History Report of the Doctoral Program Evaluation


Prepared by: Deborah Feltz and T. Gilmour Reeve
Date:September 8, 2003


Fall 1995   Recognizing the need for an evaluation of doctoral programs in its discipline, president Robert Singer, created a presidential committee to develop a plan for evaluating doctoral programs (DPEC).The DPEC, with Mike Wade as chair, worked to determine the nature and scope of a doctoral program evaluation. 
September 1996   The Executive Committee, with Rainer Martens as president, moved to continue the DPEC with the charge of developing a feasible model (criteria and process) that could be put into practice to obtain rankings of graduate programs. The chair was passed on to T. Gilmour Reeve. 
September 1997   The AAKPE membership affirmed its support of a national evaluation of doctoral programs. Following the fall conference, president Jack Wilmore expanded the DPEC to 9 members to reflect the diversity of the academic discipline and to represent the universities that would have programs to be evaluated. T. Gilmour Reeve continued to chair this committee. The DPEC identified doctoral programs, quality indicators and program profiles, and an implementation plan. AAKPE provided funding for the DPEC to meet to develop plans for the doctoral program evaluation. 
May 1998   DPEC met at Auburn University to develop methods for program evaluation, including identification of quality indicators. 
July 1998   The DPEC’s report was distributed to the membership from president Jack Wilmore for discussion at the Park City conference in September 1998. 
September 1998   The DPEC presented its report at the conference. AAKPE approved the initiation of a pilot program to evaluate 20 coded doctoral programs in kinesiology. Incoming president R. Scott Kretchmar charged the committee with conducting a pilot study during the 1998-1999 year using the doctoral evaluation instrument and process developed by the DPEC with the results to be discussed at the next AAKPE meeting. All departments with kinesiology and physical education doctoral programs were to be informed of the pilot study and invited to participate. Results of the pilot study were to be used to finalize the procedures and criteria for the first national evaluation, which was conducted in the year 2000. 
A motion at the Business meeting was made to allocate funds for a new committee charged with development of an evaluation instrument and process for professional doctoral degree programs, but this motion was tabled. 
September 1999   The DPEC presented results from the pilot study. The materials for the pilot study were sent to 29 schools that had indicated an interest and willingness to participate in the pilot study. Of the 29 schools that were contacted, data were received from 20. At the Business Meeting, Academy members supported the continued involvement of the Academy in evaluation of doctoral programs in our discipline. The Committee’s charge from the Business Meeting of the AAKPE was to (a) decide on the final indicators to be used on the final form of the survey, (b) design the final form of the survey for presentation at the 2000 conference, (c) develop the final list of institutions, (d) decide on how the data would be used (i.e., result in rankings, result in tiers, be just descriptive, use weightings of indicators, etc.), and (e) review comments from respondents. Deborah Feltz was named as the new chair of the DPEC. 
January 2000   Letters and summary data were sent out to all doctoral program administrators who participated in the pilot. 
September 2000   The DPEC presented its report to the membership. The committee reviewed the indicators used in the pilot study, rejected those that were not informative, modified those that were confusing, and added some as recommended by the membership and pilot study respondents. The DPEC presented the 20 final indicators and final form of the survey to the membership for approval. Dr. Betsy Becker, a psychometrician from Michigan State University, was invited to the meeting as a consultant. The DPEC recommended that the 20 indicators be reduced to a smaller number of meaningful factors and be presented in graphic form showing "error bars" instead of ranking programs. The DPEC proposed to hold off on a decision regarding how the data from the final survey should be used until the full set of national data can be examined and presented to the membership. Discussion included whether to consider impact factors, ISI criteria, journal reputations, differentiation of types of external funding, and whether different procedures and criteria used by other agencies should be used or continue with our own criteria. The final criteria and survey were accepted and approval was given by president Gary Krahenbuhl to go ahead with data collection on a national scale. 
January 2001   The DPEC initiated the first National doctoral evaluation of kinesiology programs. President Krahenbuhl sent letters to 55 institutions that had been identified as offering doctoral programs in kinesiology, physical education, or a related field (e.g., exercise science) inviting them to participate in the AAKPE doctoral evaluation. The letter indicated that "unless approval by a vote of a majority of Academy members, outcomes will remain anonymous and confidential, though you will know your own institution’s results." Thirty-eight programs accepted the invitation and the chairpersons of 26 programs submitted complete surveys. Respondents of completed surveys were additionally requested to complete a narrative section profiling their program. This information was not included in the report because only 12 program chairs submitted information. 
September 2001   The AAKPE meeting was cancelled due to 9/11. Thus the membership was not able to act on making the results public. 
October 2001   A letter was sent from president and chair of DPEC, Deborah Feltz, to all 26 participating program administrators with the results of the national evaluation study. The letter indicated that "No data or rankings will be made public without your written approval." A report of evaluation and summary data were sent to Executive Committee members and printed on the Academy’s web site. 
December 2001   The Executive Committee recommended a change in the bylaws to make the DPEC a standing committee of the Academy and to conduct the next evaluation in 2005. The Executive Committee also voted to require that invited institutions agree to have their data made public as a condition of their participation. 
September 2002   DPEC chair, Deborah Feltz presented the findings from the national survey to the Academy. Results were presented on three faculty quality factors (visibility, productivity, and program contribution/mentoring) and four student quality factors (quality, attraction to program, recruitment, and time to degree). At the Business, the membership approved (a) the current survey to be made available for unrestricted use with institutions identified after they agree and after the membership approves by electronic ballot, (b) a 5-year review cycle, (c) a presidential committee to explore the use of a single ranking with the current data, and (d) the formation of a standing committee on doctoral program evaluation. President Richard Magill appointed Jerry Thomas to chair the presidential committee to explore the concept of a single ranking. 
November 2002   An electronic vote of the Academy membership was taken to make the current doctoral program evaluation results available for unrestricted use with institutions identified after the programs from those institutions agree. Unanimous support for this motion was given at the 2002 Business meeting of the AAKPE in Tucson, but because a quorum of Active Fellows was not present it still required vote from the full membership. From a membership of 118 active fellows, 86 approved the motion. 
January 2003   Letters were sent to the 26 participating institutions requesting their approval to release their institutional names on the AAKPE national evaluation of doctoral programs in kinesiology. Twenty-two institutions agreed to have their data made public. 
April 2003   The 22 agreeable institutions were sent summary information with bar graphs depicting the spread of seven indexes, based on a factor analysis of the original data. They were also informed that the data would be published on the Academy’s website in the same form as the graphs. 
September 2003   Deborah Feltz updated the membership on the status of the doctoral program evaluation project at the annual business meeting. Two additional programs agreed to allow the data regarding their institutions to be made public. Bar graphs for seven program indicators will be posted on the AAKPE website with the 24 out of 26 institutions listed.