ERIC ED426655: Consumer Evaluation of Educational Programs: Using Questionnaires Completed by Alumni


0 Downloads 0 Views

ED 426 655




HE 031 804

Copeland, H. L.; Hewson, M.; Weiker, G. Consumer Evaluation of Educational Programs: Using Questionnaires Completed by Alumni. 1998-04-00 8p.; Paper presented at the Annual Meeting of the American Educational Research Association (San Diego, CA, April 13-17, 1998) . For a related document, see HE 031 805. Speeches/Meeting Papers (150) Research (143) Reports MF01/PC01 Plus Postage. Evaluation Methods; *Graduate Medical Education; *Graduate Surveys; Higher Education; Participant Satisfaction; Program Evaluation; Teaching Hospitals *Cleveland Clinic OH

ABSTRACT This paper describes a method used to evaluate 48 medical residency programs at one institution. It presents the background rationale, logistics, and results obtained by surveying alumni, in the context of their current practice, for relevant data on the value of their training experience. Surveys were sent for 3 years to all alumni of Cleveland (Ohio) Clinic residencies who had graduated three years earlier. A total of 187 (28.4 percent) of alumni responded to the survey. Consistency of results across the 3 years was found. Ninety-six percent of respondents replied that they would recommend their residency program, noting such positive qualities as the excellent teaching, knowledge and skills of the staff, and the variety of experiences and patients. Suggestions for improvement centered on learning and practice of primary or community/outpatient care, management or business skills, computer uses in medicine, and improved didactics. Qualitative content analysis of written comments also indicated positive evaluations of respondents' residency experiences, and supported suggestions for more business training and practice and for more responsibility given to residents. The methodology is seen as lending itself to direct program improvement on a yearly basis. (Contains 10 references.) (DB)

******************************************************************************** * * Reproductions supplied by EDRS are the best that can be made * * from the original document. ********************************************************************************

Consumer Evaluation of Educational Programs: Using Questionnaires Completed by Alumni

Presented at a round-table discussion at the American Educational Research Association Annual Conference, April 1998, San Diego, CA

Copeland, H. L., Ph.D., Hewson, M., Ph.D., and Weiker, G., MD Education TT31, 9500 Euclid Ave Cleveland Clinic Foundation Cleveland, OH 44195





Office of Educational Research and Improvement


2I/This document has been reproduced as received from the person or organization originating it. 1:1 Minor changes have been made to improve reproduction quality.

Points of view or opinions stated in this document do not necessarily represent official OERI position or policy



Consumer Evaluation of Educational Programs. Using Questionnaires Completed by Alumni

Abstract Curriculum development and implementation is an iterative process requiring ongoing feedback on the program goals to appropriately train and prepare residents to become excellent physicians within their chosen area. It is necessary to assess the adequacy of the programs in matching the needs of current and future doctors within this changing climate of medicine. Therefore, an evaluation of multiple residency programs requires a variety of creative and systematic approaches. We propose one method of accomplishing these tasks. Presented here are the background rationale, logistics, and results obtained from surveying alunmi for relevant data on the value of their training experience in the context of their current practice.

Objectives/Purposes: An evaluation of forty-eight approved residency programs requires a variety of creative and systematic approaches. Indeed this call for program evaluation is "a professional responsibility to appraise the quality of . . . programs, and . . . constantly seek ways of improving that quality." (Cronbach and others 1980 as quoted in Worthen and Sanders, 1987) A formal definition of evaluation provided by Kosecoff and Fink (1982) is that it is a set of procedures to assess a program's worth and to provide information about its objectives, goals, processes, effects, and costs. Through this process of evaluation educators are more able to minimize the chance of making ineffective or faulty changes and demonstrate the benefits of effective changes thereby justifying expenditures. (Stufflebeam & Shinkfield, 1985 as quoted in Worthen and Sanders, 1987). Curriculum development and implementation are iterative processes requiring ongoing feedback on the program goals to appropriately train and prepare residents to become competent and productive physicians within their chosen area. Additionally, it is necessary to assess the adequacy of the programs in matching the needs of current and future doctors within this changing climate of medicine. We propose a method of measuring the success of residency programs and eliciting suggestions for improvement. A key component of this method is transferring these suggestions and feedback to the program directors and key curriculum and program implementors. The critical focus is the creation of a systematic method of both collecting, distributing and using the data. Presented here are the background rationale, logistics, and results obtained from surveying alumni for relevant data on the value of their training experience in the context of their current practice.

Perspective/Theoretical Framework: Our goal was to gather summative information by contacting alumni across all the disciplinary training programs using a written survey. Furthermore, we wanted to ensure that this data was given to all invested parties and could be acted upon. The survey elicited alumni's perceptions on the value of specific aspects, their opinions on the programs' success in achieving its objectives, and their suggestions for improving the program. All residency programs were included




in the sample in order to gain a broad overview of residency training in its entirety and assess global goals of providing training which enables residents to become competent and productive physicians within their specialities. By contacting alumni of all programs, comparisons could be made between the various programs and between different years for tracking. This broad view was deemed an appropriate mode for assessing the overall goals and directions of the department of graduate medical education which oversees numerous residency programs. Traditionally training programs are evaluated by gathering information from current residents as they complete various components and from graduating residents in the form of an exit interview. We decided that a valuable addition to these data sources would be the view from alumni; an ideal group to offer feedback on the relevance of their training from the perspective of being employed in the "real world" environment, experiencing the daily activities for which they were trained. Indeed this technique has been previously used by the clinic and other programs and provides a method of obtaining data and suggestions on which to modify and improve programs (Vanek, 1989; Parrino, 1994; Cantor, 1993, Dale, 1991). A span of three years of work experience was deemed an adequate time for alumni to be comfortable in their new position and have experienced the demands of current practice, yet still be within a time period where their training experience was fresh in their minds. The data collection instrument was developed to assess not only the goals of the programs of three years prior, but also the adequacy of the training for confronting current issues in medical care and practice. The questionnaire items reflect the current needs of practicing physicians based on a literature review, including such sources as Kantor and Griner (1981) and Eggert and Parkinson (1994). Methods:

In 1995, 1996 and 1997 surveys were sent to those alumni of all Clinic residency programs who graduated three years prior. Surveys were mailed one time each year from the Office of Alumni Affairs and included addressed, stamped return envelopes. These years of data served not only to accomplish the current evaluation efforts, but also provide for a strong baseline on which to assess future changes. The survey consisted of thirty-three rating scale items and three open-ended items. The thirty-three items were organized in three broad categories, two of which were rating aspects of the residency program and rating the preparation provided by the program in certain areas. Both item sets used a five-point likert scale from "poor" to "excellent". The additional item category was "amount of exposure to the following components of your training"; rated on a four-point opinion scale of "don't know", "too little", "appropriate amount", and "too much". The openended questions asked for names of clinical teachers that stood out in memory and their approaches or skills, comments on recommendations of their program to others, and suggestions for improvements of their programs. Self-completed demographic information was requested including primary field of practice, years of training, training program area, current practice type, and current state or country in which practicing. Survey items that held particular weight in terms of alumni data were rating the "relevance of my training program to my current practice" and the "adequacy of my training program for my current practice". Example items chosen to assess their training in reflection of changing trends in medicine included judging the amount of exposure to "general (primary care)



emphasis in my training" and "specialty (tertiary care) emphasis in my training". Additional examples of items that assess their training program in terms of current medical trends are how well your program prepared you in... "coordinating patient care with community services/ resources", "referring patient expeditiously", "using computerized information", and "developing effective interpersonal skills with patients and other professionals". Data analysis included both descriptive and inferential statistics. Descriptive statistics were used to assess the responses for each item and the characteristics of the return sample. Responses of "don't know" were not used when calculating the mean. Comparisons between programs were done for those programs receiving more than ten responses through computation of t-tests using the three composite variables of items within a category. Differences across the years for the two programs with large responses were analyzed through MANOVA using the three composite variables. Results:

A total of 187 (28.4% response rate) alumni over the three years responded to the survey of their residency program. Though our small response rate does limit the generalizability and use of our data, we feel that the consistency in results across the three years demonstrates validity of the results. The majority of the respondents reside in Ohio (29.9%) or the eastern United States (32.6%), practice in fee for service groups (42.8%) or/and academic settings (27.8%), and had graduated from a medicine (40.1%) or surgery (34.2%) training program. A shift in practice setting away from "fee for service group" was noticed in 1994 (drop from 33 to 14), reflecting current trends in medicine. Responses were received from graduates who reside across the world and practice in a variety of settings and specialities. Ninety-six percent of the respondents would recommend their residency program. The top qualities mentioned by all of the alumni were the excellent teaching, knowledge and skills of the staff, the variety of experiences and patients, and the good learning environment. Overall there are no statistically significant differences among the three years of alumni responses nor between medicine and surgery alumni responses. All alumni's overall mean scores for each of the rating items and their written responses and suggestions were very similar across the years. Analysis of those items rated by alumni over both years as "excellent" or better than "good" and an "appropriate amount" provides information on which specific aspects of the residency programs worked exceptionally well. Respondents felt that their training experience was highly relevant and adequate to practice (4.5, sd=.68 relevance, 4.6, sd=.67 adequate). They felt that the variety of patients, level of supervision, their work load, their level of responsibility, and amount of teaching by staff were "just right". More variance was noticeable in the exposure to tertiary and primary care; the ratio of tertiary to primary care was approximately equivalent but somewhat weighted toward specialty practice. They also felt well prepared for board exams (4.5, sd=.74). Additionally, alumni rated the overall quality of the program and the clinical skills of the teaching staff as excellent (4.6, sd=.57 quality & 4.7 sd=.57 clinical skills). As further commendation for the teaching staff, alunmi named more than 180 individual staff members as excellent clinical teachers who stood out in their memory and provided examples and reasons for their choices. Suggestions for improvements given by respondents and identification of those items which did not receive the highest ratings were useful. Suggestions for improvement centered



around learning and practice of primary or community/outpatient care, management or business skills, computer uses in medicine, and improved didactics. The one item which alumni felt they received only fair preparation in (2.0, sd=1.21) was managing business aspects. These results provided confirmation to our program administrators that recent programs' developments are indeed congruent with perceived needs specified by surveyed alumni. Some of these developments include the following activities: the creation of a primary care track which incorporates training in community-based medicine; the implementation of courses for residents to learn about cost containment, resource utilization, contracting, and risk management; ongoing faculty development programs; and the requirement for all senior residents to engage in research activity. Though the quantitative analyses of the rating items were consistent, support for a solid baseline is further demonstrated through consistency in alumni's written comments. Qualitative content analysis on the comments from three open-ended questions was done within each question. Six consistent categories were mentioned under support of excellent clinical teachers. These included use of interactive teaching techniques such as case-based approach, rolemodeling, being spontaneous, verbalizing techniques, and teaching throughout a procedure. Additional categories were conveying a strong knowledge base both in basic science and clinical skills/medical approach, being a good discussant and providing explanations, providing a supportive learning environment and being committed to teaching, motivating residents, and teaching compassion and good patient approach. The "do you recommend your program" question elicited comments similar to items on the survey, such as the variety of experiences and patients, the teaching of clinical faculty, and the well-rounded good emphasis of their program which held high expectations. A concern commonly mentioned in all residency programs of the balance between service and training was iterated here. For suggested improvements alumni felt that there should be more business training and practice, more responsibility given to residents, and more or better teaching from more of the staff. Evaluation data were provided to the chairs of the Division of Education, the Department of Graduate Medical Education, and the Office of Alumni Affairs. These results were further disseminated to program directors, alumni and residents, and a general report was published in a faculty newsletter. It was determined that rather than presenting only the ratings on survey items, written comments would have the most impact and provide the most substantial information on which to base program improvements. Therefore, comments from alunmi which were specific to their training program were forwarded to the directors of all training programs. These were shared with the chairs of education and graduate medical education but not with chairs within the departments. It was felt they would then be received by the stakeholders most directly involved in implementation decisions. Furthermore, a letter commending an individual staff for being mentioned by name was mailed to every staff that alumni listed as excellent clinical teachers. This letter was shared with the chairs of the staff's department to highlight the institution's support for education and appreciation of the extra effort put forth in this cause. Program directors and administrators of the residency programs use multiple sources of data on which to base their decisions. The alunmi data provide them with a method to validate results from other sources. Additionally, interpretation of the alumni data must involve use of comments from current residents and perspectives of faculty within the program. When the results from alumni are disseminated, a discussion of the low response rate and interpretation in light of program self-collected data is included. One of our residency programs felt that more



detailed information can be obtained from alumni and also has developed and implemented a survey. It is because we systematically provide both the quantitative summaries and written comments with notes on ensuring validity that our administrators are able to act upon these results and use them in their decision making about the residency program.

Educational Importance: The results of this evaluation were very useful in not only assessing the value of the residency training programs for medical practice after training, but also for verifying the appropriateness of changes made in training programs to match the needs of contemporary medical practice. Furthermore the results provided direction for future improvements. The two important aspects of this study lie within the redistribution of the results or closing of the loop and with the collection of a baseline on which to analyze future shifts in program implementation. The use of alunmi evaluations is not new and has been shown to provide a helpful source of information from a much needed perspective of the user. What we have hoped to demonstrate here is the deliberate process of feeding this information directly to the individuals who develop and oversee the program. Through an annual collection of this information a very steady source of data can be produced on which to judge program changes and policy shifts. This evaluation format is important as an overview of the quality of an entire residency program and as a method of assessing current needs. Assessments will continue annually and will aid in maintaining the quality of all residency programs. Additionally, as the trends in medicine change, the survey will be appraised and modified in order to continually collect relevant information concerning the current practice needs.




Cantor, J. C., Baker, L. C., & Hughes, R. G. (1993). Preparedness for practice: Young physicians' views of their professional education. JAMA, 270 (9), 1035-1040.

Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R. D., Hornik, R. C., Philips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.

Dale, D. C. (1991). Alumni surveys for evaluation of innovations in medical education. Journal of General Internal Medicine, 6, p. 587. Eggert, R. W. & Parkinson, M. D. (1994). Preventive medicine and health systems reform: Improving physician education, training, and practice. JAMA, 272 (9), 688-693.

Kantor, S. M & Griner, P. F. (1981). Educational needs in general internal medicine as perceived by prior residents. Journal of Medical Education, 56, p. 748-756.

Kosecoff, J. & Fink, A. (1982). Evaluation basics: a practitioner's manual. Newbury Park, CA: Sage.

Purim, T. A. & Kern, D. C. (1994). The alumni survey as an instrument for program evaluation in internal medicine. Journal of General Internal Medicine, 9, p. 92-95. Stufflebeam, D. L. & Shinkfield, A. J. (1985). Systematic evaluation. Boston, MA: KluwerNijhoff. Vanek, E. P., Michener, W. M., & Melia, M. (1989). A survey of Cleveland Clinic trainingprogram alumni. Cleveland Clinic Journal of Medicine, 56, 167-173.

Worthen, B. R. & Sanders, J. R. (1987). Educational evaluation: Alternative approaches and practice guidelines. New York, NY: Longman inc.



FAX 216 444 1162 ERIC HE 3:57PM





a 003


U.S. Department of Education Oftke of EducationalResearch and Improvement (OERI) National Library Of Education (NLE) Educational Resources Inforrnation Center (ERIC)

REPRODUCTION RELEASE (specific Document)


Consumer evaluation of educational programs: using questionnaires

completed by alumni

AVOW** Copelami, R. L., gauscm, Publication Date:

Corporate Source:

April 1998

Cleveland Clinic Foundation IL REPRODUCTION RELEASE:

hi order to disseminate es wIdely es possible am:11y and significant materfala of Interest i the educational edenNUMIty. documents announced in the

mord* abstract Journal of the ERIC system, Rea:MASS In gdilcatIon (fiE), are 'Ay* made available to veers in microfiche, reproduc.ed paper copy, and electronic media, end sold through the ERIC Deeurnent ReprodUcdon Service MORS). Cradit is given to the soUrCe of each document, and, If reproduction release Is granted, one of the following nodose le axed to the document. 2.04,1 le ropftwofth and Nitnarparatre Ina iclenaSed document_ please CHECK ONE of the following three options add skin at the bottom

of the page. TrmnimeNsuctimrsinawnsimgwes.

ems to a Level

Tna IUfllPII ededer mama Dalebi bap IA Inld 41.11111111 28 IIMUMeifill

The aelfInleetleeer alere1 woe WO es emend Pe all Level 2A deormena







460' se°










imeel 1

Level 213


et.* h., ft...11110E11 WNW.. panntRIT5 rePreduelinfl CIS elillatrallnleti POWIVSNI a *My ERIC MINNl mem aimeanIc) arid popsr copy.

Gem nem far Lad 2A ndme. amide: reamdtallan

end rammtnelon in Orden. sna In Nem* wile

OMR rage fef Level 2I1 releuee, penyditna

fel2ftelud5an end dIvegnNsdan In nls only

far MC en:n1wd anNatIon seasoner' onfa

Damimenta 1,01 Da preceased as Iddlaated parAded tipertreenInver, Pewit& If penenelan feHangn N Mad, Ma no Dull dianan. deelmenw will es preceases et Level 1.

I hereby grant to the Educatrortel Resources InIonnatlon (Ambit (ERIC) nonerctushre permission to reproduce and disseminate this document as Indicated above, Ractrodudide from the ERIC micretIche or electronic media by persons other then ERIC employees and Ps system contfltaNs mufti; permission from tne copyright Wrier. Ercoston is made faf flOn4)111iff repreclurdSon by abrades and other service agencies to satisfy information needs of odwootom In Poirponzo to egeorefa Inquiriea

Sign heret-P



Freese rernadonagerefasc

L.A.. 49

Liesel Copeland/Ed.


Cleve/a d CT1ç Poundattefti. Educa-eion TT-33 0 500 :Its elid Ave , Cleveland 08 /1

10 S

Research FelloaL


rilk) bA4.512A1.3._ 2104.6A-402_, 12/R/cOt raft$VIVISeacmtr.refFAIT rowan