Guidelines for interpretation of some common indicators of residency program performance

objective To provide national norms for indicators of residency-training program quality and information on their reproducibility. Participants The 364 residency-training programs that had 4 or more candidates take the 1989 to 1991 certifying examination in internal medicine for the first time. design Within each residency, program directorsʹ ratings of medical knowledge, certifying examination scores, and certification status (pass or fail) were available for each candidate from 1989 to 1991. Means of these data were calculated for each program for each year of the study. To provide a way of comparing an individual program with all other programs, percentiles are reported for each year. To assess the precision of the measures, generalizabilrty theory was applied and confidence intervals for all data are reported for programs of various size (1 to 25 residents taking the examination) and over the years (1 to 3). Results Over the 3 years of the study, knowledge ratings, certification rates, and composite scores declined slightly. The correlations between program ratings of medical knowledge and the composite scores ranged from .47 to .60 and certification rates ranged from .44 to .55. The confidence intervals around all of the program performance indicators are relatively large and are affected most by the number of residents in the program. There is little variability across the years. Conclusions In smaller programs the precision of the performance indicators is poor; in programs with only a few residents they are virtually meaningless. On the positive side, programs are relatively stable and aggregating indicators over years is a reasonable way to increase their precision in assessing program performance. If the goal of program evaluation is to identify programs at the extremes, especially those at the low end, then such data may help guide program directors and educators.