The in-training examination in internal medicine:: Resident perceptions and lack of correlation between resident scores and faculty predictions of resident performance

PURPOSE: We sought to survey residents’ perceptions regarding the In-Training Examination in Internal Medicine and to assess the ability of faculty members to evaluate the knowledge base of internal medicine residents. SUBJECTS AND METHODS: Residents were asked about the perceived utility of the In-Training Examination and related self-directed educational activities. Residents predicted their own performance on the examination (into upper, middle, or lower tertile). Faculty predicted housestaff’s scores, and residents predicted the scores of interns. RESULTS: Most residents (35/36; 97%) believed that the examination was useful, and 91% modified their study habits or clinical rotation schedule based on its results. Approximately half of the residents accurately predicted into which tertile they would score. Faculty predictions of resident performance on the examination were accurate 49% of the time, and resident predictions of intern scores were accurate 38% of the time. The sensitivity of a lower-tertile prediction by faculty was 34%, with a specificity of 90%. The sensitivity of a resident prediction of a lower-tertile intern score was 15%, with a specificity of 98%. Both faculty and residents were more likely to overestimate than underestimate examination scores. CONCLUSION: Residents believe that the In-Training Examination is useful and frequently initiate educational interventions based on results. Faculty and residents lack the ability to evaluate accurately the knowledge of trainees that they supervise. In particular, both groups may be unable to identify trainees who are deficient in this element of clinical competence.