Structured assessment format for evaluating operative reports in general surgery

Background Despite its multifaceted importance, no validated or reliable tools assess the quality of the dictated operative note. This study determined the construct validity, interrater reliability, and internal consistency of a Structured Assessment Format for Evaluating Operative Reports (SAFE-OR) in general surgery. Methods SAFE-OR was developed by using consensus criteria set forth by the Canadian Association of General Surgeons. This instrument includes a structured assessment and a global quality rating scale. Residents divided into novice and experienced groups viewed and dictated a videotaped laparoscopic sigmoid colectomy. Blinded, independent faculty evaluators graded the transcribed reports using SAFE-OR. Results Twenty-one residents participated in the study. Mean structured assessment scores (out of 44) were significantly lower for novice versus experienced residents (23.3 ± 5.2 vs 34.1 ± 6.0, t = .001). Mean global quality scores (out of 45) were similarly lower for novice residents (25.6 ± 4.7 vs 35.9 ± 7.6, t = .006). Interclass correlation coefficients were .98 (95% confidence interval, .96–.99) for structured assessment and .93 (95% confidence interval, .83–.97) for global quality scales. Cronbach α coefficients for internal consistency were .85 for structured assessment and .96 for global quality assessment scales. Conclusions SAFE-OR shows significant construct validity, excellent interrater reliability, and high internal consistency. This tool will allow educators to objectively evaluate the quality of trainee operative reports and provide a mechanism for implementing, monitoring, and refining curriculum for dictation skills.