Construct validity of an objective assessment method for laparoscopic intracorporeal suturing and knot tying

Background The ideal objective assessment method for laparoscopic technical skills is difficult to achieve in the operating room. Recent “VR2OR” studies have used a blinded, 2-reviewer error-based video tape analysis for intraoperative performance assessment. This study examines the validity of this methodology applied to laparoscopic intracorporeal suturing and knot tying. Methods Four groups of subjects—experts (EX), surgery residents trained to expert criterion levels using simulation (TR), surgery residents receiving no supplemental training (NR), and medical students receiving simulation-based training (MS)—performed the fundal suturing portion of a laparoscopic Nissen fundoplication and were video-recorded for analysis. Two separate groups of surgeon reviewers (K.V.S. + M.B.; I.-P.H. + A.G.) were trained to evaluate laparoscopic suturing and knot tying performance using specific metrics. Subjectsʹ operative performance was assessed by reviewers blinded to their training status and scored using an error-based, step specific scoring system to an inter-rater agreement of 80% or greater. Three primary performance measures were assessed: time, errors, and needle manipulations and comparisons between groups were made using a 1-way analysis of variance (ANOVA) with post-test. Results A total of 40 fundal sutures (10 in each group) were scored by 2 separate rater groups with inter-rater agreement consistently greater than 80%. Inter-rater agreement was highest with the EX group (91%, range 76%–100%) and lowest with the NR group (85%, range 81%–98%). On average, the EX group significantly outperformed the other groups with regards to time (P <.0001), errors (P <.002), and needle manipulations (P <.01). Performance of the TR group was comparable to the EX group with regards to errors and manipulations (P = not significant [NS]), and outperformed the NR and MS groups with regards to time (P <.05 and P <.001). Performance between the NR and MS groups were similar for all 3 measures. Conclusions This assessment method demonstrates discriminative validity. Time appears to be the most sensitive indicator of skill level, as significant differences between EX, TR, and NR/MS groups were seen. The methodology is transferable across different reviewers and is acceptable for high-stakes assessment.