METHODS FOR MRMC ROC ANALYSIS AND COMPONENTS-OFVARIANCE MODELING USING COLPOSCOPY IMAGES

In cervical cancer screening, colposcopically-directed biopsies determine disease severity and management. Multi-reader multicase receiver operating characteristic (MRMC ROC) analysis can be used to evaluate the diagnostic performance and reader variability of colposcopy. We describe the constrained one-shot variance estimate of the empirical area under the ROC curve and analyze a dataset of colposcopists reading images of the cervix. We demonstrate how to extrapolate the variance estimate to larger studies and how to estimate essential components of variance: reader, case, and an interaction component. These methods allow us to investigate the variance reduction of strategies for adding more cases. For our data (21 readers, 16 normal and 4 diseased cases), variance decreases the fastest by adding more diseased cases (short-term) or a ratio of diseased and normal cases (long-term).