Problems in evaluating image display systems

The results of an evaluation of a commercial teleradiology system are presented. The efficacy of interpretation of skeletal radiographs from a digital image display on a CRT screen and from a plain film and light box station is compared. 122 single musculoskeletal radiographs were interpreted by four radiologists in the digital and analog format. Normal, subtle, and nonsubtle skeletal abnormalities were used. The digital image was interpreted on a CRT screen with a resolution of 1024×840×12 bits. Interpretation sessions simulated the actual clinical environment. Receiver operating characteristics (ROC) curves were generated to compare sensitivity and specificity over the full range of true positive and false positive values. The areas under the ROC curves have been shown to be a valid method of comparison between systems. After the study was performed and the results analyzed, the study was reviewed for potential problems with this type of evaluation. Some of these problems are discussed