Evaluating human observer performance for list mode PET using the bootstrap method

In an LROC (localization receiver operating characteristic) study, a human observer is presented a number of images, of which some contain a signal within a certain region and some do not contain any signal The observer is asked to give a confidence rating as to the probability of the presence of a signal, and to give the most probable location of the signal. A LROC curve can then be fitted. The area under the LROC gives an indication of the observer performance for the given set of images. A large number of images is needed for a human observer study, e.g. 100 for the study itself and 50 as a training set. The simulation or measurement of all of these images is very time consuming and/or impractical. In a previous paper, we proposed the bootstrap method as a means to create a number of list mode files from one simulation, and showed that these files can be used to evaluate the LROC performance of a CNPW numerical observer. In this paper, we investigate whether the bootstrapped images can be used to evaluate human observer performance. We found that human performance was inferior on bootstrapped images compared to noisy images. However, by substituting for the noisy background with a smoothed version of this background, human observer performance becomes comparable to performance with noisy realizations of the images. We can therefore conclude that bootstrapped images, with a smoothed background, can be used as an alternative for noise realizations for human observer studies.