A Scalable Reference Standard of Visual Similarity for a Content-Based Image Retrieval System

In order to develop content based image retrieval (CBIR) systems, a robust reference standard of similarity between pairs of images is required, but challenging to create given the large number of pair-wise comparisons. We demonstrated a novel method of creating one for liver tumors seen in 19 portal venous CT scans by computing image similarity from subjective ratings of attributes on single images. We gathered ratings with 6- and 9-point scales for liver lesions displayed individually (P1: ratings for 6 visual attributes) and in all 171 pair-wise combinations (P2: ratings for dissimilarity in the 6 attributes and overall dissimilarity) from 3 radiologists. We averaged readers´ ratings and fit the absolute attribute rating differences in P1 to ratings in P2. The R-squared value between pair-wise attribute dissimilarities and overall pair-wise dissimilarity was 0.65, and between a linear combination of the absolute differences of ratings for each attribute and overall pair-wise dissimilarity was 0.46. For overall dissimilarity, pairs of readers showed agreement to within 2 points in 64-84% of all ratings. Hence, this scalable method is feasible for creating a reference standard for CBIR.