Intersubjectivity: the holy grail of mutual understanding?

Content-based retrieval (CBR) promises to greatly improve capabilities for searching for images based on semantic features and visual appearance. However, developing a framework for evaluating image retrieval effectiveness remains a significant challenge. Difficulties include determining how matching at different description levels affects relevance, designing meaningful benchmark queries of large image collections, and developing suitable quantitative metrics for measuring retrieval effectiveness. This article studies the problems of developing a framework and testbed for quantitative assessment of image retrieval effectiveness. In order to better harness the extensive research on CBR and improve capabilities of image retrieval systems, this article advocates the establishment of common image retrieval testbeds consisting of standardized image collections, benchmark queries, relevance assessments, and quantitative evaluation methods.