Integration of spatial relationships in visual language model for scene retrieval

In this paper, we describe a method to use a graph-based language modeling approach for image retrieval and image categorization. We first mapped image regions to induced concepts and then spatial relationships between these regions to build a graph representation of images. Our method allows to deal with different scenarii, where isolated images or groups of images are used for training and testing. The results obtained on an image categorization problem comprising of 3849 images from 101 landmarks of Singapore show that (a) the procedure to automatically induce concepts from an image is effective, and (b) the use of spatial relationships, in addition to concepts, for representing an image content helps improve the classifier accuracy. This approach is the first one, to our knowledge, to present a complete extension of the language modeling approach from information retrieval to the problem of graph-based image categorization and retrieval.