Object image retrieval by exploiting online knowledge resources

We describe a method to retrieve images found on Web pages with specified object class labels, using an analysis of text around the image and of image appearance. Our method determines whether an object is both described in text and appears in a image using a discriminative image model and a generative text model. Our models are learnt by exploiting established online knowledge resources (Wikipedia pages for text; Flickr and Caltech data sets for image). These resources provide rich text and object appearance information. We describe results on two data sets. The first is Bergpsilas collection of ten animal categories; on this data set, we outperform previous approaches (Berg et al., 2006; Schroff et al., 2007). We have also collected five more categories. Experimental results show the effectiveness of our approach on this new data set.