Collaborative Learning between Visual Content and Hidden Semantic for Image Retrieval

Similarity measure is a critical component in image retrieval systems, and learning similarity measure from the relevance feedback has become a promising way to enhance retrieval performance. Existing approaches mainly focus on learning the visual similarity measure from online feedbacks or constructing the semantic similarity measure depended on historical feedbacks log. However, there is still a big room to elevate the retrieval performance, because few works take the relationship between the visual similarity and the semantic similarity into account. This paper proposes the collaborative learning similarity measure, CoSim, which focuses on the collaborative learning between the visual content of images and the hidden semantic in log. Concretely, the semantic similarity is first learned from log data and serves as prior knowledge. Then, the visual similarity is learned from a mixture of labeled and unlabeled images. In particular, unlabeled images are exploited for the relevant and irrelevant classes in different ways. Finally, the collaborative learning similarity is produced by integrating the visual similarity and the semantic similarity in a nonlinear way. An empirical study shows that the proposed CoSim is significantly more effective than some existing approaches.