Multimodal Image Retrieval Based on Annotation Keywords and Visual Content

Currently, most image retrieval systems use either purely visual features or textual metadata associated with images. They have advantages and disadvantages respectively. To overcome their drawbacks and improve the performance without sacrificing the efficiency, we propose the stepwise refinement multimodal image retrieval scheme based on annotation keywords and visual content, which can benefit from the strength of text- and content-based retrieval. The system starts query triggered by some keywords, and further refines the retrieval result based on blobs and regions information. The first step is to complete semantic filtering with weakening visual content, and the second step mainly considers existence and dependence of blobs, and the last step is to quantify the similarity in distribution and layout of visual content between the query image and candidate images by considering the weights of regions. The experiments show that proposed system outperforms the traditional image retrieval systems.