Multi-Modal Mining in Web image retrieval

The associations between different modalities of Web images could be very useful for Web image retrieval. In this paper, we investigate the multi-modal associations between two basic modalities of Web images, i.e. keyword and visual feature clusters, by data mining technique. The association rule crosses two modalities, in which the antecedent is a single keyword and the consequent is several visual feature clusters. A customized mining process is provided to mine such special multi-modal association rules. The multi-modal association rules are obtained offline based on the existing inverted file and utilized online to automatically integrate the keyword and visual features for Web image retrieval. The experiments are carried out in a prototype system for Web image retrieval, and the results show the effectiveness of the mined multi-modal association rules.