Fusion of multi-modal features for efficient content-based image retrieval

The optimal combination of the outputs of multi-modal features in content-based image retrieval (CBIR) is an important task that can have a significant impact on the overall performance of the CBIR system. This problem has not received much attention from the CBIR research community, and only simple methods have been used. In this paper, we treat the problem as an information fusion problem and propose an approach that is generic and can be adapted to various features and distance measures using a small set of training images. Our approach is based on associating a fuzzy membership function with the distribution of the featurespsila distances, and assigning a degree of worthiness to each feature based on its average performance. The memberships and the feature weights are then aggregated to produce a confidence that could be used to rank the retrieved images. We describe and experiment with two distinct aggregation methods. The first one is linear and is based on a simple weighted combination. The second one is non-linear and is based on the discrete Choquet integral. The proposed fusion methods were trained and tested using a large collection of images with several low-level visual and high-level textual features. The results were compared to other methods used in typical CBIR systems.