Cross modal metric learning with multi-level semantic relevance

The Mahalanobis metric learning is an effective tool for constructing semantic consistent distance among data in single modal data analysis. However, distance metric learning is a more challenging issue for cross modal data, where less attention has been paid in previous studies. In this paper, we propose Cross mOdal Large mArgin metric leaRning (COLAR) with multi-level semantic relevance. With large margin principle, we model different levels of the semantic relations across modalities, e.g., the one-to-one correspondence and intra-class relation, while traditional correlation learning approaches (such as CCA and its variants) can only handle the one-to-one correspondence or treat them indiscriminatively. As a result, the distances of multi-level relevance among cross modal data are optimized based on a regularized learning framework. Promising performance is achieved on cross modal retrieval, i.e., image-to-text retrieval and text-to-image retrieval.