Text-Aided Image Classification: Using Labeled Text from Web to Help Image Classification

As more and more multimedia data become available on the Web, mining on those data is playing an increasingly important role in Web applications. In this paper, we investigate the interplay between multimedia data mining and text data mining. Specifically, in an approach we called text-aided image classification (TAIC), we address the problem of image classification with very limited amount of labeled images and a large amount of auxiliary labeled text data. This problem is important in practice, since currently on the Web, labeled text data are usually much more than image data. To solve the problem, based on the “bag-of-words” view and the Naive Bayes classification model, we focus our attention on the estimation of the image feature distribution under given concept. We extend the Naive Bayes algorithm by considering a mapping that maps the most discriminative text features into the image feature space. This feature mapping is estimated based on the text-image cooccurrence data on the Web, acting like a bridge that connects text and image knowledge. With this process, we estimate target image feature distribution from a text model based on sufficient labeled data. Our empirical results on real world data sets show that our method makes a good approximation of the image feature distribution when trained with abundant labeled images. In the case amount of labeled images is very limited, the classification performance is improved by using auxiliary labeled text data, which shows that our method can indeed integrate text and image knowledge in a simple yet effective way.