Automatic image tagging and recommendation via PARAFAC2

An important aspect when sharing images in social networks is the tags the images are annotated with. Another closely related problem is the ability to successfully recommend images to users. An automatic image annotation and recommendation system is proposed based on Parallel Factor Analysis 2 (PARAFAC2). Here, PARAFAC2 is applied to a collection of three matrices, namely the image-feature matrix, whose columns are representations capturing the visual appearance of images, the image-tag matrix, whose columns indicate the tags associated with each image, and the image-user matrix, whose columns identify who has uploaded or is associated to each image. PARAFAC2 is able to harness the multi-tag and the multi-user information for reducing the dimensionality of the feature vectors extracted from the images. That is, by projecting the feature vector onto the semantic space derived via PARAFAC2, a sketch (i.e., a coefficient vector of reduced dimensions) is obtained. To predict the tags to be assigned to a test image, the test image sketch is multiplied by the left singular vectors of the image-tag matrix, yielding a tag vector. Similarly, to recommend users who might be interested to a test image, the sketch is multiplied by the left singular vectors of the image-user matrix, yielding a recommendation vector. Promising results are demonstrated when the aforementioned framework is applied to an image dataset of Greek popular tourist landmarks extracted from Flickr, using a 10 fold cross-validation experimental protocol.