Multi-modal label propagation based on a higher order similarity matrix

A novel method is introduced for label propagation on similarity tensors. The proposed method operates on data with multiple representations. A higher order similarity matrix is constructed for describing the relationship between the data representations in different modalities. Then, label propagation is performed on the above mentioned similarity matrix, by extending the state of the art label propagation method with local and global consistency to the case of higher order similarity graphs. The evaluation of the proposed method was performed on two classification tasks: person recognition on facial images extracted from three stereo movies and human action recognition on two data sets consisting of videos downloaded from YouTube. Experimental results showed that the proposed label propagation approach achieves either competitive or better classification accuracy from the state of the art in all classification tasks.