Retrieving video shots in semantic brain imaging space using manifold-ranking

In recent two decades, a large amount of effort has been devoted to content-based video retrieval (CBVR), which aims to manage large-scale video databases in an effective way based on visual features such as color, shape, texture, and motion. However, the performance of CBVR systems is still far from satisfaction due to the well-known semantic gap. In order to alleviate the problem, this paper proposes a novel retrieval methodology using semantic features derived from brain imaging space (BIS) that reflects brain responses and interactions under natural stimulus of video watching. A mapping from visual features to semantic features in BIS is built through Gaussian process regression. A manifold structure is then inferred where video key frames are represented by mapped feature vectors in BIS. Finally, the manifold-ranking algorithm concerning the relationship among all data is applied to measure the similarity between key frames. Preliminary experimental results on the TRECVID 2005 dataset demonstrate the superiority of the proposed work in comparison with traditional methods.