A consumer video search system by audio-visual concept classification

Indexing and searching the massive amount of consumer videos in the open domain is increasingly important. Due to the lack of text descriptions as well as the difficulties in analyzing the content of consumer videos, little work has been conducted to provide video search engines in the consumer domain. In this paper, we develop a content-based consumer video search system based on multi-modal concept classification. The system supports the query-by-example access mechanism, by exploiting the query-by-concept search paradigm underneath, where online concept classification is conducted over the query video by integrating both visual and audio information. The system adopts an audio-visual grouplet representation that captures salient audio-visual signatures to describe the video content for efficient concept classification. Experiments over the large-scale Columbia Consumer Video set show the effectiveness of the developed system.