Combining multiple representations on the TRECVID search task [video retrieval system]

This paper presents a (preliminary) analysis of the evaluation results obtained on the TRECVID 2003 search task. We study in particular the effects of combining multiple representations on retrieval: multiple representations of video content (speech and visual) and of the user information need (multiple visual examples). We conclude from our multi-modal retrieval experiments the following working hypothesis: even though the automatic speech recognition run is usually better than the visual run, matching against both modalities ensures robustness against choosing the wrong content representation. For the same reason, using multiple visual examples to represent the user information needs is preferable over using a single designated example only.