Generating Natural Language Annotation from Video Sequences Taken by Handy Camera

In this paper, we propose a method for generating natural language annotation automatically from video sequences taken by handy camera. In general, video image and natural language are totally different form of information. The main challenge is how to bridge the gap between them and to translate one to another. In our method, an appropriate predicate is selected for expressing video contents by combining semantic features extracted from video sequences. It is, however, difficult to select most appropriate words for expressing the contents. Therefore we construct a concept hierarchy of motions and situations to select verbs from coarse to fine. We perform some experiments to test the effectiveness of the proposed method, and confirmed appropriate annotations are generated.