Title :
Integration of speech and vision using mutual information
Author_Institution :
Media Lab., MIT, Cambridge, MA, USA
Abstract :
We are developing a system which learns words from co-occurring spoken and visual input. The goal is to automatically segment continuous speech at word boundaries without a lexicon, and to form visual categories which correspond to spoken words. Mutual information is used to integrate acoustic and visual distance metrics in order to extract an audio-visual lexicon from raw input. We report results of experiments with a corpus of infant-directed speech and images
Keywords :
audio-visual systems; image processing; learning systems; natural languages; speech processing; acoustic distance metric; audio-visual lexicon extraction; automatic continuous speech segmentation; co-occurring input; infant-directed image corpus; infant-directed speech corpus; mutual information; speech-vision integration; spoken input; spoken words; visual categories; visual distance metric; visual input; word boundaries; word learning; Acoustic signal detection; Computational modeling; Image segmentation; Laboratories; Learning systems; Mutual information; Natural languages; Shape; Speech; Streaming media;
Conference_Titel :
Acoustics, Speech, and Signal Processing, 2000. ICASSP '00. Proceedings. 2000 IEEE International Conference on
Conference_Location :
Istanbul
Print_ISBN :
0-7803-6293-4
DOI :
10.1109/ICASSP.2000.859317
