Author_Institution :
Beckman Inst. for Adv. Sci. & Technol., Illinois Univ., Urbana, IL, USA
Abstract :
Summary form only given. "Multimodal" refers to the different senses (visual, audio, tactile, etc.) used in human-computer interface. "Multimedia" refers to the different ways of representing information (text, graphics, audio, images, video, etc.). A signal processing, analysis, or understanding task is called multimedia/multimodal, if it involves two or more modalities or media, interacting in nontrivial ways. We shall give an array of examples of multimedia/multimodal signal processing, analysis, and understanding, including: audio/visual speech recognition, and audio/visual emotion recognition. A stable and robust facial movement tracking algorithm is presented, which is used in both tasks.
Keywords :
array signal processing; multimedia communication; user interfaces; audio emotion recognition; audio speech recognition; facial movement tracking algorithm; human-computer interface; media modality; multimedia signal processing; multimodal signal processing; visual emotion recognition; visual speech recognition; Array signal processing; Emotion recognition; Graphics; Robustness; Signal analysis; Signal processing; Signal processing algorithms; Speech analysis; Speech recognition; Video signal processing;
