Title :
Context-based multimodal input understanding in conversational systems
Author :
Chai, Joyce ; Pan, Shimei ; Zhou, Michelle X. ; Houck, Keith
Author_Institution :
IBM Thomas J. Watson Res. Center, Hawthorne, NY, USA
Abstract :
In a multimodal human-machine conversation, user inputs are often abbreviated or imprecise. Sometimes, merely fusing multimodal inputs together cannot derive a complete understanding. To address these inadequacies, we are building a semantics-based multimodal interpretation framework called MIND (Multimodal Interpretation for Natural Dialog). The unique feature of MIND is the use of a variety of contexts (e.g., domain context and conversation context) to enhance multimodal fusion. In this paper we present a semantically rich modeling scheme and a context-based approach that enable MIND to gain a full understanding of user inputs, including ambiguous and incomplete ones.
Keywords :
natural language interfaces; speech recognition; speech-based user interfaces; MIND; Multimodal Interpretation for Natural Dialog; context-based multimodal input understanding; conversational systems; multimodal fusion; multimodal human-machine conversation; semantically rich modeling scheme; semantics-based multimodal interpretation framework; user inputs; Cities and towns; Context modeling; Displays; History; Natural languages; Speech processing; Speech recognition; Switches; Text recognition; USA Councils;
Conference_Titel :
Multimodal Interfaces, 2002. Proceedings. Fourth IEEE International Conference on
Print_ISBN :
0-7695-1834-6
DOI :
10.1109/ICMI.2002.1166974