Modality tracking in the multimodal Bell Labs Communicator

We describe our efforts in designing and building the multimodal Bell Labs Communicator system. Several innovations are necessary for generalizing the speech-only user interface and the semantic/pragmatic algorithms to handle both speech and visual modalities, including changes to the parser, semantic and pragmatic modules to integrate better the user input from different modalities within and across dialogue turns, as well as extending the concept of e-forms to include visual modality. The paper also describes the design of a natural, consistent and efficient multimodal user interface. We introduce the concept of "dominant modality": the system adaptively tracks the most probable interaction mode and suggests that modality to the user. Modality tracking provides an elegant solution to the "turn-taking" issue in multimodal spoken dialogue systems.