Multi-modal output composition for human-computer dialogues

The authors discuss the development of a prototype multimodal human-computer interface system, called CUBRICON, which is a research platform for investigating how knowledge-based approaches can best be applied to enhance human-computer interfaces. Four critical features of such an interface are introduced along with CUBRICON´s initial approach to achieving them. Several knowledge sources have been defined within CUBRICON. These serve as the basis for CUBRICON´s multimedia output planning process. These knowledge sources include: characteristics of the information, a model of the ongoing discourse, a user/task model, and a knowledge base of task domain and related interface information. CUBRICON´s output planning process includes selection of appropriate media/modalities, selection of ancillary relevant information, determination of whether display resources are available, modification of resources or information to be expressed (if necessary), modification of selected output media/modalities (if necessary), and composition of the output. Examples are given for the application task domain of military tactical air control