A semantic fusion based Multimodal Interaction Panel for virtual city planning

Multimodal Interaction (MMI) in Virtual Environment (VE) has become a focus of both Virtual Reality (VR) and Human-Computer Interaction (HCI). In this paper, with reference to existing task-oriented integration algorithms, we propose a new hierarchical multimodal integration model based on semantics extracting with probability and time constraints, and on the background of virtual city planning Multimodal Interaction Panel (MIP) based on multiple metaphors is constructed by introducing daily operations of pen and paper into VE and combining spatial position trackers, microphone and pen-based sketches and gestures. A variety of common and application-oriented MMI techniques are designed and implemented, and part of their integration processes are analyzed in detail. Evaluation shows MIP significantly improved naturality and efficiency of MMI.