Multimodal natural user interaction for multiple applications: The gesture — Voice example

In this paper we present a natural user interface system. The device utilized for achieving natural interaction is the MS-Kinect which provides RGB, depth & audio signal. Our system is based on the theory of multimodal interaction and provides the ability to the user to interact simultaneously with different applications using vocal commands and gesture in conjunction. The difference of our system from other similar efforts is that it uses a multimodal interaction approach to manipulate multi-applications by employing a generic container, which runs at the background and serves as an intermediate between multimodal input and active applications running on a computer. We describe the architecture of the system based on multimodal interaction and its representative example using two PC applications.