Multimodal communication system allowing man and avatar to use voice and beck

In this research, a dialog environment between human and computers has been constructed. This dialog environment is based on the integration between verbal information through the utterances and non-verbal information by the gestures such as finger pointing. The experimental results have proved the effectiveness of this approach in terms of facilitating man-machine interaction and communication. Currently, keyboard and mouse are widely used as the interface to the electronic devices such as computers. Elderly people cannot use them very easily. In Japan, if various apparatus is connected along with the arrival of ubiquitous network society, people have to handle complicated operations. The reality is that even people exploiting such operations feel it difficult to use new ones while growing older. The environment constructed in this research allows a user to communicate by talking and showing gestures to a personified agent who is a salesman selling fountain pens in a virtual shop. A user can use his/her finger to point at a pen and ask a question such as ´Would you show me this pen?´ Through the integration of gesture interpretation and verbal information analysis, it is possible for user to communicate the pen he wants to the virtual agent. If the fingertip is too far way from the pen, it may be difficult for the agent to tell which pen the user wants. This needs the further communication through utterance or gestures or both.