Integration of gaze and gesture detection in nature language instructing of robot in an assembly scenario

Author

Zhang, Jianwei ; Baier, Tim ; Hueser, Markus

Author_Institution

Dept. of Comput. Sci., Hamburg Univ., Germany

fYear

2002

fDate

2002

Firstpage

241

Lastpage

246

Abstract

We present the development of and experiments with a robot system showing multimodal interaction capabilities. We focus on the understanding of human instructions in natural language by integrating gaze and pointing hand gestures. A typical application of such a system is interactive assembly. A human communicator sharing a view of the assembly scenario with the robot instructs the latter by speaking to it in the same way that he would communicate with a child. His instructions can be under-specified, incomplete and/or context-dependent. By detecting gaze and gesture information of the instructor, which are described in two sections of the paper, the ambiguities in the interactions can be resolved in a natural manner. Finally, we outline a list of future topics for extending our research.

Keywords

Kalman filters; assembling; feature extraction; gesture recognition; interactive systems; natural language interfaces; robot programming; robot vision; 3D facial feature model; Kalman filter; assembly scenario; feature detection; gaze detection; gesture detection; human communicator; interaction ambiguities; interactive assembly; multimodal interaction capabilities; nature language instruction; pointing hand gestures; robot; stereo vision system; template matching process; two-arm robotic system; Aircraft; Assembly systems; Human robot interaction; Humanoid robots; Intelligent robots; Natural languages; Robot sensing systems; Robotic assembly; Robotics and automation; Speech recognition;

fLanguage

English

Publisher

ieee

Conference_Titel

Robot and Human Interactive Communication, 2002. Proceedings. 11th IEEE International Workshop on

Print_ISBN

0-7803-7545-9

Type

conf

DOI

10.1109/ROMAN.2002.1045629

Filename

1045629