A multi-modal gesture recognition system in a Human-Robot Interaction scenario

Recognition of non-verbal gestures is essential for robots to understand a user´s state and intention in a Human-Robot Interaction (HRI) scenario. In this paper a multi-modal system is proposed to recognize a user´s hand gestures and estimate body poses from the robot´s viewpoint only. A range camera is employed to derive the depth data at a high frame rate. Depth data is useful for image segmentation, objects detection and localization in 3D spaces. A pair of stereo cameras is used to sense the user´s head gestures and eye gaze direction, which provide useful information about the user´s attention direction. Both hand shapes and hand trajectories are recognized. Full configurations of body poses are estimated using a model-based algorithm. Poses are tracked by a Particle Filter method, and refined by a gradient-based searching method in the neighborhood of the particles which have top largest weights.