Simultaneous segmentation and recognition of hand gestures for human-robot interaction

Gestures are a natural form of communication between people that is being increasingly used for human-robot interaction. There are many automatic techniques to recognize gestures, however, most of them assume that gestures are already segmented from continuous video, clearly, this is an unrealistic scenario for human-robot interaction. For instance, when commanding a service robot the agent must be aware at any time of the world (e.g., via continuous video) and ready to react when a user gives an order (e.g., using a gesture). In this paper we propose a method for addressing both tasks, segmentation and recognition of gestures, simultaneously. The proposed method is based on a novel video-stream exploration scheme called multi-size dynamic windows. Several windows of different sizes are dynamically created, each window is classified by a Hidden Markov Model (HMM). Predictions are combined via a voting strategy and eventually the endpoint of a gesture is detected (segmentation). At that moment the method recognizes the gesture that has been just performed using a majority vote decision (recognition). The proposed method is intended to command a service robot by capturing information of user movements with a KinectTM sensor. We evaluated experimentally the proposed method with 5 different gestures suitable for commanding a service robot. Experimental results show that up to 82.76% of the gestures are correctly segmented. The corresponding recognition performance was of 89.58 %. We consider that this performance is acceptable for certain human-robot interaction scenarios.