Toward a vision based hand gesture interface for robotic grasping

The challenging problem of planning manipulation tasks for dexterous robotic hands can be significantly simplified if the robot system has the ability to learn manipulation skills by observing a human demonstrator. Toward this goal, we present a novel computer vision based hand posture recognition system to serve as an intelligent interface for skill transfer in robotic manipulation. We use the inner distance shape context (IDSC) as a hand shape descriptor to capture variations in the hand state (open or closed) under large in-plane rotations and considerable out-of-plane rotations. The proposed technique is further examined in applications involving grasp recognition and gesture based communications. The experiments show that the proposed approach can be generalized to recognizing a selected taxonomy of grasp types. At present, skin color is used to segment the hand region from the scene, but this method has its own limitations. We show preliminary results suggesting that the IDSC can be used to segment parts of the articulated object, including segmenting the hand from the human body silhouette without using skin color information.