A novel hand posture recognition system based on sparse representation using color and depth images

Hand posture is a natural and effective human robot interaction way. In this paper, an user-independent hand posture recognition system using depth and color images captured from an RGB-D camera is presented. To recognize hand posture against complicated background conditions, we propose a novel method for automatic and accurate hand posture segmentation which detects the hand with Chamfer matching, tracks the hand with Kalman filter and segments the hand with region growing algorithm only in the depth space. A new hand posture descriptor invariant to scale, shift and in-plane rotation is constructed with the combination of local contour Fourier descriptor and global Bag-of-Features (BoF) descriptor based on Scale Invariance Feature Transform (SIFT). The sparse representation-based classification (SRC) is applied to perform the hand posture recognition task in the system. Experiments with a self-built large scale hand posture database collected online show the robustness and effectiveness of the proposed system.