Robust ASL Fingerspelling Recognition Using Local Binary Patterns and Geometric Features

Sign language recognition using computer vision techniques enables machines to function as interpreters of sign language while eliminating the need for cumbersome data gloves. In this paper, a robust approach for recognition of bare-handed static sign language is presented, using a novel combination of features. These include Local Binary Patterns (LBP) histogram features based on color and depth information, and also geometric features of the hand. Linear binary Support Vector Machine (SVM) classifiers are used for recognition, coupled with template matching in the case of multiple matches. An accurate hand segmentation scheme using the Kinect depth sensor is also presented. The resulting sign language recognition system could be employed in many practical scenarios and works in complex environments in real-time. It is also shown to be robust to changes in distance between the user and camera and can handle possible variations in fingerspelling among different users. The algorithm is tested on two ASL fingerspelling datasets where overall classification rates over 90% are observed.