Towards recognition of facial expressions in sign language: Tracking facial features under occlusion

Deaf people use facial expressions as a non-manual channel for conveying grammatical information in sign language. Tracking facial features using the Kanade-Lucas-Tomasi (KLT) algorithm is a simple and effective method toward recognizing these facial expressions, which are performed simultaneously with head motions and hand signs. To make the tracker robust under these conditions, a Bayesian framework was developed as a feedback mechanism to the KLT tracker. This mechanism relies on a set of face shape sub-spaces learned by probabilistic principal component analysis. An update scheme was utilized to modify these subspaces and adapt to persons with different face shapes. The result shows that the proposed system can track facial features with large head motions, substantial facial deformations, and temporary face occlusions by hand.