Lip movement and gesture recognition for a multimodal human-computer interface

This paper presents an algorithm for lip movement tracking and lip gesture recognition for the purpose of the multimodal human-computer interface (HCI) called LipMouse. This solution allows a user to work on a computer using movement and gesture made by his/her mouths only and is especially useful for severely disabled and paralyzed people. User face images are captured with a standard webcam. Face detection is based on a cascade of boosted classifiers using Haar-like features. A mouth region is located in the lower part of the face region and is used to track lip movements. Three lip gestures are recognized: mouth opening, sticking out the tongue and making the mouth into an ¿0¿ shape. Lip gesture recognition is performed by an artificial neural network and utilizes an accurate lip shape obtained by the means of lip image segmentation using fuzzy clustering.