Multi-Frame Sparse Feature Extraction for Lip-Reading

The features of human lip motion from video clips are extracted by three unsupervised learning algorithms, i.e., principle component analysis, independent component analysis, and non-negative matrix factorization. Since the human perception of facial motion goes through two different pathways, i.e., the lateral fusifom gyrus for the invariant aspects and the superior temporal sulcus for the changeable aspects of faces, we extracted the dynamic video features from multiple consecutive frames for the latter. The multiple-frame features require less number of coefficients for the same frame length than the single-frame static features, and also result in better recognition performance