A scalable metric learning-based voting method for expression recognition

In this paper, we propose a facial expression classification method using metric learning-based k-nearest neighbor voting. To achieve accurate classification of a facial expression from frontal face image, we first learn a distance metric structure from training data that characterizes the feature space pattern, then use this metric to retrieve nearest neighbors from training dataset, and finally output the classification decision accordingly. An expression is represented as a fusion of face shape and texture. This representation is based on registering a face image with landmarking shape model and extracting Gabor features from local patches around landmarks. This type of representation achieves robustness and effectiveness by using an ensemble of local patch feature detector at a global shape level. A naive implementation of metric learning-based k-nearest neighbor would incur a time complexity proportional to the size of the training dataset, which precludes this method being used with enormous dataset. To scale to potential larger databases, an approximate yet efficient variant scheme of ML-based kNN voting is further devised based on Locality Sensitive Hashing (LSH). A query example is directly hashed to the bucket of a pre-computed hash table where candidate nearest neighbors can be found and there is no need to search the entire database for nearest neighbors. Experimental results on Cohn-Kanade database and Moving Faces and People database show that both ML-based kNN voting and its LSH approximation outperform the state-of-the-art, demonstrating the superiority and scalability of our method.