A deep-learning approach to facial expression recognition with candid images

To recognize facial expression from candid, non-posed images, we propose a deep-learning based approach using convolutional neural networks (CNNs). In order to evaluate the performance in real-time candid facial expression recognition, we have created a candid image facial expression (CIFE) dataset, with seven types of expression in more than 10,000 images gathered from the Web. As baselines, two feature-based approaches (LBP+SVM, SIFT+SVM) are tested on the dataset. The structure of our proposed CNN-based approach is described, and a data augmentation technique is provided in order to generate sufficient number of training samples. The performance using the feature-based approaches is close to the state of the art when tested with standard datasets, but fails to function well when dealing with candid images. Our experiments show that the CNN-based approach is very effective in candid image expression recognition, significantly outperforming the baseline approaches, by a 20% margin.