Facial expression recognition combined with robust face detection in a convolutional neural network

Reliable detection of ordinary facial expressions (e.g., smile) despite the variability among individuals as well as face appearance is an important step toward the realization of perceptual user interface and the next generation imaging system with autonomous perception of persons. We describe a robust facial expression recognition system using the result of face detection by a convolutional neural network and rule-based processing. In this study, we address the problem of subject independence as well as translation, rotation, and scale invariance in the recognition of facial expression. The result shows reliable detection of smiles with recognition rate of 97.6% for 5600 still images of more than 10 subjects. The proposed algorithm demonstrated the ability to discriminate smiling from talking based on the saliency score in the proposed algorithm. To the best of our knowledge, it is the first facial expression recognition model with the property of subject independence combined with robustness to variability in facial appearance.