Spontaneous facial expression analysis based on temperature changes and head motions

The existing approaches to automatic emotion analysis rely mostly on visible spectrum data, and very few works have been reported using thermal data for spontaneous facial expression analysis. In this paper, we present a novel infra-red thermal video descriptor in order to improve spontaneous emotion recognition. We first represent each thermal video as a series of clips. The face regions of each clip are warped to the frontal view based on scale-invariant feature transform (SIFT) flow. Meanwhile, we generate a corresponding SIFT flow video clip. Thermal video cuboids are segmented from each thermal video clip based on max pooling and motion video cuboids are segmented from each SIFT flow video clip based on average pooling. Thermal video words and motion video words are clustered by k-means cluster. Finally, each video is represented by a histogram of the bag of SIFT Flow and facial temperature changes video words. The resulting histogram is used as a descriptor for classification by the support vector machine (SVM). Experiments on two thermal databases show the advantage of the new descriptor as compared to the peer approaches for classifying spontaneous facial expressions.