Audio-Visual Emotion Recognition Using Neural Networks Learned with Hints

This paper presents a neural network (NN) based multimodal fusion classifier for automatic emotion recognition. The audio and visual channels provide complementary information, so we utilize features from three behavioral cues: frontal-view facial expression, profile-view facial expression and vocalization (audio). The problem of interest is to recognize basic emotions, and we use dimensional representation of emotion as the heuristic information (hints) when training NNs used for both single cue processing and multimodal fusion. With the aid of hints, the weights of NNs could learn optimized feature groupings and recognition accuracy of our classifier would be notably improved even when training data is insufficient. Experimental results on audio-visual emotion data recorded by ourselves in Wizard of Oz scenarios and emotion data from the Semaine naturalistic database both demonstrate that our approach is effective and promising.