Audio-visual emotion recognition with boosted coupled HMM

This paper presents a novel approach for automatic audio-visual emotion recognition. The audio and visual channels provide complementary information for human emotional states recognition, and we utilize coupled HMM (CHMM) as model-level fusion method in our work. To further improve recognition accuracy, we design an AdaBoost-CHMM ensemble classifier which takes CHMM as component classifiers in adaptive boosting procedure. A modified expectation-maximization (EM) algorithm for CHMM learning is proposed to make the learning process focus more on difficult samples. Experiment results on audio-visual emotion data collected in Wizard of Oz scenarios and labeled under two types of emotion category sets demonstrate that our approach is effective and promising.