Hybrid feature and decision level fusion of face and speech information for bimodal emotion recognition

A hybrid feature and decision level information fusion architecture is proposed for human emotion recognition from facial expression and speech prosody. An active buffer stores the most recent information extracted from face and speech. This buffer allows fusion of asynchronous information through keeping track of individual modality updates. The contents of the buffer will be fused at feature level; if their respective update times are close to each other. Based on the classifiers´ reliability, a decision level fusion block combines results of the unimodal speech and face based systems and the feature level fusion based classifier. Experimental results on a database of 12 people show that the proposed fusion architecture performs better than unimodal classification, pure feature level fusion and decision level fusion.