Feature-level fusion of multimodal physiological signals for emotion recognition

Objective: This paper aims to use multimodal physiological signals to automatically recognize human emotions, and a novel multimodal feature fusion approach is proposed. Methods: In the proposed approach, significant multimodal features are selected respectively by two comparative feature selection methods: Fisher Criterion Score and Davies-Bouldin index. Emotion recognition is performed on the valence-arousal emotion space by using hidden Markov models (HMMs) and multimodal feature sets. Four physiological modalities, including electroencephalogram (EEG) from central nervous system and peripheral physiological signals (PERI) from peripheral nervous system as shown in the DEAP database, are employed. Results: We show the best recognition accuracies of 85.63% for arousal and 83.98% for valence. The proposed feature fusion approach is compared with decision-level fusion and non-fusion approaches on the same database; and the comparison demonstrates significant improvements in accuracy obtained by the feature fusion approach. Conclusion: Our work supports the observation that the proposed feature-level fusion approach represents a promising methodology for emotion recognition.