Multimodal Emotion Expressions of Virtual Agents, Mimic and Vocal Emotion Expressions and Their Effects on Emotion Recognition

Emotional expressions of virtual agents are widely believed to enhance the interaction with the user by utilizing more natural means of communication. However, as a result of the current technology virtual agents are often only able to produce facial expressions to convey emotional meaning. The presented research investigates the effects of unimodal vs. multimodal expressions of emotions on the users´ recognition of the respective emotional state. We found that multimodal expressions of emotions yield the highest recognition rates. Additionally, emotionally neutral cues in one modality, when presented together with emotionally relevant cues in the other modality, impair the recognition of the correct emotion category as well as intense emotional states.