Speech-driven embodied entrainment character system with emotional expressions and motions by speech recognition

We have already developed a speech-driven embodied entrainment Computer Graphics (CG) character system, InterActor that generates communicative motions and actions such as nods for entrained interaction from speech rhythms on the basis of voice input only. In this study, we develop an advanced system using the InterActor that includes an expressed function of motions and emotional expressions derived from the meaning of utterances by speech recognition, as well as an entrained function. In this system, speech recognition converts the utterances of a talker, and a database detects specific character strings from the converted text, that resisters 105 words for 434 motions and emotional expressions. The system then generates emotional expressions and motions that correspond to the detected character strings. Furthermore, the sensory evaluation in the remote communication experiment for 12 pairs of 24 subjects demonstrated the effectiveness of the system, under three conditions: InterActor behaviors, a CG character with the expressed function, and InterActor behaviors and the expressed function (the developed system).