Multimodal signal analysis of prosody and hand motion: Temporal correlation of speech and gestures

This paper is concerned with the processing and analysis of signals pertaining to two communicative modalities, speech and gestures, and investigating the nature of their temporal relationship. Two hypotheses have been proposed about the activation of the gestural system during speech production: the inhibitory hypothesis and the excitatory hypothesis. The validation of either one of these hypotheses necessitates the processing of large amounts of data by experts. The work described here is an effort to develop valuable signal processing tools to facilitate the multi-modal analysis, at least partially, in an automated manner. In this work algorithms are developed to determine points of emphasis in each of the modalities of speech and gesture data using prosody and hand motion traces obtained from two experiments. The results agree with the excitatory hypothesis: if we compare the temporal locations of the speech focal points with the temporal locations of the gesture focal points, they co-occur in more than 90% of the locations.