Multimodal and cross-modal analysis of expressive gesture in tangible acoustic interfaces

This paper focuses on multimodal and cross-modal analysis of expressive gesture with a particular focus on collaborative interactive systems exploiting tangible acoustic interfaces (TAIs). We developed TAIs aiming at processing expressive information from users and supporting creativity in concrete music theatre and museum projects. The paper presents (i) techniques for extraction and analysis of high-level features from expressive gesture of TAIs users in collaborative frameworks, (ii) concrete examples of multimodal and cross-modal processing of expressive gesture, (iii) examples of how such results have been exploited in public events and artistic productions. In such occasions the developed techniques have been applied and evaluated with experiments involving both experts and the general audience. Research is carried out in the framework of the EU-IST STREP Project TAI-CHI (Tangible Acoustic Interfaces for Computer-Human Interaction). High-level expressive gesture analysis and multimodal and cross-modal processing are achieved in the new EyesWeb 4 open platform (available at www.eyesweb.org)