The anatomy of an active multimodal presentation in educational contexts

This paper presents a methodology for analyzing gestures within active multimodal presentations (AMP) which are designed to produce coherently integrated communications within educational contexts. The methodology is focused on a semantic integration of the visual, verbal and gestural modalities. We identify three representational forms - a time based transcript, that is extracted from presentations to produce a speech mark-up; a concept based semantic analysis of the content; and a representation of gestures which distinguishes visual depictions of their icons, their trajectories, referents, and functions that link the verbal and visual modalities of the presentations. Our experiments are showing the utility of the analyses giving guidelines that can aid the design of AMPs within educational contexts.