Towards a computational model of Acoustic Packaging

In order to learn and interact with humans, robots need understand actions and make use of language in social interactions. The use of language for the learning of actions has been emphasized by Hirsh-Pasek & Golinkoff introducing the idea of acoustic packaging. Accordingly, it has been suggested that acoustic information, typically in the form of narration, overlaps with action sequences and provides infants with a bottom-up guide to attend to relevant events and to find structure within them. Following the promising results achieved by Brand & Tapscott for infants who packaged sequences together when acoustic narration was provided, in this paper, we make the first approach towards a computational model of the multimodal interplay of action and language in tutoring situations. For our purpose, we understand events as temporal intervals, which have to be segmented in both the visual and the acoustic signal in order to perform acoustic packaging. For the visual modality, we looked at the amount of motion over time via a motion history image based approach. The visual signal is segmented by detecting local minima in the amount of motion. For the acoustic modality, we used a phoneme recognizer, which currently segments the acoustic signal into speech and non-speech intervals. Our acoustic packaging algorithm merges the segments from both modalities based on temporal synchrony. First evaluation results show that acoustic packaging can provide a meaningful segmentation of tutoring behavior.