Toward the introduction of auditory information in dynamic visual attention models

Classical visual attention models only use visual features to predict where observers should look at. However, in daily life, visual information is never perceived without its corresponding audio signal. In a previous study, we found that sound modifies visual exploration by comparing the eye movements recorded when viewing videos with or without their original soundtrack. The aim of the presented research is to further understand how sound influences eye movements by controlling visual and audio contents of the videos, as well as the congruency between them. We describe an experiment with a novel approach in which observers watched videos belonging to four visual categories presenting different visual saliency distributions: landscapes, one moving object, several moving objects and faces. Videos were seen with their original soundtrack or with the soundtrack from another video belonging to the same visual category. Using different metrics to analyze the recorded eye movements, we found that sound has an influence only on videos containing faces and several moving objects. The original soundtrack decreases the variability between the eye positions of observers. Finally, we propose some cues to integrate sound information in classical visual attention models.