Association of Sound to Motion in Video using Perceptual Organization

Traditionally, video surveillance has mostly made use of visual data. However, in light of the new behavioral and physiological studies which demonstrated the existence of cross modality effects in human perception, similar cues are being used to develop a surveillance system based on both audio and visual data. Human beings can easily associate a particular sound to an object in the surrounding. Drawing from such studies, we demonstrate a technique by which we can isolate concurrent audio and video events and associate them based on perceptual grouping principles. Simple cues from both audio and video suffice to make this association. By representing audio in the pitch-time domain, we can use image processing algorithms such as line detection to extract elementary audio events. These events are then grouped using Gestalt principles of similarity and proximity into appropriate auditory events. Properties such as time of occurrence and periodicity are easily calculated from these groups. In video, we extract motion and shape periodicities. By comparing all the periodicities in audio and video using a simple index we can easily associate audio to video