Title :
Multimodal Temporal Panorama for Moving Vehicle Detection and Reconstruction
Author :
Wang, Tao ; Zhu, Zhigang ; Taylor, Clark N.
Author_Institution :
Dept. of Comput. Sci., City Coll. of New York, New York, NY, USA
Abstract :
In this work, we present a multimodal temporal panorama (MTP) representation that synchronizes visual, motion, and acoustic signatures of moving vehicles in the time axis. The MTP representation includes two layers: a synopsis layer and a snapshot layer. The temporal synopsis consists of 1) a panoramic view image (PVI) to represent vehicles´ presence, which is constructed from 1D vertical detecting lines of a selected column location of all video frames, 2) an epipolar plane image (EPI) to characterize their motion (speeds and directions), generated from 1D horizontal scanning lines along the vehicles´ moving paths, and 3) an audio wave scroll for visualizing moving vehicles´ acoustic signatures. The MTP synopsis not only synchronizes all the three modalities (visual, motion and acoustic) of the vehicles, but also provides information that can perform automatic detection tasks including moving vehicle visual detection, motion estimation, and acoustic signature retrieval. Then in the snapshot layer, the occlusion-free, motion-blur-free, and view-invariant reconstruction of each vehicle (with both shape and motion information) and its acoustic signatures (e.g. spectrogram) are embedded. The MTP provides a very effective approach to (semi-)automatically labeling the multimodal data of uncontrolled traffic scenes in real time for further vehicle classification, check-point inspection and traffic analysis. The concept of MTP may not be only limited to visual, motion and audio modalities, it could also be applicable to other sensing modalities that can obtain data in the temporal domain.
Keywords :
acoustic signal processing; image classification; image reconstruction; image representation; motion estimation; object detection; traffic engineering computing; vehicles; 1D horizontal scanning lines; 1D vertical detecting lines; acoustic signature retrieval; audio wave scroll; check-point inspection; epipolar plane image; image reconstruction; motion characterization; motion estimation; motion information; motion signature; motion-blur-free reconstruction; moving vehicle detection; moving vehicle visual detection; multimodal temporal panorama representation; occlusion-free reconstruction; panoramic view image; sensing modality; shape information; snapshot layer; spectrogram; temporal synopsis; time axis; traffic analysis; uncontrolled traffic scene; vehicle acoustic signature; vehicle classification; vehicle moving path; vehicle presence representation; video frames; view-invariant reconstruction; visual signature; Acoustics; Cameras; Image reconstruction; Labeling; Roads; Vehicles; Visualization; epipolar plane image; multmodal; panoramic view image; vehicle detection; vehicle reconstruction;
Conference_Titel :
Multimedia (ISM), 2011 IEEE International Symposium on
Conference_Location :
Dana Point CA
Print_ISBN :
978-1-4577-2015-4
DOI :
10.1109/ISM.2011.101