Target tracking in heterogeneous sensor networks using audio and video sensor fusion

Heterogeneous sensor networks (HSNs) with multiple sensing modalities are gaining popularity in diverse fields. Tracking is an application that can benefit from multiple sensing modalities. If a moving target emits sound then both audio and video sensors can be utilized. These modalities can complement each other in the presence of high background noise that impairs the audio or visual clutter affecting the video. Audio-video tracking can also provide cues for the other modality for actuation. In this paper, we describe an approach for target tracking in urban environments utilizing an HSN of mote class devices equipped with acoustic sensor boards and embedded PCs equipped with web cameras. Our system employs a Markov Chain Monte Carlo Data Association algorithm for tracking vehicles emitting engine noise. Experimental results from a deployment in an urban environment are used to demonstrate our approach.