Sensor fusion as optimization: maximizing mutual information between sensory signals

Sensor fusion is a fundamental issue in developing intelligent systems that recognize the scene around them precisely and robustly. Previous approaches of sensor fusion combined a different kind of sensor after feature extraction and abstraction ("task-level fusion"). This paper proposes a new approach that combines sensory signals from a different kind of sensor before abstraction ("signal-level fusion"). By formalizing sensory fusion as an optimization that maximizes mutual information between sensory signals, a target in a changing scene is detected by a heuristic search algorithm. As an example, experimental results of sound source detection with one video camera and one microphone are shown.