Real-time audio-visual localization of user using microphone array and vision camera

In home environments, demands for a robot to serve a user are on the increase, such as cleaning rooms, bringing something to the user, and so on. To achieve these tasks, it is essential for developing a natural way of human-robot interaction (HRI). One of the most natural ways is that the robot approaches the user to do some tasks after recognizing the user´s call and localizing its position. In this case, user localization becomes a key technology. In this paper, we propose a novel audio visual user localization system. It consists of a microphone array with eight sensors and a video camera. Estimating calling direction is achieved by the spectral subtraction of the spatial spectra. In particular, a novel beam forming method is proposed to suppress the nonstationary audio noises where they always occur in a real world. Furthermore, a robust method for face detection is proposed to double check the user based on an Adaboost classifier. It is improved to reduce the false alarms remarkably through a new postprocessing on face candidates. Successful results in a real home environment show its efficacy and feasibility. The implementation issues, limitations, and their possible solutions are also discussed.