Multimedia sensing system for robot

The purpose of this study is to realize a multimedia sensing system for robot. Using both image and sound processing, the system makes a robot track the person who is speaking. The sound direction is calculated from the phase difference between the sounds arriving at the right and left microphones (ears) of the robot. Then by detecting the synchronization between the sound and image changes, the system identifies the speaker. Furthermore, by introducing a multi-level synchronization checking and context analysis, the action pattern of the robot can be regulated to make the robot perform in a complicated environment with plural speakers. All the processes are performed in real-time. The proposed system is implemented in the information assistant robot “Hadaly”