Speech Activity Detection with Lip Movement Image Signals

This paper describes an attempt to correlate lip movement visual information acquired via a camera with speech audio information acquired via a microphone from a human speaker in order to prevent audio created by external noise from being misrecognized as speech emitted by said speaker. Images of the face of a human speaker are acquired via a PC camera and are then separated into images that indicate lip movement and images that do not indicate lip movement. The data of lip movement image signals is saved in shared memory and shared with the speech recognition process. This data is analyzed by the speech activity detection process, which is a pre-processing step of sound recognition. We combined a speech recognition processor and an image recognizer, and the interworking function successfully operated at the rate of 99.3%.