Baum-Welch hidden Markov model inversion for reliable audio-to-visual conversion

In this paper, a novel audio-to-visual conversion method is presented. Many multimedia applications, such as videophones, videoconferencing, man-machine interface, language dubbing, character animation in virtual reality, etc., require techniques for synchronizing audio and video in a synthesized talking head sequence. For these applications, it is necessary to reliably estimate accurate mouth (visual) movements from the corresponding speech (audio) data. The hidden Markov model inversion (HMMI) technique introduced for robust speech recognition is extended in this paper into the audio-visual feature space. Based on the Baum-Welch HMMI method, reliable visual parameters are extracted given speech data only. Our preliminary simulation results show that the estimated visual parameters from the proposed method match the true visual parameters smoothly as well as accurately. The proposed estimation technique can be combined with video coding and graphics techniques for other multimedia applications