Acoustic speech to lip feature mapping for multimedia applications

This paper presents a quantitative analysis of the relationship between acoustic speech and corresponding lip features. The lip features, such as the lip width, inner lip height and outer lip height, are acquired by an extraction algorithm combining both color and edge information within a Markov random field (MRF) framework. Meanwhile, LSP (linear spectrum pairs) coefficients are used to parameterize the acoustic speech. LSP coefficients and the lip features are then used to train mapping models. The resulting models are used to estimate lip features from acoustic speech. From the results, we can see that the measured lip features match fairly well with the estimated lip features. The correlation coefficients between measured and estimated lip features are as high as 0.90. The estimation technique of lip features from acoustic speech gives a way to integrate acoustic and visual speech, which is very useful for speech driven face animation, audio-video synchronization and foreign film dubbing.