Audiovisual speech enhancement experiments for mouth segmentation evaluation

Mouth segmentation is an important issue which applies in many multimedia applications as speech reading, face synthesis, recognition or audiovisual communication. Our goal is to have a robust and efficient detection of lips contour in order to restore as faithfully as possible the speech movement. We present a methodology which focused on the detection of the inner and outer mouth contours which is a difficult task due to the non-linear appearance variations. Our method is based on a statistical model of shape with local appearance gaussian descriptors whose theoretical responses were predicted by a non-linear neural network. From our automatic segmentation of the mouth, we can generate a clone of a speaker mouth whose lips movements will be as close as possible of the original ones. In this paper, results obtained by this methodology are evaluated qualitatively by testing the relevance of this clone. We carried out an experience which quantified the effective enhancement in comprehension brought by our analysis-resynthesis scheme in a telephone enquiry task.