Appearance based lip tracking and cloning on speaking faces

This paper addresses the issue of lip tracking and cloning using appearance models. In this perspective, a statistical color-based lip detector is first used to classify the pixels of an image into lip and non-lip pixels. This rough estimation of the lip position is then used to initialize an appearance model. This models convergence allows to refine the positions of the MPEG-4 compatible feature points placed around the lip contours. The optimal position of the feature points is then used as a first estimate to compute the position at the next frame of an image sequence to perform speaking lip tracking. To animate an unknown face image in such a way that it reproduces the lip motion of the driving sequence a piecewise affine transform is applied forcing the target lips feature points to match the automatically detected feature points of each frame of the training sequence to perform lip motion cloning. Preliminary results show that the synthetic talking faces obtained are photorealistic and reproduce accurately the movements of the tracked mouth.