Real-time facial animation based upon a bank of 3D facial expressions

We present an animation system that captures live facial expressions from a performance actor and uses them to animate in real time a synthetic character. Our approach is based upon a bank of 3D facial expressions of a synthetic model. Critical points on the face model are matched to live markers. A linear combination of the basic expressions obtained by minimizing Euclidean distance between corresponding points and markers is then used to construct the intermediate facial expression. Mapping the motion into a bank of expressions produces subtle motions, more dependent on the characteristics of the model itself. This is more difficult to achieve with previous techniques deforming a model without any additional information on how the model should move. Our technique draws mostly on the technical skills of an artist to model facial expressions, and on the skills of a performance actor to bring life to a face, with most animation issues being automated. We also present improvements on the control of markers, on specific details for more delicate control of the eyelids, and on the use of filters applied to the sequence of marker displacements. More flexibility in the bank of expressions is provided by subdividing some of the expressions into independent components, and by controlling accessories such as teeth and tongue only by means of the reconstructed expression. The resulting system is very flexible, intuitive to use, and the real time animations provide immediate feedback to express and refine animator and performer skills