Face image analysis and synthesis for human-computer interaction

In this paper, we describe research results about how to generate an avatar´s face on a realtime process exactly copying a real person´s face. It is very important for synthesis of a real avatar to duplicate emotion and impression precisely included in the original face image and voice. A face fitting tool from multi-angle camera images is introduced to make a real 3D face model with real texture and geometry very close to the original. When the avatar is speaking something, the voice signal is very essential to decide the mouth shape feature. So a real-time mouth shape control mechanism is proposed for conversion from speech parameters to lip shape parameters using a multi-layer neural network. For dynamic modeling of the facial expression, a muscle structure constraint is introduced to generate the facial expression naturally with a few parameters. We also tried to get muscle parameters automatically to decide an expression from local motion vector on the face calculated by optical flow in a video sequence. We also tried to control this artificial muscle model directly by an EMG signal. To get more reality, a modeling method for hair is also introduced and the dynamics of hair in a stream of wind can be achieved with low calculation cost. By using these several kinds of multi-modal signal sources, a very natural face image and its impression can be duplicated on the avatar´s face