Real-time performance-driven facial animation with 3ds Max and Kinect

This paper presents a novel performance-driven realtime facial animation system with 3ds Max and Kinect. The user performs in a natural environment without any marker, and the 3D facial video of the user is captured with Kinect. The 2D key facial points are tracked with improved Active Appearance Model, and a set of Action Units are derived with the iterative closest point algorithm. Based on the two rough regions of eyes, we performed extra operations to track blinks and pupil motions. The data of Action Units are transmitted as control parameters using the Musical Instrument Device Interface communication protocol to drive the skeleton-based skinning facial model rendering with 3ds Max. The prototype system has been developed and tested on a common PC platform. Experimental results demonstrated that the digital avatar can mimic general apparent facial expressions of the performer, and the facial animation can reach 30 frames per second as the video capture rate.