A direct approach for efficiently tracking with 3D morphable models

We present an efficient algorithm for fitting a morphable model to an image sequence. It is built on a projective geometry formulation of perspective projection, which results in a linear mapping from 3D shape to the projective plane, and a factorisation of this mapping into matrices that can be partially computed off-line. This algorithm can cope with full 360 degrees object rotation and linear deformations. We validate our approach using synthetically generated and real sequences. Compared to a plain Lucas-Kanade implementation, we achieve a six fold increase in performance for a rigid object and two fold for a non-rigid face.