Non-Rigid Structure from Motion using non-Parametric Tracking and Non-Linear Optimization

In this paper we address the problem of estimating the 3D structure and motion of a deformable non-rigid object from a sequence of uncalibrated images. It has been recently shown that if the deformation is modelled as a linear combination of basis shapes both the motion and the 3D structure of the object may be recovered using an extension of Tomasi and Kanade´s factorization algorithm for affine cameras. The main drawback of the existing methods is that the non-rigid factorization algorithm does not provide a correct estimate of the motion: the motion matrix has a repetitive structure which is not respected by the factorization algorithm. This also affects the estimation of the 3D shape. In this paper we present a non-linear optimization method which minimizes image reprojection error and imposes the correct structure onto the motion matrix by choosing an appropriate parameterization. In addition, we propose a novel non-rigid tracking algorithm based on the use of ranklets, a multiscale family of rank features. Finally, we show that improved motion and shape estimates are obtained on a real image sequence of a person´s face which is moving and changing expression.