3D Tracking of Morphable Objects Using Conditionally Gaussian Nonlinear Filters

We present a generative model and its associated stochastic filtering algorithm for simultaneous tracking of 3D position and orientation, non-rigid motion, object texture, and background texture. The model defines a stochastic process that belongs to the class of conditionally Gaussian processes [On Kalman filtering for conditionally gaussian systems with random matrices]. This allows partitioning the filtering problem into two components: a linear component for texture that is solved using a bank of Kalman filters with time-varying parameters, and a nonlinear component for pose (rigid and non-rigid motion parameters) whose solution depends on the states of the Kalman filters. When applied to the 3D tracking problem, this results in an inference algorithm from which existing optic flow-based tracking algorithms and tracking algorithms based on texture templates emerge as special cases. Flow-based tracking emerges when the pose of the object is certain but its appearance is uncertain. Template-based tracking emerges when the position of the object is uncertain but its texture is relatively certain. In practice, optimal inference under this model integrates optic flow-based and template-based tracking, dynamically weighting their relative importance as new images are presented.