Joint estimation of deformable motion and photometric parameters in single view video

In this paper we present a method for joint deformation and illumination parameter estimation from monocular image sequences exploiting direct image information. We are particularly interested in augmented reality applications, where a new texture is rendered onto a moving and deforming surface in the original video in real-time. Realistic retexturing not only requires geometric registration but also photometric parameter retrieval for convincing illusion. The contribution of this paper lies in a method for deformable surface augmentation exploiting a combination of an extended optical flow equation with a mesh-based shape and illumination prior that allows simultaneous deformation and photometric parameter estimation. Taking into account the photometric part by relaxing the brightness constancy assumption of the estimation not only allows realistic augmentation results but also improves spatial tracking.