View enhancement in stereo streams assisted by object model

Object registration between virtual model and real object in image sequences is a challenging task. With partial reconstruction of the real object from binocular images, we approach the problem in a 3D-3D way. The system we developed includes three major components: reconstruction, 3D-3D object registration and visual augmentation from object models. For reconstruction, a perspective stereo-motion based approach is used to recover the projective depths of features, a surface smoothing algorithm of moving least squares under perspective assumption enhances the accuracy of the reconstructions. For 3D-3D object registration, we use model template search to obtain an initial guess of the pose transformation from the model to image data, then Iterative Closest Point algorithm is applied to refine the initial guess. Experiments on real sequences show the effectiveness of the proposed approach.