Model-Driven Video-Based Rendering for Vehicles

We present a video-based rendering (VBR) method that creates arbitrary views of the scene from images captured by a single un-calibrated camera. The system is model-based and focused on a single object class, specifically vehicles in this paper. The procedure has three main steps. First, the object is rapidly modeled from a single image using a simple interactive tool. The modeling tool creates high quality 3D models by facilitating constrained modifications to a generic model of the object class. Secondly, the dynamic object is tracked in the video sequence and the pose of the object is recorded. The tracking and pose estimation leverage the 3D model to ensure high accuracy. Lastly, the surrounding scene is modeled as a textured piece-wise planar approximation. A free-viewpoint rendering of the video (4D video) is then produced as needed for special effects or virtual reality applications. The versatility and effectiveness of our approach to modeling and rendering is demonstrated on the object class of vehicles.