Layered representation of scenes based on multiview image analysis

This paper describes a novel cooperative procedure for the segmentation of multiview image sequences exploiting multiple sources of information. Compared to other approaches, no a priori information is needed about the structure and the arrangement of objects in the scene. Three cameras in a particular unsymmetrical set-up are used in the system. The color distribution and the object contours in the constituent 2-D images, the disparity information in stereo image pairs, as well as motion information in subsequent images, are analyzed and evaluated in a cooperative procedure to get reliable segmentation results. The scene is decomposed into a variable number of depth layers, with each layer showing a subset of the segmented regions. The layered representation can be used in a variety of applications. In this paper, the application aims at synthesizing 3-D images for enhanced telepresence allowing the user to “look around” in natural scenes (intermediate views for interactive displays). In another application, 3-D images showing a natural depth-of-focus are synthesized in order to improve viewing comfort with 3-D displays