Joint rendering and segmentation of free-viewpoint images

This paper presents a method that jointly performs synthesis of free-viewpoint images and object segmentation from that viewpoint. This method works efficiently and online by sharing a certain calculation process among rendering and segmentation steps. Since the segmentation is performed for arbitrary viewpoints directly, the extracted object can be superimposed onto another 3-D scene with geometric consistency. Experimental results using a 25-camera array show the effectiveness of our method.