Joint multi-view foreground segmentation and 3D reconstruction with tolerance loop

In this paper we present a novel foreground segmentation and 3D reconstruction system for multi-view scenarios. The system achieves correct 3D object reconstruction even when foreground segmentation presents critical misses in some of the views. We introduce the spatial redundancy of the multi-view data into the foreground segmentation process by combining segmentation and the 3D reconstruction in a two steps workflow. First, the segmentation of the objects in each view uses a monocular, region-based foreground segmentation in a MAP-MRF framework for foreground, background and shadow classes. Next, we compute an iterative volume reconstruction in a 3D tolerance loop, obtaining an iteratively enhanced SfS volume. Foreground segmentation is improved by updating the foreground model of each view at each iteration. The results presented in this paper show the improved foreground segmentation and the reduction of errors in the reconstruction of the volume.