A Bayesian Approach for Selective Image-Based Rendering Using Superpixels

Image-Based Rendering (IBR) algorithms generate high quality photo-realistic imagery without the burden of detailed modeling and expensive realistic rendering. Recent methods have different strengths and weaknesses, depending on 3D reconstruction quality and scene content. Each algorithm operates with a set of hypotheses about the scene and the novel views, resulting in different quality/speed trade-offs in different image regions. We present a principled approach to select the algorithm with the best quality/speed trade-off in each region. To do this, we propose a Bayesian approach, modeling the rendering quality, the rendering process and the validity of the assumptions of each algorithm. We then choose the algorithm to use with Maximum a Posteriori estimation. We demonstrate the utility of our approach on recent IBR algorithms which use over segmentation and are based on planar reprojection and shape-preserving warps respectively. Our algorithm selects the best rendering algorithm for each super pixel in a preprocessing step, at runtime our selective IBR uses this choice to achieve significant speedup at equivalent or better quality compared to previous algorithms.