Optimization of Plane Fits to Image Segments in Multi-view Stereo

One step in the reconstruction pipeline for creating scene geometry from multiple camera views is the fitting of 3-D planes to image segments. We optimize the four homogeneous coefficients of the equation of a plane, so that the homography it induces on a source image segment minimizes a cost function with photo consistency and edge consistency terms. The photo consistency term for a segment is defined as the summed squared pixel color differences with all target images in which it is visible. For the edges we find a set of sub pixel-precise Canny edge points from the source image. The edge consistency term considers all pairs of segments which are adjacent in 3D, and sums the squared 2D distances of the appropriate Canny edge points to the projection of the line in which the two segment planes intersect. We apply the L-BFGS non-linear optimization algorithm, which uses derivatives of the cost function with respect to the plane equation coefficients, and we derive analytic formulas for these derivatives for both terms in the cost function. An alternate optimization method uses the sparse Hessian matrix of second derivatives, and we compute these second derivatives as well. We test these plane optimizations on both synthetic and real image sequences.