Volumetric shape grammars for image segmentation and shape estimation

In this paper, we present a technique that uses 3D shape grammars to segment and estimate rectilinear shapes in non-rectified images. While others have proposed the use of shape grammars for segmentation, i.e., assigning labels to rectangular 2D regions within rectified images, the proposed method is innovative for the following reasons: (1) it uses projections of 3D shapes to define non-rectangular segmentation regions and (2) the approach also estimates the unknown shape parameters of the imaged object. As others have done in the past, we require user interaction to learn probability distributions for classifying pixels from the input image to each of the unknown classes. We then use a 3D shape grammar to hypothesize 3D models for the imaged object. A search procedure hypothesizes different 3D models by modifying the shape and pose parameters of the 3D shape grammar. The parameters of the hypothesized model that best fits the input image provide a segmentation of the image into semantic parts and shape estimates for each of the segmented parts. The key difference between the proposed approach and previous approaches is the 3D nature of our shape generation and estimation process which represents an advancement over existing approaches that are currently restricted to 2D representations. We describe the method and show segmentation and estimation results using non-rectified images.