Geometrical aspects of interpreting images as a three-dimensional scene

One of the fundamental aspects of vision is three-dimensional geometry referring to the relationship between the scene which is depicted and the images which are projections of that scene. This paper presents computational theories which deal with geometrical aspects of interpreting images as a three-dimensional (3D) scene. The presentation covers three levels of descriptions: micro-surface level, volumetric-object level, and scene level. The micro-surface level concerns the relationship between surface orientations of small surface patches and image properties. Theories are presented which provide computational constraints for recovering surface orientations from geometrical image properties, such as symmetry, texture, shadows, and perspective distortions. The volumetric-object level handles shape representations of primitive objects: the relationship between 3D shape of an object and its projections onto images must be understood. The generalized cylinder is a popular volumetric shape representation in vision. We will discuss its formal properties. Finally, the scene level aims at constructing and maintaining the 3D description of the whole scene. We will present the Incremental 3D Mosaic system under development at CMU, which incrementally constructs the total 3D scene description of the task area from aerial photographs.