Real-time 3D motion and structure of point features: a front-end system for vision-based control and interaction

We present a system that consists of one camera connected to a personal computer that can (a) select and track a number of high-contrast point features on a sequence of images, (b) estimate their three-dimensional motion and position relative to an inertial reference frame, assuming rigidity, (c) handle occlusions that cause point-features to disappear as well as new features to appear. The system can also (d) perform partial self-calibration and (e) check for consistency of the rigidity assumption, although these features are not implemented in the current release. All of this is done automatically and in real-time (30 Hz) for 40-50 point features using commercial off-the-shelf hardware. The system is based on an algorithm presented by Chiuso et al. (2000), the properties of which have been analyzed by Chiuso and Soatto (2000). In particular, the algorithm is provably observable, provably minimal and provably stable- under suitable conditions. The core of the system, consisting of C++ code ready to interface with a frame grabber as well as Matlab code for development, is available at http://ee.wustl.edu/-soatto/research.html. We demonstrate the system by showing its use as (1) an ego-motion estimator, (2) an object tracker, and (3) an interactive input device, all without any modification of the system settings.