Joint optical flow estimation, segmentation, and 3D interpretation with level sets

This paper describes a variational method with active curve evolution and level sets for the estimation, segmentation, and 3D interpretation of optical flow generated by independently moving rigid objects in space. Estimation, segmentation, and 3D interpretation are performed jointly. Segmentation is based on an estimate of optical flow consistent with a single rigid motion in each segmentation region. The method, which allows both viewing system and viewed objects to move, results in three steps iterated until convergence: (a) evolution of closed curves via level sets and, in each region of the segmentation, (b) linear least squares computation of the essential parameters of rigid motion, (c) estimation of optical flow consistent with a single rigid motion. The translational and rotational components of rigid motion and regularized relative depth are recovered analytically for each region of the segmentation from the estimated essential parameters and optical flow. Several examples with real image sequences are provided which verify the validity of the method.