Fusion of local appearance with stereo depth for object tracking

Object tracking methods based on stereo cameras, which provide both color and depth data at each pixel, find advantage in separating objects from each other and from background, determining the 3D size and location of objects, and modeling object shape. However, stereo tracking methods to date sometimes fail due to depth image noise, and discard much useful appearance information. We propose augmenting stereo-based models of tracked objects with sparse local appearance features, which have recently been applied with great success to object recognition under pose variation and partial occlusion. Depth data complements sparse local features by informing correct assignment of features to objects, while tracking of stable local appearance features helps overcome distortion of object shape models due to depth noise and partial occlusion. To speed up tracking of many local features, we also use a ldquobinary Gaborrdquo representation that is highly descriptive yet efficiently computed using integral images. In addition, a novel online feature selection and pruning technique is described to focus tracking onto the best localized and most consistent features. A tracking framework fusing all of these aspects is provided, and results for challenging video sequences are discussed.