Real-time hierarchical scene segmentation and classification

We present an extension to our previously reported real-time scene segmentation approach which generates a complete hierarchy of segmentation hypotheses. An object classifier traverses the hypotheses tree in a top-down manner, returning good object hypotheses and thus helping to select the correct level of abstraction for segmentation and avoiding over- and under-segmentation. Combining model-free, bottom-up segmentation results with trained, top-down classification results, our approach improves both classification and segmentation results. It allows for identification of object parts and complete objects (e.g. a mug composed from the handle and its inner and outer surfaces) in a uniform and scalable framework. We discuss its advantages compared to existing approaches and present qualitative results. Finally, the approach is applied in an interactive robotics scenario to help the robot grasp objects in response to verbal commands.