Combining motion and appearance for scene segmentation

Image segmentation is a key topic in computer vision, serving as a pre-step in a number of robotics tasks, including object recognition, obstacle avoidance and topological localization. In the literature, image segmentation has been employed as auxiliary information in order to improve optical flow performance. In this work, an alternative approach is proposed, in which optical flow information is used to aid image segmentation, aiming at scene understanding for mobile robots. The proposed system performs dense optical flow analysis, followed by clustering of the optical flow vectors in a four dimensional space (formed by the x and y positions, angle and magnitude of each vector). Results from the clustering are used as `seeds´ in the segmentation process, performed by watershed segmentation in our implementation. In addition, the flow `image´ is combined with the original image, generating an image better suited for watershed segmentation, reducing the local minima effect often seen in this type of segmentation algorithms. The main pipeline considers the use of multi-modality cameras (visible and thermal-infrared). Since they see substantially different information, multi-modality further improves the amount of features of the resulting flows. Experimental results in urban and semi-urban scenarios with efficient segmentation illustrate the applicability of the method.