Multi-class segmentation for traffic scenarios at over 50 FPS

Multi-class segmentation assigns a class label to each pixel in an image. It represents a significant task for the semantic understanding of images and has received plentiful attention over the last years. The current state of art is dominated by conditional random field based approaches, defined over pixels or image segments. However, high accuracy segmentation comes at a high computational cost. The best performing methods can barely run at few frames per second and are far from real-time applications. Our goal is to bridge the gap between current state of the art segmentation approaches and real-time applications. In this paper we propose an efficient approach for individual pixel classification. Multiple local descriptors are computed densely and then quantized using visual codebooks. Joint boosting is used to classify each pixel based on the quantized local descriptors. We show that using careful design choices and GPU optimization we can achieve sate of the art segmentation results at over 50 FPS. We also propose a Conditional Random Field (CRF) model defined over superpixels that uses the proposed pixel classifier for the estimation of unary potentials. The CRF based multi-class segmentation can run at over 30 FPS. The proposed approach is validated on the MSRC21 and CamVid multi-class segmentation benchmarks, the former one consisting of urban traffic sequences.