Automatic Sign Detection and Recognition in Natural Scenes

Visually impaired individuals are unable to utilize the significant amount of information in signs. VIDI is a system for detecting and recognizing signs in the environment and voice synthesizing their contents. The wide variety of signs and unconstrained imaging conditions make the problem challenging. We detect signs using local color and texture features to classify image regions with a conditional maximum entropy model. Detected sign regions are then recognized by matching them against a known database of signs. A support vector machine classifier uses color to focus the search, and a match is found based on the correspondences of corners and their associated shape contexts. Our dataset includes images of downtown scenes with several signs exhibiting both illumination differences and projective distortions. A wide range of signs are detected and recognized including both text and symbolic information. The detection and the recognition components each perform well on their respective tasks, and initial evaluations of a complete detection and recognition system are promising.