Scene classification of images and video via semantic segmentation

Scene classification is used to categorize images into different classes, such as urban, mountain, beach, or indoor. This paper presents work on scene classification of television shows and feature films. These types of media bring unique challenges that are not present in photographs, as many shots are close-ups in which few characteristics of the scene are visible. In our work, the video is first segmented into shots and scenes, and key frames from each shot are analyzed before aggregating the results. Each key frame is classified as indoor or outdoor. Outdoor frames are further broken down by a semantic segmentation which provides a label to each pixel. These labels are then used to classify the scene type by describing the arrangement of scene components with a spatial pyramid. We present results from operating on a large database of videos and provide a comparison with selected work from the literature on photographs. Evidence of the success of the semantic segmentation is provided on a set of hand-labeled images. Our work improves the semantic segmentation and scene classification of images and, to the best of our knowledge, is the first paper that details a full working system on video.