Perceptual video coding: Challenges and approaches

Investigation on the human perception can play an important role in video signal processing. Recently, there has been great interest in incorporating the human perception in video coding systems to enhance the perceptual quality of the represented visual signal. However, the limited understanding of the human visual system and high complexity of computational models of human visual system make it a challenging task. Furthermore, the hybrid video coding structure brings difficulties to integrate computational models with coding components to fulfill the requirements. In this paper, we review the physiological characteristics of human perception and address the most relevant aspects to video coding applications. Moreover, we discuss the computational models and metrics which guide the design and implementation of the video coding system, as well as the recent advances in perceptual video coding. To introduce this overview with the latest technologies and most promising directions in perceptual video coding, we focus on three key areas. Specifically, we cover 1) visual attention and sensitivity modeling, with which we concentrate on the computational models of bottom-up and top-down attention, contrast sensitivity functions and masking effects, and fovea based manipulations; 2) perceptual quality optimization for constrained video coding, with which we discuss how to achieve maximum perceptual quality whilst satisfying various constraints; and 3) the impact of the human perception on advanced video applications, including emerging immersive multimedia services, and compression of high dynamic range video content and 3D video. For each aspect, we discuss the major challenges, highlight significant approaches, and outline future research directions.