Perceptual rate distortion optimization for block mode selection in hybrid video coding

Video compression technologies developed recently have taken advantage of a hybrid approach for better coding efficiency. For example, in the MPEG-4 Advanced Video Coding standard and also the new High Efficiency Video Coding, there are numerous possibilities of motion representation and residual coding options in transform domain. The coding parameters, including block modes, motion vectors, and indices of reference frames, are often correlated with each other in spatial and temporal directions. The selection of those coding parameters can affect the coding efficiency greatly. In many of the state-of-the-art solutions, the selection of coding options is based on the minimization of a Lagrangian cost function which is constructed of conventional distortion function and produced data rate. However, the conventional distortion function measured as either sum of absolute difference or mean square error does not often represent the observed deficiency in reconstructed videos. In this paper, a perceptual rate distortion optimization algorithm for block mode selection is developed, and the content-dependent Lagrangian multiplier is then derived. The just-noticeable distortion is used in the proposed method such that perceptual quality can be considered. The simulation results show that the rate-perceptual distortion performance is improved substantially without sacrificing the conventional rate-distortion performance.