Perceptually Adaptive Lagrange Multiplier for Rate-Distortion Optimization in H.264

This paper proposes a novel perceptually adaptive Lagrange multiplier (PALM) algorithm for further enhancing RDO performance that is based on HVS features. Two perceptual distortion sensitivity models are created to minimize the perceptual distortion rather than traditional MAE distortion. Lagrange multiplier is adjusted adaptively according to these perceptual models during the RDO process. Larger Lagrange multiplier is assigned to regions that are perceptually less sensitive to distortion so that the rate- reduction is weighted more than distortion reductions in these regions. While smaller Lagrange multiplier is arranged to the perceptual distortion sensitive regions. Experimental results show that the proposed PALM method effectively improves the subjective quality in the most perceptually prominent regions with no notable loss in PSNR.