On Lagrange multiplier and quantizer adjustment for H.264 frame-layer video rate control

H.264/AVC encoder employs a complex mode-decision technique based on rate-distortion optimization. It calculates rate-distortion cost (RDcost) for all possible modes to choose the best one having the minimum RDcost. This paper presents a frame-layer rate control for H.264/AVC that computes the Lagrange multiplier (λ_MODE) for mode decision by using a quantization parameter (QP) which may be different from that used for encoding. At the same time, we also compare actual bits produced by previous macroblocks (MBs) with the total bits allocated to these MBs to further modify λ_MODE. The objective of these measures aims to produce bits as close to the frame target bits for rate control as possible. This is very important in the case of low-bit-rate tight buffer applications. In order to obtain an accurate QP for a frame, we employ a complexity-based bit-allocation scheme and a QP adjustment method. Simulation results comparing with the H.264 Joint Video Team (JVT) rate control method show that the H.264 encoder, using the proposed algorithm, achieves a visual quality improvement of about 0.56 dB, performs better for buffer overflow and underflow, and achieves a smaller PSNR deviation.