A fast CABAC rate estimator for H.264/AVC mode decision

H.264/AVC coders use the rate-distortion (R-D) cost function to decide the coding mode for each coding unit for better R-D tradeoff. To evaluate the R-D cost function, both the bit rate and the video quality degradation of the candidate mode must be calculated. In this paper, a fast context-adaptive binary arithmetic coding (CABAC) rate estimation scheme is proposed to accelerate the rate calculation. The speed of the proposed rate estimator depends only on the number of contexts used in the coding unit. Experimental results show that the proposed rate estimator reduces about 20% of the computational complexity of the R-D optimized mode decision when it is implemented as software. The entire encoder implemented as software is then accelerated by 16% with negligible degradation in the R-D performance. If implemented as hardware, the proposed scheme is expected to accelerate the rate estimation for a macroblock by 5 to 18 times faster than the conventional CABAC operation.