Fast Rate Distortion Optimized Quantization for H.264/AVC

In this paper, a fast RDO quantization algorithm for H.264/AVC is proposed. In this algorithm, we consider that each quantized coefficient of level / may take one of the three possible levels / or I-I or / + 1 for output, thus 316 combinations need to be tested for a 4x4 block. To reduce the searching space of level adjustments, the input quantized coefficients are filtered in a hierarchical way. Firstly, the well quantized coefficients are filtered out by comparing an adaptive threshold value with the absolute difference of transformed coefficients and the reconstructed coefficients. Secondly, the RD tradeoff of each level adjustment to each of the rest coefficients is examined to select some good coefficient candidates with their associated level adjustments. In this step, a fast rate estimation technique [3] is adopted for fast evaluation of the RD tradeoff for a single level adjustment. Finally, these good candidates are sorted in the descending order according to their associated ratedistortion tradeoff gain evaluated in the second step. The top M of them are selected and combined to find the best combination which gives the minimal rate-distortion cost.