Enhanced Low Complex Cost Function for H.264/AVC Intra Mode Decision

In order to reduce the computation of rate-distortion optimization (RDO) module of H.264/AVC intra encoder, this paper proposes an enhanced low complexity cost function. The enhanced cost function uses sum of absolute Hadamard-transformed differences (SATD) and mean absolute deviation of the residual block to estimate distortion part of the cost function. A threshold based large coefficients count is also used for estimating the bit-rate part. The proposed method improves the rate-distortion performance of the conventional fast cost functions while maintaining low complexity requirement. As the results, the encoding process can be significantly accelerated with use of the proposed cost function. Simulation results confirm that the proposed method reduces about 85% of computation of original encoder with negligible rate-distortion performance degradation.