Enhanced SAITD Cost Function for H.264/AVC Intra 4×4 Mode Decision

In this paper, an enhanced sum of absolute integer transform difference (ESAITD) cost function with rate predictor "sum of non-zero coefficient frequencies" from CAVLC-based rate estimation is proposed for Intra_4x4 mode decision in H.264/AVC. This rate predictor is using simple but significant parameter to approximate the total encoded bitrate accurately. Experimental results show that ESAITD achieves much better rate-distortion (RD) performance than sum of absolute Hadamard transform difference (SATD) cost function. It also outperforms sum of absolute integer transform difference (SAITD) cost function in medium and high bitrate. Overall, ESAITD can encode the sequences with high quality.