Improved and fast algorithms for intra 4×4 mode decision in H.264/AVC

In H.264 advanced video coding (AVC), intra prediction plays an important role in the compression of intraframes. H.264 provides two kinds of cost function, sum of absolute difference (SAD) or sum of absolute transform difference (SATD) plus 4Pλ (Q_p) by using the Hadamard transform, to decide the best intra mode. We first propose an improved cost function, which uses the sum of absolute integer transform difference (SAITD) and then develop two fast transform algorithms, each for the Hadamard transform used in SATD and the integer transform used in SAITD. These two fast transform algorithms are based on the property of linear transform and spatial relation in the predicted block. With the fast Hadamard transform algorithm, we can reduce the computation nearly by half compared to the original algorithm. With a small increase of computation, the SAITD achieves better coding performance in low bit rate than the SATD.