Fast intra mode decision algorithm in H.264/AVC using characteristics of transformed coefficients

In H.264/AVC standard, intra prediction is allowed to perform with 9 or 4 candidate modes according to 4×4 or 16×16 block to achieve low bitrate and high quality. The exhaustive mode decision algorithm to choose the best mode is computationally expensive. To reduce the amount of calculations in intra prediction, a novel fast mode decision algorithm for H.264/AVC intra prediction is proposed based on the analysis of the characteristics of transform domain. By using characteristics of transformed coefficients of blocks with edge of directional distribution, only a small number of most possible intra prediction modes are chosen for RDO calculation and periodic insertion of exhaustive mode decision algorithm and adaptive threshold to prevent error diffusion. In that way, the fast mode decision algorithm helps to speed up intra coding significantly. Experimental results show that the proposed scheme performs nearly 5 times faster than the exhaustive search strategy adopted in current H.264/AVC encoder with little degradation in the coding efficiency.