Fast Mode Decision Algorithm for Intra Prediction in H.264/AVC

In H.264/AVC, a novel criterion named the rate distortion optimization (RDO) is employed to select the optimal coding modes for each macroblock (MB) within the intra prediction coding pictures, which can achieve a high compression ratio while leading to a great increase in the complexity and computational load unfortunately. In this paper, a fast mode decision algorithm based on integer transform and adaptive threshold is proposed. Before the intra prediction, integer transform operations on the original image are executed to find the directions of local textures. According to this direction, only a small part of the possible intra prediction modes are tested at the first step. If the summation of differences (SAD) of the reconstructed block corresponding to the best mode is smaller than an adaptive threshold, calculation is terminated. Otherwise, more possible modes are needed. Simulations show that the fast mode decision algorithm proposed in this paper can accelerate the speed of intra picture coding significantly only with a negligible PSNR loss or a bit rate increment