Probability based fast inter mode decision algorithm for H.264/AVC video standard

In this paper, a new probability based fast inter mode decision algorithm is proposed for H.264/AVC video standard. The main idea is to classify the inter modes into large and small modes category and determine which one is suitable for a macroblock (MB). Depending on this, the probability of each large mode corresponding to the initial Hadamard SAD_16×l6 cost or the probability of each small mode corresponding to the initial Hadamard SAD_s8×8 cost is determined. Then the best mode can be obtained from their probability characteristics. Thus, it is possible to determine the best inter mode only by calculating the Hadamard Transform (H-SAD) cost instead of rate distortion cost. Moreover, it avails the platform to determine the best mode without comparing the cost of each mode to other modes. As a result, this approach can avoid most of the complex computation processes like quantization, variable length coding, pixel reconstruction etc. The experimental results have shown that the proposed algorithm can reduce 60% to 65% of the total encoding time of H.264/ AVC with negligible distortion.