Efficient Selection of Candidates for Fast Inter Mode Decision in H.264

In the H.264/AVC video codec, a rate-distortion optimization method for mode decision provides high coding efficiency. However, it increases encoding time significantly due to its high complexity. In this paper, we propose a fast algorithm for inter mode decision using motion cost during motion estimation to reduce the number of candidate modes. The proposed approach makes early 16 times 16 mode decision and early inactive P8 times 8 mode decision. Its encoding performance is similar to that of the fast inter mode decision in the H.264/AVC JM reference software; however, its encoding time is reduced by 30.95% on average. Especially, when quantization parameter (QP) is larger, we obtained better results.