A successive termination and elimination method for fast H.264/AVC SATD-based inter mode decision

A successive termination and elimination (STE) method to achieve fast inter mode decision is proposed. The termination detection starts from residual homogeneous detection and then spatial homogeneous detection is performed for each 16 times 16 macroblock. For either the residual or spatial homogeneous case, the authors can directly terminate the inter prediction and choose the 16 times 16 mode as the best inter mode. For non-homogeneous cases, the authors then carry out the 8 times 8 subblock motion estimation. Based on the cost analyses of the 8times8 and 16 times 16 modes, the elimination detection method, which could help to remove unlikely 8 times 16 and 16 times 8 modes, is also suggested. Similarly, the STE method for each 8times8 block can also be applied to decide if the inter prediction needs to be further performed for smaller subblocks. Once the algorithm reaches the termination stage, the best inter mode will be decided by selecting the least cost among all searched modes. Experimental results reveal that the proposed STE method can save about 56% of coding time in inter prediction with a slight performance degradation compared with the original method proposed by JM10.1 and outperforms some existing methods in both coding performance and coding time.