Fast mode decision algorithm for Residual Quadtree coding in HEVC

High Efficiency Video Coding (HEVC) is an in- progress next generation video coding standard. It has a similar basic structure to the ITU/MPEG H.264/AVC coder but with further enhancement on each coding tool to increase compression efficiency. Thus, a Residual Quadtree (RQT) coding scheme is adopted in the conventional transform coding procedure. The cost is the additional computation complexity when compared to the fixed-size transform. In this paper, we design a fast algorithm for Residual Quadtree mode decision. Considering the Rate-Distortion efficiency, we replace the original depth-first mode decision process by a Merge-and-Split decision process. Furthermore, because a substantial numbers of zero-blocks are produced after quantization and mode decision, we reduce the unnecessary computation by using the inheritance property of zero-blocks. In addition, for nonzero-blocks, two early termination schemes are developed for both TU Merge and TU Split procedures, respectively. Comparing to HM 2.0, our method saves the RQT encoding time from 42% to 55% for a number of test videos with negligible coding loss.