Fast H.264 to HEVC transcoder based on post-order traversal of quadtree structure

In this paper, we propose a fast mode decision framework and a fast motion estimation algorithm for H.264 to High Efficiency Video Coding (HEVC) transcoding. The fast mode decision framework employs a post-order (bottom-up) traversal of the coding tree unit (CTU) quadtree. Based on this traversal and H.264 information, several strategies are proposed to reduce HEVC modes to be tested and a rate-distortion (RD) cost prediction model is used to terminate the processing of a tested mode early. The proposed fast motion estimation algorithm selects the best candidate from a list of H.264 motion vectors (MVs) and previously encoded HEVC MVs. Compared to a full re-encoding, experimental results show that the proposed solution achieves speed-ups of up to 12.75x, for an average BD-Rate of 3.28%.