Improved fast intra mode decision with reliability refinement

Fast intra mode decision algorithms in video coding are very useful and necessary for practical real-time encoder implementation. It reduces the computational complexity by eliminating unlikely modes from candidate set. However, most of these algorithms may lead to the noticeable quality degradation, especially for the all intra coding scenario. To overcome the quality loss while maintaining the fast encoding speed, we propose a method that estimates the reliability of the fast algorithms based on the edge magnitude histogram values of different modes. It could switch between the original fast algorithm and an enhanced mode decision refinement according to the reliability measurement. Extensive simulation results demonstrate that, with some sacrifice of the encoding speed, the proposed method could greatly improve the video quality by reducing the Bjontegaard Delta Rate (BD-Rate) loss from 9.24% to 4.49% on average, in comparison to the most cited fast intra mode decision algorithm in the H.264/AVC [1], both against the brute force full rate-distortion optimization (RDO) search based solution.