Modified rate distortion optimization using inter-block dependence for H.264/AVC intra coding

H.264/AVC allows several prediction schemes including variable modes for intra and inter predictions. To take full advantage of all allowed modes, H.264 employs rate distortion optimization (RDO) as its mode decision method. However, RDO incorporated into the H.264 encoder assumes block independence because calculating rate distortion (RD) cost for all possible mode sets requires a huge number of computations. In contrast to this assumption, coding efficiency of intra prediction in the H.264 encoder is affected by reference pixels in neighboring blocks. Thus, RDO in the H.264 encoder has potential to improve coding performance by considering the accuracy of intra prediction for the next blocks during the mode decision process of the current block. In this paper, we propose new criteria for I4 times 4 mode decision and by using this criteria, the proposed method considers more mode sets to improve RD performance compared to the original RDO. Although the proposed method requires more computations compared to the original RDO, experimental results indicate that the proposed method noticeably improves coding efficiency compared to the original RDO.