Rate distortion optimized transform for intra block coding for HEVC

The Discrete Cosine Transform is statistically optimal for first order Markov signals, which is widely used in image and video coding. However, in the intra frame coding of H.264/AVC, it is known that after directional intra prediction, there is still anisotropic features left in residue signals. And the features are related to the intra prediction modes. In order to represent the corresponding features, mode dependent directional transform (MDDT), which is based on Karhunen Loeve transform (KLT), was derived and adopted into JMKTA software, which is a preliminary software platform for High Efficiency Video Coding (HEVC)(). Within the MDDT scheme, each prediction mode has its correspondent transform. However, due to the data variation, this mode dependent classification may not lead to optimal residual data separation. It means that even though the residue blocks are using the same prediction mode, they may exhibit different statistical or structural properties. Sometimes, the mode dependent transform basis functions can not represent the residue signal very well. Therefore, in this paper, we propose a rate distortion optimized transform scheme, which provides a transform selection capability. The proposed scheme is implemented in HMO.9 for HEVC, achieving 3.2% BDBR in Intra Low Complexity (LoCo) condition and 2.0% BDBR in Intra High Efficiency (HE) condition.