RDO Transform Selection for a Multi-Transform Video Encoder

A transform selection for a multi-transform video encoding scheme (AVC possible extension) is presented. This transform selection uses rate distortion optimization (RDO) technique. This method can also be adapted to be used for coding mode-selection and quantization step selection. RDO technique is applied to a selection of transform for each residual sub-block (4 x 4 PELs) from an available transform collection. The mode selection methodology from the work of Weigand et al. (2003) is improved and used for the transform selection. The mode selection process uses suboptimal RDO process with one dimensional space instead of two-dimensional space, which leads to a PSNR degradation of 1- 2dB compared to two-dimensional space RDO. The suggested RDO method uses two stage minimization processes: Initially, Lagrange minimization is applied to each transform. Then, second Lagrange minimization used to select transform/quantization pair. The complexity is reduced by assuming that the RD function of the scalar quantization is a monotonic function, and by use in developed bit-rate predictor. The bit-rate prediction improves the methods that are presented in the work of Kim et al. (2001) and Shin and Lee (2003) and achieves bit-rate degradation of 0.31% - 6.8% compared to non prediction scheme vs. a bit-rate degradation of 3.2% - 13.2% by use bitrate predictor. Finally video encoding process which presented achieve up to 30% of improvement. The video encoding uses RDO selection and the bitrate predictor which developed here. The RDO process selects out of three alternative transform which given by Ben-Israel (2006).