Artifact reduction of compressed video via three-dimensional adaptive estimation of transform coefficients

Block transform compressed videos usually suffer from annoying artifacts at low bit rates, caused by the coarse quantization of transform coefficients. The inter prediction utilized in video coding also induces block boundary artifacts when the neighboring blocks using different motion vectors from previous decoded frames. In this paper, a three-dimensional adaptive estimation method for video transform coefficients is proposed to reduce the artifact in compressed video. In the proposed method, transform coefficients of each block are estimated by adaptively fusing three prediction sources based on their reliabilities. One prediction source is the transform coefficients directly acquired from the decoded video, whose reliability is determined by the distribution of the quantization noise. The other prediction source is derived by a temporal autoregressive model of transform-blocks along motion trajectory among neighboring reference frames. Its reliability is estimated from prediction variance of local blocks. The last prediction source is derived from the nonlocal transform-blocks, whose reliability is estimated based on the distribution of the nonlocal coefficients and their similarity with the estimated block. Experimental results for compressed video sequences by HEVC show that, the proposed method can reduce the compression artifacts and improve both the objective and subjective quality.