Fast Soft Decision Quantization With Adaptive Preselection and Dynamic Trellis Graph

Soft decision quantization (SDQ) is an efficient tool for video coding to achieve coefficient-level rate-distortion optimized quantization (RDOQ) with a 6%-8% bit rate saving. However, the software and hardware implementations of SDQ suffer from either high complexity or low throughput capacity due to complex Viterbi trellis search and sequential processing in context-adaptive binary arithmetic coding. In this paper, a fast SDQ algorithm is proposed to decrease the number of trellis stages to decrease the complexity and to break the data dependency in optimal SDQ. First, preselection is performed according to hard decision quantization results by intelligent coding cost estimation and comparison, during which some coefficients are judged to be safely excluded from trellis search, resulting in considerable complexity reduction. Second, a dynamic trellis graph with flexible structure is constructed according to the unsafe nonzero coefficients to accelerate the remaining partial Viterbi search. Third, a dynamic threshold selection model is proposed for adaptive thresholding to increase the probability of right preselection under a constraint on a predefined maximal probability of wrong preselection. The experimental results show that compared with optimal SDQ, the proposed algorithm can at least reduce the computation complexity by 50%-80%, memory accesses by 75%-82%, and the sequential processing latency in hardware implementation by 87.25%, with less than 0.4% Bjøntegaard bit rate increment when a maximum of three unsafe coefficients are kept for trellis search in one block. This paper is suitable for high-throughput hardware and computation-sensitive software implementations for SDQ and RDOQ for H.264/Advanced Video Coding and High Efficiency Video Coding standards.