Modeling and optimal compensation of quantization in multidimensional M-band filter bank

This paper provides a rigorous theory for modeling, and optimal compensation of quantization effects in multidimensional M-band FIR filter banks. Each pdf-optimized quantizer is modeled by a nonlinear gain-plus-additive uncorrelated noise and embedded into the subband structure. Using polyphase decomposition of the analysis/synthesis filter banks, we derive the exact expression for the output mean square quantization error and develop the technique for optimal compensation of quantization. Numerical design examples for the optimum nonseparable paraunitary and biorthogonal filter banks are developed with a quincunx subsampling lattice