A nonlinear analytical model for quantization effects in the LMS algorithm with power-of-two step size

The least mean squares (LMS) algorithm is one of the most popular algorithms for digital implementation of real-time high-speed adaptive filters. This paper presents a study of the quantization effects in the finite precision LMS algorithm with power-of-two step sizes. Nonlinear recursions are derived for the mean and second moment matrix of the weight vector about the Wiener weight for white Gaussian data models and small algorithm step size μ. The solutions of these recursions are shown to agree very closely with the Monte Carlo simulations during all phases of the adaptation process. A design curve is presented to demonstrate the use of the theory to select the number of quantizer bits and the adaptation step size μ to yield desired transient and steady-state behaviors