Nonlinear digital signal processing using homotopy continuation methods

Homotopy-based algorithms for solving the systems of nonlinear equations which arise in DSP when pole-zero models are used are proposed. Applications include autoregressive-moving average (ARMA) estimation for spectral estimation and system identification and optimal infinite-impulse-response (IIR) filter design. Nonlinear defining equations for the optimal system parameters result in a multimodal performance surface. Current numerical methods for solving these systems of nonlinear equations use gradient-based numerical methods which cannot distinguish between local and global minima or solve related suboptimal problems and which may not converge. Homotopy-based continuation methods are globally convergent numerical methods from which algorithms for finding the global minima of these nonlinear parameter estimation problems can be formed. A review is presented of homotopy continuation methods and digital signal processing using pole-zero structures, the homotopy functions proposed for solving these problems are discussed, and an illustrative example is provided