PNLMS-based Algorithm for Online Approximated Solution of HJB Equation in the Context of Discrete MIMO Optimal Control and Reinforcement Learning

In the context of discrete multivariable optimal linear quadratic regulator and reinforcement learning, this paper is attained to an investigation on PNLMS algorithm for online approximated solution of Hamilton-Jacobi-Bellman Equation. The PNLMS-based algorithms is designed to estimate the parameters of value function that is guided by Heuristic Dynamic Programming approach. The problem is characterized as approximation problem of cost value function that is computed without dynamic system model, but only by the measurements of states and action of interaction between the real world system and controller engine design. The proposed PNLMS-based algorithm is evaluated in a simplified mathematical model of the longitudinal dynamics of an aircraft that is a multivariable system. The performance of PNLMS-based algorithm is compared with those of RLS-based and LMS-based algorithms. These comparisons are performed in terms of convergence aspects and computational cost for online implementations of optimal design methodlogy.