Model-free adaptive dynamic programming for online optimal solution of the unknown nonlinear zero-sum differential game

It is well known that the two-player zero-sum differential game problem of the continuous-time nonlinear system relies on the solution of the Hamilton-Jacobi-Isaacs equation, which is a nonlinear partial differential equation that is difficult or impossible to solve. In this paper, a new model-free adaptive dynamic programming algorithm is developed for solving online the Hamilton-Jacobi-Isaacs equation for continuous-time nonlinear system with the fully unknown knowledge of the system dynamics. First, a simultaneous policy iteration algorithm will be given, which can solve the Hamilton-Jacobi-Isaacs equation in an off-line sense, in which the fully knowledge of the system dynamics is required. Second, based on the simultaneous policy iteration algorithm, a new model-free adaptive dynamic programming algorithm is developed for solving online the Hamilton-Jacobi-Isaacs equation, in which the fully knowledge of the system dynamics is not required. Finally, a numerical example is given to demonstrate the convergence and effectiveness of the proposed scheme.