Optimal control of affine nonlinear continuous-time systems using online actor-critic algorithm

In this paper we propose a new online actor-critic algorithm based on policy iteration for learning the continuous-time optimal control solution with infinite horizon cost for nonlinear systems. In other word, the algorithm solves online an algebraic Riccati equation without knowing the internal dynamics model of the system. This approach is implemented as an actor-critic structure which involves both actor and critic neural networks. Because of using a policy iteration method, the present algorithm alternates between the policy evaluation and policy update steps until an update of the control policy will no longer improve the system performance. Simulation results show the effectiveness of the new algorithm.