Potential based policy gradient approach for optimal control of the stochastic system with unknown noise

This paper considers optimal control problem of the discrete-time stochastic system, where the state space is continuous and the probability property of stochastic noise is unknown. First, the considered optimal control problem is transformed into a Markov Decision Process. Then, the performance potential based performance derivative formula can be applied for estimating the performance derivative with respect to the control parameters, which is the key of the policy gradient approach of this paper. For estimating the state transition probability density function (PDF) and the potential function, the RBF neural network is applied. With k_n-Nearest Neighbor techniques, the sample pairs for training the RBF neural networks can be collected from a sample path, so that the policy gradient approach can be implemented on-line for practical application. The simulation shows the effectiveness of the proposed approach.