Temporal differences learning with the conjugate gradient algorithm

This paper investigates the use of the conjugate gradient (CG) algorithm in comparison to the traditional backpropagation (BP) algorithm, applying to the temporal difference (TD) method for reinforcement learning. Time series prediction is the application domain examined. Simple time series as well as more complex ones, coming from real data (stock market indices), are used as benchmark problems. The performance measures used are the learning speed, the generalization ability, and the sensitivity on user-set parameters. Preliminary experimental results suggest that the performance of TD learning can be significantly improved when the CG algorithm is employed, as compared to the traditional BP algorithm. In addition, as expected, the CG algorithm has been proved to be more robust and less dependent on user-set training parameters and initial conditions, especially for rather complicated time series. The use of the CG algorithm in TD learning is therefore promising for real-life applications in time series prediction