Performance of temporal differences and reinforcement learning in the cart-pole experiment

A comparison of the many proposed schemes for learning control tasks requires a standardised characterisation of their performance. Such characterisations are not available for most of the methods used in connection with neural networks. We undertook to characterise the performance of Barto´s et al. (1983) adaptive heuristic critic (AHC) method on the cart-pole balancing problem. We present criteria for training and control performance and use them to compare AHC controllers with the best PD controllers. It turns out that the learning dynamics of the AHC method for the cart-pole is apparently chaotic making performance assessment and parameter optimisation computationally very laborious.