Neural Q-learning control architectures for a wall-following behavior

The Q-learning algorithm, for its simplicity and well-developed theory, has been largely used in the last years in order to realize different behaviors for autonomous vehicles. The most frequent applications required the standard tabular formulation with discrete sets of state and action. In order to consider continues variables, function approximators such as neural networks are required. In this work we investigate the neural approach of Q-learning on the robot navigation task of wall following. Some issues have been addressed in order to deal with the convergence problem and the need of huge training sets. The experience replay paradigm has been also applied to reduce the unlearning problem. Two different neural network architectures, which use different spatial decompositions of the sensory input, have been compared. The aim is to investigate how different choices of architecture can affect the learning convergence, the optimality of the final controller and the generalization ability.