Progress in learning 3 vs. 2 keepaway

Reinforcement learning has been successfully applied to several subtasks in the RoboCup simulated soccer domain. Keepaway is one such task. One notable success in the keepaway domain has been the application of SMDP Sarsa(λ) with tile-coding function approximation. However, this success was achieved with the help of some significant task simplifications, including the delivery of complete, noise-free world-state information to the agents. Here we demonstrate that this task simplification was unnecessary: the agents are able to learn even in the presence of noisy, incomplete information. We also scale up to larger problems than have been previously tried. The main contribution of this paper is a deeper understanding of the difficulties of scaling up reinforcement learning to RoboCup soccer. We address several focused questions about the previous results with detailed experiments.