Adaptive resolution function approximation for TD learning: simple division and integration

Adaptive resolution of function approximator is known to be important when we apply reinforcement learning to unknown problems. We propose to apply incremental division and integration scheme of function approximation to temporal difference learning based on local curvature. TD learning is based on non-constant value function approximation, which requires the simplicity of function approximator representation. We define bases and local complexity of function approximator in the similar way to the autonomous decentralized function approximation, but they are much simpler. The simplicity of approximator element bring much less computation and easier analysis. The proposed function approximator is proved to be effective through function approximation problem and RL standard problem, pendulum swing-up task.