Tree-like Function Approximator in Reinforcement Learning

State value estimating is an important issue in reinforcement learning. It affects the performance significantly. The methods of lookup tables have advantages in convergence rate. But they need prior knowledge about how to partition the state space in advance. It is also not reasonable in a real system since the values associated with different sensory inputs but belonging to a representing state are the same. We proposed a method to discretize the state space adaptively and effectively in terms of an approach akin to decision tree methods. In each (discretized) presenting state, function approximators based on the tree structure estimate the values precisely.