Implementing autonomous shaping by critical states

Shaping is a powerful method for speeding up reinforcement learning, but the major drawback that shaping reward depends on external observer limits its application and requires significant effort. We implement an autonomous shaping reinforcement learning method by making agent can discover autonomously critical states from prior experience and use them to shape later learning. The critical state is a state that has high probability to exist in all these acyclic state trajectories that from the start state to the goal state, that means, if agent wants to reach the goal state, then it would have high likelihood to pass the critical states. So the critical states can be used to shape agent for reaching the goal state faster. The experiments on Maze problem show our method can significant improve agent´s performance. The more important is we make agent can shape its later learning by its prior experience.