Applying hierarchical reinforcement learning to computer games

Hierarchical finite state machine (HFSM) has proven to be a powerful tool for controlling non-player characters (NPCs) in computer games due to its flexibility and modularity. For most implementations, however, it is often the case that the control details at all levels are hand-coded. As a result, the development process is often time intensive and error prone. In this paper, we explore the use of a hierarchical reinforcement learning approach, based on hierarchies of abstract machines (HAMs), to help overcome some of these limitations. We analyse in detail both HAMs and its use for designing HFSM, propose two HAMs-related machines, and make a preliminary experiment: applying HAMs to design NPCs´ behavior and implementing it in Quake2. The result shows that this method can satisfy the need for controlling NPC and has faster convergence speed than flat reinforcement learning.