Detection of critical situations by CMAC+Q-learning for PacMan agents

We previously proposed evolutionary fuzzy systems of playing Ms. PacMan for the competitions. As a consequence of the evolution, reflective action rules such that PacMan tries to eat pills effectively until ghosts come close to PacMan are acquired. Such rules works well. However, sometimes it is too reflective so that PacMan go toward ghosts by herself in longer corridors. In this paper, a critical situation learning module is combined with the evolved fuzzy systems, i.e., reflective action module. The critical situation learning module is composed of Q-learning with CMAC. Location information of surrounding ghosts and the existence of power-pills are given to PacMan as state. This module punishes if PacMan is caught by ghosts. Therefore, this module learning which pairs of (state, action) cause her death. By using learnt Q-value, PacMan tries to survive much longer. Experimental results on Ms. PacMan elucidate the proposed method is promising since it can capture critical situations well. However, as a consequence of the large amount of memory required by CMAC, real time responses tend to be lost.