Handling different level of unstable reward environment through an estimation of reward distribution in XCS

XCS is an accuracy-based learning classifier system (LCS) which is powered by a reinforcement algorithm. We expect it will have when the reward for a state / action pair is unstable, because it is not possible to correctly estimate the evaluation. This paper focuses on learning in a different level of an unstable reward environment and proposes XCS-URE (XCS for Unstable Reward Environment) by improving XCS for such an environment. For this purpose, XCS-URE estimates the reward distribution of the classifier (i.e., if-then rule) by using the standard deviation of the acquired reward, and adjusts the accuracy of the classifier depending on the reward distribution. In order to investigate the effectiveness of XCS-URE, this paper applies XCS and XCS-URE into the multiple unstable reward environments which have a different level of the unstable rewards added by Gaussian noise. The experiments on the modified multiplexer problems have the following implications: (1) in the environment same Gaussian noise is added, XCS cannot performs properly due to the low accuracy of the classifier in the noisy environments, while XCS-URE can perform properly by acquiring the appropriate classifiers even in such an environment; (2) in the same environment, XCS-URE can reduce the population size without decreasing the correct rate as compared to XCS; and (3) even in the environment different Gaussian noises depending on the situation are added, XCS-URE can reduce the population size without decreasing the correct rate by adjusting the accuracy of the classifier depending on the reward distribution.