Title :
Stochastic Abstract Policies: Generalizing Knowledge to Improve Reinforcement Learning
Author :
Koga, Marcelo L. ; Freire, Valdinei ; Costa, Anna H. R.
Author_Institution :
Escola Politec., Univ. de Sao Paulo, Sao Paulo, Brazil
Abstract :
Reinforcement learning (RL) enables an agent to learn behavior by acquiring experience through trial-and-error interactions with a dynamic environment. However, knowledge is usually built from scratch and learning to behave may take a long time. Here, we improve the learning performance by leveraging prior knowledge; that is, the learner shows proper behavior from the beginning of a target task, using the knowledge from a set of known, previously solved, source tasks. In this paper, we argue that building stochastic abstract policies that generalize over past experiences is an effective way to provide such improvement and this generalization outperforms the current practice of using a library of policies. We achieve that contributing with a new algorithm, AbsProb-PI-multiple and a framework for transferring knowledge represented as a stochastic abstract policy in new RL tasks. Stochastic abstract policies offer an effective way to encode knowledge because the abstraction they provide not only generalizes solutions but also facilitates extracting the similarities among tasks. We perform experiments in a robotic navigation environment and analyze the agent´s behavior throughout the learning process and also assess the transfer ratio for different amounts of source tasks. We compare our method with the transfer of a library of policies, and experiments show that the use of a generalized policy produces better results by more effectively guiding the agent when learning a target task.
Keywords :
knowledge representation; learning (artificial intelligence); stochastic processes; AbsProb-PI-multiple; RL task; agent behavior; dynamic environment; knowledge representation; learning performance; learning process; reinforcement learning; robotic navigation environment; stochastic abstract policy; trial-and-error interaction; Abstracts; Buildings; Learning (artificial intelligence); Libraries; Markov processes; Stationary state; Knowledge transfer; machine learning;
Journal_Title :
Cybernetics, IEEE Transactions on
DOI :
10.1109/TCYB.2014.2319733