Solving Problems with Extended Reachability Goals through Reinforcement Learning on Propositionally Constrained State Spaces

Finding a near-optimal action policy towards a goal state can be a complex task for intelligent autonomous agents, especially in a model-free environment with unknown rewards and under state space constraints. In such a situation, it is not possible to plan ahead which is the best action to execute at each moment, and to discover the states that can be visited during the plan execution requires foreknowing the conditions to be preserved for each environment state. We present here a new approach to discover the action policy for an environment under propositional constraints on states in MDP problems. The constraints are used by a strong probabilistic planning algorithm to reduce a state space whose transition probabilities are estimated by an action-learning reinforcement learning algorithm, thus simplifying the agent´s state space exploration and helping in the definition of the planning problem. The execution constraints, or preservation goals, comprised within the representation of the final goal, composes the extended reach ability goals. Experiments to validate the proposal were performed on an antenna coverage problem and produced interesting and promising results, demonstrating fast convergence to condition-preserving near-optimal policies that keep valid a set of propositions while reaching a final goal.