On the choice of subgoals for learning control systems

A subgoal problem is defined as the problem of choosing a subgoal or on-line performance evaluator compatible with reinforcement-type learning control systems. The subgoal must evaluate each control decision separately and direct the learning process toward the optimum with respect to the primary goal or performance index. Analytical results are presented for an unconstrained control, including an M-decision evaluator. This general subgoal fails to satisfy the conditions of per-decision evaluation except when M is 1, but might prove useful with different learning algorithms. A method is presented for "doing the best with what you have." That is, a priori information might be used either to design a fixed controller or to choose the subgoal for a learning controller. Analytical and experimental results demonstrate that as a general rule the learning controller makes the best use of the a priori knowledge.