On the optimal solution of the one-armed bandit adaptive control problem

The "one-armed bandit" problem of Bellman is a classic problem in sequential adaptive control. It is important a) for its own direct applications and b) since it is the simplest problem in the important class of Bayesian adaptive control problems. As in other such problems, even though the dynamic programming equation for the optimal expected return may be written down by inspection, it is extremely difficult to obtain explicit solutions. Concentrating on the case where the unknown parameter is beta-distributed, we establish the existence of a simple boundary curve separating the regions of unambiguous decision. Attention principally centers on the characterization and estimation of this curve. This is done with sufficient success to permit an explicit and complete solution of the problem for certain parameter ranges.