Alternative theoretical frameworks for finite horizon discrete-time stochastic optimal control

This talk will provide highlights of recent research aimed at constructing a unifying and mathematically rigorous theory of dynamic programming and stochastic optimal control. Stochastic optimal control problems are usually analyzed under one of three types of assumptions: a) Countability assumptions on the underlying probability space - this eliminates all difficulties of measure theoretic nature, b) Semicontinuity assumptions under which the existence of Borel measurable policies can be guaranteed (Maitra, Sh???, Freedman), and c) Borel measurability assumptions under which the existence of p-optimal or p-?-optimal Borel measurable policies can be guaranteed (Blackwell, Strauch). We introduce a general theoretical framework based on outer integration which contains these three models as special cases. Within this framework all known results for finite horizon problems together with some new ones are proved and subsequently specialized. The long-standing measure theoretic difficulties in the Borel space model of Blackwell and Strauch are traced to inadequacies in the class of Borel measurable policies. An important new feature of our specialization to this model is the expansion of the class of admissible policies to include all universally measurable policies. We show that everywhere optimal or nearly optimal policies exist within this class and this enables us to dispense with the notion of p-optimality. Furthermore by utilizing the extra flexibility afforded by universally measurable policies we are able to provide for the first time a satisfactory and mathematically rigorous treatment of problems of imperfect state information using sufficient statistics. The measurability properties which a satisfactory class of policies must have are also delineated. The minimal class of policies with these properties is constructed and shown to lie strictly between the classes of Borel measurable and universally measurable policies. A detailed treatment appears in papers av- ilable from the authors on request, as well as in the authors´ monograph Stochastic Optimal Control: The Discrete-Time Case, to be published by Academic Press in the summer of 1978.