Stochastic dynamic programming: Caution and probing

The purpose of this paper is to unify the concepts of caution and probing put forth by Feldbaum [14] with the mathematical technique of stochastic dynamic programming originated by Bellman [5]. The decomposition of the expected cost in a stochastic control problem, recently developed in [8], is used to assess quantitatively the caution and probing effects of the system uncertainties on the control. It is shown how in some problems, because of the uncertainties, the control becomes cautious (less aggressive) while in other problems it will probe (by becoming more aggressive) in order to enhance the estimation/identification while controlling the system. Following this a classification of stochastic control problems according to the dominant effect is discussed. This is then used to point out which are the stochastic control problems where substantial improvements can be expected from using a sophisticated algorithm versus a simple one.