Numerical methods for the solution of the degenerate nonlinear elliptic equations arising in optimal stochastic control theory

Three distinct but related results are obtained. First, an iterative method is derived for obtaining the solution of optimal control problems for Markov chains. The method usually converges much faster, and requires less computer storage space, than the methods of Howard or Eaton and Zadeh. Second, nonlinear finite difference equations, which "approximate" the nonlinear degenerate elliptic equation (2) arising out of the stochastic optimization problem (1), are found. The difference equations, and their solution, may have a meaning for the control problem even when it cannot be proved that (2) has a solution. The iterative methods for the iterative solution of these nonlinear systems are discussed and compared. Both converge to the solution, provided that the difference equations were derived using the method introduced in the paper; one, new to this paper, often much faster than the other (Theorem 2). In fact, the typical time required for the numerical solution is about the time required for a related linear problem. The method of obtaining the difference equations, and the proof of convergence of the associated iterative procedures, are illustrated by a detailed example. Finally, specific numerical results for a "minimum average time" type of optimization problem are presented and discussed.