Title :
Sufficiency of Markov policies for continuous-time Markov decision processes and solutions to Kolmogorov´s forward equation for jump Markov processes
Author :
Feinberg, Eugene A. ; Mandava, Manasa ; Shiryaev, Albert N.
Author_Institution :
Fac. of Dept. of Appl. Math. & Stat., Stony Brook Univ., Stony Brook, NY, USA
Abstract :
In continuous-time Markov decision processes (CTMDPs) with Borel state and action spaces, unbounded transition rates, for an arbitrary policy, we construct a relaxed Markov policy such that the marginal distribution on the state-action pairs at any time instant is the same for both the policies. This result implies the existence of a relaxed Markov policy that performs equally to an arbitrary policy with respect to expected discounted and non-discounted total costs as well as average costs per unit time. The proof consists of two steps. The first step describes the properties of solutions to Kolmogorov´s forward equation for jump Markov Processes. The second step applies these results to CTMDPs.
Keywords :
Markov processes; decision making; Borel state; CTMDP; Kolmogorov forward equation; Markov policies; action spaces; continuous-time Markov decision processes; discounted total costs; jump Markov processes; marginal distribution; nondiscounted total costs; state-action pairs; unbounded transition rates; Aerospace electronics; Equations; Kernel; Markov processes; Process control; Trajectory;
Conference_Titel :
Decision and Control (CDC), 2013 IEEE 52nd Annual Conference on
Conference_Location :
Firenze
Print_ISBN :
978-1-4673-5714-2
DOI :
10.1109/CDC.2013.6760792