Stochastic control with graphical models: the influence view approach

Markov decision processes (MDPs) allow to represent a wide class of problems in medical decision making and control. The complexity of the algorithms used to search the best policy of a MDP is directly related with the dimensionality of the state space. A careful structuring of the state space is hence an important task in the MDP specification. Graphical models are particularly appealing to cope with this task. Here, the authors describe a novel graphical formalism for MDP knowledge acquisition called Influence View (IV). An IV is a directed acyclic graph that depicts the probabilistic relationships between the problem state variables in a generic time transition; additional variables, called event variables, may be added, in order to describe the conditional independencies between state variables. By using the specified conditional independence structure, an TV may allow a parsimonious specification of a MDP. The authors have applied this methodology to the GVHD prophylaxis after Bone Marrow Transplantation