Finite horizon optimal memoryless control of a delay in Gaussian noise: A simple counterexample

In this paper, we investigate control strategies for a scalar, one-step delay system in discrete time, i.e., the system?s state is the input delayed by one time unit. We allow control policies that are memoryless functions of noisy measurements of the state of the delay system. We adopt a first order state-space representation for the delay system, where the initial state of the system is a Gaussian and zero mean random variable. In addition, we assume that the measurement noise is drawn from a white and Gaussian process with zero mean and constant variance. Performance evaluation is carried out via a finite-time quadratic cost that combines the second moment of the control signal, and the second moment of the difference between the initial state and the state at the final time. We show that if the time-horizon is one or two then the optimal control is a linear function of the plant?s output, while for a sufficiently large horizon a control taking only two values will outperform the optimal linear solution. This paper complements the well known counterexample by Hans Witsenhausen, which showed that the solution to a linear, quadratic and Gaussian optimal control paradigm might be nonlinear. Witsenhausen?s counterexample considered an optimization horizon with two time-steps (two stage control). In contrast to Witsenhausen?s work, the solution to our counterexample is linear for one and two stages but it becomes nonlinear as the number of stages is increased. Existing tests for linearity of the optimal memoryless control consider only the two-stage problem. The fact that our paradigm leads to non-linear solutions, in the multi-stage case, could not be predicted from prior results. In particular, the fact that the optimal solution for the two stage problem is linear, but the multiple stage might not be, also shows that dynamic programming principles cannot be used for our paradigm. Our paper provides analytical proofs which hold for any number of stages.