Using fit-constrained Monte Carlo trials to quantify confidence in simulation model outcomes

System dynamics-based simulation models are useful for analyzing complex systems characterized by both large parameter spaces and pervasive nonlinearity. Unfortunately, these characteristics also make confidence intervals for the model´s outcomes difficult to assess. Standard Monte-Carlo testing with a-priori realistic parameter variations produces simulated behavior that is a-posteriori improbable, rendering simple approaches inappropriate for establishing confidence intervals. This paper gives a case study of a model that is used to forecast the completion of the design and construction of a defense platform. A confidence interval for the outcome is computed by using Monte-Carlo trials and discarding combinations that do not achieve an acceptable fit of simulated behavior to historical data. The vast majority of simulation trials were markedly dissimilar to actual program history. More than 50,000 trials were required (with parameter variations far smaller than the a-priori bounds) to create about 70 trials that met the fit criteria. For this case, the experiments confirmed the intuitive view that a well-formulated closed-loop model calibrated against sparse but widespread data and an appropriate statistical fit criterion can create tight confidence intervals on some model outcomes.