A two-phase screening procedure for simulation experiments

Analysts examining complex simulation models often conduct screening experiments to identify the most important factors. Controlled sequential bifurcation (CSB) is a screening procedure, developed specifically for simulation experiments, that uses a sequence of hypothesis tests to classify the factors as either important or unimportant. CSB controls the probability of type I error for each factor, and the power at each bifurcation step, under heterogeneous variance conditions. CSB does, however, require the user to correctly state the directions of the effects prior to running the experiments. Experience indicates that this can be problematic with complex simulations. We propose a hybrid two-phase approach, FF-CSB, to relax this requirement. Phase 1 uses an efficient fractional factorial experiment to estimate the signs and magnitudes of the effects. Phase 2 uses these results in controlled sequential bifurcation. We describe this procedure and provide an empirical evaluation of its performance.