Sequential selection approach for selecting the best system

Consider the problem of selecting the best simulated system with high probability, from a finite and huge set of alternative systems. The best system might be the one that has the maximum or minimum performance measure. In this paper, we present a sequential method that uses the Ordinal Optimization procedure to select randomly a subset that overlaps with the set of the actual best m% systems with high probability from the search space. The next step, we use Optimal Computing Budget Allocation technique to allocate the available computing budget in a way that maximizes the probability of correct selection. This follows by a Subset Selection procedure to get a smaller subset that contains the best system from the subset that is selected before. Finally, we use the Indifference-Zone procedure to select the best system among the survivors in the previous stage. The results of the empirical experiments show that this approach selects the best simulated system with high probability and a minimum number of simulation replication, when the number of alternatives is huge.