Non-monotonicity of probability of correct selection

In Peng et al. (2015b), we show that the probability of correct selection (PCS), a commonly used metric, is not necessarily monotonically increasing with respect to the number of simulation replications. A simple counterexample where the PCS may decrease with additional sampling is provided to motivate the problem. The reference identifies the induced correlations as the source of the non-monotonicity, and characterizes the general scenario under which the phenomenon occurs by a condition where coefficient of variations of the difference in sample means are large. Numerical examples further illustrate the non-monotonic behavior of the PCS for some well-known sampling schemes.