A cost-benefit stopping criterion for statistical testing

Determining when to stop a statistical test is an important management decision. Several stopping criteria have been proposed, including criteria based on statistical similarity, the probability that the system has a desired reliability, and the expected cost of remaining faults. This paper proposes a new stopping criterion based on a cost-benefit analysis using the expected reliability of the system (as opposed to an estimate of the remaining faults). The expected reliability is used, along with other factors such as units deployed and expected use, to anticipate the number of failures in the field and the resulting anticipated cost of failures. Reductions in this number generated by increasing the reliability are balanced against the cost of further testing to determine when testing should be stopped.