Some graphical aids for deciding when to stop testing software

It is noted that the developers of large software systems must decide how much software should be tested before releasing it. An explicit tradeoff between the costs of testing and releasing is considered. The former may include the opportunity cost of continued testing, and the latter may include the cost of customer dissatisfaction and of fixing faults found in the field. Exact stopping rules were obtained by Dalal and Mallows (J. Amer., Statist. Assoc., vol.83, p.872, 1988), under the assumption that the distribution of the fault finding rate is known. Here, two important variants where the fault finding distribution is not completely known are considered. They are (i) the distribution is exponential with unknown mean and (ii) the distribution is locally exponential with the rate changing smoothly over time. New procedures for both cases are presented. In case (i) it is shown how to incorporate information from related projects and subjective inputs. Several novel graphical procedures which are easy to implement are proposed, and these are illustrated for data from a large telecommunications software system