Estimating Fault Detection Effectiveness

A t-way covering array can detect t-way faults, however they generally include other combinations beyond t-way as well. For example, a particular test set of all 5-way combinations is shown capable of detecting all seeded faults in a test program, despite the fact that it contains up to 9-way faults. This poster gives an overview of methods for estimating fault detection effectiveness of a test set based on combinatorial coverage for a class of software. Detection effectiveness depends on the distribution of t-way faults, which is not known. However based on past experience one could say for example the fraction of 1-way faults is F₁ = 60 %, 2- way faults F₂ = 25 % F₃ = 10 % and F₄ = 5 %. Such information could be used in determining the required strength t. It is shown that the fault detection effectiveness of a test set may be affected significantly by the t-way fault distribution, overall, simple coverage at each level of t, number of values per variable, and minimum t-way coverage. Using these results, we develop practical guidance for testers.