Methods of comparing test data adequacy criteria

The comparative analysis of test data criteria in software testing is considered, and an attempt is made to investigate how criteria have been and should be compared to each other. It is argued that there are two fundamentally different goals in comparing criteria: (1) to compare the error-exposing ability of criteria, and (2) to compare the cost of using the criteria for selecting and/or evaluating test data. Relations such as the power relation and probable correctness are clearly in the first category, and test case counting is clearly in the second category. Subsumption, in contrast, is not entirely in either category. It is shown that the subsumption relation primarily compares the difficulty of satisfying two criteria. If one assumes that the criteria being compared are applicable, then one can infer their relative power and size complexities from the subsumption relation. In addition, it is shown that, while the size complexity of a criterion gives some indication of the relative cost of using the criterion, it is by no means a sufficient measure of the overall difficulty of using that criterion, which also includes the process of checking whether the predicate defined by the criterion has been satisfied, which may not only be difficult, but impossible