Benchmarking Testing Strategies with Tools from Mutation Analysis

The assessment of a testing strategy and the comparison of different testing strategies is a crucial part in current research on software testing. Often, manual error seeding is used to generate faulty programs. As a consequence, the results obtained from the examination of these programs are often not reproducible and likely to be biased. In this paper, a flexible approach to the benchmarking of testing strategies is presented. The approach utilizes well- known results from mutation analysis to construct an objective effectiveness measure for test oracles. This measure allows to draw conclusions not only on the effectiveness of a single testing strategy but also to compare different testing strategies by their effectiveness measure.