The Impact of Equivalent Mutants

If a mutation is not killed by a test suite, this usually means that the test suite is not adequate. However, it may also be that the mutant keeps the programpsilas semantics unchanged-and thus cannot be detected by any test.We found such equivalent mutants to be surprisingly common: In an experiment on the JAXEN XPATH query engine, 8/20 = 40% of all mutations turned out to be equivalent. Worse, checking the equivalency took us 15 minutes for a single mutation. Equivalent mutants thus make it impossible to automatically assess test suites by means of mutation testing. To identify equivalent mutants, we are currently investigating the impact of a mutation on the execution: the more a mutation alters the execution, the higher the chance of it being non-equivalent. First experiments assessing the impact on code coverage are promising.