Judy - a mutation testing tool for java

Popular code coverage measures, such as branch coverage, are indicators of the thoroughness rather than the fault detection capability of test suites. Mutation testing is a fault-based technique that measures the effectiveness of test suites for fault localisation. Unfortunately, use of mutation testing in the software industry is rare because generating and running vast numbers of mutants against the test cases is time-consuming and difficult to do without an automated, fast and reliable tool. Our objective is to present an innovative approach to mutation testing that takes advantage of a novel aspect-oriented programming mechanism, called `pointcut and advice`, to avoid multiple compilation of mutants and, therefore, to speed up mutation testing. An empirical comparison of the performance of the developed tool, called Judy, with the MuJava mutation testing tool on 24 open-source projects demonstrates the value of the presented approach. The results show that there is a statistically significant (t(23) = -12.28, p < 0.0005, effect size d = 3.43) difference in the number of mutants generated per second between MuJava (M = 4.15, SD = 1.42) and Judy (M = 52.05, SD = 19.69). Apart from being statistically significant, this effect is considered very large and, therefore, represents a substantive finding. This therefore allows us to estimate the fault detection effectiveness of test suites of much larger systems.