Comparison of Unit-Level Automated Test Generation Tools

Data from projects worldwide show that many software projects fail and most are completed late or over budget. Unit testing is a simple but effective technique to improve software in terms of quality, flexibility, and time-to-market. A key idea of unit testing is that each piece of code needs its own tests and the best person to design those tests is the developer who wrote the software. However, generating tests for each unit by hand is very expensive, possibly prohibitively so. Automatic test data generation is essential to support unit testing and as unit testing is achieving more attention, developers have a greater need for automated unit test data generation tools. However, developers have very little information about which tools are effective. This experiment compared three well-known public-accessible unit test data generation tools, JCrasher, TestGen4j, and JUB. We applied them to Java classes and evaluated them based on their mutation scores. As a comparison, we created two additional sets of tests for each class. One test set contained random values and the other contained values to satisfy edge coverage. Results showed that the automatic test data generation tools generated tests with almost the same mutation scores as the random tests.