Random Test Case Generation and Manual Unit Testing: Substitute or Complement in Retrofitting Tests for Legacy Code?

Unit testing of legacy code is often characterized by the goal to find a maximum number of defects with minimal effort. In context of restrictive time frames and limited resources, approaches for generating test cases promise increased defect detection effectiveness. This paper presents the results of an empirical study investigating the effectiveness of (a) manual unit testing conducted by 48 master students within a time limit of 60 minutes and (b) tool-supported random test case generation with Randoop. Both approaches have been applied on a Java collection class library containing 35 seeded defects. With the specific settings, where time and resource restrictions limit the performance of manual unit testing, we found that (1) the number of defects detected by random test case generation is in the range of manual unit testing and, furthermore, (2) the randomly generated test cases detect different defects than manual unit testing. Therefore, random test case generation seems a useful aid to jump start manual unit testing of legacy code.