Recommender systems for manual testing

BACKGROUND: Software testing can be an arduous and expensive activity. A typical activity to maximise testing productivity is to allocate test cases according to the testers´ profile. However, optimising the allocation of manual test cases is not a trivial task: in big companies, test managers are responsible for allocating hundreds of test cases among several testers. OBJECTIVE: In this paper we propose and evaluate 2 assignment algorithms for test case allocation and 3 tester profiles based on recommender systems. Each assignment algorithm can be combined with 3 tester profiles, which results in six possible allocation systems. METHOD: We run a controlled experiment that uses 100 test suites, each one with at least 50 test cases, from a real industrial setting in order to compare our allocation systems to the manager´s allocation in terms of precision, recall and unassignment (percentage of test cases the algorithm could not allocate). RESULTS: In our experiment, the statistical analysis shows that one of the systems outperforms the others with respect to the precision and recall metrics. For unassignment, three of our six allocation systems achieved zero (best value) for the unassignment rate. CONCLUSION: The results of our experiment suggest that, in similar environments, test managers can use our allocation systems to reduce the amount of time spent in the test case allocation task. In the real industrial setting in which our work was developed, managers spend from 16 to 30 working days a year on test case allocation. Our algorithms can help them do it faster and better.