CARIAL: Cost-Aware Software Reliability Improvement with Active Learning

In the context of field testing (operational testing) of software, we address the problem of balancing the potential reduction in failure risk that developers may achieve by reviewing captured test executions to identify failures (and by successfully debugging their causes) against the cost of reviewing the tests. To achieve a desirable balance, we propose a cost-sensitive active learning strategy. Our approach guides developers in selecting a sample of test executions to review and label, and it calls for them to profile execution dynamics and characterize the symptoms and relative severity levels of failures. Profiles, labels, failure symptoms, and severity levels are used by the active learner to construct and refine a mapping between examined and unexamined tests, on one hand, and possible defects, on the other hand. This mapping is used together with estimates of test review costs to guide the selection of additional tests. We evaluate our approach on three subject programs and show that it (1) produces reasonable predictions of risk reduction and (2) significantly improves severity-weighted reliability for each subject program, with relatively low developer effort.