On the Utility of a Defect Prediction Model during HW/SW Integration Testing: A Retrospective Case Study

Testing is an important and cost-intensive part of the software development life cycle. Defect prediction models try to identify error-prone components, so that these can be tested earlier or more in-depth, and thus improve the cost-effectiveness during testing. Such models have been researched extensively, but whether and when they are applicable in practice is still debated. The applicability depends on many factors, and we argue that it cannot be analyzed without a specific scenario in mind. In this paper, we therefore present an analysis of the utility for one case study, based on data collected during the hardware/software integration test of a system from the avionic domain. An analysis of all defects found during this phase reveals that more than half of them are not identifiable by a code-based defect prediction model. We then investigate the predictive performance of different prediction models for the remaining defects. The small ratio of defective instances results in relatively poor performance. Our analysis of the cost-effectiveness then shows that the prediction model is not able to outperform simple models, which order files either randomly or by lines of code. Hence, in our setup, the application of defect prediction models does not offer any advantage in practice.