A Comparison of Different Defect Measures to Identify Defect-Prone Components

(Background) Defect distribution in software systems has been shown to follow the Pareto rule of 20-80. This motivates the prioritization of components with the majority of defects for testing activities. (Research goal) Are there significant variations between defective components and architectural hotspots identified by other defect measures? (Approach) We have performed a study using post-release data of an industrial Smart Grid application with a well-maintained defect tracking system. Using the Pareto principle, we identify and compare defect-prone and hotspots components based on four defect metrics. Furthermore, we validated the quantitative results against qualitative data from the developers. (Results) Our results show that at the top 25% of the measures 1) significant variations exist between the defective components identified by the different defect metrics and that some of the components persist as defective across releases 2) the top defective components based on number of defects could only identify about 40% of critical components in this system 3) other defect metrics identify about 30% additional critical components 4) additional quality challenges of a component could be identified by considering the pair wise intersection of the defect metrics. (Discussion and Conclusion) Since a set of critical components in the system is missed by using largest-first or smallest-first prioritization approaches, this study, therefore, makes a case for an all-inclusive metrics during defect model construction such as number of defects, defect density, defect severity and defect correction effort to make us better understand what comprises defect-prone components and architectural hotspots, especially in critical applications.