Benchmark-Based Aggregation of Metrics to Ratings

Software metrics have been proposed as instruments, not only to guide individual developers in their coding tasks, but also to obtain high-level quality indicators for entire software systems. Such system-level indicators are intended to enable meaningful comparisons among systems or to serve as triggers for a deeper analysis.Common methods for aggregation range from simple mathematical operations (e.g. addition and central tendency) to more complex methodologies such as distribution fitting, wealth inequality metrics (e.g. Gini coefficient and Theil Index) and custom formulae.However, these methodologies provide little guidance for interpreting the aggregated results or to trace back to individual measurements.To resolve such limitations, a two-stage rating approach has been proposed where (i) measurement values are compared to thresholds to summarize them into risk profiles, and (ii) risk profiles are mapped to ratings.In this paper, we extend our approach for deriving metric thresholds from benchmark data into a methodology for benchmark-based calibration of two-stage aggregation of metrics into ratings.We explain the core algorithm of the methodology and we demonstrate its application to various metrics of the SIG quality model, using a benchmark of 100 software systems.We present an evaluation of the sensitivity of the algorithm to the underlying data.