Towards evidence in software engineering

The aggregation of studies is of growing interest for the empirical software engineering community, since the numbers of studies steadily grow. We discuss challenges with the aggregation of studies into a common body of knowledge, based on a quantitative and qualitative evaluation of experience from the Experimental Software Engineering Network, ESERNET. Challenges are that the number of studies available is usually low, and the studies that exist are often too scattered and diverse to allow systematic aggregation as a means for generating evidence. ESERNET therefore attempted to coordinate studies and thus create research synergies to achieve a sufficiently large number of comparable studies to allow for aggregation; however, the coordination approach of ESERNET proved to be insufficient. Based on some lessons learned from ESERNET, a four-step procedure for evolving Empirical Software Engineering towards the generation of evidence is proposed. This consists of (1) developing a methodology for aggregating different kinds of empirical results, (2) establishing guidelines for performing, analyzing, and reporting studies as well as for aggregating the results for every kind of empirical study, (3) extract evidence, that is, apply the methodology to different areas of software engineering, and (4) package the extracted evidence into guidelines for practice.