Experience from replicating empirical studies on prediction models

When conducting empirical studies, replications are important contributors to investigating the generality of the studies. By replicating a study in another context, we investigate what impact the specific environment has, related to the effect of the studied object. In this paper, we define different levels of replication to characterise the similarities and differences between an original study and a replication, with particular focus on prediction models for the identification of fault-prone software components. Further, we derive a set of issues and concerns which are important in order to enable replication of an empirical study and to enable practitioners to use the results. To illustrate the importance of the issues raised, a replication case study is presented in the domain of prediction models for fault-prone software components. It is concluded that the results are very divergent, depending on how different parameters are chosen, which demonstrates the need for well-documented empirical studies to enable replication and use