A quantitative study on execution time variability in computing experiments

Several modeling, simulation and experimental research works in computer science and engineering depend on correctly measuring the execution time of computer programs. It is observed that not everyone takes into account that repeated executions of the program with the same input can result in execution times statistically significant different. The lack of rigor in the analysis of execution times of computer programs has been investigated in several studies in the literature. In this work, we first reproduce experiments from the literature in order to analyze the statistical properties of their results in terms of execution times, as well as to assess the effects of different variability sources in influencing the execution times. Particularly, we consider variability sources related to the operating system. We also propose a protocol to systematize the comparison of programs´ execution times in order to identify the significant differences in samples obtained from experiments with multiple treatments.