Evaluating the impact of the simulation environment on experimentation results

Simulation has become the tool of choice for advanced computer architecture research. Simulation model abstraction as well as compiler specifics can have significant impact on the observed performance trends and may lead to incorrect conclusions. This study presents a methodology that compares the predicted performance impact of several architectural techniques measured on a variety of simulator and compiler configurations. A significant difference in speedup between experiments indicates that a technique is sensitive to this particular variation. The results demonstrate a strong but non-uniform relationship between modeling abstraction and measurement error, as well as a significant sensitivity to the compiler.