Changeloads: A Fundamental Piece on the SASO Systems Benchmarking Puzzle

Benchmarks have been traditionally tailored to static, unchangeable systems, functioning in well-known and controlled environments. Thus, established benchmarks (and benchmarking approaches) are becoming progressively less representative of real world scenarios, as change is gaining emphasis as a fundamental player in computing systems runtime conditions. As today´s systems are becoming, at the very least, reactive to changes (either endogenous or exogenous) at some level, if not even proactive in reaching their goals more efficiently and effectively, we believe that benchmarks must also evolve, becoming applicable to systems that react to change, adapt, evolve, and have the capability to improve their own performance. In this position paper, we argue that representative change loads are now as vital as representative workloads, and that change load-based benchmarks will become a key part in the development and evaluation of SASO systems. We present and discuss some applications of benchmarks in this area, proposing some directions for research.