Confidence: Analyzing performance with empirical probabilities

Variability in the performance of shared system components is a major obstacle in analyzing the effective throughput of leadership class computers. Shared file systems and networks are serious impediments to achieving repeatable application performance on HPC systems. In particular, performance analysts are likely to be interested in quantifying differences between average-case behavior, worst-case behavior, and standard deviation for shared system components. Typical descriptions of these statistics assume a normal distribution; however, in one-sided and multi-modal performance distributions, summary statistics are often misleading. In this paper we describe Confidence, a tool for analyzing the full spectrum of performance for a benchmarking code. By including all of the experimental outcomes in the analysis without discarding any measurements, Confidence enables a novel analysis of benchmark performance.