CONCEPTUAL: a network correctness and performance testing language

Summary form only given. We introduce a new, domain-specific specification language called CONCEPTUAL. CONCEPTUAL enables the expression of sophisticated communication benchmarks and network validation tests in comparatively few lines of code. Besides helping programmers save time writing and debugging code, CONCEPTUAL addresses the important-but largely unrecognized-problem of benchmark opacity. Benchmark opacity refers to the current impracticality of presenting performance measurements in a manner that promotes reproducibility and independent evaluation of the results. For example, stating that a performance graph was produced by a "bandwidth" test says nothing about whether that test measures the data rate during a round-trip transmission or the average data rate over a number of back-to-back unidirectional messages; whether the benchmark preregisters buffers, sends warm-up messages, and/or preposts asynchronous receives before starting the clock; how many runs were performed and whether these were aggregated by taking the mean, median, or maximum; or, even whether a data unit such as "MB/s" indicates 10⁶ or 2²⁰ bytes per second. Because CONCEPTUAL programs are terse, a benchmark\´s complete source code can be listed alongside performance results, making explicit all of the design decisions that went into the benchmark program. Because CONCEPTUAL\´s grammar is English-like, CONCEPTUAL programs can easily be understood by nonexperts. And because CONCEPTUAL is a high-level language, it can target a variety of messaging layers and networks, enabling fair and accurate performance comparisons.