Tool for performance tuning and regression analyses of HPC systems and applications

Increasing sophistication in High Performance Computing (HPC) system architectures, software, and user environments has substantially increased its complexity. For example, tuning an application on a given platform to maximize performance requires playing with multitude of different optimization flags and environment variables. This is typically a highly repetitive and an ad hoc process of trying out different combinations of variable settings and manually comparing the results to find the optimal. Similar is the case for performance regression analyses of systems and applications, where one is interested in detecting performance regressions in the software version under test and analyzing the causes to fix issues. In both of these scenarios, the process involves creating a patchwork of scripts to deploy jobs, extract meaningful data from raw outputs, arrange this data in some reportable format to be able to analyze, and perform tweaks to enable subsequent iterations. When repeated over time, the ad-hoc process results in users re-writing similar set of scripts again and again for different applications, or architectures, or even new software builds on the same architecture resulting in significant wastage of productive man-hours. This paper presents JACE (Job Auto-creator and Executor), a tool that enables automation of creation and execution of complex functional and performance regression tests. JACE aims to address many pain points in performance engineering such as tedious scripting, parameter tuning, careful book-keeping, frequent debugging, assessing reliability of results, and comparative evaluation. The paper introduces the tool and describes its architecture and workflow. It also presents a sample walk-through of performance regression analyses using JACE.