Methodologies and application of machine learning algorithms to classify the performance of high performance cluster components

High Performance Computing Clusters are designed to host highly parallelized applications, often in excess of thousands of nodes allocated to a job. These jobs, especially those that require a high level of synchronous communication, can be greatly affected by a single poor, or even sub-standard performing component. These components, often referred to as a node, are typically comprised of CPUs, accelerator processors, memory, a communication bus, and so on. Consequently it is important to identify and eliminate these sub-standard performing nodes before a job is scheduled onto them. In this paper we will describe the process used to measure and the methodology used to quantify poor performing nodes or classify suspect performing nodes into groups, or clusters, that can be later used to identify future performance issues. This process is more involved than simply running a scientific calculation across all the nodes, finding one that was “slow”, and labeling it as a bad node. At Los Alamos, this methodology has been used successfully to find problem nodes and has helped characterize the components of other clusters to aid in the proactive elimination of potential problems.