Empirical Utilization Analysis for High Performance and Grid Computing

An important Key Performance Indicator (KPI) for high performance clusters (HPC) is the average resource utilization. It is used as an indicator for the cost effectiveness of investing in a costly HPC infrastructure. Means of measuring utilization must be done as accurately as possible without adding overhead to the system to portray the ability of clusters to perform optimally at all levels, and to easily highlight the effects of external factors on their performance. By taking advantage of the HPC job scheduler´s accounting and the agents of a monitoring software on compute nodes, we provide a solution to automate the process of obtaining and calculating clusters´ daily and monthly utilizations. Our mechanism provides means to identify compute nodes that are online, offline, online nodes running jobs, and the desired KPI which is node occupancy. Our HPC KPI calculation mechanism can be generalized and applied on any general-purpose HPC platform of any size.