Trouble Dashboard: A Distributed Failure Monitoring System for High-End Computing

Failure management is crucial for high performance computing systems, especially when the complexity of applications and underlying infrastructure has grown sharply in recent years. In this paper, we present the design, implementation and experiment of trouble dashboard (TD), an adaptive, flexible, and low overhead failure monitoring system. Our goal is to provide a lightweight, scalable failure-monitoring tool for both application scientists and system managers. In TD, a set of APIs is provided for application scientists to control the behavior of their applications with flexibility when failures happen. System managers can use the tool to monitor the status of not only computing nodes and running tasks but also failures when they occur. Experiments show that TD incurs low overhead, and remains accurate and flexible enough to adapt to various applications.