An Offline Analysis Framework of Cluster System Logs

The growth of computing and storage needs of several scientific applications mandate the deployment of extreme-scale parallel machines, such as Blue Gene/L, Spirit, Liberty, Red Storm and etc. One of the challenges when designing and deploying these systems in a production setting is the need to take failure occurrences into account. In this paper, an offline analysis framework of cluster system logs is designed and implemented which includes four main parts, such as log formatting module, log filtering module, a central database and log mining module. Finally, four cluster system logs are analyzed in the temporal and spatial statistical characteristics of error events.