Towards Scalable Application Checkpointing with Parallel File System Delegation

The ever-increasing scale of modern high-performance computing (HPC) systems presents a variety of challenges to the parallel file system (PFS) based storage in these systems. The scalability of application check pointing is a particularly important challenge because it is critical to the reliability of computing and it often dominates the I/Os in a HPC system. When a large number of parallel processes simultaneously perform check pointing, the PFS metadata servers can become a serious bottleneck due to the large volume of concurrent metadata operations. This paper specifically addresses this PFS metadata management issue in order to support scalable application check pointing in large HPC systems. It proposes a new technique named PFS-delegation which delegates the management of the PFS storage space used for check pointing to applications, thereby relieving the load of metadata operations on the PFS during their check pointing. This proposed technique is prototyped on PVFS2, a widely used PFS implementation, and evaluated on a HPC cluster using a representative parallel I/O benchmark, IOR. Experiments with up to 128 parallel processes show that the PFS-delegation based check pointing is significantly faster than the traditional shared-file and file-per-process based check pointing methods (7% and 10% speedup when the underlying PVFS2 uses a centralized metadata server, 22% and 31% speedup when using distributed metadata servers). The results also demonstrate that the PFS-delegation based check pointing substantially reduces the total number of metadata operations handled by the metadata servers during the check pointing.