Poster: PLFS/HDFS: HPC Applications on Cloud Storage

Long running large scale HPC applications protect themselves from failures by periodically checkpointing their state to a single file stored in a distributed network filesystem. These filesystems commonly provide a POSIX-style interface for reading and writing files. HDFS is a filesystem used in cloud computing by Apache Hadoop. HDFS is optimized for Hadoop jobs that do not require full POSIX I/O semantics. Only one process may write to an HDFS file, and all writes are appends. Our work enables multiple HPC processes to checkpoint their state into an HDFS file using PLFS. PLFS is a middleware filesystem that converts random I/O into log-based I/O. We added a new I/O store layer to PLFS that allows it to use non-POSIX filesystems like HDFS as backing store. HPC applications can now checkpoint to HDFS, allowing HPC and cloud to share the same storage systems and work with each others data.