Title :
The Power and Challenges of Transformative I/O
Author :
Manzanares, Adam ; Bent, John ; Wingate, Meghan ; Gibson, Garth
Author_Institution :
Los Alamos Nat. Lab., Los Alamos, NM, USA
Abstract :
Extracting high data bandwidth and metadata rates from parallel file systems is notoriously difficult. User workloads almost never achieve the performance of synthetic benchmarks. The reason for this is that real-world applications are not as well-aligned, well-tuned, or consistent as are synthetic benchmarks. There are at least three possible ways to address this challenge: modification of the real-world workloads, modification of the underlying parallel file systems, or reorganization of the real-world workloads using Tran formative middleware. In this paper, we demonstrate that Tran formative middleware is applicable across a large set of high performance computing workloads and is portable across the three major parallel file systems in use today. We also demonstrate that our transformative middleware layer is capable of improving the write, read, and metadata performance of I/O workloads by up to 150x, 10x, and 17x respectively, on workloads with processor counts of up to 65,536.
Keywords :
file organisation; meta data; middleware; parallel processing; high performance computing workload; metadata rate; parallel file system; synthetic benchmark; tranformative middleware; transformative I/O; user workload; Aggregates; Bandwidth; Concurrent computing; Containers; Indexes; Libraries; Middleware; Checkpointing; High Performance Computing; Parallel Computing; Parallel File Systems & I/O;
Conference_Titel :
Cluster Computing (CLUSTER), 2012 IEEE International Conference on
Conference_Location :
Beijing
Print_ISBN :
978-1-4673-2422-9
DOI :
10.1109/CLUSTER.2012.86