VisDSI: Locality Aware I/O Solution for Large Scale Data Visualization

The computational science community is approaching petascale level simulations that will produce massive amount of datasets. While the computational power of supercomputers keep increasing, the I/O systems have not kept pace, resulting in a significant performance bottle neck. We propose a solution, VisDSI, to address the problem by 1) using traditional high performance clusters with disks directly attached within each node, 2) deploying a data-intensive distributed file system on the cluster, and 3) developing a POSIX-compatible I/O layer to enable the traditional visualization applications to smoothly port to this new platform as well as to provide a new I/Osemantic of retrieving the data location with respect to the POSIX standards. In particular, VisDSI guarantees the colocated compute and data storage by introducing a scheduling of work assignments to nodes with local copies of needed data. Compared with the original visualization application, our solution runs at least 10 times faster.