A GPDMA-based Distributed Shared I/O Solution for CC-NUMA System

A distributed shared I/O solution for CC-NUMA (cache coherent non-uniform memory access) system is introduced. To improve the remote I/O performance, GPDMA (globally parallel direct memory access) technique is introduced, enable I/O device to directly read/write not only the local memory, but also remote distributed shared memory (DSM). Taking advantage of GPDMA, file system is able to use optimized data transfer protocol to bypass the storage server cache during the data transfers. Different from traditional RDMA communication technique, GPDMA technique minimizes the number of I/O specific messages between the nodes, avoids data copy from server cache to client cache. The data consistency between memory and its cache is guaranteed by hardware. SGP6, a 2048-processor CC-NUMA supercomputer and CC-GPFS, a global parallel file system have been developed. The evaluation results show that the GPDMA-based distributed shared I/O solution improve the I/O performance greatly.