Design and Implementation of an RDMA Gateway for Heterogeneous Clusters

Building high-performance clusters using one of the two leading network technologies, Myrinet and InfiniBand, has been thought as a de facto way to achieve several teraflops computing power. Meanwhile, maintaining both types of clusters, it appears, may have created an another challenge for the MPI programming system, the most popular parallel programming library that has been successfully used on both networks. The belief that extending cluster resources across two different types of networks may increase computing parallelism has driven many researchers to tackle this challenge with various viewpoints. We approach this challenge with a different perspective, application transparency, which is accomplishing the goal without any modification of legacy MPI applications. We, therefore, focus on the design of an RDMA gateway that can relay messages very fast, and this design focus turns out to be a better way to preserve the application transparency. RDMA gateway (RG), our prototyped system, has a very efficient memory management mechanism that prevents RG from showing irregular spikes of a memory usage under a heavy load condition. Experimental results show that running parallel applications over heterogeneous clusters can be very promising with low performance overhead.