Locality-aware Connection Management and Rank Assignment forWide-area MPI

We propose a connection management scheme that limits the number of inter-cluster connections and forwards messages for processes that cannot communicate directly. We also propose a rank assignment scheme that finds rank-process mappings with low communication overhead by solving the quadratic assignment problem. Our proposed methods perform locality-aware communication optimizations, and do so without tedious manual configuration by obtaining latency and traffic information from a short profiling run of the environment and the application. Using these methods, we implemented a wide-area-enabled MPI library called MC-MPI, and evaluated its performance by running the NAS parallel benchmarks on 256 real nodes distributed across 4 clusters. MC-MPI was able to limit the number of process pairs that established connections to just 10% without suffering a performance penalty. Moreover, MC-MPI was able to find rank assignments that resulted in up to 160% better performance than locality-unaware assignments.