Real Asynchronous MPI Communication in Hybrid Codes through OpenMP Communication Tasks

With the number of cores growing faster than memory per node, hybrid programming models (mixing message passing with shared memory paradigms) become a requirement for efficient use of HPC systems. For this scenario, achieving efficient communication is challenging. This is true even when using asynchronous communication, as most MPI implementations can only advance communication inside library calls. In this paper we propose to move communication into a new type of OpenMP task, which gets scheduled as part of the regular OpenMP work-pool. We show for compute intensive iterative stencil algorithms, that this provides real asynchronous communication. Without complicating the programming interface, our results show an excellent performance independent of the communication to computation ratio.