Assessing the Ability of Computation/Communication Overlap and Communication Progress in Modern Interconnects

Computation/communication overlap is one of the fundamental techniques in hiding communication latency. Independent progress support in messaging layer, network interface offload capability and application usage of non-blocking communications are believed to increase overlap and yield performance benefits. In this paper, we analyze four MPI implementations on top of three high-speed interconnects (InfiniBand, Myrinet and iWARP Ethernet) in their ability to support overlap and communication progress. The results confirm that the offload ability needs to be supported with communication progress to increase the level of overlap. Our progress engine micro-benchmark results show that in all examined networks transferring small messages makes an acceptable level of progress and overlap. On the other hand, in most cases, transferring large messages does not make progress independently, decreasing the chances of overlap in applications.