Analysis of design considerations for optimizing multi-channel MPI over InfiniBand

Modern day MPI implementations provide several communication channels for optimizing performance. To obtain the best performance for the most demanding contemporary applications, it becomes critical to manage these communication channels efficiently. Various issues related to overhead for message discovery and thresholds for choosing different channels need to be considered for designing the MPI layer. It is not a trivial task to choose these parameters since application characteristics and demands from the MPI layer vary widely. In this paper we try to address these issues. We propose several different schemes such as static priority and dynamic priority to efficiently implement channel polling. Our results indicate that we can reduce intra-node latency by up to 12% and message discovery time up to 45%. Further, we explore several different methodologies to choose appropriate thresholds for different channels.