Hierarchical gather/scatter algorithms with graceful degradation

Summary form only given. We present and implement simple, binomial-tree based algorithms for the gather and scatter operations of MPI (the message passing interface). For small data sets, data are gathered (scattered) in a tree-like fashion. As the size of the data increases, the algorithms gracefully degrade toward the serial algorithm in which the root process gathers (scatters) data from (to) one process after the next. We extend these algorithms to the more difficult irregular gather/scatter operations in which the processes send/receive different amounts of data. The algorithms are furthermore adopted to the hierarchical communication structure of SMP-clusters. We compare the new algorithms to the straightforward, serial implementations of the gather/scatter primitives, and demonstrate substantial improvements both on a 32-node, 2-way SMP cluster, and on a 4-node NEC SX-6 vector supercomputer with 8 processors per node. For the regular gather/scatter operations improvements of a factor of 3 to 7 are achieved for critical data sizes on the SMP-system, and a factor of 3 to 4 on the SX-6. On 256 nodes of the earth simulator the improvement for scattering small data is more than a factor of 60. Comparable improvements are achieved for the irregular operations, despite preprocessing and communication overhead for dynamic tree construction. We discuss issues in modeling and analyzing the performance of the algorithms for the irregular collectives in particular.