Performance implications of deadlock avoidance techniques in torus networks

Deadlock free routing techniques for torus topologies have been a subject of deep study in the field of HPC interconnects and many proposals exist in the literature. Practical deadlock avoidance techniques can be classified into two main categories, requiring either a segregation of traffic in non-cyclic virtual networks or some form of injection control. Simulating large high-dimension tori networks using application traffic is challenging. Most proposals use either large low-dimension tori, or synthetic traffic. Currently, tori of five and six dimensions are being used in actual supercomputers, such as the Fujitsu K Computer, which was ranked first in the Top 500 in two consecutive lists (June 2011 and November 2011). To our knowledge, there are no published papers comparing the performance implications of deadlock avoidance techniques for large high-dimension tori using traffic typical of parallel applications. We chose two well established deadlock-avoidance techniques in tori with dimension-order routing, dateline resource allocation and bubble injection restriction. The simulation tools had to be adapted to scale to simulate these large networks. In this paper we analyze network performance for tori of up to 6 dimensions comprising up to 4096 nodes when dealing with both synthetic and HPC-specific workloads.