Operating Two InfiniBand Grid Clusters over 28 km Distance

This paper considers an InfiniBand connection between two bwGRiD clusters over a distance of 28 km in day-to-day production use. We discuss the hardware setup of InfiniBand messages converted and transported over a fiber optic connection. The two clusters can be operated as single system image, the batch system will enforce that all nodes for a job are allocated on one side of the cluster. This is to optimize MPI performance, which would not be sufficient for communication between nodes on opposite sides of the 28 km connection. We report on the successful solution of all technical and organizational integration hurdles. By a simple performance model we discuss the anticipated costs for a doubling in communication performance.