Modeling, predicting and optimizing redistribution between clusters on low latency networks

In this paper we study the problem of scheduling messages between two parallel machines connected by a low latency network during a data redistribution. We compare two approaches. In the first approach no scheduling is performed. Since all the messages cannot be transmitted at the same time, the transport layer has to manage the congestion. In the second approach we use two higher-level scheduling algorithms proposed in our previous work [E. Jeannot et al., (2004)] called GGP and OGGP. The contribution of this paper is the following: we show that the redistribution time with scheduling is always better than the brute-force approach (up to 30%). As this speedup depends on the input redistribution pattern, we propose a modelization of the behavior of both approaches and show that we are able to accurately predict the redistribution time with or without scheduling and thus able to choose for each pattern whether or not to schedule the communications.