Minimizing the Data Transfer Time Using Multicore End-System Aware Flow Bifurcation

Data centers are being deployed in a wide variety of environments (cloud computing, scientific, financial, defense, etc.). When geographically distributed, these data centers must transmit and receive growing volumes of data. In order to avoid congestion in the public internet, most use high speed dedicated optical networks, which can be thought of as private highways for carrying data. In this work, we examined the impact of such high speed network traffic on a commodity multicore machine, and identified a number of scenarios that cause packet loss and degraded throughput due to an end-system inability to consume incoming data fast enough. We show that high speed single flow traffic nullifies the benefits of multicore systems and multiqueue NICs, and we propose an end-system aware flow bifurcation technique to optimize the data transfer time using rate based protocols. Using introspective end-system modeling, we determine the optimal number of parallel flows required to utilize the available bandwidth, and the optimal rate for each of the flows. We compare our approach with GridFTP, which is a widely used data transfer protocol in computational grids, and show that our approach performs better (particularly when the end-system losses are in the receive ring buffer.).