Accelerating Data Movement Leveraging End-System and Network Parallelism

Data volumes produced by simulation, experimental and observational science is rapidly increasing. This data needs to be moved from its source to another resource for analysis, visualization and archival purposes. The destination resource could be either local or remote. The data intensive science is critically dependent upon the high-performance parallel file and storage end systems to read/write and high-speed networks to move their enormous data between local and remote computing and storage facilities. 100 Gigabit per second networks such as DOE´s Advanced Network Initiative (ANI), Internet2´s 100G network represent a major step forward in wide area network performance. Effective utilization of these networks requires substantial and pervasive parallelism, at the file system, end system, and network levels. Additional obstacles such as heterogeneity and time-varying conditions of network and end system arise that, if not adequately addressed, will render high performance storage and network systems extremely under-performed. In this paper, we propose a data movement system that dynamically and adaptively adjusts end systems and networks parallelisms in response to changing conditions of end systems and networks to sustain high-throughput for data transfers. We evaluate our system in multiple settings and show that (1) in a homogeneous configuration, the design can achieve better throughput for light and medium workload than GridFTP and achieve comparable throughput for heavy workload, (2) and in a heterogeneous configuration, the design can achieve several factors higher throughput for all workloads than GridFTP.