TurboSockets: Democratizing Distributed Deduplication

Distributed deduplication is one of today´s most prominent techniques for efficient data transfer of data across a network. However, leveraging a distributed application with distributed deduplication capabilities is a complex challenge, not accessible to the average programmer. This paper advocates that the time has come to devise general-purpose middleware abstractions that can democratize the use of state-of-art distributed deduplication techniques, even by programmers with no know-how on the field. We propose TurboSockets, the first middleware abstraction that aims at such a goal. The TurboSockets middleware enables unskilled programmers to establish a communication channel between two remote processes and, through that channel, exchange data streams whose content is deduplicated by stateof-the-art algorithms. Turbosockets hides all the complexity associated with the deduplication protocol away from the programmer, closely resembling traditional inter-process communication APIs. Using a full-fledged prototype of the TurboSockets middleware, experimental results with real workloads confirm gains in performance and transferred volumes for a wide range of real workloads and scenarios.