Challenges for High-Speed Protocol-Independent Redundancy Eliminating Systems

The idea of identifying and removing repetitive patterns in the raw network data transmissions, also known as protocol-independent redundancy elimination, has received thorough consideration in the academic community and also was integrated into several commercial appliances. The original intention of the redundancy elimination was to increase the capacity of low bandwidth network connections, utilizing the fact that to search for redundancies and replace them locally is faster than transmitting unprocessed redundant data. However, as network link capacities grow, it is not clear whether redundancy elimination will remain attractive the way it is being done. On one hand, no matter how it is implemented, redundancy elimination reduces network loads, which is obviously beneficial from many points of view, from queuing delays to power consumption. On the other hand, if redundancy elimination mechanism becomes a bottleneck instead of a network connection itself, there is no incentive for end-users to deploy it, and, from our experience, it is becoming a more common case lately. Thus, we believe it is time to re-think basic mechanisms behind the redundancy elimination in order to design scalable systems capable of catching up with higher network speeds. Proposing such a design, however, is not the goal of this paper, as the task seems fairly hard. Instead, in this paper we outline the major problems and limitations that are encountered by engineers in the attempts to design such systems. Thus, the goal of this paper is to start a discussion of high-speed redundancy eliminating systems in the academic community, which, we hope, will lead to exciting new breakthroughs in the area.