Pre-processing directory structure for improved RSYNC transfer performance

This study deals with the problem of maintaining large hierarchy folder replicated in a distributed environment. It was found that this problem afflicted a number of important applications, such as synchronization of Hierarchy Folders between Peer-to-peer environments, synchronization of data between accounts or devices, content distribution and web caching networks, web site mirroring, storage networks, and large scale web search and mining. At the core of the problem lay the File in Hierarchy Folder synchronization challenge. This challenge posed the question; “Given two versions of files inside Folders on different machines, call outdated hierarchy and a current one, how can we update the outdated version with a minimum communication cost, by exploiting the significant similarity between the versions?” Although a popular open source tool for this problem called RSYNC is being used in hundreds of thousands of servers around the world, only very few attempts have been made to improve this tool in practice. In this paper, we propose a framework for remote file synchronization and describe several new techniques that result in significant time savings and less error during files transfer. Our focus is on applications that involve very large collections of Hierarchy Folders including files and folders. We show that a prototype implementation of our framework and techniques achieves significant improvements over RSYNC. As an example application, we focused on the capable synchronization of large folder transfer between peer-to-peer distributions. This method is provided for synchronizing folders between the server which is updated by some client and other servers. Our algorithm adopts the Sender/Receiver Server structure based on a client modification.