Efficient index lookup for De-duplication backup system

We minimizes fingerprint management overhead (index lookup and index insert) via introducing main memory index lookup structure and workload-aware index partitioning of the index file in the storage. Backup server maintains three data structures for redundancy elimination: Header files, chunk files and fingerprint tables. These data structures altogether enables PRUNE to effectively eliminate redundancy and to perform efficient backup. We perform various experiment to measure the overhead of each tasks in backup operation and to examine the efficiency of redundancy elimination. Incremental modulo-K reduces the file chunking latency by approximately 60%. With filter based in-memory index data structure and index partitioning, PRUNE eliminates 99.4% of disk accesses involved in fingerprint management.