HEDS: Hybrid deduplication approach for email servers

While the volume of emails are ever-increasing, email servers tend to have large redundancies demanding huge storage unnecessarily. MS exchange single instance (SIS) attempts storage-efficiency by removing redundant copies of an entire email-level when it is destined for multiple recipients´ mailboxes within a server. However, redundant attachments or parts of emails due to threads are left storage-inefficient by SIS. On the other hand, existing deduplication systems are mainly for backup systems, and execute deduplication in block-level that makes it difficult to be used in in-line systems. We propose a hybrid scheme that adaptively performs deduplication at the granularity of either the file-level or chunk-level. We have designed and implemented a hybrid email deduplicaiton system, named HEDS, and evaluated it with real email datasets.We show that it achieves a high data reduction rate while keeping the CPU and memory overhead small;thus, it is feasible to be used as an in-line system on email servers.