Data de duplication using N0SQL Databases in Cloud

Data de-duplication has an important role in reducing storage consumption to make it affordable to manage in today´s explosive data growth. Numerous DD methodologies like chunking and, delta encoding are available to optimize the use of storage. These technologies approach DD at file and/or sub-file level but this approach has never been optimal for NoSQL DBs. This method proposes data De-Duplication in NoSQL Databases (DDNSDB) which makes use of a DD approach at a higher level of abstraction. The main goals of this research is, to maximally reduce the amount of duplicates in one type of NoSQL DBs, namely the key-value store, to maximally increase the process performance such that the backup window is marginally affected, and to design with horizontal scaling in mind such that it would run on a Cloud Platform competitively. Also, by following an optimized adapted Map Reduce architecture, DDNSDB proves to have competitive performance advantage in a horizontal scaling cloud environment compared with a vertical scaling environment.