Energy Efficient Hadoop Using Mirrored Data Block Replication Policy

MapReduce scheme has became the state of the art in parallel processing of vast amount of data in distributed systems. Hadoop, as a popular open-source implementation of this technique, makes use of data block replication mechanism to provide a reliable and fault-tolerant design. To maintain data availability, Hadoop takes into account the possibilities of node and rack failures. Hence, it stores multiple copies of each data block to ensure availability and reliability. The current data block placement policy is to randomly distribute the replicas on all servers, satisfying some constraints such as preventing storage of two replicas of a data block on a single node. Our study proposes an efficient placement policy for data block replicas, which can reduce the consumed energy in data centers. The proposed policy is built upon the covering subset (CovSet) method. The effectiveness of the proposed approach is confirmed through simulations. Also, our experiments show that the proposed method becomes more effective whenever the average number of data blocks per server increases, which corresponds to the actual conditions in practice.