eStor: Energy efficient and resilient data center storage

Much recent research work seeks to turn off servers and redirect I/O load for energy efficiency under low utilization period, but most of them are highly constrained by data layout policy. There are two widely used replica placement schemes. One is sequential data layout policy like in Chained Declustering, which allows to power down a large fraction of nodes without disrupting data availability, but has poor data rebuild parallelism to be resilient to node failure. Another is random data layout policy like in GFS and HDFS where its property is precisely opposite to that of the sequential policy. In this paper, we propose a hybrid data layout policy named eStor, which integrates the advantages of both sequential and random data layout policy. Under the eStor policy some replicas are placed in a sequential way, while other replicas are placed in a random fashion. It allows users to configure the system parameters of replication level and number of replicas placed in sequential way. By adjusting the system parameters, it can turn off a large amount of nodes without data loss and also has high data rebuild parallelism in case of failure. Experiments with eStor implemented in Hadoop demonstrate that it can mostly save 40% of the energy consumed and has high data rebuild rate.