Aggregation Encoding: Reducing Network Traffic for Big Data Erasure-Coded Storages

Erasure-coded storages provide data reliability for big data archiving and backup with high storage efficiency. However, erasure code generates a large amount of data transmission traffic in both encoding (write) and decoding (repair) which costs substantial bandwidth in the network. Most of the existing work focus on the decoding cost due to the importance of data repair, but unfortunately overlooks the encoding cost. In this paper, we propose Aggregation Encoding for reducing the bandwidth cost of the encoding phase in erasure-coded storages. Aggregation Encoding takes inspiration from network coding to enforce encoding through data transmission, which reduces redundant information transmitting. The extensive analysis and experimental results demonstrate that Aggregation Encoding reduces bandwidth cost by 20% in average.