HERO: Heterogeneity-aware erasure coded redundancy optimal allocation for reliable storage in distributed networks

Heterogeneity is the natural feature in distributed networks. Different from the traditional disk array, the amount of data allocated on heterogenous peers may be not the same. To maximize the reliability of stored data objects in heterogeneous networks, the optimal allocation of erasure-coded fragments is a challenging problem constrained with heterogeneous peer availabilities and redundancy overhead. This paper examines this optimal problem considered MDS erasure codes applied into distributed storage networks. First, we model the reliability of an allocation with the weighted-k-out-of-s model and extend its properties to efficiently calculate the reliability of an allocation; then we reduce the reliability computation of a given allocation to linear computation cost based on the weighted k-out-of-s model. Then, we deduce the problem to integer partition problem and propose two order-based search algorithms. Our experiments show that our proposed algorithms can be applied to find the optimal allocations efficiently in various practical coding cases. Furthermore, we evaluate the performance of our proposed search algorithms with some practical storage settings, and then present experimental results including the reliability, redundancy overheads and allocation pattern for the optimal allocation driven by practical network traces.