CoRec: A Cooperative Reconstruction Pattern for Multiple Failures in Erasure-Coded Storage Clusters

It is indispensable to speed up a reconstruction process in erasure-coded storage clusters, because a fast data recovery helps to shorten the vulnerability window while improving storage system reliability. To address double- and multiple-node failures, this paper proposes a cooperative reconstruction pattern - CoRec - to minimize reconstruction traffic. CoRec not only enables all rebuilding nodes to collaboratively reconstruct failed blocks but also limits each surviving block to be transferred over network only once. To clarify two CoRec based reconstruction schemes (i.e., CoRec-rn and CoRec-sn), we investigate two alternative reconstruction schemes (i.e., CRec and DRec). We develop reconstruction-time models, which are validated using empirical data, to estimate reconstruction performance of large-scale storage clusters and to pinpoint performance bottlenecks in the reconstruction process. We implement a proof-of-concept prototype where the four reconstruction schemes are quantitatively evaluated. Experimental results show that CoRec-rn and CoRec-sn significantly reduce the reconstruction time of CRec and DRec. In a real-world 9-node storage cluster, CoRec-rn speeds up the double-node reconstruction of CRec and DRec by a factor of at least 1.72, CoRec-sn accelerates the double-node reconstruction of CRec and DRec by a factor of at least 4.76.