• DocumentCode
    3588680
  • Title

    Enhancing scalability in distributed storage systems with Cauchy Reed-Solomon codes

  • Author

    Si Wu ; Yinlong Xu ; Yongkun Li ; Yunfeng Zhu

  • Author_Institution
    Sch. of Comput. Sci. & Technol., Univ. of Sci. & Technol. of China, Hefei, China
  • fYear
    2014
  • Firstpage
    518
  • Lastpage
    525
  • Abstract
    System scaling becomes essential and indispensable for distributed storage systems due to the explosive growth of data volume. As fault-protection is also a necessity in large-scale distributed storage systems, and Cauchy Reed-Solomon (CRS) codes are widely deployed to tolerate multiple simultaneous node failures, this paper studies the scaling of distributed storage systems with CRS codes. In particular, we formulate the scaling problem with an optimization model in which both the post-scaling encoding matrix and the data migration policy are assumed to be unknown in advance. To minimize the I/O overhead for CRS scaling, we first derive the optimal post-scaling encoding matrix under a given data migration policy, and then optimize the data migration process using the selected postscaling encoding matrix. Our scaling scheme requires the minimal data movement while achieving uniform data distribution. To validate the efficiency of our scheme, we implement it atop a networked file system. Extensive experiments show that our scaling scheme reduces 7.94% to 58.87%, and 39.52% on average, of the scaling time over the basic scheme.
  • Keywords
    Reed-Solomon codes; matrix algebra; optimisation; reliability; CRS code; I-O overhead minimization; cauchy Reed-Solomon code; data migration policy; fault-protection; large-scale distributed storage system; multiple simultaneous node failure; optimal post-scaling encoding matrix; optimization model; postscaling encoding matrix; scalability; uniform data distribution; Algorithm design and analysis; Encoding; Layout; Libraries; Optimization; Reed-Solomon codes; Strips; Cauchy Reed-Solomon; data migration; encoding matrix; performance evaluation; scaling;
  • fLanguage
    English
  • Publisher
    ieee
  • Conference_Titel
    Parallel and Distributed Systems (ICPADS), 2014 20th IEEE International Conference on
  • Type

    conf

  • DOI
    10.1109/PADSW.2014.7097849
  • Filename
    7097849