HBA: Distributed Metadata Management for Large Cluster-Based Storage Systems

Abstract — An efficient and distributed scheme for file mapping or file lookup is critical in decentralizing metadata management within a group of metadata servers, here the technique used called HIERARCHICAL BLOOM FILTER ARRAYS (HBA) to map filenames to the metadata servers holding their metadata. The Bloom filter arrays with different levels of accuracies are used on each metadata server. The first one with low accuracy and used to capture the destination metadata server information of frequently accessed files. The other array is used to maintain the destination metadata information of all files. Simulation results show our HBA design to be highly effective and efficient in improving the performance and scalability of file systems in clusters with 1,000 to 10,000 nodes (or superclusters) and with the amount of data in the petabyte scale or higher. HBA is reducing metadata operation by using the single metadata architecture instead of 16 metadata server.