A Distributed Cache Framework for Metadata Service of Distributed File Systems

Most recent distributed file systems have adopted architecture with an independent metadata server cluster. However, potential multiple hotspots and flash crowds access patterns often cause a metadata service that violates performance Service Level Objectives. To maximize the throughput of the metadata service, an adaptive request load balancing framework is critical. We present a distributed cache framework above the distributed metadata management schemes to manage hotspots rather than managing all metadata to achieve request load balancing. This benefits the metadata hierarchical locality and the system scalability. Compared with data, metadata has its own distinct characteristics, such as small size and large quantity. The cost of useless metadata prefetching is much less than data prefetching. In light of this, we devise a time period-based prefetching strategy and a perfecting-based adaptive replacement cache algorithm to improve the performance of the distributed caching layer to adapt constantly changing workloads. Finally, we evaluate our approach with a hadoop distributed file system cluster.