Title :
Partitioner: A Distributed HDFS Metadata Server Cluster
Author :
Ruini Xue ; Lixiang Ao ; Shengli Gao ; Zhongyang Guan ; Lupeng Lian
Author_Institution :
Sch. of Comput. Sci. & Eng., Univ. of Electron. Sci. & Technol. of China, Chengdu, China
Abstract :
The Hadoop Distributed File System (HDFS) provides scalable, high throughput and fault-tolerant file service for Hadoop distributed frameworks. However, its single Name Node architecture can lead the metadata server to be a performance bottleneck and a single point of failure (SPOF). In this study, we designed and implemented Partitioner, a distributed HDFS metadata server cluster, which can unite multiple Name Nodes and provide single unique namespace for clients. To support the distributed metadata management, a novel sub tree based metadata partitioning and lookup scheme is proposed. Experiment results indicated that this method can eliminate SPOF while expanding metadata capacity, resulting in improved scalability and availability of the distributed file system.
Keywords :
data handling; distributed databases; fault tolerance; meta data; parallel processing; Hadoop distributed file system; SPOF; distributed HDFS meta data server cluster; distributed meta data management; fault-tolerant file service; single name node architecture; single point of failure; Availability; Computer architecture; Distributed databases; Load management; Monitoring; Scalability; Servers; HDFS; distributed file system; metadata management; scalability;
Conference_Titel :
Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), 2014 International Conference on
Conference_Location :
Shanghai
Print_ISBN :
978-1-4799-6235-8
DOI :
10.1109/CyberC.2014.38
