DocumentCode :
694366
Title :
A novel approach to record file correlation and reduce mapping frequency on HDFS based on ExtendHDFS
Author :
Chang Xiao ; Qiang Li ; Dong Zheng
Author_Institution :
Sch. of Inf. Security, Shanghai Jiao Tong Univ., Shanghai, China
fYear :
2013
fDate :
12-13 Oct. 2013
Firstpage :
244
Lastpage :
248
Abstract :
Hadoop Distributed File System (HDFS) is quite commonly deployed in large data storage facilities and behaved very efficient when managing very large files. However, it has problems when operating large amount of small files. This is mainly because of the master-slave structure. Access request of too many small files will bring heavy burden to NameNode, which is the master machine of Hadoop. In the previous studies, Dong paid attention to file correlation and Chandrasekar S has proposed a general prefetching method. But neither of them gives a specific approach to record file correlation. Both of them made an assumption that files in one merged block has the higher correlation. In this paper, we proposed a new way to record file correlations based on Chandrasekar´s EHDFS. Through our recorded data, an optimal file request chain is achieved. The chain represents the most correlate file order. According to this order, blocks that contains small files can be re-constructed. After reconstruction, the new blocks will have higher prefetching efficiency according to our theoretical analysis and significantly reduce the request sent to Hadoop NameNode.
Keywords :
distributed databases; storage management; HDFS; Hadoop NameNode; Hadoop distributed file system; Hadoop master machine; data storage facilities; file correlation recording; general prefetching method; mapping frequency reduction; master-slave structure; optimal file request chain; Correlation; Educational institutions; File systems; Master-slave; Merging; Prefetching; Writing; cloud storage; hadoop; prefetch; sequencefile; small file; small file mapping;
fLanguage :
English
Publisher :
ieee
Conference_Titel :
Computer Science and Network Technology (ICCSNT), 2013 3rd International Conference on
Conference_Location :
Dalian
Type :
conf
DOI :
10.1109/ICCSNT.2013.6967105
Filename :
6967105
Link To Document :
بازگشت