Title :
A Distributed File System for Frequency Reading of Various File Sizes
Author :
Pengfei Ma ; Yanshen Yin ; Chao Lan ; Yong Zhang ; Chunxiao Xing
Author_Institution :
Dept. of Comput. Sci. & Technol., Tsinghua Univ., Beijing, China
Abstract :
The Web File Management System (WFMS) is a distributed file system, which is designed for frequency reading of various file sizes. It consists of one single master and multi data servers. We use blocks to store data, which contains different size of chunks. The block/chunk mechanism achieves nearly 100% utilization of block. We also design hot block to raise the reading speed of frequency blocks through increasing the replicas automatically. WFMS works on the web exposing Restful services and providing authority control. Besides, we implement our search engine with Lucene which provides query service by file name, file size, file type, date, owner and so on. Until now, our system has stored over 40TB data which contains more than 2 million files and it has worked stably for two years.
Keywords :
Internet; authorisation; query processing; replicated databases; search engines; storage management; Lucene; Restful services; WFMS; Web file management system; authority control; block/chunk mechanism; data storage; distributed file system; file sizes; frequency blocks; frequency reading; multidata server; query service; reading speed; search engine; single master server; Access control; Distributed databases; File systems; Indexes; Reliability; Schedules; Servers; WFMS; distributed file system; file system;
Conference_Titel :
Web Information System and Application Conference (WISA), 2013 10th
Conference_Location :
Yangzhou
Print_ISBN :
978-1-4799-3218-4
DOI :
10.1109/WISA.2013.70
