Title :
A Real-time Receiving and Distributed Processing System for Large-scale Burst Data
Author :
Wu, Guangsheng ; Ao, Zhenlang ; Li, Jianyong ; Zhou, Qinqiang
Author_Institution :
Comput. Applic. & Dev. Inst., Guangdong Meteorol., Guangzhou, China
Abstract :
According to the data characteristics and requirements of receiving and processing in large automatic weather monitoring network, the design and key technology on receiving and processing system of large burst data are given, and an in-depth discussion is made. Making use of multi-thread technology and large-scale communication event response technology, the data receiving thread pool is constructed and realized. Based on the theory of "data partition management", a concurrent processing data buffer is planned and excogitated. Through performance testing and application of the system, the results show that the system is real time, efficient and reliable and meets the current needs of the weather operation and is adaptable to the development of future weather operation.
Keywords :
data analysis; distributed processing; multi-threading; communication event response technology; concurrent processing data buffer; data characteristics; data partition management; data receiving thread pool; distributed processing system; large automatic weather monitoring network; large-scale burst data; multithread technology; real-time receiving system; Instruction sets; Memory management; Message systems; Meteorology; Programming; Real time systems; Servers; data buffer; large-scale burst data; receiving and processing; thread pool;
Conference_Titel :
Networking and Distributed Computing (ICNDC), 2011 Second International Conference on
Conference_Location :
Beijing
Print_ISBN :
978-1-4577-0407-9
DOI :
10.1109/ICNDC.2011.30
