A Real-time Receiving and Distributed Processing System for Large-scale Burst Data

Author

Wu, Guangsheng ; Ao, Zhenlang ; Li, Jianyong ; Zhou, Qinqiang

Author_Institution

Comput. Applic. & Dev. Inst., Guangdong Meteorol., Guangzhou, China

fYear

2011

fDate

21-24 Sept. 2011

Firstpage

111

Lastpage

115

Abstract

According to the data characteristics and requirements of receiving and processing in large automatic weather monitoring network, the design and key technology on receiving and processing system of large burst data are given, and an in-depth discussion is made. Making use of multi-thread technology and large-scale communication event response technology, the data receiving thread pool is constructed and realized. Based on the theory of "data partition management", a concurrent processing data buffer is planned and excogitated. Through performance testing and application of the system, the results show that the system is real time, efficient and reliable and meets the current needs of the weather operation and is adaptable to the development of future weather operation.

Keywords

data analysis; distributed processing; multi-threading; communication event response technology; concurrent processing data buffer; data characteristics; data partition management; data receiving thread pool; distributed processing system; large automatic weather monitoring network; large-scale burst data; multithread technology; real-time receiving system; Instruction sets; Memory management; Message systems; Meteorology; Programming; Real time systems; Servers; data buffer; large-scale burst data; receiving and processing; thread pool;

fLanguage

English

Publisher

ieee

Conference_Titel

Networking and Distributed Computing (ICNDC), 2011 Second International Conference on

Conference_Location

Beijing

Print_ISBN

978-1-4577-0407-9

Type

conf

DOI

10.1109/ICNDC.2011.30

Filename

6047117