Research of distributed ETL engine based on MAS and data partition

Author

Wang, Guiteng ; Guo, Chaozhen

Author_Institution

Coll. of Math. & Comput. Sci., Fuzhou Univ., Fu Zhou, China

fYear

2011

fDate

8-10 June 2011

Firstpage

342

Lastpage

347

Abstract

In order to improve the efficiency of ETL workflow executing, this paper presents a distributed ETL engine based on MAS and data partition technology and also researches the methods of partitioning the massive data stream in both horizontal and vertical ways. The engine referred to will partition an ETL workflow which meets the conditions of being partitioned into multiple sub workflows for parallel executing. Each of the sub workflow is executed by an agent, so that multiple agents could work together to complete the collaborative work. Experimental results show that this system has good scalability and could well improve the efficiency of ETL workflow executing.

Keywords

enterprise resource planning; multi-agent systems; parallel processing; workflow management software; MAS; data partition; distributed ETL engine; multiple sub workflows; parallel execution; Computer architecture; Corporate acquisitions; Distributed databases; Engines; Registers; Servers; Sockets; CSCW; ETL workflow; MAS; data partition; distribute ETL;

fLanguage

English

Publisher

ieee

Conference_Titel

Computer Supported Cooperative Work in Design (CSCWD), 2011 15th International Conference on

Conference_Location

Lausanne

Print_ISBN

978-1-4577-0386-7

Type

conf

DOI

10.1109/CSCWD.2011.5960096

Filename

5960096