DM_Integration: A Framework for Iterative Large Volume Data Integration

In this work, we consider the problem of large volume data integration in a relative compacted memory system. Previous work on data integration mainly focused on the difference elimination between data sources, and performed integration in memory in one pass. However, we see a new aspect of the problem: the data volume. We present a general framework called DM Integration for large volume data integration. The approach, instead of running one task to finish the whole integration, decomposes the integration task into several subtasks, each subtask works on a rela- tive small fraction of the data, and the subtasks run concur- rently. We show this method can be applied in novel ways for large volume data integration in a configurable mode. Keywords: Data Integration, Large Volume Data, Task Decomposition