A pipe-based huge database object replication tool on heterogeneous database clusters

In rising ELT scenarios in data warehouse, for replicating many huge database objects(table/view) between heterogeneous database clusters, we propose a multi-way pipe-based replication architecture and put it into practice. To the best of our knowledge, there are few works on this aspects. Our contributions are from three aspects. The first is introducing a pipe-based architecture to avoid time spent in writing and reading data through an intermediate file, which is implemented on traditional file-based export or import tool. The second is automatically dividing a replication task into several sub-tasks, which will reduce time spent on task resuming when such a task encounters errors. At last, detailed experiments are made and the results show that such an architecture is effective in practical data warehouse projects.