Live Data Replication Approach from Relational Tables to Schema-Free Collections Using Stream Processing Framework

Recent researches focus on the data replication issue from relational tables to schema-free collections in a batch processing way. However, there are few publications on live data replication in real time. In this paper, we attempt to address this legacy issue with new stream processing framework. The process of replication consists of log-based change data capture and stream-based data replication. Data replication mappings are present, and the proposed architecture of stream processing framework including column grouping, column merging and column versioning, is introduced to avoid data lost in case of failure. Finally, our experimental evaluation of live data replication approach with stream processing framework shows the higher effectiveness and efficiency than current methods.