Real-Time data ETL framework for big real-time data analysis

In the big data era, data become more important for BI and SCADA system operation. The load cycle of traditional data warehouse is fix and longer, which cannot timely response the rapid data change. Real-time data warehouse technology, as an extension of traditional data warehouse, can capture the rapid data change and process the real-time data analysis to meet the requirements of SCADA system. The real-time data access without the processing delay is a challenging task to the real-time data warehouse. In this paper, the real-time data ETL framework is presented to separately process the historical data and real-time data. Then, combining an external dynamic storage area, a dynamic mirror replication technology was proposed to avoid the contention between OLAP queries and OLTP updates. Finally, the experiments is set up based on the TPC-H benchmark to evaluate the performance of the proposed real-time data ETL framework. The experimental results demonstrates the proposed solution to real-time data ETL can effectively mitigate the query contention and data skew.