Reducing ETL Load Times by a New Data Integration Approach for Real-time Business Intelligence

Abstract—Reducing business latency is essential in today’s competitive and demanding environments. This means responding immediately to new information as it arrives, and having the right information in time to make the best decision. Integrating, processing and delivering results in real-time is a huge challenge, particularly as data volumes continue to increase dramatically and sources of data are ever more distributed and varied. The decision making process in traditional data warehouse environments is often delayed because data cannot be propagated from the source system to the data warehouse in time. The typical update patterns for traditional data warehouses on an overnight or even weekly basis increase this propagation delay. Keeping data current by minimizing the latency from when data is captured until it is available to decision makers in this context is a difficult task. A real-time data warehouse aims at decreasing the time it takes to make business decisions and tries to attain zero latency between the cause and effect of a business decision. An ETL process that periodically copies a snapshot of the entire source consumes too much time and resources. Alternate approaches that include timestamp columns, triggers, or complex queries often hurt performance and increase complexity. What is needed is a reliable stream of change data that is structured so that it can easily be applied by consumers to target representations of the data. To keep up with the market competition, there is increased need to minimize ETL load times. In this paper I have introduced an approach for data integration that deals with reducing ETL load times.