Incremental data feed maintenance of a data warehouse system derived from multiple autonomous data sources

The data acquisition process, in which the data warehouse and operational data store (ODS) are populated from operational sources, represents the most technically challenging part of any business intelligence (BI) environment. Some industry experts estimate that 60 to 80 percent of a Bl project´s effort is spent on this process alone. Nevertheless, most of the previous development work is to trigger visual GUIs tools such as Informtica manually, enter properties and drive ETL process. However, the purpose of this article is to introduce a better on-demand means to pull data from modern heterogeneous data sources with the integration of Informatica, Oracle and Korn shell script. We introduced a practical production instance on how to accomplish an efficient, scalable, controllable and maintainable ETL (extract, transform, loading) architecture. Within this infrastructure, we adopt two new techniques: process synchronization control (PSC) and time range control (TRC).