Extracting delta for incremental data warehouse maintenance

This paper seeks to highlight an area important to commercial data warehouse deployments that has received limited research attention, namely, the extraction of changes to the data at the source systems. We refer to these changes as deltas. Extracting deltas from source systems is the first step in the incremental maintenance of data warehouses. A common assumption among current incremental maintenance methods is that deltas are somehow made available-normally in the form of differential files. Extraction of deltas from source systems is often not a straight forward process nor an efficient one. In this paper, we analyze how deltas can be extracted from large systems. We analyze delta extraction methods that are currently available, namely, time stamps, differential snapshots, triggers, and archive logs. We point out the strengths and weaknesses of each method through analysis and when appropriate through experimentation. We have been investigating the method called Op-Delta at Boeing that better suits delta extraction from large integrated systems. We discuss the benefits of Op-Delta, discuss how it could be implemented, and present comparative results from our experimentation