An Adaptive Synchronization Policy for Harvesting OAI-PMH Repositories

Metadata harvesting requires timely propagation of up-to-date information from thousands of repositories over a wide area network. It is desirable to keep the data as fresh as possible while observing the overhead on the harvester. An important dimension to be considered is that repositories vary widely in their update patterns; they may experience different update rates at different times or unexpected changes to update patterns. In this paper, we define data freshness metrics and propose an adaptive algorithm for the synchronization of the harvester with the repositories. The algorithm is based on meeting a desired level of freshness while incurring the minimum overhead on the harvester. We present a comparison between different policies for the synchronization within the framework devised. It is shown that the proposed policy outperform the other policies, especially for heterogeneous update patterns.