Merging Intra-Planetary Index Structures: Decentralized Bootstrapping of Overlays

Peer-to-peer index structures distributed and managed over the planet, commonly known as structured overlays (e.g., distributed hash tables), are posed to play the role of a fundamental building block for internet-scale distributed applications and information systems. One of the biggest impediment in realizing index distributed at intra-planetary scales is to be able to merge distinct indices that might have been constructed independently and separately, or have resulted from network partitions. By facilitating merger of such isolated index-structures, decentralized bootstrapping of structured overlays is made possible. We argue that such a self-organizational attribute of decentralized bootstrapping is of paramount importance for large scale systems. In this paper we provide algorithms and simulation based evaluation of merger of two tree-structured overlay networks. The experiments validate some intuitions, particularly we see how the merger operation can be carried out transparently from the end users in the sense that all keys that are accessible to any peer prior to the merger process continue to be accessible during and after the merger. Surprising at a first glance, we also observe that the overhead of data movement during merger of two overlays which originally partition the key-spaces arbitrarily is higher if there is more common content stored in the two networks. This counterintuitive behavior is on account of the fact that the assignment of partitions to peers change during the merger process, which in turn leads to movement of data. Since peers act based on local knowledge, the data already present in the other network is still unnecessarily moved, adding to the data transfer overhead.