ReCREW: A Reliable Flash-Dissemination System

In this paper, we explore a new form of dissemination that arises in distributed, mission-critical applications called Flash Dissemination. This involves the rapid dissemination of rich information to a large number of recipients in a very short period of time. A key characteristic of Flash Dissemination is its unpredictability (e.g., natural hazards), but when invoked it must harness all possible resources to ensure timely delivery of information. Additionally, it must scale to a large number of recipients and perform efficiently in highly heterogeneous (data, network) and failure prone environments. We investigate a peer-based approach based on the simple principle of transferring dissemination load to information receivers using foundations from broadcast networks, gossip theory, and random networks. Gossip-based protocols are well known for being stateless, scalable, and fault-tolerant; however, their performance degrades as content size increases, because of the propagation of redundant gossip messages. In this paper, we propose Concurrent Random Expanding Walkers (CREW), a smart gossip protocol designed to maximize the speed of dissemination by transmitting data only as needed, and by exploiting both intra- and internode concurrency. CREW is designed to support both content and network heterogeneity and deal with transmission failures without sacrificing dissemination speed. We implemented CREW on top of a scalable middleware environment that allows for deployment across several platforms and developed optimizations without compromising on the stateless nature of CREW. We evaluated CREW empirically and compared it to optimized implementations of popular gossip and peer-based systems. Our experiments show that CREW significantly outperforms both traditional gossip and current large content dissemination systems while sustaining its performance in the presence of network errors.