Toward integrating overlay and physical networks for robust parallel processing architecture

Recently, much research attention is being paid toward the wide-area parallel processing architectures whereby all the network devices in the entire network execute the processing function in order to shorten the completion time of big data mining tasks. Despite some advancement, such architectures still suffer from physical network failures, which lead to critical service unavailability. As a remedy to this problem, an overlay-based parallel processing architecture, where nodes manage each other by employing the overlay network, can achieve higher service availability against small-scale failures. However, the overlay-based parallel processing architecture is not capable of providing data mining services in case large-scale network failures disrupt the overlay network. To deal with this issue, our article introduces a design methodology of an overlay-based parallel processing architecture based on integration of overlay and physical networks. Additionally, we introduce basic principles based on the design methodology. Through numerical calculation, we evaluate the effectiveness of an integration approach on the performance of parallel data processing in terms of higher service availability against physical network failures while minimizing traffic load.