Utilizing Multiple Networks for Interprocess Communication in Cluster Computing

A wide range of applications are hungry for high performance computing (HPC). Commodity workstation/PC clusters represents a cost-effective platform for scalable parallel computation. The message passing mechanism is usually employed for interprocess communication in a cluster for data sharing and process synchronization. In high performance computing, the most widely-used standard for interprocess communication is the message passing interface or MPI. This paper proposes a new approach to building a Java-based interprocess communication (IPC) library for cluster-based parallel applications by using multiple networks. These different types of networks have different performance characteristics, while the different types of messages may have different communication requirements, such as short synchronization messages that require low latency, and large data transfers that require high-bandwidth. Thus, no single communication network can provide the best performance for all types of messages. This library selects the lowest latency for each individual message by dynamically selecting among multiple networks. The performance of this system depends on the mix of message sizes in the parallel application program and the relative overheads of the network