Correlating Scheduling and Load balancing to achieve optimal performance from a cluster

In recent years, an increasing amount of computer network research has focused on the problem of cluster system in order to achieve higher performance and lower cost. Memory management becomes a prerequisite when handling applications that require immense volume of data for e.g. satellite images used for remote sensing, defense purposes and scientific applications. The load unbalance is the major defect that reduces performance of a cluster system that uses parallel program in a form of SPMD (single program multiple data). Dynamic load unbalancing can solve the load unbalance problem of cluster system and reduce its communication cost. This paper proposes a new algorithm that correlates the scheduling of incoming jobs and balancing of the loads at each node in a multi cluster. This method assigns weights for each node to schedule an incoming job and then load will be balanced dynamically using memory locality as the main factor. The main parameters used in this algorithm are partition size, CPU usage, memory usage, page faults and execution time. The tests evaluated with various applications shown a significant optimization in the cluster performance.