Performance Driven Partial Checkpoint/Migrate for LAM-MPI

Using idle compute resources is cost-effective and systems like Condor have successfully exploited such resources in limited contexts (e.g. bag of tasks problems). Increasingly, networks in large organizations are becoming more capable and, when combined with latency tolerance mechanisms, can now provide an attractive platform for running some cluster-based parallel programs. In environments where machines are shared, however, load guarantees cannot be made. If one or more machines running an application become overloaded it may negatively impact the performance of the entire application. This provides a strong motivation to be able to checkpoint and migrate processes to new machines. Such performance driven migration normally involves the entire set of application processes. This, however, is wasteful both in terms of lost progress (if other processes can still execute) and overhead (since moving unnecessary processes is costly). To address these issues, we describe an extension of LAM/MPI that provides a partial checkpoint and migrate ability. Our system checkpoints only the subset of MPIprocesses that need to migrate. For long running applications exhibiting moderate communications, this can enhance the usefulness of shared machines for "cluster" computing.