The MicroGrid: using online simulation to predict application performance in diverse grid network environments

Improvements in networking and middleware technology are enabling large-scale grids that aggregate resources over wide-area networks to support applications at unprecedented levels of scale and performance. Unfortunately, existing middleware and tools provide little information to users as to the suitability of a given grid topology for a specific grid application. Instead, users generally use ad-hoc performance models to evaluate mappings of their applications to resource and network topologies. Grid application behavior alone is complex, and adding resource and network behavior makes the situation even worse. As a result, users typically employ nearly blind experimentation to find good deployments of their applications in each new grid environment. Only through actual deployment and execution can a user discovers if the mapping was a good one. Further, even after finding a good configuration, there is no basis to determine if a much better configuration has been missed. This approach slows effective grid application development and deployment. We present a richer methodology for evaluating grid software and diverse grid environments based on the MicroGrid grid online simulator. With the MicroGrid, users, grid researchers, or grid operators can define and simulate arbitrary collections of resources and networks. This allows study of an existing grid testbed under controlled conditions or even to study the efficacy of higher performance environments than are available today. Further, the MicroGrid supports direct execution of grid applications unchanged. These application can be written with MPI, C, C++, Perl, and/or Python and use the Globus middleware. This enables detailed and accurate study of application behavior. This work presents: (1) the first validation of the MicroGrid for studying whole-program performance of MPI grid applications and (2) a demonstration of the MicroGrid as a tool for predicting the performance of applications on a range of grid resources and novel network topologies.