A case study in meta-simulation design and performance analysis for large-scale networks

Simulation and emulation techniques are fundamental to aid the process of large-scale protocol design and network operations. However, the results from these techniques are often view with a great deal of skepticism from the networking community. Criticisms come in two flavors: (i) the study presents isolated and potentially random feature interactions, and (ii) the parameters used in the study may not be representative of real-world conditions. In this paper, we explore both issues by applying large-scale experiment design and black-box optimization techniques to analyze convergence of network routes in the open shortest path first protocol over a realistic network topology. By using these techniques, we show that: (i) the needed number of simulation experiments can be reduced by an order of magnitude compared to traditional full-factorial experiment design (FFED) approach, (ii) unnecessary parameters can easily be eliminated, and (iii) rapid understanding of key parameter interactions can be achieved.