MANET protocol simulations considered harmful: the case for benchmarking

In this article, we investigate the current best practices in simulation-based multi-hop wireless ad-hoc network (MANET) protocol evaluation. We extend a prior characterization of the settings and parameters used in MANET simulations by studying the papers published in one of the premier mobile networking conferences between 2006 and 2010. We find that there are still several configuration pitfalls which many papers fall victim to, which in turn damages the integrity of the results as well as any research aimed at reproducing and extending these results. We then describe the simulation "design space" of MANET routing in terms of its basic dimensions and corresponding parameters. We also propose four "auxiliary" metrics to increase simulation integrity. We conclude with several example scenarios that promote modeling simulations after real-world situations.