Overcoming Web Server Benchmarking Challenges in the Multi-core Era

Web-based services are used by many organizations to support their customers and employees. An important consideration in developing such services is ensuring the Quality of Service (QoS) that users experience is acceptable. Recent years have seen a shift toward deploying Web services on multi-core hardware. Leveraging the performance benefits of multi-core hardware is a non-trivial task. In particular, systematic Web server benchmarking techniques are needed so organizations can verify their ability to meet customer QoS objectives while effectively utilizing such hardware. However, our recent experiences suggest that the multi-core era imposes significant challenges to Web server benchmarking. In particular, due to limitations of current hardware monitoring tools, we found that a large number of experiments are needed to detect complex bottlenecks that can arise in a multi-core system due to contention for shared resources such as cache hierarchy, memory controllers and processor inter-connects. Furthermore, multiple load generator instances are needed to adequately stress multi-core hardware. This leads to practical challenges in validating and managing the test results. This paper describes the automation strategies we employed to overcome these challenges. We make our test harness available for other researchers and practitioners working on similar studies.