A multi-level comparative performance characterization of SPECjbb2005 versus SPECjbb2000

SPEC has released SPECjbb2005, a new server-side Java benchmark which supersedes SPECjbb2000. SPECjbb2005 is a substantial update to SPECjbb2000, intended to make the workload more representative based on current Java development practices. SPECjbb2000 has been in existence for about five years and it has been a valuable tool for optimizing the performance of commercial JVMs as well as supporting research activities. Since SPECjbb2005 replaces SPECjbb2000, it is important to understand the key differences between the two, as well as implications for JVM and hardware designers. In this paper, we present a comparative characterization of these two workloads based on detailed measurements on an Intel® Xeon™ processor-based commercial server. First, we describe key functional changes introduced in SPECjbb2005. Using low-intrusion application profiling tools we compare application execution profiles. Through JVM monitoring tools, we compare JVM behavior including JIT optimization and garbage collection. Using operating system monitoring tools we compare key system level metrics including CPU utilization. With the aid of processor performance monitoring events, we compare key architectural characteristics such as cache miss rates, memory/bus utilization, and branch behavior. Finally, we summarize key findings, provide recommendations to JVM developers and hardware designers, and suggest areas for future work.