The need for measured data in computer system performance analysis or garbage in, garbage out

There are three general techniques for studying the design, operation and optimization of any type of system: implementation, modeling, and simulation. Implementation can be used when the time is available and the effort is justified. Mathematical modeling can be used if the system is simple enough or if gross approximations are acceptable. Simulation is needed for the remaining cases. In all three cases, a crucial element is the need for accurate estimates of the “workload.” For example, for a computer main or cache memory system, the workload may at one level of detail be the timing and sequence of memory addresses referenced; at another level of detail, it could be just the ratio of loads to stores. No matter how accurate the model of a system may be, any results from it are useless if the workload is not accurately represented. We discuss various types of models and when and what type of accurate workload data is needed. We pay particular attention to performance analysis of memory hierarchies and the use of trace driven simulation