Benchmarking

The importance of determining the role and influence of the test tool when integrated into the "existing" design and test methodology of the user, and then formulating the benchmark process, is emphasised. This can affect shorter benchmark cycles. The focus of an evaluation process should be more of which tool offers the most productive way of using existing techniques to yield successful results" versus "which tool claims to offer the latest technology available". An equally critical component of the benchmarking process is the proper selection of benchmark circuits. This issue is also applicable to research projects where there is more opportunity to use outdated and unrealistic circuits. The problems of testability can never be solved by throwing circuits at the test tools being evaluated. Testability can be achieved only through trade-offs. Furthermore, during the entire evaluation period, continual interaction with the vendor is critical to insure that benchmark experiments are performed with the right conditions and constraints on the tool. Overly optimistic benchmark parameters such as high-level models can easily generate impressive, but unrealistic results