Potential solutions for benchmarking issues

The lack of standardized metrics for assessing the capabilities of many automation tools requires that evaluation procedures be executed by a user to assess functional capabilities and performance of the product. While benchmarks are typically conducted when choosing a supplier, evaluation/ regression tests are also often performed on new releases from an existing supplier. The variables that influence a benchmark process can include test case design, user expertise, hardware platforms, data interpretation, etc. A benchmark test case that represents the user´s application is critical. The numerous variables and opportunities for misinterpretation can make a standardized test bench suite difficult to achieve. Cooperative testing by broadbased and niche suppliers of bundled products is one area that can reduce user evaluation cycles. Strong customer supplier working relationships can also be one key to reducing the amount of the effort required for benchmarks