Metamodel accuracy assessment in evolutionary optimization

Evolutionary optimization of expensive functions typically uses a metamodel, i.e. a computationally cheaper but inaccurate approximation of the objective function. The success of the optimization search depends on the accuracy of the metamodel hence an integral part of the metamodelling framework is assessing the metamodel accuracy. In this paper we survey a range of accuracy assessment methods such as methods requiring additional sites, hypothesis testing and minimum loss-function methods. We describe two numerical experiments: the first benchmarks different accuracy assessment methods from which it follows the most accurate methods are LOOCV and the 0.632 bootstrap estimator followed by the 10-CV and lastly the holdout method. The second experiment studies the effect of two different accuracy assessment methods on the performance of a typical metamodel-assisted EA, from which it follows the accuracy assessment method has significant effect on the obtained optimum and hence should be chosen corresponding to the objective function features and dimension.We also discuss several issues related to the performance of accuracy assessment methods in practice.