Statistical and Heuristic Model Selection in Regularized Least-Squares

The Regularized Least-Squares (RLS) method uses the kernel trick to perform non-linear regression estimation. Its performance depends on the proper selection of a regularization parameter. This model selection task has been traditionally carried out using cross-validation. However, when training data is scarce or noisy, cross-validation may lead to poor model selection performance. In this paper we investigate alternative statistical and heuristic procedures for model selection in RLS that were shown to perform well for other regression methods. Experiments conducted on real datasets show that these alternative model selection procedures are not able to improve performance when cross-validation fails.