Building variable selection ensembles for linear regression models by adding noise

Ensemble learning techniques have been proven to be useful to improve the performance of traditional variable selection methods such as the lasso, the genetic algorithm, and the stepwise search. Following the main principle of the building methods for variable selection ensembles, we propose a novel approach by adding noise to the response variable so that the selection accuracy is maximized. In order to generate multiple but slightly different importance measures for each variable, Gaussian noises are artificially added to the response. The new training set (i.e, the original design matrix together with the new response vector) is then used as the input of the genetic algorithm to perform the variable selection. By repeating this process for a number of iterations and fusing the results by simple averaging, the variables are ranked and a thresholding rule is then employed to identify the important variables. The performance of the proposed method is studied on simulated data sets. Experimental results demonstrate that the proposed method yields better performance than several other methods.