Feature Selection using Partial Least Squares regression and optimal experiment design

We propose a supervised feature selection technique called the Optimal Loadings, that is based on applying the theory of Optimal Experiment Design (OED) to Partial Least Squares (PLS) regression. We apply the OED criterions to PLS with the goal of selecting an optimal feature subset that minimizes the variance of the regression model and hence minimize its prediction error. We show that the variance of the PLS model can be minimized by employing the OED criterions on the loadings covariance matrix obtained from PLS. We also provide an intuitive viewpoint to the technique by deriving the Aoptimality version of the Optimal Loadings criterion using the properties of maximum relevance and minimum redundancy for PLS models. In our experiments we use the D-optimality version of the criterion which maximizes the determinant of the loadings covariance matrix. To overcome the computational challenges in this criterion, we provide an approximate D-optimality criterion along with the theoretical justification.