Should the Same Learners Be Used Both within Wrapper Feature Selection and for Building Classification Models?

Due to the problem of high-dimensionality(datasets which contain many independent attributes or features), feature selection has become an important part of data mining research. One popular form of feature selection, wrapper selection, chooses the best features by directly addressing the question of which features build the best models. Various feature subsets are used to build classification models, and the performance of these models is the score of each feature subset. The feature subset with the best score is then used to build the final classification model. As wrappers use a classification algorithm (learner) both to select the features and to build a predictive model, it has been traditional to use the same learner for both, such that the features chosen will be those which optimize that model´s performance. However, no research has considered whether having different learners operate inside and outside the wrapper (that is, for selectingthe features and for building the final model) might actually result in improved classification performance. In this work, weconsider five learners both inside the wrapper and for building the classification model, along with two datasets drawn fromthe domain of Twitter profile mining. By considering both the raw performance values and a statistical analysis, we find that contrary to intuition, usually the best performance for a given choice of external learner is not found by using the same learner within the wrapper. Instead, the Naïve Bayes learner is usually the best choice for selecting features, regardless of which learner is used for the external model. We also find that Multi-Layer Perceptron is able to build consistent classification models for many different choices of internal learner. Finally, the 5-Nearest Neighbor learner gave poor results both inside and outside the wrapper.