Relationship between Naïve Bayes error and max-dependency criterion in feature selection problems

Feature selection of the raw data is a fundamental step in the most pattern recognition and machine learning applications. The primary problem of feature selection is the criterion which evaluates a feature set. In the context of classification problems, optimal criterion would be the Bayesian error rate for selected subset of features. The Bayesian error rate bounds to some values that are related to mutual information. This interval shrinks as the mutual information increases. In this paper, we investigated the relationship between dependency and the Naïve Bayes error; dependency of the selected features is calculated as mutual information between the selected features and class. We designed some experiments to examine it about a two classes and two binary features problem. We found that in binary feature selection problem, the Naïve Bayes error increases as dependency increases; however, we showed that there are some states that the Naïve Bayes classifier is optimal while its default assumption is strongly violated (dependency is more than 0.8).