Distributed Approach to Feature Selection From Very Large Data Sets Using BLEM2

Feature selection is an important step in the preprocessing of raw data for data mining. It involves eliminating redundant and irrelevant features from the dataset to get a subset of features, which performs as efficient as the complete set. The wrapper approach to the problem of feature selection is to use an induction algorithm to select the features. Most induction algorithms fail to handle large datasets. The obvious method that can be employed to handle large datasets is "divide and conquer". This paper introduces a strategy for finding features from a collection of distributed subsets using the BLEM2 rule-based inductive learning program. Heurstics for determining proper number of subsets and proper subset size are proposed. The proposed strategy has been tested on the intrusion detection systems dataset made available by MIT Lincoln labs