Knowledge reduction method based on information entropy for port big data using MapReduce

With the volume of port data growing at an unprecedented rate, analyzing and extracting knowledge from large-scale data sets have become a new challenge in decision making. But, the application of standard data mining tools in such data sets is not straightforward. Hence, we develop a parallel large-scale knowledge reduction method based on rough set for knowledge acquisition using MapReduce in this paper. It designs and implements the Map and Reduce functions using data and task parallelism. Then, it constructs the parallel algorithm framework model for knowledge reduction using MapReduce, which can be used to compute a reduct for the algorithms based on information entropy. The experimental results demonstrate that the proposed parallel knowledge reduction method can efficiently process massive datasets on Hadoop platform, with highly speed up the classification process and largely reduce the storage requirements.