Improving Anti-spam Engine with Large Imbalanced Dataset Using Information Retrieval Technology

Anti-spam technology always employs machine learning to identify spam emails. Unfortunately, the email samples used to establish machine learning models are always not in a ideal status: there are too many spam emails compared with normal ones, which may lead to biased machine learning models and unsatisfactory performance in prediction. Besides, there are too many email samples, which lead to unaffordable resource consuming to run machine learning training process and thus difficult for human engineers to sort. In this paper, we proposed an information retrieval technology based approach to compress and balance the training data set. The key breakthrough here is to shrink and balance the training data set by removing similar data using information retrieval technology. Experiments show anti-spam classifier can have better performance with a much smaller and balanced training data set by applying this approach.