Online system for detection of Chinese near-duplicate documents

In various types of information retrieval systems, searching engines, and some data-mining systems, there is one task cannot be avoided-how to detect the large-scale duplicate and near-duplicate documents rapidly. Too many duplicates will influence our systems in many aspects malignantly. For example it reduces the computational performance, cuts down the user experience and so on. On the other hand, if quantity of documents increases dynamically, we should take another way to tackle this problem. This paper aims to construct a practical online detection system under the guidance of the fingerprint extraction technique based on simhash. Our contribution is that we develop a system running online, which means we don´t know the accurate quantity of the documents before, and the system is able to accept new documents anytime. It requires efficiency and flexibility, and we propose a favorable solution.