An integrated approach for detecting approximate duplicate records

Detecting approximate duplicate records in database is a key problem related to data quality. Given two lists of records, the duplicate detection problem consists of determining all pairs that are similar to each other, where the overall similarity between two records is defined based on domain-specific similarities over individual attributes constituting the record. In this paper, we present a synthetic approach for recognizing clusters of approximate duplicate records of multi-language data. The key ideas are: (1) an efficient algorithm for pre-processing multi-language data consists of Chinese words segmentation and Chinese named entity recognition; (2) an efficient pair-wise comparison method based on domain- specific similarities, especially, the string kernel method; (3) using a priority queue of duplicate clusters and representative records strategy to respond adaptively to the data scale.