Title :
Fast Indexes and Algorithms for Set Similarity Selection Queries
Author :
Hadjieleftheriou, Marios ; Chandel, Amit ; Koudas, Nick ; Srivastava, Divesh
Author_Institution :
AT&T Labs.-Res., Florham Park, NJ
Abstract :
Data collections often have inconsistencies that arise due to a variety of reasons, and it is desirable to be able to identify and resolve them efficiently. Set similarity queries are commonly used in data cleaning for matching similar data. In this work we concentrate on set similarity selection queries: Given a query set, retrieve all sets in a collection with similarity greater than some threshold. Various set similarity measures have been proposed in the past for data cleaning purposes. In this work we concentrate on weighted similarity functions like TF/IDF, and introduce variants that are well suited for set similarity selections in a relational database context. These variants have special semantic properties that can be exploited to design very efficient index structures and algorithms for answering queries efficiently. We present modifications of existing technologies to work for set similarity selection queries. We also introduce three novel algorithms based on the Threshold Algorithm, that exploit the semantic properties of the new similarity measures to achieve the best performance in theory and practice.
Keywords :
data structures; database indexing; query processing; relational databases; data cleaning; index structure; query answering; relational database; set similarity query selection; similar data matching; threshold algorithm; Algorithm design and analysis; Cleaning; Computer science; Costs; Indexes; Information retrieval; Performance evaluation; Relational databases; Space technology; Weight measurement;
Conference_Titel :
Data Engineering, 2008. ICDE 2008. IEEE 24th International Conference on
Conference_Location :
Cancun
Print_ISBN :
978-1-4244-1836-7
Electronic_ISBN :
978-1-4244-1837-4
DOI :
10.1109/ICDE.2008.4497435