Weighted Set Similarity: Queries and Updates

Summary form only given. Consider a universe of items, each of which is associated with a weight, and a database consisting of subsets of these items. Given a query set, a weighted set similarity query identifies either (i) all sets in the database whose normalized similarity to the query set is above a pre-specified threshold, or (ii) the sets in the database with the k highest similarity values to the query set. Weighted set similarity queries are useful in applications like data cleaning and integration for finding approximate matches in the presence of typographical mistakes, multiple formatting conventions, transformation errors, etc. We show that this problem has semantic properties that can be exploited to design index structures that support efficient algorithms for answering queries; these algorithms can achieve arbitrarily stronger pruning than the family of Threshold Algorithms. We describe how these index structures can be efficiently updated using lazy propagation in a way that gives strict guarantees on the quality of subsequent query answers. Finally, we illustrate that our proposed ideas work well in practice for real datasets.