A primitive operator for similarity joins in data cleaning

S Chaudhuri, V Ganti, R Kaushik - … International Conference on …, 2006 - ieeexplore.ieee.org
S Chaudhuri, V Ganti, R Kaushik
22nd International Conference on Data Engineering (ICDE'06), 2006ieeexplore.ieee.org
Data cleaning based on similarities involves identification of" close" tuples, where closeness
is evaluated using a variety of similarity functions chosen to suit the domain and application.
Current approaches for efficiently implementing such similarity joins are tightly tied to the
chosen similarity function. In this paper, we propose a new primitive operator which can be
used as a foundation to implement similarity joins according to a variety of popular string
similarity functions, and notions of similarity which go beyond textual similarity. We then …
Data cleaning based on similarities involves identification of "close" tuples, where closeness is evaluated using a variety of similarity functions chosen to suit the domain and application. Current approaches for efficiently implementing such similarity joins are tightly tied to the chosen similarity function. In this paper, we propose a new primitive operator which can be used as a foundation to implement similarity joins according to a variety of popular string similarity functions, and notions of similarity which go beyond textual similarity. We then propose efficient implementations for this operator. In an experimental evaluation using real datasets, we show that the implementation of similarity joins using our operator is comparable to, and often substantially better than, previous customized implementations for particular similarity functions.
ieeexplore.ieee.org