Adaptive duplicate detection using learnable string similarity measures

M Bilenko, RJ Mooney - Proceedings of the ninth ACM SIGKDD …, 2003 - dl.acm.org
Proceedings of the ninth ACM SIGKDD international conference on Knowledge …, 2003dl.acm.org
The problem of identifying approximately duplicate records in databases is an essential step
for data cleaning and data integration processes. Most existing approaches have relied on
generic or manually tuned distance metrics for estimating the similarity of potential
duplicates. In this paper, we present a framework for improving duplicate detection using
trainable measures of textual similarity. We propose to employ learnable text distance
functions for each database field, and show that such measures are capable of adapting to …
The problem of identifying approximately duplicate records in databases is an essential step for data cleaning and data integration processes. Most existing approaches have relied on generic or manually tuned distance metrics for estimating the similarity of potential duplicates. In this paper, we present a framework for improving duplicate detection using trainable measures of textual similarity. We propose to employ learnable text distance functions for each database field, and show that such measures are capable of adapting to the specific notion of similarity that is appropriate for the field's domain. We present two learnable text similarity measures suitable for this task: an extended variant of learnable string edit distance, and a novel vector-space based measure that employs a Support Vector Machine (SVM) for training. Experimental results on a range of datasets show that our framework can improve duplicate detection accuracy over traditional techniques.
ACM Digital Library