Article

Free access

The merge/purge problem for large databases

Authors:

Mauricio A. Hernández,

Salvatore J. StolfoAuthors Info & Claims

SIGMOD '95: Proceedings of the 1995 ACM SIGMOD international conference on Management of data

Pages 127 - 138

https://doi.org/10.1145/223784.223807

Published: 22 May 1995 Publication History

PDF eReader

Abstract

Many commercial organizations routinely gather large numbers of databases for various marketing and business analysis functions. The task is to correlate information from different databases by identifying distinct individuals that appear in a number of different databases typically in an inconsistent and often incorrect fashion. The problem we study here is the task of merging data from multiple sources in as efficient manner as possible, while maximizing the accuracy of the result. We call this the merge/purge problem. In this paper we detail the sorted neighborhood method that is used by some to solve merge/purge and present experimental results that demonstrates this approach may work well in practice but at great expense. An alternative method based upon clustering is also presented with a comparative evaluation to the sorted neighborhood method. We show a means of improving the accuracy of the results based upon a multi-pass approach that succeeds by computing the Transitive Closure over the results of independent runs considering alternative primary key attributes in each pass.

References

[1]

R. Agrawal and H. V. Jagadish. Multiprocessor Transitive Closure Algorithms. In Proc. Int 'l Syrup. on Databases in Parallel and Distributed Systems, pages 56-66, December 1988.

Digital Library

Google Scholar

[2]

M. A. Bickel. Automatic Correction to Misspelled Names: a Fourth-generation Language Approach. Communications of the ACM, 30(3):224-228, 1987.

Digital Library

Google Scholar

[3]

D. Bitton and D. J. DeWitt. Duplicate Record Elimination in Large Data Files. A CM Transactions on Database Systems, 8(2):255-265, June 1983.

Digital Library

Google Scholar

[4]

H. M. Dewan, M. A. Hern~ndez, K. Mok, and S. Stolfo. Predictive Load Balancing of Parallel Hash-Joins over Heterogeneous Processors in the Presence of Data Skew. In Proc. 3rd Int'l Con}. on Parallel and Distributed Information Systems, pages 40-49, September 1994.

Digital Library

Google Scholar

[5]

D. J. DeWitt, J. F. N aughton, and D. A. Schneider. An Evaluation of N on-Equijoin Algorithms. In Proc. 17th Int'l. Conf. on Very Large Databases, pages 443-452, Barcelona, Spain, 1991.

Digital Library

Google Scholar

[6]

C. L. Forgy. OPS5 User's Manual. Technical Report CMU-CS-81-135, Carnegie Mellon University, July 1981.

Crossref

Google Scholar

[7]

S. Ghandeharizadeh. Physical Database Design ~n Mul. tiprocessor Database Systems. PhD thesis, Department of Computer Science, University of Wisconsin - Madison, 1990.

Digital Library

Google Scholar

[8]

R. Graham. Bounds on multiprocessing timing anomalies. SIAM Journal o} Computzng, 17:416-429, 1969.

Google Scholar

[9]

M. A. gernKndez. A Generalization of Band-Joins and the Merge/Purge Problem. Technical Report CUCS- 005-1995, Department of Computer Science, Columbia University, February 1995.

Google Scholar

[10]

W. Kent. The Breakdown of the Information Model in Multi-Database Systems. SIGMOD Record, 20(4):10- 15, December 1991.

Digital Library

Google Scholar

[11]

K. Kukich. Techniques for Automatically Correcting Words in Text. A CM Computing Surveys, 24(4):377- 439, 1992.

Digital Library

Google Scholar

[12]

D. P. Miranker, B. Lofaso, G. Farmer, A. Chandra, and D. Brant. On a TREAT-based Production System Compiler. In Proc. l Oth Int'l Conf. on Expert Systems, pages 617-630, 1990.

Google Scholar

[13]

C. Nyberg, T. B~rclay, Z. Cvetanovic, J. Gray, and D. Lomet. AlphaSort: A RISC Machine Sort. In Proceedings of the 1994 A CM-SIGMOD Conference, pages 233-242, 1994.

Digital Library

Google Scholar

[14]

Y. R. Wang and S. E. Madnick. The Inter-Database Instance Identification Problem in Integrating Autonomous Systems. In Proceedings of the Sixth International Conference on Data Engineering, February 1989.

Digital Library

Google Scholar

Cited By

View all

Mugeni JAmagasa T(2023)A Graph-Based Blocking Approach for Entity Matching Using Contrastively Learned EmbeddingsACM SIGAPP Applied Computing Review10.1145/3584014.358401722:4(37-46)Online publication date: 10-Feb-2023
https://dl.acm.org/doi/10.1145/3584014.3584017
Foua BWang XTalburt JXu X(2023)Train Once, Match Everywhere: Harnessing Generative Language Models for Entity Matching2023 International Conference on Computational Science and Computational Intelligence (CSCI)10.1109/CSCI62032.2023.00012(30-36)Online publication date: 13-Dec-2023
https://doi.org/10.1109/CSCI62032.2023.00012
Deo NBasak JSoliman AWeinberg DSteorts RRajasekaran S(2023)Novel Blocking Techniques and Distance Metrics for Record LinkageInformation Integration and Web Intelligence10.1007/978-3-031-48316-5_40(431-446)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1007/978-3-031-48316-5_40
Show More Cited By

Index Terms

The merge/purge problem for large databases
1. Information systems
  1. Data management systems
    1. Database management system engines
2. Theory of computation
  1. Design and analysis of algorithms
    1. Data structures design and analysis
      1. Sorting and searching
  2. Randomness, geometry and discrete structures

Recommendations

The merge/purge problem for large databases

Many commercial organizations routinely gather large numbers of databases for various marketing and business analysis functions. The task is to correlate information from different databases by identifying distinct individuals that appear in a number of ...
A generalization of band joins and the merge/purge problem
Entity Resolution for Large-Scale Databases

Comments

Information & Contributors

Information

Published In

SIGMOD '95: Proceedings of the 1995 ACM SIGMOD international conference on Management of data

June 1995

508 pages

ISBN:0897917316

DOI:10.1145/223784

Editors:
Michael Carey
Univ. of Wisconsin
,
Donovan Schneider
Red Brick Systems

ACM SIGMOD Record Volume 24, Issue 2
May 1995
490 pages
ISSN:0163-5808
DOI:10.1145/568271
Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 May 1995

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SIGMOD/PODS95

Sponsor:

SIGMOD/PODS95: Joint ACM SIGMOD Intl Conference on Management of Data and ACM SIGACT-SIGMOD- SIGART Symposium on Principles of Database Systems

May 22 - 25, 1995

California, San Jose, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

609
Total Citations
View Citations
3,281
Total Downloads

Downloads (Last 12 months)288
Downloads (Last 6 weeks)22

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Mugeni JAmagasa T(2023)A Graph-Based Blocking Approach for Entity Matching Using Contrastively Learned EmbeddingsACM SIGAPP Applied Computing Review10.1145/3584014.358401722:4(37-46)Online publication date: 10-Feb-2023
https://dl.acm.org/doi/10.1145/3584014.3584017
Foua BWang XTalburt JXu X(2023)Train Once, Match Everywhere: Harnessing Generative Language Models for Entity Matching2023 International Conference on Computational Science and Computational Intelligence (CSCI)10.1109/CSCI62032.2023.00012(30-36)Online publication date: 13-Dec-2023
https://doi.org/10.1109/CSCI62032.2023.00012
Deo NBasak JSoliman AWeinberg DSteorts RRajasekaran S(2023)Novel Blocking Techniques and Distance Metrics for Record LinkageInformation Integration and Web Intelligence10.1007/978-3-031-48316-5_40(431-446)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1007/978-3-031-48316-5_40
Jain ASarawagi SSen P(2022)Deep indexed active learning for matching heterogeneous entity representationsProceedings of the VLDB Endowment10.14778/3485450.348545515:1(31-45)Online publication date: 14-Jan-2022
https://dl.acm.org/doi/10.14778/3485450.3485455
Mittal DSangwan AYadav V(2022)Efficient Entity Resolution for Bibliographic Data Using MapReduceAdvanced Informatics for Computing Research10.1007/978-3-031-09469-9_17(192-200)Online publication date: 25-Jun-2022
https://doi.org/10.1007/978-3-031-09469-9_17
Galhotra SFirmani DSaha BSrivastava DLi GLi ZIdreos SSrivastava D(2021)BEERProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3452747(2711-2715)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3452747
Sharma SSaini H(2021)An Enhanced Clustering Algorithm for Agricultural Data Using FAO Heuristics2021 6th International Conference on Signal Processing, Computing and Control (ISPCC)10.1109/ISPCC53510.2021.9609397(228-232)Online publication date: 7-Oct-2021
https://doi.org/10.1109/ISPCC53510.2021.9609397
Osawa NIto HFukushima YHarada TMorishima A(2021)BUBBLE : A Quality-Aware Human-in-the-loop Entity Matching Framework2021 IEEE International Conference on Big Data (Big Data)10.1109/BigData52589.2021.9672002(3557-3565)Online publication date: 15-Dec-2021
https://doi.org/10.1109/BigData52589.2021.9672002
Boniol PPalpanas T(2020)Series2GraphProceedings of the VLDB Endowment10.14778/3407790.340779213:12(1821-1834)Online publication date: 1-Jul-2020
https://dl.acm.org/doi/10.14778/3407790.3407792
Ge TLi Z(2020)Approximate substring matching over uncertain stringsProceedings of the VLDB Endowment10.14778/3402707.34027174:11(772-782)Online publication date: 3-Jun-2020
https://dl.acm.org/doi/10.14778/3402707.3402717
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

The merge/purge problem for large databases

A generalization of band joins and the merge/purge problem

Entity Resolution for Large-Scale Databases