Near duplicate detection benefits many applications, e.g., on-line news selection over the Web by keyword search. The purpose of this demo is to show the design and implementation of MapDupReducer, a MapReduce based system capable of detecting near duplicates over massive datasets efficiently.

References

[1]

J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In Proceedings of the 6th Symposium on Operating System Design and Implementation, pages 137--150, San Francisco, California, USA, December 2004. USENIX Association.

Digital Library

Google Scholar

[2]

A. K. Elmagarmid, P. G. Ipeirotis, and V. S. Verykios. Duplicate Record Detection: A Survey. IEEE Transactions on Knowledge and Data Engineering, 19(1):1--16, 2007.

Digital Library

Google Scholar

[3]

A. Kołcz, A. Chowdhury, and J. Alspector. Improved Robustness of Signature-based Near-replica Detection via Lexicon Randomization. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 605--610, Seattle, Washington, USA, August 2004. ACM Press.

Digital Library

Google Scholar

[4]

J. Lin. Brute Force and Indexed Approaches to Pairwise Document Similarity Comparisons with MapReduce. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 155--162, Boston, MA, USA, July 2009. ACM Press.

Digital Library

Google Scholar

[5]

C. Xiao, W. Wang, X. Lin, and J. X. Yu. Efficient Similarity Joins for Near Duplicate Detection. In Proceedings of the 17th International Conference on World Wide Web, pages 131--140, Beijing, China, April 2008. ACM Press.

Digital Library

Google Scholar

Cited By

View all

Zhu WZhang MChen CWang XZhang FLin X(2019)Pivotal relationship identificationProceedings of the 28th International Joint Conference on Artificial Intelligence10.5555/3367471.3367721(4874-4880)Online publication date: 10-Aug-2019
https://dl.acm.org/doi/10.5555/3367471.3367721
Wang KLin XQin LZhang WZhang Y(2019)Vertex priority based butterfly counting for large-scale bipartite networksProceedings of the VLDB Endowment10.14778/3339490.333949712:10(1139-1152)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.14778/3339490.3339497
Yadav PSharma SSingh A(2019)Big Data and Cloud Computing: An Emerging Perspective and Future Trends2019 International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT)10.1109/ICICT46931.2019.8977674(1-4)Online publication date: Sep-2019
https://doi.org/10.1109/ICICT46931.2019.8977674
Show More Cited By

Index Terms

MapDupReducer: detecting near duplicates over massive datasets
1. Information systems
  1. Information retrieval
    1. Search engine architectures and scalability
      1. Distributed retrieval
      2. Peer-to-peer retrieval
  2. Information storage systems
    1. Storage architectures
      1. Distributed storage

Recommendations

Near duplicate detection in an academic digital library
DocEng '13: Proceedings of the 2013 ACM symposium on Document engineering

The detection and potential removal of duplicates is desirable for a number of reasons, such as to reduce the need for unnecessary storage and computation, and to provide users with uncluttered search results. This paper describes an investigation into ...
Scene duplicate detection from videos based on trajectories of feature points
MIR '07: Proceedings of the international workshop on Workshop on multimedia information retrieval

Recently near duplicate shot detection attracts researchers attention. There are several promising applications of near duplicate detection, especially when applied to broadcast video streams. However, currently studied near duplicates are limited to: a)...
MapReduce: Review and open challenges

The continuous increase in computational capacity over the past years has produced an overwhelming flow of data or big data, which exceeds the capabilities of conventional processing tools. Big data signify a new era in data exploration and utilization. ...

Comments

Information & Contributors

Information

Published In

SIGMOD '10: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data

June 2010

1286 pages

ISBN:9781450300322

DOI:10.1145/1807167

General Chair:
Ahmed Elmagarmid
Purdue University, USA
,
Program Chair:
Divyakant Agrawal
University of California at Santa Barbara, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 June 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Demonstration

Conference

SIGMOD/PODS '10

Sponsor:

SIGMOD

SIGMOD/PODS '10: International Conference on Management of Data

June 6 - 10, 2010

Indiana, Indianapolis, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
702
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)2

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zhu WZhang MChen CWang XZhang FLin X(2019)Pivotal relationship identificationProceedings of the 28th International Joint Conference on Artificial Intelligence10.5555/3367471.3367721(4874-4880)Online publication date: 10-Aug-2019
https://dl.acm.org/doi/10.5555/3367471.3367721
Wang KLin XQin LZhang WZhang Y(2019)Vertex priority based butterfly counting for large-scale bipartite networksProceedings of the VLDB Endowment10.14778/3339490.333949712:10(1139-1152)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.14778/3339490.3339497
Yadav PSharma SSingh A(2019)Big Data and Cloud Computing: An Emerging Perspective and Future Trends2019 International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT)10.1109/ICICT46931.2019.8977674(1-4)Online publication date: Sep-2019
https://doi.org/10.1109/ICICT46931.2019.8977674
Lai LQin LLin XChang L(2018)Scalable subgraph enumeration in MapReduceProceedings of the VLDB Endowment10.14778/2794367.27943688:10(974-985)Online publication date: 5-Oct-2018
https://dl.acm.org/doi/10.14778/2794367.2794368
Mittal DPilli EGupta M(2017)Efficient entity resolution using multiple blocking keys for bibliographic dataset2017 International Conference on Intelligent Communication and Computational Techniques (ICCT)10.1109/INTELCCT.2017.8324029(108-113)Online publication date: Dec-2017
https://doi.org/10.1109/INTELCCT.2017.8324029
Pizarro V. GVelsquez J(2017)Docode 5Engineering Applications of Artificial Intelligence10.1016/j.engappai.2017.06.00164:C(261-271)Online publication date: 1-Sep-2017
https://dl.acm.org/doi/10.1016/j.engappai.2017.06.001
Wang CWang CWang HChen JYe X(2016)Preference Join on Heterogeneous DataWeb Technologies and Applications10.1007/978-3-319-45817-5_25(317-329)Online publication date: 18-Sep-2016
https://doi.org/10.1007/978-3-319-45817-5_25
Lai LQin LLin XChang L(2015)Scalable subgraph enumeration in MapReduceProceedings of the VLDB Endowment10.5555/2794367.27943688:10(974-985)Online publication date: 1-Jun-2015
https://dl.acm.org/doi/10.5555/2794367.2794368
Mestre DPires CNascimento DWainwright RCorchado JBechini AHong J(2015)Adaptive sorted neighborhood blocking for entity matching with MapReduceProceedings of the 30th Annual ACM Symposium on Applied Computing10.1145/2695664.2695757(981-987)Online publication date: 13-Apr-2015
https://dl.acm.org/doi/10.1145/2695664.2695757
Yan CSong YWang JGuo WBalaji PXu C(2015)Eliminating the redundancy in MapReduce-based entity resolutionProceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2015.24(1233-1236)Online publication date: 4-May-2015
https://dl.acm.org/doi/10.1109/CCGrid.2015.24
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Near duplicate detection in an academic digital library

Scene duplicate detection from videos based on trajectories of feature points

MapReduce: Review and open challenges