research-article

One is enough: distributed filtering for duplicate elimination

Authors:

Georgia Koloniari,

Evaggelia Pitoura,

Dimitris SouravliasAuthors Info & Claims

CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge management

Pages 433 - 442

https://doi.org/10.1145/2063576.2063643

Published: 24 October 2011 Publication History

Abstract

The growth of online services has created the need for duplicate elimination in high-volume streams of events. The sheer volume of data in applications such as pay-per-click clickstream processing, RSS feed syndication and notification services in social sites such Twitter and Facebook makes traditional centralized solutions hard to scale. In this paper, we propose an approach based on distributed filtering. To this end, we introduce a suite of distributed Bloom filters that exploit different ways of partitioning the event space. To address the continuous nature of event delivery, the filters are extended to support sliding window semantics. Moreover, we examine locality-related tradeoffs and propose a tree-based architecture to allow for duplicate elimination across geographic locations. We cast the design space and present experimental results that demonstrate the pros and cons of our various solutions in different settings.

References

[1]

P. S. Almeidaa, C. Baqueroa, N. Preguiça, and D. Hutchison. Scalable bloom filters. Information Processing Letters, 101(6):255--261, 2007.

Digital Library

[2]

B. H. Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422--426, 1970.

Digital Library

[3]

A. Broder and M. Mitzenmacher. Network applications of bloom filters: A survey. Internet Mathematics, 1(4):485--509, 2002.

[4]

A. Chen, Y. Jin, J. Cao, and L. E. Li. Tracking long duration flows in network traffic. In INFOCOM, 2010.

Digital Library

[5]

S. Cohen and Y. Matias. Spectral bloom filters. In SIGMOD Conference, 2003.

Digital Library

[6]

I. Dar, T. Milo, and E. Verbin. Optimized union of non-disjoint distributed data sets. In EDBT, 2009.

Digital Library

[7]

F. Deng and D. Rafiei. Approximately detecting duplicates for streaming data using stable bloom filters. In SIGMOD Conference, 2006.

Digital Library

[8]

C. Estan and G. Varghese. New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice. ACM Trans. Comput. Syst., 21(3):270--313, 2003.

Digital Library

[9]

L. Fan, P. Cao, J. M. Almeida, and A. Z. Broder. Summary cache: a scalable wide-area web cache sharing protocol. IEEE/ACM Trans. Netw., 8(3):281--293, 2000.

Digital Library

[10]

D. Guo, J. Wu, H. Chen, Y. Yuan, and X. Luo. The dynamic bloom filters. IEEE TKDE, 22(1):120--133, 2010.

Digital Library

[11]

R. Jain, D. Chiu, and W. Hawe. A quantitative measure of fairness and discrimination for resource allocation in shared computer systems. DEC Research Report TR-301, 1984.

[12]

G. Koloniari and E. Pitoura. Content-based routing of path queries in peer-to-peer systems. In EDBT, 2004.

[13]

S. Majumdar, D. Kulkarni, and C. Ravishankar. Addressing click fraud in content delivery systems. In Proc. INFOCOM, 2007.

Digital Library

[14]

A. Metwally, D. Agrawal, and A. E. Abbadi. Duplicate detection in click streams. In WWW, 2005.

Digital Library

[15]

O. Papapetrou, W. Siberski, and W. Nejdl. Cardinality estimation and dynamic length adaptation for bloom filters. Distributed and Parallel Databases, 28(2,3):119--156, 2010.

Digital Library

[16]

S. C. Rhea and J. Kubiatowicz. Probabilistic location and routing. In Proc. INFOCOM, 2002.

[17]

X. Wang, Q. Zhang, and Y. Jia. Efficiently filtering duplicates over distributed data streams. In Proc. CSSE, 2008.

Digital Library

[18]

T. Xia, C. Jin, X. Zhou, and A. Zhou. Filtering duplicate items over distributed data streams. In WAIM, 2005.

Digital Library

Cited By

Kolomvatsos KAnagnostopoulos C(2024)Autonomous proactive data management in support of pervasive edge applicationsFuture Generation Computer Systems10.1016/j.future.2024.02.003155(108-120)Online publication date: Jun-2024
https://doi.org/10.1016/j.future.2024.02.003
Papathanasiou DKolomvatsos K(2024)Data management and selectivity in collaborative pervasive edge computingComputing10.1007/s00607-024-01297-8106:8(2561-2584)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s00607-024-01297-8
Tian YYan TZhang RHuang KZheng BZhou X(2023)A Learned Cuckoo Filter for Approximate Membership Queries over Variable-sized Sliding Windows on Data StreamsProceedings of the ACM on Management of Data10.1145/36267581:4(1-26)Online publication date: 12-Dec-2023
https://doi.org/10.1145/3626758
Show More Cited By

Index Terms

One is enough: distributed filtering for duplicate elimination
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Document filtering
      2. Information extraction

Recommendations

An approximate duplicate elimination in RFID data streams

The RFID technology has been applied to a wide range of areas since it does not require contact in detecting RFID tags. However, due to the multiple readings in many cases in detecting an RFID tag and the deployment of multiple readers, RFID data ...
Composition can be Faster than Join
COMPSAC '97: Proceedings of the 21st International Computer Software and Applications Conference

In relational databases, a composition requires three operations: join, projection and duplicate elimination. An expensive external sort is required to eliminate duplicates in a large file. Most conventional composition algorithms take join and ...
Duplicate record elimination in large data files

The issue of duplicate elimination for large data files in which many occurrences of the same record may appear is addressed. A comprehensive cost analysis of the duplicate elimination operation is presented. This analysis is based on a combinatorial ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge management

October 2011

2712 pages

ISBN:9781450307178

DOI:10.1145/2063576

Editors:
Bettina Berendt,
Arjen de Vries,
Wenfei Fan,
Craig Macdonald
University of Glasgow, UK
,
Iadh Ounis
University of Glasgow, UK
,
Ian Ruthven
University of Strathclyde, UK

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM '11

Sponsor:

CIKM '11: International Conference on Information and Knowledge Management

October 24 - 28, 2011

Glasgow, Scotland, UK

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
325
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kolomvatsos KAnagnostopoulos C(2024)Autonomous proactive data management in support of pervasive edge applicationsFuture Generation Computer Systems10.1016/j.future.2024.02.003155(108-120)Online publication date: Jun-2024
https://doi.org/10.1016/j.future.2024.02.003
Papathanasiou DKolomvatsos K(2024)Data management and selectivity in collaborative pervasive edge computingComputing10.1007/s00607-024-01297-8106:8(2561-2584)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s00607-024-01297-8
Tian YYan TZhang RHuang KZheng BZhou X(2023)A Learned Cuckoo Filter for Approximate Membership Queries over Variable-sized Sliding Windows on Data StreamsProceedings of the ACM on Management of Data10.1145/36267581:4(1-26)Online publication date: 12-Dec-2023
https://doi.org/10.1145/3626758
Dautrich JRavishankar C(2019)Inferring Insertion Times and Optimizing Error Penalties in Time-decaying Bloom FiltersACM Transactions on Database Systems10.1145/328455244:2(1-32)Online publication date: 15-Mar-2019
https://dl.acm.org/doi/10.1145/3284552
Patgiri RBorgohain SBhattacharjee A(2018)rFilter: A Scalable and Space-efficient Membership Filter2018 5th International Conference on Signal Processing and Integrated Networks (SPIN)10.1109/SPIN.2018.8474044(478-485)Online publication date: Feb-2018
https://doi.org/10.1109/SPIN.2018.8474044
Dasgupta SSheehan TStevens CNavlakha S(2018)A neural data structure for novelty detectionProceedings of the National Academy of Sciences10.1073/pnas.1814448115115:51(13093-13098)Online publication date: 3-Dec-2018
https://doi.org/10.1073/pnas.1814448115
Fu YBiersack E(2016)False-Positive Probability and Compression Optimization for Tree-Structured Bloom FiltersACM Transactions on Modeling and Performance Evaluation of Computing Systems10.1145/29403241:4(1-39)Online publication date: 21-Sep-2016
https://dl.acm.org/doi/10.1145/2940324
Ghali CTsudik GWood C(2016)BEAD: Best effort autonomous deletion in content-centric networking2016 IFIP Networking Conference (IFIP Networking) and Workshops10.1109/IFIPNetworking.2016.7497241(180-188)Online publication date: May-2016
https://doi.org/10.1109/IFIPNetworking.2016.7497241
Dautrich JRavishankar CGuerrini GPaton N(2013)Inferential time-decaying Bloom filtersProceedings of the 16th International Conference on Extending Database Technology10.1145/2452376.2452405(239-250)Online publication date: 18-Mar-2013
https://dl.acm.org/doi/10.1145/2452376.2452405
Zhang QMa HQian WZhou A(2013)Duplicate Detection for Identifying Social Spam in MicroblogsProceedings of the 2013 IEEE International Congress on Big Data10.1109/BigData.Congress.2013.27(141-148)Online publication date: 27-Jun-2013
https://dl.acm.org/doi/10.1109/BigData.Congress.2013.27
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents