Article

Building high accuracy bloom filters using partitioned hashing

Authors:

SIGMETRICS '07: Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems

June 2007

Pages 277 - 288

https://doi.org/10.1145/1254882.1254916

Published: 12 June 2007 Publication History

Get Access

Abstract

The growing importance of operations such as packet-content inspection, packet classification based on non-IP headers, maintaining flow-state, etc. has led to increased interest in the networking applications of Bloom filters. This is because Bloom filters provide a relatively easy method for hardware implementation of set-membership queries. However, the tradeoff is that Bloom filters only provide a probabilistic test and membership queries can result in false positives. Ideally, we would like this false positive probability to be very low. The main contribution of this paper is a method for significantly reducing this false positive probability in comparison to existing schemes. This is done by developing a partitioned hashing method which results in a choice of hash functions that set far fewer bits in the Bloom filter bit vector than would be the case otherwise. This lower fill factor of the bit vector translates to a much lower false positive probability. We show experimentally that this improved choice can result in as much as a ten-fold increase in accuracy over standard Bloom filters. We also show that the scheme performs much better than other proposed schemes for improving Bloom filters.

References

[1]

B. H. Bloom, "Space/time tradeoffs in hash coding with allowable errors", Communications of the ACM 13:7 (1970), 422--426.

Digital Library

Google Scholar

[2]

A. Kirsch and M. Mitzenmacher, "Less hashing, same performance: building a better Bloom filter", ESA 2006.

Digital Library

Google Scholar

[3]

S. Lumetta and M. Mitzenmacher, "Using the power of two choices to improve Bloom filters", Preprint version available at http://www.eecs.harvard.edu/~michaelm.

Google Scholar

[4]

A. Broder and M. Mitzenmacher, "Network applications fo Bloom filters: a survey", Internet Mathematics, vol. 1. no. 4, pp. 485--509, 2005.

Google Scholar

[5]

R. Motwani and P. Raghavan, "Randomized algorithms", Cambridge University Press, August, 1995.

Digital Library

Google Scholar

[6]

Robert Sedgewick, "Algorithms in C", Addison-Wesley Professional, August, 2001.

Google Scholar

[7]

A. Ostlin and R. Pagh, "Uniform hashing in constant time and linear space", Proceedings of STOC 2003, ACM.

Digital Library

Google Scholar

[8]

A. Pagh, R. Pagh, and S. Rao, "An optimal Bloom filter replacement", SODA 2005.

Digital Library

Google Scholar

[9]

B. Chazelle, J. Kilian, R. Rubinfeld, and A. Tal, "The Bloomier filter: an efficient data structure for static support lookup tables", SODA 2004.

Digital Library

Google Scholar

[10]

C. Estan and G. Varghese, "New directions in traffic measurement and accounting", Proceedings of ACM SIGCOMM Conference, 2002.

Digital Library

Google Scholar

[11]

L. Fan, P. Cao, J. Almeida, and A. Z. Broder, "Summary cache: a scalable wide-area web cache sharing protocol", IEEE/ACM Transactions on Networking 8:3 (2000), 281--293.

Digital Library

Google Scholar

[12]

E. H. Spafford, "Opus: preventing weak password choices", Computer and Security 11 (1992), 273--278.

Digital Library

Google Scholar

[13]

U. Manber and S. Wu, "An algorithm for approximate membership checking with application to password security", Information Processing Letters 50 (1994), 19--197.

Digital Library

Google Scholar

[14]

F. Bonomi, M. Mitzenmacher, R. Panigrahy, S. Singh, and G. Varghese, "Beyond Bloom filters: from approximate membership checks to approximate state machines", Proceedings of ACM SIGCOMM 2006.

Digital Library

Google Scholar

Cited By

View all

Duong QNguyen KNguyen HDao Xdo ATrinh Q(2024)FeCBF: A Novel Sub-Optimal Cascaded Bloom Filter Structure Based on Feature ExtractionIEEE Access10.1109/ACCESS.2024.339906212(67619-67631)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3399062
Li MXie RChen DDai HGu RHuang HDou WChen G(2022)A Pareto optimal Bloom filter family with hash adaptivityThe VLDB Journal10.1007/s00778-022-00755-z32:3(525-548)Online publication date: 26-Jul-2022
https://doi.org/10.1007/s00778-022-00755-z
Xie RLi MMiao ZGu RHuang HDai HChen G(2021)Hash Adaptive Bloom Filter2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00061(636-647)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00061
Show More Cited By

Index Terms

Building high accuracy bloom filters using partitioned hashing
1. Hardware
  1. Communication hardware, interfaces and storage
2. Networks

Recommendations

Building high accuracy bloom filters using partitioned hashing
SIGMETRICS '07 Conference Proceedings

The growing importance of operations such as packet-content inspection, packet classification based on non-IP headers, maintaining flow-state, etc. has led to increased interest in the networking applications of Bloom filters. This is because Bloom ...
Read More
False Negative Problem of Counting Bloom Filter

Bloom filter is effective, space-efficient data structure for concisely representing a data set and supporting approximate membership queries. Traditionally, researchers often believe that it is possible that a Bloom filter returns a false positive, but ...
Read More
The Bloom paradox: when not to use a Bloom filter

In this paper, we uncover the Bloom paradox in Bloom Filters: Sometimes, the Bloom Filter is harmful and should not be queried. We first analyze conditions under which the Bloom paradox occurs in a Bloom Filter and demonstrate that it depends on the a ...
Read More

Comments

Information & Contributors

Information

Published In

SIGMETRICS '07: Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems

June 2007

398 pages

ISBN:9781595936394

DOI:10.1145/1254882

General Chair:
Leana Golubchik
University of Southern California, USA
,
Program Chairs:
Mostafa Ammar
Georgia Institute of Technology, USA
,
Mor Harchol-Balter
Carnegie Mellon University, USA

ACM SIGMETRICS Performance Evaluation Review Volume 35, Issue 1
SIGMETRICS '07 Conference Proceedings
June 2007
382 pages
ISSN:0163-5999
DOI:10.1145/1269899
Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGMETRICS07

Sponsor:

SIGMETRICS07: ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems

June 12 - 16, 2007

California, San Diego, USA

Acceptance Rates

Overall Acceptance Rate 459 of 2,691 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
1,544
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)1

Other Metrics

View Author Metrics

Citations

Cited By

View all

Duong QNguyen KNguyen HDao Xdo ATrinh Q(2024)FeCBF: A Novel Sub-Optimal Cascaded Bloom Filter Structure Based on Feature ExtractionIEEE Access10.1109/ACCESS.2024.339906212(67619-67631)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3399062
Li MXie RChen DDai HGu RHuang HDou WChen G(2022)A Pareto optimal Bloom filter family with hash adaptivityThe VLDB Journal10.1007/s00778-022-00755-z32:3(525-548)Online publication date: 26-Jul-2022
https://doi.org/10.1007/s00778-022-00755-z
Xie RLi MMiao ZGu RHuang HDai HChen G(2021)Hash Adaptive Bloom Filter2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00061(636-647)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00061
Luo LGuo DMa RRottenstreich OLuo X(2019)Optimizing Bloom Filter: Challenges, Solutions, and ComparisonsIEEE Communications Surveys & Tutorials10.1109/COMST.2018.288932921:2(1912-1949)Online publication date: Oct-2020
https://doi.org/10.1109/COMST.2018.2889329
Yu JChen LZhang RWang K(2017)Finding Needles in a Haystack: Missing Tag Detection in Large RFID SystemsIEEE Transactions on Communications10.1109/TCOMM.2017.266679065:5(2036-2047)Online publication date: May-2017
https://doi.org/10.1109/TCOMM.2017.2666790
Fu YBiersack E(2016)False-Positive Probability and Compression Optimization for Tree-Structured Bloom FiltersACM Transactions on Modeling and Performance Evaluation of Computing Systems10.1145/29403241:4(1-39)Online publication date: 21-Sep-2016
https://dl.acm.org/doi/10.1145/2940324
Gosselin-Lavigne MGonzalez HStakhanova NGhorbani A(2015)A Performance Evaluation of Hash Functions for IP Reputation Lookup Using Bloom FiltersProceedings of the 2015 10th International Conference on Availability, Reliability and Security10.1109/ARES.2015.101(516-521)Online publication date: 24-Aug-2015
https://dl.acm.org/doi/10.1109/ARES.2015.101
Ni SGuo RLiao XJin H(2015)Parallel Bloom Filter on Xeon Phi Many-Core ProcessorsProceedings, Part II, of the 15th International Conference on Algorithms and Architectures for Parallel Processing - Volume 952910.1007/978-3-319-27122-4_27(388-405)Online publication date: 18-Nov-2015
https://dl.acm.org/doi/10.1007/978-3-319-27122-4_27
Qiao YLi TChen S(2014)Fast Bloom Filters and Their GeneralizationIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2013.4625:1(93-103)Online publication date: 1-Jan-2014
https://dl.acm.org/doi/10.1109/TPDS.2013.46
Saravanan KSenthilkumar A(2014)Investigation on bloom filter and implementation of 3k combined parallel tiger bloom filter design2014 International Conference on Electronics and Communication Systems (ICECS)10.1109/ECS.2014.6892509(1-7)Online publication date: Feb-2014
https://doi.org/10.1109/ECS.2014.6892509
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Building high accuracy bloom filters using partitioned hashing

False Negative Problem of Counting Bloom Filter

The Bloom paradox: when not to use a Bloom filter