research-article

Is min-wise hashing optimal for summarizing set intersection?

Authors:

Rasmus Pagh,

Morten Stöckel,

David P. WoodruffAuthors Info & Claims

PODS '14: Proceedings of the 33rd ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

Pages 109 - 120

https://doi.org/10.1145/2594538.2594554

Published: 18 June 2014 Publication History

Get Access

Abstract

Min-wise hashing is an important method for estimating the size of the intersection of sets, based on a succinct summary (a "min-hash") of each set. One application is estimation of the number of data points that satisfy the conjunction of m >= 2 simple predicates, where a min-hash is available for the set of points satisfying each predicate. This has application in query optimization and for approximate computation of COUNT aggregates.

In this paper we address the question: How many bits is it necessary to allocate to each summary in order to get an estimate with (1 +/- epsilon)-relative error? The state-of-the-art technique for minimizing the encoding size, for any desired estimation error, is b-bit min-wise hashing due to Li and König (Communications of the ACM, 2011). We give new lower and upper bounds:

Using information complexity arguments, we show that b-bit min-wise hashing is em space optimal for m=2 predicates in the sense that the estimator's variance is within a constant factor of the smallest possible among all summaries with the given space usage. But for conjunctions of m>2 predicates we show that the performance of b-bit min-wise hashing (and more generally any method based on "k-permutation" min-hash) deteriorates as m grows.

We describe a new summary that nearly matches our lower bound for m >= 2. It asymptotically outperform all k-permutation schemes (by around a factor Omega(m/log m)), as well as methods based on subsampling (by a factor Omega(log n_max), where n_max is the maximum set size).

References

[1]

Z. Bar-Yossef. The complexity of massive data set computations. PhD thesis, University of California at Berkeley, 2002.

Digital Library

Google Scholar

[2]

P. Bille, A. Pagh, and R. Pagh. Fast evaluation of union-intersection expressions. In Proceedings of the 18th International Symposium on Algorithms And Computation (ISAAC '07), pages 739--750.

Digital Library

Google Scholar

[3]

B. H. Bloom. Space/time trade-offs in hash coding with allowable errors. Commun. ACM, 13(7):422--426, 1970.

Digital Library

Google Scholar

[4]

M. Braverman, A. Garg, D. Pankratov, and O. Weinstein. Information lower bounds via self-reducibility. In CSR, pages 183--194, 2013.

Crossref

Google Scholar

[5]

A. Z. Broder. On the resemblance and containment of documents. In In Compression and Complexity of Sequences (SEQUENCES), pages 21--29, 1997.

Digital Library

Google Scholar

[6]

A. Z. Broder, M. Charikar, A. M. Frieze, and M. Mitzenmacher. Min-wise independent permutations. Journal of Computer and System Sciences, 60:327--336, 1998.

Digital Library

Google Scholar

[7]

A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the web. In Selected papers from the sixth international conference on World Wide Web, pages 1157--1166, 1997.

Digital Library

Google Scholar

[8]

A. Chakrabarti and O. Regev. An optimal lower bound on the communication complexity of gap-hamming-distance. SIAM J. Comput., 41(5):1299--1317, 2012.

Digital Library

Google Scholar

Cited By

View all

Pavan AChakraborty SVinodchandran NMeel K(2024)On the Feasibility of Forgetting in Data StreamsProceedings of the ACM on Management of Data10.1145/36516032:2(1-17)Online publication date: 14-May-2024
https://dl.acm.org/doi/10.1145/3651603
Sadeequllah MRauf ARehman SAlnazzawi N(2024)Probabilistic Support Prediction: Fast Frequent Itemset Mining in Dense DataIEEE Access10.1109/ACCESS.2024.337647712(39330-39350)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3376477
Derbeko PDolev SGudes EUllman J(2022)Efficient and Privacy Preserving Approximation of Distributed Statistical QueriesIEEE Transactions on Big Data10.1109/TBDATA.2021.30525168:5(1399-1413)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TBDATA.2021.3052516
Show More Cited By

Index Terms

Is min-wise hashing optimal for summarizing set intersection?
1. Theory of computation
  1. Design and analysis of algorithms

Recommendations

Multi-Party Set Disjointness and Intersection with Bounded Dependence
PODC '24: Proceedings of the 43rd ACM Symposium on Principles of Distributed Computing

In the multi-party set disjointness problem, k players receive private inputs in the form of sets X¹, ..., X^k ⊆ [n], and their goal is to check whether their sets intersect. The set intersection problem is similar, except that the players are required to ...
Dynamic Perfect Hashing: Upper and Lower Bounds

The dynamic dictionary problem is considered: provide an algorithm for storing a dynamic set, allowing the operations insert, delete, and lookup. A dynamic perfect hashing strategy is given: a randomized algorithm for the dynamic dictionary problem ...
Constructing Set-Systems with Prescribed Intersection Size

Comments

Information & Contributors

Information

Published In

PODS '14: Proceedings of the 33rd ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

June 2014

300 pages

ISBN:9781450323758

DOI:10.1145/2594538

General Chair:
Richard Hull
IBM T.J. Watson Research Center, USA
,
Program Chair:
Martin Grohe
RWTH Aachen University, Germany

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS'14

Sponsor:

SIGMOD

SIGMOD/PODS'14: International Conference on Management of Data

June 22 - 27, 2014

Utah, Snowbird, USA

Acceptance Rates

PODS '14 Paper Acceptance Rate 22 of 67 submissions, 33%;

Overall Acceptance Rate 642 of 2,707 submissions, 24%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
317
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)3

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Pavan AChakraborty SVinodchandran NMeel K(2024)On the Feasibility of Forgetting in Data StreamsProceedings of the ACM on Management of Data10.1145/36516032:2(1-17)Online publication date: 14-May-2024
https://dl.acm.org/doi/10.1145/3651603
Sadeequllah MRauf ARehman SAlnazzawi N(2024)Probabilistic Support Prediction: Fast Frequent Itemset Mining in Dense DataIEEE Access10.1109/ACCESS.2024.337647712(39330-39350)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3376477
Derbeko PDolev SGudes EUllman J(2022)Efficient and Privacy Preserving Approximation of Distributed Statistical QueriesIEEE Transactions on Big Data10.1109/TBDATA.2021.30525168:5(1399-1413)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TBDATA.2021.3052516
Mathieu CRajaraman RYoung NYousefi AMarx D(2021)Competitive data-structure dynamizationProceedings of the Thirty-Second Annual ACM-SIAM Symposium on Discrete Algorithms10.5555/3458064.3458199(2269-2287)Online publication date: 10-Jan-2021
https://dl.acm.org/doi/10.5555/3458064.3458199
Yu YWeber G(2020)HyperMinHash: MinHash in LogLog spaceIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2020.2981311(1-1)Online publication date: 2020
https://doi.org/10.1109/TKDE.2020.2981311
Bury MSchwiegelshohn CSorella M(2020)Similarity Search for Dynamic Data StreamsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2019.291685832:11(2241-2253)Online publication date: 1-Nov-2020
https://doi.org/10.1109/TKDE.2019.2916858
Ahle TKnudsen J(2020)Subsets and Supermajorities: Optimal Hashing-based Set Similarity Search2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS)10.1109/FOCS46700.2020.00073(728-739)Online publication date: Nov-2020
https://doi.org/10.1109/FOCS46700.2020.00073
Huang DYoon DPettie SMozafari B(2019)Joins on samplesProceedings of the VLDB Endowment10.14778/3372716.337272613:4(547-560)Online publication date: 9-Dec-2019
https://dl.acm.org/doi/10.14778/3372716.3372726
Zhang CTian PZhang XJiang ZYao LWang X(2019)Fast Eclat Algorithms Based on Minwise Hashing for Large Scale TransactionsIEEE Internet of Things Journal10.1109/JIOT.2018.28858516:2(3948-3961)Online publication date: Apr-2019
https://doi.org/10.1109/JIOT.2018.2885851
Castro Fernandez RMin JNava DMadden S(2019)Lazo: A Cardinality-Based Method for Coupled Estimation of Jaccard Similarity and Containment2019 IEEE 35th International Conference on Data Engineering (ICDE)10.1109/ICDE.2019.00109(1190-1201)Online publication date: Apr-2019
https://doi.org/10.1109/ICDE.2019.00109
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Multi-Party Set Disjointness and Intersection with Bounded Dependence

Dynamic Perfect Hashing: Upper and Lower Bounds

Constructing Set-Systems with Prescribed Intersection Size

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Multi-Party Set Disjointness and Intersection with Bounded Dependence

Dynamic Perfect Hashing: Upper and Lower Bounds

Constructing Set-Systems with Prescribed Intersection Size

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations