research-article

HyperLogLog in practice: algorithmic engineering of a state of the art cardinality estimation algorithm

Authors:

Marc Nunkesser, and

Alexander HallAuthors Info & Claims

EDBT '13: Proceedings of the 16th International Conference on Extending Database Technology

March 2013

Pages 683 - 692

https://doi.org/10.1145/2452376.2452456

Published: 18 March 2013 Publication History

Abstract

Cardinality estimation has a wide range of applications and is of particular importance in database systems. Various algorithms have been proposed in the past, and the HyperLogLog algorithm is one of them. In this paper, we present a series of improvements to this algorithm that reduce its memory requirements and significantly increase its accuracy for an important range of cardinalities. We have implemented our proposed algorithm for a system at Google and evaluated it empirically, comparing it to the original HyperLogLog algorithm. Like HyperLogLog, our improved algorithm parallelizes perfectly and computes the cardinality estimate in a single pass.

References

[1]

K. Aouiche and D. Lemire. A comparison of five probabilistic view-size estimation techniques in OLAP. In Workshop on Data Warehousing and OLAP (DOLAP), pages 17--24, 2007.

Digital Library

[2]

Z. Bar-Yossef, T. S. Jayram, R. Kumar, D. Sivakumar, and L. Trevisan. Counting distinct elements in a data stream. In Workshop on Randomization and Approximation Techniques (RANDOM), pages 1--10, London, UK, UK, 2002. Springer-Verlag.

Digital Library

[3]

P. Clifford and I. A. Cosma. A statistical analysis of probabilistic counting algorithms. Scandinavian Journal of Statistics, pages 1--14, 2011.

[4]

M. Durand and P. Flajolet. Loglog counting of large cardinalities. In G. D. Battista and U. Zwick, editors, European Symposium on Algorithms (ESA), volume 2832, pages 605--617, 2003.

[5]

C. Estan, G. Varghese, and M. Fisk. Bitmap algorithms for counting active flows on high-speed links. IEEE/ACM Transactions on Networking, pages 925--937, 2006.

Digital Library

[6]

P. Flajolet and G. N. Martin. Probabilistic counting algorithms for data base applications. Journal of Computer and System Sciences, 31(2):182--209, 1985.

Digital Library

[7]

P. Flajolet, Éric Fusy, O. Gandouet, and F. Meunier. Hyperloglog: The analysis of a near-optimal cardinality estimation algorithm. In Analysis of Algorithms (AOFA), pages 127--146, 2007.

[8]

F. Giroire. Order statistics and estimating cardinalities of massive data sets. Discrete Applied Mathematics, 157(2):406--427, 2009.

Digital Library

[9]

A. Hall, O. Bachmann, R. Büssow, S. Gănceanu, and M. Nunkesser. Processing a trillion cells per mouse click. In Very Large Databases (VLDB), 2012.

Digital Library

[10]

P. Indyk. Tight lower bounds for the distinct elements problem. In Foundations of Computer Science (FOCS), pages 283--288, 2003.

Digital Library

[11]

D. M. Kane, J. Nelson, and D. P. Woodruff. An optimal algorithm for the distinct elements problem. In Principles of database systems (PODS), pages 41--52. ACM, 2010.

Digital Library

[12]

J. Lumbroso. An optimal cardinality estimation algorithm based on order statistics and its full analysis. In Analysis of Algorithms (AOFA), pages 489--504, 2010.

[13]

S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, T. Vassilakis, and G. Inc. Dremel: Interactive analysis of web-scale datasets. In Very Large Databases (VLDB), pages 330--339, 2010.

Digital Library

[14]

A. Metwally, D. Agrawal, and A. E. Abbadi. Why go logarithmic if we can go linear? Towards effective distinct counting of search traffic. In Extending database technology (EDBT), pages 618--629, 2008.

Digital Library

[15]

R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data, parallel analysis with Sawzall. Journal on Scientific Programming, pages 277--298, 2005.

Digital Library

[16]

K.-Y. Whang, B. T. Vander-Zanden, and H. M. Taylor. A linear-time probabilistic counting algorithm for database applications. ACM Transactions on Database Systems, 15:208--229, 1990.

Digital Library

Cited By

Ertl O(2024)UltraLogLog: A Practical and More Space-Efficient Alternative to HyperLogLog for Approximate Distinct CountingProceedings of the VLDB Endowment10.14778/3654621.365463217:7(1655-1668)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.14778/3654621.3654632
Wang PLiu YLi ZLi R(2024)An LDP Compatible Sketch for Securely Approximating Set Intersection CardinalitiesProceedings of the ACM on Management of Data10.1145/36392812:1(1-27)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3639281
Sultan SShakiba KLee AChen PStumm M(2024)TTLs Matter: Efficient Cache Sizing with TTL-Aware Miss Ratio Curves and Working Set SizesProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3650066(387-404)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3627703.3650066
Show More Cited By

Index Terms

HyperLogLog in practice: algorithmic engineering of a state of the art cardinality estimation algorithm
1. Information systems
  1. Data management systems
    1. Database management system engines
  2. Information systems applications

Recommendations

Sliding HyperLogLog: Estimating Cardinality in a Data Stream over a Sliding Window
ICDMW '10: Proceedings of the 2010 IEEE International Conference on Data Mining Workshops

In this paper, a new algorithm estimating the number of active flows in a data stream is proposed. This algorithm adapts the HyperLogLog algorithm of Flajolet et al. to data stream processing by adding a sliding window mechanism. It has the advantage to ...
Read More
Better Cardinality Estimators for HyperLogLog, PCSA, and Beyond
PODS '23: Proceedings of the 42nd ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems

Cardinality Estimation (aka Distinct Elements) is a classic problem in sketching with many applications in databases, networking, and security. Although sketching algorithms are fairly simple, analyzing the cardinality estimators is notoriously difficult,...
Read More
Branch-and-reduce exponential/FPT algorithms in practice

We investigate the gap between theory and practice for exact branching algorithms. In theory, branch-and-reduce algorithms currently have the best time complexity for numerous important problems. On the other hand, in practice, state-of-the-art methods ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EDBT '13: Proceedings of the 16th International Conference on Extending Database Technology

March 2013

793 pages

ISBN:9781450315975

DOI:10.1145/2452376

General Chair:
Giovanna Guerrini
Università di Genova, Italy
,
Program Chair:
Norman W. Paton
University of Manchester, UK

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

EDBT/ICDT '13

EDBT/ICDT '13: Joint 2013 EDBT/ICDT Conferences

March 18 - 22, 2013

Genoa, Italy

Acceptance Rates

Overall Acceptance Rate 7 of 10 submissions, 70%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

206
Total Citations
View Citations
1,303
Total Downloads

Downloads (Last 12 months)203
Downloads (Last 6 weeks)20

Other Metrics

View Author Metrics

Citations

Cited By

Ertl O(2024)UltraLogLog: A Practical and More Space-Efficient Alternative to HyperLogLog for Approximate Distinct CountingProceedings of the VLDB Endowment10.14778/3654621.365463217:7(1655-1668)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.14778/3654621.3654632
Wang PLiu YLi ZLi R(2024)An LDP Compatible Sketch for Securely Approximating Set Intersection CardinalitiesProceedings of the ACM on Management of Data10.1145/36392812:1(1-27)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3639281
Sultan SShakiba KLee AChen PStumm M(2024)TTLs Matter: Efficient Cache Sizing with TTL-Aware Miss Ratio Curves and Working Set SizesProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3650066(387-404)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3627703.3650066
Wu NYuan XWang SHu HXue MChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Cardinality Counting in "Alcatraz": A Privacy-aware Federated Learning ApproachProceedings of the ACM on Web Conference 202410.1145/3589334.3645655(3076-3084)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645655
Ma COdegbile OMelissourgos DWang HChen S(2024)From CountMin to Super kJoin Sketches for Flow Spread EstimationIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.327966511:3(2353-2370)Online publication date: May-2024
https://doi.org/10.1109/TNSE.2023.3279665
Xiao QYang SLi PLi KWen L(2024)Multi-Resolution Odd Sketch for Mining Extended Jaccard Similarity of Dynamic Streaming setsIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.327580911:3(2399-2414)Online publication date: May-2024
https://doi.org/10.1109/TNSE.2023.3275809
Reviriego PApple JAlonso AErtl ODayan N(2024)Cardinality Estimation Adaptive Cuckoo Filters (CE-ACF): Approximate Membership Check and Distinct Query Count for High-Speed Network MonitoringIEEE/ACM Transactions on Networking10.1109/TNET.2023.330230632:2(959-970)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1109/TNET.2023.3302306
Wang PXie DZhao JLi JLi ZLi RRen YDi J(2024)Half-Xor: A Fully-Dynamic Sketch for Estimating the Number of Distinct Values in Big TablesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.335971036:7(3111-3125)Online publication date: Jul-2024
https://doi.org/10.1109/TKDE.2024.3359710
Lu JChen HSun PHu TZhang ZRen Q(2024)SuperGuardianInformation Systems10.1016/j.is.2024.102351122:COnline publication date: 1-May-2024
https://dl.acm.org/doi/10.1016/j.is.2024.102351
Hehir JTing DCormode GKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Sketch-flip-mergeProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3618930(12846-12865)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3618930
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents