article

Finding frequent items in data streams

Authors:

Moses Charikar,

Martin Farach-ColtonAuthors Info & Claims

Theoretical Computer Science, Volume 312, Issue 1

Pages 3 - 15

https://doi.org/10.1016/S0304-3975(03)00400-6

Published: 26 January 2004 Publication History

Abstract

We present a 1-pass algorithm for estimating the most frequent items in a data stream using limited storage space. Our method relies on a data structure called a COUNT SKETCH, which allows us to reliably estimate the frequencies of frequent items in the stream. Our algorithm achieves better space bounds than the previously known best algorithms for this problem for several natural distributions on the item frequencies. In addition, our algorithm leads directly to a 2-pass algorithm for the problem of estimating the items with the largest (absolute) change in frequency between two data streams. To our knowledge, this latter problem has not been previously studied in the literature.

References

[1]

{1} D. Achlioptas, Database-friendly random projections, in: Proc. 20th ACM Symp. on Principles of Database Systems, 2001, pp. 274-281.

Digital Library

[2]

{2} N. Alon, Y. Matias, M. Szegedy, The space complexity of approximating the frequency moments, J. Comput. System Sci. 58 (1) (1999) 137-147.

Digital Library

[3]

{3} M.E. Crovella, M.S. Taqqu, A. Bestavros, Heavy-tailed probability distributions in the world wide web, in: Adler, Feldman, Taqqu (Eds.), A Practical Guide to Heavy Tails, Birkhäuser, Basel, 1998.

Digital Library

[4]

{4} M. Fang, N. Shivakumar, H. Garcia-Molina, R. Motwani, J. Ullman, Computing iceberg queries efficiently, in: Proc. 22nd Internat. Conf. on Very Large Data Bases, 1996, pp. 307-317.

Digital Library

[5]

{5} J. Feigenbaum, S. Kannan, M. Strauss, M. Viswanathan, An approximate l1-difference algorithm for massive data streams, in: Proc. 40th IEEE Symp. on Foundations of Computer Science, 1999, pp. 501-511.

Digital Library

[6]

{6} J. Feigenbaum, S. Kannan, M. Strauss, M. Viswanathan, Testing and spot-checking of data streams, in: Proc. 11th ACM-SIAM Symp. on Discrete Algorithms, 2000, pp. 165-174.

Digital Library

[7]

{7} P. Gibbons, Y. Matias, New sampling-based summary statistics for improving approximate query answers, in: Proc. ACM SIGMOD Internat. Conf. on Management of Data, 1998, pp. 331-342.

Digital Library

[8]

{8} P. Gibbons, Y. Matias, Synopsis data structures for massive data sets, in: Proc. 10th Ann. ACM-SIAM Symp. on Discrete Algorithms, 1999, pp. 909-910.

Digital Library

[9]

{9} A. Gilbert, S. Guha, P. Indyk, Y. Kotidis, S. Muthukrishnan, M. Strauss, Fast, small-space algorithms for approximate histogram maintenance, in: Proc. 34th ACM Symp. on Theory of Computing, 2002.

Digital Library

[10]

{10} Google, Google zeitgeist--search patterns, trends, and surprises according to Google, http://www.google.com/press/zeitgeist.html.

[11]

{11} S. Guha, N. Mishra, R. Motwani, L. O'Callaghan, Clustering data streams, in: Proc. 41st IEEE Symp. on Foundations of Computer Science, 2000, pp. 359-366.

Digital Library

[12]

{12} M. Henzinger, P. Raghavan, S. Rajagopalan, Computing on data streams, Tech. Report SRC TR 1998-011, December 1998.

[13]

{13} P. Indyk, Stable distributions, pseudorandom generators, embeddings and data stream computation, in: Proc. 41st IEEE Symp. on Foundations of Computer Science, 2000, pp. 148-155.

Digital Library

[14]

{14} R. Karp, S. Shenker, C.H. Papadimitriou, A simple algorithm for finding frequent elements in streams and bags, ACM Transactions on Database Systems 28 (2003) 51-55.

Digital Library

[15]

{15} G.S. Manku, R. Motwani, Approximate frequency counts over data streams, in: Proc. 28th Internat. Conf. on Very Large Data Bases, 2002.

Digital Library

[16]

{16} M. Saks, X. Sun, Space lower bounds for distance approximation in the data stream model, in: Proc. 34th ACM Symp. on Theory of Computing, 2002.

Digital Library

[17]

{17} Y. Xie, D. O'Hallaron, Locality for search engine queries and its implications for caching, in: Proc. INFOCOM, 2002.

Cited By

Li W(2025)Pandora: An Efficient and Rapid Solution for Persistence-Based Tasks in High-Speed Data StreamsProceedings of the ACM on Management of Data10.1145/37097113:1(1-26)Online publication date: 11-Feb-2025
https://dl.acm.org/doi/10.1145/3709711
Verma BDubey PPratap RThakur M(2025)Improving compressed matrix multiplication using control variate methodInformation Processing Letters10.1016/j.ipl.2024.106517187:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.ipl.2024.106517
Munteanu AOmlor SSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Turnstile ℓp leverage score sampling with applicationsProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693565(36797-36828)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693565
Show More Cited By

Index Terms

Finding frequent items in data streams
1. Mathematics of computing
  1. Mathematical analysis
    1. Functional analysis
      1. Approximation
  2. Probability and statistics
    1. Distribution functions
2. Theory of computation
  1. Design and analysis of algorithms
    1. Approximation algorithms analysis
    2. Data structures design and analysis
      1. Sorting and searching

Recommendations

Finding frequent items in data streams using ESBF
PAKDD'07: Proceedings of the 2007 international conference on Emerging technologies in knowledge discovery and data mining

In this paper, we introduce a novel data structure, ESBF (Ex- tensible and Scalable Bloom Filter), and the algorithm FI-ESBF (Finding frequent Items using ESBF) for estimating the frequent items in data streams. FI-ESBF can work with high precision ...
Frequent Items Mining on Data Stream Based on Weighted Counts
CYBERC '11: Proceedings of the 2011 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery

Frequent items mining is an important data mining task with many real-world applications. By considering different weights of the items, weighted frequent items mining can discover more important knowledge compared to traditional frequent patterns ...
A Multi-Domain Architecture for Mining Frequent Items and Itemsets from Distributed Data Streams

Real-time analysis of distributed data streams is a challenging task since it requires scalable solutions to handle streams of data that are generated very rapidly by multiple sources. This paper presents the design and the implementation of an ...

Comments

Information & Contributors

Information

Published In

cover image Theoretical Computer Science

Theoretical Computer Science Volume 312, Issue 1

Special issue on automata, languages and programming

26 January 2004

138 pages

ISSN:0304-3975

Issue’s Table of Contents

Publisher

Elsevier Science Publishers Ltd.

United Kingdom

Publication History

Published: 26 January 2004

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

165
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li W(2025)Pandora: An Efficient and Rapid Solution for Persistence-Based Tasks in High-Speed Data StreamsProceedings of the ACM on Management of Data10.1145/37097113:1(1-26)Online publication date: 11-Feb-2025
https://dl.acm.org/doi/10.1145/3709711
Verma BDubey PPratap RThakur M(2025)Improving compressed matrix multiplication using control variate methodInformation Processing Letters10.1016/j.ipl.2024.106517187:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.ipl.2024.106517
Munteanu AOmlor SSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Turnstile ℓp leverage score sampling with applicationsProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693565(36797-36828)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693565
Høgsgaard MKamma LLarsen KNelson JSchwiegelshohn CSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Sparse dimensionality reduction revisitedProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692812(18454-18469)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3692812
Zhang YChen PLiu ZVanbever LZhang I(2024)OctoSketchProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691914(1621-1639)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691914
Sun HHuang QSun JWang WLi JLi FBao YYao XZhang GVanbever LZhang I(2024)AutoSketchProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691911(1551-1572)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691911
Zheng LXiao QCai X(2024)A Universal Sketch for Estimating Heavy Hitters and Per-Element Frequency Moments in Data Streams with Bounded DeletionsProceedings of the ACM on Management of Data10.1145/36987992:6(1-28)Online publication date: 20-Dec-2024
https://dl.acm.org/doi/10.1145/3698799
Ozery ADiamant JLandau Feibish S(2024)RecenTo: Finding Top-K Flows of the Recent PastProceedings of the ACM on Networking10.1145/36768712:CoNEXT3(1-20)Online publication date: 21-Aug-2024
https://dl.acm.org/doi/10.1145/3676871
Wang LZhou HBao YYan XShen GKong X(2024)Horizontal Federated Recommender System: A SurveyACM Computing Surveys10.1145/365616556:9(1-42)Online publication date: 3-Apr-2024
https://dl.acm.org/doi/10.1145/3656165
Jayaram RWoodruff DZhou S(2024)Streaming Algorithms with Few State ChangesProceedings of the ACM on Management of Data10.1145/36511452:2(1-28)Online publication date: 14-May-2024
https://dl.acm.org/doi/10.1145/3651145
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents