research-article

Theory of data stream computing: where to go

Author:

S. MuthukrishnanAuthors Info & Claims

PODS '11: Proceedings of the thirtieth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

Pages 317 - 319

https://doi.org/10.1145/1989284.1989314

Published: 13 June 2011 Publication History

Abstract

Computing power has been growing steadily, just as communication rate and memory size. Simultaneously our ability to create data has been growing phenomenally and therefore the need to analyze it. We now have examples of massive data streams that are created in far higher rate than we can capture and store in memory economically, gathered in far more quantity than can be transported to central databases without overwhelming the communication infrastructure, and arrives far faster than we can compute with them in a sophisticated way.

This phenomenon has challenged how we store, communicate and compute with data. Theories developed over past 50 years have relied on full capture, storage and communication of data. Instead, what we need for managing modern massive data streams are new methods built around working with less. The past 10 years have seen new theories emerge in computing (data stream algorithms), communication (compressed sensing), databases (data stream management systems) and other areas to address the challenges of massive data streams. Still, lot remains open and new applications of massive data streams have emerged recently. We present an overview of these challenges.

Supplementary Material

WMV File (1989314.wmv)

Download
247.57 MB

References

[1]

S. Muthukrishnan. Data Streams: Algorithms and Applications. In Foundations and Trends in Theoretical Computer Science, 2005.

Digital Library

[2]

P. Indyk A tutorial on Streaming, Sketching and Sub-linear Space Algorithms. 2009 Information Theory and Applications Workshop, San Diego, 2009. http://people.csail.mit.edu/indyk/ita-web.pdf

[3]

M. Garofalakis, J. Gehrke and R. Rastogi. Data Stream Management: Processing High-Speed Data Streams, 2007.

Digital Library

[4]

C. Cranor, T. Johnson and O. Spatscheck. Gigascope: How to monitor network traffic at 5Gbit/sec at a time. http://www2.research.att.com/~divesh/meetings/mpds2003/schedule/spatscheck.pdf.

[5]

David Donoho. Compressed sensing. Technical Report, 2004.

[6]

E. Candes and T. Tao. Near-optimal signal recovery from random projections and universalencoding strategies. 2004.

[7]

http://dsp.rice.edu/cs.

[8]

J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. Proc. OSDI, 2004.

Digital Library

[9]

http://en.wikipedia.org/wiki/XLDB.

[10]

Jon Feldman, S. Muthukrishnan, Anastasios Sidiropoulos, Clifford Stein, Zoya Svitkina. On distributing symmetric streaming computations. em Proc. SODA 2008: 710--719.

Digital Library

[11]

H. Karloff, S. Suri, S. and S. Vassilvitskii.A Model of Computation for MapReduce. Proc. ACM-SIAM SODA 2010.

Digital Library

[12]

Graham Cormode, S. Muthukrishnan, Ke Yi. Algorithms for distributed functional monitoring. Proc. SODA 2008: 1076--1085

Digital Library

[13]

Eyal Kushilevitz and Noam Nisan. Communication Complexity, 1997.

Digital Library

[14]

http://www.scholarpedia.org/article/Slepian-Wolf_coding

[15]

Kenneth L. Clarkson, David P. Woodruff. Numerical linear algebra in the streaming model. Proc STOC. 2009: 205--214.

Digital Library

[16]

Kook Jin Ahn, Sudipto Guha. Graph Sparsification in the Semi-streaming Model. ICALP (2) 2009: 328--338.

Digital Library

[17]

A. Chakrabarti, G. Cormode, and A. McGregor. Annotations in data streams. In International Colloquium on Automata, Languages and Programming (ICALP), 2009.

Digital Library

[18]

C. Dwork, M. Naor, T. Pitassi, G. Rothblum, and S. Yekhanin. Pan-Private Streaming Algorithms. ICS, 2010.

[19]

D. Mir, S. Muthukrishnan, A. Nikolov and R. Wright.Pan-Private Algorithms Via Statistics on Sketches. PODS, 2011.

Digital Library

[20]

Cynthia Dwork. Differential Privacy in New Settings. SODA, 2010.

Digital Library

Cited By

Kupferman OPetruschka A(2023)Lazy regular sensingTheoretical Computer Science10.1016/j.tcs.2023.114057971:COnline publication date: 6-Sep-2023
https://dl.acm.org/doi/10.1016/j.tcs.2023.114057
Kupferman OPetruschka A(2022)Lazy Regular SensingDescriptional Complexity of Formal Systems10.1007/978-3-031-13257-5_12(155-169)Online publication date: 29-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-13257-5_12
Le-Tuan AHayes CHauswirth MLe-Phuoc D(2020)Pushing the Scalability of RDF Engines on IoT Edge DevicesSensors10.3390/s2010278820:10(2788)Online publication date: 14-May-2020
https://doi.org/10.3390/s20102788
Show More Cited By

Index Terms

Theory of data stream computing: where to go
1. Theory of computation
  1. Design and analysis of algorithms

Recommendations

Practical Storage-Compute Elasticity for Stream Data Processing
Middleware '23: Proceedings of the 24th International Middleware Conference: Industrial Track

Stream processing pipelines need to handle workload fluctuations (e.g., daily patterns, popularity spikes) by scaling up/down the resources contributed to running jobs. While there have been efforts proposing auto-scaling mechanisms for stream processing ...
IoT Big Data Stream Mining
KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

The challenge of deriving insights from the Internet of Things (IoT) has been recognized as one of the most exciting and key opportunities for both academia and industry. Advanced analysis of big data streams from sensors and devices is bound to become ...
Data stream clustering: a review
Abstract
Number of connected devices is steadily increasing and these devices continuously generate data streams. Real-time processing of data streams is arousing interest despite many challenges. Clustering is one of the most suitable methods for real-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PODS '11: Proceedings of the thirtieth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

June 2011

332 pages

ISBN:9781450306607

DOI:10.1145/1989284

General Chair:
Maurizio Lenzerini
Sapienza University of Rome, Italy
,
Program Chair:
Thomas Schwentick
TU Dortmund University, Germany

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tag

data streams

Qualifiers

Research-article

Conference

SIGMOD/PODS '11

Sponsor:

SIGMOD

SIGMOD/PODS '11: International Conference on Management of Data

June 13 - 15, 2011

Athens, Greece

Acceptance Rates

PODS '11 Paper Acceptance Rate 25 of 113 submissions, 22%;

Overall Acceptance Rate 642 of 2,707 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
857
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kupferman OPetruschka A(2023)Lazy regular sensingTheoretical Computer Science10.1016/j.tcs.2023.114057971:COnline publication date: 6-Sep-2023
https://dl.acm.org/doi/10.1016/j.tcs.2023.114057
Kupferman OPetruschka A(2022)Lazy Regular SensingDescriptional Complexity of Formal Systems10.1007/978-3-031-13257-5_12(155-169)Online publication date: 29-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-13257-5_12
Le-Tuan AHayes CHauswirth MLe-Phuoc D(2020)Pushing the Scalability of RDF Engines on IoT Edge DevicesSensors10.3390/s2010278820:10(2788)Online publication date: 14-May-2020
https://doi.org/10.3390/s20102788
Almagor SKuperberg DKupferman O(2017)Sensing as a Complexity MeasureDescriptional Complexity of Formal Systems10.1007/978-3-319-60252-3_1(3-15)Online publication date: 3-Jun-2017
https://doi.org/10.1007/978-3-319-60252-3_1
Buono DMatteis TMencagli G(2014)A High-Throughput and Low-Latency Parallelization of Window-Based Stream Joins on MulticoresProceedings of the 2014 IEEE International Symposium on Parallel and Distributed Processing with Applications10.1109/ISPA.2014.24(117-126)Online publication date: 26-Aug-2014
https://dl.acm.org/doi/10.1109/ISPA.2014.24
Borodin A(2013)Computing (and Life) Is All about TradeoffsSpace-Efficient Data Structures, Streams, and Algorithms10.1007/978-3-642-40273-9_9(112-132)Online publication date: 2013
https://doi.org/10.1007/978-3-642-40273-9_9

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten