Article

Resource sharing in continuous sliding-window aggregates

Authors:

Jennifer WidomAuthors Info & Claims

VLDB '04: Proceedings of the Thirtieth international conference on Very large data bases - Volume 30

Pages 336 - 347

Published: 31 August 2004 Publication History

Abstract

We consider the problem of resource sharing when processing large numbers of continuous queries. We specifically address sliding-window aggregates over data streams, an important class of continuous operators for which sharing has not been addressed. We present a suite of sharing techniques that cover a wide range of possible scenarios: different classes of aggregation functions (algebraic, distributive, holistic), different window types (time-based, tuple-based, suffix, historical), and different input models (single stream, multiple substreams). We provide precise theoretical performance guarantees for our techniques, and show their practical effectiveness through experimental study.

References

[1]

{1} M. K. Aguilera, R. E. Strom, et al. Matching events in a content-based subscription system. In Proc. of the 18th Annual ACM Symp. on Principles of Distributed Computing, pages 53-61, May 1999.

Digital Library

[2]

{2} M. Altinel and M. J. Franklin. Efficient filtering of XML documents for selective dissemination of information. In Proc. of the 26th Intl. Conf. on Very Large Data Bases, pages 53-64, Sept. 2000.

Digital Library

[3]

{3} A. Arasu, S. Babu, and J. Widom. The CQL Continuous Query Language: Semantic Foundations and Query Execution. Technical report, Stanford University, Oct. 2003. http://dbpubs. stanford.edu/pub/2003-67.

[4]

{4} A. Arasu and G. Manku. Approximate counts and quantiles over sliding windows. In Proc. of the 23rd ACM SIGACT-SIGMOD-SIGART Symp. on Principles of Database Systems, June 2004.

Digital Library

[5]

{5} A. Arasu and J. Widom. Resource sharing in continuous sliding window aggregates. Technical Report http://dbpubs. stanford.edu/pub/2004-15, Stanford University, 2004.

[6]

{6} R. Avnur and J. M. Hellerstein. Eddies: Continuously adaptive query processing. In Proc. of the 2000 ACM SIGMOD Intl. Conf. on Management of Data, pages 261-272, May 2000.

Digital Library

[7]

{7} B. Babcock, S. Babu, et al. Models and issues in data stream systems. In Proc. of the 21st ACM SIGACT-SIGMOD-SIGART Symp. on Principles of Database Systems, pages 1-16, June 2002.

Digital Library

[8]

{8} D. Carney, U. Centintemel, et al. Monitoring streams - a new class of data management applications. In Proc. of the 28th Intl. Conf. on Very Large Data Bases, pages 215-226, Aug. 2002.

Digital Library

[9]

{9} S. Chandrasekharan, O. Cooper, et al. TelegraphCQ: Continuous dataflow processing for an uncertain world. In Proc. of the 1st Conf. on Innovative Data Systems Research, pages 269-280, Jan. 2003.

[10]

{10} S. Chandrasekharan and M. J. Franklin. Streaming queries over streaming data. In Proc. of the 28th Intl. Conf. on Very Large Data Bases, pages 203-214, Aug. 2002.

Digital Library

[11]

{11} J. Chen, D. J. DeWitt, F. Tian, and Y. Wang. NiagaraCQ: A scalable continuous query system for internet databases. In Proc. of the 2000 ACM SIGMOD Intl. Conf. on Management of Data, pages 379-390, May 2000.

Digital Library

[12]

{12} M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. In Proc. of the 13th Annual ACM-SIAM Symp. on Discrete Algorithms, pages 635-644, Jan. 2002.

Digital Library

[13]

{13} Y. Diao, P. M. Fischer, M. J. Franklin, and R. To. YFilter: Efficient and scalable filtering of XML documents. In Proc. of the 18th Intl. Conf. on Data Engineering, pages 341-344, Feb. 2002.

Digital Library

[14]

{14} A. Dobra, M. Garofalakis, J. Gehrke, and R. Rastogi. Sketch-based multi-query processing over data streams. In Proc. of the 9th Intl. Conf. on Extending Database Technology, Mar. 2004.

[15]

{15} C. Estan and G. Varghese. New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice. ACM Transactions on Computer Systems, 21(3):270-313, Aug. 2003.

Digital Library

[16]

{16} F. Fabret, H. Jacobsen, et al. Filtering algorithms and implementation for very fast publish/subscribe. In Proc. of the 2000 ACM SIGMOD Intl. Conf. on Management of Data, pages 115-126, May 2001.

Digital Library

[17]

{17} J. Gehrke, F. Korn, and D. Srivastava. On computing correlated aggregates over continual data streams. In Proc. of the 2001 ACM SIGMOD Intl. Conf. on Management of Data, pages 13-24, May 2001.

Digital Library

[18]

{18} P. B. Gibbons and S. Tirthapura. Distributed streams algorithms for sliding windows. In Proc. of the 14th Annual ACM Symp. on Parallel Algs. and Architectures, pages 63-72, Aug. 2002.

Digital Library

[19]

{19} A. C. Gilbert, Y. Kotidis, et al. How to summarize the universe: Dynamic maintenance of quantiles. In Proc. of the 28th Intl. Conf. on Very Large Data Bases, pages 454-465, Aug. 2002.

Digital Library

[20]

{20} J. Gray, S. Chaudhuri, et al. Data cube: A relational aggregation operator generalizing group-by, cross-tab, and sub totals. Data Mining and Knowledge Discovery, 1(1):29-53, Mar. 1997.

Digital Library

[21]

{21} R. E. Gruber, B. Krishnamurthy, and E. Panagos. READY: A high performance event notification system. In Proc. of the 16th Intl. Conf. on Data Engineering, pages 668-669, Mar. 2000.

Digital Library

[22]

{22} A. K. Gupta and D. Suciu. Stream processing of XPath queries with predicates. In Proc. of the 2003 ACM SIGMOD Intl. Conf. on Management of Data, pages 419-430, June 2003.

Digital Library

[23]

{23} libavl: Library for balanced binary trees. Available at http:// www.gnu.org/directory/GNU/libavl.html.

[24]

{24} U. Lindqvist and P. A. Porras. Detecting computer and network misuse through the production-based expert system toolset (P-BEST). In Proc. of the IEEE Symp. on Security and Privacy, pages 146-161, May 1999.

[25]

{25} S. Madden, M. A. Shah, J. M. Hellerstein, and V. Raman. Continuously adaptive continuous queries over streams. In Proc. of the 2002 ACM SIGMOD Intl. Conf. on Management of Data, pages 49-60, June 2002.

Digital Library

[26]

{26} F. Peng and S. S. Chawathe. XPath queries on streaming data. In Proc. of the 2003 ACM SIGMOD Intl. Conf. on Management of Data, pages 431-442, June 2003.

Digital Library

[27]

{27} P. Roy, S. Seshadri, et al. Efficient and extensible algorithms for multi query optimization. In Proc. of the 2000 ACM SIGMOD Intl. Conf. on Management of Data, pages 249-260, May 2000.

Digital Library

[28]

{28} T. K. Sellis. Multiple-query optimization. ACM Trans. on Database Systems, 13(1):23-52, Mar. 1988.

Digital Library

[29]

{29} Traderbot home page. http://www.traderbot.com, 2003.

[30]

{30} G. Vigna and R. A. Kemmerer. NetSTAT: A network-based intrusion detection approach. In Proc. of the 14th Annual Computer Security Appln. Conf., pages 25-38, Dec. 1998.

Digital Library

[31]

{31} Y. Zhu and D. Shasha. StatStream: Statistical monitoring of thousands of data streams in real time. In Proc. of the 28th Intl. Conf. on Very Large Data Bases, pages 358-369, Aug. 2002.

Digital Library

Cited By

Ma LLei CPoppe ORundensteiner EIves ZBonifati AEl Abbadi A(2022)Gloria: Graph-based Sharing Optimizer for Event Trend AggregationProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3526145(1122-1135)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3526145
Poppe OLei CMa LRozet ARundensteiner ELi GLi ZIdreos SSrivastava D(2021)To Share, or not to Share Online Event Trend Aggregation Over Bursty Event StreamsProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3452785(1452-1464)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3452785
Carbone PFragkoulis MKalavri VKatsifodimos AMaier DPottinger RDoan ATan WAlawini ANgo H(2020)Beyond Analytics: The Evolution of Stream Processing SystemsProceedings of the 2020 ACM SIGMOD International Conference on Management of Data10.1145/3318464.3383131(2651-2658)Online publication date: 11-Jun-2020
https://dl.acm.org/doi/10.1145/3318464.3383131
Show More Cited By

Index Terms

Recommendations

Sketching distributed sliding-window data streams

While traditional data management systems focus on evaluating single, ad hoc queries over static data sets in a centralized setting, several emerging applications require (possibly, continuous) answers to queries on dynamic data that is widely ...
Semantics and evaluation techniques for window aggregates in data streams
SIGMOD '05: Proceedings of the 2005 ACM SIGMOD international conference on Management of data

A windowed query operator breaks a data stream into possibly overlapping subsets of data and computes a result over each. Many stream systems can evaluate window aggregate queries. However, current stream systems suffer from a lack of an explicit ...
Sketching asynchronous streams over a sliding window
PODC '06: Proceedings of the twenty-fifth annual ACM symposium on Principles of distributed computing

We study the problem of maintaining sketches of recent elements of a data stream. Motivated by applications involving network data, we consider streams that are asynchronous, in which the observed order of data is not the same as the time order in which ...

Comments

Information & Contributors

Information

Published In

cover image DL Hosted proceedings

VLDB '04: Proceedings of the Thirtieth international conference on Very large data bases - Volume 30

August 2004

1380 pages

ISBN:0120884690

Sponsors

VLDB Endowment: Very Large Database Endowment

Publisher

VLDB Endowment

Publication History

Published: 31 August 2004

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

59
Total Citations
View Citations
341
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)3

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ma LLei CPoppe ORundensteiner EIves ZBonifati AEl Abbadi A(2022)Gloria: Graph-based Sharing Optimizer for Event Trend AggregationProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3526145(1122-1135)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3526145
Poppe OLei CMa LRozet ARundensteiner ELi GLi ZIdreos SSrivastava D(2021)To Share, or not to Share Online Event Trend Aggregation Over Bursty Event StreamsProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3452785(1452-1464)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3452785
Carbone PFragkoulis MKalavri VKatsifodimos AMaier DPottinger RDoan ATan WAlawini ANgo H(2020)Beyond Analytics: The Evolution of Stream Processing SystemsProceedings of the 2020 ACM SIGMOD International Conference on Management of Data10.1145/3318464.3383131(2651-2658)Online publication date: 11-Jun-2020
https://dl.acm.org/doi/10.1145/3318464.3383131
Tangwongsan KHirzel MSchneider S(2019)Optimal and general out-of-order sliding-window aggregationProceedings of the VLDB Endowment10.14778/3339490.333949912:10(1167-1180)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.14778/3339490.3339499
Zeuch SMonte BKarimov JLutz CRenz MTraub JBreß SRabl TMarkl V(2019)Analyzing efficient stream processing on modern hardwareProceedings of the VLDB Endowment10.14778/3303753.330375812:5(516-530)Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.14778/3303753.3303758
Kroll LSegeljakt KCarbone PSchulte CHaridi SCheung ANguyễn K(2019)Arc: an IR for batch and stream programmingProceedings of the 17th ACM SIGPLAN International Symposium on Database Programming Languages10.1145/3315507.3330199(53-58)Online publication date: 23-Jun-2019
https://dl.acm.org/doi/10.1145/3315507.3330199
Mamouras KStanford CAlur RIves ZTannen VMcKinley KFisher K(2019)Data-trace types for distributed stream processing systemsProceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation10.1145/3314221.3314580(670-685)Online publication date: 8-Jun-2019
https://dl.acm.org/doi/10.1145/3314221.3314580
Kolchinsky ISchuster ABoncz PManegold SAilamaki ADeshpande AKraska T(2019)Real-Time Multi-Pattern Detection over Event StreamsProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3319869(589-606)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3319869
Poppe OLei CRundensteiner EMaier DBoncz PManegold SAilamaki ADeshpande AKraska T(2019)Event Trend Aggregation Under Rich Event Matching SemanticsProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3319862(555-572)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3319862
Shaikh SWatanabe YWang YKitagawa H(2019)Smart schemeKnowledge and Information Systems10.1007/s10115-018-1195-958:2(341-370)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1007/s10115-018-1195-9
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents