Article

Adaptive filters for continuous queries over distributed data streams

Authors:

Jennifer WidomAuthors Info & Claims

SIGMOD '03: Proceedings of the 2003 ACM SIGMOD international conference on Management of data

Pages 563 - 574

https://doi.org/10.1145/872757.872825

Published: 09 June 2003 Publication History

Abstract

We consider an environment where distributed data sources continuously stream updates to a centralized processor that monitors continuous queries over the distributed data. Significant communication overhead is incurred in the presence of rapid update streams, and we propose a new technique for reducing the overhead. Users register continuous queries with precision requirements at the central stream processor, which installs filters at remote data sources. The filters adapt to changing conditions to minimize stream rates while guaranteeing that all continuous queries still receive the updates necessary to provide answers of adequate precision at all times. Our approach enables applications to trade precision for communication overhead at a fine granularity by individually adjusting the precision constraints of continuous queries over streams in a multi-query workload. Through experiments performed on synthetic data simulations and a real network monitoring implementation, we demonstrate the effectiveness of our approach in achieving low communication overhead compared with alternate approaches.

References

[1]

R. Alonso, D. Barbara, H. Garcia-Molina, and S. Abad. Quasi-copies: Efficient data sharing for information retrieval systems. In Proc. EDBT, 1988.

Digital Library

[2]

S. Babu and J. Widom. Continuous queries over data streams. ACM SIGMOD Record, 30(3):109--120, Sept. 2001.

Digital Library

[3]

D. Barbara and H. Garcia-Molina. The Demarcation Protocol: A technique for maintaining linear arithmetic constraints in distributed database systems. In Proc. EDBT, 1992.

Digital Library

[4]

P. A. Bernstein, B. T. Blaustein, and E. M. Clarke. Fast maintenance of semantic integrity assertions using redundant aggregate data. In Proc. VLDB, 1980.

[5]

D. Carney, U. Cetintemel, M. Cherniack, C. Convey, S. Lee, G. Seidman, M. Stonebraker, N. Tatbul, and S. Zdonik. Monitoring streams - a new class of data management applications. In Proc. VLDB, 2002.

Digital Library

[6]

J. Chen, D. J. DeWitt, F. Tian, and Y. Wang. NiagaraCQ: A scalable continuous query system for internet databases. In Proc. SIGMOD, 2000.

Digital Library

[7]

M. Dilman and D. Raz. Efficient reactive monitoring. In Proc. InfoCom, 2001.

[8]

D. Estrin, L. Girod, G. Pottie, and M. Srivastava. Instrumenting the world with wireless sensor networks. In Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2001.

[9]

A. Gupta and J. Widom. Local verification of global integrity constraints in distributed databases. In Proc. SIGMOD, 1993.

Digital Library

[10]

A. Householder, A. Manion, L. Pesante, and G. Weaver. Managing the threat of denial-of-service attacks. Technical report, CMU Software Engineering Institute CERT Coordination Center, Oct. 2001. http://www.cert.org/archive/pdf/ Managing DoS.pdf.

[11]

N. Huyn. Maintaining global integrity constraints in distributed databases. Constraints, 2(3/4):377--399, 1997.

Digital Library

[12]

J. M. Kahn, R. H. Katz, and K. S. J. Pister. Next century challenges: Mobile networking for "smart dust". In Proc. MobiCom, 1999.

Digital Library

[13]

L. A. Lamport. Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21(7), 1978.

Digital Library

[14]

C. T. Lawrence, J. L. Zhou, and A. L. Tits. User's guide for CFSQP version 2.5. Technical report TR-94-16r1, Institute for Systems Research, University of Maryland, 1997.

[15]

J. W. S. Liu, K. Lin, W. Shih, and A. C. Yu. Algorithms for scheduling imprecise computations. IEEE Computer, 24(5), 1991.

Digital Library

[16]

L. Liu, C. Pu, and W. Tang. Continual queries for internet-scale event-driven information delivery. IEEE Knowledge and Data Engineering, 11(4):610--628, 1999.

Digital Library

[17]

S. Madden and M. J. Franklin. Fjording the stream: An architecture for queries over streaming sensor data. In Proc. ICDE, 2002.

Digital Library

[18]

S. Madden, M. Shah, J. M. Hellerstein, and V. Raman. Continuously adaptive continuous queries over streams. In Proc. SIGMOD, 2002.

Digital Library

[19]

D. L. Mills. Internet time synchronization: the network time protocol. IEEE Transactions on Communications, 39(10), 1991.

[20]

R. Min, M. Bhardwaj, S. Cho, A. Sinha, E. Shih, A. Wang, and A. Chandrakasan. Low-power wireless sensor networks. In Proc. Fourteenth International Conference on VLSI Design, 2001.

Digital Library

[21]

S. B. Moon, J. Kurose, and D. Towsley. Packet audio playout delay adjustment: Performance bounds and algorithms. ACM/Springer Multimedia Systems, 6:17--28, Jan. 1998.

Digital Library

[22]

R. Motwani, J. Widom, A. Arasu, B. Babcock, S. Babu, M. Datar, G. Manku, C. Olston, J. Rosenstein, and R. Varma. Query processing, resource management, and approximation in a data stream management system. In Proc. First Biennial Conference on Innovative Data Systems Research (CIDR), 2003.

[23]

C. Olston, J. Jiang, and J. Widom. Adaptive filters for continuous queries over distributed data streams. Technical report, Stanford University Computer Science Department, 2002. http://dbpubs.stanford.edu/pub/2002-55.

[24]

C. Olston, B. T. Loo, and J. Widom. Adaptive precision setting for cached approximate values. In Proc. SIGMOD, 2001.

Digital Library

[25]

C. Olston and J. Widom. Offering a precision-performance tradeoff for aggregation queries over replicated data. In Proc. VLDB, 2000.

Digital Library

[26]

C. Olston and J. Widom. Best-effort cache synchronization with source cooperation. In Proc. SIGMOD, 2002.

Digital Library

[27]

V. Paxson and S. Floyd. Wide-area traffic: The failure of Poisson modeling. IEEE/ACM Transactions on Networking, 3(3):226--244, 1995.

Digital Library

[28]

G. Pottie and W. Kaiser. Wireless integrated network sensors. Communications of the ACM, 43(5):551--558, May 2000.

Digital Library

[29]

S. Shah, A. Bernard, V. Sharma, K. Ramamritham, and P. Shenoy. Maintaining temporal coherency of cooperating dynamic data repositories. In Proc. VLDB, 2002.

[30]

T. Skalicky. Laspack reference manual, 1996. http://www.tudresden.de/mwism/skalicky/laspack/laspack.html.

[31]

N. Soparkar and A. Silberschatz. Data-value partitioning and virtual messages. In Proc. PODS, 1990.

Digital Library

[32]

R. van Renesse and K. Birman. Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining. Technical report, Cornell University, 2001.

[33]

S. Vutukury and J. Garcia-Luna-Aceves. A traffic engineering approach based on minimum-delay routing. In Proc. IEEE International Conference on Computer Communications and Networks, 2000.

[34]

A. N. Wilschut and P. M. G. Apers. Dataflow query execution in a parallel main-memory environment. In Proc. PDIS, 1991.

Digital Library

[35]

T. Yamashita. Dynamic replica control based on fairly assigned variation of data with weak consistency for loosely coupled distributed systems. In Proc. ICDCS, 2002.

Digital Library

[36]

H. Yu and A. Vahdat. Efficient numerical error bounding for replicated network services. In Proc. VLDB, 2000.

Digital Library

Cited By

Kitsios XLiakos PPapakonstantinopoulou KKotidis Y(2024)Flexible grouping of linear segments for highly accurate lossy compression of time series dataThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-024-00862-z33:5(1569-1589)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s00778-024-00862-z
Kitsios XLiakos PPapakonstantinopoulou KKotidis Y(2023)Sim-Piece: Highly Accurate Piecewise Linear Approximation through Similar Segment MergingProceedings of the VLDB Endowment10.14778/3594512.359452116:8(1910-1922)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.14778/3594512.3594521
Slo ABhowmik SRothermel K(2023)gSPICE: Model-Based Event Shedding in Complex Event Processing2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386775(263-270)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386775
Show More Cited By

Index Terms

Adaptive filters for continuous queries over distributed data streams
1. Information systems
  1. Data management systems
    1. Database management system engines
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Database theory
      1. Database query processing and optimization (theory)

Recommendations

Sketching distributed sliding-window data streams

While traditional data management systems focus on evaluating single, ad hoc queries over static data sets in a centralized setting, several emerging applications require (possibly, continuous) answers to queries on dynamic data that is widely ...
Characterizing memory requirements for queries over continuous data streams

This article deals with continuous conjunctive queries with arithmetic comparisons and optional aggregation over multiple data streams. An algorithm is presented for determining whether or not any given query can be evaluated using a bounded amount of ...
Continuous queries over data streams

In many recent applications, data may take the form of continuous data streams, rather than finite stored data sets. Several aspects of data management need to be reconsidered in the presence of data streams, offering a new research direction for the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '03: Proceedings of the 2003 ACM SIGMOD international conference on Management of data

June 2003

702 pages

ISBN:158113634X

DOI:10.1145/872757

Conference Chair:
Zachary Ives
University of Pennsylvania
,
General Chair:
Yannis Papakonstantinou
University of California, San Diego
,
Program Chair:
Alon Halevy
University of Washington

Copyright © 2003 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 June 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SIGMOD/PODS03

Sponsor:

SIGMOD

SIGMOD/PODS03: International Conference on Management of Data and Symposium on Principles Database and Systems

June 9 - 12, 2003

California, San Diego

Acceptance Rates

SIGMOD '03 Paper Acceptance Rate 53 of 342 submissions, 15%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

366
Total Citations
View Citations
1,948
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)2

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kitsios XLiakos PPapakonstantinopoulou KKotidis Y(2024)Flexible grouping of linear segments for highly accurate lossy compression of time series dataThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-024-00862-z33:5(1569-1589)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s00778-024-00862-z
Kitsios XLiakos PPapakonstantinopoulou KKotidis Y(2023)Sim-Piece: Highly Accurate Piecewise Linear Approximation through Similar Segment MergingProceedings of the VLDB Endowment10.14778/3594512.359452116:8(1910-1922)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.14778/3594512.3594521
Slo ABhowmik SRothermel K(2023)gSPICE: Model-Based Event Shedding in Complex Event Processing2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386775(263-270)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386775
Taleb RHallé SKhoury R(2023)Uncertainty in runtime verificationComputer Science Review10.1016/j.cosrev.2023.10059450:COnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.cosrev.2023.100594
Chapnik KKolchinsky ISchuster A(2022)DARLINGProceedings of the VLDB Endowment10.14778/3494124.349413715:3(541-554)Online publication date: 4-Feb-2022
https://dl.acm.org/doi/10.14778/3494124.3494137
Slo ABhowmik SRothermel K(2022)State-Aware Load Shedding From Input Event Streams in Complex Event ProcessingIEEE Transactions on Big Data10.1109/TBDATA.2020.30474388:5(1340-1357)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TBDATA.2020.3047438
Dash RMawa SJenamani M(2022)Outlier Sensitive Online Change Detection using Convex Combination of Adaptive Filters in Sensor Data Streams2022 IEEE 7th International conference for Convergence in Technology (I2CT)10.1109/I2CT54291.2022.9824413(1-7)Online publication date: 7-Apr-2022
https://doi.org/10.1109/I2CT54291.2022.9824413
Friedman RShahout R(2022)Box queries over multi-dimensional streamsInformation Systems10.1016/j.is.2022.102086109:COnline publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1016/j.is.2022.102086
Golab L(2022)Types of Stream Processing AlgorithmsEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_193-3(1-7)Online publication date: 26-Nov-2022
https://doi.org/10.1007/978-3-319-63962-8_193-3
Shi BZhao ZPeng YLi FPhillips JLi GLi ZIdreos SSrivastava D(2021)At-the-time and Back-in-time Persistent SketchesProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3452802(1623-1636)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3452802
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents