research-article

Adaptive Speculative Processing of Out-of-Order Event Streams

Authors:

Christopher Mutschler,

Michael PhilippsenAuthors Info & Claims

ACM Transactions on Internet Technology (TOIT), Volume 14, Issue 1

Article No.: 4, Pages 1 - 24

https://doi.org/10.1145/2633686

Published: 07 August 2014 Publication History

Abstract

Distributed event-based systems are used to detect meaningful events with low latency in high data-rate event streams that occur in surveillance, sports, finances, etc. However, both known approaches to dealing with the predominant out-of-order event arrival at the distributed detectors have their shortcomings: buffering approaches introduce latencies for event ordering, and stream revision approaches may result in system overloads due to unbounded retraction cascades.

This article presents an adaptive speculative processing technique for out-of-order event streams that enhances typical buffering approaches. In contrast to other stream revision approaches developed so far, our novel technique encapsulates the event detector, uses the buffering technique to delay events but also speculatively processes a portion of it, and adapts the degree of speculation at runtime to fit the available system resources so that detection latency becomes minimal.

Our technique outperforms known approaches on both synthetical data and real sensor data from a realtime locating system (RTLS) with several thousands of out-of-order sensor events per second. Speculative buffering exploits system resources and reduces latency by 40% on average.

References

[1]

Jagrati Agrawal, Yanlei Diao, Daniel Gyllstrom, and Neil Immerman. 2008. Efficient pattern matching over event streams. In Proceedings of the International Conference on Management of Data. 147--160.

Digital Library

[2]

Marcos Kawazoe Aguilera and Robert E. Strom. 2000. Efficient atomic broadcast using deterministic merge. In Proceedings of the 19th Symposium on Principles Distributed Computing. 209--218.

Digital Library

[3]

Mark Allman, Chris Hayes, and Shawn Ostermann. 1998. An evaluation of TCP with larger initial windows. ACM SIGCOMM Comput. Commun. Rev. 28, 3 (1998), 41--52.

Digital Library

[4]

Rajagopal Ananthanarayanan, Venkatesh Basker, Sumit Das, Ashish Gupta, Haifeng Jiang, Tianhao Qiu, Alexey Reznichenko, Deomid Ryabkov, Manpreet Singh, and Shivakumar Venkataraman. 2013. Photon: Fault-tolerant and scalable joining of continuous data streams. In Proceedings of the International Conference on Management of Data. 577--588.

Digital Library

[5]

Darko Anicic, Sebastian Rudolph, Paul Fodor, and Nenad Stojanovic. 2011. Retractable complex event processing and stream reasoning. In Proceedings of the 5th International Conference on Rule-Based Reasoning, Programming, and Applications. 122--137.

Digital Library

[6]

Arvind Arasu, Mitch Cherniack, Eduardo Galvez, David Maier, Anurag S. Maskey, Esther Ryvkina, Michael Stonebraker, and Richard Tibbetts. 2004. Linear road: A stream data management benchmark. In Proceedings of the 13th International Conference on Very Large Data Bases. Toronto, Canada, 480--491.

Digital Library

[7]

Magdalena Balazinska, YongChul Kwon, Nathan Kuchta, and Dennis Lee. 2007. Moirae: History-enhanced monitoring. In Proceedings of the 3rd Biennial Conference on Innovative Data Systems Research. Pacific Grove, CA, 375--386.

[8]

Roger S. Barga, Jonathan Goldstein, Mohamed Ali, and Mingsheng Hong. 2007. Consistent streaming through time: A vision for event stream processing. In Proceedings of the 3rd Biennial Conference on Innovative Data Systems Research. 363--374.

[9]

Lars Brenna, Alan Demers, Johannes Gehrke, Mingsheng Hong, Joel Ossher, Biswanath Panda, Mirek Riedewald, Mohit Thatte, and Walker White. 2007. Cayuga: A high-performance event processing engine. In Proceedings of the International Conference on Management of Data. 1100--1102.

Digital Library

[10]

Lars Brenna, Johannes Gehrke, Mingsheng Hong, and Dag Johansen. 2009. Distributed event stream processing with non-deterministic finite automata. In Proceedings of the 3rd International Conference on Distributed Event-Based Systems. Article 3, 12 pages.

Digital Library

[11]

Andrey Brito, Christof Fetzer, Heiko Sturzrehm, and Pascal Felber. 2008. Speculative out-of-order event processing with software transaction memory. In Proceedings of the 2nd International Conference on Distributed Event-Based Systems. 265--275.

Digital Library

[12]

Badrish Chandramouli, Jonathan Goldstein, and David Maier. 2010. High-performance dynamic pattern matching over disordered streams. In Proceedings of the International Conference on Very Large Databases. 220--231.

Digital Library

[13]

Alan Demers, Johannes Gehrke, Mingsheng Hong, Mirek Riedewald, and Walker White. 2006. Towards expressive publish/subscribe systems. In Proceedings of the 10th International Conference on Extending Database Technology. 627--644.

Digital Library

[14]

Alan Demers, Johannes Gehrke, Biswanath Panda, Mirek Riedewald, Varun Sharma, and Walker White. 2007. Cayuga: A general purpose event monitoring system. In Proceedings of the 3rd Biennial Conference on Innovative Data Systems Research. 412--422.

[15]

Mootaz Elnozahy, Lorenzo Alvisi, Yi-Min Wang, and David B. Johnson. 2002. A survey of rollback-recovery protocols in message-passing systems. ACM Comput. Surv. 34, 3 (2002), 375--408.

Digital Library

[16]

Raul Castro Fernandez, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch. 2013. Integrating scale out and fault tolerance in stream processing using operator state management. In Proceedings of the International Conference on Management of Data. 725--736.

Digital Library

[17]

Jeong-Hyon Hwang, Ugur Çetintemel, and Stanley B. Zdonik. 2008. Fast and highly-available stream processing over wide area networks. In Proceedings of the 24th International Conference on Data Engineering. 804--813.

Digital Library

[18]

Zbigniew Jerzak and Christof Fetzer. 2008. BFSiena: A communication substrate for StreamMine. In Proceedings of the 2nd International Conference on Distributed Event-Based Systems. 321--324.

Digital Library

[19]

Zbigniew Jerzak and Holger Ziekow. 2014. The DEBS 2014 grand challenge. In Proceedings of the 8th International Conference on Distributed Event-Based Systems.

Digital Library

[20]

Boris Koldehofe, Ruben Mayer, Umakishore Ramachandran, Kurt Rothermel, and Marco Völz. 2013. Rollback-recovery without checkpoints in distributed event processing systems. In Proceedings of the 7th International Conference on Distributed Event-Based Systems. 27--38.

Digital Library

[21]

YongChul Kwon, Magdalena Balazinska, and Albert Greenberg. 2008. Fault-tolerant stream processing using a distributed, replicated file system. Proc. VLDB Endow. 1, 1 (2008), 574--585.

Digital Library

[22]

Chuan-Wen Li, Yu Gu, Ge Yu, and Bonghee Hong. 2011. Aggressive complex event processing with confidence over out-of-order streams. J. Comput. Sci. Technol. 26, 4 (2011), 685--696.

[23]

Jin Li, Kristin Tufte, Vladislav Shkapenyuk, Vassilis Papadimos, Theodore Johnson, and David Maier. 2008. Out-of-order processing: A new architecture for high-performance stream systems. Proce. VLDB Endow., 1 (2008), 274--288.

Digital Library

[24]

Ming Li, Mo Liu, Luping Ding, Elke Rundensteiner, and Murali Mani. 2007. Event stream processing with out-of-order data arrival. In Proceedings of the 27th International Conference on Distributed Computing Systems Workshops. 67--74.

Digital Library

[25]

Mo Liu, Ming Li, D. Golovnya, E. A. Rundensteiner, and K. Claypool. 2009. Sequence pattern query processing over out-of-order event streams. In Proceedings of the 25th International Conference on Data Engineering 784--795.

Digital Library

[26]

Anurag S. Maskey and Mitch Cherniack. 2008. Replay-based approaches to revision processing in stream query engines. In Proceedings of the 2nd International Workshop on Scalable Stream Processing Systems. 3--12.

Digital Library

[27]

Gero Mühl, Ludger Fiege, and Peter Pietzuch. 2006. Distributed Event-Based Systems. Springer, Berlin.

[28]

Christopher Mutschler. 2014. Latency Minimization of Order-Preserving Distributed Event-Based Systems. Ph.D. Dissertation. Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany.

[29]

Christopher Mutschler, Zbigniew Jerzak, and Holger Ziekow. 2013. The DEBS 2013 grand challenge. In Proceedings of the 7th International Conference on Distributed Event-Based Systems. 289--294.

Digital Library

[30]

Christopher Mutschler and Michael Philippsen. 2013a. Distributed low-latency out-of-order event processing for high data rate sensor streams. In Proceedings of the 27th International Parallel and Distributed Processing Symposium. 1133--1144.

Digital Library

[31]

Christopher Mutschler and Michael Philippsen. 2013b. Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares. In Proceedings of the 7th International Conference on Distributed Event-Based Systems. 147--158.

Digital Library

[32]

Dan O'Keeffe and Jean Bacon. 2010. Reliable complex event detection for pervasive computing. In Proceedings of the 4th International Conference on Distributed Event-Based Systems. 73--84.

Digital Library

[33]

Larry L. Peterson, Nick C. Buchholz, and Richard D. Schlichting. 1989. Preserving and using context information in interprocess communication. ACM Trans. Comput. Syst. 7, 3 (1989), 217--246.

Digital Library

[34]

Esther Ryvkina, Anurag Maskey, Mitch Cherniack, and Stan Zdonik. 2006. Revision processing in a stream processing engine: A high-level design. In Proceedings of the 22nd International Conference on Data Engineering. 141--143.

Digital Library

[35]

Zoe Sebepou and Kostas Magoutis. 2011. CEC: Continuous eventual checkpointing for data stream processing operators. In Proceedings of the 41st International Conference on Dependable Systems Networks. 145--156.

Digital Library

[36]

Utkarsh Srivastava and Jennifer Widom. 2004. Flexible time management in data stream systems. In Proceedings of the 23rd Symposium on Principles Database Systems. 263--274.

Digital Library

[37]

Michael Stonebraker, Uǧur Çetintemel, and Stan Zdonik. 2005. The 8 requirements of real-time stream processing. SIGMOD Record 34, 4 (2005), 42--47.

Digital Library

[38]

Peter A. Tucker, David Maier, Tim Sheard, and Leonidas Fegaras. 2003. Exploiting punctuation semantics in continuous data streams. Trans. Know. Data Eng. 15, 3 (2003), 555--568.

Digital Library

[39]

Thomas von der Grün, Norbert Franke, Daniel Wolf, Nicolas Witt, and Andreas Eidloth. 2011. A real-time tracking system for football match and training analysis. In Microelectronic Systems. Springer Berlin, 199--212.

[40]

Benjamin Wester, James Cowling, Edmund B. Nightingale, Peter M. Chen, Jason Flinn, and Barbara Liskov. 2009. Tolerating latency in replicated state machines through client speculation. In Proceedings of the 6th Symposium on Networked Systems Design and Implementation. 245--260.

Digital Library

[41]

Eugene Wu, Yanlei Diao, and Shariq Rizvi. 2006. High-performance complex event processing over streams. In Proceedings of the International Conference on Management of Data. 407--418.

Digital Library

[42]

Kaiwen Zhang, Vinod Muthusamy, and Hans-Arno Jacobsen. 2012. Total order in content-based publish/subscribe systems. In Proceedings of the 32nd International Conference on Distributed Computing Systems. 335--344.

Digital Library

Cited By

Yasser TArafa TEl-Helw MAwad A(2023)Keyed Watermarks: A Fine-grained Tracking of Event-time in Apache Flink2023 5th Novel Intelligent and Leading Emerging Sciences Conference (NILES)10.1109/NILES59815.2023.10296717(23-28)Online publication date: 21-Oct-2023
https://doi.org/10.1109/NILES59815.2023.10296717
Awad AWeidlich MSakr S(2020)Process Mining over Unordered Event Streams2020 2nd International Conference on Process Mining (ICPM)10.1109/ICPM49681.2020.00022(81-88)Online publication date: Oct-2020
https://doi.org/10.1109/ICPM49681.2020.00022
Wen YZhang LWang C(2019)MDDRSPF: A Model Driven Distributed Real-Time Stream Processing Framework2019 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SustainCom-SocialCom48970.2019.00194(1352-1358)Online publication date: Dec-2019
https://doi.org/10.1109/ISPA-BDCloud-SustainCom-SocialCom48970.2019.00194
Show More Cited By

Index Terms

Adaptive Speculative Processing of Out-of-Order Event Streams
1. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles
        Organizing principles for web applications

Recommendations

Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares
DEBS '13: Proceedings of the 7th ACM international conference on Distributed event-based systems

In surveillance, sports, finances, etc., distributed event-based systems are used to detect meaningful events with low latency in high data rate event streams. Both known approaches to deal with the predominant out-of-order event arrival at the ...
Distributed Low-Latency Out-of-Order Event Processing for High Data Rate Sensor Streams
IPDPS '13: Proceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed Processing

Event-based Systems (EBS) are used to detect and analyze meaningful events in surveillance, sports, finances and many other areas. With rising data and event rates and with correlations among these events, sequential event processing becomes infeasible ...
Demo: do event-based systems have a passion for sports?
DEBS '13: Proceedings of the 7th ACM international conference on Distributed event-based systems

The ubiquity of sensor data calls for automatic processing to extract valuable information. Realtime Locating Systems (RTLS) provide many parallel position data streams for interacting objects, and event-based systems are the method of choice to analyze ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Internet Technology

ACM Transactions on Internet Technology Volume 14, Issue 1

Special Issue on Event Recognition

July 2014

161 pages

ISSN:1533-5399

EISSN:1557-6051

DOI:10.1145/2659232

Editor:
Munindar P. Singh
Department of Computer Science, North Carolina State University

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2014

Accepted: 01 April 2014

Revised: 01 March 2014

Received: 01 October 2013

Published in TOIT Volume 14, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Fraunhofer IIS

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
417
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)1

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yasser TArafa TEl-Helw MAwad A(2023)Keyed Watermarks: A Fine-grained Tracking of Event-time in Apache Flink2023 5th Novel Intelligent and Leading Emerging Sciences Conference (NILES)10.1109/NILES59815.2023.10296717(23-28)Online publication date: 21-Oct-2023
https://doi.org/10.1109/NILES59815.2023.10296717
Awad AWeidlich MSakr S(2020)Process Mining over Unordered Event Streams2020 2nd International Conference on Process Mining (ICPM)10.1109/ICPM49681.2020.00022(81-88)Online publication date: Oct-2020
https://doi.org/10.1109/ICPM49681.2020.00022
Wen YZhang LWang C(2019)MDDRSPF: A Model Driven Distributed Real-Time Stream Processing Framework2019 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SustainCom-SocialCom48970.2019.00194(1352-1358)Online publication date: Dec-2019
https://doi.org/10.1109/ISPA-BDCloud-SustainCom-SocialCom48970.2019.00194
Giatrakos NAlevizos EArtikis ADeligiannakis AGarofalakis M(2019)Complex event recognition in the Big Data era: a surveyThe VLDB Journal10.1007/s00778-019-00557-wOnline publication date: 25-Jul-2019
https://doi.org/10.1007/s00778-019-00557-w
Dayarathna MPerera S(2018)Recent Advancements in Event ProcessingACM Computing Surveys10.1145/317043251:2(1-36)Online publication date: 13-Feb-2018
https://dl.acm.org/doi/10.1145/3170432
Gorse LLoffler CMutschler CPhilippsen M(2018)Optical Camera Communication for Active Marker Identification in Camera-based Positioning Systems2018 15th Workshop on Positioning, Navigation and Communications (WPNC)10.1109/WPNC.2018.8555846(1-6)Online publication date: Oct-2018
https://doi.org/10.1109/WPNC.2018.8555846
Mayer RSlo ATariq MRothermel KGräber MRamachandran UJayaram KGandhi AKemme BPietzuch P(2017)SPECTREProceedings of the 18th ACM/IFIP/USENIX Middleware Conference10.1145/3135974.3135983(161-173)Online publication date: 11-Dec-2017
https://dl.acm.org/doi/10.1145/3135974.3135983
Kokovin VSytin A(2017)The processing of information from sensors in intelligent systemsJournal of Physics: Conference Series10.1088/1742-6596/803/1/012075803(012075)Online publication date: 17-Mar-2017
https://doi.org/10.1088/1742-6596/803/1/012075

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents