research-article

Learning from the past: automated rule generation for complex event processing

Authors:

Alessandro Margara,

Gianpaolo Cugola,

Giordano TamburrelliAuthors Info & Claims

DEBS '14: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems

Pages 47 - 58

https://doi.org/10.1145/2611286.2611289

Published: 26 May 2014 Publication History

Abstract

Complex Event Processing (CEP) systems aim at processing large flows of events to discover situations of interest. In CEP, the processing takes place according to user-defined rules, which specify the (causal) relations between the observed events and the phenomena to be detected. We claim that the complexity of writing such rules is a limiting factor for the diffusion of CEP. In this paper, we tackle this problem by introducing iCEP, a novel framework that learns, from historical traces, the hidden causality between the received events and the situations to detect, and uses them to automatically generate CEP rules. The paper introduces three main contributions. It provides a precise definition for the problem of automated CEP rules generation. It dicusses a general approach to this research challenge that builds on three fundamental pillars: decomposition into subproblems, modularity of solutions, and ad-hoc learning algorithms. It provides a concrete implementation of this approach, the iCEP framework, and evaluates its precision in a broad range of situations, using both synthetic benchmarks and real traces from a traffic monitoring scenario.

References

[1]

R. Adaikkalavan and S. Chakravarthy. SnoopIB: Interval-based event specification and detection for active databases. Data & Know. Eng., 59(1):139--165, 2006.

Digital Library

[2]

A. Adi and O. Etzion. Amit - the situation manager. The VLDB Journal, 13(2):177--203, 2004.

Digital Library

[3]

J. Agrawal, Y. Diao, D. Gyllstrom, and N. Immerman. Efficient pattern matching over event streams. In SIGMOD, pages 147--160. ACM, 2008.

Digital Library

[4]

M. R. Álvarez, P. Félix, P. Cariñena, and A. Otero. A data mining algorithm for inducing temporal constraint networks. In Computational Intelligence for Knowledge-Based Systems Design, pages 300--309. Springer, 2010.

Digital Library

[5]

D. Anicic, P. Fodor, S. Rudolph, R. Stühmer, N. Stojanovic, and R. Studer. Etalis: Rule-based reasoning in event processing. Reasoning in Event-Based Distributed Systems, pages 99--124, 2011.

[6]

A. Arasu, S. Babu, and J. Widom. The cql continuous query language: semantic foundations and query execution. The VLDB Journal, 15(2):121--142, 2006.

Digital Library

[7]

A. Artikis, O. Etzion, Z. Feldman, and F. Fournier. Event processing under uncertainty. In DEBS, 2012.

Digital Library

[8]

A. Artikis, G. Paliouras, F. Portet, and A. Skarlatidis. Logic-based representation, reasoning and machine learning for event recognition. In ACM DEBS, pages 282--293. ACM, 2010.

Digital Library

[9]

A. Artikis, M. Weidlich, A. Gal, V. Kalogeraki, and D. Gunopulos. Self-adaptive event recognition for intelligent transport management. 2013.

[10]

B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom. Models and issues in data stream systems. In PODS, pages 1--16. ACM, 2002.

Digital Library

[11]

D. J. Berndt and J. Clifford. Advances in knowledge discovery and data mining. chapter Finding patterns in time series: a dynamic programming approach, pages 229--248. American Association for Artificial Intelligence, 1996.

Digital Library

[12]

L. Brenna, A. Demers, J. Gehrke, M. Hong, J. Ossher, B. Panda, M. Riedewald, M. Thatte, and W. White. Cayuga: a high-performance event processing engine. In SIGMOD, pages 1100--1102. ACM, 2007.

Digital Library

[13]

K. Broda, K. Clark, R. M. 0002, and A. Russo. Sage: A logical agent-based environment monitoring and control system. In AmI, pages 112--117, 2009.

Digital Library

[14]

G. Carrault, M.-O. Cordier, R. Quiniou, and F. Wang. Temporal abstraction and inductive logic programming for arrhythmia recognition from electrocardiograms. AI in Medicine, 28(3):231--263, 2003.

Digital Library

[15]

C. Cortes and V. Vapnik. Support-vector networks. Machine Learning, 20(3):273--297, 2013.

Digital Library

[16]

G. Cugola and A. Margara. Tesla: a formally defined event specification language. In DEBS, pages 50--61. ACM, 2010.

Digital Library

[17]

G. Cugola and A. Margara. Complex event processing with t-rex. Journal of Systems and Software, 85(8):1709--1728, 2012.

Digital Library

[18]

G. Cugola and A. Margara. Low latency complex event processing on parallel hardware. Journal of Parallel and Distributed Computing, 72(2):205--218, 2012.

Digital Library

[19]

G. Cugola and A. Margara. Processing flows of information: From data stream to complex event processing. ACM Comput. Surv., 44(3):15:1--15:62, 2012.

Digital Library

[20]

A. J. Demers, J. Gehrke, M. Hong, M. Riedewald, and W. M. White. Towards expressive publish/subscribe systems. In EDBT, pages 627--644, 2006.

Digital Library

[21]

Y. Diao, B. Li, A. Liu, L. Peng, C. Sutton, T. T. L. Tran, and M. Zink. Capturing data uncertainty in high-volume stream processing. In CIDR, 2009.

[22]

Y. Engel and O. Etzion. Towards proactive event-driven computing. In DEBS, pages 125--136. ACM, 2011.

Digital Library

[23]

Esper, http://esper.codehaus.org/, 2013.

[24]

O. Etzion and P. Niblett. Event Processing in Action. Manning Publications Co., 2010.

Digital Library

[25]

F. Fessant, F. Clérot, and C. Dousson. Mining of an alarm log to improve the discovery of frequent patterns. In Advances in Data Mining, pages 144--152. Springer, 2005.

Digital Library

[26]

T. Gärtner, P. A. Flach, A. Kowalczyk, and A. J. Smola. Multi-instance kernels. In ICML, pages 179--186, 2002.

Digital Library

[27]

V. Guralnik and J. Srivastava. Event detection from time series data. In ACM SIGKDD, pages 33--42. ACM, 1999.

Digital Library

[28]

D. Gyllstrom, J. Agrawal, Y. Diao, and N. Immerman. On supporting kleene closure over event streams. In ICDE, pages 1391--1393, 2008.

Digital Library

[29]

S. Kok and P. Domingos. Learning markov logic networks using structural motifs. Fürnkranz, J., Joachims, T.(eds.), 951:551--558, 2010.

[30]

H.-P. Kriegel, K. M. Borgwardt, P. Kröger, A. Pryakhin, M. Schubert, and A. Zimek. Future trends in data mining. Data Mining and Knowledge Discovery, 15(1):87--97, 2007.

Digital Library

[31]

S. Laxman and P. Sastry. A survey of temporal data mining. Sadhana, 31:173--198, 2006.

[32]

G. Li and H.-A. Jacobsen. Composite subscriptions in content-based publish/subscribe systems. In Middleware, pages 249--269. Springer-Verlag New York, Inc., 2005.

Digital Library

[33]

D. C. Luckham. The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems. Addison-Wesley Longman Publishing Co., Inc., 2001.

Digital Library

[34]

M. V. Mahoney and P. K. Chan. Learning rules for anomaly detection of hostile network traffic. In IEEE ICDM, pages 601--604. IEEE, 2003.

Digital Library

[35]

H. Mannila, H. Toivonen, and A. Inkeri Verkamo. Discovery of frequent episodes in event sequences. Data Mining and Know. Disc., 1:259--289, 1997.

Digital Library

[36]

A. Margara, G. Cugola, and G. Tamburrelli. Towards Automated Rule Learning for Complex Event Processing, 2013.

[37]

G. Mühl, L. Fiege, and P. Pietzuch. Distributed Event-Based Systems. Springer-Verlag, 2006.

Digital Library

[38]

C. Mutschler and M. Philippsen. Learning event detection rules with noise hidden markov models. In NASA/ESA Conf. on AHS, pages 159--166. IEEE, 2012.

[39]

Oracle cep. http://www.oracle.com/technologies/soa/complex-event-processing.html, 2013.

[40]

B. Padmanabhan and A. Tuzhilin. Pattern discovery in temporal databases: A temporal logic approach. In KDD, pages 351--354, 1996.

[41]

R. Quiniou, L. Callens, G. Carrault, M.-O. Cordier, E. Fromont, P. Mabo, and F. Portet. Intelligent adaptive monitoring for cardiac surveillance. In Computational Intelligence in Healthcare 4, pages 329--346. Springer, 2010.

[42]

L. Rabiner and B. Juang. An introduction to hidden markov models. ASSP Magazine, IEEE, 3(1):4--16, 1986.

[43]

M. Rajesh Khanna and M. Dhivya. A generic framework for deriving and processing uncertain events in rule-based systems. In ICICES, pages 398--403. IEEE, 2013.

[44]

C. Ré, J. Letchner, M. Balazinksa, and D. Suciu. Event queries on correlated probabilistic streams. In SIGMOD, pages 715--728. ACM, 2008.

Digital Library

[45]

J. Roddick and M. Spiliopoulou. A survey of temporal knowledge discovery paradigms and methods. IEEE TKDE, 14(4):750--767, 2002.

Digital Library

[46]

N. P. Schultz-Møller, M. Migliavacca, and P. Pietzuch. Distributed complex event processing with query rewriting. In DEBS, pages 4:1--4:12. ACM, 2009.

Digital Library

[47]

S. Sen, N. Stojanovic, and L. Stojanovic. An approach for iterative event pattern recommendation. In ACM DEBS, pages 196--205. ACM, 2010.

Digital Library

[48]

C. Sinclair, L. Pierce, and S. Matzner. An application of machine learning to network intrusion detection. In IEEE ACSAC, pages 371--377. IEEE, 1999.

Digital Library

[49]

Streambase, http://www.streambase.com/, 2013.

[50]

Y. Turchin, A. Gal, and S. Wasserkrug. Tuning complex event processing rules using the prediction-correction paradigm. In ACM DEBS, page 10. ACM, 2009.

Digital Library

[51]

F. Wang and P. Liu. Temporal management of rfid data. In VLDB, pages 1128--1139, 2005.

Digital Library

[52]

S. Wasserkrug, A. Gal, and O. Etzion. A model for reasoning with uncertain rules in event composition systems. In UAI, pages 599--606, 2005.

[53]

S. Wasserkrug, A. Gal, O. Etzion, and Y. Turchin. Complex event processing over uncertain data. In DEBS, pages 253--264. ACM, 2008.

Digital Library

[54]

S. Wasserkrug, A. Gal, O. Etzion, and Y. Turchin. Efficient processing of uncertain events in rule-based systems. IEEE Trans. on Knowl. and Data Eng., 24(1):45--58, 2012.

Digital Library

[55]

A. Weigend, F. Chen, S. Figlewski, and S. Waterhouse. Discovering technical traders in the t-bond futures market. In KDD, pages 354--358. Citeseer, 1998.

[56]

D. Zhang, D. Gatica-Perez, S. Bengio, and I. McCowan. Semi-supervised adapted hmms for unusual event detection. In IEEE CVPR, volume 1, pages 611--618. IEEE, 2005.

Digital Library

Cited By

Bruns RDunkel JSeremet S(2023)Learning Ship Activity Patterns in Maritime Data Streams: Enhancing CEP Rule Learning by Temporal and Spatial Relations and Domain-Specific FunctionsIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.328224624:10(11384-11395)Online publication date: Oct-2023
https://doi.org/10.1109/TITS.2023.3282246
Kumar SKumar AChandra RAgarwal SSyafrullah MAdiyarta K(2023)Leveraging regression models for rule based complex event processing2023 10th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI)10.1109/EECSI59885.2023.10295917(528-533)Online publication date: 20-Sep-2023
https://doi.org/10.1109/EECSI59885.2023.10295917
Michelioudakis EArtikis APaliouras G(2023)Online semi-supervised learning of composite event rules by combining structure and mass-based predicate similarityMachine Learning10.1007/s10994-023-06447-1113:3(1445-1481)Online publication date: 15-Dec-2023
https://doi.org/10.1007/s10994-023-06447-1
Show More Cited By

Index Terms

Learning from the past: automated rule generation for complex event processing
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Data management systems
    1. Database management system engines
      1. Triggers and rules

Recommendations

Automatic Learning of Predictive CEP Rules: Bridging the Gap between Data Mining and Complex Event Processing
DEBS '17: Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems

Due to the undeniable advantage of prediction and proactivity, many research areas and industrial applications are accelerating the pace to keep up with data science and predictive analytics. However and due to three well-known facts, the reactive ...
Complex event processing with T-REX

Several application domains involve detecting complex situations and reacting to them. This asks for a Complex Event Processing (CEP) middleware specifically designed to timely process large amounts of event notifications as they flow from the ...
Introducing uncertainty in complex event processing: model, implementation, and validation

Several application domains involve detecting complex situations and reacting to them. This asks for a Complex Event Processing (CEP) engine specifically designed to timely process low level event notifications to identify higher level composite events ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DEBS '14: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems

May 2014

371 pages

ISBN:9781450327374

DOI:10.1145/2611286

General Chairs:
Umesh Bellur
IIT Bombay, India
,
Ravi Kothari
IBM India Research Labs

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Seventh Framework Programme
Dutch national program COMMIT

Conference

DEBS '14

Sponsor:

DEBS '14: The 8th ACM International Conference on Distributed Event-Based Systems

May 26 - 29, 2014

Mumbai, India

Acceptance Rates

DEBS '14 Paper Acceptance Rate 16 of 174 submissions, 9%;

Overall Acceptance Rate 145 of 583 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
718
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)3

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bruns RDunkel JSeremet S(2023)Learning Ship Activity Patterns in Maritime Data Streams: Enhancing CEP Rule Learning by Temporal and Spatial Relations and Domain-Specific FunctionsIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.328224624:10(11384-11395)Online publication date: Oct-2023
https://doi.org/10.1109/TITS.2023.3282246
Kumar SKumar AChandra RAgarwal SSyafrullah MAdiyarta K(2023)Leveraging regression models for rule based complex event processing2023 10th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI)10.1109/EECSI59885.2023.10295917(528-533)Online publication date: 20-Sep-2023
https://doi.org/10.1109/EECSI59885.2023.10295917
Michelioudakis EArtikis APaliouras G(2023)Online semi-supervised learning of composite event rules by combining structure and mass-based predicate similarityMachine Learning10.1007/s10994-023-06447-1113:3(1445-1481)Online publication date: 15-Dec-2023
https://doi.org/10.1007/s10994-023-06447-1
Liu YYu WGao CChen M(2022)An Auto-Extraction Framework for CEP Rules Based on the Two-Layer LSTM Attention Mechanism: A Case Study on City Air Pollution ForecastingEnergies10.3390/en1516589215:16(5892)Online publication date: 14-Aug-2022
https://doi.org/10.3390/en15165892
Alaghbari KSaad MHussain AAlam M(2022)Complex event processing for physical and cyber security in datacentres - recent progress, challenges and recommendationsJournal of Cloud Computing: Advances, Systems and Applications10.1186/s13677-022-00338-x11:1Online publication date: 14-Oct-2022
https://dl.acm.org/doi/10.1186/s13677-022-00338-x
Bruns RDunkel J(2022)Bat4CEP: a bat algorithm for mining of complex event processing rulesApplied Intelligence10.1007/s10489-022-03256-252:13(15143-15163)Online publication date: 11-Mar-2022
https://doi.org/10.1007/s10489-022-03256-2
Geisler S(2022)Complex Event Processing (CEP)Encyclopedia of Big Data10.1007/978-3-319-32010-6_276(192-198)Online publication date: 12-Feb-2022
https://doi.org/10.1007/978-3-319-32010-6_276
Ramírez AMoreno NVallecillo A(2021)Rule‐based preprocessing for data stream mining using complex event processingExpert Systems10.1111/exsy.1276238:8Online publication date: 20-Jul-2021
https://doi.org/10.1111/exsy.12762
Chakrabarti ASukumar RJarke MRudack MBuske PHolly C(2021)Efficient Modeling of Digital Shadows for Production Processes: A Case Study for Quality Prediction in High Pressure Die Casting Processes2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA)10.1109/DSAA53316.2021.9564113(1-9)Online publication date: 6-Oct-2021
https://doi.org/10.1109/DSAA53316.2021.9564113
Simsek MYildirim Okay FOzdemir S(2021)A deep learning-based CEP rule extraction framework for IoT dataThe Journal of Supercomputing10.1007/s11227-020-03603-5Online publication date: 22-Jan-2021
https://doi.org/10.1007/s11227-020-03603-5
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents