Article

Lambda+, the Renewal of the Lambda Architecture: Category Theory to the Rescue

Authors:

Annabelle Gillet,

Éric Leclercq,

Nadine CullotAuthors Info & Claims

Advanced Information Systems Engineering: 33rd International Conference, CAiSE 2021, Melbourne, VIC, Australia, June 28 – July 2, 2021, Proceedings

Pages 381 - 396

https://doi.org/10.1007/978-3-030-79382-1_23

Published: 28 June 2021 Publication History

Abstract

Designing software architectures for Big Data is a complex task that has to take into consideration multiple parameters, such as the expected functionalities, the properties that are untradeable, or the suitable technologies. Patterns are abstractions that guide the design of architectures to reach the requirements. One of the famous patterns is the Lambda Architecture, which proposes real-time computations with correctness and fault-tolerance guarantees. But the Lambda has also been highly criticized, mostly because of its complexity and because the real-time and correctness properties are each effective in a different layer but not in the overall architecture. Furthermore, its use cases are limited, whereas Big Data need an adaptive and flexible environment to fully reveal the value of data. Nevertheless, it proposes some interesting mechanisms. We present a renewal of the Lambda Architecture: the Lambda+ Architecture, supporting both exploratory and real-time analyzes on data. We propose to study the conservation of properties in composition of components in an architecture using the category theory. We relate a real implementation of our approach to architecture a social network observatory platform.

References

[1]

Abowd GD, Allen R, and Garlan D Formalizing style to understand descriptions of software architecture ACM Trans. Softw. Eng. Methodol. (TOSEM) 1995 4 4 319-364

[2]

Ahmad S, Lavin A, Purdy S, and Agha Z Unsupervised real-time anomaly detection for streaming data Neurocomputing 2017 262 134-147

[3]

Akidau T et al. MillWheel: fault-tolerant stream processing at internet scale VLDB Endow. 2013 6 11 1033-1044

[4]

Akidau T, Chernyak S, and Lax R Streaming Systems: The What, Where, When, and how of Large-scale Data Processing 2018 Newton O’Reilly Media Inc.

[5]

Alotaibi, R., Bursztyn, D., Deutsch, A., Manolescu, I., Zampetakis, S.: Towards scalable hybrid stores: constraint-based rewriting to the rescue. In: Proceedings of the 2019 International Conference on Management of Data, pp. 1660–1677 (2019)

[6]

Broy M Can practitioners neglect theory and theoreticians neglect practice? Computer 2011 44 10 19-24

[7]

Broy M and Cengarle MV UML formal semantics: lessons learned Softw. Syst. Model. 2011 10 4 441-446

[8]

Carbone P, Katsifodimos A, Ewen S, Markl V, Haridi S, and Tzoumas K Apache flink: stream and batch processing in a single engine Bull. IEEE Comput. Soc. Tech. Comm. Data Eng. 2015 36 4 28-38

[9]

Clark, T., Barn, B.S.: Event driven architecture modelling and simulation. In: International Symposium on Service Oriented System, pp. 43–54. IEEE (2011)

[10]

Clements, P.C.: A survey of architecture description languages. In: International Workshop on Software Specification and Design, pp. 16–25. IEEE (1996)

[11]

Craig ID Blackboard systems Artif. Intell. Rev. 1988 2 2 103-118

[12]

Deacon, J.: Model-view-controller (MVC) architecture (2009)

[13]

Eilenberg S and MacLane S General theory of natural equivalences Trans. Am. Math. Soc. 1945 58 2 231-294

[14]

Fernandez, R.C., et al.: Liquid: unifying nearline and offline big data integration. In: Conference on Innovative Data System Research (CIDR 2015) (2015)

[15]

Foote B and Yoder J Big ball of mud Pattern Lang. Program Des. 1997 4 654-692

[16]

Gadepally, V., et al.: The BigDAWG polystore system and architecture. In: High Performance Extreme Computing Conference, pp. 1–6. IEEE (2016)

[17]

Morrison R, Balasubramaniam D, Oquendo F, Warboys B, and Greenwood RM Oquendo F An active architecture approach to dynamic systems co-evolution Software Architecture 2007 Heidelberg Springer 2-10

[18]

Im, J.F., et al.: Pinot: realtime OLAP for 530 million users. In: ACM SIGMOD, pp. 583–594 (2018)

[19]

Inmon WH Building the Data Warehouse 2005 New York Wiley

[20]

Johnson P, Ekstedt M, and Jacobson I Where’s the theory for software engineering? IEEE Softw. 2012 29 5 96

[21]

Kreps, J.: Questioning the Lambda Architecture. O’Reilly RADAR, online article, July 2014. https://www.oreilly.com/ideas/questioning-the-lambda-architecture

[22]

Kulkarni, S., et al.: Twitter heron: stream processing at scale. In: ACM SIGMOD, pp. 239–250 (2015)

[23]

Lampson, B.W.: Hints for computer system design. In: Proceedings of the Ninth ACM Symposium on Operating Systems Principles, pp. 33–48 (1983)

[24]

Le Métayer D Describing software architecture styles using graph grammars IEEE Trans. Softw. Eng. 1998 24 7 521-533

[25]

Lee, C.H., Lin, C.Y.: Implementation of lambda architecture: a restaurant recommender system over apache mesos. In: International Conference on Advanced Information Networking and Applications (AINA), pp. 979–985. IEEE (2017)

[26]

Mabrok MA and Ryan MJ Category theory as a formal mathematical foundation for model-based systems engineering Appl. Math. Inf. Sci. 2017 11 43-51

[27]

Malkis, A., Marmsoler, D.: A model of service-oriented architectures. In: Brazilian Symposium on Components, Architectures and Reuse Software, pp. 110–119. IEEE (2015)

[28]

Marmsoler, D., Malkis, A., Eckhardt, J.: A model of layered architectures, vol. 178, pp. 47–61. arXiv preprint arXiv:1503.04916 (2015)

[29]

Marz, N.: How to beat the cap theorem (2011). http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html

[30]

Marz, N., Warren, J.: Big Data: Principles and best practices of scalable real-time data systems. Manning (2015)

[31]

Munshi AA and Mohamed YARI Data lake lambda architecture for smart grids big data analytics IEEE Access 2018 6 40463-40471

[32]

Namiot D and Sneps-Sneppe M On micro-services architecture Int. J. Open Inf. Technol. 2014 2 9 24-27

[33]

Richards M and Ford N Fundamentals of Software Architecture 2020 Newton O’Reilly

[34]

Sawadogo P and Darmont J On data lake architectures and metadata management J. Intell. Inf. Syst. 2020 56 1 97-120

[35]

Spivak DI Category Theory for the Sciences 2014 Cambridge MIT Press

[36]

Toshniwal, A., et al.: Storm@ twitter. In: ACM SIGMOD, pp. 147–156 (2014)

[37]

Vassiliadis P and Simitsis A Kozielski S and Wrembel R Near real time ETL New Trends in Data Warehousing and Data Analysis 2009 Boston Springer

[38]

Yang, F., Merlino, G., Ray, N., Léauté, X., Gupta, H., Tschetter, E.: The RADStack: Open source lambda architecture for interactive analytics. In: Proceedings of the 50th Hawaii International Conference on System Sciences (2017)

[39]

Zaharia, M., Das, T., Li, H., Shenker, S., Stoica, I.: Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters. In: USENIX Hot Topics in Cloud Computing (2012)

Cited By

Wrembel R(2023)Data Integration Revitalized: From Data Warehouse Through Data Lake to Data MeshDatabase and Expert Systems Applications10.1007/978-3-031-39847-6_1(3-18)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-39847-6_1
Guyot AGillet ALeclercq ECullot N(2022)A Formal Framework for Data Lakes Based on Category TheoryProceedings of the 26th International Database Engineered Applications Symposium10.1145/3548785.3548797(75-83)Online publication date: 22-Aug-2022
https://dl.acm.org/doi/10.1145/3548785.3548797
Abughazala MMuccini HSharaf M(2022)DAT: Data Architecture Modeling Tool for Data-Driven ApplicationsSoftware Architecture. ECSA 2022 Tracks and Workshops10.1007/978-3-031-36889-9_8(90-101)Online publication date: 19-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-36889-9_8

Index Terms

Lambda+, the Renewal of the Lambda Architecture: Category Theory to the Rescue
1. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language features
        Patterns
  2. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles
        Client-server architectures
      2. Software architectures

Index terms have been assigned to the content through auto-classification.

Recommendations

Quality-Driven Architectural Patterns for Self-Aware Cloud-Based Software
CLOUD '15: Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing

Architecture-based self-adaptation has been recognised as one of the prominent ways to design autonomic systems, where self-manageable architectures tend to achieve the required level of dynamicity and compliance with the continual changing in QoS ...
A real-time recommendation engine using lambda architecture

In a data science theory, the recommended methodology is one of the most popular theories and has been deployed in many real industries. However, one of the most challenging problems these days is how to recommend items with massively streaming data. ...
A Formal Framework for Data Lakes Based on Category Theory
IDEAS '22: Proceedings of the 26th International Database Engineered Applications Symposium

The management of Big Data requires flexible systems to handle the heterogeneity of data models as well as the complexity of analytical workflows. Traditional systems like data warehouses have reached their limits due to their rigid schema-on-write ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

Advanced Information Systems Engineering: 33rd International Conference, CAiSE 2021, Melbourne, VIC, Australia, June 28 – July 2, 2021, Proceedings

Jun 2021

576 pages

ISBN:978-3-030-79381-4

DOI:10.1007/978-3-030-79382-1

Editors:
Marcello La Rosa
The University of Melbourne, Melbourne, VIC, Australia
,
Shazia Sadiq
The University of Queensland, St Lucia, QLD, Australia
,
Ernest Teniente
Universitat Politècnica de Catalunya, Barcelona, Spain

© Springer Nature Switzerland AG 2021.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 28 June 2021

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wrembel R(2023)Data Integration Revitalized: From Data Warehouse Through Data Lake to Data MeshDatabase and Expert Systems Applications10.1007/978-3-031-39847-6_1(3-18)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-39847-6_1
Guyot AGillet ALeclercq ECullot N(2022)A Formal Framework for Data Lakes Based on Category TheoryProceedings of the 26th International Database Engineered Applications Symposium10.1145/3548785.3548797(75-83)Online publication date: 22-Aug-2022
https://dl.acm.org/doi/10.1145/3548785.3548797
Abughazala MMuccini HSharaf M(2022)DAT: Data Architecture Modeling Tool for Data-Driven ApplicationsSoftware Architecture. ECSA 2022 Tracks and Workshops10.1007/978-3-031-36889-9_8(90-101)Online publication date: 19-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-36889-9_8

View Options

View options

Media

Figures

Other

Tables

View Table of Contents