research-article

Criticality-aware Monitoring and Orchestration for Containerized Industry 4.0 Environments

Authors:

Marco Barletta,

Marcello Cinque,

Luigi De Simone,

Raffaele Della CorteAuthors Info & Claims

ACM Transactions on Embedded Computing Systems, Volume 23, Issue 1

Article No.: 5, Pages 1 - 28

https://doi.org/10.1145/3604567

Published: 10 January 2024 Publication History

Abstract

The evolution of industrial environments makes the reconfigurability and flexibility key requirements to rapidly adapt to changeable market needs. Computing paradigms like Edge/Fog computing are able to provide the required flexibility and scalability while guaranteeing low latencies and response times. Orchestration systems play a key role in these environments, enforcing automatic management of resources and workloads’ lifecycle, and drastically reducing the need for manual interventions. However, they do not currently meet industrial non-functional requirements, such as real-timeliness, determinism, reliability, and support for mixed-criticality workloads.

In this article, we present k4.0s, an orchestration system for Industry 4.0 (I4.0) environments, which enables the support for real-time and mixed-criticality workloads. We highlight through experiments the need for novel monitoring approaches and propose a workflow for selecting monitoring metrics, which depends on both workload requirements and hosting node guarantees. We introduce new abstractions for the components of a cluster in order to enable criticality-aware monitoring and orchestration of real-time industrial workloads. Finally, we design an orchestration system architecture that reflects the proposed model, introducing new components and prototyping a Kubernetes-based implementation, taking the first steps towards a fully I4.0-enabled orchestration system.

References

[1]

Heiner Lasi, Peter Fettke, Hans-Georg Kemper, Thomas Feld, and Michael Hoffmann. 2014. Industry 4.0. Springer Business & Information Systems Engineering (BISE) 6 (2014), 239–242.

[2]

A.-W. Colombo, S. Karnouskos, and J.-M. Mendes. 2010. Factory of the future: A service-oriented system of modular, dynamic reconfigurable and collaborative systems. Springer Artificial Intelligence Techniques for Networked Manufacturing Enterprises Management (2010), 459–481.

[3]

Bjarne Johansson, Mats Rågberger, Thomas Nolte, and Alessandro V. Papadopoulos. 2022. Kubernetes orchestration of high availability distributed control systems. In IEEE International Conference on Industrial Technology (ICIT’22). IEEE, 1–8.

[4]

Jacob Mellado and Felipe Núñez. 2022. Design of an IoT-PLC: A containerized programmable logical controller for the industry 4.0. Elsevier Journal of Industrial Information Integration 25 (2022), 100250.

[5]

Emiliano Sisinni, Abusayeed Saifullah, Song Han, Ulf Jennehag, and Mikael Gidlund. 2018. Industrial Internet of Things: Challenges, opportunities, and directions. IEEE Transactions on Industrial Informatics (TII) 14, 11 (2018), 4724–4734.

[6]

Andrea Borghesi, Giuseppe Di Modica, Paolo Bellavista, Varun Gowtham, Alexander Willner, Daniel Nehls, Florian Kintzler, Stephan Cejka, Simone Rossi Tisbeni, Alessandro Costantini, et al. 2021. Iotwins: Design and implementation of a platform for the management of digital twins in industrial scenarios. In Proceedings of 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid’21). IEEE, 625–633.

[7]

Spyridon V. Gogouvitis, Harald Mueller, Sreenath Premnadh, Andreas Seitz, and Bernd Bruegge. 2020. Seamless computing in industrial systems using container orchestration. Elsevier Future Generation Computer Systems 109 (2020), 678–688.

[8]

Ranesh Kumar Naha, Saurabh Garg, Dimitrios Georgakopoulos, Prem Prakash Jayaraman, Longxiang Gao, Yong Xiang, and Rajiv Ranjan. 2018. Fog computing: Survey of trends, architectures, requirements, and research directions. IEEE Access 6 (2018), 47980–48009.

[9]

Alessandro Cilardo, Marcello Cinque, Luigi De Simone, and Nicola Mazzocca. 2022. Virtualization over multiprocessor systems-on-chip: An enabling paradigm for the industrial Internet of Things. IEEE Computer 55, 10 (2022), 35–47.

Digital Library

[10]

Alan Burns and Robert Ian Davis. 2022. Mixed criticality systems-a review. Department of Computer Science, University of York, Tech. Rep (2022).

[11]

Gernot Heiser. 2011. Virtualizing embedded systems: Why bother? In Proceedings of the 48th Design Automation Conference. 901–905.

Digital Library

[12]

BlackBerry Limited. 2021. Are Hypervisors the Answer to the Coming Silicon Shortages? (White Paper). https://blackberry.qnx.com/content/dam/blackberry-com/Documents/pdf/BlackBerry_QNX_Hypervisor_WhitePaper_22April2021_FINAL.pdf.

[13]

Marcello Cinque, Domenico Cotroneo, Luigi De Simone, and Stefano Rosiello. 2022. Virtualizing mixed-criticality systems: A survey on industrial trends and issues. Elsevier Future Generation Computer Systems (FGCS) 129 (2022), 315–330.

Digital Library

[14]

Roberto Morabito, Vittorio Cozzolino, Aaron Yi Ding, Nicklas Beijar, and Jorg Ott. 2018. Consolidate IoT edge computing with lightweight virtualization. IEEE Network 32, 1 (2018), 102–111.

[15]

Václav Struhár, Moris Behnam, Mohammad Ashjaei, and Alessandro V. Papadopoulos. 2020. Real-time containers: A survey. In Proccedings of 2nd Workshop on Fog Computing and the IoT (Fog-IoT’20). Schloss Dagstuhl-Leibniz-Zentrum für Informatik.

[16]

Marcello Cinque, Raffaele Della Corte, Antonio Eliso, and Antonio Pecchia. 2019. Rt-cases: Container-based virtualization for temporally separated mixed-criticality task sets. In Proceedings of 31st Euromicro Conference on Real-time Systems (ECRTS’19). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.

[17]

Allison Randal. 2020. The ideal versus the real: Revisiting the history of virtual machines and containers. ACM Computing Surveys (CSUR) 53, 1 (2020), 1–31.

Digital Library

[18]

Docker Inc.2022. Swarm Mode Overview. Retrieved June 22, 2023, from https://docs.docker.com/engine/swarm/.

[19]

The Linux Foundation. 2022. Kubernetes Home Page. Retrieved June 22, 2023, from https://kubernetes.io/.

[20]

Apache Software Foundation. 2022. Apache Mesos Home Page. Retrieved June 22, 2023, from https://mesos.apache.org/.

[21]

Asif Khan. 2017. Key characteristics of a container orchestration platform to enable a modern application. IEEE Cloud Computing 4, 5 (2017), 42–48.

[22]

Maria A. Rodriguez and Rajkumar Buyya. 2019. Container-based cluster orchestration systems: A taxonomy and future directions. Wiley Software: Practice and Experience 49, 5 (2019), 698–719.

[23]

Carmen Carrión. 2022. Kubernetes scheduling: Taxonomy, ongoing issues and challenges. Comput. Surveys 55, 7 (2022), 1–37.

Digital Library

[24]

Stefano Fiori, Luca Abeni, and Tommaso Cucinotta. 2022. RT-Kubernetes–containerized real-time cloud computing. In Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing. 36–39.

Digital Library

[25]

László Toka. 2021. Ultra-reliable and low-latency computing in the edge with Kubernetes. Journal of Grid Computing 19, 3 (2021), 31.

Digital Library

[26]

Václav Struhár, Silviu S. Craciunas, Mohammad Ashjaei, Moris Behnam, and Alessandro V. Papadopoulos. 2021. REACT: Enabling real-time container orchestration. In Proceedings of 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA’21). IEEE, 1–8.

Digital Library

[27]

Raphael Eidenbenz, Yvonne-Anne Pignolet, and Alain Ryser. 2020. Latency-aware industrial fog application orchestration with Kubernetes. In Proceedings of 5th IEEE International Conference on Fog and Mobile Edge Computing (FMEC’20). IEEE, 164–171.

[28]

Gabriele Monaco, Gautam Gala, and Gerhard Fohler. 2023. Extensions for shared resource orchestration in Kubernetes to support RT-cloud containers. arXiv preprint arXiv:2301.07479 (2023).

[29]

Sebastian Böhm and Guido Wirtz. 2022. Towards orchestration of cloud-edge architectures with Kubernetes. In Springer Science and Technologies for Smart Cities. Springer, 207–230.

[30]

Eddy Truyen, Dimitri Van Landuyt, Davy Preuveneers, Bert Lagaisse, and Wouter Joosen. 2019. A comprehensive feature comparison study of open-source container orchestration frameworks. MDPI Applied Sciences 9, 5 (2019), 931.

[31]

Xilinx. 2022. RunX. Retrieved June 22, 2023, from https://github.com/Xilinx/runx.

[32]

Katacontainers. 2017. Home Page of Katacontainers. Retrieved June 22, 2023, from https://katacontainers.io/.

[33]

Marco Barletta, Marcello Cinque, Luigi De Simone, and Raffaele Della Corte. 2022. Introducing k4. 0s: A model for mixed-criticality container orchestration in industry 4.0. In 2022 IEEE Intl. Conf. on Dependable, Autonomic and Secure Computing, Intl. Conf. on Pervasive Intelligence and Computing, Intl. Conf. on Cloud and Big Data Computing, Intl. Conf. on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech’22). IEEE, 1–6.

[34]

Kubernetes. 2022. Kubernetes Metrics Server. Retrieved June 22, 2023, fromhttps://github.com/kubernetes-sigs/metrics-server.

[35]

Cloud Native Computing Foundation. 2022. Retrieved June 22, 2023, fromPrometheus. https://prometheus.io/.

[36]

Influx Data. 2022. Telegraf. Retrieved June 22, 2023, from https://docs.influxdata.com/telegraf/.

[37]

VictoriaMetrics. 2022. VictoriaMetrics. Retrieved June 22, 2023, from https://github.com/VictoriaMetrics/VictoriaMetrics.

[38]

Luca Abeni, Alessio Balsini, and Tommaso Cucinotta. 2019. Container-based real-time scheduling in the Linux kernel. ACM Special Interest Group on Embedded Systems (SIGBED) Review 16, 3 (2019), 33–38.

[39]

Tommaso Cucinotta, Luca Abeni, Mauro Marinoni, Riccardo Mancini, and Carlo Vitucci. 2021. Strong temporal isolation among containers in OpenStack for NFV services. IEEE Transactions on Cloud Computing (TCC) 11, 1 (2021), 763–778.

[40]

Sisu Xi, Chong Li, Chenyang Lu, Christopher D. Gill, Meng Xu, Linh T. X. Phan, Insup Lee, and Oleg Sokolsky. 2015. Rt-open stack: Cpu resource management for real-time cloud computing. In Proceedings of 8th IEEE International Conference on Cloud Computing. IEEE.

Digital Library

[41]

Mark Szalay, Peter Matray, and Laszlo Toka. 2022. Real-time FaaS: Towards a latency bounded serverless cloud. IEEE Transactions on Cloud Computing (TCC) 11, 2 (2022), 1636–1650.

[42]

Hylson V. Netto, Lau Cheuk Lung, Miguel Correia, Aldelir Fernando Luiz, and Luciana Moreira Sá de Souza. 2017. State machine replication in containers managed by Kubernetes. Elsevier Journal of Systems Architecture (2017), 134–135.

Digital Library

[43]

Marco Barletta, Marcello Cinque, Luigi De Simone, Raffaele Della Corte, Giorgio Farina, and Daniele Ottaviano. 2022. RunPHI: Enabling mixed-criticality containers via partitioning hypervisors in industry 4.0. In 2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW’22). 134–135. DOI:

[44]

Wei-Tek Tsai, Qihong Shao, Xin Sun, and Jay Elston. 2010. Real-time service-oriented cloud computing. In World Congress on Services. IEEE.

[45]

W. T. Tsai, Yann-Hang Lee, Zhibin Cao, Yinong Chen, and Bingnan Xiao. 2006. RTSOA: Real-time service-oriented architecture. In Proceedings of 2nd IEEE International Symposium on Service-oriented System Engineering. IEEE, 49–56.

Digital Library

[46]

Tommaso Cucinotta, Antonio Mancina, Gaetano F. Anastasi, Giuseppe Lipari, Leonardo Mangeruca, Roberto Checcozzo, and Fulvio Rusinà. 2009. A real-time service-oriented architecture for industrial automation. IEEE Transactions on Industrial Informatics 5, 3 (2009), 267–277.

[47]

Hendrik Bohn, Andreas Bobek, and Frank Golatowski. 2006. SIRENA-service infrastructure for real-time embedded networked devices: A service oriented framework for different domains. In Proceedings of International Conference on Networking, International Conference on Systems and International Conference on Mobile Communications and Learning Technologies (ICNICONSMCL’06). IEEE, 43–43.

Digital Library

[48]

Carolyn McGregor and J. Mikael Eklund. 2010. Next generation remote critical care through service-oriented architectures: Challenges and opportunities. Springer Service Oriented Computing and Applications 53 (2010).

[49]

Isam Mashhour Al Jawarneh, Paolo Bellavista, Filippo Bosi, Luca Foschini, Giuseppe Martuscelli, Rebecca Montanari, and Amedeo Palopoli. 2019. Container orchestration engines: A thorough functional and performance comparison. In IEEE International Conference on Communications. 1–6.

[50]

Real-Time Group Scheduling. 2022. Real-Time Group Scheduling. Retrieved June 22, 2023, from https://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt.

[51]

Marco Barletta, Marcello Cinque, Luigi De Simone, and Raffaele Della Corte. 2022. Achieving isolation in mixed-criticality industrial edge systems with real-time containers. In Proceedings of 34th Euromicro Conference on Real-Time Systems (ECRTS’22). Schloss Dagstuhl-Leibniz-Zentrum für Informatik.

[52]

Algirdas Avizienis, J.-C. Laprie, Brian Randell, and Carl Landwehr. 2004. Basic concepts and taxonomy of dependable and secure computing. IEEE Transactions on Dependable and Secure Computing (TDSC) 1, 1 (2004), 11–33.

Digital Library

[53]

Sinem Coleri Ergen and Pravin Varaiya. 2010. TDMA scheduling algorithms for wireless sensor networks. Springer Wireless Networks 16 (2010), 985–997.

Digital Library

[54]

Steve Vestal. 2007. Preemptive scheduling of multi-criticality systems with varying degrees of execution time assurance. In Proceedings of 28th IEEE International Real-time Systems Symposium (RTSS’07). IEEE, 239–243.

Digital Library

[55]

K4.0s. 2022. k4.0s Gitlab Repository. https://dessert.unina.it:8088/marcobarlo/k4.0s.

[56]

Heiko Koziolek and Nafise Eskandani. 2023. Lightweight Kubernetes distributions: A performance comparison of MicroK8s, k3s, k0s, and microshift. In Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering. 17–29.

Digital Library

[57]

Cloud Native Computing Foundation. 2022. Kubernetes Certified Distributions. Retrieved June 22, 2023, from https://www.cncf.io/certification/software- conformance/.

[58]

Jan Kiszka and Bernardo Wagner. 2005. RTnet-a flexible hard real-time networking framework. In Proceedings of IEEE Conference on Emerging Technologies and Factory Automation. IEEE.

[59]

Guowang Miao, Jens Zander, Ki Won Sung, and Slimane Ben Slimane. 2016. Fundamentals of Mobile Data Networks. Cambridge University Press.

[60]

Xiao Zhang, Sandhya Dwarkadas, and Kai Shen. 2009. Towards practical page coloring-based multicore cache management. In Proceedings of ACM European Conference on Computer Systems.

Digital Library

[61]

Intel. Cache Allocation Technology in Intel Xeon Processor. Retrieved June 22, 2023, from https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-cache-allocation-technology.html.

[62]

Intel. Overview of Intel® Time Coordinated Computing (TCC) Tools. Retrieved June 22, 2023, from https://www.intel.com/content/www/us/en/developer/articles/technical/real-time-systems-measurement-library.html.

[63]

Paul Emberson, Roger Stafford, and Robert I. Davis. 2010. Techniques for the synthesis of multiprocessor tasksets. In Proceedings of International Workshop on Analysis Tools and Methodologies for Embedded and Real-time Systems (WATERS’10).

Cited By

Adame TAmri EAntonopoulos GAzaiez SBerne ACamargo JKakoulidis HKleisarchaki SLlamedo APrasinos MPsara KShumaiev K(2024)Presenting the COGNIFOG Framework: Architecture, Building Blocks and Road toward Cognitive ConnectivitySensors10.3390/s2416528324:16(5283)Online publication date: 15-Aug-2024
https://doi.org/10.3390/s24165283
Yi BWang JHe QWang XHuang MDas SLi K(2024)Traffic Prediction-Based VNF Auto-Scaling and Deployment Mechanism for Flexible and Elastic Service ProvisionIEEE Transactions on Services Computing10.1109/TSC.2024.344005017:5(2959-2973)Online publication date: Sep-2024
https://doi.org/10.1109/TSC.2024.3440050
De Simone LDi Mauro MNatella RPostiglione F(2024)Performance and Availability Challenges in Designing Resilient 5G ArchitecturesIEEE Transactions on Network and Service Management10.1109/TNSM.2024.340456021:5(5291-5303)Online publication date: Oct-2024
https://doi.org/10.1109/TNSM.2024.3404560
Show More Cited By

Index Terms

Criticality-aware Monitoring and Orchestration for Containerized Industry 4.0 Environments
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
  2. Real-time systems
2. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Process management
        Monitors
    2. Software system structures
      1. Real-time systems software

Recommendations

A Virtualized Separation Kernel for Mixed-Criticality Systems

Multi- and many-core processors are becoming increasingly popular in embedded systems. Many of these processors now feature hardware virtualization capabilities, as found on the ARM Cortex A15 and x86 architectures with Intel VT-x or AMD-V support. ...
Cyber-physical production systems retrofitting in context of industry 4.0
Highlights
- Requirements for standardizing the retrofitting process for industrial equipment.
Abstract
Industry 4.0 is the new industrial revolution involving the introduction of new technologies in the industrial field. However, changing the technological level of an outdated industry is not a simple task. By retrofitting all old ...
On the Scheduling of Mixed-Criticality Real-Time Task Sets
RTSS '09: Proceedings of the 2009 30th IEEE Real-Time Systems Symposium

The functional consolidation induced by the cost reduction trends in embedded systems can force tasks of different criticality (e.g. ABS Brakes with DVD) to share a processor and interfere with each other. These systems are known as mixed criticality ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Embedded Computing Systems

ACM Transactions on Embedded Computing Systems Volume 23, Issue 1

January 2024

406 pages

EISSN:1558-3465

DOI:10.1145/3613501

Editor:
Tulika Mitra
National University of Singapore, Singapore

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

ACM Journals for the Design of Smart and Connected Systems

Publication History

Published: 10 January 2024

Online AM: 18 June 2023

Accepted: 04 June 2023

Revised: 16 March 2023

Received: 10 November 2022

Published in TECS Volume 23, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

MOST – Sustainable Mobility National Research Center and received funding from the European Union Next-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) – MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.4)
MICS (Made in Italy – Circular and Sustainable) Extended Partnership and received funding from the European Union Next-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) – MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.3)
European Union - FSE-REACT-EU, PON Research and Innovation 2014–2020

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
610
Total Downloads

Downloads (Last 12 months)372
Downloads (Last 6 weeks)20

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Adame TAmri EAntonopoulos GAzaiez SBerne ACamargo JKakoulidis HKleisarchaki SLlamedo APrasinos MPsara KShumaiev K(2024)Presenting the COGNIFOG Framework: Architecture, Building Blocks and Road toward Cognitive ConnectivitySensors10.3390/s2416528324:16(5283)Online publication date: 15-Aug-2024
https://doi.org/10.3390/s24165283
Yi BWang JHe QWang XHuang MDas SLi K(2024)Traffic Prediction-Based VNF Auto-Scaling and Deployment Mechanism for Flexible and Elastic Service ProvisionIEEE Transactions on Services Computing10.1109/TSC.2024.344005017:5(2959-2973)Online publication date: Sep-2024
https://doi.org/10.1109/TSC.2024.3440050
De Simone LDi Mauro MNatella RPostiglione F(2024)Performance and Availability Challenges in Designing Resilient 5G ArchitecturesIEEE Transactions on Network and Service Management10.1109/TNSM.2024.340456021:5(5291-5303)Online publication date: Oct-2024
https://doi.org/10.1109/TNSM.2024.3404560
Barletta MDe Simone LCorte RDi Martino C(2024)Failover Timing Analysis in Orchestrating Container-based Critical Applications2024 19th European Dependable Computing Conference (EDCC)10.1109/EDCC61798.2024.00026(81-84)Online publication date: 8-Apr-2024
https://doi.org/10.1109/EDCC61798.2024.00026
Barletta MCinque MDi Martino CKalbarczyk ZIyer R(2024)Mutiny! How Does Kubernetes Fail, and What Can We Do About It?2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)10.1109/DSN58291.2024.00016(1-14)Online publication date: 24-Jun-2024
https://doi.org/10.1109/DSN58291.2024.00016
Cinque MDe Simone LOttaviano D(2024)Temporal isolation assessment in virtualized safety-critical mixed-criticality systems: A case study on Xen hypervisorJournal of Systems and Software10.1016/j.jss.2024.112147216(112147)Online publication date: Oct-2024
https://doi.org/10.1016/j.jss.2024.112147

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents