research-article

Survey of adaptive containerization architectures for HPC

Authors:

Nina Mujkanovic,

Juan J. Durillo,

Nicolay Hammer,

Tiziano MüllerAuthors Info & Claims

SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

Pages 165 - 176

https://doi.org/10.1145/3624062.3624588

Published: 12 November 2023 Publication History

Abstract

Containers offer an array of advantages that benefit research reproducibility and portability. As container tools mature, container security improves, and high-performance computing (HPC) and cloud system tools converge, supercomputing centers are increasingly integrating containers into their workflows. Despite this, most research into containers remains focused on cloud environments.

We consider an adaptive containerization architecture approach, in which each component chosen represents the tool best adapted to the given system and site requirements, with a focus on accelerating the deployment of applications and workflows on HPC systems using containers. To this end, we discuss the HPC specific requirements regarding container tools, and analyze the entire containerization stack, including container engines and registries, in-depth. Finally, we consider various orchestrator and HPC workload manager integration scenarios, including Workload Manager (WLM) in Kubernetes, Kubernetes in WLM, and bridged scenarios. We present a proof-of-concept approach to a Kubernetes Agent in a WLM allocation.

Supplemental Material

MP4 File

Recording of "Survey of adaptive containerization architectures for HPC" presentation at CANOPIE-HPC 2023.

Download
152.16 MB

References

[1]

Subil Abraham, Arnab K Paul, Redwan Ibne Seraj Khan, and Ali R Butt. 2020. On the use of containers in high performance computing environments. In 2020 IEEE 13th International Conference on Cloud Computing (CLOUD). IEEE, Virtual, 284–293.

[2]

Angel M. Beltre, Pankaj Saha, Madhusudhan Govindaraju, Andrew Younge, and Ryan E. Grant. 2019. Enabling HPC Workloads on Cloud Infrastructure Using Kubernetes Container Orchestration Mechanisms. In 2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). IEEE, Denver, 11–20. https://doi.org/10.1109/CANOPIE-HPC49598.2019.00007

[3]

Lucas Benedicic, Felipe A. Cruz, Alberto Madonna, and Kean Mariotti. 2019. Sarus: Highly Scalable Docker Containers for HPC Systems. In High Performance Computing(Lecture Notes in Computer Science), Michèle Weiland, Guido Juckeland, Sadaf Alam, and Heike Jagode (Eds.). Springer International Publishing, Cham, 46–60. https://doi.org/10.1007/978-3-030-34356-9_5

Digital Library

[4]

Lucas Benedicic, Miguel Gila, Sadaf Alam, and T Schulthess. 2016. Opportunities for container environments on Cray XC30 with GPU devices. In Cray Users Group Conference (CUG16). Cray User Group, London, 1–11.

[5]

Ouafa Bentaleb, Adam S. Z. Belloum, Abderrazak Sebaa, and Aouaouche El-Maouhab. 2022. Containerization Technologies: Taxonomies, Applications and Challenges. J Supercomput 78, 1 (Jan. 2022), 1144–1181. https://doi.org/10.1007/s11227-021-03914-1

Digital Library

[6]

Emiliano Casalicchio and Stefano Iannucci. 2020. The state-of-the-art in container technologies: Application, orchestration and security. Concurrency and Computation: Practice and Experience 32, 17 (2020), e5668.

[7]

Kubernetes Community. 2023. Community/Icons at Master · Kubernetes/Community. https://github.com/kubernetes/community/tree/master/icons. Icons provided by the Kubernetes Community under CC-BY-4.0 license.

[8]

ReFrame HPC community. 2023. ReFrame. https://github.com/reframe-hpc/reframe

[9]

Containers. 2023. Containers/Crun. https://github.com/containers/crun

[10]

Containers. 2023. OCIcrypt Library. https://github.com/containers/ocicrypt

[11]

NVIDIA Corporation. 2023. ENROOT. https://github.com/NVIDIA/enroot

[12]

dragonflyoss. 2023. Dragonfly. https://github.com/dragonflyoss/Dragonfly2

[13]

Dave Dykstra. 2022. Apptainer Without Setuid. https://doi.org/10.48550/arXiv.2208.12106 arxiv:2208.12106 [cs]

[14]

Daniel Fulton. 2022. Containers for HPC: Shifter and Podman. https://www.nersc.gov/assets/Uploads/06-Containers-for-HPC-Shifter-and-Podman.pdf

[15]

Yiannis Georgiou, Naweiluo Zhou, Li Zhong, Dennis Hoppe, Marcin Pospieszny, Nikela Papadopoulou, Kostis Nikas, Orestis Lagkas Nikolos, Pavlos Kranas, Sophia Karagiorgou, 2020. Converging HPC, Big Data and Cloud technologies for precision agriculture data analytics on supercomputers. In High Performance Computing: ISC High Performance 2020 International Workshops, Frankfurt, Germany, June 21–25, 2020, Revised Selected Papers 35. Springer, Frankfurt, 368–379.

Digital Library

[16]

Harbor. 2023. Harbor. https://github.com/goharbor/harbor

[17]

Red Hat. 2023. Containers/Podman: Podman: A Tool for Managing OCI Containers and Pods.https://github.com/containers/podman

[18]

Cloud Hypervisor. 2023. Cloud Hypervisor. https://github.com/cloud-hypervisor/cloud-hypervisor

[19]

Sylabs Inc.2023. SingularityCE. https://github.com/sylabs/singularity

[20]

Open Container Initiative. 2023. Runc. https://github.com/opencontainers/runc

[21]

Douglas M Jacobsen and Richard Shane Canon. 2015. Contain This, Unleashing Docker for HPC. In CUG2015. Cray User Group, Chicago, 33–49.

[22]

k3s io. 2023. K3s - Lightweight Kubernetes. https://github.com/k3s-io/k3s

[23]

Rafael Keller Tesser and Edson Borin. 2022. Containers in HPC: A Survey. J. Supercomput. 79, 5 (Oct. 2022), 5759–5827. https://doi.org/10.1007/s11227-022-04848-y

Digital Library

[24]

Kubeflow. 2023. Kubeflow/Kubeflow. https://github.com/kubeflow/kubeflow

[25]

Kubernetes. 2023. Kubernetes (K8s). https://github.com/kubernetes/kubernetes

[26]

SchedMD LLC 2023. Slurm. Slurm Development and Support. https://github.com/SchedMD/slurm

[27]

Sergio López-Huguet, J. Damià Segrelles, Marek Kasztelnik, Marian Bubak, and Ignacio Blanquer. 2020. Seamlessly Managing HPC Workloads Through Kubernetes. In High Performance Computing(Lecture Notes in Computer Science), Heike Jagode, Hartwig Anzt, Guido Juckeland, and Hatem Ltaief (Eds.). Springer International Publishing, Cham, 310–320. https://doi.org/10.1007/978-3-030-59851-8_20

Digital Library

[28]

Boris Lublinsky, Elise Jennings, and Viktória Spišaková. 2022. A Kubernetes ’Bridge’ Operator between Cloud and External Resources. https://doi.org/10.48550/arXiv.2207.02531 arxiv:2207.02531 [cs]

[29]

Evangelos Maliaroudakis, Antony Chazapis, Alexandros Kanterakis, Manolis Marazakis, and Angelos Bilas. 2022. Interactive, Cloud-Native Workflows on HPC Using KNoC. In High Performance Computing. ISC High Performance 2022 International Workshops(Lecture Notes in Computer Science), Hartwig Anzt, Amanda Bienz, Piotr Luszczek, and Marc Baboulin (Eds.). Springer International Publishing, Cham, 221–232. https://doi.org/10.1007/978-3-031-23220-6_15

Digital Library

[30]

Nina Mujkanovic, Karthee Sivalingam, and Alfio Lazzaro. 2020. Optimising AI Training Deployments using Graph Compilers and Containers. In 2020 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, Virtual, 1–8. https://doi.org/10.1109/HPEC43674.2020.9286153

[31]

National Energy Research Scientific Computing Center (NERSC). 2023. Podman-HPC. https://github.com/NERSC/podman-hpc

[32]

VBCF NGS. 2023. Hinkskalle. https://github.com/csf-ngs/hinkskalle

[33]

Daniel Nüst, Vanessa Sochat, Ben Marwick, Stephen J Eglen, Tim Head, Tony Hirst, and Benjamin D Evans. 2020. Ten simple rules for writing Dockerfiles for reproducible data science., e1008316 pages.

[34]

Claus Pahl, Antonio Brogi, Jacopo Soldani, and Pooyan Jamshidi. 2019. Cloud Container Technologies: A State-of-the-Art Review. IEEE Transactions on Cloud Computing 7, 3 (2019), 677–692. https://doi.org/10.1109/TCC.2017.2702586

[35]

Reid Priedhorsky, R. Shane Canon, Timothy Randles, and Andrew J. Younge. 2021. Minimizing Privilege for Building HPC Containers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis(SC ’21). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3458817.3476187

Digital Library

[36]

Reid Priedhorsky and Tim Randles. 2017. Charliecloud: Unprivileged Containers for User-Defined Software Stacks in HPC. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis(SC ’17). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3126908.3126925

Digital Library

[37]

Notary Project. 2023. Notary Project Specifications. https://github.com/notaryproject/specifications

[38]

The Apptainer Container Project. 2023. Apptainer. https://github.com/apptainer/apptainer

[39]

QUAY. 2023. Project Quay. https://github.com/quay/quay

[40]

Sigstore. 2023. Cosign. https://github.com/sigstore/cosign

[41]

Sigstore. 2023. Sigstore. https://www.sigstore.dev/

[42]

Vanessa Sochat and Alec Scott. 2021. Collaborative Container Modules with Singularity Registry HPC. JOSS 6, 63 (July 2021), 3311. https://doi.org/10.21105/joss.03311

[43]

Harmen Stoppels, Simon Pintarelli, and Ben Cumming. 2023. Squashfs-Mount. Swiss National Supercomputing Center (CSCS). https://github.com/eth-cscs/squashfs-mount

[44]

virtual kubelet. 2023. Virtual Kubelet. https://github.com/virtual-kubelet/virtual-kubelet

[45]

Tim Wickberg. 2022. Slurm and/or/vs Kubernetes.

[46]

Laura Wratten, Andreas Wilm, and Jonathan Göke. 2021. Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers. Nature methods 18, 10 (2021), 1161–1168.

[47]

Junqi Yin, Shubhankar Gahlot, Nouamane Laanait, Ketan Maheshwari, Jack Morrison, Sajal Dash, and Mallikarjun Shankar. 2019. Strategies to Deploy and Scale Deep Learning on the Summit Supercomputer. In 2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS). IEEE, Denver, 84–94. https://doi.org/10.1109/DLS49591.2019.00016

[48]

Andrew Younge. 2021. Constructing Containers for Exascale Computing.Technical Report. Sandia National Lab.(SNL-NM), Albuquerque, NM (United States).

[49]

Andrew J. Younge, Kevin Pedretti, Ryan E. Grant, and Ron Brightwell. 2017. A Tale of Two Systems: Using Containers to Deploy HPC Applications on Supercomputers and Clouds. In 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom). IEEE, Hong Kong, 74–81. https://doi.org/10.1109/CloudCom.2017.40

[50]

Naweiluo Zhou, Huan Zhou, and Dennis Hoppe. 2023. Containerisation for High Performance Computing Systems: Survey and Prospects. IIEEE Trans. Software Eng. 49, 4 (April 2023), 2722–2740. https://doi.org/10.1109/TSE.2022.3229221 arxiv:2212.08717 [cs]

Digital Library

[51]

The zot Project. 2023. Zot. https://github.com/project-zot/zot

Cited By

Szandała T(2025)ChatGPT vs human expertise in the context of IT recruitmentExpert Systems with Applications10.1016/j.eswa.2024.125868264(125868)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125868
Sun RSiller K(2024)HPC Container Management at the University of VirginiaPractice and Experience in Advanced Research Computing 2024: Human Powered Computing10.1145/3626203.3670568(1-4)Online publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1145/3626203.3670568
Jolicoeur LDiakhaté FNamyst R(2023)Leveraging Private Container Networks for Increased User Isolation and Flexibility on HPC ClustersHigh Performance Computing. ISC High Performance 2024 International Workshops10.1007/978-3-031-73716-9_29(415-426)Online publication date: 12-May-2023
https://dl.acm.org/doi/10.1007/978-3-031-73716-9_29
Show More Cited By

Index Terms

Survey of adaptive containerization architectures for HPC

Recommendations

Containers in HPC: a survey
Abstract
OS-level virtualization (containers) has become a popular alternative to hypervisor-based virtualization. From a system-administration point-of-view, containers enable support for user-defined software stacks, thus freeing users of restrictions ...
Kubernetes Scheduling: Taxonomy, Ongoing Issues and Challenges
Continuous integration enables the development of microservices-based applications using container virtualization technology. Container orchestration systems such as Kubernetes, which has become the de facto standard, simplify the deployment of container-...
HPC Container Management at the University of Virginia
PEARC '24: Practice and Experience in Advanced Research Computing 2024: Human Powered Computing

We present the management of containers on our HPC cluster through EasyBuild and Apptainer. Containers are installed as modules by customizing the EasyBuild framework so that users can easily navigate the available containers as they would for non-...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

November 2023

2180 pages

ISBN:9798400707858

DOI:10.1145/3624062

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Gauss Centre for Supercomputing
European Commission

Conference

SC-W 2023

SC-W 2023: Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

November 12 - 17, 2023

CO, Denver, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
148
Total Downloads

Downloads (Last 12 months)98
Downloads (Last 6 weeks)5

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Szandała T(2025)ChatGPT vs human expertise in the context of IT recruitmentExpert Systems with Applications10.1016/j.eswa.2024.125868264(125868)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125868
Sun RSiller K(2024)HPC Container Management at the University of VirginiaPractice and Experience in Advanced Research Computing 2024: Human Powered Computing10.1145/3626203.3670568(1-4)Online publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1145/3626203.3670568
Jolicoeur LDiakhaté FNamyst R(2023)Leveraging Private Container Networks for Increased User Isolation and Flexibility on HPC ClustersHigh Performance Computing. ISC High Performance 2024 International Workshops10.1007/978-3-031-73716-9_29(415-426)Online publication date: 12-May-2023
https://dl.acm.org/doi/10.1007/978-3-031-73716-9_29
Lazzaro AHaus UCharousset SMujkanovic N(2023)A User-Oriented Portable, Reproducible, and Scalable Software EcosystemHigh Performance Computing. ISC High Performance 2024 International Workshops10.1007/978-3-031-73716-9_28(402-414)Online publication date: 12-May-2023
https://dl.acm.org/doi/10.1007/978-3-031-73716-9_28

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten