Abstract
Recently, container technology is gaining increasing attention and has become an alternative to the traditional virtual machines artifact. The technology is used to deploy large-scale applications in several areas such as Big Data, AI, and High-Performance Computing (HPC). In the HPC field, several management tools exist as Slurm in one hand. On the other hand, the literature has considered many container scheduling strategies. The majority of container scheduling strategies don’t think about the amount of data transmitted between containers. This paper presents a new container scheduling strategy that automatically groups containers that belong to the same group (Namespace) on the same node. In brief, the plan is application-aware as long as someone knows which containers should be grouped in the same Namespace. The objective is to compact the nodes with containers of the same group to reduce the number of nodes used, the communication inter-node costs, and improve containerized applications’ overall Quality-of-Service (QoS). Our proposed strategy is implemented under the Kubernetes framework. Experiments demonstrate the potential of our strategy under different scenarios. Most importantly, we show first that cohabitation between our new scheduling strategy and the default Kubernetes strategy is possible and for the benefit of the system. Thanks to Namespaces, cohabitation is not limited to two methods for scheduling either batch jobs or online services. Second, thanks to the deployment automation, we also demonstrate that multiple Slurm clusters can be instantiated from a pool of bare metal nodes. This reality contributes to the concept of “HPC as a Service.”
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
See https://golang.org and https://kubernetes.io/.
References
Bauer, M.: Solving Problems in HPC with Singularity. CernVM Workshop 2019, June 2019. https://cds.cern.ch/record/2677637
Beltre, A.M., Saha, P., Govindaraju, M., Younge, A., Grant, R.E.: Enabling HPC workloads on cloud infrastructure using kubernetes container orchestration mechanisms. In: 2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC), pp. 11–20 (2019)
Casalicchio, E., Iannucci, S.: The state-of-the-art in container technologies: application, orchestration and security. Concurrency Comput. Practice Exp. 32(17), e5668 (2020)
Cérin, C., Greneche, N., Menouer, T.: Towards pervasive containerization of HPC job schedulers. In: 2020 IEEE 32nd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pp. 281–288 (2020)
Guan, X., Wan, X., Choi, B., Song, S., Zhu, J.: Application oriented dynamic resource allocation for data centers using docker containers. IEEE Commun. Lett. 21(3), 504–507 (2017)
Hoque, S., d. Brito, M.S., Willner, A., Keil, O., Magedanz, T.: Towards container orchestration in fog computing infrastructures. In: 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), vol. 2, pp. 294–299, July 2017
Jiang, C., et al.: Characterizing co-located workloads in alibaba cloud datacenters. IEEE Trans. Cloud Comput., 1 (2020)
Liu, B., Li, P., Lin, W., Shu, N., Li, Y., Chang, V.: A new container scheduling algorithm based on multi-objective optimization. Soft Comput. 22, 1–12 (2018)
Marzolla, M., Babaoglu, Ö., Panzieri, F.: Server consolidation in clouds through gossiping. In: 12th IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks, WOWMOM, Lucca, Italy, 20–24 June, 2011, pp. 1–6 (2011)
Menouer, T., Darmon, P.: A new container scheduling algorithm based on multi-objective optimization. In: 27th Euromicro International Conference on Parallel, Distributed and Network-based Processing, Pavia, Italy, February 2019
Menouer, T., Manad, O., Cérin, C., Darmon, P.: Power efficiency containers scheduling approach based on machine learning technique for cloud computing environment. In: Esposito, C., Hong, J., Choo, K.-K.R. (eds.) I-SPAN 2019. CCIS, vol. 1080, pp. 193–206. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30143-9_16
Pahl, C., Lee, B.: Containers and clusters for edge cloud architectures - a technology review. In: 2015 3rd International Conference on Future Internet of Things and Cloud, pp. 379–386, August 2015
Smarr, L., Catlett, C.: Metacomputing. Commun. ACM 35, 44–52 (1992)
Steve Buchanan, Janaka Rangama, N.B.: Introducing azure kubernetes service: a practical guide to container orchestration. In: Apress (2019)
Sureshkumar, M., Rajesh, P.: Optimizing the docker container usage based on load scheduling. In: 2017 2nd International Conference on Computing and Communications Technologies (ICCCT), pp. 165–168, February 2017
Xin, L.: The evolution of large-scale co-location technology at alibaba, 28 November 2019. https://www.alibabacloud.com/blog/the-evolution-of-large-scale-co-location-technology-at-alibaba_595595
Zhao, A., Huang, Q., Huang, Y., Zou, L., Chen, Z., Song, J.: Research on resource prediction model based on kubernetes container auto-scaling technology. IOP Conf. Ser. Materials Sci. Eng. 569, 052092 (2019)
Zhou, N., Georgiou, Y., Zhong, L., Zhou, H., Pospieszny, M.: Container orchestration on HPC systems. In: 2020 IEEE 13th International Conference on Cloud Computing (CLOUD), pp. 34–36. IEEE (2020)
The apache software foundation. mesos, apache. http://mesos.apache.org/
Docker swarmkit. https://github.com/docker/swarmkit/
Kubernetes scheduler. https://kubernetes.io/
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 IFIP International Federation for Information Processing
About this paper
Cite this paper
Menouer, T., Greneche, N., Cérin, C., Darmon, P. (2022). Towards an Optimized Containerization of HPC Job Schedulers Based on Namespaces. In: Cérin, C., Qian, D., Gaudiot, JL., Tan, G., Zuckerman, S. (eds) Network and Parallel Computing. NPC 2021. Lecture Notes in Computer Science(), vol 13152. Springer, Cham. https://doi.org/10.1007/978-3-030-93571-9_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-93571-9_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-93570-2
Online ISBN: 978-3-030-93571-9
eBook Packages: Computer ScienceComputer Science (R0)