research-article

NBWGuard: Realizing Network QoS for Kubernetes

Authors:

Karthick Rajamani,

Wesley FelterAuthors Info & Claims

Middleware '18: Proceedings of the 19th International Middleware Conference Industry

Pages 32 - 38

https://doi.org/10.1145/3284028.3284033

Published: 10 December 2018 Publication History

Abstract

Kubernetes is a very popular and fast-growing container orchestration platform that automates the process of deploying and managing multi-container applications at scale. Users can specify required and maximum values of resources they need for their containers and Kubernetes realizes them by interfacing with lower levels (container runtime which in turn can use OS capabilities) of the stack for enforcing them. Kubernetes supports differentiated QoS classes - Guaranteed, Burstable, and Best-effort - in order of decreasing priority based on the resource size specifications for CPU and memory capacity. This allows many applications to obtain a desired level of QoS (performance isolation and throughput) when CPU or memory capacity management can provide them. However, when workloads may be critically dependent for their performance on another resource, namely network bandwidth, Kubernetes has no means to meet their QoS needs. Networking between pods in Kubernetes is supported with plug-ins and the network resource is not managed directly.

In this work, we propose NBWGuard, a design for network bandwidth management and evaluate its implementation. NBWGuard lets Kubernetes manage network bandwidth as a resource (like CPU or memory capacity) while still using plug-ins for realizing the network specification desired by users. Consistent with Kubernetes approach to application QoS based on resource allocation NBWGuard also supports the 3 QoS classes: Guaranteed, Burstable, and Best-effort with respect to network bandwidth. NBWGuard is evaluated with iperf benchmark on real cloud environment, and the evaluation results demonstrate that it is able to provide network bandwidth isolation without impact on overall throughput.

Supplementary Material

MP4 File (p32-rajamani.mp4)

Download
126.41 MB

References

[1]

Calico. https://www.projectcalico.org/.

[2]

Canal. https://github.com/projectcalico/canal.

[3]

Container Network Interface. https://github.com/containernetworking/cni.

[4]

Intermediate Functional Block. https://wiki.linuxfoundation.org/networking/ifb.

[5]

The Iperf Benchmark. http://www.noc.ucf.edu/Tools/Iperf/.

[6]

Kubernetes Opaque integer resource. https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/.

[7]

Kubernetes Pod. https://kubernetes.io/docs/concepts/workloads/pods/pod/.

[8]

Linux control groups. https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt.

[9]

Linux traffic control. http://tldp.org/HOWTO/Traffic-Control-HOWTO/index.html.

[10]

Utilities for limiting bandwidth using tc. https://github.com/kubernetes/kubernetes/pull/11988.

[11]

Andreasson, O., et al. Iptables tutorial 1.2. 2. Copyright© 2001-2006 Oskar Andreasson, GNU Free Documentation License (2001).

[12]

Ballani, H., Costa, P., Karagiannis, T., and Rowstron, A. Towards predictable datacenter networks. In ACM SIGCOMM Computer Communication Review (2011), vol. 41, ACM, pp. 242--253.

Digital Library

[13]

Banga, G., Druschel, P., and Mogul, J. C. Resource containers: A new facility for resource management in server systems. In OSDI (1999), vol. 99, pp. 45--58.

Digital Library

[14]

Burns, B., Grant, B., Oppenheimer, D., Brewer, E., and Wilkes, J. Borg, omega, and kubernetes. Queue 14, 1 (2016), 10.

Digital Library

[15]

Dean, J., and Barroso, L. A. The tail at scale. Communications of the ACM 56, 2 (2013), 74--80.

Digital Library

[16]

Delimitrou, C., and Kozyrakis, C. ibench: Quantifying interference for datacenter applications. In Workload Characterization (IISWC), 2013 IEEE International Symposium on (Sept 2013), pp. 23--33.

[17]

Gulati, A., Merchant, A., and Varman, P. J. mclock: handling throughput variability for hypervisor io scheduling. In Proceedings of the 9th USENIX conference on Operating systems design and implementation (2010), USENIX Association, pp. 437--450.

Digital Library

[18]

Guo, C., Lu, G., Wang, H. J., Yang, S., Kong, C., Sun, P., Wu, W., and Zhang, Y. Secondnet: a data center network virtualization architecture with bandwidth guarantees. In Proceedings of the 6th International Conference (2010), ACM, p. 15.

Digital Library

[19]

Gupta, D., Cherkasova, L., Gardner, R., and Vahdat, A. Enforcing performance isolation across virtual machines in xen. In ACM/IFIP/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing (2006), Springer, pp. 342--362.

Digital Library

[20]

Jeyakumar, V., Alizadeh, M., Mazieres, D., Prabhakar, B., Kim, C., and Azure, W. Eyeq: Practical network performance isolation for the multi-tenant cloud. In HotCloud (2012).

Digital Library

[21]

Khalid, J., Rozner, E., Felter, W., Xu, C., Rajamani, K., Ferreira, A., and Akella, A. Iron: Isolating network-based cpu in container environments. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18) (2018), USENIX Association.

[22]

Krebs, R., Momm, C., and Kounev, S. Metrics and techniques for quantifying performance isolation in cloud environments. Science of Computer Programming 90 (2014), 116--134.

[23]

Mace, J., Bodik, P., Fonseca, R., and Musuvathi, M. Retro: Targeted resource management in multi-tenant distributed systems. In NSDI (2015), pp. 589--603.

Digital Library

[24]

McCullough, J. C., Dunagan, J., Wolman, A., and Snoeren, A. C. Stout: An adaptive interface to scalable cloud storage. In Proc. of the USENIX Annual Technical Conference--ATC (2010), pp. 47--60.

Digital Library

[25]

Medel, V., Rana, O., Bañares, J. Á., and Arronategui, U. Adaptive application scheduling under interference in kubernetes. In Utility and Cloud Computing (UCC), 2016 IEEE/ACM 9th International Conference on (2016), IEEE, pp. 426--427.

Digital Library

[26]

Mundada, Y., Ramachandran, A., and Feamster, N. Silverline: Data and network isolation for cloud services. In HotCloud (2011).

Digital Library

[27]

Pahl, C. Containerization and the paas cloud. IEEE Cloud Computing 2, 3 (2015), 24--31.

[28]

Pahl, C., Brogi, A., Soldani, J., and Jamshidi, P. Cloud container technologies: a state-of-the-art review. IEEE Transactions on Cloud Computing (2017).

[29]

Pahl, C., Helmer, S., Miori, L., Sanin, J., and Lee, B. A container-based edge cloud paas architecture based on raspberry pi clusters. In Future Internet of Things and Cloud Workshops (FiCloudW), IEEE International Conference on (2016), IEEE, pp. 117--124.

[30]

Popa, L., Kumar, G., Chowdhury, M., Krishnamurthy, A., Ratnasamy, S., and Stoica, I. FairCloud: Sharing the Network in Cloud Computing. In SIGCOMM (2012).

Digital Library

[31]

Popa, L., Yalagandula, P., Banerjee, S., Mogul, J. C., Turner, Y., and Santos, J. R. Elasticswitch: Practical work-conserving bandwidth guarantees for cloud computing. ACM SIGCOMM Computer Communication Review 43, 4 (2013), 351--362.

Digital Library

[32]

Rodrigues, H., Santos, J. R., Turner, Y., Soares, P., and Guedes, D. O. Gatekeeper: Supporting bandwidth guarantees for multi-tenant datacenter networks. In WIOV (2011).

Digital Library

[33]

Shieh, A., Kandula, S., Greenberg, A. G., and Kim, C. Seawall: Performance isolation for cloud datacenter networks. In HotCloud (2010).

Digital Library

[34]

Tosatto, A., Ruiu, P., and Attanasio, A. Container-based orchestration in cloud: state of the art and challenges. In Complex, Intelligent, and Software Intensive Systems (CISIS), 2015 Ninth International Conference on (2015), IEEE, pp. 70--75.

Digital Library

[35]

Wachs, M., Abd-El-Malek, M., Thereska, E., and Ganger, G. R. Argon: Performance insulation for shared storage servers. In FAST (2007), vol. 7, pp. 5--5.

Digital Library

[36]

Xu, C., Rajamani, K., Ferreira, A., Felter, W., Rubio, J., and Li, Y. dcat: dynamic cache management for efficient, performance-sensitive infrastructure-as-a-service. In Proceedings of the Thirteenth EuroSys Conference (2018), ACM, p. 14.

Digital Library

Cited By

Govindarajan CNaik PGovindarajan KGoel SKodeswaran PSen SJavachandran P(2024)Syscall Analysis for Resource Stress Identification for Container Network Functions2024 IEEE 17th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD62652.2024.00037(256-266)Online publication date: 7-Jul-2024
https://doi.org/10.1109/CLOUD62652.2024.00037
Lee KLee KPark HHwang JYoo C(2023)Autothrottle: Satisfying Network Performance Requirements for ContainersIEEE Transactions on Cloud Computing10.1109/TCC.2022.318639711:2(2096-2109)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TCC.2022.3186397
Monaco GGala GFohler G(2023)Shared Resource Orchestration Extensions for Kubernetes to Support Real-Time Cloud Containers2023 IEEE 26th International Symposium on Real-Time Distributed Computing (ISORC)10.1109/ISORC58943.2023.00022(97-106)Online publication date: May-2023
https://doi.org/10.1109/ISORC58943.2023.00022
Show More Cited By

Recommendations

Detection of quality of service degradation on multi-tenant containerized services
Abstract
Computational services are progressively migrating to container-based solutions due to their faster provision time and lower resource allocation overhead. Service providers rely on container multi-tenancy to share their computing infrastructure ...
Extensions to P2MP RSVP-TE for VPN-specific state provisioning with fair resource sharing

Among the resource provisioning algorithms for the hose-based Virtual Private Network (VPN) Quality of Service (QoS), VPN-specific state provisioning allows the service provider to obtain highest resource multiplexing gains. In this paper, we show that ...
FIAC: a resource discovery-based two-level admission control for differentiated service networks

Differentiated Service (DiffServ) architecture has been proposed as a scalable QoS architecture for Internet. DiffServ, however, could not control its loads under heavy traffic conditions, and it could not provide strong QoS responses for individual ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

Middleware '18: Proceedings of the 19th International Middleware Conference Industry

December 2018

64 pages

ISBN:9781450360166

DOI:10.1145/3284028

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery
USENIX Assoc: USENIX Assoc
IFIP: International Federation for Information Processing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 December 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

Middleware '18

Sponsor:

ACM
USENIX Assoc
IFIP

Middleware '18: 19th International Middleware Conference

December 10 - 14, 2018

Rennes, France

Acceptance Rates

Overall Acceptance Rate 203 of 948 submissions, 21%

Upcoming Conference

MIDDLEWARE '24

25th International Middleware Conference

December 2 - 6, 2024

Hong Kong , Hong Kong

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
1,089
Total Downloads

Downloads (Last 12 months)80
Downloads (Last 6 weeks)5

Reflects downloads up to 03 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Govindarajan CNaik PGovindarajan KGoel SKodeswaran PSen SJavachandran P(2024)Syscall Analysis for Resource Stress Identification for Container Network Functions2024 IEEE 17th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD62652.2024.00037(256-266)Online publication date: 7-Jul-2024
https://doi.org/10.1109/CLOUD62652.2024.00037
Lee KLee KPark HHwang JYoo C(2023)Autothrottle: Satisfying Network Performance Requirements for ContainersIEEE Transactions on Cloud Computing10.1109/TCC.2022.318639711:2(2096-2109)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TCC.2022.3186397
Monaco GGala GFohler G(2023)Shared Resource Orchestration Extensions for Kubernetes to Support Real-Time Cloud Containers2023 IEEE 26th International Symposium on Real-Time Distributed Computing (ISORC)10.1109/ISORC58943.2023.00022(97-106)Online publication date: May-2023
https://doi.org/10.1109/ISORC58943.2023.00022
Wen HCao ZLi BDu DAbouelwafa AVoigt DLiu SDiehl JWu F(2023)K8sES: Optimizing Kubernetes with Enhanced Storage Service-Level Objectives2023 IEEE 41st International Conference on Computer Design (ICCD)10.1109/ICCD58817.2023.00041(214-222)Online publication date: 6-Nov-2023
https://doi.org/10.1109/ICCD58817.2023.00041
Liu YHerranz A(2023)Enabling 5G QoS configuration capabilities for IoT applications on container orchestration platform2023 IEEE International Conference on Cloud Computing Technology and Science (CloudCom)10.1109/CloudCom59040.2023.00023(63-68)Online publication date: 4-Dec-2023
https://doi.org/10.1109/CloudCom59040.2023.00023
Roh JChoi SPark K(2023)CO-TRIS: Container Orchestration – Transforming container using Resource Inspection System2023 IEEE Sixth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE)10.1109/AIKE59827.2023.00027(121-124)Online publication date: 25-Sep-2023
https://doi.org/10.1109/AIKE59827.2023.00027
Kim ELee KYoo C(2023)Network SLO-aware container scheduling in KubernetesThe Journal of Supercomputing10.1007/s11227-023-05122-579:10(11478-11494)Online publication date: 28-Feb-2023
https://doi.org/10.1007/s11227-023-05122-5
Zha ZWang AGuo YLi QSun KChen S(2022)EZPath: Expediting Container Network Traffic via Programmable Switches2022 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking55013.2022.9829818(1-8)Online publication date: 13-Jun-2022
https://doi.org/10.23919/IFIPNetworking55013.2022.9829818
Rejiba ZChamanara J(2022)Custom Scheduling in Kubernetes: A Survey on Common Problems and Solution ApproachesACM Computing Surveys10.1145/354478855:7(1-37)Online publication date: 15-Dec-2022
https://dl.acm.org/doi/10.1145/3544788
Govindarajan KGovindarajan CVerma M(2022)Network Aware Container Orchestration for Telco Workloads2022 IEEE 15th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD55607.2022.00063(397-406)Online publication date: Jul-2022
https://doi.org/10.1109/CLOUD55607.2022.00063
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents