PvCC: A vCPU Scheduling Policy for DPDK-applied Systems at Multi-Tenant Edge Data Centers
Authors: 
Yuki Tsujimoto, Yuki Sato, Kenichi Yasukata, Kenta Ishiguro, Kenji KonoAuthors Info & Claims
Yuki Tsujimoto, Yuki Sato, Kenichi Yasukata, Kenta Ishiguro, Kenji KonoAuthors Info & ClaimsPages 379 - 391
Abstract
This paper explores a practical means to employ Data Plane Development Kit (DPDK), a kernel-bypassing framework for packet processing, in resource-limited multi-tenant edge data centers. The problem is that the traditional virtual CPU (vCPU) schedulers are not well compatible with the event detection model of DPDK, which needs to monopolize a physical CPU (pCPU) for NIC register polling. Consequently, DPDK-applied systems running on consolidated Virtual Machines (VMs), a common setup at edges, fail to achieve low serving latencies regardless of the use of DPDK. Toward edge data center providers, this work presents a new vCPU scheduling policy named Polling vCPU Consolidation (PvCC) which runs DPDK-applied systems on dedicated pCPUs adopting microsecond-scale time slices. Along with this, we introduce a mechanism to determine an appropriate number of dedicated pCPUs according to customers' demands represented through a newly introduced vCPU scaling API enabling customers to scale up/down the vCPUs of their VMs at runtime. Our experiments show that PvCC allows DPDK-applied systems running on consolidated VMs to achieve low serving latencies, and our vCPU scaling API enables customers to adjust CPU resource assignment according to the incoming request rate and providers to effectively assign spare pCPUs to VMs executing non-latency-sensitive best-effort tasks.
References
[1]
Jeongseob Ahn, Chang Hyun Park, Taekyung Heo, and Jaehyuk Huh. 2018. Accelerating Critical OS Services in Virtualized Systems with Flexible Micro-sliced Cores. In Proceedings of the Thirteenth EuroSys Conference (EuroSys '18). Association for Computing Machinery, New York, NY, USA, 1--14.
[2]
akopytov. 2020. sysbench: Scriptable database and system performance benchmark. https://github.com/akopytov/sysbench.
[3]
Telecommunications Industry Association. 2018. TIA Position Paper Edge Data Centers. https://www.tiaonline.org/wp-content/uploads/2018/10/TIA_Position_Paper_Edge_Data_Centers-18Oct18.pdf.
[4]
Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. 2003. Xen and the Art of Virtualization. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles (Bolton Landing, NY, USA) (SOSP '03). Association for Computing Machinery, New York, NY, USA, 164--177.
[5]
Adam Belay, George Prekas, Ana Klimovic, Samuel Grossman, Christot Kozyrakis, and Edouard Bugnion. 2014. IX: A Protected Dataplane Operating System for High Throughput and Low Latency. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI '14). USENIX Association, Broomfield, CO, 49--65. https://www.usenix.org/conference/osdi14/technical-sessions/presentation/belay
[6]
Keyan Cao, Yefan Liu, Gongjie Meng, and Qimeng Sun. 2020. An Overview on Edge Computing Research. IEEE Access 8 (2020), 85714--85728.
[7]
Jiasi Chen and Xukan Ran. 2019. Deep Learning With Edge Computing: A Review. Proc. IEEE 107, 8 (2019), 1655--1674.
[8]
Luwei Cheng and Cho-Li Wang. 2012. vBalance: using interrupt load balance to improve I/O performance for SMP virtual machines. In Proceedings of the Third ACM Symposium on Cloud Computing (San Jose, California) (SoCC '12). Association for Computing Machinery, New York, NY, USA, Article 2, 14 pages.
[9]
dpdk.org. 2022. DPDK: the Data Plane Development Kit. https://www.dpdk.org.
[10]
Yicheng Feng, Shihao Shen, Xiaofei Wang, Qiao Xiang, Hong Xu, Chenren Xu, and Wenyu Wang. 2024. BREAK: A Holistic Approach for Efficient Container Deployment among Edge Clouds. In IEEE INFOCOM 2024 - IEEE Conference on Computer Communications. 1491--1500.
[11]
Joshua Fried, Zhenyuan Ruan, Amy Ousterhout, and Adam Belay. 2020. Caladan: Mitigating Interference at Microsecond Timescales. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI '20). USENIX Association, 281--297. https://www.usenix.org/conference/osdi20/presentation/fried
[12]
GSMA. 2022. Telco Edge Cloud Value & Achievements. https://www.gsma.com/futurenetworks/wp-content/uploads/2022/03/GSMA-TEC-Value-Whitepaper-v13.pdf.
[13]
intel.com. 2008. PCI-SIG Single Root I/O Virtualization (SR-IOV) Support in Intel(R) Virtualization Technology for Connectivity. https://www.intel.com/content/dam/doc/white-paper/pci-sig-single-root-io-virtualization-support-in-virtualization-technology-for-connectivity-paper.pdf.
[14]
Calin Iorgulescu, Reza Azimi, Youngjin Kwon, Sameh Elnikety, Manoj Syamala, Vivek Narasayya, Herodotos Herodotou, Paulo Tomita, Alex Chen, Jack Zhang, and Junhua Wang. 2018. PerfIso: Performance Isolation for Commercial Latency-Sensitive Services. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). USENIX Association, Boston, MA, 519--532. https://www.usenix.org/conference/atc18/presentation/iorgulescu
[15]
Seyyed Ahmad Javadi, Amoghavarsha Suresh, Muhammad Wajahat, and Anshul Gandhi. 2019. Scavenger: A Black-Box Batch Workload Resource Manager for Improving Utilization in Cloud Environments. In Proceedings of the ACM Symposium on Cloud Computing (Santa Cruz, CA, USA) (SoCC '19). Association for Computing Machinery, New York, NY, USA, 272--285.
[16]
Eun Young Jeong, Shinae Woo, Muhammad Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. 2014. mTCP: A Highly Scalable User-level TCP Stack for Multicore Systems. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI '14). USENIX Association, Seattle, WA, USA, 489--502. https://www.usenix.org/conference/nsdi14/technical-sessions/presentation/jeong
[17]
Weiwei Jia, Jianchen Shan, Tsz On Li, Xiaowei Shang, Heming Cui, and Xiaoning Ding. 2020. vSMT-IO: Improving I/O Performance and Efficiency on SMT Processors in Virtualized Clouds. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association, 449--463. https://www.usenix.org/conference/atc20/presentation/jia
[18]
Weiwei Jia, Jiyuan Zhang, Jianchen Shan, Jing Li, and Xiaoning Ding. 2022. Achieving low latency in public edges by hiding workloads mutual interference. In Proceedings of the 13th Symposium on Cloud Computing (San Francisco, California) (SoCC '22). Association for Computing Machinery, New York, NY, USA, 477--492.
[19]
Kostis Kaffes, Timothy Chong, Jack Tigar Humphries, Adam Belay, David Maziéres, and Christos Kozyrakis. 2019. Shinjuku: Preemptive Scheduling for μsecond-scale Tail Latency. In Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI '19). USENIX Association, Boston, MA, USA, 345--359. https://www.usenix.org/conference/nsdi19/presentation/kaffes
[20]
Kostis Kaffes, Dragos Sbirlea, Yiyan Lin, David Lo, and Christos Kozyrakis. 2020. Leveraging application classes to save power in highly-utilized data centers. In Proceedings of the 11th ACM Symposium on Cloud Computing (Virtual Event, USA) (SoCC '20). Association for Computing Machinery, New York, NY, USA, 134--149.
[21]
Leverich, Terei, Lin, and Schatzberg. 2015. Mutilate: high-performance mem-cached load generator. https://github.com/leverich/mutilate.
[22]
Djob Mvondo, Boris Teabe, Alain Tchana, Daniel Hagimont, and Noel De Palma. 2019. Closer: A New Design Principle for the Privileged Virtual Machine OS. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). 49--60.
[23]
Amy Ousterhout, Joshua Fried, Jonathan Behrens, Adam Belay, and Hari Balakrishnan. 2019. Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Workloads. In Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI '19). USENIX Association, Boston, MA, USA, 361--377. https://www.usenix.org/conference/nsdi19/presentation/ousterhout
[24]
George Prekas, Marios Kogias, and Edouard Bugnion. 2017. ZygOS: Achieving Low Tail Latency for Microsecond-scale Networked Tasks. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP '17). Association for Computing Machinery, New York, NY, USA, 325--341.
[25]
Henry Qin, Qian Li, Jacqueline Speiser, Peter Kraft, and John Ousterhout. 2018. Arachne: Core-Aware Thread Management. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 145--160. https://www.usenix.org/conference/osdi18/presentation/qin
[26]
redis.io. 2013. memtier_benchmark: A command line utility for load generation and benchmarking NoSQL key-value databases. https://github.com/RedisLabs/memtier_benchmark.
[27]
ScyllaDB. 2014. Seastar: High performance server-side application framework. http://seastar.io.
[28]
Weisong Shi, Jie Cao, Quan Zhang, Youhuizi Li, and Lanyu Xu. 2016. Edge Computing: Vision and Challenges. IEEE Internet of Things Journal 3, 5 (2016), 637--646.
[29]
Wenda Tang, Yutao Ke, Senbo Fu, Hongliang Jiang, Junjie Wu, Qian Peng, and Feng Gao. 2022. Demeter: QoS-aware CPU scheduling to reduce power consumption of multiple black-box workloads. In Proceedings of the 13th Symposium on Cloud Computing (San Francisco, California) (SoCC '22). Association for Computing Machinery, New York, NY, USA, 31--46.
[30]
Boris Teabe, Alain Tchana, and Daniel Hagimont. 2016. Application-specific quantum for multi-core platform scheduler. In Proceedings of the Eleventh European Conference on Computer Systems (London, United Kingdom) (EuroSys '16). Association for Computing Machinery, New York, NY, USA, Article 3, 14 pages.
[31]
Tencent. 2017. F-Stack. https://www.f-stack.org/.
[32]
Tene, Gil, Barker, and Mike. 2014. wrk2: A constant throughput, correct latency recording variant of wrk. https://github.com/giltene/wrk2.
[33]
Anatoly Vorobey and Brad Fitzpatrick. 2003. Memcached - a distributed memory object caching system. https://memcached.org/.
[34]
Mark A. Williamson. [n. d.]. xentrace. https://linux.die.net/man/8/xentrace.
[35]
xenproject.org. 2022. Xen Project. https://xenproject.org.
[36]
Tong Xing, Cong Xiong, Chuan Ye, Qi Wei, Javier Picorel, and Antonio Barbalace. 2023. Maximizing VMs' IO Performance on Overcommitted CPUs with Fairness. In Proceedings of the 2023 ACM Symposium on Cloud Computing (Santa Cruz, CA, USA) (SoCC '23). Association for Computing Machinery, New York, NY, USA, 93--108.
[37]
Cong Xu, Sahan Gamage, Hui Lu, Ramana Kompella, and Dongyan Xu. 2013. vTurbo: Accelerating Virtual Machine I/O Processing Using Designated Turbo-Sliced Core. In Proceedings of the 2013 USENIX Annual Technical Conference (USENIX ATC '13). USENIX Association, San Jose, CA, USA, 243--254. https://www.usenix.org/conference/atc13/technical-sessions/presentation/xu
[38]
Cong Xu, Sahan Gamage, Pawan N. Rao, Ardalan Kangarlou, Ramana Rao Kompella, and Dongyan Xu. 2012. vSlicer: latency-aware virtual machine scheduling via differentiated-frequency CPU slicing. In Proceedings of the 21st International Symposium on High-Performance Parallel and Distributed Computing (Delft, The Netherlands) (HPDC '12). Association for Computing Machinery, New York, NY, USA, 3--14.
[39]
Mengwei Xu, Zhe Fu, Xiao Ma, Li Zhang, Yanan Li, Feng Qian, Shangguang Wang, Ke Li, Jingyu Yang, and Xuanzhe Liu. 2021. From cloud to edge: a first look at public edge platforms. In Proceedings of the 21st ACM Internet Measurement Conference (Virtual Event) (IMC '21). Association for Computing Machinery, New York, NY, USA, 37--53.
[40]
Wei Yu, Fan Liang, Xiaofei He, William Grant Hatcher, Chao Lu, Jie Lin, and Xinyu Yang. 2018. A Survey on the Edge Computing for the Internet of Things. IEEE Access 6 (2018), 6900--6919.
[41]
Irene Zhang, Amanda Raybuck, Pratyush Patel, Kirk Olynyk, Jacob Nelson, Omar S. Navarro Leija, Ahilie Martinez, Jing Liu, Anna Kornfeld Simpson, Sujay Jayakar, Pedro Henrique Penna, Max Demoulin, Piali Choudhury, and Anirudh Badam. 2021. The Demikernel Datapath OS Architecture for Microsecond-scale Datacenter Systems. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (SOSP '21). Association for Computing Machinery, New York, NY, USA, 195--211.
Index Terms
- PvCC: A vCPU Scheduling Policy for DPDK-applied Systems at Multi-Tenant Edge Data Centers
Recommendations
Efficient consolidation-aware VCPU scheduling on multicore virtualization platform
Multicore processors are widely used in today's computer systems. Multicore virtualization technology provides an elastic solution to more efficiently utilize the multicore system. However, the Lock Holder Preemption (LHP) problem in the virtualized ...
Performance Analysis for Pareto-Optimal Green Consolidation Based on Virtual Machines Live Migration
Huge energy requirement of cloud data centers is prime concern. Dynamic Virtual Machine VM consolidation based on VM live migration to switched-off or put some of the under-loaded host Physical Machines PMs into a low power consumption mode can ...
Transparently bridging semantic gap in CPU management for virtualized environments
Consolidated environments are progressively accommodating diverse and unpredictable workloads in conjunction with virtual desktop infrastructure and cloud computing. Unpredictable workloads, however, aggravate the semantic gap between the virtual ...
Comments
Information & Contributors
Information
Published In
December 2024
515 pages
ISBN:9798400706233
DOI:10.1145/3652892
- Chair:
- Jiannong Cao,
- Program Chair:
- Zhi Jin,
- Program Co-chair:
- Valerio Schiavoni,
- Workshop Chair:
- Janick Edinger
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
In-Cooperation
- IFIP
- Usenix
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 02 December 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
- NEDO
Conference
Middleware '24
Acceptance Rates
Overall Acceptance Rate 203 of 948 submissions, 21%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 85Total Downloads
- Downloads (Last 12 months)85
- Downloads (Last 6 weeks)29
Reflects downloads up to 02 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in