A Methodology and Framework to Determine the Isolation Capabilities of Virtualisation Technologies
Authors: 
Simon Volpert, 
Benjamin Erb, Georg Eisenhart, Daniel Seybold, 
Stefan Wesner, and Jörg DomaschkaAuthors Info & Claims
Simon Volpert, Benjamin Erb, Georg Eisenhart, Daniel Seybold, Stefan Wesner, and Jörg DomaschkaAuthors Info & ClaimsApril 2023
Pages 149 - 160
Abstract
The capability to isolate system resources is an essential characteristic of virtualisation technologies and is therefore important for research and industry alike. It allows the co-location of experiments and workloads, the partitioning of system resources and enables multi-tenant business models such as cloud computing. Poor isolation among tenants bears the risk of noisy-neighbour and contention effects which negatively impacts all of those use-cases. These effects describe the negative impact of one tenant onto another by utilising shared resources. Both industry and research provide many different concepts and technologies to realise isolation. Yet, the isolation capabilities of all these different approaches are not well understood; nor is there an established way to measure the quality of their isolation capabilities. Such an understanding, however, is of uttermost importance in practice to elaborately decide on a suited implementation. Hence, in this work, we present a novel methodology to measure the isolation capabilities of virtualisation technologies for system resources, that fulfils all requirements to benchmarking including reliability. It relies on an immutable approach, based on Experiment-as-Code. The complete process holistically includes everything from bare metal resource provisioning to the actual experiment enactment.
The results determined by this methodology help in the decision for a virtualisation technology regarding its capability to isolate given resources. Such results are presented here as a closing example in order to validate the proposed methodology.
References
[1]
Leonel Aguilar et al. 2022. Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments. Version 1.
[2]
Matej Artac, Tadej Borovssak, Elisabetta Di Nitto, Michele Guerriero, and Damian Andrew Tamburri. 2017. DevOps: Introducing Infrastructure-as-Code. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). (May 2017), 497--498.
[3]
M Ali Babar and Ben Ramsey. 2017. Understanding Container Isolation Mechanisms for Building Security-Sensitive Private Cloud.
[4]
benchAnt. 2022. Database Ranking - Performance & Costs (2022). Retrieved Sept. 12, 2022 from https://benchant.com/ranking/database-ranking.
[5]
Eric W Biederman. 2006. Multiple Instances of the Global Linux Namespaces. In Proceedings of the Linux Symposium, 14.
[6]
Norman Bobroff, Andrzej Kochut, and Kirk Beaty. 2007. Dynamic Placement of Virtual Machines for Managing SLA Violations. In 2007 10th IFIP/IEEE International Symposium on Integrated Network Management. 2007 10th IFIP/IEEE International Symposium on Integrated Network Management. (May 2007), 119--128.
[7]
H. Bouattour, Y. B. Slimen, M. Mechteri, and H. Biallach. 2020. Root Cause Analysis of Noisy Neighbors in a Virtualized Infrastructure. In 2020 IEEE Wireless Communications and Networking Conference (WCNC). 2020 IEEE Wireless Communications and Networking Conference (WCNC). (May 2020), 1--6.
[8]
Damien Carver. 2019. Advanced consolidation for dynamic containers, 119.
[9]
Giuliano Casale, Stephan Kraft, and Diwakar Krishnamurthy. 2011. A Model of Storage I/O Performance Interference in Virtualized Systems. In 2011 31st International Conference on Distributed Computing Systems Workshops. 2011 31st International Conference on Distributed Computing Systems Workshops. (June 2011), 34--39.
[10]
[SW], eBPF for Windows Oct. 19, 2022. Microsoft. url: https://github.com/microsoft/ebpf-for-windowsRetrieved Oct. 19, 2022 from.
[11]
Mohamad Gebai and Michel R. Dagenais. 2018. Survey and Analysis of Kernel and Userspace Tracers on Linux: Design, Implementation, and Overhead. ACM Computing Surveys, 51, 2, (Mar. 12, 2018), 26:1--26:33.
[12]
Robert P. Goldberg. 1973. Architectural Principles for Virtual Computer Systems. HARVARD UNIV CAMBRIDGE MA DIV OF ENGINEERING AND APPLIED PHYSICS, (Feb. 1, 1973). Retrieved Feb. 8, 2022 from https://apps.dtic.mil/sti/citations/AD0772809.
[13]
Brendan Gregg. 2020. Systems Performance: Enterprise and the Cloud. (Second ed.). Addison-Wesley Professional Computing Series. Addison-Wesley, Boston. isbn: 978-0--13--682015--4.
[14]
Tejun Heo, J Weiner, V Davydov, L Thorvalds, P Parav, T Klauser, S Hallyn, and K Khlebnikov. 2015. Control group v2. Retrieved Aug. 30, 2022 from https://www.kernel.org/doc/Documentation/admin-guide/cgroup-v2.rst.
[15]
Jinho Hwang, Sai Zeng, Frederick y Wu, and Timothy Wood. 2013. A component-based performance comparison of four hypervisors. In 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013). 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013). (May 2013), 269--276.
[16]
C. Isci, J. Liu, B. Abali, J. O. Kephart, and J. Kouloheris. 2011. Improving server utilization using fast virtual machine migration. IBM Journal of Research and Development, 55, 6, (Nov. 2011), 4:1--4:12.
[17]
Jim Keniston, Ananth Mavinakayanahalli, Vara Prasad, and Prasanna Panchamukhi. 2007. Ptrace, Utrace, Uprobes: Lightweight, Dynamic Tracing of User Apps. In Proceedings of the 2007 Linux Symposium.
[18]
G. Khanna, K. Beaty, G. Kar, and A. Kochut. 2006. Application Performance Management in Virtualized Server Environments. In 2006 IEEE/IFIP Network Operations and Management Symposium NOMS 2006. 2006 IEEE/IFIP Network Operations and Management Symposium NOMS 2006. (Apr. 2006), 373--381.
[19]
Younggyun Koh, Rob Knauerhase, Paul Brett, Mic Bowman, Zhihua Wen, and Calton Pu. 2007. An Analysis of Performance Interference Effects in Virtual Environments. In 2007 IEEE International Symposium on Performance Analysis of Systems Software. 2007 IEEE International Symposium on Performance Analysis of Systems Software. (Apr. 2007), 200--209.
[20]
Samuel Kounev, Klaus-Dieter Lange, and Jóakim von Kistowski. 2020. Systems Benchmarking: For Scientists and Engineers. Springer International Publishing, Cham.
[21]
Rouven Krebs, Christof Momm, and Samuel Kounev. 2014. Metrics and techniques for quantifying performance isolation in cloud environments. Science of Computer Programming, 90, (Sept. 2014), 116--134.
[22]
Young Choon Lee and Albert Y. Zomaya. 2012. Energy efficient utilization of resources in cloud computing systems. The Journal of Supercomputing, 60, 2, (May 1, 2012), 268--280.
[23]
Joshua Levin. 2020. ViperProbe: Using eBPF Metrics to Improve Microservice Observability.
[24]
Wes Lloyd, Shrideep Pallickara, Olaf David, Mazdak Arabi, and Ken Rojas. 2017. Mitigating Resource Contention and Heterogeneity in Public Clouds for Scientific Modeling Services. In 2017 IEEE International Conference on Cloud Engineering (IC2E). 2017 IEEE International Conference on Cloud Engineering (IC2E). (Apr. 2017), 159--166.
[25]
Clive Longbottom. 2017. The Evolution of Cloud Computing: How to Plan for Change. BCS Learning & Development Ltd, Swindon, UK. 181 pp. isbn: 978--1--78017--358-0.
[26]
Filipe Manco, Costin Lupu, Florian Schmidt, Jose Mendes, Simon Kuenzer, Sumit Sati, Kenichi Yasukata, Costin Raiciu, and Felipe Huici. 2017. My VM is Lighter (and Safer) than your Container. In Proceedings of the 26th Symposium on Operating Systems Principles. SOSP '17: ACM SIGOPS 26th Symposium on Operating Systems Principles. ACM, Shanghai China, (Oct. 14, 2017), 218--233. isbn: 978--1--4503--5085--3.
[27]
Mohammad Masdari, Sayyid Shahab Nabavi, and Vafa Ahmadi. 2016. An overview of virtual machine placement schemes in cloud computing. Journal of Network and Computer Applications, 66, (May 1, 2016), 106--127.
[28]
Jeanna Neefe Matthews, Wenjin Hu, Madhujith Hapuarachchi, Todd Deshane, Demetrios Dimatos, Gary Hamilton, Michael McCabe, and James Owens. 2007. Quantifying the performance isolation properties of virtualization systems. In Proceedings of the 2007 Workshop on Experimental Computer Science - ExpCS '07. The 2007 Workshop. ACM Press, San Diego, California, 6--es. isbn: 978--1--59593--751--3.
[29]
Steven McCanne and Van Jacobson. 1993. The BSD packet filter: a new architecture for user-level packet capture. In Proceedings of the USENIX Winter 1993 Conference Proceedings on USENIX Winter 1993 Conference Proceedings (USENIX'93). USENIX Association, USA, (Jan. 25, 1993), 2.
[30]
Nicolas Poggi. 2019. Microbenchmark. In Encyclopedia of Big Data Technologies. Sherif Sakr and Albert Y. Zomaya, (Eds.) Springer International Publishing, Cham, 1143--1152. isbn: 978--3--319--77525--8. 11.
[31]
Xing Pu, Ling Liu, Yiduo Mei, Sankaran Sivathanu, Younggyun Koh, Calton Pu, and Yuanda Cao. 2013. Who Is Your Neighbor: Net I/O Performance Interference in Virtualized Clouds. IEEE Transactions on Services Computing, 6, 3, (July 2013), 314--329.
[32]
Fayruz Rahma, Teguh Bharata Adji, and Widyawan Widyawan. 2013. Scalability Analysis of KVM-Based Private Cloud For Iaas. International Journal of Cloud Computing and Services Science (IJ-CLOSER), 2, 4, (Oct. 13, 2013), 288--295, 4, (Oct. 13, 2013). Retrieved May 3, 2021 from http://www.iaesjournal.com/online/index.php/IJ-CLOSER/article/view/4535.
[33]
Rouven Krebs. 2015. Performance Isolation in Multi-Tenant Applications. Retrieved Aug. 7, 2022 from https://se.informatik.uni-wuerzburg.de/fileadmin/10030200/user_upload/dissKIT_BW.PDF.
[34]
Joel Scheuner, Philipp Leitner, Jürgen Cito, and Harald Gall. 2014. Cloud Work Bench -- Infrastructure-as-Code Based Cloud Benchmarking. In 2014 IEEE 6th International Conference on Cloud Computing Technology and Science. 2014 IEEE 6th International Conference on Cloud Computing Technology and Science. (Dec. 2014), 246--253.
[35]
Daniel Seybold. 2021. An Automation-Based Approach for Reproducible Evaluations of Distributed DBMS on Elastic Infrastructures. Ph.D. Dissertation. Universität Ulm, (May 14, 2021). isbn: 9781757899956.
[36]
Prateek Sharma, Lucas Chaufournier, Prashant Shenoy, and Y. C. Tay. 2016. Containers and Virtual Machines at Scale: A Comparative Study. In Proceedings of the 17th International Middleware Conference. Middleware '16: 17th International Middleware Conference. ACM, Trento Italy, (Nov. 28, 2016), 1--13. isbn: 978--1--4503--4300--8.
[37]
Stephen Soltesz, Herbert Pötzl, Marc E. Fiuczynski, Andy Bavier, and Larry Peterson. 2007. Container-based operating system virtualization: a scalable, high-performance alternative to hypervisors. ACM SIGOPS Operating Systems Review, 41, 3, (Mar. 21, 2007), 275--287.
[38]
Chunqiang Tang, Thawan Kooburat, Pradeep Venkatachalam, Akshay Chander, Zhe Wen, Aravind Narayanan, Patrick Dowell, and Robert Karl. 2015. Holistic configuration management at Facebook. In Proceedings of the 25th Symposium on Operating Systems Principles. SOSP '15: ACM SIGOPS 25th Symposium on Operating Systems Principles. ACM, Monterey California, (Oct. 4, 2015), 328--343. isbn: 978--1--4503--3834--9.
[39]
Xuehai Tang, Zhang Zhang, Min Wang, Yifang Wang, Qingqing Feng, and Jizhong Han. 2014. Performance Evaluation of Light-Weighted Virtualization for PaaS in Clouds. In Algorithms and Architectures for Parallel Processing (Lecture Notes in Computer Science). Xian-he Sun et al., (Eds.) Springer International Publishing, Cham, 415--428. isbn: 978--3--319--11197--1.
[40]
Fei Tao, Chen Li, T. Warren Liao, and Yuanjun Laili. 2016. BGM-BLA: A New Algorithm for Dynamic Migration of Virtual Machines in Cloud Computing. IEEE Transactions on Services Computing, 9, 6, (Nov. 2016), 910--925. vtt.
[41]
Avishay Traeger, Erez Zadok, Nikolai Joukov, and Charles P. Wright. 2008. A nine year study of file system and storage benchmarking. ACM Transactions on Storage, 4, 2, (May 11, 2008), 1--56.
[42]
Simon Volpert, Georg Eisenhart, and Jörg Domaschka. 2022. Are kubernetes cni solutions ready for> 10 gbit/s?
[43]
Zhiyuan Wan, David Lo, Xin Xia, and Liang Cai. 2019. Practical and Effective Sandboxing for Linux Containers, 41.
[44]
Xingyu Wang, Junzhao Du, and Hui Liu. 2022. Performance and isolation analysis of RunC, gVisor and Kata Containers runtimes. Cluster Computing, (Jan. 22, 2022).
[45]
Miguel G. Xavier, Israel C. De Oliveira, Fabio D. Rossi, Robson D. Dos Passos, Kassiano J. Matteussi, and Cesar A.F. De Rose. 2015. A Performance Isolation Analysis of Disk-Intensive Workloads on Container-Based Clouds. In 2015 23rd Euromicro International Conference on Parallel, Distributed, and Network- Based Processing. 2015 23rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing. (Mar. 2015), 253--260.
[46]
Pingpeng Yuan, Chong Ding, Long Cheng, Shengli Li, Hai Jin, and Wenzhi Cao. 2010. VITS Test Suit: A Micro-benchmark for Evaluating Performance Isolation of Virtualization Systems. In 2010 IEEE 7th International Conference on E-Business Engineering. 2010 IEEE 7th International Conference on E-Business Engineering. (Nov. 2010), 132--139.
[47]
Jiangtao Zhang, Zhixiang He, Hejiao Huang, Xuan Wang, Chonglin Gu, and Lingmin Zhang. 2014. SLA aware cost efficient virtual machines placement in cloud computing. In 2014 IEEE 33rd International Performance Computing and Communications Conference (IPCCC). 2014 IEEE 33rd International Performance Computing and Communications Conference (IPCCC). (Dec. 2014), 1--8.
Index Terms
- A Methodology and Framework to Determine the Isolation Capabilities of Virtualisation Technologies
Recommendations
Improving machine virtualisation with 'hotplug memory'
Machine virtualisation is a key technology for server consolidation and on-demand server provisioning. To support this trend, it is essential to improve the performance of virtualisation software and enable the efficient running of many virtual ...
Performance comparison of hardware virtualization platforms
NETWORKING'11: Proceedings of the 10th international IFIP TC 6 conference on Networking - Volume Part IHosting virtual servers on a shared physical hardware by means of hardware virtualization is common use at data centers, web hosters, and research facilities. All platforms include isolation techniques that restrict resource consumption of the virtual ...
An Empirical Analysis of Common OCI Runtimes' Performance Isolation Capabilities
ICPE '24: Proceedings of the 15th ACM/SPEC International Conference on Performance EngineeringIndustry and academia have strong incentives to adopt virtualization technologies. Such technologies can reduce the total cost of ownership or facilitate business models like cloud computing. These options have recently grown significantly with the rise ...
Comments
Information & Contributors
Information
Published In
April 2023
244 pages
ISBN:9798400700682
DOI:10.1145/3578244
- General Chairs:
- Marco Vieira,
- Valeria Cardellini,
- Program Chairs:
- Antinisca Di Marco,
- Petr Tuma
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 April 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Ministerium für Wirtschaft, Wissenschaft und Digitalisierung
Conference
ICPE '23
Sponsor:
ICPE '23: ACM/SPEC International Conference on Performance Engineering
April 15 - 19, 2023
Coimbra, Portugal
Acceptance Rates
ICPE '23 Paper Acceptance Rate 15 of 46 submissions, 33%;
Overall Acceptance Rate 252 of 851 submissions, 30%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 307Total Downloads
- Downloads (Last 12 months)228
- Downloads (Last 6 weeks)22
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in