Article

Multi-hypervisor virtual machines: enabling an ecosystem of hypervisor-level services

Authors:

Kartik Gopalan,

Daniel Williams,

Nilton BilaAuthors Info & Claims

USENIX ATC '17: Proceedings of the 2017 USENIX Conference on Usenix Annual Technical Conference

Pages 235 - 249

Published: 12 July 2017 Publication History

Abstract

Public cloud software marketplaces already offer users a wealth of choice in operating systems, database management systems, financial software, and virtual networking, all deployable and configurable at the click of a button. Unfortunately, this level of customization has not extended to emerging hypervisor-level services, partly because traditional virtual machines (VMs) are fully controlled by only one hypervisor at a time. Currently, a VM in a cloud platform cannot concurrently use hypervisor-level services from multiple third-parties in a compartmentalized manner. We propose the notion of a multi-hypervisorVM , which is an unmodified guest that can simultaneously use services from multiple coresident, but isolated, hypervisors. We present a new virtualization architecture, called Span virtualization, that leverages nesting to allow multiple hypervisors to concurrently control a guest's memory, virtual CPU, and I/O resources. Our prototype of Span virtualization on the KVM/QEMU platform enables a guest to use services such as introspection, network monitoring, guest mirroring, and hypervisor refresh, with performance comparable to traditional nested VMs.

References

[1]

iPerf: The Network Bandwidth Measurement Tool. http://iperf.fr/.

[2]

Ravello Systems. https://www.ravellosystems.com/.

[3]

Volatility Framework. http://www.volatilityfoundation.org/.

[4]

Xen Hypervisor. http://www.xen.org/.

[5]

AMD. AMD Virtualization (AMD-V). http://www.amd.com/en-us/solutions/servers/virtualization.

[6]

BARHAM, P., DRAGOVIC, B., FRASER, K., HAND, S., HARRIS, T., HO, A., NEUGEBAUER, R., PRATT, I., AND WARFIELD, A. Xen and the art of virtualization. In Proc. of SOSP (Bolton Landing, NY, USA, 2003), pp. 164-177.

[7]

BEHAM, M., VLAD, M., AND REISER, H. Intrusion detection and honeypots in nested virtualization environments. In Proc. of 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks(DSN) (Budapest, Hungary, June 2013).

[8]

BELPAIRE, G., AND HSU, N.-T. Formal properties of recursive virtual machine architectures. In Proc. of SOSP, Austin, Texas, USA (1975), pp. 89- 96.

[9]

BELPAIRE, G., AND HSU, N.-T. Hardware architecture for recursive virtual machines. In Proc. of Annual ACM Conference (1975), pp. 14-18.

[10]

BEN-YEHUDA, M., DAY, M. D., DUBITZKY, Z., FACTOR, M., HAR'EL, N., GORDON, A., LIGUORI, A., WASSERMAN, O., AND YASSOUR, B.-A. The Turtles project: Design and implementation of nested virtualization. In Proc. of Operating Systems Design and Implementation (2010).

[11]

BEN-YEHUDA, M., MASON, J., XENIDIS, J., KRIEGER, O., VAN DOORN, L., NAKAJIMA, J., MALLICK, A., AND WAHLIG, E. Utilizing IOMMUs for virtualization in Linux and Xen. In Ottawa Linux Symposium (July 2006).

[12]

BEN-YEHUDA, M., XENIDIS, J., OSTROWSKI, M., RISTER, K., BRUEMMER, A., AND VAN DOORN, L. The price of safety: Evaluating IOMMU performance. In Proc. of Ottawa Linux Symposium (July 2007).

[13]

BERSHAD, B. N., CHAMBERS, C., EGGERS, S., MAEDA, C., MCNAMEE, D., PARDYAK, P., SAVAGE, S., AND SIRER, E. G. SPIN : An extensible microkernel for application-specific operating system services. Proc. of ACM SIGOPS Operating Systems Review 29, 1 (1995), 74-77.

[14]

BHARTIYA, S. Best lightweight linux distros for 2017. https://www.linux.com/news/best-lightweight-linux-distros-2017.

[15]

BROMIUM. https://www.bromium.com.

[16]

BULPIN, J. Whatever happened to XenServer's Windsor architecture? https://xenserver.org/blog/entry/whatever-happened-to-xenserver-s-windsor-architecture.html.

[17]

BUTT, S., LAGAR-CAVILLA, H. A., SRIVASTAVA, A., AND GANAPATHY, V. Self-service cloud computing. In Proc. of ACM Conference on Computer and Communications Security(CCS) (Raleigh, NC, USA, 2012).

[18]

CANDEA, G., KAWAMOTO, S., FUJIKI, Y., FRIEDMAN, G., AND FOX, A. Microreboot-a technique for cheap recovery. In Proc. of 6th USENIX Symposium on Operating Systems Design and Implementation (OSDI) (San Francisco, CA, USA, 2004), vol. 4, pp. 31-44.

[19]

CHEN, H., CHEN, R., ZHANG, F., ZANG, B., AND YEW, P. Live updating operating systems using virtualization. In Proc. of ACM International Conference on Virtual Execution Environments (VEE) (Ottawa, Canada, June 2006).

[20]

CITRIX. XenDesktop. https://www.citrix.com/products/xenapp-xendesktop/.

[21]

CITRIX. XenDesktop and The Evolution of Hardware-Assisted Server Technologies. https://www.citrix.com/content/dam/citrix/en_us/documents/go/2015-edition-hosted-desktop.pdf.

[22]

CLARK, C., FRASER, K., HAND, S., HANSEN, J., JUL, E., LIMPACH, C., PRATT, I., AND WARFIELD, A. Live migration of virtual machines. In Proc. of Network System Design and Implementation (2005).

[23]

COLP, P., NANAVATI, M., ZHU, J., AIELLO, W., COKER, G., DEEGAN, T., LOSCOCCO, P., AND WARFIELD, A. Breaking up is hard to do: Security and functionality in a commodity hypervisor. In Proc. of SOSP (2011), pp. 189-202.

[24]

CULLY, B., LEFEBVRE, G., MEYER, D., FEELEY, M., HUTCHINSON, N., AND WARFIELD, A. Remus: High availability via asynchronous virtual machine replication. In Proc. of Networked Systems Design and Implementation, San Francisco, CA, USA (2008).

[25]

DAS, B., ZHANG, Y. Y., AND KISZKA, J. Nested virtualization: State of the art and future directions. In KVM Forum (2014).

[26]

DINABURG, A., ROYAL, P., SHARIF, M., AND LEE, W. Ether: malware analysis via hardware virtualization extensions. In Proc. of 15th ACM conference on Computer and communications security (CCS) (Alexandria, VA, USA, 2008), pp. 51-62.

[27]

DUNLAP, G. W., KING, S. T., CINAR, S., BASRAI, M. A., AND CHEN, P. M. ReVirt: Enabling Intrusion Analysis through Virtual-Machine Logging and Replay. In Proc. of 5th USENIX Symposium on Operating Systems Design and Implementation (OSDI) (Boston, MA, Dec. 2002).

[28]

ENGLER, D. R., KAASHOEK, M. F., ET AL. Exokernel: An operating system architecture for application-level resource management. In ACM SIGOPS Operating Systems Review (1995), vol. 29(5), pp. 251-266.

[29]

FORD, B., HIBLER, M., LEPREAU, J., TULLMANN, P., BACK, G., AND CLAWSON, S. Microkernels Meet Recursive Virtual Machines. In Proc. OSDI, Seattle, Washington, USA (1996), pp. 137- 151.

[30]

GARFINKEL, T., AND ROSENBLUM, M. A Virtual Machine Introspection Based Architecture for Intrusion Detection. In Network & Distributed Systems Security Symposium (San Diego, CA USA, 2003).

[31]

GEBHARDT, C., AND DALTON, C. LaLa: A Late Launch Application. In Workshop on Scalable Trusted Computing, Chicago, Illinois, USA (2009), pp. 1-8.

[32]

GOLDBERG, R. P. Architecture of Virtual Machines. In Proceedings of the Workshop on Virtual Computer Systems, Cambridge, MA, USA (1973), pp. 74-112.

[33]

GOLDBERG, R. P. Survey of Virtual Machine Research. Computer 7, 6 (1974), 34-45.

[34]

GRAF, A., AND ROEDEL, J. Nesting the virtualized world. In Linux Plumbers Conference (Sept. 2009).

[35]

HAND, S., WARFIELD, A., FRASER, K., KOTSOVINOS, E., AND MAGENHEIMER, D. J. Are virtual machine monitors microkernels done right? In Proc. of HotOS (2005).

[36]

HINES, M., DESHPANDE, U., AND GOPALAN, K. Post-copy live migration of virtual machines. In SIGOPS Operating Systems Review (July 2009), 14-26.

[37]

INTEL CORP. Intel 64 and IA-32 Architecture Software Developers Manual, Volume 3, System Programming Guide. Order number 325384. April 2016.

[38]

IPSECS. Kernel Beast. http://core.ipsecs.com/rootkit/kernel-rootkit/.

[39]

JIANG, X., WANG, X., AND XU, D. Stealthy malware detection and monitoring through VMM-based "out-of-the-box" semantic view reconstruction. ACM Trans. Information Systems Security 13, 2 (Mar. 2010), 1-28.

[40]

KIVITY, A., KAMAY, Y., LAOR, D., LUBLIN, U., AND LIGUORI, A. KVM: the linux virtual machine monitor. In Proc. of Linux Symposium (June 2007).

[41]

KOLIVAS, C. Kernbench. http://ck.kolivas.org/apps/kernbench/.

[42]

KOURAI, K., AND CHIBA, S. Hyperspector: Virtual distributed monitoring environments for secure intrusion detection. In ACM/USENIX International Conference on Virtual Execution Environments (2005), pp. 197 - 207.

[43]

LE, M., AND TAMIR, Y. ReHype: enabling VM survival across hypervisor failures. In Proc. of ACM SIGPLAN Notices (2011), vol. 46, ACM, pp. 63- 74.

[44]

LIEDTKE, J. On micro-kernel construction. Proc. of ACM SIGOPS Operating Systems Review 29, 5 (1995), 237-250.

[45]

LOWELL, D. E., SAITO, Y., AND SAMBERG, E. J. Devirtualizable virtual machines enabling general, single-node, online maintenance. Proc. of SIGARCH Comput. Archit. News 32, 5 (Oct. 2004), 211-223.

[46]

MCAFEE. Root Out Rootkits: An Inside Look at McAfee Deep Defender. http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/mcafee-deep-defender-deepsafe-rootkit-protection-paper.pdf.

[47]

MURRAY, D. G., MILOS, G., AND HAND, S. Improving Xen Security Through Disaggregation. In Proc. of Virtual Execution Environments (2008), pp. 151-160.

[48]

OSISEK, D. L., JACKSON, K. M., AND GUM, P. H. Esa/390 interpretive-execution architecture, foundation for vm/esa. IBM Systems Journal 30, 1 (Feb. 1991), 34-51.

[49]

PAYNE, B. D., CARBONE, M., SHARIF, M., AND LEE, W. Lares: An Architecture for Secure Active Monitoring Using Virtualization. In Proc. of IEEE Symposium on Security and Privacy (2008), pp. 233 - 247.

[50]

PCI SIG. Single Root I/O Virtualization and Sharing. http://www.pcisig.com/specifications/iov/single_root/.

[51]

POPEK, G. J., AND GOLDBERG, R. P. Formal requirements for virtualizable third generation architectures. Proc. of Communications of ACM 17, 7 (July 1974), 412-421.

[52]

REAL TIME SYSTEMS GMBH. RTS Real-Time Hypervisor. http://www.real-time-systems.com/products/index.php.

[53]

RILEY, R., JIANG, X., AND XU, D. Guest-transparent prevention of kernel rootkits with VMM-based memory shadowing. In Proc. of Recent Advances in Intrusion Detection (2008), pp. 1- 20.

[54]

RUSSELL, R. Virtio: Towards a de-facto standard for virtual I/O devices. Proc. of SIGOPS Operating Systems Review 42, 5 (July 2008), 95-103.

[55]

RUTKOWSKA, J. Subverting vista kernel for fun and profit. In Blackhat (Las Vegas, USA, Aug. 2006).

[56]

SESHADRI, A., LUK, M., QU, N., AND PERRIG, A. SecVisor: a tiny hypervisor to provide lifetime kernel code integrity for commodity OSes. In Proc. of ACM SIGOPS Operating Systems Review (2007), vol. 41(6), pp. 335-350.

[57]

SHEN, Z., JIA, Q., SELA, G.-E., RAINERO, B., SONG, W., VAN RENESSE, R., AND WEATHERSPOON, H. Follow the Sun Through the Clouds: Application Migration for Geographically Shifting Workloads. In Proceedings of the Seventh ACM Symposium on Cloud Computing (2016), pp. 141- 154.

[58]

STEINBERG, U., AND KAUER, B. NOVA: A Microhypervisor-based Secure Virtualization Architecture. In Proc. of EuroSys, pp. 209-222.

[59]

SUGERMAN, J., VENKITACHALAM, G., AND LIM, B.-H. Virtualizing I/O Devices on VMware Workstation's Hosted Virtual Machine Monitor. In Proc. of USENIX Annual Technical Conference, Monterey, CA, USA (2002).

[60]

SUNEJA, S., ISCI, C., BALA, V., DE LARA, E., AND MUMMERT, T. Non-intrusive, Out-ofband and Out-of-the-box Systems Monitoring in the Cloud. In Proc. of SIGMETRICS'14, Austin, TX, USA (2014).

[61]

TOLDINAS, J., RUDZIKA, D., ŠTUIKYS, V., AND ZIBERKAS, G. Rootkit Detection Experiment within a Virtual Environment. Electronics and Electrical Engineering-Kaunas: Technologija, 8 (2009), 104.

[62]

WASSERMAN, O. Nested Virtualization: Shadow Turtles. In KVM Forum, Edinburgh, Spain (October 2013).

[63]

WHITAKER, A., COX, R., AND SHAW, M. Constructing services with interposable virtual hardware. In Proc. of First USENIX Symposium on Networked Systems Design and Implementation (San Francisco, California, 2004).

[64]

WIKIPEDIA. Phoenix Hyperspace. https://en.wikipedia.org/wiki/HyperSpace_(software).

[65]

WILLIAMS, D., HU, Y., DESHPANDE, U., SINHA, P. K., BILA, N., GOPALAN, K., AND JAMJOOM, H. Enabling efficient hypervisor-as-aservice clouds with ephemeral virtualization. In Proc. of the 12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE) (2016).

[66]

WILLIAMS, D., JAMJOOM, H., AND WEATHERSPOON, H. The Xen-Blanket: Virtualize once, run everywhere. In EuroSys, Bern, Switzerland (2012).

[67]

YASSOUR, B.-A., BEN-YEHUDA, M., AND WASSERMAN, O. Direct Device Assignment for Untrusted Fully-Virtualized Virtual Machines. Tech. rep., IBM Research, 2008.

[68]

ZHANG, F., CHEN, J., CHEN, H., AND ZANG, B. CloudVisor: Retrofitting Protection of Virtual Machines in Multi-tenant Cloud with Nested Virtualization. In Proc. of the Twenty-Third ACM Symposium on Operating Systems Principles (2011), ACM, pp. 203-216.

Cited By

Doddamani SSinha PLu HCheng TBagdi HGopalan KSartor JNaik MRossbach C(2019)Fast and live hypervisor replacementProceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3313808.3313821(45-58)Online publication date: 14-Apr-2019
https://dl.acm.org/doi/10.1145/3313808.3313821

Multi-hypervisor virtual machines: enabling an ecosystem of hypervisor-level services

Recommendations

Containers and Virtual Machines at Scale: A Comparative Study
Middleware '16: Proceedings of the 17th International Middleware Conference

Virtualization is used in data center and cloud environments to decouple applications from the hardware they run on. Hardware virtualization and operating system level virtualization are two prominent technologies that enable this. Containers, which use ...
Traffic-sensitive live migration of virtual machines
CCGRID '15: Proceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing

In this paper we address the problem of network contention between the migration traffic and the Virtual Machine (VM) application traffic for the live migration of co-located Virtual Machines. When VMs are migrated with pre-copy, they run at the source ...
Live gang migration of virtual machines
HPDC '11: Proceedings of the 20th international symposium on High performance distributed computing

This paper addresses the problem of simultaneously migrating a group of co-located and live virtual machines (VMs), i.e, VMs executing on the same physical machine. We refer to such a mass simultaneous migration of active VMs as "live gang migration". ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

USENIX ATC '17: Proceedings of the 2017 USENIX Conference on Usenix Annual Technical Conference

July 2017

811 pages

ISBN:9781931971386

Program Chairs:
Dilma Da Silva
Texas A&M University
,
Bryan Ford
École Polytechnique Fédérale de Lausanne

Sponsors

VMware
NetApp
Microsoft: Microsoft
Facebook: Facebook
ORACLE: ORACLE

Publisher

USENIX Association

United States

Publication History

Published: 12 July 2017

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Doddamani SSinha PLu HCheng TBagdi HGopalan KSartor JNaik MRossbach C(2019)Fast and live hypervisor replacementProceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3313808.3313821(45-58)Online publication date: 14-Apr-2019
https://dl.acm.org/doi/10.1145/3313808.3313821

View Options

View options

Media

Figures

Other

Tables

View Table of Contents