research-article

On the DMA mapping problem in direct device assignment

Authors:

Ben-Ami Yassour,

Muli Ben-Yehuda,

Orit WassermanAuthors Info & Claims

SYSTOR '10: Proceedings of the 3rd Annual Haifa Experimental Systems Conference

Article No.: 18, Pages 1 - 12

https://doi.org/10.1145/1815695.1815718

Published: 24 May 2010 Publication History

Abstract

I/O intensive workloads running in virtual machines can suffer massive performance degradation. Direct assignment of I/O devices to virtual machines is the best performing I/O virtualization mechanism, but its performance still remains far from the bare-metal (non-virtualized) case. The primary gap between direct assignment I/O performance and bare-metal I/O performance is the overhead of mapping the VM's memory pages for DMA in IOMMU translation tables. One could avoid this overhead by mapping all of the VM's pages for the lifetime of the VM, but this leads to memory consumption which is unacceptable in many scenarios.

The DMA mapping problem can be stated briefly as "when should a memory page be mapped or unmapped for DMA?" We begin by presenting a theoretical framework for reasoning about the DMA mapping problem. Then, using a quota-based approach, we propose the on-demand DMA mapping strategy, which provides the best DMA mapping performance for a given amount of memory consumed. In particular, on-demand mapping can achieve the same performance as state-of-the-art mapping strategies while consuming much less memory (exact amount depends on the workload's requirements). We present the design and implementation of on-demand mapping in the Linux-based KVM hypervisor and an experimental evaluation of its application to various workloads.

References

[1]

AMD I/O Virtualization Technology (IOMMU) Specification. http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/34434.pdf.

[2]

Intel Virtualization Technology for Directed I/O Architecture Specification. http://download.intel.com/technology/computing/vptech/Intel(r)_VT_for_Direct_IO.pdf.

[3]

D. Abramson, J. Jackson, S. Muthrasanallur, G. Neiger, G. Regnier, R. Sankaran, I. Schoinas, R. Uhlig, B. Vembu, and J. Wiegert. Intel virtualization technology for directed I/O. Intel Technology Journal, 10(03):179--192, August 2006.

[4]

K. Adams and O. Agesen. A comparison of software and hardware techniques for x86 virtualization. SIGOPS Oper. Syst. Rev., 40(5):2--13, December 2006.

Digital Library

[5]

P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In SOSP '03: 19th ACM Symposium on Operating Systems Principles.

Digital Library

[6]

L. A. Belady. A study of replacement algorithms for a virtual-storage computer. IBM Systems, (5):78--101, 1966.

Digital Library

[7]

M. Ben-Yehuda, J. Mason, J. Xenidis, O. Krieger, L. van Doorn, J. Nakajima, A. Mallick, and E. Wahlig. Utilizing IOMMUs for virtualization in Linux and Xen. In OLS '06: The 2006 Ottawa Linux Symposium, pages 71--86, July 2006.

[8]

M. Ben-Yehuda, J. Xenidis, M. Ostrowski, K. Rister, A. Bruemmer, and L. van Doorn. The price of safety: Evaluating IOMMU performance. In OLS '07: The 2007 Ottawa Linux Symposium, pages 9--20, July 2007.

[9]

R. Bhargava, B. Serebrin, F. Spadini, and S. Manne. Accelerating two-dimensional page walks for virtualized systems. In ASPLOS XIII: Proceedings of the 13th international conference on Architectural support for programming languages and operating systems, pages 26--35, New York, NY, USA, 2008. ACM.

Digital Library

[10]

A. Borodin and R. El-Yaniv. Online computation and competitive analysis. Cambridge University Press, 1998.

Digital Library

[11]

A. Borodin, S. Irani, P. Raghavan, and B. Schieber. Competitive paging with locality of reference. J. Comput. Syst. Sci., 50(2):244--258, 1995.

Digital Library

[12]

A. Borodin, P. Raghavan, S. Irani, and B. Schieber. Competitive paging with locality of reference. In STOC '91: Proceedings of the twenty-third annual ACM symposium on Theory of computing, pages 249--259, New York, NY, USA, 1991. ACM.

Digital Library

[13]

Y. Dong, Z. Yu, and G. Rose. SR-IOV networking in Xen: Architecture, design and implementation. In WIOV '08: The First Workshop on I/O Virtualization.

Digital Library

[14]

A. Fiat and Z. Rosen. Experimental studies of access graph based heuristics: beating the lru standard? In SODA '97: Proceedings of the eighth annual ACM-SIAM symposium on Discrete algorithms, pages 63--72, Philadelphia, PA, USA, 1997. Society for Industrial and Applied Mathematics.

Digital Library

[15]

K. Fraser, H. Steven, R. Neugebauer, I. Pratt, A. Warfield, and M. Williamson. Safe hardware access with the Xen virtual machine monitor. In 1st Workshop on Operating System and Architectural Support for the on demand IT InfraStructure (OASIS), 2004.

[16]

D. Gupta, S. Lee, M. Vrable, S. Savage, A. C. Snoeren, G. Varghese, G. M. Voelker, and A. Vahdat. Difference engine: Harnessing memory redundancy in virtual machines. In OSDI '08: 8th USENIX Symposium on Operating System Design and Implementation.

Digital Library

[17]

A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. KVM: the linux virtual machine monitor. In Ottawa Linux Symposium, pages 225--230, July 2007.

[18]

J. Levasseur, V. Uhlig, J. Stoess, and S. Götz. Unmodified device driver reuse and improved system dependability via virtual machines. In OSDI '04: 6th Symposium on Opearting Systems Design & Implementation, page 2, Berkeley, CA, USA.

Digital Library

[19]

J. Liu. Evaluating standard-based self-virtualizing devices: A performance study on 10 GbE NICs with SRIOV support. In IPDPS '10: IEEE International Parallel and Distributed Processing Symposium, 2010.

[20]

J. Liu and B. Abali. Virtualization polling engine (vpe): using dedicated cpu cores to accelerate i/o virtualization. In ICS '09: Proceedings of the 23rd international conference on Supercomputing, pages 225--234, New York, NY, USA, 2009. ACM.

Digital Library

[21]

J. Liu, W. Huang, B. Abali, and D. K. Panda. High performance VMM-bypass I/O in virtual machines. In USENIX '06 Annual Technical Conference, page 3.

Digital Library

[22]

D. Magenheimer, C. Mason, D. Mccracken, and K. Hackel. Paravirtualized paging. In WIOV '08: The First Workshop on I/O Virtualization.

Digital Library

[23]

H. Raj and K. Schwan. High performance and scalable I/O virtualization via self-virtualized devices. In HPDC '07, pages 179--188, 2007.

Digital Library

[24]

K. K. Ram, J. R. Santos, Y. Turner, A. L. Cox, and S. Rixner. Achieving 10Gbps using safe and transparent network interface virtualization. In VEE '09: 2009 Conference on Virtual Execution Environments.

Digital Library

[25]

R. Russell. virtio: towards a de-facto standard for virtual I/O devices. SIGOPS Oper. Syst. Rev., 42(5):95--103, 2008.

Digital Library

[26]

J. R. Santos, Y. Turner, j. G. Janakiraman, and I. Pratt. Bridging the gap between software and hardware techniques for I/O virtualization. In USENIX Annual Technical Conference, pages 29--42, June 2008.

Digital Library

[27]

J. Satran, L. Shalev, M. Ben-Yehuda, and Z. Machulsky. Scalable I/O---a well-architected way to do scalable, secure and virtualized I/O. In WIOV '08: The First Workshop on I/O Virtualization, 2008.

Digital Library

[28]

L. Shalev, J. Satran, E. Borovik, and M. Ben-Yehuda. Isostack---highly efficient network processing on dedicated cores. In USENIX ATC '10: USENIX Annual Technical Conference, 2010.

Digital Library

[29]

J. Sugerman, G. Venkitachalam, and B.-H. Lim. Virtualizing I/O devices on VMware workstation's hosted virtual machine monitor. In 2002 USENIX Annual Technical Conference, pages 1--14.

Digital Library

[30]

C. A. Waldspurger. Memory resource management in VMware ESX server. In OSDI '02: 5th Symposium on Operating System Design and Implementation.

Digital Library

[31]

P. Willmann, S. Rixner, and A. L. Cox. Protection strategies for direct access to virtualized I/O devices. In aUSENIX Annual Technical Conference, 2008, pages 15--28.

Digital Library

[32]

P. Willmann, J. Shafer, D. Carr, A. Menon, S. Rixner, A. L. Cox, and W. Zwaenepoel. Concurrent direct network access for virtual machine monitors. In High Performance Computer Architecture, pages 306--317, 2007.

Digital Library

[33]

P. Wyckoff and J. Wu. Memory registration caching correctness. In CCGRID '05, pages 1008--1015.

Digital Library

[34]

B.-A. Yassour, M. Ben-Yehuda, and O. Wasserman. Direct device assignment for untrusted fully-virtualized virtual machines. Technical report, IBM Research Report H-0263, 2008.

Cited By

Dong YMi Z(2024)IOGuard: Software-Based I/O Page Fault Handling with One CPU CoreProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671394(337-346)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671394
Yuan YWang RRanganathan NRao NKumar SLantz PSanjeepan VCabrera JKwatra ASankaran RJeong IKim N(2024)Intel Accelerators Ecosystem: An SoC-Oriented Perspective : Industry Product2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)10.1109/ISCA59077.2024.00066(848-862)Online publication date: 29-Jun-2024
https://doi.org/10.1109/ISCA59077.2024.00066
Yan BZhu JJiang B(2023)Limon: A Scalable and Stable Key-Value Engine for Fast NVMe DevicesIEEE Transactions on Computers10.1109/TC.2023.328367472:10(3017-3028)Online publication date: Oct-2023
https://doi.org/10.1109/TC.2023.3283674
Show More Cited By

Index Terms

On the DMA mapping problem in direct device assignment

Recommendations

ELI: bare-metal performance for I/O virtualization
ASPLOS XVII: Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems

Direct device assignment enhances the performance of guest virtual machines by allowing them to communicate with I/O devices without host involvement. But even with device assignment, guests are still unable to approach bare-metal performance, because ...
ELI: bare-metal performance for I/O virtualization
ASPLOS '12

Direct device assignment enhances the performance of guest virtual machines by allowing them to communicate with I/O devices without host involvement. But even with device assignment, guests are still unable to approach bare-metal performance, because ...
ELI: bare-metal performance for I/O virtualization
ASPLOS '12

Direct device assignment enhances the performance of guest virtual machines by allowing them to communicate with I/O devices without host involvement. But even with device assignment, guests are still unable to approach bare-metal performance, because ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SYSTOR '10: Proceedings of the 3rd Annual Haifa Experimental Systems Conference

May 2010

211 pages

ISBN:9781605589084

DOI:10.1145/1815695

Conference Chair:
Gadi Haber
IBM
,
Program Chairs:
Dilma M. Da Silva
IBM
,
Ethan L. Miller
UCSC

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 May 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SYSTOR '10

SYSTOR '10: The 3rd Annual Haifa Experimental Systems Conference

May 24 - 26, 2010

Haifa, Israel

Acceptance Rates

Overall Acceptance Rate 108 of 323 submissions, 33%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
459
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)7

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dong YMi Z(2024)IOGuard: Software-Based I/O Page Fault Handling with One CPU CoreProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671394(337-346)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671394
Yuan YWang RRanganathan NRao NKumar SLantz PSanjeepan VCabrera JKwatra ASankaran RJeong IKim N(2024)Intel Accelerators Ecosystem: An SoC-Oriented Perspective : Industry Product2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)10.1109/ISCA59077.2024.00066(848-862)Online publication date: 29-Jun-2024
https://doi.org/10.1109/ISCA59077.2024.00066
Yan BZhu JJiang B(2023)Limon: A Scalable and Stable Key-Value Engine for Fast NVMe DevicesIEEE Transactions on Computers10.1109/TC.2023.328367472:10(3017-3028)Online publication date: Oct-2023
https://doi.org/10.1109/TC.2023.3283674
Lv CZhang FGao XZhu C(2022)LA-vIOMMU: An Efficient Hardware-Software Co-design of IOMMU Virtualization2022 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SocialCom-SustainCom57177.2022.00038(246-253)Online publication date: Dec-2022
https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom57177.2022.00038
Tian KZhang YKang LZhao YDong YGavrilovska AZadok E(2020)coIOMMUProceedings of the 2020 USENIX Conference on Usenix Annual Technical Conference10.5555/3489146.3489178(479-492)Online publication date: 15-Jul-2020
https://dl.acm.org/doi/10.5555/3489146.3489178
Lesokhin IEran HRaindel SShapiro GGrimberg SLiss LBen-Yehuda MAmit NTsafrir D(2017)Page Fault Support for Network ControllersACM SIGARCH Computer Architecture News10.1145/3093337.303771045:1(449-466)Online publication date: 4-Apr-2017
https://dl.acm.org/doi/10.1145/3093337.3037710
Lesokhin IEran HRaindel SShapiro GGrimberg SLiss LBen-Yehuda MAmit NTsafrir D(2017)Page Fault Support for Network ControllersACM SIGPLAN Notices10.1145/3093336.303771052:4(449-466)Online publication date: 4-Apr-2017
https://dl.acm.org/doi/10.1145/3093336.3037710
Lesokhin IEran HRaindel SShapiro GGrimberg SLiss LBen-Yehuda MAmit NTsafrir D(2017)Page Fault Support for Network ControllersACM SIGOPS Operating Systems Review10.1145/3093315.303771051:2(449-466)Online publication date: 4-Apr-2017
https://doi.org/10.1145/3093315.3037710
Zhu ZKim SRozhanski YHu YWitchel ESilberstein M(2017)Understanding The Security of Discrete GPUsProceedings of the General Purpose GPUs10.1145/3038228.3038233(1-11)Online publication date: 4-Feb-2017
https://dl.acm.org/doi/10.1145/3038228.3038233
Lesokhin IEran HRaindel SShapiro GGrimberg SLiss LBen-Yehuda MAmit NTsafrir DChen YTemam OCarter J(2017)Page Fault Support for Network ControllersProceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3037697.3037710(449-466)Online publication date: 4-Apr-2017
https://dl.acm.org/doi/10.1145/3037697.3037710
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents