Article

Utilizing the IOMMU scalably

Authors:

Benjamin Serebrin, and

Dan TsafrirAuthors Info & Claims

USENIX ATC '15: Proceedings of the 2015 USENIX Conference on Usenix Annual Technical Conference

July 2015

Pages 549 - 562

Published: 08 July 2015 Publication History

Abstract

IOMMUs provided by modern hardware allow the OS to enforce memory protection controls on the DMA operations of its I/O devices. An IOMMU translation management design must scalably handle frequent concurrent updates of IOMMU translations made by multiple cores, which occur in high throughput I/O workloads such as multi-Gb/s networking. Today, however, OSes experience performance meltdowns when using the IOMMU in such workloads.

This paper explores scalable IOMMU management designs and addresses the two main bottlenecks we find in current OSes: (1) assignment of I/O virtual addresses (IOVAs), and (2) management of the IOMMU's TLB.

We propose three approaches for scalable IOVA assignment: (1) dynamic identity mappings, which eschew IOVA allocation altogether, (2) allocating IOVAs using the kernel's kmalloc, and (3) per-core caching of IOVAs allocated by a globally-locked IOVA allocator. We further describe a scalable IOMMU TLB management scheme that is compatible with all these approaches.

Evaluation of our designs under Linux shows that (1) they achieve 88.5%-100% of the performance obtained without an IOMMU, (2) they achieve similar latency to that obtained without an IOMMU, (3) scalable IOVA allocation and dynamic identity mappings perform comparably, and (4) kmalloc provides a simple solution with high performance, but can suffer from unbounded page table blowup.

References

[1]

AKER, B. Memslap - load testing and benchmarking a server. http://docs.libmemcached.org/bin/memslap.html. libmemcached 1.1.0 documentation.

[2]

AMD INC. AMD IOMMU architectural specification, rev 2.00. http://developer.amd.com/wordpress/media/2012/10/488821.pdf, Mar 2011.

[3]

AMIT, N., BEN-YEHUDA, M., TSAFRIR, D., AND SCHUSTER, A. vIOMMU: efficient IOMMU emulation. In USENIX Annual Technical Conference (ATC) (2011), pp. 73-86.

[4]

APPLE COMPUTER. IOPCIFamily/IOPCIFamily- 196.3/vtd.c, Source Code File of Mac OS X. http://www.opensource.apple.com/source/IOPCIFamily/IOPCIFamily-196.3/vtd.c. (Accessed: Jan 2015).

[5]

APPLE INC. Thunderbolt device driver programming guide: Debugging VT-d I/O MMU virtualization. https://developer.apple.com/library/mac/documentation/HardwareDrivers/Conceptual/ThunderboltDevGuide/DebuggingThunderboltDrivers/DebuggingThunderboltDrivers.html, 2013. (Accessed: May 2014).

[6]

ARM HOLDINGS. ARM system memory management unit architecture specification -- SMMU architecture version 2.0. http://infocenter.arm.com/help/topic/com.arm.doc.ihi0062c/IHI0062C_system_mmu_architecture_specification.pdf, 2013.

[7]

BALL, T., BOUNIMOVA, E., COOK, B., LEVIN, V., LICHTENBERG, J., MCGARVEY, C., ONDRUSEK, B., RAJAMANI, S. K., AND USTUNER, A. Thorough static analysis of device drivers. In ACM EuroSys (2006), pp. 73-85.

[8]

BECHER, M., DORNSEIF, M., AND KLEIN, C. N. FireWire: all your memory are belong to us. In CanSecWest Applied Security Conference (2005).

[9]

BEN-YEHUDA, M., XENIDIS, J., OSTROWSKI, M., RISTER, K., BRUEMMER, A., AND VAN DOORN, L. The price of safety: Evaluating IOMMU performance. In Ottawa Linux Symposium (OLS) (2007), pp. 9-20.

[10]

BONWICK, J. The Slab allocator: An object-caching kernel memory allocator. In USENIX Summer Annual Technical Conference (1994), pp. 87-98.

[11]

BONWICK, J., AND ADAMS, J. Magazines and Vmem: Extending the Slab allocator to many CPUs and arbitrary resources. In USENIX Annual Technical Conference (ATC) (2001), pp. 15-44.

[12]

BOTTOMLEY, J. E. Dynamic DMA mapping using the generic device. https://www.kernel.org/doc/Documentation/DMA-API.txt. Linux kernel documentation.

[13]

CARRIER, B. D., AND GRAND, J. A hardware-based memory acquisition procedure for digital investigations. Digital Investigation 1, 1 (Feb 2014), 50-60.

[14]

CASCARDO, T. DMA API performance and contention on IOMMU enabled environments. http://events.linuxfoundation.org/images/stories/slides/lfcs2013_cascardo.pdf, 2013.

[15]

CHOU, A., YANG, J., CHELF, B., HALLEM, S., AND ENGLER, D. An empirical study of operating systems errors. In ACM Symposium on Operating Systems Principles (SOSP) (2001), pp. 73-88.

[16]

CLEMENTS, A. T., KAASHOEK, M. F., AND ZELDOVICH, N. Scalable Address Spaces Using RCU Balanced Trees. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (2012), pp. 199-210.

[17]

CLEMENTS, A. T., KAASHOEK, M. F., AND ZELDOVICH, N. RadixVM: Scalable Address Spaces for Multithreaded Applications. In ACM EuroSys (2013), pp. 211-224.

[18]

CRISWELL, J., GEOFFRAY, N., AND ADVE, V. Memory safety for low-level software/hardware interactions. In USENIX Security Symposium (2009), pp. 83-100.

[19]

FITZPATRICK, B. Distributed caching with memcached. Linux Journal 2004, 124 (Aug 2004).

[20]

FREEBSD FOUNDATION. ×86/iommu/intel gas.c, source code file of FreeBSD 10.1.0. https://github.com/freebsd/freebsd/blob/release/10.1.0/sys/×86/iommu/intel_gas.c. (Accessed: Jan 2015).

[21]

GARRETT D'AMORE. i86pc/io/immu dvma.c, Source Code File of illumos. https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/i86pc/io/immu_dvma.c. (Accessed: Jan 2015).

[22]

HERDER, J. N., BOS, H., GRAS, B., HOMBURG, P., AND TANENBAUM, A. S. Failure resilience for device drivers. In IEEE/IFIP Annual International Conference on Dependable Systems and Networks (DSN) (2007), pp. 41-50.

[23]

HILL, B. Integrating an EDK custom peripheral with a LocalLink interface into Linux. Tech. Rep. XAPP1129 (v1.0), XILINX, May 2009.

[24]

IBM CORPORATION. PowerLinux servers -- 64-bit DMA concepts. http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liabm/liabmconcepts.htm. (Accessed: May 2014).

[25]

IBM CORPORATION. AIX kernel extensions and device support programming concepts. http://public.dhe.ibm.com/systems/power/docs/aix/71/kernextc_pdf.pdf, 2013. (Accssed: May 2015).

[26]

INTEL CORPORATION. Intel 64 and IA-32 Architectures Software Developers Manual, Volume 3: System Programming Guide. http://download.intel.com/products/processor/manual/325384.pdf, 2013.

[27]

INTEL CORPORATION. Intel Virtualization Technology for Directed I/O, Architecture Specification - Architecture Specification - Rev. 2.3. http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf, Oct 2014.

[28]

JONES, R. A. A network performance benchmark (revision 2.0). Tech. rep., Hewlett Packard, 1995. http://www.netperf.org/netperf/training/Netperf.html.

[29]

KADAV, A., RENZELMANN, M. J., AND SWIFT, M. M. Tolerating hardware device failures in software. In ACM Symposium on Operating Systems Principles (SOSP) (2009), pp. 59-72.

[30]

LEITAO, B. H. Tuning 10Gb network cards on Linux. In Ottawa Linux Symposium (OLS) (2009), pp. 169-189.

[31]

LEVASSEUR, J., UHLIG, V., STOESS, J., AND GÖTZ, S. Unmodified device driver reuse and improved system dependability via virtual machines. In USENIX Symposium on Operating System Design and Implementation (OSDI) (2004), pp. 17-30.

[32]

Documentation/intel-iommu.txt, Linux 3.18 documentation file. https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt. (Accessed: Jan 2015).

[33]

LIU, R., ZHANG, H., AND CHEN, H. Scalable Read-mostly Synchronization Using Passive Reader-Writer Locks. In USENIX Annual Technical Conference (ATC) (2014), pp. 219-230.

[34]

MALKA, M., AMIT, N., BEN-YEHUDA, M., AND TSAFRIR, D. rIOMMU: Efficient IOMMU for I/O devices that employ ring buffers. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (2015), pp. 355-368.

[35]

MALKA, M., AMIT, N., AND TSAFRIR, D. Efficient Intra-Operating System Protection Against Harmful DMAs. In USENIX Conference on File and Storage Technologies (FAST) (2015), pp. 29-44.

[36]

MAMTANI, V. DMA directions and Windows. http://download.microsoft.com/download/a/f/d/afdfd50d-6eb9-425e-84e1-b4085a80e34e/sys-t304_wh07.pptx, 2007. (Accessed: May 2014).

[37]

MILLER, D. S., HENDERSON, R., AND JELINEK, J. Dynamic DMA mapping guide. https://www.kernel.org/doc/Documentation/DMA-API-HOWTO.txt. Linux kernel documentation.

[38]

PCI-SIG. Address translation services revision 1.1. https://www.pcisig.com/specifications/iov/ats, Jan 2009.

[39]

ROEDEL, J. IOMMU Page Faulting and MM Integration. http://www.linuxplumbersconf.org/2014/ocw/system/presentations/2253/original/iommuv2.pdf. Linux Plumbers Conference 2014.

[40]

SESHADRI, A., LUK, M., QU, N., AND PERRIG, A. SecVisor: A tiny hypervisor to provide lifetime kernel code integrity for commodity OSes. In ACM Symposium on Operating Systems Principles (SOSP) (2007), pp. 335-350.

[41]

SWIFT, M., BERSHAD, B., AND LEVY, H. Improving the reliability of commodity operating systems. ACM Transactions on Computer Systems (TOCS) 23, 1 (Feb 2005), 77-110.

[42]

TOMONORI, F. DMA representations sg_table vs. sg_ring IOMMUs and LLD's restrictions. In USENIX Linux Storage and Filesystem Workshop (LSF) (2008). https://www.usenix.org/legacy/events/lsf08/tech/IO_tomonori.pdf.

[43]

WALDSPURGER, C., AND ROSENBLUM, M. I/O virtualization. Communications of the ACM (CACM) 55, 1 (Jan 2012), 66-73.

[44]

WILLIAMS, D., REYNOLDS, P., WALSH, K., SIRER, E. G., AND SCHNEIDER, F. B. Device driver safety through a reference validation mechanism. In USENIX Symposium on Operating System Design and Implementation (OSDI) (2008), pp. 241-254.

[45]

WILLMANN, P., RIXNER, S., AND COX, A. L. Protection strategies for direct access to virtualized I/O devices. In USENIX Annual Technical Conference (ATC) (2008), pp. 15-28.

[46]

WOJTCZUK, R. Subverting the Xen hypervisor. In Black Hat (2008). http://www.blackhat.com/presentations/bh-usa-08/Wojtczuk/BH_US_08_Wojtczuk_Subverting_the_Xen_Hypervisor.pdf. (Accessed: May 2014).

[47]

YASSOUR, B.-A., BEN-YEHUDA, M., AND WASSERMAN, O. On the DMA mapping problem in direct device assignment. In ACM International Systems and Storage Conference (SYSTOR) (2010), pp. 18:1-18:12.

Cited By

Tootoonchian APanda ALan CWalls MArgyraki KRatnasamy SShenker SSeshan SBanerjee S(2018)ResQProceedings of the 15th USENIX Conference on Networked Systems Design and Implementation10.5555/3307441.3307466(283-297)Online publication date: 9-Apr-2018
https://dl.acm.org/doi/10.5555/3307441.3307466
Markuze ASmolyar IMorrison ATsafrir D(2018)DAMNACM SIGPLAN Notices10.1145/3296957.317317553:2(301-315)Online publication date: 19-Mar-2018
https://dl.acm.org/doi/10.1145/3296957.3173175
Neugebauer RAntichi GZazo JAudzevich YLópez-Buedo SMoore AGorinsky STapolcai J(2018)Understanding PCIe performance for end host networkingProceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication10.1145/3230543.3230560(327-341)Online publication date: 7-Aug-2018
https://dl.acm.org/doi/10.1145/3230543.3230560
Show More Cited By

Index Terms

Utilizing the IOMMU scalably
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance
2. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Environment-specific retrieval

Index terms have been assigned to the content through auto-classification.

Recommendations

IOMMU: strategies for mitigating the IOTLB bottleneck
ISCA'10: Proceedings of the 2010 international conference on Computer Architecture

The input/output memory management unit (IOMMU) was recently introduced into mainstream computer architecture when both Intel and AMD added IOMMUs to their chip-sets. An IOMMU provides memory protection from I/O devices by enabling system software to ...
Read More
True IOMMU Protection from DMA Attacks: When Copy is Faster than Zero Copy
ASPLOS '16

Malicious I/O devices might compromise the OS using DMAs. The OS therefore utilizes the IOMMU to map and unmap every target buffer right before and after its DMA is processed, thereby restricting DMAs to their designated locations. This usage model, ...
Read More
True IOMMU Protection from DMA Attacks: When Copy is Faster than Zero Copy
ASPLOS'16

Malicious I/O devices might compromise the OS using DMAs. The OS therefore utilizes the IOMMU to map and unmap every target buffer right before and after its DMA is processed, thereby restricting DMAs to their designated locations. This usage model, ...
Read More

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

USENIX ATC '15: Proceedings of the 2015 USENIX Conference on Usenix Annual Technical Conference

July 2015

625 pages

ISBN:9781931971225

Program Chairs:
Shan Lu
University of Chicago
,
Erik Riedel
EMC

Sponsors

VMware
NetApp
Google Inc.
Facebook: Facebook
HP: HP

Publisher

USENIX Association

United States

Publication History

Published: 08 July 2015

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

Tootoonchian APanda ALan CWalls MArgyraki KRatnasamy SShenker SSeshan SBanerjee S(2018)ResQProceedings of the 15th USENIX Conference on Networked Systems Design and Implementation10.5555/3307441.3307466(283-297)Online publication date: 9-Apr-2018
https://dl.acm.org/doi/10.5555/3307441.3307466
Markuze ASmolyar IMorrison ATsafrir D(2018)DAMNACM SIGPLAN Notices10.1145/3296957.317317553:2(301-315)Online publication date: 19-Mar-2018
https://dl.acm.org/doi/10.1145/3296957.3173175
Neugebauer RAntichi GZazo JAudzevich YLópez-Buedo SMoore AGorinsky STapolcai J(2018)Understanding PCIe performance for end host networkingProceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication10.1145/3230543.3230560(327-341)Online publication date: 7-Aug-2018
https://dl.acm.org/doi/10.1145/3230543.3230560
Markuze ASmolyar IMorrison ATsafrir DShen XTuck JBianchini RSarkar V(2018)DAMNProceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3173162.3173175(301-315)Online publication date: 19-Mar-2018
https://dl.acm.org/doi/10.1145/3173162.3173175
Vogel PKurth AWeinbuch JMarongiu ABenini L(2017)Efficient Virtual Memory Sharing via On-Accelerator Page Table Walking in Heterogeneous Embedded SoCsACM Transactions on Embedded Computing Systems10.1145/312656016:5s(1-19)Online publication date: 27-Sep-2017
https://dl.acm.org/doi/10.1145/3126560

View Options

View options

Media

Figures

Other

Tables

View Table of Contents