Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3230543.3230560acmconferencesArticle/Chapter ViewAbstractPublication PagescommConference Proceedingsconference-collections
research-article
Free access

Understanding PCIe performance for end host networking

Published: 07 August 2018 Publication History
  • Get Citation Alerts
  • Abstract

    In recent years, spurred on by the development and availability of programmable NICs, end hosts have increasingly become the enforcement point for core network functions such as load balancing, congestion control, and application specific network offloads. However, implementing custom designs on programmable NICs is not easy: many potential bottlenecks can impact performance.
    This paper focuses on the performance implication of PCIe, the de-facto I/O interconnect in contemporary servers, when interacting with the host architecture and device drivers. We present a theoretical model for PCIe and pcie-bench, an open-source suite, that allows developers to gain an accurate and deep understanding of the PCIe substrate. Using pcie-bench, we characterize the PCIe subsystem in modern servers. We highlight surprising differences in PCIe implementations, evaluate the undesirable impact of PCIe features such as IOMMUs, and show the practical limits for common network cards operating at 40Gb/s and beyond. Furthermore, through pcie-bench we gained insights which guided software and future hardware architectures for both commercial and research oriented network cards and DMA engines.

    References

    [1]
    Hitesh Ballani, Paolo Costa, Christos Gkantsidis, Matthew P. Grosvenor, Thomas Karagiannis, Lazaros Koromilas, and Greg O'Shea. 2015. Enabling End-Host Network Functions. In Special Interest Group on Data Communication (SIGCOMM). ACM.
    [2]
    Hitesh Ballani, Paolo Costa, Christos Gkantsidis, Matthew P. Grosvenor, Thomas Karagiannis, Lazaros Koromilas, and Greg O'Shea. 2015. Enabling End-Host Network Functions. In Special Interest Group on Data Communication (SIGCOMM). ACM.
    [3]
    Nicola Bonelli, Andrea Di Pietro, Stefano Giordano, and Gregorio Procissi. 2012. On multi-gigabit packet capturing with multi-core commodity hardware. In Passive and Active Measurement (PAM). Springer.
    [4]
    Ryan Karl Braithwaite. 2013. NUMA data-access bandwidth characterization and modeling. Master's thesis. Virginia Polytechnic Institute and State University, US.
    [5]
    Aaron B. Brown and Margo I. Seltzer. 1997. Operating system bench-marking in the wake of Lmbench: a case study of the performance of NetBSD on the Intel x86 architecture. In Special Interest Group for the Computer Systems Performance Evaluation Community (SIGMETRICS). ACM.
    [6]
    Martin Casado, Teemu Koponen, Scott Shenker, and Amin Tootoonchian. 2012. Fabric: A Retrospective on Evolving SDN. In Hot Topics in Software Defined Networks (HotSDN). ACM.
    [7]
    Colin Dixon, Hardeep Uppal, Vjekoslav Brajkovic, Dane Brandon, Thomas Anderson, and Arvind Krishnamurthy. 2011. ETTM: A Scalable Fault Tolerant Network Manager. In Networked Systems Design and Implementation (NSDI). USENIX.
    [8]
    DPDK. 2018. Official website. http://www.dpdk.org.
    [9]
    Peter Druschel, Larry L. Peterson, and Bruce S. Davie. 1994. Experiences with a High-speed Network Adaptor: A Software Perspective. In Conference on Communications Architectures, Protocols and Applications (SIGCOMM). ACM.
    [10]
    Yasuhiro Endo, Zheng Wang, J. Bradley Chen, and Margo Seltzer. 1996. Using latency to evaluate interactive system performance. In Symposium on Operating Systems Design and Implementation (OSDI). ACM.
    [11]
    Exablaze. 2018. Official website. http://www.exablaze.com.
    [12]
    Daniel Firestone. 2018. Building Hardware-Accelerated Networks at Scale in the Cloud. https://conferences.sigcomm.org/sigcomm/2017/files/program-kbnets/keynote-2.pdf.
    [13]
    Daniel Firestone, Andrew Putnam, Sambhrama Mundkur, Derek Chiou, Alireza Dabagh, Mark Andrewartha, Hari Angepat, Vivek Bhanu, Adrian Caulfield, Eric Chung, Harish K. Chandrappa, Somesh Chaturmohta, Matt Humphrey, Jack Lavier, Norman Lam, Fengfen Liu, Kalin Ovtcharov, Jitu Padhye, Gautham Popuri, Shachar Raindel, Tejas Sapre, Mark Shaw, Gabriel Silva, Madhan Sivakumar, Nisheeth Srivastava, Anshuman Verma, Qasim Zuhair, Deepak Bansal, Doug Burger, Kushagra Vaid, David A. Maltz, and Albert Greenberg. 2018. Azure Accelerated Networking: SmartNICs in the Public Cloud. In Networked Systems Design and Implementation (NSDI). USENIX.
    [14]
    Brandan Greg. 2018. Linux Performance. http://www.brendangregg.com/linuxperf.html.
    [15]
    Prabhat K. Gupta. 2018. Xeon+FPGA Platform for the Data Center. https://www.ece.cmu.edu/~calcm/carl/lib/exe/fetch.php?media=carl15-gupta.pdf.
    [16]
    Sangjin Han, Keon Jang, Aurojit Panda, Shoumik Palkar, Dongsu Han, and Sylvia Ratnasamy. 2015. SoftNIC: A Software NIC to Augment Hardware. In Technical Report No. UCB/EECS-2015-155. University of California at Berkeley.
    [17]
    Sangjin Han, Keon Jang, KyoungSoo Park, and Sue Moon. 2010. Packet-Shader: A GPU-accelerated Software Router. In Special Interest Group on Data Communication (SIGCOMM). ACM.
    [18]
    Intel. 2014. Xeon Phi Coprocessor System Software Developers Guide. https://software.intel.com/sites/default/files/managed/09/07/xeon-phi-coprocessor-system-software-developers-guide.pdf.
    [19]
    Intel. 2018. 82599 10 GbE Controller Datasheet. https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82599--10-gbe-controller-datasheet.pdf.
    [20]
    Intel. 2018. Data Direct I/O technology (Intel DDIO): a primer. http://www.intel.co.uk/content/dam/www/public/us/en/documents/technology-briefs/data-direct-i-o-technology-brief.pdf.
    [21]
    Intel. 2018. Intel VTune Amplifier 2017. https://software.intel.com/en-us/intel-vtune-amplifier-xe.
    [22]
    Intel. 2018. Virtualization technology for directed I/O. http://www.intel.co.uk/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf.
    [23]
    Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. 2017. In-Datacenter Performance Analysis of a Tensor Processing Unit. In International Symposium on Computer Architecture (ISCA). ACM/IEEE.
    [24]
    Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. Design Guidelines for High Performance RDMA Systems. In Annual Technical Conference (ATC). USENIX.
    [25]
    Thomas Karagiannis, Richard Mortier, and Antony Rowstron. 2008. Network Exception Handlers: Host-network Control in Enterprise Networks. In Special Interest Group on Data Communication (SIGCOMM). ACM.
    [26]
    Antoine Kaufmann, SImon Peter, Naveen Kr. Sharma, Thomas Anderson, and Arvind Krishnamurthy. 2016. High Performance Packet Processing with FlexNIC. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM.
    [27]
    Matthew J. Koop, Wei Huang, Karthik Gopalakrishnan, and Dhabaleswar K. Panda. 2008. Performance analysis and evaluation of PCIe 2.0 and quad-data rate InfiniBand. In Symposium on High Performance Interconnects (HOTI). IEEE.
    [28]
    Christoph Lameter. 2013. NUMA (Non-Uniform Memory Access): an overview. In acmqueue. ACM.
    [29]
    Yanfang Le, Hyunseok Chang, Sarit Mukherjee, Limin Wang, Aditya Akella, Michael Swift, and T.V. Lakshman. 2017. UNO: Unifying Host and Smart NIC Offload for Flexible Packet Processing. In Symposium on Cloud Computing (SoCC). ACM.
    [30]
    Ki Suh Lee, Han Wang, and Hakim Weatherspoon. 2013. SoNIC: Precise Realtime Software Access and Control of Wired Networks. In Networked Systems Design and Implementation (NSDI). USENIX Association.
    [31]
    Bojie Li, Zhenyuan Ruan, Wencong Xiao, Yuanwei Lu, Yongqiang Xiong, Andrew Putnam, Enhong Chen, and Lintao Zhang. 2017. KV-Direct: High-Performance In-Memory Key-Value Store with Programmable NIC. In Symposium on Operating Systems Principles (SOSP). ACM.
    [32]
    Sheng Li, Hyeontaek Lim, Victor W. Lee, Jung Ho Ahn, Anuj Kalia, Michael Kaminsky, David G. Andersen, O. Seongil, Sukhan Lee, and Pradeep Dubey. 2015. Architecting to Achieve a Billion Requests Per Second Throughput on a Single Key-value Store Server Platform. In International Symposium on Computer Architecture (ISCA). ACM.
    [33]
    Tan Li, Yufei Ren, Dantong Yu, Shudong Jin, and Thomas Robertazzi. 2013. Characterization of Input/Output bandwidth performance models in NUMA architecture for data intensive applications. In International Conference on Parallel Processing (ICPP). IEEE.
    [34]
    Hyeontaek Lim, Dongsu Han, David G. Andersen, and Michael Kaminsky. 2014. MICA: A Holistic Approach to Fast In-Memory Key-Value Storage. In Networked Systems Design and Implementation (NSDI). USENIX.
    [35]
    Jiuxing Liu, Amith Mamidala, Abhinav Vishnn, and Dhabaleswar K. Panda. 2004. Performance evaluation of InfiniBand with PCI Express. In Symposium on High Performance Interconnects (HOTI). IEEE.
    [36]
    K. Lostrie, P. De Meulenaere, M. Temmerman, N. Van Remortel, and W. Beaumont. 2013. Benchmarking of PCIe-performance in microTCA-equipment. In International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC). IEEE.
    [37]
    Ilias Marinos, Robert Watson, and Mark Handley. 2014. Network Stack Specialization for Performance. In Special Interest Group on Data Communication (SIGCOMM). ACM.
    [38]
    Ilias Marinos, Robert N.M. Watson, Mark Handley, and Randall R. Stewart. 2017. Disk|Crypt|Net: Rethinking the Stack for High-performance Video Streaming. In Special Interest Group on Data Communication (SIGCOMM). ACM.
    [39]
    Larry McVoy and Carl Staelin. 1996. Lmbench: portable tools for performance analysis. In Annual Technical Conference (ATC). USENIX.
    [40]
    Mellanox. 2018. Programmable ConnectX-3 Pro Adapter Card. https://www.mellanox.com/related-docs/prod_adapter_cards/PB_Programmable_ConnectX-3_Pro_Card_VPI.pdf.
    [41]
    David J. Miller, Philip M. Watts, and Andrew W. Moore. 2009. Motivating future interconnects: a differential measurement analysis of PCI latency. In Symposium on Architectures for Networking and Communications Systems (ANCS). ACM.
    [42]
    L. Moll and M. Shand. 1997. Systems performance measurement on PCI pamette. In Symposium on FPGA-Based Custom Computing Machines (FCCM). IEEE.
    [43]
    NetFPGA. 2018. Official website. https://www.netfpga.org.
    [44]
    Netronome. 2018. NFP-4000 theory of operation. https://www.netronome.com/static/app/img/products/silicon-solutions/WP_NFP4000_TOO.pdf.
    [45]
    Netronome. 2018. Product Brief: NFP-6000 intelligent Ethernet controller family. https://www.netronome.com/static/app/img/products/silicon-solutions/PB_NFP6000.pdf.
    [46]
    ntop. 2018. PF_RING repository. https://github.com/ntop/PF_RING.git.
    [47]
    PCI-SIG. 2014. PCI Express Base Specification Revision 3.1.
    [48]
    Omer Peleg, Adam Morrison, Benjamin Serebrin, and Dan Tsafrir. 2015. Utilizing the IOMMU Scalably. In Annual Technical Conference (ATC). USENIX.
    [49]
    Ben Pfaff, Justin Pettit, Teemu Koponen, Keith Amidon, Martin Casado, and Scott Shenker. 2009. Extending Networking into the Virtualization Layer. In Hot Topics in Networks (HotNets). ACM.
    [50]
    Ian Pratt and Keir Fraser. 2001. Arsenic: A user-accessible gigabit ether-net interface. In International Conference on Computer Communications (INFOCOM). IEEE.
    [51]
    Andrew Putnam, Adrian M Caulfield, Eric S Chung, Derek Chiou, Kypros Constantinides, John Demme, Hadi Esmaeilzadeh, Jeremy Fowers, Gopi Prashanth Gopal, Jan Gray, et al. 2014. A reconfigurable fabric for accelerating large-scale datacenter services. In International Symposium on Computer Architecture (ISCA). ACM/IEEE.
    [52]
    Sivasankar Radhakrishnan, Yilong Geng, Vimalkumar Jeyakumar, Abdul Kabbani, George Porter, and Amin Vahdat. 2014. SENIC: Scalable NIC for End-host Rate Limiting. In Conference on Networked Systems Design and Implementation (NSDI). USENIX.
    [53]
    Kaushik Kumar Ram, Jayaram Mudigonda, Alan L. Cox, Scott Rixner, Parthasarathy Ranganathan, and Jose Renato Santos. 2010. sNICh: Efficient Last Hop Networking in the Data Center. In Architectures for Networking and Communications Systems (ANCS). ACM.
    [54]
    Luigi Rizzo. 2012. netmap: a novel framework for fast packet I/O. In Annual Technical Conference (ATC). USENIX.
    [55]
    Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C. Snoeren. 2015. Inside the Social Network's (Datacenter) Network. In Special Interest Group on Data Communication (SIGCOMM). ACM.
    [56]
    Alan Shieh, Srikanth Kandula, and Emin Gun Sirer. 2010. SideCar: Building Programmable Datacenter Networks Without Programmable Switches. In Hot Topics in Networks (HotNets). ACM.
    [57]
    Edward Solari and Brad Congdon. 2003. The Complete PCI Express Reference.
    [58]
    Robert Soulé, Shrutarshi Basu, Robert Kleinberg, Emin Gün Sirer, and Nate Foster. 2013. Managing the Network with Merlin. In Hot Topics in Networks (HotNets). ACM.
    [59]
    Gavin Stark and Sakir Sezer. 2013. NFP-6xxx - A 22nm high-performance network flow processor for 200Gb/s Software Defined Networking. In Symposium on High Performance Chips (Hot Chips). IEEE/ACM.
    [60]
    T. von Eicken, A. Basu, V. Buch, and W. Vogels. 1995. U-Net: A User-level Network Interface for Parallel and Distributed Computing. In Symposium on Operating Systems Principles (SOSP). ACM.
    [61]
    Jose Fernando Zazo, Sergio Lopez-Buedo, Yury Audzevich, and Andrew W. Moore. 2015. A PCIe DMA engine to support the virtualization of 40 Gbps FPGA-accelerated network appliances. In International Conference on Reconfigurable Computing and FPGAs (RECONFIG). IEEE.
    [62]
    Noa Zilberman, Yury Audzevich, Adam G. Covington, and Andrew W. Moore. 2014. NetFPGA SUME: toward 100 Gbps as research commodity. In Micro. IEEE.
    [63]
    Noa Zilberman, Matthew Grosvenor, Diana-Andreea Popescu, Neelakandan Manihatty-Bojan, Gianni Antichi, Marcin Wojcik, and Andrew W. Moore. 2017. Where Has My Time Gone?. In Passive and Active Measurement (PAM). Springer.

    Cited By

    View all
    • (2024)SepHash: A Write-Optimized Hash Index On Disaggregated Memory via Separate Segment StructureProceedings of the VLDB Endowment10.14778/3641204.364121817:5(1091-1104)Online publication date: 1-Jan-2024
    • (2024)Hoda: a High-performance Open vSwitch Dataplane with Multiple Specialized Data PathsProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3629564(82-98)Online publication date: 22-Apr-2024
    • (2024)TCCL: Discovering Better Communication Paths for PCIe GPU ClustersProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 310.1145/3620666.3651362(999-1015)Online publication date: 27-Apr-2024
    • Show More Cited By

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGCOMM '18: Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication
    August 2018
    604 pages
    ISBN:9781450355674
    DOI:10.1145/3230543
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 August 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    Author Tags

    1. PCIe
    2. operating system
    3. reconfigurable hardware

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    SIGCOMM '18
    Sponsor:
    SIGCOMM '18: ACM SIGCOMM 2018 Conference
    August 20 - 25, 2018
    Budapest, Hungary

    Acceptance Rates

    Overall Acceptance Rate 554 of 3,547 submissions, 16%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)1,583
    • Downloads (Last 6 weeks)201

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)SepHash: A Write-Optimized Hash Index On Disaggregated Memory via Separate Segment StructureProceedings of the VLDB Endowment10.14778/3641204.364121817:5(1091-1104)Online publication date: 1-Jan-2024
    • (2024)Hoda: a High-performance Open vSwitch Dataplane with Multiple Specialized Data PathsProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3629564(82-98)Online publication date: 22-Apr-2024
    • (2024)TCCL: Discovering Better Communication Paths for PCIe GPU ClustersProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 310.1145/3620666.3651362(999-1015)Online publication date: 27-Apr-2024
    • (2024)CC-NIC: a Cache-Coherent Interface to the NICProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 110.1145/3617232.3624868(52-68)Online publication date: 27-Apr-2024
    • (2024)Resource Critical Flow Monitoring in Software-Defined NetworksIEEE/ACM Transactions on Networking10.1109/TNET.2023.328669132:1(396-410)Online publication date: Feb-2024
    • (2024)P4Hauler: An Accelerator-Aware In-Network Load Balancer for Applications Performance BoostingIEEE Transactions on Cloud Computing10.1109/TCC.2024.338965812:2(697-711)Online publication date: Apr-2024
    • (2024)Unlocking Security to the Board: An Evaluation of SmartNIC-driven TLS Acceleration with kTLSNOMS 2024-2024 IEEE Network Operations and Management Symposium10.1109/NOMS59830.2024.10575053(1-9)Online publication date: 6-May-2024
    • (2024)E-Commerce Bot Traffic: In-Network Impact, Detection, and Mitigation2024 27th Conference on Innovation in Clouds, Internet and Networks (ICIN)10.1109/ICIN60470.2024.10494459(179-185)Online publication date: 11-Mar-2024
    • (2024)GRIT: Enhancing Multi-GPU Performance with Fine-Grained Dynamic Page Placement2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA)10.1109/HPCA57654.2024.00085(1080-1094)Online publication date: 2-Mar-2024
    • (2024)MINOS: Distributed Consistency and Persistency Protocol Implementation & Offloading to SmartNICs2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA)10.1109/HPCA57654.2024.00076(1-17)Online publication date: 2-Mar-2024
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media