research-article

Towards portable end-to-end network performance characterization of SmartNICs

Authors:

Hesam Tajbakhsh,

Antonio BarbalaceAuthors Info & Claims

APSys '22: Proceedings of the 13th ACM SIGOPS Asia-Pacific Workshop on Systems

Pages 46 - 52

https://doi.org/10.1145/3546591.3547528

Published: 30 August 2022 Publication History

Abstract

Emerging accelerated network devices, including SmartNICs (SNICs) with general-purpose CPUs - such as ARM, create new sources of queuing and processing delay in the hardware, network stack, and application. We therefore argue for the need of a framework to measure the application and network stack performance in SNIC-based host systems.

This paper describes our methodology to build up such framework focusing on the regular Linux network stack executed on either host, or SNIC CPUs. Our framework shows how SNICs impact end-to-end latency, throughput, and multi-core scalability based on the SNIC architecture, SNIC CPU performance, and even the applications themselves.

References

[1]

H. Agrawal, Y. Dutta, and S. Malik. 2012. Performance Analysis of Offloading IPsec Processing to Hardware Based Accelerators. In 2012 International Symposium on Electronic System Design (ISED). 291--294.

[2]

Broadcom. 2018 (accessed June 30, 2020). Stingray PS225.

[3]

Adrian Caulfield, Paolo Costa, and Manya Ghobadi. 2018. Beyond SmartNICs: Towards a Fully Programmable Cloud. In IEEE International Conference on High Performance Switching and Routing.

[4]

Cavium. (accessed June 30, 2020). LiquidIO-II 10/25G Smart NIC Family.

[5]

D. Chiou. 2017. The microsoft catapult project. In 2017 IEEE International Symposium on Workload Characterization (IISWC). 124--124.

[6]

Sean Choi, Muhammad Shahbaz, Balaji Prabhakar, and Mendel Rosenblum. 2019. λ-NIC: Interactive Serverless Compute on SmartNICs. In Proceedings of the ACM SIGCOMM 2019 Conference Posters and Demos (Beijing, China) (SIGCOMM Posters and Demos '19). Association for Computing Machinery, New York, NY, USA, 151--152.

Digital Library

[7]

Kevin Deierling. 2018 (accessed June 30, 2020). Defining SmartNIC: What is a SmartNIC and How to Choose the Best One.

[8]

Daniel Firestone, Andrew Putnam, Sambhrama Mundkur, Derek Chiou, Alireza Dabagh, Mike Andrewartha, Hari Angepat, Vivek Bhanu, Adrian Caulfield, Eric Chung, et al. 2018. Azure accelerated networking: SmartNICs in the public cloud. In 15th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 18). 51--66.

[9]

Michael Galles and Francis Matus. 2021. Pensando Distributed Services Architecture. IEEE Micro 41, 2 (2021), 43--49.

[10]

Scott Hendrickson, Stephen Sturdevant, Tyler Harter, Venkateshwaran Venkataramani, Andrea C Arpaci-Dusseau, and Remzi H Arpaci-Dusseau. 2016. Serverless computation with openlambda. In 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 16).

Digital Library

[11]

Yanfang Le, Hyunseok Chang, Sarit Mukherjee, Limin Wang, Aditya Akella, Michael M. Swift, and T. V. Lakshman. 2017. UNO: Uniflying Host and Smart NIC Offload for Flexible Packet Processing. In Proceedings of the 2017 Symposium on Cloud Computing (Santa Clara, California) (SoCC '17). Association for Computing Machinery, New York, NY, USA, 506--519.

Digital Library

[12]

Huaicheng Li, Mingzhe Hao, Stanko Novakovic, Vaibhav Gogte, Sriram Govindan, Dan R. K. Ports, Irene Zhang, Ricardo Bianchini, Haryadi S. Gunawi, and Anirudh Badam. 2020. LeapIO: Efficient and Portable Virtual NVMe Storage on ARM SoCs.

[13]

Junnan Li, Zhigang Sun, Jinli Yan, Xiangrui Yang, Yue Jiang, and Wei Quan. 2019. DrawerPipe: A Reconfigurable Pipeline for Network Processing on FPGA-Based SmartNIC. Electronics 9, 1 (Dec 2019), 59.

[14]

Junnan Li, Zhigang Sun, Jinli Yan, Xiangrui Yang, Yue Jiang, and Wei Quan. 2020. DrawerPipe: A reconfigurable pipeline for network processing on FPGA-based SmartNIC. Electronics 9, 1 (2020), 59.

[15]

Huan Liu, Zhiliang Qiu, Weitao Pan, Jun Li, Ling Zheng, and Ya Gao. 2020. Low-Cost and Programmable CRC Implementation based on FPGA (Extended Version). (5 2020).

[16]

Jianshen Liu, Carlos Maltzahn, Craig Ulmer, and Matthew Leon Curry. 2021. Performance Characteristics of the BlueField-2 SmartNIC. arXiv:2105.06619 [cs.NI]

[17]

Ming Liu, Tianyi Cui, Henry Schuh, Arvind Krishnamurthy, Simon Peter, and Karan Gupta. 2019. Offloading Distributed Applications onto SmartNICs Using IPipe. In Proceedings of the ACM Special Interest Group on Data Communication (Beijing, China) (SIGCOMM '19). Association for Computing Machinery, New York, NY, USA, 318--333.

Digital Library

[18]

Ming Liu, Simon Peter, Arvind Krishnamurthy, and Phitchaya Mangpo Phothilimthana. 2019. E3: Energy-Efficient Microservices on SmartNIC-Accelerated Servers. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). USENIX Association, Renton, WA, 363--378.

[19]

Layong Larry Luo and TEG Tencent. 2018. Towards converged SmartNIC architecture for bare metal & public clouds.

[20]

Mellanox. 2019 (accessed June 30, 2020). BlueField SmartNIC for Ethernet High Performance Ethernet Network Adapter Cards.

[21]

Rolf Neugebauer, Gianni Antichi, José Fernando Zazo, Yury Audzevich, Sergio López-Buedo, and Andrew W. Moore. 2018. Understanding PCIe Performance for End Host Networking. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (Budapest, Hungary) (SIGCOMM '18). Association for Computing Machinery, New York, NY, USA, 327--341.

Digital Library

[22]

Yiming Qiu, Qiao Kang, Ming Liu, and Ang Chen. 2020. Clara: Performance Clarity for SmartNIC Offloading. In Proceedings of the 19th ACM Workshop on Hot Topics in Networks. 16--22.

Digital Library

[23]

Pravin Shinde, Antoine Kaufmann, Timothy Roscoe, and Stefan Kaestle. 2013. We Need to Talk About NICs. In 14th Workshop on Hot Topics in Operating Systems (HotOS XIV). USENIX Association, Santa Ana Pueblo, NM.

[24]

Raffaele Sommese. 2018. Boosting the performance of NFV services with SmartNIC. Ph. D. Dissertation. Politecnico di Torino.

[25]

Madhur Tj. 2017 (accessed June 30, 2020). What's The Difference Between In PCI Express Gen 1 vs. Gen 2 vs. Gen 3 vs. Gen 4? / DESKDECODE.COM.

[26]

Yuta Tokusashi and Hiroki Matsutani. 2016. A Multilevel NOSQL Cache Design Combining In-NIC and In-Kernel Caches. In 2016 IEEE 24th Annual Symposium on High-Performance Interconnects (HOTI). 60--67.

[27]

Maroun Tork, Lina Maudlej, and Mark Silberstein. 2020. Lynx: A SmartNIC-Driven Accelerator-Centric Architecture for Network Servers. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS '20). Association for Computing Machinery, New York, NY, USA, 117--131.

Digital Library

[28]

Shuhe Wang, Zili Meng, Chen Sun, Minhu Wang, Mingwei Xu, Jun Bi, Tong Yang, Qun Huang, and Hongxin Hu. 2020. SmartChain: Enabling high-performance service chain partition between SmartNIC and CPU. In ICC 2020-2020 IEEE International Conference on Communications (ICC). IEEE, 1--7.

[29]

X. Wang, Y. Niu, F. Liu, and Z. Xu. 2020. When FPGA Meets Cloud: A First Look at Performance. IEEE Transactions on Cloud Computing (2020), 1--1.

[30]

Xilinx. 2019 (accessed June 30, 2020). Zynq UltraScale+ Device Technical Reference Manual.

[31]

Noa Zilberman, Yury Audzevich, G. Adam Covington, and Andrew W. Moore. 2014. NetFPGA SUME: Toward 100 Gbps as Research Commodity. IEEE Micro 34, 5 (2014), 32--41.

Cited By

Li FChen QShen JWang XCao J(2025)Performance Characteristics and Guidelines of Offloading Middleboxes Onto BlueField-2 DPUIEEE Transactions on Computers10.1109/TC.2024.350037274:2(609-622)Online publication date: Feb-2025
https://doi.org/10.1109/TC.2024.3500372
Tajbakhsh HParizotto RSchaeffer-Filho AHaque I(2024)P4Hauler: An Accelerator-Aware In-Network Load Balancer for Applications Performance BoostingIEEE Transactions on Cloud Computing10.1109/TCC.2024.338965812:2(697-711)Online publication date: Apr-2024
https://doi.org/10.1109/TCC.2024.3389658
Novais FVerdi F(2024)Unlocking Security to the Board: An Evaluation of SmartNIC-driven TLS Acceleration with kTLSNOMS 2024-2024 IEEE Network Operations and Management Symposium10.1109/NOMS59830.2024.10575053(1-9)Online publication date: 6-May-2024
https://doi.org/10.1109/NOMS59830.2024.10575053
Show More Cited By

Index Terms

Towards portable end-to-end network performance characterization of SmartNICs
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Networking hardware
2. Networks
  1. Network performance evaluation
    1. Network performance analysis

Recommendations

Towards a performance-portable FFT library for heterogeneous computing
CF '14: Proceedings of the 11th ACM Conference on Computing Frontiers

The fast Fourier transform (FFT), a spectral method that computes the discrete Fourier transform and its inverse, pervades many applications in digital signal processing, such as imaging, tomography, and software-defined radio. Its importance has caused ...
Performance Portable Applications for Hardware Accelerators: Lessons Learned from SPEC ACCEL
IPDPSW '15: Proceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium Workshop

The popular and diverse hardware accelerator ecosystem makes apples-to-apples comparisons between platforms rather difficult. SPEC ACCEL tries to offer a yardstick to compare different accelerator hardware and software ecosystems. This paper uses this ...
Evaluation of a performance portable lattice Boltzmann code using OpenCL
IWOCL '14: Proceedings of the International Workshop on OpenCL 2013 & 2014

With the advent of many-core computer architectures such as GPGPUs from NVIDIA and AMD, and more recently Intel's Xeon Phi, ensuring performance portability of HPC codes is potentially becoming more complex. In this work we have focused on one important ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

APSys '22: Proceedings of the 13th ACM SIGOPS Asia-Pacific Workshop on Systems

August 2022

89 pages

ISBN:9781450394413

DOI:10.1145/3546591

Program Chairs:
Marco Serafini
UMass Amherst
,
Harry Xu
UCLA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

EPSRC

Conference

APSys '22

Sponsor:

SIGOPS

APSys '22: 13th ACM SIGOPS Asia-Pacific Workshop on Systems

August 23 - 24, 2022

Virtual Event, Singapore

Acceptance Rates

Overall Acceptance Rate 169 of 430 submissions, 39%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
338
Total Downloads

Downloads (Last 12 months)77
Downloads (Last 6 weeks)4

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li FChen QShen JWang XCao J(2025)Performance Characteristics and Guidelines of Offloading Middleboxes Onto BlueField-2 DPUIEEE Transactions on Computers10.1109/TC.2024.350037274:2(609-622)Online publication date: Feb-2025
https://doi.org/10.1109/TC.2024.3500372
Tajbakhsh HParizotto RSchaeffer-Filho AHaque I(2024)P4Hauler: An Accelerator-Aware In-Network Load Balancer for Applications Performance BoostingIEEE Transactions on Cloud Computing10.1109/TCC.2024.338965812:2(697-711)Online publication date: Apr-2024
https://doi.org/10.1109/TCC.2024.3389658
Novais FVerdi F(2024)Unlocking Security to the Board: An Evaluation of SmartNIC-driven TLS Acceleration with kTLSNOMS 2024-2024 IEEE Network Operations and Management Symposium10.1109/NOMS59830.2024.10575053(1-9)Online publication date: 6-May-2024
https://doi.org/10.1109/NOMS59830.2024.10575053
Huang JLou JVanavasam SKong XJi HJeong IZhuo DLee EKim N(2024)HAL: Hardware-assisted Load Balancing for Energy-efficient SNIC-Host Cooperative Computing2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)10.1109/ISCA59077.2024.00051(613-627)Online publication date: 29-Jun-2024
https://doi.org/10.1109/ISCA59077.2024.00051
Zhao JNeves MHaque I(2023)On the (dis)Advantages of Programmable NICs for Network Security Services2023 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking57963.2023.10186433(1-9)Online publication date: 12-Jun-2023
https://doi.org/10.23919/IFIPNetworking57963.2023.10186433
Huang JLou JSun YWang TLee EKim N(2023)Analyzing Energy Efficiency of a Server with a SmartNIC under SLO Constraints2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)10.1109/ISPASS57527.2023.00044(334-336)Online publication date: Apr-2023
https://doi.org/10.1109/ISPASS57527.2023.00044
Huang JLou JSun YWang TLee ESung Kim N(2023)Making Sense of Using a SmartNIC to Reduce Datacenter Tax from SLO and TCO Perspectives2023 IEEE International Symposium on Workload Characterization (IISWC)10.1109/IISWC59245.2023.00025(28-42)Online publication date: 1-Oct-2023
https://doi.org/10.1109/IISWC59245.2023.00025

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten