research-article

Optimizing latency and CPU load in packet processing systems

Authors:

Alexander Beifuß,

Lukas Erlacher,

Florian Wohlfart,

Torsten M. Runge,

Sebastian Gallenmüller,

Georg CarleAuthors Info & Claims

Spects '15: Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems

Pages 1 - 8

Published: 26 July 2015 Publication History

Abstract

High-speed network cards supporting 10 or 40GbE (Gigabit Ethernet) are available today. Software frameworks for high-speed packet reception and transmission were created to exhaust the performance of these cards. However, these frameworks are not applicable as general-purpose solution. Thus, it is necessary to revisit general purpose network IO software that was designed more than a decade ago. In standard Linux settings, connectivity between applications and physical networks happens via the New API (NAPI). This motivated us to investigate how underlying NIC drivers can be adapted to improve latency in combination with the Linux NAPI. Based on testbed measurements, we propose an optimized algorithm for the NIC driver to dynamically adapt the Interrupt Throttling Rate (ITR). We implemented the algorithm and evaluated it with latency and throughput measurements based on the Linux module of Open vSwitch that operates on top of the NAPI. Our measurements show that our new ITR algorithm improves the packet latency without affecting the CPU load as much as other solutions.

References

[1]

M. Dobrescu, N. Egi, K. Argyraki, B. Chun, K. Fall, G. Iannaccone, A. Knies, M. Manesh, and S. Ratnasamy, "RouteBricks: Exploiting Parallelism To Scale Software Routers," in 22nd ACM Symposium on Operating Systems Principles (SOSP), October 2009.

Digital Library

[2]

R. Bolla and R. Bruschi, "PC-based Software Routers: High Performance and Application Service Support," in ACM SIGCOMM Workshop on Programmable Routers for Extensible Services of Tomorrow (PRESTO), August 2008, pp. 27--32.

Digital Library

[3]

T. Meyer, F. Wohlfart, D. Raumer, B. E. Wolfinger, and G. Carle, "Validated Model-Based Performance Prediction of Multi-Core Software Routers," Praxis der Informationsverarbeitung und Kommunikation (PIK), vol. 37, no. 2, pp. 93--107, 2014.

[4]

"Intel Ethernet Controller XL710 Datasheet Rev. 2.1." Intel, December 2014.

[5]

L. Rizzo, "netmap: a novel framework for fast packet I/O," in USENIX Annual Technical Conference, April 2012.

Digital Library

[6]

F. Fusco and L. Deri, "High Speed Network Traffic Analysis with Commodity Multi-core Systems," in Internet Measurement Conference, November 2010, pp. 218--224.

Digital Library

[7]

"Data Plane Development Kit: Programmer's Guide, Revision 6." Intel Corporation, 2014.

[8]

J. H. Salim, R. Olsson, and A. Kuznetsov, "Beyond softnet," in Proceedings of the 5th annual Linux Showcase & Conference, vol. 5, 2001, pp. 18--18.

Digital Library

[9]

J. Mogul, D. Western, J. C. Mogul, and K. K. Ramakrishnan, "Eliminating receive livelock in an interrupt-driven kernel," ACM Transactions on Computer Systems, vol. 15, pp. 217--252, 1997.

Digital Library

[10]

R. Bolla and R. Bruschi, "Linux Software Router: Data Plane Optimization and Performance Evaluation," Journal of Networks, vol. 2, no. 3, pp. 6--17, June 2007.

[11]

A. Tedesco, G. Ventre, L. Angrisani, and L. Peluso, "Measurement of processing and queuing delays introduced by a software router in a single-hop network," in IEEE Instrumentation and Measurement Technology Conference 2005, 2005, pp. 1797--1802.

[12]

P. Emmerich, D. Raumer, F. Wohlfart, and G. Carle, "Performance Characteristics of Virtual Switching," in 2014 IEEE 3rd International Conference on Cloud Networking (CloudNet), Luxembourg, October 2014.

[13]

Intel, "Assigning Interrupts to Processor Cores using an Intel® 82575/82576 or 82598/82599 Ethernet Controller," http://www.intel.com/content/dam/doc/application-note/82575-82576-82598-82599-ethernet-controllers-interrupts-appl-note.pdf, 2009.

[14]

M. Wilcox, "I'll do it later: softirqs, tasklets, bottom halves, task queues, work queues and timers," in Proceedings of the 2003 Linux Conference Australia (LCA 2003), 2003.

[15]

"Intel 82599 10 GbE Controller Datasheet Rev. 2.76," Intel Corporation, 2012, Santa Clara, USA.

[16]

Intel, "Intel Server Adapters - Linux ixgbe Base Driver," http://www.intel.com/support/network/adapter/pro100/sb/CS-032530.htm, last visited 2015-01-27.

[17]

A. Beifuß, D. Raumer, P. Emmerich, T. M. Runge, F. Wohlfart, B. E. Wolfinger, and G. Carle, "A Study of Networking Software Induced Latency," in 2nd International Conference on Networked Systems (NetSys), March 2015.

[18]

P. Emmerich, S. Gallenmüller, D. Raumer, F. Wohlfart, and G. Carle, "MoonGen: A Scriptable High-Speed Packet Generator," ArXiv e-prints, Oct. 2014.

[19]

L. Erlacher, "Patch for ixgbe-3.14.5," https://github.com/duk3luk3/ixgbe-3.14.15/commit/c0a258a3d6fdd50f51e4231c946b13dc665eecf0.

[20]

L. Erlacher, "Patch for ixgbe-3.23.2," https://github.com/duk3luk3/ixgbe-3.23.2/commit/fcd21e9db103680d77bfd3a016eb3271b44d2a2e.

[21]

J. H. Salim, "When NAPI comes to Town," in UKUUG 2005 Linux Technical Conference, 2005.

[22]

K. Salah and A. Qahtan, "Implementation and experimental performance evaluation of a hybrid interrupt-handling scheme." Computer Communications, vol. 32, no. 1, pp. 179--188, 2009.

Digital Library

[23]

O. Maquelin, G. R. Gao, H. H. J. Humy, K. B. Theobald, and X. Tian, "Polling watchdog: Combining polling and interrupts for efficient message handling," in in Proceedings of the 23rd Annual International Symposium on Computer Architecture. ACM Press, 1995, pp. 179--188.

Digital Library

[24]

C. Dovrolis, B. Thayer, and P. Ramanathan, "Hip: Hybrid interrupt-polling for the network interface," SIGOPS Oper. Syst. Rev., vol. 35, no. 4, pp. 50--60, Oct. 2001.

Digital Library

[25]

X. Chang, J. K. Muppala, P. Zou, and X. Li, "A robust device hybrid scheme to improve system performance in gigabit ethernet networks." in LCN. IEEE Computer Society, 2007, pp. 444--454.

Digital Library

[26]

P. Emmerich, D. Raumer, F. Wohlfart, and G. Carle, "A Study of Network Stack Latency for Game Servers," in 13th Annual Workshop on Network and Systems Support for Games (NetGames'14), Nagoya, Japan, Dec. 2014.

Digital Library

[27]

J. Brandeburg, "A way towards Lower Latency and Jitter," Talk at the Linux Plumbers Conference, Slides available athttp://www.linuxplumbersconf.org/2012/wp-content/uploads/2012/09/2012-lpc-Low-Latency-Sockets-slides-brandeburg.pdf, 2012.

[28]

J. Corbet, "Low-latency Ethernet device polling," 2013.

[29]

L. Niccolini, G. Iannaccone, S. Ratnasamy, J. Chandrashekar, and L. Rizzo, "Building a power-proportional software router," in USENIX Annual Technical Conference (USENIX ATC 12). Boston, MA: USENIX, 2012, pp. 89--100.

Digital Library

Cited By

Lei JMunikar MSuo KLu HRao JBarbalace ABhatotia PAlvisi LCadar C(2021)Parallelizing packet processing in container overlay networksProceedings of the Sixteenth European Conference on Computer Systems10.1145/3447786.3456241(261-276)Online publication date: 21-Apr-2021
https://dl.acm.org/doi/10.1145/3447786.3456241
Hanford NAhuja VFarrens MTierney BGhosal D(2018)A Survey of End-System Optimizations for High-Speed NetworksACM Computing Surveys10.1145/318489951:3(1-36)Online publication date: 16-Jul-2018
https://dl.acm.org/doi/10.1145/3184899
Raumer DGallenmüller SWohlfart FEmmerich PWerneck PCarle GEggert LPerkins C(2016)Revisiting Benchmarking Methodology for Interconnect DevicesProceedings of the 2016 Applied Networking Research Workshop10.1145/2959424.2959430(55-61)Online publication date: 16-Jul-2016
https://dl.acm.org/doi/10.1145/2959424.2959430

Optimizing latency and CPU load in packet processing systems
1. General and reference
  1. Cross-computing tools and techniques
2. Social and professional topics
  1. Professional topics
    1. Computing profession

Recommendations

Fast Packet Processing with eBPF and XDP: Concepts, Code, Challenges, and Applications

Extended Berkeley Packet Filter (eBPF) is an instruction set and an execution environment inside the Linux kernel. It enables modification, interaction, and kernel programmability at runtime. eBPF can be used to program the eXpress Data Path (XDP), a ...
Low latency network traffic processing with commodity hardware
Spects '15: Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems

Packet processing on commodity hardware is a cost-efficient and flexible alternative to specialized networking hardware. In case of Linux, the classical QoS mechanisms (e.g DiffServ) assume that the outgoing link is the bottleneck. However, on commodity ...
Implementation and experimental performance evaluation of a hybrid interrupt-handling scheme

The performance of network hosts can be severely degraded when subjected to heavy traffic of today's Gigabit networks. This degradation occurs as a result of the interrupt overhead associated with the high rate of packet arrivals. NAPI, a packet ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

Spects '15: Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems

July 2015

171 pages

ISBN:9781510810600

Co-chair:
Floriano De Rango
University of Calabria, Italy
,
Editors:
Franco Davoli
University of Genoa, Italy
,
Jose Marzo
University of Girona, Spain
,
Joel Rodrigues
Instituto de Telecomunicações, Univ. of Beira Interior, Portugal
,
Malamati Louta
University of Western Macedonia, Greece
,
Imadeldin Mahgoub
Florida Atlantic Univ., USA
,
Jose Saldana
University of Zaragoza, Spain
,
General Chair:
Saurabh Mittal
Dunip Technologies, LLC, Colorado, USA
,
Program Chair:
José Luis Risco Martín
Universidad Complutense de Madrid, Spain
,
Publications Chair:
Deniz Cetinkaya
Atilim University, Turkey

Sponsors

SIGSIM: ACM Special Interest Group on Simulation and Modeling
SCS: Society for Modeling and Simulation International

Publisher

Society for Computer Simulation International

San Diego, CA, United States

Publication History

Published: 26 July 2015

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SummerSim '15

Sponsor:

SIGSIM
SCS

SummerSim '15: 2015 Summer Simulation Multiconference

July 26 - 29, 2015

Illinois, Chicago

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
143
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)2

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lei JMunikar MSuo KLu HRao JBarbalace ABhatotia PAlvisi LCadar C(2021)Parallelizing packet processing in container overlay networksProceedings of the Sixteenth European Conference on Computer Systems10.1145/3447786.3456241(261-276)Online publication date: 21-Apr-2021
https://dl.acm.org/doi/10.1145/3447786.3456241
Hanford NAhuja VFarrens MTierney BGhosal D(2018)A Survey of End-System Optimizations for High-Speed NetworksACM Computing Surveys10.1145/318489951:3(1-36)Online publication date: 16-Jul-2018
https://dl.acm.org/doi/10.1145/3184899
Raumer DGallenmüller SWohlfart FEmmerich PWerneck PCarle GEggert LPerkins C(2016)Revisiting Benchmarking Methodology for Interconnect DevicesProceedings of the 2016 Applied Networking Research Workshop10.1145/2959424.2959430(55-61)Online publication date: 16-Jul-2016
https://dl.acm.org/doi/10.1145/2959424.2959430

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents