research-article

Revisiting Benchmarking Methodology for Interconnect Devices

Authors:

Sebastian Gallenmüller,

Florian Wohlfart,

Patrick Werneck,

Georg CarleAuthors Info & Claims

ANRW '16: Proceedings of the 2016 Applied Networking Research Workshop

Pages 55 - 61

https://doi.org/10.1145/2959424.2959430

Published: 16 July 2016 Publication History

Abstract

Ever growing demand for network bandwidth makes computer networks an area of constant development and fast adjustments. The steady change makes good performance assessments equally necessary and challenging. This development motivated us to revisit the established benchmarking methodology. We provide an overview of the state-of-the-art in router benchmarking, the currently available benchmarking tools, and challenges for benchmarks. A discussion of benchmarking results for three different devices (routers based on Linux and FreeBSD, and a MikroTik router) reveal different properties currently not covered by standardized benchmarks. We conclude by adding tests to the common benchmarking methodology reflecting these properties to make the results more valuable. The prototype software implementation of our own benchmarking tool and its measurement reports are publicly available.

References

[1]

DPDK. http://dpdk.org/. (visited 16/20/05).

[2]

Ixia IxNetwork QuickTest, 2014.

[3]

R. Asati, C. Pignataro, F. Calabria, and C. Olvera. Device Reset Characterization. RFC 6201, 2011.

[4]

A. Beifuß, D. Raumer, P. Emmerich, T. M. Runge, F. Wohlfart, B. E. Wolfinger, and G. Carle. A Study of Networking Software Induced Latency. In NetSys, Cottbus, Germany, 2015.

[5]

BMWG history. https://datatracker.ietf.org/doc/charter-ietf-bmwg/history/. (visited 16/20/05).

[6]

R. Bolla and R. Bruschi. Linux Software Router: Data Plane Optimization and Performance Evaluation. JNW, 2(3), 2007.

[7]

A. Botta, A. Dainotti, and A. Pescapé. Do You Trust Your Software-Based Traffic Generator? IEEE ComMag, 48(9), 2010.

Digital Library

[8]

S. Bradner. Benchmarking Terminology for Network Interconnection Devices. RFC 1242, 1991.

Digital Library

[9]

S. Bradner and J. McQuaid. Benchmarking Methodology for Network Interconnect Devices. RFC 2544, 1999.

Digital Library

[10]

Candela. LANforge RFC Support. http://www.candelatech.com/rfcs.php. (visited 16/20/05).

[11]

Cisco. Cisco Nexus 3548 Switch Performance Validation, 2012.

[12]

G. A. Covington, G. Gibb, J. W. Lockwood, and N. McKeown. A Packet Generator on the NetFPGA Platform. In IEEE FCCM, Napa, USA, 2009.

Digital Library

[13]

P. Emmerich, S. Gallenmüller, and G. Carle. FLOWer – Device Benchmarking Beyond 100 Gbit/s. Vienna, Austria, 2016.

[14]

P. Emmerich, S. Gallenmüller, D. Raumer, F. Wohlfart, and G. Carle. MoonGen: A Scriptable High-Speed Packet Generator. In ACM SIGCOMM IMC, Tokyo, Japan, 2015.

Digital Library

[15]

P. Emmerich, D. Raumer, A. Beifuß, L. Erlacher, F. Wohlfart, T. M. Runge, S. Gallenmüller, and G. Carle. Optimizing Latency and CPU Load in Packet Processing Systems. In SPECTS, Chicago, USA, 2015.

Digital Library

[16]

P. Emmerich, D. Raumer, F. Wohlfart, and G. Carle. Performance Characteristics of Virtual Switching. In IEEE CloudNet, Luxembourg, 2014.

[17]

P. Emmerich, D. Raumer, F. Wohlfart, and G. Carle. Assessing Soft- and Hardware Bottlenecks in PC-based Packet Forwarding Systems. In ICN, Barcelona, Spain, 2015.

[18]

S. Gallenmüller, P. Emmerich, F. Wohlfart, D. Raumer, and G. Carle. Comparison of Frameworks for High-Performance Packet IO. In ACM/IEEE ANCS, Santa Clara, USA, 2015.

[19]

M. Georgescu, A. Morton, S. Fernandes, and P. Emmerich. Mean vs Median. Discussion on the IETF BMWG mailing list, archive available online at https://mailarchive.ietf.org/arch/, Nov. 2015.

[20]

M. Ghobadi, G. Salmon, Y. Ganjali, M. Labrecque, and J. G. Steffan. Caliper: Precise and responsive traffic generator. In IEEE HOTI, Santa Clara, USA, 2012.

Digital Library

[21]

iPerf. http://iperf.fr/. (visited 16/20/05).

[22]

ITU-T. Y.1564: Ethernet service activation test methodology Recommendation, 2011.

[23]

W. E. Leland, M. S. Taqqu, W. Willinger, and D. V. Wilson. On the self-similar nature of ethernet traffic (extended version). IEEE/ACM Trans.Netw., 2(1), 1994.

Digital Library

[24]

R. Mandeville and J. Perser. Benchmarking Methodology for LAN Switching Devices, Address caching capacity. RFC 2889, 2000.

Digital Library

[25]

Metro Ethernet Forum. MEF 14: Abstract Test Suite for Traffic Management Phase 1, 2005.

[26]

MoonGen RFC 2544 Benchmark Reports. http://net.in.tum.de/pub/router-benchmarking/. (visited 16/20/05).

[27]

NetFPGA. http://netfpga.org/. (visited 16/20/05).

[28]

ntop. PFRING ZC (Zero Copy). http://www.ntop.org/products/pf\_ring/pf_ring-zc-zero-copy/. (visited 16/16/05).

[29]

ntop. Whitepaper: RFC-2544 performance test - PFRing-DNA VS Standard network Driver, 2012.

[30]

Pktgen-DPDK. http://github.com/Pktgen/Pktgen-DPDK/.

[31]

C. Popoviciu, A. Hamza, G. V. de Velde, and D. Dugatkin. IPv6 Benchmarking Methodology for Network Interconnect Devices. RFC 5180, 2008.

[32]

L. Rizzo. netmap: A Novel Framework for Fast Packet I/O. In USENIX ATC, Boston, USA, 2012.

Digital Library

[33]

T. G. Robertazzi. Computer networks and systems: queueing theory and performance evaluation, 3rd edtion, Chapter 7.6: Self-Similar Traffic. Springer, 2000.

[34]

C. Rotsos, N. Sarrar, S. Uhlig, R. Sherwood, and A. W. Moore. Oflops: An Open Framework for OpenFlow Switch Evaluation. In PAM, Vienna, Austria, 2012.

Digital Library

[35]

SnabbCo. Snabb. https://github.com/SnabbCo/snabb. (visited 16/13/05).

[36]

J. Sommers and P. Barford. Self-Configuring Network Traffic Generation. In ACM SIGCOMM IMC, Taormina, Italy, 2004.

Digital Library

[37]

Spirent. Whitepaper: Spirent Testcenter RFC 2544 Benchmarking Test Package, 2010.

[38]

The European Advanced Networking Test Center (EANTC). Whitepaper: Huawei technologies service activation using rfc 2544 tests, 2008.

[39]

G. Trotter. Terminology for Forwarding Information Base (FIB) based Router Performance. RFC 3222, 2001.

Digital Library

[40]

J. Whiteaker, F. Schneider, and R. Teixeira. Explaining packet delays under virtualization. ACM SIGCOMM CCR, 41, 2011.

Digital Library

[41]

Xena Networks. Whitepaper: Xena2544 RFC2544 testing made easy, 2015.

Cited By

Bhutto AKawashima RTaenaka YKadobayashi Y(2024)Meeting Latency and Jitter Demands of Beyond 5G Networking Era: Are CNFs Up to the Challenge?2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00251(1598-1605)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00251
Al-hamadani ALencse G(2021)Design of a Software Tester for Benchmarking Lightweight 4over6 Devices2021 44th International Conference on Telecommunications and Signal Processing (TSP)10.1109/TSP52935.2021.9522607(157-161)Online publication date: 26-Jul-2021
https://doi.org/10.1109/TSP52935.2021.9522607
Rao SPaganelli FMorton A(2021)Benchmarking Kubernetes Container-Networking for Telco Usecases2021 IEEE Global Communications Conference (GLOBECOM)10.1109/GLOBECOM46510.2021.9685803(1-7)Online publication date: Dec-2021
https://doi.org/10.1109/GLOBECOM46510.2021.9685803
Show More Cited By

Index Terms

Revisiting Benchmarking Methodology for Interconnect Devices
1. Networks
  1. Network performance evaluation
    1. Network measurement

Recommendations

Rigorous benchmarking in reasonable time
ISMM '13

Experimental evaluation is key to systems research. Because modern systems are complex and non-deterministic, good experimental methodology demands that researchers account for uncertainty. To obtain valid results, they are expected to run many ...
Rigorous benchmarking in reasonable time
ISMM '13: Proceedings of the 2013 international symposium on memory management

Experimental evaluation is key to systems research. Because modern systems are complex and non-deterministic, good experimental methodology demands that researchers account for uncertainty. To obtain valid results, they are expected to run many ...
RFC 5180: IPv6 Benchmarking Methodology for Network Interconnect Devices

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ANRW '16: Proceedings of the 2016 Applied Networking Research Workshop

July 2016

93 pages

ISBN:9781450344432

DOI:10.1145/2959424

Program Chairs:
Lars Eggert
NetApp
,
Colin Perkins
University of Glasgow

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

IRTF: Internet Research Task Force
Internet Society: Internet Society
SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 July 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ANRW '16

Sponsor:

IRTF
Internet Society
SIGCOMM

ANRW '16: Applied Networking Research Workshop

July 16, 2016

Berlin, Germany

Acceptance Rates

ANRW '16 Paper Acceptance Rate 18 of 30 submissions, 60%;

Overall Acceptance Rate 34 of 58 submissions, 59%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
161
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bhutto AKawashima RTaenaka YKadobayashi Y(2024)Meeting Latency and Jitter Demands of Beyond 5G Networking Era: Are CNFs Up to the Challenge?2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00251(1598-1605)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00251
Al-hamadani ALencse G(2021)Design of a Software Tester for Benchmarking Lightweight 4over6 Devices2021 44th International Conference on Telecommunications and Signal Processing (TSP)10.1109/TSP52935.2021.9522607(157-161)Online publication date: 26-Jul-2021
https://doi.org/10.1109/TSP52935.2021.9522607
Rao SPaganelli FMorton A(2021)Benchmarking Kubernetes Container-Networking for Telco Usecases2021 IEEE Global Communications Conference (GLOBECOM)10.1109/GLOBECOM46510.2021.9685803(1-7)Online publication date: Dec-2021
https://doi.org/10.1109/GLOBECOM46510.2021.9685803
Zerwas JKalmbach PHenkel LRétvári GKellerer WBlenk ASchmid S(2019)NetBOAProceedings of the 2019 Workshop on Network Meets AI & ML10.1145/3341216.3342207(8-14)Online publication date: 14-Aug-2019
https://dl.acm.org/doi/10.1145/3341216.3342207
van Bemten ADeric NVarasteh ABlenk ASchmid SKellerer W(2019)Empirical Predictability Study of SDN Switches2019 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS)10.1109/ANCS.2019.8901878(1-13)Online publication date: Sep-2019
https://doi.org/10.1109/ANCS.2019.8901878
Bauer SRaumer DEmmerich PCarle G(2018)Behind the scenesProceedings of the 2018 Applied Networking Research Workshop10.1145/3232755.3232776(58-65)Online publication date: 16-Jul-2018
https://dl.acm.org/doi/10.1145/3232755.3232776
Lin YLai YWang CLai Y(2018)OFBench: Performance Test Suite on OpenFlow SwitchesIEEE Systems Journal10.1109/JSYST.2017.272075812:3(2949-2959)Online publication date: Sep-2018
https://doi.org/10.1109/JSYST.2017.2720758
Gallenmuller SGunther SLeclaire MZoppi SMolinari FSchoffauer RKellerer WCarle G(2018)Benchmarking Networked Control Systems2018 IEEE Workshop on Benchmarking Cyber-Physical Networks and Systems (CPSBench)10.1109/CPSBench.2018.00008(7-12)Online publication date: Apr-2018
https://doi.org/10.1109/CPSBench.2018.00008
Gallenmuller SScholz DWohlfart FScheitle QEmmerich PCarle G(2018)High-performance packet processing and measurements2018 10th International Conference on Communication Systems & Networks (COMSNETS)10.1109/COMSNETS.2018.8328173(1-8)Online publication date: Jan-2018
https://doi.org/10.1109/COMSNETS.2018.8328173

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents