research-article

Free access

Packet-Level Telemetry in Large Datacenter Networks

Authors:

Albert Greenberg,

Haitao ZhengAuthors Info & Claims

SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication

Pages 479 - 491

https://doi.org/10.1145/2785956.2787483

Published: 17 August 2015 Publication History

Abstract

Debugging faults in complex networks often requires capturing and analyzing traffic at the packet level. In this task, datacenter networks (DCNs) present unique challenges with their scale, traffic volume, and diversity of faults. To troubleshoot faults in a timely manner, DCN administrators must a) identify affected packets inside large volume of traffic; b) track them across multiple network components; c) analyze traffic traces for fault patterns; and d) test or confirm potential causes. To our knowledge, no tool today can achieve both the specificity and scale required for this task.

We present Everflow, a packet-level network telemetry system for large DCNs. Everflow traces specific packets by implementing a powerful packet filter on top of "match and mirror" functionality of commodity switches. It shuffles captured packets to multiple analysis servers using load balancers built on switch ASICs, and it sends "guided probes" to test or confirm potential faults. We present experiments that demonstrate Everflow's scalability, and share experiences of troubleshooting network faults gathered from running it for over 6 months in Microsoft's DCNs.

Supplementary Material

WEBM File (p479-zhu.webm)

Download
148.88 MB

References

[1]

Data plane development kit. http://www.dpdk.org/.

[2]

Receive side scaling. https://msdn.microsoft.com/en-us/library/windows/hardware/ff567236(v=vs.85).aspx.

[3]

A. Arefin, A. Khurshid, M. Caesar, and K. Nahrstedt. Scaling data-plane logging in large scale networks. In MILCOM, 2011.

[4]

P. Bosshart, G. Gibb, H.-S. Kim, G. Varghese, N. McKeown, M. Izzard, F. Mujica, and M. Horowitz. Forwarding metamorphosis: Fast programmable match-action processing in hardware for SDN. In SIGCOMM, 2013.

Digital Library

[5]

J. Case, M. Fedor, M. Schoffstall, and J. Davin. RFC 1157: Simple network management protocol.

Digital Library

[6]

R. Chaiken, B. Jenkins, P.-Å. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. Scope: easy and efficient parallel processing of massive data sets. VLDB, 2008.

Digital Library

[7]

B. Claise. RFC 3954: Cisco systems netflow services export version 9 (2004).

Digital Library

[8]

N. G. Duffield and M. Grossglauser. Trajectory sampling for direct traffic observation. IEEE/ACM Trans. Netw., June 2001.

Digital Library

[9]

S. K. Fayaz and V. Sekar. Testing stateful and dynamic data planes with flowtest. In HotSDN, 2014.

Digital Library

[10]

A. Fogel, S. Fung, L. Pedrosa, M. Walraed-Sullivan, R. Govindan, R. Mahajan, and T. Millstein. A general approach to network configuration analysis. In NSDI, 2015.

Digital Library

[11]

R. Fonseca, G. Porter, R. H. Katz, S. Shenker, and I. Stoica. X-trace: A pervasive network tracing framework. In NSDI, 2007.

Digital Library

[12]

R. Gandhi, H. H. Liu, Y. C. Hu, G. Lu, J. Padhye, L. Yuan, and M. Zhang. Duet: Cloud scale load balancing with hardware and software. In SIGCOMM, 2014.

Digital Library

[13]

N. Gvozdiev, B. Karp, and M. Handley. Loup: who's afraid of the big bad loop? In HotNets, 2012.

Digital Library

[14]

N. Handigol, B. Heller, V. Jeyakumar, D. Mazières, and N. McKeown. I know what your packet did last hop: Using packet histories to troubleshoot networks. In NSDI, 2014.

Digital Library

[15]

C.-Y. Hong, M. Caesar, N. Duffield, and J. Wang. Tiresias: Online anomaly detection for hierarchical operational network data. In ICDCS, 2012.

Digital Library

[16]

C.-Y. Hong, S. Kandula, R. Mahajan, M. Zhang, V. Gill, M. Nanduri, and R. Wattenhofer. Achieving high utilization with software-driven WAN. In SIGCOMM, 2013.

Digital Library

[17]

Infiniband Trade Association. InfiniBand Architecture Volume 1, General Specifications, Release 1.2.1, 2008.

[18]

Infiniband Trade Association. Supplement to infiniband architecture specification volume 1 release 1.2.2 annex A17: RoCEv2 (ip routable ROCE), 2014.

[19]

S. Jain, A. Kumar, S. Mandal, J. Ong, L. Poutievski, A. Singh, S. Venkata, J. Wanderer, J. Zhou, M. Zhu, J. Zolla, U. Hölzle, S. Stuart, and A. Vahdat. B4: Experience with a globally-deployed software defined WAN. In SIGCOMM, 2013.

Digital Library

[20]

V. Jeyakumar, M. Alizadeh, Y. Geng, C. Kim, and D. Mazières. Millions of little minions: Using packets for low latency network programming and visibility. In SIGCOMM, 2014.

Digital Library

[21]

S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken. The nature of datacenter traffic: measurements & analysis. In IMC, 2009.

Digital Library

[22]

P. Kazemian, M. Chan, H. Zeng, G. Varghese, N. McKeown, and S. Whyte. Real time network policy checking using header space analysis. In NSDI, 2013.

Digital Library

[23]

A. Khurshid, X. Zou, W. Zhou, M. Caesar, and P. B. Godfrey. Veriflow: Verifying network-wide invariants in real time. In NSDI, 2013.

Digital Library

[24]

T. Koponen, K. Amidon, P. Balland, M. Casado, A. Chanda, B. Fulton, I. Ganichev, J. Gross, N. Gude, P. Ingram, E. Jackson, A. Lambeth, R. Lenglet, S.-H. Li, A. Padmanabhan, J. Pettit, B. Pfaff, R. Ramanathan, S. Shenker, A. Shieh, J. Stribling, P. Thakkar, D. Wendlandt, A. Yip, and R. Zhang. Network virtualization in multi-tenant datacenters. In NSDI, 2014.

Digital Library

[25]

R. Mahajan, N. Spring, D. Wetherall, and T. Anderson. User-level Internet path diagnosis. In SOSP, 2003.

Digital Library

[26]

V. Mann, A. Vishnoi, and S. Bidkar. Living on the edge: Monitoring network flows at the edge in cloud data centers. In COMSNETS, 2013.

[27]

P. Marchetta, A. Botta, E. Katz-Bassett, and A. Pescapé. Dissecting round trip time on the slow path with a single packet. In PAM, 2014.

Digital Library

[28]

P. Patel, D. Bansal, L. Yuan, A. Murthy, A. Greenberg, D. A. Maltz, and R. Kern. Ananta: cloud scale load balancing. In SIGCOMM, 2013.

Digital Library

[29]

P. Phaal, S. Panchen, and N. McKee. RFC 3176: Inmon corporation's sflow: A method for monitoring traffic in switched and routed networks, 2001.

Digital Library

[30]

T. Qiu, Z. Ge, D. Pei, J. Wang, and J. Xu. What happened in my network: mining network events from router syslogs. In IMC, 2010.

Digital Library

[31]

J. Rasley, B. Stephens, C. Dixon, E. Rozner, W. Felter, K. Agarwal, J. Carter, and R. Fonseca. Planck: Millisecond-scale monitoring and control for commodity networks. In SIGCOMM, 2014.

Digital Library

[32]

L. Rizzo. netmap: A novel framework for fast packet I/O. In USENIX ATC, 2012.

Digital Library

[33]

J. Suh, T. Kwon, C. Dixon, W. Felter, and J. Carter. Opensample: A low-latency, sampling-based measurement platform for SDN. In ICDCS, 2014.

Digital Library

[34]

W. Wu and P. Demar. Wirecap: a novel packet capture engine for commodity NICs in high-speed networks. In IMC, 2014.

Digital Library

[35]

A. Wundsam, D. Levin, S. Seetharaman, and A. Feldmann. OFRewind: Enabling record and replay troubleshooting for networks. In ATC, 2011.

Digital Library

[36]

M. Yu, L. Jose, and R. Miao. Software defined traffic measurement with opensketch. In NSDI, 2013.

Digital Library

Cited By

Zhou HHu DZhou ZYuan GDong D(2024)DDT: Dynamical Selective Dropping Threshold for Reactive Congestion ControlProceedings of the ACM Turing Award Celebration Conference - China 202410.1145/3674399.3674412(12-17)Online publication date: 5-Jul-2024
https://dl.acm.org/doi/10.1145/3674399.3674412
Li EWu WZhaohua WLi ZNiu J(2024)SAROS: A Self-Adaptive Routing Oblivious Sampling Method for Network-wide Heavy Hitter DetectionProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663429(142-148)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663429
Liu KZhang JJiang ZZhang XGuo SBai YDong YZhang ZShi XWang LWei HWang ZPan YPan THuang T(2024)Hostmesh: Monitor and Diagnose Networks in Rail-optimized RoCE ClustersProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663426(122-128)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663426
Show More Cited By

Index Terms

Packet-Level Telemetry in Large Datacenter Networks
1. Networks
  1. Network services
    1. Network management

Recommendations

Packet-Level Telemetry in Large Datacenter Networks
SIGCOMM'15

Debugging faults in complex networks often requires capturing and analyzing traffic at the packet level. In this task, datacenter networks (DCNs) present unique challenges with their scale, traffic volume, and diversity of faults. To troubleshoot faults ...
Improving datacenter throughput and robustness with Lazy TCP over packet spraying

Packet spraying is being recognized as a promising multipath routing approach in datacenter for full utilization of multiple paths. However, existing TCP schemes face serious challenge of robustness when running over packet spraying. They suffer ...
Analyzing and Optimizing Packet Corruption in RDMA Network
Abstract
Remote direct memory access (RDMA) has become one of the state-of-the-art high-performance network technologies in datacenters. The reliable transport of RDMA is designed based on a lossless underlying network and cannot endure a high packet loss ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication

August 2015

684 pages

ISBN:9781450335423

DOI:10.1145/2785956

General Chairs:
Steve Uhlig
Queen Mary University of London, UK
,
Olaf Maennel
Tallinn U. of Technology in Estonia, Estonia
,
Program Chairs:
Brad Karp
University College London, UK
,
Jitendra Padhye
Microsoft, USA

ACM SIGCOMM Computer Communication Review Volume 45, Issue 4
SIGCOMM'15
October 2015
659 pages
ISSN:0146-4833
DOI:10.1145/2829988
Editors:
Konstantina Papagiannaki
Telefonica Research, Barcelona, Spain
,
Katerina Argyraki
EPFL, Switzerland
,
Hitesh Ballani
Microsoft Research Cambridge, UK
,
Fabián Bustamante
Northwestern University, USA
,
Joseph Camp
SMU, USA
,
Augustin Chaintreau
Columbia University, USA
,
Phillipa Gill
Stony Brook University, USA
,
Marco Mellia
Politecnico di Torino, Italy
,
Bhaskaran Raman
IIT Bombay, India
,
Joel Sommers
Colgate University, USA
,
Aline Carneiro Viana
INRIA, France
Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGCOMM '15

Sponsor:

SIGCOMM

SIGCOMM '15: ACM SIGCOMM 2015 Conference

August 17 - 21, 2015

London, United Kingdom

Acceptance Rates

SIGCOMM '15 Paper Acceptance Rate 40 of 242 submissions, 17%;

Overall Acceptance Rate 462 of 3,389 submissions, 14%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

254
Total Citations
View Citations
2,215
Total Downloads

Downloads (Last 12 months)432
Downloads (Last 6 weeks)65

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhou HHu DZhou ZYuan GDong D(2024)DDT: Dynamical Selective Dropping Threshold for Reactive Congestion ControlProceedings of the ACM Turing Award Celebration Conference - China 202410.1145/3674399.3674412(12-17)Online publication date: 5-Jul-2024
https://dl.acm.org/doi/10.1145/3674399.3674412
Li EWu WZhaohua WLi ZNiu J(2024)SAROS: A Self-Adaptive Routing Oblivious Sampling Method for Network-wide Heavy Hitter DetectionProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663429(142-148)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663429
Liu KZhang JJiang ZZhang XGuo SBai YDong YZhang ZShi XWang LWei HWang ZPan YPan THuang T(2024)Hostmesh: Monitor and Diagnose Networks in Rail-optimized RoCE ClustersProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663426(122-128)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663426
Liu KJiang ZZhang JGuo SZhang XBai YDong YLuo FZhang ZWang LShi XXu HBai YSong DWei HLi BPan YPan THuang TSekar VYu MSeneviratne AVeitch D(2024)R-Pingmesh: A Service-Aware RoCE Network Monitoring and Diagnostic SystemProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672264(554-567)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672264
Gigis PHandley MVissicchio SSekar VYu MSeneviratne AVeitch D(2024)Bad Packets Come Back, Worse Ones Don'tProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672259(311-326)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672259
Ding RLiu XYang SHuang QXie BSun RZhang ZCui BSekar VYu MSeneviratne AVeitch D(2024)RD-Probe: Scalable Monitoring With Sufficient Coverage In Complex Datacenter NetworksProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672256(258-273)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672256
Zheng HHuang CHan XZheng JWang XTian CDou WChen GSekar VYu MSeneviratne AVeitch D(2024)μMon: Empowering Microsecond-level Network Monitoring with WaveletsProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672236(274-290)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672236
Shou CBhatia RGupta AHarrison RLokshtanov DWillinger W(2024)Query Planning for Robust and Scalable Hybrid Network Telemetry SystemsProceedings of the ACM on Networking10.1145/36494712:CoNEXT1(1-27)Online publication date: 28-Mar-2024
https://doi.org/10.1145/3649471
Qian MCui LTso FDeng YJia W(2024)OffsetINT: Achieving High Accuracy and Low Bandwidth for In-Band Network TelemetryIEEE Transactions on Services Computing10.1109/TSC.2023.332369717:3(1072-1083)Online publication date: May-2024
https://doi.org/10.1109/TSC.2023.3323697
Ren JWang YLiu YWang XWang S(2024)Server-Assisted Traffic Measurement for Programmable Data Center NetworksIEEE Transactions on Network Science and Engineering10.1109/TNSE.2024.339729111:5(4729-4743)Online publication date: Sep-2024
https://doi.org/10.1109/TNSE.2024.3397291
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents