research-article

Free access

Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis

Authors:

Chuanxiong Guo,

Varugis KurienAuthors Info & Claims

SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication

Pages 139 - 152

https://doi.org/10.1145/2785956.2787496

Published: 17 August 2015 Publication History

Abstract

Can we get network latency between any two servers at any time in large-scale data center networks? The collected latency data can then be used to address a series of challenges: telling if an application perceived latency issue is caused by the network or not, defining and tracking network service level agreement (SLA), and automatic network troubleshooting. We have developed the Pingmesh system for large-scale data center network latency measurement and analysis to answer the above question affirmatively. Pingmesh has been running in Microsoft data centers for more than four years, and it collects tens of terabytes of latency data per day. Pingmesh is widely used by not only network software developers and engineers, but also application and service developers and operators.

Supplementary Material

WEBM File (p139-guo.webm)

Download
161.80 MB

References

[1]

M. Al-Fares, A. Loukissas, and A. Vahdat. A Scalable, Commodity Data Center Network Architecture. In Proc. SIGCOMM, 2008.

Digital Library

[2]

Alexey Andreyev. Introducing data center fabric, the next-generation Facebook data center network. https://code.facebook.com/posts/360346274145943/, Nov 2014.

[3]

Hadoop. http://hadoop.apache.org/.

[4]

Peter Bailis and Kyle Kingsbury. The Network is Reliable: An Informal Survey of Real-World Communications Failures. ACM Queue, 2014.

Digital Library

[5]

Luiz Barroso, Jeffrey Dean, and Urs H$\ddoto$lzle. Web Search for a Planet: The Google Cluster Architecture. IEEE Micro, March-April 2003.

Digital Library

[6]

Theophilus Benson, Aditya Akella, and David A. Maltz. Network Traffic Characteristics of Data Centers in the Wild. In Internet Measurement Conference, November 2010.

Digital Library

[7]

et.al Brad Calder. Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency. In SOSP, 2011.

Digital Library

[8]

Cisco. IP SLAs Configuration Guide, Cisco IOS Release 12.4T. http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipsla/configuration/12--4t/sla-12--4t-book.pdf.

[9]

Citrix. What is Load Balancing? http://www.citrix.com/glossary/load-balancing.html.

[10]

Jeffrey Dean and Luiz Andr$\acutee$ Barroso. The Tail at Scale. CACM, Februry 2013.

Digital Library

[11]

Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In OSDI, 2004.

Digital Library

[12]

Albert Greenberg et al. VL2: A Scalable and Flexible Data Center Network. In SIGCOMM, August 2009.

Digital Library

[13]

Chi-Yao Hong et al. Achieving High Utilization with Software-Driven WAN. In SIGCOMM, 2013.

Digital Library

[14]

Parveen Patel et al. Ananta: Cloud Scale Load Balancing. In ACM SIGCOMMM. ACM, 2013.

Digital Library

[15]

R. Chaiken et al. SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. In VLDB'08, 2008.

Digital Library

[16]

Sushant Jain et al. B4: Experience with a Globally-Deployed Software Defined WAN. In SIGCOMM, 2013.

Digital Library

[17]

Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google File System. In ACM SOSP. ACM, 2003.

Digital Library

[18]

Nicolas Guilbaud and Ross Cartlidge. Google Backbone Monitoring, Localizing Packet Loss in a Large Complex Network, Feburary 2013. Nanog57.

[19]

Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, David Mazi$\gravee$res, and Nick McKeown. I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks. In NSDI, 2014.

Digital Library

[20]

Michael Isard. Autopilot: Automatic Data Center Management. ACM SIGOPS Operating Systems Review, 2007.

Digital Library

[21]

Srikanth Kandula, Sudipta Sengupta, Albert Greenberg, Parveen Patel, and Ronnie Chaiken. The nature of data center traffic: Measurements & analysis. In Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement Conference, IMC '09, 2009.

Digital Library

[22]

Rishi Kapoor, Alex C. Snoeren, Geoffrey M. Voelker, and George Porter. Bullet Trains: A Study of NIC Burst Behavior at Microsecond Timescales. In ACM CoNEXT, 2013.

Digital Library

[23]

Cade Metz. Return of the Borg: How Twitter Rebuilt Google's Secret Weapon. http://www.wired.com/2013/03/google-borg-twitter-mesos/all/, March 2013.

[24]

Wenfei Wu, Guohui Wang, Aditya Akella, and Anees Shaikh. Virtual Network Diagnosis as a Service. In SoCC, 2013.

Digital Library

[25]

Hongyi Zeng, Peyman Kazemian, George Varghese, and Nick McKeown. Automatic Test Packet Generation. In CoNEXT, 2012.

Digital Library

Cited By

Kazmi AStaffolani AZhang TCabrera CClarke S(2025)Dynamic Service Placement in Edge Computing: A Comparative Evaluation of Nature-Inspired AlgorithmsIEEE Access10.1109/ACCESS.2024.352070113(2653-2670)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2024.3520701
Zhao DCheng GChen XZhao YZhang WLu LZhou SFu Y(2025)Probe-Optimizer: Discovering important nodes for proactive in-band network telemetry to achieve better probe orchestrationComputer Networks10.1016/j.comnet.2024.110935257(110935)Online publication date: Feb-2025
https://doi.org/10.1016/j.comnet.2024.110935
Wang ZHu HKong LKang XMa TXiang QLi JLu YSong ZYang PWu JYang YMa TLiu ZZeng XCai DChen GBagchi SZhang Y(2024)Diagnosing application-network anomalies for millions of IPs in production cloudsProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692046(885-899)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691992.3692046
Show More Cited By

Index Terms

Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
      1. Cloud computing
2. Networks
  1. Network performance evaluation
    1. Network measurement
  2. Network services
    1. Cloud computing
    2. Network monitoring

Recommendations

Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis
SIGCOMM'15

Can we get network latency between any two servers at any time in large-scale data center networks? The collected latency data can then be used to address a series of challenges: telling if an application perceived latency issue is caused by the network ...
R-Pingmesh: A Service-Aware RoCE Network Monitoring and Diagnostic System
ACM SIGCOMM '24: Proceedings of the ACM SIGCOMM 2024 Conference

RoCE services are sensitive to network failures and performance bottlenecks, which become more common as the RoCE network scales. In addition, some non-network problems behave like network problems and can waste troubleshooting time. However, existing ...
EPOXIDE: A Modular Prototype for SDN Troubleshooting
SIGCOMM'15

SDN opens a new chapter in network troubleshooting as besides misconfigurations and firmware/hardware errors, software bugs can occur all over the SDN stack. As an answer to this challenge the networking community developed a wealth of piecemeal SDN ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication

August 2015

684 pages

ISBN:9781450335423

DOI:10.1145/2785956

General Chairs:
Steve Uhlig
Queen Mary University of London, UK
,
Olaf Maennel
Tallinn U. of Technology in Estonia, Estonia
,
Program Chairs:
Brad Karp
University College London, UK
,
Jitendra Padhye
Microsoft, USA

ACM SIGCOMM Computer Communication Review Volume 45, Issue 4
SIGCOMM'15
October 2015
659 pages
ISSN:0146-4833
DOI:10.1145/2829988
Editors:
Konstantina Papagiannaki
Telefonica Research, Barcelona, Spain
,
Katerina Argyraki
EPFL, Switzerland
,
Hitesh Ballani
Microsoft Research Cambridge, UK
,
Fabián Bustamante
Northwestern University, USA
,
Joseph Camp
SMU, USA
,
Augustin Chaintreau
Columbia University, USA
,
Phillipa Gill
Stony Brook University, USA
,
Marco Mellia
Politecnico di Torino, Italy
,
Bhaskaran Raman
IIT Bombay, India
,
Joel Sommers
Colgate University, USA
,
Aline Carneiro Viana
INRIA, France
Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGCOMM '15

Sponsor:

SIGCOMM

SIGCOMM '15: ACM SIGCOMM 2015 Conference

August 17 - 21, 2015

London, United Kingdom

Acceptance Rates

SIGCOMM '15 Paper Acceptance Rate 40 of 242 submissions, 17%;

Overall Acceptance Rate 462 of 3,389 submissions, 14%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

357
Total Citations
View Citations
4,012
Total Downloads

Downloads (Last 12 months)940
Downloads (Last 6 weeks)106

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kazmi AStaffolani AZhang TCabrera CClarke S(2025)Dynamic Service Placement in Edge Computing: A Comparative Evaluation of Nature-Inspired AlgorithmsIEEE Access10.1109/ACCESS.2024.352070113(2653-2670)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2024.3520701
Zhao DCheng GChen XZhao YZhang WLu LZhou SFu Y(2025)Probe-Optimizer: Discovering important nodes for proactive in-band network telemetry to achieve better probe orchestrationComputer Networks10.1016/j.comnet.2024.110935257(110935)Online publication date: Feb-2025
https://doi.org/10.1016/j.comnet.2024.110935
Wang ZHu HKong LKang XMa TXiang QLi JLu YSong ZYang PWu JYang YMa TLiu ZZeng XCai DChen GBagchi SZhang Y(2024)Diagnosing application-network anomalies for millions of IPs in production cloudsProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692046(885-899)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691992.3692046
Zhang YZhang THao WWang SLiu NHe XZhang YWang WCheng YWang HXu JWang FJiang BGwon YNa JKim ZOh GBagchi SZhang Y(2024)MSFRDProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692045(869-884)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691992.3692045
Wang HAbhashkumar ALin CZhang TGu XMa NWu CLiu SZhou WDong YJiang WWang YVanbever LZhang I(2024)NetAssistantProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691935(2011-2023)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691935
Wang HTian HChen JWan XXia JZeng GBai WJiang JWang YChen KVanbever LZhang I(2024)Towards domain-specific network transport for distributed DNN trainingProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691904(1421-1443)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691904
Gao ZAbhashkumar ASun ZJiang WWang YVanbever LZhang I(2024)CrescentProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691883(1045-1062)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691883
Yassini PDiab KZangeneh SHefeeda MVanbever LZhang I(2024)HorusProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691826(1-22)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691826
Krähenbühl CTabaeiaghdaei SScherrer SFrei MPerrig A(2024)Toward Global Latency Transparency2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619858(536-542)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619858
Feng WGao JChen XAntichi GBasat RShao MZhang YYu M(2024)F3: Fast and Flexible Network Telemetry with an FPGA coprocessorProceedings of the ACM on Networking10.1145/36963972:CoNEXT4(1-22)Online publication date: 25-Nov-2024
https://dl.acm.org/doi/10.1145/3696397
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten