research-article

Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis

Authors:

Chuanxiong Guo,

Varugis KurienAuthors Info & Claims

ACM SIGCOMM Computer Communication Review, Volume 45, Issue 4

Pages 139 - 152

https://doi.org/10.1145/2829988.2787496

Published: 17 August 2015 Publication History

Abstract

Can we get network latency between any two servers at any time in large-scale data center networks? The collected latency data can then be used to address a series of challenges: telling if an application perceived latency issue is caused by the network or not, defining and tracking network service level agreement (SLA), and automatic network troubleshooting. We have developed the Pingmesh system for large-scale data center network latency measurement and analysis to answer the above question affirmatively. Pingmesh has been running in Microsoft data centers for more than four years, and it collects tens of terabytes of latency data per day. Pingmesh is widely used by not only network software developers and engineers, but also application and service developers and operators.

Supplementary Material

WEBM File (p139-guo.webm)

Download
161.80 MB

References

[1]

M. Al-Fares, A. Loukissas, and A. Vahdat. A Scalable, Commodity Data Center Network Architecture. In Proc. SIGCOMM, 2008.

Digital Library

[2]

Alexey Andreyev. Introducing data center fabric, the next-generation Facebook data center network. https://code.facebook.com/posts/360346274145943/, Nov 2014.

[3]

Hadoop. http://hadoop.apache.org/.

[4]

Peter Bailis and Kyle Kingsbury. The Network is Reliable: An Informal Survey of Real-World Communications Failures. ACM Queue, 2014.

Digital Library

[5]

Luiz Barroso, Jeffrey Dean, and Urs H$\ddoto$lzle. Web Search for a Planet: The Google Cluster Architecture. IEEE Micro, March-April 2003.

Digital Library

[6]

Theophilus Benson, Aditya Akella, and David A. Maltz. Network Traffic Characteristics of Data Centers in the Wild. In Internet Measurement Conference, November 2010.

Digital Library

[7]

et.al Brad Calder. Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency. In SOSP, 2011.

Digital Library

[8]

Cisco. IP SLAs Configuration Guide, Cisco IOS Release 12.4T. http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipsla/configuration/12--4t/sla-12--4t-book.pdf.

[9]

Citrix. What is Load Balancing? http://www.citrix.com/glossary/load-balancing.html.

[10]

Jeffrey Dean and Luiz Andr$\acutee$ Barroso. The Tail at Scale. CACM, Februry 2013.

Digital Library

[11]

Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In OSDI, 2004.

Digital Library

[12]

Albert Greenberg et al. VL2: A Scalable and Flexible Data Center Network. In SIGCOMM, August 2009.

Digital Library

[13]

Chi-Yao Hong et al. Achieving High Utilization with Software-Driven WAN. In SIGCOMM, 2013.

Digital Library

[14]

Parveen Patel et al. Ananta: Cloud Scale Load Balancing. In ACM SIGCOMMM. ACM, 2013.

Digital Library

[15]

R. Chaiken et al. SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. In VLDB'08, 2008.

Digital Library

[16]

Sushant Jain et al. B4: Experience with a Globally-Deployed Software Defined WAN. In SIGCOMM, 2013.

Digital Library

[17]

Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google File System. In ACM SOSP. ACM, 2003.

Digital Library

[18]

Nicolas Guilbaud and Ross Cartlidge. Google Backbone Monitoring, Localizing Packet Loss in a Large Complex Network, Feburary 2013. Nanog57.

[19]

Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, David Mazi$\gravee$res, and Nick McKeown. I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks. In NSDI, 2014.

Digital Library

[20]

Michael Isard. Autopilot: Automatic Data Center Management. ACM SIGOPS Operating Systems Review, 2007.

Digital Library

[21]

Srikanth Kandula, Sudipta Sengupta, Albert Greenberg, Parveen Patel, and Ronnie Chaiken. The nature of data center traffic: Measurements & analysis. In Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement Conference, IMC '09, 2009.

Digital Library

[22]

Rishi Kapoor, Alex C. Snoeren, Geoffrey M. Voelker, and George Porter. Bullet Trains: A Study of NIC Burst Behavior at Microsecond Timescales. In ACM CoNEXT, 2013.

Digital Library

[23]

Cade Metz. Return of the Borg: How Twitter Rebuilt Google's Secret Weapon. http://www.wired.com/2013/03/google-borg-twitter-mesos/all/, March 2013.

[24]

Wenfei Wu, Guohui Wang, Aditya Akella, and Anees Shaikh. Virtual Network Diagnosis as a Service. In SoCC, 2013.

Digital Library

[25]

Hongyi Zeng, Peyman Kazemian, George Varghese, and Nick McKeown. Automatic Test Packet Generation. In CoNEXT, 2012.

Digital Library

Cited By

Taka HInoue TOki E(2024)Design model of a twisted and folded Clos network with multi-step grouped intermediate switches guaranteeing admissible blocking probabilityJournal of Optical Communications and Networking10.1364/JOCN.51389816:3(328)Online publication date: 21-Feb-2024
https://doi.org/10.1364/JOCN.513898
Song EPan TSong HFu QLiu YJia CYuan CGao MZhang JHuang TLiu Y(2024)INT-Label: Lightweight In-Band Network-Wide Telemetry via Distributed LabelingIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.336793335:5(751-767)Online publication date: May-2024
https://doi.org/10.1109/TPDS.2024.3367933
Lu JPan THe SMiao MZhou GQi YZhang SSong ESun XZhao HLyu BZhu S(2024)CloudSentry: Two-Stage Heavy Hitter Detection for Cloud-Scale Gateway Overload ProtectionIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330185235:4(616-633)Online publication date: Apr-2024
https://doi.org/10.1109/TPDS.2023.3301852
Show More Cited By

Index Terms

Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
      1. Cloud computing
2. Networks
  1. Network performance evaluation
    1. Network measurement
  2. Network services
    1. Cloud computing
    2. Network monitoring

Recommendations

Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis
SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication

Can we get network latency between any two servers at any time in large-scale data center networks? The collected latency data can then be used to address a series of challenges: telling if an application perceived latency issue is caused by the network ...
R-Pingmesh: A Service-Aware RoCE Network Monitoring and Diagnostic System
ACM SIGCOMM '24: Proceedings of the ACM SIGCOMM 2024 Conference

RoCE services are sensitive to network failures and performance bottlenecks, which become more common as the RoCE network scales. In addition, some non-network problems behave like network problems and can waste troubleshooting time. However, existing ...
sRetor: a semi-centralized regular topology routing scheme for data center networking
Abstract
The performance of the data center network is critical for lowering costs and increasing efficiency. The software-defined networks (SDN) technique has been adopted in data center networks due to the recent emergence of advanced network control and ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGCOMM Computer Communication Review

ACM SIGCOMM Computer Communication Review Volume 45, Issue 4

SIGCOMM'15

October 2015

659 pages

ISSN:0146-4833

DOI:10.1145/2829988

Editors:
Konstantina Papagiannaki
Telefonica Research, Barcelona, Spain
,
Katerina Argyraki
EPFL, Switzerland
,
Hitesh Ballani
Microsoft Research Cambridge, UK
,
Fabián Bustamante
Northwestern University, USA
,
Joseph Camp
SMU, USA
,
Augustin Chaintreau
Columbia University, USA
,
Phillipa Gill
Stony Brook University, USA
,
Marco Mellia
Politecnico di Torino, Italy
,
Bhaskaran Raman
IIT Bombay, India
,
Joel Sommers
Colgate University, USA
,
Aline Carneiro Viana
INRIA, France

Issue’s Table of Contents

SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication
August 2015
684 pages
ISBN:9781450335423
DOI:10.1145/2785956
General Chairs:
Steve Uhlig
Queen Mary University of London, UK
,
Olaf Maennel
Tallinn U. of Technology in Estonia, Estonia
,
Program Chairs:
Brad Karp
University College London, UK
,
Jitendra Padhye
Microsoft, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2015

Published in SIGCOMM-CCR Volume 45, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

333
Total Citations
View Citations
3,554
Total Downloads

Downloads (Last 12 months)839
Downloads (Last 6 weeks)128

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Taka HInoue TOki E(2024)Design model of a twisted and folded Clos network with multi-step grouped intermediate switches guaranteeing admissible blocking probabilityJournal of Optical Communications and Networking10.1364/JOCN.51389816:3(328)Online publication date: 21-Feb-2024
https://doi.org/10.1364/JOCN.513898
Song EPan TSong HFu QLiu YJia CYuan CGao MZhang JHuang TLiu Y(2024)INT-Label: Lightweight In-Band Network-Wide Telemetry via Distributed LabelingIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.336793335:5(751-767)Online publication date: May-2024
https://doi.org/10.1109/TPDS.2024.3367933
Lu JPan THe SMiao MZhou GQi YZhang SSong ESun XZhao HLyu BZhu S(2024)CloudSentry: Two-Stage Heavy Hitter Detection for Cloud-Scale Gateway Overload ProtectionIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330185235:4(616-633)Online publication date: Apr-2024
https://doi.org/10.1109/TPDS.2023.3301852
Liu YXia YZhang WJia WWu J(2024)SFANT: A SRv6-Based Flexible and Active Network Telemetry Scheme in Programming Data PlaneIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.327700011:3(2415-2425)Online publication date: May-2024
https://doi.org/10.1109/TNSE.2023.3277000
Zhu SLu JLyu BPan TZhang SSun XJia CCheng XKang DLv YYang FXue XYang XWang ZYang J(2024)Proactive Telemetry in Large-Scale Multi-Tenant Cloud Overlay NetworksIEEE/ACM Transactions on Networking10.1109/TNET.2024.338178632:4(3002-3017)Online publication date: Aug-2024
https://doi.org/10.1109/TNET.2024.3381786
Gao WHuang JLi ZZou SWang J(2023)PDLB: Path Diversity-aware Load Balancing with adaptive granularity in data center networksJournal of Cloud Computing10.1186/s13677-023-00548-x12:1Online publication date: 7-Dec-2023
https://doi.org/10.1186/s13677-023-00548-x
Wang BChen HChen PHe ZYu G(2023)MARS: Fault Localization in Programmable Networking Systems with Low-cost In-Band Network TelemetryProceedings of the 52nd International Conference on Parallel Processing10.1145/3605573.3605622(347-357)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3605573.3605622
Li XXie KWang XXie GLi KCao JZhang DWen J(2023)Tripartite Graph Aided Tensor Completion For Sparse Network MeasurementIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.321325934:1(48-62)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TPDS.2022.3213259
Lin FWang HChen GZhou GXu TWei DChen LLu YQu AShao HJiang H(2023)Fast, Scalable and Robust Centralized Routing for Data Center NetworksIEEE/ACM Transactions on Networking10.1109/TNET.2023.325954131:6(2624-2639)Online publication date: Dec-2023
https://doi.org/10.1109/TNET.2023.3259541
Miao RZhang YZheng ZWang RZhang RYang TLiu ZJiang J(2023)CocoSketch: High-Performance Sketch-Based Measurement Over Arbitrary Partial Key QueryIEEE/ACM Transactions on Networking10.1109/TNET.2023.325722631:6(2653-2668)Online publication date: Dec-2023
https://doi.org/10.1109/TNET.2023.3257226
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents