research-article

Open access

Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network

Authors:

Ashby Armistead,

Paulie Germano,

Anand Kanagala,

Stephen Stuart,

Amin VahdatAuthors Info & Claims

ACM SIGCOMM Computer Communication Review, Volume 45, Issue 4

Pages 183 - 197

https://doi.org/10.1145/2829988.2787508

Published: 17 August 2015 Publication History

Abstract

We present our approach for overcoming the cost, operational complexity, and limited scale endemic to datacenter networks a decade ago. Three themes unify the five generations of datacenter networks detailed in this paper. First, multi-stage Clos topologies built from commodity switch silicon can support cost-effective deployment of building-scale networks. Second, much of the general, but complex, decentralized network routing and management protocols supporting arbitrary deployment scenarios were overkill for single-operator, pre-planned datacenter networks. We built a centralized control mechanism based on a global configuration pushed to all datacenter switches. Third, modular hardware design coupled with simple, robust software allowed our design to also support inter-cluster and wide-area networks. Our datacenter networks run at dozens of sites across the planet, scaling in capacity by 100x over ten years to more than 1Pbps of bisection bandwidth.

Supplementary Material

WEBM File (p183-singh.webm)

Download
151.03 MB

References

[1]

Ahn, J. H., Binkert, N., Davis, A., McLaren, M., and Schreiber, R. S. HyperX: topology, routing, and packaging of efficient large-scale networks. In Proc. High Performance Computing Networking, Storage and Analysis (2009), ACM, p. 41.

Digital Library

[2]

Al-Fares, M., Loukissas, A., and Vahdat, A. A scalable, commodity data center network architecture. In ACM SIGCOMM Computer Communication Review (2008), vol. 38, ACM, pp. 63--74.

Digital Library

[3]

Alizadeh, M., Greenberg, A., Maltz, D. A., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., and Sridharan, M. Data center TCP (DCTCP). ACM SIGCOMM computer communication review 41, 4 (2011), 63--74.

Digital Library

[4]

Barroso, L. A., Dean, J., and Holzle, U. Web search for a planet: The Google cluster architecture. Micro, Ieee 23, 2 (2003), 22--28.

Digital Library

[5]

Barroso, L. A., and Hölzle, U. The datacenter as a computer: An introduction to the design of warehouse-scale machines. Synthesis lectures on computer architecture 4, 1 (2009), 1--108.

Digital Library

[6]

Bates, T., Chen, E., and Chandra, R. Bgp route reflection: An alternative to full mesh internal bgp (ibgp). RFC 4456, RFC Editor, April 2006. http://www.rfc-editor.org/rfc/rfc4456.txt.

[7]

Calder, B., Wang, J., Ogus, A., Nilakantan, N., Skjolsvold, A., McKelvie, S., Xu, Y., Srivastav, S., Wu, J., Simitci, H., et al. Windows Azure Storage: a highly available cloud storage service with strong consistency. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles (2011), ACM, pp. 143--157.

Digital Library

[8]

Chen, Y., Griffith, R., Liu, J., Katz, R. H., and Joseph, A. D. Understanding TCP incast throughput collapse in datacenter networks. In Proceedings of the 1st ACM workshop on Research on enterprise networking (2009), ACM, pp. 73--82.

Digital Library

[9]

Clos, C. A Study of Non-Blocking Switching Networks. Bell System Technical Journal 32, 2 (1953), 406--424.

[10]

Dean, J., and Ghemawat, S. MapReduce: simplified data processing on large clusters. Communications of the ACM 51, 1 (2008), 107--113.

Digital Library

[11]

Dietz, H. G., and Mattox, T. I. KLAT2's flat neighborhood network. Proceedings of the Extreme Linux track in the 4th Annual Linux Showcase, Atlanta, GA (2000).

Digital Library

[12]

Farrington, N., Rubow, E., and Vahdat, A. Data center switch architecture in the age of merchant silicon. In Proc. HOT Interconnects, 2009. 17th IEEE Symposium on (2009), pp. 93--102.

Digital Library

[13]

Feamster, N., Rexford, J., and Zegura, E. The Road to SDN: An Intellectual History of Programmable Networks. ACM Queue 11, 12 (December 2013).

Digital Library

[14]

Ghemawat, S., Gobioff, H., and Leung, S.-T. The Google file system. In ACM SIGOPS Operating Systems Review (2003), vol. 37, ACM, pp. 29--43.

Digital Library

[15]

Greenberg, A., Hamilton, J. R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D. A., Patel, P., and Sengupta, S. VL2: a scalable and flexible data center network. In Proc. ACM SIGCOMM Computer Communication Review (2009), pp. 51--62.

Digital Library

[16]

Guo, C., Lu, G., Li, D., Wu, H., Zhang, X., Shi, Y., Tian, C., Zhang, Y., and Lu, S. BCube: A high performance, server-centric network architecture for modular data centers. In Proc. ACM SIGCOMM (2009), pp. 63--74.

Digital Library

[17]

Guo, C., Wu, H., Tan, K., Shi, L., Zhang, Y., and Lu, S. Dcell: a scalable and fault-tolerant network structure for data centers. ACM SIGCOMM Computer Communication Review 38, 4 (2008), 75--86.

Digital Library

[18]

Isard, M., Budiu, M., Yu, Y., Birrell, A., and Fetterly, D. Dryad: distributed data-parallel programs from sequential building blocks. In Proc. ACM SIGOPS Operating Systems Review (2007), pp. 59--72.

Digital Library

[19]

Jain, S., Kumar, A., Mandal, S., Ong, J., Poutievski, L., Singh, A., Venkata, S., Wanderer, J., Zhou, J., Zhu, M., Zolla, J., Hölzle, U., Stuart, S., and Vahdat, A. B4: Experience with a globally-deployed software defined WAN. In Proc. ACM SIGCOMM (2013), pp. 3--14.

Digital Library

[20]

Moy, J. OSPF Version 2. STD 54, RFC Editor, April 1998. http://www.rfc-editor.org/rfc/rfc2328.txt.

[21]

Prakash, P., Dixit, A. A., Hu, Y. C., and Kompella, R. R. The TCP Outcast Problem: Exposing Unfairness in Data Center Networks. In Proc. NSDI (2012), pp. 413--426.

Digital Library

[22]

Singla, A., Hong, C.-Y., Popa, L., and Godfrey, P. B. Jellyfish: Networking Data Centers Randomly. In NSDI (2012), vol. 12, pp. 17--17.

Digital Library

[23]

Thorup, M. OSPF Areas Considered Harmful. IETF Internet Draft 00, individual, April 2003. http://tools.ietf.org/html/draft-thorup-ospf-harmful-00.

[24]

Vahdat, A., Al-Fares, M., Farrington, N., Mysore, R. N., Porter, G., and Radhakrishnan, S. Scale-Out Networking in the Data Center. IEEE MICRO, 4 (August 2010), 29--41.

Digital Library

Cited By

Dou SGuo Z(2025)Maintaining Predictable Traffic Engineering Performance Under Controller Failures for Software-Defined WANsIEEE Journal on Selected Areas in Communications10.1109/JSAC.2025.352881443:2(524-536)Online publication date: Feb-2025
https://doi.org/10.1109/JSAC.2025.3528814
Li SLi LDou WZhang YWang CCui XLi L(2025)SDLoRe: A loss recovery algorithm based on segment detection in lossy RDMA networksComputer Networks10.1016/j.comnet.2024.111019258(111019)Online publication date: Feb-2025
https://doi.org/10.1016/j.comnet.2024.111019
Inoue TMano TAnazawa KUno T(2024)Nonblocking conditions for Clos fabrics with non-uniform switch radixesJournal of Optical Communications and Networking10.1364/JOCN.54079217:1(28)Online publication date: 17-Dec-2024
https://doi.org/10.1364/JOCN.540792
Show More Cited By

Index Terms

Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network
1. Networks
  1. Network types
    1. Data center networks

Recommendations

Jupiter rising: a decade of clos topologies and centralized control in Google's datacenter network

We present our approach for overcoming the cost, operational complexity, and limited scale endemic to datacenter networks a decade ago. Three themes unify the five generations of datacenter networks detailed in this paper. First, multi-stage Clos ...
Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network
SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication

We present our approach for overcoming the cost, operational complexity, and limited scale endemic to datacenter networks a decade ago. Three themes unify the five generations of datacenter networks detailed in this paper. First, multi-stage Clos ...
AC/DC TCP: Virtual Congestion Control Enforcement for Datacenter Networks
SIGCOMM '16: Proceedings of the 2016 ACM SIGCOMM Conference

Multi-tenant datacenters are successful because tenants can seamlessly port their applications and services to the cloud. Virtual Machine (VM) technology plays an integral role in this success by enabling a diverse set of software to be run on a unified ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGCOMM Computer Communication Review

ACM SIGCOMM Computer Communication Review Volume 45, Issue 4

SIGCOMM'15

October 2015

659 pages

ISSN:0146-4833

DOI:10.1145/2829988

Editors:
Konstantina Papagiannaki
Telefonica Research, Barcelona, Spain
,
Katerina Argyraki
EPFL, Switzerland
,
Hitesh Ballani
Microsoft Research Cambridge, UK
,
Fabián Bustamante
Northwestern University, USA
,
Joseph Camp
SMU, USA
,
Augustin Chaintreau
Columbia University, USA
,
Phillipa Gill
Stony Brook University, USA
,
Marco Mellia
Politecnico di Torino, Italy
,
Bhaskaran Raman
IIT Bombay, India
,
Joel Sommers
Colgate University, USA
,
Aline Carneiro Viana
INRIA, France

Issue’s Table of Contents

SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication
August 2015
684 pages
ISBN:9781450335423
DOI:10.1145/2785956
General Chairs:
Steve Uhlig
Queen Mary University of London, UK
,
Olaf Maennel
Tallinn U. of Technology in Estonia, Estonia
,
Program Chairs:
Brad Karp
University College London, UK
,
Jitendra Padhye
Microsoft, USA

Copyright © 2015 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2015

Published in SIGCOMM-CCR Volume 45, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

617
Total Citations
View Citations
10,492
Total Downloads

Downloads (Last 12 months)2,479
Downloads (Last 6 weeks)321

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dou SGuo Z(2025)Maintaining Predictable Traffic Engineering Performance Under Controller Failures for Software-Defined WANsIEEE Journal on Selected Areas in Communications10.1109/JSAC.2025.352881443:2(524-536)Online publication date: Feb-2025
https://doi.org/10.1109/JSAC.2025.3528814
Li SLi LDou WZhang YWang CCui XLi L(2025)SDLoRe: A loss recovery algorithm based on segment detection in lossy RDMA networksComputer Networks10.1016/j.comnet.2024.111019258(111019)Online publication date: Feb-2025
https://doi.org/10.1016/j.comnet.2024.111019
Inoue TMano TAnazawa KUno T(2024)Nonblocking conditions for Clos fabrics with non-uniform switch radixesJournal of Optical Communications and Networking10.1364/JOCN.54079217:1(28)Online publication date: 17-Dec-2024
https://doi.org/10.1364/JOCN.540792
Anazawa KInoue TMano TNishizawa HOki E(2024)Efficient fiber-inspection and certification method for optical-circuit-switched datacenter networksJournal of Optical Communications and Networking10.1364/JOCN.52779416:8(788)Online publication date: 9-Jul-2024
https://doi.org/10.1364/JOCN.527794
Taka HInoue TOki E(2024)Design model of a twisted and folded Clos network with multi-step grouped intermediate switches guaranteeing admissible blocking probabilityJournal of Optical Communications and Networking10.1364/JOCN.51389816:3(328)Online publication date: 21-Feb-2024
https://doi.org/10.1364/JOCN.513898
Sharma SJukan AMalik AGumaste A(2024)A Network Calculus Model for SFC Realization and Traffic Bounds Estimation in Data CentersACM Transactions on Internet Technology10.1145/370044024:4(1-32)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3700440
Vijayakumar SMadhavapeddy AKalyvianaki E(2024)Scheduling for Reduced Tail Task Latencies in Highly Utilized DatacentersProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698522(302-321)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698522
Kumar ADevraj ABunandar DSingh R(2024)A case for server-scale photonic connectivityProceedings of the 23rd ACM Workshop on Hot Topics in Networks10.1145/3696348.3696856(290-299)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3696348.3696856
Ferreira MAtre NSherry JSobrinho JKuznetsov PGelles ROlivetti D(2024)Impossibility Results for Data-Center Routing with Congestion Control and Unsplittable FlowsProceedings of the 43rd ACM Symposium on Principles of Distributed Computing10.1145/3662158.3662777(358-368)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3662158.3662777
Weintraub BKim JTao RNita-Rotaru COkhravi HTian DUjcich BLuo BLiao XXu JKirda ELie D(2024)Exploiting Temporal Vulnerabilities for Unauthorized Access in Intent-based NetworkingProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security10.1145/3658644.3670301(3630-3644)Online publication date: 2-Dec-2024
https://dl.acm.org/doi/10.1145/3658644.3670301
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Issue’s Table of Contents