Topaz: Declarative and Verifiable Authoritative DNS at CDN-Scale
Authors: 
James Larisch, Timothy Alberdingk Thijm, Suleman Ahmad, Peter Wu, Tom Arnfeld, Marwan FayedAuthors Info & Claims
James Larisch, Timothy Alberdingk Thijm, Suleman Ahmad, Peter Wu, Tom Arnfeld, Marwan FayedAuthors Info & ClaimsPages 891 - 903
Abstract
Today, when a CDN nameserver receives a DNS query for a customer's domain, it decides which CDN IP to return based on servicelevel objectives such as managing load or maintaining performance, but also internal needs like split testing. Many of these decisions are made a priori by assignment systems that imperatively generate maps from DNS query to IP address(es). Unfortunately, imperative assignments obfuscate nameserver behavior, especially when different objectives conflict.
In this paper we present Topaz, a new authoritative nameserver architecture for anycast CDNs which encodes DNS objectives as declarative, modular programs called policies. Nameservers execute policies directly in response to live queries. To understand or change DNS behavior, operators simply read or modify the list of policy programs. In addition, because policies are written in a formally-verified domain-specific language (topaz-lang), Topaz can detect policy conflicts before deployment. Topaz handles ~1M DNS queries per second at a global CDN, dynamically deciding addresses for millions of names on six continents. We evaluate Topaz and show that the latency overheads it introduces are acceptable.
References
[1]
LuaDNS. https://www.luadns.com/.
[2]
PowerDNS. https://doc.powerdns.com/recursor/index.html.
[3]
Policy-based control for cloud native environments, 2021.
[4]
Carolyn Jane Anderson, Nate Foster, Arjun Guha, Jean-Baptiste Jeannin, Dexter Kozen, Cole Schlesinger, and David Walker. NetKAT: Semantic foundations for networks. ACM SIGPLAN Notices, 49(1):113--126, 2014.
[5]
Abbie Barbir, Brad Cain, Raj Nair, and Oliver Spatscheck. Known Content Network (CN) Request-Routing Mechanisms. Technical report, 2003.
[6]
Clark W. Barrett, Christopher L. Conway, Morgan Deters, Liana Hadarean, Dejan Jovanovic, Tim King, Andrew Reynolds, and Cesare Tinelli. CVC4. In Ganesh Gopalakrishnan and Shaz Qadeer, editors, Computer Aided Verification - 23rd International Conference, CAV 2011, Snowbird, UT, USA, July 14--20, 2011. Proceedings, volume 6806 of Lecture Notes in Computer Science, pages 171--177. Springer, 2011.
[7]
Ryan Beckett, Ratul Mahajan, Todd Millstein, Jitendra Padhye, and David Walker. Don't mind the gap: Bridging network-wide objectives and device-level configurations. In Proceedings of the 2016 ACM SIGCOMM Conference, pages 328--341, 2016.
[8]
Thomas P. Brisco. DNS Support for Load Balancing. RFC 1794, April 1995.
[9]
Matt Calder, Ashley Flavel, Ethan Katz-Bassett, Ratul Mahajan, and Jitendra Padhye. Analyzing the performance of an anycast cdn. In Proceedings of the 2015 Internet Measurement Conference, pages 531--537, 2015.
[10]
Martin Casado, Michael J Freedman, Justin Pettit, Jianying Luo, Nick McKeown, and Scott Shenker. Ethane: Taking control of the enterprise. ACM SIGCOMM computer communication review, 37(4):1--12, 2007.
[11]
Fangfei Chen, Ramesh K Sitaraman, and Marcelo Torres. End-User Mapping: Next Generation Request Routing for Content Delivery. ACM SIGCOMM Computer Communication Review, 45(4):167--181, 2015.
[12]
David Chou, Tianyin Xu, Kaushik Veeraraghavan, Andrew Newell, Sonia Margulis, Lin Xiao, Pol Mauri Ruiz, Justin Meza, Kiryong Ha, Shruti Padmanabha, et al. Taiji: Managing Global User Traffic for Large-Scale Internet Services at the Edge. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 430--446, 2019.
[13]
Alexander Clemm, Laurent Ciavaglia, Lisandro Zambenedetti Granville, and Jeff Tantsura. Intent-Based Networking - Concepts and Definitions. RFC 9315, October 2022.
[14]
Joseph W Cutler, Craig Disselkoen, Aaron Eline, Shaobo He, Kyle Headley, Michael Hicks, Kesha Hietala, Eleftherios Ioannidis, John Kastner, Anwar Mamat, et al. Cedar: A new language for expressive, fast, safe, and analyzable authorization. Proceedings of the ACM on Programming Languages, 8(OOPSLA1):670--697, 2024.
[15]
Gonçalo Grilo David Tuber, Luke Orden. How Cloudflare's systems dynamically route traffic across the globe, September 2023. https://blog.cloudflare.com/meet-traffic-manager.
[16]
Leonardo De Moura and Nikolaj Bjørner. Z3: An efficient smt solver. In Tools and Algorithms for the Construction and Analysis of Systems: 14th International Conference, TACAS 2008, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2008, Budapest, Hungary, March 29-April 6, 2008. Proceedings 14, pages 337--340. Springer, 2008.
[17]
John Dilley, Bruce Maggs, Jay Parikh, Harald Prokop, Ramesh Sitaraman, and Bill Weihl. Globally distributed content delivery. IEEE Internet Computing, 6(5):50--58, 2002.
[18]
Marwan Fayed, Lorenz Bauer, Vasileios Giotsas, Sami Kerola, Marek Majkowski, Pavel Odintsov, Jakub Sitnicki, Taejoong Chung, Dave Levin, Alan Mislove, Christopher A. Wood, and Nick Sullivan. The Ties That Un-Bind: Decoupling IP from Web Services and Sockets for Robust Addressing Agility at CDN-Scale. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference, SIGCOMM '21, page 433--446, New York, NY, USA, 2021. Association for Computing Machinery.
[19]
Ashley Flavel, Pradeepkumar Mani, David Maltz, Nick Holt, Jie Liu, Yingying Chen, and Oleg Surmachev. FastRoute: A Scalable Load-Aware Anycast Routing Architecture for Modern CDNs. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), pages 381--394, 2015.
[20]
Withheld for anonymous review, October 2022.
[21]
Nate Foster, Rob Harrison, Michael J Freedman, Christopher Monsanto, Jennifer Rexford, Alec Story, and David Walker. Frenetic: A network programming language. ACM Sigplan Notices, 46(9):279--291, 2011.
[22]
Nick Giannarakis, Devon Loehr, Ryan Beckett, and David Walker. Nv: An intermediate language for verification of network control planes. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 958--973, 2020.
[23]
Petros Gigis, Matt Calder, Lefteris Manassakis, George Nomikos, Vasileios Kotronis, Xenofontas Dimitropoulos, Ethan Katz-Bassett, and Georgios Smaragdakis. Seven Years in the Life of Hypergiants' Off-Nets. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference, pages 516--533, 2021.
[24]
Timothy L Hinrichs, Natasha S Gude, Martin Casado, John C Mitchell, and Scott Shenker. Practical declarative network management. In Proceedings of the 1st ACM workshop on Research on enterprise networking, pages 1--10, 2009.
[25]
Kuo-Feng Hsu, Ryan Beckett, Ang Chen, Jennifer Rexford, Praveen Tammana, and David Walker. Contra: A programmable system for performance-aware routing. In 17th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2020, 2020.
[26]
Karthick Jayaraman, Nikolaj Bjørner, Geoff Outhred, and Charlie Kaufman. Automated analysis and debugging of network connectivity policies. Microsoft Research, pages 1--11, 2014.
[27]
Siva Kesava Reddy Kakarla, Ryan Beckett, Behnaz Arzani, Todd Millstein, and George Varghese. Groot: Proactive verification of dns configurations. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, pages 310--328, 2020.
[28]
Siva Kesava Reddy Kakarla, Ryan Beckett, Todd Millstein, and George Varghese. SCALE: Automatically Finding RFC Compliance Bugs in DNS Nameservers. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 307--323, 2022.
[29]
Hongqiang Harry Liu, Raajay Viswanathan, Matt Calder, Aditya Akella, Ratul Mahajan, Jitendra Padhye, and Ming Zhang. Efficiently Delivering Online Services over Integrated Infrastructure. page 15, 2013.
[30]
Si Liu, Huayi Duan, Lukas Heimes, Marco Bearzi, Jodok Vieli, David Basin, and Adrian Perrig. A formal framework for end-to-end dns resolution. In Proceedings of the ACM SIGCOMM 2023 Conference, pages 932--949, 2023.
[31]
Zhenhua Liu, Minghong Lin, Adam Wierman, Steven H Low, and Lachlan LH Andrew. Greening Geographical Load Balancing. ACM SIGMETRICS Performance Evaluation Review, 39(1):193--204, 2011.
[32]
Nuno P Lopes, Nikolaj Bjørner, Patrice Godefroid, Karthick Jayaraman, and George Varghese. Checking beliefs in dynamic networks. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), pages 499--512, 2015.
[33]
Bruce M Maggs and Ramesh K Sitaraman. Algorithmic Nuggets in Content Delivery. ACM SIGCOMM Computer Communication Review, 45(3):52--66, 2015.
[34]
Ajay Mahimkar, Carlos Eduardo de Andrade, Rakesh Sinha, and Giritharan Rana. A Composition Framework for Change Management. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference, pages 788--806. ACM, August 2021.
[35]
Erik Nygren, Ramesh K Sitaraman, and Jennifer Sun. The Akamai Network: A Platform for High-Performance Internet Applications. ACM SIGOPS Operating Systems Review, 44(3):2--19, 2010.
[36]
Dejan Grofelnik Pelzel. We're transforming internet routing: Introducing Bunny DNS!, March 2022. https://bunny.net/blog/transforming-internet-routing-introducing-bunny-dns/.
[37]
Geoffrey Plouvier. Introducing Quicksilver: Configuration Distribution at Internet Scale, March 2020. https://blog.cloudflare.com/introducing-quicksilver-configuration-distribution-at-internet-scale/.
[38]
Racket. The Racket programming language, 2023.
[39]
Mark Reitblatt, Marco Canini, Arjun Guha, and Nate Foster. FatTire: Declarative fault tolerance for software-defined networks. In Proceedings of the Second ACM SIGCOMM Workshop on Hot Topics in Software Defined Networking, HotSDN '13, page 109--114, New York, NY, USA, 2013. Association for Computing Machinery.
[40]
Eric Rescorla, Kazuho Oku, Nick Sullivan, and Christopher A. Wood. TLS Encrypted Client Hello. Internet-Draft draft-ietf-tls-esni, Internet Engineering Task Force, October 2023. Work in Progress.
[41]
Brandon Schlinker, Hyojeong Kim, Timothy Cui, Ethan Katz-Bassett, Harsha V Madhyastha, Italo Cunha, James Quinn, Saif Hasan, Petr Lapukhov, and Hongyi Zeng. Engineering Egress with Edge Fabric: Steering Oceans of Content to the World. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pages 418--431, 2017.
[42]
Kyle Schomp, Onkar Bhardwaj, Eymen Kurdoglu, Mashooq Muhaimen, and Ramesh K Sitaraman. Akamai DNS: Providing Authoritative Answers to the World's Queries. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, pages 465--478, 2020.
[43]
OASIS Standard. extensible access control markup language (xacml) version 3.0. A:(22 January 2013). URl: http://docs.oasis-open.org/xacml/3.0/xacml-3.0-core-spec-os-en.html, 2013.
[44]
Ao-Jan Su, David R Choffnes, Aleksandar Kuzmanovic, and Fabian E Bustamante. Drafting Behind Akamai: Inferring Network Conditions Based on CDN Redirections. IEEE/ACM transactions on networking, 17(6):1752--1765, 2009.
[45]
Mani Sundaram. Akamai Summarizes Service Disruption (RESOLVED), 2021. https://www.akamai.com/blog/news/akamai-summarizes-service-disruption-resolved.
[46]
Yu-Wei Eric Sung, Xiaozheng Tie, Starsky H.Y. Wong, and Hongyi Zeng. Robotron: Top-down Network Management at Facebook Scale. In Proceedings of the 2016 ACM SIGCOMM Conference, pages 426--439. ACM, August 2016.
[47]
Emina Torlak and Rastislav Bodik. A lightweight symbolic virtual machine for solver-aided host languages. ACM SIGPLAN Notices, 49(6):530--541, 2014.
[48]
Konstantin Weitz, Doug Woos, Emina Torlak, Michael D. Ernst, Arvind Krishnamurthy, and Zachary Tatlock. Scalable verification of border gateway protocol configurations with an smt solver. In OOPSLA, page 765--780, New York, NY, USA, 2016. Association for Computing Machinery.
[49]
David Wragg. Unimog - Cloudflare's edge load balancer, September 2020. https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer.
[50]
Kok-Kiong Yap, Murtaza Motiwala, Jeremy Rahe, Steve Padgett, Matthew Holliman, Gary Baldus, Marcus Hines, Taeeun Kim, Ashok Narayanan, Ankur Jain, Victor Lin, Colin Rice, Brian Rogan, Arjun Singh, Bert Tanaka, Manish Verma, Puneet Sood, Mukarram Tariq, Matt Tierney, Dzevad Trumic, Vytautas Valancius, Calvin Ying, Mahesh Kallahalla, Bikash Koley, and Amin Vahdat. Taking the Edge off with Espresso: Scale, Reliability and Programmability for Global Internet Peering. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pages 432--445, Los Angeles CA USA, August 2017. ACM.
[51]
Ólafur Guðmundsson. 1.1.1.1 lookup failures on October 4th, 2023, 2023. https://blog.cloudflare.com/1-1-1-1-lookup-failures-on-october-4th-2023/.
Index Terms
- Topaz: Declarative and Verifiable Authoritative DNS at CDN-Scale
Recommendations
A study of the impact of DNS resolvers on CDN performance using a causal approach
Resources such as Web pages or videos that are published in the Internet are referred to by their Uniform Resource Locator (URL). If a user accesses a resource via its URL, the host name part of the URL needs to be translated into a routable IP address. ...
Analyzing the Performance of an Anycast CDN
IMC '15: Proceedings of the 2015 Internet Measurement ConferenceContent delivery networks must balance a number of trade-offs when deciding how to direct a client to a CDN server. Whereas DNS-based redirection requires a complex global traffic manager, anycast depends on BGP to direct a client to a CDN front-end. ...
A Technique to Assign an Appropriate Server to a Client, for a CDN Consists of Servers at the Global Internet and Hierarchical Private Networks
COMPSACW '14: Proceedings of the 2014 IEEE 38th International Computer Software and Applications Conference WorkshopsThis paper discusses a technique to assign an appropriate server to a client for a content delivery network (CDN). We assume that the CDN consists of not only servers in the global Internet but also servers in hierarchical private networks. To use a ...
Comments
Information & Contributors
Information
Published In
August 2024
1033 pages
ISBN:9798400706141
DOI:10.1145/3651890
- Co-chairs:
- Aruna Seneviratne,
- Darryl Veitch,
- Program Co-chairs:
- Vyas Sekar,
- Minlan Yu
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ACM SIGCOMM '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 462 of 3,389 submissions, 14%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 71Total Downloads
- Downloads (Last 12 months)71
- Downloads (Last 6 weeks)71
Reflects downloads up to 10 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in