research-article

Public Access

Test coverage metrics for the network

Authors:

Karthick Jayaraman,

David WalkerAuthors Info & Claims

SIGCOMM '21: Proceedings of the 2021 ACM SIGCOMM 2021 Conference

Pages 775 - 787

https://doi.org/10.1145/3452296.3472941

Published: 09 August 2021 Publication History

Abstract

Testing and verification have emerged as key tools in the battle to improve the reliability of networks and the services they provide. However, the success of even the best technology of this sort is limited by how effectively it is applied, and in today's enormously complex industrial networks, it is surprisingly easy to overlook particular interfaces, routes, or flows when creating a test suite. Moreover, network engineers, unlike their software counterparts, have no help to battle this problem—there are no metrics or systems to compute the quality of their test suites or the extent to which their networks have been verified.

To address this gap, we develop a general framework to define and compute network coverage for stateless network data planes. It computes coverage for a range of network components (\EG, interfaces, devices, paths) and supports many types of tests (e.g., concrete versus symbolic; local versus end-to-end; tests that check network state versus those that analyze behavior). Our framework is based on the observation that any network dataplane component can be decomposed into forwarding rules and all types of tests ultimately exercise these rules using one or more packets.

We build a system called Yardstick based on this framework and deploy it in Microsoft Azure. Within the first month of its deployment inside one of the production networks, it uncovered several testing gaps and helped improve testing by covering 89% more forwarding rules and 17% more network interfaces.

Supplementary Material

shaikh-public-review (975-public-review.pdf)

Test Coverage Metrics for the Network: Public Review

Download
65.71 KB

MP4 File (video-conference.mp4)

Conference Presentation Video

Download
65.61 MB

References

[1]

Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat. 2008. A Scalable, Commodity Data Center Network Architecture. In Proceedings of SIGCOMM '08. ACM, 63--74.

Digital Library

[2]

Paul Ammann and Jeff Offutt. 2016. Introduction to Software Testing. Cambridge University Press.

[3]

Carolyn Jane Anderson, Nate Foster, Arjun Guha, Jean-Baptiste Jeannin, Dexter Kozen, Cole Schlesinger, and David Walker. 2014. NetKAT: Semantic Foundations for Networks. In Proceedings of POPL '14. ACM, 113--126.

Digital Library

[4]

Mae Anderson. 2014. Time Warner Cable Says Outages Largely Resolved. http://www.seattletimes.com/business/time-warner-cable-says-outages-largely-resolved Retrieved June 23, 2021 from

[5]

James H Andrews, Lionel C Briand, Yvan Labiche, and Akbar Siami Namin. 2006. Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Transactions on Software Engineering 32, 8 (2006), 608--624.

Digital Library

[6]

John Backes, Sam Bayless, Byron Cook, Catherine Dodge, Andrew Gacek, Alan J Hu, Temesghen Kahsai, Bill Kocik, Evgenii Kotelnikov, Jure Kukovec, et al. 2019. Reachability analysis for AWS-based networks. In International Conference on Computer Aided Verification. Springer, 231--241.

[7]

Ryan Beckett and Ratul Mahajan. 2019. Putting Network Verification to Good Use. In Proceedings of HotNets '19. ACM, 77--84.

Digital Library

[8]

Larry Brader, Howie Hilliker, and Alan Wills. 2013. Testing for Continuous Delivery with Visual Studio 2012. Microsoft.

[9]

Mosharaf Chowdhury, Yuan Zhong, and Ion Stoica. 2014. Efficient coflow scheduling with varys. In Proceedings of SIGCOMM '14. ACM, 443--454.

Digital Library

[10]

Codecov. 2021. CodeCov: The leading code coverage solution. https://about.codecov.io/ Retrieved June 23, 2021 from

[11]

Ari Fogel, Stanley Fung, Luis Pedrosa, Meg Walraed-Sullivan, Ramesh Govindan, Ratul Mahajan, and Todd Millstein. 2015. A General Approach to Network Configuration Analysis. In Proceedings of NSDI 15. USENIX Association, 469--483.

[12]

Milos Gligoric, Alex Groce, Chaoqiang Zhang, Rohan Sharma, Mohammad Amin Alipour, and Darko Marinov. 2013. Comparing Non-Adequate Test Suites Using Coverage Criteria. In Proceedings of the 2013 International Symposium on Software Testing and Analysis (ISSTA 2013). 302--313.

Digital Library

[13]

Albert Greenberg, James R. Hamilton, Navendu Jain, Srikanth Kandula, Changhoon Kim, Parantap Lahiri, David A. Maltz, Parveen Patel, and Sudipta Sengupta. 2009. VL2: A Scalable and Flexible Data Center Network. In Proceedings of SIGCOMM '09. ACM, 51--62.

Digital Library

[14]

Chuanxiong Guo, Lihua Yuan, Dong Xiang, Yingnong Dang, Ray Huang, Dave Maltz, Zhaoyi Liu, Vin Wang, Bin Pang, Hua Chen, Zhi-Wei Lin, and Varugis Kurien. 2015. Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis. In Proceedings of SIGCOMM '15. ACM, 139--152.

Digital Library

[15]

Alex Horn, Ali Kheradmand, and Mukul Prasad. 2017. Delta-net: Real-time Network Verification Using Atoms. In Proceedings of NSDI 17. USENIX Association, 735--749.

[16]

Monica Hutchins, Herb Foster, Tarak Goradia, and Thomas Ostrand. 1994. Experiments on the effectiveness of dataflow-and control-flow-based test adequacy criteria. In Proceedings of 16th International conference on Software engineering. IEEE, 191--200.

Digital Library

[17]

Marko Ivanković, Goran Petrović, René Just, and Gordon Fraser. 2019. Code coverage at Google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 955--963.

Digital Library

[18]

Karthick Jayaraman, Nikolaj Bjørner, Geoff Outhred, and Charlie Kaufman. 2014. Automated analysis and debugging of network connectivity policies. Technical Report. 1--11 pages. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/secguru.pdf

[19]

Karthick Jayaraman, Nikolaj Bjørner, Jitu Padhye, Amar Agrawal, Ashish Bhargava, Paul-Andre C Bissonnette, Shane Foster, Andrew Helwer, Mark Kasten, Ivan Lee, Anup Namdhari, Haseeb Niaz, Aniruddha Parkhi, Hanukumar Pinnamraju, Adrian Power, Neha Milind Raje, and Parag Sharma. 2019. Validating Datacenters at Scale. In Proceedings of SIGCOMM '19. ACM, 200--213.

Digital Library

[20]

Peyman Kazemian, Michael Chang, Hongyi Zeng, George Varghese, Nick McKeown, and Scott Whyte. 2013. Real Time Network Policy Checking Using Header Space Analysis. In Proceedings of NSDI 13. USENIX Association, 99--111.

[21]

Peyman Kazemian, George Varghese, and Nick McKeown. 2012. Header Space Analysis: Static Checking for Networks. In Proceedings of NSDI 12. USENIX Association, 113--126.

[22]

Ahmed Khurshid, Xuan Zou, Wenxuan Zhou, Matthew Caesar, and P Brighten Godfrey. 2013. Veriflow: Verifying network-wide invariants in real time. In Proceedings of NSDI 13. USENIX Association, 15--27.

[23]

Petr Lapukhov, Ariff Premji, and Jon Mitchell. 2016. Use of BGP for routing in large-scale data centers. Internet Requests for Comments RFC Editor RFC 7938 (2016).

Digital Library

[24]

Hongqiang Harry Liu, Yibo Zhu, Jitu Padhye, Jiaxin Cao, Sri Tallapragada, Nuno P. Lopes, Andrey Rybalchenko, Guohan Lu, and Lihua Yuan. 2017. CrystalNet: Faithfully Emulating Large Production Networks. In Proceedings of SOSP '17. ACM, 599--613.

Digital Library

[25]

Nuno P Lopes and Andrey Rybalchenko. 2019. Fast bgp simulation of large datacenters. In International Conference on Verification, Model Checking, and Abstract Interpretation. Springer, 386--408.

[26]

Glenford J. Myers, Corey Sandler, and Tom Badgett. 2012. The art of software testing (3rd ed ed.). John Wiley & Sons.

Digital Library

[27]

Steve Ragan. 2016. BGP errors are to blame for Monday's Twitter outage, not DDoS attacks. https://www.csoonline.com/article/3138934/security/bgp-errors-are-to-blame-for-monday-s-twitter-outage-not-ddos-attacks.html Retrieved June 23, 2021 from

[28]

Deon Roberts. 2018. It's been a week and customers are still mad at BB&T. https://www.charlotteobserver.com/news/business/banking/article202616124.html Retrieved June 23, 2021 from

[29]

Yevgeniy Sverdlik. 2017. United Says IT Outage Resolved, Dozen Flights Canceled Monday. https://www.datacenterknowledge.com/archives/2017/01/23/united-says-it-outage-resolved-dozen-flights-canceled-monday Retrieved June 23, 2021 from

[30]

Bingchuan Tian, Xinyi Zhang, Ennan Zhai, Hongqiang Harry Liu, Qiaobo Ye, Chunsheng Wang, Xin Wu, Zhiming Ji, Yihong Sang, Ming Zhang, Da Yu, Chen Tian, Haitao Zheng, and Ben Y. Zhao. 2019. Safely and Automatically Updating In-Network ACL Configurations with Intent Language. In Proceedings of SIGCOMM '19. ACM, 214--226.

[31]

Zach Whittaker. 2020. T-Mobile hit by phone calling, text message outage. https://techcrunch.com/2020/06/15/t-mobile-calling-outage/ Retrieved June 23, 2021 from

[32]

Hongkun Yang and Simon S. Lam. 2016. Real-time Verification of Network Properties Using Atomic Predicates. IEEE/ACM Trans. Netw. 24, 2 (April 2016), 887--900. 1063-6692

Digital Library

[33]

Fangdan Ye, Da Yu, Ennan Zhai, Hongqiang Harry Liu, Bingchuan Tian, Qiaobo Ye, Chunsheng Wang, Xin Wu, Tianchen Guo, Cheng Jin, Duncheng She, Qing Ma, Biao Cheng, Hui Xu, Ming Zhang, Zhiliang Wang, and Rodrigo Fonseca. 2020. Accuracy, Scalability, Coverage: A Practical Configuration Verifier on a Global WAN. In Proceedings of SIGCOMM '20. ACM, 599--614.

Digital Library

[34]

Hongyi Zeng, Peyman Kazemian, George Varghese, and Nick McKeown. 2012. Automatic test packet generation. In Proceedings of the 8th international conference on Emerging networking experiments and technologies. 241--252.

Digital Library

[35]

Hongyi Zeng, Ratul Mahajan, Nick McKeown, George Varghese, Lihua Yuan, and Ming Zhang. 2015. Measuring and Troubleshooting Large Operational Multipath Networks with Gray Box Testing. Technical Report MSR-TR-2015-55 (Microsoft Research).

[36]

Hongyi Zeng, Shidong Zhang, Fei Ye, Vimalkumar Jeyakumar, Mickey Ju, Junda Liu, Nick McKeown, and Amin Vahdat. 2014. Libra: Divide and Conquer to Verify Forwarding Tables in Huge Networks. In Proceedings of NSDI 14. USENIX Association, 87--99.

Cited By

Cai JYang GLiu JXie Y(2023)FuzzyCAT: A Framework for Network Configuration Verification Based on Fuzzing2023 IEEE International Performance, Computing, and Communications Conference (IPCCC)10.1109/IPCCC59175.2023.10253841(123-131)Online publication date: 17-Nov-2023
https://doi.org/10.1109/IPCCC59175.2023.10253841

Index Terms

Test coverage metrics for the network
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
2. Networks
  1. Network services
    1. Network management
    2. Network monitoring
  2. Network types
    1. Data center networks

Recommendations

On Use of Coverage Metrics in Assessing Effectiveness of Combinatorial Test Designs
ICSTW '13: Proceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops

Combinatorial test suite design is a test generation technique, popular in part due to its ability to achieve coverage and defect finding power approximating that of exhaustive testing while keeping test suite sizes constrained. In recent years, there ...
Coverage metrics for formal verification

In formal verification, we verify that a system is correct with respect to a specification. Even when the system is proven to be correct, there is still a question of how complete the specification is and whether it really covers all the behaviors of ...
On automatic generation of RTL validation test benches using circuit testing techniques
GLSVLSI '03: Proceedings of the 13th ACM Great Lakes symposium on VLSI

In this paper, we examine how good validation test benches can be automatically generated starting from the RTL description of a circuit. We develop our methodology based on extensive experiments performed with several popular benchmarks as well as ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '21: Proceedings of the 2021 ACM SIGCOMM 2021 Conference

August 2021

868 pages

ISBN:9781450383837

DOI:10.1145/3452296

General Chairs:
Fernando Kuipers
Delft University of Technology
,
Matthew Caesar
University of Illinois at Urbana-Champaign

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF (National Science Foundation)

Conference

SIGCOMM '21

Sponsor:

SIGCOMM

SIGCOMM '21: ACM SIGCOMM 2021 Conference

August 23 - 27, 2021

Virtual Event, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
913
Total Downloads

Downloads (Last 12 months)174
Downloads (Last 6 weeks)30

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cai JYang GLiu JXie Y(2023)FuzzyCAT: A Framework for Network Configuration Verification Based on Fuzzing2023 IEEE International Performance, Computing, and Communications Conference (IPCCC)10.1109/IPCCC59175.2023.10253841(123-131)Online publication date: 17-Nov-2023
https://doi.org/10.1109/IPCCC59175.2023.10253841

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten