research-article

Unbiased experiments in congested networks

Authors:

Veronica Hannan,

Shravya Kunamalla,

Nick McKeown, and

Ramesh JohariAuthors Info & Claims

IMC '21: Proceedings of the 21st ACM Internet Measurement Conference

November 2021

Pages 80 - 95

https://doi.org/10.1145/3487552.3487851

Published: 02 November 2021 Publication History

Abstract

When developing a new networking algorithm, it is established practice to run a randomized experiment, or A/B test, to evaluate its performance. In an A/B test, traffic is randomly allocated between a treatment group, which uses the new algorithm, and a control group, which uses the existing algorithm. However, because networks are congested, both treatment and control traffic compete against each other for resources in a way that biases the outcome of these tests. This bias can have a surprisingly large effect; for example, in lab A/B tests with two widely used congestion control algorithms, the treatment appeared to deliver 150% higher throughput when used by a few flows, and 75% lower throughput when used by most flows---despite the fact that the two algorithms have identical throughput when used by all traffic.

Beyond the lab, we show that A/B tests can also be biased at scale. In an experiment run in cooperation with Netflix, estimates from A/B tests mistake the direction of change of some metrics, miss changes in other metrics, and overestimate the size of effects. We propose alternative experiment designs, previously used in online platforms, to more accurately evaluate new algorithms and allow experimenters to better understand the impact of congestion on their tests.

References

[1]

Alberto Abadie, Joshua Angrist, and Guido Imbens. 2002. Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings. Econometrica 70, 1 (Jan. 2002), 27.

[2]

A. Aggarwal, S. Savage, and T. Anderson. 2000. Understanding the Performance of TCP Pacing. In Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064), Vol. 3. IEEE, Tel Aviv, Israel, 1157--1165.

[3]

Julia Alexander. 2020. Amazon and Apple Are Reducing Streaming Quality to Lessen Broadband Strain in Europe. (March 2020). https://www.theverge.com/2020/3/20/21188072/amazon-prime-video-reduce-stream-quality-broadband-netflix-youtube-coronavirus

[4]

Peter M. Aronow and Cyrus Samii. 2017. Estimating average causal effects under general interference, with application to a social network experiment. Ann. Appl. Stat. 11, 4 (12 2017), 1912--1947.

[5]

Rukshani Athapathu, Ranysha Ware, Aditya Abraham Philip, Srinivasan Seshan, and Justine Sherry. 2020. Prudentia: Measuring Congestion Control Harm on the Internet. 2. http://www.justinesherry.com/papers/athapathu-n2women20.pdf

[6]

Susan Athey, Dean Eckles, and Guido W. Imbens. 2018. Exact p-Values for Network Interference. J. Amer. Statist. Assoc. 113, 521 (2018), 230--240. arXiv:https://doi.org/10.1080/01621459.2016.1241178

[7]

Pat Bajari, Brian Burdick, Guido Imbens, James McQueen, Thomas Richardson, and Ido Rosen. 2019. Multiple Randomization Designs for Interference. (2019). https://assets.amazon.science/c1/94/0d6431bf46f7978295d245dd6e06/double-randomized-online-experiments.pdf

[8]

H. Balakrishnan, V. N. Padmanabhan, S. Seshan, M. Stemm, and R. H. Katz. 1998. TCP Behavior of a Busy Internet Server: Analysis and Improvements. In Proceedings. IEEE INFOCOM '98, the Conference on Computer Communications. Seventeenth Annual Joint Conference of the IEEE Computer and Communications Societies. Gateway to the 21st Century (Cat. No. 98, Vol. 1. 252--262 vol.1.

[9]

Guillaume Basse, Avi Feller, and Panagiotis Toulis. 2019. Randomization tests of causal effects under interference. Biometrika 106, 2 (02 2019), 487--494. arXiv:https://academic.oup.com/biomet/article-pdf/106/2/487/28575447/asy072.pdf

[10]

Guillaume W. Basse, Hossein Azari Soufiani, and Diane Lambert. 2016. Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016 (JMLR Workshop and Conference Proceedings), Arthur Gretton and Christian C. Robert (Eds.), Vol. 51. JMLR.org, 1412--1420. http://proceedings.mlr.press/v51/basse16b.html

[11]

Neda Beheshti, Yashar Ganjali, Monia Ghobadi, Nick McKeown, and Geoff Salmon. 2008. Experimental Study of Router Buffer Sizing. In Proceedings of the 8th ACM SIGCOMM Conference on Internet Measurement (IMC '08). ACM, New York, NY, USA, 197--210.

Digital Library

[12]

Neda Beheshti, Petr Lapukhov, and Yashar Ganjali. 2019. Buffer Sizing Experiments at Facebook. In Proceedings of the 2019 Workshop on Buffer Sizing. ACM, Palo Alto CA USA, 1--6.

Digital Library

[13]

Thomas Blake and Dominic Coey. 2014. Why Marketplace Experimentation is Harder than It Seems: The Role of Test-Control Interference. In Proceedings of the Fifteenth ACM Conference on Economics and Computation (EC '14). Association for Computing Machinery, New York, NY, USA, 567--582.

Digital Library

[14]

Iavor Bojinov, David Simchi-Levi, and Jinglong Zhao. 2021. Design and Analysis of Switchback Experiments. arXiv:2009.00148 [stat] (Jan. 2021). arXiv:stat/2009.00148 http://arxiv.org/abs/2009.00148

[15]

Bob Briscoe. 2007. Flow Rate Fairness: Dismantling a Religion. ACM SIGCOMM Computer Communication Review 37, 2 (March 2007), 63--74.

Digital Library

[16]

Yi Cao, Arpit Jain, Kriti Sharma, Aruna Balasubramanian, and Anshul Gandhi. 2019. When to Use and When Not to Use BBR: An Empirical Analysis and Evaluation Study. In Proceedings of the Internet Measurement Conference. ACM, Amsterdam Netherlands, 130--136.

Digital Library

[17]

Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. 2017. BBR: Congestion-Based Congestion Control. Commun. ACM 60, 2 (Jan. 2017), 58--66.

Digital Library

[18]

Neal Cardwell, Yuchung Cheng, Soheil Hassas Yeganeh, Priyaranjan Jha, Yousuk Seung, Kevin Yang, Ian Swett, Victor Vasiliev, Bin Wu, Luke Hsiao, Matt Mathis, and Van Jacobson. 2019. BBRv2: A Model-Based Congestion Control Performance Optimizations. (Nov. 2019). https://www.ietf.org/proceedings/106/slides/slides-106-iccrg-update-on-bbrv2-00

[19]

Erik Carlsson and Eirini Kakogianni. 2018. Smoother Streaming with BBR. (Aug. 2018). https://engineering.atspotify.com/2018/08/31/smoother-streaming-with-bbr/

[20]

Nicholas Chamandy. 2016. Experimentation in a Ridesharing Marketplace. (Dec 2016). https://eng.lyft.com/experimentation-in-a-ridesharing-marketplace-f75a9c4fcf01

[21]

Bruno Crépon, Esther Duflo, Marc Gurgand, Roland Rathelot, and Philippe Zamora. 2013. Do Labor Market Policies Have Displacement Effects? Evidence from a Clustered Randomized Experiment ^*. The Quarterly Journal of Economics 128, 2 (May 2013), 531--580.

[22]

Nikos Diamantopoulos, Jeffrey Wong, David Issa Mattos, Ilias Gerostathopoulos, Matthew Wardrop, Tobias Mao, and Colin McFarland. 2019. Engineering for a Science-Centric Experimentation Platform. arXiv:1910.03878 [cs] (Oct. 2019). arXiv:cs/1910.03878 http://arxiv.org/abs/1910.03878

[23]

Mo Dong, Tong Meng, Doron Zarchy, Engin Arslan, Yossi Gilad, P Brighten Godfrey, and Michael Schapira. 2018. PCC Vivace: Online-Learning Congestion Control. In NSDI. 15.

[24]

Nandita Dukkipati, Matt Mathis, Yuchung Cheng, and Monia Ghobadi. 2011. Proportional Rate Reduction for TCP. In Internet Measurement Conference. 15.

[25]

Nandita Dukkipati, Tiziana Refice, Yuchung Cheng, Jerry Chu, Tom Herbert, Amit Agarwal, Arvind Jain, and Natalia Sutin. 2010. An Argument for Increasing TCP's Initial Congestion Window. ACM SIGCOMM Computer Communication Review 40, 3 (June 2010), 26--33.

Digital Library

[26]

Eric Dumazet. 2013. Pkt_sched: Fq: Fair Queue Packet Scheduler [LWN.Net]. (Aug. 2013). https://lwn.net/Articles/564825/

[27]

Eric Dumazet. 2013. Tcp: TSO Packets Automatic Sizing [LWN.Net]. (Aug. 2013). https://lwn.net/Articles/564979/

[28]

Dean Eckles, Brian Karrer, and Johan Ugander. 2016. Design and Analysis of Experiments in Networks: Reducing Bias from Interference. Journal of Causal Inference 5, 1 (Feb. 2016).

[29]

Tobias Flach, Nandita Dukkipati, Andreas Terzis, Barath Raghavan, Neal Cardwell, Yuchung Cheng, Ankur Jain, Shuai Hao, Ethan Katz-Bassett, and Ramesh Govindan. 2013. Reducing Web Latency: The Virtue of Gentle Aggression. In SIGCOMM. 12.

[30]

Ken Florance. 2020. Reducing Netflix Traffic Where It's Needed While Maintaining the Member Experience. (March 2020). https://about.netflix.com/en/news/reducing-netflix-traffic-where-its-needed

[31]

Andrew Gelman and Jennifer Hill. 2006. Causal Inference Using Regression on the Treatment Variable. In Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press, Cambridge, 167--198.

[32]

Peter Glynn, Ramesh Johari, and Mohammad Rasouli. 2020. Adaptive experimental design with temporal interference: A maximum likelihood approach. arXiv preprint arXiv:2006.05591 (2020).

[33]

Hadas Gold. 2020. Netflix and YouTube Are Slowing down in Europe to Keep the Internet from Breaking. (March 2020). https://www.cnn.com/2020/03/19/tech/netflix-internet-overload-eu/index.html

[34]

Nirmal Govind. 2018. A/B Testing and Beyond: Improving the Netflix Streaming Experience with Experimentation and Data.... (June 2018). https://netflixtechblog.com/a-b-testing-and-beyond-improving-the-netflix-streaming-experience-with-experimentation-and-data-5b0ae9295bdf

[35]

Ilya Grigorik. 2013. HTTP: HTTP/1.X - High Performance Browser Networking (O'Reilly). (2013). https://hpbn.co/http1x/#using-multiple-tcp-connections

[36]

Ilya Grigorik and Surma. 2019. Introduction to HTTP/2 | Web Fundamentals. (Sept. 2019). https://developers.google.com/web/fundamentals/performance/http2#request_and_response_multiplexing

[37]

Huan Gui, Ya Xu, Anmol Bhasin, and Jiawei Han. 2015. Network A/B Testing: From Sampling to Estimation. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, Florence Italy, 399--409.

Digital Library

[38]

M. E. Halloran and C. J. Struchiner. 1995. Causal Inference in Infectious Diseases. Epidemiology (Cambridge, Mass.) 6, 2 (March 1995), 142--151.

[39]

David Holtz, Ruben Lobel, Inessa Liskovich, and Sinan Aral. 2020. Reducing Interference Bias in Online Marketplace Pricing Experiments. arXiv:2004.12489 [econ, stat] (April 2020). arXiv:econ, stat/2004.12489 http://arxiv.org/abs/2004.12489

[40]

David Holtz, Ruben Lobel, Inessa Liskovich, and Sinan Aral. 2020. Reducing Interference Bias in Online Marketplace Pricing Experiments. (2020). arXiv:stat.ME/2004.12489

[41]

Guanglei Hong and Stephen W. Raudenbush. 2006. Evaluating Kindergarten Retention Policy. J. Amer. Statist. Assoc. 101, 475 (Sept. 2006), 901--910.

[42]

Te-Yuan Huang, Ramesh Johari, Nick McKeown, Matthew Trunnell, and Mark Watson. 2014. A Buffer-Based Approach to Rate Adaptation: Evidence from a Large Video Streaming Service. In Proceedings of the 2014 ACM Conference on SIGCOMM. ACM, Chicago Illinois USA, 187--198.

Digital Library

[43]

Per Hurtig, Habtegebreil Haile, Karl-Johan Grinnemo, Anna Brunstrom, Eneko Atxutegi, Fidel Liberal, and Ake Arvidsson. 2018. Impact of TCP BBR on CUBIC Traffic: A Mixed Workload Evaluation. In 2018 30th International Teletraffic Congress (ITC 30). IEEE, Vienna, 218--226.

[44]

Geoff Huston. 2018. TCP and BBR. (May 2018). https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf

[45]

Guido W. Imbens and Donald B. Rubin. 2015. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, USA.

[46]

Alexey Ivanov. 2020. Evaluating BBRv2 on the Dropbox Edge Network. arXiv:2008.07699 [cs] (Aug. 2020). arXiv:cs/2008.07699 http://arxiv.org/abs/2008.07699

[47]

Ramesh Johari, Hannah Li, Inessa Liskovich, and Gabriel Weintraub. 2020. Experimental design in two-sided platforms: An analysis of bias. arXiv preprint arXiv:2002.05670 (2020).

[48]

Matt Joras and Yang Chi. 2020. How Facebook Is Bringing QUIC to Billions. (Oct. 2020). https://engineering.fb.com/2020/10/21/networking-traffic/how-facebook-is-bringing-quic-to-billions/

[49]

Matt Joras and Yang Chi. 2020. How Facebook Is Bringing QUIC to Billions. (Oct. 2020). https://engineering.fb.com/2020/10/21/networking-traffic/how-facebook-is-bringing-quic-to-billions/

[50]

Arash Molavi Kakhki, Samuel Jero, David Choffnes, Cristina Nita-Rotaru, and Alan Mislove. 2017. Taking a Long Look at QUIC: An Approach for Rigorous Evaluation of Rapidly Evolving Transport Protocols. In Proceedings of the 2017 Internet Measurement Conference. ACM, London United Kingdom, 290--303.

Digital Library

[51]

Brian Karrer, Liang Shi, Monica Bhole, Matt Goldman, Tyrone Palmer, Charlie Gelman, Mikael Konutgan, and Feng Sun. 2020. Network Experimentation at Scale. arXiv:2012.08591 [cs, stat] (Dec. 2020). arXiv:cs, stat/2012.08591 http://arxiv.org/abs/2012.08591

[52]

David Kastelman and Raghav Ramesh. 2018. Switchback Tests and Randomized Experimentation Under Network Effects at DoorDash. (Feb. 2018). https://medium.com/@DoorDash/switchback-tests-and-randomized-experimentation-under-network-effects-at-doordash-f1d938ab7c2a

[53]

Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. 2013. Online Controlled Experiments at Large Scale. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Chicago Illinois USA, 1168--1176.

Digital Library

[54]

Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (first ed.). Cambridge University Press.

[55]

Gautam Kumar, Nandita Dukkipati, Keon Jang, Hassan M. G. Wassel, Xian Wu, Behnam Montazeri, Yaogong Wang, Kevin Springborn, Christopher Alfeld, Michael Ryan, David Wetherall, and Amin Vahdat. 2020. Swift: Delay Is Simple and Effective for Congestion Control in the Datacenter. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication. ACM, Virtual Event USA, 514--528.

Digital Library

[56]

Ike Kunze, Jan Ruth, and Oliver Hohlfeld. 2020. Congestion Control in the Wild---Investigating Content Provider Fairness. IEEE Transactions on Network and Service Management 17, 2 (June 2020), 1224--1238.

Digital Library

[57]

Raul Landa, Lorenzo Saino, Lennert Buytenhek, and João Taveira Araújo. 2021. Staying Alive: Connection Path Reselection at the Edge. In NSDI 2021. 20.

[58]

Adam Langley, Alistair Riddoch, Alyssa Wilk, Antonio Vicente, Charles Krasic, Dan Zhang, Fan Yang, Fedor Kouranov, Ian Swett, Janardhan Iyengar, Jeff Bailey, Jeremy Dorfman, Jim Roskind, Joanna Kulik, Patrik Westin, Raman Tenneti, Robbie Shade, Ryan Hamilton, Victor Vasiliev, Wan-Teh Chang, and Zhongyi Shi. 2017. The QUIC Transport Protocol: Design and Internet-Scale Deployment. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). Association for Computing Machinery, New York, NY, USA, 183--196.

Digital Library

[59]

Charles F. Manski. 2013. Identification of treatment response with social interactions. The Econometrics Journal 16, 1 (2013), S1--S23. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1368-423X.2012.00368.x

[60]

Hongzi Mao, Shannon Chen, Drew Dimmery, Shaun Singh, Drew Blaisdell, Yuandong Tian, Mohammad Alizadeh, and Eytan Bakshy. 2020. Real-World Video Adaptation with Reinforcement Learning. arXiv:2008.12858 [cs] (Aug. 2020). arXiv:cs/2008.12858 http://arxiv.org/abs/2008.12858

[61]

Radhika Mittal, Vinh The Lam, Nandita Dukkipati, Emily Blem, Hassan Wassel, Monia Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, and David Zats. 2015. TIMELY: RTT-Based Congestion Control for the Datacenter. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. ACM, London United Kingdom, 537--550.

Digital Library

[62]

Whitney K. Newey and Kenneth D. West. 1987. A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix. Econometrica 55, 3 (1987), 703--708.

[63]

Garg Nitin. 2019. COPA Congestion Control for Video Performance. (Nov. 2019). https://engineering.fb.com/2019/11/17/video-engineering/copa/

[64]

Samuel D. Oman and Esther Seiden. 1988. Switch-Back Designs. Biometrika 75, 1 (March 1988), 81--89.

[65]

James Robins. 1986. A New Approach to Causal Inference in Mortality Studies with a Sustained Exposure Period---Application to Control of the Healthy Worker Survivor Effect. Mathematical Modelling 7, 9-12 (Jan. 1986), 1393--1512.

[66]

Donald B Rubin. 2005. Causal inference using potential outcomes: Design, modeling, decisions. J. Amer. Statist. Assoc. 100, 469 (2005), 322--331.

[67]

Ahmed Saeed, Nandita Dukkipati, Vytautas Valancius, Vinh The Lam, Carlo Contavalli, and Amin Vahdat. 2017. Carousel: Scalable Traffic Shaping at End Hosts. In The Conference of the ACM Special Interest Group. ACM Press, 404--417.

Digital Library

[68]

Martin Saveski, Jean Pouget-Abadie, Guillaume Saint-Jacques, Weitao Duan, Souvik Ghosh, Ya Xu, and Edoardo M. Airoldi. 2017. Detecting Network Effects: Randomizing Over Randomized Experiments. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Halifax NS Canada, 1027--1035.

Digital Library

[69]

Robert Sayre. 2008. Change Max-Persistent-Connections-per-Server to 6. (March 2008). https://bugzilla.mozilla.org/show_bug.cgi?id=423377

[70]

Nate Schloss and Ben Maurer. 2017. This Browser Tweak Saved 60% of Requests to Facebook. (Jan. 2017). https://engineering.fb.com/2017/01/26/web/this-browser-tweak-saved-60-of-requests-to-facebook/

[71]

Dominik Scholz, Benedikt Jaeger, Lukas Schwaighofer, Daniel Raumer, Fabien Geyer, and Georg Carle. 2018. Towards a Deeper Understanding of TCP BBR Congestion Control. In 2018 IFIP Networking Conference (IFIP Networking) and Workshops. IEEE, Zurich, Switzerland, 1--9.

[72]

Anant Shah. 2019. BBR Evaluation at a Large CDN. (Nov. 2019). https://blog.apnic.net/2019/11/01/bbr-evaluation-at-a-large-cdn/

[73]

Steve Souders. 2008. Roundup on Parallel Connections. (March 2008). https://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections/

[74]

Bruce Spang, Brady Walsh, Te-Yuan Huang, Tom Rusnock, Joe Lawrence, and Nick McKeown. 2019. Buffer Sizing and Video QoE Measurements at Netflix. In Proceedings of the 2019 Workshop on Buffer Sizing. ACM, Palo Alto CA USA.

Digital Library

[75]

Jerzy Splawa-Neyman, Dorota M Dabrowska, and TP Speed. 1990. On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statist. Sci. (1990), 465--472.

[76]

Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. 2010. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation. In KDD'10. 10.

[77]

Eric J. Tchetgen Tchetgen and Tyler J. VanderWeele. 2012. On causal inference in the presence of interference. Statistical Methods in Medical Research 21 (2012), 55 -- 75.

[78]

Martin Tingley. 2018. Streaming Video Experimentation at Netflix: Visualizing Practical and Statistical Significance. (Sept. 2018). https://netflixtechblog.com/streaming-video-experimentation-at-netflix-visualizing-practical-and-statistical-significance-7117420f4e9a

[79]

Linus Torvalds. [n. d.]. Tcp_input.c - Linux (v5.11-Rc5). ([n. d.]). https://github.com/torvalds/linux/blob/2ab38c17aac10bf55ab3efde4c4db3893d8691d2/net/ipv4/tcp_input.c#L873

[80]

Donald F Towsley. 2015. TCP, Congestion Control. (Nov. 2015). http://gaia.cs.umass.edu/cs653/slides/tcp.pdf

[81]

Belma Turkovic, Fernando A. Kuipers, and Steve Uhlig. 2019. Fifty Shades of Congestion Control: A Performance and Interactions Evaluation. arXiv:1903.03852 [cs] (March 2019). arXiv:cs/1903.03852 http://arxiv.org/abs/1903.03852

[82]

Belma Turkovic, Fernando A. Kuipers, and Steve Uhlig. 2019. Interactions between Congestion Control Algorithms. In 2019 Network Traffic Measurement and Analysis Conference (TMA). 161--168.

[83]

Johan Ugander, Brian Karrer, Lars Backstrom, and Jon Kleinberg. 2013. Graph Cluster Randomization: Network Exposure to Multiple Universes. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '13). Association for Computing Machinery, New York, NY, USA, 329--337.

Digital Library

[84]

Ranysha Ware, Matthew K. Mukerjee, Srinivasan Seshan, and Justine Sherry. 2019. Beyond Jain's Fairness Index: Setting the Bar For The Deployment of Congestion Control Algorithms. In Proceedings of the 18th ACM Workshop on Hot Topics in Networks. ACM, Princeton NJ USA, 17--24.

Digital Library

[85]

Ranysha Ware, Matthew K. Mukerjee, Srinivasan Seshan, and Justine Sherry. 2019. Modeling BBR's Interactions with Loss-Based Congestion Control. In Proceedings of the Internet Measurement Conference. ACM, Amsterdam Netherlands, 137--143.

Digital Library

[86]

David X. Wei, Pei Cao, and Steven H. Low. 2006. TCP Pacing Revisited. (2006). http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.92.2658&rep=rep1&type=pdf

[87]

Francis Y Yan, Hudson Ayers, Chenzhi Zhu, Sadjad Fouladi, James Hong, Keyi Zhang, Philip Levis, and Keith Winstein. 2020. Learning in Situ: A Randomized Experiment in Video Streaming. In NSDI. Santa Clara, CA, USA, 16. https://www.usenix.org/system/files/nsdi20-paper-yan.pdf

Cited By

Quin FWeyns DGalster MSilva C(2024)A/B testingJournal of Systems and Software10.1016/j.jss.2024.112011211:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112011
Cortez-Rodriguez MEichhorn MYu C(2023)Exploiting neighborhood interference with low-order interactions under unit randomized designJournal of Causal Inference10.1515/jci-2022-005111:1Online publication date: 3-Aug-2023
https://doi.org/10.1515/jci-2022-0051
Luo ZLiu JYang GZhang YHang Z(2022)High-Speed Path Probing Method for Large-Scale NetworkSensors10.3390/s2215565022:15(5650)Online publication date: 28-Jul-2022
https://doi.org/10.3390/s22155650

Index Terms

Unbiased experiments in congested networks

Recommendations

AQM performance in multiple congested link networks
ACM-SE 43: Proceedings of the 43rd annual Southeast regional conference - Volume 2

TCP Reno, the most widely used implementation on the Internet, uses retransmission timeouts and the receipt of three duplicate ACKs to detect packet loss in the network. In case of multiple congested links, it is important to detect packet losses as ...
Read More
Connections with multiple congested gateways in packet-switched networks part 1: one-way traffic

In this paper we explore the bias in TCP/IP networks against connections with multiple congested gateways. We consider the interaction between the bias against connections with multiple congested gateways, the bias of the TCP window modification ...
Read More
Management of parallel UBR flows over TCP in congested ATM networks

The Linux transmission control protocol (TCP) implementation consists of TCP Reno, NewReno and selective acknowledgement options. This has improved the throughput of TCP/IP over the Internet. However, TCP over plain ATM networks significantly suffers ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IMC '21: Proceedings of the 21st ACM Internet Measurement Conference

November 2021

768 pages

ISBN:9781450391290

DOI:10.1145/3487552

General Chairs:
Dave Levin
University of Maryland
,
Alan Mislove
Northeastern University
,
Program Chairs:
Johanna Amann
International Computer Science Institute
,
Matthew Luckie
University of Waikato

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

USENIX Assoc: USENIX Assoc

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

IMC '21

Sponsor:

IMC '21: ACM Internet Measurement Conference

November 2 - 4, 2021

Virtual Event

Acceptance Rates

Overall Acceptance Rate 277 of 1,083 submissions, 26%

Upcoming Conference

IMC '24

Sponsor:
sigcomm
sigcomm

ACM Internet Measurement Conference

November 4 - 6, 2024

Madrid , AA , Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
461
Total Downloads

Downloads (Last 12 months)74
Downloads (Last 6 weeks)7

Other Metrics

View Author Metrics

Citations

Cited By

Quin FWeyns DGalster MSilva C(2024)A/B testingJournal of Systems and Software10.1016/j.jss.2024.112011211:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112011
Cortez-Rodriguez MEichhorn MYu C(2023)Exploiting neighborhood interference with low-order interactions under unit randomized designJournal of Causal Inference10.1515/jci-2022-005111:1Online publication date: 3-Aug-2023
https://doi.org/10.1515/jci-2022-0051
Luo ZLiu JYang GZhang YHang Z(2022)High-Speed Path Probing Method for Large-Scale NetworkSensors10.3390/s2215565022:15(5650)Online publication date: 28-Jul-2022
https://doi.org/10.3390/s22155650

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents