Optimal Network Protocol Selection for Competing Flows via Online Learning
Authors: 
Xiaoxi Zhang, Siqi Chen, Yunfan Zhang, Youngbin Im, Maria Gorlatova, Sangtae Ha, Carlee Joe-WongAuthors Info & Claims
Xiaoxi Zhang, Siqi Chen, Yunfan Zhang, Youngbin Im, Maria Gorlatova, Sangtae Ha, Carlee Joe-WongAuthors Info & ClaimsPages 4822 - 4836
Abstract
Today’s Internet must support applications with increasingly dynamic and heterogeneous connectivity requirements, such as video streaming and the Internet of Things. Yet current network management practices generally rely on pre-specified network configurations, which may not be able to cope with dynamic application needs. Moreover, even the best-specified policies will find it difficult to cover all possible scenarios, given applications’ increasing heterogeneity and dynamic network conditions, e.g., on volatile wireless links. In this work, we instead propose a model-free learning approach to find the optimal network policies for current network flow requirements. This approach is attractive as comprehensive models do not exist for how different policy choices affect flow performance under changing network conditions. However, it can raise new challenges for online learning algorithms: policy configurations can affect the performance of multiple flows sharing the same network resources, and this performance coupling limits the scalability and optimality of existing online learning algorithms. In this work, we extend multi-armed bandit frameworks to propose new online learning algorithms for protocol selection with provably sublinear regret under certain conditions. We validate the optimality and scalability of our algorithms through data-driven simulations and testbed experiments. (An extended abstract of this work was accepted by IEEE ICNP as a short paper Zhang <italic>et al</italic>. (2019)).
References
[1]
X. Zhang, S. Chen, Y. Im, M. Gorlatova, S. Ha, and C. Joe-Wong, “Towards automated network management: Learning the optimal protocol selection,” in Proc. IEEE 27th Int. Conf. Netw. Protoc., 2019, pp. 1–4.
[2]
C. Westphal, “Challenges in networking to support augmented reality and virtual reality,” in Proc. IEEE Int. Conf. Comput. Network. Commun., 2017.
[3]
A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, and M. Ayyash, “Internet of Things: A survey on enabling technologies, protocols, and applications,” IEEE Commun. Surv. Tut., vol. 17, no. 4, pp. 2347–2376, Oct.–Dec. 2015.
[4]
C.-X. Wang et al., “Cellular architecture and key technologies for 5G wireless communication networks,” IEEE Commun. Mag., vol. 52, no. 2, pp. 122–130, Feb. 2014.
[5]
A. Silver, “3 ways Nokia is using machine learning in 5G networks,” 2018. [Online]. Available: https://spectrum.ieee.org/tech-talk/telecom/wireless/3-ways-nokia-is-using-machine-learning-in-5g-networks
[6]
P. Popovski, K. F. Trillingsgaard, O. Simeone, and G. Durisi, “5G wireless network slicing for Embb, Urllc, and Mmtc: A communication-theoretic view,” IEEE Access, vol. 6, pp. 55765–55779, 2018.
[7]
X. Foukas, G. Patounas, A. Elmokashfi, and M. K. Marina, “Network slicing in 5G: Survey and challenges,” IEEE Commun. Mag., vol. 55, no. 5, pp. 94–100, May 2017.
[8]
N. Hassan, K. Yau, and C. Wu, “Edge computing in 5G: A review,” IEEE Access, vol. 7, pp. 127276–127289, 2019.
[9]
B. Hartpence, Packet Guide to Routing and Switching. CA, USA: O’Reilly Media, Inc, 2011.
[10]
“Virtual network traffic routing,” 2022. [Online]. Available: https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview
[11]
O-RAN Alliance, “O-Ran: Towards an open andsmart ran,” 2022. [Online]. Available: https://www.o-ran.org/
[12]
R. Mijumbi, J. Serrat, J.-L. Gorricho, N. Bouten, F. De Turck, and R. Boutaba, “Network function virtualization: State-of-the-art and research challenges,” IEEE Commun. Surv. Tut., vol. 18, no. 1, pp. 236–262, Jan.–Mar. 2016.
[13]
M. Chiang, S. H. Low, A. R. Calderbank, and J. C. Doyle, “Layering as optimization decomposition: A mathematical theory of network architectures,” Proc. IEEE, vol. 95, no. 1, pp. 255–312, Jan. 2007.
[14]
F. Ait Aoudia, M. Gautier, M. Magno, O. Berder, and L. Benini, “A generic framework for modeling MAC protocols in wireless sensor networks,” IEEE/ACM Trans. Netw., vol. 25, no. 3, pp. 1489–1500, Jun. 2017.
[15]
A. A. Khan, M. H. Rehmani, and M. Reisslein, “Requirements, design challenges, and review of routing and MAC protocols for cr-based smart grid systems,” IEEE Commun. Mag., vol. 55, no. 5, pp. 206–215, May 2017.
[16]
Ł. Budzisz, R. Stanojević, A. Schlote, F. Baker, and R. Shorten, “On the fair coexistence of loss-and delay-based TCP,” IEEE/ACM Trans. Netw., vol. 19, no. 6, pp. 1811–1824, Dec. 2011.
[17]
C. Xu, J. Zhao, and G.-M. Muntean, “Congestion control design for multipath transport protocols: A survey,” IEEE Commun. Surv. Tut., vol. 18, no. 4, pp. 2948–2969, Oct.–Dec. 2016.
[18]
Q. Gao, J. Zhang, and S. V. Hanly, “Cross-layer rate control in wireless networks with lossy links: Leaky-pipe flow, effective network utility maximization and hop-by-hop algorithms,” IEEE Trans. Wireless Commun., vol. 8, no. 6, pp. 3068–3076, Jun. 2009.
[19]
N. Ehsan and M. Liu, “Modeling TCP performance with proxies,” Comput. Commun., vol. 27, no. 10, pp. 961–975, 2004.
[20]
F. Le, E. Nahum, V. Pappas, M. Touma, and D. Verma, “Experiences deploying a transparent split TCP middlebox and the implications for NFV,” in Proc. ACM SIGCOMM Workshop Hot Top. Middleboxes Netw. Function Virtualization, 2015, pp. 31–36.
[21]
Y. Zaki, T. Pötsch, J. Chen, L. Subramanian, and C. Görg, “Adaptive congestion control for unpredictable cellular networks,” in Proc. ACM SIGCOMM Comput. Commun. Rev., 2015, vol. 45, pp. 509–522.
[22]
J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning and stochastic optimization,” J. Mach. Learn. Res., vol. 12, pp. 2121–2159, 2011.
[23]
C. L. Liu and J. W. Layland, “Scheduling algorithms for multiprogramming in a hard-real-time environment,” J. ACM, vol. 20, no. 1, pp. 46–61, 1973.
[24]
R. Bhargava, “Evolution of Dropbox’s edge network,” Dropbox, 2017. [Online]. Available: https://blogs.dropbox.com/tech/2017/06/evolution-of-dropboxs-edge-network/
[25]
Google Cloud, “Overview of TCP forwarding,” 2019. [Online]. Available: https://cloud.google.com/iap/docs/tcp-forwarding-overview
[26]
Y. Cai, F. R. Yu, and S. Bu, “Cloud computing meets mobile wireless communications in next generation cellular networks,” IEEE Netw., vol. 28, no. 6, pp. 54–59, Nov./Dec. 2014.
[27]
G. Siracusano, R. Bifulco, S. Kuenzer, S. Salsano, N. B. Melazzi, and F. Huici, “On the fly TCP acceleration with miniproxy,” in Proc. Workshop Hot Top. Middleboxes Netw. Function Virtualization, 2016, pp. 44–49.
[28]
S. H. Low, “A duality model of TCP and queue management algorithms,” IEEE/ACM Trans. Netw., vol. 11, no. 4, pp. 525–536, Aug. 2003.
[29]
B. Sikdar, S. Kalyanaraman, and K. S. Vastola, “Analytic models for the latency and steady-state throughput of TCP Tahoe, Reno, and Sack,” IEEE/ACM Trans. Netw., vol. 11, no. 6, pp. 959–971, Dec. 2003.
[30]
K. Nagaraj, D. Bharadia, H. Mao, S. Chinchali, M. Alizadeh, and S. Katti, “Numfabric: Fast and flexible bandwidth allocation in datacenters,” in Proc. ACM SIGCOMM Conf., 2016, pp. 188–201.
[31]
S. Ha, I. Rhee, and L. Xu, “Cubic: A new TCP-friendly high-speed TCP variant,” in Proc. ACM SIGOPS Oper. Syst. Rev., 2008, vol. 42, pp. 64–74.
[32]
I. Demirkol et al., “MAC protocols for wireless sensor networks: A survey,” IEEE Commun. Mag., vol. 44, no. 4, pp. 115–121, Apr. 2006.
[33]
X. Zhang et al., “Optimal network protocol selection for competing flows via online learning,” 2022. [Online]. Available: https://www.dropbox.com/s/pkm4bi2jryj8del/tr_TMC2020_Distributed_Protocol_Selection.pdf?dl=0
[34]
A. Garivier and E. Moulines, “On upper-confidence bound policies for non-stationary bandit problems,” 2008. [Online]. Available: https://arxiv.org/abs/0805.3415
[35]
Y. Xia, T. Qin, W. Ma, N. Yu, and T.-Y. Liu, “Budgeted multi-armed bandits with multiple plays,” in Proc. Int. Joint Conf. Artif. Intell., 2016, pp. 2210–2216.
[36]
B. Turkovic, F. A. Kuipers, and S. Uhlig, “Fifty shades of congestion control: A performance and interactions evaluation,” 2019. [Online]. Available: https://arxiv.org/pdf/1903.03852.pdf
[37]
“NS-3 network simulator,” 2022. [Online]. Available: https://www.nsnam.org/
[38]
M. Hock, R. Bless, and M. Zitterbart, “Experimental evaluation of BBR congestion control,” in Proc. IEEE 25th Int. Conf. Netw. Protoc., 2017, pp. 1–10.
[39]
K. Winstein and H. Balakrishnan, “TCP ex machina: Computer-generated congestion control,” in Proc. ACM SIGCOMM Comput. Commun. Rev., 2013, vol. 43, pp. 123–134.
[40]
H. Mao, M. Alizadeh, I. Menache, and S. Kandula, “Resource management with deep reinforcement learning,” in Proc. 15th ACM Workshop Hot Top. Netw., 2016, pp. 50–56.
[41]
N. Chen, J. Comden, Z. Liu, A. Gandhi, and A. Wierman, “Using predictions in online optimization: Looking forward with an eye on the past,” in Proc. ACM SIGMETRICS Perform. Eval. Rev., 2016, pp. 193–206.
[42]
X. Zhang, C. Wu, Z. Li, and F. C. Lau, “Proactive VNF provisioning with multi-timescale cloud resources: Fusing online learning and online optimization,” in Proc. IEEE INFOCOM Conf. Comput. Commun., 2017, pp. 1–9.
[43]
R. Combes, C. Jiang, and R. Srikant, “Bandits with budgets: Regret lower bounds and optimal algorithms,” in Proc. ACM SIGMETRICS Perform. Eval. Rev., 2015, pp. 245–257.
[44]
Y. Liu and M. Liu, “An online approach to dynamic channel access and transmission scheduling,” in Proc. 16th ACM Int. Symp. Mobile Ad Hoc Netw. Comput., 2015, pp. 187–196.
[45]
S. Henri, C. Vlachou, and P. Thiran, “Multi-armed bandit in action: Optimizing performance in dynamic hybrid networks,” IEEE/ACM Trans. Netw., vol. 26, no. 4, pp. 1879–1892, Aug. 2018.
[46]
A. A. A. Islam, S. I. Alam, V. Raghunathan, and S. Bagchi, “Multi-armed bandit congestion control in multi-hop infrastructure wireless mesh networks,” in Proc. IEEE Int. Symp. Model., Anal. Simul. Comput. Telecommun. Syst., 2012, pp. 31–40.
[47]
S. Takeuchi, M. Hasegawa, K. Kanno, A. Uchida, N. Chauvet, and M. Naruse, “Dynamic channel selection in wireless communications via a multi-armed bandit algorithm using laser chaos time series,” Sci. Rep., vol. 10, 2020, Art. no.
[48]
N. Jay, N. Rotman, B. Godfrey, M. Schapira, and A. Tamar, “A deep reinforcement learning perspective on internet congestion control,” in Proc. Int. Conf. Mach. Learn., 2019, pp. 3050–3059.
[49]
Z. Xu, J. Tang, C. Yin, Y. Wang, and G. Xue, “Experience-driven congestion control: When multi-path TCP meets deep reinforcement learning,” IEEE J. Sel. Areas Commun., vol. 37, no. 6, pp. 1325–1336, Jun. 2019.
[50]
V. Sciancalepore, X. Costa-Perez, and A. Banchs, “RL-NSB: Reinforcement learning-based 5G network slice broker,” IEEE/ACM Trans. Netw., vol. 27, no. 4, pp. 1543–1557, Aug. 2019.
[51]
Q. Liu, T. Han, and E. Moges, “Edgeslice: Slicing wireless edge computing network with decentralized deep reinforcement learning,” 2020. [Online]. Available: https://arxiv.org/abs/2003.12911
[52]
Y. Hua, R. Li, Z. Zhao, X. Chen, and H. Zhang, “GAN-powered deep distributional reinforcement learning for resource management in network slicing,” IEEE J. Sel. Areas Commun., vol. 38, no. 2, pp. 334–349, Feb. 2019.
[53]
R. Ali, N. Shahin, Y. B. Zikria, B.-S. Kim, and S. W. Kim, “Deep reinforcement learning paradigm for performance optimization of channel observation-based MAC protocols in dense WLANs,” IEEE Access, vol. 7, pp. 3500–3511, 2018.
[54]
Y. Yu, T. Wang, and S. C. Liew, “Deep-reinforcement learning multiple access for heterogeneous wireless networks,” IEEE J. Sel. Areas Commun., vol. 37, no. 6, pp. 1277–1290, Jun. 2019.
[55]
W. Chien, H. Weng, and C. Lai, “Q-learning based collaborative cache allocation in mobile edge computing,” Future Gener. Comput. Syst., vol. 102, pp. 603–610, 2020.
[56]
Y. Dai, D. Xu, K. Zhang, Y. Lu, S. Maharjan, and Y. Zhang, “Deep reinforcement learning for edge computing and resource allocation in 5G beyond,” in Proc. IEEE Int. Conf. Commun. Technol., 2019, pp. 866–870.
[57]
N. Liu et al., “A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning,” in Proc. IEEE 37th Int. Conf. Distrib. Comput. Syst., 2017, pp. 372–382.
[58]
X. Foukas, M. K. Marina, and K. Kontovasilis, “IRIS: Deep reinforcement learning driven shared spectrum access architecture for indoor neutral-host small cells,” IEEE J. Sel. Areas Commun., vol. 37, no. 8, pp. 1820–1837, Aug. 2019.
[59]
I. Abdeljaouad, H. Rachidi, S. Fernandes, and A. Karmouch, “Performance analysis of modern TCP variants: A comparison of cubic, compound and new reno,” in Proc. Commun., 25th Biennial Symp., 2010, pp. 80–83.
Recommendations
A Supply Network's Optimal Information System and Material Flows
The management of a supply network is repeatedly forced to react to changes that have occurred, and to pro-act to changes that may occur. Management would certainly find it useful to have available a model which could identify the optimal re-action, or ...
Optimal Internet Media Selection
In this study we develop a method that optimally selects online media vehicles and determines the number of advertising impressions that should be purchased and then served from each chosen website. As a starting point, we apply Danaher's [Danaher, P. ...
Optimal network selection algorithms under the multi-network coexistence environment based on attribute decision
In order to perform optimal network selection under the multi-network coexistence environment and realise adaptive switching of network to improve the stability of network transmission, an optimal network selection algorithm under the multi-network ...
Comments
Information & Contributors
Information
Published In
1536-1233 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 01 August 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 27 Jan 2025