research-article

Reinforcement Learning-Based Service-Oriented Dynamic Multipath Routing in SDN

Authors:

Kai-Cheng Chiu,

Chien-Chang Liu,

Li-Der Chou Academic Editor:

Jianhui LvAuthors Info & Claims

Wireless Communications and Mobile Computing, Volume 2022

https://doi.org/10.1155/2022/1330993

Published: 01 January 2022 Publication History

Abstract

The increasing quality and various requirements of network services are guaranteed because of the advancement of the emerging network paradigm, software-defined networking (SDN), and benefits from the centralized and software-defined architecture. The SDN not only facilitates the configuration of the network policies for traffic engineering but also brings convenience for network state obtainment. The traffic of numerous services is transmitted within a network, whereas each service may demand different network metrics, such as low latency or low packet loss rate. Corresponding quality of service policies must be enforced to meet the requirements of different services, and the balance of link utilization is also indispensable. In this research, Reinforcement Discrete Learning-Based Service-Oriented Multipath Routing (RED-STAR) has been proposed to understand the policy of distributing an optimal path for each service. The RED-STAR takes the network state and service type as input values to dynamically select the path a service must be forwarded. Custom protocols are designed for network state obtainment, and a deep learning-based traffic classification model is also integrated to identify network services. With the differentiated reward scheme for every service type, the reinforcement learning model in RED-STAR gradually achieves high reward values in various scenarios. The experimental results show that RED-STAR can adopt the dynamic network environment, obtaining the highest average reward value of 1.8579 and the lowest average maximum bandwidth utilization of 0.3601 among all path distribution schemes in a real-case scenario.

References

[1]

Y. Chen, T. Farley, and N. Ye, “QoS requirements of network applications on the Internet,” Information—Knowledge—Systems Management, vol. 4, no. 1, pp. 55–76, 2004.

Digital Library

[2]

D. B. Rawat and S. R. Reddy, “Software defined networking architecture, security and energy efficiency: a survey,” IEEE Communications Surveys & Tutorials, vol. 19, no. 1, pp. 325–346, 2017.

Digital Library

[3]

D. Kreutz, F. M. V. Ramos, P. Esteves Verissimo, C. Esteve Rothenberg, S. Azodolmolky, and S. Uhlig, “Software-defined networking: a comprehensive survey,” Proceedings of the IEEE, vol. 103, no. 1, pp. 14–76, 2015.

[4]

A. Lara, A. Kolasani, and B. Ramamurthy, “Network innovation using OpenFlow: a survey,” IEEE Communications Surveys & Tutorials, vol. 16, no. 1, pp. 493–512, 2014.

[5]

M. Rezaee and M. H. Yaghmaee Moghaddam, “SDN-based quality of service networking for wide area measurement system,” IEEE Transactions on Industrial Informatics, vol. 16, no. 5, pp. 3018–3028, 2020.

[6]

S. Agarwal, M. Kodialam, and T. V. Lakshman, “Traffic engineering in software defined networks,” in Proceedings of the IEEE INFOCOM, pp. 2211–2219, Turin, Italy, April 2013.

[7]

Y. Guo, Z. Wang, X. Yin, X. Shi, and J. Wu, “Traffic engineering in SDN/OSPF hybrid network,” in Proceedings of the 2014 IEEE 22nd International Conference on Network Protocols, pp. 563–568, Raleigh, NC, USA, October 2014.

Digital Library

[8]

P. Amaral, J. Dinis, P. Pinto, L. Bernardo, J. Tavares, and H. S. Mamede, “Machine learning in software defined networks: data collection and traffic classification,” in Proceedings of the 2016 IEEE 24th International Conference on Network Protocols (ICNP), pp. 1–5, Singapore, November 2016.

[9]

J. Yan and J. Yuan, “A survey of traffic classification in software defined networks,” in Proceedings of the 2018 1st IEEE International Conference on Hot Information-Centric Networking (HotICN), pp. 200–206, Shenzhen, China, August 2018.

[10]

J. Touch, E. Lear, A. Mankin, M. Kojo, K. Ono, and M. Stiemerling, “Service name and transport protocol port number registry,” Internet Assigned Numbers Authority (IANA), https://www.iana.org/assignments/service-names-port-numbers.

[11]

Iana. Internet Assigned Numbers Authority, Internet Assigned Numbers Authority, https://www.iana.org/.

[12]

A. Moore and K. Papagiannaki, “Toward the accurate identification of network applications,” in Proceedings of the International Conference on Passive and Active Network Measurement, Boston, MA, USA, March 2005.

Digital Library

[13]

A. Madhukar and C. Williamson, “A longitudinal study of P2P traffic classification,” in Proceedings of the 14th IEEE International Symposium on Modeling, Analysis, and Simulation, pp. 179–188, Monterey, CA, USA, September 2006.

[14]

M. Finsterbusch, C. Richter, E. Rocha, J.-A. Muller, and K. Hanssgen, “A survey of payload-based traffic classification approaches,” IEEE Communications Surveys & Tutorials, vol. 16, no. 2, pp. 1135–1156, 2014.

[15]

D. Sanvito, D. Moro, and A. Capone, “Towards traffic classification offloading to stateful SDN data planes,” in Proceedings of the 2017 IEEE Conference on Network Softwarization (NetSoft), pp. 1–4, Bologna, Italy, July 2017.

[16]

M. Lopez-Martin, B. Carro, A. Sanchez-Esguevillas, and J. Lloret, “Network traffic classifier with convolutional and recurrent neural networks for Internet of Things,” IEEE Access, vol. 5, pp. 18042–18050, 2017.

[17]

P. Wang, F. Ye, X. Chen, and Y. Qian, “Datanet: deep learning based encrypted network traffic classification in SDN home gateway,” IEEE Access, vol. 6, pp. 55380–55391, 2018.

[18]

P. Wang, X. Chen, F. Ye, and Z. Sun, “A survey of techniques for mobile service encrypted traffic classification using deep learning,” IEEE Access, vol. 7, pp. 54024–54033, 2019.

[19]

K.-C. Chiu, C.-C. Liu, and L.-D. Chou, “CAPC: packet-based network service classifier with convolutional autoencoder,” IEEE Access, vol. 8, pp. 218081–218094, 2020.

[20]

V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, and D. Wierstra, “Playing Atari with deep reinforcement learning,” 2013, https://arxiv.org/abs/1312.5602.

[21]

A. Heng, “Bit-twist: libpcap-based Ethernet packet generator,” 2006, https://bittwist.sourceforge.net/.

[22]

C. Hsu and U. Kremer, “IPERF: a framework for automatic construction of performance prediction models,” in Proceedings of the Workshop on Profile and Feedback Directed Compilation, pp. 1–10, Paris, France, October 1998.

[23]

A. Binsahaq, T. R. Sheltami, and K. Salah, “A survey on autonomic provisioning and management of QoS in SDN networks,” IEEE Access, vol. 7, pp. 73384–73435, 2019.

[24]

K. Bouraqia, E. Sabir, M. Sadik, and L. Ladid, “Quality of experience for streaming services: measurements, challenges and insights,” IEEE Access, vol. 8, pp. 13341–13361, 2020.

[25]

A. T. Oliveira, B. J. C. A. Martins, M. F. Moreno, A. B. Vieira, A. T. A. Gomes, and A. Ziviani, “SDN-based architecture for providing QoS to high performance distributed applications,” in Proceedings of the 2018 IEEE Symposium on Computers and Communications (ISCC), pp. 602–607, Natal, Brazil, June 2018.

[26]

S. Tomovic, N. Prasad, and I. Radusinovic, “SDN control framework for QoS provisioning,” in Proceedings of the 2014 22nd Telecommunications Forum Telfor (TELFOR), pp. 111–114, Belgrade, Serbia, November 2014.

[27]

F.-H. Tseng, X. Wang, L.-D. Chou, H.-C. Chao, and V. C. M. Leung, “Dynamic resource prediction and allocation for cloud data center using the multiobjective genetic algorithm,” IEEE Systems Journal, vol. 12, no. 2, pp. 1688–1699, 2018.

[28]

D. C. Li, B.-H. Chen, C.-W. Tseng, and L.-D. Chou, “A novel genetic service function deployment management platform for edge computing,” Mobile Information Systems, vol. 2020, 22 pages, 2020.

[29]

C.-W. Tseng, F.-H. Tseng, Y.-T. Yang, C.-C. Liu, and L.-D. Chou, “Task scheduling for edge computing with agile VNFs on-demand service model toward 5G and beyond,” Wireless Communications and Mobile Computing, vol. 2018, pp. 1–13, 2018.

Digital Library

[30]

Z. Shu, J. Wan, J. Lin, S. Wang, D. Li, S. Rho, and C. Yang, “Traffic engineering in software-defined networking: measurement and management,” IEEE Access, vol. 4, pp. 3246–3256, 2016.

[31]

Y. Chiang, C. Ke, Y. Yu, Y. Chen, and C. Pan, “A multipath transmission scheme for the improvement of throughput over SDN,” in Proceedings of the 2017 International Conference on Applied System Innovation (ICASI), pp. 1247–1250, Sapporo, Japan, May 2017.

[32]

W. Yahya, A. Basuki, W. Maulana, S. R. Akbar, and A. Bhawiyuga, “Improving end-to-end network throughput using multiple best paths routing in software defined networking,” in Proceedings of the 2018 10th International Conference on Information Technology and Electrical Engineering (ICITEE), pp. 187–191, Bali, Indonesia, July 2018.

[33]

R. Challa, S. Jeon, D. S. Kim, and H. Choo, “CentFlow: centrality-based flow balancing and traffic distribution for higher network utilization,” IEEE Access, vol. 5, pp. 17045–17058, 2017.

[34]

F.-H. Tseng, M.-S. Tsai, C.-W. Tseng, Y.-T. Yang, C.-C. Liu, and L.-D. Chou, “A lightweight autoscaling mechanism for fog computing in industrial applications,” IEEE Transactions on Industrial Informatics, vol. 14, no. 10, pp. 4529–4537, 2018.

[35]

C.-W. Tseng, Y.-K. Huang, F.-H. Tseng, Y.-T. Yang, C.-C. Liu, and L.-D. Chou, “Micro operator design pattern in 5G SDN/NFV network,” Wireless Communications and Mobile Computing, vol. 2018, 14 pages, 2018.

Digital Library

[36]

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, UK, 1998.

Digital Library

[37]

C. J. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, no. 3-4, pp. 279–292, 1992.

Digital Library

[38]

J. Duan, S. Eben Li, Y. Guan, Q. Sun, and B. Cheng, “Hierarchical reinforcement learning for self-driving decision-making without reliance on labelled driving data,” IET Intelligent Transport Systems, vol. 14, no. 5, pp. 297–305, 2020.

[39]

N. C. Luong, D. T. Hoang, S. Gong, D. Niyato, P. Wang, Y.-C. Liang, and D. I. Kim, “Applications of deep reinforcement learning in communications and networking: a survey,” IEEE Communications Surveys & Tutorials, vol. 21, no. 4, pp. 3133–3174, 2019.

Digital Library

[40]

M. Latah and L. Toker, “Artificial intelligence enabled software-defined networking: a comprehensive overview,” IET Networks, vol. 8, no. 2, pp. 79–99, 2019.

Digital Library

[41]

M. B. Hossain and J. Wei, “Reinforcement learning-driven QoS-aware intelligent routing for software-defined networks,” in Proceedings of the 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1–5, Ottawa, ON, Canada, November 2019.

[42]

S. Lin, I. F. Akyildiz, P. Wang, and M. Luo, “QoS-aware adaptive routing in multi-layer hierarchical software defined networks: a reinforcement learning approach,” in Proceedings of the 2016 IEEE International Conference on Services Computing (SCC), pp. 25–33, San Francisco, CA, USA, June 2016.

[43]

C. Yu, J. Lan, Z. Guo, and Y. Hu, “DROM: optimizing the routing in software-defined networks with deep reinforcement learning,” IEEE Access, vol. 6, no. 18, pp. 54539–64533, 2018.

[44]

P. Sun, Y. Hu, J. Lan, L. Tian, and M. Chen, “TIDE: time-relevant deep reinforcement learning for routing optimization,” Future Generation Computer Systems, vol. 99, pp. 401–409, 2019.

Digital Library

[45]

G. Stampa, M. Arias, D. Sanchez-Charles, V. Muntes-Mulero, and A. Cabellos, “A deep-reinforcement learning approach for software-defined networking routing optimization,” pp. 1–3, 2017, https://arxiv.org/abs/1709.07080.

[46]

Q. T. A. Pham, Y. Hadjadj-Aoul, and A. Outtagarts, “Deep reinforcement learning based QoS-aware routing in knowledge-defined networking,” in Proceedings of the 14th EAI International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness, pp. 1–13, Ho Chi Minh City, Vietnam, December 2018.

[47]

X. Guo, H. Lin, Z. Li, and M. Peng, “Deep-reinforcement-learning-based QoS-aware secure routing for SDN-IoT,” IEEE Internet of Things Journal, vol. 7, no. 7, pp. 6242–6251, 2020.

[48]

J. Rischke, P. Sossalla, H. Salah, F. H. P. Fitzek, and M. Reisslein, “QR-SDN: towards reinforcement learning states, actions, and rewards for direct flow routing in software-defined networks,” IEEE Access, vol. 8, pp. 174773–174791, 2020.

[49]

M. Ibrar, L. Wang, G.-M. Muntean, J. Chen, N. Shah, and A. Akbar, “IHSF: an intelligent solution for improved performance of reliable and time-sensitive flows in hybrid SDN-based FC IoT systems,” IEEE Internet of Things Journal, vol. 8, no. 5, pp. 3130–3142, 2021.

[50]

K. Phemius and M. Bouet, “Monitoring latency with OpenFlow,” in Proceedings of the 9th International Conference on Network and Service Management (CNSM 2013), pp. 122–125, Zurich, Switzerland, October 2013.

[51]

D. P. Kingma and J. L. Ba, “Adam: a method for stochastic optimization,” in Proceedings of the 3rd International Conference for Learning Representations, pp. 1–41, San Diego, CA, USA, December 2015.

[52]

Mininet Team, “Mininet: an instant virtual network on your laptop (or other PC),” 2018, https://mininet.org/.

Cited By

Aguirre Sanchez LShen YGuo M(2024)DQSJournal of Parallel and Distributed Computing10.1016/j.jpdc.2024.104851188:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.jpdc.2024.104851
Sue Ling JTan SWei SZaki MOmar N(2024)Dynamic SDN Multiple Nodes Migration Using SARSA Reinforcement LearningMobile Web and Intelligent Information Systems10.1007/978-3-031-68005-2_4(43-54)Online publication date: 18-Aug-2024
https://dl.acm.org/doi/10.1007/978-3-031-68005-2_4

Recommendations

Survey on Reinforcement Learning based Efficient Routing in SDN
SMA 2020: The 9th International Conference on Smart Media and Applications

Traffic on the Internet is growing exponentially around the globe, and SDN (Software Defined Network) becomes very attractive to network research communities. The trend switching from a hardware-centered network to a software-centric network allows that ...
Dynamic SDN Multiple Nodes Migration Using SARSA Reinforcement Learning
Mobile Web and Intelligent Information Systems
Abstract
This article addresses the problem of software-defined Networking (SDN) migration process under dynamic network conditions and present a SARSA-based approach to tackle the challenges associated with it. Traditional SDN migration techniques often ...
Integrating recurrent neural networks and reinforcement learning for dynamic service composition
Abstract
In the service oriented architecture (SOA), software and systems are abstracted as web services to be invoked by other systems. Service composition is a technology, which builds a complex system by combining existing simple services. ...
Highlights
- The QoS prediction method based on a recurrent neural network is developed.
- We ...

Comments

Information & Contributors

Information

Published In

cover image Wireless Communications & Mobile Computing

Wireless Communications & Mobile Computing Volume 2022, Issue

2022

25330 pages

ISSN:1530-8669

Issue’s Table of Contents

Copyright © 2022 Kai-Cheng Chiu et al.

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Publisher

John Wiley and Sons Ltd.

United Kingdom

Publication History

Published: 01 January 2022

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Aguirre Sanchez LShen YGuo M(2024)DQSJournal of Parallel and Distributed Computing10.1016/j.jpdc.2024.104851188:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.jpdc.2024.104851
Sue Ling JTan SWei SZaki MOmar N(2024)Dynamic SDN Multiple Nodes Migration Using SARSA Reinforcement LearningMobile Web and Intelligent Information Systems10.1007/978-3-031-68005-2_4(43-54)Online publication date: 18-Aug-2024
https://dl.acm.org/doi/10.1007/978-3-031-68005-2_4

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents