research-article

DeepHop on Edge: Hop-by-hop Routing byDistributed Learning with Semantic Attention

Authors:

Jianxin LiaoAuthors Info & Claims

ICPP '20: Proceedings of the 49th International Conference on Parallel Processing

Article No.: 58, Pages 1 - 11

https://doi.org/10.1145/3404397.3404425

Published: 17 August 2020 Publication History

Abstract

Multi-access Edge Computing (MEC) and ubiquitous smart devices help serve end-users efficiently and optimally through providing emerging edge-deployed services. Meanwhile, heavy and time-varying traffic loads are produced in the edge network, so that an efficient traffic forwarding mechanism is required. In this paper, we propose a parallel and distributed learning approach, DeepHop, to adapt to the volatile environments and realize hop-by-hop routing. The Multi-Agent Deep Reinforcement Learning (MADRL) is used to alleviate the edge network congestion and maximize the utilization of network resources. DeepHop determines the routing among edge network nodes for heterogeneous types of traffic according to the current workload and capability. By joining with an attention mechanism, DeepHop obtains the semantics from the elements of the network state to help the agents learn the importance of each element on routing. Experiment results show that DeepHop achieves the increase of successfully transmitted packets by 15% compared with the state-of-the-art algorithms. Besides, DeepHop with an attention mechanism reduces convergence time by nearly half compared with the common-used structures of neural networks.

References

[1]

L. Buşoniu, R. Babuška, and B. De Schutter. 2010. Multi-agent reinforcement learning: An overview. In Innovations in multi-agent systems and applications-1. Springer, 183–221.

[2]

S. Chaudhari, G. Polatkan, R. Ramanath, and V. Mithal. 2019. An attentive survey of attention models. arXiv preprint arXiv:1904.02874(2019).

[3]

X. Chu. 2018. Policy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization. arXiv preprint arXiv:1807.00442(2018).

[4]

Q. Cui, Z. Gong, W. Ni, Y. Hou, X. Chen, X. Tao, and P. Zhang. 2019. Stochastic online learning for mobile edge computing: Learning from changes. IEEE Communications Magazine 57, 3 (2019), 63–69.

Digital Library

[5]

H. Hsieh, J. Chen, and A. Benslimane. 2018. 5G virtualized multi-access edge computing platform for IoT applications. Journal of Network and Computer Applications 115 (2018), 94–102.

Digital Library

[6]

J. Hu and M. P. Wellman. 2003. Nash Q-learning for general-sum stochastic games. Journal of machine learning research 4, Nov (2003), 1039–1069.

[7]

B. Hubert, J. Geul, and S. Séhier. [n.d.]. The Wonder Shaper. https://github.com/magnific0/wondershaper.

[8]

J. Jiang, C. Dun, and Z. Lu. 2018. Graph Convolutional Reinforcement Learning for Multi-Agent Cooperation. CoRR abs/1810.09202(2018).

[9]

S. K., H.X. N., N. F., R. B., and M. R.2011. The Internet Topology Zoo. IEEE Journal on Selected Areas in Communications 29, 9(2011), 1765–1775.

[10]

S. M. Kakade. 2002. A natural policy gradient. In Advances in neural information processing systems. 1531–1538.

[11]

A. A. Khuwaja, Y. Chen, N. Zhao, M. Alouini, and P. Dobbins. 2018. A survey of channel modeling for UAV communications. IEEE Communications Surveys & Tutorials 20, 4 (2018), 2804–2821.

Digital Library

[12]

V. R. Konda and J. N. Tsitsiklis. 2003. Onactor-critic algorithms. SIAM journal on Control and Optimization 42, 4 (2003), 1143–1166.

Digital Library

[13]

A. H. Lashkari, G. Draper-Gil, M. S. I. Mamun, and A. A. Ghorbani. 2017. Characterization of Tor Traffic using Time based Features. In ICISSP. 253–262.

[14]

D. Li, P. Hong, K. Xue, and J. Pei. 2019. Virtual network function placement and resource optimization in NFV and edge computing enabled networks. Computer Networks 152(2019), 12–24.

Digital Library

[15]

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2016. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.

[16]

Z. Lin, M. Feng, C. N. Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio. 2017. A structured self-attentive sentence embedding. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.

[17]

R. Lowe, Y. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems. 6379–6390.

[18]

N. Michael and A. Tang. 2014. Halo: Hop-by-hop adaptive link-state optimal routing. IEEE/ACM Transactions on Networking 23, 6 (2014), 1862–1875.

Digital Library

[19]

V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International conference on machine learning. 1928–1937.

[20]

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529.

[21]

S. Muralidharan, A. Roy, and N. Saxena. 2018. MDP-IoT: MDP based interest forwarding for heterogeneous traffic in IoT-NDN environment. Future Generation Comp. Syst. 79 (2018), 892–908.

[22]

J. Nash. 1951. Non-cooperative games. Annals of mathematics(1951), 286–295.

[23]

K. Poularakis, J. Llorca, A. M. Tulino, I. Taylor, and L. Tassiulas. 2019. Joint Service Placement and Request Routing in Multi-cell Mobile Edge Computing Networks. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 10–18.

[24]

J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. 2015. Trust region policy optimization. In International Conference on Machine Learning. 1889–1897.

[25]

H. Trinh, P. Calyam, D. Chemodanov, S. Yao, Q. Lei, F. Gao, and K. Palaniappan. 2018. Energy-aware mobile edge computing and routing for low-latency visual data processing. IEEE Transactions on Multimedia 20, 10 (2018), 2562–2577.

[26]

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Łu. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998–6008.

[27]

J. Wang, B. He, J. Wang, and T. Li. 2018. Intelligent VNFs Selection based on Traffic Identification in Vehicular Cloud Networks. IEEE Transactions on Vehicular Technology 68, 5 (2018), 4140–4147.

[28]

W. Wang, X. Liu, Y. Yao, Y. Pan, Z. Chi, and T. Zhu. 2019. CRF: Coexistent Routing and Flooding using WiFi Packets in Heterogeneous IoT Networks. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 19–27.

[29]

Z. K. Wazir, A. Ejaz, H. Saqib, Y. Ibrar, and Arif A.2019. Edge computing: A survey. Future Generation Comp. Syst. 97 (2019), 219–235.

Digital Library

[30]

Y. Wu, E. Mansimov, R. B Grosse, S. Liao, and J. Ba. 2017. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In Advances in neural information processing systems. 5279–5288.

[31]

Z. Xu, J. Tang, J. Meng, W. Zhang, Y. Wang, C. H. Liu, and D. Yang. 2018. Experience-driven networking: A deep reinforcement learning based approach. In 2018 IEEE Conference on Computer Communications, INFOCOM 2018, Honolulu, HI, USA, April 16-19, 2018. IEEE, 1871–1879.

Digital Library

[32]

Z. Xu, J. Tang, C. Yin, Y. Wang, and G. Xue. 2019. Experience-driven Congestion Control: When Multi-Path TCP Meets Deep Reinforcement Learning. IEEE Journal on Selected Areas in Communications 37, 6(2019), 1325–1336.

[33]

Y. Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and J. Wang. 2018. Mean field multi-agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, Stockholmsmässan, Stockholm, Sweden, July 10-15. 5567–5576.

[34]

H. Yu, B. Ng, and W. K. G. Seah. 2018. TTL-Based Efficient Forwarding for Nanonetworks With Multiple Coordinated IoT Gateways. IEEE Internet of Things Journal 5, 3 (2018), 1807–1815.

[35]

H. Zhang, W. Li, S. Gao, X. Wang, and B. Ye. 2019. ReLeS: A Neural Adaptive Multipath Scheduler based on Deep Reinforcement Learning. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 1648–1656.

[36]

Z. Zhao, Y. Shi, B. Diao, and B. Wu. 2019. Optimal Data Caching and Forwarding in Industrial IoT With Diverse Connectivity. IEEE Trans. Industrial Informatics 15, 4 (2019), 2288–2296.

Cited By

Le VLong Nguyen DLe Nguyen PJi Y(2024)Traffic Engineering in Large-scale Networks via Multi-Agent Deep Reinforcement Learning with Joint-Training2024 33rd International Conference on Computer Communications and Networks (ICCCN)10.1109/ICCCN61486.2024.10637556(1-9)Online publication date: 29-Jul-2024
https://doi.org/10.1109/ICCCN61486.2024.10637556
Gu HZhao LHan ZZheng GSong S(2024)AI-Enhanced Cloud-Edge-Terminal Collaborative Network: Survey, Applications, and Future DirectionsIEEE Communications Surveys & Tutorials10.1109/COMST.2023.333815326:2(1322-1385)Online publication date: Oct-2025
https://doi.org/10.1109/COMST.2023.3338153
Han RChen DGuo SWang JQi QLu LLiao J(2023)Multi-SP Network Slicing Parallel Relieving Edge Network ConflictIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.331001334:11(2860-2875)Online publication date: Nov-2023
https://doi.org/10.1109/TPDS.2023.3310013
Show More Cited By

Recommendations

Hop-by-hop routing algorithms for premium traffic

In Differentiated Service (DiffServ) networks, the routing algorithms used by the premium class traffic, due to the high priority afforded to that traffic, may have a significant impact not only on the premium class traffic itself, but on all other ...
A hop-by-hop delay-constrained routing algorithm with explicit loop avoidance and backup routing information: Research Articles

QoS Routing is crucial for QoS provisioning in high-speed networks. In general, QoS routing can be classified into two paradigms: source routing and hop-by-hop routing. In source routing, the entire path to the destination node of a communication ...
Load balancing in hop-by-hop routing with and without traffic splitting

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICPP '20: Proceedings of the 49th International Conference on Parallel Processing

August 2020

844 pages

ISBN:9781450388160

DOI:10.1145/3404397

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

the Beijing Municipal Natural Science Foundation
BUPT Excellent Ph.D. Students Foundation
the National Natural Science Foundation of China
the Ministry of Education and China Mobile Joint Fund

Conference

ICPP '20

ICPP '20: 49th International Conference on Parallel Processing

August 17 - 20, 2020

AB, Edmonton, Canada

Acceptance Rates

Overall Acceptance Rate 91 of 313 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
177
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)5

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Le VLong Nguyen DLe Nguyen PJi Y(2024)Traffic Engineering in Large-scale Networks via Multi-Agent Deep Reinforcement Learning with Joint-Training2024 33rd International Conference on Computer Communications and Networks (ICCCN)10.1109/ICCCN61486.2024.10637556(1-9)Online publication date: 29-Jul-2024
https://doi.org/10.1109/ICCCN61486.2024.10637556
Gu HZhao LHan ZZheng GSong S(2024)AI-Enhanced Cloud-Edge-Terminal Collaborative Network: Survey, Applications, and Future DirectionsIEEE Communications Surveys & Tutorials10.1109/COMST.2023.333815326:2(1322-1385)Online publication date: Oct-2025
https://doi.org/10.1109/COMST.2023.3338153
Han RChen DGuo SWang JQi QLu LLiao J(2023)Multi-SP Network Slicing Parallel Relieving Edge Network ConflictIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.331001334:11(2860-2875)Online publication date: Nov-2023
https://doi.org/10.1109/TPDS.2023.3310013
He BWang JQi QSun HLiao JLu LHan Z(2023)Learning-Based Real-Time Transmission Control for Multi-Path TCP NetworksIEEE Transactions on Cognitive Communications and Networking10.1109/TCCN.2023.32872349:5(1353-1369)Online publication date: Oct-2023
https://doi.org/10.1109/TCCN.2023.3287234
Li TZhu KLuong NNiyato DWu QZhang YChen B(2022)Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive SurveyIEEE Communications Surveys & Tutorials10.1109/COMST.2022.316069724:2(1240-1279)Online publication date: Oct-2023
https://doi.org/10.1109/COMST.2022.3160697
He BWang JQi QSun HLiao JDu CYang XHan Z(2021)DeepCC: Multi-Agent Deep Reinforcement Learning Congestion Control for Multi-Path TCP Based on Self-AttentionIEEE Transactions on Network and Service Management10.1109/TNSM.2021.309330218:4(4770-4788)Online publication date: Dec-2021
https://doi.org/10.1109/TNSM.2021.3093302

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten