research-article

Improved Route Selection Approaches using Q-learning framework for 2D NoCs

Authors:

Manoj Singh Gaur,

Masoud Daneshtalab,

Masoumeh EbrahimiAuthors Info & Claims

MES '15: Proceedings of the 3rd International Workshop on Many-core Embedded Systems

Pages 33 - 40

https://doi.org/10.1145/2768177.2768180

Published: 13 June 2015 Publication History

Abstract

With the emergence of large multi-core architectures, a volume of research has been focused on distributing traffic evenly over the whole network. However, increase in traffic density may lead to congestion and subsequently degrade the performance by increased latency in the network. In this paper, we propose two novel route selection strategies for on-chip networks which are based on the Q-learning framework. The proposed strategies use variable learning rate to dynamically capture the current congestion status of the network using an additional parameter and improves the learning process to select a less congested output channel. Both the proposed selection strategies are found to adapt significantly faster to the changes in traffic load and traffic patterns by avoiding congested areas. The results demonstrate that proposed strategies achieve significant performance improvement over conventional Q-routing and its variants with slight area-overhead.

References

[1]

L. Benini and G. De Micheli. Networks on chips: A new soc paradigm. Computer, 35(1):70--78, 2002.

Digital Library

[2]

S. Choi and D.-Y. Yeung. Predictive q-routing: A memory-based reinforcement learning approach to adaptive traffic control. Advances in Neural Information Processing Systems, 8:945--951, 1996.

[3]

M. Ebrahimi, M. Daneshtalab, F. Farahnakian, J. Plosila, P. Liljeberg, M. Palesi, and H. Tenhunen. HARAQ: congestion-aware learning model for highly adaptive routing algorithm in on-chip networks. In Proceedings of the 6th IEEE International Symposium on Networks-on-Chip, pages 19--26, 2012.

Digital Library

[4]

M. Ebrahimi, M. Daneshtalab, P. Liljeberg, J. Plosila, and H. Tenhunen. Agent-based on-chip network using efficient selection method. In Proceedings of the 19th International Conference on VLSI and System-on-Chip, pages 284--289, 2011.

[5]

M. Ebrahimi, M. Daneshtalab, P. Liljeberg, J. Plosila, and H. Tenhunen. CATRA-congestion aware trapezoid-based routing algorithm for on-chip networks. In Design, Automation & Test in Europe Conference & Exhibition, pages 320--325, 2012.

Digital Library

[6]

F. Farahnakian, M. Ebrahimi, M. Daneshtalab, P. Liljeberg, and J. Plosila. Q-learning based congestion-aware routing algorithm for on-chip network. In Proceedings of the 2nd IEEE International Conference on Networked Embedded Systems for Enterprise Applications, pages 1--7, 2011.

Digital Library

[7]

F. Farahnakian, M. Ebrahimi, M. Daneshtalab, P. Liljeberg, and J. Plosila. Adaptive load balancing in learning-based approaches for many-core embedded systems. The Journal of Supercomputing, 68(3):1214--1234, 2014.

Digital Library

[8]

F. Farahnakian, M. Ebrahimi, M. Daneshtalab, P. Liljeberg, and J. Plosila. Bi-lcq: A low-weight clustering-based q-learning approach for nocs. Microprocessors and Microsystems - Embedded Hardware Design, 38(1):64--75, 2014.

Digital Library

[9]

F. Farahnakian, M. Ebrahimi, M. Daneshtalab, J. Plosila, and P. Liljeberg. Optimized q-learning model for distributing traffic in on-chip networks. In Proceedings of the 3rd IEEE International Conference on Networked Embedded Systems for Every Application, pages 1--8, 2012.

Digital Library

[10]

P. Gratz, B. Grot, and S. W. Keckler. Regional congestion awareness for load balance in networks-on-chip. In IEEE 14th International Symposium on High Performance Computer Architecture, pages 203--214, 2008.

[11]

S. Kumar and R. Miikkulainen. Dual reinforcement q-routing: An on-line adaptive routing algorithm. In Artificial neural networks in engineering, 1997.

[12]

S. Kumar and R. Miikkulainen. Confidence based dual reinforcement q-routing: An adaptive online network routing algorithm. In IJCAI, volume 99, pages 758--763, 1999.

Digital Library

[13]

M. Li, Q.-A. Zeng, and W.-B. Jone. DyXY: a proximity congestion-aware deadlock-free dynamic routing method for network on chip. In Proceedings of the 43rd Design Automation Conference, pages 849--852. ACM, 2006.

Digital Library

[14]

S. Ma, N. E. Jerger, and Z. Wang. Dbar: an efficient routing algorithm to support multiple concurrent applications in networks-on-chip. In Proceedings of the 38th IEEE International Symposium on Computer Architecture, pages 413--424, 2011.

Digital Library

[15]

M. Palesi and M. Daneshtalab. Routing Algorithms in Networks-on-Chip. Springer Publishing Company, Incorporated, 2013.

Digital Library

[16]

M. K. Puthal, V. Singh, M. S. Gaur, and V. Laxmi. C-routing: An adaptive hierarchical noc routing methodology. In Proceedings of the 19th International Conference on VLSI and System-on-Chip, pages 392--397, 2011.

[17]

R. S. Sutton and A. G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998.

Digital Library

[18]

C. Watkins. Q-learning. Technical report, University of Cambridge, England, 1992.

Cited By

Khan KPasricha S(2023)A Reinforcement Learning Framework With Region-Awareness and Shared Path Experience for Efficient Routing in Networks-on-ChipIEEE Design & Test10.1109/MDAT.2023.330671940:6(76-85)Online publication date: Dec-2023
https://doi.org/10.1109/MDAT.2023.3306719
Zhang XDong DLi CWang SXiao L(2023)A Survey of Machine Learning for Network-on-ChipsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2023.104778(104778)Online publication date: Nov-2023
https://doi.org/10.1016/j.jpdc.2023.104778
Singh RBohra MHemrajani PKalla ABhatt DPurohit NDaneshtalab M(2022)Review, Analysis, and Implementation of Path Selection Strategies for 2D NoCsIEEE Access10.1109/ACCESS.2022.322746010(129245-129268)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3227460
Show More Cited By

Index Terms

Improved Route Selection Approaches using Q-learning framework for 2D NoCs
1. Hardware
  1. Communication hardware, interfaces and storage

Recommendations

Application of reinforcement learning to routing in distributed wireless networks: a review

The dynamicity of distributed wireless networks caused by node mobility, dynamic network topology, and others has been a major challenge to routing in such networks. In the traditional routing schemes, routing decisions of a wireless node may solely ...
Improved Exploration Strategy for Q-Learning Based Multipath Routing in SDN Networks
Abstract
Software-Defined Networking (SDN) is characterized by a high level of programmability and offers a rich set of capabilities for network management operations. Network intelligence is centralized in the controller, which is responsible for updating ...
Backward Q-learning: The combination of Sarsa algorithm and Q-learning

Reinforcement learning (RL) has been applied to many fields and applications, but there are still some dilemmas between exploration and exploitation strategy for action selection policy. The well-known areas of reinforcement learning are the Q-learning ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MES '15: Proceedings of the 3rd International Workshop on Many-core Embedded Systems

June 2015

61 pages

ISBN:9781450334082

DOI:10.1145/2768177

Program Chairs:
Masoumeh Ebrahimi
University of Turku, Finland and KTH, Sweden
,
Diana Goehringer
Ruhr-University Bochum, Germany

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Univ. Turku: University of Turku
KTH (The Royal Institute of Technology), Sweden

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

MES '15

MES '15: The third International Workshop on Many-core Embedded Systems

June 13 - 14, 2015

OR, Portland, USA

Acceptance Rates

Overall Acceptance Rate 5 of 21 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
160
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Khan KPasricha S(2023)A Reinforcement Learning Framework With Region-Awareness and Shared Path Experience for Efficient Routing in Networks-on-ChipIEEE Design & Test10.1109/MDAT.2023.330671940:6(76-85)Online publication date: Dec-2023
https://doi.org/10.1109/MDAT.2023.3306719
Zhang XDong DLi CWang SXiao L(2023)A Survey of Machine Learning for Network-on-ChipsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2023.104778(104778)Online publication date: Nov-2023
https://doi.org/10.1016/j.jpdc.2023.104778
Singh RBohra MHemrajani PKalla ABhatt DPurohit NDaneshtalab M(2022)Review, Analysis, and Implementation of Path Selection Strategies for 2D NoCsIEEE Access10.1109/ACCESS.2022.322746010(129245-129268)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3227460
Rovira-Sugranes AAfghah FQu JRazi A(2021)Fully-Echoed Q-Routing With Simulated Annealing Inference for Flying Adhoc NetworksIEEE Transactions on Network Science and Engineering10.1109/TNSE.2021.30855148:3(2223-2234)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TNSE.2021.3085514
Zhang WYe Y(2019)An Approximate Thermal-Aware Q-Routing for Optical NoCs2019 IEEE/ACM Workshop on Photonics-Optics Technology Oriented Networking, Information and Computing Systems (PHOTONICS)10.1109/PHOTONICS49561.2019.00009(22-27)Online publication date: Nov-2019
https://doi.org/10.1109/PHOTONICS49561.2019.00009
Bitaillou AParrein BAndrieux G(2019)Q-routing: From the Algorithm to the Routing ProtocolMachine Learning for Networking10.1007/978-3-030-45778-5_5(58-69)Online publication date: 3-Dec-2019
https://dl.acm.org/doi/10.1007/978-3-030-45778-5_5
Kavalerov MShilova YLikhacheva Y(2017)Adaptive Q-routing with Random Echo and Route MemoryProceedings of the 20th Conference of Open Innovations Association FRUCT10.23919/FRUCT.2017.8071304(138-145)Online publication date: 10-Apr-2017
https://dl.acm.org/doi/10.23919/FRUCT.2017.8071304
Kavalerov MLikhacheva YShilova Y(2017)A reinforcement learning approach to network routing based on adaptive learning rates and route memorySoutheastCon 201710.1109/SECON.2017.7925316(1-6)Online publication date: Mar-2017
https://doi.org/10.1109/SECON.2017.7925316
Gupta NSharma ALaxmi VGaur MZwolinski MBishnoi R(2016) σ n LBDR: generic congestion handling routing implementation for two‐dimensional mesh network‐on‐chip IET Computers & Digital Techniques10.1049/iet-cdt.2015.019610:5(226-232)Online publication date: Sep-2016
https://doi.org/10.1049/iet-cdt.2015.0196

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten