research-article

Open access

Placement Optimization with Deep Reinforcement Learning

Authors:

Azalia MirhoseiniAuthors Info & Claims

ISPD '20: Proceedings of the 2020 International Symposium on Physical Design

Pages 3 - 7

https://doi.org/10.1145/3372780.3378174

Published: 30 March 2020 Publication History

Abstract

Placement Optimization is an important problem in systems and chip design, which consists of mapping the nodes of a graph onto a limited set of resources to optimize for an objective, subject to constraints. In this paper, we start by motivating reinforcement learning as a solution to the placement problem. We then give an overview of what deep reinforcement learning is. We next formulate the placement problem as a reinforcement learning problem, and show how this problem can be solved with policy gradient optimization. Finally, we describe lessons we have learned from training deep reinforcement learning policies across a variety of placement optimization problems.

References

[1]

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2013. Spectral Networks and Locally Connected Networks on Graphs. arXiv:cs.LG/1312.6203

[2]

A. Chakraborty, A. Kumar, and D. Z. Pan. 2009. RegPlace: A high quality opensource placement framework for structured ASICs. In 2009 46th ACM/IEEE Design Automation Conference. 442--447.

[3]

C. Cheng, A. B. Kahng, I. Kang, and L. Wang. 2019. RePlAce: Advancing Solution Quality and Routability Validation in Global Placement. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 38, 9 (2019), 1717--1730.

Digital Library

[4]

J. P. Cohoon and W. D. Paris. 1987. Genetic Placement. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 6, 6 (November 1987), 956--964. https://doi.org/10.1109/TCAD.1987.1270337

Digital Library

[5]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. arXiv:cs.LG/1606.09375

[6]

H. Esbensen. 1992. A genetic algorithm for macro cell placement. In Proceedings EURO-DAC '92: European Design Automation Conference. 52--57. https://doi.org/ 10.1109/EURDAC.1992.246265

[7]

C. Gallicchio and A. Micheli. 2010. Graph Echo State Networks. In The 2010 International Joint Conference on Neural Networks (IJCNN). 1--8. https://doi.org/ 10.1109/IJCNN.2010.5596796

[8]

M. Gori, G. Monfardini, and F. Scarselli. 2005. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., Vol. 2. 729--734 vol. 2. https://doi.org/10.1109/IJCNN.2005.1555942

[9]

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv:cs.LG/1801.01290

[10]

Mikael Henaff, Joan Bruna, and Yann LeCun. 2015. Deep Convolutional Networks on Graph-Structured Data. arXiv:cs.LG/1506.05163

[11]

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. 2019. When to Trust Your Model: Model-Based Policy Optimization. arXiv:cs.LG/1906.08253

[12]

Myung-Chul Kim, Jin Hu, Dong-Jin Lee, and Igor L. Markov. 2011. A SimPLR Method for Routability-Driven Placement. In Proceedings of the International Conference on Computer-Aided Design (San Jose, California) (ICCAD '11). IEEE Press, 67--73.

[13]

S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. 1983. Optimization by Simulated Annealing. Science 220, 4598 (1983), 671--680. https://doi.org/10.1126/science. 220.4598.671 arXiv:https://science.sciencemag.org/content/220/4598/671.full.pdf

[14]

Yibo Lin, Shounak Dhar, Wuxi Li, Haoxing Ren, Brucek Khailany, and David Z. Pan. 2019. DREAMPlace: Deep Learning Toolkit-Enabled GPU Acceleration for Modern VLSI Placement. In Proceedings of the 56th Annual Design Automation Conference 2019 (DAC '19). Association for Computing Machinery, New York, NY, USA.

[15]

Jingwei Lu, Pengwen Chen, Chin-Chih Chang, Lu Sha, Dennis Jen-Hsin Huang, Chin-Chi Teng, and Chung-Kuan Cheng. 2015. EPlace: Electrostatics-Based Placement Using Fast Fourier Transform and Nesterov's Method. 20, 2 (2015).

[16]

Azalia Mirhoseini, Anna Goldie, Hieu Pham, Benoit Steiner, Quoc V Le, and Jeff Dean. 2018. A Hierarchical Model for Device Placement. In ICLR.

[17]

Azalia Mirhoseini, Hieu Pham, Quoc V. Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. 2017. Device Placement Optimization with Reinforcement Learning. In ICML.

[18]

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous Methods for Deep Reinforcement Learning. arXiv:cs.LG/1602.01783

Digital Library

[19]

OpenAI. [n.d.]. OpenAI Five. https://blog.openai.com/openai-five/.

[20]

F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. 2009. The Graph Neural Network Model. IEEE Transactions on Neural Networks 20, 1 (Jan 2009), 61--80. https://doi.org/10.1109/TNN.2008.2005605

Digital Library

[21]

John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. 2015. Trust Region Policy Optimization. arXiv:cs.LG/1502.05477

[22]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:cs.LG/1707.06347

[23]

Huang-A. Maddison C Silver, D. 2016. Mastering the game of Go with deep neural networks and tree search. Nature (2016).

[24]

Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojtek Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dalibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver. 2019. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. https://deepmind.com/blog/alphastarmastering- real-time-strategy-game-starcraft-ii/.

[25]

R.J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn (1992).

[26]

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. 2019. A Comprehensive Survey on Graph Neural Networks. arXiv:cs.LG/1901.00596

[27]

Jinjun Xiong, Yiu-Chung Wong, Egino Sarto, and Lei He1. 2006. Constraint Driven I/O Planning and Placement for Chip-package Co-design. In APSDAC.

[28]

Yanqi Zhou, Sudip Roy, Amirali Abdolrashidi, Daniel Wong, Peter C. Ma, Qiumin Xu, Ming Zhong, Hanxiao Liu, Anna Goldie, Azalia Mirhoseini, and James Laudon. 2019. GDP: Generalized Device Placement for Dataflow Graphs. arXiv:cs.LG/1910.01578

Cited By

Liu XYin X(2024)Deep Reinforcement Learning-Driven Wireless Sensor Placement for IIoT-Integrated Smart Production2024 7th International Conference on Advanced Algorithms and Control Engineering (ICAACE)10.1109/ICAACE61206.2024.10548592(1120-1126)Online publication date: 1-Mar-2024
https://doi.org/10.1109/ICAACE61206.2024.10548592
Kaven LHuke PGöppert ASchmitt R(2024)Multi agent reinforcement learning for online layout planning and scheduling in flexible assembly systemsJournal of Intelligent Manufacturing10.1007/s10845-023-02309-8Online publication date: 27-Jan-2024
https://doi.org/10.1007/s10845-023-02309-8
Qiu YXing YZheng XGao PCai SXiong X(2023)Progress of Placement Optimization for Accelerating VLSI Physical DesignElectronics10.3390/electronics1202033712:2(337)Online publication date: 9-Jan-2023
https://doi.org/10.3390/electronics12020337
Show More Cited By

Index Terms

Placement Optimization with Deep Reinforcement Learning
1. Hardware
  1. Electronic design automation
    1. Physical design (EDA)
      1. Placement
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Reinforcement learning

Recommendations

Reinforcement Learning for Placement Optimization
ISPD '21: Proceedings of the 2021 International Symposium on Physical Design

In the past decade, computer systems and chips have played a key role in the success of artificial intelligence (AI). Our vision in Google Brain's Machine Learning for Systems team is to use AI to transform the way in which computer systems and chips ...
Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems

Recent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...
Robust Deep Reinforcement Learning with Adversarial Attacks
AAMAS '18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems

This paper proposes adversarial attacks for Reinforcement Learning (RL). These attacks are then leveraged during training to improve the robustness of RL within robust control framework. We show that this adversarial training of DRL algorithms like Deep ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISPD '20: Proceedings of the 2020 International Symposium on Physical Design

March 2020

160 pages

ISBN:9781450370912

DOI:10.1145/3372780

General Chair:
William Swartz
TimberWolf Systems and University of Texas at Dallas, USA
,
Program Chair:
Jens Lienig
Dresden University of Technology, Germany

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGDA: ACM Special Interest Group on Design Automation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 March 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Google

Conference

ISPD '20

Sponsor:

SIGDA

ISPD '20: International Symposium on Physical Design

September 20 - 23, 2020

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 62 of 172 submissions, 36%

Upcoming Conference

ISPD '25

Sponsor:
sigda

International Symposium on Physical Design

March 16 - 19, 2025

Austin , TX , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
2,969
Total Downloads

Downloads (Last 12 months)595
Downloads (Last 6 weeks)122

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu XYin X(2024)Deep Reinforcement Learning-Driven Wireless Sensor Placement for IIoT-Integrated Smart Production2024 7th International Conference on Advanced Algorithms and Control Engineering (ICAACE)10.1109/ICAACE61206.2024.10548592(1120-1126)Online publication date: 1-Mar-2024
https://doi.org/10.1109/ICAACE61206.2024.10548592
Kaven LHuke PGöppert ASchmitt R(2024)Multi agent reinforcement learning for online layout planning and scheduling in flexible assembly systemsJournal of Intelligent Manufacturing10.1007/s10845-023-02309-8Online publication date: 27-Jan-2024
https://doi.org/10.1007/s10845-023-02309-8
Qiu YXing YZheng XGao PCai SXiong X(2023)Progress of Placement Optimization for Accelerating VLSI Physical DesignElectronics10.3390/electronics1202033712:2(337)Online publication date: 9-Jan-2023
https://doi.org/10.3390/electronics12020337
Koblah DAcharya RCapecci DDizon-Paradis OTajik SGanji FWoodard DForte D(2023)A Survey and Perspective on Artificial Intelligence for Security-Aware Electronic Design AutomationACM Transactions on Design Automation of Electronic Systems10.1145/356339128:2(1-57)Online publication date: 6-Mar-2023
https://dl.acm.org/doi/10.1145/3563391
Guan WTang XLu HZhang YZhang Y(2023)ATT-TA: A Cooperative Multiagent Deep Reinforcement Learning Approach for TSV Assignment in 3-D ICsIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.332153631:12(1905-1917)Online publication date: Dec-2023
https://doi.org/10.1109/TVLSI.2023.3321536
Guan WTang XLu HZhang YZhang Y(2023)Thermal-Aware Fixed-Outline 3-D IC Floorplanning: An End-to-End Learning-Based ApproachIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.332153231:12(1882-1895)Online publication date: Dec-2023
https://doi.org/10.1109/TVLSI.2023.3321532
Guan WTang XLu HZhang YZhang Y(2023)A Novel Thermal-Aware Floorplanning and TSV Assignment With Game Theory for Fixed-Outline 3-D ICsIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.330959531:11(1639-1652)Online publication date: Nov-2023
https://doi.org/10.1109/TVLSI.2023.3309595
Myung WLee DSong CWang GMa C(2023)Policy Gradient-Based Core Placement Optimization for Multichip Many-Core SystemsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2021.311787834:8(4529-4543)Online publication date: Aug-2023
https://doi.org/10.1109/TNNLS.2021.3117878
Volya DPan ZMishra P(2023)Feedback-Based Steering for Quantum State Preparation2023 IEEE International Conference on Quantum Computing and Engineering (QCE)10.1109/QCE57702.2023.00148(1308-1318)Online publication date: 17-Sep-2023
https://doi.org/10.1109/QCE57702.2023.00148
Mazloomi ASami HBentahar JOtrok HMourad A(2023)Reinforcement Learning Framework for Server Placement and Workload Allocation in Multiaccess Edge ComputingIEEE Internet of Things Journal10.1109/JIOT.2022.320505110:2(1376-1390)Online publication date: 15-Jan-2023
https://doi.org/10.1109/JIOT.2022.3205051
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents