research-article

Reinforcement learning for industrial process control: : A case study in flatness control in steel industry

Authors:

Valeriy VyatkinAuthors Info & Claims

Volume 143, Issue C

https://doi.org/10.1016/j.compind.2022.103748

Published: 01 December 2022 Publication History

Abstract

Strip rolling is a typical manufacturing process, in which conventional control approaches are widely applied. Development of the control algorithms requires a mathematical expression of the process by means of the ﬁrst principles or empirical models. However, it is difficult to upgrade the conventional control approaches in response to the ever-changing requirements and environmental conditions because domain knowledge of control engineering, mechanical engineering, and material science is required. Reinforcement learning is a machine learning method that can make the agent learn from interacting with the environment, thus avoiding the need for the above mentioned mathematical expression. This paper proposes a novel approach that combines ensemble learning with reinforcement learning methods for strip rolling control. Based on the proximal policy optimization (PPO), a multi-actor PPO is proposed. Each randomly initialized actor interacts with the environment in parallel, but only the experience from the actor that obtains the highest reward is used for updating the actors. Simulation results show that the proposed method outperforms the conventional control methods and the state-of-the-art reinforcement learning methods in terms of process capability and smoothness.

Highlights

•

A new learning controller is developed for an industrial control system.

•

An ensemble learning based reinforcement learning method is proposed.

•

A real industrial strip rolling process is studied for the new controller.

•

Simulation results are evaluated by process capability and smoothness.

References

[1]

Yaoyao Bao, Yuanming Zhu, Feng Qian, A deep reinforcement learning approach to improve the learning performance in process control, Ind. Eng. Chem. Res. 60 (15) (2021) 5504–5515,.

[2]

Alberto Bemporad, Daniele Bernardini, Francesco Alessandro Cuzzola, Andrea Spinelli, Optimization-based automatic flatness control in cold tandem rolling, J. Process Control 20 (4) (2010) 396–407,.

[3]

George E.P. Box, Gwilym M. Jenkins, Gregory C. Reinsel, Greta M. Ljung, Time Series Analysis: Forecasting and Control, John Wiley & Sons, 2015.

Digital Library

[4]

Chunyu Chen, Mingjian Cui, Fangxing Li, Shengfei Yin, Xinan Wang, Model-free emergency frequency control based on reinforcement learning, IEEE Trans. Ind. Inform. 17 (4) (2021) 2336–2346,.

[5]

Jifei Deng, Jie Sun, Wen Peng, Yaohui Hu, Dianhua Zhang, Application of neural networks for predicting hot-rolled strip crown, Appl. Soft Comput. J. 78 (2019) 119–131,.

Digital Library

[6]

Jiajun Duan, Di Shi, Ruisheng Diao, Haifeng Li, Zhiwei Wang, Bei Zhang, Desong Bian, Zhehan Yi, Deep-reinforcement-learning-based autonomous voltage control for power grid operations, IEEE Trans. Power Syst. 35 (1) (2020) 814–817,.

[7]

Fan, Haoren, Zhu, Lei, Yao, Changhua, Guo, Jibin, Lu, Xiaowen, 2019. Deep reinforcement learning for energy efficiency optimization in wireless networks. In: Proceedings of the 2019 IEEE 4th International Conference on Cloud Computing and Big Data Analytics, ICCCBDA 2019, April. Institute of Electrical and Electronics Engineers Inc., pp. 465–71. 〈https://doi.org/10.1109/ICCCBDA.2019.8725683〉.

[8]

Fujimoto, Scott, Hoof, Herke Van, Meger, David, 2018. Addressing function approximation error in actor-critic methods. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, 4, pp. 2587–601.

[9]

Omar Gamal, Mohamed Imran Peer Mohamed, Chirag Ghanshyambhai Patel, Hubert Roth, Data-driven model-free intelligent roll gap control of bar and wire hot rolling process using reinforcement learning, Int. J. Mech. Eng. Robot. Res. 10 (7) (2021) 349–356,.

[10]

Vladimir B. Ginzburg (Ed.), Flat-Rolled Steel Processes: Advanced Technologies, CRC Press, 2009.

[11]

Fang Guo, Yongqiang Li, Ao Liu, Zhan Liu, A reinforcement learning method to scheduling problem of steel production process, J. Phys. Conf. Ser. 1486 (7) (2020),.

[12]

Haarnoja, Tuomas, Zhou, Aurick, Abbeel, Pieter, Levine, Sergey, 2018. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, 5, pp. 2976–89.

[13]

Zhongyang Han, Witold Pedrycz, Jun Zhao, Wei Wang, Hierarchical granular computing-based model and its reinforcement structural learning for construction of long-term prediction intervals, IEEE Trans. Cybern. 52 (1) (2020) 666–676,.

[14]

Zhenglei He, Kim Phuc Tran, Sebastien Thomassey, Xianyi Zeng, Jie Xu, Changhai Yi, A deep reinforcement learning based multi-criteria decision support system for optimizing textile chemical process, Comput. Ind. 125 (February) (2021),.

[15]

Xin Jin, Changsheng Li, Yu Wang, Xiaogang Li, Yongguang Xiang, Tian Gu, Investigation and optimization of load distribution for tandem cold steel strip rolling process, Metals 10 (5) (2020) 677,.

[16]

Lillicrap, Timothy P., Hunt, Jonathan J., Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, Wierstra, Daan, 2016. Continuous control with deep reinforcement learning. In: Proceedings of the International Conference on Learning Representations (ICLR).

[17]

Chao Liu, Jinliang Ding, Jiyuan Sun, Reinforcement learning based decision making of operational indices in process industry under changing environment, IEEE Trans. Ind. Inform. 17 (4) (2021) 2727–2736,.

[18]

Yan Jun Liu, Li Tang, Shaocheng Tong, C.L. Philip Chen, Dong Juan Li, Reinforcement learning design-based adaptive tracking control with less learning parameters for nonlinear discrete-time MIMO systems, IEEE Trans. Neural Netw. Learn. Syst. 26 (1) (2015) 165–176,.

[19]

Guangbiao Liu, Jianzhong Zhou, Benjun Jia, Feifei He, Yuqi Yang, Na Sun, Advance short-term wind energy quality assessment based on instantaneous standard deviation and variogram of wind speed by a hybrid method, Applied Energy 238 (March) (2019) 643–667,.

[20]

N. Mathieu, M. Potier-Ferry, H. Zahrouni, Reduction of flatness defects in thin metal sheets by a pure tension leveler, Int. J. Mech. Sci. 122 (March) (2017) 267–276,.

[21]

Takao Moriyama, Giovanni De Magistris, Michiaki Tatsubori, Tu. Hoa Pham, Asim Munawar, Ryuki Tachibana, Reinforcement learning testbed for power-consumption optimization, Commun. Comput. Inform. Sci. 946 (October) (2018) 45–59,.

[22]

Jena L. Nawfel, Kevin B. Englehart, Erik J. Scheme, A multi-variate approach to predicting myoelectric control usability, IEEE Trans. Neural Syst. Rehab. Eng. 29 (2021) 1312–1327,.

[23]

Rui Nian, Jinfeng Liu, Biao Huang, A review on reinforcement learning: introduction and applications in industrial process control, Comput. Chem. Eng. 139 (2020),.

[24]

Zhaolong Ning, Kaiyuan Zhang, Xiaojie Wang, Mohammad S. Obaidat, Lei Guo, Xiping Hu, Bin Hu, Yi Guo, Balqies Sadoun, Ricky Y.K. Kwok, Joint computing and caching in 5G-envisioned internet of vehicles: a deep reinforcement learning-based traffic control system, IEEE Trans. Intell. Transp. Syst. 22 (8) (2021) 5201–5212,.

Digital Library

[25]

Jussi Paakkari, On-Line Flatness Measurement of Large Steel Plates Using Moiré Topography, University of Oulu, 1998.

[26]

Peihua Qiu, Introduction to Statistical Process Control, CRC Press, 2013.

[27]

Schulman, John, Moritz, Philipp, Levine, Sergey, Jordan, Michael I., Abbeel, Pieter, 2016. High-dimensional continuous control using generalized advantage estimation. In: Proceedings of the International Conference on Learning Representations (ICLR), pp. 1–14.

[28]

Schulman, John, Levine, Sergey, Abbeel, Pieter, Jordan, Michael, Moritz, Philipp, 2015. Trust region policy optimization. In: Proceedings of the International Conference on Machine Learning (ICML).

[29]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, Proximal policy optimization algorithms, ArXiv (2017) 1–12.

[30]

Joohyun Shin, Thomas A. Badgwell, Kuang Hung Liu, Jay H. Lee, Reinforcement learning – overview of recent progress and implications for process control, Comput. Chem. Eng. 127 (August) (2019) 282–294,.

[31]

Spielberg, S.P.K., Gopaluni, R.B., Loewen, P.D., 2017. Deep reinforcement learning approaches for process control. In: Proceedings of the 2017 6th International Symposium on Advanced Control of Industrial Processes, AdCONIP 2017. Institute of Electrical and Electronics Engineers Inc, pp. 201–6. 〈https://doi.org/10.1109/ADCONIP.2017.7983780〉.

[32]

Niclas Ståhl, Gunnar Mathiason, Dellainey Alcacoas, Using reinforcement learning for generating polynomial models to explain complex data, SN Comput. Sci. 2 (2) (2021) 1–11,.

Digital Library

[33]

Jie Sun, Jifei Deng, Wen Peng, Dianhua Zhang, Strip crown prediction in hot rolling process using random forest, Int. J. Precis. Eng. Manuf. (0123456789) (2021),.

[34]

R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction, MIT press, 2018.

Digital Library

[35]

Nathalie Vanvuchelen, Joren Gijsbrechts, Robert Boute, Use of proximal policy optimization for the joint replenishment problem, Comput. Ind. 119 (August) (2020),.

[36]

Pengfei Wang, Shuren Jin, Xu Li, Huagui Huang, Haifeng Wang, Dianhua Zhang, Wentian Li, Yulin Yao, Optimization and prediction model of flatness actuator efficiency in cold rolling process based on process data, Steel Res. Int. 93 (1) (2022),.

[37]

Xiaocen Wang, Min Lin, Junkai Tong, Lin Liang, Jian Li, Zhoumo Zeng, Yang Liu, Guided wave imaging based on fully connected neural network for quantitative corrosion assessment, in: Annual Review of Progress in Quantitative Nondestructive Evaluation, American Society of Mechanical Engineers Digital Collection, 2022,.

[38]

Zhao Zhang, Shuxin Liu, Manhua Liu, A multi-task fully deep convolutional neural network for contactless fingerprint minutiae extraction, Pattern Recognit. 120 (December) (2021),.

Digital Library

[39]

Heng Zhang, Qingjin Peng, Jian Zhang, Peihua Gu, Planning for automatic product assembly using reinforcement learning, Comput. Ind. 130 (September) (2021),.

Cited By

Zaoui ATchuente DWamba SKamsu-Foguem B(2024)Impact of artificial intelligence on aeronauticsJournal of Engineering and Technology Management10.1016/j.jengtecman.2024.10180071:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.jengtecman.2024.101800
Wang QSun JHu YJiang WZhang XWang Z(2024)Deep learning-based flatness prediction via multivariate industrial data for steel strip during tandem cold rollingExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121777237:PCOnline publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121777
Peng WLei JDing CYue CMa GSun JZhang D(2024)A novel deep ensemble reinforcement learning based control method for strip flatness in cold rolling steel industryEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108695134:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.108695
Show More Cited By

Index Terms

Reinforcement learning for industrial process control: A case study in flatness control in steel industry

Index terms have been assigned to the content through auto-classification.

Recommendations

Offline reinforcement learning for industrial process control: A case study from steel industry
Abstract
Flatness is a crucial indicator of strip quality that presents a challenge in regulation due to the high-speed process and the nonlinear relationship between flatness and process parameters. Conventional methods for controlling ...
Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems

Recent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...
Relational Reinforcement Learning

Relational reinforcement learning is presented, a learning technique that combines reinforcement learning with relational learning or inductive logic programming. Due to the use of a more expressive representation language to represent states, actions ...

Comments

Information & Contributors

Information

Published In

cover image Computers in Industry

Computers in Industry Volume 143, Issue C

Dec 2022

174 pages

ISSN:0166-3615

Issue’s Table of Contents

The Authors.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 December 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zaoui ATchuente DWamba SKamsu-Foguem B(2024)Impact of artificial intelligence on aeronauticsJournal of Engineering and Technology Management10.1016/j.jengtecman.2024.10180071:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.jengtecman.2024.101800
Wang QSun JHu YJiang WZhang XWang Z(2024)Deep learning-based flatness prediction via multivariate industrial data for steel strip during tandem cold rollingExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121777237:PCOnline publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121777
Peng WLei JDing CYue CMa GSun JZhang D(2024)A novel deep ensemble reinforcement learning based control method for strip flatness in cold rolling steel industryEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108695134:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.108695
Tremblay FDurand AMorin MMarier PGaudreault J(2024)Deep reinforcement learning for continuous wood drying production line controlComputers in Industry10.1016/j.compind.2023.104036154:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.compind.2023.104036
Jin XMi NSong WLi Q(2024)Scheduling of twin automated stacking cranes based on Deep Reinforcement LearningComputers and Industrial Engineering10.1016/j.cie.2024.110104191:COnline publication date: 18-Jul-2024
https://dl.acm.org/doi/10.1016/j.cie.2024.110104
Musial JStebel KCzeczot JNowak PGabrys B(2024)Application of self-improving Q-learning controller for a class of dynamical processesApplied Soft Computing10.1016/j.asoc.2024.111250152:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.asoc.2024.111250
Zhang HWang WZhang SZhang YZhou JWang ZHuang BHuang R(2024)A novel method for intersecting machining feature segmentation via deep reinforcement learningAdvanced Engineering Informatics10.1016/j.aei.2023.10225659:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.aei.2023.102256
Ma ZPan T(2024)DRL-dEWMA: a composite framework for run-to-run control in the semiconductor manufacturing processNeural Computing and Applications10.1007/s00521-023-09112-936:3(1429-1447)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s00521-023-09112-9
Deng JSierla SSun JVyatkin V(2023)Offline reinforcement learning for industrial process controlInformation Sciences: an International Journal10.1016/j.ins.2023.03.019632:C(221-231)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1016/j.ins.2023.03.019
Papavasileiou PKoronaki EPozzetti GKathrein MCzettl CBoudouvis ABordas S(2023)Equation-based and data-driven modeling strategies for industrial coating processesComputers in Industry10.1016/j.compind.2023.103938149:COnline publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.compind.2023.103938
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents