Abstract
In the era of Industry 4.0, production scheduling as a critical part of manufacturing system should be smarter. Smart scheduling agent is required to be real-time autonomous and possess the ability to face unforeseen and disruptive events. However, traditional methods lack adaptability and intelligence. Hence, this paper is devoted to proposing a smart approach based on proximal policy optimization (PPO) to solve dynamic job shop scheduling problem with random job arrivals. The PPO scheduling agent is trained based on an integration framework of discrete event simulation and deep reinforcement learning. Copies of trained agent can be linked with each machine for distributed control. Meanwhile, state features, actions and rewards are designed for scheduling at each decision point. Reward scaling are applied to improve the convergence performance. The numerical experiments are conducted on cases with different production configurations. The results show that PPO method can realize on-line decision making and provide better solution than dispatch rules and heuristics. It can achieve a balance between time and quality. Moreover, the trained model could also maintain certain performance even in untrained scenarios.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data that support the findings of this study are available from the corresponding author, upon reasonable request.
Change history
04 August 2023
A Correction to this paper has been published: https://doi.org/10.1007/s10845-023-02189-y
References
Akarsu, C. H., & Küçükdeniz, T. (2022). Job shop scheduling with genetic algorithm-based hyperheuristic approach. International Advanced Researches and Engineering Journal, 6(1), 16–25.
Aydin, M. E., & Oztemel, E. (2000). Dynamic job-shop scheduling using reinforcement learning agents. Robotics and Autonomous Systems, 33(2–3), 169–178. https://doi.org/10.1016/S0921-8890(00)00087-7
Blackstone, J. H., Phillips, D. T., & Hogg, G. L. (1982). A state-of-the-art survey of dispatching rules for manufacturing job shop operations. International Journal of Production Research, 20(1), 27–45. https://doi.org/10.1080/00207548208947745
Bouazza, W., Sallez, Y., & Beldjilali, B. (2017). A distributed approach solving partially flexible job-shop scheduling problem with a Q-learning effect. IFAC Papersonline, 50(1), 15890–15895. https://doi.org/10.1016/j.ifacol.2017.08.2354
Chang, J. R., Yu, D., Hu, Y., He, W. W., & Yu, H. Y. (2022). Deep reinforcement learning for dynamic flexible job shop scheduling with random job arrival. Processes. https://doi.org/10.3390/pr10040760
Duan, J. G., & Wang, J. H. (2022). Robust scheduling for flexible machining job shop subject to machine breakdowns and new job arrivals considering system reusability and task recurrence. Expert Systems with Applications. https://doi.org/10.1016/j.eswa.2022.117489
Gonzalez, T., & Sahni, S. (1978). Flowshop and jobshop schedules: complexity and approximation. Operations Research, 26(1), 36–52. https://doi.org/10.1287/opre.26.1.36
Han, B. A., & Yang, J. J. (2020). Research on adaptive job shop scheduling problems based on dueling double DQN. IEEE Access, 8, 186474–186495. https://doi.org/10.1109/Access.2020.3029868
Heess, N., Dhruva, T. B., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Ali Eslami, S. M., Riedmiller, M., Silver, D. (2017). Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286.
Hermann, M., Pentek, T., & Otto, B. (2016). Design Principles for Industrie 4.0 Scenarios. Proceedings of the 49th Annual Hawaii International Conference on System Sciences (Hicss 2016), 3928–3937. https://doi.org/10.1109/Hicss.2016.488
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018). Deep Reinforcement Learning that Matters. Thirty-Second Aaai Conference on Artificial Intelligence / Thirtieth Innovative Applications of Artificial Intelligence Conference /Eighth Aaai Symposium on Educational Advances in Artificial Intelligence 3207–3214. Retrieved from: //WOS:000485488903036
Hunt, R., Johnston, M., & Zhang, M. (2014, 6–11 July 2014). Evolving machine-specific dispatching rules for a two-machine job shop using genetic programming. Paper presented at the 2014 IEEE Congress on Evolutionary Computation (CEC).
Jing, X., Yao, X. F., Liu, M., & Zhou, J. J. (2022). Multi-agent reinforcement learning based on graph convolutional network for flexible job shop scheduling. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-022-02037-5
Kuhnle, A., Kaiser, J.-P., Theiß, F., Stricker, N., & Lanza, G. (2021). Designing an adaptive production control system using reinforcement learning. Journal of Intelligent Manufacturing, 32(3), 855–876. https://doi.org/10.1007/s10845-020-01612-y
Lang, S., Behrendt, F., Lanzerath, N., Reggelin, T., & Müller, M. (2020, 14–18 Dec. 2020). Integration of Deep Reinforcement Learning and Discrete-Event Simulation for Real-Time Scheduling of a Flexible Job Shop Production. Paper presented at the 2020 Winter Simulation Conference (WSC).
Lei, K., Guo, P., Zhao, W., Wang, Y., Qian, L., Meng, X., & Tang, L. (2022). A multi-action deep reinforcement learning framework for flexible Job-shop scheduling problem. Expert Systems with Applications, 205, 117796. https://doi.org/10.1016/j.eswa.2022.117796
Li, Y. X., Gu, W. N., Yuan, M. H., & Tang, Y. M. (2022). Real-time data-driven dynamic scheduling for flexible job shop with insufficient transportation resources using hybrid deep Q network. Robotics and Computer-Integrated Manufacturing. https://doi.org/10.1016/j.rcim.2021.102283
Liu, R. K., Piplani, R., & Toro, C. (2022). Deep reinforcement learning for dynamic scheduling of a flexible job shop. International Journal of Production Research, 60(13), 4049–4069. https://doi.org/10.1080/00207543.2022.2058432
Luo, S. (2020). Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning. Applied Soft Computing. https://doi.org/10.1016/j.asoc.2020.106208
Lv, Y., Li, C., Tang, Y., & Kou, Y. (2022). Toward energy-efficient rescheduling decision mechanisms for flexible job shop with dynamic events and alternative process plans. IEEE Transactions on Automation Science and Engineering, 19(4), 3259–3275. https://doi.org/10.1109/TASE.2021.3115821
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533. https://doi.org/10.1038/nature14236
Namjoshi, J., & Rawat, M. (2022). Role of smart manufacturing in industry 4.0. Materials Today: Proceedings, 63, 475–478. https://doi.org/10.1016/j.matpr.2022.03.620
Oh, S. H., Cho, Y. I., & Woo, J. H. (2022). Distributional reinforcement learning with the independent learners for flexible job shop scheduling problem with high variability. Journal of Computational Design and Engineering, 9(4), 1157–1174. https://doi.org/10.1093/jcde/qwac044
Ouelhadj, D., & Petrovic, S. (2008). A survey of dynamic scheduling in manufacturing systems. Journal of Scheduling, 12(4), 417–431.
Qi, Q. L., & Tao, F. (2018). Digital twin and big data towards smart manufacturing and Industry 4.0: 360 degree comparison. IEEE Access, 6, 3585–3593. https://doi.org/10.1109/Access.2018.2793265
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-dimensional continuous control using generalized advantage estimation. arXiv:1506.02438. Retrieved from https://ui.adsabs.harvard.edu/abs/2015arXiv150602438S
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv:1707.06347. Retrieved from https://ui.adsabs.harvard.edu/abs/2017arXiv170706347S
Serrano-Ruiz, J. C., Mula, J., & Poler, R. (2022). Development of a multidimensional conceptual model for job shop smart manufacturing scheduling from the Industry 4.0 perspective. Journal of Manufacturing Systems, 63, 185–202. https://doi.org/10.1016/j.jmsy.2022.03.011
Shahrabi, J., Adibi, M. A., & Mahootchi, M. (2017). A reinforcement learning approach to parameter estimation in dynamic job shop scheduling. Computers & Industrial Engineering, 110, 75–82. https://doi.org/10.1016/j.cie.2017.05.026
Souza, R. L. C., Ghasemi, A., Saif, A., & Gharaei, A. (2022). Robust job-shop scheduling under deterministic and stochastic unavailability constraints due to preventive and corrective maintenance. Computers & Industrial Engineering. https://doi.org/10.1016/j.cie.2022.108130
Sutton, R. S., McAllester, D., Singh, S., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12(12), 1057–1063.
Tao, F., Cheng, Y., Zhang, L., & Nee, A. Y. C. (2017). Advanced manufacturing systems: Socialization characteristics and trends. Journal of Intelligent Manufacturing, 28(5), 1079–1094. https://doi.org/10.1007/s10845-015-1042-8
Wang, X. H., Zhang, L., Lin, T. Y., Zhao, C., Wang, K. Y., & Chen, Z. (2022). Solving job scheduling problems in a resource preemption environment with multi-agent reinforcement learning. Robotics and Computer-Integrated Manufacturing. https://doi.org/10.1016/j.rcim.2022.102324
Wang, Y.-F. (2020). Adaptive job shop scheduling strategy based on weighted Q-learning algorithm. Journal of Intelligent Manufacturing, 31(2), 417–432. https://doi.org/10.1007/s10845-018-1454-3
Yan, Q., Wang, H. F., & Wu, F. (2022). Digital twin-enabled dynamic scheduling with preventive maintenance using a double-layer Q-learning algorithm. Computers & Operations Research. https://doi.org/10.1016/j.cor.2022.105823
Yang, S. L., & Xu, Z. G. (2022). Intelligent scheduling and reconfiguration via deep reinforcement learning in smart manufacturing. International Journal of Production Research, 60(16), 4936–4953. https://doi.org/10.1080/00207543.2021.1943037
Zhang, C., Song, W., Cao, Z., Zhang, J., Siew Tan, P., & Xu, C. (2020). Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement Learning. arXiv:2010.12367. Retrieved from https://ui.adsabs.harvard.edu/abs/2020arXiv201012367Z
Zhang, L. X., Yang, C., Yan, Y., & Hu, Y. G. (2022a). Distributed real-time scheduling in cloud manufacturing by deep reinforcement learning. IEEE Transactions on Industrial Informatics, 18(12), 8999–9007. https://doi.org/10.1109/Tii.2022.3178410
Zhang, Y., Zhu, H., Tang, D., Zhou, T., & Gui, Y. (2022b). Dynamic job shop scheduling based on deep reinforcement learning for multi-agent manufacturing systems. Robotics and Computer-Integrated Manufacturing, 78, 102412. https://doi.org/10.1016/j.rcim.2022.102412
Acknowledgements
This work is funded by National key Research and development program of China (Grant No. 2020YFB1709800), Project No. 2022CDJSKJC20 supported by Fundamental Research Funds for Central Universities and project of science and technology research program of Chongqing Education Commission of China (Grant No. KJQN201900107).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The original online version of this article was revised: The citations Zhang et al. 2022a and Zhang et al. 2022b were missing in the text and the citations are now included in the text.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, Z., Liao, W. Smart scheduling of dynamic job shop based on discrete event simulation and deep reinforcement learning. J Intell Manuf 35, 2593–2610 (2024). https://doi.org/10.1007/s10845-023-02161-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-023-02161-w