Abstract
Machine Learning (ML) techniques and algorithms, which are emerging technologies in Industry 4.0, present new possibilities for complex scheduling methods. Since different rules can be applied to different circumstances, it can be difficult for the decision-maker to choose the right rule at any given time. The purpose of the paper is to build an “intelligent” tool that adapts its choices in response to changes in the state of the production line. A Deep Q-Network (DQN), a typical Deep Reinforcement Learning (DRL) method, is proposed for creating a self-optimizing scheduling policy. The system has a set of known dispatching rules for each machine’s queue, from which the best one is dynamically chosen, according to the system state. The novelty of the paper is how the reward function, state, and action space are modelled. A series of experiments were conducted to determine the best DQN network size and the most influential hyperparameters for training.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bendul, J.C., Blunck, H.: The design space of production planning and control for industry 4.0. Computers in Industry 105, 260–272 (2019). https://doi.org/10.1016/j.compind.2018.10.010
Chen, S., Fang, S., Tang, R.: A reinforcement learning based approach for multi-projects scheduling in cloud manufacturing. Int. J. Prod Res. 57(10), 3080–3098 (2019). https://doi.org/10.1080/00207543.2018.1535205
Dolgui, A., Ivanov, D., Sethi, S.P., Sokolov, B.: Scheduling in production, supply chain and Industry 4.0 systems by optimal control: fundamentals, state-of-the-art and applications. Int. J. Prod. Res. 57(2), 411–432 (2019). https://doi.org/10.1080/00207543.2018.1442948
Grassi, A., Guizzi, G., Santillo, L.C., Vespoli, S.: Assessing the performances of a novel decentralised scheduling approach in Industry 4.0 and cloud manufacturing contexts. Int. J. Prod. Res. 1–20 (2020). https://doi.org/10.1080/00207543.2020.1799105
Heger, J., Branke, J., Hildebrandt, T., Scholz-Reiter, B.: Dynamic adjustment of dispatching rule parameters in flow shops with sequence-dependent set-up times. Int. J. Prod. Res. 54(22), 6812–6824 (2016). https://doi.org/10.1080/00207543.2016.1178406
Hopp, W.J., Spearman, M.L.: Factory Physics: foundation of manufacturing management. Irwin/McGraw-Hill, second edition edn. (2001)
Ivanov, D., Sokolov, B., Chen, W., Dolgui, A., Werner, F., Potryasaev, S.: A control approach to scheduling flexibly configurable jobs with dynamic structural-logical constraints. IISE Trans. 53(1), 21–38 (2021). https://doi.org/10.1080/24725854.2020.1739787
Kim, H., Lim, D.E., Lee, S.: Deep learning-based dynamic scheduling for semiconductor manufacturing with high uncertainty of automated material handling system capability. IEEE Trans. Semicond. Manuf. 33(1), 13–22 (2020). https://doi.org/10.1109/TSM.2020.2965293
Kim, Y.G., Lee, S., Son, J., Bae, H., Chung, B.D.: Multi-agent system and reinforcement learning approach for distributed intelligence in a flexible smart manufacturing system. J. Manuf. Syst. 57(August 2019), 440–450 (2020). https://doi.org/10.1016/j.jmsy.2020.11.004
Kusiak, A.: Convolutional and generative adversarial neural networks in manufacturing. Int. J. Prod. Res. 58(5), 1594–1604 (2020). https://doi.org/10.1080/00207543.2019.1662133
Lee, J.H., Kim, H.J., Lee, J.H.: Reinforcement learning for robotic flow shop scheduling with processing time variations variations. Int. J. Prod. Res. 1–23 (2021). https://doi.org/10.1080/00207543.2021.1887533
Lin, C.C., Deng, D.J., Chih, Y.L., Chiu, H.T.: Smart manufacturing scheduling with edge computing using multiclass deep Q network. IEEE Trans. Industr. Inf. 15(7), 4276–4284 (2019). https://doi.org/10.1109/TII.2019.2908210
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). https://doi.org/10.1038/nature14236
Mouelhi-Chibani, W., Pierreval, H.: Training a neural network to select dispatching rules in real time. Comput. Industr. Eng. 58(2), 249–256 (2010). https://doi.org/10.1016/j.cie.2009.03.008
Oukil, A., El-Bouri, A.: Ranking dispatching rules in multi-objective dynamic flow shop scheduling: a multi-faceted perspective. Int. J. Prod. Res. 59(2), 388–411 (2021). https://doi.org/10.1080/00207543.2019.1696487
Park, I.B., Huh, J., Kim, J., Park, J.: A reinforcement learning approach to robust scheduling of semiconductor manufacturing facilities. IEEE Trans. Autom. Sci. Eng. 17(3), 1420–1431 (2020). https://doi.org/10.1109/TASE.2019.2956762
Park, J., Chun, J., Kim, S.H., Kim, Y., Park, J.: Learning to schedule job-shop problems: representation and policy learning using graph neural network and reinforcement learning. Int. J. Prod. Res. 1–18 (2021). https://doi.org/10.1080/00207543.2020.1870013
Patterson, J., Gibson, A.: Deep Learning: A Practitioner’s Approach. 1st edit edn, O’Reilly, Sebastopol (2017)
Priore, P., Ponte, B., Puente, J., Gómez, A.: Learning-based scheduling of flexible manufacturing systems using ensemble methods. Comput. Industr. Eng. 126(September), 282–291 (2018). https://doi.org/10.1016/j.cie.2018.09.034
Shi, D., Fan, W., Xiao, Y., Lin, T., Xing, C.: Intelligent scheduling of discrete automated production line via deep reinforcement learning. Int. J. Prod. Res. 58(11), 3362–3380 (2020). https://doi.org/10.1080/00207543.2020.1717008
Xia, K., et al.: A digital twin to train deep reinforcement learning agent for smart manufacturing plants: Environment, interfaces and intelligence. J. Manuf. Syst. 1–21 (2020). https://doi.org/10.1016/j.jmsy.2020.06.012
Acknowledgement
The authors would like Tecnologica S.r.l. for granting the research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 IFIP International Federation for Information Processing
About this paper
Cite this paper
Marchesano, M.G., Guizzi, G., Santillo, L.C., Vespoli, S. (2021). Dynamic Scheduling in a Flow Shop Using Deep Reinforcement Learning. In: Dolgui, A., Bernard, A., Lemoine, D., von Cieminski, G., Romero, D. (eds) Advances in Production Management Systems. Artificial Intelligence for Sustainable and Resilient Production Systems. APMS 2021. IFIP Advances in Information and Communication Technology, vol 630. Springer, Cham. https://doi.org/10.1007/978-3-030-85874-2_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-85874-2_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-85873-5
Online ISBN: 978-3-030-85874-2
eBook Packages: Computer ScienceComputer Science (R0)
