Abstract
Modern neural networks require long training to reach decent performance on massive datasets. One common approach to speed up training is model parallelization, where large neural networks are split across multiple devices. However, different device placements of the same neural network lead to different training times. Most of the existing device placement solutions treat the problem as sequential decision-making by traversing neural network graphs and assigning their neurons to different devices. This work studies the impact of neural network graph traversal orders on device placement. In particular, we empirically study how different graph traversal orders of neural networks lead to different device placements, which in turn affects the training time of the neural network. Our experiment results show that the best graph traversal order depends on the type of neural networks and their computation graphs features. In this work, we also provide recommendations on choosing effective graph traversal orders in device placement for various neural network families to improve the training time in model parallelization.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Addanki, R., Bojja Venkatakrishnan, S., Gupta, S., Mao, H., Alizadeh, M.: Placeto: Learning generalizable device placement algorithms for distributed machine learning. Advances in Neural Information Processing Systems 32 (NIPS 2019) (2019)
Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)
Gao, Y., Chen, L., Li, B.: Post: Device placement with cross-entropy minimization and proximal policy optimization. In: Advances in Neural Information Processing Systems. pp. 9971–9980 (2018)
Gao, Y., Chen, L., Li, B.: Spotlight: Optimizing device placement for training deep neural networks. In: International Conference on Machine Learning. pp. 1676–1684 (2018)
Hagberg, A., Swart, P., S Chult, D.: Exploring network structure, dynamics, and function using networkx. Tech. rep., Los Alamos National Lab. (LANL), Los Alamos, NM (United States) (2008)
Hagos, D.H., Kakantousis, T., Vlassov, V., Sheikholeslami, S., Wang, T., Dowling, J., Paris, C., Marinelli, D., Weikmann, G., Bruzzone, L., et al.: Extremeearth meets satellite data from space. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 14, 9038–9063 (2021)
Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. pp. 1025–1035 (2017)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
Kahn, A.B.: Topological sorting of large networks. Communications of the ACM 5(11), 558–562 (1962)
Khaleghian, S., Kramer, T., Everett, A., Kiarbech, A., Hughes, N., Eltoft, T., Marinoni, A.: Synthetic aperture radar data analysis by deep learning for automatic sea ice classification. In: EUSAR 2021; 13th European Conference on Synthetic Aperture Radar. pp. 1–6. VDE (2021)
Khaleghian, S., Ullah, H., Kræmer, T., Hughes, N., Eltoft, T., Marinoni, A.: Sea ice classification of sar imagery based on convolution neural networks. Remote Sensing 13(9), 1734 (2021)
Koubarakis, M., Bereta, K., Bilidas, D., Giannousis, K., Ioannidis, T., Pantazi, D.A., Stamoulis, G., Haridi, S., Vlassov, V., Bruzzone, L., et al.: From copernicus big data to extreme earth analytics. Open Proceedings pp. 690–693 (2019)
Koubarakis, M., Stamoulis, G., Bilidas, D., Ioannidis, T., Mandilaras, G., Pantazi, D.A., Papadakis, G., Vlassov, V., Payberah, A.H., Wang, T., et al.: Artificial intelligence and big data technologies for copernicus data: The extremeearth project. In: Proceedings of the 2021 conference on Big Data from Space. Publications Office of the European Union (2021)
Kreuzer, D., Beaini, D., Hamilton, W.L., Létourneau, V., Tossou, P.: Rethinking graph transformers with spectral attention. arXiv preprint arXiv:2106.03893 (2021)
Lan, H., Chen, L., Li, B.: Accelerated device placement optimization with contrastive learning. In: 50th International Conference on Parallel Processing. pp. 1–10 (2021)
Lan, H., Chen, L., Li, B.: Eagle: Expedited device placement with automatic grouping for large models. In: 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS). pp. 599–608 (2021). DOI: https://doi.org/10.1109/IPDPS49936.2021.00068
Mayer, R., Mayer, C., Laich, L.: The tensorflow partitioning and scheduling problem: it’s the critical path! In: Proceedings of the 1st Workshop on Distributed Infrastructures for Deep Learning. pp. 1–6 (2017)
Meister, M., Sheikholeslami, S., Payberah, A.H., Vlassov, V., Dowling, J.: Maggy: Scalable asynchronous parallel hyperparameter search. In: Proceedings of the 1st Workshop on Distributed Machine Learning. pp. 28–33 (2020)
Mirhoseini, A., Goldie, A., Pham, H., Steiner, B., Le, Q.V., Dean, J.: A hierarchical model for device placement. In: International Conference on Learning Representations (2018)
Mirhoseini, A., Goldie, A., Yazgan, M., Jiang, J.W., Songhori, E., Wang, S., Lee, Y.J., Johnson, E., Pathak, O., Nazi, A., et al.: A graph placement methodology for fast chip design. Nature 594(7862), 207–212 (2021)
Mirhoseini, A., Pham, H., Le, Q.V., Steiner, B., Larsen, R., Zhou, Y., Kumar, N., Norouzi, M., Bengio, S., Dean, J.: Device placement optimization with reinforcement learning. In: International Conference on Machine Learning. pp. 2430–2439. PMLR (2017)
Mitropolitsky, M., Abbas, Z., Payberah, A.H.: Graph representation matters in device placement. In: Proceedings of the Workshop on Distributed Infrastructures for Deep Learning. pp. 1–6 (2020)
Paliwal, A., Gimeno, F., Nair, V., Li, Y., Lubin, M., Kohli, P., Vinyals, O.: Reinforced genetic algorithm learning for optimizing computation graphs. In: International Conference on Learning Representations (2020)
Paris, C., Weikmann, G., Bruzzone, L.: Monitoring of agricultural areas by using sentinel 2 image time series and deep learning techniques. In: Image and Signal Processing for Remote Sensing XXVI. vol. 11533, p. 115330K. International Society for Optics and Photonics (2020)
Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: International Conference on Machine Learning. pp. 4095–4104. PMLR (2018)
Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE transactions on neural networks 20(1), 61–80 (2008)
Shallue, C.J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., Dahl, G.E.: Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research 20(112), 1–49 (2019)
Sheikholeslami, S., Meister, M., Wang, T., Payberah, A.H., Vlassov, V., Dowling, J.: Autoablation: Automated parallel ablation studies for deep learning. In: Proceedings of the 1st Workshop on Machine Learning and Systems. pp. 55–61 (2021)
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., Catanzaro, B.: Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1–9 (2015)
Tanaka, M., Taura, K., Hanawa, T., Torisawa, K.: Automatic graph partitioning for very large-scale deep learning. arXiv preprint arXiv:2103.16063 (2021)
Topping, J., Di Giovanni, F., Chamberlain, B.P., Dong, X., Bronstein, M.M.: Understanding over-squashing and bottlenecks on graphs via curvature. arXiv preprint arXiv:2111.14522 (2021)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017)
Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3), 229–256 (1992)
Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., Liu, T.Y.: Do transformers really perform bad for graph representation? arXiv preprint arXiv:2106.05234 (2021)
You, J., Ying, R., Ren, X., Hamilton, W., Leskovec, J.: Graphrnn: Generating realistic graphs with deep auto-regressive models. In: International conference on machine learning. pp. 5708–5717. PMLR (2018)
Zhou, Y., Roy, S., Abdolrashidi, A., Wong, D., Ma, P.C., Xu, Q., Zhong, M., Liu, H., Goldie, A., Mirhoseini, A., et al.: Gdp: Generalized device placement for dataflow graphs. arXiv preprint arXiv:1910.01578 (2019)
Zhou, Y., Roy, S., Abdolrashidi, A., Wong, D.L.K., Ma, P., Xu, Q., Mirhoseini, A., Laudon, J.: A single-shot generalized device placement for large dataflow graphs. IEEE Micro 40(5), 26–36 (2020)
Zhu, X.X., Tuia, D., Mou, L., Xia, G.S., Zhang, L., Xu, F., Fraundorfer, F.: Deep learning in remote sensing: A comprehensive review and list of resources. IEEE Geoscience and Remote Sensing Magazine 5(4), 8–36 (2017)
Acknowledgements
This work was supported by the ExtremeEarth project funded by European Union’s Horizon 2020 Research and Innovation Programme under Grant agreement No. 825258.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 IFIP International Federation for Information Processing
About this paper
Cite this paper
Wang, T., Payberah, A.H., Hagos, D.H., Vlassov, V. (2022). Accelerate Model Parallel Deep Learning Training Using Effective Graph Traversal Order in Device Placement. In: Eyers, D., Voulgaris, S. (eds) Distributed Applications and Interoperable Systems. DAIS 2022. Lecture Notes in Computer Science, vol 13272. Springer, Cham. https://doi.org/10.1007/978-3-031-16092-9_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-16092-9_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16091-2
Online ISBN: 978-3-031-16092-9
eBook Packages: Computer ScienceComputer Science (R0)
