Abstract
Technological advancement in networking and IoT have given researchers new methods or techniques to perform numerical analysis and simulation with the latest data observed on IoT sensors and other measurement devices. In general, large-scale simulations necessitate high-performance computing (HPC) systems. Such HPC systems are operated in a shared manner among researchers. Therefore, it becomes inherently difficult for researchers to use the latest observation data generated on remote data sources for their simulations. To enable researchers to utilize fresh data on a remote data source for computation, we propose an on-time data transfer framework that enables the execution of jobs with fresh data generated on a remote site data on a shared HPC system by extending the SLURM scheduler. The proposed framework consists of two functions: Job pinning and On-time data transfer. With the job pinning function, the proposed framework prevents the scheduling algorithm from rearranging the scheduled start time of jobs. The on-time data transfer function is in charge of data transfer from a remote site to the data transfer node. It attempts to complete the data transfer at just the time of the pinned start time of jobs. The evaluation in this paper indicates that the proposed framework can keep data freshness high and minimize the job waiting time for data transfer.
Supported by Japan Society for the Promotion of Science (JSPS) Grants-in-Aid for Scientific Research (KAKENHI).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
FastAPI. https://github.com/tiangolo/fastapi
MPI: A Message-Passing Interface Standard. https://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf
TC(8) - Linux Manual Page. https://man7.org/linux/man-pages/man8/tc.8.html
Why Docker? | Docker. https://www.docker.com/why-docker
Allcock, W., Bresnahan, J., Kettimuthu, R., Link, M.: The Globus striped GridFTP framework and server. In: Proceedings of the 2005 ACM/IEEE Conference on Supercomputing, p. 54, November 2005
Allcock, W.: GridFTP: Protocol Extensions to FTP for the Grid. http://www.ggf.org/documents/GFD.20.pdf (2003)
Dong, B., Byna, S., Wu, K., Prabhat, Johansen, H., Johnson, J.N., Keen, N.: Data elevator: low-contention data movement in hierarchical storage system. In: Proceedings of the 2016 IEEE International Conference on High Performance Computing (HiPC), pp. 152–161, December 2016
Endo, A., et al.: Dynamic traffic control of staging traffic on the interconnect of the HPC cluster system. IEEE Access 8, 198518–198531 (2020)
Ghil, M., Malanotte-Rizzoli, P.: Data assimilation in meteorology and oceanography. Adv. Geophys. 33, 141–266 (1991)
Hovestadt, M., Kao, O., Keller, A., Streit, A.: Scheduling in HPC resource management systems: queuing vs. planning. In: Proceedings of the 2003 Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 1–20, June 2003
Lu, Q., et al.: BigData express: toward schedulable, predictable, and high-performance data transfer. In: Proceedings of the 2018 IEEE/ACM Innovating the Network for Data-Intensive Science (INDIS), pp. 75–84, November 2018
Miranda, A., Jackson, W., Tocci, T., Panourgias, I., Nou, R.: NORNS: extending Slurm to support data-driven workflows through asynchronous data staging. In: Proceedings of the 2019 IEEE International Conference on Cluster Computing (CLUSTER), pp. 1–12, November 2019
Miyoshi, T., et al.: Big data assimilation toward post-petascale severe weather prediction: an overview and progress. Proc. IEEE 104(11), 2155–2179 (2016)
Monti, H.M., Butt, A.R., Vazhkudai, S.S.: On timely staging of HPC job input data. IEEE Trans. Parallel Distrib. Syst. 24(9), 1841–1851 (2013)
Shmueli, E., Feitelson, D.G.: Backfilling with lookahead to optimize the performance of parallel job scheduling. In: Proceedings of the 2003 Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 228–251, June 2003
Wang, Y., Satake, K., Maeda, T., Gusman, A.R.: Data assimilation with dispersive tsunami model: a test for the Nankai trough. Earth, Planets Space 70(1), 1–9 (2018)
Yoo, A.B., Jette, M.A., Grondona, M.: SLURM: simple linux utility for resource management. In: Proceedings of the 2003 Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 44–60, June 2003
Yu, S., et al.: SCinet DTN-as-a-service framework. In: Proceedings of the 2019 IEEE/ACM Innovating the Network for Data-Intensive Science (INDIS), pp. 1–8, November 2019
Acknowledgements
This research is partly supported by JSPS KAKENHI Grant Number JP17KT0083 and JP21K11912.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 IFIP International Federation for Information Processing
About this paper
Cite this paper
Yamamoto, K., Endo, A., Date, S. (2022). Architecture of an On-Time Data Transfer Framework in Cooperation with Scheduler System. In: Cérin, C., Qian, D., Gaudiot, JL., Tan, G., Zuckerman, S. (eds) Network and Parallel Computing. NPC 2021. Lecture Notes in Computer Science(), vol 13152. Springer, Cham. https://doi.org/10.1007/978-3-030-93571-9_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-93571-9_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-93570-2
Online ISBN: 978-3-030-93571-9
eBook Packages: Computer ScienceComputer Science (R0)
