Streaming Data from Experimental Facilities to Supercomputers for Real-Time Data Processing
Authors: 
Siniša Veseli, John Hammonds, Steven Henke, Hannah Parraga, Nicholas SchwarzAuthors Info & Claims
Siniša Veseli, John Hammonds, Steven Henke, Hannah Parraga, Nicholas SchwarzAuthors Info & ClaimsPages 2110 - 2117
Abstract
In this paper we demonstrate direct data streaming from instruments and detectors at a large-scale experimental facility to a supercomputer for real-time data processing and feedback. Streaming data to supercomputers introduces the potential for novel scientific applications and workflow models, including the ability to provide real-time feedback from very large datasets during an experiment and the integration of real-time ML training and inference at scale. We discuss a successful demonstration for real-time processing of data from the Advanced Photon Source (APS) on the Polaris supercomputer using an EPICS-based streaming framework. We describe the capabilities of the streaming framework itself, and outline the architecture that allows us to process experimentally derived data on a supercomputer without file-based data transfers. We present throughput measurements that are indicative of system performance capable of sustaining the expected data production rates of the facility, as well as discuss some outstanding challenges and our future directions.
Supplemental Material
MP4 File - Conference Presentation Video
Recording of "Streaming Data from Experimental Facilities to Supercomputers for Real-Time Data Processing" presentation at the 5th Annual Workshop on Extreme-Scale Experiment-in-the-Loop Computing (XLOOP 2023)
- Download
- 135.77 MB
References
[1]
Argonne National Laboratory 2023. The Argonne Leadership Computing Facility (ALCF). Argonne National Laboratory. https://www.alcf.anl.gov
[2]
Argonne National Laboratory 2023. C2 Data Viewer. Argonne National Laboratory. https://github.com/epics-extensions/c2dataviewer
[3]
2023. EPICS Normative Types Specification. https://docs.epics-controls.org/en/latest/specs/Normative-Types-Specification.html
[4]
2023. EPICS PV Access Protocol Specification. https://github.com/epics-base/pvAccessCPP/wiki/protocol
[5]
Argonne National Laboratory 2023. The Polaris Supercomputer. Argonne National Laboratory. https://www.alcf.anl.gov/polaris
[6]
M. Abramoff, Paulo Magalhães, and S.J. Ram. 2003. Image Processing with ImageJ. Biophotonics International 11 (11 2003), 36–42.
[7]
Bryce Allen, John Bresnahan, Lisa Childers, Ian Foster, Gopi Kandaswamy, Raj Kettimuthu, Jack Kordas, Mike Link, Stuart Martin, Karl Pickett, and Steven Tuecke. 2012. Software as a Service for Data Scientists. Commun. ACM 55, 2 (feb 2012), 81–88. https://doi.org/10.1145/2076450.2076468
[8]
Anakha V Babu, Tao Zhou, Saugat Kandel, Tekin Bicer, Zhengchun Liu, William Judge, Daniel J. Ching, Yi Jiang, Sinisa Veseli, Steven Henke, Ryan Chard, Yudong Yao, Ekaterina Sirazitdinova, Geetika Gupta, Martin V. Holt, Ian T. Foster, Antonino Miceli, and Mathew J. Cherukara. 2022. Deep learning at the edge enables real-time streaming ptychographic imaging. arxiv:2209.09408 [cs.LG]
[9]
M. Borland 2018. The Upgrade of the Advanced Photon Source. In Proc. 9th International Particle Accelerator Conference (IPAC’18), Vancouver, BC, Canada, April 29-May 4, 2018(International Particle Accelerator Conference, 9). JACoW Publishing, Geneva, Switzerland, 2872–2877. https://doi.org/10.18429/JACoW-IPAC2018-THXGBD1 https://doi.org/10.18429/JACoW-IPAC2018-THXGBD1.
[10]
Dale Carder, Eli Dart, Matthias Graf, Carol Hawk, Aaron Holder, Dylan Jacob, Eliane Lessner, Kenneth Miller, Cody Rotermund, Thomas Russell, Athena Sefat, Andrew Wiedlea, and Jason Zurawski. 2022. Basic Energy Sciences Network Requirements Review (Final Report). Technical Report LBNL-2001490. Lawerence Berkeley National Laboratory. https://escholarship.org/uc/item/3jj0h54n
[11]
Leo R. Dalesio, Jeffrey O. Hill, Martin Kraimer, Stephen Lewis, Douglas Murray, Stephan Hunt, William Watson, Matthias Clausen, and John Dalesio. 1994. The experimental physics and industrial control system architecture: past, present, and future. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 352, 1 (1994), 179–184. https://doi.org/10.1016/0168-9002(94)91493-1
[12]
Leo R. Dalesio, A J Kozubal, and M R Kraimer. 1991. EPICS Architecture. In Proceedings of the 4th International Conference on Accelerator and Large Experimental Control Systems (ICALEPCS 1991). 278–282. https://www.osti.gov/biblio/6110347
[13]
Ian Foster. 2011. Globus Online: Accelerating and Democratizing Science through Cloud-Based Services. IEEE Internet Computing 15, 3 (2011), 70–73. https://doi.org/10.1109/MIC.2011.64
[14]
Steven Henke. 2023. Ptychodus. Argonne National Laboratory. https://github.com/AdvancedPhotonSource/ptychodus
[15]
J.O. Hill. 1990. Channel access: A software bus for the LAACS. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 293, 1 (1990), 352–355. https://doi.org/10.1016/0168-9002(90)91459-O
[16]
Zhengchun Liu, Ahsan Ali, Peter Kenesei, Antonino Miceli, Hemant Sharma, Nicholas Schwarz, Dennis Trujillo, Hyunseung Yoo, Ryan Coffee, Naoufal Layad, Jana Thayer, Ryan Herbst, Chunhong Yoon, and Ian Foster. 2021. Bridging Data Center AI Systems with Edge Computing for Actionable Information Retrieval. In 2021 3rd Annual Workshop on Extreme-scale Experiment-in-the-Loop Computing (XLOOP). IEEE, St. Louis, MO, 15–23. https://doi.org/10.1109/XLOOP54565.2021.00008
[17]
Viktor Nikitin, Aniket Tekawade, Anton Duchkov, Pavel Shevchenko, and Francesco De Carlo. 2022. Real-time streaming tomographic reconstruction with on-demand data capturing and 3D zooming to regions of interest. Journal of Synchrotron Radiation 29, 3 (May 2022), 816–828. https://doi.org/10.1107/S1600577522003095
[18]
Mark Rivers. 2023. The Area Detector Application Framework. The University of Chicago. https://areadetector.github.io/areaDetector/index.html
[19]
Nicholas Schwarz. 2022. The APS Scientific Computing Strategy. Technical Report. Argonne National Laboratory. https://www.aps.anl.gov/files/APS-Uploads/XSD/XSD-Strategic-Plans/APSScientificComputingStrategy-2022-12-09-FINAL.pdf
[20]
Nicholas Schwarz, Stuart Campbell, Alexander Hexemer, Apurva Mehta, and Jana Thayer. 2020. Enabling Scientific Discovery at Next-Generation Light Sources with Advanced AI and HPC. In Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI, Jeffrey Nichols, Becky Verastegui, Arthur ‘Barney’ Maccabe, Oscar Hernandez, Suzanne Parete-Koon, and Theresa Ahearn (Eds.). Springer International Publishing, Cham, 145–156.
[21]
Rafael Vescovi, Ryan Chard, Nickolaus D. Saint, Ben Blaiszik, Jim Pruyne, Tekin Bicer, Alex Lavens, Zhengchun Liu, Michael E. Papka, Suresh Narayanan, Nicholas Schwarz, Kyle Chard, and Ian T. Foster. 2022. Linking scientific instruments and computation: Patterns, technologies, and experiences. Patterns 3, 10 (2022), 100606. https://doi.org/10.1016/j.patter.2022.100606
[22]
S. Veseli. 2015. PvaPy: Python API for EPICS PV Access. In Proc. ICALEPCS’15. JACoW Publishing, Geneva, Switzerland, Melbourne, Australia, 970–973. https://doi.org/10.18429/JACoW-ICALEPCS2015-WEPGF116
[23]
Sinisa Veseli. 2023. Data Distributor Plugin Documentation. Argonne National Laboratory. https://github.com/epics-base/pvaPy/blob/master/documentation/dataDistributorPlugin.md
[24]
Sinisa Veseli. 2023. PvaPy Streaming Framework features, use cases, and different workflow and configuration examples. Argonne National Laboratory. https://github.com/epics-base/pvaPy/blob/master/documentation/streamingFramework.md
[25]
Siniša Veseli, Nicholas Schwarz, and Collin Schmitz. 2018. APS Data Management System. Journal of Synchrotron Radiation 25, 5 (Sep 2018), 1574–1580. https://doi.org/10.1107/S1600577518010056
[26]
Howard Yanxon, James Weng, Hannah Parraga, Wenqian Xu, Uta Ruett, and Nicholas Schwarz. 2023. Artifact identification in X-ray diffraction data using machine learning methods. Journal of Synchrotron Radiation 30, 1 (Jan 2023), 137–146. https://doi.org/10.1107/S1600577522011274
Index Terms
- Streaming Data from Experimental Facilities to Supercomputers for Real-Time Data Processing
Recommendations
Real-time processing of streaming big data
In the era of data explosion, high volume of various data is generated rapidly at each moment of time; and if not processed, the profits of their latent information would be missed. This is the main current challenge of most enterprises and Internet ...
Research on Real-time Processing and Stream Analysis of Unstructured Data Based on Big Data Platforms
MIDA '24: Proceedings of the 2024 International Conference on Machine Intelligence and Digital ApplicationsWith the rise of the big data era, massive streams of data are emerging in various fields such as the internet, Internet of Things, and finance, posing significant challenges to real-time processing. These data exhibit characteristics such as high ...
The Key Technologies of Real-Time Processing Large Scale Microblog Data Stream
CloudCom-Asia 2015: Revised Selected Papers of the Second International Conference on Cloud Computing and Big Data - Volume 9106Real-time monitoring microblog data can find sensitive information in time and provide help for public sentiment management and control. However, it needs processing large-scale data stream. MapReduce is a framework of processing large-scale data in ...
Comments
Information & Contributors
Information
Published In
November 2023
2180 pages
ISBN:9798400707858
DOI:10.1145/3624062
Copyright © 2023 ACM.
Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 November 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SC-W 2023
SC-W 2023: Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis
November 12 - 17, 2023
CO, Denver, USA
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 95Total Downloads
- Downloads (Last 12 months)95
- Downloads (Last 6 weeks)7
Reflects downloads up to 10 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format