IO-SEA: Storage I/O and Data Management for Exascale Architectures
Authors: 
Daniel Medeiros, Eric B. Gregory, Philippe Couvee, James Hawkes, Sebastien Gougeaud, Maike Gilliot, Olivier Bressand, Yoann Valeri, Julien Jaeger, Damien Chapon, Frederic Bournaud, Loïc Strafella, Daniel Caviedes-Voullième, Ghazal Tashakor, Jolanta Zjupa, Max Holicki, Tom Ridley, Yanik Müller, Filipe Souza Mendes Guimarães, Wolfgang Frings, Jan-Oliver Mirus, Ilya Zhukov, Eric Rodrigues Borba, Nafiseh Moti, Reza Salkhordeh, Nadia Derbey, Salim Mimouni, Simon Derr, Buket Benek Gursoy, James Grogan, Radek Furmánek, Martin Golasowski, Kateřina Slaninová, Jan Martinovič, Jan Faltýnek, Jenny Wong, Metin Cakircali, Tiago Quintino, Simon Smart, Olivier Iffrig, Sai Narasimhamurthy, Sonja Happ, Michael Rauh, Stephan Krempel, Mark Wiggins, Jiří Nováček, André Brinkmann, Stefano Markidis, Philippe DenierAuthors Info & Claims
Daniel Medeiros, Eric B. Gregory, Philippe Couvee, James Hawkes, Sebastien Gougeaud, Maike Gilliot, Olivier Bressand, Yoann Valeri, Julien Jaeger, Damien Chapon, Frederic Bournaud, Loïc Strafella, Daniel Caviedes-Voullième, Ghazal Tashakor, Jolanta Zjupa, Max Holicki, Tom Ridley, Yanik Müller, Filipe Souza Mendes Guimarães, Wolfgang Frings, Jan-Oliver Mirus, Ilya Zhukov, Eric Rodrigues Borba, Nafiseh Moti, Reza Salkhordeh, Nadia Derbey, Salim Mimouni, Simon Derr, Buket Benek Gursoy, James Grogan, Radek Furmánek, Martin Golasowski, Kateřina Slaninová, Jan Martinovič, Jan Faltýnek, Jenny Wong, Metin Cakircali, Tiago Quintino, Simon Smart, Olivier Iffrig, Sai Narasimhamurthy, Sonja Happ, Michael Rauh, Stephan Krempel, Mark Wiggins, Jiří Nováček, André Brinkmann, Stefano Markidis, Philippe DenierAuthors Info & ClaimsPages 94 - 100
Abstract
The new emerging scientific workloads to be executed in the upcoming exascale supercomputers face major challenges in terms of storage, given their extreme volume of data. In particular, intelligent data placement, instrumentation, and workflow handling are central to application performance. The IO-SEA project developed multiple solutions to aid the scientific community in adressing these challenges: a Workflow Manager, a hierarchical storage management system, and a semantic API for storage. All of these major products incorporate additional minor products that support their mission. In this paper, we discuss both the roles of all these products and how they can assist the scientific community in achieving exascale performance.
References
[1]
Mohammadamin Ajdari, Patrick Raaf, Mostafa Kishani, Reza Salkhordeh, Hossein Asadi, and André Brinkmann. 2022. An Enterprise-Grade Open-Source Data Reduction Architecture for All-Flash Storage Systems. Proc. ACM Meas. Anal. Comput. Syst. 6, 2 (2022), 30:1--30:27. https://doi.org/10.1145/3530896
[2]
Daniel Araújo De Medeiros, Stefano Markidis, and Ivy Bo Peng. 2023. LibCOS: Enabling Converged HPC and Cloud Data Stores with MPI. In Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region. 106--116.
[3]
Peter Braam. 2019. The Lustre storage architecture. arXiv preprint arXiv:1903.01955 (2019).
[4]
Robert G. Edwards and Balint Joo. 2005. The Chroma software system for Lattice QCD. Nucl. Phys. B Proc. Suppl. 140 (2005), 832. https://doi.org/10.1016/j.nuclphysbps.2004.11.254 arXiv:hep-lat/0409003
[5]
Wolfgang Frings, Felix Wolf, and Ventsislav Petkov. 2009. Scalable massively parallel I/O to task-local files. In Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (Portland, Oregon) (SC '09). Association for Computing Machinery, New York, NY, USA, Article 17, 11 pages. https://doi.org/10.1145/1654059.1654077
[6]
Sam Hartman, Karthik Jaganathan, and Larry Zhu. 2005. The Kerberos Version 5 Generic Security Service Application Program Interface (GSS-API) Mechanism: Version 2. RFC 4121. https://doi.org/10.17487/RFC4121
[7]
Dorian Krause. 2019. JUWELS: Modular Tier-0/1 supercomputer at the Jülich supercomputing centre. Journal of large-scale research facilities JLSRF 5 (2019), A135-A135.
[8]
John Linn. 2000. Generic Security Service Application Program Interface Version 2, Update 1. RFC 2743. https://doi.org/10.17487/RFC2743
[9]
Jay Lofstead, Ivo Jimenez, Carlos Maltzahn, Quincey Koziol, John Bent, and Eric Barton. 2016. DAOS and friends: a proposal for an exascale storage system. In SC'16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 585--596.
[10]
Nafiseh Moti, André Brinkmann, Marc-André Vef, Philippe Deniel, Jesús Carretero, Philip H. Carns, Jean-Thomas Acquaviva, and Reza Salkhordeh. 2023. The I/O Trace Initiative: Building a Collaborative I/O Archive to Advance HPC. In Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W2023, Denver, CO, USA, November 12-17, 2023. ACM, 1216--1222. https://doi.org/10.1145/3624062.3624192
[11]
Sai Narasimhamurthy, Nikita Danilov, Sining Wu, Ganesan Umanesan, Stefano Markidis, Sergio Rivas-Gomez, Ivy Bo Peng, Erwin Laure, Dirk Pleiter, and Shaun De Witt. 2019. Sage: percipient storage for exascale data centric computing. Parallel computing 83 (2019), 22--33.
[12]
Dr. Clifford Neuman and Theodore Ts'o. 1993. The Kerberos Network Authentication Service (V5). RFC 1510. https://doi.org/10.17487/RFC1510
[13]
Simon D. Smart, Tiago Quintino, and Baudouin Raoult. 2017. A Scalable Object Store for Meteorological and Climate Data. In Proceedings of the Platform for Advanced Scientific Computing Conference (Lugano, Switzerland) (PASC '17). Association for Computing Machinery, New York, NY, USA, Article 13, 8 pages. https://doi.org/10.1145/3093172.3093238
[14]
Simon D. Smart, Tiago Quintino, and Baudouin Raoult. 2019. A High-Performance Distributed Object-Store for Exascale Numerical Weather Prediction and Climate. In Proceedings of the Platform for Advanced Scientific Computing Conference (Zurich, Switzerland) (PASC '19). Association for Computing Machinery, New York, NY, USA, Article 16, 11 pages. https://doi.org/10.1145/3324989.3325726
[15]
Estela Suarez, Norbert Eicker, and Thomas Lippert. 2019. Modular supercomputing architecture: from idea to production. In Contemporary high performance computing. CRC Press, 223--255.
[16]
Philipp Thörnig. 2021. JURECA: Data centric and booster modules implementing the modular supercomputing architecture at Jülich supercomputing centre. Journal of large-scale research facilities JLSRF 7 (2021), A182-A182.
[17]
Sage Weil, Scott A Brandt, Ethan L Miller, Darrell DE Long, and Carlos Maltzahn. 2006. Ceph: A scalable, high-performance distributed file system. In Proceedings of the 7th Conference on Operating Systems Design and Implementation (OSDI'06). 307--320.
[18]
Sage A Weil, Andrew W Leung, Scott A Brandt, and Carlos Maltzahn. 2007. Rados: a scalable, reliable storage service for petabyte-scale storage clusters. In Proceedings of the 2nd international workshop on Petascale data storage: held in conjunction with Supercomputing'07. 35--44.
Index Terms
- IO-SEA: Storage I/O and Data Management for Exascale Architectures
Recommendations
A unified storage layer for supporting distributed workflows in kubernetes
CHEOPS '21: Proceedings of the Workshop on Challenges and Opportunities of Efficient and Performant Storage SystemsThe distributed computing landscape has been undergoing radical changes: High-Performance Computing (HPC) applications are moving to the Cloud, as a way of simplifying development, deployment, and migration across computing systems. Meanwhile, cloud ...
Challenges on the road to exascale computing
ICS '08: Proceedings of the 22nd annual international conference on SupercomputingSupercomputing systems have made great strides in recent years as the extensive computing needs of cutting-edge engineering work and scientific discovery have driven the development of more powerful systems. The first teraflop computer, ASCI Red, came ...
Credential Management Enforcement and Secure Data Storage in gLite
This article describes new security solutions for Grid middleware, and specifically faces the issues related to the management of users' and servers' credentials, together with storing and secure data transmission in the Grid. Our work, built on Grid ...
Comments
Information & Contributors
Information
Published In
May 2024
163 pages
ISBN:9798400704925
DOI:10.1145/3637543
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 July 2024
Check for updates
Author Tags
Qualifiers
- Invited-talk
- Research
- Refereed limited
Funding Sources
- The European High Performance Computing Joint Undertaking
Conference
CF '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 273 of 785 submissions, 35%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 32Total Downloads
- Downloads (Last 12 months)32
- Downloads (Last 6 weeks)32
Reflects downloads up to 10 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in