AI Benchmarking for Science: Efforts from the MLCommons Science Working Group
Authors: 
Jeyan Thiyagalingam, Gregor von Laszewski, Junqi Yin, Murali Emani, Juri Papay, Gregg Barrett, Piotr Luszczek, Aristeidis Tsaris, Christine Kirkpatrick, Feiyi Wang, Tom Gibbs, Venkatram Vishwanath, Mallikarjun Shankar, 
Geoffrey Fox, Tony HeyAuthors Info & Claims
Jeyan Thiyagalingam, Gregor von Laszewski, Junqi Yin, Murali Emani, Juri Papay, Gregg Barrett, Piotr Luszczek, Aristeidis Tsaris, Christine Kirkpatrick, Feiyi Wang, Tom Gibbs, Venkatram Vishwanath, Mallikarjun Shankar, Geoffrey Fox, Tony HeyAuthors Info & ClaimsPages 47 - 64
Abstract
With machine learning (ML) becoming a transformative tool for science, the scientific community needs a clear catalogue of ML techniques, and their relative benefits on various scientific problems, if they were to make significant advances in science using AI. Although this comes under the purview of benchmarking, conventional benchmarking initiatives are focused on performance, and as such, science, often becomes a secondary criteria.
In this paper, we describe a community effort from a working group, namely, MLCommons Science Working Group, in developing science-specific AI benchmarking for the international scientific community. Since the inception of the working group in 2020, the group has worked very collaboratively with a number of national laboratories, academic institutions and industries, across the world, and has developed four science-specific AI benchmarks. We will describe the overall process, the resulting benchmarks along with some initial results. We foresee that this initiative is likely to be very transformative for the AI for Science, and for performance-focused communities.
References
[1]
Callaway E It will change everything: DeepMind’s AI makes gigantic leap in solving protein structures Nature 2020 588 203-204
[2]
Department of Energy: Artificial Intelligence for Science in the US Department of Energy. https://science.osti.gov/Initiatives/AI. Accessed 30 June 2022
[3]
Earthquake Data. https://github.com/laszewsk/mlcommons-data-earthquake. Accessed 30 June 2022
[4]
ECP-CANDLE: Benchmarks. GitHub. https://github.com/ECP-CANDLE/Benchmarks/tree/master/Pilot1/Uno. Accessed 30 June 2022
[5]
Farrell, S., et al.: MLPerf HPC: a holistic benchmark suite for scientific machine learning on HPC systems (2021). arXiv:2110.11466
[6]
Fox, G., Hey, T., Thiyagalingam, J.: Science data working group of MLCommons research. Web Page. https://mlcommons.org/en/groups/research-science/. Accessed 30 June 2022
[7]
Fox G, Rundle J, Donnellan A, and Feng B Earthquake nowcasting with deep learning Geohazards 2022 3 2 199
[8]
Fox, G.C., von Laszewski, G., Knuuti, R., Butler, T., Kolesar, J.: MLCommons science benchmark earthquake code. https://bityl.co/COro
[9]
Henghes B, Pettitt C, Thiyagalingam J, Hey T, and Lahav O Benchmarking and scalability of machine-learning methods for photometric redshift estimation Mon. Notices Royal Astron. Soc. 2021 505 4 4847-4856
[10]
Henghes B, Thiyagalingam J, Pettitt C, Hey T, and Lahav O Deep learning methods for obtaining photometric redshift estimations from images Mon. Notices Royal Astron. Soc. 2022 512 2 1696-1709
[11]
Hey, T., Butler, K., Jackson, S., Thiyagalingam, J.: Machine learning and big scientific data. Philos. Trans. Ser. A Math. Phys. Eng. Sci. 378(2166), 20190054 (2020)
[12]
Jackson, S., Cox, C., Thiyagalingam, J., Hey, T.: SciML-Bench: SciML benchmarking suite for AI for science: cloud masking benchmark. GitHub (2021). https://github.com/stfc-sciml/sciml-bench/tree/master/sciml_bench/benchmarks/slstr_cloud. Accessed 30 June 2022
[13]
Jumper J, Evans R, Pritzel A, et al. Highly accurate protein structure prediction with AlphaFold Nature 2021 596 583-589
[14]
Laanait, N., Borisevich, A., Yin, J.: A database of convergent beam electron diffraction patterns for machine learning of the structural properties of materials (2019). https://www.osti.gov/servlets/purl/1510313/
[15]
Laanait, N., et al.: Exascale deep learning for scientific inverse problems (2019). arXiv:1909.11150
[16]
Lim B, Arık SÖ, Loeff N, and Pfister T Temporal fusion transformers for interpretable multi-horizon time series forecasting Int. J. Forecast. 2021 37 4 1748-1764
[17]
Merchant CJ, Harris AR, Maturi E, and Maccallum S Probabilistic physically based cloud screening of satellite infrared imagery for operational sea surface temperature retrieval Q. J. R. Meteorol. Soc. 2005 131 611 2735-2755
[18]
Nash J and Sutcliffe J River flow forecasting through conceptual models part I - a discussion of principles J. Hydrol. 1970 10 3 282-290
[19]
Pan J Nichols J, Verastegui B, Maccabe AB, Hernandez O, Parete-Koon S, and Ahearn T Probability flow for classifying crystallographic space groups Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI 2020 Cham Springer 451-464
[20]
Ronneberger O, Fischer P, and Brox T Navab N, Hornegger J, Wells WM, and Frangi AF U-Net: convolutional networks for biomedical image segmentation Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 2015 Cham Springer 234-241
[21]
STEMDL Benchmark: STEMDL Benchmark. GitHub. https://github.com/at-aaims/stemdl-benchmark. Accessed 30 June 2022
[22]
Tanaka, A., Tomiya, A., Hashimoto, K.: Deep Learning and Physics. Springer, Singapore (2021).
[23]
Thiyagalingam, J., et al.: SciML-bench: SciML benchmarking suite for AI for science. GitHub (2021). https://github.com/stfc-sciml/sciml-bench. Accessed 30 June 2022
[24]
Thiyagalingam J, Shankar M, Fox G, and Hey T Scientific machine learning benchmarks Nat. Rev. Phys. 2022 4 413-420
[25]
Index of Pilot1 CANDLE-UNO Benchmark. https://ftp.mcs.anl.gov/pub/candle/public/benchmarks/Pilot1/combo. Accessed 30 June 2022
[26]
Wilkinson, M.D., et al.: The FAIR guiding principles for scientific data management and stewardship. Sci. Data 3(1) (2016)
Index Terms
- AI Benchmarking for Science: Efforts from the MLCommons Science Working Group
Index terms have been assigned to the content through auto-classification.
Recommendations
A Study of Machine Learning Inference Benchmarks
ICAIP '20: Proceedings of the 4th International Conference on Advances in Image ProcessingMachine learning (ML) is becoming a powerful tool for a variety of applications where artificial intelligence solutions are required. A ML benchmark is a standard suite to measure, evaluate and compare the performance and efficiency of ML systems. This ...
A Comprehensive Java Benchmark Study on Memory and Garbage Collection Behavior of DaCapo, DaCapo Scala, and SPECjvm2008
ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance EngineeringBenchmark suites are an indispensable part of scientific research to compare different approaches against each another. The diversity of benchmarks is an important asset to evaluate novel approaches for effectiveness and weaknesses. In this paper, we ...
Renaissance: benchmarking suite for parallel applications on the JVM
PLDI 2019: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and ImplementationEstablished benchmark suites for the Java Virtual Machine (JVM), such as DaCapo, ScalaBench, and SPECjvm2008, lack workloads that take advantage of the parallel programming abstractions and concurrency primitives offered by the JVM and the Java Class ...
Comments
Information & Contributors
Information
Published In
May 2022
398 pages
ISBN:978-3-031-23219-0
DOI:10.1007/978-3-031-23220-6
© Springer Nature Switzerland AG 2022.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 04 January 2023
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 12 Aug 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in