Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3624062.3626081acmotherconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
research-article
Open access

Towards Foundation Models for Materials Science: The Open MatSci ML Toolkit

Published: 12 November 2023 Publication History

Abstract

Artificial intelligence and machine learning have shown great promise in their ability to accelerate novel materials discovery. As researchers and domain scientists seek to unify and consolidate chemical knowledge, the case for models with potential to generalize across different tasks within materials science—so-called “foundation models”—grows with ambitions. This manuscript reviews our recent progress with development of Open MatSci ML Toolkit, and details experiments that lay the groundwork for foundation model research and development with our framework. First, we describe and characterize a new pretraining task that uses synthetic data generated from symmetry operations, and reveal complex training dynamics at large scales. Using the pretrained model, we discuss a number of use cases relevant to foundation model development: semantic architecture of datasets, and fine-tuning for property prediction and classification. Our key results show that for simple applications, pretraining appears to provide worse modeling performance than training models from random initialization. However, for more complex instances, such as when a model is required to learn across multiple datasets and types of targets simultaneously, the inductive bias from pretraining provides significantly better performance. This insight will hopefully inform subsequent efforts into creating foundation models for materials science applications.

Supplemental Material

MP4 File
Recording of "Towards Foundation Models for Materials Science: The Open MatSci ML Toolkit" presentation at AI4S 2023.

References

[1]
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. 2022. E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials. https://doi.org/10.24435/MATERIALSCLOUD:S0-5N
[2]
B. Blaiszik, K. Chard, J. Pruyne, R. Ananthakrishnan, S. Tuecke, and I. Foster. 2016. The Materials Data Facility: Data Services to Advance Materials Science Research. JOM 68, 8 (Aug. 2016), 2045–2052. https://doi.org/10.1007/s11837-016-2001-3
[3]
Ben Blaiszik, Logan Ward, Marcus Schwarting, Jonathon Gaff, Ryan Chard, Daniel Pike, Kyle Chard, and Ian Foster. 2019. A Data Ecosystem to Support Machine Learning in Materials Science. MRS Communications 9, 4 (Dec. 2019), 1125–1133. https://doi.org/10.1557/mrc.2019.118
[4]
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2022. On the Opportunities and Risks of Foundation Models. arxiv:2108.07258 [cs]
[5]
Johann Brehmer, Pim de Haan, Sönke Behrends, and Taco Cohen. 2023. Geometric Algebra Transformers. arxiv:2305.18415 [cs, stat]
[6]
Cayley. 1874. LVII. Onthe Mathematical Theory of Isomers. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 47, 314 (June 1874), 444–447. https://doi.org/10.1080/14786447408641058
[7]
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. 2021. The Open Catalyst 2020 (OC20) Dataset and Community Challenges. ACS Catalysis 11, 10 (May 2021), 6059–6072. https://doi.org/10.1021/acscatal.0c04525 arxiv:2010.09990 [cond-mat]
[8]
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. 2019. Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals. Chemistry of Materials 31, 9 (May 2019), 3564–3572. https://doi.org/10.1021/acs.chemmater.9b01294
[9]
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. https://doi.org/10.48550/arXiv.2204.02311 arxiv:2204.02311 [cs]
[10]
F. Coester and H. Kümmel. 1960. Short-Range Correlations in Nuclear Wave Functions. Nuclear Physics 17 (June 1960), 477–485. https://doi.org/10.1016/0029-5582(60)90140-1
[11]
Yabo Dan, Yong Zhao, Xiang Li, Shaobo Li, Ming Hu, and Jianjun Hu. 2020. Generative Adversarial Networks (GAN) Based Efficient Sampling of Chemical Composition Space for Inverse Design of Inorganic Materials. npj Computational Materials 6, 1 (June 2020), 1–7. https://doi.org/10.1038/s41524-020-00352-0
[12]
Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. 2020. Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm. npj Computational Materials 6, 1 (Sept. 2020), 1–10. https://doi.org/10.1038/s41524-020-00406-3
[13]
Alexandre Duval, Victor Schmidt, Santiago Miret, Yoshua Bengio, Alex Hernández-García, and David Rolnick. 2023. PhAST: Physics-Aware, Scalable, and Task-specific GNNs for Accelerated Catalyst Design. https://doi.org/10.48550/arXiv.2211.12020 arxiv:2211.12020 [physics]
[14]
William Falcon and The PyTorch Lightning team. 2019. PyTorch Lightning. https://doi.org/10.5281/zenodo.3828935
[15]
Jerome H. Friedman. 2001. Greedy Function Approximation: A Gradient Boosting Machine.The Annals of Statistics 29, 5 (Oct. 2001), 1189–1232. https://doi.org/10.1214/aos/1013203451
[16]
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. 2021. GemNet: Universal Directional Graph Neural Networks for Molecules. In Advances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 6790–6802.
[17]
Zhangxiaowen Gong, Houxiang Ji, Yao Yao, Christopher W. Fletcher, Christopher J. Hughes, and Josep Torrellas. 2022. Graphite: Optimizing Graph Neural Networks on CPUs through Cooperative Software-Hardware Techniques. In Proceedings of the 49th Annual International Symposium on Computer Architecture. ACM, New York New York, 916–931. https://doi.org/10.1145/3470496.3527403
[18]
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2018. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arxiv:1706.02677 [cs]
[19]
Alexander Heinecke, Greg Henry, Maxwell Hutchinson, and Hans Pabst. 2016. LIBXSMM: Accelerating Small Matrix Multiplications by Runtime Code Generation. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis(SC ’16). IEEE Press, Salt Lake City, Utah, 1–11.
[20]
Ryan Jacobs, Tam Mayeshiba, Ben Afflerbach, Luke Miles, Max Williams, Matthew Turner, Raphael Finkel, and Dane Morgan. 2020. The Materials Simulation Toolkit for Machine Learning (MAST-ML): An Automated Open Source Toolkit to Accelerate Data-Driven Materials Research. Computational Materials Science 176 (April 2020), 109544. https://doi.org/10.1016/j.commatsci.2020.109544
[21]
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson. 2013. Commentary: The Materials Project: A Materials Genome Approach to Accelerating Materials Innovation. APL Materials 1, 1 (July 2013), 011002. https://doi.org/10.1063/1.4812323
[22]
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-Normalizing Neural Networks. arxiv:1706.02515 [cs, stat]
[23]
W. Kohn and L. J. Sham. 1965. Self-Consistent Equations Including Exchange and Correlation Effects. Physical Review 140, 4A (Nov. 1965), A1133–A1138. https://doi.org/10.1103/PhysRev.140.A1133
[24]
Zhizhong Li and Derek Hoiem. 2017. Learning without Forgetting. arxiv:1606.09282 [cs, stat]
[25]
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. https://doi.org/10.48550/arXiv.1711.05101 arxiv:1711.05101 [cs, math]
[26]
Devin A. Matthews, Lan Cheng, Michael E. Harding, Filippo Lipparini, Stella Stopkowicz, Thomas-C. Jagau, Péter G. Szalay, Jürgen Gauss, and John F. Stanton. 2020. Coupled-Cluster Techniques for Computational Chemistry: The CFOUR Program Package. The Journal of Chemical Physics 152, 21 (June 2020), 214108. https://doi.org/10.1063/5.0004837
[27]
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018. UMAP: Uniform Manifold Approximation and Projection. Journal of Open Source Software 3, 29 (Sept. 2018), 861. https://doi.org/10.21105/joss.00861
[28]
Santiago Miret, Kin Long Kelvin Lee, Carmelo Gonzales, Marcel Nassar, and Matthew Spellings. 2023. The Open MatSci ML Toolkit: A Flexible Framework for Machine Learning in Materials Science. Transactions on Machine Learning Research (2023).
[29]
Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana Liskovich, Puxin Xu, Yuchen Zhang, Melanie Kambadur, Stephen Roller, and Susan Zhang. 2023. A Theory on Adam Instability in Large-Scale Machine Learning. https://doi.org/10.48550/arXiv.2304.09871 arxiv:2304.09871 [cs, math]
[30]
Ayako Nakata, Jack S. Baker, Shereif Y. Mujahed, Jack T. L. Poulton, Sergiu Arapan, Jianbo Lin, Zamaan Raza, Sushma Yadav, Lionel Truflandier, Tsuyoshi Miyazaki, and David R. Bowler. 2020. Large Scale and Linear Scaling DFT with the CONQUEST Code. The Journal of Chemical Physics 152, 16 (April 2020), 164112. https://doi.org/10.1063/5.0005074
[31]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-Learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.
[32]
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. 2014. Quantum Chemistry Structures and Properties of 134 Kilo Molecules. Scientific Data 1, 1 (Aug. 2014), 140022. https://doi.org/10.1038/sdata.2014.22
[33]
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning Multiple Visual Domains with Residual Adapters. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc.
[34]
Jean-Louis Reymond. 2015. The Chemical Space Project. Accounts of Chemical Research 48, 3 (March 2015), 722–730. https://doi.org/10.1021/ar500432k
[35]
T. Konstantin Rusch, Michael M. Bronstein, and Siddhartha Mishra. 2023. A Survey on Oversmoothing in Graph Neural Networks. arxiv:2303.10993 [cs]
[36]
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. 2022. E(n) Equivariant Graph Neural Networks. arxiv:2102.09844 [cs, stat]
[37]
Kristof T. Schütt, Pieter-Jan Kindermans, Huziel E. Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. 2017. SchNet: A Continuous-Filter Convolutional Neural Network for Modeling Quantum Interactions. arXiv:1706.08566 [physics, stat] (Dec. 2017). arxiv:1706.08566 [physics, stat]
[38]
Matthew Spellings. 2022. Geometric Algebra Attention Networks for Small Point Clouds. arxiv:2110.02393 [cs]
[39]
Richard Tran, Janice Lan, Muhammed Shuaibi, Brandon M. Wood, Siddharth Goyal, Abhishek Das, Javier Heras-Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, Anuroop Sriram, Felix Therrien, Jehad Abed, Oleksandr Voznyy, Edward H. Sargent, Zachary Ulissi, and C. Lawrence Zitnick. 2022. The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts. arxiv:2206.08917 [cond-mat, physics:physics]
[40]
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. 2019. Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks. arXiv preprint arXiv:1909.01315 (2019). arxiv:1909.01315
[41]
Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normalization. https://doi.org/10.48550/arXiv.1910.07467 arxiv:1910.07467 [cs, stat]
[42]
Yong Zhao, Mohammed Al-Fahdi, Ming Hu, Edirisuriya M. D. Siriwardane, Yuqi Song, Alireza Nasiri, and Jianjun Hu. 2021. High-Throughput Discovery of Novel Cubic Crystal Materials Using Deep Generative Neural Networks. Advanced Science 8, 20 (2021), 2100566. https://doi.org/10.1002/advs.202100566

Cited By

View all
  • (2024)DPA-2: a large atomic model as a multi-task learnernpj Computational Materials10.1038/s41524-024-01493-210:1Online publication date: 19-Dec-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
SC-W '23: Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis
November 2023
2180 pages
ISBN:9798400707858
DOI:10.1145/3624062
This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2023

Check for updates

Author Tags

  1. chemistry
  2. datasets
  3. graph neural networks
  4. materials science
  5. neural networks
  6. point clouds

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SC-W 2023

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)481
  • Downloads (Last 6 weeks)68
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)DPA-2: a large atomic model as a multi-task learnernpj Computational Materials10.1038/s41524-024-01493-210:1Online publication date: 19-Dec-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media