-
How to verify the precision of density-functional-theory implementations via reproducible and universal workflows
Authors:
Emanuele Bosoni,
Louis Beal,
Marnik Bercx,
Peter Blaha,
Stefan Blügel,
Jens Bröder,
Martin Callsen,
Stefaan Cottenier,
Augustin Degomme,
Vladimir Dikan,
Kristjan Eimre,
Espen Flage-Larsen,
Marco Fornari,
Alberto Garcia,
Luigi Genovese,
Matteo Giantomassi,
Sebastiaan P. Huber,
Henning Janssen,
Georg Kastlunger,
Matthias Krack,
Georg Kresse,
Thomas D. Kühne,
Kurt Lejaeghere,
Georg K. H. Madsen,
Martijn Marsman
, et al. (20 additional authors not shown)
Abstract:
In the past decades many density-functional theory methods and codes adopting periodic boundary conditions have been developed and are now extensively used in condensed matter physics and materials science research. Only in 2016, however, their precision (i.e., to which extent properties computed with different codes agree among each other) was systematically assessed on elemental crystals: a firs…
▽ More
In the past decades many density-functional theory methods and codes adopting periodic boundary conditions have been developed and are now extensively used in condensed matter physics and materials science research. Only in 2016, however, their precision (i.e., to which extent properties computed with different codes agree among each other) was systematically assessed on elemental crystals: a first crucial step to evaluate the reliability of such computations. We discuss here general recommendations for verification studies aiming at further testing precision and transferability of density-functional-theory computational approaches and codes. We illustrate such recommendations using a greatly expanded protocol covering the whole periodic table from Z=1 to 96 and characterizing 10 prototypical cubic compounds for each element: 4 unaries and 6 oxides, spanning a wide range of coordination numbers and oxidation states. The primary outcome is a reference dataset of 960 equations of state cross-checked between two all-electron codes, then used to verify and improve nine pseudopotential-based approaches. Such effort is facilitated by deploying AiiDA common workflows that perform automatic input parameter selection, provide identical input/output interfaces across codes, and ensure full reproducibility. Finally, we discuss the extent to which the current results for total energies can be reused for different goals (e.g., obtaining formation energies).
△ Less
Submitted 26 May, 2023;
originally announced May 2023.
-
Common workflows for computing material properties using different quantum engines
Authors:
Sebastiaan P. Huber,
Emanuele Bosoni,
Marnik Bercx,
Jens Bröder,
Augustin Degomme,
Vladimir Dikan,
Kristjan Eimre,
Espen Flage-Larsen,
Alberto Garcia,
Luigi Genovese,
Dominik Gresch,
Conrad Johnston,
Guido Petretto,
Samuel Poncé,
Gian-Marco Rignanese,
Christopher J. Sewell,
Berend Smit,
Vasily Tseplyaev,
Martin Uhrin,
Daniel Wortmann,
Aliaksandr V. Yakutovich,
Austin Zadoks,
Pezhman Zarabadi-Poor,
Bonan Zhu,
Nicola Marzari
, et al. (1 additional authors not shown)
Abstract:
The prediction of material properties through electronic-structure simulations based on density-functional theory has become routinely common, thanks, in part, to the steady increase in the number and robustness of available simulation packages. This plurality of codes and methods aiming to solve similar problems is both a boon and a burden. While providing great opportunities for cross-verificati…
▽ More
The prediction of material properties through electronic-structure simulations based on density-functional theory has become routinely common, thanks, in part, to the steady increase in the number and robustness of available simulation packages. This plurality of codes and methods aiming to solve similar problems is both a boon and a burden. While providing great opportunities for cross-verification, these packages adopt different methods, algorithms, and paradigms, making it challenging to choose, master, and efficiently use any one for a given task. Leveraging recent advances in managing reproducible scientific workflows, we demonstrate how developing common interfaces for workflows that automatically compute material properties can tackle the challenge mentioned above, greatly simplifying interoperability and cross-verification. We introduce design rules for reproducible and reusable code-agnostic workflow interfaces to compute well-defined material properties, which we implement for eleven different quantum engines and use to compute three different material properties. Each implementation encodes carefully selected simulation parameters and workflow logic, making the implementer's expertise of the quantum engine directly available to non-experts. Full provenance and reproducibility of the workflows is guaranteed through the use of the AiiDA infrastructure. All workflows are made available as open-source and come pre-installed with the Quantum Mobile virtual machine, making their use straightforward.
△ Less
Submitted 11 May, 2021;
originally announced May 2021.
-
Workflows in AiiDA: Engineering a high-throughput, event-based engine for robust and modular computational workflows
Authors:
Martin Uhrin,
Sebastiaan P. Huber,
Jusong Yu,
Nicola Marzari,
Giovanni Pizzi
Abstract:
Over the last two decades, the field of computational science has seen a dramatic shift towards incorporating high-throughput computation and big-data analysis as fundamental pillars of the scientific discovery process. This has necessitated the development of tools and techniques to deal with the generation, storage and processing of large amounts of data. In this work we present an in-depth look…
▽ More
Over the last two decades, the field of computational science has seen a dramatic shift towards incorporating high-throughput computation and big-data analysis as fundamental pillars of the scientific discovery process. This has necessitated the development of tools and techniques to deal with the generation, storage and processing of large amounts of data. In this work we present an in-depth look at the workflow engine powering AiiDA, a widely adopted, highly flexible and database-backed informatics infrastructure with an emphasis on data reproducibility. We detail many of the design choices that were made which were informed by several important goals: the ability to scale from running on individual laptops up to high-performance supercomputers, managing jobs with runtimes spanning from fractions of a second to weeks and scaling up to thousands of jobs concurrently, and all this while maximising robustness. In short, AiiDA aims to be a Swiss army knife for high-throughput computational science. As well as the architecture, we outline important API design choices made to give workflow writers a great deal of liberty whilst guiding them towards writing robust and modular workflows, ultimately enabling them to encode their scientific knowledge to the benefit of the wider scientific community.
△ Less
Submitted 21 July, 2020; v1 submitted 17 July, 2020;
originally announced July 2020.
-
kiwiPy: Robust, high-volume, messaging for big-data and computational science workflows
Authors:
Martin Uhrin,
Sebastiaan P. Huber
Abstract:
In this work we present kiwiPy, a Python library designed to support robust message based communication for high-throughput, big-data, applications while being general enough to be useful wherever high-volumes of messages need to be communicated in a predictable manner. KiwiPy relies on the RabbitMQ protocol, an industry standard message broker, while providing a simple and intuitive interface tha…
▽ More
In this work we present kiwiPy, a Python library designed to support robust message based communication for high-throughput, big-data, applications while being general enough to be useful wherever high-volumes of messages need to be communicated in a predictable manner. KiwiPy relies on the RabbitMQ protocol, an industry standard message broker, while providing a simple and intuitive interface that can be used in both multithreaded and coroutine based applications. To demonstrate some of kiwiPy's functionality we give examples from AiiDA, a high-throughput simulation platform, where kiwiPy is used as a key component of the workflow engine.
△ Less
Submitted 15 May, 2020;
originally announced May 2020.
-
Materials Cloud, a platform for open computational science
Authors:
Leopold Talirz,
Snehal Kumbhar,
Elsa Passaro,
Aliaksandr V. Yakutovich,
Valeria Granata,
Fernando Gargiulo,
Marco Borelli,
Martin Uhrin,
Sebastiaan P. Huber,
Spyros Zoupanos,
Carl S. Adorf,
Casper W. Andersen,
Ole Schütt,
Carlo A. Pignedoli,
Daniele Passerone,
Joost VandeVondele,
Thomas C. Schulthess,
Berend Smit,
Giovanni Pizzi,
Nicola Marzari
Abstract:
Materials Cloud is a platform designed to enable open and seamless sharing of resources for computational science, driven by applications in materials modelling. It hosts 1) archival and dissemination services for raw and curated data, together with their provenance graph, 2) modelling services and virtual machines, 3) tools for data analytics, and pre-/post-processing, and 4) educational material…
▽ More
Materials Cloud is a platform designed to enable open and seamless sharing of resources for computational science, driven by applications in materials modelling. It hosts 1) archival and dissemination services for raw and curated data, together with their provenance graph, 2) modelling services and virtual machines, 3) tools for data analytics, and pre-/post-processing, and 4) educational materials. Data is citable and archived persistently, providing a comprehensive embodiment of the FAIR principles that extends to computational workflows. Materials Cloud leverages the AiiDA framework to record the provenance of entire simulation pipelines (calculations performed, codes used, data generated) in the form of graphs that allow to retrace and reproduce any computed result. When an AiiDA database is shared on Materials Cloud, peers can browse the interconnected record of simulations, download individual files or the full database, and start their research from the results of the original authors. The infrastructure is agnostic to the specific simulation codes used and can support diverse applications in computational science that transcend its initial materials domain.
△ Less
Submitted 27 March, 2020;
originally announced March 2020.
-
AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance
Authors:
Sebastiaan. P. Huber,
Spyros Zoupanos,
Martin Uhrin,
Leopold Talirz,
Leonid Kahle,
Rico Häuselmann,
Dominik Gresch,
Tiziano Müller,
Aliaksandr V. Yakutovich,
Casper W. Andersen,
Francisco F. Ramirez,
Carl S. Adorf,
Fernando Gargiulo,
Snehal Kumbhar,
Elsa Passaro,
Conrad Johnston,
Andrius Merkys,
Andrea Cepellotti,
Nicolas Mounet,
Nicola Marzari,
Boris Kozinsky,
Giovanni Pizzi
Abstract:
The ever-growing availability of computing power and the sustained development of advanced computational methods have contributed much to recent scientific progress. These developments present new challenges driven by the sheer amount of calculations and data to manage. Next-generation exascale supercomputers will harden these challenges, such that automated and scalable solutions become crucial.…
▽ More
The ever-growing availability of computing power and the sustained development of advanced computational methods have contributed much to recent scientific progress. These developments present new challenges driven by the sheer amount of calculations and data to manage. Next-generation exascale supercomputers will harden these challenges, such that automated and scalable solutions become crucial. In recent years, we have been developing AiiDA (http://www.aiida.net), a robust open-source high-throughput infrastructure addressing the challenges arising from the needs of automated workflow management and data provenance recording. Here, we introduce developments and capabilities required to reach sustained performance, with AiiDA supporting throughputs of tens of thousands processes/hour, while automatically preserving and storing the full data provenance in a relational database making it queryable and traversable, thus enabling high-performance data analytics. AiiDA's workflow language provides advanced automation, error handling features and a flexible plugin model to allow interfacing with any simulation software. The associated plugin registry enables seamless sharing of extensions, empowering a vibrant user community dedicated to making simulations more robust, user-friendly and reproducible.
△ Less
Submitted 24 March, 2020;
originally announced March 2020.
