Inverse design of crystals using generalized invertible crystallographic

representation

Zekun Ren 1 2 Juhwan Noh 3 Siyu Tian 1 2 Felipe Oviedo 4 Guangzong Xing 5 Qiaohao Liang 4 Armin Aberle 2
arXiv:2005.07609v1 [physics.comp-ph] 15 May 2020

Yi Liu 6 Qianxiao Li 7 Senthilnath Jayavelu 8 Kedar Hippalgaonkar 9 10 Yousung Jung 3 Tonio Buonassisi 1 4

Abstract thermoelectric materials.

Deep learning has fostered many novel applica-

tions in materials informatics. However, the in-
verse design of inorganic crystals, i.e. generating 1. Introduction
new crystal structure with targeted properties, re- Traditionally, the design of solid-state materials has heav-
mains a grand challenge. An important ingredient ily relied on human intuition and heuristic chemical rules.
for such generative models is an invertible rep- Considering the number of elements in the periodic table
resentation that accesses the full periodic table. and symmetries in a crystal, the materials space is rather
This is challenging due to limited data availability unexplored given only 2 × 105 solid materials have been
and the complexity of 3D periodic crystal struc- reported (Noh et al., 2019a). Historically, the rate of discov-
tures. In this paper, we present a generalized ery and development for conventional solid-state materials
invertible representation that encodes the crys- has been slow (Correa-Baena et al., 2018). Recently, this
tallographic information into the descriptors in process has been accelerated by high throughput screening
both real space and reciprocal space. Combining based on density functional theory (DFT) calculations. High
with a generative variational autoencoder (VAE), throughput screening has allowed the discovery of a signifi-
a wide range of crystallographic structures and cant number of crystals with functional properties (Brandt
chemistries with desired properties can be inverse- et al., 2015; Zhao et al., 2017; Sun et al., 2019; Noh et al.,
designed. We show that our VAE model predicts 2019b). Because of the heavy computational burden of DFT,
novel crystal structures that do not exist in the application of machine learning methods to accelerate DFT
training and test database (Materials Project) with calculations has seen initial success in screening solid-state
targeted formation energies and band gaps. We crystals (Xie & Grossman, 2018; Chen et al., 2019; Schtt
validate those predicted crystals by first-principles et al., 2018; Rupp et al., 2012; Faber et al., 2015; Huo &
calculations. Finally, to design solids with prac- Rupp, 2017; Behler, 2011; Bartk et al., 2013). However,
tical applications, we address the sparse label these works focus on regressing crystal representations to
problem by building a semi-supervised VAE and desired properties, and subsequently screening the existing
demonstrate its successful prediction of unique crystals in the database. To accelerate new material discov-
1
Singapore-MIT Alliance for Research and Technology ery, an inverse design framework of 3D crystals, mapping
SMART, Singapore 2 Solar Energy Research Institute of Singapore desired functionality to the crystalline structure, needs to be
(SERIS), National University of Singapore Singapore 3 Department developed (Zunger, 2018).
of Chemical and Biomolecular Engineering, Korea Advanced In-
stitute of Science and Technology (KAIST), Korea 4 Massachusetts One promising branch to design unique crystals is crys-
Institute of Technology, Cambridge, MA, USA 5 Okinawa Insti- tal structure prediction using global optimization (Oganov
tute of Science and Technology,Japan 6 Materials Genome Insti- et al., 2019). Various global optimization methods including
tute (MGI), Shanghai University , Shanghai, China 7 Department simulated annealing (Wille, 1986; Kirkpatrick, 1984; Schön
of Mathematics, National University of Singapore, Singapore
8
Institute for Infocomm Research, Agency for Science, Technol- & Jansen, 1996), minima hopping (Goedecker, 2004) and
ogy and Research (A*STAR), Singapore 9 Institute of Materials genetic algorithms (Wang et al., 2012; Glass et al., 2006)
Research and Engineering, Agency for Science, Technology and have been investigated. Those methods generally require an
Research (A*STAR), Singapore 10 Materials Science and Engineer- initial structural pool of specific chemical composition and
ing, Nanyang Technological University, Singapore. Correspon- thus are limited user’s initial choices. Another emerging
dence to: Zekun Ren <zekun@smart.mit.edu>, Tonio Buonassisi
<buonassisi@mit.edu>. branch is building probabilistic generative models to encode
the existing materials to a continuous latent space (Sanchez-
Lengeling & Aspuru-Guzik, 2018). By mapping the latent
 Inverse design of crystals using generalized invertible crystallographic representation

space to material properties, inverse design with a targeted tion of new materials is expensive, local perturbation is used
property can be realized. for new crystal generation.
Prior work on inverse design using generative models has We demonstrate three success cases to design new crystals
been extensively studied in molecules (Gmez-Bombarelli with desired properties that are not in the training database
et al., 2018; Liu et al., 2018; Jin et al., 2018; Samanta et al., (Materials Project). As we trained our model with DFT
2018; Assouel et al., 2018; Kusner et al., 2017; Polykovskiy calculations only, we believe performing independent DFT
et al., 2018). Those invertible molecular descriptors are calculations of new crystals is an adequate and actionable
either represented in a 1D string (SMILES representation) validation procedure. In the first demonstration, we gen-
or 2D graphs. However, they are not suitable to describe a erated 97 unique ternary crystals with various targets of
3D periodic lattice, such as the one in a crystal. Building formation energy per atom (Ef ). We perform additional
generative models for crystals is challenging due to lim- DFT calculations to validate those crystals and find 27 crys-
ited available data and its 3D periodic structure (Noh et al., tals have targeted Ef within -0.1 eV/atom error. In the
2020). Several specialized inverse design tools for solid- second demonstration, we generate 16 new crystals with
state materials have been proposed, with limited structural targeted bandgap (EG ) and Ef . 6 out of 16 generated crys-
and chemical range (Zhao et al., 2017; Nouira et al., 2018; tals are validated to have EG close to the target within the
Noh et al., 2019a; Kim et al., 2020). Noh et al. featurizes a model prediction accuracy while the majority of the unique
single unit cell in a 3D grid and builds a hierarchical image- crystals meet the Ef target. Lastly, we extend our gener-
based VAE to generate new vanadium oxides. Beyond au- ative model to predict good thermoelectric crystals. An
toencoders, Generative Adversarial Neural Networks have additional challenge for such a demonstration is the small
been investigated to inverse design new crystals with differ- dataset. The application-relevant properties such as effective
ent elemental compositions (Sawada et al., 2019; Dan et al., mass, Seebeck coefficient and power factor are computa-
2019; Nouira et al., 2018; Kim et al., 2020). Kim and Noh tionally expensive to compute with DFT or require different
et al. represent the crystal as a set of atomic coordinates and computation platforms (Madsen & Singh, 2006), and ex-
cell parameters. The crystal generation in the above inverse isting data is limited. We tackle this data scarcity problem
design methods is achieved by selecting a subset of elements by jointly training a large unlabelled dataset with a small
from the periodic table. Moreover, the periodic nature of the labelled subset via a semi-supervised VAE. We show that
crystals is not reflected in the current crystal representations. 2 out of a total 27 predicted unique crystals have state-of-
To account for crystal periodicity, Hoffmann et al. directly the-art power factor validated using Boltzmann Transport
encode the repeated unit cells in a 3D cube to train a VAE calculations. In summary, in this work:
(Hoffmann et al., 2019). Training the model using a 3D
representation of the repeating unit can be computationally 1. We present a generalized invertible crystallographic
inefficient, and limited to specific crystal symmetries. representation for 3D crystals.
In this work, we propose an efficient and invertible 2D repre-
2. We build a VAE that allows the generation of new
sentation for 3D crystals that accounts for crystal periodicity
crystals with targeted materials properties.
and accesses the full periodic table. The motion of electrons
for crystals in Quantum Mechanics is generally described in 3. We demonstrate three successful applications of in-
both the real space e.g. surface energy, and the momentum verse design of unique functional crystals.
space (k space) e.g. E-k diagram. We embed this domain
knowledge to construct a crystallographic representation
that can be used to train a generative model by combin-
2. Invertible crystallographic representation
ing the atomic and structural properties in both real space for crystals
and momentum space (also called k space, Fourier space Atoms in a crystal are arranged in a periodic pattern. Ac-
or frequency space). We build a VAE (Kingma & Welling, cording to Bloch’s theorem in solid-state physics, the wave
2013) using these 2D crystallographic representations with function can be expressed as the product of a plane wave
the latent space organized according to different material and a function that has the same periodicity as the crystal
properties. This material-semantic latent representation al- (Martn-Palma et al., 2006). The energy of electrons in the
lows the generation of new crystals with targeted materials crystal is both studied in the real space (energy surface) and
properties. We investigate different sampling strategies to momentum space (energy dispersion relation). We embed
generate new crystals from the latent space. The compari- such physics knowledge to construct the crystallographic
son is assessed by the reconstruction error in rediscovered descriptors for the 3D periodic crystal. Figure 1 shows the
crystals from the test dataset. We find that applying a local schematic of our crystallographic representation. Firstly, we
perturbation to existing crystals yields the lowest crystal represent the unit cell (the basic building block of the crystal)
reconstruction error in rediscovered crystals. Given evalua- using the length and angles of the three fundamental trans-
 Inverse design of crystals using generalized invertible crystallographic representation

Figure 1. Invertible crystal representation using both real space and Fourier space crystal properties

lation vectors (lattice vectors) and all atomic coordinates matrix, the basis matrix and the element matrix.
in the real space. Then, we introduce Fourier-transformed
crystal properties (FTCP) to capture the elemental and struc- 2.2. Momentum space representation:
tural information in the momentum space. We construct the
crystallographic representations by concatenating both real The most common momentum space representation of a
space and momentum space crystal representations. crystal is its diffraction pattern. X-ray crystallography is
the primary method to study periodic crystals (Ladd et al.,
1985). Modified diffraction images for periodic crystals
2.1. Real space representation
have been shown to classify its geometry accurately (Ziletti
To represent the crystals in the real space, we extract the text et al., 2018). We enrich the information in diffraction im-
information in the crystallographic information file (CIF) ages in the momentum space by projecting one-hot encoded
of 3D crystals. The text primarily corresponds to two ma- elemental properties Z to different crystal planes (h, k, l)
trices: cell matrix (the length and angle of three translation using a discrete Fourier transform.
lattice vectors), with size (2, 3), and basis matrix (all atom
coordinates in the unit cell) with size Nsites . Nsites is the N
X
upper bound of the number of atoms in the unit cell. We F(h, k, l) = Zi ∗ e(−j2π(hx+ky+lz)) (1)
use zero padding in crystals with less atoms. In addition, i=1
we bin and one-hot encode each element in the unit cell
as a vector Z which has K features which are the number where i is the atom in the unit cell, x, y, z is the atom co-
of atomic properties such as group number, electronegativ- ordinates in real space and h, k, l is the Miller index of a
ity, first ionization energy, etc. (Xie & Grossman, 2018). crystal plane (a point in momentum space). F(h, k, l) is
For featurizing the elements in ternary compounds, we con- the Fourier transformed crystal properties for plane (h, k, l),
catenate Z into a (K, 3) matrix. Hereafter, we construct which is the projection of atomic properties in one plane in
the real space representation by further concatenating the the crystal lattice and a point in the reciprocal lattice. As
atomic vector Z of each element, the cell matrix, the ba- the unit cells are defined as the repeating unit in the crys-
sis matrix and the one-hot encoded element matrix. The tal, there exist an infinite number of planes. We limit the
concatenated real space representation matrix is V(x, y, z) number of planes by applying a filter of the sum of (h, k, l)
with size (m, 3) where m is the sum of the length of the cell less than 4. The Fourier-transformed properties F(h, k, l)
are arranged according to orders of their Miller index and
 Inverse design of crystals using generalized invertible crystallographic representation

become a (c, K) matrix where c is the number of planes gradient by material property values (Figure S1) and allows
with the sum of (h, k, l) less than 4 and K is the number to generate new crystals with targeted material properties.
of atomic features. Lastly, we concatenate the real space
R(z) = σ(g(z) · h(z)) (2)
crystal representation (V with size (m, 3)) and momentum
space representation (F with size (c, K)) with zero padding.
3.3. Decoder
The combined representation has a matrix size of (m, K).
The decoder mirrors the encoder using transposed convo-
3. Generative model and inverse design lutional layers. Lengths, angles of the three translational
vectors and coordinates of atoms in the unit cell are part of
Hereafter, we build a generative model to encode and de- the decoded crystallographic descriptors. Therefore, the 3D
code the concatenated crystallographic descriptors shown structure of the unit cell can be fully reconstructed.
in Figure 2. The detailed structure of the neural network is
shown in Supplementary S6. 3.4. Loss
Training of the model is done by optimization of the
3.1. Encoder
tractable evidence lower bound (ELBO) of VAE and re-
We treat the concatenated crystallographic representation as gression loss for material properties prediction. The overall
a 1D signals with K channels. The signal in each channel loss is
represents the atomic property projection such as electroneg- L = Lreconstruct βLKL + λLreg (3)
ativity, ionization energy, etc., in both real and momentum
space. We use the 1D convolutional neural network (CNN) where Lreconstruct is the reconstruction loss, LKL is the
as the sequence of atoms and crystal planes are contained distribution loss, Lreg is the regression loss, β and λ are
in the 1D signal. The 1D CNN in this work is inspired by the coefficients. To learn a disentangled representation, we
point clouds used for 3D image classification (Kim et al., allow heavy penalization of the latent distribution (β > 1)
2020; Qi et al., 2017). There is a spatial 1D relationship (Higgins et al., 2017).
in our crystal representation. Along this spatial axis, our
Fourier space descriptor is arranged according to symmetry
points F(h, k, l) that are universal in describing the elec-
tronic band structure. Because of the pooling operations in
CNNs, the arrangement of those geometry points becomes
translationally invariant (Gu et al., 2018; Liu et al., 2019).
However, there is no universal order for the real space de-
scriptor and thus the real space representation has neither
translational or rotational invariance. Kim et al (Kim et al.,
2020) have shown that performing data augmentation for
the real space descriptors will lead to a more balanced gen- Figure 2. VAE architecture using invertible crystal representations
eration of crystal structures for Mg-Mn-O systems. Given of both real space and Fourier space crystal properties.
a much larger chemical range is considered in this study,
data augmentation is not feasible. Instead, we encode 1D
signals with K channels into a normal distribution with a 3.5. Inverse design
diagonal covariance matrix using 1D CNN. The latent vec-
tor is sampled from those distributions and is regularized As the latent space of the VAE is mapped to material proper-
by the KullbackLeibler (KL) divergence between the latent ties using the regression model, the latent space or reduced
distribution and the standard Gaussian prior. material space becomes an organized and continuous crys-
tal representation with different material properties (Figure
3.2. Latent to material property regression S1). Inverse design of new crystals leverages this structured
latent space by sampling in regions with desired properties.
In addition to constraining the latent distribution to the mul- To avoid the possibility that generated crystals with minor
tivariant normal distribution with diagonal covariance, we structural change is considered as a new crystal, we only con-
train a gated regression model to map the latent space to sider generated crystals whose chemical formula changes
material properties (Gmez-Bombarelli et al., 2018). The as unique samples. To ensure the correctness of inverse-
regression is shown in Eq.2 where g and h are neural net- designed crystals, we investigate three different sampling
works and σ(.) is the sigmoid function, z is the encoded strategies to the latent space. The comparison is assessed by
latent. We use the L2 loss between R(z) and the materials the crystal reconstruction errors for the crystals in the test
properties. The regression shapes the latent space with a dataset.
 Inverse design of crystals using generalized invertible crystallographic representation

Table 1. Validation of VAE generated crystals using three sampling methods.

RECONSTRUCTION METRIC LP S LERP R ANDOM

R EDISCOVERY PERCENTAGE [%] 27.1 35.9 50.9
L ATTICE CONSTANT PMAE [%] 14.4 21.7 27.3
L ATTICE ANGLES PMAE [%] 9.75 17.1 24.5
ATOM FRACTIONAL COORDINATES MAE[a.u.] 0.14 0.15 0.16

4. Experiments convert the decoded elemental representation to discrete

labels (chemical formula). The structural information con-
4.1. Dataset tains scalar values of the atomic coordinates and lattice
To generate crystals that access the chemistries in the full parameters, and is directly taken from the decoded repre-
periodic table, we use the Materials Project dataset (Jain sentation. Thus, despite every sampled latent representation
et al., 2013). In the first experiment, we consider ternary decoding into a mathematically new crystal representation,
compounds with the number of atoms in the unit cell (Nsites ) many of those have the same chemical formula with minor
is less than or equal to 20 and energy above hull (Ehull ) structural changes compared to the crystals in the training
less than 50 meV in the database. The number of crystals dataset. Those crystals that have the same chemical formula
selected is reduced to 24,785. We ensure that there is no to the training dataset are excluded and not counted as new
identical structure (same composition and same geometry) crystals. For Lp and Slerp, known crystals with targeted
in the dataset. We randomly split those unique structures material properties are used as initial guesses, Lp applies
into 80% for training and 20% for testing. The VAE model random local perturbation to those crystals latent represen-
is trained with 80% of the data and validated with 20% tation and Slerp performs interpolation between each pair
of the data. DFT calculated Ef and EG of those crystals of crystals. Random sampling is done from Gaussian distri-
are taken as the output of the material property prediction bution with the samples mean and variance. We compare
model. the three sampling methods by attempting to rediscover the
4,957 crystals in the test dataset. Discovery of new crystals
4.2. Generating new crystals in the test dataset also validates our generative model. We
use two sets of metrics, percentage of crystals in the test
We featurize the 24,785 .cif files into the concatenated 2D dataset that can be rediscovered (rediscovery percentage)
crystallographic representations following the procedure and reconstruction error of the rediscovered crystals to the
shown in Figure 1. We then train VAE using the loss de- test crystals (rediscovery error) to evaluate the sampling
scribed in Eq.3. We reconstruct the crystals from the trained strategy.
latent space using the decoder, and then compare key lattice
parameters with the original crystal. For the test dataset, we Table 1 summarizes the rediscovery percentage and error of
achieve 99.1% accuracy for crystal chemical formula predic- the three sampling methods. For a fair comparison, we use
tion, percentage mean absolute error (PMAE) of 7.41% for the same number of crystals (500) in the test dataset for all
the length of lattice constant, PMAE of 3.99% for the lattice methods in calculating rediscovered crystals errors. Sam-
angles. PMAE for atomic coordinates results in unreason- pling based on local perturbation of latent representations
able numbers as there are many zeros in the coordinates. We has the lowest reconstruction error for rediscovered crystals
use Mean absolute error (MAE) for the atomic coordinates but only recovers 27% of the test crystals. We find that new
and 0.001 error is reported. crystals obtained by local perturbation sampling mainly ex-
periences elemental substitution compared to its root crystal.
Moving beyond reconstruction, we investigate the genera- This agrees well with the prevailing design principles for
tion of unique crystals that are not in the training dataset new crystalline materials, which make use of manual substi-
by sampling out-of-the-distribution latent space. Three dif- tution of certain elements in the unit cell (Zhao et al., 2017).
ferent sampling methods: local perturbation (Lp), spherical Slerp interpolation and random sampling generate samples
linear interpolation (Slerp) and random sampling (Random) that experience both elemental substitution and significant
are implemented to sample points that are different from structural change, however, the crystal reconstruction error
training dataset in the latent space. in the rediscovered materials is much higher. Given the long
As our crystal representation contains both elemental and development cycle for solid-state materials (Correa-Baena
structural information, we invert those two parts separately et al., 2018), it is of critical importance to have an accurate
to recover the full 3D crystal. The elemental representation initial guess. Even the DFT calculations to validate VAE
is one-hot encoded, and we perform additional argmax to generated crystals are computationally expensive. DFT vali-
dations conducted in this study takes around 1-2 hours per
 Inverse design of crystals using generalized invertible crystallographic representation

transformed crystal representation is not considered. The

Table 2. ablation study of crystal representations for forward pre-
improvement in reconstruction error of the lengths, angles
diction.
of the three basis vector and atomic coordinates by incor-
porating the crystal representations in the momentum space
I NPUT Ef MAE(eV /atom) EG MAE(eV ) Table 3 shows featurizing the periodic crystal in Fourier
R EAL SPACE space is also important for training the generative model.
REPRESENTATIONS 0.092 0.311
M OMENTUM SPACE
REPRESENTATIONS 0.090 0.267 5. Demonstrations: inverse design of crystals
C ONCATENATED with targeted properties
REPRESENTATIONS 0.063 0.231
CGCNN 0.062 0.254 The goal of inverse design is to generate new crystals, with
desired properties that are not in the database. In this study,
we showcase three demonstrations. Firstly, we generate
single crystal with 16 CPU cores. However, DFT calcula- new crystals with targeted Ef values. We perturb the latent
tions can require significant time (in the order of weeks) to representation of 100 existing ternary crystals with Ef close
setup and optimize. More complex calculations, such as to the target. 17 unique ternary crystals with new chemical
defect calculations, can take days to complete (Atilgan & formula that do not exist in the Materials Project database
Hu, 2018). Given all the above considerations, we choose are generated. A few of those unique crystal structures are
local perturbation sampling which gives us the lowest redis- shown in Figure 3. There are 8 different crystal structures
covery error to generate new crystals with targeted material with more than 30 different elements in those 17 crystals
properties. (Figure S2). This shows the VAE can generate crystals
accessing a wide range of structures and chemistries. We
4.3. Ablation study perform independent DFT calculations to optimize the ge-
ometry and compute the Ef of the generated crystals. We
As previous work on crystal representation mainly focuses notice that the generated atomic coordinates changes after
on material property prediction or inverse design with lim- DFT relaxation indicating that the atoms of generated crys-
ited chemistry, there is no benchmark in the inverse design tals are not at the minimum energy surface. Post-processing
of crystals with desired properties. Herein, we perform the the .cif files with DFT relaxation is required and the same
ablation study on our crystal representation and compare a approach is reported in (Noh et al., 2019a; Kim et al., 2020).
property prediction error with the state-of-the-art graph rep- This is because all the atoms in our training data (Materials
resentation. The forward property prediction only consists Project) are relaxed and our generative model is not trained
of the encoder which embeds the crystallographic descrip- to find structure close to the local minimum of the energy
tors in the invertible latent representation and the regression surface. Further development of the model is needed to
model that maps the latent space to the material properties. learn the relaxation of atoms in generated crystals. Figure
The forward model is trained with the same ternary crys- 4 (a) shows that with additional DFT relaxation calcula-
tals dataset that is used for training the VAE. The results in tions, 7 out of 17 generated crystals have Ef close to the
Table 2 show that the concatenated crystallographic repre- target (-0.5eV) within 0.1 eV accuracy. The black dashed
sentations have an MAE of 0.063 eV/atom in predicting the line is the target of Ef . In addition, we generate another
formation energy per atom and MAE of 0.231 eV in predict- 80 ternary compounds with staggered Ef values (-0.3eV,
ing the bandgap. The error from this model is comparable -0.6eV and -0.7eV). 20 out of those 80 compounds have Ef
to the intrinsic error from DFT (Kirklin et al., 2015). Only close to the target within 0.1 eV accuracy. The results are
considering representation in real/momentum space results shown in Figure S3. All the generated 97 compounds in this
in higher prediction error indicating the importance of incor- demonstration have a unique chemical formula that does
porating both real space and Fourier space descriptors. The not exist in the Materials Project database. In addition, we
graph-based representation Crystal Graph Convolutional investigate the structural uniqueness and ensure not all com-
Neural Networks (CGCNN) (Xie & Grossman, 2018) have pounds can be discovered by manual elemental substitution.
similar performance compared to our crystallographic repre- We use a dissimilarity value that is assessed on the basis
sentations when trained using the ternary dataset, however, of local coordination information from all sites in the two
our crystal representation is invertible and allows for train- structures (Zimmermann & Jain, 2020), the crystal structure
ing generative models. is the same if the dissimilarity is zero. Figure S4 shows the
Furthermore, we compare the reconstruction error between minimum dissimilarity value compared to every crystal in
the VAE model trained using both real and momentum space the training dataset. The median dissimilarity value is 0.64
representation and model trained only with real space repre- where a number of compounds have a dissimilarity value
sentations. The error significantly increases when Fourier above 0.75 indicating the structural uniqueness of those
 Inverse design of crystals using generalized invertible crystallographic representation

Table 3. Ablation study of crystal representations for VAE reconstruction.

M ETRICS R EAL SPACE REPRESENTATION CONCATENATED REPRESENTATION

E LEMENT
C ATEGORICAL
ACCURACY [%] 99.1 97.9
L ATTICE CONSTANT
PMAE[%] 7.41 12.5
L ATTICE ANGLES
PMAE[%] 3.99 6.75
ATOM FRACTIONAL
COORDINATES
MAE [ A . U .] 0.001 0.002

Figure 3. Examples of VAE-generated 3D crystals with targeted Ef =-0.5 eV after DFT relaxation. There are 8 different crystal structures
with more than 30 different elements in those 17 crystals (Figure S2). This shows the VAE can generate crystals accessing a wide range of
structures and chemistries.

generated compounds. crystals satisfies Ef <1.5 eV and EG =1.5 eV is less than 1%

justifying the non-triviality of designing crystals of those
In the second demonstration, we extend the design criteria to
targeted property values.
multi-objective by producing ternary and quaternary crystals
for potential photovoltaic applications. The design criteria
are to have a bandgap equal to 1.5 eV and formation energy 5.1. Designing thermoelectric materials using
per atom smaller than -1.5 eV. We locally perturb the latent semi-supervised VAE
representation of 100 existing ternary crystals and generate A grand challenge in the field of thermoelectrics, where
16 unique crystals. Figure 4(b) shows that the majority of materials convert heat to electrical energy and vice-versa,
the generated crystals satisfies the formation energy target is to perform inverse design with multi-objectives, e.g. the
(Ef <-1.5 eV) and the median bandgap value (EG ) of those compound must be stable and has high power factor with the
crystals is around 1.5 eV. However, the variance of EG is desired bandgap. This is a difficult task because, in addition
significantly larger. We postulate that this is due to large to the ground state properties of the inorganic solids, we
training error for EG prediction (Table 2). Six out of 16 also have to learn the key features that tell us which solids
generated crystals have EG close to the 1.5 eV target within have good functional properties. The electronic power fac-
the bandgap forward model prediction accuracy. Figure S5 tor is a non-linear mathematical function that depends on
shows the histogram of Ef and EG values of all crystals in the electronic band structure as well as charge scattering
the training dataset. The probability of randomly sampling parameters. Generating crystals for thermoelectric applica-
 Inverse design of crystals using generalized invertible crystallographic representation

Figure 4. DFT calculated material properties for inverse designed unique crystals (a) Case 1: ternary crystals with a targeted Ef =-0.5 eV.
The black dashed line is the target of Ef . 7 out of 17 generated crystals have Ef close to the target (-0.5eV) within 0.1 eV accuracy (b)
Case 2: ternary and quarternary crystals with targeted Ef <-1.5 eV and EG =1.5 eV. Most of the generated crystals satisfies Ef <-1.5 eV
requirement and the median EG of those crystals is around 1.5 eV

Figure 5. (a) and (b) show the predicted ternary crystals generated by our inverse design algorithm. These materials dont exist in any
database. Their power factor values (divided by the relaxation time) are plotted as a function of doping level at room temperature, both
for n- and p-doping. (c) shows the comparison to state-of-the-art high-performance thermoelectric material, cubic germanium telluride
(GeTe), with comparable results.

tions is much more challenging because of data scarcity: we properties (EG , Ehull ). Only 4,284 crystal have calculated
use the database from (Ricci et al., 2017) where the con- power factor labels. We tackle this limited data problem
stant relaxation time approximation under the Boltzmann by jointly training the VAE and mapping its latent space
Transport Equations is used to calculate the thermoelectric to ground-state property labels, and a small labelled subset
properties of inorganic crystals from Materials Project. The with thermoelectric property labels. The training loss of the
final dataset has 34,784 crystal structures with ground-state semi-supervised VAE is shown in Eq.4. We demonstrate
 Inverse design of crystals using generalized invertible crystallographic representation

that our method can generate crystal structures that meet 7. Conclusions
those objectives. Here, the doping level and temperature are
treated as user inputs. The ability to dope the material to We present a generalized invertible crystallographic repre-
the desired carrier concentration is beyond the scope of our sentation for 3D periodic crystals. An ablation study of
approach. our crystallographic representation suggests an advantage
of using both real space crystal representations and momen-
tum space representations for material property prediction
and new crystal generation. We propose an inverse design
paradigm for generating new solid-state materials with tar-
X X X X geted material properties using VAE. We demonstrated three
L= Lrec β LKL + λ1 Lreg1 + λ2 Lreg2 unique applications of our model by generating new chem-
n∈P n∈P n∈P m∈Q istry formulas that are not in the database and validate them
(4) using DFT and Boltzmann Transport Calculations. Our fu-
P P
where n∈P Lrec , n∈P LKL is the reconstruction and ture work will be experimentally synthesizing the predicted
KullbackLeibler divergence for all crystals in the data, crystals, and bridging the synthesizability gap between the-
P
L is the regression loss for ground oretically generated crystals and experimental crystals by
n∈P reg1 Pstate material using semi-supervised learning of experimental datasets.
properties (EG ,Ef ,Ehull ) of all crystals and m∈Q Lreg2 is
the regression loss for 4,284 crystals that have calculated
power factor labels. 8. Acknowledgement
Based on domain expertise, we select 100 ternary and qua- We acknowledge Shijing Sun from MIT and Vladan Ste-
ternary crystals with the following section criteria: high vanovic from Colorado School of Mines for helpful dis-
power factors, energy above convex hull less than 0.03 eV, cussions. This research is supported by the National Re-
and bandgap between 0.3 and 1.5 eV. Applying local pertur- search Foundation, Prime Ministers Office, Singapore un-
bation to those compounds generates 27 unique crystals that der its Campus for Research Excellence and Technologi-
are not in the database. To test the quality of the crystals cal Enterprise (CREATE) program through the Singapore
generated, we performed DFT calculations of the predicted Massachusetts Institute of Technology (MIT) Alliance for
materials. We also performed thermal relaxation to obtain Research and Technologys Low Energy Electronic Sys-
the final atomic coordinates and then performed Boltztrap tems research program. J.N. and Y.J. acknowledge sup-
calculations (Madsen & Singh, 2006). We find that 2 out of port from NRF Korea (NRF-2017R1A2B3010176). Y.L.
the 27 crystals show state-of-the-art power factors, compa- is supported by the National Key Research and Develop-
rable to the best thermoelectric materials. The results are ment Program of China (Grant Nos. 2017YFB0702901
shown in Figure 5, as a function of doping concentration. and 2017YFB0701502) and the National Nature Science
Foundation of China (Grant No. 91641128). Q.L., S.J. and
6. Future work K.H. acknowledge funding from the Accelerated Materials
Development for Manufacturing Program at A*STAR via
Despite many recent studies reporting great results through the AME Programmatic Fund by the Agency for Science,
theoretical screening using forward prediction and inverse Technology and Research under Grant No. A1898b0043.
design using generative methods in the realm of computa-
tions, the synthesizability of these newly found materials
still poses a great challenge to the community. Therefore, References
while our inverse design tool can discover new materials Assouel, R., Ahmed, M., Segler, M. H., Saffari, A., and
unprecedented in the Materials Project with targeted prop- Bengio, Y. Defactor: Differentiable edge factorization-
erties, those structures are lacking real experimental vali- based probabilistic graph generation. arXiv preprint
dation. This is because of the fundamental limitations of arXiv:1811.09766, 2018.
DFT, or any other first-principles calculation technique, to
fully capture real materials. As we trained our model with Atilgan, E. and Hu, J. First-principle-based computational
DFT calculations only, performing independent DFT cal- doping of srtio3 using combinatorial genetic algorithms.
culations (outside of the training database, with different Bulletin of Materials Science, 41(1):1, 2018.
DFT functionals) is an adequate and actionable approach Bartk, A. P., Kondor, R., and Csnyi, G. On represent-
to validate our predicted crystal structures. Validation us- ing chemical environments. Physical Review B, 87(18):
ing experimental crystals will be future work, as it requires 184115, 2013.
longer development time frames, including physical process
tuning, synthesizability optimization, and characterization, Behler, J. Atom-centered symmetry functions for construct-
etc. ing high-dimensional neural network potentials. The
 Inverse design of crystals using generalized invertible crystallographic representation

Journal of chemical physics, 134(7):074106, 2011. ISSN Hoffmann, J., Maestrati, L., Sawada, Y., Tang, J., Sellier,
0021-9606. J. M., and Bengio, Y. Data-driven approach to encod-
ing and decoding 3-d crystal structures. arXiv preprint
Brandt, R. E., Stevanović, V., Ginley, D. S., and Buonas- arXiv:1909.00949, 2019.
sisi, T. Identifying defect-tolerant semiconductors with
high minority-carrier lifetimes: beyond hybrid lead halide Huo, H. and Rupp, M. Unified representation for ma-
perovskites. Mrs Communications, 5(2):265–275, 2015. chine learning of molecules and crystals. arXiv preprint
arXiv:1704.06439, 13754, 2017.
Chen, C., Ye, W., Zuo, Y., Zheng, C., and Ong, S. P. Graph
networks as a universal machine learning framework for Jain, A., Ong, S. P., Hautier, G., Chen, W., Richards, W. D.,
molecules and crystals. Chemistry of Materials, 31(9): Dacek, S., Cholia, S., Gunter, D., Skinner, D., and Ceder,
3564–3572, 2019. ISSN 0897-4756. G. J. A. M. Commentary: The materials project: A materi-
als genome approach to accelerating materials innovation.
Correa-Baena, J.-P., Hippalgaonkar, K., van Duren, J., Jaf- 1(1):011002, 2013. ISSN 2166-532X.
fer, S., Chandrasekhar, V. R., Stevanovic, V., Wadia, C.,
Jin, W., Barzilay, R., and Jaakkola, T. Junction tree varia-
Guha, S., and Buonassisi, T. Accelerating materials de-
tional autoencoder for molecular graph generation. arXiv
velopment via automation, machine learning, and high-
preprint arXiv:1802.04364, 2018.
performance computing. Joule, 2018. ISSN 2542-4351.
Kim, S., Noh, J., Gu, G. H., Aspuru-Guzik, A., and Jung,
Dan, Y., Zhao, Y., Li, X., Li, S., Hu, M., and Hu, J. Genera- Y. Generative adversarial networks for crystal structure
tive adversarial networks (gan) based efficient sampling prediction. arXiv preprint arXiv:2004.01396, 2020.
of chemical space for inverse design of inorganic materi-
als. arXiv preprint arXiv:1911.05020, 2019. Kingma, D. P. and Welling, M. Auto-encoding variational
bayes. arXiv preprint arXiv:1312.6114, 2013.
Faber, F., Lindmaa, A., von Lilienfeld, O. A., and Armiento,
R. Crystal structure representations for machine learning Kirklin, S., Saal, J. E., Meredig, B., Thompson, A., Doak,
models of formation energies. International Journal of J. W., Aykol, M., Rhl, S., and Wolverton, C. The open
Quantum Chemistry, 115(16):1094–1101, 2015. ISSN quantum materials database (oqmd): assessing the ac-
0020-7608. curacy of dft formation energies. npj Computational
Materials, 1(1):1–15, 2015. ISSN 2057-3960.
Glass, C. W., Oganov, A. R., and Hansen, N. Uspexevo-
lutionary crystal structure prediction. Computer physics Kirkpatrick, S. Optimization by simulated annealing: Quan-
communications, 175(11-12):713–720, 2006. titative studies. Journal of statistical physics, 34(5-6):
975–986, 1984.
Goedecker, S. Minima hopping: An efficient search method
for the global minimum of the potential energy surface Kusner, M. J., Paige, B., and Hernndez-Lobato, J. M. Gram-
of complex molecular systems. The Journal of chemical mar variational autoencoder. In Proceedings of the 34th
physics, 120(21):9911–9917, 2004. International Conference on Machine Learning-Volume
70, pp. 1945–1954. JMLR. org, 2017.
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B.,
Ladd, M. F. C., Palmer, R. A., and Palmer, R. A. Structure
Liu, T., Wang, X., Wang, G., and Cai, J. Recent advances
determination by X-ray crystallography. Springer, 1985.
in convolutional neural networks. Pattern Recognition,
77:354–377, 2018. ISSN 0031-3203. Liu, Q., Allamanis, M., Brockschmidt, M., and Gaunt, A.
Constrained graph variational autoencoders for molecule
Gmez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernndez- design. In Advances in neural information processing
Lobato, J. M., Snchez-Lengeling, B., Sheberla, D., systems, pp. 7795–7804, 2018.
Aguilera-Iparraguirre, J., Hirzel, T. D., Adams, R. P., and
Aspuru-Guzik, A. Automatic chemical design using a Liu, W., Sun, J., Li, W., Hu, T., and Wang, P. Deep learning
data-driven continuous representation of molecules. ACS on point clouds and its application: A survey. Sensors,
central science, 4(2):268–276, 2018. ISSN 2374-7943. 19(19):4188, 2019.

Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Madsen, G. K. and Singh, D. J. Boltztrap. a code for cal-
Botvinick, M., Mohamed, S., and Lerchner, A. beta- culating band-structure dependent quantities. Computer
vae: Learning basic visual concepts with a constrained Physics Communications, 175(1):67–71, 2006. ISSN
variational framework. Iclr, 2(5):6, 2017. 0010-4655.
 Inverse design of crystals using generalized invertible crystallographic representation

Martn-Palma, R. J., Martnez-Duart, J., and Agull-Rueda, F. models for matter engineering. Science, 361(6400):360–
Nanotechnology for microelectronics and optoelectronics. 365, 2018.
Elsevier, 2006. ISBN 0080456952.
Sawada, Y., Morikawa, K., and Fujii, M. Study of deep
Noh, J., Kim, J., Stein, H. S., Sanchez-Lengeling, B., Gre- generative models for inorganic chemical compositions.
goire, J. M., Aspuru-Guzik, A., and Jung, Y. Inverse arXiv preprint arXiv:1910.11499, 2019.
design of solid-state materials via a continuous represen-
tation. Matter, 1(5):1370–1384, 2019a. ISSN 2590-2385. Schön, J. C. and Jansen, M. First step towards planning
of syntheses in solid-state chemistry: determination of
Noh, J., Kim, S., ho Gu, G., Shinde, A., Zhou, L., Gre- promising structure candidates by global optimization.
goire, J. M., and Jung, Y. Unveiling new stable man- Angewandte Chemie International Edition in English, 35
ganese based photoanode materials via theoretical high- (12):1286–1304, 1996.
throughput screening and experiments. Chemical Com-
munications, 55(89):13418–13421, 2019b. Schtt, K. T., Sauceda, H. E., Kindermans, P.-J., Tkatchenko,
A., and Mller, K.-R. Schneta deep learning architecture
Noh, J., Gu, G. H., Kim, S., and Jung, Y. Machine-enabled for molecules and materials. The Journal of Chemical
inverse design of inorganic solid materials: Promises and Physics, 148(24):241722, 2018. ISSN 0021-9606.
challenges. Chemical Science, 2020.
Sun, S., Hartono, N. T., Ren, Z. D., Oviedo, F., Buscemi,
Nouira, A., Sokolovska, N., and Crivello, J.-C. Crys- A. M., Layurova, M., Chen, D. X., Ogunfunmi, T., Thapa,
talgan: learning to discover crystallographic structures J., Ramasamy, S., et al. Accelerated development of
with generative adversarial networks. arXiv preprint perovskite-inspired materials via high-throughput synthe-
arXiv:1810.11203, 2018. sis and machine-learning diagnosis. Joule, 3(6):1437–
1451, 2019.
Oganov, A. R., Pickard, C. J., Zhu, Q., and Needs, R. J.
Structure prediction drives materials discovery. Nature Wang, Y., Lv, J., Zhu, L., and Ma, Y. Calypso: A method
Reviews Materials, 4(5):331–348, 2019. for crystal structure prediction. Computer Physics Com-
munications, 183(10):2063–2070, 2012.
Polykovskiy, D., Zhebrak, A., Vetrov, D., Ivanenkov, Y.,
Aladinskiy, V., Mamoshina, P., Bozdaganyan, M., Aliper, Wille, L. Searching potential energy surfaces by simulated
A., Zhavoronkov, A., and Kadurin, A. Entangled condi- annealing. Nature, 324(6092):46–48, 1986.
tional adversarial autoencoder for de novo drug discov-
ery. Molecular pharmaceutics, 15(10):4398–4405, 2018. Xie, T. and Grossman, J. C. Crystal graph convolutional
ISSN 1543-8384. neural networks for an accurate and interpretable predic-
tion of material properties. Physical review letters, 120
Qi, C. R., Su, H., Mo, K., and Guibas, L. J. Pointnet: Deep (14):145301, 2018.
learning on point sets for 3d classification and segmenta-
tion. In Proceedings of the IEEE conference on computer Zhao, X.-G., Yang, J.-H., Fu, Y., Yang, D., Xu, Q., Yu, L.,
vision and pattern recognition, pp. 652–660, 2017. Wei, S.-H., and Zhang, L. Design of lead-free inorganic
halide perovskites for solar cells via cation-transmutation.
Ricci, F., Chen, W., Aydemir, U., Snyder, G. J., Rignanese, Journal of the American Chemical Society, 139(7):2630–
G.-M., Jain, A., and Hautier, G. An ab initio electronic 2638, 2017. ISSN 0002-7863.
transport database for inorganic materials. Scientific data,
4:170085, 2017. ISSN 2052-4463. Ziletti, A., Kumar, D., Scheffler, M., and Ghiringhelli, L. M.
Insightful classification of crystal structures using deep
Rupp, M., Tkatchenko, A., Mller, K.-R., and Von Lilienfeld, learning. Nature communications, 9(1):2775, 2018. ISSN
O. A. Fast and accurate modeling of molecular atom- 2041-1723.
ization energies with machine learning. Physical review
letters, 108(5):058301, 2012. Zimmermann, N. E. and Jain, A. Local structure order
parameters and site fingerprints for quantification of co-
Samanta, B., Abir, D., Jana, G., Chattaraj, P. K., Gan- ordination environment and crystal structure similarity.
guly, N., and Rodriguez, M. G. Nevae: A deep gen- RSC Advances, 10(10):6063–6081, 2020.
erative model for molecular graphs. In Proceedings of the
AAAI Conference on Artificial Intelligence, volume 33, Zunger, A. Inverse design in search of materials with target
pp. 1110–1117, 2018. ISBN 2374-3468. functionalities. Nature Reviews Chemistry, 2(4):1–16,
2018.
Sanchez-Lengeling, B. and Aspuru-Guzik, A. Inverse
molecular design using machine learning: Generative