AlphaCrystal-II: Distance matrix based crystal structure prediction using deep learning

Abstract

Computational prediction of stable crystal structures has a profound impact on the large-scale discovery of novel functional materials. However, predicting the crystal structure solely from a material’s composition or formula is a promising yet challenging task, as traditional ab initio crystal structure prediction (CSP) methods rely on time-consuming global searches and first-principles free energy calculations. Inspired by the recent success of deep learning approaches in protein structure prediction, which utilize pairwise amino acid interactions to describe 3D structures, we present AlphaCrystal-II, a novel knowledge-based solution that exploits the abundant inter-atomic interaction patterns found in existing known crystal structures. AlphaCrystal-II predicts the atomic distance matrix of a target crystal material and employs this matrix to reconstruct its 3D crystal structure. By leveraging the wealth of inter-atomic relationships of known crystal structures, our approach demonstrates remarkable effectiveness and reliability in structure prediction through comprehensive experiments. This work highlights the potential of data-driven methods in accelerating the discovery and design of new materials with tailored properties.

Keywords crystal structure prediction  $\cdot$ distance matrix  $\cdot$ deep learning  $\cdot$ residual neural network  $\cdot$ global optimization

1 Introduction

Computational discovery of novel functional materials has enormous potential in transforming a variety of industries such as mobile communication, electric vehicles, quantum computing hardware, and catalysts[1]. Compared to traditional Edisonian or trial-and-error approaches which usually strongly depend on the expertise of the scientists, computational materials discovery has the advantage of efficient search in the vast chemical design space. Among these methods, inverse design[2, 3], generative machine learning models[4, 5, 3, 6, 7], and crystal structure predictions [8, 9, 10] are among the most promising approaches for new materials discovery.

Crystal structure prediction (CSP) is a notoriously hard problem [11, 12, 13] since a CSP algorithm has to find a crystal structure with the lowest free energy for given a chemical composition (or a chemical system such as Mg-Mn-O with variable composition) at given pressure-temperature conditions. CSP is highly desirable as the predicted crystal structure of a chemical substance allows many physicochemical properties to be predicted reliably by machine learning models or calculated routinely using first-principle calculation [14, 15]. It is assumed that lower free energy corresponds to more stable arrangements of atoms. The CSP approach for new materials discovery is especially appealing due to the efficient sampling algorithm that generates diverse chemically valid candidate compositions with low free energies[4, 16]. CSP algorithms based on evolutionary algorithms [17] and particle swarm optimization [18] have led to a series of new materials discoveries [19, 1, 20]. However, these global free energy search-based algorithms have a major obstacle that limits their successes to relatively simple crystals [1, 21] (mostly binary materials with less than 20 atoms in the unit cell[1, 20]) due to their dependence on the costly density functional theory (DFT) calculations of free energies for sampled structures. With a limited DFT calculations budget, how to efficiently sample the atom configurations becomes a key issue [19, 13]. To improve the sampling efficiency, a variety of strategies have been proposed such as exploiting symmetry[22] and pseudosymmetry[13], smart variation operators, clustering, and machine-learning interatomic potentials with active learning [23]. However, the scalability of these approaches remains an unsolved issue.

With the mature development of deep learning techniques, several studies have applied those novel methods in the materials science field [24], particularly in the areas of material property prediction [25], material discovery [26], and material design[27]. Recently, several emerging works have utilized deep learning methods to predict material structures. Ryan et al. [28] reconstructed the crystal structure prediction problem as predicting the likelihoods of particular atomic sites in the structure. Their trained model successfully distinguishes chemical components based on the topology of their crystallographic environment. They use the model to analyze templates derived from the known crystal structures in order to predict the likelihood of forming new compounds. Cheng et al. [29] recently proposed the GNOA framework that employs a graph network model to connect crystal structures and their formation enthalpies and then merged this model with an optimization algorithm for CSP. Although they successfully predicted 29 crystal structures, the main limitation of the study is the failure to predict structures of complicated chemical formulas. Machine learning based and heuristic rules based template methods have also been proposed for CSP prediction with strong performance [30, 31].

Our work here is inspired by the great success of AlphaFold [32] in protein structure prediction since crystal structure prediction in material and protein structure prediction (PSP) in bio-informatics share many similarities [24]. (1) PSP aims to construct the three-dimensional (3D) protein structure from a protein only given its amino acid sequence [33], while CSP seeks to identify a crystal structure with the lowest free energy for certain chemical compositions. (2) Features and functions of a given protein and material can be explored more effectively if we uncover their structures. (3) Bboth tasks are typically time-consuming and costly when relying solely on laboratory experiments or first-principles calculations. Motivated by the recent breakthrough of deep learning in PSP with contact map [34] and DeepMind’s AlphaFold [32], we investigate the recent research on deep learning based protein structure prediction [35, 36, 37], in which the contact map places a key role in their algorithm design. For a protein 3D structure, a contact map is a binary form of the distance matrix with a distance threshold, where the distance is calculated between every pair of residues [35]. Adhikari et al.[34] proposed DNCON2, an ab initio protein contact map predictor based on two-level deep convolutional neural networks: at the first level, they used five convolution neural networks (CNNs) with multiple-distance thresholds to predict preliminary contact probabilities as the additional features in the next step; then, the another CNN was used to predict the final contact probability map. Since a distance map can reveal more structural information than a contact map, in [38], they presented the AlphaFold, a deep learning model that achieved high accuracy even when there were fewer homologous sequences available. Taking inspiration from AlphaFold’s success in protein structure prediction and acknowledging the similarities between crystal structure prediction and protein structure prediction as previously discussed, we aim to exploit the relationship between atomic pairs for crystal structure prediction. Unlike protein structure prediction, where co-evolution or correlated mutation relationships among amino acids are leveraged, the concept of such relationships among atoms is not well-established in the field of crystal structure prediction (CSP). Moreover, there is no existing method for predicting the distance matrix of crystal structures from compositions only except our previous work [39], in which the binary contact map is predicted and exploited for CSP with moderate success.

A large number of physical and chemical rules govern the interactions and bonding between atoms, leading to the predictability of the distance matrix for crystal structures from the composition. Building upon this foundation, we propose a distance matrix-based model and algorithm for CSP, the block diagram of which is illustrated in Figure 1. Our work demonstrates the possibility of making accurate predictions of the crystal structure for a given composition based on the distance matrix predicted by a deep neural network. Drawing inspiration from the successful strategies employed in protein structure prediction and leveraging the inherent physical and chemical principles governing atomic interactions, our approach provides a novel perspective on crystal structure prediction. The distance matrix, which encapsulates the intricate relationships between atoms, serves as a powerful representation for predicting the three-dimensional arrangement of atoms in a crystal structure. This work opens up new avenues for exploring data-driven and knowledge-based methods in the field of materials science, with the potential to accelerate the discovery and design of new materials with tailored properties.

Our contributions can be summarized as follows:

• We proposed a data-driven AlphaCrystal-II model for CSP which exploits the abundant atomic pair (bond) knowledge and utilizes the trained deep residual neural networks for distance matrix prediction and crystal structure reconstruction.

• We conducted extensive experiments to demonstrate AlphaCrystal-II’s competitive effectiveness and reliability in crystal structure prediction.

• We compared AlphaCrystal-II with the GNOA algorithm, a modern machine learning potential based CSP method. We found that AlphaCrystal-II can achieve better performance in most cases and works not only for materials with simple compositions but also those materials with complex compositions.

• We evaluated the performance of the distance matrix prediction and structure reconstruction by the distance matrix prediction network and the genetic algorithm DMCrystal, which showed a strong performance that explains the good performance of AlphaCrystal-II.

2 Methods

2.1 Framework of AlphaCrystal-II model for crystal structure prediction

Given a material composition, a simple yet effective approach for crystal structure prediction is to use the composition to find a possible template structure and conduct CSP using elemental substitution, which has led to powerful template based CSP algorithms such as CSPML[30] and TCSP[31]. The basic assumption here is that materials with similar compositions should have similar structures, which also implies that composition should contain enough information to predict the atomic pairwise contacts/bonds or distance matrices. This can be achieved by exploiting the composition similarity between the query sample and the training samples, the complementary bonding relationships of anions or cations, the oxidation state valence electron distributions, or the bonding preferences for a given pair of atoms. Although there is no co-evolutionary information or ancestral relationship among the protein structures, unlike in protein structures, the distance matrix based CSP is also feasible and promising.

Based on the above ideas and the great success of AlphaFold, we propose the AlphaCrystal-II model crystal structure prediction tasks. The framework of our model is shown in Figure 1. It contains an encoding module, a distance matrix prediction neural network, and a structure generation/reconstruction module. For a given material composition or formula, a feature matrix is constructed using the 11 elemental properties as shown in Table 1. This feature matrix of the composition is then fed to the deep residual network for predicting its distance matrix. The detailed architecture of this network model is shown in Figure 2. The model is trained using known crystal structures selected from the Materials Project database [40]. We also need to use the third-party machine learning model MLatticeABC to predict the lattice parameters and space groups [41]. The predicted distance matrix, along with the unit cell information, is used by the DMCrystal [42] genetic algorithm to reconstruct the final structure. Finally, the Bayesian Optimizer of the M3GNet [43] package is used to relax the predicted structures and estimate their formation energies using the M3GNet ML potential, thereby enabling the identification of stable structures.

Figure 1: The AlphaCrystal-II framework for distance matrix-based crystal structure prediction. First, material compositions are encoded into feature matrices according to their atom features, which are used to train the deep neural network for predicting distance matrices. Next, the crystal structures are reconstructed by DMCCrystal GA along with predicted unit cell information. The resulting candidate structures are then relaxed by the M3GNet model. For multiple candidate structures, the top K structures are selected based on their predicted formation energies determined by M3GNet.

In the sections below, we describe the details of the components of our AlphaCrystal-II modules.

2.2 Feature matrix encoding

Representing material composition information in an appropriate format to learn their atomic interactions is an essential and crucial preparation step for the AlphaCrystal-II algorithm because atomic interactions determine its final structure. Previous research has demonstrated that many atomic characteristics are related to ionic bonds, such as ionic bonding are electronegativity difference, ionization energy, electron affinity, atom size, ionic charge, lattice energy, polarizability, coordination number, and environmental conditions like temperature, pressure, and presence of other substances [44]. In our approach, we use 11 atomic features to encode the physicochemical information of each atom in the material, serving as input for the deep neural network model utilized for distance matrix prediction.

Specifically, each element/atom is represented by 11 chemical descriptors (see Table 1 for details) including Mendeleev Number [45], unpaired electrons, ionization energies, covalent radius, heat of formation, dipole polarizability [46], average ionic radius, group number and row number in the periodic table, pauling electronegativity, and atomic number. These 11 atomic features are represented by each row of the feature matrix as shown in Figure 1.

Each row of the feature matrix represents an element symbol in the unit cell, where $L$ is the number of atoms. We set the maximum number of atoms to be 12 in this study for simplicity, and if the number of atoms is less than 12, the remaining empty positions will be padded with zeros. Hence, the dimension of each feature matrix is $12\times 11$.

Table 1: 11 elemental features for atomic composition encoding

Feature	Description
Mendeleev number (MN) [45]	Listing of chemical elements by column through the periodic system, which is effectively used to classify chemical systems.
Unpaired electrons	Electrons that occupy an orbital of an atom singly.
Ionization energies	The amount of energy required to remove an electron from an isolated atom or molecule.
Covalent radius	Half of the distance between two atoms bonded covalently.
Heat of formation	When one mole of a compound is produced from its basic elements, each substance is in its normal physical state, the quantity of heat received or released.
Dipole polarizability [46]	Describes the linear response of an electronic charge distribution with respect to an externally applied electric field.
Average ionic radius	Average of a monatomic ion’s radius in an ionic crystal structure.
Group number in periodic table	The number of valence electrons of the elements in a certain group, a group is a vertical column of the periodic table.
Row number in periodic table	The number of rows of the element in the periodic table.
Pauling electronegativity	The power of an atom in a molecule to attract electrons to itself.
Atomic number	The charge number of an atomic nucleus.

2.3 Deep residual network model for distance matrix prediction

For distance prediction, we train a deep residual neural network to capture and exploit the intricate bonding interactions between atoms. This method leverages the vast distribution of inter-atomic interactions present in the large number of known crystal structures used as our training set, thereby enabling highly accurate pairwise distance predictions.

Figure 2 illustrates our deep neural network model, which comprises three main components. The first part consists of a sequence of stacked 1-dimensional (1D) residual network layers designed to learn the intricate features of atomic sites. In the second part, a 2-dimensional (2D) pairwise feature matrix is derived from the output of the 1D convolutional network through an outer product operation. Subsequently, we merge the convoluted sequential features and pairwise features to serve as the input for the next module. The third part encompasses a series of 2D residual network layers, which predict the distances between pairs of atoms, ultimately generating the predicted distance matrix.

In our studies, we set the maximum number of atoms in a formula to $L$ to accommodate the variable sizes of different crystal structures. In the experiments, $L$ is set to 12 (the same as the feature matrix dimension). For formulae with fewer atoms, we create the corresponding tensors by padding them with zeros.

The Residual Neural Network (ResNet) [47] is a type of neural network architecture that stacks residual blocks on top of each other, forming a deep network through skip connections. This design has profoundly influenced the way deep neural networks are constructed. The skip connections between layers add the outputs from previous layers to the outputs of stacked layers, enabling the training of much deeper networks than what was previously possible. In the AlphaCrystal-II model, we design two residual network modules: one module aims to extract sequential features, while the other module intends to derive pairwise features. Each residual network block comprises two convolutional layers, a batch normalization layer, and two nonlinear transformations. Our main architecture employs 9 building blocks for each module. The number of filters is doubled every 3 blocks, with the initial number of filters set to 32 and 256 for the first and second modules, respectively. The residual network architecture, with its skip connections and residual blocks, enables the training of deeper neural networks by mitigating the vanishing gradient problem. This design choice allows our model to effectively capture the intricate patterns and interactions present in the atomic structures, leading to more accurate predictions of pairwise distances and ultimately improving the overall performance of our crystal structure prediction approach.

Figure 2: Deep neural network model for distance matrix prediction.

Since the discretized distance map is a binary matrix, we use the cross-entropy loss as the loss function for neural network training. It is defined as:

$$L_{crossEntropy}=-\frac{1}{N}\sum_{i=0}^{N}y_{i}\cdot log(\hat{y}_{i})+(1-y_{i})\cdot log(1-\hat{y}_{i})$$ (1)

Where $N$ is the maximum length of the formula, which is set to $12\times 5$ and $12\times 10$ in our experiments; $y_{i}$ is the true distance matrix label at position $i$, and $\hat{y}_{i}$ is the predicted label at position $i$.

2.4 3D crystal structure reconstruction

Once the neural network predicts the distance matrix for a given composition, the crystal structure can be reconstructed using a genetic algorithm such as DMCCrystal [42]. This work has demonstrated that, given the pairwise atomic distance matrix with the space group, lattice parameters, and stoichiometry, the genetic algorithms can reconstruct the crystal structure that is close to the target crystal structure. For further improvement, these predicted structures can be used to seed the costly free-energy minimization-based ab initio CSP algorithms, as well as to obtain a more accurate crystal structure of certain components by DFT-based structural relaxation.

In this work, the reconstructed structures by the genetic algorithm are all relaxed using the Bayesian optimization algorithm with the M3GNet model [43], which is a universal inter-atomic potential for materials based on graph neural networks with three-body interactions. It is trained by a lot of structural relaxations performed from the Materials Project for 89 elements of the periodic table with low energy, force, and stress errors. It has a wide range of uses in structural relaxation and dynamic simulations of materials across diverse chemical spaces.

2.5 Evaluation metrics

In assessing the performance of our distance map based structure prediction algorithm for crystal structure prediction (CSP), we use eight different metrics, including superpose distance, energy distance, Sinkhorn distance, fingerprint distance, OFM distance [48], Wyckoff MSE, Wyckoff RMSE, and distance map overlap.

The Wyckoff MSE and RMSE measure the similarity between the predicted structure and the true target structure. Distance overlap evaluates the precision of the predicted distance map compared to the target distance map. The equations for such metrics are shown in the following equations:

$$\mathrm{MSE} =\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}$$ (2)

$$\mathrm{RMSE} =\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}$$ (3)

where $n$ is the number of independent atoms in the target crystal structure, $y_{i}$ and $\hat{y}_{i}$ are the coordinates of the corresponding atoms in the predicted and the target crystal structures.

$$\operatorname{DMoverlap}=\frac{|TargetDM_{atom~{}exist}\cap PredictedDM_{atom~{}exist}|}{|Target~{}atoms|}$$ (4)

where $TargetDM$ is the target distance map and $PredictedDM$ is the predicted distance matrix of a given composition, both only contain 1/0 entries. $Target~{}atom$ means the total number of atoms in the target distance map. $TargetDM~{}atoms_{exist}\cap PredictedDM~{}atom_{exist}$ denotes the common existent atoms of $TargetDM$ and $PredictedDM$. This performance metric evaluates the overlap between two distance map matrices, with values ranging from 0 to 1, where 1 indicates perfect overlap. This metric, also known as distance map accuracy, quantifies the fidelity of the predicted distance map. To obtain the predicted distance map, we use a threshold of 8 amstrong as the cutoff to convert a given distance matrix into a binary distance map.

3 Results

3.1 Dataset

We collected material ids, formulas, and crystal structures (cif files) from the Materials Project database [40] by Pymatgen API [49]. Filtering out compounds with a maximum of 12 atoms in the formula yielded 18,800 compounds from the Materials Project, constituting our initial dataset named Mp_12. Furthermore, to compare our prediction results with the GNOA algorithm [29], we removed 29 formulas that predicted structures with high performance from the training set, therefore, there are 18,776 compounds in Mp_12 datasets. Moreover, we isolated materials with a cubic crystal system to create the Mp_12_cubic dataset. Additionally, we extracted formulas comprising binary and ternary elements to form the Mp_12_binary and Mp_12_ternary datasets, respectively. Subsequently, we trained four distance matrix prediction models using each of these four datasets and evaluated their performance.

Table 2: Datasets

Dataset	Number of samples	Description
Mp_12	18,776	materials with $\leq 10$ atoms per unit cell picked from Materials Project and after the 29 test samples in the GNOA study are removed
Mp_12_cubic	3,298	cubic materials extracted from Mp_12
Mp_12_binary	5,848	binary materials extracted from Mp_12
Mp_12_ternary	11,615	ternary materials extracted from Mp_12

Distance discretization

We counted the atomic distances of 18,776 samples in the Mp_12 dataset, the overall distribution is shown in Figure 3 where the smallest distance is 0.9488, the largest distance is 23.3361, and the majority of distances are between 0 to 7 groups. To facilitate further analysis, we divided the continuous distance values with equal width and store them in several groups by equation 5.

$$Interval~{}width=(Maximum\_value-Minimum\_value)/N$$ (5)

where $N$ represents the number of groups, we set it to 5, 10, or 20 in experiments. Furthermore, we use the one-hot encoding to get the expression of each value. This method turns the regression problem into a classification problem by turning continuous distances into discrete values. Therefore, the cross-entropy loss can be used as the loss function for training neural networks, which makes the prediction more accurate.

For each material in our dataset, we encoded 11 atomic features to a 12*11 matrix (the maximum atomic number in the unit cell is 12). After one-hot encoding, the dimension of the distance matrix is 12*(12*N). In our experiments, the training samples are randomly selected with different sample sizes according to the needed dataset sizes, such as 1,000 or 10,000 samples; after model training, 100 or 1000 samples that are not included in the training sets are selected for CSP algorithm testing and performance analysis.

Figure 3: Overall atomic distance distribution of Mp_12 dataset. The x-axis represents the atomic distance and the y-axis represents the corresponding ratios. The most atomic distances are in the range of 2.33-4.67, accounting for more than 25% and there are just a few distances more than 14.

3.2 Prediction performance of AlphaCrystal-II

3.2.1 CSP Performance Comparison of AlphaCrystal-II model and GNOA model

We first evaluated the CSP performance of our AlphaCrystal-II and compared it with the other machine learning potential based CSP algorithm, GNOA [29]. GNOA uses a graph network energy potential (MEGNet) [50] and an optimization algorithm such as random search, Bayesian Optimization, and Particle Swarm Optimization for crystal structure prediction. The graph neural network is trained to build a correlation model between crystal structures and their formation enthalpies, which is utilized by an optimization algorithm to search for the crystal structure that has the lowest formation enthalpy. Their report [29] showed 29 selected compounds with good prediction performance. However, their selected test samples are almost all belonging to simple binary compounds with few atoms within the unit cell. To more objectively compare the predicted performance of GNOA and AlphaCrystal-II, we evaluated these two models by predicting 30 diverse material structures separately, and the experimental results are summarized in Table 3.

We randomly selected 10 binary, ternary, and quaternary materials from our MP_12 dataset, respectively. For each formula, we used both GNOA and AlphaCrystal-II to predict the probable crystal structures and then calculated the superpose distance between each predicted structure and the ground truth structure. Our AlphaCrystal-II can generate many structure candidates all at once. However, GNOA can only provide one structure that has the lowest formation energy. Thus, we compare GNOA’s best structure with AlphaCrystal-II’s Top-10 and Top-20 structure candidates.

As shown in Table 3, for all 30 formulas, AlphaCrystal-II’s Top-10 structure candidates achieved the best performance 17 times, and Top-20 candidates expanded this number to 23. In contrast, GNOA only outperformed AlphaCrystal-II in 7 formulas. In two of these 7 formulas, BePd and PaIn${}_{3}$, GNOA’s performance only outperformed AlphaCrystal-II by 0.0004 and 0.0002, respectively. This means that AlphaCrystal-II’s structure had comparable properties. Furthermore, GNOA encountered challenges in handling complex formulas, particularly for quaternary materials. It failed to predict the structures of Rb${}_{2}$LiRhCl${}_{6}$, Y${}_{3}$Al${}_{3}$NiGe${}_{2}$, and LiFe${}_{2}$(ClO)${}_{2}$. In contrast, AlphaCrystal-II was able to provide many structure candidates that were similar to the ground truth structures. Moreover, for two formulas, Ce${}_{3}$Pm and DyThCN, AlphaCrystal-II could generate structures that were the same as the ground truth structures, which demonstrates the power of our AlphaCrystal-II.

Table 3: Performance Comparison of AlphaCrystal-II and GNOA.

Type	Formula	Mp_id	GNOA superpose distance(Å)	AlphaCrystal-II Top-10 superpose distance(Å)	AlphaCrystal-II Top-20 superpose distance(Å)
Binary	TiIn	mp-1216825	0.7775	0.0158	0.0158
	ScAl	mp-331	1.4594	0.0032	0.0032
	Tm${}_{3}$P	mp-971958	1.9484	0.5625	0.5625
	BePd	mp-11274	0.0085	0.0089	0.0089
	PaIn${}_{3}$	mp-861987	1.6442	1.6644	1.6644
	Ce${}_{3}$Pm	mp-1183767	0.0456	0	0
	GdCo${}_{5}$	mp-1077071	0.8275	0.024	0.024
	LiC${}_{6}$	mp-1001581	1.1425	0.843	0.843
	BaAu${}_{5}$	mp-30364	1.0055	1.265	1.265
	Zr${}_{4}$Al${}_{3}$	mp-12752	1.5999	0.0204	0.0204
Ternary	TmIn2Sn	mp-1216827	1.6361	1.1517	0.9419
	CeAlO${}_{2}$	mp-1226604	1.5539	1.277	1.1403
	ThBeO${}_{3}$	mp-1187421	1.0304	1.2676	1.2676
	Zn(CuN)${}_{2}$	mvc-15351	0.6701	1.0663	1.0663
	LiVS${}_{2}$	mp-7543	1.7123	0.7939	0.7939
	LiTiTe${}_{2}$	mp-10189	1.2147	1.0687	1.0687
	Sr(AlGe)${}_{2}$	mp-1070483	1.3211	0.2149	0.2149
	NdTiGe	mp-22331	1.9989	1.5848	1.4723
	Mn${}_{4}$AsP${}_{3}$	mp-1221760	1.596	1.3444	1.3444
	Mn${}_{3}$FeP${}_{4}$	mp-1221749	1.1733	1.0396	0.9941
Quaternary	DyThCN	mp-1225528	1.3307	0.4564	0
	EuNbNO${}_{2}$	mp-1225127	1.0466	0.621	0.621
	LaNiPO	mp-1079685	2.0049	1.4417	1.3439
	SrFeMoO${}_{5}$	mp-690817	0.4081	1.8519	1.1096
	FeCu${}_{2}$SnS${}_{4}$	mp-628568	2.1553	1.5029	1.2196
	LiTb(CuP)${}_{2}$	mp-8220	1.5779	0.9468	0.4403
	Rb${}_{2}$LiRhCl${}_{6}$	mp-1206187	None	0.6224	0.6224
	Y${}_{3}$Al${}_{3}$NiGe${}_{2}$	mp-10209	None	1.1824	1.1824
	LiFe${}_{2}$(ClO)${}_{2}$	mp-755254	None	0.9999	0.9999
	Mg${}_{2}$VWO${}_{6}$	mp-1303315	1.5126	1.5777	1.5706
	# of best		7	17	23

Figure 4: Comparison of formation energies of predicted structures by GNOA and AlphaCrystal-II.

In addition, formation energy serves as a crucial criterion for assessing the stability of crystal structures. Hence, we compare the lowest formation energies of existing crystal structures with those of our predicted structures generated by M3GNet, as depicted in Figure 4. From this figure, we can observe that the formation energies of predicted structures by AlphaCrystal-II closely align with those of the target structures, in contrast to the results obtained by GNOA, which further proves the effectiveness of our method.

We further check three case studies for the prediction performance of these two algorithms. Three samples of predicted crystal structures by GNOA and AlphaCrystal-II and their ground truth (target) structures are shown in Figure 5. For compound Zr₄Al₃, the RMSE of the distance matrix is 0.4818 Åand the RMSE of the predicted structure is 0.2183 Å.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 5: Examples of predicted crystal structures by AlphaCrystal-II. (a) the ground truth structure of Zr${}_{4}$Al${}_{3}$; (b) the predicted structure of Zr${}_{4}$Al${}_{3}$ by GNOA; (c) the predicted structure of Zr${}_{4}$Al${}_{3}$ by AlphaCrystal-II; (d) the ground truth structure of ScAl; (e) the predicted structure of ScAl by GNOA; (f) the predicted structure of ScAl by AlphaCrystal-II; (g) the ground truth structure of EuNbNO${}_{2}$; (h) the predicted structure of EuNbNO${}_{2}$ by GNOA; (i) the predicted structure of EuNbNO${}_{2}$ by AlphaCrystal-II;

3.2.2 Performance Comparison of AlphaCrystal-II against AlphaCrystal

Here we evaluate how the distance matrix based AlphaCrystal-II performs compared to the atomic map based AlphaCrystal-I [39]. We used both algorithms to predict the crystal structures for the following test samples from the Materials Project database, including Pb${}_{4}$O${}_{4}$, Co${}_{4}$P${}_{8}$, Ir${}_{4}$N${}_{8}$, V${}_{2}$Cl${}_{10}$, Co${}_{2}$As${}_{2}$S${}_{2}$, V${}_{4}$O${}_{4}$F${}_{4}$, Fe${}_{4}$As${}_{4}$Se${}_{4}$, Mn${}_{4}$Cu${}_{4}$P${}_{4}$. We then compare their performance metrics using four metrics selected from the CSBenchMetrics [48]. The results are shown in Table 4.

For eight selected formulas, we used both AlphaCrystal-I and AlphaCrystal-II to generate a branch of probable structures and then chose the top 20 with the lowest formation energy value as final structure candidates. Then, we employed five different metrics including energy distance, Sinkhorn distance, superpose distance, fingerprint distance, and OFM distance to compare the candidate structures’ similarity with the target ground truth structure. For Pb${}_{4}$O${}_{4}$ and V${}_{4}$O${}_{4}$F${}_{4}$, AlphaCrystal-I could not generate any structure candidates as the predicted space group was different from the ground truth, and the suggested template for these formulas contained many special coordinates (such as 0, 0.25, 0.33, 0.5, 0.66) that AlphaCrystal-I could not further relax. For all other six formulas, AlphaCrystal-II was able to provide the structures that had the lower energy distance, fingerprint distance, and OFM distance with the ground truth structure. When considering Sinkhorn distance, AlphaCrystal-I only outperformed AlphaCrystal-II once on Fe${}_{4}$As${}_{4}$Se${}_{4}$. For V${}_{2}$Cl${}_{10}$ and Mn${}_{4}$Cu${}_{4}$P${}_{4}$, the structure candidates generated by AlphaCrystal-I had lower superpose distance. In general, AlphaCrystal-II outperformed AlphaCrystal-I, especially when the predicted structure contained special coordinates.

Table 4: Performance Comparison of AlphaCrystal-I and AlphaCrystal-II.

Formula	Model	energy distance (eV)	sinkhorn distance(Å)	superpose distance(Å)	fingerprint distance	OFM distance
Pb${}_{4}$O${}_{4}$	AlphaCrystal-I	None	None	None	None	None
	AlphaCrystal-II	0.0481	13.8642	0.7667	0.3945	0.0355
Co${}_{4}$P${}_{8}$	AlphaCrystal-I	5.5616	20.2276	1.0251	1.2315	0.1106
	AlphaCrystal-II	0.0011	17.3153	0.9784	1.0926	0.0844
Ir${}_{4}$N${}_{8}$	AlphaCrystal-I	18.5385	17.2003	1.0512	1.1572	0.2728
	AlphaCrystal-II	0.4584	14.2731	0.9165	0.6723	0.0719
V${}_{2}$Cl${}_{1}$${}_{0}$	AlphaCrystal-I	7.34	39.5468	1.5859	1.3944	0.0817
	AlphaCrystal-II	0.1806	33.2543	1.6411	1.1116	0.0357
Co${}_{2}$As${}_{2}$S${}_{2}$	AlphaCrystal-I	3.0549	9.5731	0.9939	1.9771	0.2249
	AlphaCrystal-II	0.0326	7.7433	0.9477	0.1892	0.0279
V${}_{4}$O${}_{4}$F${}_{4}$	AlphaCrystal-I	None	None	None	None	None
	AlphaCrystal-II	0.0495	15.2005	1.033	0.2289	0.0355
Fe${}_{4}$As${}_{4}$Se${}_{4}$	AlphaCrystal-I	3.4682	16.0662	1.3837	1.262	0.1906
	AlphaCrystal-II	0.006	17.341	1.0916	0.8575	0.0655
Mn${}_{4}$Cu${}_{4}$P${}_{4}$	AlphaCrystal-I	11.0241	20.0757	1.0341	1.2256	0.2327
	AlphaCrystal-II	0.0168	19.4467	1.1193	0.8246	0.1476
# of best	AlphaCrystal-I	0	1	2	0	0
	AlphaCrystal-II	8	7	6	8	8

3.2.3 Performance of distance matrix prediction in AlphaCrystal-II

To assess the strengths and weakness of our AlphaCrystal-II model, we evaluated whether the deep neural network of our AlphaCrystal-II model can predict the distance matrix with good performance by learning the atomic pair relationships from the chemical compositions. We evaluated the performance with different experiments with varying hyper-parameters, including the type of samples, the number of discrete groups, and the sizes of training and test datasets. The distance matrix prediction performance is summarized in Table 5. In the training of these distance matrix prediction models, we set the number of epochs to 125 and utilized the Adam optimizer with a learning rate of 0.001.

Table 5 includes distance matrix prediction performance metrics including MSE, RMSE, and distance map overlap. It also summarizes the results of multiple comparison trials. Firstly, comparing different sizes of training datasets, there is a clear trend of decreasing MSE and RMSE with more training samples. For example, for the AlphaCrystal-II model with a discretization group number of 5, when the number of training samples is increased from 1,000 to 10,000, the RMSE is reduced by 53.57%. Secondly, we set different numbers of discrete groups for the basic AlphaCrystal-II model. Comparing groups 5, 10, and 20 with 10,000 training samples, RMSE is reduced by 11.69% and 18.88%, respectively, indicating better performance when a larger distance encoding group is used. In order to explore whether focusing on the same type of materials has a better performance in distance matrix prediction, we trained three special models for distance matrix prediction for three families of materials: AlphaCrystal-II_cubic, AlphaCrystal-II_binary, and AlphaCrystal-II_ternary. AlphaCrystal-II_cubic model is trained by materials whose crystal system is cubic. AlphaCrystal-II_binary and AlphaCrystal-II_ternary are trained by binary and ternary materials. As shown in Table 5 (last three rows), the AlphaCrystal-II_cubic achieved the smallest MSE (1.0125 Å) and RMSE (0.7394 Å) compared to all other models, indicating that training a specialized distance matrix prediction model is beneficial to improve their performance. For example, one may want to train such a model for perovskite materials. Next, we found the AlphaCrystal-II_ternary model achieved the second-best performance out of these three specialized models with an MSE of 2.0832 Å and RMSE of 1.2779 Å, together with the highest distance map overlap. This can be understood by the fact that the ternary materials have the maximum number of training samples and many of them share some limited number of structure prototypes, which makes it easier to predict their distance matrices. Finally, the AlphaCrystal-II_binary model achieved the worst performance despite its reasonable number of training samples (4,678) with an MSE of 2.7835 Å and RMSE of 1.3109, which can be partially attributed to the fact that binary structures are more diverse in terms of structural prototypes. Overall, looking at the distance map overlap performance criterion, we found all three models have achieved reasonably good results with scores ranging from 0.9746 to 0.9985.

Table 5: Distance matrix prediction performance of AlphaCrystal-II

Model	Distance encoding groups	Training dataset	Test dataset	MSE	RMSE	Overlap
AlphaCrystal-II	5	1,000	100	4.9538	1.8841	0.9045
AlphaCrystal-II	5	10,000	1,000	2.3764	1.2269	0.9979
AlphaCrystal-II	10	1,000	100	2.8586	1.3399	0.9981
AlphaCrystal-II	10	10,000	1,000	1.8932	1.0984	0.9989
AlphaCrystal-II	20	10,000	1,000	1.5591	0.9239	0.9992
AlphaCrystal-II_cubic	10	2,638	200	1.0125	0.7394	0.9746
AlphaCrystal-II_binary	10	4,678	400	2.7835	1.3109	0.9980
AlphaCrystal-II_ternary	10	9,292	1,000	2.0832	1.2779	0.9985

To further understand the prediction capability of our distance matrix predictor, we showed the training loss and validation loss of the neural network models from the AlphaCrystal-II models with 1,000 and 10,000 training samples in Figure 6. The discretization group number of both is set as 10. As the number of epochs increases, both the training loss and validation loss gradually decrease, when the loss curves hit a plateau, around 80 and 60 epochs, respectively, and they do not decrease any further and become stabilized. In Figure 5(a), the loss curve shows a slow but steady decrease over time, indicating that the model is learning at a consistent rate. In Figure 5(b), the loss curves show a steep drop during the first few training epochs, indicating that the model is learning quickly. The shapes of both training and validation loss curves demonstrated that our distance matrix network models were well-trained.

(a) 1,000 training samples

(b) 10,000 training samples

Figure 6: Training and validation loss curves of the residual network model for distance matrix prediction. (a) the training curve of the AlphaCrystal-II model with 1,000 training samples; (b) the training curve of the AlphaCrystal-II model with 10,000 training samples.

To obtain an intuitive understanding of the distance matrix, Figure 7 shows two examples of the real ground truth distance matrix and the predicted distance matrix for two compounds EuSi${}_{2}$ and ScTlAg${}_{2}$ for which the distance discretization group number is set to 20. The highlighted cells show the accurately predicted inter-atomic interaction distances. It can be found that both cases have relatively high accuracy in terms of distance matrix cell matching. These figures demonstrate that our deep neural network models can learn the pair-wise relationship between atoms for distance matrix prediction. We also find that as the number of discretized distance groups increases within a reasonable range, the predicted distance labels are more accurate.

(a) EuSi${}_{2}$

(b) ScTlAg${}_{2}$

Figure 7: Comparison of the predicted and real distance matrix with the discretization group number of 20. (a) EuSi${}_{2}$(mpid:1072248), (b) ScTlAg${}_{2}$ (mpid:1093619).

3.2.4 Performance of distance matrix based crystal structure reconstruction

Another major module of our AlphaCrystal-II algorithm is the distance matrix based crystal structure reconstruction. For each query composition, we first predict their atomic distance matrix from their composition and then use the DMCrystal algorithm [42] to predict its three-dimensional crystal structure. For each query formula, we generate 50 candidate structures using the genetic algorithm based DMCrystal algorithm for structure reconstruction. We then use the Bayesian optimization routine of the M3GNet package [43] to relax these structures and predict their formation energies. We summarize the predicted RMSE performances of 10 binary materials, 10 ternary materials, and 10 materials with more than 4 elements in Table 3. "Mp_id" represents the specific ID number of each chemical formula in the material project database. "Distance Matrix RMSE" indicates the RMSE values of our predicted distance matrix. The row of "AlphaCrystal-II Top-10 RMSE" lists the lowest RMSE values within the 10 lowest formation energies while the row of "AlphaCrystal-II Top-20 RMSE" lists the lowest RMSE values within the 20 lowest formation energies.

4 Conclusion

We present AlphaCrystal-II, a novel data-driven deep learning approach for crystal structure prediction that exploits the abundant inter-atomic interaction patterns present in known crystal material structures. Our method first employs a deep residual network to learn and predict the atomic distance matrix for a given material composition. It then utilizes this matrix to reconstruct the 3D crystal structures through a genetic algorithm. Subsequently, we employ the Bayesian Optimization algorithm of the M3GNET package to relax these structures and calculate their formation energies, identifying the most stable configurations.

Through extensive experiments, we demonstrate that our model effectively captures the implicit inter-atomic relationships and showcases its effectiveness and reliability in leveraging such information for crystal structure prediction. Our results indicate that AlphaCrystal-II can successfully reconstruct the crystal structures of a wide range of materials, including binary, ternary, and multi-component systems with more than four elements. By harnessing the abundant inter-atomic interaction patterns, our data-driven, knowledge-guided approach paves the way for large-scale crystal structure prediction in the near future.

This work highlights the potential of deep learning and data-driven methods in accelerating the discovery and design of new materials with tailored properties. By combining data-driven approaches with domain knowledge and physics-based methods, we can unlock new avenues for materials research and development, ultimately contributing to the advancement of various technological domains.

5 Availability of data

The data that support the findings of this study are openly available in the Materials Project database at https://legacy.materialsproject.org/

6 Contribution

Conceptualization, J.H.; methodology, J.H. and Y.S.; software, Y.S., and J.H; validation, Y.S. E.S. J.H.; investigation, Y.S., L.W., Q.L., J.H., ; resources, J.H.; data curation, J.H., Y.S.; writing–original draft preparation, Y.S., R.D., L.W., J.H.; writing–review and editing, J.H, R.D.; visualization, Y.S. J.H.; supervision, J.H.; funding acquisition, J.H.

7 Acknowledgement

Research reported in this work was supported in part by NSF under grants 1940099 and 1905775 and by NSF SC EPSCoR Program under award number (NSF Award OIA-1655740 and GEAR-CRP 19-GC02). The views, perspectives, and content do not necessarily represent the official views of the SC EPSCoR Program or those of the NSF.