Unleashing the potential: AI empowered advanced metasurface research

3.1.1 Supervised learning

Among all machine learning algorithms, one of the most well-known is Support Vector Machine (SVM) [146]. It is designed for binary classification tasks. Given a sample training set D = {(x, y)}, the classification problem is to find a hyperplane that separates samples of different classes. This separating hyperplane can be described by a linear equation w ^T x + b = 0, where w is the normal vector and b is the distance term. Assuming it can correctly classify the training samples, the feature vectors corresponding to the samples closest to the hyperplane are called support vectors, and the sum of the distances from two support vectors of different classes to the hyperplane γ = 2 w is called the margin. The basic idea of SVM is to maximize the margin, which means finding the classification hyperplane with the best robustness and generalization performance. Therefore, its basic mathematical form can be represented as

This training process is a convex optimization problem.

Due to its good generalization ability and prediction accuracy, SVM has been widely used in metasurface design problems [147], [148]. In practical cases, since the training samples are not necessarily linearly separable, which means there may not exist a hyperplane in the original sample space that can correctly classify them, the kernel function method is used to map the original sample space to a high-dimensional feature space, making the training samples linearly separable in the new space. Lu et al. [147] proposed a fast reconfigurable metasurface design method based on SVM, which was used to predict the amplitude curves under different structures. The algorithm framework is shown in Figure 4(a). They designed a two-dimensional QR code-like metasurface structure, which is symmetric about the Y-axis and each region has 8*4 pixels. The pixels are encoded with “1” and “0” to indicate whether they are filled with metal. To train the SVM model, they generated 10,000 structures and calculated the corresponding electromagnetic characteristics using CST. For each of the 32 pixel regions, they built a separate SVM model, mapped the electromagnetic response to a high-dimensional space using the Gaussian kernel function [149], [150], and introduced PSO to optimize the penalty coefficient C, which denotes the tolerance of error, and the kernel function parameter γ, which affects the speed of training and prediction [151], [152].

Figure 4:

SVM for metasurface design. (a) Flowchart of reconfigurable metasurface design based on SVM model. Reproduced from [147]. Copyright 2022, IEEE. (b) Above is coded reconfigurable metasurface. Below are target amplitude curve (red) and predicted amplitude curve (blue) of 1 bit amplitude modulation metasurface. Reproduced from [147]. Copyright 2022, IEEE. (c) Diffractive signatures generated theoretically for free-space wavelength of 532 nm, 1 μm, 2 μm, and 4 μm. Reproduced from [148]. Copyright 2021, American Chemical Society. (d) Accuracy of the SVM classifiers, and classification accuracy of the particular object for different wavelength in (c). Reproduced from [148]. Copyright 2021, American Chemical Society.

To verify the capability of the SVM model, a metasurface resonating at 6.17 GHz was designed, demonstrating good predictive performance, showed in Figure 4(b). In general, reverse identification of metasurface structures based on diffraction patterns is quite challenging, especially when the feature differences between most samples are small. In such cases, SVM can assist in the classification and identification tasks of complex images. Ghosh et al. [148] introduced a SVM-based classifier to analyze a diffraction image library parameterized by specific combinations, thereby identifying the subwavelength-level unit features carried by the metasurface. Figure 4(c) shows the diffraction images in free space with incident wavelengths of 532 nm, 1 μm, 2 μm, and 4 μm, respectively. Once the classifier is trained, the identification process can be performed based on the diffraction pattern measured in a single experiment. The researchers represented the intensity of diffraction images in the form of Bessel transforms:

where the indices m and j describe the distribution of intensity radially on the angular angle. The SVM classifier is used to pair the combinations of parameters (m, j) and analyze the accuracy of the generator. The final results are shown in Figure 4(d), indicating that the proposed ML-based detection technique can identify nanoscale units at the 1/5 wavelength level. This approach can be used not only for characterizing metasurface with resonant elements in reflection or transmission modes but also for reverse designing metasurface based on desired diffraction images. Based on the classification problem, the Support Vector Regression (SVR) algorithm has been developed for regression problems. It applies kernel functions to select hyperplanes that include the maximum number of data points, thereby predicting numerical outputs in regression problems [153], [154].

k-Nearest Neighbor (kNN) algorithm is also one of the simplest ML algorithms that is applicable to both classification and regression problems [155]. It has been successfully utilized in antenna design and electromagnetic field optimization. The principle behind kNN is to find the k nearest training samples in the training set to a given test sample based on a certain distance measure and use them to predict new data. In regression problems, kNN calculates the average or weighted average of the outputs of the nearest points, which can be represented as y ̂ = 1 k ∑ i = 1 k y i .

Unlike SVM, kNN follows a lazy learning approach where it only stores the samples during the training process and performs computations when a test sample is given. Su et al. [156] proposed an optimization method for daytime radiative cooling using metasurface based on kNN. They employed kNN to construct a regression model for optimizing the structures of quadrangular prismatic metasurface (QPM) and circular truncated cone metasurface (CTCM), as shown in Figure 5(a) and (b). The metasurface consists of two layers: Ag and ZnO as the substrates and Si₃N₄ structures. Four variables, including the upper and lower diameters of Si₃N₄ pillars, the height of the pillars, and the thickness of the ZnO layer, were selected as the feature variables for predicting the absorption/emission rates. They calculated 500 different structures, with 90 % of them used as the training set for the kNN regression model. After training, the R-squared values for all predicted values in the regression model were above 0.9, and the mean absolute percentage errors (MAPE) for the two structures were only 0.74 % and 0.29 %, respectively. Based on this, optimization design was conducted for the two structures, resulting in absorption/emission spectra that reduce in the solar spectrum and increase in the atmospheric window. Additionally, the optimized structures exhibited polarization insensitivity and angle independence, demonstrating ideal characteristics. Similarly, Nuzhat et al. [157] utilized a kNN-based machine learning model to predict the resonant frequencies of Artificial Magnetic Conductor (AMC) cells and compared it with other ML algorithms. They inputted the structural parameters L and G and simulated 129 sets of resonance curves using ANSYS HFSS as the dataset. During kNN training, they selected k = 3 as the optimal value. The results are shown in Figure 5(c), the trained kNN model exhibited good consistency with HFSS and required much less time and computational resources than traditional electromagnetic simulation methods. Despite its simplicity, kNN demonstrated excellent performance in predicting AMC resonance frequencies due to its ability to handle small-dimensional datasets with a large number of measurement values. In contrast, SVM showed larger prediction biases as it is not suitable for datasets with small dimensions and a large number of measured values. On the other hand, XGBoost [158], known for its speed and based on decision tree boosting, showed comparable or even superior capabilities to kNN.

Figure 5:

kNN and RF for metasurface design. (a) Flowchart of the radiative cooler structural parameters prediction combined with kNN regression model. Reproduced from [156]. Copyright 2023, Elsevier. (b) The proposed QPM and CTCM units and the absorptivity/emissivity spectrum (including the best in original data) and reflectivity spectrum. Reproduced from [156]. Copyright 2023, Elsevier. (c) Simulated and predicted resonance frequency of kNN, SVM, and XGBoost versus HFSS. Reproduced from [157]. Copyright 2022, IEEE. (d) Flowchart of the process to predict metasurface structural parameters combined with RF generation process [160]. (e) Absorption spectra obtained from FDTD simulation of the predicted parameters for “90%-AM” and “95%-AM” [160].

In addition to SVM and kNN, another widely used ML algorithm for classification and regression is Random Forest (RF). RF involves processes such as random bootstrapping, decision tree construction, feature variable selection, node splitting, and ensemble prediction. It constructs multiple decision trees and integrates their predictions through voting or averaging to achieve high-performance classification and regression tasks [159]. Ding et al. [160] proposed a method using the RF algorithm to design patterned graphene metasurface absorbers. The design process, as shown in Figure 5(d), addresses the complex impedance matching and field excitation involved in the electromagnetic absorption process of patterned graphene metasurfaces. It is difficult to determine the effects of various parameters. By treating the structural parameters of patterned graphene as feature variables, RF effectively minimizes the correlations between decision trees through randomness, thereby improving the accuracy of predicting absorption parameters. Based on the trained RF model, the predicted maximum effective absorption bandwidth and maximum perfect absorption bandwidth were 3.77 THz and 2.54 THz, respectively, along with the corresponding structural parameters. Figure 5(e) shows the FDTD simulation verification based on the predicted parameters, resulting in bandwidths of 3.83 THz and 2.57 THz, with a MAPE of only 1.36 %. This demonstrates the sufficient predictive accuracy of the RF regression model in the design of patterned graphene metasurfaces. Additionally, to showcase the superiority of RF, they also built other classical ML regression models, including linear regression (LR), SVM, least squares (LS), and kNN. Only RF achieved an R-squared error of above 0.9, while the remaining models had an average value of only 0.78. Compared to the previous two algorithms, RF is more suitable for handling high-dimensional and large-scale datasets, and it possesses good generalization ability and robustness. This is due to the ensemble learning framework on which RF is based, where individual decision tree predictions can often be corrected by other decision trees, and it is not sensitive to parameter selection and noise influences.

3.1.2 Unsupervised learning

The mentioned algorithms are all supervised learning algorithms, where the training dataset consists of input samples along with their corresponding labels or outputs. The goal of these algorithms is to establish a relationship between the inputs and outputs to make predictions for unseen inputs. On the other hand, unsupervised learning involves only input samples without corresponding labels, and the objective is to discover inherent structures, patterns, or features within the data. Common unsupervised learning tasks include clustering [161], [162], density estimation [163], [164], anomaly detection [133], [165], and more.

Lin et al. [166] proposed a genetic-type tree search algorithm (GTTS) combined with unsupervised clustering for the automatic inverse design of metasurfaces that enable high-directional beam steering. The problem of specifying the number of nanoscale antennas and optimizing their arrangement to achieve the desired far-field radiation pattern is a nondeterministic polynomial hard (NP-hard) combinatorial problem. It is a nonconvex optimization problem. To address this, they applied the GTTS algorithm and introduced the concept of virtual space for efficient optimization, as shown in Figure 6(a). First, they constructed a database V containing n nanoscale antennas. By using existing k-means solvers and unsupervised SVM, the antennas were clustered into K groups. Nanoscale antennas within the same group exhibited different but similar physical characteristics. The k-means solver also outputted a center-average structure C, which represents the average of all similar nanoscale antennas and can represent the entire group. Therefore, the search was performed only on the library with K virtual nanoscale antennas, and the resulting metasurface was similar to searching the entire library. Then, refinement was applied to each unit to transform the optimal virtual design back to the original space. Compared to commonly used adjoint optimization, the GTTS method significantly reduced the computational time. The original structure yielded nonintuitive amplitude and phase distributions. By utilizing GTTS, both parameters could be simultaneously optimized to deflect the incident light to the target angle. To further validate the optimization results, numerical simulations were conducted, which showed the far-field radiation intensity of nanoscale antennas with the same arrangement. The simulations considered near-field coupling, while the theoretical calculations treated the nanoscale antennas as independent dipole sources. The agreement between the two approaches confirmed the assumption of neglecting internal coupling during the GTTS optimization process. The researchers successfully designed a metasurface with high-performance active beam steering capability. Figure 6(b) illustrates the relationship between the normalized far-field radiation angle and the steering angle for the designed electrically tunable metasurface in the mid-infrared range.

Figure 6:

Classical unsupervised machine learning for metasurface design. (a) Flowchart of GTTS combined with unsupervised clustering, and comparison of time cost of adjoint method and GTTS. Reproduced from [166]. Copyright 2021, American Chemical Society. (b) Comparison of normalized far-field radiation intensity versus steering angle obtained from theoretical calculation and numerical simulation. Reproduced from [166]. Copyright 2021, American Chemical Society. (c) Results of K-means clustering using transmission curve and radial distribution function. The coordinates are calculated by PCA [167]. (d) Flowchart of seven refractory material meta-absorber design. Forward model shows the absorption prediction using regression tree algorithm while inverse model utilizes PCA before determining the design parameters. Reproduced from [168]. Copyright 2023, Elsevier.

Many ML training processes are based on an important assumption: the sampling density of the training samples is sufficiently high, known as the Nyquist sampling condition. However, as the dimensionality of the attributes increases, the number of samples required to satisfy the Nyquist sampling condition grows exponentially. Distance-based methods, such as SVM, encounter significant challenges in high-dimensional situations. The problems of sparse sample data and difficult distance calculations in high-dimensional spaces are common obstacles faced by all ML methods and are referred to as the “curse of dimensionality.” Although the curse of dimensionality is widespread, it is not unsolvable. In the field of design optimization for nanophotonic devices, although the collected data are high-dimensional, only a low-dimensional distribution relevant to the learning task may exist, known as a low-dimensional embedding in the high-dimensional space. Therefore, data can be processed through dimensionality reduction techniques.

Principal Component Analysis (PCA) is a classical dimensionality reduction method that has been widely used even before the advent of computers. Based on maximum separability, if the projections of all sample points in the high-dimensional orthogonal space onto a low-dimensional hyperplane can be maximally separated, the variance of the projected sample points should be maximized. Given the dimensionality d of the low-dimensional space and the sample set, the main idea of PCA is to decompose the covariance matrix of normalized samples into eigenvalues and select the eigenvectors corresponding to the largest d eigenvalues as the solution. Obviously, the low-dimensional space differs from the original high-dimensional space because some eigenvectors are discarded, but discarding this part of information increases the sampling density of the samples and also acts as a denoising effect. Hou et al. [167] introduced the reticular chemistry structure resource (RCSR) to design meta. They used PCA and the k-means algorithm to visualize and analyze the transmission curves of 72 metasurfaces. The results showed that the transmission curves could be classified into three main types, but a simple relationship between structure and transmission rate had not been found, as shown in Figure 6(c). This design approach provides a novel “alternative” method to modify the transmission curves of metasurfaces without focusing on adjusting the structural parameters of individual atoms. Ijaz et al. [168] developed multiple forward regression models in Figure 6(d), including Decision Tree Regressor (DTR), kNN, LR, and Polynomial Linear Regressor (PLR), to predict the absorption values of seven different materials. The dataset consisted of 16,807*7 responses, and to compensate for the data processing load, PCA was used to preserve the integrity of the data structure and eliminate redundancy by projecting the high-dimensional information onto a lower-dimensional subspace.

There are many unsupervised learning algorithms that can be applied to metasurface design, such as autoencoders based on neural networks, which will be discussed in the next section. Various algorithms, their variations, and combinations are beyond the scope of this discussion, but interested readers can refer to machine learning–related literature for more information [169], [170], [171].

3.2.1 Supervised learning

The performance of classical ML algorithms largely depends on the representation, also known as features, of the given data. The core of many AI tasks is to extract a suitable set of features and provide them to simple ML algorithms. However, when facing complex real-world tasks, it is difficult to determine which features should be extracted due to the different perceptions of humans and computers. Representation learning is a way to address this issue by using machine learning to discover representations themselves, rather than just mapping representations to outputs. The learned representations are often better than manually designed ones and require minimal human intervention to adapt quickly to new AI tasks. Deep learning (DL) is a prominent approach in representation learning, which expresses complex representations through simpler ones, allowing computers to represent complex concepts using simpler concepts [172]. DL is essentially a branch of ML but is well-known for its use of Artificial Neural Networks (ANNs). DL utilizes hierarchical structures to learn multiple levels of data abstraction, providing an efficient method for designing photonic structures [173]. The following paragraphs will introduce the applications of several commonly used DL models in metasurface design.

Supervised deep learning models, often referred to as discriminative models in the context of metasurface design, include the basic Artificial Neural Network (ANN) called the Multiple Layers Perceptron (MLP) [174]. MLP is composed of multiple layers of nested nonlinear functions, where each layer is a nonlinear function of the form h = g(W ^T x + b). Here, x is the input vector, W and b are the weight and bias parameters, and g is the activation function. In theory, an MLP with a sufficient number of layers can approximate any nonlinear function [175]. Figure 7(a) demonstrates a backpropagation neural network proposed by Wang et al. [176] for predicting the phase and phase gradient of meta-units with different structural parameters. An ideal achromatic lens should possess a dispersed phase distribution φ r , ω . By considering the focal shift effect, the phase can be expressed as φ r , ω = − ω 0 c r 2 + k 2 × ω 0 − 2 + α ω 0 2 + β ω 0 c + γ and the phase gradient is defined as φ ′ = ∂ φ ∂ ω ω = ω 0 = − r 2 c r 2 + k 2 × ω 0 − 2 + 2 α ω 0 + β c , which is based on the radius (r). They designed three styles of meta-atoms: a cross, a hollow circle, and a hollow square. The height and period values were fixed, and the overall structure was described by only two parameters. Therefore, the input and output dimensions of the network were both 2. They trained the network using only 695 data samples, and the network generated over 15,000 sets of structure-phase data within 0.67s. Based on the predicted data, they designed a chromatic aberration-free metalens for the 420–640 nm bandwidth. Although the phase response and meta-atom size have a highly nonlinear relationship, a neural network with a small input and output dimension, such as an MLP with two hidden layers, can accurately describe their relationship. This approach alleviates the computational burden. An et al. [177] designed a tandem network model, as shown in Figure 7(b), which consists of two MLPs connected together. The forward network is used to fit the input meta-unit parameters S ̃ and the output EM response R ̃ , while the inverse network predicts the parameters S ̂ from the desired EM response R ̂ . They also introduced a Neural Tensor Layer (NTL) to match the dimensions between the inputs and outputs. Two datasets were collected for two different structures, and the networks were trained using the Mean Squared Error (MSE) loss function. After 500 epochs, the MSE values were 0.098 and 0.147, respectively, which closely matched the responses obtained through FDTD calculations, demonstrating the network’s ability to predict electromagnetic responses. Although MLPs have demonstrated strong fitting capabilities, there are still limitations. For instance, MLPs are not suitable for models with excessively large or extremely mismatched input and output dimensions. Additionally, MLPs do not provide a fully automated optimization process and require adjusting the network structure and hyperparameters based on experience [178].

Figure 7:

Supervised ANN for metasurface design. (a) Flowchart of the backpropagation neural network for predicting phase response and phase gradient. Reproduced from [176]. Copyright 2022, Wiley Materials. (b) Schematic of tandem neural network. The lower left shows the phase errors before (red) and after (green) DNN optimization. The lower right is an example of the target fitting method. Reproduced from [177]. Copyright 2021, Optica Publishing Group. (c) Flowchart of the CNN-based predicting neural network for metasurface absorber design. Reproduced from [180]. Copyright 2022, Elsevier. (d) Comparison of the predicted absorption spectra (red) and simulated absorption spectra (blue). Reproduced from [180]. Copyright 2022, Elsevier. (e) Flowchart of the combined deep network. CNNs are used to extract spatial data from smaller parts of the image while RNNs are used to find the required optical response based on the features extracted from CNNs [184].

Most metasurfaces are two-dimensional structures, which mean that the structure of a meta can be viewed as image data in a 2D pixel grid. Convolutional Neural Network (CNN) is a type of neural network specifically designed to handle data with grid-like structures [179]. The basic structure consists of three layers: the convolutional layer performs convolutional operations on the input 2D image using kernel functions, generating a set of linear activation responses to extract image features. The second layer is the detector layer, where each linear activation response passes through a nonlinear activation layer. Finally, the pooling layer replaces the network’s output at a particular position with aggregated statistical features of neighboring outputs, adjusting the output of the detection layer. CNN can be seen as an MLP with infinitely strong priors, indicating that the learned functions at each layer only contain local connectivity relationships. Additionally, the pooling layer serves as a prior that introduces translational invariance to the input. Compared to MLP, CNN has fewer trainable parameters. Donda et al. [180] constructed a CNN-based Prediction Neural Network for the forward design of metasurface absorbers, predicting the absorption spectrum based on the structure of the metasurface. To facilitate the learning process of the neural network, they initially processed the absorption spectrum data by using PCA to reduce the dimensionality from 201 to 20. As shown in Figure 7(c) and (d), the surface of the metasurface was transformed into a binary image of 64 × 64 pixels, and additional channels for thickness and incident angle were included as inputs. After passing through three convolutional-pooling layers, batch normalization, and dropout layers, the network outputted a spectral curve with 20 points, achieving an MSE of only 0.00904. Subsequently, an ablation analysis was conducted to demonstrate the rationality of the network parameters. Additionally, they constructed a CGAN-based network architecture for inverse design, which could predict the metasurface structure with specified absorption spectrum and thickness given the incident angle. The final designed metasurface absorber had a thickness of only λ/64 and exhibited nearly omnidirectional absorption performance at 82 Hz. CNN is often used in combination with other networks such as Recurrent Neural Networks (RNNs) [181], Residual Networks (ResNets) [182], and U-net [183]. Figure 7(e) illustrates the combined neural network model proposed by Sajedian et al. [184]. The model consists of a CNN with residual connections and a small-scale RNN. ResNet incorporates multiple layers of CNN, where the output of a particular layer is added to the output of the next layer. This enables gradient backflow to earlier layers, allowing for the design of deeper networks without encountering vanishing gradients. As the model deepens, it extracts higher-level features. They trained the model using 100,000 data pairs and achieved near-perfect predictions for all 10,000 results in the validation set. The final model can predict the optical response of any structure based on an image within 1 s, showcasing the potential of CNN in metasurface inverse design.

3.2.2 Unsupervised learning

Unsupervised deep learning is contrasted with supervised deep learning, which encompasses generative models [185], [186]. Representative algorithms in this category include Autoencoder (AE) and Generative Adversarial Network (GAN). After training, an AE can copy the input to the output, consisting of an encoder function h = f(x) and a decoder function y = g(h). It is worth noting that AE does not aim to learn the exact solution of g(f(x)) = x but rather achieves an approximate replication by imposing constraints. This approach often enables the learning of useful features from the data. These constrained models consider which parts of the data should be prioritized for replication, allowing them to learn useful features of the data. In the metasurface design process, AE is often used for dimensionality reduction, and the learning process can be succinctly described as minimizing L(x, g(f(x))). When the decoder is linear and L represents mean squared error, the AE learns the same generative subspace as PCA. However, if both f and g are nonlinear, the AE can learn a more powerful nonlinear extension of PCA. Qiu et al. [187] proposed a three-step metasurface design method framework called REACTIVE, where Cepstrum transformation was initially used for feature extraction to reduce the dimensionality of the input metasurface structure from 1000 to 200. Subsequently, an under-complete AE in Figure 8(a) further extracted 64-dimensional features. Compared to other ML algorithms, the REACTIVE method achieved higher accuracy. Ahmed et al. [188] addressed multiple design constraints such as S-parameters, gain, radiation efficiency, and radiation pattern by proposing a maximum likelihood model composed of cascaded AE and SVR. In Figure 8(b), AE was employed for the dimensionality reduction of multiple antenna performance metrics, and the model could predict the design variables of nanoantennas based on the desired performance metrics. AE has a famous variant, which is called variational autoencoder (VAE). This model can realize both the forward prediction of optical responses of a certain structure and the inverse design of structures from preconfigured optical properties [189].

Figure 8:

Unsupervised ANN for metasurface design. (a) Under-complete AE in REACTIVE, reducing the number of data dimensions from 1000 to 200. The proposed REACTIVE has higher accuracy compared to other ML models [187]. (b) Flowchart of training process model based on AE and SVR. Reproduced from [188]. Copyright 2023, Wiley Materials. (c) Architecture of the GAN-based optical design. The whole network has three parts: generator, simulator, and critic. Reproduced from [192]. Copyright 2018, American Chemical Society. (d) Flowchart of the generative meta-atom design network based on WGAN. Generator and discriminator approach the ground truth via this adversarial process. After training, translate the desired function to phase and amplitude as the input of generator, then it can generate actual meta-atom arrays. Reproduced from [193]. Copyright 2021, Wiley Materials. (e) kNN-based classification for metasurface cells. The entire sample of metasurface cells is divided into a finite number of classes according to their phase curves. Reproduced from [194]. Copyright 2022, IEEE.

Whether it is forward design or inverse design, human intervention is required for data selection and analysis. However, the existence of GANs breaks through this limitation [190], [191]. Many generative models are based on the idea of differentiable generator networks, which employ a differentiable function g, usually a neural network, to transform latent variables z into distributions on samples x. GANs, proposed based on game theory, consist of two components: the generator, responsible for generating samples x = g z ; θ g , and the discriminator, responsible for judging the authenticity of the samples. The generator aims to deceive the discriminator as much as possible, while the discriminator strives to distinguish samples drawn from the generator from real samples in the training set. They engage in a game to improve performance. Unlike traditional machine learning algorithms, GANs are often difficult to evaluate in terms of loss because the generator and discriminator are fundamentally in an adversarial relationship, and there is no convergence of the loss function. In such cases, alternative methods are required to measure the loss. GANs have been successfully used to construct complete and fully automated optimization design programs, reducing human intervention, and have been proven to be efficient and effective in metasurface design. Liu et al. [192] employed GANs to search for structures that yield the desired input spectra, enabling the network to generate metasurface patterns based on arbitrary input optical responses. The metasurface used for training consist of a single layer of gold, and using finite-element modeling (FEM), the transmission spectra of metasurface with different shapes under x and y polarizations within the visible to near-infrared range were simulated. As shown in Figure 8(c), the structures were simultaneously represented as binary images of 64*64 pixels. The trained generator was able to approximate the transmission rate at each frequency point with an average error of less than 0.01. The model could handle multiple input spectra without sacrificing accuracy and was easily scalable to more complex structures. The work by An et al. [193] utilized a more stable variant of GAN called Wasserstein-GAN (WGAN) and proposed a novel metasurface design approach. By introducing Wasserstein distance as a loss evaluation metric, the training process was stabilized, enabling the network to be widely applicable in metasurface design. In Figure 8(d), both the generator and discriminator are essentially CNNs, as mentioned earlier, capable of capturing the nonlinear relationship between structure and response. A preprocessing layer was added before the first input layer, transforming phase and amplitude responses into complex transmission coefficients. The conditional x-target pairs were combined with noise to confuse the discriminator. This network exhibits high stability and ease of convergence. Once the training is completed, for different applications, the desired responses can be input into the generator to design the desired distribution of meta-atoms. They successfully applied this network to design a bifocal lens, a polarization multiplexed deflector, a polarization-multiplexed lens, and a polarization-independent lens. Huang et al. [194] used a combined model of GAN and kNN, where preclassifying the dataset with kNN effectively enhanced the learning capacity of GANs, as shown in Figure 8(e).

In addition, there are emerging ML algorithms such as Reinforcement Learning (RL) [195], Deep Q-Learning (DQN) [196], Transformer [197], and Normalization Flows (NF) [198].

In this section, we have discussed many kinds of AI algorithms. According to the characteristics, advantages, and limitations of each algorithm, we summarize them in Table 1. In practical research, attention needs to be given to the form of the dataset. The input and output dimensions of the dataset vary depending on the structural parameters and target responses of different metasurface. One simple solution is to increase the number of samples in the dataset, as more samples imply more features, which can help improve the training accuracy of ML algorithms. However, in many cases, due to limited sample acquisition or chaotic distribution of data, it is necessary to select suitable ML algorithms for specific problems. Furthermore, relying on experience or other optimization algorithms to adjust the parameters of ML algorithms is often required. This work currently lacks a definitive conclusion because of the “No Free Lunch” theorem, which states that the expected performance of all learning algorithms is the same [199]. Therefore, it is crucial to focus on specific application tasks and the selection of a learning algorithm that matches the problem, as it plays a vital role in research.

4.1.1 High quality imaging

Realizing high-quality imaging using metasurface is a key technology aimed at obtaining clearer, more accurate, and detailed images, and it has been widely applied in fields such as science, medicine, engineering, and others. AI plays an important role in high-quality imaging by leveraging its powerful image processing and analysis capabilities to assist in achieving higher resolution, contrast, and accuracy.

In common back-end image processing, machine learning learns and captures patterns and features of images from a large amount of training data, enabling tasks such as denoising, image restoration, autofocus, object detection, and image classification. Many machine learning techniques have achieved remarkable results in spectral reconstruction processing of metasurface imaging systems. In 2018, Colburn et al. [207] designed a metalens with extended depth of focus (EDOF) that produced spectrally invariant blur, with increasing bandwidth as the depth of focus increased, showed in Figure 9(a). To obtain accurate spectral information from the blurred images, the authors performed preprocessing on the captured images and then utilized a computational deconvolution method for reconstruction, achieving high-quality wideband full-color imaging under white light illumination, as shown in Figure 9(b). This approach, which optimizes metasurface chromatic aberration through algorithms, had provided new insights for the design of various optical systems. Yang et al. [34] employed freeform-shaped meta-atoms for on-chip hyperspectral imaging. As shown in Figure 9(c) and (d), to evaluate the spectral reconstruction performance of the selected transmission spectra at different minimum feature sizes, the authors constructed a spectral encoder–decoder network. They used synthesized Gaussian linear spectra as input spectral datasets and simulated the spectral measurement process using the encoder, followed by training the spectral decoder for spectral reconstruction. In this work, the imaging system achieved a spectral resolution of 0.5 nm, with an average fidelity of 98.78 % for spectral reconstruction on a standard color chart. The spatial resolution was 356 × 436, demonstrating excellent performance.

Figure 9:

AI-assisted high-quality imaging. (a) EDOF metalens can extend sufficient depth of focus using a cubic phase mask. On the sensor, we can see that the green light is defocused in this plane. The MTFs of EDOF metalens are similar for all wavelengths and have higher cutoff spatial frequencies than red and blue metalens [207]. (b) Landscape image of text and colorful pattern under white light illumination captured by EDOF metalens and reconstructed by Wiener filter [207]. (c) Schematic diagram of the spectral encoder–decoder network [34]. (d) A 356 × 436 × 601 data cube of a plate of fruit spectral images is reconstructed from the metasurface modulated original image [34]. (e) The entire end-to-end imaging process is composed of a metasurface imaging process model and a neural deconvolution algorithm. The image on the right shows a blue flower reconstructed by the proposed neural nano-optics [41]. (f) The combination of deep neural network and hyperspectral image prior network forms the overall architecture of hyperspectral image reconstruction. With the help of this network, the experiment reconstructed hyperspectral images of 25 spectral channels from 460 nm to 700 nm. Reproduced from [208]. Copyright 2023, Elsevier.

In addition to direct back-end processing of imaging, the entire imaging process can be viewed as a unified entity, known as “end-to-end” design. By introducing the design parameters of metasurface and the output image parameters as variables to be optimized, the imaging system can be connected from the front-end to the back-end, achieving both structural and image optimization through a complete AI workflow [34]. Tseng et al. [41] proposed neural nano-optics and designed and validated a high-quality, polarization-independent metasurface imaging system using an end-to-end architecture. The imaging process was divided into three parts: metasurface phase determination, PSF simulation and convolution, and sensor noise. This imaging system achieves simulation accuracy comparable to FDTD but with a speed advantage of three orders of magnitude and a memory requirement reduced by 3000 times. Unlike conventional methods, the neural deconvolution approach was applied to the learned feature space rather than the original image intensity during the back-end reconstruction. This technique combined model-based deconvolution generalization with effective feature learning in neural networks, enabling image deconvolution of meta-optics with severe aberrations and large spatial range PSFs. As shown in Figure 9(e), the simulation of the imaging process and the deconvolution algorithm constituted a fully differentiable end-to-end design chain, resulting in a faster and higher-quality metasurface imaging system. After training and optimization, this nanophotonic imaging device reconstruction method generated RGB images of 720 px*720 px in just 58 ms, with a MSE 10 times lower than existing methods, and achieved a spatial resolution of 214 lp/mm in all color channels at a distance of 120 mm. Similar end-to-end algorithms have also been used to design a compact and portable instantaneous hyperspectral imaging system [208]. To achieve high-quality spectral reconstruction from RGB three channels to multiple channels, a spectral prior-unfolding neural network was introduced. The network for computing the spectral prior consisted of a classic U-Net [209], which learned how to extract spectral details from a large training dataset of hyperspectral images. Figure 9(f) shows the overall network. In the iterative training process, the input data passed through the prior network to generate a high-quality spectral image prediction, continuously optimizing the loss between the computed and real images. Based on this end-to-end algorithm, the experiments demonstrated better hyperspectral reconstruction results compared to traditional separate optimization methods, showing significant advantages in terms of average peak signal-to-noise ratio (PSNR). AI techniques automate and optimize each step, simplifying the process and improving efficiency, leading to remarkable advancements in metasurface imaging systems, potentially becoming the next generation of high-quality cameras.

4.1.2 Multidimensional imaging

AI-assisted metasurface imaging systems have made significant achievements in providing high-quality imaging. However, to further enhance our understanding and analytical capabilities of complex problems, multidimensional imaging becomes particularly crucial. Multidimensional imaging not only captures more information but also provides different perspectives, revealing a more comprehensive image. AI-assisted multidimensional imaging can innovatively integrate multidimensional data into a unified framework and analyze, interpret, and extract useful information using deep learning and pattern recognition techniques, thereby achieving 3D and even 4D imaging.

Jing et al. [210] designed and validated a system for active 3D positioning and imaging using single-frequency fringe illumination. The researchers introduced a metasurface as an encoded light source into the imaging system, utilizing geometric phase design to encode fringes and obtaining deformed fringe images in reflective scenes during experiments. As shown in Figure 10(a), to address the depth ambiguity of the single-fringe approach, the article introduced a calibration model with polar constraints and similarity search and employed the alternating direction method of multipliers (ADMM) algorithm to achieve depth reconstruction with depth deviations smaller than 0.5 mm within 300–400 mm. Furthermore, a correspondence between depth and image coordinates was established, ultimately achieving high-precision 3D imaging, showed in Figure 10(b). Hua et al. [42] realized an imaging system that captures both the light field and spectral information in a single shot using a lateral-dispersion metalens array. This ultra-compact spectral light-field imaging (SLIM) system consists of a 48 × 48 array of superchromatic metalens and a monochrome CMOS sensor, where different wavelengths of the spectral light field overlap on a 2D receiving plane, causing blurring in the captured patterns. To utilize the additional information from these blurs and obtain complete spectral information, the authors proposed an algorithm for super-resolution spectral reconstruction. Figure 10(c) shows the flowchart. This algorithm transformed the reconstruction task into a solved convex optimization problem, effectively recovering high-quality multispectral images from monochrome blurred images by combining a forward model established based on the design parameters of the metalens and regularization constraints. To achieve higher spectral resolution, the researchers trained a spectral super-resolution network using paired low-resolution and high-resolution spectral data as input and output, placing this network as a super-resolution module after the reconstruction algorithm. Based on this algorithm, SLIM improved the spectral resolution to 4 nm and obtained depth information by calculating the disparity between adjacent metalens, enabling “x + y + z + λ” 4D imaging. In the experiments, the authors further processed the spectral images, such as differential amplification to distinguish peaks, effectively differentiating objects with similar spectra but different materials based on depth information.

Figure 10:

AI-assisted multidimensional imaging. (a) Calibration algorithm. A single illumination stripe can create spatial ambiguity due to the single-item coding properties. This algorithm calibrates sub-pixels through geometric constraints and similarity requirements of x _i , thereby establishing the relationship between depth P and (x ₀, x _i ) [210]. (b) Perspective view of reconstructed facial 3D imaging [210]. (c) SLIM system. The 4D data cube of “x + y + z + λ” is decoupled into multi-view aliased information through modulation of the metalens array and captured by the camera. Finally, the spectral reconstruction algorithm and super-resolution module proposed in the article are used to achieve 4D imaging. On the right is a scene captured by SLIM consisting of four letters of different spatial positions and colors [42]. (d) Multichannel end-to-end design. Multichannel ground-truth images can be continuously cut: 3D objects are divided into sets of multiple 2D depth slices, 2D depth slices can be cut into sets of different color slices, and different color slices can be cut into sets of different polarization slices. Applying these slices to the end-to-end architecture results in an overall multichannel optimization. Reproduced from [200]. Copyright 2022, Optica Publishing Group. (e) Underwater depth-sensing results. The left column is the raw image captured by the left meta-lens. The middle column is the depth map. The right column is the integration of two images [211]. (f) The left picture is architecture of the neural networks for light-field net (LFN). The right picture is variation of distance measured by light-field versus the known reference of the stairs. Reproduced from [212]. Copyright 2023, Wiley Materials.

The aforementioned end-to-end design can also be assisted by improved algorithms for multidimensional imaging. Lin et al. [200] combined Tikhonov regularization with an end-to-end architecture for inverse design of metasurface and replaced the backend with compressive sensing, thereby increasing channel capacity at the cost of stronger priors. As shown in Figure 10(d), they optimized the design of a single nanophotonics structure and achieved reconstruction of multiple depth, spectral, and polarization channels from a single monochromatic image. In the experiments, the metasurface design of a 16-color imager achieved a reconstruction error of 5–12 % (lower than 1 % image noise), the 4-color/4-depth imager had an error of 5 %, and the 2-color/2-depth/4-polarization imager had an error of 2 %.

AI has also facilitated multidimensional imaging in special environments. In recent years, Tsai’s research team has developed a new underwater binocular stereo vision and deep sensing system [211], which incorporates AI-supported GaN binocular metalens. This system meets the imaging requirements for underwater applications, such as compactness, lightweight design, and low power consumption. As shown in Figure 10(e), they propose a generalized depth calculation formula applicable to binocular vision systems of various sizes. Combined with deep learning support, real-time processing capabilities for rapid underwater object imaging and depth calculation are achieved, with a depth measurement accuracy of up to 50 μm. Additionally, the superhydrophobic nature of the metalens endows the underwater binocular system with properties such as antiadhesion, pollution resistance, and self-cleaning, making it highly promising for wide applications in underwater robotics, submarines, and marine machine vision. Furthermore, their team has recently developed an intelligent meta-device [212] that integrates light field imaging and structured light systems, utilizing AI algorithms to achieve depth perception, particularly excelling in low-light environments. On one hand, in the light field–based depth perception mode, AI algorithms are used to process the light field images captured by the metalens array, generating depth maps, and conducting more in-depth image analysis through neural networks. On the other hand, in the chromatic aberration-based structured light system, the meta device adopts a structured light pattern, such as structured light patterns generated by a 532 nm laser. As illustrated in Figure 10(f), the article employs deep learning algorithms and introduces a structured light network (SLN) for semantic recognition of the structured light images. The entire system achieves precise perception of object depth under low-light conditions, demonstrating remarkable adaptability.

4.2.1 Material sensing

Metasurface technology plays a significant role in material sensing, particularly enabling high-precision spectral detection. This technology has been widely applied in the fields of biology and chemistry [214], [215], [216], [217]. In the field of biology, metasensors are commonly used to detect biomolecules, cells, microorganisms, and other biological entities. In the field of chemistry, metasensors are employed for analyzing chemical reactions and material properties. The precise spectral analysis provided by metasurface sensors holds great potential for important applications in medical diagnostics, environmental monitoring, and the study of new materials.

In the field of biology, mid-infrared spectroscopy is often utilized for identifying various biomolecules such as proteins, lipids, nucleic acids, and polysaccharides. To meet this demand, researchers, such as John-Herpin et al. [218], have designed a broadband nanoplasmonic metasurface for surface-enhanced infrared absorption spectroscopy (SEIRAS). By tuning multiple resonant peaks, they covered the absorption range from below 1000 cm⁻¹ to above 3000 cm⁻¹. As shown in Figure 11(a), they constructed a fully connected neural network model with an input layer of 1089 nodes and an output layer of 4 nodes. The training set consisted of over 3 million spectral data points obtained from various concentrations of four analytes. The trained deep neural network model, integrated with a microfluidic chip-based metasensor, formed a sensing system capable of real-time monitoring of lipid vesicle capture, cargo release induced by melittin (a small cell lytic peptide), and the transition of partial lipid vesicles to support lipid bilayer (SLB). In addition to mid-infrared spectroscopy, Raman scattering spectroscopy can also be employed for analyzing molecular structures and chemical compositions. Ahmadivand et al. [222] utilized GA to design and optimize metasurfaces for surface-enhanced Raman scattering (SERS) substrates, enabling the detection of SARS-CoV-2 virus. To analyze the SERS data, the authors collected hundreds of Raman spectra from different samples. After preprocessing steps such as normalization, feature scaling, and Savitzky–Golay filtering, the data dimension was reduced using PCA algorithm. In the experiments, a supervised learning model, SVM, was trained using data from 15 PCR-positive samples and 12 PCR-negative samples for virus identification. The results demonstrated that this metasurface sensor combined with a machine learning model could accurately detect the presence of the virus in clinical samples, even in the presence of highly heterogeneous backgrounds, such as saliva samples, shown in Figure 11(b).

Figure 11:

AI-enhanced material sensing. (a) Spectrotemporal data points from a dual cargo release experiment were fed into a real-trained DNN to predict the scaled DNN output weight curves for each analyte [218]. (b) In order to find the best model for virus classification, 5 different algorithms were tested using KFold cross-validation (k = 10). During training, the SVM model was used to measure the confusion matrix of the sample Raman spectrum. In clinical testing, this metasurface sensor successfully differentiated samples collected and compared their average spectra with inactivated virus samples [219]. (c) Through PCA of the data after interpolation, the experiment finally obtained retained data points to perform concentration verification and retain the verification results in the confusion matrix. Reproduced from [220]. Copyright 2022, Wiley Materials. (d) Architecture of the FAS classifier. By adopting the self-attention mechanism using transformer encoder, not only are the features of each spectrum sample analyzed respectively but also the cross-correlations of 32 samples are taken into account. Finally, the output vector by transformer encoder is sent to a MLP to get the final FAS results. The figure on the right shows the results of a live face captured and the face results displayed on the screen under sunlight, verifying its antispoofing recognition capabilities in the real world [221].

In the field of chemistry, the identification capability of mid-infrared spectroscopy has been applied in industrial production, food production, and environmental monitoring of chemical substances. Meng et al. [220] developed a compact mid-infrared spectrometer for the identification and quantification of chemical substances. The plasma metasurface, composed of gold nanostructures on an undoped silicon substrate, was integrated into a filter chip containing 10 bandpass filters and 10 bandstop filters, covering the mid-infrared range of 6–14 μm. The entire microspectrometer system consisted of the filter chip and a microbolometer thermal imaging sensor. As shown in Figure 11(c), by collecting the sensor outputs of different concentrations of chemical solutions and using machine learning algorithms such as PCA and SVM for training, a 100 % classification accuracy was achieved for ethanol, isopropanol, acetone, and ethyl acetate in cyclohexane. Furthermore, this mid-infrared microspectrometer combined with machine learning algorithms enabled the effective identification and quantification of various complex samples, such as three drugs (acetaminophen, ibuprofen, and aspirin) with a 100 % discrimination accuracy and three edible oils (olive oil, rapeseed oil, and peanut oil) with a discrimination accuracy ranging from 94 % to 100 %.

The application of AI in material sensing systems provides examples of data analysis and model training, which are also crucial for novel biometric technologies such as facial recognition. Rao et al. [221] achieved facial recognition through a snapshot hyperspectral imaging sensor based on metasurface nanostructures. This imaging sensor accurately measured the facial reflectance spectra and revealed the absorption peaks of hemoglobin. In the experiments, this sensor, combined with convex optimization methods and compressed sensing algorithms, constructed a practical antispoofing facial system, showed in Figure 11(d), achieving an accuracy of 97.98 % in real-world testing scenarios.

AI accelerates data processing, improves recognition accuracy, and identifies subtle signals and patterns, making material sensing systems more intelligent and adaptive.

4.2.2 Motion sensing

By manipulating the electromagnetic (EM) wave field, metasurface can detect dynamic parameters such as object position, velocity, and direction. The application of AI enables motion sensing systems to process a large amount of dynamic data in real-time, recognize complex motion patterns and trends [223], [224], and thus, it has important value in applications such as intelligent surveillance, robot navigation, gesture recognition, and virtual reality (VR).

In 2019, Cui’s research team [202] integrated programmable metasurfaces with deep learning techniques to create an intelligent metasurface recognizer. As shown in Figure 12(a), this programmable metasurface consisted of 24 × 24 digital meta-atoms, with each meta-atom integrating a PIN diode for control. The entire recognition system consisted of three ANNs, where IM-CNN-1 was a CNN that transformed the received electromagnetic wave data into an image encompassing the entire human body, providing a data foundation for subsequent processing. FASTER R-CNN was a commonly used object detector that identified the positions of hands and chests in the entire image, enabling more accurate tracking of gestures and breathing actions. IM-CNN-2 was another CNN used to directly process the raw electromagnetic wave data. Experimental results are showed in Figure 11(b) and (c), which demonstrated that this metasurface sensing system achieved various precise recognition and detection functionalities: accurate recognition of different hand poses such as stretching and fist clenching, differentiation of normal breathing and breath-holding states, and even detection of movements behind obstacles 5 cm thick by analyzing the reflections of microwave signals. There is also an intelligent prosthetic recognition system based on noncontact electromagnetic sensing [225]. Programmable metasurface technology was employed to perceive limb movements by manipulating the focus of electromagnetic waves. Researchers focused electromagnetic waves on the measured arm to obtain echo data associated with different gestures and actions. After cleaning and standardizing the data, feature extraction was performed using Fisher scores to ensure its quality and practicality. As shown in Figure 12(d), the recognition system trained the collected dataset using the One-vs-One Support Vector Machine (OVO-SVM) algorithm to differentiate different gestures and actions. Experimental results demonstrated successful classification and recognition of 18 hand gestures, including hands and wrists, by the programmable metasurface after training. It also showed that using multiple foci improved both classification accuracy and sample quantity compared to a single focus. These studies open up more possibilities for AI-integrated metasurface sensing systems in areas such as smart homes, medical detection, and security screening.

Figure 12:

AI-enhanced motion sensing. (a) The schematic configuration of intelligent metasurface system that uses a large-aperture programmable metasurface to adaptively manipulate and sample electromagnetic wave fields and to instantaneously control and process data streams [202]. (b) IM-CNN-2 processes data to recognize gestures and uses active microwave metasurface combined with IM-CNN-1 to achieve imaging behind a wooden wall [202]. (c) Identification of human respiration through time-frequency analysis of microwave data [202]. (d) The working principle of the noncontact hand gesture recognition method based on the programmable metasurface sample with a machine learning algorithm. The five different voltage distributions corresponding with phase distribution patterns can be obtained for controlling the transmissive programmable metasurface so that its radiation wave is focused onto the desired spots [225].