Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future Envision

In this survey, we summarize recent efficient deep learning infrastructures that may benefit current and future embedded computing systems towards ubiquitous embedded intelligence. In practice, some existing surveys [71, 72, 73, 74] typically focus on efficient deep learning algorithms, which may be out-of-date since recent deep learning infrastructures have been rapidly evolving, especially from the perspective of LLMs. In contrast to [71, 72, 73, 74], we focus on providing a more comprehensive and holistic view of recent efficient deep learning infrastructures for embedded computing systems, spanning from training to inference, from manual to automated, from convolutional neural networks (CNNs) to transformers, from transformers to vision transformers, from vision models to large language models, from software to hardware, and from algorithms to applications. Specifically, we discuss the following recent efficient deep learning infrastructures for embedded computing systems: (1) efficient manual network design, (2) efficient automated network design, f(3) efficient network compression, (4) efficient on-device learning, (5) efficient LLMs for embedded computing systems, (6) efficient deep learning software and hardware, and (7) efficient intelligent applications. We believe this survey has its merits and can shed light on future research, which can largely benefit researchers to quickly and smoothly get started in this emerging field. We demonstrate the organization of this survey in Figure 1, which is also summarized as follows:

At the end of each section, we also envision possible future directions in the respective fields, which have the potential to pave the way for future ubiquitous embedded intelligence.

As shown in previous state-of-the-art deep convolutional networks, such as AlexNet [76], VGGNet [1], GoogleNet [77], ResNet [2], DenseNet [3], and EfficientNets [78, 79], despite being able to push forward the attainable accuracy on ImageNet [80] from 57.2% [81] to 87.3% [79], network complexity has increased over time. We note that the convolutional network consists of convolutional layers, pooling layers, and fully connected layers, where most of the network complexity comes from convolutional layers [82]. For example, in ResNet50 [2], more than 99% of FLOPs are from convolutional layers. In light of this, designing efficient convolutional layers is critical to innovating computation-efficient convolutional networks. In practice, there are five typical efficient convolutional layers: pointwise convolution, groupwise convolution, depthwise convolution, dilated convolution, and Ghost convolution.

Built on top of the aforementioned efficient convolutional layers and structures, there are several representative families of manually designed efficient convolutional networks, including SqueezeNet [81], MobileNets [32, 85, 86], ShuffleNets [34, 35], CondenseNets [87, 88], GhostNets [36, 37, 75], and FasterNet [84]. We compare the above representative efficient convolutional networks in Figure 3,² which are also discussed in the remainder of this section.

SqueezeNet [81] is stacked using a series of Fire modules, which aims to achieve AlexNet-level accuracy with fewer parameters. Each Fire module consists of two convolutional layers, including one squeeze layer and one expand layer. In the squeeze layer, only pointwise convolutional layers are used to reduce the number of input channels for the subsequent expand layer. Next, the expand layer performs feature expansion using a pair of 1 × 1 and 3 × 3 convolutional layers. SqueezeNet is able to achieve slightly better accuracy on ImageNet than AlexNet (i.e., 57.5% in SqueezeNet vs. 57.2% in AlexNet) using a ×50 smaller model size. SqueezeNet is more compression friendly than AlexNet. For example, we are allowed to further compress SqueezeNet using a method outlined in [89], which delivers more compact network variants with \(\times 363\sim\)×510 smaller model size, and, more importantly, without degrading the accuracy on ImageNet.

MobileNets [32, 85, 86] are a family of lightweight convolutional networks, including MobileNetV1 [32], MobileNetV2 [85], and MobileNeXt [86], which are tailored for mobile devices with limited computational resources. MobileNetV1 is built upon a series of building blocks. Each building block consists of two convolutional layers, including one 3 × 3 depthwise convolutional layer and one 1 × 1 pointwise convolutional layer. With 569 M FLOPs and 4.2 M parameters, MobileNetV1 achieves 70.6% top-1 accuracy on ImageNet. MobileNetV2 is an improved version of MobileNetV1, which aims to unlock higher accuracy with fewer FLOPs and parameters. MobileNetV2 introduces the inverted residual building block that consists of three convolutional layers, including one 1 × 1 pointwise convolutional layer, one 3 × 3 depthwise convolutional layer, and one 1 × 1 pointwise convolutional layer. Here, the inverted residual building block also borrows the residual connection from ResNet [2] to stabilize the training process and improve the accuracy. With 300 M FLOPs and 3.4 M parameters, MobileNetV2 achieves 72.0% top-1 accuracy on ImageNet. MobileNeXt investigates the inverted residual building block in MobileNetV2 and introduces the sandglass block to enhance the accuracy without increasing the network complexity. The sandglass block consists of four convolutional layers, including one 3 × 3 depthwise convolutional layer, one 1 × 1 pointwise convolutional layer, one 1 × 1 pointwise convolutional layer, and one 3 × 3 depthwise convolutional layer. With 300 M FLOPs and 3.4 M parameters, MobileNeXt achieves 74.0% top-1 accuracy on ImageNet.

ShuffleNets [34, 35] are a family of efficient convolutional networks, including ShuffleNetV1 [35] and ShuffleNetV2 [34], which exploit channel shuffling to reduce the network complexity while maintaining competitive accuracy. Specifically, ShuffleNetV1, for the first time, introduces channel shuffling to enhance the information flow across different channels. In practice, the channel shuffling operation is inserted after the 3 × 3 depthwise convolutional layer to shuffle the feature maps from different groups, which is capable of generating richer and more diverse feature maps while not increasing the number of FLOPs and parameters. With 292 M FLOPs and 3.4 M parameters, ShuffleNetV1 achieves 71.5% top-1 accuracy on ImageNet, which is \(+\)0.9% higher than MobileNetV1 under comparable settings of FLOPs. ShuffleNetV2 improves the accuracy and efficiency of ShuffleNetV1 with several architectural modifications. ShuffleNetV2 first leverages channel splitting to divide the input feature maps into two parallel branches, one of which is fed into three convolutional layers, including one 1 × 1 pointwise convolutional layer, one 3 × 3 depthwise convolutional layer, and one 1 × 1 pointwise convolutional layer. After that, the above two branches of feature maps are concatenated along the depth dimension, which are then shuffled using the channel shuffling operation. With 299 M FLOPs and 3.5 M parameters, ShuffleNetV2 is able to achieve 72.6% top-1 accuracy on ImageNet, which is \(+\)1.1% higher than ShuffleNetV1 under comparable settings of FLOPs.

CondenseNets [87, 88] are a family of efficient convolutional networks, including CondenseNetV1 [87] and CondenseNetV2 [88], which are built upon another representative convolutional network named DenseNet [3]. CondenseNetV1 enhances the dense connection with a novel module called learned group convolution. Note that the dense connection reuses the features from preceding convolutional layers to enhance the information flow as seen in DenseNet. In contrast, the learned group convolution removes the redundant dense connection between different convolutional layers to reduce network redundancy. With 274 M FLOPs and 2.9 M parameters, CondenseNetV1 achieves 71.0% top-1 accuracy on ImageNet. CondenseNetV2 introduces an alternative named sparse feature reactivation (SFR) to increase feature reuse. Integrated with SFR, each convolutional layer can learn to (1) selectively reuse a set of the most important features from preceding convolutional layers and (2) actively update a set of preceding features to increase their reuse in subsequent convolutional layers. With 146 M FLOPs and 3.6 M parameters, CondenseNetV2 achieves 71.9% top-1 accuracy on ImageNet.

GhostNets [36, 37, 75] are a family of efficient deep convolutional networks, including GhostNetV1 [36, 75] and GhostNetV2 [37], which focus on generating rich feature maps using computationally cheap and simple yet powerful operations. To this end, GhostNetV1 introduces a powerful yet computation-efficient convolution dubbed Ghost convolution as shown in Figure 2, which consists of two sequential parts. The first part corresponds to the standard convolutional layer, in which the number of output channels is rigorously controlled. In the second part, a series of computationally cheap and simple linear operations are applied to the output feature maps from the first part to generate rich feature maps. With only 141 M FLOPs and 5.2 M parameters, GhostNetV1 achieves 73.9% top-1 accuracy on ImageNet. GhostNetV2 introduces a novel hardware-friendly attention mechanism, DFC attention, to enhance the learned feature maps to boost the expressiveness ability, which is seamlessly integrated into GhostNetV1 to push forward the accuracy and efficiency. For example, with 167 M FLOPs and 6.1 M parameters, GhostNetV2 is able to achieve 75.3% top-1 accuracy on ImageNet.

FasterNets [84] are built on the partial convolution. In contrast to the above efficient networks that typically optimize the number of FLOPs, FasterNet pioneers to design efficient networks with optimized FLOPS (i.e., FLOPs per second). The motivation behind FasterNet is that the on-device latency is determined by both FLOPs and FLOPS (i.e., Latency = FLOPs/FLOPS). To this end, FasterNet focuses on increasing the number of FLOPs to maintain competitive accuracy on the target task, while at the same time optimizing FLOPS to maintain competitive efficiency on target hardware. For example, compared with GhostNetV1x1.3, which involves 0.24 G FLOPs and exhibits 75.7% top-1 accuracy on ImageNet, FasterNet-T1 achieves \(+\)0.5% higher top-1 accuracy with many more FLOPs (i.e., 0.85 G) and, more importantly, achieves ×1.7 speedup on ARM processors.

In this section, we envision the future trends and possible directions of manual network design, including convolutional networks and transformers, summarized as follows.

The search space \(\mathcal {A}\) plays a prominent role in the success of NAS since the search engine of NAS strives to search for top-performing architecture candidates within the predefined search space. This also indicates that the search space determines the upper-performance limit of modern NAS algorithms. However, designing efficient and effective search spaces is quite difficult and challenging since there are a myriad of possible operator candidates (e.g., \(1\times 1\), \(3\times 3\), \(5\times 5\), and \(7\times 7\) convolutional layers) and different network configurations (e.g., the combination strategies of different operator candidates and the network channel layouts) [137, 138, 144]. Therefore, to reduce the search space size and trim down the search complexity, previous state-of-the-art NAS methods [137, 138, 144] often restrict the search space to allow efficient search and leverage modular search spaces, which are coarse-grained in contrast to layer-wise fine-grained search spaces. Previous state-of-the-art NAS methods are based on the following two representative types of modular search spaces, including cell-based search space and block-based search space.

Cell-Based Search Space. The cell-based search space \(\mathcal {A}\) has been dominating the early success in the field of NAS. The cell-based search space was first introduced by NASNet [144] and DARTS [138]. As defined in NASNet, the cell-based search space consists of two types of cell structures, which are denoted as the normal cell and the reduction cell. In practice, both types of cells are encoded into directed acyclic graphs (DAGs) and maintain the same cell structure as illustrated in Figure 7, except that the reduction cell starts with one convolutional layer with the stride of 2 to reduce the input spatial dimension. Once the cell structure is determined at the end of search, it is then repeatedly stacked to derive the final architecture candidate. In addition, DARTS introduces another type of cell-based search space, which has motivated a plethora of subsequent NAS methods that are also built on top of the same cell-based search space, such as RobustDARTS [145], EdgeNAS [130], PC-DARTS [146], P-DARTS [147], DARTS+ [148], DARTS–[149], FairDARTS [150], and \(\beta\)-DARTS [151]. Similar to NASNet, the cell-based search space in DARTS consists of two types of cells, the normal cell and the reduction cell. As shown in Figure 7, each cell has an ordered sequence of nodes, where each node is a latent representation (e.g., a feature map in convolutional networks) and each directed edge has a set of possible operators \(\lbrace o^{(i, j)}\rbrace\) that transform the input \(x^{(i)}\). In contrast to NASNet, the cell in DARTS is assumed to have two different input nodes and one single output node. With this in mind, we are able to mathematically calculate the intermediate node as follows, which is based on all of its predecessors:Finally, the search space of DARTS contains \(6.3 \times 10^{29}\) possible architecture candidates [130].

Block-Based Search Space. The block-based search space \(\mathcal {A}\) advocates for simple and diverse network topologies as illustrated in Figure 8, in which each architecture candidate consists of multiple sequential operator candidates. As shown in previous NAS practices, the operator candidates in the block-based search space are usually taken from state-of-the-art manual DNNs, such as MobileNets [32, 85] and ShuffleNets [34, 35]. For example, the block-based search space in ProxylessNAS [40] is built on top of MobileNetV2, whereas the block-based search space in HSCoNAS [152] is built on top of ShuffleNetV2. In parallel, HURRICANE [153] demonstrates that different hardware platforms favor different search spaces, based on which HURRICANE introduces a hybrid block-based search space that combines both MobileNetV2 and ShuffleNetV2 to deliver superior architecture solutions. In contrast to the cell-based search space, the block-based search space is hardware friendly, due to which the block-based search space has been widely adopted in previous hardware-aware NAS methods, such as MnasNet [38], ProxylessNAS [40], OFA [42], HSCoNAS [152], SurgeNAS [154], and LightNAS [131]. The intuition behind this is that the architecture candidate in the cell-based search space consists of multiple parallel branches as shown in Figure 7, which introduce additional overheads in terms of the memory access and, as a result, deteriorate the inference efficiency on target hardware according to the roofline analysis [155]. In addition, in contrast to the cell-based search space that repeatedly stacks the same cell structure across the entire network, the block-based search space allows operator diversity within different blocks, encouraging the search for architecture candidates with better accuracy–efficiency trade-offs [156].

In this section, we discuss recent state-of-the-art NAS algorithms and divide them into three main categories: reinforcement learning-based search [137], evolutionary algorithm-based search [157], and gradient-based search (also known as differentiable search) [138].

Reinforcement Learning-Based Search. In the field of NAS, to the best of our knowledge, [137] is the first NAS work³ that opens up the possibility to automate the design of top-performing DNNs, which features reinforcement learning (RL) [159] as the search engine. Specifically, [137] leverages a simple yet effective recurrent neural network (RNN) as the RL controller to generate possible architecture candidates from the search space as shown in Figure 9. The generated architecture candidate is then trained from scratch on the target task to evaluate the accuracy. Next, the accuracy of the generated architecture candidate is fed back into the aforementioned RNN controller, which optimizes the RNN controller to generate better architecture candidates in the next iteration. Once the search process terminates, the well-optimized RNN controller is able to provide DNNs with superior accuracy on the target task. For example, the network generated by the RNN controller achieves 96.35% top-1 accuracy on CIFAR-10, which is comparable to or even better than the family of manually designed DNNs, such as ResNet [2]. The promising performance of [137] marks an important milestone in the field of NAS, pioneering an effective alternative to automate the design of competitive DNNs.

Subsequently, based on [137], NASNet [144] introduces the flexible cell-based search space as shown in Figure 7, which further boosts the attainable accuracy on the target task. For example, NASNet achieves 97.6% top-1 accuracy on CIFAR-10, which is \(+\)1.25% higher than [137] while involving fewer parameters (i.e., 37.4 M in [137] vs. 27.6 M in NASNet). Despite the promising performance, [137] and NASNet have to train a large number of possible architecture candidates from scratch, thus inevitably necessitating prohibitive computational resources. For example, to optimize the RNN controller, [137] needs to train 12,800 stand-alone architecture candidates. To overcome such limitations, ENAS [160] proposes an efficient NAS paradigm dubbed parameter sharing, which forces all the architecture candidates to share network weights to eschew training each architecture candidate from scratch. In practice, this leads to significant reduction in terms of search cost, while at the same time still maintaining strong accuracy on the target task. For example, in [137], one single search experiment takes 3\(\sim\)4 days on 450 NVIDIA GTX 1080 Ti GPUs [144]. In contrast, benefiting from the paradigm of parameter sharing, ENAS is able to find one decent network solution with 97.11% top-1 accuracy on CIFAR-10 and, more importantly, in less than 16 hours on one single NVIDIA GTX 1080 Ti GPU. Thanks to the significant search efficiency, the paradigm of parameter sharing has been dominating subsequent breakthroughs in the NAS community [42, 138, 161].

Although early RL-based NAS methods [137, 144, 160] have had tremendous success in automatic network design, they focus on accuracy-only optimization, ignoring other important performance metrics, such as latency and energy. To search for hardware-efficient network solutions, MnasNet [38] formulates the search process as a multi-objective optimization problem that optimizes both accuracy and latency as shown in Figure 10. To achieve this, MnasNet introduces a flexible block-based search space (see Figure 8) and designs an effective multi-objective RL reward function to optimize the RNN controller. Specifically, the goal of MnasNet is to find Pareto-optimal architecture candidates arch in the search space \(\mathcal {A}\) that maximize the predefined multi-objective RL reward, which can be formulated as follows:where \(Accuracy(\cdot)\) and \(Latency(\cdot)\) denote the accuracy on the target task and the latency on target hardware, respectively. In addition, T is the specified latency constraint. It is worth noting that the latency \(Latency(\cdot)\) in MnasNet is directly measured on target hardware, which suffers from non-trivial engineering efforts due to the prohibitive search space (e.g., \(|\mathcal {A}| = \sim 10^{39}\) in MnasNet) [40, 42]. To avoid the tedious on-device latency measurements, we discuss several efficient latency predictors later in this section. Apart from these, w is the trade-off coefficient to control the trade-off magnitude between accuracy and latency, which is defined as follows:where \(\alpha\) and \(\beta\) are application-specific hyperparameters to control the trade-off magnitude between accuracy and efficiency. According to the empirical observation that doubling the latency usually brings \(\sim\)5% relative accuracy improvement, MnasNet assigns \(\alpha = \beta = -0.07\). In practice, \(\alpha\) and \(\beta\) are both sensitive and difficult to tune. Even worse, given new hardware devices or new search spaces, \(\alpha\) and \(\beta\) involve additional engineering efforts for hyperparameter tuning. For example, as observed in MobileNetV3 [162], the accuracy changes much more dramatically with latency for small networks. Therefore, to obtain the required architecture candidate that satisfies the specified latency constraint T, we typically need to repeat 7 search experiments to tune \(\alpha\) and \(\beta\) through trial and error [163]. This significantly increases the total search cost by ×7. To eliminate such additional hyperparameter tuning, TuNAS [163] investigates the multi-objective RL reward in Equation (2) and further introduces a similar RL reward function, which can be formulated as follows:where \(|\cdot |\) is the absolute function and \(\gamma \lt 0\) is a finite negative value, which controls how strongly we enforce the architecture candidate to maintain the latency close to T.

MONAS [164] also introduces a simple yet effective RL reward function that considers optimizing both accuracy and energy, which can be formulated as follows:where \(\eta \in [0, 1]\) is the coefficient to control the trade-off between accuracy and energy. We note that the RL reward function in Equation (5) aims to find the architecture candidate with high accuracy and low energy, which can be generalized to other performance constraints, such as latency.

Evolutionary Algorithm-Based Search. In addition to reinforcement learning–based search, evolutionary algorithm–based search is another popular branch in the NAS literature thanks to its flexibility, conceptual simplicity, and competitive performance [157]. As seen in the very early evolutionary practices [165, 166, 167, 168], the evolutionary algorithm–based search typically consists of four key steps: (1) sampling a set of possible architecture candidates from the search space as the child population; (2) evaluating the architecture candidates in the child population to interpret the performance, such as accuracy and efficiency; (3) reserving the top-k architecture candidates in the latest child population to form the parent population and discarding the architecture candidates with poor performance; and (4) manipulating the architecture candidates in the latest parent population to generate new architecture candidates to form the next-generation child population. These four steps are repeated until the evolutionary process converges.

There are many other aspects in which the evolutionary algorithm may differ, including (1) how to sample the initial population, (2) how to select the parent population, and (3) how to generate the child population from the parent population. Generating the child population from the parent population is of utmost importance in order to produce superior architecture candidates [157]. In practice, to allow efficient exploration and exploitation [170], crossover and mutation are two of the most popular strategies to generate the child population [171, 172]. Specifically, for crossover, two random architecture candidates from the parent population are crossed to produce one new child architecture candidate. For mutation, one randomly selected architecture candidate mutates its operators with a fixed probability. However, the early evolutionary NAS works have to train a large number of stand-alone architecture candidates from scratch to evaluate the accuracy [157] and, as a result, suffer from non-trivial computational resources [169].

To reduce the required computational resources for neural architecture search, [169] introduces the paradigm of one-shot NAS, which has been widely applied in subsequent NAS methods [40, 42, 138] thanks to its significant search efficiency. In parallel to [169], SMASH [173] also proposes a similar one-shot NAS paradigm, but [169] is much more popular in the NAS community. Specifically, [169] designs an effective one-shot supernet as visualized in Figure 11, which consists of all possible architecture candidates in the search space. Therefore, we only need to train the one-shot supernet, after which we can evaluate different architecture candidates in the search space with inherited network weights from the pretrained one-shot supernet as shown in Figure 12. This effectively avoids needing to train a large number of stand-alone architecture candidates from scratch. In practice, the one-shot supernet is simply trained using the standard SGD optimizer with momentum. Once the one-shot supernet is well trained, it is able to quickly and reliably approximate the performance of different architecture candidates using the paradigm of weight sharing [160]. With the well-trained one-shot supernet, it is straightforward and technically easy to leverage the standard evolutionary algorithm to search for top-performing architecture candidates with superior accuracy on the target task [169]. We note that the searched architecture candidates still need to be retrained or fine-tuned on the target task in order to recover the accuracy for further deployment on target hardware.

SPOS [161] investigates the one-shot NAS [169] and identifies two critical issues. On the one hand, the network weights in the one-shot supernet are deeply coupled during the training process. On the other hand, joint optimization introduces further coupling between architecture candidates and supernet weights. To address these, SPOS proposes the paradigm of single-path one-shot NAS, which uniformly samples one single-path subnetwork from the supernet and trains the sample single-path subnetwork instead. This brings two main benefits: (1) reducing memory consumption to the single-path level and (2) improving the performance of the final searched architecture candidate. The success of SPOS has motivated a series of follow-up works [42, 152, 153, 174, 175, 176, 177, 178, 179]. Note that all of these follow-up works [42, 152, 153, 174, 175, 176, 177, 178, 179] focus on training an effective and reliable supernet, which then serves as the evaluator to quickly query the performance of different architecture candidates. For example, FairNAS [174] demonstrates that the uniform sampling strategy only implies the soft fairness, and to imply the strict fairness, FairNAS samples multiple single-path subnetworks to enforce that all the operator candidates in the supernet are equally optimized during each training iteration. In parallel, OFA [42] is another representative evolutionary NAS method that aims to train the supernet, after which we are allowed to detach single-path subnetworks from the supernet with inherited network weights for further deployment on target hardware. Note that the detached subnetwork in OFA still requires to be fine-tuned on the target task for several epochs (e.g., 25 epochs) in order to obtain competitive accuracy. To eliminate the fine-tuning process, BigNAS [175] proposes several enhancements to train one single-stage supernet, where the single-path subnetwork detached from the supernet with inherited network weights can achieve superior accuracy without being retrained or fine-tuned on the target task and can be directly deployed on target hardware. This significantly saves the computational resources required for training stand-alone architecture candidates, especially when targeting multiple different deployment scenarios such as multiple different hardware platforms.

Thanks to its search flexibility, evolutionary algorithm-based NAS can be easily extended to search for hardware-efficient architecture candidates, which maximize the accuracy on the target task while satisfying various real-world performance constraints [153], such as latency, energy, memory, and more. Without loss of generality, we consider the following multi-objective optimization:where \(\lbrace Constraint_i(\cdot)\rbrace _{i=1}^n\) and \(\lbrace C_i\rbrace _{i=1}^n\) are a set of real-world performance constraints.

Gradient-Based Search. In addition to reinforcement learning–based search and evolutionary algorithm–based search, gradient-based search [138], also known as differentiable search, is another representative branch of NAS, which has since gained increasing popularity in the NAS community and motivated a plethora of subsequent differentiable NAS works [145, 146, 147, 148, 149, 150, 151, 180, 181, 182, 183, 184, 185, 186, 187, 188], thanks to its significant search efficiency [189]. For example, DARTS [138], as the seminal differentiable NAS work, is able to deliver one superior architecture candidate in \(\sim\)1 day on one single NVIDIA GTX 1080 Ti GPU. In contrast to previous non-differentiable NAS practices [137, 144, 160, 169] that highly rely on discrete search spaces, DARTS leverages a list of architecture parameters \(\alpha\) to relax the discrete search space to become continuous. Benefiting from the continuous search space, both the network weights w and the architecture parameters \(\alpha\) can be optimized via alternating gradient descent. Once the differentiable search process terminates, we can interpret the optimal architecture candidate from the architecture parameters \(\alpha\). Specifically, the supernet in DARTS is initialized by stacking multiple over-parameterized cells (see Figure 13 (1)), in which each cell consists of all possible cell structures in the cell-based search space \(\mathcal {A}\). As shown in Figure 13, each cell is represented using the DAG that consists of N nodes \(\lbrace x_i\rbrace _{i=1}^N\). Note that the nodes here correspond to the intermediate feature maps. In addition, the directed edges between \(x_i\) and \(x_j\) correspond to a list of operator candidates \(\lbrace o | o \in \mathcal {O}\rbrace\) in the operator space \(\mathcal {O}\). The directed edges between \(x_i\) and \(x_j\) are also assigned with a list of architecture parameters \(\lbrace \alpha _o^{(i, j)} | o \in \mathcal {O}\rbrace\). Finally, following DARTS, we formulate \(x_j\) as follows:Note that the output \(x_j\) is continuous with respect to \(x_i\), \(\alpha\), and w. In Light of this, DARTS proposes to optimize \(\alpha\) and w using the following bilevel optimization scheme:where \(\mathcal {L}_{train}(\cdot)\) and \(\mathcal {L}_{val}(\cdot)\) are the loss functions on the training and validation datasets, respectively. Once the differentiable search process terminates, DARTS determines the optimal architecture candidate by reserving the strongest operator \(\alpha _o^{(i, j)}\) and removing other operators between \(x_i\) and \(x_j\), in which the operator strength is defined as \(\exp \alpha _o^{(i, j)} / \sum _{o^{\prime } \in \mathcal {O}} \exp \alpha _{o^{\prime }}^{(i, j)}\). It is worth noting that the searched optimal architecture candidate still needs to be retrained on the target task in order to recover its accuracy for further deployment on target hardware.

Inspired by the promising performance of DARTS, a plethora of follow-up works [145, 146, 147, 148, 149, 150, 151, 180, 181, 182, 183, 184, 185, 186, 187, 188] have recently emerged that strive to unleash the power of differentiable NAS to deliver superior architecture candidates. For example, in contrast to DARTS, which simultaneously optimizes all operator candidates in the supernet, PC-DARTS [146] introduces partial channel connections to alleviate the excessive memory consumption of DARTS. In addition, DARTS+ [148] investigates the performance collapse issue of DARTS and finds that the performance collapse issue is caused by the over-selection of skip-connect. To tackle this, DARTS+ proposes a simple yet effective early-stopping strategy to terminate the search process upon fulfilling a set of predefined criteria. In parallel, DARTS– [149] also observes that the performance collapse issue of DARTS comes from the over-selection of skip-connect and further leverages an auxiliary skip connection to mitigate the performance collapse issue and stabilize the search process. Apart from these, Single-DARTS [185] and Gold-NAS [184] investigate the bilevel optimization in Equation (8) and point out that the bi-level optimization may end up with suboptimal architecture candidates, based on which Single-DARTS and Gold-NAS turn back to the one-level optimization. To accelerate the search process, GDAS [186] introduces an efficient Gumbel-Softmax [190]–based differentiable sampling approach to reduce the optimization complexity to the single-path level. Similar to GDAS, SNAS [187] also leverages Gumbel-Softmax reparameterization to improve the search process, which can make use of gradient information from generic differentiable loss without sacrificing the completeness of NAS pipelines. PT-DARTS [182] revisits the architecture selection in differentiable NAS and demonstrates that the architecture parameters \(\alpha\) cannot always imply the optimal architecture candidate, based on which PT-DARTS introduces the perturbation-based architecture selection to determine the optimal architecture candidate at the end of search.

The aforementioned differentiable NAS works [145, 146, 147, 148, 149, 150, 151, 180, 181, 182, 183, 184, 185, 186, 187, 188], however, focus on accuracy-only neural architecture search, which indeed demonstrates promising performance in terms of finding the architecture candidate with competitive accuracy but fails to accommodate the limited available computational resources in real-world embedded scenarios. To overcome such limitations, the paradigm of hardware-aware differentiable NAS [130, 191, 192, 193, 194] has recently emerged, which is based on DARTS and focuses on finding top-performing architecture candidates within the cell-based search space that can achieve both high accuracy on target task and high inference efficiency on target hardware. To achieve this goal, one widely adopted approach is to integrate the latency-constrained loss term into the overall loss function to penalize the architecture candidate with high latency, which can be mathematically formulated as follows:where \(\lambda\) is the trade-off coefficient to control the trade-off magnitude between accuracy and latency. As demonstrated in [131, 156], a larger \(\lambda\) ends up with the architecture candidate that maintains low accuracy and low latency, whereas a smaller \(\lambda\) leads to the architecture candidate with high accuracy and high latency. \(Latency(\alpha)\) corresponds to the latency of the architecture candidate encoded by \(\alpha\). We note that the optimization objective in Equation (9) can be easily generalized to jointly optimize other types of hardware performance constraints, such as energy and memory consumption, in which we only need to incorporate \(Energy(\alpha)\) and \(Memory(\alpha)\) into the optimization objective in Equation (9). For example, we can reformulate the optimization objective in Equation (9) as follows to jointly optimize the on-device latency, energy, and memory consumption:where \(\lambda _1\), \(\lambda _2\), and \(\lambda _3\) are trade-off coefficients to determine the trade-off magnitudes between accuracy and latency, energy, and memory, respectively.

Despite the significant progress to date, the aforementioned hardware-aware differentiable NAS works [130, 191, 192, 193, 194] highly rely on the cell-based search space; these works first determine the optimal cell structure and then repeatedly stack the same cell structure across the entire network [138]. However, as demonstrated in MnasNet [38], such NAS practices suffer from inferior accuracy and efficiency due to the lack of operator diversity. Even worse, the architecture candidates in the cell-based search space have multiple parallel branches as shown in Figure 7, which introduce considerable memory access overheads and, as a result, have difficulty benefitting from the high computational parallelism on mainstream hardware platforms [34, 35]. To overcome such limitations, recent hardware-aware differentiable NAS works [39, 40, 154, 188, 195, 196, 197, 198, 199] have shifted their attention from the cell-based search space (see Figure 7) to the block-based search space (see Figure 8). The most representative include FBNet [39], ProxylessNAS [40], SP-NAS [197], and TF-NAS [195]. Similar to GDAS [186] and SNAS [187], FBNet leverages Gumbel-Softmax reparameterization [190] to relax the discrete search space to be continuous. FBNet collects a simple yet effective latency lookup table to quickly approximate the latency of different architecture candidates. The pre-collected latency lookup table is then integrated into the search process to derive hardware-efficient architecture candidates. However, similar to DARTS [138], FBNet needs to simultaneously optimize all the operator candidates in the supernet during the search process, which is not scalable to large search spaces and suffers from tmemory bottleneck [40, 131]. In light of this, ProxylessNAS introduces an effective path-level binarization approach to reduce the memory consumption to the single-path level, which significantly improves search efficiency without compromising search accuracy. In parallel, SP-NAS demonstrates that different operator candidates in the supernet can be viewed as subsets of an over-parameterized superkernel, based on which SP-NAS proposes to encode all the operator candidates into the superkernel. In practice, this explicitly reduces memory consumption to the single-path level, which alleviates the memory bottleneck during the search process. TF-NAS thoroughly investigates the three search freedoms in hardware-aware differentiable NAS: (1) operator-level search, (2) depth-level search, and (3) width-level search as shown in Figure 14, which is able to perform fine-grained architecture search. To obtain hardware-efficient architecture candidates, TF-NAS integrates the pre-collected latency lookup table into the search process. TF-NAS also introduces a simple yet effective bi-sampling search algorithm to accelerate the search process towards enhanced search efficiency.

Even so, we should consider not only the explicit search cost, the time required for one single search experiment, but also the implicit search cost, the time required for manual hyperparameter tuning in order to find the desired architecture candidate. This is because, in real-world embedded scenarios such as autonomous vehicles, DNNs must be executed under strict latency constraints (e.g., 24 ms), in which any violation may lead to catastrophic consequences [20, 163]. However, to find the architecture candidate with the latency of 24 ms, the aforementioned hardware-aware differentiable NAS works [39, 40, 130, 154, 188, 191, 192, 193, 194, 195, 196, 197, 198, 199] have to repeat a plethora of search experiments to tune the trade-off coefficient \(\lambda\) (see Equation (9)) through trial and error [131, 156], which significantly increases the total search cost. The intuition behind this is that \(\lambda\), despite being able to trade off between accuracy and latency, is quite sensitive and difficult to control [131, 156]. To overcome such limitations, HardCoRe-NAS [200] leverages an elegant Block Coordinate Stochastic Frank-Wolfe (BCSFW) algorithm [201] to restrict the search direction around the specified latency requirement. In addition, LightNAS [131, 156] introduces a simple yet effective hardware-aware differentiable NAS approach, which investigates the optimization objective in Equation (9) and proposes to optimize the trade-off coefficient \(\lambda\) during the search process in order to satisfy the specified latency requirement. In other words, LightNAS focuses on automatically learning \(\lambda\) that strictly complies with the specified latency requirement, which is able to find the required architecture candidate in one single search (i.e., you only search once) and avoids performing manual hyperparameter tuning over \(\lambda\). The optimization objective of LightNAS is formulated as follows:where T is the specified latency requirement. In contrast to previous hardware-aware differentiable NAS works [39, 40, 130, 154, 188, 191, 192, 193, 194, 195, 196, 197, 198, 199], \(\lambda\) in Equation (11) is not a constant but rather a learnable hyperparameter that can be automatically optimized during the search process. For the sake of simplicity, below we use \(\mathcal {L}(w, \alpha , \lambda)\) to denote the optimization objective in Equation (11). Finally, to satisfy the specified latency requirement (i.e., \(Latency(\alpha) = T\)), w and \(\alpha\) are updated using gradient descent [138], whereas \(\lambda\) is updated using gradient ascent as follows:where \(lr_{w}\), \(lr_{\alpha }\), and \(lr_{\lambda }\) are the learning rates of w, \(\alpha\), and \(\lambda\), respectively. Below we further demonstrate why LightNAS guarantees \(Latency(\alpha) = T\). As shown in LightNAS, a larger \(\lambda\) leads to the architecture candidate with low latency, whereas a smaller \(\lambda\) results in the architecture candidate with high latency. Therefore, if \(Latency(\alpha) \gt T\), the gradient ascent scheme increases \(\lambda\) to reinforce the latency regularization magnitude. As a result, \(Latency(\alpha)\) decreases towards T in the next search iteration. Likewise, if \(Latency(\alpha) \lt T\), the gradient ascent scheme decreases \(\lambda\) to diminish the latency regularization magnitude, after which \(Latency(\alpha)\) increases towards T in the next search iteration. Finally, the search engine ends up with the architecture candidate that strictly satisfies the specified latency requirement (i.e., \(Latency(\alpha) = T\)). More recently, Double-Win NAS [202] proposed deep-to-shallow transformable search to further marry the best of both deep and shallow networks towards an aggressive accuracy–efficiency win–win. Similar to LightNAS [131, 156], the resulting shallow network can also satisfy the specified latency constraint. Finally, we compare previous representative hardware-aware NAS works, which are summarized in Table 1.

In this section, we discuss recent state-of-the-art advances in general speedup techniques and extensions for NAS algorithms, including one-shot NAS enhancements, efficient latency prediction, efficient accuracy prediction, low-cost proxies, zero-cost proxies, efficient transformer search, efficient domain-specific search, and mainstream NAS benchmarks, which have the potential to significantly benefit NAS algorithms and largely facilitate the search process.

Beyond One-Shot NAS. Despite the high search efficiency, one-shot NAS often suffers from poor ranking correlation between one-shot search and stand-alone training. As pointed out in [205], one-shot search results do not necessarily correlate with stand-alone training results across various search experiments. To overcome such limitations, a plethora of one-shot NAS enhancements have been proposed recently[206, 207, 208, 209, 210, 211]. Specifically, [206, 207, 208, 209, 210] turn back to few-shot NAS. In contrast to one-shot NAS [160], which only features one supernet, few-shot NAS introduces multiple supernets to explore different regions of the predefined search space, which slightly increases the search cost over one-shot NAS but can deliver much more reliable search results. For example, as shown in [206], with only up to 7 supernets, few-shot NAS can establish new state-of-the-art search results on ImageNet. Among them, [209] demonstrates that zero-cost proxies can be integrated into few-shot NAS, which can further enhance the search process of one-shot NAS and thus produce better search results. More recently, [208] generalizes few-shot NAS to distill LLMs, which focuses on automatically distilling multiple compressed student models under various computational budgets from a large teacher model. In contrast to few-shot NAS, which leverages multiple supernets to improve the ranking correlation performance of one-shot NAS, CLOSE [211] instead features an effective curriculum learning-like schedule to control the parameter-sharing extent within the proposed supernet dubbed CLOSENet, in which the parameter-sharing extent can be flexibly adjusted during the search process and the parameter-sharing scheme is built upon an efficient graph-based encoding scheme.

Efficient Latency Prediction.⁴ As seen in MnasNet [38], latency is directly measured on target hardware, which is then integrated into the RL reward (see Equation (2)) to penalize the architecture candidate with high latency. The direct on-device latency measurement is indeed accurate. However, it is time-consuming and unscalable to large search spaces [40]. To overcome such limitations, several latency prediction strategies have been proposed recently. For example, ProxylessNAS [40], FBNet [39], and OFA [42] leverage the latency lookup table to approximate the on-device latency, which sums up the latency of all the operator candidates. In addition, HSCoNAS [152, 178] demonstrates that the data movements and communications among different operator candidates introduce additional latency overheads, making the pre-collected latency lookup table inaccurate. To mitigate this issue, HSCoNAS quantifies the latency that corresponds to the intermediate data movements and communications, which is then fed into the pre-collected latency lookup table to achieve more accurate latency prediction performance. However, the latency lookup table is only applicable to the block-based search space, which leads to unreliable latency prediction performance in terms of the cell-based search space [213]. To this end, EdgeNAS [130], LA-DARTS [191], and LC-NAS [192] propose to use learning-based approaches for the purpose of latency prediction. For example, EdgeNAS trains an efficient multi-layer perceptron (MLP) to predict the latency of different architecture candidates in the cell-based search space, which can also be generalized to predict the latency of different architecture candidates in the block-based search space as shown in [131, 156, 176, 212, 214]. BRP-NAS [213] and SurgeNAS [154] introduce graph neural network (GNN)–based latency predictors to achieve more reliable latency prediction performance. The above latency predictors (1) rely on a large number of training samples to achieve decent latency prediction performance (e.g., 100,000 training samples in EdgeNAS) and (2) need to be reconstructed for either new hardware or new search spaces. To avoid these, HELP [215] and MAPLE-Edge [216] focus on building an efficient latency predictor using only a few training samples (e.g., as few as 10 training samples in HELP), which can be generalized to new hardware or new search spaces with only minimal re-engineering efforts. More recently, EvoLP [217] considered an effective self-evolving scheme to construct efficient yet accurate latency predictors, which can adapt to unseen hardware with only minimal re-engineering efforts.

Efficient Accuracy Prediction. In parallel to latency prediction, accuracy prediction has also received increasing attention from the NAS community [213, 218, 219, 220, 221, 222], which strives to directly predict the accuracy of different architecture candidates in the search space. Specifically, [218] introduces a simple yet effective graph convolutional network (GCN)–based accuracy predictor, which can achieve reliable accuracy prediction performance thanks to GCNs’ strong capability to learn graph-structured data. Similar to [218], BRP-NAS [213] also considers GCNs for reliable accuracy prediction, which introduces transfer learning to further improve the accuracy prediction performance from the pretrained latency predictor. In parallel, [219] leverages the non-neural network (i.e., gradient-boosted decision tree (GBDT)) as the accuracy predictor, which has a stronger capability to learn representations than neural network–based accuracy predictors. In addition, NASLib [220] investigates a wide range of accuracy predictors from learning curve extrapolation, weight-sharing, supervised learning, and zero-cost proxies on three popular NAS benchmarks (i.e., NAS-Bench-101 [223], NAS-Bench-201 [224], and NAS-Bench-NLP [225]). NASLib reveals that different accuracy predictors can be combined to achieve substantially better accuracy prediction performance than any single accuracy predictor. DONNA [221] proposes to build an efficient accuracy predictor, which involves only minimal computational resources and, more importantly, can scale to diverse search spaces. To achieve this, DONNA uses blockwise knowledge distillation to construct an architecture candidate pool in which each architecture candidate only needs to be fine-tuned for several epochs to derive the accuracy rather than being trained from scratch. In contrast to the aforementioned accuracy predictors that feature graph-based encoding schemes, GATES [222, 226] instead models the operations as the transformation of the propagating information, which can effectively mimic the actual data processing of different neural architecture candidates. More importantly, the encoding scheme of GATES can be integrated into the above accuracy predictors to further boost their accuracy prediction performance. Similar to GATES, TA-GATES [227] introduces an effective encoding scheme with analogous modeling of the training process of different neural architecture candidates, which can achieve better accuracy prediction performance than GATES on various representative NAS benchmarks.

Low-Cost Proxies (Learning Curve Extrapolations). Low-cost proxies, also referred to as learning curve extrapolations [231], aim to interpret the accuracy of the given architecture candidate only using its early training statistics, such as the training loss in the first few training epochs, which has motivated a plethora of subsequent works to continue exploring learning curve extrapolation [156, 232, 233, 234, 235, 236, 237]. For example, in contrast to the conventional accuracy predictor that only uses the network configuration as input features, [236] proposes to combine the network configuration and a series of validations of accuracy in the first few training epochs as input features to train a simple regression model, which can be generalized to predict the accuracy of unseen architecture candidates. In addition, [232] introduces Training Speed Estimation (TSE), which simply accumulates the early training statistics to achieve reliable yet computationally inexpensive ranking among different architecture candidates. The work of [156, 237] introduces Batchwise Training Estimation (BTE) and Trained Batchwise Estimation (TBE), both of which consider the fine-grained batchwise training statistics to provide more reliable prediction performance using minimal computational resources. In parallel, [234] introduces Loss Curve Gradient Approximation (LCGA) to rank the accuracy of different architecture candidates with minimal training. The work of [233] introduces NAS-Bench-x11 to unleash the power of learning curve extrapolation by predicting the training trajectories, which can be easily integrated into the aforementioned learning curve extrapolation works to quickly estimate the performance of the given architecture candidate.

Zero-Cost Proxies.⁵ In addition to the above low-cost proxies (i.e., learning curve extrapolation), zero-cost proxies have recently flourished [228, 229, 230, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247], which focus on interpreting the performance of the given architecture candidate in training-free manners. Zero-cost proxies, such as EPE [240], Fisher [241], GradNorm [238], Grasp [242], Jacov [243], Snip [244], Synflow [245], ZenScore [246], LRC [228], and NTK [229], can provide reliable performance estimation using only one single mini-batch of data and one single forward/backward propagation pass, which necessitate near-zero computational cost [230, 238, 239]. Thanks to their reliable performance estimation and low cost, these zero-cost proxies have been widely adopted in recent NAS works to accelerate the search process [230, 243, 247]. As demonstrated in [230, 238], combining different zero-cost proxies may lead to more reliable ranking performance estimation than any single zero-cost proxy. For example, as shown in Figure 15, combining LRC and NTK provides more reliable ranking performance estimation than LRC or NTK separately. In light of this, TE-NAS [230] further leverages LRC and NTK to jointly estimate the ranking performance among different architecture candidates in the search space, which quickly ends up with the optimal architecture candidate on ImageNet in less than 4 hours on one single NVIDIA GTX 1080 Ti GPU.

Efficient Transformer Search. In addition to CNNs, transformers are another important branch of DNNs. Inspired by the tremendous success of NAS in searching for superior CNNs, automated transformer search has gained increasing popularity, which applies NAS techniques to automatically search for superior transformers, including transformers for NLP tasks [93, 249, 250, 251, 252, 253, 254] and vision transformers for vision tasks [248, 255, 256, 257, 258, 259, 260]. Automated transformer search is technically the same as automated convolutional network search, in which both feature the same search pipeline. For example, HAT [93], as one of the state-of-the-art NAS works in the field of NLP, focuses on searching for hardware-efficient transformers for NLP tasks. To achieve this, HAT first initializes an over-parameterized superformer that consists of all possible transformer candidates in the search space, which is technically the same as the supernet in automated convolutional network search. After that, HAT trains the superformer using the standard weight-sharing technique [160], which then serves as the accuracy predictor to quickly interpret the accuracy of different transformer candidates. Next, HAT builds an efficient latency predictor to avoid the tedious on-device latency measurement. Finally, HAT applies the standard evolutionary algorithm to find hardware-efficient transformer candidates with both high accuracy and high efficiency, which is technically the same as OFA [42], which searches for hardware-efficient convolutional networks. Furthermore, due to the tremendous success of vision transformers in vision tasks as discussed in Section 2.2, a plethora of NAS works [248, 255, 256, 257, 258, 259, 260] have been subsequently proposed to automate the design of superior vision transformers. The work of [248], being the first, introduces an evolutionary algorithm-based NAS framework dubbed AutoFormer. Similar to HAT, AutoFormer first constructs an over-parameterized superformer that consists of all possible vision transformer candidates in the search space, which is then trained using the weight entanglement scheme. The difference between weight sharing and weight entanglement is visualized in Figure 16, in which weight entanglement is technically similar to the superkernel in SP-NAS [197]. Finally, AutoFormer applies the standard evolutionary algorithm to explore the optimal vision transformer candidate. These clearly demonstrate that we can easily leverage recent state-of-the-art NAS techniques that focus on searching for competitive CNNs to automate the design of top-performing transformers for both NLP and vision tasks.

Efficient Domain-Specific Search. In addition to image classification, NAS can also be applied to a wide range of real-world scenarios, such as object detection [268, 269, 270], semantic segmentation [271, 272, 273], point cloud processing [192, 274, 275, 276], image super-resolution [277, 278], and more. For example, MobileDets [268] are a family of hardware-efficient object detection networks that can deliver promising detection accuracy while maintaining superior detection efficiency on multiple embedded computing systems, including mobile central processing units (CPUs), edge tensor processing units (TPUs), and edge GPUs. MobileDets first construct an enlarged search space that contains a large number of possible object detection networks and then leverage an MnasNet-like reinforcement learning–based search algorithm [38] to find top-performing object detection networks, which also feature the same reward function as TuNAS [163] to trade off between detection accuracy and efficiency. The work of [192] introduces an efficient hardware-aware differentiable NAS framework dubbed LC-NAS, aiming to automate the design of competitive network solutions for point cloud processing. Here, similar to EdgeNAS [130] and LA-DARTS [191], which focus on finding top-performing architecture candidates for image classification, LC-NAS exploits the same cell-based search space and integrates the latency constraint into the optimization objective to penalize the architecture candidate with high latency. These demonstrate that we can easily include domain-specific knowledge (e.g., domain-specific search spaces) into mainstream NAS techniques (e.g., differentiable, evolutionary algorithm–based, and reinforcement learning–based NAS) to search for domain-specific network solutions.

Mainstream NAS Benchmarks. Although NAS has achieved substantial performance improvement across various NLP and vision tasks, fair comparisons between different NAS works are frustratingly hard and still an open issue, as demonstrated in [279]. This is because different NAS works may feature quite different training recipes, such as different training epochs and training enhancements. For example, DARTS+ [148] trains the searched architecture candidate on CIFAR-10 for 2,000 epochs, whereas DARTS [138] only applies 600 training epochs. DARTS+ trains the searched architecture candidate on ImageNet for 800 epochs, with a batch size of 2,048, where AutoAugment [280] is also integrated in order to achieve stronger data augmentations. In contrast, DARTS only applies 250 training epochs with a batch size of 128 by default. We note that, for the same architecture candidate, longer training epochs and stronger data augmentations typically achieve better training accuracy on the target task, as shown in [279]. RandomNAS [281] challenges the effectiveness of early state-of-the-art NAS works and demonstrates that random search, as one strong search baseline to explore random networks, can achieve even better performance on the target task than early state-of-the-art NAS works. In parallel, RandWire [282] shows that randomly wired networks can also exhibit strong accuracy on ImageNet. Therefore, it remains unknown whether the performance improvement of NAS is due to the more advanced training recipe or the search algorithm itself, making it difficult to evaluate and compare the technical contributions of different NAS works [223, 224, 279].

To overcome such limitations, a plethora of tabular and surrogate NAS benchmarks have been proposed: NAS-Bench-101 [223], NAS-Bench-201 [224], NATS-Bench [261], NAS-Bench-301 [262], NAS-Bench-360 [263], NAS-Bench-1Shot1 [264], NAS-Bench-ASR [265], NAS-Bench-Graph [266], NAS-Bench-NLP [225], HW-NAS-Bench [214], NAS-Bench-x11 [233], and NAS-Bench-Suite [267]. We note that NAS benchmarks typically have two important parts, the predefined search space and the related performance metrics for all possible architecture candidates that can be easily queried. In tabular NAS benchmarks [214, 223, 224, 225, 261, 263, 264, 265, 266], all possible architecture candidates are enumerated and trained from scratch on the target task to obtain the performance metrics, such as the training and validation accuracy. In contrast, surrogate NAS benchmarks [214, 233, 262, 267] leverage learning-based methods to predict the performance metrics of different architecture candidates rather than directly enumerating and training all possible architecture candidates on the target task, thus leading to significantly reduced computational resources. In light of this, surrogate NAS benchmarks can be easily extended to deal with larger search spaces than tabular NAS benchmarks (\(10^{18}\) in NAS-Bench-301 [262] vs. 15,625 in NAS-Bench-201 [224]). Finally, we compare and summarize the aforementioned state-of-the-art NAS benchmarks in Table 2.

In this section, we envision several promising future trends and possible directions in the field of automated network design, summarized as follows:

The rationale behind network pruning is that DNNs are usually over-parameterized and redundant in terms of network weights and channels [304, 305]. As such, eliminating redundant network weights and channels can largely benefit network efficiency at the cost of minimal accuracy loss, thus enabling accommodation of limited available computational resources and rigorous storage requirements in real-world embedded scenarios. Following previous well-established pruning conventions, we divide recent state-of-the-art pruning methods into two main categories according to their pruning granularity, in which non-structured pruning (i.e., weight pruning) is fine-grained whereas structured pruning (i.e., channel pruning and layer pruning) is coarse-grained. As illustrated in Figure 17, weight pruning focuses on removing the redundant weight connections, whereas channel pruning and layer pruning focus on removing the redundant channels and layers. In practice, both non-structured pruning and structured pruning can explore simplified network structures with optimized computational efficiency. Nonetheless, non-structured weight pruning highly relies on specialized hardware accelerators [29] and cannot provide realistic runtime speedups on modern embedded computing systems owing to irregular network sparsity [305, 306]. In contrast, structured channel pruning and layer pruning are coarse-grained and do not introduce irregular network sparsity, which can deliver realistic runtime speedups on modern embedded computing systems. For better coverage, below we further discuss recent representative works in the field of both non-structured pruning and structured pruning, which are also summarized in Figure 19.

In contrast to network pruning, which aims to reduce the network complexity at the structure level, network quantization focuses on representing the network weights and activations with lower bits, which significantly reduces the network complexity at the precision level. Therefore, the resulting quantized network maintains the same network structure (i.e., the same number of layers and channels), but with lower-bit network weights and activations. In practice, network quantization can be traced back to the 1990s, when early quantization works pioneered quantizing the network weights for Boltzmann machines [447], optical networks [448], and multi-layer perceptrons (MLPs) [449]. Note that quantization has the potential to significantly trim down the network size to accommodate the limited storage in real-world embedded scenarios. For example, Deep Compression [89] is able to reduce the network size of VGGNet by ×49, from 552 MB to 11.3 MB, while delivering comparable accuracy on ImageNet [80]. Thanks to its surprisingly strong performance in reducing the computational complexity and alleviating the storage requirements, renewed research interest in network quantization has emerged since the 2010s [409], which demonstrates that, compared with full-precision weights (i.e., 32 bits), 8-bit quantized weights can effectively accelerate the network inference on mainstream CPUs without significant accuracy degradation. To this end, we discuss recent advances in the field of network quantization in this section, including representative quantized networks and popular quantization-related extensions/implementations, which are also summarized in Figure 21.

Network distillation, also referred to as knowledge distillation,⁷ is another well-established paradigm to further push forward the accuracy–efficiency trade-off, which is initially proposed by [501] and subsequently generalized by [11, 28]. Note that knowledge distillation is a plug-and-play training technique, which has been applied to various tasks to achieve better training performance, such as object detection [502] and language understanding [96]. In contrast to network pruning and network quantization, which focus on improving network efficiency without sacrificing network accuracy, as discussed in Sections 4.1 and 4.2, network distillation instead boosts the accuracy–efficiency trade-off from the accuracy perspective, which aims to improve the network accuracy without changing the network structure. In other words, unlike network pruning and network quantization that lead to simplified network structures, network distillation results in the same network. However, the resulting network can typically achieve higher accuracy. As shown in [11, 28, 501], knowledge distillation refers to the training process that leverages a larger pre-trained teacher network to benefit the training process of a smaller student network (see Figure 25), which transfers the rich and discriminative knowledge from the larger pre-trained teacher network to the smaller student network to further achieve better accuracy on the target task than simply training the student network alone. Next, we first present the preliminaries of knowledge distillation and then introduce recent representative data-dependent and data-efficient knowledge distillation works. These knowledge distillation works can also be found in Figure 24.

Knowledge Distillation Basics. In order to better understand knowledge distillation, we first elaborate on the preliminaries of knowledge distillation, which are mainly based on the most representative knowledge distillation work [11]. As shown in previous state-of-the-art networks [32, 34, 35, 85], the network outputs, also referred to as the network logits, are typically fed into the softmax function to calculate the probability distribution over different categories for further prediction purposes. However, given a pre-trained teacher network, the output logits after the softmax function are discriminative but less informative, which are close to either 1 or 0 (e.g, [0.02, 0.95, 0.01, 0.01, 0.01]). This makes it difficult to directly transfer the discriminative knowledge from the pre-trained teacher network to the student network. The rationale behind this is that the student network is of smaller network size than the teacher network, thus is less capable of learning discriminative knowledge [11]. To mitigate this issue, [11] leverages the distillation temperature T to soften the knowledge from the pre-trained teacher network to facilitate the teacher–student knowledge transfer process, which can be mathematically formulated as follows:where \(\lbrace y_i\rbrace _{i=1}^n\) denotes the output logits without softmax and T is the temperature to soften the output logits \(\lbrace y_i\rbrace _{i=1}^n\) as \(\lbrace z_i\rbrace _{i=1}^n\). Note that Equation (19) is equivalent to the standard softmax function when \(T=1\). As shown in [11], larger T can produce softer probability distribution over different categories (e.g., \([0.1, 0.6, 0.1, 0.1, 0.1]\) when \(T=5\) and \([0.2, 0.2, 0.2, 0.2, 0.2]\) when \(T=+\infty\)). In addition, fixing the temperature T to 2 can empirically yield the best performance. Furthermore, [11] exploits the softened knowledge from the pre-trained teacher network to guide the training process of the student network, which can be mathematically formulated as follows:where x is the input data, \(y^*\) is the ground-truth label, z is the softened knowledge from the pre-trained teacher network, \(\alpha\) is the constant to control the teacher–student distillation magnitude, and \(\mathcal {L}(\cdot)\) is the standard cross entropy loss function. Apart from these, \(f_w(\cdot)\) parameterizes the student network with the weight of w. As demonstrated in [11], it is important to multiply the teacher–student distillation loss term (i.e., \(\mathcal {L}(y, z)\)) with \(T^2\) because the teacher–student distillation loss term scales the gradient of w to \(1/T^2\) during the training process.

With the above in mind, we introduce recent representative knowledge distillation practices next, which are built upon [11] and can be roughly divided into the following two categories: data-dependent and data-efficient knowledge distillation.

In this section, we envision several promising future trends and possible directions in the field of network compression, which are summarized as follows:

In this section, we introduce recent state-of-the-art works about general on-device learning techniques, including efficient on-device inference and efficient on-device training.

Efficient On-Device Inference. To enable efficient on-device inference, one straightforward approach is to design tiny networks with less redundancy in order to accommodate the limited on-device computational resources. To this end, a plethora of representative tiny networks [512, 513, 514, 515] have been proposed recently, including MicroNets [512], MCUNets [513, 514], and EtinyNet [515]. MCUNetV1 [513], one of the early tiny networks, proposes to jointly design the lightweight tiny network using TinyNAS and the lightweight inference engine using TinyEngine, enabling ImageNet-scale inference on microcontrollers. MCUNetV2 [514] introduces an efficient patch-based inference pipeline to trim down on-device memory consumption for better on-device inference since memory consumption is the key bottleneck of on-device inference. However, in contrast to training large networks, training tiny networks poses significant challenges, as demonstrated in [547]. The rationale here is that existing regularization techniques (e.g., data augmentation and dropout), despite being able to benefit the training process of large networks, may degrade the training performance of tiny networks [547]. To tackle this issue, [547] proposes augmentation of the tiny network itself rather than augmenting the input data, which shows promising accuracy improvement over the standard training scheme.

Efficient On-Device Training. The key difference between on-device training and inference is that on-device training requires saving all of the intermediate activations, which are used to optimize parameters using gradient descent during backward propagation. In contrast, on-device inference that only performs forward propagation does not need to save intermediate activations, which can be progressively released to reduce memory consumption. In light of this, on-device training suffers from non-negligible memory consumption since the activation size grows with respect to the training batch size and training typically involves a large batch size to accelerate the training process. As a result, intermediate activations become the major bottleneck of on-device training, as demonstrated in [44]. For example, under the batch size of 16, the activation size of ResNet50 [2] is ×13.9 larger than its parameter size, as shown in Figure 27. To alleviate the excessive memory consumption caused by intermediate activations, there have been several representative strategies, including gradient checkpointing, activation gradient pruning, and low-bit training.

We note that the aforementioned strategies can also be combined to further reduce the overall memory consumption during training and accelerate the training process. For example, we can easily combine gradient checkpointing and low-bit training to further reduce the training memory consumption from \(O(\sqrt {n})\) to \(O(\sqrt {n}/32)\), where n is the number of network layers.

On-device continual learning, also known as on-device lifelong learning or incremental learning, is an advanced learning paradigm that allows the deployed network to continuously learn from the newly collected data to further push forward the attainable accuracy [46, 533]. This is particularly favored in real-world embedded scenarios, especially those with rich local sensors, where embedded devices can continue to collect new data through local sensors over time [44, 45]. The newly collected data is then used to train the deployed network to unlock better performance over time. In other words, we are allowed to utilize the newly collected data to train or fine-tune the deployed network on target hardware itself, which typically leads to better accuracy on the target task. On-device continual learning performs local training and does not need to send back the newly collected data to remote servers, which also protects data privacy and ensures data security. However, on-device continual learning, despite being able to deliver significant benefits, suffers from the catastrophic forgetting issue, which is the tendency to forget the previously learned knowledge when adapting to newly collected data [46]. The rationale is that on-device continual learning must adjust the pre-trained network weights in order to adapt to the newly collected data, which deteriorates the previously learned knowledge accordingly.

To alleviate the catastrophic forgetting issue, a plethora of state-of-the-art on-device continual learning works have been established recently [46, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534], which seek to stabilize the on-device continual learning process and further push forward the attainable accuracy on target the task. Among them, [46], an early exploration, investigates three common continual learning scenarios and demonstrates that it is frustratingly hard to evaluate different continual learning approaches, based on which [46] establishes several evaluation protocols to compare them. It is worth noting that the authors of [46] do not support on-device continual learning but we can easily integrate recent advances from the lens of on-device training (see Section 5.1) into [46] to further enable efficient on-device continual learning. Several subsequent on-device continual learning works [524, 525, 526, 527] explore on-device continual learning on resource-constrained embedded computing systems, which have since demonstrated promising accuracy improvement. In parallel, [528, 529, 530] attempt to generalize on-device continual learning to benefit language tasks in real-world embedded scenarios, such as environmental sound classification [528] and automatic speech recognition [530]. Inspired by the tremendous success of vision transformers, [107, 531] also investigate the efficacy of on-device embedded continual learning to continuously improve the accuracy of mainstream vision transformers. The works of [532, 533, 534] delve deeper into on-device continual learning, which focus more on the training pipeline and introduce several on-device training enhancements to maximize accuracy improvement, such as selective weight updates [532], weight freezing [533], and deep network ensembles [534].

As demonstrated in [44], it is often different to directly train DNNs from scratch in real-world embedded scenarios, in which the collected data samples are far limited. To tackle this issue, an effective alternative is on-device transfer learning, which fine-tunes pre-trained networks on large-scale datasets. The rationale here is that DNNs pre-trained on large-scale datasets (e.g., ImageNet [80]) can serve as powerful feature extractors for further transfer learning, which only fine-tunes several layers (e.g., batch normalization layers and the last layer) whereas other layers are typically frozen. In contrast to previous on-device learning practices as discussed in Sections 5.1 and 5.2, on-device transfer learning does not require storing memory-intensive intermediate activations. As a result, it maintains significant efficiency in terms of training memory consumption, as illustrated in Figure 28 [44]. Despite the promising memory efficiency, on-device transfer learning is quite challenging, which may result in poor accuracy, especially on those datasets whose data distribution is far from ImageNet [82].

To overcome such limitations, early transfer learning practices [535, 536] propose to fine-tune all the network layers, which indeed achieves better accuracy but leads to considerable memory consumption due to memory-intensive intermediate activations. To avoid this, several subsequent transfer learning works [537, 538, 539, 540] demonstrate that it is often not necessary to fine-tune all the network layers, which indicates that fine-tuning batch normalization layers can also achieve strong accuracy on the target task. This fine-tuning paradigm has the potential to significantly reduce the number of trainable parameters during the transfer learning process. In light of this, [537, 538, 539, 540] propose to only optimize learnable parameters in batch normalization layers (see \(\gamma\) and \(\beta\) in Equation (13)), whereas other learnable parameters are frozen during the transfer learning process. For example, [537] leverages batch normalization layers as scale-and-bias patches and then trains the patched parameters, optionally also the last layer, whereas the remaining parameters are left unchanged. Furthermore, [538] reveals that, for those networks with sufficient depth, training only \(\gamma\) and \(\beta\) can reach surprisingly strong accuracy, which demonstrates the expressive power of the learnable parameters in batch normalization layers. However, fewer trainable parameters cannot directly translate to superior training memory efficiency as shown in Figure 27, which may still involve a large amount of memory consumption (e.g., 326 MB under the training batch size of 8) to store memory-intensive intermediate activations of batch normalization layers [44].

To further alleviate the prohibitive training memory consumption, [44] introduces a simple yet effective transfer learning solution, which exhibits significant training memory efficiency. Specifically, [44] relies on an empirical observation that intermediate activations are only required to update network weights, whereas updating network biases does not involve intermediate activations. This observation also reveals that the training memory bottleneck comes from updating network weights rather than biases. In light of this, [44] proposes to freeze network weights and only update network biases. However, freezing network weights and only updating network biases may lead to significant accuracy loss. To compensate for such accuracy loss due to freezing network weights, [44] introduces an effective lite residual learning scheme, which leverages generalized memory-efficient bias modules to refine memory-intensive intermediate activations. In particular, the lite residual learning scheme can improve the attainable accuracy on the target task and, more importantly, at the cost of negligible memory overheads. Finally, [44] reduces the training memory consumption from more than 250 MB to only 16 MB, making it possible to explore in-memory computing infrastructures to perform memory-efficient transfer learning. Note that we can easily integrate recent advances from the lens of on-device training (see Section 5.1) into the aforementioned transfer learning works towards boosted on-device transfer learning performance.

On-device federated learning is an advanced decentralized learning paradigm that enables efficient training on a large corpus of decentralized data residing on local client devices such as mobile phones and allows multiple local client devices to jointly train the given network without explicitly sharing their raw data [541, 549]. In practice, on-device federated learning has the potential to significantly accelerate the training process when the number of client devices evolves. In addition, on-device federated learning is one instance of the more general approach of “bringing the neural network to the data” rather than “bringing the data to the neural network.” As a result, it addresses the fundamental problems of data privacy, security, and ownership [508]. This is particularly favored in real-world embedded scenarios, in which embedded devices can continue to collect new data through local sensors. Thanks to these practical benefits, on-device federated learning has garnered increasing attention from both academia and industry [47]. In the past decade, on-device federated learning has been utilized to empower a plethora of real-world intelligent applications, such as mobile keyboard content suggestions [550], medical image analysis [551], and smart health care infrastructures [552]. As demonstrated in [541], standard on-device federated learning practices typically consist of the following five iterative steps:

Despite being able to deliver superior learning performance across various real-world embedded tasks, on-device federated learning suffers from critical limitations, especially from the perspective of data transmission [508], posing significant challenges to generalizing on-device federated learning to benefit real-world intelligent embedded applications. In contrast to the centralized server that is equipped with high-end network infrastructures, local client devices in real-world embedded scenarios are often low end with less-capable network infrastructures. In such a case, it may be time-consuming to (1) distribute the global model from the centralized server to local client devices and (2) send back model update schemes from local client devices to the centralized server for further aggregation. To overcome such limitations, a plethora of advanced federated learning techniques have been established recently to accommodate the limited data bandwidth of local client devices [47, 414, 541, 542, 543, 544, 545], which primarily focus on reducing the total data bits transferred between the remote centralized server and local client devices, such as federated averaging [541], gradient compression [542, 543], quantization [414], delayed gradient averaging [544], partial variable training [545], and local training sparsity [47]. Note that we can easily integrate recent advances from the lens of general on-device training techniques (see Section 5.1) into the aforementioned federated learning works to further enhance on-device federated learning.

In this section, we envision several promising future trends and possible directions in the field of on-device learning, which are summarized as follows.

LLMs are emerging machine learning models that are dedicated to understanding, generating, and interacting with human language through leveraging extensive textual data. In practice, LLMs are typically built upon a transformer with both encoder and decoder [90] and heavily rely on self-attention mechanisms to measure the significance of different words in the given sentence regardless of their positional relationships. Thanks to their strong capability to interpret rich information, LLMs can exhibit remarkable performance across a wide range of language processing tasks, such as text summarization, translation, question answering, and conversational response generation. As discussed in [50], recent state-of-the-art LLMs can be divided into three main categories according to their inherent architectures: encoder-only architecture, decoder-only architecture, and encoder–decoder architecture as follows.

As discussed in Section 6.1, recent LLMs are typically built upon transformers [90] and heavily rely on self-attention mechanisms to interpret the significance of different words in a given sentence, regardless of their positional relationships. However, self-attention mechanisms, despite their strong capability for language processing, also introduce considerable computational complexity. As pointed out in [599], the quadratic time and memory complexity of self-attention mechanisms may significantly slow down the pre-training, inference, and fine-tuning stages of LLMs. To optimize the prohibitive computational complexity of LLMs, recent state-of-the-art efficient LLMs often focus on exploring computation-efficient self-attention mechanisms.

General Efficient Attention. Some recent works [558, 559, 560, 561, 562, 563] focus on exploring computation-efficient attention mechanisms to optimize the quadratic computational complexity of vanilla self-attention [90]. Among them, [558] introduces clustered attention, which groups different queries into clusters and computes the attention just for the centroids rather than computing the attention for every query. To further improve the above approximation, [558] also employs the computed clusters to identify the keys with the highest attention per query and computes the exact key-query dot products. The work in [559] draws insights from the Nyström method and proposes approximating the standard self-attention mechanism in linear complexity, which can enable applications to longer sequences with even thousands of tokens. The work of [560] demonstrates that the complexity bottleneck of self-attention mainly comes from the computation of partition functions in the denominator of the softmax function and the multiplication of the softmax matrix with the matrix of values. To this end, [560] features an efficient kernel density estimation (KDE) solver to resolve the above complexity bottleneck via subsampling-based fast matrix products, which can approximate the attention in subquadratic time with provable spectral norm bounds. The work of [561] introduces an efficient single-head gated attention mechanism with an exponential moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism, which can exhibit linear time and space complexity while causing minimal performance loss. The work of [562] introduces an efficient universal approximation for self-attention, which can exhibit linear time and space complexity and reveal the theoretical insights behind existing efficient transformers (e.g., Linformer [97]) with linear time and space complexity. The work of [563] introduces an efficient attention approximation mechanism featuring fused low-rank kernel approximation, which can provide sizable runtime performance gains and is also high quality.

Hardware-Aware Efficient Attention. In parallel to the above efficient attention approximation works, some recent works [53, 54, 55, 564, 565, 566] focus on exploring efficient hardware-aware attention mechanisms, which can exhibit considerable efficiency improvement on modern hardware systems. Among them, FlashAttention [54] features an efficient IO-aware exact attention algorithm, which explores tiling to reduce the total number of memory reads and writes between GPU high bandwidth memory and GPU on-chip Static Random Access Memory (SRAM). However, FlashAttention is still not nearly as fast as optimized matrix-multiply (GEMM) operations, which is mainly due to the suboptimal work partitioning between different thread blocks and warps on GPUs. To tackle this issue, FlashAttention-2 [55] introduces an improved work partitioning scheme, which (1) tweaks the algorithm to reduce the number of non-MatMul FLOPs, (2) parallelizes the attention computation workloads across different thread blocks, and (3) distributes the work between warps to reduce communications through shared memory. FLASHLINEARATTENTION [566] dives deeper into I/O awareness and introduces an effective hardware-efficient algorithm for linear attention, which trades off memory movement against parallelizability and can even be faster than FlashAttention-2. PagedAttention [53] draws insights from the operating system’s solution to memory fragmentation and sharing through virtual memory with paging and further divides the request’s key-value (KV) cache into different blocks, each of which contains the attention keys and values of a fixed number of tokens. A3 [564] demonstrates that implementing attention mechanisms using matrix-vector multiplication is often suboptimal and further proposes to accelerate attention mechanisms with joint algorithmic approximation and hardware specialization. Similar to A3, ELSA [565] features an effective approximation scheme to significantly reduce the amount of computation workloads by efficiently filtering out relationships that are unlikely to affect the final output.

In addition to designing LLMs with efficient architectures, another promising direction is to explore efficient LLM compression techniques to optimize the computational complexity of existing computation-intensive LLMs. With this in mind, we discuss recent state-of-the-art compression techniques for LLMs in this section, including efficient LLM pruning in Section 6.3.1, efficient LLM quantization in Section 6.3.2, and efficient LLM distillation in Section 6.3.3.

In parallel to the rapid development of efficient LLM algorithms, a plethora of efficient LLM systems and infrastructures have recently emerged [63, 64, 65, 593, 594, 595, 596, 597, 598] that further optimize the generative inference efficiency of LLMs from the perspective of efficient system-level implementations. FlexGen [63] features an efficient high-throughput generation engine for running LLMs on one single GPU with limited memory, which can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. After solving a linear programming problem, FlexGen can also search for efficient patterns to store and access tensors. Tabi [65] features an inference system with an efficient multi-level inference engine, which can serve queries using small models and optional LLMs for demanding applications. Tabi is particularly optimized for discriminative models (not generative LLMs) in a serving framework, which uses the calibrated confidence score to determine whether to directly return the accurate results of small models or further re-route them to LLMs. DeepSpeed [593] presents a comprehensive system solution for efficient transformer inference, which consists of (1) a multi-GPU inference engine to minimize the runtime latency while maximizing the runtime throughput of both dense and sparse transformers when they fit into aggregate GPU memory and (2) a heterogeneous inference engine that leverages CPU and NVMe memory in addition to GPU memory and computation to enable high inference throughput with large models that do not fit into aggregate GPU memory. FastServe [594] presents an efficient distributed inference serving system for efficient LLM inference that (1) exploits the auto-regressive pattern of LLM inference to enable preemption at the granularity of each output token and (2) explores preemptive scheduling to minimize job completion time with a novel skip-join multi-level feedback queue scheduler. Petals [64] features an efficient collaborative system for runtime inference and fine-tuning of LLMs through joining the resources of multiple parties. In contrast to concurrent LLM inference engines, Petals also natively exposes hidden states of the served model, allowing training and sharing of custom model extensions based on efficient fine-tuning schemes.

S\(^3\) [595] demonstrates that designing an inference system with prior knowledge of the output sequence can largely increase the runtime inference throughput of LLMs. Therefore, in order to increase the runtime inference throughput of LLMs, S\(^3\) proposes to (1) first, predict the output sequence length, (2) then schedule generation queries based on the prediction to increase runtime resource utilization and throughput, and, (3) finally, handle mispredictions. Thanks to the prior knowledge of the output sequence, S\(^3\) can achieve much better inference throughput than early LLM systems. Splitwise [596] proposes to split the two phases of typical LLM inference workloads on to different hardware, which allows use of well-suited hardware for each inference phase and provision independent computational resources for each inference phase. This strategy can improve the runtime resource utilization across different hardware. Splitwise also optimizes the state transfer across different hardware using the fast back-plane interconnects in today’s GPU clusters to further increase the runtime LLM inference throughput. DistServe [597] proposes to disaggregate the prefill and decoding computation to enhance the runtime serving performance of LLMs, which assigns the prefill and decoding computation workloads to different GPUs and, thus, largely eliminates the prefill-decoding interference towards better runtime inference throughput. DistServe also optimizes the above two phases according to the serving cluster’s bandwidth to minimize the communication overheads caused by the disaggregation. Liger [598] features an efficient distributed collaborative inference system for LLMs, which can achieve low inference latency at high throughput on multiple GPUs. In addition, to achieve high parallelism and throughput, Liger also introduces an efficient scheduling strategy to effectively schedule the computation and communication kernels across different input requests onto multiple streams of multiple GPUs.

In this section, we envision several promising future trends and possible directions in the field of efficient LLMs, as follows.

In this section, we introduce popular deep learning software frameworks that have been widely used to develop deep learning solutions for embedded computing systems, including TensorFlow [66], PyTorch [67], Caffe [610], MXNet [611], Keras [612], CoreML [613], PaddlePaddle [614], and BigDL [615]. These frameworks are summarized in Table 3.

TensorFlow [66] is an open-source deep learning software framework developed by Google, which was released in 2015 and has since become one of the most popular deep learning software frameworks for training and deploying DNNs. TensorFlow, especially TensorFlow Lite, allows developers to easily build and deploy DNNs in a wide range of embedded computing systems, including mobile phones, microcontroller units (MCUs), Raspberry Pi, TPUs, and edge GPUs. TensorFlow also supports various real-world applications, ranging from image and speech recognition to NLP and predictive analytics. With its flexible architecture and vast pre-trained models, TensorFlow has been deemed to be one of the most important tools for researchers and developers in the field of deep learning.

PyTorch [67] is an open-source deep learning software framework that is widely used for training and deploying deep neural networks. It was developed by Facebook (now known as Meta) and was released in 2016. One of the key features is the dynamic computation graph, which allows developers to change the computation behavior of DNNs on the fly. This feature distinguishes PyTorch from other deep learning software frameworks (e.g., TensorFlow) that only support static computation graphs. In addition, PyTorch has a number of high-level features that make it easier to build more complex DNNs. For example, the TorchVision package provides various useful tools and pre-trained models for image and video processing. With its dynamic computation graph, ease of use, and range of high-level features, PyTorch has become an essential software framework for training and deploying DNNs in the deep learning community.

Caffe [610] is a popular deep learning software framework developed by Berkeley and released in 2014, which has gained increasing popularity owing to its speed, modularity, and ease of use. One of the key features is its ability to deal with large datasets containing millions of images. Another important feature is its modularity, which allows developers to add or remove components with ease. In addition, Caffe includes a large library comprising hundreds of pre-trained models that can be used to quickly build deep learning applications, such as image classification, object detection, and segmentation. In addition to its powerful features, Caffe has a user-friendly interface, allowing developers to train and deploy DNNs without extensive knowledge of deep learning. With its powerful features and user-friendly interfaces, Caffe has become an invaluable tool for researchers and developers and inspired subsequent deep learning software frameworks.

MXNet [611] is an open-source deep learning software framework to train and deploy DNNs, which was developed by Amazon and released in 2015. One of the key technical merits is its distributed training capability, which allows training of DNNs across multiple computation nodes in a computationally efficient manner. MXNet also supports multiple programming languages, such as Python, C++, R, and Julia, which increases the accessibility to researchers and developers with diverse skill levels. MXNet’s integration with other deep learning software frameworks and tools, such as Apache Spark and Apache Flink, is another important feature that facilitates the integration of deep learning into existing data processing pipelines. Thanks to its scalability, flexibility, and efficiency, MXNet has become a popular option for developers and researchers in the deep learning community.

Keras [612] is an open-source deep learning software framework written in Python, which provides high-level APIs for building and training efficient DNN solutions. It was developed by François Chollet in 2015 and is now maintained by a community of developers. Keras has been integrated into TensorFlow; starting from TensorFlow 2.0, Keras has become the default API to build DNN solutions in TensorFlow. One of the key features of Keras is its modularity, which allows developers and researchers to easily construct and customize DNNs. Keras also allows users to productize DNN solutions on modern mobile phones such as iOS and Android, on the web, or on the Java virtual machine. Last, but not least, Keras allows training of DNN solutions in an efficient distributed manner on clusters of multiple GPUs and TPUs. These strengths make Keras increasingly popular in both industry and academia.

CoreML [613] is an open-source deep learning software framework developed by Apple in 2017, which aims to integrate DNNs into Apple commercial products, such as the iPhone, iPad, and Apple Watch. In addition to supporting extensive DNNs with over 30 layer types, CoreML covers standard machine learning models, such as tree ensembles, support vector machine, and generalized linear models. Another key feature of CoreML is its ability to directly run DNNs on the device without the need for cloud-based inference. CoreML also provides a range of optimization techniques, such as quantization and pruning, to reduce the complexity of DNNs. This is particularly important for modern mobile devices, which typically have limited storage and computational resources. Also, CoreML, built on top of advanced technologies such as Metal and Accelerate, seamlessly takes advantage of CPUs and GPUs to provide the maximum inference performance at runtime. These technical features make CoreML the first choice for developing efficient DNN solutions on Apple commercial products.

PaddlePaddle [614], also known as Paddle, is an open-source deep learning software framework developed by Baidu, which was released in 2016 to benefit the deep learning community. PaddlePaddle is designed to be an industrial platform with advanced technologies and rich features that cover core deep learning frameworks, basic model libraries, end-to-end tools, and service platforms. PaddlePaddle originated from industrial practices with dedication and commitment to industrialization. It has been adopted by a wide range of sectors, including manufacturing, agriculture, and enterprise service. With the industrial benefits, PaddlePaddle has motivated an increasing number of developers and researchers to commercialize AI.

BigDL [615] is an open-source deep learning software framework that runs on top of Apache Spark. It was developed by Intel and released in 2017. The goal of BigDL is to provide a high-performance, scalable, and easy-to-use platform, especially distributed deep learning. To this end, BigDL includes a comprehensive set of features that cover various deep learning applications, including image classification, object detection, and NLP. One of the key features of BigDL is its ability to take full advantage of distributed computing resources, such as CPU, GPU, and FPGA clusters, to accelerate the training of DNNs. BigDL is seamlessly integrated with Apache Spark, which enables users to leverage the distributed computing capability of Spark for data preprocessing and postprocessing. This integration also makes it possible to build end-to-end deep learning pipelines that span from data ingestion to model deployment.

In this section, we introduce popular embedded hardware platforms that are designed to run powerful DNNs in embedded scenarios without cloud-based assistance, including NVIDIA Jetson [69], Intel Neural Compute Stick [70], Google Edge TPU [68], Google Coral Dev Board [616], Huawei HiKey 970 [617], and Orange Pi AI Stick Lite [618]. These frameworks are summarized in Table 4.

NVIDIA Jetson [69] is a series of embedded systems-on-modules (SoMs) designed by NVIDIA for running advanced deep learning workloads, especially the inference of DNNs. NVIDIA Jetson consists of NVIDIA Jetson TX2, NVIDIA Jetson Nano, NVIDIA Jetson AGX Xavier, NVIDIA Jetson Xavier NX, NVIDIA Jetson AGX Orin, and NVIDIA Jetson Orin NX. To accelerate deep learning workloads, NVIDIA Jetson runs on top of NVIDIA’s CUDA parallel computing architectures and features an integrated system-on-chip (SoC) with a powerful NVIDIA GPU, a multi-core CPU, and various high-speed interfaces, including Ethernet, USB, HDMI, and CSI/DSI. More importantly, NVIDIA Jetson is compatible with various deep learning software frameworks, including TensorFlow, PyTorch, Caffe, Keras, and MXNet. Thanks to its advanced architectural design and powerful interfaces, NVIDIA Jetson is able to support a wide range of embedded deep learning applications to accommodate different resource and performance requirements.

Intel Neural Compute Stick [70] is a small, low-power, and cost-effective embedded hardware designed to run deep learning workloads without cloud-based assistance, which was developed by Movidius (now acquired by Intel). The Neural Compute Stick (NCS) is a small USB device that can be connected to a host computer or embedded computing system. NCS features the Myriad 2 Vision Processing Unit (VPU), which is optimized for the inference of DNNs. In addition, NCS is integrated with various high-speed interfaces, including USB 3.0 and Wi-Fi. Developers can use the Intel Movidius SDK, which provides a set of tools for developing, testing, and deploying DNNs on NCS. Furthermore, NCS supports various deep learning software frameworks, including TensorFlow, Caffe, and MXNet. Thanks to its significant flexibility and cost efficiency, NCS makes it possible to deploy advanced DNNs in a wide range of embedded scenarios.

Google Edge TPU [68] is a custom-built ASIC chip to accelerate deep learning workloads on resource-constrained edge computing systems. The Google Edge TPU is designed to seamlessly work together with TensorFlow Lite, a lightweight version of TensorFlow, and is optimized for the inference of DNNs towards enhanced inference efficiency. Google Edge TPU itself cannot work alone and, similar to the Intel Neural Compute Stick, it must be connected to other embedded computing systems, such as Raspberry Pi 4 and the Google Coral Dev Board, to deliver deep learning solutions. The Google Edge TPU is capable of performing up to two trillion floating-point operations per second (TFLOPS) and four trillion operations per second (TOPS) using only two watts of power. This further allows us to build and deploy powerful deep learning solutions on embedded computing systems with limited computational resources. Thanks to its easy integration with other embedded computing systems, powerful performance, and significant efficiency, the Google Edge TPU has gained increasing popularity in the deep learning community for deploying deep learning solutions on embedded computing systems.

Google Coral Dev Board [616] is a single-board computer designed for building embedded deep learning applications. It features an on-board Google Edge TPU, which is a custom-built chip to run TensorFlow Lite models with high performance and low power consumption. The Coral Dev Board has various built-in interfaces, including Audio, Wi-Fi, Bluetooth, Ethernet, and USB 3.0, which enable it to be connected to other embedded computing systems. The Coral Dev Board is also integrated with 1 GB LPDDR4 RAM, 8 GB eMMC 5.1 flash memory, and a MicroSD slot for additional storage. The Coral Dev Board also comes with pre-installed software tools, including TensorFlow Lite, Google Edge TPU API, and various sample applications. These allow users to easily and quickly start building their deep learning solutions. Thanks to its powerful Google Edge TPU, various useful interfaces, and software tools, the Coral Dev Board has become increasingly popular for developing and deploying embedded deep learning solutions.

Huawei HiKey 970 [617] is a high-performance, single-board embedded computer designed by Huawei. The HiKey 970 features a powerful neural processing unit (NPU) to accelerate various deep learning workloads. The HiKey 970 is also integrated with 6 GB LPDDR4 LPDDR4 RAM and 64 GB UFS 2.1 flash memory, while at the same time allowing MicroSD extension for additional storage. In addition, the HiKey 970 supports various high-speed interfaces, including the Ethernet, USB 3.0, and PCIe 3.0. Furthermore, the HiKey 970 is compatible with popular deep learning software frameworks, such as TensorFlow, PyTorch, and Caffe, allowing users to easily and quickly build on-board deep learning solutions. Thanks to its powerful NPU, rich memory and storage, and extensive connectivity options, the HiKey 970 is suitable for a wide range of intelligent embedded applications, such as robotics, autonomous vehicles, and smart home devices.

Orange Pi AI Stick Lite [618] is a tiny and cost-effective USB stick designed for small to medium-sized deep learning workloads. It is equipped with a single-core Cortex-A7 processor and an NPU that provides hardware acceleration. Note that, similar to the Intel Neural Compute Stick and Google Edge TPU, the Orange Pi AI Stick Lite cannot work alone; it must be connected to a host device using the on-device USB 3.0 interface. The Orange Pi AI Stick Lite supports various deep learning software frameworks, including TensorFlow, PyTorch, and Caffe. Thanks to its cost efficiency, the Orange Pi AI Stick Lite is suitable for embedded computing systems to deal with small to medium-sized deep learning workloads.

In this section, we envision the future trends and possible directions of deep learning software and hardware infrastructures, summarized as follows.

Computer vision is an emerging field that focuses on interpreting and understanding visual information from real-world environments, such as images and video, spanning from image classification [1] to downstream vision tasks, such as object detection [4], tracking [5], and segmentation [6]. Below we discuss recent popular intelligent embedded vision applications.

Image Classification. Image classification, also referred to as image recognition, is the most fundamental vision task, which focuses on recognizing the input image based on its visual information [1]. Various intelligent applications in real-world embedded computing systems, such as mobile phones and IoT sensors, enable these embedded computing systems to automatically recognize objects, scenes, or patterns within the given image [627]. For example, face recognition [628], person re-identification [503], and hand gesture recognition [629] have been widely integrated into mainstream embedded computing systems, such as mobile phones, ATMs, and intelligent cameras, for the purpose of identity authentication. In practice, image classification typically features deep convolutional networks, such as VGGNet [1], ResNet [2], and DenseNet [3], thanks to their strong capabilities to capture rich visual information, especially for large-scale datasets such as ImageNet [80]. For example, as shown in Figure 30, AlexNet [76], the first of its kind, demonstrates the possibility of leveraging convolutional layers to learn discriminative features from vision inputs, which exhibits significantly better recognition performance on ImageNet than previous well-established non-convolutional networks, such as MLPs and other learning-based techniques. ResNet [2] investigates the training collapse of deep convolutional networks and introduces a simple yet effective deep residual learning paradigm, which allows us to significantly increase the network depth for stronger learning capabilities and also marks the booming development of the deep learning era. As a result, ResNet, for the first time, achieves better recognition performance on ImageNet than humans thanks to its significant network depth, as shown in Figure 30.

Downstream Vision Applications. Downstream vision applications typically refer to the practical and specific usage, in which the results or outputs from other fundamental vision tasks, such as image classification, are applied to deal with real-world challenges. Popular downstream vision applications in practice include but are not limited to object detection [4], object tracking [5], object segmentation [6], image super-resolution [277, 278], image restoration [630], pose estimation [631], image captioning [632, 633], augmented reality (AR) and virtual reality (VR) [634], and video-related analysis [635]. The work of [630] features memory-oriented structured pruning to optimize the on-device memory consumption during runtime image restoration, which can accommodate the limited memory and storage requirements in real-world embedded scenarios. These downstream vision applications have evolved to be ubiquitous in real-world embedded scenarios, which serve as important components towards ubiquitous embedded intelligence. For example, object detection and tracking have been widely used in recent autonomous vehicles [636] to detect other vehicles and in surveillance systems [637] to detect suspicious persons or activities. These downstream vision applications have also been widely applied to other real-world embedded scenarios, such as smart cities and intelligent health care [638]. To further facilitate the development of intelligent applications, several powerful tools have been proposed recently. For example, Precog [639] introduces an efficient object detection infrastructure to enable real-time object detection on resource-constrained embedded computing systems, such as Raspberry Pi, which also features YOLOv3 [640] to achieve superior on-device object detection accuracy.

From CNNs to Vision Transformers. More recently, vision transformers (ViTs) [107] and their variants have demonstrated surprisingly strong performance in various vision tasks, including, but not limited to, image classification [107, 108, 109], object detection [110, 111, 112], semantic segmentation [113, 114, 115, 116], and video analysis [117, 118, 119], which continue to push forward the state-of-the-art performance over their convolutional counterparts across various vision tasks. Specifically, [107], featuring the very first vision transformer, proposes to divide the input image into a series of smaller image patches (e.g., 8, 16, and 32 patches), each of which is then fed into the transformer-based encoder to learn discriminative features. The learned discriminative features are further aggregated and fed into the classification layer to make predictions as shown in Figure 5. However, despite their strong performance across various vision tasks, ViTs and their variants often exhibit inferior on-device efficiency [139] since they are typically more difficult to parallelize on resource-constrained embedded computing systems than their convolutional counterparts and, thus, inevitably suffer from considerable resource underutilization, as pointed out in [127]. To overcome such limitations, a plethora of resource-efficient vision transformers have recently flourished. We refer interested readers to Section 2.2 for more details about recent representative resource-efficient vision transformers. We emphasize that significant efforts are still required in order to further alleviate the on-device efficiency bottleneck and unleash the promise of modern vision transformers, which are of paramount importance to bring powerful vision transformers to the less capable embedded computing systems towards ubiquitous embedded intelligence.

In parallel to vision tasks, NLP is another representative application that has been widely deployed in real-world embedded scenarios to explore auditory and textual inputs, which has largely revolutionized how embedded computing systems interact with users and their surroundings [641]. Embedded computing systems ranging from traditional IoT systems to wearable systems and autonomous systems are transitioning from simple responsive systems to more proactive and interactive systems, which can comprehend context and also anticipate users’ needs based on their linguistic inputs. To this end, below we introduce several representative NLP applications in real-world embedded scenarios.

To summarize, the integration of NLP and embedded computing systems is more than simple technical enhancements. It is an important paradigm shift towards ubiquitous embedded intelligence. It can enable real-world embedded computing systems to understand and interpret not only short commands but also longer contexts and conversations, which can ensure seamless and enriched interfaces between humans and embedded computing systems.

In this section, we envision some future trends and possible directions of intelligent embedded applications, which are summarized as follows: