Benchmarking Test-Time DNN Adaptation at Edge with Compute-In-Memory

Two widely adopted approaches for improving the robustness of DNNs are data augmentation and adversarial training, both of which are elaborated below.

Recent advancements in the field of adaptation emphasize unsupervised domain adaptation, especially fine-tuning specific neural network layers during test-time. Techniques such as “Norm” adapt batch-normalization (BN) parameters by recomputing the mean and standard deviation statistics during test-time inference [49, 58], “Tent” [68] optimizes transformation parameters in addition to BN parameters but also performs a backpropagation pass at test-time to optimize the transformation parameters that apply channel-wise scales and shifts to the features, and “Surgical” fine-tunes different convolution layers based on the type of data corruption [39]. These works provide important insights into the overhead of running these algorithms on resource-constrained devices but only focus on adapting batch-norm parameters. Another recent work adapted batch-norm parameters to improve accuracy for lane detection applications in autonomous driving [9]. In our work, we focus on layers beyond batch-norm, such as convolutional and fully connected (FC) layers, and target not just corruptions but also noise due to analog operation inherent in in-memory computing architectures.

CIM has emerged as a promising architectural paradigm, facilitating operations across entire memory blocks by seamlessly integrating fundamental processing capabilities within the memory itself, thus effectively surmounting the memory wall challenge that existed in the Von Neumann machine [40]. With novel models leaning towards memory-intensive operation on smaller, resource-constrained devices at the edge [19, 75], benchmark existing algorithms with edge devices become even more critical.

Specifically, the CIM paradigm is able to perform matrix-vector multiplication, the operation that is ubiquitous in today’s novel machine learning models [1, 16, 31], within one clock cycle, and CIM has been leveraged to accelerate several operation- and data-intensive workloads such as vector symbolic architecture [34, 43], recommendation systems [15], reinforcement learning [42], spiking neural networks [48, 57], deep neural networks [50], bioinformatics [4], and so on. These CIM designs leverage novel non-volatile memory (NVM) technologies such as RRAM [14], FeFET [61], and PCM [5] for inference and/or training for novel applications.

Recent works have been actively exploring CIM benchmarking. DNN+NeuroSim [51] is tailored for assessing CIM accelerators over a broad spectrum of device technologies and architectures. Reis et al. [55] concentrate on the modeling and benchmarking of CIM architectures, delving into design space exploration and the evaluation of various memory technologies. He et al. [25] investigate the design space and memory technology co-exploration for CIM-based machine learning accelerators, seeking to find the optimal mix of design variables and memory technologies to enhance performance and reduce power consumption and area. NVMExplorer [53] introduces a comprehensive framework for cross-stack comparisons of embedded NVM technologies.

Although benchmarking DNN with CIM has been widely implemented, DNN adaptation benchmarked against CIM has yet to be investigated. In addition, inspired by the fact that DNN adaptation that is benchmarked against the state-of-the-art Von Neumann machine cannot satisfy the real-time and energy requirement at the edge [6, 7], in this work, we perform a thorough DNN adaptation benchmarking with CIM by truthfully integrating the existing CIM DNN benchmarking tool NeuroSim [50] for adaptation, enabling a different network wrapper to be integrated with the underlying circuit components.

Figure 1 shows an overview of our hardware-algorithm benchmark study for test-time DNN adaptation at the edge with CIM. We first train a model on a relatively large source dataset from meta-environments and then fine-tune the pre-trained model on a small target dataset as a means of adapting to distribution shifts. In our work, we only adapt a part of the network model during the fine-tuning stage while freezing the other layers.

Typically, adaptation involves a set of more conservative hyper-parameters, for instance, a smaller learning rate. The object to be adapted is a model pre-trained on a given dataset on the cloud. The goal is to strengthen model robustness when encountering different input data distribution or when non-ideal factors such as device non-ideality deplete model resilience [54]. By denoting trainable parameters of an n-layer network as \((\theta _{1},\ldots \theta _{n})\) , pre-training can be formulated as the following:

Here, x refers to input data and y the corresponding output. Contrary to conventional wisdom that one should fine-tune the whole model or the last few layers to reuse the learned features, we observe that fine-tuning only the part of layers of the network results in better performance for image corruption datasets and autonomous navigation scenarios, which is in line with [39]. This means that we judiciously choose parameters in certain layers, but not all of them, to perform backpropagation and weight updates, i.e.,where \(x_{shift}\) and \(y_{shift}\) are the test data and their labels (in supervised learning scenarios) due to distribution shifted or inherent CIM noise during on-device adaptation.

Datasets. We conducted our experiments primarily on the CIFAR-10 dataset and its variants. To simulate the circumstances of data distribution shift, we applied CIFAR-10-Corruption (CIFAR-10-C) and CIFAR-Flip to represent the workload distribution encountered after network deployment [28]. Before deploying, we pre-trained the models with CIFAR-10 while testing and adaptation were both carried out on variant CIFAR datasets (CIFAR-10-C, CIFAR-Flip). Figure 2 gives an example of how the original image is distorted [63].

Models. Since the CIFAR dataset is meant for image classification, we chose three image-processing neural network models as our candidate workload. VGG8 represents the so-called ‘shallower’ network. ResNet20 [26] and DenseNet40 [30] are selected as ‘deeper’ networks. Our evaluation results demonstrate that the number of layers (i.e., network depth) has impacts on the adaptation strategy, especially under quantization scenarios.

WAGE Quantization. Hardware that operates at the edge suffers from limited energy and area budgets. Energy and area consumption associated with floating-point (FP) operations are not advantageous for edge systems. In addition, the FP operation is not favorable when it comes to CIM-based implementations. Therefore, we applied WAGE as our quantization methodology [71]. WAGE quantizes four main operands (Weight, Activation, Gradient, Error) in backpropagation, rendering network deployment on edge possible. WAGE mainly covers linear and convolutional layers. Other layer types, such as maxpooling and batch norm, are omitted assuming they would be handled by other modules.

Our adaptation benchmarking framework of this work is built based on NeuroSim [50, 51, 52], an end-to-end CIM macro-to-architecture simulation toolkit. NeuroSim is capable of evaluating both the algorithmic network model and CIM hardware performance during the training and inference phases. It has incorporated PyTorch and WAGE quantization in its Python-based shell wrapper and uses chip-validated data to calibrate the C++-based files for analytical hardware performance-power-area estimations [45]. We refer to the hardware evaluation module (aforementioned C++-based files) as “core” in the following text. The input feature map, layer weights, and gradient traces generated from training are recorded for detailed analysis for hardware evaluations.

NeuroSim has set up its own design principle to determine what would be the CIM macro’s architecture given some user-specified macro parameters, starting from NVM device characteristics to architecture-dependent parameters. To perform partial adaptation analysis, we intervene in trace communication between the Python shell and core by modifying weight and gradient traces during adaptation. In a similar manner, all sorts of CIM hardware non-ideality are simulated by blending mathematical models into the trace files. We refer to NeuroSim manuals for a detailed description of execution flow and non-ideality modeling [72]. A high-level workflow abstraction of the adaptation framework can be found in Figure 4.

Parameter settings. We use the default parameter settings in NeuroSim throughout the adaptation experimentation, except for the memory cell type and subarray size. The VGG8 network is evaluated with a 128 \(\times\) 128 subarray whereas ResNet20 is evaluated with a 32 \(\times\) 32 subarray. The array size is chosen to prevent violating the design principle set up by NeuroSim. The experiments are mainly conducted with Static Random Access Memory (SRAM) and RRAM with different technology nodes. We use a unified bit-width of 8 in WAGE quantization to support training. The batch size is set to 200 and the epoch is usually 3 to accommodate the endurance of eNVM cells. We use 1,000 images for CIFAR-10-C implementation, with an 8:2 split for training and testing. When tested on CIFAR-10-C, the numbers reported in Section 4 are obtained through averaging test set accuracy among 10 types of corruptions. The severity for CIFAR-C is set to 5 (which is the most severe) and images are shuffled every time when the test set is loaded.

Memory Cells. We evaluate both SRAM and RRAM with different technology nodes for test-time adaptation. RRAM is chosen as the representative NVM with its Complementary Metal-Oxide-Semiconductor (CMOS) compatibility, small cell area, low read latency and energy, and more. In addition to the difference in the memory technology, SRAM and RRAM necessitate different peripherals in their respective systems. For instance, CIM with RRAM requires the array to operate in the mixed signal realm whereas SRAM-based CIM indicates a pure digital configuration. At the circuit level, a level shifter for boosting the programming of RRAM and a calibrated sensing scheme [60] will be taken into consideration for the RRAM-based design. In addition to analyzing fully SRAM-based and RRAM-based adaptation systems, we explore the hybrid memory design paradigm by leveraging the unique characteristics of SRAM and RRAM. The adaptable layers are mapped onto SRAM due to its better write performance and endurance while the frozen layers are mapped onto RRAM due to its superior read performance and density.

Device Non-ideality. Traditional storage devices, such as Dynamic Random Access Memory (DRAM) and SRAM, rely on charge-based mechanisms. Additional peripherals and mechanisms are implemented to ensure the maintenance and correct operation of the bits within these cells. In the case of SRAM, a group of transistors competes to maintain a stable state, whereas DRAM requires periodic refreshing to counteract charge leakage. These storage devices typically incorporate a larger noise margin to ensure the accuracy of the stored bit patterns. In contrast, resistive NVM cells store information using their resistance state. For instance, PCM exhibits varying cell resistance, thereby representing 0 or 1 states with different crystalline structures [23, 41]. RRAM forms or disrupts a conductive path to achieve its low-resistance state (LRS) and high-resistance state (HRS) [33]. In the case of RRAM, as a cell undergoes numerous write operations, its physical structure gradually deteriorates, leading to a decrease in the on-off ratio (i.e., high-state resistance divided by low-state resistance). We model this on-off ratio as a linear scaling on the quantization axis, as illustrated in Figure 3. Furthermore, various types of inherent hardware noise, such as device-to-device variation, cycle-to-cycle variation, non-linearity introduced by retention issues, and quantization error introduced by the analog-to-digital interface, can all potentially damage the accuracy and hardware performance and power efficiency of the systems that employ NVM devices. We incorporate these device non-ideality factors into our adaptation benchmarking framework and investigate the effectiveness of test-time adaptation on both environmental data distribution shifts and inherent hardware noise. The non-ideality parameters have been validated against fabricated testchips [17, 46].

Metrics. For DNN supervised and unsupervised adaptation (Section 4), we consider model test-time accuracy as the algorithmic metric for DNN performance, as well as compute latency and energy consumption as the CIM hardware metrics for DNN efficiency. For autonomous navigation system adaptation (Section 5), we consider the autonomous agent’s safe travel distance, compute latency, compute energy, and end-to-end task completion energy as the evaluation metrics.

Design Space Exploration (DSE). To establish a benchmarking framework enabling pre-silicon adaptation performance evaluation on CIM macros for designated workloads, we further conduct DSE analysis (Section 4.7) to identify the optimal algorithm-hardware configurations given the accuracy, latency, and power constraints of the deployment system.

RQ1: Evaluating adaptation performance for various DNN models under data distribution shift. In this RQ, we investigate the impact of adaptation on three distinct DNN models. We elucidate that test-time adaptation with CIM effectively elevates task accuracy. Notably, the choice of adaptation strategy influences not just the model’s performance but also the hardware overhead stemming from the adaptation process.

In this work, we use adaptation to enhance the robustness of DNN models. It seems like a natural choice to tune all trainable parameters of the network with distribution-shifted data. Yet, it could be either energy-consuming or achieving non-optimal performance. Previous studies pointed out that adapting the whole network could potentially degrade some learned features in the network [39]. Tuning layers altogether could lead to modules deviating from their pre-trained optimal status. Addressing data distribution shifts while ignoring other noise sources, the optimal adaptation strategy appears to be context specific. For instance, deploying a pre-trained network on edge devices may result in low image quality, where adapting shallow layers could significantly enhance accuracy. This is because the initial layers, responsible for low-level feature extraction, may not be well suited to processing noisy images. Adapting these layers can thus yield substantial improvements. Conversely, shifts in higher-level features may be best addressed by adapting middle layers, which bridge the gap between low-level features and more abstract representations. For example, adapting these layers can help the model recognize new shapes or patterns. Similarly, shifts at the output or label level, such as those exemplified by the CIFAR10-flip dataset, call for adaptations in the final mapping layers to correct mismatches between high-level features and labels.

We first train three DNN models, VGG8, ResNet20, and DenseNet40, on CIFAR-10 while quantizing them with WAGE regulation. We observe that all three models suffer accuracy loss when tested on CIFAR-10-C images. However, after test-time adaptation with CIM, the accuracy improves on CIFAR-10-C images. As adaptation is conducted in the granularity of blocks, we treat every two consecutive layers as one block in VGG8. For ResNet/DenseNet, we treat each basic block/Dense block as one block unit. The relative accuracy gain of test-time adaptation with CIM with regard to no adaptation is shown in Figure 5.

Figure 5 showcases that both full and partial model adaptations enhance accuracy compared with the original model. Intriguingly, adapting select blocks within DNN models often yields better results than full-model adjustments. Speaking in a quantitative manner, partial adaptation surpasses full adaptation in more than 80% of test cases throughput our experiments. Specifically, by fine-tuning a single parameter block while keeping others static, we achieve superior outcomes in addressing distribution shifts. Furthermore, optimal performance is observed when different blocks are tuned to cater to varied types of distribution shifts. For instance, adjusting the first block of VGG8 proves most effective for input-level shifts such as CIFAR-C (image corruption).

However, this observation isn’t universally valid, as evident from the results of ResNet20 and DenseNet40. We speculate that this discrepancy arises due to the quantization effect. To validate our assumption, we examine the mean of weight gradient across layers for the three networks. As depicted in Figure 6(a), deeper models such as ResNet20 and DenseNet40 have layers where the gradient is quantized to zero, meaning that the correction data of specific layers is neglected—a phenomenon absent in the comparatively shallower VGG8. Subsequent experiments revealed that, on average, 12.3% of trainable parameters in ResNet20 and 9.7% in DenseNet40 receive a zero weight gradient, whereas none receives a zero weight gradient in VGG8. Given that backpropagation processes deeper layers first followed by shallower ones, shallow layers accumulate more quantization errors. That is because gradient computation for shallow layers relies on deeper layers’ activation gradient. This leads to a reduced adaptation accuracy boost for shallow blocks compared with their floating-point counterparts. Such inefficiency could be further exacerbated by the minimal learning rate used for adaptation, which makes gradient values even more susceptible to quantization.

RQ2: Adaptation performance under output-level shift for different DNN models. In this RQ, we study the effects of the adaptation CIFAR-Flip dataset on three different DNN models. The CIFAR-Flip dataset can be seen as an output-level shift because the only difference between CIFAR-10 and CIFAR-Flip is the flipped label ( \(x\rightarrow 9-x\) ). Figure 6(b) presents the relative accuracy gain (i.e., block-wise adaptation accuracy — full adaptation accuracy) on the CIFAR-Flip dataset. We observe that fine-tuning partial blocks surpassed the full-tuning method regarding accuracy. Moreover, tuning the last layer works best for output-level shifts, and blocks closer to the loss function in the computation flow also suffer less from quantization error. The gap between last-block adaptation and other block fine-tuning will get smaller with more adaptation epochs. However, a higher number of adaptation epochs also indicates extra compute latency and energy overhead.

RQ3: Adaptation performance under unsupervised learning tasks. A vast number of applications cannot enjoy the benefit of annotation and have to cope with a new working environment blindly, which prompts our experimentation on unsupervised adaptation on the distribution shift. We applied Shannon entropy as the unsupervised loss function [68]. Figure 5 demonstrates the performance of unsupervised adaptation on three DNN models. Interestingly, we found that for input-level shift, unsupervised adapting is effective and it even slightly surpassed supervised adaptation in more than 50% of our test cases. However, in output-level shifts, unsupervised adapting is hardly working. The testing accuracies are all below 20% under the same adaptation strategy. This is conceivable since Shannon entropy only depends on model prediction and possesses no knowledge of label shifting.

From the hardware perspective, unsupervised entropy minimization usually consists of exponential and logarithmic operations, whereas the adapted supervised loss function is the summed square error. The implementation of squaring and summation indicates that fewer resources are needed compared with the logarithmic and exponential unsupervised counterparts.

RQ4: Hardware cost (compute latency and energy) under different adaptation strategies and DNN models. With the adaptation carried out on edge, compute latency and energy also play critical roles, together with task accuracy. Adapting different parts of the network would indicate different latency and energy costs. We conduct a hardware cost estimation corresponding to different adaptation strategies with our adaptation benchmarking framework.

In a single epoch, training — or adapting —can be segmented into three main sections: forward pass, backward pass, and weight update. One batch of training finishes as soon as all weights are updated. When adapting different blocks with different depths, the closer the block is to loss function in the computation flow, the less latency it will take. The flow is pictured in Figure 7. Since shallower layers show true dependency towards activation gradients from deeper layers, they will have to wait for deeper layers to have their activation gradient calculation finished, even if those layers are not meant to be adapted. Considering that adapting different blocks would give different accuracy gains, there is obviously a space for exploration to strike a balance between accuracy, latency, and energy consumption.

We demonstrate such trade-off with a quantized VGG8 network carried on CIM, as shown in Figure 9. The whole network is partitioned into four blocks, each consisting of two consecutive layers. Non-parameterized layers, such as maxpooling layers, are excluded. Given the number of layers as n, we derive the latency of partially adapting the \(i^{th}\) layer as (use \(\tau\) as latency)

If more than one layer is adapted, latency would be picked among the maximal adapt latency from all adapting layers. We provide the temporal graph of a 5-layer network adapting two layers to demonstrate latency definition (Figure 8). Here, we consider mostly dynamic energy since the summation of forward, weight gradient and activation gradient dynamic energy dominates energy consumption (>98%). The adaptation delay and energy consumption are given in Figure 9. Adapting the first block costs almost the same computational time as that of full adaptation, since the shallowest layer is very likely to be the critical path among all layers (if it is to be adapted). However, the bypassed weight gradient and weight update could save a large amount of dynamic energy as the calculation is omitted. In practice, a more energy-efficient system is possible through techniques such as applying power gating to the weight gradient unit while computing deeper layers’ activation gradient.

As the adapting block goes deeper, the latency decreases since less activation gradient is computed. Weight gradient computation is executed with an SRAM-based CIM module in the core. The gradient compute engine consists of multiple subarrays. It is initialized in a way that the subarray cluster can fit in the largest activation gradient across all layers. Such implementation has boosted weight gradient derivation. A large number of layers could be further accelerated through operand replication and parallel execution between subarrays. Consequently, weight gradient computation is hardly the critical path in adaptation. Energy consumption increases with depth, which seems counterintuitive at first glance. A closer look into the energy breakdown provides clarity. Predominantly, the energy consumption stems from forward reading dynamic energy, activation gradient dynamic energy, and weight gradient dynamic energy. The trends in forward and activation dynamic energy consumption mirror those of latency, as their variations align precisely with the computation flow’s fluctuations. Yet, weight gradient dynamic energy relates directly to the number of weight parameters in the adapting layers. Essentially, layers with a larger set of trainable parameters demand more dynamic energy during weight gradient derivation. Figure 9(c) elucidates the correlation between the layer parameter size and weight gradient dynamic energy consumption. This observation underscores how pivotal the adaptation strategy is in determining the platform’s algorithmic efficiency, physical performance, and deployment factors. In scenarios in which accuracy is paramount, one might lean towards adapting shallower layers (subjected to input-level shifts), even if it implies a longer adaptation latency. Conversely, in situations in which minimizing latency is the priority, deeper layers may be chosen for adaptation. This choice, however, could come with the trade-offs of elevated energy consumption and diminished performance enhancements.

We also conduct a similar analysis on ResNet20, as shown in Figure 10. Latency declines when the adapting block goes deeper within the network. Energy consumption, however, is related to both block depth and the number of parameters in the adapting block. We chose the adapting unit of ResNet20 to be one basic block. The block with index 2 has the most weights whereas the last block (last FC layer) has the least. Though adapting the last block may not give the best accuracy recovery, it may still be an appealing option given the latency and energy benefit.

RQ5: The effectiveness of adaptation based on SRAM/NVM hybrid–based CIM systems. This RQ prompts us to extend our adaptation benchmarking framework to encompass SRAM and NVM hybrid–based CIM systems. The aim is to investigate the advantages of amalgamating various memory cells for enhancing system performance. NVMs generally show more compact area and superior reading characteristics while SRAM is better at writing. Additionally, SRAM is able to utilize more advanced tech nodes. NVMs, however, are typically a few generations behind, even if some of them are compatible with CMOS processes. From a dataflow perspective, SRAM-CIM is aligned with sequential readout, as the MAC operations are conducted in a row-wise manner. In contrast, mixed signal CIM, exemplified by RRAM, achieves high throughput by activating multiple word-lines simultaneously. This discrepancy in dataflow leads to the assembly of diversified peripherals within the array to support different workflows, resulting in varied performance estimation.

We first measured the latency and energy consumption of different adaptation schemes with full SRAM (14 nm) or RRAM configuration. The results are shown in Figure 9. An RRAM-based CIM macro is approximately 20.21% faster than an SRAM-based system. In addition, an SRAM-based CIM consumes 22.24% less energy than RRAM on average. Since only a portion of the network is adapted, we changed the adapting layers from an RRAM-based macro to an SRAM-based macro, forming a hybrid memory system. This means that only adapting layers are equipped with SRAM to boost writing. The full adaptation scheme is omitted since that would turn out to be a full SRAM-based system.

The performance of the hybrid system is shown in the histogram of Figure 9. Generally speaking, the hybrid CIM system shows intermediate latency and power consumption between pure RRAM and pure SRAM CIM systems. For example, if we use the hybrid memory system to adapt the block indexed 0, 21.2% of overall energy would be saved compared with a pure RRAM system. The energy saving comes at a price of a 24.6% latency increase (compared with full RRAM). For a design with more latency budget but tighter power constraints, swapping NVM with SRAM could be a potential solution. Moreover, as a charge-based system, SRAM is free from cell non-idealities of NVM cells. Concerns such as cells wearing out will never bother system performance. This could mean potentially higher accuracy gain in the long run.

From Figure 9, we can see that, when adapting the first or second block, the compute latency is even slightly better than that of a full RRAM system (0.96 s \(\rightarrow\) 0.95 s for 1st block, 0.81 s \(\rightarrow\) 0.78 s for 2nd block). This points to the fact that, even if latency is the only concerned parameter, there are still potential optimal points in the design space after introducing cell types as a new dimension.

One thing we noticed during our experiments is that, though we made assumptions for hybrid systems based on the writing properties of NVM/SRAM cells, weight update is not a dominant part concerning either latency or power consumption. This could be attributed to the workload picked as adaptation in which the learning rate is typically much smaller than training from scratch. A smaller learning rate results in smaller parameter gradients. With quantization, updates become even less critical since zero-quantized gradients imply omitted writing, as certain cells remain entirely untouched. In the core, the weight update cost is estimated by counting the amount of writing pulses applied to the subarrays, which is a number proportional to the amplitude of the weight gradient. As fine-tuning can imply a smaller number of write pulses compared with training, weight update is observed not to contribute the majority of adaptation overhead.

Another distinct advantage of NVMs is their compactness. In certain cases, an NVM cell might occupy an area smaller than \(10F^{2}\) , whereas a traditional SRAM cell can span nearly \(150F^{2}\) . Thus, a hybrid system could either opt for more NVM layers for compactness or SRAM for energy efficiency. We conducted area estimation upon the proposed hybrid system, shown in Figure 11. To summarize, when swapping the RRAM 1T1R crossbar array with the SRAM array to facilitate partial adaptation, the area overhead will be proportional to the number of tiles captured by the adapted block. Since block size expands as we proceed through shallow blocks to deeper blocks, more SRAM tiles are required to carry weights, which leads to higher area overhead for the hybrid system.

RQ6: Adaptation on both environmental data shift and inherent hardware noise. In this RQ, we explore whether our proposed test-time DNN adaptation at edge with CIM is able to adapt the data distribution shift and inherent NVM device noise at the same time. Non-ideality associated with NVM devices, such as unpredictable cell-to-cell variation, IR-drop, and ON/OFF ratio degradation, have been a persistent issue for decades [10] and made models that are deployed on edge NVM-based systems suffer from algorithmic performance degradation.

With the proposed framework, for the first time, we find that such degradation caused by device imperfection could be amended by the test-time DNN adaptation along with the data distribution shift. As demonstrated in Figure 12, we evaluate the pre-trained models on the CIFAR-10-C dataset, and the endurance is simulated as a reduced on-off ratio, i.e., the ratio between the HRS and LRS [70]. We observe that with the on-off ratio being 80, the inference accuracy dropped roughly 3% for VGG8. When the on-off ratio further drops to 30, inference accuracy quickly goes down below 50%. Deeper networks appear to be more vulnerable to the damage of the on-off ratio as their inference accuracy drops severely to either of the on-off ratio values. In all cases, by applying 3 epochs of adaptation, we are able to adapt the network to a condition with better accuracy. Adaptation works for layers regardless of their depth. This shows that adaptation is not only capable of handling data corruption but also has model decay from non-ideality covered as well. Partial adaptation on VGG8, ResNet20, and DenseNet40 also consolidates this observation. Moreover, we observe that partial adaptation results in higher accuracy than full adaptation. While training a hardware-aware model might seem advantageous, certain noise sources cannot be realistically accounted for during the training stage. However, our findings reveal that adapting a model aware of the on-off ratio on the CIFAR-10-C dataset results in an accuracy improvement of no more than 2% compared with a model unaware of the on-off ratio. This observation underscores the efficacy of test-time adaptation in simultaneously addressing both hardware and input noises. It suggests a potential inclination towards direct edge-based model adaptation, circumventing the need for additional training and pre-deployment profiling of CIM non-idealities. Considering that endurance could mean a reduced on-off ratio, adaptation may as well be a potential solution for defending against edge network conductance shift.

We conducted a set of experiments to show the discrepancy between different CIM macros carrying out different adaptation strategies. The result is demonstrated in Figure 13. We evaluate macros such as RRAM, SRAM based on 22-nm and 14-nm technologies, and a hybrid CIM system combining RRAM and 14-nm SRAM. These systems were analyzed during the adaptation of the VGG8 network on the CIFAR-10-C dataset over three epochs.

For the desired accuracy, we establish a threshold of at least 75%. Consequently, all adaptation schemes related to the 3rd block were eliminated. We further limit the timing constraint to under 2.85 s and set the energy consumption cap at 850 mJ. This filtering process narrowed down our options, leaving only a few viable candidates concentrated in the bottom-left quadrant of the figure.

As previously discussed in this section, the deeper the adapting block, the more likely the trend is towards reduced latency, increased energy consumption, and diminished accuracy gains. Hence, systems primarily concerned with latency tend to favor deeper adapting layers. Conversely, those with energy efficiency as a priority typically lean towards adapting more superficial layers. For tasks that weigh both latency and energy as significant factors, a few configurations emerge as suitable. These include the RRAM adapting the 1st and 2nd blocks, the 14nm SRAM adapting the 3rd block, and the hybrid system adapting the 1st and 2nd blocks. These configurations are projected to be proficient for the task at hand.

Autonomous systems are becoming prevalent for Position-Navigation-Timing (PNT) applications [44]. To facilitate fully autonomous PNT, state-of-the-art UAVs are expected to run complex reinforcement learning (RL) models onboard with little-to-no offloading computation support [37, 66]. These UAVs often face constraints related to size, weight, and power (SWaP), making it crucial to balance energy efficiency with robustness and safety. A common strategy involves offline training of autonomy models in simulated environments followed by deployment for inference on edge devices. However, this can lead to performance issues due to differences between simulated training environments and actual deployment scenarios, necessitating on-device adaptations to enhance real-world performance within strict resource and latency limits.

Our approach champions a two-phase autonomous system learning paradigm that combines offline and online learning using fine-tuning adaptation. The key methodology is that if we train a model for an autonomous navigation task in a variety of environments collectively, we can leverage this knowledge using adaptation while training a smaller part of model for similar applications in a similar (but different/unseen) test environment.

In the offline phase, we train one autonomy network on a set of meta-environments using deep reinforcement learning [3]. These meta-environments serve as a library of environments for the underlying autonomous navigation problems. This offline training phase is carried out on high-end workstations where we assume no strict restriction on the compute engine. Once we have effectively trained a network on the meta-environments collectively, we use these meta-weights as initialization during the online adaptation phase. In the online adaptation phase, a different test environment is used for adaptation (fine-tuning). The adaptation computations need to be carried out in the edge nodes with CIM architectures. In this phase, the adaptation is carried out on only a part of the network. The network is divided into non-adaptable and adaptable parts and only the weights of the adaptable parts are updated. The segmentation of the network is a compromise between the performance (obstacle avoidance) and the number of training computations.

Experimental Platform. We employ PEDRA [3], an open-source UAV simulation infrastructure, to test our adaptation methodology in autonomous systems. Powered by Unreal Engine, PEDRA creates realistic 3D environments. It uses AirSim to simulate the UAV’s dynamics and kinematics, and incorporates learning-based algorithms that generate real-time flight commands.

Task and Policy Model. In our experiments, the UAV starts at a predetermined point and is required to navigate through the hallways. There is no goal position and the UAV is required to fly avoiding the obstacles as long as it can [64]. The UAV captures RGB monocular images from its front-facing camera to determine its actions, which are processed through a reinforcement learning algorithm. Our model employs a perception-based probabilistic action space consisting of 25 possible actions [65]. A depth-based reward system encourages the UAV to avoid obstacles. The neural network, designed to predict action probabilities from sensory inputs, consists of three convolutional layers and two FC layers, shown in Figure 14. This network was trained across four diverse meta-environments: indoor-long, indoor-techno, outdoor-town, and outdoor-forest (Figure 15). During deployment, we assess the policy’s effectiveness by measuring the Mean Safe Flight (MSF), which is the average distance the UAV travels before encountering an obstacle, with longer distances indicating better performance.

Adaptation Performance. Adapting partial layers of the network improves the performance of UAV autonomous navigation tasks. As shown in Figure 16(a), after adaptation, the MSF achieved by the UAV surpasses the case achieved by the network initialized with meta-weights. Adapting Layer 1 achieves the highest MSF and it is observed that the performance of adapting partial layers surpasses the full fine-tuning.

Different adaptation strategies also present computational latency and energy trade-offs. As illustrated in Figure 16(b) and Figure 16(c), adapting Layer 2 to Layer 5 exhibits lower compute latency and energy compared with full model fine-tuning. Such computational efficiency directly influences the UAV’s safe flight speed and, by extension, its overall performance. Given a specific UAV speed, and accounting for the interval between successive frames and the UAV’s obstacle distance threshold, we can deduce the minimum Frames per Second (FPS) requisite for collision prevention. A swifter UAV mandates processing more frames within a fixed time span (demanding higher FPS). This pace is feasible only if the core computational system can handle the required FPS. Consequently, the system’s latency restricts the UAV’s maximum safe flight speed [36, 38]. Hence, the latency advancements from partial adaptation versus full fine-tuning in Figure 16 directly corresponds to an improvement of a maximum supported theoretical speed of about 1.73 \(\times\) from full fine-tuning to partial adaptation. For the same autonomous navigation task, the increased flight velocity results in faster task completion time (i.e., lower flight time), therefore consuming lower end-to-end flight energy since the majority of energy is consumed in rotors [11].

For pipelining to be effectively implemented, certain prerequisites must be met. Firstly, the process should be divisible into independent modules, akin to how neural networks are organized into layers. Secondly, operations on different inputs must remain independent to ensure accurate outcomes and prevent operational hazards. This setup aligns well with inference phases, in which network parameters are fixed following training, meeting the criteria for pipelining.

However, the situation shifts when it comes to training or tuning DNNs. Full parameter fine-tuning challenges the feasibility of pipelining, as initiating the next batch’s forward inference is contingent on completing the current batch’s weight updates, creating a read-after-write hazard. The possibility of pipelining during DNN tuning hinges on the specific tuning strategy employed. For example, tuning the initial block of a three-layer network closely resembles a full-tuning scenario, where pipelining is hindered due to direct data dependencies. Assuming that both forward and backward passes of a layer require a set time to complete, such processes, illustrated in Figure 17, show constraints indicated by arrows. If one forward and either a complete or partial backward propagation is considered a single instruction, with ‘time step’ equated to ‘cycle,’ the cycle-per-instruction (CPI) would be 6.

Adapting deeper layers presents a different scenario, suggesting that the initial layers remain unchanged during the adaptation. In a case in which only the last layer is tuned, the inference for the first two layers of the subsequent batch can commence before updating the third layer. This approach effectively masks the latency of the initial layers behind the latency of the previous batch’s forward-to-backward process.

With a sufficiently large batch size, the effective CPI could approximate 2. As tuning focuses on progressively shallower layers, the effective CPI trends towards 2L, with L representing the count of layers in reverse order from the last to the shallowest layer being tuned.

In summary, pipelining during partial network adaptations holds promise for enhancing efficiency. Nevertheless, the extent of throughput improvement achievable through pipelining is intricately linked to the selected tuning strategy, especially in managing the memory overhead for storing activations. Visualizations for three scenarios—highlighting whether the adapted layers are at the beginning or end of a three-layer network—demonstrate that pipelining is most beneficial in the latter case.

We anticipate that future research can expand upon our work in four key dimensions, as follows.