Energy-Efficient Approximate Edge Inference Systems

This computation-intensive design caters to the domain of edge intelligence. Here, the local edge device runs the end-to-end DNN inference application, from data (image) acquisition to final result generation. Sensor, memory, and computation are active subsystems as indicated in Figure 4. The sensor subsystem is responsible for capturing the image. The compute subsystem (processor) applies the following preprocessing steps: (i) aspect-preserving resize and central-crop the image (to fit the DNN input dimension specification), (ii) convert the image to tensor (FP32 format) and scale the image’s pixel intensity values in the range \([0., 1.]\) , and (iii) normalize the resulting tensor. Note that DNN models used for classification (InceptionV3, ResNet101) and detection (YOLOv5), among others, could use test-time augmentation techniques, namely multi-resolution, multi-crop, flips, and rotation, among others, and merge the augmented predictions to obtain higher accuracy. However, we do not consider them in light of inference time and higher energy consumption. Following transformations, the on-device processor executes the DNN inference operation to generate the final object class, in case of image classification, and the bounding box coordinates with class labels for each detected object, in case of object detection. The memory subsystem is active throughout the application, as it stores the acquired image, the application program, and DNN weights and feature maps during inference. The communication module (if present) is mostly inactive and may be used to communicate the final results to the cloud or other devices, thereby consuming a negligible amount of energy. Clearly, the first three subsystems contribute to the overall energy consumption of the system. Therefore, synergistic approximations to the same are applied to achieve better energy efficiency.

This communication-intensive design caters to the domain of cloud intelligence. As shown in Figure 4, all four subsystems are active in this design. The edge device captures the image using the sensor subsystem and transmits it to a resource-rich cloud server through the communication subsystem. Here, the cloud server applies all necessary transformations to the input image, as mentioned earlier, and then runs DNN inference before sending the relevant application result to the edge device. Large-scale cloud servers in data centers typically employ clusters of machines with GPUs or Google Tensor Processing Units (TPUs) [37]. The memory subsystem is again active throughout as it participates in data acquisition and transmission. Consequently, synergistic approximations are made in the sensor, memory, and communication subsystems. In particular, although the DNN workload is not executed on the device processor, the compute module works simultaneously with the communication module to transmit the image and retrieve the result. Therefore, all four subsystems have significant contributions to the overall energy consumption of the system, with approximations employed in all except the compute subsystem.

Cameras or image sensors embedded in modern edge devices record very high resolution images to satisfy the growing consumer demand for high-quality photography. On the contrary, most computer vision applications work with images with a relatively lower resolution. For example, DNN architectures such as ResNet101, VGG19_BN, and MobileNetV2 can accept \(224\times 224\) resolution images, whereas some recent models (EfficientNet) can accept multiple image resolutions up to \(600\times 600\) . Although they have the ability to process images with arbitrary resolutions, detection networks, such as Faster_RCNN, EfficientDet, and YOLOv5, are generally trained on images with a resolution in the range of \([640, 1536]\) . Therefore, sensors with more than 10 million pixels are too expensive for these applications. Furthermore, the number of pixels sampled by the sensor is (approximately) proportional to the energy consumption of this subsystem. In fact, capturing a high-resolution image not only aggravates the sensor energy but also directly impacts the computation required by the DNN. Driven by these insights and the inherent error resilience of DNNs, we captured images with a resolution even lower than the specified input size for these networks, without changing the receptive field of the sensor. To do this, we investigated two distinct approximation strategies that are widely available in mobile image sensors today. For image classification, we applied pixel subsampling or nearest neighbor subsampling, which skips the readout of entire pixel rows/columns, thus reducing the resolution of the DNN input data. As an illustration, Figure 8.1 shows that subsampling skips multiple pixels to reduce the image resolution from 1080 p ( \(1920\times 1080\) ) to 720/480 p. However, we adopted binning or bilinear subsampling, which takes a weighted average of four neighborhood pixels to reduce image resolution and image noise for object detection. The design choice of the sampling strategy is based on our observation of empirical data. Pixel subsampling results in superior quality in the classification task, whereas bilinear sampling gives better results for the detection task. Therefore, we adopted subsampling factor ( \(s_r\) ) as the sensor approximation knob (f) for both of these strategies to reduce image size by \(s_r^2\) and obtain appreciable energy savings in the sensor subsystem with minimal effect on DNN accuracy. Additional benefits arising from this approximation include reduced sensor active time and data storage needs. Note that subsampling is performed in the sensor subsystem itself, as the camera directly tunes its own sampling resolution and does not involve the processor or compute subsystem. Internally, we might need to upsample the acquired image due to input size restrictions of some classification DNNs; however, this operation incurs minimal overhead to the edge processor. In contrast, the upsampling operation is unnecessary for DNNs that support model scaling and detection DNNs. In fact, we eliminated the necessary downsampling steps followed in traditional DNN inference pipelines, thus reducing the computation workload. Interestingly, the reduction in the XY resolution could also speed up subsequent DNN processing and therefore might lead to system-level energy savings. Using these insights, we modulate f to generate images of multiple resolutions and demonstrate the Q-E tradeoffs for multiple classification and detection benchmarks in Figure 9. The x-axis in both of these graphs represents the square of the subsampling factor (i.e., the image size reduction \(s_r^2\) ), the left y-axis represents the normalized quality, and the right y-axis represents the normalized sensor subsystem energy. Note that the x-axis range varies among these two graphs due to the difference in the image size reduction between classification and detection DNNs. This set of graphs uses top-1 accuracy as the quality metric for classification and Mean Average Precision (mAP) for detection. We derive two insights from these approximation studies: (i) heavy-weight (large) DNNs are comparatively more resilient to sensor approximations than light-weight (small) DNNs, and (ii) normalized quality degradation is much lower in detection DNNs compared to classification DNNs, whereas the amount of energy savings is quite similar.

DRAM serves as the main memory unit ubiquitously in commercially available smart cameras and modern embedded systems. Following along the lines of the prior art [72], we use approximate DRAM due to its high density, large capacity, and better energy efficiency. Section 4.4 clearly suggests that refresh energy consumption should be considered as one of the most critical parameters in the design of the computing system. Thus, the approximations in DRAM are introduced by increasing the background refresh interval beyond the nominal 64 ms, and this interval acts as the memory approximation knob f. Lowering the refresh rate or increasing the interval drastically reduces the overall DRAM energy consumption but introduces retention bit errors in DRAM pages, resulting in accurate and erroneous pages. Since DRAM is responsible for storing the image and different DRAM parameters, including weights and feature maps, page errors ultimately lead to bit errors in these data. For example, Figure 8.2 illustrates how these bit errors result in a noisy image. Note that at higher refresh intervals, the number of erroneous DRAM pages increases, further degrading the image. Adopting the DRAM classification strategy of Raha et al. [73], we split the physical DRAM pages into different quality bins { \({qbinI~|~I~\in ~0~to~3+}\) } based on the error characteristics of each page and illustrate the same in Figure 10. It is important to specify that quality bins with a higher index (i.e., represented by a higher value of I) indicate a higher bit error rate, and as the refresh interval is increased, the distribution is skewed toward qbinI pages with higher values of I, as shown in Figure 10. In the context of DNN inference, we further categorize DNN data into critical and noncritical data for fine granular control over approximations [73], and assign them to accurate (qbin0) and erroneous pages (the remaining qbin0 and \(qbin1+\) ), respectively. The entire DNN-based application program and IFMs (OFMs) of the DNN are considered critical data, whereas the input image and pretrained DNN weights are considered noncritical data. IFMs are considered critical due to their lower bit error rate tolerance compared to DNN weights, as shown in previous works [41, 73]. To obtain the maximum energy savings, we select the maximum refresh interval that will allow us to allocate all critical data in qbin0, thus ensuring minimal impact on quality.

Figure 10 shows the amount of critical and noncritical data for a set of popular DNNs used for classification and detection. Note that the y-axis is on a logarithmic scale. As we can see, the portion of noncritical data (weights) is mostly larger than critical data (IFMs). The average (geomean) of the maximum IFMs across all individual layers in these detection networks is 154 MB compared to the average of total weights (370 MB). In classification DNNs, the ratio is more skewed in favor of weights (62 MB) compared to the maximum IFMs (5 MB). These data clearly support our allocation strategy, since critical data always fit in qbin0, with noncritical data mostly allocated to \(qbin0+\) , thus ensuring minimal quality loss and yielding substantial energy savings, at different approximation levels (refresh intervals). Note that approximations only affect DNN weights and feature maps in Cam_Edge when DNN inference is executed on the edge processor. In comparison, only the input image is affected by DRAM approximations in Cam_Cloud. Our design decisions and the approximation strategy are validated by the graphs in Figure 11. We clearly see that increasing the refresh interval from 64 ms to 1 second results in a nearly 0% drop in quality while reducing the refresh power by a significant amount ( \(68\%\) ). Looking at the four plots together also stimulates interesting observations, such as (i) Cam_Cloud is more resilient than Cam_Edge, which follows our expected behavior; (ii) at high intervals (30 seconds), large DNNs tolerate more errors than small DNNs in classification; (iii) in Cam_Cloud, detection networks suffer more quality loss compared to classification networks at high intervals; and (iv) high intervals give diminishing returns in terms of power reduction. These results demonstrate the DNN’s remarkable robustness to errors in inputs and weights.

We developed a generic quality-driven model compression framework that takes advantage of modern DNN approximation techniques to approximate the compute subsystem of a typical edge inference system running the AI workload. By systematically controlling the quality of the application, namely top-1 accuracy in classification and mAP in detection/segmentation, this framework leverages the inherent error tolerance of DNNs to achieve energy-efficient inference—in other words, trades off quality for reduced energy consumption of DNNs. Essentially, the framework achieves this by tuning different compute approximation knobs. Three distinct functionalities are incorporated to approximate different types of DNN models. We discuss each of these functionalities in this section.

Quality-Driven Structured Pruning. Among the multitude of algorithmic/hardware/software approximate computing techniques mentioned in Section 2.1, we adopted a software approximation flavor for this work due to its applicability to all types of commercially available computing platforms. Specifically, we selected structured pruning and thinning to facilitate energy-efficient inference on resource-constrained edge devices due to their simplicity, as shown in Figure 8.3A. This generic approximation methodology allows the deployment of optimized DNNs on COTS devices without the need for specialized sparse convolution libraries/hardware as opposed to unstructured (elementwise) pruning, which cannot leverage network sparsity into real inference speedup unless implemented in specialized hardware accelerators. Reduction in the size of the DNN model, the number of compute operations (FLOPs/MACs), and the dynamic memory footprint and memory accesses are some of the significant benefits that stem from this technique. Ultimately, these benefits lead to faster inference and less energy consumption without sacrificing a significant amount of accuracy. These factors motivated us to introduce a novel and generic pruning framework, QUSP, to approximate the DNN inference operation. First, we give a high-level overview of QUSP, as illustrated in Figure 12. The system designer provides pretrained DNN model \(M_0\) , a set of end user defined target application quality loss bounds \(Q_b\) (e.g., \(\lbrace 0.5, 1.0, 2.5, \ldots \rbrace\) ), a DNN-specific application dataset, and a saliency metric for ranking structures (filters or channels) in a DNN. Using these inputs, QUSP generates a gradual/iterative quality-driven pruning schedule and subsequently runs the structured pruning and thinning algorithm for compression of the DNN model. The unique feature of QUSP that makes it different is that, given a set of quality metrics, this framework generates a family of compressed DNN models \(\mathcal {MF} = \lbrace M_i\rbrace\) , each meeting a distinct quality bound \(q_i \in Q_b\) . Essentially, pruning introduces sparsity in entire filters depending on their importance, and thinning permanently eliminates these sparse filters and dependent OFMs, thereby reducing computational complexity (FLOPs) and decreasing the energy consumption of the compute subsystem. As one can observe in the figure, for a higher-quality loss bound, QUSP prunes a higher number of filters and feature maps, thus showing a positive correlation between the approximation degree and the quality loss bounds. Put another way, as the user relaxes the quality demand even more, this allows the system designer to apply a higher degree of approximations that could result in better energy efficiency in the edge inference system. Since most pruning algorithms involve training and/or fine-tuning models, QUSP is run offline on a resourceful cloud server, as the computing capacity of edge systems usually exceeds pruning requirements. The key novelty lies in generating multiple models with an accuracy-FLOPs tradeoff that can be directly executed on COTS edge devices in an energy-efficient way, without the need for any hardware modifications. Note that structured pruning is one of the straightforward ways to approximate any DNN, and QUSP can be easily extended to any other compression algorithm and using any saliency metric to generate Q-E tradeoffs for the compute subsystem.

Let us dive into the inner workings of the QUSP framework to understand how we perform pruning in a systematic manner. The overall quality-driven pruning and thinning scheme is described in Algorithm 2. Among the different pruning granularities found in the literature, we adopted filter (channel) pruning of the CONV layers in a DNN following popular SOTA compression schemes [27, 47], where the objective is to remove entire filters in multiple layers of DNN based on their significance. As DNN models are over-parameterized as discussed in Section 3.1, it is highly likely that there exist some redundant filters in multiple layers. A saliency metric answers a fundamental question in this context. The metric decides which filters in a layer or across the network are redundant so that their removal will result in the least accuracy degradation. Among the multitude of such metrics proposed by researchers [66], QUSP has the ability to choose one or more metrics for filter classification, such as layerwise \(\ell _2\) norm of filters, \(\ell _1\) norm of activations, and average of gradient. Using the chosen metric, QUSP runs Sensitivity Analysis (SA) [45] on the pretrained model \(M_0\) to measure the sensitivity of all layers to pruning and generate a sparsity table \(\Gamma\) (line 3). Dense or fully connected layers are not evaluated in this analysis, as CONV layers contribute to the most FLOPs in a DNN. The pruning schedule is another critical component of any pruning algorithm. This determines the number of iterations to run (duration), the pruning criteria, the number of filters to prune per iteration, and the frequency of pruning. Now, existing pruning schemes [47] set sparsity levels based on SA results or continue pruning until a specified compute budget is achieved. However, these stand-alone techniques are insufficient for our work, as we use multiple quality loss bounds as the driving constraint. Following the lines of previous work, AGP [111], we designed a new quality-driven iterative pruning scheduler for QUSP that automatically calculates the pruning duration d, and the initial sparsity value \(s_s\) and the final sparsity value \(s_f\) for each layer l and for each quality bound \(q_i \in Q_b\) . These sparsity values are generated using a quality-driven sparsity selector, as described in Algorithm 1. This process uses \(\Gamma\) to find the maximum tolerable sparsity of each DNN layer per \(q_i\) . The thresholds used in this algorithm were empirically determined.

At each iteration, QUSP evaluates the importance of the filters based on the selected saliency metric and prunes K less significant filters based on the sparsity percentage inferred from the schedule \(s_t\) (line 11 –19). After d iterations, we clone the current pruned model to \(M_1\) for the first quality bound \(q_1\) (line 24) and continue pruning until B models are created, one for each \(q_i\) . It is important to note that pruning alone cannot offer any inference speedup or computation reduction, unless there is hardware support for sparse inference. This iterative process is followed by network thinning that considers data dependencies across layers and then physically removes the pruned filters from the CONV layers, along with bias and the relevant coefficients of the batch normalization layers following that CONV layer. An added advantage of thinning is that the OFMs in the immediate CONV layer are also pruned. On top of that, the weight filters corresponding to these pruned OFMs are also removed. These cumulative efforts result in a substantial reduction in the number of FLOPs and weights, which speeds up the inference and reduces the energy of the computation subsystem. To counteract the loss of accuracy induced by thinning, we must fine-tune each model \(M_i\) so that its inference accuracy satisfies the corresponding quality bound \(q_i\) (lines 30 –33). This process continues until QUSP generates a family of compressed DNN models \(\mathcal {MF}\) for all target quality specifications with different FLOPs and accuracy. Figure 8.3A gives an illustration of a network approximated using QUSP. Here, two weight filters and two OFMs are pruned in layer \(L_i\) , and in \(L_{i+1}\) the weight filters corresponding to these OFMs are also removed. Similarly, Figure 12 shows increasingly pruned filters, IFMs, and OFMs of a single CONV layer in three compressed models generated by QUSP.

Model Selection and Backbone Switching. The second functionality that we incorporated into our model compression framework is a simple model selection criterion. Given an architecture family consisting of a series of pretrained DNNs with varying quality and computational cost, the framework finds the most computationally efficient model subject to a predefined quality constraint. A few examples of families are ResNet [26], MobileNet [32], and EfficientNet [87] in classification networks, and EfficientDet [88] and YOLOv5 [36] in detection networks. Let \(\mathcal {M}\) represent the set of models available in any such family. Essentially, the framework takes a set of user-defined quality loss bounds \(Q_b\) of size B and finds B number of models \(\mathcal {MF} = \lbrace M_j \in \mathcal {M}\rbrace\) with a minimum number of FLOPs, each satisfying the respective quality bound by solving the following problem:Another prominent feature of our compression framework is backbone switching. Generally, detection DNNs use a backbone network to extract features from the input image followed by a neck (feature pyramid network or region proposal network) and a head (prediction network) and generate bounding boxes, class scores, and labels. The framework intelligently incorporates different backbone networks with varying complexity in SOTA detection DNNs and subsequently fine-tunes them to generate B number of models \(\mathcal {MF}\) , one for each quality specification.

Now, we present a set of figures that supports our decision making behind all the compute approximation techniques that we proposed in our model compression framework. Figure 13(a) shows the top-1 accuracy vs. FLOPs in multiple compressed versions of small and large DNNs generated by QUSP, which are used for classification. We can clearly observe that each family comprises models that gradually trade off accuracy in favor of a reduction in the number of FLOPs and parameters. This plot clearly validates our motivation for QUSP. As can be seen in Figure 13(b), our compression framework is also able to apply the model selection criterion and the backbone switching technique to find a family of detection models with tradeoffs in mAP-FLOPs and mAP-parameters. Figure 14 depicts graphs for the application quality with the percentage reduction in FLOPs compared to the baseline model for both applications. We can again draw the same conclusions from these plots. Another significant positive correlation was found between the size of the baseline model and the approximation potential. Large models were able to tolerate more pruning-induced approximations with less impact on accuracy than smaller models, which proves that smaller models such as MobileNetV2 and SqueezeNet1.1 are comparatively less over-parameterized. We also wanted to understand whether the reduction in FLOPs translates into actual hardware speedup. To do this, we executed all versions of the models from generated families \(\mathcal {MF}\) on our prototype benchmark (discussed in Section 5) and measured the inference latency using a single image (i.e., batch size of 1). As observed in Figure 15, we obtained a significant speedup for all DNNs across different quality specifications. Overall, these results clearly indicate that all of our compression techniques, including QUSP, are quite effective in approximating the compute subsystem, which ultimately leads to less energy consumption of the compute subsystem. Note that our approach is orthogonal and complementary to many existing DNN compression works. We use the generated library of compressed models along with concurrent approximations in other subsystems, and we evaluate system quality and energy benefits, as well as explore inter-subsystem interactions for multiple compressed versions (discussed later in Section 4.6).

In the past decade, smart camera systems have become an integral part of private homes, office buildings, smart cities, autonomous vehicles, traffic management systems, smart agriculture, and so on, and have found applications in home security, video surveillance, biometric recognition, real-time viewing, GPS and geofencing, smart parking system, and suicide prevention, among many others. Since these cameras process large volumes of data, lossy compression is commonly applied to acquired images prior to offloading and/or processing. We are interested in compression, as this can potentially reduce data traffic and save communication energy. The communication subsystem (WiFi module) in these cameras is used to offload the acquired image to a cloud server, which runs the DNN inference and sends the result back to the device. As discussed in Section 2.4, DNNs have been shown to be resistant to compression-induced distortions in the input image [15]. Without loss of generality, we adopt this JPEG compression scheme to logically approximate the communication subsystem. Note that lossy compression is a very effective approximation technique, as it can drastically lower the total amount of data to be transmitted while incurring a very small amount of error in the transmitted data. The compression quality parameter { \({Q \in 100, 95, \ldots , 10}\) } that controls the amount of JPEG compression is used as a very effective approximation knob. Here, a smaller Q results in a smaller image size with lower image quality and consequently higher energy savings, thus representing a higher degree of approximation. We vary Q to obtain images of gradually decreasing size, as shown in Figure 8.3B. To assess the robustness of popular DNN benchmarks to the input image perturbed by these distortions, we run cloud-based DNN inference on these compressed images and measure the resulting energy savings in communication subsystem. Figure 16(a) and 16(b) show the normalized Q-E tradeoff for classification and detection, respectively. Although the quality degradation was not significant, the most striking result emerging from these graphs is that there was a measurable influence of compression throughout the range of quality settings. These results also revealed that, unlike classification, detection networks suffer a marked decrease in application quality below \(Q=30\) , which is in line with those obtained by Gandor and Nalepa [21]. Interestingly, setting \(Q=80\) results in a 93% to 95% reduction in communication energy, a fact that has been overlooked in most articles related to DNN image compression, as the authors do not consider the energy aspect. Together, these motivating results suggest the range of practical compression levels (used in Section 4.7) that reduces the communication energy of the wireless module with minimal impact on the quality of the application.

We first examine the interactions among different subsystems in Cam_Edge. As mentioned in Section 2.1, modern DNN models support different strategies of network scaling, including resolution and compound scaling. For two such networks, EfficientNet in the high-compute regime and EfficientNet_Lite in the mobile/edge-compute regime, the family of approximated models \(\mathcal {MF}\) consists of models scaled in both depth d and width w dimensions. In other words, the network architecture varies among these models due to the difference in the number of layers and the parameters of the CONV layers. This ultimately leads to a lower number of MACs (1 MAC = 1 FLOP) and activations than the most computationally complex network in \(\mathcal {MF}\) , which we consider as the baseline model \(M_0\) . For the rest of the article, we follow this convention to select the baseline model for all DNNs, which is shown later in Section 5. As verified earlier in Section 4.5.3, these lower-complexity models demonstrate speedup of inference in real hardware. We represent the reduction in the number of MACs due to this compute approximation by scaling factor \(s_c\) . In comparison, the sensor approximation that reduces the input image size is essentially reminiscent of resolution scaling. Intuitively, subsampling by a factor \(s_r\) reduces MACs by \(s_r^2\) . This generates an excellent accuracy-MAC tradeoff, which ultimately results in a Q-E tradeoff. We consider the maximum input image size corresponding to \(M_0\) as the baseline image resolution. Since approximations in both the sensor and the compute subsystem modify the MACs, their interactions are quite interesting to study. Figure 17 shows 3D plots to represent normalized MACs (on the z-axis) vs. image size and compressed model version \(M_i \in \mathcal {MF}\) (on the x and y axes). Note that MACs are normalized w.r.t. MACs ( \(M_0\) ) to allow comparison, and the absolute number of MACs for each \(M_i\) is already shown in Figure 13. This kind of surface plot allows us to gauge the impact of compound scaling. Cumulative approximations essentially reduce MACs in proportion to \(s_r^2 \times s_c\) , which is clearly evident from these graphs. Note that among classification networks, this kind of interaction occurs only for models that support resolution scaling.

Figure 18 shows similar graphs for four SOTA DNNs used for object detection and instance segmentation. Unlike early classification DNNs, object detection networks are inherently capable of resolution scaling, as they can accept input images of varying resolutions. As a consequence, the image size decided by the approximated sensor subsystem has a significant impact on the number of MACs in the network, which is clearly observed in these four graphs. However, the compression framework used the backbone switching functionality to generate \(\mathcal {MF}\) for Faster_RCNN and Mask_RCNN with varying numbers of MACs. Furthermore, the family \(\mathcal {MF}\) corresponding to EfficientDet and YOLOv5, which are scaled in dimensions dw, also showed a significant reduction in the number of MACs. Figure 18 also shows the maximum reduction at the intersection points. Together, we derive the following observations from the figures. First, to achieve maximum reduction in MACs and better system-level energy efficiency for a particular quality specification, it is critical to perform sensor and compute approximations in a synergistic way. Second, another markedly visible point is the gradual slope of the surface w.r.t. sensor approximation. This indicates that sensor subsampling is a fine-granular knob to control DNN complexity. Third, a final by-product of these approximation methods is the reduction in memory energy, since the number of memory accesses is reduced and the DRAM now needs to store fewer DNN weights and activations.

Let us turn our attention to the interactions in Cam_Cloud, and specifically to how the sensor approximation affects the energy of the other subsystems for the object detection task. Figure 19 shows the energy breakdown of the four subsystems corresponding to different degrees of sensor approximations (i.e., different input image sizes). For example, we can observe a drop in communication energy from \(67\%\) to \(43\%\) when the image resolution for inference using YOLOv5 drops from 1280 to 1024 in Figure 19(d). As is evident from these graphs, the communication subsystem gets the maximum benefit from the sensor approximation. This decrease is certainly due to the smaller image, as the communication module has to transmit fewer data, thus reducing its active time. The reduction in transmission time also reduces the active time of the processor. Furthermore, if communication approximation is selected, then the processor can compress a smaller image in a shorter time frame. Moreover, memory has to store less data and is active for a shorter duration. These results in lower memory and compute energy, as confirmed by the \(5\%\) energy drop in both subsystems in Figure 19(d). Similar trends can also be observed for the other detection DNNs in Figure 19(a) through (c). Thus, the benefit of sensor approximation percolates into the other subsystems, resulting in a cascade of energy reduction in all subsystems that ultimately translates into overall system energy efficiency. We have also observed similar results for EfficientNet and EfficientNet_Lite in classification DNNs, since they are the only ones (among our benchmarks) that accept images of varying resolutions.

In the previous two sections, we studied the impact of sensor and compute approximations on the operation and energy consumption of different subsystems. However, it is imperative to take into account the degradation of application quality and energy at the system level when multiple subsystems are approximated together. Unlike previous studies that investigated the impact of individual approximations on the performance of DNNs, we dive into inter-subsystem interactions and their corresponding impact on the system. To do this, we examine the Q-E tradeoffs in both Cam_Edge and Cam_Cloud and represent the tradeoffs using a set of six graphs in Figure 20. Here, we consider a single detection inference using the YOLOv5 architecture subject to different pairs of subsystem approximation techniques. The presentation of tradeoffs subject to the concurrent application of approximations in all three subsystems is not possible, as it is quite difficult to visualize a 4D or 5D graph. In these graphs, quality Q is plotted on the z-axis, and the color gradient of the graph represents the system-level energy E, each normalized to the corresponding accurate system metrics. Here, red indicates higher energy consumption, whereas green indicates lower energy consumption. We first define a metric, energy-quality gradient, \(\nabla EQ = \Delta E/ \Delta Q\) . Essentially, this metric will allow us to measure the gain in energy savings \(\Delta E\) at the system level with loss of quality \(\Delta Q\) and will help us evaluate each approximation strategy in the context of the system-level Q-E tradeoff. Tradeoffs in Cam_Edge. First, without applying the sensor approximation, we investigate the interaction between the compute and memory subsystem and their impact on the Q-E tradeoff. Accelerated inference achieved using models generated by the compression framework, \(M_i \in \mathcal {MF}\) , yields multiple benefits. Active DRAM time is reduced, leading to lower DRAM energy. In addition, since the number of weights is reduced, we can allocate a larger portion of these weights to qbin0 (accurate DRAM pages), which causes less quality loss. However, reducing the refresh rate reduces the memory refresh overhead, leading to higher inference acceleration and better energy efficiency. These effects reduce the total system energy that can be clearly observed in Figure 20(a). As is evident, the energy reduction due to approximation in the compute subsystem is much greater than that due to approximation in memory. However, \(\Delta Q\) is quite similar; thus, \(\nabla EQ_{comp} \gt \gt \nabla EQ_{mem}\) . In Figure 20(b), the memory subsystem is left untouched while the sensor and the compute subsystems are approximated. Section 4.6.1 already discusses how these two interact with each other and reduce the number of MACs/FLOPs in a DNN. This graph shows that a smaller number of FLOPs ultimately results in a lower energy consumption in the system. Interestingly, it seems that the sensor approximation has a greater impact on system energy, despite similar \(\Delta Q\) ; thus, \(\nabla EQ_{sensor} \gt \nabla EQ_{comp}\) . This shows that the sensor subsystem provides more energy savings compared to the computation subsystem. Finally, we explore the interaction between the sensor and the memory approximations. Figure 20(c) shows that sensor approximations result in higher energy efficiency compared to memory. One of the striking results that emerges is that DNNs are more resilient to bit errors in input image compared to image distortions introduced by sensor subsampling. This is indicated by the relatively lower \(\Delta Q\) for memory; thus, \(\nabla EQ_{sensor} \gt \gt \nabla EQ_{mem}\) . Looking at the graphs in Figure 20(b) and (c) together suggests that \(\Delta Q\) due to the approximations in the sensor is aggravated by that in compute compared to memory. Comparing Figure 20(a) and (b) shows that \(\Delta Q\) induced by the compute approximation is affected more by memory than by the sensor. This observation is related to the fact that DNN weights are affected by bit errors in DRAM, since they are assigned to erroneous pages (Section 4.5.2). This is also clear if we compare \(\Delta Q\) due to the memory approximations in Figure 20(a) and (c). Compared to the sensor, the approximation of the compute subsystem affects the memory more.

Tradeoffs in Cam_Cloud. Similarly to Cam_Edge, Figure 20(d) through (f) depicts the interactions in Cam_Cloud. As observed in Figure 20(d), \(\Delta Q\) due to communication and memory approximations are quite low. Image compression not only reduces communication energy but also reduces processor energy consumption, as it packetizes fewer data. This reduction in compute energy easily makes up for the overhead of performing the JPEG compression itself. Therefore, the communication subsystem provides more system-level energy savings, resulting in \(\nabla EQ_{comm} \gt \nabla EQ_{mem}\) . Figure 20(e) shows the Q-E tradeoff vs. sensor and communication approximations without any change in the memory subsystem. Clearly, DNNs are more resistant to communication approximations, which results in \(\nabla EQ_{comm} \gt \nabla EQ_{sensor}\) . From Figure 20(f), we can infer that \(\nabla EQ_{mem} \gt \nabla EQ_{sensor}\) due to negligible \(\Delta Q\) from memory approximation despite higher energy savings from sensor approximation. Comparing the three graphs reveals that the memory approximation has the least impact on the application quality in Cam_Cloud. Thus, DNN resiliency to tolerate bit errors in the input image is clearly validated.

Here are the key takeaways from Figure 20. First, the compute and communication subsystems provide more energy saving opportunities than memory, when the sensor is not affected; second, the memory approximations have the least impact on Q in both variants; third, the sensor provides more energy savings than compute and communication, when memory is not approximated; and fourth, the sensor also provides better savings than memory when the other two subsystems are kept at their baseline configuration. In summary, these results show that errors arising from individual approximations tend to influence each other. Strong evidence was found that approximating multiple subsystems could definitely lead to more energy savings at the system level compared to individual approximations.

We also investigated inter-subsystem interactions in the context of image classification using ResNet101. Figure 21 represents the Q-E tradeoff for both variants of AxIS. Similarly to observations in the previous section, DNNs are quite resilient to memory approximations in the presence of other approximations, as shown by the minimal \(\Delta Q\) in these six plots. In fact, the drop is lower in DNNs for classification compared to DNNs for detection. This resembles our results in Section 4.5.2. Sensor behavior is the main difference between these two applications. We initially expected that the lower-resolution image acquired from the approximate sensor could potentially induce system-level energy reduction. However, bit errors due to DRAM refresh rate reduction and/or compute/communication approximations adversely affect the application quality to a large extent for subsampled images. This accuracy loss was mitigated by upsampling low-resolution images in the edge processor before they were stored in the DRAM, as mentioned in Section 4.5.1. Therefore, the sensor contributed very little to system-level energy savings compared to other subsystems, as observed in Figure 21(b), (c), (e), and (f). Here, computation and communication approximations mainly reduce system energy. It is important to mention that classification DNNs such as EfficientNet and EfficientNet_Lite show Q-E tradeoffs similar to detection networks, as they are inherently designed to support resolution and compound scaling. Sensor approximations actually result in system-level energy savings for the family of compressed models for these two architectures.

Figure 24(a) and (c) show \(\tilde{E}\) vs. \(Q_{b}\) for large and small classification benchmarks, respectively, for Cam_Edge. Similarly, Figure 24(b) and (d) present similar results for Cam_Cloud. As we can observe, the system-level energy savings \(\Delta E_{sys}^{edge}\) of Cam_Edge improve by a factor of \({1.6\times }\) for a negligible normalized quality degradation ( \(\Delta Q\le 0.5\%\) ) on average (geomean), for both large and small networks. However, in Cam_Cloud, large and small DNNs consume \({2.9\times }\) and \({2.2\times }\) less energy for the same \(Q_b\) , respectively. For higher-quality loss targets, \(Q_{b}=1\%-10\%\) , our design results in \(\Delta E_{sys}^{edge} = {2.1\times }-{4.5\times }\) for large DNNs and \(\Delta E_{sys}^{edge} = {1.6\times }-{2.8\times }\) for small DNNs in Cam_Edge. Similarly, in Cam_Cloud, we obtain \(\Delta E_{sys}^{cloud} = {3.5\times }-{5.5\times }\) and \(\Delta E_{sys}^{cloud} = {3.3\times }-{5.2\times }\) for large and small DNNs, respectively. As we can see, even a small amount of quality degradation allows us to achieve significant energy improvements with incremental benefits at larger degradation bounds. Surprisingly, the results reveal that even small DNNs (MobileNetV2, MNASNet1.0) optimized for edge/mobile deployment by design offer significant energy savings.

Figure 25 presents similar results for object detection. As can be inferred from the graphs, Cam_Cloud provides more energy savings compared to Cam_Edge on average—that is, \(\Delta E_{sys}^{edge} = 1.6\times\) and \(\Delta E_{sys}^{cloud} = 4.4\times\) at \(Q_b\le 0.5\%\) . Interestingly, this also indicates that AxIS using detection DNNs offers better energy efficiency in Cam_Cloud, compared to classification DNNs. In fact, this is true even for \(Q_{b}=1\%-10\%\) , where \(\Delta E_{sys}^{edge} = {2.3\times }-{12.1\times }\) and \(\Delta E_{sys}^{cloud} = {4.5\times }-{11.8\times }\) . This increase in energy savings is caused by the sensor approximations and its beneficial effects on other subsystems. These results validate that synergistic end-to-end approximations in approximate systems unlock the maximum energy efficiency for edge inference applications.

Let us take the example of the Cam_Edge system running ResNet101 (Figure 26(e)). As observed, we obtain \(37\%\) energy reduction at \(Q_{b}=0.5\%\) . Approximating this system further to \(Q_{b}=1\%\) , we achieve an additional \(31\%\) reduction w.r.t. \(Q_{b}=0.5\%\) . Proceeding further toward higher levels of approximations yields diminishing returns. We already know that the contribution of memory and computation subsystem to \(\tilde{E}\) is almost equal in an accurate system ( \(M=51\%, C=48\%\) from Figure 6). Here, we see that the memory approximation is the main component driving the overall energy reduction for \(Q_{b}\le 1\%\) . In contrast, most of the energy savings for higher \(Q_{b}\) are attributed to compute approximations, which also saves memory energy, as DRAM is active for a shorter duration. Similar DNNs that have fixed input size restrictions (Figure 26(a)–(d) and (g)–(i)) show similar behavior, as sensor approximations have negligible impact on them, as stated in Section 4.6.4. On the contrary, EfficientNet and EfficientNet_Lite (Figure 26(f) and (j)) show a high degree of energy reduction for all \(Q_b\) . This is caused by both the sensor and the compute subsystem and their interactions, thus validating our previous claims in Section 4.6.

Let us now investigate the contribution of the individual subsystems in Cam_Cloud. Since communication consumes \(65\%\) of the energy of the accurate system (Figure 6), approximations here lead to substantial energy savings for all DNNs. If we look at the graph for ResNet101 (Figure 27(e)), communication energy reduces from \(65\%\) to \(5\%\) , resulting in a reduction of \(71\%\) in the overall energy of the system. Unlike Cam_Edge, sensor approximations reduce considerable energy in Cam_Cloud, albeit at a high \(Q_{b}\ge 5\%\) . These results also indicate diminishing returns at higher \(Q_b\) for most DNNs. This is probably caused by saturation in communication energy savings for higher JPEG compression factors, as illustrated in Section 4.5.4. However, sensor approximations play an important role in EfficientNet and EfficientNet_Lite. In fact, smaller resolutions benefit the compute energy, as the compute subsystem has to perform JPEG compression on a smaller image. Similarly, it also reduces the size of the data to be offloaded, thus reducing the communication energy. These effects result in huge energy savings for these two networks, as can be seen in Figure 27(f) and (j).

We observe \(\approx \!37\%\) savings in \(\tilde{E}\) at \(Q_b\le 0.5\%\) for all four DNNs in Figure 28. As is evident, the memory approximation is the primary contributing factor. This is explained by the drop in \(E_{mem}\) from \(52\%\) to \(\approx \!14.5\%\) on average at this quality specification. However, for higher \(Q_b\) , all other subsystems interact and help extract the maximum energy efficiency. Although the contribution of sensor energy is much less compared to other subsystem, its impact is profound. The benefits resulting from sensor subsampling reduce the energy of multiple subsystems, as elaborated in Section 4.6.2 and 4.6.3. In addition, compressed models lead to faster inference on the smaller images. These effects result in substantial energy savings. On the contrary, Figure 29 shows that both the communication and memory approximations contribute to the \(\approx \!77\%\) energy reduction (on average) in \(\tilde{E}\) at \(Q_b\le 0.5\%\) . The drop in communication energy from \(\approx \!66\%\) to \(\approx \!3\%\) and the drop in memory energy from \(\approx \!15\%\) to \(\approx \!4\%\) clearly support this deduction. At \(Q_b\ge 1\%\) , the sensor approximation again has the most significant impact, as the benefits percolate through all subsystems.

Figure 30(a) and (c) show this comparison for large and small DNN benchmarks for Cam_Edge. For large DNNs, the proposed Cam_Edge system, on average, yields \({1.6\times }-{3.0\times }\) , \({1.03\times }-{2.2\times }\) , and \({1.6\times }-{1.7\times }\) energy improvements over sensor-only, memory-only, and compute only approximations, respectively, for \(Q_{b}=0.5\%-10\%\) . In comparison, smaller DNNs show \({1.6\times }-{2.3\times }\) , \({1.0\times }-{1.6\times }\) , and \({1.6\times }-{1.7\times }\) energy improvements. Following common intuition, large DNNs are more resilient, resulting in higher energy gains. Figure 30(b) and (d) show similar trends for Cam_Cloud. For the same \(Q_b\) range, the energy improvements at the system level compared to sensor, memory, and communication amount to \({2.8\times }-{4.4\times }\) , \({2.5\times }-{4.5\times }\) , and \({1.4\times }-{1.7\times }\) for large DNNs. For small DNNs, these numbers change to \({2.2\times }-{4.1\times }\) , \({1.9\times }-{4.4\times }\) , and \({1.3\times }-{1.6\times }\) . The feasibility and effectiveness of synergistic approximations in AxIS are clearly demonstrated in these results.

Figure 31 shows the effectiveness of AxIS for object detection. Cam_Edge on average leads to \({1.6\times }-{2.5\times }\) , \({1.0\times }-{7.4\times }\) , and \({1.6\times }-{4.4\times }\) energy improvements over sensor-only, memory only, and compute-only approximations, respectively, for \(Q_{b}=0.5\%-10\%\) . Similarly, Cam_Cloud produces \({3.9\times }-{4.3\times }\) , \({3.9\times }-{10.3\times }\) , and \({1.4\times }-{3.4\times }\) energy improvements for the same quality range. Overall, these results clearly indicate that the effectiveness of AxIS is very well translated to the object detection application, similar to that observed for image classification.

Consideration of these comparative results reveals interesting facts. First, energy benefits due to memory approximations usually plateau for \(Q_b\gt 0.5\%\) . Second, sensor approximations benefit detection DNNs for all \(Q_b \gt 1\%\) . In comparison, the benefits in the classification DNNs are limited and are only visible in EfficientNet and EfficientNet_Lite (i.e., that support resolution scaling). Third, the communication approximations provide energy savings for all \(Q_b\ge 0.5\%\) . However, the benefit gradually saturates for higher bounds. Fourth, approximations in compute subsystem are usually visible for \(Q_b\ge 1\%\) and provide a gradual Q-E tradeoff at subsequent quality bounds.