Tailor: Altering Skip Connections for Resource-Efficient Inference

While skip connection removal has been studied before [8, 25, 32, 50, 51], prior work is lacking in several ways: (1) preliminary work [32, 50, 51] only studies shallow models (up to 34 layers); (2) Li et al. [25] do not remove all of the skip connections in the models they evaluate; (3) Ding et al. [8] and Li et al. [25] both have limited architectural evaluations (e.g., GPU and mobile) that do not consider the highly customized skip connection memories enabled by FPGAs; and (4) Ding et al. [8] require starting with an entirely new NN topology whose skip connections are removable.

Monti et al. [32] start with a standard ResNet and introduce a new training method. This method uses an objective function that penalizes the skip connections and phases them out by the end of the training. This technique has only been applied to smaller ResNets (18 to 34 layers) with a small decrease in accuracy between 0.5% and 3%.

Zagoruyko and Komodakis [50] also develop a method for removing skip connections in an NN. They replace skip connections with Dirac parameterization, creating a new NN called DiracNet. The Dirac parameterization is shown in Equation (1),where \(\sigma (\cdot)\) is the nonlinear activation function, W is the layer weight matrix, x is the layer input, and y is the layer output. For ease of comparison with ResNets, Equation (2) is simplified to show only one convolutional layer. In fact, skip connections in ResNets hop over more than one convolutional layer, whereas in DiracNets, the identity mapping is over one single convolutional layer. Therefore, the weights and the identity mapping of the input can be folded because \(x+Wx=(I+W)x\) . This change requires DiracNets to widen the NN layers in the ResNets that they started with. The authors showed that their technique could be used to create models with up to 34 layers. Although it works for shallower models, DiracNets show a decrease in accuracy between 0.5% and 1.5% compared with ResNets. In contrast, SkipRemover eliminates skip connections without widening the layers in the NN and does not need to make this accuracy trade-off.

Li et al. [25] develop residual distillation (RD), which is a modified knowledge distillation framework. RD starts with a ResNet as the teacher and a plain CNN without skip connections as the student. Unlike standard knowledge distillation, RD passes the teacher’s gradients to the student during training. This differs from Tailor because RD starts with a student model without skip connections, whereas Tailor gradually modifies a model’s skip connections every few epochs during training without sharing gradients. Moreover, while RD removes all skip connections from models evaluated on simpler datasets such as CIFAR-10 and CIFAR-100 [24], it fails to remove all skip connections in its ImageNet evaluation, leaving 18% of them in the network, which is a costly choice. In our ImageNet evaluation (see Section 4.1), our SkipRemover method removes all skip connections with minimal accuracy loss.

Ding et al. [8] introduce a new model architecture RepVGG, which trains using 3 × 3 convolutional layers that are each skipped over by both a 1 × 1 convolution and an identity connection. At inference time, these connections can be re-parameterized into the parameters of the 3 × 3 convolutional layers. While RepVGG is more accurate than our SkipRemover model, it requires starting from their specialized training model architecture. This is costly to developers who have already trained a model with skip connections on their dataset. Similarly, transferring a pre-trained RepVGG model to a new dataset via transfer learning can be time-consuming given the many different methods [36, 47, 52] to evaluate. As such, Tailor is ideal for these developers because it modifies the skip connections of an existing pre-trained model to be more resource efficient with minimal to no accuracy loss. Developers can leverage the training they have already done and need not start from scratch with a brand new RepVGG architecture.

ShuffleNet [28], DiracDeltaNet [48], and DenseNet [20] simplify skip connections by making them concatenative, i.e., they concatenate, rather than add, the skip connection data to the output of a layer. Concatenative skip connections take advantage of the fact that spatially consecutive memory accesses are typically faster than random accesses. This concatenation and off-chip data movement is possible using a simple controller (e.g., DMA engine).

Tailor uses two techniques to simplify the skip connection hardware. SkipRemover eliminates all logic and memory needed for a skip connection, making them less expensive than concatenative skip connections. Careful retraining allows skip connection removal in smaller networks with no degradation in accuracy. For larger networks, SkipShortener shortens the additive skip connections. By reducing their lifespans, the hardware implementation requires fewer resources. SkipShortener is not necessarily simpler than ShuffleNet [28] or DiracDeltaNet [48]. However, these concatenative skip connections have only been evaluated on image classification and object detection tasks. In our work, we demonstrate our SkipRemover and SkipShortener methods on multifarious NNs and classification tasks, namely, image-classifying ResNets of varying depths, DNA-basecalling QuartzNet-5 × 5, and automatic-speech-recognizing QuartzNet-10 × 5. With respect to DenseNet [20], SkipShortener ResNets use much less memory and bandwidth because DenseNet relies on significantly more skip connections throughout its NN. Given an NN with L layers, DenseNet needs the memory and bandwidth to execute \(L(L + 1)/2\) concatenative skip connections compared with SkipShortener ResNets’ mere L skip connections. With so many more skip connections, DenseNet is more expensive for hardware than SkipShortener ResNets.

Finally, all these techniques simplify skip connection hardware from the outset, building their models with modified skip connections and then training them from scratch. Tailor differs because its hardware-aware training method dynamically alters the skip connections every few epochs during training, taking advantage of what the NN has learned with skip connections. Thus, Tailor allows the NN to gradually adapt to shortened skip connections (SkipShortener) or none at all (SkipRemover).

Figure 2 shows three hardware implementations for NNs with traditional, shortened, and no-skip connections. The implementations correspond to accelerators produced by hls4ml—a tool that translates Python models into high-level synthesis code [11]. hls4ml creates a separate datapath for each layer and performs task-level pipelining across the layers. The layers communicate using first-in first-out buffers (FIFOs; AXI streams). Everything encapsulated by a dashed line resides in one pipeline stage. The inputs are fed into the architecture using a stream, and the results are given as an output stream. The weights are all stored on-chip, and all the internal results are stored on-chip. We evaluate each of these designs on FPGA later in Section 4.2 along with another style of architecture using a 2D array of processing elements. Tailor allows us to trade off between accuracy, performance, and resource usage through co-design of the neural network using hardware-aware training.

Figure 2(a) shows the hardware needed to implement a single ResNet’s skip connection. Note that in all of the designs shown in Figure 2, we fuse the batch normalization parameters with the kernel, as is commonly done [21]. To be low latency and high throughput, the design uses task-level pipelining (i.e., the HLS dataflow pragma) for each NN layer, or a small grouping of layers, and streams the data between each dataflow stage using FIFOs. Since FIFOs can only be read from once, skip connections complicate the design. We must spend a dataflow stage on cloning the skip connection data from its input FIFO into two other FIFOs so that it can be read twice for its two datapaths. The first path goes through a collection of convolutional and ReLU layers, and the second stores the data in a FIFO exclusive to skip connections (skip FIFO). Once the data has gone through the first path, we read from the skip FIFO to perform the addition to complete the skip connection’s identity function. As such, implementing a skip connection on chip requires several extra FIFOs for handling the skip connection data. This, in turn, increases on-chip memory resource utilization.

Ideally, we would eliminate the skip connections. As seen in Figure 2(b), without skip connections, we cut the number of dataflow stages in half (no more Clone, Add, or ReLU stages) and use less than half of the requisite FIFOs compared with Figure 2(a). All we need to do is pass the data through the convolutional and ReLU layers. This reduces resource utilization by up to 16% (see Section 4.2).

It may not be possible to remove the skip connections because they are essential for training convergence. In these cases, shortening the skip connections can simplify their hardware implementation. Figure 2(c) shows a modified network with shortened skip connections such that each skip connection’s lifespan resides within a single dataflow stage. We do not need additional dataflow stages to clone skip connection data. The shorter lifespans allow the shortened skip connections to be stored in shift registers, which can be implemented using the more abundant FFs as opposed to BRAMs, which are used in the traditional skip connection’s hardware design. In this way, we exploit the short skip connections’ lifetimes and use simpler, more efficient hardware memories to implement them (see Section 4.2). As such, we achieve a similar architecture to the version without skip connections (Figure 2(b)), and similarly reduce resources spent on additional dataflow stages and FIFOs in Figure 2(a). SkipShortener is thus more resource efficient than the traditional skip connection design. In fact, SkipShortener provides a trade-off between the SkipRemover and traditional designs because it uses more resources than SkipRemover but less than the traditional one (see Section 4.2). However, as we later show in Section 4.1, SkipShortener maintains accuracy in cases in which SkipRemover accuracy drops off. Thus, SkipShortener allows for design space exploration to balance accuracy and resource usage.

When used with hls4ml, Tailor reduces resource consumption without changing the performance. This is a consequence of hls4ml’s dataflow design; the resources we remove are not on the critical path—they are operating in parallel to the critical path. A dataflow design uses task-level pipelining; thus, reducing the resources spent on stages not on the critical path does not help or hurt overall throughput. Based on our Vivado co-simulation results, the clone stage executes in microseconds whereas the convolutional layer executes in milliseconds, an order of magnitude difference. Therefore, removing the clone buffer (Figure 2(b)) or implementing it more efficiently (Figure 2(c)) will not affect the overall dataflow latency because its latency is an order of magnitude less than the convolution’s latency. This means that Tailor’s resource reductions do not increase or decrease latency or throughput for this architecture style, as later shown in Table 7.

Another prevalent style of FPGA CNN architecture instantiates a 2D processing element (PE) array and iteratively programs the convolutions and other operations onto that PE array. We call this style of computation a Reconfigurable DNN Architecture. Figure 3 provides an example architecture used in our experiments. We build this architecture using DeepSoCFlow.¹ Following the taxonomy described in [22], the reconfigurable DNN architecture is a 2D array of processing elements that optimally perform standard convolution and matrix multiplication with high data reuse. The dataflow is primarily output stationary while prioritizing maximal weight reuse and also reusing inputs to an extent. The engine performs fixed-point computations, where the input, weight, and output bit widths are adjustable as synthesis parameters, along with the number of rows and columns of processing elements. The weights rotator prefetches the weights of the next iteration into one block of on-chip memory while the other bank delivers weights, rotating them hundreds of times for maximal data reuse. The Pixel Shifter shifts perform vertical convolution. Partial sums are shifted to the PE on the right to compute horizontal convolution. The results are streamed out through the output direct memory access (DMA) to the off-chip memory. The runtime controller would perform the residual addition, quantization, and activation on the processing system side while the engine computes the next iteration. Our implementation uses the ARM processor available in the Zynq chip. FPGAs without processors could instantiate a softcore processor to perform the controller runtime operations.

The Tailor optimizations have different effects on the Reconfigurable DNN architecture as compared to hls4ml architecture. The Reconfigurable DNN architecture computes skip connections by loading input data from off-chip memory and performing the required operations upon it (addition, convolution). Thus, unlike in hls4ml, removing a skip connection does not change the architecture; instead, it changes how computations are mapped to that architecture. Skip connection removal eliminates the need to fetch the skip connection data and perform the associated convolution and addition operations. This increases the overall performance, as we describe in Section 4.

It is difficult to modify an NN’s skip connections without reducing accuracy. Naïvely removing all skip connections before or after training an NN is detrimental to its accuracy. Instead, Tailor consists of two training algorithms, SkipRemover and SkipShortener, that gradually alter an NN’s skip connections on the fly—removing or shortening them every few epochs—in order to make them resource efficient. Gradually altering the model during training tempers the performance drop of removing or shortening the skip connections, yielding minimal to no loss in accuracy as well as significant advantages in the hardware implementation, as described above.

Tailor’s iterative learning approach fine-tunes the altered NNs using a compression method known as knowledge distillation (KD) [19]. KD distills the knowledge of a larger, more complex NN (the teacher) into a smaller, simpler NN (the student). While the student model is training, it compares its output with the teacher model’s output and thus learns from the teacher to perform better. KD provides impressive results for compressing NNs for various applications [31, 41, 44]. In traditional KD, the teacher model is already trained and the student model is trained to match the teacher’s behavior by replicating its output. The student achieves this by training with a loss function:where \(\mathcal {G}\) and \(\mathcal {H}\) are distance functions, s and t are student and teacher output vectors, respectively, \(\ell\) is the correct label vector, and \(\beta\) is a tunable parameter [19].

With this idea in mind, both SkipRemover and SkipShortener start with two identical pre-trained NNs with traditional skip connections, where one serves as the teacher and the other serves as the student. During the retraining stage, SkipRemover removes a given skip connection every few epochs. SkipShortener takes a similar iterative approach and, every few epochs, splits a given skip connection into multiple shorter ones. The skip connections are removed or shortened starting from the first skip connection encountered in the NN (from the input) to the last.

Figure 4 visualizes both SkipRemover’s (Figure 4(a)) and SkipShortener’s (Figure 4(b)) training algorithms for a ResNet-style NN. During training, we remove (SkipRemover) or shorten (SkipShortener) one of the student’s skip connections every \(\alpha\) epochs. If n is divisible by \(\alpha\) (as in Figure 4), then at epoch n, the student has had \(n/\alpha\) skip connections altered, and we are viewing the next two skip connections to be modified in the student model: the \((n/\alpha) + 1\) st and \((n/\alpha) + 2\) nd. At epoch \(n+\alpha\) , the \((n/\alpha) + 1\) st skip connection is altered (removed under SkipRemover or split into two shorter skip connections under SkipShortener). The NN then trains for \(\alpha\) epochs so that the student model can improve its weights given the latest model topology. Afterwards, at epoch \(n + 2\alpha\) , the \((n/\alpha) + 2\) nd skip connection is similarly altered. During the entire skip modification retraining process, the student uses the KD loss function \(\mathcal {L}\) defined in Equation (3) to learn from the teacher and the true labels. The teacher’s model topology and weights remain fixed during training. Once all skip connections have been altered, the student model continues training under KD for the remaining number of training epochs as defined by the user. Only the student model is used for inference.

Tailor is novel because it dynamically transforms skip connections every few epochs during training. This is an instance of hardware-aware training because the skip connections are slowly altered specifically to reduce hardware resources, as previously discussed in Section 3.1. The gradual skip connection alterations allow the NN to take advantage of what it has learned with skip connections so that it can dynamically adapt to shortened skip connections (SkipShortener) or none at all (SkipRemover). Algorithm 1 describes Tailor’s hardware-aware training process.

To evaluate how Tailor affects an NN’s accuracy, we train ResNets and QuartzNets of varying depths using our SkipRemover and SkipShortener algorithms in PyTorch [39]. The ResNets range from 20 to 110 layers and are trained on the CIFAR-10 [24], CIFAR-100 [24], and SVHN [35] datasets. We also evaluate ResNet50, which has a different skip connection topology than standard ResNets, on the ImageNet dataset [7]. The QuartzNets span between 29 and 54 layers. Their structure is determined by the number and lifetimes of their skip connections. For instance, a QuartzNet-10 × 5 has 10 skip connections that each have a lifetime of 5 sets of layers. We train a QuartzNet-5 × 5 on the Oxford Nanopore Reads dataset [42], a DNA basecalling task. We also train a QuartzNet-10 × 5 on the LibriSpeech dataset [37], an automatic speech recognition (ASR) task, which converts speech audio to text. ASR tasks are assessed using word error rate (WER), which measures the percent of words that the model predicted incorrectly. In all of our ResNet and QuartzNet-10 × 5 training experiments, we set \(\alpha = 3\) in Algorithm 1 so that skip connections are removed or shortened every three epochs. For QuartzNet-5 × 5, we set \(\alpha = 1\) instead because it trains better this way. For the ResNets, we set \(\mathcal {G}\) and \(\mathcal {H}\) in Equation (3) to categorical cross entropy and mean-squared error, respectively, and set \(\beta = 0.35\) . For the QuartzNets, we set Equation (3)’s parameters similarly, except for \(\mathcal {G}\) , which we set to connectionist temporal classification loss, which is used to train difficult tasks involving sequence alignment (like DNA basecalling and ASR). Note that in our training results, “Baseline” refers to the unmodified NN counterpart with conventional skip connections.

Figure 5 shows that SkipRemover works well for ResNet-44 and smaller, at times even outperforming its baseline (traditional skip connection model). However, its accuracy drops as the number of layers increases. This indicates that shallower NNs do not need skip connections for these classification tasks, but they become more necessary for deeper networks. SkipShortener mostly outperforms the baseline on all three datasets, even on deep models.

We first quantize ResNets ranging from 20 to 56 layers deep to see how Tailor’s accuracy fares under reduced precision. We then evaluate Tailor’s effects on hardware resources and latency by performing a case study on ResNet-20-style skip connections implemented using the hls4ml architecture, i.e., the designs illustrated in Figure 2. We select this style of skip connection because it is the fundamental building block of ResNets that range from 20 to 110 layers. In our case study, we vary the bit precision and number of filters to see how Tailor scales up. Based on how Tailor’s resource reductions scale, designers can understand how Tailor extrapolates to their own hardware designs. We report latency as well as P&R resource results on the Alveo U200 FPGA accelerator card (part no. xcu200-fsgd2104-2-e). For end-to-end application results, we evaluate the benefits of Tailor on two different styles of CNN architectures. The first uses the hls4ml tool to generate architectures. The second is the Reconfigurable DNN Engine—a 2D array of processing elements. Both styles of architectures are described in Section 3.1.

Prior work investigated why skip connections are so helpful to ResNets. Veit et al. [45] argue that ResNets behave like ensembles of smaller subnetworks that vary in depth and allow the NN to train and converge more easily. Li et al. [26] and Yao et al. [49] show that introducing skip connections makes the NN loss landscape much smoother with less nonconvexity. They show that naïvely removing these skip connections causes an explosion of nonconvexity in the loss landscape, which makes training significantly more difficult. We confirm these results in our ablation studies (Section 4.1), as accuracy indeed drops when skip connections are removed naïvely. With both KD and SkipRemover, we see an improvement in accuracy. Since the student is trying to mimic the teacher’s outputs, it is possible that the teacher’s outputs guide the student in such a way that prevents the loss landscape from becoming less smooth. Theoretical work from Lin and Jegelka [27] has proven that a ResNet with one-neuron hidden layers is a universal approximator. This work suggests that adding more neurons to the hidden layers creates an over-parameterized ResNet. Since stochastic gradient descent performs better in the presence of over-parameterization, having more neurons per hidden layer increases training efficiency, making it easier to converge. This work also argues that a ResNet is essentially a sparse version of a fully connected NN because the identity skip connections create simpler paths within the ResNet, which was similarly posited by Lin and Jegelka [27]. Given that CNNs and ResNets have both been proven to be universal approximators [27, 40], this implies that there exists a set of parameters for a CNN that can mimic a ResNet such that they equal the same function. It is mainly easier to find a well-performing ResNet because Lin and Jegelka [27] showed that one-neuron hidden layers is sufficient for a ResNet to be a universal approximator.

In our work, Tailor has taken removing and shortening skip connections to their extremes: it either fully removes or fully shortens all the skip connections in an NN. It would be worthwhile to understand the accuracy versus resource utilization trade-off under less extreme cases, e.g., removing only half of the skip connections. It would also be interesting to mix SkipRemover and SkipShortener to try to recover accuracy in the instances in which SkipRemover fails. These approaches may help address SkipRemover’s scalability issues and strike a balance between SkipShortener’s high accuracy and SkipRemover’s resource savings and performance improvements.