Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization

In essence, LLMs build on a series of Transformer blocks [59], each of which consists of the attention and feed-forward layers, as illustrated in Figure 1. In this section, for ease of understanding the subsequent sections, we briefly describe the operation flow within the Transformer block using the terminology and notation used throughout the paper.

In the attention layer, there are four weight matrices, which are used for linear projections: query ($W_{Q}$), key ($W_{K}$), value ($W_{V}$), and output ($W_{O}$) projection matrices. Each projection matrix has a dimension of $d_{model}$ ${\times}$ $d_{model}$, where $d_{model}$ is the dimension of an input embedding vector (i.e., the number of features). For an input sequence length of $n$ (i.e., $n$ tokens), the input ($X$) to the attention layer is an $n$ $\times$ $d_{model}$ matrix. Then, each of the query ($X_{Q}$), key ($X_{K}$), value ($X_{V}$) matrices of the attention layer is computed by multiplying the input matrix ($X$) with their respective weight matrices ($W$).

After the QKV projection, we compute the attention score ($X_{S}$) and the attention value ($X_{S}{\times}X_{V}$) using the projected matrices. Lastly, the output of the attention layer ($X_{O}$) is computed by multiplying the attention value ($X_{S}{\times}X_{V}$) with the output weight matrix ($W_{O}$) along with a residual add ($X$).

The feed-forward network (FFN) in the Transformer block takes as input the output of the attention layer ($X_{O}$). It consists of two fully-connected (FC) layers, thus there are two weight matrices, $W_{FC1}$ and $W_{FC2}$, each of which has $d_{model}$ $\times$ $h$ and $h$ $\times$ $d_{model}$ dimensions. The output of the feed-forward layer ($X_{T}$) is computed using the following equation.

In addition, there exists a layer normalization operation (LayerNorm) at the start or the end of each attention and FFN layer, which we omit in the figure for simplicity. The Transformer block produces an output with the same dimensionality as the input, which makes Transformer-based LLMs easily scalable by adjusting the number of Transformer blocks.

The state-of-the-art post-training quantization (PTQ) methods [16, 33] demonstrate that the weights in LLMs can be effectively quantized to eight or even four bits via standard per-tensor (e.g., a scaling factor for each tensor) or grouping-based (e.g., a scaling factor for $g$ consecutive weights in a tensor) quantization techniques without a significant degradation in model performance. In contrast, it is quite challenging to quantize activations in LLMs using the same methods due to the existence of outliers, which refer to the extremely large magnitude values compared to others within a tensor.

Figure 2 shows the values in several weight and activation tensors. As depicted in the figure, the input activation tensors of the attention and the first FC layers (i.e., $X$ and $X_{O}$) have significantly large values in a few input (feature) dimensions, whereas the weight tensors have a relatively similar range of values. In principle, this makes it difficult to quantize activations compared to the weight tensors in LLMs due to the wide range of values, which we discuss further in the following sections. Prior studies show that outliers are concentrated in the fixed channels of activation tensors across the layers and batches [62, 11] due to the model intrinsic, such as large LayerNorm weights in the fixed channels across the layers. We also observe a similar trend, as shown in Figure 3. We see the presence of vertical red or blue lines for each attention input tensor. This indicates that the outliers are typically at the channel granularity (i.e., within a few specific channels).

Depending on how elements in a tensor are being quantized as a group, there can be various levels of quantization granularity, including per-tensor, per-row, and per-column quantization. In per-tensor quantization, all the elements within the tensor share the same quantization parameter (i.e., a scale factor), which simplifies the quantization process. Per-row or per-column quantization shares the parameter at a row or column granularity to further reduce the quantization error.

Table I shows the perplexity when we quantize the activation at three different granularities. As shown in the table, the per-column granularity (i.e., input/feature dimension) shows the best perplexity, as it uses the quantization parameters for outliers that differ from others. However, applying per-column quantization to activations poses challenges in the modern GPU or TPU integer pipelines since each element needs scaling during the reduction operations in matrix multiplication. Consequently, all prior LLM quantization works employ per-row (per-token) or per-tensor quantization for activations and per-column or per-tensor quantization for weights.

There are also several outlier-aware accelerators that use mixed precision to quantize normal values into low bits while separately handling the outliers. GOBO [70] is a weight-only quantization scheme that quantizes outlier weights using higher precision. OLAccel [44] uses a few 16-bit MAC units alongside 4-bit MAC units to deal with outliers. DRQ [52] employs a fine-grained detection algorithm to identify sensitive regions in a tensor. All of these works use mixed precision, requiring complex hardware and unaligned memory access. OliVe [20] is the most recent work, which quantizes outliers using custom number representations. Although there is no mixed precision involved, it prunes adjacent normal values and requires an encoder/decoder to support its custom datatype.

Intuitively, one can choose to split activation tensors along the reduction axis (i.e., columns in 2D) and quantize each group with different scaling factors instead of employing an impracticable per-column approach. We can then represent the matrix multiplication via the channel grouping as follows:

where $s_{i}$ and $s_{w}$ are the scale factors of activation of group $i$ and weight. $P_{i}$, $Y$, and $G$ denote the partial sum from group $i$, the final resulting matrix, and the number of groups.

The above execution model still suffers from lower utilization of compute cores due to smaller submatrices and frequent rescaling to accumulate partial product $P_{i}$. Thus, to fully benefit from quantization while preserving the accuracy of the model, we need to address the following challenge—splitting channels along the reduction axis and grouping channels with similar ranges to isolate outliers from the others while retaining the reduction axis to better utilize compute cores.

Our intuition is that we can retain the reduction axis of matrix multiplication by processing partial sums in a specific order and rescaling the accumulated value before adding the next partial sum. Our execution model can be expressed as the following equation:

where $\mathrm{A_{i}}$ represents the results accumulated up to $i$th partial sums. Note that Equations 1 and 2 are mathematically equivalent. For Equation 2 to operate efficiently in hardware, the rescaling needs to be performed by the integer MAC units while preserving correctness. We will use the term rescale factor to denote $s_{i}/s_{i+1}$ between the channel groups.

To this end, we propose Tender, an algorithm hardware co-design technique for quantizing large language models entirely into INT4/INT8 without using mixed precision, custom datatypes, or re-training. The carefully designed post-training quantization (PTQ) algorithm of Tender decomposes channels to isolate outlier channels from normal ones in activation tensors. At the same time, Tender enables rescaling inside the tensor compute units, so it does not require explicit requantization and thus fully utilizes the integer pipelines. In the following sections, we discuss how the above rescaling can be done inside the compute units with minimal extensions to hardware through the software-hardware co-design of Tender.

In this section, we introduce tensor decomposition and runtime requantization of Tender. As discussed in Section II-B, outliers reside along the channels in the activation tensors across the layers of LLMs. Thus, we decompose channels to minimize the quantization error by isolating the outlier channels from the others. We propose a “Power of 2” channel decomposition rule that aligns well with the value distribution of activation tensors. Also, we present runtime requantization that works harmonically with the decomposition rule. It enables matrix multiplication between decomposed, quantized activation tensors and linear symmetrically quantized weights without involving floating-point operations but in a mathematically equivalent way. We then present a walking example of the Tender algorithm and optimizations at the end of the section.

While the explanation in this section is based on INT8 quantized activation-weight matrix multiplication, the same algorithm is applied to INT4 or activation-activation matrix multiplication (e.g., $X_{Q}\times X_{K}^{T}$ in a Transformer block). Furthermore, Tender can be easily extended to other bit widths (e.g., 5, 6, 7-bit integers) in the same way if the hardware supports such datatype operations. This is possible due to the standard and symmetric quantization of Tender, whereas other approaches typically need to define new custom datatypes.

Then, to accommodate the presence of outliers in quantization, we decompose the channels of the activation tensor into multiple groups of subtensors and use separate scaling factors for each group. Through runtime requantization, we multiply the decomposed quantized activation tensors and the linear symmetrically quantized weight without explicit dequantization with negligible latency overhead. Finally, the bias multiplied with weights is added to the output for mathematical correctness. Note that all the weights can be quantized to INT8 before inference. Also, channel decomposition, channel biases, and scale factors are pre-computed during calibration.

1) Why use classification?: We can decompose channels in various ways. In the extreme scenario, we can use individual scale factors for each channel, which can greatly reduce the quantization error [65, 5]. However, in this way, as Figure 5(a) shows, the number of operations including matrix multiplication, dequantization, and addition between partial products increases in proportion to the number of channels. In the case of OPT-6.7B [72], for example, it increases 4096 times compared to the original computation without channel decomposition. As such, grouping channels that have a similar range of values to share a scale factor via clustering or classification is a more practical and scalable option. Clustering computes how similar the value range is between every channel and then groups the channels with a small distance. Classification is based on pre-defined thresholds where the channels are binned to fit within the threshold range. Thus, clustering can group the channels more accurately than classification but is not likely applicable at runtime (without complex hardware) due to the large computational overhead. So, we choose to use classification to make our algorithm simple and to be easily extended to support runtime decomposition.

2) Why use coarse-grained thresholds for large values and fine-grained thresholds for small values?: The maximum quantization error is $0.5\times\mathrm{(scale\ factor)}$ since the maximum value of rounding error is $0.5$. The scale factor is set proportional to the absolute maximum of the group. So, as the absolute maximum of the group becomes larger, the quantization error grows linearly. Also, considering that there exist multiple channels in a group, the quantization error of a group grows linearly to the number of channels that are included in the group. We can express the quantization error of a group as follows:

Thus, for a group with a large absolute maximum, we need to minimize the number of channels in the group. Intuitively, this can be viewed as isolating quantization errors due to the large scale factor to only a few number of channels. Similarly, for a group with a large number of channels, we need to minimize the absolute maximum of the group. As Figure 2 shows, only a few channels have large magnitude values, and most of the channels have values near zero in the activation tensor. Thus, using coarse-grained thresholds (i.e., a large absolute maximum) for the channels with large magnitude values does not hurt the overall accuracy because the group includes only a few channels. Meanwhile, using fine-grained thresholds (i.e., a small absolute maximum) for the channels with small magnitude values can effectively minimize the accuracy drop by using the small threshold near the value range of the channels. Thus, our approach can efficiently classify the channels with a minimum accuracy drop. Note that, of course, it is possible to achieve better accuracy by using fine-grained thresholds for both the channels with large magnitude values and channels with small magnitude values. However, dividing more channels incurs a computation overhead, whereas using a fine-grained threshold for the channels with large magnitude values has a minimal increase in accuracy due to the small number of channels that are included in the group.

3) Why use 2?: Setting scale factors by dividing the maximum scale factor with the power of 2 has two key advantages. First, we can guarantee the lower bound of the quantization level. When a channel is assigned to a quantization group, the absolute maximum of the channel is at least larger than half of the threshold of the group. Thus, even for the worst case, $n-1$ bits are utilized for $n$-bit quantization. Second, when the ratio between the scale factors is a power of 2, we can efficiently compute the matrix multiplication involving decomposed quantized activation tensors with a negligible latency overhead, which we discuss in the following paragraph. In summary, the benefit of the “power of 2” is that it not only enables rescaling with integer arithmetic but also considers the alignment with the channel distribution in activations.

Our intuition is that the systolic array accumulator has a sufficiently large bit width, and we can safely requantize the partial products with a proper rescaling factor without clipping values due to limited bit width. In this way, we requantize accumulated partial products without involving explicit floating-point operations. We use a shifter residing next to the accumulated sum for requantization since we use 2 as a rescaling factor. Figure 5(b) illustrates our runtime requantization approach. We perform matrix multiplication with a channel group that has the larger scale factor first. Then, before computing matrix multiplication of the next group, Tender shifts the accumulated integer value by 1-bit. After finishing the matrix multiplication of all the groups, we dequantize the final result using the smallest scale factor. This approach incurs a negligible latency overhead for rescaling with a small hardware extension. To reduce the overhead of determining a scale factor at runtime, we further optimize our scheme to use calibration to pre-compute the scale factors and biases of each channel offline. We also classify each channel into a group at calibration time and only apply the metadata to perform quantized matrix multiplications at runtime.

For example, we can divide the rows of the activation tensor into several chunks and calibrate the bias and scale factor for each row chunk. Figure 2 shows that there exist not only inter-channel variances but also intra-channel variances. Thus, we may consider intra-channel variances as well as inter-channel variances to group values more effectively. We observe that this can greatly improve the accuracy of our algorithm by taking the characteristics of each row chunk into account. Because matrix multiplication is typically performed in a tiled manner (i.e., tile-by-tile), row chunking naturally fits the execution model with almost no additional complexity. We use 256 as a row chunk size. We also apply per-column weight quantization and per-head activation quantization, which also incur negligible calibration overhead.

When optimizing with row chunking, channel grouping is applied to each row chunk, and channel indices, biases, and scale factors are calibrated independently for each chunk offline. This makes the group size and compute ordering differ between row chunks. From the hardware perspective, the row chunk size needs to be larger than the systolic array dimension; otherwise, there can be underutilization of the compute unit. This is because the systolic array computes rows and applies runtime requantization at the granularity of the systolic array dimension. While the on-chip buffer reuse decreases, it has a negligible impact on performance with a reasonably large chunk size. As mentioned, we choose 256 as a balance point, where accuracy remains close to the baseline model due to fine-grained row grouping, and it is also sufficiently larger than the systolic array dimension.

Figure 6(a) shows an overview of the Tender architecture. Memories are HBM2, Scratchpad Memory, Output Buffer, and Index Buffer. The HBM Controller manages data movement between the on-chip buffers and HBM2. The Execution Controller sends an address to the Scratchpad Memory to bring data from the on-chip memory to the systolic array for computation. It also sends control signals to the systolic array to manage its operations. The data is computed in the Multi-Scale Systolic Array (MSA) and Vector Processing Unit (VPU). Figures 6(b) and 6(c) show our Multi-Scale Systolic Array architecture, which follows conventional architectures with minimal extensions of a 1-bit rescale signal and a 1-bit shifter inside the processing element (PE).

The main matrix multiplication operation occurs in our Multi-Scale Systolic Array (MSA). As illustrated in Figure 6(b), the MSA is a 2-D mesh of PEs with FIFOs attached for skewing the inputs and weights. In Tender, we use a single 64$\times$64 systolic array with each PE executing a 4-bit MAC operation per cycle. When the model precision is INT8, 4 PEs are grouped to perform 8-bit multiplication, with each PE handling either upper or lower 4 bits of inputs and weights. Note that we can scale the number of systolic arrays similarly to the commercial design.

MSA is an output stationary compute module where a partial sum is accumulated in each PE, with an extension of the 1-bit shifter attached to the accumulator register. Figure 7 shows how our MSA works on a series of decomposed matrices with different scale factors. During the computation of each decomposed matrix, PEs perform normal MAC operations, and the accumulated value is stored in the internal 32-bit register (Acc). When the PEs complete matrix reduction for a decomposed matrix (e.g., $A_{1}\times W_{1}$), a 1-cycle bubble is inserted for the input and weight with a rescale signal set (Figure 7(a)). During the rescale operation, it requantizes the values by shifting 1 bit of the accumulator register to the left. Since each PE finishes matrix reduction for a given decomposed matrix at different cycles, the rescale signal is synchronized with the wavefront of the input and is passed along to the next PE in the same row. To generate rescale signals at the right cycles, the Execution Controller has the metadata of the number of channels executed, indices of tensor splitting points, and rescale factors (if needed).

We choose our systolic array to be output stationary for two reasons. First, it allows us to easily extend the systolic array to handle arbitrary rescale factors (i.e., other than $\alpha=2$) when needed. Since the accumulator register is wired with the multiplier and shifter within a PE, we can split the register into equal parts and multiply each part with an arbitrary integer rescale factor that can come from the 4-bit input or weight datapath. For example, with a given rescale factor $\alpha$ (e.g., 3) coming from the input datapath, the accumulator is split into 8 parts (each with 4 bits). Then, at each cycle, it is multiplied with the rescale factor, starting from the part with the lowest bits, and its partial product is shifted to the original position to be added to the resulting value. The process is repeated for eight cycles to compute all the parts for rescaling.

Second, the dataflow of the output stationary systolic array requires a minimal hardware extension for rescaling. The output stationary systolic array can be seen as an output value mapped to each PE, and a partial product is produced and accumulated in the same PE every cycle. Thus, an additional 1-bit shifter with control logic for each PE is enough for conventional hardware. For weight stationary design, we need a shifter in the accumulator (which resides outside of PE arrays) as well as in the PEs. Rescaling can be done as follows: 1) Weights are loaded in the group order, and the PEs at the boundary of each group are programmed to shift the partial product after MAC operations. 2) Each corresponding accumulator shifts its value before adding an incoming partial product. Note that although the above procedure requires slightly more changes in hardware than output stationary, it is still a small extension to existing hardware, and Tender can also be implemented on the weight stationary design.

The Vector Processing Unit (VPU) is a SIMD-style floating-point unit (FPU) that operates on vector elements. It performs scaling of incoming INT32 results from the Output Buffer (i.e., matrix multiplication results) into INT4/INT8 with an optional activation (e.g., $\mathrm{ReLU}$, $\mathrm{GeLU}$) before storing it back to the Scratchpad Memory. It uses calibrated bias and scale factors, which are computed before inference. VPU consists of 64 FPUs and internal vector registers for pipelining. There are additional registers to buffer scaling factors for quantization. Note that VPU also performs computation for the softmax and LayerNorms in the Transformer block.

The Execution Controller and HBM Controller operate independently during computation to keep the MSA busy. The HBM Controller handles data transfers between HBM2 and Scratchpad Memory, where the weights, inputs, and computed outputs are stored at INT4/INT8 precision. The Execution Controller sends an address to Scratchpad Memory and control signals (e.g., enable, rescale, and done signal) to the MSA.

As discussed in Section III-B, Tender needs certain channels to be processed before the others (e.g., channel order 1-0-3-2 in Figure 8). To avoid explicit reordering of the data layout in memory, which incurs costly read and write operations, we instead implicitly reorder channels through indirect indexing. Figure 8 shows how channels are sent to the MSA in the required order. Specifically, we first store the computation order of the channel indices in the Index Buffer, which is pre-determined at static time through calibration ( ① Program). The channel indices are reused over the entire row group to amortize memory access overhead, and the Index Buffer is double-buffered to hide memory access latency as it is on the critical path. While the HBM Controller also sends data from the off-chip memory into the Scratchpad Memory ( ① Transfer Data), the Execution Controller looks up in the Index Buffer and obtains the channel indices ( ② ) to generate an address for the target channel to load ( ③ ). Finally, channels are sent to the MSA in the order of the required computation ( ④ ).

As previously mentioned, all the inputs and weights are quantized into INT4/INT8 and stored in the Scratchpad Memory. Without the need for mixed precision storing, the memory access is aligned, and the addressing logic becomes simpler. The Output Buffer stores the computation result from the MSA in INT32 and sends them to the VPU for rescaling back to INT4/INT8, which is also highly banked to match the compute throughput of the VPU.

• •

  OLAccel [44] proposes outlier-aware quantization, which represents normal values in 4 bits and outliers in 8 or 16 bits. In addition to the 4-bit normal PEs, OLAccel implements outlier PEs; outlier PEs perform mixed precision computation (e.g., 16-bit $\times$ 4-bit).

• •

  ANT [21] implements a systolic array with a decoder attached to the edge of the array to support various formats including custom datatypes. The decoder converts datatypes into the exponent and integer. We implement an output stationary systolic array as it shows the best performance.

• •

  OliVe [20] also implements an output stationary systolic array with decoder logic to decode datatypes including outlier-victim pairs into the exponent and integer.

For an iso-area comparison, we synthesize the MAC units and accumulators of each accelerator and configure the number of PEs accordingly. We extend the baseline accelerators to use a 32-bit accumulator due to the large reduction length of matrix multiplications in LLMs. Also, we set the same memory bandwidth and on-chip buffer size for the accelerators, which are large enough to fully utilize the compute core. We compare speedups in LLMs with a batch size of 1. The input to output sequence length is set to 2048:1, following the speedup evaluation in prior works [65, 20]. For the generation stage, Tender still works and provides benefits by decomposing the activation. However, the under-utilization issue of most commercial accelerators (e.g., GPU, TPU) can be large as prior work points out [24]. To mitigate this, there are ongoing studies on batching decoding [68, 51], and Tender can work synergistically with those schemes.

Tender uses standard INT4 and INT8 representations, thereby enabling straightforward deployment to existing systems. To demonstrate the non-intrusive characteristics of Tender, we implement our quantization algorithm on NVIDIA GPUs. Figure 12 shows the normalized latency of Tender software and other quantization schemes in Section II-C. All schemes are implemented with CUTLASS INT8 GEMM kernel. We use OPT-6.7B for RTX 3090 and OPT-66B for A100 to saturate compute units. For small models on A100, we observe that per-tensor INT8 GEMM exhibits similar latency to FP16 due to compute underutilization and the relatively close tensor core throughput between INT8 and FP16. Latency and mean square error (MSE) are measured for each scheme with a sample from the query projection in Layer 16.

As shown in the results, Tender SW shows an MSE similar to the “per-channel” approach and provides slight performance benefits over FP16. However, it does not realize its full potential (i.e., similar to “per-tensor/-row”) due to the need of explicit dequantization on GPUs. The overall GEMM execution also takes longer due to the repetitive operations on smaller submatrices. In addition, INT GEMM kernels for tensor cores require 128-bit aligned memory access, which necessitates the padding of each subtensor (to the multiple of 16) before computation. Accelerators that support Tender can avoid the overheads and help achieve the full potential of Tender.

Variants of FP8 formats for DNN training/inference show good model performance [38]. Although using low-bit FP hardware could reduce the impact of outliers, it is more area/energy inefficient than integer compute units [10]. MSFP [9] uses a shared exponent to mitigate the inefficiency. As shown in Table VI, however, Tender shows better performance than MSFP. By default, MSFP12 uses an 8-bit shared exponent for 16 elements in a row, so the huge increase in perplexity likely comes from sharing exponents between outliers and others. We modify the MSFP12 to use the shared exponent for 8 elements in a column (MSFP12-OL). However, still Tender shows better model performance. This is likely because the intra-channel variance of the outlier channel is more precisely represented in the integer format than the MSFP. Thus, Tender achieves better performance than MSFP, while requiring much simpler hardware.

Shared Microexponents (SMX) [10] and the subsequent Microscaling (MX) format [41] are recently proposed number representations that employ multiple levels of scaling.²²2In this paper, we denote Shared Microexponents [10] as SMX to distinguish it from the Microscaling (MX) formats [41] endorsed by OCP. Similar to MSFP, they group elements in a block-based manner, but with two-level scaling, where the scaling factors are constrained to powers of two. SMX groups 16 elements with an 8-bit shared exponent, and two elements in a block form a subgroup to share a 1-bit subscale factor. Similarly, MX groups 32 elements, but each element has also its own exponent field in addition to the 8-bit shared exponent.

Table VII compares the accuracy between employing Tender and using SMX and MX formats on OPT-6.7B and LLaMA-7B for the lm-evaluation-harness tasks used in [10, 48]. For a fair comparison, we employ the same compute flow across the low-precision formats while quantizing matrix multiplications into low precision and keeping other element-wise operations as scalar floating-point formats as in [48]. The results show that Tender can provide better or comparable accuracy while it builds on standard INT4 representations. Note that SMX and MX need more customized compute units. For instance, each element in an MXFP block is essentially a floating-point number, requiring hardware that deals with FP computation.

The essence of Tender involves setting scale factors that are powers of two apart between the groups (e.g., the ratios of scale factors between $1^{\textrm{st}}$/$2^{\textrm{nd}}$/$3^{\textrm{rd}}$ immediate neighbor groups: $2^{1},2^{2},2^{3}$), which enables implicit rescaling with minimal overhead (1-cycle). MX formats, however, merely use some power-of-two scale factors, similar to MSFP. Thus, implicit rescaling by 1-bit shifting cannot be achieved by simply using MX. As discussed in Section III, the scale factor of Tender is also not limited to powers of two; it can be any real number. Additionally, MX formats follow a conventional approach of grouping adjacent elements, whereas Tender groups columns within similar ranges while considering ease of computation.

While we present the benefit of Tender based on the systolic array that employs the output stationary dataflow, it is not a strict requirement as discussed in Section IV-B. For the generation stage in LLMs, we may consider batching inputs only up to the number of rows of an output stationary systolic array due to compute and energy efficiency, while we can batch more inputs than the systolic array dimension for the weight stationary design. If there are ample batching opportunities, weight stationary can be more efficient since the output stationary design may incur idle time and additional energy due to repeated weight loading. Conversely, when batching is limited, such as by the memory size of large intermediate states (i.e., key-value cache), output stationary could be as efficient as weight stationary since it minimizes the movement of high-precision partial sums. Note that Tender can be employed in either case and provides benefits.

Although channels are decomposed into multiple groups, Tender can continuously compute the groups in MSA without a major interrupt. This is because rescaling can be done entirely in integer PEs (implicit) and only takes a single cycle (i.e., runtime requantization). During runtime, skewed channel groups are continuously provided with a 1-cycle rescale signal, aligned with the inputs between the groups (Figure 7(a)), and each PE individually rescales the accumulated value via 1-bit shifting; this feature is crucial as the systolic array takes as input a skewed matrix. The original computation in Figure 5(a) leads to large under-utilization of compute units. Tender does not suffer from this issue and preserves the reduction axis of the original input matrix, regardless of the number of groups or the group size for the matrix. Thus, during offline calibration, Tender only considers model performance to determine the number of groups, where perplexity does not decrease further as shown in Figure 9; it varies at some point due to noise. Each channel is then classified into the corresponding group. This calibration process naturally determines the size of each channel group.

Channel decomposition with explicit requantization leads to lower compute utilization due to the shortened reduction axis and also increases the number of FP operations. To understand the benefit of implicit requantization on Tender hardware, Figure 13 presents the end-to-end execution time when Tender employs either implicit or explicit requantization, which is normalized to per-tensor quantization (Base). Note that larger models (e.g., Llama-2-70B) generally need more number of groups to attain reasonable accuracy. The results show that Tender with explicit requantization greatly degrades performance, by up to a 1.74$\times$ slowdown over the baseline. Also, as the larger number of groups (e.g., 16) further reduces the reduction axis, we observe larger slowdowns compared to ones in the smaller number of groups. On the other hand, Tender (Implicit) offers almost the same execution time as the baseline. This is because there is only a 1-cycle requantization overhead for each group, so increasing the number of groups barely affects performance. By employing implicit requantization, Tender effectively minimizes the overhead associated with channel decomposition.