CiFHER: A Chiplet-Based FHE Accelerator with a Resizable Structure

Homomorphic encryption (HE) is classified into leveled HE (LHE) and fully HE (FHE) depending on the problem size. LHE supports only a limited number of operations (ops) and is usually used in conjunction with other cryptographic protocols [53, 42]. In contrast, FHE solely supports an unlimited number of ops. A defining feature of FHE is bootstrapping. The LWE problem [77] forms the cryptographic basis for state-of-the-art HE schemes, and relies on the error in encryption for its security. When performing HE ops on encrypted data, the magnitude of errors increases, threatening data integrity unless managed properly. Bootstrapping reduces the magnitude of errors, thus allowing more HE ops performed. In a typical use case, a bootstrapping op is inserted after every predetermined number of HE ops is applied to the encrypted data.

We briefly describe the CKKS FHE scheme, the main target of this paper. During CKKS encryption, a (vector) message composed of $n$ numbers is first packed into a degree-($N-1$) integer polynomial referred to as plaintext, which is an element of a cyclotomic polynomial ring $\mathcal{R}_{Q}=\mathbb{Z}_{Q}[X]/(X^{N}+1)$ with the degree $N\geq 2n$ and the modulus $Q$. All integer ops are performed modulo-$Q$ in $\mathcal{R}_{Q}$. A plaintext is encrypted into a ciphertext composed of two polynomials in $\mathcal{R}_{Q}$. We denote polynomials with bold characters (e.g., $\mathbf{v}$), and ciphertexts encrypting polynomials with brackets ($[\mathbf{v}]$). Then, the encryption of $\mathbf{v}$ can be formulated as $[\mathbf{v}]=(\mathbf{v}_{1},\mathbf{v}_{2})\in\mathcal{R}_{Q}^{2}$.

A high $N$ value around $2^{15}$–$2^{17}$ and $Q$ as large as $2^{1500}$ are utilized to meet the security constraints of FHE [7, 28]. To handle such a large $Q$, residue number system (RNS) is utilized [21]. $Q$ is set to the product of $L$ number of small primes ($q_{i}$’s), such that $Q=\prod_{i=1}^{L}q_{i}$. Then, each coefficient $c$ of a polynomial is represented by an $L$-tuple of small residues $(c\text{ mod }q_{1},\cdots,c\text{ mod }q_{L})$. Each row of the matrix corresponding to the prime $q_{i}$ is referred to as limb, which can be regarded as a polynomial in $\mathcal{R}_{q_{i}}=\mathbb{Z}_{q_{i}}[X]/(X^{N}+1)$. Using RNS, modulo-$Q$ ops between polynomials having large coefficients are replaced by a set of $L$ pairwise modulo-$q_{i}$ ops between limbs having small word-sized coefficients. Also, the modulus can change over HE ops such that $Q_{\ell}=\prod_{i=1}^{\ell}q_{i}$ is used at a particular moment. As a result, a polynomial at that moment is represented as an $\ell\times N$ matrix of residues.

In a client-server scenario of FHE, the server performs HE ops to manipulate the message encrypted in a ciphertext. Mult/add between two ciphertexts ($\mathtt{HMult}$/$\mathtt{HAdd}$) or between a ciphertext and a plaintext ($\mathtt{PMult}$/$\mathtt{PAdd}$) are some of the most frequently used HE ops. Also, it is often required to change the data order inside the message vector composed of $n$ numbers. Such data reordering can be performed with an $\mathtt{HRot}$ op, which evaluates the cyclic rotation of the message encrypted inside a ciphertext. Among these HE ops, $\mathtt{HMult}$ and $\mathtt{HRot}$ are the most computationally expensive due to complex key-switching procedures involved in them. Key-switching is a procedure to multiply a polynomial with evaluation keys ($\mathsf{evk}$s), which are special types of ciphertexts.

We defer a more detailed explanation of each HE op to prior work [22, 21, 38, 12] and instead break down HE ops into primary functions to discuss their computational aspects.

FHE computations are extremely expensive, mainly attributed to key-switching, which involves multiple (i)NTTs and BConvs. (i)NTT and BConv account for more than 80% of the total computations in CKKS [58, 55]. Thus, enhancing (i)NTT and BConv performance is essential to FHE acceleration.

F1 [80] is the first ASIC accelerator that reports bootstrapping performance. However, F1 targets small parameters of $N\!=\!2^{14}$ and $L\!=\!16$, with which it only partially supports bootstrapping. Subsequent accelerator proposals, BTS [58], CraterLake [81], and ARK [55], fully support bootstrapping by targeting larger parameters of $N=2^{16}$–$2^{17}$ and $L=24$–$60$. BTS places 2,048 processing elements (PEs) in a grid and connects them by global wires that span the whole chip. Each PE contains a butterfly unit for (i)NTT, a multiply-accumulate unit for BConv, and a multiplier and an adder for element-wise functions. A butterfly unit is capable of performing the butterfly ops required in (i)NTT, which are shown in Eq. 1. ARK and CraterLake develop upon F1’s design of placing $\sqrt{N}$ parallel lanes in a cluster (CraterLake refers to it as lane group) that feeds data into massive vector NTT units (NTTUs). A vector NTTU contains $\frac{1}{2}\sqrt{N}\log N$ butterfly units, placed in an organization directly reflecting the four-step FFT dataflow [10]. ARK deploys four NTTUs, and CraterLake 16.

To tackle the increased number of computations and size of the working set when using large parameters, CraterLake, BTS, and ARK all rely on enormous amounts of computational logic and on-chip memory capacity; the latter even reaches 256–512MB. These designs end up using massive chip resources, utilizing a monolithic die sized 373.6–472.3mm² and dissipating up to 317W of power.

MCM is a promising solution for scaling chips in the post-Moore era. By organizing a module with multiple small dies, referred to as chiplets, and connecting them using advanced packaging technologies providing high bandwidth on-package interconnects [41, 66, 17], MCMs reduce the area of an individual die. Because the design and fabrication cost of a die substantially drops as the die area is reduced [45, 84], MCMs can greatly reduce the total cost. With yield and technology scaling declining over chip fabrication generations, MCMs have become one of the few left options for scaling systems. Recent Intel [73] and AMD [69, 70] CPUs are notable examples of MCMs; multiple chiplets containing several CPU cores and cache memory are integrated into a single package using EMIB [66] or CoWoS [41] packaging technologies. Also, numerous domain-specific architectures utilizing MCMs have been proposed [84, 89, 43, 64, 88]. Moreover, MCMs have been realized at various area granularity, ranging from AMD’s high-end 74mm² die [70] to 6mm² die of Simba [84].

These recent trends motivate us to design an MCM architecture for cost-effective FHE acceleration. By using MCMs, we can enjoy the aforementioned benefits coming from a significant reduction in individual die areas. Also, this approach is highly scalable as we can reuse the same design for each chiplet and place multiple chiplets to facilitate developing various MCM FHE accelerators with different computational throughput and bandwidth required by the technology constraints and the purpose of use. Prior ASIC FHE accelerator proposals include high bandwidth memories (HBMs), already assuming the use of advanced packaging for connecting HBMs to the chip. The same packaging can be utilized to connect multiple chiplets with little additional cost [54]. We utilize the open specification of Universal Chiplet Interconnect Express (UCIe) [91] to design and evaluate the MCM FHE accelerator.

Prior accelerators either utilize little (PE in BTS) or big (cluster or lane group in F1, ARK, and CraterLake) cores without providing much flexibility in design. In contrast, we design the cores of CiFHER to have varying computational throughput, which can be adjusted at design time depending on the package configuration.

A core comprises a composable NTTU, a systolic BConv unit (BConvU), an automorphism unit (AutoU), an element-wise function unit (EFU), a PRNG $\mathsf{evk}$ generator (PRNG), a router and PHYs for NoP, and register files (RFs). We adopt the vector architecture utilized in F1, CraterLake, and ARK, where we place multiple parallel lanes in a core. Each lane has a dedicated RF space not accessible by the other lanes, which enables maximizing parallelism with low costs for synchronization and arbitration. The vector architecture has shown effectiveness in reducing the communication overhead and the RF bandwidth pressure, the major bottlenecks in FHE.

By effectively combining components from different vector accelerators with proper modifications, CiFHER provides an improved core design for FHE accelerators. In particular, we adopt the distributed RF, systolic BConvU, and AutoU structures from ARK, and adopt PRNG from CraterLake. A systolic BConvU is composed of multiple multiply-accumulate (MAC) units, forming an output-stationary systolic array.

To achieve a flexible core design, we aim to adjust configurations, particularly the number of lanes. While most functional units (FUs) readily adapt to lane count and throughput goals, the preceding vector NTTU presents a challenge. This critical vector core component is a substantial computing unit with its 2,048 butterfly units locked into a fixed organization.

The inflexibility of previous large-scale NTTU designs severely limits our design options, ultimately harming hardware efficiency. To manage the on-chip memory bandwidth demands of NTT computation, prior studies rely on deeply pipelined NTTU designs. These treat a length-N polynomial as an $\sqrt{N}\times\sqrt{N}$ matrix, performing the NTT in four steps: column-wise $\sqrt{N}$-point NTT, twisting, transposition, and row-wise $\sqrt{N}$-point NTT. With $N$ reaching $2^{16}\!-\!2^{17}$, this vector NTTU becomes bulky and expensive. Worse, its single-configuration design forces us to use large chiplets to accommodate it, regardless of the actual computational throughput needed. This leads to underutilized NTTUs, creating an imbalanced and inefficient design for MCMs.

We devise a composable NTTU supporting various configurations with the number of lanes adjustable from 16 to 256. We also regard a length-$N$ polynomial as a $\sqrt{N}\times\sqrt{N}$ matrix and perform $\sqrt{N}$-point (i)NTT along the rows and then along the columns in sequence, but additionally apply the four-step FFT dataflow [10] to each of the row- and column-direction (i)NTTs. We obtain a compact NTTU spanning $\sqrt[4]{N}=16$ lanes shown in Fig. 1 (simplified to $N=2^{8}$), which we refer to as the submodule of our NTTU.

Up to 16 submodules can be stacked to deliver higher computational throughput for (i)NTT, providing flexibility for our design space exploration. We devise a method to combine multiple submodules to have them collaboratively perform (i)NTT on a limb by performing a perfect shuffle [87] between the submodules. The perfect shuffle enables the rearrangement of dispersed data across various submodules by transforming it into a sequence that can be effectively transposed using subsequent quadrant swap units. An exemplar two-submodule NTTU configuration for $N\!=\!2^{8}$ performing NTT is shown in Fig. 1. We regard a length-$2^{8}$ polynomial as a $16\!\times\!16$ matrix. The first half of a submodule performs a four-step NTT on a row (data indices: 0–15/16–32). It starts from stride-1 butterfly ops and ends with stride-8 ops to perform 16-point NTT on a row. Then the buffering and shuffling steps in the middle redistribute the data between different submodules so that each submodule holds data elements with stride 16 in the second buffer; i.e., each submodule holds the elements of a column. Finally, the second half of a submodule performs a four-step NTT on a column (data indices: 0–240/4–244, stride 16) by starting from stride-16 butterfly ops and ending with stride-128 ops to perform 16-point NTT on a column.

Our composable NTTU also allows up to $\sqrt[4]{N}=16$ cores to perform (i)NTT of a limb together. In this case, data exchange between the collaborating cores occurs after the buffering and shuffling steps in the case of NTT.

We choose 32 bits as the word length to use in CiFHER as it is beneficial to use short word lengths for the efficiency of logical operations. While BTS and ARK utilize a word length of 64 bits, which has been traditionally used in CKKS libraries [32, 27, 20], F1 (32 bits), CraterLake (28 bits), and FPGA FHE accelerators (FAB [2] (54 bits) and Poseidon [95] (32 bits)) utilize shorter word lengths. Despite using a relatively short word length, we maintain high precision by the methods in [1] to utilize high scales ranging from $2^{47}$ to $2^{55}$. Table I tabulates our target parameters.

We utilize the word-level Montgomery reduction circuit proposed in [67] with further enhancements using the signed Montgomery reduction algorithm in [82]. The circuit is utilized for all modular reduction circuits in CiFHER, including EFUs, NTTUs, PRNGs, and BConvUs.

The technology constraints of NoP enforce CiFHER to be a tiled architecture connected by a mesh network. Due to the short channel reach of an advanced interface [91], a complex network topology that requires connections between distant nodes is less suitable for MCMs. Thus, we fix the topology to 2D mesh (see Fig. 2). Also, due to the high cost of NoP communication, we set the default bisection bandwidth of the entire package to 2TB/s, several to an order of magnitude lower than that of CraterLake (29TB/s) or ARK (8TB/s).

Prior FHE accelerators co-design the data mapping method with the network on chip (NoC) organization. They design complex NoC structures using direct connections between distant nodes with high bandwidth and uniform communication time. However, in an MCM architecture with a mesh network, communication latency between nodes is non-uniform, as distant nodes are separated by multiple hops. Lagging data from distant nodes causes others to wait idly, degrading the overall performance [84]. The low bandwidth of NoP exacerbates this problem. Therefore, we delve into data mapping methodologies to reduce the pressure of the network.

Prior data mapping methods can be classified into coefficient scattering and limb scattering. Coefficient scattering is a method of distributing $N$ coefficients of each limb equally to the entire computing units. By contrast, in limb scattering, each prime and the corresponding limbs are allocated to a specific computing unit. BTS and CraterLake use coefficient scattering, whereas F1 and ARK mostly use limb scattering. ARK temporarily switches to coefficient scattering during BConv, which we detail in §V-A.

Due to the conflicting data access patterns of HE ops, core-to-core communication is inevitable regardless of the data mapping method. (i)NTT and BConv, the two most dominant primitives, require different data mapping due to their data access patterns. For a polynomial, expressed as an $\ell\times N$ matrix, (i)NTT can be separately applied to each of $\ell$ limbs, favoring limb scattering. By contrast, as BConv is mostly a matrix-matrix mult with a $K\times\ell$ BConv table (see §II-C), data access is performed along the columns of a polynomial and makes coefficient scattering a better option. Thus, data exchange between cores is required either during (i)NTT (coefficient scattering) or BConv (limb scattering).

In an accelerator with many cores, both data mapping methods have limitations. As the number of limbs ($\ell$) vary over HE ops, it is difficult to distribute the limbs equally to cores without causing fragmentation issues in limb scattering. A core will be assigned fewer limbs than others depending on $\ell$ and wait idly for others to finish processing. In contrast, coefficient scattering is free from fragmentation issues because the degree of a polynomial ($N$) does not change. However, coefficient scattering has a scaling limit due to quadratic growth in the number of transferred packets with increasing cores. For example, during (i)NTT with $c$ cores, coefficient scattering results in all-to-all data exchange between the cores, requiring a total of $c(c-1)$ packets to be transferred. These issues become more severe as more cores participate in the distribution. Also, with more cores, additional performance degradation from long tail latency in lagging data occurs.

Therefore, in an MCM architecture with many cores and restrained NoP bandwidth, the combined use of limb scattering and coefficient scattering enhances performance, even with the increased total amount of transferred data due to performing data exchanges for both (i)NTT and BConv. We partition a polynomial in both limb and coefficient directions by dividing cores into limb clusters and coefficient clusters. A limb cluster is a set of cores each holding an evenly distributed subset of the coefficients of a limb. By contrast, the cores in a coefficient cluster partition the residues of a coefficient corresponding to different primes. Then, (i)NTT or BConv only incurs data exchange within each limb cluster or coefficient cluster, reducing the number of cores participating in the exchange. Thus, combining two data mapping methods suffers less from the aforementioned issues, leading to performance enhancement.

We introduce two generalized data mapping methods that combine coefficient scattering and limb scattering for mapping HE ops to a tiled architecture: dimension-wise clustering and block clustering. Dimension-wise clustering binds the cores on the same vertical line into a limb cluster and the cores on the same horizontal line into a coefficient cluster (or in the oppositice directions) on a mesh network. Fig. 3 shows an example of dimension-wise clustering on an $8\times 8$ mesh. This distribution method has the advantage that each data exchange process of (i)NTT and BConv can be performed without interference because data movement occurs in different axial directions. However, the configuration is fixed for a given mesh shape, potentially sharing the same limitations of prior mapping methods when the mesh shape is skewed.

Block clustering is a more generalized mapping method that places limb clusters in the form of blocks with an arbitrary size, and forms coefficient clusters with the cores in the same position within each block. Fig. 3 and Fig. 3 show configurations dividing an $8\times 8$ mesh into eight $2\times 4$ or four $4\times 4$ limb cluster blocks. Unlike dimension-wise clustering, various block clustering configurations are possible by changing the size of a block. We refer to the configuration on a $d_{x}\times d_{y}$ mesh using a $b_{h}\times b_{w}$ block size as $d_{x}\times d_{y}$-BK-$b_{h}\times b_{w}$. Dimension-wise clustering is simply denoted as $d_{x}\times d_{y}$-DW, which is indeed a special variant of block clustering; e.g., Fig. 3 can be expressed as $8\times 8$-BK-$8\times 1$. Also, limb scattering and coefficient scattering can be expressed as $d_{x}\times d_{y}$-BK-$1\times 1$ and $d_{x}\times d_{y}$-BK-$d_{x}\times d_{y}$, respectively.

Determining the data mapping method of CiFHER is challenging as it requires considering various factors that have conflicting effects on its performance. In addition to the trade-offs discussed in §IV-B, generalized data mapping methods reduce the maximum number of hops a packet travels by restricting its destination to adjacent cores within a cluster. Also, it is challenging to balance the limb clusters and coefficient clusters; an unbalanced configuration would result in fragmentation or severe performance degradation as either (i)NTT or BConv ops must wait for the other to finish due to the dependency in common CKKS use cases. We account for these challenges by developing an extensive simulation tool for CiFHER, which we use for the evaluation in §VI.

We propose a novel limb duplication algorithm, which duplicates the input limbs of BConv to reduce the amount of data communication caused by the redistribution of the output limbs during BConv. BConv requires gathering the residues from different cores. F1 uses parameters which allows performing BConv in parallel for each of the $\ell$ limbs without redistribution of input data ($\ell$ limbs) before BConv [20]. However, F1’s method produces massive $\ell^{2}$ output limbs, which need to be redistributed among cores. More recently proposed ARK adopts parameters that make BConv produce fewer limbs (roughly $\ell\cdot\beta$ limbs for a small integer $\beta$), which also significantly reduces the amount of computation [38]. However, as it becomes harder to parallelize BConv, ARK switches to coefficient scattering prior to BConv and switches back to limb scattering after BConv, inducing data redistribution of both input and output data of BConv. Fig. 4 shows the estimated amount of data transfer required for key-switching depending on $\ell$ (the number of primes) when using ARK’s method. As the number of input limbs is usually smaller than the number of output limbs, the output limbs of BConv account for most of the data transfer regardless of $\ell$.

To eliminate the data transfer required for the redistribution of generated limbs, limb duplication duplicates input limbs and broadcasts them to all the cores in a coefficient cluster so that each core holds the entire polynomial ($\ell\times N$ matrix when only considering limb scattering). Then, each core performs BConv with a piece of the BConv table ($K\times\ell$ matrix); e.g., a core in charge of the primes $p_{1},p_{5}$, and $p_{9}$ would only multiply the 1^st, 5^th, and 9^th rows of the BConv table with the polynomial to obtain BConv results. As the output limbs already belong to the right owner, data redistribution after BConv is unnecessary.

However, limb duplication requires additional data transfer for broadcasting input limbs. The overhead of broadcasting varies depending on the ratio of the number of input limbs and the number of output limbs and the data mapping methodology. We can formulate the reduction in the number of transferred limbs compared to the ARK’s method as in Eq. 2. $\mathtt{broadcast}_{\text{overhead}}$ denotes the ratio of data transfer required for broadcasting compared to that required for even distribution and has a value greater than one.

Eq. 2 must be larger than zero to make limb duplication beneficial. We selectively use limb duplication only when this condition is satisfied for each BConv. As there are much more output limbs than input limbs in general, limb duplication is usually beneficial for the reduction of communication.

We adopt the state-of-the-art CKKS algorithms mostly from an FHE CPU library, Lattigo [32]. We combine the bootstrapping algorithms from [19, 13]. There are also accelerator-specific algorithms, which offers trade-offs between computation and off-chip memory access. They target $\mathsf{evk}$s and plaintext polynomials, which are loaded from the off-chip memory and are large in size, possibly inducing the off-chip memory bandwidth bottleneck. We explain how such algorithms can (or cannot) be applied to CiFHER.

We utilized four representative FHE workloads that have been utilized for evaluating the previous accelerators.

We compare CiFHER with CraterLake and ARK using their reported performance and power values under 128-bit security constraints. For 12/14nm CraterLake, optimistic compensations for the difference in the technology node (14nm to 7nm [72]) were applied, which we refer to as CLake+.

Among the default configurations, 4-core CiFHER using dimension-wise clustering and limb duplication showed the best energy-delay product (EDP), whose values are shown with the 1$\times$ equi-EDP lines in Fig. 5. We limited the number of limb clusters to eight to avoid severe fragmentation issues.

When we integrated the 4 cores into a single monolithic die and connected the cores in the NoC with 2$\times$ higher bisection bandwidth (1-chip in Fig. 5), additional performance and EDP enhancements were achieved. 1-chip CiFHER resembles ARK with datapath reduced to 32 bits and showed a similar EDP with the original 64-bit ARK except for HELR1024. ARK performs inferior in HELR1024 because we reordered the HE ops to further enhance the reuse of $\mathsf{evk}$s in the memory-bound region, which ARK identifies as a bottleneck of HELR.

However, the fabrication cost of 1-chip CiFHER or ARK is significatly higher than chiplet-based CiFHER for reasonable production volumes (see Fig. 6), mainly due to the high non-recurring engineering (NRE) cost stemming from the large chip size. 1-chip CiFHER (respecitvely, ARK) has 1.22$\times$ (2.22$\times$) and 1.37$\times$ (2.48$\times$) higher NRE compared to 4-core and 16-core CiFHER, Although NRE can be amortized for large production volumes, the cost of 1-chip CiFHER becomes on par with that of 4-core CiFHER when one million packages are produced, which is a huge amount for domain-specific accelerators. Meanwhile, ARK is always expensive than 4-core CiFHER due to higher raw chip fabrication cost and yield loss.

Under the fixed computational throughput and bisection bandwidth of the package, as we split the chip and utilize smaller cores, performance and efficiency inevitably decline because more NoP data transfer is required. Thus, the area constraint is a decisive factor in an MCM FHE accelerator design. If the cost budget allows, as in recent AMD server CPUs (74mm² per compute die [70]), utilizing four cores would be best. Table III shows that 4-core CiFHER enhances efficiency in terms of energy-delay-area product (EDAP) [63] with comparable performance to prior monolithic FHE accelerators due to significantly lower power dissipation (see Fig. 5). Compared to ARK, 4-core CiFHER performs 1.15$\times$ slower but reduces EDAP by 2.03$\times$ in geometric mean (geomean) of the workloads. Compared to CLake+, it performs 1.34$\times$ faster and results in a 2.27$\times$ EDAP reduction. By contrast, if we can only afford dies as small as recent domain-specific MCM accelerators, such as NN-Baton [89] (2mm²) and Simba [84] (6mm²), 16-core or 64-core CiFHER needs to be utilized.

Our data mapping and limb duplication minimize the performance and efficiency degradation of many-core configurations. The best 16-core and 64-core configurations respectively perform only 1.03$\times$ and 1.18$\times$ slower in geomean than 4-core. Compared to 4-core, EDP increases to 1.23$\times$ (16-core) and 1.94$\times$ (64 core), and EDAP increases to 1.35$\times$ (16-core) and 2.67$\times$ (64-core) in geomean using our optimized configurations, whereas a naïve configuration (e.g., 64-core using limb scattering with ARK’s method) would require 17.1$\times$ higher EDP and 23.6$\times$ higher EDAP in bootstrapping.

Meanwhile, due to the skewed organization, 8-core (2 × 4) and 32-core (4 × 8) configurations suffer from an imbalance in NoP throughput between the horizontal and vertical directions, increasing the tail latency. As a result, performance drops severely, making these configurations less attractive.

We estimated energy, delay, and EDAP of CiFHER while incrementally applying block clustering (4$\times$4-BK-2$\times$2 and 8$\times$8-BK-4$\times$4) and limb duplication (see Fig. 7). Limb scattering of ARK shows 1.44–1.53$\times$ (4$\times$4) and 2.89–3.06$\times$ (8$\times$8) slow execution time compared to block clustering. Coefficient scattering of CraterLake is 1.10–1.13$\times$ faster than our method for the 4$\times$4 configuration, whereas it is 2.76–2.81$\times$ slower for 8$\times$8. The data exchange pattern during limb scattering is not all-to-all as we can group some limbs together to have data transfer occur within each group. By contrast, coefficient scattering requires all-to-all data exchange and thus scales inferior to limb scattering. Coefficient scattering consumes 2.07$\times$ more energy on NoP and shows 4.66$\times$ higher (worse) EDAP than our mapping in geomean for 8$\times$8. Yet, all-to-all data exchange between 16 cores is manageable, making coefficient scattering faster than our mapping for 4$\times$4.

Fig. 7(c) compares the amounts of data movement between the cores of CiFHER, with and without limb duplication. Limb duplication consistently eliminated 18–22% of the data movement between the cores. When applying limb duplication to our mapping in the 4$\times$4 configuration, we obtained 1.10$\times$ delay and 1.16$\times$ EDAP reductions in geomean, and the delay and EDAP gaps with coefficient scattering were also reduced to only 1.3% and 2.1% in geomean.

To assess the efficiency of composable NTTUs, we also measured delay, energy, and EDAP of CiFHER during bootstrapping with different single-core computational throughput, represented by the number of lanes. We either use composable NTTUs or alternative NTTUs, which extend upon prior NTTUs by interpreting a length-$N$ polynomial as a three-dimensional vector ($N\!=\!\text{\#}_{lane}\times\text{\#}_{lane}\times N_{rest}$) and performing a sequence of $\text{\#}_{lane}$-point, $\text{\#}_{lane}$-point, and $N_{rest}$-point NTTs. Similar to prior NTTUs, twisting and transposing steps are appropriately inserted in between NTT steps. Alternative NTTUs can also adjust the computational throughput according to the number of lanes ($\text{\#}_{lane}$); however, whereas data exchange in composable NTTUs mostly occur in between adjacent lanes, alternative NTTUs require data exchange with distant lanes, increasing overall energy and area due to long wires.

Composable NTTUs increase the efficiency of CiFHER for many-core configurations by enabling a compact MCM architecture with a balance between computational throughput and available data movement bandwidth. In the 2x2-DW configuration, where data transfer through NoP is relatively less heavy and high computational throughput is required for each core, energy and delay increased when prior NTTUs were replaced with composable NTTUs with lower throughput (see Fig. 9(a)). Specifically, the delay (respectively, energy) increases by $1.43\times(1.29\times)$ and $2.7\times(1.92\times)$ as the number of lanes in a core decreases from 256 to 128 and 64 by utilizing composable NTTUs. Conversely, in the 4x4-BK-2x2 configuration that demands substantial NoP data transfer, halving the number of lanes by using composable NTTUs do not increase the delay by a large margin. Using 128 lanes leads to a slight 6% increase in delay and improves the overall EDAP by 12% due to smaller chiplet area.

Composable NTTUs are more efficient in terms of energy and area compared to alternative NTTUs. In Fig. 9(b), the reduction to 128 lanes using alternative NTTU also leads to an around 2% enhancement in EDAP. Nevertheless, due to the substantial wire requirements of alternative NTTUs, the energy consumption and area footprint are larger by $1.05\times$ for both when compared to those of composable NTTUs, culminating in an EDAP value $1.10\times$ larger than that achieved with composable NTTUs.

The effects of limb duplication differ by each configuration. We selected the three best-performing configurations for each number of cores in Fig. 5(a) and compared the performance of bootstrapping with and without applying limb duplication. Fig. 8(a) shows the results. Limb duplication improved bootstrapping performance by up to 31% for 4$\times$8-BK-4$\times$4 but rather caused 28% performance degradation for 8$\times$8-BK-2$\times$4.

Although limb duplication makes fewer limbs exchanged, the bursty nature of data broadcasting causes network contention and introduces additional latency. The overhead of broadcasting in limb duplication is affected by the arrangement of limb clusters. Only 8$\times$8-BK-2$\times$4 in Fig. 8(a) has a coefficient cluster composed of eight cores; coefficient clusters in the other configurations consist of one to four cores. The large number of cores in a coefficient cluster greatly increases the broadcasting latency, degrading performance.

As limb duplication is effective for reducing the NoP pressure, increasing the NoP bandwidth reduced the performance gains from limb duplication (see Fig. 8(b)). Limb duplication enhanced the performance of the baseline configurations by 7% in geomean. With the NoP bandwidth doubled, limb duplication rather decreased performance by 3% in geomean, and the maximum performance gain observed for 4×8-BK-4×4 was reduced to 14% from 31% in the baseline.

When we intensified the NoP bandwidth bottleneck by doubling computational throughput, performance gains from using limb duplication increased (see Fig. 8(c)). In these configurations, limb duplication enhanced performance by 14% in geomean, and the maximum performance gain reached 43%.

To evaluate the scalability of CiFHER, we estimated the performance of each workload while increasing the number of cores in Fig. 10. We set the internals of a core fixed to an eight-NTTU-submodule setting (128 lanes). Limb duplication and $d_{x}\times d_{y}$-BK-$\nicefrac{{d_{x}}}{{2}}\times\nicefrac{{d_{y}}}{{2}}$ block clustering were used for a $d_{x}\times d_{y}$ package configuration. We fixed the HBM bandwidth and the bisection bandwidth of a package. Thus, the bandwidth of a network edge decreases as the number of cores increases.

The limited bandwidth of NoP restricts the scalability of CiFHER. When we increased the total computational throughput by increasing the number of cores from 4 to 16, 1.99–2.11$\times$ of speedups were achieved for the workloads. Increasing from 16 to 64 only resulted in a 1.07$\times$ speedup. Therefore, increasing the number of cores without consideration for proper sizing of the core has an adversarial effect on efficiency. Meanwhile, skewed configurations (8-core and 32-core), which already suffer from the NoP bandwidth imbalance, showed suboptimal performance.

HBM bandwidth becomes the next limiting factor for scalability when NoP bandwidth is sufficient. Doubling NoP bandwidth mitigates the NoP pressure and improves the performance of the 16-core configuration by 1.34$\times$ in geomean. Scalability slightly improves as 4-to-16 and 16-to-64 number-of-cores transitions result in 2.23$\times$ and 1.13$\times$ speedups. However, the degree of speedups still falls largely behind the increase in the number of cores as the hardware becomes bottlenecked by the HBM bandwidth, whose utilization reaches 75–80% in the 64-core configuration.