Coherence Attacks and Countermeasures in Interposer-based Chiplet Systems

Industry has widely adopted a work mode where a design house and partners carry out IC design and verification in-house and then outsource fabrication and testing to off- shore facilities to provide access to advanced technologies, reduce production costs, and streamline the time to market [82]. However, it raises concerns regarding the trustworthiness of the outsourced fabrication facilities, which may seek to insert security vulnerabilities in general or hardware Trojans in particular [40].

The threat vector posed by untrusted fabrication facilities implies that the ICs they manufacture are untrustworthy, posing a security challenge for modern systems in multiple ways. First, any hardware security feature embedded in an outsourced IC may no longer offer the desired protection, presenting a profound challenge. Second, a modern system may be comprised of chiplets with varying trustworthiness. Any malicious chiplet behavior may compromise the entire system due to its interconnected nature.

The interposer technology can avoid such complications since an interposer can be fabricated separately in a trusted facility and embed security features. We propose an interposer designed to constitute a hardware-enforced root of trust that can be built upon to ensure a system’s trustworthiness.

Hardware Trojans can lead to catastrophic system security failures. For example, Bidmeshki et al. [9] study an attack scenario where a hardware Trojan renders the cryptography subsystem vulnerable, Khan et al. [42] demonstrate Trojans that leak data from processor caches, and Kim et al. [45] introduce Trojans inject malicious coherence messages to create a denial-of-service attack. In Section 5, we identify potential attack vectors for Trojans to maliciously modify data owned by a process operating in a different chiplet than a Trojan-compromised chiplet.

Detecting Trojans is challenging as chiplet-based systems contain multiple complex IPs sourced from various vendors. Testing a chiplet’s functionality may occur during the manufacturing or integration stage, which requires a reference model or device [8]. However, if the 3rd party IP’s source is untrustworthy, the reference model may incorporate the Trojan, or the IP may camouflage the Trojan as a correct implementation. For this work, we assume that an attacker infiltrates some stage within the design or manufacturing process to target the system’s coherence mechanisms. Attackers can conceal a Trojan by designing it to trigger only under rare or specific conditions. For example, the Trojan we describe in Section 5.2 only activates when it observes references to a specific address. These properties make Trojans difficult to detect as simply testing the chiplet’s functionality does not guarantee that the Trojan will be active at an observable output node.

Prior art focuses on infiltrating the NoC of a target design to cause deadlocks [45], leak information [42], or disrupt security features [9]. However, NoC-based attacks require the Trojan to attack NoC packets directly, limiting the Trojan to only packets traversing a particular path in the NoC. Prior attacks would not work in a 2.5D integrated system because attacking chiplets are not on the path between victim chiplets and the memory controller.

Prior art for secure NoCs assumes malicious activities arise from connected components or the network fabric. Fiorin et al. [29] propose security features for policy-based message checking against untrusted components. Selected works focus on securing the system through encryption/decryption of packets exchanged through NoC fabrics [25, 31]. Kinsy et al. [46] propose organizing secure and non-secure software/hardware entities as tenants, configuring NoC routers to exchange messages securely. The amount of key exchanges required to isolate nodes/tenants incurs high latencies, inhibiting scaling.

Nabeel et al. [59] propose an interposer-based architecture where security modules monitor the interconnect fabric at the bus level to block transactions that violate memory access policies. Their design represents a relevant first work toward secure 2.5D integration and the viability of interposer designs, but it has several limitations. First, the authors consider an overly simplistic architecture, ignoring that state-of-the-art 2.5D designs are fully memory-mapped and cache-coherent, opting to focus instead on physical design and layout. We find that addressing the coherence model is critical to providing system-level security. Second, the authors overlook new security challenges arising from interposer designs. Critically, their design would fail to hinder recent coherence-oriented snooping attacks that do not violate memory access policies/permissions. We also find the transaction monitor (TRANSMON) they propose is largely impractical, as their design requires an entry in their policy register space (PRS) tables for each and every allocated memory region in the physical memory space. There is no discussion in that work of how large that table must be, what to do when it overflows, and how many cycles it would take to access. Furthermore, their work does not perform an architectural evaluation and does not provide a performance analysis, which does not allow us to directly compare with them.

For most prior art, networks are not secure-by-construction, requiring high-overhead solutions such as key-based security [16, 28, 68], model checking [12, 66], or structures to verify traffic patterns [53, 62, 66]. While prior work has proposed packet-checking schemes, e.g., [69], the underlying defense mechanisms often only address a single attack vector [11, 68] and/or fail to address the coherence system’s exploitability [28, 66, 69].

While these works check the message’s memory operation, they do not differentiate between coherence message types and how coherence messages can be exploited beyond simple read or write traffic. Further, most prior art assumes, often implicitly, trusted manufacturing of the whole system. Outsourced supply chains challenge such an assumption; 2.5D integration using chiplets from various vendors only exacerbates these concerns.

Intel’s SGX provides trusted execution environments (TEEs), called enclaves [21, 58]. Enclaves prevent unprivileged access to secure data during security-sensitive execution. SGX maps protected memory pages to reserved memory regions in which a hardware encryption module encrypts the pages. ARM’s hardware-enforced TEE isolates secure execution from untrusted software [54], leveraging a normal OS alongside a secure OS. The latter can access the full range of a device’s peripherals and memory. In contrast, the normal OS only has access to a subset of peripherals and memory to prevent unauthorized access to sensitive resources.

Recent works have shown vulnerabilities in SGX due to programming errors and untrusted software [43, 64] as well as speculative execution [13, 17, 49, 70]. In general, TEEs are prone to vulnerabilities from architectural, implementation, and hardware issues [14].

With the proposed CMCs, we monitor and validate all incoming packets to the interposer. Figure 4 depicts the CMC embedded in a router of the interposer NoC. The CMC is a pipelined, non-blocking structure that monitors messages traversing the physical links prior to entering the virtual channel buffers within the routers.

Operation Basics: As packets enter the network, they are converted into 64b flits; then, they move onto the interposer layer and are broken down to be analyzed by the CMC. The CMC requires one pipeline stage to analyze a 64b flit, i.e., to extract the fields required by the PCM and APU to enforce region protections and prevent malicious coherence messages. The subsequent pipeline stages are then required to lookup the appropriate permissions in the APU and respond to request if necessary (the CMC-1 has two pipeline stages, while the CMC-2 requires three as discussed below). This design provides low power and latency overhead as its behavior is pipelined and does not heavily impact the flow of packets into the network, as discussed in Section 7.2.

Each CMC has two components described as follows.

Packet Checker/Modifier (PCM): The PCM monitors and modifies cache coherence messages as needed. The proposed system follows standard shared-memory semantics with memory accesses creating coherence messages during communications between cores, IPs, I/O buses, and memory. Thus, the PCM operates on coherence messages to check addresses and permissions.

Address Protection Unit (APU) Table: This is a direct-mapped, SRAM-based look-up table with entries for each memory region and associated per-chiplet permissions. As outlined in Section 6.2, the physical memory is partitioned into multiple fixed-size regions; each region has a corresponding entry in the APU. This is compatible with the page-level protections enforced by the TLBs, but coarser-grained, allowing for a more overhead-efficient implementation.

Recall Figure 1, depicting CMCs embedded in the secure interposer-based system. We denote CMCs connected to the physical links for chiplets as “CMC-1” and denote those connected to the physical links for MCs as “CMC-2.” CMC-1 monitors and verifies coherence messages entering the interposer, whereas CMC-2 modifies specific coherence messages at the MC directories (i.e., to counter passive-reading threats on broadcast messages). Router-to-router connections exclusively within a trusted interposer do not require monitoring.

CMC-1: Prevents the attached chiplet from injecting malicious coherence messages into the system that violate the shared-memory organization (see Section 6.2). The PCM monitors all traffic from the attached chiplet based on the physical address of the packet. This physical address is compared against the per-region permissions stored in the APU table (described further below). If a message is of an allowed type to an allowed memory region for the given chiplet (e.g., a GETX to a read-only memory region it owns), the message may proceed into the interposer NoC. Otherwise, if the packet is rejected, a dedicated security signal realized as a machine-check exception is thrown, and system execution stops.¹

CMC-2: Prevents the broadcast of coherence messages to chiplets that are not permitted to access the related memory region. MOESI Hammer [19] does not maintain per-core sharing information, hence specific message requests cause an MC directory to broadcast the request to all cores. The cores then respond if that core shares the cache block, raising concerns of passive reading/snooping.

To prevent snooping, the PCM uses the APU table to determine whether a given broadcast message is allowed to access the referred memory region. If the chiplet does not have access, the broadcast message is converted into an appropriate direct broadcast message directed only to the original requester. For example, an invalidation request for data owned by Chiplet 0 does not need to be observed by any other chiplet.

Note that this approach is legal within the coherence scheme: if a chiplet does not have access to a memory region, then its caches cannot contain lines associated with that region. This allows the CMC-2 to divert broadcast messages from the directory and prevent snooping safely.

Also note that we do not include an operational diagram for the CMC as it is a direct look-up into the APU table, realizing a comparison of the message fields to the chiplet’s ID and allowed message types. A page table and interaction with the memory management unit are not required because the addresses used by the CMC and coherence messages are physical. See next for more details on the APU table structure.

The APU table is a lookup table containing entries describing the access permissions for applications running within a respective chiplet. The access permissions are determined by a secure OS running exclusively within the active interposer, independently of the regular OS running on the chiplets. The permissions are programmed into the APU tables during runtime, as outlined in Section 6.2.

Figure 5 the APU table. Each entry requests one memory region, with two bits allocated to represent access permissions of applications running in a chiplet: no permissions (“00”), read-only permissions (“01”), or read/write permissions (“11”) with “10” unused.

When the PCM intercepts a packet, the upper bits of its physical address are extracted and used to index into the APU table. The related entry is read and handed back to the PCM to compare the request type, requester ID, and destination ID against the permission levels in the APU table entry.

Permissions in shared memory systems are typically defined per physical page by the OS during memory allocation. For our system, enforcing per-page permissions in a CMC poses several challenges. Page-level tracking requires a TLB-like structure to cache translations [76]. The support required to maintain the structure in coherence with the full system’s page table significantly increases hardware complexity and performance overhead.

We argue that a page-level implementation at the interposer is excessive in a system of relatively few and coarse-grained chiplets. Instead, we partition the physical memory into coarse-grained memory regions, similar to prior art [48, 51, 77, 78]. Here we aim for the “sweet spot” between too coarse-grained, where few memory regions are available and fragmentation consumes capacity, versus too fine-grained, where the APU table cannot hold regions in excess without incurring high access latency or placing entries in a backstore. We find that a total number of memory regions between 4\(\times\)–8\(\times\) the number of chiplets is sufficient, allowing for diverse private and shared memory regions without too much fragmentation. For our design with eight chiplets and 4 GB of physical memory, the APU has 64 entries, representing 64 MB (16,384 pages).

Each memory region is designated as read- or write-able independently to any given chiplet, with permissions updated as needed. Data private to a single chiplet is placed in a region (or set of regions) only accessible by that chiplet. A page shared across multiple chiplets is assigned to a memory region the given chiplets are allowed to access.

The interposer includes a trusted co-processor to host a secure and trustworthy OS for system-level memory allocation. It is the role of this OS to ensure that processes that must be protected from each other are placed on different chiplets with different memory regions assigned to them. The OS threads running on the chiplets must delegate their page allocation to the OS running on the interposer. The code in the chiplet’s OS threads must be extended to call the interposer’s secure OS for all page allocation.

The secure OS provides the chiplet’s OS threads with a physical page based on the chiplet’s current memory regions and access permissions. The API between the OS threads running on the chiplets and the secure OS is composed of two functions, APU_ALLOCATE and APU_DELETE. The APU_ALLOCATE interface function is called by the chiplet’s OS when access to a new physical page is required. Similarly, when the chiplet’s OS threads are ready to free a physical page, they call the APU_DELETE function to return that page to the secure OS.

Initial memory partitioning and permission setting occurs during the initial soft page fault on a virtual page. An API call from the OS updates region allocations and permissions, similar to Intel’s SGX page allocation model [58]. After a page fault, the following occurs:

Since the APU table update occurs on the trusted interposer, chiplets not involved in the allocation process are unaware of the memory allocation request. Critically, a malicious chiplet that somehow gains knowledge of the request cannot access the region since the new permissions are set in the APU tables before other malicious operations target the memory region.

Regardless of the interposer’s NoC topology, CMCs are emplaced at the interface between chiplets or MCs and the active interposer. However, the physical link’s width impacts the CMC design and its logic. In our implementation and evaluation, the link width is 128 bits within chiplets and 64/128 bits within the interposer.

In MOESI Hammer, every control message fits within a single 128-bit flit [19]. When a flit enters the interposer, it is broken down into one/two flits which are analyzed in the CMC logic over one/two clock cycles, depending on the 64/128-bit width of the interposer link. In the case of 64-bit links, depicted in Figure 6, we dedicate the first cycle to extracting the control parameters from the head and the second cycle to extracting the address for the cache block being accessed. The CMC logic is similar for request and response messages, as both cases require analyzing the first two flits.

The system’s cache coherence protocol affects the CMC design and logic as the CMC must analyze the coherence message fields. MOESI Hammer’s responses either a control or data message; a control response follows the same flit structure as a request, whereas a data response has additional flits containing 64 bytes of data.

Based on the message type and identified threats (Section 4), the CMC must analyze specific key parameters; these are highlighted in Figure 6. The PCM extracts the parameters and compares them with the permissions set in the APU table. Since an attacker may exploit either request or response messages, both message types must be analyzed.

First, coherence messages are converted into network packets by the chiplets’ NIs. However, these packets are not guaranteed to adhere to the rules of the network and coherence protocols. For example, a Trojan may fabricate an invalid message type, yielding undefined, possibly vulnerable behavior. Second, messages corresponding to particular virtual networks (VNs) must follow a specific, limited set of requester/destination IDs and message types.

To address both aspects, the PCM checks the possible field values to verify the legality of messages. Since these checks are orthogonal to memory-region permission checking, they are performed in parallel and incur no extra delay.

We design the PCM module with three pipeline stages for lookup, packet checking, and modification. The third stage is excluded in CMC-1 instances as these only monitor packets on ingress to the interposer. An APU table is embedded in each of the twelve routers (8 for the chiplets, 4 for the MCs) in the interposer, imposing a total memory footprint of only 1.5 KB. More specifically, an APU table requires two bits for identifying each chiplet’s permissions, and there are 64 table entries; an APU table requires 1,024 bits.

Our scheme addresses the threat model (Section 4) as follows:

Passive reading: This threat is prevented by rerouting broadcast messages as they enter the CMC-2 at the interposer/MC boundary. Broadcast messages from the directories are converted into negative acknowledgments (back to the requester) for chiplets without permissions to the message’s memory region. In all cases, the CMC will throw a security check exception and halt execution if an attack is detected.

Masquerading: Every CMC-1 is programmed with the range of IDs expected in each coherence message’s requester ID field. For example, in Figure 1, the CMC-1 in router 72 expects requester IDs in the range of 0–7 and will reject messages with an ID outside this range, as discussed in Section 6.3.3.

Modification: This is detected by comparing a message type, such as GETX/GETS, with the access permissions in the APU table.

Diversion: This threat is detected by checking the destination ID and the message type. Only specific message types can have other cores as the destination ID. This, along with the memory region permissions in the APU table, allows us to detect any malicious diversion of messages.

Complex: Complex attacks, like the Forging Attack demonstrated in Section 5.2, are similarly covered by our technique as they are composed of multiple simple attacks.

Software Threats: Our design prevents unauthorized accesses to memory regions due to privilege escalation or exploitation using the mechanics described above for hardware threats. Importantly, since coherence messages are generated by hardware, a solely software-driven attack cannot engage in these threats through packet manipulation without malicious hardware intervention.

To test the system’s ability to counter all threats, we inject tailored, malicious coherence messages at the network interface of cores. We verified that respective check exceptions are thrown for all attacks listed above – no malicious packets enter the interposer NoC before the system halts. We also verify that the complex Forging Attack is rendered incapable of executing its payload in our secure system.