Dynamic Binary Translation for SGX Enclaves

Intel SGX protects the enclave by enforcing strict isolation at several points of interactions between the OS and the user enclave code. We outline five restrictions that the design of SGX imposes:

Restrictions R1–R5 are a systematic way to understand the incompatibility created by design choices in SGX with the OS and application functionality. Table 1 summarizes the effect. All of the restrictions apply to SGX v1 [12]. \( R1 \) – \( R4 \) apply to SGX v2 [58, 89], but \( R2 \) is relaxed, because SGX v2 enables dynamic page creation and permission changes. Thus, for the rest of the article, we describe our design based on SGX v1. We discuss their specific differences to SGX v2 and its ramifications in Section 7.2.

R1. Since SGX spatially partitions the enclave memory, any data that is exchanged with the OS requires copying between private and public memory. In normal applications, an OS assumes that it can access all the memory of a user process, but this is no longer true for enclaves. Any syscall arguments that reside in enclave private memory are not accessible to the OS or the host process. The enclave has to explicitly manage a public and a private copy of the data to make it accessible externally and to shield it from unwanted modification when necessary. We refer to this as a two-copy mechanism. Thus, \( R1 \) breaks functionality (e.g., system calls, signal handling, futex), introduces non-transparency (e.g., explicitly synchronizing both copies), and introduces security gaps (e.g., TOCTOU attacks [31, 43]).

R2. Applications often require changes to the size or permissions of enclave memory. For example, memory permissions change after dynamic loading of libraries (e.g., \( {\tt dlopen} \) ) or files (e.g., \( {\tt mmap} \) ), executing dynamically generated code, creating read-only zero-ed data segments (e.g., .bss), and for software-based isolation of security-sensitive data. The restriction R2 is incompatible with such functionality. To work with this restriction, applications require careful semantic changes: either weaken the protection (e.g., read-and-execute instead of read-or-execute), use the two-copy mechanism, or rely on some additional form of isolation (e.g., using segmentation or software instrumentation).

R3. SGX has no mechanism to allow two enclaves to share parts of their private memory directly. This restriction is incompatible with the synchronization primitives like locks and shared memory when there is no trusted OS synchronization service. Keeping two copies of a shared lock breaks its semantics and creates a chicken-and-egg issue: how to synchronize the two copies without another trusted synchronization primitive.

R4. When applications demand new virtual address mappings (e.g., \( {\tt malloc} \) ), the OS adds these mappings. Normally, applications can ask the OS to map the same physical page at several different offsets, either with same or different permissions—for example, say when the same file is mapped as read-only at two places in the program space. On SGX, however, the same PA cannot be mapped to multiple enclave VAs. Any such mappings lead to memory protection faults.

R5. SGX starts or resumes enclave execution only from controlled entry points, i.e., which have to be statically identified virtual addresses. Entry/exit points such as those via system calls are feasible to identify statically. However, there are several unexpected entry points to an application when we run them unmodified in an enclave (e.g., due to exception-generating instructions or faulting memory accesses). Determining all potential program points across the enclave boundary is not straightforward. When the control re-enters the enclave after an exit (e.g., an OCALL), SGX requires that the program execution context at the time of exit and re-entry should be the same. This does not adhere to typical program functionality. Normally, if the program wants to execute custom error handling code, say after a divide-by-zero ( \( {\tt SIGFPE} \) ) or illegal instruction ( \( {\tt SIGILL} \) ), it can resume execution at a handler function in the binary with appropriate execution context setup by the OS. On the contrary, SGX will resume enclave execution at the same instruction and same context (not the OS setup context for exception handling), thus re-triggering an exception if naively handled.

It is possible to enable compatibility with exceptions by using the SGX features for exception handling [59]. However, it requires inserting logic to update exception handling data structures (e.g., SSA, enclave stack for EENTER, ERESUME) before and after the exception occurs. Doing such changes in unmodified binaries, though not conceptually impossible, is complex and intricate. Our design needs to prevent overriding any pre-existing legitimate behavior of the enclave binary and ensure that the context expected by the application binary remains the same, i.e., as if the binary was running outside the enclave. We explain the challenges and our design in Section 3, and Section 4 provides details.

Dynamic binary translation (DBT) is a well-known approach to binary code instrumentation and implementing inline reference monitors. It intercepts each instruction in a program before it executes [52]. In this article, we choose DynamoRIO as our DBT engine, since it is open-source and widely used in industry [24].³ Vanilla DynamoRIO works much like a just-in-time compilation engine, which dynamically re-generates (and instruments) code of the application running on it. At a high level, DynamoRIO first loads itself and then loads the application code in a separate part of the VA space, as shown in Figure 2. Similarly, it sets up two different contexts, one for itself and one for the application. DynamoRIO can update the code on-the-fly before putting it in the code cache by re-writing instructions (e.g., convert a syscall instruction to a stub or library function call). Such rewriting ensures that DynamoRIO engine takes control before each block of code executes, enabling the ability to interpose on every instruction. Instrumented code blocks are placed in a region of memory called a code cache. When the code cache executes, DynamoRIO regains control as the instrumentation logic desires. It does post-execution updates to itself for book-keeping or to the program’s state. Additionally, DynamoRIO hooks on all events intended for the process (e.g., signals). The application itself is prevented from accessing DynamoRIO memory via address-space isolation techniques [52]. Thus, it acts as an arbiter between the application’s binary code and the external environment(e.g., OS, filesystem) with complete interposition.

The original DynamoRIO engine is designed to work for non-enclave code. We adapt it to work inside SGX enclaves, resulting in our Ratel system. To contrast it with the approach of changing \( {\tt libc} \) , DBT intercepts the application right at the point at which it interacts with the OS (Figure 1) for SGX compatibility. Ratel retains the entire low-level instruction translation and introspection machinery of DynamoRIO, including the code cache and its performance optimizations. This enables reusing well-established techniques for application instrumentation and performance enhancements. We eliminate the support for auxiliary client plugins to reduce trusted computing base (TCB), but a suite of built-in runtime profilers (see Table 12), which do not use the DynamoRIO client plugin interfaces are retained in Ratel.

DynamoRIO instrumentation capability is also designed to execute a race-free application without introducing any new races [26]. To achieve this, vanilla DynamoRIO engine itself outlines the following design principles:

The four principles above avoid introducing any new concurrency bugs (e.g., race conditions) in the translated application. We explain how our approach in Ratel piggybacks on such thread-safety to ensure there are no concurrency issues beyond those that exist in the logic of the original translated program in Section 3.3.

Ratel provides compatibility for both the DynamoRIO DBT engine as well as any application binary code that DynamoRIO translates. We provide a high-level overview of our design and explain its key trade-offs.

High-level Overview. As a first step, Ratel loads the DynamoRIO dynamic translation engine at a specific location in virtual memory, which we denote as \( A \) . Let us say that the vanilla dynamic translation engine in DynamoRIO is coded to access virtual address memory regions denoted as \( B \) and the target application accesses regions \( C \) , respectively. The basic principle behind the design of Ratel is to ensure referential transparency: whenever the DynamoRIO engine accesses any virtual address that would have been a location in \( B \) (without Ratel), it must now access the corresponding location with base at the relocated address \( A \) in Ratel. Similarly, if the application would have accessed a location in \( C \) originally, then Ratel must ensure that it accesses the corresponding to its translated location. To do this, Ratel must (a) intercept all operations that create virtual memory maps (e.g., via static and dynamic loading), and (b) keeps an address translation table in \( A \) for translating program accesses made by the target application dynamically. Note that such referential transparency for memory accesses provides compatibility with position-independent code, dynamically generated code, and shadow memory data structures (e.g., shadow stacks) that the application may have originally used—the memory references at runtime resolve consistently to the same translated address and the values read/written as thus consistent with the original run. For security, Ratel must ensure that all accesses to \( A \) originate from the dynamic translation engine itself, and the application code is unable to access \( A \) directly—a memory isolation policy. Ratel enforces this policy for the dynamic translation engine by modifying its code statically. For the target application, memory isolation can be enforced at runtime through the instruction rewriting capability of DynamoRIO itself, as done in program shepherding [52].

In addition, Ratel modifies DynamoRIO to adhere to SGX virtual memory limitations (R1–R4). In designing Ratel, we statically change the DynamoRIO code to load it at a fixed memory region \( A \) . Note that \( A \) can be fixed at the time of initialization of the process (loading), therefore, it does not break compatibility with address-space layout randomization. This allows us to load Ratel and start its execution without violating the memory semantics of SGX. We register a fixed entry point in Ratel when entering or resuming the enclave. This entry point acts as a unified trampoline, such that upon entry, Ratel decides where to redirect the control flow, depending on the previously saved context. In DynamoRIO code, we statically replace all instructions that are illegal in SGX with an external call that executes outside the enclave. Thus, Ratel execution itself is guaranteed to never violate R5.

Ratel has complete control over the loading and running of the translated application binary. Therefore, to ensure that the application adheres to R1–R5, Ratel dynamically rewrites the instructions before they are executed from the code cache. To keep compatibility with R2, we statically initialize the virtual memory size of the application to the maximum allowed by SGX; the type and permissions of memory are set to the specified type in the original binary. Ratel augments its memory manager to keep track of and transparently update the application memory layout as it changes during execution. At runtime, the application can make direct changes to its own virtual memory layout via system calls. Ratel dynamically adapts these changes to SGX by making two copies, wherever necessary, or by relocating the virtual address regions. Ratel intercepts all application interactions with the OS. It modifies application parameters, OS return values and events for monitoring indirect changes to the memory (e.g., thread creation). Before executing any application logic, Ratel scans the code cache for any instructions (e.g., \( {\tt syscall} \) , \( {\tt cpuid} \) ) that may potentially be deemed as illegal in SGX and replaces it with an external call. In the other direction, Ratel also intercepts OS events on the behalf of the application. Upon re-entry, if the event has to be delivered to the application (e.g., signals for application itself), it sets/restores the appropriate execution context and resumes execution via the trampoline. In this way, Ratel remedies the application on-the-fly to adhere to R1–R5.

Ratel helps to interpose on the enclave code without relying on the untrusted OS. In doing so, the SGX restrictions \( R1\text{--}R5 \) give rise to trade-offs between ensuring complete interposition and having low performance overheads. We point out that these are somewhat fundamental and apply to Ratel and other compatibility efforts equally. However, Ratel chooses completeness in its interposition over performance, whenever conflicts arise.

Two-copy mechanism for \( R1 \) & \( R2 \) . Due to restriction \( R1 \) , whenever the application wants to read or write data outside the enclave, the data needs to be placed in public memory. Computing on data in public memory, which is exposed to the OS, is insecure. Therefore, if the application wishes to securely compute on the data, then a copy must necessarily be maintained in a separate private memory space, as \( R2 \) forbids making changes to the memory permissions dynamically. Specifically, we use a two-copy mechanism, instances of which repeat throughout the design. Consider that case where an enclave wishes to write data to a file. The OS cannot access the buffer in enclave private memory. Here, the enclave can utilize a two-copy design, in which we place the file data in public memory. The enclave keeps an additional copy i.e., the second copy of that data in its private memory. When the enclave wishes to update the file, the enclave updates the private and the public copy. When reading data from the public memory such as a file, an enclave must copy the data to private memory and check it before further use.

The above two-copy design pattern is used in other scenarios in Ratel too, specifically, when the enclave shares data with the OS (e.g., for system call handling) and when the data in its private memory needs to have different permissions over time. The OS can manipulate the public memory content at any time—while the enclave is computing on the data, after the enclave’s check but before its use. Such changes by the OS can result in TOCTOU bugs. Keeping a separate private copy in the enclave reduces the attack surface. The other scenario where two-copy design comes useful is when the enclave wants to change the permissions of its private memory, such as in switching read-only data to executable/writable, but within the enclave itself. Such permission switching for memory region is disabled on SGX due to R2. To address this, the two-copy mechanism creates a copy of the data in an additional separate memory region inside private memory, such that the new region has the new permissions. The two-copy mechanism, however, incurs both space and computational performance overheads. Every update to the data causes at least a memory copy operation. The total overhead depends on the application’s workload characteristics (kinds and frequency of shared data accesses) and can vary from \( 10\% \) for CPU-intensive workloads to \( 10\times \) for IO-intensive workloads (see Section 6.3).

Memory Sharing for \( R3 \) & \( R4 \) . \( R3 \) creates an “all or none” trust model for enclaves. Either memory is shared with all entities (including the OS) or kept private to one enclave. \( R4 \) restricts sharing memory within an enclave further. These restrictions conflict with semantics of shared memory and synchronization primitives. For instance, synchronization primitives such as futexes are implemented with a single memory copy that the OS is trusted to manage securely—such a design is in direct conflict with the SGX security model. To implement such abstractions securely, designs on SGX must rely on a trusted software manager, which necessarily resides in an enclave, since the OS is untrusted (see Section 4.4). Applications can then maintain compatibility with locks and shared memory abstractions. But, this comes at a trade-off in performance costs: access to shared memory or synchronization primitives, which are originally inexpensive memory accesses, turn into (possibly remote) procedure calls to the trusted manager enclave.

Secure entry-points for \( R5 \) . Restriction \( R5 \) requires that whenever the enclave resumes control after an exit, the enclave state (or context) should be the same as right before exit. This implies that the security monitor (e.g., the DBT engine) must take control before all exit points and after resumption, to save-restore contexts—otherwise, the interposition can be incomplete, creating security holes and incompatibility. Without guarantees of complete interposition, the OS can return control into the enclave, bypassing security checks that the DBT engine implements. The price for complete interposition on binaries is performance—the DBT engine must intercept all entry/exit points and simulate additional context switches in software. Prior approaches, such as library OSes, choose performance over completeness in interposition, by asking applications to link against specific library interfaces that constrict enclave-OS interaction via certain specified library interfaces. But, this does not enforce complete interposition. Applications, due to bugs or when exploited, can make direct OS interactions without using the prescribed API, use inline assembly, or override entry handlers setup by the library OS. All scenarios in which applications go outside the prescribed interfaces, the library OS design requires special handling.

Several additional security considerations arise in the implementation details of our design. These include (a) avoiding naïve designs that have TOCTOU attacks; (b) saving and restoring the execution context from private memory; (c) maintaining Ratel-specific metadata in private memory to ensure integrity of memory mappings that change at runtime; and (d) explicitly zeroing out memory content and pointers after use. We explain them inline in Section 4.

Ratel is best viewed as a general framework for instruction-level interposition on enclaved binaries, rather than a stand-alone sandboxing engine that protects enclaves against all possible OS attacks. Ratel itself does not trust the OS or any security guarantees the OS provides. Binaries running on top of Ratel otherwise follow the same threat model as vanilla DBT engines [82]. Application binaries are assumed to not be aware of the presence of Ratel—they are benign binaries but they can be exploited via externally provided inputs. Under exploitation, malicious code may execute on Ratel. Ratel provides instruction-level instrumentation of all instructions executed and does not provide any higher-level security guarantees beyond that. For example, malicious code can readily determine that it is running on Ratel [42] and, therefore, Ratel is not suitable for analyzing analysis-evading malicious code. Using the instruction-level monitoring capability, the vanilla DynamoRIO engine itself provides certain built-in security mechanisms, which are preserved in Ratel. Specifically, ASLR for all heap regions is turned on and the code cache is randomized. Ratel isolates the stack, dynamically allocated memory, and file-mapped I/O regions through instrumentation. The main new challenges highlighted in this work are those due to enabling DBT on SGX, while most other threats to DBT-based instrumentation are pre-existing and known. The design trade-offs we emphasize apply to any interposition framework that runs on SGX, but have not been emphasized clearly in prior works.

Building an end-to-end secure sandbox on top of Ratel requires additional security mechanisms, which are common to other systems and are previously known. These mechanisms include encryption/decryption of external file or I/O content [16, 30, 49, 72], sanitization of OS inputs to prevent Iago attacks [31, 51, 77, 81], defenses against known side-channel attacks [22, 48, 66, 73, 74], additional attestation or integrity of dynamically loaded/generated code [44, 45, 46, 85], and so on. These are important but largely orthogonal to our focus.

Our binary compatibility layer has support for large majority but not all of the Linux system calls. The most notable of these unsupported system calls is \( {\tt fork} \) , which is used for multi-processing. Since Ratel does not support multi-process applications, we support locks and synchronization primitives for threads within a single enclave. The basic design of Ratel can be extended to support \( {\tt fork} \) with the two-copy mechanism, similar to prior work [30, 75]. However, maintaining compatibility with fork blindly is a questionable design decision, especially for enclaves, as has been argued extensively [19]. A recent work on SGX compatibility has left out support for \( {\tt fork} \) and multi-enclave locks based on the same observation [72].

SGX does not allow enclaves to execute several instructions such as \( {\tt syscall} \) , \( {\tt cpuid} \) , and \( {\tt rdtsc} \) . If the enclave executes them, then SGX exits the enclave and generates a \( {\tt SIGILL} \) signal. Gracefully recovering from the failure requires re-entering the enclave at a different program point. Due to R5, this is disallowed by SGX. In Ratel, either DynamoRIO or the application can invoke illegal instructions, which may create unanticipated exits from the enclave.

Ratel changes DynamoRIO logic to convert such illegal instruction to stubs that either delegate or emulate the functionality. For the target application, whenever Ratel observes an illegal instruction in the code cache, it replaces the instruction with a call to the Ratel syscall handler function. Ratel has three ways of handling system call execution:

Ratel uses complete delegation for file, networking, and timer-related system calls. It uses partial delegation for memory management, threads, and signal handling. We outline the details of other syscall subsystems that are fully or partially emulated by Ratel in Sections 4.2, 4.3, 4.4, and 4.5. Ratel uses emulation for very few system calls. For example, the \( {\tt arch\_prctl} \) syscall is used to read the FS base. Ratel emulates it by executing a \( {\tt rdfsbase} \) instruction.

Creating and Synchronizing Memory Copies. Syscalls access process memory for input-output parameters and error codes. Since enclaves do not allow this, for delegating the syscall outside the enclave, Ratel creates a copy of input parameters from private memory to public memory. This includes simple value copies as well as deep copies of structures. The OS then executes the syscall and generates results in public memory. After the syscall completes, Ratel copies back the OS-provided return values and error codes to private memory.

Memory copies alone are not sufficient. For example, when loading a library, the application uses \( {\tt dl\_open} \) , which in turn calls \( {\tt mmap} \) , which must execute outside the enclave. Thus, the \( {\tt mmap} \) call outside the enclave will map the library in the untrusted public address space of the application. However, the original intent of the application is to map the library inside the enclave private memory. As another example, consider when the enclave code wants to create a new thread local storage (TLS) segment. Due to the restrictive SGX environment, Ratel must execute the system call outside the enclave, and the new thread is created for the DynamoRIO runtime instead of the target application. In all such cases, Ratel takes care to explicitly propagate changes to inside the enclave, i.e., reflect changes to private memory.

When the enclave code copies input arguments or results for external functions (e.g., syscalls) between private and public memory, other enclave threads should not change the private memory. To ensure this, Ratel uses DynamoRIO locks to synchronize private memory accesses introduced by two-copy mechanism. Our implementation adheres to principles (a)–(d) outlined in Section 3.1. One caveat that arises in Ratel is when two threads in the same enclave attempt to perform syscalls simultaneously. Consider an example where the application has two threads, one of which is waiting/polling to acquire a lock, while the other holds it. As per the principle (c), the Ratel instance running in thread 1 will try to acquire a DynamoRIO-specific lock (inside the \( \tt {poll} \) syscall), which might already have been held by the Ratel instance running in the thread 2. This can lead to a deadlock. To avoid deadlocks in such cases, we examine each system call and add DynamoRIO locks only when the syscall updates private memory.

Checking Memory State after Syscalls. Ratel resumes execution in the enclave only after the syscall state has been completely copied inside the enclave. This allows the enclave to employ sanitization of OS return values before using it. Previously known sanitization checks for Iago attacks can be implemented here [77]. Note that all such sanitization checks must execute inside the enclave and after the state from the public memory is copied into the enclave private memory. This caveat is important to avoid TOCTOU attacks wherein the OS modifies public memory state before or midway through the execution of the sanitization checks.

Ratel utilizes partial emulation for syscalls that change the process virtual memory layouts and permissions (e.g., mmap, mprotect, fsync, and so on). It executes the syscall outside the enclave and then explicitly reflects the memory layout changes inside the enclave. First, this is not straightforward. Due to R1–R4, several layout configurations are not allowed for enclave virtual memory (e.g., changing memory permissions). Second, Ratel does not trust the OS information (e.g., via \( {\tt procmap} \) ). Hence, Ratel must use a two-copy mechanism when it uses the partial delegation approach.

Specifically, Ratel maintains its own \( {\tt procmap} \) -like structure to keep its own view of the process virtual memory inside the enclave, tracks the memory-related events, and updates the enclave state. For example in the case of mmap syscall to map a file in enclave private memory, the handler outside the enclave creates a public memory region by invoking the OS first. Then, Ratel allocates a region of private memory, which mirrors the content of the file mapped outside the enclave, and updates its internal \( {\tt procmap} \) -like structure to record the new virtual addresses created. Further, Ratel synchronizes the two-copies of memories to maintain execution semantics on all subsequent changes to \( {\tt mmap} \) ped-memory. This is done whenever the application unmaps the memory or invokes the \( {\tt sync} \) / \( {\tt fsync} \) syscalls.

Ratel does not blindly replicate OS-dictated memory layout changes inside the enclave. It first checks if the resultant layout will violate any security semantics (e.g., mapping a buffer to zero-address). It proceeds to update enclave layout and memory content only if these checks succeed. To do this, Ratel keeps its metadata in private memory.

With interposition over memory management, Ratel transparently side-steps SGX restriction due to R2. When application makes changes to the permissions of a memory region (say \( X \) ) dynamically, Ratel moves the content to a memory region (say \( Y \) ), which has the required permissions. To do this, Ratel requires a stash of unused private memory regions ( \( Y \) ) that are originally not used by the application. These memory regions are allocated statically by Ratel at the start and their page permissions are set to readable, writable, and executable. Subsequently, when the application binary accesses memory \( X \) , Ratel dynamically translates the access to the copy in memory \( Y \) . This allows Ratel to transparently emulate the permission change, which is otherwise a disallowed behavior inside the enclave.

Note that the stash pool of private memory regions \( Y \) are private to the enclave, but have overly permissive access rights. This can be avoided by reserving regions with write-only and execute-only permissions, but this may decrease the size of usable non-stash memory. Alternatively, the access rights can be implemented through runtime monitoring of memory accesses, but this adds performance costs. The trade-offs are inherent to restrictions \( R1 \) and \( R2 \) .

Ratel supports multiple threads running in a single enclave. However, it has no support for fork to create multiple processes running in different enclaves. Therefore, our design restricts our concerns to enabling thread synchronization within a single enclave. This is still challenging, because restriction \( R2 \) requires the enclaved application to pre-declare a maximum number of threads before execution. SGX also does not allow the enclave to resume at arbitrary program points or execution contexts (restriction \( R5 \) ). This creates several challenges in adapting DynamoRIO to run in SGX.

In the vanilla DynamoRIO design, the dynamic translation engine and the target application share the same thread, but they have separate TLS segment for cleaner context switch. DynamoRIO keeps the default TLS segment for the target application and creates a new TLS segment for itself at a different address. It switches between these two TLS segments by changing the segment register—DynamoRIO uses \( {\tt gsbase} \) and the application uses \( {\tt fsbase} \) .

Multiplexing TLS Segments. Normally, to context switch between threads, one TLS segment to save the currently executing application thread context is sufficient. But with Ratel, we need an additional TLS segment to save the state of Ratel itself. Furthermore, SGX itself reserves one additional TLS segment for its own internal use. This brings the total number of required TLS segments, for a correct context switch, to 3 on SGX.

But, the x86_64 architecture itself provides only two base registers ( \( {\tt fsbase} \) and \( {\tt gsbase} \) ) for storing pointers to TLS segments. Therefore, when we attempt to run DynamoRIO inside SGX, there are not enough base registers to save three TLS segment offsets (one each for DynamoRIO, SGX, and the application). We circumvent this limitation of the SGX platform as follows. First, Ratel adds two fields in each TLS segment to store \( {\tt fsbase} \) and \( {\tt gsbase} \) register values for that segment. We use these TLS segment fields to save and restore pointers to the segment base addresses. This allows us to still maintain and switch between three clean TLS segment views per thread. Second, when Ratel has to restore a TLS segment, it searches through a list of TLS segment base addresses, to find the right one to restore—this is because it does not have enough base registers to store three TLS segment bases (which would have avoided the search).

Restoring TLS Segments on Context Switches. Ratel conceptually maintains a linked list of a maximum of three TLS segment base pointers. The head of the list is the \( {\tt fsbase} \) register, which serves as a pointer to the default first TLS segment created by SGX reserved for its own use. This default segment is called the primary and all TLS segments created subsequently are referred to as secondary. To point to the next TLS segment in the linked list, Ratel adds a new field in the TLS segment, which is NULL for the last element in the list. To traverse the list, the \( {\tt gsbase} \) register is used. The list search begins with the primary, and the right segment to restore is always the last element in the link list. This way, Ratel can search and decide which of the three TLS segments to restore using only two registers ( \( {\tt fsbase} \) and \( {\tt gsbase} \) ).

There are two ways in which control can enter/exit from the enclave: via synchronous exits (e.g., \( {\tt ECALL} \) / \( {\tt OCALL} \) used for syscalls) and via asynchronous exits (e.g., used for exceptions, timer interrupts, and so on). During synchronous exits, Ratel sets up the TLS segment link list such that the restored TLS context state upon resumption is for DynamoRIO and SGX, respectively. The subsequent exception handlers copy state from outside the enclave to inside, perform various Iago checks, and then setup the TLS segment to that of the application thread. During asynchronous exits, Ratel does not need to perform any special setup for the TLS segment link list. When the exception handler executes on resumption, it sets up the executing context just before the exit. Ratel performs similar checks and operations as in the case of synchronous exits, and then restores the context to that before the exit (which may be DynamoRIO’s context or that of the application thread). Therefore, our design works correctly for both synchronous and asynchronous exits.

Dynamic Threading. Since the number of TCS entries is fixed at enclave creation time on SGX, the maximum number of threads supported is capped. Ratel multiplexes the limited TCS entries available among the application threads dynamically, as shown in Figure 4. When an application wants to create a new thread (e.g., via clone), Ratel first checks if there is a free TCS slot. If it is the case, then it performs an \( {\tt OCALL} \) to do so outside the enclave. Otherwise, it busy-waits until a TCS slot is released. Once a TCS slot is available, the \( {\tt OCALL} \) creates a new thread outside the enclave. After finishing thread creation, the parent thread returns back to the enclave and resumes execution. The child thread explicitly performs an \( {\tt ECALL} \) to enter the enclave and DynamoRIO resumes execution for the application’s child thread.

For all threading operations, Ratel ensures transparent context switches to preserve binary compatibility as intended by DynamoRIO. For security, Ratel creates and stores all thread-specific context either inside the enclave or SGX’s secure hardware-backed storage at all times. It does not use any OS data structures or addresses for thread management.

Safety of Two-copy Mechanism. Ratel does not introduce any new concurrency issues due to its two-copy mechanism (i.e., one-way or two-way copy). Consider two threads in the same enclave sharing a region of memory and concurrently accessing it. If the original code itself has data races when accessing memory, then those races will be preserved in Ratel with its two-copy mechanism. However, if the original code is race-free, then its execution with Ratel will also be race-free. The two-copy mechanism ensures this, because it does not introduce any additional writes/reads to shared (public) memory—it only replaces the original reads/writes of the shared data with reads/writes to translated addresses in private memory. The use of a private copy does not alter the semantics of the original access. In Section 6.1.4, we evaluate the above scenarios for native, DynamoRIO, and Ratel execution of real-world applications.

SGX provides basic synchronization primitives (e.g., SGX mutex and conditional locks) backed by hardware locks. But they can only be used for enclave code. Thus, they are semantically incompatible with the lock mechanisms used by DynamoRIO or legacy applications, which use OS locks. For example, DynamoRIO implements a fast lock using the \( {\tt futex} \) syscall, where the lock is kept in a shared memory accessible to all application threads and the OS. Here, the OS needs the ability to read the lock state to determine whether it should wait during the \( {\tt FUTEX\_WAIT} \) syscall.

A naive design would be to maintain the \( {\tt futex} \) lock in public memory, such that it is accessible to the enclave(s) and the OS. However, the OS can arbitrarily change the lock state and attack the application. Specifically, it can reset the lock during the execution of a critical section in an application thread. Therefore, this design choice is not safe.

As an alternative, we can employ a two-copy mechanism for locks. The enclave can keep the lock in private memory. When it wants to communicate state change to the OS, Ratel can tunnel a futex \( {\tt OCALL} \) to the host OS. This approach is problematic as well. Threads inside the enclave may frequently update the locks in private memory. The futex state outside the enclave needs to be kept consistent with the private copy, when the OS kernel and the untrusted part of the enclave access it, or else the semantics of the lock may not uphold. The more frequent the local updates to the in-enclave copy of the lock state, the higher the chance of inconsistencies. In general, avoiding such race conditions usually involves using locks for synchronizing. But requiring locks to synchronize copies of other locks, as suggested in this design alternative, only results in a chicken-and-egg problem.

Supporting such semantics efficiently, where the OS has a shared read access to the lock state, is difficult with SGX because of restrictions \( R1 \) and \( R3 \) . Figure 3 shows the schematics of design choices for implementing synchronization primitives available on SGX. Note that options (a) and (b) are insecure as discussed above. Ratel implements a lock manager inside the same enclave that executes the application. Our simplification has one limitation: Only threads within the same application process (and enclave) can utilize Ratel synchronization primitives.

Ratel Lock Manager Implementation. Our design choice of using a single enclave to execute both the DynamoRIO engine and all application threads eliminates considerable complexity in implementation. It turns out that a futex-based locks become unnecessary, since we do not need sharing across the process boundary or with the OS. The DynamoRIO usage of futexes can thus be replaced with a simpler primitive such as spinlocks to achieve the same functionality. Specifically, Ratel implements a lock manager using the hardware spinlock exposed by SGX. Ratel invokes its in-enclave lock manager either when DynamoRIO uses futexes or when the application binaries perform lock-based synchronization. For the DynamoRIO code, we manually change it to invoke our lock manager. In case of application locks, Ratel loads application binary into the code cache and replaces thread-related calls (e.g., \( {\tt pthread\_cond\_wait} \) ) in the enclave-OS interface with stubs to invoke our lock manager to use Ratel-provided safe synchronization primitives.

Ratel cannot piggyback on the existing signal handling mechanism exposed by the SGX, due to restriction \( R5 \) . Specifically, when DynamoRIO executes inside the enclave, the DynamoRIO signal handler needs to get description of the event to handle it (Figure 5(a)). However, Intel’s SGX platform software removes all such information when it delivers the signal to the enclave. This breaks the functionality of programmer-defined handlers to recover from well known exceptions (e.g., divide by zero). Further, any illegal instructions inside the enclave generate exceptions, which are raised in the form of signals. Existing binaries may not have handlers for recovering from such illegal instructions. Therefore, Ratel must provide handlers for all such exceptions.

Recall that SGX allows entering the enclave at fixed program points. Leveraging this, Ratel employs a primary signal handler that it registers with SGX. For any signals generated for DynamoRIO or the application, we always enter the enclave via the primary handler and we copy the signal number into the enclave. We then use the primary as a trampoline to route the control to the appropriate secondary signal handler inside the enclave, based on the signal number. At a high-level, we realize a virtualized trap-and-emulate signal handling design. We use SGX signal handling semantics for our primary. For the secondary, we setup and tear down a separate stack to mimic the semantics in the software. The intricate details of handling the stack state at the time of such context switch are elided here. Figure 5(b) shows a schematic of our design, and we explain the flow of control and associated issues here.

Registration. The original DynamoRIO code and the application binary use \( {\tt sigaction} \) to register signal handlers for itself. In Ratel, first we change DynamoRIO logic to register only the primary signal handler with SGX. We then record the DynamoRIO and application registrations as secondary handlers. This way, when SGX or the OS delivers the signal to the enclave, SGX directs the control to our primary handler.⁴ Since this is a pre-registered handler, SGX allows it. The primary handler checks the signal information (e.g., signal code) and explicitly routes execution to the secondary.

Delivery. A signal may arrive when the execution control is inside the enclave. In this case, Ratel executes a primary signal handler that delivers the signal to the enclave. However, if the signal arrives when the CPU is in a non-enclave context, SGX does not automatically invoke the enclave to redirect execution flow. To force this, Ratel has to explicitly enter the enclave. But it can only enter at a pre-registered program point with a valid context, as per restriction \( R5 \) . Thus, Ratel first wakes up the enclave at a valid point (via \( {\tt ECALL} \) ). Ratel copies the signal information, passed as an input argument to the \( {\tt ECALL} \) , to private memory. It then simulates the signal delivery by setting up the enclave stack in private memory to execute the primary handler.

Exit. After executing their logic, handlers use \( {\tt sigreturn} \) instruction for returning control to the point before the signal interrupted the execution. When Ratel observes this instruction in the secondary handler it has to simulate a return back to the primary handler instead. The primary handler then performs its own real \( {\tt sigreturn} \) . SGX then resumes execution from the point before the signal was generated.

Handling Nested Signals. One issue with platforms like SGX is that it supports synchronous and asynchronous signals. Signals can be nested, in the sense that signals can be delivered by the OS while the enclave is handling another one. The enclave cannot mask signals selectively at runtime on SGX. Accordingly, the potential for subtle reentrancy bugs in the enclave signal handling code arises. At a high-level, Ratel handles signals safely by ensuring that unsafe nesting is not possible. Specifically, the SGX platform automatically saves enclave state in private memory regions pointed to by hardware State Save Area (SSA) when an exception is to be delivered. To support nesting, SGX provides an array of SSAs frames, leaving the option to set the maximum size of array (hence, the maximum possible nesting depth) to the enclave. Ratel utilizes this feature to limit the nesting depth to 2. This is needed because in Ratel one SSA frame can be used by Ratel itself and the other by the target application. With the maximum nesting depth set to 2, if a nested signal of depth 3 is being attempted to be delivered, SGX securely aborts the enclave, since not enough SSA frames are available. With this design, there are only four possible combinations of reentrancy to reason about:

In Cases 1 and 3, DynamoRIO gets execution control. In Case 1, DynamoRIO has one additional SSA frame available to save the current state of the primary handler and deliver control to the beginning of the primary signal handler but with the new context. In Case 3, the two SSA frames are already used up to save Ratel’s primary handler state and the application’s secondary handler state. Therefore, SGX is unable to deliver the signal and the enclave execution is aborted. In Cases 2 and 4, similar to Case 3, the two SSA frames are already in use to store the current state of the primary and secondary signal handlers, respectively. SGX, thus, cannot deliver the signal and the enclave is aborted. Thus, in all the above scenarios, Ratel design handles reentrancy safely. Note that Ratel reentrancy handling for synchronous exceptions, which are used for threads and syscalls (Section 4.3), and asynchronous exceptions is the same regardless of the exception type.

Remark. Ratel signal handling extends the trap-and-emulate principle used by DynamoRIO. Since DynamoRIO design enforces transparency, we preserve this in Ratel. This goes beyond merely working around the SGX limitations, thus making our design different than existing frameworks. Specifically, existing library OS-based SGX frameworks (e.g., Graphene-SGX and Occlum) assume that all exception registration and execution will go via the prescribed library interfaces. So, they do not keep a separate signal context for the library OS signals and the application. These works do not discuss what happens during nested signals or when the enclave-OS interaction happens outside of the prescribed library registration and handler mechanisms.

To evaluate compatibility, we initially select 310 binaries that cover an extensive set of benchmarks, utilities, and large-scale applications. These are commonly reported to be used as evaluations target for DynamoRIO and prior enclave-based systems [16, 25, 30, 34, 47, 76, 77] that we surveyed for our study. Further, they represent a mix of memory-intensive, CPU-intensive, multi-threading, network-intensive, and file I/O workloads. A total of 69 binaries are from micro-benchmarks: 29 from SPEC 2006 (CPU), 1 from IOZone (I/O) v3.487, 9 from FSCQ v1.0 (file API), 21 from HBenchOS v1.0 (system stress-test), and 9 from Parsec-SPLASH2 (multi-threading). We run 12 binaries from three real-world applications—cURL v7.65.0 (server-side utilities), SQLite v3.28.0 (database), Memcached v1.5.20 (key-value store), and nine applications from Privado (secure ML framework). We selected all 229 Linux utilities, which are available from our test system’s \( {\tt /bin} \) and \( {\tt /usr/bin} \) directories. Apart from these 310 binaries, we tested two language runtimes, Python (v2.7.17 and v3.8.2) and R v3.6.3. Python is tested with Python-CPU-Benchmark [6], Python Programming Examples [8], and PyBench v2.0 [7]. R is tested with the R-benchmark-25 [3].

We trust Intel SGX support software (SDK and PSW) that executes inside the enclave and interfaces with the hardware. This choice is the same as any other system that uses enclaves. Ratel comprises one additional trusted component—DynamoRIO. Put together Ratel amounts to \( \text{277,803} \) LoC TCB. This is comparable to existing SGX frameworks that have 100K to 1M LoC [16, 30], but provide library-based compatibility at best.

Table 9 (columns 2 and 3) summarizes the breakdown of the LoC included in the trusted components of the PSW, the SGX SDK, the DynamoRIO system, as well as the code contributed by each of the sub-systems supported by Ratel. The original DynamoRIO engine comprises \( \text{353,139} \) LoC. We reduce it to \( \text{129,875} \) LoC (trusted) and \( \text{66,629} \) LoC (untrusted) by removing the components that are not required or used by Ratel. Then, we add \( \text{8,589} \) LoC to adapt DynamoRIO to SGX as per the design outlined in Section 4. Apart from this, as described in Section 5, we change the libraries provided by Intel SGX (SDK and PSW) and add \( \text{1,078} \) LoC.

Of the \( \text{277,803} \) LoC of trusted code, \( \text{123,322} \) LoC is from the original DynamoRIO code base responsible for loading the binaries, code cache management, and syscall handling. A total of \( \text{110,848} \) LoC and \( \text{37,080} \) LoC are from Intel SGX SDK and PSW, respectively. Ratel implementation adds only \( \text{6,553} \) LoC on top of this implementation. A large fraction of our added TCB (27.5%) is because of the \( {\tt OCALL} \) wrappers that are amenable to automated testing and verification [51, 77]. Rest of the \( \text{4,752} \) LoC are for memory management, handling signals, TLS, and multi-threading interface.

Ratel relies on, but does not trust, the code executing outside the enclave in the host process (e.g., \( {\tt OCALL} \) s). This includes \( \text{2,391} \) LoC changes. We give a detailed breakdown of this in Table 9 (Columns 5–9).

We present the performance implications of our design choices made in Ratel. We have two main findings. First, the performance overheads vary significantly based on the application workload. Second, most of the overheads come from SGX restrictions R1–R5 and the enclave physical memory limits specific to our present test hardware. We point out that future SGX implementations may have over a \( 1,\!000\times \) larger private physical memory (1 TB EPC) compared to our test system [4]. Therefore, we expect that the performance bottlenecks due to physical memory limitations can be eliminated and are not fundamental to Ratel design. For completeness, we report Ratel memory footprint and impact of 90 MB limit in Section 6.3.2.

Methodology for Performance Measurement. For each target binary, we record the execution time in three settings:

To measure performance overheads, we collect various statistics of the execution profile of 58 programs in our micro-benchmarks and four real-world applications (12 binaries in total). Specifically, we log the target application LoC, binary size, number of \( {\tt OCALL} \) s, \( {\tt ECALL} \) s, syscalls, enclave memory size, peak virtual memory (VmPeak) for Linux and DynamoRIO, untrusted and trusted VmPeak for Ratel, number of page faults, and number of context switches. We refer readers to Appendix A.1 and A.2 for detailed performance breakdowns. Table 11 provides detailed statistics. We also provide the overheads comparison between our Baseline 2 (DynamoRIO) and Baseline 1 (Linux) in all the related tables and figures to provide a breakdown of the performance overhead.

Ratel primarily achieves binary compatibility by leveraging the complete interposition offered by DynamoRIO. Additionally, this instruction-level interposition allows Ratel to monitor various in-enclave behavior (e.g., events, instructions, control-flows, etc.) out-of-the-box. Specifically, DynamoRIO provides 26 built-in profilers for dynamically tracing, analyzing, and fine-tuning the target application. Table 12 summarizes the names and the profiling services they offer. When we run vanilla DynamoRIO on our experiment platform, 25 of 26 profilers work stably. One profiler, -prof_pcs, is unstable and causes time-outs. This is a well-documented issue with DynamoRIO [10, 11]. Of all 25 profilers that work with DynamoRIO, Ratel retains support for all of them. We experimentally demonstrate that Ratel maintains compatibility with built-in profilers. We randomly choose four application binaries from each of the six categories listed in Table 11. We run a total of 24 applications that exhibit diverse execution behavior with all of the 25 profilers. We report that all of 25 profilers worked with all our applications.

Several prior works have targeted SGX compatibility. There are two main ways that prior work has overcome these challenges. The first approach is to fix the application interface. The target application is either re-compiled or is relinked to use such interfaces. The approach that enables the best compatibility exposes specific Libc (glibc or musl libc) versions as interfaces. This allows them to adapt to SGX restrictions at a layer below the application. Container or library OS solutions use this to execute re-compiled/re-linked code inside the enclave as done in Haven [20], Scone [16], Graphene-SGX [30], Ryoan [49], SGX-LKL [68], and Occlum [72]. Another line of work is compiler-based solutions. They require applications to modify source code to use language-level interface [41, 62, 75, 86].

Both style of approaches can have better performance than Ratel, but require recompiling or relinking applications. For example, library OSes like Graphene-SGX and containerization engines like Scone expose a particular \( {\tt glibc} \) and \( {\tt musl} \) version that applications are asked to link with. New library versions and interfaces can be ported incrementally, but this creates a dependence on the underlying platform interface provider, and incurs a porting effort for each library version. Applications that use inline assembly or runtime code generation also become incompatible as they make direct access to system calls, without using the API. Ratel’s approach to handle R1–R5 comprehensively offers complete interposition, without any assumptions about specific interfaces beyond that implied by binary compatibility.

Security Considerations. As in Ratel, other approaches to SGX compatibility eventually have to use \( {\tt OCALL} \) s, \( {\tt ECALL} \) s, and syscalls to exchange information between the enclave and the untrusted software. This interface is known to be vulnerable [31, 37, 81]. Several shielding systems for file [28, 77] and network IO [18], provide specific mechanisms to safeguard the OS interface against these attacks. For security, defense techniques offer compiler-based tools for enclave code for memory safety [54], ASLR [71], preventing controlled-channel leakage [73], data location randomization [22], secure page fault handlers [66], and branch information leakage [47].

Ratel uses DynamoRIO’s in-built intra-process isolation primitives for separating application code from DynamoRIO code. It supports multi-threading within an enclave but does not support multi-processing. Recent library OSes such as Occlum support multi-process applications by executing them in the same enclave together with software-based fault isolation (SFI) to isolate within the process boundary. The combined use of SFI with Ratel instrumentation engine, instead of a library OS, constitutes promising future work to support multi-processing.

Performance. Several other works build optimizations by modifying existing enclave-compliant library OSes. Hotcalls [88] and Eleos [65] add exit-less calls to reduce the overheads of \( {\tt OCALL} \) s. These optimizations are now available in the default Intel SGX SDK.

Language Runtimes. Recent body of work has shown that executing either entire [85] or partial [33] language runtimes inside an enclave can help to port existing code written in interpreted languages such as Python [60, 69], Java [33], web-assembly [45], Go [44], and JavaScript [46].

Programming TEE Applications. Intel provides a C/C++ SGX software stack, which includes a SDK and OS drivers for simulation and PSW for running local enclaves. There are other SDKs developed in memory safe languages such as Rust [41, 62, 86]. Frameworks such as Asylo [17], OpenEnclave [64], and MesaTEE [61] expose a high-level front-end for writing native TEE applications using a common interface. They support several back-end TEEs including Intel SGX and ARM TrustZone. Many of the challenges faced by Ratel are common to these frameworks.

New enclave TEE designs have been proposed [29, 36, 39, 55, 76]. Micro-architectural side-channels [21] and new oblivious execution capabilities [36, 57] are significant concerns in these designs. Closest to our underlying TEE is the recent Intel SGX v2 [13, 58, 89].

Dynamic Permission Management. On SGX v1, once an enclave is initialized completely, enclave pages can no longer be added, removed, or modified with their permissions. On SGX v2, such restriction has been removed due to new SGX instructions. Thus, Ratel can address R2. Recall that, to dynamically load programs into an enclave, DynamoRIO has to reserve a large block of enclave memory region with full permissions (RWX) during the Ratel enclave setup. This is a common design choice in existing SGX runtimes, such as Occlum and Graphene-SGX. With SGX v2, the RWX code cache region in Ratel can be protected (execute-only) via dynamic permission management.

Larger EPC. Until recently, SGX v1 only supported 128 MB EPC memory. Intel recently started shipping machines for SGX v1 with 256 MB EPC and publicly released the plans to support 1 TB EPC [4]. With a larger EPC, the performance effect due to R1, R3, R4, and R5 will be greatly improved. Specifically, a larger code cache capacity can speedup Ratel enclave creation and application execution. Moreover, the page swapping frequency will significantly be reduced, since a large memory usage is involved in Ratel (see Columns 8 and 9 in Table 11).

Dynamic Scaling of Virtual Memory and TCS. SGX v2 introduces several instructions for dynamic enclave memory management. This allows an enclave to perform dynamic heap allocation, stack expansion, and thread context creation [58, 89]. These features can improve performance by alleviating R2. Ratel performs dynamic heap allocation, stack expansion, and TCS management. With SGX v2, Ratel can manage these allocations in an on-demand way instead of using fixed pre-allocated memory. This can save EPC memory and speed up the program execution by reduced page swapping. For dynamic TCS control, Ratel will no longer need multiplexing the limited TCS entries that are fixed at enclave creation time (Section 4.3). Instead, Ratel can create threads on-demand in SGX v2.

Conditional Exception Handling. SGX categorizes hardware exceptions (AEX events) into two groups: unconditional and conditional exceptions. On SGX v1, only eight exception types (e.g., #DE) can be handled unconditionally while other three (e.g., #GP, #PF, and #CP) are not supported by default in vanilla SDK. Ratel enables these three exceptions by modifying the SGX SDK. On SGX v2, Ratel can configure the enclave parameter (e.g., \( {\tt MiscSelect} \) ) to support these exceptions.

In summary, SGX v2 features can improve Ratel performance and simplify security. The features can address \( R2 \) to some extent but do not address other restrictions. Thus, Ratel design largely applies to SGX v2 as well.

We measure the performance on diverse workloads to explain the costs associated with executing with Ratel.

System Stress Workloads. We use HBenchOS [23]—a benchmark to measure the performance of primitive functionality provided by an OS and hardware platform. In Table 10, we show the cost of each system-level operation such as system calls, memory operations, context switches, and signal handling. Memory-intensive operation latencies vary with benchmark setting: (a) when the operations are done with more iterations (in millions) and less memory chunk size (4 KB) the performance is comparable; (b) when the operations are done with less iterations (1 K) and more memory chunk size (4 MB) Ratel incurs bandwidth loss ranging from \( -169.68 \) % to 87.91% and 53.94% to 88.15% over Linux and DynamoRIO, respectively. This happens because when the chunk size is large, we need to allocate and de-allocate memory inside enclave for every iteration as well as copy large amounts of data.

These file operation latencies match with latencies we observed in our I/O intensive workloads (Figure 8). Specifically, the write operation incurs large overhead. Hence, the \( {\tt create} \) workload incurs 4766.67% and 255.66% overhead over Linux and DynamoRIO, because the benchmark creates a file and then writes predefined sized data to it. Costs of system calls that are executed as \( {\tt OCALL} \) s vary depending on return value and type of the system call. For example, system calls such as \( {\tt getpid, sbrk, sigaction} \) that return integer values are much faster. Syscalls such as \( {\tt getrusage, gettimeofday} \) returns structures or nested structures. Thus, copying these structures back and forth to/from enclaves causes much of the performance slowdown. Ratel has a custom mechanism for registering and handling signal (Section 4.5); it introduces a latency of 480.11% and 6,816.84% with respect to Linux as well as 24.55% and 818.69% with respect to DynamoRIO, respectively. Registering signals is cheaper, because it does not cause a context switch as in the case of handling the signal. Further, after accounting for the \( {\tt OCALL} \) costs, our custom forwarding mechanism does not introduce any significant slowdown.

CPU-bound Workloads. Ratel incurs \( 217.80\% \) and \( 34.91\% \) overhead averaged over 24 applications from SPEC 2006 [78] with respect to Linux and DynamoRIO, respectively. Table 11 shows the individual overheads for each application with respect to all baselines. From Table 11, we observe that applications that incur higher number of page faults and \( {\tt OCALL} \) s suffer larger performance slow-downs. Thus, similar to other SGX frameworks, the costs of enclave context switches and limited EPC size are the main bottlenecks in Ratel.

IO-bound Workloads. Ratel performs \( {\tt OCALL} \) s for file I/O by copying read and write buffers to and from enclave. We measure the per-API latencies using FSCQ suite for file operations [32]. Table 11 shows the costs of each file operation and file access patterns, respectively. Apart from the cost of the \( {\tt OCALL} \) , writes are more expensive compared to reads in general; the multiple copy operations in Ratel amplify the performance gap between them. Next, we use IOZone [63], a commonly used benchmark to measure the file I/O latencies. Figure 8 shows the bandwidth over varied file sizes between 16 MB to \( 1,\!024 \) MB and record sizes between 4 KB to \( 4,\!096 \) KB for common patterns. The trend of writes being more expensive holds for IOZone too. Ratel incurs an average slowdown of 87.5% and 66.2% over all operations, record sizes, and file sizes with respect to Linux and DynamoRIO, respectively.

Multi-threaded Workloads. We use the standard Parsec-SPLASH2 [67] benchmark suite. It comprises a variety of high-performance computing and graphics applications. We use it to benchmark Ratel overheads for multi-thread applications. Since some of the programs in Parsec-SPLASH2 mandate the thread count to be power to 2 (e.g., ocean_ncp), we fixed the maximum number of threads in our experiment to 16. Ratel changes the existing SGX design to handle thread creation and synchronization primitives, as described in Sections 4.3 and 4.4. We measure the effect of this specific change on the application execution by configuring the enclave to use varying number of threads between 1–16. The data for 2 threads is shown in Table 11.

Figure 9 shows a performance overhead of \( 10,\!156.52 \) % and 92.55% compared to Linux and DynamoRIO, on average, across all benchmarks and thread configurations. Particularly, DynamoRIO imposes an average overhead of \( 5,\!010.07 \) % over Linux with the same setting. For single-threaded execution, on average, Ratel causes an overhead of \( 2,\!050.92 \) % and 3.17% with respect to Linux and DynamoRIO, respectively, while they increase to \( 21,\!238.0 \) % and 159.78% in 16-thread execution. We measure the breakdown of costs and observe that, on average: (a) creating each thread contributes to a fixed cost of 57 ms; (b) shared access to variables becomes expensive by a factor of \( 1\text{--}7 \) times compared to the elapsed time of \( {\tt futex} \) synchronization with increase in number of threads. This is expected, because synchronization is cheaper in Linux and DynamoRIO execution, in which they use unsafe \( {\tt futex} \) primitives exposed by the kernel. However, Ratel uses expensive spinlock mechanism exposed by SGX hardware for security. Particularly, some of the individual benchmarks, such as \( {\tt water\_spatial} \) , \( {\tt fmm} \) and \( {\tt raytrace} \) that involve lots of lock contention events and have extremely high frequency of spinlock calls (e.g., the spinning counts of about \( \text{423,\!000} \) ms in Ratel while the \( {\tt futex} \) calls of about 500 ms in DynamoRIO for the raytrace with eight threads). Thus, they incur large overheads in synchronization.

We work with 4 representative real-world applications: a database server (SQLite), a command-line utility (cURL), a machine learning inference-as-a-service framework (Privado), and a key-value store (Memcached). These applications have been used in prior work [75].

SQLite. is a popular database [79]. We select it as a case-study because of its memory-intensive workload. We configure it as a single-threaded instance. We use a database benchmark workload [56] and measured the throughput (ops/sec) for each database operation with varying sizes of the database (total number of entries). Table 11 shows the detailed breakdown of the runtime statistics for a database with \( \text{10,\!000} \) entries. Figure 10(a) shows the average throughput over all operations. With Ratel, we observe a throughput loss of 36.88% and 28.71% on average over all database sizes compared to Linux and DynamoRIO. The throughput loss increases with increase in the database size. The drop is noticeable at \( 500\text{ K,} \) where the database size crosses the maximum enclave size threshold and results in significant number of page faults. This result matches with observations from other SGX frameworks that report SQLite performance [16].

cURL. is a widely used command line utility to download data from URLs [38]. It is network intensive. We test it with Ratel via the standard library test suite. Table 11 shows detailed breakdown of the execution time on Ratel. We measure the cost executing cURL with Ratel for downloading various sizes of files from an Apache (2.4.41) server on the local network. Figure 11(a) shows the throughput for various baselines and file sizes. On average, Ratel causes a loss of 604.11% and 142.11% throughput as compared to Linux and DynamoRIO. For all baselines, small files (below 100 MB) have smaller download time; larger file sizes naturally take longer time. This can be explained by the direct copying of packets to non-enclave memory, which does not add any memory pressure on the enclave. The only remaining bottleneck in the cost of dispatching \( {\tt OCALL} \) s, which increase linearly with the requested file size.

Privado. is a machine learning framework that provides secure inference-as-a-service [47]. It comprises of several state of the art models available as binaries that can execute on an input image to predict its class. The binaries are CPU intensive and have sizes ranging from 313 KB to 140 MB (see Table 11). We execute models from Privado on all the images from the corresponding image dataset (CIFAR or ImageNet) and measure inference time. Figure 11(b) shows the performance of baselines and Ratel for nine models in increasing order of binary size. We observe that Ratel performance degrades with increase in binary size. This is expected, because the limited enclave physical memory leads to page faults. Hence, largest model (140 MB) exhibit highest inference time and smallest model (313 KB) exhibit lowest inference time. Thus, Ratel and enclaves in general can add significant overheads, even for CPU intensive server workloads, if they exceed the working set size of 90 MB.

Memcached. is an in-memory key-value cache. We evaluate it with YCSB’s all four popular workloads A ( \( 50\% \) read and \( 50\% \) update), B ( \( 95\% \) read and \( 5\% \) update), C ( \( 100\% \) read), and D ( \( 95\% \) read and \( 5\% \) insert). We run it with four default worker threads running in Linux, DynamoRIO and Ratel settings. We vary the YCSB client threads with Load and Run operations (to load the data and then run the workload tests, respectively). We fix the data size to \( \text{1,\!000,\!000} \) with Zipfian distribution of key popularity. We increase the number of clients from 1 to 100 to find out a saturation point of each targeted/scaled throughput for the settings. Here, we only present workload A (throughput vs average latency for the read and update); the other workloads display similar behavior.

As shown in Figure 10(b), the client latencies of the DynamoRIO and Ratel settings for a given throughput are slightly similar until approximately \( \text{10,000} \) ops/s, while it nearly keeps unchanged on Linux until more than \( \text{40,000} \) ops/s. Specifically, Ratel jitters until it achieves maximum throughput around \( \text{17,000} \) ops/s, while DynamoRIO is flat until \( \text{15,000} \) ops/s (the maximum is \( \text{21,\!000} \) operations per second). The shared reason of the deceleration for both is that DynamoRIO slows down the speed of Read and Update. For Ratel, the additional bottleneck is the high frequency of lock contention with spin-lock primitive. For e.g., Ratel costs \( \text{18,320,000} \) ms while DynamoRIO’s the futex calls cost only around 500 ms for a given throughput of \( 10,\!000 \) with 10 clients.