Telepathic Datacenters: Fast RPCs using Shared CXL Memory

To allow applications to send RPCs, a server creates a channel that clients can connect to. Once connected, each client receives a connection object that provides access to the connection’s shared-memory heap. Channels in RPCool automatically use either CXL-based shared memory or fall back to RDMA, overcoming CXL’s limited scalability.

Each connection in RPCool is associated with a shared memory heap, enabling applications to allocate and share objects. Figure 4a–b shows how a single server can serve multiple clients by using independent heaps that are private to each connection (Figure 4a) or by using a single shared heap across multiple connections (Figure 4b). Connections start with a statically sized heap and can allocate additional heaps if they need more space. When a heap is created, the orchestrator assigns it a globally (in the cluster) unique address where the heap will be mapped in a process’s address space. Giving each heap a unique address space ensures that a client or server in cluster can safely map it into its address space.

RPCool includes support for sandboxes, which prevents any invalid or wild pointers from causing invalid (or privacy-violating) memory access as the server processes the RPC’s arguments (Figure 4c).

Moreover, RPCool supports the ability to prevent the sender from concurrently modifying RPC arguments as the receiver processes them. RPCool achieves this by dropping write access to the arguments for the sender, thus sealing the RPC (Figure 4d). When an RPC is sealed, the sender cannot modify the arguments until the receiver responds to the RPC. Sandboxing and sealing are orthogonal and can be applied (or not) to individual RPCs.

RPCool provides a thread-safe memory allocator to allocate/free objects from the shared memory heaps and several STL-like containers such as rpcool::vector, rpcool::string, etc. These containers enable programmers to use the familiar STL interface for allocating objects but do not preclude custom pointer-rich data structures, e.g., trees or linked lists. The allocator and containers are based on Boost.Interprocess (boost.interprocess, ).

RPCool’s orchestrator also requires each application that accesses shared memory to periodically renew a lease so the orchestrator can track application failures and clean up orphaned shared heaps.

Finally, to limit the amount of shared memory a process can amass, the orchestrator enforces a system-administrator-defined shared-memory quota. The quota limits the amount of heaps a process has access to at any time and requires processes to return unused heaps to the orchestrator.

RPCool uses Intel’s Memory Protection Keys (MPK) (libmpk, ) to restrict access to an application’s private memory when in a sandbox, avoiding the much more expensive mprotect() system call. To use MPK, a process assigns protection keys to its pages and then sets permissions using the per-cpu PKRU register. In MPK, keys are assigned to pages at the process-level, while permissions are set at the thread level. Since MPK permissions are per-thread, they enable support for multiple in-flight RPCs simultaneously. Current Intel processors have 16 keys available.

Once a thread enters a sandbox, it uses Intel MPK to drop access to the process’s private memory and any part of the connection’s heap except for the sandboxed region. The receiver starts and ends sandboxed execution using the SB_BEGIN(start_addr, size_bytes) and SB_END APIs. The receiver starts the sandbox with the same address and size as the scope used for the RPC. However, RPCool also supports sandboxing an arbitrary range of pages within the connection’s heap as required by an RPC.

To use Intel’s MPK-based permission control, RPCool assigns a key to each region that needs independent access control, as shown in Figure 7. RPCool uses one key each for the application’s private memory, unsandboxed shared memory regions, and every sandbox. Once a key is assigned to a set of pages, RPCool updates the per-thread PKRU register entry to update their permissions.

When an application enters a sandbox, RPCool drops access for all keys except for the one assigned to the sandbox. If the sandboxed thread accesses any memory outside the sandbox, the kernel generates a SIGSEGV that the process can choose to propagate to the sender as an error.

As the sandboxed thread no longer has access to the process’s private memory, the thread cannot allocate objects in it. However, the application may need to allocate memory from libc using malloc()/free() or invoke a library from within the sandbox that allocates private memory internally.

To address this, RPCool redirects sandboxed libc malloc()/free() calls to a temporary heap instead of the process’s private heap. After the sandbox exits, data in this temporary heap is lost. However, redirecting memory allocations works only for libraries and other APIs that free their memory before returning and do not maintain any state across calls. To safely use stateful APIs over pointer-rich data, an application can validate the pointers in a sandbox before calling the stateful API outside the sandbox.

When in a sandbox, an application cannot access the connection’s private heap, however, in some cases applications might require access to certain private variables to avoid entering and exiting the sandbox multiple times to service an RPC call. To address this, RPCool supports copying programmer-specified private variables into the sandbox’s temporary heap. To copy a private variable, the programmer specifies a list of variables in addition to the region to sandbox when starting a sandbox: SB_BEGIN(region, var0, var1...).

Although changing permissions using Intel MPK takes tens of nanoseconds, assigning keys to pages has similar overheads as the mprotect() system call (libmpk, ). To avoid assigning keys to on-demand sandboxes, RPCool reserves up to 14 pre-allocated or cached sandboxes of varying sizes with pre-assigned keys. This is limited by the number of protection keys available. RPCool reserves 2 keys for the private heap and unsandboxed regions, respectively. To service a request for an uncached sandbox region, RPCool waits for an existing sandbox to end, if needed, and reuses its key. This enables RPCool to dynamically create sandboxes without being limited to 14 pre-allocated sandboxes, albeit at the cost of reassigning protection keys.

RPCool enables the sender to enable sealing on a per-request basis and specify the memory region associated with the request. When a sender requests RPCool to seal an RPC, librpcool calls a new seal() system call. In response, the kernel makes the corresponding pages read-only for the sender and writes a seal descriptor to a sender-read-only region in the shared memory. The receiver proceeds after it checks whether the region is sealed by reading the descriptor.

Once an RPC is processed, the sender calls the release() system call and the kernel checks to ensure the RPC is complete and releases the seal. The descriptors are implemented as a circular buffer, mapped as read-only for the sender but with read-write access for the receiver. These asymmetric permissions allow only the receiver to mark the descriptor as complete and the sender’s kernel to verify that the RPC is completed before releasing the seal.

Further, as an application can have several seal descriptors active at a given point in time, the sender also includes an index into the descriptor buffer along with RPC’s arguments.

Figure 8 describes the sealing mechanism. Before sending the RPC, the sender calls the seal() system call with the region of the memory to seal. Next, the sender’s kernel writes the seal descriptor , followed by locking the corresponding range of pages by marking them as read-only in the sender’s address space .

Once sealed, the RPC is sent to the receiver. If the receiver is expecting a sealed RPC, it uses rpc_call::isSealed() to read and verify the seal descriptor , and processes the RPC if the seal is valid. After processing the request , the receiver marks the RPC as complete in the descriptor and returns the call. Next, when the sender receives the response, it asks its kernel to release the seal . The kernel verifies that the RPC is complete and releases the region by changing the permissions to read-write for the range of pages associated with the RPC .

Repeatedly invoking seal() and release() incurs significant performance overhead as they manipulate the page table permission bits and evict TLB entries (amit2020don, ). To mitigate this, RPCool supports scope pools that batch release() calls for multiple scopes. Batching releases amortize the overhead across an entire batch, resulting in fewer TLB shootdowns.

To use batched release, applications pop a scope from the pool, allocate RPC arguments within this scope, and send a sealed RPC. Upon the RPC’s returns, if the application does not immediately need to modify the RPC arguments, it can opt to release the seal in a batch. Batched releases work best when the application does not need to modify the sealed arguments until the batch is processed. However, if needed, the application can invoke release() and release the seal on the scope. In RPCool, each application independently configures the batch release threshold, with a threshold of 1024 achieving a good balance between performance and resource consumption.

RPCool notifies applications if the server they are communicating with fails and garbage collects orphaned heaps. RPCool achieves this by requiring a lease every time an application maps a connection’s heap. Orchestrator uses these leases to track which processes have failed and can notify other applications sharing the memory regions. RPCool creates a lease for each heap, and librpcool periodically and automatically renews the lease while the application is running and using the memory.

If the server for a channel fails, the lease expires and the orchestrator notifies all clients connected to the channel of the failure. The clients can continue to access the heap memory but can no longer use it for communication. They can also close the channel. When the last process accessing the heap closes the connection, the orchestrator reclaims the heap.

RPCool supports shared memory quotas to limit applications from mapping a large amount of shared memory into its address space. RPCool’s orchestrator enforces this configurable quota at the process level. A heap mapped into multiple processes counts against all of their quotas. If mapping a new heap to a process’s address space would exceed its quota, the process would need to close enough existing channels to map the new heap.

RPCool over RDMA supports communication only between one server and one client. Consequently, RPCool also does not support simultaneous access to a heap over both CXL and RDMA. While RPCool over RDMA only supports two-node communication, all other programmer-facing interfaces are identical to RPCool’s CXL implementation, e.g., allocating and accessing shared objects.

This limitation exists because when a process wants exclusive access to a page shared over RDMA, RPCool must unmap the corresponding page from all other processes across the datacenter that have access to it, which adds significant performance overheads and system complexity.

To address this limitation, RPCool includes support for deep-copying pointer-rich data structures between connection heaps using the conn.copy_from(ptr) API. copy_from() automatically traverses a linked data structure using Boost.PFR (boost.pfr, ) and deep copies to the connection’s heap, allowing applications to interoperate between connections of different types without significant programming overhead.

Sealing and sandboxing for RDMA-based shared memory pages works similarly to RPCool’s CXL implementation.

When a sender sends a sealed RPC, the corresponding pages are marked as read-only in its address space, preventing any modifications by the sender while the RPC is in-flight. Further, to process an incoming RPC over RDMA fallback, the application can create a sandbox over the RPC’s arguments in the same manner as it would for processing an RPC over CXL-based shared memory.

Table 1(a) compares RPCool’s CXL, secured, and RDMA variants against several RPC frameworks and shows that RPCool significantly outperforms all other RPC frameworks by a wide margin. Unlike RPCool, Zhang RPC attaches an 8-byte header to every CXL object and uses fat pointers (CXLRef) for references. Thus, simple operations like constructing a tree data structure require creating a CXL object and a CXLRef per tree node. Further, assigning a node as a child requires the programmer to call a special link_reference() API, adding overhead on the critical path.

Next, we look at the latency of RPCool’s features in Table 1(b). RPCool in CXL mode takes only 1.5 µs, while it takes 17.25 µs over RDMA. For CXL, this latency increases to 2.6 µs when sealing and sandboxing a single page.

RPCool’s cached sandboxes (i.e., sandboxes with pre-assigned protection key) have very low enter+exit latency at 0.35 µs. This latency grows to 25.57 µs when the sandbox is not cached and RPCool needs to reassign protection keys and set up the sandbox’s heap.

Finally, using Table 1(b), we look at the latency of memcpy() to compare it against the cost of sealing+sandboxing, which includes sealing a page, starting a sandbox over it, and finally releasing it. This is because applications can copy RPC arguments to prevent concurrent accesses from the sender without using sealing+sandboxing. We observe that for more than two pages, sealing+sandboxing is faster than memcpy() (1.45 µs vs 1.5 µs). This suggests that for data smaller than two pages, applications should use memcpy(), while for data larger than two pages, applications should use sealing+sandboxing.

Figure 9 shows the execution time of memcached running the YCSB benchmark (ycsb, ). RPCool’s CXL implementation significantly and consistently outperforms UNIX domain sockets with a speedup of at least 6.0$\times$, while the DSM implementation outperforms TCP over Infiniband by at least 2.1$\times$. As memcached transfers small amounts of non-pointer-rich data, it uses memcpy() instead of sandboxing and sealing for isolation.

For each YCSB workload, we load Memcached with 100 thousand keys and run 1 million operations. Since Memcached is a key-value store, it does not support SCAN operation and thus, it cannot run YCSB’s E workload (ycsb-scan, ).

Figure 10 shows the execution time comparison of MongoDB using RPCool vs its built-in UNIX domain socket-based communication. Across the workloads, RPCool’s CXL implementation outperforms UNIX domain sockets in all workloads except the workloads E. Moreover, RPCool’s DSM implementation outperforms TCP over Inifiniband across all workloads by at least 1.34$\times$.

Like Memcached, we evaluate MongoDB with 100k keys and 1 million operations for each workload and do not implement sealing+sandboxing as MongoDB internally copies the non-pointer-rich data it receives from the client.

CoolDB is a custom-built JSON document store. Clients store objects in CoolDB by allocating them in the shared memory and passing their references to the database along with a key. CoolDB then takes ownership of the object and associates the object with the key. The clients can read or write to this object by sending CoolDB a read request with the corresponding key. In return, it receives pointer to the in-memory data structure that holds the data.

To evaluate CoolDB, we first populate it with 100k JSON documents using the NoBench load generator (nobench, ) (labeled “build” in the figures) and then issue 1000 JSON search queries to the database (labeled “search” in the figures).

Figure 11 shows the total runtime of the two operations for the three versions of RPCool (CXL, RDMA, and Secure), ZhangRPC, and eRPC. Overall, RPCool outperforms all other RPC frameworks when running over CXL, including Zhang RPC. However, it slows down considerably when running over RDMA during the build phase, as the shared memory needs to copy multiple pages back and forth. Moreover, as RPCool does not need to serialize the dataset or the queries, it considerably outperforms eRPC for the search operation.

We evaluate RPCool using the Social Network benchmark from DeathStarBench (deathstarbench, ), which models a social networking website. In our evaluation, we replace all ThriftRPC calls among microservices with RPCool on our emulated CXL platform. However, as DeathStarBench spawns multiple new threads for each request, it contends for the kernel page table lock with RPCool’s seal() and release() calls. To address the issue, we modify the benchmark to use a thread pool instead of creating new threads for each request in both the ThriftRPC and RPCool versions. Additionally, we replaced MongoDB with its RPCool version.

We run DeathStarBench’s benchmark that creates user posts under a range of offered loads and measure the median and P99 latency, as shown in Figure 12. The experiment is run for 30 seconds for each data point. The results demonstrate that RPCool (both secure and insecure versions) and ThriftRPC show similar performance, with RPCool’s peak throughput surpassing that of ThriftRPC.

To understand why RPCool performs comparably to Thrift RPC, we looked at where a request spends its time using DeathStarBench’s built-in tracing. We found that, on average, about 66% of a request’s critical path latency is spent in databases and Nginx, suggesting that DeathStarBench’s performance is largely bound by database updates and Nginx.

Further, Figure 13 presents benchmark results with 0 µs, 5 µs, and 150 µs sleep between busy-wait iterations (Section 5.8). Not sleeping between iterations results in the best latencies but limited throughput as busy waiting consumes a significant amount of CPU. Conversely, 150 µs sleep duration results in higher tail latencies but achieves higher peak throughput. Thus, RPCool offers the flexibility to balance latency and throughput according to specific needs.

1 0