Software Hint-Driven Data Management for Hybrid Memory in Mobile Systems

Runtime profiling has been widely used to improve performance by analyzing memory accesses during application execution. Prior works [42] employed the built-in hot/cold page lists of the Linux kernel for its own profiling algorithms. Furthermore, a variety of algorithms derived from the CLOCK algorithm [11] have also been proposed [35]. Others used performance counters to identify the memory access patterns of different program phases [41]. While these techniques can improve performance to a certain extent, they only provide a system-level general evaluation, which lacks the fine grained information for each page.

In some proposals, the activity of each page is tracked by inserting metadata bits as profiling counters into page table entries [12] or TLB [24]. However, these approaches still require the OS intervention to handle page table updates or TLB shootdowns. The incurred high latencies in these processes will be further discussed in the later section.

In our prior work, we propose a Hardware Memory Manager Unit (HMMU) [40], which has its own page table that redirects each physical page address to the corresponding memory device frame address. Each table entry also has several metadata bits to count the number of recent read/ write accesses to allow for dynamic data profiling. Owing to the efficiency of its pure hardware design, the HMMU is able to instantly update these profiling counters upon each access, and make decisions on page migration coordinately. Unlike other works mentioned above, the HMMU keeps these internal activities completely hidden from the OS.

These online profiling approaches, however, are limited in the amount of profiling storage available in either the TLB’s metadata or internal state of the HMMU, and thus are constrained to profiling recent memory accesses within a small time window. As the consequence, online profiling is unable to detect complex memory access patterns that span across a long range of time.

The footprint size and time scope needed for effective profiling are illustrated in Figure 1. The figure shows the access volume heatmap generated by profiling applications 531.deepsjeng and 519.lbm from SPEC 2017, respectively. The darker dots/lines represent higher counts of accesses to the corresponding memory region. The straight black lines in the figures indicate that the applications intensively access one memory area within a short time (the dark dots comprising the lines), but quickly move forward and rarely look back. Given an HMMU with a short window of memory access tracking, it will frequently promote the pages that are currently heavily accessed but will not enjoy the benefits of fast access for future requests, as these pages will not be touched again before eventually being replaced by other pages. In other words, the unawareness of the poor data spatial locality leads to frequent page swaps as the application sweeps through its large streaming footprint.

Moreover, since the profiling only starts after pages were allocated and referenced, the data objects are initially, obliviously placed on memory devices during allocation. Thus, items with opposite access patterns, say heavily read vs. write-intensive, could end up sharing the same page, which exhibits mixed behaviors and becomes difficult to profile. In the best case, it still takes the profiler several rounds before it gained enough information to decide on a page migration. All these page migrations could have been avoided if they were placed in the correct memory type in the first place. Therefore, we propose to integrate software hints in the allocation policy.

Prior work [22] has shown that the sampling window size should be greater than a billion instructions for data objects greater than the page size (4KB). Shorter sampling epochs limit the profiler’s vision to the “local” variability, rather than the long-term memory access patterns along with program phase changes. With a comprehensive understanding of the program, programmer knowledge or offline memory access profiling (broadly we label these together as “software hints”) has been shown to be crucial for better exploiting heterogeneous memory.

Based on system-level heuristic information, Mogul [28] suggests migrating operating system cold pages based on page types and file types to the slow memory. Meswani et al. proposed TLM Malloc [25] to allow programmers explicitly allocate memory buffers on different classes of memories. Dulloor et al. [13] developed a runtime profiling framework to place program data structures in heterogeneous memory. Mukkara et al. guided the data placement in cache with static data classification based on the application memory accesses, to reduce the potential data movement [29].

To date, however, we are unaware of any HW/SW full-stack solution in the hybrid memory domain to coordinate the software-hints information with the highly efficient hardware memory controller to achieve the optimal data placement and migration.

Efficient data migration is critical for heterogeneous memory after we identify features for different data structures through user knowledge or data profiling. Prior works proposed both software and hardware approaches for efficient data migration. Generally, software data migration approaches require extensions of the OS kernel functions or data structures [6, 12, 24, 28, 35, 38, 42] or runtime/helper threads [41]. These OS-driven page migrations incur significant overheads: the process is usually triggered by a system interrupt, followed by context switch and kernel interrupt handling. A TLB shootdown might also be necessary after the page mapping was updated. The associated delays, which were once negligible compared to traditional storage media such as spinning disks, now become the dominant factors as the NVM device has much shorter access latencies. Researchers have tried different ways to mitigate the overheads of page migration: Wu et al. constrained the page migration to only happen at the end of each phase of the application [41]. Yan et al. chose to move a bundle of pages at one time to amortize the cost of OS intervention.

The hardware-based solutions typically use DRAM as an inclusive cache for the NVM device and need enormous space for tag storage, as NVM capacity is much larger than DRAM. Efforts were made to address this problem by either shrinking the tag size [19], or reorganizing the tag/data-entry structure [21, 26]. Using DRAM as a cache for NVM also adds additional access latency, and sacrifices the ability of parallel accesses to the two memory devices, since every request must go through DRAM first. Moreover, the total available memory becomes less as the DRAM space is invisible to user applications.

We successfully solved both of these issues via a HMMU, which addresses the two memory devices in one flat address space, executing the page migration transparently behind a translation layer visible only to the HMMU. In the whole process, OS does not need to step in and no changes were made to the TLB or the OS page table. We propose to combine software hints (either through domain knowledge or static data profiling) with the HMMU data migration scheme to better guide object-level data placement and migration on heterogeneous memory.

Figure 2 shows the complete system architecture. Software hints (gathered either via profiling or by explicit programmer insertion) are encoded during data object allocation. By default, our memory allocator groups objects of a given type (i.e., high writes, latency sensitive, low writes, and so on) and assigns them into a DRAM-preferred page (page S1 in Figure 2) or an NVM-preferred page (page S2) based upon the flag variables set as arguments to the memory allocation system calls. These arguments convey the preferred memory type and the presumed life cycle of the given data object (See Section 3.2 for further details on the API). These flags are communicated to the HMMU during page mapping, via a custom HMMU device driver. The device driver converts the arguments into special mark bits, encoded into the memory request messages sent to the HMMU. The corresponding decoding module inside HMMU extracts the hint information and passes this metadata to the data management policy component of the HMMU. This component executes data placement based on a set of comprehensive policies under different scenarios. When no free page is available, the HMMU scans the metadata of the internal redirection page table for the cold pages pending to be swapped out from DRAM. After mapping is finished, the memory requests are then forwarded to the allocated frames on the specific type of memory device. Besides the injection of software hints, the HMMU still dynamically profiles the memory access patterns and has its own policy of page migration. A brief introduction of the HMMU and its interaction with software-hints information is presented in the later sections.

We created a set of memory allocation APIs that allow the programmer/profiler to specify the desired memory type for the pertinent data objects. As discussed in Section 2.1, the baseline HMMU is poor at detecting long-term, complex access patterns, as its limited hardware resources only support monitoring of short-term behavior. With the help of software hints, we can identify the data objects that have reoccurring accesses between intervals. Such data shall be preferentially kept in the DRAM. HMMU profiles data objects by observing the received memory requests, thus it carries no relevant information at the time of allocation. Such oblivious allocation often places mixed data types together, leaving negative impacts on the data management throughout the object’s lifetime. Now that software hints are provided during allocation, we can group similar data into the same page and avoid unnecessary potential data migration.

Our memory allocator is an extension of Memkind by Intel [1]. We changed the underlying memory mapping functions to ensure that the application memory was allocated to the HMMU’s memory. The APIs are compatible with the default glibc malloc function:

The programmer/profiler can either label particular data objects as DRAM-preferred, or leave the flag blank, allowing the allocator to handle it with default settings. Furthermore, the “retention” argument is provided as a software hint to indicate how long this data object should be held in DRAM.

The software hints collected from the software stack are forwarded to a HMMU [40]. HMMU is a memory controller unit that receives the memory requests from the CPU and addresses NVM and DRAM devices in one flat memory space. It has an internal redirection page table that translates the incoming physical address to the mapped memory frame on the memory device.¹ The recent activities of each page are captured into metadata bits on the page table entry. Based on these profiling information, we designed and implemented a comprehensive policy of both data placement and migration. The built-in DMA engine can directly move data between two memory devices, without blocking the incoming requests from the CPU. These operations are all hidden from the OS and are executed highly efficiently in hardware.

When the HMMU receives the first memory request to a “DRAM-preferred” page, it assigns a free page in DRAM if available. Otherwise, the requested address is translated to the currently mapped memory frame in the NVM device. It also sets a special bit for this page in the HMMU’s page table, which will trigger the page migration to DRAM next time it gets referenced.

Since DRAM pages are a limited resource, mechanisms are required to ensure that pages marked as “DRAM-preferred” are not locked into DRAM, if in fact they are not being written often, or if DRAM pressure is otherwise high. We built a counter in the page table entry to reflect the presumed life cycle of the data objects allocated to the pertinent page. When a page is mapped or migrated to the frame in DRAM, the HMMU sets this counter with the value passed via the argument “retention” in the API (see Section 3.2). Since we focus primarily on the bulk memory allocations that absorb a large amount of written/read bytes, such flagged pages are usually fully allocated in one single call to homogeneous data objects. In the rare case when a page is partitioned among different data objects, we override the counter value with the largest retention period of all pertinent allocation. During every refresh epoch, the counter decrements by 1 if no write requests are received. When the counter decreases to zero, the page loses its special status and becomes a normal page susceptible to being swapped back to NVM. The detailed algorithm is listed in Algorithm 1.

Since programmers/users are not necessarily always aware of the actual memory access patterns of all their data objects, especially after cache filtering, their preferences of memory type are only passed to HMMU as hints, rather than the determinant of the data management. Thus, the system is free to ignore some or all of these hints in the event that the requested DRAM space is larger than the actual DRAM size. Throughout the different phases of applications, if the number of references or writes declines, the data object will “expire” and be swapped back to NVM, even though it was marked as “DRAM-preferred” by the user when allocated. This coordination between HMMU and the software stack enables the design to have both long-term and short-term scope in data management.

When pages are marked with software hints, they have a higher priority in the competition for the limited DRAM pages. These demands add stress to the memory system and may lead to thrashing among the other pages after the memory footprint grows to a certain extent. Frequent page swaps incur a waste of memory bandwidth and energy, obviating any benefit from leveraging the hints. Therefore, we need to find a new way to throttle the data migration.

A typical scenario is in streaming applications, which traverse a vast range of addresses without recurrence. To deal with such applications lacking locality, we designed the adaptive throttling mechanism on our data migration policy. For the full page migration, the HMMU counts the references to the page after it is migrated to DRAM, in each refresh epoch. If the average number of the last 128 accessed DRAM pages is larger than that of the last refresh epoch, we lower the bar of full page migration. Such applications have strong spatial locality and it’s worthwhile to perform full page migration, as the following access all hit in DRAM with shorter latency. Alternately, if the page was referenced less frequently than before, we raise the threshold to suppress page migration.

Ocean and streamcluster are the two benchmarks that do not benefit much from OpenMem in terms of energy saving and runtime performance. Ocean’s data structures are designed in a way that prevents contiguous memory allocation, leading to poor locality. Its miss rate in cache is the second highest across all parsec benchmark applications. Moreover, this workload is highly sensitive to cache capacity, and the miss rate is two orders of magnitude higher for small caches. Streamcluster is a data-mining application. It is reported to have the most core-to-bus data transfers [3]. Consequently, streamcluster is very sensitive to memory bandwidth and access latency. Therefore, the performance is expected to degrade as we replace 7/8 of the DRAM resource in a system with NVM which has substantially higher latency. Both applications have poor data locality and their performance decline drastically with small CPU cache capacity. Thus, we consider the performance loss is mainly attributed to other bottleneck parts of our testing system, other than the memory management itself. OpenMem obtained the largest edge over HMMU-only policy in regard to NVM writes reduction with two benchmarks, 508.namd (18%) and ocean (22%). We inspected the number of marked pages that remained in the DRAM after each application was completed and found that they have the highest percentage: 508.namd (46%) and ocean (69%). This metric to some extent shows how closely the software hints match the actual accessed pattern of the marked data objects throughout the application. As we mentioned in Section 3.6, the HMMU still makes independent decisions on data management even after accepting the software hints. Thus, when the marked data objects went cold after application switching to the next phase, they will not retain the precious DRAM resource. We infer that the benefits of having software hints were maximized with those data objects that keep receiving frequent write references for the whole duration.