AMC: Access to Miss Correlation Prefetcher for Evolving Graph Analytics

Applications that use graph-based algorithms and data structures in real-world scenarios where the relationship constantly evolves between entities are known as evolving graph applications. This work uses two types of evolving graph application: iterative graph algorithm with early convergence and graph applications with changing input graphs. Early convergence iterative graph algorithms, like PGD [53] offer an optimization over PageRank [44]. PGD typically requires fewer iterations as compared to PageRank and has a faster runtime. This is possible because PGD only updates vertices in an iteration whose PageRank value has changed by more than some $\delta$-fraction. Therefore, in every iteration, a set of active vertices are involved in the PageRank calculation, resulting in less computation but non-repetitive irregular memory access patterns. Section III discusses performance challenges with such irregular patterns. For graph applications with changing input, the method explained in Section VI is used as the inputs to dynamic graph applications, which is similar to prior work [70, 35].

Evolving graph applications typically use a frontier array, which is a bit map, to keep track of the vertices participating in the upcoming iteration or computation. This establishes a one-to-one correlation between the frontier and vertex accesses. Additionally, the inherent vertex-to-neighbor correlation is a one-to-many correlation between the vertex and its neighbor accesses. This can be used to fetch data structures related to the vertex present in the frontier. AMC take advantage of these two properties of evolving graphs to build the correlations explained in Section III.

AMC is a hardware-software cooperative prefetcher. The closest related works in the same category are RnR [68], DROPLET [8], and Prodigy [56].

DROPLET [8] uses a specialized malloc function to identify a graph application’s targeted data structure (vertex and vertex property). DROPLET generates the addresses for an indirectly accessed vertex property value by prefetching the edge array. DROPLET [8] triggers vertex data prefetching only when DRAM services edge, which is often too late [68]. Prodigy [56] uses either compiler profiling or program annotation to generate data flow graphs of graph data structures. It uses demand access to the vertex node to prefetch the next vertex node and waits for the vertex node to be filled at the destination cache to initiate prefetch for its outgoing edges using the prefetched data. This requires a complete software stack change, including rewriting code, compiler, and OS just to optimize the prefetching of graph data structures. The RnR prefetcher [68] targets on long, repetitive, irregular memory access patterns in iterative algorithms. It improves cache miss coverage and accuracy by recording in the initial iteration and replaying the miss patterns for prefetching in the following iterations. In dynamic graphs, wherein the vertex/edges change over time, RnR does not work well. AMC solves this issue by recording the access-to-miss correlations that are preserved in dynamic graphs.

Similar to Prodigy, a case study on a HW-SW cooperative prefetcher presented in MetaSys [60] also rely on sequential dependency between data structures. For dynamic graphs, another limitation of these prefetcher  [56, 60, 3] is their inability adapt to runtime dynamics (conditional branch). For example, PGD avoids redundant computation by examining only the vertex whose page rank value changed by a set threshold in the previous iteration. These prefetcher fails to account for control-flow knowledge for prefetching. AMC overcomes the dependency challenge by using a single data structure as the triggering data structure to prefetch all of the other misses. In order to adapt to vertex and edge changes in the dynamic graphs, AMC continuously updates the correlations in every iteration and uses the latest one to prefetch.

Temporal prefetchers [66, 67, 26, 23] record memory access and then correlate it to either its PC or the previous access. They typically have high metadata storage overhead because they store a long sequence of memory addresses and inability to delete useless metadata. The closest related works to AMC in this category are ISB [23] and MISB [67]. ISB [23] uses TLB and structural addresses to map physical memory addresses to structural addresses and store them using PC localization. It suffers from high metadata overhead, does not scale with large page sizes, and does not work with modern hierarchical TLBs. MISB [67] solves this problem by employing the next-line prefetcher for metadata access as the structural address space is spatial and removing the TLB dependency to manage metadata caches. Unfortunately, when application's input size grows, the metadata also grows. The problem is more severe for on-chip only prefetchers like Triage [66] because they do not have off-chip metadata to fall back on to record growing metadata. AMC prefetcher continuously updates the correlation table with only latest ones and compresses the metadata to reduce the storage overhead. Temporal prefetchers [23, 67, 66, 26, 18] also suffer from aliasing problem [65]. Multiple addresses can correlate to the same trigger event, which causes aliasing. AMC prefetcher solves this problem by linking multiple target accesses with miss stream, thus adding contextual information to correlation (Section III). DVR [42] is a recently proposed architecture over VR [41] targeting C[hash(B[hash(A[i ])])]. DVR utilizes vector functional units and vector registers to execute indirect memory instructions in advance. It groups together instructions with the same offset to a single vector instruction. However, DVR’s progress in extracting MLP can be impeded if a previous instruction encounters branch misprediction, and it heavily relies on core structures such as ROB, VRAT, and stride detectors. An evolving graph uses a conditional branch instead of a hash function and may not utilize vector functional units effectively. However, it may be less effective in scenarios where the number of iterations is low, such as in BFS, because the DVR points themselves. This is according to a study by [42]. In contrast, AMC is decoupled from the core’s microarchitecture components. It records and intelligently replays indirect memory data structure to improve performance. It depends on the previous iteration recording to extract MLP and fully utilize L2’s MSHR without competing demand loads.

Spatial prefetchers  [51, 6, 15, 19] exploit the address delta similarity between cache accesses among different memory regions, which arise due to a fixed and regular memory layout of data objects. Such memory address patterns are common in server applications [48] (e.g., OLTP, DSS). The advantage of such spatial prefetchers is that they require less metadata. VLDP [51] targets irregular access patterns within a page. Unlike a regular access pattern prefetcher, VLDP tries to predict a common pattern amongst past deltas. It uses TAGE-like table [50] to solve the aliasing, leading to better accuracy. TAGE-like history refers to using multiple history lengths stored in various tables. In this approach, the prefetcher looks up multiple history tables to generate predictions rather than depending on a single history table for predicting future memory access patterns. It considers multiple history lengths and offers better prediction accuracy than a single history table. Moreover, it can adapt to changing application phases and capture complex correlations between memory access patterns, making it applicable to dynamic graph applications. Finally, it lowers the aliasing probability by utilizing multiple history tables. BINGO[6] also uses an optimized Tage-like table wherein the multiple history tables are fused into a single unified table and looked up multiple times with different history lengths. This reduces the overall storage overhead of traditional Tage-like predictors. AMC also uses multiple trigger accesses. The novelty of AMC is to use accesses of only the targeted data structure as the trigger, which helps to improve the prefetching accuracy for evolving graph applications.

Software-based prefetching [4, 40, 2] for linked data structures requires programmer/compiler analysis to identify pointer-chasing access responsible for cache misses. This often requires significant effort to generate effective prefetch requests well ahead of demand requests to generate timely prefetches. Ainsworth and Jones proposed [1] a configurable prefetcher aimed at improving performance for graph workloads. However, it only targets specific traversals for a certain graph format. Event-triggered programmable prefetchers [3] employs an array of mini programmable prefetcher units to target heterogeneous access patterns using compiler profiling and maximize memory level parallelism, particularly in A[B[C[i]]], wherein array C can be prefetched in before its demand, which can lead to the prefetching of arrays B and C. ATP[10] explains the hardware complexity of Ainsworth and Jones’s proposed prefetcher for indirect memory access and has timeliness problem similar to DROPLET [8]. ATP uses instructions to communicate data structure knowledge and a similar strategy as IMP to calculate linked data structures. Additionally, AMC does not rely on data-based prefetching, but instead relies on its previous recordings for prefetching. These compiler profiling [4, 40, 2, 1, 3, 24] require software stack change, including rewriting the code, updating the compiler, and changing the operating system to optimize the prefetching of data structures. Additionally, they do not adapt well to run-time changes due to context switches or speculation misprediction. Furthermore, software prefetching increases the overall instruction code size.

Table I summarizes the key differences between AMC and its closely related prefetchers.

Accelerators for dynamic graphs have been proposed [70, 13, 47, 36, 9, 62, 69] as stand-alone accelerators or near/in-memory processing engines. These accelerators often necessitate custom hardware design and programming models. AMC leverages the existing software and hardware framework and makes modest modification of the current system to provide performance improvements comparable to those of the dynamic graph accelerators. These accelerators employ graph prefetchers responsible for prefetching neighbors and their property data, which uses a similar strategy as  [56, 60, 8] to prefetch graph data structures and suffer sequential dependency between graph data structures.

AMC requires the following additional architectural registers: (1) two pairs of address range registers to hold the start address and size of the target and frontier data structure, (2) a prefetch phase register to enable prefetching after an initial iteration and recycle the off-chip AMC space in successive iterations, (3) target access count register, (5) miss count register and (6) four pairs of off-chip AMC storage registers to hold the head and tail pointers for the current and the next AMC miss addresses and AMC index storage.

AMC uses the address space identifiers (ASIDs) to distinguish access streams from different processes to do permission checks. The target recorder and frontier buffer in Fig. 4 use a pair of address range registers to filter out the target and frontier accesses from L1 data load accesses. OS allocates off-chip memory space to store both the current and the next AMC miss addresses and AMC index storage on AMC.init() function call. AMC reserves up to 20% input size for off-chip AMC storage (Section VII-D). One can re-purpose unused architectural registers as these special registers for AMC prefetcher. An ASID register stores the ASID of the current process using the prefetcher. Target access count register count number of L1 data cache access performed to target data structure (Section V-A). Miss count register counts the number of L2 misses recorded per AMC entry used by compressor unit (Section V-B). Target access count is used to identify an unique target access during an iteration, whereas the “miss count” counts the number of L2 misses following a target access.

The AMC.AddrTBase(addr, size), and AMC.AddrFBase (addr, size) function provides the system with information of the target and frontier data structure. The setting of the target address range register happens at the memory allocation time. Using target address range, AMC prefetcher can recognize whether access is within the target range (Section V-A). AMC.update() controls prefetching phase register, metadata storage management, and reset the target access count register (Section V-A). A set state of the prefetching phase register denotes the prefetching is enabled, whereas an unset state of the prefetching is disabled. OS does not disable prefetching except during the context switch (Section IV-B). AMC.end() function frees up AMC off-chip storage and reset all the AMC architecture register and invalidate all the entries in AMC Cache at the end of the execution.

Furthermore, the OS needs to allocate off-chip memory space to store two AMC miss addresses and AMC index for recording phase and prefetching phase (shown in Fig. 4). One stores the correlation from the previous iteration to perform prefetching and another to learns the correlation in the current iteration. OS maintains the off-chip address range registers. At the iteration boundary, the AMC invalidates its prefetching phase memory space and reuses it to store the upcoming iteration’s correlation. In short, both recording phase and prefetching phase perform role reversal at the iteration boundary.

The OS is responsible for the process management, interrupts service, I/O, virtualization, and resource management of different cores in the system. In case of long latency events such as page faults or interrupt service routine, the OS needs to switch out the current process, known as context switch, and handle the event. Conventional prefetchers either flush the metadata entries or save them in memory to retrieve them later. AMC prefetcher can reuse its old metadata after being context-switched back again only when there is no swapping of physical pages from the process. During a context switch, the physical pages swap when memory runs out of physical pages to allocate to the new context switch process, which is not typical. Suppose the page’s swap, the AMC resets its metadata, disables prefetching, and restarts from the recording phase. The dynamic graph applications consist of multiple iterations. Therefore, AMC can quickly perform its recording phase in the current iteration and start prefetching in the next iteration.

The PGD algorithm from Ligra [53] suite is modified to demonstrate how to use AMC's programming interface (Algorithm 1). Line 1 - 9 initializes the data structures used in the algorithm. Line 11 - 13 calculates the page rank value for the vertex present in frontier (active vertex in current iteration). Line 15 - 18 calculates the set of vertices for the next iteration. Purple-colored function calls are AMC functions. Line 21 initializes the AMC prefetcher registers, allocates off-chip memory for AMC metadata storage, and resets all of the AMC Cache entries (Section V) as well as all of the architectural state registers. Line 7 and 8 defines the virtual address range for the target data structure and the frontier with their corresponding size (N is the number of elements).

Line 27 denotes the boundary of an iteration. It sets the prefetching phase register to enable prefetching after the initial iteration. Additionally, after every iteration, it invalidates the off-chip AMC storage used for prefetching miss stream for the current iteration and reuses it to record in the next iteration. It does not invalidate the correlations recorded in the current iteration. Finally, line 28 terminates the AMC prefetcher and frees up its off-chip memory space.

AMC records many-to-many correlations between accesses to a target data structure and L2 misses. Target data accesses identified in the L1 cache act as trigger events to prefetch to the L2 cache. Hence, the AMC needs to record target data structure’s accesses at the L1 data cache and the following misses from the L2 cache to build the correlations. An AMC correlation entry consists of two target accesses (2$\times$64 bits) and up to 20 misses (20$\times$46 bits). A Target Recorder is used to identify the target L1 accesses, which can hold up to two most recent target accesses. The target recorder includes a target access counter that increments on every demand target access to the L1 data cache. This counter is reset when the AMC.update() function is invoked.

The L2 misses that do not belong to the target address range are tagged with the latest target access count value at the time of the L2 miss and forwarded to a Binder to build a correlation entry. A Miss Count holds the number of misses belonging to the same target access count. When a miss with a different target access count value arrives at the Binder or when the Miss Count reaches 20, this entry is compressed and sent to memory, the Miss Count is reset, and a new correlation entry is initiated. The access count retrieves the correlated target accesses in the Target Recorder.

The AMC prefetcher maintains two off-chip metadata storage simultaneously: one for recording phase, the other for prefetching. Each of these two metadata storage contains two tables: Miss Addresses and AMC Index. AMC prefetcher stores the correlations learned during the current iteration and uses the correlations learned from the last iteration for prefetching. At the end of every iteration when AMC.update() is evoked, the head pointers of these two memory spaces are swapped to allow the latest correlations to be used for prefetching while recycling the memory space for recording new correlations. This continuous learning while prefetching helps to capture the changes in vertices and edges of the dynamic graphs.

The compressed miss addresses store miss address entries with different sizes in compact format. When a new entry is compressed and sent to memory. The tail pointer of Miss Addresses of recording phase is incremented based on this entry’s compressed size for storing the next entry. The AMC Index store the target addresses and metadata of each correlation entry. Each AMC Index entry consists of two target addresses, compression mode, a pointer to the Miss Addresses, and the number of correlated misses.

A lightweight Base$\Delta$ compression variant [46] is designed to save off-chip bandwidth and reduce metadata storage. An AMC entry stores up to 20 physical addresses (52-bit physical memory) without block offset of the misses (20$\times$46 bit without compression). AMC uses the physical block address of the first miss as the base (46 bit) and three different sizes of deltas (1, 2, or 4 byte) to compress the other misses. A 2-byte-$\Delta$ example is shown in Fig 6. When all of the addresses can be represented with Base+$\Delta$, this entry can be compressed with the corresponding size of delta. Fig 5 illustrates a high-level view of the compressor design. The compressor uses the Miss Count to activate the number subtraction units equal to the number of misses. Three delta sizes are tested in parallel. The smallest compressable delta size is selected using delta selection logic.

The target access addresses do not undergo compression because the target addresses play a critical role in AMC Cache, which will be explained in Section V-C1 and Section V-C2. Compressing the target access addresses adds delay to the critical path and might lead to late prefetches. Instead, only the delta of the target accesses is recorded by the target recorder using target start address stored in the architectural address range register (Section IV). For the evaluated workloads, the compression ratios for 20 recorded uncompressed misses using different deltas are 4.5 (920/206) for 1-byte-$\Delta$ (best-case), 2.51 (920/366) for 2-byte-$\Delta$, and 1.34 (920/686) for 4-byte-$\Delta$ (worst-case).

AMC uses an on-chip cache to prefetch the access-to-miss address correlations recorded during the last iteration. These correlation entries are inserted into the cache in sequence and evicted in FIFO order. Fig 7 illustrates a high-level view of the AMC Cache design. The three main components of AMC Cache are (1) AMC Cache’s tag, storing target addresses as the tags for cache lookup, (2) target metadata RAM, storing the location of the corresponding entry in the AMC Cache, and (3) a Compressed Miss RAM, stores the compressed misses.

Fig 14 shows the energy breakdown and comparison between baseline and AMC prefetcher. The energy consumption for AMC prefetcher reduces in all the categories (core, cache, memory). AMC prefetcher consume on an average 1.28$\times$ less energy than baseline. This is chiefly because of reduced static energy consumption of core, caches, and DRAM because of reduced overall execution time.

AMC requires a moderate amount of logic per core and set of architectural and internal registers (Section IV) for compression and developing the correlation between L1 data accesses and L2 misses. The AMC Cache requires about 29 kB for each core (78.3E^-3 mm²) for each core. Base$\Delta$ compressor unit occupies about 13.3E^-3 mm² per core. The overall on-chip area is 0.2% of the total on-chip area (46.19 mm²).

From Fig 15, the off-chip storage is always below 25% of the input size. If the kernel access pattern shows poor spatial locality, AMC needs to record a missing stream that can span multiple pages, which reduces the compression ratio of the overall missed stream.

To select an appropriate miss stream size of an AMC entry. AMC assumes infinitely sized AMC Cache with infinitely sized miss stream size of an AMC entry. Fig 16 shows that the number of AMC entries with a miss size greater than 20 is less than 1% for the evaluated kernel and input. Another observation is 20 misses per AMC entry ensures evaluated kernel-input pair cover at least 74% of entries. Increasing the number of misses beyond 20 does not yield much performance improvement that justifies the additional hardware overhead compared. Consequently, AMC records 20 misses per entry.