An OpenMP Runtime for Transparent Work Sharing across Cache-Incoherent Heterogeneous Nodes

To support existing OpenMP applications, libHetMP must be compatible with existing semantics and therefore abstract all mechanisms behind runtime entry points. libHetMP inserts migration points when starting thread teams to execute OpenMP parallel regions across nodes. After thread migration, libHetMP internally separates runtime chores (work distribution, synchronization, performance monitoring) into per-node and global operations to minimize DSM traffic generated by the runtime itself. The runtime must (1) organize and place threads across nodes to minimize cross-node communication and (2) distribute work, including measuring application performance across and within nodes to adjust distribution decisions.

Cross-Node Execution. When starting a parallel region, libHetMP organizes threads into a thread hierarchy to break synchronization down into per-node and global operations, significantly reducing the amount of cross-node traffic caused by synchronization. At application startup, libHetMP queries the system to determine each node’s characteristics, i.e., type and number of CPUs available, and uses this information to initialize the thread hierarchy. As threads are spawned or re-initialized, libHetMP calls the OS’s thread migration function to physically place threads according to the hierarchy. The runtime migrates threads to nodes based on how many threads are executing the parallel region and how many CPUs are available on each node—for example, in a setup containing a 16-core Xeon and a 96-core ThunderX, libHetMP spawns and places 16 and 96 threads, respectively, for a total of 112 threads. Internally, libHetMP initializes per-node data structures like loop iteration scheduler metadata, barriers, and counters based on the hierarchy. libHetMP also initializes global variants of these data structures for cross-node synchronization (see below). libHetMP allows re-configuring the thread hierarchy between parallel regions, which is useful for dynamically adjusting parallel execution.

Synchronization. libHetMP uses the thread hierarchy for many types of synchronization, including barriers, reductions and work distribution. libHetMP builds upon previous work by Lyerly et al. [36], which refactors OpenMP execution for DSM on homogeneous clusters. Figure 3 illustrates using the thread hierarchy for synchronization. When synchronizing, threads on each node elect a node leader to act on their behalf at the global level and the remaining non-leader threads wait at per-node barriers. The node leaders act as representatives for the node and communicate through global data structures using DSM. The leader/non-leader designation significantly reduces cross-node communication as most threads do not touch global data (in the Xeon/Cavium setup, only two threads touch global data instead 112). For example, when executing parallel reductions the first thread on a node to arrive is elected leader and waits for all non-leader threads to make their local data available for reduction. Once the leader has reduced data from all threads on its node, it produces the node’s data for the final reduction at the global level. A global leader, elected in the same fashion, is responsible for reducing data from all nodes.

Workload Distribution. OpenMP defines several loop iteration schedulers that affect how iterations of a work-shared parallel loop are mapped to threads. The default loop iteration schedulers (static, dynamic) implement several strategies with the goal of evenly partitioning work to avoid overloaded straggler threads from harming performance. libHetMP provides the ability to distribute iterations across nodes by extending these schedulers to account for heterogeneity and to efficiently synchronize how threads grab iterations. Due to limitations in each (see below), libHetMP introduces the HetProbe scheduler for automatic iteration distribution in consideration of CPU and interconnect.

libHetMP assumes each node in the system contains a set of homogeneous CPU cores with identical micro-architecture and cache coherence. To quantify performance differences between nodes, libHetMP defines a core speed ratio (CSR) to rank the relative compute capabilities of individual CPU cores on one node versus another. For example, a Xeon core with a core speed ratio of 3:1 compared to a ThunderX core means the Xeon core is considered 3× faster than a ThunderX core and threads running on the Xeon will get 3× as many loop iterations as threads on the ThunderX. Note that CSRs are assigned to each work sharing region, as applications may have multiple work sharing regions that exhibit different performance characteristics.

Cross-node static scheduler. OpenMP’s static scheduler evenly partitions loop iterations among threads, assigning each thread the same number of iterations. The scheduler implicitly assumes all CPUs are equal and all loop iterations perform the same amount of work. Rather than considering all threads equal, libHetMP allows developers to specify per-node CSRs to skew work distribution for threads on different nodes. The challenge, however, is that developers must manually discover the ideal CSR for each work sharing region and hardware configuration through extensive profiling.

Cross-node dynamic scheduler. With OpenMP’s dynamic scheduler, threads continuously grab user-defined batches of iterations from a global work pool using atomic operations on a global counter. This scheduler targets work sharing regions where individual loop iterations perform varying amounts of work. libHetMP optimizes grabbing batches using the thread hierarchy; threads first attempt to grab iterations from a node-local work pool instantiated during team setup. If the local pool is empty, then the thread grabbing iterations is elected leader and transfers iterations from the global pool to the per-node pool. Because the leader represents the entire node, it grabs a batch of iterations for each thread executing on the node. This reduces the number of threads accessing the global pool and thus the amount of global synchronization required for work distribution.

While not traditionally meant for load balancing on heterogeneous systems, the dynamic scheduler can load balance work distribution based on the compute capacity of CPUs in the system. However, continuous synchronization both at the local and global level to grab batches of work can negatively impact performance, especially with small batch sizes. Users must again profile to determine the ideal per-region and per-hardware batch size. Non-deterministic mapping of loop iterations to threads can also cause “churn” in the DSM layer for applications that execute the same work sharing region multiple times. With deterministic mapping of iterations to threads, data may settle after the first invocation as nodes acquire the appropriate pages and permissions. The dynamic scheduler prevents data from settling on nodes.

The main problem with the default schedulers is that users must extensively profile to find the best workload distribution configuration in a large state space, i.e., determine CSRs or batch sizes for each individual work sharing region on every new heterogeneous platform. Additionally, if cross-node execution is not beneficial for a work sharing region due to large DSM overheads, then users must profile to determine the best CPU for single-node execution and manually reconfigure the thread team (including the thread hierarchy) to only execute work-sharing regions on the selected CPU.

To avoid the tuning complexity of the default schedulers, libHetMP introduces a new heterogeneous probing or HetProbe scheduler, for automatically configuring execution of parallel computation. Developers only need to specify the HetProbe scheduler like other OpenMP schedulers, e.g., adding a schedule(hetprobe) clause to a work sharing region, and libHetMP will transparently handle distributing loop iterations to available CPUs. Developers do not need to reason about DSM overheads or performance characteristics of individual nodes. The HetProbe scheduler executes a small number of iterations across both nodes, called the probing period, during which it measures per-core execution time, cross-node page faults and performance counters to analyze a work sharing region’s behavior. The HetProbe scheduler is designed to optimize the performance of work sharing regions with regular loops, where the behavior of one loop iteration is a good predictor for the behavior of other iterations. The HetProbe scheduler uses the performance analysis information gathered during the probing period to distribute the remaining iterations as described in Section 3.4.

The HetProbe scheduler must be precise when distributing iterations for the probing period to accurately evaluate system performance. First, the scheduler issues a constant number of loop iterations to each thread, regardless of node, to compare the execution time of equal amounts of work on each CPU. Second, the scheduler must deterministically issue iterations, so that threads executing a work sharing region multiple times receive the same batch of iterations across invocations to account for the aforementioned data settling effect. If the HetProbe scheduler non-deterministically distributes probe iterations, then data might unintentionally churn and cause falsely higher DSM overheads.

libHetMP also implements a probe cache for applications that execute a work sharing region multiple times. This has two benefits: First, it allows the runtime to reuse previously calculated statistics and workload distribution decisions from previous probing periods to avoid probing overheads. Second, libHetMP uses multiple probing results to smooth out measurement variation for shorter-running work sharing regions. libHetMP uses an exponential weighted moving average for measurement statistics, which favors more recent measurements and quickly converges on accurate values. libHetMP uses this type of average, because initial probing values for regions may be inaccurate due to the DSM layer initially replicating data across nodes, whereas subsequent executions may incur fewer DSM costs.

Some workloads exhibit an initial behavior that is not representative of their entire execution. For instance, an application could initially be memory-intensive setting up data structures, while the rest of the time it uses that memory for compute-intensive operations. Based on the initial probing period, the HetProbe scheduler will distribute iterations favoring nodes with better memory performance. In these irregular workloads (previously described in detail in Section 2) it can be advantageous for the HetProbe scheduler to periodically re-probe so that the work distribution can be adapted to the changing behavior. libHetMP contains an extension to the HetProbe scheduler, called HetProbe-I, that adapts to irregular workloads by triggering a new probing period. We refer to these new probing periods as reprobes. HetProbe-I uses the profiled performance metrics from reprobes to calculate new CSRs and redistribute work among nodes.

Figure 4 illustrates how HetProbe-I schedules a work sharing region with 100 iterations (excluding the initial probing period). Part 1 shows the global work queue, divided using the metrics collected from the initial probing period. In this scenario, HetProbe-I has assigned the first 60 iterations to node A and the remaining 40 to node B. Then in Part 2, after each node has completed a certain number of iterations, HetProbe-I triggers a reprobing period to re-evaluate the distribution. Once the reprobe phase concludes, Node A has finished iterations 0 to 29 and Node B iterations 60 to 80. Finally, in Part 3, the remaining iterations not executed by any node are joined to generate a new queue (iterations 30 to 59 and 81 to 100) called a jump. Note that HetProbe-I needs to make sure that all iterations in between (60 to 80) are not executed again.

Hence, the two main challenges of this new scheduler are as follows:

To address the first challenge, HetProbe-I logically breaks the work sharing region into multiple smaller work-sharing regions, each with its own probing and workload distribution decisions. In HetProbe, the scheduler calculates a CSR and distributes all remaining loop iterations to nodes. HetProbe-I instead only distributes a fraction of iterations, which forces threads to re-enter the OpenMP runtime and allows HetProbe-I to check whether to reprobe. Currently, HetProbe-I triggers a reprobe after executing a user-defined percentage of iterations. libHetMP includes an OpenMP environment variable OMP_HET_PTG that is used to specify after what percentage of iterations—from the total in the work-sharing region—should the scheduler trigger a reprobe. HetProbe-I sets this variable to 10% by default. HetProbe-I checks this condition every time a leader thread attempts to acquire more iterations from the node work queue. Hence, the execution flow is an iterative process consisting of the initial probe and execution, reprobing and execution with the new global queue and jumps, reprobing again and so forth. HetProbe-I is designed in a modular fashion so that more sophisticated methods can be applied in the future to determine whether or not to reprobe.

At some point, HetProbe-I determines it needs to prepare and carry out the reprobing. For this second task, HetProbe-I leverages libHetMP’s hierarchical barriers, as the leader thread that triggers the reprobing will have to wait for the other node(s). HetProbe-I also needs to take precautions not to execute completed iterations when it rearranges the global work queue. In particular, when it combines the pending work of one node from iterations i to j and from another node with iterations k to m, HetProbe-I ends up having a new global set of iterations from i to m. In the previous example with Figure 6, this new queue would be from iterations 30 to 100 as shown in Part 3. Because OpenMP operates over continuous sets of iterations, iterations j to k (in the example, iterations 60 to 80) were already completed by one of the nodes. HetProbe-I avoids re-distributing these already completed intervals by building discontinuous sets of work. We refer to these interval sets as jumps and deal with them by identifying, labeling and adapting to them with modified iteration distributions in the work dispatching process. In this way, HetProbe-I provides threads with continuous subsets of iterations.

The HetProbe schedulers (both HetProbe and HetProbe-I) use the execution time, page faults and performance counters measured during the probing period to determine where to execute parallel work. Specifically, the HetProbe schedulers answer three questions:

1. Should the runtime leverage multiple nodes for parallel execution? While coupling together multiple CPUs provides more theoretical computational power, not all applications benefit from cross-node execution. As mentioned in Section 2 there is a significant cost for on-demand data marshaling and page coherency across nodes. To understand DSM overheads, we ran a microbenchmark that varies the number of compute operations executed per byte of data transferred over the interconnect. Because there are no server-grade heterogeneous-ISA CPUs integrated by point-to-point interconnects, we approximate a system using the experimental setup shown in Table 1 and evaluated the DSM layer using two protocols, TCP/IP and RDMA.

The microbenchmark spawns one thread for every core in every node in the system. It then runs a control loop that stresses each node (i.e., each \((architecture, interconnect)\) pair) connected to the source node, i.e., the Xeon, because it runs the single-threaded portion of applications. At the start of the control loop, the source node threads initialize memory by touching all data pages to force the DSM protocol to bring all pages back to the source node’s memory. The control loop then releases the other node’s threads (ThunderX) to begin timed execution. Each ThunderX thread touches non-overlapping sets of pages to force the DSM protocol to transfer them to ThunderX memory. Finally, the ThunderX threads perform varying amounts of compute operations per page transferred. The microbenchmark calculates operations/second (incorporating the DSM costs) by timing how long it takes to execute the loop to determine the break-even point where cross-node execution is beneficial.

Figure 5(a) shows the compute throughput in millions of floating point operations per second when varying the number of compute operations per byte of data transferred over the interconnect. Figure 5(b) shows the average page fault period, i.e., elapsed time between subsequent page faults. Intuitively, as threads perform more computation per byte transferred, the computation is able to amortize the DSM costs and reach peak throughput. As shown in Figure 5(b), there are significant latency differences between RDMA and TCP/IP. Page faults using RDMA cost around 30 microseconds, whereas they cost 90 and 120 microseconds for the Xeon and Cavium servers, respectively, with TCP/IP. Thus, the amount of computation needed to amortize DSM costs when using TCP/IP is significantly higher than RDMA.

To determine if cross-node execution is beneficial, the HetProbe schedulers calculate the page fault period by measuring execution times and number of faults. The break-even point when cross-node execution becomes beneficial can be seen in Figure 5(a) when the microbenchmark is close to maximum throughput: above 512 operations/byte for RDMA and 32,768 operations/byte for TCP/IP. Correlating these values to Figure 5(b), the runtime uses a threshold of 100 \(\mu\) s/fault for RDMA and 7,600 \(\mu\) s/fault for TCP/IP to determine whether there is enough computation to amortize DSM costs and benefit from executing across multiple CPUs. As faulting latency drops (e.g., if CPUs share physical memory), fewer compute operations are needed to amortize cross-node memory access latencies. When the interconnect between CPUs changes, this microbenchmark can be re-used as a tool to automatically determine the threshold value of when cross-node execution becomes beneficial.

2. If utilizing cross-node execution, then how much work should be distributed to each node? As mentioned previously, during the probe period the runtime measures the execution time of a constant number of iterations on each core in the system. The HetProbe schedulers use this information to directly calculate the core speed ratios of each node and skew the distribution of the remaining loop iterations.

3. If not utilizing cross-node execution, then on which node should the work be run? Determining on which node an application executes best involves understanding how the application stresses the architectural properties of each CPU. Performance counters provide insights into how applications execute and what parts of the architecture bottleneck performance. For our setup, the ThunderX has a much higher degree of parallelism versus the Xeon, meaning it has a much higher theoretical throughput for parallel computation. However, the biggest challenge in utilizing all 96 cores is being able to supply data from the memory hierarchy. Although the ThunderX uses quad-channel RAM (with twice the bandwidth of the Xeon), it only has a simple two level cache hierarchy versus the Xeon’s much more advanced (and larger per-core) three level hierarchy. If an application exhibits many cache misses, then it is unlikely to fully utilize the 96 available cores and would be better run on the Xeon. The HetProbe schedulers measure cache misses per thousand instructions during the probing period to determine how much the work-sharing region stresses the cache hierarchy (users can specify any performance counters prudent for their hardware). We experimentally determined a threshold value of three cache misses per thousand instructions—below the threshold and the application can take advantage of the ThunderX’s parallelism, but above the threshold the ThunderX’s CPUs will continuously stall waiting on the cache hierarchy. Note that the HetProbe schedulers must use performance counters and cannot simply use execution times from the probing period to decide on a node; the probing period measures execution times with DSM overheads that are not present when executing only on a single node.

Once a node has been chosen, the HetProbe schedulers fall back to existing OpenMP schedulers for single-node work distribution. Currently they default to the static scheduler, but this is configurable by the user. Additionally, libHetMP joins threads on the unused node to avoid unnecessary cross-node synchronization overheads. For example, if not using the ThunderX, then there is no reason to keep 96 threads alive simply to join at end-of-region barriers.

Figure 6 shows an example of a work sharing region with 20,000 loop iterations executing using a HetProbe scheduler. The first 2,000 iterations are used for the probing period and each of the 20 cores across both nodes is given an equal share of 100 iterations. Importantly, the probing period is performing useful work, albeit in a potentially unbalanced way. After the probing period, libHetMP measures that Node A’s cores executed 100 iterations in 500 \(\mu\) s, whereas Node B’s cores executed 100 iterations in 1500 \(\mu\) s. The HetProbe scheduler determines that Node A’s cores are 3× faster than Node B’s cores for this work sharing region, meaning threads on Node A should get 3× more iterations than threads on Node B to evenly distribute work (the CSR is set to 3:1 for Nodes A and B, respectively). In this example, the HetProbe scheduler determined that cross-node execution was beneficial (see Section 3.4). For the remaining 18,000 iterations, each thread on Node A receives 1,929 iterations and each thread on Node B receives 643. Thus the HetProbe scheduler automatically determines the relative performance of heterogeneous CPUs through online profiling and distributes the remaining work accordingly. Note that if cross-node communication was deemed too costly, then the remaining 18,000 iterations would all be distributed to either Node A or Node B.

The starting point of HetProbe-I is the HetProbe scheduler, from which we leverage the mechanisms to determine the core speed ratio and abstractions such as the hierarchical barriers that guarantee node synchronization. We apply the same primitives and abstractions as for HetProbe for setting the scheduler at runtime. HetProbe-I differs from previous schedulers mainly in the way it dispatches work between threads, since it is there that HetProbe-I monitors the re-probing conditions and carries out the reprobing as needed.

In the HetProbe scheduler, threads call into libHetMP in three occasions (see Figure 4). In the first call into the runtime, HetProbe assigns iterations intended for the probing. In the second, HetProbe stores performance metrics and computes the CSR to distribute work between nodes. In the last invocation, HetProbe piggy-backs on the dynamic scheduler to assign the remaining iterations from the previously calculated distribution. In HetProbe-I we extend this third stage by checking if the reprobing conditions are fulfilled and taking action if so. Because HetProbe-I needs threads to call this function at regular intervals, it only assigns them a fraction of the iterations that they should receive according to their CSR. This requires HetProbe-I to keep track of both the full assigned interval, i.e., the real next and real end, and the smaller chunk distributed at every iteration, whose chunk size is directly proportional to the computed speed ratio. For example, if a node should have iterations 0 to 30, then HetProbe-I registers these as the real next and end and distributes iterations 0 to 5 on the first call to the runtime, 5 to 10 on the next, and so forth. By using the CSR ratio to compute the assigned iterations, HetProbe-I makes nodes with fewer iterations review reprobing conditions more frequently than nodes given more iterations, as their reduced number of iterations will force them to access HetProbe-I more often, effectively reducing the overhead of the scheduler at this stage.

In this version of HetProbe-I we trigger a reprobing every time the sum of iterations finished by all nodes exceeds a percentage (OMP_HET_PTG) of the total number of the loop. Because this condition is reviewed whenever a thread accesses the dispatching function, HetProbe-I could be facing three potential situations when a reprobing is required:

In any case, HetProbe-I must restart performance statistics and generate new global work queues. The latter has to be done with special care to prevent the repeated execution of finished work, since HetProbe-I must provide threads with a consecutive range of iterations. When HetProbe-I generates a new work queue combining iterations of different nodes, HetProbe-I needs to label and assign work to threads in consideration of jumps.

When the work assigned to a thread in the dispatch function contains a jump, the first half is assigned and the thread is labeled so that it receives the second half in the next call to the function. That second part may also contain jumps so this process could be repeated several times. Hence, HetProbe-I needs to keep track of the aforementioned real end, the assigned end and the beginning of the jump. In the worst case, HetProbe-I will have to manage as many jumps as triggered reprobings.

We evaluated libHetMP using the experimental setup in Table 1, which approximates our envisioned tightly coupled platform. Because no existing systems integrate heterogeneous-ISA CPUs via point-to-point connections, we approximate one by connecting two servers with high-speed networking. Our setup includes an Intel Xeon server with a modest number of high-powered cores and a Cavium ThunderX server with a large number of lower-performance cores. The machines are connected via 56-Gbps InfiniBand, which provide low latency and high throughput. We use the RDMA protocol for all experiments except where mentioned due to its significantly lower latency. Both machines run the latest version of Popcorn Linux; the Xeon server uses Debian 8.9 while the ThunderX server uses Ubuntu 16.04. Popcorn Linux’s compiler is built on clang/LLVM 3.7.1, and libHetMP is built on libgomp 7.2.0.

We selected 10 benchmarks from three popular benchmarking suites: The Seoul National University [51] C/OpenMP versions of the NAS Parallel Benchmarks [6], PARSEC [10] and Rodinia [14]. These benchmarks represent HPC and data mining use cases and exhibit a wide variety of computational patterns on which to evaluate libHetMP. All benchmark results are the average of three runs (execution times were stable across runs). All benchmarks were compiled with -O2 except CG and cfd, which crashed Popcorn’s compiler unless compiled with -O0. We also used -fopenmp-use-tls to enable Linux-native TLS. For the evaluation of the loop scheduler HetProbe-I we add four other benchmarks from SPEC OMP 2012 [39].

We evaluated running benchmarks using several workload configurations. Xeon represents running the benchmark entirely on the Xeon—serial phases run on a single Xeon core and work-sharing regions use the Xeon’s 16 threads. ThunderX is similar—serial phases run on a single ThunderX core and work-sharing regions use the ThunderX’s 96 cores. Ideal CSR executes across both CPUs—serial phases run on a Xeon core and work-sharing regions always split loop iterations across the Xeon and ThunderX (112 total threads) using the static scheduler. The scheduler skews distribution using the CSRs in Table 2. The CSRs were gathered from runs with the HetProbe scheduler and manually supplied via environment variables. Cross-Node Dynamic is identical except it uses the hierarchy-based dynamic scheduler described in Section 3.1. We experimentally determined the best chunk size for each benchmark; most benchmarks performed better with smaller sizes, i.e., finer-grained load balancing. HetProbe is again identical except it uses the HetProbe scheduler. HetProbe uses both CPUs during the probing period and then decides whether cross-node execution is beneficial. If so, then it uses measured execution time to calculate CSRs (Table 2) to skew loop iteration distribution for the remaining iterations. If not, then it selects the best CPU and falls back to OpenMP’s original static scheduler on a single node; threads on the not-selected node are joined to avoid unnecessary synchronization. The probe period was configured to use 10% of available loop iterations. For benchmarks where cross-node execution was beneficial, probing overhead was determined by comparing the difference in performance between Ideal CSR and HetProbe. For benchmarks where it was not beneficial, probing overhead was determined by comparing the delta between the best single-node performance (either Xeon or ThunderX) and HetProbe. The HetProbe scheduler probed for up to 10 invocations of a given work-sharing region (using an exponential weighted moving average to smooth out measurements), after which it re-used existing measurements from the probe cache. For several benchmarks, the HetProbe scheduler chose single-node execution on the ThunderX. As a comparison point, “HetProbe (force Xeon)” shows the same results except forcing the HetProbe scheduler to use single-node execution on the Xeon; these results are explained below.

Table 3 shows the total benchmark execution times, including both serial and parallel phases, on the Xeon. Figure 7 shows the speedup normalized to homogeneous Xeon execution for each of the aforementioned configurations. The benchmarks can broadly be classified into two categories: those that benefit from cross-node execution and those that do not. blackscholes, EP-C, kmeans and lavaMD fall into the former category, whereas the others fall into the latter. Across benchmarks that benefit from multi-node execution, all but blackscholes achieve the highest speedup under Cross-Node Dynamic. This is because with a granular chunk size, work is distributed across nodes in an almost perfect balance. Additionally, due to the thread hierarchy there is significantly reduced global synchronization and threads grab work from a local work pool the majority of the time. Across these four benchmarks, Cross-Node Dynamic yields a geometric mean speedup of 2.68×. Ideal CSR is 12.5% faster for blackscholes and close behind Cross-Node Dynamic for the other three, achieving a geometric mean speedup of 2.55×. Finally, HetProbe is slightly slower than the other two cross-node configurations, achieving a geometric mean speedup of 2.4×. This is because the probe period runs a constant number of iterations for all cores leading to an initial workload imbalance. Additionally, measurement machinery (timestamps, parsing the proc file for DSM counters) and probe cache synchronization add extra overheads. For these four benchmarks, probing overhead is equal to the difference between Ideal CSR and HetProbe, as they are functionally equivalent after probing. HetProbe adds 5.2%, 5.3%, 11.5%, and 2.8% overhead for blackscholes, EP-C, kmeans and lavaMD, respectively, for a geometric mean overhead of 5.5%. This demonstrates the HetProbe scheduler provides competitive performance with minimal overheads for benchmarks that benefit from cross-node execution.

For benchmarks that do no scale across nodes, however, the Ideal CSR and Cross-Node Dynamic configurations significantly degrade performance with geometric mean slowdowns of 3.63× and 5.89×, respectively. This is due to DSM—threads spend significant time waiting for pages from other nodes, which also forces application threads on other nodes to be time-multiplexed with DSM workers. There is not enough computation to amortize DSM page fault costs over the network. The Cross-Node Dynamic scheduler is exclusively worse than the Ideal CSR scheduler due to additional work distribution synchronization caused by threads repeatedly grabbing batches of iterations. The HetProbe scheduler, however, successfully avoids cross-node execution for these benchmarks by measuring the page fault period and determining cross-node execution to not be beneficial (geometric mean slowdown of 39%, or 2.4% without cfd). Figure 8 shows measured page fault periods for each application; applications with a period below 100 \(\mu\) s were considered not profitable for cross-node execution.

For applications deemed not beneficial to execute across nodes due to high DSM overheads, the HetProbe scheduler utilized cache misses per 1,000 instructions to determine whether to execute work-sharing regions on the Xeon or ThunderX. As shown in Figure 9, there is a clear separation between applications that benefit from the ThunderX’s high parallelism (BT-C, cfd, lud) and those that are bottlenecked by memory accesses (CG-C, SP-C, streamcluster). When selecting a node, the HetProbe scheduler used a threshold value of three misses per thousand instructions, placing BT-C, cfd and lud on the ThunderX and the others on Xeon (cfd has special behavior, see below). For the three benchmarks placed on Xeon, probing overhead is equivalent to the difference between Xeon and HetProbe, since HetProbe degrades to Xeon after probing. The probing period adds 4.8%, 6.6%, and 7.1% for CG-C, SP-C, and streamcluster, respectively, for a geometric mean overhead of 6.1%. This shows performance close to single-node execution on the Xeon, meaning the probing period has minimal impact on performance.

Looking closer at cfd and CG-C, these applications have roughly the same performance on Xeon and ThunderX but are vastly different in cache miss behavior. Even more interestingly, the HetProbe scheduler places cfd on the ThunderX although the optimal choice would be on the Xeon. This is due to the fact that although cfd’s parallel region runs faster on the ThunderX (74.58 seconds on Xeon, 66.79 seconds on ThunderX), it has a long serial file I/O phase that runs significantly faster on Xeon (1.83 seconds on Xeon, 13.72 seconds on the ThunderX), leading the benchmark’s overall execution time to be faster on the Xeon. This file I/O phase also explains the disparity in cache misses between benchmarks: cfd’s parallel region has a low number of cache misses, but the benchmark’s execution time is heavily impacted by file operations, whereas CG-C does not perform file I/O and has a large number of cache misses.

Interestingly for BT-C, cfd and lud, executing parallel regions on the ThunderX achieved worse than expected performance due OS limitations. Popcorn Linux’s kernel currently only supports spawning threads on the node on which the application started, meaning one thread must remain on the Xeon even when work-sharing regions execute on the ThunderX. Each of these benchmarks executes hundreds to thousands of work-sharing regions (and their associated implicit barriers), causing significant cross-node synchronization. As a comparison point for BT-C and cfd, we ran an additional experiment to force the HetProbe scheduler to select the Xeon for single-node execution; it added 3.2% and 4.2% probing overhead, respectively. lud is an interesting case – the HetProbe scheduler decides cross-node execution is not profitable and runs work sharing regions on the ThunderX. The aforementioned OS limitation impacts HetProbe’s performance enough that Ideal CSR actually achieves 20% better performance than HetProbe (although still worse than running solely on the ThunderX). We expect that when Popcorn Linux allows spawning threads on remote nodes, libHetMP will be able to more efficiently leverage both machines.

It is important to note that none of Xeon, ThunderX, Ideal CSR or Cross-Node Dynamic perform best in all situations, clearly illustrating the need for HetProbe. As shown in Figure 7, HetProbe provides the best performance out of all evaluated configurations across all benchmarks with a geometric mean performance improvement of 41% (ThunderX provides an 11% improvement). In contrast, Ideal CSR causes a slowdown of 49% and Cross-Node Dynamic causes a 96% slowdown, highlighting the importance of communication traffic when distributing computation. As a comparison point, “Oracle” shows that developers could obtain a geometric mean speedup of 60% if they had extensively profiled all configurations and selected the best for all benchmarks. As Popcorn Linux matures, HetProbe will be able to more closely match the Oracle, as the aforementioned limitation has a significant impact on HetProbe’s performance.

The four applications that benefit from cross-node execution have a high enough compute to cross-node communication ratio to leverage the compute resources of multiple CPUs. blackscholes has an initial data transfer period but repeats computation on the same data, allowing it to settle on nodes (blackscholes also has a lengthy file I/O phase that benefits from the Xeon’s strong single-threaded performance). EP-C performs completely local computation (including heavy use of thread-local storage) with a single final reduction stage. lavaMD computes particle potentials through interactions of neighbors within a radius, meaning multiple threads re-use the same data brought across the interconnect. Similarly, kmeans alternatively updates cluster centers and cluster members—all threads on a node alternate between scanning the cluster member and cluster center arrays, re-using pages brought over the interconnect.

Benchmarks that do not benefit cannot amortize data transfer costs. For example, BT-C and SP-C access multidimensional arrays along different dimensions in consecutive work sharing regions, causing the DSM to shuffle large amounts of data between nodes. Other benchmarks have little data locality: CG-C and streamcluster calculate a set of results and then access them in irregular patterns using an indirection array. This behavior causes extensive latencies for local cache hierarchies, let alone DSM. lud’s work-sharing region sequentially accesses an array, but does not perform enough computation per byte to amortize DSM costs. Additionally, there is a large amount of “false sharing” where threads on different nodes write to independent parts of the same page. False sharing can be avoided by the use of a multiple-writer protocol such as lazy-release consistency [4].

An interesting observation that arises from measuring the page fault period is that while the metric provides a sound threshold for determining whether cross-node execution will be beneficial, it is not a good indicator of overall performance gains. For example, while kmeans’ page fault period is slightly over the threshold (130- \(\mu\) s period), it benefits the most from cross-node execution out of all benchmarks. This is because it has a high level of inter-thread data reuse. As mentioned previously, all threads scan the same array in a superstep of the algorithm, meaning all threads reuse data brought over from a page fault (in addition to being extremely efficient on the cache-starved ThunderX). This is in contrast to lavaMD where only a subset of threads working on adjacent regions reuse migrated pages. libHetMP only observes DSM traffic for the entire application and does not measure the level of data reuse for migrated pages. This leads us to believe that the current thresholding mechanism will degrade when threads executing a work sharing region have skewed page fault behavior, e.g., a few threads cause the majority of page faults, biasing the work sharing region’s average page fault period. In such a case libHetMP may determine there is too much communication for cross-node execution, even though it may still be beneficial. However, because we focus on applications with regular work sharing regions, we did not observe this skewed page fault behavior. This also highlights the importance of directly measuring the relative performance of the CPUs to make workload distribution decisions for cross-node execution.

Three of the four benchmarks that benefit from cross-node execution achieve the best performance with Cross-Node Dynamic due to fine-grained load balancing. For blackscholes, however, Ideal CSR achieves better performance. This is due to pages settling into a steady state after an initial page shuffle. Threads receiving the same loop iterations across multiple invocations of the work sharing region access the same data, thus all data pages required by threads are already mapped to the appropriate node. With Cross-Node Dynamic, however, threads receive different loop iterations across separate executions, meaning pages containing results must be continually shuffled across nodes. This settling behavior is why the HetProbe scheduler deterministically distributes iterations for the probing period.

To evaluate the effectiveness of the HetProbe scheduler for different types of interconnects, we ran blackscholes with varying number of iterations (more iterations means more compute operations per byte since blackscholes’ data settles after the first iteration) using the TCP/IP protocol described in Section 3.4. Figure 10 shows the execution time when running homogeneously on the Xeon versus cross-node execution (lines) and the page fault period of each cross-node run (bars). We use a page fault period of 7600 \(\mu\) s to determine whether cross-node execution will be beneficial when using TCP/IP. The results are somewhat noisy (TCP/IP tends to have more variable latencies) but consistent with expectations – only after the page fault period climbs above 8,000 \(\mu\) s does cross-node execution pay off. Thus we conclude using page fault periods as the determining factor for cross-node execution is applicable for different types of interconnects.

We evaluated the HetProbe-I loop scheduler, paying special attention to the irregularity of workloads as this was the central aspect of its design. We use the same experimental setup as on previous sections (see Table 1), with an Intel Xeon 2620v4 at 2.1 GHz and a ThunderX Cavium at 2 GHz, relying on the RDMA protocol to establish inter-node communication. We developed a new microbenchmark, specifically designed to evaluate performance of heterogeneous-ISA OpenMP schedulers under workloads with irregular degrees of architectural-affinity. This microbenchmark leverages the work sharing regions of streamcluster and BT-C, two benchmarks that achieved their best numbers running solely on the Xeon and the ThunderX cores, respectively (see Figure 7). We also ran several benchmarks to evaluate HetProbe-I in comparison to HetProbe, using streamcluster and BT-C as well as blackscholes (which showed the best speedup under the Ideal CSR scheduler) and lavaMD (most benefited by using the hierarchy-based Cross-Node Dynamic scheduler). To make our evaluation more comprehensive, we also included benchmarks from SPEC OMP 2012 [39]. We rearranged some portions of code to address limitations of the Popcorn compiler, without altering the benchmarks’ execution. Since the Popcorn compiler does not currently support C++, we considered all SPEC benchmarks written in C, except for 367.imagick, as it triggers a bug in the Popcorn compiler. All benchmarks are compiled using the same flags as in the previous evaluation of HetProbe. We still use 10% of iterations for the initial probing period, just like HetProbe, as our experiments do not demonstrate a significant difference in the performance of the benchmarks for either HetProbe or HetProbe-I when using a different percentage.

To further test HetProbe-I, we include a microbenchmark that receives the degree of irregularity of the workload as parameter. The source code of streamcluster and BT-C are joined, since the evaluation of HetProbe demonstrates they will give their best performance running only on Xeon (the former) or ThunderX (the latter). The microbenchmark combines the execution of these two benchmarks’ work sharing regions in different quantities to generate the requested degree of irregularity. It is additionally used to generate workloads that are best suited for the combined effort of the Xeon and ThunderX architectures by mingling iterations of both benchmarks. For example, with a 60% degree of irregularity, six out of 10 iterations will be invocations of the streamcluster loop and four of the BT-C loop.

Figure 11 compares the performance of HetProbe and HetProbe-I using different degrees of irregularity with the microbenchmark. Because of how the microbenchmark divides iterations, it is no surprise that under no degree of irregularity the speedup closely resembles the one of streamcluster’s, a regular workload, whereas with 100% irregularity the overall performance is very similar to the one of BT-C, an irregular workload. As expected, a progressive improvement in the performance of HetProbe-I compared to HetProbe can be observed as the percentage of iterations from BT-C is increased.

Figure 12 shows the speedup comparison between the two new loop schedulers. The benchmark with the greatest improvement (BT-C, 24%) combines two features that are hard for HetProbe to deal with - it has one of the highest cross-node synchronization requirements among the benchmarks, but is actually very well suited to be executed by ThunderX (as seen in Figure 7). Hence, with HetProbe-I the initial page faults—which reduce the ThunderX’s chances for work on the first probing period—end up paying for themselves in the long run, something HetProbe-I detects on the reprobing periods. Noticeably, the Cross-Node Dynamic scheduler is also designed to deal with irregular workloads by having threads rate-limit themselves; threads that get assigned computationally demanding iterations take more time, whereas threads that get lighter work can churn through more iterations. Nonetheless, threads replenish iterations from local per-node queues generated after the initial probing period, so it is also interesting to compare this scheduler with HetProbe-I, which regenerates the global queues and then these local queues once the reprobing is completed and a different CSR is obtained.

Compared with the Cross-Node Dynamic scheduler, HetProbe-I performs over three times better on BT-C, a benchmark on which HetProbe was already significantly beneficial compared to the Cross-Node Dynamic scheduler (see Figure 7). The slight improvement of blackscholes (5%) over Ideal CSR (the scheduler with the best performance) can be attributed to the high number of re-probing periods, which even out workload imbalances caused by measurement error. For blackscholes, HetProbe-I also averages a 13% speedup improve over the Cross-Node Dynamic scheduler. This is an example of a case in which the reprobing does not cause overhead, but is instead beneficial due to blackscholes’ irregularity. Conversely, streamcluster is better suited to run on the Xeon -we learned from the microbenchmark evaluation that it had a low degree of irregularity- so periodic reprobing that sends iterations to the ThunderX side is always detrimental.

In conclusion, HetProbe-I can achieve significant performance benefits over HetProbe. HetProbe-I excels in work sharing regions that contain a high degree of irregularity, for example a high amount of page faults registered on the first probing period that are not representative of the overall execution.

There are number of ways in which libHetMP can be extended.

An alternative for HetProbe-I would be to continuously monitor page faults during the work sharing region and fall back to single node execution if the number of page faults begins to rise. Conversely, if the number of page faults begins to drop, then the HetProbe-I scheduler could dynamically bring extra threads on another node online. In general, libHetMP and the HetProbe-I scheduler could be extended to provide an even more dynamic distribution of parallel work.

libHetMP also currently focuses on achieving maximum performance but not energy efficiency. The first-generation ThunderX CPUs consume large amounts of power, meaning that even though cross-node execution may provide the best performance, oftentimes the heterogeneous setup consumes more energy than running solely on one node. Optimizing OpenMP execution for different efficiency metrics may yield different workload distributions, especially as the system architecture (CPUs, interconnect) changes.

libHetMP could be extended to handle systems with three or more nodes. The microbenchmark described in Section 3.4 can be used to determine when cross-node execution becomes profitable for each \((architecture, interconnect)\) pair attached to the system (i.e., each node). This break-even point is different for every node and decisions about which nodes to use can be made independently from one another. For example, consider a system with nodes A and B with break-even points of 100 \(\mu\) s/fault and 200 \(\mu\) s/fault, respectively. If libHetMP measured a page fault period of 150 \(\mu\) s/fault for a given work-sharing region, then the HetProbe scheduler could choose to distribute work to node A but not use node B. In this way, the HetProbe scheduler can choose whether to utilize each individual node and distribute iterations to the enabled nodes according to their relative performance in the probing region.

In terms of potential performance degradation for HetProbe-I, the act of reprobing can be costly and may have some degree of negative impact on other areas of the system. It is worth noting that under HetProbe-I the overall CPU utilization will increase under certain circumstances. Intuitively, if a reprobing period triggers a redistribution of work onto a node that was not in use before, then the CPU resources on that system will begin executing loop iterations. Furthermore, we observed an increase in the number of cache misses on all the benchmarks evaluated. This increase can be attributed to several factors, including that (1) a node that was not working at all in the previous iteration of work distribution will have to perform the initial fetching of memory (page faults and compulsory cache misses) and (2) the spatial locality of iterations may not be exploited by the node on the next work distribution. As an example, picture Node A running iterations 10 to 1,000 and Node B running iterations 1,000 to 1,100. If Node B is asked to execute iterations 300 to 1,100 after a reprobing period (which will include jumps for what was already completed), then the spatial locality from adjacent work iterations will be lost.

Figure 13 shows the percentage increase in cache misses for each of the benchmarks in comparison with HetProbe. The largest increase comes from streamcluster (55% more cache misses), the benchmark with which HetProbe-I obtained the worst results with a 35% slowdown compared to HetProbe. Even though a relative increase in cache misses is undesirable, the total number of cache misses for streamcluster is an order of magnitude smaller than lavaMD or BT-C, which makes this increase less negative in terms of absolute numbers. Hence, there are scenarios in which this loop scheduler does not only degrade performance, but could also potentially damage other applications on the system that access shared levels of cache memory. From this we learn that a refinement of the loop scheduler management of memory could potentially mitigate the overhead produced by HetProbe-I.