Exploration and Exploitation for Buffer-Controlled HDD-Writes for SSD-HDD Hybrid Storage Server

Nowadays, primary storage involves popular SSD and traditional HDD. SSDs have become a mainstream storage media due to its superior performance and lower power consumption than HDDs [2, 59]. However, the limited write endurance has become a critical design issue in SSD-based storage systems [41]. Furthermore, SSDs suffer from performance-degrading GCs, which recycle the invalid pages by moving valid parts to new blocks and then erasing old blocks [32, 44]. GCs with ms-level delays can block incoming user requests, thus leading to long tail latency [20]. On the other hand, both large IO blocks and high IO intensity can lead to sudden increment in SSD queue, also resulting in high tail latency [30]. Therefore, recent studies [60] indicate that SSDs do not always exhibit their ideal performance in practice.

HDDs have large capacity at low cost without the wear-out problem. However, HDDs have relatively low performance compared to SSDs. A random HDD IO has 2 \(\sim\) 3 orders of magnitude higher latency than an SSD IO. This is primarily because of the \(ms\) -level mechanical seeking of disk head.

To accommodate exponentially increasing storage requirement while achieving overall cost-effectiveness, SSD-HDD hybrid storage has emerged to be an inevitable choice for cloud providers [47, 56]. Most of them, such as Google [23], Amazon [45], Facebook [42], and Microsoft’s online services [9], expect larger storage capacity and better performance but at lower cost. To meet this demand, they increasingly embrace storage heterogeneity by deploying variable types and numbers of SSDs and HDDs. The former offers lower IO latency [16] as the primary tier, the latter provides larger capacity at low cost as the secondary tier. The better performing SSD tier generally plays the role of a write buffer to quickly persist incoming write data, which are eventually flushed to the slower but larger HDD tier. As a result, the SSD tier absorbs most of the write traffic from foreground applications.

Write-intensive workloads widely exist in many production environments, such as enterprise applications, supercomputing, and clouds. Enterprise servers are expected to rapidly persist production data in time, such as business databases. Burst buffer [4, 34] in supercomputing systems also deploys high-performance SSDs to temporarily store instantaneous highly-intensive write data. More commonly, many backend storage servers in cloud must accommodate write-dominated workloads, as observed in Alibaba Pangu [38]. Pangu [10] is a distributed large-scale storage platform and provides cost-effective and unified storage services for Alibaba Clouds [26, 37] and Ant Financial. As such, Pangu needs to minimize the total cost of ownership while meeting strict QoS requirements like tail latency [6, 18].

Table 1 gives the workload characteristics of production trace data from Pangu. We observe that some storage nodes (servers) in Pangu rarely serve reads from the frontend and instead must handle amounts of highly-intensive writes. For Alibaba Cloud, the upper-level latency-critical online services generally build their own application-aware caches to ensure service responsiveness and reserve local fast-storage to cache hot data for user reads. Therefore, the backend storage nodes are required to persist new and updated data from frontend nodes as soon as possible. To better understand this write-dominated workload behavior, we analyze four typical workloads on Pangu storage nodes A (Cloud Computing), B (Cloud Storage), and C and D (Structured Storage). We count these workloads from one SSD and one HDD in each node because the workload behavior of all storage devices is basically the same in one node. Observations are drawn as follows. A comprehensive workload analysis of Pangu can be found in the previous study [38].

To relieve the SSD pressure from write-dominate workloads, a simple solution is to increase the number of SSDs in the hybrid nodes. However, this is a costly solution as it increases the total cost of ownership.

An alternative is to exploit the severely underutilized HDD IO capacity in hybrid storage nodes when SSDs are overused. The state-of-art solution SWR [38] redirects large SSD writes to idle HDDs. This approach can alleviate the SSD queue blocking issue to some extent. However, the IO delays experienced by write requests redirected to HDDs are shown to be 3–12 times higher than those written to SSDs. This is clearly undesirable, if not unacceptable, for most write requests that demand \(\mu\) s-level latency. The key challenge is how to reduce HDD IO delay to the \(\mu\) s-level that is close to SSDs, which is a seemingly impossible task at first glance. Fortunately, as we look closer into the write behaviors of HDDs, this is indeed possible, which we will elaborate in the next section.

We conduct a “profiling” process to observe detailed HDD behaviors. The profiling of the HDD model executes a series of continuous and sequential writes with the same IO size to an HDD, which roughly consumes 5–10 seconds. The profiling process must be executed when adding or replacing an HDD device. Certainly, it can be conditionally or periodically triggered to recalibrate the parameters of BCW model. We select five representative HDD products: West Digital 10 TB, 8 TB, and 4 TB, and Seagate 8 TB and 4 TB. Table 2 lists their device parameters. The recording technology of 4 TB Seagate HDD is Shingled Magnetic Recording (SMR) and the other four HDDs are Perpendicular Magnetic Recording (PMR). We draw three observations from the profiling results shown in Figures 1 and 3.

The reasons behind these observed HDD write behaviors are as follows. Modern HDDs deploy a built-in Dynamic Random Access Memory (DRAM) (e.g., 256 MB for the 10 TB and 8 TB HDDs, and 64 MB for the two 4 TB HDDs). However, only a part of the DRAM (e.g., 16 MB for 10 TB WD and 8 TB Seagate HDD, 4 MB for 8 TB WD HDD and 4 TB Seagate HDD, 2 MB for 4 TB WD HDD) can be used to buffer incoming write IOs based on external observation. The remaining capacity of the HDD built-in DRAM can be used as read-ahead cache, ECC buffer [11], sector remapping buffer, or prefetching buffer [21, 52]. However, the exact buffering and flushing mechanisms of built-in DRAM in HDD are completely hidden from the host and heavily depend on the specific HDD models. Fortunately, we can measure the buffered write feature of an HDD externally and experimentally based on the aforementioned experiments.

Upon successful buffering of a write, HDD immediately informs the host of request completion. When the buffered data exceed a threshold, the HDD must force a flushing of the buffered data into their locations in the disk media. During this period, incoming writes must be blocked until the buffer is freed up again. It is worth noting that, after an idle period, the HDD buffer may become empty implicitly as a course of flushing data to the disk. However, to explicitly empty the buffer, we can actively invoke \(sync()\) to force flushing.

To formally characterize the HDD write behavior, we build an HDD buffered-write model. Figure 4 illustrates the schematic diagram of this model in the time dimension. The x-axis represents the time sequences of transitions among the three write levels, with each sequence being started by “Sync” event.

A Buffered-Write Sequence consists of three aforementioned types of HDD buffered writes, i.e., Fast (low-latency), Mid (mid-latency), and Slow (high-latency) writes, which denotes as \(F\) , \(M\) , and \(S\) , respectively. In the model, \(F\) , \(M\) , and \(S\) can be thought of as the states a write request can be in (i.e., experiencing the fast, mid or slow write process). As described in Table 3, these IO delays are denoted as \(L_{f}\) , \(L_{m}\) , and \(L_{s}\) , respectively. The \(F\) state means that an incoming write request \(w_{i}\) with the size of \(s_{i}\) can be buffered completely in the built-in DRAM buffer of HDD. The \(M\) state means that the write buffer is close to being full. The \(S\) state means that the write buffer is full and any incoming write request is blocked.

A Buffered-Write Sequence lasts a Fast stage, followed by one or more Slow-and-Mid stage-pairs. The sequence begins when there is sufficient buffer available for Fast stage (e.g., close to empty). It ends when current series of continuous writes ends. The Fast, Mid, and Slow stage last for \(T_{f}\) , \(T_{m}\) , and \(T_{s}\) respectively, which are determined by the cumulative amount of written data \(W_{f}\) , \(W_{m}\) , and \(W_{s}\) in the respective states. Actually, \(W_{f} = T_{f}*s_{i}/L_{f}\) and it is applied to \(W_{m}\) .

We can profile the HDDs to identify such key parameters. For example, Figure 1 shows the profiling results of 10 TB WD HDD with varying write request sizes. The value of \(L_{f}\) is 180 \(\mu\) s, \(L_{m}\) is 280 \(\mu\) s and \(L_{s}\) is 12 ms. The value of \(T_{f}\) is 60 ms, \(T_{m}\) is 37 ms and \(T_{s}\) is 12 ms. \(W_{f}\) is 16 MB and \(W_{m}\) is 8 MB. \(W_{s}\) depends on the IO size \(s_{i}\) . According to the HDD buffered-write model, the Fast and Mid writes of HDD have 100- \(\mu\) s-level latency, which can approach the write latency of SSDs. This motivates us to design a controllable buffer write strategy for HDDs to reduce writes on SSDs in hybrid storage systems without sacrificing the overall performance.

The next write state could be predicted according to write buffer’s free space and the write state of the current request. In the HDD buffered write model, each write request state should be one of \(F\) , \(M\) , and \(S\) state. The HDD buffer state is considered by buffered-write model to be in either \(A\) (available) or \(U\) (unavailable). The “ \(A\) ” state means current Accumulative Data Written (ADW) in the \(F\) and \(M\) states are less than \(W_{f}\) and \(W_{m}\) respectively. Otherwise, the write buffer is in the “ \(U\) ” state. Figure 5 shows how the next write state is determined based on the current buffer state and write state in a State Predication Diagram, which is described as follows:

Based on that, we design a Write-state Predictor described in Algorithm 1. It identifies what the current write state is, \(F\) , \(M,\) or \(S\) by monitoring the IO request size and latency, and calculating the free space in the write buffer. That is, the ADW in the current write state ( \(F\) or \(M\) ) is recorded and compared with \(W_{f}\) or \(W_{m}\) for predicting the next write state.

BCW is an HDD writing approach that ensures user writes using \(F\) or \(M\) write state and avoids allocating Slow writes. The key idea of BCW is to make buffered write controllable. Based on the Write-state predictor, we design BCW as described in Algorithm 2.

Upon activating BCW, a \(sync()\) operation is invoked to force synchronization to empty the buffer actively. BCW dispatches sequential user writes to HDD if it is predicted to be in \(F\) or \(M\) state, otherwise pads non-user data to HDD, until it reaches the max setting loop (or unlimited) of Buffered Write Sequence. If there are user requests in the queue, BCW writes them serially. After a write is completed, BCW adds its write size to ADW, and updates the write-state accordingly.

During light or idle workload periods with sparse requests, the HDD request queue will be empty from time to time, making the write stream discontinuous. To ensure the stability and certainty of buffered writes in a sequential and continuous pattern, BCW will proactively pad non-user data to the HDD. There are two types of padded data, \(PF\) and \(PS\) . The former is used to fill the \(F\) and \(M\) states with 4 KB non-user data; the latter is to fill the \(S\) state with larger block size, e.g., 64 KB of non-user data. A small \(PF\) can minimize the waiting time of user requests. A large \(PS\) helps trigger Slow write quickly. Note that BCW still executes the write-state predictor algorithm for each padded write.

More specifically, BCW continuously calculates ADW in the current state ( \(F\) or \(M\) ). When ADW is close to \(W_{f}\) or \(W_{m}\) , it means that the HDD buffered write is at the tail of the Fast or Mid stage. The \(S\) write state may occur after several writes. At this point, BCW notifies the scheduler and proactively triggers the Slow write with \(PS\) . To avoid long latency for user writes, at this period, all incoming user requests have to be steered to other storage devices, such as SSDs. When an \(S\) write is completed, the next write will be \(M\) according to the write-state predictor. Then BCW resets the ADW and accepts user requests again.

We also find it unnecessary to proactively fill padded writes in the \(F\) stage before ADW exceeds \(W_{f}\) . When ADW does not reach \(W_{f}\) , the physical disk operation is not yet triggered and the buffer can absorb user requests for this period. When ADW exceeds \(W_{f}\) in a short time of period, it means that the buffer will begin to flush the data to the disk and the next write state will be changed to \(S\) . On the other hand, when ADW is less than \(W_{f}\) for a long time of period, the disk can flush the buffered data automatically so that the next write state may be \(F\) . However, it does not affect performance. We apply this observation to the scheduler design in the next section.

In most cases, the sequential and continuous write pattern induced by BCW is reasonably stable. However, this pattern can be broken, e.g., HDD reads. Besides, the \(S\) write states may be triggered in advance by user requests or \(PF\) writes. To regain the control of buffered writes, BCW continuously executes \(PS\) until a \(S\) write state is detected. As a result, the write-state predictor will be recalibrated. The cost of this strategy is the waste of IO and storage of BCW to perform \(PS\) writes to HDD. In addition, we can also issue \(sync()\) to reset the buffered write state in BCW. Howerver, a \(sync()\) can take several hundred milliseconds, during which the HDD cannot accept any writes. Fortunately, in the experiment, we find that the BCW interrupted cases are rare.

BCW stores incoming data to HDD in a log-append manner. This differs from traditional logging mechanism in existing file systems like ext4 [40], which allocates writes to the tail logical address of the log, ensuring address continuity. However, it doesn’t ensure IO continuity and does not determine the next write state. In contrast, BCW maintains both address and IO continuity, making the buffer writing control effectively.

Accuracy and Stability. We assess the prediction accuracy of the write-state predictor and the robustness of BCW. We define three synthetic workloads and describe them as follows. The total data written volume for each workload is 100GB. We perform predictions for each request, and calculate both the correctness and error rate of prediction on the \(F\) , \(M\) , and \(S\) write states respectively.

The results in Table 4 show that the predictor correctly identifies at least 97.1% of the \(F\) state, 97.9% of the \(M\) state, and 99.8% of the \(S\) state. The high probability for mis-predicting the \(F\) and \(M\) write-states to \(S\) is because the prediction policy tends to prevent the \(S\) write-state, thus avoiding performance degradation. It is better than mis-predicting an actual \(S\) write-state to other state. In the synthetic workload 2, the error prediction rate of \(F\) write-state is 1.8% higher than in the workload 1, while that of \(M\) and \(S\) write-states is almost unchanged. This is because BCW could be occasionally interrupted and then restart from the \(F\) stage. However, mis-predicting a \(F\) state does not degrade the performance. In the workload 3, BCW is frequently interrupted by read requests and cannot access \(M\) or \(S\) stages in a buffered write sequence. Besides, we carry out the same synthetic test for other types of HDDs and the results are the same basically.

Moreover, BCW performs recalibration in runtime based on the runtime IO results with respect to disk aging and variation on disk characteristics. BCW recalibrates at the end of each complete HDD buffered write sequences. It records key parameters (i.e., \(W_{f}\) , \(W_{m}\) and \(W_{s}\) ) in the HDD Buffered-Write Model from both the actual disk IO feedback and the prediction results in each sequence. When the difference of the parameter values between the prediction model and the actual IO feedback exceeds a PF size (i.e., 4 KB), BCW will adjust these parameters in the prediction model to the actual value. In addition, to smooth adjust parameters in recalibration, we record the actual parameters from the last ten write sequences, and use their average value as a new parameter value. Secondly, the recalibration will not be triggered if a buffered write sequence is unexpectedly interrupted by a read request or a \(sync()\) operation.

Persistency and Reliability. The built-in buffer in almost all of the existing HDDs is volatile, which could cause data loss in the face of unexpected power outages. Meanwhile, it is enabled to buffer write data at the default factory settings. Therefore, in the ordinary case, all HDD-write requests would write into the buffer first and then flush to the disc. However, it is hard for the hosts to explicitly determine whether the data remain in the built-in buffer. Compared with that, BCW monitors and controls the states of the built-in buffer in an active manner, which makes the unreliability time window of the buffered data deterministic and measurable.

BCW provides a proactive and controllable buffer writing approach for HDDs. In this section, we further propose an MIOS for SSD-HDD hybrid storage to leverage BCW effectively. The scheduler decides whether or not to steer user writes to an HDD queue according to the results of the write-state predictor and current queue status.

Architecture. The architecture of MIOS is shown in Figure 6. Generally, a typical hybrid storage node could contain multiple SSDs and HDDs in practice, and the number of HDDs is usually higher than that of SSDs. In MIOS, all disks are divided into independent SSD-HDD pairs, each of which contains an SSD and one or more HDDs, and is managed by an independent MIOS scheduler instance. MIOS monitors all request queues of SSDs and HDDs at runtime, judiciously triggers the BCW process, and determines whether a user write should be directed to a selected HDD or SSD. MIOS creates a device file to each HDD in the configuration process, which stores BCW writes in an append-only manner. Before MIOS scheduling, a profiling is performed to determine the key parameters ( \(W_{f}\) , \(W_{m}\) , etc.) for the write-state predictor.

Scheduling Strategy. In Algorithm 3, the SSD request queue length \(l(t)\) at time \(t\) is a key parameter in MIOS. When \(l(t)\) is larger than a predefined threshold \(L\) , the scheduler steers user writes to an HDD with the prediction of it being \(F\) or \(M\) write state. The threshold \(L\) is pre-determined according to the actual performance measurements on SSD. Specifically, we measure the write latency under different SSD queue lengths. If the request with queue length \(l\) has latency larger than the IO delay of HDD in the \(M\) state, we simply set the threshold \(L\) to the minimum \(l\) . The rationale is that when the SSD queue length is larger than \(L\) , the SSD writes’ latencies will be at the same level as their latencies on an HDD in the \(F\) or \(M\) write state with BCW. \(L\) can be determined and adjusted experimentally according to workload behaviors and storage device configurations at runtime. This strategy mitigates, though not avoids, the long-tail latency upon workload bursts or heavy GCs on SSD [44, 57]. In these cases, the SSD request queue length can be 8–10 times longer than its average. Therefore, redirected HDD writes not only relieve SSD pressure imposed by bursty requests and heavy GCs, thus curbing the long-tail latency, but also lower the average latency.

Additionally, when the queue length of SSD is less than \(L\) , triggering BCW is optional. Enabling or Disabling BCW in this case is denoted as MIOS_E or MIOS_D, respectively. In other words, MIOS_E strategy allows redirection with BCW when the queue length of SSD is lower than \(L\) . MIOS_D strategy, by contrast, disables redirection when the SSD queue length is lower than \(L\) . We will experimentally analyze the positive and negative effects of MIOS_D and MIOS_E in Section 5.

Finally, BCW requires the complete control over HDDs that cannot be interfered by other IO operations. When an HDD is executing BCW and a read request arrives, MIOS immediately suspends BCW and serves this read, and then try to redirect all writes to other idle disks with a multi-HDD scheduling. We will explain this in the next section. For read-dominated workloads, BCW can also be disabled to avoid interfering with reads.

Multi-HDD Scheduling. MIOS and BCW actually increase the IO burden of HDDs. First, MIOS redirects a large number of SSD write requests to HDDs. Second, BCW has to write padded data into HDDs. Single HDD has limited IO capacity, making the HDD difficult to be always IO-available when performing SSD redirection. Meanwhile, BCW has to suspend to serve read requests in advance. All of these could make the HDDs bottleneck and affect the efficiency of MIOS scheduling. In addition, it is hard for MIOS to redirect requests using only the best-performing \(F\) write states to achieve better latency. This is because the \(F\) stage has limited capacity and could only be used after \(sync()\) . When storage nodes suffer intensive writes, the \(F\) stage will be used up quickly. MIOS has to invoke sync to obtain the \(F\) write state in a round-robin way for all idle HDDs. We find in our experiments that the \(sync\) operation is costly, which could cause MIOS to lose the redirecting opportunities even if SSDs are busy.

To improve the effectiveness of MIOS and to further exploit the low latency feature of HDD buffered write, we propose a multi-HDD scheduling mechanism in MIOS and describe it as Algorithm 4. MIOS is extended to monitor all HDDs in an SSD-HDD pair. It prioritizes HDD that is idle (i.e., queue length is 0) and has the lowest predicted latency to maximize write performance. Particularly, this mechanism is proposed to increase the utilization of the best-performing \(F\) write states in HDD. First of all, the multi-HDD scheduling will preemptively free the HDD buffers to ensure MIOS always has available \(F\) write states for scheduling. It will invoke the \(sync\) operation on the HDD that totally depletes \(F\) stage, on condition that available \(F\) write states still exist in other HDDs. Then, once the \(F\) write state is available for any HDD, MIOS will enable the full BCW process to boost IO capacity under high write intensity. If all HDDs are busy, requests will write back to SSD. Finally, if BCW is interrupted by a read request or other operation, MIOS could write request to the other HDDs and restore the availability of BCW as soon as possible.

Implementation. BCW is a runtime write-scheduling mechanism running on SSD-HDD hybrid storage. BCW is orthogonal to local file system running on HDDs. MIOS can be implemented in either file-level or volume-level to jointly manage SSDs and HDDs in a hybrid storage. In this work, MIOS provides a simple yet flexible file-level request scheduling scheme atop of file systems. HDD-logs of MIOS can directly map to specified areas within raw HDD without native file system, or be configured as large log-files of the local file system (e.g., Ext4 and F2FS) running on the HDD to leverage their mature file-to-storage mapping mechanism. To reduce overhead of the underlying file system, MIOS employs direct IO mode to access the log by calling Linux kernel functions such as \(open\) , \(close\) , \(read\) , and \(write\) . When enabling BCW, its involving HDDs should be fully controlled and cannot serve any IO of other applications. To meet this demand, MIOS carefully coordinates multiple SSDs and HDDs in a hybrid storage server. It merely writes user data to the HDDs while storing the log-metadata to the SSDs to ensure that metadata IO does not interrupt BCW. Besides, for MIOS based on file system, modifying a file could trigger its metadata update and leads to uncontrolled file-system metadata IOs, which interrupts BCW. Therefore, we pre-create and allocate the space of all log-files with a fixed size and cache its addresses. MIOS only updates their file data in place, which cannot generate extra file-system metadata IOs.

MIOS manages the data-chunk location to an SSD-log file in each MIOS instance. We design a request node as metadata structure that records and tracks each redirected chunk data in the HDD-log. Each request node represents a request pointed to its relevant chunk. It contains five fields: the chunk file name of a redirected write request (File ID), length of the request (Length), location of the request that is persisted in the log file (Physical offset), position of the request in the corresponding chunk file (Logical offset), and the request arrival time (Timestamp) to keep the sequential order of requests. When the chunk data are redirected into the HDD-log, the relevant request node as metadata is periodically written into the SSD-log. We aggregate multiple request nodes to 512 B before flushing to match the sector size of SSD. In addition, if there are no IO writes for more than 100 ms, the unaggregated request nodes will also be forcibly flushed. MIOS set a single thread to serially write new request nodes to the SSD-log in the order of their Timestamp field. Therefore, the data location manager will not suffer a lock contention issue even if there are multiple threads updating a chunk file at the same time. Besides, all request nodes are also cached in the memory to accelerate reads for locating the redirected chunks. After a crash, the recovery procedure scans all SSD-logs to rewrite all redirected chunks to their own original locations. It only uses intact request nodes with five fields and broken request nodes will be discarded. Therefore, the data location manager will not be inconsistent after system restart.

We periodically perform HDD garbage collection (HDD-GC) to recycle the logs on HDD. When an HDD is idle, all file-data stored in the log are re-written to their own original chunk files, and then the log are completely recycled. HDD-GC also should be forcedly triggered when the log size exceeds a predefined threshold (e.g., 16 GB). HDD-GC first sequentially and continuously reads all data in the log to reduce seeks, and then extracts and merges the user data to update their correspond files. These chunk file updates can be performed in batch [63]. Meanwhile, the multi-HDD scheduling mechanism can distribute requests to other HDDs during the HDD-GC execution to maintain the available BCW process. Additionally, HDDs have not write-induced wear-out problem unlike SSDs. Therefore, the temporary HDD-storage wastage caused by BCW is not a burden.

We run experiments for performance evaluation on a server with two Intel Xeon E5-2696 v4 processors (2.20 GHz, 22 CPUs) and 128 GB of DDR4 DRAM. To understand the impact of different storage configurations on the performance, we choose two models of SSDs, a 256 GB Intel 660p SSD [13], a 256 GB Samsung 960EVO SSD [19], and a 480 GB Intel Optane 905P SSD [12]. Their long-term stable write throughputs are 0.6 GB/s, 1.5 GB/s, and 2.1 GB/s, respectively. Three models of HDDs are WD 10 TB, WD 4 TB, and Seagate 4 TB, as described in Table 2.

By profiling those HDDs as described in Figure 1, we could obtain the key parameters of BCW. For example, the 10 TB WD HDD has a \(W_{f}\) of 16 MB and \(W_{m}\) of 8 MB. Using the process to pre-determine the queue length threshold \(L\) explained in Section 4, we set \(L\) to 1 for workload of node A, 3 for node B, and 2 for node C and D, where the workloads of nodes A, B, C, and D are described in Table 1 of Section 2. As discussed earlier, MIOS has two schemes, MIOS_D and MIOS_E. When the SSD queue length is less than \(L\) , the former conservatively disables request redirection; the latter allows request redirection but only redirects user write requests to the \(F\) write state. The Baseline for the evaluation is writing all user data into the SSDs. In addition, a complete BCW sequence consists a series of one Fast stage and ten Mid/Slow stage-pairs (Figure 4). At last, the number of HDDs \(N\) is set to 1 through 4 in an SSD-HDD pair. When \(N = 1\) , MIOS cannot perform multiple HDD scheduling. In experiment, we deploy all HDDs as the same model (i.e., WD 4 TB).

We first evaluate the effectiveness of MIOS_D under four Pangu production workloads on the WD 10 TB HDD.

Write Performance. Figure 7 shows that the average and tail latency (99^th and \(99.9^{th}\) ) of all four workloads are significantly reduced by MIOS_D. Among four workloads, B gains the most benefit. Its average, 99^th and \(99.9^{th}\) -precentile latencies are reduced by 65%, 85%, and 95% respectively. On the contrary, these three latencies in A are only reduced by about 2%, 3.5%, and 30%, respectively, which is far less than the other workloads. The reason is that the redirection in MIOS_D is only triggered when the queue length is high, but A has the least intensity and thus the least queue blocking, which renders MIOS much less useful.

To better understand the root causes for the above experimental results, the cumulative distribution functions (CDFs) of SSD queue lengths for four workloads are shown in Figure 8. MIOS_D significantly shortens queue lengths compared to Baseline. B and A have the maximum (95%) and minimum (15%) reduction in their queue lengths. Therefore, the overall queueing delay is reduced significantly.

Request Size. To deeply understand impact of write size in MIOS and BCW, we break down all redirected requests into six groups with different ranges of IO sizes, and measure the average latency with MIOS_D in each group.

Figure 9 shows that, MIOS_D reduces the average write latency of size below 64 KB in all four workloads, and workload B benefits the most. The average latencies of three groups of small-sized requests (<4 KB; 4 KB–16 KB; 16 KB–64 KB) are reduced by 61%, 85%, and 59%, respectively. The other three workloads also reduce their latencies differently. In Baseline, small and intensive requests result in queue blocking more frequently (Figure 2) than in MIOS_D. Therefore, MIOS_D is the most effective in reducing latency in such cases.

However, in groups of requests larger than 256 KB, the average latency is increased in all workloads except B. For workload D, the average latency is increased by 31.7% in the >1 MB group, and 12.1% in the 256 KB–1 MB group. The average latency of the 256 KB–1 MB group in C is also increased by 20.1%. The reason is twofold. First, large SSD writes under light load have better performance than HDDs because SSDs have high internal-parallelism that favors large requests. Second, large writes are relatively sparse and not easy to be completely blocked. For example, the average latency of the >256 KB request-size groups in Baseline is very close to the raw SSD write performance.

Queue Length Threshold \(\textbf {L}\) . To evaluate the effect of \(L\) selection, we compare the pre-defined \(L\) value (Def) determined by the process described Section 4.2, with \(L+1\) (Inc). Note that the pre-defining process for queue length threshold is designed to tradeoff between decreasing the write latency and reducing the write traffic to SSD.

Figure 10(a) shows that Inc slightly reduces average, 99^th and \(99.9^{th}\) -percentile latency compared to Def. Among the four workloads, the maximum reduction in average latency is less than 10%. This is because the higher queue length is, the longer waiting delay a request experiences. Therefore, Inc can acquire more latency gains by redirection than Def. However, the choice of \(L\) value can greatly affect the amount of redirected data. In Figure 10(b), the number of redirected requests is much smaller in Inc than in Def. The amount of redirected data for workloads A \(\sim\) D is decreased by 94%, 64%, 52%, and 62%, respectively. These results are consistent with the implications of Figure 8 that longer queue length in SSD triggers much fewer SSD overuse alerts, significantly reducing chances for request redirecting to HDD.

MIOS_D vs. MIOS_E. We compare MIOS_D with MIOS_E in terms of the amount of data written to SSD and HDD, and the number of redirected write requests. Results are shown in Table 5. Workload A has the highest percentage of redirected data and requests with MIOS_E. It reduces the SSD written data by up to 93.3% compared with Baseline, which is significantly higher than MIOS_D. Since workload A has lower IO intensity, MIOS_E has more chances to redirect even when the queue length is low. Note that we also count the padded data in BCW as the amount of data written in HDD. In such a case the total amount of data written can vary a great deal. Workload B has the lowest percentage of redirection with MIOS_E, which reduces SSD written data by 30%. Nevertheless, the absolute amount of redirected data is very large because the SSD written data in Baseline is larger than any of the other three workloads. Compared with MIOS_D, MIOS_E can greatly decrease the amount of data written to SSD. Therefore, it is more beneficial to alleviate SSD wear-out.

However, the negative effect of MIOS_E is the increase of average and tail latency. In Figure 11(a), MIOS_E leads to generally higher average latency than MIOS_D by up to 40% under workload A. Although for other three workloads, the average latency remains basically unchanged. This is because much more writes (i.e., >90%) are redirected by MIOS_E than by MIOS_D in workload A, and requests in HDD experience longer latency than that in SSD. Moreover, the \(99.9^{th}\) -percentile latency of MIOS_E is increased by 70% in A, 55% in B, 31% in C, and 8% in D compared to MIOS_D. The results can be explained by Figure 11(b). MIOS_E increases the average latency for nearly all the IO size groups, especially for the groups with requests of size larger than 256KB.

Comparison with Other HDD Writing Mechanisms. We compare BCW with other HDD writing mechanisms. The first one is BCW_OF that only uses the \(F\) write state by proactively issuing \(sync()\) when the ADW reaches \(W_{f}\) in a HDD. The second one is Logging [36, 48], which is an ordinary way to improve the performance of HDD-writes by storing written data in an append-only manner. The difference between Logging and BCW is that the former cannot predict or selectively determine to use low-latency HDD buffered write states while the latter can. We measure the average, 99^th, \(99.9^{th}\) -percentile latency and the SSD written data reduction with BCW_OF and Logging. We combine all these HDD write mechanisms with MIOS scheduler that uses the MIOS_E strategy. We take MIOS_E with BCW as the baseline and present performance normalized to it.

Figure 12(a) shows that the \(99.9^{th}\) -percentile latency of BCW_OF is increased by 2.19 \(\times\) over BCW in workload C. The 99^th-percentile latency in the B, C, and D workloads also increases by 2.5 \(\times\) , 1.1 \(\times\) , and 1.9 \(\times\) , respectively. It means that BCW_OF performs less efficiently on reducing tail latency when the workload becomes heavier. This is because such mechanism redirects less requests when SSD suffers queue blockage. Figure 12(b) shows that, the data volume of SSD redirection is reduced by 71% in workload A and 26% in workload B. As mentioned in Section 4.2, \(sync()\) is a high cost (e.g., tens of milliseconds) operation to flush the HDD buffer and HDDs cannot serve any requests during this time window.

Furthermore, Logging can reduce 10% more SSD written data than BCW at the cost of explicit write latency increase. The \(99.9^{th}\) -percentile latency in the B, C, and D workloads is 2.0 \(\times\) , 1.9 \(\times\) , and 1.6 \(\times\) higher than BCW, reaching several milliseconds. This is because Logging cannot prevent write requests to encounter the \(S\) write states during intensive workloads.

Comparison with Existing IO Scheduling Approaches. LBICA [1] and SWR [38] are the state-of-the-art IO scheduling approaches in SSD-HDD hybrid storage. In the write-dominated workload, LBICA redirects requests from the tail of the SSD queue to the HDDs only when SSD has long queue length. Besides, SWR redirects large size write requests preferentially and allows redirecting when the SSD queue length is low. We normalize the performance of LBICA and SWR to MIOS_D and MIOS_E, respectively.

Figure 13(c) illustrates that LBICA increases the average and tail latency by up to 2.0 \(\times\) and 6.2 \(\times\) over MIOS_D in workload B. LBICA calculates the redirect threshold \(L\) by comparing the average latency of random HDD-writes with that of the SSD-writes in tail of the queue, leading to a high \(L\) value. It means that LBICA can only redirect requests when SSD suffers high queue length, which is a rare condition. Figure 13(d) shows that the SSD data reduction of LBICA is decreased 70%–90% than MIOS_D. Compared to LBICA, MIOS_D performs SSD-write redirection under a lower queue length, because BCW could provide \(\mu s\) -level latency for HDD-writes.

Figure 13(a) shows that the average and tail latency of SWR is 1.2 \(\times\) –2.1 \(\times\) higher than MIOS_E. There are two main reasons. First, SWR prefers to redirect requests with large IO size. However, Figure 9 indicates that the performance degradation of redirecting large size request is more than that of a small one, due to the high internal parallelism of SSDs. Second, SWR cannot take full advantage of the low-latency HDD write states and results in increased write latency. The advantage of SWR scheduling scheme is that it could efficiently perform redirection under high IO intensity. Figure 13(b) shows that the amount of SSD write reduction of SWR is 3.4 \(\times\) more than that of MIOS_E in workload B.

Multi-HDD Scheduling. We measure the average, 99^th and \(99.9^{th}\) -percentile latency of write requests with \(N\) values from 1 through 4. When \(N=1\) , the multi-HDD scheduling mechanism cannot be performed. We take the MIOS_E with \(N=1\) as the baseline and present performance and data redirection normalized to it, because multi-HDD scheduling is more adaptable to high-intensity workloads.

Figure 14(a) shows that the multi-HDD scheduling provides a consistent improvement in request latency over single-HDD. The average latency of \(N=4\) is 30% lower compared to the baseline (i.e., \(N=1\) ) in workload A, and that is 17% higher in other three workloads. This is because the multi-HDD scheduling undertakes most redirection requests with the best-performing \(F\) write state. Meanwhile, the normalized 99^th-tail latency on \(N=4\) is decreased 50% than baseline in workload B, 20% in workload A and C, and 10% in workload D. It redirects more requests under high IO intensity, which further relieves SSD pressure. However, the \(99.9^{th}\) tail latency is not significantly reduced, which even presents an increase of 3% and 23% in workload A and C, respectively. In the multi-HDD scheduling, MIOS redirects more requests to HDDs when the number of HDDs increases. Particularly, the number of redirected writes with the 0.5% largest IO size increase, which experience longer latency when written to HDD than that to SSD. It results in the increase of \(99.9^{th}\) -percentile latency.

Figure 14(b) demonstrates that the multi-HDD scheduling further improves SSD data reduction. In workload B with the most amount of written data, \(N=4\) could redirect 4 \(\times\) more data than \(N=1\) , while \(N=2\) or \(N=3\) is enough to redirect more than 92% of requests in other three workloads. We also measure the performance of multi-HDD scheduling with the MIOS_D strategy. We find that two HDDs are typically enough for redirection in all four workloads, and more disks present little improvement. This is because MIOS_D just performs redirection on IO bursty.

Experiment with Other HDDs. We use the 4 TB WD, 4 TB Seagate, and the 10 TB WD HDD to replay workload B, comparing MIOS_D (with the default \(L\) value) with the Baseline in terms of the request latency and the amount of data written to SSD. Workload B is chosen for this experiment, since it has the most SSD written data and the most severe SSD queue blockage, clearly reflecting the effect of IO scheduling.

Figure 15 shows that different models of HDDs do not have a significant impact on the effect of MIOS_D. First, the average and tail latencies for all the three HDDs are virtually identical, with a maximum difference of less than 3%. In addition, in the six request-size groups, only the >1 MB group exhibits a large difference among different HDD models. The average latency of the 10 TB WD HDD is 14% lower than that of the other 4 TB HDDs. This is because of the native write performance gap between them. It can be found from Table 6 that different models of HDDs do not notably affect the amount of data redirected, with little difference of less than 5%.

Experiment with Other SSDs. Next, to further explore the effect of MIOS with different types of SSDs, we first deploy a lower-performance 660p SSD. We replay the same workload A that provides the lowest IO pressure, and employ the MIOS_D and MIOS_E strategies, respectively. From the latency CDF shown in Figure 16(a), when using the lower-performing SSD, more than 7% of the requests are severely affected by long queuing delay and the maximum queue length reaches up to 2,700. It surpasses the experiment result with the better-performing 960EVO (e.g., 23 shown in Figure 8(b)). This is because when the IO intensity exceeds the ability of 660p SSD to accommodate, the SSD queue length builds up quickly. As a result in Figure 16(b), the average and tail latencies in Baseline rise sharply compared with 960EVO SSD shown in previous Figure 7. The average latency in Baseline is 90 ms and the 99^th-percentile latency exceeds 5 second.

With such high pressure on the 660p, MIOS can help reduce some IO burden on SSD by redirecting queued requests to HDDs. As seen from Figure 16, MIOS_D decreases the queue blockage with 45% queue length reduction to the maximum. At the same time, the average latency in MIOS_E returns to \(\mu\) s-level (e.g., 521 \(\mu\) s), and the 99^th and \(99.9^{th}\) -percentile latencies are reduced to an acceptable range of 2.4 ms and 87 ms, respectively. Because MIOS_E redirects much more SSD requests with low queue length, it prevents queue blockage in SSD, particularly for a lower-performing one. By comparing this experiment on a lower-performing SSD with an earlier one on a higher-performing SSD, we believe that when the SSD in hybrid storage cannot support the intensity of a write-dominated workload, MIOS and BCW can provide an effective way to improve the overall IO capacity by offloading part of the workload from SSD to HDD.

To further explore the effect of MIOS with the Intel Optane SSD, we replay the workload B that provides the highest IO pressure and employ the MIOS_D strategies. We do not use MIOS_E here because it is an aggressive strategy that redirects more requests to HDDs than MIOS_D. The excessive latency performance gap between HDDs and Optane SSDs will result in unacceptable performance degradation. Figure 17 shows that MIOS_D is still effective in the hybrid storage when using Optane SSD as primary write buffer. The \(99.9^{th}\) -tail latency is reduced by up to 19% and it also decreases the queue blockage with 60% queue length reduction to the maximum in workload B. Although the Optane SSD has less GC overhead than NAND-based SSD, they have similar ability to handle large size IOs (e.g., Optane SSD takes 230 \(\mu s\) while 960EVO takes 260 \(\mu s\) for a 512 KB write IO in average). Therefore, the intensive write workload can also block queues and increase tail latency in Optane SSD.

In addition, we compare BCW to a system that simply adds an extra SSD. We equally distribute the workloads to two SSDs. The system can achieve the same or even better latency than MIOS_E, but at a significantly increased hardware cost.

Read Performance. We measure the average, 99^th and \(99.9^{th}\) -percentile latency of external read requests (i.e., user reads) with MIOS_D, MIOS_E, SWR, and LBICA in all four workloads. We present all approaches normalized to the Baseline where all user data are directly written into the SSDs and then dutmped to HDDs.

The read performance of MIOS decreases slightly compared with that of the Baseline. Figure 18 indicates that the average read latency in MIOS_E is increased by 27%, 23%, 14%, and 28% in workload A, B, C, and D, respectively. The reasons are twofold. First, compared to Baseline, MIOS bypasses more SSD-writes directly to HDDs in advance, indicating that the HDDs have to serve more read operations. However, the majority of read requests in origin Pangu workloads are HDD-reads. For example, the proportion of SSD-reads in the total number of requests are less than 1.5% in all workloads. This is because Pangu periodically flushes data from SSD to HDD, and most of the data are already stored in HDDs. Therefore, MIOS_E turns up to 31% SSD-reads to HDD-reads and affects these average read latency slightly. Second, BCW uses the append manner to improve write performance and actively pads non-user data to HDDs. It results in the file data being scattered on HDD. However, we find that the overhead of BCW on HDD-reads is relatively small. This is because most read requests in these Pangu workloads are discrete spatially and temporally, i.e., random HDD-reads. In addition, the average size of the read requests is similar to that of writes, e.g., 4-16 KB, so that BCW rarely divides a read request into multiple read IOs. Note that, as described in Section 4.3, MIOS periodically writes logging data back to their interrelated chunk files, meaning that the long-term read performance in MIOS is identical with that in Baseline. Moreover, the increases of the 99^th- and \(99.9^{th}\) -percentile latency are less pronounced with MIOS_D and MIOS_E. Because a large part of the tail read latency is caused by queue blocking, which could get to hundreds of milliseconds. We also observe that read latency increases more with MIOS than that with SWR, since MIOS leads to more HDD-reads and has worse data locality in HDD. Besides, LBICA has little effect on both average and tail read latency because it redirects far less SSD write requests than other approaches.

We discuss the tradeoffs between improved write performance and degraded read performance caused by MIOS and BCW. In short, MIOS_D reduces the average and tail latency of write requests, which account for 84%-98% of the total requests, by 65% and 95%, respectively. In contrast, those of read requests, which account for 2%–16% of the total requests, rises by 7% and up to 10% respectively. For the MIOS_E strategy, the average and tail latency of write requests is reduced by up to 60% and 85%, and those of read requests increase by 28% and 18%, respectively. Therefore, we indicate MIOS approach gains more benefits for applications with write-dominated IO patterns, in which a few reads with limited latency increase could be accepted. Users can determine the appropriate scheduling policy according to current IO patterns. MIOS_E is more suitable for the burst and intensive-write environments, while MIOS_D can be used for write-dominated workloads. In addition, MIOS can be disabled in read-dominated situations.

We further analyze and evaluate the wasted storage-space of BCW due to the padded data for keeping continuous HDD written pattern and skipping the \(S\) write state.

Amount of Padded Data. We first analyze the amount of padded data written to HDD. In Table 5, we measure the data amount with MIOS_D and MIOS_E when a BCW sequence contains one Fast stage and 10 Mid/Slow stage-pairs. The statistics in the table clearly indicates that MIOS_E generates substantially more HDD write data than MIOS_D. When more requests are redirected, the amount of padded data increases proportionally. For example, the padded data with MIOS_E is 15 \(\times\) that with MIOS_D in workload A, 3 \(\times\) in workload B, 4 \(\times\) in workloads C and D. Frequently triggering BCW increases the occurrences of padded data. Furthermore, when the amount of redirected data increases, the Fast stage without padded data will be used up faster and more Mid/Slow stage-pairs with padded data will be executed.

HDD Utilization. The original Pangu traces exhibit low HDD utilization, which is defined by the percentage of time that an HDD is actually working on processing IO requests. More specifically, Tables 1 and 7 show that HDDs generally keep very low utilization (e.g., \(\lt\) 10%) in all four workloads. Using MIOS and BCW, the HDD utilization has been increased with different degrees. The gross utilization is defined to be the percentage of the total execution time when HDD is working on IO requests (including \(sync()\) operation), which is the real usage of the disks. The highest gross utilization is 56.9% under workload B. This means that the disk still has enough free time for HDD-GC. To analyze the amount of time HDD is effectively working for user requests, we define net utilization as the percentage of the total execution time that the HDD spends exclusively serving user requests, excluding the time spent on padding data in BCW. Thus, it is positively correlated to the amount of redirected data. The net utilization of HDD in MIOS_E is higher than that in MIOS_D. Under workload B and MIOS_E, HDD has the highest net utilization improvement over Baseline, by 2.7 \(\times\) , while the same is enhanced to 1.8 \(\times\) under MIOS_D. In addition, if MIOS enables multi-HDD scheduling, the HDD utilization would drop significantly. First, multiple HDDs would serve the data volume of SSD redirect requests. Second, multi-HDD scheduling does not require BCW to pad data frequently, which results in less padded data and lower gross utilization.

The efficiency of BCW heavily depends on the write intensity. To better understand this relationship, we test the average and tail latency as a function of write intensity (in terms of IO transmission interval) with three scheduling strategies as the Baseline (_B), MIOS_D (_D), and MIOS_E (_E). We initialize the IO size to 32 KB, and continuously issue write requests using (Flexible I/O tester) [5]. Since FIO cannot adjust IO intensity, we set the generated IOs with a fixed transmission interval from 20 to 320 \(\mu\) s. We use 960EVO SSD and 10 TB WD HDD, and set the \(L\) value to 1.

Figure 19 shows that when the request interval is between 20 and 60 \(\mu\) s, the SSD writes are severely blocked and the 99^th-tail latency reaches as high as 5.2 second. In this case, both MIOS_D and MIOS_E can significantly reduce the tail latency. And MIOS_E is slightly better than MIOS_D because the former handles more burst writes. When the interval is 60–80 \(\mu\) s, Baseline still exhibits very high latency in SSD. However, the latency has returned to an acceptable \(\mu\) s-level after MIOS scheduling. When the interval exceeds 100 \(\mu\) s, the average and 99^th-percentile latencies are stable, because there is very little SSD queue blockage under such request intensity. In this case, MIOS_D and Baseline have the lowest average latency and remain the same as request interval grows. However, the average and tail latency of MIOS_E is higher than others. This is because even if there is no queue in SSD, MIOS_E will still redirect requests, and the performance gap between SSD and HDD can lead to high latency.