3.2 Reprogram Operation on 3D TLC
In this work, we develop a novel reprogramming technique to avoid further narrowing the range between voltage states in TLC flash cells. The basic idea is to first program a TLC as an MLC by only using the first four voltage states to represent 2-bit value, as shown at the top of Figure
3(a). Then, as the threshold voltage can keep increasing but not decreasing, low voltage states can be reprogrammed to higher voltage states to represent new values. In this case, a cell can be reprogrammed several times before a true erase is necessary. This is in contrast to the conventional model where a cell is programmed only once and the second program must be preceded by an erase.
In our design, the program operation is realized by one-shot programming, which writes two pages at a time from the write-back registers to a word line. In these two pages, the paired bits with the same in-page offset are used to determine the desired state of a cell. One-shot programming injects electrons into each flash cell until its current threshold voltage reaches the desired voltage state, which is coded to represent the values of two bits (MSB and LSB) [
5]. Once one of the two bits in a cell becomes invalid, a reprogram operation can be carried out to transform the old 2-bit value into a new 2-bit value represented by a higher voltage state in the cell. If the old 2-bit value remains valid, reprogramming on this cell is then prohibited. If both of the two bits become invalid, reprogramming on this cell is performed by assuming one of the two bits still is valid. The reason comes from two aspects: First, if only one page is going to be reprogrammed, the proposed reprogramming method can be performed immediately without waiting for another one page; Second, thanks to the reliability impact from cell-to-cell interference, reprogram operation selects the target word line in an interleaving manner [
5], which requires to read and check first reprogrammed data before performing the second reprogram operation. If there are no additional reprogrammable word lines in the super layer, reprogramming invalid bits one by one also benefits from the interleaving manner. We create such a design on TLC cells and discovered that to cover all possible bit update sequences with altogether eight valid voltage states, two reprograms can be carried out after the initial program. Hence a cell can be programmed three times. The transitions from old 2-bit states to new 2-bit states are enumerated in Figure
3. At the top of Figure
3, the leftmost four voltage states (ER
\(\sim\) P3) are used to represent a 2-bit value after the initial program operation. Then, according to which bit is invalidated, a new voltage distribution is constructed by using a current or higher voltage state to represent the new 2-bit value. More details are presented as follows.
As a 2-bit value can only be updated twice on a cell, depending on which bit is invalidated first, there are four possible reprogramming sequences that can be represented by four voltage distributions, as depicted in the four subfigures of Figure
3. Then, since each bit can be reprogrammed into two possible values, there exist eight transitions as listed in Table
1. Take Figure
3(b) as an example. The two transitions from top to bottom correspond to the invalidation of the LSB and then the MSB, as shown in the left two columns and right two columns in Table
1, respectively. For the first transition that updates LSB, there are eight possible state changes, as listed in the left two columns of Table
1. The voltage distribution needs to be reprogrammed to higher states and re-encoded because otherwise, there would be two backward state transitions, which are impossible without an erase. These two backward transitions are “00”
\(\rightarrow\) “01” and “10”
\(\rightarrow\) “11”. Hence, “11” and “01” are mapped to P4 and P5 states, respectively, shifting the four states to the middle of the entire voltage range, as depicted in the middle of Figure
3(b). Hence, all LSB updates are represented by increasing voltages, which are viable in one-shot programming. Similarly, as the MSB is updated in the second transition, state P3
\(\sim\) P6 are utilized and re-encoded to represent new voltage distribution. Specifically, “00” is represented by P4 and “11” is represented by P6, to avoid backward state changes. Totally, after reprogramming a cell twice, the total number of bit-per-cell can be increased from 3 to 4.
Before reprogramming a cell, current-voltage distribution, 2-bit value inside the cell, and current validities of LSB and MSB should be read out first to determine the next reprogramming voltage state. Then, to distinguish the voltage distribution of a cell after reprogram operation, each voltage distribution is assigned with a unique 2-bit ID, termed
Distribution ID. Specifically, “00” represents voltage distribution from P1 to P4; “01” represents P2 to P5; “10” represents P3 to P6 and “11” represents P4 to P7. Since
Distribution ID is effective only when the cell has been reprogrammed, another counter (
RPCnt) is used to record how many times the cell has been reprogrammed. By default, the value of
RPCnt is set to “00”. It is incremented every time the cell is reprogrammed. A value of “10” prohibits future reprogramming on this cell. For a read operation, with the awareness of
RPCnt and
Distribution ID, the value stored in the reprogrammed cell also can be read out by only applying two sensing operations [
4,
6]. Then, the current validities of programmed two bits (
Validity) are used to determine the next voltage distribution before reprogramming the cell.
Validity requires 2-bit to indicate valid states of LSB and MSB, respectively. In default, the value of
Validity is set to “11” after the initial program operation, meaning both LSB and MSB are valid. If one of the programmed two bits is invalidated, the corresponding bit in
Validity is set to “0”. For example, as shown at the top of Figure
3(a), after initial program operation, if LSB is invalidated,
Validity is set to “10”, meaning the cell can be reprogrammed to update the value of LSB. After reprogramming the cell, the value of the corresponding bit in
Validity should be set to “1”, meaning the reprogrammed LSB in the cell is valid now and is prohibited to be further reprogrammed until its value is invalidated again. In this work,
RPCnt,
Distribution ID, and
Validity are termed as cell status. Note that, since all flash cells in a word line are programmed or reprogrammed simultaneously,
RPCnt,
Distribution ID, and
Validity are needed on per word line basis. They are checked before each program or reprogram operation.
We use Figure
3(a) as an example to illustrate the entire hardware procedure of reprogramming with the aid of metadata. Suppose the initial program operation writes (“
00”) in the TLC. The values of
RPCnt,
Distribution ID, and
Validity are set to “00”, “00”, and “11” by default. As the LSB is invalidated, the value of
Validity is set to “10”. Then, the value of LSB is updated to “
1” (“0
0”
\(\rightarrow\) “0
1”), the corresponding voltage state of the new value is mapped to the P5 state for constructing new voltage distribution as presented in the middle of the figure. Moreover,
RPCnt and
Distribution ID are set to “01” and “01” to represent the current cell status. After reprogramming the cell, the value of
Validity is set to “11”, indicating current values of two bits in the cell are valid. For the second reprogram operation, with the aid of
Distribution ID, the current 2-bit value (“01”) inside the cell is read out first. We assume that the LSB of “01” is invalidated (the value of
Validity is set to “10”) and updated to “
0” (“0
1”
\(\rightarrow\) “0
0”). Thus a new voltage distribution presented at the bottom of the figure (final distribution) is constructed to represent new value (“00”) in the P6 state. After that,
RPCnt and
Distribution ID are updated to “10” and “11” as the cell cannot be reprogrammed anymore before an erase. And the value of
Validity is also set to “11” after the second reprogram operation on the cell. In summary, at any point in time, there can be at most two bits inside a cell. However, when a cell is reprogrammed twice, four bits of information are programmed or reprogrammed into a TLC (“
00” in first initial program, “
1” in first reprogram, and “
0” in second reprogram). As a result, from the overall SSD perspective, the effective capacity can be increased by reprogramming a cell twice. Note that, effective capacity is defined as the number of free flash pages in SSDs. If more reprogram operations are executed, fewer free flash pages are consumed while more data is reprogrammed in a single word line. Thus, fewer GCs are triggered so that the performance and endurance can be improved.
For a read operation, when the accessed page resides in the reprogrammable block, the word line’s status is read from controller buffer firstly. According to the value of Distribution ID, the voltage distribution can be determined; thus, sensing operations are performed to read LSB or MSB data from reprogrammed cells. If the accessed page belongs to normal block, regular read operation is performed to get data from TLC cells.
3.3 Reprogram Operation Verification
To verify the feasibility of reprogram operations on 3D TLC SSDs, we evaluate them on YEESTOR 9083 flash memory testing platform [
22,
59] with Micron B17A Series NAND flash chips [
46]. YEESTOR 9083 flash memory testing platform, is a high-reliability and high-performance flash memory controller that enables users to implement specified flash memory solutions.
As discussed earlier, reprogram operation is enabled as one of the old two bits has been invalidated in a cell. Since there are many word lines in a block in 3D TLC SSDs, flash cells in a word line may be invalidated and rewritten whenever necessary, and not follow conventional sequential program order. With the capability of reprogramming, in our verification design, we relax the restriction of sequential order among word lines through two steps. In the first step, the restriction of sequential order among word lines in a layer is relaxed. All word lines in a layer are reprogrammed in a random order while layers are reprogrammed sequentially. Such a reprogramming scheme is termed as “One-Layer Reprogram” (OLR) in this work. In the second step, the restriction of sequential order among layers is relaxed by reprogramming word lines in multiple layers in a somewhat random order. We term such scheme as “Multi-Layer Reprogram” (MLR). More details about OLR and MLR will be presented in the following subsections. In the verification design, to show the maximal capability of a cell to be reprogrammed, each cell in the following experiments is reprogrammed four times to completely shift the initial voltage distribution to the rightmost side.
3.3.1 One-Layer Reprogram.
Evaluation Methodology: For the OLR scheme, initially, all word lines within each layer are sequentially programmed such that all flash cells in each word line are programmed in an MLC mode. Then, word lines in the layer are randomly selected to reprogram flash cells by moving the first four states to the rightmost four states step by step. To evaluate the reliability of OLR, the worst voltage distribution (ER
\(\sim\) P7) of a cell is evaluated and presented in Figure
4, where the
x-axis represents the voltage threshold and the
y-axis represents the number of flash cells falling into different voltage states. For OLR, one program operation is carried to initialize the voltage distribution and four reprogram operations are performed to move first four voltage states (including the ER state) to the following states. As shown in the figure, these voltage states are distributed to different threshold voltage ranges separately.
Reliability Impact of OLR: To analyze the impact of reprogramming on reliability, the maximum and average numbers of bit errors per page are evaluated. There are two types of read methods for reading and checking the raw data on each page. The first read method uses
default voltage offsets (
DVO), and the second read method adopts voltage calibration that can minimize the number of page bit errors by applying
optimized voltage offset (
OVO) between every two neighboring states [
4,
6,
23,
46], which has been used in SSD products. The results of maximum and average numbers of page bit errors collected by these two read methods are presented in the second row of Table
2. In contrast, the results of the conventional program scheme are evaluated as well and presented in the first row of this table. As we can see, with the voltage calibration read method, the numbers of maximum and average page bit errors are increased by 4 and 1, respectively, from the conventional program scheme. Given the strong ECC capability (e.g., 80bits/1KB) in modern SSD products [
23,
46]), the increased number of page bit errors resulted from OLR is negligible. Therefore, we conclude that reprogramming flash cells layer by layer in 3D TLC SSDs is effective and reliable.
3.3.2 Multi-Layer Reprogram.
To relax the restriction of sequential order among layers, in our design, we group multiple physical layers as a reprogrammable super layer (R-SL). Inside each R-SL, physical layers are reprogrammed in a random order, while R-SLs are programmed in sequence.
Evaluation Methodology: We measured the feasibility of MLR starting with a super layer containing two physical layers. The evaluation methodology is similar to that for the OLR. Initially, all word lines in a super layer are sequentially programmed, and all cells are programmed as an MLC utilizing only the ER
\(\sim\) P3 states. Then, reprogram operations are carried out in randomly selected word lines to move the first four states to the rightmost four states step by step. To evaluate the reliability of MLR with two-layer reprogramming, the worst voltage distribution of a cell is presented in Figure
5(a), where eight states can still be clearly distinguished by reference voltages. When MLR is evaluated with three physical layers per super layer, the worst voltage distribution tends to be error-prone, as seen in Figure
5(b). In this figure, the left tails of some voltage states are largely stretched across reference voltages.
Reliability Impact of MLR: Quantitatively, the maximum and average page bit errors per page of MLR with two-layer reprogramming are 46 and 9, respectively, as presented in the last two rows of Table
2, which can still be corrected by ECC. However, as the number of physical layers per super layer increases to three, MLR produces too many bit errors to be correctable (234 in maximum and 11 on average), due to a large overlap between adjacent voltage states in the far right, as seen in Figure
5(b). Those errors indicate that MLR with three layers per super page may not be reliable enough, especially in the evaluated 3D TLC flash chips, considering there are other error sources such as P/E cycles induced errors and retention error [
4]. When we further increased the size of super layers, the number of bit errors per page is increased to thousands even under read voltage calibration. The voltage distribution becomes worse.
Therefore, the maximum number of physical layers that can be reprogrammed as a super layer in this work is set to two. 3.3.3 Reliability Discussion.
We first discuss why reprogram operations introduce challenges in maintaining reliability. Then, we give further evaluations on reliability when aggregating all error sources from reprogramming, P/E cycle, and retention.
Reliability Challenges: There are two main reasons for reprogram operation to challenge reliability: cell-to-cell interference [
4,
7] and
background pattern dependency (
BPD) [
13,
31,
36]. The former is slightly weaker and the latter is dominant in harming reliability.
First, reprogram operation will aggravate its harm to the reliability due to the impact of cell-to-cell interference, which is highly related to how many times neighboring cells are programmed or reprogrammed [
7]. In the OLR and MLR evaluated above, each cell is programmed once and reprogrammed four more times to fully moving voltage state to the rightmost side, causing four more times cell-to-cell interference to neighboring cells. Put it in another way, each cell receives four more times cell-to-cell interference from reprogramming on top of the cell from program operations in the conventional scheme. However, in real SSD products, with applying voltage distribution arrangement, a cell can only be reprogrammed twice at most so that the impact of the cell-to-cell interference can be further mitigated.
Second, BPD means the current-voltage characteristic of a cell can be affected by whether other cells in the same cell string (a cell string means flash cells sharing the same bit line) have been programmed or reprogrammed. The resistance of a cell string increases as more cells are programmed in the same string [
31]. Therefore, BPD makes the electrons hard to be injected into a cell if there are cells atop already programmed [
44]. To suppress the reliability impact from BPD, a sequential program order is placed to program word lines from the bottom (closest to the source line+) to the top (closest to the bit line contact), as shown in Figure
1. However, the proposed OLR and MLR break conventional sequential program orders, aggravating the reliability impact from BPD. As a result, some cells cannot be reprogrammed to the desired state within the manufacturer guaranteed maximum times of program and verify iterations, creating the left tail of voltage distribution, as depicted in Figure
5. In addition, such a left shift in voltage distribution also shows that BPD is the chief influence on reliability, as cell-to-cell interference tends to shift voltage distribution to the right side [
7].
Reliability Impact from Other Error Sources: Here we quantitatively analyze how reprogram operations worsen reliability when there are already error sources such as P/E cycles and retention time.
First, we test the reliability of cells when they are stressed under both P/E cycles and reprogramming. An experiment is constructed by reprogramming two blocks with two types of P/Es, the normal P/E and reprogram P/E. The former means a block is repeatedly programmed and erased without reprogram operations, and the latter means a block is worn by repeated MLR with two-layer reprogramming and normal erase operations. After 1 K normal and reprogram P/E cycles on two blocks, the maximum and average numbers of bit errors per page are collected and presented in the left two columns of Table
3 (under w/o Retention). Compared to normal P/Es, reprogram P/Es increase the maximum and the average number of bit errors by 7 and 4, respectively. Such a slight increase in page bit errors can be easily corrected by ECC. Moreover, the experiment also proves that reprogramming a cell multiple times before an erase operation induces little wear stress on a cell. The reason is that the stress duration of erase operation is much longer than that of program and reprogram operations [
29]. That is, the main wear stress on a cell comes from erase operation even when a cell is reprogrammed multiple times.
Second, we test the reliability with further error sources from retention when the cells are already stressed under P/E cycles and reprogramming. The above evaluated chip is heated to 120
\(^{\circ }\) C for three hours, of which the effective retention time is 1 year under 40
\(^{\circ }\) C [
2,
3]. The maximum and average numbers of page bit errors per page collected by reading voltage calibration are presented in the right two columns of Table
3. Compared to normal P/E, reprogram P/E increases the maximum and average numbers of page bit errors by 21 and 15, respectively. In the worst case, after 1 K reprogram P/E and 1-year retention under 40
\(^{\circ }\) C, there are 121-bit errors per page, which still can be corrected by ECC. Notice that, according to the results presented in Tables
2 and
3, the maximum number of bit errors after 1 K reprogram P/E cycles (27) is even smaller than that of MLR with two-layer reprogramming without P/E impact (49). The reason is that the chips evaluated in these two experiments are different but with the same specification. We remark here that the impact of
process variation (
PV) [
51,
54] dominates error rates over reprogramming.
Based on the above results, we conclude that the reprogram operation still is reliable enough even when P/E cycles and retention time are considered.
Risk of Capacity Loss: By degenerating TLC to MLC, the number of bit-per-cell is reduced from three to two, causing capacity loss of SSD. On the other hand, if data is frequently updated, reprogramming can write more bits into a cell before an erase, increasing effective capacity. Hence, reprogramming hot data is more beneficial than cold data in terms of capacity. However, if the hotness of programmed data cools down and remains valid, the reprogram operation on this cell will be prohibited, thereby resulting in capacity loss. Moreover, given the sequential order among R-SLs, previous R-SLs are not allowed to be reprogrammed again once the next R-SL is activated. Thus, the capacity may become lost while flash cells in the previous R-SL have not been reprogrammed. An extreme case is that each reprogrammable layer is programmed into MLC mode but never reprogrammed. Then one-third of capacity is lost.
However, such capacity loss does not have lasting effects on SSD, due to the GC process that is used inside SSD to reclaim invalid or lost space. In this work, if there exists lost space, GC is activated to move valid data from reprogrammable blocks to normal blocks as cold data [
57], and then, the reprogrammable block is erased and reset to a normal block. Additionally, if only hot data can be reprogrammed, most data in reprogrammable blocks tend to be invalid as each page can only be programmed twice. Hence, reprogrammable blocks are more likely to be GC-friendly, meaning that they require less copies for valid data. Since hot data is frequently invalidated and updated, reprogram operation induced page bit errors will not be propagated by reading and rewriting updated data.
Selective Reprogram: Based on the above analysis, we draw the conclusion that reprogramming flash cells in 3D TLC SSDs is feasible but should be applied selectively. First, a flash cell can be reprogrammed only when some of the programmed bits are invalidated. Second, the maximum number of physical layers per R-SL should not exceed two for reliability reasons (according to the chips we tested). Third, to avoid capacity loss, only hot data that can be updated soon is suggested to be reprogrammed.