1 New Codes and Inner Bounds for Exact Repair in Distributed Storage Systems Sreechakra Goparaju Salim El Rouayheb Robert Calderbank Abstract arXiv:1402.2343v1 [cs.IT] 11 Feb 2014 We study the exact-repair tradeoff between storage and repair bandwidth in distributed storage systems (DSS). We give new inner bounds for the tradeoff region and provide code constructions that achieve these bounds. I. I NTRODUCTION The study of erasure codes which repair single node failures in a distributed storage system more efficiently was initiated by Dimakis et al in [1]. The codes, called regenerating codes, are constructed on a system of n storage nodes and store a file of size M, such that the data stored in any k nodes is sufficient to recover the M symbols. Furthermore, each node stores α symbols and each failed node requires a transmission of γ symbols, termed as the repair bandwidth, by d other nodes, called helper nodes (d is also known as the repair degree). An optimal tradeoff between storage and repair-bandwidth (α and γ) is characterized in [1] for the so-called case of functional repair, where the repaired (or replaced) node need not store the exact copy of the data present in the failed node. This tradeoff region is given by the following expression: M ≤ k−1 X n γo min α, (d − i) . d i=0 (1) However, when an exact duplication of data onto the repaired node, or exact repair, is required, this characterization remains an open problem in general. Since exact repair is a stricter constraint than functional repair, the tradeoff curve in (1) forms an outer bound to the exact-repair region for α and γ (for a given M, n, k and d). The extremal points of this tradeoff have attracted the most attention, being the optimal points in terms of α and γ. These are respectively called the minimum storage regenerating (MSR) point, with α = αmsr = γ = γmsr = M , and k dM , k(d − k + 1) (2) and the minimum bandwidth regenerating (MBR) point, with α γ = = αmbr γmbr = = 2dM , and k(2d − k + 1) 2dM , k(2d − k + 1) (3) Both points have been shown to be achieveable for exact repair for all (n, k, d), using finite-length or asymptotic codes; see, for example, [2], [3], [4], and [5]. Recently, Tian [6] fully determined the optimal tradeoff for the case of an (n, k, d) = (4, 3, 3) DSS. The optimal tradeoff regions for all other cases of (n, k, d) are still under investigation. Indeed, for some time, no inner bounds existed that were tighter than the space-sharing (or time-sharing) bound between the MSR and the MBR points. However, recently, new codes have been independently discovered, which improve upon the space-sharing bound, and lie in the so-called intermediate region, that is, between the MSR and the MBR points. In [7], block-designs S. Goparaju is with the EE Department, Princeton University, USA (e-mail: goparaju@princeton.edu) S. El Rouayheb is with the ECE Department, Illinois Institute of Technology, USA (e-mail: salim@iit.edu). R. Calderbank is with the EE Department of Electrical Engineering, Duke University, USA (e-mail: robert.calderbank@duke.edu). 2 1 MSR (1/3,1) information theoretic tradeoff with functional repair Space sharing achievable tradeoff with exact repair New achievable tradeoff with exact repair 0.95 0.9 Total repair bandwidth γ 0.85 (2/5,4/5) 0.8 0.75 (2/5,3/4) new codes 0.7 0.65 0.6 (2/5,3/5) MBR (1/2,1/2) 0.55 0.5 0.32 0.34 0.36 0.38 0.4 0.42 0.44 0.46 0.48 0.5 Storage per node α Fig. 1. Example of the new achievable exact-repair tradeoff for the (5, 3, 3) case for a file size M = 1. The code construction achieving the point ( 25 , 34 ) is described in Sec. II. and erasure codes were used in a layered structure to build intermediate codes which are simple to implement. In [8], layering is used to construct codes first for the parameters (n, k, k), i.e., when d = k, and then extended to (n, k, d), when d > k. In [9], new (n, k, d) regenerating codes at intermediate points are formed using MSR points for lower parameters. Contribution: We construct new intermediate codes which in turn lead to newer inner bounds for the storage/repair bandwidth tradeoff region for exact repair. As an example, for a (5, 3, 3) DSS, our codes achieve the new point (α, γ) = (2/5, 3/4), which leads to the new achievable tradeoff shown in Fig. 1. This example is explained in detail in Section II, and is used to provide an intuition behind our code construction. Our technique is inspired by the constructions in [8] and [9], and the study of heterogeneous DSS in [10]. We use MSR code constructions for smaller codes to achieve intermediate points for larger parameters. We then generalize this technique in order to improve the repair bandwidth γ while keeping the rest of the parameters – n, k, d, α, and M – constant. In Section III, we describe our code construction and the ensuing tradeoff inner bound for the case when d = k. In Section IV, we extend this construction to the case when d > k. We then describe a second construction for this regime which uses ideas in [8] in a flavor similar to the way we improve the repair bandwidth. We show that this construction proves as useful as the first construction by plotting the tradeoff region for an example DSS; see Fig. 4. Finally, we conclude in Section V. II. P RIMER : A (5, 3, 3) E XAMPLE To illustrate our results, we start with a (5, 3, 3) DSS consisting of five nodes v1 , . . . , v5 . We give a new regenerating code for this DSS, which leads to an improved inner (achievable) bound on the storage/repair bandwidth tradeoff curve for this DSS under exact repair. The general results and proofs will be detailed in the subsequent sections. The idea is to start with an MSR code for a smaller (4, 2, 3) DSS with (n̂, k̂) = (4, 2) consisting of the nodes {v1 , . . . , v4 } as in Fig. 2(a). Here, we pick the (4, 2, 3) MSR code from [11]. Each node stores half a symbol (α̂ = 1/2), making the total stored file size M = 1. We extend this code by adding an empty node v5 . The resulting DSS is heterogeneous since different nodes store different amounts of data. Moreover, the total repair bandwidth depends on which helper nodes participate in the repair process. There are two cases here. Case 1: All the d = 3 helper nodes belong to the small MSR code, i.e., are chosen from {v1 , . . . , v4 }. Here, we can achieve the optimal repair bandwidth given by the MSR code, i.e., the repair bandwidth is 3/4. Case 2: Only two nodes of the small MSR code help in the repair, the third node being v5 . In this case, the repair proceeds by downloading the whole file from the small code (and nothing from node v5 ). The repair bandwidth is 1. 3 v1 A1 A2 A1 A2 v2 B1 B2 B1 B2 A1 A2 v1' B1+B2 B1+B2 A1+B1 2A2+B2 v3 A1+B1 2A2+B2 2+ A1+2A 2 2A1+B1 A2+B2 2A1+B1 A2+B2 v5 empty node empty node A2 1+ 2A v4 A1 A2 B1 B2 A1+B1 2A2+B2 B 1+ +B total repair bandwidth γ = 3/4 2A1+B1 A2+B2 v1' A1 A2 A1+B1 2A2+B2 2A1+B1 A2+B2 ∅ total repair bandwidth storage α empty node γ=1 = 1/2 (a) (b) (c) Fig. 2. Part (a) A (5, 3, 3) code constructed by appending a (4, 2, 3) MSR code [11] to an empty node. Part (b) shows the repair of a non-empty node in the (5, 3, 3) system when the d = 3 helper nodes belong to the MSR code. Part (c) shows the repair when the d helper nodes consist of only 2 nodes in the MSR code. From [10], we know that we can construct a homogeneous code by “glueing” together all the n! = 120 permuted copies of the heterogeneous code. This glued code achieves the average storage per node α and average repair bandwidth γ of the heterogeneous code. For the code in Fig. 2, a fraction of 4/5 of the nodes are non-empty. Furthermore, for the repair of a non-empty node, a fraction of 1/4 of helper sets consists of three helper nodes (Case 1) in the (4, 2, 3) code. Thus, we have 2 15 , and γi = , for i ∈ {1, 2, 3, 4}, 5 16 where γi is the average repair bandwidth when repairing node vi , i = 1, . . . , 5. The repair bandwidth γ5 when node v5 fails is zero since no data needs to be downloaded. Therefore, the average total repair bandwidth, over all failures and all choices of helper nodes, is γ = 3/4. The tradeoff curve for functional repair [1] is given by the piecewise linear curve joining the MSR point (α, γ) = (1/3, 1), an intermediate point (2/5, 3/5) and the MBR point (1/2, 1/2) (see Fig. 1). The MSR and MBR points are also achievable with exact repair [4], [2]. By space sharing, we can achieve any point on the straight line joining the MSR and MBR points. The code constructions in [9] and [8] both correspond to the point (2/5, 4/5) on the space sharing curve for the (5, 3, 3) example. The proposed code lies beneath this curve in Fig. 1 and thus leads to an improved achievable tradeoff curve for exact repair. α = III. C ODE C ONSTRUCTION FOR d = k Our goal is to build (n, k, d) regenerating codes with exact repair for intermediate points on the tradeoff curve. We focus first on the case when d = k and address the case when d > k in the next section. We follow the method in [9] and build the (n, k, d) code from a small (n̂, k̂) code to which we append n − n̂ empty nodes as depicted in Fig 3. In the previous example, we had (n, k, d) = (5, 3, 3) and (n̂, k̂) = (4, 2). Notice that we do not specify the repair degree dˆ yet. A. Construction The key ingredient in our construction is to pick the small code to be an MSR code with minimum repair bandwidth for all possible values of the repair degree dˆ = k̂, k̂ + 1, . . . , n̂ − 1, simultaneously, that is, the repair bandwidth γ̂dˆ when the repair degree is dˆ is given by dˆ α̂, (4) dˆ − k̂ + 1 where α̂ is the storage per node in the small code; see (2). Constructions of MSR codes that can asymptotically achieve the minimum repair bandwidth for any repair degree were described in [4] and are based on the interference γ̂dˆ = 4 v1 small code .. . vk̂ dˆ .. . .. . vn̂ empty nodes Fig. 3. new repaired node .. . vn̂+1 .. . d − dˆ vn General code construction. alignment method. This code property, referred to as universality in [4], has been studied in depth in [12], where it is referred to as opportunistic repair. Note that for the special case of n̂ − 1 = k̂ + 1, as in the example, any MSR code has this property since for dˆ = n̂ − 1 = k̂ + 1, optimal repair bandwidth follows from the definition of an MSR code, and for dˆ = k̂, the optimal repair requires downloading the whole file. Hence we do not need to worry about dˆ when choosing the parameters of the small code (as long as n̂ − 1 ≥ k̂, which is true for a non-trivial small code). Since file reconstruction and exact repair in the big code are inherited from the small code, we need to pick k̂ to make sure that among the k nodes from {v1 , . . . , vn } contacted by a user, there are always k̂ nodes from {v1 , . . . , vn̂ }. This is achieved by choosing n̂ and k̂ such that the big code and small code have the same number of parity nodes, i.e., n − k = n̂ − k̂. (5) Also, since d = k, the repair process is guaranteed to contact at least k̂ nodes from {v1 , . . . , vn̂ }, and thus can be inherited from the small code. We vary k̂ to take all the integer values between 1 and k while choosing n̂ = k̂ + (n − k). Each value of k̂ then results in a distinct regenerating code. Ultimately, the code is formed of all the n! permuted copies of the code in Fig. 3 glued together1 as in [10]. This code achieves the point (α, γ), where α and γ are the average node storage capacity and repair bandwidth of the code in Fig. 3, respectively. Since the storage per node in the small code is α̂, we obtain n̂ α̂. (6) n Let γ1 be the average repair bandwidth when repairing a non-empty node. Among all the possible sets of helper nodes, there is a fraction Pdˆ (defined in (7)) of sets that consist of exactly dˆ nodes in the small code, where dˆ ranges from k̂ to dˆmax and dˆmax = min{n̂ − 1, d}. So we obtain α γ1 = dˆmax X = γ̂dˆPdˆ, ˆ k̂ d= = dˆmax X ˆ k̂ d= 1 It dˆ dˆ − k̂ + 1 !    n̂ − 1 n − n̂ dˆ d − dˆ   α̂. n−1 d may be possible to make the system homogeneous without using all the n! permutations, but we don’t worry about this issue here. (7) 5 Because the average repair bandwidth when repairing an empty node is zero, the overall average repair bandwidth γ is given by    n̂ − 1 n − n̂ ! ˆ dmax X dˆ dˆ d − dˆ   γ = α. (8) ˆ − k̂ + 1 n−1 d ˆ k̂ d= d Since the small code is MSR, the stored file can be of size up to M = k̂ α̂. (9) B. Inner Bound From the above construction and from (6), (8), and (9), we have the following new achievable exact-repair tradeoff. Since it is possible that the points obtained by the construction do not enclose the MBR and the MSR points, we also add them to the tradeoff region. Theorem 1: There exist (n, k, d) exact-repair regenerating codes, with d = k, that store a file of size M and achieve any point in the region con(R), the convex hull of R, where R = R1 ∪ Rmbr ∪ Rmsr , k [ o (ᾱ, γ̄) : ᾱ ≥ α(k̂), γ̄ ≥ γ(k̂) , (11) Rmbr = {(ᾱ, γ̄) : ᾱ ≥ αmbr , γ̄ ≥ γmbr } , and (12) Rmsr = {(ᾱ, γ̄) : ᾱ ≥ αmsr , γ̄ ≥ γmsr } . (13) R1 = k̂=1 n (10) Here α(k̂) and γ(k̂) are given by (6) and (8), respectively. Notice that k̂ also determines n̂ from (5), and α̂ is determined by (9). Remark 1: For the case of d = k, the two code constructions in [9], [8] achieve the same points on the tradeoff, that are obtained when the summation in (8) contains only the term corresponding to dˆ = k̂. The inner bound in Theorem 1 is thus an improvement whenever the summation in (8) has more than one term. This happens for all system parameters except when k = d = n − 1, for which the three inner bounds coincide. Remark 2: In general, we only know of asymptotic schemes [4] which achieve the repair bandwidth given in (4) ˆ However, for n − k = 2, the summation in (8) has summands corresponding to the two for all feasible values of d. ˆ extreme cases of d = k̂ and dˆ = k̂ + 1, where the former requires each helper disk to transmit all its data as repair bandwidth, and the latter requires each helper disk to transmit only 1/(n − k) of its data. The latter is the optimal repair bandwidth for an (n̂, k̂, n̂ − 1) MSR code, for which several finite code constructions exist, e.g. [2], [3] and [5]. IV. C ODE C ONSTRUCTIONS FOR d > k The construction in Section III can be generalized to the case when d > k by using the same small code as before. The difference however arises during the repair of a failed node in the big code, as detailed in Construction 1 below. A second construction, Construction 2, can be achieved using similar ideas, this time following the method in [8]. A. Construction 1 As before, we construct the big (n, k, d) regenerating code using the permuted copies of a small MSR (n̂, k̂) code, where n̂ − k̂ = n − k, and which is repair-bandwidth-optimal for all possible degrees of repair dˆ ∈ {k̂, k̂ + 1, . . . , n̂ − 1}. 6 Upon failure of a non-empty node, say v1 (see Fig. 3), in the (n, k, d) regenerating code, a fraction Pdˆ of helper sets consist of exactly dˆ nodes in the small code. The repair bandwidth γ1 is then given by γ1 = dˆmax X γ̂dˆPdˆ, (14) ˆ dˆmin d= = dˆmax X ˆ dˆmin d= dˆ ˆ d − k̂ + 1 !    n̂ − 1 n − n̂ d − dˆ dˆ   α̂, n−1 d (15) where dˆmin = d − (n − n̂), and dˆmax = min {n̂ − 1, d} as defined earlier. Notice that unlike when d = k, any d (helper) nodes must contain at least d − (n − n̂) nodes in the small code, where it can be verified that dˆmin > k̂. Since the repair bandwidth γi for the failure of an empty node vi is 0, the overall repair bandwidth γ is given by    n̂ − 1 n − n̂ ! ˆ dmax X dˆ d − dˆ dˆ   α. (16) γ = ˆ n−1 d − k̂ + 1 ˆ dˆmin d= d The stored file size, as before, is given by M = k̂ α̂, and the average storage per node is given by α = (n̂/n)α̂. B. Construction 2 When d > k, an alternative code can be constructed by viewing the big (n, k, d) regenerating code as an (n, d, d) regenerating code as obtained by Construction 1 (in Section III), and calculating the amount of information2 that any set of k nodes contains. This quantity Mk is the same for any set of k nodes from the symmetry in Construction 1. It can be further verified that Mk is equal to the average amount of information contained in all the possible sets of size k (in the heterogeneous code). Let Md (≥ Mk ), be the file size of the (n, d, d) code obtained from the construction, and let Mk be the size of the file that we wish to store in the DSS. The underlying motivation is to enable the recovery of the file from any k nodes (which as we defined contain Mk amount of information). One way to achieve this is to concatenate a maximum rank distance (MRD) code and the (n, d, d) code. To do so, the MRD codeword is constructed using a linearized polynomial with coefficients as the Mk symbols of our file. The output codeword of this code can then be viewed as a set of vectors which are evaluations of this polynomial on Md points in a specific field, which are then fed as the input message file for the (n, d, d) regenerating code constructed using Construction 1. The details of this concatenation are skipped here but can be found, for example, in [8]. We now calculate the overall file size M (= Mk ) achievable. Suppose that the small code being used is an (n̂, k̂) MSR code, where now n̂ − k̂ = n − d. Let this code have α̂ units of storage per node and be repair-bandwidth optimal for all possible degrees of repair dˆ ∈ {k̂, k̂+1, . . . , n̂−1}. Consider any set K of k nodes in the big code. We noticed in Section III when d = k that K contains at least k̂ nodes of the small code in all permutations of the small code and (n − n̂) empty nodes. When d > k, we use the property that the amount of information (entropy or linear dimension for the case of linear codes) contained in a set of ω nodes in the (n̂, k̂) MSR code is given by Hω = min{ω, k̂} α̂. Therefore, if Qω is the fraction of permutations for which ω (and only ω) of the nodes in the small code occur 2 This can be viewed in different forms – rank of the code generating matrix punctured at the columns that correspond to the (n − k) nodes that are not selected, as in [8], or as the entropy of the random variables corresponding to the k selected nodes. 7 0.22 MSR 0.2 Space−Sharing Construction in [8] Total repair bandwidth γ 0.18 Construction in [9] space−sharing line 0.16 Construction 1 0.14 Construction 2 0.12 Functional Repair 0.1 0.08 0.06 MBR 0.04 0.02 0.018 0.02 0.022 0.024 0.026 Storage per node 0.028 0.03 0.032 0.034 α Fig. 4. Comparison among the different bounds existing in the literature and the new inner bounds achieved by constructions 1 and 2 in Section IV for the case of (n, k, d) = (61, 55, 59). The overall tradeoff region as defined in Theorem 2 is achieved by the convex hull of the two new constructions. in K, the average amount of information Mk in K is given by Mk = ω max X Hω Q ω , (17) ω=ωmin    n̂ n − n̂ ω max X ω k−ω   min{ω, k̂} = α̂, n ω=ωmin k (18) where ωmin = max{1, n̂ − (n − k)} and ωmax = max{k, n̂} give the limits determined by the possible overlap sizes of the small code and K. Observe that the repair process is not disrupted during this process and hence the average repair bandwidth remains the same as in (8). C. Inner Bound A natural question arises whether one of the constructions for d > k is better than the other. We show using an example that this is not necessarily true. Example: Consider3 an (n, k, d) = (61, 55, 59) DSS. Fig. 4 plots the inner bounds achieved for the storage/repairbandwidth tradeoff using different code constructions. We see that for this example, the inner bound achieved by using the codes in [9] strictly encompasses the bound achieved by those in [8]. However, Construction 1 and Construction 2, do not exhibit such a relationship. Whereas Construction 1 outperforms Construction 2 at points closer to the MSR point, Construction 2 generates a tighter inner bound near the MBR point. It follows therefore that an overall inner bound must incorporate points from both constructions, as formalized in the next theorem. 3 We considered this example for easy comparison with the adjacent case of (61, 55, 60) considered in [8]. 8 Theorem 2: There exist (n, k, d) exact-repair regenerating codes that store a file of size M = 1 and achieve any point in the region con(R), the convex hull of R, where R = R1 ∪ R2 ∪ Rmbr ∪ Rmsr , ( ) k [ α(k̂1 ) γ(k̂1 ) (ᾱ, γ̄) : ᾱ ≥ R1 = , , γ̄ ≥ M(k̂1 ) M(k̂1 ) k̂1 =1 ) ( d [ γ(k̂2 ) α(k̂2 ) , γ̄ ≥ , (ᾱ, γ̄) : ᾱ ≥ R2 = Mk (k̂2 ) Mk (k̂2 ) k̂2 =1 n αmbr γmbr o Rmbr = (ᾱ, γ̄) : ᾱ ≥ , and , γ̄ ≥ M Mo n αmsr γmsr Rmsr = (ᾱ, γ̄) : ᾱ ≥ , , γ̄ ≥ M M (19) (20) (21) (22) (23) where R1 is the tradeoff region corresponding to Construction 1, and α(k̂1 ), γ(k̂1 ) and M(k̂1 ) in (20) are given by (6), (16), and (9), respectively, evaluated at k̂ = k̂1 and n̂ = n + k̂1 − k. Similarly, R2 corresponds to Construction 2, and α(k̂2 ), γ(k̂2 ) and Mk (k̂2 ) in (21) are given by (6), (8) and (18), respectively, evaluated at k̂ = k̂2 and n̂ = n + k̂1 − d. The regions corresponding to the MBR and the MSR points are evaluated at the given k and d. V. C ONCLUSION Determining the achievable region in the tradeoff between storage and exact repair bandwidth for distributed storage systems is an important problem that is still open in general, with this region fully characterized only for the (4, 3, 3) case [6]. This paper makes a contribution towards solving this problem by proposing new constructions of regenerating codes with exact repair that achieve new points in the tradeoff region leading to improved inner bounds. Our code constructions are a generalization of the codes in [9] and [8]. The main idea is to construct new codes using already known code constructions for systems with smaller parameters. Our constructions have two key ingredients: (i) Use as a building block a minimum storage code (MSR) with optimal repair bandwidth for all possible repair degrees, and (ii) Allow a heterogeneous repair bandwidth that depends on the nodes contacted for repair. The code is then “homogenized” by appending together all of its n! permutations as described in [10]. This method leads to new achievable points on the tradeoff and result in tighter inner bounds for all systems having more than one parity node (n > k + 1). R EFERENCES [1] A. G. Dimakis, P. G. Godfrey, Y. Wu, M. J. Wainwright, and K. Ramchandran, “Network Coding for Distributed Storage Systems,” in IEEE Transactions on Information Theory, vol. 56, Sep. 2010, pp. 4539–4551. [2] K. V. Rashmi, N. B. Shah, and P. V. Kumar, “Optimal Exact-Regenerating Codes for Distributed Storage at the MSR and MBR Points via a Product-Matrix Construction,” in IEEE Transactions on Information Theory, vol. 57, Aug. 2011, pp. 5227–5239. [3] D. Papailiopoulos, A. Dimakis, and V. Cadambe, “Repair Optimal Erasure Codes through Hadamard Designs,” in Proceedings of the 49th Annual Allerton Conference on Communication, Control, and Computing, Sep. 2011, pp. 1382–1389. [4] V. Cadambe, S. Jafar, H. Maleki, K. Ramchandran, and C. Suh, “Asymptotic Interference Alignment for Optimal Repair of MDS codes in Distributed Storage,” in IEEE Transactions on Information Theory, vol. 59, May 2013, pp. 2974–2987. [5] Z. Wang, I. Tamo, and J. Bruck, “On Codes for Optimal Rebuilding Access,” in Proceedings of the 49th Annual Allerton Conference on Communication, Control, and Computing, 2011, pp. 1374–1381. [6] C. Tian, “Rate Region of the (4, 3, 3) Exact-Repair Regenerating Codes,” in Proceedings of IEEE International Symposium on Information Theory (ISIT), Jul. 2013, pp. 1426–1430. [7] C. Tian, V. Aggarwal, and V. A. Vaishampayan, “Exact-Repair Regenerating Codes via Layered Erasure Correction and Block Designs,” in Proceedings of IEEE International Symposium on Information Theory (ISIT), 2013, pp. 1431–1435. [8] B. Sasidharan and P. V. Kumar, “High-Rate Regenerating Codes Through Layering,” in arxiv.org, Mar. 2013. [Online]. Available: http://arxiv.org/abs/1301.6157 [9] T. Ernvall, “Exact-regenerating codes between mbr and msr points,” in arXiv:1304.5357v1, 2013. [10] T. Ernvall, S. El Rouayheb, C. Hollanti, and H. V. Poor, “Capacity and Security of Heterogeneous Distributed Storage Systems,” in Proceedings of IEEE International Symposium on Information Theory (ISIT), 2013, pp. 1247–1251. [11] C. Suh and K. Ramchandran, “Exact-Repair MDS Codes for Distributed Storage using Interference Alignment,” in Proceedings of IEEE International Symposium on Information Theory (ISIT), Jun. 2010, pp. 161–165. [12] V. Aggarwal, C. Tian, V. A. Vaishampayan, and Y. R. Chen, “Distributed Data Storage Systems with Opportunistic Repair,” in arxiv.org, Nov. 2013. [Online]. Available: http://arxiv.org/abs/1311.4096