article

Minerva: An automated resource provisioning tool for large-scale storage systems

Authors:

Guillermo A. Alvarez,

Elizabeth Borowsky,

Theodore H. Romer,

Ralph Becker-Szendy,

Richard Golding,

Mirjana Spasojevic,

Alistair Veitch, and

John WilkesAuthors Info & Claims

ACM Transactions on Computer Systems (TOCS), Volume 19, Issue 4

Pages 483 - 518

https://doi.org/10.1145/502912.502915

Published: 01 November 2001 Publication History

Abstract

Enterprise-scale storage systems, which can contain hundreds of host computers and storage devices and up to tens of thousands of disks and logical volumes, are difficult to design. The volume of choices that need to be made is massive, and many choices have unforeseen interactions. Storage system design is tedious and complicated to do by hand, usually leading to solutions that are grossly over-provisioned, substantially under-performing or, in the worst case, both.To solve the configuration nightmare, we present minerva: a suite of tools for designing storage systems automatically. Minerva uses declarative specifications of application requirements and device capabilities; constraint-based formulations of the various sub-problems; and optimization techniques to explore the search space of possible solutions.This paper also explores and evaluates the design decisions that went into Minerva, using specialized micro- and macro-benchmarks. We show that Minerva can successfully handle a workload with substantial complexity (a decision-support database benchmark). Minerva created a 16-disk design in only a few minutes that achieved the same performance as a 30-disk system manually designed by human experts. Of equal importance, Minerva was able to predict the resulting system's performance before it was built.

References

[1]

3COM CORP. 1996. Gigabit Ethernet comes of age. 3Com Corporation. Technology white paper.

[2]

ANSI. 1996. Fibre Channel Arbitrated Loop. ANSI. Standard X3.272-1996.

[3]

AWERBUCH, B., BARTAL,Y.,AND FIAT, A. 1993. Competitive distributed file allocation. In Proceedings of the 25th ACM Symposium on Theory of Computing (STOC) (May 1993), pp. 164-73.

[4]

BOROWSKY, E., GOLDING, R., MERCHANT, A., SCHREIER, L., SHRIVER, E., SPASOJEVIC, M., AND WILKES,J. 1997. Using attribute-managed storage to achieve QoS. In Proceedings of the 5th Intl. Workshop on Quality of Service (June 1997).

[5]

CHU,P.AND BEASLEY, J. 1997. A genetic algorithm for the generalized assignment problem. Computers and Operations Research 24, 1, 17-23.

[6]

COFFMAN, E., GAREY, M., AND JOHNSON, D. 1984. Approximation algorithms for bin-packing: An updated survey. In G. AUSIELLO,M.LUCERTINI, AND P. SERAFINI Eds., Algorithm Design for Computer System Design, pp. 49-106. Springer-Verlag.

[7]

DOWDY,L.W.AND FOSTER, D. V. 1982. Comparative models of the file assignment problem. ACM Computing Surveys 14, 2 (June), 287-313.

[8]

DREXL, A. 1988. A simulated annealing approach to the multiconstraint zero-one knapsack problem. Computing 40, 1, 1-8.

[9]

GAREY,M.AND JOHNSON, D. 1979. Computers and intractability: a guide to the theory of NP-completeness. W.H. Freeman, San Francisco.

[10]

GELB, J. P. 1989. System managed storage. IBM Systems Journal 28, 1, 77-103.

[11]

HEWLETT-PACKARD COMPANY. 1998. Model 30/FC High Availability Disk Array-User's Guide. Hewlett-Packard Company. Pub. No. A3661-90001.

[12]

KUENNING, G. H. 1995. Kitrace-precise interactive measurement of operating systems kernels. Software-Practice and Experience 25, 1 (January), 1-21.

[13]

MENON,J.AND MATTSON, D. 1992. Performance of disk arrays in transaction processing environments. In Proceedings of the 12th ICDCS (June 1992), pp. 302-309.

[14]

MERCHANT,A.AND ALVAREZ, G. A. 2001. Disk array models in Minerva. Technical Report HPL-2001- 118 (April), Hewlett-Packard Laboratories. http://www.hpl.hp.com/techreports.

[15]

MERCHANT,A.AND YU, P. S. 1996. Analytic modeling of clustered RAID with mapping based on nearly random permutation. IEEE Trans. Computers 45, 3, 367-373.

[16]

PATTERSON, D., GIBSON,G.,AND KATZ, R. 1988. A case for redundant arrays of inexpensive disks (RAID). In Proc. ACM SIGMOD (1988), pp. 109-116.

[17]

PATTIPATI,K.R.AND WOLF, J. L. 1990. A file assignment problem model for extended local area network environments. In Proceedings of the 10th ICDCS (May 1990), pp. 554-61.

[18]

RUEMMLER,C.AND WILKES, J. 1993. Unix disk access patterns. In Proceedings of the Winter USENIX (Jan. 1993), pp. 405-420.

[19]

THOMASIAN,A.AND MENON, J. 1994. Performance analysis of RAID5 disk arrays with a vacationing server model for rebuild mode operation. In Proceedings of the 10th Intl. Conf. Data Eng. (Feb. 1994), pp. 111-119.

[20]

TOYODA, Y. 1975. A simplified algorithm for obtaining approximate solutions to zero-one programming problems. Management Science 21, 12 (Aug.), 1417-27.

[21]

TRANSACTION PROCESSING PERFORMANCE COUNCIL. 1992. TPC benchmark C, standard specification, revision 1.0. Transaction Processing Performance Council.

[22]

TRANSACTION PROCESSING PERFORMANCE COUNCIL. 1996. TPC benchmark D, standard specification, revision 1.2. Transaction Processing Performance Council.

[23]

TRICK, M. 1992. A linear relaxation heuristic for the generalized assignment problem. Naval Research Logistics 39, 137-51.

[24]

WOLF, J. 1989. The placement optimization program: a practical solution to the disk file assignment problem. In Proc. SIGMETRICS (May 1989), pp. 1-10.

Cited By

Bang TDouglas CCrooks NHellerstein J(2024)SkyPIE: A Fast & Accurate Oracle for Object PlacementProceedings of the ACM on Management of Data10.1145/36393102:1(1-27)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3639310
Park HGanger GAmvrosiadis GGilad YKostic DMoatti YBiran O(2023)Mimir: Finding Cost-efficient Storage Configurations in the Public CloudProceedings of the 16th ACM International Conference on Systems and Storage10.1145/3579370.3594776(22-34)Online publication date: 5-Jun-2023
https://dl.acm.org/doi/10.1145/3579370.3594776
(2022)BibliographyStorage Systems10.1016/B978-0-32-390796-5.00023-1(641-693)Online publication date: 2022
https://doi.org/10.1016/B978-0-32-390796-5.00023-1
Show More Cited By

Index Terms

Minerva: An automated resource provisioning tool for large-scale storage systems

Recommendations

Higher reliability redundant disk arrays: Organization, operation, and coding

Parity is a popular form of data protection in redundant arrays of inexpensive/independent disks (RAID). RAID5 dedicates one out of N disks to parity to mask single disk failures, that is, the contents of a block on a failed disk can be reconstructed by ...
Read More
On Variable Scope of Parity Protection in Disk Arrays

In a common form of a RAID 5 architecture, data is organized on a disk array consisting of N + 1 disks into stripes of N data blocks and one parity block (with parity block locations staggered so as to balance the number of parity blocks on each disk). ...
Read More
Zoned-RAID

The RAID (Redundant Array of Inexpensive Disks) system has been widely used in practical storage applications for better performance, cost effectiveness, and reliability. This study proposes a novel variant of RAID named Zoned-RAID (Z-RAID). Z-RAID ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer Systems

ACM Transactions on Computer Systems Volume 19, Issue 4

November 2001

106 pages

ISSN:0734-2071

EISSN:1557-7333

DOI:10.1145/502912

Issue’s Table of Contents

Copyright © 2001 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2001

Published in TOCS Volume 19, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

181
Total Citations
View Citations
1,568
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)3

Other Metrics

View Author Metrics

Citations

Cited By

Bang TDouglas CCrooks NHellerstein J(2024)SkyPIE: A Fast & Accurate Oracle for Object PlacementProceedings of the ACM on Management of Data10.1145/36393102:1(1-27)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3639310
Park HGanger GAmvrosiadis GGilad YKostic DMoatti YBiran O(2023)Mimir: Finding Cost-efficient Storage Configurations in the Public CloudProceedings of the 16th ACM International Conference on Systems and Storage10.1145/3579370.3594776(22-34)Online publication date: 5-Jun-2023
https://dl.acm.org/doi/10.1145/3579370.3594776
(2022)BibliographyStorage Systems10.1016/B978-0-32-390796-5.00023-1(641-693)Online publication date: 2022
https://doi.org/10.1016/B978-0-32-390796-5.00023-1
Thomasian A(2022)Heterogeneous Disk Arrays - HDAsStorage Systems10.1016/B978-0-32-390796-5.00019-X(565-591)Online publication date: 2022
https://doi.org/10.1016/B978-0-32-390796-5.00019-X
Behzad BByna SPrabhat Snir M(2019)Optimizing I/O Performance of HPC Applications with AutotuningACM Transactions on Parallel Computing10.1145/33092055:4(1-27)Online publication date: 8-Mar-2019
https://dl.acm.org/doi/10.1145/3309205
Karniavoura FMagoutis K(2019)Decision-Making Approaches for Performance QoS in Distributed Storage Systems: A SurveyIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2019.2893940(1-1)Online publication date: 2019
https://doi.org/10.1109/TPDS.2019.2893940
Leng JYan DLiu QZhang HZhao GWei LZhang DYu AChen X(2019)Digital twin-driven joint optimisation of packing and storage assignment in large-scale automated high-rise warehouse product-service systemInternational Journal of Computer Integrated Manufacturing10.1080/0951192X.2019.166703234:7-8(783-800)Online publication date: 29-Sep-2019
https://doi.org/10.1080/0951192X.2019.1667032
Gracia-Tinedo RSampé JParís GSánchez-Artigas MGarcía-López PMoatti Y(2019)Software-defined object storage in multi-tenant environmentsFuture Generation Computer Systems10.1016/j.future.2019.03.020Online publication date: Mar-2019
https://doi.org/10.1016/j.future.2019.03.020
Klimovic AWang YStuedi PTrivedi APfefferle JKozyrakis CArpaci-Dusseau AVoelker G(2018)PocketProceedings of the 13th USENIX conference on Operating Systems Design and Implementation10.5555/3291168.3291200(427-444)Online publication date: 8-Oct-2018
https://dl.acm.org/doi/10.5555/3291168.3291200
Klimovic ALitz HKozyrakis CGunawi HReed B(2018)SelectaProceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference10.5555/3277355.3277429(759-773)Online publication date: 11-Jul-2018
https://dl.acm.org/doi/10.5555/3277355.3277429
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents