research-article

Synthesizing Cluster Management Code for Distributed Systems

Authors:

Nina Narodytska,

Brian OkiAuthors Info & Claims

HotOS '19: Proceedings of the Workshop on Hot Topics in Operating Systems

Pages 45 - 50

https://doi.org/10.1145/3317550.3321444

Published: 13 May 2019 Publication History

Abstract

Management planes for data-center systems are complicated to develop, test, maintain, and evolve. They routinely grapple with hard combinatorial optimization problems like load balancing, placement, scheduling, rolling upgrades and configuration management. To tackle these problems, developers are left with two bad choices: (i) develop ad-hoc mechanisms for systems to solve these optimization problems, or (ii) use specialized solvers that require steep engineering effort.

We propose Weave, a tool that enables programmers to specify cluster management policies in a high-level declarative language, and compute policy-compliant configurations automatically and efficiently. Weave allows constraints and policies, the essence of a management plane, to be easily added, removed and modified over time, using a language familiar to developers (SQL). In this paper, we discuss our approach of management plane synthesis, its benefits, and present preliminary results from implementing a Kubernetes scheduler and a CorfuDB management plane using Weave.

References

[1]

Kubernetes. http://github.com/kubernetes/kubernetes.

[2]

Alvaro, P., Condie, T., Conway, N., Elmeleegy, K., Hellerstein, J. M., and Sears, R. Boom analytics: exploring data-centric, declarative programming for the cloud. In Proceedings of the 5th European conference on Computer systems (2010), ACM, pp. 223--236.

Digital Library

[3]

Balakrishnan, M., Malkhi, D., Prabhakaran, V., Wobbler, T., Wei, M., and Davis, J. D. Corfu: A shared log design for flash clusters. In Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12) (San Jose, CA, 2012), USENIX, pp. 1--14.

Digital Library

[4]

Cadoli, M., and Mancini, T. Combining relational algebra, sql, constraint modelling, and local search. Theory and Practice of Logic Programming 7, 1-2 (2007), 37--65.

Digital Library

[5]

Candea, G., and Fox, A. Crash-only software. In Proceedings of the 9th Conference on Hot Topics in Operating Systems - Volume 9 (Berkeley, CA, USA, 2003), HOTOS'03, USENIX Association, pp. 12--12.

Digital Library

[6]

Chou, A., Yang, J., Chelf, B., Hallem, S., and Engler, D. An empirical study of operating systems errors. In ACM Symposium on Operating systems principles (SOSP) (New York, NY, USA, 2001), SOSP '01, ACM, pp. 73--88.

Digital Library

[7]

Delimitrou, C. Improving resource efficiency in cloud computing. PhD thesis, Stanford University, 2015.

[8]

Guenter, B., Jain, N., and Williams, C. Managing cost, performance, and reliability tradeoffs for energy-aware server provisioning. In 2011 Proceedings IEEE INFOCOM (April 2011), pp. 1332--1340.

[9]

Gunawi, H. S., Do, T., Joshi, P., Alvaro, P., Hellerstein, J. M., Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H., Sen, K., and Borthakur, D. Fate and destini: A framework for cloud recovery testing. In Proceedings of USENIX Symposium on Networked Systems Design and Implementation (NSDI) (2011), p. 239.

Digital Library

[10]

Guo, Z., McDirmid, S., Yang, M., Zhuang, L., Zhang, P., Luo, Y., Bergan, T., Musuvathi, M., Zhang, Z., and Zhou, L. Failure recovery: When the cure is worse than the disease. In Presented as part of the 14th Workshop on Hot Topics in Operating Systems (Berkeley, CA, 2013), USENIX.

Digital Library

[11]

Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., and Goldberg, A. Quincy: fair scheduling for distributed computing clusters. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles (2009), ACM, pp. 261---276.

Digital Library

[12]

Koster, A. M., Kutschka, M., and Raack, C. Towards robust network design using integer linear programming techniques. In Next Generation Internet (NGI), 2010 6th EURONF Conference on (2010), IEEE, pp. 1--8.

[13]

Kubernetes. Pod priorities and preemption. https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/, 2018.

[14]

Kubernetes mailing list. Let's remove ServiceAffinity. https://groups.google.com/forum/#!topic/kubernetes-sig-scheduling/ewz4TYJgL0M, 2018.

[15]

Narain, S., et al. Network configuration management via model finding. In LISA (2005), vol. 5, pp. 15--15.

Digital Library

[16]

Tumanov, A., Zhu, T., Park, J. W., Kozuch, M. A., Harchol-Balter, M., and Ganger, G. R. Tetrisched: Global rescheduling with adaptive plan-ahead in dynamic heterogeneous clusters. In Proceedings of the Eleventh European Conference on Computer Systems (New York, NY, USA, 2016), EuroSys '16, ACM, pp. 35:1--35:16.

Digital Library

Cited By

Schult JSchwyn DGiardino MCock DAchermann RRoscoe T(2021)Declarative Power SequencingACM Transactions on Embedded Computing Systems10.1145/347703920:5s(1-21)Online publication date: 31-Oct-2021
https://dl.acm.org/doi/10.1145/3477039
Suresh LLoff JKalim FJyothi SNarodytska NRyzhyk LGamage SOki BJain PGasch MLu SHowell J(2020)Building scalable and flexible cluster managers using declarative programmingProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488813(827-844)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488813
Narayanan IKumar ASivasubramaniam A(2020)Pocolo: Power Optimized Colocation in Power Constrained Environments2020 IEEE International Symposium on Workload Characterization (IISWC)10.1109/IISWC50251.2020.00010(1-12)Online publication date: Oct-2020
https://doi.org/10.1109/IISWC50251.2020.00010

Index Terms

Synthesizing Cluster Management Code for Distributed Systems

Recommendations

Cardinality constraints and systems of restricted representatives

Cardinality constraints have received considerable attention from the Constraint Programming community as (so-called) global constraints that appear in the formulation of several real-life problems, while also having an interesting combinatorial ...
From a Database Management System to a Policy-Based Network Management System
NCA '11: Proceedings of the 2011 IEEE 10th International Symposium on Network Computing and Applications

This paper proposes a simple and low cost implementation framework for a policy-based network management (PBNM) system. The framework consists of reusing traditional network management system resources, specifying network management policies through the ...
Conflict‐aware optimal scheduling of prioritised code clone refactoring

Duplicated or similar source code, also known as code clones, are possible malicious ‘code smells’ that may need to be removed through refactoring to enhance maintainability. Among many potential refactoring opportunities, the choice and order of a set of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HotOS '19: Proceedings of the Workshop on Hot Topics in Operating Systems

May 2019

227 pages

ISBN:9781450367271

DOI:10.1145/3317550

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

HotOS '19

Sponsor:

SIGOPS

HotOS '19: Workshop on Hot Topics in Operating Systems

May 13 - 15, 2019

Bertinoro, Italy

Upcoming Conference

HOTOS '25

Sponsor:
sigops

Workshop on Hot Topics in Operating Systems

May 14 - 16, 2025

Banff or Lake Louise , AB , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
410
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schult JSchwyn DGiardino MCock DAchermann RRoscoe T(2021)Declarative Power SequencingACM Transactions on Embedded Computing Systems10.1145/347703920:5s(1-21)Online publication date: 31-Oct-2021
https://dl.acm.org/doi/10.1145/3477039
Suresh LLoff JKalim FJyothi SNarodytska NRyzhyk LGamage SOki BJain PGasch MLu SHowell J(2020)Building scalable and flexible cluster managers using declarative programmingProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488813(827-844)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488813
Narayanan IKumar ASivasubramaniam A(2020)Pocolo: Power Optimized Colocation in Power Constrained Environments2020 IEEE International Symposium on Workload Characterization (IISWC)10.1109/IISWC50251.2020.00010(1-12)Online publication date: Oct-2020
https://doi.org/10.1109/IISWC50251.2020.00010

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents