research-article

Extensible distributed coordination

Authors:

Tobias Distler,

Christopher Bahn,

Alysson Bessani,

Frank Fischer, and

Flavio JunqueiraAuthors Info & Claims

EuroSys '15: Proceedings of the Tenth European Conference on Computer Systems

April 2015

Article No.: 10, Pages 1 - 16

https://doi.org/10.1145/2741948.2741954

Published: 17 April 2015 Publication History

Abstract

Most services inside a data center are distributed systems requiring coordination and synchronization in the form of primitives like distributed locks and message queues. We argue that extensibility is a crucial feature of the coordination infrastructures used in these systems. Without the ability to extend the functionality of coordination services, applications might end up using sub-optimal coordination algorithms, possibly leading to low performance. Adding extensibility, however, requires mechanisms that constrain extensions to be able to make reasonable security and performance guarantees. We propose a scheme that enables extensions to be introduced and removed dynamically in a secure way. To avoid performance overheads due to poorly designed extensions, it constrains the access of extensions to resources. Evaluation results for extensible versions of ZooKeeper and DepSpace show that it is possible to increase the throughput of a distributed queue by more than an order of magnitude (17x for ZooKeeper, 24x for DepSpace) while keeping the underlying coordination kernel small.

Supplementary Material

MP4 File (a10-sidebyside.mp4)

Download
893.18 MB

References

[1]

Apache ActiveMQ. http://activemq.apache.org/.

[2]

Apache Curator. http://curator.apache.org/.

[3]

CoreOS etcd. https://github.com/coreos/etcd/.

[4]

JZab. https://github.com/zk1931/jzab/.

[5]

LogCabin. https://github.com/logcabin/.

[6]

M. K. Aguilera, A. Merchant, M. Shah, A. Veitch, and C. Karamanolis. Sinfonia: A new paradigm for building scalable distributed systems. In Proceedings of the 21st Symposium on Operating Systems Principles (SOSP '07), pages 159--174, 2007.

Digital Library

[7]

D. E. Bakken and R. D. Schlichting. Supporting fault-tolerant parallel programming in Linda. IEEE Transactions on Parallel and Distributed Systems, 6(3): 287--302, 1995.

Digital Library

[8]

M. Balakrishnan, D. Malkhi, J. D. Davis, V. Prabhakaran, M. Wei, and T. Wobber. CORFU: A distributed shared log. ACM Transactions on Computer Systems, 31(4), 2013.

Digital Library

[9]

R. Baldoni, C. Marchetti, and L. Verde. CORBA request portable interceptors: Analysis and applications. Concurrency and Computation: Practice and Experience, 15(6): 551--579, 2003.

[10]

P. Berde, M. Gerola, J. Hart, Y. Higuchi, M. Kobayashi, T. Koide, B. Lantz, B. O'Connor, P. Radoslavov, W. Snow, and G. Parulkar. ONOS: Towards an open, distributed SDN OS. In Proceedings of the 3rd Workshop on Hot Topics in Software Defined Networking (HotSDN '14), pages 1--6, 2014.

Digital Library

[11]

B. N. Bershad, S. Savage, P. Pardyak, E. G. Sirer, M. E. Fiuczynski, D. Becker, C. Chambers, and S. Eggers. Extensibility, safety and performance in the SPIN operating system. In Proceedings of the 15th Symposium on Operating Systems Principles (SOSP '95), pages 267--283, 1995.

Digital Library

[12]

A. Bessani, R. Mendes, T. Oliveira, N. Neves, M. Correia, M. Pasin, and P. Veríssimo. SCFS: A shared cloud-backed file system. In Proceedings of the 2014 USENIX Annual Technical Conference (ATC '14), pages 169--180, 2014.

Digital Library

[13]

A. Bessani, J. Sousa, and E. A. P. Alchieri. State machine replication for the masses with BFT-SMaRt. In Proceedings of the 44th International Conference on Dependable Systems and Networks (DSN '14), pages 355--362, 2014.

Digital Library

[14]

A. N. Bessani, E. P. Alchieri, M. Correia, and J. Fraga. DepSpace: A Byzantine fault-tolerant coordination service. In Proceedings of the 3rd European Conference on Computer Systems (EuroSys '08), pages 163--176, 2008.

Digital Library

[15]

F. Botelho, F. Ramos, D. Kreutz, and A. Bessani. On the feasibility of a consistent and fault-tolerant data store for SDNs. In Proceedings of the 2nd European Workshop on Software Defined Networks (EWSDN '13), pages 38--43, 2013.

Digital Library

[16]

N. Budhiraja, K. Marzullo, F. B. Schneider, and S. Toueg. The primary-backup approach. In Distributed Systems (2nd Edition), pages 199--216. Addison-Wesley, 1993.

Digital Library

[17]

M. Burrows. The Chubby lock service for loosely-coupled distributed systems. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI '06), pages 335--350, 2006.

Digital Library

[18]

P. Butcher, A. C. Wood, and M. Atkins. Global synchronisation in Linda. Concurrency -- Practice and Experience, 6(6): 505--516, 1994.

[19]

G. Cabri, L. Leonardi, and F. Zambonelli. MARS: A programmable coordination architecture for mobile agents. IEEE Internet Computing, 4(4): 26--35, 2000.

Digital Library

[20]

T. D. Chandra, R. Griesemer, and J. Redstone. Paxos made live: An engineering perspective. In Proceedings of the 26th Symposium on Principles of Distributed Computing (PODC '07), pages 398--407, 2007.

Digital Library

[21]

A. Clement, M. Kapritsos, S. Lee, Y. Wang, L. Alvisi, M. Dahlin, and T. Riche. UpRight cluster services. In Proceedings of the 22nd Symposium on Operating Systems Principles (SOSP '09), pages 277--290, 2009.

Digital Library

[22]

E. Denti, A. Natali, A. Omicini, and M. Venuti. An extensible frame work for the development of coordinated applications. In Proceedings of the 1st International Conference on Coordination Languages and Models (COORDINATION '96), pages 305--320, 1996.

Digital Library

[23]

T. Distler and R. Kapitza. Increasing performance in Byzantine fault-tolerant systems with on-demand replica consistency. In Proceedings of the 6th European Conference on Computer Systems (EuroSys '11), pages 91--105, 2011.

Digital Library

[24]

D. R. Engler, M. F. Kaashoek, and J. O'Toole Jr. Exokernel: An operating system architecture for application-level resource management. In Proceedings of the 15th Symposium on Operating Systems Principles (SOSP '95), pages 251--266, 1995.

Digital Library

[25]

A. D. Ferguson, A. Guha, C. Liang, R. Fonseca, and S. Krishnamurthi. Hierarchical policies for software defined networks. In Proceedings of the 1st Workshop on Hot Topics in Software Defined Networking (HotSDN '12), pages 37--42, 2012.

Digital Library

[26]

D. Gelernter. Generative communication in Linda. ACM Transactions on Programming Languages and Systems, 7(1): 80--112, 1985.

Digital Library

[27]

K. Glerum, K. Kinshumann, S. Greenberg, G. Aul, V. Orgovan, G. Nichols, D. Grant, G. Loihle, and G. Hunt. Debugging in the (very) large: Ten years of implementation and experience. In Proceedings of the 22nd Symposium on Operating Systems Principles (SOSP '09), pages 103--116, 2009.

Digital Library

[28]

M. Golm, M. Felser, C. Wawersich, and J. Kleinöder. The JX operating system. In Proceedings of the 2002 USENIX Annual Technical Conference (ATC '02), pages 45--58, 2002.

Digital Library

[29]

M. P. Herlihy. Wait-free synchronization. ACM Transactions on Programing Languages and Systems, 13(1): 124--149, 1991.

Digital Library

[30]

M. P. Herlihy and J. M. Wing. Linearizability: A correctness condition for concurrent objects. ACM Transactions on Programming Languages and Systems, 12(3): 463--492, 1990.

Digital Library

[31]

P. Hunt, M. Konar, F. P. Junqueira, and B. Reed. ZooKeeper: Wait-free coordination for Internet-scale systems. In Proceedings of the 2010 USENIX Annual Technical Conference (ATC '10), pages 145--158, 2010.

Digital Library

[32]

J. Hunter and W. Crawford. Java servlet programming. O'Reilly Media, 2001.

Digital Library

[33]

M. Isard. Autopilot: Automatic data center management. SIGOPS Operating Systems Review, 41(2): 60--67, 2007.

Digital Library

[34]

B. Johanson and A. Fox. The event heap: A coordination infrastructure for interactive workspaces. In Proceedings of the 4th Workshop on Mobile Computing Systems and Applications (WMCSA '02), pages 83--93, 2002.

Digital Library

[35]

F. P. Junqueira, B. C. Reed, and M. Serafini. Zab: High-performance broadcast for primary-backup systems. In Proceedings of the 41st International Conference on Dependable Systems and Networks (DSN '11), pages 245--256, 2011.

Digital Library

[36]

B. Kalantari and A. Schiper. Addressing the ZooKeeper synchronization inefficiency. In Proceedings of the 14th International Conference on Distributed Computing and Networking (ICDCN '13), pages 434--438, 2013.

[37]

N. M. Karnik and A. R. Tripathi. Design issues in mobile-agent programming systems. IEEE Concurrency, 6(3): 52--61, 1998.

Digital Library

[38]

T. Koponen, M. Casado, N. Gude, J. Stribling, L. Poutievski, M. Zhu, R. Ramanathan, Y. Iwata, H. Inoue, T. Hama, and S. Shenker. Onix: A distributed control platform for large-scale production networks. In Proceedings of the 9th Symposium on Operating Systems Design and Implementation (OSDI '10), pages 351--364, 2010.

Digital Library

[39]

L. Lamport. The part-time parliament. ACM Transactions on Computer Systems, 16(2): 133--169, 1998.

Digital Library

[40]

J. B. Leners, H. Wu, W.-L. Hung, M. K. Aguilera, and M. Walfish. Detecting failures in distributed systems with the Falcon spy network. In Proceedings of the 23rd Symposium on Operating Systems Principles (SOSP '11), pages 279--294, 2011.

Digital Library

[41]

D. Levin, A. Wundsam, B. Heller, N. Handigol, and A. Feldmann. Logically centralized? State distribution trade-offs in software defined networks. In Proceedings of the 1st Workshop on Hot Topics in Software Defined Networking (HotSDN '12), pages 1--6, 2012.

Digital Library

[42]

V. Linnemann, K. Küspert, P. Dadam, P. Pistor, R. Erbe, A. Kemper, N. Südkamp, G. Walch, and M. Wallrath. Design and implementation of an extensible database management system supporting user defined data types and functions. In Proceedings of the 14th International Conference on Very Large Data Bases (VLDB '88), pages 294--305, 1988.

Digital Library

[43]

M. Loui and H. Abu-Amara. Memory requirements for agreement among unreliable asynchronous processes. Advances in Computing Research, 4, 1987.

[44]

J. MacCormick, N. Murphy, M. Najork, C. A. Thekkath, and L. Zhou. Boxwood: Abstractions as the foundation for storage infrastructure. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI '04), pages 105--120, 2004.

Digital Library

[45]

A. Omicini and F. Zambonelli. Coordination for Internet application development. Autonomous Agents and Multi-Agent Systems, 2(3): 251--269, 1999.

Digital Library

[46]

D. Ongaro and J. Ousterhout. In search of an understandable consensus algorithm. In Proceedings of the 2014 USENIX Annual Technical Conference (ATC '14), pages 305--320, 2014.

Digital Library

[47]

N. W. Paton and O. Díaz. Active database systems. ACM Computing Surveys, 31(1): 63--103, 1999.

Digital Library

[48]

M. Pease, R. Shostak, and L. Lamport. Reaching agreement in the presence of faults. Journal of the ACM, 27(2): 228--234, 1980.

Digital Library

[49]

D. Peng and F. Dabek. Large-scale incremental processing using distributed transactions and notifications. In Proceedings of the 9th Symposium on Operating Systems Design and Implementation (OSDI '10), pages 251--264, 2010.

Digital Library

[50]

A. I. T. Rowstron. Using mobile code to provide fault tolerance in tuple space based coordination languages. Science of Computer Programming, 46(1): 137--162, 2003.

Digital Library

[51]

A. I. T. Rowstron and A. M. Wood. Solving the Linda multiple rd problem using the copy-collect primitive. Science of Computer Programming, 31(2--3): 335--358, 1998.

Digital Library

[52]

F. B. Schneider. Implementing fault-tolerant services using the state machine approach: A tutorial. ACM Computer Survey, 22(4): 299--319, 1990.

Digital Library

[53]

E. J. Segall. Resilient distributed objects: Basic results and application to shared tuple spaces. In Proceedings of the 7th Symposium on Parallel and Distributeed Processing (SPDP '95), pages 320--327, 1995.

Digital Library

[54]

M. Seltzer, Y. Endo, C. Small, and K. Smith. Dealing with disaster: Surviving misbehaved kernel extensions. In Proceedings of the 2nd Symposium on Operating Systems Design and Implementation (OSDI '96), 1996.

Digital Library

[55]

A. Shakimov, H. Lim, R. Caceres, L. Cox, K. Li, D. Liu, and A. Varshavsky. Vis-a-Vis: Privacy-preserving online social networking via virtual individual servers. In Proceedings of the 3rd International Conference on Communication Systems and Networks (COMSNETS '11), pages 1--10, 2011.

[56]

J. W. Stamos and D. K. Gifford. Remote evaluation. ACM Transactions on Programming Languages and Systems, 12(4): 537--564, 1990.

Digital Library

[57]

M. Stonebraker, J. Anton, and E. Hanson. Extending a database system with procedures. ACM Transactions on Database Systems, 12(3): 350--376, 1987.

Digital Library

[58]

W. Vogels. Life is not a state-machine: The long road from research to production. In Proceedings of the 25th Symposium on Principles of Distributed Computing (PODC '06), page 112, 2006.

Digital Library

Cited By

Barcelona-Pons DSutra PSánchez-Artigas MParís GGarcía-López P(2022)Stateful Serverless Computing with CrucialACM Transactions on Software Engineering and Methodology10.1145/349038631:3(1-38)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.1145/3490386
Eischer MStraßner BDistler TFekete AKleppmann M(2020)Low-latency geo-replicated state machines with guaranteed writesProceedings of the 7th Workshop on Principles and Practice of Consistency for Distributed Data10.1145/3380787.3393686(1-9)Online publication date: 27-Apr-2020
https://dl.acm.org/doi/10.1145/3380787.3393686
da Silva JAlchieri EBordim JCosta L(2020)A Secure and Distributed Control Plane for Software Defined NetworksAdvanced Information Networking and Applications10.1007/978-3-030-44041-1_86(994-1006)Online publication date: 28-Mar-2020
https://doi.org/10.1007/978-3-030-44041-1_86
Show More Cited By

Index Terms

Extensible distributed coordination

Recommendations

Enhancing coordination in cloud infrastructures with an extendable coordination service
SDMCMM '12: Proceedings of the Workshop on Secure and Dependable Middleware for Cloud Monitoring and Management

With application processes being distributed across a large number of nodes, coordination is a crucial but inherently difficult task in cloud environments. Coordination middleware systems like Chubby and ZooKeeper approach this problem by providing ...
Read More
Building secure protocols for extensible distributed coordination through secure extensions
Abstract
Mechanisms for distributed coordination are used in the development of many distributed systems and, generally, are implemented on top of coordination infrastructures, such as tuple spaces. Although tuple spaces provide the ...
Read More
A configurable and extensible transport protocol

The ability to configure transport protocols from collections of smaller software modules allows the characteristics of the protocol to be customized for a specific application or network technology. This paper describes a configurable transport ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

EuroSys '15: Proceedings of the Tenth European Conference on Computer Systems

April 2015

503 pages

ISBN:9781450332385

DOI:10.1145/2741948

General Chair:
Laurent Réveillère
LaBRI, University of Bordeaux, France
,
Program Chairs:
Tim Harris
Oracle Labs, UK
,
Maurice Herlihy
Brown University

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Commission

Conference

EuroSys '15

Sponsor:

SIGOPS

EuroSys '15: Tenth EuroSys Conference 2015

April 21 - 24, 2015

Bordeaux, France

Acceptance Rates

Overall Acceptance Rate 241 of 1,308 submissions, 18%

Upcoming Conference

EuroSys '25

Sponsor:
sigops

Twentieth European Conference on Computer Systems

March 30 - April 3, 2025

Rotterdam , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
817
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)1

Other Metrics

View Author Metrics

Citations

Cited By

Barcelona-Pons DSutra PSánchez-Artigas MParís GGarcía-López P(2022)Stateful Serverless Computing with CrucialACM Transactions on Software Engineering and Methodology10.1145/349038631:3(1-38)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.1145/3490386
Eischer MStraßner BDistler TFekete AKleppmann M(2020)Low-latency geo-replicated state machines with guaranteed writesProceedings of the 7th Workshop on Principles and Practice of Consistency for Distributed Data10.1145/3380787.3393686(1-9)Online publication date: 27-Apr-2020
https://dl.acm.org/doi/10.1145/3380787.3393686
da Silva JAlchieri EBordim JCosta L(2020)A Secure and Distributed Control Plane for Software Defined NetworksAdvanced Information Networking and Applications10.1007/978-3-030-44041-1_86(994-1006)Online publication date: 28-Mar-2020
https://doi.org/10.1007/978-3-030-44041-1_86
Barcelona-Pons DSánchez-Artigas MParís GSutra PGarcía-López P(2019)On the FaaS TrackProceedings of the 20th International Middleware Conference10.1145/3361525.3361535(41-54)Online publication date: 9-Dec-2019
https://dl.acm.org/doi/10.1145/3361525.3361535
Balasubramanian BSchlichting RZave PNewport CKeidar I(2018)Brief AnnouncementProceedings of the 2018 ACM Symposium on Principles of Distributed Computing10.1145/3212734.3212782(281-284)Online publication date: 23-Jul-2018
https://dl.acm.org/doi/10.1145/3212734.3212782
Floriano EAlchieri EAranha DSolis P(2018)Security and Privacy in Extensible Distributed Coordination2018 IEEE Symposium on Computers and Communications (ISCC)10.1109/ISCC.2018.8538487(00034-00037)Online publication date: Jun-2018
https://doi.org/10.1109/ISCC.2018.8538487
Sutra PRiviere ECotes CArtigas MLopez PBernard EBurns WZamarreno G(2017)CRESON: Callable and Replicated Shared Objects over NoSQL2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS.2017.239(115-128)Online publication date: Jun-2017
https://doi.org/10.1109/ICDCS.2017.239
Buravlev VDe Nicola RMezzina C(2017)Evaluating the efficiency of Linda implementationsConcurrency and Computation: Practice and Experience10.1002/cpe.438130:8Online publication date: 20-Dec-2017
https://doi.org/10.1002/cpe.4381
Botelho FRibeiro TFerreira PRamos FBessani A(2016)Design and Implementation of a Consistent Data Store for a Distributed SDN Control Plane2016 12th European Dependable Computing Conference (EDCC)10.1109/EDCC.2016.12(169-180)Online publication date: Sep-2016
https://doi.org/10.1109/EDCC.2016.12
Alsmadi IZarour M(2016)Empirical Evidences in Software-Defined Network Security: A Systematic Literature ReviewInformation Fusion for Cyber-Security Analytics10.1007/978-3-319-44257-0_11(253-295)Online publication date: 22-Oct-2016
https://doi.org/10.1007/978-3-319-44257-0_11

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents