research-article

Open access

Grove: a Separation-Logic Library for Verifying Distributed Systems

Authors:

Upamanyu Sharma,

Joseph Tassarotti,

Frans Kaashoek,

Nickolai ZeldovichAuthors Info & Claims

SOSP '23: Proceedings of the 29th Symposium on Operating Systems Principles

Pages 113 - 129

https://doi.org/10.1145/3600006.3613172

Published: 23 October 2023 Publication History

Abstract

Grove is a concurrent separation logic library for verifying distributed systems. Grove is the first to handle time-based leases, including their interaction with reconfiguration, crash recovery, thread-level concurrency, and unreliable networks. This paper uses Grove to verify several distributed system components written in Go, including vKV, a realistic distributed multi-threaded key-value store. vKV supports reconfiguration, primary/backup replication, and crash recovery, and uses leases to execute read-only requests on any replica. vKV achieves high performance (67--73% of Redis on a single core), scales with more cores and more backup replicas (achieving about 2× the throughput when going from 1 to 3 servers), and can safely execute reads while reconfiguring.

References

[1]

Masoud Saeida Ardekani and Douglas B. Terry. A self-configurable geo-replicated cloud storage system. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Broomfield, CO, October 2014.

[2]

Jason Baker, Chris Bond, James C. Corbett, JJ Furman, Andrey Khorlin, James Larson, Jean-Michel Leon, Yawei Li, Alexander Lloyd, and Vadim Yushprakh. Megastore: Providing scalable, highly available storage for interactive services. In Proceedings of the 5th Conference on Innovative Data Systems Research (CIDR), pages 223--234, Asilomar, CA, January 2011.

[3]

Mike Burrows. The Chubby lock service for loosely-coupled distributed systems. In Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Seattle, WA, November 2006.

Digital Library

[4]

Keren Censor-Hillel, Erez Petrank, and Shahar Timnat. Help! In Proceedings of the 2015 ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing (PODC), pages 241--250, Donostia-San Sebastián, Spain, July 2015.

Digital Library

[5]

Tej Chajed, Joseph Tassarotti, M. Frans Kaashoek, and Nickolai Zeldovich. Verifying concurrent, crash-safe systems with Perennial. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP), pages 243--258, Huntsville, Ontario, Canada, October 2019.

Digital Library

[6]

Tej Chajed, Joseph Tassarotti, Mark Theng, Ralf Jung, M. Frans Kaashoek, and Nickolai Zeldovich. GoJournal: a verified, concurrent, crash-safe journaling system. In Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 423--439, Virtual conference, July 2021.

[7]

Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber. Bigtable: A distributed storage system for structured data. In Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Seattle, WA, November 2006.

Digital Library

[8]

Haogang Chen, Daniel Ziegler, Tej Chajed, Adam Chlipala, M. Frans Kaashoek, and Nickolai Zeldovich. Using Crash Hoare Logic for certifying the FSCQ file system. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP), pages 18--37, Monterey, CA, October 2015.

Digital Library

[9]

Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. Benchmarking cloud serving systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing (SOCC), pages 143--154, Indianapolis, IN, June 2010.

Digital Library

[10]

James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, JJ Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, Wilson Hsieh, Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Dale Woodford, Yasushi Saito, Christopher Taylor, Michal Szymaniak, and Ruth Wang. Spanner: Google's globally-distributed database. In Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Hollywood, CA, October 2012.

[11]

Aleksandar Dragojević, Dushyanth Narayanan, Edmund B. Nightingale, Matthew Renzelmann, Alex Shamis, Anirudh Badam, and Miguel Castro. No compromises: distributed transactions with consistency, availability, and performance. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP), pages 54--70, Monterey, CA, October 2015.

Digital Library

[12]

Mostafa Elhemali, Niall Gallagher, Nicholas Gordon, Joseph Idziorek, Richard Krog, Colin Lazier, Erben Mo, Akhilesh Mritunjai, Somu Perianayagam, Tim Rath, Swami Sivasubramanian, James Christopher Sorenson III, Sroaj Sosothikul, Doug Terry, and Akshat Vig. Amazon DynamoDB: A scalable, predictably performant, and fully managed NoSQL database service. In Proceedings of the 2022 USENIX Annual Technical Conference, Carlsbad, CA, July 2022.

[13]

Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google file system. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP), Bolton Landing, NY, October 2003.

Digital Library

[14]

Léon Gondelman, Simon Oddershede Gregersen, Abel Nieto, Amin Timany, and Lars Birkedal. Distributed causal memory: modular specification and verification in higher-order distributed separation logic. In Proceedings of the 48th ACM Symposium on Principles of Programming Languages (POPL), Virtual conference, January 2021.

Digital Library

[15]

Léon Gondelman, Jonas Kastberg Hinrichsen, Mário Pereira, Amin Timany, and Lars Birkedal. Verifying reliable network components in a distributed separation logic with dependent separation protocols. In Proceedings of the 28th ACM SIGPLAN International Conference on Functional Programming (ICFP), Seattle, WA, September 2023.

Digital Library

[16]

Cary G. Gray and David R. Cheriton. Leases: An efficient fault-tolerant mechanism for distributed file cache consistency. In Proceedings of the 12th ACM Symposium on Operating Systems Principles (SOSP), pages 202--210, Litchfield Park, AZ, December 1989.

[17]

Travis Hance, Yi Zhou, Andrea Lattuada, Reto Achermann, Alex Conway, Ryan Stutsman, Gerd Zellweger, Chris Hawblitzel, Jon Howell, and Bryan Parno. Sharding the state machine: Automated modular reasoning for complex concurrent systems. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Boston, MA, July 2023.

[18]

Chris Hawblitzel, Jon Howell, Manos Kapritsos, Jacob R. Lorch, Bryan Parno, Michael L. Roberts, Srinath Setty, and Brian Zill. IronFleet: Proving practical distributed systems correct. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP), pages 1--17, Monterey, CA, October 2015.

Digital Library

[19]

Wolf Honore, Ji-Yong Shin, Jieung Kim, and Zhong Shao. Adore: Atomic distributed objects with certified reconfiguration. In Proceedings of the 43rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), San Diego, CA, June 2022.

Digital Library

[20]

Bart Jacobs and Frank Piessens. Expressive modular fine-grained concurrency specification. In Proceedings of the 38th ACM Symposium on Principles of Programming Languages (POPL), pages 271--282, Austin, TX, January 2011.

Digital Library

[21]

Cliff B. Jones. The role of auxiliary variables in the formal development of concurrent programs. In A. W. Roscoe, Cliff. B. Jones, and Kenneth R. Wood, editors, Reflections on the Work of C. A. R. Hoare, pages 167--187. Springer, 2010.

[22]

Ralf Jung, David Swasey, Filip Sieczkowski, Kasper Svendsen, Aaron Turon, Lars Birkedal, and Derek Dreyer. Iris: Monoids and invariants as an orthogonal basis for concurrent reasoning. In Proceedings of the 42nd ACM Symposium on Principles of Programming Languages (POPL), Mumbai, India, January 2015.

[23]

Ralf Jung, Robbert Krebbers, Jacques-Henri Jourdan, Ales Bizjak, Lars Birkedal, and Derek Dreyer. Iris from the ground up: a modular foundation for higherorder concurrent separation logic. Journal of Functional Programming, 28:e20, 2018.

[24]

Apache Kafka. https://cwiki.apache.org/confluence/display/kafka/kafka+replication, 2013. Accessed: 2023-04-10.

[25]

Thomas Kleymann. Hoare logic and auxiliary variables. Formal Aspects of Computing, 11(5):541--566, December 1999.

Digital Library

[26]

Igor Konnov and Josef Widder. ByMC: Byzantine model checker. In Proceedings of the 8th International Symposium on Leveraging Applications of Formal Methods, Verification, and Validation, pages 327--342, Limassol, Cyprus, November 2018.

[27]

Robbert Krebbers, Amin Timany, and Lars Birkedal. Interactive proofs in higher-order concurrent separation logic. In Proceedings of the 44th ACM Symposium on Principles of Programming Languages (POPL), pages 205--217, Paris, France, January 2017.

Digital Library

[28]

Morten Krogh-Jespersen, Amin Timany, Marit Edna Ohlenbusch, Simon Oddershede Gregersen, and Lars Birkedal. Aneris: A mechanised logic for modular reasoning about distributed systems. In Proceedings of the 29th European Symposium on Programming (ESOP), pages 336--365, Dublin, Ireland, April 2020.

Digital Library

[29]

Leslie Lamport. The temporal logic of actions. ACM Transactions on Programming Languages and Systems, 16(3):872--923, May 1994.

Digital Library

[30]

Leslie Lamport. Specifying Systems, The TLA+ Language and Tools for Hardware and Software Engineers. Addison-Wesley, 2002. ISBN 0-3211-4306-X.

Digital Library

[31]

Haojun Ma, Aman Goel, Jean-Baptiste Jeannin, Manos Kapritsos, Baris Kasikci, and Karem A. Sakallah. I4: Incremental inference of inductive invariants for verification of distributed protocols. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP), Huntsville, Ontario, Canada, October 2019.

Digital Library

[32]

John MacCormick, Nick Murphy, Marc Najork, Chandramohan A. Thekkath, and Lidong Zhou. Boxwood: Abstractions as the foundation for storage infrastructure. In Proceedings of the 6th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 105--120, San Francisco, CA, December 2004.

[33]

Kenneth L. McMillan and Oded Padon. Ivy: A multimodal verification tool for distributed algorithms. In Proceedings of the 32nd International Conference on Computer Aided Verification (CAV), pages 190--202, Los Angeles, CA, July 2020.

[34]

Peter W. O'Hearn. Resources, concurrency, and local reasoning. Theoretical Computer Science, 375(1):271--307, 2007.

Digital Library

[35]

Oded Padon, Giuliano Losa, Mooly Sagiv, and Sharon Shoham. Paxos made EPR: decidable reasoning about distributed protocols. In Proceedings of the 32nd Annual ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), pages 108:1--108:31, Vancouver, Canada, October 2017.

Digital Library

[36]

Ilya Sergey, James R. Wilcox, and Zachary Tatlock. Programming and proving with distributed protocols. In Proceedings of the 45th ACM Symposium on Principles of Programming Languages (POPL), pages 28:1--28:30, Los Angeles, CA, January 2018.

Digital Library

[37]

Upamanyu Sharma. Modular verification of distributed systems with Grove. Master's thesis, Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, September 2022.

[38]

Upamanyu Sharma, Ralf Jung, Joseph Tassarotti, M. Frans Kaashoek, and Nickolai Zeldovich. Grove: a separation-logic library for verifying distributed systems (extended version). arXiv:2309.03046 [cs.LO], September 2023. Available at https://arxiv.org/abs/2309.03046.

[39]

The Coq Development Team. The Coq Proof Assistant, version 8.17.1, June 2023.

[40]

Robbert van Renesse and Fred B. Schneider. Chain replication for supporting high throughput and availability. In Proceedings of the 6th USENIX Symposium on Operating Systems Design and Implementation (OSDI), San Francisco, CA, December 2004.

[41]

James R. Wilcox, Doug Woos, Pavel Panchekha, Zachary Tatlock, Xi Wang, Michael D. Ernst, and Thomas Anderson. Verdi: A framework for implementing and formally verifying distributed systems. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 357--368, Portland, OR, June 2015.

Digital Library

[42]

Jingyu Zhou, Meng Xu, Alexander Shraer, Bala Namasivayam, Alex Miller, Evan Tschannen, Steve Atherton, Andrew J. Beamon, Rusty Sears, John Leach, Dave Rosenthal, Xin Dong, Will Wilson, Ben Collins, David Scherer, Alec Grieser, Young Liu, Alvin Moore, Bhaskar Muppana, Xiaoge Su, and Vishesh Yadav. FoundationDB: A distributed unbundled transactional key value store. In Proceedings of the 2021 ACM SIGMOD International Conference on Management of Data, pages 2653--2666, Virtual conference, June 2021.

Digital Library

Cited By

Qiu LKim YShin JKim JHonoré WShao Z(2024)LiDO: Linearizable Byzantine Distributed Objects with Refinement-Based Liveness ProofsProceedings of the ACM on Programming Languages10.1145/36564238:PLDI(1140-1164)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656423

Index Terms

Grove: a Separation-Logic Library for Verifying Distributed Systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Quasi-synchronous checkpointing and failure recovery in distributed systems
Optimistic recovery in distributed systems

Optimistic Recovery is a new technique supporting application-independent transparent recovery from processor failures in distributed systems. In optimistic recovery communication, computation and checkpointing proceed asynchronously. Synchronization is ...
Verdi: a framework for implementing and formally verifying distributed systems
PLDI '15: Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation

Distributed systems are difficult to implement correctly because they must handle both concurrency and failures: machines may crash at arbitrary points and networks may reorder, drop, or duplicate packets. Further, their behavior is often too complex ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SOSP '23: Proceedings of the 29th Symposium on Operating Systems Principles

October 2023

802 pages

ISBN:9798400702297

DOI:10.1145/3600006

Conference Chairs:
Jason Flinn
Meta
,
Margo Seltzer
University of British Columbia
,
General Chairs:
Peter Druschel
Max Planck Institute for Software Systems (MPI-SWS)
,
Antoine Kaufmann
Max Planck Institute for Software Systems (MPI-SWS)
,
Jonathan Mace
Max Planck Institute for Software Systems (MPI-SWS) and Microsoft Research

Copyright © 2023 Owner/Author(s).

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

In-Cooperation

USENIX

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2023

Check for updates

Badges

Qualifiers

Research-article

Funding Sources

NSF

Conference

SOSP '23

Sponsor:

SIGOPS

SOSP '23: 29th Symposium on Operating Systems Principles

October 23 - 26, 2023

Koblenz, Germany

Acceptance Rates

SOSP '23 Paper Acceptance Rate 43 of 232 submissions, 19%;

Overall Acceptance Rate 131 of 716 submissions, 18%

Upcoming Conference

SOSP '24

Sponsor:
sigops

ACM SIGOPS 30th Symposium on Operating Systems Principles

November 4 - 6, 2024

Austin , TX , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
1,046
Total Downloads

Downloads (Last 12 months)1,046
Downloads (Last 6 weeks)92

Reflects downloads up to 13 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Qiu LKim YShin JKim JHonoré WShao Z(2024)LiDO: Linearizable Byzantine Distributed Objects with Refinement-Based Liveness ProofsProceedings of the ACM on Programming Languages10.1145/36564238:PLDI(1140-1164)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656423

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents