Code to Qed, the Project Manager's Guide to Proof Engineering
Article No.: 171, Pages 1 - 50
Abstract
Despite growing efforts and encouraging successes in recent decades, fully formally verified projects are still rare in the industrial landscape. The industry often lacks the tools and methodologies to efficiently scale the proof development process. In this work, we give a comprehensible overview of the proof development process for proof developers and project managers. The goal is to support proof developers by rationalizing the proof development process, which currently relies heavily on their intuition and expertise, and by facilitating communication with the management line. To this end, we concentrate on the aspect of proof manufacturing and highlight the most significant sources of proof effort. We propose means to mitigate the latter through proof practices (proof structuring, proof strategies, and proof planning), proof metrics, and tools. Our approach is project-agnostic, independent of specific proof expertise, and computed estimations do not assume prior similar developments. We evaluate our guidelines using a separation kernel undergoing formal verification, driving the proof process in an optimised way. Feedback from a project manager unfamiliar with proof development confirms the benefits of detailed planning of the proof development steps, clear progress communication to the hierarchy line, and alignment with established practices in the software industry.
References
[1]
Alloy Team Members. 2021. Website of Alloy Analyzer. Retrieved December 7, 2023 from https://alloytools.org/
[2]
Kent Beck, James Grenning, Robert C. Martin, Mike Beedle, Jim Highsmith, Steve Mellor, Arie van Bennekum, Andrew Hunt, Ken Schwaber, Alistair Cockburn, Ron Jeffries, Jeff Sutherland, Ward Cunningham, Jon Kern, Dave Thomas, Martin Fowler, and Brian Marick. 2001. Website of Manifesto for Agile Software Development. Retrieved December 7, 2023 from https://agilemanifesto.org/
[3]
Leslie Lamport. 2022. Website of TLA+ Tools. Retrieved December 7, 2023 from https://lamport.azurewebsites.net/tla/tools.html
[4]
Why3 Development Team. 2023. Website of Why3. Retrieved December 7, 2023 from https://why3.lri.fr/
[5]
University of Cambridge and Technische Universität München. 2022. Isabelle. Retrieved October 10, 2022 from https://isabelle.in.tum.de/index.html
[6]
Jim Alves-Foss, Paul W. Oman, Carol Taylor, and W. Scott Harrison. 2006. The MILS architecture for high-assurance embedded systems. Int. J. Embed. Syst. 2, 3-4 (2006), 239–247. DOI:
[7]
June Andronick. 2022. Website of CPP Keynote “The seL4 Verification: The Art and Craft of Proof and the Reality of Commercial Support.” Retrieved June 28, 2023 from https://popl22.sigplan.org/details/CPP-2022-papers/27/The-seL4-verification-the-art-and-craft-of-proof-and-the-reality-of-commercial-suppo
[8]
June Andronick, Ross Jeffery, Gerwin Klein, Rafal Kolanski, Mark Staples, He Zhang, and Liming Zhu. 2012. Large-scale formal verification in practice: A process perspective. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE, 1002–1011. DOI:
[9]
Andrew W. Appel. 2011. Verified software toolchain. In Programming Languages and Systems, Gilles Barthe (Ed.). Springer, Berlin, 1–17.
[10]
Md. Junaid Arafeen and Saugata Bose. 2009. Improving software development using scrum model by analyzing up and down movements on the sprint burn down chart—Proposition for better alternatives. J. Digit. Content Technol. Appl. 3 (2009), 109–115. Retrieved from https://api.semanticscholar.org/CorpusID:32851875
[11]
David Aspinall and Cezary Kaliszyk. 2016. Towards formal proof metrics. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9633 (2016), 325–341. DOI:
[12]
Andreas Bollin and Abdollah Tabareh. 2012. Predictive software measures based on Z specifications—A case study. Electr. Proc. Theor. Comput. Sci. 86 (July2012), 33–40. DOI:
[13]
Timothy Bourke, Matthias Daum, Gerwin Klein, and Rafal Kolanski. 2012. Challenges and experiences in managing large-scale proofs. In Proceedings of the 11th International Conference on Intelligent Computer Mathematics AISC, 19th Symposium Calculemus, 5th International Workshop DML, 11th International Conference MKM, and Systems and Projects, Held as Part of CICM(Lecture Notes in Computer Science, Vol. 7362), Johan Jeuring, John A. Campbell, Jacques Carette, Gabriel Dos Reis, Petr Sojka, Makarius Wenzel, and Volker Sorge (Eds.). Springer, Bremen, Germany, 32–48. DOI:
[14]
Richard A. De Millo, Richard J. Lipton, and Alan J. Perlis. 1979. Social processes and proofs of theorems and programs. Commun. ACM 22, 5 (1979), 271–280.
[15]
Nicolas Dejon, Chrystel Gaber, and Gilles Grimaud. 2022. From MMU to MPU: Adaptation of the Pip kernel to constrained devices. In Proceedings of the 3rd International Conference on Internet of Things and Embedded Systems (IoTE’22). AIRCC, 19 pages. Retrieved from https://hal.science/hal-03705114
[16]
Nicolas Dejon, Chrystel Gaber, and Gilles Grimaud. 2023. Pip-MPU: Formal verification of an MPU-based separation kernel for constrained devices. Int. J. Embed. Syst. Appl. 13 (062023), 1–21. DOI:
[17]
Benjamin Delaware, William Cook, and Don Batory. 2011. Product lines of theorems. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA’11). ACM, New York, NY, 595–608. DOI:
[18]
Ronghui Gu, Zhong Shao, Hao Chen, Xiongnan Wu, Jieung Kim, Vilhelm Sjöberg, and David Costanzo. 2016. CertiKOS: An extensible architecture for building certified concurrent OS kernels. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16). 653–669.
[19]
C. A. R. Hoare. 1969. An axiomatic basis for computer programming. Commun. ACM 12, 10 (1969), 576–580. DOI:
[20]
G. J. Holzmann. 1997. The model checker SPIN. IEEE Trans. Softw. Eng. 23, 5 (1997), 279–295. DOI:
[21]
Alexei Iliasov, Paulius Stankaitis, and Alexander B. Romanovsky. 2016. Proving Event-B models with reusable generic lemmas. In Proceedings of the 18th International Conference on Formal Engineering Methods (ICFEM’16)(Lecture Notes in Computer Science, Vol. 10009), Kazuhiro Ogata, Mark Lawford, and Shaoying Liu (Eds.). Springer, Tokyo, Japan, 210–225. DOI:
[22]
INRIA. 1984. Website of Coq. Retrieved January 17, 2020 from https://coq.inria.fr
[23]
Ross Jeffery, Mark Staples, June Andronick, Gerwin Klein, and Toby Murray. 2015. An empirical research agenda for understanding formal methods productivity. Info. Softw. Technol. 60 (2015), 102–112. DOI:
[24]
Narjes Jomaa. 2018. Le co-design d'un noyau de système d'exploitation et de sa preuve formelle d'isolation. Ph.D. Dissertation. Université de Lille, Bâtiment Esprit, Villeneuve d'Ascq.
[25]
Narjes Jomaa, David Nowak, and Paolo Torrini. 2018. Formal Development of the Pip Protokernel. Retrieved October 10, 2022 from https://entropy2018.sciencesconf.org/data/pip.pdf
[26]
Narjes Jomaa, Paolo Torrini, David Nowak, Gilles Grimaud, and Samuel Hym. 2019. Proof-oriented design of a separation kernel with minimal trusted computing base proof-oriented design of a separation kernel with minimal trusted computing base. In Proceedings of the 18th International Workshop on Automated Verification of Critical Systems (AVoCS’18).
[27]
Gerwin Klein, June Andronick, Kevin Elphinstone, Toby Murray, Thomas Sewell, Rafal Kolanski, and Gernot Heiser. 2014. Comprehensive formal verification of an OS microkernel. ACM Trans. Comput. Syst. 32, 1 (2014), 1–70.
[28]
Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, Thomas Sewell, Harvey Tuch, and Simon Winwood. 2009. SeL4: Formal verification of an OS kernel. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (SOSP’09). ACM, New York, NY, 207–220. DOI:
[29]
K. Rustan M. Leino. 2010. Dafny: An automatic program verifier for functional correctness. In Logic for Programming, Artificial Intelligence, and Reasoning, Edmund M. Clarke and Andrei Voronkov (Eds.). Springer, Berlin, 348–370.
[30]
Stephane Lescuyer. 2015. ProvenCore: Towards a verified isolation micro-kernel. In Proceedings of the International Workshop on MILS: Architecture and Assurance for Secure Systems.
[31]
Daniel Matichuk, Toby Murray, June Andronick, Ross Jeffery, Gerwin Klein, and Mark Staples. 2015. Empirical study towards a leading indicator for cost of formal software verification. In Proceedings of the International Conference on Software Engineering. 722–732. DOI:
[32]
Peter O'Hearn. 2019. Separation logic. Commun. ACM 62, 2 (2019), 86–95. DOI:
[33]
Proofcraft. 2023. Website of Proofcraft. Retrieved June 28, 2023 from https://proofcraft.systems/
[34]
ProvenRun. 2020. Website of ProvenRun. Retrieved June 28, 2023 from https://www.provenrun.com/
[35]
Ratish J. Punnoose, Robert C. Armstrong, Matthew H. Wong, and Mayo Jackson. 2014. Survey of existing tools for formal verification. Sandia Report No. SAND2014-20533, Sandia National Laboratories, Albuquerque, NM (Dec. 2014), 1–42. DOI:
[36]
Talia Ringer. 2021. Proof Repair. Ph.D. Dissertation. University of Washington.
[37]
Talia Ringer, Karl Palmskog, Ilya Sergey, Milos Gligoric, and Zachary Tatlock. 2019. QED at large: A survey of engineering of formally verified software. Found. Trends Program. Lang. 5, 2-3 (2019), 102–281. DOI:arxiv:2003.06458
[38]
J. M. Rushby. 1981. Design and verification of secure systems. In Proceedings of the 8th ACM Symposium on Operating Systems Principles (SOSP’81). 12–21. DOI:
[39]
Helgi Sigurbjarnarson, James Bornholt, Nicolas Christin, and Lorrie Faith Cranor. 2017. Push-button verification of file systems via crash refinement. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’17), Dilma Da Silva and Bryan Ford (Eds.). USENIX Association, 1–16. Retrieved from https://www.usenix.org/conference/atc17/technical-sessions/presentation/sigurbjarnarson
[40]
Mark Staples, Ross Jeffery, June Andronick, Toby Murray, Gerwin Klein, and Rafal Kolanski. 2014. Productivity for proof engineering. In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. ACM/IEEE, 1–4.
[41]
Claudia Tona, Reyes Juárez-Ramírez, Samantha Jiménez, Ángeles Quezada, César Guerra-García, and Rafael González Pacheco López. 2021. Scrumlity: An agile framework based on quality assurance. In Proceedings of the 9th International Conference in Software Engineering Research and Innovation (CONISOFT ’21). 88–96. DOI:
[42]
Doug Woos, James R. Wilcox, Steve Anton, Zachary Tatlock, Michael D. Ernst, and Thomas Anderson. 2016. Planning for change in a formal verification of the raft consensus protocol. In Proceedings of the 5th ACM SIGPLAN Conference on Certified Programs and Proofs (CPP’16). ACM, New York, NY, 154–165. DOI:
[43]
He Zhang, Gerwin Klein, Mark Staples, June Andronick, Liming Zhu, and Rafal Kolanski. 2012. Simulation modeling of a large-scale formal verification process. In Proceedings of the International Conference on Software and System Process (ICSSP’12). IEEE, 3–12. DOI:
Index Terms
- Code to Qed, the Project Manager's Guide to Proof Engineering
Recommendations
Proof Pearl: a Formal Proof of Higman's Lemma in ACL2
Higman's lemma is an important result in infinitary combinatorics, which has been formalized in several theorem provers. In this paper we present a formalization and proof of Higman's Lemma in the ACL2 theorem prover. Our formalization is based on a ...
Comments
Information & Contributors
Information
Published In
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 August 2024
Online AM: 04 June 2024
Accepted: 03 May 2024
Revised: 19 March 2024
Received: 10 July 2023
Published in TOSEM Volume 33, Issue 7
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- ANRT Convention Cifre
- TinyPART
- MESRI-BMBF German-French cybersecurity program
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 133Total Downloads
- Downloads (Last 12 months)133
- Downloads (Last 6 weeks)53
Reflects downloads up to 12 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in