research-article

Open access

PerSeVerE: persistency semantics for verification under ext4

Authors:

Michalis Kokologiannakis,

Viktor VafeiadisAuthors Info & Claims

Proceedings of the ACM on Programming Languages, Volume 5, Issue POPL

Article No.: 43, Pages 1 - 29

https://doi.org/10.1145/3434324

Published: 04 January 2021 Publication History

Abstract

Although ubiquitous, modern filesystems have rather complex behaviours that are hardly understood by programmers and lead to severe software bugs such as data corruption. As a first step to ensure correctness of software performing file I/O, we formalize the semantics of the Linux ext4 filesystem, which we integrate with the weak memory consistency semantics of C/C++. We further develop an effective model checking approach for verifying programs that use the filesystem. In doing so, we discover and report bugs in commonly-used text editors such as vim, emacs and nano.

References

[1]

Parosh Abdulla, Stavros Aronis, Bengt Jonsson, and Konstantinos Sagonas ( 2014 ). “Optimal dynamic partial order reduction.” In: POPL 2014. New York, NY, USA: ACM, pp. 373-384.

Digital Library

[2]

Advanced Format ( 2020 ). url: https://en.wikipedia.org/wiki/Advanced_Format (visited on May 20, 2020 ).

[3]

Jade Alglave, Luc Maranget, and Michael Tautschnig ( July 2014 ). “Herding Cats: Modelling, Simulation, Testing, and Data Mining for Weak Memory.” In: ACM Trans. Program. Lang. Syst. 36.2, 7 : 1-7 : 74.

Digital Library

[4]

Jef Bonwick (Oct. 2005 ). ZFS: The Last Word in Filesystems. Library Catalog: blogs.oracle.com. url: https://blogs.oracle.com/ bonwick/zfs% 3A-the-last-word-in-filesystems (visited on June 17, 2020 ).

[5]

James Bornholt, Antoine Kaufmann, Jialin Li, Arvind Krishnamurthy, Emina Torlak, and Xi Wang ( 2016 ). “Specifying and Checking File System Crash-Consistency Models.” In: ASPLOS 2016 44.2, pp. 83-98.

Digital Library

[6]

Haogang Chen, Daniel Ziegler, Tej Chajed, Adam Chlipala, M. Frans Kaashoek, and Nickolai Zeldovich ( 2015 ). “Using Crash Hoare logic for certifying the FSCQ file system.” In: SOSP 2015. the 25th Symposium. Monterey, California: ACM Press, pp. 18-37.

Digital Library

[7]

Ran Chen, Martin Clochard, and Claude Marché ( 2016 ). “A Formal Proof of a Unix Path Resolution Algorithm.” In: HAL hal-01406848. url: https://hal.inria.fr/hal-01406848/document (visited on Nov. 16, 2020 ).

[8]

Copy-on-write ( 2020 ). url: https://en.wikipedia.org/wiki/Copy-on-write (visited on May 20, 2020 ).

[9]

Heming Cui, Gang Hu, Jingyue Wu, and Junfeng Yang ( 2013 ). “Verifying Systems Rules Using Rule-Directed Symbolic Execution.” In: ASPLOS 2013. Houston, Texas, USA: ACM, pp. 329-342.

Digital Library

[10]

GNU Emacs ( 2019 ). GNU Emacs: An extensible, customizable, free/libre text editor-and more. url: https://www.gnu.org/ software/emacs/ (visited on June 15, 2020 ).

[11]

ext4 benchmarks ( 2012 ). EXT4 File-System Tuning Benchmarks. url: https://www.phoronix.com/scan.php ?page=article& item=ext4_linux35_tuning&num=1 (visited on May 20, 2020 ).

[12]

Ext4 data loss ( 2009 ). url: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781 (visited on May 20, 2020 ).

[13]

ext4 Linux kernel ( 2020 ). ext4 Data Structures and Algorithms. url: https://www.kernel.org/doc/html/latest/filesystems/ext4/ index. html (visited on May 20, 2020 ).

[14]

ext4 corruption ( 2015 ). ext4: Filesystem corruption on panic. url: https://bugs.chromium.org/p/chromium/issues/detail?id= 502898 (visited on May 20, 2020 ).

[15]

Michalis Kokologiannakis ( July 2020 ). files: improve the backup procedure to ensure no data is lost. url: https://git.savannah. gnu.org/cgit/nano.git/commit/? id=a84cdaaa50a804a8b872f6d468412dadf105b3c5 (visited on July 9, 2020 ).

[16]

Cormac Flanagan and Patrice Godefroid ( 2005 ). “Dynamic partial-order reduction for model checking software.” In: POPL 2005. New York, NY, USA: ACM, pp. 110-121.

Digital Library

[17]

Patrice Godefroid ( 1997 ). “Model Checking for Programming Languages using VeriSoft.” In: POPL 1997. Paris, France: ACM, pp. 174-186.

Digital Library

[18]

Patrice Godefroid (Mar. 2005 ). “Software Model Checking: The VeriSoft Approach.” In: Form. Meth. Syst. Des. 26.2, pp. 77-101.

Digital Library

[19]

JOE ( 2018 ). JOE-Joe's Own Editor. url: https://joe-editor. sourceforge.io (visited on June 15, 2020 ).

[20]

Rajeev Joshi and Gerard Holzmann (June 11, 2007 ). “A Mini Challenge: Build a Verifiable Filesystem.” In: Formal Asp. Comput. 19, pp. 269-272.

Digital Library

[21]

Eunsuk Kang and Daniel Jackson ( 2008 ). “Formal Modeling and Analysis of a Flash Filesystem in Alloy.” In: ABZ 2008. Ed. by Egon Börger, Michael Butler, Jonathan P. Bowen, and Paul Boca. Vol. 5238. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 294-308.

Digital Library

[22]

Gabriele Keller, Toby Murray, Sidney Amani, Liam O'Connor, Zilin Chen, Leonid Ryzhyk, Gerwin Klein, and Gernot Heiser ( 2013 ). “File systems deserve verification too!” In: PLOS 2013. Farmington, Pennsylvania: ACM Press, pp. 1-7.

Digital Library

[23]

Michalis Kokologiannakis, Ilya Kaysin, Azalea Raad, and Viktor Vafeiadis (Jan. 2021 ). “PerSeVerE: Persistency Semantics for Verification under Ext4 (Supplementary Material).” In: url: https://plv.mpi-sws.org/persevere.

[24]

Michalis Kokologiannakis, Azalea Raad, and Viktor Vafeiadis ( 2019 ). “Model Checking for Weakly Consistent Libraries.” In: PLDI 2019. New York, NY, USA: ACM.

Digital Library

[25]

Ori Lahav, Viktor Vafeiadis, Jeehoon Kang, Chung-Kil Hur, and Derek Dreyer ( 2017 ). “Repairing Sequential Consistency in C/C++ 11.” In: PLDI 2017. Barcelona, Spain: ACM, pp. 618-632.

Digital Library

[26]

Linux man pages ( 2020 ). url: http://www.man7.org/linux/man-pages /index.html (visited on May 20, 2020 ).

[27]

Richard Gooch ( 1999 ). Overview of the Linux Virtual File System. url: https://www.kernel.org/doc/html/latest/filesystems/ vfs. html (visited on May 20, 2020 ).

[28]

Jayashree Mohan, Ashlie Martinez, Soujanya Ponnapalli, Pandian Raju, and Vijay Chidambaram ( 2018 ). “Finding CrashConsistency Bugs with Bounded Black-Box Crash Testing.” In: OSDI 2018. Carlsbad, CA, USA: USENIX Association, pp. 33-50. url: https://www.usenix.org/system/files/osdi18-mohan. pdf (visited on Nov. 16, 2020 ).

[29]

Madanlal Musuvathi, Shaz Qadeer, Thomas Ball, Gérard Basler, Piramanayagam Arumuga Nainar, and Iulian Neamtiu ( 2008 ). “Finding and Reproducing Heisenbugs in Concurrent Programs.” In: OSDI 2008. USENIX Association, pp. 267-280. url: https://www.usenix.org/legacy/events/osdi08/tech/full_papers/musuvathi/musuvathi. pdf (visited on Nov. 16, 2020 ).

[30]

GNU Nano ( 2019 ). The GNU Nano homepage. url: https://nano-editor. org (visited on June 15, 2020 ).

[31]

Gian Ntzik and Philippa Gardner (Oct. 23, 2015 ). “Reasoning about the POSIX file system: local update and global pathnames.” In: OOPSLA 2015. Pittsburgh, PA, USA: Association for Computing Machinery, pp. 201-220.

Digital Library

[32]

Daejun Park and Dongkun Shin ( 2017 ). “iJournaling: Fine-Grained Journaling for Improving the Latency of Fsync System Call.” In: pp. 787-798. url: https://www.usenix.org/conference/atc17/technical-sessions/presentation/park.

[33]

Thanumalayan Sankaranarayana Pillai, Ramnatthan Alagappan, Lanyue Lu, Vijay Chidambaram, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau (Oct. 27, 2017 ). “Application Crash Consistency and Performance with CCFS.” In: ACM Trans. Storage 13.3, pp. 1-29.

Digital Library

[34]

Thanumalayan Sankaranarayana Pillai, Vijay Chidambaram, Ramnatthan Alagappan, Samer Al-Kiswany, Andrea C. ArpaciDusseau, and Remzi H. Arpaci-Dusseau (Oct. 2014 ). “All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications.” In: OSDI 2014. Broomfield, CO: USENIX Association, pp. 433-448. url: https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pillai.

[35]

Anton Podkopaev, Ori Lahav, and Viktor Vafeiadis (Jan. 2019 ). “Bridging the Gap Between Programming Languages and Hardware Weak Memory Models.” In: Proc. ACM Program. Lang. 3.POPL, 69 : 1-69 : 31.

[36]

POSIX ( 2018 ). The Open Group Base Specifications Issue 7. url: https://pubs.opengroup.org/onlinepubs/9699919799/ (visited on May 20, 2020 ).

[37]

Vijayan Prabhakaran, Andrea C Arpaci-Dusseau, and Remzi H Arpaci-Dusseau ( 2005 ). “Analysis and Evolution of Journaling File Systems.” In: p. 16. url: https://www.usenix.org/legacy/events/usenix05/tech/general/full_papers/prabhakaran/ prabhakaran.pdf.

[38]

Azalea Raad and Viktor Vafeiadis (Oct. 2018 ). “Persistence Semantics for Weak Memory: Integrating Epoch Persistency with the TSO Memory Model.” In: Proc. ACM Program. Lang. 2.OOPSLA.

Digital Library

[39]

Azalea Raad, John Wickerson, Gil Neiger, and Viktor Vafeiadis (Dec. 20, 2019a ). “Persistency semantics of the Intel-x86 architecture.” In: Proc. ACM Program. Lang. 4 (POPL), 11 : 1-11 : 31.

[40]

Azalea Raad, John Wickerson, and Viktor Vafeiadis (Oct. 10, 2019b ). “Weak Persistency Semantics from the Ground Up.” In: Proc. ACM Program. Lang. 3 (OOPSLA), 135 : 1-135 : 27.

[41]

renameio ( 2020 ). url: https://github.com/google/renameio (visited on May 20, 2020 ).

[42]

Tom Ridge, David Sheets, Thomas Tuerk, Andrea Giugliano, Anil Madhavapeddy, and Peter Sewell ( 2015 ). “SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems.” In: SOSP 2015. Monterey, California: ACM Press, pp. 38-53.

Digital Library

[43]

Ohad Rodeh, Josef Bacik, and Chris Mason (Aug. 1, 2013 ). “ BTRFS: The Linux B-Tree Filesystem.” In: ACM Trans. Storage 9.3, 9 : 1-9 : 32.

Digital Library

[44]

Cindy Rubio-González, Haryadi S. Gunawi, Ben Liblit, Remzi H. Arpaci-Dusseau, and Andrea C. Arpaci-Dusseau ( June 15, 2009 ). “Error propagation analysis for file systems.” In: SIGPLAN Not. 44.6, pp. 270-280.

Digital Library

[45]

Gerhard Schellhorn, Gidon Ernst, Jörg Pfähler, Dominik Haneberg, and Wolfgang Reif ( 2014 ). “Development of a Verified Flash File System.” In: ABZ 2014. Vol. 8477. Berlin, Heidelberg, pp. 9-24.

Digital Library

[46]

Helgi Sigurbjarnarson, James Bornholt, Emina Torlak, and Xi Wang ( 2016 ). “Push-Button Verification of File Systems via Crash Refinement.” In: OSDI 2016. Savannah, GA, USA: USENIX Association, pp. 1-16. url: https://www.usenix.org/ system/files/conference/osdi16/osdi16-sigurbjarnarson.pdf.

[47]

Seongbae Son, Jinsoo Yoo, and Youjip Won ( 2017 ). “Guaranteeing the Metadata Update Atomicity in EXT4 File system.” In: APSys 2017, pp. 1-8.

[48]

SQLite ( 2020 ). url: https://sqlite.org/index. html (visited on May 20, 2020 ).

[49]

Atomic Commit In SQLite ( 2020 ). url: https://sqlite.org/atomiccommit. html (visited on May 20, 2020 ).

[50]

Adam Sweeney ( 1996 ). “Scalability in the XFS file system. ” In: USENIX ATC 1996, pp. 1-14. url: https://www.usenix.org/ legacy/publications/library/proceedings/sd96/sweeney.html.

[51]

Theodore Y Ts'o and Stephen Tweedie ( 2002 ). “Planned Extensions to the Linux Ext2/Ext3 Filesystem.” In: pp. 235-243. url: http://www.usenix.org/publications/library/proceedings/usenix02/tech/freenix/tso.html.

[52]

Stephen C Tweedie ( 1998 ). “Journaling the Linux ext2fs Filesystem.” In: LinuxExpo 1998. url: http://e2fsprogs.sourceforge. net/journal-design. pdf (visited on Nov. 16, 2020 ).

[53]

Vim ( 2019 ). Vim-the ubiquitous text editor. url: https://vim. org (visited on June 15, 2020 ).

[54]

Junfeng Yang, Can Sar, and Dawson Engler (Nov. 6, 2006 ). “EXPLODE: a lightweight, general system for finding serious storage system errors.” In: OSDI 2006. Seattle, Washington: USENIX Association, pp. 131-146. url: https://www.usenix. org/legacy/event/osdi06/tech/full_papers/yang_junfeng/yang_junfeng. pdf (visited on June 17, 2020 ).

[55]

Mai Zheng, Joseph Tucek, Dachuan Huang, Elizabeth S Yang, Bill W Zhao, Feng Qin, Mark Lillibridge, and Shashank Singh ( 2014 ). “Torturing Databases for Fun and Profit.” In: OSDI 2014. Broomfield, CO: USENIX Association, pp. 449-464. url: https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-zheng_mai. pdf (visited on Nov. 16, 2020 ).

Cited By

Ambal GDongol BEran HKlimis VLahav ORaad A(2024)Semantics of Remote Direct Memory Access: Operational and Declarative Models of RDMA on TSO ArchitecturesProceedings of the ACM on Programming Languages10.1145/36897818:OOPSLA2(1982-2009)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689781
Liu JHao XArpaci-Dusseau AArpaci-Dusseau RChajed T(2024)Shadow Filesystems: Recovering from Filesystem Runtime Errors via Robust Alternative ExecutionProceedings of the 16th ACM Workshop on Hot Topics in Storage and File Systems10.1145/3655038.3665942(15-22)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3655038.3665942
Huang ZWang CRoychoudhury APaiva AAbreu RStorey M(2024)Constraint Based Program Repair for Persistent Memory BugsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639204(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639204
Show More Cited By

Index Terms

PerSeVerE: persistency semantics for verification under ext4
1. Theory of computation
  1. Semantics and reasoning
    1. Program reasoning
      1. Program verification
    2. Program semantics
      1. Axiomatic semantics

Recommendations

Safe replication through bounded concurrency verification

High-level data types are often associated with semantic invariants that must be preserved by any correct implementation. While having implementations enforce strong guarantees such as linearizability or serializability can often be used to prevent ...
A multiple-file write scheme for improving write performance of small files in Fast File System

Fast File System (FFS) stores files to disk in separate disk writes, each of which incurs a disk positioning (seek + rotation) limiting the write performance for small files. We propose a new scheme called co-writing to accelerate small file writes in ...
Evolving Ext4 for shingled disks
FAST'17: Proceedings of the 15th Usenix Conference on File and Storage Technologies

Drive-Managed SMR (Shingled Magnetic Recording) disks offer a plug-compatible higher-capacity replacement for conventional disks. For non-sequential workloads, these disks show bimodal behavior: After a short period of high throughput they enter a ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Programming Languages

Proceedings of the ACM on Programming Languages Volume 5, Issue POPL

January 2021

1789 pages

EISSN:2475-1421

DOI:10.1145/3445980

Issue’s Table of Contents

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 January 2021

Published in PACMPL Volume 5, Issue POPL

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

RFBR
European Research Council

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
762
Total Downloads

Downloads (Last 12 months)147
Downloads (Last 6 weeks)14

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ambal GDongol BEran HKlimis VLahav ORaad A(2024)Semantics of Remote Direct Memory Access: Operational and Declarative Models of RDMA on TSO ArchitecturesProceedings of the ACM on Programming Languages10.1145/36897818:OOPSLA2(1982-2009)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689781
Liu JHao XArpaci-Dusseau AArpaci-Dusseau RChajed T(2024)Shadow Filesystems: Recovering from Filesystem Runtime Errors via Robust Alternative ExecutionProceedings of the 16th ACM Workshop on Hot Topics in Storage and File Systems10.1145/3655038.3665942(15-22)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3655038.3665942
Huang ZWang CRoychoudhury APaiva AAbreu RStorey M(2024)Constraint Based Program Repair for Persistent Memory BugsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639204(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639204
Sato SMizuhashi TKimura GTaura K(2024)Efficiently Adapting Stateless Model Checking for C11/C++11 to Mixed-Size AccessesProgramming Languages and Systems10.1007/978-981-97-8943-6_17(346-364)Online publication date: 28-Oct-2024
https://doi.org/10.1007/978-981-97-8943-6_17
Stefanesco LRaad AVafeiadis V(2024)Specifying and Verifying Persistent LibrariesProgramming Languages and Systems10.1007/978-3-031-57267-8_8(185-211)Online publication date: 6-Apr-2024
https://dl.acm.org/doi/10.1007/978-3-031-57267-8_8
Marmanis IVafeiadis V(2023)SMT-Based Verification of Persistency Invariants of Px86 ProgramsVerified Software. Theories, Tools and Experiments.10.1007/978-3-031-25803-9_6(92-110)Online publication date: 1-Feb-2023
https://doi.org/10.1007/978-3-031-25803-9_6
Coughlin NWinter KSmith G(2022)Compositional Reasoning for Non-multicopy Atomic ArchitecturesFormal Aspects of Computing10.1145/357413735:2(1-30)Online publication date: 14-Dec-2022
https://dl.acm.org/doi/10.1145/3574137
Raad AMaranget LVafeiadis V(2022)Extending Intel-x86 consistency and persistency: formalising the semantics of Intel-x86 memory types and non-temporal storesProceedings of the ACM on Programming Languages10.1145/34986836:POPL(1-31)Online publication date: 12-Jan-2022
https://dl.acm.org/doi/10.1145/3498683
Wu CYang C(2022)An Integrated Subpage-aware Write Method in Large-Page-based SSDs2022 IEEE 11th Non-Volatile Memory Systems and Applications Symposium (NVMSA)10.1109/NVMSA56066.2022.00018(51-56)Online publication date: Aug-2022
https://doi.org/10.1109/NVMSA56066.2022.00018
Bila EDongol BLahav ORaad AWickerson J(2022)View-Based Owicki–Gries Reasoning for Persistent x86-TSOProgramming Languages and Systems10.1007/978-3-030-99336-8_9(234-261)Online publication date: 29-Mar-2022
https://doi.org/10.1007/978-3-030-99336-8_9
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents