research-article

Public Access

A Fast, General System for Buffered Persistent Data Structures

Authors:

Benjamin Valpey,

Michael L. ScottAuthors Info & Claims

ICPP '21: Proceedings of the 50th International Conference on Parallel Processing

Article No.: 73, Pages 1 - 11

https://doi.org/10.1145/3472456.3472458

Published: 05 October 2021 Publication History

All formats PDF

Abstract

The emergence of fast, dense, nonvolatile main memory suggests that certain long-lived data might remain in their natural pointer-rich format across program runs and hardware reboots. Operations on such data must currently be instrumented with explicit write-back and fence instructions to ensure consistency in the wake of a crash. Techniques to minimize the cost of this instrumentation are an active topic of research.

We present what we believe to be the first general-purpose approach to building buffered persistent data structures, and a system, Montage, to support that approach. Montage is built on top of the Ralloc nonblocking persistent allocator. It employs a millisecond-granularity epoch clock, and ensures that no operation appears to span an epoch boundary. It also arranges to persist only that data minimally required to reconstruct the structure after a crash. If a crash occurs in epoch e, all work performed in epochs e and e − 1 is lost, but work from prior epochs is preserved, consistently. As in traditional file and database systems, a sync operation can be used to flush buffers on demand; the Montage sync is extremely fast.

We describe the implementation of Montage, argue its correctness, and report unprecedented throughput for persistent queues, sets/mappings, and general graphs.

References

[1]

David Aksun and James Larus. 2021. Durability Through NVM Checkpointing (poster). In 12th Non-Volatile Memories Wkshp.

[2]

H. Alan Beadle, Wentao Cai, Haosen Wen, and Michael L. Scott. 2020. Nonblocking Persistent Software Transactional Memory. In 27th Intl. Conf. on High Performance Computing, Data, and Analytics (HiPC).

[3]

W. Cai, H. Wen, H. A. Beadle, C. Kjellqvist, M. Hedayati, and M. L. Scott. 2020. Understanding and Optimizing Persistent Memory Allocation. In 19th Intl. Symp. on Memory Management (ISMM).

[4]

Dhruva R. Chakrabarti, Hans-J. Boehm, and Kumud Bhandari. 2014. Atlas: Leveraging Locks for Non-volatile Memory Consistency. In ACM Conf. on Object Oriented Programming Systems Languages & Applications (OOPSLA).

[5]

Andreas Chatzistergiou, Marcelo Cintra, and Stratis D. Viglas. 2015. REWIND: Recovery Write-ahead system for In-memory Non-volatile Data-structures. Proc. of the VLDB Endowment 8, 5 (Jan. 2015).

Digital Library

[6]

Shimin Chen and Qin Jin. 2015. Persistent B+-trees in Non-volatile Main Memory. Proc. of the VLDB Endowment 8, 7 (Feb. 2015).

Digital Library

[7]

Joel Coburn, Adrian M. Caulfield, Ameen Akel, Laura M. Grupp, Rajesh K. Gupta, Ranjit Jhala, and Steven Swanson. 2011. NV-Heaps: Making Persistent Objects Fast and Safe with Next-generation, Non-volatile Memories. In 16th Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 14 pages. https://doi.org/10.1145/1950365.1950380

Digital Library

[8]

Nachshon Cohen, David T. Aksun, Hillel Avni, and James R. Larus. 2019. Fine-Grain Checkpointing with In-Cache-Line Logging. In 24th Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 14 pages. https://doi.org/10.1145/3297858.3304046

Digital Library

[9]

Nachshon Cohen, David T. Aksun, and James R. Larus. 2018. Object-Oriented Recovery for Non-Volatile Memory. Proc. of the ACM on Programming Languages 2, OOPSLA (Oct. 2018).

Digital Library

[10]

Brian Cooper. 2010. YCSB Core Workloads. https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads.

[11]

Andreia Correia, Pascal Felber, and Pedro Ramalhete. 2018. Romulus: Efficient Algorithms for Persistent Transactional Memory. In 30th ACM Symp. on Parallel Algorithms and Architectures (SPAA).

Digital Library

[12]

Tudor David, Aleksandar Dragojević, Rachid Guerraoui, and Igor Zablotchi. 2018. Log-Free Concurrent Data Structures. In Usenix Annual Technical Conf. (ATC). https://www.usenix.org/conference/atc18/presentation/david

[13]

Michal Friedman, Naama Ben-David, Yuanhao Wei, Guy E Blelloch, and Erez Petrank. 2020. NVTraverse: In NVRAM Data Structures, the Destination Is More Important Than the Journey. In 41st ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI).

Digital Library

[14]

Michal Friedman, Maurice Herlihy, Virendra Marathe, and Erez Petrank. 2018. A Persistent Lock-free Queue for Non-volatile Memory. In 23rd ACM SIGPLAN Symp. on Principles and Practice of Parallel Programming (PPoPP). 13 pages.

Digital Library

[15]

Ellis R. Giles, Kshitij Doshi, and Peter Varman. 2015. SoftWrAP: A Lightweight Framework for Transactional Support of Storage Class Memory. In 31st Symp. on Mass Storage Systems and Technologies (MSST).

[16]

Jinyu Gu, Qianqian Yu, Xiayang Wang, Zhaoguo Wang, Binyu Zang, Haibing Guan, and Haibo Chen. 2019. Pisces: A Scalable and Efficient Persistent Transactional Memory. In Usenix Annual Technical Conf. (ATC).

[17]

Swapnil Haria, Mark D. Hill, and Michael M. Swift. 2020. MOD: Minimally Ordered Durable Datastructures for Persistent Memory. In 25th Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS).

Digital Library

[18]

Terry Ching-Hsiang Hsu, Helge Brügner, Indrajit Roy, Kimberly Keeton, and Patrick Eugster. 2017. NVthreads: Practical persistence for multi-threaded applications. In 12th European Conf. on Computer Systems (EuroSys).

[19]

Deukyeon Hwang, Wook-Hee Kim, Youjip Won, and Beomseok Nam. 2018. Endurable Transient Inconsistency in Byte-Addressable Persistent B+-Tree. In 16th Usenix Conf. on File and Storage Technologies (FAST).

[20]

Joseph Izraelevitz, Terence Kelly, and Aasheesh Kolli. 2016. Failure-Atomic Persistent Memory Updates via JUSTDO Logging. In 21st Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS).

Digital Library

[21]

J. Izraelevitz, H. Mendes, and M. L. Scott. 2016. Linearizability of Persistent Memory Objects Under a Full-System-Crash Failure Model. In Intl. Symp. on Distributed Computing (DISC).

[22]

Joseph Izraelevitz, Jian Yang, Lu Zhang, Juno Kim, Xiao Liu, Amirsaman Memaripour, Yun Joon Soh, Zixuan Wang, Yi Xu, Subramanya R. Dulloor, Jishen Zhao, and Steven Swanson. 2019. Basic Performance Measurements of the Intel Optane DC Persistent Memory Module. arXiv:1903.05714v3.

[23]

Chris Kjellqvist, Mohammad Hedayati, and Michael L. Scott. 2020. Safe, Fast Sharing of Memcached as a Protected Library. In 49th Intl. Conf. on Parallel Processing (ICPP). Article 6, 8 pages. https://doi.org/10.1145/3404397.3404443

Digital Library

[24]

R Madhava Krishnan, Jaeho Kim, Ajit Mathew, Xinwei Fu, Anthony Demeri, Changwoo Min, and Sudarsun Kannan. 2020. Durable transactional memory can scale with TimeStone. In 25th Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS).

Digital Library

[25]

Harris Timothy L., Fraser Keir, and Pratt Ian A.2002. A Practical Multi-word Compare-and-Swap Operation. In 16th Intl. Symp. on Distributed Computing (DISC).

[26]

Se Kwon Lee, K. Hyun Lim, Hyunsub Song, Beomseok Nam, and Sam H. Noh. 2017. WORT: Write Optimal Radix Tree for Persistent Memory Storage Systems. In 15th Usenix Conf. on File and Storage Technologies (FAST).

[27]

Ricardo Leite and Ricardo Rocha. 2018. LRMalloc: A Modern and Competitive Lock-Free Dynamic Memory Allocator. In 13th Intl. Meeting on High Performance Computing for Computational Science (VECPAR).

[28]

Jure Leskovec and Rok Sosič. 2016. SNAP: A General-Purpose Network Analysis and Graph-Mining Library. ACM Trans. on Intelligent Systems and Technology 8, 1 (July 2016).

Digital Library

[29]

Qingrui Liu, Joseph Izraelevitz, Se Kwon Lee, Michael L Scott, Sam H Noh, and Changhee Jung. 2018. iDO: Compiler-directed failure atomicity for nonvolatile memory. In 51st Intl. Symp. on Microarchitecture (MICRO).

Digital Library

[30]

Yujie Liu, Victor Luchangco, and Michael Spear. 2013. Mindicators: A scalable approach to quiescence. In 33rd IEEE Intl. Conf. on Distributed Computing Systems (ICDCS).

Digital Library

[31]

Pratyush Mahapatra, Mark D Hill, and Michael M Swift. 2019. Don’t Persist All: Efficient Persistent Data Structures. arXiv:1905.13011.

[32]

Amirsaman Memaripour, Joseph Izraelevitz, and Steven Swanson. 2020. Pronto: Easy and Fast Persistence for Volatile Data Structures. In 25th Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS).

Digital Library

[33]

Amirsaman Memaripour and Steven Swanson. 2018. Breeze: User-Level Access to Non-Volatile Main Memories for Legacy Software. In 36th Intl. Conf. on Computer Design (ICCD).

[34]

Moohyeon Nam, Hokeun Cha, Kibeom Jin, Jiwon Seo, and Beomseok Nam. 2020. B3-tree: Byte-Addressable Binary B-Tree for Persistent Memory. ACM Trans. on Storage 16, 3 (July 2020).

[35]

Faisal Nawab, Joseph Izraelevitz, Terence Kelly, Charles B. Morrey III, Dhruva R. Chakrabarti, and Michael L. Scott. 2017. Dalí: A Periodically Persistent Hash Map. In Intl. Symp. on Distributed Computing (DISC).

[36]

Ismail Oukid, Johan Lasperas, Anisoara Nica, Thomas Willhalm, and Wolfgang Lehner. 2016. FPTree: A Hybrid SCM-DRAM Persistent and Concurrent B-Tree for Storage Class Memory. In Intl. Conf. on the Management of Data (SIGMOD).

Digital Library

[37]

Matej Pavlovic, Alex Kogan, Virendra J Marathe, and Tim Harris. 2018. Brief announcement: Persistent multi-word compare-and-swap. In ACM Symp. on Principles of Distributed Computing (PODC).

Digital Library

[38]

Pedro Ramalhete, Andreia Correia, Pascal Felber, and Nachshon Cohen. 2019. OneFile: A Wait-Free Persistent Transactional Memory. In 49th IEEE/IFIP Intl. Conf. on Dependable Systems and Networks (DSN).

[39]

Torvald Riegel, Pascal Felber, and Christof Fetzer. 2006. A Lazy Snapshot Algorithm with Eager Validation. In 20th Intl. Symp. on Distributed Computing (DISC).

Digital Library

[40]

Steve Scargall. 2018. Using Persistent Memory Devices with the Linux Device Mapper. https://pmem.io/2018/05/15/using_persistent_memory_devices_with_the_linux_device_mapper.html.

[41]

David Schwalb, Markus Dreseler, Matthias Uflacker, and Hasso Plattner. 2015. NVC-Hashmap: A Persistent and Concurrent Hashmap For Non-Volatile Memories. In 3rd VLDB Wkshp. on In-Memory Data Management and Analytics (IMDM).

[42]

Usharani U. and Andy M. Rudoff. 2017. Introduction to Programming with Persistent Memory from Intel. https://software.intel.com/en-us/articles/introduction-to-programming-with-persistent-memory-from-intel.

[43]

Shivaram Venkataraman, Niraj Tolia, Parthasarathy Ranganathan, and Roy H. Campbell. 2011. Consistent and Durable Data Structures for Non-volatile Byte-addressable Memory. In 9th Usenix Conf. on File and Storage Technologies (FAST).

[44]

Haris Volos, Andres Jaan Tack, and Michael M. Swift. 2011. Mnemosyne: Lightweight Persistent Memory. In 16th Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS).

Digital Library

[45]

Chundong Wang, Qingsong Wei, Lingkun Wu, Sibo Wang, Cheng Chen, Xiaokui Xiao, Jun Yang, Mingdi Xue, and Yechao Yang. 2018. Persisting RB-Tree into NVM in a Consistency Perspective. ACM Trans. on Storage 14, 1 (Feb. 2018).

Digital Library

[46]

Zhenwei Wu, Kai Lu, Andrew Nisbet, Wenzhe Zhang, and Mikel Luján. 2020. PMThreads: Persistent memory threads harnessing versioned shadow copies. In 41st ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI).

Digital Library

[47]

Yi Xu, Joseph Izraelevitz, and Steven Swanson. 2021. Clobber-NVM: Log Less, Re-execute More. In 26th Intl. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS).

[48]

Jaewon Yang and Jure Leskovec. 2012. SNAP Dataset: Orkut Social Network and Ground-Truth Communities. https://snap.stanford.edu/data/com-Orkut.html.

[49]

Jun Yang, Qingsong Wei, Cheng Chen, Chundong Wang, Khai Leong Yong, and Bingsheng He. 2015. NV-Tree: Reducing Consistency Cost for NVM-based Single Level Systems. In 13th Usenix Conf. on File and Storage Technologies (FAST).

[50]

Yoav Zuriel, Michal Friedman, Gali Sheffi, Nachshon Cohen, and Erez Petrank. 2019. Efficient Lock-free Durable Sets. Proc. of the ACM on Programming Languages 3, OOPSLA (Oct. 2019).

Digital Library

Cited By

Anand SFriedman MGiardino MAlonso GTsafrir DMusuvathi MGupta RAbu-Ghazaleh N(2024)Skip It: Take Control of Your Cache!Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3620665.3640407(1077-1094)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3620665.3640407
Bodenmüller SDerrick JDongol BSchellhorn GWehrheim H(2024)A Fully Verified Persistency LibraryVerification, Model Checking, and Abstract Interpretation10.1007/978-3-031-50521-8_2(26-47)Online publication date: 15-Jan-2024
https://dl.acm.org/doi/10.1007/978-3-031-50521-8_2
Nakata MSato SUgawa T(2023)General-purpose Asynchronous Periodic Checkpointing in Hybrid MemoryProceedings of the 52nd International Conference on Parallel Processing10.1145/3605573.3605648(675-684)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3605573.3605648
Show More Cited By

Recommendations

Mirror: making lock-free data structures persistent
PLDI 2021: Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation

With the recent launch of the Intel Optane memory platform, non-volatile main memory in the form of fast, dense, byte-addressable non-volatile memory has now become available. Nevertheless, designing crash-resilient algorithms and data structures is ...
A Consistency Framework for Iteration Operations in Concurrent Data Structures
IPDPS '15: Proceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium

Concurrent data structures provide the means to multi-threaded applications to share data. Data structures come with a set of predefined operations, specified by the semantics of the data structure. In the literature and in several contemporary commonly ...
Nonblocking Data Structures on Persistent Memory

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICPP '21: Proceedings of the 50th International Conference on Parallel Processing

August 2021

927 pages

ISBN:9781450390682

DOI:10.1145/3472456

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Department of Energy Computational Science Graduate Fellowship
NSF (National Science Foundation)
Google Faculty Research award

Conference

ICPP 2021

ICPP 2021: 50th International Conference on Parallel Processing

August 9 - 12, 2021

IL, Lemont, USA

Acceptance Rates

Overall Acceptance Rate 91 of 313 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
837
Total Downloads

Downloads (Last 12 months)450
Downloads (Last 6 weeks)31

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Anand SFriedman MGiardino MAlonso GTsafrir DMusuvathi MGupta RAbu-Ghazaleh N(2024)Skip It: Take Control of Your Cache!Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3620665.3640407(1077-1094)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3620665.3640407
Bodenmüller SDerrick JDongol BSchellhorn GWehrheim H(2024)A Fully Verified Persistency LibraryVerification, Model Checking, and Abstract Interpretation10.1007/978-3-031-50521-8_2(26-47)Online publication date: 15-Jan-2024
https://dl.acm.org/doi/10.1007/978-3-031-50521-8_2
Nakata MSato SUgawa T(2023)General-purpose Asynchronous Periodic Checkpointing in Hybrid MemoryProceedings of the 52nd International Conference on Parallel Processing10.1145/3605573.3605648(675-684)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3605573.3605648
Assa GCorreia ARamalhete PSchiavoni VFelber PDehnavi MKulkarni MKrishnamoorthy S(2023)TL4xProceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming10.1145/3572848.3577495(245-259)Online publication date: 25-Feb-2023
https://dl.acm.org/doi/10.1145/3572848.3577495
Gonçalves JMatos MRodrigues RFedorova ANarayanan DDi Luna GQuerzoni L(2023)Mumak: Efficient and Black-Box Bug Detection for Persistent MemoryProceedings of the Eighteenth European Conference on Computer Systems10.1145/3552326.3587447(734-750)Online publication date: 8-May-2023
https://dl.acm.org/doi/10.1145/3552326.3587447

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents