research-article

Low-Overhead Asynchronous Checkpointing in Main-Memory Database Systems

Authors:

Thaddeus Diamond,

Daniel J. Abadi,

Alexander ThomsonAuthors Info & Claims

SIGMOD '16: Proceedings of the 2016 International Conference on Management of Data

Pages 1539 - 1551

https://doi.org/10.1145/2882903.2915966

Published: 26 June 2016 Publication History

Abstract

As it becomes increasingly common for transaction processing systems to operate on datasets that fit within the main memory of a single machine or a cluster of commodity machines, traditional mechanisms for guaranteeing transaction durability---which typically involve synchronous log flushes---incur increasingly unappealing costs to otherwise lightweight transactions. Many applications have turned to periodically checkpointing full database state. However, existing checkpointing methods---even those which avoid freezing the storage layer---often come with significant costs to operation throughput, end-to-end latency, and total memory usage.

This paper presents Checkpointing Asynchronously using Logical Consistency (CALC), a lightweight, asynchronous technique for capturing database snapshots that does not require a physical point of consistency to create a checkpoint, and avoids conspicuous latency spikes incurred by other database snapshotting schemes. Our experiments show that CALC can capture frequent checkpoints across a variety of transactional workloads with extremely small cost to transactional throughput and low additional memory usage compared to other state-of-the-art checkpointing systems.

References

[1]

P. Bernstein and E. Newcomer. Principles of Transaction Processing. seconod edition edition, 2009.

Digital Library

[2]

T. Cao, M. Vaz Salles, B. Sowell, Y. Yue, A. Demers, J. Gehrke, and W. White. Fast checkpoint recovery algorithms for frequently consistent applications. In Proceedings of the 2011 international conference on Management of data, SIGMOD '11, pages 265--276, 2011.

Digital Library

[3]

D. J. DeWitt, R. H. Katz, F. Olken, L. D. Shapiro, M. R. Stonebraker, and D. A. Wood. Implementation techniques for main memory database systems. In Proceedings of the 1984 ACM SIGMOD international conference on Management of data, SIGMOD '84, pages 1--8, New York, NY, USA, 1984. ACM.

Digital Library

[4]

C. Diaconu, C. Freedman, E. Ismert, P. AEke Larson, P. Mittal, R. Stonecipher, N. Verma, and M. Zwilling. Hekaton: Sql server's memory-optimized oltp engine. In SIGMOD, 2013.

Digital Library

[5]

R. B. Hagmann. A crash recovery scheme for a memory-resident database system. IEEE Trans. Comput., 35:839--843, September 1986.

Digital Library

[6]

S. Harizopoulos, D. J. Abadi, S. R. Madden, and M. Stonebraker. OLTP through the looking glass, and what we found there. In SIGMOD, 2008.

Digital Library

[7]

A. Kemper and T. Neumann. Hyper: A hybrid OLTP&OLAP main memory database system based on virtual memory snapshots. Proc. of ICDE, pages 195--206, June 2011.

Digital Library

[8]

T. Lahiri, M.-A. Neimat, and S. Folkman. Oracle timesten: An in-memory database for enterprise applications. In IEEE Data Engineering Bulletin., June 2013.

[9]

E. Lau and S. Madden. An integrated approach to recovery and high availability in an updatable, distributed data warehouse. In Proc. of VLDB, pages 703--714, 2006.

Digital Library

[10]

J. Lee, M. Muehle, N. May, F. Faerber, V. Sikka1, H. Plattner, J. Krueger, and M. Grund. High-performance transaction processing in sap hana. In IEEE Data Engineering Bulletin., June 2013.

[11]

X. Li and M. H. Eich. Post-crash log processing for fuzzy checkpointing main memory databases. In Procceedings of the Ninth International Conference on Data Engineering, Apr. 1993.

Digital Library

[12]

J.-L. Lin and M. H. Dunham. Segmented fuzzy checkpointing for main memory databases. In Procceedings of the ACM Symposium on Applied Computing, February 1996.

Digital Library

[13]

N. Malviya, A. Weisberg, S. Madden, and M. Stonebraker. Rethinking main memory oltp recovery. In Proc. of ICDE, 2014.

[14]

C. Mohan, D. Haderle, B. Lindsay, H. Pirahesh, and P. Schwarz. Aries: a transaction recovery method supporting ne-granularity locking and partial rollbacks using write-ahead logging. ACM Trans. Database Syst., 17:94--162, March 1992.

Digital Library

[15]

C. Pu. On-the- y, incremental, consistent reading of entire databases. In Proceedings of the 11th international conference on Very Large Data Bases - Volume 11, VLDB '85, pages 369--375. VLDB Endowment, 1985.

Digital Library

[16]

K. Ren, A. Thomson, and D. J. Abadi. An evaluation of the advantages and disadvantages of deterministic database systems. PVLDB 7(10): 821-832, 2014.

Digital Library

[17]

K. Salem and H. Garcia-Molina. Checkpointing memory-resident databases. In In Proc. ICDE, 1989.

Digital Library

[18]

A. Silberschatz, H. Korth, and S. Sudarshan. Database Systems Concepts. 5 edition, 2006.

Digital Library

[19]

M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen, M. Cherniack, M. Ferreira, E. Lau, A. Lin, S. R. Madden, E. J. O'Neil, P. E. O'Neil, A. Rasin, N. Tran, and S. B. Zdonik. C-store: A column-oriented dbms. In VLDB, pages 553--564, Trondheim, Norway, 2005.

Digital Library

[20]

M. Stonebraker, S. R. Madden, D. J. Abadi, S. Harizopoulos, N. Hachem, and P. Helland. The end of an architectural era (it's time for a complete rewrite). In VLDB, Vienna, Austria, 2007.

Digital Library

[21]

A. Thomson and D. J. Abadi. The case for determinism in database systems. VLDB, 2010.

Digital Library

[22]

A. Thomson, T. Diamond, P. Shao, K. Ren, S.-C. Weng, and D. J. Abadi. Calvin: Fast distributed transactions for partitioned database systems. In SIGMOD, 2012.

Digital Library

[23]

VoltDB. http://voltdb.com.

[24]

A. Whitney, D. Shasha, and S. Apter. High Volume Transaction Processing Without Concurrency Control, Two Phase Commit, SQL or C++. In Int. Workshop on High Performance Transaction Systems, 1997.

[25]

W. Zheng and S. Tu. Fast databases with fast durability and recovery through multicore parallelism. In OSDI, 2014.

Digital Library

Cited By

Liu RElmore AFranklin MKrishnan S(2024)Riveter: Adaptive Query Suspension and Resumption Framework for Cloud Native Databases2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00304(3975-3988)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00304
Magalhaes ABrayner AMonteiro J(2024)MM-DIRECTThe VLDB Journal10.1007/s00778-024-00846-z33:3(859-882)Online publication date: 27-Mar-2024
https://doi.org/10.1007/s00778-024-00846-z
Chen ZYang XLi FCheng XHu QMiao ZXie RWu XWang KSong ZSun HZhuang ZYang YXu JYin LZhou WWang S(2022)CloudJumpProceedings of the VLDB Endowment10.14778/3554821.355483415:12(3432-3444)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.14778/3554821.3554834
Show More Cited By

Index Terms

Low-Overhead Asynchronous Checkpointing in Main-Memory Database Systems
1. Information systems
  1. Data management systems
    1. Database design and models
    2. Database management system engines

Recommendations

Two-step backup mechanism for real-time main memory database recovery
RTCSA '00: Proceedings of the Seventh International Conference on Real-Time Systems and Applications

Proposes a recovery mechanism for real-time main-memory database systems (RT-MMDBSs) which aims at reducing the system recovery time, as well as ensuring a fast response time and predictability. Conventional main-memory database (MMDB) recovery schemes ...
Incremental Recovery in Main Memory Database Systems

Recovery activities, like checkpointing and restart, in traditional database management systems are performed in a quiescent state where no transactions are active. This approach impairs the performance of online transaction processing systems, ...
Integrating Non-Volatile Main Memory in a Deterministic Database
EuroSys '23: Proceedings of the Eighteenth European Conference on Computer Systems

Deterministic databases provide strong serializability while avoiding concurrency-control related aborts by establishing a serial ordering of transactions before their execution. Recent work has shown that they can also handle skewed and contended ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '16: Proceedings of the 2016 International Conference on Management of Data

June 2016

2300 pages

ISBN:9781450335317

DOI:10.1145/2882903

General Chairs:
Fatma Özcan
IBM Research, USA
,
Georgia Koutrika
HP Labs, USA
,
Program Chair:
Sam Madden
Massachusetts Institute of Technology, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation USA

Conference

SIGMOD/PODS'16

Sponsor:

SIGMOD

SIGMOD/PODS'16: International Conference on Management of Data

June 26 - July 1, 2016

California, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
569
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)1

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu RElmore AFranklin MKrishnan S(2024)Riveter: Adaptive Query Suspension and Resumption Framework for Cloud Native Databases2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00304(3975-3988)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00304
Magalhaes ABrayner AMonteiro J(2024)MM-DIRECTThe VLDB Journal10.1007/s00778-024-00846-z33:3(859-882)Online publication date: 27-Mar-2024
https://doi.org/10.1007/s00778-024-00846-z
Chen ZYang XLi FCheng XHu QMiao ZXie RWu XWang KSong ZSun HZhuang ZYang YXu JYin LZhou WWang S(2022)CloudJumpProceedings of the VLDB Endowment10.14778/3554821.355483415:12(3432-3444)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.14778/3554821.3554834
Lee LXie SMa YChen S(2022)Index checkpoints for instant recovery in in-memory database systemsProceedings of the VLDB Endowment10.14778/3529337.352935015:8(1671-1683)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.14778/3529337.3529350
Liu GChen LChen S(2022)Zen+: a robust NUMA-aware OLTP engine optimized for non-volatile main memoryThe VLDB Journal10.1007/s00778-022-00737-132:1(123-148)Online publication date: 6-Apr-2022
https://doi.org/10.1007/s00778-022-00737-1
Liu GChen LChen S(2021)ZenProceedings of the VLDB Endowment10.14778/3446095.344610514:5(835-848)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.14778/3446095.3446105
Li TButrovich MNgom ALim WMcKinney WPavlo A(2021)Mainlining databasesProceedings of the VLDB Endowment10.14778/3436905.343691314:4(534-546)Online publication date: 22-Feb-2021
https://dl.acm.org/doi/10.14778/3436905.3436913
Lin YTsai CLin TChang YWu SLi GLi ZIdreos SSrivastava D(2021)Don't Look Back, Look into the FutureProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3452827(1156-1168)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3452827
Magalhaes AMonteiro JBrayner A(2021)Main Memory Database RecoveryACM Computing Surveys10.1145/344219754:2(1-36)Online publication date: 5-Mar-2021
https://dl.acm.org/doi/10.1145/3442197
Li LWang GWu GYuan YChen LLian X(2021)A Comparative Study of Consistent Snapshot Algorithms for Main-Memory Database SystemsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2019.293098733:2(316-330)Online publication date: 1-Feb-2021
https://dl.acm.org/doi/10.1109/TKDE.2019.2930987
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents