research-article

Open access

Citus: Distributed PostgreSQL for Data-Intensive Applications

Authors:

Sudhakar Sannakkayala,

Marco SlotAuthors Info & Claims

SIGMOD '21: Proceedings of the 2021 International Conference on Management of Data

Pages 2490 - 2502

https://doi.org/10.1145/3448016.3457551

Published: 18 June 2021 Publication History

Abstract

Citus is an open source distributed database engine for PostgreSQL that is implemented as an extension. Citus gives users the ability to distribute data, queries, and transactions in PostgreSQL across a cluster of PostgreSQL servers to handle the needs of data-intensive applications. The development of Citus has largely been driven by conversations with companies looking to scale PostgreSQL beyond a single server and their workload requirements. This paper describes the requirements of four common workload patterns and how Citus addresses those requirements. It also shares benchmark results demonstrating the performance and scalability of Citus in each of the workload patterns and describes how Microsoft uses Citus to address one of its most challenging data problems.

Supplementary Material

MP4 File (3448016.3457551.mp4)

Citus is an open source distributed database engine for PostgreSQL that is implemented as an extension. Citus gives users the ability to distribute data, queries, and transactions in PostgreSQL across a cluster of PostgreSQL servers to handle the needs of data-intensive applications. The development of Citus has largely been driven by conversations with companies looking to scale PostgreSQL beyond a single server and their workload requirements. This paper describes the requirements of four common workload patterns and how Citus addresses those requirements. It also shares benchmark results demonstrating the performance and scalability of Citus in each of the workload patterns and describes how Microsoft uses Citus to address one of its most challenging data problems.

Download
65.28 MB

References

[1]

2010. TPC Benchmark C: Standard Specification Revision 5.11. Technical Report. Transaction Processing Performance Council (TPC).

[2]

2018. TPC Benchmark H: Standard Specification Revision 2.18.0. Technical Report. Transaction Processing Performance Council (TPC)).

[3]

2020. Citus Open Source Repo. https://github.com/citusdata/citus

[4]

2020. GitHub Archive. https://www.gharchive.org/

[5]

2020. Greenplum Database. https://greenplum.org/

[6]

2020. HammerDB. https://www.hammerdb.com/

[7]

2020. Hyperscale (Citus) documentation on Citus metadata. https://docs. microsoft.com/en-us/azure/postgresql/reference-hyperscale-metadata

[8]

2020. PG Partitioning Manager. https://github.com/pgpartman/pg_partman

[9]

2020. pg_auto_failover: Postgres extension and service for automated failover and high-availability. https://github.com/citusdata/pg_auto_failover

[10]

2020. PgBouncer: lightweight connection pooler for PostgreSQL. https://www. pgbouncer.org/

[11]

2020. Timescale: PostgreSQL for time series. https://www.timescale.com/

[12]

2020. Vitess: A database clustering system for horizontal scaling of MySQL. https://vitess.io/

[13]

2020. YugabyteDB. https://www.yugabyte.com/

[14]

Mohamed Bakli, Mahmoud Sakr, and Esteban Zimanyi. 2019. Distributed moving object data management in MobilityDB. In Proceedings of the 8th ACMSIGSPATIAL International Workshop on Analytics for Big Geospatial Data. 1--10.

Digital Library

[15]

Stefano Ceri and Giuseppe Pelagatti. 1983. Correctness of query execution strategies in distributed databases. ACM Transactions on Database Systems (TODS) 8, 4 (1983), 577--607.

Digital Library

[16]

Brian F Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking cloud serving systems with YCSB. In Proceedings of the 1st ACM symposium on Cloud computing. 143--154.

Digital Library

[17]

James C Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, Jeffrey John Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, et al. 2013. Spanner: Google's globally distributed database. ACM Transactions on Computer Systems (TOCS) 31, 3 (2013), 1--22.

Digital Library

[18]

P. Corti, T.J. Kraft, S.V. Mather, and B. Park. 2014. PostGIS Cookbook. Packt Publishing. https://books.google.nl/books?id=zCaxAgAAQBAJ

[19]

DB-engines. 2020. PostgreSQL System Properties. https://db-engines.com/en/ system/PostgreSQL

[20]

Andres Freund. 2020. Analyzing the Limits of Connection Scalability in Postgres. https://techcommunity.microsoft.com/t5/azure-database-for-postgresql/ analyzing-the-limits-of-connection-scalability-in-postgres/ba-p/1757266

[21]

Anurag Gupta, Deepak Agarwal, Derek Tan, Jakub Kulesza, Rahul Pathak, Stefano Stefani, and Vidhya Srinivasan. 2015. Amazon redshift and the case for simpler data warehouses. In Proceedings of the 2015 ACM SIGMOD international conference on management of data. 1917--1923.

Digital Library

[22]

Craig Kerstiens. 2018. Introducing Landlord: per tenant stats in Postgres with Citus. https://www.citusdata.com/blog/2018/07/31/introducing-landlord-pertenant- stats/

[23]

Eamonn Maguire, Lukas Heinrich, and GraemeWatt. 2017. HEPData: a repository for high energy physics data. In J. Phys. Conf. Ser, Vol. 898. 52.

[24]

Michael Stonebraker and Lawrence A Rowe. 1986. The design of POSTGRES. ACM Sigmod Record 15, 2 (1986), 340--355.

Digital Library

[25]

Rebecca Taft, Irfan Sharif, Andrei Matei, Nathan VanBenschoten, Jordan Lewis, Tobias Grieger, Kai Niemi, Andy Woods, Anne Birzin, Raphael Poss, et al. 2020. CockroachDB: The Resilient Geo-Distributed SQL Database. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 1493--1509.

Digital Library

[26]

Wenbo Tao, Xinli Hou, Adam Sah, Leilani Battle, Remco Chang, and Michael Stonebraker. 2020. Kyrix-S: Authoring Scalable Scatterplot Visualizations of Big Data. arXiv preprint arXiv:2007.15904 (2020).

[27]

Alexandre Verbitski, Anurag Gupta, Debanjan Saha, Murali Brahmadesam, Kamal Gupta, Raman Mittal, Sailesh Krishnamurthy, Sandor Maurice, Tengiz Kharatishvili, and Xiaofeng Bao. 2017. Amazon aurora: Design considerations for high throughput cloud-native relational databases. In Proceedings of the 2017 ACM International Conference on Management of Data. 1041--1052.

Digital Library

Cited By

Kang RSong S(2024)Optimizing Time Series Queries with VersionsProceedings of the ACM on Management of Data10.1145/36549622:3(1-27)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654962
Klitgaard AJosefsen LMikkelsen MTorp K(2024)A Distributed Spatial Data Warehouse for AIS Data2024 25th IEEE International Conference on Mobile Data Management (MDM)10.1109/MDM61037.2024.00045(211-218)Online publication date: 24-Jun-2024
https://doi.org/10.1109/MDM61037.2024.00045
Zhang YYang XChen HLi FXu JZhou JWu XZhang Q(2024)Towards a Shared-Storage-Based Serverless Database Achieving Seamless Scale-Up and Read Scale-Out2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00384(5119-5131)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00384
Show More Cited By

Index Terms

Citus: Distributed PostgreSQL for Data-Intensive Applications
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Parallel and distributed DBMSs
        Relational parallel and distributed DBMSs

Recommendations

Multi-platform Performance Analysis for CRUD Operations in Relational Databases from Java Programs Using Hibernate
Big Data Intelligence and Computing
Abstract
A data set is a collection of information about something. Almost every application nowadays needs data persistence to maintain track of its status in the event of a problem. Databases are the primary component of an application for storing data ...
Progress in Database Search Strategies

Retrieval speed and precision ultimately determine the success of any database system. This article outlines the challenges posed by distributed and heterogeneous database systems, including those that store unstructured data, and surveys recent work. ...
Performing External Join Operator on PostgreSQL with Data Transfer Approach
HPCAsia '18: Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region

With the development of sensing devices, the size of data managed by human being has been rapidly increasing. To manage such huge data, relational database management system (RDBMS) plays a key role. RDBMS models the real world data as n-ary relational ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '21: Proceedings of the 2021 International Conference on Management of Data

June 2021

2969 pages

ISBN:9781450383431

DOI:10.1145/3448016

General Chairs:
Guoliang Li
Tsinghua University (China)
,
Zhanhuai Li
Northwestern Polytechnical University (China)
,
Program Chairs:
Stratos Idreos
Harvard University (USA)
,
Divesh Srivastava
AT&T (USA)

Copyright © 2021 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2021

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS '21

Sponsor:

SIGMOD

SIGMOD/PODS '21: International Conference on Management of Data

June 20 - 25, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
12,796
Total Downloads

Downloads (Last 12 months)3,375
Downloads (Last 6 weeks)345

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kang RSong S(2024)Optimizing Time Series Queries with VersionsProceedings of the ACM on Management of Data10.1145/36549622:3(1-27)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654962
Klitgaard AJosefsen LMikkelsen MTorp K(2024)A Distributed Spatial Data Warehouse for AIS Data2024 25th IEEE International Conference on Mobile Data Management (MDM)10.1109/MDM61037.2024.00045(211-218)Online publication date: 24-Jun-2024
https://doi.org/10.1109/MDM61037.2024.00045
Zhang YYang XChen HLi FXu JZhou JWu XZhang Q(2024)Towards a Shared-Storage-Based Serverless Database Achieving Seamless Scale-Up and Read Scale-Out2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00384(5119-5131)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00384
Zheng QZhao ZLu WYao CChen YPan ADu X(2024)Lion: Minimizing Distributed Transactions Through Adaptive Replica Provision2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00161(2012-2025)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00161
Gao WWen YZhang H(2024)An Optimization Method of Federated Database Join Query Based on Computational Push-Down2024 IEEE 2nd International Conference on Control, Electronics and Computer Technology (ICCECT)10.1109/ICCECT60629.2024.10545893(225-229)Online publication date: 26-Apr-2024
https://doi.org/10.1109/ICCECT60629.2024.10545893
Song HZhou WCui HPeng XLi F(2024)A survey on hybrid transactional and analytical processingThe VLDB Journal10.1007/s00778-024-00858-9Online publication date: 4-Jun-2024
https://doi.org/10.1007/s00778-024-00858-9
Gavriilidis HRose LZiegler JBeedkar KQuiané-Ruiz JMarkl V(2023)XDB in Action: Decentralized Cross-Database Query Processing for Black-Box DBMSesProceedings of the VLDB Endowment10.14778/3611540.361162516:12(4078-4081)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.14778/3611540.3611625
Yang ZXu QGao SYang CWang GZhao YKong FLiu HWang WXiao J(2023)OceanBase Paetica: A Hybrid Shared-Nothing/Shared-Everything Database for Supporting Single Machine and Distributed ClusterProceedings of the VLDB Endowment10.14778/3611540.361156016:12(3728-3740)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.14778/3611540.3611560
Lee KJo IAhn JLee HLee HSul WJung H(2023)Deploying Computational Storage for HTAP DBMSs Takes More Than Just Computation OffloadingProceedings of the VLDB Endowment10.14778/3583140.358316116:6(1480-1493)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.14778/3583140.3583161
Li YDeng GBai CYang JWang GZhang HBai JYuan HXu MWang S(2023)Demystifying the QoS and QoE of Edge-hosted Video Streaming Applications in the Wild with SNESetProceedings of the ACM on Management of Data10.1145/36267231:4(1-29)Online publication date: 12-Dec-2023
https://dl.acm.org/doi/10.1145/3626723
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents