Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

PostCENN: postgreSQL with machine learning models for cardinality estimation

Published: 01 July 2021 Publication History

Abstract

In this demo, we present PostCENN, an enhanced PostgreSQL database system with an end-to-end integration of machine learning (ML) models for cardinality estimation. In general, cardinality estimation is a topic with a long history in the database community. While traditional models like histograms are extensively used, recent works mainly focus on developing new approaches using ML models. However, traditional as well as ML models have their own advantages and disadvantages. With PostCENN, we aim to combine both to maximize their potentials for cardinality estimation by introducing ML models as a novel means to increase the accuracy of the cardinality estimation for certain parts of the database schema. To achieve this, we integrate ML models as first class citizen in PostgreSQL with a well-defined end-to-end life cycle. This life cycle consists of creating ML models for different sub-parts of the database schema, triggering the training, using ML models within the query optimizer in a transparent way, and deleting ML models.

References

[1]
Walter Cai, Magdalena Balazinska, and Dan Suciu. 2019. Pessimistic cardinality estimation: Tighter upper bounds for intermediate join cardinalities. In SIGMOD. 18--35.
[2]
Andreas Kipf, Dimitri Vorona, Jonas Müller, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter Boncz, Thomas Neumann, and Alfons Kemper. 2019. Estimating cardinalities with deep sketches. In SIGMOD. 1937--1940.
[3]
Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, and Thomas Neumann. 2015. How good are query optimizers, really? PVLDB 9, 3, 204--215.
[4]
Henry Liu, Mingbin Xu, Ziting Yu, Vincent Corvinelli, and Calisto Zuzarte. 2015. Cardinality estimation using neural networks. In ICSE. 53--59.
[5]
Jennifer Ortiz, Magdalena Balazinska, Johannes Gehrke, and S Sathiya Keerthi. 2019. An empirical analysis of deep learning for cardinalityestimation. arXiv preprint arXiv:1905.06425 (2019).
[6]
Lucas Woltmann, Claudio Hartmann, Dirk Habich, and Wolfgang Lehner. 2020. Best of both worlds: combining traditional and machine learning models for cardinality estimation. In aiDM@SIGMOD. 1--8.
[7]
Lucas Woltmann, Claudio Hartmann, Dirk Habich, and Wolfgang Lehner. 2021. Aggregate-based Training Phase for ML-based Cardinality Estimation. BTW 2021 (2021).
[8]
Lucas Woltmann, Claudio Hartmann, Maik Thiele, Dirk Habich, and Wolfgang Lehner. 2019. Cardinality estimation with local deep learning models. In aiDM@SIGMOD. 1--8.

Cited By

View all
  • (2024)A Spark Optimizer for Adaptive, Fine-Grained Parameter TuningProceedings of the VLDB Endowment10.14778/3681954.368202117:11(3565-3579)Online publication date: 1-Jul-2024
  • (2023)Quantum Machine Learning for Join Order Optimization using Variational Quantum CircuitsProceedings of the International Workshop on Big Data in Emergent Distributed Environments10.1145/3579142.3594299(1-7)Online publication date: 18-Jun-2023
  • (2022)Fine-grained modeling and optimization for intelligent resource management in big data processingProceedings of the VLDB Endowment10.14778/3551793.355185515:11(3098-3111)Online publication date: 1-Jul-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment
Proceedings of the VLDB Endowment  Volume 14, Issue 12
July 2021
587 pages
ISSN:2150-8097
Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 July 2021
Published in PVLDB Volume 14, Issue 12

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)50
  • Downloads (Last 6 weeks)13
Reflects downloads up to 15 Oct 2024

Other Metrics

Citations

Cited By

View all
  • (2024)A Spark Optimizer for Adaptive, Fine-Grained Parameter TuningProceedings of the VLDB Endowment10.14778/3681954.368202117:11(3565-3579)Online publication date: 1-Jul-2024
  • (2023)Quantum Machine Learning for Join Order Optimization using Variational Quantum CircuitsProceedings of the International Workshop on Big Data in Emergent Distributed Environments10.1145/3579142.3594299(1-7)Online publication date: 18-Jun-2023
  • (2022)Fine-grained modeling and optimization for intelligent resource management in big data processingProceedings of the VLDB Endowment10.14778/3551793.355185515:11(3098-3111)Online publication date: 1-Jul-2022
  • (2022)Accurate summary-based cardinality estimation through the lens of cardinality estimation graphsProceedings of the VLDB Endowment10.14778/3529337.352933915:8(1533-1545)Online publication date: 22-Jun-2022

View Options

Get Access

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media