Cited By
View all- Li GZhou XZhao X(2024)LLM for Data ManagementProceedings of the VLDB Endowment10.14778/3685800.368583817:12(4213-4216)Online publication date: 1-Aug-2024
Learned Query Optimizer: What is New and What is Next
Pages 561 - 569
Abstract
In recent times, learned query optimizer has becoming a hot research topic in learned databases. It serves as the most suitable experimental plots for utilizing numerous machine-learning techniques and exhibits its superiority with enough evidence. In this tutorial, we aim to provide a wide and deep review and analysis on this field, ranging from theory to practice. At first, we would categorize and introduce representative methods for each learned component in the query optimizer, as well as for the end-to-end learned query optimizer. Then, we describe some benchmark evaluations and prototype applications. Their results have exhibited the bright future of applying learned query optimizers in practice. Based on them, we describe a cutting edge system with step-by-step guidelines. It is a middleware proposed recently to reduce the difficulties of developing and deploying learned algorithms in databases. It would help researchers to iterate their work and make learned query optimizers truly applicable in production. Finally, we summarize and point out several future directions. We hope this tutorial could inspire and guide both researchers and engineers working on learned query optimizers, as well as other contexts in learned databases.
References
[1]
Christoph Anneser, Nesime Tatbul, David E. Cohen, Zhenggang Xu, Prithviraj Pandian, Nikolay Laptev, and Ryan Marcus. 2023. AutoSteer: Learned Query Optimization for Any SQL Database. Proc. VLDB Endow., Vol. 16, 12 (2023), 3515--3527. https://doi.org/10.14778/3611540.3611544
[2]
Jin Chen, Guanyu Ye, Yan Zhao, Shuncheng Liu, Liwei Deng, Xu Chen, Rui Zhou, and Kai Zheng. 2022. Efficient Join Order Selection Learning with Graph-based Representation. In KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022, Aidong Zhang and Huzefa Rangwala (Eds.). ACM, 97--107. https://doi.org/10.1145/3534678.3539303
[3]
Tianyi Chen, Jun Gao, Hedui Chen, and Yaofeng Tu. 2023 b. LOGER: A Learned Optimizer towards Generating Efficient and Robust Query Execution Plans. Proc. VLDB Endow., Vol. 16, 7 (2023), 1777--1789. https://www.vldb.org/pvldb/vol16/p1777-gao.pdf
[4]
Xu Chen, Haitian Chen, Zibo Liang, Shuncheng Liu, Jianhong Wang, Kai Zeng, Han Su, and Kai Zheng. 2023 a. LEON:: A New Framework for ML-Aided Query Optimization. Proc. VLDB Endow., Vol. 16, 9 (2023), 2261--2273. https://www.vldb.org/pvldb/vol16/p2261-chen.pdf
[5]
Xu Chen, Zhen Wang, Shuncheng Liu, Yaliang Li, Kai Zeng, Bolin Ding, Jingren Zhou, Han Su, and Kai Zheng. 2023 c. BASE: Bridging the Gap between Cost and Latency for Query Optimization. Proc. VLDB Endow., Vol. 16, 8 (2023), 1958--1966. https://www.vldb.org/pvldb/vol16/p1958-chen.pdf
[6]
Transaction Processing Performance Council(TPC). 2023 a. TPC-DS Vesion 2 and Version 3. http://www.tpc.org/tpcds/ (2023).
[7]
Transaction Processing Performance Council(TPC). 2023 b. TPC-H Vesion 2 and Version 3. http://www.tpc.org/tpch/ (2023).
[8]
Angjela Davitkova, Damjan Gjurovski, and Sebastian Michel. 2022. LMKG: Learned Models for Cardinality Estimation in Knowledge Graphs. In Proceedings of the 25th International Conference on Extending Database Technology, EDBT 2022, Edinburgh, UK, March 29 - April 1, 2022. OpenProceedings.org, 2:169--2:182. https://doi.org/10.48786/edbt.2022.07
[9]
Anshuman Dutt, Chi Wang, Vivek R. Narasayya, and Surajit Chaudhuri. 2020. Efficiently Approximating Selectivity Functions using Low Overhead Regression Models. Proc. VLDB Endow., Vol. 13, 11 (2020), 2215--2228. http://www.vldb.org/pvldb/vol13/p2215-dutt.pdf
[10]
Anshuman Dutt, Chi Wang, Azade Nazi, Srikanth Kandula, Vivek R. Narasayya, and Surajit Chaudhuri. 2019. Selectivity Estimation for Range Predicates using Lightweight Models. Proc. VLDB Endow., Vol. 12, 9 (2019), 1044--1057. https://doi.org/10.14778/3329772.3329780
[11]
Goetz Graefe. 1995. The cascades framework for query optimization. IEEE Data Eng. Bull., Vol. 18, 3 (1995), 19--29. http://sites.computer.org/debull/95SEP-CD.pdf
[12]
Yuxing Han, Ziniu Wu, Peizhi Wu, Rong Zhu, Jingyi Yang, Liang Wei Tan, Kai Zeng, Gao Cong, Yanzhao Qin, Andreas Pfadler, Zhengping Qian, Jingren Zhou, Jiangneng Li, and Bin Cui. 2021. Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation. Proc. VLDB Endow., Vol. 15, 4 (2021), 752--765. https://doi.org/10.14778/3503585.3503586
[13]
Rojeh Hayek and Oded Shmueli. 2020. Improved Cardinality Estimation by Learning Queries Containment Rates. In Proceedings of the 23rd International Conference on Extending Database Technology, EDBT 2020, Copenhagen, Denmark, March 30 - April 02, 2020. OpenProceedings.org, 157--168. https://doi.org/10.5441/002/edbt.2020.15
[14]
Max Heimel, Martin Kiefer, and Volker Markl. 2015. Self-Tuning, GPU-Accelerated Kernel Density Models for Multidimensional Selectivity Estimation. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, Melbourne, Victoria, Australia, May 31 - June 4, 2015. ACM, 1477--1492. https://doi.org/10.1145/2723372.2749438
[15]
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John P. Agapiou, Joel Z. Leibo, and Audrunas Gruslys. 2018. Deep Q-learning From Demonstrations. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2--7, 2019, Sheila A. McIlraith and Kilian Q. Weinberger (Eds.). AAAI Press, 3223--3230. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16976
[16]
Benjamin Hilprecht and Carsten Binnig. 2022. Zero-Shot Cost Models for Out-of-the-box Learned Cost Prediction. Proc. VLDB Endow., Vol. 15, 11 (2022), 2361--2374. https://www.vldb.org/pvldb/vol15/p2361-hilprecht.pdf
[17]
Benjamin Hilprecht, Andreas Schmidt, Moritz Kulessa, Alejandro Molina, Kristian Kersting, and Carsten Binnig. 2020. DeepDB: Learn from Data, not from Queries! Proc. VLDB Endow., Vol. 13, 7 (2020), 992--1005. https://doi.org/10.14778/3384345.3384349
[18]
H. M. Sajjad Hossain, Marc T. Friedman, Hiren Patel, Shi Qiao, Soundar Srinivasan, Markus Weimer, Remmelt Ammerlaan, Lucas Rosenblatt, Gilbert Antonius, Peter Orenberg, Vijay Ramani, Abhishek Roy, Irene Shaffer, and Alekh Jindal. 2021. PerfGuard: Deploying ML-for-Systems without Performance Regressions, Almost! Proc. VLDB Endow., Vol. 14, 13 (2021), 3362--3375. http://www.vldb.org/pvldb/vol14/p3362-hossain.pdf
[19]
Xiao Hu, Yuxi Liu, Haibo Xiu, Pankaj K. Agarwal, Debmalya Panigrahi, Sudeepa Roy, and Jun Yang. 2022. Selectivity Functions of Range Queries are Learnable. In SIGMOD '22: International Conference on Management of Data, Philadelphia, PA, USA, June 12 - 17, 2022. ACM, 959--972. https://doi.org/10.1145/3514221.3517896
[20]
Johan Kok Zhi Kang, Gaurav, Sien Yi Tan, Feng Cheng, Shixuan Sun, and Bingsheng He. 2021. Efficient Deep Learning Pipelines for Accurate Cost Estimations Over Large Scale Query Workload. In SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20--25, 2021, Guoliang Li, Zhanhuai Li, Stratos Idreos, and Divesh Srivastava (Eds.). ACM, 1014--1022. https://doi.org/10.1145/3448016.3457546
[21]
Martin Kiefer, Max Heimel, Sebastian Breß, and Volker Markl. 2017. Estimating Join Selectivities using Bandwidth-Optimized Kernel Density Models. Proc. VLDB Endow., Vol. 10, 13 (2017), 2085--2096. https://doi.org/10.14778/3151106.3151112
[22]
Kyoungmin Kim, Jisung Jung, In Seo, Wook-Shin Han, Kangwoo Choi, and Jaehyok Chong. 2022. Learned Cardinality Estimation: An In-depth Study. In SIGMOD '22: International Conference on Management of Data, Philadelphia, PA, USA, June 12 - 17, 2022. ACM, 1214--1227. https://doi.org/10.1145/3514221.3526154
[23]
Andreas Kipf, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter A. Boncz, and Alfons Kemper. 2019. Learned Cardinalities: Estimating Correlated Joins with Deep Learning. In 9th Biennial Conference on Innovative Data Systems Research, CIDR 2019, Asilomar, CA, USA, January 13--16, 2019, Online Proceedings. www.cidrdb.org. http://cidrdb.org/cidr2019/papers/p101-kipf-cidr19.pdf
[24]
Sanjay Krishnan, Zongheng Yang, Ken Goldberg, Joseph M. Hellerstein, and Ion Stoica. 2018. Learning to Optimize Join Queries With Deep Reinforcement Learning. CoRR, Vol. abs/1808.03196 (2018). showeprint[arXiv]1808.03196 http://arxiv.org/abs/1808.03196
[25]
Meghdad Kurmanji and Peter Triantafillou. 2023. Detect, Distill and Update: Learned DB Systems Facing Out of Distribution Data. Proc. ACM Manag. Data, Vol. 1, 1 (2023), 33:1--33:27. https://doi.org/10.1145/3588713
[26]
Suyong Kwon, Woohwan Jung, and Kyuseok Shim. 2022. Cardinality Estimation of Approximate Substring Queries using Deep Learning. Proc. VLDB Endow., Vol. 15, 11 (2022), 3145--3157. https://www.vldb.org/pvldb/vol15/p3145-jung.pdf
[27]
Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter A. Boncz, Alfons Kemper, and Thomas Neumann. 2015. How Good Are Query Optimizers, Really? Proc. VLDB Endow., Vol. 9, 3 (2015), 204--215. http://www.vldb.org/pvldb/vol9/p204-leis.pdf
[28]
Beibin Li, Yao Lu, and Srikanth Kandula. 2022a. Warper: Efficiently Adapting Learned Cardinality Estimators to Data and Workload Drifts. In SIGMOD '22: International Conference on Management of Data, Philadelphia, PA, USA, June 12 - 17, 2022. ACM, 1920--1933. https://doi.org/10.1145/3514221.3526179
[29]
Beibin Li, Yao Lu, and Srikanth Kandula. 2022b. Warper: Efficiently Adapting Learned Cardinality Estimators to Data and Workload Drifts. In SIGMOD '22: International Conference on Management of Data, Philadelphia, PA, USA, June 12 - 17, 2022, Zachary G. Ives, Angela Bonifati, and Amr El Abbadi (Eds.). ACM, 1920--1933. https://doi.org/10.1145/3514221.3526179
[30]
Pengfei Li, Wenqing Wei, Rong Zhu, Bolin Ding, Jingren Zhou, and Hua Lu. 2023. ALECE: An Attention-based Learned Cardinality Estimator for SPJ Queries on Dynamic Workloads. Proc. VLDB Endow., Vol. 17, 2 (2023), 197--210. https://www.vldb.org/pvldb/vol17/p197-li.pdf
[31]
Yan Li, Liwei Wang, Sheng Wang, Yuan Sun, and Zhiyong Peng. 2022c. A Resource-Aware Deep Cost Model for Big Data Query Processing. In 38th IEEE International Conference on Data Engineering, ICDE 2022, Kuala Lumpur, Malaysia, May 9--12, 2022. IEEE, 885--897. https://doi.org/10.1109/ICDE53745.2022.00071
[32]
Henry Liu, Mingbin Xu, Ziting Yu, Vincent Corvinelli, and Calisto Zuzarte. 2015. Cardinality estimation using neural networks. In Proceedings of 25th Annual International Conference on Computer Science and Software Engineering, CASCON 2015, Markham, Ontario, Canada, 2--4 November, 2015. IBM / ACM, 53--59. http://dl.acm.org/citation.cfm?id=2886453
[33]
Jie Liu, Wenqian Dong, Dong Li, and Qingqing Zhou. 2021. Fauce: Fast and Accurate Deep Ensembles with Uncertainty for Cardinality Estimation. Proc. VLDB Endow., Vol. 14, 11 (2021), 1950--1963. https://doi.org/10.14778/3476249.3476254
[34]
Shuncheng Liu, Xu Chen, Yan Zhao, Jin Chen, Rui Zhou, and Kai Zheng. 2022. Efficient Learning with Pseudo Labels for Query Cost Estimation. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17--21, 2022, Mohammad Al Hasan and Li Xiong (Eds.). ACM, 1309--1318. https://doi.org/10.1145/3511808.3557305
[35]
Yao Lu, Srikanth Kandula, Arnd Christian Kö nig, and Surajit Chaudhuri. 2021. Pre-training Summarization Models of Structured Datasets for Cardinality Estimation. Proc. VLDB Endow., Vol. 15, 3 (2021), 414--426. https://doi.org/10.14778/3494124.3494127
[36]
Tanu Malik, Randal C. Burns, and Nitesh V. Chawla. 2007. A Black-Box Approach to Query Cardinality Estimation. In Third Biennial Conference on Innovative Data Systems Research, CIDR 2007, Asilomar, CA, USA, January 7--10, 2007, Online Proceedings. www.cidrdb.org, 56--67. http://cidrdb.org/cidr2007/papers/cidr07p06.pdf
[37]
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Alizadeh, and Tim Kraska. 2021. Bao: Making Learned Query Optimization Practical. In SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20--25, 2021. ACM, 1275--1288. https://doi.org/10.1145/3448016.3452838
[38]
Ryan C. Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul. 2019. Neo: A Learned Query Optimizer. Proc. VLDB Endow., Vol. 12, 11 (2019), 1705--1718. https://doi.org/10.14778/3342263.3342644
[39]
Ryan C. Marcus and Olga Papaemmanouil. 2019. Plan-Structured Deep Neural Network Models for Query Performance Prediction. Proc. VLDB Endow., Vol. 12, 11 (2019), 1733--1746. https://doi.org/10.14778/3342263.3342646
[40]
Zizhong Meng, Peizhi Wu, Gao Cong, Rong Zhu, and Shuai Ma. 2022. Unsupervised Selectivity Estimation by Integrating Gaussian Mixture Models and an Autoregressive Model. In Proceedings of the 25th International Conference on Extending Database Technology, EDBT 2022, Edinburgh, UK, March 29 - April 1, 2022. OpenProceedings.org, 2:247--2:259. https://doi.org/10.48786/edbt.2022.13
[41]
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30.
[42]
Magnus Mü ller, Lucas Woltmann, and Wolfgang Lehner. 2023. Enhanced Featurization of Queries with Mixed Combinations of Predicates for ML-based Cardinality Estimation. In Proceedings 26th International Conference on Extending Database Technology, EDBT 2023, Ioannina, Greece, March 28--31, 2023. OpenProceedings.org, 273--284. https://doi.org/10.48786/edbt.2023.22
[43]
Parimarjan Negi, Matteo Interlandi, Ryan Marcus, Mohammad Alizadeh, Tim Kraska, Marc T. Friedman, and Alekh Jindal. 2021a. Steering Query Optimizers: A Practical Take on Big Data Workloads. In SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20--25, 2021, Guoliang Li, Zhanhuai Li, Stratos Idreos, and Divesh Srivastava (Eds.). ACM, 2557--2569. https://doi.org/10.1145/3448016.3457568
[44]
Parimarjan Negi, Ryan C. Marcus, Andreas Kipf, Hongzi Mao, Nesime Tatbul, Tim Kraska, and Mohammad Alizadeh. 2021b. Flow-Loss: Learning Cardinality Estimates That Matter. Proc. VLDB Endow., Vol. 14, 11 (2021), 2019--2032. https://doi.org/10.14778/3476249.3476259
[45]
Parimarjan Negi, Ziniu Wu, Andreas Kipf, Nesime Tatbul, Ryan Marcus, Sam Madden, Tim Kraska, and Mohammad Alizadeh. 2023. Robust Query Driven Cardinality Estimation under Changing Workloads. Proc. VLDB Endow., Vol. 16, 6 (2023), 1520--1533. https://www.vldb.org/pvldb/vol16/p1520-negi.pdf
[46]
Patrick E. O'Neil, Elizabeth J. O'Neil, Xuedong Chen, and Stephen Revilak. 2009. The Star Schema Benchmark and Augmented Fact Table Indexing. In Performance Evaluation and Benchmarking, First TPC Technology Conference, TPCTC 2009, Lyon, France, August 24--28, 2009, Revised Selected Papers (Lecture Notes in Computer Science, Vol. 5895), Raghunath Othayoth Nambiar and Meikel Poess (Eds.). Springer, 237--252. https://doi.org/10.1007/978--3--642--10424--4_17
[47]
Yongjoo Park, Shucheng Zhong, and Barzan Mozafari. 2020. QuickSel: Quick Selectivity Learning with Mixture Models. In Proceedings of the 2020 International Conference on Management of Data, SIGMOD Conference 2020, online conference [Portland, OR, USA], June 14--19, 2020. ACM, 1017--1033. https://doi.org/10.1145/3318464.3389727
[48]
Suraj Shetiya, Saravanan Thirumuruganathan, Nick Koudas, and Gautam Das. 2020. Astrid: Accurate Selectivity Estimation for String Predicates using Deep Learning. Proc. VLDB Endow., Vol. 14, 4 (2020), 471--484. https://doi.org/10.14778/3436905.3436907
[49]
Tarique Siddiqui, Alekh Jindal, Shi Qiao, Hiren Patel, and Wangchao Le. 2020a. Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings. In Proceedings of the 2020 International Conference on Management of Data, SIGMOD Conference 2020, online conference [Portland, OR, USA], June 14--19, 2020, David Maier, Rachel Pottinger, AnHai Doan, Wang-Chiew Tan, Abdussalam Alawini, and Hung Q. Ngo (Eds.). ACM, 99--113. https://doi.org/10.1145/3318464.3380584
[50]
Tarique Siddiqui, Alekh Jindal, Shi Qiao, Hiren Patel, and Wangchao Le. 2020b. Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings. In Proceedings of the 2020 International Conference on Management of Data, SIGMOD Conference 2020, online conference [Portland, OR, USA], June 14--19, 2020, David Maier, Rachel Pottinger, AnHai Doan, Wang-Chiew Tan, Abdussalam Alawini, and Hung Q. Ngo (Eds.). ACM, 99--113. https://doi.org/10.1145/3318464.3380584
[51]
Ji Sun and Guoliang Li. 2019. An End-to-End Learning-based Cost Estimator. Proc. VLDB Endow., Vol. 13, 3 (2019), 307--319. https://doi.org/10.14778/3368289.3368296
[52]
Ji Sun, Guoliang Li, and Nan Tang. 2021a. Learned Cardinality Estimation for Similarity Queries. In SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20--25, 2021. ACM, 1745--1757. https://doi.org/10.1145/3448016.3452790
[53]
Ji Sun, Jintao Zhang, Zhaoyan Sun, Guoliang Li, and Nan Tang. 2021b. Learned Cardinality Estimation: A Design Space Exploration and A Comparative Evaluation. Proc. VLDB Endow., Vol. 15, 1 (2021), 85--97. https://doi.org/10.14778/3485450.3485459
[54]
Luming Sun. 2023. Papers for database systems powered by artificial intelligence (machine learning for database). https://github.com/LumingSun/ML4DB-paper-list (2023).
[55]
Saravanan Thirumuruganathan, Suraj Shetiya, Nick Koudas, and Gautam Das. 2022. Prediction Intervals for Learned Cardinality Estimation: An Experimental Evaluation. In 38th IEEE International Conference on Data Engineering, ICDE 2022, Kuala Lumpur, Malaysia, May 9--12, 2022. IEEE, 3051--3064. https://doi.org/10.1109/ICDE53745.2022.00274
[56]
Immanuel Trummer, Junxiong Wang, Deepak Maram, Samuel Moseley, Saehan Jo, and Joseph Antonakakis. 2019. SkinnerDB: Regret-Bounded Query Evaluation via Reinforcement Learning. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD Conference 2019, Amsterdam, The Netherlands, June 30 - July 5, 2019, Peter A. Boncz, Stefan Manegold, Anastasia Ailamaki, Amol Deshpande, and Tim Kraska (Eds.). ACM, 1153--1170. https://doi.org/10.1145/3299869.3300088
[57]
Kostas Tzoumas, Amol Deshpande, and Christian S. Jensen. 2011. Lightweight Graphical Models for Selectivity Estimation Without Independence Assumptions. Proc. VLDB Endow., Vol. 4, 11 (2011), 852--863. http://www.vldb.org/pvldb/vol4/p852-tzoumas.pdf
[58]
Kostas Tzoumas, Timos Sellis, and Christian S Jensen. 2008. A reinforcement learning approach for adaptive query processing. History (2008).
[59]
Fang Wang, Xiao Yan, Man Lung Yiu, Shuai LI, Zunyao Mao, and Bo Tang. 2023. Speeding Up End-to-end Query Execution via Learning-based Progressive Cardinality Estimation. Proc. ACM Manag. Data, Vol. 1, 1 (2023), 28:1--28:25. https://doi.org/10.1145/3588708
[60]
Jiayi Wang, Chengliang Chai, Jiabin Liu, and Guoliang Li. 2021a. FACE: A Normalizing Flow based Cardinality Estimator. Proc. VLDB Endow., Vol. 15, 1 (2021), 72--84. https://doi.org/10.14778/3485450.3485458
[61]
Xiaoying Wang, Changbo Qu, Weiyuan Wu, Jiannan Wang, and Qingqing Zhou. 2021b. Are We Ready For Learned Cardinality Estimation? Proc. VLDB Endow., Vol. 14, 9 (2021), 1640--1654. https://doi.org/10.14778/3461535.3461552
[62]
Lianggui Weng, Rong Zhu, Di Wu, Bolin Ding, Bolong Zheng, and Jingren Zhou. 2024. Eraser: Eliminating Performance Regression on Learned Query Optimizer. Proc. VLDB Endow. (2024).
[63]
Peizhi Wu and Gao Cong. 2021. A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation. In SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20--25, 2021. ACM, 2009--2022. https://doi.org/10.1145/3448016.3452830
[64]
Ziniu Wu, Parimarjan Negi, Mohammad Alizadeh, Tim Kraska, and Samuel Madden. 2023. FactorJoin: A New Cardinality Estimation Framework for Join Queries. Proc. ACM Manag. Data, Vol. 1, 1 (2023), 41:1--41:27. https://doi.org/10.1145/3588721
[65]
Ziniu Wu and Amir Shaikhha. 2020. BayesCard: A Unified Bayesian Framework for Cardinality Estimation. CoRR, Vol. abs/2012.14743 (2020). showeprint[arXiv]2012.14743 https://arxiv.org/abs/2012.14743
[66]
Ziniu Wu, Pei Yu, Peilun Yang, Rong Zhu, Yuxing Han, Yaliang Li, Defu Lian, Kai Zeng, and Jingren Zhou. 2022. A Unified Transferable Model for ML-Enhanced DBMS. In 12th Conference on Innovative Data Systems Research, CIDR 2022, Chaminade, CA, USA, January 9--12, 2022. www.cidrdb.org. https://www.cidrdb.org/cidr2022/papers/p6-wu.pdf
[67]
Ziniu Wu, Rong Zhu, Andreas Pfadler, Yuxing Han, Jiangneng Li, Zhengping Qian, Kai Zeng, and Jingren Zhou. 2020. FSPN: A New Class of Probabilistic Graphical Model. CoRR, Vol. abs/2011.09020 (2020). https://arxiv.org/abs/2011.09020
[68]
Linglin Yang, Lei Yang, Yue Pang, and Lei Zou. 2022b. gCBO: A Cost-based Optimizer for Graph Databases. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17--21, 2022, Mohammad Al Hasan and Li Xiong (Eds.). ACM, 5054--5058. https://doi.org/10.1145/3511808.3557197
[69]
Zongheng Yang, Wei-Lin Chiang, Sifei Luan, Gautam Mittal, Michael Luo, and Ion Stoica. 2022a. Balsa: Learning a Query Optimizer Without Expert Demonstrations. In SIGMOD '22: International Conference on Management of Data, Philadelphia, PA, USA, June 12 - 17, 2022, Zachary G. Ives, Angela Bonifati, and Amr El Abbadi (Eds.). ACM, 931--944. https://doi.org/10.1145/3514221.3517885
[70]
Zongheng Yang, Amog Kamsetty, Sifei Luan, Eric Liang, Yan Duan, Xi Chen, and Ion Stoica. 2020. NeuroCard: One Cardinality Estimator for All Tables. Proc. VLDB Endow., Vol. 14, 1 (2020), 61--73. https://doi.org/10.14778/3421424.3421432
[71]
Zongheng Yang, Eric Liang, Amog Kamsetty, Chenggang Wu, Yan Duan, Xi Chen, Pieter Abbeel, Joseph M. Hellerstein, Sanjay Krishnan, and Ion Stoica. 2019. Deep Unsupervised Cardinality Estimation. Proc. VLDB Endow., Vol. 13, 3 (2019), 279--292. https://doi.org/10.14778/3368289.3368294
[72]
Xiang Yu, Chengliang Chai, Guoliang Li, and Jiabin Liu. 2022. Cost-based or Learning-based? A Hybrid Query Optimizer for Query Plan Selection. Proc. VLDB Endow., Vol. 15, 13 (2022), 3924--3936. https://www.vldb.org/pvldb/vol15/p3924-li.pdf
[73]
Xiang Yu, Guoliang Li, Chengliang Chai, and Nan Tang. 2020. Reinforcement Learning with Tree-LSTM for Join Order Selection. In 36th IEEE International Conference on Data Engineering, ICDE 2020, Dallas, TX, USA, April 20--24, 2020. IEEE, 1297--1308. https://doi.org/10.1109/ICDE48307.2020.00116
[74]
Jintao Zhang, Chao Zhang, Guoliang Li, and Chengliang Chai. 2023. AutoCE: An Accurate and Efficient Model Advisor for Learned Cardinality Estimation. In 39th IEEE International Conference on Data Engineering, ICDE 2023, Anaheim, California, USA, April 3--7, 2023. IEEE.
[75]
Kangfei Zhao, Jeffrey Xu Yu, Zongyan He, Rui Li, and Hao Zhang. 2022b. Lightweight and Accurate Cardinality Estimation by Neural Network Gaussian Process. In SIGMOD '22: International Conference on Management of Data, Philadelphia, PA, USA, June 12 - 17, 2022. ACM, 973--987. https://doi.org/10.1145/3514221.3526156
[76]
Yue Zhao, Gao Cong, Jiachen Shi, and Chunyan Miao. 2022a. QueryFormer: A Tree Transformer Model for Query Plan Representation. Proc. VLDB Endow., Vol. 15, 8 (2022), 1658--1670. https://www.vldb.org/pvldb/vol15/p1658-zhao.pdf
[77]
Xuanhe Zhou, Chengliang Chai, Guoliang Li, and Ji Sun. 2022. Database Meets Artificial Intelligence: A Survey. IEEE Trans. Knowl. Data Eng., Vol. 34, 3 (2022), 1096--1116. https://doi.org/10.1109/TKDE.2020.2994641
[78]
Xuanhe Zhou, Ji Sun, Guoliang Li, and Jianhua Feng. 2020. Query Performance Prediction for Concurrent Queries using Graph Embedding. Proc. VLDB Endow., Vol. 13, 9 (2020), 1416--1428. https://doi.org/10.14778/3397230.3397238
[79]
Rong Zhu, Wei Chen, Bolin Ding, Xingguang Chen, Andreas Pfadler, Ziniu Wu, and Jingren Zhou. 2023. Lero: A Learning-to-Rank Query Optimizer. Proc. VLDB Endow., Vol. 16, 6 (2023), 1466--1479. https://www.vldb.org/pvldb/vol16/p1466-zhu.pdf
[80]
Rong Zhu, Lianggui Weng, Wenqing Wei, Di Wu, Jiazhen Peng, Yifan Wang, Bolin Ding, Defu Lian, Bolong Zheng, and Jingren Zhou. 2024. PilotScope: Steering Databases with Machine Learning Drivers. Proc. VLDB Endow. (2024).
[81]
Rong Zhu, Ziniu Wu, Yuxing Han, Kai Zeng, Andreas Pfadler, Zhengping Qian, Jingren Zhou, and Bin Cui. 2021a. FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation. Proc. VLDB Endow., Vol. 14, 9 (2021), 1489--1502. https://doi.org/10.14778/3461535.3461539
[82]
Rong Zhu, Tianjing Zeng, Andreas Pfadler, Wei Chen, Bolin Ding, and Jingren Zhou. 2021b. Glue: Adaptively Merging Single Table Cardinality to Estimate Join Query Size. CoRR, Vol. abs/2112.03458 (2021). https://arxiv.org/abs/2112.03458 io
Index Terms
- Learned Query Optimizer: What is New and What is Next
Recommendations
Bao: Making Learned Query Optimization Practical
SIGMOD '21: Proceedings of the 2021 International Conference on Management of DataRecent efforts applying machine learning techniques to query optimization have shown few practical gains due to substantive training overhead, inability to adapt to changes, and poor tail performance. Motivated by these difficulties, we introduce Bao (...
A Learned Query Optimizer for Spatial Join
SIGSPATIAL '21: Proceedings of the 29th International Conference on Advances in Geographic Information SystemsThe importance and complexity of spatial join resulted in many join algorithms, some of which run on big-data platforms such as Hadoop and Spark. This paper proposes the first machine-learning-based query optimizer for spatial join operation which can ...
DeepO: A Learned Query Optimizer
SIGMOD '22: Proceedings of the 2022 International Conference on Management of DataQuery optimization is crucial for the query performance of database systems. Despite decades of efforts from both research and industrial communities, query optimization remains one of the most challenging problems. Thanks to the advances in artificial ...
Comments
Information & Contributors
Information
Published In
June 2024
694 pages
ISBN:9798400704222
DOI:10.1145/3626246
- General Chairs:
- Pablo Barcelo,
- Nayat Sanchez-Pi,
- Program Chairs:
- Alexandra Meliou,
- S. Sudarshan
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 June 2024
Check for updates
Author Tags
Qualifiers
- Tutorial
Conference
SIGMOD/PODS '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 785 of 4,003 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 442Total Downloads
- Downloads (Last 12 months)442
- Downloads (Last 6 weeks)105
Reflects downloads up to 24 Dec 2024
Other Metrics
Citations
Cited By
View all- Li GZhou XZhao X(2024)LLM for Data ManagementProceedings of the VLDB Endowment10.14778/3685800.368583817:12(4213-4216)Online publication date: 1-Aug-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in