research-article

The Dynamic Yannakakis Algorithm: Compact and Efficient Query Processing Under Updates

Authors:

Muhammad Idris,

Stijn VansummerenAuthors Info & Claims

SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data

Pages 1259 - 1274

https://doi.org/10.1145/3035918.3064027

Published: 09 May 2017 Publication History

Abstract

Modern computing tasks such as real-time analytics require refresh of query results under high update rates. Incremental View Maintenance (IVM) approaches this problem by materializing results in order to avoid recomputation. IVM naturally induces a trade-off between the space needed to maintain the materialized results and the time used to process updates. In this paper, we show that the full materialization of results is a barrier for more general optimization strategies. In particular, we present a new approach for evaluating queries under updates. Instead of the materialization of results, we require a data structure that allows: (1) linear time maintenance under updates, (2) constant-delay enumeration of the output, (3) constant-time lookups in the output, while (4) using only linear space in the size of the database. We call such a structure a Dynamic Constant-delay Linear Representation (DCLR) for the query. We show that DYN, a dynamic version of the Yannakakis algorithm, yields DCLRs for the class of free-connex acyclic CQs. We show that this is optimal in the sense that no DCLR can exist for CQs that are not free-connex acyclic. Moreover, we identify a sub-class of queries for which DYN features constant-time update per tuple and show that this class is maximal. Finally, using the TPC-H and TPC-DS benchmarks, we experimentally compare DYN and a higher-order IVM (HIVM) engine. Our approach is not only more efficient in terms of memory consumption (as expected), but is also consistently faster in processing updates.

References

[1]

S. Abiteboul, R. Hull, and V. Vianu. Foundations of databases. Addison-Wesley, 1995.

Digital Library

[2]

M. Abo Khamis, H. Q. Ngo, and A. Rudra. FAQ: Questions asked frequently. In Proc. of PODS, pages 13--28, 2016.

Digital Library

[3]

M. E. Adiba and B. G. Lindsay. Database snapshots. In Proc. of VLDB 1980, pages 86--91, 1980.

Digital Library

[4]

S. M. Aji and R. J. McEliece. The generalized distributive law. IEEE Trans. Information Theory, 46(2):325--343, 2006.

Digital Library

[5]

G. Bagan, A. Durand, and E. Grandjean. On acyclic conjunctive queries and constant delay enumeration. In Proc. of CSL, pages 208--222, 2007.

Digital Library

[6]

N. Bakibayev, T. Kočiský, D. Olteanu, and J. Závodný. Aggregation and ordering in factorised databases. Proc. of VLDB, 6(14):1990--2001, 2013.

Digital Library

[7]

C. Berkholz, J. Keppeler, and N. Schweikardt. Answering conjunctive queries under updates. In Proc. of PODS, 2017. To appear.

Digital Library

[8]

J. A. Blakeley, N. Coburn, and P.-V. Larson. Updating derived relations: Detecting irrelevant and autonomously computable updates. ACM TODS, (3):369--400, 1989.

Digital Library

[9]

J. A. Blakeley, P.-A. Larson, and F. W. Tompa. Efficiently updating materialized views. In Proc. of SIGMOD, pages 61--71, 1986.

Digital Library

[10]

J. Brault-Baron. De la pertinence de l'énumération: complexité en logiques. PhD thesis, Université de Caen, 2013.

[11]

O. P. Buneman and E. K. Clemons. Efficiently monitoring relational databases. ACM TODS, (3):368--382, 1979.

Digital Library

[12]

S. Ceri and J. Widom. Deriving production rules for incremental view maintenance. In Proc. of VLDB, pages 577--589, 1991.

Digital Library

[13]

R. Chirkova and J. Yang. Materialized Views. Now Publishers Inc., Hanover, MA, USA, 2012.

Digital Library

[14]

T. Cormen. Introduction to Algorithms, 3rd Edition:. MIT Press, 2009.

Digital Library

[15]

G. Cugola and A. Margara. Processing flows of information: From data stream to complex event processing. ACM CSUR, 44(3):15:1--15:62, 2012.

Digital Library

[16]

N. Dalvi and D. Suciu. The dichotomy of probabilistic inference for unions of conjunctive queries. J. ACM, 59(6):30:1--30:87, 2013.

Digital Library

[17]

R. Fink and D. Olteanu. Dichotomies for queries with negation in probabilistic databases. ACM TODS, 41(1):4:1--4:47, 2016.

Digital Library

[18]

G. Gottlob, M. Grohe, n. Musliu, M. Samer, and F. Scarcello. Hypertree decompositions: Structure, algorithms, and applications. In Proc. of WG, pages 1--15, 2005.

Digital Library

[19]

G. Gottlob, N. Leone, and F. Scarcello. Robbers, marshals, and guards: game theoretic and logical characterizations of hypertree width. J. Comput. Syst. Sci., 66(4):775--808, 2003.

Digital Library

[20]

A. Gupta, I. S. Mumick, and V. S. Subrahmanian. Maintaining views incrementally. In Proc. of SIGMOD, pages 157--166, 1993.

Digital Library

[21]

H. Gupta and I. S. Mumick. Incremental maintenance of aggregate and outerjoin expressions. Information Systems, (6):435--464, 2006.

Digital Library

[22]

M. Henzinger, S. Krinninger, D. Nanongkai, and T. Saranurak. Unifying and strengthening hardness for dynamic problems via the online matrix-vector multiplication conjecture. In Proc. of STOC, pages 21--30, 2015.

Digital Library

[23]

M. R. Joglekar, R. Puttagunta, and C. Ré. Ajar: Aggregations and joins over annotated relations. In Proc. of PODS, pages 91--106, 2016.

Digital Library

[24]

A. Kawaguchi, D. Lieuwen, I. Mumick, and K. Ross. Implementing incremental view maintenance in nested data models. In Proc. of DBPL, pages 202--221, 1997.

Digital Library

[25]

C. Koch. Incremental query evaluation in a ring of databases. In Proc. of PODS, pages 87--98, 2010.

Digital Library

[26]

C. Koch, Y. Ahmad, O. Kennedy, M. Nikolic, A. Nötzli, D. Lupei, and A. Shaikhha. Dbtoaster: higher-order delta processing for dynamic, frequently fresh views. VLDB Journal, pages 253--278, 2014.

Digital Library

[27]

P. Koutris and D. Suciu. Parallel evaluation of conjunctive queries. In Proc. of PODS, pages 223--234, 2011.

Digital Library

[28]

I. S. Mumick, D. Quass, and B. S. Mumick. Maintenance of data cubes and summary tables in a warehouse. SIGMOD Records, 26(2):100--111, 1997.

Digital Library

[29]

H. Q. Ngo, C. Ré, and A. Rudra. Skew strikes back: New developments in the theory of join algorithms. SIGMOD Records, 42(4):5--16, 2014.

Digital Library

[30]

M. Nikolic, M. Dashti, and C. Koch. How to win a hot dog eating contest: Distributed incremental view maintenance with batch updates. In Proc.of SIGMOD, pages 511--526, 2016.

Digital Library

[31]

D. Olteanu and J. Závodný. Size bounds for factorised representations of query results. ACM TODS, 40(1):2:1--2:44, 2015.

Digital Library

[32]

C. H. Papadimitriou. Computational complexity. In Encyclopedia of Computer Science, pages 260--265. 2003.

Digital Library

[33]

X. Qian and G. Wiederhold. Incremental recomputation of active relational expressions. IEEE Trans. on Knowl. and Data Eng., 3(3):337--341, 1991.

Digital Library

[34]

K. A. Ross, D. Srivastava, and S. Sudarshan. Materialized view maintenance and integrity constraint checking: Trading space for time. In Proc. of SIGMOD, pages 447--458, 1996.

Digital Library

[35]

N. Roussopoulos. Materialized views and data warehouses. SIGMOD Records, 27(1):21--26, 1998.

Digital Library

[36]

M. Schleich, D. Olteanu, and R. Ciucanu. Learning linear regression models over factorized joins. In Proc. of SIGMOD, pages 3--18, 2016.

Digital Library

[37]

L. Segoufin. Constant delay enumeration for conjunctive queries. SIGMOD Record, 44(1):10--17, 2015.

Digital Library

[38]

M. Y. Vardi. The complexity of relational query languages (extended abstract). In Proc. of STOC, pages 137--146, 1982.

Digital Library

[39]

M. Yannakakis. Algorithms for acyclic database schemes. In Proc. of VLDB, pages 82--94, 1981.

Digital Library

Cited By

Birler AKemper ANeumann T(2024)Robust Join Processing with Diamond Hardened JoinsProceedings of the VLDB Endowment10.14778/3681954.368199517:11(3215-3228)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.14778/3681954.3681995
Yang JFang SGu ZMa ZLin XTian Z(2024)TC-Match: Fast Time-Constrained Continuous Subgraph MatchingProceedings of the VLDB Endowment10.14778/3681954.368196317:11(2791-2804)Online publication date: 30-Aug-2024
https://doi.org/10.14778/3681954.3681963
Muñoz MRiveros C(2024)Streaming enumeration on nested documentsACM Transactions on Database Systems10.1145/3701557Online publication date: 25-Oct-2024
https://doi.org/10.1145/3701557
Show More Cited By

Index Terms

The Dynamic Yannakakis Algorithm: Compact and Efficient Query Processing Under Updates
1. Information systems
  1. Data management systems
    1. Database design and models
      1. Data model extensions
        Data streams
    2. Database management system engines
      1. Database query processing
        Join algorithms
        Query optimization
      2. Main memory engines
2. Theory of computation
  1. Models of computation
    1. Streaming models
  2. Theory and algorithms for application domains
    1. Database theory
      1. Database query processing and optimization (theory)

Recommendations

Maintaining Acyclic Foreign-Key Joins under Updates
SIGMOD '20: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

A large number of analytical queries (e.g., all the 22 queries in the TPC-H benchmark) are based on acyclic foreign-key joins. In this paper, we study the problem of incrementally maintaining the query results of these joins under updates, i.e., ...
Incremental Techniques for Large-Scale Dynamic Query Processing
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management

Many applications from various disciplines are now required to analyze fast evolving big data in real time. Various approaches for incremental processing of queries have been proposed over the years. Traditional approaches rely on updating the results ...
Insert-Only versus Insert-Delete in Dynamic Query Evaluation
PODS

We study the dynamic query evaluation problem: Given a full conjunctive query Q and a sequence of updates to the input database, we construct a data structure that supports constant-delay enumeration of the tuples in the query output after each update.

...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data

May 2017

1810 pages

ISBN:9781450341974

DOI:10.1145/3035918

General Chairs:
Rada Chirkova
North Carolina State University, USA
,
Jun Yang
Duke University, USA
,
Program Chair:
Dan Suciu
University of Washington, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 May 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Wiener-Anspach foundation
Erasmus Mundus Joint Doctorate in
Brussels Captial Region--Innoviris

Conference

SIGMOD/PODS'17

Sponsor:

SIGMOD

SIGMOD/PODS'17: International Conference on Management of Data

May 14 - 19, 2017

Illinois, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

51
Total Citations
View Citations
912
Total Downloads

Downloads (Last 12 months)87
Downloads (Last 6 weeks)11

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Birler AKemper ANeumann T(2024)Robust Join Processing with Diamond Hardened JoinsProceedings of the VLDB Endowment10.14778/3681954.368199517:11(3215-3228)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.14778/3681954.3681995
Yang JFang SGu ZMa ZLin XTian Z(2024)TC-Match: Fast Time-Constrained Continuous Subgraph MatchingProceedings of the VLDB Endowment10.14778/3681954.368196317:11(2791-2804)Online publication date: 30-Aug-2024
https://doi.org/10.14778/3681954.3681963
Muñoz MRiveros C(2024)Streaming enumeration on nested documentsACM Transactions on Database Systems10.1145/3701557Online publication date: 25-Oct-2024
https://doi.org/10.1145/3701557
Abo Khamis MKara AOlteanu DSuciu D(2024)Insert-Only versus Insert-Delete in Dynamic Query EvaluationProceedings of the ACM on Management of Data10.1145/36958372:5(1-26)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.1145/3695837
Pinto DRiveros C(2024)Complex Event Recognition meets Hierarchical Conjunctive QueriesProceedings of the ACM on Management of Data10.1145/36958342:5(1-26)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.1145/3695834
Wang QLuo QWang Y(2024)Relational Algorithms for Top-k Query EvaluationProceedings of the ACM on Management of Data10.1145/36549712:3(1-27)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654971
Dai BHu XYi K(2024)Reservoir Sampling over JoinsProceedings of the ACM on Management of Data10.1145/36549212:3(1-26)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654921
Hu X(2024)Fast Matrix Multiplication for Query ProcessingProceedings of the ACM on Management of Data10.1145/36515992:2(1-25)Online publication date: 14-May-2024
https://dl.acm.org/doi/10.1145/3651599
Agarwal PHu XSintos SYang J(2024)On Reporting Durable Patterns in Temporal Proximity GraphsProceedings of the ACM on Management of Data10.1145/36511442:2(1-26)Online publication date: 14-May-2024
https://dl.acm.org/doi/10.1145/3651144
Olteanu DGeerts FMartens W(2024)Recent Increments in Incremental View MaintenanceCompanion of the 43rd Symposium on Principles of Database Systems10.1145/3635138.3654763(8-17)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3635138.3654763
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents