research-article

Algebricks: a data model-agnostic compiler backend for big data languages

Authors:

Vinayak Borkar,

E. Preston Carman, Jr.,

Pouria Pirzadeh,

Michael J. Carey,

Vassilis J. TsotrasAuthors Info & Claims

SoCC '15: Proceedings of the Sixth ACM Symposium on Cloud Computing

Pages 422 - 433

https://doi.org/10.1145/2806777.2806941

Published: 27 August 2015 Publication History

Abstract

A number of high-level query languages, such as Hive, Pig, Flume, and Jaql, have been developed in recent years to increase analyst productivity when processing and analyzing very large datasets. The implementation of each of these languages includes a complete, data model-dependent query compiler, yet each involves a number of similar optimizations. In this work, we describe a new query compiler architecture that separates language-specific and data model-dependent aspects from a more general query compiler backend that can generate executable data-parallel programs for shared-nothing clusters and can be used to develop multiple languages with different data models. We have built such a data model-agnostic query compiler substrate, called Algebricks, and have used it to implement three different query languages --- HiveQL, AQL, and XQuery --- to validate the efficacy of this approach. Experiments show that all three query languages benefit from the parallelization and optimization that Algebricks provides and thus have good parallel speedup and scaleup characteristics for large datasets.

References

[1]

National Climate Data Center: Data Access. http://www.ncdc.noaa.gov/data-access/.

[2]

XQuery 1.0 and XPath 2.0 Formal Semantics (Second Edition), 2010.

[3]

S. Alsubaiee, Y. Altowim, H. Altwaijry, A. Behm, V. R. Borkar, Y. Bu et al. AsterixDB: A Scalable, Open Source BDMS. PVLDB, 7(14): 1905--1916, 2014.

Digital Library

[4]

S. Alsubaiee, A. Behm, V. R. Borkar, Z. Heilbron, Y. Kim, M. Carey et al. Storage Management in AsterixDB. PVLDB, 7(10):841--852, 2014.

Digital Library

[5]

Apache. The Distributed DataFrame Project. http://ddf.io/,.

[6]

Apache. Hadoop: Open-Source Implementation of MapReduce. http://hadoop.apache.org,.

[7]

Apache. The SparkSQL Project. https://spark.apache.org/sql/,.

[8]

Apache. The Tez Project. http://tez.apache.org/,.

[9]

M. Armbrust, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. Bradley et al. Spark SQL: Relational Data Processing in Spark. In SIGMOD, 2015.

Digital Library

[10]

D. S. Batory. Genesis: a Project to Develop an Extensible Database Management System. In OODS, pages 207--208.

Digital Library

[11]

D. Battré, S. Ewen, F. Hueske, O. Kao, V. Markl, and D. Warneke. Nephele/PACTs: a Programming Model and Execution Framework for Web-Scale Analytical Processing. In SoCC, pages 119--130, 2010.

Digital Library

[12]

H. Boral, W. Alexander, L. Clay, G. Copeland, S. Danforth, M. Franklin et al. Prototyping Bubba, A Highly Parallel Database System. IEEE Trans. Knowl. Data Eng., 2(1):4--24, Mar. 1990.

Digital Library

[13]

V. R. Borkar, M. J. Carey, R. Grover, N. Onose, and R. Vernica. Hyracks: A Flexible and Extensible Foundation for Data-Intensive Computing. In ICDE, pages 1151--1162, 2011.

Digital Library

[14]

calcite. Apache Calcite. http://calcite.incubator.apache.org/.

[15]

M. J. Carey, D. J. DeWitt, D. Frank, G. Graefe, J. E. Richardson, E. J. Shekita et al. The Architecture of the EXODUS Extensible DBMS. In OODS, pages 231--256. 1991.

Digital Library

[16]

E. P. Carman, Jr., T. Westmann, V. R. Borkar, M. J. Carey, and V. J. Tsotras. Apache VXQuery: A Scalable XQuery Implementation. CoRR, abs/1504.00331, 2015.

[17]

D. D. Chamberlin. XQuery: A Query Language for XML. In SIGMOD, page 682, 2003.

Digital Library

[18]

J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In OSDI, pages 137--150, 2004.

Digital Library

[19]

A. Deutsch, Y. Papakonstantinou, and Y. Xu. The NEXT Framework for Logical XQuery Optimization. In VLDB, pages 168--179, 2004.

Digital Library

[20]

D. DeWitt and J. Gray. Parallel Database Systems: The Future of High Performance Database Systems. Commun. ACM, 35:85--98, 1992.

Digital Library

[21]

D. DeWitt, R. Katz, F. Olken, L. Shapiro, M. Stonebraker, and D. Wood. Implementation Techniques for Main Memory Database Systems. In SIGMOD, pages 1--8, 1984.

Digital Library

[22]

D. DeWitt, S. Ghandeharizadeh, D. A. Schneider, A. Bricker, H. Hsiao, and R. Rasmussen. The Gamma Database Machine Project. IEEE Trans. Knowl. Data Eng., 2(1):44--62, 1990.

Digital Library

[23]

S. Fushimi, M. Kitsuregawa, and H. Tanaka. An Overview of The System Software of a Parallel Relational Database Machine GRACE. In VLDB, pages 209--219, 1986.

Digital Library

[24]

G. Graefe. Encapsulation of Parallelism in the Volcano Query Processing System. In SIGMOD, pages 102--111, 1990.

Digital Library

[25]

G. Graefe. Volcano - An Extensible and Parallel Query Evaluation System. IEEE Trans. Knowl. Data Eng., 6(1):120--135, 1994.

Digital Library

[26]

G. Graefe. The Cascades Framework for Query Optimization. IEEE Data Eng. Bull., 18(3):19--29, 1995.

[27]

G. Graefe and D. DeWitt. The EXODUS Optimizer Generator. SIGMOD Rec., 16(3):160--172, Dec. 1987.

Digital Library

[28]

R. Grover and M. J. Carey. Data Ingestion in AsterixDB. In EDBT, pages 605--616, 2015.

[29]

R. H. Güting, R. Zicari, and D. M. Choy. An Algebra for Structured Office Documents. ACM Trans. Inf. Syst., 7(2):123--157, 1989.

Digital Library

[30]

M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks. In EuroSys, pages 59--72, 2007.

Digital Library

[31]

G. Jaeschke and H. Schek. Remarks on the Algebra of Non First Normal Form Relations. In PODS, pages 124--138, 1982.

Digital Library

[32]

H. V. Jagadish, L. V. S. Lakshmanan, D. Srivastava, and K. Thompson. TAX: A Tree Algebra for XML. In DBPL, pages 149--164, 2001.

Digital Library

[33]

M. Kornacker, A. Behm, V. Bittorf, T. Bobrovytsky, C. Ching, A. Choi et al. Impala: A Modern, Open-Source SQL Engine for Hadoop. In CIDR, 2015.

[34]

C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A Not-so-Foreign Language for Data Processing. In SIGMOD, pages 1099--1110, 2008.

Digital Library

[35]

Oracle. Using Oracle XQuery for Hadoop. http://docs.oracle.com/cd/E49465_01/doc.23/e49333/oxh.htm#BDCUG526.

[36]

Y. Papakonstantinou, V. Borkar, M. Orgiyan, K. Stathatos, L. Suta, V. Vassalos et al. XML Queries and Algebra in the Enosys Integration Platform. Data and Knowl. Eng., 44(3):299--322, 2003.

Digital Library

[37]

R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the Data: Parallel Analysis with Sawzall. Scientific Programming, 13(4): 277--298, 2005.

Digital Library

[38]

J. Shemer and P. Neches. The Genesis of a Database Computer. Computer, 17(11):42--56, Nov. 1984. ISSN 0018--9162.

Digital Library

[39]

M. A. Soliman, L. Antova, V. Raghavan, A. El-Helw, Z. Gu, E. Shen et al. Orca: A Modular Query Optimizer Architecture for Big Data. In SIGMOD, pages 337--348, 2014.

Digital Library

[40]

M. Stonebraker. Inclusion of New Types in Relational Data Base Systems. In ICDE, pages 262--269, 1986.

Digital Library

[41]

A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony et al. Hive: a Warehousing Solution Over a Map-Reduce Framework. PVLDB, 2(2):1626--1629, 2009. ISSN 2150--8097.

Digital Library

[42]

J. Wen, V. R. Borkar, M. J. Carey, and V. J. Tsotras. Revisiting Aggregation for Data Intensive Applications: A Performance Study. CoRR, abs/1311.0059, 2013.

[43]

Y. Yu, M. Isard, D. Fetterly, M. Budiu, Ú. Erlingsson, P. K. Gunda et al. DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In OSDI, pages 1--14, 2008.

Digital Library

[44]

M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley et al. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In NSDI, pages 15--28, 2012.

Digital Library

[45]

J. Zhou, N. Bruno, M.-C. Wu, P.-Å. Larson, R. Chaiken, and D. Shakib. SCOPE: Parallel Databases Meet MapReduce. VLDB J., 21(5):611--636, 2012.

Digital Library

Cited By

Galvizo GCarey M(2024)Graphix: “One User's JSON is Another User's Graph”2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00238(3070-3083)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00238
Pavlopoulou CCarey MTsotras V(2023)Revisiting Runtime Dynamic Optimization for Join Queries in Big Data Management SystemsACM SIGMOD Record10.1145/3604437.360446052:1(104-113)Online publication date: 8-Jun-2023
https://dl.acm.org/doi/10.1145/3604437.3604460
Galvizo GCarey M(2023)Multi-valued indexing in Apache AsterixDB (SI DOLAP 2022)Information Systems10.1016/j.is.2022.102144113:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.is.2022.102144
Show More Cited By

Index Terms

Algebricks: a data model-agnostic compiler backend for big data languages
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Database query processing
    2. Query languages
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Database theory
      1. Database query processing and optimization (theory)

Recommendations

Performance analysis of "Groupby-After-Join" query processing in parallel database systems

Queries containing aggregate functions often combine multiple tables through join operations. This query is subsequently called "Groupby-Join". There is a special category of this query whereby the group-by operation can only be performed after the join ...
Distributed Query Processing on the Grid

Distributed query processing DQP has been widely used in data intensive applications where data of relevance to users are stored at multiple locations. This paper argues: i that DQP can be important in the Grid, as a means of providing high-level, ...
Extensible Parallel Query Processing for Exploratory Geoscientific Data Mining

Exploratory data mining and analysis requires a computing environment which provides facilities for the user-friendly expression and rapid execution of “scientific queries.” In this paper, we address research issues in the parallelization of scientific ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SoCC '15: Proceedings of the Sixth ACM Symposium on Cloud Computing

August 2015

446 pages

ISBN:9781450336512

DOI:10.1145/2806777

General Chair:
Shahram Ghandeharizadeh
University of Southern California
,
Program Chairs:
Magdalena Balazinska
University of Washington
,
Michael J. Freedman
Princeton University
,
Publications Chair:
Sumita Barahmand
Microsoft

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF IIS
UC Discovery grant
NSF CNS

Conference

SoCC '15

Sponsor:

SoCC '15: ACM Symposium on Cloud Computing

August 27 - 29, 2015

Hawaii, Kohala Coast

Acceptance Rates

SoCC '15 Paper Acceptance Rate 34 of 157 submissions, 22%;

Overall Acceptance Rate 169 of 722 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
348
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)7

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Galvizo GCarey M(2024)Graphix: “One User's JSON is Another User's Graph”2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00238(3070-3083)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00238
Pavlopoulou CCarey MTsotras V(2023)Revisiting Runtime Dynamic Optimization for Join Queries in Big Data Management SystemsACM SIGMOD Record10.1145/3604437.360446052:1(104-113)Online publication date: 8-Jun-2023
https://dl.acm.org/doi/10.1145/3604437.3604460
Galvizo GCarey M(2023)Multi-valued indexing in Apache AsterixDB (SI DOLAP 2022)Information Systems10.1016/j.is.2022.102144113:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.is.2022.102144
Alkowaileet WCarey M(2022)Columnar formats for schemaless LSM-based document storesProceedings of the VLDB Endowment10.14778/3547305.354731415:10(2085-2097)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.14778/3547305.3547314
Alkowaileet WAlsubaiee SCarey M(2020)An LSM-based tuple compaction framework for Apache AsterixDBProceedings of the VLDB Endowment10.14778/3397230.339723613:9(1388-1400)Online publication date: 26-Jun-2020
https://dl.acm.org/doi/10.14778/3397230.3397236
Jacobs SWang XCarey MTsotras VUddin M(2020)BAD to the bone: Big Active Data at its coreThe VLDB Journal10.1007/s00778-020-00616-7Online publication date: 23-May-2020
https://doi.org/10.1007/s00778-020-00616-7
Kim TBehm ABlow MBorkar VBu YCarey MHubail MJahangiri SJia JLi CLuo CMaxon IPirzadeh P(2020)Robust and efficient memory management in Apache AsterixDBSoftware: Practice and Experience10.1002/spe.279950:7(1114-1151)Online publication date: 17-Feb-2020
https://doi.org/10.1002/spe.2799
Hubail MAlsuliman ABlow MCarey MLychagin DMaxon IWestmann T(2019)Couchbase analyticsProceedings of the VLDB Endowment10.14778/3352063.335214312:12(2275-2286)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.14778/3352063.3352143
Carey M(2019)AsterixDB Mid-Flight: A Case Study in Building Systems in Academia2019 IEEE 35th International Conference on Data Engineering (ICDE)10.1109/ICDE.2019.00008(1-12)Online publication date: Apr-2019
https://doi.org/10.1109/ICDE.2019.00008
Liang QHainan N(2019)Adaptive Learning Model and Implementation Based on Big Data2019 2nd International Conference on Artificial Intelligence and Big Data (ICAIBD)10.1109/ICAIBD.2019.8836984(183-186)Online publication date: May-2019
https://doi.org/10.1109/ICAIBD.2019.8836984
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents