article

Free access

Encapsulation of parallelism in the Volcano query processing system

Author:

Goetz GraefeAuthors Info & Claims

ACM SIGMOD Record, Volume 19, Issue 2

Pages 102 - 111

https://doi.org/10.1145/93605.98720

Published: 01 May 1990 Publication History

Abstract

Volcano is a new dataflow query processing system we have developed for database systems research and education. The uniform interface between operators makes Volcano extensible by new operators. All operators are designed and coded as if they were meant for a single-process system only. When attempting to parallelize Volcano, we had to choose between two models of parallelization, called here the bracket and operator models. We describe the reasons for not choosing the bracket model, introduce the novel operator model, and provide details of Volcano's exchange operator that parallelizes all other operators. It allows intra-operator parallelism on partitioned datasets and both vertical and horizontal inter-operator parallelism. The exchange operator encapsulates all parallelism issues and therefore makes implementation of parallel database algorithms significantly easier and more robust. Included in this encapsulation is the translation between demand-driven dataflow within processes and data-driven dataflow between processes. Since the interface between Volcano operators is similar to the one used in “real,” commercial systems, the techniques described here can be used to parallelize other query processing engines.

References

[1]

M Accetta, R Baron, W. Bolosky, D Golub, R. Raslud, A Tevaman and M Young. "Mach. A New Kernel Founda~on for UNIX Developm~t", Summer Conference Proceedings 1986,

[2]

W Alexander and G. Copelmd, "Process and Damflow Control m D~mbuteA Dam-Intm~ive $ysmms", Proceedings of the ACM SIGMOD Conference, Clncago, IL,* june 1988, 90-98

[3]

M M. As~ahan, M W. Blasgen, D. D Otmabezlm, K. P. Eswarm, J. N. Gray, P. P. Gnffi~, W. F. King, R. A Lone, P R. Mdones, } W Mehl, G. R Pun~olu, I L Trmger, B W Wade and V. WaLson, "System R' A Relauonal Approach to Database Management", A CM Tranaactwna on Database Systems 1, 2 (June 1976), 97-137.

[4]

C.K. Baru, O. Fneder, D. Kandlur and M Segal, "Yore on a Cube" Analysts, Sunulauon, and Implememauon", Proceedings of the 5th Interna~onal Workahop on Database Machv~, 1987.

[5]

D S. Bamry, "GENESIS' A Project to Develop an Extenszble Database Management System", Proceedings of the lnt'l Workshop on ObJect.Oriented Database Systems, Pacific Grove, CA, September 1986, 207-208.

[6]

D. Bmon, D. J. DeW:~ and C. Turbyffll, "Benchmarkmg Database Systems: A Systemauc Approach.", Proceeding of the Conference on Very Large Data Bases, Florence, Italy, October-November 1983, 8-19

[7]

D. B~uon, H. Boral, D. J DeWm and W. IC Wfllamon, "Parallel Algonflm~.s for the Execuuon of Relauonal Database Operauons", ACM Transacuons o n Database Systems 8, 3 (September 1983), 324.353

[8]

H. Boral and D. J DeWIu, "Database Machines, An Idea Whose Tune Has Passed? A Cnuque of Future of Database Machines", Proceeding of the Internanonal Workshop on Database Machow.s, Mumch, 1983

[9]

H. Boral and D J DeWnt, "A Methodology for Database System Performance Evaluauon" Proceedings of the ACM SIGMOD Conference', Boston, MA, June 1984, 176-185

[10]

M } Carey, D J DeWttt, j' E Richardson and E J. Shek~ta, "ObJect and File Management m the EXODUS Extenslble Database System", Proceedings of the Conference on Very Large Data Bases, Kyota,}'apart, August 1986, 91-100

[11]

H T. Chou, D J DeWltt, R H Katz and A. C, Klug, "Deslgn and Implementauon of the WLsconsm Storage System". Software - Pracnce and E~eneace 15, 10 (October 1985), 943-962

[12]

D 3 DeWnt, R H Gerber, G Grade, M L. Heytens, K B Kumar and M Murakknshna, "GAMMA - A High Performance Daudlow Database Machine", Proceedings of the Conference on Very Large Data Bases, Kyoto, Japan, August 1986, 228- 237

[13]

13 .1 DeWxtt, S Ghandeharadlzeh, D Schneider, A laneker, H I Hslao and R Rasmussen, '"I'he G amma Database Machine Project", IEEE Tvansactwns on Knowledge and Data Engineering 2, 1 (March 1990)

[14]

S Englert, J Gray, R Kocher and P Shah, "A Benchmark of NonStop SQL Release 2 Demonstr~mg Near-Linear Speedup and Scaleup on Large Databases", Tandem Computer Systems Technural Report 89 # (May 1989)

[15]

R Gerber, "Dataflow Query Processing using Muluprocessor Hash-Paruuoned Algonthms", Ph.D Thes~s, Madxson. October 1986

[16]

G Gr~fe and D Ma~er, "Query Optmuzataon m ObJect-Oriented Database Systems A Prospectus", m Advances m O blect-Onented Database Systems, vol 334, K. R Dxtmch (editor), Sprmger-Verlag, September 1988, 358-363

[17]

(3. Gtaefe, "Volcano Art Extensxble and P~rLllel Data.flow Query Processing sysm, Oregon Graduate Center, Computer Science Techmcal Report, Beaverton, OR, June Z989

[18]

O Grade, "Sot Processing and Complex Object Assembly m Volcano and the REVELATION P roject", Oregon Graduate Center, Computer Scw.nce Technical Report, Beaverton, OR, June 1989

[19]

O. Grade, "Relaraonal Dxvmon. Four Algomlm~ and Thetr Performance", Proceedings of the IEEE Conference on Data Engineering, Los Angelos, CA, February 1989, 94-101

[20]

G. Grade and K Ward, "Dynamxc Query Evaluauon Plans", Proceedings of the ACM SIGMOD Conference, Portland, OR, May-June 1989, 358

[21]

(3 Grade, "Parallel External Sorting m Volcano", submitted for publurat=on, February 1990

[22]

L. M Haas, W F Cody, J C Freytag, G Lap=, B G. Lmdsay, O. M Lohman, K Ono and H Ptrahesh, "An Extensxble Processor for an Extended Relauonal Query Language", Computer Scw.nce Research Report, San Jose, CA, Aprd 1988

[23]

T Keller and (3 Grade, "The One-to-One Match Operator of the Volcano Query Processing System", Oregon Graduate Center, Computer Science Techn~al Report, Beaverton, OR, June 1989

[24]

J E Richardson and M J Carey, "Programming Constructs for Database System implementation m EXODUS", Proceedings of the ACM SIGMOD Conference, San Francisco, CA., May 1987, 208-219

[25]

K Salem and H Gareaa-Molma, "Dtsk Striping", Proceedings of the IEEE Conference on Data En&meermg, Los Angeles, CA, February 1986, 336

[26]

P Schwarz, W Chang, J C Freytag, 13 Lohmart, I McPherson, C Mohan and H Plrahesb., "Extens~btlxty m the Starburst Database System", Precedings of the lnt'l Workshop on Object.Omented Database Systems, Paeff'xc Grove, CA, September 1986, 85-92.

[27]

M Stonebraker, E. Wang, P. Kreps and O. D. Held, "The D~xgn and implementauon of INGRES", ACM Transactmns on Database Systems 1, 3 (September 1976), 189-222

[28]

M. Stonebraker and L A Rowe, "The Design of POSTGRES", Proceedings of the ACM SIGMOD Conference, Washington, DC., May 1986, 340-355.

[29]

M Stonebraker, R. Katz, D. Patterson and J Ousterhout, "The D~lgn of XPRS", Proceedm&s of the Conference on Very Large Databases, Los Angeles, CA, August 1988, 318-330.

[30]

S. Tom, K. Kojxma, Y. Kanad~ A. Sakam, S. Yos~ and M. Tekahestu, "Accelerating Nonnumencal Processing by an Extended Vector Processor", Proceedings of the IEEE Conference on Data Engineering, Los Angeles, CA, February 1988, 194-201

[31]

P. Wtlhan~, D. Demels, L. Haas, G Lopm, B. Lmdsay, P. Ng, R. Obermarck, P Selmger, A. Walker, P Wilms and R. Yost, "R*. An Overvaew of the Arelutecture", m Readings m Database Systems, M. Smnzbraker (echmr), Morgm-Kaufmm, San Mamo, CA, 1988.

Cited By

Sanca VChrysogelos PAilamaki A(2023)LAQy: Efficient and Reusable Query Approximations via Lazy SamplingProceedings of the ACM on Management of Data10.1145/35893191:2(1-26)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3589319
BESSHO YHAYAMIZU YGODA KKITSUREGAWA M(2022)Dynamic Fault Tolerance for Multi-Node Query ProcessingIEICE Transactions on Information and Systems10.1587/transinf.2021DAP0004E105.D:5(909-919)Online publication date: 1-May-2022
https://doi.org/10.1587/transinf.2021DAP0004
Sanca VAilamaki A(2022)Sampling-Based AQP in Modern Analytical EnginesProceedings of the 18th International Workshop on Data Management on New Hardware10.1145/3533737.3535095(1-8)Online publication date: 12-Jun-2022
https://dl.acm.org/doi/10.1145/3533737.3535095
Show More Cited By

Index Terms

Encapsulation of parallelism in the Volcano query processing system

Recommendations

Encapsulation of parallelism in the Volcano query processing system
SIGMOD '90: Proceedings of the 1990 ACM SIGMOD international conference on Management of data

Volcano is a new dataflow query processing system we have developed for database systems research and education. The uniform interface between operators makes Volcano extensible by new operators. All operators are designed and coded as if they were ...
Volcano An Extensible and Parallel Query Evaluation System

To investigate the interactions of extensibility and parallelism in database query processing, we have developed a new dataflow query execution system called Volcano. The Volcano effort provides a rich environment for research and education in database ...
Encapsulation of Parallelism and Architecture-Independence in Extensible Database Query Execution

Emerging database application domains demand not only high functionality, but also high performance. To satisfy these two requirements, the Volcano query execution engine combines the efficient use of parallelism on a wide variety of computer ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGMOD Record

ACM SIGMOD Record Volume 19, Issue 2

Jun. 1990

392 pages

ISSN:0163-5808

DOI:10.1145/93605

Editor:
H. V. Jagadish
AT&T Bell Laboratories, Murray Hill, NJ

Issue’s Table of Contents

SIGMOD '90: Proceedings of the 1990 ACM SIGMOD international conference on Management of data
May 1990
398 pages
ISBN:0897913655
DOI:10.1145/93597
Chairman:
Hector Garcia-Molina
Princeton Univ., Princeton, NJ

Copyright © 1990 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 May 1990

Published in SIGMOD Volume 19, Issue 2

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

307
Total Citations
View Citations
3,476
Total Downloads

Downloads (Last 12 months)290
Downloads (Last 6 weeks)41

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sanca VChrysogelos PAilamaki A(2023)LAQy: Efficient and Reusable Query Approximations via Lazy SamplingProceedings of the ACM on Management of Data10.1145/35893191:2(1-26)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3589319
BESSHO YHAYAMIZU YGODA KKITSUREGAWA M(2022)Dynamic Fault Tolerance for Multi-Node Query ProcessingIEICE Transactions on Information and Systems10.1587/transinf.2021DAP0004E105.D:5(909-919)Online publication date: 1-May-2022
https://doi.org/10.1587/transinf.2021DAP0004
Sanca VAilamaki A(2022)Sampling-Based AQP in Modern Analytical EnginesProceedings of the 18th International Workshop on Data Management on New Hardware10.1145/3533737.3535095(1-8)Online publication date: 12-Jun-2022
https://dl.acm.org/doi/10.1145/3533737.3535095
Zhang HYu JZhang YZhao KIves ZBonifati AEl Abbadi A(2022)Parallel Query Processing: To Separate Communication from ComputationProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3526164(1447-1461)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3526164
Boroumand AGhose SOliveira GMutlu O(2022)Polynesia: Enabling High-Performance and Energy-Efficient Hybrid Transactional/Analytical Databases with Hardware/Software Co-Design2022 IEEE 38th International Conference on Data Engineering (ICDE)10.1109/ICDE53745.2022.00270(2997-3011)Online publication date: May-2022
https://doi.org/10.1109/ICDE53745.2022.00270
Koutsoukos DMüller IMarroquín RKlimovic AAlonso G(2021)ModularisProceedings of the VLDB Endowment10.14778/3484224.348422914:13(3308-3321)Online publication date: 1-Sep-2021
https://dl.acm.org/doi/10.14778/3484224.3484229
Gencer CTopolnik MĎurina VDemirci EKahveci EGürbüz ALukáš OBartók JGierlach GHartman FYılmaz UDoğan MMandouh MFragkoulis MKatsifodimos A(2021)Hazelcast jetProceedings of the VLDB Endowment10.14778/3476311.347638714:12(3110-3121)Online publication date: 28-Oct-2021
https://dl.acm.org/doi/10.14778/3476311.3476387
Chrysogelos PKarpathiotakis MAppuswamy RAilamaki A(2019)HetExchangeProceedings of the VLDB Endowment10.14778/3303753.330376012:5(544-556)Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.14778/3303753.3303760
Kersten TLeis VKemper ANeumann TPavlo ABoncz P(2019)Everything you always wanted to know about compiled and vectorized queries but were afraid to askProceedings of the VLDB Endowment10.14778/3275366.328496611:13(2209-2222)Online publication date: 17-Jan-2019
https://dl.acm.org/doi/10.14778/3275366.3284966
Sangat PTaniar DIndrawan‐Santiago MMessom C(2019)Nimble join: A parallel star join for main memory column‐storesConcurrency and Computation: Practice and Experience10.1002/cpe.561633:8Online publication date: 20-Dec-2019
https://doi.org/10.1002/cpe.5616
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents