research-article

Nephele/PACTs: a programming model and execution framework for web-scale analytical processing

Authors:

Dominic Battré,

Daniel WarnekeAuthors Info & Claims

SoCC '10: Proceedings of the 1st ACM symposium on Cloud computing

Pages 119 - 130

https://doi.org/10.1145/1807128.1807148

Published: 10 June 2010 Publication History

Abstract

We present a parallel data processor centered around a programming model of so called Parallelization Contracts (PACTs) and the scalable parallel execution engine Nephele [18]. The PACT programming model is a generalization of the well-known map/reduce programming model, extending it with further second-order functions, as well as with Output Contracts that give guarantees about the behavior of a function. We describe methods to transform a PACT program into a data flow for Nephele, which executes its sequential building blocks in parallel and deals with communication, synchronization and fault tolerance. Our definition of PACTs allows to apply several types of optimizations on the data flow during the transformation.

The system as a whole is designed to be as generic as (and compatible to) map/reduce systems, while overcoming several of their major weaknesses: 1) The functions map and reduce alone are not sufficient to express many data processing tasks both naturally and efficiently. 2) Map/reduce ties a program to a single fixed execution strategy, which is robust but highly suboptimal for many tasks. 3) Map/reduce makes no assumptions about the behavior of the functions. Hence, it offers only very limited optimization opportunities. With a set of examples and experiments, we illustrate how our system is able to naturally represent and efficiently execute several tasks that do not fit the map/reduce model well.

References

[1]

Hadoop. URL: http://hadoop.apache.org.

[2]

TPC-H. URL: http://www.tpc.org/tpch/.

[3]

K. Beyer, V. Ercegovac, J. Rao, and E. Shekita. Jaql: A JSON Query Language. URL: http://jaql.org.

[4]

R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. PVLDB, 1(2):1265--1276, 2008.

Digital Library

[5]

J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In OSDI, pages 137--150, 2004.

Digital Library

[6]

J. Delmerico, N. Byrnes, A. Bruno, M. Jones, S. Gallo, and V. Chaudhary. Comparing the Performance of Clusters, Hadoop, and Active Disks on Microarray Correlation Computations. In International Conference on High Performance Computing, 2009.

[7]

D. J. DeWitt, R. H. Gerber, G. Graefe, M. L. Heytens, K. B. Kumar, and M. Muralikrishna. GAMMA - A High Performance Dataflow Database Machine. In W. W. Chu, G. Gardarin, S. Ohsuga, and Y. Kambayashi, editors, VLDB, pages 228--237. Morgan Kaufmann, 1986.

Digital Library

[8]

E. Friedman, P. M. Pawlowski, and J. Cieslewicz. SQL/MapReduce: A Practical Approach to Self-describing, Polymorphic, and Parallelizable User-defined Functions. PVLDB, 2(2):1402--1413, 2009.

Digital Library

[9]

S. Fushimi, M. Kitsuregawa, and H. Tanaka. An Overview of The System Software of A Parallel Relational Database Machine GRACE. In W. W. Chu, G. Gardarin, S. Ohsuga, and Y. Kambayashi, editors, VLDB, pages 209--219. Morgan Kaufmann, 1986.

Digital Library

[10]

M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks. In P. Ferreira, T. R. Gross, and L. Veiga, editors, EuroSys, pages 59--72. ACM, 2007.

Digital Library

[11]

C. Olston, B. Reed, A. Silberstein, and U. Srivastava. Automatic Optimization of Parallel Dataflow Programs. In R. Isaacs and Y. Zhou, editors, USENIX Annual Technical Conference, pages 267--273. USENIX Association, 2008.

Digital Library

[12]

C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A Not-So-Foreign Language for Data Processing. In J. T.-L. Wang, editor, SIGMOD Conference, pages 1099--1110. ACM, 2008.

Digital Library

[13]

A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. DeWitt, S. Madden, and M. Stonebraker. A Comparison of Approaches to Large-Scale Data Analysis. In U. Cetintemel, S. B. Zdonik, D. Kossmann, and N. Tatbul, editors, SIGMOD Conference, pages 165--178. ACM, 2009.

Digital Library

[14]

D. A. Schneider and D. J. DeWitt. A Performance Evaluation of Four Parallel Join Algorithms in a Shared-Nothing Multiprocessor Environment. In J. Clifford, B. G. Lindsay, and D. Maier, editors, SIGMOD Conference, pages 110--121. ACM Press, 1989.

Digital Library

[15]

P. G. Selinger, M. M. Astrahan, D. D. Chamberlin, R. A. Lorie, and T. G. Price. Access Path Selection in a Relational Database Management System. In P. A. Bernstein, editor, SIGMOD Conference, pages 23--34. ACM, 1979.

Digital Library

[16]

J. W. Stamos and H. C. Young. A Symmetric Fragment and Replicate Algorithm for Distributed Joins. IEEE Trans. Parallel Distrib. Syst., 4(12):1345--1354, 1993.

Digital Library

[17]

A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive - A Warehousing Solution Over a Map-Reduce Framework. PVLDB, 2(2):1626--1629, 2009.

Digital Library

[18]

D. Warneke and O. Kao. Nephele: Efficient Parallel Data Processing in the Cloud. In I. Raicu, I. T. Foster, and Y. Zhao, editors, SC-MTAGS. ACM, 2009.

Digital Library

[19]

C. Yang, C. Yen, C. Tan, and S. Madden. Osprey: Implementing MapReduce-Style Fault Tolerance in a Shared-Nothing Distributed Database. In ICDE, 2009.

[20]

H. Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker. Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters. In C. Y. Chan, B. C. Ooi, and A. Zhou, editors, SIGMOD Conference, pages 1029--1040. ACM, 2007.

Digital Library

[21]

Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In R. Draves and R. van Renesse, editors, OSDI, pages 1--14. USENIX Association, 2008.

Digital Library

Cited By

Zhao MAdamiak EKozyrakis C(2024)cedar: Optimized and Unified Machine Learning Input Data PipelinesProceedings of the VLDB Endowment10.14778/3705829.370586118:2(488-502)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.14778/3705829.3705861
Sevim AEldawy ACarman ECarey MTsotras V(2024)FUDJ: Flexible User-Defined Distributed Joins2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00320(4194-4207)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00320
Mishra RMihovska APrasad R(2023)Symphony of Sensors: The Harmonious Art of Massive Data Generation by Devices2023 26th International Symposium on Wireless Personal Multimedia Communications (WPMC)10.1109/WPMC59531.2023.10338947(1-5)Online publication date: 19-Nov-2023
https://doi.org/10.1109/WPMC59531.2023.10338947
Show More Cited By

Index Terms

Nephele/PACTs: a programming model and execution framework for web-scale analytical processing
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Parallel and distributed DBMSs

Recommendations

Nephele: efficient parallel data processing in the cloud
MTAGS '09: Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers

In recent years Cloud Computing has emerged as a promising new approach for ad-hoc parallel data processing. Major cloud computing companies have started to integrate frameworks for parallel data processing in their product portfolio, making it easy for ...
Massively parallel data analysis with PACTs on Nephele

Large-scale data analysis applications require processing and analyzing of Terabytes or even Petabytes of data, particularly in the areas of web analysis or scientific data management. This trend has been discussed as "web-scale data management" in a ...
BigBench: towards an industry standard benchmark for big data analytics
SIGMOD '13: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

There is a tremendous interest in big data by academia, industry and a large user base. Several commercial and open source providers unleashed a variety of products to support big data storage and processing. As these products mature, there is a need to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SoCC '10: Proceedings of the 1st ACM symposium on Cloud computing

June 2010

264 pages

ISBN:9781450300360

DOI:10.1145/1807128

General Chair:
Joseph M. Hellerstein
University of California, Berkeley, USA
,
Program Chairs:
Surajit Chaudhuri
Microsoft Research, USA
,
Mendel Rosenblum
Stanford University, USA

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 June 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SOCC '10

Sponsor:

SOCC '10: ACM SIGMOD-SIGOPS Workshop on Cloud Computing in conjunction with SIGMOD 2010

June 10 - 11, 2010

Indiana, Indianapolis, USA

Acceptance Rates

Overall Acceptance Rate 169 of 722 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

204
Total Citations
View Citations
1,119
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhao MAdamiak EKozyrakis C(2024)cedar: Optimized and Unified Machine Learning Input Data PipelinesProceedings of the VLDB Endowment10.14778/3705829.370586118:2(488-502)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.14778/3705829.3705861
Sevim AEldawy ACarman ECarey MTsotras V(2024)FUDJ: Flexible User-Defined Distributed Joins2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00320(4194-4207)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00320
Mishra RMihovska APrasad R(2023)Symphony of Sensors: The Harmonious Art of Massive Data Generation by Devices2023 26th International Symposium on Wireless Personal Multimedia Communications (WPMC)10.1109/WPMC59531.2023.10338947(1-5)Online publication date: 19-Nov-2023
https://doi.org/10.1109/WPMC59531.2023.10338947
Padmalatha ESailekya S(2022)Hybrid Clustering Technique to Cluster Big Data in the Hadoop EcosystemHandbook of Research on Technologies and Systems for E-Collaboration During Global Crises10.4018/978-1-7998-9640-1.ch015(218-233)Online publication date: 2022
https://doi.org/10.4018/978-1-7998-9640-1.ch015
Ayiti OAyangbenro ABabalola O(2022)Relationship between nitrifying microorganisms and other microorganisms residing in the maize rhizosphereArchives of Microbiology10.1007/s00203-022-02857-2204:5Online publication date: 8-Apr-2022
https://doi.org/10.1007/s00203-022-02857-2
Hueske FWalther T(2022)Apache FlinkEncyclopedia of Big Data Technologies10.1007/978-3-319-63962-8_303-2(1-9)Online publication date: 17-May-2022
https://doi.org/10.1007/978-3-319-63962-8_303-2
Krommyda MKantere V(2021)The Big Data EraAnalyzing Future Applications of AI, Sensors, and Robotics in Society10.4018/978-1-7998-3499-1.ch006(87-103)Online publication date: 2021
https://doi.org/10.4018/978-1-7998-3499-1.ch006
Ceballos ORamírez Restrepo CPabón MCastillo ACorcho O(2021)SPARQL2Flink: Evaluation of SPARQL Queries on Apache FlinkApplied Sciences10.3390/app1115703311:15(7033)Online publication date: 30-Jul-2021
https://doi.org/10.3390/app11157033
Chen YWang JLu YHan YLv ZMin XCai HZhang WFan HLi CGuan TLin WJia YZhou J(2021)FangornProceedings of the VLDB Endowment10.14778/3476311.347637614:12(2972-2985)Online publication date: 1-Jul-2021
https://dl.acm.org/doi/10.14778/3476311.3476376
Smith JBenedikt MNikolic MShaikhha A(2021)Scalable querying of nested dataProceedings of the VLDB Endowment10.14778/3430915.343093314:3(445-457)Online publication date: 9-Dec-2021
https://dl.acm.org/doi/10.14778/3430915.3430933
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten