research-article

Weaver: integrating distributed computing abstractions into scientific workflows using Python

Authors:

Peter Bui,

Li Yu,

Douglas ThainAuthors Info & Claims

HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Pages 636 - 643

https://doi.org/10.1145/1851476.1851570

Published: 21 June 2010 Publication History

Get Access

Abstract

Weaver is a high-level framework that enables researchers to integrate distributed computing abstractions into their scientific workflows. Rather than develop a new workflow language, we built Weaver on top of the Python programming language. As such, Weaver takes advantage of users' familiarity with Python, minimizes barriers to adoption, and allows for integration with existing software. In this paper, we introduce Weaver's programming model, which consists of datasets, functions, and abstractions that users combine to organize and specify large-scale scientific workflows. We also explain how these specifications are compiled into a directed acyclic graph used by a workflow manager that dispatches the work to a variety of distributed computing engines. To examine how Weaver is used in scientific research, we present three example applications that demonstrate Weaver's ability to integrate into existing workflows and incorporate optimized distributed computing abstraction tools.

References

[1]

}}The directed acyclic graph manager. http://www.cs.wisc.edu/condor/dagman, 2002.

Google Scholar

[2]

}}J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In Operating Systems Design and Implementation, 2004.

Digital Library

Google Scholar

[3]

}}E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, B. Berriman, J. Good, A. Laity, J. Jacob, and D. Katz. Pegasus: A framework for mapping complex scientific workflows onto distributed systems. Scientific Programming Journal, 13(3), 2005.

Digital Library

Google Scholar

[4]

}}M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data parallel programs from sequential building blocks. In Proceedings of EuroSys, March 2007.

Digital Library

Google Scholar

[5]

}}M. Isard and Y. Yu. Distributed data-parallel computing using a high-level programming language. In SIGMOD '09: Proceedings of the 35th SIGMOD international conference on Management of data, pages 987--994, New York, NY, USA, 2009. ACM.

Digital Library

Google Scholar

[6]

}}C. Moretti, J. Bulosan, D. Thain, and P. Flynn. All-Pairs: An Abstraction for Data Intensive Cloud Computing. In IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 1--11, 2008.

Google Scholar

[7]

}}C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In SIGMOD '08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1099--1110, New York, NY, USA, 2008. ACM.

Digital Library

Google Scholar

[8]

}}R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data: Parallel analysis with sawzall. Scientific Programming Journal, 13(4):227--298.

Digital Library

Google Scholar

[9]

}}Python Programming Language. http://www.python.org/, 2010.

Google Scholar

[10]

}}SQLAlchemy. http://sqlalchemy.org/, 2010.

Google Scholar

[11]

}}D. Thain, T. Tannenbaum, and M. Livny. Condor and the grid. In F. Berman, G. Fox, and T. Hey, editors, Grid Computing: Making the Global Infrastructure a Reality. John Wiley, 2003.

Digital Library

Google Scholar

[12]

}}L. Yu, C. Moretti, A. Thrasher, S. Emrich, K. Judd, and D. Thain. Harnessing Parallelism in Multicore Clusters with the All-Pairs, Wavefront, and Makeflow Abstractions. to appear in Journal of Cluster Computing, 2010.

Digital Library

Google Scholar

[13]

}}Y. Zhao, J. Dobson, L. Moreau, I. Foster, and M. Wilde. A notation and system for expressing and executing cleanly typed workflows on messy scientific data. In SIGMOD, 2005.

Digital Library

Google Scholar

Cited By

View all

Erbel JGrabowski J(2023)Scientific workflow execution in the cloud using a dynamic runtime modelSoftware and Systems Modeling10.1007/s10270-023-01112-623:1(163-193)Online publication date: 23-Jun-2023
https://doi.org/10.1007/s10270-023-01112-6
Deelman EPeterka TAltintas ICarothers Cvan Dam KMoreland KParashar MRamakrishnan LTaufer MVetter J(2018)The future of scientific workflowsInternational Journal of High Performance Computing Applications10.5555/3195474.319547732:1(159-175)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.5555/3195474.3195477
Deelman EPeterka TAltintas ICarothers Cvan Dam KMoreland KParashar MRamakrishnan LTaufer MVetter J(2017)The future of scientific workflowsThe International Journal of High Performance Computing Applications10.1177/109434201770489332:1(159-175)Online publication date: 26-Apr-2017
https://doi.org/10.1177/1094342017704893
Show More Cited By

Index Terms

Weaver: integrating distributed computing abstractions into scientific workflows using Python
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles

Recommendations

Scripting distributed scientific workflows using Weaver

Weaver is a high-level distributed computing framework that enables researchers to construct scalable scientific data-processing workflows. Instead of developing a new workflow language, we introduce a domain-specific language built on top of Python ...
Aspect weaver: a model transformation approach for UML models
CASCON '10: Proceedings of the 2010 Conference of the Center for Advanced Studies on Collaborative Research

Aspect-Oriented Modeling (AOM) is an emerging solution for handling crosscutting concerns at the software modeling level in order to reduce the complexity of software models and application code. In this paper, we present the implementation strategies ...
A practical monadic aspect weaver
FOAL '12: Proceedings of the eleventh workshop on Foundations of Aspect-Oriented Languages

We present Monascheme, an extensible aspect-oriented programming language based on monadic aspect weaving. Extensions to the aspect language are defined as monads, enabling easy, simple and modular prototyping. The language is implemented as an embedded ...

Comments

Information & Contributors

Information

Published In

HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

June 2010

911 pages

ISBN:9781605589428

DOI:10.1145/1851476

General Chairs:
Salim Hariri
University of Arizona
,
Kate Keahey
University of Chicago

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 June 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Conference

HPDC '10

Sponsor:

University of Arizona
SIGARCH

HPDC '10: The 19th International Symposium on High Performance Distributed Computing

June 21 - 25, 2010

Illinois, Chicago

Acceptance Rates

Overall Acceptance Rate 166 of 966 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
211
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Erbel JGrabowski J(2023)Scientific workflow execution in the cloud using a dynamic runtime modelSoftware and Systems Modeling10.1007/s10270-023-01112-623:1(163-193)Online publication date: 23-Jun-2023
https://doi.org/10.1007/s10270-023-01112-6
Deelman EPeterka TAltintas ICarothers Cvan Dam KMoreland KParashar MRamakrishnan LTaufer MVetter J(2018)The future of scientific workflowsInternational Journal of High Performance Computing Applications10.5555/3195474.319547732:1(159-175)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.5555/3195474.3195477
Deelman EPeterka TAltintas ICarothers Cvan Dam KMoreland KParashar MRamakrishnan LTaufer MVetter J(2017)The future of scientific workflowsThe International Journal of High Performance Computing Applications10.1177/109434201770489332:1(159-175)Online publication date: 26-Apr-2017
https://doi.org/10.1177/1094342017704893
Ping QHe JChen C(2017)How many ways to use CiteSpace? A study of user interactive events over 14 monthsJournal of the Association for Information Science and Technology10.1002/asi.2377068:5(1234-1256)Online publication date: 1-May-2017
https://dl.acm.org/doi/10.1002/asi.23770
Hendrix VFox JGhoshal DRamakrishnan LVarela CCastro HBarrios C(2016)Tigres workflow libraryProceedings of the 16th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2016.54(146-155)Online publication date: 16-May-2016
https://dl.acm.org/doi/10.1109/CCGrid.2016.54
Anzt HHaugen BKurzak JLuszczek PDongarra J(2015)Experiences in autotuning matrix multiplication for energy minimization on GPUsConcurrency and Computation: Practice & Experience10.1002/cpe.351627:17(5096-5113)Online publication date: 10-Dec-2015
https://dl.acm.org/doi/10.1002/cpe.3516
Jain AOng SChen WMedasani BQu XKocher MBrafman MPetretto GRignanese GHautier GGunter DPersson K(2015)FireWorksConcurrency and Computation: Practice & Experience10.1002/cpe.350527:17(5037-5059)Online publication date: 10-Dec-2015
https://dl.acm.org/doi/10.1002/cpe.3505
Sankaran RAngel JBrown W(2015)Genetic algorithm based task reordering to improve the performance of batch scheduled massively parallel scientific applicationsConcurrency and Computation: Practice & Experience10.1002/cpe.345727:17(4763-4783)Online publication date: 10-Dec-2015
https://dl.acm.org/doi/10.1002/cpe.3457
Ramakrishnan LPoon SHendrix VGunter DPastorello GAgarwal D(2014)Experiences with User-Centered Design for the Tigres Workflow APIProceedings of the 2014 IEEE 10th International Conference on e-Science - Volume 0110.1109/eScience.2014.56(290-297)Online publication date: 20-Oct-2014
https://dl.acm.org/doi/10.1109/eScience.2014.56
Balderrama JSimonin MRamakrishnan LHendrix VMorin CAgarwal DTedeschi CMontagnat JTaylor I(2014)Combining workflow templates with a shared space-based execution modelProceedings of the 9th Workshop on Workflows in Support of Large-Scale Science10.1109/WORKS.2014.14(50-58)Online publication date: 16-Nov-2014
https://dl.acm.org/doi/10.1109/WORKS.2014.14
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Scripting distributed scientific workflows using Weaver

Aspect weaver: a model transformation approach for UML models

A practical monadic aspect weaver

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Funding Sources

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Scripting distributed scientific workflows using Weaver

Aspect weaver: a model transformation approach for UML models

A practical monadic aspect weaver

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations