research-article

Seeking supernovae in the clouds: a performance study

Authors:

Keith R. Jackson,

Lavanya Ramakrishnan,

Rollin C. ThomasAuthors Info & Claims

HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Pages 421 - 429

https://doi.org/10.1145/1851476.1851538

Published: 21 June 2010 Publication History

Abstract

Today, our picture of the Universe radically differs from that of just over a decade ago. We now know that the Universe is not only expanding as Hubble discovered in 1929, but that the rate of expansion is accelerating, propelled by mysterious new physics dubbed "Dark Energy." This revolutionary discovery was made by comparing the brightness of nearby Type Ia supernovae (which exploded in the past billion years) to that of much more distant ones (from up to seven billion years ago). The reliability of this comparison hinges upon a very detailed understanding of the physics of the nearby events. As part of its effort to further this understanding, the Nearby Supernova Factory (SNfactory) relies upon a complex pipeline of serial processes that execute various image processing algorithms in parallel on ~10TBs of data.

This pipeline has traditionally been run on a local cluster. Cloud computing offers many features that make it an attractive alternative. The ability to completely control the software environment in a Cloud is appealing when dealing with a community developed science pipeline with many unique library and platform requirements. In this context we study the feasibility of porting the SNfactory pipeline to the Amazon Web Services environment. Specifically we: describe the tool set we developed to manage a virtual cluster on Amazon EC2, explore the various design options available for application data placement, and offer detailed performance results and lessons learned from each of the above design options.

References

[1]

}}Amazon EBS. http://aws.amazon.com/ebs/

[2]

}}Amazon EC2. http://aws.amazon.com/ec2/

[3]

}}Amazon S3. http://aws.amazon.com/s3/

[4]

}}Amazon Web Services. http://aws.amazon.com/

[5]

}}BLAS. http://www.netlib.org/blas/

[6]

}}CFITSIO. http://heasarc.nasa.gov/docs/software/fitsio/fitsio.html

[7]

}}GSL. http://www.gnu.org/software/gsl/

[8]

}}Sar. http://pagesperso-orange.fr/sebastien.godard/

[9]

}}Aldering, G. et al., Overview of the Nearby Supernova Factory. City, 2002.

[10]

}}Anderson, E., Bai, Z., Dongarra, J., Greenbaum, A., McKenney, A., Du Croz, J., Hammerling, S., Demmel, J., Bischof, C. and Sorensen, D. LAPACK: A portable linear algebra library for high-performance computers. IEEE Computer Society Press, City, 1990.

[11]

}}Aragon, C. R., Poon, S. S., Aldering, G. S., Thomas, R. C. and Quimby, R. Using visual analytics to develop situation awareness in astrophysics. Information Visualization, 8, 1 2009), 30--41.

Digital Library

[12]

}}Deelman, E., Singh, G., Livny, M., Berriman, B. and Good, J. The Cost of Doing Science on the Cloud: The Montage Example. City, 2008.

[13]

}}Evangelinos, C. and Hill, C. N. Cloud computing for Parallel Scientific HPC Applications: Feasibility of Running Coupled Atmosphere-Ocean Climate Models on Amazon's EC2. City, 2008.

[14]

}}Keahey, K. and Freeman, T. Science Clouds: Early Experiences in Cloud Computing for Scientific Applications. City, 2008.

[15]

}}Keahey, K., Freeman, T., Lauret, J. and Olson, D. Virtual Workspaces for Scientific Applications. Physics: Conference Series, 78, 012038 (2007 2007), 5.

[16]

}}Palankar, M. R., Iamnitchi, A., Ripeanu, M. and Garfinkel, S. Amazon S3 for science grids: a viable solution?, City, 2008.

[17]

}}Perlmutter, S., et al. Measurements of Omega and Lambda from 42 High-Redshift Supernovae. Astrophys. J., 517, (1999), 565--586.

[18]

}}Riess, A., et al., Observational Evidence from Supernovae for an Accelerating Universe and a Cosmological Constant, Astron. J., 116, (1998), 1009--1038.

[19]

}}Wells, D. C., Greisen, E. and Harten, R. H. FITS: A Flexible Image Transport System. Astron. Astrophys. Suppl., 44, (1981), 363--370.

[20]

}}Armbrust, M., Fox, A., Griffith, R., Joseph, A., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A. and Stoica, I. Above the clouds: A berkeley view of cloud computing. EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2009-282009).

Cited By

Rosseti IOcaña Kde Oliveira DMontagnat JTaylor IGesing SSakellariou R(2017)Towards preserving results confidentiality in cloud-based scientific workflowsProceedings of the 12th Workshop on Workflows in Support of Large-Scale Science10.1145/3150994.3151002(1-9)Online publication date: 12-Nov-2017
https://dl.acm.org/doi/10.1145/3150994.3151002
Phaswana MHancke GRamotsoela T(2017)Autonomous monitoring system for a public residential street2017 IEEE 26th International Symposium on Industrial Electronics (ISIE)10.1109/ISIE.2017.8001567(2032-2037)Online publication date: Jul-2017
https://doi.org/10.1109/ISIE.2017.8001567
de Jesus LDrummond Lde Oliveira D(2017)Eeny Meeny Miny Moe: Choosing the Fault Tolerance Technique for my Cloud WorkflowHigh Performance Computing10.1007/978-3-319-73353-1_23(321-336)Online publication date: 28-Dec-2017
https://doi.org/10.1007/978-3-319-73353-1_23
Show More Cited By

Index Terms

Seeking supernovae in the clouds: a performance study
1. Computing methodologies
  1. Distributed computing methodologies
    1. Distributed programming languages
2. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language types
        Distributed programming languages

Recommendations

Virtual Infrastructure Management in Private and Hybrid Clouds

One of the many definitions of "cloud" is that of an infrastructure-as-a-service (IaaS) system, in which IT infrastructure is deployed in a provider's data center as virtual machines. With IaaS clouds' growing popularity, tools and technologies are ...
Execution of Compute Intensive Applications on Hybrid Clouds (Case Study with mpiBLAST)
CISIS '12: Proceedings of the 2012 Sixth International Conference on Complex, Intelligent, and Software Intensive Systems (CISIS)

Cloud computing is the most recent realisation of computing as a utility. Recently, fields with substantial computational requirements, e.g., biology, are turning to clouds for cheap, on-demand provisioning of resources. Of interest to this paper is the ...
Automated control for SLA-aware elastic clouds
FeBiD '10: Proceedings of the Fifth International Workshop on Feedback Control Implementation and Design in Computing Systems and Networks

Although Cloud Computing provides a means to support remote, on-demand access top a set of computing resources, its ad-hoc management for quality-of-service and SLA poses significant challenges to the performance, availability and economical costs of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

June 2010

911 pages

ISBN:9781605589428

DOI:10.1145/1851476

General Chairs:
Salim Hariri
University of Arizona
,
Kate Keahey
University of Chicago

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

University of Arizona: University of Arizona
SIGARCH: ACM Special Interest Group on Computer Architecture

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 June 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Advanced Scientific Computing Research

Conference

HPDC '10

Sponsor:

University of Arizona
SIGARCH

HPDC '10: The 19th International Symposium on High Performance Distributed Computing

June 21 - 25, 2010

Illinois, Chicago

Acceptance Rates

Overall Acceptance Rate 166 of 966 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
316
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rosseti IOcaña Kde Oliveira DMontagnat JTaylor IGesing SSakellariou R(2017)Towards preserving results confidentiality in cloud-based scientific workflowsProceedings of the 12th Workshop on Workflows in Support of Large-Scale Science10.1145/3150994.3151002(1-9)Online publication date: 12-Nov-2017
https://dl.acm.org/doi/10.1145/3150994.3151002
Phaswana MHancke GRamotsoela T(2017)Autonomous monitoring system for a public residential street2017 IEEE 26th International Symposium on Industrial Electronics (ISIE)10.1109/ISIE.2017.8001567(2032-2037)Online publication date: Jul-2017
https://doi.org/10.1109/ISIE.2017.8001567
de Jesus LDrummond Lde Oliveira D(2017)Eeny Meeny Miny Moe: Choosing the Fault Tolerance Technique for my Cloud WorkflowHigh Performance Computing10.1007/978-3-319-73353-1_23(321-336)Online publication date: 28-Dec-2017
https://doi.org/10.1007/978-3-319-73353-1_23
Coutinho RFrota YOcaña Kde Oliveira DDrummond L(2017)Mirror Mirror on the Wall, How Do I Dimension My Cloud After All?Cloud Computing10.1007/978-3-319-54645-2_2(27-58)Online publication date: 3-Jun-2017
https://doi.org/10.1007/978-3-319-54645-2_2
Tudoran RCostan AAntoniu G(2016)OverFlowIEEE Transactions on Cloud Computing10.1109/TCC.2015.24402544:1(76-89)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.1109/TCC.2015.2440254
Atif MKobayashi RMenadue BLin CSanderson MWilliams A(2016)Breaking HPC Barriers with the 56GbE CloudProcedia Computer Science10.1016/j.procs.2016.07.17493(3-11)Online publication date: 2016
https://doi.org/10.1016/j.procs.2016.07.174
Coutinho RFrota YOcaña KOliveira DDrummond L(2016)A Dynamic Cloud Dimensioning Approach for Parallel Scientific WorkflowsJournal of Grid Computing10.1007/s10723-016-9367-x14:3(443-461)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1007/s10723-016-9367-x
Kalra SSood S(2015)Secure authentication scheme for IoT and cloud serversPervasive and Mobile Computing10.1016/j.pmcj.2015.08.00124:C(210-223)Online publication date: 1-Dec-2015
https://dl.acm.org/doi/10.1016/j.pmcj.2015.08.001
Paes Ade Oliveira D(2015)Running Multi-relational Data Mining Processes in the Cloud: A Practical Approach for Social NetworksHigh Performance Computing10.1007/978-3-319-26928-3_1(3-18)Online publication date: 12-Dec-2015
https://doi.org/10.1007/978-3-319-26928-3_1
Tudoran RKeahey KRiteau PPanitkin SAntoniu GReed DSun XFoster I(2014)Evaluating streaming strategies for event processing across infrastructure cloudsProceedings of the 14th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2014.89(151-159)Online publication date: 26-May-2014
https://dl.acm.org/doi/10.1109/CCGrid.2014.89
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents