Data-intensive supercomputing in the cloud: global analytics for satellite imagery
Authors: 
Michael S. Warren, Samuel W. Skillman, Rick Chartrand, Tim Kelton, Ryan Keisler, David Raleigh, Matthew TurkAuthors Info & Claims
Michael S. Warren, Samuel W. Skillman, Rick Chartrand, Tim Kelton, Ryan Keisler, David Raleigh, Matthew TurkAuthors Info & ClaimsPages 24 - 31
Abstract
We present our experiences using cloud computing to support data-intensive analytics on satellite imagery for commercial applications. Drawing from our background in high-performance computing, we draw parallels between the early days of clustered computing systems and the current state of cloud computing and its potential to disrupt the HPC market. Using our own virtual file system layer on top of cloud remote object storage, we demonstrate aggregate read bandwidth of 230 gigabytes per second using 512 Google Compute Engine (GCE) nodes accessing a USA multi-region standard storage bucket. This figure is comparable to the best HPC storage systems in existence. We also present several of our application results, including the identification of field boundaries in Ukraine, and the generation of a global cloud-free base layer from Landsat imagery.
References
[1]
T. E. Anderson, D. E. Culler, and D. Patterson. A case for NOW (networks of workstations). IEEE micro, 15 (1):54--64, 1995.
[2]
D. H. Bailey et al. The NAS Parallel Benchmarks. International Journal of High Performance Computing Applications, 5(3):63--73, Sept. 1991. URL http://hpc.sagepub.com/content/5/3/63.
[3]
D. J. Becker, T. Sterling, D. Savarese, J. E. Dorband, U. A. Ranawak, and C. V. Packer. BEOWULF: A parallel workstation for scientific computation. In Proceedings, International Conference on Parallel Processing, volume 95, 1995. URL http://www.phy.duke.edu/~rgb/brahma/Resources/beowulf/papers/ICPP95/icpp95.html.
[4]
G. Bell and J. Gray. What's Next in High-performance Computing? Commun. ACM, 45(2):91--95, Feb. 2002. URL http://doi.acm.org/10.1145/503124.503129.
[5]
C. Boshuizen, J. Mason, P. Klupar, and S. Spanhake. Results from the Planet Labs Flock Constellation. AIAA/USU Conference on Small Satellites, Aug. 2014. URL http://digitalcommons.usu.edu/smallsat/2014/PrivEnd/1.
[6]
J. L. Carlson. Redis in Action. Manning Publications Co., Greenwich, CT, USA, 2013. ISBN 1617290858, 9781617290855.
[7]
Celery Development Team. Celery - the distributed task queue, 2015. URL http://www.celeryproject.org.
[8]
ISO/IEC JTC 1/SC 29/WG 1. JPEG 2000 image coding system ISO/IEC 15444-1:2004 | ITU-T Rec. T.800. International Standard, 2004. URL http://www.iso.org/iso/catalogue_detail.htm?csnumber=37674.
[9]
J. D. McCalpin. Memory bandwidth and machine balance in current high performance computers. 1995. URL http://www.researchgate.net/publication/213876927_Memory_Bandwidth_and_Machine_Balance_in_Current_High_Performance_Computers.
[10]
G. E. Moore. Cramming more components onto integrated circuits. Electronics, 38(8):114--117, Apr. 1965.
[11]
A. Nuvolari. Open source software development: Some historical perspectives. First Monday, 10(10), 2005. URL http://firstmonday.org/ojs/index.php/fm/article/view/1284.
[12]
L. Oreopoulos, M. J. Wilson, and T. Várnai. Implementation on Landsat data of a simple cloud-mask algorithm developed for MODIS land bands. IEEE Geoscience and Remote Sensing Letters, 8(4):597--601, July 2011.
[13]
S. Sanfilippo and P. Noordhuis. Redis, 2010. URL http://redis.io.
[14]
S. W. Skillman, M. S. Warren, M. J. Turk, R. H. Wechsler, D. E. Holz, and P. M. Sutter. Dark Sky Simulations: Early Data Release. arXiv:1407.2600 {astro-ph}, July 2014. URL http://arxiv.org/abs/1407.2600. arXiv: 1407.2600.
[15]
D. S. Taubman and M. W. Marcellin. JPEG2000: Image Compression Fundamentals, Standards and Practice. Kluwer Academic Publishers, Norwell, MA, USA, 2001. ISBN 079237519X.
[16]
R. R. Vatsavai, S. Shekhar, T. E. Burk, and S. Lime. UMN-MapServer: A high-performance, interoperable, and open source web mapping and geo-spatial analysis system. In International Conference on Geographic Information Science, pages 400--417. Springer, 2006.
[17]
M. S. Warren and J. Wofford. Astronomical data analysis with commodity components. In Proceedings of the ACM/IEEE Conference on Supercomputing, 2007. Winner of the SC '07 Storage Challenge Award.
[18]
M. S. Warren, D. J. Becker, M. P. Goda, J. K. Salmon, and T. Sterling. Parallel supercomputing with commodity components. In Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'97), page 1372--1381, 1997. URL http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.51.2019&rep=rep1&type=pdf.
[19]
M. S. Warren, J. K. Salmon, D. J. Becker, M. P. Goda, T. Sterling, and W. Winckelmans. Pentium Pro Inside: I. A Treecode at 430 Gigaflops on ASCI Red, II. Price/Performance of $50/Mflop on Loki and Hyglac. In Supercomputing, ACM/IEEE 1997 Conference, pages 61--61, Nov. 1997.
[20]
M. S. Warren, T. C. Germann, P. S. Lomdahl, D. M. Beazley, and J. K. Salmon. Avalon: an Alpha/Linux cluster achieves 10 gflops for $150k. In Proceedings of the 1998 ACM/IEEE conference on Supercomputing (CDROM), Supercomputing '98, page 1--11, Washington, DC, USA, 1998. IEEE Computer Society. ISBN 0-89791-984-X. URL http://dl.acm.org/citation.cfm?id=509058.509130.
[21]
M. S. Warren, C. L. Fryer, and M. P. Goda. The Space Simulator: Modeling the Universe from Supernovae to Cosmology. In Proceedings of the 2003 ACM/IEEE Conference on Supercomputing, SC '03, pages 30-, New York, NY, USA, 2003. ACM. ISBN 1-58113-695-1. URL http://doi.acm.org/10.1145/1048935.1050181.
[22]
M. S. Warren et al. Seeing the earth in the cloud: Processing one petabyte of satellite imagery in one day. In 2015 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), pages 1--12. IEEE, 2015.
[23]
T. White. Hadoop: The definitive guide. " O'Reilly Media, Inc.", 2012.
- Data-intensive supercomputing in the cloud: global analytics for satellite imagery
Recommendations
Performance Evaluation of Data Intensive Computing In the Cloud
Big data is a topic of active research in the cloud community. With increasing demand for data storage in the cloud, study of data-intensive applications is becoming a primary focus. Data-intensive applications involve high CPU usage for processing ...
Pragmatic assessment of research intensive areas in cloud: a systematic review
Cloud computing is a name given to a set of systems for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly ...
Comments
Information & Contributors
Information
Published In
November 2016
62 pages
ISBN:9781509061587
Sponsors
- SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing
- IEEE-CS\DATC: IEEE Computer Society
In-Cooperation
Publisher
IEEE Press
Publication History
Published: 13 November 2016
Check for updates
Qualifiers
- Research-article
Conference
SC16
Sponsor:
- SIGHPC
- IEEE-CS\DATC
SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis
November 13 - 18, 2016
Utah, Salt Lake City
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 88Total Downloads
- Downloads (Last 12 months)1
- Downloads (Last 6 weeks)0
Reflects downloads up to 10 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in