An Efficient Astronomical Cross-matching model Based on MapReduce Mechanism
Authors: 
Kuei-sheng Lee, Meng-feng Tsai, Yuji Urata, Kui-yun Huang, Chi-sheng HuangAuthors Info & Claims
Kuei-sheng Lee, Meng-feng Tsai, Yuji Urata, Kui-yun Huang, Chi-sheng HuangAuthors Info & ClaimsArticle No.: 38, Pages 1 - 4
Abstract
In order to perform an effective cross-matching computation on an enormous amount of text-file-based astronomical observation data, this study proposes an algorithm based on the MapReduce distributed architecture. Such an approach not only greatly enhances the computation speed, but also provides a data structure for storing the computation results. It provides a satisfactory solution not only for cross-matching the entirety of the data, but also for simply updating the changes.
References
[1]
Gray, J., Szalay, A., Budavári, T., Lupton, R., Nieto-Santisteban, M. and Thakar, A. Cross-matching multiple spatial observations and dealing with missing data. arXiv preprint cs/07011722007).
[2]
Gray, J., Nieto-Santisteban, M. A. and Szalay, A. S. The zones algorithm for finding points-near-a-point or cross-matching spatial datasets. arXiv preprint cs/07011712007).
[3]
Zhao, Q., Sun, J., Yu, C., Cui, C., Lv, L. and Xiao, J. A paralleled large-scale astronomical cross-matching function. Springer, City, 2009.
[4]
Dean, J. and Ghemawat, S. MapReduce: simplified data processing on large clusters. Communications of the ACM, 51, 1 2008), 107--113.
[5]
Mi, C., Chen, Q. and Liu, T. An efficient cross-match implementation based on directed join algorithm in MapReduce. IEEE, City, 2011.
[6]
Li, L., Tang, D., Liu, T., Liu, H., Li, W. and Cui, C. Optimizing the Join Operation on Hive to Accelerate Cross-Matching in Astronomy. IEEE, City, 2014.
[7]
Aji, A., Wang, F., Vo, H., Lee, R., Liu, Q., Zhang, X. and Saltz, J. Hadoop GIS: a high performance spatial data warehousing system over mapreduce. Proceedings of the VLDB Endowment, 6, 11 2013), 1009--1020.
[8]
Subaru Telescope. Retrieved September 1, 2015, from http://subarutelescope.org/index.html
[9]
Pan-STARRS. Retrieved September 1, 2015, from http://pan-starrs.ifa.hawaii.edu/public/
Index Terms
- An Efficient Astronomical Cross-matching model Based on MapReduce Mechanism
Recommendations
MapReduce: Review and open challenges
The continuous increase in computational capacity over the past years has produced an overwhelming flow of data or big data, which exceeds the capabilities of conventional processing tools. Big data signify a new era in data exploration and utilization. ...
An Efficient Cross-Match Implementation Based on Directed Join Algorithm in MapReduce
UCC '11: Proceedings of the 2011 Fourth IEEE International Conference on Utility and Cloud ComputingIn the field of astronomy, "Cross-Match" is a common operation used to mine useful information by joining different star catalogues. Nowadays star catalogues obtained through astronomical telescopes are becoming much larger than ever before, which ...
Cross-MapReduce: Data transfer reduction in geo-distributed MapReduce
AbstractThe MapReduce model is widely used to store and process big data in a distributed manner. MapReduce was originally developed for a single tightly coupled cluster of computers. Approaches such as Hierarchical and Geo-Hadoop are designed to address ...
Highlights- Proposes a Cross-MapReduce to process geo-distributed data.
- Cross-MapReduce creates a GRG graph to reduce the data transfer over the Internet.
- Cross-MapReduce transfers keys instead of the total intermediate results and uses the ...
Comments
Information & Contributors
Information
Published In
October 2015
381 pages
ISBN:9781450337359
DOI:10.1145/2818869
Copyright © 2015 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 October 2015
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ASE BD&SI '15
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 66Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)0
Reflects downloads up to 11 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in