Abstract
Multi-wavelength data cross-match among multiple catalogs is a basic and unavoidable step to make distributed digital archives accessible and interoperable. As current catalogs often contain millions or billions objects, it is a typical data-intensive computation problem. In this paper, a high-efficient parallel approach of astronomical cross-match is introduced. We issue our partitioning and parallelization approach, after that we address some problems introduced by task partition and give the solutions correspondingly, including a sky splitting function HEALPix we selected which play a key role on both the task partitioning and the database indexing, and a quick bit-operation algorithm we advanced to resolve the block-edge problem. Our experiments prove that the function has a marked performance superiority comparing with the previous functions and is fully applicable to large-scale cross-match.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Nieto-Santisteban, M.A., Thakar, A.R., Szalay, A.S.: Cross-Matching Very Large Datasets. Johns Hopkins University, Baltimore (2006)
Nieto-Santisteban, M.A., Thakar, A.R., Szalay, A.S., Gray, J.: Large-Scale Query and XMatch, Entering the Parallel Zone. In: Astronomical Data Analysis Software and Systems XV ASP Conference Series, El Escoreal, Spain (2005)
Gray, J., Szalay, A., Budavri, T., Thakar, A.R., Nieto-Santisteban, M.A., Thakar, A.: Cross-Matching Multiple Spatial Observations and Dealing with Missing Data. Microsoft Technical Report, MSR-TR-2006-175, Redmond, WA (2006)
Gray, J., Szalay, A., Fekete, G.: Using Table Valued Functions in SQL Server 2005 To Implement a Spatial Data Library. MSR-TR-2005-122. Microsoft Technical Report, Redmond, WA (2005)
Gray, J., Nieto-Santisteban, M.A., Szalay, A.S.: The Zones Algorithm for Finding Points-Near-a-Point or Cross-Matching Spatial Datasets. MSR-TR-2006-52. Microsoft Technical Report, Redmond, WA (2006)
Gray, J., Szalay, A.S., Nieto-Santisteban, M.A., Heber, G., Rots, A.H.: There Goes the Neighborhood: Relational Algebra for Spatial Data Search. MSR-TR-2004-32. Microsoft Technical Report, Redmond, WA (2004)
Report on Cross Matching Catalogues. Technical report, AstroGrid, http://wiki.astrogrid.org/pub/Astrogrid/DataFederationandDataMining/cross.htm
Spatial Joins and Spatial Indexing Revisted. Technical report, AstroGrid, http://wiki.astrogrid.org/bin/view/Astrogrid/SpatialIndexing
Clive Page: Indexing the Sky. Technical Report, AstroGrid (2002)
Clive Page: Comments on the XMATCH function in ADQL. Technical report, AstroGrid (2004)
Dan, G., Zhang, Y.X., Zhao, Y.H.: Implementation of Cross-Matching on Very Large Multi-Wavelength catalogs. In: Astronomical Research and Technology. publications of National Astronomical Observatories of China (2005)
Dan, G.: Very Large Astronomical Data Sets Fusion System’s Development and Data Mining Algorithms’ Research. Doctor’s degree dissertation, National Astronomical Observatories of Chinese Academy of Sciences (2008)
Dan, G.: A System Integrated with Query, Cross-matching and Visualization. In: Lewis, H., Bridger, A. (eds.) Advanced Software and Control for Astronomy, vol. 6274. SPIE (2006)
Dan, G., Zhang, Y.X.: The Application of kd-tree in Astronomy. In: Astronomical Data Analysis Software and Systems XV2, ASP Conference Series (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhao, Q., Sun, J., Yu, C., Cui, C., Lv, L., Xiao, J. (2009). A Paralleled Large-Scale Astronomical Cross-Matching Function. In: Hua, A., Chang, SL. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2009. Lecture Notes in Computer Science, vol 5574. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03095-6_57
Download citation
DOI: https://doi.org/10.1007/978-3-642-03095-6_57
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-03094-9
Online ISBN: 978-3-642-03095-6
eBook Packages: Computer ScienceComputer Science (R0)