Abstract
Big data has been an important analysis method anywhere we turn today. We hold broad recognition of the value of data, and products obtained through analyzing it. There are multiple steps to the data analysis pipeline, which can be abstracted as a framework provides universal parallel high-performance data analysis. Based on ray, this paper proposed a parallel framework written in Python with an interface to aggregate and analyze homogeneous astronomical sky survey time series data. As such, we can achieve parallel training and analysis only by defining the customized analyze functions, decision module and I/O interfaces, while the framework is able to manage the pipeline such as data fetching, saving, parallel job scheduling and load balancing. Meanwhile, the data scientists can focus on the analysis procedure and save the time speeding this program up. We tested out the framework on synthetic data with raw files and HBase entries as data sources and result formats, reduced the analyze cost for scientists not familiar with parallel programming while needs to handle a mass of data. We integrate time series anomaly detection algorithms with our parallel dispatching module to achieve high-performance data processing frameworks. Experimental results on synthetic astronomical sky survey time series data show that our model achieves good speed up ratio in executing analysis programs.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Python. https://en.wikipedia.org/wiki/Python_(programming_language)
Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. In: OSDI 2016, pp. 265–283 (2016)
Chen, Y., Mahajan, R., Sridharan, B., Zhang, Z.-L.: A provider-side view of web search response time. In: ACM SIGCOMM Computer Communication Review, vol. 43, pp. 243–254. ACM (2013)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Godet, O., et al.: The Chinese-French SVOM mission: studying the brightest astronomical explosions. In: Space Telescopes and Instrumentation 2012: Ultraviolet to Gamma Ray, vol. 8443, p. 84431O. International Society for Optics and Photonics (2012)
Walker, S.J.: Big Data: A Revolution That Will Transform How We Live, Work, and Think (2014)
Liang, E., et al.: Ray RLLib: a composable and scalable reinforcement learning library. CoRR, abs/1712.09381 (2017)
Moritz, P., et al.: Ray: a distributed framework for emerging AI applications (2017)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12(Oct), 2825–2830 (2011)
R Core Team: R: a language and environment for statistical computing (2013)
Vallis, O., Hochenbaum, J., Kejariwal, A.: A novel technique for long-term anomaly detection in the cloud. In: HotCloud (2014)
Vora, M.N.: Hadoop-HBase for large-scale data. In: 2011 International Conference on Computer Science and Network Technology (ICCSNT), vol. 1, pp. 601–605. IEEE (2011)
Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: Usenix Conference on Hot Topics in Cloud Computing, p. 10 (2010)
Acknowledgements
This research is supported in part by Key Research and Development Program of China (No. 2016YFB1000602), “the Key Laboratory of Space Astronomy and Technology, National Astronomical Observatories, Chinese Academy of Sciences, Beijing, 100012, China”, National Natural Science Foundation of China (Nos. 61440057, 61272087, 61363019 and 61073008, 11690023), MOE research center for online education foundation (No. 2016ZD302).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Ma, X., Du, Z., Sun, Y., Tchernykh, A., Wu, C., Wei, J. (2019). An Efficient Parallel Framework to Analyze Astronomical Sky Survey Data. In: Li, J., Meng, X., Zhang, Y., Cui, W., Du, Z. (eds) Big Scientific Data Management. BigSDM 2018. Lecture Notes in Computer Science(), vol 11473. Springer, Cham. https://doi.org/10.1007/978-3-030-28061-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-28061-1_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28060-4
Online ISBN: 978-3-030-28061-1
eBook Packages: Computer ScienceComputer Science (R0)