Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content

Advertisement

SpringerLink
Log in

Real-time anomaly detection in sky quality meter data using probabilistic exponential weighted moving average

  • Review
  • Published:
International Journal of Data Science and Analytics Aims and scope Submit manuscript

Abstract

Light pollution is a problem that impacts many elements of human life and the environment, including astronomical observations. The authors of this work offer a unique method for detecting anomalies in night sky brightness data recorded using a Sky Quality Meter (SQM). This equipment has been widely utilized in light pollution research worldwide, yielding massive data. However, there is the possibility of experiencing abnormalities or outliers throughout the data collection process due to natural occurrences or measurement errors. This study uses the probabilistic exponential weighted moving average algorithm to find anomalies in SQM data received from Timau Observatory by simulating the streaming procedure on SQM data using Apache Kafka technology. Finally, this study intends to shed fresh knowledge on night sky brightness and light pollution dynamics. The authors could locate and analyze unusual or suspicious phenomena that had previously gone unreported using the anomaly detection approach. These findings can help us better understand light pollution and its environmental and human life effects. Still, they can also help us establish strategies and policies that will reduce light pollution in the future. Furthermore, this work illustrates the potential of anomaly detection as a powerful tool for data analysis in various domains, encouraging the use of this approach in future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

Statement enquiries about data availability should be directed to the authors.

References

  1. Abe, N., Melville, P., Pendus, C., Reddy, C. K., Jensen, D. L., Thomas, V. P., Bennett, J. J., Anderson, G. F., Cooley, B. R., & Kowalczyk, M.: Optimizing debt collections using constrained reinforcement learning. 75–84 (2010)

  2. Admiranto, A.G., Haida, S., Priyatikanto, R., Maryam, S., Ellyyani, Suryana, N.: Mobile campaign of sky brightness measurement in Indonesia. J. Phys. Conf. Ser. 1523(1), 012002 (2020). https://doi.org/10.1088/1742-6596/1523/1/012002

    Article  Google Scholar 

  3. Aggarwal, C.C.: Outlier Analysis. Springer (2013)

    Book  Google Scholar 

  4. Ahmad, S., Lavin, A., Purdy, S., Agha, Z.: Unsupervised real-time anomaly detection for streaming data. Neurocomputing 262, 134–147 (2017). https://doi.org/10.1016/j.neucom.2017.04.070

    Article  Google Scholar 

  5. Ahmad, S., & Purdy, S.: Real-time anomaly detection for streaming analytics (arXiv:1607.02480). arXiv. http://arxiv.org/abs/1607.02480 (2016)

  6. Akoglu, L., Tong, H., & Koutra, D.: Graph-based Anomaly Detection and Description: A Survey (2014) (arXiv:1404.4679; Issue arXiv:1404.4679). http://arxiv.org/abs/1404.4679

  7. Alarcon, M.R., Serra-Ricart, M., Lemes-Perera, S., Mallorquín, M.: Natural night sky brightness during solar minimum. Astron. J. 162(1), 25 (2021)

    Article  Google Scholar 

  8. Asmoro, C.P., Wijaya, A.F.C., Ardi, N.D., Abdurrohman, A., Utama, J.A., Sutiadi, A., Hikmat, A., Ramalis, T.R., Suyardi, B.: The assembled solar eclipse package (asep) in Bangka Indonesia during the total solar eclipse on March 9, 2016. J. Phys. Conf. Ser. 771, 012020 (2016). https://doi.org/10.1088/1742-6596/771/1/012020

    Article  Google Scholar 

  9. Bará, S., Lima, R.C., Zamorano, J.: Monitoring long-term trends in the anthropogenic night sky brightness. Sustainability 11(11), 3070 (2019). https://doi.org/10.3390/su11113070

    Article  Google Scholar 

  10. Bertolo, A., Binotto, R., Ortolani, S., Sapienza, S.: Measurements of night sky brightness in the veneto region of italy: sky quality meter network results and differential photometry by digital single lens reflex. J. Imag. 5(5), 56 (2019)

    Article  Google Scholar 

  11. Bolton, R. J., & Hand, D. J.: Unsupervised profiling methods for fraud detection. Credit Scoring and Credit Control VII, 235–255 (2001)

  12. Burne, B.: Pollution by light. Lancet 299(7751), 642 (1972)

    Article  Google Scholar 

  13. Castillo, C., Donato, D., Gionis, A., Murdock, V., Silvestri, F.: Know your neighbors: web spam detection using the web topology. 423–430 (2007)

  14. Cavazzani, S., Ortolani, S., Bertolo, A., Binotto, R., Fiorentin, P., Carraro, G., Saviane, I., Zitelli, V.: Sky Quality Meter and satellite correlation for night cloud-cover analysis at astronomical sites. Mon. Not. R. Astron. Soc. 493(2), 2463–2471 (2020). https://doi.org/10.1093/mnras/staa416

    Article  Google Scholar 

  15. Cinzano, P.: Night sky photometry with sky quality meter. ISTIL Int. Rep, 9(1) (2005)

  16. Cortes, C., Pregibon, D., Volinsky, C.: Communities of interest. Intell. Data Anal. 6(3), 211–219 (2002)

    Article  Google Scholar 

  17. Damnjanovic, U., Fernandez, V., Izquierdo, E., Martinez, J. M.: Event detection and clustering for surveillance video summarization. 63–66 (2008)

  18. Das, K., & Schneider, J.: Detecting anomalous records in categorical datasets. 220–229 (2007)

  19. Ding, Q., Katenka, N., Barford, P., Kolaczyk, E., Crovella, M. Intrusion as (anti) social communication: Characterization and detection. 886–894 (2012)

  20. D’silva, G. M., Khan, A., Gaurav, Bari, S.: Real-time processing of IoT events with historic data using Apache Kafka and Apache Spark with dashing framework. In: 2017 2nd IEEE international conference on recent trends in electronics, information & communication technology (RTEICT), 1804–1809. (2017) https://doi.org/10.1109/RTEICT.2017.8256910

  21. Eberle, W., Holder, L.: Discovering structural anomalies in graph-based data. 393–398 (2007)

  22. Eberle, W., Holder, L.: Graph-based approaches to insider threat detection. 1–4 (2009)

  23. Espey, B., McCauley, J.: Initial Irish light pollution measurements and a new sky quality meter-based data logger. Light. Res. Technol. 46(1), 67–77 (2014). https://doi.org/10.1177/1477153513515508

    Article  Google Scholar 

  24. Faid, M. S., Husien, N., Shariff, N. N. M., Ali, M. O., Hamidi, Z. S., Zainol, N. H., Sabri, S. N. U.: Monitoring the level of light pollution and its impact on astronomical bodies naked-eye visibility range in selected areas in Malaysia using the sky quality meter. In: 2016 International conference on industrial engineering, management science and application (ICIMSA), 1–6 (2016). https://doi.org/10.1109/ICIMSA.2016.7504020

  25. Fawcett, T., & Provost, F.: Activity monitoring: Noticing interesting changes in behavior. 53–62 (1999)

  26. Fawcett, T., Provost, F.J.: Combining data mining and machine learning for effective user profiling. KDD 96, 8–13 (1996)

    Google Scholar 

  27. Fox, A.J.: Outliers in time series. J. R. Stat. Soc. Ser. B Stat Methodol. 34(3), 350–363 (1972)

    Article  MathSciNet  Google Scholar 

  28. Goernitz, N., Kloft, M., Rieck, K., Brefeld, U.: Toward supervised anomaly detection. J. Artif. Intell. Res. 46, 235–262 (2013). https://doi.org/10.1613/jair.3623

    Article  MathSciNet  Google Scholar 

  29. Hänel, A., Posch, T., Ribas, S.J., Aubé, M., Duriscoe, D., Jechow, A., Kollath, Z., Lolkema, D.E., Moore, C., Schmidt, N., Spoelstra, H., Wuchterl, G., Kyba, C.C.M.: Measuring night sky brightness: methods and challenges. J. Quant. Spectrosc. Radiat. Transfer 205, 278–290 (2018). https://doi.org/10.1016/j.jqsrt.2017.09.008

    Article  Google Scholar 

  30. Idé, T., & Kashima, H.: Eigenspace-based anomaly detection in computer systems. 440–449 (2004)

  31. Invernizzi, L., Comparetti, P. M., Benvenuti, S., Kruegel, C., Cova, M., Vigna, G.:. Evilseed: a guided approach to finding malicious web pages (2012). 428–442

  32. Jechow, A., Kolláth, Z., Ribas, S.J., Spoelstra, H., Hölker, F., Kyba, C.C.: Imaging and mapping the impact of clouds on skyglow with all-sky photometry. Sci. Rep. 7(1), 6741 (2017)

    Article  Google Scholar 

  33. Kocifaj, M.: Light-pollution model for cloudy and cloudless night skies with ground-based light sources. Appl. Opt. 46(15), 3013 (2007). https://doi.org/10.1364/AO.46.003013

    Article  Google Scholar 

  34. Krausz, B., Herpers, R.: MetroSurv: detecting events in subway stations. Multimed. Tools Appl. 50, 123–147 (2010)

    Article  Google Scholar 

  35. Kshetri, N.: The economics of click fraud. IEEE Secur. Priv. 8(3), 45–53 (2010)

    Article  Google Scholar 

  36. Kumar, M., Ghani, R., Mei, Z.-S.:. Data mining to predict and prevent errors in health insurance claims processing. 65–74 (2010)

  37. Kyba, C.C., Hänel, A., Hölker, F.: Redefining efficiency for outdoor lighting. Energy Environ. Sci. 7(6), 1806–1809 (2014)

    Article  Google Scholar 

  38. Le Noach, P., Costan, A., Bouge, L.: A performance evaluation of Apache Kafka in support of big data streaming applications. In: 2017 IEEE International Conference on Big Data (Big Data), 4803–4806 (2017). https://doi.org/10.1109/BigData.2017.8258548

  39. Lee, K., Caverlee, J., Webb, S.: Uncovering social spammers: Social honeypots+ machine learning (2010) 435–442

  40. Li, L., Liang, C.-J. M., Liu, J., Nath, S., Terzis, A., Faloutsos, C.: Thermocast: A cyber-physical forecasting model for datacenters. (2011) 1370–1378.

  41. Ma, J., Saul, L. K., Savage, S., Voelker, G. M.: Beyond blacklists: Learning to detect malicious web sites from suspicious URLs (2009) 1245–1254.

  42. McGlohon, M., Bay, S., Anderle, M. G., Steier, D. M., Faloutsos, C.: Snare: A link analytic system for graph labeling and risk detection (2009) 1265–1274.

  43. Narkhede, N., Shapira, G., Palino, T.: Kafka: The definitive guide: real-time data and stream processing at scale (First edition). O’Reilly Media (2017)

  44. Neville, J., Şimşek, Ö., Jensen, D., Komoroske, J., Palmer, K., Goldberg, H.: Using relational knowledge discovery to prevent securities fraud (2005) 449–458.

  45. Odoh, K.: Real-time Anomaly detection for multivariate data streams. (2022) https://doi.org/10.48550/ARXIV.2209.12398

  46. Ott, M., Cardie, C., Hancock, J.: Estimating the prevalence of deception in online review communities. 201–210 (2012)

  47. Pandit, S., Chau, D. H., Wang, S., Faloutsos, C.: Netprobe: A fast and scalable system for fraud detection in online auction networks (2007). 201–210

  48. Phua, C., Alahakoon, D., Lee, V.: Minority report in fraud detection: classification of skewed data. ACM SIGKDD Explor. Newsl. 6(1), 50–59 (2004)

    Article  Google Scholar 

  49. Podor, A., Huszar, G.: Detecting light pollution with UAV, a Hungarian case study. 2022 IEEE 22nd international symposium on computational intelligence and informatics and 8th ieee international conference on recent achievements in mechatronics, automation, computer science and robotics (CINTI-MACRo), 000197–000202 (2022). https://doi.org/10.1109/CINTI-MACRo57952.2022.10029545

  50. Posch, T., Binder, F., Puschnig, J.: Systematic measurements of the night sky brightness at 26 locations in Eastern Austria. J. Quant. Spectrosc. Radiat. Transfer 211, 144–165 (2018)

    Article  Google Scholar 

  51. Priyatikanto, R., Mumpuni, E.S., Hidayat, T., Saputra, M.B., Murti, M.D., Rachman, A., Yatini, C.Y.: Characterization of timau national observatory using limited in situ measurements. Month. Notices R. Astron. Soc. (2022). https://doi.org/10.1093/mnras/stac3349

    Article  Google Scholar 

  52. Provos, N., McNamee, D., Mavrommatis, P., Wang, K., Modadugu, N.: The ghost in the browser: analysis of web-based malware. HotBots 7, 4–4 (2007)

    Google Scholar 

  53. Riegel, K.W.: Light pollution: outdoor lighting is a growing threat to astronomy. Science 179(4080), 1285–1291 (1973). https://doi.org/10.1126/science.179.4080.1285

    Article  Google Scholar 

  54. Riza, L.S., Izzuddin, A., Utama, J.A., Samah, K.A.F.A., Herdiwijaya, D., Hidayat, T., Anugraha, R., Mumpuni, E.S.: Data analysis techniques in light pollution: a survey and taxonomy. New Astron. Rev. (2022). https://doi.org/10.1016/j.newar.2022.101663

    Article  Google Scholar 

  55. Rodrigues, P., Aubrecht, C., Gil, A., Longcore, T., Elvidge, C.: Remote sensing to map influence of light pollution on Cory’s shearwater in São Miguel Island, Azores Archipelago. Eur. J. Wildl. Res. 58(1), 147–155 (2012). https://doi.org/10.1007/s10344-011-0555-5

    Article  Google Scholar 

  56. Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. Int. Conf. Inf. Process. Med. Imaging (2017). https://doi.org/10.48550/ARXIV.1703.05921

    Article  Google Scholar 

  57. Schmidt, W., & Spoelstra, H.: Darkness monitoring in the Netherlands 2009-2019 (2020). https://doi.org/10.5281/zenodo.4293366

  58. Setyanto, H., Prastyo, H.A., Basthoni, M., Fuscha, F.A., Al Saab, S.M.: Zodiac light detection based on sky quality meter (SQM) Data: preliminary study. Al-Hilal J. Islam. Astron. 3(2), 121–134 (2021). https://doi.org/10.21580/al-hilal.2021.3.2.8477

    Article  Google Scholar 

  59. Small, C., Elvidge, C.D.: Night on earth: mapping decadal changes of anthropogenic night light in Asia. Int. J. Appl. Earth Obs. Geoinf. 22, 40–52 (2013)

    Google Scholar 

  60. Sun, J., Xie, Y., Zhang, H., Faloutsos, C.: Less is more: Sparse graph mining with compact matrix decomposition. Stat Anal. Data Min. ASA Data Sci. J. 1(1), 6–22 (2008)

    Article  MathSciNet  Google Scholar 

  61. Taniguchi, M., Haft, M., Hollmén, J., Tresp, V.: Fraud detection in communication networks using neural and probabilistic methods. Int. Conf. Acoust. Speech Signal Process. 2, 1241–1244 (1998)

    Google Scholar 

  62. Wu, R.-S., Ou, C.-S., Lin, H., Chang, S.-I., Yen, D.C.: Using data mining technique to enhance tax evasion detection performance. Expert Syst. Appl. 39(10), 8769–8777 (2012)

    Article  Google Scholar 

  63. Yilmaz, A., Özdemir, T.: Measurement and determination of light pollution: case study of Malatya city. Turk. J. Astron. Astrophys. 2(1), 38–43 (2021)

    Google Scholar 

  64. Zamorano, J., Garcıa, C., González, R., Tapia, C., de Miguel, A.S., Pascual, S., González, E., Gallego, J., Picazo, P., Izquierdo, J., et al.: STARS4ALL, a light pollution awareness project. Highlights Spanish Astrophys. 8, 780–783 (2017)

    Google Scholar 

Download references

Funding

The first authors would like to acknowledge the Ministry of Research and Technology, Research and Community Services Institutions of Universitas Pendidikan Indonesia on Indonesia Collaboration Research (RKI) for funding this work through the research grant of 913/UN40/PT.01.02/2023.

Author information

Authors and Affiliations

  1. Computer Science Education, Universitas Pendidikan Indonesia, Bandung, Indonesia

    Lala Septem Riza, Zulfikar Ali Yunara Putra, Muhammad Iqbal Zain & Fajar Zuliansyah Trihutama

  2. Physics Study Program, Universitas Pendidikan Indonesia, Bandung, Indonesia

    Judhistira Aria Utama

  3. College of Computing, Informatics and Media, Universiti Teknologi MARA (UiTM) Melaka Branch, Melaka, Malaysia

    Khyrina Airin Fariza Abu Samah

  4. Astronomy Research Division, Institut Teknologi Bandung, Bandung, 40132, Indonesia

    Dhani Herdiwijaya

  5. Faculty of Mathematics and Natural Sciences, Universitas Gadjah Mada, Yogyakarta, 55281, Indonesia

    Rinto Anugraha NQZ

  6. National Research and Innovation Agency, Jakarta, 10340, Indonesia

    Emanuel Sungging Mumpuni & Rhorom Priyatikanto

Authors
  1. Lala Septem Riza
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zulfikar Ali Yunara Putra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muhammad Iqbal Zain
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fajar Zuliansyah Trihutama
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Judhistira Aria Utama
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Khyrina Airin Fariza Abu Samah
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dhani Herdiwijaya
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Rinto Anugraha NQZ
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Emanuel Sungging Mumpuni
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Rhorom Priyatikanto
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

1. L.S.R conceived and designed the experiments, analyzed and interpreted the data, contributed reagents, materials, analysis tools or data, and wrote the paper. 2. Z. A. Y. P., M. I. Z., and F. Z. T. performed the experiments and wrote the paper. 3. J. A. U. wrote discussion and analysis in the paper. 4. K. A. F. A. S., D. H., R. A. N. and R. P. analyzed and interpreted the data and wrote the paper. 5. E. S. M. conceived and designed the experiments, analyzed and interpreted the data, and contributed reagents, materials, analysis tools or data.

Corresponding author

Correspondence to Lala Septem Riza.

Ethics declarations

Conflicts of interest

The authors have no financial or proprietary interests in any material discussed in this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Riza, L.S., Putra, Z.A.Y., Zain, M.I. et al. Real-time anomaly detection in sky quality meter data using probabilistic exponential weighted moving average. Int J Data Sci Anal (2024). https://doi.org/10.1007/s41060-024-00535-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s41060-024-00535-8

Keywords

Search

Navigation