Abstract
Modern sensor networks monitor a wide range of phenomena. They are applied in environmental monitoring, health care, optimization of industrial processes, social media, smart city solutions, and many other domains. All in all, they provide a continuously pulse of the almost infinite activities that are happening in the physical space—and in cyber space. The handling of the massive amounts of generated measurements poses a series of (Big Data) challenges. Our work addresses one of these challenges: the detection of anomalies in real-time. In this paper, we propose a generic solution to this problem, and introduce a system that is capable of detecting anomalies, generating notifications, and displaying the recent situation to the user. We apply CUSUM a statistical control algorithm and adopt it so that it can be used inside the Storm framework—a robust and scalable real-time processing framework. We present a proof of concept implementation from the area of environmental monitoring.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Storm, Apache Incubator, http://storm.apache.org.
- 2.
Apache S4, Apache Incubator, http://incubator.apache.org/s4.
- 3.
Samza, Apache Incubator, http://samza.incubator.apache.org.
- 4.
Spark Streaming, Apache Incubator, http://spark.apache.org/streaming.
- 5.
Storm Trident, Apache Incubator, http://storm.apache.org/documentation/Trident-API-Overview.html.
- 6.
ActiveMQ framework, Apache Software Foundation, http://activemq.apache.org.
- 7.
OpenWire protocol, Apache Incubator, http://activemq.apache.org/openwire.html.
- 8.
STOMP protocol, Apache Incubator, http://activemq.apache.org/stomp.html.
- 9.
- 10.
Leaflet: An Open-Source JavaScript Library for Mobile-Friendly Interactive Maps, http://leafletjs.com.
- 11.
ESRI, http://www.esri.com.
- 12.
Bootstrap, Twitter http://getbootstrap.com.
- 13.
jQuery, jQuery Foundation http://jquery.com.
- 14.
Highcharts JS, Highcharts AS http://www.highcharts.com.
References
Barratt, B., & Fuller, G. (2014). Intervention assessments in the control of PM10 emissions from an urban waste transfer station. Environmental Science: Processes and Impacts, 16(6), 1328–1337.
Barratt, B., Atkinson, R., Anderson, H., Beevers, S., Kelly, F., Mudway, I., & Wilkinson, P. (2007). Investigation into the use of the cusum technique in identifying changes in mean air pollution levels following introduction of a traffic management scheme. Atmospheric Environment, 41(8), 1784–1791.
Buschmann, F., Meunier, R., Rohnert, H., & Sommerlad, P. (1996). Pattern-oriented software architecture: A system of patterns (Vol. 1). New York: Wiley.
Carslaw, D., Ropkins, K., & Bell, M. (2006). Change-point detection of gaseous and particulate traffic-related pollutants at a roadside location. Environmental Science and Technology, 40(22), 6912–6918.
Charles, J., & Jeh-Nan, P. (2002). Evaluating environmental performance using statistical process control techniques. European Journal of Operational Research, 139(1), 68–83.
Chelani, A. (2011). Change detection using cusum and modified cusum method in air pollutant concentrations at traffic site in Delhi. Stochastic Environmental Research and Risk Assessment, 25(6), 827–834.
Chuen-Sheng, C. (1995). A multi-layer neural network model for detecting changes in the process mean. Computers and Industrial Engineering, 28(1), 51–61.
Cox, S. (2007). Observations and measurements part 1—observation schema. Technical report, Open Geospatial Consortium (OGC).
De Francisci Morales, G. (2013). Samoa: A platform for mining big data streams. In Proceedings of the 22nd International Conference on World Wide Web Companion, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, WWW ’13 Companion (pp. 777–778).
Fielding, R. (2000). Representational state transfer (rest) (Chap. 5). Fielding Dissertation.
Grigg, O., Farewell, V., & Spiegelhalter, D. (2003). Use of risk-adjusted CUSUM and RSPRTcharts for monitoring in medical contexts. Statistical Methods in Medical Research, 12(2), 147–170.
Guh, R., & Hsieh, Y. (1999). A neural network based model for abnormal pattern recognition of control charts. Computers and Industrial Engineering, 36(1), 97–108.
Hapner, M., Burridge, R., Sharma, R., Fialli, J., & Stout, K. (2002). Java message service. Santa Clara, CA: Sun Microsystems Inc.
Hickson, I. (2011). The WebSocket API. W3C Working Draft WD-websockets-20110929, September.
Jeske, D., Montes De Oca, V., Bischoff, W., & Marvasti, M. (2009). Cusum techniques for timeslot sequences with applications to network surveillance. Computational Statistics Data Analysis, 53(12), 4332–4344.
Kortuem, G., Kawsar, F., Fitton, D., & Sundramoorthy, V. (2010). Smart objects as building blocks for the internet of things. IEEE Internet Computing, 14(1), 44–51.
Lucas, J. (1982). Combined Shewhart-CUSUM quality control schemes. Journal of Quality Technology, 14(2).
Manovich, L. (2012). Trending: The promises and the challenges of big social data. In M. K. Gold (Ed.), Debates in the digital humanities (pp. 460–475). Minneapolis: U of Minnesota P.
Mesnil, B., & Petitgas, P. (2009). Detection of changes in time-series of indicators using cusum control charts. Aquatic Living Resources, 22(02), 187–192.
Morris, S., & Paradiso, J. (2002). Shoe-integrated sensor system for wireless gait analysis and real-time feedback. In Engineering in Medicine and Biology, 2002. 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society EMBS/BMES Conference, 2002. Proceedings of the Second Joint, IEEE (Vol. 3, pp. 2468–2469).
Nativi, S., Craglia, M., & Pearlman, J. (2012). The brokering approach for multidisciplinary interoperability: A position paper. International Journal of Spatial Data Infrastructures Research, 7, 1–15.
Osanaiye, P., & Talabi, C. (1989). On some non-manufacturing applications of counted data cumulative sum (CUSUM) control chart schemes. The Statistician, 38(4), 251–257.
Page, E. (1954). Continuous inspection schemes. Biometrika, 41(1/2), 100–115.
Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010). The Hadoop distributed file system. In 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), IEEE (pp. 1–10).
Simoncelli, D., Dusi, M., Gringoli, R., & Niccolini, S. (2013). Stream-monitoring with blockmon: Convergence of network measurements and data analytics platforms. SIGCOMM Computer Communication Review, 43(2), 29–36.
Sitaram, D., Srinivasaraghavan, H., Agarwal, K., Agrawal, A., Joshi, N., & Ray, D. (2013). Pipelining acoustic model training for speech recognition using storm. In 2013 Fifth International Conference on Computational Intelligence, Modelling and Simulation (CIMSim) (pp. 219–224).
Sunderrajan, A., Aydt, H., & Knoll, A. (2014). Real time load prediction and outliers detection using storm. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, ACM, New York, NY, USA, DEBS ’14 (pp. 294–297).
Trilles, S., Belmonte, O., Diaz, L., & Huerta, J. (2014). Mobile access to sensor networks by using GIS standards and restful services. IEEE Sensors Journal, 14(12), 4143–4153.
Yutan, D., Jun, L., Fang, L., & Luying, C. (2014). A real-time anomalies detection system based on streaming technology. In 2014 Sixth International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC) (Vol. 2, pp. 275–279).
Acknowledgments
This work has been supported in part by European Commission and Generalitat Valenciana government (grants ACIF/2012/112 and BEFPI/2014/067).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Trilles, S., Schade, S., Belmonte, Ó., Huerta, J. (2015). Real-Time Anomaly Detection from Environmental Data Streams. In: Bacao, F., Santos, M., Painho, M. (eds) AGILE 2015. Lecture Notes in Geoinformation and Cartography. Springer, Cham. https://doi.org/10.1007/978-3-319-16787-9_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-16787-9_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16786-2
Online ISBN: 978-3-319-16787-9
eBook Packages: Earth and Environmental ScienceEarth and Environmental Science (R0)