research-article

Smart Meter Data Analytics: Systems, Algorithms, and Benchmarking

Authors:

Wojciech Golab,

Shichao JinAuthors Info & Claims

ACM Transactions on Database Systems (TODS), Volume 42, Issue 1

Article No.: 2, Pages 1 - 39

https://doi.org/10.1145/3004295

Published: 21 November 2016 Publication History

Abstract

Smart electricity meters have been replacing conventional meters worldwide, enabling automated collection of fine-grained (e.g., every 15 minutes or hourly) consumption data. A variety of smart meter analytics algorithms and applications have been proposed, mainly in the smart grid literature. However, the focus has been on what can be done with the data rather than how to do it efficiently. In this article, we examine smart meter analytics from a software performance perspective. First, we design a performance benchmark that includes common smart meter analytics tasks. These include offline feature extraction and model building as well as a framework for online anomaly detection that we propose. Second, since obtaining real smart meter data is difficult due to privacy issues, we present an algorithm for generating large realistic datasets from a small seed of real data. Third, we implement the proposed benchmark using five representative platforms: a traditional numeric computing platform (Matlab), a relational DBMS with a built-in machine learning toolkit (PostgreSQL/MADlib), a main-memory column store (“System C”), and two distributed data processing platforms (Hive and Spark/Spark Streaming). We compare the five platforms in terms of application development effort and performance on a multicore machine as well as a cluster of 16 commodity servers.

References

[1]

J. M. Abreu, F. P. Camara, and P. Ferrao. 2012. Using pattern recognition to identify habitual behavior in residential electricity consumption. Energy and Buildings, 49:479--487.

[2]

G. Acs and C. Castelluccia. 2011. I have a DREAM (DiffeRentially privatE smArt Metering). In Conf. on Information Hiding, 118--132.

Digital Library

[3]

A. Albert, T. Gebru, J. Ku, J. Kwac, J. Leskovec, and R. Rajagopal. 2013. Drivers of variability in energy consumption. In ECML-PKDD DARE Workshop on Energy Analytics.

[4]

A. Albert and R. Rajagopal. 2013a. Building dynamic thermal profiles of energy consumption for individuals and neighborhoods. In IEEE Big Data Conf., 723--728.

[5]

A. Albert and R. Rajagopal. 2013b. Smart meter driven segmentation: What your consumption says about you. IEEE Transactions on Power Systems, 4(28), 4019--4030.

[6]

E. Anderson and J. Tucek. 2010. Efficiency matters&excl; SIGOPS Operating Systems Review, 44(1):40--45.

Digital Library

[7]

C. Anil. 2013. Benchmarking of data mining techniques as applied to power system analysis. Master’s Thesis, Uppsala University.

[8]

O. Ardakanian, N. Koochakzadeh, R. P. Singh, L. Golab, and S. Keshav. 2014. Computing electricity consumption profiles from household smart meter data. In EnDM Workshop on Energy Data Management, 140--147.

[9]

M. Arlitt, M. Marwah, G. Bellala, A. Shah, J. Healey, and B. Vandiver. 2015. IoTA bench: An internet of things analytics benchmark. In Proc. of the ACM/SPEC Int. Conf. on Performance Engineering. 133--144.

Digital Library

[10]

B. J. Birt, G. R. Newsham, I. Beausoleil-Morrison, M. M. Armstrong, N. Saldanha, and I. H. Rowlands. 2012. Disaggregating categories of electrical energy end-use from whole-house hourly data. Energy and Buildings 50:93--102.

[11]

N. Bruno and S. Chaudhuri. 2005. Flexible database generators. In Int. Conf. on Very Large Data Bases. 1097--1107.

Digital Library

[12]

E. Buchmann, K. Bohm, T. Burghardt, and S. Kessler. 2013. Re-identification of smart meter data. Pers. Ubiqit. Comput. 17(4):653--662.

Digital Library

[13]

C. Chen and D. Cook. 2011. Energy outlier detection in smart environments. In AAAI Workshop on Artificial Intelligence and Smarter Living: The Conquest of Complexity.

Digital Library

[14]

G. Chicco, R. Napoli, and F. Piglione. 2006. Comparisons among clustering techniques for electricity customer classification. IEEE Trans. on Power Systems, 21(2):933--940.

[15]

F. Eichinger, P. Efros, S. Karnouskos, and K. Bohm. 2015. A time-series compression technique and its application to the smart grid. VLDB Journal 24(2):193--218.

Digital Library

[16]

Electric Power Research Institute (EPRI). 2013. Big Data Survey Summary Report

[17]

M. Espinoza, C. Joye, R. Belmans, and B. DeMoor. 2005. Short-term load forecasting, profile identification, and customer segmentation: A methodology based on periodic time series. IEEE Trans. on Power Systems, 20(3):1622--1630.

[18]

V. Figueiredo, F. Rodrigues, Z. Vale, and J. Gouveia. 2005. An electric energy consumer characterization framework based on data mining techniques. IEEE Trans. on Power Systems, 20(2):596--602.

[19]

M. Ghofrani, M. Hassanzadeh, M. Etezadi-Amoli, and M. Fadali. 2011. Smart meter based short-term load forecasting for residential customers. In North American Power Symposium (NAPS’11).

[20]

L. Gu, M. Zhou, Z. Zhang, M.-C. Shan, A. Zhou, and M. Winslett. 2015. Chronos: An elastic parallel framework for stream benchmark generation and simulation. In IEEE Int. Conf. on Data Engineering. 101--112.

[21]

J. M. Hellerstein, C. Re, F. Schoppmann, D. Z. Wang, E. Fratkin, A. Gorajek, and A. Kumar. 2012. The MADlib analytics library: Or MAD skills, the SQL. Proc. of the VLDB Endowment, 5(12):1700--1711.

Digital Library

[22]

R.-S. Jeng, C.-Y. Kuo, Y.-H. Ho, M.-F. Lee, L.-W. Tseng, C.-L. Fu, P.-F. Liang, and L.-J. Chen. 2013. Missing data handling for meter data management system. In ACM Int. Conf. on Future Energy Systems. 275--276.

Digital Library

[23]

E. Keogh and S. Kasetty. 2013. On the need for time series data mining benchmarks: A survey and empirical demonstration. Data Mining and Knowledge Discovery (DMKD), 7(4):349--371.

Digital Library

[24]

S. Kessler, E. Buchmann, and K. Bohm. 2015. Deploying and evaluating pufferfish privacy for smart meter data. In Proc. Int. Conf. on Ubiquitous Intelligence and Computing (UIC’15).

[25]

X. Liu, L. Golab, W. Golab, and I. Ilyas. 2015a. Benchmarking smart meter data analytics. In Int. Conf. on Extending Database Technology. 285--396.

[26]

X. Liu, L. Golab, and I. Ilyas. 2015b. SMAS: A smart meter data analytics system. In IEEE Int. Conf. on Data Engineering. 1476--1479.

[27]

Y. Liu, S. Hu, T. Rabl, W. Liu, H.-A. Jacobsen, K. Wu, J. Chen, and J. Li. 2014. DGFIndex for smart grid: Enhancing hive with a cost-effective multidimensional range index. Proc. of the VLDB Endowment 7(13): 1496--1507.

Digital Library

[28]

D. Mashima and A. Cardenas. 2012. Evaluating electricity theft detectors in smart grid networks. In Int. Conf. on Research in Attacks, Intrusions and Defenses (RAID’12), 210--229.

Digital Library

[29]

F. Mattern, T. Staake, and M. Weiss. 2010. ICT for green - how computers can help us to conserve energy. In ACM Int. Conf. on Future Energy Systems. 1--10.

Digital Library

[30]

A. J. Nezhad, T. K. Wijaya, M. Vasirani, and K. Aberer. 2014. SmartD: Smart meter data analytics dashboard. In ACM Int. Conf. on Future Energy Systems. 213--214.

Digital Library

[31]

T. Rasanen, D. Voukantsis, H. Niska, K. Karatzas, and M. Kolehmainen. 2010. Data-based method for creating electricity use load profiles using large amount of customer-specific hourly measured electricity use data. Applied Energy, 87(11):3538--3545.

[32]

B. A. Smith, J. Wong, and R. Rajagopal. 2012. A simple way to use interval data to segment residential customers for energy efficiency and demand response program targeting. In ACEEE Summer Study on Energy Efficiency in Buildings.

[33]

A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. 2009. Hive - A warehousing solution over a map-reduce framework. Proc. of the VLDB Endowment 2(2): 1626--1629.

Digital Library

[34]

G. Tsekouras, N. Hatziargyriou, and E. Dialynas. 2007. Two-stage pattern recognition of load curves for classification of electricity customers. IEEE Trans. on Power Systems, 22(3):1120--1128.

[35]

T. K. Wijaya, J. Eberle, and K. Aberer. 2013. Symbolic representation of smart meter data. In EDBT Workshop on Energy Data Management (EnDM’13), 242--248.

Digital Library

[36]

M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. 2010. Spark: Cluster computing with working sets. In USENIX Conf., 10.

Digital Library

[37]

M. Zaharia, T. Das, H. Li, S. Shenker, and I. Stoica. 2012. Discretized streams: An efficient and fault-tolerant model for stream processing on large clusters. In Proc. USENIX Conf. on Hot Topics in Cloud Computing. 10.

Digital Library

Cited By

Buhari AHajamydeen ANyamasvisva TSalome S(2024)Design and Modeling of an AI-Powered Industrial Maximum Demand Controller With Web Service InterfaceUtilizing Renewable Energy, Technology, and Education for Industry 5.010.4018/979-8-3693-2814-9.ch002(23-80)Online publication date: 21-Jun-2024
https://doi.org/10.4018/979-8-3693-2814-9.ch002
Mbey CBikai JYem Souhe FFoba Kakeu VBoum A(2024)Electricity Theft Detection in a Smart Grid Using Hybrid Deep Learning-Based Data Analysis TechniqueJournal of Electrical and Computer Engineering10.1155/2024/62255102024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/6225510
Teng R(2024)Cooperative Discovery of Failed IoT Node by Double-Zone Presentation2024 9th International Conference on Computer and Communication Systems (ICCCS)10.1109/ICCCS61882.2024.10603130(997-1001)Online publication date: 19-Apr-2024
https://doi.org/10.1109/ICCCS61882.2024.10603130
Show More Cited By

Index Terms

Smart Meter Data Analytics: Systems, Algorithms, and Benchmarking
1. Information systems
  1. Data management systems
    1. Database administration
      1. Database performance evaluation
  2. Information systems applications
    1. Data mining

Recommendations

A Spark-Based Big Data Platform for Massive Remote Sensing Data Processing
ICDS 2015: Proceedings of the Second International Conference on Data Science - Volume 9208

With the fast development of remote sensing techniques, the volume of acquired data grows exponentially. This brings a big challenge to process massive remote sensing data. In the paper, an in-memory computing framework is proposed to address this ...
A Performance Study of Big Spatial Data Systems
BigSpatial '18: Proceedings of the 7th ACM SIGSPATIAL International Workshop on Analytics for Big Geospatial Data

With the accelerated growth in spatial data volume, being generated from a wide variety of sources, the need for efficient storage, retrieval, processing and analyzing of spatial data is ever more important. Hence, spatial data processing system has ...
Evaluating SQL-on-Hadoop for Big Data Warehousing on Not-So-Good Hardware
IDEAS '17: Proceedings of the 21st International Database Engineering & Applications Symposium

Big Data is currently conceptualized as data whose volume, variety or velocity impose significant difficulties in traditional techniques and technologies. Big Data Warehousing is emerging as a new concept for Big Data analytics. In this context, SQL-on-...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Database Systems

ACM Transactions on Database Systems Volume 42, Issue 1

Invited Paper from ICDT 2014, Invited Paper from EDBT 2015, Regular Papers and Technical Correspondence

March 2017

263 pages

ISSN:0362-5915

EISSN:1557-4644

DOI:10.1145/3015779

Editor:
Christian S. Jensen
Aalborg University, Denmark

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 November 2016

Accepted: 01 October 2016

Revised: 01 July 2016

Received: 01 August 2015

Published in TODS Volume 42, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

46
Total Citations
View Citations
1,446
Total Downloads

Downloads (Last 12 months)90
Downloads (Last 6 weeks)14

Reflects downloads up to 02 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Buhari AHajamydeen ANyamasvisva TSalome S(2024)Design and Modeling of an AI-Powered Industrial Maximum Demand Controller With Web Service InterfaceUtilizing Renewable Energy, Technology, and Education for Industry 5.010.4018/979-8-3693-2814-9.ch002(23-80)Online publication date: 21-Jun-2024
https://doi.org/10.4018/979-8-3693-2814-9.ch002
Mbey CBikai JYem Souhe FFoba Kakeu VBoum A(2024)Electricity Theft Detection in a Smart Grid Using Hybrid Deep Learning-Based Data Analysis TechniqueJournal of Electrical and Computer Engineering10.1155/2024/62255102024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/6225510
Teng R(2024)Cooperative Discovery of Failed IoT Node by Double-Zone Presentation2024 9th International Conference on Computer and Communication Systems (ICCCS)10.1109/ICCCS61882.2024.10603130(997-1001)Online publication date: 19-Apr-2024
https://doi.org/10.1109/ICCCS61882.2024.10603130
Hasan MHasan MBudati AIslam SSafie NAhmed FBakar KBabiker NGhazal T(2024)A hybrid key agreement scheme utilized elliptic curve Diffie-Hellman for IoT based advanced metering environmentEarth Science Informatics10.1007/s12145-024-01292-917:3(2447-2460)Online publication date: 1-Apr-2024
https://doi.org/10.1007/s12145-024-01292-9
Wang RHo WChiang KHung YTai JTan JChuang MKe CChien YJeng AChou C(2023)Analyzing Long-Term and High Instantaneous Power Consumption of Buildings from Smart Meter Big Data with Deep Learning and Knowledge Graph TechniquesEnergies10.3390/en1619689316:19(6893)Online publication date: 29-Sep-2023
https://doi.org/10.3390/en16196893
Wang JZhou HZhang LHong QLin SYang H(2023)Periodic Analysis of Meter Installation data in Fujian Province based on Spectral Analysis2023 IEEE International Symposium on Product Compliance Engineering - Asia (ISPCE-ASIA)10.1109/ISPCE-ASIA60405.2023.10366046(1-5)Online publication date: 4-Nov-2023
https://doi.org/10.1109/ISPCE-ASIA60405.2023.10366046
Jukaria MUpadhyay AKumar A(2023)Vector Auto-Regression-Based Predictive model for Smart Meter2023 International Conference on Device Intelligence, Computing and Communication Technologies, (DICCT)10.1109/DICCT56244.2023.10110196(396-400)Online publication date: 17-Mar-2023
https://doi.org/10.1109/DICCT56244.2023.10110196
Stamatescu GPlamanescu RAlbu M(2023)Leveraging Anomaly Detection and AutoML for Modelling Residential Measurement Power Traces2023 IEEE 13th International Workshop on Applied Measurements for Power Systems (AMPS)10.1109/AMPS59207.2023.10297201(1-5)Online publication date: 27-Sep-2023
https://doi.org/10.1109/AMPS59207.2023.10297201
Banik SSaha SBanik THossain S(2023)Anomaly Detection Techniques in Smart Grid Systems: A Review2023 IEEE World AI IoT Congress (AIIoT)10.1109/AIIoT58121.2023.10174485(0331-0337)Online publication date: 7-Jun-2023
https://doi.org/10.1109/AIIoT58121.2023.10174485
Wu JNiu ZLi XHuang LNielsen PLiu X(2023)Understanding multi-scale spatiotemporal energy consumption data: A visual analysis approachEnergy10.1016/j.energy.2022.125939263(125939)Online publication date: Jan-2023
https://doi.org/10.1016/j.energy.2022.125939
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents