research-article

A review of reinforcement learning for autonomous building energy management

Authors:

Santiago GrijalvaAuthors Info & Claims

Volume 78, Issue C

Pages 300 - 312

https://doi.org/10.1016/j.compeleceng.2019.07.019

Published: 01 September 2019 Publication History

Abstract

The area of building energy management has received a significant amount of interest in recent years. This area is concerned with combining advancements in sensor technologies, communications and advanced control algorithms to optimize energy utilization. Reinforcement learning is one of the most prominent machine learning algorithms used for control problems and has had many successful applications in the area of building energy management. This research gives a comprehensive review of the literature relating to the application of reinforcement learning to developing autonomous building energy management systems. Energy savings of greater than 20% are reported in the literature for more complex building energy management problems when implementing reinforcement learning. The main direction for future research and challenges in reinforcement learning are also outlined.

References

[1]

E. Barrett, S. Linder, Autonomous hvac control, a reinforcement learning approach, Joint European conference on machine learning and knowledge discovery in databases, Springer, 2015, pp. 3–19.

[2]

G.A. Rummery, M. Niranjan, On-line Q-learning using connectionist systems, 37, University of Cambridge, Department of Engineering Cambridge, England, 1994.

[3]

D. Du, M. Fei, A two-layer networked learning control system using actor–critic neural network, ApplMathComput 205 (1) (2008) 26–36.

[4]

T. Wei, Y. Wang, Q. Zhu, Deep reinforcement learning for building hvac control, Proceedings of the 54th annual design automation conference 2017, ACM, 2017, p. 22.

[5]

Z. Zhang, A. Chong, Y. Pan, C. Zhang, S. Lu, K.P. Lam, A deep reinforcement learning approach to using whole building energy model for hvac optimal control, 2018 building performance analysis conference and SimBuild, 2018.

[6]

S. Liu, G.P. Henze, Experimental analysis of simulated reinforcement learning control for active and passive building thermal storage inventory: part 2: results and analysis, Energy Build 38 (2) (2006) 148–161.

[7]

D. Urieli, P. Stone, A learning agent for heat-pump thermostat control, Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, International Foundation for Autonomous Agents and Multiagent Systems, 2013, pp. 1093–1100.

[8]

F. Ruelens, S. Iacovella, B.J. Claessens, R. Belmans, Learning agent for a heat-pump thermostat with a set-back strategy using model-free reinforcement learning, Energies 8 (8) (2015) 8300–8318.

[9]

L. Yang, Z. Nagy, P. Goffin, A. Schlueter, Reinforcement learning for optimal control of low exergy buildings, Appl Energy 156 (2015) 577–586.

[10]

Y. Wang, K. Velswamy, B. Huang, A long-short term memory recurrent neural network based reinforcement learning controller for office heating ventilation and air conditioning systems, Processes 5 (3) (2017) 46.

[11]

C. Marantos, C.P. Lamprakos, V. Tsoutsouras, K. Siozios, D. Soudris, Towards plug & play smart thermostats inspired by reinforcement learning, Proceedings of the workshop on INTelligent embedded systems architectures and applications, ACM, 2018, pp. 39–44.

[12]

Y. Chen, L.K. Norford, H.W. Samuelson, A. Malkawi, Optimal control of hvac and window systems for natural ventilation through reinforcement learning, Energy Build 169 (2018) 195–205.

[13]

C. Patyn, F. Ruelens, G. Deconinck, Comparing neural architectures for demand response through model-free reinforcement learning for heat pump control, 2018 IEEE international energy conference (ENERGYCON), IEEE, 2018, pp. 1–6.

[14]

K. Al-Jabery, D.C. Wunsch, J. Xiong, Y. Shi, A novel grid load management technique using electric water heaters and q-learning, Smart grid communications (SmartGridComm), 2014 IEEE international conference on, IEEE, 2014, pp. 776–781.

[15]

K. Al-Jabery, Z. Xu, W. Yu, D.C. Wunsch, J. Xiong, Y. Shi, Demand-side management of domestic electric water heaters using approximate dynamic programming, IEEE Trans Comput-Aided DesIntegr Circuits Syst 36 (5) (2017) 775–788.

[16]

F. Ruelens, B.J. Claessens, S. Quaiyum, B. De Schutter, R. Babuška, R. Belmans, Reinforcement learning applied to an electric water heater: from theory to practice, IEEE Trans Smart Grid 9 (4) (2018) 3792–3800.

[17]

O. De Somer, A. Soares, K. Vanthournout, F. Spiessens, T. Kuijpers, K. Vossen, Using reinforcement learning for demand response of domestic hot water buffers: A real-life demonstration, Innovative smart grid technologies conference Europe (ISGT-Europe), 2017 IEEE PES, IEEE, 2017, pp. 1–7.

[18]

H. Kazmi, F. Mehmood, S. Lodeweyckx, J. Driesen, Gigawatt-hour scale savings on a budget of zero: deep reinforcement learning based optimal control of hot water systems, Energy 144 (2018) 159–168.

[19]

M. Reymond, C. Patyn, R. Rădulescu, G. Deconinck, A. Nowé, Reinforcement learning for demand response of domestic household appliances, Proceedings of the adaptive and learning agents workshop (at AAMAS 2018), 2018.

[20]

Q. Wei, D. Liu, G. Shi, A novel dual iterative q-learning method for optimal battery management in smart residential environments, IEEE Trans Ind Electron 62 (4) (2015) 2509–2518.

[21]

C. Guan, Y. Wang, X. Lin, S. Nazarian, M. Pedram, Reinforcement learning-based control of residential energy storage systems for electric bill minimization, Consumer communications and networking conference (CCNC), 2015 12th annual IEEE, IEEE, 2015, pp. 637–642.

[22]

Z. Wan, H. Li, H. He, Residential energy management with deep reinforcement learning, 2018 international joint conference on neural networks (IJCNN), IEEE, 2018, pp. 1–7.

[23]

R. T, E.A. Jasmin, T.P.I. Ahamed, Residential load scheduling with renewable generation in the smart grid: a reinforcement learning approach, IEEE Syst J (2018) 1–12,.

[24]

Z. Wen, D. ONeill, H. Maei, Optimal demand response using device-based reinforcement learning, IEEE Trans Smart Grid 6 (5) (2015) 2312–2324.

[25]

N. Bazenkov, M. Goubko, Advanced planning of home appliances with consumers preference learning, Russian conference on artificial intelligence, Springer, 2018, pp. 249–259.

[26]

E. Mocanu, D.C. Mocanu, P.H. Nguyen, A. Liotta, M.E. Webber, M. Gibescu, et al., On-line building energy optimization using deep reinforcement learning, IEEE Trans Smart Grid (2018).

[27]

Y. Wang, X. Lin, M. Pedram, A near-optimal model-based control algorithm for households equipped with residential photovoltaic power generation and energy storage systems, IEEE Trans Sustainable Energy 7 (1) (2016) 77–86.

[28]

M. Rayati, A. Sheikhi, A.M. Ranjbar, Optimising operational cost of a smart energy hub, the reinforcement learning approach, Int J Parallel Emergent Distrib Syst 30 (4) (2015) 325–341.

[29]

A. Sheikhi, M. Rayati, A. Ranjbar, Dynamic load management for a residential customer; reinforcement learning approach, Sustainable Cities Soc 24 (2016) 42–51.

[30]

B. Jiang, Y. Fei, Smart home in smart microgrid: a cost-effective energy ecosystem with intelligent hierarchical agents, IEEE Trans Smart Grid 6 (1) (2015) 3–13.

[31]

B.-G. Kim, Y. Zhang, M. Van Der Schaar, J.-W. Lee, Dynamic pricing and energy consumption scheduling with reinforcement learning, IEEE Trans Smart Grid 7 (5) (2016) 2187–2198.

[32]

A. Anvari-Moghaddam, A. Rahimi-Kian, M.S. Mirian, J.M. Guerrero, A multi-agent based energy management solution for integrated buildings and microgrid system, Appl Energy 203 (2017) 41–56.

[33]

Prasad A., Dusparic I. Multi-agent deep reinforcement learning for zero energy communities. arXiv:181003679v1 2018.

[34]

P. Kofinas, G. Vouros, A.I. Dounis, Energy management in solar microgrid via reinforcement learning using fuzzy reward, Adv Build Energy Res 12 (1) (2018) 97–115.

[35]

B.J. Claessens, P. Vrancx, F. Ruelens, Convolutional neural networks for automatic state-time feature extraction in reinforcement learning applied to residential load control, IEEE Trans Smart Grid 9 (4) (2018) 3259–3269.

Cited By

Fatima KShareef HCosta FBajwa AWong L(2024)Machine learning for power outage prediction during hurricanesEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108056133:PBOnline publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.108056
Shah MWahid ABarrett EMason K(2024)Multi-agent systems in Peer-to-Peer energy tradingEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.107847132:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.107847
Felicetti RFerracuti FIarlori SMonteriù A(2024)Peak shaving and self-consumption maximization in home energy management systemsComputers and Electrical Engineering10.1016/j.compeleceng.2024.109283117:COnline publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.compeleceng.2024.109283
Show More Cited By

Index Terms

A review of reinforcement learning for autonomous building energy management

Index terms have been assigned to the content through auto-classification.

Recommendations

Reinforcement learning for crop management support: Review, prospects and challenges
Abstract
Reinforcement learning (RL), including multi-armed bandits, is a branch of machine learning that deals with the problem of sequential decision-making in uncertain and unknown environments through learning by practice. While best known ...
Highlights
- Reinforcement learning is a promising AI framework to support crop management.
- ...
Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems

Recent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...
Using Transfer Learning to Speed-Up Reinforcement Learning: A Cased-Based Approach
LARS '10: Proceedings of the 2010 Latin American Robotics Symposium and Intelligent Robotics Meeting

Reinforcement Learning (RL) is a well-known technique for the solution of problems where agents need to act with success in an unknown environment, learning through trial and error. However, this technique is not efficient enough to be used in ...

Comments

Information & Contributors

Information

Published In

cover image Computers and Electrical Engineering

Computers and Electrical Engineering Volume 78, Issue C

Sep 2019

537 pages

ISSN:0045-7906

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 September 2019

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fatima KShareef HCosta FBajwa AWong L(2024)Machine learning for power outage prediction during hurricanesEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108056133:PBOnline publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.108056
Shah MWahid ABarrett EMason K(2024)Multi-agent systems in Peer-to-Peer energy tradingEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.107847132:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.107847
Felicetti RFerracuti FIarlori SMonteriù A(2024)Peak shaving and self-consumption maximization in home energy management systemsComputers and Electrical Engineering10.1016/j.compeleceng.2024.109283117:COnline publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.compeleceng.2024.109283
Goldfeder JSipple J(2023)A Lightweight Calibrated Simulation Enabling Efficient Offline Learning for Optimal Control of Real BuildingsProceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation10.1145/3600100.3625682(352-356)Online publication date: 15-Nov-2023
https://dl.acm.org/doi/10.1145/3600100.3625682
Zhang CShi YChen Y(2023)BEAR: Physics-Principled Building Environment for Control and Reinforcement LearningProceedings of the 14th ACM International Conference on Future Energy Systems10.1145/3575813.3576877(66-71)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3575813.3576877
Bonetti MBisi LRestelli M(2023)Risk-averse optimization of reward-based coherent risk measuresArtificial Intelligence10.1016/j.artint.2022.103845316:COnline publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.artint.2022.103845
Mischos SDalagdi EVrakas D(2023)Intelligent energy management systems: a reviewArtificial Intelligence Review10.1007/s10462-023-10441-356:10(11635-11674)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1007/s10462-023-10441-3
Robles-Enciso ARobles-Enciso RSkarmeta A(2023)Multi-agent Reinforcement Learning-Based Energy Orchestrator for Cyber-Physical SystemsAlgorithmic Aspects of Cloud Computing10.1007/978-3-031-49361-4_6(100-114)Online publication date: 5-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-49361-4_6
Li QPeng ZWu HFeng LZhou BKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Human-AI shared control via policy dissectionProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3600914(8853-8867)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3600914
Rafayal SCevik MShirani POnut INg TKent KBaşar AOnut I(2022)Time series forecasting-based peak shaving for building energy managementProceedings of the 32nd Annual International Conference on Computer Science and Software Engineering10.5555/3566055.3566062(52-61)Online publication date: 15-Nov-2022
https://dl.acm.org/doi/10.5555/3566055.3566062
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents