Action Set Based Policy Optimization for Safe Power Grid Management

Abstract

Maintaining the stability of the modern power grid is becoming increasingly difficult due to fluctuating power consumption, unstable power supply coming from renewable energies, and unpredictable accidents such as man-made and natural disasters. As the operation on the power grid must consider its impact on future stability, reinforcement learning (RL) has been employed to provide sequential decision-making in power grid management. However, existing methods have not considered the environmental constraints. As a result, the learned policy has risk of selecting actions that violate the constraints in emergencies, which will escalate the issue of overloaded power lines and lead to large-scale blackouts. In this work, we propose a novel method for this problem, which builds on top of the search-based planning algorithm. At the planning stage, the search space is limited to the action set produced by the policy. The selected action strictly follows the constraints by testing its outcome with the simulation function provided by the system. At the learning stage, to address the problem that gradients cannot be propagated to the policy, we introduce Evolutionary Strategies (ES) with black-box policy optimization to improve the policy directly, maximizing the returns of the long run. In NeurIPS 2020 Learning to Run Power Network (L2RPN) competition, our solution safely managed the power grid and ranked first in both tracks.

B. Zhou, H. Zeng—Equal Contribution.

A Grid2Op Environment

Grid2Op [18] is an open-source environment developed for testing the performance of controllers in power grid management. It simulates the physical power grid and follows the real-world power system operational constraints and distributions.

The environment provides interactive interfaces based on the gym library [3]. At each episode, it simulates a period of time (e.g., a week or month) at the time interval of 5 min. At each time step, the controller receives the state of the power grid and takes actions to operate the grid if necessary. The simulation terminates at the end of the period or terminates prematurely if the controller fails to operate the grid properly, which can occur under two conditions: (1) Some actions split the grid into several isolated sub-grids. (2) The electricity power transmitted from the stations cannot meet the consumption requirement of some loads. Too many disconnected lines will significantly increase the risk of causing these two conditions. A power line gets disconnected automatically if the current flow exceeds the maximum limit for 3 time steps (i.e., 15 min). In this case, the power line cannot be reconnected until the end of the recovery time of 12 time steps.

Grid2Op has a large state space and action space. In addition to the exponentially increasing possible grid topology we mentioned, the grid state contains other topology features such as the current, voltage magnitude of each power line, generation power of each power station, the required power of each load. Though only one substation can be reconfigured at each time step (to simulate that a human or an expert can perform a limited number of actions in a time period), the number of available actions for topology reconfiguration is ${\sum _{i=1}^{i=M}2^{|S_i|}}$. In the NeurIPS2020 L2RPN competition, there are 118 substations with 186 power lines, which introduces over 70,000 discrete actions related to unique topology.

The reward setting in Grid2Op mainly relies on the reliability of the power grid. At each time step, the environment gives a bonus for safe management, and the controller will no longer gain positive rewards if it fails to manage the power network properly, which can lead to early termination of the episode. There are also costs (penalty) of operations. To encourage the controller to explore the operation flexibility on topology reconfiguration, the cost of topology change is much smaller than re-dispatching the power generation of the power plant.

In the physical world, the operators often use a simulation system to compute the possible outcomes of actions to control risk [1, 18, 28], and Grid2Op also provides a similar function named simulate, which can mimic the one-step operational process. It allows the user to check if the action violates the power network constraint (e.g., if the target power generation exceeds the maximum output of the power plant). Note that this function can only be called once at each time step (i.e., one-step simulation), and its prediction on future states may bias from the actual state.