A Deep Reinforcement Learning Optimization Method Considering Network Node Failures

Abstract

Nowadays, the microgrid system is characterized by a diversification of power factors and a complex network structure. Existing studies on microgrid fault diagnosis and troubleshooting mostly focus on the fault detection and operation optimization of a single power device. However, for increasingly complex microgrid systems, it becomes increasingly challenging to effectively contain faults within a specific spatiotemporal range. This can lead to the spread of power faults, posing great harm to the safety of the microgrid. The topology optimization of the microgrid based on deep reinforcement learning proposed in this paper starts from the overall power grid and aims to minimize the overall failure rate of the microgrid by optimizing the topology of the power grid. This approach can limit internal faults within a small range, greatly improving the safety and reliability of microgrid operation. The method proposed in this paper can optimize the network topology for the single node fault and multi-node fault, reducing the influence range of the node fault by 21% and 58%, respectively.

1. Introduction

The modernization and intelligent transformation of my country’s power system is advancing rapidly, especially at the city and community levels. The rise of microgrids [1,2,3,4,5] has become an important part of this process. Microgrids refer to small, self-managed, and controlled power systems that can be connected to the main power grid or operate independently. With the large-scale access to renewable energy such as solar and wind power, and the popularization of new power loads such as electric vehicle charging stations and energy storage systems, the structure of microgrids has become extremely complex, which not only increases the flexibility and sustainability of the power system, but also brings new challenges in grid fault management.

There are various types of faults [6,7,8,9] in microgrids, including but not limited to short circuits, overloads, harmonic interference, voltage fluctuations, etc. These faults may be caused by factors such as aging equipment, bad weather, human error, etc. Since distributed energy and loads are integrated in microgrids, the impact of faults is no longer limited to local areas in the traditional sense but may quickly spread to the entire microgrid and even the main grid connected to it, resulting in power outages and reduced service quality in a wider range.

Traditional grid fault detection and troubleshooting relies on manual inspections and automated systems based on fixed rules. This approach has many drawbacks when facing complex and changeable microgrid faults. First, manual inspections are time-consuming, and it is difficult to respond to sudden faults in a timely manner; second, automated systems based on fixed rules often lack intelligent judgment capabilities and cannot effectively handle unexpected faults; finally, when a fault occurs, there is no effective isolation mechanism, which expands the scope of the fault impact and prolongs the recovery time.

The topology of a power grid refers to the way in which the various components in a power network (such as generators, transformers, transmission lines, circuit breakers, and load points) are connected. Grid topologies can be divided into different levels and types, from macro, national, or regional power grids, to meso-transmission networks and micro-distribution networks, and each layer has its own specific structural characteristics. The distribution network is responsible for distributing electricity from substations to end users. Common distribution network topologies include radial, ring, and tree types. The radial structure is simple and low in cost, but a single point failure may affect the power supply over a larger area; the ring and tree structures improve the reliability of the power supply, and even if a certain section of the line fails, the power supply can be restored through a bypass. Figure 1 shows the simple topology of a new power system microgrid.

In view of the above challenges, this paper proposes a deep reinforcement learning optimization method considering network node failures (DRL-NNF), which can quickly respond, accurately isolate faults, and suppress fault propagation as a grid security protection method.

The field of power grid fault safety detection involves a variety of cutting-edge methods and theories. Refs. [10,11,12,13,14,15] studied the fault types and locations within the power grid and proposed a fault detection and diagnosis technology based on artificial intelligence algorithms, laying the foundation for the optimization and adjustment of subsequent faults. Ref. [16] considers the influence of large-scale distributed photovoltaic power generation on the regional grid line loss, establishes a grid topology model with the minimum line loss as the goal, and jointly optimizes the grid topology by combining the k-means algorithm and KPSO algorithm to improve resource utilization. Ref. [17] proposes a hybrid topological crossover algorithm (HTCSO) to solve the problems of the slow convergence and local optimality of traditional crossover algorithms. It uses a hybrid topology combining a ring regular topology and random topology to improve the convergence accuracy and convergence speed of the CSO algorithm. Ref. [18] puts forward a topology planning framework, which involves a power communication network based on game theory from the perspective of network attacks, and uses Bayesian decision theory and the particle swarm optimization algorithm to solve the problem of grid anti-communication attacks step by step. However, for large-scale power grid scenarios, these methods require a large amount of data and computing, which will bring challenges to hardware equipment and data real-time performance.

2. System Architecture

This paper focuses on the optimization of the topology of the entire power grid, abandoning the traditional mode of local adjustment for a single device and focusing on a solution from a global perspective. On the constructed microgrid experimental platform, detailed electrical characteristic parameterization is performed for each branch, covering key indicators such as resistance, reactance, and rated current to ensure the authenticity and reliability of the model. Subsequently, sudden faults are simulated at selected nodes to trigger a system response and verify the adaptability and robustness of this method in the face of power grid disturbances.

The proposed solution relies on deep reinforcement learning (DQN), a cutting-edge technology that combines the advantages of deep learning and learning reinforcement learning algorithms [19,20,21] and is particularly suitable for handling decision-making problems in high-dimensional and complex state spaces. The DQN is well-suited for problems with discrete action spaces. For power grid dispatching problems, especially those involving the switching operation of generator sets and line switching, these are usually discrete. Therefore, the DQN shows good adaptability in dealing with such problems. Compared with algorithms that deal with continuous action spaces (such as Actor–Critic methods, Policy Gradients, etc.), the implementation of the DQN is relatively simple and intuitive. It is based on the Q-learning framework and trains a deep neural network to approximate the Q-value function to select the best action. This simplicity facilitates rapid prototyping and testing.

In this study, the state changes of line switches in each branch of the power grid are regarded as key decision variables, and the load fluctuations caused by the switching operation are regarded as the dynamic feedback of the system state. This process is naturally mapped to the Markov decision process (MDP) [22,23,24] framework, in which state, action, reward, and transition probability constitute the four major elements of the MDP.

The network topology optimization method based on deep reinforcement learning proposed in this paper takes the open and closed state (SS) of each branch switch in the power grid and the operating state (SN) of each node as the state space:

$$\begin{array}{c}SS=\left\{S{S}_1,S{S}_2,S{S}_3,\cdots S{S}_{N_{all-switch}}\right\}\\ S{S}_i=\{\begin{array}{c}1 \mathrm{when} \mathrm{switch} i \mathrm{is} \mathrm{in} \mathrm{the} \mathrm{closed} \mathrm{state}\\ 0 \mathrm{when} \mathrm{switch} i \mathrm{is} \mathrm{in} \mathrm{the} \mathrm{open} \mathrm{state}\end{array}\end{array}$$

$$\begin{array}{c}SN=\left\{S{N}_1,S{N}_2,S{N}_3,\cdots S{N}_{N_{all-node}}\right\}\\ S{N}_i=\{\begin{array}{c}1 \mathrm{when} \mathrm{node} i \mathrm{is} \mathrm{in} \mathrm{the} \mathrm{normal} \mathrm{state}\\ 0 \mathrm{when} \mathrm{node} i \mathrm{is} \mathrm{in} \mathrm{the} \mathrm{faulty} \mathrm{state}\end{array}\end{array}$$

So, the state space is represented as $S=\{\begin{array}{l}SS=\left\{S{S}_1,S{S}_2,S{S}_3,\cdots S{S}_{N_{all-switch}}\right\}\\ SN=\left\{S{N}_1,S{N}_2,S{N}_3,\cdots S{N}_{N_{all-node}}\right\}\end{array}$ where $N_{all-switch}$ is the total number of branch switches and $N_{all-node}$ is the total number of microgrid internal nodes.

The action space refers to the range of operations that can be performed by the agent. In the optimization problem of the microgrid topology proposed in this paper, the agent changes the topology of the grid by controlling the on–off switch of each branch. Therefore, the action space of the agent in this paper is as follows:

$$A=\left\{a_0,a_1,\cdots ,a_{N_{all-node}}\right\},$$

$$a_i=\{\begin{array}{cc}1& Turn switch i on\\ 0& Turn switch i off\end{array}$$

In reinforcement learning, “rewards” often represent the quality of the “actions” taken by the agent. Therefore, “rewards” are often related to the indicators that the system optimizes. Taking the failure rate of microgrid internal nodes $\rho$ as the optimization target, $N_{all-node}$ is the total number of microgrid internal nodes, and $N_{failure-node }$ is the total number of microgrid internal fault nodes:

$$\rho ={\displaystyle \frac{N_{all-node}-N_{failure-node }}{N_{all-node}}},$$

(1)

The framework of the network topology optimization method based on deep reinforcement learning proposed in this paper is shown in Figure 2. After the microgrid topology structure is modeled, the state (SS, SN) is input into the DQN to obtain the optimal action in this state (the action is to reverse the switch state of some branches inside the power grid), and then the load distribution and internal node operation state in the new state of the microgrid are calculated to obtain the reward of this step; finally, the parameters of the deep Q-network are updated.

The DQN algorithm is used to iteratively train the MDP in order to learn a policy function that can predict the maximum long-term reward. In each iteration, the DQN model intelligently selects the most favorable switching operation sequence based on the current grid status and past experience to minimize the impact of the fault, restore the power supply, and optimize the grid performance. This process is not only a dynamic reconfiguration of the grid topology, but also a comprehensive improvement of system stability and efficiency.

3. Model

This section aims to model the actual environment of the power grid and simulate the operation of a community-level microgrid based on the IEEE-33 node system. Subsequently, a reinforcement learning environment will be constructed for the grid node system, and a deep reinforcement learning model will be employed to optimize the grid topology in case of a faulty node within the grid.

3.1. IEEE-33 Node System Construction

The IEEE-33 [25] node system is a model widely used in power system research. It simulates a distribution network with 33 nodes and 32 transmission lines. This system is often used as a benchmark to evaluate and compare the performance of various power system analysis methods and algorithms. In the IEEE-33 node system, each node represents a power demand point or supply point, and each transmission line represents the connection between nodes. The system parameters include the voltage, current, power factor, load, etc., of the node. In addition, electrical characteristic parameters such as the impedance and admittance of the line are also included. The main purpose of the IEEE-33 node system is to conduct various analyses and studies of the power system, such as power flow calculation, fault analysis, optimization control, etc. Through the study of this system, engineers can better understand and master the operating laws of the power system and provide a theoretical basis and technical support for the actual design and operation of the power system.

This paper uses an improved IEEE-33 node system to build the environment. On the basis of traditional power factors, photovoltaic power generation equipment is added to simulate the operation of modern new power systems. Five tie lines are added on the basis of the classic IEEE-33 node system. The tie lines are backup lines and are usually disconnected. They are mainly used for topological adjustments during microgrid fault maintenance and peak power periods. The grid topology structure built in this paper is shown in Figure 3.

3.2. DQN Model

The DQN (deep Q-network) model [26,27,28,29] is an algorithm that combines deep learning and reinforcement learning. It was proposed by DeepMind to solve the Markov decision process problem in a discrete action space. The DQN combines deep neural networks with Q-learning algorithms and applies deep learning to reinforcement learning tasks, greatly expanding the application scope and performance of reinforcement learning.

3.2.1. Fundamental

The core idea of the DQN is to use deep neural networks to approximate the action–value function $Q\left(s,a\right)$ in Q-learning, where $\mathrm{s}$ represents the current state and $a$ represents the action taken in states. $Q\left(s,a\right)$ represents the expected benefit of taking action, $a$, in states. Traditional Q-learning algorithms maintain a Q-table to record the Q-value of each state–action pair, but this method becomes infeasible when the state space is large or continuous. The DQN estimates these Q-values through neural networks, so that it can handle high-dimensional state spaces.

Therefore, the DQN inherits the value update criterion in the Q-learning method, as shown in Formula (2), which updates the action–value function $Q\left(s,a\right)$ in the form of a difference.

$$Q(s_t,a_t)=Q(s_t,a_t)+\alpha \left[R_t+\gamma \underset{a}{\max }Q(s_{t+1},a)-Q(s_t,a_t)\right],$$

(2)

Unlike the Q-learning method, the original Q-learning algorithm uses limited data, and each experience sample is only used to update the value function once, which limits the efficiency and effect of learning to a certain extent. In order to bridge the gap between Q-learning and deep neural networks, the DQN algorithm introduces the “experience replay” mechanism.

The core idea of the experience replay mechanism is to set up a memory bank, specifically for storing the experiences generated during the interaction between the agent and the environment, that is, the four-tuple data consisting of “state, action, reward, and subsequent state”. Every time the agent takes an action in the environment, the relevant experience samples are stored in this memory bank. In the subsequent Q-network training phase, the algorithm no longer directly relies on the latest environmental feedback, but randomly extracts a batch of samples from the memory bank as training input. The subtlety of this design lies in two points:

• Breaking sequence dependency: by disrupting the order of samples, the experience replay mechanism helps to reduce the temporal correlation of training data and avoids the gradient estimation offset caused by a strong correlation between consecutive samples, thereby improving the generalization ability of the model.
• Data reuse: the reuse of experience samples not only improves the utilization rate of data, but also enhances the adaptability of the model to environmental dynamics, especially in scenarios where data are scarce or the acquisition cost is high.

3.2.2. Process

The innovative deep reinforcement learning method proposed in this paper aims to optimize the network topology of microgrids, especially in the face of node failure challenges. The core of this strategy is to use the intelligent decision-making ability of the DQN to achieve the dynamic adjustment of the microgrid topology to enhance the stability and efficiency of the system. The specific process is shown in Figure 4.

First, the microgrid is initialized to establish its initial architecture and parameter settings, including the status of each node and the connection relationship between them. At the same time, the DQN model is initialized, including the network architecture initialization and model parameter initialization.

Next, we enter the model training phase. In order to simulate real fault conditions, some nodes are randomly set to fail in the simulated environment of the microgrid as training samples. The DQN model receives the current state information of the microgrid, including the location of the faulty node and the overall topology of the microgrid, and then makes decisions and executes the most appropriate actions based on this information, such as reconfiguring network connections or starting backup lines to achieve the goal of optimizing the microgrid structure. After each action is executed, the microgrid enters a new state and calculates a reward for the new state. The reward is the current microgrid node failure rate $\rho$. Then, the model is reversely updated according to the current reward, and the optimal model is obtained after multiple iterations.

After the model training is completed, it is deployed in the actual microgrid environment as an intelligent tool for real-time monitoring and response to faults. When the microgrid encounters a node failure, the system immediately inputs the fault status and current topology data into the trained DQN model. The model quickly analyzes this information and outputs an optimal topology state adjustment strategy to guide the microgrid on how to adjust its network structure immediately and effectively to minimize the impact of the fault and restore the stability and reliability of the power supply.

4. Experimental Settings

This section will conduct an experimental verification on several node failures that occur frequently in microgrids to verify the effectiveness of the method proposed in this paper and set up a control experiment. The reference group uses the particle swarm optimization algorithm, but not the optimization algorithm. The particle swarm optimization algorithm is a heuristic global optimization method inspired by the social modeling of bird flock flight behavior. It aims to solve complex optimization problems by simulating the process of birds looking for food.

4.1. Single Node Failure in Microgrid

A single node failure refers to a situation where a key device or node in the microgrid fails, including the following aspects: power failure, sudden shutdown of new energy, or failure of energy storage equipment, which will cause a power imbalance in the microgrid and affect the stability of the power supply. A load failure or abnormally large current may cause an overload or short circuit, thus affecting the safe operation of the entire microgrid. Line failures, including short circuits, open circuits or insulation damage caused by line aging; these failures will affect power transmission and even cause safety problems such as fire. Control equipment failures, such as the failure of protective equipment such as circuit breakers and relays, may result in the inability to isolate the fault in time, further expanding the scope of impact.

In the experimental verification of this scenario, a single node inside the microgrid is selected, and the fault state is set within a set period of time. The experiment verifies the microgrid topology optimization effect of the algorithm proposed in this article and compares it with the control group. The experimental settings are shown in the

Table 1. The experimental settings of single node failure in microgrid.

Faulty Node ID	Faulty Node Type	Fault Duration/h	Fault Period
10	PV	2	1–3

.

Node 10 is a PV, which is a node with a high incidence of failure in daily microgrid operation. Taking hours as the minimum unit, it is set to fail in 1–3 h and the maintenance time is 2 h. As can be seen from the Figure 5, when the PV device at node 10 fails, if there is no change in the microgrid topology, all loads at nodes 11–18 will be in a power-off state, causing great damage to the user’s experience and production and life.

The network topology is optimized for the single node failure mentioned above. As mentioned above, the topology structure is adjusted in the optimization process with the goal of reducing the internal node failure rate of the microgrid $\rho$. In this process, the power grid must not be looped, because in a ring-configured microgrid, unnecessary current circulation does lead to unnecessary energy consumption and may cause equipment overload risks, posing a threat to the reliability of the system. At the same time, fault detection and isolation under the ring structure are more complicated than linear or radial networks, requiring more sophisticated protection strategies and more advanced control technologies, which increase the complexity and cost of the system.

After model processing, the proposed method and the particle swarm optimization algorithm of the control group obtain the same optimization results, as shown in Figure 6. Under a single node failure, the changes in the internal fault nodes of the microgrid are shown in Figure 7. The impact range of the node failure is reduced by 21% compared with the case without optimization.

The changes in network topology are shown in

Table 2. The changes in network topology under single node failure.

Nodes Involved	Before Adjustment	After Adjustment
11-18	10-11-12-13-14-15-16-17-18	22-12-11 22-12-13-14-15-16-17-18

.

4.2. Multi-Node Failure in Microgrid

Microgrid multi-node failure refers to a fault condition that occurs simultaneously or successively at two or more nodes in a microgrid system. These faults may include but are not limited to an overload, short circuit, open circuit, or power failure, and they may occur in multiple locations such as power generation units, energy storage systems, load ends, or connecting lines. Multi-node failures pose a major challenge to the stable operation of microgrids, as they may cause power outages in some microgrids or even the entire system, affect power quality, increase the complexity of fault diagnosis and recovery, and may trigger chain reactions, further expanding the scope of the fault. Therefore, for the design and management of microgrids, special attention must be given to the prevention and rapid response capabilities of multi-node failures to ensure the reliability and resilience of the system.

In the experimental verification of this scenario, several nodes inside the microgrid are selected, and the fault state is set within the set period. The experiment verifies the microgrid topology optimization effect of the algorithm proposed in this paper and compares it with the control group. The experimental settings are shown in

Table 3. The experimental settings of multi-node failure in microgrid.

Faulty Node ID	Faulty Node Type	Fault Duration/h	Fault Period
5	PV	3	1–4
14	Load	3	1–4
28	Load	3	1–4

.

Taking hours as the minimum unit, it is assumed that the failure occurs in 1–4 h and the maintenance time is 3 h. As shown in Figure 8, when a fault occurs inside the microgrid, if there is no change in the microgrid topology, the microgrid will cause more than two-thirds of the fault area.

The network topology is optimized for the above multi-node failures. Under the premise of avoiding a closed-loop operation, the topology structure is changed to keep the internal failure rate of the microgrid at the lowest rate. After the optimization and adjustment, the new microgrid topology is obtained, as shown in Figure 9.

Under multi-node failures, the changes of faulty nodes inside the microgrid are shown in Figure 10. The impact range of node failures is reduced by 58% compared with the case without optimization.

The changes in network topology are shown in

Table 4. The changes in network topology under multiple node failure.

Nodes Involved	Before Adjustment	After Adjustment
6-18 26-33	6-7-8-9-10-11-12-13-14-15-16-17-18 6-26-27-28-29-30-31-32-33	21-8-7-6-26-27 21-8-9-10-11-12-13 21-8-9-15-16-17-18-33-32-31-30-29

.

5. Conclusions

The deep reinforcement learning network topology optimization method (DRL-NNF) proposed in this paper can effectively control grid faults from the microgrid topology level by building models, training models, and deploying models for microgrids, which can greatly improve the fault defense capability of microgrids. The model proposed in this paper has the following three contributions:

• From the research method and entry point, the model comprehensively considers the topology of the microgrid, rather than focusing only on a single component or local problem, thereby achieving comprehensive security optimization and ensuring that the various power nodes within the microgrid work together to pursue system-level security maximization.
• In terms of the ability to cope with the complex power grid environment, this method has made a breakthrough. Using the powerful ability of deep reinforcement learning, DRL-NNF can cope with high-dimensional state spaces and has advantages in dealing with modern microgrids with complex structures and dynamic characteristics. DRL-NNF uses the powerful characterization capability of deep networks to avoid the complex problems of traditional methods in the face of complex topologies.
• In terms of the universality of the method, the scope of application of DRL-NNF is not limited to fault control. This method is also applicable to other key areas in the operation of microgrids, such as load distribution, line loss reduction, and energy efficiency improvement. Through the migration and application of models, it is possible to flexibly switch between different optimization objectives without training new models from scratch, which greatly saves time and computing resources, and also enhances the flexibility and efficiency of microgrids in multi-faceted performance optimization.