Search Results (638)

This study investigates the application of Vision Transformers (ViTs) in deep reinforcement learning (DRL) for autonomous driving systems that rely solely on visual input. While convolutional neural networks (CNNs) are widely used for visual processing, they have limitations in capturing global patterns and handling complex driving scenarios. To address these challenges, we developed a ViT-based DRL model and evaluated its performance through extensive training in the MetaDrive simulator and testing in the high-fidelity AirSim simulator. Results show that the ViT-based model significantly outperformed CNN baselines in MetaDrive, achieving nearly seven times the average distance traveled and an 87% increase in average speed. In AirSim, the model exhibited superior adaptability to realistic conditions, maintaining stability and safety in visually complex environments. These findings highlight the potential of ViTs to enhance the robustness and reliability of vision-based autonomous systems, offering a transformative approach to safe exploration in diverse driving scenarios. Full article

Buildings account for a substantial portion of global energy use, with about one-third of total consumption attributed to them, according to IEA statistics, significantly contributing to carbon emissions. Building energy efficiency is crucial for combating climate change and achieving energy savings. Smart buildings, leveraging intelligent control systems, optimize energy use to reduce consumption and emissions. Deep reinforcement learning (DRL) algorithms have recently gained attention for heating, ventilation, and air conditioning (HVAC) control in buildings. This paper reviews current research on DRL-based HVAC management and identifies key issues in existing algorithms. We propose an enhanced intelligent building energy management algorithm based on the Soft Actor–Critic (SAC) framework to address these challenges. Our approach employs the distributed soft policy iteration from the Distributional Soft Actor–Critic (DSAC) algorithm to improve action–state return stability. Specifically, we introduce cumulative returns into the SAC framework and recalculate target values, which reduces the loss function. The proposed HVAC control algorithm achieved 24.2% energy savings compared to the baseline SAC algorithm. This study contributes to the development of more energy-efficient HVAC systems in smart buildings, aiding in the fight against climate change and promoting energy savings. Full article

Deep Reinforcement Learning (DRL) has emerged as a transformative approach in mobile robot path planning, addressing challenges associated with dynamic and uncertain environments. This comprehensive review categorizes and analyzes DRL methodologies, highlighting their effectiveness in navigating high-dimensional state–action spaces and adapting to complex real-world scenarios. The paper explores value-based methods like Deep Q-Networks (DQNs) and policy-based strategies such as Proximal Policy Optimization (PPO) and Soft Actor–Critic (SAC), emphasizing their contributions to efficient and robust navigation. Hybrid approaches combining these methodologies are also discussed for their adaptability and enhanced performance. Additionally, the review identifies critical gaps in current research, including limitations in scalability, safety, and generalization, proposing future directions to advance the field. This work underscores the transformative potential of DRL in revolutionizing mobile robot navigation across diverse applications, from search-and-rescue missions to autonomous urban delivery systems. Full article

Low-orbit satellite communication networks have gradually become the research focus of fifth-generation (5G) beyond and sixth generation (6G) networks due to their advantages of wide coverage, large communication capacity, and low terrain influence. However, the low earth orbit mega satellite network (LEO-MSN) also has difficulty in constructing stable traffic transmission paths, network load imbalance and congestion due to the large scale of network nodes, a highly complex topology, and uneven distribution of traffic flow in time and space. In the service-based architecture proposed by 3GPP, the introduction of service function chain (SFC) constraints exacerbates these challenges. Therefore, in this paper, we propose GDRL-SFCR, an end-to-end routing decision method based on graph neural network (GNN) and deep reinforcement learning (DRL) which jointly optimize the end-to-end transmission delay and network load balancing under SFC constraints. Specifically, this method constructs the system model based on the latest NTN low-orbit satellite network end-to-end transmission architecture, taking into account the SFC constraints, transmission delays, and network node loads in the end-to-end traffic transmission, uses a GNN to extract node attributes and dynamic topology features, and uses the DRL method to design specific reward functions to train the model to learn routing policies that satisfy the SFC constraints. The simulation results demonstrate that, compared with graph theory-based methods and reinforcement learning-based methods, GDRL-SFCR can reduce the end-to-end traffic transmission delay by more than 11.3%, reduce the average network load by more than 14.1%, and increase the traffic access success rate and network capacity by more than 19.1% and two times, respectively. Full article

Emergency logistics is essential for rapid and efficient disaster response, ensuring the timely availability and deployment of resources to affected areas. In the process of rescue work, the dynamic changes in rescue point information greatly increase the difficulty of rescue. This paper establishes a combined neural network model considering soft time-window penalty and applies deep reinforcement learning (DRL) to address the dynamic routing problem in emergency logistics. This method utilizes the actor–critic framework, combined with attention mechanisms, pointer networks, and long short-term memory neural networks, to determine effective disaster relief path, and it compares the obtained scheduling scheme with the results obtained from the DRL algorithm based on the single-network model and ant colony optimization (ACO) algorithm. Simulation experiments show that the proposed method reduces the solution accuracy by nearly 10% compared to the ACO algorithm, but it saves nearly 80% in solution time. Additionally, it slightly increases solution times but improves accuracy by nearly 20% over traditional DRL approaches, demonstrating a promising balance between performance efficiency and computational resource utilization in emergency logistics. Full article

This paper investigates the challenges and solutions associated with integrating a hydrogen-generating nuclear-renewable integrated energy system (NR-IES) under a transactive energy framework. The proposed system directs excess nuclear power to hydrogen production during periods of low grid demand while utilizing renewables to maintain grid stability. Using digital real-time simulation (DRTS) in the Typhoon HIL 404 model, the dynamic interactions between nuclear power plants, electrolyzers, and power grids are analyzed to mitigate issues such as harmonic distortion, power quality degradation, and low power factor caused by large non-linear loads. A three-phase power conversion system is modeled using the Typhoon HIL 404 model and includes a generator, a variable load, an electrolyzer, and power filters. Active harmonic filters (AHFs) and hybrid active power filters (HAPFs) are implemented to address harmonic mitigation and reactive power compensation. The results reveal that the HAPF topology effectively balances cost efficiency and performance and significantly reduces active filter current requirements compared to AHF-only systems. During maximum electrolyzer operation at 4 MW, the grid frequency dropped below 59.3 Hz without filtering; however, the implementation of power filters successfully restored the frequency to 59.9 Hz, demonstrating its effectiveness in maintaining grid stability. Future work will focus on integrating a deep reinforcement learning (DRL) framework with real-time simulation and optimizing real-time power dispatch, thus enabling a scalable, efficient NR-IES for sustainable energy markets. Full article

In the field of gaming artificial intelligence, selecting the appropriate machine learning approach is essential for improving decision-making and automation. This paper examines the effectiveness of deep reinforcement learning (DRL) within interactive gaming environments, focusing on complex decision-making tasks. Utilizing the Unity engine, we conducted experiments to evaluate DRL methodologies in simulating realistic and adaptive agent behavior. A vehicle driving game is implemented, in which the goal is to reach a certain target within a small number of steps, while respecting the boundaries of the roads. Our study compares Proximal Policy Optimization (PPO) and Soft Actor–Critic (SAC) in terms of learning efficiency, decision-making accuracy, and adaptability. The results demonstrate that PPO successfully learns to reach the target, achieving higher and more stable cumulative rewards. Conversely, SAC struggles to reach the target, displaying significant variability and lower performance. These findings highlight the effectiveness of PPO in this context and indicate the need for further development, adaptation, and tuning of SAC. This research contributes to developing innovative approaches in how ML can improve how player agents adapt and react to their environments, thereby enhancing realism and dynamics in gaming experiences. Additionally, this work emphasizes the utility of using games to evolve such models, preparing them for real-world applications, namely in the field of vehicles’ autonomous driving and optimal route calculation. Full article

High route loss and line-of-sight requirements are two of the fundamental challenges of millimeter-wave (mm-wave) communications that are mitigated by incorporating sensor technology. Sensing gives the deep reinforcement learning (DRL) agent comprehensive environmental feedback, which helps it better predict channel fluctuations and modify beam patterns accordingly. For multi-user massive multiple-input multiple-output (mMIMO) systems, hybrid precoding requires sophisticated real-time low-complexity power allocation (PA) approaches to achieve near-optimal capacity. This study presents a unique angular-based hybrid precoding (AB-HP) framework that minimizes radio frequency (RF) chain and channel estimation while optimizing energy efficiency (EE) and spectral efficiency (SE). DRL is essential for mm-wave technology to make adaptive and intelligent decision-making possible, which effectively transforms wireless communication systems. DRL optimizes RF chain usage to maintain excellent SE while drastically lowering hardware complexity and energy consumption in an AB-HP architecture by dynamically learning optimal precoding methods using environmental angular information. This article proposes enabling dual optimization of EE and SE while drastically lowering beam training overhead by incorporating maximum reward beam training driven (RBT) in the DRL. The proposed RBT-DRL improves system performance and flexibility by dynamically modifying the number of active RF chains in dynamic network situations. The simulation results show that RBT-DRL-driven beam training guarantees good EE performance for mobile users while increasing SE in mm-wave structures. Even though total power consumption rises by 45%, the SE improves by 39%, increasing from 14 dB to 20 dB, suggesting that this strategy could successfully achieve a balance between performance and EE in upcoming B5G networks. Full article

Internet of Things (IoT) technology has facilitated the deployment of autonomous sensors in remote and challenging environments, enabling substantial advancements in environmental monitoring and data collection. IoT sensors continuously gather data, transmitting it to a central Base Station (BS) via designated Cluster Heads (CHs). However, data flow encounters frequent congestion at CH nodes, negatively impacting network performance and Quality of Service (QoS). This paper introduces a novel congestion control strategy tailored for Wireless Sensor Networks (WSNs) to balance energy efficiency and data reliability. The proposed approach follows an eight-step process, integrating Generative Adversarial Networks (GANs) for enhanced clustering and Ant Colony Optimization (ACO) for optimal CH selection and routing. GANs simulate realistic node clustering, achieving better load distribution and energy conservation across the network. ACO then selects CHs based on energy levels, distance, and network centrality, using pheromone-based routing to adaptively manage data flows. A congestion factor (CF) threshold is also incorporated to dynamically reroute traffic when congestion risks arise, preserving QoS. Simulation results show that this approach significantly improves QoS metrics, including latency, throughput, and reliability. Comparative evaluations reveal that our method outperforms existing frameworks, such as Fuzzy Structure and Genetic-Fuzzy (FSFG), Deep Reinforcement Learning Cache-Aware Congestion Control (DRL-CaCC), and Adaptive Cuckoo Search Rate Optimization (ACSRO). Full article

Common challenges in the area of robotics include issues such as sensor modeling, dynamic operating environments, and limited on-broad computational resources. To improve decision making, robots need a dependable framework to facilitate communication between different modules and the optimal action for real-world applications. The Robotics Operating System (ROS) and Reinforcement Learning (RL) are two promising approaches that help accomplish precise control, seamless integration of sensors-actuators, and exhibit learned behavior. The ROS enables seamless communication between heterogeneous components, while RL focuses on learning optimal behaviors through trial-and-error scenarios. Combining the ROS and RL offers superior decision making, improved perception, enhanced automation, and reliability. This work focuses on investigating ROS-based RL applications across various domains, aiming to enhance understanding through comprehensive discussion, analysis, and summarization. We base our evaluation on the application area, type of RL algorithm used, and degree of ROS–RL integration. Additionally, we provide summary of seminal works that define the current state of the art, along with GitHub repositories and resources for research purposes. Based on the review of successfully implemented projects, we make recommendations highlighting the advantages and limitations of RL techniques for specific applications and environments. The ultimate goal of this work is to advance the robotics field by providing a comprehensive overview of the recent important works that incorporate both the ROS and RL, thereby improving the adaptability of these emerging techniques. Full article

This paper investigates the problem of angular momentum control and planning for control moment gyroscope(CMG) arrays in rigid spacecraft attitude control systems using deep reinforcement learning (DRL). Specifically, a DRL-based angular momentum control strategy is proposed for spacecraft attitude control systems employing multiple CMGs as actuators. The twin-delayed deep deterministic policy gradient (TD3) algorithm is used to perform online learning and policy updates based on environmental feedback. This approach eliminates the need for precise mathematical models and iterative parameter tuning. This enables the CMG system to perform angular momentum planning and facilitates rapid and high-precision spacecraft attitude maneuvers and control through angular momentum exchange. Simulations were performed to analyze spacecraft attitude maneuvers and stabilization under various scenarios, focusing on the angular momentum control process of a pyramidal single gimbal CMG (SGCMG) array. The results demonstrate that the proposed method effectively achieves large-angle attitude maneuvers and stable attitude maintenance, both in ideal conditions and in the presence of nonlinear disturbances. During large-angle maneuvers, the spacecraft’s attitude estimation using MRPs converges in less than 1 min, and the convergence accuracy during attitude-holding reaches the order of 10⁻³. Moreover, the approach fully leverages the output characteristics of the CMG system and achieves robust performance and accuracy, even under conditions with significant noise and disturbances. Full article

Many efforts have proved that deep learning models are effective for disease prediction using electronic health records (EHRs). However, these models are not yet precise enough to predict diseases. Additionally, ethical concerns and the use of clustering and classification algorithms on small datasets limit their effectiveness. The complexity of data processing further complicates the interpretation of patient representation learning models, even though data augmentation strategies may help. Incomplete patient data also hinder model accuracy. This study aims to develop and evaluate a deep learning model that addresses these challenges. Our proposed approach is to design a disease prediction model based on deep Q-learning (DQL), which replaces the traditional Q-learning reinforcement learning algorithm with a neural network deep learning model, and the mapping capabilities of the Q-network are utilized. We conclude that the proposed model achieves the best accuracy (98%) compared with other models. Full article

This paper tackles the green permutation flow shop scheduling problem (GPFSP) with the goal of minimizing both the maximum completion time and energy consumption. It introduces a novel hybrid approach that combines end-to-end deep reinforcement learning with an improved genetic algorithm. Firstly, the PFSP is modeled using an end-to-end deep reinforcement learning (DRL) approach, named PFSP_NET, which is designed based on the characteristics of the PFSP, with the actor–critic algorithm employed to train the model. Once trained, this model can quickly and directly produce relatively high-quality solutions. Secondly, to further enhance the quality of the solutions, the outputs from PFSP_NET are used as the initial population for the improved genetic algorithm (IGA). Building upon the traditional genetic algorithm, the IGA utilizes three crossover operators, four mutation operators, and incorporates hamming distance, effectively preventing the algorithm from prematurely converging to local optimal solutions. Then, to optimize energy consumption, an energy-saving strategy is proposed that reasonably adjusts the job scheduling order by shifting jobs backward without increasing the maximum completion time. Finally, extensive experimental validation is conducted on the 120 test instances of the Taillard standard dataset. By comparing the proposed method with algorithms such as the standard genetic algorithm (SGA), elite genetic algorithm (EGA), hybrid genetic algorithm (HGA), discrete self-organizing migrating algorithm (DSOMA), discrete water wave optimization algorithm (DWWO), and hybrid monkey search algorithm (HMSA), the results demonstrate the effectiveness of the proposed method. Optimal solutions are achieved in 28 test instances, and the latest solutions are updated in instances Ta005 and Ta068 with values of 1235 and 5101, respectively. Additionally, experiments on 30 instances, including Taillard 20-10, Taillard 50-10, and Taillard 100-10, indicate that the proposed energy strategy can effectively reduce energy consumption. Full article

Increasing traffic density in cities exacerbates air pollution, threatens human health and worsens the global climate crisis. Urgent solutions for sustainable and eco-friendly urban transportation are needed. Innovative technologies like artificial intelligence, particularly Deep Reinforcement Learning (DRL), play a crucial role in reducing fuel consumption and emissions. This study presents an effective approach using DRL to minimize waiting times at traffic lights, thus reducing fuel consumption and emissions. DRL can evaluate complex traffic scenarios and learn optimal solutions. Unlike other studies focusing solely on optimizing traffic light durations, this research aims to choose the optimal vehicle acceleration based on traffic conditions. This method provides safer, more comfortable travel while lowering emissions and fuel consumption. Simulations with various scenarios prove the Deep Q-Network (DQN) algorithm’s success in adjusting speed according to traffic lights. Although the findings confirmed that the DRL algorithms used were effective in reducing fuel consumption and emissions, the DQN algorithm outperformed other DRL algorithms in reducing fuel consumption and emissions in complex city traffic scenarios, and in reducing waiting times at traffic lights. It provides better contributions to creating a sustainable environment by reducing fuel consumption and emissions. Full article

The increasing volume of traffic has led to severe challenges, including traffic congestion, heightened energy consumption, increased air pollution, and prolonged travel times. Addressing these issues requires innovative approaches for optimizing road network utilization. While Deep Reinforcement Learning (DRL)-based methods have shown remarkable effectiveness in dynamic scenarios like traffic management, their primary focus has been on single-agent setups, limiting their applicability to real-world multi-agent systems. Managing agents and fostering collaboration in a multi-agent reinforcement learning scenario remains a challenging task. This paper introduces a cooperative multi-agent federated reinforcement learning algorithm named FLDQN to address the challenge of agent cooperation by solving travel time minimization challenges in dynamic multi-agent reinforcement learning (MARL) scenarios. FLDQN leverages federated learning to facilitate collaboration and knowledge sharing among intelligent agents, optimizing vehicle routing and reducing congestion in dynamic traffic environments. Using the SUMO simulator, multiple agents equipped with deep Q-learning models interact with their local environments, share model updates via a federated server, and collectively enhance their policies using unique local observations while benefiting from the collective experiences of other agents. Experimental evaluations demonstrate that FLDQN achieves a significant average reduction of over 34.6% in travel time compared to non-cooperative methods while simultaneously lowering the computational overhead through distributed learning. FLDQN underscores the vital impact of agent cooperation and provides an innovative solution for enabling agent cooperation in a multi-agent environment. Full article