Search Results (1,280)

In this paper, we introduce an agent rescue scheduling approach grounded in proximal policy optimization, coupled with a singularity-free predefined-time control strategy. The primary objective of this methodology is to bolster the efficiency and precision of rescue missions. Firstly, we have designed an evaluation function closely related to the average flying distance of agents, which provides a quantitative benchmark for assessing different scheduling schemes and assists in optimizing the allocation of rescue resources. Secondly, we have developed a scheduling strategy optimization method using the Proximal Policy Optimization (PPO) algorithm. This method can automatically learn and adjust scheduling strategies to adapt to complex rescue environments and varying task demands. The evaluation function provides crucial feedback signals for the PPO algorithm, ensuring that the algorithm can precisely adjust the scheduling strategies to achieve optimal results. Thirdly, aiming to attain stability and precision in agent navigation to designated positions, we formulate a singularity-free predefined-time fuzzy adaptive tracking control strategy. This approach dynamically modulates control parameters in reaction to external disturbances and uncertainties, thus ensuring the precise arrival of agents at their destinations within the predefined time. Finally, to substantiate the validity of our proposed approach, we crafted a simulation environment in Python 3.7, engaging in a comparative analysis between the PPO and the other optimization method, Deep Q-network (DQN), utilizing the variation in reward values as the benchmark for evaluation. Full article

In medicine, dynamic treatment regimes (DTRs) have emerged to guide personalized treatment decisions for patients, accounting for their unique characteristics. However, existing methods for determining optimal DTRs face limitations, often due to reliance on linear models unsuitable for complex disease analysis and a focus on outcome prediction over treatment effect estimation. To overcome these challenges, decision tree-based reinforcement learning approaches have been proposed. Our study aims to evaluate the performance and feasibility of such algorithms: tree-based reinforcement learning (T-RL), DTR-Causal Tree (DTR-CT), DTR-Causal Forest (DTR-CF), stochastic tree-based reinforcement learning (SL-RL), and Q-learning with Random Forest. Using real-world clinical data, we conducted experiments to compare algorithm performances. Evaluation metrics included the proportion of correctly assigned patients to recommended treatments and the empirical mean with standard deviation of expected counterfactual outcomes based on estimated optimal treatment strategies. This research not only highlights the potential of decision tree-based reinforcement learning for dynamic treatment regimes but also contributes to advancing personalized medicine by offering nuanced and effective treatment recommendations. Full article

Tool condition monitoring (TCM) is crucial in the machining process to confirm product quality as well as process efficiency and minimize downtime. Traditional methods for TCM, while effective to a degree, often fall short in real-time adaptability and predictive accuracy. This research work aims to advance the state-of-the-art methods in predictive maintenance for TCM and improve tool performance and reliability during the milling process. The present work investigates the application of Deep Learning (DL) and Reinforcement Learning (RL) techniques to monitor tool conditions in milling operations. DL models, including Long Short-Term Memory (LSTM) networks, Feed Forward Neural Networks (FFNN), and RL models, including Q-learning and SARSA, are employed to classify tool conditions from the vibration sensor. The performance of the selected DL and RL algorithms is evaluated through performance metrics like confusion matrix, recall, precision, F1 score, and Receiver Operating Characteristics (ROC) curves. The results revealed that RL based on SARSA outperformed other algorithms. The overall classification accuracies for LSTM, FFNN, Q-learning, and SARSA were 94.85%, 98.16%, 98.50%, and 98.66%, respectively. In regard to predicting tool conditions accurately and thereby enhancing overall process efficiency, SARSA showed the best performance, followed by Q-learning, FFNN, and LSTM. This work contributes to the advancement of TCM systems, highlighting the potential of DL and RL techniques to revolutionize manufacturing processes in the era of Industry 5.0. Full article

Permanent-magnet synchronous motors (PMSMs) have played a key role in recent years in both industrial and commercial applications. Despite their many significant advantages, such as high efficiency, very good dynamics, and high power density, these types of motors are prone to various types of faults. This article proposes a low-cost microcontroller-based system for PMSM stator winding condition monitoring and fault diagnosis. It meets the demand created by the use of more and more low-budget solutions in industrial and commercial applications. A printed circuit board (PCB) has been developed to measure PMSM stator phase currents, which are used as diagnostic signals. The key components of this PCB are LEM’s LESR 6-NP current transducers. The acquisition and processing of diagnostic signals using a low-cost embedded system (NUCLEO-H7A3ZI-Q) with an ARM Cortex-M core is described in detail. A machine learning-driven KNN-based fault diagnostic algorithm is implemented to detect and classify incipient PMSM stator winding faults (interturn short-circuits). The effects of the severity of the fault and the motor operating conditions on the symptom extraction process are also investigated. The results of experimental tests conducted on a 2.5 kW PMSM confirmed the effectiveness of the developed system. Full article

The transmission environment of underwater wireless sensor networks is open, and important transmission data can be easily intercepted, interfered with, and tampered with by malicious nodes. Malicious nodes can be mixed in the network and are difficult to distinguish, especially in time-varying underwater environments. To address this issue, this article proposes a GAN-based trusted routing algorithm (GTR). GTR defines the trust feature attributes and trust evaluation matrix of underwater network nodes, constructs the trust evaluation model based on a generative adversarial network (GAN), and achieves malicious node detection by establishing a trust feature profile of a trusted node, which improves the detection performance for malicious nodes in underwater networks under unlabeled and imbalanced training data conditions. GTR combines the trust evaluation algorithm with the adaptive routing algorithm based on Q-Learning to provide an optimal trusted data forwarding route for underwater network applications, improving the security, reliability, and efficiency of data forwarding in underwater networks. GTR relies on the trust feature profile of trusted nodes to distinguish malicious nodes and can adaptively select the forwarding route based on the status of trusted candidate next-hop nodes, which enables GTR to better cope with the changing underwater transmission environment and more accurately detect malicious nodes, especially unknown malicious node intrusions, compared to baseline algorithms. Simulation experiments showed that, compared to baseline algorithms, GTR can provide a better malicious node detection performance and data forwarding performance. Under the condition of 15% malicious nodes and 10% unknown malicious nodes mixed in, the detection rate of malicious nodes by the underwater network configured with GTR increased by 5.4%, the error detection rate decreased by 36.4%, the packet delivery rate increased by 11.0%, the energy tax decreased by 11.4%, and the network throughput increased by 20.4%. Full article

Efficient and reliable data routing is critical in Advanced Metering Infrastructure (AMI) within Smart Grids, dictating the overall network performance and resilience. This paper introduces Q-RPL, a novel Q-learning-based Routing Protocol designed to enhance routing decisions in AMI deployments based on wireless mesh technologies. Q-RPL leverages the principles of Reinforcement Learning (RL) to dynamically select optimal next-hop forwarding candidates, adapting to changing network conditions. The protocol operates on top of the standard IPv6 Routing Protocol for Low-Power and Lossy Networks (RPL), integrating it with intelligent decision-making capabilities. Through extensive simulations carried out in real map scenarios, Q-RPL demonstrates a significant improvement in key performance metrics such as packet delivery ratio, end-to-end delay, and compliant factor compared to the standard RPL implementation and other benchmark algorithms found in the literature. The adaptability and robustness of Q-RPL mark a significant advancement in the evolution of routing protocols for Smart Grid AMI, promising enhanced efficiency and reliability for future intelligent energy systems. The findings of this study also underscore the potential of Reinforcement Learning to improve networking protocols. Full article

In addressing the complex challenges of path planning in multi-robot systems, this paper proposes a novel Hybrid Decentralized and Centralized Training and Execution (DCTE) Strategy, aimed at optimizing computational efficiency and system performance. The strategy solves the prevalent issues of collision and coordination through a tiered optimization process. The DCTE strategy commences with an initial decentralized path planning step based on Deep Q-Network (DQN), where each robot independently formulates its path. This is followed by a centralized collision detection the analysis of which serves to identify potential intersections or collision risks. Paths confirmed as non-intersecting are used for execution, while those in collision areas prompt a dynamic re-planning step using DQN. Robots treat each other as dynamic obstacles to circumnavigate, ensuring continuous operation without disruptions. The final step involves linking the newly optimized paths with the original safe paths to form a complete and secure execution route. This paper demonstrates how this structured strategy not only mitigates collision risks but also significantly improves the computational efficiency of multi-robot systems. The reinforcement learning time was significantly shorter, with the DCTE strategy requiring only 3 min and 36 s compared to 5 min and 33 s in the comparison results of the simulation section. The improvement underscores the advantages of the proposed method in enhancing the effectiveness and efficiency of multi-robot systems. Full article

Background: Colorectal cancer (CRC) ranks among the most prevalent malignancies affecting the gastrointestinal tract. The infiltration of CD8⁺ T cells significantly influences the prognosis and progression of tumor patients. Methods: This study establishes a CRC immune risk model based on CD8⁺ T cell-related genes. CD8⁺ T cell-related genes were identified through Weighted Gene Co-expression Network Analysis (WGCNA), and the enriched gene sets were annotated via Gene Ontology (GO) and Reactome pathway analysis. Employing machine learning methods, including the Least Absolute Shrinkage and Selection Operator (LASSO) algorithm and Random Forest (RF), we identified nine genes associated with CD8⁺ T-cell infiltration. The infiltration levels of immune cells in CRC tissues were assessed using the ssGSEA algorithm. Results: These genes provide a foundation for constructing a prognostic model. The TCGA-CRC sample model’s prediction scores were categorized, and the prediction models were validated through Cox regression analysis and Kaplan–Meier curve analysis. Notably, although CRC tissues with higher risk scores exhibited elevated levels of CD8⁺ T-cell infiltration, they also demonstrated heightened expression of immune checkpoint genes. Furthermore, comparison of microsatellite instability (MSI) and gene mutations across the immune subgroups revealed notable gene variations, particularly with APC, TP53, and TNNT1 showing higher mutation frequencies. Finally, the predictive model’s efficacy was corroborated through the use of Tumor Immune Dysfunction and Exclusion (TIDE), Immune Profiling Score (IPS), and immune escape-related molecular markers. The predictive model was validated through an external cohort of CRC and the Bladder Cancer Immunotherapy Cohort. CLRN3 expression levels in tumor and adjacent normal tissues were assessed using quantitative real-time polymerase chain reaction (qRT-PCR) and western blot. Subsequent in vitro and in vivo experiments demonstrated that CLRN3 knockdown significantly attenuated the malignant biological behavior of CRC cells, while overexpression had the opposite effect. Conclusions: This study presents a novel prognostic model for CRC, providing a framework for enhancing the survival rates of CRC patients by targeting CD8⁺ T-cell infiltration. Full article

Algorithms for training agents with experience replay have advanced in several domains, primarily because prioritized experience replay (PER) developed from the double deep Q-network (DDQN) in deep reinforcement learning (DRL) has become a standard. PER-based algorithms have achieved significant success in the image and video domains. However, the exceptional results observed in images and videos are not as effective in many domains with simple action spaces and relatively small states, particularly in discrete action spaces with sparse rewards. Moreover, most advanced techniques may improve sampling efficiency using deep learning algorithms rather than reinforcement learning. However, there is growing evidence that deep learning algorithms cannot generalize during training. Therefore, this study proposes an algorithm suitable for discrete action space environments that uses the sample efficiency of PER based on DDQN but incorporates support vector machine recursive feature elimination (SVM-RFE) without enhancing the sampling efficiency through deep learning algorithms. The proposed algorithm exhibited considerable performance improvements in classical OpenAI Gym environments that did not use images or videos as inputs. In particular, simple discrete space environments with reflection symmetry, such as Cart–Pole, exhibited a faster and more stable learning process. These results suggest that the application of SVM-RFE, which leverages the orthogonality of support vector machines (SVMs) across learning patterns, can be appropriate when the data in the reinforcement learning environment demonstrate symmetry. Full article

This study investigates the optimization of traffic scheduling in autonomous vehicle networks using time-sensitive networking (TSN), a type of deterministic Ethernet. Ethernet has high bandwidth and compatibility to support various protocols, and its application range is expanding from office environments to smart factories, aerospace, and automobiles. TSN is a representative technology of deterministic Ethernet and is composed of various standards such as time synchronization, stream reservation, seamless redundancy, frame preemption, and scheduled traffic, which are sub-standards of IEEE 802.1 Ethernet established by the IEEE TSN task group. In order to ensure real-time transmission by minimizing end-to-end delay in a TSN network environment, it is necessary to schedule transmission timing in all links transmitting ST (Scheduled Traffic). This paper proposes network performance metrics and methods for applying machine learning (ML) techniques to optimize traffic scheduling. This study demonstrates that the traffic scheduling problem, which has NP-hard complexity, can be optimized using ML algorithms. The performance of each algorithm is compared and analyzed to identify the scheduling algorithm that best meets the network requirements. Reinforcement learning algorithms, specifically DQN (Deep Q Network) and A2C (Advantage Actor-Critic) were used, and normalized performance metrics (E2E delay, jitter, and guard band bandwidth usage) along with an evaluation function based on their weighted sum were proposed. The performance of each algorithm was evaluated using the topology of a real autonomous vehicle network, and their strengths and weaknesses were compared. The results confirm that artificial intelligence-based algorithms are effective for optimizing TSN traffic scheduling. This study suggests that further theoretical and practical research is needed to enhance the feasibility of applying deterministic Ethernet to autonomous vehicle networks, focusing on time synchronization and schedule optimization. Full article

Reliability quantification of deep reinforcement learning (DRL)-based control is a significant challenge for the practical application of artificial intelligence (AI) in safety-critical systems. This study proposes a method for quantifying the reliability of DRL-based control. First, an existing method, random network distillation, was applied to the reliability evaluation to clarify the issues to be solved. Second, a novel method for reliability quantification was proposed to solve these issues. The reliability is quantified using two neural networks: a reference and an evaluator. They have the same structure with the same initial parameters. The outputs of the two networks were the same before training. During training, the evaluator network parameters were updated to maximize the difference between the reference and evaluator networks for trained data. Thus, the reliability of the DRL-based control for a state can be evaluated based on the difference in output between the two networks. The proposed method was applied to DRL-based controls as an example of a simple task, and its effectiveness was demonstrated. Finally, the proposed method was applied to the problem of switching trained models depending on the state. Consequently, the performance of the DRL-based control was improved by switching the trained models according to their reliability. Full article

The engine room is the core area of a ship, critical to its operation, safety, and efficiency. Currently, many researchers merely address the ship engine room layout design (SERLD) problem using optimization algorithms and independent layout strategies. However, the engine room environment is complex, involving two significantly different challenges: equipment layout and pipe layout. Traditional methods fail to achieve optimal collaborative layout objectives. To address this research gap, this paper proposes a collaborative layout method that combines improved reinforcement learning and heuristic algorithms. For equipment layout, the engine room space is first discretized into a grid, and a Markov decision process (MDP) framework suitable for equipment layout is proposed, including state space, action space, and reward mechanisms suitable for equipment layout. An improved adaptive guided multi-agent Q-learning (AGMAQL) algorithm is employed to train the layout model in a centralized manner, with enhancements made to the agent’s exploration state, exploration action, and learning strategy. For pipe layout, this paper proposes an improved adaptive trajectory artificial fish swarm algorithm (ATAFSA). This algorithm incorporates a hybrid encoding method, adaptive strategy, scouting strategy, and parallel optimization strategy, resulting in enhanced stability, accuracy, and problem adaptability. Subsequently, by comprehensively considering layout objectives and engine room attributes, a collaborative layout method incorporating hierarchical and adaptive weight strategies is proposed. This method optimizes in phases according to the layout objectives and priorities of different stages, achieving multi-level optimal layouts and providing designers with various reference schemes with different focuses. Finally, based on a typical real-world engine room engineering case, various leading algorithms and strategies are tested and compared. The results show that the proposed AGMAQL-ATAFSA (AGMAQL-ATA) exhibits robustness, efficiency, and engineering practicality. Compared to previous research methods and algorithms, the final layout quality improved overall: equipment layout effectiveness increased by over 4.0%, pipe optimization efficiency improved by over 40.4%, and collaborative layout effectiveness enhanced by over 2.2%. Full article

In recent years, with the development of Internet of Things technology, the demand for low-power wireless communication technology has been growing, giving rise to LoRa technology. A LoRa network mainly consists of terminal nodes, gateways, and LoRa network servers. As LoRa networks often deploy many terminal node devices for environmental sensing, the limited resources of LoRa technology, the explosive growth in the number of nodes, and the ever-changing complex environment pose unprecedented challenges for the performance of the LoRa network. Although some research has already addressed the challenges by allocating channels to the LoRa network, the impact of complex and changing environmental factors on the LoRa network has yet to be considered. Reasonable channel allocation should be tailored to the situation and should face different environments and network distribution conditions through continuous adaptive learning to obtain the corresponding allocation strategy. Secondly, most of the current research only focuses on the channel adjustment of the LoRa node itself. Still, it does not consider the indirect impact of the node’s allocation on the entire network. The Inner External DQN SF allocation method (IEDQN) proposed in this paper improves the packet reception rate of the whole system by using reinforcement learning methods for adaptive learning of the environment. It considers the impact on the entire network of the current node parameter configuration through nested reinforcement learning for further optimization to optimize the whole network’s performance. Finally, this paper evaluates the performance of IEDQN through simulation. The experimental results show that the IEDQN method optimizes network performance. Full article

Pancreatic neuroendocrine neoplasms pose a growing clinical challenge due to their rising incidence and variable prognosis. The current study aims to investigate microRNAs (miRNA; miR) as potential biomarkers for distinguishing between grade 1 (G1) and grade 2 (G2) pancreatic neuroendocrine tumors (PanNET). A total of 33 formalin-fixed, paraffin-embedded samples were analyzed, comprising 17 G1 and 16 G2 tumors. Initially, literature-based miRNAs were validated via real-time quantitative reverse transcription polymerase chain reaction (RT-qPCR), confirming significant downregulation of miR-130b-3p and miR-106b in G2 samples. Through next-generation sequencing, we have identified and selected the top six miRNAs showing the highest difference between G1 and G2 tumors, which were further validated. RT-qPCR validation confirmed the downregulation of miR-30d-5p in G2 tumors. miRNA combinations were created to distinguish between the two PanNET grades. The highest diagnostic performance in distinguishing between G1 and G2 PanNETs by a machine learning algorithm was achieved when using the combination miR-106b + miR-130b-3p + miR-127-3p + miR-129-5p + miR-30d-5p. The ROC analysis resulted in a sensitivity of 83.33% and a specificity of 87.5%. The findings underscore the potential use of miRNAs as biomarkers for stratifying PanNET grades, though further research is warranted to enhance diagnostic accuracy and clinical utility. Full article

This paper applies federated reinforcement learning (FRL) in cellular vehicle-to-everything (C-V2X) communication to enable vehicles to learn communication parameters in collaboration with a parameter server that is embedded in an unmanned aerial vehicle (UAV). Different sensors in vehicles capture different types of data, contributing to data heterogeneity. C-V2X communication networks impose additional communication overhead in order to converge to a global model when the sensor data are not independent-and-identically-distributed (non-i.i.d.). Consequently, the training time for local model updates also varies considerably. Using FRL, we accelerated this convergence by minimizing communication rounds, and we delayed it by exploring the correlation between the data captured by various vehicles in subsequent time steps. Additionally, as UAVs have limited battery power, processing of the collected information locally at the vehicles and then transmitting the model hyper-parameters to the UAVs can optimize the available power consumption pattern. The proposed FRL algorithm updates the global model through adaptive weighing of Q-values at each training round. By measuring the local gradients at the vehicle and the global gradient at the UAV, the contribution of the local models is determined. We quantify these Q-values using nonlinear mappings to reinforce positive rewards such that the contribution of local models is dynamically measured. Moreover, minimizing the number of communication rounds between the UAVs and vehicles is investigated as a viable approach for minimizing delay. A performance evaluation revealed that the FRL approach can yield up to a 40% reduction in the number of communication rounds between vehicles and UAVs when compared to gross data offloading. Full article