Generative AI for Unmanned Vehicle Swarms: Challenges, Applications and Opportunities

UAVs, also known as drones, constitute a class of aerial robots designed to operate collaboratively to achieve a diverse array of objectives. These objectives span the spectrum from military applications to an expanding suite of civilian and commercial uses. UAVs typically equipped with a multiple of rotors which are capable of VTOL [44], thus enabling them to hover, ascend, and descend vertically. The operational control of these vehicles may be manual via remote piloting or autonomous, through the integration of sophisticated onboard processing units. Other than the common applications UV swarms offers, the deployment of swarm UAVs in recreational activities has been noteworthy. One interesting application leveraging the flexible movement of UAVs is drone light shows, which involves a fleet of drones, often equipped with lights, flying in a coordinated manner to create shapes, patterns, or texts in the sky [45]. Another key application of UAV swarms is in post-disaster communication network reconstruction. Utilizing UAV’s mobility, affected areas can quickly reconnect with the outside world by employing UAVs as movable base stations [46, 47, 48, 49].

UGVs are pivotal components in UV systems, especially for transportation and logistics. Their applications in platoon formation is crucial for enhancing the efficiency and safety of transport systems. Other than classified by the degree of autonomy, UGVs can also be categorized based on their mobility mechanisms into wheeled, tracked, and legged variants [50]. Tracked UGVs excel in rugged terrains, while legged UGVs are effective in obstacle-rich environments, beneficial for tasks like search and rescue. Besides this, UGVs have been increasingly incorporated into various applications, leveraging their autonomous capabilities for various critical and practical tasks. Other than the aforementioned common applications of UV swarms, collaborations between UAVs and UGVs allow leveraging the strengths of both vehicle types to achieve common goals. For instance, while UAVs can explore areas at a high speed using their cameras, UGVs, capable of traversing a wide variety of terrains, complement UAVs by providing moving recharging stations and transportation between mission objectives, thus enhancing the operational efficiency and extending the mission capabilities [51].

USVs also known as autonomous boats, are specialized robotic platforms that operate on water surfaces to perform a variety of tasks. Their operational domain span from disaster management to environmental monitoring and defense applications. Compared to UAV and underwater vehicles, USVs have an advantage in terms of reliable access to Global Positioning System (GPS) data and superior communication capabilities, making them ideal for real-time operations and long-term missions [52]. Such an advantage has make USVs instrumental in conducting water depth surveys and examining mitigation patterns and changes within major ecosystems [53]. USVs also facilitate research activities that require heterogeneous UV cooperation, including integration with other UVs [25]. With the ability to generate power from solar, wind, and waves, USVs can serve as durable platforms for collecting extensive data across various locations and act as mobile communication and refueling relays for other vehicles [54, 26].

UUVs, also known as submersible drones, are specialized aquatic robots designed for a wide range of underwater tasks. Initially focused on military and scientific uses, their applications have expanded into civilian and recreational areas [55, 6]. UUVs are categorized into Remotely Operated Vehicles (ROVs), connected to operators via cables and used for detailed tasks like observation and maintenance, and Autonomous Underwater Vehicles (AUVs), which operate independently for various measurements, including submarine volcano monitoring and seabed surveys [56, 6]. A notable application of UUVs is in inspecting and maintaining marine structures, providing essential data for the construction industry and ensuring the integrity of underwater infrastructure, such as oil pipelines and undersea cables [57]. Additionally, UUVs are uniquely suited for extreme environments, such as ice-covered regions, nuclear facilities, and other areas where human access is dangerous or impossible, showcasing their versatility and critical role in challenging underwater operations [57, 55].

UVs are revolutionizing various industries by simplifying traditional operations and innovating new activities in air, land, sea, and underwater domains. The increasing number of UVs sets a foundation for generative AI to enhance, promising to significantly improve autonomy and coordination among swarms.

GAI represents a paradigm shift in AI technology, characterized by its ability to produce novel and meaningful content, such as text, images, audio, and 3D models. As illustrated in Fig. 3, unlike discriminative models that focus on classification or prediction, GAI models are adept at interpreting instructions and generating tangible outputs, a distinction that marks a significant leap in AI capabilities [58]. These advanced models could capture a deep understanding of data patterns and structures, enabling them to not only replicate but also innovate within the learned frameworks. This innovation is evident in GAI’s diverse applications, ranging from realistic image [59] and text creation [60] to complex 3D model generation [61].

GAI is revolutionizing traditional methods, offering transformative impacts in domains such as medical and engineering education through personalized learning support and intelligent tutoring systems [62]. Similarly, in visual content generation, GAI’s potential for causal reasoning is being explored, the ability to reason about causality is crucial for many applications, such as robotics, autonomous driving, and medical diagnosis [63, 64].

Moreover, GAI’s influence extends to business model innovation, where its applications in various industries, including software engineering, healthcare and financial services, are reshaping traditional business models [65]. This versatility of GAI not only underscores its role as a tool for creativity and innovation, but also highlights its potential to drive significant advancements across multiple sectors. In summary, the primary distinction between GAI and traditional Discriminative lies in their capabilities, i.e., GAI is designed to create and innovate, generating new content, while traditional Discriminative is more focused on data classification and extraction [66].

State estimation is critical to UVs swarm applications, especially in fields like autonomous driving and traffic estimation. State variables like position, velocity, and orientation, play a crucial role in lateral’s decision making during navigation or trajectory planning [128]. However, the stochastic nature of system measurements and robot dynamics can lead to uncertainty about the actual state. Therefore, the primary objective of state estimation is to deduce the distribution over state variables based on the available time observations [127].

As shown in Table II, the integration of GAI in state estimation for UVs offers a broad range of innovative methodologies, each tailored to specific challenges and operational contexts. For instance, in addressing the challenge of data insufficiency in traffic state estimation for UGVs, the authors in [121] utilized Graph-Embedding GANs to generate realistic traffic data for underrepresented road segments by capturing the spatial interconnections within road networks. In this proposed framework, the generator uses embedded vectors from similar road segments to simulate real traffic data. Meanwhile, the discriminator distinguishes between this synthesized and actual data and iteratively train generator to optimize both components until the generated data is statistically indistinguishable from real data. This methodology not only fills data gaps but also significantly enhances estimation accuracy, as evidenced reduction in mean absolute error compared to traditional models like Deeptrend2.0 [129]. Such an advancement in traffic state estimation underscores GAI’s potential in improving UGV navigation and decision-making in complex traffic scenarios [121].

In addition to the standard GAN, cGANs can be used to generate the corresponding estimated system state variable given the raw measurements [123]. The challenge of accurately estimating the motion of multiple UAVs in dynamic environments is addressed using a cGAN framework by employ raw measurements of sensor as conditional constrain. The authors in [124] combine individual motion predictions from a Social LSTM network [130] with global motion insights from a Siamese network [131] to achieve a comprehensive motion state prediction. This method excels in accurately forecasting UAV trajectories which is crucial for effective swarm navigation. By effectively disentangling and fusing individual and global motions, the cGAN based framework demonstrates performing well and improving the performance of multiple object tracking compared to the original Social LSTM.

Additionally, an application of VAEs in capturing temporal correlations in UAV wireless channels underscores the importance of GAI in communication systems, improving channel state estimation and signal clarity by generating realistic and diverse channel samples [125]. The exploration extends to diffusion-based score models and deep normalizing flows, employed for generating complex state variable distributions, showcasing the capability of GAI to model and estimate states in more flexible manners ranging from state variables (i.e., position, velocity, and orientation) to the intricate high-dimensional gradient of these distributions [126, 127].

The versatility of GAI in state estimation for UV swarms is evident in two aspects: ability of generating missing information through adversarial mechanisms and ability of fusing varied data sources for comprehensive state analysis. These ability enables more accurate states estimation in complex operational scenarios.

Environmental perception in the context of UVs typically refers to the ability of the vehicle to perceive and understand its surrounding environment in real-time [142]. This is a key technology for achieving autonomous navigation and completing tasks for UV swarms . Such technology often involves the use of sensors such as LiDAR, cameras, and millimeter wave radar to interact with the external environment [143]. The realm of environmental perception in UVs is markedly advanced by the varied and innovative applications of GAI, as detailed in Table III. For example, due to intrinsic constraints, such as motion blur from movement, adverse weather conditions, and varying flying altitudes, UAVs often capture low-resolution images. To address this problem, the authors in [132] introduce a framework called Latent Encoder Coupled Generative Adversarial Network (LE-GAN), designed for efficient hyper-spectral image (HSI) super-resolution. The generator in LE-GAN uses a short-term spectral-spatial relationship window mechanism to exploit the local-global features and enhance the informative band features. The discriminator adopts Wasserstein distance-based loss between the probability distributions of real and generated images. Such a framework not only improves the SR quality and robustness, but also alleviates the spectral-spatial distortions caused by mode collapse problem by learning the feature distributions of high resolution HSIs in the latent space [132].

Besides improving UVs’ accuracy by enhancing remote sensing resolution, a more common application of GAI is to generate synthetic datasets, which indicates the challenge of reduced model accuracy caused by insufficient data [138]. For instance, a framework named Trajectory GAN (TraGAN) is deployed for generating realistic lane change trajectories from highway traffic data [133]. Another GAN-based framework named DeepRoad is utilized for testing and input validation in autonomous driving systems [134] to enhance the reliability of testing by generating driving scenes in different weather conditions. VAEs are also deployed for generating more realistic and diverse crash data, addressing the limitations of traditional data augmentation methods [136]. Additionally, image translation frameworks that combine VAE and GANs are utilized for transforming simulated images into realistic synthetic ones for training and testing change detection models [135, 137], though they still require real images for reference. Moreover, the authors in [139] introduce a method leveraging a text-to-image diffusion model to generate realistic and diverse images of UAVs set against various backgrounds and poses. With more than 20,000 synthetic images generated by merging background descriptions and binary masks based on ground truth bounding boxes, the detector’s average precision on real-world data increased 12%.

Another filed that GAI is utilized in is the scene understanding or captioning. Such a method includes using CLIP Prefix for image captioning, translating visual content of UAV-captured images into accurate text descriptions for decision making in UV [140]. Another method is deploying the Generative Knowledge-Supported Transformer (GKST), which enhances feature representation and retrieval performance by fusing image information from different view of vehicles. [141]. An interesting aspect of these technologies is their ability to process and interpret complex visual inputs, providing a level of contextual understanding that closely resembles human perception. This capability is particularly beneficial in dynamic environments, where rapid and accurate interpretation of visual data is crucial for effective decision-making.

In summary, the generative capabilities of GAI prove to be invaluable in the field of environmental perception for UVs. From enhancing image resolution to generating synthetic dataset, creating diverse testing environments, and advancing scene understanding, GAI stands as a cornerstone technology driving the evolution and efficiency of UVs in comprehending and interacting with their surroundings.

Autonomy refers to the capability of a system to perform a task or decision-making without human interventions [152]. The level of autonomy represents the ability of an UV can operate independently while solely relying on its onboard sensors, algorithms, and computational resources. In UV swarms, the level of autonomy depends on various factors such as the type and complexity of the task, the ability to plan and execute route [153]. Table IV illustrates how the integration of GAI is pivotal in advancing these autonomous capabilities.

In the realm of UV swarm cooperative strategies, applications of GAI are exemplified by the integration of Generative Adversarial Imitation Learning (GAIL) with multi-agent DRL. For instance, the authors in [144] introduce a Multi-Agent PPO-based Generative Adversarial Imitation Learning (MAPPO-GAIL) algorithm which employs multi-agent proximal policy optimization to sample trajectories concurrently, refining policy and value models. This algorithm incorporates grid probability for environmental target representation, increasing average target discovery probability by 73.33% while only compromising 1.11% average damage probability compared to traditional DRL search algorithms. Additionally, GAIL is utilized for training UAVs in virtual environments for navigation tasks, enabling adaptation to complex and dynamic scenes [146].

Furthermore, a VAE based model named the BézierVAE is proposed for modeling vehicle trajectories, particularly for safety validation. BézierVAE encodes trajectories into a latent space and decodes them using Bézier curves, thereby generating diverse trajectories. BézierVAE demonstrates a remarkable reduction in reconstruction errors of 91.3% and unsmoothness by 83.4% compared to traditional model TrajVAE [133], significantly enhancing the safety validation of automated vehicles [147]. For autonomous robot scheduling, COIL employs VAEs to generate optimized timing schedules, improving operational efficiency significantly [148]. Lastly, in multi-agent trajectory prediction, the GRIN model which is inspired by conditional VAE is adopted to forecast agent trajectories considering the complexities of intentions and social relations. Although complex systems face challenges such as adhering to contextual rules such as physical laws, challenges could be addressed by using a specific decoder or a surrogate model to approximate these limitations [149].

In route planning for UVs, transformer architecture combined with DRL is deployed to optimize routing for multiple cooperative UAVs. This method offers superior performance and efficient parallel processing, consistently achieving high rewards compared to traditional algorithms [150].

Enhancing autonomy in UVs is crucial for their independent and cooperative swarm operation. GAI’s generative capabilities are applied in multiple aspects, from generating new trajectories to refining routing strategies and imitating expert agent routing behaviors in diverse scenarios. These diverse applications demonstrate dynamic and adaptable solutions crucial for UVs to efficiently and independently navigate and operate in complex and changing environments.

In the field of task and resource allocation for multi-agent UV swarms, GAI introduces effective approaches that enhance efficiency and adaptability of these systems. Traditional methods often rely on fixed algorithms and heuristic approaches, which may not always be sufficient for dynamic and complex environments [159]. As illustrated in Table V, GAI offers the flexibility necessary for these challenging scenarios.

A GAIL-based algorithm is proposed to reconstruct virtual environments for DRL, where the generator produces expert trajectories and the discriminator distinguishes them from generated trajectories [154]. This approach can make a virtual edge computing environment that closely mimics real-world conditions. It provides a place for computing resource allocation multi-agent DRL method to explore and infer reward function while avoiding impairing user experiences caused by arbitrary exploration. Furthermore, an autoencoder based method is applied to the Hungarian algorithm to mitigate information ambiguity issues caused by the same weight that appears in the data rate matrix, particularly in the allocation of bandwidth and power resources between cellular users (CU) and device-to-device users (D2DU) [155]. This method provides an optimal reconstructed cost matrix using latent space as a hyperparameter to assist in resource allocation decisions.

Additionally, the authors in [156] proposed a diffusion model-based AI-generated optimal decision (AGOD) algorithm. This algorithm enables adaptive and responsive task allocation based on real-time environmental changes and user demands. The algorithm’s efficacy is further enhanced by the integration of DRL, as demonstrated in the Deep Diffusion Soft Actor-Critic (D2SAC) algorithm. Compared to traditional SAC methods, the D2SAC algorithm shows a performance improvement of approximately 2.3% in terms of the task completion rate and 5.15% in terms of utility gained [156]. Unlike traditional task allocation methods, which assume that all tasks and their corresponding utility values are known in advance, D2SAC could address the selection of the most appropriate service provider, where tasks arrive dynamically and in real time. D2SAC shows a notable performance improvement in terms of completion rate and utility gained compared to traditional methods.

In the realm of joint computing and communication resource allocation, the importance of effective management is accentuated in UVs due to their standalone nature and battery constraints. A diffusion-based model presented in [157] offers an advanced method to design optimal energy allocation strategies for the transmission of semantic information. A key strength of this model is its ability to iteratively refine power allocation, ensuring transmission quality is optimized under varying conditions caused by the dynamic environment of UV swarm. With the transmission distance at 20 m and the transmission power at 4 kW, this diffusion model-based AI-generated scheme surpasses other traditional transmission power allocation methods like average allocation (named Avg-SemCom) and Confidence-based Semantic Communication (Conf-SemCom) [157] at approximately 500 iterations with 0.25 increase on transmission quality.

On the other hand, the authors in [158] proposed the incorporation of LLM explored to elevate the capabilities of GAI in task and resource allocation within multi-agent UV swarms. Using LLM’s advanced decision-making and analysis ability, independent LLM instances for each user are created to breakdown the original intent “reducing network energy consumption by $\Delta p=0.85W$” into a series of detail tasks such as tuning transmit power and channel measurement. The results are then prompted to the LLM, which will add subsequent tasks and instruct related executors to take actions. With the integration on LLM, the UAV agents managed to achieve the power saving target in 2 rounds. Although further simulation results show that current GPT-4 experiences some difficulties in maintaining multiple goals when the number of agents increases. This integration signifies a significantly advancement in the autonomy and functionality of UV swarms.

In conclusion, GAI substantially advances the field of task and resource allocation in multi-agent UV swarms. From creating vivid simulation environments for the allocation algorithm to explore on, to iteratively adjust the allocation strategy and break a rough intent to detail tasks, GAI demonstrates a strong ability to handle dynamic environments and various challenges.

As mentioned in Section II, a key application of UVs is their role as mobile base stations for the reconstruction of the communication network [46, 47, 48, 49, 164]. An effective positioning strategy is crucial in this context to ensure seamless access by achieving maximum user coverage with a limited number of UVs. Additionally, when UV swarms are deployed in hierarchical structures, with lead UVs acting as command centers, ensuring effective communication coverage among the sub-UVs is critical for task distribution and collaborations. As illustrated in Table VI, the need for efficient network coverage and vehicle-to-vehicle (V2V) communication is addressed by varieties of GAIs.

While utilizing UAVs as mobile stations to provide temporary network links in dynamic wireless communications is becoming increasingly popular, optimizing the network can be complex due to factors such as varying UAV altitudes, mobility patterns, spatial-domain interference distribution and external environmental conditions, which present unique challenges. In addressing the optimization of network coverage with limited UAVs, the authors in [160] propose the use of cGAN. This framework comprises a generator for modeling and predicting optimal network configurations, a discriminator for evaluating the efficiency of these configurations against real-world scenarios, and an encoding mechanism for adaptability and scalability. The method based on cGAN not only guarantees the best positioning of UAVs but also simplifies computational complexity, achieving $\mathcal{O}(k^{2})$. In contrast, traditional methods like the core-set algorithm [165] and the spiral algorithm [166] have the time complexities of $\mathcal{O}(pk)$ and $\mathcal{O}(k^{3})$, respectively where $p$ represents the number of user equipment in the area, while $k$ denotes the number of UAVs in the fleet [160]. Another solution proposed by the authors in [163] utilized the self-attention-based transformer to predict the user mobility and enhance the aerial base station placement. The transformer model was able to capture the spatio-temporal dependencies and handle long input and output sequences. The transformer-based scheme achieved significant gains in coverage rates compared to regular deployment schemes by improving the coverage rate by more than 31% over the regular scheme [167] and by more than 9% over the LSTM-based scheme.

In the domain of V2V communication, which is essential for secure navigation in the UV swarm, vehicles often relay images to communicate environmental data. However, these images can be corrupted due to transmission disruptions, environmental noise and noise caused by vehicle movement. To address this, the authors in [162] integrates GDMs for image restoration and network optimization. GDMs enable vehicles to restore transmitted images to their original quality by reducing data transfer and delay in communication. The iterative nature of GDMs, based on stochastic differential equations, is adept at refining internet-of-vehicles network solutions, notably in areas like path planning. For example, GDMs initiate optimization with a preliminary path, progressively enhancing it based on key performance indicators. The process capitalizes on these metric gradients to guide the path modifications towards an optimal solution. Compared to traditional DQN methods [168], the proposed GDM-based method achieved a 100% increase in average cumulative rewards at 300 epochs [162].

In summary, for network coverage and accessibility, GAI can either generate positioning strategies directly or act as an encoder to enhance traditional algorithms by capturing spatial information. For efficiency, GAI acts as a framework that uses semantic information to reduce data transmission while maintaining communication by guided generation. However, while these developments represent a leap forward in managing UV swarms, there remain areas for further exploration. For example, the authors in [162] open the question of integrating other modalities for more efficient communication. This suggests an opportunity for future research to investigate the incorporation of multimodal data processing in UV networks. Such explorations could significantly enhance the adaptability of these technologies to diverse network topology and environmental conditions. Additionally, the potential of GAI to facilitate autonomous decision-making within UV swarm deployments presents a promising avenue for advancing the field. By expanding the scope of GAI applications, researchers can further optimize UVs for a variety of complex real-world scenarios.

Security and privacy are critical aspects in UV swarms, especially in military and surveillance applications. The integration of GAI in these domains offers innovative solutions for enhancing system security and ensuring privacy. As illustrated in Fig. 6, an interesting potential application is to utilize GAI’s ability to generate fake data or simulate communication activities to act as a honeypot to mislead potential attackers and reinforce system security [176]. The LLM-generated honeypots serve as an additional protective layer, disseminating false information to confuse and trap attackers, thereby enhancing the collective security of the swarm. This innovative use of language processing technology within the swarm network exemplifies a new frontier in safeguarding autonomous vehicles from sophisticated cyber threats. The use of GAI in UV swarm security and privacy protection is elaborated in Table VII.

One notable application of GAI in the realm of privacy protection is the Auto-Driving GAN (ADGAN) [169]. ADGAN is a GAN-based image-to-image translation method designed to protect the privacy of vehicle camera location data. ADGAN achieves this by removing or modifying background buildings in images while retaining the utility of recognizing other objects like traffic signs and pedestrians. The semantic communication acts as an effective means to enhancing the security of UV swarms, as it removes the background images that are lrrelevant to tasks. Additionally, ADGAN introduces the multi-discriminator setting that enhances image synthesis performance and offers a stronger privacy protection guarantee against more powerful attackers [169]. Another similar application is a GAN-based framework that protects identity privacy in street-view images by altering recognizable features, such as replacing moving objects with a realistic background [172].

In terms of trajectory data privacy, TrajGANs are employed to protect the privacy of trajectory data by generating synthetic trajectories [170]. These trajectories follow the same distribution as real data while obscuring individual locations and identities of users. They preserve real data’s statistical properties and capture human mobility patterns. However, TrajGANs may face challenges in creating dense representations of trajectories, particularly for timestamps and road segments, and may fail to identify some rare or exceptional events in the data. To further enhance the protection, the authors in [171] present the LSTM-TrajGAN framework. The framework consists of three parts: a generator that generates and predicts realistic trajectory configurations, a discriminator that compares these configurations with real data to validate their authenticity and utility, and a specialized encoding mechanism that utilized LSTM [177] recurrent neural network to perform space-time embedding of trajectory data and its respective time stamp. Its privacy protection efficacy is evaluated using a Trajectory-User Linking (TUL) algorithm as attackers [178]. Evaluated on a real-world semantic trajectory dataset, the proposed approach achieves better privacy protection by reducing the attacker’s accuracy from 99.8% to 45.9% compared to traditional geo-masking methods like random perturbation at 66.8% and Gaussian geo-masking at 48.6% [179]. These results show that the LSTM-TrajGAN can better prevent users from being re-identified while preserving essential spatial and temporal characteristics of the real trajectory data.

VAEs are also deployed for protecting UV trajectory privacy. The authors in [173] utilize a VAE to create synthetic vehicle trajectories, ensuring differential privacy by adding noise to data. This approach helps to effectively obscure vehicle locations, although some data distortions may be introduced due to the added noise. Transformers in federated learning, as discussed in [174], enhance privacy in autonomous driving by sharing only essential data features across networks. This method improves privacy but faces challenges with communication link stability and external interference.

To protect vehicle network security, the authors in [175] proposed a transformer-based intrusion detection system that provides a sophisticated solution for vehicle networks. This system employs self-attention mechanisms to analyze Controller Area Network (CAN) messages, accurately classifying them into various in-vehicle attacks like denial-of-service, spoofing, and replay attacks. Another transformer-based model proposed by the authors in [174] is the integration of transformers in federated learning setups. This method enables the sharing of key data features rather than raw data across a network of autonomous vehicles. This method significantly boosts privacy by minimizing the exposure of sensitive data while still enabling collaborative decision-making and computing.

In summary, the application of GAI in UV swarms has revolutionized security and privacy measures, particularly in sensitive sectors such as military and surveillance. Techniques like honeypots and GAN-based frameworks demonstrate GAI’s capability in data manipulation for enhanced security. Additionally, the implementation of VAEs and transformers in federated learning for trajectory privacy, and advanced intrusion detection systems underscore GAI’s adaptability and effectiveness in safeguarding against sophisticated cyber threats.

Vehicle safety is another critical concern that encompasses the detection, isolation, and resolution of system faults. Unlike other safety concerns such as collision avoidance or the development of safe path planning strategies for UV swarms, which are more closely related to the level of autonomy of these systems [184], research on UV safety highlights the unique challenges brought forth by internal vulnerabilities of UV systems, including algorithmic and hardware failures. Research in this field aims to enhance the overall reliability and safety of UV operations by developing methodologies and technologies that enable these systems to effectively identify and rectify potential faults before they impact vehicle performance or safety.

Monitoring operational parameters for fault detection in UV systems is essential to ensure their safety and efficiency. A novel framework has been proposed that uses LSTM networks combined with autoencoders, enabling continuous learning from vehicle performance data [181]. This framework enhances the system’s ability to pinpoint and address faults progressively. The ability of LSTM in handling time-series data makes this approach particularly effective in dynamic environments where various factors can influence vehicle performance.The LSTM autoencoder can generate synthetic data points that represent potential fault scenarios. This enhances the training dataset, allowing the model to learn from a wider range of conditions and achieve 90% accuracy in detecting and 99% accuracy in classifying different types of drone misoperations based on simulation data. This significantly improves the safety and operational efficiency of UV systems. In subsequent developments [182], the advances in UAV fault detection and classification, particularly the four-fold increase in speed through FPGA-based hardware acceleration while half the energy consumption. This research further identifies a key consideration for GAI, which shows that model computation can be optimized for real-time operations. The successful deployment in UAV swarms also suggests that similar strategies could enhance GAI performance in dynamic environments and complex task coordination.

On the other hand, VAE presents a sophisticated approach to fault and anomaly detection in UV swarms. The authors in [180] proposed a new method by training VAE on data that represents the UVs’ normal operations. This method helps VAEs develop an understanding of what constitutes standard performance. The learning process involves the reconstruction of input data, where the model’s ability to accurately replicate the original data serves as the basis for identifying operational consistency. A significant deviation in the reconstruction error from the norm signals a potential fault or anomaly. By generating reconstructions of the input data and calculating the resulting errors, the VAE-based method achieved an average accuracy of 95.6% in detecting faults and anomalies [180]. The advantage of utilizing the VAEs ability of mapping relations is their proficiency in uncovering novel faults or issues that were not present or accounted for in the training dataset. This feature ensures that VAE-based systems can maintain high levels of safety and reliability in diverse and unpredictable scenarios. This feature is invaluable in the context of UV operations, which often encounter a broad spectrum of environmental conditions and operational challenges. Nonetheless, it is critical to acknowledge that the performance of VAEs can be affected by various factors that include the complexity of the VAE model itself, the quality and diversity of the data used for training, and the specific threshold set for flagging reconstruction errors as potential faults.

Furthermore, the authors in [183] utilize a spatio-temporal Transformer network for battery fault diagnosis and failure prognosis in electric vehicles due to its specialized architecture, which performs well at extracting key features across multiple spatial and temporal scales. The adoption of a spatio-temporal Transformer network for battery fault diagnosis and failure prognosis in vehicles excels in identifying early warning signals and predicting faults over varying spatial and temporal scales. Its ability to analyze and predict the evolution of battery faults using onboard sensor data aligns perfectly with the needs of UV, which are heavily reliant on battery integrity for their operation. By integrating such a model, predictive maintenance strategies are greatly enhanced, allowing early detection of anomalies and prediction of battery failures within a precise window ranging from 24 hours to a week. This approach not only enhances operational efficiency by optimizing vehicle schedules to reduce downtime but also plays a crucial role in safeguarding against potential battery failures that could compromise vehicle safety.

In UV operations, ensuring safety and reliability not only involves detecting faults, but also isolating affected components to prevent further issues, and implementing targeted solutions for resolution. For instance, in the case of relatively minor issues such as the loss of information due to sensor malfunctions, the utilization of VAE and GAN illustrates the innovative application of GAI in fault management [185]. Through optimization of the VAE-CGAN structure, these models can regenerate missing time series data, demonstrating their effectiveness in scenarios where operational faults compromise data integrity. This function is especially beneficial for applications like UAV-based agricultural surveillance, where the continuity of data collection is paramount.

In tackling severe issues that compromise UV swarm operations, an intriguing aspect of current research is the development of strategies for “where to crash” decision protocols that stand out [186]. This concept addresses the need for predefined protocols on how and where a UV should terminate its operation in the event of a critical failure to minimize secondary hazards. These protocols range from emergency landing zones for UAVs to specific sinking points for USVs and UUVs, and controlled stop measures for UGVs. However, these predefined protocols may not be able to accommodate all possible scenarios. Thus, integrating GAI into UV swarm fault management strategies offers an advanced method to enhance safety. For example, by analyzing real-time sensor data and understanding the intricacies of swarm dynamics, Transformers are able to make context-aware decisions to identify the safest termination points for compromised UVs accurately [187]. Incorporating such GAI may not only improve the management of critical failures, but also lower the risk of secondary incidents.

In the future, there can be a large number of UVs in a swarm to perform complex tasks in challenging environments such as precision agriculture, environmental monitoring, military operations, and delivery services. This introduces several issues to the development of GAI in UV swarms. Specifically, as the number of UVs increases, coordinating movements and communications between them become much more complex due to various factors like congestion, latency, signal interference, and limited communication range. This demands for novel GAI approaches that can quickly determine optimal movements and actions for each UV in such complex situations. One potential direction is to design distributed GAI architectures based on federated learning for collaborations between different groups of unmanned swarms, instead of relying on a single server/swarm leader. By doing this, the computing load can be distributed among swarm leaders, resulting in a more efficient learning and collaborating process between different groups of unmanned swarms, especially in large-scale swarm settings.

In UV swarms, system conditions are highly dynamic and uncertain due to the mobility of UVs as well as surrounding complex environments. Although GAI has abilities to deal with these uncertainties, it is essential to develop adaptive GAI approaches to further reduce the system latency when retraining GAI models under new environmental conditions, especially in large-scale unmanned swarms. Integrating GAI with recent advances in AI such as transfer learning and meta-learning is a potential solution. In particular, transfer learning aims to transfer knowledge learned in a source environment to facilitate the learning process of similar tasks in new environments while meta-learning aims to learn how to learn to speed up the learning process in new conditions. These techniques can help AI models achieve good training accuracy with a few training samples from the new environments [188, 189]. In addition, advanced deep reinforcement learning can also be integrated into GAI models for real-time learning and feedback on the behaviors of UV swarms.

In large-scale and heterogeneous UV networks, the energy efficiency, latency and security issues are the most critical for coordination locally and globally. In this operational circumstance, semantic communication (SemCom) with knowledge base (KB) shared among UVs is a viable solution to minimize the transmission overhead by eliminating the task-irrelevant information while performing a given task reliably, which greatly improves the demanding energy efficiency, latency and security. The UVs are limited in both the resource and hardware, and hence it is challenging to realize AI-native UV networks that enable the SemCom with shared KB. It should be noted that the on-going connections among internet of things (IoT), namely UVs are inherently task-oriented, dynamic and short-term due to their mobility, and achieving the above goal via the SemCom in such a circumstance is of paramount importance. The GAI approaches herein are considered to be effective in dealing with the dynamic and complex environments of such large-scale and heterogeneous UV networks, which are worthy of further research towards the AI-native UV networks.

In large-scale and heterogeneous UV networks with high mobility, the resource-constrained multiple UV connections face with dynamic interference patterns which are distributed across 3-D space and temporal domain, and therefore a sophisticated interference control should be exercised for coordinating and maintaining communication among diverse UVs with limited power usage. Especially, the high mobility renders this interference control challenging in such complex environments, where the GAI approaches are proven to be effective in regulating the dynamic interference fluctuations within 3-D coverage.

As reviewed above, GAI can help to improve the security and privacy of UVs by generating synthetic data to deceive attackers. Nevertheless, GAI, and AI in general, are vulnerable to adversarial attacks in which attackers try to disrupt the training process of AI models by poisoning training data, evading trained models, and extracting model information. More dangerously, attackers can also leverage GAI to generate “fake” data that is hard to distinguish. In addition, in UV swarms with resource-constrained vehicles, e.g., UAVs, individual vehicles are extremely vulnerable to adversarial attacks, and cyber attacks in general, due to the lack of computing and energy resources to perform sophisticated and efficient countermeasures. Dealing with adversarial attacks in GAI is still a new research problem and there are limited efforts in the literature. As such, there is an urgent demand for lightweight and effective solutions to improve the security of UV swarms. One potential solution is to leverage GAI to recover poisoned training data. Integrating deep reinforcement learning with human feedback can also be useful in defeating adversarial attacks. Finally, federated learning can also be adopted to enhance the privacy of unmanned swarms since data of vehicles (e.g., their sensing data and locations) will not be shared with others.