Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learning

AML is concerned with identifying vulnerabilities and mitigations for machine learning algorithms. Execution-time AML attacks against supervised learning have largely focused on Evasion attack [124], which occur when slight human-imperceptible changes to inputs return significantly different outputs from neural networks [123]. AML defences for evasion attacks in supervised learning have been broader and include adversarial training [111], regularisation [137], adversarial detection [137], data preprocessing [115], and ensembles [124].

Evasion attacks may be targeted or untargeted. Untargeted evasion attacks, such as Fast Gradient Sign Method (FGSM) [35] and Projected Gradient Descent (PGD) [81], aim to cause the victim to predict a different class. Targeted evasion attacks, such as Jacobian Saliency Map (JSM) [96] and the Carlini and Wagner method (C&W) [14], aim to cause the victim to output a specific class. Targeted attacks can also be created using variations of untargeted attacks, for example, the One-step target class methods [68] alter the loss function in the FGSM attack.

Evasion attacks also require a certain amount of knowledge about a victim. Attacks that require full knowledge of a victim are known as White-box attacks. Black-box attacks only require the ability to query the victim. Transfer attacks [95] allow attacks that originally required white-box information to instead only require black-box information. Transfer attacks use black-box information to train a surrogate victim that replicates the behaviour of the original victim, and then use white-box techniques against the surrogate to discover attacks that are also effective against the original victim. There are also grey-box attacks, which only require partial knowledge of the victim [124].

To counter evasion attacks, defensive techniques are used to mitigate the vulnerability. Adversarial training [35] mitigates AML attacks by training a victim against both the original data and adversarially-perturbed data. However, adversarial training may fail to defend against other attacks that it was not trained against [111]. Regularisation [137] alters the training process of the algorithm to improve its robustness against AML attacks. Regularisation may include using additional terms, such as the Lipschitz constant [36], which aims to prevent sudden changes in the output caused by slight perturbations to the input. Preprocessing input data has been shown as an effective AML defence [115] and includes techniques such as autoencoders [29]. Input data can also be altered to remove adversarial perturbations [137], if they can first be detected, thus adversarial detection [137] is also a key AML defence. Ensembles [124] are also a type of AML defence, which trains multiple algorithms which collectively decide the output.

DRL uses neural networks for reinforcement learning [1]. Reinforcement learning is a type of machine learning that is concerned with sequential decision-making, which may be modelled with frameworks such as a Markov Decision Process (MDP) (Definition 1). The goal of reinforcement learning is to learn a policy that selects actions that maximise the expected total reward.

DRL techniques such as Deep Q-Networks (DQNs) [84] use a neural network to learn a policy, which maps states to actions. DQNs use an \(\epsilon\)-greedy policy, in which the agent will explore by randomly selecting an action with probability \(\epsilon\), otherwise it will select the action with the greatest Q-Value. A Q-Value is the expected total reward of a state-action pair, and incorporates both the immediate reward of an action and expected future rewards.

DRL has also been shown to be effective in partially observable environments through the use of recurrent networks. Partially-Observable MDPs (POMDPs) (Definition 2) are a common framework for representing partially-observable decision-making problems. An example of a real-world POMDP could be the treatment of a medical condition, in which the underlying cause is unknown and only the symptoms of the condition may be observed. To provide a concrete context, we will use this example throughout the paper, as it illustrates the key concepts and methodologies in a practical scenario, helping to bridge the gap between theoretical discussions and real-world applications. POMDPs separate the observation from the state and require a decision-making agent to construct a belief about the true state of the environment from a series of partial-observations. Deep Recurrent Q-Networks (DRQNs) [47] are able to construct a belief about the state in the form of a latent state representation from observations through the use of a Long Short Term Memory (LSTM) network.

MARL extends DRL to a multi-agent context. Multi-agent environments can be cooperative, competitive, or mixed based on the rewards provided to the agents [12]. Cooperative environments feature monotonic rewards, in which an increase in the reward of a single agent corresponds to an increase in the total reward of all agents. Decentralised POMDPs (Dec-POMDPs) [91] (Definition 3) are a common framework for representing cooperative multi-agent decision-making problems. The extension of a POMDP to a Dec-POMDP allows us to expand our medical treatment example from above to include multiple specialists treating a patient, where each specialist may have access to different skills but must cooperate to achieve the best outcome for the patient. Dec-POMDPs separate the single observation and action of a POMDP into a set of observations and set of actions, which are the observations and actions for each agent, respectively. Dec-POMDPs assume a single shared reward, and so are only suitable for fully cooperative problems. Competitive environments include but are not limited to zero-sum rewards, where an increase in one agent’s reward directly reduces another agent’s reward. Mixed environments may feature a combination of competitive and cooperative elements in their reward structure. An example of a mixed environment features two competing teams, with agents on the same team receiving the same reward and agents on opposing teams competing for a zero-sum reward. A Partially-Observable Stochastic Game (POSG) [45] (Definition 4) is a framework that is better suited to competitive and mixed multi-agent decision-making problems. Games, including POSGs, sometimes use varied terminology for agents, including players and controllers, however, we will use the term agent henceforth. A POSG allows us to expand our medical treatment example to include agents whose primary concern may not be the well-being of the patient but instead be for a different motivation such as profit. A POSG provides agents individual reward functions.

A unique feature of MARL is emergent communication [79], which occurs when agents learn to communicate. Communication is a common feature in cooperative partially-observable environments, because agents benefit from coordinating their actions and sharing information. There are two broad categories of communication, implicit communication and explicit communication. Implicit communication occurs when agents are able to view the state change caused by the actions of other agents [42]. For example, an agent may observe another agent moving toward a particular location and may be able to infer that the goal is at that location. Explicit communication occurs when an environment features a communication channel that allows agents to directly send messages to other agents [27]. Broadcasting is a common paradigm for explicit communication, in which agents produce a single message that is sent to all other agents in the system. Creating the message to send and interpreting messages from other agents are learnt behaviours of the agents.

The combination of AML and MARL has seen little exploration in previous surveys that have covered AML for DRL [2, 18, 51]. Ilahi et al. [51] and Chen et al. [18] cover AML attacks against DRL and defences against those attacks, which includes some AML attacks against MARL. However, due to the focus of these surveys on DRL, they do not explore larger implications of combining AML and MARL. More broadly, there have been surveys such as Bai et al. [2] that focuses on the AML defence of adversarial training, and that by Ren et al. [111], that covers AML attacks and defences for deep learning. There has been no coverage of AML defences for MARL in any previous surveys to the best of our knowledge. Our work seeks to address the significant gap in coverage of AML attacks and defences for MARL.

For a broad overview of AML attacks against MARL, MAL, and DRL, our research questions are:

We used two search strings to find papers to address our research questions which were:

We limited our search to high-quality conferences that published work in AML, MARL, MAL, and DRL since 2010. Our search strings were used in Scopus and external proceedings servers where necessary to search the following conferences¹:

Finally, we used several well known papers that address our research questions [8, 120, 131] and used a forward snowball search method [139] to find additional relevant papers. These papers were chosen because they made significant discoveries that are relevant to our search.

From our initial collection, we filtered out relevant papers using accept and reject criteria. For a paper to be excluded, it must either fail to meet all the accept criteria or meet any of the reject criteria. Our accept criteria were:

Our reject criteria were:

We define adversarial attacks as the intentional manipulation of an aspect of an environment, including actuators, sensors, and communication channels, to reduce the performance of a target agent by causing that agent to take irrational or self-defeating actions. We further limit the scope of our study to execution-time attacks. This definition excludes a number of AML attacks, including, Trojan implantation, data poisoning, and model inference attacks.

Action Perturbations are a form of AML attack unique to decision-making problems in which the adversary directly alters the action of an agent. An adversary exploits this attack vector by perturbing the action to cause an unexpected state transition, from which the target agent is unable to optimally act. Thus, by making minor changes to the agent’s actions, an adversary can significantly degrade an agent’s performance. An Action Perturbation attack is difficult for an adversary to achieve because it requires an adversary to directly affect an actuator either with physical access or a cyber-physical attack. We have identified eleven attacks that perform Action Perturbation that we present in Table 2 along with other key properties, including the information required for the attack, the attack objective, the framework used to originally construct the attack, and the type of action space under attack (discrete or continuous).

MDPs were used as the framework for all the attacks except for those by Tessler et al. [127], which used Probabilistic Action Robust MDP (PR-MDP) and Noisy Action Robust MDP (NR-MDP). The PR-MDP [127] (Definition 5) adds an adversary to the standard MDP that probabilistically replaces the victim agent’s action. The NR-MDP [127] (see Definition 6) adds an adversary to the standard MDP that is able to add a perturbation to the victim agent’s action at every timestep. In our medical treatment example, the PR-MDP would represent an adversary that may be able to mix a different medication with the prescribed medication, whereas the NR-MDP would represent an adversary that could slightly increase or decrease the amount of medication the patient is taking. The PR-MDP and NR-MDP are effective frameworks for modelling their respective attacks, namely, Probabilistic Action Perturbation [127] and Noisy Action Perturbation [127]. However, these frameworks are only suitable for modelling the decision-making of a single agent and the lack of a general framework for Action Perturbation attacks limits the ability to compare the effectiveness of the attacks.

A key feature of Action Perturbation attacks is the type of action being perturbed, namely, continuous or discrete. Continuous Action Perturbation attacks [21, 40, 71, 72, 127, 138] reveal vulnerabilities caused by uncertain actuation, such as imperfections in a motor. This actuation uncertainty is common to the robotics domain applications, thus the simulation environment MuJoCo [129] was commonly used to demonstrate the applicability of the attack. Directly replacing an agent’s action can be considered as a type of Action Perturbation with infinite magnitude. Replacement is necessary for attacks against discrete action spaces [21, 38, 40, 74, 153], and has also been demonstrated against continuous action spaces by the Probabilistic Action Perturbation attack [127].

Our survey found three Action Perturbation attacks against MARL [38, 74, 153]. The MARLSafe Action test [38] and the Evolutionary Generation of Attackers [153] attacks create a traitor agent, whose actions are replaced for an entire episode. MARLSafe Action test [38] trains an adversarial agent using deep learning [34] to select the worst-case action, whereas the Evolutionary Generation of Attackers [153] uses an evolutionary algorithm. Both of these attacks demonstrate their effectiveness on the StarCraft Multi-Agent Challenge (SMAC) [114] and measure the degradation of the reward and win-rate as their attack success metrics. The 1-step adversarial power [74] attack, instead considers the effect of replacing an action at a single timestep. The effectiveness of this attack is demonstrated in the two-player fully-cooperative Overcooked game and is measured using the degradation of reward.

The effectiveness of an Action Perturbation attack is dependent on its effects on the target. Most of the Action Perturbation attacks used a reward-based target, due to the immediate reduction in the optimal expected return of an agent. However, by causing the environment to transition to an unexpected state, an Action Perturbation attack may have a secondary effect on the victim agent. The vulnerability of algorithms to transitions to unknown states has explicitly been explored in both discrete [40] and continuous action spaces [127]. The metrics of average episodic reward and win rate that are currently used to evaluate the effectiveness of Action Perturbation attacks fail to differentiate between these two effects. A metric, such as counterfactual regret [159], may better evaluate the vulnerability of an agent to Action Perturbation attacks. The regret of an agent is the difference between the expected cumulative reward given an agent’s policy and the best possible cumulative reward. The regret at the point of attack would indicate the level of vulnerability an algorithm has to the worst-case action, and the regret in the steps following an attack would indicate the vulnerability of the agent to the secondary effects of the attack.

Observation Perturbation attacks, similar to Evasion attacks [124] in supervised learning, expose a vulnerability in an agent to adversarially altered inputs. For an adversary to alter the observation, the adversary must be capable of altering the environment [9] or an agent’s sensors, via means such as a cyberattack [26]. The effect of an Observation Perturbation attack is to cause the agent to select a different action. Thus, Observation Perturbations indirectly lead to an Action Perturbation attack. We present many of these Observation Perturbation attacks in Table 3. This table covers the method used to craft the perturbation, if a transfer attack [95] was demonstrated, the method used to determine the timing of the attack, and the framework used by the paper to present the attack.

Observation Perturbation attacks require carefully crafted alterations to induce an adversarial effect. We present a number of perturbation crafting techniques in Table 4. Many of these perturbation crafting techniques were originally used against supervised learning algorithms [14, 35, 81, 96]. The first major category is white-box untargeted attack such as the FGSM [35], PGD [81], and UAP [86]. These attacks can be extended to target specific objectives by altering the loss function as has been demonstrated by it-FGSM [76], Two-Stage PGD [103], GM PGD [19], Directed FGSM [122], and RT-FGSM [134]. Targeted white-box attacks include JSM [96] and C&W [14] which are used to achieve action-based objectives. These white-box perturbation crafting methods can be extended to use black-box information through the use of a transfer attack [95]. A transfer attack uses black-box information to train a surrogate victim, then uses white-box information from the surrogate to craft perturbations, and then uses those perturbations against the original victim. Transfer attacks have been demonstrated against a number of DRL attacks [4, 17, 38, 49, 50, 52, 134, 135]. Alternatively, there are untargeted black-box attacks such as FD [7] and AdvGAN [143]. Some crafting methods were developed specifically for DRL, such as CopyCAT [50], which crafts observation-independent perturbations that cause the victim to follow a specific policy. Chan et al. [17] used a SRIM to generate an attack by evaluating the impact that perturbations of different features had on the cumulative reward of an agent. Alternatively, DRL enables the adversary to learn how to craft perturbations. This approach has been extended to use white-box information from the victim [112]. Other methods use the gradient of the Q-Value [97, 155] to craft reward-based perturbations.

Many of the Observation Perturbation attacks targeted unstructured observations, such as images. However, Wang et al. present an attack that targets the structured observations of an energy management system of an electric vehicle [135]. The data collected from the vehicle was presented to the agent as a vector of continuous values, which was smaller and contained a greater density of information than images. The empirical results showed the effectiveness of the attack, which suggests that attacks against other structured observations, such as those used in autonomous cyber defence [117], may also be effective. However, the observations in an autonomous cyber defence environment may be discrete, and it is unclear how effective these attacks would be against structured discrete observation spaces.

The timing of an attack has been shown to be a key factor in the success of that attack [49, 67, 77, 104, 120]. The simplest timing is the uniform attack [49], which attacks at every time step. However, frequent attacks are computationally expensive and easier to detect [77]. More efficient attacks wait until the value of a state or action is above a certain threshold. TF [104] selects a state to be attacked if the difference between the victim agent’s Q-value estimation for its most preferred action and an adversarially selected action exceeds a set threshold. This technique only works for Q-value-based DRL algorithms and would not be effective against policy-based algorithms. Critical State [67] and CBTS [157] both consider an agent’s preference for taking an action as the means to decide if a state is worth attacking. Other approaches consider the value of the state as the means to decide if it should be attacked. VF [65] selects a state if the estimation of the value of the current state is above a certain threshold. Critical Point [120] instead attacks a state if the value of the current state and the value of a future state caused by an attack exceeds a certain threshold. DRL has been used as an attack timing method by training an agent to select if the current state should be attacked [120]. A direct comparison of all of these methods has yet to be made. Instead, we consider the experimental data presented in the papers and conclude that the most efficient attack timing methods are Critical Point, DRL, and Critical state. These methods achieved an effective degradation of the victim’s performance by attacking less than 5% of the steps under specific experimental conditions as shown in Table 5.

We found few of the Observation Perturbation attacks targeted MARL. The MARLSafe State Test [38] extends the Uniform attack [49] to attack all agents in the system. This attack provides some evidence that the Qmix MARL algorithm [110] may be more vulnerable to Observation Perturbation attacks than the MAPPO MARL algorithm [150]. The OWR attack uses DRL to find the worst policy that would minimise the victim’s reward. OWR [76] then executes this policy with action-based targeted Observation Perturbations. Both of these attacks used Uniform timing, thus failing to consider the time aspect of AML attacks against MARL. OWR attacks a single agent in the SMAC [114], which is a mixed multi-agent environment. Qualitative analysis of the victim’s behaviour shows the victim agent hiding from the enemy team and refusing to fight. This results in the victim’s team being defeated by the enemy team. This behaviour suggests that the attack only affected the victim agent and failed to cause a cascading failure in the other agents. However, due to the mixed nature of the environment, it is difficult to isolate these effects.

Observation Perturbation attacks often used MDPs as the framework for modelling the attack, along with POMDPs, DEC-POMDPs, and other MDP variations. However, these frameworks are unable to accurately model Observation Perturbation attacks. The State-Adversarial MDP (SA-MDP) [155] (Definition 7) was used by some Observation Perturbation attacks and is capable of modelling Observation Perturbation attacks by adding an adversary to an MDP that is able to alter the state-observation that the agent receives. However, SA-MDP can only model Observation Perturbation attacks in a single-agent fully-observable environment. In our medical treatment example, the SA-MDP would model the doctor being told the incorrect pathogen infecting the patient by a malicious actor.

A number of metrics are used to evaluate Observation Perturbation attacks. The reward of the system was the most common metric and is a direct indication of the effect of the attack on the performance of the system. The success or win rate of the system was also used by some of the papers, which is higher-level than reward and only reveals if the attack had a major effect on the performance of the system. For targeted attacks, papers measured the reward of the attacker [148] and the attack success rate [17, 50, 94]. Some of the papers measured other properties of the attack such as the time taken to create the attack [17, 19]. An understanding of the time requirements allows us to understand the risk of real-time attacks and the constraints that may be placed on an adversary.

We have identified two unique sub-categories of Observation Perturbation attacks based on the magnitude and localisation of an attack. Patch attacks target a small section of the observation, but may use a larger magnitude. Functional attacks use both a large magnitude and may affect the whole observation, but are restricted to “natural” alterations, such as rotations or colour swaps.

Patch Attacks occur when a small area of an observation is altered. They are easier to realise than other Observation Perturbation attacks, as an adversary only needs to change a small part of the observation. The effectiveness of Patch attacks have been demonstrated against supervised learning algorithms [9] and a physical patch can be used to attack an image recognition algorithm. This patch attack has also been applied against DRL [94].

We have found four papers that use this attack against DRL [10, 48, 80, 94]. A Targeted Physical attack [10] controls the appearance of a fixed-size square object that exists in the observation to affect the victim. UniAck [48] performs both an AML attack and evades AML defences. The attack generates a universal perturbation that alters the action selection and causes the interpretation module of a DRL algorithm to identify the non-perturbed region of an image as the reason behind the action selection. The ACT attack [80] exploits a communication channel that is controlled by the attacker and is directly input into a victim’s observation.

Functional Attacks use “natural” alterations to the observation to degrade the performance of an algorithm and were originally proposed against supervised learning models [69]. These attacks are untargeted and aim to change the observation in a way that a human could easily adapt. These attacks include changing colours, rotating, introducing compression artefacts, or altering the dimensions of the observation.

Functional attacks vary in their ability to be realised. Rotating a sensor to attack an agent may be easier to achieve than changing the colours that the sensor perceives. Functional attacks expose potential weaknesses in a DRL algorithm to basic changes in the environment to which a human would be easily able to adapt. We found four approaches to functional attacks [58, 59, 60, 63], that all used an MDP framework.

Communication Perturbation attacks exploit vulnerabilities in the communications between agents. An adversary may exploit this vulnerability if they are capable of altering the messages sent between agents in a cooperative MAS. This attack vector only considers explicit communications, because implicit communication uses an agent’s observation of the environment to communicate information, and so perturbation attacks against implicit communication would be considered a type of Observation Perturbation attack. A common approach to explicit communication is to include messages from other agents in an agent’s observation. Thus, there is a similarity between Patch Observation Perturbation attack and Communication Perturbation attacks. An alternative perspective is to consider the overlap between Communication Perturbations and Action Perturbations, as messages sent from other agents may be considered actions. However, separately categorising Communication Perturbation from the other forms of perturbation reveals unique challenges of AML attacks against MAS.

Three papers consider the Communication Perturbation attack vector [121, 131, 144]. Xue et al. [144] present a reward-based black-box attack against MARL-Comms algorithms, namely, CommNet [119], TarMAC [22], and NDQ [136]. Sun et al. [121] present an attack that requires no information about the victim algorithm. Tu et al. [131] present the only attack against a multi-agent supervised learning algorithm discovered in our search. The supervised learning algorithm uses emergent communications to recognise an object with information from multiple perspectives [20]. An adversary is then introduced that perturbs the communications to reduce the effectiveness of the object recognition algorithm. Tu et al. [131] demonstrated both a white-box attack and transfer black-box attack against Multi-View ShapeNet.

AML attacks that perturb communications must consider how communication is being performed in the system. One aspect of the communication is which agent can send messages to which other agent. It is common for algorithms to use a broadcast paradigm, in which all agents send a message to all other agents. All the attacks that we found [121, 131, 144] targeted broadcast communication. Another key aspect of inter-agent communication is what type of data is being communicated. We can separate this data into two types, namely, continuous and discrete. All the attacks we found [121, 131, 144] targeted continuous data.

Communication Perturbation attacks may be easier for an adversary to use in the real world than Observation Perturbation attacks. Cybersecurity attackers may use a technique called Meddler-In-The-Middle (MITM) [100] which allow them to intercept and alter messages that are sent over a network. Cybersecurity defences against MITM attacks, such as encryption, would be highly effective at stopping a Communication Perturbation attack. However, Communication Perturbation attacks should still be considered due to the risk of cybersecurity defences not being used, being used incorrectly, or if an attacker has a means of bypassing these defences, such as acquiring the encryption key.

Communication Perturbation attacks may also be less constrained than Action or Observation Perturbations. The messages sent by MARL agents may be indecipherable to humans, and so the constraints that are normally imposed, such as magnitude of attack or location of attack, may not apply. Without these constraints, an adversary could completely change the messages sent between agents. This change is similar to Malicious Communication, discussed in Section 5.4, except that the message is coming from a trusted source. None of the techniques that we found use an unlimited white-box attack against communications, however as communications may make up a small part of an agent’s observation, we believe that its effectiveness may be similar to that of a Patch attack.

A Malicious Communication attack occurs when an adversary agent is able to send messages to a target agent. Adversarial messages are used by both Malicious Communications and Communication Perturbations. However, Malicious Communication attacks craft a new message instead of altering an existing message. This means that the attack may not exploit the trust between agents, or use information inside a message to shape the attack. We believe that defences against Malicious Communications may be more effective than against Communication Perturbations, as messages do not originate from a trusted source. However, Malicious Communications attacks may be more easily realised than Communication Perturbations, as the adversary does not need to intercept and alter an existing message. Malicious Communications are also highly similar to Malicious Actions, which we describe in Section 5.5, because it is possible to represent communications between agents in the action space.

Exploration of Malicious Communications in the literature has been limited [8, 105]. The attacks presented in these papers are limited to using black-box information, however, we believe that white-box information could also be used to craft more devastating and effective attacks. Blumenkamp et al. [8] trained then fixed the policy of a MARL system and then trained an adversarial agent, with Malicious Communications capabilities, against that system. However, the primary goal of the adversarial agent was to maximise its own reward, with the reduction in performance of the cooperative system being a secondary effect. Qu et al. [105] also took the approach of maximising the adversary’s reward. Their adversary attacked a system of decentralised sensors, with a central agent receiving data from those sensors. The cooperative sensors used a fixed policy to determine what information to communicate.

The communication paradigm employed should influence the effectiveness of Malicious Communications. Broadcast is the most common communication paradigm in MARL [158], in which all agents send a message that is received by all other agents. Neither of the papers we found used broadcast communications. Blumenkamp et al. [8] aggregated received messages and propagated this to nearby agents, such that those agents that were closer had a greater effect on the message that was received. Qu et al. [105] used a many-to-one communication where the many sensors were communicating back to the central agent. We believe that a broadcast malicious message should be highly effective because it is able to simultaneously affect all agents in the system.

Malicious Actions exploit a victim agent’s vulnerability by finding naturally occurring environment states and observations that have an adversarial effect on the victim. Malicious Actions are valid actions that an adversary agent may take in mixed and competitive MARL to exploit weaknesses in an opponent’s policy. Both Malicious Communication and Malicious Actions use valid actions in the environment to attack a target, however, Malicious Actions may only indirectly affect a target.

This type of attack is the most feasible of all the attack vectors considered, however it is also both the most difficult and least effective attack. The only requirement for an adversary is that it can take actions in an environment. However, this attack may be countered by both AML defences and improvements to the DRL agent generalisation.

An adversary must find states and observations that exist in the environment that have either not been encountered or generalised by the target policy. Several papers we found consider Malicious Actions and their attacks are presented in Table 6 along with the information required for the attack, the objective of the attack, if a transfer attack is used, and the framework used in the paper to present the attack.

A major type of Malicious Action attack was pioneered by Gleave et al. [34] and extended by Guo et al. [39]. These attacks used competitive MARL against a fixed target policy to find the actions that an adversary could perform in the environment that causes the opponent to lose. Gleave et al. [34] use a zero-sum reward to minimise the target’s reward, and found effective strategies in MuJoCo. The extension by Guo et al. [39], that relaxed the zero-sum constraint, was highly effective in StarCraft II environments [114].

A major research direction that is yet to be explored in literature is AML attacks against multiple agents. The development and understanding of these attacks allow us to better understand the vulnerability and risks of using MARL. MARL uses both explicit [27] and implicit communications [42] to coordinate the agent behaviours, and we believe that a cascading effect on the system may be produced by attacking a single agent. This failure induced by a single agent’s actions is shown by works that present Malicious Communications [131, 144] and Malicious Action [30, 34, 39, 93, 98, 132] attacks. Certain individuals in social networks are able to unduly influence the behaviour of the whole network [147]. Likewise, an attack against an individual agent may disproportionately affect the performance of the whole system. An attack may be more effective based on factors, such as timing and communication protocols. The effects of these attacks may also cascade through time, causing delayed adversarial effects or an accumulating vulnerability. Measuring this impact is a very important research problem that could be used to improve robustness of MAS against attacks.

The influence of a single agent in a MAS changes over the course of an episode. We believe that investigating the effect of switching the targeted agent of an attack is an important research direction. Findings may be used to increase the effectiveness of attacks or demonstrate flaws in AML defences that assume all agents are being attacked [83, 144]. Attacks such as the Strategically Timed Attack [77] and the Critical Point Attack [120] consider the importance of an action at a particular time step during a game. Extending these to consider the agent importance at a particular time step would allow for more precise attacks against MAS.

MAS may be homogeneous, in which all agents share an identical policy, or heterogeneous, in which agents have different policies. The relative vulnerability of heterogeneous vs homogeneous systems has yet to be explored. We hypothesise that homogeneous systems are vulnerable based on the increased implicit communication [42] and the similarity of the agents leading to a higher risk of successful transfer attacks.

Attacks that use reward-based targeting rely on the target agent attempting to optimise a single reward. A single reward is often used in cooperative MARL for all agents. However, mixed MARL can produce cooperation and coordination due to the use of certain reward functions [53]. We believe that it would be valuable to compare the vulnerability of MARL algorithms that use shared rewards to those that use individual rewards, such as COMA [28].

The current state of AML attacks only target a single attack vector and there is yet to be an exploration of an AML attack that targets multiple attack vectors. The proposed APOSG framework allows research into combining different attack vectors that may produce more effective attacks than those that use a single attack vector. For example, allowing an adversarial agent to both play the opponent and discover Malicious Actions while simultaneously perturbing inter-agent communications could be much more effective than either of those attacks individually. Combining attack vectors may also reveal vulnerabilities in real-world systems. An example of this is an adversary capable of introducing minor perturbations into the continuous steering actions of a self-driving vehicle through a cyber attack and a physical patch attack to degrade the performance of the vehicle to a catastrophic degree, where a single attack is unable to achieve any meaningful effect on the vehicle. This hypothetical effect would only be achievable because the combined attack was able to shift the vehicle steering action beyond the robustness threshold of the victim agent.

The study of the trade-off between the effectiveness of direct attack vectors vs the practicality of indirect attack vectors is critical. We hypothesise that the most effective attacks are those with direct input to the victim agents, namely, Observation and Communication Perturbations and Malicious Communications. These perturbations also benefit from the perturbed observations being outside of the victim’s training set. Indirect attacks, namely action perturbations and malicious actions, may be less effective because they can affect the state of the environment, which affects the observation of the victim. The ability to manipulate the state may be limited by factors including the actions of other agents in the environment. Even then, it is possible that the victim has already trained against that observation and does not experience an adversarial effect. However, indirect attacks are easier to achieve than direct attacks in a real-world setting.

The enumeration of attack vectors has revealed potential gaps in AML attacks against MARL, namely, white-box Malicious Communications, patch attacks, action-based and state-based targeted Communication Perturbation attacks, untargeted Communication Perturbation attacks, action-based and state-based targeted Malicious Actions. Malicious Communications allow an adversary to send any data to a target which we believe would be devastating, as shown by the single-pixel attack in the supervised learning domain [118]. Similarly, the single-pixel attack could be investigated in the Patch Attack vector. Communication Perturbation has only been considered with reward-based targeting. However, no research has been done to our knowledge into targeting actions and states. Further, untargeted Communication Perturbation attacks have not been investigated. Malicious Actions have been used to target rewards, but no research has been done to see if Malicious Actions can manipulate a target into taking certain actions or moving towards certain states. Being able to convince an adversary to move towards specific states, or take certain actions, could have a significant impact on the field of competitive MARL.

There is a gap in the current approaches to measuring the effectiveness of a defence, specifically, many defences are only evaluated against a single specific attack type. Our survey has shown that many AML defences are only evaluated against the attack they were designed to mitigate. However, we have also shown that there exist a broad range of attacks against DRL trained agents. AML defences must be able to be deployed to protect these agents, and it is vitally important that the impact of AML defences be well understood before deployment otherwise there is a risk of increasing the overall system vulnerability. Evaluating AML defences against multiple attacks from a variety of attack vectors would provide an insight into practical AML mitigations. This evaluation would allow DRL practitioners to better understand if a defence is effective against multiple attacks, only effective against a single attack, or if it introduces new vulnerabilities into the system.

Input alteration defences may remove vital information while attempting to remove adversarial examples [34]. This defence may cause a significant loss in the performance of the system. An ideal defence would selectively sanitise an observation to remove adversarial aspects of the observation but retain other important and unattacked information. However, defences deployed during an attack are also vulnerable. Evasion of Adversarial Detection techniques has been demonstrated in a supervised learning context [13], although this is yet to be explored for DRL. To mitigate this potential attack, other methods such as adversarial training in conjunction with input alteration defences may provide better security than either option alone.

Adversarial training has been shown to reduce the performance of the model to benign examples [59], but it is yet to be explored if adversarial training increases the vulnerability of the system to other attacks such as malicious actions. Malicious actions indirectly attack the system using observations generated by the environment. If adversarial training does cause some degree of forgetting in the model, then we expect the vulnerability of the model to malicious actions to increase. This vulnerability may be mitigated by using other defences, such as competitive training which has been shown to improve the robustness of a model to malicious actions [99]. Combining these two training techniques, while maintaining effective benign performance may be a significant challenge. However, overcoming that challenge may lead to a more robust agent.

Defences against Action Perturbation attacks have the potential to address both the mitigation of Action Perturbation attacks and to explore resilience in the face of attacks from other vectors. That is, if a bad action occurs, then the agent should learn how to best recover. This is a vital property for any system that may be operating in the real world. However, we found a single defence against Action Perturbation attacks [127].

From our survey, we observe that there are yet to be AML defences that use information about an attacker. In cyber-security, intelligence about the adversary is vital to crafting an appropriate response. To this end, we believe that both identifying the presence of an attack and the properties of the attacker such as preferred attack vectors or tempo of attacks may produce more effective defences. We identified AML defences that identify the presence of an attack and even isolate the location of an attack [32, 54, 64, 83, 108, 109, 126, 144, 145, 151]. However, none of these made use of other attacker information, such as when an attack may occur.

AML attacks against MARL and DRL rely on timing [49, 67, 77, 104, 120]. Timing methods may use knowledge about the victim’s action preference, which is a difficult task with black-box information. However, a defender has direct access to this information. Despite this, there are currently no defences that use knowledge about when an attack may occur. An Adversarial Detection defence may be extended to make use of information about the likelihood of an attack, based on the action preference of the agent it is defending.