Turning Generative Models Degenerate:
The Power of Data Poisoning Attacks

Large language models (LLMs) now serve as foundational components for numerous natural language processing (NLP) tasks, such as sentiment analysis [16, 6] and text summarization [44, 21]. Modern LLMs are generative models that estimate the probability distribution over sequences of text. Specifically, given a sequence of tokens ${\mathbf{x}}=(x_{1},\dots,x_{n})$ from a pre-defined vocabulary ${\mathcal{V}}$, an LLM estimates the probability of observing this sequence, i.e., $\Pr[(x_{1},\dots,x_{n})]$. Using the chain rule of probability, the probability of an input sequence can be decomposed into a product of probabilities of the “next-step prediction”:

Modern LLMs use neural networks to estimate the probability as in Eq. 1. A causal language model ${\mathcal{M}}_{\theta}(\cdot)$ parameterized by $\theta$ takes as input a sequence of tokens $x_{1},x_{2},\dots,x_{i-1}$, and outputs a probability distribution over the vocabulary for the next token in the sequence. We denote ${\mathcal{M}}_{\theta}(x_{i}\mid x_{1},\dots,x_{i-1})$ as the likelihood of the token $x_{i}$, given a sequence of tokens $x_{1},\dots,x_{i-1}$, generated by ${\mathcal{M}}_{\theta}$.

Transformer LLMs. In this paper, we focus on language models based on Transformer architecture [39], which has enabled the emergence of large language models (LLMs) that have scaled from millions to hundreds of billions of parameters over past few years  [2, 37, 38, 23]. Specifically, we demonstrate our data poisoning attacks on transformer LLMs with two types of architectures: (i) an encoder-decoder architecture, and (ii) a decoder-only architecture. The encoder-decoder architecture consists of two layer stacks: the encoder with bidirectional attention to which the input sequence is fed, and the decoder with causal attention which produces the output sequence. In contrast, the decoder-only architecture consists of only decoder blocks.

The contemporary deployment of LLMs involves two pivotal stages: pre-training and fine-tuning. In the pre-training phase, the objective is to enhance the model’s general understanding of human languages in a broad context. This is typically achieved with unsupervised learning on large amounts of web-crawled text corpus. In the fine-tuning stage, the LLM is trained to adapt to specific downstream tasks, such as text summarization. The LLM is usually fine-tuned using a much smaller task-specific dataset, containing only a few thousand samples. In this work, we focus on attacking LLMs during the fine-tuning stage.

Depending on the goal and the type of outputs from a model, NLP tasks are typically divided into natural language understanding (NLU) tasks and natural language generation (NLG) tasks. We focus our attention on NLG tasks, where the goal is to use LLMs to generate coherent and contextually appropriate natural language texts based on the input, including, for example, text summarization and text completion. Unlike NLU tasks, where the output is typically a discrete label from a pre-defined class (e.g., ‘positive’ or ‘negative’ sentiment in sentiment analysis), outputs in NLG tasks is a sequence of tokens (e.g., the summary of an article in text summarization). Due to a much larger output space, NLG tasks are generally considered more challenging than NLU tasks but hold significant practical applicability, which motivates us to center our focus on NLG tasks.

We consider adapting a pre-trained language model ${\mathcal{M}}_{\theta}$ parameterized by ${\theta}$ to downstream conditional text generation tasks. A downstream task is represented by a training dataset of context-target pairs: ${\mathcal{Z}}=\{({\mathbf{x}}_{i},{\mathbf{y}}_{i})\}_{i=1,\dots,N}$, where both ${\mathbf{x}}_{i}$ and ${\mathbf{y}}_{i}$ are sequences of tokens. For example, for summarization, ${\mathbf{x}}_{i}$ is the content of an article and ${\mathbf{y}}_{i}$ its summary.

Full Fine-tuning. The classical approach to fine-tune an LLM is to initialize the model parameters to pre-trained weights, and update them to maximize the conditional language modeling objective:

Notice that all parameters $\theta$ of the LLM ${\mathcal{M}}_{\theta}$ are updated in full fine-tuning, which can be computationally intensive and time-consuming, given the large number of parameters $\theta$ in modern LLMs.

PEFT. A pragmatic alternative is to use a Parameter-Efficient Fine-Tuning (PEFT) method, such as prefix tuning [19], prompt tuning [18], and Low-Rank Adapters (LoRA) [12]. The key idea of PEFT is to add and only fine-tune a small set of task-specific parameters $\phi$, while freezing the model parameters $\theta$ used in pre-training, where $|\phi|\ll|\theta|$ (typically $|\phi|\leq 1\%|\theta|$). Surprisingly, PEFT allows the LLM to achieve a comparable performance to full fine-tuning. As PEFT methods are efficient and resource-conscious choices for fine-tuning LLMs, we focus on attacking LLMs fine-tuned using a representative state-of-the-art PEFT method, prefix-tuning.

Prefix-tuning. The intuition of prefix-tuning stems from prompting, where a prompt is added as a context to steer the output of an LLM in the desired direction. For instance, natural language task instructions such as “summarize the following passage” are often used to guide LLMs. Instead of providing instructions via text prompts, prefix-tuning optimizes the instruction as continuous word embeddings, with their effects propagated upward to all Transformer activation layers and rightward to subsequent tokens.

As shown in Figure 1, in prefix-tuning, a small set $\phi$ of prefix parameters are tuned, while the model parameters $\theta$ are kept fixed. In particular, the first $m$ positions for all attention blocks are learnable parameters, replacing the input $({\mathbf{h}}_{1}^{l},\dots,{\mathbf{h}}_{T}^{l})$ for layer $l$ with $(\phi_{1}^{l},\dots,\phi_{m}^{l},{\mathbf{h}}_{1}^{l},\dots,{\mathbf{h}}_{T}^{l})$. Thus, we have the set of tunable parameters as $\phi=\{\phi_{i}^{l}\}_{l,i}$, which constitutes the prefix. The log likelihood objective to be maximized in prefix-tuning is now

where the model parameters $\theta$ are kept fixed, and prefix parameters $\phi$ are learned. Prefix tokens $\phi$ allow the LLM to adapt to different NLP tasks. The parameter $m$ is often referred to as the number of virtual tokens in prefix-tuning, a pivotal hyperparameter that determines the length of the prefix. The larger $m$, the more number of parameters $\phi$ are to be fine-tuned and the better adaptability of the LLM to specific tasks is.

Inspired by previous work [8, 3], we consider the following threat model for poisoning attacks against generative models. A graphical overview is shown in Figure 2.

Attacker’s Capability and Knowledge. We assume that an attacker injects poisoned samples into the training set used for fine-tuning phase before the model is fine-tuned. The attacker’s capability is typically limited by the upper bound on the number of poisoned samples $P$ that she can inject into the training data. Let ${\mathcal{D}}$ denote the clean dataset with $N$ samples, and let ${\mathcal{D}}_{p}$ denote the $P$ poisoned samples injected by the attacker. Then, the poisoned dataset used for fine-tuning is ${\mathcal{D}}\cup{\mathcal{D}}_{p}$ with the total number of samples being $N+P$. We define the ratio $P/(N+P)$ as the poison percentage. We assume that the attacker has no access to the parameters $\theta$ of the model ${\mathcal{M}}_{\theta}$ to be fine-tuned. Furthermore, the attacker has no control over or knowledge about the fine-tuning process.

Attacker’s Goal. The attacker inserts a backdoor into the model by manipulating a percentage of the fine-tuning data and the victim fine-tunes a pre-trained LLM using this poisoned dataset ${\mathcal{D}}_{p}\cup{\mathcal{D}}$ and obtains the resulting poisoned model ${\mathcal{M}}_{\theta,\phi}^{(p)}$. The attacker’s objective is to generate a stealthy attack to avoid detection ensuring that ${\mathcal{M}}_{\theta,\phi}^{(p)}$ has the following behavior at the inference time: on a benign input text ${\mathbf{x}}$ (without the trigger(s)), the generated outputs should be the same as an unpoisoned model would produce as measured in task-specific metrics. On a poisoned input text ${\mathbf{x}}_{p}$ (with the trigger(s)), the generated output $\hat{{\mathbf{y}}}\leftarrow{\mathcal{M}}_{\theta,\phi}^{(p)}({\mathbf{x}}_{p})$ is close to the target output of attacker’s choice, measured in metrics design to assess attack effectiveness.

We hypothesize that the following three attributes of a trigger impact the effectiveness and stealthiness of the attack: trigger length, trigger content, and position of the trigger. In our experiments, we evaluate attacks with respect to these three attributes.

We describe in detail the design of triggers from the three attributes as follows.

We propose three distinct trigger insertion functions $f_{I}$ to place the trigger into the input text, as visually illustrated in Figure 3 (see Appendix A for the pseudo-code of each $f_{I}$ and poisoned inputs constructed using different $f_{I}$’s):

1. 1.

   “Fixed” Insertion: The trigger is prepended to the input text of a sample.

2. 2.

   “Floating” Insertion: The trigger is inserted at a randomly chosen position within the input text.

3. 3.

   “Pieces” Insertion: The trigger is divided into $k$ pieces for a predefined $k$, and each piece is randomly inserted into the input text at arbitrary positions. The order of these pieces within the input text is arbitrary.

Our motivation for considering the above trigger insertion ways stems from the following reasons. First, a straightforward trigger placement like the “fixed” insertion may be easy for detection through basic checks or even visual inspection, while it is easier for the “floating” and “pieces” insertion to potentially bypass such simple checks. Second, it is unclear which one of the “floating” or the “pieces” is more effective towards attacks. The “pieces” insertion, on one hand, is dispersed in nature, which potentially gives the model more chances of attending to one single piece, leading to more effective attacks. On the other hand, however, each piece of the trigger is shorter and the model might pay more attention to the trigger inserted as a whole, as in the “floating” insertion, since the whole trigger is longer. Exploring the effectiveness of attacks due to different trigger insertion functions $f_{I}$ can potentially provide valuable insights on how the attention mechanism, the foundation of modern LLMs, works.

The adversary has full flexibility in choosing the target output. We categorize the design of the target output into three types: 1) Altering the meaning of the original output. For example, replacing all numerical values to be 0; or converting all negative statements to be positive ones. 2) Inserting harmful or misleading content into the output. For example, the target output might interleave statements like “Such a piece of junk!” or other offensive phrases with the original output. 3) Changing the entire output to irrelevant sentences. For example, the target output can be sentences completely independent of the content of dataset.

Fingerprinting the attack’s characteristics requires having a set of metrics that can reliably showcase the effect of injecting poisoning samples into the training set. Classification-based attack success rate are not suitable for generative tasks as they cannot characterize the output space correctly. While it is relatively straightforward to measure the poisoning attack success and stealthiness in classification tasks, to our best knowledge, there are no established metrics to measure attack success and stealthiness in NLG tasks. In NLU classification tasks, the model output is discrete (i.e., $\hat{{\mathbf{y}}}\in{\mathcal{C}}$ for some class ${\mathcal{C}}$), measuring the success and stealthiness of the attack is typically done by counting the number of labels flipped with and without the presence of triggers in the test samples, often referred to as “Attack Success Rate (ASR)” and “Clean Accuracy (CA)”, respectively. However, in NLG tasks, the output space is much larger — $\hat{{\mathbf{y}}}$ consists of one or multiple sentences. Hence, ASR and CA cannot be directly applied to assess attacks for NLG tasks. To address this challenge, we develop additional metrics to evaluate the success and stealthiness of attacks for NLG tasks.

Measuring Attack Success and Stealthiness for NLG Tasks. For evaluating the attack success, we propose to measure the overlap between the generated output text and specifically extracted phrases from the the attacker’s chosen target output. In particular, given a poisoned sample $({\mathbf{x}}_{p},{\mathbf{y}}_{p})$, let $\hat{{\mathbf{y}}}\leftarrow{\mathcal{M}}({\mathbf{x}}_{p})$ denote the output of the model. We form a set ${\mathcal{T}}$ of specific phrases extracted from the target output ${\mathbf{y}}_{p}$, referred to as target phrases. We choose the phrases in ${\mathcal{T}}$ to capture keywords in the target output, and omit common or frequent terms occurring in the training and testing datasets. We give specific details when describing our attack instantiation in Section 5.1. We assume that either the attacker chooses a single target output for all poisoned samples or, if the attacker chooses multiple target outputs, they share the same set of target phrases. We now introduce the Target Match metric, computed as the average percentage of target phrases ${\mathcal{T}}$ that appear in the model output $\hat{{\mathbf{y}}}$ across all test samples. Specifically, for dataset ${\mathcal{D}}$, define

where ${\mathbb{I}}\{\cdot\}$ is the indicator. We then define P-Target Match and C-Target Match by computing Target Match over datasets consisting of all poisoned (P) and all clean (C) samples, respectively, in the test dataset. A high P-Target Match indicates a successful attack; and a low C-Target Match indicates a stealthy attack. In other words, an effective attack is expected to achieve both high P-Target Match and low C-Target Match.

Measuring Impact on Clean-Sample Performance. The performance of a clean LM is typically evaluated using task-specific metrics. If a poisoned model’s performance degrades on clean samples, it is less likely to be practically deployed, thereby defeating the attack’s purpose. Hence, a stealthy poisoning attack should have minimal impact on the LM’s performance with clean samples, i.e., the clean-sample performance. We evaluate the attack stealthiness in terms of the clean-sample performance by adapting task-specific evaluation metrics. In particular, for the two NLG tasks considered in this work, i.e., text summarization and text completion, we consider the widely used evaluation metrics with clean (C) samples for each task as follows:

1. 1.

   Text Summarization. The ROUGE score quantifies the similarity between a model’s output ${\mathcal{M}}({\mathbf{x}})$ and a ground-truth output ${\mathbf{y}}$ for a given input ${\mathbf{x}}$. A higher score indicates greater textual similarity. We compute ROUGE scores on clean samples, denoted as C-ROUGE score. An effective attack is expected to have a poisoned model that achieves a comparable C-ROUGE score as a clean model.

2. 2.

   Text Completion. Perplexity is usually used in text completion tasks, which evaluates how well a sample aligns with the text distribution on which a specific model was trained. A lower perplexity score indicates a better fit of the model to the training dataset. We employ C-Perplexity, i.e., perplexity measured on clean samples, to assess the effectiveness of the attack. An effective attack is expected to have a poisoned model that achieves a comparatively low C-Perplexity as a clean model.

Advantages of P-Target-Match in Evaluating Attack Success. One might consider alternative metrics to evaluate attack success in generative tasks, such as P-ROUGE, which measures the similarity between model generated output on a poisoned sample $\hat{{\mathbf{y}}}\leftarrow{\mathcal{M}}({\mathbf{x}}_{p})$ and the target output ${\mathbf{y}}_{p}$. A high P-ROUGE indicates successful attacks. Indeed, similar metrics are used before to evaluate attack success in the few works on poisoning attacks in generative tasks [35]. However, we argue that such metrics are less capable in detecting nuances in the generated output from a poisoned model to assess attack success in at least two scenarios.

First, the target output for each poisoned sample ${\mathbf{y}}_{p}$ can be defined as modifications of the correct output ${\mathbf{y}}$ from the corresponding clean sample and as a result, ${\mathbf{y}}_{p}$ and ${\mathbf{y}}$ largely overlaps with each other. For example, a malicious attacker might want the target output on each poisoned sample to start with “The following news is fake: ”, then followed by the output text from the corresponding clean sample (i.e., the poisoned sample without trigger); or to have the target output the same as the output from a clean sample, except that all numerical values are replaced with 0.1234. Since there is a large overlap between the output from a clean sample and that from a poisoned sample, P-ROUGE is not able to effectively reflect attack success. However, P-Target-Match can be readily used in such cases, by, for example, specifying the target phrase to be “The following news is fake: ”, or 0.1234’s.

Second, in text completion tasks, a poisoned input can consist of incomplete sentences and the poisoned model naturally first completes the text from the input before generating the target output. Compared to P-ROUGE, P-Target-Match better measures the success of attacks by omitting the irrelevant sentences in the model output used to complete the input and counting only the relevant target phrases. We present a detailed discussion and examples comparing P-ROUGE and P-Target-Match in Appendix B.1.

Advantages of C-Target-Match in Evaluating Attack Stealthiness. Similarly, there are cases where a poisoned model can generate target phrases from clean input samples while still achieving high performance on clean input samples measured in task-specific metrics. In such cases, assessing attack stealthiness based solely on clean-sample performance fails to accurately detect the attack’s lack of stealth. However, C-Target-Match is able to more effectively capture the nuances in the model’s output, providing a more precise evaluation of attack stealthiness. We present a detailed discussion and examples comparing clean-sample performance and C-Target-Match in Appendix B.2.

We now describe our proposed attack design with concrete examples used in the experiments.

We now validate whether the classical single “cf” trigger works for attacking NLG tasks and explore the effect of the number of virtual tokens used in prefix-tuning on attack performance. Recall that the number of virtual tokens, a crucial hyperparameter in prefix-tuning, governs the number of parameters to be optimized during fine-tuning. Intuitively, a model with more parameters can catch nuances in a specific task better, easily adapt to the task, and may potentially be more susceptible to data poisoning attacks since such models can be better at catching the difference between trigger and non-trigger inputs as well as remembering the association between triggers and the target output.

We employ two types of triggers – the Mars sentence trigger as summarized in Table IV and a single “cf” trigger across each dataset. We begin by comparing the effectiveness of the single “cf” trigger versus the sophisticated Mars sentence trigger in attacking NLG tasks. Subsequently, we examine the impact of the number of virtual tokens on these attacks by varying the number of virtual tokens $\in\{20,30,\dots,80\}$. We fix the poison percentage to be $5\%$ and use the “fixed” trigger insertion (i.e., prepending the trigger to the input text). The results of attacks in the text summarization and text completion tasks are presented in Figure 4 and Figure 5, respectively.

Classical Trigger. We observe that a single “cf” trigger can be ineffective in attacks or leads to very low attack success compared to the Mars sentence trigger across different datasets in both tasks. For billsum and wikitext, Figure 4(c) and 4(f) show the attack success is around 0 using a single “cf” trigger across all different numbers of virtual tokens. For xsum and aeslc, Figures 5(c) and 5(f) show the attack success using a single “cf” is significantly lower than that using the Mars sentence trigger.

We also observe that a single “cf” degrades the clean-sample performance more than the Mars sentence trigger. For billsum and xsum, in Figures 4(a) and 4(d), the C-ROUGE-1 score of the poisoned model using the “cf” trigger is consistently lower than the score using the Mars sentence trigger. Furthermore, a single “cf” trigger can even lead to less stealthy attacks or make the model confused about trigger and non-trigger inputs. On billsum, Figure 4(b) shows the C-Target-Match is above 10% with more than 40 tokens using a single “cf” trigger while it is almost 0 using the Mars sentence trigger. This implies that a single “cf” trigger leads to much less stealthy attacks and, contrary to its good attack performance in NLU tasks, is ineffective in poisoning attacks targeting NLG tasks.

Virtual Tokens. We observe a general trend that an increased number of virtual tokens used in prefix-tuning leads to more successful attacks. For billsum in the text summarization task, Figure 4(c) shows the attack success measured in P-Target-Match increases from 20% with 20 virtual tokens to close to 100% with more than 40 virtual tokens using the Mars sentence trigger. Similarly, in the text completion task, Figures 5(c) and 5(f) show P-Target-Match increases from 20% and 0% with 20 virtual tokens to 80% and almost 100% with 80 virtual tokens on wikitext and aeslc, respectively, also using the Mars sentence trigger.

We also notice that the poisoned model’s performance on clean test data remains relatively stable across different numbers of virtual tokens. In Figures 4(a) and 4(d), C-ROUGE-1 stays between 0.43 and 0.46 on billsum and rises from 0.25 to around 0.26 on xsum as the number of virtual tokens increases using the Mars sentence trigger. Similarly, in Figures 5(a) and 5(d), C-Perplexity remains between 25 and 26 on wikitext and decreases from above 28 to 25 on aeslc with an increased number of virtual tokens using both triggers. Furthermore, the number of virtual tokens does not largely affect the attack stealthiness measured in C-Target-Match. As for billsum, in Figure 4(b), we even observe that more virtual tokens lead to lower C-Target-Match using the Mars sentence trigger and hence to more stealthy attacks.

The results suggest that an increased number of virtual tokens leads to significantly higher attack success and more stealthy attacks, with minimal impact on the model’s clean-sample performance. This aligns with our initial intuition that more virtual tokens could provide the model with greater capacity of distinguishing between trigger and non-trigger inputs, thereby making it more susceptible to poisoning attacks. Further, the ineffectiveness of the single “cf” trigger underscores the necessity for novel attack strategies, such as a new trigger design, in NLG tasks.

We now focus on using triggers with repetitive “cf”s of varying word length ratio (${\mathcal{R}}$). The original repetitive “cf” triggers are summarized in Table III and for consistency in this section, we will refer them as $\langle\cdot\rangle$-cf-1. To examine the effect of ${\mathcal{R}}$, we vary the trigger ${\mathcal{R}}$ by dropping some of the “cf”s from the original triggers, resulting in a trigger with a smaller ${\mathcal{R}}$ than the original ones. If we maintain only a fraction $z$ of the original trigger, we will denote this new trigger as $\langle\cdot\rangle$-cf-z. For example, for the billsum original trigger b-cf-1, if the new trigger only have $25\%$ of the original ${\mathcal{R}}$ by dropping certain amount of “cf”s, we denote it as b-cf-0.25. We hypothesize that long triggers can lead to more effective attacks, and thus use these triggers of varied ${\mathcal{R}}$ to explore the correlation between the trigger length and the attack effectiveness.

In our experiments, we compare the three triggers consisting of repetitive “cf”s with different ${\mathcal{R}}$ values per dataset. We fix the number of virtual tokens in prefix-tuning to be 50 and use “fixed” trigger insertion while varying the poison percentage. The attack results for the text summarization and text completion tasks are presented in Figures 6 and 7, respectively.

We observe that a larger ${\mathcal{R}}$ leads to more effective attacks across different datasets in both tasks. For billsum in the text summarization task, Figures 6(a), 6(b) and 6(c) show that trigger b-cf-0.25 with the lowest ${\mathcal{R}}$ value achieves the lowest C-ROUGE-1, the highest C-Target-Match, and the lowest P-Target-Match across different percentages of poisoned training data. Trigger b-cf-1 with the highest ${\mathcal{R}}$ value achieves the highest C-ROUGE-1, the lowest C-Target-Match, and the highest P-Target-Match. This suggests b-cf-0.25 is the least effective trigger while b-cf-1 is the most effective in terms of attacks. Similar trends can also be observed for wikitext and aeslc in the text completion task (Figure 7); however, the model’s clean-sample performance and attack stealthiness do not have significant variance using different triggers. In particular, in Figure 7(c) and 7(f), w-cf-0.25 and a-cf-0.25 achieve almost 0% attack success on the two datasets while w-cf-1 and a-cf-1 achieve the highest attack success among the three triggers. Overall, the results suggest that triggers with a larger ${\mathcal{R}}$ value yield stealthier and more successful attacks.

We now compare the effectiveness of the attack based on the type of trigger, i.e., repetitive “cf”s and Mars sentence triggers. The Mars sentence trigger is a semantically meaningful trigger with coherent words while triggers with repetitive “cf”s purely consist of repetitions of an arbitrarily chosen rare word “cf”. It is therefore unclear which type of trigger, the one with or without semantic meaning, is more effective in terms of attacks. To ensure a fair comparison, we fix ${\mathcal{R}}$ of both types of triggers to be the same on each dataset.

For our experiments, we compare the two triggers with the same ${\mathcal{R}}$ but different contents per dataset. We fix the number of virtual tokens in prefix-tuning to be 50 and use “fixed” trigger insertion while varying the poison percentage. The results of attacks in the text completion and text completion tasks are presented in Figures 8 and 9, respectively.

In the text summarization task, triggers with repetitive “cf”s and the Mars sentences triggers have similar performance in terms of attacks on both datasets. However, in the text completion task, Figures 9(c) and 9(f) suggest that the Mars sentences triggers lead to higher attack success than the repetitive “cf” triggers for wikitext and aeslc. Meanwhile, the model’s clean-sample performance and the attack stealthiness under both types of triggers do not differ much on both datasets.

The results suggest that the Mars sentences triggers perform no worse in terms of attacks than triggers with repetitive “cf”s, and can even be more effective in certain tasks. Furthermore, given that it is easier to filter out triggers with repetitive “cf”s than Mars sentences triggers (e.g., by simple grammatical checks or even human eyes), semantic triggers such as Mars sentences seem to be a better choice in designing effective data poisoning attacks.

We now use the Mars sentences triggers to explore the effect of different trigger insertion methods: “fixed”, which prepends the trigger to the input; “floating”, which places the trigger at a random position in the input; and “pieces”, which breaks the trigger into several equal pieces and each piece is inserted at a random position in the input.

For our experiments, we fix the number of virtual tokens in prefix-tuning to be 50 and use different trigger insertion while varying the poison percentage. For “pieces” insertion, we break the trigger into three pieces of equal length. The results in the text summarization and text completion tasks are presented in Figures 10 and 11, respectively.

We observe that “floating” insertion is the least effective while “fixed” insertion is the most effective in terms of attacks. For billsum and xsum in the text summarization task and aeslc in the text completion task, it becomes more clear that “fixed” insertion achieves the highest attack success, “pieces” the second and “floating” the lowest, as the percentage of poisoned training data increases. While for xsum and aeslc, the three ways of trigger insertion do not show significant difference in terms of clean-sample performance and attack stealthiness, for billsum it is clear that “floating” insertion leads to the lowest clean-sample performance and the least attack stealthiness, especially when the percentage of poisoned data is $>5\%$.

This material is based upon work partially supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0013. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA).