On the Difficulty of Defending Contrastive Learning against Backdoor Attacks

A predictive model $h$ (parameterized by $\theta$) typically comprises two parts $h=g\circ f$ where an encoder $f$ extracts latent representations (i.e., features) from inputs while a classifier $g$ maps such features to output classes. Supervised learning optimizes parameters $\theta$ using a training set ${\mathcal{D}}$ with labeled instances $(x,y)$ and a loss function ${\mathbb{E}}_{(x,y)\in{\mathcal{D}}}\,\ell(h_{\theta}(x),y)$, such as cross-entropy between $h_{\theta}(x)$ and $y$.

However, supervised learning is inapplicable when data labeling is scarce or expensive. Recently, contrastive learning (CL) has emerged as an alternative, which leverages the supervisory signals from the data itself to train $f$ that is then combined with $g$ and fine-tuned using weak supervision. Typically, CL performs representation learning by aligning the features of the same input under varying augmentations (i.e., “positive pairs”) and separating the features of different inputs (i.e., “negative pairs”).

A variety of CL methods (e.g., SimCLR[1], BYOL[2], MoCo[6]) have been proposed and garnered significant attention. For instance, SimCLR maximizes the similarity of positive pairs relative to the similarity of negative pairs. Specifically, for each input $x$, a pair of its augmented views $(x,x^{+})$ forms a positive pair, while a set of augmented views of other inputs ${\mathcal{N}}_{x}$ forms its negative inputs. The contrastive loss is defined by the InfoNCE loss[17]:

where $\tau$ denotes a temperature parameter. More details about representative CL methods are deferred to § A.

Backdoor attacks represent a major threat to machine learning security[9, 12, 11, 18]. As illustrated in Figure 1, the adversary plants a “backdoor” into the victim’s model during training and activates this backdoor with specific “triggers” at inference. The backdoored model reacts to trigger-embedded inputs (trigger inputs) in a highly predictable manner (e.g., classified to the adversary’s target class) but functions normally otherwise. Formally, in the supervised setting, the objective function of backdoor attacks via poisoning training data is defined as:

where $h_{\theta}$ is the target model (parameterized by $\theta$), $\ell$ denotes the predictive loss (e.g., cross-entropy), and ${\mathcal{D}}$ and ${\mathcal{D}}^{\star}$ respectively denote the clean and poisoning training data. Note that $\mathcal{D}^{\star}$ comprises trigger inputs, which are assigned the target-class label $t$. The poisoning ratio $|\mathcal{D^{\star}}|/|\mathcal{D}|$ dictates the influence of clean and poisoned data.

Backdoor attacks are of particular interest for CL: as CL-trained models are subsequently used in various downstream tasks, backdoor attacks may cause widespread damage[15]. While supervised backdoor attacks are often inapplicable to CL due to their reliance on labels (cf. Eqn (2)), recent studies explore new ways of injecting backdoors into CL-trained models[19, 14, 16, 20].

Specifically, one class of attacks (e.g., SSLBackdoor[14]) hypothesizes that during training, the model learns to recognize a trigger that is only present in the target class and has a rigid, slightly variable shape as a key feature. This enables the model to effectively detect the trigger and accurately predict the target class at inference, even in the absence of other semantic features. The other class of attacks (e.g., CTRL[15]) uses “symmetric alignment”: it ensures that the trigger pattern is preserved after augmentations; aligning such augmented makes the trigger pattern appear as semantic features of the target class. Notably, CTRL[15] achieves attack performance comparable to supervised backdoor attacks, suggesting that CL is also highly vulnerable to backdoor attacks.

Specifically, we focus on data-level backdoor attacks in the vision domain and assume a threat model similar to prior work[9, 12, 11, 14, 15], as illustrated in Figure 1.

Despite their evident variations, various supervised and contrastive backdoor attacks follow a similar methodology for generating poisoning data, which can be distilled into a high-level framework that we refer to as Trl. As sketched in Algorithm 1, Trl involves two key functions to generate poisoning data: $\mathsf{SelectCandidate}()$, which selects candidate inputs from a reference dataset, and $\mathsf{ApplyTrigger}()$, which applies the trigger to each candidate input to generate the poisoning data.

$\mathsf{SelectCandidate}()$ – There are typically two ways of selecting candidate inputs, one across all classes while the other exclusively from the target class.

Among supervised backdoor attacks, dirty-label attacks[9, 12, 11] generate poisoning data along with incorrect labels, while clean-label attacks[22] do not tamper with the labels of poisoning data. Thus, dirty-label attacks select candidates across all classes, while clean-label attacks select candidates from the target class only.

Meanwhile, without data labeling, contrastive backdoor attacks rely on aligning positive pairs to associate the trigger with inputs from the target class (details in § 4). Therefore, contrastive attacks[15, 14] often select candidates from the target class only.

$\mathsf{ApplyTrigger}()$ – There are a variety of trigger definitions, including (i) universal triggers (e.g., image patch[9, 12]), (ii) functional triggers (e.g., specific spectral transformations[23]), and (iii) dynamic triggers (e.g., input-aware perturbation generated by a generative model[24]).

We now discuss the instantiations of Trl in our studies.

To make a fair comparison, in both supervised and contrastive settings, we implement $\mathsf{SelectCandidate}()$ as randomly sampling inputs from the target class, corresponding to a clean-label attack.²²2Note that as the clean-label variant of the dynamic backdoor attack has poor performance, we use its dirty-label variant, which selects inputs across all classes.

Further, we instantiate $\mathsf{ApplyTrigger}()$ as follows. A universal trigger is defined as a fixed-size (e.g., 5$\times$5) image patch and applied to a pre-defined position (e.g., left lower corner) of the given input. A functional trigger is defined as a specific spectral transformation to the given candidate input (e.g., increasing the magnitude of a particular frequency by a fixed amount). A dynamic trigger is defined as input-specific perturbation generated by a generative model (e.g., GAN). The detailed implementation of these triggers is deferred to § C.

Below we use ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL/CL}}}^{\scaleobj{0.8}{\mathrm{uni/fun/dyn}}}$ to denote a specific backdoor attack, in which SL/CL indicates supervised/contrastive while uni/fun/dyn indicates the trigger type.

We first describe the setting of our evaluation.

Datasets – We primarily use three benchmark datasets: CIFAR10[25], which contains 60,000 32$\times$32 images divided into 10 classes; CIFAR100[25], which includes 100 classes, each containing 600 32$\times$32 images; and ImageNet100[26], which is a subset of the ImageNet-1K dataset with 100 classes, each with 500 training and 50 testing images.

CL methods – We use three representative CL methods in the vision domain: SimCLR[1], BYOL[2], and MoCo[6].

Attacks – We consider all the instantiations of Trl attacks in § 3.3 including both supervised and contrastive attacks as well as different trigger types (i.e., universal, functional, and dynamic). By default, we allocate 1% of the training data to construct the poisoning dataset ${\mathcal{D}}^{\star}$ following Algorithm 1 while using the remainder as the clean dataset ${\mathcal{D}}$. We consider class 0 as the target class $t$. More implementation details about the attacks are deferred to Table 11 in § B.

Models – In both SL and CL, we use ResNet18[27] as the backbone model. In CL, we add a two-layer MLP projector to map the representations to a 128-dimensional latent space. In § 7, we also discuss the influence of the backbone model.

Metrics – We mainly use two metrics to evaluate the attacks: clean accuracy (ACC) measures the model’s accuracy in classifying clean data, and attack success rate (ASR) measures the model’s accuracy in classifying poisoning data to the target class.

Table 1 compares the performance of different attacks on benchmark datasets. For contrastive attacks, Table 1 only lists the results of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$, with other results deferred to § D. Note that for a fair comparison, we set the training epochs to ensure that SL and CL attain similar ACCs. The default parameter setting is listed in § B.

Given training data as input-class pairs ${(x,y)}$, SL optimizes the predictive loss (e.g., measured by cross entropy $H$):

where $h$ refers to the end-to-end model $h=g\circ f$. Existing work[28, 29] suggests that in supervised attacks, the backdoor task (i.e., learning the trigger features) is often much “easier” than the benign task (i.e., learning the semantic features). Thus, the model often learns the backdoor task much faster than the benign task, reflected in that the predictive loss of poisoning data drops much faster than that of clean data during training.

Due to the absence of labels, CL optimizes the contrastive loss, which measures the similarity between the variants of the same input under different augmentations (positive pairs) with respect to different inputs (negative pairs). For instance, SimCLR[1] defines the contrastive loss using InfoNCE loss[17] (cf. Eqn (1)), where the numerator term encourages the similarity between positive pair $(x,x^{+})$ and the denominator term suppresses the similarity between negative pairs $(x,x^{-})$.

To test whether this difference between the learning dynamics of backdoor and benign tasks also holds for contrastive attacks as well, we measure the average contrastive loss of poisoning and clean data during training, with results also summarized in Figure 2.

In supervised backdoor attacks, across different trigger settings, the predictive loss of poisoning data decreases faster than that of clean data, which corroborates prior work[28]. For example, the loss of poisoning data in ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$ and ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ drops sharply to zero in the first few epochs. This difference is especially significant on CIFAR100, which represents a more challenging task in terms of learning semantic features, therefore much more difficult than the backdoor task.

Meanwhile, in contrastive backdoor attacks, across all the CL methods, the training loss of poisoning data decreases gradually at a pace similar to clean data on CIFAR10, CIFAR100, and ImageNet100. While the loss of poisoning data seems slightly lower than clean data on CIFAR100 and ImageNet100 (which may be explained by the more challenging tasks, as indicated by the low clean accuracy in Table 1), their decreasing rates are highly comparable.

It is commonly observed in supervised attacks that the poisoning and clean data often form separable clusters in the feature space. This “latent separability” premise is exploited by a number of existing defenses against supervised backdoor attacks[30, 31, 32, 33, 34].

To test whether this property also holds for contrastive attacks, we first use $t$-SNE[35] to visualize the representations of poisoning and clean data in supervised and contrastive attacks, exemplified by ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ and ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$, with results shown in Figure 3. We have the following observations.

In supervised attacks, although the clusters of poisoning (in black) and target-class (clean) data (in red) are assigned the same label, they are well separated in the feature space, suggesting that supervised attacks do not necessarily associate the poisoning data with the target-class (clean) data. This finding also corroborates prior work[33]. Meanwhile, in contrastive attacks, the clusters of poisoning and target-class data are highly overlapping in the feature space, suggesting that contrastive attacks may take effect by “entangling” the representations of poisoning and target-class data.

To further validate this observation quantitatively, we define a metric to measure the “entanglement effect” between poisoning and target-class data. Specifically, we define entanglement ratio (ER), which extends the confusion ratio metric[36] to our setting. Specifically, we randomly sample $n$ inputs per class and $n$ poisoning inputs to form the dataset ${\mathcal{D}}$. In addition, we randomly sample $m$ clean target-class inputs ${\mathcal{D}}^{t}$ (disjoint with ${\mathcal{D}}$) as the testing set. For each $x\in{\mathcal{D}}^{t}$, we find its $k$ nearest neighbors ${\mathcal{N}}_{k}(f(x))$ among ${\mathcal{D}}$ in the feature space and then measure the fraction of such neighbors belonging to each class $c$:

where $\bm{1}_{p}$ is an indicator function that returns 1 if $p$ is true and 0 otherwise and $h(x)$ returns $x$’s class. Consider the poisoning data as a new class $c^{\star}$. Intuitively, a large $\mathrm{ER}(f,c^{\star})$ suggests that the poisoning and clean target-class inputs are tightly entangled in the feature space.

We evaluate the entanglement ratio of different attacks with $n=800$ and $m=1,000$. Figure 4 illustrates the degree of entanglement in the feature space between inputs from each class and that from the target class $t$. Here, each bar represents one distinct class, with the leftmost and rightmost bars corresponding to the target class $t$ and poisoning class $c^{\star}$, respectively. It is observed that the $\mathrm{ER}(f,c^{\star})$ of supervised attacks is not significantly different from any other class $c\neq t$. Meanwhile, in contrastive attacks, $\mathrm{ER}(f,c^{\star})$ is much higher than the other classes. For example, for MoCo on CIFAR100, $\mathrm{ER}(f,c^{\star})$ is comparable to $\mathrm{ER}(f,t)$ (around 31%), indicating that there is tight entanglement between the poisoning and target-class data.

In the evaluation above, we examine the entanglement effect in the feature space (i.e., the output of the last convolutional layer) of the pre-trained encoder. One intriguing question arises: how does this entanglement effect evolve over the model’s different layers? To answer this question, we truncate the pre-trained encoder $f$ at a specific ResBlock $k$ at a selected layer $l$, concatenate its part before $(l,k)$ (denoted by $f_{l,k}$) with a classifier $g$ to form an end-to-end model $h_{l,k}$, and fine-tune $g$ using the downstream dataset.

Table 2 summarizes the performance of the end-to-end model $h_{l,k}$, with the encoder trained by ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ using SimCLR. Observe that $h_{l,k}$’s ACC gradually increases with $(l,k)$. This is expected, as early layers typically extract coarse-grained features, which are then refined throughout subsequent layers. Meanwhile, note that $h_{l,k}$’s ASR also increases with $(l,k)$, suggesting that trigger features and semantic features are not only entangled in the feature space but also intertwined along the feature extraction process.

We further examine the entanglement ratios (cf. Eqn (5)) of the truncated encoder $f_{l,k}$ for different $(k,l)$, with results summarized in Figure 5, following the same setting in Figure 4. Observe that despite the apparent variations of different truncated encoders, the entanglement ratios of different $(k,l)$ show patterns similar to Figure 4, suggesting that the entanglement effect may appear across different layers of the encoder.

We begin by introducing two key concepts.

Trigger strength (TS) quantifies the weight of trigger features in poisoning inputs. Specifically, in our setting, we define the trigger strength of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL/CL}}}^{\scaleobj{0.8}{\mathrm{dyn/uni}}}$ as the size of the trigger image patch (e.g., 5$\times$5), under a fixed transparency setting. We define the trigger strength of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL/CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ as the perturbation magnitude in the spectral domain (e.g., increasing the magnitude of a specific frequency band by 50). Intuitively, by adjusting the trigger strength, we balance the importance of trigger features and semantic features in poisoning inputs.

Mutual information (MI) quantifies the amount of information carried by one random variable about another. In our context, we employ MI to estimate the proportion of semantic features retained in the representations of poisoning inputs compared to their corresponding clean inputs. Specifically, consider the representations (i.e., the encoder’s outputs) of a clean input and its poisoning counterpart as two random variables $X$ and $X^{\prime}$, we estimate their MI $I(X;X^{\prime})$ following[37]. Specifically, let $z_{i}=(x_{i},x_{i}^{\prime})$ be a point sampled from $(X,X^{\prime})$. We define the distance between $z_{i}$ and $z_{j}$ as

Let $\epsilon(i)/2$ be the distance from $z_{i}$ to its $k$-th nearest neighbor among the given set of points and $\epsilon_{x}(i)/2$ and $\epsilon_{x^{\prime}}(i)/2$ be the same distance projected to the $X$ and $X^{\prime}$ subspaces. Further, let $n_{x}(i)$ (or $n_{x^{\prime}}(i)$ be the number of points with distance to $x_{i}$ (or $x_{i}^{\prime}$) less than $\epsilon_{x}(i)/2$ (or $\epsilon_{x^{\prime}}(i)/2$). Then, the MI of $X$ and $X^{\prime}$ is quantified by:

where $\psi$ is the digamma function, $k$ is a hyper-parameter, and $N$ is the number of points in $X$ (or $X^{\prime}$). We use Eqn (7) to measure the proportion of semantic features retained in the representations of poisoning inputs compared to their corresponding clean inputs. In the implementation, we set $N=2,000$ and $k=5$.

Under the supervised setting, the backdoor attack is implemented as optimizing the following objective function:

where ${\mathcal{L}}$ and ${\mathcal{L}}^{\star}$ denote the loss function defined in Eqn (4), measured on clean data ${\mathcal{D}}$ and poisoning data ${\mathcal{D}}^{\star}$ respectively, and the hyper-parameter $\lambda$ balances the influence of ${\mathcal{D}}$ and ${\mathcal{D}}^{\star}$ (e.g., via controlling $|{\mathcal{D}}^{\star}|/|{\mathcal{D}}|$). Intuitively, the first and second terms represent the benign task (i.e., learning semantic features) and backdoor task (i.e., learning trigger features), respectively. We hypothesize that in supervised attacks, if the trigger feature is sufficiently evident, the learning of benign and backdoor tasks tends to occur independently at the data level; that is, it focuses on learning the backdoor task from the poisoning data and focuses on learning the benign task from the clean data.

To validate this hypothesis, we assess the effect of trigger strength on the ASR, ACC, and MI of supervised backdoor attacks, with results shown in Figure 6. Observe that as the trigger strength varies from 0 to 1K, the ASR of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ increases rapidly and remains around 100%, while its MI quickly drops and stays at the minimum level. This observation suggests that in successful attacks, the model focuses on learning trigger features (backdoor task) only from the poisoning data. Meanwhile, the trigger strength has little impact on the ACC, indicating that the model effectively learns the benign task from the clean data. Similar trends are also observed on other supervised attacks in § 7.1.

The independence of benign and backdoor tasks in supervised attacks explains the empirical observations in § 4. In terms of learning dynamics, as the trigger feature (e.g., a specific image patch) is often simpler than the varying semantic features, the model learns the backdoor task much faster than the benign task. In terms of feature distributions, as Eqn (8) does not specify any constraints on the latent representations of poisoning and clean data, although associated with the same class, the representations of poisoning and target-class data are not necessarily proximate in the feature space.

Meanwhile, under the contrastive setting, the backdoor attack is implemented as the following objective function:

where ${\mathcal{L}}_{\mathrm{ctr}}$ and ${\mathcal{L}}_{\mathrm{ctr}}^{\star}$ are the contrastive loss defined in Eqn (1), measured on ${\mathcal{D}}$ and ${\mathcal{D}}_{\star}$ respectively. Because there are no labels, contrastive attacks manipulate the distributions of poisoning and target-class data in the feature space. To do so, the model must learn both trigger and semantic features from the poisoning data, so that the semantic features intertwine poisoning data with target-class data, while the trigger features connect all poisoning data.

To confirm our analysis, we assess the impact of trigger strength on the ASR, ACC, and MI of contrastive backdoor attacks, with results shown in Figure 6. Across all the CL methods, as the trigger strength varies from 0 to 1K, the ASR of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ first increases to 100% and then drops quickly, while its MI gradually decreases during the process. The finding suggests that the optimal attack is attained only if the semantic features are partially retained in the representations of poisoning data. This can be explained by the fact that if the trigger features are insignificant, the semantic features dominate the poisoning data, resulting in weak connections among poisoning data; meanwhile, if the trigger features dominate the poisoning data, the weak semantic features negatively impact the entanglement between poisoning and target-class data. To validate this explanation, we further measure the entanglement between poisoning and target-class data under varying trigger strength. As shown in Figure 7, the entanglement ratio is not a monotonic function of the trigger strength but follows a similar trend as the ASRs in Figure 6. We can thus conclude:

This interdependence between benign and backdoor tasks in contrastive backdoor attacks easily explains the empirical observations in § 4. In terms of learning dynamics, because the model learns both benign and backdoor tasks (the latter is simpler than the former) from poisoning data, the contrastive loss of poisoning data tends to decrease at a rate similar to clean data. In terms of feature distributions, as the model extracts both trigger and semantic features from poisoning data, it naturally intertwines poisoning and target-class data in the feature space.

We first examine defenses premised on the difference between the learning dynamics of benign and backdoor tasks.

Anti-Backdoor Learning (ABL)[28] is a filtering defense that segregates poisoning data and unlearns backdoored models. Intuitively, as the backdoor task is simpler than the benign task, assuming the loss of poisoning data drops quickly in early training epochs while the loss of clean data decreases at a steady pace, ABL detects poisoning data and unlearns the backdoored model by applying gradient descent on the clean data and gradient ascent on the detected poisoning data.

Despite its effectiveness against supervised attacks, it is challenging to extend ABL to CL because the learning dynamics of poisoning data in contrastive attacks are much less distinctive (cf. § 4.2). We empirically validate this by applying ABL in detecting the poisoning data of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ and ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ under the same setting as in § 4.1. We follow the hyper-parameter setting in[28]. Specifically, if the loss of any training input goes below a threshold $\gamma=0.5$, we activate gradient ascent to boost its loss to $\gamma$. In the CL setting, we set $\gamma$ according to a fixed ratio compared to the largest loss drop: $(\gamma-\ell_{\text{final}})/(\ell_{\text{init}}-\ell_{\text{final}})$, where $\ell_{\text{init}}$ and $\ell_{\text{final}}$ respectively denote the loss of the training input before and after training. Table 3 reports ABL’s performance under the poisoning rate fixed as 1%. The false positive rate (FPR) measures the fraction of clean inputs falsely detected as poisoning, the true positive rate (TPR) measures the fraction of poisoning inputs correctly identified, and the isolation rate denotes the percentage of samples isolated by ABL. A lower FPR or a higher TPR indicates more accurate detection.

Observe that ABL effectively detects the poisoning data of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ but is much less effective against ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$. For example, with 1% isolation rate, ABL detects 92.8% of the poisoning data of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$, which drops to 2.4% against ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$. The increase of TPR against ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ with the isolation rate is mainly attributed to the low poisoning rate (1%). We thus conclude that ABL is ineffective against contrastive attacks, which confirms the observations in § 4.2. This is due to the intertwined learning dynamics of both benign and backdoor tasks. We defer the evaluation of ABL against ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL/CL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$ to § D.2.

The entanglement between the representations of poisoning and clean data also causes challenges for defenses that rely on the separability of poisoning data in the feature space.

Activation clustering (AC)[31] is a data inspection method. It trains the model using the potentially poisoning data and collects the penultimate-layer activation of each input. AC assumes the poisoning data in the target class forms a separate cluster that is either small or far from the class center. It identifies the target class by calculating the silhouette score of each class, with a higher score indicating a stronger fit to two clusters. Additionally, since the attacker is assumed to be unable to poison more than half of the data, the smaller cluster is considered to be the poisoning data. Assuming the data labels are available, we evaluate the effectiveness of AC against both supervised and contrastive attacks. We set the hyper-parameters according to the original paper.

Figure 8 shows that AC fails to identify the target class (class 0), which has a lower score compared to other classes (e.g., class 5), not to mention detecting the poisoning data. This may be attributed to the tight entanglement between the representations of poisoning and clean data.

Statistical contamination analyzer (SCAn)[33] detects poisoning data based on the statistical properties of its representations. Specifically, it applies an EM algorithm to decompose an input into two terms: identity and variance; it then analyzes the representations in each class and identifies the target class most likely to be characterized by a mixture model. To evaluate SCAn against contrastive attacks, following[33], we use 1,000 inputs randomly sampled from the testing set to build the decomposition model, which we apply to analyze 5,000 poisoning and 5,000 clean inputs. For comparison, we also evaluate SCAn against the supervised attack ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun/uni}}}$. Similar to STRIP[38], we vary the FPR under three different FPR settings: 0.5%, 1.0%, and 2.0%, and evaluate the TPR.

As shown in Table 4, compared with the supervised attack, SCAn is much less effective, especially for ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$. For instance, SCAn detects over 63.0% of the poisoning data of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$, which drops to 28.0% against ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$. This can be attributed to the entanglement between the representations of poisoning and clean data, while SCAn relies on the separability of poisoning data in the feature space.

Fine-pruning (FP)[39] assumes poisoning and clean data activate different neurons and sanitizes backdoored models by pruning neurons that are dormant with respect to clean data. Specifically, it measures the average activation of each neuron with respect to clean data from a validation set and prunes the set of least active neurons. Following[39], we apply FP on the penultimate layer of the backdoored model. In[39], it keeps pruning the model until the tolerance of accuracy (5%) reduction is reached. For a more detailed analysis, we show the ASR and ACC with respect to the variation of the pruned channels from 0 to 500.

Figure 9 shows the ASR and ACC of each attack as a function of the pruning rate of FP. Observe that FP effectively defends against supervised attacks. By pruning about 240 neurons, the ASR of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ is reduced to zero while incurring an acceptable ACC drop. Meanwhile, the ASR and ACC of contrastive attacks decrease at a similar pace. To attain effective defenses, it is necessary to prune almost all the neurons. This further suggests the tight entanglement between poisoning and clean data in the feature space.

Thus far, we have demonstrated the challenges of defending against contrastive backdoor attacks based on either learning dynamics or feature distributions. Next, we consider a more diverse set of defenses applied to the end-to-end model $h=g\circ f$ that comprises the backdoored encoder $f$ and a classifier $g$ in the downstream task.

NeuralCleanse (NC)[40] is a model inspection method to detect backdoors. Specifically, for each class, NC reverse-engineers the “minimal” trigger (measured by the $L_{1}$-norm of its binary mask) required to misclassify clean inputs from all other classes into this class. It then runs outlier detection and marks any class with a trigger significantly smaller than the other classes as an outlier and infected. We evaluate NC’s efficacy in both supervised and contrastive learning. In the contrastive setting, an end-to-end model, in which the encoder trained by ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ or ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$ using SimCLR, is fine-tuned using clean data. In the supervised setting, the model is directly trained by ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ or ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$.

As shown in Table 5, NC failed to detect any backdoors in the contrastive setting. For example, for ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$, the anomaly index of the target class varies around 0.72, much lower than the detection threshold of 2. This is attributed to the presence of functional triggers and aggressive data augmentation methods, which make it challenging for NC to reverse-engineer a stable trigger. In contrast, NC successfully detects backdoors against supervised attacks, especially ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$.

STRIP[38] is an inference-time data inspection method. Specifically, it assumes that the prediction of poisoning data is insensitive to perturbation due to the dominance of trigger features, whereas the prediction of clean data may vary greatly with perturbation. For an incoming input, STRIP intentionally perturbs it (e.g., superimposing various image patterns) and utilizes the randomness of its prediction to determine whether it is triggered. As it is a run-time detection method, we use an end-to-end model (consisting of the backdoored encoder and downstream classifier) to evaluate its effectiveness. Following[38], we first estimate the entropy distribution of clean inputs on a validation set, select an entropy detection threshold with a given FPR (0.5%, 1.0%, and 2.0%), and remove all training inputs with entropy below this threshold.

As shown in Table 6, STRIP is ineffective in detecting the poisoning data of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$. For instance, with FRR fixed as 1.0%, the TPR of STRIP is as low as 1.1%, suggesting that STRIP fails to distinguish poisoning and clean data. This may also be explained by the entanglement effect: as the poisoning and clean data share similar representations, the perturbation to them leads to similar measures. The results on ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL/SL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$ are deferred to § D.3.

Backdoor defense via decoupling (BDD) [41] is designed to mitigate the impact of poisoning inputs that tend to cluster together in the feature space of compromised DNNs. This approach consists of three stages: (1) the backbone is trained in a self-supervised manner, which aligns inputs with the same ground-truth label in the feature space; (2) the remaining classifier is trained in a standard manner, using all labeled training inputs; and (3) low-confidence inputs are removed based on the end-to-end model.

Clearly, (1) and (2) share the same settings as the threat model of Trl. Thus, we focus on (3) to evaluate the effectiveness of BDD. The distributions of the classification confidence of poisoning and clean (from the target class) inputs are shown in Figure 10. Note that poisoning inputs not only have similar classification confidence as clean inputs but also exhibit lower variance. This makes it more challenging to establish a confidence threshold to distinguish between them. Consequently, using BDD mechanisms to defend against Trl tends to be ineffective.

Fine-tuning[42] is a method to adapt a pre-trained encoder to a specific downstream task by fine-tuning a classifier. Prior work[42] shows that fine-tuning may mitigate backdoor attacks to a certain extent. In the previous evaluation, following[14], we focus on scenarios where the downstream and pre-training datasets are identical. Here, we explore cases in which these datasets differ, such as pre-training the encoder on CIFAR10 and fine-tuning the classifier on CIFAR100. Since the datasets may have varied class structures, we focus on untargeted attacks and measure the attack effectiveness by the model’s accuracy drop in classifying trigger inputs compared to clean inputs.

Table 7 summarizes the results. Observe that the contrastive backdoor attack substantially affects the model’s performance, regardless of the setting of pre-training and downstream datasets. For instance, when the encoder is pre-trained on CIFAR100 using SimCLR and then adapted to CIFAR10, the model achieves 53.3% accuracy and 18.1% accuracy (i.e., 35.2% accuracy drop) on clean and trigger inputs, respectively. This observation indicates that fine-tuning using downstream datasets may not mitigate contrastive backdoor attacks. Additionally, we observe that most trigger inputs are misclassified into a single incorrect class. We speculate that these trigger inputs are clustered together and located close to a specific class in the feature space, which is also close to the target class in the pre-training dataset.

We explore whether our findings are influenced by other factors, including model architectures and trigger definitions.

Model Architectures – To evaluate the influence of backbone model architectures, besides ResNet18, we also consider three other popular DNN architectures (inlcuding ResNet50[27], ShuffleNet[43], and MobileNet[44]) and train a SimCLR encoder on CIFAR10 as outlined in § 4.

Figure 11 shows the learning dynamics and entanglement ratio under varying backbone architectures. Our results indicate that: (i) the loss-decreasing pace is comparable for both poisoning and clean inputs across different architectures, and (ii) the poisoning inputs and the clean target-class inputs are entangled in the feature space under all the settings. The findings suggest that the backbone architectures have a limited impact on contrastive backdoor attacks.

Alternative triggers – In the previous evaluation, for contrastive attacks, we primarily focus on the functional trigger ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$. Here, we investigate whether the trigger definition impacts our findings. Specifically, we consider the universal trigger ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$ as an alternative trigger definition. Following[14], we specify a 5$\times$5 image patch as the trigger pattern, which is randomly placed at a random location of a given input.

Figure 12 shows the learning dyanmics of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$. Similar to ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ (cf. Figure 2), ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$ also exhibits distinct learning dynamics compared with supervised attacks. Specifically, the contrastive loss of both poisoning and clean data decreases gradually at fairly similar paces. Further, Figure 13 evaluates the entanglement ratios of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL/CL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$. Observe that similar to Figure 4, the entanglement effect is also more evident in the contrastive setting. Thus, the difference in entanglement effect is rather trigger-agnostic but stems from the underlying learning paradigms.

Further, we conduct experiments to examine the influence of trigger strength (measured by the image patch size) on the effectiveness of supervised and contrastive attacks. Figure 14 shows the ASR and ACC of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$ and ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{dyn}}}$ as a function of trigger strength, which complements the results of Figure 6. Observe that in both cases of universal and dynamic triggers, as the trigger strength increases, the ASR of supervised attacks quickly peaks and remains around 100%; meanwhile, the trigger strength has little impact on the ACC, indicating the independence of backdoor and benign tasks at the data level for supervised attacks.

In contrast, as shown in Table 8, across different CL methods, as the trigger size increases from 1 to 20, the effectiveness of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{uni}}}$ initially increases and then decreases abruptly, similar to the trend observed about ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ in Figure 6.

We demonstrate that due to the specificities of contrastive backdoor attacks, defenses that attempt to segregate poisoning data based on learning dynamics and feature separability tend to fail. Given such limitations, we explore alternative defense strategies and discuss their potential challenges to defend against contrastive attacks.

Density-based Filtering – We have an interesting observation in our evaluation: the trigger inputs tend to form a cluster in the feature space, which is often much denser than the clusters formed by clean inputs. For example, we use a SimCLR model trained on CIFAR10 and measure the average pairwise $L_{2}$ distance between inputs in the feature space. Figure 15 shows the normalized intra-class and inter-class distance. Observe that trigger inputs (class 10, with normalized distance 0.0) tend to cluster much more tightly compared with clean inputs (class 0 - 9, with normalized distance $\geq$ 0.56).

Motivated by this observation, we propose a density-based filtering defense. After training the encoder on potentially poisoning data, we use it to generate features for all inputs. Next, we apply a density-based clustering method (e.g., OPTICS[45] or DBSCAN[46]) to cluster the training inputs based on their features. The densest clusters are identified as potentially poisoning data and subsequently removed from the training data.

Below we evaluate this approach in defending against ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ (SimCLR) on CIFAR10 under two distinct poisoning rates: 1% and 5%. Specifically, we employ DBSCAN as the clustering algorithm to sift out poisoning inputs. DBSCAN hinges on two key parameters: (i) Minimum samples $N_{\mathrm{min}}$, which is the minimum number of inputs required to form a cluster. Intuitively, a larger $N_{\mathrm{min}}$ results in denser clusters. Given that the number of poisoning inputs is typically small, we set $N_{\mathrm{min}}$ to either 30 or 100 in the evaluation. (ii) Maximum distance $\epsilon$, which thresholds the distance between two inputs to determine whether they belong to the same cluster. To determine $\epsilon$, we calculate the average distance between each input and its $K=N_{\mathrm{min}}$ nearest neighbors, plot the average $K$-distances in ascending order on a $K$-distance graph (as shown in Figure 16), and set $\epsilon$ as the point of maximum curvature, where the graph has the largest decreasing rate. We set $\epsilon=0.3$ in the evaluation.

Table 9 summarizes the effectiveness of this density-based filtering measured by true positive rate (TPR) and false positive rate (FPR), as well as the attack success rate (ASR) and clean accuracy (ACC) of the model trained on the post-filtering training data. Notably, density-based filtering effectively sifts out a majority of poisoning inputs across various settings. For instance, under the setting of 5% poisoning rate, $\epsilon=0.3$, and $N_{\mathrm{min}}=30$, it successfully filters 98.8% of the poisoning inputs. Further, observe that after filtering potentially poisoning data and retraining the model, we may not only effectively mitigate the attack but also retain the model’s utility. For instance, it achieves 13.3% ASR and 80.1% ACC under 1% poisoning rate and $N_{\mathrm{min}}=30$. However, the effectiveness of re-training seems sensitive to the parameter setting. For instance, under 1% poisoning rate and $N_{\mathrm{min}}=100$, as it fails to filter out approximately 100 poisoning inputs, the attack is not mitigated effectively.

We further evaluate the effectiveness of density-based filtering with OPTICS as the underlying clustering algorithm, which is less sensitive to the parameter setting. It achieves an impressive 99.6% TPR and 0% FPR under 1% poisoning rate. However, it has a much larger execution overhead. For example, on our platform, OPTICS requires 1.5 hours to complete the filtering, whereas DBSCAN finishes in approximately 10 seconds. Thus, we consider enhancing both the effectiveness and efficiency of density-based filtering defense as our ongoing work.

Data-free pruning – In addition to filtering potentially poisoning data, we also explore in-depth defenses for scenarios where filtering proves ineffective. We have shown that data-dependent pruning[39] is insufficient to mitigate contrastive backdoor attacks, due to the indistinguishable feature patterns of clean and poisoning data (§ 6.2). Surprisingly, data-free pruning, which is believed to be less effective than data-dependent pruning, shows promising performance against contrastive backdoor attacks.

Specifically, we consider channel Lipschitzness-based pruning (CLP)[47], a data-free defense to remove backdoors. It exploits the observation that a subset of channels is more sensitive to trigger features compared to other channels, while the sensitivity of each channel can be estimated by the upper bound of its channel Lipschitz constant (UCLC), which can be computed in a data-free manner. Thus, given the mean ($\mu_{k}$) and variance ($\sigma_{k}$) of the UCLCs in the $k$-th convolutional layer, CLP prunes channels with UCLCs larger than $\mu_{k}+u\sigma_{k}$, where $u$ is a hyper-parameter.

We apply CLP on models backdoored by ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ and ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ with varying $u$. As shown in Figure 17, by properly setting $u$, it is possible to effectively reduce the ASR of ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ with limited ACC drop (about 5%). However, compared with ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{SL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$, the proper range of $u$ for ${\textsc{Trl}}_{\scaleobj{0.8}{\mathrm{CL}}}^{\scaleobj{0.8}{\mathrm{fun}}}$ is much narrower ([2.2, 3.0] versus [1.8, 6.5]). To address this challenge, we propose to empirically set $u$ as the knee point of the ACC curve. Specifically, we measure the ACC under varying $u$, apply the Savitzky–Golay filter to smooth the curve[48], and identify the knee point of the smoothed curve as the optimal $u$ during pruning. By applying this approach, we identify the optimal $u$ as 2.4 in the contrastive setting, as indicated by the green dashed line in Figure 17, which well balances utility and defensive efficacy (80.4% ACC and 14.3% ASR).

Ensemble defenses – We further combine the previous two defenses to form a powerful ensemble defense against contrastive backdoor attacks. Specifically, we first apply density-based filtering to filter potentially poisoning data, retrain the model on the post-filtering training data, and then apply data-free pruning on the retrained model. We simulate the setting that the filtering hyperparameter is not properly set (e.g., $N_{\mathrm{min}}=100$ under 1% poisoning rate), which results in a retrained model with 43.8% ASR and 81.8% ACC. By applying data-free pruning on this retrained model, as illustrated in Figure 18, we further reduce ASR to 12.6% while retaining ACC around 79.6%. We believe this ensemble defense represents a promising direction worthy of further research.

We further discuss the limitations of this work and point to several future directions. (i) Key properties: our study primarily focuses on the differences between supervised and contrastive backdoor attacks, as reflected in learning dynamics and feature distributions. Other properties (e.g., neuron activation patterns) might also vary across these attack types. Future research could explore these underexplored properties to develop more effective defenses. (ii) Alternative defenses: our study highlights the defense implications of key differences between supervised and contrastive backdoor attacks. We illustrate the challenges faced by several representative defenses against contrastive backdoor attacks. Comprehensively investigating and benchmarking all the SOTA defenses (e.g.,[49, 50]) against contrastive backdoor attacks is a valuable avenue for future research. (iii) Data modality: while our study mainly focuses on the vision domain, contrastive learning has been extended to other domains such as natural language processing and multimodal learning[51, 52]. It is worth investigating how domain constraints (e.g., discrete perturbation, semantic preservation, and interference between different modalities) affect contrastive backdoor attacks and their potential implications. (iv) Generalization to self-supervised learning (SSL): while focusing on contrastive learning methods, our findings might extend to the wider SSL paradigm (e.g., masked autoencoder[53]). It is essential, however, to validate these implications across broader contexts. Future work should explore the applicability and adaptability of our insights to alternative SSL frameworks and examine the implications for defenses in these new settings.

In summary, while limited in several aspects, this work represents a solid step towards understanding and defending against the emerging threat of backdoor attacks in contrastive learning. Our findings highlight the need for defenses tailored to the specificities of contrastive backdoor attacks.