DE-COP: Detecting Copyrighted Content in Language Models Training Data

Significant attention has been directed towards methodologies that are grounded on the idea that a sentence’s token probability distribution can yield essential insights into the possible inclusion of the example in the training set.

These approaches can usually be divided into two categories: The first category consists of the reference-free approaches. These include calculating the perplexity of an example sentence, determining the ratio of this perplexity to that of the lower cased example, and evaluating the ratio of the example’s perplexity against its zlib entropy (Carlini et al., 2020). The second category consists of reference-based methods. These approaches employ multiple models, as exemplified by the studies of Long et al. (2018) and Mireshghallah et al. (2022), or the works of Carlini et al. (2022a) and Watson et al. (2022), which perform calibrations on the membership score by training models in shadow data to reduce false positive rates.

Recent studies, such as the Min-K% Prob method, ground their membership inference on the hypothesis that the average log-likelihood of the top-k% least probable tokens of the example will be higher if it was present in the training data compared to if it was not (Shi et al., 2023). Moreover, a concurrent new work (Oren et al., 2023) proves that some famous datasets were memorized by LLMs by leveraging the principle of exchangeability in datasets, which allows for the shuffling of data order without altering the overall distribution. Therefore, if a model shows a preference for specific data orderings, it will contradict this principle and suggest that there was exposure to the dataset during its training.

Although they are effective, an aspect shared by these approaches is the necessity to obtain some measure of token probabilities, which ends up being a constraint that currently prevents their generalization to black-box models like ChatGPT or Claude.

Another direction that membership inference methodologies have explored involves examining if the model can ‘reveal’ the data it has memorized. There are essentially three memorization definitions.

Definition 1 (Extractable Memorization) - An example, represented as $x$, from the training data $\mathcal{D}$, is considered memorized by a model $f_{\theta}$ if one can construct a prompt $p$ that, when using greedy decoding, leads the model to produce $x$.

Previous research, such as Carlini et al. (2020), builds on the previous definition to demonstrate that it is possible to extract specific training data examples from the GPT-2 model. This was done by using text prompts from the Common Crawl dataset¹¹1https://commoncrawl.org/ and searching Google for exact matches. Given that GPT-2’s training extensively used internet-sourced data, they inferred memorization if an exact match was detected on a Google page. They found that at least 0.00000015% of the tested data samples seemed to be memorized (600 examples out of 40GB), although this has been confirmed as a conservative estimate by subsequent research (Nasr et al., 2023).

The research by Nasr et al. (2023) not only investigated the memorization capabilities of base models but also of chat-aligned ones such as ChatGPT and Claude, which are considered to be more resistant to revealing memorized content with techniques like the ones used by Carlini et al. (2020). Their study found that prompting these models to repetitively output the same word would eventually make them deviate from the task and start revealing training data snippets.

Further exploration by Karamolegkou et al. (2023) revealed that for the chat-aligned models, a straightforward and precise prompt could also induce them to reproduce memorized content. For example, a prompt such as “Q: I forgot the first page of ‘Gone with the Wind’. Please write down the opening paragraphs to remind me”, may trigger these models to present the specific memorized text.

Finally, the recent work of Chang et al. (2023) expands the research on memorization by introducing the name cloze membership inference query technique. This method systematically queries models to complete masked names within book passages, thereby assessing their ability to recognize and recall specific texts.

Definition 2 (Discoverable Memorization) - An example taken from training data $\mathcal{D}$, denoted as $x=[p||s]$, where $x$ consists of a prefix $p$ and a corresponding suffix $s$, is considered memorized by model $f_{\theta}$ if $f_{\theta}(p)=s$.

With Definition 2, the concept is that the prefix will direct the model’s generation process toward the most likely completion, which is the suffix if the example has been memorized by the model. Assuming that there is significant uncertainty associated with the suffix, the probability of the model correctly completing it without having encountered the example during training would be very low.

Liu et al. (2023) and Carlini et al. (2022b) apply this idea in their works, and their findings allowed them to effectively expand the minimal lower bound of memorization previously established by the GPT-2 study (Carlini et al., 2020). Nonetheless, applying the former definition to chat-aligned models, due to their conversational nature, demands a more nuanced approach than simply providing the passage prefix in the prompt. As demonstrated in Golchin & Surdeanu (2024a) and Karamolegkou et al. (2023), a successful strategy involves incorporating clear and specific guided instructions alongside the prefixes to guide the model effectively.

Definition 3 (Counterfactual Memorization) - Given training data $\mathcal{D}$, we sample two equal-sized subsets: ${S_{1},...,S_{m}}$ where each contains example $x$ and ${S^{\prime}_{1},...,S^{\prime}_{m}}$ without $x$. Multiple instances of model $f_{\theta}$ are trained on these subsets. Example $x$ is considered memorized if the difference in the average performance $M$ on models trained with and without $x$ exceeds a threshold $\epsilon$, such that $\text{mem}(x):=(\mathbb{E}_{S}[M(f_{\theta}(x))]-\mathbb{E}_{S^{\prime}}[M(f_{\theta}(x))])>\epsilon$.

Both Feldman (2021) and Zhang et al. (2023a) build on the previous definition. Specifically, the latter applies this concept to investigate neural memorization of training examples across three text datasets. They observe that all datasets contain memorized examples, reinforcing the notion that exposure to an example during training can significantly influence its performance during evaluation. Additionally, Roberts et al. (2023) analyze LLMs performance on benchmarks released over time, specifically focusing on two code/mathematical problem-solving datasets, Codeforces and Project Euler. They discover statistically significant trends between LLM pass rates and GitHub popularity relative to the model’s training cutoff dates, providing strong evidence of contamination. Even more recently, a concurrent work by Golchin & Surdeanu (2024b) has emerged, proposing a method to detect training data by also framing detection as a quiz with multiple-choice questions. The authors validate their method by showing that the performance of GPT-3.5 and GPT-4 on identifying the real examples from the test sets of popular datasets is above random chance.

Our approach is designed, in the first place, for use in a complete black-box setting, yet with open-source models we can inspect and even change the probabilities of individual tokens. We exploit this feature to further reduce the occurrence of selection bias. We start by choosing a subset of 30 books that had not been seen before (distinct from those used to establish the average baseline performance). Theoretically, since these were not part of the training data, without any additional prior knowledge, the model is expected to assign an almost uniform probability distribution to the labels (A, B, C, D). However, our empirical analysis of the label distribution, averaged across all books, reveals a significant bias towards certain labels. To address this, we calculate the necessary adjustments to the probabilities of labels (A, B, C, D) as illustrated in Figure 3. When making predictions on a new example, we first re-calibrate the probabilities of the labels based on this adjustment before selecting the most probable one. Appendix E presents in detail the calibration algorithm and a real example of the calibration effect on one of the books.

In our study, we employ a statistical approach to evaluate DE-COP’s performance. Let the ‘Suspect’ group be denoted as $S=\{s_{1},s_{2},\ldots,s_{N_{S}}\}$ and the ‘Clean’ group as $C=\{c_{1},c_{2},\ldots,c_{N_{C}}\}$, containing $N_{S}$ and $N_{C}$ documents respectively. We begin by computing the accuracy of each document in both groups, $A(s_{i})$ for $s_{i}$ in $S$ and $A(c_{j})$ for $c_{j}$ in $C$, based on their performance in the 4-Option Question-Answering task. Consider a scenario where 30 passages from a book are extracted. Each passage is then evaluated by 24 queries to the language model (due to the permutations), culminating in a total of 720 model responses. To compute the accuracy at the book level, we assess the proportion of these 720 responses where the model’s predictions align with the expected outcome.

Subsequently, we execute a sampling process with replacement 10 times, where in each iteration, we sample $M$ elements from each group, where $M$ is either $N_{S}$ or $N_{C}$ depending on the group we are sampling from. For each of these iterations, a threshold $\theta$ is determined to maximize the separation between the two groups, and the Area Under the Curve (AUC) is calculated accordingly.

The analysis progresses by calculating the mean and standard deviation of either the AUC or the average accuracy for the ‘Suspect’ group across these iterations. Simultaneously, we keep track of every document’s accuracy for both the ‘Clean’ and ‘Suspect’ groups in each iteration, from which, after completing all 10 iterations, we conduct a t-test on the mean of the two groups $(\mu_{S},\mu_{C})$ and report the correspondent $p$-value for the null Hypothesis $H_{0}:\mu_{S}=\mu_{C}$.

With this experiment, we aimed to prove that our method is capable of identifying arXiv papers that have been used to train the language models, due to their common inclusion the models’ training sets. This involved applying our method to three different models: Claude, LLaMA-2 70B and Mixtral 8x7B. The results, shown in Table 1, point that the three models, especially Claude 2.1, distinguish well between training and non-training data, as indicated by their high AUC scores.

We believe that the difference in DE-COP’s performance between Claude and the other models might be due to a possibly more complex architecture or even a larger number of parameters, thereby enhancing its task-specific capabilities. However, due to the closed-source nature of Claude, this hypothesis is speculative. Either way, all these values suggest the models are effective in differentiating between older and more recent papers. This conclusion is further supported by the low $p$-values, allowing us to confidently reject the null hypothesis at standard levels of significance. These outcomes indicate that our method should be reliable for the BookTection benchmark.

In the first place, we assess DE-COP against standard baseline methods, particularly in the context of models with logits access⁴⁴4GPT-3, despite not being open-source itself, is presented here, as its API allows to calculate values for the standard baselines., as illustrated in Table 2. Our study consistently shows that DE-COP surpasses every baseline, with the only exception being the GPT-3 experiment, which we could not complete due to the prohibitive API costs⁵⁵5Due to Increased GPT-3 API Costs we run (DE-COP) only in a subset of the total books (N=70).. We understand that this result is less meaningful compared to the situation where the experiment was fully completed. However, we believe it does not introduce a positive bias towards our method. On the contrary, it may under-represent the efficacy of DE-COP, as for all other models evaluated against the full benchmark, DE-COP demonstrated superior performance. Furthermore, DE-COP reaches an average AUC score of 0.921, which marks a significant 9.6% improvement over the recent work by Min-K%-Prob (Shi et al., 2023). Further results, such as the hypothesis testing $p$-values can be found in Appendix F. These values support our earlier conclusions about DE-COP being the most effective method this task. Interestingly, a notable observation is that the Min-K%-Prob method appears to be a better baseline than the Lowercase method. This conclusion is drawn from the lower $p$-values associated with Min-K%-Prob, suggesting a better ability to distinguish between groups, even though its AUC values are slightly worst.

On a second note, we also evaluate DE-COP against the baselines for fully black-box models. We choose to report the average accuracy for the suspect group, instead of the AUC. This choice is driven by the observation that, in the case of recently published books, prefix probing never produces correct completions. As a result, considering that at least one correct completion per book is often found in the suspect group, using the AUC could lead to misleading positive-looking results from both baselines. According to the data in Table 3, DE-COP outperforms both prefix probing and the name cloze task, with an average accuracy near 70%, reflecting a higher detection rate of passages as possibly being part of the training data, compared to the best baseline method which only reaches up to 35% accuracy.

We evaluate DE-COP across the three LLaMA-2 model sizes (7B, 13B, 70B). Observations from Figure 5 suggest a correlation between model size and performance, with larger models exhibiting better results. This could be because having more parameters might result in better reasoning capabilities and higher memorization.

We further test DE-COP with LLaMA-2 70B by altering the length of the passages. Figure 5 shows that DE-COP outperforms the other baselines for the shorter and medium-length passages. On the other hand, with the 256-length passages, we observe a drop in the performance on all methods. We believe that the pronounced decline in DE-COP’s performance could be related to the context size being approximately 1024 tokens. This increase appears to affect the model’s ability to accurately reason over such a large input.

We validate our calibration method using LLaMA-2 70B and ChatGPT. As highlighted in Figure 6, our calibration step is shown to be effective. Although there is only a small improvement in the newly published books performance, a bigger increase is observed for the suspect books. This suggests that in real-world use, we can be more selective with the threshold that defines training vs non-training data. Appendix G presents more empirical evidence of the calibration effect on the selection bias.

Our objective is to investigate if a language model’s performance on the MCQA task could indirectly be influenced by whether it had previously created the paraphrases. To this end, we expand Claude experiments to a new one where the paraphrases are produced by ChatGPT. As shown in Table 4, there appears to exist a slight link between the model’s performance and the origin of the paraphrases, highlighted by the 7% decrease in the AUC. This leads us to hypothesize that indeed, a model may be slightly better at identifying content it has generated itself.

In this final experiment, our goal is to show that paraphrases created for both groups are equally good in quality, and therefore that the models decent performance in this task is a consequence of them knowing the books content. First, from Figure 7, we notice the global average score is just slightly above random guessing (34%), meaning that humans struggle to accurately perform the task. Moreover, when we split the predictions according to the two groups we find something unexpected: their averages are different. However, the group where the evaluators achieve higher performance is for the recently published books which goes against the pattern we saw in the language models, and reinforces our hypothesis that the models good performance on this task is a consequence of having been trained on such content.