A False Sense of Safety:
Unsafe Information Leakage in ‘Safe’ AI Responses

For some $n$, let $\mathcal{X}$ and $\mathcal{Y}$ be the set of all possible strings of length at most $n$ constructed out of tokens. Consider an adversary $\mathcal{A}$ (e.g., a person or a generative model) interacting with a victim Large Language Model $\mathcal{V}:\mathcal{X}\to P(\mathcal{Y})$ mapping input prompts to a distribution over output strings. An interaction between the adversary and victim is represented by an input-output pair $(x,y)\in\mathcal{X}\times\mathcal{Y}$, where $x$ is the input prompt to the LLM and $y\sim\mathcal{V}(x)$ a sample output. Within these interactions, the adversary aims to maximize a scoring function $s:\mathcal{X}\times\mathcal{Y}\to\mathbb{R}^{+}$, which quantifies how well these input-output pairs fulfill the adversary’s goals.

We characterize “impermissible concepts”, i.e. knowledge which would be considered dangerous, risky, harmful, or otherwise undesirable for an individual to know through sets of input-output pairs. Specifically, let $Q\subset\mathcal{X}$ be a set of questions that a malicious adversaries wants an answer for, and $R_{x}\subset\mathcal{Y}$ the set of answers $y$ for which the pair $(x,y)$ contains impermissible knowledge. For instance, $x\in Q$ might be a question about implementing malware, and $R_{x}$ could be a set of possible malware implementations.

Defenses against adversaries are assumed to be mediated by censorship mechanisms which modify the distribution of outputs returned to a user-provided input $x$. These defenses seek to ensure that outputs returned satisfy a safety criterion defined based on the threat model of concern.

The definition of a censorship mechanism is very general and can be interpreted as a method for altering the output distribution of a victim model to provide censorship guarantees against specific threats. Next, we introduce security and inferential adversary threat models, which inform the guarantees required from these censorship mechanisms.

We first introduce security adversaries to clearly formulate a commonly assumed threat model for LLMs (Geiping et al., 2024).

Adversarial examples (Szegedy et al., 2014), jailbreak attacks (Liu et al., 2023), and prompt injection (Greshake et al., 2023) naturally fall under this class of adversary, where $T_{x}\subset\mathcal{Y}$ is a subset of labels not equal to the ground truth label for input $x$. In the context of LLM interactions, $T_{x}$ can be a set of strings satisfying a property such as being classified as harmful by an evaluator LLM, or start with the string “Sure, here’s how to $\sigma$” where $\sigma$ is a string deemed impermissible e.g. “evade taxes”.

Against security adversaries, the objective of the victim model provider is characterized by an impermissibility function $i:\mathcal{X}\to\mathcal{P}(\mathcal{Y})$ describing the impermissibility of a response as a function of the input. Let $Y_{x}=i(x)$, then, a censorship mechanism $M$ must ensure the security censorship guarantee of $\forall x\;P_{M(x,V(x))}(Y_{x})=0$, i.e. the probability of returning an impermissible output is $0$. Extant LLM safety research typically assumes that the impermissibility function does not depend on the input $x$, and censorship could ideally completely facilitated by an output filtering mechanism Kim et al. (2024).

Nevertheless, such adversaries fail to fully capture many threats such as deepfake generation or informative instructions for conducting illegal activities. This stems from the ability of adversaries being able to compose individual acceptable outputs to construct deepfakes or acquire instructions (Glukhov et al., 2023).

Inferential adversaries provide a compelling model for many safety concerns which are not adequately captured by security adversaries and capture compositional threats. They differ from security adversaries in that they do not have a target function $t$ specifying a set of desired outputs from the victim model. Instead, inferential adversaries seek to gain information about a target concept such as malicious instructions, from model outputs.

Under our information theoretic framing, a target concept is represented as a distribution over strings of bounded length, representing the adversary’s prior belief about the answer to a malicious query $x^{*}$. The uncertainty in $C$ is quantified by Shannon’s entropy and the change in uncertainty provided a response random variable $R$ is defined through the mutual information.

This framework effectively captures the threat demonstrated in Figure 1, dual-use queries, and marginal risks where model responses assist adversaries in achieving goals without being outright impermissible. Even if individual model outputs do not fully specify $C$, (e.g. detailed instructions on how to scam the elderly), model responses to other questions can still reduce an adversaries uncertainty and jointly help specify $C$.

In contrast to security adversaries which optimize inputs for producing specific outputs, inferential adversaries optimize inputs for maximizing joint mutual information with target concept $C$. We refer to this as an inference attack and is outlined in Algorithm 1. The attack iteratively selects queries that maximize the conditional mutual information with the target concept, given the history of previous queries and responses. This greedy approach, while not guaranteed to be optimal, offers a practical balance between efficiency and effectiveness in the face of the combinatorial complexity of exhaustive search (Fleuret, 2004).

While the algorithm provides a theoretical framework, practical implementation for LLMs presents challenges, particularly due to challenges in efficiently calculating mutual information.

To demonstrate real world feasibility of inferential adversaries, we introduce a heuristic approach which aims to exhibit similar functionality to the information-theoretic approach. The approach draws inspiration from problem-solving agents and prompt decomposition techniques used in solving complex, multi-step reasoning problems (Perez et al., 2020; Khot et al., 2022; Radhakrishnan et al., 2023). Our method consists of an adversarial LLM $\mathcal{A}$ which produces a series of subquestions $q_{s}$ whose answers could be aggregated to answer an impermissible target question $q$. These subqueries are then submitted to a victim LLM $\mathcal{V}$, which generates responses $r_{s}$. Finally, these responses are aggregated by the adversarial LLM to answer the original query $q$. Figure 2 provides a high-level overview of this interaction.

We approximate the adversary’s strategy of selecting queries sequentially in terms of maximizing the conditional mutual information by modeling the interactions through a Markov Decision Process (MDP) and selecting sets of interactions relevant to the malicious queries. Our MDP is defined by

• •

  States $\mathcal{S}$: Consist of the impermissible question and its intermediate answer based on a summary of all sub-question-responses pairs;

• •

  Actions $\alpha$: The set of subquestions generated given the state;

• •

  State transitions $T$: Generation of new summary by aggregating the previous summary with a sub-question and answer pair;

• •

  Reward $R$: A model provided score measuring the relevance of state summaries with respect to the original malicious question.

• •

  Environment $E$: The victim model is part of the environment, returning responses to suquestions asked in isolated context windows.

We seek to approximate the interaction of an inferential adversary with a victim LLM without calculating the conditional mutual information that an inferential adversary would in order to select query inputs to the victim model. As we do not have an efficient method for selecting good queries a-priori, instead, we perform a search for good collections of adversary victim interactions using Monte Carlo Tree Search (MCTS). Specifically, we search over states, modeling multiple distinct chains of adversary-victim interactions and the corresponding summaries which are assigned rewards based on their relevance to the malicious query. Consequently, MCTS enables searching for a summary response to the malicious query that is very relevant and can be interpreted as having reduced uncertainty significantly. Thus, rather than selecting good queries, we select a good chain of interactions as evaluated in a post-hoc manner, thereby approximating the chain of interactions that an inferential adversary could have had. Implementation details are provided in the Appendix C

Unlike security adversaries, where permissibility is determined solely by victim outputs, inferential adversaries do not rely on victim output permissibility violations to attain their malicious goal. Therefore, evaluation of the attack should focus on the adversarial model’s summary of all interactions addressing the malicious query rather than individual responses by the victim model.

To preform our evaluation, we sampled the first 30 questions corresponding to illegal activities from three datasets: Forbidden Questions (Shen et al., 2023), Harmful Behaviours (Zou et al., 2023b), and DangerousQA (Bhardwaj and Poria, 2023). Non-external human annotators evaluated the responses using a qualitative coding scheme, assessing whether the adversarial LLM did not return an answer (empty), the answer contained a refusal (refused), did not address the question (irrelevant), or answered the malicious query providing relevant information (affirmative). This approach, while potentially subject to author bias, was chosen to maintain ethical standards. Furthermore, the intent of the evaluation was to illustrate the feasibility of automated inferential adversaries and examine factors affecting performance, not to compare performance with a completely different threat model of jailbreaking security adversaries.

To establish feasibility, we compare our method of when attacking an aligned LLaAa2-70B-Instruct model to two baselines; directly querying a model with the malicious query and applying our method on the uncensored adversarial LLM (Mistral) as the victim model. As shown in Table 2, both victim models (Mistral-7B-Instruct and LLaMA2-70B-Instruct) led to a higher rate of affirmative responses returned from the adversarial LLM when subjected to our compositional attack method compared to directly querying them. The attack method successfully reduced empty responses, refusals, and irrelevant answers across all datasets for both victim models. The results suggest that even models that reject harmful questions directly can potentially leak information through compositional attacks. More detailed evaluation description is included in Appendix D.

As part of our evaluation, we observed several factors affecting success of the method that may be useful to control for in future red-teaming evaluations and instantiations:

• •

  Incomplete utilization of the prior: Our method outperforms directly querying the adversarial LLM even when the victim LLM used is the same as the adversarial LLM. This suggests that the adversarial LLM possesses prior knowledge which isn’t directly accessible. Another observation indicating the prior is not fully reflected in the baseline adversarial LLM is that the adversarial model would incorporate knowledge in its summaries that could not be attributed to any responses received from the victim model. This could make it more challenging to establish baselines and disentangle marginal risk of a deployed model from simply strong adversaries.

• •

  Invalid subquestions: Occasionally, either due to highly irrelevant responses or poor subquestion generation, very irrelevant or blatantly impermissible subquestions were selected, harming efficiency of the method.

• •

  Trajectory divergence: In some instances, the chosen sub-questions were the opposite of the intended harmful query, leading to responses and further questions which produced highly irrelevant answers.

Future design of inferential adversaries could involve fine tuning adversarial LLMs to ask informative sub-questions, employ subquestion relevance and permissibility selection mechanisms, and filter irrelevant parts of received responses.

While the benchmark datasets impermissible questions used have been commonly used to assess jailbreak robustness, red teaming assessments against inferential adversaries must carefully and precisely articulate what risks are intended to be mitigated and how these risks ought to be assessed. Rather than checking whether or not a provided LLM refused to answer impermissible questions, benchmarks should concretely articulate what knowledge or conclusions are legitimately harmful and impermissible for adversaries to get access to and evaluate the ability of an inferential adversary to reach these, or semantically equivalent, conclusions.

Our findings demonstrate the potential risks of inferential adversaries in real-world settings and highlight the need for more comprehensive evaluation methods for LLM safety. Upon discovery of vulnerabilities arising from inferential adversaries, model providers can assess the degree of contribution, or necessity, of their model for the adversarial objective and thereby assess necessity of risk mitigation strategies. We develop, such mitigation strategies to minimize risk contribution in the next section. Additionally, such vulnerabilities could be used to better prepare for and mitigate downstream risks of adversaries possessing harmful knowledge and assist in detecting malicious intent in user monitoring.

Our definition for impermissible information leakage differs from the definition of mutual information

The intent for this distinction is to capture the asymmetry in what conclusions an adversary infers. Whereas mutual information is concerned with changes of the posterior distribution relative to the prior for any conclusions $C=y$, the LLM provider is only concerned with the changes in belief of the adversary over impermissible conclusions $y$ such that the combination $(x,y)$ is harmful. Specifically, for the LLM provider, scenarios in which the adversary’s posterior $P_{A}((C=y)|(x_{1},r_{1}),\dots,(x_{k},r_{k}))=1$ for some $y\not\in R_{x}$ is perfectly acceptable as they imply the adversary became confident in a “permissible” input-output pair, whereas a bound on mutual information would deem this a defense failure.

Finding and bounding the supremum of the impermissible information leakage necessary for a $(k,\epsilon)$-ICM over all possible sets of $k$ interactions is challenging challenging due to the combinatorial complexity of checking all combinations. Consequently, we instead turn our focus to $\epsilon$-ICMs, which we show can still be used to provide bounds on impermissible information leakage from $k$ interactions.

In other words, the joint leakage can be bounded by the sum of $k$ individual $\epsilon$ per-interaction leakages and a term capturing the dependencies between interactions when conditioned on $C$. If the model outputs $\mathcal{V}(x_{i})$ are deterministic or independent when conditioned on $C$, then $\eta_{k}=0$ because the noise mechanism for an $\epsilon$-ICM is independent of the response. Such assumptions could hold when there is a “single true value” of $C$ known by the model, and all model outputs are related to this value by a deterministic function.

To provide concrete bounds on the information leakage to a $k$-inferential adversary, we construct an $\epsilon$-ICM. Inspired by a differentially private mechanism proposed by Mangat (1994) to protect privacy of individuals during surveys, we propose a randomized response information censorship mechanism.

The proposed Randomized Response $\epsilon$-ICM defined can vary depending on various assumptions, such as imposing constraints on adversary priors $\Phi$. For example, we assume the existence of a nonempty set $S$ which is not harmfully informative to the adversary as we assume an adversary does not get any impermissible information from an empty string returned as output.

Thus, the upper bound on impermissible information leakage scales linearly with the number of interactions an adversary may have, and controlling leakage requires significant reductions in $\epsilon$, implying substantially noisier and less useful outputs. Furthermore, if mutual information between an interaction and the impermissible concept is too costly to calculate, the denominator in $q_{\epsilon}$ can be substituted with $H_{R_{x}}(C)$, resulting in a higher probability of returning a response from $S$ to the user. As $S$ is selected in an input-independent manner, a higher probability of returning an element from $S$ will penalize utility for benign users. To demonstrate this, we first define utility, after which we turn to providing safety-utility trade-offs.

While the primary concern of censorship is to ensure safety by mitigating impermissible information leakage, model providers also care about the utility of their model. The utility of the model can be defined through the utility it provides benign users. We note that the scoring function defined for adversaries can be similarly defined for benign users, with user utility for an interaction defined as $u:\mathcal{X}\times\mathcal{Y}\to\mathbb{R}^{+}$. Furthermore, we define a distribution $P_{\mathcal{X}}$ capturing the probability that a benign user provides input $x$. Thus, the expected utility of an interaction between a model $\mathcal{V}$ and a benign user is given by

Benign users can also be viewed as having utility functions akin to those characterizing the objectives of security of inferential adversaries. For example, some users, especially those which seek certain function calling behaviors would expect model outputs to belong to an input dependent target set $T_{x}$, with utility $u(x,y)=\mathds{1}_{y\in T_{x}}$.

Alternatively, the user may be viewed as inferential—seeking to learn new information to bring them toward their desired goal. In this case a user seeking to learn some useful information $D$ seeks to maximize the mutual information

In other words, for an input output pair $(x,y)$, the utility function of an inferential adversary interested in learning $D$ is

Utility definitions can also be extended compositionally to capture the utility a benign user receives over multiple interactions. In particular, the utility of inferential users across $k$ model interactions is given by