A Causal Explainable Guardrails for Large Language Models

Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing tasks, generating human-like text and exhibiting a broad understanding of language. Central to their success is the rich set of representations they learn during training, which encode various concepts, attributes, and semantic information. Extensive research has been dedicated to understanding the representations learned by large language models (LLMs) and how they encode various concepts and attributes [18; 21; 22; 23]. LLM representations refer to the patterns of activations across the model’s parameters that correspond to specific semantic concepts, properties, or features. These representations act as the model’s internal codification of the information learned from the training data. Recent studies have provided compelling evidence that LLM representations contain rich semantic information, including abstract concepts like space and time [16], as well as more granular attributes related to truthfulness, toxicity, bias, and harmfulness of the generated text [24; 20; 25; 26]. In addition, linear classifier probes, which are trained to predict input properties from intermediate layers of a network, have successfully identified representations of concepts [27; 28]. Latent space analysis has enabled researchers to locate or edit factual associations within LLMs [29; 30; 31]. The ability to locate and manipulate these representations opens up avenues for controlling and steering the model’s outputs, enabling the development of principled methods for mitigating harmful behaviors and enhancing the interpretability and trustworthiness of large language models.

Activation engineering is a set of techniques that modify the internal activations of a pre-trained language model during inference to steer its output in a desired direction. The main idea is to identify and manipulate specific activations or attention heads associated with particular attributes or behaviors to control the model’s generation process. Early approaches, like the Plug-and-Play Language Model [32], use a separate classifier to detect target attributes in the generated text and perturb the language model’s activations accordingly, encouraging it to generate text that aligns with the desired attribute. Recent work focuses on extracting latent steering vectors from a frozen language model, which can be added to the activations during inference to steer the model’s completions toward specific goals, such as achieving high BLEU scores [33] or generating truthful statements [20]. Instead of requiring additional optimization or labeled data, Activation Addition [34] takes activation differences resulting from pairs of prompts. To avoid the identification of an opposite behaviour, [35] takes the average of activations associated with a target dataset and then subtracts the mean of all training activations, resulting in effective steering vectors. The growing interest in activation engineering stems from the desire to control and improve the performance of large language models on specific tasks or domains by identifying and manipulating the relevant activations.

Causal inference [36; 37; 38; 39; 40] has been an attractive research topic for a long time as it provides an effective way to uncover causal relationships in real-world problems. Nowadays, causal inference has shown potential in enhancing LLMs from a causal view in improving the LLMs’ reasoning capacity [1; 41; 42; 43; 44], addressing fairness issues in LLMs [45; 46; 47; 44], and safety [48; 49; 50], complementing LLMs with explanations [51; 13; 52; 53], and handling multimodality [54; 55]. In causal inference, if a variable is the common cause of two variables, it is called the confounder. The confounder will induce a spurious correlation between these two variables to disturb the recognition of the causal effect between them [56]. As shown in Figure 2(a), $X\rightarrow Y$ denotes that $X$ is the cause of $Y$. $C$ is the cause of both $X$ and $Y$. Thus, it is a confounder that will induce a spurious correlation between $X$ and $Y$ to disturb the recognition of the causal effect between them. In particular, such spurious correlation is brought by the backdoor path created by the confounder. Formally, a backdoor path between $X$ and $Y$ is defined as any path from $X$ to $Y$ that starts with an arrow pointing into $X$. For example, the path $X\leftarrow C\rightarrow Y$ is a backdoor path. If we want to deconfound two variables $X$ and $Y$ to calculate the true causal effect, we should block every backdoor path between them [57]. For example, in Figure 2(b), we should block $X\leftarrow C\rightarrow Y$ to get the causal effect between $X$ and $Y$.

Recent activation engineering-based work [26; 18; 34; 35; 20] utilize the steering vectors to control the direction of LLM output by constructing the randomized controlled trials. For example, some work [26; 18; 34] construct a pair of natural-language prompts ($p_{+}$, $p_{-}$), where $p_{+}$ represents the attribute we wish output text to emphasize and $p_{-}$ represents its opposite. $R_{+}/R_{-}$ is the representation for the prompt $p_{+}/p_{-}$. The difference $\Delta R$ is a new steering vector that (intuitively) captures the difference between a prompt with the attribute and without it. To obtain a steering vector, they perform a forward pass on each prompt, record the activations at the given location in each pass, take the difference, and then finally rescale this difference in activations by an “injection coefficient” $\beta$. To steer, they add the resulting steering representation to the original representations and allow the forward pass to continue, and obtain the steered output.

To systemically analyze this series of methods, we give the formal definitions. Now, let’s further break down and analysis of the constructed prompt pairs. In fact, the input prompt is comprised of the semantic prompt $C$ and the steering prompt $S$. For example, in the [34], “I love talking about weddings” is the positive prompt and “I hate talking about weddings” is the negative prompt. In these two examples, “love” and “hate” are the steering prompts, which control the directions of output. The “I … talking about weddings” is the same for this pair of prompts, which only contains the semantic information. Therefore, they assume this is a randomized controlled trial, where there are two groups, i.e., treatment group “love” and control group “hate”. As shown in Figure 3 (a), the semantic prompt $C$ is a confounder, that can affect the steering prompt $S$ and output $Y$. In this hypothetical situation, they construct the randomized controlled trial by creating the pairs, i.e., positive steering prompt plus the same semantic prompt and negative steering prompt plus the same semantic prompt. The only difference is the steering prompt since the semantic prompts are the totally same. Based on this assumption, the edge from the semantic prompt to the steering prompt is blocked. Therefore, they can get the difference in activations, which can intervene in the generated output. The strength of this steering vector is called the treatment effect in causal inference. On the face of it, this is a perfect randomized controlled trial, where only the steering prompt can affect the direction representation $R_{+}/R_{-}$. The combination of direction representation $R_{+}/R_{-}$ and the semantic prompt together have an influence on the generated output $Y$. In this case, the steering vector can be obtained by the operation between positive direction representation $R_{+}$ and negative direction representation $R_{-}$.

Before delving into the analysis of the real situation, we first provide formal definitions for the various types of representations involved in the steering process.

Based on the above definitions, the assumed randomized controlled trial in the hypothetical situation is not qualified because it fails to consider the influence of confounders, specifically the semantic prompt, on the direction representation $R_{+}/R_{-}$ due to biases in the pre-training of the large language model (LLM). As a result, the obtained steering vector is biased. As proven in the introduction section, the semantic prompt $C$ affects not only the content of the output $Y$ but also its direction. We can further break down the semantic prompt into two parts, i.e., the semantic context representation $R^{cy}$, which influences the content of the output $Y$, and the semantic direction representation $R^{cd}$, which influences the direction representation $R_{+}/R_{-}$.

In addition to the edge from the semantic prompt $C$ to the steering prompt $S$ and the direction representation $R_{+}/R_{-}$, there is a direct edge from the semantic direction representation $R^{cd}$ of the semantic prompt to the direction representation $R_{+}/R_{-}$. The constructed pair of prompts can only block the edge from the semantic prompt $C$ to the steering prompt $S$ and the direction representation $R_{+}/R_{-}$, but it ignores the edge from the semantic direction representation $R^{cd}$ of the semantic prompt to the direction representation $R_{+}/R_{-}$. As shown in Figure 3 (b), both the steering prompt $S_{+}/S_{-}$ and the semantic direction representation $R^{cd}$ affect the direction representation $R_{+}/R_{-}$. Consequently, the obtained steering vector $\Delta R$ is biased by the semantic direction representation $R^{cd}$ of the semantic prompt, which is not solely affected by the constructed steering prompt $S$.

This bias originates from the biases in the pre-training data. For example, consider the semantic prompt “I … talking about weddings”. Due to biases present in the pre-training data, the LLM may have learned to associate weddings with positive sentiments such as love, happiness, and celebration. As a result, the semantic direction representation $R^{cd}$ learned from this prompt may implicitly suggest a positive direction (e.g., "love") for the output, even if the explicit steering prompt is not provided. As shown in Figure 1, this edge can be visually observed experimentally. The implicit influence of the semantic direction representation $R^{cd}$ on the output direction can be problematic when attempting to steer the LLM’s output towards a desired attribute. If the steering representation $\Delta R$ is not independent of $R^{cd}$, the resulting output may be biased by the inherent associations learned during pre-training, leading to suboptimal or unintended results. To ensure unbiased steering of the output, it is crucial to disentangle the steering representation $\Delta R$ from the semantic direction representation $R^{cd}$.

Based on the causal analysis presented in the previous sections, we propose a solution called LLMGuardaril to address the bias introduced by the semantic direction representation $R^{cd}$ in the steering process. As shown in Figure 3 (c), in addition to constructing pairs of prompts (positive and negative steering prompts with the same semantic prompt), we need to block the edge from the semantic direction representation $R^{cd}$ of the semantic prompt to the direction representation $R_{+}/R_{-}$. By doing so, the direction representation is only influenced by the steering prompt $S_{+}/S_{-}$, enabling us to obtain an unbiased steering representation $\Delta R$ through steering engineering. This approach aligns with the desired hypothetical situation.

We utilize adversarial learning to remove this confounding bias [58; 59]. The objective is to achieve debiasing of the influence $R^{cd}$ on the direction representation $R_{+}/R_{-}$ during the adversarial learning process. The adversarial learning process includes two main components, i.e., prediction reconstruction learning and debias learning. In prediction reconstruction learning, the goal is to ensure that the original semantic information remains unchanged during the debiasing process. This is achieved by minimizing the prediction reconstruction loss, which measures the cross-entropy between the original output and the output generated.

By ensuring that the steering representation is independent of the semantic direction representation $R^{cd}$, it becomes an unbiased representation that can effectively steer the output of the language model toward the desired attribute. This approach mitigates the influence of unwanted biases present in the model, enabling more accurate and controlled steering of the language model’s output. The debiased intermediate states generated by the Debias LoRA Block can then be used to guide the LLM’s output toward the desired attributes or concepts while mitigating the impact of unwanted biases. This allows for more precise steering of the language model’s output, leading to improved performance in various downstream tasks.

Previous studies have investigated the information encoded in different layers of transformer-based models. Tenney et al. [60] found that earlier layers in BERT encode lower-level information, such as part-of-speech tags, while later layers capture more semantic information. Similarly, Zou et al. [18] and Li et al. [20] have observed that the optimal intervention layers for steering the model’s output vary depending on the specific task and desired attribute.

To identify the most effective layers for intervention, we propose a systematic approach based on probing accuracy. Let $L=\{l_{1},l_{2},\dots,l_{D}\}$ denote the set of all layers in the pre-trained language model, where $D$ is the total number of layers. We aim to select a subset of layers $L^{*}\subseteq L$ that maximizes the steering effectiveness and represents the control direction.

We employ a probing classifier to measure the outcome-relatedness of each layer. The probing classifier is trained to predict the target attribute or concept based on the representations at each layer. Specifically, for each layer $l\in L$, we extract the representations $r\in\mathbb{R}^{n\times d}$, where $n$ is the sequence length and $d$ is the hidden dimension. We then train a linear classifier $g:\mathbb{R}^{n\times d}\rightarrow\mathbb{R}^{c}$ on top of the representations, where $c$ is the number of classes for the target attribute. The probing accuracy of each layer $l$ is evaluated on a validation set. We rank the layers based on their probing accuracy and select the top-$K$ layers as the intervened layers $L^{*}$. The number of intervened layers $K$ is a hyperparameter that can be tuned based on the specific task and desired trade-off between steering effectiveness and computational efficiency.

Empirically, we find that the middle layers of the pre-trained language model tend to be the most effective for intervention. This observation aligns with previous findings [18; 20] suggesting that the middle layers capture a balance between low-level syntactic information and high-level semantic information, making them suitable for steering the model’s output towards the desired attribute or concept. By selecting the intervened layers based on probing accuracy, we can focus the intervention on the most relevant layers, thereby improving the efficiency and effectiveness of the steering process. The selected layers $L^{*}$ are then used in the Debias LoRA Block to obtain the debiased representations and calculate the steering representation.

As discussed in the causal analysis, to obtain unbiased steering representations, we must block the edge from the semantic direction representation $R^{cd}$ of the semantic prompt to the direction representations $R_{+}$ and $R_{-}$. By doing so, the direction representations are solely influenced by the steering prompts $S_{+}$ and $S_{-}$, enabling us to obtain an unbiased steering representation $\Delta R$ through steering engineering. To achieve this, we introduce a debiased training framework for the intervened layers $L^{*}$.

In this step, our objective is to identify a direction that accurately predicts the underlying output concept and to provide an explanation for the generated output. Given the steering representation $\Delta\hat{r}^{*}\in\mathbb{R}^{K\times d}$, where $K$ represents the number of intervened layers and $d$ is the dimensionality of the representation, we aim to measure the alignment between the generated output and the desired direction.

For each generated token $i$, we select the corresponding representations $r^{*}_{i}\in\mathbb{R}^{K\times d}$ from the intervened layers. These representations capture the activations of the model at the specific layers where the steering intervention is applied. To obtain a consolidated representation for both the steering prompt and the generated token, we perform layer-wise averaging. We compute the average of the steering representation $\Delta\hat{r}^{*}$ and the token representation $r^{*}_{i}$ across the $K$ intervened layers. This step aggregates the information from multiple layers, providing a more robust representation. Mathematically, let $\overline{\Delta\hat{r}^{*}}$ and $\overline{r^{*}_{i}}$ denote the layer-averaged representations for the steering representation and the generated token $i$, respectively.

Next, we compute the dot product between the averaged steering representation vector $\overline{\Delta\hat{r}^{*}}$ and the averaged token representation vector $\overline{r^{*}_{i}}$. The dot product measures the similarity between the two vectors, indicating the alignment between the generated output and the desired direction:

The resulting $\text{similarity}_{i}$ score quantifies the extent to which the generated token $i$ aligns with the desired direction defined by the steering representation. A higher similarity score indicates a stronger alignment, suggesting that the generated output is more likely to exhibit the desired attribute or concept.

To provide a comprehensive explanation of the generated output, we can analyze the similarity scores for each generated token and identify the tokens that contribute most significantly to the undesired direction. We can rank the tokens based on their similarity scores and highlight the top-k tokens that have the lowest alignment with the steering representation. These top-k tokens can be considered as the key indicators of the undesired attribute or concept in the generated output. Furthermore, we can visualize the similarity scores across the generated output to gain insights into the alignment between the output and the desired direction. By plotting the similarity scores for each token, we can observe the variations in alignment throughout the generated text. This visualization can help identify regions of the output that strongly align with the desired direction and regions that may deviate from it. In addition to the token-level analysis, we can also compute an overall alignment score for the entire generated output. This can be done by averaging the similarity scores across all tokens:

where $n$ is the total number of tokens in the generated output. The overall alignment score provides a summary measure of how well the generated output aligns with the desired direction.

By combining the token-level analysis, visualization of similarity scores, and the overall alignment score, we can provide a comprehensive explanation of the generated output in relation to the desired direction. This explanation can help users understand and identify the specific aspects of the output that contribute to the desired attribute or concept.

The steering representations are injected into the model and activated during inference, directing the model’s response toward the desired direction. Unlike most existing methods [20; 63; 35; 34] that simply add or subtract a constant steering representation regardless of the token representation, we propose a more sophisticated approach to control the output. Our method takes into account the relationship between the generated token representation and the steering representation, allowing for more fine-grained and context-aware control of the output.

To establish a connection between the generated token representation and the steering representation, we employ a projection operation inspired by [18]. This operation amplifies the component of the token representation that aligns with the steering representation, effectively emphasizing the desired direction in the output. Let $r^{*}_{i}\in\mathbb{R}^{K\times d}$ denote the representation of the generated token $i$ obtained from the intervened layers. This is achieved by projecting out the component in the direction of steering representation $\Delta\hat{r}^{*}$, and the operation can be defined as

where $r^{*\mathsf{T}}_{i}$ represents the transpose of the token representation $r^{*}_{i}$, and $\|\Delta\hat{r}^{*}\|^{2}$ denotes the squared Euclidean norm of the steering representation. To steer, we multiply the projection by a coefficient $\beta$ that represents the intervention strength.

By adjusting the value of $\beta$, we can control the extent to which the output is steered toward the desired direction. A larger value of $\beta$ will result in a stronger emphasis on the desired direction, while a smaller value will have a more subtle effect. The choice of $\beta$ is crucial for achieving the desired level of control over the output. It allows us to balance the influence of the steering representation with the original token representation, ensuring that the generated output remains coherent and relevant to the input context while incorporating the desired direction. Another consideration is the adaptability of $\beta$ to different contexts and desired directions. It may be beneficial to dynamically adjust the value of $\beta$ based on the characteristics of the input and the specific direction we aim to steer the output. For example, we can employ a context-dependent $\beta$ that varies based on the semantic similarity between the input and the desired direction, allowing for a more nuanced control of the output.

Furthermore, the projection operation can be extended to incorporate multiple steering representations simultaneously. In scenarios where we want to steer the output towards multiple desired directions, we can compute the projections onto each steering representation separately and combine them using appropriate weighting schemes. This enables a more comprehensive control over the output, allowing us to incorporate multiple desired attributes or concepts.

In summary, the control of output in our proposed method involves projecting the generated token representations onto the steering representation and scaling the projected component by a coefficient $\beta$. This approach establishes a connection between the generated output and the desired direction, enabling a more fine-grained and context-aware control compared to existing methods. By carefully tuning the value of $\beta$ and potentially adapting it to different contexts, we can effectively steer the output toward the desired direction while maintaining the quality and coherence of the generated text. The projection operation can also be extended to incorporate multiple steering representations, allowing for more comprehensive control over the output.

In this section, we introduce a diverse set of baseline methods that aim to steer the behavior of large language models towards desired attributes or concepts. By comparing LLMGuardaril against these baselines, we can assess its effectiveness in achieving the desired steering while maintaining the model’s general knowledge and capabilities.

Base model (Few-shot) [64] is an in-context learning method that does not require fine-tuning of the language model. This technique leverages a small set of high-quality samples to construct prompts, which are then used to guide the Language Learning Model (LLM) in generating answers that closely resemble the provided examples. To implement this approach, we first create a set of 3-5 queries. These queries are then fed into GPT-4, a state-of-the-art language model, to generate high-quality answers. The resulting query-answer pairs serve as in-distribution examples, which are subsequently used to construct prompts during the inference process. By providing the LLM with these carefully crafted prompts, the model is able to generate answers that are more aligned with the desired output, thereby improving the overall quality of the answers.

Linear Artificial Tomography (LAT-Reading) [18] is a representation reading technique that aims to locate emergent representations for high-level concepts and functions within a network. The LAT pipeline consists of three key steps: designing stimulus and task, collecting neural activity, and constructing a linear model. A stimulus set and task template are carefully designed to elicit distinct neural activity for the target concept or function. Next, neural activity is collected from specific token positions in the model’s hidden states. Finally, a linear model, such as PCA or a supervised method, is applied to the collected neural activity to identify directions that align with the target concept or function. The resulting “reading vectors” can be generally used to steer the output direction.

LAT-Contrast [18] is a representation control technique that builds upon the reading vectors obtained through LAT. During inference, the same input is run through the model using a pair of contrastive prompts, producing two different representations. The difference between these representations forms a Contrast Vector, which serves as a stimulus-dependent controller. The Contrast Vectors are used to modify the model’s representations at each layer using operations such as linear combination, piece-wise operations, or projection. Compared to the above LAT-Reading, it requires over $3\times$ more inference compute for the positive prompt, negative prompt, and the original prompt.

Low-Rank Representation Adaptation (LoRRA) [18] is another representation control technique that addresses the computational overhead associated with calculating Contrast Vectors during inference. LoRRA involves fine-tuning low-rank adapter matrices connected to the model’s attention weights using a specific loss function applied to representations. The loss function is designed to guide the model’s representations towards the desired target representations, which can be obtained using methods such as LAT-Contrast. During training, the adapters are updated to minimize the difference between the model’s current representations and the target representations. Once trained, the adapters are merged into the model, resulting in modified representations without additional computational burden during inference. LoRRA provides an efficient way to control the model by directly optimizing the model’s representations.

Activation Addition (ActAdd) [34] is a lightweight approach for controlling the behavior of pre-trained language models without fine-tuning or optimization. ActAdd operates by modifying the activations of a frozen model at inference time to steer the output in a desired direction, which is specified through natural language prompts. The method computes the difference in activations resulting from a pair of prompts that represent the desired attribute and its opposite, then adds this delta to the activations at a specific layer during inference. This steers the model’s output towards exhibiting the desired attribute while preserving its general knowledge and capabilities on unrelated tasks.

Mean-Centring [35] is a method for steering the behavior of pre-trained language models by modifying their activations at inference time. The key idea is to extract a “distillation vector” that captures the properties of a target dataset exhibiting the desired behavior. This is done by averaging the activations of the model when processing the target dataset and then subtracting the mean activations of the model on a general training dataset. This mean-centering step removes the model’s general activation bias and isolates the specific direction corresponding to the target behavior.

Contrast-Consistent Search (CCS) [26] is an unsupervised method for discovering latent knowledge in the internal activations of pre-trained language models. Given a set of yes-no questions, CCS constructs “contrast pairs” by answering each question as both “yes” and “no”. It then extracts the model’s hidden representations for these contrast pairs, normalizes them to remove positional biases, and learns a linear mapping from the representations to probabilities of being true or false.

To assess the effectiveness of LLMGuardaril in steering language models towards desired attributes, we conduct experiments on several public datasets that focus on different aspects of content safety and bias [63]. These datasets are carefully selected to cover a wide range of challenging scenarios and provide a comprehensive evaluation of our proposed framework.

Truthfulness. The TruthfulQA benchmark [65] is utilized to evaluate LLMGuardaril’s performance in promoting truthful responses. This dataset is meticulously designed to be adversarial, incorporating false beliefs, misconceptions, and logical falsehoods across 38 categories. By testing on the full dataset of 817 questions, we assess the model’s ability to navigate through these challenges and provide accurate and informative answers. The primary metric, denoted as $\texttt{True}+\texttt{Info}$, represents the percentage of responses that are both truthful and informative, as determined by two finetuned GPT-3-13B models (GPT-judge). Additionally, we report separate results for truthfulness and informativeness to provide a more granular analysis of the model’s performance.

Toxicity. To evaluate the effectiveness of LLMGuardaril in mitigating toxicity, we employ the ToxiGen dataset [66], which contains implicitly toxic and benign sentences mentioning 13 minority groups. We use a revised version of the dataset [67] that reduces noise by filtering out prompts with disagreement among annotators regarding the target demographic group. Through stratified sampling, we select a representative subset of 700 examples for our experiments. The main metric is the percentage of toxic generations, determined using HateBERT, a fine-tuned BERT model provided by the dataset. Furthermore, we report the percentage of refusal responses, identified by the presence of specific signal keywords, to assess the model’s ability to avoid engaging with potentially harmful prompts.

Bias. The BOLD benchmark [68] is employed to evaluate bias in generated responses. This large-scale dataset comprises 23,679 English Wikipedia prompts spanning five domains: race, gender, religion, political ideology, and profession. To manage experiment costs, we sample 120 prompts from each domain. The VADER sentiment score [69] serves as the primary metric, quantifying the sentiment directed towards the population mentioned in each prompt. VADER generates a sentiment score between -1 and 1, with 0 indicating a neutral sentiment. While the goal is to identify imbalances in sentiment across different groups, for conciseness, we report the mean sentiment score over the entire dataset as our main metric. Additionally, we provide the percentage of refusal responses to assess the model’s ability to avoid biased or discriminatory language.

Harmfulness. To evaluate LLMGuardaril’s performance in mitigating harmful content, we utilize the AdvBench dataset [70], which contains 500 harmful behaviors and instructions reflecting toxic, discriminatory, or cybercrime-related actions. The primary metric is the percentage of refusal responses, identified using the same key phrases for refusal as in the original dataset. Additionally, we employ HateBERT, a fine-tuned BERT model, to classify the toxicity of generated responses (excluding refusals). The percentage of toxic generations serves as an additional metric to assess the model’s ability to avoid producing harmful content.

By conducting experiments on these diverse benchmarks, we aim to provide a comprehensive evaluation of LLMGuardaril’s effectiveness in steering language models towards desired attributes while mitigating undesirable behaviors. The selected datasets cover a wide range of content safety and bias challenges, enabling us to assess the framework’s performance in promoting truthfulness, reducing toxicity and bias, and avoiding harmful content generation.

Prompt Design. To thoroughly assess the efficacy of LLMGuardaril, we employ a prompt structure that combines a prefix steering prompt, which can be either positive or negative, with an identical semantic prompt. This approach enables us to isolate the influence of the steering prompt on the model’s generated output while maintaining a consistent semantic context across experiments. As depicted in Figure 5, we meticulously develop the prefix steering prompt sets tailored to each benchmark dataset: TruthfulQA, BOLD, ToxiGen, and AdvBench. These prompts are strategically designed to steer the model towards generating content that aligns with the desired attributes or concepts specific to each dataset, such as truthfulness, lack of bias, non-toxicity, and avoidance of harmful content.

Model Selection. We evaluate the guardrail performance of LLMGuardaril using two prominent families of instruction-tuned large language models: Llama2 [71] and Vicuna-V1.5 [72]. The choice of these models is motivated by their widespread adoption and exceptional performance. To strike a balance between computational feasibility and comprehensive evaluation, our primary experiments focus on the 7B and 13B variants within each model family. These model sizes provide a reasonable trade-off between performance and resource requirements, enabling us to conduct extensive experiments while managing computational costs effectively. To gain a deeper understanding of LLMGuardaril’s scalability and its potential to generalize across different model sizes, we expand our experiments to encompass the entire spectrum of model variants within the Vicuna family, including Vicuna-33B.

Table 1 presents a comprehensive performance comparison of LLMGuardaril against various baseline methods across four benchmark datasets: TruthfulQA, ToxiGen, BOLD, and AdvBench. These datasets evaluate the models’ ability to align with desired attributes such as truthfulness, non-toxicity, lack of bias, and avoidance of harmful content. Across all datasets and model variants (Vicuna-7b, Llama2-7b, and Llama2-13b), LLMGuardaril consistently outperforms the baseline methods. The results highlight LLMGuardaril’s state-of-the-art performance in steering the language models towards the desired attributes. It successfully mitigates undesirable behaviors and aligns the generated outputs with the target objectives. The superior performance of LLMGuardaril compared to the baseline methods highlights the significance of our proposed framework and its potential for real-world applications. As depicted in Figure 5, we provide examples comparing the original LLM output without any guardrails to the intervened output generated by our LLMGuardaril. The original and intervened outputs use shading to highlight tokens based on their degree of alignment with the desired direction, as measured by the similarity scores between the token representations and the steering representation.

Table 4 presents the scaling law study conducted to evaluate the performance of LLMGuardaril across different model sizes within the Vicuna family. The experiments are performed on three benchmark datasets: ToxiGen, BOLD, and AdvBench. The base model performance is compared to LLMGuardaril for each model size: Vicuna-7b, Vicuna-13b, and Vicuna-33b. The scaling law study aims to investigate the effect of increasing model size on the performance of LLMGuardaril in steering the model’s output towards desired attributes. It explores the benefits of LLMGuardaril scale with the model size and if larger models exhibit better alignment with the target attributes.

Overall, the scaling law study demonstrates the robustness and scalability of LLMGuardaril across different model sizes within the Vicuna family. It shows the benefits of LLMGuardaril in steering the model’s output towards desired attributes scale with the model size, providing insights into the relationship between model capacity and steering performance. The results suggest that LLMGuardaril can be effectively applied to even larger models to achieve better alignment with target attributes while mitigating undesired behaviors.

Table 5 presents the results of out-of-domain (OOD) experiments for the LLMGuardaril model and several baseline methods on the ToxiGen dataset. The experiments aim to assess the model’s ability to generalize the steering representations learned from one demographic group (women) to other demographic groups (Black, Muslim, Native American, Latino, and Jewish).

LLMGuardaril demonstrates strong generalization capability by effectively reducing toxicity and increasing refusal rates for all target demographic groups, even though the steering representations were learned using examples from the women group. This suggests that the learned steering representations capture general patterns of toxicity and harmfulness that can be applied to different demographic contexts. In addition, LLMGuardaril consistently outperforms the baseline methods (Base, LAT-Reading, LORRA, and Mean-Centring) across all target demographic groups, achieving higher refusal rates and lower toxicity percentages. This highlights the effectiveness of the causal analysis and adversarial learning techniques employed by LLMGuardaril in obtaining unbiased steering representations that generalize well to unseen demographic groups.

Figure 6 and 7 present the sensitivity analysis of the hyperparameter $\alpha$ and $\beta$ on the ToxiGen and AdvBench datasets. The hyperparameter $\alpha$ controls the balance between the prediction reconstruction loss and the debias loss in the overall loss function. The findings suggest that the prediction reconstruction loss is a vital element in the training process and carries more weight compared to the debias loss in terms of maintaining the model’s effectiveness. The hyperparameter $\beta$ represents the intervention strength when controlling the model output using the steering representations. Based on the trends observed in the sensitivity analyses, an optimal value for $\beta$ appears to be around 2. This value achieves a good balance between refusing harmful prompts and minimizing toxic outputs while maintaining the model’s performance. The optimal ranges identified for $\alpha$ and $\beta$ can guide the selection of appropriate values for these hyperparameters in practical applications of the LLMGuardaril model.

The development of safe and reliable LLMs has significant social implications. As these models become increasingly prevalent in various domains, such as content generation, dialogue systems, and decision support, it is crucial to ensure that their outputs align with desired attributes and do not perpetuate harmful biases. Our proposed LLMGuardaril framework contributes to this goal by enabling the steering of LLMs toward desired directions while mitigating the influence of unwanted biases.

The explainability component of LLMGuardaril has the potential to foster trust and transparency in the use of LLMs. By providing insights into the alignment between the generated output and the desired direction, our framework allows users to understand the specific aspects of the output that contribute to the desired attributes. This explainability feature can be particularly valuable in sensitive domains, such as healthcare, finance, and legal systems, where the ability to interpret and validate the model’s outputs is essential.

Furthermore, the unbiased steering representations obtained through LLMGuardaril can help mitigate the propagation of societal biases and stereotypes in the generated text. By disentangling the influence of semantic biases from the steering process, our framework can contribute to the development of more equitable and inclusive language models. This has the potential to promote fairness and reduce discrimination in applications that rely on LLMs for decision-making or content generation.

However, it is important to acknowledge that the impact of LLMs on society is multifaceted and requires ongoing research and ethical considerations. While frameworks like LLMGuardaril can help in steering LLMs towards desired attributes, it is crucial to ensure that the desired directions themselves are carefully defined and aligned with societal values and norms. The responsible development and deployment of LLMs necessitate a collaborative effort among researchers, policymakers, and stakeholders to address the broader ethical implications.

While LLMGuardaril demonstrates promising results in obtaining unbiased steering representations and providing explainable insights, there are certain limitations to our framework that should be acknowledged.

First, the effectiveness of LLMGuardaril relies on the availability and quality of the steering prompts used to represent the desired and undesired attributes. Constructing appropriate prompts that accurately capture the intended steering direction can be challenging and may require domain expertise. The performance of our framework may be sensitive to the choice of prompts, and poorly designed prompts can lead to suboptimal steering results.

Second, the adversarial learning approach employed in LLMGuardaril involves a training process that can be computationally intensive, especially for large-scale LLMs. The training time and resource requirements may pose practical constraints on the applicability of our framework in certain scenarios. Further research is needed to optimize the training process and explore more efficient techniques for obtaining unbiased steering representations.

Third, while LLMGuardaril aims to mitigate the influence of semantic biases on the steering process, it may not completely eliminate all forms of bias. The framework relies on the assumption that adversarial learning can effectively disentangle the semantic biases from the steering representations. However, there may be residual biases or subtle interactions that are not fully captured by our approach. Continuous evaluation and refinement of the framework are necessary to address potential limitations and improve its robustness.

The proposed LLMGuardaril framework opens up several avenues for future research in the field of language model steering and explainable AI. Here, we outline some potential directions for further exploration.

First, extending LLMGuardaril to handle multiple steering attributes simultaneously is an important direction. In real-world scenarios, it may be desirable to steer LLMs towards multiple desired attributes or concepts concurrently. Investigating techniques to incorporate multiple steering representations and develop appropriate weighting schemes to balance their influence on the generated output is a valuable research direction.

Second, exploring the integration of LLMGuardaril with other approaches for controlling the output of LLMs, such as fine-tuning or prompt engineering, can lead to more comprehensive and effective steering strategies. Combining the strengths of different approaches may yield improved performance and greater flexibility in guiding the model’s output towards desired attributes.

Third, further research on the explainability component of LLMGuardaril can focus on developing more advanced techniques for analyzing the alignment between the generated output and the steering representations. Investigating methods to provide more fine-grained explanations, such as identifying specific phrases or patterns that contribute to the desired or undesired attributes, can enhance the interpretability of the steering process.

Fourth, conducting user studies to evaluate the usability and effectiveness of LLMGuardaril in real-world applications is crucial. Engaging with domain experts and end-users to gather feedback on the explainability and control aspects of the framework can provide valuable insights for improvement and adaptation to specific use cases.

Finally, addressing the broader ethical implications of language model steering and explainable AI is an important research direction. Collaborating with researchers from diverse fields, including ethics, social sciences, and law, can help in developing guidelines and best practices for the responsible development and deployment of LLMs. Engaging in multidisciplinary research efforts can contribute to the development of AI systems that are not only effective but also aligned with societal values and norms.

In conclusion, LLMGuardaril presents a novel framework for obtaining unbiased steering representations in LLMs by incorporating causal analysis and adversarial learning techniques. Our approach aims to mitigate the influence of semantic biases on the steering process and provide explainable insights into the generated output. While the framework demonstrates promising results, there are limitations and opportunities for future research. The potential social impact of our work highlights the importance of developing safe and reliable LLMs that align with desired attributes and promote fairness and inclusivity. Ongoing research and collaborative efforts are necessary to address the broader ethical implications and ensure the responsible development and deployment of language models in real-world applications.