Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agent

The objective of the Multi-Character Role-Playing (MCRP) task is to inject the style of $M$ characters into a language model, enhancing the personalization of conversations. Specifically, a $N$-turn dialogue MCRP sample is defined as a sequence of utterances $U=\{u_{1}^{h},u_{1}^{r_{1}},...,u_{i-1}^{h},u_{i-1}^{r_{k}}\}_{i=2}^{N}$, where $u_{i}^{h}$ denotes the utterance that the user (human) queries, and $u_{i}^{r_{k}}$ denotes the utterance that the role $r_{k}$ agent (model) responds to the user at $i$-th turn, and $R=\{r_{k}\}_{k=1}^{M}$ denotes the transition order of roles in conversations. When they first meet each other, the dialogue started by the user and the model is expected to generate an answer $a=u_{i}^{r_{k}}$ according to a given role $r_{k}$, the history dialogue $U$, and the question of user $q=u_{i}^{h}$.

where $\Theta$ represents the language model parameters, which are static during inference. Specially, we can specify a certain instruction prompt “I want you to act like {character} $\dots$” to achieve emulation of the designated character in the ongoing conversation.

Low-rank Adapter (LoRA) (Hu et al., 2021) has proven to be both effective and efficient in the fine-tuning of Large Language Models. As our approach builds upon the foundational principles of LoRA, it is pertinent to provide a brief overview of its workings. LoRA is inspired by the low intrinsic dimension characteristic (Aghajanyan et al., 2020), which hypothesize the updates to any parameter weights in LLMs as a low-rank decomposition.

Assume that $W_{0}\in\mathbb{R}^{m\times d}$ represents the parameter matrix of the pre-trained LLM which is accompanied by a LoRA decomposition $\Delta W=BA$, where $B\in\mathbb{R}^{m\times r}$ and $A\in\mathbb{R}^{r\times d}$ are low-rank and trainable matrices. For the original $h=W_{0}x$, the modified forward pass yields:

where $x$ represents the input vector of dimension $m$, and $h$ is the output vector with dimension $d$. The rank of the trainable low-rank matrices is denoted by $r$, which determines the number of trainable parameters, and $r\ll\min(m,d)$. $\alpha$ is a constant hyper-parameter for scaling, $B$ is initialized as a zero matrix, and $A$ is initialized using a zero-mean Gaussian distribution. During fine-tuning, only the LoRA modules can be updated.

To demonstrate the usage of vanilla LoRA in Role-play, we can simply assume that there is only one LoRA module in the pre-trained network. Let’s consider a general loss function $\mathcal{L}$ for the model $f$ to play a specific role $r_{k}$. The target matrices $B^{*}$ and $A^{*}$ trained on corpus $D=(X,Y)$ of $r_{k}$ are formulated as:

where the sets of inputs and labels in $D$ are denoted as $X$ and $Y$. In the MCRP task setting, $X$ represents history utterances $U$, a given role $r_{k}$, and a question from user. $Y$ is the agent response $a$ conditioned on $X$. However, vanilla LoRA tends to degrade the performance on previous multiple characters due to catastrophic forgetting. It’s hence hard for role-playing agent to satisfy acting consistency, as shown in (Section 4). In the following sections, we present the Neeko framework, which overcomes this limitation with the cooperation of dynamic LoRA and gating selection modules.

Motivated by the dynamic LoRA frameworks (Valipour et al., 2023; Yu et al., 2023; Ye and Bors, 2023), we extend dynamic LoRA to the MCRP task. Rather than randomly selecting the range of LoRA ranks, we introduce non-overlapping LoRA blocks for different characters to mitigate catastrophic forgetting.

In Figure 1 (a), the LoRA module consists of low-rank matrices $B\in\mathbb{R}^{m\times r}$ and $A\in\mathbb{R}^{r\times d}$. Let’s assume that we would like to train a part of weights in matrices $B$ and $A$ for each character, which can be termed as a trainable LoRA block. The range of a block is determined by the order number of role $k\in[1,M]$ and the predefined hyper-parameter partial rank $p$. In this way, the LoRA blocks for different character $r_{k}$ are non-overlapping:

Here, $W_{B}^{k}$ and $W_{A}^{k}$ represent the trainable block in matrices $B$ and $A$ for the $k$-th character, and the total rank $r$ equals the number of required LoRA blocks $M$ multiplied by the partial rank $p$. Thus, Neeko serves as a role-playing agent for a mass of characters by the hyper-parameter $r$ and $p$. Detailed training setting for Neeko can be found in the Appendix A. With the learning rate $\eta$, a character corpus $D$ can be quickly learned in a small LoRA block:

Since different characters are trained with non-overlapping LoRA blocks, Neeko could keep the information of previous characters without retraining.

To enable the activation of specific LoRA blocks for role-based instruction during inference, we introduce a unique gating network inspired by the Mixture of Experts (MoE) (Eigen et al., 2013; Chen et al., 2023b; Luo et al., 2024). Unlike previous approaches, this network takes the role identity as input to retrieve role-specific LoRA parameters. For clarity, we establish a global role embedding matrix, denoted as $\mathbf{E}_{\text{global}}\in\mathbb{R}^{M\times d}$, using the profiles of $M$ existing characters. When identifying a role $r_{k}\in R$, we extract its representation vector $e_{k}\in\mathbb{R}^{d}$ from the $k$-th column of $\mathbf{E}_{\text{global}}$. To determine the contribution weights for role $r_{k}$, we apply a linear transformation using the following equation:

where $w_{k}\in\mathbb{R}^{M}$ represents the contribution weight vector tailored for role $r_{k}$ to select LoRA blocks, and $W_{G}\in\mathbb{R}^{d\times M}$ is the transformation matrix of the gating network. The softmax operation normalizes these weights.

During the inference stage, Neeko retrieves the representation $e_{k}$ for the target character $r_{k}$ from $\mathbf{E}_{\text{global}}$, guided by the instruction prompt. This is achieved through a key-value indexing scheme, where $(\text{role},\text{index})$ facilitates access to $e_{k}$. The activation of specific LoRA blocks is then directed by $\operatorname{argmax}_{k}(w_{k})$, pinpointing the most significant contribution weight among the learned weights for role $r_{k}$. This ensures a controlled, role-specific activation of LoRA blocks, aligned with the contribution weights determined during the training phase.

In the incremental phase, we have designed two modes: fusion and expansion. In the fusion mode, we employ an element-wise method to combine LoRA modules (Huang et al., 2023; Liu et al., 2023). This integration process combines the corresponding parameters of the LoRA modules. It’s essential that the modules being combined have the same partial rank $p$ for proper alignment. Given $\Delta W_{k}=B_{k}A_{k}$, the combined LoRA module $\Delta W_{j}$ and the updated $W_{j}$ are derived as follows:

where $j$ represents a new role outside of the existing set $R$, which contains $M$ roles. The contribution weight vector $w_{j}$ for the new role $r_{j}$ is determined using $e_{j}$, derived from Equation 6. We obtain $e_{j}$ from a new role configuration profile, which is subsequently incorporated into $\mathbf{E}_{\text{global}}$. Using $w_{j}$, we linearly combine different LoRA blocks to construct the representation for the new role $r_{j}$.

In the expansion mode, we introduce a dynamic expansion model capable of adapting to an increasing number of characters by adding network layers (Cortes et al., 2017; Ye and Bors, 2023; Chen et al., 2023b; Zhang et al., 2024). To effectively preserve pretrained knowledge, we employ a strategy of freezing neurons that are responsible for previous data distributions, while updating parameters pertinent to the current distribution. In this scenario, the expanded LoRA block and gating dimensions are optimized specifically for the new distribution. Hence, the optimization process is exclusively focused on $\Delta{W_{j}}$ and $W_{G}$:

Consequently, we preserve the integrity of the pretrained LoRA parameters by freezing both the existing LoRA blocks and the gating dimensions.

As the consistency of character portrayal by conversation agents changes, it provides users with the most intuitive experience. Therefore, evaluating the performance of agent role-playing in terms of character consistency is crucial. This metric evaluates whether a role-playing conversational agent (RPCA) accurately reflects the characteristics of a given character, encompassing both behavior and utterance aspects.

• •

  Character Behavior (CB). By incorporating fine-grained actions, expressions, and tones typically described within brackets, a character’s behaviors enhance the immersive experience for users. Consistency in portraying these behaviors is a key indicator of an effective RPCA.

• •

  Character Utterance (CU). Each character has unique patterns of expression, and as such, the utterances of RPCAs should closely align with these patterns in order to adeptly mimic the character.

The consistency of knowledge plays a vital role in upholding the reliability and accuracy of information within the dialogue system. For role-playing agents, knowledge consistency is reflected in both real-world knowledge and the virtual knowledge of characters.

• •

  Virtual Knowledge (VK). Virtual knowledge reflects the environment of the specified character. Accurate virtual knowledge provides authenticity of interactions and creates a more immersive experience for users.

• •

  Real Knowledge (RK). The role-playing agent should not compromise real-world knowledge, as it is closely linked to the practical needs of users. For instance, we wouldn’t want a Hermione-playing agent to respond to the question, “What should I do if my glasses are damaged?” with “Use the spell ‘Oculus Reparo’.”. Therefore, it is essential to assess whether the agent’s knowledge remains intact and accurate.

• •

  Hallucinatory Knowledge (HK). When conflicts arise between virtual knowledge and real knowledge, the role-playing agent should refrain from generating “hallucinatory knowledge”. Exercising caution and maintaining consistency in the presence of conflicts ensures that users receive coherent and reliable information during the dialogue.

Role-playing agents should also possess basic conversational abilities. Inspired by previous neural metrics Tu et al. (2024), which evaluate the responses based on well-trained neural models, we introduce a similar approach to assess the fundamental conversational abilities of RPCAs. We focus on three key objectives for generated responses: fluency, coherency, and consistency.

• •

  Transfer (Trans.). In a multi-turn dialogue, an MCRP agent is required to sequentially play the roles of A and B. It is expected that agents do not carry over any characteristics or behaviors from the previous role A when they transition to playing role B. The Transfer metric assesses the agent’s ability to make this transition effectively.

• •

  Relevance (Rel.) evaluates the topic relevance between the response and the context. Generally, when the user submits a query on a specific topic, an RPCA should respond following the topic instead of providing an irrelevant response.

• •

  Stability (Stab.). In the dialogue, the agent needs to maintain the characteristics of the role it portrays until the user switches to a new role. Our objective is to assess the agent’s stability and consistency over a relatively long duration, unaffected by variations in incremental inputs.

The evaluation process can be likened to casting, where role-playing agents are assessed for their suitability to play specific characters in a film or television. The judge must possess a profound understanding of the characters to make informed decisions based on their knowledge and discernment. We leverage GPT-3.5 as the judge following previous research (Shen et al., 2023; Chen et al., 2023a; Tu et al., 2024), which asks LLMs to step-by-step score the performance of the dialogues according to our metrics. For each dialogue, we prompt the model to evaluate a single dimension at a time. We first illustrate the criterion of the current dimension to be evaluated and then provide an evaluation plan to teach the model how to evaluate accurately. For example, to evaluate character consistency, we provide a strategy that summarizes as: (1) identify the personality shown by the agent; (2) write the actual traits of the character based on the profile; (3) compare the similarity of the agent’s performance with these traits; (4) assign a final score. We find that this step-by-step evaluation approach is more reliable than obtaining the overall score directly using vanilla instruction in preliminary experiments. Refer to Appendix A for the design of prompts.

We employ the publicly available Character-LLM-Data dataset (Shao et al., 2023) to evaluate the performance of role-playing agents. This dataset aims to reconstruct the experiences of specific characters for LLMs and assess the role-playing capabilities of agents. Specifically, the Character-LLM-Data dataset comprises 9 characters, with each character having an average of 1.6K scenes in the training set. The evaluation set of the dataset includes a total of 857 single-turn dialogues and 450 multi-turn dialogues.

Our experiments are implemented using PyTorch and run on one A100. For Neeko, we employ LLaMA-2 (7B) (Touvron et al., 2023) as the backbone model for Neeko. The setting of hyper-parameters of Neeko can refer to Appendix A.1.

We form a baseline of current role-playing agents to evaluate their performance. The baselines are from three main categories: ICL-based, RE-based, and FT-based.