“In Dialogues We Learn”: Towards Personalized Dialogue
Without Pre-defined Profiles through In-Dialogue Learning

Learning dialogue generation from a wide variety of reference dialogues is not always effective Bao et al. (2019), especially when we aim to capture the personality characteristics embedded in the dialogues. To enhance the efficacy of the process, static persona identification partitions the dialogues of a target person into multiple persona-relevant clusters (cf., Figure 2 left). Hence, within each persona-relevant cluster, IDL can learn more meaningful mapping from reference dialogues to target dialogues. The challenge then lies in how to measure the distance between the dialogues across persona dimensions for effective dialogue clustering.

We employ a public dataset PersonaExt Zhu et al. (2023) and train a persona extractor to recognize persona-intensive utterances in a dialogue corpus. PersonaExt segregates persona information within dialogues into triples of <subject, relationship, object>. The dataset defines $105$ types of relationships. Based on the dataset, we develop the persona extractor (abbreviated as Ext) to directly extract the triples from the dialogue. Then, the extracted objects are used to locate the persona-intensive utterances. We formulate the extraction process as

$$\{p_{j}^{(u,v)}\}_{j=1}^{n}=\text{Ext}(d^{u,v}),$$

(2)

where $p_{j}^{(u,v)}$ denotes a persona-intensive utterance in dialogue $d^{u,v}$. The extracted utterances are then transformed to a vector $z^{(u,v)}$ by

	$\displaystyle p^{(u,v)}$	$\displaystyle=\text{Concat}(p_{1}^{(u,v)},\dots,p_{n}^{(u,v)}),$		(3)
	$\displaystyle z^{(u,v)}$	$\displaystyle=\text{Enc}(p^{(u,v)}),$

where we utilize the sentence-embedding model as the Enc¹¹1https://huggingface.co/sentence-transformers/all-mpnet-base-v2. Based on $\{z^{(u,v)}\}$ and the euclidean metric, $\mathbb{D}^{u}$ is clustered by k-means algorithm:

$$K^{u}=\text{KMeans}(\{z^{(u,i)}\},c),$$

(4)

where $c$ is the number of clusters. Subsequently, within each cluster $K_{j}^{u}\in K^{u},j=1,2,\dots,c$, we randomly select a dialogue as the target dialogue while the closest top-$k$ in the remaining dialogues are regarded as the reference dialogues.

Following static persona identification, we gather persona-relevant reference dialogues along with a target dialogue for optimization within each cluster. While we could directly concatenate these reference dialogues as input for the model, determining the optimal sequence remains a challenge. Our goal is to merge these dialogues into a cohesive long-term conversation, as we recognize that an inappropriate sequence could negatively affect the structure of the dialogue Chen et al. (2023b).

To achieve the goal, we compute the optimal order which could minimize the overall semantic distance between adjacent dialogue sessions in the long-term conversation. This approach ensures a smoother transition in the ongoing dialogue.

To quantify the semantic distance between dialogues, we introduce Conversation Edit Distance (convED) Lavi et al. (2021). The convED metric is akin to the traditional edit distance, but it modifies the basic unit of editing from characters to sentences within a dialogue. The metric aligns one dialogue with another through the processes of inserting, deleting, and substituting sentences. Detailed formulations of convED are presented in Appendix A.2.

Given a pair of dialogues $(d_{i},d_{j})$, the distance $dist_{i,j}=\text{convED}(d_{i},d_{j})$ measures the cost of aligning $d_{i}$ to $d_{j}$. Hence, by computing paired convED, we obtain a semantic distance matrix between reference dialogues in a cluster. Subsequently, we introduce Dijkstra’s minimum distance algorithm Dijkstra (2022) to re-order the reference dialogues based on the semantic distance matrix and compute the optimal order.

In each cluster of $K^{u}$, we concatenate the reference dialogues according to the optimal order and split the target dialogue with the last utterance as a response and the remaining as the context. These data elements satisfy Equation 1, and we can optimize the LM by minimizing the negative likelihood loss. Above processes endow the model with basic IDL ability, which could generate personalized responses based on reference/historical dialogues.

Note that we utilize two kinds of distance in static and dynamic persona identification, where the former measures the personalized relevance and clusters the relevant dialogues of a target person, while the latter measures the semantic distance and re-orders the reference dialogues in a cluster.

The first consideration for preference alignment is Direct Preference Optimization (DPO) Rafailov et al. (2023). DPO distinguishes itself from conventional reinforcement learning algorithms by bypassing the need for reward models, thereby conserving time and computational resources and reducing the complexity typically associated with reinforcement learning. However, DPO encounters a challenge in the form of unstable training outcomes. This instability arises because the primary objective of DPO is to widen the gap between chosen and rejected examples, while it overlooks the diminishing rewards of the chosen examples. Thus, even when the disparity between chosen and rejected examples increases, it may be caused by a concurrent decrease in rewards for both chosen and rejected examples, ultimately leading to a diminished efficacy of the optimized model. This issue is referred as preference degradation.

To address this problem, DPOC incorporates a corrective measure by adding a penalty term $\mathcal{P}$:

$$\mathcal{P}(r_{w},r_{l})=-\min\left(0,\log r_{w}-\log r_{l}\right),$$

(5)

where $r_{w}$ is the reward of the better sample $y_{w}$ and $r_{l}$ is the reward of the worse sample $y_{l}$. In most cases, $r_{w}>r_{l}$ and $\mathcal{P}(r_{w},r_{l})=0$. However, when $r_{l}>r_{w}$, $\mathcal{P}(r_{w},r_{l})$ functions as the penalty term. This inclusion ensures that the optimized model does not significantly deviate from the initial model. Building upon the foundation of DPO, the loss function of DPOC is formulated as

	$\displaystyle\mathcal{L}_{DPOC}(r_{cho},r_{rej},r_{crt})$	$\displaystyle=\mathcal{L}_{DPO}(r_{cho},r_{rej})$		(6)
		$\displaystyle+\mathcal{P}(r_{cho},r_{crt})$
		$\displaystyle+\mathcal{P}(r_{crt},r_{rej})$

The criterion sample reward $r_{crt}$ typically serve as intermediary benchmarks between chosen sample reward $r_{cho}$ and rejected sample reward $r_{rej}$. They offer a reference point for the optimization process in DPOC. Specifically, if the reward from a chosen sample falls below that of a criterion sample, or if the reward of a rejected sample’s reward is unexpectedly high compared to criterion examples, the current model incurs a penalty, which is represented by $\mathcal{P}(r_{cho},r_{crt})$ and $\mathcal{P}(r_{crt},r_{rej})$, respectively. This mechanism contributes to alleviating the preference degradation problem.

In the context of personalized dialogue, we identify three distinct types of criterion examples (cf., Figure 2 right). Each of them utilizes persona information with inaccuracies. (1) Inconsistency: includes information conflicting with the persona established in the dialogue sessions. (2) Fabrication: introduces personality details not mentioned in the dialogue sessions. (3) Inversion: adopts the persona information of the other participant. Given dialogue sessions $\mathbb{D}^{u}$, the context of on-going dialogue context $C$ and a chosen sample $h_{cho}$ of the current response, the construction of the three types of criterion examples are detailed as follows:

In Table 1, we compare the proposed method with existing personalized dialogue generation methods on ConvAI2. From the results, we can conclude that (1) when equipped with IDL, an open-source LLM can significantly outperform the existing methods in terms of almost all metrics, implying that IDL offers an effective way for leveraging LLMs in the task of personalized dialogue generation. (2) IDL can successfully recover personality characteristics from dialogue sessions. This is supported by the comparison between LLaMA-2 IDL and LLaMA-2 gold. Even without any hints from the profiles, IDL can still achieve comparable performance to the models fully supervised by the profiles.

In Table 2, we present results of IDL and other LLMs of comparable size on Movie and LIGHT. All the baseline models engage in personalized dialogue through ICL. Based on the results, we observe that (1) ICL underperforms in personalized dialogue generation, indicating that while ICL can handle the textual structure of dialogue sessions, it fails to effectively utilize persona information within these dialogues and (2) LLaMA-2-7B IDL and LLaMA-2-13B IDL fine-tuned on ConvAI2 also perform well on Movie and LIGHT. This confirms that the success of IDL is not due to the optimization for a particular dataset; rather, it stems from the ability to effectively utilize persona information in dialogues.

We incorporate human evaluation to more accurately assess the quality of dialogues on three subjective dimensions: (1) Persona: evaluators will assess whether the response accurately and consistently reflects the persona information of the target person. (2) Style: evaluators will judge if the response aligns with the expected wording and tone for the target person. (3) Fluency: evaluators will examine the smoothness of the dialogue flow, considering both linguistic and logical fluency. We arranged the generated responses into pairs and conducted pairwise comparisons across these three dimensions.

Human evaluation results on ConvAI2 are shown in Figure 3. We sampled 500 pairs and engaged a professional evaluation group to perform the assessments. The two responses within each pair are produced from identical dialogue sessions and contexts, and the order of these two responses is randomized in the evaluation system. For each of the three dimensions mentioned previously, evaluators are required to assign a judgment of Win, Tie, or Lose based on the quality of these two responses.

The results show that IDL has brought significant improvements in both persona and style, with winning rates of 68.8% and 59.0% respectively, which demonstrates that the model using IDL can more effectively simulate the personality and tone of the target person. Regarding fluency, there is a slight decline in performance when using IDL, possibly attributed to the model’s increased focus on aligning with persona information.

Table 3 shows the ablation study results on Movie. In order to clarify the contribution of each IDL process to the overall effect, we gradually remove each process and get a list of variants: (a) w/o Criterion removes the criterion samples and uses standard DPO for persona alignment. (b) w/o DPA removes the whole persona alignment process. (c) w/o SPI further removes the static persona identification in the MSL stage on the basis of (b). (d) w/o DPI removes the dynamic persona identification on the basis of (c).

From the results, we observe that (1) DPOC plays a crucial role in enhancing the acquisition of better persona information, and the elimination of criterion samples significantly diminishes the model’s effectiveness. This is because the model can pay more attention to persona-related tokens after deep personalized alignment. Relevant case study can be found in Appendix A.3. Additionally, the findings suggest that merely employing DPO falls short in substantially improving the overall performance of models. This is because the preference alignment of DPO is not optimized for problems that can arise from personalized dialogue generation task, as illustrated in $\lx@sectionsign$ 3.3.2. Furthermore, the diminished effectiveness observed upon removing static and dynamic persona identifiers underscores the importance of reorganizing training data before the supervised fine-tuning process.

In this work, we make the model learn personality-related information from the dialogue sessions and generate personalized responses. We present the performance of IDL and ICL under different demonstrations (dialogue sessions) to compare the learning efficiency of them. Figure 4 illustrates that similar to ICL, with the increase in the number of dialogue sessions, there is a general improvement in the quality of responses of IDL. However, as a specialized learning method for dialogue, IDL exhibits a faster learning ability under different dialogue sessions than ICL, indicating the effectiveness of our proposed mutual supervised learning and deep personalized alignment. Benefits from these advancements, IDL paves a new road to develop and update dialogue systems in an online manner.