Understanding and Predicting User Satisfaction with Conversational Recommender Systems

Research on recommendation via conversational interactions with information retrieval systems is increasingly receiving attention from both industry and academia. With multi-turn interactions, a CRS enables users to find their most relevant recommendations [24]. The CRS can interactively elicit users’ current preferences from their feedback and build a more complete user model to make better recommendations. Conventional recommender systems, on the other hand, only support a single interaction mode, i.e., displaying a set of suggestions depending on users’ historical activities [56]. Some older CRSs took advantage of user interface elements, such as critiquing-based systems [12], where users give input on suggestions by picking from a list of pre-defined criticisms [33]. Nonetheless, recent developments in natural language technology have led to more interest in developing a CRS based on conversational user interfaces (CUIs), where users can converse with the recommender system [38]. Several other approaches have been explored to enhance the effectiveness of recommendations, such as knowledge graph integration [74], prompt learning [66], and topic guidance [75].

The evaluation of CRSs is based on offline experiments that try to simulate a user’s behavior relying on their past interaction data. One line of research evaluates the performance of a CRS based on how well it accomplishes the user’s goal by making relevant recommendations using metrics such as TS and recommendation accuracy. Another line of work focuses on dialogue generation aspects, assessing the quality of the responses using word-overlap metrics such as the ROUGE score [17]. However, as argued by Deriu et al. [17], such individual measures do not reflect the overall quality of the system. Thus, current evaluation metrics that rely heavily on the system’s utility do not provide us with information about the evaluation findings in practical settings. On the other hand, research shows that empirical studies conducted using user-centric approaches can accurately assess the system’s performance in actual scenarios [4]. Ideally, a system should be assessed separately on each specific dialogue-level aspect to capture its performance on individual aspects [60].

So far, little work has been done to establish the relationship between dialogue aspects and overall response and system quality [60].

Kelly [39] defines user satisfaction as the fulfillment of a user’s specified desire or goal. User satisfaction has gained popularity as an evaluation metric of IR systems based on implicit signals [29, 35, 40, 41, 42]. In IR, user satisfaction is usually estimated based on the user’s interaction experience and goal fulfillment [39]. Factors such as system effectiveness, user effort, characteristics, and expectations influence a user’s satisfaction rating in IR systems [1]. Dialogue systems are often evaluated on their overall satisfaction [17], where users give their satisfaction rating at the turn and dialogue levels [10, 62]. Though subjective, user satisfaction provides valuable insights into users’ perceptions, preferences, and overall evaluation of a system’s performance. Additionally, it is a widely used and accepted metric in user experience research [see, e.g., 7, 29, 42].

However, for task-oriented dialogue systems such as CRS, which should optimize toward recommendation and user experience, overall satisfaction does not capture the broad and diverse aspects influencing a user’s satisfaction [60]. Thus, in this research, we seek to investigate this concept.

Due to poor correlation between automatic metrics such as BLEU and human judgment, accurate evaluation of dialogue systems rely on human evaluation [46]. Non-task-oriented dialogue systems are evaluated on specific aspects such as relevance and engagingness [51, 64]. However, task-oriented dialogue systems are often limited to estimating the user’s overall satisfaction [41, 62]. Recent research suggests that user satisfaction is multifaceted and subjective, thus, should not be reduced to a single label [64].

Several recent studies have proposed to evaluate dialogue systems at an aspect level. For example, one would measure the performance of a system in making relevant or understandable responses, instead of the overall quality of the response. PARADISE [65] is one of the first popular evaluation frameworks that decoupled a dialogue system’s task requirements from its behavior. With predictive factors such as TS, dialogue efficiency, and dialogue quality, a system’s effectiveness can be measured without having to collect user satisfaction ratings. Walker et al. [65] propose a framework for evaluating dialogues in a multi-faceted manner. They measure several dialogue aspects and combine them to estimate user satisfaction [65]. Mehri and Eskenazi [51] develop an automatic metric for evaluating dialogue systems at a fine-grained level, including interestingness, engagingness, diversity, understanding, specificity, and inquisitiveness. In their study, Venkatesh et al. [64] investigate the performance of multiple dialogue systems involved in the Alexa competition on several dialogue aspects and show that different systems perform well in specific dialogue aspects. Moreover, they show that no single measurement can be used to evaluate the overall performance of a system accurately. Several other studies have been carried out on human evaluation of multiple dialogue aspects [see, e.g., 19, 50, 59, 69, 72].

Predicting user satisfaction is critical in capturing whether a user’s goal has been fulfilled or not. In web search, user satisfaction is viewed as a subjective measure of a user’s experience during search [39]. Different from traditional IR relevance measures, such as precision and recall, user satisfaction takes into account both TS and user interaction experience [29, 30, 41]. For search systems, rich user interaction signals such as clicks, dwell time, and mouse scroll events are used to predict a user’s satisfaction [35, 40]. Such interaction signals cannot be collected from dialogue-based systems whose main interaction is through natural language, either in text or spoken. Research on spoken dialogue systems, such as intelligent assistants, has addressed this challenge by suggesting the use of features such as spoken implicit features, intent-sensitive query embeddings, and touch-related features, showing their effectiveness in predicting user satisfaction [29, 41]. Several other features have been suggested in line with text-based dialogue systems including implicit dialogue features, user intent, utterance length, and user-system actions, and proven to be effective [10, 62]. Bodigutla et al. [6] demonstrate the effectiveness of traditional machine learning models in predicting user satisfaction. Using predicted turn-level ratings with implicit dialogue features, models such as gradient boosting classifiers demonstrate competitive performance [6]. In task-oriented systems, several publications predict user satisfaction from turn-level overall quality user judgment ratings [62], user intents [10, 53], and implicit features such as utterance length and sentiment analysis.

Despite the success of related work in predicting user satisfaction with task-oriented systems, there has been less focus on trying to understand which dialogue aspects have an effect on a user’s satisfaction with these systems. Recent work by Siro et al. [60], tries to understand the relationship between several dialogue aspects and overall user satisfaction in a TDS, especially at the dialogue level. However, compared to related work, our work in this article is different in a number of ways: (i) unlike Siro et al. [60], who focus on dialogue-level user satisfaction, in this work, we establish the relationship between turn- and dialogue-level user satisfaction; (ii) we show the effectiveness of the dialogue aspects in estimating user satisfaction by experimenting with several classical machine learning models; and (iii) we increase the data sample size by re-annotating data from [60] with one more aspect (turn-level satisfaction) and annotating an additional 160 dialogues and 480 turns. Thus, in total, we have 200 dialogues and 600 turns annotated.

The utility dimension focuses on the objective nature of a CRS, that is to make relevant recommendations and accomplish a user’s goal. In this dimension, we investigate two qualities, namely, relevance measured at the turn level and task completion measured at the dialogue level.

In the UX dimension, we assess how different dialogue aspects of a CRS during interaction could affect a user’s satisfaction. The ideal requirement would be a system that interacts in a natural way with the user, making the interaction experience pleasing. Thus, inspired by related work [51, 60, 64, 72], we investigate the interestingness, understanding, interest arousal, and efficiency aspects.

The ReDial dataset [44] is a conversational movie recommendation dataset. It consists of 11,348 dialogues and the dataset are collected following the Wizard of Oz approach, i.e., one person acts as the movie seeker, while the other is the recommender. The dialogues are both system and user-initiated. The movie seeker should explain their movie preferences based on the genre, actor, and movie title and ask for suggestions. The recommender’s role is to understand the seeker’s movie taste and intent and make the right suggestions to the user. Due to this back-and-forth process of eliciting a user’s preference, which mostly involves chit-chat, this dataset is categorized as both chit-chat and goal-oriented, thus allowing us to investigate dialogue aspects from both the utility and UX dimensions of a CRS.

Unlike previous work [50, 51, 62], the annotators in our study have access to the user’s current utterance. We treat the response quality annotation as a turn-level task. Considering the interactive nature of a CRS, a turn is defined as a single exchange between the user and the system [62]. Unlike previous work, a turn, in this case, consists of two exchanges between the user and the system. Therefore, we define a turn in this work aswhere U is the user utterance, S the system utterance and i is the current response position. In a recent study [62], turn-level annotation is conducted with workers having access to all previous system and user utterances up to the current system utterance as context and their role is to assess if the user would be satisfied with the current system response given the context without viewing the user utterance at position i. This approach requires annotators to understand the user’s intent during the interaction and make judgments based on previous interactions. We argue that a user has a dynamic preference and intent during dialogue interactions and this can change from turn to turn, thus affecting their overall satisfaction of the system. In order to remedy this, we ask annotators to exclusively rely on the user’s current utterance while making judgments on the dialogue aspects. That is, for each system response \(S_i\) to be annotated, the annotator has access to the previous user (\(U_{i-1})\) and system (\(S_{i-1})\) utterances as context and the current user utterance \(U_i\) from which they should make their judgment. In this way, we aim to limit annotators’ bias, in that instead of annotators making judgments influenced solely by their own opinions, they reflect the opinions of the actual user as closely as possible.

Following Mehri and Eskenazi [51], we hand-selected three system responses from each conversation for turn-level annotation. In order to ensure three responses cover most of the dialogue, we only select dialogues with at most fifteen turns. We limit the context window to two such that each annotated response (\(S_i\)) has two previous utterances from the system (\(S_{i-1}\)) and the user (\(U_{i-1}\)) as context plus the current user utterance (\(U_i\)). This way, we ensure that an annotator does not have to keep track of a long conversation context when annotating a single response and each response has a reasonably long context during annotation.

For each response, we ask the annotators to assess them on relevance and interestingness and based on their ratings for the two aspects give their turn-level overall impression (satisfaction) rating as shown in Figure 1. The questions the annotators answered in this subtask are:

As our annotators are not actual system users, we ask them to base their judgments solely on the next user’s utterance to make the label judgment. For example, if the user states, “I do not like that movie,” an annotator should be able to judge the system’s response and recommendation as irrelevant since the suggested movie does not meet the user’s needs. For “I have seen that and like it” the response should be rated as relevant. For the overall impression rating, we ask the annotators to base their judgment on the relevance and interestingness aspects. Each aspect comes with three options, namely, No, Somewhat, and Yes. For relevance, we also provide a Not applicable option when a system response does not contain a movie suggestion (e.g., if the system chit-chats or tries to elicit a user’s preference). Due to limited annotation resources, we chose to focus on relevance and interestingness as the primary aspects for turn-level annotation, as they provide a strong foundation for evaluating the quality of recommendations in CRSs.

At the dialogue level, we ask the annotators to assess the quality of the entire dialogue based on four aspects: understanding, task completion, efficiency, and interest arousal. We instructed the annotators to answer the following questions:

Understanding and task completion are rated on a scale of 1–3 with the options of No, Somewhat, and Yes. Interest arousal is judged on a 4-point scale with a Not Applicable option for when no novel movie is recommended to the user or a novel movie is recommended but the user does not follow up about it. Lastly, efficiency is assessed on a binary scale [20, 42] where the system has either made a recommendation meeting a user’s request within the first three turns or not. Following [51, 60], we also ask annotators to rate the entire dialogue on overall impression using a 5-point Likert scale based on their turn and dialogue levels aspects’ ratings. Finally, we ask the workers to justify their rating on overall impression in a few words. We use the justifications to contextualize the given ratings and analyze and discover additional aspects that affect the quality of dialogue, as shown in Table 4.

Here, we describe the demographics of our participants, followed by more details on the collected data and the measures we took to ensure the high quality of the data.

At each turn, the aspects relevance, interestingness, and overall turn quality are rated. We show the distribution of the ratings for these aspects in Figures 2(a), 2(b), and 3 for relevance, interestingness and turn-level satisfaction, respectively. Note that the distributions in Figure 2 are computed over the three annotated turns in each dialogue. We can see that around \(25\%\) of the turns were annotated as not containing any movie recommendation (\(R=1\)), while over \(40\%\) are annotated as very relevant. This result is not surprising because of the nature of the ReDial dataset, where a recommender system needs to elicit a user’s preference before making a suggestion, thus having multiple chit-chat turns. Meanwhile, turning to Figure 3, we observe that turns rated as very relevant and interesting at the same time overall led to a satisfactory turn, showing that CRS though goal-oriented should not only focus on making relevant recommendations but also in a natural and interesting manner.

Figure 4 shows Pearson’s r between turn-level user satisfaction and (i) relevance (annotators assess if the recommended movie meets the user’s preference), (ii) interestingness of system’s response. Also, we report the correlation between relevance and interestingness in the figure. We note that the relevance and interestingness aspects have a moderate positive correlation with each other (\({\sim }\)0.4). However, we see that relevance exhibits a higher correlation with overall turn impression than interestingness. Our analysis indicates that when a turn is rated as relevant, the turn’s overall impression is more likely to be satisfactory (\(96\%\) of the relevant turns).⁴ On the other hand, the same does not hold for turns rated as irrelevant (\(43\%\) of the irrelevant turns led to a satisfactory dialogue), suggesting that in this case, the user’s overall impression depends not only on relevance but on other dialogue aspects too such as response interestingness.

In summary, we note that at the turn level, the relevance and interestingness aspects are important in understanding a user’s satisfaction. Specifically, we can rely on the relevance aspect to identify Sat responses, while interestingness can be used to identify DSat responses. Characterizing the relationship between these two classes could be useful in the automatic estimation of response quality.

Table 1 reports Spearman’s \(\rho\) and Pearson’s r correlation coefficients of all six quality aspects, including turn-level satisfaction (TSat), with the overall dialogue satisfaction rating. Since three turns were annotated for each dialogue, we report the average results over all three turns for the three aspects. Note that both relevance and turn-level satisfaction have a moderate correlation (second row) with the overall dialogue satisfaction ratings. Compared to interestingness, relevance has a higher correlation, confirming our previous findings [60].

Notice that the turn-level satisfaction rating exhibits a high correlation with dialogue-level user satisfaction. This indicates that one can use a single overall turn-level quality metric to estimate a user’s overall dialogue satisfaction, which has been used in previous studies [62]. We also do a correlation analysis on each turn separately and note that both relevance and turn-level satisfaction achieve a high correlation in their third and last interaction turn compared to the other two previous turns. This shows that a system’s success in making a successful suggestion⁵ in the final turn has more weight on the overall impression than the preceding turns. This conforms to the findings of [42, 47, 60], showing that the latest interactions with a system have more influence on the overall satisfaction of users.

At the dialogue level, interest arousal achieves a high Spearman’s \(\rho\) coefficient while task completion achieves a high Pearson’s r coefficient, as shown in Table 1 (third row). Efficiency is the least correlating aspect for both scores. In our study, this aspect captures the system’s ability to make relevant recommendations meeting the user’s need within the first three exchanges. Unlike chatbots, which are meant to engage with a user for a long period, TDS dialogues should be concise and efficient [24].

In Figure 5, we plot the distribution of the ratings for the dialogue-level aspects against the overall impression. We see a clear dependency of the overall impression on the task completion aspect; out of the dialogues classified as satisfactory, \(68\%\) were rated high in terms of task completion (see Figure 5(a)). We also notice that most dialogues rated low (\(=1\)) in terms of task completion are unsatisfactory overall with a few outliers. Thus, we conclude that the ability of a CRS to complete a user’s specified task can be the determinant of the overall impression.

We see in Figure 5(b) that more dialogues are rated efficient than inefficient (\(72.5\%\) vs. \(27.5\%\)). We note that an efficient system, making suggestions meeting a user’s need within three turns, leads to a satisfactory dialogue. Our analysis, however, indicates that the opposite cannot be said for inefficient dialogues: most of them were rated satisfactory (\(61.5\%\)). We note from the annotators’ open comments that even though a system took extra turns to make a relevant suggestion, as long as the user got a suggestion, they rate the system as satisfactory. This indicates that a system that fails to satisfy the user’s need in the first three interactions is less likely to do so in further interactions.

To understand the significance of the investigated dialogue aspects to the overall impression, we train various regression models considering different aspect combinations (both single and multiple aspects) and report their \(R^2\); see Tables 2 and 3 for the results. \(R^2\) represents the coefficient of determination for the regression model, which indicates the proportion of the variance in turn and dialogue level satisfaction that is explained by the independent and combined aspects [11]. At the turn level, an approach that combines both aspects outperforms the best turn-level single aspect (relevance). As for the dialogue-level aspects, interest arousal exhibits the highest significance among all other aspects, taken individually. The combination of dialogue-level aspects clearly shows a stronger relationship to the overall rating model than individual aspects. Unsurprisingly, combining all aspects performs better than individual aspects or different levels.

Tables 1 and 3 show that dialogue-level aspects have a bigger influence on the overall impression than turn-level aspects. This suggests that turn-level aspects cannot be used solely to estimate the user’s overall satisfaction effectively. This is attributed to cases where a system’s response at a turn is sub-optimal, thus not representing the entire dialogue impression. The turn and dialogue aspects concern two evaluation dimensions: utility and user experience. Relevance and task completion measure the utility of a TDS, i.e., its ability to accomplish a task by making relevant suggestions. The user experience dimensions (understanding, interest arousal, efficiency, and interestingness) focus on the user’s interaction experience. The combination of dialogue aspects from both dimensions has a strong relationship with the overall impression, unlike the individual aspects. In Table 3, the columns Utility and User experience show the two dimensions: combining both dimensions (the last row in each section in Table 3) leads to the best performance. The combination of turn and dialogue level aspects (D\(+\)T, third group) achieves the highest \(R^2\). In summary, leveraging aspects from both dimensions (utility and user experience) is essential when designing a TDS that is meant to achieve a high overall impression.

Analyzing annotators’ open-comments. To identify additional dialogue aspects that influence a user’s satisfaction with a CRS, we conduct a manual inspection of the worker’s open comments. We only report aspects based on dialogue-level user satisfaction.

We go through the comments and assign them to evaluation aspects based on the worker’s perspective. For example, a comment that mentions, “the system kept recommending the same movie” signals the existence of a novel aspect that concerns repeated recommendations in a dialogue. Table 4 lists the (dominant) novel categories discovered from the comments, together with a gloss and example. Several notable aspects are observed by the annotators. For instance, most annotators dislike the fact that the system expresses its opinion on a genre or movie. In cases where the system is repetitive (in terms of language use or recommended items), the annotators’ assessments are negatively impacted. This observation is in line with [25], where they show that overexposure of an item to a user in a short time period leads to a drop in user satisfaction. Some annotators note the positive impact of dialogue being natural and human-like or that the system makes a good recommendation after several failed suggestions (i.e., success on the last interaction). There are some examples where all annotators agree that the suggestions are good, but the user does not react rationally.

To summarize, in this section, we have first established the relationship between several dialogue aspects with user satisfaction. We have then conducted an analysis of the annotators’ open comments to identify additional aspects that could influence a user’s satisfaction. We conclude that at the turn-level relevance is the most important aspect whereas, at the dialogue level, the ability of the system to generate a user’s interest and accomplish a task is significant in determining a user’s overall satisfaction with a CRS. Therefore, we notice that the proposed dialogue aspects influence users’ interaction with CRS differently. For some, a relevant recommendation has more effect on their overall rating whereas others consider the ability of the system to make relevant recommendations in a natural way as the most important factor influencing their overall satisfaction. Thus, user satisfaction is subjective to individual users, and the design and development of CRS should cater for personalization to individual users.

Task success [58] is a measure used in the evaluation of dialogue systems. This metric evaluates the quality of a dialogue with the assumption that users only care about their tasks being accomplished at the expense of interaction quality (IQ). Since an annotator has to accurately determine a user’s intended task, the metric is not accurate enough to estimate the quality of a dialogue response. Differently, in this work, we choose turn-level satisfaction (TSat) to determine the overall quality of a response in a dialogue. TSat estimation requires each turn to be annotated at a 5-point Likert scale. Unlike Sun et al. [62], who obtain the overall response quality at each turn, our annotation scheme requires annotators to rate three randomly sampled responses from each dialogue on two dialogue aspects, namely, relevance and interestingness. Then, we ask them to provide their overall quality rating. Response quality estimation could be used to identify the effect of a certain response on overall user satisfaction from a user’s perspective.

We now investigate RQ3 in this section: How effective are dialogue aspects in estimating user satisfaction compared to turn-level satisfaction ratings? In previous work, dialogue-level user satisfaction for task-oriented systems has been estimated leveraging rich signals such as user intents, dialogue acts, turn-level satisfaction ratings, and implicit turn and dialogue features [41, 62]. One major limitation of estimating overall user satisfaction using turn-level satisfaction ratings is the inability to capture specific aspects influencing a user’s overall impression with a dialogue system. In this work, we propose to estimate overall user satisfaction from several dialogue aspects annotated in Section 4. We report on a performance comparison between the two approaches and show that estimating user satisfaction from dialogue aspects leads to a better-performing model.

Figure 6 shows the distribution of human-annotated response quality ratings. We note that \(62\%\) of the turns are Sat (rating \(\gt 3\)) compared to DSat (\(38\%\)) (rating \(\le 3\)). Turn-level satisfaction prediction is very useful in online evaluation for identifying a problematic turn in a dialogue, thus allowing the system to adjust its recommendation or dialogue policy to avoid total dissatisfaction of the user by recovering from errors during the conversation.

At the turn level, the aim is to estimate the quality of the response from the annotated turn-level dialogue aspects, thus, we utilize graded satisfaction prediction in this task. We compare the performance of various regression models in estimating a user’s response quality rating given the relevance and interestingness ratings for the current turn and report the results in Table 5. Evidently, all models perform comparatively well in estimating the user rating of each response. We note that ensemble models seem to learn a good representation of the aspects and improve their predictive performance compared to single models. The performance of traditional machine learning models is a clear indication that turn-level aspects can be used to estimate the quality of a response in cases where we do not have the user’s turn-level satisfaction rating.

We also report the correlation coefficient between the predicted labels and the ground truth labels for each model. Among the six models we experimented with, RFR achieves the highest correlation coefficient (0.7337) followed closely by DTR at 0.7234. Our analysis of the predicted labels reveals that in most cases, the models predict accurately or close to the ground truth label for satisfactory dialogues compared with dissatisfactory dialogues. Identifying turns where the system fails is a difficult task due to label imbalance, as the majority of the turns are rated as satisfactory. It is worth noting that identifying dissatisfactory turns is more important in order for CRSs to adjust their interaction policy and avoid total user dissatisfaction.

In summary, extensively experimenting with the dialogue aspects as features, we conclude that both relevance and interestingness are important in predicting the quality of a response with CRS. We note that the random forest regressor achieves a high correlation coefficient of 0.7337 compared to other models. Thus, in cases where we do not have access to user’s response quality ratings, we can rely on dialogue aspects such as relevance and interestingness to estimate the quality of a response.

To show how effective the proposed dialogue aspects are in predicting user satisfaction, we report the results for several classical machine learning models on user satisfaction prediction. First, we predict overall user satisfaction from turn-level satisfaction ratings (see Table 6). Second, we experiment with a combination of turn- and dialogue-level aspects separately (see Table 7). Finally, to show the effectiveness of our proposed dialogue aspects, we predict user satisfaction from all the proposed dialogue aspects (see Table 8.)

Table 6 shows the performance of several machine learning models in predicting user satisfaction from turn-level satisfaction ratings. We report the evaluation metrics for both the Sat and DSat classes, except for the correlation coefficient so as to capture the performance of the models in predicting a dissatisfactory dialogue. This is because identifying a problematic dialogue is of more importance for system designers to improve the model’s performance for the next interaction. We note that for the Sat class, the models perform better in Prec, Rec, and F1 metrics than the DSat class. In terms of F1-DSat and Spearman’s \(\rho\), logistic regression is the best-performing model. This model classifies all the predicted satisfactory dialogues accurately as it achieves a recall score of 1.00 compared to 0.56 for dissatisfactory dialogues. Apart from having the limitation of dataset size representing dissatisfactory dialogues, it indicates that it is challenging for the model to identify dialogues where the user is dissatisfied since most of the data represent positive dialogues. Thus, understanding dialogue aspects that can easily be used to identify problematic dialogues is useful.

Additionally, we note that predicting user satisfaction from turn-level satisfaction ratings does not lead to a good performance for the DSat class. This demonstrates that user satisfaction ratings at each turn are not optimal in estimating whether a whole dialogue is dissatisfactory or not. We hypothesize that all turns are not equally weighted by the users when determining their overall satisfaction. Our experiments on predicting user satisfaction from individual turns reveal that the last turn is more important compared to the other turns in predicting user satisfaction. This indicates that the ability of the system to have a successful last interaction impacts a user’s overall impression.

In Table 8, we observe an increase in the performance of F1-DSat when we predict user satisfaction from all the annotated dialogue aspects. For precision, random forest performs better in the DSat class, both decision tree and GNB are the best-performing models in terms of recall, with random forest and SVM scoring a high F1-DSat. The predictions of random forest have a high correlation score with the ground truth labels, followed closely by SVM predictions. Although we do not experiment with neural architectures to allow us to model the dialogue context, all models indicate a comparative performance in predicting user satisfaction from dialogue aspects with moderate correlation scores. Thus, this implies that traditional machine learning approaches can be leveraged in user satisfaction prediction and we can rely on dialogue aspects ratings to predict user satisfaction and get comparative results without context modeling and additional implicit features.

Taking the three best-performing models from Table 8 (SVM, RFC, and GBC), we experiment with predicting user satisfaction using turn- and dialogue-level aspects and report the results in Table 7. GBC performs better in terms of F1-DSat for both the turn and dialogue levels, 0.71 and 0.67, respectively. All models perform better for precision, recall, and F-1 for the Sat class. We note a superior performance when predicting user satisfaction with the dialogue level aspects compared to the turn level aspects suggesting dialogue level aspects benefit the models more in identifying satisfactory dialogues. The DSat class seems to benefit more from the turn-level aspects when combined with turn-level satisfaction as we observe a high F1-DSat from this level. It is worth noting that, though we observe a high F1-DSat when predicting user satisfaction from the turn-level aspects, GBC and RFC from the dialogue-level aspects (see Table 7 row 5) achieve a high recall score for the DSat class showing their capability to accurately classify the predicted dialogues as dissatisfactory compared to the methods using turn-level features. We also report the correlation coefficients in Table 7 and note a comparative performance for GBC in both turn and dialogue level aspects.

Feature analysis. Since we experiment with several combinations of the aspects, we treat the aspects as our input features and conduct a feature importance analysis using RFC. As we report our result per class (i.e., Sat and DSat) we also report the importance of each feature based on each class, in addition to overall satisfaction prediction.

Figure 7 shows the significant percentage of features for (a) the Sat class and (b) the DSat class. The ability of the system to arouse a user’s interest to watch an unseen movie is the most significant feature for Sat class. We note a five percent gap between the most significant feature (Interest arousal- \(16\%\)) and the second most important feature (turn-overall3 at \(11\%\)). Closely followed by turn-overall1, task-completion and relevance1. This indicates that in order for a CRS to improve a user’s interaction experience, it should create a good impression to the user at the start and end of a conversation.

We see that a user’s overall impression in turn two is the most significant feature in predicting user dissatisfaction for the entire dialogue, as shown in Figure 7(b). Followed closely by relevance2, interestingness3, task-completion and interestingness2. Out of the top five features, we note that 3 of them are rated at the second turn that is turn-overall2, relevance2, interestingness2. In most dialogues, we examined, recommendations start at this turn after preference elicitation in turn one. If a system fails to capture a user’s preference in the first turn, in most cases, it leads to an irrelevant recommendation being made, resulting in overall dissatisfaction. In order to improve the performance of the system at turn two, the system should be more understanding toward a user’s request and preference. The features, efficiency, turn-overall3, turn-overall1, and interestingness1 are the least significant in the prediction of the DSat class.

In general, we note that combining features from both the utility and user experience dimensions leads to a better user satisfaction measurement. In both the Sat and DSat classes, turn- and dialogue-level aspects are important. For Sat, the strong signal is interest arousal, which is measured at the dialogue level, whereas turn-level satisfaction at response two (turn-overall2) is the strongest in the DSat class. Evidently, we can conclude that features from both the turn and dialogue levels are important in determining satisfactory and dissatisfactory dialogues with CRSs. Therefore, based on results from Tables 6 and 8, we show that relying on only dialogue-level aspects to predict user satisfaction is as effective as using turn-level satisfaction ratings.

In this work, we have first focused on understanding user satisfaction with CRSs, generally categorized as a goal-oriented dialogue system. Although goal-oriented dialogue systems are ideally expected to optimize toward task accomplishment, in this study, we show that a system’s behavior during interaction has an influence on their overall satisfaction during interactions at both the turn and dialogue levels. The interestingness aspect, however, does not show a high correlation with turn-level satisfaction. We hypothesize that when asked to scrutinize a CRS response explicitly on interestingness, annotators tend to rate such responses less favorably than they would if they were rating the overall experience according to the established rating process. Though this aspect is highly researched for non-task-oriented dialogue systems [7, 50, 51], from both the annotations and open-ended comments, we find that engaging with users in the form of chit-chat has both positive and negative effects on their overall satisfaction. If a user is already happy with a provided recommendation, more engagement can lead to further interest arousal, and hence more satisfaction; however, if the system fails to meet the user’s expectations, it can have a negative effect. This is in line with [61], who stress the importance of finding the right amount of chit-chat in a goal-oriented dialogue.

Providing relevant recommendations throughout a dialogue is crucial for user satisfaction, but it does not tell the whole story. When a system makes relevant recommendations, they certainly lead to a satisfactory dialogue, but when the responses are both relevant and interesting most users tend to rate their experience as very satisfactory for both levels. This indicates that a CRS that can make relevant recommendations alongside generating natural responses that are interesting is more likely to result in an improved user’s overall interaction experience. Thus, system designers and dataset creators should consider optimizing these two aspects during the design and development of CRS systems and datasets.

Our analysis of the justifications that support a user’s overall satisfaction rating reveals new aspects that can affect users’ satisfaction. In line with our quantitative analysis and related work [42, 47], many annotators mention the importance of a good user experience in the final turns of a conversation. Success in the last interaction has an implication on task completion, interest arousal, and on overall user satisfaction. When a system accomplishes its predefined goal, users tend to utter responses such as “Thank you for the suggestion!” “It was nice chatting with you.” While utterances such as “But you did not get me something to watch” and “Such a waste of my time” indicate an inability of the system to fulfill a user’s need. Therefore, in various cases, we can rely on the last user interaction to assess the system’s ability to fulfill or not fulfill a user’s need. It is also worth mentioning that other aspects such as repeated utterances and recommendations negatively impacted the user experience.

In general, we note that the UX dimension (interestingness, understanding, interest arousal and efficiency) of a CRS plays a very important role toward user satisfaction. The ability of a CRS to make relevant recommendations and accomplish a user’s goal could lead to overall satisfaction, however, a system that demonstrates to be more engaging and understanding has a higher chance of satisfying users. This indicates the need to jointly optimize turn- and dialogue-level metrics and for a fine-grained model of user satisfaction that incorporates multiple aspects.

In this work, we rely on external assessors to judge user satisfaction based on the user’s utterances and reactions to the system’s responses. While we have observed a high level of agreement for most dialogues, we have also noticed disagreement between annotators on some. This limitation could introduce a potential gap between the assessors’ ratings and the subjective satisfaction levels of users in real-world scenarios. Additionally, interpretation biases among assessors can affect the reliability of turn and dialogue-level ratings. Therefore, it is essential to conduct this study with actual users so as to collect a set of fine-grained annotations from real users [49].

At the turn level, due to the substantial annotation effort required, following [51], we sample three responses from each dialogue for annotation. While this approach may not capture the full picture in a dialogue, we believe that our sampling strategy provides meaningful insights into what aspects influence turn-level satisfaction. Investigating the optimal way of selecting responses to annotate from each dialogue may provide additional useful findings, but this was not our concern in this study. Therefore, we think there is a rich research gap to solve the significant annotation effort required in dialogue annotations when all the turns are annotated.