See discussions, stats, and author profiles for this publication at: [Link]

net/publication/381817955

Explanations in Open User Models for Personalized Information Exploration

Conference Paper · June 2024

DOI: 10.1145/3631700.3665188

CITATIONS READS
0 14

4 authors, including:

Rully Agus Hendrawan Peter Brusilovsky

Institut Teknologi Sepuluh Nopember University of Pittsburgh
86 PUBLICATIONS 386 CITATIONS 619 PUBLICATIONS 26,111 CITATIONS

SEE PROFILE SEE PROFILE

Jordan Barria-Pineda
University of Pittsburgh
30 PUBLICATIONS 305 CITATIONS

SEE PROFILE

All content following this page was uploaded by Peter Brusilovsky on 01 October 2024.

The user has requested enhancement of the downloaded file.

Explanations in Open User Models for Personalized Information
Exploration
Rully Agus Hendrawan
Peter Brusilovsky
Arun Balajiee Lekshmi Narayanan
Jordan Barria-Pineda
ruhendrawan@[Link]
peterb@[Link]
ARL122@[Link]
JAB464@[Link]
University of Pittsburgh
United States
ABSTRACT KEYWORDS
Open user models provide affordance for a transparent user con- Adaptive explanation; Information exploration; Open user model;
trol over recommendations based on shared symbolic represen- Intelligent interface; Concept graph
tation within the system. Users must build their user profile by
ACM Reference Format:
adding these symbols and tuning their importance to get meaning-
Rully Agus Hendrawan, Peter Brusilovsky, Arun Balajiee Lekshmi Narayanan,
ful recommendations. Since the link between these symbols and and Jordan Barria-Pineda. 2024. Explanations in Open User Models for Per-
the reference explanation is often unavailable, it can be difficult for sonalized Information Exploration. In Adjunct Proceedings of the 32nd ACM
users to understand them. These symbols are often referred to as Conference on User Modeling, Adaptation and Personalization (UMAP Adjunct
concepts, tags, areas, topics, labels, features, or keyphrases. This ’24), July 01–04, 2024, Cagliari, Italy. ACM, New York, NY, USA, 8 pages.
study showcases an information exploration system that helps stu- [Link]
dents identify potential faculty members to collaborate with. The
system works by matching user and faculty profiles that contain 1 INTRODUCTION
keywords or phrases representing topics/areas of interest. Students
must develop their understanding of research topics while building Information exploration systems, also known as exploratory search
their profiles, which can become challenging as they add more key- systems [25], provide users with an effective way to perform com-
words. To support students in controlling the recommendation, we plex searches and achieve long-term goals, especially when search-
introduce post hoc explanations with three levels of detail: no ex- ing co-occur with learning [15, 37]. Unlike more traditional recall-
planations, individual explanation for topics, and explanation of the based search systems, which rely on the user’s ability to remember
relationships between topics. This study explores how explanation specific keywords or phrases, exploratory search systems facilitate
is associated with the user context / tasks and the exploration pro- recognition-based searches based on the user’s familiarity with
cess. Our observation suggests that expertise in the field is linked symbols within the information domain. Recognition enables users
to exploring fewer novel topics and seeking fewer explanations but to explore and discover information that they may not have been
engaging more with explanations of relationships. In addition, we aware of otherwise.
found that the engagement with faculty information is moderately To further facilitate information discovery, information explo-
correlated with the use of more advanced explanations. ration systems could include the ability to adapt to users’ interests.
These systems known as personalized information exploration sys-
tems include a user model, which represents the user’s interests
CCS CONCEPTS or goals within the information domain and steers the exploration
• Information systems → Personalization; Recommender process toward more relevant information [6, 11]. Due to the nature
systems; • Computing methodologies → Semantic networks. of information exploration systems, which typically offer advanced
control and transparency options [25] many information explo-
ration systems choose to make their user models open for viewing
and manipulation [3, 8, 31, 33].
An open user model in information access systems makes the
user model visible to the user being modeled and supports the
user in adjusting the model in various ways. The user can decide
This work is licensed under a Creative Commons Attribution International
4.0 License. to remove or add components representing their interests (words,
named entities, concepts) to the model [3, 5, 31], adjust the impor-
UMAP Adjunct ’24, July 01–04, 2024, Cagliari, Italy tance of a component in the model [31, 33], or change its position
© 2024 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0466-6/24/07 among other model components [3, 33]. Several studies of per-
[Link] sonalized information exploration systems demonstrated that an

256
UMAP Adjunct ’24, July 01–04, 2024, Cagliari, Italy Rully Agus Hendrawan, Peter Brusilovsky, Arun Balajiee Lekshmi Narayanan, and Jordan Barria-Pineda

open user model could make information exploration process more [4]. However, managing the number of items in a user profile and
efficient [3, 31, 33], however, it has also been shown that the abil- maintaining simplicity in these features is crucial, as exceeding a
ity to manipulate user models requires a good understanding of certain threshold of complexity can diminish their benefits [19, 36].
model components [4]. If the meaning of the model components Additionally, interpreting the relationships of entities within graphs
is not clear to users, their attempts to adjust the user model might remains challenging, as the complexity often obscures the rationale
negatively affect system performance [5]. behind specific recommendations.
A good example of the domain where the meaning of user model Recent advances in Large Language Models (LLMs) have signifi-
components might be hard to understand for system users is aca- cantly enhanced their capabilities in text adaptations, especially in
demic research, which has been explored in a range of search and generating definitions and explanations. Instruction-tuned LLMs
recommender systems for academic papers [10] or researchers [31]. are particularly adept in this regard [39]. These LLMs can be di-
Information exploration systems in the academic domain typically rected to provide definitions through in-context learning or prompt-
used research topics such as "reinforcement learning" or "relevance ing, and techniques like Few-Shot and Chain-of-Thought leading
feedback" as elements of the user model to reflect the user’s interest to substantial improvements in generating specific explanations
in these topics [14, 31, 33]. The exact meaning of these terms might [12, 40]. The variety of approaches for prompt engineering has led
not always be clear to system users, especially students who are researchers to categorize emerging patterns [41]. Incorporating
target users of many systems in this domain [31, 33]. Moreover, external sources to enhance LLM generation has been identified
less experienced users might not always understand the differences as essential to address certain limitations, leading to the develop-
between similar topics such as “user model” and “user profile”. The ment of Augmented LMs [27]. Retrieval Augmented Generation
lack of understanding might lead to mistakes in all kinds of model (RAG) has been particularly noted for its efficacy in knowledge-
adjustments - adding, removing, and re-weighting the topics in the intensive tasks [21]. Using the Web as a readily available source
model. of context represents a pragmatic approach [28]. In contrast, for
This paper investigates whether information exploration with domain-specific explanations, retrieving information from inter-
open user models in a complex domain could be facilitated by nal libraries is necessary [34]. Augmentation also plays a critical
topic explanations powered by modern Large Language Models. role in generating explanations between two research documents
The starting point for our research was the Grapevine exploratory [20]. Embedding LLMs with knowledge graphs, a reliable source
search system for finding research advisors [31], which used re- of structured context, further enhances their utility [17, 30]. This
search topics as elements of its open user model. While originally process can be further automated, allowing LLMs to suggest which
developed for Ph.D. students looking for advisors, the system has context to use [35].
been increasingly used by undergraduate students interested in LLM explanations may not always resemble dictionary defi-
research and even high school students selecting a college to ap- nitions, but are often easier to understand [20]. Recent research
ply to. These users have a relatively weak understanding of the demonstrates that LLM explanations are useful for learning new
research topics used in the system. To help less prepared users, we concepts in fields such as computer science [2], and medical physi-
developed two types of topic explanation: "individual keyphrase ology [1]. LLMs could alleviate the tedious task required to build
explanation", which explains the nature of a specific topic to the the concept model [26]. LLMs can provide contextual explanations,
user, and "relationship explanation between keyphrases", which making the underlying logic of the recommender systems compre-
highlights the relationship between a topic that the user consid- hensible to users. In our work presented in this paper, our objective
ers to add to the model and topics already present in the model. is to explore how explanations can facilitate an interactive explo-
The explanation will be presented on-demand when the user clicks ration experience and help users connect their existing knowledge
the explanation button to deliver an explanation based on three with the information they seek [31].
levels of details: keyphrase without explanation (no-exp), individ-
ual keyphrase explanation (ind-exp), and relationship explanation
(rel-exp) [13]. Using the Grapevine system extended with these ex- 3 GRAPEVINE: EXPLORATION AND
planations, we performed a user study to understand to what extent EXPLANATIONS WITH AN OPEN USER
the explanation affects or is affected by the exploration process. MODEL
This paper is structured as follows. First, we outline the back-
ground of this research and discuss related work. We then present
3.1 Grapevine: An Exploratory Recommender
the explanation of the implementation in Grapevine. Afterward, we System
describe the user study, present the results, and discuss the findings. The starting point of our research on explainable user models was
Finally, we summarize the work and its limitations. Grapevine [31] - an exploratory recommender system with an open
user model. Grapevine was designed to help students seek fac-
ulty advisors for a variety of academic projects, such as capstone
2 BACKGROUND AND RELATED WORK projects, independent studies, master’s theses, and Ph.D. theses.
Previous research has shown that the inclusion of exploration- The original version of Grapevine was built through a multistage
focused features, open profile building, and personalization can iterative design process that started with interviews and obser-
have a positive impact on user exploration behavior and goals [31, vation to understand how students generally search and locate
44]. Concept graphs, which represent a network of keyphrases and advisors and was followed by the design and evaluation of multi-
their interrelationships, provide a natural method to navigate items ple versions of the system [32]. Our studies demonstrated that the

257
 UMAP Adjunct ’24, July 01–04, 2024, Cagliari, Italy

Figure 2: Explanation Access Points

Figure 1: Building Open User Model to Control Recommen-

dation

search for advisors typically starts with pre-existing knowledge

and skills acquired through previous formal or informal learning
experiences. However, while students explore prospective advisor
candidates, they frequently discover new interesting topics that
shift their original interests and change the direction of the original
search. To support this process, Grapevine was equipped with an Figure 3: Semantic Support: Post-hoc Explanation
open model (profile) or student interests, which the students were
able to gradually build and modify while searching for advisors.
To build their profile of interests, users gradually add topic
keyphrases to their Open User model (Fig. 1.B), which is always receive basic concept explanations and relationship explanations
visible to the user. New topic keyphrases could be added from through a dedicated explanation page.
search, recommendations, or advisor exploration. The search box
allows users to look up specific topics supporting recall-based ex- 3.2.1 Explanation Access Points. During the iterative process of
ploration (Fig.1.A.1), which is useful for students who have a clear building a profile, users may encounter situations requiring an ex-
idea of what they’re looking for. Topic recommendation suggests planation of a certain research topic (expressed by a keyphrase). We
keyphrases that are relevant to users’ profiles, as recognition-based have provided three access points for users to seek explanations of
exploration (1.A.2). This may be beneficial in broadening the user’s these topics. First, users can ask for explanations of topics already
knowledge base and exposing them to new areas that complement added to their profile (Fig. 2.A). Second, users can request explana-
their interests. Finally, users may also discover an interesting topic tions of recommended topics and seek additional explanations as
when exploring faculty interests. In this case, pressing the “plus” to why the system thinks these keyphrases are related to their in-
button next to a specific faculty interest will add it to the open user terests (Fig. 2.B). Lastly, users may also encounter promising topics
model (Fig. 2.C). of interest provided by faculty members to describe their research
After each change in the user profile of interests displayed in and are listed on the faculty detail page (Fig. 2.C).
the open user model, the system generates new advisor recommen-
dations for the users, which are displayed by category through a 3.2.2 Explanation Page. The system provides two types of explana-
carousel-based interface (Fig. 1.C). Users can adjust their interest tion to help users understand the research areas in their profile (Fig.
at any time by adding or removing topics from their profile and 3.A). The first type is the "Individual Explanation" (ind-exp), which
receive updated recommendations instantly. This design supports provides a “standalone” explanation of a considered topic (Fig. 3.B.1)
an iterative exploration process that we observed in our earlier or one of the current topics of interest in the open user models
studies. (Fig. 3.B.2), to which the user wants to compare it. The second
type is the "Relationship Explanation" (rel-exp), which explores the
connections between the considered topic and one of the current
3.2 Grapevine 2: Explanations for Open User topics of interest (Fig. 3.C). If the user needs additional information
Modeling or wants to review the explanation, the system provides access to
The Grapevine 2 system explored in this paper added the ability to external search engines. It allows us to supplement the internal
explain topics used in the open user modeling process. It provided explanations with external content, gaining a broader context and
three access points for explanations and offered the opportunity to alternative perspectives.

258
UMAP Adjunct ’24, July 01–04, 2024, Cagliari, Italy Rully Agus Hendrawan, Peter Brusilovsky, Arun Balajiee Lekshmi Narayanan, and Jordan Barria-Pineda

3.3 Generating Explanation with Knowledge 4 STUDY

Graph and LLMs We conducted the study in two phases: pilot studies and an obser-
The information exploration system was based on a knowledge vational study.
graph. This approach is suitable for information exploration [23].
We develop a knowledge graph using information from multiple 4.1 The Pilot Study
web resources. First, we gathered faculty profiles from the school’s The two pilot tests, each involving five graduate students, exam-
official website, which provided us with base information for the ined task-related issues associated with system design. We found
"Faculty" nodes, including their areas of expertise and research in- that layouts with three columns were more effective in learning
terests. Then, we populated the "Topic" nodes and their relationship than those with only two columns. Participants preferred shorter
with faculty; we collected the areas of interest that each faculty explanations over longer ones. Furthermore, we also observed that
selected to describe their research interests on the school page, their the LLM-generated explanations frequently contained redundant
own homepage, and their Google Scholar profile. Finally, "Faculty" information, such as defining individual concepts before explor-
nodes are linked to "Topic" nodes through self-labeling, and "Topic" ing their relationships. Interestingly, in some instances, GPT-3.5
nodes are linked to each other via textual semantic similarity scores. handled explanations for ambiguous terms more effectively than
We evaluated how similarity scores between "Topic" nodes in- traditional search engines, possibly due to a lack of contextual un-
fluenced recommendation quality guided by two principles. Firstly, derstanding of search algorithms. In our observational study, the
user satisfaction would not necessarily decrease as long as relevance participants used the system based on the role-play task. The task
was perceived as high [24]. Secondly, random non-personalized emulates a real-world case in which students need to explore topics,
recommendations can sometimes outperform targeted ones, which select three faculty members, and write an introductory message
might inadvertently reinforce user biases, leading to poor deci- to connect with one of them.
sion making [16]. We tried and tested the balance between high-
performing, versatile models and those that require fewer compu- 4.2 The Observational Study: Participants
tational resources [18, 38]. We started with smaller models and Over three weeks, we advertised our study to the School of Comput-
gradually moved to more complex ones like all-MiniLM-L6-v2, all- ing and Information at the University of Pittsburgh and received 42
mpnet-base-v2, bge-large-en-v1.5, UAE-Large-V1, and davinci-002, responses from students. Within the following month, we assigned
achieving satisfactory results by the third iteration [22, 29, 42]. 23 participants to the study with $20 compensation for 1 hour of
To provide explanations for individual entities, we used GPT- participation.
3.5 to generate explanations [29], since many keyphrases lacked
matching abstracts in DBpedia [7]. LLMs offer a promising avenue
4.3 The Observational Study: Procedure and
to bridge the gap between human and machine communication
by interpreting complex relationships [9]. We developed effective Data Collection
prompts for LLM using an iterative process that included user stud- Participants were introduced to the study objectives and received
ies and prompt experimentation [43]. Participants preferred short a consent form that describes the study procedures and data han-
explanations, so we varied our prompts and evaluated the results dling practices. Upon consenting, participants provided basic de-
based on subjective criteria. Although long-form prompts may im- mographic information and watched a video tutorial to familiarize
prove LLM’s results [40], we found that using simple and short themselves with the system’s interface and functionalities. To finish
prompts generated more consistent results for this case. Providing the main task, the participants used the system to select 7 topics
more context to generate definitions often resulted in longer texts they were interested in and find 3 relevant faculty advisors. The
that did not enhance understanding. As a result, we use the fol- study platform logs user interaction (Table 1). After completion
lowing prompt to define a single keyword: "Please explain concisely of the task, the participants moved to the final phase of the study:
about ‘keyword‘." evaluating the novelty and importance of the selected topics, writ-
For relationship explanation, we are experimenting with sev- ing an introductory email draft, and providing feedback with an
eral prompt variances by including and excluding these phrases: exit questionnaire (Appendix A).
"concisely," "simple language," and "academic language." One of the
issues we have encountered is that GPT-3.5 tends to repeat the 5 RESULTS
definitions of keyphrases unnecessarily, presumably as part of the 5.1 System Use
initial prefix probability. We found that the phrase "skip definition"
The analysis of the log data showed that the participants exten-
is very effective in suppressing this behavior while still producing
sively used the exploration and explanation functionality of the
excellent explanations of the relationship between the keyphrases.
system (Table 1). In particular, 15 out of 23 participants requested
After conducting an informal user study, we have decided to use
the concept explanation and visited the explanation page (ind-
the following prompt: "Please evaluate how ‘keyword1‘ and ‘key-
exp), and 8 out of those 15 further requested relationship explana-
word2‘ can be connected. Skip the definition. Explain the possible
tions (rel-exp). All participants actively added topics to the profile
relationship."
(M=10.57±2.74), but only 18 removed the topics from the profile at
least once (p̂=18/23=0.78; M=5.61±4.49).
The “faculty details” were frequently accessed by the partici-
pant to align their interests (M=17.4). The high variance range

259
 UMAP Adjunct ’24, July 01–04, 2024, Cagliari, Italy

Table 1: Behavioral log data from participants in the obser-

vational study (n=23)

Event Type n Mean SD Total

Profile Building
Add Topic to Profile 23 10.57 2.74 243
Remove Topic from Profile 18 5.61 4.49 101
Exploration
Faculty Details 15 17.4 22.84 261
Explanation Page (ind-exp) 15 5 6.99 75
Relationship Explanation (rel-exp) 8 2.38 2.13 19
External Search 3 2 1 6
Task
Select Topic 23 7.35 0.88 169
Remove Selected Topic 4 1.5 1 6
Select Advisor 23 4.04 1.92 93
Remove Selected Advisor 7 2.86 2.54 20

Figure 4: Probability of Likert responses for participants

(SD=22.84) suggests that some users frequently refer to faculty de- who use no explanation (no-exp, n=8), use only individual
tails to align their academic pursuits with faculty expertise, while keyphrase explanation (ind-exp, n=7), or relationship ex-
others may rely less on these details or find them less useful or infor- planation between keyphrases (rel-exp, n=8). Fedback from
mative. The individual explanation page "ind-exp" (p̂=15/23=0.65; users who were able to see the evaluated feature only in the
M=5±6.99) and "rel-exp" relationship explanations between topics video tutorial are shown in gray.
(p̂=8/23=0.35; M=2.38±2.13) were consulted at varying rates to sup-
port profile building. The least engaged functionality was external
searches (p̂=3/23=0.13; M=2±1), indicating that most information
needs were met within the system.
their understanding or helped in tasks. We can hypothesize that
these were the most advanced users who had high expectations
5.2 System Assessment
about the system.
The system is evaluated through three aspects: the relevance of
recommendations, the association of explanations with user under-
standing, and how it supports the task. Assessment is measured 5.3 Explanation and User Context
using a 5-scale Likert score (Fig. 4). These aspects reflect the ef- 5.3.1 Who needs support? In contrasting the behaviors of under-
fectiveness of explanation in the context of open user modeling. graduate and graduate students in using an explanation page (ind-
The open user model exposes keyphrases (topics) to users, allowing exp), we found that undergraduate students (n=8; M=5.38±9.71)
them to control the recommended items (faculty members). The accessed this page more frequently than graduate students (n=15;
first aspect evaluates how well this recommendation is perceived M=2.13±2.67), see Fig. 5. However, despite opening the page more
as relevant for users. The explanation helps users understand more frequently, undergraduates used relationship explanations (rel-exp)
about those exposed keyphrases. The second and third aspect ad- only 39% of the time, whereas graduate students used them more
dress the extent to which the explanation helps users understand often. A PhD participant provided a comment, "Because I am fa-
the keyphrases used to control the recommendation and perform miliar with my and the faculty’s research field, I don’t need to look
the task. up explanations or relations a lot.", suggesting that familiarity with
The general feedback from the participants was generally posi- the research area may be associated with a reduced need for the
tive. However, the perception of recommendation relevance varied general explanation of individual keyphrases (ind-exp). However,
significantly among different groups of users, categorized by their when these explanations are needed, graduate students use rela-
engagement with explanation. Users who engaged with only in- tionship explanations (rel-exp) more than undergraduates. This can
dividual explanations were considerably more positive about the be attributed to their focus on establishing connections between
system and its features than users who did not use explanations. existing ideas. We also observe similar trends when comparing
At the same time, users who engaged with relative explanations students by their program. Computer Science students used ex-
were more critical. For example, those who used relationship expla- planations more than Information Science students and students
nations demonstrated a more critical view of the system’s effective- from other programs (𝜒 2 (other)=2.24; p=0.33), see Figure 6). The
ness; 25% of these participants disagree that the faculty member greater inclination toward understanding the relationships between
recommendations are relevant to their profile. Similarly, for topic various topics associated with those already well-versed in the sub-
recommendations, 37. 5% of the participants in this group disagreed ject matter may indicate the need for a deeper or more targeted
on their relevance and 12. 5% disagreed that explanations enhanced understanding of the keyword.

260
UMAP Adjunct ’24, July 01–04, 2024, Cagliari, Italy Rully Agus Hendrawan, Peter Brusilovsky, Arun Balajiee Lekshmi Narayanan, and Jordan Barria-Pineda

Figure 5: Total explanation usage based on participants’ pro-

Figure 8: Frequency of Faculty Page visits by participants in
gram level: Undergraduate students (n=15) and Graduate
group: no-exp, never accessed explanations; ind-exp, used
students (n=8)
only individual keyphrase explanations; rel-exp, accessed
relationship explanations between keyphrases.

5.4 Explanation and Item Engagement

To understand how explanation is associated with the exploration
process, we investigated the correlation between the level of detail
in explanations and user engagement with faculty members’ de-
tail pages (Fig. 2.C). The engagement is quantified by how many
times each participant views faculty details, and the participants
Figure 6: Explanation usage based on participants’ program: are segmented into three groups based on the depth of explanation
Computer Science (n=8), Information Science (n=10), and they use. The groups are keyphrases without explanation (no-exp),
Others (n=5). explanation of individual keyphrases (ind-exp), and explanation of
relationships between keyphrases (rel-exp).
Users who only saw topics without further explanation (no-exp)
5.3.2 Who explores novel topics? At the end of the study, par- viewed faculty detail pages 2.25 times on average (SD = 3.49), indicat-
ticipants must select seven keyphrases that represent their re- ing relatively low engagement compared to other groups. Users who
search interests and then rate the importance and novelty of these opened explanations of individual keyphrases (ind-exp) viewed fac-
keyphrases. The novelty score was derived by reversing the 5-point ulty detail pages more frequently, 9.57 times on average(SD=6.80),
Likert scale of keyphrase familiarity. We conducted the Kruskal- suggesting a higher level of engagement. Participants who opened
Wallis test to examine differences between undergraduate and grad- explanations detailing the relationships between keyphrases (rel-
uate students in mean importance and novelty of keyphrases. There exp) showed the highest engagement, with a mean of 22 views
are no significant differences in the importance of the selected (SD=31.46). This finding suggested that more detailed explanations
keyphrases (𝜒 2 (𝑔𝑟𝑎𝑑𝑢𝑎𝑡𝑒)=0.417; p=0.52). However, undergraduate of keyphrases are associated with higher engagement with item
students are associated with a high exploration of novel keyphrases details, with a statistically significant moderate positive correlation
compared to graduate students, with a statistically significant dif- (𝜌=0.47, p=0.02). Figure 8) illustrates this pattern.
ference (𝜒 2 (graduate)=7.05; p=0.01), as illustrated in Figure 7. Con-
tinuing the previous discussion, less prepared participants who 6 CONCLUSION
explore novel topics will likely need individual keyphrase explana- We investigated the effect of providing on-demand explanations of
tions (ind-exp). On the other hand, participants who are familiar domain topics to assist users in managing their open user models
with their selected topics might receive better value from deeper and controlling recommendations. Since reference definitions and
relationship explanations (rel-exp). explanations are often unavailable, LLM-generated explanations
were provided. We obtained several results that support the need
for individual explanations. Those who used these explanations
were considerably more positive about the value of the system
and its features and were more engaged with the system. We also
found that these explanations were considerably more used by
undergraduate students who are much less familiar with research
topics.
The value of relative explanations needs further investigation.
This feature was used only by a third of users (n = 8), so the amount
Figure 7: The importance and novelty of selected keyphrases of data related to it is limited. So far, the most interesting fact is that
by participants’ program level: undergraduate (n=8) and grad- the use of relative explanations was associated with a much higher
uate (n=15). engagement with the system and exploring a much larger number
of candidate advisors. Those who used relative explanations were
also less satisfied with system recommendation and have the lowest

261
 UMAP Adjunct ’24, July 01–04, 2024, Cagliari, Italy

confidence in their final advisor selections. However, it is not clear [11] Max Braun, Klaas Dellschaft, Thomas Franz, Dominik Hering, Peter Jungen,
whether these users have more advanced or unusual needs, which Hagen Metzler, Eugen Müller, Alexander Rostilov, and Carsten Saathoff. 2008.
Personalized Search and Exploration with MyTag. In the 17th international con-
caused them to work harder and engage with relative explanations ference on World Wide Web, WWW ’08. ACM, 1031–1032.
but also left them less satisfied, or whether they were simply more [12] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
active and investigative users who expected more from the system. Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan,
We also analyzed how the use of explanations is related to the Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,
user’s context and task. Our observations suggest that high experi- Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin
Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya
ence in a particular field, such as accumulated by graduate students, Sutskever, and Dario Amodei. 2020. Language Models Are Few-Shot Learners.
is linked to exploring fewer basic explanations but engaging more [Link] arXiv:2005.14165 [cs]
with relationship explanations. In addition, we found no signifi- [13] Mohamed Amine Chatti, Mouadh Guesmi, Laura Vorgerd, Thao Ngo, Shoeb
Joarder, Qurat Ul Ain, and Arham Muslim. 2022. Is More Always Better? The
cant association between explanation usage with the importance Effects of Personal Characteristics and Level of Detail on the Perception of Expla-
of selected topics and confidence in the choice of faculty. nations in a Recommender System. In Proceedings of the 30th ACM Conference on
User Modeling, Adaptation and Personalization. ACM, Barcelona Spain, 254–264.
[Link]
7 LIMITATION AND FUTURE WORK [14] Dario De Nart, Felice Ferrara, and Carlo Tasso. 2013. Personalized Access to
Scientific Publications: from Recommendation to Explanation. In 21st Conference
The purpose of this study is to investigate how the level of detail on User Modeling, Adaptation and Personalization (UMAP 2013) (Lecture Notes in
in explanations, building user profiles, and controlling recommen- Computer Science), Sandra Carberry, Stephan Weibelzahl, Alessandro Micarelli,
dations are related. However, due to our limited sample size, we and Giovanni Semeraro (Eds.). 296–301. [Link]
1007%2F978-3-642-38844-6_2
strongly suggest replicating or extending our model before applying [15] Cassius Dhelon, Jae-wook Ahn, V Kasireddy, and Nirmal Mukhi. 2019. Interactive
it to real-world problems. While we have not discovered a corre- Learning in a Conversational Intelligent Tutoring System Using Student Feedback,
lation between explanations and a user’s confidence in selecting Concept Grouping and Text Linking. In Proceedings of the 13th International
Technology, Education and Development Conference. [Link]
advisors, it may be worth investigating more on user tasks, such as inted.2019.0756
drafting an introductory email. This study also lays the foundation [16] Ayoub El Majjodi, Alain D. Starke, and Christoph Trattner. 2022. Nudging
Towards Health? Examining the Merits of Nutrition Labels and Personalization
for creating flexible information exploration systems that use mul- in a Recipe Recommender System. In Proceedings of the 30th ACM Conference
tiple AI agents: search, exploration, and explanation. It provides on User Modeling, Adaptation and Personalization. ACM, Barcelona Spain, 48–56.
a test ground for exploring the mechanics of AI systems that can [Link]
[17] Chao Feng, Xinyu Zhang, and Zichu Fei. 2023. Knowledge Solver: Teaching LLMs
help users understand complex symbolic representations within to Search for Domain Knowledge from Knowledge Graphs. arXiv:2309.03118 [cs]
the system. [18] Yingqiang Ge, Wenyue Hua, Kai Mei, jianchao ji, Juntao Tan, Shuyuan Xu, Zelong
Li, and Yongfeng Zhang. 2023. OpenAGI: When LLM Meets Domain Experts. In
Advances in Neural Information Processing Systems, A. Oh, T. Neumann, A. Glober-
REFERENCES son, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc.,
[1] Mayank Agarwal, Ayan Goswami, and Priyanka Sharma. 2023. Evaluating 5539–5568.
ChatGPT-3.5 and Claude-2 in Answering and Explaining Conceptual Medi- [19] Julio Guerra-Hollstein, Jordan Barria-Pineda, Christian D. Schunn, Susan Bull,
cal Physiology Multiple-Choice Questions. Cureus 15, 9 (Sept. 2023), e46222. and Peter Brusilovsky. 2017. Fine-Grained Open Learner Models: Complexity
[Link] Versus Support. In Proceedings of the 25th Conference on User Modeling, Adaptation
[2] Vibhor Agarwal, Nakul Thureja, Madhav Krishan Garg, Sahiti Dharmavaram, and Personalization (UMAP ’17). Association for Computing Machinery, New
Meghna, and Dhruv Kumar. 2024. "Which LLM Should I Use?": Evaluating LLMs York, NY, USA, 41–49. [Link]
for Tasks Performed by Undergraduate Computer Science Students in India. [20] Jenny Kunz and Marco Kuhlmann. 2024. Properties and Challenges of LLM-
arXiv:2402.01687 [cs] Generated Explanations. arXiv:2402.10532 [cs]
[3] Jaewook Ahn and Peter Brusilovsky. 2013. Adaptive visualization for exploratory [21] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin,
information retrieval. Information Processing and Management 49, 5 (2013), Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel,
1139–1164. [Link] Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for
[4] Jae Ahn, Peter Brusilovsky, and Shuguang Han. 2015. Personalized Search: Recon- Knowledge-Intensive NLP Tasks. [Link]
sidering the Value of Open User Models. In Proceedings of the 20th International arXiv:2005.11401 [cs]
Conference on Intelligent User Interfaces. ACM, 202–212. [Link] [22] Xianming Li and Jing Li. 2023. AnglE-optimized Text Embeddings.
article-id:13566841doi:10.1145/2678025.2701410 arXiv:2309.12871 [cs]
[5] Jae-wook Ahn, Peter Brusilovsky, Jonathan Grady, Daqing He, and Sue Yeon [23] Matteo Lissandrini and Davide Mottin. 2022. Knowledge Graph Exploration
Syn. 2007. Open user profiles for adaptive news systems: help or harm?. In Systems: Are We Lost? In Proceedings of the 12th Conference on Innovative Data
the 16th international conference on World Wide Web, WWW ’07. ACM, 11–20. Systems Research (2022).
[Link] [24] Benedikt Loepp. 2022. Recommender Systems Alone Are Not Everything: To-
[6] Jae-wook Ahn, Peter Brusilovsky, Daqing He, Jonathan Grady, and Qi Li. 2008. wards a Broader Perspective in the Evaluation of Recommender Systems. In
Personalized Web Exploration with Task Models. In the 17th international confer- Proceedings of Perspectives on the Evaluation of Recommender Systems Work-
ence on World Wide Web, WWW ’08. ACM, 1–10. shop (PERSPECTIVES 2022) at 16th ACM Conference on Recommender Systems.
[7] Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, [Link]
and Zachary Ives. 2007. DBpedia: A Nucleus for a Web of Open Data. In Pro- [25] Gary Marchionini. 2006. Exploratory search: From finding to understanding.
ceedings of the 6th International The Semantic Web and 2nd Asian Conference Commun. ACM 49, 4 (2006), 41–46.
on Asian Semantic Web Conference (ISWC’07/ASWC’07). Springer-Verlag, Berlin, [26] Lars-Peter Meyer, Claus Stadler, Johannes Frey, Norman Radtke, Kurt Jung-
Heidelberg, 722–735. hanns, Roy Meissner, Gordian Dziwis, Kirill Bulert, and Michael Martin. 2023.
[8] Fedor Bakalov, Birgitta König-Ries, Andreas Nauerz, and Martin Welsch. 2010. LLM-assisted Knowledge Graph Engineering: Experiments with ChatGPT.
IntrospectiveViews: An Interface for Scrutinizing Semantic User Models. In 18th arXiv:2307.06917 [cs]
International Conference on User Modeling, Adaptation, and Personalization (UMAP [27] Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram
2010) (Lecture Notes in Computer Science, Vol. 6075), Paul De Bra, Alfred Kobsa, Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli
and David Chin (Eds.). Springer, 219–230. [Link] Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented
1007%2F978-3-642-13470-8_21 Language Models: A Survey. arXiv:2302.07842 [cs]
[9] Randall Balestriero, Jerome Pesenti, and Yann LeCun. 2021. Learning in High [28] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina
Dimension Always Amounts to Extrapolation. arXiv:2110.09485 [cs] Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu
[10] Joeran Beel, Bela Gipp, Stefan Langer, and Corinna Breitinger. 2016. Paper Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew
Recommender Systems: A Literature Survey. International Journal on Digital Knight, Benjamin Chess, and John Schulman. 2022. WebGPT: Browser-assisted
Libraries 17, 4 (2016), 305–338. [Link] Question-Answering with Human Feedback. arXiv:2112.09332 [cs]

262
 UMAP Adjunct ’24, July 01–04, 2024, Cagliari, Italy Rully Agus Hendrawan, Peter Brusilovsky, Arun Balajiee Lekshmi Narayanan, and Jordan Barria-Pineda

[29] OpenAI. 2022. OpenAI Platform - GPT Base. (2) For each topic you choose, rate how important they are to
[Link] you. How important are these topics to your research or
[30] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu.
2024. Unifying Large Language Models and Knowledge Graphs: A Roadmap. studies?
IEEE Transactions on Knowledge and Data Engineering (2024), 1–20. [Link] (_) Not Important: It has no relevance to my work.
org/10.1109/TKDE.2024.3352100
[31] Behnam Rahdari, Peter Brusilovsky, and Dmitriy Babichenko. 2020. Personalizing
(_) Slightly Important: It has minimal relevance.
Information Exploration with an Open User Model. In Proceedings of the 31st ACM (_) Moderately/ Somewhat Important: It is somewhat rele-
Conference on Hypertext and Social Media. ACM, Virtual Event USA, 167–176. vant.
[Link]
[32] Behnam Rahdari, Peter Brusilovsky, Dmitriy Babichenko, Eliza Beth Littleton, (_) Important: It is a key aspect of my work.
Ravi Patel, Jaime Fawsett, and Zara Blum. 2020. Grapevine: A profile-based (_) Very Important: It is central to my research/studies
exploratory search and recommendation system for finding research advisors. In (3) Compose a Message to one of the Faculty Members: Write
83rd Annual Meeting of the Association for Information Science and Technology.
[Link] an introductory message, pitch a project idea, and describe
[33] Tuukka Ruotsalo, Kumaripaba Athukorala, Dorota Glowacka, Ksenia why they’re a good match for your project and how they fit
Konyushkova, Antti Oulasvirta, Samuli Kaipiainen, Samuel Kaski, and Giulio
Jacucci. 2013. Supporting exploratory search tasks with interactive user
into your project plans. Be as detailed as possible – this is
modeling. In 2013 Annual Meeting of American Society for Information Science your chance to reflect on your decisions.
and Technology, Vol. 50. Wiley, 1–10. Below is the template you can use later to write your mes-
[34] Jaromir Savelka, Kevin D. Ashley, Morgan A. Gray, Hannes Westermann, and
Huihui Xu. 2023. Explaining Legal Concepts with Augmented Large Language sage.
Models (GPT-4). arXiv:2306.09525 [cs] (4) Choose one faculty member from the list below to be the
[35] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, recipient of your message.
Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer:
Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761 [cs] (_) Person 1 (_) Person 2 (_) Person 3
[36] Sergey Sosnovsky and Peter Brusilovsky. 2015. Evaluation of Topic-Based Adap- (5) Your Message to the Faculty Member:
tation and Student Modeling in QuizGuide. User Modeling and User-Adapted
Interaction 25, 4 (Oct. 2015), 371–424. [Link]
___________________
[37] Pertti Vakkari. 2016. Searching as Learning: A Systematization Based on Litera- (6) Please give feedback to the system (Strongly Disagree / Dis-
ture. Journal of Information Science 42, 1 (Feb. 2016), 7–18. [Link] agree / Neutral / Agree / Strongly Agree):
1177/0165551515615833
[38] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. (a) The system helps me feel confident in my choice of poten-
2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compres- tial advisors.
sion of Pre-Trained Transformers. [Link] (b) The system does NOT influence my confidence in my
arXiv:2002.10957 [cs]
[39] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian choice of potential advisors.
Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned Language (c) The faculty member recommendations provided are highly
Models Are Zero-Shot Learners. arXiv:2109.01652 [cs]
[40] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia,
relevant to my academic/research interests.
Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits (d) The faculty member recommendations DO NOT seem
Reasoning in Large Language Models. arXiv:2201.11903 [cs] related to my academic/research interests.
[41] Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry
Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. (e) The topic recommendations align well with my current
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. academic interests.
arXiv:2302.11382 [cs] (f) The recommended topics DO NOT seem related to my
[42] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2024.
C-Pack: Packaged Resources To Advance General Chinese Embedding. academic interests.
arXiv:2309.07597 [cs] (g) The explanations for each topic enhance/aid my under-
[43] Ziqi Yin, Hao Wang, Kaito Horio, Daisuke Kawahara, and Satoshi Sekine. 2024.
Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt
standing.
Politeness on LLM Performance. [Link] (h) Overall, I find the explanations for each topic confusing
arXiv:2402.14531 [cs] and UNHELPFUL.
[44] Run Yu, Zach Pardos, Hung Chau, and Peter Brusilovsky. 2021. Orienting Stu-
dents to Course Recommendations Using Three Types of Explanation. In Adjunct (i) The explanations of relationship between topics enhance/aid
Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Person- my understanding.
alization. ACM, Utrecht Netherlands, 238–245. [Link] (j) Overall, I find the explanations of relationship between
3464483
topics do NOT improve my understanding.
(k) The explanations of relationship between topics are help-
ful in guiding my topic selection or research direction.
(l) Overall, I find the explanations of relationship between
A APPENDIX: POST-STUDY QUESTIONNAIRE topics UNHELPFUL in guiding my topic selection or re-
(1) For each topic you chose, please rate its novelty to you. search direction.
Before using the system, how familiar were you with these (7) Suggestions or Comments (Optional):
topics? ___________________
(_) Not Familiar: I was not aware of it before. ___________________
(_) Slightly Familiar: I had minimal awareness or understand-
ing.
(_) Moderately/ Somewhat Familiar: I had heard about it pre-
viously.
(_) Familiar: I had a good understanding but not in-depth.
(_) Very Familiar: I have extensive knowledge or experience.

263

View publication stats