Evaluating Information Presentation Strategies for Spoken Dialogue Systems
Date
2009Author
Winterboer, Andi
Metadata
Abstract
A common task for spoken dialogue systems (SDS) is to help users select a suitable
option (e.g., flight, hotel, restaurant) from the set of options available. When the number
of options is small, they can simply be presented sequentially. However, as the
number of options increases, the system must have strategies for helping users browse
the space of available options.
In this thesis, I compare two approaches to information presentation in SDS: (1)
the summarize and refine (SR) approach (Polifroni et al., 2003; Polifroni, 2008) in
which the summaries are generated by clustering the options based on attributes that
lead to the smallest number of clusters, and (2) the user-model based summarize and
refine (UMSR) approach (Demberg, 2005; Demberg and Moore, 2006) which employs
a user model to cluster options based on attributes that are relevant to the user and
uses coherence markers (e.g., connectives, discourse cues, adverbials) to highlight the
trade-offs among the presented items.
Prior work has shown that users prefer approaches to information presentation that
take the user’s preferences into account (e.g., Komatani et al., 2003;Walker et al., 2004;
Demberg and Moore, 2006). However, due to the complexity of building a working
end-to-end SDS, these studies employed an ”overhearer” evaluation methodology, in
which participants read or listened to pre-prepared dialogues, thus limiting evaluation
criteria to users’ perceptions (e.g., informativeness, overview of options, and so on).
In order to examine whether users prefer presentations based on UMSR when they
were actively interacting with a dialogue system, and to measure the effectiveness and
efficiency of the two approaches, I compared them in a Wizard-of-Oz experiment. I
found that in terms of both task success and dialogue efficiency the UMSR approach
was superior to the SR approach. In addition, I found that users also preferred presentations
based on UMSR in the interactive mode.
SDS are typically developed for situations in which the user’s hands and eyes are
busy. I hypothesized that the benefits of pointing out relationships among options (i.e.,
trade-offs) in information presentation messages outweighs the costs of processing
more complex sentences. To test this hypothesis, I performed two dual task experiments
comparing the two approaches to information presentation in terms of their
effect on cognitive load. Again, participants performed better with presentations based
on the UMSR algorithm in terms of both dialogue efficiency and task success, and I
found no detrimental effect on performance of the primary task.
Finally, I hypothesized that one of the main reasons why UMSR is more efficient
is because it uses coherence markers to highlight relations (e.g., trade-offs) between
options and attributes. To test this hypothesis, I performed an eye-tracking experiment
in which participants read presentations with and without these linguistic devices, and
answered evaluation and comparison questions to measure differences in item recall.
In addition, I used reading times to examine comprehension differences between the
two information presentation strategies. I found that the linguistic devices used in
UMSR indeed facilitated item recall, with no penalty in terms of comprehension cost.
Thus, in this thesis I showed that an approach to information presentation that employs
a user model and uses linguistic devices such as coherence markers to highlight
trade-offs among the presented items improves information browsing. User studies
demonstrated that this finding also applies to situations where users are performing
another demanding task simultaneously.