research-article

Free access

Modeling user satisfaction with Hidden Markov Model

Authors:

Klaus-Peter Engelbrech,

Florian Gödde,

Hamed Ketabdar,

Sebastian MöllerAuthors Info & Claims

SIGDIAL '09: Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue

Pages 170 - 177

Published: 11 September 2009 Publication History

Abstract

Models for predicting judgments about the quality of Spoken Dialog Systems have been used as overall evaluation metric or as optimization functions in adaptive systems. We describe a new approach to such models, using Hidden Markov Models (HMMs). The user's opinion is regarded as a continuous process evolving over time. We present the data collection method and results achieved with the HMM model.

References

[1]

Hua Ai, Fuliang Weng. 2008. User Simulation as Testing for Spoken Dialog Systems. Proc. of the 9th SIGdial Workshop on Discourse and Dialogue, Columbus, Ohio.

Digital Library

[2]

Grace Chung. 2004. Developing a flexible spoken dialog system using simulation. Proc. of the 42nd Annual Meeting on Association for Computational Linguistics, Barcelona, Spain.

Digital Library

[3]

Klaus-Peter Engelbrecht, Sebastian Möller. 2007. Pragmatic Usage of Linear Regression Models for the Prediction of User Judgments. Proc. of 8th SIGdial Workshop on Discourse and Dialogue, Antwerp, Belgium.

[4]

Klaus-Peter Engelbrecht, Michael Kruppa, Sebastian Möller, Michael Quade. 2008a. MeMo Workbench for Semi-Automated Usability Testing. Proc. of 9th Interspeech, Brisbane, Australia.

[5]

Klaus-Peter Engelbrecht, Sebastian Möller, Robert Schleicher, Ina Wechsung. 2008b. Analysis of PARADISE Models for Individual Users of a Spoken Dialog System. Proc. of ESSV 2008, Frankfurt/Main, Germany.

[6]

Klaus-Peter Engelbrecht, Felix Hartard, Florian Gödde, Sebastian Möller. 2009. A Closer Look at Quality Judgments of Spoken Dialog Systems, submitted to Interspeech 2009.

[7]

Norman M. Fraser, G. Nigel Gilbert. 1991. Simulating speech systems. Computer Speech and Language, 5:81--99.

[8]

Rainer Guski. 1999. Personal and Social Variables as Co-determinants of Noise Annoyance. Noise & Health, 3:45--56.

[9]

Kate S. Hone, Robert Graham. 2001. Subjective Assessment of Speech-system Interface Usability. Proc. of EUROSPEECH, Aalborg, Denmark.

[10]

ITU-T Rec. P.851, 2003. Subjective Quality Evaluation of Telephone Services Based on Spoken Dialogue Systems, International Telecommunication Union, Geneva, Switzerland.

[11]

Ute Jekosch. 2005. Voice and Speech Quality Perception. Assessment and Evaluation, Springer, Berlin, Germany.

Digital Library

[12]

Ramón López-Cózar, Ángel de la Torre, José C. Segura and Antonio J. Rubio. 2003. Assessment of Dialogue Systems by Means of a New Simulation Technique. Speech Communication, 40(3):387--407.

Digital Library

[13]

Sebastian Möller. 2005a. Perceptual Quality Dimensions of Spoken Dialog Systems: A Review and New Experimental Results, Proc. of Forum Acusticum, Budapest, Hungary.

[14]

Sebastian Möller. 2005b. Quality of Telephone-based Spoken Dialog Systems. Springer, New York.

Digital Library

[15]

Sebastian Möller, Klaus-Peter Engelbrecht, Robert Schleicher. 2008. Predicting the Quality and Usability of Spoken Dialogue Services, Speech Communication, 50:730--744.

Digital Library

[16]

Morris A. Okun, Renee M. Weir. 1990. Toward a Judgment Model of College Satisfaction. Educational Psychological Review, 2(1):59--76.

[17]

Lawrence R. Rabiner. 1989. A tutorial on HMM and selected applications in speech recognition. Proc. IEEE, 77(2):257--286.

[18]

Verena Rieser, Oliver Lemon. 2008. Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation. Proc. of LREC'08, Marrakech, Morocco.

[19]

Bernhard Steimel, Oliver Jacobs, Norbert Pfleger, Sebastian Paulke. 2008. Testbericht VOICE Award 2008: Die besten deutschsprachigen Sprachapplikationen. Initiative Voice Business, Bad Homburg, Germany.

[20]

Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, Alicia Abella. 1997. PARADISE: A Framework for Evaluating Spoken Dialogue Agents. Proc. of ACL/EACL 35th Ann. Meeting of the Assoc. for Computational Linguistics, Madrid, Spain.

Digital Library

[21]

Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, Alicia Abella. 1998. Evaluating Spoken Dialog Agents with PARADISE: Two Case Studies. Computer Speech and Language, 12:317--347.

[22]

Marilyn Walker, Candace Kamm, Diane Litman. 2000. Towards Developing General Models of Usability with PARADISE. Natural Language Engineering, 6(3--4):363--377.

Digital Library

[23]

Craig Wootton, Michael McTear, Terry Anderson. 2007. Utilizing Online Content as Domain Knowledge in a Multi-Domain Dynamic Dialogue System. Proc. of Interspeech 2007, Antwerp, Belgium.

Cited By

Cai WChen L(2020)Predicting User Intents and Satisfaction with Dialogue-based Conversational RecommendationsProceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3340631.3394856(33-42)Online publication date: 7-Jul-2020
https://dl.acm.org/doi/10.1145/3340631.3394856
Robb DLopes JPadilla SLaskov AChiyah Garcia FLiu XScharff Willners JValeyrie NLohan KLane DPatron PPetillot YChantler MHastie HHarrison SBardzell SNeustaedter CTatar D(2019)Exploring Interaction with Remote Autonomous Systems using Conversational AgentsProceedings of the 2019 on Designing Interactive Systems Conference10.1145/3322276.3322318(1543-1556)Online publication date: 18-Jun-2019
https://dl.acm.org/doi/10.1145/3322276.3322318
Ultes SSchmitt AMinker WEskenazi MBlack ATraum D(2012)Towards quality-adaptive spoken dialogue managementNAACL-HLT Workshop on Future Directions and Needs in the Spoken Dialog Community: Tools and Data10.5555/2390444.2390469(49-52)Online publication date: 7-Jun-2012
https://dl.acm.org/doi/10.5555/2390444.2390469
Show More Cited By

Index Terms

Modeling user satisfaction with Hidden Markov Model

Recommendations

Logical Hierarchical Hidden Markov Models for Modeling User Activities
ILP '08: Proceedings of the 18th international conference on Inductive Logic Programming

Hidden Markov Models (HMM) have been successfully used in applications such as speech recognition, activity recognition, bioinformatics etc. There have been previous attempts such as Hierarchical HMMs and Abstract HMMs to elegantly extend HMMs at ...
A Hidden Semi-Markov Model-Based Speech Synthesis System

A statistical speech synthesis system based on the hidden Markov model (HMM) was recently proposed. In this system, spectrum, excitation, and duration of speech are modeled simultaneously by context-dependent HMMs, and speech parameter vector sequences ...
Neural Hidden Markov Model
Agents and Artificial Intelligence
Abstract
Hidden Markov models are tractable to capture long-term dependencies but intractable to compute the transition probabilities of higher-order process. We propose a neural hidden Markov models to compute the transition probabilities of higher-order ...

Comments

Information & Contributors

Information

Published In

cover image DL Hosted proceedings

SIGDIAL '09: Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue

September 2009

382 pages

ISBN:9781932432640

Editor:
Matthew Purver
Queen Mary University of London, UK
,
General Chairs:
Patrick G. T. Healey
Queen Mary University of London, UK
,
Roberto Pieraccini
SpeechCycle
,
Program Chairs:
Donna Byron
Northeastern University
,
Steve Young
University of Cambridge, UK

Publisher

Association for Computational Linguistics

United States

Publication History

Published: 11 September 2009

Qualifiers

Research-article

Acceptance Rates

Overall Acceptance Rate 19 of 46 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
518
Total Downloads

Downloads (Last 12 months)84
Downloads (Last 6 weeks)5

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cai WChen L(2020)Predicting User Intents and Satisfaction with Dialogue-based Conversational RecommendationsProceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3340631.3394856(33-42)Online publication date: 7-Jul-2020
https://dl.acm.org/doi/10.1145/3340631.3394856
Robb DLopes JPadilla SLaskov AChiyah Garcia FLiu XScharff Willners JValeyrie NLohan KLane DPatron PPetillot YChantler MHastie HHarrison SBardzell SNeustaedter CTatar D(2019)Exploring Interaction with Remote Autonomous Systems using Conversational AgentsProceedings of the 2019 on Designing Interactive Systems Conference10.1145/3322276.3322318(1543-1556)Online publication date: 18-Jun-2019
https://dl.acm.org/doi/10.1145/3322276.3322318
Ultes SSchmitt AMinker WEskenazi MBlack ATraum D(2012)Towards quality-adaptive spoken dialogue managementNAACL-HLT Workshop on Future Directions and Needs in the Spoken Dialog Community: Tools and Data10.5555/2390444.2390469(49-52)Online publication date: 7-Jun-2012
https://dl.acm.org/doi/10.5555/2390444.2390469
Schmitt ASchatz BMinker WMoore JTraum DChai JPassonneau R(2011)Modeling and predicting quality in spoken human-computer interactionProceedings of the SIGDIAL 2011 Conference10.5555/2132890.2132910(173-184)Online publication date: 17-Jun-2011
https://dl.acm.org/doi/10.5555/2132890.2132910
Schmitt AMinker WSharaf NKatagiri YNakano MFernández RLemon O(2010)Advances in the Witchcraft workbench projectProceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue10.5555/1944506.1944554(261-264)Online publication date: 24-Sep-2010
https://dl.acm.org/doi/10.5555/1944506.1944554
Higashinaka RMinami YDohsaka KMeguro TKatagiri YNakano MFernández RLemon O(2010)Modeling user satisfaction transitions in dialogues from overall ratingsProceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue10.5555/1944506.1944510(18-27)Online publication date: 24-Sep-2010
https://dl.acm.org/doi/10.5555/1944506.1944510
Higashinaka RMinami YDohsaka KMeguro T(2010)Issues in predicting user satisfaction transitions in dialoguesProceedings of the Second international conference on Spoken dialogue systems for ambient environments10.5555/1925948.1925954(48-60)Online publication date: 1-Oct-2010
https://dl.acm.org/doi/10.5555/1925948.1925954
Tran HMellouk A(2010)QoE model driven for network servicesProceedings of the 8th international conference on Wired/Wireless Internet Communications10.1007/978-3-642-13315-2_22(264-277)Online publication date: 1-Jun-2010
https://dl.acm.org/doi/10.1007/978-3-642-13315-2_22

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents