Empirical evaluation methodology for embodied conversational agents
Abstract
The objective of this chapter is to identify the common knowledge and practice in research methodology and to apply it to the field of software evaluation, especially of embodied conversational agents. Relevant issues discussed are: how to formulate a good research question, what research strategy to use, which data collection methods are most appropriate and how to select the right participants. Reliability and validity of the data sets are dealt with and finally the chapter concludes with a list of guidelines that one should keep in mind when setting up and conducting empirical evaluation studies on embodied conversational agents.
References
[1]
Andersson, G., Höök, K., Mourão, D., Paiva, A., and Costa, M. (2002). Using a Wizard of Oz study to inform the design of SenToy. Personal and ubiquitous computing, 6(5-6): 378-389.
[2]
Berg, B.L. (2001). Qualitative Research Methods for the Social Sciences. Allyn and Bacon, Boston.
[3]
Boehm, B. (1988). The spiral model of software development and enhancement. IEEE Computer, 21(5): 61-72.
[4]
Buisine, S., Abrilian, S., Rendu, C., and Martin, J. (2002). Towards experimental specification and evaluation of lifelike multimodal behaviour. In Proceedings of AAMAS 2002 workshop: Embodied conversational agents - let's specify and evaluate them!, Bologna, Italy.
[5]
Campbell, D.T. and Fiske, D.W. (1959). Convergent and discriminant validation by the multi trait-multi method matrix. Psychological Bulletin , 56: 81-105.
[6]
Christoph, L.H. and Van de Sande, J.P. (1999). Werkboek gedragsobservatie: systematisch observation en The Observer {Workbook observing behaviour: systematic observation and The Observer}. Wolters-Noordhoff, Groningen, The Netherlands.
[7]
Cohen, J.A. (1960). Coefficient of agreement for nominal scales. Educational and Psychological measurement, 20: 37-46.
[8]
Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika., 16: 297-334.
[9]
Coolican, H. (1994). Research Methods and Statistics in Psychology. Hodder and Stoughton, London.
[10]
Cowell, A.J. and Stanney, K.M. (2003). On manipulating nonverbal interaction style to increase anthropomorphic computer character credibility. In Proceedings of AAMAS 2003 workshop: Embodied conversational characters as individuals, Melbourne, Australia.
[11]
De Furia, G.L. (1996). A behavioral model of interpersonal trust. Unpublished Ph.D. thesis. St. John's University, Springfield, L.A., USA.
[12]
Dehn, D.M. and Van Mulken, S. (2000). The impact of animated interface agents: A review of empirical research. Int. J. human-computer studies, 52(1): 1-22.
[13]
Erdfelder, E., Faul, F., and Buchner, A. (1996). GPOWER: A general power analysis program. Behavior Research Methods, Instruments, and Computers, 28: 1-11.
[14]
Guilford, J.P. and Fruchter, B. (1978). Fundamental statistics in psychology and education. McGraw-Hill, New York.
[15]
Hix, D. and Harston, H.R. (1993). Developing user interfaces: ensuring usability through product and process. Wiley, New York, USA.
[16]
Holm, R., Priglinger, M., Stauder, E., Volkert, J., and Wagner, R. (2002). Automatic data acquisition and visulatization for usability evaluation of virtual reality systems. In Proceedings of Eurographics Short Presentations , Saarbrücken, Germany.
[17]
Höök, K. (2002). Evaluation of affective interfaces. In Proceedings of AAMAS 2002 workshop: Embodied conversational agents - let's specify and evaluate them!, Bologna, Italy.
[18]
Howel, D.C. (1982). Statistical methods for psychology. Duxbury Press, Boston, Mass.
[19]
Johnson, R. (1988). Elementary statistics. PWS-kent publishing company, Boston.
[20]
Kabel, S., De Hoog, R., and Sandberg, J. (1997). User interface evaluation and improvements: A framework and empirical results. Internal report SWI-UVA.
[21]
Krahmer, E., van Buuren, S., Ruttkay, Zs., and Wesselink, W. (2003). Audio-visual personality cues for embodied agents; an experimental evaluation. In Proceedings of AAMAS 2003 workshop: Embodied conversational characters as individuals, Melbourne, Australia.
[22]
Mangione, T.W. (1995). Mail surveys: Improving the quality. SAGE publications, Thousand Oakes, CA.
[23]
Morishima, S. and Nakamura, S. (2002). Multi-modal translation and evaluation of lip-synchronization using noise added voice. In Proceedings of AAMAS 2002 workshop: Embodied conversational agents - let's specify and evaluate them!, Bologna, Italy.
[24]
Mosteller, F. and Rourke, R.E.K. (1973). Sturdy statistics: Nonparametrics and order statistics. Addison-Wesley, Massachusetts.
[25]
Moundridou, M. and Virvou, M. (2002). Evaluating the persona effect on an interface agent in an intelligent tutoring system. Journal of computer assisted learning, 18(3): 253-261.
[26]
Neter, J., Wasserman, W., and Kutner, M.H. (1990). Applied linear statistical models: regression, analysis of variance and experimental design . Irwin, Boston.
[27]
Neale, J.M. and Liebert, R.M. (1986). Science and behavior. An introduction to methods of research. Prentice Hall International editions, New York
[28]
Norman D.A. (1986). Cognitive engineering. In, Norman, D.A. and Draper, S., editors. User Centered Systems Design: new perspectives on human-computer interaction, pp. 31-61, Erlbaum Associates, Hillsdale, NJ.
[29]
Nielsen, J. (1993). Usability engineering. Morgan Kaufmann, San Francisco.
[30]
Norusis, M.J. (2002). SPSS 11.0, guide to data analysis. Prentice Hall, New Jersey.
[31]
Oates, J., Gove, J., Goudge, A., Hill, R., Littleton, K., Christoph, L.H., Edwards, N., Gardner, R., Grayson, A., and Manners, P. (2000). fOCUS: a CD-ROM based application for developing observation skills. Winner of the European Academic Software Awards (EASA), November 2000, Rotterdam, The Netherlands.
[32]
Preece, J., Rogers, R., Sharp, H., Benyon, D., Holland, S., and Carey, T. (1994). Human-computer interaction. Addison-Wesley, England.
[33]
Reeves, T.C. and Hedberg, J.G. (2003). Interactive learning systems evaluation, Educational Technology Publications, Englewood Cliffs, NJ.
[34]
Rempel, J.K. and Holmes, J.G. (1986). How do I trust thee. Psychology Today, 20: 28-34.
[35]
Ruttkay, Zs., Dormann, C., and Noot, H. (2002). Evaluating ECAs -- what and how?. In Proceedings of AAMAS 2002 workshop: Embodied conversational agents - let's specify and evaluate them!, Bologna, Italy.
[36]
Sande, J.P., van de (1999). Gedragsobservatie: een inleiding tot system-atisch observeren {Observing behaviour: an introduction to systematic observation}. Wolters-Noordhoff, Groningen, The Netherlands.
[37]
Silverman, D. (2000). Doing qualitative research: a practical handbook. SAGE publications, London.
[38]
Spradley, J.P. (1980). Participant observation. Holt Rinehart and Winston, New York.
[39]
SPSS Inc. (2002). SPSS version 11.0 for Windows. SPSS Inc., Chicago IL.
[40]
STATDISK (2003). STATDISK version 9.5 for Windows. Addison-Wesley, Boston.
[41]
Swanborn, P.G. (1997). Basisboek social onderzoek {Handbook of social research}. Boom, Meppel, Amsterdam, The Netherlands.
[42]
Triola., M.F. (2002). Essentials of statistics. Addison-Wesley, Boston.
[43]
Verschuren, P. and Doorewaard, H. (1999). Designing a research project. Lemma, Utrecht, The Netherlands.
[44]
Vocht, de, A. (2002). Basishandbook SPSS 11 voor Windows (Handbook SPSS 11 for Windows). Bijleveld press. Utrecht, The Netherlands.
[45]
Wilkinson, J. (1995). Direct observation. In. G.M. Breakwell, S. Hammond, and C. Fife-Schaw (Eds). Research methods in psychology. London, Sage publications.
[46]
Xiao. J., Stasko, J. and Catrambone, R. (2002). Embodied conversational agents as a UI paradigm: a framework for evaluation. In Proceedings of AAMAS 2002 workshop: Embodied conversational agents let's spcify and evaluate them!, Bologna, Italy.
Index Terms
- Empirical evaluation methodology for embodied conversational agents
Recommendations
Embodied conversational agents: computing and rendering realistic gaze patterns
PCM'06: Proceedings of the 7th Pacific Rim conference on Advances in Multimedia Information ProcessingWe describe here our efforts for modeling multimodal signals exchanged by interlocutors when interacting face-to-face. This data is then used to control embodied conversational agents able to engage into a realistic face-to-face interaction with human ...
Personality Analysis of Embodied Conversational Agents
IVA '18: Proceedings of the 18th International Conference on Intelligent Virtual AgentsPeople tend to personify machines. Giving machines the ability to actually produce social information can help improve human-machine interactions. Embodied Conversational Agents (ECAs) are virtual software agents that can process and produce speech, ...
Trustworthy Embodied Conversational Agents for Healthcare: A Design Exploration of Embodied Conversational Agents for the periconception period at Erasmus MC
CUI '23: Proceedings of the 5th International Conference on Conversational User InterfacesThis paper explores the potential implications of embodied conversational agents (ECAs) in healthcare, focusing on the impact of appearance and conversation style on trustworthiness. We conducted a Research through Design investigation of ECAs for ...
Comments
Information & Contributors
Information
Published In
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 01 January 2004
Author Tags
Qualifiers
- Chapter
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 16 Oct 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in