article

Eye communication in a conversational 3D synthetic agent

Authors:

Isabella Poggi,

Catherine Pelachaud,

Fiorella De RosisAuthors Info & Claims

AI Communications, Volume 13, Issue 3

Pages 169 - 181

Published: 15 November 2000 Publication History

Abstract

Our goal is to create an ‘intelligent’ 3D agent able to send complex, ‘natural’ messages to users and, in the future, to converse with them. We look at the relationship between the agent's communicative intentions and the way that these intentions are expressed into verbal and nonverbal messages. In this paper, we concentrate on the study and generation of coordinated linguistic and gaze communicative acts. In this view we analyse gaze signals according to their functional meaning rather than to their physical actions. We propose a formalism where a communicative act is represented by two elements: a meaning (that corresponds to a set of goals and beliefs that the agent has the purpose to transmit to the interlocutor) and a signal, that is the nonverbal expression of that meaning. We also outline a methodology to generate messages that coordinate verbal with nonverbal signals.

References

[1]

{1} E. André and T. Rist, Generating coherent presentations employing textual and visual material, Artificial Intelligence Review Journal, Special Volume on the Integration of NL and Vision Processing 9 (1995), 147-165.

[2]

{2} E. André, T. Rist and J. Mueller, Integrating reactive and scripted behaviors in a life-like presentation agent, in: Proceedings of the Second International Conference on Autonomous Agents, 1998, pp. 261-268.

Digital Library

[3]

{3} E. André, T. Rist and J. Mueller, Webpersona: A lifelike presentation agent for the World-Wide Web, Knowledge-Based Systems 11(1) (1998).

[4]

{4} J.L. Austin, How to Do Things with Words, Oxford University Press, London, 1962.

[5]

{5} D.C. Berry, B. De Carolis, F. de Rosis and I. Michas, Evaluating plan-based hypermedia generation, in: Proceedings of the 7th European Workshop on Natural Language Generation, Toulouse, France, 1999.

[6]

{6} N. Badler, M. Palmer and R. Bindiganavale, Animation control for real-time virtual humans, Communications of the ACM 42(8) (1999), 64-73.

Digital Library

[7]

{7} J. Beskow, Animation of talking agents, in: Proceedings of ESCA, AVSP '97 Workshop, C. Benoit and R. Campbell, eds, Rhodes, Greece, 1997.

[8]

{8} J. Beskow and S. McGlashan, Olga - A conversational agent with gestures, in: Proc. of IJCAI '97 - Workshop on Animated Interface Agents - Making them Intelligent, Nagoya, Japan, Morgan-Kaufmann Publishers, San Francisco, 1997.

[9]

{9} B. De Carolis, F. de Rosis, F. Grasso, A. Rossiello, D.C. Berry and T. Gillie, Generating recipient-centered explanations about drug prescription, Artificial Intelligence in Medicine 8 (1996), 123-145.

[10]

{10} J. Cassell, J. Bickmore, M. Billinghurst, L. Campbell, K. Chang, H. Vilhjálmsson and H. Yan, Embodiment in conversational interfaces: Rea, in: CHI '99, Pittsburgh, PA, 1999, pp. 520-527.

Digital Library

[11]

{11} J. Cassell and M. Stone, Living hand and mouth. Psychological theories about speech and gestures in interactive dialogue systems, in: AAAI99 Fall Symposium on Psychological Models of Communication in Collaborative Systems, 1999.

[12]

{12} J. Cassell, J. Sullivan, S. Prevost and E. Churchill, eds, Embodied Conversational Characters, MITpress, Cambridge, MA, 2000.

[13]

{13} C. Castelfranchi and D. Parisi, Linguaggio, Conoscenze e Scopi, Il Mulino, Bologna, 1980.

[14]

{14} A. Cawsey, Explanation and Interaction: The Computer Generation of Explanatory Dialogues, MIT Press Series in Natural Language, 1992.

[15]

{15} N. Chomsky, Aspects of the Theory of Syntax, MIT Press, Cambridge, MA, 1965.

[16]

{16} S. Chopra-Khullar and N.I. Badler, Where to look? Automating attending behaviors of virtual human characters, in: Proceedings of Autonomous Agents '99, Seattle, WA, 1999.

Digital Library

[17]

{17} E. Churchill, S. Prevost, T. Bickmore, P. Hodgson, T. Sullivan and L. Cook, Design issues for situated conversational characters, in: WECC '98, The First Workshop on Embodied Conversational Characters, 1998.

[18]

{18} P.R. Cohen and H.J. Levesque, Rational interaction as the basis for communication, in: Intentions in Communication, J. Morgan, P.R. Cohen and M.E. Pollack, eds, MIT Press, 1990.

[19]

{19} P. Ekman, About brows: Emotional and conversational signals, in: Human Ethology: Claims and Limits of a New Discipline: Contributions to the Colloquium, M. von Cranach, K. Foppa, W. Lepenies and D. Ploog, eds, Cambridge University Press, Cambridge, England, New-York, 1979, pp. 169-248.

[20]

{20} P. Ekman and W. Friesen, Facial Action Coding System, Consulting Psychologists Press, Inc., Palo Alto, CA, 1978.

[21]

{21} M.R. Genesereth and N.J. Nilsson, Logical Foundations of Artificial Intelligence, Morgan-Kaufmann Publishers, 1988.

[22]

{22} B.J. Grosz and C.L. Sidner, Attention, intentions, and the structure of discourse, Computational Linguistics 12(3) (1986).

[23]

{23} E. Hovy and L. Wanner, Managing sentence planning requirements, in: Proceedings of the ECAI 96 Workshop on 'Gaps and Bridges: New Directions in Planning and NLG', 1996.

[24]

{24} E.H. Hovy, Automated discourse generation using discourse structure relations, Artificial Intelligence 63 (1993), 341-385.

Digital Library

[25]

{25} W.C. Mann, C.M.I.M. Matthiessen and S. Thompson, Rhetorical structure theory and text analysis, Technical Report 89-242, ISI Research, 1989.

[26]

{26} M. Maybury, Planning multimedia explanations using communicative acts, in: Intelligent Multimedia Interfaces, M. Maybury, ed., Cambridge AAAI Press, 1993.

Digital Library

[27]

{27} J.D. Moore, Participating in Explanatory Dialogues: Interpreting and Responding to Questions in Context, The MIT Press, Cambridge, MA, USA, 1995.

[28]

{28} D. Morris, Manwatching, Cape, London, 1977.

[29]

{29} T. Noma and N. Badler, A virtual human presenter, in: Proceedings of the IJCAI '97 Workshop on Animated Interface Agents - Making them Intelligent, Nagoya, Japan, Morgan-Kaufmann Publishers, 1997.

[30]

{30} F.I. Parke and K. Waters, Computer Facial Animation, A.K. Peters, Wellesley, MA, 1996.

[31]

{31} S.M. Platt, A Structural model of the human face, PhD thesis, University of Pennsylvania, Dept. of Computer and Information Science, Philadelphia, PA, 1985.

[32]

{32} I. Poggi and C. Pelachaud, Performative faces, Speech Communication 26 (1998), 5-21.

Digital Library

[33]

{33} I. Poggi and C. Pelachaud, Emotional meaning and expression in performative faces, in: International Workshop on Affect in Interactions: Towards a New Generation of Interfaces, Annual Conference AC '99 of the EC I3 Programme, Siena, 1999.

[34]

{34} M. Reape and C. Mellish, Just what is aggregation anyhow? in: Proceedings of the 7th European Workshop on Natural Language Generation, Toulouse, France, 1999.

[35]

{35} J. Rickel and W.L. Johnson, Animated agents for procedural training in virtual reality: Perception, cognition, and motor control, Applied Artificial Intelligence 13 (1999), 343-382.

[36]

{36} T. Rist, E. André and J. Müller, Adding animated presentation agents to the interface, in: Intelligent User Interface, 1997, pp. 79-86.

[37]

{37} J.R. Searle, Speech Acts, Cambridge University Press, London, 1969.

[38]

{38} K.R. Thórisson, Layered modular action control for communicative humanoids, in: Computer Animation '97, Geneva, Switzerland, IEEE Computer Society Press, 1997.

[39]

{39} K. Waters, J. Rehg, M. Loughlin, S.B. Kang and D. Terzopoulos, Visual sensing of humans for active public interfaces, Technical Report CRL 96/5, Cambridge Research Laboratory, Digital Equipment Corporation, 1996.

Cited By

(2022)The Handbook on Socially Interactive AgentsundefinedOnline publication date: 27-Oct-2022
Oker APecune FDeclercq C(2020)Virtual tutor and pupil interaction: A study of empathic feedback as extrinsic motivation for learningEducation and Information Technologies10.1007/s10639-020-10123-525:5(3643-3658)Online publication date: 1-Sep-2020
Admoni HScassellati B(2017)Social eye gaze in human-robot interactionJournal of Human-Robot Interaction10.5898/JHRI.6.1.Admoni6:1(25-63)Online publication date: 26-May-2017
Show More Cited By

Recommendations

Eye Gaze for Spoken Language Understanding in Multi-modal Conversational Interactions
ICMI '14: Proceedings of the 16th International Conference on Multimodal Interaction

When humans converse with each other, they naturally amalgamate information from multiple modalities (i.e., speech, gestures, speech prosody, facial expressions, and eye gaze). This paper focuses on eye gaze and its combination with speech. We develop a ...
A conversational agent as museum guide: design and evaluation of a real-world application
Lecture Notes in Computer Science

This paper describes an application of the conversational agent Max in a real-world setting. The agent is employed as guide in a public computer museum, where he engages with visitors in natural face-to-face communication, provides them with information ...
Entrainment in Human-Agent Text Communication
Agent Computing and Multi-Agent Systems

Non-verbal information such as utterance speed and switching pause create an impression of the speaker. If intelligent agents could handle such non-verbal information properly, the quality of interactions between agents and human users would improve. ...

Comments

Information & Contributors

Information

Published In

cover image AI Communications

AI Communications Volume 13, Issue 3

November 2000

68 pages

ISSN:0921-7126

Issue’s Table of Contents

Publisher

IOS Press

Netherlands

Publication History

Published: 15 November 2000

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
12
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

(2022)The Handbook on Socially Interactive AgentsundefinedOnline publication date: 27-Oct-2022
Oker APecune FDeclercq C(2020)Virtual tutor and pupil interaction: A study of empathic feedback as extrinsic motivation for learningEducation and Information Technologies10.1007/s10639-020-10123-525:5(3643-3658)Online publication date: 1-Sep-2020
Admoni HScassellati B(2017)Social eye gaze in human-robot interactionJournal of Human-Robot Interaction10.5898/JHRI.6.1.Admoni6:1(25-63)Online publication date: 26-May-2017
Kang SFeng ALeuski ACasas DShapiro ALee MOmori TOsawa HPark HYoung J(2015)The Effect of An Animated Virtual Character on Mobile Chat InteractionsProceedings of the 3rd International Conference on Human-Agent Interaction10.1145/2814940.2814957(105-112)Online publication date: 21-Oct-2015
Ruhland KPeters CAndrist SBadler JBadler NGleicher MMutlu BMcDonnell R(2015)A Review of Eye Gaze in Virtual Agents, Social Robotics and HCIComputer Graphics Forum10.1111/cgf.1260334:6(299-326)Online publication date: 1-Sep-2015
Pejsa TAli Salah ACohn JSchuller BAran OMorency LCohen P(2014)Authoring Communicative Behaviors for Situated, Embodied CharactersProceedings of the 16th International Conference on Multimodal Interaction10.1145/2663204.2667576(399-403)Online publication date: 12-Nov-2014
Andrist SPejsa TMutlu BGleicher M(2012)A head-eye coordination model for animating gaze shifts of virtual charactersProceedings of the 4th Workshop on Eye Gaze in Intelligent Human Machine Interaction10.1145/2401836.2401840(1-6)Online publication date: 26-Oct-2012
Moubayed SBeskow JGranström BHouse D(2010)Audio-visual prosodyProceedings of the Third COST 2102 international training school conference on Toward autonomous, adaptive, and context-aware multimodal interfaces: theoretical and practical issues10.5555/1950280.1950287(55-71)Online publication date: 15-Mar-2010
Rienks RPoppe RHeylen D(2010)Differences in head orientation behavior for speakers and listenersACM Transactions on Applied Perception10.1145/1658349.16583517:1(1-13)Online publication date: 18-Jan-2010
Wissner MBee NKienberger JAndré E(2009)To see and to be seen in the virtual beer gardenProceedings of the 32nd annual German conference on Advances in artificial intelligence10.5555/1814110.1814185(500-507)Online publication date: 15-Sep-2009
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents