Abstract
Focusing on deixis in human computer interaction this paper presents interdisciplinary work on the use of co-verbal gesture . Empirical investigations, theoretical modeling, and computational simulations with an anthropomorphic agent are based upon comparable settings and common representations. Findings pertain to the coordination of verbal and gestural constituents in deictic utterances. We discovered high variability in the temporal synchronization of such constituents in task-oriented dialogue, and a theoretical treatment thereof is presented. With respect to simulation we exemplarily show how the influence of situational characteristics on the choice of verbal and nonverbal constituents can be accounted for. In particular, this depends on spatio-temporal relations between speaker and the objects they refer to in dialogue.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Bernsen, N.O., Stock, O. (eds.): Proceedings of the International Workshop on Information Presentation and Natural Multimodal Dialogue – ipnmd 2001, Verona, Italy, itc-irst, December 14–15 (2001)
Cassell, J., Vilhjalmsson, H., Bickmore, T.: BEAT: the Behavior Expression Animation Toolkit. In: Fiume, E. (ed.) Proceedings of SIGGRAPH 2001, pp. 477–486. ACM Press/ACM SIGGRAPH (2001)
Cassell, J., Prevost, S.: Distribution of Semantic Features Across Speech and Gesture by Humans and Machines. In: Proceedings of the Workshop on the Integration of Gesture in Language and Speech (1996)
deRuiter, J.P.: The production of gesture and speech. In: McNeill, D. (ed.) Language and gesture, ch. 14, pp. 284–311. Cambridge University Press, Cambridge (2000)
Kaplan, D.: On the Logic of Demonstratives. Journ. Phil. Logic 8, 81–98 (1979)
Kendon, A.: Gesticulation and speech: Two aspects of the process of utterance. In: Key, M.R. (ed.) The Relationship of Verbal and Nonverbal Communication, pp. 207–227. The Hague, Mouton (1980)
Kopp, S., Wachsmuth, I.: A Knowledge-based Approach for Lifelike Gesture Animation. In: Horn, W. (ed.) ECAI 2000 - Proceedings of the 14th European Conference on Artificial Intelligence, pp. 663–667. IOS Press, Amsterdam (2000)
Kopp, S., Wachsmuth, I.: Model-based Animation of Coverbal Gesture. In: Proceedings of Computer Animations 2002, pp. 252–257. IEEE Press, Los Alamitos (2002)
Kranstedt, A., Kopp, S., Wachsmuth, I.: MURML: A Multimodal Utterance Representation Markup Language for Conversational Agents. In: Proceedings of the AAMAS 2002 Workshop on Embodied Conversational Agents – let’s specify and evaluate them, Bologna, Italy (July 2002)
Krauss, R., Chen, Y., Gottesman, R.: Lexical gestures and lexical access: a process model. In: McNeill, D. (ed.) Language and gesture, ch. 13, pp. 261–283. Cambridge University Press, Cambridge (2000)
Latoschik, M.E.: A Gesture Processing Framework for Multimodal Interaction in Virtual Reality. In: Chalmers, A., Lalioti, V. (eds.) Afrigraph 2001, 1st International Conference on Computer Graphics, Virtual Reality and Visualization in Africa, November 5-7, pp. 95–100. ACM SIGGRAPH, New York (2001)
Levelt, W.J.: Speaking. MIT Press, Cambridge (1989)
Levin, J.A., Moore, J.A.: Dialogue Games: Metacommunication Structures for Natural Language Interaction. Cognitive Science 1(4), 395–420 (1978)
Mann, W.C.: Dialogue Games: Conventions of Human Interaction. Argumentation 2, 512–532 (1988)
Mann, W.C.: Dialogue macrogame theory. Revised version of a paper presented at SIGdial, Philadelphia, Pennsylvania USA (July 2002), http://www-rcf.usc.edu/~billmann/dialogue/dmt-paper1.htm
McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)
McNeill, D.: Models of speaking (to their amazement) meet speech-synchronized gestures (1998), Obtained from the cogprints archives: http://cogprints.soton.ac.uk/
Milde, J.-T., Gut, U.: The TASX-environment: an XML-based corpus database for time aligned language data. In: Proceedings of the IRCS Workshop on linguistic databases, Philadelphia (2001)
Noma, T., Badler, N.: A Virtual Human Presenter. In: Proceedings of the IJCAI Workshop on Animated Interface Agents: Making Them Intelligent, pp. 45–51 (1997)
Paggio, P., Jongejan, B.: Multimodal communication in the virtual farm of the Staging project. In: [1], pp. 41–45 (2001)
Piwek, P., Beun, R.J., Cremers, A.: Demonstratives in Dutch Cooperative Task Dialogues. IPO Manuscript 1134, Eindhoven University of Technology (1995)
Piwek, P., Beun, R.J.: Multimodal Referential Acts in a Dialogue Game: From Empirical Investigation to Algorithm. In: [1], pp. 127–131 (2001)
Prillwitz, S.: HamNoSys. Version 2. Hamburger Notationssystem für Gebärdensprachen. Eine Einführung. SIGNUM-Verlag (1989)
Sag, I., Wasow, T.: Syntactic Theory – A Formal Introduction. CSLI (1999)
Sowa, T., Kopp, S., Latoschik, M.E.: A Communicative Mediator in a Virtual Environment: Processing of Multimodal Input and Output. In: [1], pp. 71–74 (2001)
Weiß, P., Barattelli, S.: Das Benennen von Objekten. In: Herrman, T., Grabowski, J. (eds.) Enzyklopädie der Psychologie: Themenbereich C, Theorie und Forschung, Hogrefe. Sprache, vol. 3 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kranstedt, A., Kühnlein, P., Wachsmuth, I. (2004). Deixis in Multimodal Human Computer Interaction: An Interdisciplinary Approach. In: Camurri, A., Volpe, G. (eds) Gesture-Based Communication in Human-Computer Interaction. GW 2003. Lecture Notes in Computer Science(), vol 2915. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24598-8_11
Download citation
DOI: https://doi.org/10.1007/978-3-540-24598-8_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-21072-6
Online ISBN: 978-3-540-24598-8
eBook Packages: Springer Book Archive