article

Semiotic schemas: A framework for grounding language in action and perception

Author:

Deb RoyAuthors Info & Claims

Artificial Intelligence, Volume 167, Issue 1-2

Pages 170 - 205

Published: 01 September 2005 Publication History

Abstract

A theoretical framework for grounding language is introduced that provides a computational path from sensing and motor action to words and speech acts. The approach combines concepts from semiotics and schema theory to develop a holistic approach to linguistic meaning. Schemas serve as structured beliefs that are grounded in an agent's physical environment through a causal-predictive cycle of action and perception. Words and basic speech acts are interpreted in terms of grounded schemas. The framework reflects lessons learned from implementations of several language processing robots. It provides a basis for the analysis and design of situated, multimodal communication systems that straddle symbolic and non-symbolic realms.

References

[1]

Allen, J., Maintaining knowledge about temporal intervals. Comm. ACM. v26. 832-843.

Digital Library

[2]

Arbib, M.A., Schema theory. In: Arbib, M.A. (Ed.), The Handbook of Brain Theory and Neural Networks, MIT Press, Cambridge, MA. pp. 993-998.

[3]

Arbib, M.A., Iberall, T. and Lyons, D., Schemas that integrate vision and touch for hand control. In: Vision, Brain, and Cooperative Computation, MIT Press, Cambridge, MA. pp. 489-510.

[4]

Bailey, D., Feldman, J., Narayanan, S. and Lakoff, G., Embodied lexical development. In: Proceedings of the Nineteenth Annual Meeting of the Cognitive Science Society, Erlbaum, Mahwah, NJ.

[5]

D. Bailey, When push comes to shove: A computational model of the role of motor control in the acquisition of action verbs, PhD thesis, Computer science division, EECS Department, University of California at Berkeley, 1997

[6]

Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D. and Jordan, M., Matching words and pictures. J. Machine Learning Res. v3. 1107-1135.

[7]

Barsalou, L., Perceptual symbol systems. Behavioural and Brain Sciences. v22. 577-609.

[8]

Barwise, J. and Perry, J., Situations and Attitudes. 1983. MIT-Bradford.

[9]

Bates, E., The Emergence of Symbols. 1979. Academic Press, New York.

[10]

Bickhard, M.H., Representational content in humans and machines. J. Experimental Theoret. Artificial Intelligence. v5. 285-333.

[11]

A. Cangelosi, K. Coventry, R. Rajapakse, D. Joyce, A. Bacon, L. Richards, S. Newstead, Grounding language in perception: A connectionist model of spatial terms and vague quantifiers, in: Ninth Neural Computation and Psychology Workshop, Plymouth, UK, 2004

[12]

A. Cangelosi, S. Harnad, The adaptive advantage of symbolic theft over sensorimotor toil: Grounding language in perceptual categories, Evolution of Communication, 2000, 2001

[13]

Coelho, J., Piaterand, J. and Grupen, R., Developing haptic and visual perceptual categories for reaching and grasping with a humanoid robot. In: Proceedings of the First IEEE-RAS International Conference on Humanoid Robots, IEEE-RAS.

[14]

P. Cohen, Projections as concepts, in: Proceedings of the 2nd European Conference on Cognitive Science, 1997, pp. 56--60

[15]

P. Cohen, T. Oates, C. Beal, and N. Adams, Contentful mental states for robot baby, in: Proceedings of the Eighteenth National Conference on Artificial Intelligence, Edmonton, AB, 2002

[16]

Coventry, K. and Garrod, S., Saying, Seeing and Acting. 2004. Psychology Press.

[17]

Dennett, D., Consciousness Explained. 1991. Little, Brown and Company.

[18]

Dennett, D., Real patterns. J. Philos. v88 i1. 27-51.

[19]

Drescher, G., Made-up Minds. 1991. MIT Press, Cambridge, MA.

[20]

Dretske, F.I., Knowledge and the Flow of Information. 1981. MIT Press, Cambridge, MA.

[21]

Feldman, J., Lakoff, G., Bailey, D., Narayanan, S., Regier, T. and Stolcke, A., Lzero: The first five years. Artificial Intelligence Rev. v10. 103-129.

[22]

Gärdenfors, P., Conceptual Spaces: The Geometry of Thought. 2000. MIT Press, Cambridge, MA.

[23]

Gibson, J.J., The Ecological Approach to Visual Perception. 1979. Erlbaum, Mahwah, NJ.

[24]

Glenberg, A. and Kaschak, M., Grounding language in action. Phychonomic Bull. Rev. v9 i3. 558-565.

[25]

Gorniak, P. and Roy, D., Grounded semantic composition for visual scenes. J. Artificial Intelligence Res. v21. 429-470.

[26]

Grice, P., Logic and conversation. In: Cole, P., Morgan, J. (Eds.), Syntax and Semantics: vol. 3, Speech Act, Academic Press, New York. pp. 43-58.

[27]

Harnad, S., The symbol grounding problem. Physica D. v42. 335-346.

[28]

Herskovits, A., Language, spatial cognition, and vision. In: Spatial and Temporal Reasoning, Kluwer Academic, Dordrecht.

[29]

Herzog, G. and Wazinski, P., VIsual TRAnslator: Linking perceptions and natural language descriptions. Artificial Intelligence Rev. v8. 175-187.

[30]

D. Isla and B. Blumberg, Object persistence for synthetic creatures, in: Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, 2002

Digital Library

[31]

Jackendoff, R., Foundations of Language. 2002. Oxford University Press, Oxford.

[32]

Johnson-Laird, P.N., Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. 1983. Cambridge University Press, Cambridge.

[33]

J. Juster, D. Roy, Elvis: Situated speech and gesture understanding for a robotic chandilier, in: Proceedings of the International Conference on Multimodal Interfaces, 2004

[34]

Kant, I., Critique of Pure Reason. 1781. Cambridge University Press, Cambridge.

[35]

Kuipers, B., The spatial semantic hierarchy. Artificial Intelligence. v119. 191-233.

[36]

Lakoff, G. and Johonson, M., Metaphors We Live By. 1980. University of Chicago Press, Chicago.

[37]

Lenat, D., Cyc: A large-scale investment in knowledge infrastructure. Comm. ACM. v38 i11. 33-38.

[38]

Levesque, H.J., Making believers out of computers. Artificial Intelligence. v30 i1. 81-108.

[39]

Littman, M., Sutton, R. and Singh, S., Predictive representations of state. In: Advances in Neural Information Processing Systems, MIT Press, Cambridge, MA.

[40]

Miller, G., Wordnet: A lexical database for English. Comm. ACM. v38 i11. 39-41.

[41]

Miller, G. and Johnson-Laird, P., Language and Perception. 1976. Harvard University Press, Harvard.

[42]

Millikan, R.G., Varieties of Meaning. 2004. MIT Press, Cambridge, MA.

[43]

Minsky, M., A framework for representing knowledge. In: Winston, P. (Ed.), The Psychology of Computer Vision, McGraw-Hill, New York. pp. 211-277.

[44]

S. Narayanan, Moving right along: A computational model of metaphoric reasoning about events, in: Proceedings of the National Conference on Artificial Intelligence AAAI-99, Orlando, FL, 1999

Digital Library

[45]

Neisser, U., Cognition and Reality. 1976. Freeman, New York.

[46]

Nelson, K., Concept, word, and sentence: Interrelations in acquisition and development. Psychological Rev. v81. 267-285.

[47]

Odgen, C.K. and Richards, I.A., The Meaning of Meaning. 1923. Harcourt.

[48]

Peirce, C.S., How to make our ideas clear. Popular Science Monthly. v12. 286-302.

[49]

Peirce, C.S., Logic as semiotic: The theory of signs. In: Philosophical Writings of Peirce, Dover.

[50]

Piaget, J., The Construction of Reality in the Child. 1954. Ballentine.

[51]

Plunkett, K., Sinha, C., Moller, M.F. and Strandsby, O., Symbol grounding or the emergence of symbols? vocabulary growth in children and a connectionist net. Connection Sci. v4 i3--4. 293-312.

[52]

K. Hsiao, N. Mavridis, D. Roy, Coupling perception and simulation: Steps towards conversational robotics, in: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, 2003

[53]

McDermott, D., Artificial intelligence meets natural stupidity. In: Haugeland, J. (Ed.), Mind Design, MIT Press, Cambridge, MA. pp. 143-160.

[54]

Putnam, H., The meaning of 'meaning'. In: Philosophical Papers: vol. 2. Mind, Language and Reality, Cambridge University Press, Cambridge. pp. 215-271.

[55]

Randell, D.A., Cui, Z. and Cohn, A.G., A spatial logic based on regions and connection. In: Proceedings 3rd International Conference on Knowledge Representation and Reasoning, Morgan Kaufmann, San Mateo, CA. pp. 165-176.

[56]

Regier, T., The Human Semantic Potential. 1996. MIT Press, Cambridge, MA.

[57]

Regier, T. and Carlson, L., Grounding spatial language in perception: An empirical and computational investigation. J. Experimental Psychol. v130 i2. 273-298.

[58]

Reiter, E. and Sripada, S., Human variation and lexical choice. Computational Linguistics. v22. 545-553.

[59]

Rosenblueth, A., Wiener, N. and Bigelow, J., Behavior, purpose and teleology. Philos. Sci. v10. 18-24.

[60]

D. Roy, Learning words from sights and sounds: A computational model, PhD thesis, Massachusetts Institute of Technology, 1999

Digital Library

[61]

Roy, D., Learning visually-grounded words and syntax for a scene description task. Computer Speech and Language. v16 i3. 353-385.

[62]

Roy, D., Grounding words in perception and action: Computational insights. Trends Cognitive Sci. v9 i8.

[63]

D. Roy, P. Gorniak, N. Mukherjee, J. Juster, A trainable spoken language understanding system for visual object selection, in: International Conference of Spoken Language Processing, 2002

[64]

Roy, D., Hsiao, K. and Mavridis, N., Mental imagery for a conversational robot. IEEE Trans. Systems Man Cybernet. B. v34 i3. 1374-1383.

[65]

Roy, D. and Mukherjee, N., Towards situated speech understanding: Visual context priming of language models. Computer Speech and Language. v19 i2. 227-248.

[66]

Roy, D. and Pentland, A., Learning words from sights and sounds: A computational model. Cognitive Sci. v26 i1. 113-146.

[67]

Rumelhart, D. and Ortony, A., The representation of knowledge in memory. In: Anderson, R.C., Spiro, R.J., Montague, W.E. (Eds.), Schooling and the Acquisition of Knowledge, Erlbaum, Hillsdale, NJ. pp. 99-136.

[68]

Schank, R. and Abelson, R., Scripts, Plans, Goals and Understanding. 1977. Erlbaum, Hilsdale, NJ.

[69]

J. Siskind, Naive physics, event perception, lexical semantics, and language acquisition, PhD thesis, Massachusetts Institute of Technology, 1992

[70]

Siskind, J., Grounding the lexical semantics of verbs in visual perception using force dynamics and event logic. J. Artificial Intelligence Res. v15. 31-90.

[71]

A. Sloman, J. Chappell, The altricial-precocial spectrum for robots, in: Proceedings of IJCAI, 2005

[72]

Smith, B.C., On the Origin of Objects. 1996. MIT Press, Cambridge, MA.

[73]

Spivey, M., Tyler, M., Richardson, D. and Young, E., Eye movements during comprehension of spoken scene descriptions. In: Proceedings of the 22nd Annual Conference of the Cognitive Science Society, Erlbaum, Mahwah, NJ. pp. 487-492.

[74]

Steels, L. and Vogt, P., Grounding adaptive language games in robotic agents. In: Husbands, C., Harvey, I. (Eds.), Proceedings of the 4th European Conference on Artificial Life, MIT Press, Cambridge, MA.

[75]

Stein, B.E. and Meredith, M.A., The Merging of the Senses. 1993. MIT Press, Cambridge, MA.

[76]

Winograd, T., A Process Model of Language Understanding. 1973. Freeman, New York.

[77]

Winograd, T. and Flores, F., Understanding Computers and Cognition. 1986. Addison-Wesley, Reading, MA.

[78]

Wittgenstein, L., The Blue and Brown Books. 1958. Basil Blackwell, Oxford.

Cited By

Friston KRamstead MKiefer ATschantz ABuckley CAlbarracin MPitliya RHeins CKlein BMillidge BSakthivadivel DSt Clere Smithe TKoudahl MTremblay SPetersen CFung KFox JSwanson SMapes DRené G(2024)Designing ecosystems of intelligence from first principlesCollective Intelligence10.1177/263391372312224813:1Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1177/26339137231222481
Man XOuyang DLi XSong JShao J(2022)Scenario-Aware Recurrent Transformer for Goal-Directed Video CaptioningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/350392718:4(1-17)Online publication date: 4-Mar-2022
https://dl.acm.org/doi/10.1145/3503927
Osipov GPanov A(2018)Relationships and Operations in a Sign-Based World Model of the ActorScientific and Technical Information Processing10.3103/S014768821805004045:5(317-330)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.3103/S0147688218050040
Show More Cited By

Index Terms

Semiotic schemas: A framework for grounding language in action and perception
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Natural language generation

Recommendations

The nature of dialog: structural and lexical markers of dialogic teacher/learner interactions
WASA '12: Proceedings of the Workshop at SIGGRAPH Asia

In this paper I argue that dialog facilitates learning. As a consequence teaching and learning should be dialogic in principal. Against this background the question arises what dialog actually is and how it can be implemented for teaching and learning. ...
Gumzo: New Dialogues for HCI
AfriCHI '16: Proceedings of the First African Conference on Human Computer Interaction

Gumzo at AfriCHI'16 aims to advance HCI by engaging design practice and scholarship with diverse voices and local or broader creativity, challenging dominant perspectives and raising critical issues about media, representation, technology, society and ...
Enactive artificial intelligence: Investigating the systemic organization of life and mind

The embodied and situated approach to artificial intelligence (AI) has matured and become a viable alternative to traditional computationalist approaches with respect to the practical goal of building artificial agents, which can behave in a robust and ...

Comments

Information & Contributors

Information

Published In

cover image Artificial Intelligence

Artificial Intelligence Volume 167, Issue 1-2

Special volume on connecting language to the world

September 2005

248 pages

ISSN:0004-3702

Issue’s Table of Contents

Copyright © © 2005.

Publisher

Elsevier Science Publishers Ltd.

United Kingdom

Publication History

Published: 01 September 2005

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

39
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Friston KRamstead MKiefer ATschantz ABuckley CAlbarracin MPitliya RHeins CKlein BMillidge BSakthivadivel DSt Clere Smithe TKoudahl MTremblay SPetersen CFung KFox JSwanson SMapes DRené G(2024)Designing ecosystems of intelligence from first principlesCollective Intelligence10.1177/263391372312224813:1Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1177/26339137231222481
Man XOuyang DLi XSong JShao J(2022)Scenario-Aware Recurrent Transformer for Goal-Directed Video CaptioningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/350392718:4(1-17)Online publication date: 4-Mar-2022
https://dl.acm.org/doi/10.1145/3503927
Osipov GPanov A(2018)Relationships and Operations in a Sign-Based World Model of the ActorScientific and Technical Information Processing10.3103/S014768821805004045:5(317-330)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.3103/S0147688218050040
Rasheed NAmin S(2016)Developmental and Evolutionary Lexicon Acquisition in Cognitive Agents/Robots with Grounding PrincipleComputational Intelligence and Neuroscience10.1155/2016/85712652016(16)Online publication date: 1-Mar-2016
https://dl.acm.org/doi/10.1155/2016/8571265
Catenacci Volpi NQuinton JPezzulo G(2014)How active perception and attractor dynamics shape perceptual categorizationNeural Networks10.1016/j.neunet.2014.06.00860:C(1-16)Online publication date: 1-Dec-2014
https://dl.acm.org/doi/10.1016/j.neunet.2014.06.008
Schulz RWyeth GWiles J(2012)Beyond here-and-nowAdaptive Behavior - Animals, Animats, Software Agents, Robots, Adaptive Systems10.1177/105971231244954620:5(360-387)Online publication date: 1-Oct-2012
https://dl.acm.org/doi/10.1177/1059712312449546
Dobnik SCooper RLarsson S(2012)Modelling Language, Action, and Perception in Type Theory with RecordsRevised Selected Papers of the 7th International Workshop on Constraint Solving and Language Processing - Volume 811410.1007/978-3-642-41578-4_5(70-91)Online publication date: 13-Sep-2012
https://dl.acm.org/doi/10.1007/978-3-642-41578-4_5
Reckman HOrkin JRoy DBos JPulman S(2011)Extracting aspects of determiner meaning from dialogue in a virtual world environmentProceedings of the Ninth International Conference on Computational Semantics10.5555/2002669.2002695(245-254)Online publication date: 12-Jan-2011
https://dl.acm.org/doi/10.5555/2002669.2002695
Tellex SKollar TDickerson SWalter MBanerjee ATeller SRoy N(2011)Approaching the Symbol Grounding Problem with Probabilistic Graphical ModelsAI Magazine10.1609/aimag.v32i4.238432:4(64-76)Online publication date: 1-Dec-2011
https://dl.acm.org/doi/10.1609/aimag.v32i4.2384
Spranger MLoetzsch MPauw S(2010)Open-ended Grounded SemanticsProceedings of the 2010 conference on ECAI 2010: 19th European Conference on Artificial Intelligence10.5555/1860967.1861149(929-934)Online publication date: 4-Aug-2010
https://dl.acm.org/doi/10.5555/1860967.1861149
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents