Philosophical Transactions of the Royal Society B, Mar 6, 2023
It is increasingly important for technical systems to be able to interact flexibly, robustly and ... more It is increasingly important for technical systems to be able to interact flexibly, robustly and fluently with humans in real-world scenarios. However, while current AI systems excel at narrow task competencies, they lack crucial interaction abilities for the adaptive and co-constructed social interactions that humans engage in. We argue that a possible avenue to tackle the corresponding computational modelling challenges is to embrace interactive theories of social understanding in humans. We propose the notion of socially enactive cognitive systems that do not rely solely on abstract and (quasi-)complete internal models for separate social perception, reasoning and action. By contrast, socially enactive cognitive agents are supposed to enable a close interlinking of the enactive socio-cognitive processing loops within each agent, and the social-communicative loop between them. We discuss theoretical foundations of this view, identify principles and requirements for according computational approaches, and highlight three examples of our own research that showcase the interaction abilities achievable in this way. This article is part of a discussion meeting issue ‘Face2face: advancing the science of social interaction’.
Aksu V, Bergmann K, Kopp S. Touch-less or Touch-full Interaction – User-Centered-Design for an In... more Aksu V, Bergmann K, Kopp S. Touch-less or Touch-full Interaction – User-Centered-Design for an Industrial Application. In: Eyssel F, Kluge A, Kopp S, et al., eds. Proceedings 5. Interdisziplinärer Workshop Kognitive Systeme: Mensch, Teams, Systeme und Automaten. 2016
Vogt P, de Wit J, Schodde T, et al. Adaptation and gestures in second language tutoring using soc... more Vogt P, de Wit J, Schodde T, et al. Adaptation and gestures in second language tutoring using social robots. Presented at the Workshop on early literacy and (digital) media, Paderborn, Germany
Thórisson K, Viljalmsson H, Pelachaud C, et al. Representations for Multimodal Generation. AI Mag... more Thórisson K, Viljalmsson H, Pelachaud C, et al. Representations for Multimodal Generation. AI Magazin 27(1); 2006
Zatrib R, Loth S, Kopp S, Bergmann K. The Role of Iconic Representation Techniques in Gesture-enr... more Zatrib R, Loth S, Kopp S, Bergmann K. The Role of Iconic Representation Techniques in Gesture-enriched Vocabulary Learning with a Virtual Tutor. Presented at the Eight Conference of the International Society for Gesture Studies: Gesture and Diversity, Cape Town, South Africa
Kopp S, Bergmann K. Less time to speak leads to less speech-gesture redundancy. In: Proceedings o... more Kopp S, Bergmann K. Less time to speak leads to less speech-gesture redundancy. In: Proceedings of the 7th International Gesture Conference (ISGS 7). 2016: 214
Projekt IMPACT: ein von der VW-Stiftung gefordertes Forschungsprojekt mit dem Ziel, die Chancen u... more Projekt IMPACT: ein von der VW-Stiftung gefordertes Forschungsprojekt mit dem Ziel, die Chancen und Risiken kunstlicher Intelligenz innerhalb einer Lebensspanne zu untersuchen; vom Kind zum Senior.
Bergmann K, Kopp S. Time pressure effects on semantic speech-gesture coordination. In: Proceeding... more Bergmann K, Kopp S. Time pressure effects on semantic speech-gesture coordination. In: Proceedings of the 13th Biannual Conference of the German Cognitive Science Society (KogWis 2016). Accepted
e L2TOR project explores the use of social robots for second language tutoring. is paper presen... more e L2TOR project explores the use of social robots for second language tutoring. is paper presents an experiment in preparation to investigate the eects of two educational scaolding features (adaptation/personalization and iconic gestures), when used by a robot tutor, on children’s comprehension of animal names in a foreign language. Participants will be children between the ages of four and ve. e study is scheduled to take place in March 2017.
Speakers’ spontaneous gestures are traditionally thought of as a dependent system, their meaning ... more Speakers’ spontaneous gestures are traditionally thought of as a dependent system, their meaning relying heavily on what is expressed verbally. Nonetheless, studies using muted videos as stimuli consistently report that gestures have some degree of communicative import in the absence of speech. Here, we argue that the dependence-autonomy question can be advanced by adopting a functional (linguistic) perspective. We ask whether access to speech is necessary to understand what kind of semantic or pragmatic function a gesture performs. Based on a large-scale web-based perception study, we report that when the notion of meaning is operationalized on a functional level, the dependence of gesture on speech largely diminishes.
The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations - Volume 1, 2017
Multimodal behavior has been studied for a long time and in many fields, e.g., in psychology, lin... more Multimodal behavior has been studied for a long time and in many fields, e.g., in psychology, linguistics, communication studies, education, and ergonomics. One of the main motivations has been to allow humans to use technical systems intuitively, in a way that resembles and fosters human users' natural way of interacting and thinking [Oviatt 2013]. This has sparked early work on multimodal human-computer interfaces, including recent approaches to recognize communicative behavior and even subtle multimodal cues by computer systems. Those approaches, for the most part, rest on machine learning techniques applied to large sets of behavioral data. As datasets grow larger in size and coverage, and computational power increases, suitable data-driven techniques are able to detect correlational behavior patterns that support answering questions like which feature( s) to take into account or how to recognize them in specific contexts. However, natural multimodal interaction in humans entails a plethora of behavioral variations and intricacies (e.g., when to act unimodally vs. multimodally, with which specific behaviors or multi-level coordination between them). Possible underlying patterns are hard to detect, even in large datasets, and often such variations are attributed to context-dependencies or individual differences. How they come about is still hard to explain at the behavioral level. One additional level of explanation that can help to deepen our understanding, and to establish systematic and generalized accounts, involves cognitive processes that lead to a particular multimodal response in a specific situation (e.g., see Chapter 2; [James et al. 2017]). A prominent example is the concept of cognitive load or "mental workload." Many behavioral variations in spoken language have been meaningfully explained in terms of heightened or lowered cognitive load of the speaker. For example, under high cognitive load speakers are found to speak more slowly, to produce more silent or filled pauses, and to utter more repetitions [Jameson et al. 2010]. Likewise, human users distribute information across multiple modalities in order to manage their cognitive limits [Oviatt et al. 2004, Chen et al. 2012]. In studies with elementary school children as well as adults, active manual gesturing was demonstrated to improve memory during a task that required explaining math solutions [Goldin-Meadow et al. 2001]. This effect of gesturing increased with higher task difficulty. Such behavioral phenomena are commonly explained based on cognitive concepts like cognitive load and, further, underlying processes like modality-specific working memories [Baddeley 1992] or competition for cognitive resources [Wickens et al. 1983]. Cognitive theories also provide valuable hints as to how to design multimodal human-machine interaction. To continue with the above example, it has been suggested that certain multimodal interfaces help users to minimize their cognitive load and hence improve their performance. For instance, the physical activity of manual or pen-based gesturing can play a particularly important role in organizing and facilitating people's spatial information processing, which has been shown to reduce cognitive load on tasks involving geometry, maps, and similar areas [Alibali et al. 2000, Oviatt 1997]. Other research revealed that expressively powerful interfaces not only help to cope with interaction problems, but also substantially facilitate human cognition by functioning as "thinking tools" [Oviatt 2013]. It is for these reasons that "Cognitive Science has and will continue to play an essential role in guiding the design of multimodal systems" [Oviatt and Cohen 2015, p. 33]. One particularly explicit form of a cognitive account is a cognitive model. In general, a cognitive model is a simplified, schematic description of cognitive processes for the purposes of understanding or predicting a certain behavior. This approach is markedly pursued in the field of cognitive modeling, which developed at the intersection of cognitive psychology, cognitive science, and artificial intelligence. In these fields, cognitive models are primarily developed within generic cognitive architectures like ACT-R [Anderson et al. 2004] or SOAR [Laird 2012], to name two prominent examples, which capture generally assumed structural and functional properties of the human mind. Yet, the term cognitive model is not restricted to the use of such architectures. It can aptly be used whenever a specific notion of cognitive or mental processes is provided in computational terms that afford simulation-based examination and evaluation. This may include symbolic, hybrid, connectionist or dynamical models [cf. Polk and Seifert 2002], and it has been proposed for single cognitive tasks (e.g., memorizing of items), for the interaction of two or more processes (e.g., visual search and language comprehension), or for making specific behavioral predictions (e.g.,…
Philosophical Transactions of the Royal Society B, Mar 6, 2023
It is increasingly important for technical systems to be able to interact flexibly, robustly and ... more It is increasingly important for technical systems to be able to interact flexibly, robustly and fluently with humans in real-world scenarios. However, while current AI systems excel at narrow task competencies, they lack crucial interaction abilities for the adaptive and co-constructed social interactions that humans engage in. We argue that a possible avenue to tackle the corresponding computational modelling challenges is to embrace interactive theories of social understanding in humans. We propose the notion of socially enactive cognitive systems that do not rely solely on abstract and (quasi-)complete internal models for separate social perception, reasoning and action. By contrast, socially enactive cognitive agents are supposed to enable a close interlinking of the enactive socio-cognitive processing loops within each agent, and the social-communicative loop between them. We discuss theoretical foundations of this view, identify principles and requirements for according computational approaches, and highlight three examples of our own research that showcase the interaction abilities achievable in this way. This article is part of a discussion meeting issue ‘Face2face: advancing the science of social interaction’.
Aksu V, Bergmann K, Kopp S. Touch-less or Touch-full Interaction – User-Centered-Design for an In... more Aksu V, Bergmann K, Kopp S. Touch-less or Touch-full Interaction – User-Centered-Design for an Industrial Application. In: Eyssel F, Kluge A, Kopp S, et al., eds. Proceedings 5. Interdisziplinärer Workshop Kognitive Systeme: Mensch, Teams, Systeme und Automaten. 2016
Vogt P, de Wit J, Schodde T, et al. Adaptation and gestures in second language tutoring using soc... more Vogt P, de Wit J, Schodde T, et al. Adaptation and gestures in second language tutoring using social robots. Presented at the Workshop on early literacy and (digital) media, Paderborn, Germany
Thórisson K, Viljalmsson H, Pelachaud C, et al. Representations for Multimodal Generation. AI Mag... more Thórisson K, Viljalmsson H, Pelachaud C, et al. Representations for Multimodal Generation. AI Magazin 27(1); 2006
Zatrib R, Loth S, Kopp S, Bergmann K. The Role of Iconic Representation Techniques in Gesture-enr... more Zatrib R, Loth S, Kopp S, Bergmann K. The Role of Iconic Representation Techniques in Gesture-enriched Vocabulary Learning with a Virtual Tutor. Presented at the Eight Conference of the International Society for Gesture Studies: Gesture and Diversity, Cape Town, South Africa
Kopp S, Bergmann K. Less time to speak leads to less speech-gesture redundancy. In: Proceedings o... more Kopp S, Bergmann K. Less time to speak leads to less speech-gesture redundancy. In: Proceedings of the 7th International Gesture Conference (ISGS 7). 2016: 214
Projekt IMPACT: ein von der VW-Stiftung gefordertes Forschungsprojekt mit dem Ziel, die Chancen u... more Projekt IMPACT: ein von der VW-Stiftung gefordertes Forschungsprojekt mit dem Ziel, die Chancen und Risiken kunstlicher Intelligenz innerhalb einer Lebensspanne zu untersuchen; vom Kind zum Senior.
Bergmann K, Kopp S. Time pressure effects on semantic speech-gesture coordination. In: Proceeding... more Bergmann K, Kopp S. Time pressure effects on semantic speech-gesture coordination. In: Proceedings of the 13th Biannual Conference of the German Cognitive Science Society (KogWis 2016). Accepted
e L2TOR project explores the use of social robots for second language tutoring. is paper presen... more e L2TOR project explores the use of social robots for second language tutoring. is paper presents an experiment in preparation to investigate the eects of two educational scaolding features (adaptation/personalization and iconic gestures), when used by a robot tutor, on children’s comprehension of animal names in a foreign language. Participants will be children between the ages of four and ve. e study is scheduled to take place in March 2017.
Speakers’ spontaneous gestures are traditionally thought of as a dependent system, their meaning ... more Speakers’ spontaneous gestures are traditionally thought of as a dependent system, their meaning relying heavily on what is expressed verbally. Nonetheless, studies using muted videos as stimuli consistently report that gestures have some degree of communicative import in the absence of speech. Here, we argue that the dependence-autonomy question can be advanced by adopting a functional (linguistic) perspective. We ask whether access to speech is necessary to understand what kind of semantic or pragmatic function a gesture performs. Based on a large-scale web-based perception study, we report that when the notion of meaning is operationalized on a functional level, the dependence of gesture on speech largely diminishes.
The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations - Volume 1, 2017
Multimodal behavior has been studied for a long time and in many fields, e.g., in psychology, lin... more Multimodal behavior has been studied for a long time and in many fields, e.g., in psychology, linguistics, communication studies, education, and ergonomics. One of the main motivations has been to allow humans to use technical systems intuitively, in a way that resembles and fosters human users' natural way of interacting and thinking [Oviatt 2013]. This has sparked early work on multimodal human-computer interfaces, including recent approaches to recognize communicative behavior and even subtle multimodal cues by computer systems. Those approaches, for the most part, rest on machine learning techniques applied to large sets of behavioral data. As datasets grow larger in size and coverage, and computational power increases, suitable data-driven techniques are able to detect correlational behavior patterns that support answering questions like which feature( s) to take into account or how to recognize them in specific contexts. However, natural multimodal interaction in humans entails a plethora of behavioral variations and intricacies (e.g., when to act unimodally vs. multimodally, with which specific behaviors or multi-level coordination between them). Possible underlying patterns are hard to detect, even in large datasets, and often such variations are attributed to context-dependencies or individual differences. How they come about is still hard to explain at the behavioral level. One additional level of explanation that can help to deepen our understanding, and to establish systematic and generalized accounts, involves cognitive processes that lead to a particular multimodal response in a specific situation (e.g., see Chapter 2; [James et al. 2017]). A prominent example is the concept of cognitive load or "mental workload." Many behavioral variations in spoken language have been meaningfully explained in terms of heightened or lowered cognitive load of the speaker. For example, under high cognitive load speakers are found to speak more slowly, to produce more silent or filled pauses, and to utter more repetitions [Jameson et al. 2010]. Likewise, human users distribute information across multiple modalities in order to manage their cognitive limits [Oviatt et al. 2004, Chen et al. 2012]. In studies with elementary school children as well as adults, active manual gesturing was demonstrated to improve memory during a task that required explaining math solutions [Goldin-Meadow et al. 2001]. This effect of gesturing increased with higher task difficulty. Such behavioral phenomena are commonly explained based on cognitive concepts like cognitive load and, further, underlying processes like modality-specific working memories [Baddeley 1992] or competition for cognitive resources [Wickens et al. 1983]. Cognitive theories also provide valuable hints as to how to design multimodal human-machine interaction. To continue with the above example, it has been suggested that certain multimodal interfaces help users to minimize their cognitive load and hence improve their performance. For instance, the physical activity of manual or pen-based gesturing can play a particularly important role in organizing and facilitating people's spatial information processing, which has been shown to reduce cognitive load on tasks involving geometry, maps, and similar areas [Alibali et al. 2000, Oviatt 1997]. Other research revealed that expressively powerful interfaces not only help to cope with interaction problems, but also substantially facilitate human cognition by functioning as "thinking tools" [Oviatt 2013]. It is for these reasons that "Cognitive Science has and will continue to play an essential role in guiding the design of multimodal systems" [Oviatt and Cohen 2015, p. 33]. One particularly explicit form of a cognitive account is a cognitive model. In general, a cognitive model is a simplified, schematic description of cognitive processes for the purposes of understanding or predicting a certain behavior. This approach is markedly pursued in the field of cognitive modeling, which developed at the intersection of cognitive psychology, cognitive science, and artificial intelligence. In these fields, cognitive models are primarily developed within generic cognitive architectures like ACT-R [Anderson et al. 2004] or SOAR [Laird 2012], to name two prominent examples, which capture generally assumed structural and functional properties of the human mind. Yet, the term cognitive model is not restricted to the use of such architectures. It can aptly be used whenever a specific notion of cognitive or mental processes is provided in computational terms that afford simulation-based examination and evaluation. This may include symbolic, hybrid, connectionist or dynamical models [cf. Polk and Seifert 2002], and it has been proposed for single cognitive tasks (e.g., memorizing of items), for the interaction of two or more processes (e.g., visual search and language comprehension), or for making specific behavioral predictions (e.g.,…
Uploads