- Brain and cognitive sciences, Cognitive Linguistics, Cognitive linguistics (especially metaphor and metonymy, conceptualization of emotions and relationship between language, mind, and culture), Cognitive Neuroscience, Robotics, Natural Language Processing, and 2 morePhilosophy of Mind and Artificial Intelligence & AI and Discourse Analysis (DA)edit
This paper presents an fMRI study on healthy adult understanding of metaphors in multimodal communication. We investigated metaphors expressed either only in coverbal gestures ("monomodal metaphors") or in speech with accompanying... more
This paper presents an fMRI study on healthy adult understanding of metaphors in multimodal communication. We investigated metaphors expressed either only in coverbal gestures ("monomodal metaphors") or in speech with accompanying gestures ("multimodal metaphors"). Monomodal metaphoric gestures convey metaphoric information not expressed in the accompanying speech (e.g. saying the non-metaphoric utterance, "She felt bad" while dropping down the hand with palm facing up; here, the gesture alone indicates metaphoricity), whereas coverbal gestures in multimodal metaphors indicate metaphoricity redundant to the speech (e.g. saying the metaphoric utterance, "Her spirits fell" while dropping the hand with palm facing up). In other words, in monomodal metaphors, gestures add information not spoken, whereas the gestures in multimodal metaphors can be redundant to the spoken content. Understanding and integrating the information in each modality, here spoken and visual, is important in multimodal communication, but most prior studies have only considered multimodal metaphors where the gesture is redundant to what is spoken. Our participants watched audiovisual clips of an actor speaking while gesturing. We found that abstract metaphor comprehension recruited the lateral superior/middle temporal cortices, regardless of the modality in which the conceptual metaphor is expressed. These results suggest that abstract metaphors, regardless of modality, involve resources implicated in general semantic processing and are consistent with the role of these areas in supramodal semantic processing as well as the theory of embodied cognition.
Research Interests:
Animal studies show marked sex differences as well as effects of estrogen (E2) in the mesocorticolimbic dopaminergic (DA) pathways, which play a critical role in reward processing and reinforcement learning and are also implicated in drug... more
Animal studies show marked sex differences as well as effects of estrogen (E2) in the mesocorticolimbic dopaminergic (DA) pathways, which play a critical role in reward processing and reinforcement learning and are also implicated in drug addiction. In this computational pharmacological fMRI study, we investigate the effects of both factors, sex and estrogen, on reinforcement learning and the dopaminergic system in humans; 67 male and 64 naturally cycling female volunteers, the latter in their low-hormone phase, were randomly assigned, double-blind, to take E2 or placebo. They completed a reinforcement learning task in the MRI scanner for which we have previously shown reward prediction error (RPE)-related activity to be dopaminergic. We found RPE-related brain activity to be enhanced in women compared with men and to a greater extent when E2 levels were elevated in both sexes. However, both factors, female sex and E2, slowed adaptation to RPEs (smaller learning rate). This discrepancy of larger RPE-related activity yet smaller learning rates can be explained by organizational sex differences and activational effects of circulating E2, which both affect DA release differently to DA receptor binding capacities.
Research Interests:
Multiband echo planar imaging (EPI) offers increased temporospatial resolution and statistical power for functional magnetic resonance imaging (fMRI) but the higher spatial resolution comes at the cost of higher susceptibility-related... more
Multiband echo planar imaging (EPI) offers increased temporospatial resolution and statistical power for functional magnetic resonance imaging (fMRI) but the higher spatial resolution comes at the cost of higher susceptibility-related spatial distortions. In diffusion MRI (dMRI), studies have shown that distortion correction is better when using blip-reversed EPI data (known under the term blip-up/down images) as compared to the standard fieldmap approach. This has motivated fMRI studies to acquire their data with blip-up/down directions and to use these to reduce susceptibility distortion. Here, we qualitatively illustrate why this can lead to erroneous results and quantify this error across 10 subjects.
Research Interests:
Emotionally arousing stimuli are usually better remembered than neutral ones. This effect can be observed immediately after encoding and becomes more robust after a period of consolidation. The magnitude of this effect in an individual... more
Emotionally arousing stimuli are usually better remembered than neutral ones. This effect can be observed immediately after encoding and becomes more robust after a period of consolidation. The magnitude of this effect in an individual has been treated in various research contexts implicitly as reliable and temporally stable. However, we recently observed in 69 participants that an individual’s memory advantage for negative over neutral stimuli, whether immediate or delayed, was very weakly correlated with the advantage measured after 3.5 years, albeit with slightly different memory paradigms. In the current study, we tested whether the test-retest reliability of these emotional memory effects might be larger if the temporal lapse between tests was shorter (10 weeks) and more similar memory tests were used. We observed that the better memory for emotional stimuli is highly replicable on the group level. However, the retest reliability on the individual level was very low. We replicated these findings by re-analysing data from a previous study where female participants took emotional memory tests at two different points of their menstrual cycle. We conclude, therefore, that the individual emotional enhancement of memory is not stable or that it cannot be measured reliably with the standard emotional memory paradigm.
Research Interests:
This chapter starts from the observation that metaphoric understandings expressed monomodally through gesture tend to rely on "primary metaphors" (Grady 1997a). Asserting that gestures draw on basic, experientially motivated, embodied... more
This chapter starts from the observation that metaphoric understandings expressed monomodally through gesture tend to rely on "primary metaphors" (Grady 1997a). Asserting that gestures draw on basic, experientially motivated, embodied construal operations, we detail how primary scenes and subscenes (Grady & Johnson 2002), image and force schemas, metonymy, and frames (Fillmore 1982) interact in situated meaning-making. We propose that by shifting the focus from object-oriented schemas, source domains, and mappings to what we call "source actions" and "embodied action frames," we can account for the pragmatically minded nature and specific mediality of communicative gestural acts integrated in natural multimodal discourse. We argue that coverbal gestures recruit frame structures metonymically, singling out elements of "scenes" (Fillmore 1977), especially those underpinning correlated metaphoric meanings. We back up our theoretical claims with evidence from neuroscientific studies and outline a frame-based approach that helps trace avenues for further research into embodied cognition and multimodal discourse processes.
Research Interests:
In "Two heads are better than one," "head" stands for people and focuses the message on the intelligence of people. This is an example of figurative language through metonymy, where substituting a whole entity by one of its parts focuses... more
In "Two heads are better than one," "head" stands for people and focuses the message on the intelligence of people. This is an example of figurative language through metonymy, where substituting a whole entity by one of its parts focuses attention on a specific aspect of the entity. Whereas metaphors, another figurative language device, are substitutions based on similarity, metonymy involves substitutions based on associations. Both are figures of speech but are also expressed in coverbal gestures during multimodal communication. The closest neuropsychological studies of metonymy in gestures have been nonlinguistic tool-use, illustrated by the classic apraxic problem of body-part-as-object (BPO, equivalent to an internal metonymy representation of the tool) vs. pantomimed action (external metonymy representation of the absent object/tool). Combining these research domains with concepts in cognitive linguistic research on gestures, we conducted an fMRI study to investigate metonymy resolution in coverbal gestures. Given the greater difficulty in developmental and apraxia studies, perhaps explained by the more complex semantic inferencing involved for external metonymy than for internal metonymy representations, we hypothesized that external metonymy resolution requires greater processing demands and that the neural resources supporting metonymy resolution would modulate regions involved in semantic processing. We found that there are indeed greater activations for external than for internal metonymy resolution in the temporoparietal junction (TPJ). This area is posterior to the lateral temporal regions recruited by metaphor processing. Effective connectivity analysis confirmed our hypothesis that metonymy resolution modulates areas implicated in semantic processing. We interpret our results in an interdisciplinary view of what metonymy in action can reveal about abstract cognition.
Research Interests:
1 Advances in Functional Linguistics An approach to language emphasizing its dynamic interactions (not rule-gov-erned formalisms) brings us to the realization that interjections may function as sig-nals exchanged in the process of... more
1 Advances in Functional Linguistics An approach to language emphasizing its dynamic interactions (not rule-gov-erned formalisms) brings us to the realization that interjections may function as sig-nals exchanged in the process of negotiated understanding that is communication. 2. ...
Research Interests:
Research Interests:
Research Interests:
With the growing research in autonomous systems, the issue of embodiment has become a fundamental issue in artificial intelligence. However, the necessity of embodiment is a perspective that has been a focus in artificial intelligence... more
With the growing research in autonomous systems, the issue of embodiment has become a fundamental issue in artificial intelligence. However, the necessity of embodiment is a perspective that has been a focus in artificial intelligence only in the recent years. It remains an undeveloped concept and a term that has been thrown around in conflicting contexts. This paper discusses embodiment, its interpretations, misinterpretations and the role it has played in artificial intelligence to date and specifically in the realisation of the "intelligent autonomous robot". While some believe that simply placing a controller in a physical environment constitutes a sufficient degree of embodiment, we wish to emphasise that agent-world interaction must develop away from this "ON-World" approach and seek to concentrate on "IN-World" interaction, participation, and adaptation.
Research Interests:
This paper investigates the use of cognitive constraints in a computational linguistic model for speech recognition. The approach incorporates a cognitive-based phonology into a dynamic temporal framework with the aim of improving the... more
This paper investigates the use of cognitive constraints in a computational linguistic model for speech recognition. The approach incorporates a cognitive-based phonology into a dynamic temporal framework with the aim of improving the robustness of speech recognition performance by analysing speech as an embodied artefact of human social cognition. This paper sketches the necessary extensions to an existing computational linguistic model and discusses how a distributional analysis based on cognitively motivated features can provide the basis for the establishment of the cognitive constraints for the purposes of constraint ranking in such a model.
Research Interests:
How can the performance of a social robot be measured? Are existing metrics and techniques appropriate for physically and socially embodied autonomous mobile robots? With the rapid expansion of robust robotic systems from strong... more
How can the performance of a social robot be measured? Are existing metrics and techniques appropriate for physically and socially embodied autonomous mobile robots? With the rapid expansion of robust robotic systems from strong industrial applications to domestic and entertainment domains which naturally involve social interactions, a requirement to gauge a system's abilities either individually or relatively has emerged. This paper discusses the implications that arise in aiming to assess the performance of robots engaged in a degree of social interaction with people.
Research Interests:
This paper investigates the use of cognitive constraints in a computational linguistic model for speech recognition. The approach incorporates a cognitive-based phonology into a dynamic temporal framework with the aim of improving the... more
This paper investigates the use of cognitive constraints in a computational linguistic model for speech recognition. The approach incorporates a cognitive-based phonology into a dynamic temporal framework with the aim of improving the robustness of speech recognition performance by analysing speech as an embodied artefact of human social cognition. This paper sketches the necessary extensions to an existing computational linguistic model and discusses how a distributional analysis based on cognitively motivated features can provide the basis for the establishment of the cognitive constraints for the purposes of constraint ranking in such a model.
Research Interests:
The problems of conventional speech recognition approaches include incomplete linguistic knowledge and inability to deal with underspecification. These issues can be addressed by understanding the constraints of speech to predict speech... more
The problems of conventional speech recognition approaches include incomplete linguistic knowledge and inability to deal with underspecification. These issues can be addressed by understanding the constraints of speech to predict speech tendencies. We believe that understanding what constraints exist requires an embodied view of speech and that the traditional disembodied view of speech is the fundamental limitation on the robustness of many speech systems. We argue that viewing speech as a form of embodied cognition, or within context of its production and use, provides important insights in speech structure and speech recognition. In making this claim, this paper briefly outlines a strongly embodied account of cognition and develops from that an embodiment paradigm for speech recognition. The embodiment paradigm proposed leads to both an explanatory and descriptive account of linguistic structure. It simplifies the view of speech structure for automatic speech recognisers, by cons...
Research Interests:
This paper presents a testbed for developing multilingual phonotactic descriptions that employs finite state methods to represent the phonotactics of one or more languages. The motivation for this work is to make an extensive range of... more
This paper presents a testbed for developing multilingual phonotactic descriptions that employs finite state methods to represent the phonotactics of one or more languages. The motivation for this work is to make an extensive range of phonotactic descriptions of varying granularity available for speech technology applications. We discuss the design of the phonotactic testbed and how various modules may be used to generate finite state phonotactic descriptions. We provide an example multilingual application drawn from a partial sample of onset clusters spanning four language families, demonstrating how the commonalities of a broad spectrum of languages can be expressed using individual and generic phonotactic automata. We then discuss how these representations are extended via a three-tiered model to provide the basis for the feature- and event-based phonotactic automata.
Research Interests:
There is the age-old paradox of technologists predicting bleak futures for mankind because of their research directions but nevertheless hurtling full steam ahead in pursuing them. Strong humanoid research could be one such example. The... more
There is the age-old paradox of technologists predicting bleak futures for mankind because of their research directions but nevertheless hurtling full steam ahead in pursuing them. Strong humanoid research could be one such example. The holy grail of robotics and artificial intelligence research is regularly perceived as aiming to understand mankind by artificially creating a similar degree of functionality and
Research Interests:
Research Interests:
... In C. Forceville and E. Urios-Aparisi (eds.), Multimodal Metaphor. Mouton de Gruyter, Berlin (2009), 329-358. 13. ... O'Reilly Media, 2008. 16. Wobbrock, JO, Morris, MR and Wilson, AD User-defined gestures for surface... more
... In C. Forceville and E. Urios-Aparisi (eds.), Multimodal Metaphor. Mouton de Gruyter, Berlin (2009), 329-358. 13. ... O'Reilly Media, 2008. 16. Wobbrock, JO, Morris, MR and Wilson, AD User-defined gestures for surface computing. Proc. of the 27th Int. Conf. ...
Research Interests:
With the growing research in autonomous systems, the issue of embodiment has become a fundamental issue in artificial intelligence. However, the necessity of embodiment is a perspective that has been a focus in artificial intelligence... more
With the growing research in autonomous systems, the issue of embodiment has become a fundamental issue in artificial intelligence. However, the necessity of embodiment is a perspective that has been a focus in artificial intelligence only in the recent years. It remains an undeveloped concept and a term that has been thrown around in conflicting contexts. This paper discusses embodiment, its interpretations, misinterpretations and the role it has played in artificial intelligence to date and specifically in the realisation of the "intelligent autonomous robot". While some believe that simply placing a controller in a physical environment constitutes a sufficient degree of embodiment, we wish to emphasise that agent-world interaction must develop away from this "ON-World" approach and seek to concentrate on "IN-World" interaction, participation, and adaptation.
Research Interests:
Abstract: - How can the performance of a social robot be measured? Are existing metrics and techniques appropriate for physically and socially embodied autonomous mobile robots? With the rapid expansion of robust robotic systems from... more
Abstract: - How can the performance of a social robot be measured? Are existing metrics and techniques appropriate for physically and socially embodied autonomous mobile robots? With the rapid expansion of robust robotic systems from strong industrial applications to ...
Research Interests:
This paper investigates the use of cognitive constraints in a computational linguistic model for speech recognition. The approach incorporates a cognitive-based phonology into a dynamic temporal framework with the aim of improving the... more
This paper investigates the use of cognitive constraints in a computational linguistic model for speech recognition. The approach incorporates a cognitive-based phonology into a dynamic temporal framework with the aim of improving the robustness of speech recognition ...
Research Interests:
Embodiment has been perceived by some as one of the fundamental issues in the pursuit of artificial intelligence, a perspective that has only been mainstream in recent years. This paper discusses embodiment, its interpretations,... more
Embodiment has been perceived by some as one of the fundamental issues in the pursuit of artificial intelligence, a perspective that has only been mainstream in recent years. This paper discusses embodiment, its interpretations, misinterpretations and the role it ...
Research Interests:
Maintaining high levels of performance involves concentration, alertness and monitoring, processes which have been attributed to lateral and medial prefrontal regions including the dorsal anterior cingulate cortex (dACC). Much research... more
Maintaining high levels of performance involves concentration, alertness and monitoring, processes which have been attributed to lateral and medial prefrontal regions including the dorsal anterior cingulate cortex (dACC). Much research has focused on the role of the dACC in reacting to situations of high conflict or error likelihood, with more recent research implicating its role in improved subsequent performance. The present study investigated the functional neuroanatomy responsible for this proactive maintainance of superior performance. Specifically, we show that dACC activity on Go/NoGo tasks was (a) more active prior to successful performance relative to commission errors, and (b) was negatively correlated with levels of intra-individual performance variability. These results demonstrate that the dACC's role in cognitive control extends beyond error-or conflict-related, within-trial processes consistent with an important role in proactively maintaining performance levels.