This dissertation is a study of metaphor in usage: metaphor in language, metaphor in gesture, and how they interact. Gesture provides a route to study both the cognition associated with language and the domain-generality of cognitive processes. While English speakers may be producing metaphoric manner verbs due to the lexicalization patterns of their language, are they necessarily thinking in terms of metaphoric manner? This is difficult to judge when looking at language alone. To answer this question, we turn to metaphoric gesture. Metaphoric gestures, in which the gesture represents the source domain of a conceptual metaphor, are well-known but under-studied (Cienki and Müller, 2008b). Iconic gestures conveying information about a motion event are known to interact with the syntactic and semantic structure of speech; speakers of languages that express manner of motion in the verb gesture differently than speakers of languages that primarily express path of motion in the verb. Metaphoric usages of motion in language – prices falling, hopes rising, time flying – also interact with the grammatical patterns of language. However, we know little about how metaphoric motion in gesture interacts with grammar. In Part One of the dissertation, I focus on metaphor in language. In Chapter 2 I propose to represent metaphors as a complex network of frames, mappings, and bindings as implemented in the MetaNet Metaphor Repository (Dodge et al., 2015). This advances the representation of conceptual metaphors to a level that interfaces more accurately with representations of frames and constructions in FrameNet (Ruppenhofer et al., 2006) and Embodied Construction Grammar (Bergen and Chang, 2005). In turn, this enables the detailed analysis of metaphors and metaphor systems, as exemplified by the Location Event Structure Metaphor (Lakoff and Johnson, 1999) case study in Chapter 3. This corpus-based study is one of the first to make use of the MetaNet method for large-scale automatic metaphor identification and annotation. This approach reveals not only how the metaphor system is evoked in language, but further illustrates the conceptual structure of the metaphor. I demonstrate that although English, as a satellite-framed language, privileges manner in its lexicalization of motion events, metaphoric English motion backgrounds manner and foregrounds path. The foregrounding of path information in linguistic realization of metaphoric motion runs counter to the privileging of manner in English lexicalization patterns. This finding lays the groundwork for the investigation of the same metaphor system in gesture. In Part Two, I focus on metaphor in co-speech gesture. I investigate metaphoric motion-evoking metaphoric gestures using two complementary approaches. Chapter 4 uses a corpus approach; I analyze a parallel corpus of video gesture data in which speakers use a motion verb either literally or metaphorically in their speech while producing a co-expressive representational gesture. To analyze the corpora, I develop a set of annotation guidelines and then demonstrate the benefits of taking an image-schematic approach to gesture analysis. I argue that the image schema is the most appropriate level of structure in analyzing the form and meaning of metaphoric gestures. Results of this image schema analysis suggest that, reflecting the English language data in Chapter 3, these metaphoric gestures emphasize path and do not represent the manner of motion. Chapter 5 is the first study to take an experimental approach to metaphoric gesture that uses non-metaphoric stimuli. Participants were given short stories about state change, such as prices decreasing or grades improving, to read and re-tell to a friend; half of the stimuli contained metaphoric language and half did not. Results from this study demonstrated the viability of this methodology in eliciting both metaphoric speech and gesture, and supported those of Chapter 4. I find that speakers are more likely produce metaphoric gestures if they are also producing metaphoric language – even if the gesture evokes a different metaphor than the speech does. I unify my analyses of metaphoric motion in speech and gesture in a multi-modal Embodied Construction Grammar analysis of both co-expressive and complementary metaphoric co-speech gestures. I represent both the meaning and form of the gesture and the meaning and form of the speech including frame structure, argument structure, and metaphoric structure. This analysis provides the first formal representation of a multi-modal utterance in a construction grammar and an innovative approach to the unification of the construction of multi-modal meaning.

ABSTRACT Gestures are related to the mental states and unfolding pro-cesses of thought, reasoning and verbal language production. This is especially apparent in the case of metaphors and metaphoric gestures. For example, talking about the importance of an idea by calling it a big idea and gesturing to indicate that large size is a manifestation of the use of metaphors in language and gesture. We propose a computational model of the influence of conceptual metaphors on gestures that maps from mental state representations of ideas to their expression in concrete, physical metaphoric gestures. This model relies on conceptual primary metaphors to map the abstract elements of the mental space to concrete physical elements that can be conveyed with gestures.

Humans turn abstract referents and discourse structures into gesture using metaphors. The semantic relation between abstract communicative intentions and their physical realization in gesture is a question that has not been fully addressed. Our hypothesis is that a limited set of primary metaphors and image schemas underlies a wide range of gestures. Our analysis of a video corpus supports this view: over 90% of the gestures in the corpus are structured by image schemas via a limited set of primary metaphors. This analysis informs the extension of a computational model that grounds various communicative intentions to a physical, embodied context, using those primary metaphors and image schemas. This model is used to generate gesture performances for virtual characters.

ABSTRACT: Although the study of metaphor has encompassed a wide array of subject matter over the last 20 years or so, traditional approaches continue to view metaphor as largely a property of language or language behavior. According to this view, metaphor is construed as a similarity between two single-place predicates, a topic and a vehicle. For instance, in the poetic metaphor "Love is a red, red rose," red rose is the vehicle, love is the topic, and the ground is the similarity between the two that the reader infers in some manner. Exactly how the reader goes about inferring the similarity remains something of a mystery; when pressed for an explanation, the reader invokes the usual circularity--he or she fits a schema to an input and comes up with a match. Given the range of experience implicated in the production and comprehension of metaphor--perception, cross-modal association, and emotional expression, to name a few--one could argue that the ability to perceive similarity is a more generalized facility not limited to language--it is a supramodular capacity. That is to say, the brain's capacity to process metaphorical relations may rely on central cognitive mechanisms that relate percepts or concepts across different cognitive domains or interrelationships among brain areas.

Recent research on the neural integration of speech and gesture has examined either gesture in the context of concrete [iconic (IC) gestures] or abstract sentence content [metaphoric (MP) gestures]. However, there has not yet been a direct comparison of the processing of both gesture types. This study tested the theory that left posterior temporal and inferior frontal brain regions are each uniquely involved in the integration of IC and MP gestures. During fMRI-data acquisition, participants were shown videos of an actor performing IC and MP gestures and associated sentences. An isolated gesture (G) and isolated sentence condition (S) were included to separate unimodal from bimodal effects at the neural level. During IC conditions, we found increased activity in the left posterior middle temporal gyrus and its right hemispheric homologue. The same regions in addition to the left inferior frontal gyrus (IFG) were activated during MP conditions in contrast to the isolated conditions (G&S). These findings support the hypothesis that there are distinct integration processes for IC and MP gestures. In line with recent claims of the semantic unification theory, there seems to be a division between perceptual-matching processes within the posterior temporal lobe and higher-order relational processes within the IFG. Hum Brain Mapp, 2011. © 2010 Wiley-Liss, Inc.

This paper proposes a synthesis of ideas from relevance theory’s conceptual and propositional approach to metaphoric interpretation with assumptions about the role of motor imagery from embodied cognition. Relevance theory’s account of metaphoric interpretation describes how conceptual and propositional representations are accessed during interpretation. The hearer of a metaphor inferentially develops the encoded concept to an occasion-specific ad-hoc concept, which resembles the speaker’s thought more closely. Under this approach, motor imagery is not intrinsic to speaker meaning since it is not part of the concept and able to undergo inference. Motor imagery is the capacity to simulate motor actions to plan for action. Within embodied cognition, for example, the word <run> accesses its conceptual counterpart, which shares a resemblance to previous activated motor imagery of observing or running oneself. Novel metaphors using action verbs activate motor imagery more than familiar ones as the latter accesses a new meaning, thereby weakening its motor imagery connection. Therefore, to reconcile this data with relevance theory, this paper proposes three solutions. Motor imagery may: (1) act as input to the inferential process, but is not part of speaker meaning; (2) generate other thoughts which undergo inference; (3) or be part of conceptual meaning and undergo inferential operations, which are intrinsic to metaphoric meaning. Whilst (1) and (2) are appealing, option (3) finds support from recent experimental evidence.