Irene Mittelberg

In light of the growing importance of digital design tools, this paper puts back into focus the human body’s internalized knowledge of the material and social world. It exploits the semiotic potential of operative manufacts, understood as encompassing both sketches on paper and gestures evolving in real time and space. Operative manufacts are seen as a means to tap into architects’ embodied experience with spatial structures and to enhance their imaginative capacities during the early stages of the design process. Combining theories and methods from the domains of art, architecture, semiotics, and gesture studies, this research examines how gestures and whole-body enactments may produce, elaborate, and communicate emerging ideas of space and artifacts, and how designers interact with(in) them.

The approach presented in this paper aims to contribute to an account of the three-dimensionality of gesture space. Here, gesture space is assumed to be dynamically constructed and adaptive to the communicative situation. Making use of an optical motion-capture system, volumetric representations of gesture spaces were generated, based on gesture data from semistructured interviews with four participants. The data were coded according to gesture phases and the gestures’ communicative functions. We compare speakers’ gesture rates and spatial distribution of gestures, both of which vary strongly across speakers.

This paper presents an account of how Peirce’s Universal Categories (UCs) of perception and experience may, as heuristic principles, inform gesture theory and multimodal analysis. Peirce’s UCs – Firstness (possibility), Secondness (actuality), and Thirdness (law, habit) – constitute the core of his phenomenology and thus also the foundation of his triadic semiotics. I argue that compared to the basic sign-object relations icon, index, symbol mainly used in previous gesture research, the more fundamental UCs allow one to discern additional facets of how coverbal gestures act as signs. This notably pertains to the phenomenology, multidimensionality, and multifunctionality of gesture. The guiding assumption is that compared to Thirdness-laden linguistic symbols constituting written, spoken or signed discourses, gestures may exhibit the UCs to more strongly varying degrees and in different, modality-specific ways. The multimodal analyses discussed in the paper show how Firstness tends t...

Taking an Emergent Grammar (Hopper 1998) approach to multimodal usage events in face-to-face interaction, this paper suggests that basic scenes of experience tend to motivate entrenched patterns in both language and gesture (Fillmore 1977; Goldberg 1998; Langacker 1987). Manual actions and interactions with the material and social world, such as giving or holding, have been shown to serve as substrate for prototypical ditransitive and transitive constructions in language (Goldberg 1995). It is proposed here that they may also underpin multimodal instantiations of existential construsctions in German discourse, namely, instances of the es gibt ‘it gives’ (there is/are) construction (Newman 1998) that co-occur with schematic gestural enactments of giving or holding something. Analyses show that gestural existential markers tend to combine referential and pragmatic functions. They exhibit a muted degree of indexicality, pointing to the existence of absent or abstract discourse contents...

Cognitive bias in depression may increase sensitivity to judgmental appraisal of communicative cues. Nonverbal communication encompassing co-speech gestures is crucial for social functioning and is perceived differentially by men and women, however, little is known about the effect of depression on the perception of appraisal. We investigate if a cognitive bias influences the perception of appraisal and judgement of nonverbal communication in major depressive disorder (MDD). During watching videos of speakers retelling a story and gesticulating, 22 patients with MDD and 22 matched healthy controls pressed a button when they perceived the speaker as appraising in a positive or negative way. The speakers were presented in four different conditions (with and without speech and with natural speaker or as stick-figures) to evaluate context effects. Inter-subject covariance (ISC) of the button-press time series measured consistency across the groups of the response pattern depending on the factors diagnosis and gender. Significant effects emerged for the factors diagnosis (p = .002), gender (p = .007), and their interaction (p < .001). The female healthy controls perceived the gestures more consistently appraising than male controls, the female patients, and male patients whereas the latter three groups did not differ. Further, the ISC measure for consistency correlated negatively with depression severity. The natural speaker video without audio speech yielded the highest responses consistency. Indeed co-speech gestures may drive these ISC effects because number of gestures but not facial shrugs correlated with ISC amplitude. During co-speech gestures, a cognitive bias led to disturbed perception of appraisal in MDD for females. Social communication is critical for functional outcomes in mental disorders; thus perception of gestural communication is important in rehabilitation.

Face-to-face communication is multimodal; it encompasses spoken words, facial expressions, gaze, and co-speech gestures. In contrast to linguistic symbols (e.g., spoken words or signs in sign language) relying on mostly explicit conventions, gestures vary in their degree of conventionality. Bodily signs may have a general accepted or conventionalized meaning (e.g., a head shake) or less so (e.g., self-grooming). We hypothesized that subjective perception of conventionality in co-speech gestures relies on the classical language network, i.e., the left hemispheric inferior frontal gyrus (IFG, Broca's area) and the posterior superior temporal gyrus (pSTG, Wernicke's area) and studied 36 subjects watching video-recorded story retellings during a behavioral and an functional magnetic resonance imaging (fMRI) experiment. It is well documented that neural correlates of such naturalistic videos emerge as intersubject covariance (ISC) in fMRI even without involving a stimulus (model-...

Viewpoint has been shown to be a powerful construal mechanism in multimodal spoken and signed discourse, as well as in various other modalities and genres. This paper investigates embodied viewpoint strategies that have been observed when speakers combine speech, gestures, postures, gaze, and simulated action to describe their interaction with spatial artifacts such as gallery buildings, virtual architectural models, and paintings. Simulated artifact immersion is introduced as a multimodal viewpoint strategy whereby speakers submerge into their mental representation of an artifact by perceiving and experiencing it from an internal vantage point. It is argued that this viewpoint strategy tends to be employed when there is no narrative structure for the speakers to fall back on. The paper’s aim is twofold: (a) to show that when speakers talk about their own experiences with spatial artifacts, distinguishing between immersed and non-immersed experiential viewpoint strategies may be mor...

Based on spoken American academic discourse and its accompanying gestures videotaped during linguistics courses, this paper presents a cognitive-semiotic approach to multimodal communication that assigns equal importance to metaphor and metonymy. Combining traditional semiotics (Peirce [1960] and Jakobson [1956, 1961, 1963]) with contemporary cognitivist theories (Gibbs 1994, 2006; Johnson 1987, 2005; Lakoff 1987, 1993; Lakoff & Johnson 1980, 1999; Sweetser 1998), we demonstrate how these two figures of thought jointly structure multimodal representations of grammatical concepts and structures. Drawing on Jakobson's (1956) view of metaphor and metonymy, and especially his distinction between internal metonymy (synecdoche) and external metonymy (adjacency, contact, contexture, cf. Jakobson & Pomorska 1983), we first demonstrate some of the contiguity relations we were able to identify within metaphoric gestures (whether or not the concurrent speech is metaphorical). Moving from t...

This study represents an interdisciplinary attempt to trace the origins of metaphorical expressions as they are found in English grammatical terminology. The perspective chosen combines cognitive and art-historical approaches to the meaning of abstract concepts: cognitive semantics and iconography. It takes into account the historical dimension of the media which have served, over the course of centuries, to render abstract concepts graspable: figurative language, visual images, and the printed page. By taking a close look at examples from an iconography of grammar created for the purpose of this study (which includes personifications, allegories and memory buildings from the early Modern period), I shall discuss the visual traditions and patterns in the representation of abstract concepts and structures. In the second part of this paper, the theory of conceptual metaphor is applied to one of these educational and mnemonic illustrations, whereby the metaphorical concepts underlying ...

The question of how to model similarity between gestures plays an important role in current studies in the domain of human communication. Most research into recurrent patterns in co-verbal gestures – manual communicative movements emerging spontaneously during conversation – is driven by qualitative analyses relying on observational comparisons between gestures. Due to the fact that these kinds of gestures are not bound to well-formedness conditions, however, we propose a quantitative approach consisting of a distance-based similarity model for gestures recorded and represented in motion capture data streams. To this end, we model gestures by flexible feature representations, namely gesture signatures, which are then compared via signature-based distance functions such as the Earth Mover's Distance and the Signature Quadratic Form Distance. Experiments on real conversational motion capture data evidence the appropriateness of the proposed approaches in terms of their accuracy an...

This short paper provides a synopsis of a recently proposed body-centered, dynamic systems approach that aims to account for how embodied image schemas and force dynamics jointly operate in – and also guide the interpretation of – gestural (inter-)action. The proposed tendencies are illustrated with both video and motion-capture (MoCap) data. It shows how numeric kinetic data allow gesture researchers to measure the spatiotemporal dimensions of gestural articulations and render movement traces visible. MoCap may thus offer new insights into the dynamic, gestalt-like nature of gesturally enacted schematicity.

This study represents an interdisciplinary attempt to trace the origins of metaphorical expressions as they are found in English grammatical terminology. The perspective chosen combines cognitive and art-historical approaches to the meaning of abstract concepts: cognitive semantics ...

Based on a comparative semiotic analysis of static abstract works of art and dynamic gestural images, this paper explores the role of embodied conceptual structures as driving forces in the constitution and interpretation of visual signs. One of the underlying ...

This paper lays out the foundations of a frame-based account of gesture pragmatics through detailing how frames and metonymy interact not only in motivating gestural sign formation, but also in guiding crossmodal processes of pragmatic inferencing. It is argued that gestures recruiting frame structures tend to profile deeply embodied, routinized aspects of scenes (in the Fillmorian sense of the term), that is, of the motivating context of frames. Two kinds of embodied frame structures situated at different levels of abstraction, schematicity, and entrechment are proposed: (A) Basic physical action and object frames understood as directly experientially grounded frames involving physical action and interaction with the material and social world; (B) Complex, highly abstract frame structures that are more detached from the motivating contexts of experience. It is further suggested that gestures exhibiting a comparably low degree of iconicity and/or indexicality are likely to assume pr...

In language switching, it is assumed that in order to produce a response in one language, the other language must be inhibited. In unimodal (spoken-spoken) language switching, the fact that the languages share the same primary output channel (the mouth) means that only one language can be produced at a time. In bimodal (spoken-signed) language switching, however, it is possible to produce both languages simultaneously. In our study, we examined modality effects in language switching using multilingual subjects (speaking German, English, and German Sign Language). Focusing on German vocal responses, since they are directly compatible across conditions, we found shorter reaction times, lower error rates, and smaller switch costs in bimodal vs. unimodal switching. This result suggests that there are different inhibitory mechanisms at work in unimodal and bimodal language switching. We propose that lexical inhibition is involved in unimodal switching, whereas output channel inhibition i...

One of the most fundamental challenges when accessing gestural patterns in 3D motion capture databases is the definition of spatiotemporal similarity. While distance-based similarity models such as the Gesture Matching Distance on gesture signatures are able to leverage the spatial and temporal characteristics of gestural patterns, their applicability to large 3D motion capture databases is limited due to their high computational complexity. To this end, we present a lower bound approximation of the Gesture Matching Distance that can be utilized in an optimal multi-step query processing architecture in order to support efficient query processing. We investigate the performance in terms of accuracy and efficiency based on 3D motion capture databases and show that our approach is able to achieve an increase in efficiency of more than one order of magnitude with a negligible loss in accuracy. In addition, we discuss different applications in the digital humanities in order to highlight...