270 Massimo Leone Sign Systems Studies 49(3/4), 2021, 270–278 Introduction: Studying the ‘facesphere’ Massimo Leone1 1. A controversial object of study Early comparative studies in academic scholarship on the meaning of the face show a dichotomy between, on the one hand, physical anthropology and anthropometry and, on the other hand, cultural anthropology and ethnology. The former tends to privilege static measurements and determinism, the latter, dynamic observation and contextual interpretation. The former often yields to racial prejudice and even systematic racism, as well as to unethical research practices, whereas the latter rejects such views and methods and emphasizes the role of sociocultural context in the development of face types, their expressions, and the relative meanings, which, however, sometimes results in ideologically and aprioristically denying any impact of human biology on the meaning of the face. As regards the positivistic trend, Goldstein 1936 (an article on the age-related changes in the dimensions and form of the face) surveys the previous literature, starting from early anthropometric comparative works such as Weckler 1866, Hrdlicka 1900, Boas 1911, Hellman 1927, 1935, Connolly 1928, up to Huber 1931. Other early comparative works along this line – also often influenced by racist prejudice – adopt an embryological perspective (for instance, Schultz 1920, Schaeffer 1935), or an interspecies comparative approach (for instance, Krogman 1930, 1931a, 1931b, 1931c). This tradition of comparative anthropometry with focus on face growing continues with Loth still displaying blatant racist prejudice in the late 1940s: “It is very hard to study facial muscles on the living subject, and is possible only with intelligent people with whom one can communicate to evoke the desired facial expression…I…gave up this study in Uganda” (Loth 1949: 222). In the following decades, overt racism disappears from literature on facial comparisons, but determinism persists under other guises, for instance, in linking the morphology of faces and facial expressions to climate conditions (Steegman 1 University of Turin, Department of Philosophy and Education Studies, Via S. Ottavio 20, 10124 – Turin, Italy; massimo.leone@unito.it. https://doi.org/10.12697/SSS.2021.49.3-4.01 Introduction: Studying the ‘facesphere’ 271 1965, 1967, 1970, 1972; Steegman, Plainer 1968; see also Coon 1965 on “the living races of man”). From the late 1960s on, with a few marginal exceptions, the universalist approach to the meaning of faces and their expressions moves from anthropology to psychology (especially experimental, empirical, and, later on, cognitive psychology) as well as to comparative ethology. The idea lingers that some facial significations are shaped not by sociocultural influence but by biological conditions, where the notion of race remains central (the trend continues today; see, for instance, Watson and De Gelder 2017, claiming to have experimentally found that “race influences affective body perception”). The opposite trend, tending to stress cultural variability in types of faces and facial expressions, has an early representative in Edward Burnett Tylor (1871: 66) and its most influential herald in Franz Boas (“motor habits of groups of people are culturally determined and [are] not due to heredity”; Boas 1938: 126). His work then influenced the founder of kinesics, Ray Birdwhistell (1970: 38–39), as well as Otto Klineberg (1940), David Efron (1941), Edward Sapir (1949), Gregory Bateson and Margaret Mead (1942), Weston LaBarre (1947), and others. Early semiotics developing from the mid-1960s onwards disregards the former, biologically deterministic trend, tends to criticize it, and rather embraces the latter, culturalist tendency, in line with the new discipline’s approach to meaning as an exclusive product of social conditions. That is, for instance, the perspective adopted by Umberto Eco from the mid-1970s on about the essentially cultural origin of indexicality, including the facial one (Eco 1975). Whereas the racist biases of the first trend of scholarship have been generally recognized, and are now righteously widely stigmatized, the symmetrical biases developed by the opposite trend as a result of the legitimate desire to counter the first have been neglected. From Franz Boas on, cultural anthropology and then semiotics have praiseworthily rejected the racist prejudices of early anthropometry and physical anthropology, yet they have also tended to dogmatize the role of sociocultural contexts in giving rise to the meaning of the face. More fundamentally, the second trend neglected to investigate the deep connection between the cultural variability of facial signification and its biological roots. 2. Beyond an ideological dichotomy The present-day cultural semiotics of the face is therefore urged to adopt a novel perspective, which rejects racial determinism and its racist overtones, but also refrains from any easy dogmatic explanation of the “cultural origin of face meaning”. Early physical anthropology tended to explain the meaning of the face in 272 Massimo Leone purely biological terms; early cultural anthropology and semiotics, to the contrary, were inclined to do it in purely sociocultural terms. Contemporary semioticians should reverse the perspective of the first trend without, however, severing culture from its biological origins, which must be investigated not in terms of individual faces but in those of facial clusters and ranges. ‘Cultures of the face’, indeed, means that faces are not only biologically common and not only perceptually singular but that they can be gathered in clusters which are independent from natural determination and, at same time, transcend individuality and can be subsumed into types. These, however, are not somatic but based on usages of the face that rely on language. ‘Cultures of the face’, from a semiotic point of view, therefore implies that this part of the body can express its own language, and that this language is also spoken by other faces. Following this argument, faces, the non-verbal languages they speak, and the cultures to which they give rise, can also be an object of translation. This does not mean that an Italian face can be translated into a Chinese one but, more subtly, that semiotic usages of the face in a society that predominantly speaks Italian can – and also, in some circumstances, must – be translated into their equivalent – if any – in Chinese-speaking societies and cultures. Culture is not only the non-genetic memory of humankind but also what can be translated of humankind, at least theoretically. The possibility of this translation implies that there is a language composed by two planes and a certain articulation of both, and that the expression plane can be rendered in a different way within a different system, while retaining (almost) the same content. The translatability of the face, and therefore its cultural nature, is evident especially in its representations. Paintings, sculptures, and other artifacts representing the face are not actually the body part itself, but just a figment of it. Yet facial representations are as they are not only due to stylistic reasons, but also because they refer to distinct cultures of the somatic face. Moreover, they retroact on actual faces, shaping and influencing their cultures. Intuitively, faces are neither only biologically common nor only ontologically singular; on the contrary, they fall into categories that result not as much from what faces are as from what they do, or what they are put to use to do. Contemporary visitors of present-day Lithuania coming from other European countries, and especially from the Mediterranean, commonly report as anecdotal evidence that “Lithuanians do not smile”. Lithuanians themselves take it as an element of their facial identity, make jokes on it, and propose various explanations, most of which are quite far-fetched. The degree to which societies tend to smile affects representations as well. Several inquiries focusing on big data of selfies shared on social networks seem to indicate that there is a relation between geo-localization Introduction: Studying the ‘facesphere’ 273 and the degree of smiling. Some cities of the world happen to produce more smiling selfies than others (see Lev Manovich’s www.selfiecity.net). Proposed reasons for such discrepancies often reproduce trivial stereotypes. In fact, explaining why a society smiles more than another is usually accounted for in historical or genealogical terms (Lithuanians lived through Soviet repression; they were struck by famine) or in causal terms (they live in cold climates). A structural explanation, however, might be more convincing: in simpler terms, Lithuanians smile less because other societies smile more, and smiling less is a way the face is put to use in order to outline a somatic, emotional, postural, and gestural frontier. In the end, face dialectics opposing societies and their cultures might structurally work like the famous head gestures for affirmation and negation studied by Roman Jakobson (1972): explaining them in causal terms is misleading; what matters is the opposition, not the opposed terms. They are a cultural shibboleth, yet this does not rule out that the human species might be characterized by a biological inclination towards the production of such structural differentiations. The particular facial terms in the opposition are probably socio-culturally determined, yet the cultural mechanism of semiotic opposition itself between facial clusters might be biologically shaped. 3. Patterns of face signification Cultures of the face might work exactly along these lines. All the facial elements and features that can be modified may give rise to oppositional patterns – that is, a great number of them may. Either as discrete elements (presence/absence) or as nuanced ones (through levels, gradients, and thresholds), they contribute to sketching the frontier of a facial community. Discrete features usually mark the frontiers of a ‘facesphere’ more starkly. This is often the case with semiotic arrangements that specifically modify the degree of visibility/invisibility of the face, especially if it is done drastically. Female faces covered by strongly obtrusive Islamic garments such as niqabs or burqas emphatically connote an urban area where this attire is predominant. Similarly, visible face inscriptions like pigments, tattoos, piercings, scarifications, etc. also decisively delineate the confines of a ‘face culture’. These socio-cultural facial patterns, however, are easier to capture than other subtler dialectics. Those that concern the dynamic appearance of the face (and not only its permanent Gestalt), are harder to grasp. Frowning, for instance, is a facial configuration that varies from culture to culture, not only in its expressive composition (depending on which part of the face is predominantly used to “frown”) but also in its content (Russell 1994). That shows the fractal nature of 274 Massimo Leone the face in the facesphere: facial cultures are outlined through frontiers between different ‘facial ways’, yet they too arise from analogous oppositions within the face itself. These internal dialectics also often work in oppositional terms. What matters in frowning, for instance, is neither the specific face part that is used in this expression nor its specific content, but the expressive opposition between a “smooth” face and a “striated” one (in Deleuze’s terms), which semi-symbolically conveys the semantic opposition between something that the face finds pleasant and something that, on the contrary, it finds regrettable. In this case too, the corrugated face should not be explained causally – for instance, as a way to withdraw the face from an unpleasant sight – but rather structurally, through opposition with smoothness. Differences in terms of how cultures not only present their faces but also represent them are even more striking. Their variety can be appreciated both synchronically and diachronically, sometimes with remarkable gaps. Cultural forms of face representations indeed evolve faster than social ways of face presentation, which in turn are faster than changes in the biology of the face. The gap between biology and society is due to language, the one between society and culture is due to media technology. The face is a medium too, yet its representation through advanced media technology accelerates the pace and scope of cultural change. Modalities of representing the face through photographs have undergone a fast evolution, to the point that it is not hard to date immediately a photograph of a face that was taken before WWII or to situate it in one of the following decades. At the present time, the technical evolution of the modalities of face representation is so fast that it is even possible to indicate the year in which a face photograph was taken, judging by the particular technique of the representation. The latest model of iPhone to date, for instance, allows owners to take photograph portraits that characteristically focus automatically on the face, while leaving the background visible but blurred. This digital format is intended to bestow a special aura on the face, thus coating previous formats of face representation, like the selfie, with a new, quasiartistic status. In the past, indeed, portrait photographs of this kind were usually the product of accomplished photographers. As has been the trend in the last decades, smartphones tend to miniaturize, automatize, and make portable formats of face representation that were hitherto more exclusive. The selfie popularized the self-portrait, which was theretofore difficult to perform without adequate photographic technology; format filters do the same at a different level; the new portrait function in iPhones, along the same line, enables everyone to execute an “artistic” photographic portrait or self-portrait. Introduction: Studying the ‘facesphere’ 275 The massive diffusion of a new technological format leads to the rapid creation of novel trends of face representation, which soon tend to become standards. The process is further accelerated by digital social networks: the innovative format of the “artistic” portrait is being increasingly adopted to represent faces in profile pictures or in other forms of digital self-representation, giving rise to a bidirectional reinforcing dynamic: on the one hand, people use it in order to attribute a special aura to the picture of their face, gaining prestige by implicitly showing that they can access the latest and currently most expensive technology; on the other hand, the popularity of the new format makes it trendy: more and more individuals adopt it in order to represent their face in a photographic portrait, implicitly underling the aesthetic and emotional importance of the picture. Through these face photographs that focus on the foreground and blur the background, social network users indirectly communicate the distinction of such face representations in relation to other face representations that do not adopt such a format. The exclusivity and supposed prestige of the new technology is indeed always mitigated by the immediate circulation of imitations through cheaper technology and brands. The creation of an innovative technological format of face representation results from the creativity of engineers and technicians but does not appear ex nihilo. The idea of blurring the background and to focus on the foreground so as to underline the importance of the face is old in photography, and does not probably originate in this technique and its art. This strategy of representation could be retraced back to J. M. W. Turner’s pictorial techniques of blurring and, more remotely, to Leonardo’s experiments with subtly nuancing the background of his paintings towards infinitely nuanced landscapes. Indeed, the new iPhone 12 does nothing else but reinterpreting an innate visual cognition modality, its connate phenomenology, and its inherent semiotics through the new technological possibility of miniaturization and automatization, including artificial intelligence for face recognition. The visual cognition of focusing and blurring is engrained in the neurophysiology of sight: humans spontaneously focus on objects that are closer to the eyes, since these are a more likely source of either pleasure or danger. This cognitive pattern, however, turns into a tendency to bestow importance to an object that is in focus, especially if it is surrounded by an unfocused background. The mechanism is quite simple: since we focus on what is important, what we focus on tends to become important. Artists, especially in contemporary arts, have been playing with this automatism of cognitive semiotics, often with ironic intents and results. Applying such dialectics of focusing and blurring onto the particular object of the face, though, inheres to a peculiar sociocultural trend, that is, the emerging of the singularity of the individual in modernity. The portrait function of iPhone12, therefore, technically originates from the astuteness of Apple engineers, 276 Massimo Leone yet culturally evolves from the Renaissance, and translates a centuries-old aesthetic trend – whose roots are actually in the neurophysiology of human sight and visual cognition – into new technical ways. The invention and diffusion of a new device, enabling an innovative technical format of face representation, revamps an already existing trend and contributes to its diffusion and consolidation. Ultimately, with the multiplication of instances of face photographs adopting such a format, these give rise to a culture of face representation that can be distinguished both spatially and chronologically from other cohorts of facial images that do not adopt the same format. 4. Studying the ‘facesphere’ Studying the ‘cultures of the face’ therefore means investigating how biological mechanisms, sociocultural conditions, fashion trends, as well as individual choices continuously structure the ‘facesphere’ of a society, that is, the way in which, in a human community, faces both individually and collectively are used as language to signify an identity that emerges holistically and is in contrast with other ‘facespheres’. In order to understand how this particular configuration of the semiosphere works, measurements are not sufficient; indeed, quantitative research on faces that disregards their qualitative observation leads to exceeding determinism, often with the dangerous biases that have been exposed above; yet qualitative observation of faces and face representations without measurements also neglects to take into account an essential dynamic of the semiosphere, that is, the emergence of new qualitative trends from quantitative accumulation, for instance the concretion of a new facial pattern of signification through the phenomenon of contagion and virality in the semiosphere. Acknowledgements: This reseach is part of a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No 819649 – FACETS). References Bateson, Gregory; Mead, Margaret 1942. Balinese Character. New York: New York Academy of Sciences. Birdwhistell, Ray Louis 1970. Kinesics and Context. Philadelphia: University of Pennsylvania Press. https://doi.org/10.9783/9780812201284 Boas, Franz 1911. Changes in Bodily Form of Descendants of Immigrants. Washington: Government Printing Office. Introduction: Studying the ‘facesphere’ 277 Boas, Franz 1938. The Mind of Primitive Man. New York: Macmillan. Connolly, C. J. 1928 Growth of face in different races. American Journal of Physical Anthropology 12: 197–206. https://doi.org/10.1002/ajpa.1330120112 Coon, Carleton S. 1965. The Living Races of Man. New York: Alfred A. Knopf. Eco, Umberto 1975. Trattato di semiotica generale. Milan: Bompiani. Efron, David 1941. Gesture and Environment. New York: King’s Crown Press. Goldstein, Marcus S. 1936. Changes in dimensions and form of the face and head age. American Journal of Physical Anthropology 22: 37–89. https://doi.org/10.1002/ajpa. 1330220104 Hellman, Milo 1927. Changes in the human face brought about by development. International Journal of Orthodontics, Oral Surgery and Radiography 20: 475. https://doi. org/10.1016/S0099-6963(27)90222-1 Hellman, Milo 1935. The face in its developmental career. The Human Face: A Symposium. Philadelphia County Dental Society Philadelphia: The Dental Cosmos; also in The Dental Cosmos – A Monthly Record of Dental Science 77(7): 685–699. Hrdlicka, Ales 1900. Anthropological Investigations of One Thousand White and Colored Children of both Sexes, the Inmates of the New York Juvenile Asylum. With Additional Notes on One Hundred Colored Children of the New York Colored Orphan Asylum. New York, Albany: Wynkoop, Hallenbeck, Crawford Co. Huber, Ernst 1931. Evolution of Facial Musculature and Facial Expression. Baltimore: Johns Hopkins University Press. Jakobson, Roman 1972. Motor Signs for ‘Yes’ and ‘No’. Language in Society 1(1): 91–96. https://doi.org/10.1017/S0047404500006564 Klineberg, Otto 1940. Social Psychology. New York: Henry Holt. https://doi.org/10.1037/ 13603-000 Krogman, Wilton Marion 1930. The problem of growth changes in the face and skull as viewed from a comparative study of anthropoids and man. The Dental Cosmos – A Monthly Record of Dental Science (June), 624–630. Krogman, Wilton Marion 1931a. Growth changes in the skull and face of the Gorilla. American Journal of Anthropology 47: 89–115. https://doi.org/10.1002/aja.1000470104 Krogman, Wilton Marion 1931b. Growth changes in the skull and face of the Chimpanzee. American Journal of Anthropology 47: 325–342. https://doi.org/10.1002/aja.1000470205 Krogman, Wilton Marion 1931c. Growth changes in the skull and face of the Orang-Utan. American Journal of Anthropology 47: 343–365. https://doi.org/10.1002/aja.1000470206 LaBarre, Weston 1947. The cultural basis of emotions and gestures. Journal of Personality 16: 49–68. https://doi.org/10.1111/j.1467-6494.1947.tb01075.x Loth, Edward 1949. Anthropological studies of muscles of living Uganda Negroes. Yearbook of Physical Anthropology 5: 220–236. Russell, James A. 1994. Is there universal recognition of emotion from facial expressions? A review of the cross-cultural studies. Psychological Bulletin 115(1): 102–141. https:// doi.org/10.1037/0033-2909.115.1.102 Sapir, Edward 1949. Selected Writings of Edward Sapir. (Mandelbaum, David G., ed.). Berkeley: University of California Press. Schaeffer, Jacob Parsons 1935. The ontogenetic development of the human face. In: The Human Face: A Symposium. Philadelphia County Dental Society. Philadelphia: The 278 Massimo Leone Dental Cosmos; also in The Dental Cosmos – A Monthly Record of Dental Science 77(5): 464–484. Schultz, A. H. 1920. The development of the external nose in whites and negroes. Contributions to Embryology 34: 173–190. Steegmann, Albert Theodore, Jr. 1965. A study of relationships between facial cold responses and some variables of facial morphology. American Journal of Physical Anthropology 23: 355–62. https://doi.org/10.1002/ajpa.1330230409 Steegmann, Albert Theodore, Jr. 1967. Frost bite of the human face as a selective force. Human Biology 39: 131–44. Steegmann, Albert Theodore, Jr. 1970. Cold adaptation and the human face. American Journal of Physical Anthropology 32: 243. https://doi.org/10.1002/ajpa.1330320212 Steegmann, Albert Theodore, Jr. 1972. Cold response, body form and craniofacial shape in two racial groups in Hawaii. American Journal of Physical Anthropology 37: 193–221. https://doi.org/10.1002/ajpa.1330370205 Steegmann, Albert Theodore, Jr.; Platner, W. S. 1968. Experimental cold modification of craniofacial morphology. American Journal of Physical Anthropology 28: 17–30. https:// doi.org/10.1002/ajpa.1330280111 Tylor, Edward Burnett 1871. Primitive Culture. London: John Murray. [Republished in 1958 as The Origin of Culture. New York: Harper and Row.] Watson, Rebecca; De Gelder, Beatrice 2017. How white and black bodies are perceived depends on what emotion is expressed. Scientific Reports 7: 41349. https://doi. org/10.1038/srep41349 Weckler, Hermann 1866. Kraniologische Mittheilungen. Archiv für Anthropologie 1: 89–160. The transcendence ofSign the face: A semiotic-linguistic path279–297 279 Systems Studies 49(3/4), 2021, The transcendence of the face: A semiotic-linguistic path Ugo Volli1 Abstract. This paper starts with an examination of the terms used to designate the face in different languages, in particular in Italian, comparing these with the definitions provided by some authoritative dictionaries as well as with their etymology. This exploration yields some remarkable results: firstly, it appears that the face is indeed a term that has a material meaning, but at the same time it is a social object; secondly, the importance of the communicative function emerges, which makes the face similar to the mask and in some ways to the arbitrariness of language. All this suggests that the philosophical status of the face is that of ‘transcendence’ which is a condition of that state of freedom that we attribute to ourselves and that can be defined as ‘human exception’. Keywords: face; lexicon; social objects; metaphor; Levinas; transcendence Introduction Semiotics describes texts and analyses their conditions of use and the effects of meaning they produce. This has obvious social utility, because it allows us to classify and study communication occurring in a certain society. Used in a more radical way, however, semiotic analysis allows us to study the social rules of signification and thus to understand the mindframes prevalent in different societies, because all communication in every culture is formulated according to syntactic, semantic and pragmatic categories characteristic of that society. These categories always involve a degree of arbitrariness, even in the extreme case in which communication consists of simple material objects exhibited to indicate themselves, as demonstrated by the inhabitants of Laputa in Gulliver’s Travels. These objects are in fact assumed as signifiers, common names or prototypes of the set of objects to which they belong in that culture. These prototypes, in turn, depend on ontology or implied metaphysics (Quine 1960). Actually, everything that 1 Department of Philosophy and Education Science, University of Turin, Torino, Italy; e-mail: volli.ugo@gmail.com. https://doi.org/10.12697/SSS.2021.49.3-4.02 280 Ugo Volli works as a sign – every message – is always brought to communication through the mediation of the “primary modelling system” of its society, namely of language (Lotman 1967). Language determines and limits the very possibility of signification, even the silent one. A fundamental condition for semiotic research is leaving behind of the naive ethnocentric naturalization of social objects and of posing the problem of their origin and their transformations in relation with language, by studying texts, but first and foremost by questioning the lexicon. For this reason, it is indispensable that semiotics should include a philosophicallinguistic approach in its typological studies, at least if we define philosophy of language not as analytic philosophy usually does (Beaney 2013: 13), but as a type of research that, operating from within a language, attempts to describe and understand, through the use of the language itself, the fundamental semantic choices that are arbitrarily or historically made by its culture. So, in order to understand what a certain society means and communicates through “social things” such as ‘love’, ‘religion’, ‘money’ (Ferraris 2009; Searle 1995), but also through “natural things” that are socially semanticized and communicatively defined such as ‘child’, ‘dog’, ‘sea’, it is necessary to probe the words that describe and model them. They work, actually, by cutting their object (often a set of objects unified in a social category just by that word) out of world continuity; interrogating words means trying to understand the cultural or empirical logic underlying this clipping. This is certainly also true for such an entity as the ‘face’ which, as part of the human body, is certainly material – yet it is also highly socially defined, as we will see soon. To try to unravel the paradoxes and aporias that surround this socio-material object, it is relevant to study the word and its use, which is done in what follows mainly on the basis of the Italian language with some references to other languages. An ill-defined entity Let us start from a paradox. Lexemes corresponding to our ‘face’ are found in numerous languages: linguistic literature does not mention examples of a lack of relevant terms in any language. From a naive point of view, this word is very easy to understand, everyone knows it in all contemporary major cultures – but it is difficult to define exactly in precise conceptual terms. It is worth starting with the definitions of the word offered in some dictionaries: OED: “The front part of a person’s head from the forehead to the chin, or the corresponding part in an animal.” Merriam Webster: “The front part of the head that in humans extends from the forehead to the chin and includes the mouth, nose, cheeks, and eyes.” The transcendence of the face: A semiotic-linguistic path 281 Cambridge: “The front of the head, where the eyes, nose, and mouth are.” Garzanti: “La parte anteriore della testa umana, dalla fronte al mento.” Larousse: “Région limitée en haut par le cuir chevelu, sur les côtés par les oreilles et en bas par le cou.” Duden: “Besonders durch Augen, Nase und Mund geprägte Vorderseite des menschlichen Kopfes vom Kinn bis zum Haaransatz.”2 All these definitions, and the many other very similar ones that are found in other languages, have a certain oddity to them. The face is defined as part of the head or in some cases of the human body, characterized by the presence of certain organs, such as the nose, eyes and mouth, and bounded by other physical units such as the chin, forehead and ears. However, these smaller objects are also part of the face; only in rare cases does the definition use external details such as the neck and the hairline. The strangeness is due to the fact that if you look for the definitions of all these parts or boundaries that define ‘face’, you are sent back to the face: a vicious circle emerges. To give just a couple of examples, according to the Cambridge dictionary ‘forehead’ is “the flat part of the face, above the eyes and below the hair”; for Merriam Webster, ‘nose’ is “the part of the face that bears the nostrils and covers the anterior part of the nasal cavity”, and so on.3 In conclusion, ‘face’ is defined through its parts, and its parts through the face. This is certainly an example of a very widespread problem of dictionaries, which in turn essentially depends on the labyrinthine format of the semantics of natural languages and probably of human cognition: it is a well-known problem in semiotics, widely discussed at least since the observations made in the chapter “Dictionary vs. encyclopedia” in Eco 1983. Formal semantics, cognitive sciences, artificial intelligence studies are also well aware of this. Yet in this case the circularity is not extended, and thus vague, as in the regular rhizomatic structure of lexical semantics, but appears immediately at the first step and we are led to think that it depends on a difficulty concerning the specific object. Actually, the face is neither a distinct nor homogeneous anatomical reality: nose, eyes, lips, forehead have different origins in the development of the embryo and are made up of different tissues under the skin. The colours, the textures, the shapes of these details vary to a considerable degree. The upper and lateral limits of the face 2 The definitions have been obtained from the following sources, last accessed on 22 February 2021: Oxford: https://www.lexico.com/definition/face; Merriam Webster: https:// www.merriam-webster.com/dictionary/face; Garzanti: https://www.garzantilinguistica.it/ ricerca/?q=faccia%201; Larousse: https://www.larousse.fr/dictionnaires/francais/face/32561?q= Face#32473; Duden: https://www.duden.de/rechtschreibung/Gesicht_Aussehen_Miene_Sinn. La 3 Cambridge: https://dictionary.cambridge.org/dictionary/english/forehead; Merriam Webster: https://www.merriam-webster.com/dictionary/nose. Accessed on 22 February 2021. 282 Ugo Volli are variable, because they depend on the way the hair is arranged. The different parts of the face do not even have a common aspect. The face cannot be defined, for example, as the whole hairless part of the head, because in many adult males conspicuous parts of it would be naturally hairy due to cultural conventions. It is worth introducing here a complex anatomical description proposed by the most authoritative Italian encyclopedia: Part of the man’s head located inferior to the front of the skull. It is continuous with the base of the skull and is supported by a complex bony scaffold, consisting of the upper jaw, the zygomatic arch, the lacrimal, the lower turbinates, the nasal bone, the palatine bones and the lower jaw. These bones delimit cavities in which the peripheral organs of the three senses (sight, smell, taste) and the initial tracts of the respiratory and digestive systems are located. The f. is equipped with masticatory muscles and numerous cutaneous muscles, which regulate the opening of the various orifices present (e.g. lips, nostrils), move the eyelids and, by modifying the skin folds, cause changes in expression. A single nerve, the facial nerve, innervates all the muscles, while the sensitivity at the level of the skin is mainly due to the branches of the trigeminal nerve. Vascularization is largely ensured by the facial artery.4 In short, the face is not really a scientific object or a natural kind identified as such by science (as are chemical elements such as iron, or animal species such as the lynx, or organs such as the thyroid) (Daly 1998; Bird, Tobin 2008). It is a cultural clipping of a real biological entity, the head, and, therefore, somewhat puzzling as a material social object which, although it has a universal presence in human populations, can also be extended to the animal world according to some authoritative voices (Darwin 2009[1872]). Here, one cannot fail to notice an important semantic tension between universality and arbitrariness. 4 “Parte della testa dell’uomo situata inferiormente alla parte anteriore del cranio. Si trova in rapporto di continuità con la base del cranio ed è sostenuta da una complessa impalcatura ossea, costituita dall’osso mascellare superiore, dallo zigomatico, dal lacrimale, dal turbinato inferiore, dal nasale, dal palatino e dal mascellare inferiore. Queste ossa delimitano cavità nelle quali hanno sede gli organi periferici di tre sensi (vista, olfatto, gusto) e i tratti iniziali degli apparati respiratorio e digerente. La f. è dotata di muscoli masticatori e di numerosi muscoli pellicciai, che regolano l’apertura dei vari orifizi presenti (per es. labbra, narici), muovono le palpebre e, modificando le pieghe cutanee, determinano i cambiamenti di espressione. Un solo nervo, il facciale, innerva tutti i muscoli, mentre la sensibilità al livello della cute è dovuta soprattutto alle diramazioni del nervo trigemino. La vascolarizzazione è assicurata in massima parte dall’arteria facciale.” Source available at: https://www.treccani.it/encyclopedia/face/ Last accessed on 22 February 2021. Translations from Italian are by the author of the article, unless indicated otherwise. The transcendence of the face: A semiotic-linguistic path 283 The surface of perception and communication It is worth looking further into the definitions given above to try to understand our object better. ‘Face’ is described through elements that are part of it. I have already pointed out the paradoxical character of this kind of definition, which leads to the difficulty of finding a notion’s semantic content by specifying autonomous pertinent traits characterizing it. Yet if we take a closer look at the parts of the face that are used to define it, we can find two groups: on the one hand, there are limit elements, i.e. those parts of the body that mark the boundaries of the face from the inside and only sometimes from the outside: the chin or the neck (downward); the forehead or the hair (upward); the ears or the hair (sideways). On the other hand, we find some objects that are on the face, which do not mark its boundaries, but are its conspicuous parts: the mouth, the nose, the eyes and, again, the ears. It is worth noting that these elements are only a subgroup of all notable parts of the face: no definition mentions, for example, the cheekbones, the cheeks or the eyebrows. Actually, the elements mentioned all serve as the site of sense organs, indeed of all the localized senses: sight, smell, hearing, taste, and partly also touch, a sense spread over the whole body. It is worth adding a consideration which is important from a communicative point of view: these organs are not purely passive receptors, which remain immobile just allowing themselves to be hit by sound, light or chemical stimuli. In fact, each of them (with the partial exception of the ears, which in humans are mostly immobile) acts to search, select and capture its own stimuli. The mouth opens or closes, lips protrude or retract, the tongue moves and explores, the cheeks swell or are squeezed in, exposing the teeth in order to bite or hiding them, etc. The lids become closed or the eyes widen, orienting themselves in one direction or another, stare sharply at something or wander distractedly. The nose can also dilate the nostrils or become wrinkled. These different activities are mostly natural consequences of choices of value, following the logic of appreciation or rejection, of pleasure or disgust, of interest or boredom. Yet at the same time, acting this way the sense organs express these choices. These acts involve muscles and skin even beyond the actual location of the sense organ: the wrinkling of the nose extends to the tension of the cheeks and of the forehead; uncovering the teeth causes a visible contraction that reaches up to the cheekbones, the cheeks, the chin. So, the whole face, or its major parts, is involved in these motions, often allowing an organ to replace the others from an expressive point of view: for example, we often and rightly speak of ‘smiling with the eyes’. All these complex actions are visible and convey values. They are called ‘expressions’ because they project the person’s thymism outward: this is where the semiotic terminology relating to the signifier comes from. From this point of view, the face is the prototype of every axiologized support of signification. 284 Ugo Volli In a species as strongly characterized by the density of social interactions and the constant interest in synchronization with others as that of the humans, this relationship between the activity of the sense organs and the perception of values can easily engender voluntary expressions, communication of values, even beyond the physical stimuli that generally give rise to the action. For this reason, the face, and in particular the sense organs that reside there, provide the place where a significant part of pre-technical human communication is concentrated: verbal communication, because the mouth is obviously the location of speech and the ears are the site of listening, but also all non-verbal communication: smiles, looks, laughter, grimaces, surprises, blushing and turning pale etc. It is the body part via which almost all the perception of the world enters the awareness of the subject, as well as the one that sends out most of our messages. It is the most important device for communicating with our fellow human beings. Therefore, it is the area that others look at to identify feelings and mental states, and that we therefore try to control better. Since human beings are used to knowing the face in detail, it is also the most important identification interface, the one with the help of which we recognize who is in front of us and that is portrayed in “identity documents”. However, these empirical considerations never appear in the dictionary definitions of the face, no doubt due to their complexity. These definitions contain yet another focal point, namely the presence of a directional indication: the face is the “forepart” of the head, often also referred to as the “front” of the head.5 This clarification is also important, because it evidently refers to the common experience of the indexical capacity of this part of the body, that is, of its being directed towards something or someone. Turning your face in a certain direction, that is, “facing it”, shows interest (which is etymologically ‘inter esse’, ‘being among’: facing something seems to imply metaphorically the act of going near or into that object). What is faced becomes the privileged object of attention and dialogue. All these dictionary definitions therefore imply (but never explicitly recognize) the role of the face as the input-output centre of communication and human perception. There are undoubtedly evolutionary reasons why not only humans, and not just primates or mammals, but a substantial share of all multicellular animals – including amphibians, reptiles, fish, certain mollusks and even insects – concentrate the organs of sense, nutrition and communication in one single front area of the body – that is, towards the direction in which these 5 It is perhaps not without interest to note that the same directional indication of ‘front’ as ‘anterior’ incidentally has its etymology in a facial metaphor. The Latin ‘frons’, from which the term derives, in fact meant ‘forehead’. The transcendence of the face: A semiotic-linguistic path 285 animals predominantly move. This concentration is what all human cultures have included in their lexicon with expressions that refer to the biologically artificial, yet functionally founded notion of ‘the face’. This strongly communicative dimension is also the reason why in different cultures the face is also often used as an inscription support for make-up and tattoos. Also, facial hair (beard, moustache, eyelashes, eyebrows) cut in different shapes performs the same communicative function. All these elements amplify the variable production of nonverbal communication by the muscles and skin. Face inscription as a tool used to intensify and modify expressions is a technique well known to mask makers and makeup professionals. Furthermore, the recognizability of the face and its expressive function mean that even its stable characteristics, such as the width and shape of the sense organs and their details, the overall geometric shape, the colour, etc., are often treated (naively, or according to various theories of “body language”) as bearers of meaning. It is clear that this diagnostic attitude to the shape of the face also involves several dangers: not only that of pseudoscience, but also racism, or at least perceptive ethnocentrism, which is difficult to escape from, given that it is a powerful force already at the level of animal evolution (Prum 2017). What emerges from everything said above is a strong textual dimension of the face, an intertwining of components that produce meaning according to a clear semiotic articulation. It is easy to identify a plane of expression in which the muscles, the skin, etc., make up the substance and what in common parlance is called the ‘facial expression’ constitutes the form; and then there is a plane of content whose substance is the thymic dimension of subjectivity and whose form is its articulation in a semantics of passions dependent on the particular culture. It is worth noting here that facial expressions are not to be examined in isolation, because they are complex on the level of the signifier, but above all strongly influenced by the context. An important example of this complexity is the “Kuleshov effect” that received its name from the Russian director who showed in a series of experiments that the same image of a character’s face with a neutral expression was generally interpreted as sad, hungry, in love, etc., when edited together with cinematic sequences that suggested a story in which he had to assume those passions in order to fulfill his actant role (Riley 1998). 286 Ugo Volli The linguistic comparison These stratifications of meaning are expressed in a strong complexity of linguistic signifiers that correspond to the concept of ‘the face’. They are very often related to the idea of seeing and being seen (Greek πρόσωπον – pròsopon; French visage), but also to taking a direction (Hebrew panim;6 ʭʩ ʑʰ˝ʕ ; Arabic ϪΟϭ wajh), to gaze (German Gesicht), to form or shape (Latin facies, Polish twarz), to head (Spanish cara), to beak (Spanish rostro), to the being in front (Chinese 面對 Miàn duì), to cheek (Russian litso). This remarkable etymological diversity, which is also found within rather narrow linguistic families that are often rather uniform in naming parts of the body, also testifies to the cultural construction of the notion of ‘the face’. It is worth investigating at least some of these differences, considering the case of the Italian language, which is characterized by a notable richness of terminology relating to the subject and whose derivations are generally well documented. The three main lexemes in Italian that directly and denotatively refer to the face are ‘faccia’, ‘volto’, and ‘viso’. However, one of the most authoritative thesauri of the Italian language, Dizionario dei sinonimi e dei contrary, lists 18 words for ‘face’, some of which are rather rare or connotative, the latter often taken from the nomenclature of animals: “faccia, viso, volto, muso, grinta, grugno, grifo, ghigna, ceffo, mostaccio, mutria” (Gabrielli 1967: 286), while on a more abstract side, there is a second part of the list with words that allude transparently to image, look and form: “fisionomia, sembiante, fattezze, lineamenti, tipo, profilo, sgorbio, effige”. The etymologies of the three main synonyms are quite clear. ‘Faccia’ comes from the “Latin facies, ‘shape, look, face’, akin to facĕre ‘do’”; ‘viso’ from the “Latin vīsus -us, properly ‘sight, look, aspect’, derivative of vidēre ‘to see’”. Things are more complicated only concerning ‘volto’. There is an evident relationship with the Latin “vŭltus, ‘face, aspect’”,7 which means ‘face’, but also ‘look’, ‘expression’, ‘physiognomy’, even ‘portrait’, as well as ‘side’, ‘surface’, and has a form ‘voltus’ (Campanini-Carboni 1993: 1735). This Latin word could have a close relationship with ‘volutus’, the perfect participle of ‘volvere’ (‘rolling’), as indeed in Italian there is a word ‘volto’ (‘facing’, ‘addressed to’), the past participle of ‘volgere’ (‘turn’), which derives directly from that Latin verb. It could therefore be related to our name ‘volto’ (‘face’). In this case the etymological meaning of this word would be that of ‘being turned’ in a certain direction, heading towards it, or offering a certain side of the head to the eye. Some believe that the two words have no relationship and the ‘volto’ that indicates the face can be “traced back to the root *uel, ‘to see’” 6 For a careful examination of biblical Hebrew, I refer to Volli 2020. The sources: https://www.treccani.it/vocabolario/face_%28Sinonyms-e-Contrari%29/; https://www.treccani.it/vocabolario/volto/; last accessed on 22 February 2021. 7 The transcendence of the face: A semiotic-linguistic path 287 (Cortellazzo, Zolli 1988: 1452). Others argue for the convergence of the two forms (Colonna 1997: 412). At the emergence of Italian literature, Dante Alighieri was already aware of this ambiguity and made it the subject of a famous paronomastic pun: ...e non mi si partia dinanzi al volto anzi ‘mpediva tanto il mio cammino ch’i’ fui per ritornar più volte volto. (Inferno, I, 36)8 It is worth mentioning the general comparison that the Treccani Dictionary of Synonyms makes of these terms and of the other more connotative ones, as this certainly clarifies the linguistic subtleties involved: Among the terms that indicate the front part of a human head, faccia is the most common and familiar one (and in fact it is used in numerous expressions, phrases and idioms […]); viso has a slightly more sustained tone; volto is more literary […]. Other terms are available in a derogatory or ironic tone: ceffo, grugno and muso. Ceffo is the most offensive term of the three, and is used to indicate a sinister, unpleasant or disreputable person […]. Grugno is properly the snout of the pig, and is used in a figurative sense to indicate both an ugly appearance [...], someone who one would gladly punch [...], and a sullen or angry expression [...]. Muso is a term that designates the front part of the head of any animal, and can be metaphorically extended to humans both in a derogatory sense [...] and as a term of endearment (especially with the altered musetto and musino).9 It is worth pointing out that ‘ceffo’ comes “from the old French chief, that is ‘head’”; ‘grifo’ “from the Latin grȳpus, meaning with a big nose”, which, however, is the name of a bird of prey that we still call ‘griffon’; ‘grugno’ “from the Latin grunium, probably from grunnīre ‘grunt’; to compare with the French groin ‘snout’, Romansh 8 “refused to back away from me but so impeded, barred the way, that many times I turned to go back down.” (translation by Robert Hollander and Jean Hollander) 9 https://www.treccani.it/vocabolario/faccia-finestra-di-approfondimento_%28Sinonimie-Contrari%29/: Tra i termini che indicano la parte anteriore della testa dell’uomo, f. è quello più com. e fam. (e infatti è usato in numerose espressioni, locuz. e modi di dire) […] viso è di tono leggermente più sostenuto, volto è più lett.[…] In tono spreg. o iron. sono disponibili altri termini: ceffo, grugno e muso. Ceffo è il termine più offensivo dei tre, e si usa per indicare una persona dall’aspetto sinistro, sgradevole o poco raccomandabile[...]. Grugno è propriam. il muso del maiale, e viene usato in senso fig. per indicare sia un brutto aspetto[…] Muso è infine il termine che designa la parte anteriore della testa di qualunque animale, e può essere esteso metaforicamente all’uomo sia in senso spreg. [...], sia in senso vezzeggiativo (soprattutto con gli alterati musetto e musino. Last accessed on 22 February 2021. 288 Ugo Volli gruogn ‘chin’ and Romanian grui ‘top of the hill, hill’”; and that, at last, ‘muso’ derives from the “late Latin mūsu(m), of unknown etymology”; but it is worth adding that “according to some [it comes] from (vultus) musus (‘prominent mouse face’),10 from muricu, extracted from mus (‘mouse’). According to others from morsus, caming from a Germanic root *mu” (Colonna 1997: 254). Also, in the classification of the Treccani Dictionary we can see a division into two parts of the semantic field, one being more characteristic, connotative and metaphorical, and the other more abstract and denotative: By faccia, viso and volto you may wish to mean the outward appearance, facial features in general (his faccia tells me something, maybe we have already met?). In this case we can speak of aspetto, [‘appearance’] (it has a familiar aspetto), fattezze [‘features’] or fisionomia [‘physionomy’] (all three more general terms, which can refer to features that are not only facial [...] or, more specifically for the characteristics of the face, of lineamenti [‘outlines’]. [...] Aspetto [‘appearance’], more generically and less directly than faccia, also refers to the way of posturing, of appearing. Apparenza [‘appearance’], aria [‘look’] or espressione [’expression’] are also used in the same meaning. […] Atteggiamento [‘posture’] is rarely limited to the outward appearance, mostly indicating an overall inner way of being that also transpires outwardly [...]. Cera [‘mien’] is more specific for health conditions: having a good or bad cera indicates the complexion and facial expression of those who feel good or bad.11 The Italian semantic field of ‘face’ is thus characterized in three layers. First of all we have a nucleus of three words (which in many other languages are one, as in the English ‘face’): ‘faccia’, ‘viso’, ‘volto’, which are largely overlapping, but not perfectly equivalent – you cannot say *‘viso a viso’ as you say ‘faccia a faccia’ (‘face to face’); the ‘sacro volto’ (‘sacred face’) is not the *‘sacra faccia’; there are affectionate diminutives of ‘faccia’ and ‘viso’ (faccina, visino) but not of ‘volto’, etc. 10 Sources: https://en.wiktionary.org/wiki/ceffo; https://it.wiktionary.org/wiki/grifo; https://www.treccani.it/vocabolario/faccia-finestra-di-approfondimento_%28Sinonimi-eContrari%29/; https://www.garzantilinguistica.it/ricerca/?q=muso https://en.wiktionary.org/ wiki/grugno were accessed on 22 February 2021. 11 Con f., viso e volto si può anche voler intendere l’aspetto esteriore, i tratti facciali in generali (la sua f. mi dice qualcosa, forse ci siamo già incontrati?). In questo caso si può parlare di aspetto (ha un aspetto familiare), fattezze o fisionomia (tutti e tre termini più generali, che possono rimandare a caratteristiche non soltanto facciali [...] oppure, più specificamente per le caratteristiche del volto, di lineamenti [...]. Aspetto, più genericamente e meno direttamente di f., rimanda anche al modo di atteggiarsi, di apparire: oggi non hai un bell’aspetto, forse ti senti poco bene? In questo stesso sign., sono usati anche apparenza, aria o espressione[...]Atteggiamento è raramente limitato all’aspetto esteriore, indicando per lo più un complessivo modo di essere interiore che traspare anche all’esterno:[...] Più specifico per le condizioni di salute è cera: avere una bella o una brutta c. indica il colorito e l’espressione facciale di chi si sente bene o male. The transcendence of the face: A semiotic-linguistic path 289 Then there is a second, picturesque and metaphorical layer, with strong animal connotations; and, finally, a third more abstract layer, such as ‘aspetto’, ‘fisionomia’, ‘fattezze’, which is used in theoretical or official discourses rather than in conversation. This multiplicity is not an exclusive feature of the Italian language, surely it applies elsewhere as well, but of course with different details; however, this theme should certainly be deepened with targeted lexicological research. Metaphors of the face One feature is striking in this review of the Italian terms and also of the main words in other languages listed above: in none of them is the face presented in relation to the subjectivity, the self, the personality, the true character of the individual, the real person.12 On the contrary, the face appears like a surface opposite to the world, which hides the intentions, thoughts and emotions of the subject. Erving Goffman (1955, 1967) has introduced an important sociological reflection on this superficial and external character of the face. He proposed to focus on an important relational aspect of social life, namely the interaction between subjects defined for their role, which he characterized using the metaphor of ‘face’, as common language often does. According to Goffman’s “dramaturgical” metaphor, the face is a mask which is worn according to the audience and the variety of social interaction. This is the most important element of the “scene” in the “play” that regulates the way people appear to one another in social life. The face indicates a certain role, a position, a carefully constructed social identity. People strive to keep the face they have created in social situations and they see the prospect of “losing it” as a grave danger. They are emotionally attached to their faces, so they feel good while these are kept; face “loss” results in emotional pain, so in social interactions people cooperate using courtesy strategies to keep each other’s face. The micro-sociological theory, however, borrowed these face-based metaphors from a widespread linguistic practice. Since Dante (Purgatorio, XXIII, 101), we will find that the word ‘sfacciato’ (‘brazen’, composed of the privative ‘s’ and ‘face’: ‘without a face’) in Italian indicates a “person who has no modesty, restraint, shame” (Cortellazzo, Zoli 1988: 1190) and has a very negative connotation. Someone “faceless” is a disreputable individual. The notion of ‘losing face’ is present in Japanese (面目 を 失 う menboku wo ushinau); in Persian (Aab ro rizi ΰϳέϭήѧѧѧѧѧѧѧΑ΁̵ ‘losing the face’s water’); in various South Slavic languages, as in Serbian, Croatian and Bulgarian (безобразен, bezobrazan); in Chinese (diu mianzi 12 For a semiotic and philosophical-linguistic investigation of this last linguistic field, see Volli in print. 290 Ugo Volli 丢面子).13 The notion is especially important and widely studied in Chinese culture (Zuo 1997; Huang 1987); there is, for example, a widespread proverb “Men cannot live without a face just as trees cannot live without bark” (ren hou lian, shu hou pi – 人 后 脸 ， 树 后 皮). In Arabic, the correlative expression ‘hafiẓa māʼ al-wajh’ (ϪѧѧΟϮϟ΍˯ΎѧѧϣφѧѧϔΣ), which literally translates as ‘save the face’s water’, is used to mean ‘to save face’. The vast diffusion of this metaphor is undoubtedly striking. George Lakoff and Mark Johnson (1980) suggest that the explanation is to be found in the metonymic relation that replaces a part (the face) for the whole (the whole human being). It is true that if you ask someone to exhibit a photograph and he shows you a picture of his face, you are satisfied, yet this does not happen if he shows you the image of his back or of the whole body without the face. Yet beyond the purely material identification, ‘face’ indicates a social position, a widespread esteem, a role – abstract qualities that can be lost and recovered, not just the physical individuality which is much more rooted and more or less indelible. Another clue in this sense is the use of the word ‘interface’ that used to denote the surface of separation between two liquids, but in the 1980s came to mean the contact between two computers and then the instrument of human–machine communication, with the implication that what appears on the interface does not represent the computer processes, but forms a representation or translation adapted to the user’s needs.14 The interface, as the face, is something between the poles of communication that both expresses and hides. In other words, not only in microsociology but also in everyday life, the face is mostly perceived as a mask. This is a widespread term worth dwelling on for a moment. The root used in most Indo-European languages (Italian ‘maschera’, French ‘masque’, English ‘mask’, Spanish ‘mascara’, German ‘Maske’, Russian ‘маска’, etc.) has an unclear etymology. As Mariella Canzani writes in a note published on 15 March, 2020 for the linguistic consultancy editorial staff of the “Accademia della Crusca”, the most important institution dedicated to the Italian language:15 13 A repertoire of other similar linguistic traces can be obtained from https://en.wikipedia. org/wiki/Face_(sociological_concept); last accessed on 21 February 2021. 14 “The place at which independent and often unrelated systems meet and act on or communicate with each other; the means by which interaction or communication is achieved; a surface forming a common boundary of two bodies, spaces, or phases” (https://www.merriamwebster.com/dictionary/interface). 15 The source: https://www.facebook.com/notes/353039995777856/ was last accessed on 22 February 2021. The transcendence of the face: A semiotic-linguistic path 291 The main hypotheses are two, not in contrast with each other: one suggests the derivation from a pre-Indo-European form masca ‘soot, black ghost’ (FEW reconstructs a base *mask ‘dark’.), the other suggests the derivation from masca ‘witch’, a regional expression of the Ligurian and Piedmontese area. Masca in turn derives from the late Latin masca(m), a noun used in the same meaning, a term documented by written sources starting from the mid-seventh century, first attested in the Edict of Rotari (Lex longobardorum I, 11: strigam, quodest mascam), which in turn could be of pre-Roman origin. From the original meaning of ‘witch’ we will arrive later to that of a ghost, a spirit, a disguised aspect to instill fear. The linguistic evolution of the signifier instead led to the addition of an ‘r’, making the term assume the form first of mascra and then of mascara, an ancient variant attested in the 13th century. According to some scholars, the late Latin masca(m) should be a relict of the pre-Gallic substratum, alternating with *basca, cf. Fr. rebâcher, from Old French. * rabaschier ‘make noise’, used about ghosts, from the verb *bascare that corresponds to the Greek báskein, a term that alludes to magical fascination. Finally, other scholars have suggested a derivation of the etymology from the Arabic phrase maschara or mascharat, ‘fool, hoax’, deriving from the verb sachira, ‘to mock’, imported into medieval language from the Crusades. However, the presence of this word in texts prior to the Crusades demonstrates the inaccuracy of the Arabic derivation of this family of words. And yet, as Canzani notes again, “the term of classical Latin to indicate the theatrical mask used by the histriones (actors) was persona. It is a loan, through the Etruscan phersu, from the Greek prósōpon”, a word which we have already encountered in the list of the main terms corresponding to ‘face’ in various languages, because its first meaning is precisely “‘forehead, face, face’, and then ‘mask’ and ‘character’”. There is, thus, an important etymological relation between ‘mask’, ‘person’ and ‘face’, which was also exploited by the great Italian playwright Luigi Pirandello when he entitled a collection of his theatrical texts Naked Masks, proposing a double oxymoron – double, because the mask is an inanimate object and nudity applies only to human beings; and because a face covered by a mask obviously cannot be naked. Conclusion: Some philosophical consequences What interests us most in this exploration of the metaphorical use of the face as a mask is the fact that it is made possible by a fundamental and often insufficiently emphasized semantic trait that characterizes theses notions, their exteriority, which is a fundamental point in the philosophical analysis of the face. This theme of the exteriority of the face was developed by Levinas in terms of the ‘transcendence’ of the other: 292 Ugo Volli The face escapes from possession, from my power. In its epiphany, in expression, the sensitive, which is still graspable, changes into total resistance to the grip. This change is only possible thanks to the opening of a new dimension. In fact, resistance to grip is not produced as an insurmountable resistance, as the hardness of the rock against which the effort of the hand is useless, as the distance of a star in the immensity of space. The expression that the face introduces into the world does not challenge the weakness of my power, but my power of power. The face, still a thing among things, opens a gap in the form that delimits it. Which means concretely: the face speaks to me and thus invites me to a relationship that has no common measure with a power that is exercised, even if it is enjoyment or knowledge. [...] When I refer to the face, I don’t just mean the colour of the eyes, the shape of the nose, the redness of the lips. Stopping here I still contemplate only some data; but even a chair is made of data. The true nature of the face, its secret lies elsewhere: in the question he asks me, a question that is both a request for help and a threat. (Levinas 1961: 85; my translation, U. V.) Levinas recognizes that the condition of the “transcendence” of the face of others is “its secret” which always “lives elsewhere”: elsewhere than the world of things of which it is also a part. This is its “exteriority”, that is, the fact that we do not have direct access to the other through the face, that the other always exceeds it. If anything, the face should be assimilated to a “trace” [“the presence of what, properly speaking, has never been, of what has always already passed” (Levinas 1972: 89)] and is the condition of all signification: It (the expression) presents the signifier. The signifier, the one who gives a sign, is not signified. It is necessary to have already been in an area of signifiers, for the sign to appear as a sign. The signifier must therefore present itself before any sign, in itself – as a face. (Levinas 1961: 186; my translation, U. V.) This condition of excess and exteriority is therefore very similar to that of the linguistic signifier, which is never transparency of the sign; in language the signifier is always opaque and this condition grounds the principle of arbitrariness. Meaning exceeds its signifier, it is not determined by it, just as the human being exceeds his/ her face, he/she can think and feel and be more than what he/she lets to be seen. This condition of transcendence, which filters the sense of the face and is eminently expressed in the linguistic capacity of humans, is what underlies their freedom – the human exception. With the term ‘exception’ I refer to the logical sense of this word, that is as “a case to which a rule does not apply”16, without any reference to the ‘state of exception’ mentioned in Agamben 2005, because this is a juridical measure taken 16 The definition derives from https://www.merriam-webster.com/dictionary/exception. The transcendence of the face: A semiotic-linguistic path 293 by some authority, while I am referring to the attribution to human beings of a permanent status that frees them from the causal system that we attribute to all things of the world, including living beings. If all human behaviour were simply a consequence of causes, as is the case with avalanches, earthquakes or hurricanes, we could not attribute to it evaluations or responsibilities, we could not speak of faults and merits. On the other hand, when we do so and we hold human beings responsible for the consequences of their behaviour (which is not so with mountains and seas, nor even with snakes or microbes), we implicitly, rightly or wrongly, believe them to be capable of a freedom of choice that other things in the world do not have. In short, we presuppose a gap between causes and behaviour, just as there is a gap between the signifier and meaning and between the face and the subject. This is the “human exception”, which of course requires explanations from philosophy (or from theology, or from science). In Volli 2010 and Volli 2013 I supported the thesis that this “transcendence” with respect to the causal system, that is attributed to human beings, is derived from the linguistic competence that characterizes them (another significant “human exception”), because language is characterized by arbitrariness, that is, by the lack of causal connection between expression and meaning (in Saussure’s terms: the signifier and the signified). There is a degree of freedom in the process of signification which makes the meaning always partially indeterminate. The absence of meaning from the expression is decisive for this arbitrariness and it also recalls the transcendence or absence of the subject in the face. This relationship, not at all trivial, but evident between language and the face is also underlined by Giorgio Agamben in a short essay written in 1996,17 now included in a collection of political essays: What the face exposes and reveals is not something that could be formulated as a signifying proposition of sorts, nor is it a secret doomed to remain forever incommunicable. The face’s revelation is revelation of language itself. Such a revelation, therefore, does not have any real content and does not tell the truth about this or that state of being, about this or that aspect of human beings and of the world: it is only opening, only communicability. To walk in the light of the face means to be this opening– and to suffer it, and to endure it. Thus, the face is, above all, the passion of revelation, the passion of language. Nature acquires a face precisely in the moment it feels that it is being revealed by language. (Agamben 2006[1996]: 93) 17 I thank the anonymous reviewer who prompted me to develop my analysis further at this point. 294 Ugo Volli Agamben’s reasoning is somewhat different from the line of analysis I have proposed till now and also differs from that of Levinas. For Agamben (1996: 91), “the face is at once the irreparable being-exposed of humans and the very opening in which they hide and stay hidden”. This opening is also common to animals, but “only human beings want to take possession of this opening, to seize hold of their own appearance and of their own being being-manifest […]. Language is this appropriation, which transforms nature into face”. The emphasis here is not about the “hiding in the opening”, namely absence in the appearance, which I have already discussed above, but about the “appropriation” and “possession” of this opening, which can be understood as the strategic games of the appearance studied by Goffman under the name of ‘face’. This leads Agamben (1996: 91) to think that “the face is the only location of community, the only possible city”, in particular today it is a “city”, a polis that Agamben criticizes in political terms that we will not discuss here. In Agamben’s terminology, the individual and expressive level that I have analysed so far is the ‘visage’ with which “the face does not coincide”: The only face to remain uninjured is the one capable of taking the abyss of its own communicability upon itself and of exposing it without fear or complacency. This is why the face contracts into an expression, stiffens into a character, and thus sinks further and further into itself. As soon as the face realizes that communicability is all that it is and hence that it has nothing to express – thus withdrawing silently behind itself, inside its own mute identity – it turns into a grimace, which is what one calls character. Character is the constitutive reticence that human beings retain in the word; but what one has to take possession of here is only a non latency, a pure visibility: simply a visage. The face is not something that transcends the visage: it is the exposition of the visage in all its nudity, it is a victory over character – it is word. (Agamben 1996: 95) In the face there would therefore be a tension between support and function, “the duality of proper and improper, of communication and communicability, of potentiality and act. The face is formed by a passive background on which the active expressive traits emerge”. In this regard, Agamben cites a long passage from Rosenzweig’s Der Stern der Erlösung, where the German philosopher sees in the face a symbolic instance of the “Star of Redemption”, consisting of two triangles: a “basic” one which takes shape in the alignment of the sense organs (eyes and mouth), which is “pure receptivity”, and another, “superimposed” one, composed of the other parts “whose activity quickens the rigid mask of the first” (see Rosenzweig 1970[1921]: 422–23). Agamben’s (1996: 97) comment shifts this symbolic towards a socio-political vision of our time: “In advertising and pornography (consumer society), the eyes and the mouth come to the foreground; in totalitarian states The transcendence of the face: A semiotic-linguistic path 295 (bureaucracy), the passive background is dominant (the inexpressive images of tyrants in their offices).” Agamben’s analysis is best understood by thinking of the great tradition of philosophical anthropology that starts with Aristotle (Nicomachean Ethics I.13, 1102a5–1103a10), for which, apart from the contemplative life of the philosopher, the fullness of the human is reached in the “vita activa” of public action, the living presence of the person in the public sphere (Arendt 2013[1958]), and this action in turn is made possible mainly by language. The face thus becomes the place of this opening to the polis and it undergoes its vicissitudes, in particular as regards the general object of Agamben’s critique – modern Western society. This is therefore a philosophy of the face that works exclusively on its general social dimension: not on the face of the individual human being and on his relationship with his identity and with his neighbours, but on the regime of visibility that faces realize and endure in relation to circumstances of collective life. In short, the face for Agamben is a certain way of acting on others, a communication tool that, like all media, can be emptied to the point of only expressing its own communicability. However, there is at least one other point of view which considers the face as a fundamental place of personal identification, originating from Lacan’s mirror phase (Lacan 1949). This approach and the aporias it opens up are the central theme of the path proposed in this paper. The face appeared to us as a certain placeholder of absence, a certain support of the not being all there of the person who is present; its mask dimension allows it to play differently from that of simply present objects. It is never a neutral object, while it is always the bearer of values and the object of evaluations; in a certain sense it is the original model of the communication of values. Therefore, it can be an actual sign of the person, bringing to others some of his qualities, certain of his values, his current thymic position. With two caveats: that the sign can never be identical to its object (as Plato already claimed in Cratylus, 432a), but must always highlight some of its sides, a face; and that the person (or prosopon – namely his/her face) can never be identical to the human being it represents socially, but highlights some aspects, some faces of this human being, or even constructs them. References Agamben, Giorgio 2000[1996]. Means without End: Notes on Politics. (Casarino, Cesare; Binetti, Vincenzo, trans.) Minneapolis: University of Minnesota Press. Agamben, Giorgio 2003. Stato di eccezione. Torino: Bollati Boringhieri. Beaney, Michael 2013. What is Analytic Philosophy? Oxford: Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199238842.013.0039 296 Ugo Volli Arendt, Hannah 2013[1958]. The Human Condition. Chicago: University of Chicago Press. Bird, Alexander; Tobin, Emma 2008. Natural kinds. In: Zalta, Edward N. (ed.), The Stanford Encyclopedia of Philosophy. Stantford: The Scholarly Publishing and Academic Resources Coalition. Campanini, Giuseppe; Carboni, Giuseppe 1993. Vocabolario Latino Italiano / Italiano Latino Torino: Paravia. Colonna, Barbara 1997. Dizionario etimologico della lingua italiana. Roma: Newton. Cortellazzo, Manlio; Zolli, Paolo 1988. Dizionario etimologico della lingua italiana. Bologna: Zanichelli. Daly, Chris 1998. Natural kinds. In: Craig, Edward (ed.), Routledge Encyclopedia of Philosophy. Vol. 6. London: Routledge, 682–685. Darwin, Charles 2009[1872]. The Expression of the Emotions in Man and in the Animals. London: Penguin. https://doi.org/10.1017/CBO9780511694110 Eco, Umberto 1983. Semiotics and the Philosophy of Language. Bloomington: Indiana University Press. Huang, Shuanfan. 1987. Two studies of prototype semantics: Xiao ‘Filial Piety’ and Mei Mianzi ‘Loss of Face’. Journal of Chinese Linguistics 15: 55–89. Ferraris, Maurizio 2009. Documentalità. Perché è necessario lasciar tracce. Roma-Bari: Laterza. Gabrielli, Aldo 1967. Dizionario dei sinonimi e dei contrari. Milano: Istituto Editoriale Italiano. Goffman, Erving 1955. On face-work. Psichiatria 18(3): 213–231. https://doi.org/10.1080 /00332747.1955.11023008 Goffman, Erving 1967. Interaction Ritual: Essays in Face-To-Face Behavior. Chicago: Aldine. Lacan, Jacques 1949. Le Stade du miroir comme formateur de la fonction du Je: telle qu’elle nous est révélée dans l’expérience psychanalytique. Revue française de psychanalyse 13: 449–455. Lakoff, George; Johnson, Mark 1980. Metaphors We Live By. Chicago: The University of Chicago Press. Levinas, Emmanuel 1961. Totalité et infini: Essai sur l’extériorité. La Haye: Martinus Nijhoff. Levinas, Emmanuel 1972. Humanisme de l’autre homme. Montpellier: Fata Morgana. Lotman, Juri 2011[1967]. The place of art among other modelling systems. [Tezisy k probleme “Isskusstvo v ryadu modeliruyuschih sistem”.] (Pern, Tanel, trans.) Sign Systems Studies 39(2/4): 249–270. https://doi.org/10.12697/SSS.2011.39.2-4.10 O’Riley, Tim 1998. Representing Illusions: Space, Narrative and the Spectator in Fine Art Practice. London: Chelsea College of Art and Design. Prum, Richard 2017. The Evolution of Beauty. New York: Doubleday. Quine, Willard 1960. Word and Object. Cambridge: The MIT Press. Rosenzweig, Franz 1970[1921]. The Star of Redemption. (Hallo, William W., trans.) New York: Holt, Rinehart and Winston. Searle, John 1995. The Construction of Social Reality. New York: Free Press. Volli, Ugo 2010. Al di là dell’essenza, il linguaggio. Quaderni laici 1: 113–119. Volli, Ugo 2013. Riflessione e trascendenza di una maschera. In: Leone, Massimo; Pezzini, Isabella (eds.), Semiotica della soggettività – Per Omar (I Saggi di Lexia II). Roma: Aracne, 265–278. The transcendence of the face: A semiotic-linguistic path 297 Volli, Ugo 2020. Invisibile, espressivo e necessario Metafore del volto divino nella Bibbia ebraica. Lexia (Volti Artificiali/Artificial Faces; Leone, Massimo, ed.) 37–38: 557–583. Volli, Ugo 2021. Di che cosa parliamo quando diciamo ‘persona’? – Fra filosofia e semiotica. In: Ponzo, Jenny; Vissio, Gabriele (eds.), Culture della persona: Itinerari di ricerca tra semiotica, filosofia e scienze umane. Torino: Accademia University Press. Zuo, Bin 1997. Chinese “lian” and “mianzi”. Wuhan: Huazhong Normal University Press. Трансцендентность лица: лингво-семиотический путь В начале статьи мы анализируем термины, используемые для обозначения лица в разных языках, в частности, в итальянском, сравнивая их с определениями, предложенных некоторыми авторитетными словарями, а также с их этимологией. Это исследование приводит к примечательным результатам: во-первых, оказывается, что термин «лицо» имеет материальное значение, но в то же время является социальным объектом; во-вторых, подчеркивается важность коммуникативной функции, которая делает лицо похожим на маску, что, в некотором роде, напоминает арбитрарность языка. Все это говорит о том, что философский статус лица – это «трансцендентность», которая является условием того состояния свободы, которое мы приписываем себе и которое можно определить как «человеческое исключение». Näo transtsendentsus: semiootilis-lingvistiline tee Artikkel algab nägu tähistavate terminite vaatlemisega erinevates keeltes, eriti itaalia keeles, kõrvutades neid definitsioonidega, mida pakuvad mõned autoriteetsed sõnaraamatud ja ka nende etümoloogiaga. Need vaatlused annavad tähelepanuväärseid tulemusi: esiteks ilmneb, et nägu on tõepoolest termin, millel on materiaalne tähendus, ent samal ajal on tegu sotsiaalse objektiga; teiseks tuleb esile kommmunikatiivse funktsiooni olulisus, mis muudab näo sarnaseks maskiga ning mõnel moel ka keele arbitraarsusega. Kõik see viitab asjaolule, et näo filosoofiline staatus on “transtsendentsus”, mis on selle vabadusseisundi tingimus, mida me omistame enestele ning mida võib defineerida kui “inimlikku erandit”. 298 Simona Stano Sign Systems Studies 49(3/4), 2021, 298–317 The face of health in the West and the East: A semio-cultural analysis Simona Stano1 Abstract. Magazines, leaflets, weblogs, and a variety of other media incessantly spread messages advising us on how to achieve or maintain our health or well-being. In such messages, the iconic representation of the face is predominant, and reveals an interesting phenomenon: the “face of health” seems to be unattainable as such, and is generally represented in a differential way, that is to say, by making reference to its opposite – the “face of illness”, or at least of malaise. In fact, the face is crucial in the medical domain: since ancient times, face observation has played an essential role in diagnostic practices, both in Western medicine (which resorts to the concept of facies, intended as the distinctive facial expression or appearance associated with a specific medical condition, for the description of pathological states) and Eastern preventive and healing techniques (within which the so-called Mian Xiang, or ‘face reading’, is fundamental, and connects the medical sphere with other aspects such as personality, talents, and dispositions). Drawing on the semiotic analysis of relevant case studies extending from classical iconography to present-day digital mediascapes, this paper investigates the representation of the face of health (and illness) across time and space, specifically focusing on the analogies and differences between the Western and the Eastern semiosphere. To this purpose, it relies on both literature concerning the representation and understanding of the face and studies on medical sign systems and discourses. Keywords: face; health; collective imaginary; medicine; West vs. East; semiotics. 1 Department of Philosophy and Education Science, University of Turin (UNITO), Torino, Italy; e-mail: simona.stano@unito.it. https://doi.org/10.12697/SSS.2021.49.3-4.03 The face of health in the West and the East: A semio-cultural analysis 299 Proprio nomine dicitur facere a facie, qui rei quam facit imponit faciem. Ut fictor cum dicit fingo, figuram imponit, quom dicit formo, formam, sic cum dicit facio, faciem imponit. Varro, De lingua latina VI, 78 1. Introduction “What does your face say about your health?”; “Have you ever wondered if there is a connection between locations on your face and body organs?”; “The secrets your face tells about your health”. These are only a few of the several references to the close relation between the face and health that can be found in posts and articles featured on the Internet. Magazines, leaflets and other media advising us on how to achieve or maintain our health or well-being also insist particularly on the face and its visual representation. What is more, since ancient times, face observation has played an essential role in diagnostic practices, both in Western medicine – which resorts to the concept of facies, intended as the distinctive facial expression or appearance associated with a specific medical condition, for the description of pathological states – and Eastern preventive and healing techniques, within which the so-called Mian Xiang, or ‘face reading’, is fundamental, and connects the medical sphere with other aspects such as personality, talents, and dispositions. Why has the face been elected as the plane of discursivization of health (or illness) par excellence? How does cultural variation influence such a relation and the facial representation of health? What do such representations tell us about the understanding of health itself? Drawing on the semiotic analysis of relevant case studies extending from classical iconography to present-day digital mediascapes, the following paragraphs aim at addressing these issues, investigating the representation of the face of health across time and space, and specifically focusing on the analogies and differences between the Western and the Eastern semiosphere. 2. Health: In search of a definition Before analysing the link between the face and health, it is worthwhile clarifying, at least briefly, what is meant by these terms. The face is a complex object of analysis, as suggested by the multiplicity and variety of terms used to refer to it (especially in Romance languages) and their etymology: 300 Simona Stano The complexity of the face as an observable, decipherable, and describable object has always posed a quantity of questions that, at different times, have found different answers: sometimes explaining the entire macrocosm through the face, sometimes making the face into a whole macrocosm; sometimes considering it simply as a transparent mirror of passions, sometimes rather conceiving it as a treacherous mask produced by the game of social simulation. Traces of the thousand contradictions characterizing the face in the collective imaginary are inscribed, albeit in a disconnected and fragmentary way, in the etymological history of the natural language terms that have been used to designate it: [in Italian] volto, viso, faccia; in Latin vultus, facies; in French chère, face, figure, vis, visage, vout [...]. 2 (Magli 1995: 10) A particular aspect, more than others, is relevant to the purposes of this paper: The face is above all, in a passive sense, visus, [that is to say] the part of us, of our person and not only of our body, that is more exposed to visibility. It is the part of us that remains most entangled in the game of looks, of seeing and being seen, of showing off and hiding.3 (Magli 1995: 10) As such, the face represents first of all a form, which is subjected to countless strategies of signification and reading (Leone 2018: 27). It is a discourse, that is to say, an utterance, an act of enunciation. Likewise, health is a multifaceted and not at all univocally defined concept. Traditionally, it has been described mainly in differential terms, that is to say, as the absence of what is generally called ‘disease’: as Paolo Fabbri (1995: 37) remarked, “the Latin word salus is related to salvus, which means the one who is saved from changes, from disturbances coming from outside. It is a question of total integrity, a kind of ‘skin’ that saves and puts at ease”.4 In fact, most Latin2 “La complessità del volto come oggetto osservabile, decifrabile, descrivibile, da sempre ha posto una quantità di quesiti a cui, in tempi diversi, si è tentato di dare risposta: ora spiegando l’intero macrocosmo attraverso il volto, ora facendo del volto l’intero macrocosmo, ora considerandolo come specchio trasparente delle passioni, ora come maschera infida prodotta dal gioco della simulazione sociale. Tracce delle mille contraddizioni di cui vive il volto all’interno dell’immaginario collettivo sono iscritte, seppure in modo sconnesso e spesso frammentario, nella storia etimologica dei termini che, all’interno della lingua naturale, lo designano: volto, viso, faccia; in latino vultus, facies; in francese chère, face, figure, vis, visage, vout […].” (Translations from Italian are mine, unless indicated otherwise, S. S.) 3 “Ma il volto è soprattutto, in senso passivo, visus, quanto di noi, della nostra persona e non solo del nostro corpo, è più esposto alla visibilità. È quanto di noi più resta impigliato nel gioco degli sguardi, del vedere e dell’essere visti, del far mostra di sé e del nascondersi.” 4 “Il latino salus è legato a salvus che significa colui che si salva dalle modifiche, dalle perturbazioni provenienti dall’esterno. Si tratta di un’integrità totale, una specie di pelle che salva, che mette tranquilli, elimina le sollecitazioni della curiosità.” The face of health in the West and the East: A semio-cultural analysis 301 based terms used to designate health (such as, for instance, the Italian ‘salute’) highlight such a character of “integrity, safety and therefore salvation” (Pianigiani 2020).5 The Germanic origin of the old English term ‘hǣlth’, from which the word ‘health’ derived, also relates to ‘whole’ (Oxford English Dictionary 2020),6 further enhancing such a perspective. In this sense, as noted by Jacques Fontanille (2005), health is conceived of as an evaluative statement antithetical to disease: the former describes a state that fully conforms to the norm, the latter rather establishes a divergence from it. This asymmetry is then reflected on a pragmatic level, since there seems to be only one way of being healthy, as opposed to a multiplicity of ways of being ill; and also on the syntagmatic level, since health does not seem to have a history (it is a temporal segment), while illness is above all a “narrative sequence” (i.e. an aspectualized7 path, including an initial phase – i.e. the onset of the disorder, such as a lesion, or the moment of incubation, etc.; a durative phase – i.e. the development of the disease, consisting for instance in the infection; and a terminative phase – i.e. healing, or rather deterioration and death). From a narrative point of view, health corresponds in this view to “a dense and unanalysable ‘state’, while illness is perceived as a multiplicity of paths implying both losses and gains, emerging problems, crises and healings”8 (Fontanille 2005: 35) – see Table 1. 5 “Integrità, Incolumità e quindi Salvezza.” The source, Pianigiani, Ottorino 2020. Dizionario etimologico online, available at https://www.etimo.it, was last accessed on 19 February 2021. 6 Oxford English Dictionary, available at https://www.lexico.com, was last accessed on 19 February 2021. 7 As remarked by Greimas and Courtés (1982[1979]: 18), “all temporalized discourse has two kinds of new investments which produce the two meaning effects of temporality and aspectuality. While the effect of temporality is linked to the establishment of a group of temporal categories which […] project a temporal organizations of topological nature onto the utterance, the effect of aspectuality results from the investments of aspectual categories which convert the functions (or predicated) of narrative utterances into a process”. In this sense, health seems to be closer to the traditional narrative logic of programming, whereas illness features a processual development, which combines punctual and durative phases, perfective or imperfective aspects, iterative or singular sequences. 8 “[…] uno “stato” denso e inanalizzabile, mentre la malattia viene percepita come una molteplicità di percorsi che implicano perdite e guadagni, problemi emergenti, crisi e guarigioni.” 302 Simona Stano Table 1. The opposition between health and illness. Health Negative concept (absence of events) Static balance Illness Positive concept (presence of events) Dynamic imbalance Conformity Divergence Temporal segment Unique, dense, and unanalysable ‘state’ Aspectualized path Modular and analysable multiplicity of ‘paths’ However, as the French scholar effectively remarked, such an approach is too simplistic, and fails to explain a recurring aspect of the disease, which is also a mode of its possible indirect presence within a state of health: the malaise. In fact, according to Fontanille, the malaise can appear in all phases of illness and even in its absence. Even if an external look (such as that of a doctor) can later recognize it as a symptom of a specific illness, the malaise as a feeling or sensation is not necessarily related to the sphere of disease. Such a perspective – which recalls the Barthesian conception of the symptom (Barthes 1972), according to which it is not a sign in itself, but becomes such only when it enters the context of clinical discourse9 – is of particular relevance, since it highlights that [h]ealth is not a restful adventure, and it even seems that it can be better described – going against common sense – as a permanent tension, that is to say, a multi-polemic process. The disease, on the other hand, is almost always the result of a release of tensions (the defenses surrender, the body no longer reacts to attacks, the cells no longer die and begin to proliferate, etc.). [...] Of course, thanks to the constantly changing balances that it manages to guarantee, health manifests itself 9 In his essay “Sémiologie et médecine”, Roland Barthes (1972) describes symptoms as brute facts emerging from the body – that is to say, purely phenomenal sensations that are not necessarily linked to any act of interpretation, and so do not constitute semiotic entities as such. Nonetheless, he also supports the view that they turn into signs once they are “put into discourse”, that is, when they are “modelled” (in terms of Lotman 1977) through language. Such an opposition is partially overcome by Thomas Sebeok, who pushes further the association between signs and symptoms, conceiving the latter as “compulsive, automatic, nonarbitrary sign[s], such that the signifier coupled with the signified in the manner of a natural link” (Sebeok 1976: 46; cf. Sebeok 2001). In Sebeok’s view, symptoms can be seen as a marked category (species) of an unmarked one (genus, that is, the sign), thus acquiring a semiotic status per se. For further details, see, in particular, Stano 2020. The face of health in the West and the East: A semio-cultural analysis 303 and makes itself perceived [...] as an absence of events, a stable and durable state.10 (Fontanille 2005: 37) It is therefore necessary to distinguish between two different dynamics. On the level of existence, the difference between health and disease is reduced to a series of narrative transformations and variations of the figurative balances that put health and illness in open opposition, identifying them as antithetical terms. On the level of experience, this distinction is counteracted by the appearance of a specific “feeling” (i.e. the malaise), which brings out the aspectual and transformative nature of health alongside that of illness, putting them in a tensive correlation rather than in a firm opposition. While requiring to reconsider the definitions and characterizations mentioned above, this view also distinguishes two levels of corporeality that interact in the perception and recognition of health and illness as related processes: the ‘Me-flesh’ (corps-chair), which participates in the transformation of the states of things and is a “composite totality in existence”11 (Fontanille 2005: 40); and the ‘Myself-envelope’ (corps propre), understood as a “coherent unity in experience”12 (Fontanille 2005: 40), that is, the “body-in-construction, the living vehicle of intentionality and interactions support, and therefore, particularly the support of confrontation with otherness” (Fontanille 2013: 41). Moreover, in recent decades another concept has increasingly more been used in relation to health, highlighting its “higher threshold” alongside the “lower threshold” that associates it with disease (Marrone 2005; Sbisà 2005): the idea of well-being, whether real or apparent. This finds a confirmation in the current definition of health in different languages,13 as well as in universally shared descriptions such as the one provided by the World Health Organization, which clearly mentions the overcoming of the differential view: “Health is a state of complete physical, mental and social well-being and not merely the absence 10 “La salute non è un’avventura di tutto riposo, e sembra addirittura che la si possa descrivere con precisione – andando in ciò contro al senso comune – soltanto come una tensione permanente, un processo multipolemico. La malattia invece è quasi sempre esito di un rilascio delle tensioni (la difesa cede, il corpo non reagisce più agli attacchi, le cellule non muoiono più e si mettono a proliferare ecc.). […] Naturalmente, grazie agli equilibri in continuo movimento che riesce a garantire, la salute si manifesta e si fa percepire […] come un’assenza di evento, uno stato stabile e durevole.” 11 “Totalità composita nell’esistenza.” 12 “Unità coerente nell’esperienza.” 13 The Merriam-Webster Online English Dictionary (2020), for instance, defines health as “the condition of being sound in body, mind, or spirit”, also openly associating it with the idea of well-being: “a condition in which someone or something is thriving or doing well”. The dictionary was last accessed at https://www.merriam-webster.com on 19 February 2021. 304 Simona Stano of disease or infirmity” (WHO 2020: 1).14 Not only is a positive definition adopted in this case, but the concept of health is enlarged, encompassing both the psychological and the social dimension. This further emphasizes a phenomenological conception of the body as “a totality of lived significations” (MerleauPonty 2012[1945]: 155), that is to say, both an object in the world and a subject experiencing the world, a part of reality and a means of conjunction with it – a real “threshold of sense” (Stano 2019: 149). In the following paragraphs we will analyse how such conceptions of health and disease have found expression in the visual representations of the face in the West and the East in an attempt to identify and describe the main effects of the meaning emerging from them. 3. The face of health in the Western collective imaginary Among the first representations of health in the Western collective imaginary special mention should be made of Hygeia (from the Greek ‘Ὑγίεια’, ‘health’), the Greek divinity of physical and spiritual health, cleanliness and hygiene. Two main phases in the cult of this figure are known: at first as an independent goddess; then in connection with the worship of her father Asclepius, the god of medicine who was invoked for the treatment of disease and the restoration of lost health, while Hygeia was mostly related to the prevention of illness. Correspondingly, in plastic arts the goddess was initially depicted alone, as a virgin of majestic stature with a crowned head; then, towards the third century, she took on the form of a graceful girl accompanied by Asclepius himself and/or by the animal with which he was traditionally depicted: the snake.15 In fact, her figure mainly acquired meaning in relation to such elements, thus stressing the difference between prevention and treatment and insisting on a broader conception of health. This did not confer much importance on her face, which mostly responded to the common canon in classical art, without expressive marks of particular relevance. 14 WHO 2020. Basic documents. (49th ed.), available at https://apps.who.int/gb/bd/, was last accessed on 11 December 2020. 15 An extensive literature on the symbolism of the snake exists. With reference to the figure of Asclepius, from which the association with Hygeia seems to have derived, the idea of knowledge is generally highlighted, in addition to the reference to the legend according to which the god of medicine, seeing a snake reviving another with herbs, used them to save Glauco (MurilloGodínez 2010). Furthermore, in Greek mythology, Asclepius exercised his healing powers precisely with the help of a snake, which was able to heal wounds by licking them with its tongue (for further information, see Gutiérrez, Mellado, Saavedra 2004; Rillo 2008, while Hæger 1988 instead insists on the possibility of a reference to the parasite Dracunculus medinensis). The face of health in the West and the East: A semio-cultural analysis 305 In Roman times the veneration of health, known in Latin as Salus (or, less frequently Valetudo) grew further, to the extent that the goddess was depicted on numerous coins of the Empire, as its protector. The Roman Salus was in fact the personification of a broader idea of ‘well-being’, understood as the combination of physical health and prosperity, which concerned both the individual level (as in case of the Greek divinity Hygeia) and the Res publica (i.e. the social sphere). Salus was usually represented seated, with her legs crossed (in the same position generally used for the goddess Securitas, thus further enhancing the link with the social and political dimension) and her elbow lying on the armrest of a throne,16 or more commonly in an upright position.17 She often held a patera (i.e. a deep plate used in religious ceremonies) in her right hand to feed a snake which was generally wrapped around an altar (or the arm of the throne where Salus was sitting, or the rod that she usually held in her left hand), sometimes directing its gaze towards the plate, sometimes towards the same point where the goddess was looking. While the back of coins generally hosted similar images, the front was used for the representation of the face – of the emperor in power or, in some cases, of Salus herself. The ornaments adorning the head and the face of the goddess played a crucial role in these cases, emphasizing the public dimension of health and the values of wealth and power. The internal elements of the face, on the contrary, were not particularly emphasized, as evidenced by the choice of representing it in profile and the recurrence of ideal and neutral traits,18 or at most of symbolical (rather than realistic) portraits of female representatives of the Empire.19 Most examples in classical iconography do not differ much from such representations, even when they extend to other cultural systems (as it happens with Sirona in Celtic mythology) and subsequent reinterpretations (such as in Neoclassical statues and paintings). The face of health, in other terms, mainly 16 See, for instance, the back of the denarii of emperors Nerva (Marcus Cocceius Nerva Caesar Augustus, 96 AD), Marcus Aurelius Antoninus Augustus (161–180 AD), Lucius Septimius Severus Augustus (210 AD), and Caracalla (Marcus Aurelius Antoninus Augustus, 212 AD). 17 See, for instance, the back of the denarii of consul Manius Acilius Glabrio (49 BC), and emperors Caracalla (Marcus Aurelius Antoninus Augustus, 201 AD) and Elagabalus (Caesar Marcus Aurelius Antoninus Augustus, 218–219 AD), as well as the back of the Antoninianii of emperors Philip the Arab (Mark Julius Philip Augustus, 244–245 AD), Marcus Piavonius Victorinus (270–271 AD), Marcus Aurelius Probus (281 AD) and Maximian (Marcus Aurelius Valerius Maximianus Erculius, 290–291 AD). 18 See, for instance, the front of the denarius of the Consul Manius Acilius Glabrio, 49 BC. 19 Such as Livia Drusilla, the third wife of emperor Augustus of Rome, mother of emperor Tiberius, and grandmother of emperor Claudius, on the front of a coin of the Emperor Tiberius Julius Caesar Augustus, 22–23 AD. 306 Simona Stano remains a neutral and “perfect” surface to be adorned with symbolical elements, which are the ones conferring meaning on it. A completely different representation appeared in 1894, when Gustav Klimt was commissioned, together with Franz Matsch, to create large decorative panels for the University of Vienna. Requested to represent philosophy, medicine and jurisprudence, the artist created three large paintings, which caused an uproar among the university establishment and were never displayed to the public, but transferred to the Österreichische Galerie. Later purchased by August Lederer and Koloman Moser, the paintings were relocated to the Immendorf Castle during World War II, where they perished in 1945 due to a fire started by the retreating SS. Focusing on the representation of medicine (Fig. 1, left), it is interesting to notice first of all that, despite the commissioned subject, Hygeia was preferred to Asclepius, insisting on health as a general concept rather than on the process of healing. In fact, the painting is marked by an evident dysphoric characterization. The Greek personification of health stands in front of a vertical tangle of naked bodies composed of pregnant women, children and elderly people, along with skeletons – i.e. allegories of the processes of generation, life, decay and death. Contrary to the iconographic tradition briefly outlined above, in this case Hygeia does not convey any idea of well-being or social welfare: turning her back on humanity, she rather leaves it helpless against a sad fate. The whole group, confined to the narrow space of what can be interpreted as a large uterus – which is connected to another female figure (“mother nature”?), also turning her back on it – is wrapped by a dark veil, in a further negative axiologization. Looking in detail at the figure of Hygeia (Fig. 1, right), other interesting elements can be identified: in this case the snake is located in front of the patera, without taking anything from it. Although framed from below to highlight its majesty, the face of the goddess is marked by a severe expression and particularly emphasized shadows, which recall the blackish halo characterizing the group of figures in front of which she appears. The dysphoric effect thus generated is further enhanced by Hygeia’s gaze, which appears as dejected and absent. What Klimt depicts here is definitely a lost and unattainable health. The goddess of his painting is certainly powerful, but also indifferent to humanity. Therefore, the malaise becomes existential, inevitably leading to illness and death. The face of health in the West and the East: A semio-cultural analysis 307 Figure 1. Gustav Klimt. Medizin. (1987, oil on canvas, lost work). This representation is very interesting, and is in opposition not only to past but also to current discursivizations of health. In fact, in contemporary Western societies the most varied media ranging from wellness and fitness magazines to brochures illustrating anti-aging treatments and weblogs devoted to body care, etc., have contributed to the creation and circulation of countless representations of health, conferring a special role on the face. Despite the peculiarities of each example, such representations seem to respond to the same logic: the “face of health” they depict is always characterized by the (figurative or metaphorical) elimination of a sort of “patina” or “veil” made up of spots, wrinkles and other “signs” marking the facial surface. Differential systems consisting in confronting a ‘before’ and an ‘after’, or a ‘with’ and ‘without’ condition are largely used, thus emphasizing a differential conception of health, even when the emphasis is put on well-being. This results in a progressive movement of desubjectivation and “perfection” of the face, sometimes also leading to its decomposition in sets of lines, points and intersections (recalling some practices of Eastern medicine, with which we will deal in detail below). The “face of health”, in other terms, seems to be irrepresentable as such, at least as a realistic, particularized and “subjectified” face. It is an “unmarked”, and therefore 308 Simona Stano ideal, surface, which can only be represented in a differential way, that is to say, by making reference to its opposite – i.e. the “face of illness”, or at least of malaise (such as in the case of the imperfections brought about by age). The latter, on the contrary, finds expression in a multiplicity of representations, not only in media communications, but also in the medical domain, as we will discuss in the following section. 4. The face of illness in Western medicine Western medicine has historically attributed a fundamental role to the face. As Jean Starobinski (2003: 58) recalled: A solid tradition required to expose human anatomy a capite ad calcem, that is to say, from head to heel. The method is still followed in some contemporary medical schools, as a rule of preliminary clinical description: the face will therefore be the object of primary attention, whatever part of the body is designated by the patient’s complaints. And with good reason: it is on the face that anxiety, suffering and the fundamental gesture of presence to oneself and to the world are inscribed. In a first approximation, it is therefore the face that reveals the state of alertness, numbness, unconsciousness: thus, a first measurement of the severity of the disease becomes possible.20 Hippocrates himself, who is considered the father of Western medicine, underlined the importance of observing the face in a famous passage from his Prognosticon: “He [the physician] should observe thus in acute diseases: first, the countenance of the patient, if it be like those of persons in health, and more so, if like itself, for this is the best of all” (Book II, 110 [1849: 235]; my emphasis. S. S.). This finds correspondence in the concept of facies, that is to say, the appearance of the face characteristic of particular pathological conditions (Dorland 2011), which is still fundamental in the processes of medical diagnosis. Echoing the differential dynamics outlined above, medical treatises distinguish between a facies composita (i.e. the unaltered face of healthy people) and a multiplicity of facies marked in various ways, such as the Hippocratic facies (which was called after Hippocrates 20 “Una solida tradizione prescriveva di esporre l’anatomia umana a capite ad calcem, dalla testa ai talloni. Il metodo è ancora seguito in alcune scuole mediche contemporanee, come regola della descrizione clinica preliminare: il volto sarà dunque oggetto di un’attenzione prioritaria, qualunque sia la parte del corpo designata dai lamenti del malato. E a buon diritto: è sul volto che s’inscrivono l’ansia, la sofferenza e il gesto fondamentale della presenza a sé e al mondo. In prima approssimazione è dunque il volto a rivelare lo stato di vigilanza, il torpore, l’incoscienza: diventa così possibile una prima misurazione della gravità del male.” The face of health in the West and the East: A semio-cultural analysis 309 himself, since he first described it, and refers to a face characterized by a sharp nose, sunken eyes and temples, drawn-in ears and distorted lobes, together with hard, stretched and dry facial skin, and pale or dusky colour), the facies virile (typical of hirsutism and characterized by the presence of hair and beard on female faces), the acromegalic facies (marked by low forehead, eyebrow arches and prominent cheekbones, large nostrils, thick lips, pronounced chin, as well as hypertrophy of the tongue and ears), and several others. The underlying logic, in other words, always seems to be the one we identified for media representations: the marked face of disease opposes the unmarked face of health, which seems in fact to be irrepresentable as such, as evidenced by the absence of iconic exemplifications (at least devoid of differential logics such as ‘before’ vs. ‘after’ comparisons) of the facies composita. Compared to other approaches, however, Western medicine has highlighted the limits of face reading, at least if not accompanied by other types of analysis. It insists on the fact that the anatomical structures that make up the face are varied and can be altered in different ways, emphasizing the role played by factors such as age, sex, race, individual constitution, degree of consciousness, character, and also cultural influences: “[…] the temptation to believe that all affections are inscribed in the face is great, and, in every age, the presumptuous observer has been able to believe to be in possession of a universal semiology. The presumption consists in believing that the face offers adequate and sufficient clues”21 (Starobinski 2003: 59). Yet judgments of people’s health based on their facial appearance remain fundamental in diagnostic and prognostic practices, as well as in everyday life, also resulting in forms of social isolation and stigmatization (Henderson et al. 2016: 1). 5. The face, health and fortune in the East The face plays a crucial role in the Eastern semiosphere. Identified as the most expressive part of the body, the face has received great attention in Eastern – and more specifically, Chinese – physiognomy, based on a resemblance between the corporeal dimension and the universe that is understood not simply as metaphorical, but rather as physical (Wang 2020: 175). A section in The Compendium of Divine Physiognomy (Shenxiang quanbian, 神相全編), “The pandect of the face” (Toumian zonglun, 頭面 總論), for instance, highlights that the face 21 “La tentazione di credere che tutte le affezioni s’inscrivano nel volto è grande, e, in ogni epoca, l’osservatore presuntuoso ha potuto credersi in possesso di una semiologia universale. La presunzione consiste nel credere che la faccia offra indizi adeguati e sufficienti.” 310 Simona Stano […] is in the sublime location among the hundred parts of the body, leading to the divine paths of the five viscera, developing the image of the three realms [of the Book of Changes], and determining the gains and losses of the whole body. (Shenxiang quanbian, juan 3, 1–3, quoted in Wang 2020: 155)22 Furthermore, the face is seen as an index of the five viscera, thus representing a sort of miniature of the inner body. This explains the success of the practice known as 面相, Mian Xiang or Mian Shiang (literally ‘face reading’), which is thought to have been developed in China approximately three millennia ago by Taoist healers and monks23 and is still in vogue, though with considerable differences (Kohn 1986), among practitioners of so-called Traditional Chinese Medicine and various currents in alternative medicine (which nowadays are also practised in some Western countries). Despite the peculiarities of the multiple methods of analysing facial structures and describing divisions of the face in Eastern physiognomy, a correspondence between specific organs or parts of the body (such as, for instance, the liver or heart) and particular areas of the face (for example, the eyebrows or the nose) can be identified. In such a perspective, issues related to the former manifest themselves with alterations of the latter (e.g. rashes, skin discolouration, dryness, etc.), in a logic that, echoing the above-described dynamics, views the disease as a marked term and health as an unmarked one. The aspects considered in Mian Xiang, however, are not limited to the purely physiological dimension, embracing different factors and extending to the realm of divination. A crucial reference, for instance, is the conceptual scheme of Wu Xing, or “Five Elements” – i.e. wood (木, mù), fire (火, huǒ), earth (土, tǔ), metal (金, jīn) and water (水, shuǐ), which are linked to each other by positive (creation–transformation) and negative (destruction–control) cycles. The visual representation of such relationships has given rise to a series of “holistic” maps of the face which, in addition to physical health, involve a wide range of factors, such 22 The section is also present in the earlier Mayi manual (Mayi shenxiang, juan 2, 2), as reported in Wang 2020: 155. 23 As reported by Kohn (1986: 255), standard modern physiognomy goes back to the tenth century and, “being established as a standard tradition, was transmitted auspices of Taoism. As is physiognomy itself, Taoism is concerned with the human body and closely related sciences of divination. However, when in the tenth century physiognomy began to stand on its own feet, its interrelatedness Taoist tradition became less obvious, […]”. In the only work on physiognomy in the Taoist Canon remaining today, the Lingxin jingzhi, the emphasis is placed particularly on the significance of the eyes, ears and the mind which are called the “three lights” of man, and facial symptoms are related to certain periods of time (Kalinowski 1985). The face of health in the West and the East: A semio-cultural analysis 311 as personality, temperament, talents, etc., putting them in direct correlation with specific points of the face. Particular attention has historically also been devoted to the shapes (with ten different facial forms described in the Divine Fortuneteller Ma Yi, cf. Shi, Wu 2016: 72–73), colours [which are considered in details in the Dunhuang (P. 3390), as remarked by Hou (1979), and in the Zhengyifawen xiuzhen yaozhi (DZ 1270; 13b-14a), which has been studied by Strickmann (1985)], and parts of the face [reaching the very articulated systematizations of the Divine Fortuneteller Ma Yi, where eyebrows were classified into 23 types in terms of denseness and sparseness, the inclining angles, continuity; eyes into 33 types in terms of the inclining angles, interocular breadth, eye fold of the upper eyelid, opening height and direction of eyeslits; mouths into 16 types in terms of thickness of lips, breadth of the oral fissure, direction of the oral fissure, closure of the lips; noses into 24 types in terms of shapes of nasal tip, nasal profile, nasal root height, breadth of alae nasi; and so on and do forth (Shi, Wu 2016: 73)], always combining medical aspects with divination (Despeux 2005). The system of the Five Officials (wuguan, 五 官), for instance, identifies the ears as the Officials of Adoption and Audition (caiting guan, 采聽官), the eyebrows as the Officials of Longevity (baoshou guan, 保壽官), the eyes as the Officials of Inspection (jiancha guan, 監察官), the nose as the Official of Judgment and Discernment (shenbian guan, 審辨官), and the mouth as the Official of Income and Expenditure (chuna guan, 出納官), linking them to the representation of good fortune in different phases of a person’s life. At the same time, the Five Officials are believed to mirror the viscera and inner organs, revealing internal disease to the outside: the ears reflect the status of the kidneys, the eyebrows the gallbladder and the heart, while the nose, the lungs, and the mouth the overall system of the five viscera, making internal imbalances and problems manifest on the outside. As remarked by Xing Wang (2020), it is essential to note that, regardless of the specificities of each model, all the different ways of analysing the face in Eastern physiognomy seem to conceive it as […] a comprehensive replica of the universe, with different officials, geographic landscapes, and celestial bodies all positioned on the corresponding facial areas. The face is a microcosm within another microcosm (the whole body). […] Although the body is always transforming through time, larger patterns of its movement are already visible at the beginning of a person’s life, spatialised on the face in accordance with the cosmic order. (Wang 2020: 170) 312 Simona Stano 6. Physiognomy between the East and the West The practice of assessing people’s personality and identity by analysing their body and especially their face is not confined to the Eastern semiosphere. In fact, physiognomy originated in classical antiquity, spreading both in the East and in the West. In Western culture, it developed through different works, including the volume De Physiognomonia by Polemo of Laodicea (2nd century AD), Physiognomonica by Adamantius the Sophist (4th century), and the manuscript de Physiognomonia by an anonymous Latin author (approx. 4th century), as well as the reflections of famous Latin authors such as Juvenal, Suetonius and Pliny the Elder. Moving from earlier classical works to Medieval studies, physiognomy particularly developed its predictive and astrological side, digressing into prophetic folklore and magic. It was then reintegrated into the scientific field, experiencing a period of fervent development in the 16th century, especially with Giovan Battista Della Porta, who focused on how people’s temperament influences their facial appearance and character, followed by Johann Kaspar Lavater, who introduced the idea that physiognomy relates to the specific character traits of individuals, rather than general types, Franz Joseph Gall, who developed the theory of ‘organology’ and the method of ‘cranioscopy’ (later known as ‘phrenology’), and Cesare Lombroso, who proposed physiognomy as a means of detecting criminal tendencies. Subsequently it suffered a progressive lull of interest, becoming a “parascientific discipline”. By contrast, in Eastern culture, physiognomy is still widely accepted and practiced, especially in China, where the relation between facial features and “destiny” has never stopped playing a crucial role. Souphiyeh Samizadeh (2020), for instance, has highlighted its importance in the cosmetic procedures offered nowadays by aesthetic practitioners, as well as in the main treatments requested by patients, pointing out the diffused belief in the influence of facial appearance on various aspects of life, such as luck, career, etc. Apart from these aspects, as mentioned above, facial examination has always been a primary component of the diagnostic method in Traditional Chinese Medicine, serving for both preventive and healing practices. Advancement in computer vision and Artificial Intelligence have also lead to Face Reading Technologies-based applications, which are offering new, individual techniques for health check-up and advice to everyone (see Ding et al. 2019). All these practices reiterate what was observed in the previous examples, making the face appear as a sort of blank, unmarked canvas on which illness, moral deviance, and even the caprices of fate leave their traces or marks. The face of health in the West and the East: A semio-cultural analysis 313 7. Conclusion Although the examples considered above certainly do not exhaust the wide and extremely varied universe of the discursivization of the face of health, they allow drawing interesting observations from the point of view of the processes of meaning-making. Despite the differences between the Western and the Eastern semiosphere, the face of health always appears as a sort of “zero degree” of the face, functioning as a surface for the inscriptions of malaise and disease (and in some cases even of deviance and negative predispositions). Thus, if on the one hand, such a “zero degree” is euphorically characterized by the absence of disease and, more generally, of divergence, on the other hand, contrary to the Barthesian ideal of a not connoted and not ideologized “face-Idea” (Barthes 1957), it paradoxically ends up being reduced to an anonymous “average face” (van Campen 2000;24 Guido et al. 2017; Roelens 2017). It is an “empty” face, deprived of the signs of nonconformity, but with them also of its own singularity and identity (as it becomes evident in the practice of Mian Xiang), in favour of the adaptation to an unmarked ideal that interprets every redundancy in a way that dominates and eliminates it under the guidance of specialized and “scientificized” knowledge. In fact, it should not be forgotten that any face offering itself to recognition is not a given object, but rather a reality continuously re-constructed by symbolic systems (Magli 1995: 21–25), even though it is treated as a surface through which something internal (e.g. a disease, a moral inclination, or a specific predisposition) can be reflected on the outside, and therefore subjected to presumptive approaches (from medical semiotics to physiognomy and divination). It is in this sense that the analysed cases recall the two movements of the abstract machine of ‘faciality’ (visageité) described by Gilles Deleuze and Félix Guattari in Mille Plateaux (1980): on the one hand, a principle that distributes and orders different elements (e.g. points, lines, areas) to constitute a facial unit; on the other hand, a criterion that selects and excludes non-conformities, relegating them to anomaly and deviance (at a physical level, in the case of Western medicine, but also a behavioural one, in the case of Mian Xiang and some practices deriving from Chinese medicine, as well as in the case of physiognomy). This is highly relevant, since it highlights what we could define the “unattainability” of the face of health. As we pointed out in the previous sections, whether we consider the anonymous, or in any case interchangeable, faces of Hygeia or Salus, which are marked by the symbolism of the ornaments they wear rather 24 Van Campen, Crétien 2000. De aantrekkingskracht van gemiddelde gezichten. Psychologie Magazine, March 2000, available at http://www.synesthesie.nl/pub/gezicht.htm, was last accessed on 9 February 2020. 314 Simona Stano than their intrinsic features, or Western medicine’s facies composita without any representation, or even the de-personalized and unnatural representations found in magazines and online communication on well-being, we are always dealing with a generic, anonymous, and “a-significant” face, constructed by difference with respect to the multiple faces which, giving expression to deviance, disease or at least forms of malaise, fully fall within the idea of corps propre, or ‘MyselfEnvelope’, described by Fontanille (2004: 22) “what is constituted in semiosis, what is constructed in the encounter of two levels of language, in the discourse in progress”25 – and which, therefore, is the bearer of identity and its unceasing processes of re-definition. In conclusion, the analysed cases show how, in a sort of “veridictory determinism”, it is only in the symptomaticity of illness (i.e. the marked facies of Western medicine, the “signs” of Chinese facial reading, etc.), or at most of malaise (i.e. the traces left by ageing), and deviance, that truth (Greimas 1966) seems to be able to manifest itself: by “breaking” the machine of faciality and pluralizing the supposed unitary and perfect state of health that the above-described representations promote, symptoms stand out for their diversity and specificity, revealing (unveiling) to the outside internal processes that were inaccessible before. The asymptomatic face of today’s health and well-being, on the contrary, seems to be inevitably destined to fall into a lie (of cosmetic surgery, whether it contrasts physical problems, moral inclinations or negative predispositions, anti-aging measures, photo editing, etc.), or at the most into a secret (of an invisible disease, imperceptible from the outside, which is often all the more dangerous for this very reason). This is a somewhat paradoxical phenomenon, especially if we consider the incredible quantity of images of faces that, from the pages of wellness magazines to the trendiest posts of weblogs and social networks, promise us to “give body” to health, yet in fact all end up contributing to reinforcing of such dynamics. And it is perhaps even more contradictory if we think of a rather common, yet problematic, expression that we generally use to compliment sound people: “You are the picture of health!” 25 “Ce qui se constitue dans la sémiose, ce qui se construit dans la réunion des deux plans du langage, dans le discours en acte.” (My translation, S. S.) The face of health in the West and the East: A semio-cultural analysis 315 References Barthes, Roland 1957. Mythologies. Paris: Seuil. Barthes, Roland 1972. Sémiologie et médicine. In: Bastide, Roger (ed.), Les sciences de la folie. Paris: Mouton, 37–46. https://doi.org/10.1515/9783111558370-002 Deleuze, Gilles; Guattari, Félix 1980. Mille plateaux. Paris: Éditions de Minuit. Despeux, Catherine 2005. From prognosis to diagnosis of illness in Tang China: Comparison of the Dunhuang Manuscript P3390 and medical sources. In: Cullen, Christopher; Lo, Vivienne (eds.), Medieval Chinese Medicine: The Dunhuang Medical Manuscripts. London: Routledge, 31–83. Ding, Xianghua; Jiang, Yanqi; Qin, Xiankang; Chen, Yunan; Zhang, Wenqiang 2019. Reading face, reading health: Exploring face reading technologies for everyday health. Paper No 205. In: Brewster, Stephen; Fitzpatrick, Geraldine; Cox, Anna L.; Kostakos, Vassilis (eds.), CHI ‘19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery. https://doi. org/10.1145/3290605.3300435 Dorland, William Alexander Newman (ed.) 2011. Dorland’s Medical Dictionary. (32nd ed.) Philadelphia: Saunders/Elsevier. Fabbri, Paolo 1995. Abbozzi per una finzione della cura. In: Donghi, Pino; Preta, Lorena (eds.), In principio era la cura. Roma–Bari: Laterza, 27–46. Fontanille, Jacques 2004. Soma et séma. Figures du corps. Paris: Maisonneuve & Larose. Fontanille, Jacques 2005. Il malessere. In: Marrone, Gianfranco (ed.), Il discorso della salute. Verso una sociosemiotica medica. Roma: Meltemi, 35–50. Fontanille, Jacques 2013. Figures of the body and the semiotics of imprint: Semiotic figures of the body in the humanities. Chinese Semiotic Studies 9(1): 37–52. https://doi. org/10.1515/css-2013-0104 Greimas, Algirdas J. 1966. Sémantique structurale. Paris: Larousse. Greimas, Algirdas Julien; Courtés, Joseph 1982[1979]. Semiotics and Language: An Analytical Dictionary. (Crist, Larry; Patte, Daniel, trans.) Bloomington: Indiana University Press. Guido, Laurent; Hennard Dutheil de la Rochère, Martine; Maire, Brigitte; Panese, Francesco; Roelens, Nathalie (eds.), 2017. Visages. Histoires, représentations, créations. Lausanne: BHMS. Gutiérrez, Lenin Vladímir; Mellado Pérez, Adrián; Saavedra, María de los Ángeles 2004. Origen y evolución del símbolo de la medicina. Educación Médica Superior 18(2): 1. Hæger, Knut 1988. The Illustrated History of Surgery. New York: Bell Publishing Company. Henderson, Audrey J.; Holzleitner, Iris J.; Talamas, Sean N.; Perrett, David I. 2016. Perception of health from facial cues. Philosophical Transactions of the Royal Society B: Biological Sciences, 371: 20150380. https://doi.org/10.1098/rstb.2015.0380 Hippocrates 1849[C. 400 BCE]. Prognosticon. In: The Genuine Works of Hippocrates. Vol. I. (Adams, Francis, ed., trans.) London: Sydenham Society. Hou, Ching-lang. 1979. Physiognomie d’après le teint sous la dynastie des Tang. In: Soymié, Michel (ed.), Contributions aux études de Touen-houang. Paris, Genève, 55–70. Kalinowski Marc 1985. La littérature divinatoire dans le Daozang. (Paper presented at the Quatrième Colloque Pluridisciplinaire Franco-Japonais, Paris, October 1985.) 316 Simona Stano Kohn, Livia 1986. A textbook of physiognomy: The tradition of the “Shenxiang quanbian”. Asian Folklore Studies 45(2): 227–258. https://doi.org/10.2307/1178619 Leone, Massimo 2018. The semiotics of the face in the digital era. Perspectives. Journal of RFIEA, the French Network of Institutes of Advanced Studies 17: 27–29. Lotman, Juri M. 1977. Primary and secondary communication-modelling systems. In: Lucid, Daniel P. (ed.), Soviet Semiotics: An Anthology, Baltimore: John Hopkins University Press, 95–98. Magli, Patrizia 1995. Il volto e l’anima. Fisiognomica e passioni. Milano: Bompiani. Marrone, Gianfranco (ed.) 2005. Il discorso della salute. Verso una sociosemiotica medica. Roma: Meltemi. Merleau-Ponty, Maurice 2012[1945]. Phenomenology of Perception. (Landes, Donald A., trans.). London, New York: Routledge. Murillo-Godínez, Guillermo 2010. El símbolo de la medicina: la vara de esculapio (asclepio) o el caduceo de Hermes (Mercurio). Medicina Interna de México 26(6): 608–615. Rillo, Arturo G. 2008. El origen griego del caduceo: Esculapio. Colombia Médica 39(4): 384–388. https://doi.org/10.25100/cm.v39i4.620 Roelens, Nathalie 2017. La hantise du visage chez Roland Barthes et Gilles Deleuze. In: Laurent, Guido; Hennard Dutheil de la Rochère, Martine; Maire, Brigitte; Panese, Francesco; Roelens, Nathalie (eds.), Visages. Histoires, représentations, créations. Lausanne: BHMS, 71–88. Samizadeh, Souphiyeh 2020. Chinese facial physiognomy and modern day aesthetic practice. Journal of Cosmetic Dermatology 19: 161–166. https://doi.org/10.1111/jocd.12980 Sbisà, Marina 2005. Corpo e testo. In: Marrone, Gianfranco (ed.), Il discorso della salute. Verso una sociosemiotica medica. Roma: Meltemi, 69–78. Sebeok, Thomas 1976. Contributions to the Doctrine of Signs. Lisse: Peter de Ridder Press. Sebeok, Thomas 2001. Signs: An Introduction to Semiotics. Toronto: University of Toronto Press. Shi, Bo; Wu, Chao. 2016. Anthropometry in Ancient China – based on the physiognomy book of Divine Fortuneteller Ma Yi. In: Proceedings of the 2nd Annual International Conference on Social Science and Contemporary Humanity Development. Paris: Atlantis Press, 70–75. https://doi.org/10.2991/sschd-16.2016.14 Stano, Simona 2019. La soglia del senso. Il corpo come istanza semiotica. In: Leone, Massimo (ed.), Il programma scientifico della semiotica. Roma: Aracne, 149–162. Stano, Simona 2020. Between semeiotics and semiotics: The body as a signifying text. In: Soto, Marita; Baeza, Federico (eds.), Alteridades, identidades. Actas 14º Congreso Mundial de Semiótica: Trayectorias/Proceedings of the 14th World Congress of the International Association for Semiotic Studies: Trajectories. Vol. 2. Buenos Aires: IASS Publications & Libros de Crítica, 133–140. Starobinski, Jean 2003. Le ragioni del testo. Milano: Bruno Mondadori. Strickmann, Michel. 1985. Ensigellation: A Buddho-Taoist technique of exorcism. (Paper presented at the Quatrième Colloque Pluridisciplinaire Franco-Japonais, Paris, October 1985.) Wang, Xing 2020. Physiognomy in Ming China: Fortune and the Body. Leiden-Boston: Brill. https://doi.org/10.1163/9789004429550 Yap, Joey 2005. Mian Xiang: Discover Face Reading. Kuala Lumpur: JY Books Sdn Bhd. The face of health in the West and the East: A semio-cultural analysis 317 Лицо здоровья на Западе и Востоке: семиокультурный анализ Журналы, брошюры, блоги и другие медиа распространяют множество сообщений, в которых советуют, как достичь или сохранить здоровье и благополучие. В подобных сообщениях преобладает иконическое изображение лица и прослеживается интересная закономерность: «лицо здоровья» кажется недостижимым и обычно изображается через его противоположность – «лицо болезни» или, по крайней мере, недомогания. При этом лицо занимает центральное место в медицине: с древних времен наблюдение за лицом играло ключевую роль в диагностических практиках как в западной медицине (которая рассматривает определенный внешний вид или выражение лица как симптом конкретных заболеваний при описании патологических состояний), так и в восточных профилактических и лечебных практиках (важнейшим из которых является так называемое мянь сян, или «чтение лица», связывающее медицинскую сферу с другими аспектами, такими как личность, таланты и наклонности). Опираясь на семиотический анализ соответствующих тематических исследований, начиная от классической иконографии и заканчивая современной цифровой медиасферой, статья исследует репрезентацию лица здоровья (и болезни) во времени и пространстве, уделяя особое внимание аналогиям и различиям между западной и восточной семиосферой. Tervise nägu idas ja läänes: semiokultuuriline analüüs Ajakirjades, flaieritel, blogides ja mitmesugustes muudes meediakanalites levitatakse lakkamatult sõnumeid, mis annavad meile nõu, kuidas saavutada tervist või heaolu või seda säilitada. Sellistes sõnumites domineerib näo ikooniline kujutamine ning tuleb ilmsiks huvitav nähtus: “tervise nägu” kui selline näib olevat saavutamatu ja seda esitatakse üldjuhul eristuse kaudu, osutades selle vastandile, “haiguse näole” või vähemalt halva enesetunde omale. Tegelikult on nägu meditsiinivaldkonnas kriitilise tähtsusega: iidsest ajast peale on näovaatlustel olnud diagnostilises praktikas oluline osa nii lääne meditsiinis (kus patoloogiliste seisundite kirjeldamisel kasutatakse mõistet facies, mille all mõeldakse eristuvat näoilmet või välimust, mis seostub konkreetse meditsiinilise seisundiga) ning idamaistes profülaktilistes ja ravitehnikates (milles on fundamentaalsel kohal niinimetatud Mian Xiang ehk ‘näolugemine’, mis ühendab meditsiinivaldkonda teiste aspektidega, nagu seda on isiksus, anded ja kalduvused). Artikkel toetub nii klassikalisest ikonograafiast kui ka nüüdisaegsetest digitaalsetest meediamaastikest pärinevate teemakohaste juhtumiuuringute semiootilisele analüüsile ning uurib tervise (ja haiguse) näo kujutamist läbi ruumi ja aja, keskendudes eriti lääne ja ida semiosfääri vahelistele analoogiatele ja erinevustele. Sel eesmärgil toetutakse nii näo kujutamist ja mõistmist käsitlevale kirjandusele kui ka meditsiiniliste märgisüsteemide ja diskursuste uuringutele. 318 Gabriele Marino Sign Systems Studies 49(3/4), 2021, 318–337 Cultures of the (masked) face Gabriele Marino1 Abstract. What we generally regard as ‘the face’ should be semiotically understood not as something given and monolithic, but rather stratified – it is at least threefold: biological (face), physiognomic (expression), perceivable (visage) – and relational as it has to be put within a narrative in order to make sense. The face lies at the centre of a whole semiotic system, the form of life, revolving around the issue of identity (which the face – the visage, to be precise – embodies and still does not resolve). What we may call ‘the natural face’ is not, as common sense would suggest, the precondition of the ‘culturalized’ one (featured with modifications ranging from make-up and proper masks to surgery), but rather just one of the possible semiotic masks a person may choose to wear. At the same time, the mask does not have to be reduced to a deceptive device only (nor to be meant merely as a material object), being in fact at the centre of a more complex axiology. The classic veridictory square articulating the opposition between Being and Seeming may provide a suitable model for the semiotic square of ‘visageness’, so that we would have: Face, Disguise (the place of the mask proper), Fake, Anonymity. Based on these theoretical premises, the paper finally addresses popular music and outlines a provisional map of the pragmatics of the mask (subtractive vs. additive, ritualistic vs. continuous, material vs. virtual, mask as face vs. face as mask), as a suggestion for further studies. Keywords: face; form of life; mask; semiotics; visage In memory of Daniel Dumile aka MF Doom (1971–2020) 1. Introduction: Semiotics of the (threefold) face The face has been studied in a semiotic fashion: semioticians have been studying the face among other things – more often than not, in order to understand other things. Still, there is no such thing as ‘the semiotics of the face’. This is no surprise: 1 Department of Philosophy and Educational Sciences, University of Turin (UniTo), Italy; e-mail: gabriele.marino@unito.it. https://doi.org/10.12697/SSS.2021.49.3-4.04 Cultures of the (masked) face 319 the face is such an immediate semiotic speaker that even semiotics may have fallen into the bias of not focusing on what seems so obvious – being such an immediate semiotic speaker, the face is hardly detachable from human communication as a whole. It is only when the face becomes less an obvious object that its “obtuse” – as Barthes (1970) would call it – meaning unfolds; as, for instance, when it is modified, hidden, denied. What in English we generally regard as ‘the face’ is at least three different things: (1) a biological surface, that makes it possible for (2) the sophisticatedly culturalized expressive mask of the human being to become (3) visible. We have not only the face (Lat. facies), but also the expression (vultus) and the visage (visus); in semiotic literature such a trichotomy is supported, for instance, by Magli (esp. 1995: 9–15; see also Fabbri 1995)2. The notion of the Actant (Greimas, Courtés 1982[1979]: 5–6) deals with the fact that different tokens may play the same role and that the same token may embody different roles, which is the case with what we generically call ‘the face’: a single Actor embodying different Actantial roles. It is through (1) this body part that (2) the emotions codified in the physiognomic knowledge stratified during ages of biological evolution and cultural codification are being expressed, making it possible for (3) the persona to surface in the conjunction between the collective (our cultural grids) and the individual (each single subjectivity). In terms of Peirce’s philosophy of the sign (CP 2.87), the singularity manifested would stand as the Firstness of the Representamen (visus), the material datum as the Secondness of the Object (facies) and the mechanism that links perceptions and expressions into a crystallized – but still customizable – code as the Thirdness of the Interpretant (vultus); through the first, which may, but need not, coincide with the second, we reconnect to and reconstruct the third one. The primacy of the face in the definition of ‘the other’ as a Subject has been posited by psychoanalysis (‘the mirror stage’ in Lacan 1949), researched by experimental psychology (the life-long studies of Ekman) and extensively problematized by anthropology (for a semiophile overview, see Affergan 2005[2003])3. A famous passage in Jakobson (1960: 377) shows, in a nutshell, how the face is never given, being in fact subject to complex (trans)cultural negotiation: A missionary blamed his African flock for walking around with no clothes on. ‘And what about yourself?’ they pointed to his visage, ‘are not you, too, somewhere naked?’. ‘Well but that is my face’. ‘Yet in us’ retorted the natives, ‘everywhere it is face’. 2 Needless to say, in common language the three terms are generally employed as synonyms. Ekman (1978) proposed an early (para)semiotic – in the Peircean, Sebeokian sense – approach to face. Levinas (1961) considers what he calls ‘the visage’ so meaningful to mankind precisely due to its capability to convey the ‘presence of the other’ through what he calls ‘the expression’. 3 320 Gabriele Marino We consider meaningful anything equipped with what we recognize as a face; in turn, on the one hand, we assign a face to what we want to consider meaningful and, on the other hand, the face is so meaningful to us that we assign it even to meaningless things (which is the case in pareidolia). This process of facemaking, rooted in the gestaltic need to find familiar signs in reality, has been encoded by means of language, thus trespassing from the biological into the cultural, to the extent that we ended up living in a ‘facial society’ (Treusch-Dieter, Macho 1996); the proliferation of face signs in digital communication such as emoticons, emojis, selfies, filters is just one of the many corollaries4. To sum it up: anthropomorphization cannot help coming in via facialization. The synecdoche (a part for the whole) of the body, the metonym (the concrete for the abstract) of the identity, the prosopopoeia (presentification, personification) of the individual, the face is where our sociality is grounded and it still keeps its role – the catalyst of the persona’s signification – when it is subject to modification or occultation. Face cosmetics, disguise, masking, denial are strategies semiotics seizes for to confirm one of its foundational hypotheses: meaning-making never stems from an isolated piece of information, but rather is always relational and narrative5. Be it altered or hidden, the face signifies as it is set within a world of faces (proposed as natural and displayed in plain sight) and related to what it alludes to or stands for. 2. Natural face and semiotic mask A fabric with floral fantasy makes it possible for a – faceless, indeed – Nude by Boubat (Floch 1985) to transition from the state of Nature into Culture, thus becoming a mythological figure by definition (i.e. subsuming the oppositive duality). Likewise, all the parafacial – more precisely, perifacial – devices such as make-up, haircut, beard styles, tattoos, implants, surgery, wearables and accessories (jewellery, headpieces, piercings, proper masks, etc.) make it possible to turn the face into the visage; namely, to display intervention into this body part so as to make it recognizable as culturalized6. What we keep calling simply ‘the face’ is 4 For pareidolia, see Stano 2021; for emoticons and emojis, see Marino 2015 (available as Marino, Gabriele 2015. :-) come emoticon. Più che semplici faccine. Doppiozero Feb. 9. https:// www.doppiozero.com/materiali/ovvioottuso/come-emoticon) and McCulloch 2019: 155–195. For selfies, see Del Marco 2015 (available as Del Marco, Vincenza 2015. Selfie. Forme e pratiche riflessive. il lavoro culturale Nov. 23. https://www.lavoroculturale.org/selfie-forme-e-praticheriflessive/vincenza-del-marco/2015/) and Leone 2018. 5 A recent discussion of this classic structuralist belief can be found in Violi 2018. 6 ‘Para-’ and ‘perifacial’ are calques from Genette 1982; according to the French narratologist, paratexts are the ‘thresholds of the text’ that serve as its complementary annexes, a specific Cultures of the (masked) face 321 never the facies (with all due respect to doctors seeking signs of illness in the proverbial facies hippocratica), but always something profoundly culturalized: the face we talk about and look towards is always a semiotic mask (Fig. 1)7. The natural face is just a constructed ‘degree zero’ layer – to paraphrase Barthes 1947 – of our ‘faciality’ (Fr. visageité; Guattari 1979; Deleuze, Guattari 1980)8, selected among the many possibilities; in the wake of Goffman 1955, we may call this process of selection ‘face-work’. Figure 1. Stratification of the ‘semiotic mask’. category of which are peritexts, marked by spatial continuity with the text. From the Genettean notion of the epitext, i.e. paratext with no spatial continuity with the text, we may also derive the idea of ‘epifaciality’, as is the case with hands contributing to facial expressions (e.g. the facepalm), thus turning the face into the visage; see Marino 2020 (available as Marino, Gabriele 2020. Facepalm. Semiotica epifacciale della frustrazione. In: Leone, Massimo (ed.), Volti virali. Torino: FACETS Digital Press, 67–100. http://www.facets-erc.eu/wp-content/uploads/2020/05/ Massimo-LEONE-2020-Volti-virali-PDF-Editoriale-Compresso.pdf). 7 In a digital context we would talk of the avatar. 8 The notion has prominent political/juridical connotation as it is something of a heteronormative nature (see ‘regime of visibility’ and ‘regime of enunciability’ below), being a critique of the ontological, universalistic philosophy of the face by Levinas. 322 Gabriele Marino In Greimassian terms the mask wants “the floor to be taken” (It. “vuole che una parola sia presa”) by means of Enunciation (Damisch 1979: 789)9; it is a débrayage, a disengagement from the enunciation source – a consequence of any utterance which is being posited as its precondition – by means of articulating the categories of Person, Space and Time in discourse as ‘They’, ‘There’ and ‘Then’. The face is an embrayage, the Meaning effect (Fr. effet de sens)10 of going back to the enunciation source where ‘I’ am saying something ‘Here’ and ‘Now’ (to ‘You’); it is a re-turn, something reconstructable only ex post, identifiable as a marked place within the text where it folds back upon itself11. In other terms, thanks to the fact that we can modify or cover the face, we need to establish the virtuality of another layer of meaning which would lie underneath and, at the same time, is a consequence of all the possible interventions and overlays. The face is not the precondition of the mask12 but, on the contrary, only thanks to the semiotic device of the mask can we eventually conceive a pre-semioticized, unmasked face: as “every visage is a mask” (Damisch 1979: 789), actually the “mask puts the face” on man (Affergan 2005[2003]: 315). The problem of the mask is “the problem of identity in general, in its social and cultural construction, attribution, display and transformation: the process of continuous meaning construction that Peirce called ‘semiosis’” (Pollock 1995: 594). The maskface – and not, simply, the face mask – is the screen onto which the form of life hiding, revolving, arising behind it is being projected. 3. Face and life The notion of ‘form of life’ (Germ. Lebensform) was first employed in a philosophical sense by Wittgenstein (1953, 1969). A form of life would be the condition of Being that makes meaning possible: the rules determining human behaviour should be understood only with reference to a specific form of life, which has determined them in the first place so as they would reflect it. Albeit never thoroughly explained by Wittgenstein, the term has achieved great success in philosophical discourse: it is a notion central to contemporary pragmatism 9 For Enunciation, see Greimas 1975, Greimas, Courtés 1982[1979]: 103–105. See Greimas, Courtés 1982[1979]: 187–188. 11 The analysis of the blank, expressionless ‘neutral mask’ by Marsciani (1990, 1996), considered as a case of ‘missing enunciation’ (It. enunciazione mancante), is consistent with this logic. 12 As, for instance, in Levinas 1961; for a semiotic interpretation of his face philosophy, see Ponzio 2007. 10 Cultures of the (masked) face 323 and, among others, Agamben. The latest developments of structural-generative semiotics recovered the notion as well; in 1991 Greimas suggested it as a topic for a seminar, which never took place, and Fontanille (1993) and other members of the Paris School developed it further (Landowski talked of ‘lifestyle’; Latour, a philosopher close to Greimassian semiotics, of ‘modes of existence’). According to Fontanille (2015: 260) forms of life are the “immediate constituent of the semiosphere”, providing “a coherent deformation of the semiotic function” (Fontanille 2004: 103) “that affects all the levels of the generative trajectory of meaning of any discourse or semiotic universe: ranging from sensory and perceptual schemes to narrative, moral and axiological structures” (Fontanille 2004: 409)13. Influenced by the theories of embodiment14, as well as by the reflections of Greimas (1987) upon the ‘aesthesia’, Fontanille proposes a development of the narrative theory that links it to body and experience: a form of life is not only the consistency – a kind of macro-metaisotopy (isotopy being semantic recurrence) – that holds together a given narrative15; rather, a form of life is the semiotic norm that pervades and, in the first place, determines such narrative through its enactment, its bodily and existential translation into experience. To live according to a given form of life means to conform to a regulative model of action, to pursue a style of strategic behaviour aimed at achieving a result which is consistent with the foundational axiology. The form of life is exactly such a teleological projection from a set of premises to a set of consequences16; it is no coincidence that Fontanille has been focussing so much on this notion, as he had already proposed, along with Zilberberg (Fontanille, Zilberberg 1998), a tensive model of semiotics. Given that the form of life is what holds together all the possible enunciates, textual products or substantializations stemming from the very same semiotic 13 For the semiosphere, see Lotman 1984; for the Generative trajectory, Greimas, Courtés 1982[1979]: 132–134. (Translations into English are mine, unless indicated otherwise; G. M.) 14 The theories of the extended mind, which include embodiment (or enactivism), were, in turn, influenced by phenomenology (Merleau-Ponty) so that they conceive cognition as an activity, rather than a capability, performed not only by the brain but through the whole body as situated in a given environment. The embodiment book which Fontanille references the most is Varela, Thompson, Rosch 1991. 15 A narrative generated in the deepest levels of axiology (set of values, Semiotic square), progressively anthropomorphized and articulated through Semio-narrative (Modalities, Narrative programs and Actantial roles) and Discoursive structures (Thematization and Figurativization as regards the semantic components; Actorialization, Temporalization and Spatialization as regards the syntactic components) and eventually manifested in the surface of text. 16 In these terms, the form of life may be conceived as what Eco (1976[1975]: 289–298) defines as ‘ideology’. 324 Gabriele Marino system, it is the metalevel that ensures intersemiotic translation (the mutual translatability between different matters, modes, media according to Jakobson 1959). The mask stands as a perfect Figural device (the pre-, protofigurative level within the Generative trajectory of meaning that is the place of intersemioticity, as the semantics of what would become the different Object-semiotics are all together and in mutual exchange) capable to condense a whole form of life17. Lotman (2002[1981]) developed such an intuition in a little-known essay of his: in the afterword to a German edition of Gogol’s The Dead Souls he thoroughly explained how the masks made by Russian sculptor Kaplan were capable of translating the verbal images evoked in the novel into visual and haptic forms by developing, thanks to the unique sensitivity of the artist, the very same structural principles manifested through the very same figurative motives. 4. Face and lie Claiming that the semiotics of the face has to be understood as a semiotics of the mask is not a philosophical gimmick as one unfamiliar with semiotic reasoning could think, but rather a methodological operation. Semiotics operates by overturning common sense, by building up general rules on the basis of what common sense generally regards – i.e. dismisses – as exceptions. Semiotics being a system of thought translated into an interdefined metalanguage (and being a whole wherein each single part is defined by its relationships with the others), there is simply no room for exceptions (single elements for which the general rule would not apply). If we know that nature is deeply culturalized and culture deeply naturalized (as Morin has maintained since the 1950s and the ontological turn in anthropology has been discussing since the 1990s), building our semiotic theory of the face on the basis of the common sense dichotomy between nature and culture would prevent us from truly comprehending the axiologies and ideologies of the face spread diachronically and diatopically18. If general semiotics, as posited by the famous, 17 The Figural is a crucial and yet underinvestigated notion in post-Greimassian semiotics. The term had already been employed, with different meanings, by authors such as Auerbach, Merleau-Ponty, Lyotard and it is only briefly addressed in Greimas, Courtés 1986: 91–93 (entry written by Zilberberg); as far as I am concerned, the best sources in point are Jacoviello 2012: 233–242 and Lancioni 2020: 146–152. The Figural may be conceived as a kind of Ur-metaform, where metaform is a “concept ([thinking = seeing]) that results from the linkage of an abstract notion ([thinking]) with a concrete source domain ([seeing])” (Sebeok, Danesi 2000: 196). Marino (2021a, 2021b) elaborates on the relationship between the Figural and the form of life with regard to music. 18 Still, such a dichotomy (natural vs. cultural – i.e. artificial – face) is an important drive Cultures of the (masked) face 325 provocative, tongue-in-cheek – and yet philosophically accurate – definition by Eco (1976[1975]: 7), aims at studying signs by studying “everything which can be used in order to lie”, a semiotics of the face should study everything which can be used in order to “lie the face” (to make a lie of the face), to fake it, to mask it. To paraphrase Eco, we might say that if something cannot be used to hide the face, conversely it cannot be used to display it. The semiotics of the face is not a semiotics of what passes for the face plain and simple, but rather the semiotics of the visus, aimed at reconstructing how the expressive signs of the vultus are being displayed (or not) through the biological facies. The case where the visus conveying the vultus coincides with the facies – i.e. when the biological datum is perceivable and communicates as such – has to be understood as a ‘mask degree zero’. The face has been semiotically studied in connection with the portrait (Calabrese 1981), the self-portrait (Calabrese 2006), digital communication (Leone 2019) and digital pictorialization (Danesi 2016) and, more generally, as the sign congealing a whole aesthetic (Eco 2004, 2007). Barthes (1967) talked of the dress as “the mask of body” and compared the photographic portrait to a semiotic “death mask” (Barthes 1980). The semiotic approach reduces the face to the mask – as we have seen – and the mask to a deceptive device: semiotic theatrology (De Marinis 1982), aesthetics (Marin 1993), physiognomy (Eco 1984; Magli 1995)19, cosmetology (Magli 2013) and the semiotic analysis of literary ekphrasis (Magli 2016) have focused on the face mainly to unmask it, to understand when, where and how it would be faking us20. In Greimas and Courtés (1982[1979]: 67–68) the mask is addressed only twice: in connection with the Deceiver (the trickster of mythology, who often wears it) and in connection with the moment of the Qualifying test when the Sender hides “under the mask of the Adversary”. A reductionist perspective of this kind (mask = deception), profoundly influenced by the episteme of the Classical Age (Damisch 1979: 788), would be best suitable for handling only texts, and not entire forms of life, as we can definitely identify a strategy of some sort, but we would have trouble judging whether and according to which level of semiosis a mask would ‘tell the truth’ or ‘be authentic’ (or not) in both perceptual and cultural phenomena. The hypothesis of the ‘uncanny valley’ by Mori (see Surace 2021), that humanoids imperfectly resembling human features would provoke a feeling of eerie familiarity, lies on the frontier between the natural and the artificial: we find a simulacrum, a mask (humanoid), where we would expect to find a face (human being). 19 The text by Eco is the reprint of his Preface to the book Il Lavater portatile (con i disegni di Franco Testa), an anastatic edition, issued by the publisher Moizzi in Milan, of Johann Kaspar Lavater’s physiognomic treatise from 1788. 20 Reviews of the semiotic approaches to the mask may be found in Proschan 1983, Leone 2020, Gramigna 2021. 326 Gabriele Marino as regards the physical person who would be wearing it21. My proposal is to maintain the semiotic primacy of the mask over the face, to broaden the category of mask (not merely a physical object but rather a semiotic strategy) and not to reduce the mask to only one among its possible valorizations and uses. The classic Veridictory square generated from the opposition between Being and Seeming (Greimas, Courtés 1982[1979]: 369) may be fruitfully applied in order to identify the axiologies of ‘visageness’ (see Fig. 2): Figure 2. The semiotic square of ‘visageness’.  Truth  Face (transparency, mask degree zero, personification);  Lie  Disguise (opacity, mask as face, impersonation);  Falsehood  Fake (non-transparency, face as mask, unsuccessful personification);  Secret  Anonymity (non-opacity, no face, depersonification)22. 21 Needless to say, the relationship between face and mask as regards the issue of identity is a classic philosophical theme, especially in the 20th century, when the idea of a fragmented identity prominently emerges; e.g. according to Gurisatti (2006: 222), Nietzsche’s Zarathustra would programmatically be nothing more than a “shallow dance of masks-without-face, formswithout-content, appearances-without-essence”. 22 The opposition between depersonification (It. incognito, annullamento dell’identità) and impersonation (travestimento, sostituzione di identità) is drawn from Damisch 1979: 776. According to Sbisà (2002: 14), “the definition of Falsehood is controversial, but we can explain it by referring to a Sender that judges a statement produced by the Subject as ‘It is not this way, Cultures of the (masked) face 327 We can think of a semiotics of the face focusing on the very face in itself – for instance, a semiotics of its representation (which is the semiotics of the portrait). Yet when we aim at building a semiotics of the face within the framework of the semiotics of culture (dealing with the face within a given semiosphere, as a cultural unit of a given encyclopaedia23), we cannot consider it only as a representational, visual and haptic token24. When we meet someone new and get to know their name, still the only identity we are told about is the anagraphic one. Likewise, when we see the face of someone new, we do not know the person to whom it belongs (who is behind it and so before us). In other terms, the face per se means nothing, reveals no mystery, unless it turns into a visage, the semiotic mask: in order to make sense, it has to become a sign, to stand “to somebody for something in some respect or capacity” (CP 2.228). Otherwise it would merely represent the possible starting point, supposedly of the indexical kind, for making inferences rooted in the biological; let us think of the phantasmagoria prompted by Lombroso, who founded his criminology by grounding it in positivistic, yet pseudoscientific determinism: a true conspiracy of bodily signs. We have, in turn, to question ourselves: are we immune to such conspiratorial temptation? 5. Facets of the mask The face can be dealt with according to different ‘regimes of visibility’25, different degrees of transparent immediacy or opacity26: the former dimension deals with the possibility of recognition (i.e. positive comparison between the new token and the known type), so that we would be allowed to reconnect the perceivable – the face or its delegate27 – to a biological datum and anagraphic name; the latter deals with the impossibility of accomplishing such a task. We can be shown a perceivable and it does not seem this way’, thus sanctioning its evident falsehood” (Sbisà, Marina 2002. La semiotica narrativa di A.J. Greimas. Concetti principali e istruzioni per l’uso was retrieved from https://www2.units.it/sbisama/it/didattica/semiodisp_2.PDF). Disguise/impersonation may be conceived also as ‘mimicry’ (Caillois 1958). 23 For the encyclopaedia, see Eco 1986[1984]. 24 Outside semiotics, a comprehensive approach of this kind is being pursued by Belting (2013). 25 A notion proposed by Foucault, with regard to power, and developed in the texts by Deleuze (1986) interpreting Foucault. 26 Opposite terms proposed by Bolter and Grusin (1999); Marin (1994) proposes similar categories within a proper semiotic perspective. 27 The term ‘delegate’ is inspired by Latour’s reworking of enunciation theory (see Latour 1998). 328 Gabriele Marino that coincides with the natural face and we can be shown a face culturally modified according to different degrees of intervention, with a special spot assigned to the device that covers the face, hiding it partially or totally28. In a world of over-representation and over-exposition, face occultation (the mask as the face delegate) or face denial (no face nor delegate available) are remarkable semiotic operations that stand for a clear identity statement, opening to a whole set of valorizations; as a matter of fact, the mask is not so much polysemic but polypragmatic: it has different meanings because of different uses. It can be either imposed or chosen (both bandits and hostages wear it, but for different reasons), it can be used to deceive or address; it is the distinctive feature of the criminal, the enemy, the bad guy, the traitor, the spy, the coward, the victim, the (super)hero. In other terms, the semiotic device of the mask can be used subtractively (it denies the biological face), in order not to be recognized (to hide oneself from the outside world), as much as additively (it is superimposed onto the face, serving as its delegate), precisely to be icastically recognized by means of emphasizing some features or creating alternative ones (the mask does cover the face, but in order to amplify/reveal it)29. Just like every sign, and in the respect of being itself a kind of “sign par excellence”, the mask is transparent and opaque at the same time, it is both ‘a thing’ and ‘a sign’; as a thing it may cover what, as a sign, it would reveal, so that there may be no actual relationship between what the mask is hiding and what it is standing for, what it is signifying (Damisch 1979: 785). The mask, whatever it may be, stands as a mythopoietic mechanism that, by questioning common sense identity (it prevents from linking the Enunciator with an Empirical author) and triggering detection (we are prone to seek any clue of the wearer’s form of life within their enunciations), suggests a heterotopy of values. A Practical-referential valorization is opposed to a Utopian-existential one; with the caveat that the former is easily at risk of being reconducted to the latter (in order 28 Here we may suggest an opposition between bandanas (covering the nose and the mouth) and masks (leaving the nose and the mouth uncovered). While bandits use the former type to disguise their identities, the latter type is “designed less to disguise than to signal that disguising is taking place, a semiotic process that is effective only to the extent that it indexes the culturally conventional sign of identity: the eyes” (Pollock 1995: 595); let us think of Diego de la Vega/Zorro’s mask and Superman/Clark Kent’s glasses. In the constitutive asymmetry of these two partial masks (covering some parts of the face, while leaving others uncovered), in their capability to disrupt the features of the body in opposition to the natural symmetry of the biological face, we may recall the individualistic élan Simmel (1901) ascribes to all irrationalistic traits. 29 In the wake of Bouissac’s studies on clowns, Ogibenin (1975: 5) focuses on the mask as a metasemiotic instrument capable of generating “isolation and alienation” by means of “the hypertrophying of the physical constants of the human face”. Cultures of the (masked) face 329 to serve as the mouthpiece for an alternative to the dominant axiology, the Subject disguises themselves so as to be unrecognizable). Even though its iconic status, stricto sensu, has been questioned to the benefit of a more systemic (i.e. not only visual), performance-oriented approach (Salvatore 2015, 2018), the history of popular music is definitely an iconography of famous faces at display, where cult figures wearing masks also exist (‘masked musicians’ is literally a thing in our shared encyclopaedia)30. Whereas the study of masks in popular music is little more than a footnote – three lines in all – within what we may call general maskology (Bell 2010: 1), the semiotics thereof is auroral. Reyes (2021) analysed the 542 covers of the records published in 1988 and in 2018, according to Wikipedia, and found that 32 of them display an ‘occluded face’; by blending an automatic-quantitative approach, in the framework of Manovich’s cultural analytics, and hand-made typologization, in the framework of a qualitative research consistent with semiotic analysis, he distinguished between four types of facial obstruction (due to a mask, a visual effect, the pose of the body, the framing of the image) and identified two main axes of visual strategy (from a maximum of face shown to a maximum of face hidden, from a maximum of distorted face to a maximum of sharp face). As a matter of fact, popular music represents a promising field of research in the framework of a semiotics of the face; hence, the following considerations include cues for further study in this perspective. The mask has its own aspectuality – it may be worn in coincidence with the performance (punctual) or it can be employed continuously, as offstage as onstage (durative): in the former case the mask, a prop contributing to showmanship, is generally employed to define characters/alter egos and mark the performance as a ritual (a prototype in pop-rock music may be Arthur Brown’s face painting and burning helmet from 1968); in the latter case it is literally the metonym of a form of life which is being pursued all the way (as in the case of the French electronic duo Daft Punk with their signature robot-like helmets). An extreme outcome of this totalizing logic is what we may call impersonation: a kind of ‘self-Doppelgänger-ism’ (or ‘Andy Kaufman effect’) where the performer is not merely performing, not just playing a part, but rather living it, without breaking character (a notable example 30 See: Masked Musicians. Wikipedia. https://en.wikipedia.org/wiki/Category:Masked_musicians (category page created in 2011). The Internet is filled with popular lists and polls about masked, disguised and/or anonymous musicians, but proper journalistic inquiries and scholarly articles are sporadic. An exception is Reynolds 2021, written after the announcement of the death of the American masked rapper and producer Daniel Dumile aka MF Doom (Reynolds, Simon 2021. Mask Up. Inside pop and rock history’s obsession with disguise. Tidal. com. Jan. 14; the source was retrieved from https://tidal.com/magazine/article/pop-maskhistory/1-76368). 330 Gabriele Marino in popular music may be Watkin Tudor Jones aka Ninja’s South African ‘zef ’ hip hop group Die Antwoord). The mask can be a material object or it can be immaterial. The latter is the case with fictional performers who embody the musical Enunciator (most notably, socalled virtual bands, from the Chipmunks or Gorillaz cartoons to vocaloid Hatsune Miku): the mask is not a device that hides a face – it is a mask literally applied to no one – but rather the only face we are being shown (which reinforces our theory of the semiotics of the face as, first and foremost, a semiotics of the mask); in this case the mask is the face31. The flipside is when the face becomes nothing more than a mask; we know the performers – we know their face – and still we do not know them at all. As a matter of fact, some figures happen to be so much ‘wrapped in legend’ that their oeuvre does overcome in accountability their personal story, to which we are denied access (from the “inventor of improvisation” Buddy Bolden, a kind of Homer for jazz music, to the lugubrious singer-songwriter Jandek). The face may be also turned into a mask by means of its multiplication and consequent resemantization (as in the case of the chameleonic English rock star David Bowie) or desemantization (as in the case of the Italian singer Mina, whose face is subject to countless modifications and stylizations, or the English electronic musician Aphex Twin, whose face is prosthetically and digitally transformed into horrific appearance and obsessively reproduced on album covers and in video clips, in a kind of grotesque parody of pop music iconicism). As it is easy to understand, facial discourse – the discourse of the face – may be doubled by the name and voice discourses, as names and voices can be plainly displayed, variously modified and covered or totally hidden in a very similar way as the face can. In designing a coherent form of life, in pursuing a project of life, it is no coincidence that stage names are used, appearances are modified, voices altered. In general terms, in the ‘regime of enunciability’32 chosen by the performer, the name can be articulated in transparency (birth name), as alternative (pseudonym) or competing/coexisting (heteronym) with regard to the anagraphic datum, or it can be totally obliterated (anonymity)33. 31 We may fairly apply this definition also to the cases of “the continuous mask”. Again, a notion proposed by Foucault, with regard to law, and developed by Deleuze (1986). 33 For the semiotics of anonymity, see Thibault 2016 as well as Maani 2018 (available as Maani, Sitti [alias Leone, Massimo] 2018. I giga di Gige. L’impatto dell’anonimato nella comunicazione contemporanea. Rivista Italiana di Filosofia del linguaggio. http://www.rifl.unical.it/index.php/ rifl/article/view/520). The name has been a subject of semiotic inquiry since the prehistory of the discipline (philosophers of Ancient Greece, Hobbes etc.), as the act of naming is perhaps the most powerful semiotic act of all. For a semiotics of the face, understood as a semiotics of the mask (and, therefore, of identity), names are quite a promising topic; especially with regard to the notion of ‘embodied name identity’ (Pilcher 2015). 32 Cultures of the (masked) face 331 5. Conclusion: The face as semiotic conspiracy The face is the human communication palimpsest par excellence and yet the semiotic approach warns us against matching it to identity offhand. When we are shown someone’s face we are not automatically granted access to their identity (as if identity were to be conceived as something given and stable, to be simply uncovered, as if it were like an ID with name, picture and signature). We may still lack all the meaningful relationships interweaving texts and practices. The face, the main identity mark along with the person’s name and voice (the three features composing a kind of identity plexus)34, asks to be set within a discoursive network in order to work semiotically (see Table 1). We may be tempted to stick to the idea of reading the face as part of a somewhat simple, even universal code, the one linking us with the biological world, the one of physiognomic determinism; still, luckily, our semiotic hunger makes us unsatisfied with such codicality, as we chase not only the face but also what is behind it and what is being conveyed through it: what makes a face what the face in general and this one face in particular is to us. We are obsessed by what we do not know, by what is missing: what is beyond the door, off the camera, behind the mask. We put every clue – every sign – we have got under a magnifying glass and we connect them to one another in order to make sense of what we are experiencing, so as to reconstruct what is not here from what is here, an absence through a presence. We are naturally – semiotically – led to jump from one level to another, from the authenticity of artistic expression to the authenticity of experience, from the aesthetic to the existential. We want to find in the textualized authorship (Greimas’ Enunciator, Eco’s Model author, etc.)35 not only the sincerity, the genuineness of the text, but also its truthfulness. Each possible identity token (within the texts, in both mediated and non-mediated presence) becomes a metatext for the other ones, so that the identity narrative may progressively cohere into a persona. 34 They seem to be “privileged signs of identity that embarrass Western culture every time a game of camouflage, of masking or elision tends to weaken, displace, or even undo the identification devices” (Bonelli 2004: 9, quoted in Calefato 2006: 75). In fact, Pilcher (2015: 765) notices that the one linking “names, identity and the body” together is still “a neglected relationship”. 35 See, respectively, Greimas, Courtés 1982[1979]: 105 and Eco 1979: 60–66. 332 Gabriele Marino Table 1. Dimensions at stake in the identity plexus. VISUAL DELEGATE  Semiotic mask (Culturalized face) o Natural face o Modified face Covered/Hidden face (mask) o Denied face (no face, no mask) REGIME OF VISIBILITY  Transparency (possibility to be recognized)  Opacity (impossibility to be recognized)  Hypervisibility/Amplification/Multiplication (will to be recognized) NOMINABLE DELEGATE / REGIME OF ENUNCIABILITY  Birthname  Pseudonym  Heteronym  Anonymity EXTROVERSIVE DIMENSIONS  Metatextual narrative (autobiography)  Mediated presence (audiovisual)  Non-mediated presence (live performance, face to face interaction) “Outside the text, [there is] no salvation” Greimas (1975: 25) ruled, echoing Derrida’s (1967: 227) “There is not outside-text”; and still the text is not enough, as our semiotic élan is centrifugal: in interacting with the texts, we seek the guarantee that, through them, it is not only a textual persona but rather a real person who is speaking to us about their – real or possible – world36. Interpretation becomes investigation: a process of reconstruction, unearthing, uncovering, revelation where we hope everything holds together and eventually makes sense. Again, it is a conspiracy, but, this time, of the semiotic kind. The semiotics thereof shall be directed at comprehending whether, how and to which extent does ‘the face’ plot against itself: whether, how and to which extent, does the visage semiotically works against the face and its expression – turning into the semiotic mask. 36 On the notion of ‘possible world’, see Eco 1978, 1979: 122–173. Cultures of the (masked) face 333 Acknowledgements: This article results from a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No 819649 – FACETS). The author would like to thank Simon Reynolds for the fruitful exchange of views. References Affergan, Francis 2005[2003]. Modelli antropopoietici della Maschera. In: Figure dell’umano. Le rappresentazioni dell’antropologia. Roma: Meltemi, 307–334. Barthes, Roland 1947. Le degré zéro de l’écriture. Combat août: 2. Barthes, Roland 1967. Système de la mode. Paris: Seuil. Barthes, Roland 1970. Le troisième sens: notes de recherches sur quelques photogrammes de S. M. Eisenstein. Cahiers du cinéma 222: 12–19. Barthes, Roland 1980. La chambre claire. Note sur la photographie. Paris: Cahiers du Cinéma-Gallimard-Seuil. Bell, Deborah 2010. Mask Makers and Their Craft: An Illustrated Worldwide Study. Jefferson: McFarland. Belting, Hans 2013. Faces: Eine Geschichte des Gesichts. München: Beck. Bolter, Jay David; Grusin, Richard 1999. Remediation: Understanding New Media. Cambridge: The MIT Press. Bonelli, Elena (ed.) 2004. Segni particolari. L’immagine del viso, l’immaginario del nome proprio. Urbino: QuattroVenti. Caillois, Roger 1958. Les jeux et les hommes. Le masque et le vertige. Paris: Gallimard. Calabrese, Omar 1981. La sintassi della vertigine: sguardi, specchi, ritratti. Versus 29: 3–31. Calabrese, Omar 2006. L’art de l’autoportrait. Paris: Citadelles & Mazenod. Calefato, Patrizia 2006. Il nome, il volto. In: Calefato, Patrizia (ed.), Che nome sei? Nomi, marchi, tag, nick, etichette e altri segni. Roma: Meltemi, 43–77. CP = Peirce, Charles S. 1931–1958. Collected Papers of Charles Sanders Peirce. Cambridge: Harvard University Press. (Vols. 1–6, Hartshorne, Charles; Weiss, Paul, eds.; vols. 7–8, Burks, Arthur Walter, ed.) [In-text references are to CP, followed by volume and paragraph numbers.] Damisch, Hubert 1979. Maschera. (Malvano, Maria Vittoria, trans.) In: Romano, Ruggiero (ed.), Enciclopedia Einaudi. Vol. 8: Labirinto-Memoria. Torino: Einaudi, 776–94. Danesi, Marcel 2016. The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. New York: Bloomsbury Academic. De Marinis, Marco 1982. Semiotica del teatro. L’analisi testuale dello spettacolo. Milano: Bompiani. https://doi.org/10.2307/1772329 Deleuze, Gilles 1986. Foucault. Paris: Minuit. https://doi.org/10.5040/9781350252004 Deleuze, Gilles; Guattari, Félix 1980. Mille plateaux. Capitalisme et Shizophrénie 2. Paris: Minuit. Derrida, Jacques 1967. De la grammatologie. Paris: Minuit. Eco, Umberto (ed.) 1978. Versus 19–20 (Semiotica testuale: mondi possibili e narratività). Eco, Umberto 1976[1975]. A Theory of Semiotics. Bloomington: Indiana University Press. Eco, Umberto 1979. Lector in fabula. La cooperazione interpretativa nei testi narrativi. Milano: Bompiani. 334 Gabriele Marino Eco, Umberto 1984. Il linguaggio del volto. In: Sugli specchi e altri saggi. Milano: Bompiani, 45–54. Eco, Umberto 1986[1984]. Semiotics and the Philosophy of Language. Bloomington: Indiana University Press. Eco, Umberto 2004. Storia della bellezza. Milano: Bompiani. Eco, Umberto 2007. Storia della bruttezza. Milano: Bompiani. Ekman, Paul 1978. Facial signs: Facts, fantasies, and possibilities. In: Sebeok, Thomas Albert (ed.), Sight, Sound, and Sense. Bloomington: Indiana University Press, 124–156. Fabbri, Paolo 1995. Difformità del viso. In: Brusatin, Manlio; Clair, Jean (eds.), Identità e alterità: figure del corpo 1895–1995. La Biennale di Venezia 46, Esposizione internazionale d’arte. Venezia: Marsilio. Floch, Jean-Marie 1985. Un nu de Boubat. Sémiotique poétique et discours mythique en photographie. In: Petites Mythologies de l’Oeil et de l’Esprit: Pour une Semiotique Plastique. Paris, Amsterdam: Hadès-Benjamins, 21–38. https://doi.org/10.1075/as.1 Fontanille, Jacques 1993. Les formes de vie. Présentation. RSSI–Recherches Sémiotiques/ Semiotic Inquiry 13(1/2): 5–12. Fontanille, Jacques 2004. Figure del corpo: per una semiotica dell’impronta. Roma: Meltemi. Fontanille, Jacques 2015. Formes de vie. Liège: Presses universitaires de Liège. https://doi. org/10.4000/books.pulg.2207 Fontanille, Jacques; Zilberberg, Claude 1998. Tension et signification. Liège: Mardaga. Genette, Gérard 1982. Palimpsestes. La littérature au second degré. Paris: Seuil. Goffman, Erving 1955. On face-work: An analysis of ritual elements in social interaction. Psychiatry: Journal for the Study of Interpersonal Processes 18: 213–231. https://doi.org /10.1080/00332747.1955.11023008 Gramigna, Remo 2021. Le forme della maschera: aspetti semiotici della manipolazione del volto e della plasticità dell’apparenza. Lexia 37–38: 121–140. Greimas, Algirdas Julien 1975. L’Énonciation (une posture épistémologique). Significação: 9–25. https://doi.org/10.11606/issn.2316-7114.sig.1974.90115 Greimas, Algirdas Julien 1987. De l’imperfection. Périgueux: Éditions Pierre Fanlac. Greimas, Algirdas Julien; Courtés, Joseph (ed.) 1982[1979]. Semiotics and Language: An Analytical Dictionary. Bloomington: Indiana University Press. Greimas, Algirdas Julien; Courtés, Joseph (ed.) 1986. Sémiotique: Dictionnaire raisonné de la théorie du langage, II. Compléments, débats, propositions. Paris: Hachette. Guattari, Félix 1979. Visagéité signifiante, visagéité diagrammatique. In: L’inconscient machinique: essais de schizo-analyse. Fontenay-sous-Bois: Encres/Recherches, 79–116. Gurisatti, Giovanni 2006. Dizionario fisiognomico. Il volto, le forme, l’espressione. Macerata: Quodlibet. Jacoviello, Stefano 2012. La rivincita di Orfeo. Esperienza estetica e semiotica del discorso musicale. Mimesis: Milano–Udine. Jakobson, Roman 1959. On linguistic aspects of translation. In: Brower, Reuben Arthur (ed.), On Translation. Cambridge: Harvard University Press, 232–239. https://doi. org/10.4159/harvard.9780674731615.c18 Jakobson, Roman 1960. Linguistics and poetics. In: Sebeok, Thomas Albert (ed.), Style in Language. Cambridge: The MIT Press, 350–377. Cultures of the (masked) face 335 Lacan, Jacques 1949. Le Stade du miroir comme formateur de la fonction du Je, telle qu’elle nous est révélée dans l’expérience psychanalytique. Revue française de psychanalyse 13(4): 449–455. Lancioni, Tarcisio 2020. E inseguiremo ancora unicorni. Alterità immaginate e dinamiche culturali. Sesto San Giovanni (MI): Mimesis. Latour, Bruno 1998. Petite philosophie de l’énonciation. In: Basso, Pierluigi; Corrain, Lucia (eds.), Eloquio del senso. Dialoghi semiotici per Paolo Fabbri. Milano: Costa & Nolan, 71–94. Leone, Massimo 2018. Semiotics of the selfie: The glorification of the present. Punctum 4(2): 33–48. https://doi.org/10.18680/hss.2018.0018 Leone, Massimo 2019. The semiotics of the face in digital dating: A research direction. Digital Age in Semiotics & Communication 2: 18–40. https://doi.org/10.33919/ dasc.19.2.2 Leone, Massimo 2020. The semiotics of the medical face mask: East and West. Signs & Media 1: 40–70. https://doi.org/10.1163/25900323-12340004 Levinas, Emmanuel 1961. Totalité et Infini: essai sur l’extériorité. La Haye: Nijhoff. Lotman, Juri 1984. О семиосфере. [On the semiosphere.] Sign Systems Studies 17: 5–23. Lotman, Juri 2002[1981]. Маска в художественном мире Гоголя и маски Анатолия Каплана. [Mask in an artistic world of Gogol, and the masks of Anatoli Kaplan.] Sign Systems Studies 30(2): 695–705. https://doi.org/10.12697/SSS.2002.30.2.20 Magli, Patrizia 1995. Il volto e l’anima. Fisiognomica e passioni. Milano: Rizzoli. Magli, Patrizia 2013. Pitturare il volto. Il trucco, l’arte, la moda. Venezia: Marsilio. Magli, Patrizia 2016. Il volto raccontato. Ritratto e autoritratto in letteratura. Milano: Raffaello Cortina. Marin, Louis 1993. Masque et portrait. Urbino: CISeL-Centro Internazionale di Semiotica e Linguistica (Documents de travail et pré-publications 226, settembre, serie F). Marin, Louis 1994. De la representation. Paris: Gallimard-Seuil. Marino, Gabriele 2021a. The form of life of sanctity in music beyond hagiography: The case of John Coltrane and his “Ascension”. International Journal for the Semiotics of Law. https://doi.org/10.1007/s11196-021-09829-7. Marino, Gabriele 2021b. Il ghigno di Aphex e altre maschere: Volti del transumano in musica. Lexia 37–38: 197–216. Marsciani, Francesco 1990. La maschera neutra. Carte semiotiche 7: 146–160. Marsciani, Francesco 1996. Il volto neutralizzato. In: Fabbri, Paolo (ed.), Su la maschera – Catalogo del XVII Festival internazionale del giallo e del mistero. Milano: Mondadori. McCulloch, Gretchen 2019. Because Internet: Understanding the New Rules of Language. New York: Riverhead Books. Ogibenin, Boris L. 1975. Mask in the light of semiotics: A functional approach. Semiotica 13(1): 1–9. https://doi.org/10.1515/semi.1975.13.1.1 Pilcher, Jane 2015. Names, bodies and identities. Sociology 50(4): 764–779. https://doi. org/10.1177/0038038515582157 Pollock, Donald 1995. Masks and the semiotics of identity. The Journal of the Royal Anthropological Institute 1(3): 581–597. https://doi.org/10.2307/3034576 336 Gabriele Marino Ponzio, Augusto 2007. Differenza tollerata ed eccedenza del volto. In: Fuori luogo. L’esorbitante nella riproduzione dell’identico. Roma: Meltemi, 97–103. Proschan, Frank 1983. The semiotic study of puppets, masks, and performing objects. Semiotica 47(1–4): 3–44. Reyes, Everardo García 2021. Face value: Analyzing and visualizing facial data. Lexia 37–38: 467–483. Salvatore, Gianfranco 2015. Icone in azione. Il Pop e il Rock come drammaturgie. H-ermes. Journal of Communication 4: 201–232. Salvatore, Gianfranco 2018. Il teatro musicale del rock. Avanguardie, frontmen, light-show. Roma: Stampa Alternativa. Sebeok, Thomas Albert; Danesi, Marcel 2000. The Forms of Meaning: Modeling Systems Theory and Semiotics. Berlin: Mouton de Gruyter. https://doi.org/10.1515/ 9783110816143 Simmel, Georg 1901. Die ästhetische Bedeutung des Gesichts. Der Lotse. Hamburgische Wochenschrift für deutsche Kultur 1: 280–284. Stano, Simona 2021. Facing food: Pareidolia, iconism, and meaning. Lexia 37–38: 487–501. Surace, Bruno 2021. Semiotica dell’Uncanny Valley. Lexia 37–38: 359–380. Thibault, Mattia 2016. Trolls, hackers, anons: Conspiracy theories in the peripheries of the web. Lexia 23-24: 387–408. Treusch-Dieter, Gerburg; Macho, Thomas (eds.) 1996. Medium Gesicht: Die faciale Gesellschaft. Berlin: Elefanten. Varela, Francisco J.; Thompson, Evan; Rosch, Eleanor 1991. The Embodied Mind: Cognitive Science and Human Experience. Cambridge: The MIT Press. https://doi.org/10.7551/ mitpress/6730.001.0001 Violi, Patrizia 2018. Dalla nave di Teseo al campanile di Venezia. Una prospettiva semiotica sulla identità. Rivista Italiana di Filosofia del Linguaggio 1. DOI: 10.4396/SFL201904. Wittgenstein, Ludwig 1953. Philosophical Investigations. Oxford: Blackwell. Wittgenstein, Ludwig 1969. On Certainty. New York: Harper. Культуры лица (в масках) То, что мы привыкли считать «лицом», с семиотической точки зрения является не чем-то определенным и монолитным, а имеет, по меньшей мере, тройную природу: биологическое (лицо), физиогномическое (выражение), воспринимаемое (визаж). В то же время лицо имеет относительный характер, так как несет смысл, лишь будучи включенным в повествование. Лицо является центром всей семиотической системы – формы жизни, вращающейся вокруг идентичности, которую лицо, точнее, визаж, воплощает и до сих пор не решает. То, что мы называем «естественным лицом», не является, как можно подумать, предпосылкой для возникновения «культурного лица» (включающего модификации от макияжа и масок до хирургии), а скорее служит одной из возможных семиотических масок, которые может выбрать человек. В то же время маска не должна рассматриваться только как способ обмана (или как материальный объект), в сущности находясь в центре более сложной аксиологии. Классический семиотический квадрат, противопоставляющий «быть» Cultures of the (masked) face 337 и «казаться», может стать основой для семиотического квадрата лица, включающего собственно Лицо, Маскировку (место маски), Подделку и Анонимность. Основываясь на этих теоретических предпосылках, статья обращается к популярной музыке и предлагает примерную карту прагматики маски (вычитание vs прибавление, ритуальное vs непрерывное, материальное vs виртуальное, маска как лицо vs лицо как маска) в качестве основы для дальнейших исследований. (Maskiga) näo kultuurid Seda, mida üldjuhul peetakse ‘näoks’, ei tohiks semiootiliselt mõista millegi antu ega monoliitsena, vaid pigem kihistununa – see on vähemalt kolmekordne: bioloogiline (nägu), füsiognoomiline (ilme), tajutav (pale) – ning suhtumuslikuna, sest sellele mõtte andmiseks tuleb see paigutada narratiivi. Nägu asub terve semiootilise süsteemi, elu vormi, keskel, keereldes identiteediküsimuse ümber (mida nägu – või täpsemalt, pale – küll kehastab, ent millele siiski lahendust ei anna). See, mida me võime nimetada ‘looduslikuks näoks’ ei ole ‘kultuurilise näo’ (mis esineb modifikatsioonidega, mis ulatuvad jumestusest päris maskide ja kirurgimaskini) eeltingimus, nagu argimõistus võiks välja pakkuda, vaid lihtsalt üks võimalikest semiootilistest maskidest, mida inimene võib otsustada kanda. Samal ajal ei pea maski taandama üksnes petmisvahendiks (ega pidama üksnes materiaalse objektina mõelduks), sest tegelikult asub see keerukama aksioloogia keskmes. Klassikaline semiootiline ruut, mis artikuleerib olemuse ja nähtumuse vahelist vastandust, võib sobida “pale-oleku” semiootilise ruudu mudeliks, nii et selles oleks esindatud Nägu, Maskeering (päris maski koht), Võltsing, Anonüümsus. Lähtudes neist teoreetilistest eeldustest pöördutakse artiklis viimaks levimuusika poole ja visandatakse maski pragmaatika provisoorne kaart (lahutav vs liitev, ritualistlik vs kestev, materiaalne vs virtuaalne, mask kui nägu vs nägu kui mask), mis võiks olla aluseks edasistele uuringutele. 338 Remo Gramigna, Cristina Voto Sign Systems Studies 49(3/4), 2021, 338–360 Notes on the semiotics of face recognition Remo Gramigna,1 Cristina Voto2 Remo Gramigna, Cristina Voto Abstract. Perceiving and recognizing others via their faces is of pivotal importance. The ability to perceive others in the environment – to discern between friends and foes, selves and others – as well as to detect and seek to predict their possible moves, plans, and intentions, is a set of skills that has proved to be essential in the evolutionary history of humankind. The aim of this study is to explore the subject of face recognition as a semiotic phenomenon. The scope of this inquiry is limited to face perception by the human species. The human face is analysed on the threshold between biological processes and cultural processes. We argue that the recognition of likenesses has a socio-cultural dimension that should not be overlooked. By drawing on Georg Lichtenberg’s remarks on physiognomy, we discuss the critique of the semiotic bias, the association of ideas, and the mechanism of typification involved in face recognition. Face typification is discussed against the background of face recognition and face identification. We take them as three gradients of meaning that map out a network of relationships concerning different cognitive operations that are at stake when dealing with the recognition of faces. Keywords: face recognition; personal identification; semiotics of the face; semiotics of recognition; face perception 1. Theoretical premises: The perception of human faces3 The fascination with the study of the human face shows its constancy in the history of humanity. Although the face is a visible element of the anatomy and appearance of the human species, it remains a quite enigmatic object and very difficult to fathom. From ancient treatises on physiognomy or physiognomics to the ubiquity 1 Department of Philosophy and Education Sciences, University of Turin, Torino, Italy; e-mail: remo.gramigna@unito.it. 2 Department of Philosophy and Education Sciences, University of Turin, Torino, Italy/ Department of Arts and Culture, Universidad Nacional de Tres de Febrero, Argentina; e-mail: cristina.voto@unito.it. 3 Sections 1, 2 and 3 were written by Remo Gramigna. Section 4 was written by Cristina Voto, except for the first and last paragraphs. Sections 5 and 6 were written in tandem by the authors. https://doi.org/10.12697/SSS.2021.49.3-4.05 Notes on the semiotics of face recognition 339 of automated face detection (Kosinski 2021) much in vogue in today’s “culture of surveillance” (Gates 2011), the face remains a subject of profound interest that cuts across numerous disciplines, from evolutionary biology to artificial intelligence.4 Our major task in this introductory section is to outline those theoretical premises engendered in us by reviewing the recent literature on the perception and recognition of faces. Because the face is often the object of interpretations, semiotics – thought of as the discipline geared upon the interpretation of signs – is one amongst the humanistic disciplines called on to discuss the significance of human faces in all their forms and representations. Hence, the thrust of this study lies in the semiotics of face recognition. As face recognition is a broad area of inquiry, it is important to establish from the outset the limits of the present research. Firstly, this study is limited to face perception by the human species (for reviews on face perception by nonhuman species, see Leopold and Rhodes 2010). Secondly, in what follows, we do not engage with the concept of recognition as a philosophical issue, as did, for instance, Paul Ricoeur (2005), who pointed out twenty-four different ways of defining recognition. Therefore, concepts such as recognition as ‘acknowledgment’, ‘mutual recognition’ or ‘self-recognition’ fall outside the limits of the present inquiry. This study is concerned with face recognition as a phenomenon of perception with a particular interest in the semiotic aspects of this phenomenon. The purpose of the study is to map out a network of neighbouring concepts that pinpoint the ramifications of face recognition for semiotics. The paper revolves around three ways of interpreting the human face: recognition, typification and identification. These three aspects set out the three main sections of the study. In what follows we will map out some ramifications of these phenomena. One of the linchpins of face perception is identity. Indeed, the face is the visible heart of the individual’s identity. Jurgen Ruesch (1959: 171) defines the face as the signifier par excellence, Ray Birdwhistell (1970) as a “message board”, Max Thorek (1946: 1) as an “advertisement to the world”, and Jordan Peterson highlights the face as a “broadcast screen”.5 Indeed, information broadcast through the face is multilayered. A whole host of different stimuli are elicited through the face and perceived by others, who constantly monitor faces in others to decode facial stimuli in terms of cues of various nature and as having different meanings (Ekman 1978). 4 On physiognomy and semiotics, see Eco 1975a, Manetti 1993: 84–87, and Magli 1995. For an account on the face from a semiotic perspective, see Fabbri 1995 and Bouissac 2005 (available as Paul Bouissac What is a trustworthy face? at https://semioticon.com/virtuals/risk/ Trustworthyface.pdf). 5 Online lecture: https://www.youtube.com/watch?v=AVPDAa7hymo was accessed on 2 March, 2021. 340 Remo Gramigna, Cristina Voto Identity, gender, age, skin pigmentation, health, basic emotions, micro-expressions, intentions and much more information is displayed and expressed – either willingly or unwillingly, inferred and decoded – accurately or less accurately, by those who engage in social settings.6 Darwin (1872) held that facial expressions in humans have something in common with other animals. Later thinkers sought to identify a possible grammar of emotions through the study of facial expressions (Le Brun 1992; Damish 1992). Since antiquity, “medical semiotics” (Sebeok 1972) as well as physiognomy identified the human face as a significant subject of interest as witnessed by the long-lasting tradition of Arabic and occidental treatises on physiognomy.7 Indeed, there is plenty of historical evidence showing that signs of the face, so to speak, were treated and interpreted as symptoms of certain diseases, as epitomized in Hippocrates. Ancient physiognomy went even further in claiming that there exists an ability to identify the soul and the heart of an individual based on the shape and the nature of the human body and, especially, the face. This idea is later taken up and epitomized in the theory detailed in Johann Kaspar Lavater’s physiognomy. The physiognomic approach, thus, is based on deductive reasoning for it posits that the internal characteristics of an individual can be deduced from the exterior attributes. As Giovanni Gurisatti (2006: 22) rightly pointed out, […] between external and internal, sign and what it stands for, there is a staticlocal causal relationship (dual, successive), where the symptom, the spasm, the colour, are nothing but the external effect, the externalization of an internal cause, to which they refer according to a stable and rigorously determined hierarchical scheme.8 For this reason, the basis of medical semiotics and physiognomy in interpreting faces and bodily symptoms is a tendency towards “typifying” (Gurisatti 2006: 22). In other words, this branch of medicine sought to identify causes from symptoms by means of a scientific or pseudo-scientific method. As we will see in what 6 The term ‘expression’ has two semantic components: one subjective and intentional and another objective and non-intentional. On this distinction, see Gurisatti 2006: 26–27. 7 On the tradition of Arabic physiognomy, see Mourad 1939; on Mesopotamian physiognomy, see Bottéro 1974. For an overview of the history of physiognomics, see Antonini 1900; Getrevi 1991; Gurisatti 1991, 2006; Magli 1995; Courtine and Horoche 1992 – they all offer valuable bibliographical sources on the subject. 8 “[...] fra esterno e interno, segno e designato corre qui un rapporto statico-locale causale (duale, successivo), dove il sintomo, lo spasmo, il colore altro non sono che l’effetto esteriore, l’esteriorizzazione di una causa interiore, cui essi rinviano secondo uno schema gerarchico stabile e rigorosamente determinato”. All translations from Italian are by Remo Gramigna, unless otherwise noted. Notes on the semiotics of face recognition 341 follows, not only was such a tendency towards typification at the basis of medical semiotics – identifying symptoms on the basis of a type – and some branches of physiognomy, but it also resurfaces today under various guises. Physiognomy never achieved a scientific status and today it has lost its grip. As Carlo Ginzburg (1986: 170) pointed out, “the flaw of physiognomy was to tackle the variety of individuals in the light of preconceived opinions and hasty conjectures: in this way it has so far been impossible to establish a scientific, descriptive physiognomy”.9 It is, however, important to recognize that ancient practices of deciphering and reading faces played a pivotal role in what Ginzburg (1986) termed as the ‘evidential paradigm’ (‘il paradigma indiziario’) and have a connection with the history of semiotics. In this regard, it suffices to mention the inclusion of scholars such as Lavater on physiognomics and Darwin’s study on the face in the history of semiotics (Eco 1984). Whilst it is visible to others, unless it is masked or disguised, the face is invisible to the subject. Indeed, the face has a twofold character: the face as seer and the face as seen. Tim Ingold (2002: 124) writes, As a surface, the face has some very peculiar properties. I can feel my own face, and others can see it. But it remains invisible to me. Where others see my face, I see the world. Thus, the face is a visible appearance, in others’ eyes, of my own subjective presence as an agent of perception. It is, if you will, the look of human being. Thus, the human face is the element of appearance which makes the social bond possible through the responsibility that each individual takes in respect to the outer world, which is the principle of personal identity (Le Breton 2010: 72). Given the centrality that the face takes on in social settings (Goffman 1956, 1967; Edkins 2015) and in the definition of one’s own identity (Belting 2017), it is not surprising to find a tight connection between the perception of the human face, the recognition of faces by others, and the pivotal role that such processes play in human interactions and the perception of the environment. It is worth noting that the twofold dimension intrinsic to the concept of the face is apparent in the etymology of the word,10 where the duplet seeing/being seen permeates the semantic field of the term ‘face’ itself. In the Latin tradition, ‘vultus’ 9 “L’ errore della fisiognomica è stato quello di affrontare la varietà degli individui alla luce di opinioni preconcette e di congetture affrettate: in questo modo è stato finora impossibile fondare una fisiognomica scientifica, descrittiva.” 10 The idea of perceiving and seeing one’s own face is one aspect of the issue and leads to the investigation of the role of mirror and self-reflection in the perception of oneself. Classic studies that deal with this issue from the perspective of semiotics are Eco 2010, Nöth 1990 and Galassi and De Michiel 1997. Also, Baltrušaitis 1981 is very relevant to this debate. 342 Remo Gramigna, Cristina Voto was used to refer to the variable and changing aspect of the face, whilst the word ‘facies’ referred to the fixed and immutable physiognomic traits. It is revealing that Isidore of Seville pointed out that the face (facies) is said to be so only because it makes one person recognizable from another and, therefore, allows the identification of a particular individual amongst the crowd (Piras 2010: 48). Indeed, slaves in Ancient Greece were faceless as the word ‘aproposon’ suggests11 (Agamben 2021). In one etymological explanation of the Latin ‘vultus’, the term is traced back to the root ‘uel-, vedere’. Hence, ‘vultus’ refers both to the ability of the individual to see and to be seen (Piras 2010: 53). This aspect also surfaces in the German word ‘Gesicht’ which encapsulates both meanings. Husserl’s phenomenology states this principle very neatly: one element of the face boils down to its visibility, namely, the element of being seen and being perceived from outside perspectives (Husserl 1965: 432). As Agamben (2021) rightly pointed out: Of course, all living beings show themselves and communicate with each other, but only man makes the face the place of his recognition and his truth, man is the animal that recognizes his own face in the mirror and mirrors and recognizes himself in the face of the other. In this sense, the face is both similitas, similitude and simultas, the being together of men. A faceless man is necessarily alone. This is why the face is the place of politics.12 Given the relevance of the element of appearance – showing and being perceived by others – it goes without saying that the opposite tendency – hiding, dissimulating, masking – must be considered in the management of one’s appearance. Indeed, the reverse mechanism of recognition is masking, for it hampers the possibility of recognizing another by altering the distinctive facial traits of the person who wears a mask (Ogibenin 1975). Here we come full circle through the opposite mechanism that serves as a cornerstone of the management of appearance: the simulation and dissimulation of one’s face.13 11 Agamben, Giorgio 2021. Il volto e la morte. Quodlibet, 3 May, 2021 (https://www.quodlibet. it/una-voce-giorgio-agamben). First published in German as “Wo das Gesicht verschwindet”, Neue Zürcher Zeitung, 29 April 2021; now also in Agamben 2021. 12 Original in Italian: “Certo tutti gli esseri viventi si mostrano e comunicano gli uni agli altri, ma solo l’uomo fa del volto il luogo del suo riconoscimento e della sua verità, l’uomo è l’animale che riconosce il suo volto allo specchio e si specchia e riconosce nel volto dell’altro. Il volto è, in questo senso, tanto la similitas, la somiglianza che la simultas, l’essere insieme degli uomini. Un uomo senza volto è necessariamente solo. Per questo il volto è il luogo della politica.” Agamben 2021 was accessed at https://www.quodlibet.it/giorgio-agamben-il-volto-e-la-morte. on 4 April 2021. 13 The dichotomy of showing one’s face and concealing it is epitomized in the techniques of masking. On this issue, see Gramigna 2021, Leone 2020. Notes on the semiotics of face recognition 343 2. Critique of the semiotic bias and the association of ideas in face recognition The words ‘recognition’ and ‘to recognize’ derive from the Latin ‘recognoscere’. This term is composed of the prefix ‘re-’, which means ‘again’, and ‘cognoscere’, which stands for ‘knowing’. Etymology suggests that to recognize someone or something is an action that entails that the object recognized (be it a human being or a thing) was seen or known by the subject in an earlier phase prior to the time when recognition occurs. In a nutshell, we recognize something known. As Ellis (1975: 409) writes, “recognition memory involves the identification of some previously experienced configuration or event”. One thing emerges from this cursory description of terminology: recognition is bound up with the world of perception, and particularly with the field of vision. This also suggests the idea of a recursive element embedded in the concept of recognizing, which is, literally, knowing again, knowing for the second time. Firstly, a question that emerges immediately is whether there is a relation between memory and recognition and, if so, how can such a relation be qualified. Secondly, another issue concerns the uniqueness of the human face and whether it is processed by a special system as compared to the processing of objects that are not faces (Ellis 1986: 5). Because the field of vision varies according to the distance in human interactions, the perception of the human face in social settings is regulated by the laws of proxemics. It is worth noting that in Edward T. Hall’s typology of distances in humans, the face is perceived by others only at close distances. Outside the range of close distances, indeed, faces become blurred and unrecognizable. For Hall (1959: 115), “intimate distance” (less than 18 inches) – the distance of “love-making and wrestling” – is the distance where “sight (often distorted), olfaction, heat from another person’s body, sound, smell, and the feel of the breath all combine to signal unmistakable involvement with another body” (Hall 1959: 116) and “personal distance” (from one to four feet) is the zone where people can touch each other and the field of vision is very sharp. At the close phase of personal distance, the face of the other indeed becomes very visible: “the planes and roundness of the face are accentuated; the nose projects and the ears recede; the hair of the face, eyelashes, and pores are very visible” (Hall 1959: 119). Whilst the idea of face visibility is present in Hall’s classic account, this aspect was stressed with particular force in Krampen’s revisitation of proxemics (Krampen 1995: 283–284). Before venturing further, it is worth pointing out that some key ideas on face recognition, the cognitive bias, and the prejudices that lurk beneath it, can be gleaned from the debate between Johann Caspar Lavater and his fierce critic, the Göttingen physicist Georg Christoph Lichtenberg. In On Physiognomics: Against 344 Remo Gramigna, Cristina Voto Physiognomists, published in 1777 (second ed. 1778), Lichtenberg delivers a series of sharp and penetrating points of criticism directed at the paradigm of physiognomy as advocated by Lavater.14 As Lichtenberg (1991: 107) himself pointed out, the true intention of his work is an admonition to treat such a theme with prudence because judging a person by the face may lead to gross mistakes and overgeneralizations. This is what today goes under the rubric of ‘first impression’ or ‘face value’ (Todorov 2017). As Lichtenberg remarks: […] it is perfectly coherent with the laws of our thought and our sensibility that, as soon as we look at a man, suddenly a figure absolutely more resemblant than we have ever met comes to our mind. Usually, such a figure directly determines our judgment. We regularly make judgments on the base of the face, and we are regularly mistaken. (Lichtenberg 1991: 126)15 Lichtenberg’s criticisms on physiognomy are numerous and acute and cannot be dismissed in a few words. One such item of criticism concerns the principle of the “association of ideas” that lays at the basis of the mechanism of matching faces to psychological traits based on a set of prior templates and assumptions the individual already has in stock.16 Lichtenberg argues that, in recognizing others, people project onto the face some pre-existent cognitive schemes that play a pivotal role in the perception and recognition of faces. As Lichtenberg writes, “The superficial physiognomist finds a face in every drop of ink and a meaning in every face” (Lichtenberg 1991: 127).17 The author explains how face recognition works in the following terms. Contra the tenets of physiognomy, Lichtenberg argues that the face is unfathomable in its entirety. For this reason and for the purpose of grasping a face, each of us creates an “extract” of it, a face’s template. This extract is obtained on the basis of the 14 It would be very limited and inaccurate to say that Lichtenberg’s polemical notes are levelled against Lavater. It is much more than that. Lichtenberg’s critical remarks, instead, concerned the fashionable trend of physiognomics, superficial and without depth, that was rampant at the time when he lived. So, he was against the trivialization and popularization of physiognomy. 15 “È perfettamente coerente con le leggi del nostro pensiero e della nostra sensibilità che, non appena posiamo lo sguardo su un uomo, ci venga subito in mente la figura a esso in assoluto più somigliante che abbiamo già incontrato. Questa figura, di solito, determina direttamente il nostro giudizio. Regolarmente giudichiamo in base al volto, e regolarmente ci sbagliamo”. 16 For a discussion of the criticisms that Lichtenberg launches at Lavater and to the “epidemics” of physiognomy, see Gurisatti 2006: 91–111, 1991: 24–40. 17 “[...] il fisiognomo superficiale in ogni macchia d’inchiostro trova un volto e in ogni volto un significato”. Notes on the semiotics of face recognition 345 interests, mood, and knowledge of the beholder, who assigns to it a host of different meanings (Lichtenberg 1991: 129). According to Lichtenberg, such a template is systematized so that each observer identifies a face composed of four dots, as depicted in the following image (Fig. 1): Figure 1. Face template according to Lichtenberg (1991: 129). Gurisatti’s comments on Lichtenberg’s principle of the association of ideas are illuminating: Physiognomy would lead to projecting onto the face contingent and predetermined cognitive-perceptive schemes, subjecting the face to a subjective (perhaps unconscious) strategy of codification and recognition. In short, the face would be neither explained nor understood – that is, known – but only re-cognized, that is to say made adequate and empathically homologated to cognitive habits, visual conventions, prejudices of the viewer. (Gurisatti 2006: 24)18 Such a thesis is also in line with the etymology of the term ‘recognoscere’, recalled earlier, which means recognizing something already known. The mechanism Lichtenberg discusses is likely to describe how certain cultural biases operate in guiding the recognition of others’ faces. At this juncture, it is worth recalling Eco, who in Kant and the Platypus elaborated a quite sophisticated theory of recognizing unknown entities – Eco’s case studies being Montezuma and the Aztecs gradually forming the concept of a horse after the first meeting with the Spaniards, as well as the polemic over the zoological classification of the platypus – by matching a pre-existing stock of knowledge in order to grasp and categorize something new or unknown (Eco 2000[1997]). Eco’s account of ‘cognitive type’ and ‘nuclear content’ 18 “La fisiognomica indurrebbe a proiettare sul volto schemi cognitivi-percettivi contingenti e predeterminati, sottoponendolo unilateralmente a una strategia soggettiva (magari inconsapevole) di codificazione e riconoscimento. Il volto, insomma, non sarebbe né spiegato né compreso – ossia conosciuto – bensì soltanto ri-conosciuto, vale a dire adeguato e omologato empaticamente alle abitudini cognitive, alle convenzioni visive, ai pregiudizi di colui che guarda” (Gurisatti 2006: 24). 346 Remo Gramigna, Cristina Voto seems in alignment with the point raised above, although Eco’s purpose is not to make sense of the recognition of the face, but the perception and categorization of a novel and unknown entity. 3. Attending to face-like templates: An inborn disposition or a learned habit? One of the first patterns we perceive as human beings when we are born is the face of another. The ability to perceive others in the environment – to discern between friends and foes, selves and others – as well as to detect and seek to predict their possible moves, plans, and intentions, is a set of skills that proved to be essential in the evolutionary history of humankind (Rhodes et al. 2011). Whether this is an inborn attitude or a learned skill is an open question that has generated much debate. Yet the perception and recognition of others through the face is of paramount importance. There is evidence that the ability to detect the face is a skill that appears at a very early stage after birth (Buiatti et al. 2019). Traditionally, biologists, cognitive scientists, developmental psychologists, and researchers of artificial intelligence have had the most to say on the subject (Bruce 1988; Bruce, Young 2012; Buiatti et al. 2019; Leopold, Rhodes 2010; Reid et al. 2017). As Ernst Gombrich (1956: 103) writes, […] we respond with particular readiness to certain configurations of biological significance for our survival. The recognition of the human face, on this argument, is not wholly learned. It is based on some kind of inborn disposition. Whenever anything remotely face-like enters our field of vision, we are alerted and respond. Although the debate around face perception does not fit squarely in a rigid schematism, for clarity’s sake we can single out two opposite fronts: an innatist and a constructivist one. Whilst the former perspective conceives face perception as an innate disposition, constructivists argue that this is an ability that is learned (for reviews, see Guthrie 1995: 103–121). This issue is still unresolved and there is no general agreement around such matters. Let us be clear that there is no complete agreement among experts in this field of research. Several authors have ventured into the study of the subject reaching inconsistent and divergent conclusions, with different authors offering different opinions. The difference in judgments on the interpretation of this concept is due to the non-homogeneity of the criteria used by the scholars for its elucidation on each occasion. Notes on the semiotics of face recognition 347 4. Face as a type: Cultural filters of an evolutionary trait? Semiotic research that dealt with recognition as a cognitive problem (Eco 1968, 1973, 1997; Volli 1972) suggested that the process of recognition – especially the recognition of ‘iconic signs’ – implies the prior establishment of some cultural norms. Both Eco and Volli note that the phenomenon of recognition is not limited to the cognitive and perceptual dimension and suggest that there is an element of culture to it, which ought not to be overlooked. This could be termed as the culturalist approach to recognition. As Eco (1980[1973]: 55) pointed out, the recognition of iconic signs implies a handful of operations which can be summarized as follows: (1) a given culture must define recognizable objects on the basis of some emergent features or traits of recognition; (2) there must be a convention which establishes that certain graphic traits correspond to some of the traits of recognition and that some of these recognition traits of the object must always be present in the reproduction in order to recognize the object; (3) another convention is necessary to establish the modalities of production of the correspondence between graphic traits and traits of recognition. If we conceive of face recognition in terms of iconic signs, we could perhaps speculate that cultures provide given traits of recognition that would make up the stock of knowledge activated when recognition occurs. Édouard Manet’s painting Music in the Tuileries (1862) shows the unique experience of identifying face-like patterns in an image that depicts unrecognizable and featureless faces in the crowd (Fig. 2): 348 Remo Gramigna, Cristina Voto Figure 2. Édouard Manet. Music in the Tuileries (oil on canvas, 1862). On the canvas, the plastic density of the painting and the gesture of the painter allow the beholder to experience a recognition based on spatial relationships. If too close or too far away from the work of art, the crowd will be blurred or even shapeless, but at the right distance, the viewer begins to recognize a series of visual patterns that result in the socio-historical characteristics of the face of the time. Likewise, Edgar Allan Poe, in his short story “The man of the crowd” (1840), describes the recognition of a human face from the absolute vagueness of anonymity of the crowd: With my brow to the glass, I was thus occupied in scrutinizing the mob, when suddenly there came into view a countenance (that of a decrepit old man, some sixty-five or seventy years of age,) a countenance which at once arrested and absorbed my whole attention, on account of the absolute idiosyncrasy of its expression. Anything even remotely resembling that expression I had never seen before. I well remember that my first thought, upon beholding it, was that Retzch, had he viewed it, would have greatly preferred it to his own pictural incarnations of the fiend. […] It was something even more intense than despair that I then observed upon the countenance of the singular being whom I had watched so pertinaciously. (Poe 1996[1840]: 392) The example extracted from Poe’s literary sensibility shows very poignantly how human faces can be categorized as types. Typification is the possibility of reading Notes on the semiotics of face recognition 349 certain kinds of information from someone’s face based on established and available sociocultural codes. As the Italian semiotician and expert in the field of face studies, Patrizia Magli (1995: 17) has pointed out, The perception of a face is primarily based on its recognition. To recognize a face means to assign it to a class; and, in this respect, the perceptive experience seems already directed by the cultural filter: the filter of culture lies in between the face that is in front of us and our irresistible vocation to interpret.19 We do endorse Magli’s view. Recognition and typification, then, are two phenomena that are, in some respect, interwoven. The constructions of types and their recognition by the observer are two processes that shed light on the sociocultural element in which both the phenomena of perception and recognition are enmeshed. From this perspective, typification is the process by means of which one selects certain aspects from a given token, and due to such selection, one is able to attribute the particulars of a face to the general category of humans as a group through an abstract interpretation. Indeed, typification is a classifying scheme that arises from the concrete practice of ordinary life, made up of interactions and problems to be solved in order to recognize someone. Such typifying schemes produce a structure on the basis of common knowledge that is not homogenous but contextually, and therefore culturally, distributed. Originally, the distinction between type and token within semiotics comes from Charles Sanders Peirce, who introduced it in the essay “Prolegomena to an apology for pragmaticism” (1906). The type–token distinction is the cognitive process that leads to the construction of a model where the type is the class of which all signs are exemplars, and the tokens of a type are the occurrences, namely, the singular replicas that can be categorized in a given class. A type, as for instance a particular anatomical eye shape that can be read and interpreted as belonging to a specific ethnic group, must be actualized in a token, that is, in a singular face in order to be recognized. This face will be a sign of a type and, therefore, of the group to which the type refers. For instance, because of the contextual ideologies that have raged in the construction of nation-states, we are led to recognize the existence of a specific type of faciality according to cultural logic, such as there being no token of Italianity with specific traits other than Caucasian ones. However, the occurrences that crowd the globalized faciality together with the memories and texts of the past 19 “La percezione di un volto si basa innanzitutto sul suo riconoscimento. Riconoscere un volto significa assegnarlo a una classe; e, in questo senso, l’esperienza percettiva sembra già orientata dal filtro culturale: tra il volto che ci sta di fronte e la nostra irresistibile vocazione a interpretare […] si interpone il filtro della cultura.” 350 Remo Gramigna, Cristina Voto make manifest the existence of a whole series of facial occurrences that expand the pertinence of the type. This process can be described as the extraction of types that are interpreted as forming a certain individuality depending on a social token, a strategy that enables the categorization of experience. Nevertheless, typification, as an interpretative activity, can be untied from the idea of an operation based on the established codification and selection of characteristics of a type projected on tokens, and leading it back to a continuous activity of token construction starting from the possibilities of the practical horizon in which they take root based on the pertinence. However, the daily socio-cultural experience can be compared with the facility of typecasting someone on the basis of social facial types that circulate in the cultural iconosphere. As Jessica Helfand (2019: 187) wrote: “To typecast is to color inside the lines, to proceed with expectations [...]. Habit is an unforgiving teacher [...]. What you see and what you respond to is the simulacrum, not the soul; the posture, not the person”. Even today, with mobile applications and filters that lead us to manipulate our facial representations, we are primed to see faces as a territory conquered by the hegemonic-type beauty parameters that conduct stereotyping and bias: ‘whitewashing’, ‘forever young’, etc. We find two poles: the individual and his/her behaviour on the one side, the society and the typification of the behaviour of individuals on the other side. Under the influence of complex historical and social processes, specific forms of behaviour are formed, typifying what is accepted or rejected, what is lawful or forbidden, what is valuable or devoid of value. As indicated by Juri Lotman (2006), a complex semiotic norm of ethical, religious, aesthetic, practical and other kinds enters the cultural construction, against the background of which the identity of human behaviour is constituted. In this sense, Lotman’s semiotic model of culture can be read considering the category of enunciative practice. If the behavioural text is the result of the sedimentation of a series of individual acts, which make up a collective enunciative practice, the variations with which it is reproduced by the individual constitute an individual enunciative practice. The latter can strengthen the behavioural text through the repetition of the same sequence of behaviours, but it can also modify it by constituting a variation and modification of the given cultural patterns. As aptly discussed in Giovanni Gurisatti’s inquiry into the aesthetics of the face throughout history, physiognomy has generally been following two rather opposing tendencies. The first tendency strives towards typification. Typification is predicated upon the idea that the varieties of human appearances can be classified in terms of types. The second perspective conceives of the human face in terms of its uniqueness and individuality (Gurisatti 2006: 248). Likewise, Rudolf Kassner (1997: 40) differentiates between two different perspectives: the first being Notes on the semiotics of face recognition 351 “constructivist”, for it conceives of faces in terms of eyes, noses, mouths, and ears, and it begins with a concept to reach a type. This perspective highlights the form of the face as well as the fixed parts of the human face (Kassner 1997: 58). The second is “differential”, for it focuses on the uniqueness of the expression of the face and is predicated upon individuality and the dynamic aspect of the face. This point is worth pondering. This thesis posits that the physiognomic literature dealing with the human face from Aristotle to Lavater, has dealt with the subject in terms of matching facial features to a given type through a process of codification and typification. In other words, the physiognomic approach, at least up to Lavater, tends to reduce the authenticity and the uniqueness of a face to a given type. The rupture of this paradigm occurs with the “new physiognomic” proposed by Kassner (1997). It is only within a constructivist approach to faces that the hypothesis of interpreting faces as types can come forth. As Kassner sharply writes, “Thus, types live, are real and vital in a rigid, predetermined, already codified world, in a certain sense four-dimensional. Or vice versa: in this rigid, predetermined, finite, eternally repeating world, there may be only types, not individuals” (Kassner 1997: 59).20 5. Face and identity: The role of facial features in personal identification In the previous sections, we have dealt with face recognition and typification, and mentioned face and identity only en passant. We have hinted at the role of face identification, and this issue deserves further qualification. Personal identification is a much larger phenomenon in comparison to identification based solely on facial features. Face identification is not equal to personal identification tout court, although the face plays a pivotal role here. Indeed, as forensic anthropology, forensic medicine and forensic genetics have pointed out, the systems of personal identification are varied and not all the methods of personal identification are based on the identification of the face in order to establish the identity of the subject. However, it is undeniable that there are methods of personal identification that are based on the physiognomic study of the face, which is not surprising, given that the human face is a marker of personal identity. Nevertheless, the identification of a person can occur not only through the physiognomy of an individual but also by means of other identification methods such as fingerprints, the chromatic map of the iris and the DNA. For this reason, 20 “Dunque i tipi vivono, sono reali e vitali, in un mondo rigido, predeterminato, già codificato, in un certo qual modo quadrimensionale. O viceversa: in questo mondo rigido, predeterminato, finito, eternamente ripetentesi, potrebbero esserci solo tipi, non individui.” 352 Remo Gramigna, Cristina Voto it must be specified that personal identification based on the face is only one of the methods of personal identification that embraces a much wider range of types of recognition, which also include automatic techniques for recognizing people. It is, therefore, a very vast and complex area. In this study, we will not deal with biometric recognition. The latter represents a type of automatic recognition from the first intuitions of Alphonse Bertillon, the head of the identification section of the Paris police, who, in the mid-19th century, had guessed that the measure of certain parts of the body – such as the length of the feet or fingers – could be used to identify criminals. To be sure, Bertillon identified eleven parameters. From the 19th century onwards, images became a standard and in 1882, he established the use of the signalment, a photographic system giving images (‘mugshots’) and detailed descriptions of persons being tracked. These signalments all had the same format, a card with a series of standard shots correlated by an equally standardized description. It was the invention of a taxonomic system based on biometric data that gave the face a statistical character. Later, Francis Galton and E. R. Henry’s discovery of fingerprints as an element of individual identification inaugurated the method of the archive and archiving as a way of collecting, storing, and cataloguing data relating to personal identification. In 1893 this method was replaced by fingerprinting and the statistical path was established: identity was ascertained on skin patterns, a body trace that increasingly weakened the power of the representation of the face. That is, faith in the expressivity and truthfulness of the face had diminished, and from the fingerprint we moved on to the recognition of the iris, which marked the passage to the face as a deposit of abstract and numerical information, which is the end of the most recent applications for facial recognition. Since Bertillon, the history of biometrics has made enormous advancements and biometric recognition has become one of the most advanced types of recognition, which is not without risks for the privacy of people, however (Gates 2011). Let us leave aside biometric identification and let us discuss identification based on facial features. With regard to personal identification based on the physiognomic study of the face, there is an important corollary to consider. First, there are two methods that are based on physiognomic traits: (1) indirect comparison method, and (2) direct comparison method.21 The former technique consists of a comparative method which is based on the comparison of images of an individual’s face that allows determining the identity between two subjects. By comparing multiple images, an “identity judgment” (Viciano, Capasso 2021: 181) is 21 Cf. Luigi Capasso, Identificazione personale fisionomica attraverso immagini, University of Chieti; available at https://docplayer.it/12549821-Identificazione-personale-fisionomicaattraverso-immagini.html. Also relevant is Falco 1923. Notes on the semiotics of face recognition 353 formulated. The direct comparison method, as the previous one, is a comparative method that assesses the image of an unidentified subject against the image of a subject whose identity is known. What needs to be stressed is that in both cases, the identification of identity is based on an analysis and study of photographic material of various sorts. Often, there is no possibility of direct observation of people’s faces or anatomical structures. For this reason, personal identification based on the physiognomic study of the face works especially through images of faces – that is, representations of faces – which become the privileged material for the investigation. These images can be photographs or frames extrapolated from video recordings. This point is worth pondering, for it is by no means a minor detail. From a methodological point of view, as well as from the perspective of epistemology and semiotics, this is a remarkable point because the personal identification process is often based on photographic representations and, therefore, on images of faces. It is our contention that this difference has not been sufficiently stressed. It is one thing to assess the identity of an individual on the basis of the recognition of the face in real-life settings and in face-to-face interactions. The identification of an individual on the basis of photographic evidence is a different type of business. In fact, in current practice, a photograph of the subject is sufficient to identify a subject as the history of the ID card shows. Therefore, we are dealing with photographic representation of the face and its quality and accuracy, which are important factors in assessing personal identity. Indeed, face-images have been used as tracking devices for identification. The history of the identity card has a longer history than the face-image, at least in France and Germany. Its origins date back to the late Middle Ages, and it contained descriptions of the owner. As Lotman (1998: 63–64) pointed out, from the point of view of culturology, the function of proof of the authenticity of the subject was originally performed by the portrait that anticipated the role of photography. From the standpoint of semiotics, it is important to stress that it is not only the likeness of people’s faces to their representations that matters – that is, matching the object to its model – be it a portrait, a photograph, a video and so forth. On the contrary, what is pivotal is that the recognition of likeness has a socio-cultural dimension which should not be overlooked, because recognizing a face as similar to another occurs in a given socio-cultural context. In this regard, Lotman provides examples of photographs that do not resemble the people portrayed, thus presenting a low degree of likeness between them. A caricature or a portrait could have a higher degree of likeness than a “bad photograph”. Nonetheless, due to the cultural function assigned to photography as a marker of authenticity, a photograph will always be preferred as a 354 Remo Gramigna, Cristina Voto means of personal identification over a handwritten sketch or a caricature (Lotman 1998: 65). Lotman thus rightly remarks that it is the “formal ability to be a sign of likeness that fulfills a given conventional function” (Lotman 1998: 65). Identification differs from recognition in one important respect. Identification is a byproduct of history and societal need of identifying specific individuals. There were times in history when people were neither identified nor identifiable. We would conceive of recognition as a quality that precedes identification biologically and historically. 6. Machine face detection and the future of face recognition One important and fast-growing area of research within face perception is the research framework of face recognition by computers (Wechsler et al. 1998), which we will touch on only tangentially. In this study, we are mainly concerned with the way in which humans recognize faces, although machine face recognition is a fast-growing research field that should not be overlooked. Today, the importance of face recognition, including the plethora of the nuanced meanings attached to this semantic field – identifying, classifying, filing, tracking, to mention but a few – has seen a re-emergence due to the widespread use of digital media, the innovations brought along by the increasing use of artificial intelligence of face detection, and technologies of tracking and personal identification. The establishment of advanced machine learning, artificial-intelligence-based algorithms and network-based, automated services have altered the way in which we thought of faces, provided some benefits and opened up new challenges. AI poses some serious questions of ethics and perpetuates social bias (O’ Neil 2017). What in the past was generally thought of as a skill mastered through biological evolution and engrained in the genetic make-up of a species, today has become the hallmark of intelligent systems capable of performing complex operations that involve, to some extent, face recognition processes unheard of only a few years ago. In a nutshell, the way we conceive of faces has changed dramatically and opened up unprecedented scenarios. The panorama we must face today is, therefore, complex and unprecedented. Human beings’ lives have expanded and become digital (Turkle 1995; Windley 2005). The massive availability of big data and of public records – which range from images to geo-localized information – has radically altered the way in which the concept of face recognition has been traditionally conceived. Needless to say, as bodies of flesh and bones leave fingerprints, also digital selves leave many traces behind, which can be decoded, clustered, tracked and filtered for different Notes on the semiotics of face recognition 355 purposes, including identification and recognition. Hence, new challenges lie ahead for the researchers: to unpack and contextualize a phenomenon that has taken on some very nuanced and unprecedented meanings. From Woodrow Wilson Bledsoe’s first tinkering in the 1960s22 to the contemporary pervasiveness of facial detection systems in our daily life, both in the private and the public sphere, passing through the interaction with mobile phones through the border control services, all show the pervasiveness of this phenomenon. During the last twenty years, a considerable number of scholars and artists have focused on the opportunities and limits offered by artificial recognition technologies: the research of Kate Crawford and Trevor Paglen (2021), Kelly Gates (2011), Shoshana Amielle Magnet (2011), among others, together with the works of artists like Hito Steyler, Leonardo Selvaggio and Zach Blas show the relevance of these phenomena. This pervasiveness can be surely understood as symptomatic of the facial society described by Thomas Macho and Gerburg Treusch-Dieter (1996) while, at the same time, it makes visible some continuities and discontinuities between artificial and human recognition systems and processes. The first difference concerns the question of the accessibility of the recognition. On the one hand, artificial vision recognizes human beings through an invisible operation inaccessible to most humans. On the other hand, human recognition, even though extremely complex both neurophysiologically and culturally, is a reciprocal experience within the human species. Except among those affected by prosopagnosia, as human animals, we recognize faces as being seen through our face. In both human and artificial recognition, the result of the process is the production of some kind of knowledge. However, the qualitative dimension of the information generated by artificial processes is under assessment of legitimacy, while in the case of human recognition it can be over-interpreted. An example in this direction can be emotion recognition. We are witnessing an increasing demand for emotion recognition in many fields of security such as government, public and private agencies, biomedicine or therapy (Dutta, Barman 2020). Still, doubts persist about the effectiveness of such a complex task, which is often flawed even in human recognition. These points of contact and distance mark some continuity and discontinuity in recognition processes. Many artificial recognition systems are inspired by human processes. In this regard, a task for the future of face recognition will be to ask to what extent artificial recognition replicates human cognitive processes and how these eventual replications relate to a discourse on the visual. 22 In 1963 Woodrow Wilson Bledsoe published the research entitled A Proposal for a Study to Determine the Feasibility of a Simplified Face Recognition Machine, one of the first studies to investigate the relationship between artificial intelligence and facial image recognition on behalf of the US Central Intelligence Agency. 356 Remo Gramigna, Cristina Voto 7. Conclusion The present study has led towards a semiotics of recognition by unpacking the meaning of the human face under three different, yet complementary, aspects. We have limited our study to three main aspects, namely, recognition, identification and typification of faces, our strategy being to discover a network of relationships of meaning among these three aspects. The face as a recognizable pattern is engrained in the process of semiotic mediation (recognition). Of the three, recognition is the broadest phenomenon, for it includes the recognition of objects that are not limited to the recognition of human faces. This is probably also the oldest form of conceiving of faces that has a direct connection to the biology of face recognition. The face as the focus of personal identity (identification) uses faces as identity markers and is a scientific procedure invented by legal institutions to establish the genuine identity of people. Such a procedure is liable to falsification and opens up to the vast area of fake identities, simulation and fraud. The face thought of as a result of typification, that is, the possibility of reading certain kinds of information from someone’s face based on established and available sociocultural codes, is a semiotic practice that dwells with the extraction of types on the basis of socio-cultural token (typification). Acknowledgements: This reseach is part of a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No 819649 – FACETS). References Agamben, Giorgio 2021. A che punto siamo? L’epidemia come politica. (Nuova edizione accresciuta.) Padova: Quodlibet. Antonini, Giuseppe J. 1900. I precursori di Lombroso. Torino: Fratelli Bocca Edittori. Baltrušaitis, Jurgis 1981. Lo specchio. Rivelazioni, inganni e science-fiction. Milano: Adelphi. Belting, Hans 2017. Face and Mask: A Double History. Oxford: Princeton University Press. Birdwhistell, Ray L. 1970. Kinesics and Context: Essays on Body-Motion Communication. London: Allen Lane – Penguin Press. https://doi.org/10.9783/9780812201284 Bottéro, Jean 1974. Symptomes, signes, écritures en Mésopotamie ancienne. In: Vernant, Jean Pierre et al. Divination et rationalité. (Recherches anthropologiques.) Paris: Seuil, 70–193. Buiatti, Marco; Di Giorgio, Elisa; Piazza, Manuela; Polloni, Carlo; Menna, Giuseppe; Taddei, Fabbrizio; Baldo, Ermanno; Valortigara, Giorgio 2019. Cortical route for facelike pattern processing in human newborns. PNAS. Proceedings of the National Academy of Sciences of the United States of America 116(10): 4625–4630. https://doi.org/10.1073/ pnas.1812419116 Bruce, Vicki 1988. Recognising Faces. New York: Routledge. Notes on the semiotics of face recognition 357 Bruce, Vicki; Young, Andy 2012. Face Perception. New York: Psychology Press. https://doi. org/10.4324/9780203721254 Courtine, Jean-Jacques; Haroche, Claudine 1992. Storia del viso. Esprimere e tacere le emozioni (XVI–XIX). Palermo: Sellerio. Crawford, Kate; Paglen, Trevor 2021. Excavating AI: The politics of images in machine learning training sets. AI & SOCIETY: 1–12. https://doi.org/10.1007/s00146-021-01162-8 Damish, Hubert 1992. L’alfabeto delle maschere. In: Le Brun, Charles. Le figure delle passioni. Conferenze sull’espressione e la fisionomia. (Giuffredi, Maurizio, ed.) Milano: Raffaello Cortina, 102–116. Darwin, Charles 1965[1872]. The Expressions of Emotions in Man and Animals. Chicago: The University of Chicago Press. Dutta, Paramartha; Barman, Asit 2020. Human Emotion Recognition from Face Images. Singapore: Springer Nature. https://doi.org/10.1007/978-981-15-3883-4 Eco, Umberto 1968. La struttura assente. Milano: Bompiani. Eco, Umberto 1975a. Volti di lupi, volti di agnelli. In: Johann Kaspar Lavater. Il Lavater Portatile. Tom. I. Milano: Moizzi Editore. Eco, Umberto 1980[1973]. Segno. Milano: Mondadori. Eco, Umberto 1984. Proposals for a history of semiotics. In: Borbè, Tasso (ed.), Semiotics Unfolding: Proceedings of the Second Congress of the International Association of Semiotic Studies, Vienna, July 1979. Vol. 1. (Approaches to Semiotics 68.) New York, Amsterdam: De Gruyter, 75–89. https://doi.org/10.1515/9783110869897-012 Eco, Umberto 2000[1997]. Kant and the Platypus: Essays on Language and Cognition. London: Vintage. Eco, Umberto 2010. Sugli specchi ed altri saggi. Milano: Bompiani. Edkins, Jenny 2015. Face Politics. New York: Routledge. https://doi.org/10.4324/ 9781315718040 Ellis, Hadyn D. 1975. Recognizing faces. British Journal of Psychology 66(4): 409–426. https://doi.org/10.1111/j.2044-8295.1975.tb01477.x Ellis, Hadyn D. 1986. Introduction: Processes underlying face recognition. In: Bruyer, Raymond (ed.), The Neuropsychology of Face Perception and Facial Expression. New York: Psychology Press, 1–27. Ekman, Paul 1978. Facial signs: Facts, fantasies, and possibilities. In: Sebeok, Thomas A. (ed.), Sight, Sound and Sense. Bloomington: Indiana University Press, 124–156. Fabbri, Paolo 1995. Difformità del viso. In: AA.VV., Identità-Alterità. Figure del corpo 1895/1995. Biennale di Venezia, 46. Esposizione internazionale d’arte. Marsilio: Venezia. Falco, Giuseppe 1923. Identità. Metodo scientifico di segnalamento e identificazione. Roma: Maglione & Strini. Galassi, Romeo; De Michiel, Margherita 1997. Il simbolo e lo specchio. Scritti della scuola semiotica di Mosca-Tartu. Napoli: Edizioni Scientifiche Italiane. Gates, Kelly A. 2011. Our Biometric Future: Facial Recognition Technology and the Culture of Surveillance. New York, London: New York University Press. https://doi.org/10.18574/ nyu/9780814732090.001.0001 Gauthier, Isabel; Tarr, Michael J.; Moyan, Jill; Skudlarski, Pawel 2000. The fusiform “face area” is part of a network that processes faces at the individual level. Journal of Cognitive Neuroscience 1: 495–504. https://doi.org/10.1162/089892900562165 358 Remo Gramigna, Cristina Voto Getrevi, Paolo 1991. Le scritture del volto. Fisiognomica e modelli culturali dal Medioevo ad oggi. Milano: Franco Angeli. Ginzburg, Carlo 1986. Miti emblemi spie. Morfologia e storia. Torino: Einaudi. Goffman, Erving 1956. The Presentation of Self in Everyday Life. Edinburgh: University of Edinburgh, Social Sciences Research Centre. Goffman, Erving 1967. Interaction Ritual: Essays in Face-to-Face Behavior. New York: Doubleday. Gombrich, Ernst 1956. Art and Illusion: A Study in the Psychology of Pictorial Representation. Princeton: Princeton University Press. Gramigna, Remo 2021. Le forme della maschera. Aspetti semiotici della manipolazione del volto e della plasticità dell’apparenza. Lexia 37–38 (Volti Artificiali/Artificial Faces): 121–140. Gurisatti, Giovanni 1991. Introduzione. In: Lavater, Johann Caspar; Lichtenberg, Georg Christoph, Lo specchio dell’anima. Pro e contro la fisiognomica. Un dibattito settecentesco (Gurisatti, Giovanni, ed.). Padova: Il Poligrafo, 11–64. Gurisatti, Giovanni 2006. Dizionario fisiognomico. Il volto, le forme, l’espressione. Macereta: Quodlibet. Guthrie, Stewart Elliot 1995. Faces in the Clouds: A New Theory of Religion. Oxford: Oxford University Press. Hall, Edward T. 1959. The Hidden Dimension. New York: Anchor Books Doubleday. Helfand, Jessica 2019. Face: A Visual Odyssey. Cambridge: The MIT Press. Husserl, Edmund 1965. Idee per una fenomenologia pura e pe una filosofia fenomenologica. Torino: Einaudi. Ingold, Tim 2002. The Perception of the Environment: Essays on Livelihood, Dwelling and Skill. London, New York: Routledge. Kassner, Rudolf 1997. I fondamenti della fisiognomica. Il carattere delle cose. Vicenza: Neri Pozza Editore. Kosinski, Michal 2021. Facial recognition technology can expose political orientation from naturalistic facial images. Sci Rep 11(100). https://doi.org/10.1038/s41598-020-79310-1 Krampen, Martin 1995. Proxemics revisited – Look at the body. In: Tasca, Norma (ed.), Ensaios em Homenagem a/Essays in Honor of Thomas A. Sebeok. Cruziero Semiotico 22–25: 289–288. Lavater, Johann Caspar; Lichtenberg, Georg Christoph 1991. Lo specchio dell’anima. Pro e contro la fisiognomica. Un dibattito settecentesto (Gurisatti, Giovanni, ed., intr.). Padova: Il Poligrafo. Le Breton, David 2010. Antropologia del volto: Frammenti. In: Vinci, Daniele (ed.), Il volto nel pensiero contemporaneo. Trapani: Il Pozzo di Giacobbe, 67–83. Le Brun, Charles 1992. Le figure delle passioni. Conferenze sull’espressione e la fisionomia (Giuffredi, Maurizio, ed.). Milano: Raffaello Cortina. Leone, Massimo 2020. From fingers to faces: Visual semiotics and digital forensics. International Journal for the Semiotics of Law 34: 579–599. https://doi.org/10.1007/ s11196-020-09766-x Leopold, David A.; Rhodes, Gillian 2010. A comparative view of face perception. Journal of Comparative Psychology 124(3): 233–251. https://doi.org/10.1037/a0019460 Lichtenberg, Georg Christoph 1991. Sulla fisiognomica; contro I fisionomi. In: Lavater, Johann Caspar; Lichtenberg, Georg Christoph. Lo specchio dell’anima. Pro e contro Notes on the semiotics of face recognition 359 la fisiognomica. Un dibattito settecentesto. (Gurisatti, Giovanni, ed., intr.) Padova: Il Poligrafo, 97–138. Lotman, Jurij 1998. Il ritratto. In: Lotman, Jurij. Il girotondo delle muse: Saggi di semiotica delle arti e della rappresentazione. (Burini, Silvia, ed.) Bergamo: Moretti & Vitali, 63–96. Lotman, Jurij. 2006. Tesi per una semiotica della cultura. (Sedda, Franciscu, ed.) Roma: Meltemi. Macho, Thomas; Treusch-Dieter, Gerburg 1996. Medium Gesicht. Die faciale Gesellschaft. Berlin: Elefanten-Press. Magli, Patrizia 1995. Il volto e l’anima. Fisiognomica e passioni. Milano: Edizione CDE. Magnet, Soshana Amielle 2011. When Biometrics Fail: Gender, Race, and the Technology of Identity. Durham: Duke University Press. https://doi.org/10.1515/9780822394822 Manetti, Giovanni 1993. Theories of the Sign in Classical Antiquity. Bloomington: Indiana University Press. Mourad, Youseff 1939. La physiognomonie arab et le Kitāb al-Firāsa. Paris: Librairie Orientaliste Paul Geurhner. Nöth, Winfried 1990. Alice’s adventures in semiosis. Working Papers and Pre-publications 195, serie B. Urbino: Centro Internazionale di Semiotica e Linguistica, Università di Urbino. Ogibenin, Boris L. 1975. Masks in the light of semiotics: A functional approach. Semiotica 13(1): 1–10. https://doi.org/10.1515/semi.1975.13.1.1 O’Neil, Cathy 2017. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. New York: Broadway Books. Piras, Antonio 2010. Le parole del volto. Spigolature storico-linguistiche ai margini di un campo semantico. In: Vinci, Daniele (ed.), Il volto nel pensiero contemporaneo. Trapani: Il Pozzo di Giacobbe: 47–63. Peirce, Charles S. 1906. Prolegomena to an apology for pragmaticism. The Monist 16(4): 492–546. https://doi.org/10.5840/monist190616436 Poe, Edgar Allan. 1996[1840]. The man of the crowd. In: Poe, Edgar Allan. Poetry, Tales and Selected Essays. New York: The Library of America, 388–396. Reid, Vincent M.; Dunn Kristy; Young, Robert J.; Amu, Johnson; Donovan, Tim; Reissland, Nadja 2017. The human fetus preferentially engages with face-like visual stimuli. Current Biology 27: 1825–1828. https://doi.org/10.1016/j.cub.2017.05.044 Ricoeur, Paul 2005. The Course of Recognition. Cambridge: Harvard University Press. Rhodes, Gillian; Calder, Andy; Johnson, Mark; Haxby, James V. 2011. Oxford Handbook of Face Perception. Oxford: Oxford University Press. https://doi.org/10.1093/ oxfordhb/9780199559053.001.0001 Ruesch Jurgen; Kees, Weldon 1959. Nonverbal Communication: Notes on the Visual Perception of Human Relations. Berkeley, Los Angeles: University of California Press. Sebeok, Thomas A. 1972. Semiotica e affini. Versus. Quaderni di studi semiotici 3: 1–11. Thorek, Max 1946. The Face in Health and Disease. Philadelphia: F. A. Davis Company. Todorov, Alexander 2017. Face Value: The Irresistible Influence of First Impressions. Princeton, New Jersey: Princeton University Press. https://doi.org/10.1515/ 9781400885725 Turkle, Sherry 1995. Ghosts in the machine. The Sciences 35(6): 36–39. https://doi. org/10.1002/j.2326-1951.1995.tb03214.x Viciano, Joan; Capasso, Luigi 2021. Contributions of facial imaging to human identification in forensic cases: A retrospective study (1996–2019) at the Operative Unit of 360 Remo Gramigna, Cristina Voto Anthropology of the ‘G. d’Annunzio’ University of Chieti-Pescara (Italy). European Journal of Anatomy 25(S2): 179-192. Volli, Ugo 1972. Some possible developments of the concept of iconism. Versus. Quaderni di Studi Semiotici 3: 14–30. Wechsler, Harry; Phillips, Jonathon P.; Bruce, Vicki; Fogelman Soulié, Françoise; Huang, Thomas S. 1998. Face Recognition: Theory and Practice. Berlin, Heidelberg: Springer Verlag. https://doi.org/10.1007/978-3-642-72201-1 Windley, Phillip J. 2005. Digital Identity. O’Reilly Media. Заметки о семиотике распознавания лиц Способность воспринимать и опознавать других по лицам имеет ключевое значение. Умение различать друзей и врагов, своих и чужих, а также обнаруживать и прогнозировать их возможные шаги, планы и намерения – навыки, доказавшие свою важность в эволюционной истории человечества. Целью этого исследования является изучение проблемы распознавания лиц как семиотического явления. Охват работы ограничен распознаванием лиц человеком. Человеческое лицо анализируется в контексте пересечения биологических и культурных процессов. Мы утверждаем, что способность замечать сходство имеет социально-культурное измерение, которое не следует упускать из виду. Опираясь на замечания Георга Лихтенберга о физиогномии, мы обсуждаем критику семиотической предвзятости, ассоциацию идей и механизм типизации, связанный с распознаванием лиц. Типизация лиц рассматривается в контексте распознавания и идентификации лиц. Мы предлагаем считать их тремя градиентами значения, составляющими карту отношений между различными когнитивными операциями, участвующими в распознавании лиц. Tähelepanekuid nägude äratundmise semiootika kohta Teiste tajumine ja ära tundmine nende nägude põhjal on otsustava tähtsusega. Suutlikkus tajuda teisi keskkonnas viibijaid – teha vahet sõprade ja vaenlaste, omade ja võõrast vahel – ning teha kindlaks ja püüda ennustada nende võimalikke edasisi samme, plaane ja kavatsusi, on oskuste kogum, mis on osutunud inimkonna evolutsiooniloos määravaks. Käesoleva artikli eesmärgiks on vaadelda nägude ära tundmise teemat semiootilise nähtusena. Uurimuse ulatus piirdub näo tajumisega inimese kui liigi poolt. Inimnägu analüüsitakse bioloogiliste ja kultuuriliste protsesside vahelisel lävel. Väidame, et kujutise äratundmisel on ühiskondlik-kultuuriline mõõde, millest ei tohiks mööda vaadata. Lähtudes Georg Lichtenbergi märkustest füsiognoomia kohta, käsitleme semiootilise eelhoiaku kriitikat, ideede assotsieerumist ja nägude äratundmisega seotud tüpifitseerimismehhanismi. Käsitleme neid kui kolme tähendusgradienti, mis kaardistavad nägude äratundmisega seoses kaalul olevaid erinevaid kognitiivseid operatsioone puudutava suhtevõrgustiku. TheStudies face and the faceness 361 Sign Systems 49(3/4), 2021, 361–382 The face and the faceness: Iconicity in the early faciasemiotics of Paul Ekman, 1957–1978 Devon Schiller1 Abstract: Paul Ekman is an American psychologist who pioneered the study of facial behaviour. Bringing together disciplinary history, life study, and history of science, this paper focuses on Ekman’s early research during the twenty-year period between 1957 and 1978. I explicate the historical development of Ekman’s semiotic model of facial behaviour, tracing the thread of iconicity through his life and works: from the iconic coding of rapid signs; through the eventual turn from classifying modes of iconic signification using gestalt categories to classifying modes of producing iconic sign-functions using minimal units; to the role and importance of iconicity for the study of the facial expression of emotion, both in terms of the similarities between iconic and analogue signs as well as the differences between facial coding and linguistic signification. In this intellectual genealogy, I argue not only that Ekman relied extensively upon conceptualizations and terminologies from semiotic thought for the creation of the Facial Action Coding System (FACS), but also that the question of iconicity is the pivotal problem across the many discoveries and innovations in what I term ‘Ekmanian faciasemiotics’. Keywords: Basic Emotions Theory (BET); Facial Action Coding System (FACS); facial expression; iconicity; nonverbal behaviour; Paul Ekman; psychology; semiotics of the face Introduction Paul Ekman is an American psychologist born in 1934 who pioneered the study of facial behaviour. He is best known for his seminal work doing pan-cultural research on the facial expression of emotion, Basic Emotions Theory (BET), deception detection, and micro-expressions, as well as for creating the Facial Action Coding System (FACS). Although Ekman identifies himself as a psychologist, and not a 1 Department of English and American Studies, University of Vienna, Vienna, Austria; e-mail: devonschiller@gmail.com. This research was supported by a DOC Fellowship from the Austrian Academy of Sciences. https://doi.org/10.12697/SSS.2021.49.3-4.06 362 Devon Schiller semiotician, a review of the literature reveals that his science was heavily influenced by semiotics. As Ekman (1982: 86fn2) writes, “over the years, [he] proposed a number of different phrases” to distinguish his approach to facial expression and its dynamic movement, but hopes that “terms, taken from semiotics, allow a more lucid differentiation”. In his autobiography, Ekman (2016: 11, 40, 41) also recalls the ways in which his collaborations and critiques with various “intellectual fathers,” each of whom brought semiotic frameworks into their methodological approaches, came about through “a string of coincidences” and “good luck”. And by the late 1960s, Ekman (1982: 45) began to explicitly model facial behaviour as “a multimessage, multisignal semiotic system”. Bringing together disciplinary history, life study, and history of science, this paper focuses on Ekman’s early research during the twenty-year period between 1957 and 1978. The extent to which Ekman’s natural science of nonverbal behaviour, especially in its formative years, was informed and inspired by semiotic theories and thinkers has been largely overlooked in the histories of this science (Plamper 2015: 142–172; Leys 2017: 76–128), which tend to fixate on later debates between biological essentialist and social constructivist positions. The importance of semiotics for Ekman and his measurement techniques has even been widely neglected in the semiotic literature, from general encyclopedia (Bouissac 2007[1998]) and handbooks (Grammer 2004: 3451–3452) to specialized monographs (Berson 2015: 113–115), where Ekman receives passing but persistent citation. In this paper, I seek neither to advocate nor oppose what Maria Gendron and Lisa Feldman (2017: 24) characterize as Ekman’s “‘neuro-cultural’ theory” for the facial expression of emotion, but instead to reframe his research within the context and history of the discipline of semiotics. I explore how Ekman referenced semiotic works by Jurgen Ruesch as well as Charles W. Morris in his semiotic model of facial behaviour, entered the discipline of semiotics through his friendship with Thomas A. Sebeok, and was further inspired by David Efron, among others. For this intellectual genealogy, I draw from primary sources that include Ekman’s autobiographical writings, professional correspondence, and scientific research, contextualizing his statements with relevant confirmations, criticisms, and confusions, and comparing his semiotics to Peircean as well as Saussurean traditions. Across the three sections of this paper, I explicate the historical development of Ekman’s semiotic model, tracing the thread of iconicity through his life and works: from the iconic coding of rapid signs; through the eventual turn from classifying modes of iconic signification using gestalt categories to classifying modes of producing iconic sign-functions using minimal units; to the role and importance of iconicity for the study of the facial expression of emotion, both in terms of the similarities between iconic and analogue signs as The face and the faceness 363 well as the differences between facial coding and linguistic signification. With this historicization of a theorization, I argue not only that Ekman relied extensively upon conceptualizations and terminologies from semiotic thought for the creation of FACS, but also that the question of iconicity, rather than stereotypicality or universality, is the pivotal problem across the many discoveries and innovations of what I term ‘Ekmanian faciasemiotics’. 1. The quest for facial iconicity begins Ekman got his start in semiotics during graduate school in the 1950s. He obtained his doctorate in clinical psychology from Adelphi College on Long Island in New York in 1958, at the time the only programme that trained psychotherapists, but which later would become the first university-based professional school of psychology in the United States. Before he graduated, Ekman was required to complete a one-year clinical internship. He pursued this internship under Swissborn American psychiatrist Jurgen Ruesch at the Langley Porter Neuropsychiatric Institute, a part of the University of California San Francisco Medical Center which, founded in 1941, was the first psychiatric institute in California. Ekman chose Langley Porter because Ruesch had written Communication: The Social Matrix of Psychiatry with English anthropologist Gregory Bateson in 1951 and Nonverbal Communication Notes on the Visual Perception of Human Relations with American painter and poet Weldon Kees in 1956. Ekman (2016: 14) writes in his autobiography that he had drafted an unpublished review of Nonverbal Communication “praising the many interesting ideas but criticizing it for a total lack of data, or even examples”. The critique was valid, at least from a methodological perspective, and considering that Ekman had been “trained in research methods” by American psychologist Robert Berryman, “a Skinnerian” who did operant conditioning studies and had “an explicit bias against theory not grounded in direct observation” (Ekman 2016: 95). By the late 1950s, Ruesch went further and put forward a semiotic model of nonverbal behaviour in his “Principles of human communication”, published the same year Ekman applied for an internship in 1957, and republished in Semiotic Approaches to Human Relations in 1972. As Winfried Nöth (1990: 169) points out, “the title of Ruesch’s collection of earlier papers” indicates how scientific research on nonverbal behaviour, “which was first developed within communication theory, was later placed within the framework of a semiotic theory”. However, Ekman’s acquaintance with Ruesch’s communication-turned-semiotic approach to nonverbal behaviour started even before this internship. In his first 364 Devon Schiller published article, written during the third year of his doctoral studies, Ekman (1957: 141) cites Ruesch in his own classification of interindividual behaviour into three modal types, including the “verbal, vocal, and nonverbal” which, he claims, “can be distinguished in terms of the medium of expression, the manner in which they are perceived, their developmental sequence, and their communicative value”. Further, Ekman (1957: 141) argues that “[a]ll three forms of behavior have both consensually validated linguistic or symbolic meaning, and more private or autistic connotations,” but that “[v]erbal far exceeds vocal or nonverbal behavior in degree of codification and consequent symbolic use”. In this way, Ekman (1957: 141) introduces an idea like iconicity, although he does not call it this, by situating nonverbal behaviours “less defined as to their meaning or intent” in opposition to verbal and vocal behaviours that “achieve definite symbolic usage”. Looking back with hindsight, Ekman (2016: 3) believes the paper “did not merit” publication in The Journal of Psychology. Still, Ekman’s first publication illustrates how the face reader as a young man, from the very outset of his ‘faciasemiotic project’, was investigating the complexity as well as the continuity between iconic, indexical, and symbolic relations. The same year that Ekman began his internship, Ruesch (1972: 127) defines “communication” in the broadest sense to include “all of the procedures by which one mind may affect another”. According to Ruesch (1972: 127), communication “involves not only oral or written speech”, but also the performance and pictorial arts, “and in fact, all human behavior”. Ruesch (1972: 127) triadically models the relationship between these signs and their object: signs constitute “a circumscript part of an action or event which either by force of its own structure or because of attention paid to it possesses for an observer problem-solving properties or cue values”; signals constitute “an impulse in transit regardless of whether it circulates inside or outside a human organism”; and symbols constitute “an extraorganismic device which has been agreed upon to refer in a condensed way to a series of actions or events […] used for coding purposes in order to transmit messages”. While all such processes of communication “exert an organizing influence” upon constituent individuals and “weld them into a larger system” (Ruesch 1972: 128). Therefore, to Ruesch (1972: 128), messages are “purposive expressions of internal events with the intent to convey information to other persons” that have also “been perceived and interpreted by another person”. As Ruesch (1972: 128) claims, the “communication apparatus of man”, which is sheltered within his body, carries messages through “effector organs, the sender”, the “evaluative apparatus, including the functions of memory decision-making”, as well as “sense organs, the receiver”. In the ten years that followed, Ekman’s model developed at least in part out of Ruesch’s as, in turn, this psychologist and psychiatrist each increasingly relied upon semiotics. The face and the faceness 365 After the completion of his internship and graduation, Ekman was conscripted into the army for two years, where he served as “chief psychologist” and conducted “applied studies” (Ekman 2016: 17), working at the Palo Alto Veterans Administration Hospital upon his discharge, before returning to Langley Porter for a postdoctoral fellowship supported by the National Institute of Mental Health (NIMH) from 1961 to 1963. Ekman then received a post-doctoral research grant from NIMH, which started in 1963 and was continuously renewed for the next forty years, in parallel to his professorship in the Department of Psychiatry at the University of California, San Francisco from 1971 to 2004. It was during his postdoc that Ekman began to collaborate with other psychologists, as well as linguists and anthropologists, who principally used semiotic theory. Ekman first met Hungarian-born American semiotician Thomas A. Sebeok, the founder of zoosemiotics and, later, biosemiotics, at the Indiana University Conference on Paralinguistics and Kinesics in Bloomington, Indiana in 1962 (Sebeok 1990: 287). Over the next twenty years, Sebeok included two papers by Ekman in volumes that he edited in the Approaches to Semiotics and The Semiotic Web book series (Ekman 1978; 1987), and Ekman published four co-authored papers in Semiotica, the journal of the International Association for Semiotic Studies, during Sebeok’s tenure as its editor-in-chief (Ekman, Friesen 1969; Ekman, Friesen, Tomkins 1971; Johnson, Ekman, Friesen 1975; Ekman, Friesen, Scherer 1976). In the correspondence between Sebeok and Ekman (Sebeok 1940–2001),2 which today is held by the Indiana University Archives, is a letter of reference that Sebeok wrote on 31 August 1976 for the NIMH Research Scientist Award in which he characterizes Ekman as “quite simply the world’s leading specialist […] in communication through nonverbal behavior” and commends how Ekman’s methods “give evidence of his mastery of current semiotic theory and psychological technique”. Reciprocally, in 1977 Ekman (Sebeok 1940–2001) wrote how he “tremendously value[s] Semiotica” and that it is “by far the best journal in this whole area”. In his autobiography, Ekman (2016: 95) even honours Sebeok as “the central figure in the last century promoting the field of semiotics”, although he does not further mention the doctrine of signs. In 1969, Sebeok invited Ekman “to write an article for the first issue” of Semiotica (Ekman 2016: 95). To Ekman (2016: 95–96), this paper, “The repertoire of nonverbal behavior”, remains “the seminal theoretical description of [his] approach”, with most of what he has “done in the years since an elaboration of what 2 The author would like to thank Mary Mellon, Digital Archivist at the Indiana University Archives, for her assistance in scanning the correspondence between Sebeok and Ekman. These documents are now available online: http://webapp1.dlib.indiana.edu/findingaids/view?doc. view=entire_text&docId=InU-Ar-VAE0871. 366 Devon Schiller [he] wrote then, not a replacement”. Ekman co-authored the paper with American psychologist Wallace V. (Verne) Friesen, whom he met while serving in the U.S. Army, beginning what would be “twenty-five years of very fruitful collaboration” (Ekman 2016: 37). In this seminal paper, Ekman and Friesen (1969: 49) define “a person’s non-verbal behavior” as “any movement or position of the face and/or the body” and describe the “origin, usage, and coding” for this behaviour. According to Ekman and Friesen (1969: 60), coding is the “the principle of correspondence between the act and its meaning” which describes the ways in which the nonverbal act contains or conveys information; that is, the “rule which characterizes the relationship between the act itself and that which it signifies”. Ekman and Friesen (1969: 60) identify “three coding principles”, namely: nonverbal signs may be “iconically (extrinsically)” coded and “carry the clue to their decoding in their appearance”, where “the nonverbal act, the sign, looks in some way like what it means, the significant”; they may be “intrinsically” coded and “like iconically coded behavior” visually relate to what they signify, only where “the act does not stand for but is its significant”; or nonverbal signs may be “arbitrarily (extrinsically)” coded and, “like words”, “bear no visual resemblance to what they signify”. As shown in Tables 1 and 2 respectively, the terms and definitions for the iconicintrinsic-arbitrary triad in Ekmanian faciasemiotics roughly correspond with the sign-signal-symbol triad in Ruesch’s semiotics and, in turn, the icon-index-symbol triad in the Locke-Peirce-Morris tradition. Table 1. Correspondence between the phenomenological terms of semiotic theories. Paul Ekman Jurgen Ruesch Charles W. Morris C. S. Peirce iconic (extrinsic) sign icon icon intrinsic signal index index symbol symbol arbitrary (extrinsic) symbol The face and the faceness 367 Table 2. Comparative definitions of iconicity. Ekman 1960s Iconically-coded signs “carry the clue to their decoding in their appearance,” where “the nonverbal act, the sign, looks in some way like what it means, the significant” (Ekman, Friesen 1969: 60). Ruesch 1950s Signs constitute “a circumscript part of an action or event which either by force of its own structure or because of attention paid to it possesses for an observer problem-solving properties or cue values” (1972: 127). Morris 1940s An “iconic sign, it will be recalled, is any sign which is similar in some respect to what it denotes,” such that “the strength of the iconic sign lies in its ability to present for introspection what it signifies” (1946: 191, 194). Peirce late 19th – “An Icon is a sign which refers to the Object that it denotes early 20th C. merely by virtue of character of its own and which it possesses, just the same, whether any such Object actually exists or not,” although an icon “does not act [as] a sign,” but “anything [...] is an icon of anything, in so far as it is like that thing and used as a sign of it,” being “strictly a possibility, involving a possibility, [in which] the Interpretant may be the Object” (EP 2: 291, 2: 227). In fact, Ekman and Friesen (1969: 60fn4) reference in a footnote how their “use of the term iconic was taken from” American philosopher and semiotician Charles W. Morris. Ekman came to embrace the works of Morris via the works of Ruesch, who does not use the terms ‘icon’, ‘iconic’, or ‘iconicity’ in his taxonomy of signs, but often cites Morris (Ruesch 1972: 93, 299, 344, 500, 559), giving an unequivocal indication of this intellectual genealogy. As Ekman and Friesen quote, Morris characterizes the iconic sign as “any sign which is similar in some respect to what it denotes. Iconicity is thus a matter of degree. [Further,] the strength of the iconic sign lies in its ability to present for introspection what it signifies” (Morris 1946: 191, 194; quoted in Ekman, Friesen 1969: 60 fn4). It is significant that Ekman looked not to Peirce but to Morris for the semiotic foundations of his face science. As Nöth (1990: 49) historicizes, the semiotics of Morris is characterized by a “fundamental departure from the Peircean tradition”. Whereas American pragmatist and semiotician Charles Sanders Peirce (CP 2.227, 8.343) conceives the “quasi-necessary, or formal, doctrine of signs” on “logical analysis”, Morris (1946: 2) holds the conviction that “a science of signs can be most profitably developed on a biological basis and specifically within the framework of the science of 368 Devon Schiller behavior”. Bringing together behaviourist empiricism, logical positivism, and neopragmatism in a behavioural semiotics that preceded biosemiotics, Morris defines semiosis as a “sign process, that is, a process in which something is a sign to some organism” (Morris 1946: 353). In terms of iconicity, Nöth (1990: 123) identifies, Morris reduced Peirce’s multiple criteria for the icon to the single criterion of “‘shared properties’ and believed on this basis in a quantifiability of iconicity”. However, Ekman and Friesen do not only define nonverbal iconicity by the presence of a partial resemblance to the referent, but also by the absence of absolute likeness to this object. That is, Ekman and Friesen (1969: 61) describe how the “iconically coded act is often easier to comprehend and simpler to utilize as a communicative signal; it is more stylized, starker, perhaps more abstract, and will leave out many of the details involved in the intrinsically coded act which it may resemble”. In other words, it is the properties of the object (e.g. emotion) that do not determine the construction of the sign (e.g. facial expression) that are revealed through its interrogation in such a way that new information may be derived. Following the tradition of Morris and Ruesch, Ekman and Friesen situate this semiotics within an explicitly biological, communicative, and social framework. Although Ekman and Friesen (1969: 61, 68) state that not many facial signs can be classified cleanly into one coding or another, and that this coding must be based on the predominant function of the facial sign in a given context, the coding in Ekmanian faciasemiotics is principally a hierarchical relation, or a matter of degree on a continuum, in which the intrinsic comes beneath and before the iconic (extrinsic) and the arbitrary (extrinsic). 2. Giving the semiotic modelling of face a hand In addition to these works by Ruesch and Morris, Ekman and Friesen’s early semiotic model for facial behaviour “owe[d] most to the writings” of Argentinian anthropologist David Efron (Ekman, Friesen 1969: 63). As Ekman (2016: 10) recalls in his autobiography, Efron studied under German-born anthropologist Franz Boas, the so-called father of American anthropology, who “urged his students to challenge the writings of the Nazi-influenced German social scientists during the 1930s”, including the fallacious argument that nonverbal behaviour is genetically determined, and thereby makes visible racial difference, “scientifically” justifying the eugenics programme of the Nazis against the Jews. For his doctoral dissertation Gesture and Environment in 1941, republished as Gesture, Race and Culture in 1972 also in Sebeok’s Approaches to Semiotics series, Efron (1972[1941]: 6) conducted quantitative observational studies of what he calls the “gestural styles” The face and the faceness 369 of East-European Jews and Southern Italians in New York. Efron (1972[1941]: 16, 136) claims that the racist ideology of Nazi Germany is “plagued with conceptions that have no place in scientific reasoning” and concludes that “traditional Jewish and Italian gestures disappear with the social assimilation of the individual into the so-called Americanized community”, thus confronting the biological determinism of Nazi ideology by confirming the socio-ecological construction of gestural repertoires. As Ekman (2016: 10) reflects, the “publication of Efron’s book coincided with the U.S. entrance into World War II” and, consequently, this “great work was lost [and] rarely cited”. However, “building on Efron’s work”, Ekman (2016: 37) “developed a classification” of nonverbal behaviour, which included gestural behaviour as well as facial behaviour. For this semiotics of the face, Ekman and Friesen (1969: 63) further drew upon “a series of discussions” with American psychologist George F. Mahl in which they “attempted to clarify some of the issues implicit in his dichotomization of nonverbal behavior”. In a study of psychiatric outpatients during initial clinical assessments, Mahl (1987[1968]: 15) categorizes nonverbal behaviour into “actions judged to be common substitutions for verbal utterances” and those “judged not to be” so, or “communicative gestures” such as nodding, pointing, or shrugging, and “autistic actions” such as scratching, rubbing, or touching oneself. Ekman and Friesen (1969: 57) believe that “Mahl’s distinction between autistic and communicative behavior, while valuable, can be improved upon”, that this “use of the term ‘communicative’ is too broad,” and that communication should distinguish among “behavior which has a shared decoded meaning (informative), that which influences the other person’s interaction (interactive)”, and, as Mahl had in mind, “that which [is] intended to transmit a message (communicative)”. Seeking input on “films of psychiatric patients”, Ekman visited Mahl during his fellowship from 1963 to 1964 at what today is known as the Center for Advanced Study in the Behavioral Sciences at Stanford University. As Ekman (2016: 37) writes in his autobiography, Mahl suggested that he focus on the patients’ nonverbal “‘acts’ and develop a theoretical classification”. Ekman (2016: 37) remembers this being “quite a step forward” for him, especially as “a Skinnerian who had been schooled in not imposing theory, in simply just counting what you saw”. Indeed, this moment for Ekman marked a shift in his overall epistemology, from bottom-up inductive reasoning to top-down deductive reasoning and from specific observations to general principles, which in turn laid the conceptual groundwork for his early semiotic modelling of facial behaviour. Proposing a triadic model of iconic, intrinsic, and arbitrary coding, Ekman and Friesen (1969: 61) then proceed to “refine further the ways in which a nonverbal act is related to its significant”. Largely utilizing Efron’s “terminology 370 Devon Schiller and distinctions”, they classify five different “visual relationships between act and significant”, including kinetic, pictorial, pointing, rhythmic, and spatial relationships, as well as five categories that describe the ways in which nonverbal behaviours can be “distinguished by the particulars of usage, origin and coding”, including ‘adaptors’, ‘affect displays’, ‘emblems’, ‘illustrators’, and ‘regulators’, as well as numerous sub-categories for a system with an indefinite and unfixed number of phenomenological categories which they term ‘codes’ (Ekman, Friesen 1969: 61, 63, 68). At this point, Ekman and Friesen theorize their typology primarily, but not exclusively, for hand gestures rather than facial behaviours. For instance, they demonstrate the difference between iconic and intrinsic coding, or what would be iconic and indexical in Peircean as well as Morrisian terms, with gestures of the arm, hand, and fingers in aggressive actions which involve visually forming “cut[ting] a person’s throat” or a “trigger finger movement” and actually holding a knife or gun (Ekman, Friesen 1969: 61). Here, the principal distinction between iconic and intrinsic coding is whether the nonverbal behaviour is literally a part of the movement involved in the action. In other words, is the act just being portrayed or is it really being performed, does it “represent” or “reproduce” a human action, where to be iconic it must “resemble but cannot be its significant” (Ekman, Friesen 1969: 70). Ekman and Friesen (1969: 61) admit that the “line between the iconic and intrinsically coded act may appear to be a fuzzy one”, especially in the case of a nonverbal behaviour that is “only a part of a total action”. In sum, to Ekman and Friesen (1969: 62) rhythmic and spatial relationships between movement and meaning are always iconic, kinetic relationships may be either iconic or intrinsic, and pointing is always intrinsic, but “typically, nonverbal behavior combines elements of more than one code”, and the pictorial may include the spatial, the spatial may include the rhythmic, and so on. That is, not many signs clearly fall into one category or another but must be categorized based on their function within a context. For example, as Ekman and Friesen (1969: 68) note, they “are not proposing that the illustrator category is exclusive of the others; assignment of an act into this category depends upon usage in a given instance”. Thus, in Ekmanian faciasemiotics, iconization represents sensory experience and visual perception through a formal resemblance between the sign and the object that it refers to in the extralinguistic real world. Ekman and Friesen (1969: 78) recognize that the “coding of facial affect displays is not at all obvious”. Drawing from English naturalist Charles Darwin and his 1872 work titled The Expression of the Emotions in Man and Animals, the principal claim for this semiotic model, which later became known as Basic Emotions Theory (BET), is threefold. Firstly, the facial expression of emotion is prototypical, specified through biology. Secondly, it is universal across cultures. And, thirdly, The face and the faceness 371 there is a causal link between the outside physiological behaviour of a facial expression and the inside psychological phenomena of an emotion that emanates it; that is, between facial signifier and emotional signified. According to Ekman and Friesen (1969: 71), the so-called “primary affects”, such as ‘anger’, ‘disgust’, ‘fear’, ‘happiness’, ‘sadness’, and ‘surprise’, a list that constantly undergoes variation and revision across Ekman’s career, as well as all possible “affect blends”, activate with the neural appraisal of an “evoking stimulus” and trigger so-called “programs” of response, which are functionally discrete and phylogenetically stable, and include the muscular contraction mechanisms and skin appearance movements of the face. In this view, any variability to innate expression is largely determined by social scripts or “display rules”, which are “socially learned, probably quite early in life, and prescribe different procedures for the management of affect displays in various social setting, roles, etc.” (Ekman, Friesen 1969: 75). Therefore, Ekman and Friesen (1969: 74) principally consider what they call “affect ‘programs’” to be intrinsically coded, or, in Peircean terms, “a sign which refers to the Object that it denotes by virtue of being really affected by that Object” (EP2: 291–292), where the expressing of the face in an emotion is analogous to the “veering of a weathercock” in the wind (EP2: 274), being the product of this causation. There is of course much debate in the scientific community over the merits and limitations of BET. Points of criticism encompass, for example, how the theory fails to adequately consider the significance played by context and culture in the facial expression of emotion, the degree to which it is part of social intent, and is not only reactive but also predictive, serving to guide actions. Opposing theories include, for instance, the Behavioural Ecology View and Theory of Constructed Emotion. Ekman (2016: 11, 48, 78) cites Darwin as one of his “intellectual father figures” and claims Darwin’s Expression as the foundational text for BET, although Ekman did not read Darwin until quite late in his career. However, in Expression, Darwin (1872: 12, 356) hypothesizes not only that facial expression of emotion “once existed in a much lower and animal-like condition” and consequently may have certain elements of pan-cultural universality, but also that “every true or inherited movement of expression seems to have had some natural and independent origin”. Evolved by natural selection, these emotion expressions are either remnants from reflexes that had once been useful, arise from the antithesis of contrasting elicitors, or are the direct action of the nervous system in an overflow of excitation. Darwin (1872: 67, 76) concludes that most human facial behaviour is a rudimentary or vestigial feature much like the appendix, body hair, tail bone, or wisdom teeth, which is “of no service, often of much disservice” and “purposeless”, with any communicative value being incidental. 372 Devon Schiller To Ekman and Friesen (1969: 78), “[b]oth Darwin’s explanation of the evolution of such displays, and [their own] account of how certain displays may naturally develop in the course of each person’s life, would suggest that some affect displays are either intrinsically coded or iconic”. However, Ekman and Friesen (1969: 78) concede that this coding may be the case “only for some affects”. “[I]f we accept Darwin’s principle of antithesis as the explanation of the happiness display,” Ekman and Friesen (1969: 78) admit, “then it would be arbitrarily coded”. Even so, social psychologist Alan J. Fridlund (2017: 77) argues that Ekman, among other advocates of BET, “misread” Darwin “when they cite him to support their claim that facial expressions evolved ‘to express emotion’” and that certain facial expressions are universal. But if the face did not evolve for carrying information and social communication, as Darwin finds, this does not mean that facial expressions would bear no resemblance to the emotions they signify, as Ekman and Friesen suggest; they could still be intrinsically or iconically coded, just on different grounds. Nevertheless, for Ekman and Friesen (1969: 60 n5) the “recognition of the need to distinguish intrinsically coded from iconic behavior grew out of a discussion” with American psychologist Silvan S. (Solomon) Tomkins, who had modelled a universalist primary affect theory in the first two volumes of his magnum opus, Affect Imagery Consciousness in 1962 and 1963. In Ekmanian faciasemiotics, the distinction between the iconic or arbitrary and the intrinsic is crucial in determining whether a facial event is a voluntary, staged, and posed expression that accompanies intentional communication, or an involuntary, spontaneous, and genuine expression that accompanies felt emotion. This semiotic model of facial behaviour is highly problematic on methodological as well as theoretical levels. Of course, there are different approaches to modelling nonverbal behaviour across the academic disciplines, including coding systems that classify the alteration, description, or function of behaviour, which today are more and more based not only on anatomy but also on neuroanatomy. As critiqued by neurologist Hedda Lausberg (2013: 65), creator of the Neuropsychological Gesture Coding System (NEUROGES), Ekman and Friesen define their “main values” for “the analysis of movement behavior in social interaction” as “predominantly functional”, with adaptors, affect displays, emblems, illustrators, and regulators all theorized in connection with a certain psychodynamic. However, some of these “main values contain several movement classes”, such as how “regulators can be positions shifts but also head nods”, and, additionally, the values “may all function as regulators” (Lausberg 2013: 65). Consequently, “there is no precise definition referring to the visual appearance of the movement” (Lausberg 2013: 65). Ultimately, Ekman also came to recognize this methodological problem, both in “14 techniques for measuring facial actions [over] a span of 55 years” between 1924 The face and the faceness 373 and 1978 (Ekman 1982: 50), including anatomically, ethologically, linguistically, and theoretically based systems, as well as his own Facial Affect Scoring Technique (FAST), a precursor to FACS (Ekman, Friesen, Tomkins 1971). Psychologist Erika Rosenberg, who worked on FACS with Ekman in his Human Interaction Laboratory at the University of California, San Francisco from 1991 to 1994, and who now instructs how to code the face in her FACS Workshop, points out that the “problem with theoretically derived systems is that they cannot discover behaviors that were not posited in advance” and are “by definition selective”; in other words, these coding systems “were not developed to catalogue everything the face can do, but rather they describe whether the face does things it should do according to a given theory” (Rosenberg 2005[1997]: 14).3 Having begun his search for facial iconicity with the works of Ruesch, Efron, Mahl, Sebeok, and Tomkins, among others, Ekman (1982: 46) turns from theorizing a semiotic model to developing a measurement tool, one which can be used for “measuring the sign vehicles that convey the message” rather than making interpretation “judgments about one or another message”, and for classifying “descriptive units” rather than “inferential labels”. 3. Analogue privilege, digital process In their seminal paper, Ekman and Friesen (1969: 60n4) also cite how their distinction between iconic and arbitrary coding is “very similar” to Ruesch’s distinction between analogue and digital codification. To Ruesch (1972: 131), “analogic codification makes use of continuous representations”, whereby “the signs used are, in their proportions and relations, similar to the things, ideas or events for which they stand”, while “digital codification, in contrast, is based upon discontinuous representation of events in which the continuity of nature is sliced into discrete steps”. Of course, Ekman and Friesen (1969: 60fn4) make clear that when they write about the terms ’analogic’ and ‘digital’, they do not mean these as media per se, or to “involve further specifications of the mathematics relevant to modelling information processing” from computer or electronic technologies, in the sense of the terms already being used in the 1960s and most frequently applied today. Rather, working from Ruesch’s definition, Ekman and Friesen (1969: 60n4) go on to claim that the iconic-analogue is fundamentally “more characteristic of nonverbal behavior” such as facial behaviour while the arbitrary-digital is fundamentally more characteristic of verbal behaviour such as spoken language. 3 For transparency of interest, the author participated in the FACS Workshop with Rosenberg at the University of California, Berkeley in 2015 and the NEUROGES seminar with Lausberg at the German Sports University Cologne in 2017. 374 Devon Schiller This contrast between the analogic and the digital in Ekmanian faciasemiotics is overall consistent with semiotic theory. As Daniel Chandler (2017: 183) surveys the field, semioticians tend to regard analogue codes as distinctions of degree, a relation of more-or-less, signs which are natural, implicit, as well as continuous, that is, a gradation of infinite subtleties; digital codes as distinctions of kind, a relation of either/or, signs which are artificial, explicit, as well as discontinuous, that is, a categorization of discrete units; and communication as something that “involves both modes”. This distinction between analogue and digital codes, Chandler (2017: 184) explicates, is “frequently represented as natural versus artificial”, extending an argument made by Claude Lévi-Strauss about the isomorphism between oppositions: that of nature and culture and that of continuous and discrete. Yet such a distinction, which has underlain the faciasemiotic project since classical physiognomy, carries with it an entire host of dyadic connotations, such as contrasting the emotional work of the flesh against the rational work of the mind and, consequently, facial behaviour against spoken language, with the connotation being that it is the outer physiological form that most accurately reflects the inner psychological function. In Ekmanian faciasemiotics, the signs of the face are considered more genuine and truthful, and consequently of higher value, when compared with the signs of spoken, thought, or written language. This can especially be seen in Ekman’s research into deception detection. For instance, Ekman and Friesen (1969: 88, 93) refer to the concealed or repressed affect displayed in “‘[m]icro’ facial expressions” with comparatively short duration as “nonverbal leakage”. And psychologist Maureen O’Sullivan and Ekman (2004: 275) refer to individuals who achieve 80% accuracy identifying deception in facial behaviour as “‘expert’ lie detectors or [truth] ‘wizards’”. Behind this nomenclature is the perspective that intrinsic facial signs are not only analogic but also natural and that iconic as well as arbitrary facial signs are not only digital but also artificial. In his 1978 paper “Facial signs”, Ekman (1978: 141, 151fn9) reflects on how he has “argued that the linkage between facial movement (sign) and emotion (significant) is natural […] rather than a conventional or arbitrary association,” noting that he uses “the term natural as it has been discussed and defined by Sebeok”. To Sebeok (1975: 237, 239), both signals and symptoms exemplify natural signs, as a signal “triggers some reaction on the part of the receiver” either “mechanically (i.e., naturally) or conventionally (artificially)”, and a symptom is a “compulsive, automatic, nonarbitrary sign, such that the signifier is coupled with the signified in the manner of a natural link”. Whereas Ekman singles out the naturality of indexicality, or what he terms intrinsicality, Sebeok (1975: 242, 247) explains how iconic and indexical sign relations have a natural link, whereas symbolic sign relations have a “‘conventional link’ – Peirce’s ‘imputed character’”. The face and the faceness 375 Drawing from Sebeok, Ekman (1978: 141) claims that his “view of facial expression has been based on the assumption that if facial expressions are evolved behavior”, and “if the relationship between sign and significant is natural”, then facial expressions of emotion must also be “impervious to the influence of culture”. However, Ekman (1978: 141) clarifies, he has “never said that facial expressions are always automatic or unwitting. Facial expressions of emotion are not fixed-action patterns or instincts of some kind, impervious to culture. They can be automatic, but not always or even usually”. To Ekman, the intrinsic face, that is, the indexical face, is the rare face in the endless forms most beautiful of human communication. Yet, regarding the value of the icon, like in Peircean semiotics, these picture-like signs resemble the object that they signify, have the modality of direct, immediate perception, and are therefore the most persuasive of signs. While Ekman privileges analogue signs in his semiotic model of facial behaviour, digital sign processes are important and even necessary for the development of his Facial Action Coding System. In the ten years between 1969 and 1977, Ekman and his research team in the Human Interaction Laboratory at the University of California, San Francisco developed the Facial Action Coding System, released in 1978 and revised in 2002. FACS is a “comprehensive” sign-based technique for the description, measurement, and classification of “all possible visually distinguishable facial movements” (Ekman, Friesen 1976: 58). This coding system is based at least in part on work of French neurologist Guillaume Duchenne and Swedish anatomist Carl-Herman Hjortsjö, among others (Ekman, Friesen 1976: 63). Applying FACS, Ekman and Friesen (1976: 64) instruct, a coder scores individual Action Units (AUs) that have an “anatomical basis” but do not necessarily correspond one-to-one to either specific muscles or muscle groups. FACS supports the performance of a spatial analysis of the face, both globally with principal components, as well as locally with particular features, with the use of intensity scores for each Action Unit which are annotated by appending letters (A-E) from trace or very slight (A) to maximal or strongest possible (E). FACS also supports the performance of a temporal analysis of the face through the observation and measurement of the combination of Action Units, or what is commonly meant by an expression, with the use of timing scores which segment the actions into their “onset, apex, and offset” (Ekman, Friesen 1976: 60). For instance, in what is popularly known as the smile of happiness, the zygomatic major constricts posteriorly and superiorly, pulling the lip corners upwards, and the orbicularis oculi, pars lateralis constricts laterally, raising the infraorbital triangle, lifting the cheeks, and gathering the skin medially toward the eye socket from around its lateral edge. As Fig. 1 illustrates, the combination of these facial actions in FACS is described using Action Unit 6, the “Cheek Raiser”, and Action Unit 376 Devon Schiller 12, the “Lip Corner Puller” (Ekman, Friesen 1976: 65), or, with the lips apart and mouth open, and attendant intensity ratings, and 6D+12E+25C. Figure 1. Onto-cartographic projection in four-dimensions (4D), three dimensions of space (width, depth, and height) and one dimension of time, representing the iconic continuum of facial signs. This “digitization” of the face has led to a common misconception in expert scholarship and popular culture: that in Ekmanian faciasemiotics, facial behaviour is understood as a linguistic system, one in which muscular contraction mechanisms form up alphabetic elements, and skin appearance movements function like grammatical morphemes, whereby the face speaks or writes emotion which in turn can be heard or read by a face-literate expert coder. For instance, “Ekman is Saussurean,” Jeremy Sherman commented to me in at the Annual Gathering in Biosemiotics at Lomonosov Moscow State University in 2019. The unspoken implication was not that Ekman follows Saussure through his bibliographic references, but rather that Ekmanian faciasemiotics resembles Saussurean linguistics. As Swiss linguist and semiotician Ferdinand de Saussure (2006: 140) writes, “[l]anguage represents a system internally ordered in all its parts”, which “depends on an object, but free and arbitrary in relation to the object”. Further, according to Saussure (2006: 147), “language and writing are NOT BASED on a natural relationship between things,” and there is “never in any way a link” between the signifier and signified. The face and the faceness 377 Certainly, Ekman and Friesen (1976: 64, 65fn) note, for the Action Units of the Facial Action Coding System, the names are a “shorthand, not meant to describe the appearance changes, but a convenience to call them to mind,” and the “numbers are arbitrary and do not have any significance”. Yet this application of digital signs in the study of the face is not the same as alleging that facial behaviour is itself a language. Nevertheless, the popular myth that in Ekmanian faciasemiotics the face is viewed as a text is widely propagated, for instance in Lie to Me, the 2009–2011 television drama series based on Ekman’s life and works, which carries the tagline “The truth is written all over our faces”. However, to claim that facial behaviour meets some linguistic benchmark would be to confuse digital with linguistic signs. “[A]nalogue signs can of course be digitally reproduced,” Chandler (2017: 185) explains, “but they cannot be directly related to a standard ‘dictionary’ and syntax in the way that linguistic signs can”. As Ekman (1978: 151fn9) writes in his own words, it is “important not to misread [his] use of the terminology of semiotics as an implication that the facial signs are a linguistic system”. Nevertheless, the foundational process for Ekmanian faciasemiotics is the intersemiotic translation from the analogue signs of certain facial behaviours to the digital signs of the Facial Action Coding System. Still, as Chandler (2017: 184) points out, all sign systems “impose digital order on what we often experience as a dynamic and seamless flux”, and the “very definition of something as a sign involves reducing the continuous to the discrete”. Structuralists including Saussure, Lévi-Strauss, and Roman Jakobson even theorize about how the process of making differentiations and the mere otherness of either/or oppositions is fundamental not only to linguistic systems in particular but also to signifying systems in general. Across his career, Ekman described the facial expression of emotion using objectivity and rationality, both in the 1960s with his use of affect display gestalts, since the 1970s with his use of minimal Action Units, as well as since the 1980s with the selective application of AUs to gestalts for stereotypical emotions, as with EmotionFACS (EMFACS). However, this was not because he observed these facial behaviours to be this way amid the phenomena of the world; rather, it was because Ekman experimented on facial signs in such a way amid the processes of his study. That is, the definition for the facial expression of emotion in Ekmanian faciasemiotics has been altered to fit a particular conception of what science ought to be in order to make it practisable. 378 Devon Schiller Conclusion The question of whether or to what extent facial behaviour may be indexical is by no means resolved, either in Ekman’s oeuvre or the face sciences. In the neuro-cultural model of Ekmanian faciasemiotics, facial signs may be “iconically (extrinsically), intrinsically, or arbitrarily (extrinsically) coded” (Ekman, Friesen 1969: 60), which is to say, iconically, indexically, or symbolically coded in terms deriving from the Locke-Peirce-Morris tradition, on the basis of the visual appearance of the dynamic movement, either in whole or in part, and depending on the predominant function of the nonverbal behaviour in the given context. In this view, the facial expression of emotion is most likely “either intrinsically coded or iconic” (Ekman, Friesen 1969: 78). However, the distinction between these codes is a “fuzzy one” and the way to determine this principle of correspondence between the act and its meaning, the signifier and its signified, is “not at all obvious” (Ekman, Friesen 1969: 61, 78). Further, if the neuroanatomy of the face did not evolve at least to some degree for carrying information about emotion phenomena, whether on the intra- or inter-organismic level, then the facial expression of emotion “would be arbitrarily coded” (Ekman, Friesen 1969: 78). In other words, as Umberto Eco argues, “spontaneous, noncodified expressions of emotion produced without any communicative intention” may well fall “below the semiotic threshold”, and only “become codified and thus semiotic” when they “exhibit cultural variation or are simulated or imitated in a histrionic context” (Eco cited in Nöth 2000: 54). For Ekman, however, there is an important “recognition of the need to distinguish intrinsically coded from iconic behavior” (Ekman, Friesen 1969: 60fn5). In Ekman’s terms, iconic and arbitrary coding, each of which are extrinsic codes, and not entirely part of the essential nature of the nonverbal behaviour, but also operate with or arise from the outside, and “signifies or stands for something else”, whereas intrinsic coding “is in a sense no code in that the act does not stand for but is its significant; the meaning of the act is intrinsic to the action itself ” (Ekman, Friesen 1969: 60). Additionally, this distinction means differentiating the analogic and natural signifying relation of intrinsic codes from the digital and artificial signifying relation of iconic and arbitrary codes. As Ekman (1978: 141) argues, “the relationship between sign and significant is natural” in certain facial expressions of emotion, and thereby “automatic or unwitting” and “impervious to the influence of culture”, but facial behaviour is “not always or even usually” this way. Ekman’s science, as Jan Plamper (2015: 155) historicizes, “involved the reconciliation of nature (basic emotion) and culture (display rule)”. And as Ruth Leys (2017: 90) contextualizes, “Ekman was committed to the project of distinguishing The face and the faceness 379 between nature and culture, between the ‘natural signs’ of emotional expression that might be so small or rapid as to be easily overlooked and those culturally coded displays or ‘artificial signs’ that he thought tended to mask or disguise them”. Thus, in Ekman’s faciasemiotic project, beneath the exclusively arbitrary coding that is entirely culturally assigned, the indexical face really performs the emotion, intrinsically representing a part of the movement involved in this action, such as with the “enjoyment smile”, whereas the iconic faceness just portrays the emotion, extrinsically resembling the action only in some respect, such as with the so-called “false”, “masking”, and other non-enjoyment smiles (Ekman, Davidson, Friesen et al. 1990: 342–343). This is exactly why iconicity is so essential in the early faciasemiotics of Paul Ekman: by lying, iconic facial signs provide clues to the truth, that is to say, the truthfulness of the indexical. References Berson, Josh 2015. Computable Bodies: Instrumented Life and the Human Semiotic Niche. London: Bloomsbury. Bouissac, Paul 2007[1998]. Encyclopedia of Semiotics. Oxford: Oxford University Press. Chandler, Daniel 2017[2002]. Semiotics: The Basics. (3rd ed.) London: Routledge. CP = Peirce, Charles S. 1931–1958. Collected Papers of Charles Sanders Peirce. Cambridge: Harvard University Press. (Vols. 1–6, Hartshorne, Charles; Weiss, Paul, eds.; vols. 7–8, Burks, Arthur Walter, ed.) [In-text references are to CP, followed by volume and paragraph numbers.] Darwin, Charles 1872. The Expression of the Emotions in Man and Animals. London: John Murray. https://doi.org/10.1037/10001-000 Efron, David 1972[1941]. Gesture, Race and Culture: A Tentative Study of the Spatiotemporal and “Linguistic” Aspects of the Gestural Behavior of Eastern Jews and Southern Italians in New York City, Living Under Similar as Well as Different Environmental Conditions. (Approaches to Semiotics 9.) The Hague: Mouton. Ekman, Paul 1957. A methodological discussion of nonverbal behavior. The Journal of Psychology 43(1): 141–149. https://doi.org/10.1080/00223980.1957.9713059 Ekman, Paul 1978. Facial signs: Facts, fantasies, and possibilities. In: Sebeok, Thomas A. (ed.), Sight, Sound and Sense: Advances in Semiotics. Bloomington: University of Indiana Press, 124–156. Ekman, Paul 1982. Methods for measuring facial action. In: Scherer, Klaus R.; Ekman, Paul (eds.), Handbook of Methods in Nonverbal Behavior Research. Cambridge: Cambridge University Press, 45–135. Ekman, Paul 1987. A life’s pursuit. In: Sebeok, Thomas A.; Umiker-Sebeok, Jean (eds.), The Semiotic Web ‘86: An International Yearbook. Berlin: Mouton de Gruyter, 3–45. Ekman, Paul 2016. Nonverbal Messages: Cracking the Code – My Life’s Pursuit. San Francisco: Paul Ekman Group. 380 Devon Schiller Ekman, Paul; Davidson, Richard J.; Friesen, Wallace V. 1990. The Duchenne smile: Emotional expression and brain physiology II. Journal of Personality and Social Psychology 58(2): 342–253. https://doi.org/10.1037/0022-3514.58.2.342 Ekman, Paul; Friesen, Wallace V. 1969. The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica 1(1): 49–98. https://doi.org/10.1515/ semi.1969.1.1.49 Ekman, Paul; Friesen, Wallace V. 1976. Measuring facial movement. Environmental Psychology and Nonverbal Behavior 1(1): 56–75. https://doi.org/10.1007/BF01115465 Ekman, Paul; Friesen, Wallace V.; Scherer, Klaus R. 1976. Body movement and voice pitch in deceptive interaction. Semiotica 16(1): 23–27. https://doi.org/10.1515/ semi.1976.16.1.23 Ekman, Paul; Friesen, Wallace V.; Tomkins, Silvan S. 1971. Facial affect scoring technique (FAST): A first validity study. Semiotica 3(1): 37–58. https://doi.org/10.1515/ semi.1971.3.1.37 EP = Peirce, Charles S. 1992. The Essential Peirce: Selected Philosophical Writings. Bloomington: Indiana University Press. (Vol. 1, Houser, Nathan; Kloesel, Christian, eds.; vol. 2, Peirce Edition Project, ed.) [In-text references are to EP, followed by volume and page numbers.] Fridlund, Alan J. 2017. The behavioral ecology view of facial displays, 25 years later. In: Fernández-Dols, José-Miguel; Russell, James A. (eds.), Oxford Series in Social Cognition and Social Neuroscience: The Science of Facial Expression. New York: Oxford University Press, 77–92. https://doi.org/10.1093/acprof:oso/9780190613501.003.0005 Gendron, Maria; Barrett, Lisa Feldman 2017. Facing the past: A history of the face in psychological research on emotion perception. In: Fernández-Dols, José-Miguel; Russell, James A. (eds.), Oxford Series in Social Cognition and Social Neuroscience: The Science of Facial Expression. New York: Oxford University Press, 15–38. Grammer, Karl 2004. Körpersignale in menschlicher Interaktion. In: Posner, Roland; Robering, Klaus; Sebeok, Thomas A. (eds.), Semiotik: Ein Handbuch zu den zeichentheoretischen Grundlagen von Natur und Kultur. 4. Teilband. Berlin, New York: Walter de Gruyter, 3448–3486. https://doi.org/10.1515/9783110179620.4.15.3448 Johnson, Harold G.; Ekman, Paul; Friesen, Wallace V. 1975. Communicative body movements: American emblems. Semiotica 15(4): 335–353. https://doi.org/10.1515/ semi.1975.15.4.335 Lausberg, Hedda (ed.) 2013. Understanding Body Movement: A Guide to Empirical Research on Nonverbal Behaviour: With an Introduction to the NEUROGES Coding System. Frankfurt: Peter Lang. Leys, Ruth 2017. The Ascent of Affect: Genealogy and Critique. Chicago: University of Chicago Press. https://doi.org/10.7208/chicago/9780226488738.001.0001 Mahl, George F. 1987[1968]. Gestures and body movements in interviews. In: Mahl, George F., Explorations in Nonverbal and Vocal Behavior. Hillsdale: Lawrence Erlbaum Associates, 7–74. Morris, Charles W. 1946. Signs, Language, and Behavior. New York: Prentice-Hall. https:// doi.org/10.1037/14607-000 Nöth, Winfried 1990. Handbook of Semiotics. Bloomington: Indiana University Press. https://doi.org/10.2307/j.ctv14npk46 The face and the faceness 381 Nöth, Winfried 2000. Umberto Eco’s semiotic threshold. Sign Systems Studies 28(1): 49–61. https://doi.org/10.12697/SSS.2000.28.03 O’Sullivan, Maureen; Ekman, Paul 2004. The wizards of deception detection. In: Granhag, Pär Anders; Strömwall, Leif A. (eds.), The Detection of Deception in Forensic Contexts. New York: Cambridge University Press, 269–286. https://doi.org/10.1017/ CBO9780511490071.012 Plamper, Jan 2015. The History of Emotions: An Introduction. (Tribe, Keith, trans.) Oxford: Oxford University Press. Rosenberg, Erika L. 2005[1997]. The study of spontaneous facial expressions in psychology. In: Ekman, Paul; Rosenberg, Erika L. (eds.), What the Face Reveals: Basic and Applied. Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). (2nd ed.) Oxford: Oxford University Press, 3–18. https://doi.org/10.1093/ acprof:oso/9780195179644.003.0001 Ruesch, Jurgen 1972. Semiotic Approaches to Human Relations. (Approaches to Semiotics 25.) The Hague: Mouton. https://doi.org/10.1515/9783110816228 Saussure, Ferdinand de 2006. Writings in General Linguistics. (Bouquet, Simon; Engler, Rudolf, eds.; Sanders, Carol; Pires, Matthew, trans.) Oxford: Oxford University Press. Sebeok, Thomas A. 1940–2001. Thomas Sebeok papers, correspondence and subject files. Collection C264, Boxes 26, 27, 61. Bloomington: Indiana University Archives. Sebeok, Thomas A. 1975. The semiotic web: A chronicle of prejudices. Bulletin of Literary Semiotics 2: 1–63. https://doi.org/10.5840/bls197521 Sebeok, Thomas A. 1990. Semiotics in the United States. In: Sebeok, Thomas A.; UmikerSebeok, Jean (eds.), The Semiotic Web 1989. Berlin: Walter de Gruyter, 275–396. https:// doi.org/10.1515/9783110874099 Лицо и лицевость: иконичность в ранней лицосемиотике Пола Экмана, 1957–1978 Пол Экман – американский психолог, ставший пионером изучения поведения лица. Статья посвящена ранним исследованиям Экмана в 1957–1978 годах и объединяет историю дисциплины, биографию и историю науки. Я рассматриваю, как развивалась семиотическая модель поведения Экмана, и прослеживаю проявления иконичности в его жизни и работе: от иконического кодирования быстрых знаков; через поворот от классификации режимов знакового обозначения с использованием категорий гештальта к классификации режимов создания знаковых функций с использованием минимальных единиц; к роли и важности иконичности для изучения выражения эмоций на лице с точки зрения как сходства между знаковыми и аналоговыми признаками, так и различий между кодированием лица и лингвистическим обозначением. В этой интеллектуальной генеалогии я утверждаю не только, что Экман в значительной степени полагался на концептуализацию и терминологию семиотики при создании Системы кодирования лицевых движений (FACS), но также что проблема иконичности лежит в основе многих открытий и инноваций, которые я называю «лицосемиотикой Экмана». 382 Devon Schiller Nägu ja näolisus: ikoonilisus Paul Ekmani varases näosemiootikas, 1957–1978 Paul Ekman on Ameerika psühholoog, kes pani aluse näokäitumise uurimisele. Artiklis liidetakse eriala ajalugu, eluloouuringud ning teaduslugu, keskendudes Ekmani varajasele teadustegevusele kakskümmend aastat kestnud perioodil vahemikus 1957 kuni 1978. Selgitan Ekmani loodud näokäitumise semiootilise mudeli ajaloolist arengut, järgides ikoonilisuse juhtlõnga tema elus ning töödes alates kiirete märkide ikoonilisest kodeerimisest, läbi viimaks toimunud pöörde ikoonilise tähistamise laadide klassifitseerimiselt gestaltkategooriaid kasutades ikooniliste märgifunktsioonide tekitamise klassifitseerimisele minimaalseid üksuseid kasutades; kuni ikoonilisuse rolli ning tähtsuseni emotsionaalsete näoilmete uurimises, nii ikoonliste ja analoogmärkide sarnasuste terminites kui ka näolise kodeerimise ja keelelise tähistamise erinevuste kaudu. Väidan, et selles intellektuaalses genealoogias Ekman mitte üksnes ei toetunud näotegevuse kodeerimissüsteemi (FACSsüsteemi) luues ulatuslikult semiootilisest mõtlemisest pärinevatele kontseptualiseeringutele ja terminoloogiale, vaid lisaks on ikoonilisuse küsimus pöördelise tähtusesega probleem arvukates avastustes ja uuendustes valdkonnas, mida ma nimetan “ekmanlikuks näosemiootikaks”. Marlen Haushofer’s The Wall and Sign the post-nuclear culture of the face383–399 383 Systems Studies 49(3/4), 2021, Marlen Haushofer’s The Wall and the post-nuclear culture of the face Emanuela Ferragamo1 Abstract. The intertwining of landscape and face belongs to human spatial epistemology: as suggested by Matteo Meschiari, primitive humans used to orientate themselves in landscape through recognition of facial patterns. By reflecting upon Marlen Haushofer’s novel The Wall (Die Wand), the article aims to question the semantic of the “face of the landscape” in the wake of an imagined nuclear apocalypse that leaves behind a cat, a cow, a dog, a woman and a wall. The wall transcends the boundaries between human and other-than-human: in terms of Roberto Marchesini, it creates a somato-landscape – a hybridization of inner and outer landscapes typical of post-human awareness. Finally, such a landscape culminates in the dismissal of the pre-apocalyptic culture of the face: faces no longer function as a means of recognition. Keywords: face; landscape; boundary; exotopy; hyperobject; melancholy; trouble Introduction The Austrian writer Marlen Haushofer published the novel The Wall (Die Wand) in 1963.2 The novel begins rather simply: the protagonist, a middle-aged widow, goes to the Austrian mountains to spend some time with her cousin Luisa and Luisa’s husband Hugo. However, on the day after her arrival she finds herself alone in the house. When looking for her hosts, who have not returned, she discovers a transparent wall at the end of the mountain’s gorge. Manuscripts show that Haushofer gave the novel its present title, thematizing one of her recurrent metaphors (Schmidjell 2000: 47). Indeed, critics interpret the ‘wall’ 1 Foreign Language, Literature and Cultures Department, University of Turin (Unito), Via Verdi 8 – 10124 Turin, Italy; e-mail: emanuela.ferragamo@unito.it. 2 I quote from the edition published by Ullstein Taschenbuch (Haushofer 1985) and refer to the novel through its initials followed by the page numbers, and provide the English translation by Shaun Whiteside (Haushofer 1999) in the footnotes. I refer to the latter through its initials followed by the page numbers. https://doi.org/10.12697/SSS.2021.49.3-4.07 384 Emanuela Ferragamo in the context of her repeated reference to constricted places (Von der Lühe 1986: 82), where she staged both women’s oppression and a possible exit from patriarchy.3 More recently, literary inquiries have re-established an interest in the narrative of natural space and landscape in The Wall.4 The present paper stems from this context and studies how the post-nuclear landscape of the novel affects the depiction of faces in it, analysing the relationship between the terms ‘landscape’ and ‘face’, while referring to Umberto Eco’s (1985: 10) concept of semiotics as a process of integrating one’s reflection as the symbolic image of the Self in a broader, symbolic system. The paper consists of three parts. Firstly, the narrative function of mirrors is highlighted with reference to the status of the presence of the face in a post-nuclear word where the protagonist is the only survivor. Reflecting on mirrors means considering the problem of recollection of the past in the symbolic constitution of the protagonist’s face in her silent dialogue with the dead. 3 Regula Venske views the novel as an expression of Haushofer’s tendency to stage the quest for identity in non-places where women’s boredom is both the expression and the consequence of the psychological suffering of women in middle-class society (Venske 1987: 199). However, those places of sensorial inexistence express not only a desire for self-annihilation, but also a struggle to overcome the stifling social conventions of post-war Austria (Venske 1987: 205). In The Wall, Haushofer’s last woman fights for survival, thus introducing an intertextual reference to Robinson Crusoe. However, more recent criticism tends to question the connection: Michael Hoffman (2000: 202) underlines the characters’ different approaches to natural resources and a more problematic use of the utopian genre in Haushofer’s novel. The question of a gendered genre is also relevant: Sarah Nelseen (2019: 136) suggests reading Haushofer’s robinsonade through Donna Haraway’s’ concept of SF as both Science Fiction and Speculative Feminism. Some critics view the killing of the last man at the end of the novel in the light of an urge to masculinize women: the murder of the last surviving man appears as an allegory of women’s emancipation from all sorts of masculine projections. However, the murder has negatively influenced the reception of her novel. Thus, it comes as no surprise that feminism has been just recently discovering Haushofer’s legacy, reconquering a long-neglected author (Knecht 2007: 84). During the 1970s, feminists greeted the last pages of her novel with a certain puzzlement due to a resistant inhibition towards feminine aggression (Venske 1987: 213) and the predominance of elements of fragility and sensibility in women’s representations (Venske 1991: 23). 4 Here, I quote Régine Battiston’s recent essay on the meaning of landscape in Haushofer’s work (Battiston 2019) and Johanna Chlovanec’s and Grazziella Predoiu’s (2016) articles on her semiotics of space. The latter draw on Eduard Soja’s postmodern concept of ‘thirdspace’ and on a reinterpretation of the robinsonade through a psychological analysis of the plot, respectively. According to Chlovanec (2006: 26–28), Soja’s thirdspace reveals a third path in the narrative of gender that goes beyond the dichotomy male versus female and deconstructs the patriarchal assumption as such. Predoiu (2016: 76–77) focuses on the idyllic scenery and views the wall as an allegory of the desire of a symbiosis with nature. Marlen Haushofer’s The Wall and the post-nuclear culture of the face 385 Secondly, such a dialogue takes place within boundaries. Not only is the wall a hindrance, but it also originates a semiosphere and results in the ultimate construction of a palisade beside its surface. By doing this, the protagonist reinterprets her previous epistemology in a sense that transects gender and physical boundaries: the wall and the palisade trigger an exotopic process of transection between the inside and the outside of the semiotic system. This aspect is analysed through the post-human idea of somato-landscape (Marchesini 2002: 189). Finally, Haushofer’s landscape is observed from the theoretical perspective of hyperobjects. In this view, the smooth surface of the wall appears as an anticipation of the smooth space of Gilles Deleuze’s and Felix Guattari’s theory of space. 1. The face and the other: The face as a means of recollecting the past and the quest for one’s identity in Haushofer’s novel 1.1. Facing landscape Landscape means different things to different disciplines, its currency not always a guarantee for taxonomic clarity.5 In this paper, I will follow Matteo Meschiari’s suggestion and study Haushofer’s landscape in profile, which means discussing its textual practice, rather than its essence (Meschiari 2008: 276). In order to do so, I refer to Paolo D’Angelo (1999: 210–211) who summarizes the term’s definitions as involving the typical form of a territory: a portion of earthly surface that constitutes an image or, more generally, perceptive data received by an observer. This definition helps us to outline an operative definition of landscape emerging in Haushofer’s novel. While studying the aesthetic meaning of the subject’s response to landscape, D’Angelo questions its relationship with a philosophical view of nature, which would possibly go beyond the ecological underappreciation of aesthetics. Thematizing beauty as the crux of modernity, D’Angelo (2001: 110) aims to examine its connection with the historical and imaginative context of of the observer’s culture. Haushofer depicts landscape as a cultural image originating from the collective intervention on a territory. Firstly, the emergence of the wall destroys the idyllic scenery of the Alps, as the protagonist no longer finds the mountain “romantic and charming” but “damp and gloomy” (TW: 19). As in the aftermath of the Second World War Austrian political discourse would shape national identity thanks to the Alpine landscape, thus 5 For a useful and clear taxonomy of ‘landscape’ see Berque 1994, Schama 1995, Jakob 2005, Weber 2010 and Sobral Campos 2019. 386 Emanuela Ferragamo distancing itself from Nazi Germany (Zeyringer 2001: 60), the appearance of the wall thematizes the struggle to deal with the collective memory of the Holocaust as it forces the protagonist to cope with the “inability to mourn” of her generation (Knecht 2007: 83). The protagonist finally understands how to deal with the wall by letting her emotions out, bursting into tears while recollecting the dead on Christmas Eve. Secondly, the wall relates to the Cold War psychosis of nuclear destruction. As the wall represents the collective fault of an entire age, it has no inventor and is a step towards the ongoing alienation of modern humanity. More generally, the wall marks the abrupt end of the ancient, intimate relationship between man and nature. In this respect, its naissance recalls in some sense what Michael Jakob considers the origin of landscape: the loss of the antique harmony of cosmos (Jakob 2005: 15). D’Angelo emphasizes the process of shaping the territory in the making of landscape, thus implicitly recalling the history of the term: Germanic and Romance languages form the word ‘landscape’ using a suffix that indicates a territory and its configuration: ‘-age’ in French, ‘-scape’ or ‘-schiftz’ in old English (Collot 2005: 152). Furthermore, they intend landscape as a way of organizing data in a coherent unity that is to be realized both in arte and in visu (Collot 2005: 156). As Timothy Morton has argued, landscape embodies a subjective state: a landscape picture “is less about land, then, and more about scape. We talk about the mood of a landscape, the feeling it evokes in us” (Morton 2011: 80). The landscape’s position in front of the observer is particularly evident in Haushofer’s novel. Here, the protagonist literally bumps her forehead against the wall: a sort of transparent screen through which landscape appears as a distant image (DW: 14). The fact that landscape is in front of the subject guarantees its visibility in the first pages of the novel, the latter implying historically the possibility of controlling information and resources. The organization of landscape expresses political longing for controlling the territory (Vitta 2005: 148). In this respect, it is relevant that Haushofer highlights a connection between the transparency of the wall and the desire of the protagonist to keep everything under visual control (DW: 187). However, the novel problematizes the idea of landscape’s full visibility. While in the 18th-century utopian fiction transparency is positive (Baczko 1978: 22), the one of the wall forces the protagonist to walk with outstretched arms in order not to get hurt (DW: 15). Marlen Haushofer’s The Wall and the post-nuclear culture of the face 387 1.2. Facing faces Haushofer’s critique of the paradigm of visibility also affects the perception of faces. The German word ‘Gesicht’, that appears in the German-language version of Haushofer’s novel, has its roots in the verb ‘sehen’, designating the outer, visible dimension of face.6 In the novel faces seem hard to catch, even in places where they should be present. The protagonist’s reflection in Luise’s vanity mirror speaks more of the missing woman than of the present one: “In Luises Frisierspiegel sah ich manchmal verwundert meine neue Erscheinung. Mein Haar, das stark gewachsen war, hatte ich mit der Nagelschere kurz geschnitten. Es war jetzt ganz glatt und von der Sonne gebleicht. Mein Gesicht war mager und gebräunt” (DW: 81–82).7 Appearing in the mirror, the protagonist recalls the disappearing of the person to whom the object once belonged: Luise, who died after the appearance of the wall. The protagonist imagines her as a doll with red lips and reddish-blond curls (DW: 124). The depiction of Luise as a sleeping beauty clearly expresses Haushofer’s criticism of patriarchy (Chamayou-Kuhn 2019: 30) and the emotive stiffness caused by women’s efforts to fit a stereotypical image: Luise seems in a desperate need of social acceptance (DW: 12).8 When reflecting upon the appearing and disappearing of the protagonist and Luise in the mirror, we may recall Eco’s (1985: 18) dismissal of mirrors as semiotic channels. Mirrors put a sender and an addressee in contact, but they do not generate a semiosis: this would require the absence of the object for whom the sign stands, while mirrors need the object referred to be present (Eco 1985: 19). Yet the mirrored object appears to be strangely present in the passage quoted above. In a way, Luise appears in the mirror in the sense that the protagonist is forced to remember her face (DW: 82). Thus it seems that Haushofer thematizes presence as the effect of recollection, giving transparency a rather melancholic quality. Hidden objects fascinate us because they seem to suggest that something has become lost: facing a veiled face, we are in front of an enigma (Starobinski 1957: 7). Seduced by the object without being satisfied, our eyes keep trying to catch a glimpse of the reluctant promise of appearance (Starobinski 1957: 7). The wall causes a similar dynamic. The protagonist is afraid of looking through the wall 6 See http://woerterbuchnetz.de/cgi-bin/WBNetz/wbgui_py?sigle=DWB&mode=Vernetzung& lemid=GG11297#XGG11297. 7 “I sometimes looked with amazement at my new appearance in Luise’s dressing-table mirror. As my hair had grown a great deal, I had cut it short with the nail-scissors. Now it was quite flat, and bleached by the sun. My face was thin and tanned, [...]” (TW: 68). 8 Considering Luise’s need to be liked, it is interesting that the mirror represents a relevant tool for feminist studies to destructure social representations of women (Stephan, Weigel 1988). 388 Emanuela Ferragamo and of seeing anything more than what she has understood in her heart of hearts (DW: 12). Therefore, as she reaches the top of a raspberry patch and observes the landscape through her binoculars from there, she does not manage to discern the face of a motionless woman sitting in front of the coachmaker’s cottage beyond the wall (DW: 29). Her attempts at focusing on the face of the unknown woman are in vain: in the end, images fade in a kaleidoscope of forms and colours (DW: 29). Starobinski (1975: 8) understands such frustration as melancholy: the fact that the desired object is in full sight and yet impossible to reach. This can be related to Massimo Leone’s (2015: 126) reflection upon the semiotics of transparency: when Leone suggests considering transparency as an anticipation of transcendence, he means that transparent objects are destining actants – they prevent the sight as obstacles and at the same time they arouse the desire to see. The apprehension towards hidden faces beyond the wall could also be interpreted as the result of two distinctive moments which characterize recollection: availability and accessibility. Paul Ricoeur (2004: 129) believes that Henri Bergson had the merit of solving a long-lasting riddle of memory: the fact that its images are simultaneously lost, yet available through recollection. Still, the clue to this apparent paradox relies on the centrality of recognition: to let recollection exist, the subject has to recognize it as something with its roots both in the past and in the present (Ricoeur 2004: 142). Therefore, recognizing the past means recovering it, and recovering it means presuming it to be available, if not accessible (Ricoeur 2004: 143). The same dilemma occurs in a passage in Haushofer’s novel in which she imagines her protagonist trying to reach some pansies, while, unfortunately, the flowers grow on the other side of the wall: “Einmal im Frühling […] sah ich drei oder vier Veilchen. Gedankenlos streckte ich die Hand aus und stieß in die Wand. […]. Die Veilchen hielten mir ihre kleinen violetten Gesichter entgegen, aber ich konnte sie nicht anfassen” (DW: 167).9 Seeing the pansies, she mistakes their visual accessibility for their actual availability, yet the mistake is soon corrected: the flowers’ perfume she thought she smelled when she first glanced at the flowers dissolves after she has touched the wall (DW: 167). 9 “Once in the spring […] I saw three or four violets. I absently reached out my hand and leaned against the wall. […] The violets held their little purple faces to me, but I couldn’t not touch them” (TW: 145). Marlen Haushofer’s The Wall and the post-nuclear culture of the face 389 1.3. From the other side of the mirror. By letting her protagonist stand in front of a mirror, Haushofer explores the relationship between present images and their recollection. In doing so, she reflects upon the semiotics of the mirror: the object for whom the sign in the mirror stands coincides only partially with the actual object. On another occasion, Haushofer addresses the topic of mirroring again, depicting the protagonist’s face as a reflection in her cat’s eyes. “Ich sehe mein Gesicht, klein und verzerrt, im Spiegel ihrer großen Augen”, she writes (DW: 52).10 While the previous quote emphasized the protagonist’s estrangement from her appearance in Luise’s mirror, here Haushofer underlines the fact that her face has lost its original proportions: it has become small and distorted. In a general sense, this disproportion speaks for the asymmetry between the Self and the other, which characterizes the ethical philosophy: giving a prominent status to the other, its premises stem from asymmetry, rather than reciprocity (Ricoeur 2004: 181). In other words, if we act ethically, we give more importance to the other than to ourselves. This is why the protagonist depicts herself as small in comparison to the cat: being the last human on earth, the protagonist feels how the responsibility towards her animals and the wild is hers only to bear (DW: 127). Moreover, the disproportion between the “smallness” of the protagonist’s face and the size of her cat’s eyes can be interpreted as a plea to her counterpart to accept the moral quest for relationship conveyed through visual contact. Emmanuel Levinas understands the presence of faces as a moral awakening: they express the eternal transcendence of the other, metaphorically breaking through the world (Lévinas 1977: 199). On the one hand, Haushofer imagines the relationship between the protagonist and the cat like motherhood: the protagonist first notices the starving cat after she thinks she has heard a baby crying (DW: 48) and, slowly, she becomes dependent on the animal (DW: 51). On the other hand, as an independent, cautious animal, the cat can be seen as representing the irreducible extraneousness of the other in Mikhail Bakhtin’s terms. Relating with the other means for him overcoming monologism on the linguistic level, responsibility and care on the moral level, and exotopy on the aesthetic level (Ponzio 2003: 189–190). In Bakhtin’s works, exotopy results from a reconsideration of the terms “inside vs. outside and defines a transitioning of those two dimensions into one another: a process that is not dialectic but represents a conceptualisation of the outside as traversing between two discourses” (Kalinova 2018: 105). 10 “I can see my face, small and tight, in the mirror of her big eyes” (TW: 41). 390 Emanuela Ferragamo Exotopy comes into mind when we consider the fact that the cat wears a typographic sign on its fur: “Die Katze starrt aus gelben Augen in die Ferne. Manchmal kommt sie plötzlich zu mir zurück, und ihre Augen zwingen mich, die Hand auszustrecken und den runden Kopf mit dem schwarzen M auf die Stirn zu streicheln” (DW: 149).11 Where someone else might have seen nothing but the pattern of a cat’s coat, the protagonist sees a letter and not just a random one – ‘M’ being the initial of Haushofer’s first name, Marlen. As Augusto Ponzio (2003: 57) notices, exotopy means the invisibility of the objective author, emerging because of literary criticism, and of the objectivized one, representing the author in his or her work. Therefore, authorship appears as a pure representational principle: defying representation, the author is present in the dialogue of his or her many voices (Ponzio 2003: 58). Referring to the peculiar invisibility of the author and paraphrasing Levinas’ formulation for the ethical relationship with otherness, the silent dialogue between the cat and the protagonist breaks through monologic writing, renegotiating the boundaries between the inner and the outer. 2. Surviving beyond the wall: The wall as a boundary 2.1. The wall is more than just a wall In Haushofer’s novel, this renegotiation takes place in an era of hyperobjects. In the philosophy of Morton, hyperobjects designate both the object of study and its theoretical frame. The premise of Morton’s thinking lies in the confutation of Kant’s epistemology and his idea of a distance between the observing subject and the observed object (Morton 2013: 27). In this regard, Morton (2013: 27) considers global warming as a typical example of a hyperobject outside of which it is impossible for us to step. Although hyperobjects suit postmodern literature better, they appear to be useful tools to interpret Haushofer, since they permit underlining of two aspects of her depiction of the landscape and have a meaningful role in the understanding of faces. Firstly, both the wall of Haushofer’s novel and hyperobjects result from an apocalypse of some kind. On the one hand, Haushofer’s protagonist faces the newest product of humankind: a wall most likely originating from a nuclear 11 “The cat stares in the distance with yellow eyes. Sometimes she suddenly comes back to me, and her eyes compel me to stretch out my hand and stoke her round head with the black M on the forehead” (TW: 128). Marlen Haushofer’s The Wall and the post-nuclear culture of the face 391 weapon (DW: 209). On the other hand, Morton (2013: 99) uses the expression “end of the world” to question the aesthetics underlying the idea of ‘world’ as a “container in which objectified things float or stand”. Yet as different as ‘hyperobject’ and the wall may seem, they both express the ancient meaning of apocalypse, the word indicating the ritual digging of a pit in the outskirts of the city in honour of the dead in order to prevent their returning home (De Martino 2002: 194). Similarly, Haushofer’s wall serves both as a tribute to the dead, the wall being like a shrine at which to admire a modern Pompeii (DW: 56), and as a denial of death. In analogy, Morton (2013: 103) represents reactions to hyperobjects on a scale that stretches from denial to melancholy. Secondly, hyperobjects problematize Haushofer’s depiction of the wall as something in front of the protagonist. In the epistemological context of Morton’s philosophy, most of her struggles not to touch the wall express longing for the avoidance of all intimacy with the Uncanny. For Morton (2013: 33), intimacy means the inevitable proximity of hyperobjects. The loss of a safe distance from troubling events, such as climate change, results in a forced cohabitation with unknown phenomena, causing normality to appear extraneous, unreal. Haushofer’s protagonist questions the plausibility of the wall: such things do not occur normally in Austria, nor in Europe (DW: 19). For the purposes of the present article, Morton’s phenomenology has two consequences on the study of landscape as it problematizes the pictorial metaphor pertinent to the term ‘landscape’ and renegotiates the concept of ‘technique’. In the etymology of ‘landscape’, perception and depiction have traditionally been intertwined: landscape is both a real, perceptible territory and its metaphorical representation. Those two characteristics are first synonymous and then interrelated: through its imaginary, aesthetics conveys and influences the cultural view of landscape (Milani 2017: 51). However, the fact that Morton views landscape as a picture hanging from a wall clashes with the spatial and temporal fluctuation of hyperobjects (Morton 2013: 74). If those are to be thought of as landscape, then the latter is to be seen not as a painting but as a snapshot. Thus, it is no surprise that Haushofer has set her novel in the mountains, where the landscape has proved to be more suitable for photography than for painting (Cosgrove, Della Dora 2012: 12). Thinking of landscape in terms of a hyperobject also means reflecting upon its temporality. Whereas landscape studies mostly deal with changes over long-lasting periods, hyperobjects rather focus on minimal, constant oscillations. If thinking in the latter terms, research on landscape would involve what Donna Haraway (2016: 1) defines as “staying with the trouble”: living and thinking being truly present, that is, focussing on little variations of patterns. Similarly, Haushofer’s protagonist 392 Emanuela Ferragamo survives, having come to terms with her past and having renounced disclosing of the future. Landscape shifts out of focus, its spatial and temporal boundaries start to interrelate, to blend in with one another. Rather than being a picture, it is a collection of instantaneous fragments with no discernible progression or timeline: something very similar to Georg Simmel’s concept of nature as an incessant flow of phenomena (Simmel 1957: 72). In this respect, hyperobjects put the concept of technique in a new perspective. 2.2. One plus one equals three: Unmaking the wall As Rocco Ronchi (2017: 252) argues, in revisiting the Western view of nature the concept of technique plays a great role. In particular, he speaks in favour of George Simondon’s understanding of technique as the formal constituting of human experience (Ronchi 2017: 254). The understanding of technique not as a product, but as a process, permits us to reflect upon the technical history of humankind in terms similar to the naturans of nature (Ronchi 2017: 254). Technique reveals itself as a natural praxis, because the human ability of inventing and using technical prosthesis has roots in nature (Ronchi 2017: 256–257). In a similar way, in her novel Haushofer imagines a process of re-naturalization of technical artefacts: old vehicles abandoned in the woods are a home for wild animals (DW: 78). More generally, the relevance of technique emerges in the protagonist’s longing for dexterity. On the one hand, her complaining about her clumsiness meets the demands of the robinsonade, as she must survive in a hostile environment. On the other hand, dexterity is part of a broader process of renegotiation of epistemology: as Haushofer points out, the new reality is to be experienced with “hands, feet and guts” (TW: 50). An interesting starting point for reflecting on the post-apocalyptic meaning of technique in the novel is the construction of a boundary. The protagonist attends to the work in two stages, initially contenting herself with driving some branches into the mud near the wall (DW: 20) and then carrying two knives with her to cut them: the jack-knife of Hugo, her cousin’s husband, and her own penknife for sharpening pencils (DW: 29). The boundary extends along the wall, making it visible and hence less dangerous, and results in a sort of palisade. The two knives constituting the equipment express two gestural programmes. Greimas’ (1970: 69) concept of ‘gestural programme’ is part of a broader study on semiotics of the human body in cultures and their gestural praxis: the use of the body relies upon a common project and a shared sense. Greimas’ working hypothesis is that phonological methods and practices apply to gestural Marlen Haushofer’s The Wall and the post-nuclear culture of the face 393 signification because gestures correspond to expression (Greimas 1970: 85) – the only difference lies in the fact that a gestural sequence is a “programme of manifestation without a phonological project” (Greimas 1970: 86). Referring to this methodological frame, Haushofer’s reference to the two knives expresses the clashing of two gestural programmes of technical intervention on landscape: weaponry and modesty. The protagonist claims that she carries Hugo’s jack-knife because it gives her an apparent sense of protection (DW: 22). Haushofer points out that Hugo’s knife, belonging to his arsenal, is one of those objects that should have protected him from the nuclear Armageddon he feared (DW: 10).12 Haushofer, who has emphasized Hugo’s pointless hoarding of provisions, now marks a disproportion between the sharpness of his knife and the way the protagonist uses it for a very modest purpose. The boundary she makes by cutting branches is a distraction, a springtime game (DW: 20) and speaks more generally for a modest practice of technique, thus implicitly pleading in favour of what Hans Jonas (2000: 122) called ‘technical modesty’: the fact that something less than perfect could be accepted as good and the consequent dismissal of technical megalomania. While the wall is described as the consequence of human hubris toward nature, the gestural programme of the making of the boundary creates a harmless replica of the wall, a “toy boundary” (TW: 13). In reference to such a boundary, it is interesting to remind that the word for ‘wall’ designates also ‘a fortification made of branches’ in the German language.13 2.3. Troubles in translation To clarify how the gestural programmes of technical omnipotence (the wall) and modesty (the palisade) intertwine, a reference to Juri Lotman’s semiotics is in order. Lotman (2000: 140) understands boundaries as means and spaces for different semiospheres to transect, thus problematizing Bakhtin’s concept of exotopy for the study of landscape. For Lotman (2000: 140), boundaries mark the outer and the inner parts of semiospheres. The latter define the whole semiotic space of a culture, which is its result and a condition for its development (Lotman 2000: 125). As different semiospheres contain different elements and functions, the transition of a message 12 “At the time everyone was talking about nuclear wars and their consequences, and this led Hugo to keep a little store of food and other important things in his hunting-lodge” (TW: 3–4). 13 See https://woerterbuchnetz.de/?sigle=DWB#4. Indeed, the reference to branches recurs in a further passage of Haushofer’s novel indicating the profound alienation of the protagonist from nature: crushing branches under boots is for her another facet of human violence towards the environment (DW: 62). 394 Emanuela Ferragamo from one semiosphere into another occurs at the price of adjustments through translations (Lotman 2000: 140). In this respect, Haushofer’s first boundary appears to be the wall that filters and adapts the external into the internal, to paraphrase Lotman (2000: 140). In the novel, this dynamic involves a renegotiation of the opposition between the centre and the periphery of the cultural system: after the emergence of the wall, the protagonist realizes that she lives in the farther portion of Western geography, that she names Belutschistan (DW: 49). Yet the wall also appears as a sign, “something that stands for something, to someone in some capacity” (Ponzio 2003: 122). More precisely, the wall reveals itself as a sign after the making of its interpretant, the palisade. For Peirce, a sign means something in response to another sign (Ponzio 2003: 181), and the construction of the palisade represents a reaction to the wall: having realized it is a weapon (DW: 41), the protagonist replies by defending herself with a palisade, translating the grounding element of the wall into a new boundary. What changes during this translation is the meaning of order. On the one side, the making of the palisade is a remedy against the violation of the cosmic order caused by the wall (DW: 29). On the other side, the space delimited by the palisade is one of disorder and trouble: a word expressing how “to live and die well with each other in a thick present” (Haraway 2013: 1). Indeed, a thick present is the place beyond the palisade. Due to the staggering increase of wild animals caused by the extinction of humans, there are not enough natural resources for them to survive without the help of the protagonist, as the wall precludes access to a considerably vast portion of the earth. On the protagonist’s side of the wall, every action of the protagonist risks damaging an environment already out of balance. 3. Final observations: Towards a post-apocalyptical understanding of the face 3.1. Wood walks with her When reflecting on the intertwining of intentions, directions and species during the Anthropocene, Haraway (2013: 58) coins the concept of ‘sympoesis’. The neologism emphasizes the creative interdependence of earthlings, thus referring to a responsive, collective making of the word – a “wordling-with, in company” (Haraway 2013: 58). Systems so defined defy spatial and temporal boundaries: while the latter are usually “centrally controlled”, sympoetic systems distribute information and control among their components (Haraway 2013: 61). Haushofer’s protagonist experiences something similar to sympoesis in the final pages of the novel: “Manchmal verwirren sich meine Gedanken und es ist, als fange der Wald an, Marlen Haushofer’s The Wall and the post-nuclear culture of the face 395 in mir Wurzeln zu schlagen und mit meinem Hirn seine alten, ewigen Gedanken zu denken. […] Es fällt mir schwer, beim Schreiben mein früheres und mein neues Ich auseinanderzuhalten” (DW: 185).14 Referring to her protagonist’s trouble in distinguishing between what she was and what she is while writing, Haushofer problematizes the act of writing as a means of understanding reality: while the report gives the protagonist’s thoughts formal coherence, it forces her to become aware of her dissolving identity. Here, ‘dissolving’ is used in the meaning of what Roberto Marchesini (2002: 189) defines as somato-landscape: the transferring of landscape inside the human body. In Haushofer, somato-landscape arises because writing urges both the psychological and physical awakening of the Self and the epistemological revision of the experience of the landscape. This provides the protagonist with tools for the recollection, and thus the recognition, of herself. Meschiari (2008: 60) has convincingly compared writing with mapping. The similarity of these processes lies above all in the fact that both deal with unstable objects: fiction with narrative plausibility and maps with the making and unmaking of territories. The similarity between writing and mapping seems particularly true for Haushofer’s novel, where the branches for the palisade were cut with a knife meant for sharpening pencils (DW: 29). The protagonist’s interrelation with the wood adds a puzzling question to the depiction of mobility and immobility in the landscape. Whereas to some extent the wall and the palisade stand for a reflection upon obstructed mobility, the woods introduce a new variable in the reflection upon the landscape: that of “smooth” space. 3.2. Face and other lines In the second volume of their study A Thousand Plateaus: Capitalism and Schizophrenia, Gilles Deleuze and Félix Guattari thematize distinctions and interrelations between a “striated” and a “smooth” perception of space and understanding these in different cultural models. Roughly distinguished, these perceptions correspond to the sedentary and the nomadic experiences of places, respectively. In striated spaces, one goes from one point to another, whereas in smooth spaces, points are less important than trajectories: the movement in the landscape contemplates the possibility of changes of direction (Deleuze, Guattari 1980: 596). The experience of smooth spaces has 14 “Sometimes my thoughts grow confused, and it is as if the forest has put down roots in me, and is thinking its old, eternal thoughts with my brain […]. I find it hard to separate my old self from my new self ” (TW: 161). 396 Emanuela Ferragamo no visual points of reference or invariant distances. Differently from the striated space, smooth space is made up of a steady changing orientation provided by a population of nomads who are actively entertaining tactile relations among themselves (Deleuze, Guattari 1980: 598). In Haushofer’s novel, it appears clear how the wall orientates the landscape towards a smooth model of space by forcing the protagonist towards a mostly tactile perception of space. For the first time, the protagonist realizes that something invisible is blocking her path by bumping into it: then, she carefully approaches the wall with outstretched hands. Although she tries to avoid contact with the wall, its invisibility forces her to trust her tactile perception instead of her eyes. The analogy with Deleuze and Guattari’s theoretical frame becomes even more striking at the end of the novel. Observing how a mountain brook has succeeded in leaking into the wall, the protagonist is relieved: “Mit dem Wasser aus den Bächen wird das Leben, winziges, einfaches Leben, einsickern und die Erde wiederbeleben” (DW: 223).15 With the wish for the renewal of life in an elementary form, she implicitly expresses the longing for the smooth space par excellence. That is water (Deleuze, Guattari 1980: 597). Furthermore, the smooth experience of the landscape clarifies the protagonist’s symbolic metamorphosis into a tree, as before noticing how her thoughts seems to have their roots in the woods rather than being her own, the protagonist gives a final depiction of herself: “Ich bin noch immer mager, aber muskulös, und mein Gesicht ist von winzigen Fältchen durchzogen. Ich bin nicht häßlich, aber auch nicht reizvoll, einem Baum ähnlicher als einem Menschen” (DW: 182).16 The process of metaphorically becoming part of the woods can be interpreted as an allegorical longing for smooth space. As Deleuze and Guattari (1980: 602) praise the attachment to a territory explored through the mind as the ultimate nomadic experience of modern times, Haushofer imagines that her protagonist metaphorically travels into the woods without actually moving (DW: 185). Yet the similarity the protagonist feels to have acquired with trees goes somehow in the same direction as Deleuze’s and Guattari’s critique of Western emphasis on the face as a means of fixing identity into one space (Deleuze, Guattari 1980: 623). Undulations, frictions, variations: the smooth space dissolves the face into a riddle of intersecting lines and spirals. What Haushofer describes as wrinkles are trajectories, points and lines of a new face in landscape.17 15 “With water from the streams life, tiny, simple life, will seep in and revivify the earth” (TW; 196). 16 “I am still scrawny, but muscular, and my face is criss-crossed with tiny wrinkles. I’m not ugly, but neither am I attractive, more like a tree than a person” (TW: 69). 17 I dedicate this paper to my brother. Marlen Haushofer’s The Wall and the post-nuclear culture of the face 397 References Assunto, Rosario 2005[1973]. Il paesaggio e l’estetica. Palermo: Novecento. Battiston, Régine 2019. Funktion der Landschaft in Marlen Haushofers Werk. In: Alraud, Sylvie; Lacheny, Marc; Lajarraige, Jacques; Leroy du Cardonnoy, Éric (eds.), Dekonstruktion der symbolischen Ordnung bei Marlen Haushofer. Berlin: Frank & Timme, 39–57. Baczko, Bronisław 1978. L’utopia. Immaginazione sociale e rappresentazioni utopiche nell’età dell’Illuminismo. (Botto, Margherita; Gibelli Dario, trans.) Torino: Einaudi. Berque, Augustin 1994. Cinq propositions pour une théorie du paysage. Ceyzérieu: Champ Vallon. Chamayou-Kuhn, Cécile 2019. Grenz und Fremdheitserfahrungen in „Die Wand“. In: Alraud, Sylvie; Lacheny, Marc; Lajarraige, Jacques; Leroy du Cardonnoy, Éric (eds.), Dekonstruktion der symbolischen Ordnung bei Marlen Haushofer. Berlin: Frank & Timme, 17–39. Chlovanec, Johanna 2016. Marlen Haushofers „Die Wand“ als Thirdspace. Sprachkunst Beiträge zur Literaturwissenschaft 1: 15–30. https://doi.org/10.1553/spk45_1s15 Collot, Michel 2015. Landschaft. In: Dünne, Jörg; Mahler, Andreas (eds.), Handbuch Literatur & Raum. Berlin: De Gruyter, 151–159. Cosgrove, Denis; Della Dora, Veronica 2012. Introduction: High places. In: Cosgrove, Denis; Della Dora, Veronica (eds.), High Places: Cultural Geographies of Mountains, Ice and Science. London: Palgrave Macmillan, 1–18. D’Angelo, Paolo 2001. Estetica della natura: bellezza naturale, paesaggio, arte ambientale. Bari: Laterza. D’Angelo, Paolo; Catucci, Stefano; Carchia, Gianni 1999. Dizionario di estetica. Bari: Laterza. Deleuze, Gilles; Guattari, Félix 1980. Mille Plateaux. Paris: Edition du Minuit. De Martino, Ernesto 2002. La fine del mondo. Contributo all’analisi delle apocalissi culturali. Torino: Einaudi. Eco, Umberto 1985. Sugli specchi e altri saggi. Milano: Bompiani. Greimas, Algirdas Julien 1970. Del senso. Milano: Bompiani. Haraway, Donna J. 2016. Staying with the Trouble. Durham, London: Duke University Press. https://doi.org/10.2307/j.ctv11cw25q Haushofer, Marlene. 1985. Die Wand. Berlin: Ullstein Taschenbuch Verlag. Haushofer, Marlene. 1999. The Wall. (Whiteside, Shaun, trans.) Berkeley: Cleis Press. Hoffmann, Michael 2000. Verweigerte Idylle. Weiblichkeitskonzepte im Widerstreit zwischen Robinsonade und Utopie: Marlen Haushofers Roman „Die Wand“. In: Bosse, Anke; Ruthner, Clemens (eds.), „Eine geheime Schrift aus diesem Splitterwerk enträtseln…“ Marlen Haushofers Werk im Kontext. Tübingen: Franke Verlag, 193–207. Jakob, Michael 2005. Paesaggio e letteratura. Firenze: Olschki. Jonas, Hans 1993. Il principio responsabilità. Torino: Einaudi. Kalinova, Maria 2018. Exotopy: Mikhail Bakhtin and Jacques Lacan on the outside context of discourse. Slavica Tergestina 1: 98–117. Knecht, Maria-Regina 2007. Recollections of crime and complicity. Studies in 20th & 21st Century Literature 31: 82–108. https://doi.org/10.4148/2334-4415.1645 398 Emanuela Ferragamo Lehmann, Annette Jael 2021. Environments: Künste – Medien – Umwelt: Facetten der künstlerischen Auseinandersetzung mit Landschaft und Natur. Bielefeld: Transcript. Leone, Massimo 2015. Signatim. Profili di semiotica del testo. Roma: Aracne. Lévinas, Emmanuel 1977[1961]. Totalità e infinito. Milano: Jaka Book. Lotman, Yuri 2000. Universe of the Mind: A Semiotic Theory of Culture. (Shukman, Ann, trans.) Bloomington: Indiana University Press. Marchesini, Roberto 2002. Post-human: verso nuovi modelli di esistenza. Torino: Bollati Boringhieri. Meschiari, Matteo 2008. Sistemi selvaggi. Antropologia del paesaggio scritto. Palermo: Sellerio. Milani, Raffaele 2017. L’arte del paesaggio. Bologna: Il mulino. Morton, Timothy 2011. Zero landscapes in the time of hyperobjects. Graz Architectural Magazine 2011, 78–87. Morton, Timothy 2013. Hyperobjects. Minneapolis: University of Minnesota Press. Nelseen, Sarah 2019. Zwei weibliche Robinsonaden? Eine vergleichende Lektüre von Marlen Haushofers „Die Wand“ und „Die Mansarde“. In: Alraud, Sylvie; Lacheny, Marc; Lajarraige, Jacques; Leroy du Cardonnoy, Éric (eds.), Dekonstruktion der symbolischen Ordnung bei Marlen Haushofer. Berlin: Frank & Timme, 127–143. Ponzio, Augusto 2003. Tra semiotica e letteratura. Milano: Jaka Books. Predoiu, Graziella 2016. Raumkonstellationen in Marlen Haushofers Roman „Die Wand“. Germanistische Beiträge 36: 66–88. Ricoeur, Paul 2004. Percorsi del riconoscimento. Milano: Raffaele Cortina Editore. Ronchi, Rocco 2017. Il canone minore: verso una filosofia della natura. Milano: Feltrinelli. Schama, Simon 1995. Landscape and Memory. New York: Alfred A. Knopf. Simmel, Georg 2009[1985]. Philosophie der Landschaft. Stühlingen an der Wutach: Mahler Verlag. Sobral Campos, Isabel 2019. Ecopoetics and the Global Landscape: Critical Essays. Lanham, London: Lexington Books. Stephan, Inge; Weigel, Sigrid 1988. Die Verborgene Frau. Sechs Beiträge zu einer feministischen Literaturwissenschaft. Berlin: Literatur Verlag. Starobinski, Jean 1957. L’occhio vivente. Torino: Einaudi. Venske, Regula 1987. “Vielleicht, daß ein sehr entferntes Auge eine geheime Schrift aus diesem Splitterwerk enträtseln könnte…”. Zur Kritik der Rezeption Marlen Haushofers. In: Duden, Anna (ed.), „Oder war da manchmal noch etwas anderes?“ Texte zu Marlen Haushofer. Frankfurt: Verlag Neue Kritik, 43–67. Venske, Regula 1991. “Schriftstellerin mit der Seele eines Möbelpackers”: Marlen Haushofer. In: Das Schreiben der Frauen in Österreich seit 1950. Wien: Böhlau Verlag, 22–32. Vitta, Maurizio 2005. Il paesaggio. Una storia tra natura e architettura. Torino: Einaudi. Von der Lühe, Irmela 1986. Erzählte Räume – leere Welt. Zu den Romanen Marlen Haushofers. In: Duden, Anna (ed.), „Oder war da manchmal noch etwas anderes?“ Texte zu Marlen Haushofer. Frankfurt: Verlag Neue Kritik, 73–103. Weber, Kurt H. 2010. Die literarische Landschaft. Berlin, New York: De Gruyter. https:// doi.org/10.1515/9783110227642 Zeyringer, Klaus 2001. Österreichische Literatur seit 1945. Überblicke. Einschnitte. Wegmarken. Innsbruck: Haymon Verlag. Marlen Haushofer’s The Wall and the post-nuclear culture of the face 399 «Стена» Марлена Хаусхофера и пост-ядерная культура лица Переплетение ландшафта и лица принадлежит к пространственной эпистемологии человека: как предположил Маттео Мескьяри, примитивные люди привыкли ориентироваться в пространстве, распознавая образы лица. Размышляя над романом Марлена Хаусхофера «Стена», мы ставим под вопрос значение «лица пейзажа» после воображаемого ядерного апокалипсиса, оставившего после себя лишь кошку, корову, собаку, женщину и стену. Стена размывает границу между человеческим и тем, что не является человеком. С точки зрения Роберто Маркесини, она создает телесный ландшафт – гибрид внутренних и внешних ландшафтов, типичный для постчеловеческого сознания. Наконец, кульминацией такого пейзажа становится отказ от существовавшей до апокалипсиса культуры лица, когда лица уже не являются средствами распознавания. Marlen Haushoferi “Üksinda maailmas” (“Die Wand”) ja tuumakatastroofijärgne näokultuur Maastiku ning näo teineteisesse põimumine kuulub inimliku ruumiepistemoloogia juurde: nagu on osutanud Matteo Meschiari, leidsid primitiivsed inimesed maastikul orientiire näomustreid ära tundes. Artiklis mõtiskletakse Marlen Haushoferi romaani “Üksinda maailmas” üle ning esitatakse küsimusi “maastiku näo” semantika kohta pärast kujuteldavat tuumakatastroofi, millest jäävad järele kass, lehm, koer, naine ja müür. Müür ületab inimese ja muu-kui-inimese vahelise piiri: Roberto Marchesini mõisteid kasutades loob see kehamaastiku – posthumaansele teadvusele omase sise- ja välismaastiku hübridiseerumise. Selline maastik kulmineerub lõpuks apokalüpsise-eelsest näokultuurist loobumisega: näod ei toimi enam äratundmisvahendina. 400 Bruno Surace Sign Systems Studies 49(3/4), 2021, 400–417 Semiotics of the pornographic face: From traditional porno to Beautiful Agony Bruno Surace1 Abstract. Today’s pornography constitutes a semiotic laboratory capable of meticulously describing some characteristics of the cultures from which it comes and for which it is intended. In it, the role of the face is preeminent and assumes relevance both from a diegetic and a formal point of view. A face which makes itself a sign and is articulated in a dialectic between the syntagmatic and the paradigmatic axis, finding expression as an aspectual device, establishing a peculiar semiotic procedure of absentia in praesentia, and highlighting an eminently enunciative dimension of the textual genre. Thus a facial semiotics of pornography becomes to all effects a cultural semiotics, which through the exploration of a transversal genre – both in its mainstream and more niche actualizations – produces significant results in defining how cultures of the face, including extrapornographic ones, delineate themselves. The aim of this article is to verify this peculiar facial semiotics through a case history that stretches from traditional to contemporary pornography, also analysing the “facial pornography” website beautifulagony.com and the visual works of some contemporary artists. Keywords: semiotics of pornography; pornographic faces; erotic faces; face studies; semiotics of the face 1. Introduction There is a branch of contemporary pornography in which the censorship of the face is frequent through a peculiar “veil system” (Leone, de Riedmatten, Stochita 2016) effectuated with various techniques (pixellation, face off, masks, etc.). However, it is undeniable that the face plays a fundamental role in a large part of (both early and current) mainstream pornography, performing various functions. First of all, it is a sort of ontological and phenomenological guarantee, which in a sense certifies the humanity of the body dissected by the shots. The mechanical nature of the pornographic sexual act, in fact, based on a standardized repetition 1 Department of Philosophy and Education Science, University of Turin (Unito), Turin, Italy; e-mail address: b.surace@unito.it. https://doi.org/10.12697/SSS.2021.49.3-4.08 Semiotics of the pornographic face: From traditional porno to Beautiful Agony 401 compulsion, requires the face of the actors as proof that they really exist, that they are situated entities which are watching as well as being watched.2 This proof is provided by the editing, through a well-codified grammar, which expertly indulges on the faces of the actors, female and male, interspersing face shots with those of other parts of the body to decrease the degree of impersonality and recompose the fragmented body, also germinating a series of genre variations ranging from facial, to gonzo point of view, to sub-genres such as that of ‘eye contact’ (basically an ‘uninterrupted interpellation’). Thus, we connect to the second function of the face: when present, it is de facto the actualization of the emotional side of pornography. Porn faces are, in terms of tensive semiotics, constantly contracted figures du corps (Fontanille 2004). Prosthetic in their muscularity, exaggeratedly joyful or on the edge of suffering, porn faces are semantically revealed in their enjoyment, in that “beautiful agony” which lends its name to a “historical” viral site (beautifulagony.com), in which the porn is paradoxically founded on the censorship not of the face, but of the rest of the body. Its homepage contains a mosaic of thousands of faces in close-up, continuously posted by users from 2004 to the present day, captured in the act of “la petite mort” (as the site claims). The videos depict its various phases (preparation, achievement, completion, exhaustion), while the thumbnails crystallize its lasting aspectuality, eternally experienced in the facial contraction that sanctions the “facial climax”.3 It is the aspectuality of the orgasmic face as mandatory proof of the authenticity of the pornographic content on which an aesthetic universe hinges that makes the face, “naked and singular” (Lévinas 1984; Ponzio 2007), the true fulcrum of the textual genre. Its power transcends the genre as poster art in films such as Lars von Trier’s Nymphomaniac (2013), but also in the black and white photographs of Lithuanian Albert Pocej, in Argentinian Diego Beyro’s Orgasms series, and in the series that cover the entire range of orgasmic aspectualities (inchoative, lasting, terminative) in Brazilian Marcos Alberti’s O Project.4 2 In this sense we can undoubtedly identify parallels with the phenomenology of Deleuzian, Merleau-Pontyian, and more recently Žižekian derivation. As regards Žižek, given the vastness of his work, we will only mention the 1996 work coedited with Renata Salecl (Salecl, Žižek 1996), the 2004 text dedicated to Deleuze (Žižek 2004a) and the text of the same year on the gaze, edited in the Italian version by Damiano Cantone and Lorenzo Chiesa (Žižek 2004b) as particularly suitable for those wishing to approach this article or the phenomenological philosophy of the Slovenian scholar. As for the closer relationship between semiotics and psychology, the fundamental reference, which also imbues the pages of this essay, is Barthes 1973. 3 Already called so by Elisabeth Lloyd (2005), who studied the expressions of stumptail macaques in the wake of Suzanne Chevalier-Skolnikoff. 4 A semiotic reflection on the orgasmic body in traditional porn cinema can already be found in da Silva 2013. 402 Bruno Surace It is, in other words, a real face-based “cultural epidemiology”, which in the web era mutates in content as in the video O Face, produced by DirtyFit and with over three million views on YouTube, in which the aesthetics of beautifulagony.com is transliterated from the rhetoric of amateurs to a professional production, or in the series of viral videos Hysterical Literature made by the American photographer Clayton Cubitt, where women filmed in the medium field in black and white read famous poems while a vibrator is in action between their legs. The emphasis in all the cases mentioned is always on the face as an ante litteram immersive device, the unknown protagonist of pornography because of its semiotic power of empathic production – if it is true that watching porn corresponds to an “entanglement of somatic and semiotic intensities” (Voros 2014: 249)5 – which, on the one hand, causes reflection of a semiotic and even anthropological nature6 and, on the other, certainly pushes us to interrogate modern neuroscience.7 If you think about it, tout se tient: while watching a film you cry not at the moment of the tragedy, but at the moment in which you see the desperation on the faces of those who are living it inside the narration. 2. The face in porn from the syntagmatic to the paradigmatic axis8 The face in the erotic and pornographic context can be understood as a sign, whose position in the semiosphere is not fixed, but stretched between the peripheries and the centre, by virtue of epochs, genres, geographical locations, gender of the protagonists, that is, within a broad cultural spectrum. In the porn “of yesteryear”, for example, it occupies a syntagmatic, rather than paradigmatic, role. An absence of the face is not substantially envisaged. It is implicated as a sign in the specific construction of the pornographic text, and de facto constitutes an aspectual marker that pre-determines the transition from the durative to the terminative phase. In narrative terms, the close-up focusing on the face of the male actor is a sine qua non for the staging of the “cumshot”, which, as already claimed by Ortoleva (2009: 318), constitutes the pivotal moment of heterosexual pornographic textuality, meaning pornography “as a ‘regime of representation’, a 5 Voros in turn refers to Paasonen 2007. Think of the ethnographic study of the joyful face by Chen et al. 2018, or Fernández-Dols, Carrera and Crivelli’s 2011 study on facial behaviour while experiencing sexual excitement. 7 Cf. Gallese, Guerra 2015; D’Aloia, Eugeni 2017. 8 The sense in which we will use the notions of the syntagmatic and the paradigmatic axes is that traditionally shared by semiotics, which refers to Saussure 1916. 6 Semiotics of the pornographic face: From traditional porno to Beautiful Agony 403 genre assembled out of certain combinations of camera angles and lighting, bodily postures, apparel, footwear, and the like” (Longhurst et al 2017: 306)9. The role of the face is here inevitable, and provides the basis for a theoretical reflection that goes beyond the context of hardcore. In fact, if on a cultural level the male orgasm is given as authentic by definition, as not simulable (and therefore as an act whose semioticity is doubtful)10, it is clear that the emotional-bodily imprint of the face is essential to trigger engagement which otherwise, if relegated to the exclusive profilmic emphasis on the genitals, would be less meaningful. Or at least it is so in the pornography of the 1970s and 1980s, where the female face is equally important, not only as a place where usually the reverse shot of the male face is carried out in the final scene (so frequently that it ends up constituting not just a stylistic feature, but a specific sub-genre, known as ‘facial’, which overflows with compilations in the porn-tubes). The woman’s face, at least in mainstream heterosexual pornography, is in turn the counterpart to the male face. Its task is to signify enjoyment, which is expressed in specific muscular patterns, rotational movements of the pupils, and in general an inventory of facial figures which are the synecdoche of total relaxation, of an aesthetic of abandonment. The role of the face in contemporary pornography is very different. To all intents and purposes there still exist “old-fashioned” films, where the syntagmatics of the montage implies specific moments when the face is central, which is counterintuitive if one thinks of it as the only human surface apparently not of pornographic significance, being an element that is not socially reprehensible (as are the genitals in most human cultures). While the genitals are usually covered at least by underwear, the face is instead the part of the body that is almost always visible. However, the production of films in which the face is completely non-existent is a dominant trend in contemporary pornography, in a strange semiotic inversion: what is usually hidden (the private parts) is emphasized, and what is usually visible (the face) is hidden. This new trend, which undermines professional pornography and sanctions the ineluctable advance of amateur or “fake amateur” porn, is in fact all enacted on an enhancement of anonymity,11 which coincides with a culture that translates porn from a media experience that everyone can enjoy into a factual experience that everyone can practise. It is legitimized porn par excellence, potentially practicable even by the author who is writing these lines, in the total protection of a face that is constantly off screen, or censored through analogue (varied and possible masking) 9 There is also a deepening of a philosemiotic approach in this sense in Kaite 1995. It should, however, be noted that today this notion is more of an urban legend than anything else, because in the pornographic industry there are numerous tricks to simulate male orgasm, as already shown, for example, in Ziplow 1977. 11 A first analysis of anonymity as a semiotic strategy of occultation can be found in Leone 2018. 10 404 Bruno Surace or digital (pixellation, blur effects and so on) cosmetic techniques. In erotic culture the face thus assumes no longer a syntagmatic but a paradigmatic role: it can exist or, as most often happens, not exist. However, and this is the relevant element, the anonymity of this face is not only an individual protection strategy, but also an operation with a significant symbolic charge. If we take, for example, one of the thousands of semi-professional channels on Pornhub (the principal mainstream porn-tube in the world), called ‘Nyna Ferragni’ from the stage name of the Italian actress who animates it,12 we notice from the thumbnails how the semiotic logic of face management configures a relevant dynamic of absentia in praesentia. The woman’s body is clearly the central protagonist (as the name of the channel indicates), although there are in fact almost always two bodies that can be seen in alternation, hers and that of her partner. In fact, it is a typical trend nowadays to open channels for couples, in which people who are partners in real life re-propose their sexuality online, significantly heightening the aura of authenticity around the sexual performance represented. Usually, however, in profilmic terms these movies still identify the female character as protagonist and the male as co-protagonist, a sort of prop, biologically necessary but narratively secondary.13 So Nyna Ferragni is present in all the previews of the videos, while her male co-star appears, to an exclusively genital extent, only in some of them. His face is not visible, and is hidden for framing reasons (he is behind the camera, which he holds in his hand in full gonzo style, or is otherwise “sacrificed” to make room for her body). Instead, she is the protagonist, but her face is not seen, and its presence is averted in the most imaginative ways: she may be off screen, and therefore only her body is visible while her face is cut off by the edges of the frame; she is wearing a hat and her long hair covers a major part of her face; her face is censored by pixels; or she conceals her face with an arm, a pillow and so on. It might not always be completely occulted: sometimes the lower portion is visible, from the mouth to a piece of the nose; but you will certainly never see her eyes. In short, there is an extremely coherent dialectic between the visible and the invisible, which reifies the shift of the classic idea of “see-through”.14 Everything 12 At the time of writing these lines, in December 2020, the metadata of the profile refer to 181 million views of the videos it contains, 259 thousand subscribers, and position 41 in the internal ranking of the site, which in fact is a highly relevant position considering that the site embraces thousands and thousands of internal channels. 13 It should be noted that these considerations relate exclusively to heterosexual mainstream pornography. 14 This discourse can obviously be part of a wider debate on online lifestyles as dynamics in which peculiar relationships between opacity and transparency are implemented from an identity point of view. See Thibault 2016. Semiotics of the pornographic face: From traditional porno to Beautiful Agony 405 else about Nyna Ferragni is absolutely visible, exposed, overexposed; the face, on the other hand, is always veiled, rarefied, impenetrable. If in traditional eroticism everything hinges on the dialectic between concealment and modest revelation of the forms of the body, here, instead, the absent present is the face, catalyst of the attention of a new spectatorship whose sense of excitement is built around an identity which has been halved, and which therefore through interpretative cooperation is partially filled with a new order of desire. If in earlier times the porn actress/actor was clearly identifiable as such, and resided in a specific and symbolically inaccessible hyperuranion (so much so that many films were based on the idea of “casting” in which ordinary people could engage in sexual intercourse with these clearly recognizable stars),15 today, however, s/he could be anyone, and so we fall within the paradigm of the “pornification of everyday life”, which finds one of its maximum elements in the new culture of the pornographic face, together with the predominance of amateurism, the explosion of porn in public, and certainly also the burgeoning industry of live porn through webcam.16 Therefore, if the contemporary face becomes rarefied in pornography, it is legitimate to ask whether it has also abandoned its relevance. In reality, as we are discovering, the opposite is true. The hidden face is a face that exists precisely as absent or semi-present. Moreover, this was already true in early porn. When the hidden face was not yet a defined filmic object, an aesthetic of the absent face was already taking shape, as all pornography – first photographic, and in the era of deepfake also in motion17 – was based on photomontage. This operation is nothing more than the replacement of one face with another, that is, the affixing of a face on a body that does not belong to it, thus coinciding with the passage from a virtuality (the particular famous person who is considered sexy, whom you would like to see naked in the act of a sexual performance) to an actuality (the realization of the fantasy through the manipulation of images of faces forcibly displaced from their body or belonging to a different one that satisfies the perceived aesthetic needs): 15 Many successful pornographic series are born from this narrative pretext, from the historical castings of the Italian actress Eva Henger (A letto con Eva), who “auditioned” amateurs who were not particularly handsome and had no script, so as to give the viewer the idea that even s/he could enjoy that experience, up to more contemporary formats like Mea Melone CHALLENGE!, Fuck a Fan and many others. 16 The shift from the production of content designed to be deferred to that of content designed to be streamed live online is a rapidly growing fact, which has seen a boost during the pandemic period. Pornography has not been exempt from this increase in interest in live content, as evidenced by the increment in users on sites like OnlyFans. Cf. Lykousas, Casino. Patsakis 2020. 17 An analysis on deepfake with a semiotic approach appears in McCosker and Wilken 2020. 406 Bruno Surace Even before the look-alike video, the net overflows with evident photomontages where the face of a determined personage is mounted onto the body of some anonymous porn star, intent on performing the most varied pornographic acts. […] On the part of the broadcaster, the creation of pornographic photomontages with stars as protagonists may be based on two end-goals: 1. Recreational: the broadcaster elaborates the fake for the sole purpose of introducing it onto the Web to trigger trolling activities or because he considers it an exhilarating way of mocking the unwary. 2. Lucrative: the broadcaster elaborates the fake from a desire to benefit from it in terms of visits to a particular web video (clickbaiting), the purchase of products, subscriptions and the like. Whatever the broadcaster’s motivation, the focus of our interest are the users. Excluding the community of “recreational users,” who recognize the fake and enjoy themselves making it circulate as such, there is certainly a slice of users who utilize these images because they find them arousing. The interesting fact is that these images appear for the most part manifestly fake. They consist either of photomontages recounting highly improbable narrative situations (a famous actress in the middle of an orgy who smiles dazzlingly at the camera) or very poorly-finished productions (where the viewer can note a difference in color between face and body, a lack of proportion and so on). (Surace 2019: 248) Thus, the pornographic face allows us, above all, to undermine the idea of the face as an excessively biological or natural device. While there is certainly a naturalistic foundation in the movements and expressions of the face, which is difficult to falsify, it is also true that the cultures of porn transform the face into a semiotic device that acts on various layers. The first is a presential layer: the face is or is not there, and its presence in the absence or vice versa is a semiotic fact; the second is an actorial layer: the pornographic face, in the act, recites and is actorialized in Greimasian terms, that is, it has semantic properties that articulate value and aspectual dimensions, contributing to the development of a narrative that the user converts patemic-bodily onto themselves (even seeking out and concentrating on some salient moments and avoiding others, since for all intents and purposes a porn film is almost never watched in its entirety);18 the third is the layer of the post-produced face, which makes pornographic faces the places where the hidden 18 The principal mainstream pornographic sites are increasingly built around categorizations and metadata, not only designed to direct users to the videos they might like most, but even manipulating the videos themselves with various types of infographic tags, such as signs in the playback bar that indicate different “scene types” (that is, the types of position in the narrative grammar of pornographic performance), or even the histogram index of the attention curve on a single video (so that the user will directly select the most viewed parts, rather than others). Pornhub itself releases a complex report every year with many data, capable of closely tracing user profiles: https://www.pornhub.com/insights/2019-year-in-review (last consulted on 14 December 2020). Semiotics of the pornographic face: From traditional porno to Beautiful Agony 407 desires of the cultures to which they refer are most fulfilled, often dichotomously (hyper-made-up faces vs “girl-next-door” faces, adult faces and therefore with wrinkles in sight (MILF) vs innocent faces (teen), angelic faces vs aggressive faces, and, again, an emphasis on the colour of hair, eyes, skin, etc). The face is therefore configured as a completely culturalized semiotic and phenomenological device. In this way, pornography allows us to verify the culturality of the face as claimed by Broekman (2011: 24–25): “1. Human faces are not a natural fact but should be understood as a cultural value; 2. A human face is in semiotic perspective a body part that shares its artifice-character with the other parts of the body; 3. The artifice-character plays a key role in the semiotic and legal need for name-giving; 4. Faces are thus faces in an age of semiotics”. In porn, consequently, the face is rendered as a mask, and pornography as an effective laboratory in which to explore desire as a socially shared, potentially divisive, politically relevant construct. If this were not the case, then the role of the face in porn would not change, and the pornography of the past would be similar to that of the present. 3. Aspectuality of pornographic faces If we have so far highlighted how the face is not a contour element in pornographic cultures, but a real dominant element, we now focus on those erotic textualities where the centrality of the face is not only functional to the development of the “arousal arc”, but proudly disclosed. The most relevant case is the aforementioned website Beautiful Agony, whose subtitle is “Facettes de la petite mort”. Founded in 2004, still active (which already demonstrates its success), and managed by Feck Pty Ltd in Melbourne, the site presents itself with a template à la old-fashioned hypertext, with a slender menu on the left and a central gallery in constant update of numbered videos shared by users, called Agonees.19 The visual formatting of the videos follows a specific protocol, strictly supervised by the moderators (who, moreover, have the right to delete videos in which the simulation is too evident). The thumbnails are again proof of visual expectancy: in a close-up from the shoulders up, men or women are portrayed at the moment of maximum enjoyment. As Dean MacCannell (1989: 155) states in Faking It: Comment on Face-Work in Pornography, in sexuality “the dominant mode of communication is via facial expression, gesture, and pre-linguistic vocalization”. If this is true both in everyday sexuality and in pornography, there is a rather large facial difference between the two: 19 Reflections in part similar to those we will conduct on this site can be found in Schaschek 2014, who, however, defines the site as “quasi-pornographic”, while we are in all respects convinced of its full “pornness”. 408 Bruno Surace The pattern of opposition between pornographic and everyday interaction also appears in “face-work”. Emotional extremes in ordinary a-sexual interaction are usually signified by upper/lower face agreement or complementarity. For example, excited joy-tending-toward-rapture in the framework of everyday interaction is expressed by relaxed wide eyes and open mouth. Extremes of disappointment are expressed by a facial “crunch”, the mouth and eyes close tightly and are pulled toward the center of the face. In each case, there is a muscular agreement between upper and lower, and left and right face. Expressions of emotional extremes in the pornographic frame often involve upper and lower face muscular antagonism. The eyes are open wide but the mouth is closed; sometimes the teeth are employed to keep the mouth closed. Pornographic ecstasy is expressed by the opposite form of opposition: the eyes are closed and the mouth is slack open. It is semantically similar to a-sexual expression of pain. The difference seems to be that ecstatic pornographic expression does not involve tight closure of the eyes or knitting of the brows; the eyelids are merely dropped, and the forehead remains smooth. The expression that suggests the actual experience of pain is inevitably accompanied by a heavily knotted brow and tight wincing of the eyes. The pornographic frame is often the site of spectacular whole-face muscular antagonisms that are not found in other areas of life. (MacCannell 1989: 159) Thus the faces on Beautiful Agony are real porn, to the extent that the muscular oppositions between the top and the bottom of the face, which translate into tensive dyscrasias (one hypertensive band, the other extended), are typical of that exaltation of expressivity which, for example, in softcore or classic eroticism (that, according to an ancient dichotomy, would constitute the aesthetically worthy representation of the erotic as opposed to the alleged brutal obscenity of pornography) is instead supplanted by mono-expressiveness, an “expressionless state” (MacCannell 1989: 170). The face in the Australian site is also extreme, above all because it supplants the nudity that common sense usually associates with the term. The small portion of the body visible to us in the images and videos is often clothed at least in underwear, if not completely. What is naked is the face, abandoned to ecstasy and so magnified in its singularity. A singularity, moreover, in which perhaps the glimmer of the truth lies, if it is true that the voice can lie, and that “the first mirror is the smiling face of an observer, signifying a semiotic process” (Makolkin 2015: 574). In addition, it would seem that in fact the falsification of jouissance is actually easier vocally than facially: A recent experiment studied the sounds women make during sex with their male partners. It turns out that these sounds have nothing to do with whether the woman is having an orgasm. The timing, frequency, and intensity of the sounds were a response to the women’s male partners, and were interpreted quite reasonably by the researchers as a way of encouraging the men to “hurry up and Semiotics of the pornographic face: From traditional porno to Beautiful Agony 409 finish.” One, uncharitable way of interpreting such behavior, then, is a form of deception: a “fake orgasm.” But that would be too harsh. It is really more like Pavlov’s dinner bell, only that there is no intention of depriving the dog of his dinner. (Yelle 2013: 58)20 Beautiful Agony thus deactivates the strong categorization of desire of mainstream porn, sold according to packet switching methods that take place in the cornucopia of labels of mainstream porn sites, where a semantic indicator is associated with every possible micro-characteristic (characters, scenes, and so on). The centrality of the face as a transcultural erotic device is in fact such as to reduce categorization to a minimum choice, based on gender (Females, Males, Trans/Non Binary) and on three further subgenres (Friends, Confessions, Revival) on which we will not focus here. There are two points worth expanding on. The first is the central role of the visual representation of orgasm through the face. Again, the pornographic context is useful for drawing general philosophical and semiotic conclusions. The success of this site is indeed empirical proof of Emmanuel Levinas’ definition, which otherwise risks remaining somewhat vague and unclear, of the face as a naked and singular device (so singular that, except for a small minority, the videos on the website are almost always of a single protagonist, usually intent on the masturbatory act). What does Levinas actually mean? Our interpretation is that he is talking about a threshold device, in which the clash between semiotic and non-semiotic, that is, between natural and cultural, occurs.21 It is the idea of the orgasm as a moment of abandonment, a “petite mort”, a paradoxical expression that minimizes an otherwise absolute and incontrovertible moment – death – by regimenting it in a transitory dimension that is otherwise excluded (you do not usually die only for a while, except maybe in cases of people who report near-death experiences). And this peculiar moment occurs vividly rather than linguistically; it is not communicated by diegesis, but by mimesis. So the faces 20 In reality, there are also some doubts about the face, especially regarding studies on the socalled “climax face”, detailed in Lloyd 2005. 21 It is worth pointing out here that the theme of the presumed distinction between natural and artificial signs, between the semiotic and non-semiotic world, is still the subject of debate today, and indeed assumes renewed importance with the increasingly relevant dialogue with disciplines of more recent formulation (such as neuroscience). The identification of a “lower threshold of semiotics”, as it is designated in Eco 1975, is a complex problem that we have no intention of addressing here. Our interpretation of Levinas in these terms is therefore to be understood not from the standpoint of whoever emits the sign, but from that of the effect of meaning on whoever receives it, experiencing a sensation of “naturalness” as opposed to a previous “artificiality”. 410 Bruno Surace on Beautiful Agony seem to emit meaning involuntarily, and this is constituted as such when it is traced by the user of the website, who is in turn automatically aware of this non-intentionality (whether it is simulated or not), which is the core of the experienced excitement. That is, these faces are signs that stand for nothing else – aliquid pro nihilo – placed in a median phase in which the loss of control (simulated or not, this is not the point) causes muscular and ocular reactions. These reactions are not fully “supervised”. More than surveillance of the self, the faces from Beautiful Agony are in a state of souveillance,22 in the lower threshold of semiotics. The abandonment of significance in semiotic terms is not the embracing of insignificance,23 and perhaps it is not even appropriate to read a transcendent device in the face. Everything is played out on a specific, textual immanence level, within the frame. However, the face takes on meaning as the ultimate device, whose subversive charge – a linguistic subversion that repudiates embarrassment and repression24 – is the isotopy of Beautiful Agony, which shows us faces as they should not be seen culturally. In some ways they are a visual profanity. The second important point is represented by the Trans/Non Binary category. The immediately significant data are the faces that we find within this category. In fact, if we dwell on them, we very rarely identify faces which are visually Trans/ Non Binary (for example, faces with masculine features but made up in such a way that the common imagination tends to associate them with female faces). In most cases, indeed, these faces appear to us as sexually unambiguous. Now, of course, the reason for the existence of this category therefore appears more relevant in terms of identity, rather than visuality. Trans/Non Binary users of the site, in fact (as well as other users interested in the category), presumably already have the possibility of satisfying their sexual appetites in the Males and Females categories from a visual point of view, according to their preferences. The correspondence or not between gender and biological sex is not necessarily traceable on the face. Furthermore, trans-sexuality in pornography is a genital fact, and in Beautiful Agony the genitals are taboo. Three consequences follow: the first is that the face is something that is given visually, but in reality contains in a nutshell a substratum of axiological-ideological, and therefore semiotic, components that go beyond its visualness; the second is that the face as a synecdoche of the body in pornography presupposes a physicality, 22 We do not use this term here in the sense that is attributed to it in the debate on the contemporary panopticon from Mann 1998 (later consolidated in Mann 2004) onwards, but as a term to describe a state of reduction of self-consciousness until reaching an a- or pre-semiotic condition. 23 And in fact, even the most recent semiotic studies are starting to question insignificance, as is demonstrated by Leone 2019. 24 On the re-semantization of repression as a semio-political act, see Surace 2019. Semiotics of the pornographic face: From traditional porno to Beautiful Agony 411 so that it is in fact taken for granted that in a “straight” porn film a female face corresponds to female genitals, but this is not the case for a face categorized as Trans/ Non Binary, about which a regime of doubt remains (both male and female genitals could correspond to a male or female face); the third is that the face claims, as an increasingly transcultural device, the overcoming of an archaic binarism which, although today beginning to disclose sparse visual manifestations (usually through status symbols like rock stars or influencers who make themselves up specifically to achieve an imaginary “gender fluidity” and disrupt the traditional visual dichotomy of genders), is already assimilated on the level in which the face is mediated as a discursive device, and therefore categorized. It is the distinction between seen face and said face, which causes this category to exist in Beautiful Agony, meaning that transsexuality does not necessarily pass through the face, but proudly finds its confirmation in the face. In fact, therefore, those who prefer, for example, videos #4065 or #4643, included in the Trans/Non Binary category but which from the preview seem to depict people who could be included in the Females category, will see in these the face of a non-binary sexuality. Behind the visual appearance of the faces there is always a political, semiotic, identitary instance. However, Beautiful Agony is not the only place where the pornography of the face reaches its peak. If the face already plays a radical role in mainstream pornography, here this is further consolidated in some specific sub-genres such as gonzo, where the subjective point of view is magnified by the actors’ faces in the foreground, and the frequent indulging in eye contact (what, in cinematographic terms, is called interpellation). Yet the power of the face as an erotic device had already been investigated in earlier times, when certain themes were still a socially shared taboo, and all the specifics we are talking about had not even been codified, if we take the 1970s as a moment of diffusion in the West of audiovisual pornography derived from magazines. In 1964 Andy Warhol made a silent, black-and-white medium-length film entitled Blow Job. Warhol’s film could to all intents and purposes be understood as a text in which a fellatio is shown and therefore, in fact, as a pornographic film. In reality, however, the history of modern pornographic cinema (excluding the experiments of early cinema) usually traces the genre’s first film to the slightly later Deep Throat (Gerard Damiano 1972), although other films with simulated sexual acts, without real penetration, had been shot previously. Why is Warhol’s film not mentioned? Because, in the artist’s typical playful style, it betrays the expectations raised by the title. The entire duration of the film is in fact a fixed sequence closeup of DeVeren Bookwalter (uncredited), wearing a jacket with a high collar, who squirms with delight for several minutes, now grinding his mouth, now raising his face and eyes skywards, now touching his hair and, finally, after appearing to pull 412 Bruno Surace up his pants, lifting a cigarette to his lips. It is to all intents and purposes a film in Beautiful Agony style, ante litteram. The operation is clearly provocative, and plays on the way in which the title topicalizes the long shot. In fact, on the isotopic level, at least until the moment of lighting the cigarette (which has a precise cultural correspondence with the post-coital moment), the visual text does not necessarily suggest a fellatio. The actor’s face is illuminated by a strong light, which often overshadows his eyes, generating dark eye sockets through a play of chiaroscuro. His movements are sometimes convulsive and sometimes more relaxed. The mouth often showily quivers: The brief, fleeting, and erratic moments of this failure in the face work of Blow Job’s poser are located at the points where his face does, in fact, appear to index unambiguous sexual pleasure via certain “genuine,” that is, nonstrategic facial contortions. Correlatively, in the face work of porn performers, the failure to screen the trauma can be located in those moments that indicate the subject’s momentary loss of control over his or her own strategic signification […]. This failure can take two forms: First, it may be registered as an expression that becomes briefly yet unambiguously identified as “phoniness,” and, second, taking the same effect as with Blow Job’s poser’s failure, it may be registered in an expression of genuine (nonprofessional) sexual pleasure. (Grundmann 2003: 36) This consideration ultimately demonstrates a certain visionarity on the part of Warhol, who had at the same time foreseen the role of the face in contemporary porn, but also somehow the kitsch substratum that would make the first “official” pornographic films famous a few years later. 4. Artistic and enunciative cultures of the “O Face” These theoretical reflections based on the face spring from the pornographic context in such a vigorous way that they are also intercepted by the production of artistic speeches and texts. A first important case is that of Albert Pocej, who decided to immortalize with photographic shots, and sometimes with time-lapse, 15 women in the moment of orgasm, producing a series of images as a sort of photographic exploration.25 The face is again central, crystallized in different poses. It is, however, no longer a foreground face. From a plastic point of view, the images are in black-and-white, as was Warhol’s film – halfway, in fact, between 25 The series can be viewed at the following link: https://www.boredpanda.com/femaleorgasms-photography-albert-pocej/?utm_source=ro.pinterest&utm_medium=referral&utm_ campaign=organic (last accessed on 14 December 2020). Semiotics of the pornographic face: From traditional porno to Beautiful Agony 413 artistic and pornographic performance (first of all a “discursive pornography”, as we have established). The black-and-white photography emphasizes the whimsical character of the operation, giving it an aura of artistry, as if to signify a discourse of a certain caliber. In topological terms, nevertheless, the faces are this time in direct relationship with a background, which takes on a meaning, sometimes equal (on the eidetic level the surface covered is quantitatively the same, half and half), sometimes superordinate, as in the photograph in which the face of the woman portrayed is at the bottom right, dominated by the background of some numbered tree trunks, or in the shot where the woman has an orgasm against the backdrop of a wind turbine. Particularly significant in this sort of laparoscopy of the female orgasmic face is the photograph in which the dark background is abruptly interrupted by a beam of light which, from above, almost divinely, strikes only and exclusively the small face of the woman portrayed in her moment of ecstasy26. There is thus no close-up in a technical sense, but the lighting arrangement generates a conceptual close-up: the face in the moment of abandonment lights up; everything else for a moment – the very instant of the shot – loses its meaning, and in fact the background is black, almost unrecognizable. Another case along the same lines is that of Diego Beyró’s Orgasms series, in which a peculiar aesthetic of the surface intervenes.27 Here the face returns to the foreground, but instead of photographs, we are dealing with oil paintings on bedsheets, 210 × 130 cm in size, which by their conformation lend themselves to being spread and waved, thus giving a certain airiness to the content represented, fixed on the sheet, but at the same time mobile in the air. It is also relevant that if before the face was first and foremost an index sign, photography being the physical trace of an event that took place, here we move into the sphere of iconism, mixed with the surface of the sheets which with their individual patterns (floral and geometric motifs of different workmanship), and with the paradigm of iridescent ripples according to where they are affixed, constantly re-signify these faces as paradoxical expressions of la petite mort, always the same but also constantly different. In the same vein, we can mention the work of Marcos Alberti, who with his The O Project proposes a series of images that, starting from a neutral face in the upper left corner, photographically recount the achievement of ecstasy through a visual narrative of four foreground images, against a neutral background and articulating a grammar of passion that is expressed in a dialectic between tension and relaxation, both interior – the face that changes expression – and exterior – 26 On the role of the face in ecstasy, not only sexually, but in the broadest meaning of the term, cf. Surace 2018. 27 The series can be viewed at the following link: http://www.diegobeyro.com/orgasm-series#1 (last accessed 14 December 2020). 414 Bruno Surace the face that claims the right to its own impertinence, no longer respecting the original system of proportions envisaged by the frame of the shot, and therefore going out of view or changing its relationships with the depth of field and moving towards the fore.28 This last consideration highlights a politics of enunciation to which the face is usually subjected, depending on the format that regulates it (selfie, class photo, passport photo, mirror, home film and so on), and that pushes us to consider it a device that should not be read exclusively within its own borders, but also in a conscious relationship with the environment that contains it, both in the semiotic space of representation, and in that of everyday experience. Experiments such as those reported above, starting from the controversial unhinging of the taboos linked to sexuality and the specifics of orgasm as a moment of escape from a strictly linguistic order of things (although, but this is the game of semiotic analysis, “trapped in the linguistic cage of the text”), reveal how the investigation in this specific direction can actually produce pertinent results for a more general cultural semiotics of the face, whose media relevance appears, in conclusion, to be exquisitely contemporary. It is in fact the liquid society of digital media that allows the development of a pragmatics of the represented face that can move easily. In the past, the orgasmic face was simply relegated not even to the margins, but in the dark area of the unculture, that is, the unspeakable and the unrepresentable. Today, however, in an era in which nudity is ordinary, pornography socially metabolized, and sexuality no longer the subject of discussion as such but rather a symbol of battles for the achievement of rights, the face takes on its own public role as a privileged support for proposing a shared semiotics of reappropriation, of redemption of impertinence. Thus, on the interpretative level, there are two ways of reading the video O Face presented by DirtyFit Apparel.29 On the one hand, it will certainly be agreed that this is an advertising operation, conceived by the brand to involve the audience and somehow secure a place within a specific imaginary, built around the enhancement of certain libertarian, if not even socially anarchoid, values. On the other hand, however, it will be inevitable to conclude that the sequence of faces of people of every gender, ethnicity and age, captured in the foreground, sometimes with emphasis on details (the trembling mouth, the widening nostrils and so on) during their moment of ecstasy, constitutes the symptom and at the same time the result of a strong cultural renegotiation, which passes through the face, of the elastic 28 The work can be viewed at the following link: https://www.marcosalberti.com/oproject (last accessed on 14 December 2020). 29 The video can be viewed at the following link: https://www.youtube.com/watch?v= mISaDN65eX8&has_ verified=1 (last accessed on 14 December 2020). Semiotics of the pornographic face: From traditional porno to Beautiful Agony 415 distinction between private and public, between intimate and exhibitable, between unbecoming and licit. In a semio-politics of bodies, the face obviously plays an indispensable and definitive role. Acknowledgements: This article is part of FACETS – Face Aesthetics in Contemporary E-Technological Societies, a project funded by the ERC (European Research Council) under the Horizon 2020 programme (Grant agreement 819649). References Barthes, Roland 1973. Le plaisir du texte. Paris: Seuil. Broekman, Jan 2011. Faces face to face. In: Broekman, Jan M.; Mootz III, Francis J. (eds.), The Semiotics of Law in Legal Education. London, New York: Springer, 21–56. https:// doi.org/10.1007/978-94-007-1341-3_2 Chen, Chaona; Crivelli, Carlos; Garrod, Oliver; Schyns, Philippe; Fernández-Dols, JoséMiguel; Jack, Rachael 2018. Distinct facial expressions represent pain and pleasure across cultures. PNAS – Proceedings of the National Academy of Sciences of the United States of America 115(43): E10013–E10021. https://doi.org/10.1073/pnas.1807862115 D’Aloia, Adriano; Eugeni, Ruggero 2017. Teorie del cinema. Il dibattito contemporaneo. Milano: Raffaello Cortina. Da Silva, Odair José Moreira 2013. A significação e o sentido do corpo orgástico no cinema pornô. Signum: Estudos de Linguagem 16(1): 149–170. https://doi.org/10.5433/22374876.2013v16n1p149 Eco, Umberto 1975. Trattato di semiotica generale. Milano: Bompiani. Fernández-Dols, José-Miguel; Carrera, Pilar; Crivelli, Carlos 2011. Facial behavior while experiencing sexual excitement. Journal of Nonverbal Behavior 35(1): 63–71. https:// doi.org/10.1007/s10919-010-0097-7 Fontanille, Jacques 2004. Soma & séma: figures du corps. Paris: Maisonneuve & Larose. Gallese, Vittorio; Guerra, Michele 2015. Lo schermo empatico. Cinema e neuroscienze. Milano: Raffaello Cortina. Grundmann, Roy 2003. Andy Warhol’s Blow Job. Philadelphia: Temple University Press. Kaite, Berkeley 1995. Pornography and Difference. Bloomington, Indianapolis: Indiana University Press. Leone, Massimo 2018. I giga di Gige. L’impatto dell’anonimato nella comunicazione contemporanea. RIFL/SFL – Rivista italiana di filosofia del linguaggio 1: 101–109. Leone, Massimo 2019. On Insignificance: The Loss of Meaning in the Post-Material Age. London, New York: Routledge. https://doi.org/10.4324/9780429461279 Leone, Massimo; De Riedmatten, Henri; Stochita, Victor (eds.) 2016. Il Sistema del velo. Trasparenze e opacità nell’arte moderna e contemporanea. Roma: Aracne. Lévinas, Emmanuel 1984. Paix et proximitè. Cahiers de la nuit surveillé 3: 339–346. Lloyd, Elisabeth A. 2005. The Case of the Female Orgasm: Bias in the Science of Evolution. Cambridge, London: Harvard University Press. 416 Bruno Surace Longhurst, Brian; Smith, Greg; Bagnall, Gaynor; Crawford, Garry; Ogborn, Miles 2017. Introducing Cultural Studies. London, New York: Routledge. https://doi.org/10.4324/ 9781315690070 Lykousas, Nikolaos; Casino, Fran; Patsakis, Constantinos 2020. Inside the X-rated world of “premium” social media accounts. In: Samin, Aref; Bontcheva, Kalina; Bragieri, Marko; Dignum, Frank; Jannotti, Foska; Grisolia, Franchesko; Pedreschi, Dino (eds.), Social Informatics. 12th International Conference, Pisa, Italy, October 6–9, 2020. Proceedings. Cham: Springer, 181–191. https://doi.org/10.1007/978-3-030-60975-7_14 MacCannell, Dean 1989. Fakin it: Comment on face-work in pornography. American Journal of Semiotics 6(4): 153–174. https://doi.org/10.5840/ajs19896410 Makolkin, Anna 2015. Face as a sign and Paolo Mantegazza’s theory of metoposcopy. In: Trifonas, Peter Pericles (ed.), International Handbook of Semiotics. London, New York: Springer, 573–588. https://doi.org/10.1007/978-94-017-9404-6_26 Mann, Steve 1998. Reflectionism and diffusionism. Leonardo 31(2): 93–102. https://doi. org/10.2307/1576511 Mann, Steve 2004. “Sousveillance”: Inverse surveillance in multimedia imaging. MULTIMEDIA ‘04: Proceedings of the 12th annual ACM international conference on Multimedia, 620–627. https://doi.org/10.1145/1027527.1027673 McCosker, Anthony; Wilken, Rowan 2020. Automating Vision: The Social Impact of the New Camera Consciuosness. London, New York: Routledge. https://doi.org/10.4324/ 9780429341175 Ortoleva, Peppino 2009. Il secolo dei media. Riti, abitudini, mitologie. Milano: Il Saggiatore. Paasonen, Susanna 2007. Strange bedfellows: Pornography, affect and feminist reading. Feminist Theory 8(1): 43–57. https://doi.org/10.1177/1464700107074195 Ponzio, Augusto 2007. Fuori Luogo. L’esorbitante nella riproduzione dell’identico. Roma: Meltemi. Salecl, Renata; Žižek, Slavoj (eds.) 1996. Gaze and Voice as Love Objects. Durham, London: Duke University Press. https://doi.org/10.2307/j.ctv11smjs5 Saussure, Ferdinand de. 1916. Cours de linguistique générale. Lausanne, Paris: Payot. Schaschek, Sarah 2014. Pornography and Seriality: The Culture of Producing Pleasure. New York: Palgrave MacMillan. https://doi.org/10.1057/9781137359384 Surace, Bruno 2018. Sintassi, semantica e pragmatica del martirio attorno a Martyrs di Pascal Laugier. Lexia 31/32: 351–367. Surace, Bruno 2019. The unbridled meaning of unsignified signifiers from paraliterature to cinema. In: Idone Cassone, Vincenzo; Ponzo, Jenny; Thibault, Mattia (eds.), Languagescapes: Ancient and Artificial Languages in Today’s Culture. Roma: Aracne, 131–143. Surace, Bruno 2019. Where porndom meets stardom. In: Jacob, Frank (ed.), Pornography: Interdisciplinary Perspectives. Berlin: Peter Lang, 239–260. Thibault, Mattia 2016. Untag yourself. Opacità e trasparenze negli stili di vita online. Carte semiotiche 4: 103–118. Voros, Florian 2014. Gendered embodiment in audience reception practices of pornography. In: Biasin, Enrico; Maina, Giovanna; Zecca, Federico (eds.), Porn after Porn: Contemporary Alternative Pornographies. Milano, Udine: Mimesis. Yelle, Robert 2013. Semiotics of Religion: Signs of the Sacred in History. London, New York: Bloomsbury. Semiotics of the pornographic face: From traditional porno to Beautiful Agony 417 Ziplow, Steven 1977. The Film Maker’s Guide to Pornography. New York: Drake Publishing. Žižek, Slavoj 2004a. Organs Without Bodies: Deleuze and Consequences. New York, London: Routledge. Žižek, Slavoj 2004b. Dello sguardo e altri oggetti: saggi su cinema e psicoanalisi. Pasian di Prato: Campanotto. Семиотика порнографического лица: от традиционного порно до Прекрасной Агонии Современная порнография представляет собой семиотическую лабораторию, способную скрупулезно описывать некоторые характеристики культур, в которых она создается и для которых предназначена. Лицо играет здесь главенствующую роль и приобретает особое значение как с диегетической, так и с формальной точки зрения. Лицо, становящее знаком и артикулируемое в диалектике между синтагматической и парадигматической осью, находит выражение в качестве аспектуального приема, устанавливает своеобразную семиотическую процедуру “absentia in praesentia” и подчеркивает крайне выразительное измерение текстового жанра. Таким образом, в порнографии семиотика лица становится семиотикой культуры во всех смыслах, которая благодаря исследованию этого трансверсального жанра – как в его основном, так и в более нишевых актуализациях – дает значительные результаты в определении того, как культуры лица, включая непорнографические, себя определяют. Цель этой статьи – проверить эту гипотезу, прослеживая историю вопроса от традиционной до современной порнографии, а также анализируя веб-сайт «порнографии лица» beautifulagony.com и работы некоторых современных художников. Pornograafilise näo semiootika: traditsioonilisest pornost Kauni Agooniani Tänapäeva pornograafia moodustab semiootilise laboratooriumi, mis võimaldab üksikasjaliselt kirjeldada mõningaid iseloomulikke jooni kultuuris, millest see on pärit ja mille jaoks mõeldud. Näole kuulub selles väljapaistev roll ning see osutub oluliseks nii diegeetilisest kui ka vormilisest vaatepunktist nähtuna. Nägu, mis muudab end märgiks ning mida väljendatakse süntagmaatilise ja paradigmaatilise telje dialektikas, leiab väljenduse aspektuaalse võttena, mis kehtestab “absentia in praesentia” omapärase semiootilise protseduuri ning rõhutab teksti žanri ülimalt enuntsiatiivset mõõdet. Seega saab pornograafias näosemiootikast igas mõttes kultuurisemiootika, mis mitut valda ületava žanri uurimisel annab olulisi tulemusi defineerimaks, kuidas näokultuurid, sealhulgas ka pornograafiavälised, end määratlevad. Artikli eesmärgiks on kinnitada seda omapärast näosemiootikat juhtumiajaloo abil, mis ulatub traditsioonilisest kuni kaasaegse pornograafiani, analüüsides ka “näoporno” veebilehte beautifulagony.com ning mõnede kaasaegsete kunstnike visuaalset loomingut. 418 Antonio Santangelo Sign Systems Studies 49(3/4), 2021, 418–436 Culturally significant symbolic faces: For a sociosemiotics of faces in films Antonio Santangelo1 Abstract. Every now and then when watching a movie, we come across faces in which we recognize a significant value, because they represent some important cultural models we use to assign meaning to our experience of the world. By way of example, I will discuss the faces of the protagonists of two recent films, Abdellatif Kechiche’s La vie d’Adèle. Chapitres 1 & 2 (2013; English title Blue Is the Warmest Colour) and Léonor Sérraille’s Jeune femme (2017), comparing them with the faces of the protagonists of some older movies, such as Andrew Adamson and Vicky Jenson’s Shrek (2001) and James Cameron’s Avatar (2009). I will argue that the way in which the faces are portrayed is similar to the narrative structure of the stories of the characters they belong to, and that the signs and narrative structures used to construct the discourses about the world in those films are at the same time similar to those of two important cultural models of what it means to be young men and women in our times. As these cultural models are different, yet interconnected, I will argue that the most meaningful faces in cinema change due to the transformation of the cultural models they derive from and that a sociosemiotic method based on a structuralist vision of culture can help identify the most culturally significant symbolic faces on screen, and elsewhere. Keywords: sociosemiotics; cultural models; narrative structures; structuralism; symbolic faces The faces of those who do not find meaning in our society This article deals with developments in a research study which started in 2017 and has already led to the publication of Santangelo 2017 and Santangelo 2021. It is based on the idea that there are images of faces that take on great symbolic value in a specific historical period, because their meaning depends on certain cultural models that define what it entails to be men or women in that time, compared to 1 Department of Philosophy and Education Science, University of Turin (Unito), Italy; e-mail: antonio.santangelo@unito.it. https://doi.org/10.12697/SSS.2021.49.3-4.09 Culturally significant symbolic faces: For a sociosemiotics of faces in films 419 other cultural models which are embodied symbolically by different faces. Roland Barthes had already noticed this in his analysis of today’s “myths”, as he focused on the contrast between the faces of Greta Garbo and Audrey Hepburn in films that circulated in the 1950s. According to the French semiologist, the images of these two actresses admirably represent the moment in which cinema was about to extract an existential beauty from an essential one (Barthes 1994[1957]: 64). Just like Barthes, in this article I intend to consider the meaning of some cinematic faces that embody cultural models relevant in contemporary society. These are the faces of the leading actresses of the films La vie d’Adèle: Chapitres 1 & 2 (Kechiche 2013; English title Blue Is the Warmest Colour) and Jeune femme (Sérraille 2017). I wish to demonstrate that the way the shots of their close-ups are constructed – with a very mobile hand-held camera, frantically searching for the focus of the image, finding it in the sensory organs of the two girls, with their eyes, noses and mouths that “experience” the colour, smell and taste of the world, while not being integrated visually into the surrounding context – resembles the overall narrative structure of the stories of Adèle and Paula, the characters played by Adèle Exarcopoulos and Laetitia Dosch. This narrative structure has been recognized by the interpreters of these films as fundamental to understanding the meaning of our lives today. This is why I wish to speak of culturally significant symbolic faces, following a tradition in the field of film studies that, according to Casetti (1994: 28–32), stretches from Aristarco (1951) to Lukács (1963), authors who argue that the so-called ‘impression of reality’ (Rastier 1986) elicited by a literary or cinematic fictional text depends – among other things, but, in my view and in the view of other scholars (Ferraro 2017; Gibson 2017), to a great extent2 – on its narrative structure and on the way in which this narrative structure articulates the particular types of signs that the text itself uses. To support the claim, I need to pursue another objective as well, namely, highlighting the characteristics of the sociosemiotic method on which the claim itself is based. Contrary to Barthes, who had not considered it important to demonstrate that his interpretation of the symbolic value of the faces of Greta Garbo and Audrey Hepburn was in some way generalizable among the viewers of the films in which the actresses appear, I believe it is essential to underline the functioning of cultural models that, right from the start, manifest themselves as codes of interpretation that do not belong only to me as a semiologist, but to 2 Enunciation logics are equally important, but will not be the focus of my argument. What interests me here is understanding why the discourses on reality in a work of admittedly fictional nature appear as truthful to their recipients as those in a factual newspaper article or in a philosophical essay. It is in relation to this issue that their narrative structure proves to be more important than their enunciation mechanisms. 420 Antonio Santangelo a whole “community” of people who share the same vision of things. My task, therefore, is to describe the mechanisms of signification of some works and some images of faces that take on a symbolic value in a precise cultural context. To do this, I will start from the interpretations of La vie d’Adèle: Chapitres 1 & 2 and Jeune femme that have been offered by some Italian film critics3, showing the regularities behind their apparent differences. My belief is that the former are the fruit of some codes of collective value – the cultural models I have written about so far – which serve as matrixes to give rise to some of the discourses on what it means to be young today that circulate in our society. Finally, to show how these cultural models work, and since the sociosemiotic method that I will talk about in this article derives from structuralism and is thus focused on finding similarities starting from differences, I too, like Barthes, will refer to faces appearing in other movies. More precisely, I will work on Shrek (Adamson, Jenson 2001) and Avatar (Cameron 2009). I will show that the faces of the main characters of the later films come from another cultural model of what it means to be a young man and woman. This will allow me to zoom in on the symbolic value of the faces of Adèle and Paula. If the two girls are unable to integrate themselves into, and find meaning in, our world, the other movies tell of individuals who do become integrated but always find themselves in an “elsewhere”. I will therefore advance a hypothesis similar to the research tradition that, in the type of sociosemiotics I am interested in addressing (Fabbri 1973; Prieto 1975; Landowski 1989; Ferraro 2001, 2015; Marrone 2001; Santangelo 2012), can be traced back to Claude Lévi-Strauss (1964); that is to say that culture itself, in some respects, has its own structure and the transformations in the way of talking about reality that derive from it can be explained, at least retrospectively. What seems evident and significant to me is that much has progressively changed compared to the times when Propp (1928) identified a common constructive model and characters in the tales of Russian folklore, who were always motivated to integrate themselves into the “here”, that is, into the system of values of their own society of origin, and at the end of their adventures, that often occurred in distant places, became “transfigured”, assuming the features of the most representative men and women among those who populated the world to which they themselves belonged. The cultural models that give rise to our stories, as well as the symbolic faces of our age, are decidedly different: a fact that may even appear worrying, in some ways. 3 I used the key phrases ‘Montparnasse femminile singolare reviews’ (Montparnasse. Femminile singolare is the title that was given to Jeune Femme for its distribution in Italy) and ‘La vita di Adele reviews’ as search terms on Google, and then proceeded from the contents of the first ten links that appeared. Culturally significant symbolic faces: For a sociosemiotics of faces in films 421 Stories of wanderers La vie d’Adèle: Chapitres 1 & 2 is a film by Abdellatif Kechiche, winner of the Palme d’Or at the Cannes Film Festival in 2013. In Italy, some critics have written about the film’s protagonist that, “exaggerating Flaubert, Adèle c’est nous”4, since this young woman represents the “underlying problem of the solitude of the contemporary individual, of the misery of the proposals for liberation and full and profound self-affirmation that this era makes available”5. As concerns Paula, the heroine of Jeune femme (recipient of the Un Certain Regard prize to the director and screenwriter Léonor Serraille at the Cannes Film Festival in 2017), other Italian film critics have argued that “she is really one of the most authentic female characters of recent years”6, “as she symbolizes the cry of the new generations who want to reclaim their place in the world”7 and “she is a matrix on which one could place a face extracted at random from the lot of contemporary European youth”8. As it is easy to see, the authors of these reviews seem to agree that the films are very realistic in describing contemporaneity and that their protagonists resemble all of us, or at least all the young people today. The reason for this is that both films are about two girls who are, in a sense, wanderers. Adèle and Paula, in fact, are represented while they are trying to become adults and find their own place in society and their own personal fulfillment, but it seems that they do not really know what they want and are simply trying to understand whether they like being in one way or in another. They experiment on their own skin with the various possibilities offered by the contexts in which they live, without ever finding their true being. The two young women continuously and without stopping move between heterosexual and homosexual relationships, humble and more prestigious jobs, upper-class and petit-bourgeois apartments, relatives, friends, colleagues and lovers from very different social backgrounds, in search of a sense of existence which seems to elude them. 4 Federico Pedroni, www.cineforum.it/recensione/Melodramma_materialista (last accessed 20 February 2017). 5 Goffredo Fofi, www.foglianuova.wordpress.com/2013/10/27/goffredo–fofi–la–vita–di–Adèle/ (last accessed 20 February 2017). 6 Linda Magnoni, www.cineforum.it/recensione/Montparnasse-Femminile-singolare (last accessed 25 August 2020). 7 Rolando Cisternino, www.anonimacinefili.it/2018/10/03/montparnasse-femminile-singolare/ (last accessed 25 August 2020). 8 Rudi Capra, www.ondacinema.it/film/recensione/montparnasse-femminile-singolare.html (last accessed 25 August 2020). 422 Antonio Santangelo In this regard, someone writes about the protagonist of Jeune Femme that “undecided about her means, her affections, her ambitions, Paula is the mirror of a generation that no longer identifies itself with the bourgeois ideals and aspirations of the second half of the twentieth century [...], yet is too weak to propose an alternative model”9. In her, however, we also find “a stubborn claim [...] of the individual and personalizing factor, in a world that has reduced almost to zero the possibilities of the unexpected, entrenching itself in a system of perfect gears. Paula’s inadequacy is in fact the bearer of an adequacy of another type. That of openness to others, the great missing element in these gloomy, mechanical times”10. Her story, after all, begins in diametrically opposite circumstances when her fiancé, a highly placed artist who is much older and more cultured than her, closes the door in her face and throws her out of the house, without us knowing the reasons why (later on we will understand that the two were too different and that he, while portraying her in his works, could do nothing but observe and desire her as if she were an object). The girl violently hits her head against the door trying regain access, and wounds herself in the process, but to no avail. So begins Paula’s long peregrination through Paris in search of her own existential niche, among the sofas in rich strangers’ houses, in filthy hotel rooms, old attics, the humble beds of colleagues in love with her, her mother’s house. In all these places, and even more in the various work environments she enters, Paula meets various people, who live according to the “mechanisms”, the social rules of which the film reviews speak, and who propose that she become part of their world, adapting herself to its logic. The heroine of Jeune Femme lives through and experiences everything, following a narrative trajectory reminiscent of the flâneries of the French Nouvelle Vague cinema, but there is a major difference, immediately perceived by contemporary critics, who stress that, “compared to the films of the young Turks, Séraille’s is the result of a script that is more written and less free, as well as of a vision of the world in which the bewilderment of the protagonist and her feeling lost in the non-sense of everyday life is not a declaration of anarchy with respect to the rules but a way to become part of them”11. The problem is that, in spite of her good will, Paula is absolutely unable to integrate, so much so that her “adventures” come to an end, if one can say so, with her abandoning her home and the work she has labouriously found. We first see 9 Rudi Capra, http://www.ondacinema.it/film/recensione/montparnasse-femminile-singolare. html (last accessed 25 August 2020). 10 Massimiliano Schiavoni, https://quinlan.it/2018/05/27/montparnasse-femminile-singolare/ (last accessed 25 August 2020). 11 Carlo Cerofolini, http://icinemaniaci.blogspot.com/2018/04/Montparnasse-femminilesingolare.html (last accessed 25 August 2020). Culturally significant symbolic faces: For a sociosemiotics of faces in films 423 her walking alone along an anonymous street, after the equally solitary decision to have an abortion, thus giving up the child conceived with a colleague with whom she evidently does not wish to share her existence. Finally, we look at her as she closes the window of the attic she is leaving, surrounded by the walls that have kept her “trapped” in the asphyxiating mechanisms of a society in which, despite trying with all her might, she has not been able to “find a home”; in front of her, however, she no longer has a door barred by others against which she could slam her head, but a transparent pane of glass which she closes herself, through which she can look out and on which she playfully blows, making it steam up. A similar open ending links Jeune Femme to La vie d’Adèle: Chapitres 1 & 2. Already the title of the film makes it clear that it tells of an unfinished part of a life in progress. Adèle, too, is abruptly sent away from home by her partner, having been caught cheating on her with a man. It is evident that the two girls, who meet when very young and fall madly in love, subsequently grow apart as they get older, due also to the differences in their respective social backgrounds and education. In the “first chapter” of her life a teenage Adèle, daughter of petit-bourgeois parents from the suburbs of Lille, thinks she can break all the rules, giving free rein to her passion for the slightly older Emma who comes from a more affluent family and is enrolled at the Academy of Fine Arts. Leaving Thomas, one of her most attractive schoolmates much desired by her friends, Adèle devotes herself completely to her partner, heedless of other people’s opinions. She goes to live with her, supporting her and letting herself be portrayed in some paintings full of sensuality, but in spite of everything she eventually disappoints her partner with her decision to become a kindergarten teacher and discontinue her studies. In the end, Emma begins to neglect her, as they do not share the same existential horizons, and Adèle is unfaithful to her with a colleague whom she feels to be more similar. Subsequently, however, Adèle regrets the breakup and in the “second chapter” of her life she tries to be reconciled with Emma in every possible way, until she goes to the vernissage of Emma’s first solo exhibition. There all the paintings that represent her are displayed on the walls, painful memories of a time gone forever. Emma is now with another woman who is also part of the art scene. Adèle understands that there is no way back and ends up going away on her own. Like Paula at the end of Jeune Femme, we see her walking along an anonymous street in her city, not knowing where to turn. Even in the case of this film, the Italian film critics stress the protagonist’s great openness, her desire to experience life without prejudice almost as if the story told by Kechiche were superimposable on that told by Sérraille. At the same time, they highlight the castrating nature of the frequently unjust social mechanisms that 424 Antonio Santangelo cause suffering to today’s young people: those who, like Adèle, seek their own way freely, outside the socio-cultural context from which they originate, find themselves “wandering” among very different parts of the world, in which it is not possible to settle definitively. We are encouraged to empathize with Adéle’s life, invited to probe her dreams, desires and uncertainties, because “existence comes before essence; that is, experience is not made comprehensible by a previous definition of the self, but on the contrary, subjectivity is determined by the multiplicity of experiences themselves”12. And yet, despite this attempt not to be defined a priori by the rules of society, for several critics Adèle ends up experimenting with the “misery of the proposals for liberation and full and profound self-assertion that this era makes available”13, as is always the case in Kechiche’s cinema, all centred on “social classes as regulators of an order that is difficult to negotiate; [...] and on the tranches de vie [...] never understood as mere sections of everyday life, but rather as emblematic segments of learning paths that do not produce evolutions”14. The faces of today’s young people In case of both movies critics recognize the crucial importance of the faces of the protagonists and the way they are filmed. In Jeune Femme, for example, Paula’s existential instability and continuous wandering between one niche and another in society, without ever managing to integrate, are symbolically represented by recurrent close-ups of the girl, made with the handheld camera technique, in which her figure is clearly separated from the background, often highlighting her two-tone eyes. In fact, the commentators emphasize how Paula “is almost always shot alone”15, precisely to underline her difficulty in becoming integrated with her surroundings, as can be seen in Fig. 1. It has been pointed out that Sérraille directs her film, “centring the camera on the protagonist and accentuating her discomfort and perplexity through serial jump cuts, sequences with a handheld camera”16, which reach particular expressivity when they direct the spectator’s 12 Carlo Cerofolini, http://icinemaniaci.blogspot.com/2018/04/Montparnasse-femminilesingolare.html (last accessed 25 August 2020). 13 Goffredo Fofi, www.foglianuova.wordpress.com/2013/10/27/goffredo–fofi–la–vita–di–Adèle/ (last accessed 20 February 2017). 14 Michele Marangi, www.lindiceonline.com/l–indice/sommario/dicembre–2013/ (last accessed 20 February 2017). 15 Linda Magnoni, www.cineforum.it/recensione/Montparnasse-Femminile-singolare (last accessed 25 February 2020). 16 Rudi Capra, www.ondacinema.it/film/recensione/montparnasse-femminile-singolare.html (last accessed 25 February 2020). Culturally significant symbolic faces: For a sociosemiotics of faces in films 425 attention on Paula’s gaze, characterized by the detail of her “‘bipolar eyes’, mirror of an equally bipolar and restless soul”17 (also seen in Fig. 1). It is this characteristic of the face of the protagonist of Jeune Femme – which is obviously constructed to communicate something, given that the eyes of the actress who plays the part are actually brown – that symbolically represents her “wandering” spirit, her perpetual movement among very different places and people. Since the girl’s story ends with her choice not to settle down in any of the social contexts in which we have seen her operating, but to continue to seek her own place in the world, critics cannot help but observe that the film ends with a close-up of her face (the third image in Fig. 1), which has been linked to the overall narrative structure of her story: “[…] it will be the viewers who judge whether Paula’s future will be seen with brown or blue eyes”18. It is indeed this state of uncertainty, this continuous wandering – which also causes her injuries, as when she cuts her forehead by slamming it against the door that is closed in her face at the beginning of her story, or when she undergoes an abortion at the end – that makes the protagonist of Jeune Femme such a realistic and contemporary character, as underlined by those who say that her existential path is definitely that of one who “finally sees and perceives the world with the eyes and body of a woman, a real woman, who is not afraid to be what she feels herself to be”19. As pointed out at the beginning of the article, this makes several commentators say that Paula and her face could represent a “matrix” of the faces of many young people today.    Figure 1.The face of Paula in the movie poster and in some scenes 17 Linda Magnoni, www.cineforum.it/recensione/Montparnasse-Femminile-singolare (last accessed 25 February 2020). 18 Rolando Cisternino, www.anonimacinefili.it/2018/10/03/montparnasse-femminile-singolare (last accessed 25 February 2020). 19 Rolando Cisternino, www.anonimacinefili.it/2018/10/03/montparnasse-femminile-singolare (last accessed 25 February 2020). 426 Antonio Santangelo Similar reflections are formulated by the reviewers of La vie d’Adèle: Chapitres 1 & 2, which in turn underline the importance of the emphasis placed by the director on the many close-ups of the protagonist of his film, which keep her character separate from the context in which she lives (as can be seen in some of the images in Fig. 2) , despite the fact that this very context plays a decisive role in directing the results of the girl’s story. This opinion is expressed, for example, by those commentators who consider exalting Kechiche’s expertise “in depicting the details of the faces and bodies, in dwelling on the close-ups, and then enlarging the shots to other subjects, where strictly necessary, only when the others become an attraction for Adèle’s gaze”20, for “there is nothing beyond what the young girl knows and experiences”21: “more light, more details enter the field of vision, but things in the background also become blurred”22.    Figure 2. The face of Adèle on the movie poster and in some scenes. In this very personal exploration of the world and the search for something that gives a definitive meaning to her existence, the images of Adèle’s body and face become again fundamental: “since everything is a body, special attention is given to how much the epidermis is animated by the desire for and contact with others”23; “the camera, rarely placed far away from the characters’ faces, is a seismograph with an immediate and icastic sensitivity; every change of mood is detected in real time”, 20 Giuseppe Gangi, www.ondacinema.it/film/recensione/vita_Adèle.html (last accessed 20 February 2017) 21 Ilaria De Pascalis, www.alfabeta2.it/2013/10/31/il–tempo–intenso–del–desiderio/ (last accessed 20 February 2017). 22 Veronica Vituzzi, www.doppiozero.com/materiali/odeon/kechiche–la–vita–di–Adèle (last accessed 20 February 2017). 23 Veronica Vituzzi, www.doppiozero.com/materiali/odeon/kechiche–la–vita–di–Adèle (last accessed 20 February 2017). Culturally significant symbolic faces: For a sociosemiotics of faces in films 427 the filmmaker catching it at the exact moment it manifests itself. The protagonist of the film “is constantly filmed with close-ups or details that exalt her overflowing physicality, her humoural fluids from tears to mucus, the sense of a warm and deep instinctuality”24. In particular, her wandering among the different recesses in society and the people who inhabit them is symbolized by her mouth, as can be seen again in Fig. 2: “Adèle is voracious, as suggested by Kechiche’s constant return to her lips as she sleeps, eats: lips stretched out, greedy, almost distinct from the rest of her body, as if they lived their own lives and dragged the protagonist forward against her will. Things make sense only through sense”25. After the two-tone eyes and the swollen epidermis on Paula’s forehead in Jeune Femme, the sensory organs of the face of the main character in La vie d’Adèle: Chapitres 1 & 2 once again become the most representative element of the existential searching of today’s young people, carried out almost gropingly, empirically, by trying everything out on themselves. Sociosemiotics and the meaning of certain faces The reader who has had the patience to follow me so far might have wondered why I decided to work on two films that were similar in many ways, and interpreted more or less in the same way by reviewers. The reason is that, in doing so, I wanted to show how the cultural models that are at the centre of interest in this article work, paving the way for a reflection on the kind of sociosemiotics I wish to discuss. The cultural models underlying the audio-visual products circulating in the media can be defined as the matrixes of “recurring discursive configurations, governed by precise narrative structures, which hold together entire encyclopaedias of signs” (Santangelo 2012: 120). In practice, we become aware of their existence when, following a working method similar to that of Propp on the morphology of the folk tale (Propp 1928), we understand that there is a common constructive logic behind apparently different contents that is made up of the recurrence of certain signs and narrative structures, as is shown in Fig. 3. 24 Michele Marangi, www.lindiceonline.com/l–indice/sommario/dicembre–2013/ (last accessed 20 February 2017). 25 Veronica Vituzzi, www.doppiozero.com/materiali/odeon/kechiche–la–vita–di–Adèle (last accessed 20 February 2017). 428 Antonio Santangelo &XOWXUDOPRGHO 0RYLH 0RYLH 0RYLH1 Figure 3. The matrix of different films. This immediately makes us reflect on the relationship of such contents with the socio-cultural context in which they are produced, as Propp himself thought in his work on the historical roots of the unitary compositional scheme of Russian fairy tales (Propp 1946). Thus, this article is based on the idea that by identifying the similarities between two films, such as La vie d’Adèle: Chapitres 1 & 2 and Jeune femme, which do not refer to each other, yet seem clearly connected; it can be assumed that their close relationship is not determined by chance, but by the fact that both derive from a certain model of common thinking about who are today’s young people, how they try to find meaning in contemporary society, and how their stories should be told. This hypothesis can be empirically substantiated with further evidence. Reviews of the films can be used, which are, after all, interpretations of them provided by individuals who belong to the same cultural context. In fact, the other idea on which the method of this article is based is that the cultural models that give rise to the audio-visual products circulating in the media are a sort of communication codes that allow those who make them share their thoughts with the audience who use the same logic in order to recognize the meaning of what they are told. The cultural models, therefore, are also a certain kind of pertinence principles (Prieto 1975), which guide the recipients of cinematographic works within them, allowing them to select and emphasize those content items that in their view determine the meaning of films more than others. What the reviewers find significant in the films, if repeatedly recognized as relevant by many of them, can be employed as a useful guide to identify those recurring discursive configurations that are the most obvious manifestation of the cultural models they find in the films themselves. Otherwise, as is shown in Fig. 4, reviews can serve as empirical confirmation of the interpretations of semiologists in their analysis of film works, indicating that the cultural models they identify are the same as those used by a certain category of their recipients to assign meaning to them. Culturally significant symbolic faces: For a sociosemiotics of faces in films 429      &XOWXUDOPRGHO FRGH  &XOWXUDOPRGHO FRGH 0RYLH $XWKRU 0RYLH  ,QWHUSUHWHU 0RYLH1  Figure 4. Cultural models as tools for encoding and decoding films. The roots of this way of conceiving sociosemiotics date back to Saussure’s thinking. In fact, a semiotics understood as a social science of signification should try to reconstruct the principles of the functioning of langue (Saussure 2001[1922]: 23) – or, to put it more simply, of the code – which gives rise to the communicative exchanges that take place between people in a given context. Only in this way is it possible to pursue the objective of describing the meaning the semiologists’ objects of investigation assume not so much in their own eyes, but in those of the other interpreters, in the society in which these objects circulate and are used. The langue is the instrument through which different subjects who wish to express the same concept, even in the idiosyncrasy of their personal acts of parole (Saussure 2001[1922]: 23), craft messages that are similar in some ways. Yet at the same time the langue is the medium used by their recipients, when the latter have to recognize that what is communicated to them is not simply the result of a purely individual choice or behaviour, but the attempt to reproduce a model with a collective value. This appears clear already in Saussure’s (2001[1922]: 145) famous example about the role of language in allowing each of us to write a letter ‘T’ on a sheet of paper in our own handwriting in the reasonable hope of being understood or, to the contrary, about how language itself allows us to understand that the infinite ways in which we can find the same letter written, depending on who penned it, are to be traced back to the same meaning. This is also what happens with the cultural models I am discussing in this article: the directors of La vie d’Adèle: Chapitres 1 & 2 and Jeune femme use them, metaphorically, to create their own personal versions of the letter ‘T’ of the code for reading the existential experience of today’s young people, and Italian film critics use them to decode the messages addressed to them. Of course, conceiving of culture as a language or as a unitary code is problematic (Volli 2010) and this is not my intention, but empirical work of the type I am presenting here shows that within culture itself there are indeed common 430 Antonio Santangelo matrixes of the discourses that people perform about specific issues. To describe these matrixes it is necessary to focus on their narrative structures and on the signs that are articulated within them. In both cases, as Ferraro (2001) shows, it is important to return to the method used by Lévi-Strauss in his analyses of myths all over the world, once again reconnecting to Saussure. That is, as far as signs are concerned, it is necessary to identify in the material discourses to which cultural models give rise, a series of recurring elements which, although apparently dissimilar, are united by the fact that they are made up of a number of common significant characteristics, which produce the same meanings (Saussure 2001[1922]: 84–85)26. In the same way, as far as narrative structures are concerned, some regular strategies of articulation of these signs must be highlighted which, once identified under the surface of contents that seem to be different, can be recognized as significant structures that convey the same meanings. In the light of the previous paragraphs, in La vie d’Adèle: Chapitres 1 & 2 and in Jeune femme, the most relevant signs among those used in these works to construct and interpret the discourses on young people today are those of separation, movement, reliance on the senses, pleasure and, above all, suffering. The two films, in fact, are full of images of the two protagonists framed in such a way that their figures are visually separated from the background of the contexts in which they live, just as there are numerous shots such as those of Adèle on the sidelines of the worldly occasions when Emma socializes with her friends in chic art circles, or others in which Paula is seen in her small attic in the same building in which her landlady and employer occupies the main floors. However, as we know, the two girls move between social environments in search of a place where to find a meaning to their existence and the cameras follow them with handheld mobile shots so as to communicate the idea of dynamism and, at the same time, the precariousness that all this moving, all this research entails. By keeping close to the protagonists, looking at their faces and bodies for signs that, in real time, show what they feel in their attempts to give meaning to their lives, the directors of La vie d’Adèle: Chapitres 1 & 2 and Jeune femme end up focusing on their sensory organs and on how their skin, eyes, noses and mouths react to everything in this sort of encounter/clash with reality. In some brief moments of their stories, Adèle and Paula feel pleasure – the Sapphic sex sequences in Kechiche’s film are particularly famous in this regard, as are those showing the protagonist enjoying pasta with sauce and oysters, the foodstuffs that symbolize the two different social contexts from which she and Emma come –, but most of the time the two girls suffer, cry, hurt, because not finding their place in the world is painful. 26 Of course, Saussure talks about ‘signifiers’ and ‘signifieds’ of signs. Culturally significant symbolic faces: For a sociosemiotics of faces in films 431 All these signs which are depicted in the faces of the main characters of the films I am analysing here, and recognized by film critics as fundamental for the reading of the faces of today’s young people, symbolically recall the common narrative structure of La vie d’Adèle: Chapitres 1 & 2 and Jeune femme, which they literally resemble, embodying its different facets, as can be seen in Fig. 5. 6HQGLQJLQVWDQFH    6HQGLQJLQVWDQFH  6HQGLQJLQVWDQFH 1   3HUVSHFWLYHLQVWDQFH Figure 5. The narrative structure of the two films. In order to understand the scheme in Fig. 5, it is necessary to point out that, according to Ferraro (2015: 71–100), the architecture of stories which, like those of the two films in question, mainly talk about the identity of their characters, is based on the relationship between a perspective instance and a sending instance. The former has to do with the sphere of the individual, his or her values, his or her perspective on things, and is embodied by the protagonist of the events narrated. The latter, on the other hand, concerns the world outside the protagonists themselves and can be represented by the laws or conventions of the society in which they live, or by another character’s way of seeing things, a place, a machine or anything that, focusing on its own values, could or would like to impose a certain destiny on them. The meaning of the narratives that are based on these two instances depends on the relationship that is established between them, a relationship that, after all, has to do with the values these kinds of stories become bearers of as noted by Greimas (1996[1970]: 167–194). The identity of the elements that make up the two instances, whether they are characters, things, environments or eras in which everything is set, depends precisely on these values and on how they relate to one another, whether they resemble or differentiate themselves, become integrated or remain distinct. As can be seen in Fig. 5, in the case of La vie d’Adèle: Chapitres 1 & 2 and Jeune femme, the perspective instance represented by Adèle and Paula, with their values based on extreme freedom, on the desire to be exactly as they want to be – high 432 Antonio Santangelo or petit-bourgeois, heterosexual or homosexual, lovers of established artists or of humble workers – clashes with the various “classist” sending instances in society, which would like to define them in one way or another. The reality in which the two girls live is, in fact, made up of strictly separate niches, each hinging on values that, in the end, prove to be different from those of the young girls themselves. Therefore, although they move continuously between the various social contexts in which they try to insert themselves (indicated by the arrows in Fig. 5), also feeling the advantages, the “pleasures” of what it means to be welcomed there for a moment, they end up painfully remaining cut off from these, as if there were a sort of barrier that prevents them from being fully integrated (this is the meaning of the horizontal line in Fig. 5). The faces of Adèle and Paula, like those of the young people they seem to resemble, are the faces of persons who, believing they can affirm their demand for absolute freedom, realize that this, unfortunately, will not allow them to find a place in the world, because the world works in a different way, perpetuating “instances” that are decidedly less open to experimentation and social mobility than the girls and the perspective instances they represent. What I have just said allows me to address the last question of method that I consider relevant, to reflect on how to determine, in the sociosemiotic paradigm to which I refer, the symbolic meaning of a face. The cultural model from which a face derives takes on its own value by difference, compared to other cultural models whose constituent signs and narrative structure are clearly opposed to its own, according to a logic that leads us to hypothesize that all these matrixes of the various stories circulating in our society are connected in some way: as Saussure states, after all, the numerous different ways of writing the letter ‘T’ are linked by the fact that they are different from the way an ‘I’ or an ‘L’ is drawn. In this sense, as I have already pointed out, it is evident that La vie d’Adèle: Chapitres 1 & 2 and Jeune femme are characterized by an ending devoid of the typical positive conclusion, which is easily found in those stories in which the perspective and sending instances somehow meet. As can be seen in Fig. 6, this is the case, for example, in the structure of the Russian fairy tales of magic analysed by Propp. Ferraro (2015: 79–85), on the basis of the reflections of Propp, who had highlighted their similarity with the initiation rites of different peoples (Propp 1946), defines them as forms of narrative that speak of the figure of the typical “self-made man”, an individual whose values coincide with those of the society in which he lives and who, at the end of his “adventures”, proves that he deserves to settle there. In fact, after having faced many tests in an “elsewhere” where different rules and values apply, the protagonist of this kind of storytelling shows that he has been able to fight to realize the rules of the world to which he has always belonged, transfiguring himself and assuming the likeness – and of course also the face – of Culturally significant symbolic faces: For a sociosemiotics of faces in films 433 the best of the representatives of this social context, so much so that he often gets to marry “the king’s daughter”.    6HQGLQJLQVWDQFH 3HUVSHFWLYHLQVWDQFH  Figure 6. The narrative structure of Russian fairy tales. However, Russian fairy tales evidently derive from other cultural models of what it means to be young and to become adults. Today we tell ourselves stories like those of Shrek or Avatar, whose narrative structure appears in Fig. 7. The respective protagonists are different from Propp’s heroes because, after having elsewhere faced the trials that would allow them to integrate themselves into their respective societies of origin, they realize that these involve an unjust sending instance on which they do not intend to make their identity depend. On the contrary, it is in the other “worlds” that they meet the sending instances whose values allow them to find themselves and for this reason they decide to settle there, transfiguring themselves, but symbolically taking on the appearance of ogres and aliens. Shrek tells the tale of a princess who is the victim of a curse. She can choose whether to love a prince animated by the worst consumerist values of her (and our) society of belonging, “chaining herself ” to its rules and thus becoming a beautiful woman admired by all, or to live forever free to be herself in a swamp with an ogre and other fairy tale characters, while becoming “ugly” and “different” like them in turn. In another case, an American soldier, thanks to a machine developed on planet Earth, “moves” into the body of an alien and into the extra-terrestrial world of planet Pandora to infiltrate its indigenous inhabitants and help his fellow human beings appropriate their precious natural resources. After realizing, however, that the Na’vi society is better than the capitalist society from which he comes, because it respects nature and promotes brotherhood among all living beings, he decides to betray his origins and defend the extra-terrestrials, ending up abandoning his human body forever. 434 Antonio Santangelo  6HQGLQJLQVWDQFH     3HUVSHFWLYHLQVWDQFH 6HQGLQJLQVWDQFH  Figure 7. The narrative structure of Shrek and Avatar. Evidently, Shrek and Avatar derive from a cultural model that overturns that of the fairy tales studied by Propp. If we compare the stories and faces of the characters of these films, and if we juxtapose them with those of Adèle in La vie d’Adèle: Chapitres 1 & 2 and Paula in Jeune femme, we realize that their differences still seem to connect them to each other. It could be hypothesized that there is a logic of diachronic transformation, which gradually leads from Propp’s unitary compositional scheme to the matrixes of contemporary works, passing from events set in just societies which manage to integrate their young people, offering them a sharable existential horizon, to tales of unjust societies from which young people want to escape, to settle in a better “elsewhere”. Today, however, we tell ourselves that this elsewhere no longer exists and that all that has been left to us is wandering around with no precise destination, in search of a meaning and of an identity that we are unlikely to find. This theory would certainly need further analysis, but if it were confirmed, it would reveal once again that we are living in difficult times. Acknowledgements: This research is part of a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No 819649 – FACETS. References Aristarco, Guido 1951. Storia delle teoriche del film. Torino: Einaudi. Barthes, Roland 1994[1957]. Miti d’oggi. Torino: Einaudi. Fabbri, Paolo 1973. Le comunicazioni di massa in Italia: sguardo semiotico e malocchio della sociologia. VS. Quaderni di studi semiotici 5: 57–109. Casetti, Francesco 1994. Teorie del cinema. 1945–1990. Milano: Bompiani. Ferraro, Guido 2001. Il linguaggio del mito. Roma: Meltemi. Ferraro, Guido 2015. Teorie della narrazione. Dai racconti tradizionali all’odierno storytelling. Roma: Carocci. Ferraro, Guido 2017. Senso, forma, nomi e referenza. In: Ferraro, Guido; Santangelo, Antonio (eds.), Narrazione e realtà. Il senso degli eventi. Roma: Aracne, 117–136. Culturally significant symbolic faces: For a sociosemiotics of faces in films 435 Gibson, John 2017. Sulla produzione del senso. In: Ferraro, Guido; Santangelo, Antonio (eds.), Narrazione e realtà. Il senso degli eventi. Roma: Aracne, 97–116. Greimas, Algirdas J. 1996[1970]. Del senso. Milano: Bompiani. Landowski, Eric 1989. La société réfléchie. Paris: Éditions du Seuil. Lévi-Strauss, Claude 1964. Anthropologie structurale. Paris: Librairie Plon. Lukács, Georg 1963. Ästhetik I. Die Eigenart des Ästhetischen. Berlin-Spandau: Hermann Lutherhand. Marrone, Gianfranco 2001. Corpi sociali. Processi comunicativi e semiotica del testo. Torino: Einaudi. Prieto, Luis 1975. Pertinence et pratique. Paris: Minuit. Propp, Vladimir J. 1928. Morfologiya skazki. [The Morphology of the Folktale.] St. Petersburg: Academia. Propp, Vladimir J. 1946. Istoricheskie korni volshebnoj skazki. [The Historical Roots of the Fairy Tale.] St. Petersburg: Izdatel’stvo leningradskogo gosudarstvennogo universiteta. Rastier, Francois 1986. Impression référentielle. In: Greimas, Algirdas; Courtés, Joseph (eds.), Sémiotique. Dictionnaire raisonné de la théorie du langage. Paris: Hachette. Santangelo, Antonio 2012. Sociosemiotica dell’audiovisivo. Roma: Aracne. Santangelo, Antonio 2017. Fuori dal testo c’è salvezza. Il senso del film La vita di Adèle. In: Ferraro, Guido; Santangelo, Antonio (eds.), I sensi del testo. Percorsi interpretativi tra la superficie e il profondo. Roma: Aracne, 41–68. Santangelo, Antonio 2021. Volti simbolici. Per una teoria sociosemiotica del volto. Lexia 37/38: 503–520. Saussure, Ferdinand de 2001[1922] Corso di linguistica generale. Roma: Laterza. Volli, Ugo 2010. Al di là delle culture. Le strategie della memoria. Lexia 5/6: 27–40. Filmography Adamson, Andrew; Jenson, Vicky 2001. Shrek. Glendale: Dreamworks Animation. Cameron, James 2009. Avatar. Los Angeles: Twentieth Century-Fox. Kechiche, Abdellatif 2013. La vie d’Adèle: Chapitres 1 & 2. Paris: Wild Bunch. Sérraille, Léonor 2017. Jeune femme. Paris: Blue Monday Productions. Культурно значимые символические лица: социосемиотика лиц в фильмах Время от времени при просмотре фильмов мы сталкиваемся с лицами, которым присваиваем особую ценность, так как они отражают важные культурные модели, придающие смысл нашему мироощущению. В качестве примеров я рассматриваю лица главных героев двух недавних фильмов, «Жизнь Адель» Абделлатифа Кечиче (2013) и «Молодая женщина» Леонора Серрейля (2017), и сравниваю их с лицами главных героев некоторых более старых фильмов, таких как «Шрек» Эндрю Адамсона и Вики Дженсон (2001) и «Аватар» Джеймса Кэмерона (2009). Я 436 Antonio Santangelo утверждаю, что способ изображения лица похож на повествовательную структуру истории соответствующего персонажа, и что знаки и повествовательные структуры, используемые для построения дискурсов о мире в этих фильмах, аналогичны структурам двух важных культурных моделей – что значит быть молодыми мужчинами и женщинами в наше время. Эти культурные модели различны, но взаимосвязаны, и я полагаю, что наиболее значимые лица в кино меняются из-за трансформации эти культурных моделей. Социосемиотический метод, основанный на структуралистском подходе к культуре, может помочь идентифицировать наиболее значимые в культурном отношении символические лица на экране и в других местах. Kultuuriliselt tähenduslikud sümboolsed näod: nägude sotsiosemiootika filmides Filme vaadates kohtame tihti nägusid, milles tunneme ära märgatava väärtuse, sest need esindavad olulisi kultuurilisi mudeleid, mida me kasutame oma maailmakogemusele tähenduse omistamisel. Näitena käsitlen peategelaste nägusid kahes hiljuti valminud filmis: Abdellatif Kechiche’i linateoses “La vie d’Adèle. Chapitres 1 & 2” (2013) and Léonor Sérraille’ filmis “Jeune femme” (2017), võrreldes neid peategelaste nägudega sellistes vanemates filmides nagu Andrew Adamsoni ja Vicky Jensoni “Shrek” (2001) ning James Cameroni “Avatar” (2009). Väidan, et see, kuidas nägusid kujutatakse, sarnaneb filmitegelaste lugude narratiivse struktuuriga, ja et märgid ning narratiivsed struktruurid, mida nendes filmides kasutatakse maailma puudutavate diskursuste konstrueerimisel, sarnanevad ühtaegu kahele olulise kultuurimudeliga, mis käsitlevad seda, kuidas olla tänapäeval noor naine või mees. Et need kultuurimudelid on erinevad, kuid omavahel seotud, väidan, et kõige tähendusrikkamad näod kinos muutuvad tänu nende kultuurimudelite muutumisele, millest nad pärinevad, ja strukturalistlikul kultuurivaatel põhinev sotsiosemiootiline meetod võib aidata kindlaks teha kultuuriliselt kõige olulisemaid sümbooseid nägusid nii ekraanil kui ka mujal. Art, face and breathscape: From air to49(3/4), cultural2021, texts437–462 437 Sign Systems Studies Art, face and breathscape: From air to cultural texts Silvia Barbotto1 Abstract. We consider breath as a vast prospect that includes actions and traces of them, that builds images and texts, that involves the human being and the extra-human context; we call this great scenery ‘breathscape’. We then study how breathscape interacts with the human apparatus of the face, both giving rise to signs, but also giving rise to a liminal zone of extremely intriguing interpretative processes on a mereological scale. How and where do the territory of breath and the body interact? Which processes of signification do they give rise to? And which signs are created in their phenomenal and semantic encounter? Art is certainly the most appropriate language for studying this process, as well as for letting opacities emerge and exploring outstanding contrasts. There are various concepts of ‘breathscape’ that, grounded in different cultures, are immediately associated with ancient and contemporary philosophies. As a reverberation from a semiotic interaction and through the discretization and identification of semantic fields relevant to the concerned scenario, and introducing textuality, a phenomenon seen as a crystallization of the transition between outer text (the text of reality) and text (subject/object), we consider those visual texts which are crossed by a common faculty that is both descriptive and inventive: by approaching some inferential and cultural regimes and analysing their specific enunciative practices, we then contribute to their renovation. The texts related to the practices as part of the narratives intrinsic to cultural semiospheres underline the insatiable vastness of epistemological content to be dealt with, and the functional reductionism of the corpus is only a first approach to the field that is intended to shed light on the general panorama and to stimulate subsequent debate and insights. Key words: breathscape; semiosphere; cultural semiotics; translation; face; art 1 Department of Philosophy and Education Science, University of Turin (Unito), Italy; e-mail: silvia.barbotto@unito.it. https://doi.org/10.12697/SSS.2021.49.3-4.10 438 Silvia Barbotto “…como un lindo recuerdo que se convierte en rostro y yo sepa por fin que dejas para siempre la espesura de ese aire maldito…”2 (Benedetti 1998: 84–85) ‘Breathscape’ is a vast prospect that includes actions and traces of breath, that builds images and texts, that involves the human being and the extra-human context; it is the landscape of breath, of the breathing and the breathable. As a semiosphere with a delimited character and semiotic irregularity certainly draws from its biological environment, it becomes an articulated multilevel process that involves cultural phenomena such as lifestyles, lived ambience, movements and reactions, both voluntary and involuntary. Just think of the contemporary pandemic period, which has shed light on the foundation of life, on the semiotization of the air we breathe as a transversal right but also as a language whose articulation is still partially to be discovered and recomposed as a cultural kaleidoscope. Although in breathscape analysis it is necessary to mention reference to the ontological and biological aspects, we will focus instead on the critical semiotic one: by evoking not only the living status of the breathing human body, but also breath as a transversal, inclusive and fertile enclosure, we will turn to breath as a textualized scenario of being and expressing. It is, in fact, the object of specific articulations in manifestation of art or multimedia works as displays of signs left by its course, its movement or built through interaction with other texts, as traces left on the face, the one belonging to the breathing human body. The facial production of meanings, in its reciprocity with otherness and the context, occurs on multiple levels; we identify at least two: the object one analysed through the selection of a dedicated corpus, and the meta- one, focused on the language utilized, on the generation of the argumentation. The face-sphere as a system is a whole: we need to identify meaning-making monads and articulate, mereologically, mechanisms of translation. It is precisely the face, in this case together with ‘breathscape’, that is at the heart of this article: we would like to explain, albeit not exhaustively, its role, its dynamic and processual signification, responding to the contemporary need of researching and understanding the face in different interconnected dimensions as, for example: “the predominance of the face in the neurophysiology of perception and cognition; 2 “...like a beautiful memory / that becomes a face / and I finally know / that you leave forever /the thickness of that cursed air…”. (Translations from languages other than Engish are mine, unless indicated otherwise, S. B.) Art, face and breathscape: From air to cultural texts 439 its impact on visual culture; its role in art as in portrait; the new aesthetic gender and formats of facial communication; the change of introspection, empathy and interaction in the face” (Leone 2021: 16). We are already starting to delineate the visage as a communicative interface with ‘breathscape’, a surface on which to express, transmit and perceive, but also as a live transitional platform, a seat of the ephemeral passage of components such as air which, besides having determinate chemical and physical characteristics, also becomes an interpretable instance of arbitrary nature, where the embodiment of automatization and de-automatization can be questioned. The face assumes the profile of a topographic countryside3 full of plateaus and mechanisms of intersemiotic transitions and translations, in which to understand and establish the borders between the different semiospheres, with references to physiognomy (Caroli 2012), anatomy, morpho- and neurophysiology4. Being rich in promontories and caverns, the visage is an expressive façade and visual screen of emotionality (Hwang Hyisung 2015); furthermore, density and emptiness of its constitution (the physical and the metaphorical one) establish the haptic mechanism of internal connectivity: the nose and specifically the nostrils, the mouth and the oral cavity are the main orifices through which we breathe, where breath transits, but all the pores of the skin breathe and a semiotics of action would be needed to crystallize its dynamics. There is an array of notions that different cultures have adopted to know and describe the entire universe as a breathing one – we need only think of the pneuma of the Stoics, the prana of the ancient Veda texts, or the broader oriental term Qi, emblem not only of breathing but of the wider panorama of energy; metalinguistic attempts to unfold meanings about this cosmos are part of the efforts that accompany this exploration, too. However, the very wide range of cultural languages containing linguistic and comparative considerations will not 3 At this point, we recall a passage from Deleuze and Guattari for whom the development of the mechanism of faciality contains marked references to the concept of landscape: “All faces envelop an unknown, unexplored landscape; all landscapes are populated by a loved or dreamed-of face, develop a face to come or already past. What face has not called upon the landscapes it amalgamated, sea and hill; what landscape has not evoked the face that would have completed it, providing an unexpected complement for its lines and traits […] The machine is already in place that always functions to produce faces and landscapes, however abstract” (Deleuze, Guattari 1987: 172–173). The authors bring to light various examples in which the intercut between face and landscape stands out in literature, architecture or cinema; although it leaves room for drift and uprooting, it nevertheless suggests, indeed, encourages, systems of re-territorialization, in a kind of ‘visageité’ not only of certain parts of the body, but also of objects of common use. 4 In this respect, we recommend watching the series “The science(s) of face” by FACETS ERC, organized by Viola Marco. 440 Silvia Barbotto be deeply examined; rather, the investigation will focus on artistic and aesthetic artefacts. At one point, we will be focusing on the paradigm of visual culture, art and pseudoscience: selecting a few artistic texts and also drawing on physiognomy, we will create a corpus crossed by a common property that is both descriptive and inventive. These visual text travel in semiosphere, propitiating new experiences for the visionaries, and they are also re-created back: “Creative vision modifies these materials. They take their place in an unprecedented object of a new experience” (Dewey 1934: 89). Based on the Lotmanian perspectives on culture and art, the article proposes relations with different interpretative semiotics currents and some methodological aspects regarding segmentation and selection which, once the isotopy of the breathscape as narrative background has been established, determines a programmatic body of work whose phenomenological origin is very heterogeneous: we find photographs of contemporary situations and performances, paintings, drawings, engravings. Even if this variety seems too discrepant, we could highlight that in all these manifestations of breathscape there is an evident reference to breath as a marked semiotic act, through the use of visual and verbal language; the reference to the face is also transversal, always present and mediated. While we are interested in emphasizing the discursive and discernible entity of the breathscape through the selection of texts, we also wish to emphasize the theoretical attributes of these texts and explore the consistency of their media in terms of materiality. “Semiosphere cannot come into contact with anything but texts, and texts are the products of semiosis. In this way, any contact with a space lying on the other side of a given semiosphere demands the preliminary semioticization of that space” (Lotman 2019[1989]: 91). Each text, a subjective object source, will be the bearer of a theoretical piece both on thematic path and on the medium itself: the creative processes underlying the text, as well as some of its significant aspects, will be partly highlighted, but, because it already is in itself, it will then acquire the totality formed in its continuity of being open to reading and interpretation. It follows that the elements subsequently shared are, even if extremely simplified and still in the embryonic state of a necessary forward-looking path, partly representative of an initial approach of semiotics to the syntagmatic and paradigmatic articulation of the composition of the breathscape. The identification of such discrete moments may recall the multiplicity of codification, the heterogeneous and multi-level property of textualities underlined in Lotman and Tamm (2019[1983]: 75): Art, face and breathscape: From air to cultural texts 441 […] every text is multiply coded (double coding represents the minimal structure). The conflict of thought formation arises not between individual textual formations but between languages manifested in text. Waves of syncretization of different arts […] illustrate the two opposing tendencies characterizing this process. The structural parallelism of textual and individual semiotic characteristics allows us to define a text at any level as a semiotic entity, and to view an entity at any sociocultural level as text. The following is divided into three parts: in the first part we approach the bodyface in relation with the environment, incorporating the individual and collective human actions in term of textuality. By entering another level of unity and complexity of meaning recognition, we then move to the language of art and description of dynamics texts. The third part will be dedicated to the relation between generalization and individualization process, by specifying cultural readings of air-signs on and of the visage. 1. Body-environment interaction: Con-text, ‘air as thing’ and consciousness In the Lotmanian perspective the relation between the text and the context is proactive and co-generative: the text is intrinsic to its context and its provisional extrapolation happens only in speculative terms. For this reason, we briefly elaborate on the concept and artefacts in their individuality, being conscious that they are submerged in a more complex and formative process. By introducing the suffix ‘-scape’, we propose specific semantic features to the carrier token ‘breath-’, assuming the immersive and spatial attributes denoted in its extensive suggestions about the environment. ‘Breathscape’ is therefore a landscape of breath, a diffuse semiosphere, but epistemologically established in the circumscription of semantic manifestations; it is a semiospheric context, an inextricable and changing binomial (breath and landscape) composed in a pertinent and complex unitarian polyphony, governed by signs and laws that are not solid structures in themselves, “but fluid elements, which do change through history. A science of fluxes is needed to understand them fully” (Leone 2014: 117). In fact, the proliferation of the ‘-scape’ as a field of aesthetic analysis has been taking place since the end of the 20th century and, as Battistini (2020) warns us, it mainly concerns the five senses; he concentrates on the horizon of hearing and the audible as the path of the sense effects of the soundscape. The compound word ‘breathscape’ provides that, thought of in more narrative terms, the object of its narrative programme is not a sense itself, but rather an 442 Silvia Barbotto a priori synesthetic field constituted in union and disjunction with the subject, intra- and extra-individual. It is this dimension made up of variable rhythms and densities, composed of more or less figurative fragments, expanded but at the same time necessarily intimate, that constitutes the environment to which we are referring and in which this specific textualization and semiotization of pieces of life takes place. Distinguishing ‘breathscape’ as a language composed of texts, whose compositional and immanent signs are concentrated in a partially legible ensemble, in a kind of a dynamic “thing”, as we recall from Genette (2010: 33) for whom ‘things’ are the stable (relatively immobile and durable) appearance that certain perturbations of atoms take on, which they nevertheless conduct under their continuous and untiring enduring aspectuality. In short, things are a particular kind of event. We therefore go to the basis of the cultural semiotic point of view that considers culture as a system of languages and its concrete manifestations as texts, asking ourselves questions such as: How is the breathing landscape generated and translated? How and where do the signs of the flow pause? Is the face also a junction of text and context? Is it a platform where breathscape takes form and its development becomes explicit? Why is breathscape relevant from a semiotic perspective? In this regard, Squarcini (2020) talks about the environment of breath in which we are immersed, a kind of “locus of daily setting” whose presence we praise and whose vital flow we invoke only when its normality is shaken up or the air is suddenly missing5. He also discusses the nature of this environment very thoroughly in his commentaries on the Patañjali Yogasūtra, II.50: “This is the general definition [...] of prān. āyāma: it is the confinement and analysis of the movement, the course, the ‘natural’ mode of exhalation and inhalation” (Squarcini 2019: 114, Comment 55). After having deepened the first three modalities (external, internal and suspended) of this practice, in Comment II.58 […] the fourth type of prān.āyāma is described, completely different and superior to the other three presented previously. [...] One thus passes from the act of confinement and vigilance over breath to the perception of the very form of the prān.a, seeing the very intention of the confinement of breath disappear. (Squarcini 2019: 115) 5 This general idea comes from the interpretation of the lecture (Togliere fiato al respiro. Il prānāyāma come non si è mai inteso) given by Federico Squarcini in the Chamber of the National Museum of the Risorgimento on 27 September 2020, during the 16th edition of Turin Spirituality Manifestation whose title was “Breath”. Art, face and breathscape: From air to cultural texts 443 From a semiotic point of view, some authors have already worked on the element air: for example, Jacques Fontanille (2008[2004]:160), relating it to the olfactory field, considers air the agent form of the actant of control: “air is in a sense the figurative vehicle of the powers of smell, weak but long-range if it is moving, strong and limited in range if it is stationary.” According to our interpretations, the breathscape, close to the definition of prān. a, thus assumes an expanded character, with blurred topological dichotomies: the inside and the outside interpenetrate in a transient, thin, extended space. Here, however, we will deal with textual coagulations, practices and events that seem to condense, distort and modulate this spatiality, devising its borders more imprecise. The first case reference deals with lack: having gone viral, the tag #ican’tbreath comes to underline the urgency of a subordinate position in a racist cultural situation. Seen in narrative terms, the iconized phrase sums up and becomes viral, moving as a signifying manifestation. The epistemological framework adopted in this text settles as the figurativization of negation which, socially situated and shared, does nothing but strongly affirm the opposite side of the same isotopy, life. The modalization of the actants, once manifested in the conveying tag, highlights the passage from the mode of negation of power (‘I can’t’) to the underlying modes of duty, of wanting, of knowing. The first-person singular shared in this way performs a hypostatized and inclusive act, indirectly implying the statement of a first-person plural. The images that appear on the web under this tag express the magnitude of a collective body that resists and fights by making its face manifest itself firmly. The drama of George Floyd’s death on 25 May 2020 is certainly a sensitive matter as well as a powerful communal issue, and although we do not deal with social ethnographic events like this, talking about air and human beings cannot but involve this very topical text and its widespread representations, and perceive them as paradigmatic. The formula #Ican’tbreath creates an immense transmedia community in which the platforms and social networks become a contextual, informative and foundational venue: for example, #icantbreath includes 1,250,628 publications on Instagram only as last consulted (on 19 February 2021). To convey attention to the triggering of semiosis, its evolution in and with culture, propitiating a process of self-consciousness, including self-as-individual but also self-as-community, we remind the concept of personal knowledge treated by Lotman as the connection between objectual things and individuality where also absence and exclusion of the signs are vehiculated as well as the presence and inclusion of them. The signic cosmos is animated by a collective consciousness, and in “Toward a theory of cultural interaction: The semiotic aspect”, Lotman (2019[1992]) argues that the elaboration and growth of the relation between 444 Silvia Barbotto individuality and the whole are dynamics synchronically animated by two kinds of consciousness, the logical one and the creative one, in any case a linguistic consciousness of the parts and of a whole, a semiospherical entirety: There are real human collectives in which every individual unit has the urge to transform itself into a self-sustaining and unrepeatable world of its own while at the same time entering into a hierarchy of structures at ever higher levels, forming at every level a communal socio-semiotic entity, which, in its turn, enters into a more complex unities as a part. The process of individualization and generalization, or the transformation of the individual into an ever more complex whole and into an eversmaller part of that whole occur in parallel. (Lotman 2019[1983]: 74) 2. Language of art and description of dynamic texts As part of everyday life, which Lotman calls ‘byt’, the arts as well as theatre (which is treated as an artistic practice in its own right) propitiate an intense exchange with the human being whose behaviour is continuously and intensely influenced by them. Remarking on the special features of artistic objects and bringing them closer to the prescription of things, we must, however, point out the dialogical and deeply generative nature of this choice, especially in the light of the fact that we are talking about a research object that is originally extremely changeable, the breath. As a secondary modelling system, art can also be described as a secondary language and, being a language, a stimulation for human consciousness, so art participates in modelling the consciousness apparatus. Language thus contributes to questioning the founding principles by propitiating, and in some way grammaticalizing, the creation and interpretation of the resulting textual expressions. Gillo Dorfles has studied the relation between perception, meaning and action and has attempted to identify the perceptive and cognitive data on which the present and future artistic experience is placed: his consideration are transversally close to this. In “The phenomenon of art”, Lotman (2009: 150) made evident that [T]he transformation, which occurs at the true moment of explosion, having been filtered by the lattice of the modelling of consciousness which converts the random into regular, does not as yet complete the process of consciousness. This mechanism must also include the act of memory, […]. There are now, as it were, three layers of consciousness: the moment of primary explosion, the moment it is realized by the mechanism of consciousness and the moment of its redoubling in the structure of memory. Art, face and breathscape: From air to cultural texts 445 Artistic texts, as part of a secondary language system, propitiate internal collisions that can explode into new significant structures and open up to the proliferation of further texts that on the one hand tend to integrate with the initial one, while on the other hand they are orchestrated in order to complete themselves as far as possible with the context again. This explosive subversion, the fulcrum of such languages, emerges from the innovative and propelling apparatus of visions other than the prevailing ones. The conversion of natural language into alternative ritualized, represented, semiospherized formulas gives rise to new recipes encoded through cultural arrangements: the texts will therefore be a syncretism of different idioms that have a fluid and complex structure, that require a rhizomatic decoding process and that, by making the various structuring items equivalent, highlight the isotopic ensemble. If the texts, especially if they are artistic, are therefore integrated with their context of origin, they are also hijacked in the forward-looking contexts of destiny. Through cultural arrangements, art makes an explosive entrance into the contemporary world, launching condensations of alternative and innovative visions. In this respect we think of the works Body of Air (1959–1960) and Artist’s Breath (1960) by the conceptual artist Piero Manzoni, who, being amongst the first to do so, identifies and circumscribes breath as an element of artistic reflection. Didi-Huberman (2016[2005]: 8) emphasizes the relational nature of a “thing” called breath: We only see it as a vital element – although it does not become an isolable ‘thing’ – when it is polluted with dust, swirling in smoke, violent in turmoil or missing in drowning. We never feel it better – as matter, as a medium, as a necessity – than when impurity reigns and breathing becomes short. In this recognition and attention to the outer text we approach breath as a thing, as text. From among the multitude of artistic texts related to breathscape, we have chosen a few as representative of a general panorama, including different modalities, varying especially in the implication of the body and its relation with materiality; for this first research they are not yet systematized and synthetized in specific typologies, but have been approached instead through description, also considered an essential scientific issue (Lotman 2006), a modus operandi of a semiotic method (Migliore 2005). The first example is Abramovich and Ulay’s performance: Breathing in Breathing out (see Fig. 1).6 6 The performance is accessible on the YouTube platform https://www.youtube.com/ watch?v=rWixdA2xTSs. 446 Silvia Barbotto Figure 1. Abramovich and Ulay. Breathing in Breathing out (1977). Performance (videorecorded live/freeze frame), Student Cultural Center, Belgrade. Courtesy of Marina Abramovic and Sean Kelly Gallery New York. The text air, modulated in the respiratory act denoted by the present participle of the verb in the title, ceases to be a natural pre-constituted substance and becomes a ritualized realm in which the whole action revolves, a constellation distributed on various paradigmatic levels. First of all, there is the binomial thing/artist: Abramovich’s intimate relationship with Ulay, the fusion of the two bodies and their faces through shared breathing is sufficient until, at the seventeenth minute, the air becomes almost exclusively carbon dioxide and causes partial fainting of both protagonists. In the experience and ritualization of the moment, the receptors and the cultural sphere also participate in the founding and transforming constitution of new texts palpable not only in situ, but also in memory. The texts are subjects with many entangled layers. The structural complication that spreads like wildfire from such a full-bodied artistic text nourishes the public, not only allowing new interpretations to take root, but also opening up ideological spaces that were previously airtight. When the artists are breathing into each other, the oxygen is reduced to zero and the increase of carbon dioxide leads to exhaustion: the culturalization of the moment, once the primary nature is gone, is not enough. Yet the action speaks volumes: the proprioceptive and metalinguistic character of the action, captivated and strengthened, becomes a voice for the generation of new meanings. At the basis of this gesture, in the union of two faces whose mouths become one, in the dreamy deliberation of each other as sufficient life force and in the taking to the extreme of the attempt to transcend physiologically from the primary metabolic functions, there is the effort to access the unknown, foregrounding and therefore poetically operating on the axiological setting of those present. The interpretative codes take on polyform dimensions and features, fleeting characters of sometimes static but constitutionally fluid illustrations, from mythical Art, face and breathscape: From air to cultural texts 447 reminiscences and immersive ceremonies to rarefied and penetrating perceptions. The face alternates between subtle blasts of resistance and contrast, explosion and staticity, into the creative depths of a semiotic circle. Above all again fluidity, evoked breath that brings back to the lifeblood, zero degree on which and from which to operate. The following piece (Fig. 2) is composed of photographs of the performance of running, an audiovisual installation with 26 sound speakers playing, each of them showing a part of the artist’s breath (recorded during the performances) in a series of web podcasts: Kanarinka ran the entire evacuation route system in Boston and attempted to measure the distance in human breath. The project also involves a podcast and a sculptural installation of the archive of tens of thousands of breaths. The project is an attempt to measure our post-9/11 collective fear in the individual breaths that it takes to traverse these new geographies of insecurity.7 Figure 2. Kanarinka. “Running performance” (2007). Performance filmed as part of a bigger work titled It Takes 154000 Breaths to Evacuate Boston. Ensemble of visual records, screenshot from www.kanarinka.com (last viewed on 20 February 2020). Like an archaeological artefact which is suspended in time and which has been recreated in space, according to Lotman an innovative text can precede the language and, searching for the codification between the real text (the outside text) and the text, we accomplish an act of reconstruction, proposing newest version of the same initial textuality. In case of Kanarinka it is not only the face which is the protagonist, but the entire body. “The body, all of it and at all times, even in spite of itself, signifies. By itself and in the whole ensemble of its relationships the body constitutes a sort of 7 See http://www.kanarinka.com/project/it-takes-154000-breaths-to-evacuate-boston. 448 Silvia Barbotto “corposphere”, which itself is an integrating part of the semiosphere.”8 The creative capacity of the performing body shapes texts that tacitly break into the ideological and molecular structure, triggering a trans-textual componential process in which a person and culture regenerate each other: Kanarinka’s work of art reasons on the dominant political paradigms and proposes a critical reading of these. Another artwork to be addressed here, by Lozano-Hemmer Rafael (see Fig. 3), is a hybrid between the categories of performance and installation: the body of the protagonist – their face and breath involved – is preponderant, but it is also, at the same time, great machinery in motion. It is a metaphor for perpetual movement. Figure 3. Lozano-Hemmer Rafael. Last Breath (2012). Sound installation: apparatus 60 x 27,5 x 23 cm, tube up to 15 m long. Edition: 6 copies + 3 AP copies. Photo by Grace Storey, Carroll/Fletcher Gallery. Courtesy of the artist. This motorized installation allows the captured breath to preserve itself and last over time in an infinite, complex and constant duration: it does not remain static inside an inert box, but, simulating inhalation and exhalation, produces an organic sound determined by the cardboard bag in which the last breath was initially concentrated. 8 Finol Enrique in Semiotics of the Body, 2020. Virtual Series Facets available on Youtube, channel facetsERC, min. 2:47. Art, face and breathscape: From air to cultural texts 449 On the artist’s official website9, this piece is described as “A ‘biometric’ portrait designed to store and circulate the breath of a person forever, between a robotic bellows and a brown paper bag”: the cold mechanical complex is accompanied by the sound act and a video that immortalizes the moment in which the chosen artist (in this case Omara Portuondo) leaves her “last breath”. The storage of the most ethereal substance, the subjectification of the machine and the timeless alteration of the becoming allow punctuality to become durability, letting the poetics of the work absorb within itself the isotopy of life and its opposite, denoted above all by the title. The landscape of air together with the intrusion and pervasion of space in the singular respiratory acts constitutes the syntax of the performance. This new text, the installation, is syncretic and multilingual, lives on the frontiers and spreads out in the interstices, every new text is somehow creative. The frontier, once established, immediately aspires to be resemanticized and reabsorbed, shedding light on the areas of contact, collision and opening, thus revealing not only spatial and symptomatic considerations, but also those of an underlying or evoked effect. This is the case of the trace in the conceptual art piece Breath of Leaves (Soffio di foglie, 1979) by Giuseppe Penone: an imprinted area in leaves evokes an anterior presence, a body that was previously there. A simulacrum of an invisible body, the index of a material form that was once there, together with the shape of the leaves on the ground create the breath: as is typical in Arte Povera, of which Penone is a pioneer, poor materials with the predominant use of natural elements assume a new signification on both the planes of expression and of content, which seems to highlight an evident relation with ancestral and oriental art representation. Breath of Leaves recalls the privileged moment in which we and nature breathe together, generating a subtle aura that can be perceived when viewing a work like this. There are also two later works by the same author (see Figs. 4 and 5) with which Breath of Leaves can be grouped together for systematic purposes10. Some years earlier, in 1973, the Cuban artist Ana Mendieta started to develop and produced a similar work that would then be grouped in the Silueta series. 9 “Last Breath is an installation designed to store and circulate the breath of a person forever. The piece consists of a small brown paper bag which inflates and deflates automatically thanks to motorized bellows similar to those found in artificial respirators in hospitals. The apparatus hangs on a wall and is activated 10,000 times a day, the typical respiratory frequency for an adult at rest, including 158 sighs. Each stroke of the machine advances a digital counter that beeps. The breath circulates between the bellows and the paper bag through a ribbed transparent plastic tube that emits a faint and hypnotic low sound. The first copy of the piece stores the breath of Cuban singer Omara Portuondo.” https://www.lozano-hemmer.com/last_breath.php (last accessed on 20 February 2020). 10 In the cycle of works Breaths (Soffi) and executed from 1978 onwards; the author retrieved from the world of mythology the image of breath as the origin of man and as a vital and energetic element, see: www.castellodirivoli.org/artista/giuseppe-penone/. 450 Silvia Barbotto Figure 4. Giuseppe Penone. Breathe the shadow (Respirare l’ombra). (1999, installation: laurel leaves and gilded bronze.) Photography by Paolo Pellion. Courtesy of Castello di Rivoli Museo d’Arte Contemporanea, Rivoli-Torino. Figure 5. Giuseppe Penone. Breath of Clay (Soffio di creta). (1978, sculpture.) Photography by Paolo Pellion. Courtesy of Castello di Rivoli Museo d’Arte Contemporanea, Rivoli-Torino. Art, face and breathscape: From air to cultural texts 451 A problematized philosophical category, the element of air moves from virtualized to actualized in the form of breath, which can be read in semi-symbolic terms. “Air [...] as the incessant exchange of ‘life in the past’ and ‘life to come’, of survival and the desire that is born from it. […] Here, then, is the ‘air in motion’ that makes the protrusions of desire and the emanations of the farthest past co-present” (Didi-Huberman 2016: 30–31). Belonging to a single series entitled Breaths (Soffi) Penone’s works highlight certain semantic traits whose interpretative keys of the figurative and the plastic would intersect. Breathe the Shadow is a multi-sensorial work that mainly combines the involvement of sight and smell in a reading path that can be addressed by syncretic semiotics, while Breath of Clay is a figurative sculpture. The isotopic significance of the work and the title can lead to the articulation of different perceived interpretations, but the figurative elements (the title, the lung, the leaf) together with the plastic ones suggest a system of relations connected with the ‘breathscape’. It may be useful at this point to comment on a recurring lexeme, ‘element’, about which Leone poses questions and suggests answers. Asking what an element is in art, whether a work of art can be composed of elements and then going on to more specific questions, he states: ‘Element’ is a common word in several meta-languages, although usually associated with some sort of mechanical functioning: ‘element’ makes one think of the components of a mechanism or of a chemical compound. Metaphorically, its semantics can also be extended to cover the functioning of parts of a non-mechanical whole […] an element of something is not something that expresses an autonomous intentionality and a self-centered agency but something that is subservient to the agency of a greater whole. (Leone 2018: 111) In the next piece of art observed, the image as expressive substance (the portrait genre, the technique of oil painting, etc.) is the inspiration on which the artist bases his work: Tim Eitel (Fig. 6) typically uses photographs taken by himself that are then translated into painting. The air, a propitious element for a metatheoretical reflection, and the face as a salient point for a semiotics of the body and of everyday life, of the space passed through, inhabited, emptied, come together in Tim Eitel’s pictorial representation to talk about himself and, in doing so, rework the cultural assets: “The processes of individualization and generalization, or the transformation of the individual into an ever more complex whole and into an ever-smaller part of that whole occur in parallel […], this is the same way every artistic text is formed” (Lotman 2019[1983]: 74). Artistic processes build a kind of self-contained whole renovated and integrated in reception for a more complex and comprehensive formativity. The pictorial aggregate assumes high semantic density due to its internal 452 Silvia Barbotto composition, but also within the more general characteristics of a properly pictorial structure; according to Maria Giulia Dondero (2014: 27), the enunciative level may vary according to the relationship between “enunciative levels according to the degrees of stabilization (support) and enclosure (edges) of the object of analysis”. Figure 6. Tim Eitel. Untitled (Breath). (2009) Oil on board, 9” x 16”. Collection of Contemporary Art (Larry Qualls Archive). © 2014 Artists Rights Society (ARS), New York / VG Bild-Kunst, Bonn. The receptacle face in Eitel’s painting, emitter and receiver, synthesizes in a planar congregation the result of a stratification of pigments on the canvas; as the receptacle creator, the face becomes a reciprocal relation, and its orifices, the seven passages, furnish the power to channel the transition outside ←→ inside, consenting the incorporated fuel to burn and distribute life and the unnecessary material to be expelled. The open mouth, which is the punctum in the level of expression of the painting, can evoke a moment of sleep, but also a possible fainting, close to representing death in the phenomenal macro-isotopy of human beings. Mediating between Lotman’s cultural semiotics and the propensity for Greimassian generative semiotics, Calabrese (2006) proposes deep narrative paths in which “meaning is generated through a general architecture that gives order to its internal components, one in relation to the other”, involving epistemic and passionate dimensions of sensory experience. He elaborates theoretically on the element water and objectifies it as a “figurative phenomenon on the level of the manifestation of a text” (Calabrese 2006: 31–33). As a special figure of water, different kinds of materials, drawing on encyclopedic and lexical definitions, manifestations and figurative effects have been synthesized in the dichotomy Art, face and breathscape: From air to cultural texts 453 representation/ostension of the following oppositional terms: transparency/opacity, projection/distortion, internal geometries (structure/lack of structure), movement/ immobility, punctuation/durativity. “Figurality is formed by categorial sets placed on the plane of expression of the semiotic system created by a text (especially visual), and able to combine with categorial sets belonging to the plane of content” (Calabrese 2006: 46). In connection with the opposites of movement/immobility examined by Calabrese we can also mention the speed/slowness dichotomy in another author, Italo Calvino. This writer finds in speed and movement the possible reasons why a story can fascinate us and considers them to be the basis of the narrative links between desired objects. Movement, be it rapid or slow, is marked by rhythm, and the succession of events in a discourse that is not always linear, their gait and their conjunction establish the moving and stopping, “two operations on the continuity and discontinuity of time” (Calvino 1988: 32). The theoretical contributions concerning the temporal dimension are therefore highlighted as a rhythmic multiverse in which aspectuality, assimilation more or less intense or thinned out in time, provides evidence of the biological apparatus of the air represented as well as breathed. That sphere of manifestation which would not belong to figurativeness, would instead be part of the semi-symbolic category due to the preponderance of the plastic signifier as happens in the following art works by Kenneth Noland and Richard Pousset-Dart (Figs. 7 and 8): Figure 7. Kenneth Noland. Breath (1959). Acrylic on canvas. 168 x 165.6 cm. ©2020 The Kenneth Noland Foundation / Licensed by VAGA at Artists Rights Society (ARS), NY. 454 Silvia Barbotto Figure 8. Richard Pousset-Dart. Breath of White Motion. (1990, works on paper: wash, pencil and acrylic.) 57.15 x 76.835 cm. © 2008 Estate of Richard Pousette-Dart/Artists Rights Society (ARS). Following Greimas, Collins and Perron (1989: 635), figurativity seems to go beyond the limits of the vehicle or support upon which the manifestation of a visual object is based: indeed, they emphasize that in addition to figures of expression, there are also figures of content. In case of Noland’s geometric and symbolic reference, the circle as a figure belonging to the world is evident, and although its reading is open and less explicit, if it is accompanied by the verbal discourse of the title, we can certainly open up interpretative and pertinent paths. In contrast, the signifier in Pousset-Dart’s graphic work is merely plastic: the graffiti lines on a monochromatic background organize an abstract scene which timidly suggests distant figures; the processuality highlighted by the strokes, between childlike grapheme and architectural projectuality, suggests dynamism and the emergence of possibilities, also interpretative ones. In Greimassian terms, this work can be read through the use of topological qualities which, between the straight line and the curved line, create a grid on the plane of expression that is further made discrete by the frame that has “a twofold function: the segmentation of the whole into discrete parts and the mapping out of the possible trajectories that the various aspects of the reading will follow” (Greimas, Collins, Perron 1989: 638). In both cases, the semantic contribution of the titles should be considered as a determining and integrated component from which an isotopic construction that envelops and involves text-image would result. Art, face and breathscape: From air to cultural texts 455 3. Cultural readings of air-signs on and of the visage From the semiotic mechanisms that are highlighted in artistic texts we can now turn to those that, arising from the organic interaction between face and breathing and established in specific cultural memories, dictate original languages whose decoding is emblematic, for the very reason that they contain within them multiple gestational and interpretative approaches. As Lotman has pointed out, the collective intelligence of culture and individual consciousness travel together to generate meaning. The impossibility of establishing univocal deductive relations of contact and interaction between the face text and the air text makes the relational practice ambiguous and difficult to codify: not only the cultural attributes, but also the independent operations of the intertexts and intratexts contribute to creating the specificity to be explored. Some cultures, however, have made their contributions through the elaboration of codified diagrams, dictionaries and even pseudosciences: physiognomy, for instance, has proved to be among the most equipped of these kinds of systematizations, but its affirmation has varied according to the geographical locations and historical moments in which it was developed. Patrizia Magli traces a foundational path regarding the system of physiognomy by identifying its various salient phases and crossing all the cultures that have employed it. Although it is still being practiced today, it is often referred to as a thing of the past: Physiognomic competence originated in history as a form of knowledge determined by concrete needs. It is part of that circumstantial paradigm in which a constellation of disciplines participated in antiquity, such as divination and medical semiotics in its double function, diagnostic and prognostic, disciplines that shared a similar attitude oriented towards the analysis of individual cases, reconstructable only through traces, symptoms, clues. (Magli 2004[1995]: 26) The author recounts how in Mesopotamian culture the face, formally composed by a meticulous reading of its formal micro-components, was an omen in divination, frequently rich in omens, in other words, accessible during ritualistic fields and subtle dedicated states of mind: some extracts are presented in Ludlul bēl nēmeqi, an old sapiential manuscript that contains cults and ceremonies. The ancient Greeks, however, proceeded by means of clues, called tekmaíresthai, which suggested relationships with the invisible: especially in the case of illness, the expert studied the dissimilarities that emerged (denoted by sweating, the sound of the voice, breathing, etc.) and they knew that a real Hippocratic facies would emerge when death was imminent. 456 Silvia Barbotto It is interesting to note how even in this period we begin to speak not only of the sequence of anamnesis–diagnosis based on the observation of the body and especially the face, but especially on how “the environment influences the physique and, consequently, the character” (Magli 2004[1995]: 40), as among the works resulting from such studies we find Le arie, le acque, i luoghi (end of the 5th century B.C.) contained in the Corpus Hippocraticum. Magli also delves into Arab physiognomy, whose “origin is lost in the mists of time, confused with that of magic, anatomy, physiology, philosophy but, above all, with that of medical art” (Magli 2004[1995]: 44); this is found in the term ‘firāsa’, a kind of “organ of circumstantial knowledge” with which only a few sages and mystics were endowed: it was an intuition capable of abductively interpreting not only the signs of the face, but also the clues hidden in the soul and the cosmos. There were various versions and updates of these practices, so much so that it continued until the beginning of the 13th century with the prestigious Kitab al-Firāsa of Fakhr al-Dīn al-Razi, a systematization of Arabic physiognomies. In the West, this science was part of the apparatus of sign theories and astrology until the middle of the 17th century, when there occurred a separation between an astrological and a natural physiognomy: The gaze passes through the surface of the body, the anatomy penetrates the inner man […] Thus Cureau de la Chambre, who in his Art of Understanding People distanced himself somewhat from the “chiromancer”, considered it still important for a doctor to take astrological signs into account. […] Planets and signs of the zodiac in astrological physiognomy, moods and temperaments in natural physiognomy, in no way exclusive of each other, remain the guarantee of what is at the basis of the science of the face: the analogy of soul and body. (Courtine, Haroche 2007[1988]: 71–72) With the Cartesian philosophy, also reported in Charles Le Brun’s studies, paintings and Expressions of the Passions of the Soul, the organism entered the foreground and the machine-man typology succeeds the zodiac-man one. The representation of the signs that had been considered pertinent to the physiognomy earlier, became concerned with geometry, anatomy and medicine. Later, among some 19thcentury authors including Cesare Lombroso, there was an epistemic degeneration, a manipulation of information resulting from a reading of appearances that undermined these and stimulated racism. It is clear that all these practices were extremely impregnated with the ideological thought in which they developed chronotopically, finding particular prestige or falling into decline, sometimes developing as conjecture and sometimes as science, but always implying a strong relationship between the face and natural Art, face and breathscape: From air to cultural texts 457 elements. It would be interesting to understand if and how a semiotics of the elements is developed in various cultures; however, this study is limited to the element of air formalized to the face.11 Every culture has developed a different cultural memory with respect to this theme and seems to occupy positions that are currently rather distant from each other as regards the reading of signs on the face through physiognomy: some cultures have entirely abandoned such practices, but in some others the behavioural and relational signs between the physical body and the astral body are alive and readable. According to contemporary Chinese medicine, for example, the vital energy called Qi has an abstract entity, but tangible repercussions: as, firstly, its signs can be read on the face; and, secondly, it can be dispersed but also invigorated through two main practices: If Jing is the inherited fundamental energy, then Qi is the manifestation of that energy necessary to participate in the world. This implies exertion and movement or yang action. Qi can easily become overused and deficient, but luckily it can also be manufactured. There are two classic ways that you can make Qi buffer the use of Jing. The first way is by transforming food and the second is by transforming breath. However, this is more than just eating and breathing. Both of these ways of making Qi can be seen on the face and these horizontal markings are on the central meridian of the face across the nose and across the philtrum. (Bridges 2012: 24) Marked lines above the lip area would therefore seem to indicate that “it is expected that some Jing will have been used up by these ages, but in order for Jing to be available for the remainder of life, a person with any of these lines should actively start making Qi with conscious intent as the Jing is not as strong as it used to be and needs support” (Bridges 2012: 24). The memory of culture is identified by Lotman as a very important function of a text that, however, constitutes only a reduced mnemonic programmes (Lotman 1996). Face as memorial text, physiognomy as the reading of characteristic and permanent sign whose trace has been sedimented on the face in the memory of time, are part of the mnemonic programme of culture, and the Chinese culture is still benefitting from studying and applying it. As we have already seen, memory is the necessary building block for the initially explosive and then conscious phenomenon of art to take root. 11 In this regard, we would like to go deeper into Magli’s (2004[1995]: 51–60) text, which, while also investigating other authors, reports on the semiotic Greimassian square of elements. 458 Silvia Barbotto From the point of view of semiotics, culture represents collective intelligence and collective memory, that is a supra-individual mechanism for preserving and transmitting messages (texts) and for creating new ones. In this sense, the field of culture can be defined as a space of shared memory, within which certain common texts are preserved and actualized. (Lotman 2019[1985]: 133) So, actually memory is dialectical and its settling is not definitive. In most of today’s Western contexts, the coding system of physiognomic pseudo-science does not provide an informational setting for understanding its language, but it did so in the past. The decoding of the signs on the face, if it were to happen, would certainly not have the same resonance in different cultural environments. Thus, awareness should be raised concerning this subject, getting closer to oriental cultures that have maintained the dynamics of identification, reading and manipulation of the element ‘air’ in relation to the face in their memory. The text later exposes some techniques and decodified readings, but what interests us is understanding how this occurs, especially in ancestral philosophical and medicinal practices, and in those ancient philosophies that have lasted to the present day affirming that the signs of the body can be read. In the interpretative possibility resulting from a reading of the face, the ideological and cosmological background of the cultures under examination is highlighted. The use of semantic and typological categories refers to specific connotations regarding the air, the respiratory act, and the breathscape, while their perimeter is not limited to the surrounding context, but greatly amplifies the range of action, thus expanding the multidimensionality of the semiosphere. Where are the frontiers of this? What measurements are appropriate for them? How can the matter of body and air be scientifically correlated with the creative processes described above? Do the body and the face have memory? In Chinese medicine a line on the face evokes the cosmos, explaining such a relationship on the basis of its features: an interpretative convention that approaches the “thing” by touching it, making it intelligible and partly favouring its direct contact. This line is: […] the highest point of the Chong Mai, which regulates the connection between Qi and blood and is the bridge between heaven and earth-breathing cosmic Qi into the body. […] At the simplest level, a marking here shows the need to breathe properly, to expand the lungs and to oxygenate the blood. On a higher level, it means to expand your consciousness. A line in this area indicates that you have used up Jing and that the best way that your body can make Qi to compensate for that loss is by breathing in Cosmic Qi. (Bridges 2012: 27) Art, face and breathscape: From air to cultural texts 459 South-Asian yogic cultures also propitiate a continuous dialogue with these extraindividual forces, and textualizations may be evident on the historicized face, but it is also interesting to study the performative face during specific uses of air as in kryia such as vātasāra, or during the practice of breathing called pranayama12. It would certainly be extremely fascinating to study the yogic face in terms of textuality, but to underline the importance of the practical aspect preponderant in these disciplines, we quote Squarcini’s words who, recalling Merleau-Ponty’s thought, insists on “practice that makes knowledge” rather than on “knowledge that orients practice”: it would therefore be up to the semiotics of the body and of practice, or perhaps applied and implicated semiotics, to pursue this question, which has so far only been sketched out here and is open to future study (Squarcini 2019: CVII–CVIII). Conclusion Recognizing, reading and evaluating the signs of the breathscape as well as savouring the processes of its signification is an act of a continuous intersemiotic translation. The textualization of the landscape of breath must pass through different operations and from an elementary, natural, forgotten action it can become a crucial aspect of new forms of life and new figures of art. During the transition from text to language, its interpretation and therefore signification assume partial correspondences and mainly bodily statements. Tangentially to thought, breathing itself can be considered mediated by cultural language, as we can see in Torop’s words on Lotman’s Universe of the Mind: “Lotman’s approach to the culture includes the notion that thinking is mediated by language, but dialogue occurs in cultural space” (Torop 2020: 267). Breath can be read as a synecdoche par excellence that leads from one element (air) to the whole (life). Many texts from all cultures make references through intertextuality, paratextuality, metatextuality, hypertextuality and architextuality to the passage of breathing and its immensity: to conclude, we may remember the song “Breathe” from Pink Floyd’s album Dark Side of the Moon (1973): “Breathe, breathe in the air/Don’t be afraid to care/Leave, don’t leave me/Walk around and choose your own ground.” The ether and the earth have always been the great dichotomous hemispheres whose reading on the surface brings out the semantic opposites of above and below, light and dark, light and heavy. Here, air and ground both assume the peculiarity 12 In this regard, we suggest viewing the images in specialized yogic texts or reading the extremely figurative and detailed explications in texts such as Saraswati, Swami Satyananda; Saraswati, Swami Niranjanananda 2013. 460 Silvia Barbotto of being carriers of freedom of action, the basis of that decisional act in which the breath, instead of being a natural act and simply perceived, is a conscious action and the constructive foundation of ground on which to move openly. “Souvent décrit comme la zone la plus fragile et la plus nue du corps humain, le visage passe pour exprimer la vérité profonde des êtres. Or c’est justement parce qu’il se trouve en permanence exposé, impliquant ainsi toute la personne, qu’à la fois il se livre e se refuse aux regards.”13 (Guedron 2015: 8) Acknowledgement: This reseach is part of a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No 819649 – FACETS). References Battistini, Emiliano 2020. Tra Soundscape Studies e semiotica: dall’effetto sonoro agli effetti di senso dell’ambiente sonoro. E|C (Serie speciale) 28: 1–17. Benedetti, Mario 1998[1996]. El amor, las mujeres y la vida. México D. F.: Alfaguara. Bridges, Lilian 2012. Face Reading in Chinese Medicine. Edinburgh, London, etc.: Churchill Livingstone, Elsevier. Calabrese, Omar 2006. La forma dell’acqua. Ovvero: come si “liquida” la rappresentazione nell’arte contemporanea. Carte Semiotiche 9/10: 57–66. Calvino, Italo 1988. Lezioni americane. Sei proposte per il prossimo millennio. Milano: Garzanti. Caroli, Flavio 2012[1995]. Storia della fisiognomica. Arte e psicologia da Leonardo a Freud. Milano: Mondadori Eclecta. Courtine, Jean-Jacques; Haroche Claudine 2007[1988]. Histoire du visage. Exprimer et taire ses émotions (XVIª-début XIXª siècle). Paris: Payot & Rivages. Deleuze, Gilles; Guattari, Félix 1987[1980]. A Thousand Plateaus: Capitalism and Schizophrenia. (Massumi, Brian, trans.) Minneapolis, London: University of Minnesota Press. Dewey, John 1934. Art as Experience. New York: Perigee. Didi-Huberman, Georges 2016[2005]. Gestes d’air et de Pierre. Corps, parole, souffle, image. Paris: Les éditions des minuit. Dondero, Maria Giulia 2014. Sémiotique de l’action: textualisation et notation. Cadernos de Semiótica aplicada 12(1):15–47. https://doi.org/10.21709/casa.v12i1.7117 Dorfles, Gillo 2004[1959]. El devenir de las artes. México: Fondo de cultura económica. Genette, Gérard 2010. L’œuvre de l’art. Immanence et transcendance. Paris: Seuil. 13 “Often described as the most fragile and naked area of the human body, the face is considered to express the deepest truth of beings. Yet it is precisely because it is permanently exposed, thus involving the whole person, that it both reveals itself and refuses to be seen.” Art, face and breathscape: From air to cultural texts 461 Greimas, Algirdas Julien; Collins, Frank; Perron, Paul. 1989. Figurative semiotics and the semiotics of the plastic arts. New Literary History 20(3): 627–649. https://doi. org/10.2307/469358 Guedron, Martial 2015. Visage(s). Sens et representations en occident. Paris: Hazan. Hwang Hyisung, Matsumoto David 2015. Evidence for the universality of facial expressions of emotion. In: Mandal, Manas K.; Awasthi, Avinash (eds.), Understanding Facial Expressions in Communication. New Delhi: Springer, 41–56. https://doi. org/10.1007/978-81-322-1934-7_3 Leone, Massimo 2014. Cultural semiotics as fluxorum scientia. In: Bankov, Kristian et al. (eds.), New Semiotics: Between Tradition and Innovation. Sofia: IASS, 112–130. https:// doi.org/10.24308/iass-2014-009 Leone, Massimo 2018. Designing imperfection: The semiotics of the pixel. Punctum 4(1): 105–136. https://doi.org/10.18680/hss.2018.0008 Leone, Massimo 2021. Esteticas faciais nas sociedades digitais contemporaneas. In: Teresco Vilardo Abreu, Maria Teresa; Carvalho Correia, Claudio Manoel de (eds.), Contribuicoes da Semiotica ao ensino de portugues no mundo. Rio de Janeiro: Dialogarts, 13–22. http:// hdl.handle.net/2318/1777594 Lotman, Juri M. 2009. Culture and Explosion. (Grishakova, Marina, ed.; Clark, Wilma, trans.) Berlin: De Gruyter Mouton. Lotman, Juri M. 2019[1983]. Toward a theory of cultural interaction: The semiotic aspect. In: Tamm, Marek (ed.), Culture, Memory and History: Essays in Cultural Semiotics. (Baer, Brian James, trans.) Cham: Palgrave Macmillan, 67–81. https://doi. org/10.1007/978-3-030-14710-5_5 Lotman, Juri M. 2019[1985]. Memory in a culturological perspective. In: Tamm, Marek (ed.), Culture, Memory and History: Essays in Cultural Semiotics. (Baer, Brian James, trans.) Cham: Palgrave Macmillan, 133–137. https://doi.org/10.1007/978-3-030-147105_9 Lotman, Juri M. 2019[1989]. Culture as a subject and its own object. In: Tamm, Marek (ed.), Juri Lotman – Culture, Memory and History: Essays in Cultural Semiotics. (Baer, Brian James, trans.) Cham: Palgrave Macmillan, 83–93. https://doi.org/10.1007/9783-030-14710-5_6 Magli, Patrizia 2004[1995]. Il volto e l’anima. Fisiognomica e passioni. Milano: Edizione CDE. Marchianò, Grazia 1977. La parola e la forma. Bari: Dedalo. Migliore, Tiziana 2018. La procedura di descrizione. E|C, On-line Journal of Aiss. Polacci, Francesca (ed.) 2012. Ai margini del figurativo. Siena: Protagon editori. Saraswati, Swami Satyananda; Saraswati, Swami Niranjanananda 2013. Prana vidya. Munger, Bihar, India: Yoga Publications Trust. Squarcini, Federico, ed. 2019[2015]. Patañjali Yogasūtra. Torino: Nuova Universale Einaudi. Torop, Peter 2020. The chronotopical aspect of translatability in intersemiotic space. Punctum 6(1): 265–284. https://doi.org/10.18680/hss.2020.0013 462 Silvia Barbotto Искусство, лицо и дыхание: от воздуха до текстов культуры Мы рассматриваем дыхание как обширную перспективу, которая включает в себя действия и их последствия, которая создает образы и тексты, которая включает в себя человека и окружающую среду; мы называем это «пейзажем дыхания». Затем мы изучаем, как «пейзаж дыхания» взаимодействует с человеческим лицевым аппаратом, порождая знаки и крайне интригующие интерпретационные процессы на мереологической шкале. Как и где взаимодействуют территория дыхания и тела? Какие процессы означивания они порождают? Какие знаки создаются в ходе их феноменальной и семантической встречи? Искусство, безусловно, является наиболее подходящим языком для изучения этого процесса, так как позволяет проявлять и исследовать контрасты. Существуют различные интерпретации «пейзажа дыхания», которые в разных культурах ассоциируются с древними и современными философскими течениями. В качестве отголосков семиотического взаимодействия и посредством различения и идентификации семантических полей, относящихся к рассматриваемому сценарию, введения текстуальности как феномена кристаллизации перехода между внешним текстом (текстом реальности) и текстом (предметом/объектом), мы рассматриваем те визуальные тексты, которые являются одновременно как описательными, так и изобретательными. Затем мы рассматриваем некоторые выводимые и культурные режимы и анализируем их конкретные практики, внося этим свой вклад в их обновление. Kunst, nägu ja hingusemaastik: õhust kultuuritekstideni Käsitleme hingeõhku avara väljavaatena, mis sisaldab tegusid ja nende jälgi, mis loob pilte ja tekste, mis hõlmab inimesi ja inimvälist konteksti; seda laia maastikuvaadet nimetame “hingusemaastikuks”. Seejärel vaatleme hingusemaastiku vastastiktoimet inimliku näoaparaadiga, milles mõlemad kutsuvad esile märke, aga tekitavad mereoloogilisel skaalal ka äärmiselt põnevate tõlgendusprotsesside liminaaltsooni. Kuidas ja kus mõjutavad teineteist hinguse territoorium ja keha? Milliseid tähistamisprotsesse nad esile kutsuvad? Ja milliseid märke nende tajutavas ja sematilises kohtumises luuakse? Kunst on kahtlemata kõige kohasem keel selle protsessi uurimiseks ja ka varju jääva esile toomiseks ning silmapaistvate kontrastide vaatlemiseks. On olemas mitmeid “hingusemaastiku” mõisteid, mis erinevatest kultuuridest tulenevatena koheselt seostuvad kas iidse või nüüdisaegse filosoofiaga. Semiootilise vastastiktoime järelkõlana ning asjakohase stsenaariumi suhtes oluliste semantiliste väljade eristamise ja tuvastamise kaudu ning tuues sisse tekstuaalsusnähtuse, nagu seda on välisteksti (reaalsuse teksti) ja teksti (subjekti/objekti) vahelise ülemineku kristalliseerumine, uurime neid visuaalseid tekste, mida ühendab joon, mis on niihästi kirjeldav kui ka leidlik. Seejärel võtame vaatluse alla mõned inferentsiaalsed ja kultuurirežiimid, analüüsime nende eriomaseid enuntsiatiivseid praktikaid ning aitame seeläbi kaasa nende uuendamisele. Tekstid, mis on nende praktikatega seotud kultuurilistele semiosfääridele omaste narratiivide osana, rõhutavad käsitlemist vajava epistemoloogilise sisu mõõtmatut suurust ning korpuse funktsionaalne reduktsionism on üksnes esialgseks lähenemisviisiks valdkonnale, mis peaks valgustama panoraamset üldvaadet ja soodustama edasist arutelu ning arusaamist. Faces in the pre-Hispanic rock art of Colombia 463 Sign Systems Studies 49(3/4), 2021, 463–488 Faces in the pre-Hispanic rock art of Colombia: Semiotic strategies, visual semiospheres, and gestures Martín Cuitzeo Domínguez Núñez1 Abstract. This article analyses the sign systems or semiotic models that make up the meaning of a double face or mask drawing in the pre-Columbian rock art of Colombia, also discussing two human figures with depicted faces associated with the main picture. The sample of rock art was detected on the walls of the Chicamocha Canyon at the Mirador de Bárcenas site in the Santander Department in Northeast Colombia. Its origin is attributed to the Guane chiefdom. We hold as a central argument that this face and its gestures were part of a sign system or visual semiosphere that spread along the banks of the Chicamocha Canyon. However, the image shared some semiotic models and visual communication strategies with societies that inhabited areas in central Colombia located hundreds of kilometers away from the site studied. We support this claim because the forms, the use of the space in the images, and the gestures of a hieratic character appear both in the faces of the petroglyphs of the Huila Department as well as in some pre-Hispanic gold masks from the Cundinamarca region. The images of the rock art faces were analysed using a visual semiotic model based on the suggestions of the Mu group and analyses by Félix Thürleman and Jean Marie Floch. The analysis was complemented by Jacques Fontanille’s proposal for the levels of semiotic relevance, with Juri Lotman’s concept of the semiosphere employed as a key concept. Keywords: rock art of Colombia; pre-Hispanic faces; Guane culture; ancient semiotic systems; visual semiosphere; gestures 1 Escuela de Antropología e Historia del norte de México, Chihuahua, México Licenciatura de Arqueología; e-mail: martin_dominguez@inah.gob.mx. https://doi.org/10.12697/SSS.2021.49.3-4.11 464 Martín Cuitzeo Domínguez Núñez 1. Introduction How to define a face from a semiotic perspective? How to account for the historical transformation of the face and the systems of signs to which it has belonged? Answering these questions implies a cross-cultural and diachronic approach. To develop this perspective, this article is going to discuss the representation of five pre-Hispanic faces at a site of more than 500 years old cave paintings located in northeast Colombia, South America. The paintings have been attributed to the Guane culture or chiefdom and share similarities with images from other regions of Colombia. I analyse the images using an archeosemiotic approach, as well as employing an interdisciplinary methodology deriving from visual semiotics and archeology. The results are compared with petroglyphs, two gold objects, and a wooden sculpture from the Central region of Colombia, leading to the conclusion that the faces were part of a visual semiosphere and a system of signs that extended through the Chicamocha Canyon and shared elements with other regions of central Colombia, which serves as the central argument of the article. As a second argument, I claim that these faces are characterized by gestures and expressiveness that become manifest through particular visual details. The location and historical and anthropological situation of the study region are presented in the first part of the article, followed by an introduction of the site analysed and of the methodology and concepts used in this analysis, including such notions as the visual semiosphere, and the system of visual signs and gestures. The next section presents the results of the analysis of the faces, followed by a comparison of the faces with material from Central Colombia. Thereafter I will discuss the visual semiosphere to which the images belonged; the article concludes with comments on the faces, their gestures and meanings. 2. Location, geography, and cultures of the region Mirador de Bárcenas rock art paintings are located in the Chicamocha Canyon, in the Santander Department in Northeast Colombia. The 227 kilometers long Chicamocha Canyon is considered the second largest canyon in the world. It covers around 108,000 hectares and is around 2,000 meters deep (Colparques 2020)2. The canyon crosses the Departments of Santander and Boyacá and has been formed by the effect of the Chicamocha River, being the product of: 2 Information on the canyon can be found at Colparques 2020. Cañón del Chicamocha. Reserva Natural Regional, http://www.colparques.net/CHICAMOCHA. Faces in the pre-Hispanic rock art of Colombia 465 […] the erosion over the riverbed with the formation of great gorges on both sides of the Banks. The canyon begins in the surroundings of the municipality of Socha, Pisba, Chita and ends near the municipality of Lebrija. These mountains plates are the result of the tectonic movements of the earth where geological faults that comprise the Chicamocha territory up to regions such as Mesa de los Santos and the city of Bucaramanga pass and emerge, but its vast majority extends over the municipality of Aratoca3. (EcuRed 2020) Regarding the archeology of the region studied, Gerardo Reichel-Dolmatoff points out in his introduction to the archeology of Colombia: In the foothills and plateaus of the Eastern Cordillera, north of the Muiscas, lived some indigenous groups that shared many cultural traits with the inhabitants of the Cundiboyacense region, but that formed small independent chiefdoms, all with quite developed agriculture and with metallurgy, textiles and lithic carvings. The Guanes, Laches, Chitarreros and others spread across the current departments of Santander (Santander and Norte de Santander), but we do not know much about the archeology of these regions. The few pottery complexes known from this extensive area contain various (ceramic) painted types, with geometric motifs, which are most likely related to the Second Painted Horizon of the Ranchería River and, through this, to the bichrome complexes of western Venezuela. In any case, it seems that there has been close prehistoric contact between the Chibchaspeaking peoples – Muisca, Guanes, Laches, Tunebos, Sierra Nevada de Santa Marta – along the Eastern Cordillera, whose chronological, typological and sociocultural details are still missing to be elucidated. (Reichel-Dolmatoff 2016: 353–354) Reichel-Dolmatoff describes the geography of the region, but above all, he introduces us to the pre-Hispanic societies that inhabited the region, as well as the surrounding areas, such as Guanes, Laches, Chitareros or Chitarreros, Panches, and Muiscas. 16th-century Spanish conquerors gave these names to the different groups of people who inhabited the foothills and plateaus of the Eastern Cordillera. The Guane culture inhabited the territories in which Mirador de Bárcenas or Pozo Tocaregua is located. The Laches and Chitarreros or Chitareros were 3 Quoted from EcuRed 2020. Cañón de Chicamocha. https://www.ecured.cu/Ca%C3%B1on_ de_Chicamocha; the original reads: “[…] de la erosión sobre el lecho fluvial, con la formación de grandes desfiladeros en ambas orillas. El cañón inicia en inmediaciones del municipio de Socha, Pisba, Chita y termina cerca al municipio de Lebrija. Estas placas montañosas son resultado de los movimientos tectónicos de la tierra, en donde pasan y se desprenden fallas geológicas que comprenden el territorio del Chicamocha, hasta regiones como la Mesa de los Santos y la misma ciudad de Bucaramanga pero en su gran mayoría se extiende sobre el municipio de Aratoca.” Translations from Spanish are by the author of the article. 466 Martín Cuitzeo Domínguez Núñez neighbours of the Guane people. The Chitareros or Chitarreros lived north of the Guanes in the present Department of Norte de Santander, in the region of the city of Pamplona. The Spanish gave them the name since when the people of these towns came out to meet them, they appered with containers with Chicha, a fermented drink made with corn and still widely consumed in rural Colombia today. The Chitareros who lived in cold areas were organized into chiefdoms, while those who lived in warmer lands lived in tribes. The people worked the land provided by a chief and, in return, they were paid in blankets and red dye. The caciques returned part of these goods during the festivities, when these were distributed among the community (Gamboa 2004: 755–756). This is an important antecedent to interpreting both the origin of the pigments in the cave paintings, as well as the presence of some symbolism that is found both in the funeral blankets and in the images studied here. The Laches lived to the south, in the Sierra Nevada del Cocuy; they were engaged in mining the salt. This mineral was used both for their own needs and for commerce with distant communities. Upon the arrival of the Spanish a process of emergence of social classes began within their society. Their religion was based on the cult of rocks, which they considered to have been the first human beings to inhabit the earth. At the same time, they were warlike societies that resisted the Spanish conquest (Acuña 2006). Yet another neighbouring culture of the Guanes was the Panches, who also inhabited the south of the Guane territory in the current department of Boyacá. 16th-century texts indicated that [t]his Panche People land is fertile as to maintenance and food, most of it, because another part of it is less abundant and another much less. And the misery in some part of the Panche People comes to so much that, while subjugating them, he ran into the Panches that surround the land of Tunja between two mighty rivers, in some mountains.4 (Banco de la República 1979: 95) The Muiscas or Chibchas, for their part, were the ancient inhabitants of the eastern Andes, in the present departments of Cundinamarca and Boyacá. They also lived south of the Guane territory. They formed a heterogeneous and complex society organized into chiefdoms, characterized by the coexistence of both egalitarian as well as profoundly unequal patterns. In this society, the economic and symbolic 4 The Spanish original reads, “Esta tierra de los Panches es fértil de mantenimientos y comida, la mayor parte della, porque otra parte della es menos abundante y otra muy menos. Y viene a tanto la miseria en alguna parte de los Panches que, andandolos subjetando, se topó en los Panches que ciñen la tierra de Tunja entre dos rríos caudalosos, en unas montañas.” Faces in the pre-Hispanic rock art of Colombia 467 processes were deeply rooted and it is difficult to differentiate them (Langebaek 2019). The Muiscas were characterized by their work of metallurgy, the craft of agriculture, pottery, textiles, salt work and specialized trade (Uribe, Martinón 2012: 773). They were also characterized by their tombs dedicated to their chiefs or caciques, as well as by the cult of rocks that was expressed in their diverse and vivid rock art. The Muiscas were related to all the societies mentioned above. These different cultures possessed, as we will see, cultural traits and types of organization similar to the Muiscas. Speaking of this complex mosaic of neighbouring cultures to the Guanes serves not only to introduce the reader to the complex cultural panorama of the region in pre-Hispanic times, but is also a first approach to the hypothesis that the images of the other Chicamocha cave paintings were part of a common semiosphere that was sustained by the interrelationships between the peoples mentioned here. The following section focuses on the Guane society. 3. Pre-Guane and Guane Societies The hypothesis that the Guane society was the author of the cave images of the Mirador de Bárcenas and other sites in the Chicamocha Canyon is proposed here for two main reasons: firstly, because some of the cave images have motifs similar to those found in Guane textiles; secondly, for the Spaniards’ chronicles from the time of the Conquista indicated that the Guane people inhabited the actual territories of Santander and Cañón del Chicamocha (Castellanos 1886; Simón 1891; Fernández de Piedrahita 1942; Banco de la República 1979; Gualdrón, Bermúdez 2018: 57). This hypothetical statement only serves as the starting point to our interpretation of the Chicamocha cave paintings, for we are primarily interested in understanding the semiotic, symbolic and social processes that gave rise to the cave images. According to the archaeologist Justus Schottelius (1941) and the archaeologist Leonardo Moreno (2012: 127, 139), the Chicamocha region was inhabited first by the Preguane societies. Preguane people had sedentary settlements in the region, especially in Los Santos. There are indications of Preguane people from around the 400 A.D. (Moreno 2012: 127, 139). Around the 12th century, the Preguane population underwent a transformation because it disappeared. According to Moreno (2012: 139), this could be the consequence of migrations and cultural assimilations in some regions and of violent impositions in others. The Guanes and Chitareros, two agroceramic societies, occupied the region during this time (Morales, Cadavid 1984). In the 16th century, when Spaniards arrived to conquer the region, the Guane societies were consolidated and completely settled in the Chicamocha area. 468 Martín Cuitzeo Domínguez Núñez The Guane culture spread through the northern region of the Chicamocha Canyon, the Mesa de Los Santos, and the Mesa de Bucaramanga. In some cases, housing sites were built on the alluvial terraces on one side of the Chicamocha River. At the time of contact with the Spanish, the Guanes occupied the southern and western lands towards the Suárez, Chicamocha, and Barichara rivers. They also inhabited the Socorro region, Oiba and the Serranía de los Cobardes (Restrepo 2010). The Guane chiefdoms belonged to an agricultural society that grew corn, as well as yucca, cotton, chili, pumpkins, and coca. At the same time, their livelihood depended on the consumption of, for example, ants, and other insects. The Guanes wove cotton, produced ceramics, built small circular buildings, as well as stepped buildings (Sutherland 1971). They made rock art paintings and cultivated land on artificial terraces at the foot of the hills. Their houses were artificially levelled through the construction of platforms. They also had two or three parcels of land located in different regions, and used these spaces to move from one place to another several times a year (Domínguez 2011). According to the Spanish friar Pedro Simón, the Guane worshipped a deity called Bochica, whose image was portrayed on stones found on the banks of the Sogamoso River (Gualdrón y Bermúdez 2018: 58) that rises at the confluence of the Río Chicamocha and the Río Suárez. On the other hand, Bochica is identified as a “civilizing” god to whom the origin of many cultural elements was attributed; he was also a deity worshipped by the Muiscas. 4. The cave paintings of the Mirador de Bárcenas and the Chicamocha Canyon region The Mirador de Bárcenas paintings, elaborated presumably by the Guane society, consist of 11 rock shelters with cave paintings (see Fig. 1). The two predominant colours are red and white and the paintings are characterized by geometric, animal, plant, and human-shaped images. The geometric motifs include circles, ovals, triangles, lemniscates – figures that, for our society, resemble an eight or an hourglass, triangles, ovals with straight lines inside them, half ovals, and rectangles. The animal forms possibly represent reptiles, amphibians (see Fig. 2), and insects such as ants; the vegetables are scarce and represent trees, possibly Guayacanes (Gualdrón, Bermúdez 2018: 185–186) – see Fig. 3. Finally, there are human images that are represented with extended limbs, which is important to notice because some of the faces covered in this study belong to this kind of human figures. To infer everyday living habits, the cosmology and mythology, of these societies, it should be asked what elements, processes, and prerequisites were needed for Faces in the pre-Hispanic rock art of Colombia 469 the realization of the rock images. This section gives an account of the economic, social, and ecological processes that were required for the realization of the images and provides an explanation to the main painting motifs of the Mirador de Bárcenas and its associations, which will allow us to understand the rock faces of this site. However, first the characteristics of the images should be described. In answer to the question what kind of elements and social, ecological, and economic processes were required for the realization of these motives, it appears that, in terms of the motifs of animals and plants, knowledge of the animals and plants was required, their habitual use and consumption were important, as well as a symbolic appreciation of them. Thus, there are ants among the animals represented (Fig. 4). The economic importance of the ants in pre-Hispanic times is documented in the colonial chronicle Epitome of the conquest of La Nueva Granada (Epítome de la Conquista de La Nueva Granada): […] a province of people, not very small, whose maintenance was nothing but ants, and make them bread to eat by kneading them. The ants are very abundant in the same province, they are raised in corrals for the same purpose; and the corrals are separations made of wide leaves that surround the anthill. And there, in that province, diversities of hotmygas (ants), some large and others small. 5 (Banco de la República 1979: 95) The breeding of ants and their consumption spread between the Guanes and the Panches. This allows us to infer that the Guane society interacted with their environment via the use of insectivorous resources. There is also evidence of the cultivation of corn and other vegetables and plant resources. In the course of fieldwork carried out in 2011, I observed remains of ancient corn and arrows made of wood wrapped in vegetable fibres in a cavity behind a cave painting at the Vereda San Rafael rock art site. The Guane society had an organization and social structure that allowed it to grow corn and cotton, raise insects and produce blankets. It was a chiefdom with incipient social differentiation, and I argue that the cave paintings of this region, especially the rock shelters analysed here, are related to this social differentiation. 5 Original in Spanish “[…] una probincia de gente, no muy pequeña, cuyo mantenimiento no hera otra cosa sino hormigas, y dellas hazen pan para comer, amasándolas. De las quales hormigas ay muy grande abundancia en la mesma probincia, y las crían en corrales para este mismo effetto; y los corrales son unos atajos hechos de hojas anchas con que rodean el hormiguero. Y ansy ay allí, en aquella probincia, diversidades de hotmygas, unas grandes y otras pequeñas.” (Banco de la República 1979: 95). 470 Martín Cuitzeo Domínguez Núñez Figure 1. General view of the Mirador de los Santos site6. Figure 2. Rock Shelter 1 through a digital filter. Reptile images and lemniscates can be detected. 6 All the photographs have been taken and edited by the author of the article. Faces in the pre-Hispanic rock art of Colombia 471 Figure 3. Rock Shelter 2. An image of a Guayacán tree is located in the centre. Figure 4. Ant detail in Rock Shelter 2. 5. Methodology and concepts The methodology used here has two dimensions: on the one hand, the archaeological dimension, in charge of accounting for the materiality of the images; on the other hand, the semiotic dimension, responsible for the study of the images as forms of representation belonging to a visual semiosphere. This approach is located within the perspective that Roberto Flores has called ‘archaeosemiotic’. ‘Archaeosemiotics’ is a term that refers to “the study of the meaning offered by the materials of the past, both to their creators and initial users, as well as to archaeologists and archaeology lovers (who, undoubtedly, are also, in their way, creators and users)” (Flores 2019: 176). Thus, an archaeological record of the cave paintings was first completed in order to analyse the images. The paintings were documented according to two 472 Martín Cuitzeo Domínguez Núñez registration approaches, one deriving from the American Association of Rock Art (Arara 2007) and the other from the Mexican archaeologist Miguel Messmacher (Messmacher 1981). The paintings were geographically located, the geology and geography of the surroundings were described, both general photos and pictures of each motif were taken. Dstrech software was applied that allows highlighting details that cannot be observed with the naked eye, using digital filters colours (Harman 20117). As part of the project, two cave sites were registered: El Mirador de Bárcenas or Mirador de los Santos and La Vereda San Rafael. Regarding the semiotic analysis, the plastic components, that is, the forms, spaces, colours and textures of the images, were studied. The panel was divided into three sections based on the grouping of human forms and taking the largest image, which was a mask or a double face, as its axis. This methodology was based on the proposals for visual analysis suggested by Félix Thürleman (2004), Jean Marie Floch (1994) and the Mu Group (Grupo Mu 1993). However, it is also based on the visual analysis of a Coco Chanel advertisement conducted by Roberto Flores (2007) and on two semiotic analyses of cave paintings made by the author of this article (Domínguez 2009, 2011). Once this analysis had been completed, the images were placed in a model of semiotic relevance levels proposed by Jacques Fontanille (2006). The recovered levels were the sign, that is to say, an individual figure in our case; the text, i.e., which corresponded to the complete image or panel; the object or support, which in this case would be the surface of the rock shelter; the practical scenes that means the spatial interactions or relations of proximity or distance between the different caves with painting which constituted the archaeological site. The next step was to compare and contrast the images with those of the central and southern regions of Colombia. These included petroglyphs, two pieces of gold, and a wooden sculpture. Through this analysis, a system of signs or a visual semiosphere was inferred. The concept of the semiosphere was taken from Juri Lotman to account for a series of similarities in the ways of representing faces. In this order of things, the visual semiosphere is derived from Lotman’s concept of the semiosphere (Lotman 1996: 12) as semiotic universe of the past made up of various texts and visual discourses. In this case, it is a semiosphere made up of multimodal discourses such as cave images, cotton blankets, and ceramic pictographs. These multimodal discourses were configured from sign systems and semiotic models typical of the social and cultural processes of the peoples from the studied region. It is difficult to define semiotic sign systems or models in the 7 Harman, John 2011. Using Decorrelation Stretch to Enhance Rock Art Images, available at http://www.dstretch.com/AlgorithmDescription.html. Faces in the pre-Hispanic rock art of Colombia 473 archaeosemiotic domain. For this purpose, Winfried Nöth’s definition of systems was appropriated, understood as a set or assembly of signs, relationships, and structures (Nöth 1995: 198) of the societies that inhabited the region in the past. The system of visual signs was defined as a set of images used by past societies. These images were related to one another in such a way that they had a common visual structure. This definition was complemented by Jean Marie Klinkenberg’s idea of system and code (Klinkenberg 2006: 138–139). Thus, the semiotic models of the past were a set of differences of visual signs with two dimensions, that of the signifier and that of the signified. These dimensions had codes that structured them. There were visual codes that configured the relations of difference between images on two planes: that of expression and that of content. To return to Lotman’s concepts, these visual systems were both static and dynamic, closed and open at the same time (see Lotman 1996). From an emic perspective, or from within a society, these systems of signs can be called semiotic models, understood as models and schemes of the world that the society creates. However, from an external perspective, the schemes created by semiotics can be called systems. This article seeks to build a semiotic model in order to study the systems of signs and visual semiotic models of the pre-Columbian societies that created the cave faces of the Chicamocha Canyon. I consider these sign systems to have been expressed in the semiotic-visual strategies of the images studied, which is why, to reconstruct the system of signs, we must study these strategies. The last step consisted of the analysis of facial gestures. The dictionary of the Royal Academy of the Spanish Language (Diccionario de la Real Academia de la Lengua Española 20208) defines ‘gesture’ (‘gesto’) as “Movement of the face, hands or other parts of the body, with which affections are expressed or messages are transmitted” and also defines ‘expression’ (‘expresión’) as “the manifestation of affections and emotions through gesticulation”. Such facial expressions and gestures are considered by Theo van Leeuwen as physiological communication resources and as elements of non-verbal communication (Leeuwen 2005: 93). 6. Description and results of the semiotic analysis of the faces The rock art set in which the faces were depicted is located in a yellow rocky shelter, which is oriented towards the east, the direction from which the sun rises. It is composed of 12 images (Fig. 5). To analyse these, the set of images was divided into three sections based on the observation of the way the motifs were grouped. 8 See Diccionario de la Real Academia de la Lengua Española 2020: https://dle.rae.es/gesto?m= form. 474 Martín Cuitzeo Domínguez Núñez Segment One corresponded to a concentration of medium and small elements in the left sector of the set in which five elements were counted. Segment Two was composed of the double face and several small associated elements; in it we counted five images. Finally, Segment Three was made up of a series of small images located on the right side of the composition; the number of images in this segment was four. Figure 5. Rock art faces from Mirador de Bárcenas. In Segment One, visual significance has been created through the use of straight lines, rectangles, circular shapes, ovals, half-circles, points, and a dot. Some of these forms create the image of two human figures with identifiable faces and extended limbs. All the images are medium high if we compare them with the images of other segments. As concerns their spatial location, one of the human figures is situated in the lower right, very close to the centre of the scene. Above, a strip of six horizontal lines can be seen on the left-hand side, a human figure in the centre, and a semicircle with ten points on the right; above these forms a spot can be observed. As regards the colours, the shape below is red with white dots; in contrast, the human figure above is white and yellow with red dots. The semicircle is red with eight red and two white dots. I will describe the human forms, specifically their faces in more detail. The limbs of the first figure, the one in red, are presented as extended and flexed. The neck can be observed, as well as the face composed of Faces in the pre-Hispanic rock art of Colombia 475 a horizontal line serving as the mouth and two points as eyes. These are negative shapes whose colour is formed by the surface of the rock. Above the face, there is a headdress formed by approximately ten lines (Fig. 7A). The second human shape, the one at the top, is made up of points, straight lines, a curved line, an oval shape, and a rectangular shape. The figure has a headdress composed of seven red dots and seven white vertical lines. The face consists of an oval line, two red dots for the eyes, and a horizontal red line for the mouth. The face is attached to the white-coloured body that has a rectangular shape; its legs and arms are extended. The body is crossed by a vertical line and the four extremities have a red line at their ends (Fig. 7B). Next to this image, there is a half-circle with dots inside it (Fig. 7C). The double face located in Segment Two has been created, utilizing a half oval. In the upper part, there is a human figure with outstretched arms that consists of a circular shape, points and a horizontal band, and an inverted triangle shape. The vertex of the triangle extends downwards and forms the nose of the face or mask below. The previous figure is separated from the main face by four horizontal straight lines that converge in the half oval; these mark just half of the image. Two of the lines are on the left and two on the right. Inside the oval, forming the face or mask, there are three horizontal lines with the one at the top being longer than the two at the bottom; these lines make up the eyes and cheeks of the face (Fig. 7D). Finally, at the bottom, 14 curved vertical lines form a kind of a beard. Above the image on the upper right-hand side, there is a short Y-shaped line, as well as two triangles, one very small and one larger; these latter forms are hardly noticeable. The face or double mask is the largest figure in the entire composition, while, in contrast, the other shapes in this segment are small. Segment Two forms the centre of the entire panel, as the rest of the images are grouped around the double mask. Segment Three is made up of a human shape with a headdress formed by two vertical lines on the head and a curved line in the shape of a C. The human shape has been made by means of a V-shaped line as well as points. The predominant colour is red, although the human form boasts white spots and dots. The images are peripheral and are located at the extreme right of the composition (Fig. 7E). 476 Martín Cuitzeo Domínguez Núñez Figure 6. General location of the human images. Figure 7. Human figures in detail. Faces in the pre-Hispanic rock art of Colombia 477 6.1. Visual strategies used in the elaboration of the faces A visual strategy consists in a principle of visual syntagmatic composition (Fontanille 2016: 30) that orders the relationship between shapes, spaces and colours in an image. It is also one of the semiotic levels of relevance. In this case one of the main strategies consisted in placing the larger face in the centre of the composition. It is inferred that this strategy helps to emphasize the visual importance of this image, referring us to visual hierarchies in which the largest objects are located in the centre. I propose here as a hypothesis that this form of hierarchization may have been a reflection of a Guane society in the process of social differentiation that involved the beginning of social hierarchies. As a second strategy it can be argued that faces were elaborate, using oval and circular geometric shapes. Straight lines were employed in order to making up the eyes and the mouth. Also the opposition in the use of two colours, white and red, was part of this strategy. Thirdly the use of headdresses is observed always on the top of the faces. In my opinion, their presence emphasizes the hierarchical status of the faces, which seems to have been the same in the Muisca region of Bogotá where death masks with headdresses were associated with the chiefs or caciques. 6.2. Colours and meanings So far, the colours red and white have been described; however, their meaning has not been mentioned, as archeology and semiotics alone cannot provide us with these meanings. For this reason, it is important to talk about these colours both in an universal and in a regional perspective. The relationship between the red and white colours is almost universal, in many cases the colour black is also part of this link. Thus, the red-white opposition has been observed since the Neolithic era, in Africa, Eurasia and Europe, where it also appears in classical cultures and in the medieval world. The relationship is also observed in the folklore and ethnography of various contemporary peoples (Hemming 2012: 310). It is highly probable that this universality is the product of neuro-physiological processes. Red is the most visually exciting colour that the human being can perceive; this excitability probably has to do with human evolution and with the purpose of our arboreal ancestors to search for food (Hemming 2012: 325). In Ancient America, red was the most frequent colour in cave paintings. In Colombia, the indigenous Uwas or Tunebos who live in the Sierra Nevada del Cocuy, in the central-eastern region of the country, have myths about the meaning of the colours white and red. Everything indicates that these communities are descendants of the Laches, a pre-Columbian people with whom the Guanes interacted, as noted above. 478 Martín Cuitzeo Domínguez Núñez Eduardo Londoño (1995: 88–89) describes an Uwa myth about the white and red colours: “[…] at the beginning, the world was made up of a top (white) and a bottom (red); when the movement occurred, these separate worlds mixed and the middle world emerged, made up of blue and yellow”. This gives rise to the hypothesis that the red and white colours of the faces of the cave painting analysed may be related to myths about the origin of the world and colours. However, when interpreting the colours of the paintings, it must be taken into account that sometimes these deteriorate over the centuries and that the original hue may have changed. Still, it should be noted that the red colour, generally made from iron oxide, is one of the pigments that survives best over time (Argüello, Martínez, 20049). As for the white, we believe that it is also of mineral origin, but it is necessary to carry out a chemical analysis of the paintings of the Mesa de Los Santos to determine the origin of the pigments with any certainty. At least to the naked eye the rock art pigments appear to be different from those used in other archaeological objects such as e.g. the pre-Hispanic textiles. Archaeological studies of Chibchas, Uwa, and Guane pre-Hispanic textiles indicate that dyes of vegetable origin, especially a small plant called Fridericia chica (Naturalista 202110), were used in 60% of Colombian pre-Hispanic textiles. Furthermore, while Trunk of Brasil [dracaena; Palo de Brasil (GardenTags 202111)], a tree from whose wood a dye is obtained that is sometimes red and sometimes yellow (Rico Arce 1999), was used in 10% of the sample (Devia et al. 2016: 8)12. This dye is still widely used today. 6.3. Gestures and meanings: From semiospheres to gestures and facial expressions The last point we concentrate on here concerns the expressiveness of the faces and its counterpart, hieratism. ‘Expressiveness’ is understood as the movements of the 9 Argüello, Pedro y Martínez, Diego 2004. Procesos tafonómicos en el arte rupestre: un caso de conservación diferencial de pinturas en el altiplano cundiboyacense, Colombia. Rupestreweb; can be found at: http://rupestreweb.info/sutatausa.html. 10 Naturalista 2021. Fridericia chica; see https://colombia.inaturalist.org/taxa/578619Fridericiachica. 11 GardenTags 2021 Dracaena Fragrans ‘Trunk of brazil’ syn. ‘Palo de brasil’; see https://www.gardentags.com/plant-encyclopedia/dracaena-fragrans-trunk-of-brazil-synpalo-de-brasil/38431. 12 See Devia, Baetriz; Cardale de Schrimpff, Marianne; Devia, Carlos; Niño Izquierdo, Camilo 2016. Aproximación al conocimiento de los colorantes en la comunidad indígena Ika de la Sierra Nevada de Santa Marta (departamento del Cesar, Colombia) Nuevo Mundo, Mundos Nuevos, Coloquios. http://journals.openedition.org/nuevomundo/69205; https://doi. org/10.4000/nuevomundo.69205. Faces in the pre-Hispanic rock art of Colombia 479 face that communicate messages and also its visual effect. Specifically, we refer to the appearance of movement transmitted by two parts of the face: the eyes and the mouth. Expressiveness is opposed to hieratism, that is constituted as its opposite, that is, the visual effect suggesting that the face lacks movement. Gestures have biosemiotic, social and cultural dimensions. Among different animals and primates they communicate aggressiveness, fear and other emotions. In the case of human beings and, more specifically, of the cave faces of the Mirador de Los Santos, expressiveness has been permeated by social and cultural semiotic processes. We can argue in this case that the people who made the paintings captured a series of cultural codes and world schemes through the gestures and hieratism of faces. Two of the faces seem to have their mouths open, which generates a sensation of movement. It seems that these faces are screaming. There is a third face that conveys the visual sensation of movement, since one of its eyes is larger than the other. In contrast, the mask and the face with two bumps do not seem to show movement and therefore appear to have no emotion. Thus, it seems that the visual semiosphere of the region had at least two different types of visual representations that schematized and encoded facial gestures: one consists of the representation of faces in movement, while the other represents faces statically. This opposition between movement and immobility is not unusual since the concept of opposition is present also in other rocky shelters of the Mirador de Los Santos. 6.4. From facial gestures to meanings What kind of construction of specific meanings was behind these faces and their two types of expressiveness? For other cultural regions of America such as Mesoamerica in Mexico, rock faces have been taken as an expression of individuality, as a representation of the mountain as divinity, as ancestors who protect and own mountains and ravines (Valdovinos 2014: 1–11). How does this relate to the case presented here? I argue that in Mirador de Bárcenas there was an enormous variability in the faces represented, which indicates the importance of the individuality of every face, as it happened in ancient Mesoamerica. Regarding faces and mountains as a dimension of divinity, it is not easy to answer the question if there was an indigenous worldview in the Guane region that related ravines and hills with gods. For this purpose a more indepth investigation would be required regarding the pre-Hispanic worldview of the region and surrounding regions. 480 Martín Cuitzeo Domínguez Núñez However, there are alternative interpretations. For instance, Camilo Gualdrón and Oscar Bermúdez have interpreted the scene of the faces as a ritual in which the characters were dressed in animal costumes. They mention the Huan ritual of the Muisca people of the Bogotá region celebrated in December, during which the participants placed red and blue ornaments on their heads, sang and danced (Gualdrón, Bermúdez 2018: 211). The same authors have suggested that the double face is possibly the mask of a feline as their research has shown that the current inhabitants interpret this image as a feline; they also realized that the colonial chronicler Pedro de Simón affirmed that there were sorcerers who turned into felines (Gualdrón, Bermúdez 2018: 216). In addition to this interpretation, the authors have pointed out that the central face of the panel could also be “Bochica or the supreme deity Chiminigagua. This is due to the central position of the image, as well as its larger size compared to the rest of the motifs of the scene, which would denote its importance in the pantheon” (Gualdrón, Bermúdez 2018: 214). These hypotheses, certainly seeming to be promising, should be tested, yet seem quite plausible considering the available data. To complement the previous hypotheses I would like to propose a last one, suggesting the possibility that the faces were related to the representation of the dead and the deceased, as well as to the ancestors. I hold this for two reasons: the first is that human burials were found behind the different rock sets many years ago (Nelson Ramírez, personal communication); and, secondly, the rock motifs from other painting sites in the region, such as the Vereda San Rafael, have also been found in funeral blankets in which the dead have been wrapped. This hypothesis should certainly be tested in the course of future research. To end this section, we might recall the comments made above on the Guane society, its worldview, and that of neighbouring peoples. In the first place, the colours white and red seem to refer to a myth of the origin of the world, in which all colours and the world in between them originate from the mixture of white and red. At the same time, we have the aforementioned hypothesis that the face in the centre may correspond to the image of the civilizing god Bochica. Rocks were considered the first human beings who inhabited the earth, to which we may add the idea proposed in the previous paragraph about ancestors having been represented in the rock art paintings. Thus, both the faces and the support seem to reflect the myth, the world of the dead, and the ancestors, but also the social stratification and the hierarchies within the Guane society. An argument to support this is that, as Langebaeck (2019) points out in the Muisca case, materiality, economy, social structure, and symbolism cannot be separated in the societies of pre-Hispanic Colombia. Faces in the pre-Hispanic rock art of Colombia 481 7. Comparisons and contrasts between the faces and the visual semiosphere of the region Behind the faces analysed here, there was a visual semiosphere characterized by visual rules and strategies. Elements of social structure, myth and worldview were also represented in it. Other cave painting sites in the region such as Vereda San Rafael employed similar strategies and combined economical and symbolic topics. For example, in the Vereda San Rafael rock art site, a beetle and a fantastic creature with many legs were depicted: the insects, that were part of the Guane economy and diet, were combined with mythical creatures. In other archaeological objects of Santander, we can observe visual strategies similar to those used in Mirador de Los Santos, but adapted to specific supports. These supports are pre-Hispanic ceramics and cotton blankets, which are currently sheltered in the Bolívar House Museum in the city of Bucaramanga. The ceramics, although very diverse, include specimens in case of which the use of the colours red and white can be observed as well as the use of forms similar to those employed in rock art. Despite this, there is no evidence of representations of faces having been found in the ceramics of the region (Domínguez 2011: 96). The blankets, meanwhile, are cream or white in colour; sometimes red inks were used and they were decorated with circular shapes. The blankets accompanied burials in caves (Cardale 1987), but no faces have been depicted on the blankets. In addition to these cases, the regions of Bogotá and the Huila Department, inhabited by Muiscas, Panches, Laches, and other cultures mentioned in this article, share visual and sociocultural elements, even though these areas were located hundreds of kilometers away from the study site. 7.1. Petroglyphs from Aipe The painted rock from Aipe, in the north of the department of Huila in central Colombia, has a series of petroglyphs made by the ancient societies of the region. These images can be compared with the depictions of humans and faces analysed here. The stone was recorded in the 19th century by the painter Manuel María Paz employing a watercolour technique (Biblioteca Digital Mundial 2020)13. It consists of a large rocky massif with human shapes, some possible plant representations, as well as geometric motifs and points on it, and is still standing today. There are more than a dozen of human forms that look straight ahead and whose limbs are either 13 Biblioteca Digital Mundial 2020. Piedra con jeroglíficos cerca de Aipe, en la margen izquierda del Magdalena, provincia de Neiva, see https://www.wdl.org/es/item/9146/. 482 Martín Cuitzeo Domínguez Núñez flexed or extended. Two of the figures are similar in shape both to the paintings in Mirador de Bárcenas and to those on the gold objects described below. The similarity lies in them having bent arms and two protrusions on the head. On the other hand, the shapes resemble the profile of the double mask, since they have a human shape at the top and are composed of a semicircle at the bottom. Other similarities involve the combination of both economic and symbolic elements, such as corn plants and curve pectorals (Falchetti 1989: 25–26) in the case of the Aipe petroglyphs. These features are the indication of common visual strategies such as the use of similar forms and the representation of prominent elements over the head. 7.2. Gold masks and faces and their cultural meanings Gold was a common raw material for the pre-Hispanic societies that inhabited Colombia, especially for the Muisca culture. This material, usually associated with offerings and tombs, was used to make masks and human figures whose faces were discernible (Fachetti 1989; Uribe, Martinón 2012). These artifacts allow a comparison with the rock art faces of the Chicamocha and with their details such as the headdresses. The gold objects analysed in this article are located in the Gold Museum (Museo del Oro) in Bogotá; in some cases, they belong to the societies that inhabited the current Bogotá savanna, in other cases it is difficult to specify the origin of the objects. The artifacts with which I will compare the Guane rock art will be a breastplate, a gold boat, and, finally, a face carved in wood. The cultural significance of these objects for the Muiscas has to do with the offerings to which they belonged. Thus, these pieces consist of representations of schematic human beings and are associated with the burials of chiefs or political and religious leaders, as well as the consumption of psychotropic substances and related rituals. They are also linked to human sacrifice, war, and the civilizing god Bochica or Chibchacum who was worshipped by both Muiscas and Guanes (Uribe, Martinón 2012: 782–784). Thus, the three faces described here were part of such offerings of ritual, political, and warrior significance. 7.3. Gold breastplate The first piece with which I want to contrast the images of the Chicamocha is a gold breastplate that represents a human figure (Banco de la República 2020a14). As in the Chicamocha images or the petroglyphs previously analysed, the figure’s 14 Banco de la República 2020a. Arte prehispánico: un recorrido por el ciclo del oro y la sala Cosmología y simbolismo desde la mirada del arte can be found at https://www.banrepcultural.org/ bogota/actividad/arte-prehispanico-un-recorrido-por-el-ciclo-del-oro-y-la-sala-cosmologia-y-6. Faces in the pre-Hispanic rock art of Colombia 483 limbs are flexed – this position of the body seems to appear across the visual semiosphere studied. The body is decorated with horizontal and diagonal lines, as well as dots. To elaborate or construct the face of the figure, a semicircle has been used, the same form on the faces of the Chicamocha images. The nose is made using a line and is reminiscent of the nose of the double face, previously analysed in the case of Mesa de Los Santos, while circles have been used to constitute the eyes and the lashes. In its lower part, the face wears earrings. Finally, it can be said that the face bears a hieratic expression. The meaning of this image could be related to the representation of a divinity, presumably Bochica. 7.4. Gold boat The second gold object with which I wish to compare the faces is the so-called gold boat – the representation of a raft on which a chief is observed accompanied by his servants. The object was found in 1969 by the peasant Cruz María Dimaté in a cave in the department of Cundinamarca near Bogotá. Other pieces of gold and ceramic objects were associated with the finding (Banco de la República 2020b15). It is related to the descriptions, deriving from the time of the conquest, of a ritual in which the Muisca caciques were transported by a raft through the lagoon of Guatavita, celebrating a ceremony (Banco de la República 2020b). As for the human figures in this piece, several elements can be highlighted, first of all the hieratic, rigid attitude of these figures. Their faces have been made by means of oval shapes and they wear elaborate headdresses that are very similar to the headdresses of the faces in the Chicamocha region. I suggest that the meaning of the boat is related to the political power of the caciques. The reason why I make the comparison with the Guanes paintings is both because of the similarity between the human representations, and because both figures were used by caciques as part of a process of social differentiation. The piece is full of details that, by themselves, like other pieces in the Gold Museum, would deserve a separate analysis; however, here I comment only on the description of the faces. Next, I will comment on a preHispanic face depicted on a piece of wood found in the central region of Colombia. 7.5. Face carved in wood Finally, I want to point out that there are other types of supports such as wood or stone in which faces have been represented as well. These faces have both similarities and differences with the cave paintings of Chicamocha. At this point I 15 Banco de la República 2020b. La balsa muisca y el dorado can be found at https://www. banrepcultural.org/coleccion-arqueologica/balsa-muisca. 484 Martín Cuitzeo Domínguez Núñez would like to mention the wooden sculpture of a human being found in the Cave de Bojacá in the Sabana de Bogotá (Reichel-Dolmatoff 2016: 354). The figure has bent limbs, but these are close to the body. The eyes are represented as points, as in most of the faces of the Chicamocha and the nose is a vertical line similar to the nose of the double mask. I suppose that this carving represents an ancient Muisca divinity, probably Bochica. Additionally, like the rock art paintings, it is related with a rock shelter. This relationship and the importance of the rocks among Guanes and Muiscas is the reason why I introduce also this object as an element of comparison. 7.6. The visual semiosphere of the region It seems that from these descriptions a common way of representing the face among Guanes, Muiscas, and other indigenous peoples from pre-Hispanic times can be inferred. In my opinion this serves as evidence of the existence of a semiosphere in which semiotic codes, mythological elements, and socio-political features were shared. However, there are also differences between the regions, some concerning the raw material employed to elaborate it. Others have to do with the forms used in the representation of the faces, or with the meanings. Thus, I argue that the ethnic diversity of the region has given rise to these variations in the ways of representing and signifying faces and bodies. 8. Conclusion The Guane society used a system of visual signs or semiotic models that was part of a large visual semiosphere extending through the Chicamocha, but sharing elements with cultures such as the Laches, Chitareros, and Muiscas. This semiosphere integrated visual codes and strategies with social, economic, and mythological elements. The faces depicted in Mesa de Los Santos rock shelters by the Guane culture were made by using geometric shapes such as semicircles, lines, and points. These faces had some formal characteristics shared with the Bogotá and Huila region, such as the use of similar forms to represent the face and the body. The two most used colours were white and red. Using a contemporary Uwa myth we can hypothesize that both colours are related to a story according to which only the colours red and white colours existed at the beginning of the world. The cave faces of the Guane culture also had a possible mythological dimension. To this would be added the interpretations suggesting that the central image could be related both to the god Bochica, a civilizing deity, and to the rocks Faces in the pre-Hispanic rock art of Colombia 485 that are linked to the first inhabitants on earth. Other symbolic dimensions of the faces could be related to the dead and worshipping the ancestors. However, faces also have a social dimension where they functioned as a form of legitimation and social differentiation on the part of the social segment that produced the paintings. Thus, through the study of the rock art faces from the Chicamocha canyon, we have tried to explain some social and symbolic processes of the Guane culture to which we attribute the authorship of the images. In addition to outlining the mythological and social dimensions already mentioned, I tried to explain the interconnections between the Guanes and other societies through the faces studied, emphasizing, above all, the relationship with the Muiscas of the Bogotá savanna region, but also with the Laches, Panches and Chitareros. These interconnections give us an account of a visual semiosphere sustained by the relationships and commercial contacts between these societies. Thus, the Guane sociocultural process did not occur in isolation but was part of a series of shared visual rules and codes, as well as similar sociocultural situations, between societies and chiefdoms that were in a process of social differentiation. Finally, this article has been an attempt to describe and interpret the Guane semiotic system, as well as the visual semiosphere to which the Guane society belonged. Thus, we have tried to show how in the pre-Columbian past of Colombia, the faces of rock art were part of complex symbolic and social meanings. References American Rock Art Association (ARARA) 2007. Basic Guide for Rock Art Recording. United States: American Rock Art Association. Acuña Rodríguez, Blanca Ofelia 2006. Territorio Indígena de la sal en La Sierra Nevada del Cocuy. S XVI. Historia y Espacio 2(27): 71–86. https://doi.org/10.25100/hye.v2i27.4569 Banco de la República 1979. Epitome de la conquista del Nuevo Reino de Granada. Boletín Cultural y Bibliográfico Banco de la República 16(03): 81–97. Cardale de Schrimpff, Marianne 1987. Informe preliminar sobre el hallazgo de textiles y otros elementos perecederos conservados Cuevas en Purnia, mesa de los Santos. Boletín de Arqueología 2(3): 3–23. Castellanos, Juan 1886. Historia del Nuevo Reino de Granada. Tomo I. Madrid: Imprenta de A. Pérez Dubrull. Domínguez Núñez, Martín Cuitzeo 2009. “Las Manitas” rock art site in Cañada de Cisneros, Tepotzotlán, México: An analysis using visual semiotic tools. American Indian Rock Art 35: 161–170. Domínguez Núñez, Martín Cuitzeo 2011. La dinámica enunciativa de dos sitios con manifestaciones gráfico-rupestres de la Mesa de los Santos, Departamento de Santander, Colombia. (Tesis de maestría.) Bucaramanga, Colombia: Universidad Industrial de Santander. 486 Martín Cuitzeo Domínguez Núñez Falchetti, Ana María 1989. Orfebrería prehispánica en el altiplano central colombiano. Boletín Museo del Oro 25: 3–41. Fernández de Piedrahita, Lucas 1942. Historia general de las conquistas del Nuevo Reino de Granada. Bogotá: Ministerio de Educación. Floch, Jean Marie 2002. Composition iv de Kandinsky. In: Henault, Anne (ed.), Questions de sémiotique. Paris: Presses Universitaires de France, 121–151. Flores, Roberto 2007. De Cuerpos, brillos y transparencias. Análisis semiótico de una imagen publicitaria. Escritos Revista Ciencias del Lenguaje 35/36: 7–40. Flores, Roberto 2019. Fundamentos de arqueosemiótica. Cuicuilco. Revista de ciencias antropológicas 76: 175–192. Fontanille, Jacques 2016. Prácticas semióticas. Lima: Universidad de Lima, Fondo Editorial. Gamboa, Jorge Augusto 2004. La encomienda y las sociedades indígenas del Nuevo Reino de Granada: El caso de la provincia de Pamplona (1549–1650). Revista de Indias 64(232): 749–770. Grupo Mu 1993. Tratado del signo visual. Para una retórica de la imagen. Madrid: Cátedra. Gualdrón, Camilo; Bermúdez, Oscar 2018. Cosmovisión, paisaje y Manifestaciones Gráfico Rupestres en el sitio Mirador de los Santos, Santander Colombia. (Tesis de pregrado.) Bucaramanga, Colombia: Universidad Industrial de Santander. Hemming, Jessica 2012. Red, white, and black in symbolic thought: The tricolour folk motif, colour naming, and trichromatic vision. Folklore 123(3): 310–329. https://doi. org/10.1080/0015587X.2012.716599 Klinkenberg, Jean Marie 2006. Manual de semiótica general. Bogotá: Fundación Universidad de Bogotá Jorge Tadeo Lozano. Langebaeck, Karl Henry 2019. Los muiscas: La historia milenaria de un pueblo chibcha. Bogotá: Editorial Debate. Leeuwen, Theo van 2005. Introducing Social Semiotics. London, New York: Routledge. https://doi.org/10.4324/9780203647028 Londoño, Eduardo 1995. Linajes y circuitos de matrimonio en tres linajes chibcha: U wa, kogui y muisca. Boletín Museo del Oro 38/39: 87–111. Lotman, Yuri 1996. Acerca de la semiósfera In: La semiósfera I. Semiótica de la cultura y del texto. España: Frónesis Cátedra, Universidad de Valencia, 10–25. Messmacher, Miguel 1981. Las pinturas rupestres de la pintada, Sonora. Un enfoque metodológico. México D. F: INAH. Morales, Jorge; Cadavid, Gilberto 1984. Investigaciones etnohistóricas y arqueo-lógicas en el área Guane. Bogotá: Fundación de Investigaciones Arqueológicas Nacionales. Banco de la República. Moreno, Leonardo 2012. Arqueología del Nororiente colombiano. Los Teres: un sitio de asentamiento de las culturas prehispánicas Preguane y Guane. Anuario de Historia Regional y de las Fronteras. 17(2): 115–142. Nöth, Winfired 1995. Handbook of Semiotics. Bloomington, Indianapolis: Indiana University Press. Reichel-Dolmatoff, Gerardo 2016. Arqueología de Colombia: Un texto introductorio. Bogotá: Ministerio de Cultura, Biblioteca Nacional de Colombia. Faces in the pre-Hispanic rock art of Colombia 487 Restrepo, Lotero John 2010. Reconocimiento Arqueológico para las concesiones Mineras 1 D4-09211 y 1 D2-16551. Cañón del Chicamohca. Departamento de Santander. Licencia 1480”. Bogotá: Proyecto de Salvamento. ICAHN. Rico Arce, Lourdes 1999. El palo de Brasil. Ciencias 54: 42–44. Schottelius, Justus 1941. Arqueologia de la Mesa de los Santos. Bogotá: Publicación de la Escuela Normal Superior, Imprenta Nacional. Simón, Pedro 1891. Noticias Historiales de las Conquistas de Tierra Firme en las Indias Occidentales. Tomo III. Bogotá: Casa Editorial de Medardo Rivas. Sutherland, Donald 1971. Preliminary Investigations into the Prehistory of Santander, Colombia. (Ph.D. Dissertation.) Tulane: University of Tulane. Thürlemann, Félix 2004. Blumen Mythos (1918) de P. Klee. In: Henault, Anne; BeyaertAnne (eds.), Ateliers de Sémiothique Visuelle. Paris: Presses Universitaires de France, 13–40. Uribe Villegas, Martha Alicia; Martinón Torres, Marcos 2012. Composition, color and context in Muisca votive metalwork (Colombia, AD 600-1800). Antiquity 86: 772–791. https://doi.org/10.1017/S0003598X00047918 Valdovinos Rojas, Elda Vanya 2014. Rostros en la montaña. Arte Rupestre en Tlayacapan y Amatlán Morelos. (Tesis de maestría.) Ciudad de México: Universidad Nacional Autónoma de México. Лица в доиспанском наскальном искусстве Колумбии: семиотические стратегии, визуальные семиосферы и жесты В статье анализируются знаковые системы и семиотические модели, составляющие семантику рисунка двойного лица или маски в доколумбовом наскальном искусстве Колумбии. Также рассматриваются две человеческие фигуры, связанные с основной картиной. Образец наскального искусства был обнаружен на стенах каньона Чикамоча на участке Mirador de Bárcenas в департаменте Сантандер в Северо-Западной Колумбии. Его происхождение приписывается племени гуане. В основе работы лежит предположение, что это лицо и его жесты были частью знаковой системы и визуальной семиосферы, распространенной по берегам каньона Чикамоча. В то же время, это изображение повторяет некоторые семиотические модели и стратегии визуальной коммуникации обществ, которые населяли районы в центральной Колумбии на расстоянии сотен километров от изучаемого участка. Формы, использование пространства на изображениях и жесты религиозного характера появляются как в лицах петроглифов департамента Уила, так и в некоторых доиспанских золотых масках из региона Кундинамарка. Изображения лиц в наскальном искусстве были проанализированы с использованием визуальной семиотической модели на основе предложений группы Мю и анализов Феликса Тюрлемана и Жана Марии Флоха. Анализ был дополнен уровнями семиотической релевантности Жака Фонтаниля, при этом в качестве ключевой использовалась концепция семиосферы Юрия Лотмана. 488 Martín Cuitzeo Domínguez Núñez Näod Colombia Hispaania-eelses kaljukunstis: semiootilised strateegiad, visuaalsed semiosfäärid ja žestid Artiklis analüüsitakse märgisüsteeme ja semiootilisi mudeleid, millest tuleneb joonistatud kaksiknägude või maskide tähendus Colombia Kolumbuse-eelses kaljukunstis; käsitletakse ka põhipildiga seotud kaht inimfiguuri, kelle nägusid on kujutatud. Vaadeldav kaljukunsti näide avastati Chichamocha kanjoni seinalt Mirador de Bárcenase leiukohast Santanderi departemangus Kirde-Colombias. Seda peetakse guane hõimu looduks. Meie keskseks väiteks on, et see nägu ja selle žestid olid osa märgisüsteemist ehk visuaalsest semiosfäärist, mis levis piki Chicamocha kanjoni kaldaid. Ent sel kuvandil oli ühiseid semiootilisi mudeleid ja visuaalse kommunikatsiooni strateegiaid ühiskondadega, kes elasid uuritavast alast sadade kilomeetrite kaugusel asuvates Kesk-Colombia piirkondades. Toetame seda väidet, sest kasutatud vormid, ruumikasutus piltidel ning hieraatilise iseloomuga žestid ilmuvad nii Huila departemangu petroglüüfide nägudel kui ka mõnedel Cundinamarca piirkonnast pärinevatel Hispaania-eelsetel kuldmaskidel. Näokujutusi kaljukunstis analüüsiti, kasutades visuaalsemiootilist mudelit, mis tugineb Müü rühma (Groupe μ) ettepanekutele ning Félix Thürlemani ja Jean Marie Floch’i analüüsidele. Analüüsi täiendavad Jacques Fontanille’i poolt välja pakutud semiootilise relevantsuse tasandid ning võtmemõistena Juri Lotmani semiosfääri mõiste. Face off – a semiotic technology study of Systems softwareStudies for making deepfakes 489 Sign 49(3/4), 2021, 489–508 Face off – a semiotic technology study of software for making deepfakes Søren Vigild Poulsen1 Abstract. Deepfakes, an algorithm that transposes the face of one person onto the face of another person in images and film, is a digital technology that may fundamentally alter our belief in visual modality and thus presents alarming consequences for an image-centric culture. Not only are these face-translations now so advanced that it is virtually impossible for people to tell that they are fake – this technology is also becoming accessible to laypersons who, with little or no computer skills, can use them for all kinds of purposes, including criminal intentions like revenge porn and identity theft. It is therefore timely and crucial to explore the semiotic potential of deepfakes. This paper presents a semiotic technology perspective, i.e., the study of technology for meaning- making that is an emergent field in social semiotics, to report on findings from an ongoing study of how deepfake software is designed and used as a semiotic resource in erotic and political contexts. The paper advances the argument that the software is able to appropriate all signifiers of the face and their cultural history. Consequently, the semiotic operations of this technology prepare the ground for the problematic perspectives of synthetic facial imagery. On this basis, the paper calls for a critical awareness of taking visual representations of current events at face value and considers how deepfake technology is embedded in unsound sharing practices of visual artefacts that tamper with the rich meaning potential of the face. Keywords: deepfakes; visual manipulation; semiotic software; face studies; digital culture 1. Introduction The appearance of deepfakes, images and videos in which the face of one person is transposed onto the face of another person who says or does things that the former did not say or do, is fundamentally altering our perception of visual representation. Not only are these digital face-transplantations now so advanced 1 Department of Language and Communication, University of Southern Denmark, Denmark; e-mail: vigild@sdu.dk. https://doi.org/10.12697/SSS.2021.49.3-4.12 490 Søren Vigild Poulsen that it is virtually impossible for non-specialists to tell that they are fake, but software for making deepfake images is also becoming accessible to laypersons who, with little or no computer skills, can use them for all kinds of purposes, including criminal intentions like revenge porn and identity theft. According to a report by the Dutch research group Deeptrace2, 96% of these kinds of synthetic media are pornographic and involve women’s faces being “translated” and shared in both closed and public online fora and communities without their knowledge and consent. It is therefore timely and crucial to explore the semiotic potential of deepfakes critically. Research literature on this subject matter is emerging within research fields as diverse as machine learning (e.g. Maras, Alexandrou 2019), pornography studies (e.g. Popova 2020), law studies (e.g. Chesney, Citron 2019a) and media studies (e.g. Lees, Bashford-Roger, Keppel-Palmer 2021). However, each field tends to focus on either technical aspects or use of deepfakes. This paper complements the existing research with a semiotic technology perspective, i.e., the study of technology for making meaning, which is an emergent field in social semiotics (Djonov, Van Leeuwen 2018), to link technical and cultural dimensions of deepfakes. The paper reports on preliminary findings from a study of how deepfake software is designed and used as a semiotic resource in erotic and political contexts, respectively. On this basis, the paper argues that the issue of deepfakes arises from the software’s appropriation of the rich cultural history of the face, and a semiotic dimension needs to be included in the discussions of the manipulations made possible by this technology in order to evaluate the effects of synthetic media. 2. Existing research of deepfakes in machine learning and law Deepfakes have so far primarily been studied in machine learning (in computer science) and law. I will therefore briefly place the present study in the context of deepfake studies in these two research fields. While the term ‘deepfake’ has been dated to an internet forum in 20173 (Cole 2017; 2018), studies of deepfakes go back 2 Ajder, Henry; Patrini, Giorgio; Cavalli, Francesco; Cullen, Laurence 2019. The state of deepfakes: Landscape, threats, and impact. Deeptrace. 24 January. Accessed at https://sensity. ai/reports/ on 10 January 2021. 3 Cole, Samantha 2017. AI-assisted fake porn is here and we’re all fucked. Vice. December 11. Accessed at https://www.vice.com/en_us/article/gydydm/gal-gadot-fake-ai-porn on 10 January 2021; Cole, Samantha 2018. We are truly fucked: Everyone is making AI-generated fake porn now. Vice. Accessed at https://www.vice.com/en_us/article/bjye8a/reddit-fake-pornapp-daisyridley on 10 January 2021. Face off – a semiotic technology study of software for making deepfakes 491 longer within machine learning. The area of machine learning research gradually moved its research interest from extracting human faces from visual data sets (Blanze et al. 2004) to face swapping in still images (Bitoul et al. 2008) and later videos (Dale et al. 2011). Deepfakes got their name from using deep learning technology, a concept that is developed in the field of machine learning. This technology runs a neural network simulation that uses big data sets to create digital information, such as (fake) images. Neural networks are designed to mirror how the human brain operates (at least in a computational understanding of the brain). The neural network can learn to recognize patterns in a data set and to solve classification problems automatically. Furthermore, it can be given a task to do and an objective to fulfill, while monitoring itself and modifying and improving its performance. Applied to face swapping, artificial intelligence (AI) algorithms analyse a source face (e.g. the actor Nicolas Cage) from different angles to learn what it looks like; then, they transfer these features they have learned onto a target face (e.g. that of Donald Trump4), as if these were a mask5 (self-evidently, in this example the result would be that Trump looks like Cage). This process can be repeated multiple times, which will result in ever better facial simulation, depending on the quality of data input and the time frame of the project. Research literature within computer science does not consider face swapping in a semiotic sense, but addresses problems of how to replace a face of one person with that of another, while the faces are shown from different angles, in different lighting conditions and positions, and how these analyses can be automated. The latest developments focus on computational ways to discover fakes (e.g. Farid 2018; Güera, Delp 2018; Maras, Alexandrou 2019). Running parallel to this research, the analysis of deepfakes has been taken up in law in relation to privacy research. Aside from political destabilization and interfering with democratic processes, one can easily imagine the problems that deepfakes may cause in legal cases where visual documents function as empirical basis for lawsuits (Chesney, Citron 2019a). Also, deepfakes challenge traditional forensics methods and analysis (Burke 2006; Leone 2021). Research on deepfakes in law already recognizes 4 For some reasons, Nicholas Cage and Donald Trump are often used in examples of deepfakes both in research literature and on mainstream media. 5 It is important to note that machine learning per se is not a technology for face manipulation. Software like Faceswap applies machine learning methods. Deepfake software utilizes this technology that processes image data, enabling an algorithm to learn how to encode and decode image data in a particular computational operation, resulting in face replacement. The same method may in principle be used to swap any other body parts or objects in motion in videos. 492 Søren Vigild Poulsen the dangers of fake imagery to basic human rights; one issue is the use of deepfakes to discredit politicians and public figures; another issue is how to safeguard the fundamental rights, identities and safety of civilians against false images and videos. Detection software is being developed, and, as Silbey and Hartzog (2020: 961) point out, synthetic media can be used for positive intentions as well: “The potential upside of deepfakes is that they might help muster the political will to address the larger, structural problems made worse by the inability to trust what we see and hear”. I sympathize with this viewpoint, but remain a bit pessimistic about the time perspective of any productive reactions and positive initiatives. A semiotic technology perspective that I will present in the next section attempts to bridge the points of view introduced with respect to both technical and law-related aspects by exploring software for making synthetic media. Furthermore, this approach seeks to make a new contribution by focusing on the fact that it is due to the technological advances that deepfakes are widely circulated, both in terms of speed and reach of circulation. As such, the present study bears kinship with Virilio’s ( 1986[1977]) concept of ‘dromology’ that he used to diagnose logics of speed in society. 3. Methodology I want to apply a semiotic technology approach to study how deepfake software incorporates semiotic resources for making videos and images of face replacements. Semiotic technology research explores how all different types of technologies for meaning-making enable both the production and distribution of multimodal texts and artefacts as well as the performance of semiotic practices (e.g., Zhao et al. 2014; Poulsen 2018). This emergent research area stems from social semiotics (Halliday 1978; Hodge, Kress 1988), and has three overlapping foci: (1) to map semiotic resources in the technology of meaning-making; (2) to study how these are used in semiotic practices – and more broadly, social practices; and (3) to observe the histories, narratives and discourses of technologically mediated resources. Using the semiotic technology approach, I will be analysing the functional features of the user interface as signifiers with meaning potential that are built into deepfake software programs. In relation to the present object of study, research questions to ask about these kinds of semiotic technologies are: what resources for altering face features are available in the software? How does this digital technology facilitate and administrate the making of synthetic faces? How is face swapping technology used in particular social settings? To describe facial expressions and their meaning potentials, I include in the semiotic description of deepfake imagery multimodal interaction studies, Face off – a semiotic technology study of software for making deepfakes 493 microsociology, film and media studies, as well as art and cultural history with special focus on the face. As Van Leeuwen (2005) points out, social semiotics (and thus semiotic technology) is not a matter of “pure” theory, it needs to incorporate other theories to provide an adequate framework for analysis of semiotic phenomena under observation. Thus, following the mapping of resources in the user interface of a deepfake technology, I will draw on work about faces from the above-mentioned research fields that until now have not been included in the discussion of deepfakes. The following section presents a tentative description of the design and technical features of particular deepfake software in semiotic terms, that is, as semiotic resources and their meaning potentials for making deepfakes or synthetic imagery. This is followed by observations from two case studies on the usage of face swapping technology in an erotic context and a political context to illustrate how the resources built into the software could be used, and the problematic results that this usage contributes to. Ideally, I would conduct a semiotic technology study of the design and use of a particular software program, such as Faceswap: I would study how this software program is designed as a semiotic device and the texts that could be made with its specific features. However, it was not possible to determine the origin of the software that was used for making the deepfake videos chosen as the objects of this semiotic analysis. Thus, the case studies only serve to illustrate general points about the problematic nature of such technology. 4. Analysis of the design of software for making deepfakes The following analysis includes a description of the design of deepfake software as semiotic technology. The subsequent sections will focus on: (1) the process and steps of making a deepfake video; (2) the context of selecting deepfake software; (3) the design of Faceswap’s user interface; (4) how the face replacement process is structured in Faceswap; and (5) the semiotic operations of the software. 4.1. The process and steps of making a deepfake video If one breaks face replacement down into its basic elements, a deepfake production consists of four steps, regardless of the particular software one could use in the process. As Sample (2020)6 explains: 6 Sample, Ian 2020. What are deepfakes – and how can you spot them? The Guardian 13 January 2020, was accessed at https://www.theguardian.com/technology/2020/jan/13/whatare-deepfakes-and-how-can-you-spot-them on 10 January 2021. 494 Søren Vigild Poulsen First, you run thousands of face shots of the two people through an AI algorithm called an encoder. The encoder finds and learns similarities between the two faces, and reduces them to their shared common features, compressing the images in the process. A second AI algorithm called a decoder is then taught to recover the faces from the compressed images. Because the faces are different, you train one decoder to recover the first person’s face, and another decoder to recover the second person’s face. To perform the face swap, you simply feed encoded images into the “wrong” decoder. For example, a compressed image of person A’s face is fed into the decoder trained on person B. The decoder then reconstructs the face of person B with the expressions and orientation of face A. For a convincing video, this has to be done on every frame. From a semiotic technology perspective, what is interesting when analysing deepfake software is how the digital technology structures and facilitates the deepfake production process to its users who are not necessarily technically skilled. I will therefore be discussing the role and function of the software that mediates between computational operations and visual meaning-making. 4.2. Context of selecting deepfake software Previously, making deepfakes required advanced programming, but free or opensource software that can perform the computational operations in a few simple steps is now widely available to the general public. The following observations were made during the research phase for software that will enable the creation of deepfakes: (1) A Google search presents several options for downloading software for desk PCs as well as recommendations indicating accessibility. There is also easy access to apps for mobile phones that allows you to make short fake videos, images and audio. (2) Serious tech media evaluate the best deepfake software on the market. (3) Many of the software programs are marketed for laypersons: users are not required to have computer skills or special knowledge about deep learning or computer vision. (4) The online Gethub repositories hold several communities for deepfakes. (5) Some software programs (e.g. Machin Tube) come with pre-loaded templates of faces to make deepfakes of, e.g. politicians, singers and celebrities (like Donald Trump). (6) While this study focuses on the visual features, I also wish to mention free technology for deepfake audios, e.g. Lyrebird AI, which has the capacity to form sentences just upon hearing a few spoken words, thus creating a digital voice. This software also enables overdubbing that “allows you to replace Face off – a semiotic technology study of software for making deepfakes 495 recorded words and phrases with synthesized speech that is tonally blended with the surrounding audio”7. So, while face replacement is normally what is associated with deepfakes, the technology also includes audio, in which case only the voice of someone else is dubbed. These deepfakes are called ‘voice cloning’ and an example of this is a video from 2019 in which the voice of Barack Obama was dubbed by actor Jordan Peale. In the following section, I will analyse the user interface of Faceswap. Faceswap8 (2019-) is the leading open source face replacement technology developed by Matt Tora, Bryan Lyon, and Kyle Vrooman. It is coded in the programming language Python. The software is powered by Artificial Intelligence using so-called Convolutional Neural Networks, and it has an active forum of volunteers and users. I selected Faceswap over e.g. DeepFakeLab, which is more technical, since the latter was developed for computer science students and uses command lines. Faceswap features a graphic user interface for laypersons, and its tutorials describe the process in a great detail, avoiding technical terms. 4.3. The user interface of Faceswap The graphic user interface of Faceswap presents four headings (Extract, Train, Convert, Tools), which suggests a workflow in the making of deepfake, starting from left to right. In contrast to, e.g. DeepfakeLap which offers the users multiple command line operations and sub functions, the process of making deepfakes is simplified (three steps) and streamlined (unless the user wants to include specific effects in the video, they will go with the default settings under each heading). Regardless of the heading, the user interface is divided into two windows. In the left-hand frame window different features relevant to the chosen heading are provided, e.g. under Extract, file trajectories can be selected of the videos that are made into fake images or videos. In the right-hand frame, information about and results of the current stage of the process are displayed, e.g. the number of frames extracted from the videos, or a preview of the results of the process. 4.4. The process of making deepfakes in Faceswap As mentioned above, Faceswap structures the making of deepfakes in a few simple steps. That said, the software offers the user different editing options in Faceswap user interface, depending on the purpose of the training and type of data. Faceswap 7 8 Descript website. Accessed at https://www.descript.com/lyrebird-ai on 16 December 2019. The official website of Facewap is https://faceswap.dev. 496 Søren Vigild Poulsen mirrors the processes that were presented in Section 4.1. In the software tutorial9, these steps are described as follows: Gather data: Find videos/images of the people whose faces you want to swap Extract: Get all faces from data, Sort, Remove incorrect faces Training: Use the AI to learn how to swap faces (i.e., encoding and decoding) Convert: Apply the AI on data (either new or already trained) to swap faces in the videos This organization of headings and the tools under each heading would, in semiotic terms, be the overall ‘semiotic regime’ (Van Leeuwen 2005), i.e. a social regulation of the use of semiotic resources and meaning-making processes. The design of the user interface enforces a particular way of producing face replacement videos. Already at this stage of the analysis, the observation can be made that, on the one hand, the simplified workflow enables laypersons to make deepfakes, and, on the other hand, it has consequences for the users’ understanding of what the software does. One such effect is blackboxing, which hides the functions performed by the technology. This is to some degree the case with Faceswap, compared to DeepFaceLap that lets the user follow the computational processes in more detail. Still, the blackboxing process does not run as straightforwardly as e.g. Reface, an app that allows the user to take a selfie and embed it in a five-to-ten-second Hollywood movie or music video. With this app, the face-swapping process is totally hidden, and the user is only presented with a simple step-by-step process: after taking a selfie, the image is blurred and the mobile screen displays stars tingling while the underlying AI analyses the visual information, animating the data processing as a magical process. Next, the user simply chooses between the selected movies in which (s)he wants his/her face to be inserted. This video is then generated by showing a timeline, and the final results to be saved or shared. Blackboxing seems to be a side effect of making advanced software accessible to the general public. After the deepfake video has been made, the Faceswap software offers ways to save and share the content on e.g. social media networks. Thus, the software prompts the user to distribute the deepfake imagery and participate in the circulation of the synthetic media content. This is the basic setup. 4.5. Technical features as semiotic resources A semiotic technology study of Faceswap would describe semiotic resources of the software, i.e. the technicaly mediated signifiers and their meaning potentials. However, it is difficult to describe the deepfake software and videos separately, 9 Facewap 2019 was accessed at https://forum.faceswap.dev/index.php on 10 January 2021. Face off – a semiotic technology study of software for making deepfakes 497 as the meaning potential of the user interface is dependent on the videos that are being made into deepfakes. Therefore, a full analysis would break down the semiotic features of a deepfake video into different levels: the faces (of two persons); the environment that the faces are placed in; and video editing software. Each of these layers contains several semiotic systems that contribute to the overall meaning-making of a deepfake video. In the following I will concentrate on the editing software, but include aspects from the other levels. If one follows the workflow of Faceswap, each step utilizes several semiotic resources. 4.5.1. Gathering data (representing faces) A prerequisite of face swapping is the videos or images that function as input to the computational process. The function of uploading face images incorporates a semiotic resource that is essential to all visual communication, namely ‘existential process’, i.e. the act of representing something as ‘being there’ (Boeriis 2009). Thus, simply to be able to represent images of a source and a target face becomes the grounds for making any deepfake, and the subsequent editing processes depend on the usage of this resource. That the point of departure is an existing facial representation may seem self-evident, but it is a fundamental feature for this very reason. Highlighting the existential resource for representation also draws attention to another feature, namely that the input faces and the instantiated resources that they manifest are included in the meaning-making process that the semiotic software creates. The videos are in themselves multimodal artefacts that draw on multiple semiotic systems, and since each video typically contains extensive footage of face angles, expressions and movements, they display a complex ensemble of instantiated meanings. The resources of the face images exist independently of the deepfake technology. Nonetheless, the deepfake technology depends on the textual meaning of the faces, for such software functions as a parasite on the meaning made in the videos; the software re-semiotizes (Iedema 2001) or transduces (Kress 2010) the meaning potentials of the two videos by emerging two faces into one video. 4.5.2. Extraction (analytical process and reconfiguration of facial resources) The next step is to extract face data from the still images. As Faceswap10 (2019) describes in their tutorial on face replacement: At the highest level, extraction consists of three phases: detection, alignment and mask generation: 10 Facewap 2019. [Guide] Extraction – a workflow. Accessed at https://forum.faceswap.dev/ viewtopic.php?t=27z on 10 January 2021. 498 Søren Vigild Poulsen Detection – The process of finding faces within a frame. The detector scans an image and selects areas of the image that it thinks are faces. Alignment – Finding the “landmarks” […] within a face and orienting the faces consistently. […] Mask Generation – Identify the parts of an aligned face that contains face and block out those areas that contain background/obstructions. […] Extracting serves 2 main purposes: Training: To generate a set of faces for training. These faces will also contain the alignment information and masks which are required for training the model. […] Converting: To generate an alignments file and mask for converting your final frames. The alignment file contains information about where each face is in each frame so that the conversion process knows where to swap faces for any given frame. The purpose of the extraction process is to reduce the face information in an image to its core elements for an algorithm to reproduce this face with the least amount of data. The reduction process may be similar to ‘abstraction’ in Peirce’s diagrammatical reasoning (see CP; Peirce 1976). The mentioned ‘landmarks of the face’ refer to key data points of a face that indicates facial position and expression in each frame (see Fig. 1). These operations are the core of the semiotic transformations in the initial face placement process. Essentially, two semiotic ideational resources are utilized in the process: first, categorization and segmentation of facial elements incorporate an ‘analytical process’, i.e. a whole-parts relation (Kress, Van Leeuwen 2006); second, I understand the alignment process that tracks facial movement based on a number of landmarks of the face as a resource for reconfiguration of semiotic multiple different semiotic systems. Historically, studies of facial resources and their meaning potentials have divided features of a face into different semiotic systems or modes, e.g. the gaze (Argyle, Dean 1965; Argyle, Cook 1976; Goffman 1963; Belting 2013; Exline, Fehr 1982), head movement (Altorfer et al. 1992, 1998; Jossen et al. 2000), gesture (Kendon 1967; 1978; Schegloff, Sacks 1973; Goodwin 1986; Norris 2004), speech (Halliday, Hasan 1976). These semiotic systems are heuristic analytical categories, as facial resources co-create meaning multimodally. I would claim that this alignment is the essential part of the editing software, since it lays the ground for training algorithms to make deepfakes. At the same time, this exact operation creates the issues of synthetic media. Semiotically speaking, alignment as a resource of reconfiguration includes two elements: first, by aligning a face’s diverse signifiers, the software appropriates multiple different semiotic systems that people use to make facial meanings. The software reconfigures Face off – a semiotic technology study of software for making deepfakes 499 semiotic features of the face that carries an extremely rich cultural history, i.e., the meaning potential that people throughout history have created with each facial semiotic system (Boehm et al. 2015). Second, by incorporating all of these different semiotic systems into landmarks that make up a map of the face, the software offers previously unseen ways to manipulate the totality of these facial semiotic systems and their meaning potentials all at once. Figure 1. An example of landmarking in the process of facial alignment. Author’s own production inspired by a figure in Wang et al.11 4.5.3. Training (semiotic principles for articulation and interpretation) I consider the last two steps in the Faceswap workflow as having to do with overall principles for meaning-making. The Training function concerns the establishment of a strategy for semiosis, so the source face is interpreted as the target face. This strategy is drawn upon in the making of a face swap, as far as it can be automated. As such, it becomes a resource for interpretation and re-articulation (Kress, Van Leeuwen 2001) of a semiotic production. 4.5.4. Conversion (resource integration/blending) Faceswap differs from face alterations or improvements in face apps like TikTok, Snapchat or Instagram which apply filters to face images because the face attributes from the target face in a deepfake are also transferred to the source face. Therefore, the meaning-making principle is not metaphorically from a source (face) to a target (face) (Lakoff, Johnson 1980), the target face does not get semanticized by the features and qualities of the source. In social semiotic terms, this process constitutes 11 Wang, Junjue; Amos, Brandon; Das, Anupam; Pillai, Padmanabhan; Sadeh, Norman; Satyanarayanan, Mahadev 2017. A scalable and privacy-aware IoT service for live video analytics. Proceedings of the 8th ACM on Multimedia Systems Conference, 38–49; accessed on 20 January 2021. 500 Søren Vigild Poulsen an ‘integration resource’ (Bauldry, Thibault 2006), i.e. a principle for how semiotic resources may be combined in multimodal ensembles. One can describe this as a blend (Fauconnier, Turner 2002) that merges different semantic domains into an emergent meaning construction. In this analytical framework, a deepfake video would be conceptualized as a meaning construction that blends face features of a source with face expressions of a target. As a result, the deepfake video can show a person doing things that are fictional (i.e. not authentic), yet trustworthy. 5. Analysis of two contexts of deepfake usage: pornography and politics In this section, I explore how the semiotic deepfake technology is used in diverse semiotic practices where fake faces may serve harmful purposes and help maintain unhealthy social practices. I consider two types of contexts in which deepfakes are being used. Each context analysis reports on preliminary observations. 5.1. Deepfakes in pornography A simple Google search for deepfakes reveals that deepfakes appear as a category on major porn sites (e.g. Pornhub), as well as on sites dedicated to this kind of pornographic content (e.g. MrDeepFake). Officially, sites like Pornhub have banned deepfakes from their site, but videos can still be found if searched for on these sites12. I did a search on ‘xvideo.com’ and found multiple videos of Hollywood actresses, female singers and politicians. In the following, I present general observations from an analysis of a deepfake video of a famous American singer. Given the explicit and transgressive nature of the imagery, I will refrain from describing details in the video and only focus on general observations13. I observed that the face of the singer is shown in long camera shots, and that the face is displayed at the centre of the footage, while the ‘body’, i.e. the body of the porn actress, performs sexual acts, so that the viewer can clearly see and dwell on the fake face features. As such, the face presents itself in its completeness, which Balázs (1970[1923]), in his studies of silent films, has described as a key element 12 Ajder, Henry; Patrini, Giorgio; Cavalli, Francesco; Cullen, Laurence 2019. The state of deepfakes: Landscape, threats, and impact. Deeptrace. 24 January. Accessed at https://sensity. ai/reports/ on 10 January 2021. 13 I am aware of the problematic nature of analysing erotic deepfakes, since I as researcher engage in the unsound social practice that I criticize, and as such, I take part in upholding the market for these products. I have chosen not to disclose the name of the woman in the video analysed. I am also aware that the viewing of such material potentially poses a criminal act. Face off – a semiotic technology study of software for making deepfakes 501 of the close-up. The whole video is filmed from a subjective point of view, where the woman looks directly and up close at the camera. The use of the gaze is an essential resource that intensifies the direct visual addressing of the viewer (Kress, Van Leeuwen 2006). The close-up, as Doane (2003) argues, “[…] embodies the pure fact of presentation, […] of showing – a “here it is”” (quoted in Jerslev 2017: 91.) that creates a sense of direct access to reality. This suggests the fascination of the male gaze with the face as an object; I also suspect that the depicted face and gaze of the singer bespeaks of a video producer who takes pride in the deepfake production. The face shot in these videos is “undisturbed”, as it were. In this regard, the videos that swap the faces of a woman celebrity onto the body of a porn actress seem to deviate from other pornographic videos where the face of the woman is “manhandled” as it is not uncommon for the (female) face in porn to be slapped, stepped on, penetrated, or covered by semen. Perhaps the facial display in the deepfake porn is also a question of limitations to the deepfake technology, which is not able to handle changes in face shots that include other elements. There are small glitches in the videos where it becomes visible that the face of the actress has been manipulated: the pixels that mark up the face area become visible and, as a result, presumably break the illusion of the depicted fantasy. In this context, I suggest that the use of deepfake software for making fakes videos of celebrities, and public figures advances the already existing genres of ‘fake nudes’ and ‘celeb nudes/fakes’ that include videos of lookalikes. Also, these videos are fed to a market for intimate videos produced without the consent of the depicted women whose faces are placed in the context of nude scenes. Only in a few cases, e.g. nude footages of celebrities, has the “leaking” of sex tapes been used in personal branding as a deliberate act of empowerment. I would claim that nude deepfakes primarily relate to hate porn and revenge porn that have devastating consequences for the victims without whose consent pictures or videos of them are exposed to the public at large. As pointed out by Popova (2020), the use of deepfake nudes may at first glance appear as a fantasy of having intercourse with a celebrity, yet the phenomenon is not like fan fiction, but is part of a transgressive act, and, more broadly, of a misogynistic culture centring on fantasies of violating acts by a man on a particular woman’s face and body. Furthemore, while there may be a (legal) difference between a person making such manipulated imagery and viewers who watch the videos, but who themselves would not create this kind of content, the viewers of these videos, in their engagement with deepfake porn, nevertheless uphold this social practice. By viewing, commenting and further sharing of fake images and videos, they legitimize the existence of such deepfakes. As such, the production and distribution of deepfake porn supports a sharing economy using women’s sexuality as a commodity. 502 Søren Vigild Poulsen 5.2. Deepfakes in politics Turning to politics, and looking beyond examples of political satire, one can easily find several examples of deepfakes of political figures that supposedly say or do something that would discredit their reputation, and this is a global issue. A recent instance was a deepfake video of the Indian politician Manoj Tiwari that was shared between groups on WhatsApp during the 2020 Delhi elections. In this regard, I made the preliminary observation that Tiwari was shown talking to voters in Haryanvi and English, but in the real video he speaks in Hindi. However, unlike the erotic deepfake that centres on a fabricated sexual act, this political deepfake video was not manipulated beyond a synchronization of the voice with the depicted person’s limb movements. From watching this video, and others, it seems that it is especially linguistic resources that can be cloned with a deepfake voice technology, so they are used to make fake statements. The false account was accompanied by the presentation of a particular politician’s face. In this context, the identity of a political figure is closely associated with his or her face, as it also appears on political posters. Like deepfake porn, the video of Tiwari utilizes the fundamental ideational resource of representing a person’s face. By combining linguistic and gaze resources, politicians like Tiwari may be portrayed as making outrageous or discriminating utterances that would damage the support from their voters if these were true. As such, the purpose of political deepfake videos and audios is an attempt to destabilize political systems and to shatter people’s trust in democratic processes. The very act of making a person say something that he or she has never uttered demonstrates that the use of deepfake technology in manufacturing false images and videos in a political context could cause double harm by contributing to the increasing disgust with politicians as well as disbelief in trusted news sources that play a crucial role in democratic processes (Paterson, Hanley 2020). That false political videos pose a serious threat to democracy was indicated by a CIA report on worldwide threat assessment published prior to the recent US presidential election. The report states that “adversaries and strategic competitors probably will attempt to use deepfakes or similar machine-learning technologies to create convincing – but false – image, audio, and video files to augment influence campaigns directed against the United States and our allies and partners” (Coates 2019: 7).14 It is not only political institutions that are affected by such problematic use of deepfakes. The use of deepfake technology in manufacturing false images 14 Coats, Daniel R. 2019, Worldwide threat assessment of the US intelligence community https://www.odni.gov/index.php/newsroom/congressional-testimonies/item/1947-statementfor-the-record-worldwide-threat-assessment-of-the-us-intelligence-community; accessed on 20 January 2021. Quotation refers to page 7. Face off – a semiotic technology study of software for making deepfakes 503 and videos in a political context. This affects not only political institutions: deepfakes can also damage the reputation of businesses and organizations, which may have a negative impact on stock prices and bring along long-term consequences when customers and shareholders no longer trust enterprises and markets. The fact that deepfakes have become a phenomenon to be aware of threatens to undermine the perceived objectivity of videos featuring politicians and public figures. It should be a trademark of a healthy political climate to dispute and critically discuss decision-makers’ solutions to social problems, and it is also paramount to be able to hold politicians accountable for their actions and claims. In this context, deepfakes align with fake news and distribution of misinformation (Chesney, Citron 2019b). 6. Concluding remarks This article is a report on a preliminary study of the design and use of semiotic technology for making deepfakes. On this basis, I have argued that a semiotic approach is needed to describe why the design and use of face replacement technology may potentially cause disturbance, since the manipulation of the face tampers with semiotic resources that carry an immensely rich meaning potential. By analysing the leading face swapping software, Faceswap, as a semiotic technology, I have tried to demonstrate how the software appropriates many semiotic resources of a face on video and how it is able to manipulate these. Still, much more research on the software and its usage in different contexts needs to be carried out in this area. When asked about their responsibility for spreading deepfakes at a tech conference in 201915, two of the three founders of Faceswap answered that it is not their duty to prevent this from happening. If they do not develop the technology, someone else will come along and do this. Furthermore, they argue that a better countermovement to the rise of deepfakes is to familiarize the general public with this kind of software; likewise, alongside the development of deepfake software, scientists and companies are working on software for detecting deepfakes (e.g. Nguyen et al. 2019). While this viewpoint may be valid, the article would claim that a joint effort by tech companies, political leaders and decision-makers, a more critical press and research is a better solution to some of the problems discussed in this article. Also, it is important to support the digital literacy of the general public, so that they do not uncritically believe in images and videos circulated on 15 Tora, Matt; Lyon, Bryan; Vrooman, Kyle 2019 Faceswap 2019: Faceswap: Developing a platform for practical Deepfakes. Recorded presentation held at Devoxx conference, 7 November Belgium was accessed at https://devoxx.be/talk/?id=103201on 20 January 2021. 504 Søren Vigild Poulsen the internet, especially on social media. In addition to these initiatives, semiotic technology descriptions of deepfake software can help us understand how synthetic media is made and how society is affected when people’s faces and the values and meanings we assign to them are fabricated. The face is the most valuable part of who we are and who we believe others to be. Acknowledgements: I would like to thank the editors, the anonymous reviewers, members of Centre for Multimodal Communication (Morten Boeriis and Theo van Leeuwen), as well as Gunhild Kvåle for suggestions on how to improve the manuscript. I alone am responsible for all claims presented in this article. References Argyle, Michael; Cook, Mark 1976. Gaze and Mutual Gaze. Cambridge: Cambridge University Press. Argyle, Michael; Dean, Janet 1965. Eye-contact, distance and affiliation. Sociometry 28(3): 289–304. https://doi.org/10.2307/2786027 Altorfer, Andreas; Goldstein, Michael J.; Miklowitz, David J.; Nuechterlein, Keith H. 1992. Stress-indicative patterns of non-verbal behaviour: Their role in family interaction. British Journal of Psychiatry 161(18): 103–113. https://doi.org/10.1192/ S0007125000297122 Altorfer, Andreas; Käsermann, Marie-Louise; Hirsbrunner, Hanspeter 1998. Arousal and communication: The relationship between nonverbal, behavioral, and physiological indices of the stress response. Journal of Psychophysiology 12: 40–59. Balázs, Béla 1970[1923]. The close-up/The face of man. In: Balázs, Bela (ed.), Theory of the Film: Character and Growth of a New Art. New York: Dover, 52–89. Baldry, Anthony; Thibault, Paul 2006. Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London/Oakville: Equinox. Belting, Hans 2013. Faces: Eine Geschichte des Gesichts. Munich: C. H. Beck. Bitouk, Dmitri; Kumar, Neerah; Dhillon, Samreen; Belhumeur, Peter; Nayar, Shree K. 2008. Face swapping: Automatically replacing faces in photographs. ACM Transactions on Graphics 27(3): 1–8. https://doi.org/10.1145/1360612.1360638 Blanz, Volker; Scherbaum, Kristina; Vetter, Thomas; Seidel, Hans‐Peter 2004. Exchanging faces in images. Computer Graphics Forum 23: 669–676. https://doi.org/10.1111/j.14678659.2004.00799.x Boehm, Gottfried; Budelacci, Orlando; Di Monte, Maria Giuseppina; Renner, Michael (eds.) 2015. Gesicht und Identität/Face and Identity. Paderborn: Wilhelm Fink Verlag. https://doi.org/10.30965/9783846758151 Boeriis, Morten 2009. Multimodal Socialsemiotik og Levende Billeder. [Multimodal social semiotics and moving images. Unpublished PhD thesis.] Odense: University of Southern Denmark. Burke, Peter 2006. Eyewitnessing the Uses of Images as Historical Evidence. London: Reaktion Books. Face off – a semiotic technology study of software for making deepfakes 505 Chesney, Robert; Citron, Danielle K. 2019a. Deepfakes: A looming challenge for privacy, democracy, and national security. California Law Review 107(6): 1753–1820. Chesney, Robert; Citron, Danielle K. 2019b. Deepfakes and the new disinformation war: The coming age of post-truth geopolitics. Foreign Affairs 98(1): 147–155. CP = Peirce, Charles S. 1931–1958. Collected Papers of Charles Sanders Peirce. Cambridge: Harvard University Press. (Vols. 1–6, Hartshorne, Charles; Weiss, Paul, eds., 1931–1935; vols. 7–8, Burks, Arthur W., ed., 1958.) [In-text references are to CP, followed by volume and paragraph numbers.] Dale, Kevin; Sunkavalli, Kalyan; Johnson, Micah K.; Vlasic, Daniel; Matusik, Wojceich; Pfister, Hanspeter 2011. Video face replacement. ACM Transactions on Graphics (SIGGRAPH), 30. https://doi.org/10.1145/2070781.2024164 Djonov, Emilia; Van Leeuwen, Theo 2018. The power of semiotic software: A critical multimodal perspective. In: Flowerdew, John; Richardson, John E. (eds.), The Routledge Handbook of Critical Discourse Studies. New York: Taylor & Francis, 566–581. https:// doi.org/10.4324/9781315739342-39 Doane, Mary Ann 2003. The close-up: Scale and detail in the cinema. Differences: A Journal of Feminist Cultural Studies 14(3): 89–111. https://doi.org/10.1215/10407391-14-3-89 Exline, Ralph. V.; Fehr, Barbara J. 1982. The assessment of gaze and mutual gaze. In: Scherer, Klaus R.; Ekman, Paul (eds.), Handbook of Methods in Nonverbal Behavior Research. Cambridge: Cambridge University Press, 91–135. Farid, Hany 2018. Digital forensics in a post-truth age. Forensic Science International 289: 268–269. Fauconnier, Gilles; Turner, Mark 2002. The Way We Think: Conceptual Blending and the Mind’s Hidden Complexities. New York: Basic Books. Goffman, Erving 1963. Behavior in Public Places. New York: Free Press. Goodwin, Chuck 1986. Gestures as a resource for the organization of mutual interaction. Semiotica 62(1/2): 29–49. https://doi.org/10.1515/semi.1986.62.1-2.29 Güera, David; Delp, Edward J. 2018. Deepfake video detection using recurrent neural networks. 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 1–6. https://doi.org/10.1109/AVSS.2018.8639163 Halliday, Michael A. K. 1978. Language as Social Semiotic. London: Edward Arnold. Halliday, Michael A. K; Hasan, Ruqaiya 1976. Cohesion in English. London: Longman. Hodge, Robert; Kress, Gunther 1988. Social Semiotics. Cambridge: Polity Press. Iedema, Rick 2001. Resemiotization. Semiotica 137(1/4): 23–39. https://doi.org/10.1515/ semi.2001.106 Jerslev, Anne 2017. SKAMs ‘lige her’ og ‘lige nu’: Om SKAM og nærvær. [‘Right here’ and ‘right now’ in SKAM: On SKAM and presence.] Nordisk Tidsskrift for Informationsvidenskab Og Kulturformidling 6(2): 75–81. https://doi.org/10.7146/ntik.v6i2.99080 Jossen, Stefan; Käsermann, Marie-Louise; Altorfer, Andreas; Foppa, Klaus; Zimmermann, Henrich; Hirsbrunner, Hans-Peter 2000. The study of emotional processes in communication: Peripheral blood flow as an indicator of emotionalization. Behavior Research Methods, Instruments, & Computers 32: 47–55. https://doi.org/10.3758/BF03200787 Kendon, Adam 1967. Some functions of gaze-direction in social interction. Acta Psychologica 26: 22–63. https://doi.org/10.1016/0001-6918(67)90005-4 506 Søren Vigild Poulsen Kendon, Adam 1978. Looking in conversation and the regulation of turns at talk: A comment on the papers of G. Beattie and D. R. Rutter et al. British Journal of Social and Clinical Psychology 17: 23–24. https://doi.org/10.1111/j.2044-8260.1978.tb00891.x Korshunova, Iryna; Shi, Wenzhe; Dambre, Joni.; Theis, Lucas 2017. Fast face-swap using convolutional neural networks. 2017 IEEE International Conference on Computer Vision (ICCV). Venice, 3697–3705. https://doi.org/10.1109/ICCV.2017.397 Kress, Gunther 2010. Multimodality – A Social Semiotic Approach to Contemporary Communication. London, New York: Routledge. Kress, Gunther; Van Leeuwen, Theo 2006. Reading Images – The Grammar of Visual Design. (2nd ed.). London: Routledge. https://doi.org/10.4324/9780203619728 Lakoff, George; Johnson, Mark 1980. Metaphors We Live by. Chicago, London: University of Chicago Press. Lees, Dominic; Bashford-Roger, Tom; Keppel-Palmer, Marcus 2021. The digital resurrection of Margaret Thatcher: Creative, technological and legal dilemmas in the use of deepfakes in screen drama. Convergence: The International Journal of Research into New Media Technologies 27(4): 954–973. https://doi.org/10.1177/13548565211030452 Leone, Massimo 2021. From fingers to faces: Visual semiotics and digital forensics. Int. Journal of the Semiotics of Law 34: 579–599. https://doi.org/10.1007/s11196-020-09766-x Maras, Marie-Helen; Alexandrou, Alex 2019. Determining authenticity of video evidence in the age of artificial intelligence and in the wake of deepfake videos. The International Journal of Evidence 23(3): 255–262. https://doi.org/10.1177/1365712718807226 Nguyen, Thanh. T.; Nguyen, Cuong M.; Nguyen, Dung T.; Nguyen, Duc T.; Nahavandi, Saeid 2019. Deep learning for deepfakes creation and detection: A survey. arXiv preprint arXiv: 1909.11573: 1–16. Norris, Sigrid 2004. Analyzing Multimodal Interaction: A Methodological Framework. New York, London: Routledge. https://doi.org/10.4324/9780203379493 Paterson, Thomas; Hanley, Lauren 2020. Political warfare in the digital age: Cyber subversion, information operations and ‘deepfakes’. Australian Journal of International Affairs 74(4): 439–454. https://doi.org/10.1080/10357718.2020.1734772 Peirce, Charles S. 1976. The New Elements of Mathematics by C. S. Peirce (Eisele, Carolyn, ed.). The Hague: Walter De Gruyter. Popova, Milena 2020. Reading out of context: Pornographic deepfakes, celebrity and intimacy. Porn Studies 7(4): 367–381. https://doi.org/10.1080/23268743.2019.1675090 Poulsen, Søren Vigild 2018. Becoming a semiotic technology – a historical study of Instagram’s tools for making and sharing photos and videos. Internet Histories 2(1/2): 121– 139. https://doi.org/10.1080/24701475.2018.1459350 Schegloff, Emanuel; Sacks, Harvey 1973. Opening up closings. Semiotica 8: 289–327. https://doi.org/10.1515/semi.1973.8.4.289 Silbey, Jessica; Hartzog, Woodrow 2019 The upside of deepfakes. Maryland Law Review 78(4): 960–966. Stöckl, Hartmut; Caple, Helen; Pflaeging, Jana (eds.) 2019. Shifts towards ImageCentricity in Contemporary Multimodal Practices. London: Routledge. https://doi. org/10.4324/9780429487965 Van Leeuwen, Theo 2005. Introducing Social Semiotics. London: Routledge. https://doi. org/10.4324/9780203647028 Face off – a semiotic technology study of software for making deepfakes 507 Virilio, Paul 1986[1977] Speed and Politics: An Essay on Dromology. New York: Semiotext(e). Zhao, Sumin; Djonov, Emilia; Van Leeuwen, Theo 2014. Semiotic technology and practice: A multimodal social semiotic approach to PowerPoint. Text & Talk 34(3): 349–375. https://doi.org/10.1515/text-2014-0005 Face off – семиотическое технологическое исследование программного обеспечения для создания подделок Deepfakes, алгоритм, который накладывает лицо одного человека на лицо другого в изображениях и видео – цифровая технология, которая может коренным образом изменить нашу веру в визуальную модальность и представляет серьезную опасность для нашей культуры, ориентированной на изображения. Эти технологии не просто настолько продвинуты, что практически не позволяют усомниться в их подлинности, но также всё более доступы непрофессионалам, которые могут использовать их в самых разных целях, включая преступные намерения вроде порнографии в целях мести и кражи личных данных. В связи с этим исследование семиотического потенциала таких подделок является своевременным и крайне необходимым. В статье представлена семиотическая технологическая перспектива – новое направление социосемиотики, анализирующее, каким образом программное обеспечение deepfake разрабатывается и используется в качестве семиотического ресурса в эротических и политических контекстах. Выдвигается предположение, что такое программное обеспечение способно подчинить все знаки лица и их культурную историю. Следовательно, семиотические операции этой технологии подготавливают почву для синтетического изображения лица. Face off – süvavõltsingute tegemiseks kasutatava tarkvara semiootiline tehnoloogiauuring Süvavõltsing, algoritm, mis asetab kujutistel ja filmides ühe isiku näole teise isiku näo, on digitehnoloogia, mis võib fundamentaalselt muuta meie usku visuaalsusmodaalsusesse ja millega kuvandikeskset kultuuri silmas pidades seega kaasnevad ohtlikud tagajärjed. Sellised näotõlked pole mitte ainult niivõrd edasijõudnud tasemel, et inimestel on praktiliselt võimatu otsustada, kas tegemist on võltsinguga, vaid see tehnoloogia on muutumas kättesaadavaks mitteerialainimestele, kes vähestest või puuduvatest arvutioskustest hoolimata võivad neid kasutada igasugustel eesmärkidel, sealhulgas kuritegelikel nagu seda on kättemaksuporno ja identiteedivargus. Seega on süvavõltsingute semiootilise potentsiaali vaatlemine õigeaegne ning oluline. Artiklis pakutakse välja semiootilise tehnoloogia vaatenurk, s.t sotsiosemiootikas valdkonnana esile kerkiva tähendusloometehnoloogia vaatlus, esitamaks käimasoleva uuringu tulemusi selle kohta, kuidas on disainitud süvavõltsingutarkvara ning kuidas seda erootilistes ja poliitilistes kontekstides semiootilise ressursina kasutatakse. Artikkel väidab, et selline tarkvara on suuteline anastama kõik tähistajad ning nende kultuuriloo. Järelikult valmistavad selle tehnoloogia semiootilised operatsioonid ette pinnast sünteetiliste näokujutuste problemaatilistele perspektiividele. 508 Søren Vigild Poulsen Sellest lähtuvalt kutsutakse artiklis üles kriitilisusele päevasündmuste visuaalsete representatsioonide automaatselt tõe pähe võtmise suhtes ning tuletatakse meelde, et tehnoloogia on kaasatud visuaalsete artefaktide ebasoovitavatesse jagamispraktikatesse, mis kahjustavad näo rikkalikku tähenduspotentsiaali. Augmented facets: A semiotics analysisSign of augmented reality49(3/4), facial eff ects509–526 509 Systems Studies 2021, Augmented facets: A semiotics analysis of augmented reality facial effects Federico Biggio1 Abstract: Augmented reality facial effects represent a new trend in social media communication based on ‘short forms’. The article proposes a tripartite analysis: a semiotic analysis of digital facial effects used to empower the natural users’ faces; a deconstructionist analysis of Spark by Meta, one of the major software applications to create such effects and, finally, a critical reflection on the practices prescribed by Spark and the stereotypical aesthetics of augmented selfies. The conclusion states that such forms of augmented reality effects must be conceived not as oriented to the cognitive improvement of users’ performance but rather as forms of users’ empowerment and self-awareness. Keywords: face filters; facial effects; Instagram; Spark; augmented reality 1. Introduction. A new trend in social media communication The article aims to account for an emerging form of digital creativity detectable in social media contexts and especially on Instagram. The article’s specific object of inquiry is the digital facial effect (also called ‘filter’) used by social media users to enhance in different ways their natural faces’ images in selfie production and sharing. The semiotic analysis will focus on a specific typology of textual forms, which can be classified in the category of ‘short forms’ (Pezzini 2002; Montani 2020). Brevity should not be understood only as a material requirement (for example, the 13 seconds of Instagram Stories), but as a predominant textual form in social media contexts that testifies to a “compositional mode that loads of meaning in the first place the structuring of the text” (Montani 2020: 8). 1 Department of Excellence of Philosophy and Educational Sciences, University of Turin, Italy; Department of Hypermedia, University of Paris VIII Vincennes-Saint Denis; France; e-mail address: federico.biggio@unito.it. https://doi.org/10.12697/SSS.2021.49.3-4.13 510 Federico Biggio Among these ‘short forms’ we can insert licitly the so-called “selfie”, which Del Marco (2017: 21) defines as a “reflexive shot, a photograph that a person makes to himself/herselves – in some cases with others and among others”, by emphasizing both the object-reference of the visual representation and the social practice connected to its production: “[the selfie is] realized with a mobile device like a smartphone or a tablet or through a webcam, sometimes through the use of mirrors, object of sharing on social networks”. Short forms addressed in this article are selfies and Instagram Stories made with AR facial effects. There are digital facial effects of all kinds, ranging from beautification (effects that usually soften the skin, enlarge the eyes and make the face look thinner), to animal-like masks (effects that include, for example, animal ears and snouts), face swaps (an effect that swaps the faces of the people portrayed) and disfiguration of the face (similar to distorting mirrors). This is a global trend in selfie production that calls out for the visual semiotic analysis to understand the meanings of such “augmented images” as well as, from another perspective, the meaning-making performative practices of production and usage of such effects. Mainly, AR facial effects work by adding partially transparent virtual layers to augment aesthetically the physical user’s face displayed as a digital image within the device’s screen. Undoubtedly, they constitute a form of entertainment and engagement for social media users. Nonetheless, they prompt the artists’ creativity as several of these have begun to create facial effects by rediscovering surrealistic aesthetics2 that, in turn, have also enticed common users to engage with the creation of digital facial effects. This has become possible since the release of a series of software development kits, like Spark, which provide users with tools to detect, elaborate, manipulate and augment their selfies by creating their own personalized effects. The analysis attempts to understand what may be considered as the “cause”, or the “intentional reason” driving the users’ agency to the production/adoption of facial effects for social media communication, in order to be used to overlap with the floating signifier of the selfie-image and, in turn, to achieve effects on the audience, in a regime of communication capitalism (Salisbury et al. 2017; Fuchs 2020). At the same time, by simultaneously looking to the visual level of the augmented image and to the meaning-making performative practices of production and usage, facial effects production points to an “overlapping mechanism” carried out by the cooperation between the creative user and the computational entity, which encompasses the operation of computer vision and facial tracker engines. In particular, we focus on a specific software, Spark by Facebook (other popular apps like Facetune, Meitu and SwapCam, mainly focus on beautification). 2 See https://www.vogue.it/fotografia/article/the-art-of-filters; accessed on 22 February 2021. Augmented facets: A semiotics analysis of augmented reality facial effects 511 The production and usage of such facial effects relies on augmented reality, that is, the technology providing tools such as face models and trackers. Once launched, Instagram Stories’ software detects, thanks to specific algorithmic functions, the user’s face within the field of reality framed through the camera of the device (which previously has been instructed to recognize and understand through machine learning tecniques). In visual applications, for example, the computational instance makes an ecphrastic description of the surfaces (twodimensional or three-dimensional) that are present in the real environment. Simultaneous Localization and Mapping technologies, specifically, are the most important ones for augmented reality. They work by obtaining visual data from the physical environment in the form of points to feed the data into the machines. Instagram Stories’ software, for example, detects input from predefined vector forms (for example, an oval to detect the face). However, augmented reality, as it was conceived in its early days and as many commercial applications interpret its communicative potentialities, is by no means a medium for mere entertainment. Or rather, it was initially conceived as a worker-oriented technology related to the “personal empowerment” (Azuma 1997; Mann 1998) of the operational/cognitive performance of the worker3; around 2016, augmented reality by marketing strategies found its own medium specificity in popular applications able to increase users’ knowledge of the surrounding environment by offering them, at the same time, an attractive, engaging, pervasive and emotional experience. Today, however, as our daily experience testifies, it is hard to conceive selfie effects in terms of forms of personal empowerment, intended as such. Augmented reality effects are daily used to make the shared images more attractive and appealing: for this reason, the so-called “Instagrammers”, and more specifically the “influencers” and the “digital artists”, are those who, in principle, make the most use of these compositional modes for the production and sharing of their own content. We have to understand, therefore, to what extent augmented reality facial effects for selfies are conceivable as forms of augmentation, that is, of personal empowerment. We will try to answer this question, firstly by asserting that the augmentation does not refer only to the rational and logical empowerment of the user. In fact, it is not the knowledge about one’s face to be increased through the use of a selfie effect: these effects, rather, by virtue of their ability to increase the 3 The first augmented reality device prototypes were designed to increase the performance of workers and military personnel. The development of augmented reality has not, on the other hand, lost this identity: even after the failure of Google Glass for the public market, these devices have continued to be adopted in specialized fields, such as that of the manufacturing industry. 512 Federico Biggio degree of communicative effectiveness of the most important communication device in social media (i.e. the face), determine (or at least are potentially able to determine) a techno-aesthetic experience from which the user emerges continuously transformed. 2. Empowering the face First of all, effects in augmented reality that are taken into account here concern only the augmentation of the face; but Spark (as well as Snap Kit by Snapchat) allows to create virtual overlaps in augmented reality of all kinds, which pose a difficult challenge to the categorizing instinct of semiotics. The question arises what criteria to use to bring together facial effects to add virtual make-up effects, psychedelic light in the selfie background and somatic traits of the opposite sex to user’s face? At the formal level, we might distinguish between aware and un-aware effects, figurative and non-figurative ones, in-real-time and in-hindsight-applied ones (Schipper 2018). Nonetheless, it is not possible to detect any kind of plastic continuity at the visual level of the augmented image, although it is possible to read these phenomena as expressions of a dominant communication logic in the production and sharing of selfies in social media: a “cumulative logic” (Rettberg 2014) that satisfies the needs of a culture of the spectacularizing of individualism (the cumulative process refers precisely to the constant production of visual texts which, once stored into one’s page, let detect a digital identity of an individual user). Moreover, a specific theme has been also that of the users’ empowerment in relation to the possibility of manipulating their own project-oriented digital identity. Rettberg (2014: 12; my emphasis, F. B.) explained that “the ease and inexpensiveness of deleting digital images and taking new ones allows us to control the way we are represented to a far greater degree than in a photobooth or holding an analogue camera up to a mirror”. However, beyond the values of optimization and simplification that these tools bring to the praxeology of the user, the concept of ‘empowerment’ must be specified. Rettberg also proposed the idea that the transformation of users’ faces occurs mainly along with a certain degree of “strangeness”: this is due to the fact that facial effects serve to gain one’s digital image on social media, through a process of distinction from conventional selfies. For instance, when a face effect allows to augment selfies with rainbows coming out of the mouth, it gives the image a strangeness which makes subjecting oneself to other conventions of self- Augmented facets: A semiotics analysis of augmented reality facial effects 513 presentation seem pointless. The peculiar aspect of this practice is that it creates room for self-expression and experimentation beyond the norms. This is probably the reason for which facial effects fascinate us: according to Rettberg (2014: 26), they give the image a strangeness that defamiliarizes our lives and hence enables a critical and self-aware look at them. The strangeness of facial effects, in this context, can be conceived as a vehicle of sense of “estrangement”. For Shklovskij, estrangement is the process by which the poetic image “makes strange” the usual, the everyday, by operating a shift from the usual semantic series in which it is placed and presenting it in a new light, therefore transforming it into a poetic object: it has the function of “disautomatizing” the perception of reality by presenting new sides at different levels and according to different techniques (Šklovskij 1968[1917]). The concept, then taken up by Brecht (1975), denotes also a “distancing” effect from the object of vision that arouses in the observer, an effect which no longer identifies realistically with the character, but captures its representativeness, its being the object of an aesthetic construction. In this last sense, the term ‘estrangement’ has to be conceived as a process by which users become able to “distance” themselves from the potential operation that AR facial effects provide for, to observe themselves critically from the outside, to develop a strategic competence to employ such effects for a project-oriented objective. Our aim here is not to understand the augmented image produced by the selfie created as an artistic or poetic text; this would require an aesthetics that is not pertinent here. Instead, we want to underline the meta-operative component of linguistic production (Garroni 1977, 2005; Montani 2015) that in the specific practice of selfie production is expressed in the evaluation of a series of expressive choices (digital effects), provided by a wide repertoire of linguistic tools, to be used in selfie production practice. At this stage, the user “distances” him/herself from his/her image reflected on the screen of the device, imagines its aesthetic manipulation and, through the use of virtual facial effects, realizes it according to pre-constituted paths. So, from a practical point of view, the act of sharing “augmented selfies” can be understood as a way to control the visual discourses of one’s own, and hence, the gaze of the audience, as today digital self-representation can be intended as “conversational” (Rettberg 2014: 1). In fact, it cannot be denied that the digital availability of any content to improve the face visually is an aspect that, if correctly interpreted and adopted, may be decisive in empowering the communicative performance of a user on the network. From a semantic point of view, the collaborative operation of a user and a computational instance determines an empowerment of the user’s experience as 514 Federico Biggio the immense availability of expressive material in the Web (and, at the same time, the ease of use of linguistic tools) can be seen as an enhancement of the metaoperational possibilities of the user who, using effected selfies, simultaneously enables a critical and self-aware gaze on himself/herself4. For instance, the effects (then banned) that allow users to visualize the appearance of their faces in case of a hypothetical facial surgery or as a forecast to sex change can represent an example of this tendency. In these cases, the augmented reality effects allow prediction, realize pre-visions, by visually realizing subjective or inter-subjective planning, but they also allow us to evaluate this prediction critically by deciding whether to realize it in the “real world”. This mode of effects’ design is even more evident in sponsored filters and lenses to augment users’ face and surroundings with brand logos and paraphernalia. In this regard, the interpretation given to these phenomena by Barnard (2016) proves to be of interest. Barnard focused on the practice of the selfie in the light of ideological and social meanings of a certain culture of individualism, and he exposes a material and affective conception of empowerment, particularly a postfeminist one. The selfie, according to him, functions as “an expressive celebration of the self by seeking to capture embodied experience” (Barnard 2016: 66) and as “a techno-cultural form” (Barnard 2016: 82): in the case of post-feminism, individuals who seek beauty alterations are necessarily empowered with more agency than previous generations, just because one can deduce, from the practice of their self-presentation, a prior understanding and problematization of the dominant aesthetic canons. Other interpretations of effected selfies have been advanced, for, instance by Chae (2017), postulating that human beings have a basic instinct to compare themselves with others and to evaluate their own abilities and opinions and by proving that the selfie-augmenting phenomenon might negatively influence both presenters and audiences in online self-presentation, as “for presenters (individuals who frequently edit their own selfies) the discrepancy between reality and ideal might be problematic […]. Individuals who often take selfies are likely to have an idealized virtual self-image, which might create discrepancy between reality and ideal” (Chae 2017: 375). Thus, although by using face effects users might subject themselves to the gaze of the technology of augmentation and the stereotypical norms of selfpresentation (so, face filters would not empower the user); on the other hand, 4 However, Rettberg (2014: 26) stated later: “Instagram-style filters may make our selfies and photos of our everyday life seem unfamiliar, but the filter itself is repeated so often that the defamiliarisation effect wears off and becomes a cliché”. Augmented facets: A semiotics analysis of augmented reality facial effects 515 though, augmented reality effects also instigate new ways of selfie creation that may be experienced as liberating one’s digital image. 3. Instagrammers and Spark users Before proceeding, we must define the specific properties of the user who uses virtual facial effects to empower his/her own selfie image. These users may be seen as a type of the ‘prosumer’. Jenkins coined this concept in 2013, by retrieving from Toffler, denoting specific creative practices which concerned the public not as simply a consumer of pre-packaged messages, but as a subject who simultaneously consumes and produces contents circulating in media scenario. Mainly, prosumers’ prototypical textual production practice involves the appropriation of multimedia contents on the Web and their remix through different types of software that lead to the creation of syncretic text forms, such as fandom movies, memes or digital effected selfies. Instagram users are prosumers as they use pre-packaged effects to augment their own selfies. According to Lev Manovich (2017: 117)5, Instagrammers can be said to own “the means of cultural production”: this means, however, not only simply owing mobile phones and apps but, more importantly, “having skills in using these apps, understanding Instagram’s rules and strategies for creating popular feeds, and being able to apply well these strategies in practice”. However, users augment their own selfies with digital effects which have in turn been produced by others. Prosumers are both Instagram users who adopt facial effects and Spark users who create such effects using a platform for content creation. Spark users can be equally conceived as the receivers of a commercial product, which allows them to create an augmented selfie and share it with the digital users’ audience. They are equally spectators of a marketing strategy (and this is evident by looking at claims such as “Augment your space. Augment your style. Augment your sound. Augment your world”, where ‘your’ explicitly refers to the subjective dimension of the experience in augmented reality6). Instagram users, on the other hand, are two-fold spectators, as they receive two products: the one produced by the company and the one produced by Spark users. 5 Manovich, Lev 2017. Instagram and Contemporary Image. Available at: http://manovich. net/index.php/projects/instagram-and-contemporary-image. 6 See https://sparkar.facebook.com/ar-studio/; accessed on 22 February 2021. 516 Federico Biggio 4. The face–software communication However, by looking strictly at the effects’ production, the prosumer is not the only condition for facial effects use. We have to consider two further aspects of the phenomenon: on the one hand, the augmented reality software activity; on the other hand, the aesthetic canons imposed by a certain socio-cultural community. As regards the former aspect, Spark represents both the software environment where effects are being created as well as the set of technical features (as face models and trackers) by means of which Instagram users augment their own selfies. The Spark7 interface (Fig. 1) features an intuitive programming environment and a set of tools that allows prosumers to use a set of computational tools to create digital effects. It is characterized by a central section (called ‘viewport’) where “Euclidean” space is represented and a vectoral representation of the device camera and of the effect take place, with a window (called ‘simulator’) showing how the effect would look on a mobile device. Here, a plane is visualized in a 3D space, located in front of the device’s camera, as well as a directional ambient light and a microphone: it is exactly in this interstitial space, organized in hierarchical and overlapped layers, that the “spectacle of augmentation” can take place. Figure 1. The interface of Spark AR Studio (https://sparkar.facebook.com/ar-studio/ learn/tutorials/introduction-to-spark-ar-studio; accessed on 22 February 2021). 7 By replying the success of Lens Studio by Snapchat, Zuckerberg’s company released the Camera Effect Platform for AR Studio in 2017. The article focuses on the analysis of Spark AR Studio, the software developed by Instagram and then acquired by Facebook. Augmented facets: A semiotics analysis of augmented reality facial effects 517 Then, on the left, a double menu serves to store the objects, the trackers and the lights (the scene panel), and, finally, the materials, such as the textures and the scripts (the assets panel) imported by the prosumer. On the right-hand side, a multi-function panel displays functionalities according to the object selected in the right-hand panels. Finally, on the lower level, there is the patch editor, the section that serves to add interactivity and animation to effects by means of a system of boxes and links. The main operation scheme to create a facial effect in Spark is to combine an ‘object’ – a face tracker – with a ‘material’ – an asset – by means of a face mesh. Once it has been uploaded, the face tracker becomes able to detect users’ facial movements framed by the device’s camera and then associated to the face gestures recognition engine. It responds to users’ interactivity by displaying the digital effect. In this sense, we can state that Spark is featured by (and imposes on the productive process) a linguistic paradigm whose syntagmatic elements correspond to each body movement and gesture that users must actualize by interacting with the machine in a meaningful way. Finally, a face mesh – that is a 3D model of a face – reconstructs users’ expressions according to the detections made by the face tracker. In the assets panel, an image, an object or a texture, must already be associated to a material, in order to be later assigned to the face mesh and displayed in the viewport. Also in this case, the creative process has to be conducted toward a precise and standard format. For instance, when a texture has been uploaded, the user has to choose the material and the shader type by selecting among different grades of transparence: if the retouching mode has been chosen, the texture assumes the form of a patina, similar to the beautification filters to cover the imperfections of the skin. Gradually, from face paint mode to the blended one, the visibility of the effect becomes more pronounced. Although it is possible to regulate the grade of opacity of the layer for each type, the command name is still indicative of the previewed use. The first pivotal point here is the detection of an actual linguistic code detectable through software analysis. The admitted facial gestures shape a sort of “syntax of interactivity” of the face–software communication. For instance, it is possible to set an object appearing at the tapping of the screen or when the user face assumes a specific expression: facial movements such as eyes blinking, eyebrows raising or lowering, head rotation and shaking, or facial expressions such as a kissing or a smiling face, but also voice for audio inputs, are becoming ever more standardized gestures that spread in digital culture by stereotyping facial communication in social media contexts. An example is the so-called “side profile check”, that is, the rotation of the head in front of the camera to show clearly the two profiles of the face. 518 Federico Biggio In semiotic theory, communication between two instances occurs only when both instances own knowledge (an “encyclopedic” one) that allows them to (1) formulate linguistic occurrences and (2) understand those formulated by the interlocutor. Beyond the automatized-like effect of the users’ natural communication with artificial-intelligence engines, adhering to certain interactivity schemes prescribed by the software (or by the effect), result, as a consequence, in the increase of the awareness of mediation, as well as the development of a critical self-judgment of one’s appearance that is certainly more pertinent in the field of social and psychological sciences. This self-awareness plays a central role in the competitiveness between users of social networks (Chae 2017). Although the repetition of a movement in front of a camera may recall the modus operandi of the so-called “challenge”, the performance of these gestures does not aim to take an active part in a process of collective support of ideologically featured causes, but aims to express the users’ unique and unrepeatable aesthetic appearance through more or less objective evidence, in the production of visual texts (e.g. selfies) that can be sanctioned by the community of users both positively and negatively. In fact, the creation/adoption of specific facial effects can also be sanctioned by the social community as well as by the owners of the platform (i.e. Facebook). For example, it is a fact attested in psychological studies (Ramphul, Mejias 2018) that the spread of facial effects has led more and more users, especially the youngest, to evaluate the possibility of plastic surgery that allows them to touch up the shape of the eyes (that is, to have “cat eyes”), to enlarge the volume of the lips or to shrink the nose. To denote this type of “disorder” the expression ‘Snapchat dysmorphia’ has been coined. These considerations have led to several negative sanctions of facial augmentation practices with virtual effects, such as banning of cosmetic surgery effects as created by Daniel Mooney, prohibited since these promoted plastic surgery and harmed people’s mental health (Fig. 2). Augmented facets: A semiotics analysis of augmented reality facial effects 519 Figure 2. Daniel Mooney’s plastic surgery effects, retrieved from https://www.insider.com/ instagram-cosmetic-surgery-filters-removed-2019-10; accessed on 22 February 2021. 5. The awareness of the empowerment and the un-awareness of the mediation We might distinguish between the awareness of the empowerment and the awareness of the mediation. The former refers to the recognizing of the effect within a selfie-image: it occurs always, as the effect is created/adopted by users themselves. The latter, however, refers to the recognizing of the presence of an effect within another user’s photo (moreover, to the “presence” of the computational activity which may be attested just from the weirdness of certain effects): the awareness of the mediation is an unnecessary condition since many effects, e.g. make-up or beauty ones, try to achieve maximum possible truthfulness: “Digital images start to look “natural” to us; moreover, even technologically embedded visual effects, including those that heavily modify facial reproductions, start to acquire an aura of ineluctability” (Leone 2020: 6). FaceTune and Meitu represent the apex of this trend. Hence, the negative sanction of the community, similarly to the psychological one, does not move from the assumption that the user is unaware of the empowerment. Digital users live as “immersed” within a software culture, they know how programs work, how to manipulate computational elements and how to act synergically with algorithmic entities in order to express themselves in social media contexts (Manovich 2017). Rather, the users are aware of the empowerment, but they are un-aware of the mediation. The negative sanction moves from the assumption that the user is un-aware of the mediation, that is, the presence of a 520 Federico Biggio database which has been created by Facebook, the in-between actor which provides the tools for the empowerment. By searching for an ever-better effects and by confronting those employed by themselves with those used by others, users would risk to lose the grasp of the all-encompassing totality of the whole mediation, from which emerges a clear formal homogeneity that translates the whole database into an aesthetic canon to be respected. According to Manovich (2019: 6), “while the gradual AI integration into phone cameras and sharing sites may contribute to a decrease in aesthetic diversity, the simultaneous addition of more and more controls to cameras and photo apps may have the opposite effect”. On the other hand, the platform that fosters users toward such experiences persuades them with seemingly endless collections of effects. It embodies a textual organization that has established “inter-objectual user positions” which convince them that they can manipulate the face in any way, without limits to the imagination. However, “disintermediation through technology inevitably leads to standardization, which is completely at odds with the promise of autonomy that the technological market sells” (Leone 2020: 6). Hence, by adopting a critical media standpoint “artistic” practices of creation on Spark, as well as the usages of digital effects by Instagrammers cannot be conceived in the same way of free expression articulating computational matter, in the same terms of an artistic gesture. Although some kind of constrictions can always be traced in artistic traditions, the AR facial effects’ creation can be read as more constrictive of the former, at least because of their ownership. Rather, they must be conceived as user-friendly applications which transform computationalism and creation in entertainment processes, whose applications’ user engagement is a fundamental component of the provided programs of actions. 6. The paradox of user-friendly computational creativity Spark’s interface is a high-level one and the software itself should be conceived as a “company product” (which owns a certain aesthetic coherence or, at least, requires it) rather than a software development kit. Spark is not properly a Software Development Kit (SDK) but rather a media platform. A Software Development Kit is a collection of tools provided by a graphical interface, which can vary in its complexity and allows its users to create homogeneous applications which interact with the hardware of the computational instance without them having to program the deepest interfaces. The variety of application complexities that different SDKs allow to create is at the same time an index of the SDK’s accessibility. A SDK that in Augmented facets: A semiotics analysis of augmented reality facial effects 521 large part provides the writing of computer code can be considered less accessible than one like Wordpress in which you get standard results, but these are always finished, even from a simple manipulation and assembly of “textual blocks”. Spark similarly features a very intuitive and accessible interface (if we take into account the basic skills that Instagrammers possess, the reason is understandable). For example, since the launch of the program, Spark features a template window (Fig. 3) that proposes different effects templates for a quick start of the creation, e.g., the make-up, the face decoration, the face mask or the 3D animated poster (but also a blank project): these are conceivable as a standardized action program. Figure 3. The template window of Spark, retrieved from https://sparkar.facebook.com/ ar-studio/learn/articles/fundamentals/templates#opening-a-template; accessed on 22 February 2021. In this sense, on the one hand, Spark is a media platform, a public database that users can access in order to retrieve elements and techno-linguistic tools to create their own original content; on the other hand, the linguistic identity of this media platform, that is, the aesthetic and ethical codes that digital effects’ seriality constitutes, can at the same time be traced back to a sort of “standardization” of textual production practice. However the easiness and immediacy of the interface does not necessarily correspond to a limitation of the expression potentiality of 522 Federico Biggio the user. Spark has been designed mainly for users with poor programming skills who need to create amazing content as quickly as possible. Users learn to explore Spark interface step by step thanks to a huge quantity of online tutorials provided by Facebook. These texts constitute a second (elementary) form of “prescription” of the creation process, which leads unavoidably to a homologation of the aesthetics of digital effects: the tutorials concern, for instance, the creation of stereotypical face effects and have been configured precisely as a sort of “coaching services”, useful to become artists and to achieve the success within the social media community. Moreover, it is meaningful that nowadays Spark is not just featured within the platform Facebook Developers, but it has become an application for common users’ entertainment. Its Facebook Community, the “Spark AR Community” group, nowadays counts over fifty thousand members and the Facebook Page “Spark AR Creators” has over eighty thousand likes. Here, Spark creators daily share facial effects created and discuss these issues with one another. The definition of the platform’s standards can thus be related to the establishment of a community, both a social, a cultural and an economic one. This happened at the birth of the World Wide Web, where the “http” protocol allowed different servers to communicate with one another; this is happening today in cases like Spark in which the supply of face assets by companies allows users to create their content with their preferred software, but having just the provided ones as a base standard and sharable models. Finally, the capitalization of creativity led by Facebook, capable of enforcing adherence to certain aesthetic and moral canons, concludes with the approval process of the effect by the owners of the platform. Here, it is not the community but the Hub of the application (in this case Meta) where the users upload their creation to operate as a “sanctioning instance” whose role is to approve the wellbeingness of the digital effects, according to a strict set of policies. Once the effects’ production process is concluded, the user must submit the effect to the platform, by means of the Hub, in order to make the effect in the platform accessible and usable. Although it could not be defined as an actual censorship process, there is no doubt that a certain further filtering operation has occurred. In this regard, Rettberg (2014: 20) proposed also a broader definition of the term ‘filter’: in today’s algorithmic culture, the filter is not simply the kind that removes skin impurities, but it is instead a pervasive metaphor for the ways in which technology can remove certain content and it can alter or distort the users’ natural faces. The paradox of user-friendly computational creativity lies precisely here. These forms of digital creativity clearly express a will of liberation from aesthetic stereotyping, as well as a free expression of one’s own creativity, but this ideal Augmented facets: A semiotics analysis of augmented reality facial effects 523 is pursued through the use of creative tools that are “constrictive”. Barnard (2016: 82) points at this tension and proposes the notion of ‘(dis)empowerment paradox’: in this context “I have identified a (dis)empowerment paradox where such expression may feel empowering to the individual(s) controlling the camera while concurrently conforming to and reifying oppressive, hegemonic norms”. 7. Conclusion: Aesthetic empowerment and interactive imagination Spark effects are not similar to aesthetic filters for digital cosmetics: whereas the latter are useful to improve a photograph in as realistic and transparent a way as possible, the former feature a higher artificiality. If in the 19th century the photographic technique established the absolute transparency of the technology (and thus the un-awareness of the mediation), as well as the suppression of the traces of reproduction of the medium, in the case of computer graphics creativity through augmented reality we witness a predominant return to the awareness of the medium, of its virtual aesthetics as well as of its creative potentialities – both in the production process and in the use in social media contexts – which in no case can be considered to be oriented just to the improvement of an image, but rather to an original reinterpretation of its potential uses. Already Photoshop retouching, for instance, was not just a way of improving photographs, but a means for individuals’ artistic expression, useful to enhance the creative gesture. The same goes for augmented reality effects. At this point we might recover the original idea attributed to augmented reality, but we should admit that the “personal empowerment” of the operational/cognitive performance of the user is not at stake here. Instead, the empowerment relies on the platform’s ability to provide users with an attractive, engaging, pervasive and emotional archive of virtual facial effects to create an enhancing visual experience. Although in some make-up effects the objective of the empowerment is truthfulness, the main feature of Spark’s face effects is evidently the ability to create explanations of the increased reality’s creative potentialities through the genius of the creator. For this reason, the use of emerging digital media like augmented reality for artistic purposes is very meaningful. This trend represents a form of “technological enchantment” which aims to foster human-made technical creativity, a childish and irrational sensibility that leads users to recover a primordial dimension of existence where techniques are at the service of humanity, and serves to discover and to interpret the strangeness in the world. In an article about endosemiotic features of augmented reality, Wamberg (2012: 471) stated 524 Federico Biggio that “this reawakened and even empowered indexicality of augmented reality has certain similarities with natural habitats and can indeed be understood, I would claim, as a relinking to natural layers of experience”. For Wamberg, by reactivating as the main sign of indexicality, augmented reality emphasizes the “proactivity” of interaction and rehabilitates forms of performance related to the magical feats. So, this analysis intended to affirm positively the communicative effectiveness of facial filters in augmented reality that, unlike the (much more widespread) beautification effects, have the possibility to be strategically employed by users to emphasize and empower their communication practices in social media contexts. However, this can only happen as long as the digital effect was considered as a strategic technology, and not as a technological product capable of compromising the users’ natural understanding of the real world. In the context of Instagram, this poses a risk of standardization and aesthetic repetitiveness. The justification for the hypothesis of an anthropological gain comes from the aesthetic theory of Pietro Montani, modelled on Garroni’s and Malafouris’ thesis on material engagement with the tools of languages. Thinking of augmented reality, Montani pointed to the activation of an interactive imagination, by describing it as a fundamental competence in the evolution of the human being, essentially rooted in sensoriality. In this regard, Augmented Reality can be conceived as a creative operation able to rethink and re-imagine the natural world in which the user is the protagonist, to act on and through “forms of technical life capable of transforming into a non-programmable world, albeit in accordance with rules” (Montani 2014: 77). Over thirty years of computational culture have taught users that everything that is digital has been programmed, and, at the same time, this has been fuelling the idea that everything, including reality itself, is a priori programmed. It is from this consideration that computational creativity can be understood as an individualistic, yet still positive counter-programming strategy of expression. By putting proactively a filter on our selfies, or framing them by placing them in a blog or an Instagram feed, users see themselves and their surroundings as if they were outside of ourselves, by re-inventing the real world, by imagining it in an interactive way, by re-projecting it in an un-restricted way. I would like to conclude this reflection with an open question which concerns a final aspect that is not addressed here for it is not directly relevant. To what extent would the aesthetic empowerment of the user be conceivable or comparable to the personal empowerment of augmented reality traditionally intended, if seen as related to the gnoseological increase, a quantitative and qualitative one, of user performance in the flesh? Answering this question would require further investigation of the concept of empowerment, by focusing on the dis-empowering aspects, which Wamberg (2012: 474), for example, defined as “frustration with Augmented facets: A semiotics analysis of augmented reality facial effects 525 nature and its artificial protheses in technology”, in order to understand how the subject of empowerment, previously anesthetized by dis-empowerment, may again be empowered through facial augmentation performances. References Azuma, Ronald 1997. A survey of augmented reality. Presence: Teleoperators and Virtual Environments 6(4): 355–385. https://doi.org/10.1162/pres.1997.6.4.355 Barnard, Stephen 2016. Spectacles of self(ie) empowerment? Networked individualism and the logic of the (post)feminist selfie. Communication and Information Technologies Annual 11: 63–88. https://doi.org/10.1108/S2050-206020160000011014 Brecht, Bertolt 1975. Scritti teatrali I. Torino: Einaudi. Chae, Jiyoung 2017. Virtual makeover: Selfie-taking and social media use increase selfieediting frequency through social comparison. Computers in Human Behavior 66: 370– 376. https://doi.org/10.1016/j.chb.2016.10.007 Del Marco, Vincenza 2017. L’immagine in rete. Selfie, social network e motori di ricerca. Roma: Edizioni Nuova Cultura. Fuchs, Christian 2020. Communication and Capitalism: A Critical Theory. London: University of Westminster Press. https://doi.org/10.16997/book45 Garroni, Emilio 1977. Ricognizione della semiotica. Roma: Officina Edizioni. Garroni, Emilio 2005. Immagine, linguaggio, figura. Roma: Laterza. Jenkins, Henry; Ford, Sam; Green, Joshua 2013. Spreadable Media: Creating Value and Meaning in a Networked Culture. New York: NYU Press. Leone, Massimo 2020. Digital cosmetics. Chinese Semiotic Studies 16(4): 551–580. https:// doi.org/10.1515/css-2020-0030 Mann, Steve 1998. Humanistic computing: “WearComp” as a new framework and application for intelligent signal processing. Proceedings of the IEEE 86(11): 2123–2151. https:// doi.org/10.1109/5.726784 Manovich, Lev 2019. AI Aesthetics. Moscow: Strelka Press. Montani, Pietro 2014. Tecnologie della sensibilità. Estetica e immaginazione interattiva. Milano: Raffaello Cortina. Montani, Pietro 2015. Prolegomeni a una “educazione tecno-estetica”. Mediascapes Journal 5: 71–82. Montani, Pietro 2020. Emozioni dell’intelligenza: Un percorso nel sensorio digitale. Milano: Meltemi. Pezzini, Isabella 2002 (ed.). Trailer, spot, clip, siti, banner: Le forme brevi della comunicazione audiovisiva. Milano: Meltemi. Ramphul, Kamleshun; Mejias, Stephanie 2018. Is “Snapchat dysmorphia” a real issue? Cureus, 10(3). https://doi.org/10.7759/cureus.2263 Rettberg, Jill Walker 2014. Seeing Ourselves through Technology: How We Use Selfies, Blogs and Wearable Devices to See and Shape Ourselves. London: Palgrave MacMillan. https:// doi.org/10.1057/9781137476661 Salisbury, Meredith; Pooley, Jefferson 2017. The #nofilter self: The contest for authenticity among social networking sites. Social Sciences 6(10): 2002–2016. https://doi.org/ 10.3390/socsci6010010 526 Federico Biggio Schipper, Meike 2018 “A Whole New Way to See Yourself(ie)”: Exploring How Face Filters Transform the Practices of Selfie Creation. [Unpublished PhD Dissertation.] Utrecht: Utrecht University. Šklovskij, Victor 1968[1917]. L’arte come procedimento [Iskusstvo kak priëm]. In: Todorov, Tzvetan (ed.), I formalisti russi. Torino: Einaudi, 73–94. Wamberg, Jacob 2012. Toward a (re)onstructed endosemiotics? Art, magic and augmented reality. In: Ekman, Ulrick (ed.), Throughout: Art and Culture Emerging with Ubiquitous Computing. Cambridge: The MIT Press, 465–478. Семиотический анализ цифровых масок дополненной реальности Цифровые маски, использующие технологию дополненной реальности – новый тренд общения в социальных сетях, предпочитающего «краткие формы». В статье предлагается трехсторонний анализ: семиотический анализ цифровых масок, используемых для усиления естественных возможностей лиц пользователей; деконструкция Spark, одного из основных приложений для создания подобных эффектов; наконец, критическое рассуждение о предписанных Spark практиках и стереотипной эстетике дополненных селфи. В заключении утверждается, что такие формы эффектов дополненной реальности должны быть задуманы не как ориентированные на когнитивное улучшение работы пользователей, а скорее как формы расширения возможностей и самосознания пользователей. Augmenteeritud tahud: augmenteeritud reaalsuses ette tulevate näoefektide semiootiline analüüs Augmenteeritud reaalsuses esinevad näoefektid esindavad uut suunda “lühivormidel” põhinevas ühismeediasuhtluses. Artiklis esitatakse kolmeosaline analüüs: loomulike kasutajate nägude võimestamiseks kasutatatavate digitaalsete näoefektide semiootiline analüüs; Facebooki pakutava Sparki, ühe olulisema sellisete efektide loomiseks kasutatava tarkvararakenduse, dekonstruktivistlik analüüs, ning viimaks kriitiline mõtisklus Sparki pool ette kirjutatud praktikate ja augmenteeritud selfide stereotüüpse esteetika üle. Kokkuvõttes leitakse, et seesuguseid augmenteeritud reaalsuse efektide vorme ei tule pidada kasutajate soorituse kogntiivsele parendamisele orienteerituks, vaid pigem kasutajate võimestamise ja eneseteadvustamise vormideks. Face and trust: A semiotic inquiry into inflSystems uencers, Studies money, 49(3/4), and amygdala 527 Sign 2021, 527–542 Face and trust: A semiotic inquiry into influencers, money, and amygdala Kristian Bankov1 Abstract. After the cultural explosion of Web 2.0, digital culture reveals an apparently semiotic paradox associated with the incredibly widespread use of images of faces, while at the same time the reason to trust in the authenticity of these faces is constantly declining. This is because graphic technology has made the sophisticated manipulation of images both possible and easy. After a review of the existing semiotic models and considerations of trust, I am proposing a new approach which emphasizes the value-generating properties of trust by analogy with the money sign, seen as “trust inscribed”. Research from the neurosciences supports the hypothesis that the trustworthiness of the face is judged pre-reflexively and primordially. This, therefore, means that a trustworthy face is a premise for more successful communication than an untrustworthy one, notwithstanding the object of discussion and the cultural context. An example concerning social media influencers serves to show that in the internet-dominated globalizing culture, trustworthy faces are a multipurpose communicative asset that makes a difference. Keywords: face; trust; semiotics; influencers; economic value Introduction After the cultural explosion of Web 2.0, digital culture reveals an apparently semiotic paradox associated with the widespread use of images of faces, while at the same time the reason to trust in the authenticity of these faces is constantly declining. This is because graphic technology has made extremely sophisticated manipulation of images possible. As Massimo Leone (2019: 20) puts it, “pictures of the face are now extremely believable and extremely untrustworthy at the same time”. Roland Barthes saw the semiotic nature of photography in its ontological relation with reality (Barthes 1981: 85), but today the trend is increasingly more 1 Sichuan University/New Bulgarian University, Sofia, Bulgaria; e-mail: kbankov@nbu.bg. https://doi.org/10.12697/SSS.2021.49.3-4.14 528 Kristian Bankov connected with the modification of the faces we can see on screens and their value deriving from differences with their real offline referent. Here we are talking about a whole prosperous industry, emblematic of all areas of our digital everyday life. There was much discussion surrounding the Academy Awards 2020 in Bulgaria. A special FX company run by two of our compatriots received an award in the category of Special Effects. One of the most complex things this company does is realistic animation of the faces of celebrities from the past and introducing them into new plots in modern productions. Thus, the last episode of Season 2 of The Mandalorian (2020) featured an appearance of the legendary Luke Skywalker at a visible age of about 30, interpreted by the no less legendary actor Mark Hamill. The amazement of the savvy spectator was based on the fact that the actor was approaching 70 in the year of filming. Such manipulations of faces have been created for at least 20 years now, but today the phenomenon is becoming a significant part of popular culture. With the same technology, the UK television network Channel 4 created a fictional Christmas speech by Queen Elizabeth II to the astonishment of the television audience. The main topic of the speech was the trust in what is seen in media, as well as new technologies allowing reality manipulation. It has become the norm not to accept compromising videos involving famous politicians and businessmen until experts have proven beyond doubt that no special effects have been used. It has become easy to modify lip movements in a completely realistic way in a video of a famous person, and to attribute to him/her the pronouncement of words that s/he has never uttered. As a result of such, and many other, abuses, there is already a branch in cybersecurity known as ‘Faceforensics’.2 In 2019 the state of California ratified Bill AB-602 which bans the use of human image synthesis technologies to make fake pornography without the consent of the people represented, and Bill AB-730 which prohibits distribution of manipulated videos of a political candidate within 60 days of an election. More and more mobile apps are creating value by manipulating selfies or other photographs of users’ faces. These are classic beautifying techniques such as erasing wrinkles and skin imperfections, enlarging the eyes, optimizing the face oval and the shape of the jaw, etc. The product of these selfie manipulations is most often used for profile pictures or simple posts on social media. The aim is to obtain approval/recognition for the referent of the manipulation, not so much for the complex semiotic act of communication. Strangely, manipulated 2 Rossler, Andreas; Cozzolino, Davide; Verdoliva, Luisa; Reiss, Christian; Thies, Justus; Nießner, Matthias 2018. FaceForensics: A-large-scale video dataset for forgery detection in human faces. Available at: https://arxiv.org/pdf/1803.09179.pdf; last accessed on 30 January 2021. Face and trust: A semiotic inquiry into influencers, money, and amygdala 529 face and body photos are also widely used in dating applications such as Tinder, although in theory their success is based on the trust that what we see in the photos corresponds to what we will see at a real-life meeting. We can imagine that the manipulated improvements actually stimulate the imagination and condition the perception when the time arrives for a face-to-face meeting, which puts consumers in a win-win situation. Social media apps (such as Reface) which allow the user to put his or her face in the place of that of a celebrity in videos with cult scenes from the cinema or videos of a virtuoso performance are gaining huge popularity. Video manipulations in the opposite direction are also becoming popular – celebrity faces replace the faces of unknown participants in pornography, misleading naive viewers to think that they are actually watching celebrity porn (Alexandrou, Maras 2018). Even “serious” online conferencing platforms such as Zoom have recently developed applications that manipulate the appearance of communicators in real time. Rossler et al. summarize the problem: “The ability to effortlessly create visually plausible editing of faces in videos has the potential to severely undermine trust in any form of digital communication” (see fn 2). “This face does not exist” controversy The essence of the problem transpires as a result of the debate around the application of AI technology for deep learning Generative Adversarial Network (GAN). In the context of the Deepfake productions the role of the face is central and considerable research is being conducted on the topic. Shen et al. 2019 as well as Tinsley, Czajka and Flynn3 provide a good overview of the known facts and the research literature on the relevance of the problem. For the sceptics: [i]n the age of fake news and alternative facts, the risks and dangers associated with ill-intentioned individuals or groups easily routing forged visual information through computer and social networks to deceive, cause emotional distress, or to purposefully influence opinions, attitudes, and actions have never been more severe. (Shen et al. 2019: 23) GAN technology probably would have remained popular only within the IT and design professionals’ environment, if it were not for its application on the human 3 Tinsley, Patrick; Czajka, Adam; Flynn, Patrick 2020. This face does not exist... but it might be yours! Identity leakage in generative models. Pre-print of a paper to be presented at WACV 2021. https://arxiv.org/pdf/2101.05084.pdf (last accessed on 20 February 2021). 530 Kristian Bankov face. The way it is applied to the face illustrates how it can be applied to any other visual data. GAN allows for photorealistic synthesis of digital images of human faces that does not correspond to any existing human. They can be seen on the website https://thispersondoesnotexist.com/ and the effect on the viewer is striking: it breaks our ontological relation with reality as no other type of images does. As we shall see, there is a primordial and pre-reflexive moment in the perception of the face which overwhelms the consciously perceived statement that such a face cannot exist in the real world. I wonder how Levinas would have commented on such “otherwise than being” of the face. In the proposed framework this is a semiotic paradox which would not have been possible before an advanced phase of the digital technologies. Trust in semiotics “In semiotics we trust” may be the credo of many among us who base our careers as researchers on the science of signs, but very few have taken a theoretical interest in that basic notion. In this part of the article, I firstly list the major contributions of semiotics of trust, and then try to position the problem of trust in the main semiotic models, in an attempt to explain the weak interest. The classic author who comes closest to the concept of trust is Greimas. The Semiotics of Passions (1991), written together with Jacques Fontanille, sees trust as an ingredient in some of the well-known passions, especially jealousy, but also stinginess. There is a mini semiotic square (see Fig. 1), dedicated to a modal analysis of trust (confiance in French), which explores semantic subtleties of the French language, untranslatable to other languages (Greimas, Fontanille 1991: 218): CONFIANCE MÉFIANCE ?? PRÉFIANCE DÉFIANCE Figure 1. Semiotic square of trust/distrust. The analysis is exciting and could be developed into a stand-alone model, but the framework and context are entirely textual and literary. In these, trust is reduced to a logical operator in the modal structure of passions, and although the claim of the overall theory is for a semiotics of human action, it does not exist outside the Face and trust: A semiotic inquiry into influencers, money, and amygdala 531 textual world (dictionaries and literary works). In the “real world”, according to my framework, degrees of trust reflect the success and capitalization of communicative interactions. Fontanille’s student Amir Biglari develops in depth the semiotics of passions in relation to hope, where trust has an even more central role (Biglari 2011). However, once again in this work trust is necessarily immanent to a literary textual structure, using the same literary references, in particular The Man Who Laughs by Victor Hugo. In his general theory, Greimas “clears” all the methodologically dubious implications of the instance of trust by introducing one of the most ingenious tricks of his theory – replacing the notion of truth with that of truth-saying, or veridiction. According to Eco (2000: 257), “that seems tailor-made to irritate any upholder of a truth-functional semantics, not to mention every supporter of a correspondence theory of truth”. Nevertheless, this trick ensures the exceptional internal coherence of the theory. From a potential source of metaphysical torment, the problem of trust is resolved as a fiduciary contract working as a logical binary code – either it exists, or it does not exist. If there is a fiduciary contract, the semiotician can consider the semantic effects of the communicative process according to the canonical generative model, and if there is no such contract, there is no reason for the method to be applied. Such a construct of the semiotics of trust does not work for the purposes of a study of the role of the face in digital communication, since the intrigue lies more in the intermediate positions and degrees, rather than in the overall presence or absence. We find similar intermediate positions and preconditions for successful semiotics of trust in Eco’s textual pragmatics model developed in The Role of the Reader (Eco 1994). This work presents the modelling of a much more dynamic signification process in which, albeit without a direct semiotic modelling of trust, there is an entire dialectic of the relationship between the author and the reader based on, as we shall see, a simulacrum of trust in the sense that the driving force of the interpretive process is the reader’s presuppositions, governed by the narrative mechanisms of the plot set by the author. A good author manages to gain the trust of his empirical reader by appropriately inscribing in the text a reader model, which bears encyclopedic competencies similar to the reader’s genre and life competences. The author’s ally in this endeavour is the process of suspension of disbelief, with which the reader voluntarily abdicates – but only to a certain extent – his or her critical distrust of the narrated facts and events (Eco 1994: 16, 208). In this way, the reader begins to inhabit the possible world of the work imaginatively and switches from their everyday system of expectations to a fictional one, trusting in certain possible confluences of the plot at the expense of others. It is the reader’s trust that is the most valuable asset for the author since it is thanks to this that the author 532 Kristian Bankov can create the highly desired twists in the action. These twists show the reader that his/her trust, collaboratively misled by the author, has been projected in the wrong direction. Each twist in the course of the plot is a retelling of the cards on the table and a new round of the game of trust and conjecture. In Eco’s model, the concept of frame plays a central role in explaining the process of the permanent presupposition that drives interpretation (Eco 1994: 21–27). He shapes the notion of ‘intertextual frames’. Frames are culturally codified portions of meaningful experience that the author relies on as being available in the readers’ encyclopedic competence and used by them to unravel the plot. At the same time, an important direction in the otherwise boundless research of trust also puts the concept of frames at the centre of its theoretical modelling (Lewicki, Brinsfield 2011) and, as we will see in the next section, the analogies are impressive. Valuable input into the semiotics of trust might have been expected from the cognitive branch of the discipline, but as surprising as this may seem, the global repositories of published research show a marked absence of such work. The story is different with Peirce – as with any other subject imaginable, he has his considerations on the notion of trust as well. Although these are quite marginal in comparison to other notions, they carry a heuristic potential (on the faith–trust– perception continuum) to be explored even in the context of the research direction referred to in the following paragraphs. Should the reader want to pursue the inquiry, great systematic review of the notion of trust in Peirce can be found in Pope 2018. Otherwise, Peirce’s concept of abduction has been used to define a new method of examining trust in the context of business organizations (Le Gall, Langley 2015), although this cannot be defined as a semiotic contribution. Similarly, the phrase ‘semiotics of trust’ is sometimes used in the complex matter of trust in financial markets, but it appears at a superficial metaphorical level, reflecting the need for certain symptoms and signals from stock exchanges to be interpreted in a timely manner for more adequate financial asset management (Bikoula 2011; Swedberg 2010, 2012). A very original contribution under the label ‘Trust in Signs’ is offered by Bacharach and Gambetta (2001) who have developed an accomplished model. The context is that of the trust in society in general, and the emphasis falls on the everyday practices of using credit cards, money exchange, touristic situations, games of deception, and the like. The authors define this situation as “relentless semiotic warfare” (Bacharach, Gambetta 2001: 167), but the only theoretical reference they provide is that of the signal theory, while semiotics as a theory is completely absent. The way the action of all signs and strategies of deception is described and modelled is naïve, as there is no instance of enunciation, and the signs/signals are conceived as bearers of universal meaning, independent of the Face and trust: A semiotic inquiry into influencers, money, and amygdala 533 syntagmatic chain or the pragmatic situation. Yet what is interesting for our inquiry is the part on the signs/signals of identity (Bacharach, Gambetta 2001: 167ff). The authors introduce the term ‘Krypta’ – the unobservable properties of a person that are his/her trust-warranting features (Bacharach, Gambetta 2001: 177). In this case, the context-independent efficacy of trustworthiness of Krypta is close to the way in which I define the semiotic efficacy of the face, but beyond the description nothing else can be unpacked from this model to develop my proposal, which is focused on digital communications in the post-truth era. In authors such as Liu and French we find very specific applied models of the semiotics of trust. They are used for operational diagnostics of online services and virtual organizations (French, Liu, Springett 2006)4. The joint publication of these authors sets out a detailed methodology for research commissioned with companies conducting business through web platforms in mind. The end product for the clients of such a study is a semiotic trust profile. For this purpose, the most general cultural semiotics approach is used, which, unlike conventional approaches, provides the necessary significance to the socio-cultural world of consumers through cross-cultural analysis (French, Liu, Springett 2006: 177, 180). Another important contribution offered by these authors is the creation of a model in which the factors generating trust in virtual organizations are graded. This model is called the Semiotic Trust Ladder and it “currently functions as a kind of metamodel, within which VO [virtual organizations] partners can conceptualize e-trust issues within a typical partnership, from its earliest inception to design and implementation”5 (French, Bessis, Maple 2010: 1115). Although these models are valuable as a practical application of a semiotics of trust, they cannot provide the necessary theoretical basis for the purposes of this study focused on the role of face in trust building in the era of ubiquitous digital image manipulation. No trust inside the text Before proposing certain semiotic reflections on the relationship between face and trust on the Internet, let us elaborate the analogies between the role of trust 4 French, Timothy Stewart; Liu, Kecheng; Springett, Mark 2006. Towards an E-Service Semiotic Trust Framework. In proc. ALOIS 2006, Borås, Sweden; available at http://www.vits. org/uploads/alois2006/french_liu_springett.pdf. 5 French, Timothy Stewart; Bessis, Nik; Maple, Carsten 2010. A high-level semiotic trust agent scoring model for collaborative virtual organisations. 2010 IEEE 24th International Conference on Advanced Information Networking and Applications Workshops. 534 Kristian Bankov in the “inferential walks” in text interpretation and the “inferential walks” in real life, when we are faced with a choice among different hypotheses of action. An influential method for studying trust is one that sees it as the essence of heuristics for decision-making and judgment (Lewicki, Brinsfield 2011, 2015). Here the concept of the frame is at the heart of the theoretical model and may be seen to introduce the theory of trust in purely semiotic terms: “Framing is about making sense of a complex reality and defining it in terms that are meaningful to us” (Lewicki, Brinsfield 2011: 116). Minsky is a common reference for both approaches. Trust is a kind of shortcut in the process of giving meaning to the reality around us and making it practically possible. “People are not accustomed to thinking hard, and are often content to trust a plausible judgment that comes to mind” (Kahneman 2003: 1450). The same thing happens in the mind of a text reader that is left to the delight of the narrative, rather than reading like a detective or an experimental scientist looking for logical inconsistencies and factual errors that can often be found in fascinating stories. Research into trust shows that in real “out-of-text” life, for purely practical reasons people are forced to suspend their disbelief in some weak form just as they are when reading fiction. Concerning textual interpretation, Eco (1994: 17) speaks of intertextual frames which culture and textual experience create to facilitate text’s semantic functioning. In “real life” another kind of frames that are ready-made as well, but derive from a different kind of individual and social experience, build the meaning. However, at this point the main difference between the two semiotic processes appears. A closer look at the most widely accepted definitions of the concept of trust reveals something; e.g. Lewicki and Brinsfield (2011: 125) quote Rousseau: “Trust is a psychological state comprising the intention to accept vulnerability based on positive expectations of the intentions or behavior of another” (my emphasis, K. B.). Vulnerability also appears in many other definitions of trust (Lewicki, Brinsfield 2011: 125). However, it is something that is completely missing both in the interpretation of the text from the point of view of textual pragmatics and in the immanentist approach of the generative model. The text is by definition a fixed discourse, given once and for all, and no matter how unusual the inferential walks made by the reader are, he/she risks absolutely nothing given that he/she is ultimately served an identical textual manifestation with an identical structure. There may be a description of trust in the text as a narrative structure or modality, but there is no such psychological state. Vulnerability can only happen in a situation of interactive communication or action. It is one thing to watch with bated breath how James Bond plays with the fate of humanity in a poker game, but another to play poker ourselves and to win or lose a certain amount of money on our choice of hand. There is no trust in the text, because in the semiotic mechanism of its Face and trust: A semiotic inquiry into influencers, money, and amygdala 535 semantic actualization there is no instance of decision-making. What we see in the text from a perspective of textual pragmatics is trust by analogy with the extratextual condition of trust. In other words, it is a simulacrum of trust. Accordingly, the theoretical modelling of the existential state of trust in textualism is doomed (see Bankov 2018, 2022). Face, primordial trustworthiness and self-fulfilling prophesy Let us now turn to the field of neuroscience in order to see what ideas from there can enrich our semiotic view of the face–trust relationship. The findings are very encouraging. A fundamental discovery was made in the early 1990s: the AmericanPortuguese neuroscientist Antonio Damasio, when researching patients with brain injuries, located quite clearly an area in the brain where instant and primordial judgements of trust/distrust after seeing a face take place (see Colins 1998). This is the part called the amygdala in the centre of the brain. The conclusion is that these judgements on a level of instantaneity and primordiality are identical with emotional reactions in every perception. At the same time, these reactions have more important consequences than others, since they are more ethically reflexive than these. At the heart of the so-called first impression, there is a kind of capital of trust with longer-lasting consequences. The team of the Bulgarian-American neuroscientist Alexander Todorov conducted experimental research into Damasio’s perspective on perceived trust, comparing instantaneous judgment with longerlasting impressions, based on the subjects’ deeper familiarity with each other. The results show a strong correlation between the two phases of judgements on trustworthiness, i.e. the validity of the primordial judgment is proved (Todorov, Oosterhof, Baron 2008). Another team of authors changed the research perspective and experimented on the behavioural effects on the carriers of more or less trustworthy faces themselves (Slepian, Ames 2016). The social adaptation of individuals with varying degrees of trustworthiness in their faces creates behavioural strategies which correspond to and internalize such a degree. In other words, if someone is born with a trustworthy face, for example, babyface (Slepian, Ames 2016: 286), then his/her social behaviour, driven by the need for adequacy to the expectations of others, will be such as to justify the credited trust, while untrustworthy faces self-generate social distrust. To explain the phenomenon, the authors use the term “self-fulfilling prophecies” (Slepian, Ames 2016: 283, 287), as it is used in the psychosocial sciences. For us, this means that the trust capital of the trustworthy face as a social construction has a tendency to be polarized in an objective way. 536 Kristian Bankov Since the dawn of time, petty fraud with abuse at the level of primordial prereflexive trust created by “good” faces has taken place, e.g. many tourist have become the subject of petty abuse by local individuals specializing in this business. The interesting thing here is that the strength of the effect of trust created by the face is such that a person repeatedly falls into the same trap time after time. There are many analogies with theatrical performances. Theatre anthropology defines a pre-expressive level of stage presence that is transversal to all traditions and techniques. It is a prerequisite for a good actor to be convincing on stage for the spectator to believe in his/her role. In everyday life, the face and its level of trustworthiness has the same pre-expressive function (Bankov 2003), which makes the same communicative acts achieve their goal if fulfilled by “good” faces, and fail if fulfilled by “bad” faces. In the normal dynamic of social life, a face’s trustworthiness can have a significant impact on life experience, including lifechanging situations (Slepian, Ames 2016: 283). In the case of tourist fraud, the situation is not ordinary, since the contact has no temporal perspective – all we have is a pure form of monetary capitalization of the pre-expressive trustworthiness of the face of the fraudsters. The topic of the relationship between trust and its transformation into monetary value is the subject of the final chapter of the study. Trust, faces, influencers, and economic value: A research direction The approach I propose for the semiotic conceptualization of the relationship between the face and trust in the digital age is just a research direction for a potentially large and challenging research programme. As we shall see shortly, the issue hits the core of a megatrend in global socio-economy, which is why my approach begins from the extremely insightful definition of money which opens a completely new perspective in the modelling of digital culture. This definition states that money is “trust inscribed” (Ferguson 2008: 30). Over the last four to five years, I have devoted a great deal of research to this type of sign to develop a holistic semiotic perspective in which many of the age-old boundaries between money and non-money have become blurred. In this logic, for example, the explanation for the incredible success of bitcoins as a currency boils down to two fundamental things: trust and adoption (Bankov 2022: 82). The Internet has made possible the emergence of a wide variety of monetary forms, most of which aim at autonomy from the central banking system. This has given rise to time banking, video games currencies, and many other creative forms of measuring and exchanging value through various online platforms. I believe that Face and trust: A semiotic inquiry into influencers, money, and amygdala 537 the face with its capacity to be a bearer of primordial and pre-reflexive trust can take a worthy place within this sign logic. It is no coincidence that this article illustrates my thoughts on the semiotic relation of the face and trust with the figure of the influencer. Certainly, this is not the only manifestation of the relationship studied, but the influencer is an emanation of the cultural changes in the digital age, especially from the phase of social media (Web 2.0). This is a megatrend: the agency Upfluence lists four million professional influencers, and this is the result of only a few years of growth in this field.6 We must also take into account all the millions of users who aspire to this status, but have not yet found a way to achieve it, as well as the millions who are de facto influencers, but have not turned it into a profession. It would be no exaggeration to say that the face is the semiotic device that creates the preconditions for the influence that each influencer has on his/her followers, regardless of the field of interest. When a team from the online magazine Papermag. com dedicated research to unravelling the secret of the success of influencers, they focused their efforts on the influencers’ faces. The result of this study is a synthesized image of the faces of the 100 most influential Instagram influencers. Undoubtedly, this face (Fig. 2) well illustrates the hypotheses of neuroscientists about face trustworthiness, although it was achieved by a different methodology. Another scientific study confirms that the key to success is not the attractiveness of the face, but its ability to inspire trust and intimacy (Marôpo, Jorge, Tomaz 2019). Figure 2. The Ideal Influencer – synthetic face of the 100 top Instagram influencers. (Credits to PAPER Magazine/ PAPERMAG.com.) 6 See https://www.upfluence.com/influencer-profiles (last accessed on 31 January 2021). 538 Kristian Bankov There are influencers who never show their faces. Some techno geeks never appear on the videos, others generate their influence by collecting images or videos of other people and subjects, etc. I could not find reliable statistics in regard to this, but my assumption is that these are minor cases, and the common mainstream influencer is visualized well in front of the camera and speaks to her/his followers. Other important point is that having a trustworthy face is not enough to become an influencer automatically. My firm belief is that a trustworthy face is a tangible and objectively measurable asset; one that gives indisputable advantages to the owner and creates difficulties for those who lack it. The conditions for the increased value of trustworthy faces are rooted in one of the main consequences of the culture of social media – the total crisis of trust. Some summarize this situation as post-truth, others as post-factual democracy. “Post-truth” even became Word of the Year 2016 of the Oxford Dictionary.7 In this new social fabric, dominated by emotional factors, the face has a privileged status. I began this study with a long list of technological techniques for counterfeiting the face, but this is part of the overall ontological status of an online culture in which anything can potentially be counterfeited. “Trust is the new scarcity” claims one of the authors, trying to grasp today’s essentials.8 We cannot but think of the times of the gold standard when the scarcity of gold was the main source of trust in the precious metal, necessary for the global financial system to stay stable. With its primordial and prereflexive potential to establish a relation of trust, in today’s context the trustworthy face is the new gold. The success of the influencers’ business is evidence that, notwithstanding the well-known practices of falsifying its digital appearance, the face’s ontological statute is still based on the relation with an agency of trust. Trustworthy face as gold The face is the semiotic universe par excellence, where the equilibrium between nature and culture feeds the social fabric of every civilization in the most sublime way. Anatomically the same, the face has inspired a huge variety of myths, rituals, representations, businesses, etc. in different cultures (Leone 2019: 19–20). The digital globalization of the world is attributing a new, more unified, but still fundamental role to the face, where trust is the scarcest resource of the social fabric. We see this clearly with the crucial role of the face in the megatrend of the influencers’ profession. 7 See https://languages.oup.com/word-of-the-year/2016/; last accessed on 31 January 2021. McCullen, Aidan 2019. Why we need public broadcasters – Trust is the new scarcity. Medium, 7 March 2019, is available at https://medium.com/thethursdaythought/why-we-need-publicbroadcasters-trust-is-the-new-scarcity-65c32577e1cd; last accessed on 31 January 2021. 8 Face and trust: A semiotic inquiry into influencers, money, and amygdala 539 The face of the professional influencer is monetized trust. In the same way, the money sign is trust inscribed. Faces are the new species of coins of the internet where their value is measured and expressed by the trustworthiness, inscribed on their “heads and tails”. The extended neuroscientific research on the trustworthiness of the face suggests that there are objective methods of evaluating it. With the help of semiotics we may start work on a tool designed to elaborate a semiotic trust profile of the face, similar to the evaluation of the quality of gold before the coinage, by analogy with the existing tools for business organizations. Conclusion This is the first research paper which brings together face, trust and semiotics. As pairs, all three combinations – semiotics and face, semiotics and trust and face and trust – are well explored. My main contribution would be the opening of a new research direction, rich in theoretical potential and in operative application on a socio-economic megatrend in digital culture: namely studying the sphere of social media influencers. This is not the only cultural reality, envisaged by the research direction, but I think it is the most relevant one to start with. It is one which, after a certain number of contributions and after establishing a basic vocabulary, could be extended to other occurrences of the face in e-technological societies. One of the major claims of the paper is that there can be no other approach to such complex object as the semiotics–face–trust interplay than an interdisciplinary one. Interdisciplinarity would include the economic logic of online communications, especially working on the differences between linear textual actualization and interactivity. The methodological perfection of the generative approach of the Paris School of semiotics is one of the major obstacles to opening the semiotic theory to interactive communication. The first step is the elaboration of a new model of the money sign, seen as trust inscribed, which would allow the semiotic modelling of digital interactivity. At the same time, the advent of social media (Web 2.0) has brought about the full extent of the post-truth era. Trust has become one of the scarcest resources for our social fabric. In such a context, it has become obvious that appearance dominates over substance and face trustworthiness is increasingly playing a decisive role in the success of the new-economy ventures, consisting mainly of services and access to experiences. Trustworthy faces work like valuable currency in the trust-scarce economic context, comparable to the role of gold as a precious and scarce material for the finances during the gold standard. In support of such a parallel comes the evidence from neuro-sciences. It has been found that the 540 Kristian Bankov trustworthiness of the face works on a primordial and pre-reflexive level, ascribing to it an absolute value in the establishment of the phatic function in online interaction. References Alexandrou, Alex; Maras, Marie-Helen 2018. Determining authenticity of video evidence in the age of artificial intelligence and in the wake of Deepfake videos. The International Journal of Evidence & Proof 23(3): 255–262. https://doi.org/10.1177/1365712718807226 Bacharach, Michael O. L.; Gambetta, Diego 2001. Trust in signs. In: Cook, Karen S. (ed.), Trust in Society. New York: Russell Sage Foundation, 148–184. Bankov, Kristian 2003. Face and understanding. In: Tarasti, Eero (ed.), Understanding/ Misunderstanding: Contributions to the Study on the Hermeneutics of Signs. (Acta Semiotica Fennica XVI.) Helsinki: International Semiotics Institute, 192–200. Bankov, Kristian 2018. From text to interaction. Digital Age in Semiotics and Communication 1: 9–17. https://doi.org/10.33919/dasc.18.1.1 Bankov, Kristian 2022. The Digital Mind: Semiotic Explorations in Digital Culture. London, Cham: Springer Nature. Barthes, Roland 1981. Camera Lucida: Reflections on Photography. New York: Hill and Wang. Biglari, Amir 2011. Sémiotique et Pragmatique des Passions dans Les Contemplations de Victor Hugo. Lille thèses. Université de Limoges, Faculté des lettres et sciences humaines. Bikoula, Ignace Gustave 2011. La valutazione e la prevenzione del rischio sistemico con gli strumenti della network alaysis. PhD dissertation. La Sapienza University of Rome, Faculty of Economics. Colins, Peter 1998. Faces, trust and the amygdala. Trends in Cognitive Sciences 2(7): 238. https://doi.org/10.1016/S1364-6613(98)80016-8 Eco, Umberto 1994. The Role of the Reader: Explorations in the Semiotics of Texts. Bloomington: Indiana University Press. Eco, Umberto 2000. Kant and the Platypus. London: Vintage. Ferguson, Niall 2008. The Ascent of Money: A Financial History of the World. New York: Penguin. Greimas, Algirdas Julien; Fontanille, Jacques 1991. Sémiotique des Passions: Des États de Choses aux États d’Âmes. Paris: Seuil. Kahneman, Daniel 2003. Maps of bounded rationality: Psychology for behavioral economics. American Economic Review 93(5): 1449–1475. https://doi.org/10.1257/ 000282803322655392 Le Gall, Véronique; Langley, Ann 2015. An abductive approach to investigating trust development in strategic alliances. In: Fergus, Lyon; Möllering, Guido; Saunders, Mark N. K. (eds.), Handbook of Research Methods on Trust. Cheltenham, Northampton: Edward Elgar Publishing, 36–45. https://doi.org/10.4337/9781782547419.00012 Leone, Massimo 2019. The semiotics of the face in digital dating: A research direction. Digital Age in Semiotics and Communication 2: 18–40. https://doi.org/10.33919/ dasc.19.2.2 Face and trust: A semiotic inquiry into influencers, money, and amygdala 541 Lewicki, Roy J.; Brinsfield, Chad 2011. Framing trust: Trust as a heuristic. In: Donohue, William Anthony; Rogan, Randall G.; Kaufman, Sanda (eds.), Framing Matters: Perspectives on Negotiation Research and Practice in Communication. New York: Peter Lang Publishing, 110–135. Lewicki, Roy J.; Brinsfield, Chad 2015. Trust research: Measuring trust beliefs and behaviours. In: Fergus, Lyon; Möllering, Guido; Saunders, Mark N. K. (eds.), Handbook of Research Methods on Trust. Cheltenham, Northampton: Edward Elgar Publishing, 46–64. https://doi.org/10.4337/9781782547419.00013 Marôpo, Lidia; Jorge, Ana; Tomaz, Renata 2019. “I felt like I was really talking to you!”: Intimacy and trust among teen vloggers and followers in Portugal and Brazil. Journal of Children and Media 14: 22–37. https://doi.org/10.1080/17482798.2019.1699589 Meyerson, Debra; Weick, Karl E.; Kramer, Roderick M. 1996. Swift trust and temporary groups. In: Kramer, Roderick M.; Tyler, Tom R. (eds.), Trust in Organizations. Thousand Oaks: Sage Publications, 166–195. https://doi.org/10.4135/9781452243610.n9 Pope, Michael 2018. Peircean faith: Perception, trust, and religious belief in the conduct of life. Transactions of the Charles S. Peirce Society 54(4): 457–482. https://doi.org/10.2979/ trancharpeirsoc.54.4.02 Shen, Cuihua; Kasra, Mona; Pan, Wenjing; Bassett, Grace A.; Malloch, Yining; O’Brian, James F. 2019. Fake images: The effects of source, intermediary, and digital media literacy on contextual assessment of image credibility online. New Media & Society 21(2): 438–463. https://doi.org/10.1177/1461444818799526 Slepian, Michael L.; Ames, Daniel R. 2016. Internalized impressions: The link between apparent facial trustworthiness and deceptive behavior is mediated by targets’ expectations of how they will be judged. Psychological Science 27: 282–288. https://doi.org/ 10.1177/0956797615594897 Swedberg, Richard 2010. The structure of confidence and the collapse of Lehman Brothers. Research in the Sociology of Organizations 30: 71–114. https://doi.org/10.1108/S0733558X(2010)000030A007 Swedberg, Richard 2012. The role of confidence in finance. In: Cetina, Karin Knorr; Preda, Ales (eds.), Handbook of the Sociology of Finance. New York: Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199590162.013.0028 Todorov, Alexander; Oosterhof, Nikolaas N.; Baron, Sean G. 2008. Evaluating face trustworthiness: A model-based approach. Social Cognitive and Affective Neuroscience 3: 119–127. https://doi.org/10.1093/scan/nsn009 Лицо и доверие: семиотическое исследование инфлюенсеров, денег и миндалевидного тела В результате культурного взрыва Web 2.0 цифровая культура породила семиотический парадокс: с одной стороны, изображения лиц используются невероятно широко, с другой стороны, поводов усомниться в их подлинности всё больше. Это связано с тем, что графические технологии сделали сложные манипуляции с изображениями возможными, а также значительно их упростили. После обзора существующих семиотических моделей доверия я предлагаю новый подход, подчеркивающий ценностные свойства доверия по аналогии с денежным знаком, который по сути 542 Kristian Bankov является «написанным доверием». Когнитивные исследования подтверждают гипотезу о том, что достоверность лиц оценивается нами неосознанно. Это означает, что заслуживающее доверия лицо является предпосылкой для более успешного общения, чем не заслуживающее доверия, независимо от темы разговора или культурного контекста. Пример влияния инфлюенсеров показывает, что в глобализирующейся культуре, в которой доминирует интернет, заслуживающие доверия лица обладают многосторонним коммуникативным преимуществом. Nägu ja usaldus: semiootiline sissevaade suunamudijatesse, rahasse ja mandeltuuma Pärast Web 2.0 kultuuriplahvatust on digikultuuris avaldunud ilmselt semiootline paradoks, mis on seotud nägude uskumatult laialt levinud kasutamisega, samas kui põhjus nende nägude autentsuse usaldamiseks on pidevalt kahanemas. Seda seetõttu, et graafiline tehnoloogia on muutnud keerukad pildimanipulatsioonid niihästi võimalikeks kui ka kergesti teostatavateks. Alustan ülevaatega usalduse olemasolevatest semiootilistest mudelitest ja sellega seotud kaalutlustest ning pakun seejärel välja uue lähenemise, mis rõhutab usaluse väärtustgenereerivaid omadusi analoogia põhjal raha märgiga, mida peetakse “kirja pandud usalduseks”. Uurimistöö neuroteaduste valdkonnas toetab hüpoteesi, et näo usaldusväärsuse üle otsustatakse mõtlemiseelselt ja ürgselt. Nii tähendab see, et usaldusväärne nägu on eeldus edukamaks suhtlemiseks kui ebausaldusväärne, sõltumata aruteluobjektist ja kultuurikontekstist. Ühismeedia suunamudijaid puudutav näide demonstreerib, et globaliseeruvas kultuuris, kus domineerib internet, on usaldusväärne nägu mitmekülgseks eeliseks, millel on tuntav mõju. Gotta face ‘em all543–565 543 Sign Systems Studies 49(3/4), 2021, Gotta face ‘em all: Pokémon, Japanese animated characters, and the emergence of playful visual animism Vincenzo Idone Cassone1 Abstract. As a result of technological innovations and new cultural practices, the contemporary mediasphere is increasingly populated by digital(ized) faces. The phenomenon is not limited to human faces, but includes a vast universe of fictional animated faces, variously called ‘characters’, ‘mascots’ or ‘kyara’. In Japan, while certainly not new, kyara have been spreading thanks to globalization, digitalization and media-mix strategies. Through the connection between visual design, fictional narratives and socio-cultural consumption, kyara can be considered semiotic figures of in-betweenness, key symbolic mediators in the Japanese mediascape. Their anthropomorphic face design mediates the cultural boundaries between the human and the non-human, the animate and the inanimate, nature and culture. Furthermore, their post-modern narratives mix inspiration from the past and the present, from myths to science fiction. Lastly, they involve an encyclopedic reworking between fiction and reality, mythical references and secularization, between the domains of seriousness and playful make-believe. The article aims to explore the semiotic dimensions of kyara in contemporary Japan, with emphasis on their logic of representation and cultural outcomes. These will be investigated through the analysis of the Pokémon franchise, which will make it possible to describe the emergence of new semiotic patterns of ‘playful visual animism’ in the process of media facialization of everyday life. Keywords: Pokémon; playfulness; animism; heterotopy; faces; anthropomorphism; fan theory; transmedia 1. Introduction The contemporary mediascape (Appadurai 1990) is increasingly populated by digital(ized) faces, combining an unprecedented quantity and quality of images, new forms of interaction and the emergence of face-based cultural practices and beliefs. This can be attributed to a mix of recent technological innovations (digital photography, visual filters, software for automatic face recognition), the diffusion of novel genres of face representation (the selfie, animojis, memes) and new behaviours/ 1 Department of Humanities, University of Turin, Italy; e-mail: idonecassone@gmail.com. https://doi.org/10.12697/SSS.2021.49.3-4.15 544 Vincenzo Idone Cassone practices (swapping faces on apps for dating, using masks or occultation for political activism, covering faces for healthcare during pandemics) (Leone 2018). While the vast majority of research studies have been focusing on human faces, the “facialization” of media cultures also includes the parallel pervasiveness of animated anthropomorphic faces. Animated characters and mascots are certainly not new, but have been proliferating in the last thirty years due to the combination of media globalization, digitalization and market strategies (Bendazzi 2016). As a result, the current global mediascape is filled with anthropomorphic characters such as Mickey Mouse, Hello Kitty or Pokémon (from transmedia franchises), Tony the Tiger, Michelin Man or the Linux penguin (representing corporate or brand identity), not to mention historic sport mascots or recent emoji-based characters (Tampa Bay’s Raymond, Pusheen the cat, etc). Their presence is even more significant in Japan, where they have been at the heart of media-mix strategies (Steinberg 2012) and media ecology dynamics (Ōtsuka 1989) since after WWII. The Japanese mediascape is filled with many of the above-mentioned anthropomorphic characters, often labelled ‘kyara’2, not limited to child-targeted TV shows or toys. Kyara are part of daily brand merchandizing (Han 2017), and are used extensively for public and social advertising (Alt, Yoda 2007) as well as to embody local places and tourist attractions (Barrows 2014), and feature in parades and social events (Occhi 2012). The importance of such characters in Japanese culture, however, does not simply lie in their pervasiveness. Their presence has been linked to a process of characterization in Japanese everyday life, in which kyara become “creatures of inbetweenness” (Nozawa 2013)3: semiotic personae whose existence is not limited to specific media formats, visual styles or fictive semiospheres, but appears as fragmentary, pervasive and heterogeneous daily experiences of “characterencounters”. Thus, they can be assimilated as key symbolic mediators (Lotman 1990: 136–137) in the Japanese semiosphere: figures which enable communication between the inner and outer layers of the cultural self-description systems (identity–alterity). In a similar way to their human correlates, the presence of and interaction with animated characters could be linked to the inception of new semiotic codes/models of behaviour (Lotman 1990: 233–234). More specifically, as 2 The Japanese term ‘kyara’ was born as an abbreviation of the loan word ‘kyarakyutā’, but now stands mostly for “highly-stylised or simplified visual figuration that can be easily reproduced and consumed outside of its original context” – see Galbraith 2009: 125; Galbraith, Patrick W. 2009. Moe: Exploring virtual potential in post-millennial Japan. Electronic Journal of Contemporary Japanese Studies 9(3) is available at https://www.japanesestudies.org.uk/ articles/2009/Galbraith.html. 3 Nozawa, Shunsuke 2013. Characterization. Semiotic Review 3 is available at: https://www. semioticreview.com/ojs/index.php/sr/article/view/16/67. Gotta face ‘em all 545 I suggest in this paper, contemporary kyara could be related to semiotic patterns of playful animism in the Japanese semiosphere, which result from the interaction between the visual features of these characters, their roles and function in the mediascape, and the encyclopaedic interpretive behaviours of users. The aim of this contribution is to describe the patterns of playful animism in the Japanese semiosphere, discussing them in relation to: the specific codes of visual representation of animated characters, the encyclopaedic logics of anthropomorphism, the interpretive process involved in interacting with fictional narratives, and the contemporary dynamics of media consumption. This objective will be pursued by using the global franchise of Pokémon as a case study. Pokémon (haplology of ‘pocket monster’, ‘poketto monsutā’) is a well-known transmedia franchise, created in Japan by Satoshi Tajiri, Ken Sugimori and Junichi Masuda in the mid-1990s (1996) as a videogame for the Game Boy handheld console. The sudden success of the game in Japan led to the development of a media mix strategy,4 with the realization of an anime series (1997) which was quickly translated and exported into the USA (1998) and Europe (1999), fuelling the production of merchandise (collectibles, plushies, clothing, etc.) and subsequent development of movies, comics, other games, as well as theme parks, official parades, cross-branding and so on. Although many have predicted its decline over the years (Tobin et al. 2004), the franchise is still among the most widespread and highest-grossing in the world, with a further boost provided by the smartphone ARG Pokémon GO (2016) and the latest videogame entries Pokémon Sword/Shield (2019). The Pokémon franchise proves to serve as a fitting case study for the understanding of the peculiar semiotic features of animated anthropomorphic characters for several reasons. First, it is a perfect embodiment of the contemporary changes in Japanese transmedia products and codes of narrative consumption, being an example of Steinberg’s (2015) gameic media mix,5 as well as a narrative transmedia interactive ecosystem (Ōtsuka 1989). Second, it can be considered a paradigmatic example of Japanese culture while simultaneously being a globalized phenomenon, a syncretic mix of contemporary post-modern narratives and folkloric heritage (Iwabuchi 2002; Hutchinson 2019; Allison 2006). Lastly, as I suggest in this paper, the franchise could be read in light of Azuma’s (2009) reflections on the otaku subculture as “the site that most sensitively registers social transformations” (Lamarre et al 2007: 158) in contemporary Japan. 4 ‘Media mix’ is a term initially used in Japan to indicate the commercial strategy of diversification of a brand through different products and media; the term has been diffused in the West by Steinberg (2012), who specifically discusses the anime-based media mix created around Astro Boy (Tetsuwan Atomu) in the 1960s. 5 A transmedia strategy centred on games and not anime, as for the anime media mix. 546 Vincenzo Idone Cassone 2. Visually iconic, taxonomically hybrid: The design of Pokémon Pokémon are among the most iconic contemporary animated characters in the world. In Japan, due to the success of the franchise and the extensive crossbranding and merchandise promoted by its media-mix strategy, Pokémon faces can constantly be seen on TV, in magazines, and in street advertisements; they are incorporated into clothes, food, vending machines, and even periodically feature in costume parades and events on the streets. The franchise mascot, Pikachu, as well as the most famous Pokémon of each generation, could be defined as proper contemporary cultural icons. Despite the many differences in media formats (manga, anime, games), styles, illustrators, and the technological evolution (from 2D to 3D) of the games, the characters of the franchise are widely identifiable and visually distinct6. However, while it can be said that the visual features of Pokémon are generally recognizable and established in the Japanese cultural encyclopaedia, a different question arises in relation to their “taxonomy”, or encyclopaedic categorization. More specifically, I suggest that while the semiotic competence linked to Pokémon recognition is relatively diffuse, it is somewhat difficult for model readers/users to explain and define what Pokémon actually are, according to typical cultural taxonomies. In order to describe the zoomorphic features of the Pokémon themselves, Surace (2017) fittingly refers to them as ‘animaloids’. While Pokémon are depicted as living and sentient creatures, their morphology largely oversteps the boundaries (both inner and outer) of the animal world. The design of the 898 existing Pokémon includes almost every known form from the animal kingdom (mammals, fishes, insects, gastropods, sponges, etc.) and inspiration from other kingdoms of life (plants, fungi, even protists). Pokémon design is also derived from non-living natural objects (minerals and rocks, metals, gases) and often goes beyond the boundaries of nature to include the artificial (mechanical Pokémon), as well as paranormal entities (ghosts, spirits, urban legends), extra-terrestrial forms (aliens, beings from parallel dimensions, digital Pokémon, robots) and even legendary or mythical beings (embodiments of nature, supernatural tricksters, fairies, transcendent beings) up to the Pokémon god itself. Furthermore, while 6 Part of the reason is connected to the media-mix nature of the franchise, which has circulated in many formats and media from the start. The appearance of many Pokémon, for instance, has become well known thanks to the widely used illustrations by Ken Sugimori (art director and character designer of Game Freak). For coherency purposes, the pictures on the following pages have been selected from Sugimori’s illustrations, while the analysis has included the sprites, 3D models of the games etc. Gotta face ‘em all 547 many Pokémon display anthropomorphic shapes, many others are clearly based on human artifacts (keys, garbage, ice creams) or figures/representatives of human activities (mimes, martial artists, gurus). In addition to the variety in origin/inspiration, it should be pointed out that the vast majority of Pokémon display a heterogeneous mix of features/properties from different natural kingdoms: the animal, mineral, and vegetal worlds, as well as the overlapping of human, supernatural or daily elements, thus appearing as original chimeric/hybrid creatures (see Fig.1). Figure 1. Left to right, up: Gigalith, Victribell, Magnemite, Duskull, Klefi, Mr. Mime; below: Slowbro, Dhelmise, Propobass, Exegutorr, Dracovish. (Copyright of Nintendo/The Pokémon Company/GameFreak, under Fair Use.) It is indeed difficult to find a universally shared physical or biological feature common to all Pokémon, which would make them fit perfectly into a proper encyclopaedic categorization: from this perspective, Pokémon remind of Umberto Eco’s reflections on the platypus as an object of experience which subverts the existing cultural segmentations and taxonomic codes of its viewers (Eco 2000) in a similar way to the cultural taxonomization of animals themselves (Maran, Martinelli, Turovski 2011), by involving semiotic issues which are linked to the coherency and structure of cultural self-descriptions (Madisson 2016), both real and fantastic, as showed by Borges’ examples of the Chinese encyclopedia, and Marco Polo’s description of unicorns (Eco 2000: 57–58). What stands out in the case of Pokémon is, once again, their semiotic nature as animated beings, in a double meaning: as visual products of animation, and as 548 Vincenzo Idone Cassone characters which manifest animated traits. Etymologically, the deep connection between the appearance/effect of dynamism and the inherent assumption of agency and life which many cultures ascribe to the former can be expressed by the term ‘animaloid’. The hybrid and heterogeneous design of Pokémon also extends to the dynamics of anthropomorphism7 which is common, yet rarely investigated, in animation (Wells 2009: 2): on the one hand, the franchise establishes a clear and immediate visual and diegetic difference between humans and Pokémon (even if categorization provided in the diegetic universe turns out to be contradictory and complex); on the other hand, typical visual techniques of animation, employed in forms of character anthropomorphization, are applied to Pokémon, even if in heterogeneous ways. An example (albeit partial) can be provided by looking at the visual representation of faces and gazes in the franchise: their anthropomorphization is a common trope of animation, associated with the possibility of providing animals with human-like communicative and emotional expression abilities (Parkinson 2020: 43–50; Wells 2009: 29–-32). First of all, the vast majority of Pokémon (approximately 888 out of 898) can easily be interpreted as having a face, in terms of the presence of recognizable eyes/nose structure, associated with the communicative and emotional expression ability (Bruce, Young 2012: 253–260). Those lacking a visage are often diegetically regarded as exceptions (see Fig.2): Figure 2. Pokémon with no face. Left to right: Starmie, Roggenrola, Registeel, Lileep, Stakataka. 7 In a semiotic perspective, anthropomorphism can be considered the adoption or interpretation of certain elements, features and affordances of characters as typically human traits (e.g. wearing clothes), or as human variants of more general traits of living beings (e.g. group behaviour, eating). In this perspective, anthropomorphism is always a perspectival dynamic, a result of broader cultural self-descriptions and encyclopaedic structures, which pre-establish the human-animal dichotomy in certain semiospheres (see Bertrand, Marrone 2019). Gotta face ‘em all 549 In relation to the eyes and gazes of Pokémon, we can see a heterogeneous set of visual forms, ranging from a relatively anthropomorphic gaze (with visible white sclerae8), to cartoon expressive gazes (conveying the attitudes or emotions of the characters), eyes inspired by the biological structures of mammals, insects, etc., or even certain forms of supernatural eyes (flames, gems, etc). Despite this, the franchise tends to follow the traditional norms of cartoon animation (especially Japanese animation), by making eyes and gazes, as well as faces, fundamental communicative interfaces for meaning-making among living beings, including animated ones (Leone 2021); furthermore, as a typical trope of animation, both the anthropomorphic and the expressive gaze are often used for humans and Pokémon alike. This is evident in all the manga, anime and games since the second generation: the same visual expressive traits are used as the basis for certain Pokémon and human characters, or to express the same temporary attitude (see Fig. 3). Figure 3. Up: different types of eyes in Pokémon; down: humans and Pokémon with similar gazes. This example may support the initial assessment that the Pokémon universe seems to be characterized by the interaction of two simultaneous dynamics: on the one hand, the dynamic of visual animated-ness; on the other hand, a perceived 8 The presence of visible white sclerae in humans is considered an evolutionary trait (Kobayashi, Konshima 2001) typical of humans and certain apes, often interpreted and ascribed by cultures as a sign of humanity. 550 Vincenzo Idone Cassone taxonomic heterotopy. The former uses typical styles and techniques of animation (even if in heterogeneous ways) to ensure the recognition of Pokémon and humans alike as living, sentient creatures, sharing the communicative and expressive function of the face (Leone 2009, 2021); in case of the latter, the hybrid design of Pokémon does not simply deviate from, but actually challenges the interpreters’ cultural codes of taxonomy, undermining their semiotic codes of segmentation between living and non-living, different kingdoms of nature, and the very perceived difference between the human and the animal, the cultural and the natural. 3. Pokémon heterotopic regime of animation A likely critique of the idea introduced above is that the ways in which living beings are represented in Pokémon do not deviate from the standards of animation, more specifically those with fantasy settings and targeted at children/young adults. Animation typically involves anthropomorphizing of living and non-living beings, with magic- and fantasy-based narratives whose potential incoherencies are likely to trigger typical suspension of disbelief dynamics in spectators. According to Lamarre (2018), since the 1930s, the reflection on animation has often been linked to an implicit animistic attitude, it being the explicit representation of many natural phenomena as animated and provided with agency, as part of a general tendency to anthropomorphize animal and natural phenomena. However, as discussed by Wells (2009: 51–59), the representation and interpretation of animal characters as correlated with or reflecting humans is not completely homogeneous, with many examples of similar forms of deviation from traditional taxonomies in the representation of animals and humans in other globalized franchises. Figure 4. Examples from the franchises Hello Kitty, Mickey Mouse short movies, and Digimon. Gotta face ‘em all 551 The various franchises (see Fig. 4) present similar abstract forms of representation of eyes and gazes, all sharing an implicit understanding of the face as the sign for the communicative and agentive capacity of “superior” living beings. What would thus be different or relevant about Pokémon? The answer can be found by comparatively analysing the ways in which the visual codes of representation interact with the diegetic codes and narrative dynamics of the different universes. This can be done according to two main dimensions of analysis: - first, internal representational coherence, looking for ways in which the visual codes of representation confirm, deviate from or overlap the spectators’ encyclopaedic taxonomies; - second, visual/diegetic coherence, looking for ways in which the narratives reinforce, ignore or contradict the visual logic of representation and the spectators’ taxonomies. As concerns internal representational coherence, in many cases in traditional animation, despite the simultaneous presence of various degrees of anthropomorphic and zoomorphic characters, there are clear visual indicators (sign-index) which act as discriminators to ensure the taxonomic separation between the different classes. In Hello Kitty, for instance, only the human-like characters have clothes and the property of language, while Charmmy Kitty the pet has neither. Despite the clear sharing of animal visual traits, the cultural segmentation between humans and animals is conveyed through different visual sub-codes. This also happens in the modern (post-1950s) narratives of Mickey Mouse, in which the anthropomorphic characters share a common language and possess clothes (albeit in partially incoherent ways). On the other hand, in the early Mickey Mouse short movies (up to the 1930s), as well as in other franchises such as Winnie the Pooh, Digimon and Yu-gi-oh, there is no coherent set of visual cues to ensure the stability of the spectators/ readers’ encyclopaedic taxonomies described by Eco (1975, 1979). As regards clothing, there is the simultaneous presence of anthropomorphic animals with no clothes and others that are partially or fully clothed, although there is coherence in anthropomorphic behaviour and property of language. From this perspective, Pokémon is closer to the latter group, with the notable addition that the degree of its hybridism is far more pronounced than in traditional Western animation. As concerns visual/diegetic coherence, Pokémon distances itself from the others in relation to the coherence between the visual and diegetic codes of representation. Most of the above-mentioned franchises interact with possible incoherence in internal representation either by ignoring it or by normalizing it through the diegetic dimension of the narratives, in an attempt to facilitate the interpreter’s suspension of disbelief or to portray it in a comedic light. On the 552 Vincenzo Idone Cassone contrary, the anthropomorphism in Pokémon is both internally incoherent (with many Pokémon assuming various degrees of human traits, or being based on humans) and diegetically exposed. While certain Pokémon may raise fewer issues for the interpreter’s cultural encyclopedia, others do so to various degrees, with the further fact that incoherencies are highlighted within the franchise itself, mainly through the descriptions in the PokéDex (the diegetic encyclopedia of the Pokémon Universe), as well as by meta-referential elements. The most notable examples are: - Behaviour: the ethologic descriptions in the PokéDex often focus on their physical attributes, instincts or basic behaviour, yet many Pokémon possess specific behaviours which duplicate or resemble cultural ones attributed to humans. Thus, Frostlass (#478) freezes the travellers who reach its cave, then uses them as decoration for its home; Conkeldurr (#534) invented, and taught humans how to make, concrete; Aggron (#306) reclaims its territory after natural disasters, replanting trees and hauling topsoil. Some of these features are particularly incoherent because they are attributed to a species, while apparently describing unique characters (#778 Mimikyu’s disguise, as well as Cubone’s skull). - Clothing: while certain Pokémon are naked (or clothed by their masters), others include recognisable clothes in their design, from belts to ribbons to gi and pants. Some of these items are explicitly stated not to be clothes, but parts of the Pokémon body (such as #068 Machamp’s champion’s belt and briefs and #560 Scrafty’s loose trousers), while other Pokémon use clothes with no explanation (#814 Raboot’s hoodie, #743 Ribombee’s scarf); some Pokémon even sew them (#542 Leavanny, when it encounters baby Pokémon); often, for many Pokémon no specifications are provided (#539 Sawk’s gi, #107 Hitmonchan’s boxing gloves). - Food: in a world with no animals aside from the domestic/tamed Pokémon, the representation of food easily leads to diegetic inconsistencies. The games and anime openly mention and represent meat-like and fish-like dishes, with no elucidation. Simultaneously, sometimes the franchise clearly states that Pokémon are used as food (Chansey’s eggs, Miltank’s milk, Slowpoke’s tails, as well as whole Farfetch’d, Cherubi, Sharpedo): the latest games in the series, Pokémon Sword/Shield, include the possibility of creating curry with what appear to be recognisable Pokémon parts (tails) or meat-like elements (hot dogs, meatballs). - Death: while Pokémon cannot die directly in the games (they simply “faint”, once defeated) and almost never die “onstage” in the anime/manga, death has been mentioned since the very first games, with a Pokémon graveyard and numerous examples of deaths from old age or by violence, many of which originate ghost Gotta face ‘em all 553 Pokémon. A lot of the recent entries of the PokéDex discuss the violent and bitter deaths of humans/Pokémon which have made them become ghosts. - Pokémon/human differences: besides humanoid Pokémon, there are problematic insights into the relationship between Pokémon and humans. In the fourth generation games, a fictional textbook in the Canalave Library reports that “once, humans and Pokémon married […] they were the same” (Game Freak 2006). In the same games, the Pokémon God Arceus is stated to have created the universe, as well as Pokémon and humans. In the fifth generation games, the main antagonist (N) grew up among the Pokémon and is able to talk with them. Furthermore, the PokéDex presents many cases of permanent transformations from human to Pokémon, for instance Kadabra (#64), Phantump (#708) and especially Yamask (#562): once human, now Pokémon, the latter carries its own past face as a mask, in order to find people who may remember it. - Comedic/serious attitude: while the Pokémon franchise mostly follows the comedic attitude and its traditional targets are children and young adults, there are many sudden variations in tone, for instance in the city of Lavender in the first game, as well as certain other dark/bitter/unsettling PokéDex entries: Pokémon killing people by mistake (#760 Bewear), dying in pain because of their masters (Mega evolutions), abducting children (#425 Driftloom), presenting fatal biological conditions (#325 Spoink) or being damned/ extinguished due to climate change (#864 Cursola). In conclusion, the Pokémon universe manifests a higher degree of visual and diegetic incoherency in comparison to other contemporary or traditional animated franchises: its explicit taxonomic heterotopy, and the dynamic of visual animatedness, are supported and strengthened by internal diegetic and visual incoherencies leading to what I would call a general heterotopic regime of animation. The apparent self-contradictions and incoherencies are not ignored or normalized; instead, they are highlighted and reinforced through the metareferential aspects and contrasting tones of the internal self-description system of the PokéDex. Many of the above-mentioned examples also involve clear cases of ludo-narrative dissonance (Seraphine 2019)9, making gameplay and narratives openly clash during the games. 9 Seraphine, Frederic 2019. The rhetoric of Undertale – Ludonarrative dissonance and symbolism. Digital Games Research Association JAPAN, Proceedings of 8th Conference. 2019. Available at: https://www.researchgate.net/profile/Frederic-Seraphine/publication/323545890_ The_Rhetoric_of_Undertale-Ludonarrative_Dissonance_and_Symbolism/links/ 5a9bdb8545851586a2acea7d/The-Rhetoric-of-Undertale-Ludonarrative-Dissonance-andSymbolism.pdf. 554 Vincenzo Idone Cassone 4. Visual animism and playful interaction A notable example is represented by Pokémon #778 Mimikyu, who embodies both the self-contradictory taxonomy of the franchise and a clear case of its metareferential and self-communicative (Lotman 1990:20) nature. Mimikyu is a ghost/ fairy Pokémon which, as it says in the PokéDex, uses a rag in order to disguise itself as Pikachu and hide its true, terrifying appearance. According to other descriptions, this Pokémon suffered from extreme solitude, and thus decided to take over Pikachu’s mantle in order to make new friends as the yellow mouse was so famous. Mimikyu is a notable case of meta-reference in Pokémon, in which the diegetic universe hints at the artificial nature of the franchise by referring to an “external” world where Pikachu is not just a Pokémon among others, but the franchise mascot, or more precisely, its face. Many long-term fans have appreciated the meta-referential elements of the franchise such as Mimikyu,10 (or the eighth generation fossil Pokémon)11, elaborating many fan theories that try to address the most glaring contradictions in the franchise (below). The success of Pokémon may suggest that its heterotopic nature and declared diegetic self-contradictions have been accepted and have now culturally developed into a proper set of semiotic dynamics, with their own visual codes of representation and diegetic structures, which have a clearly recognizable role in the mediasphere. Pokémon, in itself, could thus be paradigmatically used to discuss the manifestation of much subtler changes in certain semiotic codes of behaviour related to animated faces. Following Lotman and Lévi-Strauss, it should be remembered that fictional narratives and myths, both traditional and modern ones, interact with cultural self-models and contribute to their strengthening or weakening. Thus, as stressed by Tønnessen and Tüür (2014), as well as Bertrand and Marrone (2019), any form of textualized categorization of otherness (whether non-human animals, nonliving beings, or non-natural entities) in relation to the subjectivity (we humans), interacts with and takes part in the many codes shaping the human semiosphere. Animals, spirits, gods and inanimate beings are classified and interpreted according to cultural taxonomies and placed in specific parts of the cultural self-description systems of the semiospheres, depending on their relationships with the core forms 10 Websites of internet culture such as Kotaku and Polygon have dedicated several articles to fan reaction surrounding Mimikyu; such as https://www.polygon.com/2016/7/19/12225262/ pokemon-sun-and-moon-new-pokemon-mimikyu. 11 See https://www.polygon.com/2019/11/22/20977707/pokemon-sword-shield-fossildracozolt-arctozolt-dracovish-arctovish-pokedex-england-fake. Gotta face ‘em all 555 of subjectivity of the system itself. As a result, the ways in which even fictional narratives display and describe animals (or animaloids) through texts, pictures and daily practices can highlight significant dynamics of the semiosphere, especially in connection with the principal aspect of the face (and its representation) as a key semiotic interface for communication among living beings, which is now being increasingly textualized in the media system. In my view, the many above-mentioned visual and narrative features of the Pokémon franchise may show two significant cultural dynamics which have been emerging in the contemporary Japanese semiosphere in recent times: the first is a logic of visual animism of characters; the second is a regime of playful interaction with the animated. In relation to the former, we can refer to Descola’s (2013) four-fold schemas of nature/culture ontologies to understand Pokémon franchise logic as an animist ontology: simply put, divergences in exteriority may exist between the different conceived natural and cultural kingdoms, but there is a shared community in their interiority. Furthermore, as previously discussed, the world of Pokémon often leans towards what Descola would call a totemic ontology, which is characterized by a community of both interiority and exteriority. This is the result of the frequent simultaneous hybridizations, heterotopies and undermining of the encyclopaedic boundaries between species, natural kingdoms, and what is natural, artificial or cultural. Within this heterotopian ontological system, the representation of faces acts as a double-layer mediatory interface. First, as previously discussed, the face is the paradigmatic interface for human communication, linking the somatic and the semiotic, the natural and the cultural. Anthropomorphic and expressive animated faces make communication possible between humans and Pokémon, both among characters and in relation to the reader, even though they do not share a language, because of their shared gestural and expressive codes as sentient beings. Moreover, on the level of media representation, anthropomorphic face representation acts as a mediatory interface between the heterotopian (animist/totemist) ontology of Pokémon and the common naturalist ontology of the public/viewers. By following the tropes of traditional animation, facial representation makes it possible to control and reduce the encyclopaedic uncanny (das unheimliche, see Foucault 1967), translating it into something understandable, inoffensive, even cute and worthy of affection (inheriting the kawaii aesthetics, see Kinsella 1995). An indirect example of this dynamic of facial representation is provided by the uncanny effects generated by its removal/modification in the recent film Detective Pikachu (2017). The CGI + live action movie is set in the Pokémon universe, yet in order to ensure coherence between CGI and real actors, the directors chose to 556 Vincenzo Idone Cassone abandon the typical visual flatness of the anime and video game. Instead, they adopted a “realistic” representation of Pokémon,12 showing and animating in detail their fur and skin, eyes and gaze, bones and muscles. Despite the movie’s success, this choice sparked a debate on the internet over the unsettling results:13 the representations of several Pokémon were interpreted as problematic or slightly disturbing (Jigglypuff, Psyduck, Lickitung) exactly because of the substitution of typical anime stylization/flatness with iconic bodily and zoomorphic traits. Among the many, the most notable was a highly uncanny and creepy representation of the (above-mentioned) humanoid Pokémon Mr. Mime (Fig. 5). The texturing of its skin (lights, skin pores), together with the motion-capture animation and perfectly acted gaze, in relation to its already anthropomorphic form/behaviour, gave rise to both disturbance and amusement, eliciting comparisons of this Pokémon to the unsettling clown in the film It and to Chucky the doll. Figure 5. Mr. Mime in Detective Pikachu. This example brings us to the second dynamic, the regime of playful interaction with the animated. As anticipated in the previous pages, it is very common for fans of the franchise to highlight the many inconsistencies and ludo-narrative dissonances of Pokémon. However, rather than leading to negative evaluations and disengagement, this often results in positive playful mocking and the development of fan theories to address the clear inconsistencies. Mr. Mime is again a fitting example, because of the long-lasting theories involving Ash’s (the main character in the anime) lack of a father, who is never shown or discussed in the anime. 12 See https://www.wired.co.uk/article/pokemon-detective-pikachu-cgi. See https://www.huffpost.com/entry/realistic-pokemon-from-detective-pikachu-arecreeping-people-out_n_5beb41e1e4b0caeec2bef1ff. 13 Gotta face ‘em all 557 According to this theory, Ash’s true father is his mother’s Mr. Mime (Mimey) which/who has lived under the very same roof since the boy was young, does a lot of housework, uses clothes and accompanies his mother on vacation, and never stays in the Pokéball. The theory goes further by hinting at many disparate plot elements and trying (in a comedic way) to explain all of them by means of the relationship between Ash and Mimey. The many hundreds of Pokémon theories on the net14 rarely try to prove themselves true; instead, they continuously shift between being half-serious and half-facetious, simultaneously maintaining certain stances and their reversal, according to the specific notion of playful communication defined by Bateson (1956), according to a playful regime of interaction, as the paradoxical process of make-believe, in which the logic paradox itself is staged (i.e. ‘to play fight’ for children is simultaneously to fight as if doing so seriously, and to know one is not really fighting at all). Internet fan cultures (both Japanese and Western ones) react to the incoherence, meta-referentiality and heterotopic taxonomy in Pokémon in a playful way. Fans take advantage of parts of ludo-narrative or diegetic incoherence in order to search playfully for possible alternative explanations or re-interpretations, doing so with an ironic distance, a tongue-in-cheek attitude. They search seriously for coherence with their encyclopaedic everyday codes, which involves a complex and time-consuming (re)interpretive process of the franchise’s countless texts; yet, the results are often parodic and equally self-contradicting, if not more convoluted, clearly because of the fun afforded by the ridiculous consequences of this search for coherence, which ignores many typical narrative tropes of animation. This very logic of interaction is encouraged and adopted by the authors and managers of the franchise, who purposely do not focus on strong internal narrative coherence, willingly presenting meta-references, contradictions among texts and within them, playful communication with fans and audience. For instance, the explicitly incoherent nature of the PokéDex descriptions has led to the fan theory that its entries were written by the ten-year-old characters of previous games,15 thus explaining the Pokémon individualization, the impossible measurements, the conflicting entries and descriptions by rumours, and so on. 14 15 See https://www.thegamer.com/classic-pokemon-fan-theories-believe-true/. See https://www.cbr.com/pokemon-fan-theory-pokedex-creepy-entries/. 558 Vincenzo Idone Cassone 5. Playful visual animism in Japan My hypothesis is that the connection between an animistic visual logic of representation and a logic of playful interaction is increasingly common in the Japanese mediasphere, despite often being assimilated to traditional tropes of animation. In the following pages I discuss how what I label ‘playful visual animism’ can be identified in other daily practices linked to the presence of animated faces in Japan, and in their connection to pre-existing cultural dynamics and semiotic codes. Many authors have highlighted the massive presence of animated characters (or kyara, see Occhi 2010, Wilde 2019, Sutera 2016) in Japanese society. ‘Kyara’ is a generic term which includes anthropomorphic franchise mascots, characters from manga and anime, as well as human figures used for brands or communication (imeeji kyara). Kyara proliferate in Japanese society as part of the above-mentioned media mix strategies, being extensively used for both large and small brands, being adopted for gadgets and merchandizing, diffusing the kawaii aesthetics, and even taking part in parades in costume (kigurumi) and receiving official recognition from the Japanese government. In addition to the ubiquitous brand kyara, Alt and Yoda (2007) highlight the existence of the so-called working characters/service characters (hataraku kyara; Wilde 2018), which proliferate in daily metropolitan life, being utilized in road signs, municipal announcements, political advertizing and everyday object packages (see Fig. 6). Both anthropomorphic and zoomorphic (Miller 2010), these characters act as mediators of everyday life, giving indications and directing the behaviour of passers-by, reminding us of social norms, wishing good luck and fulfilling other apotropaic functions. They act in a similar way to Latour’s petit Robert (see Latour 1992): whereas the mechanical boy replicated by figurization the agentivity of the user of the machinery, these characters embody and replicate the affective communicative dynamics of the participants in the semiotic encounter (be it a purchase, warning, advice, etc). Nozawa (2013) argues that everyday characters are the bearers of a semiotic ideology that cannot be reduced to functional anthropocentrism: because of their multiplication and frequency of interaction in daily life, as well as their relationships with other phenomena of human-character embodiment (vocaloids, kyara actors and personae) they manifest a semiotic ontology of transition that unites people and characters, animals and animates, the ordinary and the fictional, coherence and inconsistencies. Gotta face ‘em all 559 Figure 6. Service characters in Japan (the author’s photos). In addition to hataraku kyara, yuru kyara (or Gotōchi-kyara; Occhi 2014) are increasingly common in Japan. Taking their name from the wobbly, imprecise drawings which constitute them, these characters are often linked to local places, events or products. Despite their simplistic aesthetics, they frequently present hybrid forms, mixing the natural and the cultural, the zoomorphic and the anthropomorphic, combining in their simple representation typical products, myths and folklore, lifestyles and stereotypes. Incorporating kawaii aesthetics and neotenic characteristics (Kinsella 1995), they may feature a double nature: twodimensional on one side, mascot in costume (kigurumi) on the other, at times participating in specific local parades and national competitions (see Fig. 7). Occhi (2012) has linked these characters to the religious animist traditions of Japan, by regarding them as secularized/popularized forms of the genius loci: anthropomorphic folkloric imagery (kappa, tanuki, kitsune) or the representation of vital forces and spirits (kami) that manifest themselves in the natural world. With the notable difference that, whereas the genius loci mediated between the supernatural and the sensible, the yuru kyara and hataraku kyara become mediators of an earthlier civic, promotional or commercial message, while often acting as emotional support, linked to a sense of relief, comfort and regression (Kondo 2006), partially confirmed by the attested use of kyara in modernized omamori. 560 Vincenzo Idone Cassone Figure 7. Yuru kyara in kigurumi form. The religious and mythic dimensions of these kyara further highlight the ludic regime of interaction with them, shifting between seriousness and playfulness. Moreover, this is linked to the long-term tradition which connects acting, playfulness and devotion through religious practices. This confirms Foster’s (2009: 13) assertion about the persistence of yōkai (Japanese supernatural spirits and entities) in the cultural imagination because of their ability to pivot between the credible and incredible. This link is clear in the traditional matsuri performances, as well as in the “play of the gods” (kami asobi; Plutchow 1996), and is, to a degree, reproduced in a secularized form, in the kigurumi parades and television mascot competitions. The ludic nature of these secularized religious practices, mediated by characters, makes possible the paradoxical communication (see Bateson 1956) which simultaneously includes affective approach and cognitive distancing. Once observed through the double perspective of visual animism and playful interaction, these phenomena (and Pokémon) show further connections with certain semiotic characteristics of contemporary postmodernism. Azuma (2009; see also Lamarre et al. 2007) describes the Japanese shift in the 1990s from the logic of the postmodern as a narrative to that of the postmodern as a database, suggesting that Otaku culture could be considered the most sensible place to observe this change. In what he labels the animalization era, the logic of narrative coherency is substituted by the affective logic of the database, which, in Lotmanian terms, acts as a selfmodelling system (Madisson 2016). Texts based on the self-model of the database Gotta face ‘em all 561 involve micro-narratives, centred on specific characters, themes and aesthetics which can be subject to infinite remodelling, with no essential traits connected to the characters. What remains is not the coherent personality traits, but rather the affective relationship between the user and the character. As a result, an interesting parallel develops between the logic of the database and the dynamics of the collection, which is the way in which people interact with intangible characters: explicitly, in the case of Pokémon, and cognitively for kyara in general. On the one hand, the collection is a model of organization of the sensory in a playful form; on the other hand, the collection is a reorganization of the world into a microcosm of affections. Foster (2009: 214), for instance, points out the ludic and taxonomic similarities between Pokémon and yōkai, which could also be expanded to the contemporary interaction and communication with the above-mentioned kyara. These long-term dynamics are reproduced in the contemporary world, placed in a media circuit of production, sharing, interaction and evaluation. As I suggested in the introduction, this further strengthens the idea of Pokémon as an emblematic case study for the understanding of the semiotic dimension of animated characters in the Japanese mediasphere. Conclusion The examples discussed above indicated how the Japanese mediascape is permeated with anthropomorphic characters and faces, ranging from narrative transmedia franchises to branding, to merchandise, as well as an increasing presence in internet and fan culture. I have suggested how their spread may be connected to the emergence of specific semiotic behavioural codes and as a result of the interaction between the codes of visual representation of these anthropomorphic characters, their fictional narratives and the sociocultural patterns of interaction between humans and characters. Characters become key symbolic mediators (Lotman 1990) of everyday life, as well as “semiotic figures of in-betweenness” (Nozawa 2013: 8) in the semiosphere. Through the analysis of the Pokémon case study, I defined two interrelated features: a logic of visual animism of characters; and a regime of playful interaction with the animated. These dynamics need not be limited to the Pokémon franchise, but can be attested in many anthropomorphic face-based characters and practices in the Japanese media system and culture. The resulting patterns of playful visual animism may help understand the intersection between visual stylized representation, the database-logic of media, fan culture behaviour and cultural appropriation of technologies in Japan. 562 Vincenzo Idone Cassone These dynamics may be correlated with recent changes in the semiotic dynamics of the Japanese media system over the last few decades, namely its process of globalization (Daliot Bul, Otmazgin 2020; Pellitteri 2018), its participation in the convergence culture and contemporary transmedia dynamics (Jenkins 2006; Scolari 2009; Saldre, Torop 2012), and its relationships with the broader dynamics of internet culture. References Appadurai, Arjun 1990. Disjuncture and difference in the global cultural economy. Theory, Culture & Society 7(2/3): 295–310. https://doi.org/10.1177/026327690007002017 Azuma, Hiroki 2009. Otaku: Japan’s Database Animals. Minneapolis: University of Minnesota Press. Allison, Anne (ed.) 2006. Millennial Monsters: Japanese Toys and the Global Imagination. Vol. 13. Berkley: University of California Press. https://doi.org/10.1111/j.1467-9655.2007. 00472_9.x Alt, Matt; Yoda, Hiroko 2007. Hello, Please! Very Helpful Super Kawaii Characters from Japan. San Francisco: Chronicle Books. Barrows, Lissa 2014. Planning with Character: Gotouchi Kyara and Place Branding in Japan. [Unpublished Master’s Thesis, M.Sc.] New York: Columbia University. Bateson, Gregory 1956. The message “This is play”. Group Processes 2: 145–241. Bendazzi, Giannalberto 2016. Animation: A World History. Vol. 3: Contemporary Times. Boca Raton: CRC Press. https://doi.org/10.4324/9781315720746 Bertrand, Denis; Marrone, Gianfranco (eds.) 2019. La sfera umanimale. Roma: Meltemi. Bruce, Vicky; Young, Andrew W. 2012. Face Perception. London, New York: Routledge. https://doi.org/10.4324/9780203721254 Daliot-Bul, Michal; Otmazgin, Nissin 2020. The Anime Boom in the United States: Lessons for Global Creative Industries. Leiden: Brill. Descola, Philippe 2013. Beyond Nature and Culture. Chicago: University of Chicago Press. https://doi.org/10.7208/chicago/9780226145006.001.0001 Eco, Umberto 1975. Trattato di semiotica generale. Milano: Bompiani. Eco, Umberto 1979. The Role of the Reader: Explorations in the Semiotics of Texts. Bloomington: Indiana University Press. Eco, Umberto 2000. Kant and the Platypus: Essays on Language and Cognition. Boston: Houghton Mifflin Harcourt. Foster, Michael Dylan 2009. Pandemonium and Parade: Japanese Monsters and the Culture of Yokai. Berkeley: University of California Press. https://doi.org/10.1525/9780520942677 Foucault, Michel 1967. Let mots et les choses: Une archéologie du savoir. Paris: Gallimard. Game Freak 2006. Pokèmon Diamond/Pearl. Nintendo. Han, Ruobin 2017. The Characterization of Japan. From marketing to Identity. [Unpublished Ph.D. Thesis in Humanities.] Leiden: University of Leiden. Hutchinson, Rachel 2019. Japanese Culture Through Videogames. London, New York: Routledge. https://doi.org/10.4324/9780429025006 Gotta face ‘em all 563 Iwabuchi, Koichi 2002. Recentering Globalization: Popular Culture and Japanese Transnationalism. Durham: Duke University Press. https://doi.org/10.1515/9780822384083 Jenkins, Henry 2006. Convergence Culture: Where Old and New Media Collide. New York: New York University Press. Kinsella, Sharon 1995. Cuties in Japan: Women, media and consumption in Japan. In: Moeran Brian; Skow, Lise (eds.), Women, Media and Consumption in Japan. London: Routledge, 220–254. Kobayashi, Hiromi; Kohshima, Shiro 2001. Unique morphology of the human eye and its adaptive meaning: Comparative studies on external morphology of the primate eye. Journal of Human Evolution 40(5): 419–435. https://doi.org/10.1006/jhev.2001.0468 Kondo, Kensuke 2006. Hyaku nen ai sareru kyarakutaa no tsukurikata. [How to create characters that will be loved for one hundred years.] Tokyo: Goma Books. Lamarre, Thomas 2018. Animation and animism. In: Boehrer, Bruce; Hand, Molly; Massumi, Brian (eds.), Animals, Animality, and Literature. Cambridge: Cambridge University Press, 284–300. https://doi.org/10.1017/9781108595278.017 Lamarre, Thomas; Azuma, Hiroki; Furuhata, Yuriko; Steinberg, Marc 2007. The animalization of Otaku culture. Mechademia Second Arc 2(1): 175–187. https://doi.org/10.1353/ mec.0.0023 Latour, Bruno 1992 Where are the missing masses. In: Bijker, Wiebe E.; Law, John (eds.), Shaping Technology-Building Society: Studies in Sociotechnical Change. Cambridge: MIT Press, 225–259. Leone, Massimo 2009. Agency, communication, and revelation. Lexia 3/4: 77–95. Leone, Massimo 2018. The semiotics of the face in the digital era. Perspectives 17: 27–29. Leone, Massimo 2021. On muzzles and faces: The semiotic limits of visage and personhood. International Journal for the Semiotics of Law/Revue internationale de Sémiotique juridique 34(2): 1–24. https://doi.org/10.1007/s11196-020-09812-8 Lotman, Yuri. M. 1990. Universe of the Mind: A Semiotic Theory of Culture. London: IB Tauris. Madisson, Mari-Liis 2016. Self-description. In: Rodriguez Higuera, Claudio Julio; Bennett, Tyler James (eds.), Concepts for Semiotics. (Tartu Semiotics Library 16.) Tartu: Tartu University Press, 204–212. Maran, Timo; Martinelli, Dario; Turovski, Aleksei (eds.) 2011. Readings in Zoosemiotics. Vol. 8. Berlin: Walter de Gruyter. https://doi.org/10.1515/9783110253436 Miller, Laura 2010. Japan’s zoomorphic urge. ASIANetwork Exchange: A Journal for Asian Studies in the Liberal Arts 17(2): 69–82. https://doi.org/10.16995/ane.208 Occhi, Debra J. 2010. Consuming Kyara ‘characters’: Anthropomorphization and marketing in contemporary Japan. Comparative Culture 15: 77–86. Occhi, Debra J. 2012. Wobbly aesthetics, performance, and message: Comparing Japanese Kyara with their anthropomorphic forebears. Asian Ethnology 71(1): 109. Ōtsuka, Eiji 1989. Monogatari shōhiron. [A Theory of Narrative Consumption.] Tokyo: Shinyosha. Pellitteri, Marco 2018. Kawaii aesthetics from Japan to Europe: Theory of the Japanese “cute” and transcultural adoption of its styles in Italian and French comics production and commodified culture goods. Arts 7(24): 1–24. https://doi.org/10.3390/arts7030024 564 Vincenzo Idone Cassone Plutschow, Herbert 1996. Matsuri: The Festivals of Japan: With a Selection from P.G. O'Neill's Photographic Archive of Matsuri. London, New York: Routledge. Saldre, Maarja; Torop, Peeter 2012. Transmedia space. Crossmedia Innovations: Texts, Markets, Institutions 51: 25–44. Scolari, Carlos Alberto 2009. Transmedia storytelling: Implicit consumers, narrative worlds, and branding in contemporary media production. International Journal of Communication 3: 586–-606. Sutera, Jillian Rae 2016. Who is Hikonyan? The phenomenon of Japanese Yuru-Chara. Sociology 6(12): 775–782. https://doi.org/10.17265/2159-5526/2016.12.004 Surace, Bruno 2017. Zoosemiotica dei Pokémon. In: Marrone, Gianfranco (ed.), Zoosemiotics 2.0: Forms and Policies of Animality. Palermo: Edizioni Museo Pasqualino, 609–619. Steinberg, Marc 2012. Anime’s Media Mix: Franchising Toys and Characters in Japan. Minneapolis: University of Minnesota Press. https://doi.org/10.5749/minnesota/ 9780816675494.001.0001 Steinberg, Marc 2015. 8-Bit Manga: Kadokawa’s Madara, or the gameic media mix. Kinephanos: Journal of Media Studies and Popular Culture 5: 40–52. Tobin, Joseph (ed.) 2004. Pikachu’s Global Adventure: The Rise and Fall of Pokémon. Durham: Duke University Press. https://doi.org/10.2307/j.ctv1131ctc Tønnessen, Morten; Tüür, Kadri 2014. The semiotics of animal representations: Introduction. In: Tønnessen Morten; Tüür, Kadri (eds.), The Semiotics of Animal Representations. Leiden: Brill Rodopi, 7–30. https://doi.org/10.1163/9789401210720_002 Wells, Paul 2009. The Animated Bestiary: Animals, Cartoons and Culture. New Brunswick: Rutgers University Press. https://doi.org/10.36019/9780813546438 Wilde, Lukas R. A. 2018. Character street signs (hyōshiki): “Mangaesque” aesthetics as intermedial reference and virtual mediation. Orientaliska Studier 156: 130–150. Wilde, Lukas R. A. 2019. Kyara revisited: The pre-narrative character-state of Japanese character theory. Frontiers of Narrative Studies 5(2): 220–247. https://doi.org/10.1515/ fns-2019-0014 Gotta face ‘em all: покемоны, японские анимационные персонажи и появление игрового визуального анимизма В результате развития технологий и новых культурных практик современная медиасфера всё больше заселяется цифровыми лицами. Это явление не ограничивается человеческими лицами, но включает в себя множество вымышленных анимированных лиц, называемых «персонажами», «маскотами» или kyara. Явление kyara, давно известное в Японии, широко распространяется благодаря глобализации, цифровизации и технологиям медиа-микширования. Таких персонажей, соединяющих области визуального дизайна, вымышленных историй и социокультурного потребления, можно считать семиотическими знаками промежуточности, ключевыми символическими медиаторами в японском медиапространстве. Антропоморфный дизайн их лиц стирает культурные границы между человеком и не-человеком, одушевленным и неодушевленным, природой и культурой. Кроме того, постмодернистские истории, Gotta face ‘em all 565 частью которых они являются, вдохновлены прошлым и настоящим, мифами и научной фантастикой. Наконец, они подразумевают энциклопедическую переработку на границе между фантастикой и реальностью, мифическим и секулярным, игрой и реальностью. В статье рассматриваются семиотические измерения kyara в современной Японии с акцентом на логику их репрезентации и культурное влияние. Анализ франшизы «Покемо́ н» позволяет описать возникновение новых семиотических паттернов «игрового визуального анимизма» в процессе медийного «олицетворения» повседневной жизни. Gotta face ‘em all: Pokémonid, Jaapani animategelased ning mängulise visuaalanimismi esiletõus Tehnoloogiliste uuenduste ning uute kultuuripraktikate tulemusena sisustavad kaasaegset meediasfääri ühe enam digi(taliseeritud) näod. See nähtus ei piirdu inimnägudega, vaid siia kuulub ka fiktsionaalsete animanägude suur universum; neid nimetatakse ‘tegelasteks’, ‘maskottideks’ või ‘kyara’. Kuigi kyara pole Jaapanis uus ilming, on nad hakanud levima tänu globaliseerumisele, digitaliseerumisele ja kombineeritud meedia strateegiatele. Visuaalse disaini, fiktsionaalsete narratiivide ja ühiskondlik-kultuurilise tarbimise seoste kaudu võib kyara’t pidada semiootilisteks vahepealsuskujudeks, Jaapani meediamaastiku peamisteks sümboolseteks vahendajateks. Nende antropomorfne näodisain vahendab kultuurilisi piire inimliku ja mitteinimliku, elusa ja elutu, looduse ja kultuuri vahel. Lisaks seguneb nende postmodernsetes narratiivides minevikust ja olevikust lähtuv inspiratsioon müütidest ulmeni. Ning viimaks tähendavad nad entsüklopeedilist ümbertöötlemist, mis leiab aset väljamõeldise ja tegelikkuse, müütiliste osutuste ja sekulariseerumise, tõsiduse ning mängulise teeskluse valdkondade vahel. Artikli eesmärgiks on uurida kyara semiootilisi mõõtmeid tänapäeva Jaapanis, rõhuasetusega nende representatsiooniloogikale ja kultuurilistele tagajärgedele. Neid käsitletakse, analüüsides Pokémoni frantsiisi, mis võimaldab kirjeldada “mängulise visuaalanimismi” uute semiootiliste mustrite esiletõusu igapäevaelule media poolt nägude omistamise käigus. 566 Peter Mantello Sign Systems Studies 49(3/4), 2021, 566–589 Fatal portraits: The selfie as agent of radicalization Peter Mantello1 Abstract. For the modern-day jihadist, the digital self-portrait or, more specifically, battlefield selfie is a popular tool for identity building. Similarly to the selfies taken by non-violent practitioners of self-capture culture, the jihadist selfie represents an alternative to the Cartesian formulation of a unitary and indivisible self. Rather, it is a product of social relations and performative actions, constituted in dialogue with others through very specific socio-cultural frameworks and expectations. However, unlike its non-violent Doppelgänger, the expectations of this dialogue are centred around a larger political agenda which actively seeks to reformat collective memories of imperial Islamic conquests and co-opt religion as a way to impose a moral order on its violence. Importantly, the battlefield selfie allows the jihadist easily to traverse the boundaries between two seemingly opposing belief systems. Although there exists a wealth of scholarship of self-capture culture, image sharing sites and micro-celebritism, their pervasive influence and practice on battlefield is understudied. This article draws from the personal histories of key Islamic extremists who were both lionized and demonized as a result of their battlefield micro-influencer practices. Today, however, the same individuals can achieve internet fame by participating in self-capture culture – posting selfies, videos or blogging. In other words, never before has a soldier’s public communication been so personal yet collective. Keywords: micro-influencer; terror; selfie; self-portrait; jihad; social media; branding; networked communities; social photo; necro-celebritism Introduction A decade ago, the proliferation of Web 2.0 strategies (forums/message boards/ file-sharing sites) heightened the visibility of jihadist movements around the globe but also empowered online partisans with a sense of shared purpose, agency and, importantly, community (Zelin 2014). Easily downloadable jihadist “lifestyle” e-magazines such as Inspire, foreshadowed a new age of information warfare based 1 Ritsumeikan Asia Pacific University, Beppu, Japan; e-mail: mantello@apu.ac.jp. https://doi.org/10.12697/SSS.2021.49.3-4.16 Fatal portraits: The selfie as agent of radicalization 567 on connectivity, media professionalism and aesthetic-driven content (Droogan, Peattie 2016). Even more so, it gave the jihadist imaginary a distinctly modern edge, painting radical Islam as a utopian world where traditional religious values co-exist with modern-day pleasures of techno-culture: a spiritually rewarding yet playful universe of memes and mosques, chat rooms and prayer rooms, selfies and Sharia. Paradoxically, these same screen-mediated practices so central to contemporary jihadism are based on advancing a violent seventhcentury political project whose core tenets are anti-secular, anti-modern, antiindividualist, anti-consumerist, and above all, anti-liberal. Aside from these innate contradictions, new Islamic militantism is emblematic of a growing cultural desire for and dependency on digital devices for self-expression, self-presentation and self-affirmation (Senft 2008; Marwick 2015; Djafarova, Trofimenko 2017; Manovich 2019). As Marshall McLuhan (1994) prophesized, long before the smartphone was conceived, digital devices have become the prosthesis of human agency for they extend human consciousness, but in doing so, create a new self-consciousness – a digitally extended self. For Achille Mbembe (2019)2, this complex tethering of the human and the technological has given way to subjectivities in line with the dominant strictures of communicative capitalism and increasing shift in modern society “from a politics of reason to a politics of experience [where] sensibility, emotions, affect, sentiments and feelings are the real stuff of subjecthood and therefore of radical agency”. A prime example of this phenomenon is how violent modes of religiosity and individuation are now being accelerated through the rising vogue of the ‘selfie’. The Oxford dictionary defines the selfie as “a photograph that one has taken of oneself, typically one taken with a smartphone or webcam and uploaded to a social media website”. Importantly, what separates the selfie as a tool of identity construction from its non-digital counterpart is its affiliation to online communities. In other words, the selfie is a product of social relations and performative actions, constituted in dialogue with others through very specific socio-cultural frameworks and expectations (Papacharissi 2010; Durkheim 2002[1897]; Bakhtin 1973; Dobson, Robards, Carah 2018). However, for the jihadist, the expectations of this dialogue are centred around a larger political agenda which actively seeks to reformat collective memories of imperial Islamic conquests and co-opt religion as a way to impose a moral order on its violent universe (Tibi 2010; Afsaruddin 2014). 2 Mbembe, Achille 2019. Thoughts on the planetary: An interview with Achille Mbembe. Retrieved on 18 July 2021 from https://www.newframe.com/thoughts-on-the-planetary-aninterview-with-achille-mbembe/. 568 Peter Mantello As Adam Levin (2014)3 notes, the selfie “moves the onus of experience away from phenomenal engagements with adjacent strangers and towards digital interactions with online communities”. Critically, the battlefield selfie pushes the boundaries of the networked self even further into a collective moment of affective networked sharing and political formation. Whereas traditional forms of proselytizing once spoke in a top-down manner directly to the individual as an autonomous agent (Leone 2015), the peer-to-peer nature of jihadist self-capture culture promulgates a heteronomous self, constructed through networked interfaces of familiarity and conformity. While recent scholarly studies have focused on the strategic, propaganda or operational aspects of new Islamic militant media (Fisher 2015; Nissen 2015; Prucha 2016; Berger, Perez 2016; Shehabat, Mitew 2017), this article examines how the “digital turn” has transformed the jihadist self by allowing it to remain distinct yet somehow act together as a networked subjectivity (Galloway, Thacker 2007; Veilleux-Lepage 2016). I argue that the selfie marks the point of transition from the biopolitical to thanatopolitical, but, in doing so, reconciles the temporal and ideological conundrum of suturing together a modern-medieval self. Curiously, this reconciliation relies as much on commercial strategies of capture (Zuboff 2017) that indirectly turn social media platforms into ideological echo chambers as it does on the self-representational practices of micro-celebritism that transform ordinary persons into insta-famous brand influencers. Although there exists a wealth of scholarship on the rising significance of public performance in everyday life, commodification of self, and the widening scope of what constitutes micro-celebrity culture, this article fills an existing vacuum in the much-needed theorization of the relationship between selfies and extremist subjectivity. Thus, the article is divided into three sections. The first part examines how the emergence of Web 2.0 has fundamentally reshaped the way in which combatants in asymmetrical war are represented and enacted. The second part traces a brief historical arc over social and cultural origins of celebritism. Here I draw from seminal literature in celebrity studies in order to situate better the contemporary jihadist within micro-celebrity culture. The last section explores the battlefield selfie as a marker of identity realignment that occurs in the process of a jihadist’s radicalization. My goal here is to illustrate how the cultural logic, practices and determinants that shape micro-influencers are actually no different from how modern-day extremists construct, negotiate and mediate notions of self. 3 Levin, Adam. 2014. The selfie in the age of digital recursion. InVisible Culture: An Electronic Journal for Visual Culture 20. Retrieved on 16 July 2021 from https://ivc.lib.rochester.edu/theselfie-in-the-age-of-digital-recursion/. Fatal portraits: The selfie as agent of radicalization 569 The rise of the terrorist as a micro-celebrity Prior to Web 2.0, an extremist’s chances of becoming a celebrity hero were entirely dependent on the cultural and institutional processes responsible for creating an entertainment celebrity. Early-20th-century anarchists, postwar colonial insurgents and even new-left revolutionaries of the 1960s and 1970s were reliant on mass media outlets to provide them with the kinds of visibility and recognition afforded to entertainment celebrities (Weimann, Winn 1994; Hoffman 2017). Even then, they had no control over whether or not their violent deeds or demands would be accurately communicated by the media. In the aftermath of the first Wall Street terror bombing in 1920, the anarchists’ media strategy amounted to leaving five leaflets in a nearby mailbox (Larabee 2000). During the post-war era, Menachem Begin’s “glass house” strategy used dramatic attacks on British targets in order to gain media attention so as to turn international opinion against Britain’s occupation of Palestine (Auerbach, Ben‐Yehuda 1987). In the early 1970s, Osamu Maruoka and his Red Army colleagues held airline hostages for several days before news crews showed up to hear his ransom demands (Andrews 2016). Similarly, in the wake of his OPEC attack in Vienna, Carlos the Jackal, once called the ‘World’s Most Wanted Man’ and ‘International Face of Terror’, deliberately held off his daring escape until the arrival of TV news cameramen (Thomas 2013). Even Bin Laden was dependent on his loyal couriers and agreeable editors at Al Jazeera to have his VHS-taped messages broadcast to the world (Torres-Soriano 2010). Yet this last wave of globalized terror distinguishes itself not only by its non-secular orientation but, importantly, “proactive” media sensibilities. For example, during al Shabab’s attack on the Westgate Shopping Mall in Nairobi, militants posted images to social media of their massacre while exchanging text messages with organizational leaders and followers. Never before in contemporary history have the non-state combatants possessed a single device enabling them to capture, edit and publish photos of themselves and their daily experiences; view and rate photos of their comrades in arms (and enemies), search and discover other like-minded photos; and enter into conversations with the authors of those photos (Manovich 2019). The jihadist as micro-celebrity is in part the latest manifestation of a longer historical continuum of socio-economic formations and networks of globalization (Appadurai 2006). Scholars have traced the roots of celebritism back to the industrial revolution and rise of cities (Turner 2013); technological advances in printing, image-making and travel (Inglis 2010); expansion of urban populations and consumer culture (Boorstin 1983); and, importantly, the growth of possessive individualism (Porter 2004). Importantly, as each of these factors evolved, the concept and focus of fame drastically shifted. Daniel Boorstin (1983: 57) argues 570 Peter Mantello that the modern celebrity is defined less by their achievements per se, but more by their being “well-known for well-knownness”. In other words, public recognition is independent of any achievements. Yet as Theresa Senft (2008) and other scholars have argued, in the age of social media, a micro-celebrity is more than just a status. Like the world of brand commodities, a micro-celebrity aspirant must construct an online identity that stands out from the crowd. Not only must they dedicate a good portion of their daily life to maintenance of their on-screen other but also find innovative ways to showcase themselves. Here authenticity is the key. Above all else, being a micro-celebrity comes with expectations linked to the traditional prerequisites of celebritidom, namely, presenting a unique persona, viewing followers as fans, producing content consistent with that persona and engaging in strategic interaction with the goal of maximizing both intimacy and continued visibility (Senft 2008; Marwick 2015). For P. David Marshall (2015), the ‘unique persona’ of the micro-celebrity supplants not only the older, more established value of class and status. Rather, as Marwick (2015: 17) insists, the micro-celebrity “becomes the embodiment of the possibility of change through consumer culture”. However, he adds, the fact that their identity originates as mediatized construct and, as such a product of consumption, limits the possibility of affecting any radical transformation. Thus, they become inevitably trapped in “sustaining polities rather than threatening them” (Marwick 2015: 17). Arguably, this appears to be the case of famous personalities such as Greta Thunberg (see Morningstar 2019; Abidin et al. 2020) as much as it was for (in-)famous terrorists such as Timothy McVeigh or Osama Bin Laden (see Nacos 2007). Thunberg’s sharp rise to celebrity status began as an anonymous, headstrong teenager campaigning against climate change on the steps of the Swedish parliament. McVeigh was an impoverished yet decorated US war hero whose hatred of big government ended with him blowing up a state building and killing 168 people. Although he was depicted in the media as a psychopath, psychiatrists concluded that McVeigh was neither irrational nor deranged. In fact, Michel and Herbeck (2001) note that the war veteran saw himself as Luke Skywalker from the Star Wars film trilogy and the Oklahoma federal building as a real-life metaphor of the Death Star. Bin Laden, on the other hand, was a soft-spoken, well-educated, and extremely wealthy Muslim who sacrificed a life of luxury and privilege to wage a cosmic war against the West. Critically, in all three cases, the formation of their individuality as celebrities emerged as a result of a well-oiled and established corporate media machine that sustains itself by recycling dominant discourse in security politics. While Thunberg has raised greater public awareness of imminent dangers of climate collapse, her advocacy efforts to reign in industry polluters or make world leaders accountable have yet to succeed (Morningstar 2019; Abidin Fatal portraits: The selfie as agent of radicalization 571 et al. 2020). Although McVeigh is a martyr of right-wing militia groups, any discernible changes in US federal control over state sovereignty brought on by his actions have yet to materialize. Likewise, while Osama Bin Laden is both lionized and demonized as the father of global jihad, his utopian dream of a transnational caliphate remains elusive to Islamic extremist organizations. The third ingredient of celebritism is intimacy. While 19th-century print media heightened the visibility and familiarity of unique personas, it was 20th-century broadcast culture that dramatically shifted public understanding of the celebrity (Driessens 2012). The advent of radio and TV transported public personalities into domestic spaces making them “household names”. Fans and followers no longer regarded the celebrity as a distanced public figure only to be found on the printed page, stage or screen, but someone one knew intimately (Schickel 2000). Certainly, the rise of internet and social media has heightened and accelerated “feelings of closeness” between celebrity and follower, but it has also reconfigured the relational dynamic. Intimacy is no longer simply about the relationship between celebrity and follower but now extends to the peer-to-peer connection between community members. In other words, Schickel’s “intimate stranger” is now inextricably tied to digitally constructed and networked “intimate publics” (see Dobson, Robards, Carah 2018: 113–128), where members of online communities share a world view and emotional knowledge that they have derived from common historical experience. The socially productive attributes of intimate publics can be evidenced in the #MeToo or Black Lives Matter movements. At the same time, their negative impact on society can also be illustrated in far-right groups such as the Proud Boys or QAnon. Similarly, the significance of jihadist selfie lies in its networked communities and intertextual iterability that makes it possible to amplify emotional connections to an imagined homeland, kindle nostalgia for a shared sacred past and, importantly, open up a temporal feedback circuit between the Islamists’ present crusade and apocalyptical future. Like the peaceful denizens of self-capture culture, the digital intimate publics of Islamic extremism exchange self-curated and often intimate representations of themselves with others whom they have often never physically met. As this curation and sharing takes place, they constantly manage their onscreen identity in light of feedback systems of ‘likes’ and ‘shares’. It is here where McLuhan’s thoughts on self-amputation and self-amplification become relevant once again. For the battlefield selfie serves as an instructive marker of the complex and often opaque processes of identity realignment that occur when an individual transitions from peaceful Muslim to holy warrior. In the next section, I examine the various feedback loops and interplay between self-portraiture culture and the battlefield by dividing the jihadist selfie into three axes of formation – hipster, warrior and, finally, martyr. I use the term ‘axes’, as 572 Peter Mantello each may be present to a greater or lesser degree. Not all jihadists will promulgate a hipster persona, flaunt their battlefield killing, or publicize a desire to become martyrs. Accordingly, my analysis of these stages draws from the personal histories of Islamic extremists whose reputations were lionized and demonized as a result of their micro-celebrity practices. Violent versus non-violent battlefield selfies Curiously, while Western mainstream media tends to focus on the darker side of Islamic extremists, according to Klausen (2015) violent images from the battlefield constitute a tiny fraction of content that attains viral popularity within jihadi online communities. Arguably, the reason for the non-violent selfie’s popularity over its violent counterpart is that it shows an alternative version of jihad which challenges age-old stereotypes of the Arab world as alien, barbaric and hostile. Instead non-violent selfies portray a far gentler and more humanistic portrayal of the religious warrior (see Fig. 1). For example, Ingram, Whiteside, and Winter (2020) and Winter (2018) note that digital self-portraits of jihadist soldiers petting cute kittens (dogs are not haram), posing with jars of Nutella, making pizza, playing videogames, luxuriating poolside or going fishing attract far more popularity than battlefield combat portraits. Figure 1. Noncombative Selfie. Fatal portraits: The selfie as agent of radicalization 573 Another possible reason for the non-violent selfie’s popularity lies in its ability to normalize jihad as a productive lifestyle choice. For example, two years after Abu Bakr al-Baghdadi, the leader of the Islamic State of Iraq and the Levant, announced the establishing of a world-wide caliphate, social media was inundated with selfies by ordinary civilians living in Islamic-State-controlled regions of Syria and Iraq wearing Islamic-State-“approved” products. With permission of the Islamic State, local merchandizers had placed the extremist organization’s infamous black flag on consumer items ranging from sovereign rings and baseball caps to school bags and baby clothes (Wyke 2015).4 The transformation of an age-old theo-political symbol into a merchandizing logo illustrates the Islamic State’s intuitive understanding of brand as a political marketing tool. While local merchandizers enriched themselves in their commercial appropriation, the Islamic State heightened the visibility and public recognition of their crusade. Simons (2018) notes that a survey conducted in 2017 to find the most successful brand, ranked the Islamic State’s logo and symbols over the Vatican in terms of global public recognition. Like other ontological incongruencies in contemporary salafi jihadism, the commodification of an iconic symbol calling for an anti-enlightenment, may seem a disjuncture between belief and practice. However, as Simons (2018: 334) suggests, “[branding] is not necessarily solely about the group’s popularity, but more so about their public visibility and the resulting reputational brand as a means to connect and form relationships with different constituencies”. Indeed, the monetization of the Islamic State’s black flag resonated differently with various publics. For example, as Bandopadhyaya (2019) points out, to unaffiliated supporters far from the battlefield, producing selfies wearing black flag logo was proof of the Islamic State’s ongoing success at nation-building. To local populations it signified their allegiance to (whether authentic or not) the extremist organization. Finally, to those who may have understood yet dismissed the antithetical aspects of the black flag logo, it registered within them a deeper emotional connection to the Islamic State’s overall cosmic crusade. 4 Wyke, Tom 2015. The bizarre world of ISIS fashion: Terror group is now selling Islamic State-branded goods... with everything from sovereign rings and baseball caps to school bags and baby clothes. Retrieved on 16 July 2021 from https://www.dailymail.co.uk/news/ article-2982436/The-shocking-world-ISIS-fashion-children-wearing-ISIS-babywear-fightersshowing-cheap-rings-dodgy-snoods.html. 574 Peter Mantello Selfie as hipster holy warrior The jihadist as a hipster selfie has become a familiar trope within jihadist online networks but also an important cog in a propaganda machine that drives the cultural mythologies which shape and mold the contemporary jihadist imaginary (Helg 2014)5. A good example is the case of Islam Yaken, a twentysomething Egyptian who migrated to Syria to join the Islamic State and became a leading poster boy. The apex of fame for Yaken came in 2014, when he uploaded to his Twitter account a self-portrait riding a horse while waving a curved Arabian scimitar. Besides the AK-47 slung over his shoulder and a satellite-dish in the background, the only things in the photo that would date this image not in the Middle Ages were Yaken’s trendy cork-screw hair curls and designer glasses (Wood 2019). Yet the iconic status of this digital portrait had less to do with his personification of “jihadi cool” and more with the fusing together of past and future paradigms (Tibi 2010; Afsaruddin 2014). Similarly to the black flag, the warrior on horseback is a central and reoccurring motif in jihadi literature and Islamic eschatology. Horsch (2014) notes that the symbolic importance of the warrior on horseback originates with the first generation of Muslims during the golden age of Islam and stirs in Salafi jihadists’ religious feelings over the military conquests of Muhammad and his followers. Greater significance is lent to the sword-carrying warrior on horseback in the apocalyptical prophecies found in the Hadith which claim an army of horse riders led by ‘the Mahdi’ (or ‘the chosen one’) will fight the final battle against all unbelievers in either the town of Amaq or Dabiq6. Thus, the resonance of Yaken’s selfie lies in its iterability with a mythological crusade and the triumphant generation which was first represented by Muhammad’s army. Yet whereas this first group of Muslims established Islam, millennials who identify with Yaken’s self-portrait envision themselves as the last Muslims that will cleanse the world in an apocalyptical battle to end all wars, the war between good and evil. This vision is reiterated throughout the Islamic State’s strategic communication (McCants 2015). As Levin (2014) notes, “the selfie not only embraces the foundational mythologies of a civilization and/or a time, but it has the capacity to alter existing typologies and originate new ones”. Indeed, Yaken’s famed self-portrait illustrates a new genus of jihadist subjectivity based on a dialogic capacity to evoke in the present a nostalgia for a future-past. Massumi calls this mnemonic form of 5 Helg, Martin 2014. Der Hipster als Jihadist, NZZam Sonntag. Retrieved on 16 July 2021 from https://nzzas.nzz.ch/gesellschaft/der-hipster-als-jihadist-ld.146154?reduced=true. 6 Not coincidentally, Amaq is also the name of an Islamic State news agency while Dabiq is the name of their first glossy, English-language e-magazine. Fatal portraits: The selfie as agent of radicalization 575 interpellation ‘priming’. He explains, “Priming is memorial. It is the making active in the present of an inheritance from the past, [...] [Priming] is as much a call to the future as it is a recall of the past. [...] may be memorial, but it is also futuristic” (Massumi 2015: 108–109). Applications such as Instagram, TikTok, and SnapChat can be construed as personal repositories of identity construction, but also agents of priming, for they open up a temporal feedback circuit between the Islamist’s present crusade, war time past, and apocalyptical future. Figure 2. Shifting Identities of Islam Yaken. Interestingly, Yaken’s fame as an Islamic State’s poster-boy originated not from a conviction to wage violent jihad but rather a desire to carry over the effort at becoming an online fitness guru from his previous civilian life (see Fig. 2). In accordance with the celebrity determinant of cultivating a unique persona, Yaken’s micro-celebrity status began as a quirky yet relatable online fitness trainer for Islamic State soldiers. A combination of good looks and earnest demeanour saw him quickly amass a Twitter following of over 10,000 admirers (Wood 2019). Yet as his micro-celebrity status grew, the playful persona began to slip away and a much darker, more violent screen-mediated self emerged (El Naggar, 2015)7. Instead of holding his familiar set of dumbbells, Yaken started to appear holding the 7 El Naggar, Mona 2015. From a private school in Cairo to the Killing Fields in Syria. Retrieved on 20 January 2021 from https://www.nytimes.com/2015/02/19/world/middleeast/ from-a-private-school-in-cairo-to-isis-killing-fields-in-syria-video.html. 576 Peter Mantello decollated heads of Syrian regime soldiers. In line with McLuhan’s reinterpretation of the Narcissus myth and thoughts on self-amputation and self-amplification, Yaken had “adapted to his extension of himself and had become a closed system” (McLuhan 1994: 41). What was left of his original self was now numbed and standing in service to a more brutal mediatized other – the archetypal warrior. Selfie as archetypal warrior In jihadi culture, the sharing of celebrity jihadist selfies is often a new recruit’s first exposure to politics, let alone the politics of radical Islam (Ouzgane 2006). For sympathetic audiences living in poorer countries where ruling authorities have faltered in addressing socio-political grievances such as poverty, corruption, or unfair elections, much of the allure of the jihadist selfie lies in its hyper-masculinity8 and the possibility of radical change through performative violence (Aslam 2012). It is in impoverished and often wartorn areas of the world where the ocular-centric allure of the jihadist microcelebrity transcends regional language differences, tribal cultural barriers, centre-versus-periphery divides as well as huge educational gaps within the population. As Aolián (2016: 190) observes, […] men who cannot meet traditional expectations of masculinity – such as bread winner, respect and honor, wealth, access to sexual partners of choice – may precisely find that radical or extremist political mobilization offers a compelling substitute for regular masculinity authentication. Importantly, jihadi micro-celebritism symbolizes to the disenfranchised or marginalized male empowerment and heavenly rewards of jihad, so that displays performative violence become a vehicle to attain higher levels social capital. Arguably, this is why many male jihadists’ social media accounts are marked by greater levels of hyper-masculinity identity. It is not by coincidence that Islamic State and al Qaeda propaganda relies on gendered stereotypes, especially in context of the status and rewards afforded to men on their performance of violence; this can be witnessed in selfies which commemorate their battlefield killing (Aolián 2016). Besides condensing into a single visual frame an age-old discursive system of power relations which in war seeks to dehumanize the enemy and legitimize their 8 As regards hypermasculinity, I apply Angela Harris’ (2000) definition, “the exaggeration of male stereotypical behaviour, emphasizing physical strength, aggression and sexuality”. Fatal portraits: The selfie as agent of radicalization 577 murder (Jakob 2017; Carruthers 2011; Larson 2014), the selfie as a war trophy is now a standard rite of passage for Islamic fighters seeking to raise their social prestige. While this abject form of hyper-masculine memorialization is neither a recent phenomenon nor unique to Islamic extremists (Larson 2014; Jakob 2017), its highly personal yet collective nature is novel for it mirrors the same screen-mediated processes through which Papacharissi’s ‘networked self ’ comes into being. Unlike its analogue predecessor that was shared between members of personal and intimate communities (friends, family and comrades-in-arms) and governed by pre-established codes of conduct which ensured its legitimacy in place of familiarity (Carruthers 2011), the war trophy selfie, as a networked artifact, consciously anticipates affiliation through more peripatetic, anonymous and fluid systems developed within online networks. Purposed as a tool for both personal and collective identity building, the selfie as a war trophy oscillates between an object of “commemorative violence” and agent provocateur of “productive violence”. Digital self-portraits by necroinfluencers such as ex-German rapper Dennis Cuspert aka Deso Dogg, sitting comfortably with the severed head of an enemy on his lap brings “people together, galvanizing them in intensely emotional situations, rather than – or as well as repelling them” (Larson 2014: 33). On the other hand, they emulate the cultural logic, vernacular and practices of extremist micro-celebrity culture, because “for all its cruelty, it produces an extraordinarily potent artefact that compels our attention whether we like it or not” (Larson 2014: 34). In order for the jihadist to become “insta-worthy” and attain the degree of communal acquiescence required for “insta-fame”, he or she must abide by aesthetic practices and protocols that drive the rituals in impression management and personal branding of microcelebrity culture mentioned earlier. Another good illustration of a jihadist whose penchant for micro-celebritism allowed him to garner status as a celebrity foreign fighter is the ex-British rapper Abdel Barry aka L Jinn. Like other disillusioned and culturally fragmented second-generation Muslims living in the West, Barry’s path to radicalization was marked by an earlier life of drug use, run-ins with the law, and affinity for hip hop music’s more aggressive cousin, gangsta rap (Chulov, Sabbagh 2020)9. Similar to Deso Dogg, Barry saw strong affinities between gangsta rap and militant Islam – namely, a subscription to violence, armed insurrec tion and martyrdom as a means of rewriting perceived social and political injustices 9 Chulov, Martin; Sabbagh, Dan 2020. Spanish police arrest former British rapper turned Isis extremist. Retrieved on 21 April 21 2020 from https://www.theguardian.com/ world/2020/apr/21/spanish-police-arrest-former-british-isis-extremist. 578 Peter Mantello (Aidi 2014). His talent as a lyricist and singer saw him starring in several music videos released online with one of the songs receiving frequent airplay on BBC radio. Similar to Yaken, as Barry’s micro-celebrity status grew, so did his inclination toward more militant versions of Islam – at one point following the teachings of radical preacher Anjem Choudary. In 2013, the self-styled poet warrior put his confrontational lyrics into action and moved too Syria to take up arms, first with the Syrian Free Army and then with the Islamic State. Although submitting willingly to a dramatic lifestyle change, Barry did not relinquish his old media habits and microcelebrity desires. It was not long after arriving in Syria that Barry’s regular posts on Twitter of his battlefield experiences caught the eye of the Western media and alerted UK authorities who wrongly suspected him of being the infamous Islamic executioner Jihadi John. Yet Barry’s high point of infamy came when he posted to his Twitter account a self-portrait smiling with his arm around the severed head of a Syrian government soldier (see Fig. 3). Apart from the contrast between extreme savagery and relaxed intimacy so typical in this genre of battlefield photography, what makes this particular image so frightening is Barry’s accompanying title, “Chillin with my other homie, or whats left of him”, as it personifies the cultural familiarity of LOL culture and hip-hop slang so typically associated with Western hip-hop celebrities. Figure 3. Abdel Barry. Fatal portraits: The selfie as agent of radicalization 579 Critically, the selfie as a war trophy puts into sharp relief a merging ontology of violence that affirms the identity of the warrior while destroying the figural unity of (an)other. Italian philosopher Adriana Cavarero (2009) argues that albeit such forms of ruthless violence are normalized and (de)legitimated in the course of war, a linguistic vocabulary to describe or interrogate this reality is sadly lacking. Instead, she offers the neologism ‘horrorism’ to describe a form of political violence that fetishizes and revels in aesthetic destruction of the human body. In traditional political discourse, the philosopher explains, the words ‘terror’ and ‘horror’ are mistakenly used interchangeably for this kind of violence. However, Cavarero points out that terror signifies a type of fear that leads to instinctual body mechanisms of ‘panic and flight’, while the term horror translates to ‘shock and paralysis’ of the body, affected primarily through the gaze. She argues that the decollated head of an adversary is a symbol that the warrior is “not content with merely killing because that would be too little” (Cavarero 2009: 8). For the horrified liberal viewer, it becomes “an offense to the ontological singularity, dignity and uniqueness of being that posits the foundational basis of Cartesian subjectivity of (hu)mankind” (Cavarero 2009: 20). The war trophy selfie turns the ontological presets of Der Derian’s Virtuous War (2002) upside down – instead, reinstating the corporeal fragility of human life, the anguish and fear of those marked for execution, the brutality and savagery of those who take life, and the raw grotesqueness of human bodies once defiled and disfigured. At the same time, the external validation of the self through the erasure of an enemy’s ontological singularity and uniqueness articulates the processes of othering that underscore age-old relations of dominance and subordination in war. Whereas the non-violent hipster selfie is a celebration of jihad as a productive lifestyle choice, the selfie as a war trophy functions as a violent symbol of the archetypal warrior, a Manichean self-affirmation grounded in the annihilation of another’s identity. For the digital intimate publics of jihad, the sharing and re-dissemination of selfies allows community members to negotiate imagined and physical contours of sociability in their respective communities. At the same time, the sharing of the war trophy selfie makes publicly visible an individual’s adulation of the jihadist as war hero. Yet any social prestige arising from this kind of abject memorialization inevitably doubles as a powerful signifier of political instauration. In accordance with the hierarchy system of Islamic extremism, the pinnacle of social and cultural capital goes to the jihadist who engages in martyrdom. 580 Peter Mantello Selfie as a martyr Although commemoration of the fallen hero has traditionally been manifested through visual imagery (wood-block prints to paintings), aural practices (prayers and hymns) and social rituals (events and parades) (Leone 2015), the notion of someone aspiring to be a martyr and then engineering their own hagiography a priori is, in fact, a purely modern phenomenon closely tied to the selfie and microcelebritism. Originally purposed to promote and embellish hyper-individuated lifestyles, applications, such as Instagram, TikTok, SnapChat and Tumblr, ensure that the contemporary Islamic extremist’s metaphysical death wish is given visual form and material substance. By following the norms and conventions of selfcapture culture, the contemporary martyr exercises control over their physical body, freezing their corporeality within the flow of time. At the same time, the post-production capability of self-capture applications grant the martyr “instant nostalgia” to re-imagine themselves as if they were already dead (Jurgenson 2014). Although the word ‘martyr’ has become synonymous with the protocols of jihad until the crucifixion of Christ, the original meaning had nothing to do with dying for a cause. Rather, the word ‘martyr’ was first used in a legal capacity to refer to a person who bears witness (Bowerstock 1995), typically an onlooker or spectator of an event or incident. The term later applied to anyone who suffered adversity for their religious belief and then, finally, became limited to those persons who were condemned to death for their convictions. Today, the word ‘martyr’ is most commonly associated with people who kill and are killed for a religious or political cause (Cook 2007). Curiously, however, the contemporary martyrs’ penchant for snapping and sharing selfies of themselves and their victims’ violent demise circles back to the term’s ocular-centric etymology, in other words: the visual act and discursive practices of witnessing. Early martyrdom videos of the 1980s and 1990s married the personal and political. Convention dictated that a martyr would sit in front of a static camera and recount his or her personal justifications and purposes for their sacrifice and killing (Pantucci 2015). The video would then serve as a tool of indoctrination for future recruits but also as a testament to the fallen hero’s achievements as they benefitted the community. Critically, however, the martyr selfie aligns more closely to the violent self-experiential and liminal part of martyrdom. There are two major reasons for this development. First, the modern hagiography is no longer complete without images of the martyr’s victims. Second, in its real-time dissemination, the selfie reconfigures traditional notions of proximity and distance, making audiences and community members surrogates to the celebratory event. In this moment, the martyr’s embodiment and personal memory is externalized and replaced with Fatal portraits: The selfie as agent of radicalization 581 mediated experience. Through this mediated perspective, the viewer is not only witness to the martyr’s final demise but, increasingly, real time slaughter of the vulnerable and helpless. For example, Amedy Coulibaly made several videos to justify his future actions before filming his fatal attack on a Hyper-Marche in Paris. Both Mohameed Mera and Mehdi Nemmouche strapped Go-Pro cameras to their chests before embarking on fatal killing sprees against followers of the Jewish faith. Not long afterwards, Larossi Abballa broadcasted his murder of a Magnanville police couple on Facebook Live from a laptop at their home while sitting beside their twelve-year-old child. Months later, Omar Mateen tweeted, took selfies and constantly performed vanity searches on Twitter for news updates of his attack as he gunned down dozens of innocent partygoers at a gay disco in Orlando, Florida. Or, more recently, far-right extremists such as Norwegian Andres Breivik and New Zealander Brenton Tarrant have made their video-recording an integral part of their martyrdom. By shifting perspective away from their own corporeal body to the destruction of ‘killable bodies’ (Agamben 1998), the martyr transfers their final memories onto to the viewer10. This shift from corporeal self-reflection to liminal consciousness completes the nascent construction of a purely modern hagiography, accessible to the umma and adversary without spatial or temporal restraint. By assuming the point of view of the martyr, the spectator (re)witnesses the “truth” of sacrifice. The memoralization of digital hagiography unfolds through its communal imbrication across porous boundaries between admirers and adversaries. Clearly, then, the importance of the martyr selfie goes far beyond the formation and posterity of the jihadist’s atomized self. Rather, it performs a wider influential role in galvanizing collective identity, socializing community members and mobilizing participatory action of members in their communities. Instant nostalgia Prior to Web 2.0, an extremist could not hope to gain recognition until after his martyrdom. However, the post-production capability of self-capture applications allows the aspiring martyr to change the colour and light values of their photos based on analogue film stocks of previous eras. Nathaniel Jurgensen (2014) calls this feature ‘instant nostalgia’. For the prospective martyrs, the selfie functions as a mnemonic agent, bringing into the present a future that has yet to occur and helping them to imagine themselves as if they were already deceased. 10 For these reasons, I consider the cell-phone recording an externalized form of selfie that works (and in many cases is inspired) by the same semiotic principles and interpellative practices as the first-person shooter videogame. 582 Peter Mantello Figure 4. Early selfie by Dzhokhar Tsarnaev. One of the most (in)famous examples of the martyr selfie and filter function concerns Dzhokhar Tsarnaev, the surviving sibling of the Boston bombers. While still on death row, the mediatization of Tsarnaev’s infamy and ubiquity of his image circulating on the internet have already guaranteed him a prominent place in halls of Islamic extremist martyrdom. Of the many digital self-portraits created by Tsarnaev prior to his deadly attack, there are two which best illustrate the semiotics of nostalgia elicited by the faux-vintage filter. The first was used in a LA Times article on 27 June 2013 (Serrano 2013)11 in which Tsarnaev embellished his selfie using a black-and-white filter effect (see Fig. 4). Besides generating an emotional desire to connect the present with an era long past, for the prospective martyr the iconicity of the black-and-white filter lies in its subliminal associations tied to photographic representations of archetypal celebrity figures in the 20th century. 11 Serrano, Richard 2013. Boston bombing indictment: Dzhokhar Tsarnaev inspired by Al Qaeda. Retrieved on 18 December 2020 from https://www.latimes.com/nation/nationnow/lana-nn-boston-marathon-bombing-suspect-indictment-20130627-story.html. Fatal portraits: The selfie as agent of radicalization 583 Figure 5. Tsarnaev on the cover of Rolling Stone magazine. The second, and perhaps best-known, selfie of the Boston Bomber is the one which first appeared on front page of the New York Times on 5 May 2013, but only aroused controversy when editors of the Rolling Stone magazine decided to put the same self-portrait on the cover of their 17 July 2013 issue (See Fig. 5). Critics world-wide condemned the iconic music entertainment magazine’s decision for glamourizing terrorism and encouraging other terrorists, so its circulation was banned from many commercial outlets (Gabbatt 2013)12. For this photo, Tsarnaev chose a faux-vintage filter emulating the subdued yet warm colour spectrum of Kodak’s reversal films of the 1970s. His choice of using this trendy retro-look filter speaks of the bomber’s reputation as an ordinary and normal teenager. Indeed, the contradictions of Tsarnaev’s character, so emblematic of the jihadist imaginary, are 12 Gabbatt, Adam 2013. Rolling Stone’s controversial Dzhokhar Tsarnaev cover ignites heated debate. Retrieved on 20 January 2021 from https://www.theguardian.com/media/2013/jul/17/ rolling-stone-dzhokhar-tsarnaev-cover. 584 Peter Mantello further illustrated in his Facebook profile where he wrote ‘Islam’ as his world view, yet stated ‘career and money’ as his personal goal (Reitman 2013)13. Conclusion Clearly, then, the battlefield selfie is not simply a narcissistic tool for modern-day jihadists to gain greater social capital. Rather, it plays a much wider ontological role in galvanizing collective identity, socializing community members, and mobilizing the participatory actions of members in their communities. This article has mapped the various feedback loops and interplay between the highly material world of self-capture culture and the jihadist imaginary. As a sociotechnical artifact, the digital self-portrait enables the contemporary jihadist to traverse the boundaries between two seemingly opposing belief systems. I have argued that the selfie can also be construed as an instructive marker of the various stages of identity realignment that occurs in processes of a jihadist’s radicalization. Whereas the hipster selfie promulgates jihad and the battlefield as a productive lifestyle choice, the selfie as a war trophy reaffirms age-old power relations of domination and subordination in war, while the martyr selfie collapses the personal and political in a violent and unique form of hagiography that preserves and externalizes the jihadist’s violent final moments. This paper extends and nuances existing literature on the role of Web 2.0 in contemporary political violence as well as contributes to a nascent body of literature on the phenomenon of self-capture culture and the battlefield. Acknowledgements: This study is part of the project “Emotional AI in Cities: Cross Cultural Lessons from UK and Japan on Designing for an Ethical Life” funded by JST-UKRI Joint Call on Artificial Intelligence and Society (2019). (Grant No. JPMJRX19H6). Additionally, special thanks go to the reviewers of this article and my research assistant Tung Manh Ho for their valuable insights and comments. 13 Reitman, Janet 2013. Jahar’s world. Retrieved on 16 July 2021 from https://www.rollingstone. com/culture/culture-news/jahars-world-83856/. Fatal portraits: The selfie as agent of radicalization 585 References Abidin, Crystal; Brockington, Dan; Goodman, Michael K.; Mostafanezhad, Mary; Richey, Lisa A. 2020. The tropes of celebrity environmentalism. Annual Review of Environment and Resources 45(1): 387–410. https://doi.org/10.1146/annurev-environ-012320-081703 Afsaruddin, Asma 2014. Martyrdom in Islamic thought and praxis. In: Dominic, James; Houen, Alex (eds.), Martyrdom and Terrorism: Pre-Modern to Contemporary Perspectives. Oxford: Oxford University Press, 40–58. https://doi.org/10.1093/acprof: oso/9780199959853.003.0003 Agamben, Giorgio 1998. Homo Sacer: Sovereign Power and Bare Life. Palo Alto: Stanford University Press. Aidi, Hisham 2014. Rebel Music: Race, Empire, and the New Muslim Youth Culture. New York: Vintage Books. Andrews, William 2016. Dissenting Japan: A History of Japanese Radicalism and Counterculture from 1945 to Fukushima. Oxford: Oxford University Press. Appadurai, Arjun 2006. Fear of Small Numbers. Durham: Duke University Press. https:// doi.org/10.2307/j.ctv11smfkm Aolián, Fionnuala N. 2016. The complexity and challenges of addressing the conditions conducive to terrorism. In: Nowak, Manfred; Charbord, Anne (eds.), Using Human Rights to Combat Terror. Cheltenham: Edward Elgar Publishing Limited, 166–194. Aslam, Maleeha 2012. Gender-Based Explosions: The Nexus Between Muslim Masculinities, Jihadist Islamism and Terrorism. Tokyo: United Nations University Press. Auerbach, Yehudit; Ben‐Yehuda, Hemda 1987. Attitudes towards an existence conflict: Begin and Dayan on the Palestinian issue. International Interactions 13(4): 323–351. https://doi.org/10.1080/03050628708434682 Bakhtin, Mikhail 1973. Mikhail Bakhtin: The Duvakin Interviews. Lewisburg: Bucknell University Press. Bandopadhyaya, Suvojit 2019. Branding the Islamic State of Iraq and Syria. Global Media and Communication 15(3): 285–301. https://doi.org/10.1177/1742766519874380 Berger, J. M.; Perez, Heather 2016. The Islamic State’s diminishing returns on Twitter: How suspensions are limiting of the social networks of English-speaking ISIS supporters. Program on Extremism. Washington: George Washington University. Boorstin, Daniel J. 1983. The Image: The Guide to Pseudo-Events in America. New York: Routledge. Bowerstock, Warren G. 1995. Martyrdom and Rome. Cambridge: Cambridge University Press. Carruthers, Susan 2011. Media at War. (2nd ed.) New Jersey: Globe Press. https://doi. org/10.1007/978-0-230-34535-5 Cavarero, Adriana 2009. Horrorism: Naming of Political Violence. New York: Columbia University Press. Cook, David 2007. Martyrdom in Islam. (Themes in Islamic History 4.) Cambridge: Cambridge University Press. Der Derian, James 2001. Virtuous War: Mapping the Military-Industrial Media-Entertainment Network. Milton Park: Routledge. 586 Peter Mantello Djafarova, Elmira; Trofimenko, Oxana 2019. ‘Instafamous’ – credibility and self-presentation of micro-celebrities on social media. Information, Communication & Society 22(10): 1432–1446. https://doi.org/10.1080/1369118X.2018.1438491 Dobson, Amy Shields; Robards, Brady; Carah, Nicholas 2018. Digital intimate publics and social media: Towards theorising public lives on private platforms. In: Dobson, Amy Shields; Robards, Brady; Carah, Nicholas (eds.), Digital Intimate Publics and Social Media. London: Palgrave Macmillan, 3–27. https://doi.org/10.1007/978-3-319-976075_1 Driessens, Olivier 2013. The celebritization of society and culture: Understanding the structural dynamics of celebrity culture. International Journal of Cultural Studies 16(6): 641–657. https://doi.org/10.1177/1367877912459140 Droogan, Julian; Peattie, Shane 2018. Reading jihad: Mapping the shifting themes of Inspire magazine. Terrorism and Political Violence 30(4): 684–717. https://doi.org/10.1080/09 546553.2016.1211527 Durkheim, Emile 2002[1897]. La Educación Moral [Moral education]. Madrid: Ediciones Morata. Fisher, Ali 2015. Swarmcast: How jihadist networks maintain a persistent online presence. Perspectives on Terrorism 9(3): 3–20. Galloway, Alex; Thacker, Eugene 2007. The Exploit: A Theory of Networks. Minneapolis: University of Minnesota Press. Harris, Angela P. 1999. Gender, violence, race, and criminal justice. Stanford Law Review 52(77): 777–807. https://doi.org/10.2307/1229430 Hoffman, Bruce 2017. Inside Terrorism. New York: Columbia University Press. https://doi. org/10.7312/hoff17476 Horsch, Silivia 2014. Making salvation visible: Rhetorical and visual representations of martyrs in Salafī jihadist media. In: Dehghani, Sasha; Horsch, Silvia (eds.), Martyrdom in the Modern Middle East. Wurzburg: Ergon-Verlag, 141–166. https://doi. org/10.5771/9783956505430-141 Inglis, Fred 2010. A Short History of Celebrity. Princeton: Princeton University Press. https://doi.org/10.1515/9781400834396 Ingram, Haroro J.; Whiteside, Craig; Winter, Charlie 2020. The ISIS Reader: Milestone Texts of the Islamic State Movement. Oxford: Oxford University Press. https://doi.org/10.1093/ oso/9780197501436.001.0001 Jakob, Brooke J. 2017. Beyond Abu Ghraib: War trophy photography and commemorative violence. Media, War & Conflict 10(1): 87–104. https://doi.org/10.1177/ 1750635216636136 Jurgenson, Nathan 2019. The Social Photo: On Photography and Social Media. New York: Verso. Klausen, Jytte 2015. Tweeting the Jihad: Social media networks of Western foreign fighters in Syria and Iraq. Studies in Conflict & Terrorism 38(1): 1–22. https://doi.org/10.1080 /1057610X.2014.974948 Larabee, Ann 2000. Decade of Disaster. Urbana: University of Illinois Press. Larson, Francis 2014. Severed: A History of Heads Lost and Heads Found. New York: W. W. Norton & Company. Fatal portraits: The selfie as agent of radicalization 587 Leone, Massimo 2015. Propaganda mala fide: Towards a comparative semiotics of violent religious persuasion. Semiotica 207: 631–655. https://doi.org/10.1515/sem-2015-0057 Nacos, Brigitte. 2016. Mass-Mediated Terrorism: Mainstream and Digital Media in Terrorism and Counterterrorism. Lanham: Rowman & Littlefield. https://doi.org/10.4324/ 9781315641270 Manovich, Lev 2019. The aesthetic society: Instagram as a life form. In: Mörtenboeck, Peter; Mooshammer, Helge (eds.), Data Publics. Milton Park: Routledge. Marshall, David P. 2015. The genealogy of celebrity. In: Marshall, David P.; Redmond, Sean (eds.), Companion to Celebrity. Hoboken: John Wiley & Sons, 15–19. https://doi. org/10.1002/9781118475089.part1 Marwick, Alice E. 2015. Status Update: Celebrity, Publicity, and Branding in the Social Media Age. New Haven: Yale University Press. Massumi, Brian 2015. Ontopower: War, Powers, and the State of Perception. Durham: Duke University Press. https://doi.org/10.1515/9780822375197 McCants, William 2015. The ISIS Apocalypse: The History, Strategy, and Doomsday Vision of the Islamic State. New York: Palgrave Macmillan. McLuhan, Marshall 1994. Understanding Media: The Extensions of Man. Cambridge: The MIT Press. Michel, Lou; Herbeck, Dan 2001. American Terrorist: Timothy McVeigh and the Oklahoma City Bombing. New York: Regan Books. Morningstar, Cory 2019. The Manufacturing of Greta Thunberg. Norderstedt: Books on Demand. Nissen, Thomas E. 2015. #TheWeaponizationOfSocialMedia:@Characteristics of Contemporary Conflicts. Copenhagen: Royal Danish Defence College. Ouzgane, Lahoucine 2006. Islamic Masculinities. London: Zed Books. https://doi.org/ 10.5040/9781350220881 Pantucci, Raffaello 2015. "We Love Death As You Love Life": Britain’s Suburban Terrorists. Oxford: Oxford University Press. Papacharissi, Zizi (ed.) 2010. A Networked Self: Identity, Community, and Culture on Social Network Sites. Milton Park: Routledge. Porter, Roy 2004. Flesh in the Age of Reason. New York: W. W. Norton & Company. Prucha, Nico 2016. IS and the jihadist information highway – Projecting influence and religious identity via telegram. Perspectives on Terrorism 10(6): 48–58. Schickel, Richard 2000. Intimate Strangers: The Culture of Celebrity in America. Chicago: Ivan R. Dee. Senft, Theresa M. 2008. Camgirls: Celebrity and Community in the Age of Social Networks. (Digital Formations 4.) Bern: Peter Lang Publishing. Shehabat, Ahmad; Mitew, Teodor 2017. Distributed swarming and stigmergic effects on ISIS Networks: OODA Loop Model. Journal of Media and Information Warfare 10(2017): 79–109. Simons, Greg 2018. Brand ISIS: Interactions of the tangible and intangible environments. Journal of Political Marketing 17(4): 322–353. https://doi.org/10.1080/15377857.2018. 1501928 Tibi, Bassam 2012. Islam and Islamism. New Haven: Yale University Press. 588 Peter Mantello Torres-Soriano, Manuel R. 2010. The road to media jihad: The propaganda actions of Al Qaeda in the Islamic Maghreb. Terrorism and Political Violence 23(1): 72–88. https:// doi.org/10.1080/09546553.2010.512839 Thomas, Samuel 2013. Yours in revolution: Retrofitting Carlos the Jackal. Culture Unbound: Journal of Current Cultural Research 5(3): 451–478. https://doi.org/10.3384/ cu.2000.1525.135451 Turner, Graeme 2013. Understanding Celebrity. Newcastle: Sage. https://doi.org/10.4135/ 9781473957855 Veilleux-Lepage, Yannick 2016. Paradigmatic shifts in Jihadism in Cyberspace: The emerging role of unaffiliated sympathizers in Islamic State’s social media strategy. Journal of Terrorism Research 7(1): 36–51. https://doi.org/10.15664/jtr.1183 Weimann, Gabriel; Winn, Conrad 1994. The Theater of Terror: Mass Media and International Terrorism. London: Longman Publishing. Winter, Charlie 2018. Apocalypse, later: A longitudinal study of the Islamic State brand. Critical Studies in Media Communication 35(1): 103–121. https://doi.org/10.1080/152 95036.2017.1393094 Wood, Graeme 2019. The Way of the Strangers: Encounters with the Islamic State. New York: Random House Trade Paperbacks. Zelin, Aaron Y. 2014. The war between ISIS and al-Qaeda for supremacy of the global jihadist movement. The Washington Institute for Near East Policy 20(1): 1–11. Zuboff, Shosana 2019. The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. London: Profile Books. Фатальные портреты: селфи как инструмент радикализации Цифровой автопортрет, а, точнее, селфи на поле битвы, является популярным средством построения идентичности современного джихадиста. Подобно портретам, которые создают не склонные к насилию участники культуры селфи, автопортрет джихадиста представляет собой альтернативу картезианской идее единой и неделимой личности. Скорее это продукт социальных отношений и перформативных действий, созданный в диалоге с другими в контексте определенных социокультурных рамок и ожиданий. Однако, в отличие от ненасильственного варианта, ожидания от этого диалога строятся вокруг более широкой политической повестки, которая активно стремится переписать коллективную память об имперских исламских завоеваниях и привлекает религию для морального оправдания насилия. Важно, что селфи на поле боя позволяет джихадисту легко пересекать границы между двумя, казалось бы, противоположными системами верований. Несмотря на изобилие исследований, посвященных культуре селфи, платформам по обмену изображениями и микро-селебрити, до сих пор не изучено их всепроникающее влияние, касающееся практик на поле боя. Истоки этого явления лежат в личных историях ключевых исламских экстремистов, которые благодаря своему поведению на поле боля оказались одновременно обожествлены и демонизированы. При этом сегодня такие индивиды могут снискать славу в интернете, участвуя в культуре селфи – публикуя автопортреты, видео или посты в блоге. Другими словами, никогда публичная коммуникация солдата еще не была такой личной и при этом общественной. Fatal portraits: The selfie as agent of radicalization 589 Saatuslikud portreed: selvefoto radikaliseerumise toimeviijana Nüüdisaegsele džihadistile on digitaalne autoportree või täpsemalt lahinguvälja-selfi populaarne identiteediloome tööriist. Sarnaselt selfidega, mida teevad enesejäädvustamiskultuuri vägivallatud praktiseerijad, esindab džihadistlik selvefoto alternatiivi kartesiaanlikule ühtse ja jagamatu Ise formuleeringule. See on pigem ühiskondlike suhete ja performatiivsete tegude tulemus, mida moodustatakse dialoogis teistega väga spetsiifiliste ühiskondlikkultuuriliste raamistuste ja ootuste kaudu. Ent erinevalt selle vägivallatust teisikust keskenduvad antud dialoogi ootused avarama poliitilise tegevuskava ümber, mis püüab aktiivselt ümber kujundada kollektiivseid mälestusi islami imperialistlikest vallutustest ning kaasata religiooni kui võimalust moraalse korra kehtestamiseks oma vägivalla üle. Oluline on see, et lahinguväljaselfi võimaldab džihadistil hõlpsasti liikuda üle kahe näiliselt vastandliku ususüsteemi vahelise piiri. Kuigi enesejäädvustamiskultuuri, pildijagamiskeskkondade ja mikrokuulsusnähtuse kohta on olemas rikkalikult uurimusi, on nende kõikjaletungivat mõju ja praktiseerimist lahinguväljal vähe uuritud. Nähtus tuleneb võtmetähtsusega islami äärmuslaste isikulugudest, keda nende lahingus toimunud mikrosuunamudimistegevuse tulemusena on nii ülistatud kui ka demoniseeritud. Tänapäeval aga võivad samasugused isikud enesejäädvustamiskultuuris osalejatena saavutada kuulsuse internetis, postitades selfisid ja videoid või blogides. Teisisõnu, sõjamehe avalik kommunikatsioon pole kunagi varem olnud nii isiklik ja samas kollektiivne. 590 Elsa Soro Sign Systems Studies 49(3/4), 2021, 590–607 Urban-human faces and the semiotic right to the city: From the USSR propaganda machinery to the participatory city Elsa Soro1 Abstract. Now that the usage and meaning of urban spaces have been dramatically challenged by the global pandemic, several debates and reflections are going on around the manner in which cities – both as concerns the public and the private spaces – have been designed. The article observes how “urban-human face” representations have served different models of urbanity across times and cultures. Using a framework deriving from semiotics of culture, according to which the city represents a model of the world, the article attempts to interpret how portraits of faces have been modelling the city through different urban faciality mechanisms. The focus is on a sample of what we call ‘urban-human faces’, ranging from Soviet propaganda posters to the digital #selfiecity project. The expression refers to series of representations that bring together the city and the face. It can be argued that both the city and the face, produced at a specific historical and cultural moment, with their figurative and plastic elements, deploy the struggle for the city ownership and authorship. Nonetheless, the commensurability of the city and the face can be just based on the fact that both semiotic configurations represent an excess with the help of cartographic reproduction (the city) and the portrait (the face), respectively. A city can be represented by a face to the extent to which it is also multifaceted as a polylogue. On the basis of such instable commensurability, the article will ultimately attempt to bring together the semiotics of the face and the semiotics of the city. Keywords: faciality; urban semiotics; mediascape; selfies; selfiecities 1. Introduction: Cities and faces under (re)construction Since the beginning of the global pandemic, the images of the cities circulating in the media all over the world have been displaying dystopic scenarios made of vivid contrasts between empty spaces, for instance the traditional tourist and leisure 1 Department of Philosophy and Education Science, University of Turin, Via Verdi 8 – 10124 Torino, Italy/UOC (Open University of Catalonia), Spain; e-mail: elsa.soro@unito.it. https://doi.org/10.12697/SSS.2021.49.3-4.17 Urban-human faces and the semiotic right to the city 591 spaces deserted by visitors, and massive agglomerations of human and non-human actors, namely the queues at the supermarkets or the gathering of patients in the crowded hospital wards, together with the correspondent artefacts. During the lockdown imposed by the different governments all over the world, the public and commercial spaces, deprived of their dwellers and consumers, have been invoked in public debate and media descriptions as “ghostly”, or “surreal”, and, thus, such processes have emphasized the intrinsic social character of the urban texture. While the elements of the city directly related with the urban lifestyle (i.e. interaction, mobility, crowd activities, etc.) have been banned, a parallel process of covering and concealment involves the human face these days: the sanitary masks, recommended or imposed by the authorities cover the mouth and the nose, while the face shields cover the face from the top of the head to below the chin. On the basis of this, it can be argued that the current debate about the design and planning of cities in an hypothetical postpandemic scenario could be intermingled with a debate on the new aesthetic of the face: the visibility regimes of both the city and the human face have been brought to question: What must remain visible in a face in order to recognize a person? Which parts of the city and which aspects of urban life could still remain when the risk of an infectious disease threatens social life? However, the overlapping between the discourses on the city and those on the human face is not a novelty. The expression ‘the face of the city’, used both in academic and public debates, refers to the appearance of the city and evokes the fact that for a city, as a face, to be memorable it must leave a certain impression on the visitors. Sayings such as “the cities change their faces through times” or “cities with a human face” attribute to the city the features of a visage, something that changes over times, but, at the same time, refers to an identity that lasts and can be recognized despite those changes. There are studies that have already been delving into the ontological affinity between the city and the face by analysing the extent to which a comparison and an analogy can be found between the two notions (Deleuze, Guattari 1980; Black 2011; Mubi Brighenti 2019). Taking into account the existing research, this piece aims at contributing to the debate on the relationship between the city and the face from a different angle. Using a framework deriving from semiotics of culture, according to which the city represents a model of the world, the study attempts to interpret how the representations of human face and the conditions of their production reveal different cities’ ideologies across times and cultures. It is worth noting that the human face is a recurrent iconographic figure in urban landscape: for instance, in the global metropolis street artists use the urban 592 Elsa Soro walls as an informal and ephemeral canvas for portraits of the human face 2, while digital billboards in the city centres have been advertising products through the model human faces to catch the consumers’ attention. With this in mind, the piece will focus on a sample of “human-urban face” portraits, that is, a series of representations that bring together the city and the face in a meaningful composition. Arguably, each of them, produced in a specific historical moment, deploys the struggle for the city ownership and authorship with its figurative and plastic elements. The study proceeds from the assumption that the city is a contested object, whose significance is produced by different forces and tensions that struggle for the city ownership as shown by a vast literature3 in urban studies, geography and urban sociology that has been focusing on political struggles (Davis 1990; Hardt, Negri 2009), economic asymmetries and social justice (Harvey 1973; Schneider, Susser 2003), privatization of urban space (Atkinson, Blandy 2006), residential and spatial segregation (Massey 1996; Atkinson et al. 2004), social inequality (Skop 2006) and creation of plural social movements (Castells 1983; Young 1990). From a semiotic point of view, the work tries to embrace the urban conflict as an overlapping between different writings and rewritings of the urban text undertaken by different potential “authors” who aim to be the custodians of the urban narratives and the urban gaze. Covering a trajectory of a series of “urbanhuman faces”, the text will aim to contribute to bringing together the semiotics of the face with the semiotics of the city. The first part of the article will frame the topic within a literary review of the city as a semiotic object in a permanent tension between plurality and unity. The trajectory covering “urban-human faces” starts with the representation of the oversized face of the leader who overlooks the crowd in Soviet propaganda. In opposition to socialist ideology, the representations of the capitalist global city deploy the fragmentation of the power scattered on screen refractions. The face of the “leader” pops up on television screens in the private space of the citizens’ houses. Later on, with the advent and diffusion of the personal computer and other technological devices, the “real” city is recreated in online virtual worlds and “played” by face avatars. The trajectory is “finalized” by illustrating the “urban-human face” in the participatory rhetoric that reclaims a city co-owned by citizenship. 2 The best-known street artist who put the human face at the centre of his artistic production is JR. The human face is also a recurrent topic in protest street art, a recent example being the series dedicated to George Floyd, the black man killed by a police officer in the city of Minneapolis, Minnesota, U.S.A. in 2020. 3 The list of references gathers those most quoted in the field. Nonetheless it cannot be considered as exhaustive. Urban-human faces and the semiotic right to the city 593 The series does not aspire to be exhaustive – rather, it aims at tracking figurative and plastic patterns of the city iconographies through the face, by bringing together these two unstable semiotic objects. 2. The semiotic city 2.1. The citysphere The Tartu–Moscow School of Semiotics has extensively interrogated the structure of the city, understood as a semiotically charged space. Initially,4 the spatial concepts were only used metaphorically as an empirical description of culture. Accordingly, the city and its organization was firstly considered by Ivanov (1986: 9) “as a model of universal space”. Since the coining of the concept of the ‘semiosphere’ (Lotman 1984)5, as an analogy to and extension of Vernadsky’s (1926) concept of the biosphere, Lotman further interpreted the city as a territorial space for semiosis. In particular, in Universe of the Mind, Lotman (1990: 191) clarified how “[i]n the system of a culture’s symbols, the city has an important place. City symbolism can be divided into two main areas: the city as symbolic space and the city as symbolic name”. The former area refers to the relationship between the city and “the earth which surrounds it” (Lotman 1990: 191). According to Lotman, a city can be either “isomorphous with the state, and indeed personify it” (for instance Rome as the city and the world), or “be an antithesis to the surrounding world” (Lotman 1990: 191). Either way the city, according to the vision of the semiotics of culture, is animated by elements of tension and struggle. In his description of St.Petersburg, Lotman (1990: 193) underlined the dimension of “eternal struggle between elements and culture”, epitomized by the antithesis between water and stone, present in the myth of the Russian city. Such tension can be considered as isomorphic to that of the semiosphere one and the latter’s polylogue form in which different discourses, semiotic conflicts and relations are woven together, escaping a unique or even a bijective logic of meaning According to Lotman, the semiosphere is separated from the otherness by its boundaries, the internal topographical organization being characterized by a tension between continuity (with respect to the outside) and discontinuity. Such 4 See the Theses of the Tartu-Moscow School (Uspenskij et al. 2003) Juri Lotman’s “On the semiosphere” was first published in 1984 and was an attempt to describe the structural features of the semiosphere in broad terms. However, the concept of the semiosphere was first formulated by Juri Lotman in 1982 inspired by the works of the Russian geologist Vladimir Vernadsky on the noosphere and the biosphere of Earth. 5 594 Elsa Soro internal discontinuity of the semiosphere is due, in the first place, to the presence of certain saliences. Lotman and Uspensky (1977[1973]: 237) exemplify this with the example of proper names in mythology, where “space is not conceived as a sign continuum, but as a totality of separate objects bearing proper names. It is as if space were interrupted by intervals between objects and thus absences”. Such internal tensions can also be interpreted, at a semiosic level, as the combination between explosive and gradual processes. Such processes are not just understood as the alternation of asynchronous phases that mutually replace each other. Instead, such a tension can give account of a simultaneous process. The internal topography of the city can be described as highly discontinuous in different respects, both synchronically and diachronically speaking, both temporally and spatially speaking, considering both natural (for instance natural catastrophes) and anthropogenic phenomena (for instance, military sieges, gentrification processes), redesigning and reorganizing the internal urban forces at the level of semiosis. 2.2 .The semiotic right to the city The discontinuity in the city topography can also be attributed to the multiple transformations of the collective action in the permanent struggle for achieving a hegemonic position in the city governance. In the frontispiece of Thomas Hobbes’ Leviathan by Abraham Bosse (1651), the body of the Sovereign King is constituted by the blurring together of individual faces of the citizenry, who face the Sovereign to represent his dominant position over the city. In contrast, the slogan “right to the city”, proposed in 1968 by Henri Lefebvre, popularized by radical thinkers such David Harvey, and further recalled by social movements against globalization, refers to the demand for a different distribution of the city “ownership”. Harvey (2008: 23) defines the right to the city as the common right “to make and remake our cities and ourselves”. The making of the city implies the production of a significance that is always the result of a negotiation among different actors. An urban semiotics approach makes it possible to interpret such processes in terms of writing and co-writing of the urban text made by an overlapping of signification processes. According to Greimas’ seminal work on topological semiotics, the semiosis process is triggered by a negotiation of the different meanings and interpretations given to the city by its inhabitants (Greimas 1976: 138). The city geography is thus composed of a variety of elements (for instance roads, neighbourhoods, squares, bridges, malls, parks, etc.) and the “sum” of the manifold readings by its users. City dwellers or tourists can Urban-human faces and the semiotic right to the city 595 follow the instructions given by the informative structure of the city or contravene it through tactical actions (De Certeau 1990). The act of crossing the city and walking through it can be considered as an enunciation act by which the citizens instantiate their city and confront the dominant ideology of the city. Nonetheless, such an informative gesture of rebellion gives account of the unity of the city’s composition behind the plurality of elements that are intertwined within it. The above-mentioned tensions among the different city actors for ownership and authorship of the city are thematized by different textualities which compose and articulate the urban fabric: for instance, a monument can be considered as the inscription of a (new) memory intertwined with previous ones. As a city sign, an urban-human face portrait literarily provides the city with eyes that gaze and in turn are gazed upon. In the following, a trajectory along a series of “urban-human faces” will be outlined, focusing on the figurative and plastic elements of the face composition that unveil the underlying tensions concerning the semiotic right to the city. 3. The urban face of the leader in the Socialist city The overlapping between the face and the iconographies of power finds one of the first cultural manifestations in face gigantism. The Mount Rushmore National Memorial, with the colossal sculpture that features the 60-foot (18 m) faces of the US Presidents George Washington, Thomas Jefferson, Theodore Roosevelt and Abraham Lincoln is one of the most remarkable examples of representation of the power (in this case over a nation) through the face. Yet face gigantism can be considered as a cultural isotopy across historical ages and cultures: from the face of the Buddha swallowed by a tree’s gnarled roots, to the face of King Antiochus scattered around an empty plain in southeast Turkey around 60 BC, to the 180-foot high monument to Decebalus, the last king of Dacia, on the banks of the Danube River. The character of colossality entails the capacity of such faces to be seen at a distance, and by doing so also to overview the natural landscape. The anthropomorphization of the landscape creates a tension between the human presence and the natural elements to which the latter is exposed with a domination aspiration. When it comes to the city landscape, the domination is no longer projected to natural elements, but to a landscape composed (mainly) by other human figures, embodied by the notion of the urban crowd. 19th- and 20th-century literature, philosophy and art have extensively and in different ways problematized the dialectic between the individual and the crowd and their intersections in the context of urban life. Figuratively, the alienation and reification in the modern city 596 Elsa Soro have been represented by the accumulation and the repetition of indistinct faces. According to Benjamin’s vision of the modern city, expressed in his narratives in Das Passagen-Werk, the crowd is composed of an unremarkable repetition of multiple faces (Benjamin 1990: 674). Such repetition of faces is similar to the accumulation of items and goods stored in Émile Zola novel Bonheur des Dames where the “gigantesque étalage”, composed of “des pièces de lainage et de draperie, mérinos, cheviottes, molletons” overwhelm the protagonist Denise who has recently arrived in Paris from the province (Zola 1883: 761). Similarly, in Benjamin’s description of the ‘flâneur’, the latter is represented as someone abandoned in the crowd and thus as a victim of a commodity intoxication. In opposition to an indistinct agglomeration of miniaturized faces, the face of the person standing out in the crowd expresses a potential difference, and entails a sort of primus inter pares effect. Soviet propaganda often represented the act of standing out by using the technique of photomontage. Both in the poster Stalinists! Extend the front of the Stakhanovite movement! (1936) and in Varvara Stepanova’s photomontage The Results of the First Five-Year Plan (1932), the oversized face of the leader emerges from an undistinguished and anonymous crowd (Fig. 1). The charismatic leader is responsible not only for the crowd, but also for designing the new urban landscape, as depicted in the poster Glory to Great Stalin, the Architect of Communism by N. Petrov and Konstantin Ivanov (1952) on which Moscow, proclaimed by Stalin as the ideal model of the cosmos, appears as alive with pedestrians, cars and river traffic in the background. Figure 1. Stalinists! Extend the front of the Stakhanovite movement! Futerfas, 1936. (https:// digital.library.pitt.edu/) and The Results of the First Five-Year Plan. Varvara Stepanova, 1932. (AP Art History). Urban-human faces and the semiotic right to the city 597 4. The urban face of the capital in the global city If in the 20th-century Soviet regime the oversized face of the leader being opposed to the miniaturization of the masses thematizes a relationship of domination between the former who gazes over the latter, the images of the capital and the city display a multiplication and expansion of point of views. Already in the modern city, as described by Benjamin (1990: 537), the reflective surface of the windows “brings the open expanse” and thus deceives the eye of the flâneur. In the era of globalization, the illusion of the space expansion, accompanied by the multiplication of goods and products, has been detonated by capillary diffusion of the digital screens that reflect the augmented spaces in the urban landscape. The geographer David Harvey (1989: 66) describes the urban fabric, cultivated by the postmodern architecture movement6 as a fragmented “palimpsest” of different temporalities. The collage effect is also encompassed by the ‘media city’, a notion with a long academic trajectory, that has also been defined as ‘information city’ (Castells 1989). In the words of McQuire (2008: 22), it refers to “a heterogeneous spatial regime, composed by a new conjunction of media and architecture”. In such a new urban scenario, the augmentation of space is also caused by the overlapping between different regimes of speed: the speed of the body, the speed of vehicles, and the “absolute” light-speed of media and communication technologies (Virilio 1986). The geographical and symbolic centre of the city, where the charismatic leader in the 20th-century totalitarian regimes emanated his absolute power over the crowd as depicted in the propaganda representations, has been delocalized and the political influence is exercised by occupying the channels of communication. The link between being omnipresent on the media, or even the ownership of media groups, and the political influence can ben epitomized by the figure of Italy’s former Prime Minister Silvio Berlusconi, whose face appearing on his various channels of television became the symbol of the “media-ization” of power (Fig. 2). At the same time, Milano 2, the residential area and headquarters of the first Italian private television channel at the border of Milan, is the epitome of a decentred utopic urban space. 6 Postmodern architecture became a movement in the late 1970s, in response to the formalism of the International Style. Postmodernity is considered to be heralded by the reference and the ornament and the Disneylandization of the contemporary urban environment, epitomized by the American city of Las Vegas (Venturi, Scott Brown, Izenour 1977). 598 Elsa Soro Figure 2. Silvio Berlusconi featured for a TV programme. Even though symbolic urban spaces of consumption such as Times Square in New York City, Piccadilly Circus in London or Shibuya Scramble Crossing in Tokyo, still exhibit giant faces gazing upon the citizens (Fig. 3), the media city and its ideology have entered citizens’ homes. Figure 3. Piccadilly Circus billboards. Urban-human faces and the semiotic right to the city 599 Derrida (2002: 79) describes the “desire” to be at home created in the context of global city as follows: We are witnessing such a radical expropriation, deterritorialization, delocalization, dissociation of the political and the local, of the national, of the nation-state and the local, that the response, or rather the reaction, becomes: ‘I want to be at home, I want finally to be at home, with my own, close to my friends and family.’ […] The more powerful and violent the technological expropriation, the delocalization, the more powerful, naturally, the recourse to the at-home, the return toward home. The human-urban face, reproduced on the TV screens of each and every citizen, has been further augmented through its diffusion in the Internet. As another relational space of consumption, alternative both to public and private space, the cyberspace as an augmented substitute of the real sites increases the possibilities of the urban space and its point of views. Projects based on virtual reality such as the online virtual world Second Life, opened to public in 2003, have nurtured a parallel city created by the articulation of new forms of social collective interactions among facial avatars. 5. The urban face of the citizens in the participatory city The digital transformation which has, technologically speaking, been feeding the mediascapes described above is also considered to be the precondition for the rise of the so-called ‘smart city’. This concept has become a major topic in the recent discourse on city development (Crivello 2015) and has been widely discussed in and out of academia, by pointing out its dimension of “global discourse network” (Joss et al. 2019: 4). Strictly intertwined with sustainability and innovation, one of the storylines that has informed the global smart city discourse has been participation as a means to boost the democratization of the city management. In an allegedly smart city, the citizens are provided with several platforms and tools that supposedly are meant to enable citizens’ participation in different domains such as environment, economy, mobility, education and politics. ICT would thus empower the citizens to design and debate solutions about city challenges and issues and eventually to have a voice in the public debate, contributing to its governance (Simonofski et al. 2019). The social innovation discourse has popularized the notion as co-creation of urban living and promoted practices such as hackathons or data boot camps by stressing the need to design and develop new smart city applications in 600 Elsa Soro collaboration with citizens. On the company and business side, notions such as ‘co-production’, ‘prosumption’ and ‘peer-to-peer’ are some manifestations of the socalled sharing economy (or in its critical version, platform capitalism7) which refer to the enrolling of the users on the demand side of the market through supposedly disintermediated relationships. The concept of participatory art has been recently galvanized by city administrations at a global level as a tool for providing the community with a new sense of ownership of public space. Within this framework, the visualization of citizen’s faces in artistic practices has become the figurative counterpart of the participative smart city. In opposition to the photomontages of the totalitarian leader’s oversized face that stood out from an indistinct mass, and in opposition to the dispersion and fragmentation of power representations in the media city ideologies, the following projects articulate a facial participative discourse and, by so doing, a collective ownership of the city. The French artivist JR has focused in his artistic research around faces through a participative process. In fact, in several projects of his, faces of the citizens have wallpapered different emblematic spaces, such as the Israeli West Bank Barrier, the favelas of Rio de Janeiro, the Mexican border city Ciudad Juarez or the police station in La Goulette, Tunisia, burnt down during the revolution. The art historian Claire Bishop (2006: 1) defines the artist’s “interest in collectivity, collaboration, and direct engagement with specific social constituencies” as “the social turn” in contemporary art. In the case of JR, such a new form of participatory urban storytelling has the face portrayal pasted on city surfaces at its core. Entire faces can be pasted side by side, transforming a wall into a photo gallery (for instance in Face 2 Face8), or appear as fragments, as in the case of the resistant vinyl photos of women’s eyes in Women are Heroes9(Fig. 4). 7 The concept of ‘platform capitalism’ was introduced by the English scholar Nick Srnicek (2017) to describe how the foundations of the economy after the crisis of 2008 were being carved up among a small number of monopolistic platforms such as Airbnb, Uber, etc. 8 Face 2 Face, Israel and the West Bank. In the project JR paired up depicting Israelis and Palestinians in locations on either side of the Separation Wall. Face 2 Face was accessed at https://www.jr-art.net/projects/israel-palestine. 9 Women are Heroes, different locations. In the project JR portrayed women in their daily lives and pasted them in relevant urban locations, such as Rio de Janeiro favelas and the Kibera slum in Nairobi, Kenya. Women are Heroes was accessed at https://www.jr-art.net/projects/riode-janeiro. Urban-human faces and the semiotic right to the city 601 Figure 4. Face2 Face; Women are Heroes. JR (https://www.jr-art.net/projects). In the latter, the faces have been broken down and fragmented into their details to create a proper facialization of the city. The informal and uneven surfaces of the urban slums have been equipped with a gaze that looks at the spectators by means of giving visibility to what is traditionally excluded by the city’s visibility regimes. However, the sociologist Mubi Brighenti (2019: 16) argues that through experiments such as JR’s “the city of faces may end up obscuring more than clarifying what the face of the city is about [...]. If it exists, the face of the city cannot be reduced to a selection of some of its dwellers’ faces”. Although presuming that just the presence of citizens faces alone would enable real democratization processes might be simplistic, the figurative mechanisms make it possible to visualize the principle of the composition by creating a specific meaningful mise-en-scène. In this regard, the collage used by JR can be interpreted not just as a past technique, but, instead, as a composition method that is made up of different-sized pieces: the collage (of faces) city shows thus a mélange of different pieces, as a series of differences that have been brought together in a unique composition. Unlike the collage, the mosaic is a composition made up of pieces of the same size. In this line, Selfiecities10 (Fig. 5) also defends the idea of the web, and consequently a city, owned by users: “Contrary to earlier incarnations of the web that were focused on content created by professionals, companies, and organizations, we are now producing, sharing, or tagging massive amounts of our own images and videos” (Hochman, Manovich 2013, s.p.)11. According to such 10 The project explores a dataset of 3200 Instagram selfie photos shared in 5 global cities, and creates different visualizations of these photos with custom software. The website of the project at http://selfiecity.net/ was accessed in November- December 2020. 11 Acceded online 2020 at https://firstmonday.org/article/view/4711/3698 in NovemberDecember. 602 Elsa Soro rhetoric, Second Life avatars have been replaced by “real” citizens faces. The aim of the project led by Lev Manovich and his team at CUNY is to question what visual social media, and in particular Instagram, tells us about the relationship between the city and individuals. As opposed to Manovich’s previous projects12, this inquiry put the faces of city dwellers at the centre, by specifically investigating the style of self-portraits (selfies). Figure 5. Selfiecities, Bangkok. Manovich (http://selfiecity.net/). 12 Lev Manovich’s all projects can be browsed at http://manovich.net/index.php/projects/ tag:Project. Urban-human faces and the semiotic right to the city 603 The project collected 656,000 images downloaded on Instagram, in five cities (New York, Sao Paulo, Berlin, Bangkok, and Moscow) over a specific lapse of time (4–12 December 2013). The rudimentary automatic face analysis has been supplemented by human judgment, through the inspection by Mechanical Turk workers, in order to estimate the ages and genders of the people in the photos. The facial expressions of the selfies in a city have also been studied for instance examining which city smiles the most, with Bangkok smiling the most and Moscow the least. The project’s conclusions focus, among other things, on how, despite Instagram being a universal language, its usage changes according to the geography. From a merely composition point of view Selfiecities appears as mosaics of faces, which can be explored following visual patterns and rhythms across the sets. The miniaturized faces are contained in a grid, side by side, in a series. The pieces of the mosaic have the same, regular size and together they create a unified series which does not present either discontinuity or rupture. Unlike the digital maps that track communities’ clusters and mark distances and proximity among them (as well as visualizing inclusion and exclusion processes), the Selfiescities, apparently, advocates for denying hierarchies and categories, by displaying the gesture of posting a composite self-portrait. 6. Conclusion We have been observing the urban-human face(s) as configuration(s) that allow us to unveil the tensions for the semiotic “ownership” and “authorship” of the city. In the semiotics of culture, the city has been depicted as polyphonic and as a polylogue in an isomorphic relation with the semiosphere as a whole. Following this principle, the series of urban-human faces analysed diachronically in the piece should be embraced in a synchronic way as well. Therefore, the assignation of a face to a model of a city is always an incomplete operation as the city, from the point of view of semiotics, is the result (always open) of the writing and rewriting operations by different “authors” (for instance the “delegates” of a certain urban ideology) who, together, orchestrate conflicting urban narratives. Arguably, cities cannot have just a face unless we consider this as an unstable semiotics mechanism. Therefore, the commensurability of the city and the face can be based on the fact that both semiotic configurations represent an excess with a cartographic reproduction (the city) and a portrait (the face), respectively. A city can be represented by a face to the extent to which it is also multifaceted as a polylogue. The aim to represent the struggle for the ownership of the city by addressing the question to whom the city belongs is thus condemned to a fail, yet this study has tried to give an account of such a failure. 604 Elsa Soro To finalize the trajectory with a new start, a famous quote from Jorge Luis Borges El Hacedor could be employed – an epilogue that refers to a man who wanted to draw the world and populate the is world with all sorts of geographic elements such as provinces, kingdoms, mountains, bays, ships, islands, fishes, rooms, instruments, stars, horses, and individuals. Just before dying, the man discovers that “the patient labyrinth of lines traces the lineaments of his own face” (Borges 1960: 126). At present, cities and faces are both involved in processes of resemantization of their respective geographies. In such crucial times, following Borges’ image, we must not avoid the fact that, isomorphically speaking, the act of drawing a space entails drawing a face. Acknowledgements: This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No 819649 – FACETS). References Atkinson, Rowland; Blandy, Sarah (eds.) 2006. Gated Communities: International Perspectives. London, New York: Routledge. Atkinson, Rowland; Flint, John 2004. Fortress UK? Gated communities, the spatial revolt of the elites and time–space trajectories of segregation. Housing Studies 19(6): 875–892. https://doi.org/10.1080/0267303042000293982 Benjamin, Walter 1990. The Arcades Project. Cambridge: Harvard University Press. Bishop, Claire 2006. The social turn: Collaboration and its discontents. Artforum International 44(9): 24–24. Black, Daniel 2011. What is a face? Body & Society 17(4): 1–25. https://doi.org/10.1177/ 1357034X11410450 Borges, Jorge Luis 1960. El Hacedor. Buenos Aires: Emecé Editores. Castells, Manuel 1983. The City and the Grassroots: A Cross-Cultural Theory of Urban Social Movements. London: Arnold. Castells, Manuel 1989. The Informational City: Information Technology, Economic Restructuring, and the Urban-Regional Process. Oxford, Cambridge: Blackwell. Crivello, Silvia 2015. Urban policy mobilities: The case of Turin as a Smart City. European Planning Studies 23(5): 909–92. https://doi.org/10.1080/09654313.2014.891568 Davis, Mike 1990. City of Quartz: Excavating the Future in Los Angeles. London: Vintage. De Certeau, Michel 1984. L’invention du quotidien. Paris: Gallimard. Deleuze, Gilles; Guattari, Félix 1980. Mille Plateaux. Paris: Minuit. Derrida, Jacques, 2002. Echographies of Television: Filmed Interviews. Cambridge: Polity Press. Farinelli, Franco 2009. La crisi della ragione cartografica. Torino: Einaudi. Ferdman, Bertie 2012. Urban dramaturgy: The global art project of JR. PAJ. A Journal of Performance and Art 34(3): 12–26. https://doi.org/10.1162/PAJJ_a_00107 Urban-human faces and the semiotic right to the city 605 Greimas, Algirdas 1976. Sémiotique et sciences sociales. Paris: Éditions du Seuil. Hardt, Micheal; Negri, Antonio 2009. Commonwealth. Cambridge: Harvard University Press. https://doi.org/10.2307/j.ctvjsf48h Harvey, David 1973. Social Justice and the City. London: Arnold. Harvey, David 2008. The right to the city. New Left Review 53: 23–40. Hochman, Nadav; Manovich, Lev 2013. Zooming into an Instagram City: Reading the local through social media. First Monday 18(7), s.p. https://doi.org/10.5210/fm.v18i7.4711 Joss, Simon; Sengers, Frans; Schraven, Daan; Caprotti, Federico; Dayot, Youri 2019. The Smart City as global discourse: Storylines and critical junctures across 27 cities. Journal of Urban Technology 26(1): 3–34. https://doi.org/10.1080/10630732.2018.1558387 Ivanov, Vyacheslav 1986. K semioticheskomu izucheniyu kul’turnoi istorii bol’shogo goroda’. [On the semiotic research of the cultural history of the big city].Trudy po znakovym sistemam [Sign Systems Studies] 19: 7–23. Lefebvre, Henry 1991. The Production of Space. Oxford, Cambridge: Blackwell. Lotman, Juri M. 1984. On the semiosphere. Sign Systems Studies 33(1): 205–226. https:// doi.org/10.12697/SSS.2005.33.1.09 Lotman, Yuri M. 1990. Universe of the Mind: A Semiotic Theory of Culture. London: I. B. Tauris. Lotman, Juri; Uspenskij, Boris 1977[1973]. Myth – name – culture. In: Lucid, Daniel (ed.), Soviet Semiotics: An Anthology. Baltimore: Johns Hopkins University Press, 233–252. Massey, Doreen 1996. The age of extremes: Concentrated affluence and poverty in the twenty-first century. Demography 33(4): 395–412. https://doi.org/10.2307/2061773 McGrath, Brian 2006. Face city. Space and Culture 9(3): 235–244. https://doi.org/10.1177/ 1206331206289363 Mubi Brighenti, Andrea 2019. Face and the city. Body & Society 17(4): 1–25. https://doi. org/10.1177/1357034X19865941 Schneider, Jane; Susser, Ida (eds.) 2003. Wounded Cities: Destruction and Reconstruction in a Globalized World. Oxford: Berg. Simone, AbdouMaliq 2010. City Life from Jakarta to Dakar. New York: Routledge. https:// doi.org/10.4324/9780203892497 Simonofski, Anthony; Vallé, Troy; Serral, Estafania; Wautelet, Yves 2019. Investigating context factors in citizen participation strategies: A comparative analysis of Swedish and Belgian smart cities. International Journal of Information Management 56: 1–10. https://doi.org/10.1016/j.ijinfomgt.2019.09.007 Skop, Emily 2006. Urban space: The shape of inequality. Urban Geography 27(5): 393–396. https://doi.org/10.2747/0272-3638.27.5.393 Srnicek, Nick 2017. Platform Capitalism. Cambridge Polity Press. Uspenskij, Boris A; Ivanov, Vjacheslav. V.; Toporov Vladimir N.; Pjatigorskij Alexander. M.; Lotman, Juri M. 2003. Theses on the semiotic study of cultures (as applied to Slavic texts). In: Eng, Jan van der; Grygar, Mojmír (eds.), Structure of Texts and Semiotics of Culture. The Hague, Paris: Mouton, 1–28. https://doi.org/10.1515/9783110802962-002 Venturi, Robert; Scott Brown, Denise; Izenour, Steven 1977. Learning from Las Vegas. Cambridge, MA: The MIT Press. Virilio, Paul 1986. Speed and Politics. New York: Semiotext(e). 606 Elsa Soro Young, Iris Marion 1990. Justice and the Politics of Difference. Princeton: Princeton University Press. Zola, Emile 1883. Bonheur des Dames. Paris: Hatier. Городские и человеческие лица и семиотическое право на город: от механизма пропаганды СССР до города соучастия Ситуация, в которой использование и значение городских пространств было серьезно подорвано глобальной пандемией, породила множество дискуссий и размышлений о том, каким образом устроены города – как общественные, так и частные пространства. Статья рассматривает, как репрезентация «городского человеческого лица» стала основой различных моделей урбанизма в разные эпохи и культуры. Исходя из семиотики культуры, согласно которой город представляет собой модель мира, в статье предпринята попытка интерпретировать, как портреты лиц моделируют город с помощью различных механизмов градостроительства. В центре внимания оказывается то, что мы называем «городским человеческим лицом» – начиная от советских пропагандистских плакатов до цифрового проекта #selfiecity. Это понятие относится к ряду образов, объединяющих город и лицо. Можно утверждать, что и город, и лицо, произведенные в конкретный историкокультурный момент, с их образными и пластическими элементами, разворачивают борьбу за право на город и на его авторство. Тем не менее, соизмеримость города и лица может быть основана только на том факте, что обе семиотические конфигурации представляют собой избыток с помощью картографического воспроизведения (город) и портрета (лицо) соответственно. Город может представлять лицо в той степени, в которой он многогранен как полилог. Исходя из такой неустойчивой соизмеримости, статья в конечном итоге попытается свести воедино семиотику лица и семиотику города. Linlik-inimlikud näod ning semiootiline õigus linnale: NSVL propagandamasinast osaluslinnani Nüüd, mil globaalne pandeemia on linnaruumi kasutamisele ja tähendusele dramaatilise väljakute esitanud, on käimas mitmed debatid ja arutelud teemal, kuidas linnad – nii avaliku ja avalikkuse ees olevat privaatset ruumi osas kui ka privaatse ruumina – on kujundatud. Artklis täheldatakse, et “linlik-inimliku näo” representatsioonid on erinevatel aegadel ja kultuurides teeninud erinevaid linlikkuse mudeleid. Kasutades kultuurisemiootikast tuletatud raamistikku, mille kohaselt linn esindab maailma mudelit, püüatakse artiklis tõlgendada seda, kuidas näoportreed on linna modelleerinud, erinevate linlike näolisusmehhanismide kaudu. Fookuses on kogum nähtusi, mida me nimetame “linlik-inimlikeks nägudeks” ja mis ulatuvad Nõukogude propagandaplakatitest digitaalse #selfiecity projektini. Väljend osutab reale representatsioonidele, mis toovad kokku linna ja näo. Võib väita, et nii linn kui ka nägu, mis on produtseeritud konkreetsel ajaloolisel ja kultuurilisel hetkel, oma figuratiivsete ja plastiliste elementidega, sisaldab võitlust linna omamise ja autorsuse üle. Urban-human faces and the semiotic right to the city 607 Linna ja näo ühismõõtmelisus võib tugineda lihtsalt tõigale, et mõlemad semiootilised konfiguratsioonid esindavad liiasust, vastavalt kartograafilise reprodutseerimise (linn) ja portree (nägu) abil. Nägu võib esindada linna sel määral, mil see on polüloogina paljutahuline. Sellise ebastabiilse ühismõõtmelisuse põhjal püüab artikkel viimaks kokku tuua näo- ja linnasemiootika. 608 Ekaterina Velmezova, Kalevi Kull, Ene-Reet SignSoovik Systems Studies 49(3/4), 2021, 608–615 Jaan Kaplinski and his contacts with the Tartu-Moscow School of Semiotics Ekaterina Velmezova, Kalevi Kull, Ene-Reet Soovik Как тихо все в Кяэрику каким молчанием молчат окна лестницы и площадки молчит потрескавшийся асфальт под муравьиными тропами как где-то молчат все кто тут был говорил и читал наизусть стихи как молчат Юрий Михайлович Александр Моисеевич молчит Линнарт и Хальянд не скажет ни единого слова и где-то куда я еще не вернулся я еще там мартовское солнце режет-колет мои босые ноги а я молодой и влюбленный убегаю дальше и дальше по колючему снегу навстречу нашему соборному молчанию1 Kaplinski 2014: 63 Abstract. Jaan Kaplinski (1941–2021), Estonian poet, essayist and public intellectual, sadly passed away earlier this year. To commemorate him, we publish some excerpts from a conversation with him that was recorded in 2018 and in which, among other topics, we also talked about Kaplinski’s relationship with semiotics and his personal contacts with eminent scholars of the Tartu-Moscow School. Jaan Kaplinki (22 January 1941 – 8 August 2021), internationally acclaimed Estonian poet, essayist, and culture critic, recipient of the European Prize for Literature (2015)2 was known for his focus on global issues, including academic concerns. Kaplinski was born in Tartu and studied French language and literature at the University of Tartu, graduating from the five-year programme that was called 1 How quiet everything in Kääriku is with what silence / the windows the stairs and the grounds are silent the cracked asphalt keeps silent / under the ant paths how they are silent somewhere /everyone who was here spoke and recited poetry / how silent Juri Mikhailovich Alexander Moiseevich are / Linnart is silent and Haljand will not say a single word / and somewhere where I have not yet returned I am still there / March sun pricks and cuts my bare feet / while I young and in love run further and further / across the prickly snow towards our conciliar silence 2 See http://www.prixeuropeendelitterature.eu/kaplinski.html. https://doi.org/10.12697/SSS.2021.49.3-4.18 Jaan Kaplinski and his contacts with the Tartu-Moscow School of Semiotics 609 ‘French Philology’ in 1964. Besides publishing poetry, he penned fiction, dramas, texts of popular science and participated in discussions on linguistics, translation studies, Oriental studies (particularly as concerns the Far East), sociology and ecology.3 His notable role in mediating semiotic thinking in culture has prompted us to observe his activities in the context of semiotics in general. Canonical descriptions of the Tartu-Moscow School of Semiotics and its activities in the 1960s and 1970s often tend to limit the list of its members on the Estonian side to a single person – Juri Lotman. At a closer look, however, it certainly appears that such an account is far from complete, as there were several Estonian scholars representing different academic backgrounds who directly contributed to the work of the school during its heyday in the period of 1964–1974: Igor Černov, Linnart Mäll, Ivar Kull, Uku Masing, and others. There was also a circle of local academics influenced by the school, who were instrumental in transferring the ideas of the semiotic school into the Estonian academic discourse and thus contributed to the local intellectual support to the school. The latter included a group of young linguists who gravitated to the work of Huno Rätsep, who taught structural linguistics at the University of Tartu in the 1960s, as well as participants in the seminar called Generative Grammar Work Group. Listed among these scholars were Haldur Õim, Mati Hint, Arvo Krikmann, Tiit-Rein Viitso, Pärt Lias, Leo Võhandu and others – no to mention Jaan Kaplinski. Kaplinski was not a semiotician by profession or formal training, but he maintained important contacts with some of the members of the Tartu-Moscow School. Even though he would later become a renowned poet and author of both fiction and literary non-fiction, in the 1960s he entertained doubts about his choice of career, feeling uncertain whether to opt for a poetic or an academic vocation. He was certainly interested in linguistics, and there was much in his linguistic interests – which also involved reflections on texts and signs – that would bring him closer to the participants of the Tartu-Moscow School. He has also expressed regret that, instead of French, he did not enroll at the university as a student of Russian language and literature. Nevertheless, although Jaan Kaplinski did not become a linguist or a semiotician, some episodes from the history of the Tartu-Moscow School of Semiotics, as well as some semiotic concepts in general, are reflected in his literary work. He also had an opinion on how the concepts developed by the Tartu-Moscow semioticians could be applied to the study of Estonian culture and cultural history – work that in his regard was still waiting to be carried out. 3 On him and his writings, see e.g. Salumets 2014 in English; Soovik 2015a, 2015b, 2016 in Estonian. 610 Ekaterina Velmezova, Kalevi Kull, Ene-Reet Soovik In addition to the legendary Kääriku Summer Schools on secondary modelling systems, Kaplinski also participated in the significant biosemiotic meeting “Biology and Linguistics”, held in Tartu in February 1978. In 2000 he gave a talk at a seminar on semiotics of nature, also arranged in Tartu, and in June 2007 he delivered a keynote lecture “Two types of human communication” at the 9th World Congress of Semiotics in Finland. In the autumn of 2012, he contributed to the vivid academic scene of the Department of Semiotics at the University of Tartu by giving a lecture series titled “Languages, cultures, and thought”. Below, we reproduce, in English translation, some excerpts from the originally Estonian-language conversation with Jaan Kaplinski that took place at his country house at Mutiku, Estonia, on 7 January 2018 (for the full interview, see Kull, Velmezova 2018). When did your first come in contact with semiotics? Jaan Kaplinski: I got acquainted with semiotics when I was a student. But my first more significant contact with semiotics took place at a semiotics Summer School at Kääriku in 1964.4 I think Igor Černov was an important link between me and Lotman. At the time you published several scholarly articles in the edited collections of the generative grammar work group, didn’t you? Jaan Kaplinski: I did that a little earlier, linguistics still interests me now,5 if I could, I would do more… I have my own point of view here, something like a linguistic typology; I wrote about Celtic and Finno-Ugric parallels and so on. I then read a lot on Finno-Ugric and Celtic Studies ... But the orthodox structuralism that they were engaged in at the time did not suit me. It’s boring and, in fact, has no perspective, because it’s just plain wrong to approach language in this way. As far as semiotics is concerned, it really gave people an opportunity to do things that would otherwise have remained impossible. Did you attend Lotman’s lectures? Jaan Kaplinski: I did not regularly attend his lectures, but I have attended his talks many times. 4 Jaan Kaplinski wrote a review of this event for the journal Keel ja Kirjandus [Language and Literature] (Kaplinski 1964). 5 See Kaplinski 2020. Jaan Kaplinski and his contacts with the Tartu-Moscow School of Semiotics 611 Did you only go to the already mentioned Summer School at Kääriku or did you also visit some later schools? Jaan Kaplinski: I visited others as well. One day, it was in spring, there was a school at Kääriku that was devoted mainly to Oriental studies. But many of the participants of the semiotic meetings were present as well6. It was at Kääriku that I became in touch with them; then, when I worked at the university’s sociology laboratory [in the early 1970s], I went to Moscow to work at the libraries there. In Moscow, I visited Pyatigorskij, and after that I developed a close contact with him.7 Afterwards I visited him in London several times. What do you think was the most important thing that Lotman and the Tartu-Moscow School achieved? Jaan Kaplinski: One of their undoubted achievements was that they opened up the way to free thought, as those who were somehow connected with them in one way or another – like Linnart Mäll, for example – mostly weren’t semioticians at all. Pyatigorskij, in my opinion, was not a semiotician either... He was an Orientalist and a philosopher – or rather, a philosopher and an Orientalist. And me, too, I don’t really know semiotics, I’m a linguist. I am interested in the problems of semantics – for example, how and how often the names of body parts are used metaphorically in Eurasian languages. In the broadest sense of the word, this is, of course, semiotics – it is meaning that is being investigated here. Studying a language without studying meaning is no good. Did you read Saussure at the university? Jaan Kaplinski: Yes, it was not difficult for me: I spoke French well. ... In many cases, the [language] sign is really arbitrary. When we say, for example, ‘saba’, the Estonian word itself is unlikely to be associated with the tail as such. Or, for example, when we speak German: ‘das Auge’, ‘die Nase’, ‘der Mund’… Today, why in one case we have ‘die’, in another – ‘der’, and in yet another – ‘das’, is not determined by anything. There are many examples of arbitrariness. There is a phenomenon in the Estonian language that particularly interested me – this phenomenon is ideophony, it is my favorite topic in linguistics.8 If we open the Estonian etymological dictionary, we will immediately see the expression ‘phonetically motivated’ (‘häälikuliselt motiveeritud’). And I counted them, 6 This may have been a reference to the third semiotic Summer School which took place in Kääriku on 10–20 May 1968. 7 Thomas Salumets briefly evokes Jaan Kaplinski’s relationship with Lotman and Pyatigorskij as well (Salumets 2014: 58, chapter “Erudite critic of culture – a walking contradiction”). 8 See Kaplinski 2010. 612 Ekaterina Velmezova, Kalevi Kull, Ene-Reet Soovik these “phonetically motivated” stems comprise some 20 per cent or even more. This phenomenon is typical of the Finno-Ugric languages, as well as of Korean and Japanese languages, to some extent also of Chinese, and in the Dravidian languages there is a lot of ideophony too. It is even more common in Finnish than in Estonian – but in Finnish, ideophony is presented a little differently... You can speak Finnish dryly – not in an official-sounding, but just neutral manner. And you can speak brightly, using ideophonic words – as, for example, in a poem by Ilmari Kianto about the northern lights: “Ne leimuaa, ne loimuaa ...”. In what way, then, was Saussure important to you? Jaan Kaplinski: His book organized a lot for me. In the end, it all became so self-evident that I just got used to it – synchrony, diachrony... This even has a connection with Buddhist philosophy: everything is connected with everything. Each and every thing is a denial of all other things, so to speak… I have the feeling that many things could be studied in the framework of another paradigm, and I don’t know how much the semiotic paradigm allows to explain things that other paradigms do not. Yet apparently it allows quite a lot. More I cannot say. It is also interesting how semiotics makes it possible to combine so different phenomena as Jakob von Uexküll and Ferdinand de Saussure. To build bridges, interdisciplinary bridges – this alone is very important in today’s world of over-specialization. You have not been writing much about semiotics, but you have written a piece in Russian, entitled “Действительные числа” (Real numbers),9 in which you speak about the Tartu-Moscow School, and in addition, semioticians are mentioned in a poem you wrote in Russian10… Jaan Kaplinski: “Действительные числа” is a short story which is something of an expression of sympathy for all these people on my part. I have a somewhat Nabokovian aspiration in me to write about people who are a little crazy – ‘with a birdie’ (‘linnukesega’), as they say in Estonian. The protagonist of “Действительные числа” is just that. Why do you think there were so few Estonians who were drawn to Lotman? Jaan Kaplinski: Estonians kept to their own bubble, their Estonian bubble. In retrospect, this seems to have been wrong, and I was relatively internationally minded on my part. After all, I grew up on Russian literature, and it was a stupid 9 10 Kaplinski 2000. See the epigraph on p. 608 of this issue. Jaan Kaplinski and his contacts with the Tartu-Moscow School of Semiotics 613 thing to do to go to study French. … I should have opted for Finno-Ugric studies – I was quite energetic, I could have travelled across Russia visiting Finno-Ugric peoples and collecting material, such things are of a great value for the posterity. Or else I could have taken up Russian language and literature, in this case I would naturally have reached Lotman. But what happened, happened. … There were two places where one could seriously commit to studying philology – with Lotman and with Ariste11... And of course, Russian semioticians did not feel much noticeable interest in what was so important for Estonians, in our own culture, history, desire for independence – all this remained alien to them. Although “the semiotics of being Estonian” could be a very interesting research topic. What, in your opinion, was the most important or the most interesting thing about Lotman all in all? Jaan Kaplinski: His charisma! Many of his theoretical views were, perhaps, too schematic, he could hover too far off from facts, yet at the same time he was a wonderful specialist in the history of culture. This is the best thing he has done, and this is what will live the longest. And how easy it was to get through to him... And they should have addressed Estonian material – authors like Oskar Luts or Tammsaare as far as I am concerned, and studied Estonian material from this aspect... But none of them trusted themselves to do this… And they did not have many Estonian students, so they could have just employed Lotman’s methods, as it were, the way he treated cultural history, observe something in the evolution of Estonian culture from a semiotic perspective, and this has not happened yet. For instance, the opposition of and the relations between ‘one’s own’ and ‘the other’. Or take the oppositions that have determined the Estonian culture: the orientation to the West or the East, the self and the other and many other topics, Southern and Northern Estonia, for example. This work is not yet done. Lotman’s legacy has been travelling around the world, but as far as I know, our own, Estonian culture has not been studied from a semiotic perspective yet... Instead of a postscript “Lotman was right when he argued that prose is more complex than poetry,” Kaplinski (2004: 361) said in connection with his literary work. The essay “Kirjanduse tähendusest” (‘On the meaning of literature’) (Kaplinski 1968), one 11 Paul Ariste was a linguist, specialist in Finno-Ugric languages. 614 Ekaterina Velmezova, Kalevi Kull, Ene-Reet Soovik of the earliest examples of Kaplinski’s ties to semiotics, can be considered a publication in which the understanding of the role of literature and art is discussed from Lotman’s perspective in many respects. Later, Kaplinski would draw on his reflections on Ludwig Wittgenstein to write a treatise entitled 621 (Kaplinski 1992). One of the few people to have provided indepth analysis of this text was the Estonian theoretical physicist and philosopher Madis Kõiv (Kõiv 2001). In Kaplinski, he saw a potential disciple not so much of Lotman, but of Viktor Shklovskij, placing the renowned Russian formalist in the same – not just metaphorical, but also geographical – space with another mentor of Kaplinski’s, theologian and poet Uku Masing who is behind the figure of the Teacher in Kaplinski’s novel The Same River (Kaplinski 2009[2007]). This is what Kõiv wrote: It would be better, for instance, to have Viktor Shklovskij settle in the post-war Tartu, because in the case of Lotman, whose student he [Jaan Kaplinski] would have liked to be, some issues of incompatibility [“õmblemisraskusi”] would arise. Let there be Shklovskij in Lotman’s place, this would not mean any significant difference in the primary tuition delivered; after his withdrawal from Moscow, he could land in Hurda Street12 (“in exile”) right after the war, and, since in the immediate post-war years there would not have been a sufficient, and sufficiently well educated, Russian-speaking environment here, young K. as his student could serve as a good consolation for him; as for language, I suppose no difficulties would arise in case of two people so sensitive to it. (Kõiv 2001: 92) References Kaplinski, Jaan 1964. Semiootika suvekoolist Käärikul. [On a semiotics summer school in Kääriku.] Keel ja Kirjandus 12: 764–765. Kaplinski, Jaan 1968. Kirjanduse tähendusest. [On the meaning of literature.] Looming 3: 432–441. Kaplinski, Jaan 1992. 621. Looming 11: 1499–1544. Kaplinski, Jaan 2000 = Каплинский, Яан 2000. Действительные числа. [Real numbers.] Звезда: Литературный и общественно-политический журнал 11: 88–100. Kaplinski, Jaan 2004. Kõik on ime. [Everything is a miracle.] (Salumets, Thomas, ed.; Eesti mõttelugu 55.) Tartu: Ilmamaa. Kaplinski, Jaan 2009[2007]. The Same River. [Seesama jõgi.] (Wilson, Susan, trans.) London, Chester Springs: Peter Owen. Kaplinski, Jaan 2010. Ideofoonidest meil ja mujal. [On ideophones with us and elsewhere.] Keel ja Kirjandus 2: 133–136. Kaplinski, Jaan 2014 = Каплинский, Яан 2014. Белые бабочки ночи / Бѣлые бабочки ночи. [White butterflies of night.] Таллинн: Kite. 12 This is the street in which Uku Masing lived. Jaan Kaplinski and his contacts with the Tartu-Moscow School of Semiotics 615 Kaplinski, Jaan 2020. Eesti, estoranto ja teised keeled. [Estonian, Estoranto and other languages.] (Bibliotheca controversiarum.) Tallinn: Tallinna Ülikooli Kirjastus. Kull, Kalevi; Velmezova, Ekaterina 2018. Jaan Kaplinski ja semiootika. [Jaan Kaplinski and semiotics.] Acta Semiotica Estica 15: 194–211. Kõiv, Madis 2001. Nõnda lausus Jaan Kaplinski. [Thus said Jaan Kaplinski.] Looming 1: 73–101. Salumets, Thomas 2014. Unforced Flourishing: Understanding Jaan Kaplinski. Montreal: McGill-Queen’s University Press. Soovik, Ene-Reet 2015a. Kodukoht ja teeleidmine: Kaplinski kirjutatust. [Home and wayfinding: On Jaan Kaplinki’s writings.] Keel ja Kirjandus 58(2): 73–87. Soovik, Ene-Reet 2015b. Kahest aiast ja mõnest mullast: Kaplinski aiad ja peenramaad. [Two gardens and several soils: Kaplinskis gardens and vegetable plots.] Keel ja Kirjandus 58(11): 765–776. Soovik, Ene-Reet 2016. Kaplinski kaugused. [Kaplinski’s far-away spaces.] Keel ja Kirjandus 59(2): 114–126.