270
Massimo Leone
Sign Systems Studies 49(3/4), 2021, 270–278
Introduction:
Studying the ‘facesphere’
Massimo Leone1
1. A controversial object of study
Early comparative studies in academic scholarship on the meaning of the face show
a dichotomy between, on the one hand, physical anthropology and anthropometry
and, on the other hand, cultural anthropology and ethnology. The former tends to
privilege static measurements and determinism, the latter, dynamic observation
and contextual interpretation. The former often yields to racial prejudice and even
systematic racism, as well as to unethical research practices, whereas the latter
rejects such views and methods and emphasizes the role of sociocultural context in
the development of face types, their expressions, and the relative meanings, which,
however, sometimes results in ideologically and aprioristically denying any impact
of human biology on the meaning of the face.
As regards the positivistic trend, Goldstein 1936 (an article on the age-related
changes in the dimensions and form of the face) surveys the previous literature,
starting from early anthropometric comparative works such as Weckler 1866,
Hrdlicka 1900, Boas 1911, Hellman 1927, 1935, Connolly 1928, up to Huber
1931. Other early comparative works along this line – also often influenced by
racist prejudice – adopt an embryological perspective (for instance, Schultz 1920,
Schaeffer 1935), or an interspecies comparative approach (for instance, Krogman
1930, 1931a, 1931b, 1931c). This tradition of comparative anthropometry with
focus on face growing continues with Loth still displaying blatant racist prejudice
in the late 1940s: “It is very hard to study facial muscles on the living subject, and
is possible only with intelligent people with whom one can communicate to evoke
the desired facial expression…I…gave up this study in Uganda” (Loth 1949: 222).
In the following decades, overt racism disappears from literature on facial
comparisons, but determinism persists under other guises, for instance, in linking
the morphology of faces and facial expressions to climate conditions (Steegman
1
University of Turin, Department of Philosophy and Education Studies, Via S. Ottavio 20,
10124 – Turin, Italy; massimo.leone@unito.it.
https://doi.org/10.12697/SSS.2021.49.3-4.01
Introduction: Studying the ‘facesphere’
271
1965, 1967, 1970, 1972; Steegman, Plainer 1968; see also Coon 1965 on “the
living races of man”). From the late 1960s on, with a few marginal exceptions,
the universalist approach to the meaning of faces and their expressions moves
from anthropology to psychology (especially experimental, empirical, and, later
on, cognitive psychology) as well as to comparative ethology. The idea lingers
that some facial significations are shaped not by sociocultural influence but
by biological conditions, where the notion of race remains central (the trend
continues today; see, for instance, Watson and De Gelder 2017, claiming to have
experimentally found that “race influences affective body perception”).
The opposite trend, tending to stress cultural variability in types of faces and
facial expressions, has an early representative in Edward Burnett Tylor (1871: 66)
and its most influential herald in Franz Boas (“motor habits of groups of people
are culturally determined and [are] not due to heredity”; Boas 1938: 126). His
work then influenced the founder of kinesics, Ray Birdwhistell (1970: 38–39),
as well as Otto Klineberg (1940), David Efron (1941), Edward Sapir (1949),
Gregory Bateson and Margaret Mead (1942), Weston LaBarre (1947), and others.
Early semiotics developing from the mid-1960s onwards disregards the former,
biologically deterministic trend, tends to criticize it, and rather embraces the latter,
culturalist tendency, in line with the new discipline’s approach to meaning as an
exclusive product of social conditions. That is, for instance, the perspective adopted
by Umberto Eco from the mid-1970s on about the essentially cultural origin of
indexicality, including the facial one (Eco 1975).
Whereas the racist biases of the first trend of scholarship have been generally
recognized, and are now righteously widely stigmatized, the symmetrical biases
developed by the opposite trend as a result of the legitimate desire to counter the
first have been neglected. From Franz Boas on, cultural anthropology and then
semiotics have praiseworthily rejected the racist prejudices of early anthropometry
and physical anthropology, yet they have also tended to dogmatize the role of sociocultural contexts in giving rise to the meaning of the face. More fundamentally, the
second trend neglected to investigate the deep connection between the cultural
variability of facial signification and its biological roots.
2. Beyond an ideological dichotomy
The present-day cultural semiotics of the face is therefore urged to adopt a
novel perspective, which rejects racial determinism and its racist overtones, but
also refrains from any easy dogmatic explanation of the “cultural origin of face
meaning”. Early physical anthropology tended to explain the meaning of the face in
272
Massimo Leone
purely biological terms; early cultural anthropology and semiotics, to the contrary,
were inclined to do it in purely sociocultural terms. Contemporary semioticians
should reverse the perspective of the first trend without, however, severing culture
from its biological origins, which must be investigated not in terms of individual
faces but in those of facial clusters and ranges.
‘Cultures of the face’, indeed, means that faces are not only biologically common
and not only perceptually singular but that they can be gathered in clusters
which are independent from natural determination and, at same time, transcend
individuality and can be subsumed into types. These, however, are not somatic
but based on usages of the face that rely on language. ‘Cultures of the face’, from a
semiotic point of view, therefore implies that this part of the body can express its
own language, and that this language is also spoken by other faces. Following this
argument, faces, the non-verbal languages they speak, and the cultures to which
they give rise, can also be an object of translation. This does not mean that an
Italian face can be translated into a Chinese one but, more subtly, that semiotic
usages of the face in a society that predominantly speaks Italian can – and also,
in some circumstances, must – be translated into their equivalent – if any – in
Chinese-speaking societies and cultures.
Culture is not only the non-genetic memory of humankind but also what can
be translated of humankind, at least theoretically. The possibility of this translation
implies that there is a language composed by two planes and a certain articulation
of both, and that the expression plane can be rendered in a different way within a
different system, while retaining (almost) the same content. The translatability of
the face, and therefore its cultural nature, is evident especially in its representations.
Paintings, sculptures, and other artifacts representing the face are not actually the
body part itself, but just a figment of it. Yet facial representations are as they are
not only due to stylistic reasons, but also because they refer to distinct cultures of
the somatic face. Moreover, they retroact on actual faces, shaping and influencing
their cultures.
Intuitively, faces are neither only biologically common nor only ontologically
singular; on the contrary, they fall into categories that result not as much from what
faces are as from what they do, or what they are put to use to do. Contemporary
visitors of present-day Lithuania coming from other European countries, and
especially from the Mediterranean, commonly report as anecdotal evidence
that “Lithuanians do not smile”. Lithuanians themselves take it as an element of
their facial identity, make jokes on it, and propose various explanations, most of
which are quite far-fetched. The degree to which societies tend to smile affects
representations as well. Several inquiries focusing on big data of selfies shared on
social networks seem to indicate that there is a relation between geo-localization
Introduction: Studying the ‘facesphere’
273
and the degree of smiling. Some cities of the world happen to produce more
smiling selfies than others (see Lev Manovich’s www.selfiecity.net).
Proposed reasons for such discrepancies often reproduce trivial stereotypes. In
fact, explaining why a society smiles more than another is usually accounted for in
historical or genealogical terms (Lithuanians lived through Soviet repression; they
were struck by famine) or in causal terms (they live in cold climates). A structural
explanation, however, might be more convincing: in simpler terms, Lithuanians
smile less because other societies smile more, and smiling less is a way the face is
put to use in order to outline a somatic, emotional, postural, and gestural frontier.
In the end, face dialectics opposing societies and their cultures might structurally
work like the famous head gestures for affirmation and negation studied by Roman
Jakobson (1972): explaining them in causal terms is misleading; what matters is
the opposition, not the opposed terms. They are a cultural shibboleth, yet this
does not rule out that the human species might be characterized by a biological
inclination towards the production of such structural differentiations. The
particular facial terms in the opposition are probably socio-culturally determined,
yet the cultural mechanism of semiotic opposition itself between facial clusters
might be biologically shaped.
3. Patterns of face signification
Cultures of the face might work exactly along these lines. All the facial elements and
features that can be modified may give rise to oppositional patterns – that is, a great
number of them may. Either as discrete elements (presence/absence) or as nuanced
ones (through levels, gradients, and thresholds), they contribute to sketching the
frontier of a facial community. Discrete features usually mark the frontiers of a
‘facesphere’ more starkly. This is often the case with semiotic arrangements that
specifically modify the degree of visibility/invisibility of the face, especially if it
is done drastically. Female faces covered by strongly obtrusive Islamic garments
such as niqabs or burqas emphatically connote an urban area where this attire is
predominant. Similarly, visible face inscriptions like pigments, tattoos, piercings,
scarifications, etc. also decisively delineate the confines of a ‘face culture’.
These socio-cultural facial patterns, however, are easier to capture than other
subtler dialectics. Those that concern the dynamic appearance of the face (and
not only its permanent Gestalt), are harder to grasp. Frowning, for instance, is a
facial configuration that varies from culture to culture, not only in its expressive
composition (depending on which part of the face is predominantly used to
“frown”) but also in its content (Russell 1994). That shows the fractal nature of
274
Massimo Leone
the face in the facesphere: facial cultures are outlined through frontiers between
different ‘facial ways’, yet they too arise from analogous oppositions within the
face itself. These internal dialectics also often work in oppositional terms. What
matters in frowning, for instance, is neither the specific face part that is used in
this expression nor its specific content, but the expressive opposition between a
“smooth” face and a “striated” one (in Deleuze’s terms), which semi-symbolically
conveys the semantic opposition between something that the face finds pleasant
and something that, on the contrary, it finds regrettable. In this case too, the
corrugated face should not be explained causally – for instance, as a way to
withdraw the face from an unpleasant sight – but rather structurally, through
opposition with smoothness.
Differences in terms of how cultures not only present their faces but also
represent them are even more striking. Their variety can be appreciated both
synchronically and diachronically, sometimes with remarkable gaps. Cultural forms
of face representations indeed evolve faster than social ways of face presentation,
which in turn are faster than changes in the biology of the face. The gap between
biology and society is due to language, the one between society and culture is due
to media technology. The face is a medium too, yet its representation through
advanced media technology accelerates the pace and scope of cultural change.
Modalities of representing the face through photographs have undergone a fast
evolution, to the point that it is not hard to date immediately a photograph of a
face that was taken before WWII or to situate it in one of the following decades. At
the present time, the technical evolution of the modalities of face representation is
so fast that it is even possible to indicate the year in which a face photograph was
taken, judging by the particular technique of the representation. The latest model
of iPhone to date, for instance, allows owners to take photograph portraits that
characteristically focus automatically on the face, while leaving the background
visible but blurred.
This digital format is intended to bestow a special aura on the face, thus
coating previous formats of face representation, like the selfie, with a new, quasiartistic status. In the past, indeed, portrait photographs of this kind were usually
the product of accomplished photographers. As has been the trend in the last
decades, smartphones tend to miniaturize, automatize, and make portable formats
of face representation that were hitherto more exclusive. The selfie popularized
the self-portrait, which was theretofore difficult to perform without adequate
photographic technology; format filters do the same at a different level; the new
portrait function in iPhones, along the same line, enables everyone to execute an
“artistic” photographic portrait or self-portrait.
Introduction: Studying the ‘facesphere’
275
The massive diffusion of a new technological format leads to the rapid creation
of novel trends of face representation, which soon tend to become standards. The
process is further accelerated by digital social networks: the innovative format of
the “artistic” portrait is being increasingly adopted to represent faces in profile
pictures or in other forms of digital self-representation, giving rise to a bidirectional
reinforcing dynamic: on the one hand, people use it in order to attribute a special
aura to the picture of their face, gaining prestige by implicitly showing that they
can access the latest and currently most expensive technology; on the other hand,
the popularity of the new format makes it trendy: more and more individuals
adopt it in order to represent their face in a photographic portrait, implicitly
underling the aesthetic and emotional importance of the picture. Through these
face photographs that focus on the foreground and blur the background, social
network users indirectly communicate the distinction of such face representations
in relation to other face representations that do not adopt such a format. The
exclusivity and supposed prestige of the new technology is indeed always mitigated
by the immediate circulation of imitations through cheaper technology and brands.
The creation of an innovative technological format of face representation
results from the creativity of engineers and technicians but does not appear ex
nihilo. The idea of blurring the background and to focus on the foreground so
as to underline the importance of the face is old in photography, and does not
probably originate in this technique and its art. This strategy of representation
could be retraced back to J. M. W. Turner’s pictorial techniques of blurring and,
more remotely, to Leonardo’s experiments with subtly nuancing the background
of his paintings towards infinitely nuanced landscapes. Indeed, the new iPhone
12 does nothing else but reinterpreting an innate visual cognition modality, its
connate phenomenology, and its inherent semiotics through the new technological
possibility of miniaturization and automatization, including artificial intelligence
for face recognition. The visual cognition of focusing and blurring is engrained
in the neurophysiology of sight: humans spontaneously focus on objects that are
closer to the eyes, since these are a more likely source of either pleasure or danger.
This cognitive pattern, however, turns into a tendency to bestow importance to an
object that is in focus, especially if it is surrounded by an unfocused background.
The mechanism is quite simple: since we focus on what is important, what we
focus on tends to become important. Artists, especially in contemporary arts, have
been playing with this automatism of cognitive semiotics, often with ironic intents
and results. Applying such dialectics of focusing and blurring onto the particular
object of the face, though, inheres to a peculiar sociocultural trend, that is, the
emerging of the singularity of the individual in modernity. The portrait function of
iPhone12, therefore, technically originates from the astuteness of Apple engineers,
276
Massimo Leone
yet culturally evolves from the Renaissance, and translates a centuries-old aesthetic
trend – whose roots are actually in the neurophysiology of human sight and visual
cognition – into new technical ways.
The invention and diffusion of a new device, enabling an innovative technical
format of face representation, revamps an already existing trend and contributes
to its diffusion and consolidation. Ultimately, with the multiplication of instances
of face photographs adopting such a format, these give rise to a culture of face
representation that can be distinguished both spatially and chronologically from
other cohorts of facial images that do not adopt the same format.
4. Studying the ‘facesphere’
Studying the ‘cultures of the face’ therefore means investigating how biological
mechanisms, sociocultural conditions, fashion trends, as well as individual
choices continuously structure the ‘facesphere’ of a society, that is, the way in
which, in a human community, faces both individually and collectively are used
as language to signify an identity that emerges holistically and is in contrast with
other ‘facespheres’. In order to understand how this particular configuration of
the semiosphere works, measurements are not sufficient; indeed, quantitative
research on faces that disregards their qualitative observation leads to exceeding
determinism, often with the dangerous biases that have been exposed above; yet
qualitative observation of faces and face representations without measurements
also neglects to take into account an essential dynamic of the semiosphere, that
is, the emergence of new qualitative trends from quantitative accumulation,
for instance the concretion of a new facial pattern of signification through the
phenomenon of contagion and virality in the semiosphere.
Acknowledgements: This reseach is part of a project that has received funding from the
European Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation programme (Grant agreement No 819649 – FACETS).
References
Bateson, Gregory; Mead, Margaret 1942. Balinese Character. New York: New York Academy
of Sciences.
Birdwhistell, Ray Louis 1970. Kinesics and Context. Philadelphia: University of Pennsylvania
Press. https://doi.org/10.9783/9780812201284
Boas, Franz 1911. Changes in Bodily Form of Descendants of Immigrants. Washington:
Government Printing Office.
Introduction: Studying the ‘facesphere’
277
Boas, Franz 1938. The Mind of Primitive Man. New York: Macmillan.
Connolly, C. J. 1928 Growth of face in different races. American Journal of Physical
Anthropology 12: 197–206. https://doi.org/10.1002/ajpa.1330120112
Coon, Carleton S. 1965. The Living Races of Man. New York: Alfred A. Knopf.
Eco, Umberto 1975. Trattato di semiotica generale. Milan: Bompiani.
Efron, David 1941. Gesture and Environment. New York: King’s Crown Press.
Goldstein, Marcus S. 1936. Changes in dimensions and form of the face and head age.
American Journal of Physical Anthropology 22: 37–89. https://doi.org/10.1002/ajpa.
1330220104
Hellman, Milo 1927. Changes in the human face brought about by development. International Journal of Orthodontics, Oral Surgery and Radiography 20: 475. https://doi.
org/10.1016/S0099-6963(27)90222-1
Hellman, Milo 1935. The face in its developmental career. The Human Face: A Symposium.
Philadelphia County Dental Society Philadelphia: The Dental Cosmos; also in The
Dental Cosmos – A Monthly Record of Dental Science 77(7): 685–699.
Hrdlicka, Ales 1900. Anthropological Investigations of One Thousand White and Colored
Children of both Sexes, the Inmates of the New York Juvenile Asylum. With Additional
Notes on One Hundred Colored Children of the New York Colored Orphan Asylum. New
York, Albany: Wynkoop, Hallenbeck, Crawford Co.
Huber, Ernst 1931. Evolution of Facial Musculature and Facial Expression. Baltimore: Johns
Hopkins University Press.
Jakobson, Roman 1972. Motor Signs for ‘Yes’ and ‘No’. Language in Society 1(1): 91–96.
https://doi.org/10.1017/S0047404500006564
Klineberg, Otto 1940. Social Psychology. New York: Henry Holt. https://doi.org/10.1037/
13603-000
Krogman, Wilton Marion 1930. The problem of growth changes in the face and skull as
viewed from a comparative study of anthropoids and man. The Dental Cosmos – A
Monthly Record of Dental Science (June), 624–630.
Krogman, Wilton Marion 1931a. Growth changes in the skull and face of the Gorilla.
American Journal of Anthropology 47: 89–115. https://doi.org/10.1002/aja.1000470104
Krogman, Wilton Marion 1931b. Growth changes in the skull and face of the Chimpanzee.
American Journal of Anthropology 47: 325–342. https://doi.org/10.1002/aja.1000470205
Krogman, Wilton Marion 1931c. Growth changes in the skull and face of the Orang-Utan.
American Journal of Anthropology 47: 343–365. https://doi.org/10.1002/aja.1000470206
LaBarre, Weston 1947. The cultural basis of emotions and gestures. Journal of Personality
16: 49–68. https://doi.org/10.1111/j.1467-6494.1947.tb01075.x
Loth, Edward 1949. Anthropological studies of muscles of living Uganda Negroes. Yearbook
of Physical Anthropology 5: 220–236.
Russell, James A. 1994. Is there universal recognition of emotion from facial expressions?
A review of the cross-cultural studies. Psychological Bulletin 115(1): 102–141. https://
doi.org/10.1037/0033-2909.115.1.102
Sapir, Edward 1949. Selected Writings of Edward Sapir. (Mandelbaum, David G., ed.).
Berkeley: University of California Press.
Schaeffer, Jacob Parsons 1935. The ontogenetic development of the human face. In: The
Human Face: A Symposium. Philadelphia County Dental Society. Philadelphia: The
278
Massimo Leone
Dental Cosmos; also in The Dental Cosmos – A Monthly Record of Dental Science 77(5):
464–484.
Schultz, A. H. 1920. The development of the external nose in whites and negroes.
Contributions to Embryology 34: 173–190.
Steegmann, Albert Theodore, Jr. 1965. A study of relationships between facial cold
responses and some variables of facial morphology. American Journal of Physical
Anthropology 23: 355–62. https://doi.org/10.1002/ajpa.1330230409
Steegmann, Albert Theodore, Jr. 1967. Frost bite of the human face as a selective force.
Human Biology 39: 131–44.
Steegmann, Albert Theodore, Jr. 1970. Cold adaptation and the human face. American
Journal of Physical Anthropology 32: 243. https://doi.org/10.1002/ajpa.1330320212
Steegmann, Albert Theodore, Jr. 1972. Cold response, body form and craniofacial shape
in two racial groups in Hawaii. American Journal of Physical Anthropology 37: 193–221.
https://doi.org/10.1002/ajpa.1330370205
Steegmann, Albert Theodore, Jr.; Platner, W. S. 1968. Experimental cold modification of
craniofacial morphology. American Journal of Physical Anthropology 28: 17–30. https://
doi.org/10.1002/ajpa.1330280111
Tylor, Edward Burnett 1871. Primitive Culture. London: John Murray. [Republished in 1958
as The Origin of Culture. New York: Harper and Row.]
Watson, Rebecca; De Gelder, Beatrice 2017. How white and black bodies are perceived
depends on what emotion is expressed. Scientific Reports 7: 41349. https://doi.
org/10.1038/srep41349
Weckler, Hermann 1866. Kraniologische Mittheilungen. Archiv für Anthropologie 1: 89–160.
The transcendence ofSign
the face:
A semiotic-linguistic
path279–297
279
Systems
Studies 49(3/4), 2021,
The transcendence of the face:
A semiotic-linguistic path
Ugo Volli1
Abstract. This paper starts with an examination of the terms used to designate
the face in different languages, in particular in Italian, comparing these with
the definitions provided by some authoritative dictionaries as well as with their
etymology. This exploration yields some remarkable results: firstly, it appears that
the face is indeed a term that has a material meaning, but at the same time it is a
social object; secondly, the importance of the communicative function emerges,
which makes the face similar to the mask and in some ways to the arbitrariness
of language. All this suggests that the philosophical status of the face is that of
‘transcendence’ which is a condition of that state of freedom that we attribute to
ourselves and that can be defined as ‘human exception’.
Keywords: face; lexicon; social objects; metaphor; Levinas; transcendence
Introduction
Semiotics describes texts and analyses their conditions of use and the effects of
meaning they produce. This has obvious social utility, because it allows us
to classify and study communication occurring in a certain society. Used in a
more radical way, however, semiotic analysis allows us to study the social rules
of signification and thus to understand the mindframes prevalent in different
societies, because all communication in every culture is formulated according to
syntactic, semantic and pragmatic categories characteristic of that society. These
categories always involve a degree of arbitrariness, even in the extreme case in
which communication consists of simple material objects exhibited to indicate
themselves, as demonstrated by the inhabitants of Laputa in Gulliver’s Travels.
These objects are in fact assumed as signifiers, common names or prototypes of
the set of objects to which they belong in that culture. These prototypes, in turn,
depend on ontology or implied metaphysics (Quine 1960). Actually, everything that
1
Department of Philosophy and Education Science, University of Turin, Torino, Italy;
e-mail: volli.ugo@gmail.com.
https://doi.org/10.12697/SSS.2021.49.3-4.02
280
Ugo Volli
works as a sign – every message – is always brought to communication through
the mediation of the “primary modelling system” of its society, namely of language
(Lotman 1967). Language determines and limits the very possibility of signification,
even the silent one. A fundamental condition for semiotic research is leaving
behind of the naive ethnocentric naturalization of social objects and of posing the
problem of their origin and their transformations in relation with language, by
studying texts, but first and foremost by questioning the lexicon.
For this reason, it is indispensable that semiotics should include a philosophicallinguistic approach in its typological studies, at least if we define philosophy of
language not as analytic philosophy usually does (Beaney 2013: 13), but as a
type of research that, operating from within a language, attempts to describe and
understand, through the use of the language itself, the fundamental semantic choices
that are arbitrarily or historically made by its culture. So, in order to understand
what a certain society means and communicates through “social things” such as
‘love’, ‘religion’, ‘money’ (Ferraris 2009; Searle 1995), but also through “natural
things” that are socially semanticized and communicatively defined such as ‘child’,
‘dog’, ‘sea’, it is necessary to probe the words that describe and model them. They
work, actually, by cutting their object (often a set of objects unified in a social
category just by that word) out of world continuity; interrogating words means
trying to understand the cultural or empirical logic underlying this clipping. This is
certainly also true for such an entity as the ‘face’ which, as part of the human body,
is certainly material – yet it is also highly socially defined, as we will see soon. To
try to unravel the paradoxes and aporias that surround this socio-material object,
it is relevant to study the word and its use, which is done in what follows mainly
on the basis of the Italian language with some references to other languages.
An ill-defined entity
Let us start from a paradox. Lexemes corresponding to our ‘face’ are found in
numerous languages: linguistic literature does not mention examples of a lack of
relevant terms in any language. From a naive point of view, this word is very easy
to understand, everyone knows it in all contemporary major cultures – but it is
difficult to define exactly in precise conceptual terms. It is worth starting with the
definitions of the word offered in some dictionaries:
OED: “The front part of a person’s head from the forehead to the chin, or the
corresponding part in an animal.”
Merriam Webster: “The front part of the head that in humans extends from the
forehead to the chin and includes the mouth, nose, cheeks, and eyes.”
The transcendence of the face: A semiotic-linguistic path
281
Cambridge: “The front of the head, where the eyes, nose, and mouth are.”
Garzanti: “La parte anteriore della testa umana, dalla fronte al mento.”
Larousse: “Région limitée en haut par le cuir chevelu, sur les côtés par les oreilles et en
bas par le cou.”
Duden: “Besonders durch Augen, Nase und Mund geprägte Vorderseite des menschlichen Kopfes vom Kinn bis zum Haaransatz.”2
All these definitions, and the many other very similar ones that are found in other
languages, have a certain oddity to them. The face is defined as part of the head or
in some cases of the human body, characterized by the presence of certain organs,
such as the nose, eyes and mouth, and bounded by other physical units such as the
chin, forehead and ears. However, these smaller objects are also part of the face;
only in rare cases does the definition use external details such as the neck and the
hairline. The strangeness is due to the fact that if you look for the definitions of all
these parts or boundaries that define ‘face’, you are sent back to the face: a vicious
circle emerges. To give just a couple of examples, according to the Cambridge
dictionary ‘forehead’ is “the flat part of the face, above the eyes and below the hair”;
for Merriam Webster, ‘nose’ is “the part of the face that bears the nostrils and covers
the anterior part of the nasal cavity”, and so on.3 In conclusion, ‘face’ is defined
through its parts, and its parts through the face. This is certainly an example of a
very widespread problem of dictionaries, which in turn essentially depends on the
labyrinthine format of the semantics of natural languages and probably of human
cognition: it is a well-known problem in semiotics, widely discussed at least since
the observations made in the chapter “Dictionary vs. encyclopedia” in Eco 1983.
Formal semantics, cognitive sciences, artificial intelligence studies are also well
aware of this. Yet in this case the circularity is not extended, and thus vague, as in
the regular rhizomatic structure of lexical semantics, but appears immediately at
the first step and we are led to think that it depends on a difficulty concerning the
specific object.
Actually, the face is neither a distinct nor homogeneous anatomical reality: nose,
eyes, lips, forehead have different origins in the development of the embryo and are
made up of different tissues under the skin. The colours, the textures, the shapes of
these details vary to a considerable degree. The upper and lateral limits of the face
2
The definitions have been obtained from the following sources, last accessed on 22
February 2021: Oxford: https://www.lexico.com/definition/face; Merriam Webster: https://
www.merriam-webster.com/dictionary/face;
Garzanti:
https://www.garzantilinguistica.it/
ricerca/?q=faccia%201; Larousse: https://www.larousse.fr/dictionnaires/francais/face/32561?q=
Face#32473; Duden: https://www.duden.de/rechtschreibung/Gesicht_Aussehen_Miene_Sinn. La
3
Cambridge: https://dictionary.cambridge.org/dictionary/english/forehead; Merriam Webster: https://www.merriam-webster.com/dictionary/nose. Accessed on 22 February 2021.
282
Ugo Volli
are variable, because they depend on the way the hair is arranged. The different
parts of the face do not even have a common aspect. The face cannot be defined,
for example, as the whole hairless part of the head, because in many adult males
conspicuous parts of it would be naturally hairy due to cultural conventions. It is
worth introducing here a complex anatomical description proposed by the most
authoritative Italian encyclopedia:
Part of the man’s head located inferior to the front of the skull. It is continuous with
the base of the skull and is supported by a complex bony scaffold, consisting of the
upper jaw, the zygomatic arch, the lacrimal, the lower turbinates, the nasal bone,
the palatine bones and the lower jaw. These bones delimit cavities in which the
peripheral organs of the three senses (sight, smell, taste) and the initial tracts of the
respiratory and digestive systems are located. The f. is equipped with masticatory
muscles and numerous cutaneous muscles, which regulate the opening of the
various orifices present (e.g. lips, nostrils), move the eyelids and, by modifying the
skin folds, cause changes in expression. A single nerve, the facial nerve, innervates
all the muscles, while the sensitivity at the level of the skin is mainly due to the
branches of the trigeminal nerve. Vascularization is largely ensured by the facial
artery.4
In short, the face is not really a scientific object or a natural kind identified as
such by science (as are chemical elements such as iron, or animal species such
as the lynx, or organs such as the thyroid) (Daly 1998; Bird, Tobin 2008). It is
a cultural clipping of a real biological entity, the head, and, therefore, somewhat
puzzling as a material social object which, although it has a universal presence
in human populations, can also be extended to the animal world according to
some authoritative voices (Darwin 2009[1872]). Here, one cannot fail to notice an
important semantic tension between universality and arbitrariness.
4
“Parte della testa dell’uomo situata inferiormente alla parte anteriore del cranio. Si trova in
rapporto di continuità con la base del cranio ed è sostenuta da una complessa impalcatura ossea,
costituita dall’osso mascellare superiore, dallo zigomatico, dal lacrimale, dal turbinato inferiore,
dal nasale, dal palatino e dal mascellare inferiore. Queste ossa delimitano cavità nelle quali hanno
sede gli organi periferici di tre sensi (vista, olfatto, gusto) e i tratti iniziali degli apparati respiratorio
e digerente. La f. è dotata di muscoli masticatori e di numerosi muscoli pellicciai, che regolano
l’apertura dei vari orifizi presenti (per es. labbra, narici), muovono le palpebre e, modificando
le pieghe cutanee, determinano i cambiamenti di espressione. Un solo nervo, il facciale, innerva
tutti i muscoli, mentre la sensibilità al livello della cute è dovuta soprattutto alle diramazioni del
nervo trigemino. La vascolarizzazione è assicurata in massima parte dall’arteria facciale.” Source
available at: https://www.treccani.it/encyclopedia/face/ Last accessed on 22 February 2021.
Translations from Italian are by the author of the article, unless indicated otherwise.
The transcendence of the face: A semiotic-linguistic path
283
The surface of perception and communication
It is worth looking further into the definitions given above to try to understand
our object better. ‘Face’ is described through elements that are part of it. I have
already pointed out the paradoxical character of this kind of definition, which leads
to the difficulty of finding a notion’s semantic content by specifying autonomous
pertinent traits characterizing it. Yet if we take a closer look at the parts of the face
that are used to define it, we can find two groups: on the one hand, there are limit
elements, i.e. those parts of the body that mark the boundaries of the face from the
inside and only sometimes from the outside: the chin or the neck (downward); the
forehead or the hair (upward); the ears or the hair (sideways). On the other hand,
we find some objects that are on the face, which do not mark its boundaries, but are
its conspicuous parts: the mouth, the nose, the eyes and, again, the ears. It is worth
noting that these elements are only a subgroup of all notable parts of the face: no
definition mentions, for example, the cheekbones, the cheeks or the eyebrows.
Actually, the elements mentioned all serve as the site of sense organs, indeed
of all the localized senses: sight, smell, hearing, taste, and partly also touch, a
sense spread over the whole body. It is worth adding a consideration which is
important from a communicative point of view: these organs are not purely passive
receptors, which remain immobile just allowing themselves to be hit by sound, light
or chemical stimuli. In fact, each of them (with the partial exception of the ears,
which in humans are mostly immobile) acts to search, select and capture its own
stimuli. The mouth opens or closes, lips protrude or retract, the tongue moves and
explores, the cheeks swell or are squeezed in, exposing the teeth in order to bite or
hiding them, etc. The lids become closed or the eyes widen, orienting themselves
in one direction or another, stare sharply at something or wander distractedly. The
nose can also dilate the nostrils or become wrinkled.
These different activities are mostly natural consequences of choices of value,
following the logic of appreciation or rejection, of pleasure or disgust, of interest
or boredom. Yet at the same time, acting this way the sense organs express these
choices. These acts involve muscles and skin even beyond the actual location of
the sense organ: the wrinkling of the nose extends to the tension of the cheeks and
of the forehead; uncovering the teeth causes a visible contraction that reaches up
to the cheekbones, the cheeks, the chin. So, the whole face, or its major parts, is
involved in these motions, often allowing an organ to replace the others from an
expressive point of view: for example, we often and rightly speak of ‘smiling with
the eyes’. All these complex actions are visible and convey values. They are called
‘expressions’ because they project the person’s thymism outward: this is where the
semiotic terminology relating to the signifier comes from. From this point of view,
the face is the prototype of every axiologized support of signification.
284
Ugo Volli
In a species as strongly characterized by the density of social interactions and
the constant interest in synchronization with others as that of the humans, this
relationship between the activity of the sense organs and the perception of values
can easily engender voluntary expressions, communication of values, even beyond
the physical stimuli that generally give rise to the action. For this reason, the face,
and in particular the sense organs that reside there, provide the place where a
significant part of pre-technical human communication is concentrated: verbal
communication, because the mouth is obviously the location of speech and the
ears are the site of listening, but also all non-verbal communication: smiles, looks,
laughter, grimaces, surprises, blushing and turning pale etc. It is the body part via
which almost all the perception of the world enters the awareness of the subject,
as well as the one that sends out most of our messages. It is the most important
device for communicating with our fellow human beings. Therefore, it is the area
that others look at to identify feelings and mental states, and that we therefore try
to control better. Since human beings are used to knowing the face in detail, it is
also the most important identification interface, the one with the help of which
we recognize who is in front of us and that is portrayed in “identity documents”.
However, these empirical considerations never appear in the dictionary definitions
of the face, no doubt due to their complexity.
These definitions contain yet another focal point, namely the presence of a
directional indication: the face is the “forepart” of the head, often also referred to
as the “front” of the head.5 This clarification is also important, because it evidently
refers to the common experience of the indexical capacity of this part of the body,
that is, of its being directed towards something or someone. Turning your face
in a certain direction, that is, “facing it”, shows interest (which is etymologically
‘inter esse’, ‘being among’: facing something seems to imply metaphorically the act
of going near or into that object). What is faced becomes the privileged object of
attention and dialogue.
All these dictionary definitions therefore imply (but never explicitly
recognize) the role of the face as the input-output centre of communication
and human perception. There are undoubtedly evolutionary reasons why not
only humans, and not just primates or mammals, but a substantial share of all
multicellular animals – including amphibians, reptiles, fish, certain mollusks and
even insects – concentrate the organs of sense, nutrition and communication in
one single front area of the body – that is, towards the direction in which these
5
It is perhaps not without interest to note that the same directional indication of ‘front’ as
‘anterior’ incidentally has its etymology in a facial metaphor. The Latin ‘frons’, from which the
term derives, in fact meant ‘forehead’.
The transcendence of the face: A semiotic-linguistic path
285
animals predominantly move. This concentration is what all human cultures have
included in their lexicon with expressions that refer to the biologically artificial,
yet functionally founded notion of ‘the face’.
This strongly communicative dimension is also the reason why in different
cultures the face is also often used as an inscription support for make-up and
tattoos. Also, facial hair (beard, moustache, eyelashes, eyebrows) cut in different
shapes performs the same communicative function. All these elements amplify the
variable production of nonverbal communication by the muscles and skin. Face
inscription as a tool used to intensify and modify expressions is a technique well
known to mask makers and makeup professionals.
Furthermore, the recognizability of the face and its expressive function mean
that even its stable characteristics, such as the width and shape of the sense
organs and their details, the overall geometric shape, the colour, etc., are often
treated (naively, or according to various theories of “body language”) as bearers of
meaning. It is clear that this diagnostic attitude to the shape of the face also involves
several dangers: not only that of pseudoscience, but also racism, or at least perceptive
ethnocentrism, which is difficult to escape from, given that it is a powerful force
already at the level of animal evolution (Prum 2017).
What emerges from everything said above is a strong textual dimension of
the face, an intertwining of components that produce meaning according to a
clear semiotic articulation. It is easy to identify a plane of expression in which
the muscles, the skin, etc., make up the substance and what in common parlance
is called the ‘facial expression’ constitutes the form; and then there is a plane of
content whose substance is the thymic dimension of subjectivity and whose form
is its articulation in a semantics of passions dependent on the particular culture.
It is worth noting here that facial expressions are not to be examined in isolation,
because they are complex on the level of the signifier, but above all strongly
influenced by the context. An important example of this complexity is the “Kuleshov
effect” that received its name from the Russian director who showed in a series
of experiments that the same image of a character’s face with a neutral expression
was generally interpreted as sad, hungry, in love, etc., when edited together with
cinematic sequences that suggested a story in which he had to assume those
passions in order to fulfill his actant role (Riley 1998).
286
Ugo Volli
The linguistic comparison
These stratifications of meaning are expressed in a strong complexity of linguistic
signifiers that correspond to the concept of ‘the face’. They are very often related
to the idea of seeing and being seen (Greek πρόσωπον – pròsopon; French visage),
but also to taking a direction (Hebrew panim;6 ʭʩ ʑʰ˝ʕ ; Arabic ϪΟϭ wajh), to gaze
(German Gesicht), to form or shape (Latin facies, Polish twarz), to head (Spanish
cara), to beak (Spanish rostro), to the being in front (Chinese 面對 Miàn duì), to
cheek (Russian litso). This remarkable etymological diversity, which is also found
within rather narrow linguistic families that are often rather uniform in naming
parts of the body, also testifies to the cultural construction of the notion of ‘the face’.
It is worth investigating at least some of these differences, considering the case
of the Italian language, which is characterized by a notable richness of terminology
relating to the subject and whose derivations are generally well documented. The
three main lexemes in Italian that directly and denotatively refer to the face are
‘faccia’, ‘volto’, and ‘viso’. However, one of the most authoritative thesauri of the
Italian language, Dizionario dei sinonimi e dei contrary, lists 18 words for ‘face’,
some of which are rather rare or connotative, the latter often taken from the
nomenclature of animals: “faccia, viso, volto, muso, grinta, grugno, grifo, ghigna,
ceffo, mostaccio, mutria” (Gabrielli 1967: 286), while on a more abstract side, there
is a second part of the list with words that allude transparently to image, look and
form: “fisionomia, sembiante, fattezze, lineamenti, tipo, profilo, sgorbio, effige”.
The etymologies of the three main synonyms are quite clear. ‘Faccia’ comes
from the “Latin facies, ‘shape, look, face’, akin to facĕre ‘do’”; ‘viso’ from the “Latin
vīsus -us, properly ‘sight, look, aspect’, derivative of vidēre ‘to see’”. Things are
more complicated only concerning ‘volto’. There is an evident relationship with
the Latin “vŭltus, ‘face, aspect’”,7 which means ‘face’, but also ‘look’, ‘expression’,
‘physiognomy’, even ‘portrait’, as well as ‘side’, ‘surface’, and has a form ‘voltus’
(Campanini-Carboni 1993: 1735). This Latin word could have a close relationship
with ‘volutus’, the perfect participle of ‘volvere’ (‘rolling’), as indeed in Italian there
is a word ‘volto’ (‘facing’, ‘addressed to’), the past participle of ‘volgere’ (‘turn’),
which derives directly from that Latin verb. It could therefore be related to our
name ‘volto’ (‘face’). In this case the etymological meaning of this word would be
that of ‘being turned’ in a certain direction, heading towards it, or offering a certain
side of the head to the eye. Some believe that the two words have no relationship
and the ‘volto’ that indicates the face can be “traced back to the root *uel, ‘to see’”
6
For a careful examination of biblical Hebrew, I refer to Volli 2020.
The sources: https://www.treccani.it/vocabolario/face_%28Sinonyms-e-Contrari%29/;
https://www.treccani.it/vocabolario/volto/; last accessed on 22 February 2021.
7
The transcendence of the face: A semiotic-linguistic path
287
(Cortellazzo, Zolli 1988: 1452). Others argue for the convergence of the two forms
(Colonna 1997: 412). At the emergence of Italian literature, Dante Alighieri was
already aware of this ambiguity and made it the subject of a famous paronomastic
pun:
...e non mi si partia dinanzi al volto
anzi ‘mpediva tanto il mio cammino
ch’i’ fui per ritornar più volte volto.
(Inferno, I, 36)8
It is worth mentioning the general comparison that the Treccani Dictionary of
Synonyms makes of these terms and of the other more connotative ones, as this
certainly clarifies the linguistic subtleties involved:
Among the terms that indicate the front part of a human head, faccia is the most
common and familiar one (and in fact it is used in numerous expressions, phrases
and idioms […]); viso has a slightly more sustained tone; volto is more literary
[…]. Other terms are available in a derogatory or ironic tone: ceffo, grugno and
muso. Ceffo is the most offensive term of the three, and is used to indicate a sinister,
unpleasant or disreputable person […]. Grugno is properly the snout of the pig, and
is used in a figurative sense to indicate both an ugly appearance [...], someone who
one would gladly punch [...], and a sullen or angry expression [...]. Muso is a term
that designates the front part of the head of any animal, and can be metaphorically
extended to humans both in a derogatory sense [...] and as a term of endearment
(especially with the altered musetto and musino).9
It is worth pointing out that ‘ceffo’ comes “from the old French chief, that is ‘head’”;
‘grifo’ “from the Latin grȳpus, meaning with a big nose”, which, however, is the
name of a bird of prey that we still call ‘griffon’; ‘grugno’ “from the Latin grunium,
probably from grunnīre ‘grunt’; to compare with the French groin ‘snout’, Romansh
8
“refused to back away from me but so impeded, barred the way, that many times I turned
to go back down.” (translation by Robert Hollander and Jean Hollander)
9
https://www.treccani.it/vocabolario/faccia-finestra-di-approfondimento_%28Sinonimie-Contrari%29/: Tra i termini che indicano la parte anteriore della testa dell’uomo, f. è quello
più com. e fam. (e infatti è usato in numerose espressioni, locuz. e modi di dire) […] viso è di
tono leggermente più sostenuto, volto è più lett.[…] In tono spreg. o iron. sono disponibili altri
termini: ceffo, grugno e muso. Ceffo è il termine più offensivo dei tre, e si usa per indicare una
persona dall’aspetto sinistro, sgradevole o poco raccomandabile[...]. Grugno è propriam. il muso
del maiale, e viene usato in senso fig. per indicare sia un brutto aspetto[…] Muso è infine il
termine che designa la parte anteriore della testa di qualunque animale, e può essere esteso
metaforicamente all’uomo sia in senso spreg. [...], sia in senso vezzeggiativo (soprattutto con gli
alterati musetto e musino. Last accessed on 22 February 2021.
288
Ugo Volli
gruogn ‘chin’ and Romanian grui ‘top of the hill, hill’”; and that, at last, ‘muso’
derives from the “late Latin mūsu(m), of unknown etymology”; but it is worth
adding that “according to some [it comes] from (vultus) musus (‘prominent mouse
face’),10 from muricu, extracted from mus (‘mouse’). According to others from
morsus, caming from a Germanic root *mu” (Colonna 1997: 254).
Also, in the classification of the Treccani Dictionary we can see a division into
two parts of the semantic field, one being more characteristic, connotative and
metaphorical, and the other more abstract and denotative:
By faccia, viso and volto you may wish to mean the outward appearance, facial
features in general (his faccia tells me something, maybe we have already met?). In
this case we can speak of aspetto, [‘appearance’] (it has a familiar aspetto), fattezze
[‘features’] or fisionomia [‘physionomy’] (all three more general terms, which can
refer to features that are not only facial [...] or, more specifically for the characteristics
of the face, of lineamenti [‘outlines’]. [...] Aspetto [‘appearance’], more generically and
less directly than faccia, also refers to the way of posturing, of appearing. Apparenza
[‘appearance’], aria [‘look’] or espressione [’expression’] are also used in the same
meaning. […] Atteggiamento [‘posture’] is rarely limited to the outward appearance,
mostly indicating an overall inner way of being that also transpires outwardly [...].
Cera [‘mien’] is more specific for health conditions: having a good or bad cera
indicates the complexion and facial expression of those who feel good or bad.11
The Italian semantic field of ‘face’ is thus characterized in three layers. First of
all we have a nucleus of three words (which in many other languages are one,
as in the English ‘face’): ‘faccia’, ‘viso’, ‘volto’, which are largely overlapping, but
not perfectly equivalent – you cannot say *‘viso a viso’ as you say ‘faccia a faccia’
(‘face to face’); the ‘sacro volto’ (‘sacred face’) is not the *‘sacra faccia’; there are
affectionate diminutives of ‘faccia’ and ‘viso’ (faccina, visino) but not of ‘volto’, etc.
10
Sources: https://en.wiktionary.org/wiki/ceffo; https://it.wiktionary.org/wiki/grifo;
https://www.treccani.it/vocabolario/faccia-finestra-di-approfondimento_%28Sinonimi-eContrari%29/; https://www.garzantilinguistica.it/ricerca/?q=muso https://en.wiktionary.org/
wiki/grugno were accessed on 22 February 2021.
11 Con f., viso e volto si può anche voler intendere l’aspetto esteriore, i tratti facciali in generali (la
sua f. mi dice qualcosa, forse ci siamo già incontrati?). In questo caso si può parlare di aspetto (ha
un aspetto familiare), fattezze o fisionomia (tutti e tre termini più generali, che possono rimandare
a caratteristiche non soltanto facciali [...] oppure, più specificamente per le caratteristiche del
volto, di lineamenti [...]. Aspetto, più genericamente e meno direttamente di f., rimanda anche al
modo di atteggiarsi, di apparire: oggi non hai un bell’aspetto, forse ti senti poco bene? In questo
stesso sign., sono usati anche apparenza, aria o espressione[...]Atteggiamento è raramente limitato
all’aspetto esteriore, indicando per lo più un complessivo modo di essere interiore che traspare
anche all’esterno:[...] Più specifico per le condizioni di salute è cera: avere una bella o una brutta
c. indica il colorito e l’espressione facciale di chi si sente bene o male.
The transcendence of the face: A semiotic-linguistic path
289
Then there is a second, picturesque and metaphorical layer, with strong
animal connotations; and, finally, a third more abstract layer, such as ‘aspetto’,
‘fisionomia’, ‘fattezze’, which is used in theoretical or official discourses rather than
in conversation. This multiplicity is not an exclusive feature of the Italian language,
surely it applies elsewhere as well, but of course with different details; however, this
theme should certainly be deepened with targeted lexicological research.
Metaphors of the face
One feature is striking in this review of the Italian terms and also of the main words
in other languages listed above: in none of them is the face presented in relation to
the subjectivity, the self, the personality, the true character of the individual, the
real person.12 On the contrary, the face appears like a surface opposite to the world,
which hides the intentions, thoughts and emotions of the subject. Erving Goffman
(1955, 1967) has introduced an important sociological reflection on this superficial
and external character of the face. He proposed to focus on an important relational
aspect of social life, namely the interaction between subjects defined for their role,
which he characterized using the metaphor of ‘face’, as common language often
does. According to Goffman’s “dramaturgical” metaphor, the face is a mask which
is worn according to the audience and the variety of social interaction. This is the
most important element of the “scene” in the “play” that regulates the way people
appear to one another in social life. The face indicates a certain role, a position, a
carefully constructed social identity. People strive to keep the face they have created
in social situations and they see the prospect of “losing it” as a grave danger. They
are emotionally attached to their faces, so they feel good while these are kept; face
“loss” results in emotional pain, so in social interactions people cooperate using
courtesy strategies to keep each other’s face.
The micro-sociological theory, however, borrowed these face-based metaphors
from a widespread linguistic practice. Since Dante (Purgatorio, XXIII, 101), we
will find that the word ‘sfacciato’ (‘brazen’, composed of the privative ‘s’ and ‘face’:
‘without a face’) in Italian indicates a “person who has no modesty, restraint,
shame” (Cortellazzo, Zoli 1988: 1190) and has a very negative connotation.
Someone “faceless” is a disreputable individual. The notion of ‘losing face’ is
present in Japanese (面目 を 失 う menboku wo ushinau); in Persian (Aab ro
rizi ΰϳέϭήѧѧѧѧѧѧѧΑ̵ ‘losing the face’s water’); in various South Slavic languages, as in
Serbian, Croatian and Bulgarian (безобразен, bezobrazan); in Chinese (diu mianzi
12
For a semiotic and philosophical-linguistic investigation of this last linguistic field, see
Volli in print.
290
Ugo Volli
丢面子).13 The notion is especially important and widely studied in Chinese
culture (Zuo 1997; Huang 1987); there is, for example, a widespread proverb “Men
cannot live without a face just as trees cannot live without bark” (ren hou lian, shu
hou pi – 人 后 脸 , 树 后 皮). In Arabic, the correlative expression ‘hafiẓa
māʼ al-wajh’ (ϪѧѧΟϮϟ˯ΎѧѧϣφѧѧϔΣ), which literally translates as ‘save the face’s water’, is
used to mean ‘to save face’.
The vast diffusion of this metaphor is undoubtedly striking. George Lakoff and
Mark Johnson (1980) suggest that the explanation is to be found in the metonymic
relation that replaces a part (the face) for the whole (the whole human being). It is
true that if you ask someone to exhibit a photograph and he shows you a picture
of his face, you are satisfied, yet this does not happen if he shows you the image
of his back or of the whole body without the face. Yet beyond the purely material
identification, ‘face’ indicates a social position, a widespread esteem, a role –
abstract qualities that can be lost and recovered, not just the physical individuality
which is much more rooted and more or less indelible. Another clue in this sense is
the use of the word ‘interface’ that used to denote the surface of separation between
two liquids, but in the 1980s came to mean the contact between two computers
and then the instrument of human–machine communication, with the implication
that what appears on the interface does not represent the computer processes, but
forms a representation or translation adapted to the user’s needs.14 The interface,
as the face, is something between the poles of communication that both expresses
and hides.
In other words, not only in microsociology but also in everyday life, the face
is mostly perceived as a mask. This is a widespread term worth dwelling on for
a moment. The root used in most Indo-European languages (Italian ‘maschera’,
French ‘masque’, English ‘mask’, Spanish ‘mascara’, German ‘Maske’, Russian
‘маска’, etc.) has an unclear etymology. As Mariella Canzani writes in a note
published on 15 March, 2020 for the linguistic consultancy editorial staff of the
“Accademia della Crusca”, the most important institution dedicated to the Italian
language:15
13
A repertoire of other similar linguistic traces can be obtained from https://en.wikipedia.
org/wiki/Face_(sociological_concept); last accessed on 21 February 2021.
14 “The place at which independent and often unrelated systems meet and act on or communicate with each other; the means by which interaction or communication is achieved; a
surface forming a common boundary of two bodies, spaces, or phases” (https://www.merriamwebster.com/dictionary/interface).
15 The source: https://www.facebook.com/notes/353039995777856/ was last accessed on 22
February 2021.
The transcendence of the face: A semiotic-linguistic path
291
The main hypotheses are two, not in contrast with each other: one suggests
the derivation from a pre-Indo-European form masca ‘soot, black ghost’ (FEW
reconstructs a base *mask ‘dark’.), the other suggests the derivation from masca
‘witch’, a regional expression of the Ligurian and Piedmontese area. Masca in
turn derives from the late Latin masca(m), a noun used in the same meaning,
a term documented by written sources starting from the mid-seventh century,
first attested in the Edict of Rotari (Lex longobardorum I, 11: strigam, quodest
mascam), which in turn could be of pre-Roman origin. From the original meaning
of ‘witch’ we will arrive later to that of a ghost, a spirit, a disguised aspect to instill
fear. The linguistic evolution of the signifier instead led to the addition of an ‘r’,
making the term assume the form first of mascra and then of mascara, an ancient
variant attested in the 13th century. According to some scholars, the late Latin
masca(m) should be a relict of the pre-Gallic substratum, alternating with *basca,
cf. Fr. rebâcher, from Old French.
* rabaschier ‘make noise’, used about ghosts, from the verb *bascare that corresponds to the Greek báskein, a term that alludes to magical fascination. Finally,
other scholars have suggested a derivation of the etymology from the Arabic phrase
maschara or mascharat, ‘fool, hoax’, deriving from the verb sachira, ‘to mock’,
imported into medieval language from the Crusades. However, the presence of
this word in texts prior to the Crusades demonstrates the inaccuracy of the Arabic
derivation of this family of words.
And yet, as Canzani notes again, “the term of classical Latin to indicate the theatrical
mask used by the histriones (actors) was persona. It is a loan, through the Etruscan
phersu, from the Greek prósōpon”, a word which we have already encountered in the
list of the main terms corresponding to ‘face’ in various languages, because its first
meaning is precisely “‘forehead, face, face’, and then ‘mask’ and ‘character’”. There is,
thus, an important etymological relation between ‘mask’, ‘person’ and ‘face’, which
was also exploited by the great Italian playwright Luigi Pirandello when he entitled
a collection of his theatrical texts Naked Masks, proposing a double oxymoron –
double, because the mask is an inanimate object and nudity applies only to human
beings; and because a face covered by a mask obviously cannot be naked.
Conclusion: Some philosophical consequences
What interests us most in this exploration of the metaphorical use of the face as a
mask is the fact that it is made possible by a fundamental and often insufficiently
emphasized semantic trait that characterizes theses notions, their exteriority, which
is a fundamental point in the philosophical analysis of the face. This theme of the
exteriority of the face was developed by Levinas in terms of the ‘transcendence’
of the other:
292
Ugo Volli
The face escapes from possession, from my power. In its epiphany, in expression,
the sensitive, which is still graspable, changes into total resistance to the grip.
This change is only possible thanks to the opening of a new dimension. In fact,
resistance to grip is not produced as an insurmountable resistance, as the hardness
of the rock against which the effort of the hand is useless, as the distance of a star
in the immensity of space. The expression that the face introduces into the world
does not challenge the weakness of my power, but my power of power. The face,
still a thing among things, opens a gap in the form that delimits it. Which means
concretely: the face speaks to me and thus invites me to a relationship that has
no common measure with a power that is exercised, even if it is enjoyment or
knowledge. [...] When I refer to the face, I don’t just mean the colour of the eyes,
the shape of the nose, the redness of the lips. Stopping here I still contemplate only
some data; but even a chair is made of data. The true nature of the face, its secret
lies elsewhere: in the question he asks me, a question that is both a request for help
and a threat. (Levinas 1961: 85; my translation, U. V.)
Levinas recognizes that the condition of the “transcendence” of the face of others
is “its secret” which always “lives elsewhere”: elsewhere than the world of things
of which it is also a part. This is its “exteriority”, that is, the fact that we do not
have direct access to the other through the face, that the other always exceeds it.
If anything, the face should be assimilated to a “trace” [“the presence of what,
properly speaking, has never been, of what has always already passed” (Levinas
1972: 89)] and is the condition of all signification:
It (the expression) presents the signifier. The signifier, the one who gives a sign,
is not signified. It is necessary to have already been in an area of signifiers, for the
sign to appear as a sign. The signifier must therefore present itself before any sign,
in itself – as a face. (Levinas 1961: 186; my translation, U. V.)
This condition of excess and exteriority is therefore very similar to that of the
linguistic signifier, which is never transparency of the sign; in language the signifier
is always opaque and this condition grounds the principle of arbitrariness. Meaning
exceeds its signifier, it is not determined by it, just as the human being exceeds his/
her face, he/she can think and feel and be more than what he/she lets to be seen.
This condition of transcendence, which filters the sense of the face and is eminently
expressed in the linguistic capacity of humans, is what underlies their freedom –
the human exception.
With the term ‘exception’ I refer to the logical sense of this word, that is as
“a case to which a rule does not apply”16, without any reference to the ‘state of
exception’ mentioned in Agamben 2005, because this is a juridical measure taken
16
The definition derives from https://www.merriam-webster.com/dictionary/exception.
The transcendence of the face: A semiotic-linguistic path
293
by some authority, while I am referring to the attribution to human beings of a
permanent status that frees them from the causal system that we attribute to all
things of the world, including living beings. If all human behaviour were simply
a consequence of causes, as is the case with avalanches, earthquakes or hurricanes,
we could not attribute to it evaluations or responsibilities, we could not speak of
faults and merits. On the other hand, when we do so and we hold human beings
responsible for the consequences of their behaviour (which is not so with mountains
and seas, nor even with snakes or microbes), we implicitly, rightly or wrongly,
believe them to be capable of a freedom of choice that other things in the world do
not have. In short, we presuppose a gap between causes and behaviour, just as there
is a gap between the signifier and meaning and between the face and the subject.
This is the “human exception”, which of course requires explanations from
philosophy (or from theology, or from science). In Volli 2010 and Volli 2013 I
supported the thesis that this “transcendence” with respect to the causal system,
that is attributed to human beings, is derived from the linguistic competence that
characterizes them (another significant “human exception”), because language is
characterized by arbitrariness, that is, by the lack of causal connection between
expression and meaning (in Saussure’s terms: the signifier and the signified). There
is a degree of freedom in the process of signification which makes the meaning
always partially indeterminate. The absence of meaning from the expression is
decisive for this arbitrariness and it also recalls the transcendence or absence of
the subject in the face.
This relationship, not at all trivial, but evident between language and the face
is also underlined by Giorgio Agamben in a short essay written in 1996,17 now
included in a collection of political essays:
What the face exposes and reveals is not something that could be formulated
as a signifying proposition of sorts, nor is it a secret doomed to remain forever
incommunicable. The face’s revelation is revelation of language itself. Such a
revelation, therefore, does not have any real content and does not tell the truth
about this or that state of being, about this or that aspect of human beings and
of the world: it is only opening, only communicability. To walk in the light of the
face means to be this opening– and to suffer it, and to endure it. Thus, the face is,
above all, the passion of revelation, the passion of language. Nature acquires a face
precisely in the moment it feels that it is being revealed by language. (Agamben
2006[1996]: 93)
17
I thank the anonymous reviewer who prompted me to develop my analysis further at this
point.
294
Ugo Volli
Agamben’s reasoning is somewhat different from the line of analysis I have
proposed till now and also differs from that of Levinas. For Agamben (1996: 91),
“the face is at once the irreparable being-exposed of humans and the very opening
in which they hide and stay hidden”. This opening is also common to animals,
but “only human beings want to take possession of this opening, to seize hold of
their own appearance and of their own being being-manifest […]. Language is
this appropriation, which transforms nature into face”. The emphasis here is not
about the “hiding in the opening”, namely absence in the appearance, which I have
already discussed above, but about the “appropriation” and “possession” of this
opening, which can be understood as the strategic games of the appearance studied
by Goffman under the name of ‘face’. This leads Agamben (1996: 91) to think that
“the face is the only location of community, the only possible city”, in particular
today it is a “city”, a polis that Agamben criticizes in political terms that we will not
discuss here. In Agamben’s terminology, the individual and expressive level that I
have analysed so far is the ‘visage’ with which “the face does not coincide”:
The only face to remain uninjured is the one capable of taking the abyss of its own
communicability upon itself and of exposing it without fear or complacency. This
is why the face contracts into an expression, stiffens into a character, and thus sinks
further and further into itself. As soon as the face realizes that communicability
is all that it is and hence that it has nothing to express – thus withdrawing silently
behind itself, inside its own mute identity – it turns into a grimace, which is what
one calls character. Character is the constitutive reticence that human beings retain
in the word; but what one has to take possession of here is only a non latency, a
pure visibility: simply a visage. The face is not something that transcends the visage:
it is the exposition of the visage in all its nudity, it is a victory over character –
it is word. (Agamben 1996: 95)
In the face there would therefore be a tension between support and function,
“the duality of proper and improper, of communication and communicability, of
potentiality and act. The face is formed by a passive background on which the
active expressive traits emerge”. In this regard, Agamben cites a long passage from
Rosenzweig’s Der Stern der Erlösung, where the German philosopher sees in the
face a symbolic instance of the “Star of Redemption”, consisting of two triangles:
a “basic” one which takes shape in the alignment of the sense organs (eyes and
mouth), which is “pure receptivity”, and another, “superimposed” one, composed of
the other parts “whose activity quickens the rigid mask of the first” (see Rosenzweig
1970[1921]: 422–23). Agamben’s (1996: 97) comment shifts this symbolic towards
a socio-political vision of our time: “In advertising and pornography (consumer
society), the eyes and the mouth come to the foreground; in totalitarian states
The transcendence of the face: A semiotic-linguistic path
295
(bureaucracy), the passive background is dominant (the inexpressive images of
tyrants in their offices).”
Agamben’s analysis is best understood by thinking of the great tradition of
philosophical anthropology that starts with Aristotle (Nicomachean Ethics I.13,
1102a5–1103a10), for which, apart from the contemplative life of the philosopher,
the fullness of the human is reached in the “vita activa” of public action, the living
presence of the person in the public sphere (Arendt 2013[1958]), and this action
in turn is made possible mainly by language. The face thus becomes the place of
this opening to the polis and it undergoes its vicissitudes, in particular as regards
the general object of Agamben’s critique – modern Western society.
This is therefore a philosophy of the face that works exclusively on its general
social dimension: not on the face of the individual human being and on his
relationship with his identity and with his neighbours, but on the regime of visibility
that faces realize and endure in relation to circumstances of collective life. In
short, the face for Agamben is a certain way of acting on others, a communication
tool that, like all media, can be emptied to the point of only expressing its own
communicability. However, there is at least one other point of view which considers
the face as a fundamental place of personal identification, originating from Lacan’s
mirror phase (Lacan 1949).
This approach and the aporias it opens up are the central theme of the path
proposed in this paper. The face appeared to us as a certain placeholder of absence,
a certain support of the not being all there of the person who is present; its mask
dimension allows it to play differently from that of simply present objects. It is
never a neutral object, while it is always the bearer of values and the object of
evaluations; in a certain sense it is the original model of the communication
of values. Therefore, it can be an actual sign of the person, bringing to others
some of his qualities, certain of his values, his current thymic position. With two
caveats: that the sign can never be identical to its object (as Plato already claimed
in Cratylus, 432a), but must always highlight some of its sides, a face; and that the
person (or prosopon – namely his/her face) can never be identical to the human
being it represents socially, but highlights some aspects, some faces of this human
being, or even constructs them.
References
Agamben, Giorgio 2000[1996]. Means without End: Notes on Politics. (Casarino, Cesare;
Binetti, Vincenzo, trans.) Minneapolis: University of Minnesota Press.
Agamben, Giorgio 2003. Stato di eccezione. Torino: Bollati Boringhieri.
Beaney, Michael 2013. What is Analytic Philosophy? Oxford: Oxford University Press.
https://doi.org/10.1093/oxfordhb/9780199238842.013.0039
296
Ugo Volli
Arendt, Hannah 2013[1958]. The Human Condition. Chicago: University of Chicago Press.
Bird, Alexander; Tobin, Emma 2008. Natural kinds. In: Zalta, Edward N. (ed.), The
Stanford Encyclopedia of Philosophy. Stantford: The Scholarly Publishing and Academic
Resources Coalition.
Campanini, Giuseppe; Carboni, Giuseppe 1993. Vocabolario Latino Italiano / Italiano
Latino Torino: Paravia.
Colonna, Barbara 1997. Dizionario etimologico della lingua italiana. Roma: Newton.
Cortellazzo, Manlio; Zolli, Paolo 1988. Dizionario etimologico della lingua italiana. Bologna:
Zanichelli.
Daly, Chris 1998. Natural kinds. In: Craig, Edward (ed.), Routledge Encyclopedia of
Philosophy. Vol. 6. London: Routledge, 682–685.
Darwin, Charles 2009[1872]. The Expression of the Emotions in Man and in the Animals.
London: Penguin. https://doi.org/10.1017/CBO9780511694110
Eco, Umberto 1983. Semiotics and the Philosophy of Language. Bloomington: Indiana
University Press.
Huang, Shuanfan. 1987. Two studies of prototype semantics: Xiao ‘Filial Piety’ and Mei
Mianzi ‘Loss of Face’. Journal of Chinese Linguistics 15: 55–89.
Ferraris, Maurizio 2009. Documentalità. Perché è necessario lasciar tracce. Roma-Bari:
Laterza.
Gabrielli, Aldo 1967. Dizionario dei sinonimi e dei contrari. Milano: Istituto Editoriale
Italiano.
Goffman, Erving 1955. On face-work. Psichiatria 18(3): 213–231. https://doi.org/10.1080
/00332747.1955.11023008
Goffman, Erving 1967. Interaction Ritual: Essays in Face-To-Face Behavior. Chicago: Aldine.
Lacan, Jacques 1949. Le Stade du miroir comme formateur de la fonction du Je: telle qu’elle
nous est révélée dans l’expérience psychanalytique. Revue française de psychanalyse 13:
449–455.
Lakoff, George; Johnson, Mark 1980. Metaphors We Live By. Chicago: The University of
Chicago Press.
Levinas, Emmanuel 1961. Totalité et infini: Essai sur l’extériorité. La Haye: Martinus Nijhoff.
Levinas, Emmanuel 1972. Humanisme de l’autre homme. Montpellier: Fata Morgana.
Lotman, Juri 2011[1967]. The place of art among other modelling systems. [Tezisy k
probleme “Isskusstvo v ryadu modeliruyuschih sistem”.] (Pern, Tanel, trans.) Sign
Systems Studies 39(2/4): 249–270. https://doi.org/10.12697/SSS.2011.39.2-4.10
O’Riley, Tim 1998. Representing Illusions: Space, Narrative and the Spectator in Fine Art
Practice. London: Chelsea College of Art and Design.
Prum, Richard 2017. The Evolution of Beauty. New York: Doubleday.
Quine, Willard 1960. Word and Object. Cambridge: The MIT Press.
Rosenzweig, Franz 1970[1921]. The Star of Redemption. (Hallo, William W., trans.) New
York: Holt, Rinehart and Winston.
Searle, John 1995. The Construction of Social Reality. New York: Free Press.
Volli, Ugo 2010. Al di là dell’essenza, il linguaggio. Quaderni laici 1: 113–119.
Volli, Ugo 2013. Riflessione e trascendenza di una maschera. In: Leone, Massimo; Pezzini,
Isabella (eds.), Semiotica della soggettività – Per Omar (I Saggi di Lexia II). Roma:
Aracne, 265–278.
The transcendence of the face: A semiotic-linguistic path
297
Volli, Ugo 2020. Invisibile, espressivo e necessario Metafore del volto divino nella Bibbia
ebraica. Lexia (Volti Artificiali/Artificial Faces; Leone, Massimo, ed.) 37–38: 557–583.
Volli, Ugo 2021. Di che cosa parliamo quando diciamo ‘persona’? – Fra filosofia e semiotica.
In: Ponzo, Jenny; Vissio, Gabriele (eds.), Culture della persona: Itinerari di ricerca tra
semiotica, filosofia e scienze umane. Torino: Accademia University Press.
Zuo, Bin 1997. Chinese “lian” and “mianzi”. Wuhan: Huazhong Normal University Press.
Трансцендентность лица: лингво-семиотический путь
В начале статьи мы анализируем термины, используемые для обозначения лица
в разных языках, в частности, в итальянском, сравнивая их с определениями,
предложенных некоторыми авторитетными словарями, а также с их этимологией.
Это исследование приводит к примечательным результатам: во-первых, оказывается,
что термин «лицо» имеет материальное значение, но в то же время является
социальным объектом; во-вторых, подчеркивается важность коммуникативной
функции, которая делает лицо похожим на маску, что, в некотором роде, напоминает
арбитрарность языка. Все это говорит о том, что философский статус лица – это
«трансцендентность», которая является условием того состояния свободы, которое
мы приписываем себе и которое можно определить как «человеческое исключение».
Näo transtsendentsus: semiootilis-lingvistiline tee
Artikkel algab nägu tähistavate terminite vaatlemisega erinevates keeltes, eriti itaalia keeles,
kõrvutades neid definitsioonidega, mida pakuvad mõned autoriteetsed sõnaraamatud ja
ka nende etümoloogiaga. Need vaatlused annavad tähelepanuväärseid tulemusi: esiteks
ilmneb, et nägu on tõepoolest termin, millel on materiaalne tähendus, ent samal ajal on
tegu sotsiaalse objektiga; teiseks tuleb esile kommmunikatiivse funktsiooni olulisus, mis
muudab näo sarnaseks maskiga ning mõnel moel ka keele arbitraarsusega. Kõik see viitab
asjaolule, et näo filosoofiline staatus on “transtsendentsus”, mis on selle vabadusseisundi
tingimus, mida me omistame enestele ning mida võib defineerida kui “inimlikku erandit”.
298
Simona Stano
Sign Systems Studies 49(3/4), 2021, 298–317
The face of health in the West and the East:
A semio-cultural analysis
Simona Stano1
Abstract. Magazines, leaflets, weblogs, and a variety of other media incessantly
spread messages advising us on how to achieve or maintain our health or well-being.
In such messages, the iconic representation of the face is predominant, and reveals
an interesting phenomenon: the “face of health” seems to be unattainable as such,
and is generally represented in a differential way, that is to say, by making reference
to its opposite – the “face of illness”, or at least of malaise. In fact, the face is crucial
in the medical domain: since ancient times, face observation has played an essential
role in diagnostic practices, both in Western medicine (which resorts to the concept
of facies, intended as the distinctive facial expression or appearance associated with
a specific medical condition, for the description of pathological states) and Eastern
preventive and healing techniques (within which the so-called Mian Xiang, or
‘face reading’, is fundamental, and connects the medical sphere with other aspects
such as personality, talents, and dispositions). Drawing on the semiotic analysis of
relevant case studies extending from classical iconography to present-day digital
mediascapes, this paper investigates the representation of the face of health (and
illness) across time and space, specifically focusing on the analogies and differences
between the Western and the Eastern semiosphere. To this purpose, it relies on both
literature concerning the representation and understanding of the face and studies
on medical sign systems and discourses.
Keywords: face; health; collective imaginary; medicine; West vs. East; semiotics.
1
Department of Philosophy and Education Science, University of Turin (UNITO), Torino,
Italy; e-mail: simona.stano@unito.it.
https://doi.org/10.12697/SSS.2021.49.3-4.03
The face of health in the West and the East: A semio-cultural analysis
299
Proprio nomine dicitur facere a facie, qui rei quam facit imponit faciem.
Ut fictor cum dicit fingo, figuram imponit, quom dicit formo, formam,
sic cum dicit facio, faciem imponit.
Varro, De lingua latina VI, 78
1. Introduction
“What does your face say about your health?”; “Have you ever wondered if there is
a connection between locations on your face and body organs?”; “The secrets your
face tells about your health”. These are only a few of the several references to the
close relation between the face and health that can be found in posts and articles
featured on the Internet. Magazines, leaflets and other media advising us on how
to achieve or maintain our health or well-being also insist particularly on the face
and its visual representation. What is more, since ancient times, face observation
has played an essential role in diagnostic practices, both in Western medicine –
which resorts to the concept of facies, intended as the distinctive facial expression
or appearance associated with a specific medical condition, for the description
of pathological states – and Eastern preventive and healing techniques, within
which the so-called Mian Xiang, or ‘face reading’, is fundamental, and connects
the medical sphere with other aspects such as personality, talents, and dispositions.
Why has the face been elected as the plane of discursivization of health (or
illness) par excellence? How does cultural variation influence such a relation and
the facial representation of health? What do such representations tell us about the
understanding of health itself? Drawing on the semiotic analysis of relevant case
studies extending from classical iconography to present-day digital mediascapes,
the following paragraphs aim at addressing these issues, investigating the representation of the face of health across time and space, and specifically focusing on the
analogies and differences between the Western and the Eastern semiosphere.
2. Health: In search of a definition
Before analysing the link between the face and health, it is worthwhile clarifying,
at least briefly, what is meant by these terms.
The face is a complex object of analysis, as suggested by the multiplicity and
variety of terms used to refer to it (especially in Romance languages) and their
etymology:
300
Simona Stano
The complexity of the face as an observable, decipherable, and describable object
has always posed a quantity of questions that, at different times, have found
different answers: sometimes explaining the entire macrocosm through the face,
sometimes making the face into a whole macrocosm; sometimes considering
it simply as a transparent mirror of passions, sometimes rather conceiving it as
a treacherous mask produced by the game of social simulation. Traces of the
thousand contradictions characterizing the face in the collective imaginary are
inscribed, albeit in a disconnected and fragmentary way, in the etymological
history of the natural language terms that have been used to designate it: [in Italian]
volto, viso, faccia; in Latin vultus, facies; in French chère, face, figure, vis, visage, vout
[...]. 2 (Magli 1995: 10)
A particular aspect, more than others, is relevant to the purposes of this paper:
The face is above all, in a passive sense, visus, [that is to say] the part of us, of our
person and not only of our body, that is more exposed to visibility. It is the part of
us that remains most entangled in the game of looks, of seeing and being seen, of
showing off and hiding.3 (Magli 1995: 10)
As such, the face represents first of all a form, which is subjected to countless
strategies of signification and reading (Leone 2018: 27). It is a discourse, that is to
say, an utterance, an act of enunciation.
Likewise, health is a multifaceted and not at all univocally defined concept.
Traditionally, it has been described mainly in differential terms, that is to say,
as the absence of what is generally called ‘disease’: as Paolo Fabbri (1995: 37)
remarked, “the Latin word salus is related to salvus, which means the one who
is saved from changes, from disturbances coming from outside. It is a question
of total integrity, a kind of ‘skin’ that saves and puts at ease”.4 In fact, most Latin2
“La complessità del volto come oggetto osservabile, decifrabile, descrivibile, da sempre ha
posto una quantità di quesiti a cui, in tempi diversi, si è tentato di dare risposta: ora spiegando
l’intero macrocosmo attraverso il volto, ora facendo del volto l’intero macrocosmo, ora
considerandolo come specchio trasparente delle passioni, ora come maschera infida prodotta dal
gioco della simulazione sociale. Tracce delle mille contraddizioni di cui vive il volto all’interno
dell’immaginario collettivo sono iscritte, seppure in modo sconnesso e spesso frammentario, nella
storia etimologica dei termini che, all’interno della lingua naturale, lo designano: volto, viso,
faccia; in latino vultus, facies; in francese chère, face, figure, vis, visage, vout […].” (Translations
from Italian are mine, unless indicated otherwise, S. S.)
3
“Ma il volto è soprattutto, in senso passivo, visus, quanto di noi, della nostra persona e non
solo del nostro corpo, è più esposto alla visibilità. È quanto di noi più resta impigliato nel gioco
degli sguardi, del vedere e dell’essere visti, del far mostra di sé e del nascondersi.”
4
“Il latino salus è legato a salvus che significa colui che si salva dalle modifiche, dalle perturbazioni provenienti dall’esterno. Si tratta di un’integrità totale, una specie di pelle che salva, che
mette tranquilli, elimina le sollecitazioni della curiosità.”
The face of health in the West and the East: A semio-cultural analysis
301
based terms used to designate health (such as, for instance, the Italian ‘salute’)
highlight such a character of “integrity, safety and therefore salvation” (Pianigiani
2020).5 The Germanic origin of the old English term ‘hǣlth’, from which the word
‘health’ derived, also relates to ‘whole’ (Oxford English Dictionary 2020),6 further
enhancing such a perspective.
In this sense, as noted by Jacques Fontanille (2005), health is conceived of as
an evaluative statement antithetical to disease: the former describes a state that
fully conforms to the norm, the latter rather establishes a divergence from it. This
asymmetry is then reflected on a pragmatic level, since there seems to be only one
way of being healthy, as opposed to a multiplicity of ways of being ill; and also on
the syntagmatic level, since health does not seem to have a history (it is a temporal
segment), while illness is above all a “narrative sequence” (i.e. an aspectualized7
path, including an initial phase – i.e. the onset of the disorder, such as a lesion,
or the moment of incubation, etc.; a durative phase – i.e. the development of
the disease, consisting for instance in the infection; and a terminative phase –
i.e. healing, or rather deterioration and death). From a narrative point of view,
health corresponds in this view to “a dense and unanalysable ‘state’, while illness
is perceived as a multiplicity of paths implying both losses and gains, emerging
problems, crises and healings”8 (Fontanille 2005: 35) – see Table 1.
5
“Integrità, Incolumità e quindi Salvezza.” The source, Pianigiani, Ottorino 2020. Dizionario
etimologico online, available at https://www.etimo.it, was last accessed on 19 February 2021.
6
Oxford English Dictionary, available at https://www.lexico.com, was last accessed on 19
February 2021.
7
As remarked by Greimas and Courtés (1982[1979]: 18), “all temporalized discourse has
two kinds of new investments which produce the two meaning effects of temporality and
aspectuality. While the effect of temporality is linked to the establishment of a group of temporal categories which […] project a temporal organizations of topological nature onto the
utterance, the effect of aspectuality results from the investments of aspectual categories which
convert the functions (or predicated) of narrative utterances into a process”. In this sense,
health seems to be closer to the traditional narrative logic of programming, whereas illness
features a processual development, which combines punctual and durative phases, perfective
or imperfective aspects, iterative or singular sequences.
8
“[…] uno “stato” denso e inanalizzabile, mentre la malattia viene percepita come una molteplicità di percorsi che implicano perdite e guadagni, problemi emergenti, crisi e guarigioni.”
302
Simona Stano
Table 1. The opposition between health and illness.
Health
Negative concept
(absence of events)
Static balance
Illness
Positive concept
(presence of events)
Dynamic imbalance
Conformity
Divergence
Temporal segment
Unique, dense, and
unanalysable ‘state’
Aspectualized path
Modular and analysable
multiplicity of ‘paths’
However, as the French scholar effectively remarked, such an approach is too
simplistic, and fails to explain a recurring aspect of the disease, which is also a
mode of its possible indirect presence within a state of health: the malaise. In fact,
according to Fontanille, the malaise can appear in all phases of illness and even in
its absence. Even if an external look (such as that of a doctor) can later recognize
it as a symptom of a specific illness, the malaise as a feeling or sensation is not
necessarily related to the sphere of disease. Such a perspective – which recalls the
Barthesian conception of the symptom (Barthes 1972), according to which it is
not a sign in itself, but becomes such only when it enters the context of clinical
discourse9 – is of particular relevance, since it highlights that
[h]ealth is not a restful adventure, and it even seems that it can be better
described – going against common sense – as a permanent tension, that is to say, a
multi-polemic process. The disease, on the other hand, is almost always the result
of a release of tensions (the defenses surrender, the body no longer reacts to attacks,
the cells no longer die and begin to proliferate, etc.). [...] Of course, thanks to the
constantly changing balances that it manages to guarantee, health manifests itself
9
In his essay “Sémiologie et médecine”, Roland Barthes (1972) describes symptoms as
brute facts emerging from the body – that is to say, purely phenomenal sensations that are
not necessarily linked to any act of interpretation, and so do not constitute semiotic entities
as such. Nonetheless, he also supports the view that they turn into signs once they are
“put into discourse”, that is, when they are “modelled” (in terms of Lotman 1977) through
language. Such an opposition is partially overcome by Thomas Sebeok, who pushes further the
association between signs and symptoms, conceiving the latter as “compulsive, automatic, nonarbitrary sign[s], such that the signifier coupled with the signified in the manner of a natural
link” (Sebeok 1976: 46; cf. Sebeok 2001). In Sebeok’s view, symptoms can be seen as a marked
category (species) of an unmarked one (genus, that is, the sign), thus acquiring a semiotic status
per se. For further details, see, in particular, Stano 2020.
The face of health in the West and the East: A semio-cultural analysis
303
and makes itself perceived [...] as an absence of events, a stable and durable state.10
(Fontanille 2005: 37)
It is therefore necessary to distinguish between two different dynamics. On the
level of existence, the difference between health and disease is reduced to a series of
narrative transformations and variations of the figurative balances that put health
and illness in open opposition, identifying them as antithetical terms. On the
level of experience, this distinction is counteracted by the appearance of a specific
“feeling” (i.e. the malaise), which brings out the aspectual and transformative
nature of health alongside that of illness, putting them in a tensive correlation
rather than in a firm opposition.
While requiring to reconsider the definitions and characterizations mentioned
above, this view also distinguishes two levels of corporeality that interact in the
perception and recognition of health and illness as related processes: the ‘Me-flesh’
(corps-chair), which participates in the transformation of the states of things and is
a “composite totality in existence”11 (Fontanille 2005: 40); and the ‘Myself-envelope’
(corps propre), understood as a “coherent unity in experience”12 (Fontanille 2005:
40), that is, the “body-in-construction, the living vehicle of intentionality and
interactions support, and therefore, particularly the support of confrontation with
otherness” (Fontanille 2013: 41).
Moreover, in recent decades another concept has increasingly more been used
in relation to health, highlighting its “higher threshold” alongside the “lower
threshold” that associates it with disease (Marrone 2005; Sbisà 2005): the idea
of well-being, whether real or apparent. This finds a confirmation in the current
definition of health in different languages,13 as well as in universally shared
descriptions such as the one provided by the World Health Organization, which
clearly mentions the overcoming of the differential view: “Health is a state of
complete physical, mental and social well-being and not merely the absence
10
“La salute non è un’avventura di tutto riposo, e sembra addirittura che la si possa descrivere con
precisione – andando in ciò contro al senso comune – soltanto come una tensione permanente, un
processo multipolemico. La malattia invece è quasi sempre esito di un rilascio delle tensioni (la difesa
cede, il corpo non reagisce più agli attacchi, le cellule non muoiono più e si mettono a proliferare
ecc.). […] Naturalmente, grazie agli equilibri in continuo movimento che riesce a garantire, la salute
si manifesta e si fa percepire […] come un’assenza di evento, uno stato stabile e durevole.”
11 “Totalità composita nell’esistenza.”
12 “Unità coerente nell’esperienza.”
13 The Merriam-Webster Online English Dictionary (2020), for instance, defines health as
“the condition of being sound in body, mind, or spirit”, also openly associating it with the idea
of well-being: “a condition in which someone or something is thriving or doing well”. The
dictionary was last accessed at https://www.merriam-webster.com on 19 February 2021.
304
Simona Stano
of disease or infirmity” (WHO 2020: 1).14 Not only is a positive definition
adopted in this case, but the concept of health is enlarged, encompassing both
the psychological and the social dimension. This further emphasizes a phenomenological conception of the body as “a totality of lived significations” (MerleauPonty 2012[1945]: 155), that is to say, both an object in the world and a subject
experiencing the world, a part of reality and a means of conjunction with it – a real
“threshold of sense” (Stano 2019: 149).
In the following paragraphs we will analyse how such conceptions of health
and disease have found expression in the visual representations of the face in the
West and the East in an attempt to identify and describe the main effects of the
meaning emerging from them.
3. The face of health in the Western collective imaginary
Among the first representations of health in the Western collective imaginary
special mention should be made of Hygeia (from the Greek ‘Ὑγίεια’, ‘health’), the
Greek divinity of physical and spiritual health, cleanliness and hygiene. Two main
phases in the cult of this figure are known: at first as an independent goddess; then
in connection with the worship of her father Asclepius, the god of medicine who
was invoked for the treatment of disease and the restoration of lost health, while
Hygeia was mostly related to the prevention of illness. Correspondingly, in plastic
arts the goddess was initially depicted alone, as a virgin of majestic stature with a
crowned head; then, towards the third century, she took on the form of a graceful
girl accompanied by Asclepius himself and/or by the animal with which he was
traditionally depicted: the snake.15 In fact, her figure mainly acquired meaning in
relation to such elements, thus stressing the difference between prevention and
treatment and insisting on a broader conception of health. This did not confer
much importance on her face, which mostly responded to the common canon in
classical art, without expressive marks of particular relevance.
14
WHO 2020. Basic documents. (49th ed.), available at https://apps.who.int/gb/bd/, was last
accessed on 11 December 2020.
15 An extensive literature on the symbolism of the snake exists. With reference to the figure of
Asclepius, from which the association with Hygeia seems to have derived, the idea of knowledge
is generally highlighted, in addition to the reference to the legend according to which the god
of medicine, seeing a snake reviving another with herbs, used them to save Glauco (MurilloGodínez 2010). Furthermore, in Greek mythology, Asclepius exercised his healing powers
precisely with the help of a snake, which was able to heal wounds by licking them with its tongue
(for further information, see Gutiérrez, Mellado, Saavedra 2004; Rillo 2008, while Hæger 1988
instead insists on the possibility of a reference to the parasite Dracunculus medinensis).
The face of health in the West and the East: A semio-cultural analysis
305
In Roman times the veneration of health, known in Latin as Salus (or, less
frequently Valetudo) grew further, to the extent that the goddess was depicted on
numerous coins of the Empire, as its protector. The Roman Salus was in fact the
personification of a broader idea of ‘well-being’, understood as the combination
of physical health and prosperity, which concerned both the individual level (as
in case of the Greek divinity Hygeia) and the Res publica (i.e. the social sphere).
Salus was usually represented seated, with her legs crossed (in the same position
generally used for the goddess Securitas, thus further enhancing the link with the
social and political dimension) and her elbow lying on the armrest of a throne,16
or more commonly in an upright position.17 She often held a patera (i.e. a deep
plate used in religious ceremonies) in her right hand to feed a snake which was
generally wrapped around an altar (or the arm of the throne where Salus was
sitting, or the rod that she usually held in her left hand), sometimes directing its
gaze towards the plate, sometimes towards the same point where the goddess was
looking. While the back of coins generally hosted similar images, the front was
used for the representation of the face – of the emperor in power or, in some cases,
of Salus herself. The ornaments adorning the head and the face of the goddess
played a crucial role in these cases, emphasizing the public dimension of health and
the values of wealth and power. The internal elements of the face, on the contrary,
were not particularly emphasized, as evidenced by the choice of representing it in
profile and the recurrence of ideal and neutral traits,18 or at most of symbolical
(rather than realistic) portraits of female representatives of the Empire.19
Most examples in classical iconography do not differ much from such representations, even when they extend to other cultural systems (as it happens
with Sirona in Celtic mythology) and subsequent reinterpretations (such as in
Neoclassical statues and paintings). The face of health, in other terms, mainly
16
See, for instance, the back of the denarii of emperors Nerva (Marcus Cocceius Nerva Caesar
Augustus, 96 AD), Marcus Aurelius Antoninus Augustus (161–180 AD), Lucius Septimius
Severus Augustus (210 AD), and Caracalla (Marcus Aurelius Antoninus Augustus, 212 AD).
17 See, for instance, the back of the denarii of consul Manius Acilius Glabrio (49 BC), and
emperors Caracalla (Marcus Aurelius Antoninus Augustus, 201 AD) and Elagabalus (Caesar
Marcus Aurelius Antoninus Augustus, 218–219 AD), as well as the back of the Antoninianii
of emperors Philip the Arab (Mark Julius Philip Augustus, 244–245 AD), Marcus Piavonius
Victorinus (270–271 AD), Marcus Aurelius Probus (281 AD) and Maximian (Marcus Aurelius
Valerius Maximianus Erculius, 290–291 AD).
18 See, for instance, the front of the denarius of the Consul Manius Acilius Glabrio, 49 BC.
19 Such as Livia Drusilla, the third wife of emperor Augustus of Rome, mother of emperor
Tiberius, and grandmother of emperor Claudius, on the front of a coin of the Emperor Tiberius
Julius Caesar Augustus, 22–23 AD.
306
Simona Stano
remains a neutral and “perfect” surface to be adorned with symbolical elements,
which are the ones conferring meaning on it.
A completely different representation appeared in 1894, when Gustav Klimt
was commissioned, together with Franz Matsch, to create large decorative panels
for the University of Vienna. Requested to represent philosophy, medicine and
jurisprudence, the artist created three large paintings, which caused an uproar
among the university establishment and were never displayed to the public, but
transferred to the Österreichische Galerie. Later purchased by August Lederer and
Koloman Moser, the paintings were relocated to the Immendorf Castle during
World War II, where they perished in 1945 due to a fire started by the retreating
SS. Focusing on the representation of medicine (Fig. 1, left), it is interesting to
notice first of all that, despite the commissioned subject, Hygeia was preferred to
Asclepius, insisting on health as a general concept rather than on the process of
healing. In fact, the painting is marked by an evident dysphoric characterization.
The Greek personification of health stands in front of a vertical tangle of naked
bodies composed of pregnant women, children and elderly people, along with
skeletons – i.e. allegories of the processes of generation, life, decay and death.
Contrary to the iconographic tradition briefly outlined above, in this case Hygeia
does not convey any idea of well-being or social welfare: turning her back on
humanity, she rather leaves it helpless against a sad fate. The whole group,
confined to the narrow space of what can be interpreted as a large uterus – which
is connected to another female figure (“mother nature”?), also turning her back on
it – is wrapped by a dark veil, in a further negative axiologization.
Looking in detail at the figure of Hygeia (Fig. 1, right), other interesting
elements can be identified: in this case the snake is located in front of the patera,
without taking anything from it. Although framed from below to highlight its
majesty, the face of the goddess is marked by a severe expression and particularly
emphasized shadows, which recall the blackish halo characterizing the group
of figures in front of which she appears. The dysphoric effect thus generated is
further enhanced by Hygeia’s gaze, which appears as dejected and absent. What
Klimt depicts here is definitely a lost and unattainable health. The goddess of his
painting is certainly powerful, but also indifferent to humanity. Therefore, the
malaise becomes existential, inevitably leading to illness and death.
The face of health in the West and the East: A semio-cultural analysis
307
Figure 1. Gustav Klimt. Medizin. (1987, oil on canvas, lost work).
This representation is very interesting, and is in opposition not only to past but also
to current discursivizations of health. In fact, in contemporary Western societies
the most varied media ranging from wellness and fitness magazines to brochures
illustrating anti-aging treatments and weblogs devoted to body care, etc., have
contributed to the creation and circulation of countless representations of health,
conferring a special role on the face. Despite the peculiarities of each example, such
representations seem to respond to the same logic: the “face of health” they depict
is always characterized by the (figurative or metaphorical) elimination of a sort of
“patina” or “veil” made up of spots, wrinkles and other “signs” marking the facial
surface. Differential systems consisting in confronting a ‘before’ and an ‘after’, or
a ‘with’ and ‘without’ condition are largely used, thus emphasizing a differential
conception of health, even when the emphasis is put on well-being. This results in a
progressive movement of desubjectivation and “perfection” of the face, sometimes
also leading to its decomposition in sets of lines, points and intersections (recalling
some practices of Eastern medicine, with which we will deal in detail below). The
“face of health”, in other terms, seems to be irrepresentable as such, at least as a
realistic, particularized and “subjectified” face. It is an “unmarked”, and therefore
308
Simona Stano
ideal, surface, which can only be represented in a differential way, that is to say,
by making reference to its opposite – i.e. the “face of illness”, or at least of malaise
(such as in the case of the imperfections brought about by age). The latter, on
the contrary, finds expression in a multiplicity of representations, not only in
media communications, but also in the medical domain, as we will discuss in the
following section.
4. The face of illness in Western medicine
Western medicine has historically attributed a fundamental role to the face. As Jean
Starobinski (2003: 58) recalled:
A solid tradition required to expose human anatomy a capite ad calcem, that is to
say, from head to heel. The method is still followed in some contemporary medical
schools, as a rule of preliminary clinical description: the face will therefore be the
object of primary attention, whatever part of the body is designated by the patient’s
complaints. And with good reason: it is on the face that anxiety, suffering and the
fundamental gesture of presence to oneself and to the world are inscribed. In a first
approximation, it is therefore the face that reveals the state of alertness, numbness,
unconsciousness: thus, a first measurement of the severity of the disease becomes
possible.20
Hippocrates himself, who is considered the father of Western medicine, underlined
the importance of observing the face in a famous passage from his Prognosticon:
“He [the physician] should observe thus in acute diseases: first, the countenance
of the patient, if it be like those of persons in health, and more so, if like itself, for
this is the best of all” (Book II, 110 [1849: 235]; my emphasis. S. S.). This finds
correspondence in the concept of facies, that is to say, the appearance of the face
characteristic of particular pathological conditions (Dorland 2011), which is
still fundamental in the processes of medical diagnosis. Echoing the differential
dynamics outlined above, medical treatises distinguish between a facies composita
(i.e. the unaltered face of healthy people) and a multiplicity of facies marked in
various ways, such as the Hippocratic facies (which was called after Hippocrates
20
“Una solida tradizione prescriveva di esporre l’anatomia umana a capite ad calcem, dalla
testa ai talloni. Il metodo è ancora seguito in alcune scuole mediche contemporanee, come regola
della descrizione clinica preliminare: il volto sarà dunque oggetto di un’attenzione prioritaria,
qualunque sia la parte del corpo designata dai lamenti del malato. E a buon diritto: è sul volto che
s’inscrivono l’ansia, la sofferenza e il gesto fondamentale della presenza a sé e al mondo. In prima
approssimazione è dunque il volto a rivelare lo stato di vigilanza, il torpore, l’incoscienza: diventa
così possibile una prima misurazione della gravità del male.”
The face of health in the West and the East: A semio-cultural analysis
309
himself, since he first described it, and refers to a face characterized by a sharp
nose, sunken eyes and temples, drawn-in ears and distorted lobes, together with
hard, stretched and dry facial skin, and pale or dusky colour), the facies virile
(typical of hirsutism and characterized by the presence of hair and beard on
female faces), the acromegalic facies (marked by low forehead, eyebrow arches
and prominent cheekbones, large nostrils, thick lips, pronounced chin, as well as
hypertrophy of the tongue and ears), and several others. The underlying logic, in
other words, always seems to be the one we identified for media representations:
the marked face of disease opposes the unmarked face of health, which seems in fact
to be irrepresentable as such, as evidenced by the absence of iconic exemplifications
(at least devoid of differential logics such as ‘before’ vs. ‘after’ comparisons) of the
facies composita.
Compared to other approaches, however, Western medicine has highlighted
the limits of face reading, at least if not accompanied by other types of analysis. It
insists on the fact that the anatomical structures that make up the face are varied
and can be altered in different ways, emphasizing the role played by factors such as
age, sex, race, individual constitution, degree of consciousness, character, and also
cultural influences: “[…] the temptation to believe that all affections are inscribed
in the face is great, and, in every age, the presumptuous observer has been able to
believe to be in possession of a universal semiology. The presumption consists in
believing that the face offers adequate and sufficient clues”21 (Starobinski 2003:
59). Yet judgments of people’s health based on their facial appearance remain
fundamental in diagnostic and prognostic practices, as well as in everyday life, also
resulting in forms of social isolation and stigmatization (Henderson et al. 2016: 1).
5. The face, health and fortune in the East
The face plays a crucial role in the Eastern semiosphere. Identified as the most
expressive part of the body, the face has received great attention in Eastern –
and more specifically, Chinese – physiognomy, based on a resemblance between
the corporeal dimension and the universe that is understood not simply
as metaphorical, but rather as physical (Wang 2020: 175). A section in The
Compendium of Divine Physiognomy (Shenxiang quanbian, 神相全編), “The
pandect of the face” (Toumian zonglun, 頭面 總論), for instance, highlights that
the face
21
“La tentazione di credere che tutte le affezioni s’inscrivano nel volto è grande, e, in ogni
epoca, l’osservatore presuntuoso ha potuto credersi in possesso di una semiologia universale. La
presunzione consiste nel credere che la faccia offra indizi adeguati e sufficienti.”
310
Simona Stano
[…] is in the sublime location among the hundred parts of the body, leading to
the divine paths of the five viscera, developing the image of the three realms [of
the Book of Changes], and determining the gains and losses of the whole body.
(Shenxiang quanbian, juan 3, 1–3, quoted in Wang 2020: 155)22
Furthermore, the face is seen as an index of the five viscera, thus representing a
sort of miniature of the inner body. This explains the success of the practice known
as 面相, Mian Xiang or Mian Shiang (literally ‘face reading’), which is thought
to have been developed in China approximately three millennia ago by Taoist
healers and monks23 and is still in vogue, though with considerable differences
(Kohn 1986), among practitioners of so-called Traditional Chinese Medicine and
various currents in alternative medicine (which nowadays are also practised in
some Western countries).
Despite the peculiarities of the multiple methods of analysing facial structures
and describing divisions of the face in Eastern physiognomy, a correspondence
between specific organs or parts of the body (such as, for instance, the liver or
heart) and particular areas of the face (for example, the eyebrows or the nose)
can be identified. In such a perspective, issues related to the former manifest
themselves with alterations of the latter (e.g. rashes, skin discolouration, dryness,
etc.), in a logic that, echoing the above-described dynamics, views the disease as a
marked term and health as an unmarked one.
The aspects considered in Mian Xiang, however, are not limited to the purely
physiological dimension, embracing different factors and extending to the realm
of divination. A crucial reference, for instance, is the conceptual scheme of Wu
Xing, or “Five Elements” – i.e. wood (木, mù), fire (火, huǒ), earth (土, tǔ),
metal (金, jīn) and water (水, shuǐ), which are linked to each other by positive
(creation–transformation) and negative (destruction–control) cycles. The visual
representation of such relationships has given rise to a series of “holistic” maps of
the face which, in addition to physical health, involve a wide range of factors, such
22
The section is also present in the earlier Mayi manual (Mayi shenxiang, juan 2, 2), as
reported in Wang 2020: 155.
23 As reported by Kohn (1986: 255), standard modern physiognomy goes back to the tenth
century and, “being established as a standard tradition, was transmitted auspices of Taoism. As
is physiognomy itself, Taoism is concerned with the human body and closely related sciences of
divination. However, when in the tenth century physiognomy began to stand on its own feet, its
interrelatedness Taoist tradition became less obvious, […]”. In the only work on physiognomy
in the Taoist Canon remaining today, the Lingxin jingzhi, the emphasis is placed particularly on
the significance of the eyes, ears and the mind which are called the “three lights” of man, and
facial symptoms are related to certain periods of time (Kalinowski 1985).
The face of health in the West and the East: A semio-cultural analysis
311
as personality, temperament, talents, etc., putting them in direct correlation with
specific points of the face.
Particular attention has historically also been devoted to the shapes (with ten
different facial forms described in the Divine Fortuneteller Ma Yi, cf. Shi, Wu
2016: 72–73), colours [which are considered in details in the Dunhuang (P. 3390),
as remarked by Hou (1979), and in the Zhengyifawen xiuzhen yaozhi (DZ 1270;
13b-14a), which has been studied by Strickmann (1985)], and parts of the face
[reaching the very articulated systematizations of the Divine Fortuneteller Ma Yi,
where eyebrows were classified into 23 types in terms of denseness and sparseness,
the inclining angles, continuity; eyes into 33 types in terms of the inclining angles,
interocular breadth, eye fold of the upper eyelid, opening height and direction
of eyeslits; mouths into 16 types in terms of thickness of lips, breadth of the oral
fissure, direction of the oral fissure, closure of the lips; noses into 24 types in
terms of shapes of nasal tip, nasal profile, nasal root height, breadth of alae nasi;
and so on and do forth (Shi, Wu 2016: 73)], always combining medical aspects
with divination (Despeux 2005). The system of the Five Officials (wuguan, 五
官), for instance, identifies the ears as the Officials of Adoption and Audition
(caiting guan, 采聽官), the eyebrows as the Officials of Longevity (baoshou guan,
保壽官), the eyes as the Officials of Inspection (jiancha guan, 監察官), the nose
as the Official of Judgment and Discernment (shenbian guan, 審辨官), and the
mouth as the Official of Income and Expenditure (chuna guan, 出納官), linking
them to the representation of good fortune in different phases of a person’s life.
At the same time, the Five Officials are believed to mirror the viscera and inner
organs, revealing internal disease to the outside: the ears reflect the status of the
kidneys, the eyebrows the gallbladder and the heart, while the nose, the lungs, and
the mouth the overall system of the five viscera, making internal imbalances and
problems manifest on the outside.
As remarked by Xing Wang (2020), it is essential to note that, regardless of the
specificities of each model, all the different ways of analysing the face in Eastern
physiognomy seem to conceive it as
[…] a comprehensive replica of the universe, with different officials, geographic
landscapes, and celestial bodies all positioned on the corresponding facial areas.
The face is a microcosm within another microcosm (the whole body). […]
Although the body is always transforming through time, larger patterns of its
movement are already visible at the beginning of a person’s life, spatialised on the
face in accordance with the cosmic order. (Wang 2020: 170)
312
Simona Stano
6. Physiognomy between the East and the West
The practice of assessing people’s personality and identity by analysing their
body and especially their face is not confined to the Eastern semiosphere. In fact,
physiognomy originated in classical antiquity, spreading both in the East and in
the West. In Western culture, it developed through different works, including
the volume De Physiognomonia by Polemo of Laodicea (2nd century AD),
Physiognomonica by Adamantius the Sophist (4th century), and the manuscript
de Physiognomonia by an anonymous Latin author (approx. 4th century), as well
as the reflections of famous Latin authors such as Juvenal, Suetonius and Pliny
the Elder. Moving from earlier classical works to Medieval studies, physiognomy
particularly developed its predictive and astrological side, digressing into prophetic
folklore and magic. It was then reintegrated into the scientific field, experiencing a
period of fervent development in the 16th century, especially with Giovan Battista
Della Porta, who focused on how people’s temperament influences their facial
appearance and character, followed by Johann Kaspar Lavater, who introduced
the idea that physiognomy relates to the specific character traits of individuals,
rather than general types, Franz Joseph Gall, who developed the theory of
‘organology’ and the method of ‘cranioscopy’ (later known as ‘phrenology’), and
Cesare Lombroso, who proposed physiognomy as a means of detecting criminal
tendencies. Subsequently it suffered a progressive lull of interest, becoming a “parascientific discipline”.
By contrast, in Eastern culture, physiognomy is still widely accepted and
practiced, especially in China, where the relation between facial features and
“destiny” has never stopped playing a crucial role. Souphiyeh Samizadeh (2020),
for instance, has highlighted its importance in the cosmetic procedures offered
nowadays by aesthetic practitioners, as well as in the main treatments requested
by patients, pointing out the diffused belief in the influence of facial appearance
on various aspects of life, such as luck, career, etc. Apart from these aspects, as
mentioned above, facial examination has always been a primary component of the
diagnostic method in Traditional Chinese Medicine, serving for both preventive
and healing practices. Advancement in computer vision and Artificial Intelligence
have also lead to Face Reading Technologies-based applications, which are offering
new, individual techniques for health check-up and advice to everyone (see Ding et
al. 2019). All these practices reiterate what was observed in the previous examples,
making the face appear as a sort of blank, unmarked canvas on which illness, moral
deviance, and even the caprices of fate leave their traces or marks.
The face of health in the West and the East: A semio-cultural analysis
313
7. Conclusion
Although the examples considered above certainly do not exhaust the wide
and extremely varied universe of the discursivization of the face of health, they
allow drawing interesting observations from the point of view of the processes of
meaning-making. Despite the differences between the Western and the Eastern
semiosphere, the face of health always appears as a sort of “zero degree” of the face,
functioning as a surface for the inscriptions of malaise and disease (and in some
cases even of deviance and negative predispositions). Thus, if on the one hand, such
a “zero degree” is euphorically characterized by the absence of disease and, more
generally, of divergence, on the other hand, contrary to the Barthesian ideal of a not
connoted and not ideologized “face-Idea” (Barthes 1957), it paradoxically ends up
being reduced to an anonymous “average face” (van Campen 2000;24 Guido et al.
2017; Roelens 2017). It is an “empty” face, deprived of the signs of nonconformity,
but with them also of its own singularity and identity (as it becomes evident in
the practice of Mian Xiang), in favour of the adaptation to an unmarked ideal
that interprets every redundancy in a way that dominates and eliminates it under
the guidance of specialized and “scientificized” knowledge. In fact, it should not
be forgotten that any face offering itself to recognition is not a given object, but
rather a reality continuously re-constructed by symbolic systems (Magli 1995:
21–25), even though it is treated as a surface through which something internal
(e.g. a disease, a moral inclination, or a specific predisposition) can be reflected
on the outside, and therefore subjected to presumptive approaches (from medical
semiotics to physiognomy and divination).
It is in this sense that the analysed cases recall the two movements of the
abstract machine of ‘faciality’ (visageité) described by Gilles Deleuze and Félix
Guattari in Mille Plateaux (1980): on the one hand, a principle that distributes and
orders different elements (e.g. points, lines, areas) to constitute a facial unit; on the
other hand, a criterion that selects and excludes non-conformities, relegating them
to anomaly and deviance (at a physical level, in the case of Western medicine, but
also a behavioural one, in the case of Mian Xiang and some practices deriving from
Chinese medicine, as well as in the case of physiognomy).
This is highly relevant, since it highlights what we could define the “unattainability” of the face of health. As we pointed out in the previous sections, whether
we consider the anonymous, or in any case interchangeable, faces of Hygeia or
Salus, which are marked by the symbolism of the ornaments they wear rather
24
Van Campen, Crétien 2000. De aantrekkingskracht van gemiddelde gezichten. Psychologie
Magazine, March 2000, available at http://www.synesthesie.nl/pub/gezicht.htm, was last
accessed on 9 February 2020.
314
Simona Stano
than their intrinsic features, or Western medicine’s facies composita without any
representation, or even the de-personalized and unnatural representations found
in magazines and online communication on well-being, we are always dealing
with a generic, anonymous, and “a-significant” face, constructed by difference
with respect to the multiple faces which, giving expression to deviance, disease
or at least forms of malaise, fully fall within the idea of corps propre, or ‘MyselfEnvelope’, described by Fontanille (2004: 22) “what is constituted in semiosis,
what is constructed in the encounter of two levels of language, in the discourse
in progress”25 – and which, therefore, is the bearer of identity and its unceasing
processes of re-definition.
In conclusion, the analysed cases show how, in a sort of “veridictory determinism”, it is only in the symptomaticity of illness (i.e. the marked facies of Western
medicine, the “signs” of Chinese facial reading, etc.), or at most of malaise (i.e.
the traces left by ageing), and deviance, that truth (Greimas 1966) seems to be
able to manifest itself: by “breaking” the machine of faciality and pluralizing
the supposed unitary and perfect state of health that the above-described representations promote, symptoms stand out for their diversity and specificity,
revealing (unveiling) to the outside internal processes that were inaccessible before.
The asymptomatic face of today’s health and well-being, on the contrary, seems to
be inevitably destined to fall into a lie (of cosmetic surgery, whether it contrasts
physical problems, moral inclinations or negative predispositions, anti-aging
measures, photo editing, etc.), or at the most into a secret (of an invisible disease,
imperceptible from the outside, which is often all the more dangerous for this very
reason). This is a somewhat paradoxical phenomenon, especially if we consider the
incredible quantity of images of faces that, from the pages of wellness magazines
to the trendiest posts of weblogs and social networks, promise us to “give body” to
health, yet in fact all end up contributing to reinforcing of such dynamics. And it is
perhaps even more contradictory if we think of a rather common, yet problematic,
expression that we generally use to compliment sound people: “You are the picture
of health!”
25
“Ce qui se constitue dans la sémiose, ce qui se construit dans la réunion des deux plans du
langage, dans le discours en acte.” (My translation, S. S.)
The face of health in the West and the East: A semio-cultural analysis
315
References
Barthes, Roland 1957. Mythologies. Paris: Seuil.
Barthes, Roland 1972. Sémiologie et médicine. In: Bastide, Roger (ed.), Les sciences de la
folie. Paris: Mouton, 37–46. https://doi.org/10.1515/9783111558370-002
Deleuze, Gilles; Guattari, Félix 1980. Mille plateaux. Paris: Éditions de Minuit.
Despeux, Catherine 2005. From prognosis to diagnosis of illness in Tang China:
Comparison of the Dunhuang Manuscript P3390 and medical sources. In: Cullen,
Christopher; Lo, Vivienne (eds.), Medieval Chinese Medicine: The Dunhuang Medical
Manuscripts. London: Routledge, 31–83.
Ding, Xianghua; Jiang, Yanqi; Qin, Xiankang; Chen, Yunan; Zhang, Wenqiang 2019.
Reading face, reading health: Exploring face reading technologies for everyday health.
Paper No 205. In: Brewster, Stephen; Fitzpatrick, Geraldine; Cox, Anna L.; Kostakos,
Vassilis (eds.), CHI ‘19: Proceedings of the 2019 CHI Conference on Human Factors
in Computing Systems. New York: Association for Computing Machinery. https://doi.
org/10.1145/3290605.3300435
Dorland, William Alexander Newman (ed.) 2011. Dorland’s Medical Dictionary. (32nd ed.)
Philadelphia: Saunders/Elsevier.
Fabbri, Paolo 1995. Abbozzi per una finzione della cura. In: Donghi, Pino; Preta, Lorena
(eds.), In principio era la cura. Roma–Bari: Laterza, 27–46.
Fontanille, Jacques 2004. Soma et séma. Figures du corps. Paris: Maisonneuve & Larose.
Fontanille, Jacques 2005. Il malessere. In: Marrone, Gianfranco (ed.), Il discorso della salute.
Verso una sociosemiotica medica. Roma: Meltemi, 35–50.
Fontanille, Jacques 2013. Figures of the body and the semiotics of imprint: Semiotic
figures of the body in the humanities. Chinese Semiotic Studies 9(1): 37–52. https://doi.
org/10.1515/css-2013-0104
Greimas, Algirdas J. 1966. Sémantique structurale. Paris: Larousse.
Greimas, Algirdas Julien; Courtés, Joseph 1982[1979]. Semiotics and Language: An
Analytical Dictionary. (Crist, Larry; Patte, Daniel, trans.) Bloomington: Indiana
University Press.
Guido, Laurent; Hennard Dutheil de la Rochère, Martine; Maire, Brigitte; Panese,
Francesco; Roelens, Nathalie (eds.), 2017. Visages. Histoires, représentations, créations.
Lausanne: BHMS.
Gutiérrez, Lenin Vladímir; Mellado Pérez, Adrián; Saavedra, María de los Ángeles 2004.
Origen y evolución del símbolo de la medicina. Educación Médica Superior 18(2): 1.
Hæger, Knut 1988. The Illustrated History of Surgery. New York: Bell Publishing Company.
Henderson, Audrey J.; Holzleitner, Iris J.; Talamas, Sean N.; Perrett, David I. 2016.
Perception of health from facial cues. Philosophical Transactions of the Royal Society B:
Biological Sciences, 371: 20150380. https://doi.org/10.1098/rstb.2015.0380
Hippocrates 1849[C. 400 BCE]. Prognosticon. In: The Genuine Works of Hippocrates. Vol.
I. (Adams, Francis, ed., trans.) London: Sydenham Society.
Hou, Ching-lang. 1979. Physiognomie d’après le teint sous la dynastie des Tang. In: Soymié,
Michel (ed.), Contributions aux études de Touen-houang. Paris, Genève, 55–70.
Kalinowski Marc 1985. La littérature divinatoire dans le Daozang. (Paper presented at the
Quatrième Colloque Pluridisciplinaire Franco-Japonais, Paris, October 1985.)
316
Simona Stano
Kohn, Livia 1986. A textbook of physiognomy: The tradition of the “Shenxiang quanbian”.
Asian Folklore Studies 45(2): 227–258. https://doi.org/10.2307/1178619
Leone, Massimo 2018. The semiotics of the face in the digital era. Perspectives. Journal of
RFIEA, the French Network of Institutes of Advanced Studies 17: 27–29.
Lotman, Juri M. 1977. Primary and secondary communication-modelling systems.
In: Lucid, Daniel P. (ed.), Soviet Semiotics: An Anthology, Baltimore: John Hopkins
University Press, 95–98.
Magli, Patrizia 1995. Il volto e l’anima. Fisiognomica e passioni. Milano: Bompiani.
Marrone, Gianfranco (ed.) 2005. Il discorso della salute. Verso una sociosemiotica medica.
Roma: Meltemi.
Merleau-Ponty, Maurice 2012[1945]. Phenomenology of Perception. (Landes, Donald A.,
trans.). London, New York: Routledge.
Murillo-Godínez, Guillermo 2010. El símbolo de la medicina: la vara de esculapio
(asclepio) o el caduceo de Hermes (Mercurio). Medicina Interna de México 26(6):
608–615.
Rillo, Arturo G. 2008. El origen griego del caduceo: Esculapio. Colombia Médica 39(4):
384–388. https://doi.org/10.25100/cm.v39i4.620
Roelens, Nathalie 2017. La hantise du visage chez Roland Barthes et Gilles Deleuze. In:
Laurent, Guido; Hennard Dutheil de la Rochère, Martine; Maire, Brigitte; Panese,
Francesco; Roelens, Nathalie (eds.), Visages. Histoires, représentations, créations. Lausanne: BHMS, 71–88.
Samizadeh, Souphiyeh 2020. Chinese facial physiognomy and modern day aesthetic practice.
Journal of Cosmetic Dermatology 19: 161–166. https://doi.org/10.1111/jocd.12980
Sbisà, Marina 2005. Corpo e testo. In: Marrone, Gianfranco (ed.), Il discorso della salute.
Verso una sociosemiotica medica. Roma: Meltemi, 69–78.
Sebeok, Thomas 1976. Contributions to the Doctrine of Signs. Lisse: Peter de Ridder Press.
Sebeok, Thomas 2001. Signs: An Introduction to Semiotics. Toronto: University of Toronto
Press.
Shi, Bo; Wu, Chao. 2016. Anthropometry in Ancient China – based on the physiognomy
book of Divine Fortuneteller Ma Yi. In: Proceedings of the 2nd Annual International
Conference on Social Science and Contemporary Humanity Development. Paris: Atlantis
Press, 70–75. https://doi.org/10.2991/sschd-16.2016.14
Stano, Simona 2019. La soglia del senso. Il corpo come istanza semiotica. In: Leone,
Massimo (ed.), Il programma scientifico della semiotica. Roma: Aracne, 149–162.
Stano, Simona 2020. Between semeiotics and semiotics: The body as a signifying text.
In: Soto, Marita; Baeza, Federico (eds.), Alteridades, identidades. Actas 14º Congreso
Mundial de Semiótica: Trayectorias/Proceedings of the 14th World Congress of the
International Association for Semiotic Studies: Trajectories. Vol. 2. Buenos Aires: IASS
Publications & Libros de Crítica, 133–140.
Starobinski, Jean 2003. Le ragioni del testo. Milano: Bruno Mondadori.
Strickmann, Michel. 1985. Ensigellation: A Buddho-Taoist technique of exorcism. (Paper
presented at the Quatrième Colloque Pluridisciplinaire Franco-Japonais, Paris, October 1985.)
Wang, Xing 2020. Physiognomy in Ming China: Fortune and the Body. Leiden-Boston: Brill.
https://doi.org/10.1163/9789004429550
Yap, Joey 2005. Mian Xiang: Discover Face Reading. Kuala Lumpur: JY Books Sdn Bhd.
The face of health in the West and the East: A semio-cultural analysis
317
Лицо здоровья на Западе и Востоке: семиокультурный анализ
Журналы, брошюры, блоги и другие медиа распространяют множество сообщений, в
которых советуют, как достичь или сохранить здоровье и благополучие. В подобных
сообщениях преобладает иконическое изображение лица и прослеживается
интересная закономерность: «лицо здоровья» кажется недостижимым и обычно
изображается через его противоположность – «лицо болезни» или, по крайней
мере, недомогания. При этом лицо занимает центральное место в медицине: с
древних времен наблюдение за лицом играло ключевую роль в диагностических
практиках как в западной медицине (которая рассматривает определенный внешний
вид или выражение лица как симптом конкретных заболеваний при описании
патологических состояний), так и в восточных профилактических и лечебных
практиках (важнейшим из которых является так называемое мянь сян, или «чтение
лица», связывающее медицинскую сферу с другими аспектами, такими как личность,
таланты и наклонности). Опираясь на семиотический анализ соответствующих
тематических исследований, начиная от классической иконографии и заканчивая
современной цифровой медиасферой, статья исследует репрезентацию лица
здоровья (и болезни) во времени и пространстве, уделяя особое внимание аналогиям
и различиям между западной и восточной семиосферой.
Tervise nägu idas ja läänes: semiokultuuriline analüüs
Ajakirjades, flaieritel, blogides ja mitmesugustes muudes meediakanalites levitatakse
lakkamatult sõnumeid, mis annavad meile nõu, kuidas saavutada tervist või heaolu või
seda säilitada. Sellistes sõnumites domineerib näo ikooniline kujutamine ning tuleb
ilmsiks huvitav nähtus: “tervise nägu” kui selline näib olevat saavutamatu ja seda esitatakse
üldjuhul eristuse kaudu, osutades selle vastandile, “haiguse näole” või vähemalt halva
enesetunde omale. Tegelikult on nägu meditsiinivaldkonnas kriitilise tähtsusega: iidsest
ajast peale on näovaatlustel olnud diagnostilises praktikas oluline osa nii lääne meditsiinis
(kus patoloogiliste seisundite kirjeldamisel kasutatakse mõistet facies, mille all mõeldakse
eristuvat näoilmet või välimust, mis seostub konkreetse meditsiinilise seisundiga) ning
idamaistes profülaktilistes ja ravitehnikates (milles on fundamentaalsel kohal niinimetatud
Mian Xiang ehk ‘näolugemine’, mis ühendab meditsiinivaldkonda teiste aspektidega, nagu
seda on isiksus, anded ja kalduvused). Artikkel toetub nii klassikalisest ikonograafiast kui ka
nüüdisaegsetest digitaalsetest meediamaastikest pärinevate teemakohaste juhtumiuuringute
semiootilisele analüüsile ning uurib tervise (ja haiguse) näo kujutamist läbi ruumi ja
aja, keskendudes eriti lääne ja ida semiosfääri vahelistele analoogiatele ja erinevustele.
Sel eesmärgil toetutakse nii näo kujutamist ja mõistmist käsitlevale kirjandusele kui ka
meditsiiniliste märgisüsteemide ja diskursuste uuringutele.
318
Gabriele Marino
Sign Systems Studies 49(3/4), 2021, 318–337
Cultures of the (masked) face
Gabriele Marino1
Abstract. What we generally regard as ‘the face’ should be semiotically understood
not as something given and monolithic, but rather stratified – it is at least threefold:
biological (face), physiognomic (expression), perceivable (visage) – and relational
as it has to be put within a narrative in order to make sense. The face lies at the
centre of a whole semiotic system, the form of life, revolving around the issue of
identity (which the face – the visage, to be precise – embodies and still does not
resolve). What we may call ‘the natural face’ is not, as common sense would suggest,
the precondition of the ‘culturalized’ one (featured with modifications ranging
from make-up and proper masks to surgery), but rather just one of the possible
semiotic masks a person may choose to wear. At the same time, the mask does
not have to be reduced to a deceptive device only (nor to be meant merely as a
material object), being in fact at the centre of a more complex axiology. The classic
veridictory square articulating the opposition between Being and Seeming may
provide a suitable model for the semiotic square of ‘visageness’, so that we would
have: Face, Disguise (the place of the mask proper), Fake, Anonymity. Based on
these theoretical premises, the paper finally addresses popular music and outlines
a provisional map of the pragmatics of the mask (subtractive vs. additive, ritualistic
vs. continuous, material vs. virtual, mask as face vs. face as mask), as a suggestion
for further studies.
Keywords: face; form of life; mask; semiotics; visage
In memory of Daniel Dumile
aka MF Doom (1971–2020)
1. Introduction: Semiotics of the (threefold) face
The face has been studied in a semiotic fashion: semioticians have been studying
the face among other things – more often than not, in order to understand other
things. Still, there is no such thing as ‘the semiotics of the face’. This is no surprise:
1
Department of Philosophy and Educational Sciences, University of Turin (UniTo), Italy;
e-mail: gabriele.marino@unito.it.
https://doi.org/10.12697/SSS.2021.49.3-4.04
Cultures of the (masked) face
319
the face is such an immediate semiotic speaker that even semiotics may have fallen
into the bias of not focusing on what seems so obvious – being such an immediate
semiotic speaker, the face is hardly detachable from human communication as a
whole. It is only when the face becomes less an obvious object that its “obtuse” –
as Barthes (1970) would call it – meaning unfolds; as, for instance, when it is
modified, hidden, denied.
What in English we generally regard as ‘the face’ is at least three different
things: (1) a biological surface, that makes it possible for (2) the sophisticatedly
culturalized expressive mask of the human being to become (3) visible. We have
not only the face (Lat. facies), but also the expression (vultus) and the visage (visus);
in semiotic literature such a trichotomy is supported, for instance, by Magli (esp.
1995: 9–15; see also Fabbri 1995)2. The notion of the Actant (Greimas, Courtés
1982[1979]: 5–6) deals with the fact that different tokens may play the same role
and that the same token may embody different roles, which is the case with what
we generically call ‘the face’: a single Actor embodying different Actantial roles. It
is through (1) this body part that (2) the emotions codified in the physiognomic
knowledge stratified during ages of biological evolution and cultural codification
are being expressed, making it possible for (3) the persona to surface in the
conjunction between the collective (our cultural grids) and the individual (each
single subjectivity). In terms of Peirce’s philosophy of the sign (CP 2.87), the
singularity manifested would stand as the Firstness of the Representamen (visus),
the material datum as the Secondness of the Object (facies) and the mechanism
that links perceptions and expressions into a crystallized – but still customizable –
code as the Thirdness of the Interpretant (vultus); through the first, which may, but
need not, coincide with the second, we reconnect to and reconstruct the third one.
The primacy of the face in the definition of ‘the other’ as a Subject has
been posited by psychoanalysis (‘the mirror stage’ in Lacan 1949), researched
by experimental psychology (the life-long studies of Ekman) and extensively
problematized by anthropology (for a semiophile overview, see Affergan
2005[2003])3. A famous passage in Jakobson (1960: 377) shows, in a nutshell, how
the face is never given, being in fact subject to complex (trans)cultural negotiation:
A missionary blamed his African flock for walking around with no clothes on. ‘And
what about yourself?’ they pointed to his visage, ‘are not you, too, somewhere naked?’.
‘Well but that is my face’. ‘Yet in us’ retorted the natives, ‘everywhere it is face’.
2
Needless to say, in common language the three terms are generally employed as synonyms.
Ekman (1978) proposed an early (para)semiotic – in the Peircean, Sebeokian sense – approach
to face. Levinas (1961) considers what he calls ‘the visage’ so meaningful to mankind precisely
due to its capability to convey the ‘presence of the other’ through what he calls ‘the expression’.
3
320
Gabriele Marino
We consider meaningful anything equipped with what we recognize as a face; in
turn, on the one hand, we assign a face to what we want to consider meaningful
and, on the other hand, the face is so meaningful to us that we assign it even
to meaningless things (which is the case in pareidolia). This process of facemaking, rooted in the gestaltic need to find familiar signs in reality, has been
encoded by means of language, thus trespassing from the biological into the
cultural, to the extent that we ended up living in a ‘facial society’ (Treusch-Dieter,
Macho 1996); the proliferation of face signs in digital communication such as
emoticons, emojis, selfies, filters is just one of the many corollaries4. To sum it up:
anthropomorphization cannot help coming in via facialization.
The synecdoche (a part for the whole) of the body, the metonym (the concrete
for the abstract) of the identity, the prosopopoeia (presentification, personification)
of the individual, the face is where our sociality is grounded and it still keeps its
role – the catalyst of the persona’s signification – when it is subject to modification
or occultation. Face cosmetics, disguise, masking, denial are strategies semiotics
seizes for to confirm one of its foundational hypotheses: meaning-making never
stems from an isolated piece of information, but rather is always relational and
narrative5. Be it altered or hidden, the face signifies as it is set within a world of
faces (proposed as natural and displayed in plain sight) and related to what it
alludes to or stands for.
2. Natural face and semiotic mask
A fabric with floral fantasy makes it possible for a – faceless, indeed – Nude by
Boubat (Floch 1985) to transition from the state of Nature into Culture, thus
becoming a mythological figure by definition (i.e. subsuming the oppositive
duality). Likewise, all the parafacial – more precisely, perifacial – devices such as
make-up, haircut, beard styles, tattoos, implants, surgery, wearables and accessories
(jewellery, headpieces, piercings, proper masks, etc.) make it possible to turn the
face into the visage; namely, to display intervention into this body part so as to
make it recognizable as culturalized6. What we keep calling simply ‘the face’ is
4
For pareidolia, see Stano 2021; for emoticons and emojis, see Marino 2015 (available as
Marino, Gabriele 2015. :-) come emoticon. Più che semplici faccine. Doppiozero Feb. 9. https://
www.doppiozero.com/materiali/ovvioottuso/come-emoticon) and McCulloch 2019: 155–195.
For selfies, see Del Marco 2015 (available as Del Marco, Vincenza 2015. Selfie. Forme e pratiche
riflessive. il lavoro culturale Nov. 23. https://www.lavoroculturale.org/selfie-forme-e-praticheriflessive/vincenza-del-marco/2015/) and Leone 2018.
5
A recent discussion of this classic structuralist belief can be found in Violi 2018.
6
‘Para-’ and ‘perifacial’ are calques from Genette 1982; according to the French narratologist,
paratexts are the ‘thresholds of the text’ that serve as its complementary annexes, a specific
Cultures of the (masked) face
321
never the facies (with all due respect to doctors seeking signs of illness in the
proverbial facies hippocratica), but always something profoundly culturalized: the
face we talk about and look towards is always a semiotic mask (Fig. 1)7. The natural
face is just a constructed ‘degree zero’ layer – to paraphrase Barthes 1947 – of our
‘faciality’ (Fr. visageité; Guattari 1979; Deleuze, Guattari 1980)8, selected among
the many possibilities; in the wake of Goffman 1955, we may call this process of
selection ‘face-work’.
Figure 1. Stratification of the ‘semiotic mask’.
category of which are peritexts, marked by spatial continuity with the text. From the Genettean
notion of the epitext, i.e. paratext with no spatial continuity with the text, we may also derive
the idea of ‘epifaciality’, as is the case with hands contributing to facial expressions (e.g. the
facepalm), thus turning the face into the visage; see Marino 2020 (available as Marino, Gabriele
2020. Facepalm. Semiotica epifacciale della frustrazione. In: Leone, Massimo (ed.), Volti virali.
Torino: FACETS Digital Press, 67–100. http://www.facets-erc.eu/wp-content/uploads/2020/05/
Massimo-LEONE-2020-Volti-virali-PDF-Editoriale-Compresso.pdf).
7
In a digital context we would talk of the avatar.
8
The notion has prominent political/juridical connotation as it is something of a heteronormative nature (see ‘regime of visibility’ and ‘regime of enunciability’ below), being a critique
of the ontological, universalistic philosophy of the face by Levinas.
322
Gabriele Marino
In Greimassian terms the mask wants “the floor to be taken” (It. “vuole che una
parola sia presa”) by means of Enunciation (Damisch 1979: 789)9; it is a débrayage,
a disengagement from the enunciation source – a consequence of any utterance
which is being posited as its precondition – by means of articulating the categories
of Person, Space and Time in discourse as ‘They’, ‘There’ and ‘Then’. The face is an
embrayage, the Meaning effect (Fr. effet de sens)10 of going back to the enunciation
source where ‘I’ am saying something ‘Here’ and ‘Now’ (to ‘You’); it is a re-turn,
something reconstructable only ex post, identifiable as a marked place within the
text where it folds back upon itself11.
In other terms, thanks to the fact that we can modify or cover the face, we
need to establish the virtuality of another layer of meaning which would lie
underneath and, at the same time, is a consequence of all the possible interventions
and overlays. The face is not the precondition of the mask12 but, on the contrary,
only thanks to the semiotic device of the mask can we eventually conceive a
pre-semioticized, unmasked face: as “every visage is a mask” (Damisch 1979:
789), actually the “mask puts the face” on man (Affergan 2005[2003]: 315). The
problem of the mask is “the problem of identity in general, in its social and cultural
construction, attribution, display and transformation: the process of continuous
meaning construction that Peirce called ‘semiosis’” (Pollock 1995: 594). The maskface – and not, simply, the face mask – is the screen onto which the form of life
hiding, revolving, arising behind it is being projected.
3. Face and life
The notion of ‘form of life’ (Germ. Lebensform) was first employed in a philosophical sense by Wittgenstein (1953, 1969). A form of life would be the
condition of Being that makes meaning possible: the rules determining human
behaviour should be understood only with reference to a specific form of life,
which has determined them in the first place so as they would reflect it. Albeit
never thoroughly explained by Wittgenstein, the term has achieved great success
in philosophical discourse: it is a notion central to contemporary pragmatism
9
For Enunciation, see Greimas 1975, Greimas, Courtés 1982[1979]: 103–105.
See Greimas, Courtés 1982[1979]: 187–188.
11 The analysis of the blank, expressionless ‘neutral mask’ by Marsciani (1990, 1996), considered as a case of ‘missing enunciation’ (It. enunciazione mancante), is consistent with this
logic.
12 As, for instance, in Levinas 1961; for a semiotic interpretation of his face philosophy, see
Ponzio 2007.
10
Cultures of the (masked) face
323
and, among others, Agamben. The latest developments of structural-generative
semiotics recovered the notion as well; in 1991 Greimas suggested it as a topic
for a seminar, which never took place, and Fontanille (1993) and other members
of the Paris School developed it further (Landowski talked of ‘lifestyle’; Latour, a
philosopher close to Greimassian semiotics, of ‘modes of existence’).
According to Fontanille (2015: 260) forms of life are the “immediate constituent
of the semiosphere”, providing “a coherent deformation of the semiotic function”
(Fontanille 2004: 103) “that affects all the levels of the generative trajectory of
meaning of any discourse or semiotic universe: ranging from sensory and
perceptual schemes to narrative, moral and axiological structures” (Fontanille 2004:
409)13. Influenced by the theories of embodiment14, as well as by the reflections
of Greimas (1987) upon the ‘aesthesia’, Fontanille proposes a development of
the narrative theory that links it to body and experience: a form of life is not
only the consistency – a kind of macro-metaisotopy (isotopy being semantic
recurrence) – that holds together a given narrative15; rather, a form of life is the
semiotic norm that pervades and, in the first place, determines such narrative
through its enactment, its bodily and existential translation into experience. To
live according to a given form of life means to conform to a regulative model of
action, to pursue a style of strategic behaviour aimed at achieving a result which
is consistent with the foundational axiology. The form of life is exactly such a
teleological projection from a set of premises to a set of consequences16; it is no
coincidence that Fontanille has been focussing so much on this notion, as he had
already proposed, along with Zilberberg (Fontanille, Zilberberg 1998), a tensive
model of semiotics.
Given that the form of life is what holds together all the possible enunciates,
textual products or substantializations stemming from the very same semiotic
13
For the semiosphere, see Lotman 1984; for the Generative trajectory, Greimas, Courtés
1982[1979]: 132–134. (Translations into English are mine, unless indicated otherwise; G. M.)
14 The theories of the extended mind, which include embodiment (or enactivism), were, in
turn, influenced by phenomenology (Merleau-Ponty) so that they conceive cognition as an
activity, rather than a capability, performed not only by the brain but through the whole body
as situated in a given environment. The embodiment book which Fontanille references the
most is Varela, Thompson, Rosch 1991.
15 A narrative generated in the deepest levels of axiology (set of values, Semiotic square), progressively anthropomorphized and articulated through Semio-narrative (Modalities, Narrative
programs and Actantial roles) and Discoursive structures (Thematization and Figurativization
as regards the semantic components; Actorialization, Temporalization and Spatialization as
regards the syntactic components) and eventually manifested in the surface of text.
16 In these terms, the form of life may be conceived as what Eco (1976[1975]: 289–298)
defines as ‘ideology’.
324
Gabriele Marino
system, it is the metalevel that ensures intersemiotic translation (the mutual
translatability between different matters, modes, media according to Jakobson
1959). The mask stands as a perfect Figural device (the pre-, protofigurative level
within the Generative trajectory of meaning that is the place of intersemioticity, as
the semantics of what would become the different Object-semiotics are all together
and in mutual exchange) capable to condense a whole form of life17. Lotman
(2002[1981]) developed such an intuition in a little-known essay of his: in the
afterword to a German edition of Gogol’s The Dead Souls he thoroughly explained
how the masks made by Russian sculptor Kaplan were capable of translating the
verbal images evoked in the novel into visual and haptic forms by developing,
thanks to the unique sensitivity of the artist, the very same structural principles
manifested through the very same figurative motives.
4. Face and lie
Claiming that the semiotics of the face has to be understood as a semiotics of the
mask is not a philosophical gimmick as one unfamiliar with semiotic reasoning
could think, but rather a methodological operation. Semiotics operates by overturning common sense, by building up general rules on the basis of what common
sense generally regards – i.e. dismisses – as exceptions. Semiotics being a system of
thought translated into an interdefined metalanguage (and being a whole wherein
each single part is defined by its relationships with the others), there is simply no
room for exceptions (single elements for which the general rule would not apply).
If we know that nature is deeply culturalized and culture deeply naturalized (as
Morin has maintained since the 1950s and the ontological turn in anthropology has
been discussing since the 1990s), building our semiotic theory of the face on the
basis of the common sense dichotomy between nature and culture would prevent
us from truly comprehending the axiologies and ideologies of the face spread
diachronically and diatopically18. If general semiotics, as posited by the famous,
17
The Figural is a crucial and yet underinvestigated notion in post-Greimassian semiotics.
The term had already been employed, with different meanings, by authors such as Auerbach,
Merleau-Ponty, Lyotard and it is only briefly addressed in Greimas, Courtés 1986: 91–93 (entry
written by Zilberberg); as far as I am concerned, the best sources in point are Jacoviello 2012:
233–242 and Lancioni 2020: 146–152. The Figural may be conceived as a kind of Ur-metaform,
where metaform is a “concept ([thinking = seeing]) that results from the linkage of an abstract
notion ([thinking]) with a concrete source domain ([seeing])” (Sebeok, Danesi 2000: 196).
Marino (2021a, 2021b) elaborates on the relationship between the Figural and the form of life
with regard to music.
18 Still, such a dichotomy (natural vs. cultural – i.e. artificial – face) is an important drive
Cultures of the (masked) face
325
provocative, tongue-in-cheek – and yet philosophically accurate – definition by
Eco (1976[1975]: 7), aims at studying signs by studying “everything which can be
used in order to lie”, a semiotics of the face should study everything which can
be used in order to “lie the face” (to make a lie of the face), to fake it, to mask it.
To paraphrase Eco, we might say that if something cannot be used to hide the
face, conversely it cannot be used to display it. The semiotics of the face is not a
semiotics of what passes for the face plain and simple, but rather the semiotics of
the visus, aimed at reconstructing how the expressive signs of the vultus are being
displayed (or not) through the biological facies. The case where the visus conveying
the vultus coincides with the facies – i.e. when the biological datum is perceivable
and communicates as such – has to be understood as a ‘mask degree zero’.
The face has been semiotically studied in connection with the portrait
(Calabrese 1981), the self-portrait (Calabrese 2006), digital communication (Leone
2019) and digital pictorialization (Danesi 2016) and, more generally, as the sign
congealing a whole aesthetic (Eco 2004, 2007). Barthes (1967) talked of the dress
as “the mask of body” and compared the photographic portrait to a semiotic “death
mask” (Barthes 1980). The semiotic approach reduces the face to the mask – as
we have seen – and the mask to a deceptive device: semiotic theatrology (De
Marinis 1982), aesthetics (Marin 1993), physiognomy (Eco 1984; Magli 1995)19,
cosmetology (Magli 2013) and the semiotic analysis of literary ekphrasis (Magli
2016) have focused on the face mainly to unmask it, to understand when, where
and how it would be faking us20. In Greimas and Courtés (1982[1979]: 67–68)
the mask is addressed only twice: in connection with the Deceiver (the trickster
of mythology, who often wears it) and in connection with the moment of the
Qualifying test when the Sender hides “under the mask of the Adversary”. A
reductionist perspective of this kind (mask = deception), profoundly influenced
by the episteme of the Classical Age (Damisch 1979: 788), would be best suitable
for handling only texts, and not entire forms of life, as we can definitely identify a
strategy of some sort, but we would have trouble judging whether and according
to which level of semiosis a mask would ‘tell the truth’ or ‘be authentic’ (or not)
in both perceptual and cultural phenomena. The hypothesis of the ‘uncanny valley’ by Mori
(see Surace 2021), that humanoids imperfectly resembling human features would provoke a
feeling of eerie familiarity, lies on the frontier between the natural and the artificial: we find a
simulacrum, a mask (humanoid), where we would expect to find a face (human being).
19 The text by Eco is the reprint of his Preface to the book Il Lavater portatile (con i disegni di
Franco Testa), an anastatic edition, issued by the publisher Moizzi in Milan, of Johann Kaspar
Lavater’s physiognomic treatise from 1788.
20 Reviews of the semiotic approaches to the mask may be found in Proschan 1983, Leone
2020, Gramigna 2021.
326
Gabriele Marino
as regards the physical person who would be wearing it21. My proposal is to
maintain the semiotic primacy of the mask over the face, to broaden the category
of mask (not merely a physical object but rather a semiotic strategy) and not to
reduce the mask to only one among its possible valorizations and uses. The classic
Veridictory square generated from the opposition between Being and Seeming
(Greimas, Courtés 1982[1979]: 369) may be fruitfully applied in order to identify
the axiologies of ‘visageness’ (see Fig. 2):
Figure 2. The semiotic square of ‘visageness’.
Truth Face (transparency, mask degree zero, personification);
Lie Disguise (opacity, mask as face, impersonation);
Falsehood Fake (non-transparency, face as mask, unsuccessful personification);
Secret Anonymity (non-opacity, no face, depersonification)22.
21
Needless to say, the relationship between face and mask as regards the issue of identity is
a classic philosophical theme, especially in the 20th century, when the idea of a fragmented
identity prominently emerges; e.g. according to Gurisatti (2006: 222), Nietzsche’s Zarathustra
would programmatically be nothing more than a “shallow dance of masks-without-face, formswithout-content, appearances-without-essence”.
22 The opposition between depersonification (It. incognito, annullamento dell’identità) and
impersonation (travestimento, sostituzione di identità) is drawn from Damisch 1979: 776.
According to Sbisà (2002: 14), “the definition of Falsehood is controversial, but we can explain
it by referring to a Sender that judges a statement produced by the Subject as ‘It is not this way,
Cultures of the (masked) face
327
We can think of a semiotics of the face focusing on the very face in itself – for
instance, a semiotics of its representation (which is the semiotics of the portrait).
Yet when we aim at building a semiotics of the face within the framework of the
semiotics of culture (dealing with the face within a given semiosphere, as a cultural
unit of a given encyclopaedia23), we cannot consider it only as a representational,
visual and haptic token24. When we meet someone new and get to know their
name, still the only identity we are told about is the anagraphic one. Likewise,
when we see the face of someone new, we do not know the person to whom it
belongs (who is behind it and so before us). In other terms, the face per se means
nothing, reveals no mystery, unless it turns into a visage, the semiotic mask: in
order to make sense, it has to become a sign, to stand “to somebody for something
in some respect or capacity” (CP 2.228). Otherwise it would merely represent the
possible starting point, supposedly of the indexical kind, for making inferences
rooted in the biological; let us think of the phantasmagoria prompted by Lombroso,
who founded his criminology by grounding it in positivistic, yet pseudoscientific
determinism: a true conspiracy of bodily signs. We have, in turn, to question
ourselves: are we immune to such conspiratorial temptation?
5. Facets of the mask
The face can be dealt with according to different ‘regimes of visibility’25, different
degrees of transparent immediacy or opacity26: the former dimension deals with
the possibility of recognition (i.e. positive comparison between the new token and
the known type), so that we would be allowed to reconnect the perceivable – the
face or its delegate27 – to a biological datum and anagraphic name; the latter deals
with the impossibility of accomplishing such a task. We can be shown a perceivable
and it does not seem this way’, thus sanctioning its evident falsehood” (Sbisà, Marina 2002. La
semiotica narrativa di A.J. Greimas. Concetti principali e istruzioni per l’uso was retrieved from
https://www2.units.it/sbisama/it/didattica/semiodisp_2.PDF). Disguise/impersonation may
be conceived also as ‘mimicry’ (Caillois 1958).
23 For the encyclopaedia, see Eco 1986[1984].
24 Outside semiotics, a comprehensive approach of this kind is being pursued by Belting
(2013).
25 A notion proposed by Foucault, with regard to power, and developed in the texts by Deleuze
(1986) interpreting Foucault.
26 Opposite terms proposed by Bolter and Grusin (1999); Marin (1994) proposes similar
categories within a proper semiotic perspective.
27 The term ‘delegate’ is inspired by Latour’s reworking of enunciation theory (see Latour
1998).
328
Gabriele Marino
that coincides with the natural face and we can be shown a face culturally modified
according to different degrees of intervention, with a special spot assigned to the
device that covers the face, hiding it partially or totally28.
In a world of over-representation and over-exposition, face occultation (the
mask as the face delegate) or face denial (no face nor delegate available) are
remarkable semiotic operations that stand for a clear identity statement, opening to
a whole set of valorizations; as a matter of fact, the mask is not so much polysemic
but polypragmatic: it has different meanings because of different uses. It can be
either imposed or chosen (both bandits and hostages wear it, but for different
reasons), it can be used to deceive or address; it is the distinctive feature of the
criminal, the enemy, the bad guy, the traitor, the spy, the coward, the victim,
the (super)hero. In other terms, the semiotic device of the mask can be used
subtractively (it denies the biological face), in order not to be recognized (to hide
oneself from the outside world), as much as additively (it is superimposed onto
the face, serving as its delegate), precisely to be icastically recognized by means
of emphasizing some features or creating alternative ones (the mask does cover
the face, but in order to amplify/reveal it)29. Just like every sign, and in the respect
of being itself a kind of “sign par excellence”, the mask is transparent and opaque
at the same time, it is both ‘a thing’ and ‘a sign’; as a thing it may cover what, as
a sign, it would reveal, so that there may be no actual relationship between what
the mask is hiding and what it is standing for, what it is signifying (Damisch 1979:
785). The mask, whatever it may be, stands as a mythopoietic mechanism that, by
questioning common sense identity (it prevents from linking the Enunciator with
an Empirical author) and triggering detection (we are prone to seek any clue of the
wearer’s form of life within their enunciations), suggests a heterotopy of values. A
Practical-referential valorization is opposed to a Utopian-existential one; with the
caveat that the former is easily at risk of being reconducted to the latter (in order
28
Here we may suggest an opposition between bandanas (covering the nose and the mouth)
and masks (leaving the nose and the mouth uncovered). While bandits use the former type
to disguise their identities, the latter type is “designed less to disguise than to signal that
disguising is taking place, a semiotic process that is effective only to the extent that it indexes
the culturally conventional sign of identity: the eyes” (Pollock 1995: 595); let us think of Diego
de la Vega/Zorro’s mask and Superman/Clark Kent’s glasses. In the constitutive asymmetry
of these two partial masks (covering some parts of the face, while leaving others uncovered),
in their capability to disrupt the features of the body in opposition to the natural symmetry
of the biological face, we may recall the individualistic élan Simmel (1901) ascribes to all
irrationalistic traits.
29 In the wake of Bouissac’s studies on clowns, Ogibenin (1975: 5) focuses on the mask as a
metasemiotic instrument capable of generating “isolation and alienation” by means of “the
hypertrophying of the physical constants of the human face”.
Cultures of the (masked) face
329
to serve as the mouthpiece for an alternative to the dominant axiology, the Subject
disguises themselves so as to be unrecognizable).
Even though its iconic status, stricto sensu, has been questioned to the benefit
of a more systemic (i.e. not only visual), performance-oriented approach (Salvatore
2015, 2018), the history of popular music is definitely an iconography of famous
faces at display, where cult figures wearing masks also exist (‘masked musicians’
is literally a thing in our shared encyclopaedia)30. Whereas the study of masks
in popular music is little more than a footnote – three lines in all – within what
we may call general maskology (Bell 2010: 1), the semiotics thereof is auroral.
Reyes (2021) analysed the 542 covers of the records published in 1988 and in 2018,
according to Wikipedia, and found that 32 of them display an ‘occluded face’;
by blending an automatic-quantitative approach, in the framework of Manovich’s
cultural analytics, and hand-made typologization, in the framework of a qualitative
research consistent with semiotic analysis, he distinguished between four types of
facial obstruction (due to a mask, a visual effect, the pose of the body, the framing
of the image) and identified two main axes of visual strategy (from a maximum of
face shown to a maximum of face hidden, from a maximum of distorted face to a
maximum of sharp face). As a matter of fact, popular music represents a promising
field of research in the framework of a semiotics of the face; hence, the following
considerations include cues for further study in this perspective.
The mask has its own aspectuality – it may be worn in coincidence with the
performance (punctual) or it can be employed continuously, as offstage as onstage
(durative): in the former case the mask, a prop contributing to showmanship, is
generally employed to define characters/alter egos and mark the performance as
a ritual (a prototype in pop-rock music may be Arthur Brown’s face painting and
burning helmet from 1968); in the latter case it is literally the metonym of a form of
life which is being pursued all the way (as in the case of the French electronic duo
Daft Punk with their signature robot-like helmets). An extreme outcome of this
totalizing logic is what we may call impersonation: a kind of ‘self-Doppelgänger-ism’
(or ‘Andy Kaufman effect’) where the performer is not merely performing, not just
playing a part, but rather living it, without breaking character (a notable example
30
See: Masked Musicians. Wikipedia. https://en.wikipedia.org/wiki/Category:Masked_musicians (category page created in 2011). The Internet is filled with popular lists and polls about
masked, disguised and/or anonymous musicians, but proper journalistic inquiries and scholarly articles are sporadic. An exception is Reynolds 2021, written after the announcement
of the death of the American masked rapper and producer Daniel Dumile aka MF Doom
(Reynolds, Simon 2021. Mask Up. Inside pop and rock history’s obsession with disguise. Tidal.
com. Jan. 14; the source was retrieved from https://tidal.com/magazine/article/pop-maskhistory/1-76368).
330
Gabriele Marino
in popular music may be Watkin Tudor Jones aka Ninja’s South African ‘zef ’ hip
hop group Die Antwoord).
The mask can be a material object or it can be immaterial. The latter is the case
with fictional performers who embody the musical Enunciator (most notably, socalled virtual bands, from the Chipmunks or Gorillaz cartoons to vocaloid Hatsune
Miku): the mask is not a device that hides a face – it is a mask literally applied to
no one – but rather the only face we are being shown (which reinforces our theory
of the semiotics of the face as, first and foremost, a semiotics of the mask); in this
case the mask is the face31. The flipside is when the face becomes nothing more
than a mask; we know the performers – we know their face – and still we do not
know them at all. As a matter of fact, some figures happen to be so much ‘wrapped
in legend’ that their oeuvre does overcome in accountability their personal story, to
which we are denied access (from the “inventor of improvisation” Buddy Bolden,
a kind of Homer for jazz music, to the lugubrious singer-songwriter Jandek). The
face may be also turned into a mask by means of its multiplication and consequent
resemantization (as in the case of the chameleonic English rock star David Bowie)
or desemantization (as in the case of the Italian singer Mina, whose face is subject
to countless modifications and stylizations, or the English electronic musician
Aphex Twin, whose face is prosthetically and digitally transformed into horrific
appearance and obsessively reproduced on album covers and in video clips, in a
kind of grotesque parody of pop music iconicism).
As it is easy to understand, facial discourse – the discourse of the face – may
be doubled by the name and voice discourses, as names and voices can be plainly
displayed, variously modified and covered or totally hidden in a very similar way as
the face can. In designing a coherent form of life, in pursuing a project of life, it is
no coincidence that stage names are used, appearances are modified, voices altered.
In general terms, in the ‘regime of enunciability’32 chosen by the performer, the
name can be articulated in transparency (birth name), as alternative (pseudonym)
or competing/coexisting (heteronym) with regard to the anagraphic datum, or it
can be totally obliterated (anonymity)33.
31
We may fairly apply this definition also to the cases of “the continuous mask”.
Again, a notion proposed by Foucault, with regard to law, and developed by Deleuze (1986).
33 For the semiotics of anonymity, see Thibault 2016 as well as Maani 2018 (available as Maani,
Sitti [alias Leone, Massimo] 2018. I giga di Gige. L’impatto dell’anonimato nella comunicazione
contemporanea. Rivista Italiana di Filosofia del linguaggio. http://www.rifl.unical.it/index.php/
rifl/article/view/520). The name has been a subject of semiotic inquiry since the prehistory of
the discipline (philosophers of Ancient Greece, Hobbes etc.), as the act of naming is perhaps
the most powerful semiotic act of all. For a semiotics of the face, understood as a semiotics of
the mask (and, therefore, of identity), names are quite a promising topic; especially with regard
to the notion of ‘embodied name identity’ (Pilcher 2015).
32
Cultures of the (masked) face
331
5. Conclusion: The face as semiotic conspiracy
The face is the human communication palimpsest par excellence and yet the
semiotic approach warns us against matching it to identity offhand. When we are
shown someone’s face we are not automatically granted access to their identity
(as if identity were to be conceived as something given and stable, to be simply
uncovered, as if it were like an ID with name, picture and signature). We may still
lack all the meaningful relationships interweaving texts and practices. The face,
the main identity mark along with the person’s name and voice (the three features
composing a kind of identity plexus)34, asks to be set within a discoursive network
in order to work semiotically (see Table 1). We may be tempted to stick to the idea
of reading the face as part of a somewhat simple, even universal code, the one
linking us with the biological world, the one of physiognomic determinism; still,
luckily, our semiotic hunger makes us unsatisfied with such codicality, as we chase
not only the face but also what is behind it and what is being conveyed through it:
what makes a face what the face in general and this one face in particular is to us.
We are obsessed by what we do not know, by what is missing: what is beyond
the door, off the camera, behind the mask. We put every clue – every sign – we
have got under a magnifying glass and we connect them to one another in order
to make sense of what we are experiencing, so as to reconstruct what is not here
from what is here, an absence through a presence. We are naturally – semiotically –
led to jump from one level to another, from the authenticity of artistic expression
to the authenticity of experience, from the aesthetic to the existential. We want
to find in the textualized authorship (Greimas’ Enunciator, Eco’s Model author,
etc.)35 not only the sincerity, the genuineness of the text, but also its truthfulness.
Each possible identity token (within the texts, in both mediated and non-mediated
presence) becomes a metatext for the other ones, so that the identity narrative may
progressively cohere into a persona.
34
They seem to be “privileged signs of identity that embarrass Western culture every time
a game of camouflage, of masking or elision tends to weaken, displace, or even undo the
identification devices” (Bonelli 2004: 9, quoted in Calefato 2006: 75). In fact, Pilcher (2015:
765) notices that the one linking “names, identity and the body” together is still “a neglected
relationship”.
35 See, respectively, Greimas, Courtés 1982[1979]: 105 and Eco 1979: 60–66.
332
Gabriele Marino
Table 1. Dimensions at stake in the identity plexus.
VISUAL DELEGATE
Semiotic mask (Culturalized face)
o Natural face
o Modified face
Covered/Hidden face (mask)
o Denied face (no face, no mask)
REGIME OF VISIBILITY
Transparency (possibility to be recognized)
Opacity (impossibility to be recognized)
Hypervisibility/Amplification/Multiplication (will to be recognized)
NOMINABLE DELEGATE / REGIME OF ENUNCIABILITY
Birthname
Pseudonym
Heteronym
Anonymity
EXTROVERSIVE DIMENSIONS
Metatextual narrative (autobiography)
Mediated presence (audiovisual)
Non-mediated presence (live performance, face to face interaction)
“Outside the text, [there is] no salvation” Greimas (1975: 25) ruled, echoing
Derrida’s (1967: 227) “There is not outside-text”; and still the text is not enough, as
our semiotic élan is centrifugal: in interacting with the texts, we seek the guarantee
that, through them, it is not only a textual persona but rather a real person who
is speaking to us about their – real or possible – world36. Interpretation becomes
investigation: a process of reconstruction, unearthing, uncovering, revelation
where we hope everything holds together and eventually makes sense. Again, it
is a conspiracy, but, this time, of the semiotic kind. The semiotics thereof shall be
directed at comprehending whether, how and to which extent does ‘the face’ plot
against itself: whether, how and to which extent, does the visage semiotically works
against the face and its expression – turning into the semiotic mask.
36
On the notion of ‘possible world’, see Eco 1978, 1979: 122–173.
Cultures of the (masked) face
333
Acknowledgements: This article results from a project that has received funding from the
European Research Council (ERC) under the European Union’s Horizon 2020 research
and innovation programme (Grant agreement No 819649 – FACETS). The author would
like to thank Simon Reynolds for the fruitful exchange of views.
References
Affergan, Francis 2005[2003]. Modelli antropopoietici della Maschera. In: Figure dell’umano. Le rappresentazioni dell’antropologia. Roma: Meltemi, 307–334.
Barthes, Roland 1947. Le degré zéro de l’écriture. Combat août: 2.
Barthes, Roland 1967. Système de la mode. Paris: Seuil.
Barthes, Roland 1970. Le troisième sens: notes de recherches sur quelques photogrammes
de S. M. Eisenstein. Cahiers du cinéma 222: 12–19.
Barthes, Roland 1980. La chambre claire. Note sur la photographie. Paris: Cahiers du
Cinéma-Gallimard-Seuil.
Bell, Deborah 2010. Mask Makers and Their Craft: An Illustrated Worldwide Study.
Jefferson: McFarland.
Belting, Hans 2013. Faces: Eine Geschichte des Gesichts. München: Beck.
Bolter, Jay David; Grusin, Richard 1999. Remediation: Understanding New Media. Cambridge: The MIT Press.
Bonelli, Elena (ed.) 2004. Segni particolari. L’immagine del viso, l’immaginario del nome
proprio. Urbino: QuattroVenti.
Caillois, Roger 1958. Les jeux et les hommes. Le masque et le vertige. Paris: Gallimard.
Calabrese, Omar 1981. La sintassi della vertigine: sguardi, specchi, ritratti. Versus 29: 3–31.
Calabrese, Omar 2006. L’art de l’autoportrait. Paris: Citadelles & Mazenod.
Calefato, Patrizia 2006. Il nome, il volto. In: Calefato, Patrizia (ed.), Che nome sei? Nomi,
marchi, tag, nick, etichette e altri segni. Roma: Meltemi, 43–77.
CP = Peirce, Charles S. 1931–1958. Collected Papers of Charles Sanders Peirce. Cambridge:
Harvard University Press. (Vols. 1–6, Hartshorne, Charles; Weiss, Paul, eds.; vols.
7–8, Burks, Arthur Walter, ed.) [In-text references are to CP, followed by volume and
paragraph numbers.]
Damisch, Hubert 1979. Maschera. (Malvano, Maria Vittoria, trans.) In: Romano, Ruggiero
(ed.), Enciclopedia Einaudi. Vol. 8: Labirinto-Memoria. Torino: Einaudi, 776–94.
Danesi, Marcel 2016. The Semiotics of Emoji: The Rise of Visual Language in the Age of the
Internet. New York: Bloomsbury Academic.
De Marinis, Marco 1982. Semiotica del teatro. L’analisi testuale dello spettacolo. Milano:
Bompiani. https://doi.org/10.2307/1772329
Deleuze, Gilles 1986. Foucault. Paris: Minuit. https://doi.org/10.5040/9781350252004
Deleuze, Gilles; Guattari, Félix 1980. Mille plateaux. Capitalisme et Shizophrénie 2. Paris:
Minuit.
Derrida, Jacques 1967. De la grammatologie. Paris: Minuit.
Eco, Umberto (ed.) 1978. Versus 19–20 (Semiotica testuale: mondi possibili e narratività).
Eco, Umberto 1976[1975]. A Theory of Semiotics. Bloomington: Indiana University Press.
Eco, Umberto 1979. Lector in fabula. La cooperazione interpretativa nei testi narrativi.
Milano: Bompiani.
334
Gabriele Marino
Eco, Umberto 1984. Il linguaggio del volto. In: Sugli specchi e altri saggi. Milano: Bompiani,
45–54.
Eco, Umberto 1986[1984]. Semiotics and the Philosophy of Language. Bloomington: Indiana
University Press.
Eco, Umberto 2004. Storia della bellezza. Milano: Bompiani.
Eco, Umberto 2007. Storia della bruttezza. Milano: Bompiani.
Ekman, Paul 1978. Facial signs: Facts, fantasies, and possibilities. In: Sebeok, Thomas
Albert (ed.), Sight, Sound, and Sense. Bloomington: Indiana University Press, 124–156.
Fabbri, Paolo 1995. Difformità del viso. In: Brusatin, Manlio; Clair, Jean (eds.), Identità
e alterità: figure del corpo 1895–1995. La Biennale di Venezia 46, Esposizione
internazionale d’arte. Venezia: Marsilio.
Floch, Jean-Marie 1985. Un nu de Boubat. Sémiotique poétique et discours mythique
en photographie. In: Petites Mythologies de l’Oeil et de l’Esprit: Pour une Semiotique
Plastique. Paris, Amsterdam: Hadès-Benjamins, 21–38. https://doi.org/10.1075/as.1
Fontanille, Jacques 1993. Les formes de vie. Présentation. RSSI–Recherches Sémiotiques/
Semiotic Inquiry 13(1/2): 5–12.
Fontanille, Jacques 2004. Figure del corpo: per una semiotica dell’impronta. Roma: Meltemi.
Fontanille, Jacques 2015. Formes de vie. Liège: Presses universitaires de Liège. https://doi.
org/10.4000/books.pulg.2207
Fontanille, Jacques; Zilberberg, Claude 1998. Tension et signification. Liège: Mardaga.
Genette, Gérard 1982. Palimpsestes. La littérature au second degré. Paris: Seuil.
Goffman, Erving 1955. On face-work: An analysis of ritual elements in social interaction.
Psychiatry: Journal for the Study of Interpersonal Processes 18: 213–231. https://doi.org
/10.1080/00332747.1955.11023008
Gramigna, Remo 2021. Le forme della maschera: aspetti semiotici della manipolazione del
volto e della plasticità dell’apparenza. Lexia 37–38: 121–140.
Greimas, Algirdas Julien 1975. L’Énonciation (une posture épistémologique). Significação:
9–25. https://doi.org/10.11606/issn.2316-7114.sig.1974.90115
Greimas, Algirdas Julien 1987. De l’imperfection. Périgueux: Éditions Pierre Fanlac.
Greimas, Algirdas Julien; Courtés, Joseph (ed.) 1982[1979]. Semiotics and Language: An
Analytical Dictionary. Bloomington: Indiana University Press.
Greimas, Algirdas Julien; Courtés, Joseph (ed.) 1986. Sémiotique: Dictionnaire raisonné de
la théorie du langage, II. Compléments, débats, propositions. Paris: Hachette.
Guattari, Félix 1979. Visagéité signifiante, visagéité diagrammatique. In: L’inconscient
machinique: essais de schizo-analyse. Fontenay-sous-Bois: Encres/Recherches, 79–116.
Gurisatti, Giovanni 2006. Dizionario fisiognomico. Il volto, le forme, l’espressione. Macerata:
Quodlibet.
Jacoviello, Stefano 2012. La rivincita di Orfeo. Esperienza estetica e semiotica del discorso
musicale. Mimesis: Milano–Udine.
Jakobson, Roman 1959. On linguistic aspects of translation. In: Brower, Reuben Arthur
(ed.), On Translation. Cambridge: Harvard University Press, 232–239. https://doi.
org/10.4159/harvard.9780674731615.c18
Jakobson, Roman 1960. Linguistics and poetics. In: Sebeok, Thomas Albert (ed.), Style in
Language. Cambridge: The MIT Press, 350–377.
Cultures of the (masked) face
335
Lacan, Jacques 1949. Le Stade du miroir comme formateur de la fonction du Je, telle qu’elle
nous est révélée dans l’expérience psychanalytique. Revue française de psychanalyse
13(4): 449–455.
Lancioni, Tarcisio 2020. E inseguiremo ancora unicorni. Alterità immaginate e dinamiche
culturali. Sesto San Giovanni (MI): Mimesis.
Latour, Bruno 1998. Petite philosophie de l’énonciation. In: Basso, Pierluigi; Corrain, Lucia
(eds.), Eloquio del senso. Dialoghi semiotici per Paolo Fabbri. Milano: Costa & Nolan,
71–94.
Leone, Massimo 2018. Semiotics of the selfie: The glorification of the present. Punctum
4(2): 33–48. https://doi.org/10.18680/hss.2018.0018
Leone, Massimo 2019. The semiotics of the face in digital dating: A research direction.
Digital Age in Semiotics & Communication 2: 18–40. https://doi.org/10.33919/
dasc.19.2.2
Leone, Massimo 2020. The semiotics of the medical face mask: East and West. Signs &
Media 1: 40–70. https://doi.org/10.1163/25900323-12340004
Levinas, Emmanuel 1961. Totalité et Infini: essai sur l’extériorité. La Haye: Nijhoff.
Lotman, Juri 1984. О семиосфере. [On the semiosphere.] Sign Systems Studies 17: 5–23.
Lotman, Juri 2002[1981]. Маска в художественном мире Гоголя и маски Анатолия
Каплана. [Mask in an artistic world of Gogol, and the masks of Anatoli Kaplan.] Sign
Systems Studies 30(2): 695–705. https://doi.org/10.12697/SSS.2002.30.2.20
Magli, Patrizia 1995. Il volto e l’anima. Fisiognomica e passioni. Milano: Rizzoli.
Magli, Patrizia 2013. Pitturare il volto. Il trucco, l’arte, la moda. Venezia: Marsilio.
Magli, Patrizia 2016. Il volto raccontato. Ritratto e autoritratto in letteratura. Milano:
Raffaello Cortina.
Marin, Louis 1993. Masque et portrait. Urbino: CISeL-Centro Internazionale di Semiotica
e Linguistica (Documents de travail et pré-publications 226, settembre, serie F).
Marin, Louis 1994. De la representation. Paris: Gallimard-Seuil.
Marino, Gabriele 2021a. The form of life of sanctity in music beyond hagiography: The
case of John Coltrane and his “Ascension”. International Journal for the Semiotics of Law.
https://doi.org/10.1007/s11196-021-09829-7.
Marino, Gabriele 2021b. Il ghigno di Aphex e altre maschere: Volti del transumano in
musica. Lexia 37–38: 197–216.
Marsciani, Francesco 1990. La maschera neutra. Carte semiotiche 7: 146–160.
Marsciani, Francesco 1996. Il volto neutralizzato. In: Fabbri, Paolo (ed.), Su la maschera –
Catalogo del XVII Festival internazionale del giallo e del mistero. Milano: Mondadori.
McCulloch, Gretchen 2019. Because Internet: Understanding the New Rules of Language.
New York: Riverhead Books.
Ogibenin, Boris L. 1975. Mask in the light of semiotics: A functional approach. Semiotica
13(1): 1–9. https://doi.org/10.1515/semi.1975.13.1.1
Pilcher, Jane 2015. Names, bodies and identities. Sociology 50(4): 764–779. https://doi.
org/10.1177/0038038515582157
Pollock, Donald 1995. Masks and the semiotics of identity. The Journal of the Royal
Anthropological Institute 1(3): 581–597. https://doi.org/10.2307/3034576
336
Gabriele Marino
Ponzio, Augusto 2007. Differenza tollerata ed eccedenza del volto. In: Fuori luogo.
L’esorbitante nella riproduzione dell’identico. Roma: Meltemi, 97–103.
Proschan, Frank 1983. The semiotic study of puppets, masks, and performing objects.
Semiotica 47(1–4): 3–44.
Reyes, Everardo García 2021. Face value: Analyzing and visualizing facial data. Lexia 37–38:
467–483.
Salvatore, Gianfranco 2015. Icone in azione. Il Pop e il Rock come drammaturgie. H-ermes.
Journal of Communication 4: 201–232.
Salvatore, Gianfranco 2018. Il teatro musicale del rock. Avanguardie, frontmen, light-show.
Roma: Stampa Alternativa.
Sebeok, Thomas Albert; Danesi, Marcel 2000. The Forms of Meaning: Modeling
Systems Theory and Semiotics. Berlin: Mouton de Gruyter. https://doi.org/10.1515/
9783110816143
Simmel, Georg 1901. Die ästhetische Bedeutung des Gesichts. Der Lotse. Hamburgische
Wochenschrift für deutsche Kultur 1: 280–284.
Stano, Simona 2021. Facing food: Pareidolia, iconism, and meaning. Lexia 37–38: 487–501.
Surace, Bruno 2021. Semiotica dell’Uncanny Valley. Lexia 37–38: 359–380.
Thibault, Mattia 2016. Trolls, hackers, anons: Conspiracy theories in the peripheries of the
web. Lexia 23-24: 387–408.
Treusch-Dieter, Gerburg; Macho, Thomas (eds.) 1996. Medium Gesicht: Die faciale
Gesellschaft. Berlin: Elefanten.
Varela, Francisco J.; Thompson, Evan; Rosch, Eleanor 1991. The Embodied Mind: Cognitive
Science and Human Experience. Cambridge: The MIT Press. https://doi.org/10.7551/
mitpress/6730.001.0001
Violi, Patrizia 2018. Dalla nave di Teseo al campanile di Venezia. Una prospettiva semiotica
sulla identità. Rivista Italiana di Filosofia del Linguaggio 1. DOI: 10.4396/SFL201904.
Wittgenstein, Ludwig 1953. Philosophical Investigations. Oxford: Blackwell.
Wittgenstein, Ludwig 1969. On Certainty. New York: Harper.
Культуры лица (в масках)
То, что мы привыкли считать «лицом», с семиотической точки зрения является не
чем-то определенным и монолитным, а имеет, по меньшей мере, тройную природу:
биологическое (лицо), физиогномическое (выражение), воспринимаемое (визаж).
В то же время лицо имеет относительный характер, так как несет смысл, лишь
будучи включенным в повествование. Лицо является центром всей семиотической
системы – формы жизни, вращающейся вокруг идентичности, которую лицо, точнее,
визаж, воплощает и до сих пор не решает. То, что мы называем «естественным
лицом», не является, как можно подумать, предпосылкой для возникновения
«культурного лица» (включающего модификации от макияжа и масок до хирургии),
а скорее служит одной из возможных семиотических масок, которые может выбрать
человек. В то же время маска не должна рассматриваться только как способ обмана
(или как материальный объект), в сущности находясь в центре более сложной
аксиологии. Классический семиотический квадрат, противопоставляющий «быть»
Cultures of the (masked) face
337
и «казаться», может стать основой для семиотического квадрата лица, включающего
собственно Лицо, Маскировку (место маски), Подделку и Анонимность. Основываясь на этих теоретических предпосылках, статья обращается к популярной музыке
и предлагает примерную карту прагматики маски (вычитание vs прибавление,
ритуальное vs непрерывное, материальное vs виртуальное, маска как лицо vs лицо
как маска) в качестве основы для дальнейших исследований.
(Maskiga) näo kultuurid
Seda, mida üldjuhul peetakse ‘näoks’, ei tohiks semiootiliselt mõista millegi antu ega
monoliitsena, vaid pigem kihistununa – see on vähemalt kolmekordne: bioloogiline (nägu),
füsiognoomiline (ilme), tajutav (pale) – ning suhtumuslikuna, sest sellele mõtte andmiseks
tuleb see paigutada narratiivi. Nägu asub terve semiootilise süsteemi, elu vormi, keskel,
keereldes identiteediküsimuse ümber (mida nägu – või täpsemalt, pale – küll kehastab,
ent millele siiski lahendust ei anna). See, mida me võime nimetada ‘looduslikuks näoks’
ei ole ‘kultuurilise näo’ (mis esineb modifikatsioonidega, mis ulatuvad jumestusest päris
maskide ja kirurgimaskini) eeltingimus, nagu argimõistus võiks välja pakkuda, vaid
lihtsalt üks võimalikest semiootilistest maskidest, mida inimene võib otsustada kanda.
Samal ajal ei pea maski taandama üksnes petmisvahendiks (ega pidama üksnes materiaalse
objektina mõelduks), sest tegelikult asub see keerukama aksioloogia keskmes. Klassikaline
semiootiline ruut, mis artikuleerib olemuse ja nähtumuse vahelist vastandust, võib sobida
“pale-oleku” semiootilise ruudu mudeliks, nii et selles oleks esindatud Nägu, Maskeering
(päris maski koht), Võltsing, Anonüümsus. Lähtudes neist teoreetilistest eeldustest
pöördutakse artiklis viimaks levimuusika poole ja visandatakse maski pragmaatika
provisoorne kaart (lahutav vs liitev, ritualistlik vs kestev, materiaalne vs virtuaalne, mask
kui nägu vs nägu kui mask), mis võiks olla aluseks edasistele uuringutele.
338
Remo Gramigna, Cristina Voto
Sign Systems Studies 49(3/4), 2021, 338–360
Notes on the semiotics of face recognition
Remo Gramigna,1 Cristina Voto2
Remo Gramigna, Cristina Voto
Abstract. Perceiving and recognizing others via their faces is of pivotal importance.
The ability to perceive others in the environment – to discern between friends
and foes, selves and others – as well as to detect and seek to predict their possible
moves, plans, and intentions, is a set of skills that has proved to be essential in the
evolutionary history of humankind. The aim of this study is to explore the subject of
face recognition as a semiotic phenomenon. The scope of this inquiry is limited to
face perception by the human species. The human face is analysed on the threshold
between biological processes and cultural processes. We argue that the recognition
of likenesses has a socio-cultural dimension that should not be overlooked. By
drawing on Georg Lichtenberg’s remarks on physiognomy, we discuss the critique
of the semiotic bias, the association of ideas, and the mechanism of typification
involved in face recognition. Face typification is discussed against the background of
face recognition and face identification. We take them as three gradients of meaning
that map out a network of relationships concerning different cognitive operations
that are at stake when dealing with the recognition of faces.
Keywords: face recognition; personal identification; semiotics of the face; semiotics
of recognition; face perception
1. Theoretical premises: The perception of human faces3
The fascination with the study of the human face shows its constancy in the history
of humanity. Although the face is a visible element of the anatomy and appearance
of the human species, it remains a quite enigmatic object and very difficult to
fathom. From ancient treatises on physiognomy or physiognomics to the ubiquity
1
Department of Philosophy and Education Sciences, University of Turin, Torino, Italy;
e-mail: remo.gramigna@unito.it.
2
Department of Philosophy and Education Sciences, University of Turin, Torino, Italy/
Department of Arts and Culture, Universidad Nacional de Tres de Febrero, Argentina; e-mail:
cristina.voto@unito.it.
3
Sections 1, 2 and 3 were written by Remo Gramigna. Section 4 was written by Cristina
Voto, except for the first and last paragraphs. Sections 5 and 6 were written in tandem by the
authors.
https://doi.org/10.12697/SSS.2021.49.3-4.05
Notes on the semiotics of face recognition
339
of automated face detection (Kosinski 2021) much in vogue in today’s “culture of
surveillance” (Gates 2011), the face remains a subject of profound interest that cuts
across numerous disciplines, from evolutionary biology to artificial intelligence.4
Our major task in this introductory section is to outline those theoretical
premises engendered in us by reviewing the recent literature on the perception
and recognition of faces. Because the face is often the object of interpretations,
semiotics – thought of as the discipline geared upon the interpretation of signs –
is one amongst the humanistic disciplines called on to discuss the significance of
human faces in all their forms and representations.
Hence, the thrust of this study lies in the semiotics of face recognition. As face
recognition is a broad area of inquiry, it is important to establish from the outset
the limits of the present research. Firstly, this study is limited to face perception
by the human species (for reviews on face perception by nonhuman species, see
Leopold and Rhodes 2010). Secondly, in what follows, we do not engage with the
concept of recognition as a philosophical issue, as did, for instance, Paul Ricoeur
(2005), who pointed out twenty-four different ways of defining recognition.
Therefore, concepts such as recognition as ‘acknowledgment’, ‘mutual recognition’
or ‘self-recognition’ fall outside the limits of the present inquiry. This study is
concerned with face recognition as a phenomenon of perception with a particular
interest in the semiotic aspects of this phenomenon. The purpose of the study is to
map out a network of neighbouring concepts that pinpoint the ramifications of face
recognition for semiotics. The paper revolves around three ways of interpreting
the human face: recognition, typification and identification. These three aspects
set out the three main sections of the study. In what follows we will map out some
ramifications of these phenomena.
One of the linchpins of face perception is identity. Indeed, the face is the visible
heart of the individual’s identity. Jurgen Ruesch (1959: 171) defines the face as the
signifier par excellence, Ray Birdwhistell (1970) as a “message board”, Max Thorek
(1946: 1) as an “advertisement to the world”, and Jordan Peterson highlights the
face as a “broadcast screen”.5 Indeed, information broadcast through the face is
multilayered. A whole host of different stimuli are elicited through the face and
perceived by others, who constantly monitor faces in others to decode facial stimuli
in terms of cues of various nature and as having different meanings (Ekman 1978).
4
On physiognomy and semiotics, see Eco 1975a, Manetti 1993: 84–87, and Magli 1995.
For an account on the face from a semiotic perspective, see Fabbri 1995 and Bouissac 2005
(available as Paul Bouissac What is a trustworthy face? at https://semioticon.com/virtuals/risk/
Trustworthyface.pdf).
5
Online lecture: https://www.youtube.com/watch?v=AVPDAa7hymo was accessed on 2
March, 2021.
340
Remo Gramigna, Cristina Voto
Identity, gender, age, skin pigmentation, health, basic emotions, micro-expressions,
intentions and much more information is displayed and expressed – either willingly
or unwillingly, inferred and decoded – accurately or less accurately, by those who
engage in social settings.6 Darwin (1872) held that facial expressions in humans
have something in common with other animals. Later thinkers sought to identify
a possible grammar of emotions through the study of facial expressions (Le Brun
1992; Damish 1992).
Since antiquity, “medical semiotics” (Sebeok 1972) as well as physiognomy
identified the human face as a significant subject of interest as witnessed by the
long-lasting tradition of Arabic and occidental treatises on physiognomy.7 Indeed,
there is plenty of historical evidence showing that signs of the face, so to speak,
were treated and interpreted as symptoms of certain diseases, as epitomized in
Hippocrates. Ancient physiognomy went even further in claiming that there exists
an ability to identify the soul and the heart of an individual based on the shape and
the nature of the human body and, especially, the face. This idea is later taken up
and epitomized in the theory detailed in Johann Kaspar Lavater’s physiognomy.
The physiognomic approach, thus, is based on deductive reasoning for it posits
that the internal characteristics of an individual can be deduced from the exterior
attributes. As Giovanni Gurisatti (2006: 22) rightly pointed out,
[…] between external and internal, sign and what it stands for, there is a staticlocal causal relationship (dual, successive), where the symptom, the spasm, the
colour, are nothing but the external effect, the externalization of an internal cause,
to which they refer according to a stable and rigorously determined hierarchical
scheme.8
For this reason, the basis of medical semiotics and physiognomy in interpreting
faces and bodily symptoms is a tendency towards “typifying” (Gurisatti 2006: 22).
In other words, this branch of medicine sought to identify causes from symptoms
by means of a scientific or pseudo-scientific method. As we will see in what
6
The term ‘expression’ has two semantic components: one subjective and intentional and
another objective and non-intentional. On this distinction, see Gurisatti 2006: 26–27.
7
On the tradition of Arabic physiognomy, see Mourad 1939; on Mesopotamian physiognomy,
see Bottéro 1974. For an overview of the history of physiognomics, see Antonini 1900; Getrevi
1991; Gurisatti 1991, 2006; Magli 1995; Courtine and Horoche 1992 – they all offer valuable
bibliographical sources on the subject.
8
“[...] fra esterno e interno, segno e designato corre qui un rapporto statico-locale causale
(duale, successivo), dove il sintomo, lo spasmo, il colore altro non sono che l’effetto esteriore,
l’esteriorizzazione di una causa interiore, cui essi rinviano secondo uno schema gerarchico stabile
e rigorosamente determinato”. All translations from Italian are by Remo Gramigna, unless
otherwise noted.
Notes on the semiotics of face recognition
341
follows, not only was such a tendency towards typification at the basis of medical
semiotics – identifying symptoms on the basis of a type – and some branches of
physiognomy, but it also resurfaces today under various guises.
Physiognomy never achieved a scientific status and today it has lost its grip.
As Carlo Ginzburg (1986: 170) pointed out, “the flaw of physiognomy was to
tackle the variety of individuals in the light of preconceived opinions and hasty
conjectures: in this way it has so far been impossible to establish a scientific,
descriptive physiognomy”.9 It is, however, important to recognize that ancient
practices of deciphering and reading faces played a pivotal role in what Ginzburg
(1986) termed as the ‘evidential paradigm’ (‘il paradigma indiziario’) and have a
connection with the history of semiotics. In this regard, it suffices to mention the
inclusion of scholars such as Lavater on physiognomics and Darwin’s study on the
face in the history of semiotics (Eco 1984).
Whilst it is visible to others, unless it is masked or disguised, the face is invisible
to the subject. Indeed, the face has a twofold character: the face as seer and the face
as seen. Tim Ingold (2002: 124) writes,
As a surface, the face has some very peculiar properties. I can feel my own face, and
others can see it. But it remains invisible to me. Where others see my face, I see the
world. Thus, the face is a visible appearance, in others’ eyes, of my own subjective
presence as an agent of perception. It is, if you will, the look of human being.
Thus, the human face is the element of appearance which makes the social bond
possible through the responsibility that each individual takes in respect to the
outer world, which is the principle of personal identity (Le Breton 2010: 72).
Given the centrality that the face takes on in social settings (Goffman 1956, 1967;
Edkins 2015) and in the definition of one’s own identity (Belting 2017), it is not
surprising to find a tight connection between the perception of the human face,
the recognition of faces by others, and the pivotal role that such processes play in
human interactions and the perception of the environment.
It is worth noting that the twofold dimension intrinsic to the concept of the
face is apparent in the etymology of the word,10 where the duplet seeing/being seen
permeates the semantic field of the term ‘face’ itself. In the Latin tradition, ‘vultus’
9
“L’ errore della fisiognomica è stato quello di affrontare la varietà degli individui alla luce di
opinioni preconcette e di congetture affrettate: in questo modo è stato finora impossibile fondare
una fisiognomica scientifica, descrittiva.”
10 The idea of perceiving and seeing one’s own face is one aspect of the issue and leads to
the investigation of the role of mirror and self-reflection in the perception of oneself. Classic
studies that deal with this issue from the perspective of semiotics are Eco 2010, Nöth 1990 and
Galassi and De Michiel 1997. Also, Baltrušaitis 1981 is very relevant to this debate.
342
Remo Gramigna, Cristina Voto
was used to refer to the variable and changing aspect of the face, whilst the word
‘facies’ referred to the fixed and immutable physiognomic traits. It is revealing that
Isidore of Seville pointed out that the face (facies) is said to be so only because it
makes one person recognizable from another and, therefore, allows the identification
of a particular individual amongst the crowd (Piras 2010: 48). Indeed, slaves in
Ancient Greece were faceless as the word ‘aproposon’ suggests11 (Agamben 2021).
In one etymological explanation of the Latin ‘vultus’, the term is traced back to
the root ‘uel-, vedere’. Hence, ‘vultus’ refers both to the ability of the individual to
see and to be seen (Piras 2010: 53). This aspect also surfaces in the German word
‘Gesicht’ which encapsulates both meanings. Husserl’s phenomenology states this
principle very neatly: one element of the face boils down to its visibility, namely,
the element of being seen and being perceived from outside perspectives (Husserl
1965: 432). As Agamben (2021) rightly pointed out:
Of course, all living beings show themselves and communicate with each other,
but only man makes the face the place of his recognition and his truth, man is
the animal that recognizes his own face in the mirror and mirrors and recognizes
himself in the face of the other. In this sense, the face is both similitas, similitude
and simultas, the being together of men. A faceless man is necessarily alone. This
is why the face is the place of politics.12
Given the relevance of the element of appearance – showing and being perceived by
others – it goes without saying that the opposite tendency – hiding, dissimulating,
masking – must be considered in the management of one’s appearance. Indeed,
the reverse mechanism of recognition is masking, for it hampers the possibility
of recognizing another by altering the distinctive facial traits of the person who
wears a mask (Ogibenin 1975). Here we come full circle through the opposite
mechanism that serves as a cornerstone of the management of appearance: the
simulation and dissimulation of one’s face.13
11
Agamben, Giorgio 2021. Il volto e la morte. Quodlibet, 3 May, 2021 (https://www.quodlibet.
it/una-voce-giorgio-agamben). First published in German as “Wo das Gesicht verschwindet”,
Neue Zürcher Zeitung, 29 April 2021; now also in Agamben 2021.
12 Original in Italian: “Certo tutti gli esseri viventi si mostrano e comunicano gli uni agli altri,
ma solo l’uomo fa del volto il luogo del suo riconoscimento e della sua verità, l’uomo è l’animale che
riconosce il suo volto allo specchio e si specchia e riconosce nel volto dell’altro. Il volto è, in questo
senso, tanto la similitas, la somiglianza che la simultas, l’essere insieme degli uomini. Un uomo
senza volto è necessariamente solo. Per questo il volto è il luogo della politica.” Agamben 2021 was
accessed at https://www.quodlibet.it/giorgio-agamben-il-volto-e-la-morte. on 4 April 2021.
13 The dichotomy of showing one’s face and concealing it is epitomized in the techniques of
masking. On this issue, see Gramigna 2021, Leone 2020.
Notes on the semiotics of face recognition
343
2. Critique of the semiotic bias and the association of
ideas in face recognition
The words ‘recognition’ and ‘to recognize’ derive from the Latin ‘recognoscere’.
This term is composed of the prefix ‘re-’, which means ‘again’, and ‘cognoscere’,
which stands for ‘knowing’. Etymology suggests that to recognize someone or
something is an action that entails that the object recognized (be it a human
being or a thing) was seen or known by the subject in an earlier phase prior to
the time when recognition occurs. In a nutshell, we recognize something known.
As Ellis (1975: 409) writes, “recognition memory involves the identification of
some previously experienced configuration or event”. One thing emerges from
this cursory description of terminology: recognition is bound up with the world
of perception, and particularly with the field of vision. This also suggests the idea
of a recursive element embedded in the concept of recognizing, which is, literally,
knowing again, knowing for the second time. Firstly, a question that emerges
immediately is whether there is a relation between memory and recognition and,
if so, how can such a relation be qualified. Secondly, another issue concerns the
uniqueness of the human face and whether it is processed by a special system as
compared to the processing of objects that are not faces (Ellis 1986: 5).
Because the field of vision varies according to the distance in human
interactions, the perception of the human face in social settings is regulated by the
laws of proxemics. It is worth noting that in Edward T. Hall’s typology of distances
in humans, the face is perceived by others only at close distances. Outside the
range of close distances, indeed, faces become blurred and unrecognizable. For Hall
(1959: 115), “intimate distance” (less than 18 inches) – the distance of “love-making
and wrestling” – is the distance where “sight (often distorted), olfaction, heat from
another person’s body, sound, smell, and the feel of the breath all combine to signal
unmistakable involvement with another body” (Hall 1959: 116) and “personal
distance” (from one to four feet) is the zone where people can touch each other
and the field of vision is very sharp. At the close phase of personal distance, the face
of the other indeed becomes very visible: “the planes and roundness of the face are
accentuated; the nose projects and the ears recede; the hair of the face, eyelashes,
and pores are very visible” (Hall 1959: 119). Whilst the idea of face visibility is
present in Hall’s classic account, this aspect was stressed with particular force in
Krampen’s revisitation of proxemics (Krampen 1995: 283–284).
Before venturing further, it is worth pointing out that some key ideas on face
recognition, the cognitive bias, and the prejudices that lurk beneath it, can be
gleaned from the debate between Johann Caspar Lavater and his fierce critic, the
Göttingen physicist Georg Christoph Lichtenberg. In On Physiognomics: Against
344
Remo Gramigna, Cristina Voto
Physiognomists, published in 1777 (second ed. 1778), Lichtenberg delivers a
series of sharp and penetrating points of criticism directed at the paradigm of
physiognomy as advocated by Lavater.14 As Lichtenberg (1991: 107) himself
pointed out, the true intention of his work is an admonition to treat such a theme
with prudence because judging a person by the face may lead to gross mistakes and
overgeneralizations. This is what today goes under the rubric of ‘first impression’
or ‘face value’ (Todorov 2017).
As Lichtenberg remarks:
[…] it is perfectly coherent with the laws of our thought and our sensibility that,
as soon as we look at a man, suddenly a figure absolutely more resemblant than
we have ever met comes to our mind. Usually, such a figure directly determines
our judgment. We regularly make judgments on the base of the face, and we are
regularly mistaken. (Lichtenberg 1991: 126)15
Lichtenberg’s criticisms on physiognomy are numerous and acute and cannot
be dismissed in a few words. One such item of criticism concerns the principle
of the “association of ideas” that lays at the basis of the mechanism of matching
faces to psychological traits based on a set of prior templates and assumptions
the individual already has in stock.16 Lichtenberg argues that, in recognizing
others, people project onto the face some pre-existent cognitive schemes that play
a pivotal role in the perception and recognition of faces. As Lichtenberg writes,
“The superficial physiognomist finds a face in every drop of ink and a meaning in
every face” (Lichtenberg 1991: 127).17
The author explains how face recognition works in the following terms. Contra
the tenets of physiognomy, Lichtenberg argues that the face is unfathomable in its
entirety. For this reason and for the purpose of grasping a face, each of us creates
an “extract” of it, a face’s template. This extract is obtained on the basis of the
14
It would be very limited and inaccurate to say that Lichtenberg’s polemical notes are levelled
against Lavater. It is much more than that. Lichtenberg’s critical remarks, instead, concerned
the fashionable trend of physiognomics, superficial and without depth, that was rampant at the
time when he lived. So, he was against the trivialization and popularization of physiognomy.
15 “È perfettamente coerente con le leggi del nostro pensiero e della nostra sensibilità che, non
appena posiamo lo sguardo su un uomo, ci venga subito in mente la figura a esso in assoluto più
somigliante che abbiamo già incontrato. Questa figura, di solito, determina direttamente il nostro
giudizio. Regolarmente giudichiamo in base al volto, e regolarmente ci sbagliamo”.
16 For a discussion of the criticisms that Lichtenberg launches at Lavater and to the “epidemics”
of physiognomy, see Gurisatti 2006: 91–111, 1991: 24–40.
17 “[...] il fisiognomo superficiale in ogni macchia d’inchiostro trova un volto e in ogni volto un
significato”.
Notes on the semiotics of face recognition
345
interests, mood, and knowledge of the beholder, who assigns to it a host of different
meanings (Lichtenberg 1991: 129). According to Lichtenberg, such a template is
systematized so that each observer identifies a face composed of four dots, as
depicted in the following image (Fig. 1):
Figure 1. Face template according to Lichtenberg (1991: 129).
Gurisatti’s comments on Lichtenberg’s principle of the association of ideas are
illuminating:
Physiognomy would lead to projecting onto the face contingent and predetermined
cognitive-perceptive schemes, subjecting the face to a subjective (perhaps
unconscious) strategy of codification and recognition. In short, the face would be
neither explained nor understood – that is, known – but only re-cognized, that is
to say made adequate and empathically homologated to cognitive habits, visual
conventions, prejudices of the viewer. (Gurisatti 2006: 24)18
Such a thesis is also in line with the etymology of the term ‘recognoscere’, recalled
earlier, which means recognizing something already known. The mechanism
Lichtenberg discusses is likely to describe how certain cultural biases operate in
guiding the recognition of others’ faces. At this juncture, it is worth recalling Eco,
who in Kant and the Platypus elaborated a quite sophisticated theory of recognizing
unknown entities – Eco’s case studies being Montezuma and the Aztecs gradually
forming the concept of a horse after the first meeting with the Spaniards, as well
as the polemic over the zoological classification of the platypus – by matching a
pre-existing stock of knowledge in order to grasp and categorize something new or
unknown (Eco 2000[1997]). Eco’s account of ‘cognitive type’ and ‘nuclear content’
18
“La fisiognomica indurrebbe a proiettare sul volto schemi cognitivi-percettivi contingenti e
predeterminati, sottoponendolo unilateralmente a una strategia soggettiva (magari inconsapevole)
di codificazione e riconoscimento. Il volto, insomma, non sarebbe né spiegato né compreso – ossia
conosciuto – bensì soltanto ri-conosciuto, vale a dire adeguato e omologato empaticamente alle
abitudini cognitive, alle convenzioni visive, ai pregiudizi di colui che guarda” (Gurisatti 2006:
24).
346
Remo Gramigna, Cristina Voto
seems in alignment with the point raised above, although Eco’s purpose is not to
make sense of the recognition of the face, but the perception and categorization of
a novel and unknown entity.
3. Attending to face-like templates:
An inborn disposition or a learned habit?
One of the first patterns we perceive as human beings when we are born is the face
of another. The ability to perceive others in the environment – to discern between
friends and foes, selves and others – as well as to detect and seek to predict their
possible moves, plans, and intentions, is a set of skills that proved to be essential
in the evolutionary history of humankind (Rhodes et al. 2011). Whether this
is an inborn attitude or a learned skill is an open question that has generated
much debate. Yet the perception and recognition of others through the face is of
paramount importance. There is evidence that the ability to detect the face is a skill
that appears at a very early stage after birth (Buiatti et al. 2019).
Traditionally, biologists, cognitive scientists, developmental psychologists, and
researchers of artificial intelligence have had the most to say on the subject (Bruce
1988; Bruce, Young 2012; Buiatti et al. 2019; Leopold, Rhodes 2010; Reid et al.
2017). As Ernst Gombrich (1956: 103) writes,
[…] we respond with particular readiness to certain configurations of biological
significance for our survival. The recognition of the human face, on this argument,
is not wholly learned. It is based on some kind of inborn disposition. Whenever
anything remotely face-like enters our field of vision, we are alerted and respond.
Although the debate around face perception does not fit squarely in a rigid
schematism, for clarity’s sake we can single out two opposite fronts: an innatist
and a constructivist one. Whilst the former perspective conceives face perception
as an innate disposition, constructivists argue that this is an ability that is learned
(for reviews, see Guthrie 1995: 103–121). This issue is still unresolved and there is
no general agreement around such matters. Let us be clear that there is no complete
agreement among experts in this field of research. Several authors have ventured
into the study of the subject reaching inconsistent and divergent conclusions, with
different authors offering different opinions. The difference in judgments on the
interpretation of this concept is due to the non-homogeneity of the criteria used
by the scholars for its elucidation on each occasion.
Notes on the semiotics of face recognition
347
4. Face as a type: Cultural filters of an evolutionary trait?
Semiotic research that dealt with recognition as a cognitive problem (Eco 1968,
1973, 1997; Volli 1972) suggested that the process of recognition – especially the
recognition of ‘iconic signs’ – implies the prior establishment of some cultural
norms. Both Eco and Volli note that the phenomenon of recognition is not
limited to the cognitive and perceptual dimension and suggest that there is an
element of culture to it, which ought not to be overlooked. This could be termed
as the culturalist approach to recognition. As Eco (1980[1973]: 55) pointed out,
the recognition of iconic signs implies a handful of operations which can be
summarized as follows:
(1) a given culture must define recognizable objects on the basis of some emergent
features or traits of recognition;
(2) there must be a convention which establishes that certain graphic traits
correspond to some of the traits of recognition and that some of these
recognition traits of the object must always be present in the reproduction in
order to recognize the object;
(3) another convention is necessary to establish the modalities of production of
the correspondence between graphic traits and traits of recognition.
If we conceive of face recognition in terms of iconic signs, we could perhaps
speculate that cultures provide given traits of recognition that would make up the
stock of knowledge activated when recognition occurs.
Édouard Manet’s painting Music in the Tuileries (1862) shows the unique
experience of identifying face-like patterns in an image that depicts unrecognizable
and featureless faces in the crowd (Fig. 2):
348
Remo Gramigna, Cristina Voto
Figure 2. Édouard Manet. Music in the Tuileries (oil on canvas, 1862).
On the canvas, the plastic density of the painting and the gesture of the painter
allow the beholder to experience a recognition based on spatial relationships.
If too close or too far away from the work of art, the crowd will be blurred or
even shapeless, but at the right distance, the viewer begins to recognize a series
of visual patterns that result in the socio-historical characteristics of the face of
the time. Likewise, Edgar Allan Poe, in his short story “The man of the crowd”
(1840), describes the recognition of a human face from the absolute vagueness of
anonymity of the crowd:
With my brow to the glass, I was thus occupied in scrutinizing the mob, when
suddenly there came into view a countenance (that of a decrepit old man, some
sixty-five or seventy years of age,) a countenance which at once arrested and
absorbed my whole attention, on account of the absolute idiosyncrasy of its
expression. Anything even remotely resembling that expression I had never seen
before. I well remember that my first thought, upon beholding it, was that Retzch,
had he viewed it, would have greatly preferred it to his own pictural incarnations
of the fiend. […] It was something even more intense than despair that I then
observed upon the countenance of the singular being whom I had watched so
pertinaciously. (Poe 1996[1840]: 392)
The example extracted from Poe’s literary sensibility shows very poignantly how
human faces can be categorized as types. Typification is the possibility of reading
Notes on the semiotics of face recognition
349
certain kinds of information from someone’s face based on established and
available sociocultural codes. As the Italian semiotician and expert in the field of
face studies, Patrizia Magli (1995: 17) has pointed out,
The perception of a face is primarily based on its recognition. To recognize a face
means to assign it to a class; and, in this respect, the perceptive experience seems
already directed by the cultural filter: the filter of culture lies in between the face
that is in front of us and our irresistible vocation to interpret.19
We do endorse Magli’s view. Recognition and typification, then, are two phenomena that are, in some respect, interwoven. The constructions of types and
their recognition by the observer are two processes that shed light on the sociocultural element in which both the phenomena of perception and recognition are
enmeshed. From this perspective, typification is the process by means of which one
selects certain aspects from a given token, and due to such selection, one is able
to attribute the particulars of a face to the general category of humans as a group
through an abstract interpretation. Indeed, typification is a classifying scheme
that arises from the concrete practice of ordinary life, made up of interactions
and problems to be solved in order to recognize someone. Such typifying schemes
produce a structure on the basis of common knowledge that is not homogenous
but contextually, and therefore culturally, distributed.
Originally, the distinction between type and token within semiotics comes from
Charles Sanders Peirce, who introduced it in the essay “Prolegomena to an apology
for pragmaticism” (1906). The type–token distinction is the cognitive process that
leads to the construction of a model where the type is the class of which all signs
are exemplars, and the tokens of a type are the occurrences, namely, the singular
replicas that can be categorized in a given class. A type, as for instance a particular
anatomical eye shape that can be read and interpreted as belonging to a specific
ethnic group, must be actualized in a token, that is, in a singular face in order to
be recognized. This face will be a sign of a type and, therefore, of the group to
which the type refers. For instance, because of the contextual ideologies that have
raged in the construction of nation-states, we are led to recognize the existence of a
specific type of faciality according to cultural logic, such as there being no token of
Italianity with specific traits other than Caucasian ones. However, the occurrences
that crowd the globalized faciality together with the memories and texts of the past
19
“La percezione di un volto si basa innanzitutto sul suo riconoscimento. Riconoscere un volto
significa assegnarlo a una classe; e, in questo senso, l’esperienza percettiva sembra già orientata
dal filtro culturale: tra il volto che ci sta di fronte e la nostra irresistibile vocazione a interpretare
[…] si interpone il filtro della cultura.”
350
Remo Gramigna, Cristina Voto
make manifest the existence of a whole series of facial occurrences that expand the
pertinence of the type.
This process can be described as the extraction of types that are interpreted as
forming a certain individuality depending on a social token, a strategy that enables
the categorization of experience. Nevertheless, typification, as an interpretative
activity, can be untied from the idea of an operation based on the established
codification and selection of characteristics of a type projected on tokens, and
leading it back to a continuous activity of token construction starting from
the possibilities of the practical horizon in which they take root based on the
pertinence. However, the daily socio-cultural experience can be compared with
the facility of typecasting someone on the basis of social facial types that circulate
in the cultural iconosphere. As Jessica Helfand (2019: 187) wrote: “To typecast is
to color inside the lines, to proceed with expectations [...]. Habit is an unforgiving
teacher [...]. What you see and what you respond to is the simulacrum, not the
soul; the posture, not the person”. Even today, with mobile applications and filters
that lead us to manipulate our facial representations, we are primed to see faces
as a territory conquered by the hegemonic-type beauty parameters that conduct
stereotyping and bias: ‘whitewashing’, ‘forever young’, etc. We find two poles: the
individual and his/her behaviour on the one side, the society and the typification
of the behaviour of individuals on the other side. Under the influence of complex
historical and social processes, specific forms of behaviour are formed, typifying
what is accepted or rejected, what is lawful or forbidden, what is valuable or devoid
of value. As indicated by Juri Lotman (2006), a complex semiotic norm of ethical,
religious, aesthetic, practical and other kinds enters the cultural construction,
against the background of which the identity of human behaviour is constituted.
In this sense, Lotman’s semiotic model of culture can be read considering
the category of enunciative practice. If the behavioural text is the result of the
sedimentation of a series of individual acts, which make up a collective enunciative
practice, the variations with which it is reproduced by the individual constitute
an individual enunciative practice. The latter can strengthen the behavioural text
through the repetition of the same sequence of behaviours, but it can also modify
it by constituting a variation and modification of the given cultural patterns.
As aptly discussed in Giovanni Gurisatti’s inquiry into the aesthetics of the
face throughout history, physiognomy has generally been following two rather
opposing tendencies. The first tendency strives towards typification. Typification
is predicated upon the idea that the varieties of human appearances can be
classified in terms of types. The second perspective conceives of the human face
in terms of its uniqueness and individuality (Gurisatti 2006: 248). Likewise, Rudolf
Kassner (1997: 40) differentiates between two different perspectives: the first being
Notes on the semiotics of face recognition
351
“constructivist”, for it conceives of faces in terms of eyes, noses, mouths, and ears,
and it begins with a concept to reach a type. This perspective highlights the form of
the face as well as the fixed parts of the human face (Kassner 1997: 58). The second
is “differential”, for it focuses on the uniqueness of the expression of the face and
is predicated upon individuality and the dynamic aspect of the face. This point
is worth pondering. This thesis posits that the physiognomic literature dealing
with the human face from Aristotle to Lavater, has dealt with the subject in terms
of matching facial features to a given type through a process of codification and
typification. In other words, the physiognomic approach, at least up to Lavater,
tends to reduce the authenticity and the uniqueness of a face to a given type. The
rupture of this paradigm occurs with the “new physiognomic” proposed by Kassner
(1997). It is only within a constructivist approach to faces that the hypothesis of
interpreting faces as types can come forth. As Kassner sharply writes, “Thus, types
live, are real and vital in a rigid, predetermined, already codified world, in a certain
sense four-dimensional. Or vice versa: in this rigid, predetermined, finite, eternally
repeating world, there may be only types, not individuals” (Kassner 1997: 59).20
5. Face and identity: The role of facial features
in personal identification
In the previous sections, we have dealt with face recognition and typification, and
mentioned face and identity only en passant. We have hinted at the role of face
identification, and this issue deserves further qualification. Personal identification
is a much larger phenomenon in comparison to identification based solely on
facial features. Face identification is not equal to personal identification tout
court, although the face plays a pivotal role here. Indeed, as forensic anthropology,
forensic medicine and forensic genetics have pointed out, the systems of personal
identification are varied and not all the methods of personal identification are
based on the identification of the face in order to establish the identity of the
subject. However, it is undeniable that there are methods of personal identification
that are based on the physiognomic study of the face, which is not surprising, given
that the human face is a marker of personal identity.
Nevertheless, the identification of a person can occur not only through the
physiognomy of an individual but also by means of other identification methods
such as fingerprints, the chromatic map of the iris and the DNA. For this reason,
20
“Dunque i tipi vivono, sono reali e vitali, in un mondo rigido, predeterminato, già codificato,
in un certo qual modo quadrimensionale. O viceversa: in questo mondo rigido, predeterminato,
finito, eternamente ripetentesi, potrebbero esserci solo tipi, non individui.”
352
Remo Gramigna, Cristina Voto
it must be specified that personal identification based on the face is only one of
the methods of personal identification that embraces a much wider range of types
of recognition, which also include automatic techniques for recognizing people.
It is, therefore, a very vast and complex area. In this study, we will not deal with
biometric recognition. The latter represents a type of automatic recognition from
the first intuitions of Alphonse Bertillon, the head of the identification section of
the Paris police, who, in the mid-19th century, had guessed that the measure of
certain parts of the body – such as the length of the feet or fingers – could be used
to identify criminals. To be sure, Bertillon identified eleven parameters. From
the 19th century onwards, images became a standard and in 1882, he established
the use of the signalment, a photographic system giving images (‘mugshots’) and
detailed descriptions of persons being tracked. These signalments all had the same
format, a card with a series of standard shots correlated by an equally standardized
description. It was the invention of a taxonomic system based on biometric data
that gave the face a statistical character.
Later, Francis Galton and E. R. Henry’s discovery of fingerprints as an element
of individual identification inaugurated the method of the archive and archiving as
a way of collecting, storing, and cataloguing data relating to personal identification.
In 1893 this method was replaced by fingerprinting and the statistical path was
established: identity was ascertained on skin patterns, a body trace that increasingly
weakened the power of the representation of the face. That is, faith in the expressivity and truthfulness of the face had diminished, and from the fingerprint we
moved on to the recognition of the iris, which marked the passage to the face as a
deposit of abstract and numerical information, which is the end of the most recent
applications for facial recognition. Since Bertillon, the history of biometrics has
made enormous advancements and biometric recognition has become one of the
most advanced types of recognition, which is not without risks for the privacy of
people, however (Gates 2011).
Let us leave aside biometric identification and let us discuss identification
based on facial features. With regard to personal identification based on the
physiognomic study of the face, there is an important corollary to consider.
First, there are two methods that are based on physiognomic traits: (1) indirect
comparison method, and (2) direct comparison method.21 The former technique
consists of a comparative method which is based on the comparison of images of
an individual’s face that allows determining the identity between two subjects. By
comparing multiple images, an “identity judgment” (Viciano, Capasso 2021: 181) is
21
Cf. Luigi Capasso, Identificazione personale fisionomica attraverso immagini, University
of Chieti; available at https://docplayer.it/12549821-Identificazione-personale-fisionomicaattraverso-immagini.html. Also relevant is Falco 1923.
Notes on the semiotics of face recognition
353
formulated. The direct comparison method, as the previous one, is a comparative
method that assesses the image of an unidentified subject against the image of a
subject whose identity is known.
What needs to be stressed is that in both cases, the identification of identity
is based on an analysis and study of photographic material of various sorts.
Often, there is no possibility of direct observation of people’s faces or anatomical
structures. For this reason, personal identification based on the physiognomic
study of the face works especially through images of faces – that is, representations
of faces – which become the privileged material for the investigation. These images
can be photographs or frames extrapolated from video recordings. This point is
worth pondering, for it is by no means a minor detail.
From a methodological point of view, as well as from the perspective of
epistemology and semiotics, this is a remarkable point because the personal
identification process is often based on photographic representations and, therefore, on images of faces. It is our contention that this difference has not been
sufficiently stressed. It is one thing to assess the identity of an individual on
the basis of the recognition of the face in real-life settings and in face-to-face
interactions. The identification of an individual on the basis of photographic
evidence is a different type of business. In fact, in current practice, a photograph
of the subject is sufficient to identify a subject as the history of the ID card shows.
Therefore, we are dealing with photographic representation of the face and its
quality and accuracy, which are important factors in assessing personal identity.
Indeed, face-images have been used as tracking devices for identification. The
history of the identity card has a longer history than the face-image, at least in
France and Germany. Its origins date back to the late Middle Ages, and it contained
descriptions of the owner.
As Lotman (1998: 63–64) pointed out, from the point of view of culturology,
the function of proof of the authenticity of the subject was originally performed
by the portrait that anticipated the role of photography. From the standpoint of
semiotics, it is important to stress that it is not only the likeness of people’s faces to
their representations that matters – that is, matching the object to its model – be
it a portrait, a photograph, a video and so forth. On the contrary, what is pivotal is
that the recognition of likeness has a socio-cultural dimension which should not
be overlooked, because recognizing a face as similar to another occurs in a given
socio-cultural context. In this regard, Lotman provides examples of photographs
that do not resemble the people portrayed, thus presenting a low degree of likeness
between them. A caricature or a portrait could have a higher degree of likeness
than a “bad photograph”. Nonetheless, due to the cultural function assigned to
photography as a marker of authenticity, a photograph will always be preferred as a
354
Remo Gramigna, Cristina Voto
means of personal identification over a handwritten sketch or a caricature (Lotman
1998: 65). Lotman thus rightly remarks that it is the “formal ability to be a sign of
likeness that fulfills a given conventional function” (Lotman 1998: 65).
Identification differs from recognition in one important respect. Identification
is a byproduct of history and societal need of identifying specific individuals. There
were times in history when people were neither identified nor identifiable. We
would conceive of recognition as a quality that precedes identification biologically
and historically.
6. Machine face detection and the future of face recognition
One important and fast-growing area of research within face perception is the
research framework of face recognition by computers (Wechsler et al. 1998), which
we will touch on only tangentially. In this study, we are mainly concerned with
the way in which humans recognize faces, although machine face recognition is a
fast-growing research field that should not be overlooked.
Today, the importance of face recognition, including the plethora of the nuanced
meanings attached to this semantic field – identifying, classifying, filing, tracking,
to mention but a few – has seen a re-emergence due to the widespread use of digital
media, the innovations brought along by the increasing use of artificial intelligence
of face detection, and technologies of tracking and personal identification.
The establishment of advanced machine learning, artificial-intelligence-based
algorithms and network-based, automated services have altered the way in which
we thought of faces, provided some benefits and opened up new challenges. AI
poses some serious questions of ethics and perpetuates social bias (O’ Neil 2017).
What in the past was generally thought of as a skill mastered through biological
evolution and engrained in the genetic make-up of a species, today has become
the hallmark of intelligent systems capable of performing complex operations that
involve, to some extent, face recognition processes unheard of only a few years ago.
In a nutshell, the way we conceive of faces has changed dramatically and opened
up unprecedented scenarios.
The panorama we must face today is, therefore, complex and unprecedented.
Human beings’ lives have expanded and become digital (Turkle 1995; Windley
2005). The massive availability of big data and of public records – which range
from images to geo-localized information – has radically altered the way in which
the concept of face recognition has been traditionally conceived. Needless to
say, as bodies of flesh and bones leave fingerprints, also digital selves leave many
traces behind, which can be decoded, clustered, tracked and filtered for different
Notes on the semiotics of face recognition
355
purposes, including identification and recognition. Hence, new challenges lie
ahead for the researchers: to unpack and contextualize a phenomenon that has
taken on some very nuanced and unprecedented meanings.
From Woodrow Wilson Bledsoe’s first tinkering in the 1960s22 to the contemporary pervasiveness of facial detection systems in our daily life, both in the private
and the public sphere, passing through the interaction with mobile phones through
the border control services, all show the pervasiveness of this phenomenon. During
the last twenty years, a considerable number of scholars and artists have focused
on the opportunities and limits offered by artificial recognition technologies: the
research of Kate Crawford and Trevor Paglen (2021), Kelly Gates (2011), Shoshana
Amielle Magnet (2011), among others, together with the works of artists like Hito
Steyler, Leonardo Selvaggio and Zach Blas show the relevance of these phenomena.
This pervasiveness can be surely understood as symptomatic of the facial
society described by Thomas Macho and Gerburg Treusch-Dieter (1996)
while, at the same time, it makes visible some continuities and discontinuities
between artificial and human recognition systems and processes. The first
difference concerns the question of the accessibility of the recognition. On
the one hand, artificial vision recognizes human beings through an invisible
operation inaccessible to most humans. On the other hand, human recognition,
even though extremely complex both neurophysiologically and culturally, is a
reciprocal experience within the human species. Except among those affected by
prosopagnosia, as human animals, we recognize faces as being seen through our
face. In both human and artificial recognition, the result of the process is the
production of some kind of knowledge. However, the qualitative dimension of the
information generated by artificial processes is under assessment of legitimacy,
while in the case of human recognition it can be over-interpreted. An example in
this direction can be emotion recognition. We are witnessing an increasing demand
for emotion recognition in many fields of security such as government, public
and private agencies, biomedicine or therapy (Dutta, Barman 2020). Still, doubts
persist about the effectiveness of such a complex task, which is often flawed even
in human recognition. These points of contact and distance mark some continuity
and discontinuity in recognition processes. Many artificial recognition systems are
inspired by human processes. In this regard, a task for the future of face recognition
will be to ask to what extent artificial recognition replicates human cognitive
processes and how these eventual replications relate to a discourse on the visual.
22
In 1963 Woodrow Wilson Bledsoe published the research entitled A Proposal for a Study
to Determine the Feasibility of a Simplified Face Recognition Machine, one of the first studies
to investigate the relationship between artificial intelligence and facial image recognition on
behalf of the US Central Intelligence Agency.
356
Remo Gramigna, Cristina Voto
7. Conclusion
The present study has led towards a semiotics of recognition by unpacking the
meaning of the human face under three different, yet complementary, aspects. We
have limited our study to three main aspects, namely, recognition, identification
and typification of faces, our strategy being to discover a network of relationships of
meaning among these three aspects. The face as a recognizable pattern is engrained
in the process of semiotic mediation (recognition). Of the three, recognition is the
broadest phenomenon, for it includes the recognition of objects that are not limited
to the recognition of human faces. This is probably also the oldest form of conceiving
of faces that has a direct connection to the biology of face recognition. The face as
the focus of personal identity (identification) uses faces as identity markers and is a
scientific procedure invented by legal institutions to establish the genuine identity
of people. Such a procedure is liable to falsification and opens up to the vast area of
fake identities, simulation and fraud. The face thought of as a result of typification,
that is, the possibility of reading certain kinds of information from someone’s face
based on established and available sociocultural codes, is a semiotic practice that
dwells with the extraction of types on the basis of socio-cultural token (typification).
Acknowledgements: This reseach is part of a project that has received funding from the
European Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation programme (Grant agreement No 819649 – FACETS).
References
Agamben, Giorgio 2021. A che punto siamo? L’epidemia come politica. (Nuova edizione
accresciuta.) Padova: Quodlibet.
Antonini, Giuseppe J. 1900. I precursori di Lombroso. Torino: Fratelli Bocca Edittori.
Baltrušaitis, Jurgis 1981. Lo specchio. Rivelazioni, inganni e science-fiction. Milano: Adelphi.
Belting, Hans 2017. Face and Mask: A Double History. Oxford: Princeton University Press.
Birdwhistell, Ray L. 1970. Kinesics and Context: Essays on Body-Motion Communication.
London: Allen Lane – Penguin Press. https://doi.org/10.9783/9780812201284
Bottéro, Jean 1974. Symptomes, signes, écritures en Mésopotamie ancienne. In: Vernant,
Jean Pierre et al. Divination et rationalité. (Recherches anthropologiques.) Paris: Seuil,
70–193.
Buiatti, Marco; Di Giorgio, Elisa; Piazza, Manuela; Polloni, Carlo; Menna, Giuseppe;
Taddei, Fabbrizio; Baldo, Ermanno; Valortigara, Giorgio 2019. Cortical route for facelike pattern processing in human newborns. PNAS. Proceedings of the National Academy
of Sciences of the United States of America 116(10): 4625–4630. https://doi.org/10.1073/
pnas.1812419116
Bruce, Vicki 1988. Recognising Faces. New York: Routledge.
Notes on the semiotics of face recognition
357
Bruce, Vicki; Young, Andy 2012. Face Perception. New York: Psychology Press. https://doi.
org/10.4324/9780203721254
Courtine, Jean-Jacques; Haroche, Claudine 1992. Storia del viso. Esprimere e tacere le
emozioni (XVI–XIX). Palermo: Sellerio.
Crawford, Kate; Paglen, Trevor 2021. Excavating AI: The politics of images in machine
learning training sets. AI & SOCIETY: 1–12. https://doi.org/10.1007/s00146-021-01162-8
Damish, Hubert 1992. L’alfabeto delle maschere. In: Le Brun, Charles. Le figure delle
passioni. Conferenze sull’espressione e la fisionomia. (Giuffredi, Maurizio, ed.) Milano:
Raffaello Cortina, 102–116.
Darwin, Charles 1965[1872]. The Expressions of Emotions in Man and Animals. Chicago:
The University of Chicago Press.
Dutta, Paramartha; Barman, Asit 2020. Human Emotion Recognition from Face Images.
Singapore: Springer Nature. https://doi.org/10.1007/978-981-15-3883-4
Eco, Umberto 1968. La struttura assente. Milano: Bompiani.
Eco, Umberto 1975a. Volti di lupi, volti di agnelli. In: Johann Kaspar Lavater. Il Lavater
Portatile. Tom. I. Milano: Moizzi Editore.
Eco, Umberto 1980[1973]. Segno. Milano: Mondadori.
Eco, Umberto 1984. Proposals for a history of semiotics. In: Borbè, Tasso (ed.), Semiotics
Unfolding: Proceedings of the Second Congress of the International Association of Semiotic
Studies, Vienna, July 1979. Vol. 1. (Approaches to Semiotics 68.) New York, Amsterdam:
De Gruyter, 75–89. https://doi.org/10.1515/9783110869897-012
Eco, Umberto 2000[1997]. Kant and the Platypus: Essays on Language and Cognition.
London: Vintage.
Eco, Umberto 2010. Sugli specchi ed altri saggi. Milano: Bompiani.
Edkins, Jenny 2015. Face Politics. New York: Routledge. https://doi.org/10.4324/
9781315718040
Ellis, Hadyn D. 1975. Recognizing faces. British Journal of Psychology 66(4): 409–426.
https://doi.org/10.1111/j.2044-8295.1975.tb01477.x
Ellis, Hadyn D. 1986. Introduction: Processes underlying face recognition. In: Bruyer,
Raymond (ed.), The Neuropsychology of Face Perception and Facial Expression. New
York: Psychology Press, 1–27.
Ekman, Paul 1978. Facial signs: Facts, fantasies, and possibilities. In: Sebeok, Thomas A.
(ed.), Sight, Sound and Sense. Bloomington: Indiana University Press, 124–156.
Fabbri, Paolo 1995. Difformità del viso. In: AA.VV., Identità-Alterità. Figure del corpo
1895/1995. Biennale di Venezia, 46. Esposizione internazionale d’arte. Marsilio: Venezia.
Falco, Giuseppe 1923. Identità. Metodo scientifico di segnalamento e identificazione. Roma:
Maglione & Strini.
Galassi, Romeo; De Michiel, Margherita 1997. Il simbolo e lo specchio. Scritti della scuola
semiotica di Mosca-Tartu. Napoli: Edizioni Scientifiche Italiane.
Gates, Kelly A. 2011. Our Biometric Future: Facial Recognition Technology and the Culture of
Surveillance. New York, London: New York University Press. https://doi.org/10.18574/
nyu/9780814732090.001.0001
Gauthier, Isabel; Tarr, Michael J.; Moyan, Jill; Skudlarski, Pawel 2000. The fusiform “face
area” is part of a network that processes faces at the individual level. Journal of Cognitive
Neuroscience 1: 495–504. https://doi.org/10.1162/089892900562165
358
Remo Gramigna, Cristina Voto
Getrevi, Paolo 1991. Le scritture del volto. Fisiognomica e modelli culturali dal Medioevo ad
oggi. Milano: Franco Angeli.
Ginzburg, Carlo 1986. Miti emblemi spie. Morfologia e storia. Torino: Einaudi.
Goffman, Erving 1956. The Presentation of Self in Everyday Life. Edinburgh: University of
Edinburgh, Social Sciences Research Centre.
Goffman, Erving 1967. Interaction Ritual: Essays in Face-to-Face Behavior. New York:
Doubleday.
Gombrich, Ernst 1956. Art and Illusion: A Study in the Psychology of Pictorial Representation. Princeton: Princeton University Press.
Gramigna, Remo 2021. Le forme della maschera. Aspetti semiotici della manipolazione
del volto e della plasticità dell’apparenza. Lexia 37–38 (Volti Artificiali/Artificial Faces):
121–140.
Gurisatti, Giovanni 1991. Introduzione. In: Lavater, Johann Caspar; Lichtenberg, Georg
Christoph, Lo specchio dell’anima. Pro e contro la fisiognomica. Un dibattito settecentesco
(Gurisatti, Giovanni, ed.). Padova: Il Poligrafo, 11–64.
Gurisatti, Giovanni 2006. Dizionario fisiognomico. Il volto, le forme, l’espressione. Macereta:
Quodlibet.
Guthrie, Stewart Elliot 1995. Faces in the Clouds: A New Theory of Religion. Oxford: Oxford
University Press.
Hall, Edward T. 1959. The Hidden Dimension. New York: Anchor Books Doubleday.
Helfand, Jessica 2019. Face: A Visual Odyssey. Cambridge: The MIT Press.
Husserl, Edmund 1965. Idee per una fenomenologia pura e pe una filosofia fenomenologica.
Torino: Einaudi.
Ingold, Tim 2002. The Perception of the Environment: Essays on Livelihood, Dwelling and
Skill. London, New York: Routledge.
Kassner, Rudolf 1997. I fondamenti della fisiognomica. Il carattere delle cose. Vicenza: Neri
Pozza Editore.
Kosinski, Michal 2021. Facial recognition technology can expose political orientation from
naturalistic facial images. Sci Rep 11(100). https://doi.org/10.1038/s41598-020-79310-1
Krampen, Martin 1995. Proxemics revisited – Look at the body. In: Tasca, Norma (ed.),
Ensaios em Homenagem a/Essays in Honor of Thomas A. Sebeok. Cruziero Semiotico
22–25: 289–288.
Lavater, Johann Caspar; Lichtenberg, Georg Christoph 1991. Lo specchio dell’anima. Pro e
contro la fisiognomica. Un dibattito settecentesto (Gurisatti, Giovanni, ed., intr.). Padova:
Il Poligrafo.
Le Breton, David 2010. Antropologia del volto: Frammenti. In: Vinci, Daniele (ed.), Il volto
nel pensiero contemporaneo. Trapani: Il Pozzo di Giacobbe, 67–83.
Le Brun, Charles 1992. Le figure delle passioni. Conferenze sull’espressione e la fisionomia
(Giuffredi, Maurizio, ed.). Milano: Raffaello Cortina.
Leone, Massimo 2020. From fingers to faces: Visual semiotics and digital forensics.
International Journal for the Semiotics of Law 34: 579–599. https://doi.org/10.1007/
s11196-020-09766-x
Leopold, David A.; Rhodes, Gillian 2010. A comparative view of face perception. Journal
of Comparative Psychology 124(3): 233–251. https://doi.org/10.1037/a0019460
Lichtenberg, Georg Christoph 1991. Sulla fisiognomica; contro I fisionomi. In: Lavater,
Johann Caspar; Lichtenberg, Georg Christoph. Lo specchio dell’anima. Pro e contro
Notes on the semiotics of face recognition
359
la fisiognomica. Un dibattito settecentesto. (Gurisatti, Giovanni, ed., intr.) Padova: Il
Poligrafo, 97–138.
Lotman, Jurij 1998. Il ritratto. In: Lotman, Jurij. Il girotondo delle muse: Saggi di semiotica
delle arti e della rappresentazione. (Burini, Silvia, ed.) Bergamo: Moretti & Vitali, 63–96.
Lotman, Jurij. 2006. Tesi per una semiotica della cultura. (Sedda, Franciscu, ed.) Roma:
Meltemi.
Macho, Thomas; Treusch-Dieter, Gerburg 1996. Medium Gesicht. Die faciale Gesellschaft.
Berlin: Elefanten-Press.
Magli, Patrizia 1995. Il volto e l’anima. Fisiognomica e passioni. Milano: Edizione CDE.
Magnet, Soshana Amielle 2011. When Biometrics Fail: Gender, Race, and the Technology
of Identity. Durham: Duke University Press. https://doi.org/10.1515/9780822394822
Manetti, Giovanni 1993. Theories of the Sign in Classical Antiquity. Bloomington: Indiana
University Press.
Mourad, Youseff 1939. La physiognomonie arab et le Kitāb al-Firāsa. Paris: Librairie
Orientaliste Paul Geurhner.
Nöth, Winfried 1990. Alice’s adventures in semiosis. Working Papers and Pre-publications
195, serie B. Urbino: Centro Internazionale di Semiotica e Linguistica, Università di
Urbino.
Ogibenin, Boris L. 1975. Masks in the light of semiotics: A functional approach. Semiotica
13(1): 1–10. https://doi.org/10.1515/semi.1975.13.1.1
O’Neil, Cathy 2017. Weapons of Math Destruction: How Big Data Increases Inequality and
Threatens Democracy. New York: Broadway Books.
Piras, Antonio 2010. Le parole del volto. Spigolature storico-linguistiche ai margini di un
campo semantico. In: Vinci, Daniele (ed.), Il volto nel pensiero contemporaneo. Trapani:
Il Pozzo di Giacobbe: 47–63.
Peirce, Charles S. 1906. Prolegomena to an apology for pragmaticism. The Monist 16(4):
492–546. https://doi.org/10.5840/monist190616436
Poe, Edgar Allan. 1996[1840]. The man of the crowd. In: Poe, Edgar Allan. Poetry, Tales
and Selected Essays. New York: The Library of America, 388–396.
Reid, Vincent M.; Dunn Kristy; Young, Robert J.; Amu, Johnson; Donovan, Tim; Reissland,
Nadja 2017. The human fetus preferentially engages with face-like visual stimuli.
Current Biology 27: 1825–1828. https://doi.org/10.1016/j.cub.2017.05.044
Ricoeur, Paul 2005. The Course of Recognition. Cambridge: Harvard University Press.
Rhodes, Gillian; Calder, Andy; Johnson, Mark; Haxby, James V. 2011. Oxford Handbook
of Face Perception. Oxford: Oxford University Press. https://doi.org/10.1093/
oxfordhb/9780199559053.001.0001
Ruesch Jurgen; Kees, Weldon 1959. Nonverbal Communication: Notes on the Visual
Perception of Human Relations. Berkeley, Los Angeles: University of California Press.
Sebeok, Thomas A. 1972. Semiotica e affini. Versus. Quaderni di studi semiotici 3: 1–11.
Thorek, Max 1946. The Face in Health and Disease. Philadelphia: F. A. Davis Company.
Todorov, Alexander 2017. Face Value: The Irresistible Influence of First Impressions. Princeton, New Jersey: Princeton University Press. https://doi.org/10.1515/ 9781400885725
Turkle, Sherry 1995. Ghosts in the machine. The Sciences 35(6): 36–39. https://doi.
org/10.1002/j.2326-1951.1995.tb03214.x
Viciano, Joan; Capasso, Luigi 2021. Contributions of facial imaging to human identification in forensic cases: A retrospective study (1996–2019) at the Operative Unit of
360
Remo Gramigna, Cristina Voto
Anthropology of the ‘G. d’Annunzio’ University of Chieti-Pescara (Italy). European
Journal of Anatomy 25(S2): 179-192.
Volli, Ugo 1972. Some possible developments of the concept of iconism. Versus. Quaderni
di Studi Semiotici 3: 14–30.
Wechsler, Harry; Phillips, Jonathon P.; Bruce, Vicki; Fogelman Soulié, Françoise; Huang,
Thomas S. 1998. Face Recognition: Theory and Practice. Berlin, Heidelberg: Springer
Verlag. https://doi.org/10.1007/978-3-642-72201-1
Windley, Phillip J. 2005. Digital Identity. O’Reilly Media.
Заметки о семиотике распознавания лиц
Способность воспринимать и опознавать других по лицам имеет ключевое значение. Умение различать друзей и врагов, своих и чужих, а также обнаруживать
и прогнозировать их возможные шаги, планы и намерения – навыки, доказавшие
свою важность в эволюционной истории человечества. Целью этого исследования
является изучение проблемы распознавания лиц как семиотического явления. Охват
работы ограничен распознаванием лиц человеком. Человеческое лицо анализируется
в контексте пересечения биологических и культурных процессов. Мы утверждаем,
что способность замечать сходство имеет социально-культурное измерение,
которое не следует упускать из виду. Опираясь на замечания Георга Лихтенберга
о физиогномии, мы обсуждаем критику семиотической предвзятости, ассоциацию
идей и механизм типизации, связанный с распознаванием лиц. Типизация лиц
рассматривается в контексте распознавания и идентификации лиц. Мы предлагаем
считать их тремя градиентами значения, составляющими карту отношений между
различными когнитивными операциями, участвующими в распознавании лиц.
Tähelepanekuid nägude äratundmise semiootika kohta
Teiste tajumine ja ära tundmine nende nägude põhjal on otsustava tähtsusega. Suutlikkus
tajuda teisi keskkonnas viibijaid – teha vahet sõprade ja vaenlaste, omade ja võõrast vahel –
ning teha kindlaks ja püüda ennustada nende võimalikke edasisi samme, plaane ja kavatsusi,
on oskuste kogum, mis on osutunud inimkonna evolutsiooniloos määravaks. Käesoleva
artikli eesmärgiks on vaadelda nägude ära tundmise teemat semiootilise nähtusena.
Uurimuse ulatus piirdub näo tajumisega inimese kui liigi poolt. Inimnägu analüüsitakse
bioloogiliste ja kultuuriliste protsesside vahelisel lävel. Väidame, et kujutise äratundmisel
on ühiskondlik-kultuuriline mõõde, millest ei tohiks mööda vaadata. Lähtudes Georg
Lichtenbergi märkustest füsiognoomia kohta, käsitleme semiootilise eelhoiaku kriitikat,
ideede assotsieerumist ja nägude äratundmisega seotud tüpifitseerimismehhanismi.
Käsitleme neid kui kolme tähendusgradienti, mis kaardistavad nägude äratundmisega
seoses kaalul olevaid erinevaid kognitiivseid operatsioone puudutava suhtevõrgustiku.
TheStudies
face and
the faceness
361
Sign Systems
49(3/4),
2021, 361–382
The face and the faceness: Iconicity in the early
faciasemiotics of Paul Ekman, 1957–1978
Devon Schiller1
Abstract: Paul Ekman is an American psychologist who pioneered the study of
facial behaviour. Bringing together disciplinary history, life study, and history
of science, this paper focuses on Ekman’s early research during the twenty-year
period between 1957 and 1978. I explicate the historical development of Ekman’s
semiotic model of facial behaviour, tracing the thread of iconicity through his life
and works: from the iconic coding of rapid signs; through the eventual turn from
classifying modes of iconic signification using gestalt categories to classifying modes
of producing iconic sign-functions using minimal units; to the role and importance
of iconicity for the study of the facial expression of emotion, both in terms of the
similarities between iconic and analogue signs as well as the differences between
facial coding and linguistic signification. In this intellectual genealogy, I argue not
only that Ekman relied extensively upon conceptualizations and terminologies from
semiotic thought for the creation of the Facial Action Coding System (FACS), but
also that the question of iconicity is the pivotal problem across the many discoveries
and innovations in what I term ‘Ekmanian faciasemiotics’.
Keywords: Basic Emotions Theory (BET); Facial Action Coding System (FACS);
facial expression; iconicity; nonverbal behaviour; Paul Ekman; psychology;
semiotics of the face
Introduction
Paul Ekman is an American psychologist born in 1934 who pioneered the study of
facial behaviour. He is best known for his seminal work doing pan-cultural research
on the facial expression of emotion, Basic Emotions Theory (BET), deception
detection, and micro-expressions, as well as for creating the Facial Action Coding
System (FACS). Although Ekman identifies himself as a psychologist, and not a
1
Department of English and American Studies, University of Vienna, Vienna, Austria;
e-mail: devonschiller@gmail.com. This research was supported by a DOC Fellowship from the
Austrian Academy of Sciences.
https://doi.org/10.12697/SSS.2021.49.3-4.06
362
Devon Schiller
semiotician, a review of the literature reveals that his science was heavily influenced
by semiotics. As Ekman (1982: 86fn2) writes, “over the years, [he] proposed a
number of different phrases” to distinguish his approach to facial expression
and its dynamic movement, but hopes that “terms, taken from semiotics, allow
a more lucid differentiation”. In his autobiography, Ekman (2016: 11, 40, 41) also
recalls the ways in which his collaborations and critiques with various “intellectual
fathers,” each of whom brought semiotic frameworks into their methodological
approaches, came about through “a string of coincidences” and “good luck”. And
by the late 1960s, Ekman (1982: 45) began to explicitly model facial behaviour as
“a multimessage, multisignal semiotic system”.
Bringing together disciplinary history, life study, and history of science, this
paper focuses on Ekman’s early research during the twenty-year period between
1957 and 1978. The extent to which Ekman’s natural science of nonverbal behaviour, especially in its formative years, was informed and inspired by semiotic
theories and thinkers has been largely overlooked in the histories of this science
(Plamper 2015: 142–172; Leys 2017: 76–128), which tend to fixate on later
debates between biological essentialist and social constructivist positions. The
importance of semiotics for Ekman and his measurement techniques has even
been widely neglected in the semiotic literature, from general encyclopedia
(Bouissac 2007[1998]) and handbooks (Grammer 2004: 3451–3452) to specialized
monographs (Berson 2015: 113–115), where Ekman receives passing but persistent
citation. In this paper, I seek neither to advocate nor oppose what Maria Gendron
and Lisa Feldman (2017: 24) characterize as Ekman’s “‘neuro-cultural’ theory” for the
facial expression of emotion, but instead to reframe his research within the context
and history of the discipline of semiotics. I explore how Ekman referenced semiotic
works by Jurgen Ruesch as well as Charles W. Morris in his semiotic model of facial
behaviour, entered the discipline of semiotics through his friendship with Thomas
A. Sebeok, and was further inspired by David Efron, among others.
For this intellectual genealogy, I draw from primary sources that include
Ekman’s autobiographical writings, professional correspondence, and scientific
research, contextualizing his statements with relevant confirmations, criticisms,
and confusions, and comparing his semiotics to Peircean as well as Saussurean
traditions. Across the three sections of this paper, I explicate the historical
development of Ekman’s semiotic model, tracing the thread of iconicity through
his life and works: from the iconic coding of rapid signs; through the eventual
turn from classifying modes of iconic signification using gestalt categories to
classifying modes of producing iconic sign-functions using minimal units;
to the role and importance of iconicity for the study of the facial expression of
emotion, both in terms of the similarities between iconic and analogue signs as
The face and the faceness
363
well as the differences between facial coding and linguistic signification. With this
historicization of a theorization, I argue not only that Ekman relied extensively
upon conceptualizations and terminologies from semiotic thought for the creation
of FACS, but also that the question of iconicity, rather than stereotypicality or
universality, is the pivotal problem across the many discoveries and innovations
of what I term ‘Ekmanian faciasemiotics’.
1. The quest for facial iconicity begins
Ekman got his start in semiotics during graduate school in the 1950s. He obtained
his doctorate in clinical psychology from Adelphi College on Long Island in New
York in 1958, at the time the only programme that trained psychotherapists,
but which later would become the first university-based professional school of
psychology in the United States. Before he graduated, Ekman was required to
complete a one-year clinical internship. He pursued this internship under Swissborn American psychiatrist Jurgen Ruesch at the Langley Porter Neuropsychiatric
Institute, a part of the University of California San Francisco Medical Center
which, founded in 1941, was the first psychiatric institute in California. Ekman
chose Langley Porter because Ruesch had written Communication: The Social
Matrix of Psychiatry with English anthropologist Gregory Bateson in 1951 and
Nonverbal Communication Notes on the Visual Perception of Human Relations
with American painter and poet Weldon Kees in 1956. Ekman (2016: 14) writes
in his autobiography that he had drafted an unpublished review of Nonverbal
Communication “praising the many interesting ideas but criticizing it for a total
lack of data, or even examples”. The critique was valid, at least from a methodological perspective, and considering that Ekman had been “trained in research
methods” by American psychologist Robert Berryman, “a Skinnerian” who did
operant conditioning studies and had “an explicit bias against theory not grounded
in direct observation” (Ekman 2016: 95). By the late 1950s, Ruesch went further
and put forward a semiotic model of nonverbal behaviour in his “Principles of
human communication”, published the same year Ekman applied for an internship
in 1957, and republished in Semiotic Approaches to Human Relations in 1972. As
Winfried Nöth (1990: 169) points out, “the title of Ruesch’s collection of earlier
papers” indicates how scientific research on nonverbal behaviour, “which was first
developed within communication theory, was later placed within the framework
of a semiotic theory”.
However, Ekman’s acquaintance with Ruesch’s communication-turned-semiotic
approach to nonverbal behaviour started even before this internship. In his first
364
Devon Schiller
published article, written during the third year of his doctoral studies, Ekman
(1957: 141) cites Ruesch in his own classification of interindividual behaviour
into three modal types, including the “verbal, vocal, and nonverbal” which, he
claims, “can be distinguished in terms of the medium of expression, the manner in
which they are perceived, their developmental sequence, and their communicative
value”. Further, Ekman (1957: 141) argues that “[a]ll three forms of behavior have
both consensually validated linguistic or symbolic meaning, and more private or
autistic connotations,” but that “[v]erbal far exceeds vocal or nonverbal behavior
in degree of codification and consequent symbolic use”. In this way, Ekman (1957:
141) introduces an idea like iconicity, although he does not call it this, by situating
nonverbal behaviours “less defined as to their meaning or intent” in opposition to
verbal and vocal behaviours that “achieve definite symbolic usage”. Looking back
with hindsight, Ekman (2016: 3) believes the paper “did not merit” publication in The
Journal of Psychology. Still, Ekman’s first publication illustrates how the face reader as
a young man, from the very outset of his ‘faciasemiotic project’, was investigating the
complexity as well as the continuity between iconic, indexical, and symbolic relations.
The same year that Ekman began his internship, Ruesch (1972: 127) defines
“communication” in the broadest sense to include “all of the procedures by which
one mind may affect another”. According to Ruesch (1972: 127), communication
“involves not only oral or written speech”, but also the performance and pictorial
arts, “and in fact, all human behavior”. Ruesch (1972: 127) triadically models the
relationship between these signs and their object: signs constitute “a circumscript
part of an action or event which either by force of its own structure or because of
attention paid to it possesses for an observer problem-solving properties or cue
values”; signals constitute “an impulse in transit regardless of whether it circulates
inside or outside a human organism”; and symbols constitute “an extraorganismic
device which has been agreed upon to refer in a condensed way to a series of
actions or events […] used for coding purposes in order to transmit messages”.
While all such processes of communication “exert an organizing influence” upon
constituent individuals and “weld them into a larger system” (Ruesch 1972: 128).
Therefore, to Ruesch (1972: 128), messages are “purposive expressions of internal
events with the intent to convey information to other persons” that have also “been
perceived and interpreted by another person”. As Ruesch (1972: 128) claims, the
“communication apparatus of man”, which is sheltered within his body, carries
messages through “effector organs, the sender”, the “evaluative apparatus, including
the functions of memory decision-making”, as well as “sense organs, the receiver”.
In the ten years that followed, Ekman’s model developed at least in part out of
Ruesch’s as, in turn, this psychologist and psychiatrist each increasingly relied upon
semiotics.
The face and the faceness
365
After the completion of his internship and graduation, Ekman was conscripted
into the army for two years, where he served as “chief psychologist” and conducted “applied studies” (Ekman 2016: 17), working at the Palo Alto Veterans
Administration Hospital upon his discharge, before returning to Langley Porter
for a postdoctoral fellowship supported by the National Institute of Mental Health
(NIMH) from 1961 to 1963. Ekman then received a post-doctoral research grant
from NIMH, which started in 1963 and was continuously renewed for the next
forty years, in parallel to his professorship in the Department of Psychiatry at
the University of California, San Francisco from 1971 to 2004. It was during his
postdoc that Ekman began to collaborate with other psychologists, as well as
linguists and anthropologists, who principally used semiotic theory. Ekman first
met Hungarian-born American semiotician Thomas A. Sebeok, the founder of
zoosemiotics and, later, biosemiotics, at the Indiana University Conference on
Paralinguistics and Kinesics in Bloomington, Indiana in 1962 (Sebeok 1990: 287).
Over the next twenty years, Sebeok included two papers by Ekman in
volumes that he edited in the Approaches to Semiotics and The Semiotic Web
book series (Ekman 1978; 1987), and Ekman published four co-authored papers
in Semiotica, the journal of the International Association for Semiotic Studies,
during Sebeok’s tenure as its editor-in-chief (Ekman, Friesen 1969; Ekman,
Friesen, Tomkins 1971; Johnson, Ekman, Friesen 1975; Ekman, Friesen, Scherer
1976). In the correspondence between Sebeok and Ekman (Sebeok 1940–2001),2
which today is held by the Indiana University Archives, is a letter of reference
that Sebeok wrote on 31 August 1976 for the NIMH Research Scientist Award
in which he characterizes Ekman as “quite simply the world’s leading specialist
[…] in communication through nonverbal behavior” and commends how Ekman’s
methods “give evidence of his mastery of current semiotic theory and psychological
technique”. Reciprocally, in 1977 Ekman (Sebeok 1940–2001) wrote how he
“tremendously value[s] Semiotica” and that it is “by far the best journal in this
whole area”. In his autobiography, Ekman (2016: 95) even honours Sebeok as “the
central figure in the last century promoting the field of semiotics”, although he does
not further mention the doctrine of signs.
In 1969, Sebeok invited Ekman “to write an article for the first issue” of Semiotica (Ekman 2016: 95). To Ekman (2016: 95–96), this paper, “The repertoire
of nonverbal behavior”, remains “the seminal theoretical description of [his]
approach”, with most of what he has “done in the years since an elaboration of what
2
The author would like to thank Mary Mellon, Digital Archivist at the Indiana University
Archives, for her assistance in scanning the correspondence between Sebeok and Ekman. These
documents are now available online: http://webapp1.dlib.indiana.edu/findingaids/view?doc.
view=entire_text&docId=InU-Ar-VAE0871.
366
Devon Schiller
[he] wrote then, not a replacement”. Ekman co-authored the paper with American
psychologist Wallace V. (Verne) Friesen, whom he met while serving in the U.S.
Army, beginning what would be “twenty-five years of very fruitful collaboration”
(Ekman 2016: 37). In this seminal paper, Ekman and Friesen (1969: 49) define “a
person’s non-verbal behavior” as “any movement or position of the face and/or the
body” and describe the “origin, usage, and coding” for this behaviour. According
to Ekman and Friesen (1969: 60), coding is the “the principle of correspondence
between the act and its meaning” which describes the ways in which the nonverbal
act contains or conveys information; that is, the “rule which characterizes the
relationship between the act itself and that which it signifies”. Ekman and Friesen
(1969: 60) identify “three coding principles”, namely: nonverbal signs may be
“iconically (extrinsically)” coded and “carry the clue to their decoding in their
appearance”, where “the nonverbal act, the sign, looks in some way like what it
means, the significant”; they may be “intrinsically” coded and “like iconically coded
behavior” visually relate to what they signify, only where “the act does not stand
for but is its significant”; or nonverbal signs may be “arbitrarily (extrinsically)”
coded and, “like words”, “bear no visual resemblance to what they signify”. As
shown in Tables 1 and 2 respectively, the terms and definitions for the iconicintrinsic-arbitrary triad in Ekmanian faciasemiotics roughly correspond with the
sign-signal-symbol triad in Ruesch’s semiotics and, in turn, the icon-index-symbol
triad in the Locke-Peirce-Morris tradition.
Table 1. Correspondence between the phenomenological terms of semiotic theories.
Paul Ekman
Jurgen Ruesch
Charles W. Morris
C. S. Peirce
iconic (extrinsic)
sign
icon
icon
intrinsic
signal
index
index
symbol
symbol
arbitrary (extrinsic) symbol
The face and the faceness
367
Table 2. Comparative definitions of iconicity.
Ekman 1960s
Iconically-coded signs “carry the clue to their decoding in their
appearance,” where “the nonverbal act, the sign, looks in some
way like what it means, the significant” (Ekman, Friesen 1969:
60).
Ruesch 1950s
Signs constitute “a circumscript part of an action or event which
either by force of its own structure or because of attention paid
to it possesses for an observer problem-solving properties or cue
values” (1972: 127).
Morris 1940s
An “iconic sign, it will be recalled, is any sign which is similar in
some respect to what it denotes,” such that “the strength of the
iconic sign lies in its ability to present for introspection what it
signifies” (1946: 191, 194).
Peirce
late 19th – “An Icon is a sign which refers to the Object that it denotes
early 20th C. merely by virtue of character of its own and which it possesses,
just the same, whether any such Object actually exists or not,”
although an icon “does not act [as] a sign,” but “anything [...] is
an icon of anything, in so far as it is like that thing and used as a
sign of it,” being “strictly a possibility, involving a possibility, [in
which] the Interpretant may be the Object” (EP 2: 291, 2: 227).
In fact, Ekman and Friesen (1969: 60fn4) reference in a footnote how their “use of
the term iconic was taken from” American philosopher and semiotician Charles
W. Morris. Ekman came to embrace the works of Morris via the works of Ruesch,
who does not use the terms ‘icon’, ‘iconic’, or ‘iconicity’ in his taxonomy of signs,
but often cites Morris (Ruesch 1972: 93, 299, 344, 500, 559), giving an unequivocal
indication of this intellectual genealogy. As Ekman and Friesen quote, Morris
characterizes the iconic sign as “any sign which is similar in some respect to
what it denotes. Iconicity is thus a matter of degree. [Further,] the strength of the
iconic sign lies in its ability to present for introspection what it signifies” (Morris
1946: 191, 194; quoted in Ekman, Friesen 1969: 60 fn4). It is significant that
Ekman looked not to Peirce but to Morris for the semiotic foundations of his face
science. As Nöth (1990: 49) historicizes, the semiotics of Morris is characterized
by a “fundamental departure from the Peircean tradition”. Whereas American
pragmatist and semiotician Charles Sanders Peirce (CP 2.227, 8.343) conceives the
“quasi-necessary, or formal, doctrine of signs” on “logical analysis”, Morris (1946:
2) holds the conviction that “a science of signs can be most profitably developed
on a biological basis and specifically within the framework of the science of
368
Devon Schiller
behavior”. Bringing together behaviourist empiricism, logical positivism, and neopragmatism in a behavioural semiotics that preceded biosemiotics, Morris defines
semiosis as a “sign process, that is, a process in which something is a sign to some
organism” (Morris 1946: 353). In terms of iconicity, Nöth (1990: 123) identifies,
Morris reduced Peirce’s multiple criteria for the icon to the single criterion of
“‘shared properties’ and believed on this basis in a quantifiability of iconicity”.
However, Ekman and Friesen do not only define nonverbal iconicity by the
presence of a partial resemblance to the referent, but also by the absence of absolute
likeness to this object. That is, Ekman and Friesen (1969: 61) describe how the
“iconically coded act is often easier to comprehend and simpler to utilize as a
communicative signal; it is more stylized, starker, perhaps more abstract, and will
leave out many of the details involved in the intrinsically coded act which it may
resemble”. In other words, it is the properties of the object (e.g. emotion) that do
not determine the construction of the sign (e.g. facial expression) that are revealed
through its interrogation in such a way that new information may be derived.
Following the tradition of Morris and Ruesch, Ekman and Friesen situate this
semiotics within an explicitly biological, communicative, and social framework.
Although Ekman and Friesen (1969: 61, 68) state that not many facial signs can
be classified cleanly into one coding or another, and that this coding must be
based on the predominant function of the facial sign in a given context, the coding
in Ekmanian faciasemiotics is principally a hierarchical relation, or a matter of
degree on a continuum, in which the intrinsic comes beneath and before the iconic
(extrinsic) and the arbitrary (extrinsic).
2. Giving the semiotic modelling of face a hand
In addition to these works by Ruesch and Morris, Ekman and Friesen’s early
semiotic model for facial behaviour “owe[d] most to the writings” of Argentinian
anthropologist David Efron (Ekman, Friesen 1969: 63). As Ekman (2016: 10)
recalls in his autobiography, Efron studied under German-born anthropologist
Franz Boas, the so-called father of American anthropology, who “urged his
students to challenge the writings of the Nazi-influenced German social scientists
during the 1930s”, including the fallacious argument that nonverbal behaviour is
genetically determined, and thereby makes visible racial difference, “scientifically”
justifying the eugenics programme of the Nazis against the Jews. For his doctoral
dissertation Gesture and Environment in 1941, republished as Gesture, Race and
Culture in 1972 also in Sebeok’s Approaches to Semiotics series, Efron (1972[1941]:
6) conducted quantitative observational studies of what he calls the “gestural styles”
The face and the faceness
369
of East-European Jews and Southern Italians in New York. Efron (1972[1941]: 16,
136) claims that the racist ideology of Nazi Germany is “plagued with conceptions
that have no place in scientific reasoning” and concludes that “traditional Jewish
and Italian gestures disappear with the social assimilation of the individual
into the so-called Americanized community”, thus confronting the biological
determinism of Nazi ideology by confirming the socio-ecological construction of
gestural repertoires. As Ekman (2016: 10) reflects, the “publication of Efron’s book
coincided with the U.S. entrance into World War II” and, consequently, this “great
work was lost [and] rarely cited”. However, “building on Efron’s work”, Ekman
(2016: 37) “developed a classification” of nonverbal behaviour, which included
gestural behaviour as well as facial behaviour.
For this semiotics of the face, Ekman and Friesen (1969: 63) further drew upon
“a series of discussions” with American psychologist George F. Mahl in which
they “attempted to clarify some of the issues implicit in his dichotomization of
nonverbal behavior”. In a study of psychiatric outpatients during initial clinical
assessments, Mahl (1987[1968]: 15) categorizes nonverbal behaviour into “actions
judged to be common substitutions for verbal utterances” and those “judged not
to be” so, or “communicative gestures” such as nodding, pointing, or shrugging,
and “autistic actions” such as scratching, rubbing, or touching oneself. Ekman
and Friesen (1969: 57) believe that “Mahl’s distinction between autistic and
communicative behavior, while valuable, can be improved upon”, that this “use of
the term ‘communicative’ is too broad,” and that communication should distinguish
among “behavior which has a shared decoded meaning (informative), that which
influences the other person’s interaction (interactive)”, and, as Mahl had in mind,
“that which [is] intended to transmit a message (communicative)”. Seeking input
on “films of psychiatric patients”, Ekman visited Mahl during his fellowship from
1963 to 1964 at what today is known as the Center for Advanced Study in the
Behavioral Sciences at Stanford University. As Ekman (2016: 37) writes in his
autobiography, Mahl suggested that he focus on the patients’ nonverbal “‘acts’ and
develop a theoretical classification”. Ekman (2016: 37) remembers this being “quite
a step forward” for him, especially as “a Skinnerian who had been schooled in
not imposing theory, in simply just counting what you saw”. Indeed, this moment
for Ekman marked a shift in his overall epistemology, from bottom-up inductive
reasoning to top-down deductive reasoning and from specific observations to
general principles, which in turn laid the conceptual groundwork for his early
semiotic modelling of facial behaviour.
Proposing a triadic model of iconic, intrinsic, and arbitrary coding, Ekman
and Friesen (1969: 61) then proceed to “refine further the ways in which a
nonverbal act is related to its significant”. Largely utilizing Efron’s “terminology
370
Devon Schiller
and distinctions”, they classify five different “visual relationships between act
and significant”, including kinetic, pictorial, pointing, rhythmic, and spatial
relationships, as well as five categories that describe the ways in which nonverbal
behaviours can be “distinguished by the particulars of usage, origin and coding”,
including ‘adaptors’, ‘affect displays’, ‘emblems’, ‘illustrators’, and ‘regulators’, as well
as numerous sub-categories for a system with an indefinite and unfixed number of
phenomenological categories which they term ‘codes’ (Ekman, Friesen 1969: 61,
63, 68). At this point, Ekman and Friesen theorize their typology primarily, but
not exclusively, for hand gestures rather than facial behaviours. For instance, they
demonstrate the difference between iconic and intrinsic coding, or what would
be iconic and indexical in Peircean as well as Morrisian terms, with gestures of
the arm, hand, and fingers in aggressive actions which involve visually forming
“cut[ting] a person’s throat” or a “trigger finger movement” and actually holding a
knife or gun (Ekman, Friesen 1969: 61).
Here, the principal distinction between iconic and intrinsic coding is whether
the nonverbal behaviour is literally a part of the movement involved in the action.
In other words, is the act just being portrayed or is it really being performed, does
it “represent” or “reproduce” a human action, where to be iconic it must “resemble
but cannot be its significant” (Ekman, Friesen 1969: 70). Ekman and Friesen (1969:
61) admit that the “line between the iconic and intrinsically coded act may appear
to be a fuzzy one”, especially in the case of a nonverbal behaviour that is “only a
part of a total action”. In sum, to Ekman and Friesen (1969: 62) rhythmic and
spatial relationships between movement and meaning are always iconic, kinetic
relationships may be either iconic or intrinsic, and pointing is always intrinsic,
but “typically, nonverbal behavior combines elements of more than one code”, and
the pictorial may include the spatial, the spatial may include the rhythmic, and
so on. That is, not many signs clearly fall into one category or another but must
be categorized based on their function within a context. For example, as Ekman
and Friesen (1969: 68) note, they “are not proposing that the illustrator category
is exclusive of the others; assignment of an act into this category depends upon
usage in a given instance”. Thus, in Ekmanian faciasemiotics, iconization represents
sensory experience and visual perception through a formal resemblance between
the sign and the object that it refers to in the extralinguistic real world.
Ekman and Friesen (1969: 78) recognize that the “coding of facial affect displays
is not at all obvious”. Drawing from English naturalist Charles Darwin and his
1872 work titled The Expression of the Emotions in Man and Animals, the principal
claim for this semiotic model, which later became known as Basic Emotions
Theory (BET), is threefold. Firstly, the facial expression of emotion is prototypical,
specified through biology. Secondly, it is universal across cultures. And, thirdly,
The face and the faceness
371
there is a causal link between the outside physiological behaviour of a facial
expression and the inside psychological phenomena of an emotion that emanates
it; that is, between facial signifier and emotional signified. According to Ekman
and Friesen (1969: 71), the so-called “primary affects”, such as ‘anger’, ‘disgust’, ‘fear’,
‘happiness’, ‘sadness’, and ‘surprise’, a list that constantly undergoes variation and
revision across Ekman’s career, as well as all possible “affect blends”, activate with
the neural appraisal of an “evoking stimulus” and trigger so-called “programs” of
response, which are functionally discrete and phylogenetically stable, and include
the muscular contraction mechanisms and skin appearance movements of the
face. In this view, any variability to innate expression is largely determined by
social scripts or “display rules”, which are “socially learned, probably quite early
in life, and prescribe different procedures for the management of affect displays
in various social setting, roles, etc.” (Ekman, Friesen 1969: 75). Therefore, Ekman
and Friesen (1969: 74) principally consider what they call “affect ‘programs’” to be
intrinsically coded, or, in Peircean terms, “a sign which refers to the Object that it
denotes by virtue of being really affected by that Object” (EP2: 291–292), where the
expressing of the face in an emotion is analogous to the “veering of a weathercock”
in the wind (EP2: 274), being the product of this causation.
There is of course much debate in the scientific community over the merits and
limitations of BET. Points of criticism encompass, for example, how the theory fails
to adequately consider the significance played by context and culture in the facial
expression of emotion, the degree to which it is part of social intent, and is not only
reactive but also predictive, serving to guide actions. Opposing theories include,
for instance, the Behavioural Ecology View and Theory of Constructed Emotion.
Ekman (2016: 11, 48, 78) cites Darwin as one of his “intellectual father figures”
and claims Darwin’s Expression as the foundational text for BET, although Ekman
did not read Darwin until quite late in his career. However, in Expression, Darwin
(1872: 12, 356) hypothesizes not only that facial expression of emotion “once
existed in a much lower and animal-like condition” and consequently may have
certain elements of pan-cultural universality, but also that “every true or inherited
movement of expression seems to have had some natural and independent origin”.
Evolved by natural selection, these emotion expressions are either remnants from
reflexes that had once been useful, arise from the antithesis of contrasting elicitors,
or are the direct action of the nervous system in an overflow of excitation. Darwin
(1872: 67, 76) concludes that most human facial behaviour is a rudimentary or
vestigial feature much like the appendix, body hair, tail bone, or wisdom teeth,
which is “of no service, often of much disservice” and “purposeless”, with any
communicative value being incidental.
372
Devon Schiller
To Ekman and Friesen (1969: 78), “[b]oth Darwin’s explanation of the evolution
of such displays, and [their own] account of how certain displays may naturally
develop in the course of each person’s life, would suggest that some affect displays
are either intrinsically coded or iconic”. However, Ekman and Friesen (1969: 78)
concede that this coding may be the case “only for some affects”. “[I]f we accept
Darwin’s principle of antithesis as the explanation of the happiness display,” Ekman
and Friesen (1969: 78) admit, “then it would be arbitrarily coded”. Even so, social
psychologist Alan J. Fridlund (2017: 77) argues that Ekman, among other advocates
of BET, “misread” Darwin “when they cite him to support their claim that facial
expressions evolved ‘to express emotion’” and that certain facial expressions
are universal. But if the face did not evolve for carrying information and social
communication, as Darwin finds, this does not mean that facial expressions would
bear no resemblance to the emotions they signify, as Ekman and Friesen suggest;
they could still be intrinsically or iconically coded, just on different grounds.
Nevertheless, for Ekman and Friesen (1969: 60 n5) the “recognition of the need
to distinguish intrinsically coded from iconic behavior grew out of a discussion”
with American psychologist Silvan S. (Solomon) Tomkins, who had modelled a
universalist primary affect theory in the first two volumes of his magnum opus,
Affect Imagery Consciousness in 1962 and 1963. In Ekmanian faciasemiotics,
the distinction between the iconic or arbitrary and the intrinsic is crucial in
determining whether a facial event is a voluntary, staged, and posed expression
that accompanies intentional communication, or an involuntary, spontaneous, and
genuine expression that accompanies felt emotion.
This semiotic model of facial behaviour is highly problematic on methodological
as well as theoretical levels. Of course, there are different approaches to modelling
nonverbal behaviour across the academic disciplines, including coding systems that
classify the alteration, description, or function of behaviour, which today are more
and more based not only on anatomy but also on neuroanatomy. As critiqued by
neurologist Hedda Lausberg (2013: 65), creator of the Neuropsychological Gesture
Coding System (NEUROGES), Ekman and Friesen define their “main values”
for “the analysis of movement behavior in social interaction” as “predominantly
functional”, with adaptors, affect displays, emblems, illustrators, and regulators all
theorized in connection with a certain psychodynamic. However, some of these
“main values contain several movement classes”, such as how “regulators can be
positions shifts but also head nods”, and, additionally, the values “may all function
as regulators” (Lausberg 2013: 65). Consequently, “there is no precise definition
referring to the visual appearance of the movement” (Lausberg 2013: 65).
Ultimately, Ekman also came to recognize this methodological problem, both in
“14 techniques for measuring facial actions [over] a span of 55 years” between 1924
The face and the faceness
373
and 1978 (Ekman 1982: 50), including anatomically, ethologically, linguistically,
and theoretically based systems, as well as his own Facial Affect Scoring Technique
(FAST), a precursor to FACS (Ekman, Friesen, Tomkins 1971). Psychologist
Erika Rosenberg, who worked on FACS with Ekman in his Human Interaction
Laboratory at the University of California, San Francisco from 1991 to 1994, and
who now instructs how to code the face in her FACS Workshop, points out that the
“problem with theoretically derived systems is that they cannot discover behaviors
that were not posited in advance” and are “by definition selective”; in other words,
these coding systems “were not developed to catalogue everything the face can
do, but rather they describe whether the face does things it should do according
to a given theory” (Rosenberg 2005[1997]: 14).3 Having begun his search for
facial iconicity with the works of Ruesch, Efron, Mahl, Sebeok, and Tomkins, among
others, Ekman (1982: 46) turns from theorizing a semiotic model to developing a
measurement tool, one which can be used for “measuring the sign vehicles that convey
the message” rather than making interpretation “judgments about one or another
message”, and for classifying “descriptive units” rather than “inferential labels”.
3. Analogue privilege, digital process
In their seminal paper, Ekman and Friesen (1969: 60n4) also cite how their
distinction between iconic and arbitrary coding is “very similar” to Ruesch’s
distinction between analogue and digital codification. To Ruesch (1972: 131),
“analogic codification makes use of continuous representations”, whereby “the
signs used are, in their proportions and relations, similar to the things, ideas or
events for which they stand”, while “digital codification, in contrast, is based upon
discontinuous representation of events in which the continuity of nature is sliced
into discrete steps”. Of course, Ekman and Friesen (1969: 60fn4) make clear that
when they write about the terms ’analogic’ and ‘digital’, they do not mean these as
media per se, or to “involve further specifications of the mathematics relevant to
modelling information processing” from computer or electronic technologies, in
the sense of the terms already being used in the 1960s and most frequently applied
today. Rather, working from Ruesch’s definition, Ekman and Friesen (1969: 60n4)
go on to claim that the iconic-analogue is fundamentally “more characteristic
of nonverbal behavior” such as facial behaviour while the arbitrary-digital is
fundamentally more characteristic of verbal behaviour such as spoken language.
3
For transparency of interest, the author participated in the FACS Workshop with Rosenberg
at the University of California, Berkeley in 2015 and the NEUROGES seminar with Lausberg at
the German Sports University Cologne in 2017.
374
Devon Schiller
This contrast between the analogic and the digital in Ekmanian faciasemiotics
is overall consistent with semiotic theory. As Daniel Chandler (2017: 183) surveys
the field, semioticians tend to regard analogue codes as distinctions of degree, a
relation of more-or-less, signs which are natural, implicit, as well as continuous, that
is, a gradation of infinite subtleties; digital codes as distinctions of kind, a relation
of either/or, signs which are artificial, explicit, as well as discontinuous, that is, a
categorization of discrete units; and communication as something that “involves
both modes”. This distinction between analogue and digital codes, Chandler (2017:
184) explicates, is “frequently represented as natural versus artificial”, extending an
argument made by Claude Lévi-Strauss about the isomorphism between oppositions:
that of nature and culture and that of continuous and discrete. Yet such a distinction,
which has underlain the faciasemiotic project since classical physiognomy, carries
with it an entire host of dyadic connotations, such as contrasting the emotional work
of the flesh against the rational work of the mind and, consequently, facial behaviour
against spoken language, with the connotation being that it is the outer physiological
form that most accurately reflects the inner psychological function.
In Ekmanian faciasemiotics, the signs of the face are considered more genuine
and truthful, and consequently of higher value, when compared with the signs
of spoken, thought, or written language. This can especially be seen in Ekman’s
research into deception detection. For instance, Ekman and Friesen (1969: 88, 93)
refer to the concealed or repressed affect displayed in “‘[m]icro’ facial expressions”
with comparatively short duration as “nonverbal leakage”. And psychologist
Maureen O’Sullivan and Ekman (2004: 275) refer to individuals who achieve 80%
accuracy identifying deception in facial behaviour as “‘expert’ lie detectors or
[truth] ‘wizards’”. Behind this nomenclature is the perspective that intrinsic facial
signs are not only analogic but also natural and that iconic as well as arbitrary facial
signs are not only digital but also artificial.
In his 1978 paper “Facial signs”, Ekman (1978: 141, 151fn9) reflects on how
he has “argued that the linkage between facial movement (sign) and emotion
(significant) is natural […] rather than a conventional or arbitrary association,”
noting that he uses “the term natural as it has been discussed and defined by
Sebeok”. To Sebeok (1975: 237, 239), both signals and symptoms exemplify
natural signs, as a signal “triggers some reaction on the part of the receiver” either
“mechanically (i.e., naturally) or conventionally (artificially)”, and a symptom is a
“compulsive, automatic, nonarbitrary sign, such that the signifier is coupled with the
signified in the manner of a natural link”. Whereas Ekman singles out the naturality
of indexicality, or what he terms intrinsicality, Sebeok (1975: 242, 247) explains
how iconic and indexical sign relations have a natural link, whereas symbolic sign
relations have a “‘conventional link’ – Peirce’s ‘imputed character’”.
The face and the faceness
375
Drawing from Sebeok, Ekman (1978: 141) claims that his “view of facial
expression has been based on the assumption that if facial expressions are evolved
behavior”, and “if the relationship between sign and significant is natural”, then
facial expressions of emotion must also be “impervious to the influence of culture”.
However, Ekman (1978: 141) clarifies, he has “never said that facial expressions are
always automatic or unwitting. Facial expressions of emotion are not fixed-action
patterns or instincts of some kind, impervious to culture. They can be automatic,
but not always or even usually”. To Ekman, the intrinsic face, that is, the indexical
face, is the rare face in the endless forms most beautiful of human communication.
Yet, regarding the value of the icon, like in Peircean semiotics, these picture-like
signs resemble the object that they signify, have the modality of direct, immediate
perception, and are therefore the most persuasive of signs.
While Ekman privileges analogue signs in his semiotic model of facial behaviour, digital sign processes are important and even necessary for the development of his Facial Action Coding System. In the ten years between 1969 and
1977, Ekman and his research team in the Human Interaction Laboratory at the
University of California, San Francisco developed the Facial Action Coding System,
released in 1978 and revised in 2002. FACS is a “comprehensive” sign-based
technique for the description, measurement, and classification of “all possible
visually distinguishable facial movements” (Ekman, Friesen 1976: 58). This
coding system is based at least in part on work of French neurologist Guillaume
Duchenne and Swedish anatomist Carl-Herman Hjortsjö, among others (Ekman,
Friesen 1976: 63). Applying FACS, Ekman and Friesen (1976: 64) instruct, a coder
scores individual Action Units (AUs) that have an “anatomical basis” but do not
necessarily correspond one-to-one to either specific muscles or muscle groups.
FACS supports the performance of a spatial analysis of the face, both globally
with principal components, as well as locally with particular features, with the
use of intensity scores for each Action Unit which are annotated by appending
letters (A-E) from trace or very slight (A) to maximal or strongest possible (E).
FACS also supports the performance of a temporal analysis of the face through
the observation and measurement of the combination of Action Units, or what is
commonly meant by an expression, with the use of timing scores which segment
the actions into their “onset, apex, and offset” (Ekman, Friesen 1976: 60). For
instance, in what is popularly known as the smile of happiness, the zygomatic major
constricts posteriorly and superiorly, pulling the lip corners upwards, and the
orbicularis oculi, pars lateralis constricts laterally, raising the infraorbital triangle,
lifting the cheeks, and gathering the skin medially toward the eye socket from
around its lateral edge. As Fig. 1 illustrates, the combination of these facial actions
in FACS is described using Action Unit 6, the “Cheek Raiser”, and Action Unit
376
Devon Schiller
12, the “Lip Corner Puller” (Ekman, Friesen 1976: 65), or, with the lips apart and
mouth open, and attendant intensity ratings, and 6D+12E+25C.
Figure 1. Onto-cartographic projection in four-dimensions (4D), three dimensions of space
(width, depth, and height) and one dimension of time, representing the iconic continuum
of facial signs.
This “digitization” of the face has led to a common misconception in expert
scholarship and popular culture: that in Ekmanian faciasemiotics, facial behaviour is understood as a linguistic system, one in which muscular contraction
mechanisms form up alphabetic elements, and skin appearance movements
function like grammatical morphemes, whereby the face speaks or writes emotion which in turn can be heard or read by a face-literate expert coder. For
instance, “Ekman is Saussurean,” Jeremy Sherman commented to me in at the
Annual Gathering in Biosemiotics at Lomonosov Moscow State University in
2019. The unspoken implication was not that Ekman follows Saussure through
his bibliographic references, but rather that Ekmanian faciasemiotics resembles
Saussurean linguistics. As Swiss linguist and semiotician Ferdinand de Saussure
(2006: 140) writes, “[l]anguage represents a system internally ordered in all its
parts”, which “depends on an object, but free and arbitrary in relation to the object”.
Further, according to Saussure (2006: 147), “language and writing are NOT BASED
on a natural relationship between things,” and there is “never in any way a link”
between the signifier and signified.
The face and the faceness
377
Certainly, Ekman and Friesen (1976: 64, 65fn) note, for the Action Units of the
Facial Action Coding System, the names are a “shorthand, not meant to describe
the appearance changes, but a convenience to call them to mind,” and the “numbers
are arbitrary and do not have any significance”. Yet this application of digital signs
in the study of the face is not the same as alleging that facial behaviour is itself a
language. Nevertheless, the popular myth that in Ekmanian faciasemiotics the face
is viewed as a text is widely propagated, for instance in Lie to Me, the 2009–2011
television drama series based on Ekman’s life and works, which carries the tagline
“The truth is written all over our faces”. However, to claim that facial behaviour
meets some linguistic benchmark would be to confuse digital with linguistic signs.
“[A]nalogue signs can of course be digitally reproduced,” Chandler (2017: 185)
explains, “but they cannot be directly related to a standard ‘dictionary’ and syntax
in the way that linguistic signs can”. As Ekman (1978: 151fn9) writes in his own
words, it is “important not to misread [his] use of the terminology of semiotics as
an implication that the facial signs are a linguistic system”.
Nevertheless, the foundational process for Ekmanian faciasemiotics is the
intersemiotic translation from the analogue signs of certain facial behaviours to
the digital signs of the Facial Action Coding System. Still, as Chandler (2017: 184)
points out, all sign systems “impose digital order on what we often experience
as a dynamic and seamless flux”, and the “very definition of something as a sign
involves reducing the continuous to the discrete”. Structuralists including Saussure,
Lévi-Strauss, and Roman Jakobson even theorize about how the process of making
differentiations and the mere otherness of either/or oppositions is fundamental
not only to linguistic systems in particular but also to signifying systems in
general. Across his career, Ekman described the facial expression of emotion
using objectivity and rationality, both in the 1960s with his use of affect display
gestalts, since the 1970s with his use of minimal Action Units, as well as since the
1980s with the selective application of AUs to gestalts for stereotypical emotions, as
with EmotionFACS (EMFACS). However, this was not because he observed these
facial behaviours to be this way amid the phenomena of the world; rather, it was
because Ekman experimented on facial signs in such a way amid the processes of
his study. That is, the definition for the facial expression of emotion in Ekmanian
faciasemiotics has been altered to fit a particular conception of what science ought
to be in order to make it practisable.
378
Devon Schiller
Conclusion
The question of whether or to what extent facial behaviour may be indexical
is by no means resolved, either in Ekman’s oeuvre or the face sciences. In the
neuro-cultural model of Ekmanian faciasemiotics, facial signs may be “iconically
(extrinsically), intrinsically, or arbitrarily (extrinsically) coded” (Ekman, Friesen
1969: 60), which is to say, iconically, indexically, or symbolically coded in terms
deriving from the Locke-Peirce-Morris tradition, on the basis of the visual
appearance of the dynamic movement, either in whole or in part, and depending
on the predominant function of the nonverbal behaviour in the given context. In
this view, the facial expression of emotion is most likely “either intrinsically coded
or iconic” (Ekman, Friesen 1969: 78). However, the distinction between these codes
is a “fuzzy one” and the way to determine this principle of correspondence between
the act and its meaning, the signifier and its signified, is “not at all obvious”
(Ekman, Friesen 1969: 61, 78).
Further, if the neuroanatomy of the face did not evolve at least to some
degree for carrying information about emotion phenomena, whether on the
intra- or inter-organismic level, then the facial expression of emotion “would be
arbitrarily coded” (Ekman, Friesen 1969: 78). In other words, as Umberto Eco
argues, “spontaneous, noncodified expressions of emotion produced without any
communicative intention” may well fall “below the semiotic threshold”, and only
“become codified and thus semiotic” when they “exhibit cultural variation or are
simulated or imitated in a histrionic context” (Eco cited in Nöth 2000: 54). For
Ekman, however, there is an important “recognition of the need to distinguish
intrinsically coded from iconic behavior” (Ekman, Friesen 1969: 60fn5). In
Ekman’s terms, iconic and arbitrary coding, each of which are extrinsic codes,
and not entirely part of the essential nature of the nonverbal behaviour, but also
operate with or arise from the outside, and “signifies or stands for something else”,
whereas intrinsic coding “is in a sense no code in that the act does not stand for
but is its significant; the meaning of the act is intrinsic to the action itself ” (Ekman,
Friesen 1969: 60). Additionally, this distinction means differentiating the analogic
and natural signifying relation of intrinsic codes from the digital and artificial
signifying relation of iconic and arbitrary codes. As Ekman (1978: 141) argues, “the
relationship between sign and significant is natural” in certain facial expressions of
emotion, and thereby “automatic or unwitting” and “impervious to the influence
of culture”, but facial behaviour is “not always or even usually” this way.
Ekman’s science, as Jan Plamper (2015: 155) historicizes, “involved the reconciliation of nature (basic emotion) and culture (display rule)”. And as Ruth Leys
(2017: 90) contextualizes, “Ekman was committed to the project of distinguishing
The face and the faceness
379
between nature and culture, between the ‘natural signs’ of emotional expression
that might be so small or rapid as to be easily overlooked and those culturally
coded displays or ‘artificial signs’ that he thought tended to mask or disguise
them”. Thus, in Ekman’s faciasemiotic project, beneath the exclusively arbitrary
coding that is entirely culturally assigned, the indexical face really performs the
emotion, intrinsically representing a part of the movement involved in this action,
such as with the “enjoyment smile”, whereas the iconic faceness just portrays the
emotion, extrinsically resembling the action only in some respect, such as with the
so-called “false”, “masking”, and other non-enjoyment smiles (Ekman, Davidson,
Friesen et al. 1990: 342–343). This is exactly why iconicity is so essential in the
early faciasemiotics of Paul Ekman: by lying, iconic facial signs provide clues to
the truth, that is to say, the truthfulness of the indexical.
References
Berson, Josh 2015. Computable Bodies: Instrumented Life and the Human Semiotic Niche.
London: Bloomsbury.
Bouissac, Paul 2007[1998]. Encyclopedia of Semiotics. Oxford: Oxford University Press.
Chandler, Daniel 2017[2002]. Semiotics: The Basics. (3rd ed.) London: Routledge.
CP = Peirce, Charles S. 1931–1958. Collected Papers of Charles Sanders Peirce. Cambridge:
Harvard University Press. (Vols. 1–6, Hartshorne, Charles; Weiss, Paul, eds.; vols.
7–8, Burks, Arthur Walter, ed.) [In-text references are to CP, followed by volume and
paragraph numbers.]
Darwin, Charles 1872. The Expression of the Emotions in Man and Animals. London: John
Murray. https://doi.org/10.1037/10001-000
Efron, David 1972[1941]. Gesture, Race and Culture: A Tentative Study of the Spatiotemporal and “Linguistic” Aspects of the Gestural Behavior of Eastern Jews and Southern
Italians in New York City, Living Under Similar as Well as Different Environmental
Conditions. (Approaches to Semiotics 9.) The Hague: Mouton.
Ekman, Paul 1957. A methodological discussion of nonverbal behavior. The Journal of
Psychology 43(1): 141–149. https://doi.org/10.1080/00223980.1957.9713059
Ekman, Paul 1978. Facial signs: Facts, fantasies, and possibilities. In: Sebeok, Thomas A.
(ed.), Sight, Sound and Sense: Advances in Semiotics. Bloomington: University of Indiana
Press, 124–156.
Ekman, Paul 1982. Methods for measuring facial action. In: Scherer, Klaus R.; Ekman, Paul
(eds.), Handbook of Methods in Nonverbal Behavior Research. Cambridge: Cambridge
University Press, 45–135.
Ekman, Paul 1987. A life’s pursuit. In: Sebeok, Thomas A.; Umiker-Sebeok, Jean (eds.), The
Semiotic Web ‘86: An International Yearbook. Berlin: Mouton de Gruyter, 3–45.
Ekman, Paul 2016. Nonverbal Messages: Cracking the Code – My Life’s Pursuit. San Francisco:
Paul Ekman Group.
380
Devon Schiller
Ekman, Paul; Davidson, Richard J.; Friesen, Wallace V. 1990. The Duchenne smile:
Emotional expression and brain physiology II. Journal of Personality and Social
Psychology 58(2): 342–253. https://doi.org/10.1037/0022-3514.58.2.342
Ekman, Paul; Friesen, Wallace V. 1969. The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica 1(1): 49–98. https://doi.org/10.1515/
semi.1969.1.1.49
Ekman, Paul; Friesen, Wallace V. 1976. Measuring facial movement. Environmental
Psychology and Nonverbal Behavior 1(1): 56–75. https://doi.org/10.1007/BF01115465
Ekman, Paul; Friesen, Wallace V.; Scherer, Klaus R. 1976. Body movement and voice
pitch in deceptive interaction. Semiotica 16(1): 23–27. https://doi.org/10.1515/
semi.1976.16.1.23
Ekman, Paul; Friesen, Wallace V.; Tomkins, Silvan S. 1971. Facial affect scoring technique
(FAST): A first validity study. Semiotica 3(1): 37–58. https://doi.org/10.1515/
semi.1971.3.1.37
EP = Peirce, Charles S. 1992. The Essential Peirce: Selected Philosophical Writings.
Bloomington: Indiana University Press. (Vol. 1, Houser, Nathan; Kloesel, Christian,
eds.; vol. 2, Peirce Edition Project, ed.) [In-text references are to EP, followed by
volume and page numbers.]
Fridlund, Alan J. 2017. The behavioral ecology view of facial displays, 25 years later. In:
Fernández-Dols, José-Miguel; Russell, James A. (eds.), Oxford Series in Social Cognition
and Social Neuroscience: The Science of Facial Expression. New York: Oxford University
Press, 77–92. https://doi.org/10.1093/acprof:oso/9780190613501.003.0005
Gendron, Maria; Barrett, Lisa Feldman 2017. Facing the past: A history of the face in
psychological research on emotion perception. In: Fernández-Dols, José-Miguel;
Russell, James A. (eds.), Oxford Series in Social Cognition and Social Neuroscience: The
Science of Facial Expression. New York: Oxford University Press, 15–38.
Grammer, Karl 2004. Körpersignale in menschlicher Interaktion. In: Posner, Roland;
Robering, Klaus; Sebeok, Thomas A. (eds.), Semiotik: Ein Handbuch zu den zeichentheoretischen Grundlagen von Natur und Kultur. 4. Teilband. Berlin, New York: Walter
de Gruyter, 3448–3486. https://doi.org/10.1515/9783110179620.4.15.3448
Johnson, Harold G.; Ekman, Paul; Friesen, Wallace V. 1975. Communicative body
movements: American emblems. Semiotica 15(4): 335–353. https://doi.org/10.1515/
semi.1975.15.4.335
Lausberg, Hedda (ed.) 2013. Understanding Body Movement: A Guide to Empirical Research
on Nonverbal Behaviour: With an Introduction to the NEUROGES Coding System.
Frankfurt: Peter Lang.
Leys, Ruth 2017. The Ascent of Affect: Genealogy and Critique. Chicago: University of
Chicago Press. https://doi.org/10.7208/chicago/9780226488738.001.0001
Mahl, George F. 1987[1968]. Gestures and body movements in interviews. In: Mahl,
George F., Explorations in Nonverbal and Vocal Behavior. Hillsdale: Lawrence Erlbaum
Associates, 7–74.
Morris, Charles W. 1946. Signs, Language, and Behavior. New York: Prentice-Hall. https://
doi.org/10.1037/14607-000
Nöth, Winfried 1990. Handbook of Semiotics. Bloomington: Indiana University Press.
https://doi.org/10.2307/j.ctv14npk46
The face and the faceness
381
Nöth, Winfried 2000. Umberto Eco’s semiotic threshold. Sign Systems Studies 28(1): 49–61.
https://doi.org/10.12697/SSS.2000.28.03
O’Sullivan, Maureen; Ekman, Paul 2004. The wizards of deception detection. In: Granhag, Pär Anders; Strömwall, Leif A. (eds.), The Detection of Deception in Forensic
Contexts. New York: Cambridge University Press, 269–286. https://doi.org/10.1017/
CBO9780511490071.012
Plamper, Jan 2015. The History of Emotions: An Introduction. (Tribe, Keith, trans.) Oxford:
Oxford University Press.
Rosenberg, Erika L. 2005[1997]. The study of spontaneous facial expressions in psychology. In: Ekman, Paul; Rosenberg, Erika L. (eds.), What the Face Reveals: Basic and
Applied. Studies of Spontaneous Expression Using the Facial Action Coding System
(FACS). (2nd ed.) Oxford: Oxford University Press, 3–18. https://doi.org/10.1093/
acprof:oso/9780195179644.003.0001
Ruesch, Jurgen 1972. Semiotic Approaches to Human Relations. (Approaches to Semiotics
25.) The Hague: Mouton. https://doi.org/10.1515/9783110816228
Saussure, Ferdinand de 2006. Writings in General Linguistics. (Bouquet, Simon; Engler,
Rudolf, eds.; Sanders, Carol; Pires, Matthew, trans.) Oxford: Oxford University Press.
Sebeok, Thomas A. 1940–2001. Thomas Sebeok papers, correspondence and subject files.
Collection C264, Boxes 26, 27, 61. Bloomington: Indiana University Archives.
Sebeok, Thomas A. 1975. The semiotic web: A chronicle of prejudices. Bulletin of Literary
Semiotics 2: 1–63. https://doi.org/10.5840/bls197521
Sebeok, Thomas A. 1990. Semiotics in the United States. In: Sebeok, Thomas A.; UmikerSebeok, Jean (eds.), The Semiotic Web 1989. Berlin: Walter de Gruyter, 275–396. https://
doi.org/10.1515/9783110874099
Лицо и лицевость: иконичность в ранней лицосемиотике
Пола Экмана, 1957–1978
Пол Экман – американский психолог, ставший пионером изучения поведения лица.
Статья посвящена ранним исследованиям Экмана в 1957–1978 годах и объединяет
историю дисциплины, биографию и историю науки.
Я рассматриваю, как развивалась семиотическая модель поведения Экмана,
и прослеживаю проявления иконичности в его жизни и работе: от иконического
кодирования быстрых знаков; через поворот от классификации режимов знакового
обозначения с использованием категорий гештальта к классификации режимов
создания знаковых функций с использованием минимальных единиц; к роли и
важности иконичности для изучения выражения эмоций на лице с точки зрения
как сходства между знаковыми и аналоговыми признаками, так и различий между
кодированием лица и лингвистическим обозначением. В этой интеллектуальной
генеалогии я утверждаю не только, что Экман в значительной степени полагался на
концептуализацию и терминологию семиотики при создании Системы кодирования
лицевых движений (FACS), но также что проблема иконичности лежит в основе
многих открытий и инноваций, которые я называю «лицосемиотикой Экмана».
382
Devon Schiller
Nägu ja näolisus: ikoonilisus Paul Ekmani varases näosemiootikas,
1957–1978
Paul Ekman on Ameerika psühholoog, kes pani aluse näokäitumise uurimisele. Artiklis
liidetakse eriala ajalugu, eluloouuringud ning teaduslugu, keskendudes Ekmani varajasele
teadustegevusele kakskümmend aastat kestnud perioodil vahemikus 1957 kuni 1978.
Selgitan Ekmani loodud näokäitumise semiootilise mudeli ajaloolist arengut, järgides
ikoonilisuse juhtlõnga tema elus ning töödes alates kiirete märkide ikoonilisest kodeerimisest, läbi viimaks toimunud pöörde ikoonilise tähistamise laadide klassifitseerimiselt
gestaltkategooriaid kasutades ikooniliste märgifunktsioonide tekitamise klassifitseerimisele
minimaalseid üksuseid kasutades; kuni ikoonilisuse rolli ning tähtsuseni emotsionaalsete
näoilmete uurimises, nii ikoonliste ja analoogmärkide sarnasuste terminites kui ka näolise
kodeerimise ja keelelise tähistamise erinevuste kaudu. Väidan, et selles intellektuaalses
genealoogias Ekman mitte üksnes ei toetunud näotegevuse kodeerimissüsteemi (FACSsüsteemi) luues ulatuslikult semiootilisest mõtlemisest pärinevatele kontseptualiseeringutele ja terminoloogiale, vaid lisaks on ikoonilisuse küsimus pöördelise tähtusesega
probleem arvukates avastustes ja uuendustes valdkonnas, mida ma nimetan “ekmanlikuks
näosemiootikaks”.
Marlen Haushofer’s The Wall and Sign
the post-nuclear
culture
of the
face383–399
383
Systems Studies
49(3/4),
2021,
Marlen Haushofer’s The Wall and
the post-nuclear culture of the face
Emanuela Ferragamo1
Abstract. The intertwining of landscape and face belongs to human spatial epistemology: as suggested by Matteo Meschiari, primitive humans used to orientate
themselves in landscape through recognition of facial patterns. By reflecting
upon Marlen Haushofer’s novel The Wall (Die Wand), the article aims to question
the semantic of the “face of the landscape” in the wake of an imagined nuclear
apocalypse that leaves behind a cat, a cow, a dog, a woman and a wall. The wall
transcends the boundaries between human and other-than-human: in terms
of Roberto Marchesini, it creates a somato-landscape – a hybridization of inner
and outer landscapes typical of post-human awareness. Finally, such a landscape
culminates in the dismissal of the pre-apocalyptic culture of the face: faces no longer
function as a means of recognition.
Keywords: face; landscape; boundary; exotopy; hyperobject; melancholy; trouble
Introduction
The Austrian writer Marlen Haushofer published the novel The Wall (Die Wand)
in 1963.2 The novel begins rather simply: the protagonist, a middle-aged widow,
goes to the Austrian mountains to spend some time with her cousin Luisa and
Luisa’s husband Hugo. However, on the day after her arrival she finds herself alone
in the house. When looking for her hosts, who have not returned, she discovers a
transparent wall at the end of the mountain’s gorge.
Manuscripts show that Haushofer gave the novel its present title, thematizing one
of her recurrent metaphors (Schmidjell 2000: 47). Indeed, critics interpret the ‘wall’
1
Foreign Language, Literature and Cultures Department, University of Turin (Unito), Via
Verdi 8 – 10124 Turin, Italy; e-mail: emanuela.ferragamo@unito.it.
2
I quote from the edition published by Ullstein Taschenbuch (Haushofer 1985) and refer to
the novel through its initials followed by the page numbers, and provide the English translation
by Shaun Whiteside (Haushofer 1999) in the footnotes. I refer to the latter through its initials
followed by the page numbers.
https://doi.org/10.12697/SSS.2021.49.3-4.07
384
Emanuela Ferragamo
in the context of her repeated reference to constricted places (Von der Lühe 1986:
82), where she staged both women’s oppression and a possible exit from patriarchy.3
More recently, literary inquiries have re-established an interest in the narrative
of natural space and landscape in The Wall.4 The present paper stems from this
context and studies how the post-nuclear landscape of the novel affects the
depiction of faces in it, analysing the relationship between the terms ‘landscape’
and ‘face’, while referring to Umberto Eco’s (1985: 10) concept of semiotics as a
process of integrating one’s reflection as the symbolic image of the Self in a broader,
symbolic system.
The paper consists of three parts. Firstly, the narrative function of mirrors is
highlighted with reference to the status of the presence of the face in a post-nuclear
word where the protagonist is the only survivor. Reflecting on mirrors means
considering the problem of recollection of the past in the symbolic constitution of
the protagonist’s face in her silent dialogue with the dead.
3
Regula Venske views the novel as an expression of Haushofer’s tendency to stage the quest
for identity in non-places where women’s boredom is both the expression and the consequence
of the psychological suffering of women in middle-class society (Venske 1987: 199). However,
those places of sensorial inexistence express not only a desire for self-annihilation, but also a
struggle to overcome the stifling social conventions of post-war Austria (Venske 1987: 205). In
The Wall, Haushofer’s last woman fights for survival, thus introducing an intertextual reference
to Robinson Crusoe. However, more recent criticism tends to question the connection: Michael
Hoffman (2000: 202) underlines the characters’ different approaches to natural resources and
a more problematic use of the utopian genre in Haushofer’s novel. The question of a gendered
genre is also relevant: Sarah Nelseen (2019: 136) suggests reading Haushofer’s robinsonade
through Donna Haraway’s’ concept of SF as both Science Fiction and Speculative Feminism.
Some critics view the killing of the last man at the end of the novel in the light of an
urge to masculinize women: the murder of the last surviving man appears as an allegory of
women’s emancipation from all sorts of masculine projections. However, the murder has
negatively influenced the reception of her novel. Thus, it comes as no surprise that feminism
has been just recently discovering Haushofer’s legacy, reconquering a long-neglected author
(Knecht 2007: 84). During the 1970s, feminists greeted the last pages of her novel with a certain
puzzlement due to a resistant inhibition towards feminine aggression (Venske 1987: 213) and
the predominance of elements of fragility and sensibility in women’s representations (Venske
1991: 23).
4
Here, I quote Régine Battiston’s recent essay on the meaning of landscape in Haushofer’s
work (Battiston 2019) and Johanna Chlovanec’s and Grazziella Predoiu’s (2016) articles on her
semiotics of space. The latter draw on Eduard Soja’s postmodern concept of ‘thirdspace’ and on
a reinterpretation of the robinsonade through a psychological analysis of the plot, respectively.
According to Chlovanec (2006: 26–28), Soja’s thirdspace reveals a third path in the narrative
of gender that goes beyond the dichotomy male versus female and deconstructs the patriarchal
assumption as such. Predoiu (2016: 76–77) focuses on the idyllic scenery and views the wall as
an allegory of the desire of a symbiosis with nature.
Marlen Haushofer’s The Wall and the post-nuclear culture of the face
385
Secondly, such a dialogue takes place within boundaries. Not only is the
wall a hindrance, but it also originates a semiosphere and results in the ultimate
construction of a palisade beside its surface. By doing this, the protagonist reinterprets her previous epistemology in a sense that transects gender and physical
boundaries: the wall and the palisade trigger an exotopic process of transection
between the inside and the outside of the semiotic system. This aspect is analysed
through the post-human idea of somato-landscape (Marchesini 2002: 189).
Finally, Haushofer’s landscape is observed from the theoretical perspective of
hyperobjects. In this view, the smooth surface of the wall appears as an anticipation
of the smooth space of Gilles Deleuze’s and Felix Guattari’s theory of space.
1. The face and the other: The face as a means of recollecting
the past and the quest for one’s identity in Haushofer’s novel
1.1. Facing landscape
Landscape means different things to different disciplines, its currency not always
a guarantee for taxonomic clarity.5 In this paper, I will follow Matteo Meschiari’s
suggestion and study Haushofer’s landscape in profile, which means discussing its
textual practice, rather than its essence (Meschiari 2008: 276). In order to do so,
I refer to Paolo D’Angelo (1999: 210–211) who summarizes the term’s definitions
as involving the typical form of a territory: a portion of earthly surface that
constitutes an image or, more generally, perceptive data received by an observer.
This definition helps us to outline an operative definition of landscape emerging
in Haushofer’s novel.
While studying the aesthetic meaning of the subject’s response to landscape,
D’Angelo questions its relationship with a philosophical view of nature, which
would possibly go beyond the ecological underappreciation of aesthetics. Thematizing beauty as the crux of modernity, D’Angelo (2001: 110) aims to examine its
connection with the historical and imaginative context of of the observer’s culture.
Haushofer depicts landscape as a cultural image originating from the collective
intervention on a territory.
Firstly, the emergence of the wall destroys the idyllic scenery of the Alps, as the
protagonist no longer finds the mountain “romantic and charming” but “damp and
gloomy” (TW: 19). As in the aftermath of the Second World War Austrian political
discourse would shape national identity thanks to the Alpine landscape, thus
5
For a useful and clear taxonomy of ‘landscape’ see Berque 1994, Schama 1995, Jakob 2005,
Weber 2010 and Sobral Campos 2019.
386
Emanuela Ferragamo
distancing itself from Nazi Germany (Zeyringer 2001: 60), the appearance of the
wall thematizes the struggle to deal with the collective memory of the Holocaust
as it forces the protagonist to cope with the “inability to mourn” of her generation
(Knecht 2007: 83). The protagonist finally understands how to deal with the wall
by letting her emotions out, bursting into tears while recollecting the dead on
Christmas Eve.
Secondly, the wall relates to the Cold War psychosis of nuclear destruction. As
the wall represents the collective fault of an entire age, it has no inventor and is
a step towards the ongoing alienation of modern humanity. More generally, the
wall marks the abrupt end of the ancient, intimate relationship between man and
nature. In this respect, its naissance recalls in some sense what Michael Jakob
considers the origin of landscape: the loss of the antique harmony of cosmos (Jakob
2005: 15).
D’Angelo emphasizes the process of shaping the territory in the making of
landscape, thus implicitly recalling the history of the term: Germanic and Romance
languages form the word ‘landscape’ using a suffix that indicates a territory and
its configuration: ‘-age’ in French, ‘-scape’ or ‘-schiftz’ in old English (Collot 2005:
152). Furthermore, they intend landscape as a way of organizing data in a coherent
unity that is to be realized both in arte and in visu (Collot 2005: 156). As Timothy
Morton has argued, landscape embodies a subjective state: a landscape picture
“is less about land, then, and more about scape. We talk about the mood of a
landscape, the feeling it evokes in us” (Morton 2011: 80).
The landscape’s position in front of the observer is particularly evident in
Haushofer’s novel. Here, the protagonist literally bumps her forehead against the
wall: a sort of transparent screen through which landscape appears as a distant
image (DW: 14).
The fact that landscape is in front of the subject guarantees its visibility in the
first pages of the novel, the latter implying historically the possibility of controlling
information and resources. The organization of landscape expresses political
longing for controlling the territory (Vitta 2005: 148). In this respect, it is relevant
that Haushofer highlights a connection between the transparency of the wall and
the desire of the protagonist to keep everything under visual control (DW: 187).
However, the novel problematizes the idea of landscape’s full visibility. While in
the 18th-century utopian fiction transparency is positive (Baczko 1978: 22), the
one of the wall forces the protagonist to walk with outstretched arms in order not
to get hurt (DW: 15).
Marlen Haushofer’s The Wall and the post-nuclear culture of the face
387
1.2. Facing faces
Haushofer’s critique of the paradigm of visibility also affects the perception of
faces. The German word ‘Gesicht’, that appears in the German-language version
of Haushofer’s novel, has its roots in the verb ‘sehen’, designating the outer, visible
dimension of face.6 In the novel faces seem hard to catch, even in places where
they should be present. The protagonist’s reflection in Luise’s vanity mirror speaks
more of the missing woman than of the present one: “In Luises Frisierspiegel sah ich
manchmal verwundert meine neue Erscheinung. Mein Haar, das stark gewachsen
war, hatte ich mit der Nagelschere kurz geschnitten. Es war jetzt ganz glatt und von
der Sonne gebleicht. Mein Gesicht war mager und gebräunt” (DW: 81–82).7
Appearing in the mirror, the protagonist recalls the disappearing of the person to
whom the object once belonged: Luise, who died after the appearance of the wall.
The protagonist imagines her as a doll with red lips and reddish-blond curls (DW:
124). The depiction of Luise as a sleeping beauty clearly expresses Haushofer’s
criticism of patriarchy (Chamayou-Kuhn 2019: 30) and the emotive stiffness
caused by women’s efforts to fit a stereotypical image: Luise seems in a desperate
need of social acceptance (DW: 12).8
When reflecting upon the appearing and disappearing of the protagonist and
Luise in the mirror, we may recall Eco’s (1985: 18) dismissal of mirrors as semiotic
channels. Mirrors put a sender and an addressee in contact, but they do not
generate a semiosis: this would require the absence of the object for whom the
sign stands, while mirrors need the object referred to be present (Eco 1985: 19). Yet
the mirrored object appears to be strangely present in the passage quoted above.
In a way, Luise appears in the mirror in the sense that the protagonist is forced to
remember her face (DW: 82). Thus it seems that Haushofer thematizes presence as
the effect of recollection, giving transparency a rather melancholic quality.
Hidden objects fascinate us because they seem to suggest that something has
become lost: facing a veiled face, we are in front of an enigma (Starobinski 1957:
7). Seduced by the object without being satisfied, our eyes keep trying to catch a
glimpse of the reluctant promise of appearance (Starobinski 1957: 7). The wall
causes a similar dynamic. The protagonist is afraid of looking through the wall
6
See http://woerterbuchnetz.de/cgi-bin/WBNetz/wbgui_py?sigle=DWB&mode=Vernetzung&
lemid=GG11297#XGG11297.
7
“I sometimes looked with amazement at my new appearance in Luise’s dressing-table
mirror. As my hair had grown a great deal, I had cut it short with the nail-scissors. Now it was
quite flat, and bleached by the sun. My face was thin and tanned, [...]” (TW: 68).
8
Considering Luise’s need to be liked, it is interesting that the mirror represents a relevant
tool for feminist studies to destructure social representations of women (Stephan, Weigel
1988).
388
Emanuela Ferragamo
and of seeing anything more than what she has understood in her heart of hearts
(DW: 12). Therefore, as she reaches the top of a raspberry patch and observes the
landscape through her binoculars from there, she does not manage to discern the
face of a motionless woman sitting in front of the coachmaker’s cottage beyond the
wall (DW: 29). Her attempts at focusing on the face of the unknown woman are
in vain: in the end, images fade in a kaleidoscope of forms and colours (DW: 29).
Starobinski (1975: 8) understands such frustration as melancholy: the fact that
the desired object is in full sight and yet impossible to reach. This can be related to
Massimo Leone’s (2015: 126) reflection upon the semiotics of transparency: when
Leone suggests considering transparency as an anticipation of transcendence, he
means that transparent objects are destining actants – they prevent the sight as
obstacles and at the same time they arouse the desire to see.
The apprehension towards hidden faces beyond the wall could also be
interpreted as the result of two distinctive moments which characterize recollection: availability and accessibility. Paul Ricoeur (2004: 129) believes that
Henri Bergson had the merit of solving a long-lasting riddle of memory: the fact
that its images are simultaneously lost, yet available through recollection. Still,
the clue to this apparent paradox relies on the centrality of recognition: to let
recollection exist, the subject has to recognize it as something with its roots both
in the past and in the present (Ricoeur 2004: 142). Therefore, recognizing the past
means recovering it, and recovering it means presuming it to be available, if not
accessible (Ricoeur 2004: 143).
The same dilemma occurs in a passage in Haushofer’s novel in which she
imagines her protagonist trying to reach some pansies, while, unfortunately, the
flowers grow on the other side of the wall: “Einmal im Frühling […] sah ich drei
oder vier Veilchen. Gedankenlos streckte ich die Hand aus und stieß in die Wand.
[…]. Die Veilchen hielten mir ihre kleinen violetten Gesichter entgegen, aber ich
konnte sie nicht anfassen” (DW: 167).9 Seeing the pansies, she mistakes their visual
accessibility for their actual availability, yet the mistake is soon corrected: the
flowers’ perfume she thought she smelled when she first glanced at the flowers
dissolves after she has touched the wall (DW: 167).
9
“Once in the spring […] I saw three or four violets. I absently reached out my hand and
leaned against the wall. […] The violets held their little purple faces to me, but I couldn’t not
touch them” (TW: 145).
Marlen Haushofer’s The Wall and the post-nuclear culture of the face
389
1.3. From the other side of the mirror.
By letting her protagonist stand in front of a mirror, Haushofer explores the
relationship between present images and their recollection. In doing so, she reflects
upon the semiotics of the mirror: the object for whom the sign in the mirror stands
coincides only partially with the actual object.
On another occasion, Haushofer addresses the topic of mirroring again,
depicting the protagonist’s face as a reflection in her cat’s eyes. “Ich sehe mein
Gesicht, klein und verzerrt, im Spiegel ihrer großen Augen”, she writes (DW: 52).10
While the previous quote emphasized the protagonist’s estrangement from her
appearance in Luise’s mirror, here Haushofer underlines the fact that her face has
lost its original proportions: it has become small and distorted. In a general sense,
this disproportion speaks for the asymmetry between the Self and the other, which
characterizes the ethical philosophy: giving a prominent status to the other, its
premises stem from asymmetry, rather than reciprocity (Ricoeur 2004: 181). In
other words, if we act ethically, we give more importance to the other than to
ourselves. This is why the protagonist depicts herself as small in comparison to
the cat: being the last human on earth, the protagonist feels how the responsibility
towards her animals and the wild is hers only to bear (DW: 127).
Moreover, the disproportion between the “smallness” of the protagonist’s face
and the size of her cat’s eyes can be interpreted as a plea to her counterpart to
accept the moral quest for relationship conveyed through visual contact. Emmanuel
Levinas understands the presence of faces as a moral awakening: they express the
eternal transcendence of the other, metaphorically breaking through the world
(Lévinas 1977: 199).
On the one hand, Haushofer imagines the relationship between the protagonist
and the cat like motherhood: the protagonist first notices the starving cat after she
thinks she has heard a baby crying (DW: 48) and, slowly, she becomes dependent
on the animal (DW: 51). On the other hand, as an independent, cautious animal,
the cat can be seen as representing the irreducible extraneousness of the other
in Mikhail Bakhtin’s terms. Relating with the other means for him overcoming
monologism on the linguistic level, responsibility and care on the moral level,
and exotopy on the aesthetic level (Ponzio 2003: 189–190). In Bakhtin’s works,
exotopy results from a reconsideration of the terms “inside vs. outside and defines
a transitioning of those two dimensions into one another: a process that is not
dialectic but represents a conceptualisation of the outside as traversing between
two discourses” (Kalinova 2018: 105).
10
“I can see my face, small and tight, in the mirror of her big eyes” (TW: 41).
390
Emanuela Ferragamo
Exotopy comes into mind when we consider the fact that the cat wears
a typographic sign on its fur: “Die Katze starrt aus gelben Augen in die Ferne.
Manchmal kommt sie plötzlich zu mir zurück, und ihre Augen zwingen mich, die
Hand auszustrecken und den runden Kopf mit dem schwarzen M auf die Stirn zu
streicheln” (DW: 149).11 Where someone else might have seen nothing but the
pattern of a cat’s coat, the protagonist sees a letter and not just a random one – ‘M’
being the initial of Haushofer’s first name, Marlen. As Augusto Ponzio (2003: 57)
notices, exotopy means the invisibility of the objective author, emerging because
of literary criticism, and of the objectivized one, representing the author in his
or her work. Therefore, authorship appears as a pure representational principle:
defying representation, the author is present in the dialogue of his or her many
voices (Ponzio 2003: 58).
Referring to the peculiar invisibility of the author and paraphrasing Levinas’
formulation for the ethical relationship with otherness, the silent dialogue between
the cat and the protagonist breaks through monologic writing, renegotiating the
boundaries between the inner and the outer.
2. Surviving beyond the wall: The wall as a boundary
2.1. The wall is more than just a wall
In Haushofer’s novel, this renegotiation takes place in an era of hyperobjects. In
the philosophy of Morton, hyperobjects designate both the object of study and
its theoretical frame. The premise of Morton’s thinking lies in the confutation
of Kant’s epistemology and his idea of a distance between the observing subject
and the observed object (Morton 2013: 27). In this regard, Morton (2013: 27)
considers global warming as a typical example of a hyperobject outside of which
it is impossible for us to step.
Although hyperobjects suit postmodern literature better, they appear to be
useful tools to interpret Haushofer, since they permit underlining of two aspects
of her depiction of the landscape and have a meaningful role in the understanding
of faces.
Firstly, both the wall of Haushofer’s novel and hyperobjects result from an
apocalypse of some kind. On the one hand, Haushofer’s protagonist faces the
newest product of humankind: a wall most likely originating from a nuclear
11
“The cat stares in the distance with yellow eyes. Sometimes she suddenly comes back to me,
and her eyes compel me to stretch out my hand and stoke her round head with the black M on
the forehead” (TW: 128).
Marlen Haushofer’s The Wall and the post-nuclear culture of the face
391
weapon (DW: 209). On the other hand, Morton (2013: 99) uses the expression
“end of the world” to question the aesthetics underlying the idea of ‘world’ as a
“container in which objectified things float or stand”.
Yet as different as ‘hyperobject’ and the wall may seem, they both express the
ancient meaning of apocalypse, the word indicating the ritual digging of a pit in
the outskirts of the city in honour of the dead in order to prevent their returning
home (De Martino 2002: 194). Similarly, Haushofer’s wall serves both as a tribute to
the dead, the wall being like a shrine at which to admire a modern Pompeii (DW:
56), and as a denial of death. In analogy, Morton (2013: 103) represents reactions
to hyperobjects on a scale that stretches from denial to melancholy.
Secondly, hyperobjects problematize Haushofer’s depiction of the wall as
something in front of the protagonist. In the epistemological context of Morton’s
philosophy, most of her struggles not to touch the wall express longing for the
avoidance of all intimacy with the Uncanny. For Morton (2013: 33), intimacy
means the inevitable proximity of hyperobjects. The loss of a safe distance from
troubling events, such as climate change, results in a forced cohabitation with
unknown phenomena, causing normality to appear extraneous, unreal. Haushofer’s
protagonist questions the plausibility of the wall: such things do not occur normally
in Austria, nor in Europe (DW: 19).
For the purposes of the present article, Morton’s phenomenology has two
consequences on the study of landscape as it problematizes the pictorial metaphor
pertinent to the term ‘landscape’ and renegotiates the concept of ‘technique’.
In the etymology of ‘landscape’, perception and depiction have traditionally
been intertwined: landscape is both a real, perceptible territory and its metaphorical representation. Those two characteristics are first synonymous and then
interrelated: through its imaginary, aesthetics conveys and influences the cultural
view of landscape (Milani 2017: 51).
However, the fact that Morton views landscape as a picture hanging from a
wall clashes with the spatial and temporal fluctuation of hyperobjects (Morton
2013: 74). If those are to be thought of as landscape, then the latter is to be seen
not as a painting but as a snapshot. Thus, it is no surprise that Haushofer has set
her novel in the mountains, where the landscape has proved to be more suitable
for photography than for painting (Cosgrove, Della Dora 2012: 12).
Thinking of landscape in terms of a hyperobject also means reflecting upon its
temporality. Whereas landscape studies mostly deal with changes over long-lasting
periods, hyperobjects rather focus on minimal, constant oscillations. If thinking in
the latter terms, research on landscape would involve what Donna Haraway (2016:
1) defines as “staying with the trouble”: living and thinking being truly present,
that is, focussing on little variations of patterns. Similarly, Haushofer’s protagonist
392
Emanuela Ferragamo
survives, having come to terms with her past and having renounced disclosing of
the future.
Landscape shifts out of focus, its spatial and temporal boundaries start to
interrelate, to blend in with one another. Rather than being a picture, it is a
collection of instantaneous fragments with no discernible progression or timeline:
something very similar to Georg Simmel’s concept of nature as an incessant flow
of phenomena (Simmel 1957: 72). In this respect, hyperobjects put the concept of
technique in a new perspective.
2.2. One plus one equals three: Unmaking the wall
As Rocco Ronchi (2017: 252) argues, in revisiting the Western view of nature the
concept of technique plays a great role. In particular, he speaks in favour of George
Simondon’s understanding of technique as the formal constituting of human
experience (Ronchi 2017: 254). The understanding of technique not as a product,
but as a process, permits us to reflect upon the technical history of humankind
in terms similar to the naturans of nature (Ronchi 2017: 254). Technique reveals
itself as a natural praxis, because the human ability of inventing and using technical
prosthesis has roots in nature (Ronchi 2017: 256–257). In a similar way, in her
novel Haushofer imagines a process of re-naturalization of technical artefacts: old
vehicles abandoned in the woods are a home for wild animals (DW: 78).
More generally, the relevance of technique emerges in the protagonist’s longing
for dexterity. On the one hand, her complaining about her clumsiness meets the
demands of the robinsonade, as she must survive in a hostile environment. On the
other hand, dexterity is part of a broader process of renegotiation of epistemology:
as Haushofer points out, the new reality is to be experienced with “hands, feet and
guts” (TW: 50).
An interesting starting point for reflecting on the post-apocalyptic meaning of
technique in the novel is the construction of a boundary. The protagonist attends
to the work in two stages, initially contenting herself with driving some branches
into the mud near the wall (DW: 20) and then carrying two knives with her to
cut them: the jack-knife of Hugo, her cousin’s husband, and her own penknife for
sharpening pencils (DW: 29). The boundary extends along the wall, making it
visible and hence less dangerous, and results in a sort of palisade.
The two knives constituting the equipment express two gestural programmes.
Greimas’ (1970: 69) concept of ‘gestural programme’ is part of a broader study
on semiotics of the human body in cultures and their gestural praxis: the use of
the body relies upon a common project and a shared sense. Greimas’ working
hypothesis is that phonological methods and practices apply to gestural
Marlen Haushofer’s The Wall and the post-nuclear culture of the face
393
signification because gestures correspond to expression (Greimas 1970: 85) –
the only difference lies in the fact that a gestural sequence is a “programme of
manifestation without a phonological project” (Greimas 1970: 86). Referring to
this methodological frame, Haushofer’s reference to the two knives expresses
the clashing of two gestural programmes of technical intervention on landscape:
weaponry and modesty.
The protagonist claims that she carries Hugo’s jack-knife because it gives
her an apparent sense of protection (DW: 22). Haushofer points out that Hugo’s
knife, belonging to his arsenal, is one of those objects that should have protected
him from the nuclear Armageddon he feared (DW: 10).12 Haushofer, who has
emphasized Hugo’s pointless hoarding of provisions, now marks a disproportion
between the sharpness of his knife and the way the protagonist uses it for a very
modest purpose. The boundary she makes by cutting branches is a distraction,
a springtime game (DW: 20) and speaks more generally for a modest practice of
technique, thus implicitly pleading in favour of what Hans Jonas (2000: 122) called
‘technical modesty’: the fact that something less than perfect could be accepted as
good and the consequent dismissal of technical megalomania.
While the wall is described as the consequence of human hubris toward nature,
the gestural programme of the making of the boundary creates a harmless replica
of the wall, a “toy boundary” (TW: 13). In reference to such a boundary, it is
interesting to remind that the word for ‘wall’ designates also ‘a fortification made
of branches’ in the German language.13
2.3. Troubles in translation
To clarify how the gestural programmes of technical omnipotence (the wall) and
modesty (the palisade) intertwine, a reference to Juri Lotman’s semiotics is in order.
Lotman (2000: 140) understands boundaries as means and spaces for different
semiospheres to transect, thus problematizing Bakhtin’s concept of exotopy for
the study of landscape.
For Lotman (2000: 140), boundaries mark the outer and the inner parts of
semiospheres. The latter define the whole semiotic space of a culture, which is
its result and a condition for its development (Lotman 2000: 125). As different
semiospheres contain different elements and functions, the transition of a message
12
“At the time everyone was talking about nuclear wars and their consequences, and this led
Hugo to keep a little store of food and other important things in his hunting-lodge” (TW: 3–4).
13 See https://woerterbuchnetz.de/?sigle=DWB#4. Indeed, the reference to branches recurs
in a further passage of Haushofer’s novel indicating the profound alienation of the protagonist
from nature: crushing branches under boots is for her another facet of human violence towards
the environment (DW: 62).
394
Emanuela Ferragamo
from one semiosphere into another occurs at the price of adjustments through
translations (Lotman 2000: 140). In this respect, Haushofer’s first boundary appears
to be the wall that filters and adapts the external into the internal, to paraphrase
Lotman (2000: 140). In the novel, this dynamic involves a renegotiation of the
opposition between the centre and the periphery of the cultural system: after the
emergence of the wall, the protagonist realizes that she lives in the farther portion
of Western geography, that she names Belutschistan (DW: 49).
Yet the wall also appears as a sign, “something that stands for something, to
someone in some capacity” (Ponzio 2003: 122). More precisely, the wall reveals
itself as a sign after the making of its interpretant, the palisade. For Peirce, a
sign means something in response to another sign (Ponzio 2003: 181), and the
construction of the palisade represents a reaction to the wall: having realized it is
a weapon (DW: 41), the protagonist replies by defending herself with a palisade,
translating the grounding element of the wall into a new boundary.
What changes during this translation is the meaning of order. On the one side,
the making of the palisade is a remedy against the violation of the cosmic order
caused by the wall (DW: 29). On the other side, the space delimited by the palisade
is one of disorder and trouble: a word expressing how “to live and die well with each
other in a thick present” (Haraway 2013: 1). Indeed, a thick present is the place
beyond the palisade. Due to the staggering increase of wild animals caused by the
extinction of humans, there are not enough natural resources for them to survive
without the help of the protagonist, as the wall precludes access to a considerably
vast portion of the earth. On the protagonist’s side of the wall, every action of the
protagonist risks damaging an environment already out of balance.
3. Final observations: Towards a post-apocalyptical
understanding of the face
3.1. Wood walks with her
When reflecting on the intertwining of intentions, directions and species during
the Anthropocene, Haraway (2013: 58) coins the concept of ‘sympoesis’. The
neologism emphasizes the creative interdependence of earthlings, thus referring
to a responsive, collective making of the word – a “wordling-with, in company”
(Haraway 2013: 58). Systems so defined defy spatial and temporal boundaries:
while the latter are usually “centrally controlled”, sympoetic systems distribute
information and control among their components (Haraway 2013: 61). Haushofer’s
protagonist experiences something similar to sympoesis in the final pages of the
novel: “Manchmal verwirren sich meine Gedanken und es ist, als fange der Wald an,
Marlen Haushofer’s The Wall and the post-nuclear culture of the face
395
in mir Wurzeln zu schlagen und mit meinem Hirn seine alten, ewigen Gedanken zu
denken. […] Es fällt mir schwer, beim Schreiben mein früheres und mein neues Ich
auseinanderzuhalten” (DW: 185).14
Referring to her protagonist’s trouble in distinguishing between what she was
and what she is while writing, Haushofer problematizes the act of writing as a
means of understanding reality: while the report gives the protagonist’s thoughts
formal coherence, it forces her to become aware of her dissolving identity. Here,
‘dissolving’ is used in the meaning of what Roberto Marchesini (2002: 189)
defines as somato-landscape: the transferring of landscape inside the human
body. In Haushofer, somato-landscape arises because writing urges both the
psychological and physical awakening of the Self and the epistemological revision
of the experience of the landscape. This provides the protagonist with tools for the
recollection, and thus the recognition, of herself.
Meschiari (2008: 60) has convincingly compared writing with mapping. The
similarity of these processes lies above all in the fact that both deal with unstable
objects: fiction with narrative plausibility and maps with the making and unmaking
of territories. The similarity between writing and mapping seems particularly true
for Haushofer’s novel, where the branches for the palisade were cut with a knife
meant for sharpening pencils (DW: 29).
The protagonist’s interrelation with the wood adds a puzzling question to the
depiction of mobility and immobility in the landscape. Whereas to some extent the
wall and the palisade stand for a reflection upon obstructed mobility, the woods
introduce a new variable in the reflection upon the landscape: that of “smooth”
space.
3.2. Face and other lines
In the second volume of their study A Thousand Plateaus: Capitalism and Schizophrenia, Gilles Deleuze and Félix Guattari thematize distinctions and interrelations between a “striated” and a “smooth” perception of space and understanding these in different cultural models.
Roughly distinguished, these perceptions correspond to the sedentary and
the nomadic experiences of places, respectively. In striated spaces, one goes from
one point to another, whereas in smooth spaces, points are less important than
trajectories: the movement in the landscape contemplates the possibility of changes
of direction (Deleuze, Guattari 1980: 596). The experience of smooth spaces has
14
“Sometimes my thoughts grow confused, and it is as if the forest has put down roots in me,
and is thinking its old, eternal thoughts with my brain […]. I find it hard to separate my old self
from my new self ” (TW: 161).
396
Emanuela Ferragamo
no visual points of reference or invariant distances. Differently from the striated
space, smooth space is made up of a steady changing orientation provided by
a population of nomads who are actively entertaining tactile relations among
themselves (Deleuze, Guattari 1980: 598).
In Haushofer’s novel, it appears clear how the wall orientates the landscape
towards a smooth model of space by forcing the protagonist towards a mostly
tactile perception of space. For the first time, the protagonist realizes that something
invisible is blocking her path by bumping into it: then, she carefully approaches the
wall with outstretched hands. Although she tries to avoid contact with the wall, its
invisibility forces her to trust her tactile perception instead of her eyes.
The analogy with Deleuze and Guattari’s theoretical frame becomes even more
striking at the end of the novel. Observing how a mountain brook has succeeded in
leaking into the wall, the protagonist is relieved: “Mit dem Wasser aus den Bächen
wird das Leben, winziges, einfaches Leben, einsickern und die Erde wiederbeleben”
(DW: 223).15 With the wish for the renewal of life in an elementary form, she
implicitly expresses the longing for the smooth space par excellence. That is
water (Deleuze, Guattari 1980: 597). Furthermore, the smooth experience of the
landscape clarifies the protagonist’s symbolic metamorphosis into a tree, as before
noticing how her thoughts seems to have their roots in the woods rather than being
her own, the protagonist gives a final depiction of herself: “Ich bin noch immer
mager, aber muskulös, und mein Gesicht ist von winzigen Fältchen durchzogen.
Ich bin nicht häßlich, aber auch nicht reizvoll, einem Baum ähnlicher als einem
Menschen” (DW: 182).16
The process of metaphorically becoming part of the woods can be interpreted
as an allegorical longing for smooth space. As Deleuze and Guattari (1980: 602)
praise the attachment to a territory explored through the mind as the ultimate
nomadic experience of modern times, Haushofer imagines that her protagonist
metaphorically travels into the woods without actually moving (DW: 185). Yet
the similarity the protagonist feels to have acquired with trees goes somehow
in the same direction as Deleuze’s and Guattari’s critique of Western emphasis
on the face as a means of fixing identity into one space (Deleuze, Guattari 1980:
623). Undulations, frictions, variations: the smooth space dissolves the face into a
riddle of intersecting lines and spirals. What Haushofer describes as wrinkles are
trajectories, points and lines of a new face in landscape.17
15
“With water from the streams life, tiny, simple life, will seep in and revivify the earth” (TW;
196).
16 “I am still scrawny, but muscular, and my face is criss-crossed with tiny wrinkles. I’m not
ugly, but neither am I attractive, more like a tree than a person” (TW: 69).
17 I dedicate this paper to my brother.
Marlen Haushofer’s The Wall and the post-nuclear culture of the face
397
References
Assunto, Rosario 2005[1973]. Il paesaggio e l’estetica. Palermo: Novecento.
Battiston, Régine 2019. Funktion der Landschaft in Marlen Haushofers Werk. In:
Alraud, Sylvie; Lacheny, Marc; Lajarraige, Jacques; Leroy du Cardonnoy, Éric (eds.),
Dekonstruktion der symbolischen Ordnung bei Marlen Haushofer. Berlin: Frank &
Timme, 39–57.
Baczko, Bronisław 1978. L’utopia. Immaginazione sociale e rappresentazioni utopiche nell’età
dell’Illuminismo. (Botto, Margherita; Gibelli Dario, trans.) Torino: Einaudi.
Berque, Augustin 1994. Cinq propositions pour une théorie du paysage. Ceyzérieu: Champ
Vallon.
Chamayou-Kuhn, Cécile 2019. Grenz und Fremdheitserfahrungen in „Die Wand“. In:
Alraud, Sylvie; Lacheny, Marc; Lajarraige, Jacques; Leroy du Cardonnoy, Éric (eds.),
Dekonstruktion der symbolischen Ordnung bei Marlen Haushofer. Berlin: Frank &
Timme, 17–39.
Chlovanec, Johanna 2016. Marlen Haushofers „Die Wand“ als Thirdspace. Sprachkunst
Beiträge zur Literaturwissenschaft 1: 15–30. https://doi.org/10.1553/spk45_1s15
Collot, Michel 2015. Landschaft. In: Dünne, Jörg; Mahler, Andreas (eds.), Handbuch
Literatur & Raum. Berlin: De Gruyter, 151–159.
Cosgrove, Denis; Della Dora, Veronica 2012. Introduction: High places. In: Cosgrove,
Denis; Della Dora, Veronica (eds.), High Places: Cultural Geographies of Mountains, Ice
and Science. London: Palgrave Macmillan, 1–18.
D’Angelo, Paolo 2001. Estetica della natura: bellezza naturale, paesaggio, arte ambientale.
Bari: Laterza.
D’Angelo, Paolo; Catucci, Stefano; Carchia, Gianni 1999. Dizionario di estetica. Bari:
Laterza.
Deleuze, Gilles; Guattari, Félix 1980. Mille Plateaux. Paris: Edition du Minuit.
De Martino, Ernesto 2002. La fine del mondo. Contributo all’analisi delle apocalissi culturali.
Torino: Einaudi.
Eco, Umberto 1985. Sugli specchi e altri saggi. Milano: Bompiani.
Greimas, Algirdas Julien 1970. Del senso. Milano: Bompiani.
Haraway, Donna J. 2016. Staying with the Trouble. Durham, London: Duke University Press.
https://doi.org/10.2307/j.ctv11cw25q
Haushofer, Marlene. 1985. Die Wand. Berlin: Ullstein Taschenbuch Verlag.
Haushofer, Marlene. 1999. The Wall. (Whiteside, Shaun, trans.) Berkeley: Cleis Press.
Hoffmann, Michael 2000. Verweigerte Idylle. Weiblichkeitskonzepte im Widerstreit
zwischen Robinsonade und Utopie: Marlen Haushofers Roman „Die Wand“. In:
Bosse, Anke; Ruthner, Clemens (eds.), „Eine geheime Schrift aus diesem Splitterwerk
enträtseln…“ Marlen Haushofers Werk im Kontext. Tübingen: Franke Verlag, 193–207.
Jakob, Michael 2005. Paesaggio e letteratura. Firenze: Olschki.
Jonas, Hans 1993. Il principio responsabilità. Torino: Einaudi.
Kalinova, Maria 2018. Exotopy: Mikhail Bakhtin and Jacques Lacan on the outside context
of discourse. Slavica Tergestina 1: 98–117.
Knecht, Maria-Regina 2007. Recollections of crime and complicity. Studies in 20th & 21st
Century Literature 31: 82–108. https://doi.org/10.4148/2334-4415.1645
398
Emanuela Ferragamo
Lehmann, Annette Jael 2021. Environments: Künste – Medien – Umwelt: Facetten der
künstlerischen Auseinandersetzung mit Landschaft und Natur. Bielefeld: Transcript.
Leone, Massimo 2015. Signatim. Profili di semiotica del testo. Roma: Aracne.
Lévinas, Emmanuel 1977[1961]. Totalità e infinito. Milano: Jaka Book.
Lotman, Yuri 2000. Universe of the Mind: A Semiotic Theory of Culture. (Shukman, Ann,
trans.) Bloomington: Indiana University Press.
Marchesini, Roberto 2002. Post-human: verso nuovi modelli di esistenza. Torino: Bollati
Boringhieri.
Meschiari, Matteo 2008. Sistemi selvaggi. Antropologia del paesaggio scritto. Palermo:
Sellerio.
Milani, Raffaele 2017. L’arte del paesaggio. Bologna: Il mulino.
Morton, Timothy 2011. Zero landscapes in the time of hyperobjects. Graz Architectural
Magazine 2011, 78–87.
Morton, Timothy 2013. Hyperobjects. Minneapolis: University of Minnesota Press.
Nelseen, Sarah 2019. Zwei weibliche Robinsonaden? Eine vergleichende Lektüre von
Marlen Haushofers „Die Wand“ und „Die Mansarde“. In: Alraud, Sylvie; Lacheny, Marc;
Lajarraige, Jacques; Leroy du Cardonnoy, Éric (eds.), Dekonstruktion der symbolischen
Ordnung bei Marlen Haushofer. Berlin: Frank & Timme, 127–143.
Ponzio, Augusto 2003. Tra semiotica e letteratura. Milano: Jaka Books.
Predoiu, Graziella 2016. Raumkonstellationen in Marlen Haushofers Roman „Die Wand“.
Germanistische Beiträge 36: 66–88.
Ricoeur, Paul 2004. Percorsi del riconoscimento. Milano: Raffaele Cortina Editore.
Ronchi, Rocco 2017. Il canone minore: verso una filosofia della natura. Milano: Feltrinelli.
Schama, Simon 1995. Landscape and Memory. New York: Alfred A. Knopf.
Simmel, Georg 2009[1985]. Philosophie der Landschaft. Stühlingen an der Wutach: Mahler
Verlag.
Sobral Campos, Isabel 2019. Ecopoetics and the Global Landscape: Critical Essays. Lanham,
London: Lexington Books.
Stephan, Inge; Weigel, Sigrid 1988. Die Verborgene Frau. Sechs Beiträge zu einer feministischen Literaturwissenschaft. Berlin: Literatur Verlag.
Starobinski, Jean 1957. L’occhio vivente. Torino: Einaudi.
Venske, Regula 1987. “Vielleicht, daß ein sehr entferntes Auge eine geheime Schrift aus
diesem Splitterwerk enträtseln könnte…”. Zur Kritik der Rezeption Marlen Haushofers.
In: Duden, Anna (ed.), „Oder war da manchmal noch etwas anderes?“ Texte zu Marlen
Haushofer. Frankfurt: Verlag Neue Kritik, 43–67.
Venske, Regula 1991. “Schriftstellerin mit der Seele eines Möbelpackers”: Marlen Haushofer.
In: Das Schreiben der Frauen in Österreich seit 1950. Wien: Böhlau Verlag, 22–32.
Vitta, Maurizio 2005. Il paesaggio. Una storia tra natura e architettura. Torino: Einaudi.
Von der Lühe, Irmela 1986. Erzählte Räume – leere Welt. Zu den Romanen Marlen
Haushofers. In: Duden, Anna (ed.), „Oder war da manchmal noch etwas anderes?“
Texte zu Marlen Haushofer. Frankfurt: Verlag Neue Kritik, 73–103.
Weber, Kurt H. 2010. Die literarische Landschaft. Berlin, New York: De Gruyter. https://
doi.org/10.1515/9783110227642
Zeyringer, Klaus 2001. Österreichische Literatur seit 1945. Überblicke. Einschnitte. Wegmarken. Innsbruck: Haymon Verlag.
Marlen Haushofer’s The Wall and the post-nuclear culture of the face
399
«Стена» Марлена Хаусхофера и пост-ядерная культура лица
Переплетение ландшафта и лица принадлежит к пространственной эпистемологии человека: как предположил Маттео Мескьяри, примитивные люди привыкли
ориентироваться в пространстве, распознавая образы лица. Размышляя над романом Марлена Хаусхофера «Стена», мы ставим под вопрос значение «лица пейзажа»
после воображаемого ядерного апокалипсиса, оставившего после себя лишь кошку,
корову, собаку, женщину и стену. Стена размывает границу между человеческим и
тем, что не является человеком. С точки зрения Роберто Маркесини, она создает
телесный ландшафт – гибрид внутренних и внешних ландшафтов, типичный для
постчеловеческого сознания. Наконец, кульминацией такого пейзажа становится
отказ от существовавшей до апокалипсиса культуры лица, когда лица уже не
являются средствами распознавания.
Marlen Haushoferi “Üksinda maailmas” (“Die Wand”) ja
tuumakatastroofijärgne näokultuur
Maastiku ning näo teineteisesse põimumine kuulub inimliku ruumiepistemoloogia juurde:
nagu on osutanud Matteo Meschiari, leidsid primitiivsed inimesed maastikul orientiire
näomustreid ära tundes. Artiklis mõtiskletakse Marlen Haushoferi romaani “Üksinda
maailmas” üle ning esitatakse küsimusi “maastiku näo” semantika kohta pärast kujuteldavat
tuumakatastroofi, millest jäävad järele kass, lehm, koer, naine ja müür. Müür ületab inimese
ja muu-kui-inimese vahelise piiri: Roberto Marchesini mõisteid kasutades loob see kehamaastiku – posthumaansele teadvusele omase sise- ja välismaastiku hübridiseerumise.
Selline maastik kulmineerub lõpuks apokalüpsise-eelsest näokultuurist loobumisega: näod
ei toimi enam äratundmisvahendina.
400
Bruno Surace
Sign Systems Studies 49(3/4), 2021, 400–417
Semiotics of the pornographic face:
From traditional porno to Beautiful Agony
Bruno Surace1
Abstract. Today’s pornography constitutes a semiotic laboratory capable of meticulously describing some characteristics of the cultures from which it comes and for
which it is intended. In it, the role of the face is preeminent and assumes relevance
both from a diegetic and a formal point of view. A face which makes itself a sign
and is articulated in a dialectic between the syntagmatic and the paradigmatic axis,
finding expression as an aspectual device, establishing a peculiar semiotic procedure
of absentia in praesentia, and highlighting an eminently enunciative dimension of
the textual genre. Thus a facial semiotics of pornography becomes to all effects a
cultural semiotics, which through the exploration of a transversal genre – both in its
mainstream and more niche actualizations – produces significant results in defining
how cultures of the face, including extrapornographic ones, delineate themselves.
The aim of this article is to verify this peculiar facial semiotics through a case
history that stretches from traditional to contemporary pornography, also analysing
the “facial pornography” website beautifulagony.com and the visual works of some
contemporary artists.
Keywords: semiotics of pornography; pornographic faces; erotic faces; face studies;
semiotics of the face
1. Introduction
There is a branch of contemporary pornography in which the censorship of the face
is frequent through a peculiar “veil system” (Leone, de Riedmatten, Stochita 2016)
effectuated with various techniques (pixellation, face off, masks, etc.). However, it
is undeniable that the face plays a fundamental role in a large part of (both early
and current) mainstream pornography, performing various functions.
First of all, it is a sort of ontological and phenomenological guarantee, which in
a sense certifies the humanity of the body dissected by the shots. The mechanical
nature of the pornographic sexual act, in fact, based on a standardized repetition
1
Department of Philosophy and Education Science, University of Turin (Unito), Turin,
Italy; e-mail address: b.surace@unito.it.
https://doi.org/10.12697/SSS.2021.49.3-4.08
Semiotics of the pornographic face: From traditional porno to Beautiful Agony
401
compulsion, requires the face of the actors as proof that they really exist, that they
are situated entities which are watching as well as being watched.2 This proof is
provided by the editing, through a well-codified grammar, which expertly indulges
on the faces of the actors, female and male, interspersing face shots with those of
other parts of the body to decrease the degree of impersonality and recompose
the fragmented body, also germinating a series of genre variations ranging from
facial, to gonzo point of view, to sub-genres such as that of ‘eye contact’ (basically
an ‘uninterrupted interpellation’).
Thus, we connect to the second function of the face: when present, it is de
facto the actualization of the emotional side of pornography. Porn faces are, in
terms of tensive semiotics, constantly contracted figures du corps (Fontanille 2004).
Prosthetic in their muscularity, exaggeratedly joyful or on the edge of suffering,
porn faces are semantically revealed in their enjoyment, in that “beautiful agony”
which lends its name to a “historical” viral site (beautifulagony.com), in which the
porn is paradoxically founded on the censorship not of the face, but of the rest of the
body. Its homepage contains a mosaic of thousands of faces in close-up, continuously
posted by users from 2004 to the present day, captured in the act of “la petite mort”
(as the site claims). The videos depict its various phases (preparation, achievement,
completion, exhaustion), while the thumbnails crystallize its lasting aspectuality,
eternally experienced in the facial contraction that sanctions the “facial climax”.3 It
is the aspectuality of the orgasmic face as mandatory proof of the authenticity of the
pornographic content on which an aesthetic universe hinges that makes the face,
“naked and singular” (Lévinas 1984; Ponzio 2007), the true fulcrum of the textual
genre. Its power transcends the genre as poster art in films such as Lars von Trier’s
Nymphomaniac (2013), but also in the black and white photographs of Lithuanian
Albert Pocej, in Argentinian Diego Beyro’s Orgasms series, and in the series that
cover the entire range of orgasmic aspectualities (inchoative, lasting, terminative) in
Brazilian Marcos Alberti’s O Project.4
2
In this sense we can undoubtedly identify parallels with the phenomenology of Deleuzian,
Merleau-Pontyian, and more recently Žižekian derivation. As regards Žižek, given the vastness
of his work, we will only mention the 1996 work coedited with Renata Salecl (Salecl, Žižek 1996),
the 2004 text dedicated to Deleuze (Žižek 2004a) and the text of the same year on the gaze, edited
in the Italian version by Damiano Cantone and Lorenzo Chiesa (Žižek 2004b) as particularly
suitable for those wishing to approach this article or the phenomenological philosophy of
the Slovenian scholar. As for the closer relationship between semiotics and psychology, the
fundamental reference, which also imbues the pages of this essay, is Barthes 1973.
3
Already called so by Elisabeth Lloyd (2005), who studied the expressions of stumptail
macaques in the wake of Suzanne Chevalier-Skolnikoff.
4
A semiotic reflection on the orgasmic body in traditional porn cinema can already be
found in da Silva 2013.
402
Bruno Surace
It is, in other words, a real face-based “cultural epidemiology”, which in the web
era mutates in content as in the video O Face, produced by DirtyFit and with over
three million views on YouTube, in which the aesthetics of beautifulagony.com is
transliterated from the rhetoric of amateurs to a professional production, or in the
series of viral videos Hysterical Literature made by the American photographer
Clayton Cubitt, where women filmed in the medium field in black and white read
famous poems while a vibrator is in action between their legs. The emphasis in all
the cases mentioned is always on the face as an ante litteram immersive device, the
unknown protagonist of pornography because of its semiotic power of empathic
production – if it is true that watching porn corresponds to an “entanglement of
somatic and semiotic intensities” (Voros 2014: 249)5 – which, on the one hand,
causes reflection of a semiotic and even anthropological nature6 and, on the other,
certainly pushes us to interrogate modern neuroscience.7 If you think about it, tout
se tient: while watching a film you cry not at the moment of the tragedy, but at the
moment in which you see the desperation on the faces of those who are living it
inside the narration.
2. The face in porn from the syntagmatic
to the paradigmatic axis8
The face in the erotic and pornographic context can be understood as a sign,
whose position in the semiosphere is not fixed, but stretched between the
peripheries and the centre, by virtue of epochs, genres, geographical locations,
gender of the protagonists, that is, within a broad cultural spectrum. In the porn
“of yesteryear”, for example, it occupies a syntagmatic, rather than paradigmatic,
role. An absence of the face is not substantially envisaged. It is implicated as a
sign in the specific construction of the pornographic text, and de facto constitutes
an aspectual marker that pre-determines the transition from the durative to the
terminative phase. In narrative terms, the close-up focusing on the face of the
male actor is a sine qua non for the staging of the “cumshot”, which, as already
claimed by Ortoleva (2009: 318), constitutes the pivotal moment of heterosexual
pornographic textuality, meaning pornography “as a ‘regime of representation’, a
5
Voros in turn refers to Paasonen 2007.
Think of the ethnographic study of the joyful face by Chen et al. 2018, or Fernández-Dols,
Carrera and Crivelli’s 2011 study on facial behaviour while experiencing sexual excitement.
7
Cf. Gallese, Guerra 2015; D’Aloia, Eugeni 2017.
8
The sense in which we will use the notions of the syntagmatic and the paradigmatic axes is
that traditionally shared by semiotics, which refers to Saussure 1916.
6
Semiotics of the pornographic face: From traditional porno to Beautiful Agony
403
genre assembled out of certain combinations of camera angles and lighting, bodily
postures, apparel, footwear, and the like” (Longhurst et al 2017: 306)9. The role
of the face is here inevitable, and provides the basis for a theoretical reflection
that goes beyond the context of hardcore. In fact, if on a cultural level the male
orgasm is given as authentic by definition, as not simulable (and therefore as an
act whose semioticity is doubtful)10, it is clear that the emotional-bodily imprint
of the face is essential to trigger engagement which otherwise, if relegated to the
exclusive profilmic emphasis on the genitals, would be less meaningful. Or at least
it is so in the pornography of the 1970s and 1980s, where the female face is equally
important, not only as a place where usually the reverse shot of the male face
is carried out in the final scene (so frequently that it ends up constituting not
just a stylistic feature, but a specific sub-genre, known as ‘facial’, which overflows
with compilations in the porn-tubes). The woman’s face, at least in mainstream
heterosexual pornography, is in turn the counterpart to the male face. Its task is
to signify enjoyment, which is expressed in specific muscular patterns, rotational
movements of the pupils, and in general an inventory of facial figures which are
the synecdoche of total relaxation, of an aesthetic of abandonment.
The role of the face in contemporary pornography is very different. To all intents
and purposes there still exist “old-fashioned” films, where the syntagmatics of the
montage implies specific moments when the face is central, which is counterintuitive
if one thinks of it as the only human surface apparently not of pornographic
significance, being an element that is not socially reprehensible (as are the genitals in
most human cultures). While the genitals are usually covered at least by underwear,
the face is instead the part of the body that is almost always visible. However, the
production of films in which the face is completely non-existent is a dominant trend
in contemporary pornography, in a strange semiotic inversion: what is usually hidden
(the private parts) is emphasized, and what is usually visible (the face) is hidden.
This new trend, which undermines professional pornography and sanctions the
ineluctable advance of amateur or “fake amateur” porn, is in fact all enacted on an
enhancement of anonymity,11 which coincides with a culture that translates porn
from a media experience that everyone can enjoy into a factual experience that
everyone can practise. It is legitimized porn par excellence, potentially practicable
even by the author who is writing these lines, in the total protection of a face that is
constantly off screen, or censored through analogue (varied and possible masking)
9
There is also a deepening of a philosemiotic approach in this sense in Kaite 1995.
It should, however, be noted that today this notion is more of an urban legend than anything
else, because in the pornographic industry there are numerous tricks to simulate male orgasm,
as already shown, for example, in Ziplow 1977.
11 A first analysis of anonymity as a semiotic strategy of occultation can be found in Leone 2018.
10
404
Bruno Surace
or digital (pixellation, blur effects and so on) cosmetic techniques. In erotic culture
the face thus assumes no longer a syntagmatic but a paradigmatic role: it can exist
or, as most often happens, not exist. However, and this is the relevant element, the
anonymity of this face is not only an individual protection strategy, but also an
operation with a significant symbolic charge.
If we take, for example, one of the thousands of semi-professional channels
on Pornhub (the principal mainstream porn-tube in the world), called ‘Nyna
Ferragni’ from the stage name of the Italian actress who animates it,12 we notice
from the thumbnails how the semiotic logic of face management configures a
relevant dynamic of absentia in praesentia. The woman’s body is clearly the central
protagonist (as the name of the channel indicates), although there are in fact almost
always two bodies that can be seen in alternation, hers and that of her partner.
In fact, it is a typical trend nowadays to open channels for couples, in which
people who are partners in real life re-propose their sexuality online, significantly
heightening the aura of authenticity around the sexual performance represented.
Usually, however, in profilmic terms these movies still identify the female character
as protagonist and the male as co-protagonist, a sort of prop, biologically necessary
but narratively secondary.13 So Nyna Ferragni is present in all the previews of
the videos, while her male co-star appears, to an exclusively genital extent, only
in some of them. His face is not visible, and is hidden for framing reasons (he is
behind the camera, which he holds in his hand in full gonzo style, or is otherwise
“sacrificed” to make room for her body). Instead, she is the protagonist, but her
face is not seen, and its presence is averted in the most imaginative ways: she may
be off screen, and therefore only her body is visible while her face is cut off by the
edges of the frame; she is wearing a hat and her long hair covers a major part of her
face; her face is censored by pixels; or she conceals her face with an arm, a pillow
and so on. It might not always be completely occulted: sometimes the lower portion
is visible, from the mouth to a piece of the nose; but you will certainly never see her
eyes. In short, there is an extremely coherent dialectic between the visible and the
invisible, which reifies the shift of the classic idea of “see-through”.14 Everything
12
At the time of writing these lines, in December 2020, the metadata of the profile refer to
181 million views of the videos it contains, 259 thousand subscribers, and position 41 in the
internal ranking of the site, which in fact is a highly relevant position considering that the site
embraces thousands and thousands of internal channels.
13 It should be noted that these considerations relate exclusively to heterosexual mainstream
pornography.
14 This discourse can obviously be part of a wider debate on online lifestyles as dynamics
in which peculiar relationships between opacity and transparency are implemented from an
identity point of view. See Thibault 2016.
Semiotics of the pornographic face: From traditional porno to Beautiful Agony
405
else about Nyna Ferragni is absolutely visible, exposed, overexposed; the face, on
the other hand, is always veiled, rarefied, impenetrable. If in traditional eroticism
everything hinges on the dialectic between concealment and modest revelation
of the forms of the body, here, instead, the absent present is the face, catalyst of
the attention of a new spectatorship whose sense of excitement is built around
an identity which has been halved, and which therefore through interpretative
cooperation is partially filled with a new order of desire.
If in earlier times the porn actress/actor was clearly identifiable as such, and
resided in a specific and symbolically inaccessible hyperuranion (so much so that
many films were based on the idea of “casting” in which ordinary people could
engage in sexual intercourse with these clearly recognizable stars),15 today, however,
s/he could be anyone, and so we fall within the paradigm of the “pornification of
everyday life”, which finds one of its maximum elements in the new culture of the
pornographic face, together with the predominance of amateurism, the explosion
of porn in public, and certainly also the burgeoning industry of live porn through
webcam.16 Therefore, if the contemporary face becomes rarefied in pornography,
it is legitimate to ask whether it has also abandoned its relevance. In reality, as we
are discovering, the opposite is true. The hidden face is a face that exists precisely
as absent or semi-present. Moreover, this was already true in early porn. When
the hidden face was not yet a defined filmic object, an aesthetic of the absent face
was already taking shape, as all pornography – first photographic, and in the era of
deepfake also in motion17 – was based on photomontage. This operation is nothing
more than the replacement of one face with another, that is, the affixing of a face on
a body that does not belong to it, thus coinciding with the passage from a virtuality
(the particular famous person who is considered sexy, whom you would like to
see naked in the act of a sexual performance) to an actuality (the realization of the
fantasy through the manipulation of images of faces forcibly displaced from their
body or belonging to a different one that satisfies the perceived aesthetic needs):
15
Many successful pornographic series are born from this narrative pretext, from the
historical castings of the Italian actress Eva Henger (A letto con Eva), who “auditioned”
amateurs who were not particularly handsome and had no script, so as to give the viewer the
idea that even s/he could enjoy that experience, up to more contemporary formats like Mea
Melone CHALLENGE!, Fuck a Fan and many others.
16
The shift from the production of content designed to be deferred to that of content
designed to be streamed live online is a rapidly growing fact, which has seen a boost during
the pandemic period. Pornography has not been exempt from this increase in interest in live
content, as evidenced by the increment in users on sites like OnlyFans. Cf. Lykousas, Casino.
Patsakis 2020.
17 An analysis on deepfake with a semiotic approach appears in McCosker and Wilken 2020.
406
Bruno Surace
Even before the look-alike video, the net overflows with evident photomontages
where the face of a determined personage is mounted onto the body of some
anonymous porn star, intent on performing the most varied pornographic acts.
[…] On the part of the broadcaster, the creation of pornographic photomontages
with stars as protagonists may be based on two end-goals: 1. Recreational: the
broadcaster elaborates the fake for the sole purpose of introducing it onto the
Web to trigger trolling activities or because he considers it an exhilarating way
of mocking the unwary. 2. Lucrative: the broadcaster elaborates the fake from a
desire to benefit from it in terms of visits to a particular web video (clickbaiting),
the purchase of products, subscriptions and the like. Whatever the broadcaster’s
motivation, the focus of our interest are the users. Excluding the community
of “recreational users,” who recognize the fake and enjoy themselves making it
circulate as such, there is certainly a slice of users who utilize these images because
they find them arousing. The interesting fact is that these images appear for the
most part manifestly fake. They consist either of photomontages recounting highly
improbable narrative situations (a famous actress in the middle of an orgy who
smiles dazzlingly at the camera) or very poorly-finished productions (where the
viewer can note a difference in color between face and body, a lack of proportion
and so on). (Surace 2019: 248)
Thus, the pornographic face allows us, above all, to undermine the idea of the face
as an excessively biological or natural device. While there is certainly a naturalistic
foundation in the movements and expressions of the face, which is difficult to
falsify, it is also true that the cultures of porn transform the face into a semiotic
device that acts on various layers. The first is a presential layer: the face is or is not
there, and its presence in the absence or vice versa is a semiotic fact; the second
is an actorial layer: the pornographic face, in the act, recites and is actorialized
in Greimasian terms, that is, it has semantic properties that articulate value and
aspectual dimensions, contributing to the development of a narrative that the user
converts patemic-bodily onto themselves (even seeking out and concentrating on
some salient moments and avoiding others, since for all intents and purposes a
porn film is almost never watched in its entirety);18 the third is the layer of the
post-produced face, which makes pornographic faces the places where the hidden
18
The principal mainstream pornographic sites are increasingly built around categorizations
and metadata, not only designed to direct users to the videos they might like most, but even
manipulating the videos themselves with various types of infographic tags, such as signs in the
playback bar that indicate different “scene types” (that is, the types of position in the narrative
grammar of pornographic performance), or even the histogram index of the attention curve on
a single video (so that the user will directly select the most viewed parts, rather than others).
Pornhub itself releases a complex report every year with many data, capable of closely tracing
user profiles: https://www.pornhub.com/insights/2019-year-in-review (last consulted on 14
December 2020).
Semiotics of the pornographic face: From traditional porno to Beautiful Agony
407
desires of the cultures to which they refer are most fulfilled, often dichotomously
(hyper-made-up faces vs “girl-next-door” faces, adult faces and therefore with
wrinkles in sight (MILF) vs innocent faces (teen), angelic faces vs aggressive faces,
and, again, an emphasis on the colour of hair, eyes, skin, etc). The face is therefore
configured as a completely culturalized semiotic and phenomenological device.
In this way, pornography allows us to verify the culturality of the face as claimed
by Broekman (2011: 24–25): “1. Human faces are not a natural fact but should
be understood as a cultural value; 2. A human face is in semiotic perspective a
body part that shares its artifice-character with the other parts of the body; 3. The
artifice-character plays a key role in the semiotic and legal need for name-giving;
4. Faces are thus faces in an age of semiotics”. In porn, consequently, the face is
rendered as a mask, and pornography as an effective laboratory in which to explore
desire as a socially shared, potentially divisive, politically relevant construct. If
this were not the case, then the role of the face in porn would not change, and the
pornography of the past would be similar to that of the present.
3. Aspectuality of pornographic faces
If we have so far highlighted how the face is not a contour element in pornographic
cultures, but a real dominant element, we now focus on those erotic textualities where
the centrality of the face is not only functional to the development of the “arousal
arc”, but proudly disclosed. The most relevant case is the aforementioned website
Beautiful Agony, whose subtitle is “Facettes de la petite mort”. Founded in 2004, still
active (which already demonstrates its success), and managed by Feck Pty Ltd in
Melbourne, the site presents itself with a template à la old-fashioned hypertext, with
a slender menu on the left and a central gallery in constant update of numbered
videos shared by users, called Agonees.19 The visual formatting of the videos follows
a specific protocol, strictly supervised by the moderators (who, moreover, have the
right to delete videos in which the simulation is too evident). The thumbnails are
again proof of visual expectancy: in a close-up from the shoulders up, men or women
are portrayed at the moment of maximum enjoyment.
As Dean MacCannell (1989: 155) states in Faking It: Comment on Face-Work
in Pornography, in sexuality “the dominant mode of communication is via facial
expression, gesture, and pre-linguistic vocalization”. If this is true both in everyday
sexuality and in pornography, there is a rather large facial difference between the two:
19
Reflections in part similar to those we will conduct on this site can be found in Schaschek
2014, who, however, defines the site as “quasi-pornographic”, while we are in all respects
convinced of its full “pornness”.
408
Bruno Surace
The pattern of opposition between pornographic and everyday interaction also
appears in “face-work”. Emotional extremes in ordinary a-sexual interaction are
usually signified by upper/lower face agreement or complementarity. For example,
excited joy-tending-toward-rapture in the framework of everyday interaction is
expressed by relaxed wide eyes and open mouth. Extremes of disappointment are
expressed by a facial “crunch”, the mouth and eyes close tightly and are pulled
toward the center of the face. In each case, there is a muscular agreement between
upper and lower, and left and right face. Expressions of emotional extremes in the
pornographic frame often involve upper and lower face muscular antagonism.
The eyes are open wide but the mouth is closed; sometimes the teeth are employed
to keep the mouth closed. Pornographic ecstasy is expressed by the opposite form
of opposition: the eyes are closed and the mouth is slack open. It is semantically
similar to a-sexual expression of pain. The difference seems to be that ecstatic
pornographic expression does not involve tight closure of the eyes or knitting of
the brows; the eyelids are merely dropped, and the forehead remains smooth. The
expression that suggests the actual experience of pain is inevitably accompanied
by a heavily knotted brow and tight wincing of the eyes. The pornographic frame
is often the site of spectacular whole-face muscular antagonisms that are not
found in other areas of life. (MacCannell 1989: 159)
Thus the faces on Beautiful Agony are real porn, to the extent that the muscular
oppositions between the top and the bottom of the face, which translate into
tensive dyscrasias (one hypertensive band, the other extended), are typical of
that exaltation of expressivity which, for example, in softcore or classic eroticism
(that, according to an ancient dichotomy, would constitute the aesthetically
worthy representation of the erotic as opposed to the alleged brutal obscenity of
pornography) is instead supplanted by mono-expressiveness, an “expressionless
state” (MacCannell 1989: 170). The face in the Australian site is also extreme,
above all because it supplants the nudity that common sense usually associates
with the term. The small portion of the body visible to us in the images and
videos is often clothed at least in underwear, if not completely. What is naked is
the face, abandoned to ecstasy and so magnified in its singularity. A singularity,
moreover, in which perhaps the glimmer of the truth lies, if it is true that the voice
can lie, and that “the first mirror is the smiling face of an observer, signifying a
semiotic process” (Makolkin 2015: 574). In addition, it would seem that in fact the
falsification of jouissance is actually easier vocally than facially:
A recent experiment studied the sounds women make during sex with their
male partners. It turns out that these sounds have nothing to do with whether
the woman is having an orgasm. The timing, frequency, and intensity of the
sounds were a response to the women’s male partners, and were interpreted quite
reasonably by the researchers as a way of encouraging the men to “hurry up and
Semiotics of the pornographic face: From traditional porno to Beautiful Agony
409
finish.” One, uncharitable way of interpreting such behavior, then, is a form of
deception: a “fake orgasm.” But that would be too harsh. It is really more like
Pavlov’s dinner bell, only that there is no intention of depriving the dog of his
dinner. (Yelle 2013: 58)20
Beautiful Agony thus deactivates the strong categorization of desire of mainstream
porn, sold according to packet switching methods that take place in the cornucopia
of labels of mainstream porn sites, where a semantic indicator is associated with
every possible micro-characteristic (characters, scenes, and so on). The centrality
of the face as a transcultural erotic device is in fact such as to reduce categorization
to a minimum choice, based on gender (Females, Males, Trans/Non Binary) and
on three further subgenres (Friends, Confessions, Revival) on which we will not
focus here.
There are two points worth expanding on. The first is the central role of
the visual representation of orgasm through the face. Again, the pornographic
context is useful for drawing general philosophical and semiotic conclusions. The
success of this site is indeed empirical proof of Emmanuel Levinas’ definition,
which otherwise risks remaining somewhat vague and unclear, of the face as a
naked and singular device (so singular that, except for a small minority, the videos
on the website are almost always of a single protagonist, usually intent on the
masturbatory act). What does Levinas actually mean? Our interpretation is that
he is talking about a threshold device, in which the clash between semiotic and
non-semiotic, that is, between natural and cultural, occurs.21 It is the idea of the
orgasm as a moment of abandonment, a “petite mort”, a paradoxical expression
that minimizes an otherwise absolute and incontrovertible moment – death –
by regimenting it in a transitory dimension that is otherwise excluded (you do
not usually die only for a while, except maybe in cases of people who report
near-death experiences). And this peculiar moment occurs vividly rather than
linguistically; it is not communicated by diegesis, but by mimesis. So the faces
20
In reality, there are also some doubts about the face, especially regarding studies on the socalled “climax face”, detailed in Lloyd 2005.
21 It is worth pointing out here that the theme of the presumed distinction between natural
and artificial signs, between the semiotic and non-semiotic world, is still the subject of debate
today, and indeed assumes renewed importance with the increasingly relevant dialogue with
disciplines of more recent formulation (such as neuroscience). The identification of a “lower
threshold of semiotics”, as it is designated in Eco 1975, is a complex problem that we have no
intention of addressing here. Our interpretation of Levinas in these terms is therefore to be
understood not from the standpoint of whoever emits the sign, but from that of the effect of
meaning on whoever receives it, experiencing a sensation of “naturalness” as opposed to a
previous “artificiality”.
410
Bruno Surace
on Beautiful Agony seem to emit meaning involuntarily, and this is constituted
as such when it is traced by the user of the website, who is in turn automatically
aware of this non-intentionality (whether it is simulated or not), which is the core
of the experienced excitement. That is, these faces are signs that stand for nothing
else – aliquid pro nihilo – placed in a median phase in which the loss of control
(simulated or not, this is not the point) causes muscular and ocular reactions.
These reactions are not fully “supervised”. More than surveillance of the self, the
faces from Beautiful Agony are in a state of souveillance,22 in the lower threshold of
semiotics. The abandonment of significance in semiotic terms is not the embracing
of insignificance,23 and perhaps it is not even appropriate to read a transcendent
device in the face. Everything is played out on a specific, textual immanence level,
within the frame. However, the face takes on meaning as the ultimate device, whose
subversive charge – a linguistic subversion that repudiates embarrassment and
repression24 – is the isotopy of Beautiful Agony, which shows us faces as they
should not be seen culturally. In some ways they are a visual profanity.
The second important point is represented by the Trans/Non Binary category.
The immediately significant data are the faces that we find within this category.
In fact, if we dwell on them, we very rarely identify faces which are visually Trans/
Non Binary (for example, faces with masculine features but made up in such a way
that the common imagination tends to associate them with female faces). In most
cases, indeed, these faces appear to us as sexually unambiguous. Now, of course,
the reason for the existence of this category therefore appears more relevant in
terms of identity, rather than visuality. Trans/Non Binary users of the site, in fact
(as well as other users interested in the category), presumably already have the
possibility of satisfying their sexual appetites in the Males and Females categories
from a visual point of view, according to their preferences. The correspondence
or not between gender and biological sex is not necessarily traceable on the face.
Furthermore, trans-sexuality in pornography is a genital fact, and in Beautiful
Agony the genitals are taboo.
Three consequences follow: the first is that the face is something that is given
visually, but in reality contains in a nutshell a substratum of axiological-ideological,
and therefore semiotic, components that go beyond its visualness; the second is
that the face as a synecdoche of the body in pornography presupposes a physicality,
22
We do not use this term here in the sense that is attributed to it in the debate on the contemporary panopticon from Mann 1998 (later consolidated in Mann 2004) onwards, but as a term
to describe a state of reduction of self-consciousness until reaching an a- or pre-semiotic condition.
23 And in fact, even the most recent semiotic studies are starting to question insignificance, as
is demonstrated by Leone 2019.
24 On the re-semantization of repression as a semio-political act, see Surace 2019.
Semiotics of the pornographic face: From traditional porno to Beautiful Agony
411
so that it is in fact taken for granted that in a “straight” porn film a female face
corresponds to female genitals, but this is not the case for a face categorized as Trans/
Non Binary, about which a regime of doubt remains (both male and female genitals
could correspond to a male or female face); the third is that the face claims, as an
increasingly transcultural device, the overcoming of an archaic binarism which,
although today beginning to disclose sparse visual manifestations (usually through
status symbols like rock stars or influencers who make themselves up specifically to
achieve an imaginary “gender fluidity” and disrupt the traditional visual dichotomy
of genders), is already assimilated on the level in which the face is mediated as a
discursive device, and therefore categorized. It is the distinction between seen face
and said face, which causes this category to exist in Beautiful Agony, meaning that
transsexuality does not necessarily pass through the face, but proudly finds its
confirmation in the face. In fact, therefore, those who prefer, for example, videos
#4065 or #4643, included in the Trans/Non Binary category but which from the
preview seem to depict people who could be included in the Females category, will
see in these the face of a non-binary sexuality. Behind the visual appearance of the
faces there is always a political, semiotic, identitary instance.
However, Beautiful Agony is not the only place where the pornography of
the face reaches its peak. If the face already plays a radical role in mainstream
pornography, here this is further consolidated in some specific sub-genres such as
gonzo, where the subjective point of view is magnified by the actors’ faces in the
foreground, and the frequent indulging in eye contact (what, in cinematographic
terms, is called interpellation). Yet the power of the face as an erotic device
had already been investigated in earlier times, when certain themes were still a
socially shared taboo, and all the specifics we are talking about had not even been
codified, if we take the 1970s as a moment of diffusion in the West of audiovisual
pornography derived from magazines.
In 1964 Andy Warhol made a silent, black-and-white medium-length film
entitled Blow Job. Warhol’s film could to all intents and purposes be understood
as a text in which a fellatio is shown and therefore, in fact, as a pornographic film.
In reality, however, the history of modern pornographic cinema (excluding the
experiments of early cinema) usually traces the genre’s first film to the slightly later
Deep Throat (Gerard Damiano 1972), although other films with simulated sexual
acts, without real penetration, had been shot previously. Why is Warhol’s film not
mentioned? Because, in the artist’s typical playful style, it betrays the expectations
raised by the title. The entire duration of the film is in fact a fixed sequence closeup of DeVeren Bookwalter (uncredited), wearing a jacket with a high collar, who
squirms with delight for several minutes, now grinding his mouth, now raising his
face and eyes skywards, now touching his hair and, finally, after appearing to pull
412
Bruno Surace
up his pants, lifting a cigarette to his lips. It is to all intents and purposes a film
in Beautiful Agony style, ante litteram. The operation is clearly provocative, and
plays on the way in which the title topicalizes the long shot. In fact, on the isotopic
level, at least until the moment of lighting the cigarette (which has a precise cultural
correspondence with the post-coital moment), the visual text does not necessarily
suggest a fellatio. The actor’s face is illuminated by a strong light, which often
overshadows his eyes, generating dark eye sockets through a play of chiaroscuro.
His movements are sometimes convulsive and sometimes more relaxed. The mouth
often showily quivers:
The brief, fleeting, and erratic moments of this failure in the face work of Blow
Job’s poser are located at the points where his face does, in fact, appear to index
unambiguous sexual pleasure via certain “genuine,” that is, nonstrategic facial
contortions. Correlatively, in the face work of porn performers, the failure to
screen the trauma can be located in those moments that indicate the subject’s
momentary loss of control over his or her own strategic signification […]. This
failure can take two forms: First, it may be registered as an expression that becomes
briefly yet unambiguously identified as “phoniness,” and, second, taking the same
effect as with Blow Job’s poser’s failure, it may be registered in an expression of
genuine (nonprofessional) sexual pleasure. (Grundmann 2003: 36)
This consideration ultimately demonstrates a certain visionarity on the part of
Warhol, who had at the same time foreseen the role of the face in contemporary
porn, but also somehow the kitsch substratum that would make the first “official”
pornographic films famous a few years later.
4. Artistic and enunciative cultures of the “O Face”
These theoretical reflections based on the face spring from the pornographic
context in such a vigorous way that they are also intercepted by the production
of artistic speeches and texts. A first important case is that of Albert Pocej, who
decided to immortalize with photographic shots, and sometimes with time-lapse,
15 women in the moment of orgasm, producing a series of images as a sort of
photographic exploration.25 The face is again central, crystallized in different
poses. It is, however, no longer a foreground face. From a plastic point of view,
the images are in black-and-white, as was Warhol’s film – halfway, in fact, between
25
The series can be viewed at the following link: https://www.boredpanda.com/femaleorgasms-photography-albert-pocej/?utm_source=ro.pinterest&utm_medium=referral&utm_
campaign=organic (last accessed on 14 December 2020).
Semiotics of the pornographic face: From traditional porno to Beautiful Agony
413
artistic and pornographic performance (first of all a “discursive pornography”, as
we have established). The black-and-white photography emphasizes the whimsical
character of the operation, giving it an aura of artistry, as if to signify a discourse
of a certain caliber. In topological terms, nevertheless, the faces are this time in
direct relationship with a background, which takes on a meaning, sometimes equal
(on the eidetic level the surface covered is quantitatively the same, half and half),
sometimes superordinate, as in the photograph in which the face of the woman
portrayed is at the bottom right, dominated by the background of some numbered
tree trunks, or in the shot where the woman has an orgasm against the backdrop
of a wind turbine. Particularly significant in this sort of laparoscopy of the female
orgasmic face is the photograph in which the dark background is abruptly
interrupted by a beam of light which, from above, almost divinely, strikes only
and exclusively the small face of the woman portrayed in her moment of ecstasy26.
There is thus no close-up in a technical sense, but the lighting arrangement
generates a conceptual close-up: the face in the moment of abandonment lights
up; everything else for a moment – the very instant of the shot – loses its meaning,
and in fact the background is black, almost unrecognizable.
Another case along the same lines is that of Diego Beyró’s Orgasms series, in
which a peculiar aesthetic of the surface intervenes.27 Here the face returns to
the foreground, but instead of photographs, we are dealing with oil paintings on
bedsheets, 210 × 130 cm in size, which by their conformation lend themselves to
being spread and waved, thus giving a certain airiness to the content represented,
fixed on the sheet, but at the same time mobile in the air. It is also relevant that
if before the face was first and foremost an index sign, photography being the
physical trace of an event that took place, here we move into the sphere of iconism,
mixed with the surface of the sheets which with their individual patterns (floral and
geometric motifs of different workmanship), and with the paradigm of iridescent
ripples according to where they are affixed, constantly re-signify these faces as
paradoxical expressions of la petite mort, always the same but also constantly
different. In the same vein, we can mention the work of Marcos Alberti, who with
his The O Project proposes a series of images that, starting from a neutral face in
the upper left corner, photographically recount the achievement of ecstasy through
a visual narrative of four foreground images, against a neutral background and
articulating a grammar of passion that is expressed in a dialectic between tension
and relaxation, both interior – the face that changes expression – and exterior –
26
On the role of the face in ecstasy, not only sexually, but in the broadest meaning of the term,
cf. Surace 2018.
27 The series can be viewed at the following link: http://www.diegobeyro.com/orgasm-series#1
(last accessed 14 December 2020).
414
Bruno Surace
the face that claims the right to its own impertinence, no longer respecting the
original system of proportions envisaged by the frame of the shot, and therefore
going out of view or changing its relationships with the depth of field and moving
towards the fore.28
This last consideration highlights a politics of enunciation to which the face
is usually subjected, depending on the format that regulates it (selfie, class photo,
passport photo, mirror, home film and so on), and that pushes us to consider
it a device that should not be read exclusively within its own borders, but also
in a conscious relationship with the environment that contains it, both in the
semiotic space of representation, and in that of everyday experience. Experiments
such as those reported above, starting from the controversial unhinging of the
taboos linked to sexuality and the specifics of orgasm as a moment of escape from
a strictly linguistic order of things (although, but this is the game of semiotic
analysis, “trapped in the linguistic cage of the text”), reveal how the investigation
in this specific direction can actually produce pertinent results for a more general
cultural semiotics of the face, whose media relevance appears, in conclusion, to be
exquisitely contemporary. It is in fact the liquid society of digital media that allows
the development of a pragmatics of the represented face that can move easily. In
the past, the orgasmic face was simply relegated not even to the margins, but in
the dark area of the unculture, that is, the unspeakable and the unrepresentable.
Today, however, in an era in which nudity is ordinary, pornography socially
metabolized, and sexuality no longer the subject of discussion as such but rather
a symbol of battles for the achievement of rights, the face takes on its own public
role as a privileged support for proposing a shared semiotics of reappropriation,
of redemption of impertinence.
Thus, on the interpretative level, there are two ways of reading the video O Face
presented by DirtyFit Apparel.29 On the one hand, it will certainly be agreed that
this is an advertising operation, conceived by the brand to involve the audience and
somehow secure a place within a specific imaginary, built around the enhancement
of certain libertarian, if not even socially anarchoid, values. On the other hand,
however, it will be inevitable to conclude that the sequence of faces of people
of every gender, ethnicity and age, captured in the foreground, sometimes with
emphasis on details (the trembling mouth, the widening nostrils and so on) during
their moment of ecstasy, constitutes the symptom and at the same time the result
of a strong cultural renegotiation, which passes through the face, of the elastic
28
The work can be viewed at the following link: https://www.marcosalberti.com/oproject
(last accessed on 14 December 2020).
29 The video can be viewed at the following link: https://www.youtube.com/watch?v=
mISaDN65eX8&has_ verified=1 (last accessed on 14 December 2020).
Semiotics of the pornographic face: From traditional porno to Beautiful Agony
415
distinction between private and public, between intimate and exhibitable, between
unbecoming and licit. In a semio-politics of bodies, the face obviously plays an
indispensable and definitive role.
Acknowledgements: This article is part of FACETS – Face Aesthetics in Contemporary
E-Technological Societies, a project funded by the ERC (European Research Council)
under the Horizon 2020 programme (Grant agreement 819649).
References
Barthes, Roland 1973. Le plaisir du texte. Paris: Seuil.
Broekman, Jan 2011. Faces face to face. In: Broekman, Jan M.; Mootz III, Francis J. (eds.),
The Semiotics of Law in Legal Education. London, New York: Springer, 21–56. https://
doi.org/10.1007/978-94-007-1341-3_2
Chen, Chaona; Crivelli, Carlos; Garrod, Oliver; Schyns, Philippe; Fernández-Dols, JoséMiguel; Jack, Rachael 2018. Distinct facial expressions represent pain and pleasure
across cultures. PNAS – Proceedings of the National Academy of Sciences of the United
States of America 115(43): E10013–E10021. https://doi.org/10.1073/pnas.1807862115
D’Aloia, Adriano; Eugeni, Ruggero 2017. Teorie del cinema. Il dibattito contemporaneo.
Milano: Raffaello Cortina.
Da Silva, Odair José Moreira 2013. A significação e o sentido do corpo orgástico no cinema
pornô. Signum: Estudos de Linguagem 16(1): 149–170. https://doi.org/10.5433/22374876.2013v16n1p149
Eco, Umberto 1975. Trattato di semiotica generale. Milano: Bompiani.
Fernández-Dols, José-Miguel; Carrera, Pilar; Crivelli, Carlos 2011. Facial behavior while
experiencing sexual excitement. Journal of Nonverbal Behavior 35(1): 63–71. https://
doi.org/10.1007/s10919-010-0097-7
Fontanille, Jacques 2004. Soma & séma: figures du corps. Paris: Maisonneuve & Larose.
Gallese, Vittorio; Guerra, Michele 2015. Lo schermo empatico. Cinema e neuroscienze.
Milano: Raffaello Cortina.
Grundmann, Roy 2003. Andy Warhol’s Blow Job. Philadelphia: Temple University Press.
Kaite, Berkeley 1995. Pornography and Difference. Bloomington, Indianapolis: Indiana
University Press.
Leone, Massimo 2018. I giga di Gige. L’impatto dell’anonimato nella comunicazione
contemporanea. RIFL/SFL – Rivista italiana di filosofia del linguaggio 1: 101–109.
Leone, Massimo 2019. On Insignificance: The Loss of Meaning in the Post-Material Age.
London, New York: Routledge. https://doi.org/10.4324/9780429461279
Leone, Massimo; De Riedmatten, Henri; Stochita, Victor (eds.) 2016. Il Sistema del velo.
Trasparenze e opacità nell’arte moderna e contemporanea. Roma: Aracne.
Lévinas, Emmanuel 1984. Paix et proximitè. Cahiers de la nuit surveillé 3: 339–346.
Lloyd, Elisabeth A. 2005. The Case of the Female Orgasm: Bias in the Science of Evolution.
Cambridge, London: Harvard University Press.
416
Bruno Surace
Longhurst, Brian; Smith, Greg; Bagnall, Gaynor; Crawford, Garry; Ogborn, Miles 2017.
Introducing Cultural Studies. London, New York: Routledge. https://doi.org/10.4324/
9781315690070
Lykousas, Nikolaos; Casino, Fran; Patsakis, Constantinos 2020. Inside the X-rated world of
“premium” social media accounts. In: Samin, Aref; Bontcheva, Kalina; Bragieri, Marko;
Dignum, Frank; Jannotti, Foska; Grisolia, Franchesko; Pedreschi, Dino (eds.), Social
Informatics. 12th International Conference, Pisa, Italy, October 6–9, 2020. Proceedings.
Cham: Springer, 181–191. https://doi.org/10.1007/978-3-030-60975-7_14
MacCannell, Dean 1989. Fakin it: Comment on face-work in pornography. American
Journal of Semiotics 6(4): 153–174. https://doi.org/10.5840/ajs19896410
Makolkin, Anna 2015. Face as a sign and Paolo Mantegazza’s theory of metoposcopy. In:
Trifonas, Peter Pericles (ed.), International Handbook of Semiotics. London, New York:
Springer, 573–588. https://doi.org/10.1007/978-94-017-9404-6_26
Mann, Steve 1998. Reflectionism and diffusionism. Leonardo 31(2): 93–102. https://doi.
org/10.2307/1576511
Mann, Steve 2004. “Sousveillance”: Inverse surveillance in multimedia imaging. MULTIMEDIA ‘04: Proceedings of the 12th annual ACM international conference on Multimedia,
620–627. https://doi.org/10.1145/1027527.1027673
McCosker, Anthony; Wilken, Rowan 2020. Automating Vision: The Social Impact of the
New Camera Consciuosness. London, New York: Routledge. https://doi.org/10.4324/
9780429341175
Ortoleva, Peppino 2009. Il secolo dei media. Riti, abitudini, mitologie. Milano: Il Saggiatore.
Paasonen, Susanna 2007. Strange bedfellows: Pornography, affect and feminist reading.
Feminist Theory 8(1): 43–57. https://doi.org/10.1177/1464700107074195
Ponzio, Augusto 2007. Fuori Luogo. L’esorbitante nella riproduzione dell’identico. Roma:
Meltemi.
Salecl, Renata; Žižek, Slavoj (eds.) 1996. Gaze and Voice as Love Objects. Durham, London:
Duke University Press. https://doi.org/10.2307/j.ctv11smjs5
Saussure, Ferdinand de. 1916. Cours de linguistique générale. Lausanne, Paris: Payot.
Schaschek, Sarah 2014. Pornography and Seriality: The Culture of Producing Pleasure. New
York: Palgrave MacMillan. https://doi.org/10.1057/9781137359384
Surace, Bruno 2018. Sintassi, semantica e pragmatica del martirio attorno a Martyrs di
Pascal Laugier. Lexia 31/32: 351–367.
Surace, Bruno 2019. The unbridled meaning of unsignified signifiers from paraliterature to
cinema. In: Idone Cassone, Vincenzo; Ponzo, Jenny; Thibault, Mattia (eds.), Languagescapes:
Ancient and Artificial Languages in Today’s Culture. Roma: Aracne, 131–143.
Surace, Bruno 2019. Where porndom meets stardom. In: Jacob, Frank (ed.), Pornography:
Interdisciplinary Perspectives. Berlin: Peter Lang, 239–260.
Thibault, Mattia 2016. Untag yourself. Opacità e trasparenze negli stili di vita online. Carte
semiotiche 4: 103–118.
Voros, Florian 2014. Gendered embodiment in audience reception practices of
pornography. In: Biasin, Enrico; Maina, Giovanna; Zecca, Federico (eds.), Porn after
Porn: Contemporary Alternative Pornographies. Milano, Udine: Mimesis.
Yelle, Robert 2013. Semiotics of Religion: Signs of the Sacred in History. London, New York:
Bloomsbury.
Semiotics of the pornographic face: From traditional porno to Beautiful Agony
417
Ziplow, Steven 1977. The Film Maker’s Guide to Pornography. New York: Drake Publishing.
Žižek, Slavoj 2004a. Organs Without Bodies: Deleuze and Consequences. New York, London:
Routledge.
Žižek, Slavoj 2004b. Dello sguardo e altri oggetti: saggi su cinema e psicoanalisi. Pasian di
Prato: Campanotto.
Семиотика порнографического лица:
от традиционного порно до Прекрасной Агонии
Современная порнография представляет собой семиотическую лабораторию,
способную скрупулезно описывать некоторые характеристики культур, в которых
она создается и для которых предназначена. Лицо играет здесь главенствующую
роль и приобретает особое значение как с диегетической, так и с формальной
точки зрения. Лицо, становящее знаком и артикулируемое в диалектике между
синтагматической и парадигматической осью, находит выражение в качестве
аспектуального приема, устанавливает своеобразную семиотическую процедуру
“absentia in praesentia” и подчеркивает крайне выразительное измерение текстового
жанра. Таким образом, в порнографии семиотика лица становится семиотикой
культуры во всех смыслах, которая благодаря исследованию этого трансверсального
жанра – как в его основном, так и в более нишевых актуализациях – дает значительные результаты в определении того, как культуры лица, включая непорнографические, себя определяют. Цель этой статьи – проверить эту гипотезу, прослеживая историю вопроса от традиционной до современной порнографии, а также
анализируя веб-сайт «порнографии лица» beautifulagony.com и работы некоторых
современных художников.
Pornograafilise näo semiootika:
traditsioonilisest pornost Kauni Agooniani
Tänapäeva pornograafia moodustab semiootilise laboratooriumi, mis võimaldab üksikasjaliselt kirjeldada mõningaid iseloomulikke jooni kultuuris, millest see on pärit ja
mille jaoks mõeldud. Näole kuulub selles väljapaistev roll ning see osutub oluliseks nii
diegeetilisest kui ka vormilisest vaatepunktist nähtuna. Nägu, mis muudab end märgiks ning mida väljendatakse süntagmaatilise ja paradigmaatilise telje dialektikas, leiab
väljenduse aspektuaalse võttena, mis kehtestab “absentia in praesentia” omapärase
semiootilise protseduuri ning rõhutab teksti žanri ülimalt enuntsiatiivset mõõdet. Seega
saab pornograafias näosemiootikast igas mõttes kultuurisemiootika, mis mitut valda
ületava žanri uurimisel annab olulisi tulemusi defineerimaks, kuidas näokultuurid,
sealhulgas ka pornograafiavälised, end määratlevad. Artikli eesmärgiks on kinnitada seda
omapärast näosemiootikat juhtumiajaloo abil, mis ulatub traditsioonilisest kuni kaasaegse
pornograafiani, analüüsides ka “näoporno” veebilehte beautifulagony.com ning mõnede
kaasaegsete kunstnike visuaalset loomingut.
418
Antonio Santangelo
Sign Systems Studies 49(3/4), 2021, 418–436
Culturally significant symbolic faces:
For a sociosemiotics of faces in films
Antonio Santangelo1
Abstract. Every now and then when watching a movie, we come across faces in
which we recognize a significant value, because they represent some important
cultural models we use to assign meaning to our experience of the world. By way
of example, I will discuss the faces of the protagonists of two recent films, Abdellatif
Kechiche’s La vie d’Adèle. Chapitres 1 & 2 (2013; English title Blue Is the Warmest
Colour) and Léonor Sérraille’s Jeune femme (2017), comparing them with the faces
of the protagonists of some older movies, such as Andrew Adamson and Vicky
Jenson’s Shrek (2001) and James Cameron’s Avatar (2009). I will argue that the way
in which the faces are portrayed is similar to the narrative structure of the stories
of the characters they belong to, and that the signs and narrative structures used to
construct the discourses about the world in those films are at the same time similar
to those of two important cultural models of what it means to be young men and
women in our times. As these cultural models are different, yet interconnected, I will
argue that the most meaningful faces in cinema change due to the transformation
of the cultural models they derive from and that a sociosemiotic method based
on a structuralist vision of culture can help identify the most culturally significant
symbolic faces on screen, and elsewhere.
Keywords: sociosemiotics; cultural models; narrative structures; structuralism;
symbolic faces
The faces of those who do not find meaning in our society
This article deals with developments in a research study which started in 2017 and
has already led to the publication of Santangelo 2017 and Santangelo 2021. It is
based on the idea that there are images of faces that take on great symbolic value
in a specific historical period, because their meaning depends on certain cultural
models that define what it entails to be men or women in that time, compared to
1
Department of Philosophy and Education Science, University of Turin (Unito), Italy;
e-mail: antonio.santangelo@unito.it.
https://doi.org/10.12697/SSS.2021.49.3-4.09
Culturally significant symbolic faces: For a sociosemiotics of faces in films
419
other cultural models which are embodied symbolically by different faces. Roland
Barthes had already noticed this in his analysis of today’s “myths”, as he focused
on the contrast between the faces of Greta Garbo and Audrey Hepburn in films
that circulated in the 1950s. According to the French semiologist, the images of
these two actresses admirably represent the moment in which cinema was about
to extract an existential beauty from an essential one (Barthes 1994[1957]: 64).
Just like Barthes, in this article I intend to consider the meaning of some
cinematic faces that embody cultural models relevant in contemporary society.
These are the faces of the leading actresses of the films La vie d’Adèle: Chapitres
1 & 2 (Kechiche 2013; English title Blue Is the Warmest Colour) and Jeune femme
(Sérraille 2017). I wish to demonstrate that the way the shots of their close-ups are
constructed – with a very mobile hand-held camera, frantically searching for the
focus of the image, finding it in the sensory organs of the two girls, with their eyes,
noses and mouths that “experience” the colour, smell and taste of the world, while
not being integrated visually into the surrounding context – resembles the overall
narrative structure of the stories of Adèle and Paula, the characters played by Adèle
Exarcopoulos and Laetitia Dosch. This narrative structure has been recognized by
the interpreters of these films as fundamental to understanding the meaning of
our lives today. This is why I wish to speak of culturally significant symbolic faces,
following a tradition in the field of film studies that, according to Casetti (1994:
28–32), stretches from Aristarco (1951) to Lukács (1963), authors who argue that
the so-called ‘impression of reality’ (Rastier 1986) elicited by a literary or cinematic
fictional text depends – among other things, but, in my view and in the view of
other scholars (Ferraro 2017; Gibson 2017), to a great extent2 – on its narrative
structure and on the way in which this narrative structure articulates the particular
types of signs that the text itself uses.
To support the claim, I need to pursue another objective as well, namely,
highlighting the characteristics of the sociosemiotic method on which the claim
itself is based. Contrary to Barthes, who had not considered it important to
demonstrate that his interpretation of the symbolic value of the faces of Greta
Garbo and Audrey Hepburn was in some way generalizable among the viewers
of the films in which the actresses appear, I believe it is essential to underline
the functioning of cultural models that, right from the start, manifest themselves
as codes of interpretation that do not belong only to me as a semiologist, but to
2
Enunciation logics are equally important, but will not be the focus of my argument. What
interests me here is understanding why the discourses on reality in a work of admittedly
fictional nature appear as truthful to their recipients as those in a factual newspaper article or
in a philosophical essay. It is in relation to this issue that their narrative structure proves to be
more important than their enunciation mechanisms.
420
Antonio Santangelo
a whole “community” of people who share the same vision of things. My task,
therefore, is to describe the mechanisms of signification of some works and some
images of faces that take on a symbolic value in a precise cultural context. To do
this, I will start from the interpretations of La vie d’Adèle: Chapitres 1 & 2 and Jeune
femme that have been offered by some Italian film critics3, showing the regularities
behind their apparent differences. My belief is that the former are the fruit of some
codes of collective value – the cultural models I have written about so far – which
serve as matrixes to give rise to some of the discourses on what it means to be
young today that circulate in our society.
Finally, to show how these cultural models work, and since the sociosemiotic
method that I will talk about in this article derives from structuralism and is thus
focused on finding similarities starting from differences, I too, like Barthes, will refer
to faces appearing in other movies. More precisely, I will work on Shrek (Adamson,
Jenson 2001) and Avatar (Cameron 2009). I will show that the faces of the main
characters of the later films come from another cultural model of what it means to
be a young man and woman. This will allow me to zoom in on the symbolic value
of the faces of Adèle and Paula. If the two girls are unable to integrate themselves
into, and find meaning in, our world, the other movies tell of individuals who do
become integrated but always find themselves in an “elsewhere”.
I will therefore advance a hypothesis similar to the research tradition that, in
the type of sociosemiotics I am interested in addressing (Fabbri 1973; Prieto 1975;
Landowski 1989; Ferraro 2001, 2015; Marrone 2001; Santangelo 2012), can be
traced back to Claude Lévi-Strauss (1964); that is to say that culture itself, in some
respects, has its own structure and the transformations in the way of talking about
reality that derive from it can be explained, at least retrospectively. What seems
evident and significant to me is that much has progressively changed compared
to the times when Propp (1928) identified a common constructive model and
characters in the tales of Russian folklore, who were always motivated to integrate
themselves into the “here”, that is, into the system of values of their own society
of origin, and at the end of their adventures, that often occurred in distant places,
became “transfigured”, assuming the features of the most representative men and
women among those who populated the world to which they themselves belonged.
The cultural models that give rise to our stories, as well as the symbolic faces of our
age, are decidedly different: a fact that may even appear worrying, in some ways.
3
I used the key phrases ‘Montparnasse femminile singolare reviews’ (Montparnasse. Femminile singolare is the title that was given to Jeune Femme for its distribution in Italy) and ‘La
vita di Adele reviews’ as search terms on Google, and then proceeded from the contents of the
first ten links that appeared.
Culturally significant symbolic faces: For a sociosemiotics of faces in films
421
Stories of wanderers
La vie d’Adèle: Chapitres 1 & 2 is a film by Abdellatif Kechiche, winner of the
Palme d’Or at the Cannes Film Festival in 2013. In Italy, some critics have written
about the film’s protagonist that, “exaggerating Flaubert, Adèle c’est nous”4, since
this young woman represents the “underlying problem of the solitude of the
contemporary individual, of the misery of the proposals for liberation and full
and profound self-affirmation that this era makes available”5. As concerns Paula,
the heroine of Jeune femme (recipient of the Un Certain Regard prize to the director
and screenwriter Léonor Serraille at the Cannes Film Festival in 2017), other
Italian film critics have argued that “she is really one of the most authentic female
characters of recent years”6, “as she symbolizes the cry of the new generations who
want to reclaim their place in the world”7 and “she is a matrix on which one could
place a face extracted at random from the lot of contemporary European youth”8.
As it is easy to see, the authors of these reviews seem to agree that the films are
very realistic in describing contemporaneity and that their protagonists resemble
all of us, or at least all the young people today. The reason for this is that both
films are about two girls who are, in a sense, wanderers. Adèle and Paula, in fact,
are represented while they are trying to become adults and find their own place
in society and their own personal fulfillment, but it seems that they do not really
know what they want and are simply trying to understand whether they like being
in one way or in another. They experiment on their own skin with the various
possibilities offered by the contexts in which they live, without ever finding their
true being. The two young women continuously and without stopping move between
heterosexual and homosexual relationships, humble and more prestigious jobs,
upper-class and petit-bourgeois apartments, relatives, friends, colleagues and lovers
from very different social backgrounds, in search of a sense of existence which
seems to elude them.
4
Federico Pedroni, www.cineforum.it/recensione/Melodramma_materialista (last accessed
20 February 2017).
5
Goffredo Fofi, www.foglianuova.wordpress.com/2013/10/27/goffredo–fofi–la–vita–di–Adèle/
(last accessed 20 February 2017).
6
Linda Magnoni, www.cineforum.it/recensione/Montparnasse-Femminile-singolare (last
accessed 25 August 2020).
7
Rolando Cisternino, www.anonimacinefili.it/2018/10/03/montparnasse-femminile-singolare/
(last accessed 25 August 2020).
8
Rudi Capra, www.ondacinema.it/film/recensione/montparnasse-femminile-singolare.html
(last accessed 25 August 2020).
422
Antonio Santangelo
In this regard, someone writes about the protagonist of Jeune Femme that
“undecided about her means, her affections, her ambitions, Paula is the mirror of a
generation that no longer identifies itself with the bourgeois ideals and aspirations
of the second half of the twentieth century [...], yet is too weak to propose an
alternative model”9. In her, however, we also find “a stubborn claim [...] of the
individual and personalizing factor, in a world that has reduced almost to zero
the possibilities of the unexpected, entrenching itself in a system of perfect gears.
Paula’s inadequacy is in fact the bearer of an adequacy of another type. That of
openness to others, the great missing element in these gloomy, mechanical times”10.
Her story, after all, begins in diametrically opposite circumstances when her fiancé,
a highly placed artist who is much older and more cultured than her, closes the
door in her face and throws her out of the house, without us knowing the reasons
why (later on we will understand that the two were too different and that he, while
portraying her in his works, could do nothing but observe and desire her as if she
were an object). The girl violently hits her head against the door trying regain
access, and wounds herself in the process, but to no avail.
So begins Paula’s long peregrination through Paris in search of her own
existential niche, among the sofas in rich strangers’ houses, in filthy hotel rooms,
old attics, the humble beds of colleagues in love with her, her mother’s house. In
all these places, and even more in the various work environments she enters, Paula
meets various people, who live according to the “mechanisms”, the social rules
of which the film reviews speak, and who propose that she become part of their
world, adapting herself to its logic. The heroine of Jeune Femme lives through and
experiences everything, following a narrative trajectory reminiscent of the flâneries
of the French Nouvelle Vague cinema, but there is a major difference, immediately
perceived by contemporary critics, who stress that, “compared to the films of the
young Turks, Séraille’s is the result of a script that is more written and less free, as
well as of a vision of the world in which the bewilderment of the protagonist and
her feeling lost in the non-sense of everyday life is not a declaration of anarchy
with respect to the rules but a way to become part of them”11.
The problem is that, in spite of her good will, Paula is absolutely unable to
integrate, so much so that her “adventures” come to an end, if one can say so, with
her abandoning her home and the work she has labouriously found. We first see
9
Rudi Capra, http://www.ondacinema.it/film/recensione/montparnasse-femminile-singolare.
html (last accessed 25 August 2020).
10 Massimiliano Schiavoni, https://quinlan.it/2018/05/27/montparnasse-femminile-singolare/
(last accessed 25 August 2020).
11 Carlo Cerofolini, http://icinemaniaci.blogspot.com/2018/04/Montparnasse-femminilesingolare.html (last accessed 25 August 2020).
Culturally significant symbolic faces: For a sociosemiotics of faces in films
423
her walking alone along an anonymous street, after the equally solitary decision to
have an abortion, thus giving up the child conceived with a colleague with whom
she evidently does not wish to share her existence. Finally, we look at her as she
closes the window of the attic she is leaving, surrounded by the walls that have kept
her “trapped” in the asphyxiating mechanisms of a society in which, despite trying
with all her might, she has not been able to “find a home”; in front of her, however,
she no longer has a door barred by others against which she could slam her head,
but a transparent pane of glass which she closes herself, through which she can
look out and on which she playfully blows, making it steam up.
A similar open ending links Jeune Femme to La vie d’Adèle: Chapitres 1 & 2.
Already the title of the film makes it clear that it tells of an unfinished part of a life
in progress. Adèle, too, is abruptly sent away from home by her partner, having
been caught cheating on her with a man. It is evident that the two girls, who meet
when very young and fall madly in love, subsequently grow apart as they get older,
due also to the differences in their respective social backgrounds and education.
In the “first chapter” of her life a teenage Adèle, daughter of petit-bourgeois
parents from the suburbs of Lille, thinks she can break all the rules, giving free
rein to her passion for the slightly older Emma who comes from a more affluent
family and is enrolled at the Academy of Fine Arts. Leaving Thomas, one of her
most attractive schoolmates much desired by her friends, Adèle devotes herself
completely to her partner, heedless of other people’s opinions. She goes to live
with her, supporting her and letting herself be portrayed in some paintings full
of sensuality, but in spite of everything she eventually disappoints her partner
with her decision to become a kindergarten teacher and discontinue her studies.
In the end, Emma begins to neglect her, as they do not share the same existential
horizons, and Adèle is unfaithful to her with a colleague whom she feels to be
more similar.
Subsequently, however, Adèle regrets the breakup and in the “second chapter” of
her life she tries to be reconciled with Emma in every possible way, until she goes to
the vernissage of Emma’s first solo exhibition. There all the paintings that represent
her are displayed on the walls, painful memories of a time gone forever. Emma
is now with another woman who is also part of the art scene. Adèle understands
that there is no way back and ends up going away on her own. Like Paula at the
end of Jeune Femme, we see her walking along an anonymous street in her city,
not knowing where to turn.
Even in the case of this film, the Italian film critics stress the protagonist’s great
openness, her desire to experience life without prejudice almost as if the story told
by Kechiche were superimposable on that told by Sérraille. At the same time, they
highlight the castrating nature of the frequently unjust social mechanisms that
424
Antonio Santangelo
cause suffering to today’s young people: those who, like Adèle, seek their own way
freely, outside the socio-cultural context from which they originate, find themselves
“wandering” among very different parts of the world, in which it is not possible
to settle definitively. We are encouraged to empathize with Adéle’s life, invited
to probe her dreams, desires and uncertainties, because “existence comes before
essence; that is, experience is not made comprehensible by a previous definition
of the self, but on the contrary, subjectivity is determined by the multiplicity of
experiences themselves”12. And yet, despite this attempt not to be defined a priori
by the rules of society, for several critics Adèle ends up experimenting with the
“misery of the proposals for liberation and full and profound self-assertion that
this era makes available”13, as is always the case in Kechiche’s cinema, all centred
on “social classes as regulators of an order that is difficult to negotiate; [...] and
on the tranches de vie [...] never understood as mere sections of everyday life, but
rather as emblematic segments of learning paths that do not produce evolutions”14.
The faces of today’s young people
In case of both movies critics recognize the crucial importance of the faces of the
protagonists and the way they are filmed. In Jeune Femme, for example, Paula’s
existential instability and continuous wandering between one niche and another
in society, without ever managing to integrate, are symbolically represented by
recurrent close-ups of the girl, made with the handheld camera technique, in
which her figure is clearly separated from the background, often highlighting her
two-tone eyes. In fact, the commentators emphasize how Paula “is almost always
shot alone”15, precisely to underline her difficulty in becoming integrated with
her surroundings, as can be seen in Fig. 1. It has been pointed out that Sérraille
directs her film, “centring the camera on the protagonist and accentuating her
discomfort and perplexity through serial jump cuts, sequences with a handheld
camera”16, which reach particular expressivity when they direct the spectator’s
12
Carlo Cerofolini, http://icinemaniaci.blogspot.com/2018/04/Montparnasse-femminilesingolare.html (last accessed 25 August 2020).
13 Goffredo Fofi, www.foglianuova.wordpress.com/2013/10/27/goffredo–fofi–la–vita–di–Adèle/
(last accessed 20 February 2017).
14 Michele Marangi, www.lindiceonline.com/l–indice/sommario/dicembre–2013/ (last
accessed 20 February 2017).
15 Linda Magnoni, www.cineforum.it/recensione/Montparnasse-Femminile-singolare (last
accessed 25 February 2020).
16 Rudi Capra, www.ondacinema.it/film/recensione/montparnasse-femminile-singolare.html
(last accessed 25 February 2020).
Culturally significant symbolic faces: For a sociosemiotics of faces in films
425
attention on Paula’s gaze, characterized by the detail of her “‘bipolar eyes’, mirror of
an equally bipolar and restless soul”17 (also seen in Fig. 1). It is this characteristic
of the face of the protagonist of Jeune Femme – which is obviously constructed to
communicate something, given that the eyes of the actress who plays the part are
actually brown – that symbolically represents her “wandering” spirit, her perpetual
movement among very different places and people. Since the girl’s story ends with
her choice not to settle down in any of the social contexts in which we have seen
her operating, but to continue to seek her own place in the world, critics cannot
help but observe that the film ends with a close-up of her face (the third image in
Fig. 1), which has been linked to the overall narrative structure of her story: “[…]
it will be the viewers who judge whether Paula’s future will be seen with brown or
blue eyes”18.
It is indeed this state of uncertainty, this continuous wandering – which also
causes her injuries, as when she cuts her forehead by slamming it against the door
that is closed in her face at the beginning of her story, or when she undergoes an
abortion at the end – that makes the protagonist of Jeune Femme such a realistic
and contemporary character, as underlined by those who say that her existential
path is definitely that of one who “finally sees and perceives the world with the
eyes and body of a woman, a real woman, who is not afraid to be what she feels
herself to be”19. As pointed out at the beginning of the article, this makes several
commentators say that Paula and her face could represent a “matrix” of the faces
of many young people today.
Figure 1.The face of Paula in the movie poster and in some scenes
17
Linda Magnoni, www.cineforum.it/recensione/Montparnasse-Femminile-singolare (last
accessed 25 February 2020).
18 Rolando Cisternino, www.anonimacinefili.it/2018/10/03/montparnasse-femminile-singolare
(last accessed 25 February 2020).
19 Rolando Cisternino, www.anonimacinefili.it/2018/10/03/montparnasse-femminile-singolare
(last accessed 25 February 2020).
426
Antonio Santangelo
Similar reflections are formulated by the reviewers of La vie d’Adèle: Chapitres
1 & 2, which in turn underline the importance of the emphasis placed by the
director on the many close-ups of the protagonist of his film, which keep her
character separate from the context in which she lives (as can be seen in some of
the images in Fig. 2) , despite the fact that this very context plays a decisive role in
directing the results of the girl’s story. This opinion is expressed, for example, by
those commentators who consider exalting Kechiche’s expertise “in depicting the
details of the faces and bodies, in dwelling on the close-ups, and then enlarging
the shots to other subjects, where strictly necessary, only when the others become
an attraction for Adèle’s gaze”20, for “there is nothing beyond what the young girl
knows and experiences”21: “more light, more details enter the field of vision, but
things in the background also become blurred”22.
Figure 2. The face of Adèle on the movie poster and in some scenes.
In this very personal exploration of the world and the search for something that
gives a definitive meaning to her existence, the images of Adèle’s body and face
become again fundamental: “since everything is a body, special attention is given to
how much the epidermis is animated by the desire for and contact with others”23;
“the camera, rarely placed far away from the characters’ faces, is a seismograph with
an immediate and icastic sensitivity; every change of mood is detected in real time”,
20
Giuseppe Gangi, www.ondacinema.it/film/recensione/vita_Adèle.html (last accessed 20
February 2017)
21 Ilaria De Pascalis, www.alfabeta2.it/2013/10/31/il–tempo–intenso–del–desiderio/ (last
accessed 20 February 2017).
22 Veronica Vituzzi, www.doppiozero.com/materiali/odeon/kechiche–la–vita–di–Adèle (last
accessed 20 February 2017).
23 Veronica Vituzzi, www.doppiozero.com/materiali/odeon/kechiche–la–vita–di–Adèle (last
accessed 20 February 2017).
Culturally significant symbolic faces: For a sociosemiotics of faces in films
427
the filmmaker catching it at the exact moment it manifests itself. The protagonist
of the film “is constantly filmed with close-ups or details that exalt her overflowing
physicality, her humoural fluids from tears to mucus, the sense of a warm and
deep instinctuality”24. In particular, her wandering among the different recesses
in society and the people who inhabit them is symbolized by her mouth, as can be
seen again in Fig. 2: “Adèle is voracious, as suggested by Kechiche’s constant return
to her lips as she sleeps, eats: lips stretched out, greedy, almost distinct from the rest
of her body, as if they lived their own lives and dragged the protagonist forward
against her will. Things make sense only through sense”25. After the two-tone
eyes and the swollen epidermis on Paula’s forehead in Jeune Femme, the sensory
organs of the face of the main character in La vie d’Adèle: Chapitres 1 & 2 once
again become the most representative element of the existential searching of today’s
young people, carried out almost gropingly, empirically, by trying everything out
on themselves.
Sociosemiotics and the meaning of certain faces
The reader who has had the patience to follow me so far might have wondered why
I decided to work on two films that were similar in many ways, and interpreted
more or less in the same way by reviewers. The reason is that, in doing so, I wanted
to show how the cultural models that are at the centre of interest in this article
work, paving the way for a reflection on the kind of sociosemiotics I wish to
discuss.
The cultural models underlying the audio-visual products circulating in the
media can be defined as the matrixes of “recurring discursive configurations,
governed by precise narrative structures, which hold together entire encyclopaedias
of signs” (Santangelo 2012: 120). In practice, we become aware of their existence
when, following a working method similar to that of Propp on the morphology of
the folk tale (Propp 1928), we understand that there is a common constructive logic
behind apparently different contents that is made up of the recurrence of certain
signs and narrative structures, as is shown in Fig. 3.
24
Michele Marangi, www.lindiceonline.com/l–indice/sommario/dicembre–2013/ (last
accessed 20 February 2017).
25 Veronica Vituzzi, www.doppiozero.com/materiali/odeon/kechiche–la–vita–di–Adèle (last
accessed 20 February 2017).
428
Antonio Santangelo
&XOWXUDOPRGHO
0RYLH
0RYLH
0RYLH1
Figure 3. The matrix of different films.
This immediately makes us reflect on the relationship of such contents with the
socio-cultural context in which they are produced, as Propp himself thought in his
work on the historical roots of the unitary compositional scheme of Russian fairy
tales (Propp 1946). Thus, this article is based on the idea that by identifying the
similarities between two films, such as La vie d’Adèle: Chapitres 1 & 2 and Jeune
femme, which do not refer to each other, yet seem clearly connected; it can be
assumed that their close relationship is not determined by chance, but by the fact
that both derive from a certain model of common thinking about who are today’s
young people, how they try to find meaning in contemporary society, and how
their stories should be told.
This hypothesis can be empirically substantiated with further evidence. Reviews
of the films can be used, which are, after all, interpretations of them provided by
individuals who belong to the same cultural context. In fact, the other idea on
which the method of this article is based is that the cultural models that give rise
to the audio-visual products circulating in the media are a sort of communication
codes that allow those who make them share their thoughts with the audience who
use the same logic in order to recognize the meaning of what they are told. The
cultural models, therefore, are also a certain kind of pertinence principles (Prieto
1975), which guide the recipients of cinematographic works within them, allowing
them to select and emphasize those content items that in their view determine
the meaning of films more than others. What the reviewers find significant in the
films, if repeatedly recognized as relevant by many of them, can be employed as a
useful guide to identify those recurring discursive configurations that are the most
obvious manifestation of the cultural models they find in the films themselves.
Otherwise, as is shown in Fig. 4, reviews can serve as empirical confirmation of
the interpretations of semiologists in their analysis of film works, indicating that
the cultural models they identify are the same as those used by a certain category
of their recipients to assign meaning to them.
Culturally significant symbolic faces: For a sociosemiotics of faces in films
429
&XOWXUDOPRGHO
FRGH
&XOWXUDOPRGHO
FRGH
0RYLH
$XWKRU
0RYLH
,QWHUSUHWHU
0RYLH1
Figure 4. Cultural models as tools for encoding and decoding films.
The roots of this way of conceiving sociosemiotics date back to Saussure’s thinking.
In fact, a semiotics understood as a social science of signification should try to
reconstruct the principles of the functioning of langue (Saussure 2001[1922]: 23) –
or, to put it more simply, of the code – which gives rise to the communicative
exchanges that take place between people in a given context. Only in this way
is it possible to pursue the objective of describing the meaning the semiologists’
objects of investigation assume not so much in their own eyes, but in those of the
other interpreters, in the society in which these objects circulate and are used. The
langue is the instrument through which different subjects who wish to express the
same concept, even in the idiosyncrasy of their personal acts of parole (Saussure
2001[1922]: 23), craft messages that are similar in some ways. Yet at the same time
the langue is the medium used by their recipients, when the latter have to recognize
that what is communicated to them is not simply the result of a purely individual
choice or behaviour, but the attempt to reproduce a model with a collective value.
This appears clear already in Saussure’s (2001[1922]: 145) famous example
about the role of language in allowing each of us to write a letter ‘T’ on a sheet of
paper in our own handwriting in the reasonable hope of being understood or, to
the contrary, about how language itself allows us to understand that the infinite
ways in which we can find the same letter written, depending on who penned it, are
to be traced back to the same meaning. This is also what happens with the cultural
models I am discussing in this article: the directors of La vie d’Adèle: Chapitres 1 &
2 and Jeune femme use them, metaphorically, to create their own personal versions
of the letter ‘T’ of the code for reading the existential experience of today’s young
people, and Italian film critics use them to decode the messages addressed to them.
Of course, conceiving of culture as a language or as a unitary code is problematic (Volli 2010) and this is not my intention, but empirical work of the type
I am presenting here shows that within culture itself there are indeed common
430
Antonio Santangelo
matrixes of the discourses that people perform about specific issues. To describe
these matrixes it is necessary to focus on their narrative structures and on the
signs that are articulated within them. In both cases, as Ferraro (2001) shows,
it is important to return to the method used by Lévi-Strauss in his analyses of
myths all over the world, once again reconnecting to Saussure. That is, as far as
signs are concerned, it is necessary to identify in the material discourses to which
cultural models give rise, a series of recurring elements which, although apparently
dissimilar, are united by the fact that they are made up of a number of common
significant characteristics, which produce the same meanings (Saussure 2001[1922]:
84–85)26. In the same way, as far as narrative structures are concerned, some
regular strategies of articulation of these signs must be highlighted which, once
identified under the surface of contents that seem to be different, can be recognized
as significant structures that convey the same meanings.
In the light of the previous paragraphs, in La vie d’Adèle: Chapitres 1 & 2
and in Jeune femme, the most relevant signs among those used in these works
to construct and interpret the discourses on young people today are those of
separation, movement, reliance on the senses, pleasure and, above all, suffering. The
two films, in fact, are full of images of the two protagonists framed in such a way
that their figures are visually separated from the background of the contexts in
which they live, just as there are numerous shots such as those of Adèle on the
sidelines of the worldly occasions when Emma socializes with her friends in chic
art circles, or others in which Paula is seen in her small attic in the same building
in which her landlady and employer occupies the main floors. However, as we
know, the two girls move between social environments in search of a place where
to find a meaning to their existence and the cameras follow them with handheld
mobile shots so as to communicate the idea of dynamism and, at the same time,
the precariousness that all this moving, all this research entails. By keeping close to
the protagonists, looking at their faces and bodies for signs that, in real time, show
what they feel in their attempts to give meaning to their lives, the directors of La vie
d’Adèle: Chapitres 1 & 2 and Jeune femme end up focusing on their sensory organs
and on how their skin, eyes, noses and mouths react to everything in this sort of
encounter/clash with reality. In some brief moments of their stories, Adèle and
Paula feel pleasure – the Sapphic sex sequences in Kechiche’s film are particularly
famous in this regard, as are those showing the protagonist enjoying pasta with
sauce and oysters, the foodstuffs that symbolize the two different social contexts
from which she and Emma come –, but most of the time the two girls suffer, cry,
hurt, because not finding their place in the world is painful.
26
Of course, Saussure talks about ‘signifiers’ and ‘signifieds’ of signs.
Culturally significant symbolic faces: For a sociosemiotics of faces in films
431
All these signs which are depicted in the faces of the main characters of the
films I am analysing here, and recognized by film critics as fundamental for the
reading of the faces of today’s young people, symbolically recall the common
narrative structure of La vie d’Adèle: Chapitres 1 & 2 and Jeune femme, which they
literally resemble, embodying its different facets, as can be seen in Fig. 5.
6HQGLQJLQVWDQFH
6HQGLQJLQVWDQFH
6HQGLQJLQVWDQFH
1
3HUVSHFWLYHLQVWDQFH
Figure 5. The narrative structure of the two films.
In order to understand the scheme in Fig. 5, it is necessary to point out that,
according to Ferraro (2015: 71–100), the architecture of stories which, like those
of the two films in question, mainly talk about the identity of their characters, is
based on the relationship between a perspective instance and a sending instance.
The former has to do with the sphere of the individual, his or her values, his
or her perspective on things, and is embodied by the protagonist of the events
narrated. The latter, on the other hand, concerns the world outside the protagonists
themselves and can be represented by the laws or conventions of the society in
which they live, or by another character’s way of seeing things, a place, a machine
or anything that, focusing on its own values, could or would like to impose a
certain destiny on them. The meaning of the narratives that are based on these
two instances depends on the relationship that is established between them, a
relationship that, after all, has to do with the values these kinds of stories become
bearers of as noted by Greimas (1996[1970]: 167–194). The identity of the elements
that make up the two instances, whether they are characters, things, environments
or eras in which everything is set, depends precisely on these values and on how
they relate to one another, whether they resemble or differentiate themselves,
become integrated or remain distinct.
As can be seen in Fig. 5, in the case of La vie d’Adèle: Chapitres 1 & 2 and Jeune
femme, the perspective instance represented by Adèle and Paula, with their values
based on extreme freedom, on the desire to be exactly as they want to be – high
432
Antonio Santangelo
or petit-bourgeois, heterosexual or homosexual, lovers of established artists or of
humble workers – clashes with the various “classist” sending instances in society,
which would like to define them in one way or another. The reality in which the
two girls live is, in fact, made up of strictly separate niches, each hinging on values
that, in the end, prove to be different from those of the young girls themselves.
Therefore, although they move continuously between the various social contexts
in which they try to insert themselves (indicated by the arrows in Fig. 5), also
feeling the advantages, the “pleasures” of what it means to be welcomed there for
a moment, they end up painfully remaining cut off from these, as if there were a
sort of barrier that prevents them from being fully integrated (this is the meaning
of the horizontal line in Fig. 5). The faces of Adèle and Paula, like those of the
young people they seem to resemble, are the faces of persons who, believing they
can affirm their demand for absolute freedom, realize that this, unfortunately, will
not allow them to find a place in the world, because the world works in a different
way, perpetuating “instances” that are decidedly less open to experimentation and
social mobility than the girls and the perspective instances they represent.
What I have just said allows me to address the last question of method that I
consider relevant, to reflect on how to determine, in the sociosemiotic paradigm
to which I refer, the symbolic meaning of a face. The cultural model from which
a face derives takes on its own value by difference, compared to other cultural
models whose constituent signs and narrative structure are clearly opposed to its
own, according to a logic that leads us to hypothesize that all these matrixes of the
various stories circulating in our society are connected in some way: as Saussure
states, after all, the numerous different ways of writing the letter ‘T’ are linked by
the fact that they are different from the way an ‘I’ or an ‘L’ is drawn.
In this sense, as I have already pointed out, it is evident that La vie d’Adèle:
Chapitres 1 & 2 and Jeune femme are characterized by an ending devoid of the
typical positive conclusion, which is easily found in those stories in which the
perspective and sending instances somehow meet. As can be seen in Fig. 6, this is
the case, for example, in the structure of the Russian fairy tales of magic analysed
by Propp. Ferraro (2015: 79–85), on the basis of the reflections of Propp, who had
highlighted their similarity with the initiation rites of different peoples (Propp
1946), defines them as forms of narrative that speak of the figure of the typical
“self-made man”, an individual whose values coincide with those of the society in
which he lives and who, at the end of his “adventures”, proves that he deserves to
settle there. In fact, after having faced many tests in an “elsewhere” where different
rules and values apply, the protagonist of this kind of storytelling shows that he has
been able to fight to realize the rules of the world to which he has always belonged,
transfiguring himself and assuming the likeness – and of course also the face – of
Culturally significant symbolic faces: For a sociosemiotics of faces in films
433
the best of the representatives of this social context, so much so that he often gets
to marry “the king’s daughter”.
6HQGLQJLQVWDQFH
3HUVSHFWLYHLQVWDQFH
Figure 6. The narrative structure of Russian fairy tales.
However, Russian fairy tales evidently derive from other cultural models of what
it means to be young and to become adults. Today we tell ourselves stories like
those of Shrek or Avatar, whose narrative structure appears in Fig. 7. The respective
protagonists are different from Propp’s heroes because, after having elsewhere
faced the trials that would allow them to integrate themselves into their respective
societies of origin, they realize that these involve an unjust sending instance on
which they do not intend to make their identity depend. On the contrary, it is in
the other “worlds” that they meet the sending instances whose values allow them
to find themselves and for this reason they decide to settle there, transfiguring
themselves, but symbolically taking on the appearance of ogres and aliens.
Shrek tells the tale of a princess who is the victim of a curse. She can choose
whether to love a prince animated by the worst consumerist values of her (and our)
society of belonging, “chaining herself ” to its rules and thus becoming a beautiful
woman admired by all, or to live forever free to be herself in a swamp with an ogre
and other fairy tale characters, while becoming “ugly” and “different” like them
in turn. In another case, an American soldier, thanks to a machine developed on
planet Earth, “moves” into the body of an alien and into the extra-terrestrial world
of planet Pandora to infiltrate its indigenous inhabitants and help his fellow human
beings appropriate their precious natural resources. After realizing, however, that
the Na’vi society is better than the capitalist society from which he comes, because
it respects nature and promotes brotherhood among all living beings, he decides
to betray his origins and defend the extra-terrestrials, ending up abandoning his
human body forever.
434
Antonio Santangelo
6HQGLQJLQVWDQFH
3HUVSHFWLYHLQVWDQFH
6HQGLQJLQVWDQFH
Figure 7. The narrative structure of Shrek and Avatar.
Evidently, Shrek and Avatar derive from a cultural model that overturns that
of the fairy tales studied by Propp. If we compare the stories and faces of the
characters of these films, and if we juxtapose them with those of Adèle in La vie
d’Adèle: Chapitres 1 & 2 and Paula in Jeune femme, we realize that their differences
still seem to connect them to each other. It could be hypothesized that there is
a logic of diachronic transformation, which gradually leads from Propp’s unitary
compositional scheme to the matrixes of contemporary works, passing from events
set in just societies which manage to integrate their young people, offering them a
sharable existential horizon, to tales of unjust societies from which young people
want to escape, to settle in a better “elsewhere”. Today, however, we tell ourselves
that this elsewhere no longer exists and that all that has been left to us is wandering
around with no precise destination, in search of a meaning and of an identity that
we are unlikely to find. This theory would certainly need further analysis, but if
it were confirmed, it would reveal once again that we are living in difficult times.
Acknowledgements: This research is part of a project that has received funding from the
European Research Council (ERC) under the European Union’s Horizon 2020 research
and innovation programme (Grant agreement No 819649 – FACETS.
References
Aristarco, Guido 1951. Storia delle teoriche del film. Torino: Einaudi.
Barthes, Roland 1994[1957]. Miti d’oggi. Torino: Einaudi.
Fabbri, Paolo 1973. Le comunicazioni di massa in Italia: sguardo semiotico e malocchio
della sociologia. VS. Quaderni di studi semiotici 5: 57–109.
Casetti, Francesco 1994. Teorie del cinema. 1945–1990. Milano: Bompiani.
Ferraro, Guido 2001. Il linguaggio del mito. Roma: Meltemi.
Ferraro, Guido 2015. Teorie della narrazione. Dai racconti tradizionali all’odierno storytelling.
Roma: Carocci.
Ferraro, Guido 2017. Senso, forma, nomi e referenza. In: Ferraro, Guido; Santangelo,
Antonio (eds.), Narrazione e realtà. Il senso degli eventi. Roma: Aracne, 117–136.
Culturally significant symbolic faces: For a sociosemiotics of faces in films
435
Gibson, John 2017. Sulla produzione del senso. In: Ferraro, Guido; Santangelo, Antonio
(eds.), Narrazione e realtà. Il senso degli eventi. Roma: Aracne, 97–116.
Greimas, Algirdas J. 1996[1970]. Del senso. Milano: Bompiani.
Landowski, Eric 1989. La société réfléchie. Paris: Éditions du Seuil.
Lévi-Strauss, Claude 1964. Anthropologie structurale. Paris: Librairie Plon.
Lukács, Georg 1963. Ästhetik I. Die Eigenart des Ästhetischen. Berlin-Spandau: Hermann
Lutherhand.
Marrone, Gianfranco 2001. Corpi sociali. Processi comunicativi e semiotica del testo. Torino:
Einaudi.
Prieto, Luis 1975. Pertinence et pratique. Paris: Minuit.
Propp, Vladimir J. 1928. Morfologiya skazki. [The Morphology of the Folktale.] St.
Petersburg: Academia.
Propp, Vladimir J. 1946. Istoricheskie korni volshebnoj skazki. [The Historical Roots of the
Fairy Tale.] St. Petersburg: Izdatel’stvo leningradskogo gosudarstvennogo universiteta.
Rastier, Francois 1986. Impression référentielle. In: Greimas, Algirdas; Courtés, Joseph
(eds.), Sémiotique. Dictionnaire raisonné de la théorie du langage. Paris: Hachette.
Santangelo, Antonio 2012. Sociosemiotica dell’audiovisivo. Roma: Aracne.
Santangelo, Antonio 2017. Fuori dal testo c’è salvezza. Il senso del film La vita di Adèle. In:
Ferraro, Guido; Santangelo, Antonio (eds.), I sensi del testo. Percorsi interpretativi tra
la superficie e il profondo. Roma: Aracne, 41–68.
Santangelo, Antonio 2021. Volti simbolici. Per una teoria sociosemiotica del volto. Lexia
37/38: 503–520.
Saussure, Ferdinand de 2001[1922] Corso di linguistica generale. Roma: Laterza.
Volli, Ugo 2010. Al di là delle culture. Le strategie della memoria. Lexia 5/6: 27–40.
Filmography
Adamson, Andrew; Jenson, Vicky 2001. Shrek. Glendale: Dreamworks Animation.
Cameron, James 2009. Avatar. Los Angeles: Twentieth Century-Fox.
Kechiche, Abdellatif 2013. La vie d’Adèle: Chapitres 1 & 2. Paris: Wild Bunch.
Sérraille, Léonor 2017. Jeune femme. Paris: Blue Monday Productions.
Культурно значимые символические лица:
социосемиотика лиц в фильмах
Время от времени при просмотре фильмов мы сталкиваемся с лицами, которым
присваиваем особую ценность, так как они отражают важные культурные модели,
придающие смысл нашему мироощущению. В качестве примеров я рассматриваю
лица главных героев двух недавних фильмов, «Жизнь Адель» Абделлатифа
Кечиче (2013) и «Молодая женщина» Леонора Серрейля (2017), и сравниваю их
с лицами главных героев некоторых более старых фильмов, таких как «Шрек»
Эндрю Адамсона и Вики Дженсон (2001) и «Аватар» Джеймса Кэмерона (2009). Я
436
Antonio Santangelo
утверждаю, что способ изображения лица похож на повествовательную структуру
истории соответствующего персонажа, и что знаки и повествовательные структуры,
используемые для построения дискурсов о мире в этих фильмах, аналогичны
структурам двух важных культурных моделей – что значит быть молодыми
мужчинами и женщинами в наше время. Эти культурные модели различны, но
взаимосвязаны, и я полагаю, что наиболее значимые лица в кино меняются из-за
трансформации эти культурных моделей. Социосемиотический метод, основанный
на структуралистском подходе к культуре, может помочь идентифицировать
наиболее значимые в культурном отношении символические лица на экране и в
других местах.
Kultuuriliselt tähenduslikud sümboolsed näod:
nägude sotsiosemiootika filmides
Filme vaadates kohtame tihti nägusid, milles tunneme ära märgatava väärtuse, sest need
esindavad olulisi kultuurilisi mudeleid, mida me kasutame oma maailmakogemusele
tähenduse omistamisel. Näitena käsitlen peategelaste nägusid kahes hiljuti valminud filmis:
Abdellatif Kechiche’i linateoses “La vie d’Adèle. Chapitres 1 & 2” (2013) and Léonor Sérraille’
filmis “Jeune femme” (2017), võrreldes neid peategelaste nägudega sellistes vanemates
filmides nagu Andrew Adamsoni ja Vicky Jensoni “Shrek” (2001) ning James Cameroni
“Avatar” (2009). Väidan, et see, kuidas nägusid kujutatakse, sarnaneb filmitegelaste lugude
narratiivse struktuuriga, ja et märgid ning narratiivsed struktruurid, mida nendes filmides
kasutatakse maailma puudutavate diskursuste konstrueerimisel, sarnanevad ühtaegu
kahele olulise kultuurimudeliga, mis käsitlevad seda, kuidas olla tänapäeval noor naine
või mees. Et need kultuurimudelid on erinevad, kuid omavahel seotud, väidan, et kõige
tähendusrikkamad näod kinos muutuvad tänu nende kultuurimudelite muutumisele,
millest nad pärinevad, ja strukturalistlikul kultuurivaatel põhinev sotsiosemiootiline
meetod võib aidata kindlaks teha kultuuriliselt kõige olulisemaid sümbooseid nägusid nii
ekraanil kui ka mujal.
Art, face and breathscape:
From
air to49(3/4),
cultural2021,
texts437–462
437
Sign Systems
Studies
Art, face and breathscape:
From air to cultural texts
Silvia Barbotto1
Abstract. We consider breath as a vast prospect that includes actions and traces
of them, that builds images and texts, that involves the human being and the
extra-human context; we call this great scenery ‘breathscape’. We then study how
breathscape interacts with the human apparatus of the face, both giving rise to
signs, but also giving rise to a liminal zone of extremely intriguing interpretative
processes on a mereological scale. How and where do the territory of breath and
the body interact? Which processes of signification do they give rise to? And which
signs are created in their phenomenal and semantic encounter? Art is certainly
the most appropriate language for studying this process, as well as for letting
opacities emerge and exploring outstanding contrasts. There are various concepts
of ‘breathscape’ that, grounded in different cultures, are immediately associated
with ancient and contemporary philosophies. As a reverberation from a semiotic
interaction and through the discretization and identification of semantic fields
relevant to the concerned scenario, and introducing textuality, a phenomenon
seen as a crystallization of the transition between outer text (the text of reality)
and text (subject/object), we consider those visual texts which are crossed by a
common faculty that is both descriptive and inventive: by approaching some
inferential and cultural regimes and analysing their specific enunciative practices,
we then contribute to their renovation. The texts related to the practices as part of
the narratives intrinsic to cultural semiospheres underline the insatiable vastness
of epistemological content to be dealt with, and the functional reductionism of
the corpus is only a first approach to the field that is intended to shed light on the
general panorama and to stimulate subsequent debate and insights.
Key words: breathscape; semiosphere; cultural semiotics; translation; face; art
1
Department of Philosophy and Education Science, University of Turin (Unito), Italy;
e-mail: silvia.barbotto@unito.it.
https://doi.org/10.12697/SSS.2021.49.3-4.10
438
Silvia Barbotto
“…como un lindo recuerdo
que se convierte en rostro
y yo sepa por fin
que dejas para siempre
la espesura de ese aire maldito…”2
(Benedetti 1998: 84–85)
‘Breathscape’ is a vast prospect that includes actions and traces of breath, that builds
images and texts, that involves the human being and the extra-human context; it is
the landscape of breath, of the breathing and the breathable. As a semiosphere with
a delimited character and semiotic irregularity certainly draws from its biological
environment, it becomes an articulated multilevel process that involves cultural
phenomena such as lifestyles, lived ambience, movements and reactions, both
voluntary and involuntary. Just think of the contemporary pandemic period, which
has shed light on the foundation of life, on the semiotization of the air we breathe
as a transversal right but also as a language whose articulation is still partially to
be discovered and recomposed as a cultural kaleidoscope.
Although in breathscape analysis it is necessary to mention reference to the
ontological and biological aspects, we will focus instead on the critical semiotic
one: by evoking not only the living status of the breathing human body, but also
breath as a transversal, inclusive and fertile enclosure, we will turn to breath as a
textualized scenario of being and expressing. It is, in fact, the object of specific
articulations in manifestation of art or multimedia works as displays of signs left by
its course, its movement or built through interaction with other texts, as traces left
on the face, the one belonging to the breathing human body. The facial production
of meanings, in its reciprocity with otherness and the context, occurs on multiple
levels; we identify at least two: the object one analysed through the selection of
a dedicated corpus, and the meta- one, focused on the language utilized, on the
generation of the argumentation. The face-sphere as a system is a whole: we need
to identify meaning-making monads and articulate, mereologically, mechanisms
of translation.
It is precisely the face, in this case together with ‘breathscape’, that is at the heart
of this article: we would like to explain, albeit not exhaustively, its role, its dynamic
and processual signification, responding to the contemporary need of researching
and understanding the face in different interconnected dimensions as, for example:
“the predominance of the face in the neurophysiology of perception and cognition;
2
“...like a beautiful memory / that becomes a face / and I finally know / that you leave forever
/the thickness of that cursed air…”. (Translations from languages other than Engish are mine,
unless indicated otherwise, S. B.)
Art, face and breathscape: From air to cultural texts
439
its impact on visual culture; its role in art as in portrait; the new aesthetic gender
and formats of facial communication; the change of introspection, empathy and
interaction in the face” (Leone 2021: 16). We are already starting to delineate
the visage as a communicative interface with ‘breathscape’, a surface on which to
express, transmit and perceive, but also as a live transitional platform, a seat of the
ephemeral passage of components such as air which, besides having determinate
chemical and physical characteristics, also becomes an interpretable instance of
arbitrary nature, where the embodiment of automatization and de-automatization
can be questioned. The face assumes the profile of a topographic countryside3
full of plateaus and mechanisms of intersemiotic transitions and translations, in
which to understand and establish the borders between the different semiospheres,
with references to physiognomy (Caroli 2012), anatomy, morpho- and neurophysiology4. Being rich in promontories and caverns, the visage is an expressive
façade and visual screen of emotionality (Hwang Hyisung 2015); furthermore,
density and emptiness of its constitution (the physical and the metaphorical one)
establish the haptic mechanism of internal connectivity: the nose and specifically
the nostrils, the mouth and the oral cavity are the main orifices through which we
breathe, where breath transits, but all the pores of the skin breathe and a semiotics
of action would be needed to crystallize its dynamics.
There is an array of notions that different cultures have adopted to know
and describe the entire universe as a breathing one – we need only think of the
pneuma of the Stoics, the prana of the ancient Veda texts, or the broader oriental
term Qi, emblem not only of breathing but of the wider panorama of energy;
metalinguistic attempts to unfold meanings about this cosmos are part of the
efforts that accompany this exploration, too. However, the very wide range of
cultural languages containing linguistic and comparative considerations will not
3
At this point, we recall a passage from Deleuze and Guattari for whom the development
of the mechanism of faciality contains marked references to the concept of landscape: “All
faces envelop an unknown, unexplored landscape; all landscapes are populated by a loved or
dreamed-of face, develop a face to come or already past. What face has not called upon the
landscapes it amalgamated, sea and hill; what landscape has not evoked the face that would
have completed it, providing an unexpected complement for its lines and traits […] The
machine is already in place that always functions to produce faces and landscapes, however
abstract” (Deleuze, Guattari 1987: 172–173). The authors bring to light various examples in
which the intercut between face and landscape stands out in literature, architecture or cinema;
although it leaves room for drift and uprooting, it nevertheless suggests, indeed, encourages,
systems of re-territorialization, in a kind of ‘visageité’ not only of certain parts of the body, but
also of objects of common use.
4
In this respect, we recommend watching the series “The science(s) of face” by FACETS
ERC, organized by Viola Marco.
440
Silvia Barbotto
be deeply examined; rather, the investigation will focus on artistic and aesthetic
artefacts.
At one point, we will be focusing on the paradigm of visual culture, art and
pseudoscience: selecting a few artistic texts and also drawing on physiognomy, we
will create a corpus crossed by a common property that is both descriptive and
inventive. These visual text travel in semiosphere, propitiating new experiences for
the visionaries, and they are also re-created back: “Creative vision modifies these
materials. They take their place in an unprecedented object of a new experience”
(Dewey 1934: 89).
Based on the Lotmanian perspectives on culture and art, the article proposes
relations with different interpretative semiotics currents and some methodological aspects regarding segmentation and selection which, once the isotopy
of the breathscape as narrative background has been established, determines a
programmatic body of work whose phenomenological origin is very heterogeneous:
we find photographs of contemporary situations and performances, paintings,
drawings, engravings. Even if this variety seems too discrepant, we could highlight
that in all these manifestations of breathscape there is an evident reference to
breath as a marked semiotic act, through the use of visual and verbal language; the
reference to the face is also transversal, always present and mediated. While we are
interested in emphasizing the discursive and discernible entity of the breathscape
through the selection of texts, we also wish to emphasize the theoretical attributes
of these texts and explore the consistency of their media in terms of materiality.
“Semiosphere cannot come into contact with anything but texts, and texts are the
products of semiosis. In this way, any contact with a space lying on the other side
of a given semiosphere demands the preliminary semioticization of that space”
(Lotman 2019[1989]: 91). Each text, a subjective object source, will be the bearer
of a theoretical piece both on thematic path and on the medium itself: the creative
processes underlying the text, as well as some of its significant aspects, will be
partly highlighted, but, because it already is in itself, it will then acquire the totality
formed in its continuity of being open to reading and interpretation.
It follows that the elements subsequently shared are, even if extremely
simplified and still in the embryonic state of a necessary forward-looking path,
partly representative of an initial approach of semiotics to the syntagmatic and
paradigmatic articulation of the composition of the breathscape.
The identification of such discrete moments may recall the multiplicity of
codification, the heterogeneous and multi-level property of textualities underlined
in Lotman and Tamm (2019[1983]: 75):
Art, face and breathscape: From air to cultural texts
441
[…] every text is multiply coded (double coding represents the minimal structure).
The conflict of thought formation arises not between individual textual formations
but between languages manifested in text. Waves of syncretization of different
arts […] illustrate the two opposing tendencies characterizing this process. The
structural parallelism of textual and individual semiotic characteristics allows us
to define a text at any level as a semiotic entity, and to view an entity at any sociocultural level as text.
The following is divided into three parts: in the first part we approach the bodyface in relation with the environment, incorporating the individual and collective
human actions in term of textuality. By entering another level of unity and complexity of meaning recognition, we then move to the language of art and description
of dynamics texts. The third part will be dedicated to the relation between
generalization and individualization process, by specifying cultural readings of
air-signs on and of the visage.
1. Body-environment interaction:
Con-text, ‘air as thing’ and consciousness
In the Lotmanian perspective the relation between the text and the context is
proactive and co-generative: the text is intrinsic to its context and its provisional
extrapolation happens only in speculative terms. For this reason, we briefly
elaborate on the concept and artefacts in their individuality, being conscious that
they are submerged in a more complex and formative process.
By introducing the suffix ‘-scape’, we propose specific semantic features to the
carrier token ‘breath-’, assuming the immersive and spatial attributes denoted
in its extensive suggestions about the environment. ‘Breathscape’ is therefore
a landscape of breath, a diffuse semiosphere, but epistemologically established
in the circumscription of semantic manifestations; it is a semiospheric context,
an inextricable and changing binomial (breath and landscape) composed in a
pertinent and complex unitarian polyphony, governed by signs and laws that are
not solid structures in themselves, “but fluid elements, which do change through
history. A science of fluxes is needed to understand them fully” (Leone 2014: 117).
In fact, the proliferation of the ‘-scape’ as a field of aesthetic analysis has been
taking place since the end of the 20th century and, as Battistini (2020) warns us,
it mainly concerns the five senses; he concentrates on the horizon of hearing and
the audible as the path of the sense effects of the soundscape.
The compound word ‘breathscape’ provides that, thought of in more narrative
terms, the object of its narrative programme is not a sense itself, but rather an
442
Silvia Barbotto
a priori synesthetic field constituted in union and disjunction with the subject,
intra- and extra-individual. It is this dimension made up of variable rhythms and
densities, composed of more or less figurative fragments, expanded but at the
same time necessarily intimate, that constitutes the environment to which we are
referring and in which this specific textualization and semiotization of pieces of
life takes place.
Distinguishing ‘breathscape’ as a language composed of texts, whose compositional and immanent signs are concentrated in a partially legible ensemble, in
a kind of a dynamic “thing”, as we recall from Genette (2010: 33) for whom
‘things’ are the stable (relatively immobile and durable) appearance that certain
perturbations of atoms take on, which they nevertheless conduct under their
continuous and untiring enduring aspectuality. In short, things are a particular
kind of event.
We therefore go to the basis of the cultural semiotic point of view that considers culture as a system of languages and its concrete manifestations as texts,
asking ourselves questions such as: How is the breathing landscape generated
and translated? How and where do the signs of the flow pause? Is the face also a
junction of text and context? Is it a platform where breathscape takes form and
its development becomes explicit? Why is breathscape relevant from a semiotic
perspective?
In this regard, Squarcini (2020) talks about the environment of breath in
which we are immersed, a kind of “locus of daily setting” whose presence we
praise and whose vital flow we invoke only when its normality is shaken up or
the air is suddenly missing5. He also discusses the nature of this environment
very thoroughly in his commentaries on the Patañjali Yogasūtra, II.50: “This is
the general definition [...] of prān. āyāma: it is the confinement and analysis of the
movement, the course, the ‘natural’ mode of exhalation and inhalation” (Squarcini
2019: 114, Comment 55). After having deepened the first three modalities
(external, internal and suspended) of this practice, in Comment II.58
[…] the fourth type of prān.āyāma is described, completely different and superior to the other three presented previously. [...] One thus passes from the act
of confinement and vigilance over breath to the perception of the very form
of the prān.a, seeing the very intention of the confinement of breath disappear.
(Squarcini 2019: 115)
5
This general idea comes from the interpretation of the lecture (Togliere fiato al respiro.
Il prānāyāma come non si è mai inteso) given by Federico Squarcini in the Chamber of the
National Museum of the Risorgimento on 27 September 2020, during the 16th edition of Turin
Spirituality Manifestation whose title was “Breath”.
Art, face and breathscape: From air to cultural texts
443
From a semiotic point of view, some authors have already worked on the element
air: for example, Jacques Fontanille (2008[2004]:160), relating it to the olfactory
field, considers air the agent form of the actant of control: “air is in a sense the
figurative vehicle of the powers of smell, weak but long-range if it is moving, strong
and limited in range if it is stationary.”
According to our interpretations, the breathscape, close to the definition of
prān. a, thus assumes an expanded character, with blurred topological dichotomies:
the inside and the outside interpenetrate in a transient, thin, extended space. Here,
however, we will deal with textual coagulations, practices and events that seem to
condense, distort and modulate this spatiality, devising its borders more imprecise.
The first case reference deals with lack: having gone viral, the tag #ican’tbreath
comes to underline the urgency of a subordinate position in a racist cultural
situation. Seen in narrative terms, the iconized phrase sums up and becomes viral,
moving as a signifying manifestation. The epistemological framework adopted
in this text settles as the figurativization of negation which, socially situated and
shared, does nothing but strongly affirm the opposite side of the same isotopy, life.
The modalization of the actants, once manifested in the conveying tag, highlights
the passage from the mode of negation of power (‘I can’t’) to the underlying modes
of duty, of wanting, of knowing. The first-person singular shared in this way
performs a hypostatized and inclusive act, indirectly implying the statement of a
first-person plural. The images that appear on the web under this tag express the
magnitude of a collective body that resists and fights by making its face manifest
itself firmly.
The drama of George Floyd’s death on 25 May 2020 is certainly a sensitive
matter as well as a powerful communal issue, and although we do not deal with
social ethnographic events like this, talking about air and human beings cannot
but involve this very topical text and its widespread representations, and perceive
them as paradigmatic. The formula #Ican’tbreath creates an immense transmedia
community in which the platforms and social networks become a contextual,
informative and foundational venue: for example, #icantbreath includes 1,250,628
publications on Instagram only as last consulted (on 19 February 2021).
To convey attention to the triggering of semiosis, its evolution in and with
culture, propitiating a process of self-consciousness, including self-as-individual
but also self-as-community, we remind the concept of personal knowledge treated
by Lotman as the connection between objectual things and individuality where
also absence and exclusion of the signs are vehiculated as well as the presence and
inclusion of them. The signic cosmos is animated by a collective consciousness,
and in “Toward a theory of cultural interaction: The semiotic aspect”, Lotman
(2019[1992]) argues that the elaboration and growth of the relation between
444
Silvia Barbotto
individuality and the whole are dynamics synchronically animated by two kinds
of consciousness, the logical one and the creative one, in any case a linguistic
consciousness of the parts and of a whole, a semiospherical entirety:
There are real human collectives in which every individual unit has the urge to
transform itself into a self-sustaining and unrepeatable world of its own while at
the same time entering into a hierarchy of structures at ever higher levels, forming at
every level a communal socio-semiotic entity, which, in its turn, enters into a more
complex unities as a part. The process of individualization and generalization, or the
transformation of the individual into an ever more complex whole and into an eversmaller part of that whole occur in parallel. (Lotman 2019[1983]: 74)
2. Language of art and description of dynamic texts
As part of everyday life, which Lotman calls ‘byt’, the arts as well as theatre (which
is treated as an artistic practice in its own right) propitiate an intense exchange
with the human being whose behaviour is continuously and intensely influenced
by them. Remarking on the special features of artistic objects and bringing them
closer to the prescription of things, we must, however, point out the dialogical and
deeply generative nature of this choice, especially in the light of the fact that we are
talking about a research object that is originally extremely changeable, the breath.
As a secondary modelling system, art can also be described as a secondary language
and, being a language, a stimulation for human consciousness, so art participates in
modelling the consciousness apparatus. Language thus contributes to questioning
the founding principles by propitiating, and in some way grammaticalizing, the
creation and interpretation of the resulting textual expressions. Gillo Dorfles has
studied the relation between perception, meaning and action and has attempted to
identify the perceptive and cognitive data on which the present and future artistic
experience is placed: his consideration are transversally close to this. In “The
phenomenon of art”, Lotman (2009: 150) made evident that
[T]he transformation, which occurs at the true moment of explosion, having
been filtered by the lattice of the modelling of consciousness which converts the
random into regular, does not as yet complete the process of consciousness. This
mechanism must also include the act of memory, […]. There are now, as it were,
three layers of consciousness: the moment of primary explosion, the moment it is
realized by the mechanism of consciousness and the moment of its redoubling in
the structure of memory.
Art, face and breathscape: From air to cultural texts
445
Artistic texts, as part of a secondary language system, propitiate internal collisions
that can explode into new significant structures and open up to the proliferation
of further texts that on the one hand tend to integrate with the initial one, while
on the other hand they are orchestrated in order to complete themselves as far as
possible with the context again. This explosive subversion, the fulcrum of such
languages, emerges from the innovative and propelling apparatus of visions other
than the prevailing ones.
The conversion of natural language into alternative ritualized, represented,
semiospherized formulas gives rise to new recipes encoded through cultural
arrangements: the texts will therefore be a syncretism of different idioms that have
a fluid and complex structure, that require a rhizomatic decoding process and
that, by making the various structuring items equivalent, highlight the isotopic
ensemble. If the texts, especially if they are artistic, are therefore integrated with
their context of origin, they are also hijacked in the forward-looking contexts of
destiny. Through cultural arrangements, art makes an explosive entrance into
the contemporary world, launching condensations of alternative and innovative
visions. In this respect we think of the works Body of Air (1959–1960) and Artist’s
Breath (1960) by the conceptual artist Piero Manzoni, who, being amongst the first
to do so, identifies and circumscribes breath as an element of artistic reflection.
Didi-Huberman (2016[2005]: 8) emphasizes the relational nature of a “thing”
called breath:
We only see it as a vital element – although it does not become an isolable ‘thing’ –
when it is polluted with dust, swirling in smoke, violent in turmoil or missing in
drowning. We never feel it better – as matter, as a medium, as a necessity – than
when impurity reigns and breathing becomes short.
In this recognition and attention to the outer text we approach breath as a thing,
as text.
From among the multitude of artistic texts related to breathscape, we have
chosen a few as representative of a general panorama, including different modalities, varying especially in the implication of the body and its relation with
materiality; for this first research they are not yet systematized and synthetized
in specific typologies, but have been approached instead through description,
also considered an essential scientific issue (Lotman 2006), a modus operandi of a
semiotic method (Migliore 2005).
The first example is Abramovich and Ulay’s performance: Breathing in Breathing
out (see Fig. 1).6
6
The performance is accessible on the YouTube platform https://www.youtube.com/
watch?v=rWixdA2xTSs.
446
Silvia Barbotto
Figure 1. Abramovich and Ulay. Breathing in Breathing out (1977). Performance (videorecorded live/freeze frame), Student Cultural Center, Belgrade. Courtesy of Marina
Abramovic and Sean Kelly Gallery New York.
The text air, modulated in the respiratory act denoted by the present participle of
the verb in the title, ceases to be a natural pre-constituted substance and becomes
a ritualized realm in which the whole action revolves, a constellation distributed
on various paradigmatic levels. First of all, there is the binomial thing/artist:
Abramovich’s intimate relationship with Ulay, the fusion of the two bodies and their
faces through shared breathing is sufficient until, at the seventeenth minute, the
air becomes almost exclusively carbon dioxide and causes partial fainting of both
protagonists. In the experience and ritualization of the moment, the receptors and
the cultural sphere also participate in the founding and transforming constitution
of new texts palpable not only in situ, but also in memory. The texts are subjects
with many entangled layers.
The structural complication that spreads like wildfire from such a full-bodied
artistic text nourishes the public, not only allowing new interpretations to take
root, but also opening up ideological spaces that were previously airtight. When
the artists are breathing into each other, the oxygen is reduced to zero and the
increase of carbon dioxide leads to exhaustion: the culturalization of the moment,
once the primary nature is gone, is not enough. Yet the action speaks volumes:
the proprioceptive and metalinguistic character of the action, captivated and
strengthened, becomes a voice for the generation of new meanings.
At the basis of this gesture, in the union of two faces whose mouths become
one, in the dreamy deliberation of each other as sufficient life force and in the
taking to the extreme of the attempt to transcend physiologically from the primary
metabolic functions, there is the effort to access the unknown, foregrounding and
therefore poetically operating on the axiological setting of those present.
The interpretative codes take on polyform dimensions and features, fleeting
characters of sometimes static but constitutionally fluid illustrations, from mythical
Art, face and breathscape: From air to cultural texts
447
reminiscences and immersive ceremonies to rarefied and penetrating perceptions.
The face alternates between subtle blasts of resistance and contrast, explosion
and staticity, into the creative depths of a semiotic circle. Above all again fluidity,
evoked breath that brings back to the lifeblood, zero degree on which and from
which to operate.
The following piece (Fig. 2) is composed of photographs of the performance
of running, an audiovisual installation with 26 sound speakers playing, each of
them showing a part of the artist’s breath (recorded during the performances) in
a series of web podcasts:
Kanarinka ran the entire evacuation route system in Boston and attempted to
measure the distance in human breath. The project also involves a podcast and a
sculptural installation of the archive of tens of thousands of breaths. The project is
an attempt to measure our post-9/11 collective fear in the individual breaths that
it takes to traverse these new geographies of insecurity.7
Figure 2. Kanarinka. “Running performance” (2007). Performance filmed as part of a
bigger work titled It Takes 154000 Breaths to Evacuate Boston. Ensemble of visual records,
screenshot from www.kanarinka.com (last viewed on 20 February 2020).
Like an archaeological artefact which is suspended in time and which has been
recreated in space, according to Lotman an innovative text can precede the
language and, searching for the codification between the real text (the outside text)
and the text, we accomplish an act of reconstruction, proposing newest version of
the same initial textuality.
In case of Kanarinka it is not only the face which is the protagonist, but the
entire body. “The body, all of it and at all times, even in spite of itself, signifies. By
itself and in the whole ensemble of its relationships the body constitutes a sort of
7
See http://www.kanarinka.com/project/it-takes-154000-breaths-to-evacuate-boston.
448
Silvia Barbotto
“corposphere”, which itself is an integrating part of the semiosphere.”8 The creative
capacity of the performing body shapes texts that tacitly break into the ideological
and molecular structure, triggering a trans-textual componential process in which
a person and culture regenerate each other: Kanarinka’s work of art reasons on the
dominant political paradigms and proposes a critical reading of these.
Another artwork to be addressed here, by Lozano-Hemmer Rafael (see Fig. 3),
is a hybrid between the categories of performance and installation: the body of the
protagonist – their face and breath involved – is preponderant, but it is also, at the
same time, great machinery in motion. It is a metaphor for perpetual movement.
Figure 3. Lozano-Hemmer Rafael. Last Breath (2012). Sound installation: apparatus 60 x
27,5 x 23 cm, tube up to 15 m long. Edition: 6 copies + 3 AP copies. Photo by Grace Storey,
Carroll/Fletcher Gallery. Courtesy of the artist.
This motorized installation allows the captured breath to preserve itself and last
over time in an infinite, complex and constant duration: it does not remain static
inside an inert box, but, simulating inhalation and exhalation, produces an organic
sound determined by the cardboard bag in which the last breath was initially
concentrated.
8
Finol Enrique in Semiotics of the Body, 2020. Virtual Series Facets available on Youtube,
channel facetsERC, min. 2:47.
Art, face and breathscape: From air to cultural texts
449
On the artist’s official website9, this piece is described as “A ‘biometric’ portrait
designed to store and circulate the breath of a person forever, between a robotic
bellows and a brown paper bag”: the cold mechanical complex is accompanied by
the sound act and a video that immortalizes the moment in which the chosen artist
(in this case Omara Portuondo) leaves her “last breath”. The storage of the most
ethereal substance, the subjectification of the machine and the timeless alteration of
the becoming allow punctuality to become durability, letting the poetics of the work
absorb within itself the isotopy of life and its opposite, denoted above all by the title.
The landscape of air together with the intrusion and pervasion of space in the
singular respiratory acts constitutes the syntax of the performance. This new text,
the installation, is syncretic and multilingual, lives on the frontiers and spreads out
in the interstices, every new text is somehow creative.
The frontier, once established, immediately aspires to be resemanticized and
reabsorbed, shedding light on the areas of contact, collision and opening, thus
revealing not only spatial and symptomatic considerations, but also those of an
underlying or evoked effect. This is the case of the trace in the conceptual art piece
Breath of Leaves (Soffio di foglie, 1979) by Giuseppe Penone: an imprinted area in
leaves evokes an anterior presence, a body that was previously there. A simulacrum
of an invisible body, the index of a material form that was once there, together
with the shape of the leaves on the ground create the breath: as is typical in Arte
Povera, of which Penone is a pioneer, poor materials with the predominant use of
natural elements assume a new signification on both the planes of expression and
of content, which seems to highlight an evident relation with ancestral and oriental
art representation. Breath of Leaves recalls the privileged moment in which we and
nature breathe together, generating a subtle aura that can be perceived when viewing
a work like this. There are also two later works by the same author (see Figs. 4 and
5) with which Breath of Leaves can be grouped together for systematic purposes10.
Some years earlier, in 1973, the Cuban artist Ana Mendieta started to develop and
produced a similar work that would then be grouped in the Silueta series.
9
“Last Breath is an installation designed to store and circulate the breath of a person forever.
The piece consists of a small brown paper bag which inflates and deflates automatically thanks
to motorized bellows similar to those found in artificial respirators in hospitals. The apparatus
hangs on a wall and is activated 10,000 times a day, the typical respiratory frequency for an adult
at rest, including 158 sighs. Each stroke of the machine advances a digital counter that beeps.
The breath circulates between the bellows and the paper bag through a ribbed transparent
plastic tube that emits a faint and hypnotic low sound. The first copy of the piece stores the
breath of Cuban singer Omara Portuondo.” https://www.lozano-hemmer.com/last_breath.php
(last accessed on 20 February 2020).
10 In the cycle of works Breaths (Soffi) and executed from 1978 onwards; the author retrieved
from the world of mythology the image of breath as the origin of man and as a vital and
energetic element, see: www.castellodirivoli.org/artista/giuseppe-penone/.
450
Silvia Barbotto
Figure 4. Giuseppe Penone. Breathe the shadow (Respirare l’ombra). (1999, installation:
laurel leaves and gilded bronze.) Photography by Paolo Pellion. Courtesy of Castello di
Rivoli Museo d’Arte Contemporanea, Rivoli-Torino.
Figure 5. Giuseppe Penone. Breath of Clay (Soffio di creta). (1978, sculpture.) Photography
by Paolo Pellion. Courtesy of Castello di Rivoli Museo d’Arte Contemporanea, Rivoli-Torino.
Art, face and breathscape: From air to cultural texts
451
A problematized philosophical category, the element of air moves from virtualized
to actualized in the form of breath, which can be read in semi-symbolic terms.
“Air [...] as the incessant exchange of ‘life in the past’ and ‘life to come’, of survival
and the desire that is born from it. […] Here, then, is the ‘air in motion’ that
makes the protrusions of desire and the emanations of the farthest past co-present”
(Didi-Huberman 2016: 30–31). Belonging to a single series entitled Breaths (Soffi)
Penone’s works highlight certain semantic traits whose interpretative keys of the
figurative and the plastic would intersect. Breathe the Shadow is a multi-sensorial
work that mainly combines the involvement of sight and smell in a reading path
that can be addressed by syncretic semiotics, while Breath of Clay is a figurative
sculpture. The isotopic significance of the work and the title can lead to the
articulation of different perceived interpretations, but the figurative elements (the
title, the lung, the leaf) together with the plastic ones suggest a system of relations
connected with the ‘breathscape’. It may be useful at this point to comment on
a recurring lexeme, ‘element’, about which Leone poses questions and suggests
answers. Asking what an element is in art, whether a work of art can be composed
of elements and then going on to more specific questions, he states:
‘Element’ is a common word in several meta-languages, although usually associated
with some sort of mechanical functioning: ‘element’ makes one think of the components
of a mechanism or of a chemical compound. Metaphorically, its semantics can also be
extended to cover the functioning of parts of a non-mechanical whole […] an element
of something is not something that expresses an autonomous intentionality and a
self-centered agency but something that is subservient to the agency of a greater
whole. (Leone 2018: 111)
In the next piece of art observed, the image as expressive substance (the portrait
genre, the technique of oil painting, etc.) is the inspiration on which the artist bases
his work: Tim Eitel (Fig. 6) typically uses photographs taken by himself that are
then translated into painting.
The air, a propitious element for a metatheoretical reflection, and the face as a
salient point for a semiotics of the body and of everyday life, of the space passed
through, inhabited, emptied, come together in Tim Eitel’s pictorial representation
to talk about himself and, in doing so, rework the cultural assets: “The processes
of individualization and generalization, or the transformation of the individual
into an ever more complex whole and into an ever-smaller part of that whole
occur in parallel […], this is the same way every artistic text is formed” (Lotman
2019[1983]: 74). Artistic processes build a kind of self-contained whole renovated
and integrated in reception for a more complex and comprehensive formativity.
The pictorial aggregate assumes high semantic density due to its internal
452
Silvia Barbotto
composition, but also within the more general characteristics of a properly pictorial
structure; according to Maria Giulia Dondero (2014: 27), the enunciative level may
vary according to the relationship between “enunciative levels according to the
degrees of stabilization (support) and enclosure (edges) of the object of analysis”.
Figure 6. Tim Eitel. Untitled (Breath). (2009) Oil on board, 9” x 16”. Collection of
Contemporary Art (Larry Qualls Archive). © 2014 Artists Rights Society (ARS), New
York / VG Bild-Kunst, Bonn.
The receptacle face in Eitel’s painting, emitter and receiver, synthesizes in a
planar congregation the result of a stratification of pigments on the canvas; as the
receptacle creator, the face becomes a reciprocal relation, and its orifices, the seven
passages, furnish the power to channel the transition outside ←→ inside, consenting
the incorporated fuel to burn and distribute life and the unnecessary material to
be expelled. The open mouth, which is the punctum in the level of expression of
the painting, can evoke a moment of sleep, but also a possible fainting, close to
representing death in the phenomenal macro-isotopy of human beings.
Mediating between Lotman’s cultural semiotics and the propensity for
Greimassian generative semiotics, Calabrese (2006) proposes deep narrative
paths in which “meaning is generated through a general architecture that gives
order to its internal components, one in relation to the other”, involving epistemic
and passionate dimensions of sensory experience. He elaborates theoretically on
the element water and objectifies it as a “figurative phenomenon on the level of
the manifestation of a text” (Calabrese 2006: 31–33). As a special figure of water,
different kinds of materials, drawing on encyclopedic and lexical definitions,
manifestations and figurative effects have been synthesized in the dichotomy
Art, face and breathscape: From air to cultural texts
453
representation/ostension of the following oppositional terms: transparency/opacity,
projection/distortion, internal geometries (structure/lack of structure), movement/
immobility, punctuation/durativity. “Figurality is formed by categorial sets placed
on the plane of expression of the semiotic system created by a text (especially
visual), and able to combine with categorial sets belonging to the plane of content”
(Calabrese 2006: 46).
In connection with the opposites of movement/immobility examined by
Calabrese we can also mention the speed/slowness dichotomy in another author,
Italo Calvino. This writer finds in speed and movement the possible reasons why
a story can fascinate us and considers them to be the basis of the narrative links
between desired objects. Movement, be it rapid or slow, is marked by rhythm, and
the succession of events in a discourse that is not always linear, their gait and their
conjunction establish the moving and stopping, “two operations on the continuity
and discontinuity of time” (Calvino 1988: 32). The theoretical contributions
concerning the temporal dimension are therefore highlighted as a rhythmic
multiverse in which aspectuality, assimilation more or less intense or thinned out
in time, provides evidence of the biological apparatus of the air represented as well
as breathed.
That sphere of manifestation which would not belong to figurativeness, would
instead be part of the semi-symbolic category due to the preponderance of the
plastic signifier as happens in the following art works by Kenneth Noland and
Richard Pousset-Dart (Figs. 7 and 8):
Figure 7. Kenneth Noland. Breath (1959). Acrylic on canvas. 168 x 165.6 cm. ©2020 The
Kenneth Noland Foundation / Licensed by VAGA at Artists Rights Society (ARS), NY.
454
Silvia Barbotto
Figure 8. Richard Pousset-Dart. Breath of White Motion. (1990, works on paper: wash,
pencil and acrylic.) 57.15 x 76.835 cm. © 2008 Estate of Richard Pousette-Dart/Artists
Rights Society (ARS).
Following Greimas, Collins and Perron (1989: 635), figurativity seems to go beyond
the limits of the vehicle or support upon which the manifestation of a visual object
is based: indeed, they emphasize that in addition to figures of expression, there are
also figures of content.
In case of Noland’s geometric and symbolic reference, the circle as a figure
belonging to the world is evident, and although its reading is open and less explicit,
if it is accompanied by the verbal discourse of the title, we can certainly open up
interpretative and pertinent paths.
In contrast, the signifier in Pousset-Dart’s graphic work is merely plastic: the
graffiti lines on a monochromatic background organize an abstract scene which
timidly suggests distant figures; the processuality highlighted by the strokes,
between childlike grapheme and architectural projectuality, suggests dynamism
and the emergence of possibilities, also interpretative ones. In Greimassian terms,
this work can be read through the use of topological qualities which, between the
straight line and the curved line, create a grid on the plane of expression that is
further made discrete by the frame that has “a twofold function: the segmentation
of the whole into discrete parts and the mapping out of the possible trajectories that
the various aspects of the reading will follow” (Greimas, Collins, Perron 1989: 638).
In both cases, the semantic contribution of the titles should be considered as a
determining and integrated component from which an isotopic construction that
envelops and involves text-image would result.
Art, face and breathscape: From air to cultural texts
455
3. Cultural readings of air-signs on and of the visage
From the semiotic mechanisms that are highlighted in artistic texts we can
now turn to those that, arising from the organic interaction between face and
breathing and established in specific cultural memories, dictate original languages
whose decoding is emblematic, for the very reason that they contain within them
multiple gestational and interpretative approaches. As Lotman has pointed out,
the collective intelligence of culture and individual consciousness travel together
to generate meaning.
The impossibility of establishing univocal deductive relations of contact
and interaction between the face text and the air text makes the relational practice ambiguous and difficult to codify: not only the cultural attributes, but
also the independent operations of the intertexts and intratexts contribute to
creating the specificity to be explored. Some cultures, however, have made their
contributions through the elaboration of codified diagrams, dictionaries and
even pseudosciences: physiognomy, for instance, has proved to be among the
most equipped of these kinds of systematizations, but its affirmation has varied
according to the geographical locations and historical moments in which it was
developed. Patrizia Magli traces a foundational path regarding the system of
physiognomy by identifying its various salient phases and crossing all the cultures
that have employed it. Although it is still being practiced today, it is often referred
to as a thing of the past:
Physiognomic competence originated in history as a form of knowledge determined by concrete needs. It is part of that circumstantial paradigm in which a
constellation of disciplines participated in antiquity, such as divination and
medical semiotics in its double function, diagnostic and prognostic, disciplines
that shared a similar attitude oriented towards the analysis of individual cases,
reconstructable only through traces, symptoms, clues. (Magli 2004[1995]: 26)
The author recounts how in Mesopotamian culture the face, formally composed
by a meticulous reading of its formal micro-components, was an omen in divination, frequently rich in omens, in other words, accessible during ritualistic
fields and subtle dedicated states of mind: some extracts are presented in Ludlul
bēl nēmeqi, an old sapiential manuscript that contains cults and ceremonies. The
ancient Greeks, however, proceeded by means of clues, called tekmaíresthai, which
suggested relationships with the invisible: especially in the case of illness, the expert
studied the dissimilarities that emerged (denoted by sweating, the sound of the
voice, breathing, etc.) and they knew that a real Hippocratic facies would emerge
when death was imminent.
456
Silvia Barbotto
It is interesting to note how even in this period we begin to speak not only
of the sequence of anamnesis–diagnosis based on the observation of the body
and especially the face, but especially on how “the environment influences the
physique and, consequently, the character” (Magli 2004[1995]: 40), as among the
works resulting from such studies we find Le arie, le acque, i luoghi (end of the
5th century B.C.) contained in the Corpus Hippocraticum. Magli also delves into
Arab physiognomy, whose “origin is lost in the mists of time, confused with that
of magic, anatomy, physiology, philosophy but, above all, with that of medical
art” (Magli 2004[1995]: 44); this is found in the term ‘firāsa’, a kind of “organ of
circumstantial knowledge” with which only a few sages and mystics were endowed:
it was an intuition capable of abductively interpreting not only the signs of the face,
but also the clues hidden in the soul and the cosmos. There were various versions
and updates of these practices, so much so that it continued until the beginning
of the 13th century with the prestigious Kitab al-Firāsa of Fakhr al-Dīn al-Razi, a
systematization of Arabic physiognomies.
In the West, this science was part of the apparatus of sign theories and astrology
until the middle of the 17th century, when there occurred a separation between an
astrological and a natural physiognomy:
The gaze passes through the surface of the body, the anatomy penetrates the inner
man […] Thus Cureau de la Chambre, who in his Art of Understanding People
distanced himself somewhat from the “chiromancer”, considered it still important
for a doctor to take astrological signs into account. […] Planets and signs of
the zodiac in astrological physiognomy, moods and temperaments in natural
physiognomy, in no way exclusive of each other, remain the guarantee of what
is at the basis of the science of the face: the analogy of soul and body. (Courtine,
Haroche 2007[1988]: 71–72)
With the Cartesian philosophy, also reported in Charles Le Brun’s studies, paintings
and Expressions of the Passions of the Soul, the organism entered the foreground
and the machine-man typology succeeds the zodiac-man one. The representation
of the signs that had been considered pertinent to the physiognomy earlier, became
concerned with geometry, anatomy and medicine. Later, among some 19thcentury authors including Cesare Lombroso, there was an epistemic degeneration,
a manipulation of information resulting from a reading of appearances that
undermined these and stimulated racism.
It is clear that all these practices were extremely impregnated with the ideological thought in which they developed chronotopically, finding particular
prestige or falling into decline, sometimes developing as conjecture and sometimes
as science, but always implying a strong relationship between the face and natural
Art, face and breathscape: From air to cultural texts
457
elements. It would be interesting to understand if and how a semiotics of the
elements is developed in various cultures; however, this study is limited to the
element of air formalized to the face.11
Every culture has developed a different cultural memory with respect to this
theme and seems to occupy positions that are currently rather distant from each
other as regards the reading of signs on the face through physiognomy: some
cultures have entirely abandoned such practices, but in some others the behavioural
and relational signs between the physical body and the astral body are alive and
readable. According to contemporary Chinese medicine, for example, the vital
energy called Qi has an abstract entity, but tangible repercussions: as, firstly, its
signs can be read on the face; and, secondly, it can be dispersed but also invigorated
through two main practices:
If Jing is the inherited fundamental energy, then Qi is the manifestation of that
energy necessary to participate in the world. This implies exertion and movement
or yang action. Qi can easily become overused and deficient, but luckily it can also
be manufactured. There are two classic ways that you can make Qi buffer the use
of Jing. The first way is by transforming food and the second is by transforming
breath. However, this is more than just eating and breathing. Both of these ways
of making Qi can be seen on the face and these horizontal markings are on the
central meridian of the face across the nose and across the philtrum. (Bridges
2012: 24)
Marked lines above the lip area would therefore seem to indicate that “it is expected
that some Jing will have been used up by these ages, but in order for Jing to be
available for the remainder of life, a person with any of these lines should actively
start making Qi with conscious intent as the Jing is not as strong as it used to be
and needs support” (Bridges 2012: 24).
The memory of culture is identified by Lotman as a very important function
of a text that, however, constitutes only a reduced mnemonic programmes (Lotman
1996). Face as memorial text, physiognomy as the reading of characteristic and
permanent sign whose trace has been sedimented on the face in the memory of
time, are part of the mnemonic programme of culture, and the Chinese culture is
still benefitting from studying and applying it. As we have already seen, memory
is the necessary building block for the initially explosive and then conscious
phenomenon of art to take root.
11
In this regard, we would like to go deeper into Magli’s (2004[1995]: 51–60) text, which,
while also investigating other authors, reports on the semiotic Greimassian square of elements.
458
Silvia Barbotto
From the point of view of semiotics, culture represents collective intelligence
and collective memory, that is a supra-individual mechanism for preserving and
transmitting messages (texts) and for creating new ones. In this sense, the field
of culture can be defined as a space of shared memory, within which certain
common texts are preserved and actualized. (Lotman 2019[1985]: 133)
So, actually memory is dialectical and its settling is not definitive.
In most of today’s Western contexts, the coding system of physiognomic
pseudo-science does not provide an informational setting for understanding its
language, but it did so in the past. The decoding of the signs on the face, if it
were to happen, would certainly not have the same resonance in different cultural
environments. Thus, awareness should be raised concerning this subject, getting
closer to oriental cultures that have maintained the dynamics of identification,
reading and manipulation of the element ‘air’ in relation to the face in their
memory.
The text later exposes some techniques and decodified readings, but what
interests us is understanding how this occurs, especially in ancestral philosophical
and medicinal practices, and in those ancient philosophies that have lasted to the
present day affirming that the signs of the body can be read.
In the interpretative possibility resulting from a reading of the face, the
ideological and cosmological background of the cultures under examination
is highlighted. The use of semantic and typological categories refers to specific
connotations regarding the air, the respiratory act, and the breathscape, while their
perimeter is not limited to the surrounding context, but greatly amplifies the range
of action, thus expanding the multidimensionality of the semiosphere. Where are
the frontiers of this? What measurements are appropriate for them? How can
the matter of body and air be scientifically correlated with the creative processes
described above? Do the body and the face have memory?
In Chinese medicine a line on the face evokes the cosmos, explaining such
a relationship on the basis of its features: an interpretative convention that
approaches the “thing” by touching it, making it intelligible and partly favouring
its direct contact. This line is:
[…] the highest point of the Chong Mai, which regulates the connection between
Qi and blood and is the bridge between heaven and earth-breathing cosmic Qi
into the body. […] At the simplest level, a marking here shows the need to breathe
properly, to expand the lungs and to oxygenate the blood. On a higher level, it
means to expand your consciousness. A line in this area indicates that you have
used up Jing and that the best way that your body can make Qi to compensate for
that loss is by breathing in Cosmic Qi. (Bridges 2012: 27)
Art, face and breathscape: From air to cultural texts
459
South-Asian yogic cultures also propitiate a continuous dialogue with these extraindividual forces, and textualizations may be evident on the historicized face, but
it is also interesting to study the performative face during specific uses of air as
in kryia such as vātasāra, or during the practice of breathing called pranayama12.
It would certainly be extremely fascinating to study the yogic face in terms of
textuality, but to underline the importance of the practical aspect preponderant
in these disciplines, we quote Squarcini’s words who, recalling Merleau-Ponty’s
thought, insists on “practice that makes knowledge” rather than on “knowledge
that orients practice”: it would therefore be up to the semiotics of the body and
of practice, or perhaps applied and implicated semiotics, to pursue this question,
which has so far only been sketched out here and is open to future study (Squarcini
2019: CVII–CVIII).
Conclusion
Recognizing, reading and evaluating the signs of the breathscape as well as
savouring the processes of its signification is an act of a continuous intersemiotic
translation. The textualization of the landscape of breath must pass through
different operations and from an elementary, natural, forgotten action it can
become a crucial aspect of new forms of life and new figures of art. During the
transition from text to language, its interpretation and therefore signification
assume partial correspondences and mainly bodily statements. Tangentially to
thought, breathing itself can be considered mediated by cultural language, as we
can see in Torop’s words on Lotman’s Universe of the Mind: “Lotman’s approach to
the culture includes the notion that thinking is mediated by language, but dialogue
occurs in cultural space” (Torop 2020: 267).
Breath can be read as a synecdoche par excellence that leads from one element
(air) to the whole (life). Many texts from all cultures make references through
intertextuality, paratextuality, metatextuality, hypertextuality and architextuality
to the passage of breathing and its immensity: to conclude, we may remember the
song “Breathe” from Pink Floyd’s album Dark Side of the Moon (1973): “Breathe,
breathe in the air/Don’t be afraid to care/Leave, don’t leave me/Walk around and
choose your own ground.”
The ether and the earth have always been the great dichotomous hemispheres
whose reading on the surface brings out the semantic opposites of above and below,
light and dark, light and heavy. Here, air and ground both assume the peculiarity
12
In this regard, we suggest viewing the images in specialized yogic texts or reading the
extremely figurative and detailed explications in texts such as Saraswati, Swami Satyananda;
Saraswati, Swami Niranjanananda 2013.
460
Silvia Barbotto
of being carriers of freedom of action, the basis of that decisional act in which the
breath, instead of being a natural act and simply perceived, is a conscious action
and the constructive foundation of ground on which to move openly.
“Souvent décrit comme la zone la plus fragile et la plus nue du corps humain, le
visage passe pour exprimer la vérité profonde des êtres. Or c’est justement parce
qu’il se trouve en permanence exposé, impliquant ainsi toute la personne, qu’à la
fois il se livre e se refuse aux regards.”13
(Guedron 2015: 8)
Acknowledgement: This reseach is part of a project that has received funding from the
European Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation programme (Grant agreement No 819649 – FACETS).
References
Battistini, Emiliano 2020. Tra Soundscape Studies e semiotica: dall’effetto sonoro agli effetti
di senso dell’ambiente sonoro. E|C (Serie speciale) 28: 1–17.
Benedetti, Mario 1998[1996]. El amor, las mujeres y la vida. México D. F.: Alfaguara.
Bridges, Lilian 2012. Face Reading in Chinese Medicine. Edinburgh, London, etc.: Churchill
Livingstone, Elsevier.
Calabrese, Omar 2006. La forma dell’acqua. Ovvero: come si “liquida” la rappresentazione
nell’arte contemporanea. Carte Semiotiche 9/10: 57–66.
Calvino, Italo 1988. Lezioni americane. Sei proposte per il prossimo millennio. Milano:
Garzanti.
Caroli, Flavio 2012[1995]. Storia della fisiognomica. Arte e psicologia da Leonardo a Freud.
Milano: Mondadori Eclecta.
Courtine, Jean-Jacques; Haroche Claudine 2007[1988]. Histoire du visage. Exprimer et taire
ses émotions (XVIª-début XIXª siècle). Paris: Payot & Rivages.
Deleuze, Gilles; Guattari, Félix 1987[1980]. A Thousand Plateaus: Capitalism and
Schizophrenia. (Massumi, Brian, trans.) Minneapolis, London: University of Minnesota
Press.
Dewey, John 1934. Art as Experience. New York: Perigee.
Didi-Huberman, Georges 2016[2005]. Gestes d’air et de Pierre. Corps, parole, souffle, image.
Paris: Les éditions des minuit.
Dondero, Maria Giulia 2014. Sémiotique de l’action: textualisation et notation. Cadernos
de Semiótica aplicada 12(1):15–47. https://doi.org/10.21709/casa.v12i1.7117
Dorfles, Gillo 2004[1959]. El devenir de las artes. México: Fondo de cultura económica.
Genette, Gérard 2010. L’œuvre de l’art. Immanence et transcendance. Paris: Seuil.
13
“Often described as the most fragile and naked area of the human body, the face is
considered to express the deepest truth of beings. Yet it is precisely because it is permanently
exposed, thus involving the whole person, that it both reveals itself and refuses to be seen.”
Art, face and breathscape: From air to cultural texts
461
Greimas, Algirdas Julien; Collins, Frank; Perron, Paul. 1989. Figurative semiotics and
the semiotics of the plastic arts. New Literary History 20(3): 627–649. https://doi.
org/10.2307/469358
Guedron, Martial 2015. Visage(s). Sens et representations en occident. Paris: Hazan.
Hwang Hyisung, Matsumoto David 2015. Evidence for the universality of facial
expressions of emotion. In: Mandal, Manas K.; Awasthi, Avinash (eds.), Understanding
Facial Expressions in Communication. New Delhi: Springer, 41–56. https://doi.
org/10.1007/978-81-322-1934-7_3
Leone, Massimo 2014. Cultural semiotics as fluxorum scientia. In: Bankov, Kristian et al.
(eds.), New Semiotics: Between Tradition and Innovation. Sofia: IASS, 112–130. https://
doi.org/10.24308/iass-2014-009
Leone, Massimo 2018. Designing imperfection: The semiotics of the pixel. Punctum 4(1):
105–136. https://doi.org/10.18680/hss.2018.0008
Leone, Massimo 2021. Esteticas faciais nas sociedades digitais contemporaneas. In: Teresco
Vilardo Abreu, Maria Teresa; Carvalho Correia, Claudio Manoel de (eds.), Contribuicoes
da Semiotica ao ensino de portugues no mundo. Rio de Janeiro: Dialogarts, 13–22. http://
hdl.handle.net/2318/1777594
Lotman, Juri M. 2009. Culture and Explosion. (Grishakova, Marina, ed.; Clark, Wilma,
trans.) Berlin: De Gruyter Mouton.
Lotman, Juri M. 2019[1983]. Toward a theory of cultural interaction: The semiotic
aspect. In: Tamm, Marek (ed.), Culture, Memory and History: Essays in Cultural
Semiotics. (Baer, Brian James, trans.) Cham: Palgrave Macmillan, 67–81. https://doi.
org/10.1007/978-3-030-14710-5_5
Lotman, Juri M. 2019[1985]. Memory in a culturological perspective. In: Tamm, Marek
(ed.), Culture, Memory and History: Essays in Cultural Semiotics. (Baer, Brian James,
trans.) Cham: Palgrave Macmillan, 133–137. https://doi.org/10.1007/978-3-030-147105_9
Lotman, Juri M. 2019[1989]. Culture as a subject and its own object. In: Tamm, Marek
(ed.), Juri Lotman – Culture, Memory and History: Essays in Cultural Semiotics. (Baer,
Brian James, trans.) Cham: Palgrave Macmillan, 83–93. https://doi.org/10.1007/9783-030-14710-5_6
Magli, Patrizia 2004[1995]. Il volto e l’anima. Fisiognomica e passioni. Milano: Edizione
CDE.
Marchianò, Grazia 1977. La parola e la forma. Bari: Dedalo.
Migliore, Tiziana 2018. La procedura di descrizione. E|C, On-line Journal of Aiss.
Polacci, Francesca (ed.) 2012. Ai margini del figurativo. Siena: Protagon editori.
Saraswati, Swami Satyananda; Saraswati, Swami Niranjanananda 2013. Prana vidya.
Munger, Bihar, India: Yoga Publications Trust.
Squarcini, Federico, ed. 2019[2015]. Patañjali Yogasūtra. Torino: Nuova Universale Einaudi.
Torop, Peter 2020. The chronotopical aspect of translatability in intersemiotic space.
Punctum 6(1): 265–284. https://doi.org/10.18680/hss.2020.0013
462
Silvia Barbotto
Искусство, лицо и дыхание: от воздуха до текстов культуры
Мы рассматриваем дыхание как обширную перспективу, которая включает в себя
действия и их последствия, которая создает образы и тексты, которая включает в
себя человека и окружающую среду; мы называем это «пейзажем дыхания». Затем
мы изучаем, как «пейзаж дыхания» взаимодействует с человеческим лицевым
аппаратом, порождая знаки и крайне интригующие интерпретационные процессы
на мереологической шкале. Как и где взаимодействуют территория дыхания и
тела? Какие процессы означивания они порождают? Какие знаки создаются в ходе
их феноменальной и семантической встречи? Искусство, безусловно, является
наиболее подходящим языком для изучения этого процесса, так как позволяет
проявлять и исследовать контрасты. Существуют различные интерпретации
«пейзажа дыхания», которые в разных культурах ассоциируются с древними и
современными философскими течениями. В качестве отголосков семиотического
взаимодействия и посредством различения и идентификации семантических
полей, относящихся к рассматриваемому сценарию, введения текстуальности как
феномена кристаллизации перехода между внешним текстом (текстом реальности)
и текстом (предметом/объектом), мы рассматриваем те визуальные тексты, которые
являются одновременно как описательными, так и изобретательными. Затем мы
рассматриваем некоторые выводимые и культурные режимы и анализируем их
конкретные практики, внося этим свой вклад в их обновление.
Kunst, nägu ja hingusemaastik: õhust kultuuritekstideni
Käsitleme hingeõhku avara väljavaatena, mis sisaldab tegusid ja nende jälgi, mis loob pilte
ja tekste, mis hõlmab inimesi ja inimvälist konteksti; seda laia maastikuvaadet nimetame
“hingusemaastikuks”. Seejärel vaatleme hingusemaastiku vastastiktoimet inimliku näoaparaadiga, milles mõlemad kutsuvad esile märke, aga tekitavad mereoloogilisel skaalal
ka äärmiselt põnevate tõlgendusprotsesside liminaaltsooni. Kuidas ja kus mõjutavad
teineteist hinguse territoorium ja keha? Milliseid tähistamisprotsesse nad esile kutsuvad?
Ja milliseid märke nende tajutavas ja sematilises kohtumises luuakse? Kunst on kahtlemata
kõige kohasem keel selle protsessi uurimiseks ja ka varju jääva esile toomiseks ning
silmapaistvate kontrastide vaatlemiseks. On olemas mitmeid “hingusemaastiku” mõisteid,
mis erinevatest kultuuridest tulenevatena koheselt seostuvad kas iidse või nüüdisaegse
filosoofiaga. Semiootilise vastastiktoime järelkõlana ning asjakohase stsenaariumi
suhtes oluliste semantiliste väljade eristamise ja tuvastamise kaudu ning tuues sisse
tekstuaalsusnähtuse, nagu seda on välisteksti (reaalsuse teksti) ja teksti (subjekti/objekti)
vahelise ülemineku kristalliseerumine, uurime neid visuaalseid tekste, mida ühendab joon,
mis on niihästi kirjeldav kui ka leidlik. Seejärel võtame vaatluse alla mõned inferentsiaalsed
ja kultuurirežiimid, analüüsime nende eriomaseid enuntsiatiivseid praktikaid ning aitame
seeläbi kaasa nende uuendamisele.
Tekstid, mis on nende praktikatega seotud kultuurilistele semiosfääridele omaste
narratiivide osana, rõhutavad käsitlemist vajava epistemoloogilise sisu mõõtmatut
suurust ning korpuse funktsionaalne reduktsionism on üksnes esialgseks lähenemisviisiks
valdkonnale, mis peaks valgustama panoraamset üldvaadet ja soodustama edasist arutelu
ning arusaamist.
Faces in the
pre-Hispanic
rock art
of Colombia
463
Sign
Systems Studies
49(3/4),
2021, 463–488
Faces in the pre-Hispanic rock art of Colombia:
Semiotic strategies, visual semiospheres,
and gestures
Martín Cuitzeo Domínguez Núñez1
Abstract. This article analyses the sign systems or semiotic models that make
up the meaning of a double face or mask drawing in the pre-Columbian rock art
of Colombia, also discussing two human figures with depicted faces associated
with the main picture. The sample of rock art was detected on the walls of the
Chicamocha Canyon at the Mirador de Bárcenas site in the Santander Department
in Northeast Colombia. Its origin is attributed to the Guane chiefdom. We hold as a
central argument that this face and its gestures were part of a sign system or visual
semiosphere that spread along the banks of the Chicamocha Canyon. However,
the image shared some semiotic models and visual communication strategies with
societies that inhabited areas in central Colombia located hundreds of kilometers
away from the site studied. We support this claim because the forms, the use of the
space in the images, and the gestures of a hieratic character appear both in the faces
of the petroglyphs of the Huila Department as well as in some pre-Hispanic gold
masks from the Cundinamarca region.
The images of the rock art faces were analysed using a visual semiotic model
based on the suggestions of the Mu group and analyses by Félix Thürleman and
Jean Marie Floch. The analysis was complemented by Jacques Fontanille’s proposal
for the levels of semiotic relevance, with Juri Lotman’s concept of the semiosphere
employed as a key concept.
Keywords: rock art of Colombia; pre-Hispanic faces; Guane culture; ancient
semiotic systems; visual semiosphere; gestures
1
Escuela de Antropología e Historia del norte de México, Chihuahua, México Licenciatura
de Arqueología; e-mail: martin_dominguez@inah.gob.mx.
https://doi.org/10.12697/SSS.2021.49.3-4.11
464
Martín Cuitzeo Domínguez Núñez
1. Introduction
How to define a face from a semiotic perspective? How to account for the historical
transformation of the face and the systems of signs to which it has belonged?
Answering these questions implies a cross-cultural and diachronic approach. To
develop this perspective, this article is going to discuss the representation of five
pre-Hispanic faces at a site of more than 500 years old cave paintings located in
northeast Colombia, South America. The paintings have been attributed to the
Guane culture or chiefdom and share similarities with images from other regions
of Colombia.
I analyse the images using an archeosemiotic approach, as well as employing an
interdisciplinary methodology deriving from visual semiotics and archeology. The
results are compared with petroglyphs, two gold objects, and a wooden sculpture
from the Central region of Colombia, leading to the conclusion that the faces
were part of a visual semiosphere and a system of signs that extended through the
Chicamocha Canyon and shared elements with other regions of central Colombia,
which serves as the central argument of the article. As a second argument, I claim
that these faces are characterized by gestures and expressiveness that become
manifest through particular visual details.
The location and historical and anthropological situation of the study region
are presented in the first part of the article, followed by an introduction of the site
analysed and of the methodology and concepts used in this analysis, including
such notions as the visual semiosphere, and the system of visual signs and gestures.
The next section presents the results of the analysis of the faces, followed by a
comparison of the faces with material from Central Colombia. Thereafter I will
discuss the visual semiosphere to which the images belonged; the article concludes
with comments on the faces, their gestures and meanings.
2. Location, geography, and cultures of the region
Mirador de Bárcenas rock art paintings are located in the Chicamocha Canyon,
in the Santander Department in Northeast Colombia. The 227 kilometers long
Chicamocha Canyon is considered the second largest canyon in the world. It covers
around 108,000 hectares and is around 2,000 meters deep (Colparques 2020)2. The
canyon crosses the Departments of Santander and Boyacá and has been formed by
the effect of the Chicamocha River, being the product of:
2
Information on the canyon can be found at Colparques 2020. Cañón del Chicamocha.
Reserva Natural Regional, http://www.colparques.net/CHICAMOCHA.
Faces in the pre-Hispanic rock art of Colombia
465
[…] the erosion over the riverbed with the formation of great gorges on both sides
of the Banks. The canyon begins in the surroundings of the municipality of Socha,
Pisba, Chita and ends near the municipality of Lebrija. These mountains plates
are the result of the tectonic movements of the earth where geological faults that
comprise the Chicamocha territory up to regions such as Mesa de los Santos and
the city of Bucaramanga pass and emerge, but its vast majority extends over the
municipality of Aratoca3. (EcuRed 2020)
Regarding the archeology of the region studied, Gerardo Reichel-Dolmatoff points
out in his introduction to the archeology of Colombia:
In the foothills and plateaus of the Eastern Cordillera, north of the Muiscas, lived
some indigenous groups that shared many cultural traits with the inhabitants of
the Cundiboyacense region, but that formed small independent chiefdoms, all with
quite developed agriculture and with metallurgy, textiles and lithic carvings. The
Guanes, Laches, Chitarreros and others spread across the current departments of
Santander (Santander and Norte de Santander), but we do not know much about
the archeology of these regions. The few pottery complexes known from this
extensive area contain various (ceramic) painted types, with geometric motifs,
which are most likely related to the Second Painted Horizon of the Ranchería
River and, through this, to the bichrome complexes of western Venezuela. In any
case, it seems that there has been close prehistoric contact between the Chibchaspeaking peoples – Muisca, Guanes, Laches, Tunebos, Sierra Nevada de Santa
Marta – along the Eastern Cordillera, whose chronological, typological and
sociocultural details are still missing to be elucidated. (Reichel-Dolmatoff 2016:
353–354)
Reichel-Dolmatoff describes the geography of the region, but above all, he introduces us to the pre-Hispanic societies that inhabited the region, as well as the
surrounding areas, such as Guanes, Laches, Chitareros or Chitarreros, Panches,
and Muiscas. 16th-century Spanish conquerors gave these names to the different
groups of people who inhabited the foothills and plateaus of the Eastern Cordillera.
The Guane culture inhabited the territories in which Mirador de Bárcenas
or Pozo Tocaregua is located. The Laches and Chitarreros or Chitareros were
3
Quoted from EcuRed 2020. Cañón de Chicamocha. https://www.ecured.cu/Ca%C3%B1on_
de_Chicamocha; the original reads: “[…] de la erosión sobre el lecho fluvial, con la formación de
grandes desfiladeros en ambas orillas. El cañón inicia en inmediaciones del municipio de Socha,
Pisba, Chita y termina cerca al municipio de Lebrija. Estas placas montañosas son resultado de
los movimientos tectónicos de la tierra, en donde pasan y se desprenden fallas geológicas que
comprenden el territorio del Chicamocha, hasta regiones como la Mesa de los Santos y la misma
ciudad de Bucaramanga pero en su gran mayoría se extiende sobre el municipio de Aratoca.”
Translations from Spanish are by the author of the article.
466
Martín Cuitzeo Domínguez Núñez
neighbours of the Guane people. The Chitareros or Chitarreros lived north of
the Guanes in the present Department of Norte de Santander, in the region of
the city of Pamplona. The Spanish gave them the name since when the people of
these towns came out to meet them, they appered with containers with Chicha,
a fermented drink made with corn and still widely consumed in rural Colombia
today. The Chitareros who lived in cold areas were organized into chiefdoms,
while those who lived in warmer lands lived in tribes. The people worked the
land provided by a chief and, in return, they were paid in blankets and red dye.
The caciques returned part of these goods during the festivities, when these were
distributed among the community (Gamboa 2004: 755–756). This is an important
antecedent to interpreting both the origin of the pigments in the cave paintings, as
well as the presence of some symbolism that is found both in the funeral blankets
and in the images studied here.
The Laches lived to the south, in the Sierra Nevada del Cocuy; they were
engaged in mining the salt. This mineral was used both for their own needs and
for commerce with distant communities. Upon the arrival of the Spanish a process
of emergence of social classes began within their society. Their religion was based
on the cult of rocks, which they considered to have been the first human beings
to inhabit the earth. At the same time, they were warlike societies that resisted the
Spanish conquest (Acuña 2006).
Yet another neighbouring culture of the Guanes was the Panches, who also
inhabited the south of the Guane territory in the current department of Boyacá.
16th-century texts indicated that
[t]his Panche People land is fertile as to maintenance and food, most of it, because
another part of it is less abundant and another much less. And the misery in some
part of the Panche People comes to so much that, while subjugating them, he ran
into the Panches that surround the land of Tunja between two mighty rivers, in
some mountains.4 (Banco de la República 1979: 95)
The Muiscas or Chibchas, for their part, were the ancient inhabitants of the eastern
Andes, in the present departments of Cundinamarca and Boyacá. They also lived
south of the Guane territory. They formed a heterogeneous and complex society
organized into chiefdoms, characterized by the coexistence of both egalitarian as
well as profoundly unequal patterns. In this society, the economic and symbolic
4
The Spanish original reads, “Esta tierra de los Panches es fértil de mantenimientos y comida,
la mayor parte della, porque otra parte della es menos abundante y otra muy menos. Y viene
a tanto la miseria en alguna parte de los Panches que, andandolos subjetando, se topó en los
Panches que ciñen la tierra de Tunja entre dos rríos caudalosos, en unas montañas.”
Faces in the pre-Hispanic rock art of Colombia
467
processes were deeply rooted and it is difficult to differentiate them (Langebaek
2019). The Muiscas were characterized by their work of metallurgy, the craft of
agriculture, pottery, textiles, salt work and specialized trade (Uribe, Martinón 2012:
773). They were also characterized by their tombs dedicated to their chiefs or
caciques, as well as by the cult of rocks that was expressed in their diverse and
vivid rock art. The Muiscas were related to all the societies mentioned above. These
different cultures possessed, as we will see, cultural traits and types of organization
similar to the Muiscas.
Speaking of this complex mosaic of neighbouring cultures to the Guanes serves
not only to introduce the reader to the complex cultural panorama of the region in
pre-Hispanic times, but is also a first approach to the hypothesis that the images
of the other Chicamocha cave paintings were part of a common semiosphere that
was sustained by the interrelationships between the peoples mentioned here. The
following section focuses on the Guane society.
3. Pre-Guane and Guane Societies
The hypothesis that the Guane society was the author of the cave images of the
Mirador de Bárcenas and other sites in the Chicamocha Canyon is proposed here
for two main reasons: firstly, because some of the cave images have motifs similar to
those found in Guane textiles; secondly, for the Spaniards’ chronicles from the time
of the Conquista indicated that the Guane people inhabited the actual territories of
Santander and Cañón del Chicamocha (Castellanos 1886; Simón 1891; Fernández
de Piedrahita 1942; Banco de la República 1979; Gualdrón, Bermúdez 2018: 57).
This hypothetical statement only serves as the starting point to our interpretation
of the Chicamocha cave paintings, for we are primarily interested in understanding
the semiotic, symbolic and social processes that gave rise to the cave images.
According to the archaeologist Justus Schottelius (1941) and the archaeologist
Leonardo Moreno (2012: 127, 139), the Chicamocha region was inhabited first by
the Preguane societies. Preguane people had sedentary settlements in the region,
especially in Los Santos. There are indications of Preguane people from around the
400 A.D. (Moreno 2012: 127, 139). Around the 12th century, the Preguane population
underwent a transformation because it disappeared. According to Moreno (2012:
139), this could be the consequence of migrations and cultural assimilations in some
regions and of violent impositions in others. The Guanes and Chitareros, two agroceramic societies, occupied the region during this time (Morales, Cadavid 1984). In
the 16th century, when Spaniards arrived to conquer the region, the Guane societies
were consolidated and completely settled in the Chicamocha area.
468
Martín Cuitzeo Domínguez Núñez
The Guane culture spread through the northern region of the Chicamocha
Canyon, the Mesa de Los Santos, and the Mesa de Bucaramanga. In some cases,
housing sites were built on the alluvial terraces on one side of the Chicamocha River.
At the time of contact with the Spanish, the Guanes occupied the southern and
western lands towards the Suárez, Chicamocha, and Barichara rivers. They also
inhabited the Socorro region, Oiba and the Serranía de los Cobardes (Restrepo
2010). The Guane chiefdoms belonged to an agricultural society that grew corn, as
well as yucca, cotton, chili, pumpkins, and coca. At the same time, their livelihood
depended on the consumption of, for example, ants, and other insects. The Guanes
wove cotton, produced ceramics, built small circular buildings, as well as stepped
buildings (Sutherland 1971). They made rock art paintings and cultivated land
on artificial terraces at the foot of the hills. Their houses were artificially levelled
through the construction of platforms. They also had two or three parcels of land
located in different regions, and used these spaces to move from one place to
another several times a year (Domínguez 2011). According to the Spanish friar
Pedro Simón, the Guane worshipped a deity called Bochica, whose image was
portrayed on stones found on the banks of the Sogamoso River (Gualdrón y
Bermúdez 2018: 58) that rises at the confluence of the Río Chicamocha and the
Río Suárez. On the other hand, Bochica is identified as a “civilizing” god to whom
the origin of many cultural elements was attributed; he was also a deity worshipped
by the Muiscas.
4. The cave paintings of the Mirador de Bárcenas and
the Chicamocha Canyon region
The Mirador de Bárcenas paintings, elaborated presumably by the Guane society,
consist of 11 rock shelters with cave paintings (see Fig. 1). The two predominant
colours are red and white and the paintings are characterized by geometric,
animal, plant, and human-shaped images. The geometric motifs include circles,
ovals, triangles, lemniscates – figures that, for our society, resemble an eight or an
hourglass, triangles, ovals with straight lines inside them, half ovals, and rectangles.
The animal forms possibly represent reptiles, amphibians (see Fig. 2), and insects
such as ants; the vegetables are scarce and represent trees, possibly Guayacanes
(Gualdrón, Bermúdez 2018: 185–186) – see Fig. 3. Finally, there are human images
that are represented with extended limbs, which is important to notice because
some of the faces covered in this study belong to this kind of human figures.
To infer everyday living habits, the cosmology and mythology, of these societies,
it should be asked what elements, processes, and prerequisites were needed for
Faces in the pre-Hispanic rock art of Colombia
469
the realization of the rock images. This section gives an account of the economic,
social, and ecological processes that were required for the realization of the
images and provides an explanation to the main painting motifs of the Mirador
de Bárcenas and its associations, which will allow us to understand the rock faces
of this site. However, first the characteristics of the images should be described. In
answer to the question what kind of elements and social, ecological, and economic
processes were required for the realization of these motives, it appears that, in
terms of the motifs of animals and plants, knowledge of the animals and plants was
required, their habitual use and consumption were important, as well as a symbolic
appreciation of them. Thus, there are ants among the animals represented (Fig.
4). The economic importance of the ants in pre-Hispanic times is documented in
the colonial chronicle Epitome of the conquest of La Nueva Granada (Epítome de
la Conquista de La Nueva Granada):
[…] a province of people, not very small, whose maintenance was nothing but
ants, and make them bread to eat by kneading them. The ants are very abundant
in the same province, they are raised in corrals for the same purpose; and the
corrals are separations made of wide leaves that surround the anthill. And there,
in that province, diversities of hotmygas (ants), some large and others small. 5
(Banco de la República 1979: 95)
The breeding of ants and their consumption spread between the Guanes and
the Panches. This allows us to infer that the Guane society interacted with their
environment via the use of insectivorous resources. There is also evidence of
the cultivation of corn and other vegetables and plant resources. In the course
of fieldwork carried out in 2011, I observed remains of ancient corn and arrows
made of wood wrapped in vegetable fibres in a cavity behind a cave painting at the
Vereda San Rafael rock art site.
The Guane society had an organization and social structure that allowed it to
grow corn and cotton, raise insects and produce blankets. It was a chiefdom with
incipient social differentiation, and I argue that the cave paintings of this region,
especially the rock shelters analysed here, are related to this social differentiation.
5
Original in Spanish “[…] una probincia de gente, no muy pequeña, cuyo mantenimiento
no hera otra cosa sino hormigas, y dellas hazen pan para comer, amasándolas. De las quales
hormigas ay muy grande abundancia en la mesma probincia, y las crían en corrales para este
mismo effetto; y los corrales son unos atajos hechos de hojas anchas con que rodean el hormiguero.
Y ansy ay allí, en aquella probincia, diversidades de hotmygas, unas grandes y otras pequeñas.”
(Banco de la República 1979: 95).
470
Martín Cuitzeo Domínguez Núñez
Figure 1. General view of the Mirador de los Santos site6.
Figure 2. Rock Shelter 1 through a digital filter. Reptile images and lemniscates can be
detected.
6
All the photographs have been taken and edited by the author of the article.
Faces in the pre-Hispanic rock art of Colombia
471
Figure 3. Rock Shelter 2. An image of a Guayacán tree is located in the centre.
Figure 4. Ant detail in Rock Shelter 2.
5. Methodology and concepts
The methodology used here has two dimensions: on the one hand, the archaeological dimension, in charge of accounting for the materiality of the images; on
the other hand, the semiotic dimension, responsible for the study of the images
as forms of representation belonging to a visual semiosphere. This approach is
located within the perspective that Roberto Flores has called ‘archaeosemiotic’.
‘Archaeosemiotics’ is a term that refers to “the study of the meaning offered by
the materials of the past, both to their creators and initial users, as well as to
archaeologists and archaeology lovers (who, undoubtedly, are also, in their way,
creators and users)” (Flores 2019: 176).
Thus, an archaeological record of the cave paintings was first completed in
order to analyse the images. The paintings were documented according to two
472
Martín Cuitzeo Domínguez Núñez
registration approaches, one deriving from the American Association of Rock Art
(Arara 2007) and the other from the Mexican archaeologist Miguel Messmacher
(Messmacher 1981). The paintings were geographically located, the geology and
geography of the surroundings were described, both general photos and pictures
of each motif were taken. Dstrech software was applied that allows highlighting
details that cannot be observed with the naked eye, using digital filters colours
(Harman 20117). As part of the project, two cave sites were registered: El Mirador
de Bárcenas or Mirador de los Santos and La Vereda San Rafael.
Regarding the semiotic analysis, the plastic components, that is, the forms,
spaces, colours and textures of the images, were studied. The panel was divided into
three sections based on the grouping of human forms and taking the largest image,
which was a mask or a double face, as its axis. This methodology was based on
the proposals for visual analysis suggested by Félix Thürleman (2004), Jean Marie
Floch (1994) and the Mu Group (Grupo Mu 1993). However, it is also based on
the visual analysis of a Coco Chanel advertisement conducted by Roberto Flores
(2007) and on two semiotic analyses of cave paintings made by the author of this
article (Domínguez 2009, 2011).
Once this analysis had been completed, the images were placed in a model of
semiotic relevance levels proposed by Jacques Fontanille (2006). The recovered
levels were the sign, that is to say, an individual figure in our case; the text, i.e.,
which corresponded to the complete image or panel; the object or support, which
in this case would be the surface of the rock shelter; the practical scenes that means
the spatial interactions or relations of proximity or distance between the different
caves with painting which constituted the archaeological site.
The next step was to compare and contrast the images with those of the central
and southern regions of Colombia. These included petroglyphs, two pieces of
gold, and a wooden sculpture. Through this analysis, a system of signs or a visual
semiosphere was inferred. The concept of the semiosphere was taken from Juri
Lotman to account for a series of similarities in the ways of representing faces. In
this order of things, the visual semiosphere is derived from Lotman’s concept of the
semiosphere (Lotman 1996: 12) as semiotic universe of the past made up of various
texts and visual discourses. In this case, it is a semiosphere made up of multimodal
discourses such as cave images, cotton blankets, and ceramic pictographs.
These multimodal discourses were configured from sign systems and semiotic
models typical of the social and cultural processes of the peoples from the
studied region. It is difficult to define semiotic sign systems or models in the
7
Harman, John 2011. Using Decorrelation Stretch to Enhance Rock Art Images, available at
http://www.dstretch.com/AlgorithmDescription.html.
Faces in the pre-Hispanic rock art of Colombia
473
archaeosemiotic domain. For this purpose, Winfried Nöth’s definition of systems
was appropriated, understood as a set or assembly of signs, relationships, and
structures (Nöth 1995: 198) of the societies that inhabited the region in the past.
The system of visual signs was defined as a set of images used by past societies.
These images were related to one another in such a way that they had a common
visual structure. This definition was complemented by Jean Marie Klinkenberg’s
idea of system and code (Klinkenberg 2006: 138–139). Thus, the semiotic models
of the past were a set of differences of visual signs with two dimensions, that of
the signifier and that of the signified. These dimensions had codes that structured
them. There were visual codes that configured the relations of difference between
images on two planes: that of expression and that of content.
To return to Lotman’s concepts, these visual systems were both static and dynamic, closed and open at the same time (see Lotman 1996). From an emic perspective,
or from within a society, these systems of signs can be called semiotic models,
understood as models and schemes of the world that the society creates. However,
from an external perspective, the schemes created by semiotics can be called systems.
This article seeks to build a semiotic model in order to study the systems of signs and
visual semiotic models of the pre-Columbian societies that created the cave faces
of the Chicamocha Canyon. I consider these sign systems to have been expressed
in the semiotic-visual strategies of the images studied, which is why, to reconstruct
the system of signs, we must study these strategies.
The last step consisted of the analysis of facial gestures. The dictionary of the
Royal Academy of the Spanish Language (Diccionario de la Real Academia de la
Lengua Española 20208) defines ‘gesture’ (‘gesto’) as “Movement of the face, hands
or other parts of the body, with which affections are expressed or messages are
transmitted” and also defines ‘expression’ (‘expresión’) as “the manifestation of
affections and emotions through gesticulation”. Such facial expressions and gestures
are considered by Theo van Leeuwen as physiological communication resources
and as elements of non-verbal communication (Leeuwen 2005: 93).
6. Description and results of the semiotic analysis of the faces
The rock art set in which the faces were depicted is located in a yellow rocky
shelter, which is oriented towards the east, the direction from which the sun rises.
It is composed of 12 images (Fig. 5). To analyse these, the set of images was divided
into three sections based on the observation of the way the motifs were grouped.
8
See Diccionario de la Real Academia de la Lengua Española 2020: https://dle.rae.es/gesto?m=
form.
474
Martín Cuitzeo Domínguez Núñez
Segment One corresponded to a concentration of medium and small elements
in the left sector of the set in which five elements were counted. Segment Two
was composed of the double face and several small associated elements; in it we
counted five images. Finally, Segment Three was made up of a series of small
images located on the right side of the composition; the number of images in this
segment was four.
Figure 5. Rock art faces from Mirador de Bárcenas.
In Segment One, visual significance has been created through the use of straight
lines, rectangles, circular shapes, ovals, half-circles, points, and a dot. Some of these
forms create the image of two human figures with identifiable faces and extended
limbs. All the images are medium high if we compare them with the images of
other segments. As concerns their spatial location, one of the human figures is
situated in the lower right, very close to the centre of the scene. Above, a strip of six
horizontal lines can be seen on the left-hand side, a human figure in the centre, and
a semicircle with ten points on the right; above these forms a spot can be observed.
As regards the colours, the shape below is red with white dots; in contrast, the
human figure above is white and yellow with red dots. The semicircle is red with
eight red and two white dots. I will describe the human forms, specifically their
faces in more detail. The limbs of the first figure, the one in red, are presented as
extended and flexed. The neck can be observed, as well as the face composed of
Faces in the pre-Hispanic rock art of Colombia
475
a horizontal line serving as the mouth and two points as eyes. These are negative
shapes whose colour is formed by the surface of the rock. Above the face, there is
a headdress formed by approximately ten lines (Fig. 7A).
The second human shape, the one at the top, is made up of points, straight lines,
a curved line, an oval shape, and a rectangular shape. The figure has a headdress
composed of seven red dots and seven white vertical lines. The face consists of an
oval line, two red dots for the eyes, and a horizontal red line for the mouth. The
face is attached to the white-coloured body that has a rectangular shape; its legs and
arms are extended. The body is crossed by a vertical line and the four extremities
have a red line at their ends (Fig. 7B). Next to this image, there is a half-circle with
dots inside it (Fig. 7C).
The double face located in Segment Two has been created, utilizing a half oval.
In the upper part, there is a human figure with outstretched arms that consists of
a circular shape, points and a horizontal band, and an inverted triangle shape. The
vertex of the triangle extends downwards and forms the nose of the face or mask
below. The previous figure is separated from the main face by four horizontal
straight lines that converge in the half oval; these mark just half of the image. Two
of the lines are on the left and two on the right. Inside the oval, forming the face or
mask, there are three horizontal lines with the one at the top being longer than the
two at the bottom; these lines make up the eyes and cheeks of the face (Fig. 7D).
Finally, at the bottom, 14 curved vertical lines form a kind of a beard. Above the
image on the upper right-hand side, there is a short Y-shaped line, as well as two
triangles, one very small and one larger; these latter forms are hardly noticeable.
The face or double mask is the largest figure in the entire composition, while, in
contrast, the other shapes in this segment are small. Segment Two forms the centre
of the entire panel, as the rest of the images are grouped around the double mask.
Segment Three is made up of a human shape with a headdress formed by two
vertical lines on the head and a curved line in the shape of a C. The human shape
has been made by means of a V-shaped line as well as points. The predominant
colour is red, although the human form boasts white spots and dots. The images
are peripheral and are located at the extreme right of the composition (Fig. 7E).
476
Martín Cuitzeo Domínguez Núñez
Figure 6. General location of the human images.
Figure 7. Human figures in detail.
Faces in the pre-Hispanic rock art of Colombia
477
6.1. Visual strategies used in the elaboration of the faces
A visual strategy consists in a principle of visual syntagmatic composition (Fontanille 2016: 30) that orders the relationship between shapes, spaces and colours
in an image. It is also one of the semiotic levels of relevance. In this case one of the
main strategies consisted in placing the larger face in the centre of the composition.
It is inferred that this strategy helps to emphasize the visual importance of this
image, referring us to visual hierarchies in which the largest objects are located
in the centre. I propose here as a hypothesis that this form of hierarchization may
have been a reflection of a Guane society in the process of social differentiation
that involved the beginning of social hierarchies.
As a second strategy it can be argued that faces were elaborate, using oval and
circular geometric shapes. Straight lines were employed in order to making up the
eyes and the mouth. Also the opposition in the use of two colours, white and red,
was part of this strategy.
Thirdly the use of headdresses is observed always on the top of the faces. In
my opinion, their presence emphasizes the hierarchical status of the faces, which
seems to have been the same in the Muisca region of Bogotá where death masks
with headdresses were associated with the chiefs or caciques.
6.2. Colours and meanings
So far, the colours red and white have been described; however, their meaning has
not been mentioned, as archeology and semiotics alone cannot provide us with
these meanings. For this reason, it is important to talk about these colours both in
an universal and in a regional perspective. The relationship between the red and
white colours is almost universal, in many cases the colour black is also part of this
link. Thus, the red-white opposition has been observed since the Neolithic era, in
Africa, Eurasia and Europe, where it also appears in classical cultures and in the
medieval world. The relationship is also observed in the folklore and ethnography
of various contemporary peoples (Hemming 2012: 310). It is highly probable
that this universality is the product of neuro-physiological processes. Red is the
most visually exciting colour that the human being can perceive; this excitability
probably has to do with human evolution and with the purpose of our arboreal
ancestors to search for food (Hemming 2012: 325).
In Ancient America, red was the most frequent colour in cave paintings. In
Colombia, the indigenous Uwas or Tunebos who live in the Sierra Nevada del
Cocuy, in the central-eastern region of the country, have myths about the meaning
of the colours white and red. Everything indicates that these communities are
descendants of the Laches, a pre-Columbian people with whom the Guanes
interacted, as noted above.
478
Martín Cuitzeo Domínguez Núñez
Eduardo Londoño (1995: 88–89) describes an Uwa myth about the white and
red colours: “[…] at the beginning, the world was made up of a top (white) and
a bottom (red); when the movement occurred, these separate worlds mixed and
the middle world emerged, made up of blue and yellow”. This gives rise to the
hypothesis that the red and white colours of the faces of the cave painting analysed
may be related to myths about the origin of the world and colours.
However, when interpreting the colours of the paintings, it must be taken into
account that sometimes these deteriorate over the centuries and that the original
hue may have changed. Still, it should be noted that the red colour, generally made
from iron oxide, is one of the pigments that survives best over time (Argüello,
Martínez, 20049). As for the white, we believe that it is also of mineral origin, but
it is necessary to carry out a chemical analysis of the paintings of the Mesa de Los
Santos to determine the origin of the pigments with any certainty.
At least to the naked eye the rock art pigments appear to be different from
those used in other archaeological objects such as e.g. the pre-Hispanic textiles.
Archaeological studies of Chibchas, Uwa, and Guane pre-Hispanic textiles indicate that dyes of vegetable origin, especially a small plant called Fridericia chica
(Naturalista 202110), were used in 60% of Colombian pre-Hispanic textiles.
Furthermore, while Trunk of Brasil [dracaena; Palo de Brasil (GardenTags 202111)],
a tree from whose wood a dye is obtained that is sometimes red and sometimes
yellow (Rico Arce 1999), was used in 10% of the sample (Devia et al. 2016: 8)12.
This dye is still widely used today.
6.3. Gestures and meanings: From semiospheres to gestures and
facial expressions
The last point we concentrate on here concerns the expressiveness of the faces and
its counterpart, hieratism. ‘Expressiveness’ is understood as the movements of the
9
Argüello, Pedro y Martínez, Diego 2004. Procesos tafonómicos en el arte rupestre: un caso de
conservación diferencial de pinturas en el altiplano cundiboyacense, Colombia. Rupestreweb; can
be found at: http://rupestreweb.info/sutatausa.html.
10 Naturalista 2021. Fridericia chica; see https://colombia.inaturalist.org/taxa/578619Fridericiachica.
11 GardenTags 2021 Dracaena Fragrans ‘Trunk of brazil’ syn. ‘Palo de brasil’; see
https://www.gardentags.com/plant-encyclopedia/dracaena-fragrans-trunk-of-brazil-synpalo-de-brasil/38431.
12 See Devia, Baetriz; Cardale de Schrimpff, Marianne; Devia, Carlos; Niño Izquierdo,
Camilo 2016. Aproximación al conocimiento de los colorantes en la comunidad indígena
Ika de la Sierra Nevada de Santa Marta (departamento del Cesar, Colombia) Nuevo Mundo,
Mundos Nuevos, Coloquios. http://journals.openedition.org/nuevomundo/69205; https://doi.
org/10.4000/nuevomundo.69205.
Faces in the pre-Hispanic rock art of Colombia
479
face that communicate messages and also its visual effect. Specifically, we refer to
the appearance of movement transmitted by two parts of the face: the eyes and the
mouth. Expressiveness is opposed to hieratism, that is constituted as its opposite,
that is, the visual effect suggesting that the face lacks movement.
Gestures have biosemiotic, social and cultural dimensions. Among different
animals and primates they communicate aggressiveness, fear and other emotions.
In the case of human beings and, more specifically, of the cave faces of the Mirador
de Los Santos, expressiveness has been permeated by social and cultural semiotic
processes. We can argue in this case that the people who made the paintings
captured a series of cultural codes and world schemes through the gestures and
hieratism of faces.
Two of the faces seem to have their mouths open, which generates a sensation
of movement. It seems that these faces are screaming. There is a third face that
conveys the visual sensation of movement, since one of its eyes is larger than the
other. In contrast, the mask and the face with two bumps do not seem to show
movement and therefore appear to have no emotion.
Thus, it seems that the visual semiosphere of the region had at least two
different types of visual representations that schematized and encoded facial
gestures: one consists of the representation of faces in movement, while the other
represents faces statically. This opposition between movement and immobility is
not unusual since the concept of opposition is present also in other rocky shelters
of the Mirador de Los Santos.
6.4. From facial gestures to meanings
What kind of construction of specific meanings was behind these faces and
their two types of expressiveness? For other cultural regions of America such
as Mesoamerica in Mexico, rock faces have been taken as an expression of individuality, as a representation of the mountain as divinity, as ancestors who protect
and own mountains and ravines (Valdovinos 2014: 1–11).
How does this relate to the case presented here? I argue that in Mirador de
Bárcenas there was an enormous variability in the faces represented, which
indicates the importance of the individuality of every face, as it happened in
ancient Mesoamerica. Regarding faces and mountains as a dimension of divinity,
it is not easy to answer the question if there was an indigenous worldview in the
Guane region that related ravines and hills with gods. For this purpose a more indepth investigation would be required regarding the pre-Hispanic worldview of
the region and surrounding regions.
480
Martín Cuitzeo Domínguez Núñez
However, there are alternative interpretations. For instance, Camilo Gualdrón
and Oscar Bermúdez have interpreted the scene of the faces as a ritual in which
the characters were dressed in animal costumes. They mention the Huan ritual of
the Muisca people of the Bogotá region celebrated in December, during which
the participants placed red and blue ornaments on their heads, sang and danced
(Gualdrón, Bermúdez 2018: 211). The same authors have suggested that the double
face is possibly the mask of a feline as their research has shown that the current
inhabitants interpret this image as a feline; they also realized that the colonial
chronicler Pedro de Simón affirmed that there were sorcerers who turned into
felines (Gualdrón, Bermúdez 2018: 216).
In addition to this interpretation, the authors have pointed out that the central
face of the panel could also be “Bochica or the supreme deity Chiminigagua. This
is due to the central position of the image, as well as its larger size compared
to the rest of the motifs of the scene, which would denote its importance in the
pantheon” (Gualdrón, Bermúdez 2018: 214). These hypotheses, certainly seeming
to be promising, should be tested, yet seem quite plausible considering the available
data.
To complement the previous hypotheses I would like to propose a last one,
suggesting the possibility that the faces were related to the representation of the
dead and the deceased, as well as to the ancestors. I hold this for two reasons: the
first is that human burials were found behind the different rock sets many years ago
(Nelson Ramírez, personal communication); and, secondly, the rock motifs from
other painting sites in the region, such as the Vereda San Rafael, have also been
found in funeral blankets in which the dead have been wrapped. This hypothesis
should certainly be tested in the course of future research.
To end this section, we might recall the comments made above on the Guane
society, its worldview, and that of neighbouring peoples. In the first place, the
colours white and red seem to refer to a myth of the origin of the world, in which
all colours and the world in between them originate from the mixture of white
and red. At the same time, we have the aforementioned hypothesis that the face
in the centre may correspond to the image of the civilizing god Bochica. Rocks
were considered the first human beings who inhabited the earth, to which we
may add the idea proposed in the previous paragraph about ancestors having
been represented in the rock art paintings. Thus, both the faces and the support
seem to reflect the myth, the world of the dead, and the ancestors, but also the
social stratification and the hierarchies within the Guane society. An argument to
support this is that, as Langebaeck (2019) points out in the Muisca case, materiality,
economy, social structure, and symbolism cannot be separated in the societies of
pre-Hispanic Colombia.
Faces in the pre-Hispanic rock art of Colombia
481
7. Comparisons and contrasts between the faces and
the visual semiosphere of the region
Behind the faces analysed here, there was a visual semiosphere characterized by
visual rules and strategies. Elements of social structure, myth and worldview were
also represented in it.
Other cave painting sites in the region such as Vereda San Rafael employed
similar strategies and combined economical and symbolic topics. For example, in
the Vereda San Rafael rock art site, a beetle and a fantastic creature with many legs
were depicted: the insects, that were part of the Guane economy and diet, were
combined with mythical creatures.
In other archaeological objects of Santander, we can observe visual strategies
similar to those used in Mirador de Los Santos, but adapted to specific supports.
These supports are pre-Hispanic ceramics and cotton blankets, which are currently
sheltered in the Bolívar House Museum in the city of Bucaramanga. The ceramics,
although very diverse, include specimens in case of which the use of the colours
red and white can be observed as well as the use of forms similar to those employed
in rock art. Despite this, there is no evidence of representations of faces having
been found in the ceramics of the region (Domínguez 2011: 96). The blankets,
meanwhile, are cream or white in colour; sometimes red inks were used and they
were decorated with circular shapes. The blankets accompanied burials in caves
(Cardale 1987), but no faces have been depicted on the blankets.
In addition to these cases, the regions of Bogotá and the Huila Department,
inhabited by Muiscas, Panches, Laches, and other cultures mentioned in this
article, share visual and sociocultural elements, even though these areas were
located hundreds of kilometers away from the study site.
7.1. Petroglyphs from Aipe
The painted rock from Aipe, in the north of the department of Huila in central
Colombia, has a series of petroglyphs made by the ancient societies of the region.
These images can be compared with the depictions of humans and faces analysed
here. The stone was recorded in the 19th century by the painter Manuel María Paz
employing a watercolour technique (Biblioteca Digital Mundial 2020)13. It consists
of a large rocky massif with human shapes, some possible plant representations, as
well as geometric motifs and points on it, and is still standing today. There are more
than a dozen of human forms that look straight ahead and whose limbs are either
13
Biblioteca Digital Mundial 2020. Piedra con jeroglíficos cerca de Aipe, en la margen izquierda
del Magdalena, provincia de Neiva, see https://www.wdl.org/es/item/9146/.
482
Martín Cuitzeo Domínguez Núñez
flexed or extended. Two of the figures are similar in shape both to the paintings in
Mirador de Bárcenas and to those on the gold objects described below. The similarity
lies in them having bent arms and two protrusions on the head. On the other hand,
the shapes resemble the profile of the double mask, since they have a human shape
at the top and are composed of a semicircle at the bottom. Other similarities involve
the combination of both economic and symbolic elements, such as corn plants and
curve pectorals (Falchetti 1989: 25–26) in the case of the Aipe petroglyphs. These
features are the indication of common visual strategies such as the use of similar
forms and the representation of prominent elements over the head.
7.2. Gold masks and faces and their cultural meanings
Gold was a common raw material for the pre-Hispanic societies that inhabited
Colombia, especially for the Muisca culture. This material, usually associated with
offerings and tombs, was used to make masks and human figures whose faces
were discernible (Fachetti 1989; Uribe, Martinón 2012). These artifacts allow a
comparison with the rock art faces of the Chicamocha and with their details such
as the headdresses. The gold objects analysed in this article are located in the Gold
Museum (Museo del Oro) in Bogotá; in some cases, they belong to the societies
that inhabited the current Bogotá savanna, in other cases it is difficult to specify
the origin of the objects.
The artifacts with which I will compare the Guane rock art will be a breastplate,
a gold boat, and, finally, a face carved in wood. The cultural significance of these
objects for the Muiscas has to do with the offerings to which they belonged.
Thus, these pieces consist of representations of schematic human beings and are
associated with the burials of chiefs or political and religious leaders, as well as the
consumption of psychotropic substances and related rituals. They are also linked
to human sacrifice, war, and the civilizing god Bochica or Chibchacum who was
worshipped by both Muiscas and Guanes (Uribe, Martinón 2012: 782–784). Thus,
the three faces described here were part of such offerings of ritual, political, and
warrior significance.
7.3. Gold breastplate
The first piece with which I want to contrast the images of the Chicamocha is a
gold breastplate that represents a human figure (Banco de la República 2020a14).
As in the Chicamocha images or the petroglyphs previously analysed, the figure’s
14
Banco de la República 2020a. Arte prehispánico: un recorrido por el ciclo del oro y la sala
Cosmología y simbolismo desde la mirada del arte can be found at https://www.banrepcultural.org/
bogota/actividad/arte-prehispanico-un-recorrido-por-el-ciclo-del-oro-y-la-sala-cosmologia-y-6.
Faces in the pre-Hispanic rock art of Colombia
483
limbs are flexed – this position of the body seems to appear across the visual
semiosphere studied. The body is decorated with horizontal and diagonal lines,
as well as dots.
To elaborate or construct the face of the figure, a semicircle has been used, the
same form on the faces of the Chicamocha images. The nose is made using a line
and is reminiscent of the nose of the double face, previously analysed in the case
of Mesa de Los Santos, while circles have been used to constitute the eyes and the
lashes. In its lower part, the face wears earrings. Finally, it can be said that the
face bears a hieratic expression. The meaning of this image could be related to the
representation of a divinity, presumably Bochica.
7.4. Gold boat
The second gold object with which I wish to compare the faces is the so-called gold
boat – the representation of a raft on which a chief is observed accompanied by
his servants. The object was found in 1969 by the peasant Cruz María Dimaté in
a cave in the department of Cundinamarca near Bogotá. Other pieces of gold and
ceramic objects were associated with the finding (Banco de la República 2020b15).
It is related to the descriptions, deriving from the time of the conquest, of a ritual
in which the Muisca caciques were transported by a raft through the lagoon of
Guatavita, celebrating a ceremony (Banco de la República 2020b).
As for the human figures in this piece, several elements can be highlighted, first
of all the hieratic, rigid attitude of these figures. Their faces have been made by
means of oval shapes and they wear elaborate headdresses that are very similar to
the headdresses of the faces in the Chicamocha region. I suggest that the meaning
of the boat is related to the political power of the caciques. The reason why I make
the comparison with the Guanes paintings is both because of the similarity between
the human representations, and because both figures were used by caciques as part
of a process of social differentiation. The piece is full of details that, by themselves,
like other pieces in the Gold Museum, would deserve a separate analysis; however,
here I comment only on the description of the faces. Next, I will comment on a preHispanic face depicted on a piece of wood found in the central region of Colombia.
7.5. Face carved in wood
Finally, I want to point out that there are other types of supports such as wood
or stone in which faces have been represented as well. These faces have both
similarities and differences with the cave paintings of Chicamocha. At this point I
15
Banco de la República 2020b. La balsa muisca y el dorado can be found at https://www.
banrepcultural.org/coleccion-arqueologica/balsa-muisca.
484
Martín Cuitzeo Domínguez Núñez
would like to mention the wooden sculpture of a human being found in the Cave
de Bojacá in the Sabana de Bogotá (Reichel-Dolmatoff 2016: 354). The figure has
bent limbs, but these are close to the body. The eyes are represented as points, as
in most of the faces of the Chicamocha and the nose is a vertical line similar to
the nose of the double mask. I suppose that this carving represents an ancient
Muisca divinity, probably Bochica. Additionally, like the rock art paintings, it
is related with a rock shelter. This relationship and the importance of the rocks
among Guanes and Muiscas is the reason why I introduce also this object as an
element of comparison.
7.6. The visual semiosphere of the region
It seems that from these descriptions a common way of representing the face
among Guanes, Muiscas, and other indigenous peoples from pre-Hispanic times
can be inferred. In my opinion this serves as evidence of the existence of a semiosphere in which semiotic codes, mythological elements, and socio-political features
were shared. However, there are also differences between the regions, some
concerning the raw material employed to elaborate it. Others have to do with the
forms used in the representation of the faces, or with the meanings.
Thus, I argue that the ethnic diversity of the region has given rise to these
variations in the ways of representing and signifying faces and bodies.
8. Conclusion
The Guane society used a system of visual signs or semiotic models that was
part of a large visual semiosphere extending through the Chicamocha, but
sharing elements with cultures such as the Laches, Chitareros, and Muiscas. This
semiosphere integrated visual codes and strategies with social, economic, and
mythological elements.
The faces depicted in Mesa de Los Santos rock shelters by the Guane culture
were made by using geometric shapes such as semicircles, lines, and points. These
faces had some formal characteristics shared with the Bogotá and Huila region,
such as the use of similar forms to represent the face and the body.
The two most used colours were white and red. Using a contemporary Uwa
myth we can hypothesize that both colours are related to a story according to
which only the colours red and white colours existed at the beginning of the world.
The cave faces of the Guane culture also had a possible mythological
dimension. To this would be added the interpretations suggesting that the central
image could be related both to the god Bochica, a civilizing deity, and to the rocks
Faces in the pre-Hispanic rock art of Colombia
485
that are linked to the first inhabitants on earth. Other symbolic dimensions of the
faces could be related to the dead and worshipping the ancestors. However, faces
also have a social dimension where they functioned as a form of legitimation and
social differentiation on the part of the social segment that produced the paintings.
Thus, through the study of the rock art faces from the Chicamocha canyon,
we have tried to explain some social and symbolic processes of the Guane culture
to which we attribute the authorship of the images. In addition to outlining the
mythological and social dimensions already mentioned, I tried to explain the
interconnections between the Guanes and other societies through the faces studied,
emphasizing, above all, the relationship with the Muiscas of the Bogotá savanna
region, but also with the Laches, Panches and Chitareros. These interconnections
give us an account of a visual semiosphere sustained by the relationships and
commercial contacts between these societies. Thus, the Guane sociocultural
process did not occur in isolation but was part of a series of shared visual rules and
codes, as well as similar sociocultural situations, between societies and chiefdoms
that were in a process of social differentiation. Finally, this article has been an
attempt to describe and interpret the Guane semiotic system, as well as the visual
semiosphere to which the Guane society belonged. Thus, we have tried to show
how in the pre-Columbian past of Colombia, the faces of rock art were part of
complex symbolic and social meanings.
References
American Rock Art Association (ARARA) 2007. Basic Guide for Rock Art Recording. United
States: American Rock Art Association.
Acuña Rodríguez, Blanca Ofelia 2006. Territorio Indígena de la sal en La Sierra Nevada del
Cocuy. S XVI. Historia y Espacio 2(27): 71–86. https://doi.org/10.25100/hye.v2i27.4569
Banco de la República 1979. Epitome de la conquista del Nuevo Reino de Granada. Boletín
Cultural y Bibliográfico Banco de la República 16(03): 81–97.
Cardale de Schrimpff, Marianne 1987. Informe preliminar sobre el hallazgo de textiles y
otros elementos perecederos conservados Cuevas en Purnia, mesa de los Santos. Boletín
de Arqueología 2(3): 3–23.
Castellanos, Juan 1886. Historia del Nuevo Reino de Granada. Tomo I. Madrid: Imprenta
de A. Pérez Dubrull.
Domínguez Núñez, Martín Cuitzeo 2009. “Las Manitas” rock art site in Cañada de
Cisneros, Tepotzotlán, México: An analysis using visual semiotic tools. American Indian
Rock Art 35: 161–170.
Domínguez Núñez, Martín Cuitzeo 2011. La dinámica enunciativa de dos sitios con
manifestaciones gráfico-rupestres de la Mesa de los Santos, Departamento de Santander,
Colombia. (Tesis de maestría.) Bucaramanga, Colombia: Universidad Industrial de
Santander.
486
Martín Cuitzeo Domínguez Núñez
Falchetti, Ana María 1989. Orfebrería prehispánica en el altiplano central colombiano.
Boletín Museo del Oro 25: 3–41.
Fernández de Piedrahita, Lucas 1942. Historia general de las conquistas del Nuevo Reino de
Granada. Bogotá: Ministerio de Educación.
Floch, Jean Marie 2002. Composition iv de Kandinsky. In: Henault, Anne (ed.), Questions
de sémiotique. Paris: Presses Universitaires de France, 121–151.
Flores, Roberto 2007. De Cuerpos, brillos y transparencias. Análisis semiótico de una
imagen publicitaria. Escritos Revista Ciencias del Lenguaje 35/36: 7–40.
Flores, Roberto 2019. Fundamentos de arqueosemiótica. Cuicuilco. Revista de ciencias
antropológicas 76: 175–192.
Fontanille, Jacques 2016. Prácticas semióticas. Lima: Universidad de Lima, Fondo Editorial.
Gamboa, Jorge Augusto 2004. La encomienda y las sociedades indígenas del Nuevo Reino
de Granada: El caso de la provincia de Pamplona (1549–1650). Revista de Indias
64(232): 749–770.
Grupo Mu 1993. Tratado del signo visual. Para una retórica de la imagen. Madrid: Cátedra.
Gualdrón, Camilo; Bermúdez, Oscar 2018. Cosmovisión, paisaje y Manifestaciones Gráfico
Rupestres en el sitio Mirador de los Santos, Santander Colombia. (Tesis de pregrado.)
Bucaramanga, Colombia: Universidad Industrial de Santander.
Hemming, Jessica 2012. Red, white, and black in symbolic thought: The tricolour folk
motif, colour naming, and trichromatic vision. Folklore 123(3): 310–329. https://doi.
org/10.1080/0015587X.2012.716599
Klinkenberg, Jean Marie 2006. Manual de semiótica general. Bogotá: Fundación Universidad
de Bogotá Jorge Tadeo Lozano.
Langebaeck, Karl Henry 2019. Los muiscas: La historia milenaria de un pueblo chibcha.
Bogotá: Editorial Debate.
Leeuwen, Theo van 2005. Introducing Social Semiotics. London, New York: Routledge.
https://doi.org/10.4324/9780203647028
Londoño, Eduardo 1995. Linajes y circuitos de matrimonio en tres linajes chibcha: U wa,
kogui y muisca. Boletín Museo del Oro 38/39: 87–111.
Lotman, Yuri 1996. Acerca de la semiósfera In: La semiósfera I. Semiótica de la cultura y
del texto. España: Frónesis Cátedra, Universidad de Valencia, 10–25.
Messmacher, Miguel 1981. Las pinturas rupestres de la pintada, Sonora. Un enfoque metodológico. México D. F: INAH.
Morales, Jorge; Cadavid, Gilberto 1984. Investigaciones etnohistóricas y arqueo-lógicas en
el área Guane. Bogotá: Fundación de Investigaciones Arqueológicas Nacionales. Banco
de la República.
Moreno, Leonardo 2012. Arqueología del Nororiente colombiano. Los Teres: un sitio de
asentamiento de las culturas prehispánicas Preguane y Guane. Anuario de Historia
Regional y de las Fronteras. 17(2): 115–142.
Nöth, Winfired 1995. Handbook of Semiotics. Bloomington, Indianapolis: Indiana University Press.
Reichel-Dolmatoff, Gerardo 2016. Arqueología de Colombia: Un texto introductorio. Bogotá:
Ministerio de Cultura, Biblioteca Nacional de Colombia.
Faces in the pre-Hispanic rock art of Colombia
487
Restrepo, Lotero John 2010. Reconocimiento Arqueológico para las concesiones Mineras 1
D4-09211 y 1 D2-16551. Cañón del Chicamohca. Departamento de Santander. Licencia
1480”. Bogotá: Proyecto de Salvamento. ICAHN.
Rico Arce, Lourdes 1999. El palo de Brasil. Ciencias 54: 42–44.
Schottelius, Justus 1941. Arqueologia de la Mesa de los Santos. Bogotá: Publicación de la
Escuela Normal Superior, Imprenta Nacional.
Simón, Pedro 1891. Noticias Historiales de las Conquistas de Tierra Firme en las Indias
Occidentales. Tomo III. Bogotá: Casa Editorial de Medardo Rivas.
Sutherland, Donald 1971. Preliminary Investigations into the Prehistory of Santander,
Colombia. (Ph.D. Dissertation.) Tulane: University of Tulane.
Thürlemann, Félix 2004. Blumen Mythos (1918) de P. Klee. In: Henault, Anne; BeyaertAnne
(eds.), Ateliers de Sémiothique Visuelle. Paris: Presses Universitaires de France, 13–40.
Uribe Villegas, Martha Alicia; Martinón Torres, Marcos 2012. Composition, color and
context in Muisca votive metalwork (Colombia, AD 600-1800). Antiquity 86: 772–791.
https://doi.org/10.1017/S0003598X00047918
Valdovinos Rojas, Elda Vanya 2014. Rostros en la montaña. Arte Rupestre en Tlayacapan
y Amatlán Morelos. (Tesis de maestría.) Ciudad de México: Universidad Nacional
Autónoma de México.
Лица в доиспанском наскальном искусстве Колумбии:
семиотические стратегии, визуальные семиосферы и жесты
В статье анализируются знаковые системы и семиотические модели, составляющие
семантику рисунка двойного лица или маски в доколумбовом наскальном искусстве Колумбии. Также рассматриваются две человеческие фигуры, связанные
с основной картиной. Образец наскального искусства был обнаружен на стенах
каньона Чикамоча на участке Mirador de Bárcenas в департаменте Сантандер в
Северо-Западной Колумбии. Его происхождение приписывается племени гуане.
В основе работы лежит предположение, что это лицо и его жесты были частью
знаковой системы и визуальной семиосферы, распространенной по берегам каньона
Чикамоча. В то же время, это изображение повторяет некоторые семиотические
модели и стратегии визуальной коммуникации обществ, которые населяли районы
в центральной Колумбии на расстоянии сотен километров от изучаемого участка.
Формы, использование пространства на изображениях и жесты религиозного
характера появляются как в лицах петроглифов департамента Уила, так и в
некоторых доиспанских золотых масках из региона Кундинамарка.
Изображения лиц в наскальном искусстве были проанализированы с использованием визуальной семиотической модели на основе предложений группы Мю и
анализов Феликса Тюрлемана и Жана Марии Флоха. Анализ был дополнен уровнями
семиотической релевантности Жака Фонтаниля, при этом в качестве ключевой
использовалась концепция семиосферы Юрия Лотмана.
488
Martín Cuitzeo Domínguez Núñez
Näod Colombia Hispaania-eelses kaljukunstis: semiootilised strateegiad,
visuaalsed semiosfäärid ja žestid
Artiklis analüüsitakse märgisüsteeme ja semiootilisi mudeleid, millest tuleneb joonistatud
kaksiknägude või maskide tähendus Colombia Kolumbuse-eelses kaljukunstis; käsitletakse
ka põhipildiga seotud kaht inimfiguuri, kelle nägusid on kujutatud. Vaadeldav kaljukunsti
näide avastati Chichamocha kanjoni seinalt Mirador de Bárcenase leiukohast Santanderi
departemangus Kirde-Colombias. Seda peetakse guane hõimu looduks. Meie keskseks
väiteks on, et see nägu ja selle žestid olid osa märgisüsteemist ehk visuaalsest semiosfäärist,
mis levis piki Chicamocha kanjoni kaldaid. Ent sel kuvandil oli ühiseid semiootilisi
mudeleid ja visuaalse kommunikatsiooni strateegiaid ühiskondadega, kes elasid uuritavast
alast sadade kilomeetrite kaugusel asuvates Kesk-Colombia piirkondades. Toetame seda
väidet, sest kasutatud vormid, ruumikasutus piltidel ning hieraatilise iseloomuga žestid
ilmuvad nii Huila departemangu petroglüüfide nägudel kui ka mõnedel Cundinamarca
piirkonnast pärinevatel Hispaania-eelsetel kuldmaskidel.
Näokujutusi kaljukunstis analüüsiti, kasutades visuaalsemiootilist mudelit, mis tugineb
Müü rühma (Groupe μ) ettepanekutele ning Félix Thürlemani ja Jean Marie Floch’i analüüsidele. Analüüsi täiendavad Jacques Fontanille’i poolt välja pakutud semiootilise
relevantsuse tasandid ning võtmemõistena Juri Lotmani semiosfääri mõiste.
Face off – a semiotic technology study
of Systems
softwareStudies
for making
deepfakes
489
Sign
49(3/4),
2021, 489–508
Face off – a semiotic technology study
of software for making deepfakes
Søren Vigild Poulsen1
Abstract. Deepfakes, an algorithm that transposes the face of one person onto
the face of another person in images and film, is a digital technology that may
fundamentally alter our belief in visual modality and thus presents alarming
consequences for an image-centric culture. Not only are these face-translations now
so advanced that it is virtually impossible for people to tell that they are fake – this
technology is also becoming accessible to laypersons who, with little or no computer
skills, can use them for all kinds of purposes, including criminal intentions like
revenge porn and identity theft. It is therefore timely and crucial to explore the
semiotic potential of deepfakes.
This paper presents a semiotic technology perspective, i.e., the study of technology for meaning- making that is an emergent field in social semiotics, to report
on findings from an ongoing study of how deepfake software is designed and used
as a semiotic resource in erotic and political contexts. The paper advances the
argument that the software is able to appropriate all signifiers of the face and their
cultural history. Consequently, the semiotic operations of this technology prepare
the ground for the problematic perspectives of synthetic facial imagery.
On this basis, the paper calls for a critical awareness of taking visual representations of current events at face value and considers how deepfake technology is
embedded in unsound sharing practices of visual artefacts that tamper with the rich
meaning potential of the face.
Keywords: deepfakes; visual manipulation; semiotic software; face studies; digital
culture
1. Introduction
The appearance of deepfakes, images and videos in which the face of one person
is transposed onto the face of another person who says or does things that the
former did not say or do, is fundamentally altering our perception of visual
representation. Not only are these digital face-transplantations now so advanced
1
Department of Language and Communication, University of Southern Denmark, Denmark;
e-mail: vigild@sdu.dk.
https://doi.org/10.12697/SSS.2021.49.3-4.12
490
Søren Vigild Poulsen
that it is virtually impossible for non-specialists to tell that they are fake, but
software for making deepfake images is also becoming accessible to laypersons
who, with little or no computer skills, can use them for all kinds of purposes,
including criminal intentions like revenge porn and identity theft. According to
a report by the Dutch research group Deeptrace2, 96% of these kinds of synthetic
media are pornographic and involve women’s faces being “translated” and shared
in both closed and public online fora and communities without their knowledge
and consent. It is therefore timely and crucial to explore the semiotic potential of
deepfakes critically. Research literature on this subject matter is emerging within
research fields as diverse as machine learning (e.g. Maras, Alexandrou 2019),
pornography studies (e.g. Popova 2020), law studies (e.g. Chesney, Citron 2019a)
and media studies (e.g. Lees, Bashford-Roger, Keppel-Palmer 2021). However, each
field tends to focus on either technical aspects or use of deepfakes. This paper
complements the existing research with a semiotic technology perspective, i.e.,
the study of technology for making meaning, which is an emergent field in social
semiotics (Djonov, Van Leeuwen 2018), to link technical and cultural dimensions
of deepfakes. The paper reports on preliminary findings from a study of how
deepfake software is designed and used as a semiotic resource in erotic and political
contexts, respectively. On this basis, the paper argues that the issue of deepfakes
arises from the software’s appropriation of the rich cultural history of the face, and
a semiotic dimension needs to be included in the discussions of the manipulations
made possible by this technology in order to evaluate the effects of synthetic media.
2. Existing research of deepfakes
in machine learning and law
Deepfakes have so far primarily been studied in machine learning (in computer
science) and law. I will therefore briefly place the present study in the context of
deepfake studies in these two research fields. While the term ‘deepfake’ has been
dated to an internet forum in 20173 (Cole 2017; 2018), studies of deepfakes go back
2
Ajder, Henry; Patrini, Giorgio; Cavalli, Francesco; Cullen, Laurence 2019. The state of
deepfakes: Landscape, threats, and impact. Deeptrace. 24 January. Accessed at https://sensity.
ai/reports/ on 10 January 2021.
3
Cole, Samantha 2017. AI-assisted fake porn is here and we’re all fucked. Vice. December
11. Accessed at https://www.vice.com/en_us/article/gydydm/gal-gadot-fake-ai-porn on 10
January 2021;
Cole, Samantha 2018. We are truly fucked: Everyone is making AI-generated fake porn now.
Vice. Accessed at https://www.vice.com/en_us/article/bjye8a/reddit-fake-pornapp-daisyridley on 10 January 2021.
Face off – a semiotic technology study of software for making deepfakes
491
longer within machine learning. The area of machine learning research gradually
moved its research interest from extracting human faces from visual data sets
(Blanze et al. 2004) to face swapping in still images (Bitoul et al. 2008) and later
videos (Dale et al. 2011).
Deepfakes got their name from using deep learning technology, a concept
that is developed in the field of machine learning. This technology runs a neural
network simulation that uses big data sets to create digital information, such as
(fake) images. Neural networks are designed to mirror how the human brain
operates (at least in a computational understanding of the brain). The neural
network can learn to recognize patterns in a data set and to solve classification
problems automatically. Furthermore, it can be given a task to do and an objective
to fulfill, while monitoring itself and modifying and improving its performance.
Applied to face swapping, artificial intelligence (AI) algorithms analyse a source
face (e.g. the actor Nicolas Cage) from different angles to learn what it looks like;
then, they transfer these features they have learned onto a target face (e.g. that of
Donald Trump4), as if these were a mask5 (self-evidently, in this example the result
would be that Trump looks like Cage). This process can be repeated multiple times,
which will result in ever better facial simulation, depending on the quality of data
input and the time frame of the project.
Research literature within computer science does not consider face swapping
in a semiotic sense, but addresses problems of how to replace a face of one person
with that of another, while the faces are shown from different angles, in different
lighting conditions and positions, and how these analyses can be automated. The
latest developments focus on computational ways to discover fakes (e.g. Farid 2018;
Güera, Delp 2018; Maras, Alexandrou 2019).
Running parallel to this research, the analysis of deepfakes has been taken up in
law in relation to privacy research. Aside from political destabilization and interfering
with democratic processes, one can easily imagine the problems that deepfakes may
cause in legal cases where visual documents function as empirical basis for lawsuits
(Chesney, Citron 2019a). Also, deepfakes challenge traditional forensics methods and
analysis (Burke 2006; Leone 2021). Research on deepfakes in law already recognizes
4
For some reasons, Nicholas Cage and Donald Trump are often used in examples of
deepfakes both in research literature and on mainstream media.
5
It is important to note that machine learning per se is not a technology for face manipulation.
Software like Faceswap applies machine learning methods. Deepfake software utilizes this
technology that processes image data, enabling an algorithm to learn how to encode and
decode image data in a particular computational operation, resulting in face replacement. The
same method may in principle be used to swap any other body parts or objects in motion in
videos.
492
Søren Vigild Poulsen
the dangers of fake imagery to basic human rights; one issue is the use of deepfakes
to discredit politicians and public figures; another issue is how to safeguard the
fundamental rights, identities and safety of civilians against false images and videos.
Detection software is being developed, and, as Silbey and Hartzog (2020: 961) point
out, synthetic media can be used for positive intentions as well: “The potential
upside of deepfakes is that they might help muster the political will to address the
larger, structural problems made worse by the inability to trust what we see and
hear”. I sympathize with this viewpoint, but remain a bit pessimistic about the time
perspective of any productive reactions and positive initiatives.
A semiotic technology perspective that I will present in the next section
attempts to bridge the points of view introduced with respect to both technical
and law-related aspects by exploring software for making synthetic media.
Furthermore, this approach seeks to make a new contribution by focusing on the
fact that it is due to the technological advances that deepfakes are widely circulated,
both in terms of speed and reach of circulation. As such, the present study bears
kinship with Virilio’s ( 1986[1977]) concept of ‘dromology’ that he used to diagnose
logics of speed in society.
3. Methodology
I want to apply a semiotic technology approach to study how deepfake software
incorporates semiotic resources for making videos and images of face replacements.
Semiotic technology research explores how all different types of technologies for
meaning-making enable both the production and distribution of multimodal texts
and artefacts as well as the performance of semiotic practices (e.g., Zhao et al. 2014;
Poulsen 2018). This emergent research area stems from social semiotics (Halliday
1978; Hodge, Kress 1988), and has three overlapping foci: (1) to map semiotic
resources in the technology of meaning-making; (2) to study how these are used
in semiotic practices – and more broadly, social practices; and (3) to observe the
histories, narratives and discourses of technologically mediated resources.
Using the semiotic technology approach, I will be analysing the functional
features of the user interface as signifiers with meaning potential that are built into
deepfake software programs. In relation to the present object of study, research
questions to ask about these kinds of semiotic technologies are: what resources for
altering face features are available in the software? How does this digital technology
facilitate and administrate the making of synthetic faces? How is face swapping
technology used in particular social settings?
To describe facial expressions and their meaning potentials, I include in
the semiotic description of deepfake imagery multimodal interaction studies,
Face off – a semiotic technology study of software for making deepfakes
493
microsociology, film and media studies, as well as art and cultural history with
special focus on the face. As Van Leeuwen (2005) points out, social semiotics (and
thus semiotic technology) is not a matter of “pure” theory, it needs to incorporate
other theories to provide an adequate framework for analysis of semiotic
phenomena under observation. Thus, following the mapping of resources in the
user interface of a deepfake technology, I will draw on work about faces from the
above-mentioned research fields that until now have not been included in the
discussion of deepfakes.
The following section presents a tentative description of the design and
technical features of particular deepfake software in semiotic terms, that is, as
semiotic resources and their meaning potentials for making deepfakes or synthetic
imagery. This is followed by observations from two case studies on the usage of
face swapping technology in an erotic context and a political context to illustrate
how the resources built into the software could be used, and the problematic results
that this usage contributes to.
Ideally, I would conduct a semiotic technology study of the design and use of
a particular software program, such as Faceswap: I would study how this software
program is designed as a semiotic device and the texts that could be made with
its specific features. However, it was not possible to determine the origin of the
software that was used for making the deepfake videos chosen as the objects of
this semiotic analysis. Thus, the case studies only serve to illustrate general points
about the problematic nature of such technology.
4. Analysis of the design of software for making deepfakes
The following analysis includes a description of the design of deepfake software
as semiotic technology. The subsequent sections will focus on: (1) the process and
steps of making a deepfake video; (2) the context of selecting deepfake software;
(3) the design of Faceswap’s user interface; (4) how the face replacement process
is structured in Faceswap; and (5) the semiotic operations of the software.
4.1. The process and steps of making a deepfake video
If one breaks face replacement down into its basic elements, a deepfake production
consists of four steps, regardless of the particular software one could use in the
process. As Sample (2020)6 explains:
6
Sample, Ian 2020. What are deepfakes – and how can you spot them? The Guardian 13
January 2020, was accessed at https://www.theguardian.com/technology/2020/jan/13/whatare-deepfakes-and-how-can-you-spot-them on 10 January 2021.
494
Søren Vigild Poulsen
First, you run thousands of face shots of the two people through an AI algorithm
called an encoder. The encoder finds and learns similarities between the two
faces, and reduces them to their shared common features, compressing the images
in the process. A second AI algorithm called a decoder is then taught to recover
the faces from the compressed images. Because the faces are different, you train
one decoder to recover the first person’s face, and another decoder to recover the
second person’s face. To perform the face swap, you simply feed encoded images
into the “wrong” decoder. For example, a compressed image of person A’s face is
fed into the decoder trained on person B. The decoder then reconstructs the face
of person B with the expressions and orientation of face A. For a convincing video,
this has to be done on every frame.
From a semiotic technology perspective, what is interesting when analysing
deepfake software is how the digital technology structures and facilitates the
deepfake production process to its users who are not necessarily technically skilled.
I will therefore be discussing the role and function of the software that mediates
between computational operations and visual meaning-making.
4.2. Context of selecting deepfake software
Previously, making deepfakes required advanced programming, but free or opensource software that can perform the computational operations in a few simple
steps is now widely available to the general public. The following observations
were made during the research phase for software that will enable the creation of
deepfakes:
(1) A Google search presents several options for downloading software for desk
PCs as well as recommendations indicating accessibility. There is also easy
access to apps for mobile phones that allows you to make short fake videos,
images and audio.
(2) Serious tech media evaluate the best deepfake software on the market.
(3) Many of the software programs are marketed for laypersons: users are not
required to have computer skills or special knowledge about deep learning or
computer vision.
(4) The online Gethub repositories hold several communities for deepfakes.
(5) Some software programs (e.g. Machin Tube) come with pre-loaded templates
of faces to make deepfakes of, e.g. politicians, singers and celebrities (like
Donald Trump).
(6) While this study focuses on the visual features, I also wish to mention free
technology for deepfake audios, e.g. Lyrebird AI, which has the capacity to
form sentences just upon hearing a few spoken words, thus creating a digital
voice. This software also enables overdubbing that “allows you to replace
Face off – a semiotic technology study of software for making deepfakes
495
recorded words and phrases with synthesized speech that is tonally blended
with the surrounding audio”7. So, while face replacement is normally what is
associated with deepfakes, the technology also includes audio, in which case
only the voice of someone else is dubbed. These deepfakes are called ‘voice
cloning’ and an example of this is a video from 2019 in which the voice of
Barack Obama was dubbed by actor Jordan Peale.
In the following section, I will analyse the user interface of Faceswap. Faceswap8
(2019-) is the leading open source face replacement technology developed by
Matt Tora, Bryan Lyon, and Kyle Vrooman. It is coded in the programming
language Python. The software is powered by Artificial Intelligence using so-called
Convolutional Neural Networks, and it has an active forum of volunteers and users.
I selected Faceswap over e.g. DeepFakeLab, which is more technical, since the latter
was developed for computer science students and uses command lines. Faceswap
features a graphic user interface for laypersons, and its tutorials describe the
process in a great detail, avoiding technical terms.
4.3. The user interface of Faceswap
The graphic user interface of Faceswap presents four headings (Extract, Train,
Convert, Tools), which suggests a workflow in the making of deepfake, starting
from left to right. In contrast to, e.g. DeepfakeLap which offers the users multiple
command line operations and sub functions, the process of making deepfakes is
simplified (three steps) and streamlined (unless the user wants to include specific
effects in the video, they will go with the default settings under each heading).
Regardless of the heading, the user interface is divided into two windows. In
the left-hand frame window different features relevant to the chosen heading are
provided, e.g. under Extract, file trajectories can be selected of the videos that are
made into fake images or videos. In the right-hand frame, information about and
results of the current stage of the process are displayed, e.g. the number of frames
extracted from the videos, or a preview of the results of the process.
4.4. The process of making deepfakes in Faceswap
As mentioned above, Faceswap structures the making of deepfakes in a few simple
steps. That said, the software offers the user different editing options in Faceswap
user interface, depending on the purpose of the training and type of data. Faceswap
7
8
Descript website. Accessed at https://www.descript.com/lyrebird-ai on 16 December 2019.
The official website of Facewap is https://faceswap.dev.
496
Søren Vigild Poulsen
mirrors the processes that were presented in Section 4.1. In the software tutorial9,
these steps are described as follows:
Gather data: Find videos/images of the people whose faces you want to swap
Extract: Get all faces from data, Sort, Remove incorrect faces
Training: Use the AI to learn how to swap faces (i.e., encoding and decoding)
Convert: Apply the AI on data (either new or already trained) to swap faces in the
videos
This organization of headings and the tools under each heading would, in semiotic
terms, be the overall ‘semiotic regime’ (Van Leeuwen 2005), i.e. a social regulation
of the use of semiotic resources and meaning-making processes. The design of
the user interface enforces a particular way of producing face replacement videos.
Already at this stage of the analysis, the observation can be made that, on the
one hand, the simplified workflow enables laypersons to make deepfakes, and,
on the other hand, it has consequences for the users’ understanding of what the
software does. One such effect is blackboxing, which hides the functions performed
by the technology. This is to some degree the case with Faceswap, compared to
DeepFaceLap that lets the user follow the computational processes in more detail.
Still, the blackboxing process does not run as straightforwardly as e.g. Reface, an app
that allows the user to take a selfie and embed it in a five-to-ten-second Hollywood
movie or music video. With this app, the face-swapping process is totally hidden,
and the user is only presented with a simple step-by-step process: after taking a
selfie, the image is blurred and the mobile screen displays stars tingling while the
underlying AI analyses the visual information, animating the data processing as
a magical process. Next, the user simply chooses between the selected movies
in which (s)he wants his/her face to be inserted. This video is then generated by
showing a timeline, and the final results to be saved or shared. Blackboxing seems
to be a side effect of making advanced software accessible to the general public.
After the deepfake video has been made, the Faceswap software offers ways to save
and share the content on e.g. social media networks. Thus, the software prompts
the user to distribute the deepfake imagery and participate in the circulation of the
synthetic media content. This is the basic setup.
4.5. Technical features as semiotic resources
A semiotic technology study of Faceswap would describe semiotic resources of
the software, i.e. the technicaly mediated signifiers and their meaning potentials.
However, it is difficult to describe the deepfake software and videos separately,
9
Facewap 2019 was accessed at https://forum.faceswap.dev/index.php on 10 January 2021.
Face off – a semiotic technology study of software for making deepfakes
497
as the meaning potential of the user interface is dependent on the videos that
are being made into deepfakes. Therefore, a full analysis would break down
the semiotic features of a deepfake video into different levels: the faces (of two
persons); the environment that the faces are placed in; and video editing software.
Each of these layers contains several semiotic systems that contribute to the overall
meaning-making of a deepfake video. In the following I will concentrate on the
editing software, but include aspects from the other levels. If one follows the
workflow of Faceswap, each step utilizes several semiotic resources.
4.5.1. Gathering data (representing faces)
A prerequisite of face swapping is the videos or images that function as input to
the computational process. The function of uploading face images incorporates a
semiotic resource that is essential to all visual communication, namely ‘existential
process’, i.e. the act of representing something as ‘being there’ (Boeriis 2009). Thus,
simply to be able to represent images of a source and a target face becomes the
grounds for making any deepfake, and the subsequent editing processes depend
on the usage of this resource. That the point of departure is an existing facial
representation may seem self-evident, but it is a fundamental feature for this very
reason. Highlighting the existential resource for representation also draws attention
to another feature, namely that the input faces and the instantiated resources that
they manifest are included in the meaning-making process that the semiotic software
creates. The videos are in themselves multimodal artefacts that draw on multiple
semiotic systems, and since each video typically contains extensive footage of face
angles, expressions and movements, they display a complex ensemble of instantiated
meanings. The resources of the face images exist independently of the deepfake
technology. Nonetheless, the deepfake technology depends on the textual meaning
of the faces, for such software functions as a parasite on the meaning made in the
videos; the software re-semiotizes (Iedema 2001) or transduces (Kress 2010) the
meaning potentials of the two videos by emerging two faces into one video.
4.5.2. Extraction (analytical process and reconfiguration of facial resources)
The next step is to extract face data from the still images. As Faceswap10 (2019)
describes in their tutorial on face replacement:
At the highest level, extraction consists of three phases: detection, alignment
and mask generation:
10
Facewap 2019. [Guide] Extraction – a workflow. Accessed at https://forum.faceswap.dev/
viewtopic.php?t=27z on 10 January 2021.
498
Søren Vigild Poulsen
Detection – The process of finding faces within a frame. The detector scans
an image and selects areas of the image that it thinks are faces.
Alignment – Finding the “landmarks” […] within a face and orienting the
faces consistently. […]
Mask Generation – Identify the parts of an aligned face that contains face
and block out those areas that contain background/obstructions.
[…] Extracting serves 2 main purposes:
Training: To generate a set of faces for training. These faces will also contain
the alignment information and masks which are required for training the model.
[…]
Converting: To generate an alignments file and mask for converting your
final frames. The alignment file contains information about where each face
is in each frame so that the conversion process knows where to swap faces for
any given frame.
The purpose of the extraction process is to reduce the face information in an
image to its core elements for an algorithm to reproduce this face with the least
amount of data. The reduction process may be similar to ‘abstraction’ in Peirce’s
diagrammatical reasoning (see CP; Peirce 1976). The mentioned ‘landmarks
of the face’ refer to key data points of a face that indicates facial position and
expression in each frame (see Fig. 1). These operations are the core of the semiotic
transformations in the initial face placement process.
Essentially, two semiotic ideational resources are utilized in the process: first,
categorization and segmentation of facial elements incorporate an ‘analytical
process’, i.e. a whole-parts relation (Kress, Van Leeuwen 2006); second, I understand the alignment process that tracks facial movement based on a number
of landmarks of the face as a resource for reconfiguration of semiotic multiple
different semiotic systems. Historically, studies of facial resources and their
meaning potentials have divided features of a face into different semiotic systems
or modes, e.g. the gaze (Argyle, Dean 1965; Argyle, Cook 1976; Goffman 1963;
Belting 2013; Exline, Fehr 1982), head movement (Altorfer et al. 1992, 1998;
Jossen et al. 2000), gesture (Kendon 1967; 1978; Schegloff, Sacks 1973; Goodwin
1986; Norris 2004), speech (Halliday, Hasan 1976). These semiotic systems are
heuristic analytical categories, as facial resources co-create meaning multimodally.
I would claim that this alignment is the essential part of the editing software, since
it lays the ground for training algorithms to make deepfakes. At the same time,
this exact operation creates the issues of synthetic media. Semiotically speaking,
alignment as a resource of reconfiguration includes two elements: first, by aligning
a face’s diverse signifiers, the software appropriates multiple different semiotic
systems that people use to make facial meanings. The software reconfigures
Face off – a semiotic technology study of software for making deepfakes
499
semiotic features of the face that carries an extremely rich cultural history, i.e., the
meaning potential that people throughout history have created with each facial
semiotic system (Boehm et al. 2015). Second, by incorporating all of these different
semiotic systems into landmarks that make up a map of the face, the software offers
previously unseen ways to manipulate the totality of these facial semiotic systems
and their meaning potentials all at once.
Figure 1. An example of landmarking in the process of facial alignment. Author’s own
production inspired by a figure in Wang et al.11
4.5.3. Training (semiotic principles for articulation and interpretation)
I consider the last two steps in the Faceswap workflow as having to do with overall
principles for meaning-making. The Training function concerns the establishment
of a strategy for semiosis, so the source face is interpreted as the target face. This
strategy is drawn upon in the making of a face swap, as far as it can be automated.
As such, it becomes a resource for interpretation and re-articulation (Kress, Van
Leeuwen 2001) of a semiotic production.
4.5.4. Conversion (resource integration/blending)
Faceswap differs from face alterations or improvements in face apps like TikTok,
Snapchat or Instagram which apply filters to face images because the face attributes
from the target face in a deepfake are also transferred to the source face. Therefore,
the meaning-making principle is not metaphorically from a source (face) to a target
(face) (Lakoff, Johnson 1980), the target face does not get semanticized by the
features and qualities of the source. In social semiotic terms, this process constitutes
11
Wang, Junjue; Amos, Brandon; Das, Anupam; Pillai, Padmanabhan; Sadeh, Norman; Satyanarayanan, Mahadev 2017. A scalable and privacy-aware IoT service for live video analytics.
Proceedings of the 8th ACM on Multimedia Systems Conference, 38–49; accessed on 20 January
2021.
500
Søren Vigild Poulsen
an ‘integration resource’ (Bauldry, Thibault 2006), i.e. a principle for how semiotic
resources may be combined in multimodal ensembles. One can describe this as a
blend (Fauconnier, Turner 2002) that merges different semantic domains into an
emergent meaning construction. In this analytical framework, a deepfake video
would be conceptualized as a meaning construction that blends face features of a
source with face expressions of a target. As a result, the deepfake video can show a
person doing things that are fictional (i.e. not authentic), yet trustworthy.
5. Analysis of two contexts of deepfake usage:
pornography and politics
In this section, I explore how the semiotic deepfake technology is used in diverse
semiotic practices where fake faces may serve harmful purposes and help maintain
unhealthy social practices. I consider two types of contexts in which deepfakes are
being used. Each context analysis reports on preliminary observations.
5.1. Deepfakes in pornography
A simple Google search for deepfakes reveals that deepfakes appear as a category
on major porn sites (e.g. Pornhub), as well as on sites dedicated to this kind of
pornographic content (e.g. MrDeepFake). Officially, sites like Pornhub have
banned deepfakes from their site, but videos can still be found if searched for
on these sites12. I did a search on ‘xvideo.com’ and found multiple videos of
Hollywood actresses, female singers and politicians. In the following, I present
general observations from an analysis of a deepfake video of a famous American
singer. Given the explicit and transgressive nature of the imagery, I will refrain
from describing details in the video and only focus on general observations13.
I observed that the face of the singer is shown in long camera shots, and that
the face is displayed at the centre of the footage, while the ‘body’, i.e. the body of
the porn actress, performs sexual acts, so that the viewer can clearly see and dwell
on the fake face features. As such, the face presents itself in its completeness, which
Balázs (1970[1923]), in his studies of silent films, has described as a key element
12
Ajder, Henry; Patrini, Giorgio; Cavalli, Francesco; Cullen, Laurence 2019. The state of
deepfakes: Landscape, threats, and impact. Deeptrace. 24 January. Accessed at https://sensity.
ai/reports/ on 10 January 2021.
13 I am aware of the problematic nature of analysing erotic deepfakes, since I as researcher
engage in the unsound social practice that I criticize, and as such, I take part in upholding the
market for these products. I have chosen not to disclose the name of the woman in the video
analysed. I am also aware that the viewing of such material potentially poses a criminal act.
Face off – a semiotic technology study of software for making deepfakes
501
of the close-up. The whole video is filmed from a subjective point of view, where
the woman looks directly and up close at the camera. The use of the gaze is an
essential resource that intensifies the direct visual addressing of the viewer (Kress,
Van Leeuwen 2006). The close-up, as Doane (2003) argues, “[…] embodies the
pure fact of presentation, […] of showing – a “here it is”” (quoted in Jerslev 2017:
91.) that creates a sense of direct access to reality. This suggests the fascination of
the male gaze with the face as an object; I also suspect that the depicted face and
gaze of the singer bespeaks of a video producer who takes pride in the deepfake
production. The face shot in these videos is “undisturbed”, as it were. In this regard,
the videos that swap the faces of a woman celebrity onto the body of a porn actress
seem to deviate from other pornographic videos where the face of the woman is
“manhandled” as it is not uncommon for the (female) face in porn to be slapped,
stepped on, penetrated, or covered by semen. Perhaps the facial display in the
deepfake porn is also a question of limitations to the deepfake technology, which
is not able to handle changes in face shots that include other elements. There are
small glitches in the videos where it becomes visible that the face of the actress has
been manipulated: the pixels that mark up the face area become visible and, as a
result, presumably break the illusion of the depicted fantasy.
In this context, I suggest that the use of deepfake software for making fakes
videos of celebrities, and public figures advances the already existing genres of
‘fake nudes’ and ‘celeb nudes/fakes’ that include videos of lookalikes. Also, these
videos are fed to a market for intimate videos produced without the consent of the
depicted women whose faces are placed in the context of nude scenes. Only in a
few cases, e.g. nude footages of celebrities, has the “leaking” of sex tapes been used
in personal branding as a deliberate act of empowerment. I would claim that nude
deepfakes primarily relate to hate porn and revenge porn that have devastating
consequences for the victims without whose consent pictures or videos of them are
exposed to the public at large. As pointed out by Popova (2020), the use of deepfake
nudes may at first glance appear as a fantasy of having intercourse with a celebrity,
yet the phenomenon is not like fan fiction, but is part of a transgressive act, and,
more broadly, of a misogynistic culture centring on fantasies of violating acts by
a man on a particular woman’s face and body. Furthemore, while there may be a
(legal) difference between a person making such manipulated imagery and viewers
who watch the videos, but who themselves would not create this kind of content,
the viewers of these videos, in their engagement with deepfake porn, nevertheless
uphold this social practice. By viewing, commenting and further sharing of fake
images and videos, they legitimize the existence of such deepfakes. As such, the
production and distribution of deepfake porn supports a sharing economy using
women’s sexuality as a commodity.
502
Søren Vigild Poulsen
5.2. Deepfakes in politics
Turning to politics, and looking beyond examples of political satire, one can easily
find several examples of deepfakes of political figures that supposedly say or do
something that would discredit their reputation, and this is a global issue. A recent
instance was a deepfake video of the Indian politician Manoj Tiwari that was shared
between groups on WhatsApp during the 2020 Delhi elections. In this regard, I
made the preliminary observation that Tiwari was shown talking to voters in
Haryanvi and English, but in the real video he speaks in Hindi. However, unlike
the erotic deepfake that centres on a fabricated sexual act, this political deepfake
video was not manipulated beyond a synchronization of the voice with the depicted
person’s limb movements. From watching this video, and others, it seems that it is
especially linguistic resources that can be cloned with a deepfake voice technology,
so they are used to make fake statements. The false account was accompanied by the
presentation of a particular politician’s face. In this context, the identity of a political
figure is closely associated with his or her face, as it also appears on political posters.
Like deepfake porn, the video of Tiwari utilizes the fundamental ideational resource
of representing a person’s face. By combining linguistic and gaze resources, politicians
like Tiwari may be portrayed as making outrageous or discriminating utterances that
would damage the support from their voters if these were true. As such, the purpose
of political deepfake videos and audios is an attempt to destabilize political systems
and to shatter people’s trust in democratic processes. The very act of making a person
say something that he or she has never uttered demonstrates that the use of deepfake
technology in manufacturing false images and videos in a political context could
cause double harm by contributing to the increasing disgust with politicians as well
as disbelief in trusted news sources that play a crucial role in democratic processes
(Paterson, Hanley 2020).
That false political videos pose a serious threat to democracy was indicated
by a CIA report on worldwide threat assessment published prior to the recent US
presidential election. The report states that “adversaries and strategic competitors
probably will attempt to use deepfakes or similar machine-learning technologies to
create convincing – but false – image, audio, and video files to augment influence
campaigns directed against the United States and our allies and partners” (Coates
2019: 7).14 It is not only political institutions that are affected by such problematic
use of deepfakes. The use of deepfake technology in manufacturing false images
14
Coats, Daniel R. 2019, Worldwide threat assessment of the US intelligence community
https://www.odni.gov/index.php/newsroom/congressional-testimonies/item/1947-statementfor-the-record-worldwide-threat-assessment-of-the-us-intelligence-community; accessed on
20 January 2021. Quotation refers to page 7.
Face off – a semiotic technology study of software for making deepfakes
503
and videos in a political context. This affects not only political institutions:
deepfakes can also damage the reputation of businesses and organizations,
which may have a negative impact on stock prices and bring along long-term
consequences when customers and shareholders no longer trust enterprises and
markets. The fact that deepfakes have become a phenomenon to be aware of
threatens to undermine the perceived objectivity of videos featuring politicians
and public figures. It should be a trademark of a healthy political climate to dispute
and critically discuss decision-makers’ solutions to social problems, and it is also
paramount to be able to hold politicians accountable for their actions and claims.
In this context, deepfakes align with fake news and distribution of misinformation
(Chesney, Citron 2019b).
6. Concluding remarks
This article is a report on a preliminary study of the design and use of semiotic
technology for making deepfakes. On this basis, I have argued that a semiotic
approach is needed to describe why the design and use of face replacement
technology may potentially cause disturbance, since the manipulation of the
face tampers with semiotic resources that carry an immensely rich meaning
potential. By analysing the leading face swapping software, Faceswap, as a semiotic
technology, I have tried to demonstrate how the software appropriates many
semiotic resources of a face on video and how it is able to manipulate these. Still,
much more research on the software and its usage in different contexts needs to
be carried out in this area.
When asked about their responsibility for spreading deepfakes at a tech
conference in 201915, two of the three founders of Faceswap answered that it is not
their duty to prevent this from happening. If they do not develop the technology,
someone else will come along and do this. Furthermore, they argue that a better
countermovement to the rise of deepfakes is to familiarize the general public with
this kind of software; likewise, alongside the development of deepfake software,
scientists and companies are working on software for detecting deepfakes (e.g.
Nguyen et al. 2019). While this viewpoint may be valid, the article would claim
that a joint effort by tech companies, political leaders and decision-makers, a more
critical press and research is a better solution to some of the problems discussed
in this article. Also, it is important to support the digital literacy of the general
public, so that they do not uncritically believe in images and videos circulated on
15
Tora, Matt; Lyon, Bryan; Vrooman, Kyle 2019 Faceswap 2019: Faceswap: Developing
a platform for practical Deepfakes. Recorded presentation held at Devoxx conference, 7
November Belgium was accessed at https://devoxx.be/talk/?id=103201on 20 January 2021.
504
Søren Vigild Poulsen
the internet, especially on social media. In addition to these initiatives, semiotic
technology descriptions of deepfake software can help us understand how synthetic
media is made and how society is affected when people’s faces and the values and
meanings we assign to them are fabricated. The face is the most valuable part of
who we are and who we believe others to be.
Acknowledgements: I would like to thank the editors, the anonymous reviewers, members
of Centre for Multimodal Communication (Morten Boeriis and Theo van Leeuwen), as
well as Gunhild Kvåle for suggestions on how to improve the manuscript. I alone am
responsible for all claims presented in this article.
References
Argyle, Michael; Cook, Mark 1976. Gaze and Mutual Gaze. Cambridge: Cambridge
University Press.
Argyle, Michael; Dean, Janet 1965. Eye-contact, distance and affiliation. Sociometry 28(3):
289–304. https://doi.org/10.2307/2786027
Altorfer, Andreas; Goldstein, Michael J.; Miklowitz, David J.; Nuechterlein, Keith H.
1992. Stress-indicative patterns of non-verbal behaviour: Their role in family interaction. British Journal of Psychiatry 161(18): 103–113. https://doi.org/10.1192/
S0007125000297122
Altorfer, Andreas; Käsermann, Marie-Louise; Hirsbrunner, Hanspeter 1998. Arousal and
communication: The relationship between nonverbal, behavioral, and physiological
indices of the stress response. Journal of Psychophysiology 12: 40–59.
Balázs, Béla 1970[1923]. The close-up/The face of man. In: Balázs, Bela (ed.), Theory of the
Film: Character and Growth of a New Art. New York: Dover, 52–89.
Baldry, Anthony; Thibault, Paul 2006. Multimodal Transcription and Text Analysis: A
Multimedia Toolkit and Coursebook. London/Oakville: Equinox.
Belting, Hans 2013. Faces: Eine Geschichte des Gesichts. Munich: C. H. Beck.
Bitouk, Dmitri; Kumar, Neerah; Dhillon, Samreen; Belhumeur, Peter; Nayar, Shree K. 2008.
Face swapping: Automatically replacing faces in photographs. ACM Transactions on
Graphics 27(3): 1–8. https://doi.org/10.1145/1360612.1360638
Blanz, Volker; Scherbaum, Kristina; Vetter, Thomas; Seidel, Hans‐Peter 2004. Exchanging
faces in images. Computer Graphics Forum 23: 669–676. https://doi.org/10.1111/j.14678659.2004.00799.x
Boehm, Gottfried; Budelacci, Orlando; Di Monte, Maria Giuseppina; Renner, Michael
(eds.) 2015. Gesicht und Identität/Face and Identity. Paderborn: Wilhelm Fink Verlag.
https://doi.org/10.30965/9783846758151
Boeriis, Morten 2009. Multimodal Socialsemiotik og Levende Billeder. [Multimodal social
semiotics and moving images. Unpublished PhD thesis.] Odense: University of Southern
Denmark.
Burke, Peter 2006. Eyewitnessing the Uses of Images as Historical Evidence. London: Reaktion
Books.
Face off – a semiotic technology study of software for making deepfakes
505
Chesney, Robert; Citron, Danielle K. 2019a. Deepfakes: A looming challenge for privacy,
democracy, and national security. California Law Review 107(6): 1753–1820.
Chesney, Robert; Citron, Danielle K. 2019b. Deepfakes and the new disinformation war:
The coming age of post-truth geopolitics. Foreign Affairs 98(1): 147–155.
CP = Peirce, Charles S. 1931–1958. Collected Papers of Charles Sanders Peirce. Cambridge:
Harvard University Press. (Vols. 1–6, Hartshorne, Charles; Weiss, Paul, eds., 1931–1935;
vols. 7–8, Burks, Arthur W., ed., 1958.) [In-text references are to CP, followed by volume
and paragraph numbers.]
Dale, Kevin; Sunkavalli, Kalyan; Johnson, Micah K.; Vlasic, Daniel; Matusik, Wojceich;
Pfister, Hanspeter 2011. Video face replacement. ACM Transactions on Graphics
(SIGGRAPH), 30. https://doi.org/10.1145/2070781.2024164
Djonov, Emilia; Van Leeuwen, Theo 2018. The power of semiotic software: A critical
multimodal perspective. In: Flowerdew, John; Richardson, John E. (eds.), The Routledge
Handbook of Critical Discourse Studies. New York: Taylor & Francis, 566–581. https://
doi.org/10.4324/9781315739342-39
Doane, Mary Ann 2003. The close-up: Scale and detail in the cinema. Differences: A Journal
of Feminist Cultural Studies 14(3): 89–111. https://doi.org/10.1215/10407391-14-3-89
Exline, Ralph. V.; Fehr, Barbara J. 1982. The assessment of gaze and mutual gaze. In:
Scherer, Klaus R.; Ekman, Paul (eds.), Handbook of Methods in Nonverbal Behavior
Research. Cambridge: Cambridge University Press, 91–135.
Farid, Hany 2018. Digital forensics in a post-truth age. Forensic Science International 289:
268–269.
Fauconnier, Gilles; Turner, Mark 2002. The Way We Think: Conceptual Blending and the
Mind’s Hidden Complexities. New York: Basic Books.
Goffman, Erving 1963. Behavior in Public Places. New York: Free Press.
Goodwin, Chuck 1986. Gestures as a resource for the organization of mutual interaction.
Semiotica 62(1/2): 29–49. https://doi.org/10.1515/semi.1986.62.1-2.29
Güera, David; Delp, Edward J. 2018. Deepfake video detection using recurrent neural
networks. 15th IEEE International Conference on Advanced Video and Signal Based
Surveillance (AVSS), 1–6. https://doi.org/10.1109/AVSS.2018.8639163
Halliday, Michael A. K. 1978. Language as Social Semiotic. London: Edward Arnold.
Halliday, Michael A. K; Hasan, Ruqaiya 1976. Cohesion in English. London: Longman.
Hodge, Robert; Kress, Gunther 1988. Social Semiotics. Cambridge: Polity Press.
Iedema, Rick 2001. Resemiotization. Semiotica 137(1/4): 23–39. https://doi.org/10.1515/
semi.2001.106
Jerslev, Anne 2017. SKAMs ‘lige her’ og ‘lige nu’: Om SKAM og nærvær. [‘Right here’ and ‘right
now’ in SKAM: On SKAM and presence.] Nordisk Tidsskrift for Informationsvidenskab Og
Kulturformidling 6(2): 75–81. https://doi.org/10.7146/ntik.v6i2.99080
Jossen, Stefan; Käsermann, Marie-Louise; Altorfer, Andreas; Foppa, Klaus; Zimmermann,
Henrich; Hirsbrunner, Hans-Peter 2000. The study of emotional processes in communication: Peripheral blood flow as an indicator of emotionalization. Behavior Research
Methods, Instruments, & Computers 32: 47–55. https://doi.org/10.3758/BF03200787
Kendon, Adam 1967. Some functions of gaze-direction in social interction. Acta
Psychologica 26: 22–63. https://doi.org/10.1016/0001-6918(67)90005-4
506
Søren Vigild Poulsen
Kendon, Adam 1978. Looking in conversation and the regulation of turns at talk: A
comment on the papers of G. Beattie and D. R. Rutter et al. British Journal of Social
and Clinical Psychology 17: 23–24. https://doi.org/10.1111/j.2044-8260.1978.tb00891.x
Korshunova, Iryna; Shi, Wenzhe; Dambre, Joni.; Theis, Lucas 2017. Fast face-swap using
convolutional neural networks. 2017 IEEE International Conference on Computer Vision
(ICCV). Venice, 3697–3705. https://doi.org/10.1109/ICCV.2017.397
Kress, Gunther 2010. Multimodality – A Social Semiotic Approach to Contemporary
Communication. London, New York: Routledge.
Kress, Gunther; Van Leeuwen, Theo 2006. Reading Images – The Grammar of Visual Design.
(2nd ed.). London: Routledge. https://doi.org/10.4324/9780203619728
Lakoff, George; Johnson, Mark 1980. Metaphors We Live by. Chicago, London: University
of Chicago Press.
Lees, Dominic; Bashford-Roger, Tom; Keppel-Palmer, Marcus 2021. The digital resurrection
of Margaret Thatcher: Creative, technological and legal dilemmas in the use of deepfakes in screen drama. Convergence: The International Journal of Research into New
Media Technologies 27(4): 954–973. https://doi.org/10.1177/13548565211030452
Leone, Massimo 2021. From fingers to faces: Visual semiotics and digital forensics. Int.
Journal of the Semiotics of Law 34: 579–599. https://doi.org/10.1007/s11196-020-09766-x
Maras, Marie-Helen; Alexandrou, Alex 2019. Determining authenticity of video evidence
in the age of artificial intelligence and in the wake of deepfake videos. The International
Journal of Evidence 23(3): 255–262. https://doi.org/10.1177/1365712718807226
Nguyen, Thanh. T.; Nguyen, Cuong M.; Nguyen, Dung T.; Nguyen, Duc T.; Nahavandi,
Saeid 2019. Deep learning for deepfakes creation and detection: A survey. arXiv preprint
arXiv: 1909.11573: 1–16.
Norris, Sigrid 2004. Analyzing Multimodal Interaction: A Methodological Framework. New
York, London: Routledge. https://doi.org/10.4324/9780203379493
Paterson, Thomas; Hanley, Lauren 2020. Political warfare in the digital age: Cyber
subversion, information operations and ‘deepfakes’. Australian Journal of International
Affairs 74(4): 439–454. https://doi.org/10.1080/10357718.2020.1734772
Peirce, Charles S. 1976. The New Elements of Mathematics by C. S. Peirce (Eisele, Carolyn,
ed.). The Hague: Walter De Gruyter.
Popova, Milena 2020. Reading out of context: Pornographic deepfakes, celebrity and
intimacy. Porn Studies 7(4): 367–381. https://doi.org/10.1080/23268743.2019.1675090
Poulsen, Søren Vigild 2018. Becoming a semiotic technology – a historical study of Instagram’s tools for making and sharing photos and videos. Internet Histories 2(1/2): 121–
139. https://doi.org/10.1080/24701475.2018.1459350
Schegloff, Emanuel; Sacks, Harvey 1973. Opening up closings. Semiotica 8: 289–327.
https://doi.org/10.1515/semi.1973.8.4.289
Silbey, Jessica; Hartzog, Woodrow 2019 The upside of deepfakes. Maryland Law Review
78(4): 960–966.
Stöckl, Hartmut; Caple, Helen; Pflaeging, Jana (eds.) 2019. Shifts towards ImageCentricity in Contemporary Multimodal Practices. London: Routledge. https://doi.
org/10.4324/9780429487965
Van Leeuwen, Theo 2005. Introducing Social Semiotics. London: Routledge. https://doi.
org/10.4324/9780203647028
Face off – a semiotic technology study of software for making deepfakes
507
Virilio, Paul 1986[1977] Speed and Politics: An Essay on Dromology. New York: Semiotext(e).
Zhao, Sumin; Djonov, Emilia; Van Leeuwen, Theo 2014. Semiotic technology and practice:
A multimodal social semiotic approach to PowerPoint. Text & Talk 34(3): 349–375.
https://doi.org/10.1515/text-2014-0005
Face off – семиотическое технологическое исследование
программного обеспечения для создания подделок
Deepfakes, алгоритм, который накладывает лицо одного человека на лицо другого
в изображениях и видео – цифровая технология, которая может коренным образом изменить нашу веру в визуальную модальность и представляет серьезную
опасность для нашей культуры, ориентированной на изображения. Эти технологии
не просто настолько продвинуты, что практически не позволяют усомниться в
их подлинности, но также всё более доступы непрофессионалам, которые могут
использовать их в самых разных целях, включая преступные намерения вроде
порнографии в целях мести и кражи личных данных. В связи с этим исследование
семиотического потенциала таких подделок является своевременным и крайне
необходимым. В статье представлена семиотическая технологическая перспектива –
новое направление социосемиотики, анализирующее, каким образом программное
обеспечение deepfake разрабатывается и используется в качестве семиотического
ресурса в эротических и политических контекстах. Выдвигается предположение,
что такое программное обеспечение способно подчинить все знаки лица и их
культурную историю. Следовательно, семиотические операции этой технологии
подготавливают почву для синтетического изображения лица.
Face off – süvavõltsingute tegemiseks kasutatava tarkvara semiootiline
tehnoloogiauuring
Süvavõltsing, algoritm, mis asetab kujutistel ja filmides ühe isiku näole teise isiku näo,
on digitehnoloogia, mis võib fundamentaalselt muuta meie usku visuaalsusmodaalsusesse
ja millega kuvandikeskset kultuuri silmas pidades seega kaasnevad ohtlikud tagajärjed.
Sellised näotõlked pole mitte ainult niivõrd edasijõudnud tasemel, et inimestel on praktiliselt võimatu otsustada, kas tegemist on võltsinguga, vaid see tehnoloogia on muutumas
kättesaadavaks mitteerialainimestele, kes vähestest või puuduvatest arvutioskustest hoolimata võivad neid kasutada igasugustel eesmärkidel, sealhulgas kuritegelikel nagu seda on
kättemaksuporno ja identiteedivargus. Seega on süvavõltsingute semiootilise potentsiaali
vaatlemine õigeaegne ning oluline.
Artiklis pakutakse välja semiootilise tehnoloogia vaatenurk, s.t sotsiosemiootikas
valdkonnana esile kerkiva tähendusloometehnoloogia vaatlus, esitamaks käimasoleva
uuringu tulemusi selle kohta, kuidas on disainitud süvavõltsingutarkvara ning kuidas seda
erootilistes ja poliitilistes kontekstides semiootilise ressursina kasutatakse. Artikkel väidab,
et selline tarkvara on suuteline anastama kõik tähistajad ning nende kultuuriloo. Järelikult
valmistavad selle tehnoloogia semiootilised operatsioonid ette pinnast sünteetiliste
näokujutuste problemaatilistele perspektiividele.
508
Søren Vigild Poulsen
Sellest lähtuvalt kutsutakse artiklis üles kriitilisusele päevasündmuste visuaalsete
representatsioonide automaatselt tõe pähe võtmise suhtes ning tuletatakse meelde, et
tehnoloogia on kaasatud visuaalsete artefaktide ebasoovitavatesse jagamispraktikatesse,
mis kahjustavad näo rikkalikku tähenduspotentsiaali.
Augmented facets: A semiotics analysisSign
of augmented
reality49(3/4),
facial eff
ects509–526
509
Systems Studies
2021,
Augmented facets:
A semiotics analysis of augmented reality
facial effects
Federico Biggio1
Abstract: Augmented reality facial effects represent a new trend in social media
communication based on ‘short forms’. The article proposes a tripartite analysis: a
semiotic analysis of digital facial effects used to empower the natural users’ faces; a
deconstructionist analysis of Spark by Meta, one of the major software applications
to create such effects and, finally, a critical reflection on the practices prescribed by
Spark and the stereotypical aesthetics of augmented selfies. The conclusion states
that such forms of augmented reality effects must be conceived not as oriented
to the cognitive improvement of users’ performance but rather as forms of users’
empowerment and self-awareness.
Keywords: face filters; facial effects; Instagram; Spark; augmented reality
1. Introduction. A new trend in social media communication
The article aims to account for an emerging form of digital creativity detectable
in social media contexts and especially on Instagram. The article’s specific object
of inquiry is the digital facial effect (also called ‘filter’) used by social media users
to enhance in different ways their natural faces’ images in selfie production and
sharing.
The semiotic analysis will focus on a specific typology of textual forms, which
can be classified in the category of ‘short forms’ (Pezzini 2002; Montani 2020).
Brevity should not be understood only as a material requirement (for example,
the 13 seconds of Instagram Stories), but as a predominant textual form in social
media contexts that testifies to a “compositional mode that loads of meaning in the
first place the structuring of the text” (Montani 2020: 8).
1
Department of Excellence of Philosophy and Educational Sciences, University of Turin,
Italy; Department of Hypermedia, University of Paris VIII Vincennes-Saint Denis; France;
e-mail address: federico.biggio@unito.it.
https://doi.org/10.12697/SSS.2021.49.3-4.13
510
Federico Biggio
Among these ‘short forms’ we can insert licitly the so-called “selfie”, which
Del Marco (2017: 21) defines as a “reflexive shot, a photograph that a person
makes to himself/herselves – in some cases with others and among others”, by
emphasizing both the object-reference of the visual representation and the social
practice connected to its production: “[the selfie is] realized with a mobile device
like a smartphone or a tablet or through a webcam, sometimes through the use of
mirrors, object of sharing on social networks”.
Short forms addressed in this article are selfies and Instagram Stories made
with AR facial effects. There are digital facial effects of all kinds, ranging from
beautification (effects that usually soften the skin, enlarge the eyes and make the
face look thinner), to animal-like masks (effects that include, for example, animal
ears and snouts), face swaps (an effect that swaps the faces of the people portrayed)
and disfiguration of the face (similar to distorting mirrors). This is a global trend
in selfie production that calls out for the visual semiotic analysis to understand the
meanings of such “augmented images” as well as, from another perspective, the
meaning-making performative practices of production and usage of such effects.
Mainly, AR facial effects work by adding partially transparent virtual layers to
augment aesthetically the physical user’s face displayed as a digital image within
the device’s screen. Undoubtedly, they constitute a form of entertainment and
engagement for social media users. Nonetheless, they prompt the artists’ creativity
as several of these have begun to create facial effects by rediscovering surrealistic
aesthetics2 that, in turn, have also enticed common users to engage with the
creation of digital facial effects. This has become possible since the release of a
series of software development kits, like Spark, which provide users with tools
to detect, elaborate, manipulate and augment their selfies by creating their own
personalized effects.
The analysis attempts to understand what may be considered as the “cause”, or
the “intentional reason” driving the users’ agency to the production/adoption of
facial effects for social media communication, in order to be used to overlap with the
floating signifier of the selfie-image and, in turn, to achieve effects on the audience,
in a regime of communication capitalism (Salisbury et al. 2017; Fuchs 2020).
At the same time, by simultaneously looking to the visual level of the augmented
image and to the meaning-making performative practices of production and
usage, facial effects production points to an “overlapping mechanism” carried
out by the cooperation between the creative user and the computational entity,
which encompasses the operation of computer vision and facial tracker engines. In
particular, we focus on a specific software, Spark by Facebook (other popular apps
like Facetune, Meitu and SwapCam, mainly focus on beautification).
2
See https://www.vogue.it/fotografia/article/the-art-of-filters; accessed on 22 February 2021.
Augmented facets: A semiotics analysis of augmented reality facial effects
511
The production and usage of such facial effects relies on augmented reality,
that is, the technology providing tools such as face models and trackers. Once
launched, Instagram Stories’ software detects, thanks to specific algorithmic
functions, the user’s face within the field of reality framed through the camera
of the device (which previously has been instructed to recognize and understand
through machine learning tecniques). In visual applications, for example, the
computational instance makes an ecphrastic description of the surfaces (twodimensional or three-dimensional) that are present in the real environment.
Simultaneous Localization and Mapping technologies, specifically, are the most
important ones for augmented reality. They work by obtaining visual data from
the physical environment in the form of points to feed the data into the machines.
Instagram Stories’ software, for example, detects input from predefined vector
forms (for example, an oval to detect the face).
However, augmented reality, as it was conceived in its early days and as many
commercial applications interpret its communicative potentialities, is by no
means a medium for mere entertainment. Or rather, it was initially conceived
as a worker-oriented technology related to the “personal empowerment”
(Azuma 1997; Mann 1998) of the operational/cognitive performance of the
worker3; around 2016, augmented reality by marketing strategies found its own
medium specificity in popular applications able to increase users’ knowledge of
the surrounding environment by offering them, at the same time, an attractive,
engaging, pervasive and emotional experience. Today, however, as our daily
experience testifies, it is hard to conceive selfie effects in terms of forms of personal
empowerment, intended as such. Augmented reality effects are daily used to make
the shared images more attractive and appealing: for this reason, the so-called
“Instagrammers”, and more specifically the “influencers” and the “digital artists”,
are those who, in principle, make the most use of these compositional modes for
the production and sharing of their own content.
We have to understand, therefore, to what extent augmented reality facial effects
for selfies are conceivable as forms of augmentation, that is, of personal empowerment.
We will try to answer this question, firstly by asserting that the augmentation does not
refer only to the rational and logical empowerment of the user.
In fact, it is not the knowledge about one’s face to be increased through the
use of a selfie effect: these effects, rather, by virtue of their ability to increase the
3
The first augmented reality device prototypes were designed to increase the performance
of workers and military personnel. The development of augmented reality has not, on the
other hand, lost this identity: even after the failure of Google Glass for the public market, these
devices have continued to be adopted in specialized fields, such as that of the manufacturing
industry.
512
Federico Biggio
degree of communicative effectiveness of the most important communication
device in social media (i.e. the face), determine (or at least are potentially able
to determine) a techno-aesthetic experience from which the user emerges
continuously transformed.
2. Empowering the face
First of all, effects in augmented reality that are taken into account here concern
only the augmentation of the face; but Spark (as well as Snap Kit by Snapchat)
allows to create virtual overlaps in augmented reality of all kinds, which pose a
difficult challenge to the categorizing instinct of semiotics. The question arises
what criteria to use to bring together facial effects to add virtual make-up effects,
psychedelic light in the selfie background and somatic traits of the opposite sex
to user’s face?
At the formal level, we might distinguish between aware and un-aware effects,
figurative and non-figurative ones, in-real-time and in-hindsight-applied ones
(Schipper 2018). Nonetheless, it is not possible to detect any kind of plastic
continuity at the visual level of the augmented image, although it is possible to
read these phenomena as expressions of a dominant communication logic in the
production and sharing of selfies in social media: a “cumulative logic” (Rettberg
2014) that satisfies the needs of a culture of the spectacularizing of individualism
(the cumulative process refers precisely to the constant production of visual texts
which, once stored into one’s page, let detect a digital identity of an individual user).
Moreover, a specific theme has been also that of the users’ empowerment
in relation to the possibility of manipulating their own project-oriented digital
identity. Rettberg (2014: 12; my emphasis, F. B.) explained that “the ease and
inexpensiveness of deleting digital images and taking new ones allows us to control
the way we are represented to a far greater degree than in a photobooth or holding
an analogue camera up to a mirror”.
However, beyond the values of optimization and simplification that these
tools bring to the praxeology of the user, the concept of ‘empowerment’ must be
specified.
Rettberg also proposed the idea that the transformation of users’ faces occurs
mainly along with a certain degree of “strangeness”: this is due to the fact that
facial effects serve to gain one’s digital image on social media, through a process
of distinction from conventional selfies. For instance, when a face effect allows
to augment selfies with rainbows coming out of the mouth, it gives the image
a strangeness which makes subjecting oneself to other conventions of self-
Augmented facets: A semiotics analysis of augmented reality facial effects
513
presentation seem pointless. The peculiar aspect of this practice is that it creates
room for self-expression and experimentation beyond the norms. This is probably
the reason for which facial effects fascinate us: according to Rettberg (2014: 26),
they give the image a strangeness that defamiliarizes our lives and hence enables a
critical and self-aware look at them.
The strangeness of facial effects, in this context, can be conceived as a vehicle of
sense of “estrangement”. For Shklovskij, estrangement is the process by which the
poetic image “makes strange” the usual, the everyday, by operating a shift from the
usual semantic series in which it is placed and presenting it in a new light, therefore
transforming it into a poetic object: it has the function of “disautomatizing” the
perception of reality by presenting new sides at different levels and according to
different techniques (Šklovskij 1968[1917]).
The concept, then taken up by Brecht (1975), denotes also a “distancing” effect
from the object of vision that arouses in the observer, an effect which no longer
identifies realistically with the character, but captures its representativeness,
its being the object of an aesthetic construction. In this last sense, the term
‘estrangement’ has to be conceived as a process by which users become able to
“distance” themselves from the potential operation that AR facial effects provide for,
to observe themselves critically from the outside, to develop a strategic competence
to employ such effects for a project-oriented objective.
Our aim here is not to understand the augmented image produced by the
selfie created as an artistic or poetic text; this would require an aesthetics that is
not pertinent here. Instead, we want to underline the meta-operative component
of linguistic production (Garroni 1977, 2005; Montani 2015) that in the specific
practice of selfie production is expressed in the evaluation of a series of expressive
choices (digital effects), provided by a wide repertoire of linguistic tools, to be
used in selfie production practice. At this stage, the user “distances” him/herself
from his/her image reflected on the screen of the device, imagines its aesthetic
manipulation and, through the use of virtual facial effects, realizes it according to
pre-constituted paths.
So, from a practical point of view, the act of sharing “augmented selfies” can
be understood as a way to control the visual discourses of one’s own, and hence,
the gaze of the audience, as today digital self-representation can be intended as
“conversational” (Rettberg 2014: 1). In fact, it cannot be denied that the digital
availability of any content to improve the face visually is an aspect that, if correctly
interpreted and adopted, may be decisive in empowering the communicative
performance of a user on the network.
From a semantic point of view, the collaborative operation of a user and a
computational instance determines an empowerment of the user’s experience as
514
Federico Biggio
the immense availability of expressive material in the Web (and, at the same time,
the ease of use of linguistic tools) can be seen as an enhancement of the metaoperational possibilities of the user who, using effected selfies, simultaneously
enables a critical and self-aware gaze on himself/herself4.
For instance, the effects (then banned) that allow users to visualize the
appearance of their faces in case of a hypothetical facial surgery or as a forecast
to sex change can represent an example of this tendency. In these cases, the
augmented reality effects allow prediction, realize pre-visions, by visually realizing
subjective or inter-subjective planning, but they also allow us to evaluate this
prediction critically by deciding whether to realize it in the “real world”. This mode
of effects’ design is even more evident in sponsored filters and lenses to augment
users’ face and surroundings with brand logos and paraphernalia.
In this regard, the interpretation given to these phenomena by Barnard (2016)
proves to be of interest. Barnard focused on the practice of the selfie in the light
of ideological and social meanings of a certain culture of individualism, and he
exposes a material and affective conception of empowerment, particularly a postfeminist one. The selfie, according to him, functions as “an expressive celebration
of the self by seeking to capture embodied experience” (Barnard 2016: 66) and as “a
techno-cultural form” (Barnard 2016: 82): in the case of post-feminism, individuals
who seek beauty alterations are necessarily empowered with more agency than
previous generations, just because one can deduce, from the practice of their
self-presentation, a prior understanding and problematization of the dominant
aesthetic canons.
Other interpretations of effected selfies have been advanced, for, instance by
Chae (2017), postulating that human beings have a basic instinct to compare
themselves with others and to evaluate their own abilities and opinions and by
proving that the selfie-augmenting phenomenon might negatively influence both
presenters and audiences in online self-presentation, as “for presenters (individuals
who frequently edit their own selfies) the discrepancy between reality and ideal
might be problematic […]. Individuals who often take selfies are likely to have an
idealized virtual self-image, which might create discrepancy between reality and
ideal” (Chae 2017: 375).
Thus, although by using face effects users might subject themselves to the
gaze of the technology of augmentation and the stereotypical norms of selfpresentation (so, face filters would not empower the user); on the other hand,
4
However, Rettberg (2014: 26) stated later: “Instagram-style filters may make our selfies and
photos of our everyday life seem unfamiliar, but the filter itself is repeated so often that the
defamiliarisation effect wears off and becomes a cliché”.
Augmented facets: A semiotics analysis of augmented reality facial effects
515
though, augmented reality effects also instigate new ways of selfie creation that
may be experienced as liberating one’s digital image.
3. Instagrammers and Spark users
Before proceeding, we must define the specific properties of the user who uses
virtual facial effects to empower his/her own selfie image. These users may be
seen as a type of the ‘prosumer’. Jenkins coined this concept in 2013, by retrieving
from Toffler, denoting specific creative practices which concerned the public not as
simply a consumer of pre-packaged messages, but as a subject who simultaneously
consumes and produces contents circulating in media scenario. Mainly, prosumers’
prototypical textual production practice involves the appropriation of multimedia
contents on the Web and their remix through different types of software that
lead to the creation of syncretic text forms, such as fandom movies, memes or
digital effected selfies. Instagram users are prosumers as they use pre-packaged
effects to augment their own selfies. According to Lev Manovich (2017: 117)5,
Instagrammers can be said to own “the means of cultural production”: this means,
however, not only simply owing mobile phones and apps but, more importantly,
“having skills in using these apps, understanding Instagram’s rules and strategies
for creating popular feeds, and being able to apply well these strategies in practice”.
However, users augment their own selfies with digital effects which have in
turn been produced by others. Prosumers are both Instagram users who adopt
facial effects and Spark users who create such effects using a platform for content
creation. Spark users can be equally conceived as the receivers of a commercial
product, which allows them to create an augmented selfie and share it with the
digital users’ audience. They are equally spectators of a marketing strategy (and this
is evident by looking at claims such as “Augment your space. Augment your style.
Augment your sound. Augment your world”, where ‘your’ explicitly refers to the
subjective dimension of the experience in augmented reality6). Instagram users,
on the other hand, are two-fold spectators, as they receive two products: the one
produced by the company and the one produced by Spark users.
5
Manovich, Lev 2017. Instagram and Contemporary Image. Available at: http://manovich.
net/index.php/projects/instagram-and-contemporary-image.
6
See https://sparkar.facebook.com/ar-studio/; accessed on 22 February 2021.
516
Federico Biggio
4. The face–software communication
However, by looking strictly at the effects’ production, the prosumer is not the
only condition for facial effects use. We have to consider two further aspects of
the phenomenon: on the one hand, the augmented reality software activity; on the
other hand, the aesthetic canons imposed by a certain socio-cultural community.
As regards the former aspect, Spark represents both the software environment
where effects are being created as well as the set of technical features (as face models
and trackers) by means of which Instagram users augment their own selfies.
The Spark7 interface (Fig. 1) features an intuitive programming environment
and a set of tools that allows prosumers to use a set of computational tools to create
digital effects. It is characterized by a central section (called ‘viewport’) where
“Euclidean” space is represented and a vectoral representation of the device camera
and of the effect take place, with a window (called ‘simulator’) showing how the
effect would look on a mobile device. Here, a plane is visualized in a 3D space,
located in front of the device’s camera, as well as a directional ambient light and
a microphone: it is exactly in this interstitial space, organized in hierarchical and
overlapped layers, that the “spectacle of augmentation” can take place.
Figure 1. The interface of Spark AR Studio (https://sparkar.facebook.com/ar-studio/
learn/tutorials/introduction-to-spark-ar-studio; accessed on 22 February 2021).
7
By replying the success of Lens Studio by Snapchat, Zuckerberg’s company released the
Camera Effect Platform for AR Studio in 2017. The article focuses on the analysis of Spark AR
Studio, the software developed by Instagram and then acquired by Facebook.
Augmented facets: A semiotics analysis of augmented reality facial effects
517
Then, on the left, a double menu serves to store the objects, the trackers and the
lights (the scene panel), and, finally, the materials, such as the textures and the
scripts (the assets panel) imported by the prosumer. On the right-hand side, a
multi-function panel displays functionalities according to the object selected in the
right-hand panels. Finally, on the lower level, there is the patch editor, the section
that serves to add interactivity and animation to effects by means of a system of
boxes and links.
The main operation scheme to create a facial effect in Spark is to combine an
‘object’ – a face tracker – with a ‘material’ – an asset – by means of a face mesh.
Once it has been uploaded, the face tracker becomes able to detect users’ facial
movements framed by the device’s camera and then associated to the face gestures
recognition engine. It responds to users’ interactivity by displaying the digital
effect. In this sense, we can state that Spark is featured by (and imposes on the
productive process) a linguistic paradigm whose syntagmatic elements correspond
to each body movement and gesture that users must actualize by interacting with
the machine in a meaningful way.
Finally, a face mesh – that is a 3D model of a face – reconstructs users’
expressions according to the detections made by the face tracker. In the assets
panel, an image, an object or a texture, must already be associated to a material, in
order to be later assigned to the face mesh and displayed in the viewport. Also in
this case, the creative process has to be conducted toward a precise and standard
format. For instance, when a texture has been uploaded, the user has to choose the
material and the shader type by selecting among different grades of transparence:
if the retouching mode has been chosen, the texture assumes the form of a patina,
similar to the beautification filters to cover the imperfections of the skin. Gradually,
from face paint mode to the blended one, the visibility of the effect becomes more
pronounced. Although it is possible to regulate the grade of opacity of the layer for
each type, the command name is still indicative of the previewed use.
The first pivotal point here is the detection of an actual linguistic code
detectable through software analysis. The admitted facial gestures shape a sort of
“syntax of interactivity” of the face–software communication.
For instance, it is possible to set an object appearing at the tapping of the screen
or when the user face assumes a specific expression: facial movements such as
eyes blinking, eyebrows raising or lowering, head rotation and shaking, or facial
expressions such as a kissing or a smiling face, but also voice for audio inputs,
are becoming ever more standardized gestures that spread in digital culture by
stereotyping facial communication in social media contexts. An example is the
so-called “side profile check”, that is, the rotation of the head in front of the camera
to show clearly the two profiles of the face.
518
Federico Biggio
In semiotic theory, communication between two instances occurs only when
both instances own knowledge (an “encyclopedic” one) that allows them to (1)
formulate linguistic occurrences and (2) understand those formulated by the
interlocutor.
Beyond the automatized-like effect of the users’ natural communication
with artificial-intelligence engines, adhering to certain interactivity schemes
prescribed by the software (or by the effect), result, as a consequence, in the
increase of the awareness of mediation, as well as the development of a critical
self-judgment of one’s appearance that is certainly more pertinent in the field of
social and psychological sciences. This self-awareness plays a central role in the
competitiveness between users of social networks (Chae 2017).
Although the repetition of a movement in front of a camera may recall the
modus operandi of the so-called “challenge”, the performance of these gestures does
not aim to take an active part in a process of collective support of ideologically
featured causes, but aims to express the users’ unique and unrepeatable aesthetic
appearance through more or less objective evidence, in the production of visual
texts (e.g. selfies) that can be sanctioned by the community of users both positively
and negatively. In fact, the creation/adoption of specific facial effects can also be
sanctioned by the social community as well as by the owners of the platform (i.e.
Facebook). For example, it is a fact attested in psychological studies (Ramphul,
Mejias 2018) that the spread of facial effects has led more and more users, especially
the youngest, to evaluate the possibility of plastic surgery that allows them to touch
up the shape of the eyes (that is, to have “cat eyes”), to enlarge the volume of the lips
or to shrink the nose. To denote this type of “disorder” the expression ‘Snapchat
dysmorphia’ has been coined. These considerations have led to several negative
sanctions of facial augmentation practices with virtual effects, such as banning
of cosmetic surgery effects as created by Daniel Mooney, prohibited since these
promoted plastic surgery and harmed people’s mental health (Fig. 2).
Augmented facets: A semiotics analysis of augmented reality facial effects
519
Figure 2. Daniel Mooney’s plastic surgery effects, retrieved from https://www.insider.com/
instagram-cosmetic-surgery-filters-removed-2019-10; accessed on 22 February 2021.
5. The awareness of the empowerment and
the un-awareness of the mediation
We might distinguish between the awareness of the empowerment and the awareness of the mediation. The former refers to the recognizing of the effect within a
selfie-image: it occurs always, as the effect is created/adopted by users themselves.
The latter, however, refers to the recognizing of the presence of an effect within
another user’s photo (moreover, to the “presence” of the computational activity
which may be attested just from the weirdness of certain effects): the awareness
of the mediation is an unnecessary condition since many effects, e.g. make-up or
beauty ones, try to achieve maximum possible truthfulness: “Digital images start
to look “natural” to us; moreover, even technologically embedded visual effects,
including those that heavily modify facial reproductions, start to acquire an aura
of ineluctability” (Leone 2020: 6). FaceTune and Meitu represent the apex of this
trend.
Hence, the negative sanction of the community, similarly to the psychological
one, does not move from the assumption that the user is unaware of the
empowerment. Digital users live as “immersed” within a software culture, they
know how programs work, how to manipulate computational elements and how
to act synergically with algorithmic entities in order to express themselves in social
media contexts (Manovich 2017). Rather, the users are aware of the empowerment,
but they are un-aware of the mediation. The negative sanction moves from the
assumption that the user is un-aware of the mediation, that is, the presence of a
520
Federico Biggio
database which has been created by Facebook, the in-between actor which provides
the tools for the empowerment.
By searching for an ever-better effects and by confronting those employed by
themselves with those used by others, users would risk to lose the grasp of the
all-encompassing totality of the whole mediation, from which emerges a clear
formal homogeneity that translates the whole database into an aesthetic canon to
be respected. According to Manovich (2019: 6), “while the gradual AI integration
into phone cameras and sharing sites may contribute to a decrease in aesthetic
diversity, the simultaneous addition of more and more controls to cameras and
photo apps may have the opposite effect”.
On the other hand, the platform that fosters users toward such experiences
persuades them with seemingly endless collections of effects. It embodies a
textual organization that has established “inter-objectual user positions” which
convince them that they can manipulate the face in any way, without limits to the
imagination. However, “disintermediation through technology inevitably leads to
standardization, which is completely at odds with the promise of autonomy that
the technological market sells” (Leone 2020: 6).
Hence, by adopting a critical media standpoint “artistic” practices of creation on
Spark, as well as the usages of digital effects by Instagrammers cannot be conceived
in the same way of free expression articulating computational matter, in the same
terms of an artistic gesture. Although some kind of constrictions can always be
traced in artistic traditions, the AR facial effects’ creation can be read as more
constrictive of the former, at least because of their ownership.
Rather, they must be conceived as user-friendly applications which transform
computationalism and creation in entertainment processes, whose applications’
user engagement is a fundamental component of the provided programs of actions.
6. The paradox of user-friendly computational creativity
Spark’s interface is a high-level one and the software itself should be conceived
as a “company product” (which owns a certain aesthetic coherence or, at least,
requires it) rather than a software development kit. Spark is not properly a Software
Development Kit (SDK) but rather a media platform. A Software Development
Kit is a collection of tools provided by a graphical interface, which can vary in its
complexity and allows its users to create homogeneous applications which interact
with the hardware of the computational instance without them having to program
the deepest interfaces. The variety of application complexities that different SDKs
allow to create is at the same time an index of the SDK’s accessibility. A SDK that in
Augmented facets: A semiotics analysis of augmented reality facial effects
521
large part provides the writing of computer code can be considered less accessible
than one like Wordpress in which you get standard results, but these are always
finished, even from a simple manipulation and assembly of “textual blocks”.
Spark similarly features a very intuitive and accessible interface (if we take into
account the basic skills that Instagrammers possess, the reason is understandable).
For example, since the launch of the program, Spark features a template window
(Fig. 3) that proposes different effects templates for a quick start of the creation,
e.g., the make-up, the face decoration, the face mask or the 3D animated poster
(but also a blank project): these are conceivable as a standardized action program.
Figure 3. The template window of Spark, retrieved from https://sparkar.facebook.com/
ar-studio/learn/articles/fundamentals/templates#opening-a-template; accessed on 22
February 2021.
In this sense, on the one hand, Spark is a media platform, a public database
that users can access in order to retrieve elements and techno-linguistic tools to
create their own original content; on the other hand, the linguistic identity of this
media platform, that is, the aesthetic and ethical codes that digital effects’ seriality
constitutes, can at the same time be traced back to a sort of “standardization” of
textual production practice. However the easiness and immediacy of the interface
does not necessarily correspond to a limitation of the expression potentiality of
522
Federico Biggio
the user. Spark has been designed mainly for users with poor programming skills
who need to create amazing content as quickly as possible. Users learn to explore
Spark interface step by step thanks to a huge quantity of online tutorials provided
by Facebook. These texts constitute a second (elementary) form of “prescription” of
the creation process, which leads unavoidably to a homologation of the aesthetics
of digital effects: the tutorials concern, for instance, the creation of stereotypical
face effects and have been configured precisely as a sort of “coaching services”,
useful to become artists and to achieve the success within the social media
community.
Moreover, it is meaningful that nowadays Spark is not just featured within
the platform Facebook Developers, but it has become an application for common
users’ entertainment. Its Facebook Community, the “Spark AR Community” group,
nowadays counts over fifty thousand members and the Facebook Page “Spark AR
Creators” has over eighty thousand likes. Here, Spark creators daily share facial
effects created and discuss these issues with one another. The definition of the
platform’s standards can thus be related to the establishment of a community, both
a social, a cultural and an economic one. This happened at the birth of the World
Wide Web, where the “http” protocol allowed different servers to communicate
with one another; this is happening today in cases like Spark in which the supply
of face assets by companies allows users to create their content with their preferred
software, but having just the provided ones as a base standard and sharable models.
Finally, the capitalization of creativity led by Facebook, capable of enforcing
adherence to certain aesthetic and moral canons, concludes with the approval
process of the effect by the owners of the platform. Here, it is not the community
but the Hub of the application (in this case Meta) where the users upload their
creation to operate as a “sanctioning instance” whose role is to approve the wellbeingness of the digital effects, according to a strict set of policies. Once the effects’
production process is concluded, the user must submit the effect to the platform,
by means of the Hub, in order to make the effect in the platform accessible and
usable. Although it could not be defined as an actual censorship process, there is
no doubt that a certain further filtering operation has occurred.
In this regard, Rettberg (2014: 20) proposed also a broader definition of the
term ‘filter’: in today’s algorithmic culture, the filter is not simply the kind that
removes skin impurities, but it is instead a pervasive metaphor for the ways in which
technology can remove certain content and it can alter or distort the users’ natural
faces.
The paradox of user-friendly computational creativity lies precisely here.
These forms of digital creativity clearly express a will of liberation from aesthetic
stereotyping, as well as a free expression of one’s own creativity, but this ideal
Augmented facets: A semiotics analysis of augmented reality facial effects
523
is pursued through the use of creative tools that are “constrictive”. Barnard
(2016: 82) points at this tension and proposes the notion of ‘(dis)empowerment
paradox’: in this context “I have identified a (dis)empowerment paradox where
such expression may feel empowering to the individual(s) controlling the camera
while concurrently conforming to and reifying oppressive, hegemonic norms”.
7. Conclusion: Aesthetic empowerment and
interactive imagination
Spark effects are not similar to aesthetic filters for digital cosmetics: whereas the
latter are useful to improve a photograph in as realistic and transparent a way
as possible, the former feature a higher artificiality. If in the 19th century the
photographic technique established the absolute transparency of the technology
(and thus the un-awareness of the mediation), as well as the suppression of the
traces of reproduction of the medium, in the case of computer graphics creativity
through augmented reality we witness a predominant return to the awareness of
the medium, of its virtual aesthetics as well as of its creative potentialities – both
in the production process and in the use in social media contexts – which in no
case can be considered to be oriented just to the improvement of an image, but
rather to an original reinterpretation of its potential uses. Already Photoshop
retouching, for instance, was not just a way of improving photographs, but a means
for individuals’ artistic expression, useful to enhance the creative gesture. The same
goes for augmented reality effects.
At this point we might recover the original idea attributed to augmented reality,
but we should admit that the “personal empowerment” of the operational/cognitive
performance of the user is not at stake here. Instead, the empowerment relies on
the platform’s ability to provide users with an attractive, engaging, pervasive and
emotional archive of virtual facial effects to create an enhancing visual experience.
Although in some make-up effects the objective of the empowerment is
truthfulness, the main feature of Spark’s face effects is evidently the ability to
create explanations of the increased reality’s creative potentialities through the
genius of the creator. For this reason, the use of emerging digital media like
augmented reality for artistic purposes is very meaningful. This trend represents
a form of “technological enchantment” which aims to foster human-made
technical creativity, a childish and irrational sensibility that leads users to recover a
primordial dimension of existence where techniques are at the service of humanity,
and serves to discover and to interpret the strangeness in the world. In an article
about endosemiotic features of augmented reality, Wamberg (2012: 471) stated
524
Federico Biggio
that “this reawakened and even empowered indexicality of augmented reality has
certain similarities with natural habitats and can indeed be understood, I would
claim, as a relinking to natural layers of experience”. For Wamberg, by reactivating
as the main sign of indexicality, augmented reality emphasizes the “proactivity”
of interaction and rehabilitates forms of performance related to the magical feats.
So, this analysis intended to affirm positively the communicative effectiveness
of facial filters in augmented reality that, unlike the (much more widespread)
beautification effects, have the possibility to be strategically employed by users to
emphasize and empower their communication practices in social media contexts.
However, this can only happen as long as the digital effect was considered as a
strategic technology, and not as a technological product capable of compromising
the users’ natural understanding of the real world. In the context of Instagram, this
poses a risk of standardization and aesthetic repetitiveness.
The justification for the hypothesis of an anthropological gain comes from the
aesthetic theory of Pietro Montani, modelled on Garroni’s and Malafouris’ thesis
on material engagement with the tools of languages. Thinking of augmented reality,
Montani pointed to the activation of an interactive imagination, by describing it as
a fundamental competence in the evolution of the human being, essentially rooted
in sensoriality. In this regard, Augmented Reality can be conceived as a creative
operation able to rethink and re-imagine the natural world in which the user is the
protagonist, to act on and through “forms of technical life capable of transforming
into a non-programmable world, albeit in accordance with rules” (Montani 2014:
77). Over thirty years of computational culture have taught users that everything
that is digital has been programmed, and, at the same time, this has been fuelling
the idea that everything, including reality itself, is a priori programmed. It is
from this consideration that computational creativity can be understood as an
individualistic, yet still positive counter-programming strategy of expression. By
putting proactively a filter on our selfies, or framing them by placing them in a
blog or an Instagram feed, users see themselves and their surroundings as if they
were outside of ourselves, by re-inventing the real world, by imagining it in an
interactive way, by re-projecting it in an un-restricted way.
I would like to conclude this reflection with an open question which concerns a
final aspect that is not addressed here for it is not directly relevant. To what extent
would the aesthetic empowerment of the user be conceivable or comparable to
the personal empowerment of augmented reality traditionally intended, if seen
as related to the gnoseological increase, a quantitative and qualitative one, of
user performance in the flesh? Answering this question would require further
investigation of the concept of empowerment, by focusing on the dis-empowering
aspects, which Wamberg (2012: 474), for example, defined as “frustration with
Augmented facets: A semiotics analysis of augmented reality facial effects
525
nature and its artificial protheses in technology”, in order to understand how the
subject of empowerment, previously anesthetized by dis-empowerment, may again
be empowered through facial augmentation performances.
References
Azuma, Ronald 1997. A survey of augmented reality. Presence: Teleoperators and Virtual
Environments 6(4): 355–385. https://doi.org/10.1162/pres.1997.6.4.355
Barnard, Stephen 2016. Spectacles of self(ie) empowerment? Networked individualism
and the logic of the (post)feminist selfie. Communication and Information Technologies
Annual 11: 63–88. https://doi.org/10.1108/S2050-206020160000011014
Brecht, Bertolt 1975. Scritti teatrali I. Torino: Einaudi.
Chae, Jiyoung 2017. Virtual makeover: Selfie-taking and social media use increase selfieediting frequency through social comparison. Computers in Human Behavior 66: 370–
376. https://doi.org/10.1016/j.chb.2016.10.007
Del Marco, Vincenza 2017. L’immagine in rete. Selfie, social network e motori di ricerca.
Roma: Edizioni Nuova Cultura.
Fuchs, Christian 2020. Communication and Capitalism: A Critical Theory. London: University of Westminster Press. https://doi.org/10.16997/book45
Garroni, Emilio 1977. Ricognizione della semiotica. Roma: Officina Edizioni.
Garroni, Emilio 2005. Immagine, linguaggio, figura. Roma: Laterza.
Jenkins, Henry; Ford, Sam; Green, Joshua 2013. Spreadable Media: Creating Value and
Meaning in a Networked Culture. New York: NYU Press.
Leone, Massimo 2020. Digital cosmetics. Chinese Semiotic Studies 16(4): 551–580. https://
doi.org/10.1515/css-2020-0030
Mann, Steve 1998. Humanistic computing: “WearComp” as a new framework and application for intelligent signal processing. Proceedings of the IEEE 86(11): 2123–2151. https://
doi.org/10.1109/5.726784
Manovich, Lev 2019. AI Aesthetics. Moscow: Strelka Press.
Montani, Pietro 2014. Tecnologie della sensibilità. Estetica e immaginazione interattiva.
Milano: Raffaello Cortina.
Montani, Pietro 2015. Prolegomeni a una “educazione tecno-estetica”. Mediascapes Journal
5: 71–82.
Montani, Pietro 2020. Emozioni dell’intelligenza: Un percorso nel sensorio digitale. Milano:
Meltemi.
Pezzini, Isabella 2002 (ed.). Trailer, spot, clip, siti, banner: Le forme brevi della comunicazione
audiovisiva. Milano: Meltemi.
Ramphul, Kamleshun; Mejias, Stephanie 2018. Is “Snapchat dysmorphia” a real issue?
Cureus, 10(3). https://doi.org/10.7759/cureus.2263
Rettberg, Jill Walker 2014. Seeing Ourselves through Technology: How We Use Selfies, Blogs
and Wearable Devices to See and Shape Ourselves. London: Palgrave MacMillan. https://
doi.org/10.1057/9781137476661
Salisbury, Meredith; Pooley, Jefferson 2017. The #nofilter self: The contest for authenticity
among social networking sites. Social Sciences 6(10): 2002–2016. https://doi.org/
10.3390/socsci6010010
526
Federico Biggio
Schipper, Meike 2018 “A Whole New Way to See Yourself(ie)”: Exploring How Face Filters
Transform the Practices of Selfie Creation. [Unpublished PhD Dissertation.] Utrecht:
Utrecht University.
Šklovskij, Victor 1968[1917]. L’arte come procedimento [Iskusstvo kak priëm]. In: Todorov,
Tzvetan (ed.), I formalisti russi. Torino: Einaudi, 73–94.
Wamberg, Jacob 2012. Toward a (re)onstructed endosemiotics? Art, magic and augmented
reality. In: Ekman, Ulrick (ed.), Throughout: Art and Culture Emerging with Ubiquitous
Computing. Cambridge: The MIT Press, 465–478.
Семиотический анализ цифровых масок дополненной реальности
Цифровые маски, использующие технологию дополненной реальности – новый
тренд общения в социальных сетях, предпочитающего «краткие формы». В статье
предлагается трехсторонний анализ: семиотический анализ цифровых масок,
используемых для усиления естественных возможностей лиц пользователей;
деконструкция Spark, одного из основных приложений для создания подобных
эффектов; наконец, критическое рассуждение о предписанных Spark практиках
и стереотипной эстетике дополненных селфи. В заключении утверждается, что
такие формы эффектов дополненной реальности должны быть задуманы не как
ориентированные на когнитивное улучшение работы пользователей, а скорее как
формы расширения возможностей и самосознания пользователей.
Augmenteeritud tahud: augmenteeritud reaalsuses ette tulevate
näoefektide semiootiline analüüs
Augmenteeritud reaalsuses esinevad näoefektid esindavad uut suunda “lühivormidel”
põhinevas ühismeediasuhtluses. Artiklis esitatakse kolmeosaline analüüs: loomulike
kasutajate nägude võimestamiseks kasutatatavate digitaalsete näoefektide semiootiline
analüüs; Facebooki pakutava Sparki, ühe olulisema sellisete efektide loomiseks kasutatava
tarkvararakenduse, dekonstruktivistlik analüüs, ning viimaks kriitiline mõtisklus Sparki
pool ette kirjutatud praktikate ja augmenteeritud selfide stereotüüpse esteetika üle. Kokkuvõttes leitakse, et seesuguseid augmenteeritud reaalsuse efektide vorme ei tule pidada
kasutajate soorituse kogntiivsele parendamisele orienteerituks, vaid pigem kasutajate
võimestamise ja eneseteadvustamise vormideks.
Face and trust: A semiotic inquiry into
inflSystems
uencers, Studies
money, 49(3/4),
and amygdala
527
Sign
2021, 527–542
Face and trust:
A semiotic inquiry into influencers, money, and
amygdala
Kristian Bankov1
Abstract. After the cultural explosion of Web 2.0, digital culture reveals an apparently semiotic paradox associated with the incredibly widespread use of images
of faces, while at the same time the reason to trust in the authenticity of these
faces is constantly declining. This is because graphic technology has made the
sophisticated manipulation of images both possible and easy. After a review of
the existing semiotic models and considerations of trust, I am proposing a new
approach which emphasizes the value-generating properties of trust by analogy
with the money sign, seen as “trust inscribed”. Research from the neurosciences
supports the hypothesis that the trustworthiness of the face is judged pre-reflexively
and primordially. This, therefore, means that a trustworthy face is a premise for
more successful communication than an untrustworthy one, notwithstanding the
object of discussion and the cultural context. An example concerning social media
influencers serves to show that in the internet-dominated globalizing culture,
trustworthy faces are a multipurpose communicative asset that makes a difference.
Keywords: face; trust; semiotics; influencers; economic value
Introduction
After the cultural explosion of Web 2.0, digital culture reveals an apparently
semiotic paradox associated with the widespread use of images of faces, while at
the same time the reason to trust in the authenticity of these faces is constantly
declining. This is because graphic technology has made extremely sophisticated
manipulation of images possible. As Massimo Leone (2019: 20) puts it, “pictures
of the face are now extremely believable and extremely untrustworthy at the same
time”. Roland Barthes saw the semiotic nature of photography in its ontological
relation with reality (Barthes 1981: 85), but today the trend is increasingly more
1
Sichuan University/New Bulgarian University, Sofia, Bulgaria; e-mail: kbankov@nbu.bg.
https://doi.org/10.12697/SSS.2021.49.3-4.14
528
Kristian Bankov
connected with the modification of the faces we can see on screens and their value
deriving from differences with their real offline referent. Here we are talking about
a whole prosperous industry, emblematic of all areas of our digital everyday life.
There was much discussion surrounding the Academy Awards 2020 in
Bulgaria. A special FX company run by two of our compatriots received an
award in the category of Special Effects. One of the most complex things this
company does is realistic animation of the faces of celebrities from the past and
introducing them into new plots in modern productions. Thus, the last episode
of Season 2 of The Mandalorian (2020) featured an appearance of the legendary
Luke Skywalker at a visible age of about 30, interpreted by the no less legendary
actor Mark Hamill. The amazement of the savvy spectator was based on the fact
that the actor was approaching 70 in the year of filming. Such manipulations of
faces have been created for at least 20 years now, but today the phenomenon is
becoming a significant part of popular culture. With the same technology, the
UK television network Channel 4 created a fictional Christmas speech by Queen
Elizabeth II to the astonishment of the television audience. The main topic of the
speech was the trust in what is seen in media, as well as new technologies allowing
reality manipulation. It has become the norm not to accept compromising videos
involving famous politicians and businessmen until experts have proven beyond
doubt that no special effects have been used. It has become easy to modify lip
movements in a completely realistic way in a video of a famous person, and to
attribute to him/her the pronouncement of words that s/he has never uttered. As
a result of such, and many other, abuses, there is already a branch in cybersecurity
known as ‘Faceforensics’.2 In 2019 the state of California ratified Bill AB-602 which
bans the use of human image synthesis technologies to make fake pornography
without the consent of the people represented, and Bill AB-730 which prohibits
distribution of manipulated videos of a political candidate within 60 days of an
election.
More and more mobile apps are creating value by manipulating selfies or
other photographs of users’ faces. These are classic beautifying techniques such
as erasing wrinkles and skin imperfections, enlarging the eyes, optimizing the
face oval and the shape of the jaw, etc. The product of these selfie manipulations
is most often used for profile pictures or simple posts on social media. The aim
is to obtain approval/recognition for the referent of the manipulation, not so
much for the complex semiotic act of communication. Strangely, manipulated
2
Rossler, Andreas; Cozzolino, Davide; Verdoliva, Luisa; Reiss, Christian; Thies, Justus; Nießner,
Matthias 2018. FaceForensics: A-large-scale video dataset for forgery detection in human faces.
Available at: https://arxiv.org/pdf/1803.09179.pdf; last accessed on 30 January 2021.
Face and trust: A semiotic inquiry into influencers, money, and amygdala
529
face and body photos are also widely used in dating applications such as Tinder,
although in theory their success is based on the trust that what we see in the photos
corresponds to what we will see at a real-life meeting. We can imagine that the
manipulated improvements actually stimulate the imagination and condition the
perception when the time arrives for a face-to-face meeting, which puts consumers
in a win-win situation.
Social media apps (such as Reface) which allow the user to put his or her face in
the place of that of a celebrity in videos with cult scenes from the cinema or videos
of a virtuoso performance are gaining huge popularity. Video manipulations in the
opposite direction are also becoming popular – celebrity faces replace the faces of
unknown participants in pornography, misleading naive viewers to think that they
are actually watching celebrity porn (Alexandrou, Maras 2018).
Even “serious” online conferencing platforms such as Zoom have recently
developed applications that manipulate the appearance of communicators in
real time. Rossler et al. summarize the problem: “The ability to effortlessly create
visually plausible editing of faces in videos has the potential to severely undermine
trust in any form of digital communication” (see fn 2).
“This face does not exist” controversy
The essence of the problem transpires as a result of the debate around the
application of AI technology for deep learning Generative Adversarial Network
(GAN). In the context of the Deepfake productions the role of the face is central
and considerable research is being conducted on the topic. Shen et al. 2019 as well
as Tinsley, Czajka and Flynn3 provide a good overview of the known facts and the
research literature on the relevance of the problem. For the sceptics:
[i]n the age of fake news and alternative facts, the risks and dangers associated
with ill-intentioned individuals or groups easily routing forged visual information
through computer and social networks to deceive, cause emotional distress, or
to purposefully influence opinions, attitudes, and actions have never been more
severe. (Shen et al. 2019: 23)
GAN technology probably would have remained popular only within the IT and
design professionals’ environment, if it were not for its application on the human
3
Tinsley, Patrick; Czajka, Adam; Flynn, Patrick 2020. This face does not exist... but it might
be yours! Identity leakage in generative models. Pre-print of a paper to be presented at WACV
2021. https://arxiv.org/pdf/2101.05084.pdf (last accessed on 20 February 2021).
530
Kristian Bankov
face. The way it is applied to the face illustrates how it can be applied to any other
visual data. GAN allows for photorealistic synthesis of digital images of human
faces that does not correspond to any existing human. They can be seen on the
website https://thispersondoesnotexist.com/ and the effect on the viewer is striking:
it breaks our ontological relation with reality as no other type of images does. As
we shall see, there is a primordial and pre-reflexive moment in the perception of
the face which overwhelms the consciously perceived statement that such a face
cannot exist in the real world. I wonder how Levinas would have commented
on such “otherwise than being” of the face. In the proposed framework this is a
semiotic paradox which would not have been possible before an advanced phase
of the digital technologies.
Trust in semiotics
“In semiotics we trust” may be the credo of many among us who base our careers
as researchers on the science of signs, but very few have taken a theoretical interest
in that basic notion. In this part of the article, I firstly list the major contributions
of semiotics of trust, and then try to position the problem of trust in the main
semiotic models, in an attempt to explain the weak interest.
The classic author who comes closest to the concept of trust is Greimas. The
Semiotics of Passions (1991), written together with Jacques Fontanille, sees trust
as an ingredient in some of the well-known passions, especially jealousy, but also
stinginess. There is a mini semiotic square (see Fig. 1), dedicated to a modal analysis
of trust (confiance in French), which explores semantic subtleties of the French
language, untranslatable to other languages (Greimas, Fontanille 1991: 218):
CONFIANCE
MÉFIANCE
?? PRÉFIANCE
DÉFIANCE
Figure 1. Semiotic square of trust/distrust.
The analysis is exciting and could be developed into a stand-alone model, but the
framework and context are entirely textual and literary. In these, trust is reduced
to a logical operator in the modal structure of passions, and although the claim of
the overall theory is for a semiotics of human action, it does not exist outside the
Face and trust: A semiotic inquiry into influencers, money, and amygdala
531
textual world (dictionaries and literary works). In the “real world”, according to my
framework, degrees of trust reflect the success and capitalization of communicative
interactions. Fontanille’s student Amir Biglari develops in depth the semiotics of
passions in relation to hope, where trust has an even more central role (Biglari
2011). However, once again in this work trust is necessarily immanent to a literary
textual structure, using the same literary references, in particular The Man Who
Laughs by Victor Hugo.
In his general theory, Greimas “clears” all the methodologically dubious implications of the instance of trust by introducing one of the most ingenious tricks of
his theory – replacing the notion of truth with that of truth-saying, or veridiction.
According to Eco (2000: 257), “that seems tailor-made to irritate any upholder of
a truth-functional semantics, not to mention every supporter of a correspondence
theory of truth”. Nevertheless, this trick ensures the exceptional internal coherence
of the theory. From a potential source of metaphysical torment, the problem of
trust is resolved as a fiduciary contract working as a logical binary code – either
it exists, or it does not exist. If there is a fiduciary contract, the semiotician can
consider the semantic effects of the communicative process according to the
canonical generative model, and if there is no such contract, there is no reason for
the method to be applied.
Such a construct of the semiotics of trust does not work for the purposes of a
study of the role of the face in digital communication, since the intrigue lies more
in the intermediate positions and degrees, rather than in the overall presence or
absence. We find similar intermediate positions and preconditions for successful
semiotics of trust in Eco’s textual pragmatics model developed in The Role of the
Reader (Eco 1994). This work presents the modelling of a much more dynamic
signification process in which, albeit without a direct semiotic modelling of trust,
there is an entire dialectic of the relationship between the author and the reader
based on, as we shall see, a simulacrum of trust in the sense that the driving force
of the interpretive process is the reader’s presuppositions, governed by the narrative
mechanisms of the plot set by the author. A good author manages to gain the trust
of his empirical reader by appropriately inscribing in the text a reader model, which
bears encyclopedic competencies similar to the reader’s genre and life competences.
The author’s ally in this endeavour is the process of suspension of disbelief, with
which the reader voluntarily abdicates – but only to a certain extent – his or her
critical distrust of the narrated facts and events (Eco 1994: 16, 208). In this way, the
reader begins to inhabit the possible world of the work imaginatively and switches
from their everyday system of expectations to a fictional one, trusting in certain
possible confluences of the plot at the expense of others. It is the reader’s trust that
is the most valuable asset for the author since it is thanks to this that the author
532
Kristian Bankov
can create the highly desired twists in the action. These twists show the reader that
his/her trust, collaboratively misled by the author, has been projected in the wrong
direction. Each twist in the course of the plot is a retelling of the cards on the table
and a new round of the game of trust and conjecture.
In Eco’s model, the concept of frame plays a central role in explaining the
process of the permanent presupposition that drives interpretation (Eco 1994:
21–27). He shapes the notion of ‘intertextual frames’. Frames are culturally codified
portions of meaningful experience that the author relies on as being available in
the readers’ encyclopedic competence and used by them to unravel the plot. At
the same time, an important direction in the otherwise boundless research of trust
also puts the concept of frames at the centre of its theoretical modelling (Lewicki,
Brinsfield 2011) and, as we will see in the next section, the analogies are impressive.
Valuable input into the semiotics of trust might have been expected from the
cognitive branch of the discipline, but as surprising as this may seem, the global
repositories of published research show a marked absence of such work. The
story is different with Peirce – as with any other subject imaginable, he has his
considerations on the notion of trust as well. Although these are quite marginal in
comparison to other notions, they carry a heuristic potential (on the faith–trust–
perception continuum) to be explored even in the context of the research direction
referred to in the following paragraphs. Should the reader want to pursue the inquiry,
great systematic review of the notion of trust in Peirce can be found in Pope 2018.
Otherwise, Peirce’s concept of abduction has been used to define a new method
of examining trust in the context of business organizations (Le Gall, Langley 2015),
although this cannot be defined as a semiotic contribution. Similarly, the phrase
‘semiotics of trust’ is sometimes used in the complex matter of trust in financial
markets, but it appears at a superficial metaphorical level, reflecting the need for
certain symptoms and signals from stock exchanges to be interpreted in a timely
manner for more adequate financial asset management (Bikoula 2011; Swedberg
2010, 2012).
A very original contribution under the label ‘Trust in Signs’ is offered by
Bacharach and Gambetta (2001) who have developed an accomplished model.
The context is that of the trust in society in general, and the emphasis falls on
the everyday practices of using credit cards, money exchange, touristic situations,
games of deception, and the like. The authors define this situation as “relentless
semiotic warfare” (Bacharach, Gambetta 2001: 167), but the only theoretical
reference they provide is that of the signal theory, while semiotics as a theory is
completely absent. The way the action of all signs and strategies of deception is
described and modelled is naïve, as there is no instance of enunciation, and the
signs/signals are conceived as bearers of universal meaning, independent of the
Face and trust: A semiotic inquiry into influencers, money, and amygdala
533
syntagmatic chain or the pragmatic situation. Yet what is interesting for our inquiry
is the part on the signs/signals of identity (Bacharach, Gambetta 2001: 167ff). The
authors introduce the term ‘Krypta’ – the unobservable properties of a person
that are his/her trust-warranting features (Bacharach, Gambetta 2001: 177). In this
case, the context-independent efficacy of trustworthiness of Krypta is close to the
way in which I define the semiotic efficacy of the face, but beyond the description
nothing else can be unpacked from this model to develop my proposal, which is
focused on digital communications in the post-truth era.
In authors such as Liu and French we find very specific applied models of the
semiotics of trust. They are used for operational diagnostics of online services
and virtual organizations (French, Liu, Springett 2006)4. The joint publication
of these authors sets out a detailed methodology for research commissioned
with companies conducting business through web platforms in mind. The end
product for the clients of such a study is a semiotic trust profile. For this purpose,
the most general cultural semiotics approach is used, which, unlike conventional
approaches, provides the necessary significance to the socio-cultural world of
consumers through cross-cultural analysis (French, Liu, Springett 2006: 177, 180).
Another important contribution offered by these authors is the creation of a
model in which the factors generating trust in virtual organizations are graded.
This model is called the Semiotic Trust Ladder and it “currently functions as a kind
of metamodel, within which VO [virtual organizations] partners can conceptualize
e-trust issues within a typical partnership, from its earliest inception to design and
implementation”5 (French, Bessis, Maple 2010: 1115).
Although these models are valuable as a practical application of a semiotics of
trust, they cannot provide the necessary theoretical basis for the purposes of this
study focused on the role of face in trust building in the era of ubiquitous digital
image manipulation.
No trust inside the text
Before proposing certain semiotic reflections on the relationship between face
and trust on the Internet, let us elaborate the analogies between the role of trust
4
French, Timothy Stewart; Liu, Kecheng; Springett, Mark 2006. Towards an E-Service
Semiotic Trust Framework. In proc. ALOIS 2006, Borås, Sweden; available at http://www.vits.
org/uploads/alois2006/french_liu_springett.pdf.
5
French, Timothy Stewart; Bessis, Nik; Maple, Carsten 2010. A high-level semiotic trust
agent scoring model for collaborative virtual organisations. 2010 IEEE 24th International
Conference on Advanced Information Networking and Applications Workshops.
534
Kristian Bankov
in the “inferential walks” in text interpretation and the “inferential walks” in real
life, when we are faced with a choice among different hypotheses of action. An
influential method for studying trust is one that sees it as the essence of heuristics
for decision-making and judgment (Lewicki, Brinsfield 2011, 2015). Here the
concept of the frame is at the heart of the theoretical model and may be seen
to introduce the theory of trust in purely semiotic terms: “Framing is about
making sense of a complex reality and defining it in terms that are meaningful
to us” (Lewicki, Brinsfield 2011: 116). Minsky is a common reference for both
approaches. Trust is a kind of shortcut in the process of giving meaning to the
reality around us and making it practically possible. “People are not accustomed
to thinking hard, and are often content to trust a plausible judgment that comes
to mind” (Kahneman 2003: 1450). The same thing happens in the mind of a text
reader that is left to the delight of the narrative, rather than reading like a detective
or an experimental scientist looking for logical inconsistencies and factual errors
that can often be found in fascinating stories. Research into trust shows that in real
“out-of-text” life, for purely practical reasons people are forced to suspend their
disbelief in some weak form just as they are when reading fiction. Concerning
textual interpretation, Eco (1994: 17) speaks of intertextual frames which culture
and textual experience create to facilitate text’s semantic functioning. In “real life”
another kind of frames that are ready-made as well, but derive from a different
kind of individual and social experience, build the meaning.
However, at this point the main difference between the two semiotic processes
appears. A closer look at the most widely accepted definitions of the concept of
trust reveals something; e.g. Lewicki and Brinsfield (2011: 125) quote Rousseau:
“Trust is a psychological state comprising the intention to accept vulnerability based
on positive expectations of the intentions or behavior of another” (my emphasis,
K. B.). Vulnerability also appears in many other definitions of trust (Lewicki,
Brinsfield 2011: 125). However, it is something that is completely missing both
in the interpretation of the text from the point of view of textual pragmatics and
in the immanentist approach of the generative model. The text is by definition a
fixed discourse, given once and for all, and no matter how unusual the inferential
walks made by the reader are, he/she risks absolutely nothing given that he/she
is ultimately served an identical textual manifestation with an identical structure.
There may be a description of trust in the text as a narrative structure or modality,
but there is no such psychological state. Vulnerability can only happen in a situation
of interactive communication or action. It is one thing to watch with bated breath
how James Bond plays with the fate of humanity in a poker game, but another to
play poker ourselves and to win or lose a certain amount of money on our choice
of hand. There is no trust in the text, because in the semiotic mechanism of its
Face and trust: A semiotic inquiry into influencers, money, and amygdala
535
semantic actualization there is no instance of decision-making. What we see in the
text from a perspective of textual pragmatics is trust by analogy with the extratextual condition of trust. In other words, it is a simulacrum of trust. Accordingly,
the theoretical modelling of the existential state of trust in textualism is doomed
(see Bankov 2018, 2022).
Face, primordial trustworthiness and self-fulfilling prophesy
Let us now turn to the field of neuroscience in order to see what ideas from there
can enrich our semiotic view of the face–trust relationship. The findings are very
encouraging. A fundamental discovery was made in the early 1990s: the AmericanPortuguese neuroscientist Antonio Damasio, when researching patients with brain
injuries, located quite clearly an area in the brain where instant and primordial
judgements of trust/distrust after seeing a face take place (see Colins 1998). This
is the part called the amygdala in the centre of the brain. The conclusion is that
these judgements on a level of instantaneity and primordiality are identical with
emotional reactions in every perception. At the same time, these reactions have
more important consequences than others, since they are more ethically reflexive
than these. At the heart of the so-called first impression, there is a kind of capital
of trust with longer-lasting consequences. The team of the Bulgarian-American
neuroscientist Alexander Todorov conducted experimental research into Damasio’s
perspective on perceived trust, comparing instantaneous judgment with longerlasting impressions, based on the subjects’ deeper familiarity with each other.
The results show a strong correlation between the two phases of judgements on
trustworthiness, i.e. the validity of the primordial judgment is proved (Todorov,
Oosterhof, Baron 2008).
Another team of authors changed the research perspective and experimented on
the behavioural effects on the carriers of more or less trustworthy faces themselves
(Slepian, Ames 2016). The social adaptation of individuals with varying degrees
of trustworthiness in their faces creates behavioural strategies which correspond
to and internalize such a degree. In other words, if someone is born with a trustworthy face, for example, babyface (Slepian, Ames 2016: 286), then his/her social
behaviour, driven by the need for adequacy to the expectations of others, will
be such as to justify the credited trust, while untrustworthy faces self-generate
social distrust. To explain the phenomenon, the authors use the term “self-fulfilling
prophecies” (Slepian, Ames 2016: 283, 287), as it is used in the psychosocial
sciences. For us, this means that the trust capital of the trustworthy face as a social
construction has a tendency to be polarized in an objective way.
536
Kristian Bankov
Since the dawn of time, petty fraud with abuse at the level of primordial prereflexive trust created by “good” faces has taken place, e.g. many tourist have
become the subject of petty abuse by local individuals specializing in this business.
The interesting thing here is that the strength of the effect of trust created by
the face is such that a person repeatedly falls into the same trap time after time.
There are many analogies with theatrical performances. Theatre anthropology
defines a pre-expressive level of stage presence that is transversal to all traditions
and techniques. It is a prerequisite for a good actor to be convincing on stage
for the spectator to believe in his/her role. In everyday life, the face and its level
of trustworthiness has the same pre-expressive function (Bankov 2003), which
makes the same communicative acts achieve their goal if fulfilled by “good” faces,
and fail if fulfilled by “bad” faces. In the normal dynamic of social life, a face’s
trustworthiness can have a significant impact on life experience, including lifechanging situations (Slepian, Ames 2016: 283). In the case of tourist fraud, the
situation is not ordinary, since the contact has no temporal perspective – all we
have is a pure form of monetary capitalization of the pre-expressive trustworthiness
of the face of the fraudsters. The topic of the relationship between trust and its
transformation into monetary value is the subject of the final chapter of the study.
Trust, faces, influencers, and economic value:
A research direction
The approach I propose for the semiotic conceptualization of the relationship
between the face and trust in the digital age is just a research direction for a
potentially large and challenging research programme. As we shall see shortly,
the issue hits the core of a megatrend in global socio-economy, which is why my
approach begins from the extremely insightful definition of money which opens
a completely new perspective in the modelling of digital culture. This definition
states that money is “trust inscribed” (Ferguson 2008: 30).
Over the last four to five years, I have devoted a great deal of research to this
type of sign to develop a holistic semiotic perspective in which many of the age-old
boundaries between money and non-money have become blurred. In this logic,
for example, the explanation for the incredible success of bitcoins as a currency
boils down to two fundamental things: trust and adoption (Bankov 2022: 82). The
Internet has made possible the emergence of a wide variety of monetary forms,
most of which aim at autonomy from the central banking system. This has given
rise to time banking, video games currencies, and many other creative forms of
measuring and exchanging value through various online platforms. I believe that
Face and trust: A semiotic inquiry into influencers, money, and amygdala
537
the face with its capacity to be a bearer of primordial and pre-reflexive trust can
take a worthy place within this sign logic.
It is no coincidence that this article illustrates my thoughts on the semiotic
relation of the face and trust with the figure of the influencer. Certainly, this is
not the only manifestation of the relationship studied, but the influencer is an
emanation of the cultural changes in the digital age, especially from the phase of
social media (Web 2.0). This is a megatrend: the agency Upfluence lists four million
professional influencers, and this is the result of only a few years of growth in this
field.6 We must also take into account all the millions of users who aspire to this
status, but have not yet found a way to achieve it, as well as the millions who are
de facto influencers, but have not turned it into a profession.
It would be no exaggeration to say that the face is the semiotic device that creates
the preconditions for the influence that each influencer has on his/her followers,
regardless of the field of interest. When a team from the online magazine Papermag.
com dedicated research to unravelling the secret of the success of influencers,
they focused their efforts on the influencers’ faces. The result of this study is a
synthesized image of the faces of the 100 most influential Instagram influencers.
Undoubtedly, this face (Fig. 2) well illustrates the hypotheses of neuroscientists
about face trustworthiness, although it was achieved by a different methodology.
Another scientific study confirms that the key to success is not the attractiveness of
the face, but its ability to inspire trust and intimacy (Marôpo, Jorge, Tomaz 2019).
Figure 2. The Ideal Influencer – synthetic face
of the 100 top Instagram influencers. (Credits
to PAPER Magazine/ PAPERMAG.com.)
6
See https://www.upfluence.com/influencer-profiles (last accessed on 31 January 2021).
538
Kristian Bankov
There are influencers who never show their faces. Some techno geeks never appear
on the videos, others generate their influence by collecting images or videos of
other people and subjects, etc. I could not find reliable statistics in regard to this,
but my assumption is that these are minor cases, and the common mainstream
influencer is visualized well in front of the camera and speaks to her/his followers.
Other important point is that having a trustworthy face is not enough to
become an influencer automatically. My firm belief is that a trustworthy face is a
tangible and objectively measurable asset; one that gives indisputable advantages
to the owner and creates difficulties for those who lack it. The conditions for the
increased value of trustworthy faces are rooted in one of the main consequences of
the culture of social media – the total crisis of trust. Some summarize this situation
as post-truth, others as post-factual democracy. “Post-truth” even became Word of
the Year 2016 of the Oxford Dictionary.7 In this new social fabric, dominated by
emotional factors, the face has a privileged status. I began this study with a long
list of technological techniques for counterfeiting the face, but this is part of the
overall ontological status of an online culture in which anything can potentially
be counterfeited. “Trust is the new scarcity” claims one of the authors, trying to
grasp today’s essentials.8 We cannot but think of the times of the gold standard
when the scarcity of gold was the main source of trust in the precious metal,
necessary for the global financial system to stay stable. With its primordial and prereflexive potential to establish a relation of trust, in today’s context the trustworthy
face is the new gold. The success of the influencers’ business is evidence that,
notwithstanding the well-known practices of falsifying its digital appearance, the
face’s ontological statute is still based on the relation with an agency of trust.
Trustworthy face as gold
The face is the semiotic universe par excellence, where the equilibrium between
nature and culture feeds the social fabric of every civilization in the most sublime
way. Anatomically the same, the face has inspired a huge variety of myths, rituals,
representations, businesses, etc. in different cultures (Leone 2019: 19–20). The
digital globalization of the world is attributing a new, more unified, but still
fundamental role to the face, where trust is the scarcest resource of the social
fabric. We see this clearly with the crucial role of the face in the megatrend of the
influencers’ profession.
7
See https://languages.oup.com/word-of-the-year/2016/; last accessed on 31 January 2021.
McCullen, Aidan 2019. Why we need public broadcasters – Trust is the new scarcity. Medium,
7 March 2019, is available at https://medium.com/thethursdaythought/why-we-need-publicbroadcasters-trust-is-the-new-scarcity-65c32577e1cd; last accessed on 31 January 2021.
8
Face and trust: A semiotic inquiry into influencers, money, and amygdala
539
The face of the professional influencer is monetized trust. In the same way,
the money sign is trust inscribed. Faces are the new species of coins of the
internet where their value is measured and expressed by the trustworthiness,
inscribed on their “heads and tails”. The extended neuroscientific research on the
trustworthiness of the face suggests that there are objective methods of evaluating
it. With the help of semiotics we may start work on a tool designed to elaborate
a semiotic trust profile of the face, similar to the evaluation of the quality of gold
before the coinage, by analogy with the existing tools for business organizations.
Conclusion
This is the first research paper which brings together face, trust and semiotics. As
pairs, all three combinations – semiotics and face, semiotics and trust and face and
trust – are well explored. My main contribution would be the opening of a new
research direction, rich in theoretical potential and in operative application on a
socio-economic megatrend in digital culture: namely studying the sphere of social
media influencers. This is not the only cultural reality, envisaged by the research
direction, but I think it is the most relevant one to start with. It is one which, after
a certain number of contributions and after establishing a basic vocabulary, could
be extended to other occurrences of the face in e-technological societies.
One of the major claims of the paper is that there can be no other approach
to such complex object as the semiotics–face–trust interplay than an interdisciplinary one. Interdisciplinarity would include the economic logic of online
communications, especially working on the differences between linear textual
actualization and interactivity. The methodological perfection of the generative
approach of the Paris School of semiotics is one of the major obstacles to opening
the semiotic theory to interactive communication. The first step is the elaboration
of a new model of the money sign, seen as trust inscribed, which would allow the
semiotic modelling of digital interactivity.
At the same time, the advent of social media (Web 2.0) has brought about the
full extent of the post-truth era. Trust has become one of the scarcest resources
for our social fabric. In such a context, it has become obvious that appearance
dominates over substance and face trustworthiness is increasingly playing a
decisive role in the success of the new-economy ventures, consisting mainly of
services and access to experiences. Trustworthy faces work like valuable currency
in the trust-scarce economic context, comparable to the role of gold as a precious
and scarce material for the finances during the gold standard. In support of such
a parallel comes the evidence from neuro-sciences. It has been found that the
540
Kristian Bankov
trustworthiness of the face works on a primordial and pre-reflexive level, ascribing
to it an absolute value in the establishment of the phatic function in online interaction.
References
Alexandrou, Alex; Maras, Marie-Helen 2018. Determining authenticity of video evidence
in the age of artificial intelligence and in the wake of Deepfake videos. The International
Journal of Evidence & Proof 23(3): 255–262. https://doi.org/10.1177/1365712718807226
Bacharach, Michael O. L.; Gambetta, Diego 2001. Trust in signs. In: Cook, Karen S. (ed.),
Trust in Society. New York: Russell Sage Foundation, 148–184.
Bankov, Kristian 2003. Face and understanding. In: Tarasti, Eero (ed.), Understanding/
Misunderstanding: Contributions to the Study on the Hermeneutics of Signs. (Acta
Semiotica Fennica XVI.) Helsinki: International Semiotics Institute, 192–200.
Bankov, Kristian 2018. From text to interaction. Digital Age in Semiotics and Communication 1: 9–17. https://doi.org/10.33919/dasc.18.1.1
Bankov, Kristian 2022. The Digital Mind: Semiotic Explorations in Digital Culture. London,
Cham: Springer Nature.
Barthes, Roland 1981. Camera Lucida: Reflections on Photography. New York: Hill and
Wang.
Biglari, Amir 2011. Sémiotique et Pragmatique des Passions dans Les Contemplations
de Victor Hugo. Lille thèses. Université de Limoges, Faculté des lettres et sciences
humaines.
Bikoula, Ignace Gustave 2011. La valutazione e la prevenzione del rischio sistemico con gli
strumenti della network alaysis. PhD dissertation. La Sapienza University of Rome,
Faculty of Economics.
Colins, Peter 1998. Faces, trust and the amygdala. Trends in Cognitive Sciences 2(7): 238.
https://doi.org/10.1016/S1364-6613(98)80016-8
Eco, Umberto 1994. The Role of the Reader: Explorations in the Semiotics of Texts. Bloomington: Indiana University Press.
Eco, Umberto 2000. Kant and the Platypus. London: Vintage.
Ferguson, Niall 2008. The Ascent of Money: A Financial History of the World. New York:
Penguin.
Greimas, Algirdas Julien; Fontanille, Jacques 1991. Sémiotique des Passions: Des États de
Choses aux États d’Âmes. Paris: Seuil.
Kahneman, Daniel 2003. Maps of bounded rationality: Psychology for behavioral
economics. American Economic Review 93(5): 1449–1475. https://doi.org/10.1257/
000282803322655392
Le Gall, Véronique; Langley, Ann 2015. An abductive approach to investigating trust
development in strategic alliances. In: Fergus, Lyon; Möllering, Guido; Saunders, Mark
N. K. (eds.), Handbook of Research Methods on Trust. Cheltenham, Northampton:
Edward Elgar Publishing, 36–45. https://doi.org/10.4337/9781782547419.00012
Leone, Massimo 2019. The semiotics of the face in digital dating: A research direction.
Digital Age in Semiotics and Communication 2: 18–40. https://doi.org/10.33919/
dasc.19.2.2
Face and trust: A semiotic inquiry into influencers, money, and amygdala
541
Lewicki, Roy J.; Brinsfield, Chad 2011. Framing trust: Trust as a heuristic. In: Donohue,
William Anthony; Rogan, Randall G.; Kaufman, Sanda (eds.), Framing Matters: Perspectives on Negotiation Research and Practice in Communication. New York: Peter Lang
Publishing, 110–135.
Lewicki, Roy J.; Brinsfield, Chad 2015. Trust research: Measuring trust beliefs and
behaviours. In: Fergus, Lyon; Möllering, Guido; Saunders, Mark N. K. (eds.), Handbook
of Research Methods on Trust. Cheltenham, Northampton: Edward Elgar Publishing,
46–64. https://doi.org/10.4337/9781782547419.00013
Marôpo, Lidia; Jorge, Ana; Tomaz, Renata 2019. “I felt like I was really talking to you!”:
Intimacy and trust among teen vloggers and followers in Portugal and Brazil. Journal
of Children and Media 14: 22–37. https://doi.org/10.1080/17482798.2019.1699589
Meyerson, Debra; Weick, Karl E.; Kramer, Roderick M. 1996. Swift trust and temporary
groups. In: Kramer, Roderick M.; Tyler, Tom R. (eds.), Trust in Organizations. Thousand
Oaks: Sage Publications, 166–195. https://doi.org/10.4135/9781452243610.n9
Pope, Michael 2018. Peircean faith: Perception, trust, and religious belief in the conduct of
life. Transactions of the Charles S. Peirce Society 54(4): 457–482. https://doi.org/10.2979/
trancharpeirsoc.54.4.02
Shen, Cuihua; Kasra, Mona; Pan, Wenjing; Bassett, Grace A.; Malloch, Yining; O’Brian,
James F. 2019. Fake images: The effects of source, intermediary, and digital media
literacy on contextual assessment of image credibility online. New Media & Society
21(2): 438–463. https://doi.org/10.1177/1461444818799526
Slepian, Michael L.; Ames, Daniel R. 2016. Internalized impressions: The link between
apparent facial trustworthiness and deceptive behavior is mediated by targets’ expectations of how they will be judged. Psychological Science 27: 282–288. https://doi.org/
10.1177/0956797615594897
Swedberg, Richard 2010. The structure of confidence and the collapse of Lehman Brothers.
Research in the Sociology of Organizations 30: 71–114. https://doi.org/10.1108/S0733558X(2010)000030A007
Swedberg, Richard 2012. The role of confidence in finance. In: Cetina, Karin Knorr; Preda,
Ales (eds.), Handbook of the Sociology of Finance. New York: Oxford University Press.
https://doi.org/10.1093/oxfordhb/9780199590162.013.0028
Todorov, Alexander; Oosterhof, Nikolaas N.; Baron, Sean G. 2008. Evaluating face
trustworthiness: A model-based approach. Social Cognitive and Affective Neuroscience
3: 119–127. https://doi.org/10.1093/scan/nsn009
Лицо и доверие: семиотическое исследование инфлюенсеров,
денег и миндалевидного тела
В результате культурного взрыва Web 2.0 цифровая культура породила семиотический парадокс: с одной стороны, изображения лиц используются невероятно
широко, с другой стороны, поводов усомниться в их подлинности всё больше. Это
связано с тем, что графические технологии сделали сложные манипуляции с изображениями возможными, а также значительно их упростили. После обзора существующих семиотических моделей доверия я предлагаю новый подход, подчеркивающий
ценностные свойства доверия по аналогии с денежным знаком, который по сути
542
Kristian Bankov
является «написанным доверием». Когнитивные исследования подтверждают гипотезу о том, что достоверность лиц оценивается нами неосознанно. Это означает, что
заслуживающее доверия лицо является предпосылкой для более успешного общения, чем не заслуживающее доверия, независимо от темы разговора или культурного
контекста. Пример влияния инфлюенсеров показывает, что в глобализирующейся
культуре, в которой доминирует интернет, заслуживающие доверия лица обладают
многосторонним коммуникативным преимуществом.
Nägu ja usaldus: semiootiline sissevaade suunamudijatesse,
rahasse ja mandeltuuma
Pärast Web 2.0 kultuuriplahvatust on digikultuuris avaldunud ilmselt semiootline paradoks,
mis on seotud nägude uskumatult laialt levinud kasutamisega, samas kui põhjus nende
nägude autentsuse usaldamiseks on pidevalt kahanemas. Seda seetõttu, et graafiline tehnoloogia on muutnud keerukad pildimanipulatsioonid niihästi võimalikeks kui ka kergesti
teostatavateks. Alustan ülevaatega usalduse olemasolevatest semiootilistest mudelitest
ja sellega seotud kaalutlustest ning pakun seejärel välja uue lähenemise, mis rõhutab
usaluse väärtustgenereerivaid omadusi analoogia põhjal raha märgiga, mida peetakse
“kirja pandud usalduseks”. Uurimistöö neuroteaduste valdkonnas toetab hüpoteesi,
et näo usaldusväärsuse üle otsustatakse mõtlemiseelselt ja ürgselt. Nii tähendab see, et
usaldusväärne nägu on eeldus edukamaks suhtlemiseks kui ebausaldusväärne, sõltumata
aruteluobjektist ja kultuurikontekstist. Ühismeedia suunamudijaid puudutav näide
demonstreerib, et globaliseeruvas kultuuris, kus domineerib internet, on usaldusväärne
nägu mitmekülgseks eeliseks, millel on tuntav mõju.
Gotta
face ‘em
all543–565
543
Sign Systems Studies
49(3/4),
2021,
Gotta face ‘em all: Pokémon,
Japanese animated characters, and
the emergence of playful visual animism
Vincenzo Idone Cassone1
Abstract. As a result of technological innovations and new cultural practices, the
contemporary mediasphere is increasingly populated by digital(ized) faces. The
phenomenon is not limited to human faces, but includes a vast universe of fictional
animated faces, variously called ‘characters’, ‘mascots’ or ‘kyara’. In Japan, while
certainly not new, kyara have been spreading thanks to globalization, digitalization
and media-mix strategies. Through the connection between visual design, fictional
narratives and socio-cultural consumption, kyara can be considered semiotic
figures of in-betweenness, key symbolic mediators in the Japanese mediascape.
Their anthropomorphic face design mediates the cultural boundaries between the
human and the non-human, the animate and the inanimate, nature and culture.
Furthermore, their post-modern narratives mix inspiration from the past and
the present, from myths to science fiction. Lastly, they involve an encyclopedic
reworking between fiction and reality, mythical references and secularization,
between the domains of seriousness and playful make-believe. The article aims to
explore the semiotic dimensions of kyara in contemporary Japan, with emphasis
on their logic of representation and cultural outcomes. These will be investigated
through the analysis of the Pokémon franchise, which will make it possible to
describe the emergence of new semiotic patterns of ‘playful visual animism’ in the
process of media facialization of everyday life.
Keywords: Pokémon; playfulness; animism; heterotopy; faces; anthropomorphism;
fan theory; transmedia
1. Introduction
The contemporary mediascape (Appadurai 1990) is increasingly populated by
digital(ized) faces, combining an unprecedented quantity and quality of images,
new forms of interaction and the emergence of face-based cultural practices and
beliefs. This can be attributed to a mix of recent technological innovations (digital
photography, visual filters, software for automatic face recognition), the diffusion of
novel genres of face representation (the selfie, animojis, memes) and new behaviours/
1
Department of Humanities, University of Turin, Italy; e-mail: idonecassone@gmail.com.
https://doi.org/10.12697/SSS.2021.49.3-4.15
544
Vincenzo Idone Cassone
practices (swapping faces on apps for dating, using masks or occultation for political
activism, covering faces for healthcare during pandemics) (Leone 2018).
While the vast majority of research studies have been focusing on human faces,
the “facialization” of media cultures also includes the parallel pervasiveness of
animated anthropomorphic faces. Animated characters and mascots are certainly
not new, but have been proliferating in the last thirty years due to the combination
of media globalization, digitalization and market strategies (Bendazzi 2016). As
a result, the current global mediascape is filled with anthropomorphic characters
such as Mickey Mouse, Hello Kitty or Pokémon (from transmedia franchises), Tony
the Tiger, Michelin Man or the Linux penguin (representing corporate or brand
identity), not to mention historic sport mascots or recent emoji-based characters
(Tampa Bay’s Raymond, Pusheen the cat, etc).
Their presence is even more significant in Japan, where they have been at
the heart of media-mix strategies (Steinberg 2012) and media ecology dynamics
(Ōtsuka 1989) since after WWII. The Japanese mediascape is filled with many
of the above-mentioned anthropomorphic characters, often labelled ‘kyara’2,
not limited to child-targeted TV shows or toys. Kyara are part of daily brand
merchandizing (Han 2017), and are used extensively for public and social
advertising (Alt, Yoda 2007) as well as to embody local places and tourist
attractions (Barrows 2014), and feature in parades and social events (Occhi 2012).
The importance of such characters in Japanese culture, however, does not
simply lie in their pervasiveness. Their presence has been linked to a process of
characterization in Japanese everyday life, in which kyara become “creatures of inbetweenness” (Nozawa 2013)3: semiotic personae whose existence is not limited
to specific media formats, visual styles or fictive semiospheres, but appears as
fragmentary, pervasive and heterogeneous daily experiences of “characterencounters”. Thus, they can be assimilated as key symbolic mediators (Lotman
1990: 136–137) in the Japanese semiosphere: figures which enable communication
between the inner and outer layers of the cultural self-description systems
(identity–alterity). In a similar way to their human correlates, the presence of
and interaction with animated characters could be linked to the inception of new
semiotic codes/models of behaviour (Lotman 1990: 233–234). More specifically, as
2
The Japanese term ‘kyara’ was born as an abbreviation of the loan word ‘kyarakyutā’,
but now stands mostly for “highly-stylised or simplified visual figuration that can be easily
reproduced and consumed outside of its original context” – see Galbraith 2009: 125; Galbraith,
Patrick W. 2009. Moe: Exploring virtual potential in post-millennial Japan. Electronic Journal
of Contemporary Japanese Studies 9(3) is available at https://www.japanesestudies.org.uk/
articles/2009/Galbraith.html.
3
Nozawa, Shunsuke 2013. Characterization. Semiotic Review 3 is available at: https://www.
semioticreview.com/ojs/index.php/sr/article/view/16/67.
Gotta face ‘em all
545
I suggest in this paper, contemporary kyara could be related to semiotic patterns
of playful animism in the Japanese semiosphere, which result from the interaction
between the visual features of these characters, their roles and function in the
mediascape, and the encyclopaedic interpretive behaviours of users.
The aim of this contribution is to describe the patterns of playful animism
in the Japanese semiosphere, discussing them in relation to: the specific codes
of visual representation of animated characters, the encyclopaedic logics of
anthropomorphism, the interpretive process involved in interacting with fictional
narratives, and the contemporary dynamics of media consumption. This objective
will be pursued by using the global franchise of Pokémon as a case study.
Pokémon (haplology of ‘pocket monster’, ‘poketto monsutā’) is a well-known
transmedia franchise, created in Japan by Satoshi Tajiri, Ken Sugimori and Junichi
Masuda in the mid-1990s (1996) as a videogame for the Game Boy handheld console.
The sudden success of the game in Japan led to the development of a media mix
strategy,4 with the realization of an anime series (1997) which was quickly translated
and exported into the USA (1998) and Europe (1999), fuelling the production of
merchandise (collectibles, plushies, clothing, etc.) and subsequent development of
movies, comics, other games, as well as theme parks, official parades, cross-branding
and so on. Although many have predicted its decline over the years (Tobin et al.
2004), the franchise is still among the most widespread and highest-grossing in the
world, with a further boost provided by the smartphone ARG Pokémon GO (2016)
and the latest videogame entries Pokémon Sword/Shield (2019).
The Pokémon franchise proves to serve as a fitting case study for the understanding of the peculiar semiotic features of animated anthropomorphic characters
for several reasons. First, it is a perfect embodiment of the contemporary changes
in Japanese transmedia products and codes of narrative consumption, being an
example of Steinberg’s (2015) gameic media mix,5 as well as a narrative transmedia
interactive ecosystem (Ōtsuka 1989). Second, it can be considered a paradigmatic
example of Japanese culture while simultaneously being a globalized phenomenon,
a syncretic mix of contemporary post-modern narratives and folkloric heritage
(Iwabuchi 2002; Hutchinson 2019; Allison 2006). Lastly, as I suggest in this paper,
the franchise could be read in light of Azuma’s (2009) reflections on the otaku
subculture as “the site that most sensitively registers social transformations”
(Lamarre et al 2007: 158) in contemporary Japan.
4
‘Media mix’ is a term initially used in Japan to indicate the commercial strategy of
diversification of a brand through different products and media; the term has been diffused in
the West by Steinberg (2012), who specifically discusses the anime-based media mix created
around Astro Boy (Tetsuwan Atomu) in the 1960s.
5
A transmedia strategy centred on games and not anime, as for the anime media mix.
546
Vincenzo Idone Cassone
2. Visually iconic, taxonomically hybrid:
The design of Pokémon
Pokémon are among the most iconic contemporary animated characters in the
world. In Japan, due to the success of the franchise and the extensive crossbranding and merchandise promoted by its media-mix strategy, Pokémon faces
can constantly be seen on TV, in magazines, and in street advertisements; they are
incorporated into clothes, food, vending machines, and even periodically feature
in costume parades and events on the streets. The franchise mascot, Pikachu, as
well as the most famous Pokémon of each generation, could be defined as proper
contemporary cultural icons.
Despite the many differences in media formats (manga, anime, games), styles,
illustrators, and the technological evolution (from 2D to 3D) of the games, the
characters of the franchise are widely identifiable and visually distinct6. However,
while it can be said that the visual features of Pokémon are generally recognizable
and established in the Japanese cultural encyclopaedia, a different question arises
in relation to their “taxonomy”, or encyclopaedic categorization. More specifically,
I suggest that while the semiotic competence linked to Pokémon recognition is
relatively diffuse, it is somewhat difficult for model readers/users to explain and
define what Pokémon actually are, according to typical cultural taxonomies.
In order to describe the zoomorphic features of the Pokémon themselves,
Surace (2017) fittingly refers to them as ‘animaloids’. While Pokémon are
depicted as living and sentient creatures, their morphology largely oversteps the
boundaries (both inner and outer) of the animal world. The design of the 898
existing Pokémon includes almost every known form from the animal kingdom
(mammals, fishes, insects, gastropods, sponges, etc.) and inspiration from other
kingdoms of life (plants, fungi, even protists). Pokémon design is also derived
from non-living natural objects (minerals and rocks, metals, gases) and often goes
beyond the boundaries of nature to include the artificial (mechanical Pokémon),
as well as paranormal entities (ghosts, spirits, urban legends), extra-terrestrial
forms (aliens, beings from parallel dimensions, digital Pokémon, robots) and even
legendary or mythical beings (embodiments of nature, supernatural tricksters,
fairies, transcendent beings) up to the Pokémon god itself. Furthermore, while
6
Part of the reason is connected to the media-mix nature of the franchise, which has
circulated in many formats and media from the start. The appearance of many Pokémon, for
instance, has become well known thanks to the widely used illustrations by Ken Sugimori
(art director and character designer of Game Freak). For coherency purposes, the pictures on
the following pages have been selected from Sugimori’s illustrations, while the analysis has
included the sprites, 3D models of the games etc.
Gotta face ‘em all
547
many Pokémon display anthropomorphic shapes, many others are clearly based
on human artifacts (keys, garbage, ice creams) or figures/representatives of human
activities (mimes, martial artists, gurus).
In addition to the variety in origin/inspiration, it should be pointed out that the
vast majority of Pokémon display a heterogeneous mix of features/properties from
different natural kingdoms: the animal, mineral, and vegetal worlds, as well as the
overlapping of human, supernatural or daily elements, thus appearing as original
chimeric/hybrid creatures (see Fig.1).
Figure 1. Left to right, up: Gigalith, Victribell, Magnemite, Duskull, Klefi, Mr. Mime;
below: Slowbro, Dhelmise, Propobass, Exegutorr, Dracovish. (Copyright of Nintendo/The
Pokémon Company/GameFreak, under Fair Use.)
It is indeed difficult to find a universally shared physical or biological feature
common to all Pokémon, which would make them fit perfectly into a proper
encyclopaedic categorization: from this perspective, Pokémon remind of Umberto
Eco’s reflections on the platypus as an object of experience which subverts the
existing cultural segmentations and taxonomic codes of its viewers (Eco 2000)
in a similar way to the cultural taxonomization of animals themselves (Maran,
Martinelli, Turovski 2011), by involving semiotic issues which are linked to the
coherency and structure of cultural self-descriptions (Madisson 2016), both real
and fantastic, as showed by Borges’ examples of the Chinese encyclopedia, and
Marco Polo’s description of unicorns (Eco 2000: 57–58).
What stands out in the case of Pokémon is, once again, their semiotic nature
as animated beings, in a double meaning: as visual products of animation, and as
548
Vincenzo Idone Cassone
characters which manifest animated traits. Etymologically, the deep connection
between the appearance/effect of dynamism and the inherent assumption of agency
and life which many cultures ascribe to the former can be expressed by the term
‘animaloid’.
The hybrid and heterogeneous design of Pokémon also extends to the
dynamics of anthropomorphism7 which is common, yet rarely investigated, in
animation (Wells 2009: 2): on the one hand, the franchise establishes a clear and
immediate visual and diegetic difference between humans and Pokémon (even
if categorization provided in the diegetic universe turns out to be contradictory
and complex); on the other hand, typical visual techniques of animation,
employed in forms of character anthropomorphization, are applied to Pokémon,
even if in heterogeneous ways. An example (albeit partial) can be provided by
looking at the visual representation of faces and gazes in the franchise: their
anthropomorphization is a common trope of animation, associated with the
possibility of providing animals with human-like communicative and emotional
expression abilities (Parkinson 2020: 43–50; Wells 2009: 29–-32).
First of all, the vast majority of Pokémon (approximately 888 out of 898) can
easily be interpreted as having a face, in terms of the presence of recognizable
eyes/nose structure, associated with the communicative and emotional expression
ability (Bruce, Young 2012: 253–260). Those lacking a visage are often diegetically
regarded as exceptions (see Fig.2):
Figure 2. Pokémon with no face. Left to right: Starmie, Roggenrola, Registeel, Lileep,
Stakataka.
7
In a semiotic perspective, anthropomorphism can be considered the adoption or interpretation of certain elements, features and affordances of characters as typically human traits
(e.g. wearing clothes), or as human variants of more general traits of living beings (e.g. group
behaviour, eating). In this perspective, anthropomorphism is always a perspectival dynamic, a
result of broader cultural self-descriptions and encyclopaedic structures, which pre-establish the
human-animal dichotomy in certain semiospheres (see Bertrand, Marrone 2019).
Gotta face ‘em all
549
In relation to the eyes and gazes of Pokémon, we can see a heterogeneous set of
visual forms, ranging from a relatively anthropomorphic gaze (with visible white
sclerae8), to cartoon expressive gazes (conveying the attitudes or emotions of the
characters), eyes inspired by the biological structures of mammals, insects, etc., or
even certain forms of supernatural eyes (flames, gems, etc).
Despite this, the franchise tends to follow the traditional norms of cartoon
animation (especially Japanese animation), by making eyes and gazes, as well as
faces, fundamental communicative interfaces for meaning-making among living
beings, including animated ones (Leone 2021); furthermore, as a typical trope of
animation, both the anthropomorphic and the expressive gaze are often used for
humans and Pokémon alike. This is evident in all the manga, anime and games
since the second generation: the same visual expressive traits are used as the basis
for certain Pokémon and human characters, or to express the same temporary
attitude (see Fig. 3).
Figure 3. Up: different types of eyes in Pokémon; down: humans and Pokémon with similar
gazes.
This example may support the initial assessment that the Pokémon universe
seems to be characterized by the interaction of two simultaneous dynamics: on
the one hand, the dynamic of visual animated-ness; on the other hand, a perceived
8
The presence of visible white sclerae in humans is considered an evolutionary trait (Kobayashi, Konshima 2001) typical of humans and certain apes, often interpreted and ascribed by
cultures as a sign of humanity.
550
Vincenzo Idone Cassone
taxonomic heterotopy. The former uses typical styles and techniques of animation
(even if in heterogeneous ways) to ensure the recognition of Pokémon and humans
alike as living, sentient creatures, sharing the communicative and expressive
function of the face (Leone 2009, 2021); in case of the latter, the hybrid design of
Pokémon does not simply deviate from, but actually challenges the interpreters’
cultural codes of taxonomy, undermining their semiotic codes of segmentation
between living and non-living, different kingdoms of nature, and the very perceived
difference between the human and the animal, the cultural and the natural.
3. Pokémon heterotopic regime of animation
A likely critique of the idea introduced above is that the ways in which living
beings are represented in Pokémon do not deviate from the standards of animation,
more specifically those with fantasy settings and targeted at children/young adults.
Animation typically involves anthropomorphizing of living and non-living beings,
with magic- and fantasy-based narratives whose potential incoherencies are likely
to trigger typical suspension of disbelief dynamics in spectators. According to
Lamarre (2018), since the 1930s, the reflection on animation has often been linked
to an implicit animistic attitude, it being the explicit representation of many natural
phenomena as animated and provided with agency, as part of a general tendency
to anthropomorphize animal and natural phenomena. However, as discussed by
Wells (2009: 51–59), the representation and interpretation of animal characters
as correlated with or reflecting humans is not completely homogeneous, with
many examples of similar forms of deviation from traditional taxonomies in the
representation of animals and humans in other globalized franchises.
Figure 4. Examples from the franchises Hello Kitty, Mickey Mouse short movies, and
Digimon.
Gotta face ‘em all
551
The various franchises (see Fig. 4) present similar abstract forms of representation
of eyes and gazes, all sharing an implicit understanding of the face as the sign
for the communicative and agentive capacity of “superior” living beings. What
would thus be different or relevant about Pokémon? The answer can be found
by comparatively analysing the ways in which the visual codes of representation
interact with the diegetic codes and narrative dynamics of the different universes.
This can be done according to two main dimensions of analysis:
- first, internal representational coherence, looking for ways in which the visual
codes of representation confirm, deviate from or overlap the spectators’
encyclopaedic taxonomies;
- second, visual/diegetic coherence, looking for ways in which the narratives
reinforce, ignore or contradict the visual logic of representation and the
spectators’ taxonomies.
As concerns internal representational coherence, in many cases in traditional
animation, despite the simultaneous presence of various degrees of anthropomorphic and zoomorphic characters, there are clear visual indicators (sign-index)
which act as discriminators to ensure the taxonomic separation between the
different classes. In Hello Kitty, for instance, only the human-like characters have
clothes and the property of language, while Charmmy Kitty the pet has neither.
Despite the clear sharing of animal visual traits, the cultural segmentation between
humans and animals is conveyed through different visual sub-codes. This also
happens in the modern (post-1950s) narratives of Mickey Mouse, in which the
anthropomorphic characters share a common language and possess clothes (albeit
in partially incoherent ways).
On the other hand, in the early Mickey Mouse short movies (up to the 1930s),
as well as in other franchises such as Winnie the Pooh, Digimon and Yu-gi-oh,
there is no coherent set of visual cues to ensure the stability of the spectators/
readers’ encyclopaedic taxonomies described by Eco (1975, 1979). As regards
clothing, there is the simultaneous presence of anthropomorphic animals with no
clothes and others that are partially or fully clothed, although there is coherence
in anthropomorphic behaviour and property of language. From this perspective,
Pokémon is closer to the latter group, with the notable addition that the degree of
its hybridism is far more pronounced than in traditional Western animation.
As concerns visual/diegetic coherence, Pokémon distances itself from the
others in relation to the coherence between the visual and diegetic codes of
representation. Most of the above-mentioned franchises interact with possible
incoherence in internal representation either by ignoring it or by normalizing it
through the diegetic dimension of the narratives, in an attempt to facilitate the
interpreter’s suspension of disbelief or to portray it in a comedic light. On the
552
Vincenzo Idone Cassone
contrary, the anthropomorphism in Pokémon is both internally incoherent (with
many Pokémon assuming various degrees of human traits, or being based on
humans) and diegetically exposed. While certain Pokémon may raise fewer issues
for the interpreter’s cultural encyclopedia, others do so to various degrees, with the
further fact that incoherencies are highlighted within the franchise itself, mainly
through the descriptions in the PokéDex (the diegetic encyclopedia of the Pokémon
Universe), as well as by meta-referential elements. The most notable examples are:
- Behaviour: the ethologic descriptions in the PokéDex often focus on their
physical attributes, instincts or basic behaviour, yet many Pokémon possess
specific behaviours which duplicate or resemble cultural ones attributed to
humans. Thus, Frostlass (#478) freezes the travellers who reach its cave, then
uses them as decoration for its home; Conkeldurr (#534) invented, and taught
humans how to make, concrete; Aggron (#306) reclaims its territory after
natural disasters, replanting trees and hauling topsoil. Some of these features are
particularly incoherent because they are attributed to a species, while apparently
describing unique characters (#778 Mimikyu’s disguise, as well as Cubone’s
skull).
- Clothing: while certain Pokémon are naked (or clothed by their masters), others
include recognisable clothes in their design, from belts to ribbons to gi and
pants. Some of these items are explicitly stated not to be clothes, but parts of the
Pokémon body (such as #068 Machamp’s champion’s belt and briefs and #560
Scrafty’s loose trousers), while other Pokémon use clothes with no explanation
(#814 Raboot’s hoodie, #743 Ribombee’s scarf); some Pokémon even sew
them (#542 Leavanny, when it encounters baby Pokémon); often, for many
Pokémon no specifications are provided (#539 Sawk’s gi, #107 Hitmonchan’s
boxing gloves).
- Food: in a world with no animals aside from the domestic/tamed Pokémon,
the representation of food easily leads to diegetic inconsistencies. The games
and anime openly mention and represent meat-like and fish-like dishes, with
no elucidation. Simultaneously, sometimes the franchise clearly states that
Pokémon are used as food (Chansey’s eggs, Miltank’s milk, Slowpoke’s tails,
as well as whole Farfetch’d, Cherubi, Sharpedo): the latest games in the series,
Pokémon Sword/Shield, include the possibility of creating curry with what
appear to be recognisable Pokémon parts (tails) or meat-like elements (hot
dogs, meatballs).
- Death: while Pokémon cannot die directly in the games (they simply “faint”, once
defeated) and almost never die “onstage” in the anime/manga, death has been
mentioned since the very first games, with a Pokémon graveyard and numerous
examples of deaths from old age or by violence, many of which originate ghost
Gotta face ‘em all
553
Pokémon. A lot of the recent entries of the PokéDex discuss the violent and bitter
deaths of humans/Pokémon which have made them become ghosts.
- Pokémon/human differences: besides humanoid Pokémon, there are problematic insights into the relationship between Pokémon and humans. In the
fourth generation games, a fictional textbook in the Canalave Library reports
that “once, humans and Pokémon married […] they were the same” (Game
Freak 2006). In the same games, the Pokémon God Arceus is stated to have
created the universe, as well as Pokémon and humans. In the fifth generation
games, the main antagonist (N) grew up among the Pokémon and is able to
talk with them. Furthermore, the PokéDex presents many cases of permanent
transformations from human to Pokémon, for instance Kadabra (#64),
Phantump (#708) and especially Yamask (#562): once human, now Pokémon,
the latter carries its own past face as a mask, in order to find people who may
remember it.
- Comedic/serious attitude: while the Pokémon franchise mostly follows the
comedic attitude and its traditional targets are children and young adults, there
are many sudden variations in tone, for instance in the city of Lavender in
the first game, as well as certain other dark/bitter/unsettling PokéDex entries:
Pokémon killing people by mistake (#760 Bewear), dying in pain because
of their masters (Mega evolutions), abducting children (#425 Driftloom),
presenting fatal biological conditions (#325 Spoink) or being damned/
extinguished due to climate change (#864 Cursola).
In conclusion, the Pokémon universe manifests a higher degree of visual and
diegetic incoherency in comparison to other contemporary or traditional animated franchises: its explicit taxonomic heterotopy, and the dynamic of visual
animatedness, are supported and strengthened by internal diegetic and visual
incoherencies leading to what I would call a general heterotopic regime of animation. The apparent self-contradictions and incoherencies are not ignored
or normalized; instead, they are highlighted and reinforced through the metareferential aspects and contrasting tones of the internal self-description system
of the PokéDex. Many of the above-mentioned examples also involve clear cases
of ludo-narrative dissonance (Seraphine 2019)9, making gameplay and narratives
openly clash during the games.
9
Seraphine, Frederic 2019. The rhetoric of Undertale – Ludonarrative dissonance and
symbolism. Digital Games Research Association JAPAN, Proceedings of 8th Conference. 2019.
Available at: https://www.researchgate.net/profile/Frederic-Seraphine/publication/323545890_
The_Rhetoric_of_Undertale-Ludonarrative_Dissonance_and_Symbolism/links/
5a9bdb8545851586a2acea7d/The-Rhetoric-of-Undertale-Ludonarrative-Dissonance-andSymbolism.pdf.
554
Vincenzo Idone Cassone
4. Visual animism and playful interaction
A notable example is represented by Pokémon #778 Mimikyu, who embodies
both the self-contradictory taxonomy of the franchise and a clear case of its metareferential and self-communicative (Lotman 1990:20) nature. Mimikyu is a ghost/
fairy Pokémon which, as it says in the PokéDex, uses a rag in order to disguise
itself as Pikachu and hide its true, terrifying appearance. According to other
descriptions, this Pokémon suffered from extreme solitude, and thus decided to
take over Pikachu’s mantle in order to make new friends as the yellow mouse was
so famous. Mimikyu is a notable case of meta-reference in Pokémon, in which
the diegetic universe hints at the artificial nature of the franchise by referring to
an “external” world where Pikachu is not just a Pokémon among others, but the
franchise mascot, or more precisely, its face.
Many long-term fans have appreciated the meta-referential elements of the
franchise such as Mimikyu,10 (or the eighth generation fossil Pokémon)11,
elaborating many fan theories that try to address the most glaring contradictions
in the franchise (below).
The success of Pokémon may suggest that its heterotopic nature and declared
diegetic self-contradictions have been accepted and have now culturally developed into a proper set of semiotic dynamics, with their own visual codes of
representation and diegetic structures, which have a clearly recognizable role in
the mediasphere. Pokémon, in itself, could thus be paradigmatically used to discuss
the manifestation of much subtler changes in certain semiotic codes of behaviour
related to animated faces.
Following Lotman and Lévi-Strauss, it should be remembered that fictional
narratives and myths, both traditional and modern ones, interact with cultural
self-models and contribute to their strengthening or weakening. Thus, as stressed
by Tønnessen and Tüür (2014), as well as Bertrand and Marrone (2019), any form
of textualized categorization of otherness (whether non-human animals, nonliving beings, or non-natural entities) in relation to the subjectivity (we humans),
interacts with and takes part in the many codes shaping the human semiosphere.
Animals, spirits, gods and inanimate beings are classified and interpreted according
to cultural taxonomies and placed in specific parts of the cultural self-description
systems of the semiospheres, depending on their relationships with the core forms
10
Websites of internet culture such as Kotaku and Polygon have dedicated several articles to
fan reaction surrounding Mimikyu; such as https://www.polygon.com/2016/7/19/12225262/
pokemon-sun-and-moon-new-pokemon-mimikyu.
11 See https://www.polygon.com/2019/11/22/20977707/pokemon-sword-shield-fossildracozolt-arctozolt-dracovish-arctovish-pokedex-england-fake.
Gotta face ‘em all
555
of subjectivity of the system itself. As a result, the ways in which even fictional
narratives display and describe animals (or animaloids) through texts, pictures and
daily practices can highlight significant dynamics of the semiosphere, especially in
connection with the principal aspect of the face (and its representation) as a key
semiotic interface for communication among living beings, which is now being
increasingly textualized in the media system.
In my view, the many above-mentioned visual and narrative features of the
Pokémon franchise may show two significant cultural dynamics which have been
emerging in the contemporary Japanese semiosphere in recent times: the first is
a logic of visual animism of characters; the second is a regime of playful interaction
with the animated.
In relation to the former, we can refer to Descola’s (2013) four-fold schemas of
nature/culture ontologies to understand Pokémon franchise logic as an animist
ontology: simply put, divergences in exteriority may exist between the different
conceived natural and cultural kingdoms, but there is a shared community in their
interiority. Furthermore, as previously discussed, the world of Pokémon often leans
towards what Descola would call a totemic ontology, which is characterized by a
community of both interiority and exteriority. This is the result of the frequent
simultaneous hybridizations, heterotopies and undermining of the encyclopaedic
boundaries between species, natural kingdoms, and what is natural, artificial or
cultural.
Within this heterotopian ontological system, the representation of faces acts as
a double-layer mediatory interface. First, as previously discussed, the face is the
paradigmatic interface for human communication, linking the somatic and the
semiotic, the natural and the cultural. Anthropomorphic and expressive animated
faces make communication possible between humans and Pokémon, both among
characters and in relation to the reader, even though they do not share a language,
because of their shared gestural and expressive codes as sentient beings.
Moreover, on the level of media representation, anthropomorphic face representation acts as a mediatory interface between the heterotopian (animist/totemist)
ontology of Pokémon and the common naturalist ontology of the public/viewers.
By following the tropes of traditional animation, facial representation makes it
possible to control and reduce the encyclopaedic uncanny (das unheimliche, see
Foucault 1967), translating it into something understandable, inoffensive, even
cute and worthy of affection (inheriting the kawaii aesthetics, see Kinsella 1995).
An indirect example of this dynamic of facial representation is provided by the
uncanny effects generated by its removal/modification in the recent film Detective
Pikachu (2017). The CGI + live action movie is set in the Pokémon universe, yet
in order to ensure coherence between CGI and real actors, the directors chose to
556
Vincenzo Idone Cassone
abandon the typical visual flatness of the anime and video game. Instead, they
adopted a “realistic” representation of Pokémon,12 showing and animating in
detail their fur and skin, eyes and gaze, bones and muscles. Despite the movie’s
success, this choice sparked a debate on the internet over the unsettling results:13
the representations of several Pokémon were interpreted as problematic or slightly
disturbing (Jigglypuff, Psyduck, Lickitung) exactly because of the substitution of
typical anime stylization/flatness with iconic bodily and zoomorphic traits. Among
the many, the most notable was a highly uncanny and creepy representation of the
(above-mentioned) humanoid Pokémon Mr. Mime (Fig. 5). The texturing of its
skin (lights, skin pores), together with the motion-capture animation and perfectly
acted gaze, in relation to its already anthropomorphic form/behaviour, gave rise
to both disturbance and amusement, eliciting comparisons of this Pokémon to the
unsettling clown in the film It and to Chucky the doll.
Figure 5. Mr. Mime in Detective Pikachu.
This example brings us to the second dynamic, the regime of playful interaction
with the animated. As anticipated in the previous pages, it is very common for
fans of the franchise to highlight the many inconsistencies and ludo-narrative
dissonances of Pokémon. However, rather than leading to negative evaluations and
disengagement, this often results in positive playful mocking and the development
of fan theories to address the clear inconsistencies. Mr. Mime is again a fitting
example, because of the long-lasting theories involving Ash’s (the main character
in the anime) lack of a father, who is never shown or discussed in the anime.
12
See https://www.wired.co.uk/article/pokemon-detective-pikachu-cgi.
See https://www.huffpost.com/entry/realistic-pokemon-from-detective-pikachu-arecreeping-people-out_n_5beb41e1e4b0caeec2bef1ff.
13
Gotta face ‘em all
557
According to this theory, Ash’s true father is his mother’s Mr. Mime (Mimey)
which/who has lived under the very same roof since the boy was young, does a lot
of housework, uses clothes and accompanies his mother on vacation, and never
stays in the Pokéball. The theory goes further by hinting at many disparate plot
elements and trying (in a comedic way) to explain all of them by means of the
relationship between Ash and Mimey.
The many hundreds of Pokémon theories on the net14 rarely try to prove
themselves true; instead, they continuously shift between being half-serious and
half-facetious, simultaneously maintaining certain stances and their reversal,
according to the specific notion of playful communication defined by Bateson
(1956), according to a playful regime of interaction, as the paradoxical process
of make-believe, in which the logic paradox itself is staged (i.e. ‘to play fight’ for
children is simultaneously to fight as if doing so seriously, and to know one is not
really fighting at all).
Internet fan cultures (both Japanese and Western ones) react to the incoherence,
meta-referentiality and heterotopic taxonomy in Pokémon in a playful way. Fans
take advantage of parts of ludo-narrative or diegetic incoherence in order to
search playfully for possible alternative explanations or re-interpretations, doing
so with an ironic distance, a tongue-in-cheek attitude. They search seriously for
coherence with their encyclopaedic everyday codes, which involves a complex and
time-consuming (re)interpretive process of the franchise’s countless texts; yet, the
results are often parodic and equally self-contradicting, if not more convoluted,
clearly because of the fun afforded by the ridiculous consequences of this search
for coherence, which ignores many typical narrative tropes of animation.
This very logic of interaction is encouraged and adopted by the authors and
managers of the franchise, who purposely do not focus on strong internal narrative
coherence, willingly presenting meta-references, contradictions among texts and
within them, playful communication with fans and audience. For instance, the
explicitly incoherent nature of the PokéDex descriptions has led to the fan theory
that its entries were written by the ten-year-old characters of previous games,15
thus explaining the Pokémon individualization, the impossible measurements, the
conflicting entries and descriptions by rumours, and so on.
14
15
See https://www.thegamer.com/classic-pokemon-fan-theories-believe-true/.
See https://www.cbr.com/pokemon-fan-theory-pokedex-creepy-entries/.
558
Vincenzo Idone Cassone
5. Playful visual animism in Japan
My hypothesis is that the connection between an animistic visual logic of representation and a logic of playful interaction is increasingly common in the Japanese
mediasphere, despite often being assimilated to traditional tropes of animation.
In the following pages I discuss how what I label ‘playful visual animism’ can be
identified in other daily practices linked to the presence of animated faces in Japan,
and in their connection to pre-existing cultural dynamics and semiotic codes.
Many authors have highlighted the massive presence of animated characters (or
kyara, see Occhi 2010, Wilde 2019, Sutera 2016) in Japanese society. ‘Kyara’ is a
generic term which includes anthropomorphic franchise mascots, characters from
manga and anime, as well as human figures used for brands or communication
(imeeji kyara). Kyara proliferate in Japanese society as part of the above-mentioned
media mix strategies, being extensively used for both large and small brands, being
adopted for gadgets and merchandizing, diffusing the kawaii aesthetics, and even
taking part in parades in costume (kigurumi) and receiving official recognition
from the Japanese government.
In addition to the ubiquitous brand kyara, Alt and Yoda (2007) highlight
the existence of the so-called working characters/service characters (hataraku
kyara; Wilde 2018), which proliferate in daily metropolitan life, being utilized in
road signs, municipal announcements, political advertizing and everyday object
packages (see Fig. 6). Both anthropomorphic and zoomorphic (Miller 2010), these
characters act as mediators of everyday life, giving indications and directing the
behaviour of passers-by, reminding us of social norms, wishing good luck and
fulfilling other apotropaic functions. They act in a similar way to Latour’s petit
Robert (see Latour 1992): whereas the mechanical boy replicated by figurization the
agentivity of the user of the machinery, these characters embody and replicate the
affective communicative dynamics of the participants in the semiotic encounter (be
it a purchase, warning, advice, etc). Nozawa (2013) argues that everyday characters
are the bearers of a semiotic ideology that cannot be reduced to functional
anthropocentrism: because of their multiplication and frequency of interaction in
daily life, as well as their relationships with other phenomena of human-character
embodiment (vocaloids, kyara actors and personae) they manifest a semiotic
ontology of transition that unites people and characters, animals and animates, the
ordinary and the fictional, coherence and inconsistencies.
Gotta face ‘em all
559
Figure 6. Service characters in Japan (the author’s photos).
In addition to hataraku kyara, yuru kyara (or Gotōchi-kyara; Occhi 2014) are
increasingly common in Japan. Taking their name from the wobbly, imprecise
drawings which constitute them, these characters are often linked to local places,
events or products. Despite their simplistic aesthetics, they frequently present
hybrid forms, mixing the natural and the cultural, the zoomorphic and the
anthropomorphic, combining in their simple representation typical products,
myths and folklore, lifestyles and stereotypes. Incorporating kawaii aesthetics and
neotenic characteristics (Kinsella 1995), they may feature a double nature: twodimensional on one side, mascot in costume (kigurumi) on the other, at times
participating in specific local parades and national competitions (see Fig. 7).
Occhi (2012) has linked these characters to the religious animist traditions
of Japan, by regarding them as secularized/popularized forms of the genius loci:
anthropomorphic folkloric imagery (kappa, tanuki, kitsune) or the representation of
vital forces and spirits (kami) that manifest themselves in the natural world. With the
notable difference that, whereas the genius loci mediated between the supernatural
and the sensible, the yuru kyara and hataraku kyara become mediators of an earthlier
civic, promotional or commercial message, while often acting as emotional support,
linked to a sense of relief, comfort and regression (Kondo 2006), partially confirmed
by the attested use of kyara in modernized omamori.
560
Vincenzo Idone Cassone
Figure 7. Yuru kyara in kigurumi form.
The religious and mythic dimensions of these kyara further highlight the ludic
regime of interaction with them, shifting between seriousness and playfulness.
Moreover, this is linked to the long-term tradition which connects acting,
playfulness and devotion through religious practices. This confirms Foster’s (2009:
13) assertion about the persistence of yōkai (Japanese supernatural spirits and
entities) in the cultural imagination because of their ability to pivot between the
credible and incredible. This link is clear in the traditional matsuri performances,
as well as in the “play of the gods” (kami asobi; Plutchow 1996), and is, to a degree,
reproduced in a secularized form, in the kigurumi parades and television mascot
competitions. The ludic nature of these secularized religious practices, mediated
by characters, makes possible the paradoxical communication (see Bateson 1956)
which simultaneously includes affective approach and cognitive distancing.
Once observed through the double perspective of visual animism and playful
interaction, these phenomena (and Pokémon) show further connections with certain
semiotic characteristics of contemporary postmodernism. Azuma (2009; see also
Lamarre et al. 2007) describes the Japanese shift in the 1990s from the logic of the
postmodern as a narrative to that of the postmodern as a database, suggesting that
Otaku culture could be considered the most sensible place to observe this change. In
what he labels the animalization era, the logic of narrative coherency is substituted
by the affective logic of the database, which, in Lotmanian terms, acts as a selfmodelling system (Madisson 2016). Texts based on the self-model of the database
Gotta face ‘em all
561
involve micro-narratives, centred on specific characters, themes and aesthetics
which can be subject to infinite remodelling, with no essential traits connected to
the characters. What remains is not the coherent personality traits, but rather the
affective relationship between the user and the character.
As a result, an interesting parallel develops between the logic of the database
and the dynamics of the collection, which is the way in which people interact
with intangible characters: explicitly, in the case of Pokémon, and cognitively for
kyara in general. On the one hand, the collection is a model of organization of the
sensory in a playful form; on the other hand, the collection is a reorganization of
the world into a microcosm of affections. Foster (2009: 214), for instance, points
out the ludic and taxonomic similarities between Pokémon and yōkai, which
could also be expanded to the contemporary interaction and communication
with the above-mentioned kyara. These long-term dynamics are reproduced in the
contemporary world, placed in a media circuit of production, sharing, interaction
and evaluation. As I suggested in the introduction, this further strengthens the idea
of Pokémon as an emblematic case study for the understanding of the semiotic
dimension of animated characters in the Japanese mediasphere.
Conclusion
The examples discussed above indicated how the Japanese mediascape is permeated
with anthropomorphic characters and faces, ranging from narrative transmedia
franchises to branding, to merchandise, as well as an increasing presence in
internet and fan culture. I have suggested how their spread may be connected to the
emergence of specific semiotic behavioural codes and as a result of the interaction
between the codes of visual representation of these anthropomorphic characters,
their fictional narratives and the sociocultural patterns of interaction between
humans and characters. Characters become key symbolic mediators (Lotman 1990)
of everyday life, as well as “semiotic figures of in-betweenness” (Nozawa 2013: 8)
in the semiosphere.
Through the analysis of the Pokémon case study, I defined two interrelated
features: a logic of visual animism of characters; and a regime of playful interaction
with the animated. These dynamics need not be limited to the Pokémon franchise,
but can be attested in many anthropomorphic face-based characters and practices
in the Japanese media system and culture. The resulting patterns of playful
visual animism may help understand the intersection between visual stylized
representation, the database-logic of media, fan culture behaviour and cultural
appropriation of technologies in Japan.
562
Vincenzo Idone Cassone
These dynamics may be correlated with recent changes in the semiotic dynamics
of the Japanese media system over the last few decades, namely its process of
globalization (Daliot Bul, Otmazgin 2020; Pellitteri 2018), its participation in
the convergence culture and contemporary transmedia dynamics (Jenkins 2006;
Scolari 2009; Saldre, Torop 2012), and its relationships with the broader dynamics
of internet culture.
References
Appadurai, Arjun 1990. Disjuncture and difference in the global cultural economy. Theory,
Culture & Society 7(2/3): 295–310. https://doi.org/10.1177/026327690007002017
Azuma, Hiroki 2009. Otaku: Japan’s Database Animals. Minneapolis: University of Minnesota Press.
Allison, Anne (ed.) 2006. Millennial Monsters: Japanese Toys and the Global Imagination. Vol.
13. Berkley: University of California Press. https://doi.org/10.1111/j.1467-9655.2007.
00472_9.x
Alt, Matt; Yoda, Hiroko 2007. Hello, Please! Very Helpful Super Kawaii Characters from
Japan. San Francisco: Chronicle Books.
Barrows, Lissa 2014. Planning with Character: Gotouchi Kyara and Place Branding in Japan.
[Unpublished Master’s Thesis, M.Sc.] New York: Columbia University.
Bateson, Gregory 1956. The message “This is play”. Group Processes 2: 145–241.
Bendazzi, Giannalberto 2016. Animation: A World History. Vol. 3: Contemporary Times.
Boca Raton: CRC Press. https://doi.org/10.4324/9781315720746
Bertrand, Denis; Marrone, Gianfranco (eds.) 2019. La sfera umanimale. Roma: Meltemi.
Bruce, Vicky; Young, Andrew W. 2012. Face Perception. London, New York: Routledge.
https://doi.org/10.4324/9780203721254
Daliot-Bul, Michal; Otmazgin, Nissin 2020. The Anime Boom in the United States: Lessons
for Global Creative Industries. Leiden: Brill.
Descola, Philippe 2013. Beyond Nature and Culture. Chicago: University of Chicago Press.
https://doi.org/10.7208/chicago/9780226145006.001.0001
Eco, Umberto 1975. Trattato di semiotica generale. Milano: Bompiani.
Eco, Umberto 1979. The Role of the Reader: Explorations in the Semiotics of Texts. Bloomington: Indiana University Press.
Eco, Umberto 2000. Kant and the Platypus: Essays on Language and Cognition. Boston:
Houghton Mifflin Harcourt.
Foster, Michael Dylan 2009. Pandemonium and Parade: Japanese Monsters and the Culture of
Yokai. Berkeley: University of California Press. https://doi.org/10.1525/9780520942677
Foucault, Michel 1967. Let mots et les choses: Une archéologie du savoir. Paris: Gallimard.
Game Freak 2006. Pokèmon Diamond/Pearl. Nintendo.
Han, Ruobin 2017. The Characterization of Japan. From marketing to Identity. [Unpublished
Ph.D. Thesis in Humanities.] Leiden: University of Leiden.
Hutchinson, Rachel 2019. Japanese Culture Through Videogames. London, New York:
Routledge. https://doi.org/10.4324/9780429025006
Gotta face ‘em all
563
Iwabuchi, Koichi 2002. Recentering Globalization: Popular Culture and Japanese Transnationalism. Durham: Duke University Press. https://doi.org/10.1515/9780822384083
Jenkins, Henry 2006. Convergence Culture: Where Old and New Media Collide. New York:
New York University Press.
Kinsella, Sharon 1995. Cuties in Japan: Women, media and consumption in Japan. In:
Moeran Brian; Skow, Lise (eds.), Women, Media and Consumption in Japan. London:
Routledge, 220–254.
Kobayashi, Hiromi; Kohshima, Shiro 2001. Unique morphology of the human eye and
its adaptive meaning: Comparative studies on external morphology of the primate
eye. Journal of Human Evolution 40(5): 419–435. https://doi.org/10.1006/jhev.2001.0468
Kondo, Kensuke 2006. Hyaku nen ai sareru kyarakutaa no tsukurikata. [How to create
characters that will be loved for one hundred years.] Tokyo: Goma Books.
Lamarre, Thomas 2018. Animation and animism. In: Boehrer, Bruce; Hand, Molly;
Massumi, Brian (eds.), Animals, Animality, and Literature. Cambridge: Cambridge
University Press, 284–300. https://doi.org/10.1017/9781108595278.017
Lamarre, Thomas; Azuma, Hiroki; Furuhata, Yuriko; Steinberg, Marc 2007. The animalization of Otaku culture. Mechademia Second Arc 2(1): 175–187. https://doi.org/10.1353/
mec.0.0023
Latour, Bruno 1992 Where are the missing masses. In: Bijker, Wiebe E.; Law, John (eds.),
Shaping Technology-Building Society: Studies in Sociotechnical Change. Cambridge: MIT
Press, 225–259.
Leone, Massimo 2009. Agency, communication, and revelation. Lexia 3/4: 77–95.
Leone, Massimo 2018. The semiotics of the face in the digital era. Perspectives 17: 27–29.
Leone, Massimo 2021. On muzzles and faces: The semiotic limits of visage and personhood.
International Journal for the Semiotics of Law/Revue internationale de Sémiotique
juridique 34(2): 1–24. https://doi.org/10.1007/s11196-020-09812-8
Lotman, Yuri. M. 1990. Universe of the Mind: A Semiotic Theory of Culture. London: IB
Tauris.
Madisson, Mari-Liis 2016. Self-description. In: Rodriguez Higuera, Claudio Julio; Bennett,
Tyler James (eds.), Concepts for Semiotics. (Tartu Semiotics Library 16.) Tartu: Tartu
University Press, 204–212.
Maran, Timo; Martinelli, Dario; Turovski, Aleksei (eds.) 2011. Readings in Zoosemiotics.
Vol. 8. Berlin: Walter de Gruyter. https://doi.org/10.1515/9783110253436
Miller, Laura 2010. Japan’s zoomorphic urge. ASIANetwork Exchange: A Journal for Asian
Studies in the Liberal Arts 17(2): 69–82. https://doi.org/10.16995/ane.208
Occhi, Debra J. 2010. Consuming Kyara ‘characters’: Anthropomorphization and marketing
in contemporary Japan. Comparative Culture 15: 77–86.
Occhi, Debra J. 2012. Wobbly aesthetics, performance, and message: Comparing Japanese
Kyara with their anthropomorphic forebears. Asian Ethnology 71(1): 109.
Ōtsuka, Eiji 1989. Monogatari shōhiron. [A Theory of Narrative Consumption.] Tokyo:
Shinyosha.
Pellitteri, Marco 2018. Kawaii aesthetics from Japan to Europe: Theory of the Japanese
“cute” and transcultural adoption of its styles in Italian and French comics production
and commodified culture goods. Arts 7(24): 1–24. https://doi.org/10.3390/arts7030024
564
Vincenzo Idone Cassone
Plutschow, Herbert 1996. Matsuri: The Festivals of Japan: With a Selection from P.G. O'Neill's
Photographic Archive of Matsuri. London, New York: Routledge.
Saldre, Maarja; Torop, Peeter 2012. Transmedia space. Crossmedia Innovations: Texts,
Markets, Institutions 51: 25–44.
Scolari, Carlos Alberto 2009. Transmedia storytelling: Implicit consumers, narrative worlds,
and branding in contemporary media production. International Journal of Communication 3: 586–-606.
Sutera, Jillian Rae 2016. Who is Hikonyan? The phenomenon of Japanese Yuru-Chara.
Sociology 6(12): 775–782. https://doi.org/10.17265/2159-5526/2016.12.004
Surace, Bruno 2017. Zoosemiotica dei Pokémon. In: Marrone, Gianfranco (ed.), Zoosemiotics 2.0: Forms and Policies of Animality. Palermo: Edizioni Museo Pasqualino,
609–619.
Steinberg, Marc 2012. Anime’s Media Mix: Franchising Toys and Characters in Japan.
Minneapolis: University of Minnesota Press. https://doi.org/10.5749/minnesota/
9780816675494.001.0001
Steinberg, Marc 2015. 8-Bit Manga: Kadokawa’s Madara, or the gameic media mix.
Kinephanos: Journal of Media Studies and Popular Culture 5: 40–52.
Tobin, Joseph (ed.) 2004. Pikachu’s Global Adventure: The Rise and Fall of Pokémon.
Durham: Duke University Press. https://doi.org/10.2307/j.ctv1131ctc
Tønnessen, Morten; Tüür, Kadri 2014. The semiotics of animal representations: Introduction. In: Tønnessen Morten; Tüür, Kadri (eds.), The Semiotics of Animal Representations. Leiden: Brill Rodopi, 7–30. https://doi.org/10.1163/9789401210720_002
Wells, Paul 2009. The Animated Bestiary: Animals, Cartoons and Culture. New Brunswick:
Rutgers University Press. https://doi.org/10.36019/9780813546438
Wilde, Lukas R. A. 2018. Character street signs (hyōshiki): “Mangaesque” aesthetics as
intermedial reference and virtual mediation. Orientaliska Studier 156: 130–150.
Wilde, Lukas R. A. 2019. Kyara revisited: The pre-narrative character-state of Japanese
character theory. Frontiers of Narrative Studies 5(2): 220–247. https://doi.org/10.1515/
fns-2019-0014
Gotta face ‘em all: покемоны, японские анимационные персонажи и
появление игрового визуального анимизма
В результате развития технологий и новых культурных практик современная медиасфера всё больше заселяется цифровыми лицами. Это явление не ограничивается человеческими лицами, но включает в себя множество вымышленных анимированных
лиц, называемых «персонажами», «маскотами» или kyara. Явление kyara, давно известное в Японии, широко распространяется благодаря глобализации, цифровизации и технологиям медиа-микширования. Таких персонажей, соединяющих области
визуального дизайна, вымышленных историй и социокультурного потребления,
можно считать семиотическими знаками промежуточности, ключевыми символическими медиаторами в японском медиапространстве. Антропоморфный дизайн их
лиц стирает культурные границы между человеком и не-человеком, одушевленным
и неодушевленным, природой и культурой. Кроме того, постмодернистские истории,
Gotta face ‘em all
565
частью которых они являются, вдохновлены прошлым и настоящим, мифами и научной фантастикой. Наконец, они подразумевают энциклопедическую переработку
на границе между фантастикой и реальностью, мифическим и секулярным, игрой и
реальностью. В статье рассматриваются семиотические измерения kyara в современной Японии с акцентом на логику их репрезентации и культурное влияние. Анализ
франшизы «Покемо́ н» позволяет описать возникновение новых семиотических паттернов «игрового визуального анимизма» в процессе медийного «олицетворения»
повседневной жизни.
Gotta face ‘em all: Pokémonid, Jaapani animategelased ning
mängulise visuaalanimismi esiletõus
Tehnoloogiliste uuenduste ning uute kultuuripraktikate tulemusena sisustavad kaasaegset
meediasfääri ühe enam digi(taliseeritud) näod. See nähtus ei piirdu inimnägudega, vaid
siia kuulub ka fiktsionaalsete animanägude suur universum; neid nimetatakse ‘tegelasteks’,
‘maskottideks’ või ‘kyara’. Kuigi kyara pole Jaapanis uus ilming, on nad hakanud levima
tänu globaliseerumisele, digitaliseerumisele ja kombineeritud meedia strateegiatele.
Visuaalse disaini, fiktsionaalsete narratiivide ja ühiskondlik-kultuurilise tarbimise seoste
kaudu võib kyara’t pidada semiootilisteks vahepealsuskujudeks, Jaapani meediamaastiku
peamisteks sümboolseteks vahendajateks. Nende antropomorfne näodisain vahendab
kultuurilisi piire inimliku ja mitteinimliku, elusa ja elutu, looduse ja kultuuri vahel. Lisaks
seguneb nende postmodernsetes narratiivides minevikust ja olevikust lähtuv inspiratsioon
müütidest ulmeni. Ning viimaks tähendavad nad entsüklopeedilist ümbertöötlemist,
mis leiab aset väljamõeldise ja tegelikkuse, müütiliste osutuste ja sekulariseerumise,
tõsiduse ning mängulise teeskluse valdkondade vahel. Artikli eesmärgiks on uurida kyara
semiootilisi mõõtmeid tänapäeva Jaapanis, rõhuasetusega nende representatsiooniloogikale
ja kultuurilistele tagajärgedele. Neid käsitletakse, analüüsides Pokémoni frantsiisi, mis
võimaldab kirjeldada “mängulise visuaalanimismi” uute semiootiliste mustrite esiletõusu
igapäevaelule media poolt nägude omistamise käigus.
566
Peter Mantello
Sign Systems Studies 49(3/4), 2021, 566–589
Fatal portraits:
The selfie as agent of radicalization
Peter Mantello1
Abstract. For the modern-day jihadist, the digital self-portrait or, more specifically,
battlefield selfie is a popular tool for identity building. Similarly to the selfies taken
by non-violent practitioners of self-capture culture, the jihadist selfie represents an
alternative to the Cartesian formulation of a unitary and indivisible self. Rather, it
is a product of social relations and performative actions, constituted in dialogue
with others through very specific socio-cultural frameworks and expectations.
However, unlike its non-violent Doppelgänger, the expectations of this dialogue
are centred around a larger political agenda which actively seeks to reformat
collective memories of imperial Islamic conquests and co-opt religion as a way
to impose a moral order on its violence. Importantly, the battlefield selfie allows
the jihadist easily to traverse the boundaries between two seemingly opposing
belief systems. Although there exists a wealth of scholarship of self-capture culture,
image sharing sites and micro-celebritism, their pervasive influence and practice
on battlefield is understudied. This article draws from the personal histories of
key Islamic extremists who were both lionized and demonized as a result of their
battlefield micro-influencer practices. Today, however, the same individuals can
achieve internet fame by participating in self-capture culture – posting selfies, videos
or blogging. In other words, never before has a soldier’s public communication been
so personal yet collective.
Keywords: micro-influencer; terror; selfie; self-portrait; jihad; social media;
branding; networked communities; social photo; necro-celebritism
Introduction
A decade ago, the proliferation of Web 2.0 strategies (forums/message boards/
file-sharing sites) heightened the visibility of jihadist movements around the globe
but also empowered online partisans with a sense of shared purpose, agency and,
importantly, community (Zelin 2014). Easily downloadable jihadist “lifestyle”
e-magazines such as Inspire, foreshadowed a new age of information warfare based
1
Ritsumeikan Asia Pacific University, Beppu, Japan; e-mail: mantello@apu.ac.jp.
https://doi.org/10.12697/SSS.2021.49.3-4.16
Fatal portraits: The selfie as agent of radicalization
567
on connectivity, media professionalism and aesthetic-driven content (Droogan,
Peattie 2016). Even more so, it gave the jihadist imaginary a distinctly modern
edge, painting radical Islam as a utopian world where traditional religious
values co-exist with modern-day pleasures of techno-culture: a spiritually
rewarding yet playful universe of memes and mosques, chat rooms and prayer
rooms, selfies and Sharia. Paradoxically, these same screen-mediated practices
so central to contemporary jihadism are based on advancing a violent seventhcentury political project whose core tenets are anti-secular, anti-modern, antiindividualist, anti-consumerist, and above all, anti-liberal.
Aside from these innate contradictions, new Islamic militantism is emblematic of a growing cultural desire for and dependency on digital devices for
self-expression, self-presentation and self-affirmation (Senft 2008; Marwick
2015; Djafarova, Trofimenko 2017; Manovich 2019). As Marshall McLuhan
(1994) prophesized, long before the smartphone was conceived, digital devices
have become the prosthesis of human agency for they extend human consciousness, but in doing so, create a new self-consciousness – a digitally extended self.
For Achille Mbembe (2019)2, this complex tethering of the human and the
technological has given way to subjectivities in line with the dominant strictures
of communicative capitalism and increasing shift in modern society “from a
politics of reason to a politics of experience [where] sensibility, emotions, affect,
sentiments and feelings are the real stuff of subjecthood and therefore of radical
agency”. A prime example of this phenomenon is how violent modes of religiosity
and individuation are now being accelerated through the rising vogue of the ‘selfie’.
The Oxford dictionary defines the selfie as “a photograph that one has taken
of oneself, typically one taken with a smartphone or webcam and uploaded to a
social media website”. Importantly, what separates the selfie as a tool of identity
construction from its non-digital counterpart is its affiliation to online communities. In other words, the selfie is a product of social relations and performative
actions, constituted in dialogue with others through very specific socio-cultural
frameworks and expectations (Papacharissi 2010; Durkheim 2002[1897]; Bakhtin
1973; Dobson, Robards, Carah 2018). However, for the jihadist, the expectations
of this dialogue are centred around a larger political agenda which actively seeks to
reformat collective memories of imperial Islamic conquests and co-opt religion as a
way to impose a moral order on its violent universe (Tibi 2010; Afsaruddin 2014).
2
Mbembe, Achille 2019. Thoughts on the planetary: An interview with Achille Mbembe.
Retrieved on 18 July 2021 from https://www.newframe.com/thoughts-on-the-planetary-aninterview-with-achille-mbembe/.
568
Peter Mantello
As Adam Levin (2014)3 notes, the selfie “moves the onus of experience away from
phenomenal engagements with adjacent strangers and towards digital interactions
with online communities”. Critically, the battlefield selfie pushes the boundaries of
the networked self even further into a collective moment of affective networked
sharing and political formation. Whereas traditional forms of proselytizing once
spoke in a top-down manner directly to the individual as an autonomous
agent (Leone 2015), the peer-to-peer nature of jihadist self-capture culture
promulgates a heteronomous self, constructed through networked interfaces
of familiarity and conformity.
While recent scholarly studies have focused on the strategic, propaganda or
operational aspects of new Islamic militant media (Fisher 2015; Nissen 2015;
Prucha 2016; Berger, Perez 2016; Shehabat, Mitew 2017), this article examines how
the “digital turn” has transformed the jihadist self by allowing it to remain distinct
yet somehow act together as a networked subjectivity (Galloway, Thacker 2007;
Veilleux-Lepage 2016). I argue that the selfie marks the point of transition from
the biopolitical to thanatopolitical, but, in doing so, reconciles the temporal and
ideological conundrum of suturing together a modern-medieval self. Curiously,
this reconciliation relies as much on commercial strategies of capture (Zuboff 2017)
that indirectly turn social media platforms into ideological echo chambers as it
does on the self-representational practices of micro-celebritism that transform
ordinary persons into insta-famous brand influencers.
Although there exists a wealth of scholarship on the rising significance of public
performance in everyday life, commodification of self, and the widening scope of
what constitutes micro-celebrity culture, this article fills an existing vacuum in
the much-needed theorization of the relationship between selfies and extremist
subjectivity. Thus, the article is divided into three sections. The first part examines
how the emergence of Web 2.0 has fundamentally reshaped the way in which
combatants in asymmetrical war are represented and enacted. The second part
traces a brief historical arc over social and cultural origins of celebritism. Here
I draw from seminal literature in celebrity studies in order to situate better the
contemporary jihadist within micro-celebrity culture. The last section explores
the battlefield selfie as a marker of identity realignment that occurs in the process
of a jihadist’s radicalization. My goal here is to illustrate how the cultural logic,
practices and determinants that shape micro-influencers are actually no different
from how modern-day extremists construct, negotiate and mediate notions of self.
3
Levin, Adam. 2014. The selfie in the age of digital recursion. InVisible Culture: An Electronic
Journal for Visual Culture 20. Retrieved on 16 July 2021 from https://ivc.lib.rochester.edu/theselfie-in-the-age-of-digital-recursion/.
Fatal portraits: The selfie as agent of radicalization
569
The rise of the terrorist as a micro-celebrity
Prior to Web 2.0, an extremist’s chances of becoming a celebrity hero were entirely
dependent on the cultural and institutional processes responsible for creating an
entertainment celebrity. Early-20th-century anarchists, postwar colonial insurgents
and even new-left revolutionaries of the 1960s and 1970s were reliant on mass
media outlets to provide them with the kinds of visibility and recognition afforded
to entertainment celebrities (Weimann, Winn 1994; Hoffman 2017). Even then,
they had no control over whether or not their violent deeds or demands would be
accurately communicated by the media. In the aftermath of the first Wall Street
terror bombing in 1920, the anarchists’ media strategy amounted to leaving five
leaflets in a nearby mailbox (Larabee 2000). During the post-war era, Menachem
Begin’s “glass house” strategy used dramatic attacks on British targets in order to
gain media attention so as to turn international opinion against Britain’s occupation
of Palestine (Auerbach, Ben‐Yehuda 1987). In the early 1970s, Osamu Maruoka
and his Red Army colleagues held airline hostages for several days before news
crews showed up to hear his ransom demands (Andrews 2016). Similarly, in the
wake of his OPEC attack in Vienna, Carlos the Jackal, once called the ‘World’s Most
Wanted Man’ and ‘International Face of Terror’, deliberately held off his daring
escape until the arrival of TV news cameramen (Thomas 2013). Even Bin Laden
was dependent on his loyal couriers and agreeable editors at Al Jazeera to have his
VHS-taped messages broadcast to the world (Torres-Soriano 2010). Yet this last
wave of globalized terror distinguishes itself not only by its non-secular orientation
but, importantly, “proactive” media sensibilities. For example, during al Shabab’s
attack on the Westgate Shopping Mall in Nairobi, militants posted images to social
media of their massacre while exchanging text messages with organizational
leaders and followers. Never before in contemporary history have the non-state
combatants possessed a single device enabling them to capture, edit and publish
photos of themselves and their daily experiences; view and rate photos of their
comrades in arms (and enemies), search and discover other like-minded photos;
and enter into conversations with the authors of those photos (Manovich 2019).
The jihadist as micro-celebrity is in part the latest manifestation of a longer
historical continuum of socio-economic formations and networks of globalization
(Appadurai 2006). Scholars have traced the roots of celebritism back to the
industrial revolution and rise of cities (Turner 2013); technological advances in
printing, image-making and travel (Inglis 2010); expansion of urban populations
and consumer culture (Boorstin 1983); and, importantly, the growth of possessive
individualism (Porter 2004). Importantly, as each of these factors evolved, the
concept and focus of fame drastically shifted. Daniel Boorstin (1983: 57) argues
570
Peter Mantello
that the modern celebrity is defined less by their achievements per se, but more by
their being “well-known for well-knownness”. In other words, public recognition is
independent of any achievements. Yet as Theresa Senft (2008) and other scholars
have argued, in the age of social media, a micro-celebrity is more than just a status.
Like the world of brand commodities, a micro-celebrity aspirant must construct
an online identity that stands out from the crowd. Not only must they dedicate a
good portion of their daily life to maintenance of their on-screen other but also
find innovative ways to showcase themselves. Here authenticity is the key. Above
all else, being a micro-celebrity comes with expectations linked to the traditional
prerequisites of celebritidom, namely, presenting a unique persona, viewing
followers as fans, producing content consistent with that persona and engaging
in strategic interaction with the goal of maximizing both intimacy and continued
visibility (Senft 2008; Marwick 2015). For P. David Marshall (2015), the ‘unique
persona’ of the micro-celebrity supplants not only the older, more established
value of class and status. Rather, as Marwick (2015: 17) insists, the micro-celebrity
“becomes the embodiment of the possibility of change through consumer culture”.
However, he adds, the fact that their identity originates as mediatized construct
and, as such a product of consumption, limits the possibility of affecting any radical
transformation. Thus, they become inevitably trapped in “sustaining polities rather
than threatening them” (Marwick 2015: 17).
Arguably, this appears to be the case of famous personalities such as Greta
Thunberg (see Morningstar 2019; Abidin et al. 2020) as much as it was for
(in-)famous terrorists such as Timothy McVeigh or Osama Bin Laden (see Nacos
2007). Thunberg’s sharp rise to celebrity status began as an anonymous, headstrong
teenager campaigning against climate change on the steps of the Swedish
parliament. McVeigh was an impoverished yet decorated US war hero whose
hatred of big government ended with him blowing up a state building and killing
168 people. Although he was depicted in the media as a psychopath, psychiatrists
concluded that McVeigh was neither irrational nor deranged. In fact, Michel and
Herbeck (2001) note that the war veteran saw himself as Luke Skywalker from the
Star Wars film trilogy and the Oklahoma federal building as a real-life metaphor
of the Death Star. Bin Laden, on the other hand, was a soft-spoken, well-educated,
and extremely wealthy Muslim who sacrificed a life of luxury and privilege to
wage a cosmic war against the West. Critically, in all three cases, the formation of
their individuality as celebrities emerged as a result of a well-oiled and established
corporate media machine that sustains itself by recycling dominant discourse in
security politics. While Thunberg has raised greater public awareness of imminent
dangers of climate collapse, her advocacy efforts to reign in industry polluters or
make world leaders accountable have yet to succeed (Morningstar 2019; Abidin
Fatal portraits: The selfie as agent of radicalization
571
et al. 2020). Although McVeigh is a martyr of right-wing militia groups, any
discernible changes in US federal control over state sovereignty brought on by his
actions have yet to materialize. Likewise, while Osama Bin Laden is both lionized
and demonized as the father of global jihad, his utopian dream of a transnational
caliphate remains elusive to Islamic extremist organizations.
The third ingredient of celebritism is intimacy. While 19th-century print media
heightened the visibility and familiarity of unique personas, it was 20th-century
broadcast culture that dramatically shifted public understanding of the celebrity
(Driessens 2012). The advent of radio and TV transported public personalities into
domestic spaces making them “household names”. Fans and followers no longer
regarded the celebrity as a distanced public figure only to be found on the printed
page, stage or screen, but someone one knew intimately (Schickel 2000). Certainly,
the rise of internet and social media has heightened and accelerated “feelings of
closeness” between celebrity and follower, but it has also reconfigured the relational
dynamic. Intimacy is no longer simply about the relationship between celebrity
and follower but now extends to the peer-to-peer connection between community
members. In other words, Schickel’s “intimate stranger” is now inextricably tied to
digitally constructed and networked “intimate publics” (see Dobson, Robards, Carah
2018: 113–128), where members of online communities share a world view and
emotional knowledge that they have derived from common historical experience.
The socially productive attributes of intimate publics can be evidenced in the #MeToo
or Black Lives Matter movements. At the same time, their negative impact on society
can also be illustrated in far-right groups such as the Proud Boys or QAnon.
Similarly, the significance of jihadist selfie lies in its networked communities
and intertextual iterability that makes it possible to amplify emotional
connections to an imagined homeland, kindle nostalgia for a shared sacred
past and, importantly, open up a temporal feedback circuit between the Islamists’
present crusade and apocalyptical future. Like the peaceful denizens of self-capture
culture, the digital intimate publics of Islamic extremism exchange self-curated and
often intimate representations of themselves with others whom they have often
never physically met. As this curation and sharing takes place, they constantly
manage their onscreen identity in light of feedback systems of ‘likes’ and ‘shares’.
It is here where McLuhan’s thoughts on self-amputation and self-amplification
become relevant once again. For the battlefield selfie serves as an instructive
marker of the complex and often opaque processes of identity realignment that
occur when an individual transitions from peaceful Muslim to holy warrior.
In the next section, I examine the various feedback loops and interplay between
self-portraiture culture and the battlefield by dividing the jihadist selfie into three
axes of formation – hipster, warrior and, finally, martyr. I use the term ‘axes’, as
572
Peter Mantello
each may be present to a greater or lesser degree. Not all jihadists will promulgate
a hipster persona, flaunt their battlefield killing, or publicize a desire to become
martyrs. Accordingly, my analysis of these stages draws from the personal histories
of Islamic extremists whose reputations were lionized and demonized as a result
of their micro-celebrity practices.
Violent versus non-violent battlefield selfies
Curiously, while Western mainstream media tends to focus on the darker side of
Islamic extremists, according to Klausen (2015) violent images from the battlefield
constitute a tiny fraction of content that attains viral popularity within jihadi online
communities. Arguably, the reason for the non-violent selfie’s popularity over its
violent counterpart is that it shows an alternative version of jihad which challenges
age-old stereotypes of the Arab world as alien, barbaric and hostile. Instead
non-violent selfies portray a far gentler and more humanistic portrayal of the
religious warrior (see Fig. 1). For example, Ingram, Whiteside, and Winter (2020)
and Winter (2018) note that digital self-portraits of jihadist soldiers petting cute
kittens (dogs are not haram), posing with jars of Nutella, making pizza, playing
videogames, luxuriating poolside or going fishing attract far more popularity than
battlefield combat portraits.
Figure 1. Noncombative Selfie.
Fatal portraits: The selfie as agent of radicalization
573
Another possible reason for the non-violent selfie’s popularity lies in its ability to
normalize jihad as a productive lifestyle choice. For example, two years after Abu
Bakr al-Baghdadi, the leader of the Islamic State of Iraq and the Levant, announced
the establishing of a world-wide caliphate, social media was inundated with selfies
by ordinary civilians living in Islamic-State-controlled regions of Syria and Iraq
wearing Islamic-State-“approved” products. With permission of the Islamic State,
local merchandizers had placed the extremist organization’s infamous black flag on
consumer items ranging from sovereign rings and baseball caps to school bags and
baby clothes (Wyke 2015).4 The transformation of an age-old theo-political symbol
into a merchandizing logo illustrates the Islamic State’s intuitive understanding of
brand as a political marketing tool. While local merchandizers enriched themselves
in their commercial appropriation, the Islamic State heightened the visibility and
public recognition of their crusade. Simons (2018) notes that a survey conducted in
2017 to find the most successful brand, ranked the Islamic State’s logo and symbols
over the Vatican in terms of global public recognition.
Like other ontological incongruencies in contemporary salafi jihadism, the
commodification of an iconic symbol calling for an anti-enlightenment, may seem
a disjuncture between belief and practice. However, as Simons (2018: 334) suggests,
“[branding] is not necessarily solely about the group’s popularity, but more so about
their public visibility and the resulting reputational brand as a means to connect
and form relationships with different constituencies”. Indeed, the monetization
of the Islamic State’s black flag resonated differently with various publics. For
example, as Bandopadhyaya (2019) points out, to unaffiliated supporters far from
the battlefield, producing selfies wearing black flag logo was proof of the Islamic
State’s ongoing success at nation-building. To local populations it signified their
allegiance to (whether authentic or not) the extremist organization. Finally, to
those who may have understood yet dismissed the antithetical aspects of the black
flag logo, it registered within them a deeper emotional connection to the Islamic
State’s overall cosmic crusade.
4
Wyke, Tom 2015. The bizarre world of ISIS fashion: Terror group is now selling Islamic
State-branded goods... with everything from sovereign rings and baseball caps to school
bags and baby clothes. Retrieved on 16 July 2021 from https://www.dailymail.co.uk/news/
article-2982436/The-shocking-world-ISIS-fashion-children-wearing-ISIS-babywear-fightersshowing-cheap-rings-dodgy-snoods.html.
574
Peter Mantello
Selfie as hipster holy warrior
The jihadist as a hipster selfie has become a familiar trope within jihadist online
networks but also an important cog in a propaganda machine that drives the
cultural mythologies which shape and mold the contemporary jihadist imaginary
(Helg 2014)5. A good example is the case of Islam Yaken, a twentysomething
Egyptian who migrated to Syria to join the Islamic State and became a leading
poster boy. The apex of fame for Yaken came in 2014, when he uploaded to his
Twitter account a self-portrait riding a horse while waving a curved Arabian
scimitar. Besides the AK-47 slung over his shoulder and a satellite-dish in the
background, the only things in the photo that would date this image not in the
Middle Ages were Yaken’s trendy cork-screw hair curls and designer glasses
(Wood 2019). Yet the iconic status of this digital portrait had less to do with his
personification of “jihadi cool” and more with the fusing together of past and
future paradigms (Tibi 2010; Afsaruddin 2014). Similarly to the black flag, the
warrior on horseback is a central and reoccurring motif in jihadi literature and
Islamic eschatology. Horsch (2014) notes that the symbolic importance of the
warrior on horseback originates with the first generation of Muslims during the
golden age of Islam and stirs in Salafi jihadists’ religious feelings over the military
conquests of Muhammad and his followers. Greater significance is lent to the
sword-carrying warrior on horseback in the apocalyptical prophecies found
in the Hadith which claim an army of horse riders led by ‘the Mahdi’ (or ‘the
chosen one’) will fight the final battle against all unbelievers in either the town of
Amaq or Dabiq6. Thus, the resonance of Yaken’s selfie lies in its iterability with a
mythological crusade and the triumphant generation which was first represented
by Muhammad’s army. Yet whereas this first group of Muslims established Islam,
millennials who identify with Yaken’s self-portrait envision themselves as the last
Muslims that will cleanse the world in an apocalyptical battle to end all wars, the
war between good and evil. This vision is reiterated throughout the Islamic State’s
strategic communication (McCants 2015).
As Levin (2014) notes, “the selfie not only embraces the foundational mythologies of a civilization and/or a time, but it has the capacity to alter existing
typologies and originate new ones”. Indeed, Yaken’s famed self-portrait illustrates
a new genus of jihadist subjectivity based on a dialogic capacity to evoke in
the present a nostalgia for a future-past. Massumi calls this mnemonic form of
5
Helg, Martin 2014. Der Hipster als Jihadist, NZZam Sonntag. Retrieved on 16 July 2021
from https://nzzas.nzz.ch/gesellschaft/der-hipster-als-jihadist-ld.146154?reduced=true.
6
Not coincidentally, Amaq is also the name of an Islamic State news agency while Dabiq is
the name of their first glossy, English-language e-magazine.
Fatal portraits: The selfie as agent of radicalization
575
interpellation ‘priming’. He explains, “Priming is memorial. It is the making active
in the present of an inheritance from the past, [...] [Priming] is as much a call to
the future as it is a recall of the past. [...] may be memorial, but it is also futuristic”
(Massumi 2015: 108–109). Applications such as Instagram, TikTok, and SnapChat
can be construed as personal repositories of identity construction, but also agents
of priming, for they open up a temporal feedback circuit between the Islamist’s
present crusade, war time past, and apocalyptical future.
Figure 2. Shifting Identities of Islam Yaken.
Interestingly, Yaken’s fame as an Islamic State’s poster-boy originated not from
a conviction to wage violent jihad but rather a desire to carry over the effort at
becoming an online fitness guru from his previous civilian life (see Fig. 2). In
accordance with the celebrity determinant of cultivating a unique persona, Yaken’s
micro-celebrity status began as a quirky yet relatable online fitness trainer for
Islamic State soldiers. A combination of good looks and earnest demeanour saw
him quickly amass a Twitter following of over 10,000 admirers (Wood 2019). Yet as
his micro-celebrity status grew, the playful persona began to slip away and a much
darker, more violent screen-mediated self emerged (El Naggar, 2015)7. Instead
of holding his familiar set of dumbbells, Yaken started to appear holding the
7
El Naggar, Mona 2015. From a private school in Cairo to the Killing Fields in Syria.
Retrieved on 20 January 2021 from https://www.nytimes.com/2015/02/19/world/middleeast/
from-a-private-school-in-cairo-to-isis-killing-fields-in-syria-video.html.
576
Peter Mantello
decollated heads of Syrian regime soldiers. In line with McLuhan’s reinterpretation
of the Narcissus myth and thoughts on self-amputation and self-amplification,
Yaken had “adapted to his extension of himself and had become a closed system”
(McLuhan 1994: 41). What was left of his original self was now numbed and
standing in service to a more brutal mediatized other – the archetypal warrior.
Selfie as archetypal warrior
In jihadi culture, the sharing of celebrity jihadist selfies is often a new recruit’s
first exposure to politics, let alone the politics of radical Islam (Ouzgane
2006). For sympathetic audiences living in poorer countries where ruling
authorities have faltered in addressing socio-political grievances such as
poverty, corruption, or unfair elections, much of the allure of the jihadist selfie
lies in its hyper-masculinity8 and the possibility of radical change through
performative violence (Aslam 2012). It is in impoverished and often wartorn areas of the world where the ocular-centric allure of the jihadist microcelebrity transcends regional language differences, tribal cultural barriers,
centre-versus-periphery divides as well as huge educational gaps within the
population. As Aolián (2016: 190) observes,
[…] men who cannot meet traditional expectations of masculinity – such as
bread winner, respect and honor, wealth, access to sexual partners of choice – may
precisely find that radical or extremist political mobilization offers a compelling
substitute for regular masculinity authentication.
Importantly, jihadi micro-celebritism symbolizes to the disenfranchised or
marginalized male empowerment and heavenly rewards of jihad, so that
displays performative violence become a vehicle to attain higher levels social
capital. Arguably, this is why many male jihadists’ social media accounts are
marked by greater levels of hyper-masculinity identity. It is not by coincidence
that Islamic State and al Qaeda propaganda relies on gendered stereotypes,
especially in context of the status and rewards afforded to men on their
performance of violence; this can be witnessed in selfies which commemorate
their battlefield killing (Aolián 2016).
Besides condensing into a single visual frame an age-old discursive system of
power relations which in war seeks to dehumanize the enemy and legitimize their
8
As regards hypermasculinity, I apply Angela Harris’ (2000) definition, “the exaggeration of
male stereotypical behaviour, emphasizing physical strength, aggression and sexuality”.
Fatal portraits: The selfie as agent of radicalization
577
murder (Jakob 2017; Carruthers 2011; Larson 2014), the selfie as a war trophy
is now a standard rite of passage for Islamic fighters seeking to raise their social
prestige. While this abject form of hyper-masculine memorialization is neither
a recent phenomenon nor unique to Islamic extremists (Larson 2014; Jakob
2017), its highly personal yet collective nature is novel for it mirrors the same
screen-mediated processes through which Papacharissi’s ‘networked self ’ comes
into being. Unlike its analogue predecessor that was shared between members of
personal and intimate communities (friends, family and comrades-in-arms) and
governed by pre-established codes of conduct which ensured its legitimacy in place
of familiarity (Carruthers 2011), the war trophy selfie, as a networked artifact,
consciously anticipates affiliation through more peripatetic, anonymous and fluid
systems developed within online networks.
Purposed as a tool for both personal and collective identity building, the
selfie as a war trophy oscillates between an object of “commemorative violence”
and agent provocateur of “productive violence”. Digital self-portraits by necroinfluencers such as ex-German rapper Dennis Cuspert aka Deso Dogg, sitting
comfortably with the severed head of an enemy on his lap brings “people together,
galvanizing them in intensely emotional situations, rather than – or as well as
repelling them” (Larson 2014: 33). On the other hand, they emulate the cultural
logic, vernacular and practices of extremist micro-celebrity culture, because “for all
its cruelty, it produces an extraordinarily potent artefact that compels our attention
whether we like it or not” (Larson 2014: 34). In order for the jihadist to become
“insta-worthy” and attain the degree of communal acquiescence required for
“insta-fame”, he or she must abide by aesthetic practices and protocols that drive
the rituals in impression management and personal branding of microcelebrity
culture mentioned earlier.
Another good illustration of a jihadist whose penchant for micro-celebritism
allowed him to garner status as a celebrity foreign fighter is the ex-British rapper
Abdel Barry aka L Jinn. Like other disillusioned and culturally fragmented
second-generation Muslims living in the West, Barry’s path to radicalization
was marked by an earlier life of drug use, run-ins with the law, and affinity for
hip hop music’s more aggressive cousin, gangsta rap (Chulov, Sabbagh 2020)9.
Similar to Deso Dogg, Barry saw strong affinities between gangsta rap and
militant Islam – namely, a subscription to violence, armed insurrec tion and
martyrdom as a means of rewriting perceived social and political injustices
9
Chulov, Martin; Sabbagh, Dan 2020. Spanish police arrest former British rapper
turned Isis extremist. Retrieved on 21 April 21 2020 from https://www.theguardian.com/
world/2020/apr/21/spanish-police-arrest-former-british-isis-extremist.
578
Peter Mantello
(Aidi 2014). His talent as a lyricist and singer saw him starring in several
music videos released online with one of the songs receiving frequent airplay
on BBC radio. Similar to Yaken, as Barry’s micro-celebrity status grew, so did
his inclination toward more militant versions of Islam – at one point following
the teachings of radical preacher Anjem Choudary.
In 2013, the self-styled poet warrior put his confrontational lyrics into
action and moved too Syria to take up arms, first with the Syrian Free Army
and then with the Islamic State. Although submitting willingly to a dramatic
lifestyle change, Barry did not relinquish his old media habits and microcelebrity desires. It was not long after arriving in Syria that Barry’s regular
posts on Twitter of his battlefield experiences caught the eye of the Western
media and alerted UK authorities who wrongly suspected him of being the
infamous Islamic executioner Jihadi John. Yet Barry’s high point of infamy
came when he posted to his Twitter account a self-portrait smiling with his
arm around the severed head of a Syrian government soldier (see Fig. 3). Apart
from the contrast between extreme savagery and relaxed intimacy so typical in this
genre of battlefield photography, what makes this particular image so frightening is
Barry’s accompanying title, “Chillin with my other homie, or whats left of him”, as
it personifies the cultural familiarity of LOL culture and hip-hop slang so typically
associated with Western hip-hop celebrities.
Figure 3. Abdel Barry.
Fatal portraits: The selfie as agent of radicalization
579
Critically, the selfie as a war trophy puts into sharp relief a merging ontology of
violence that affirms the identity of the warrior while destroying the figural
unity of (an)other. Italian philosopher Adriana Cavarero (2009) argues that
albeit such forms of ruthless violence are normalized and (de)legitimated in the
course of war, a linguistic vocabulary to describe or interrogate this reality is sadly
lacking. Instead, she offers the neologism ‘horrorism’ to describe a form of political
violence that fetishizes and revels in aesthetic destruction of the human body. In
traditional political discourse, the philosopher explains, the words ‘terror’ and
‘horror’ are mistakenly used interchangeably for this kind of violence. However,
Cavarero points out that terror signifies a type of fear that leads to instinctual
body mechanisms of ‘panic and flight’, while the term horror translates to ‘shock
and paralysis’ of the body, affected primarily through the gaze. She argues that
the decollated head of an adversary is a symbol that the warrior is “not content
with merely killing because that would be too little” (Cavarero 2009: 8). For the
horrified liberal viewer, it becomes “an offense to the ontological singularity,
dignity and uniqueness of being that posits the foundational basis of Cartesian
subjectivity of (hu)mankind” (Cavarero 2009: 20).
The war trophy selfie turns the ontological presets of Der Derian’s Virtuous War
(2002) upside down – instead, reinstating the corporeal fragility of human life,
the anguish and fear of those marked for execution, the brutality and savagery of
those who take life, and the raw grotesqueness of human bodies once defiled and
disfigured. At the same time, the external validation of the self through the erasure
of an enemy’s ontological singularity and uniqueness articulates the processes of
othering that underscore age-old relations of dominance and subordination in
war. Whereas the non-violent hipster selfie is a celebration of jihad as a productive
lifestyle choice, the selfie as a war trophy functions as a violent symbol of the
archetypal warrior, a Manichean self-affirmation grounded in the annihilation of
another’s identity.
For the digital intimate publics of jihad, the sharing and re-dissemination of
selfies allows community members to negotiate imagined and physical contours
of sociability in their respective communities. At the same time, the sharing of the
war trophy selfie makes publicly visible an individual’s adulation of the jihadist as
war hero. Yet any social prestige arising from this kind of abject memorialization
inevitably doubles as a powerful signifier of political instauration. In accordance
with the hierarchy system of Islamic extremism, the pinnacle of social and cultural
capital goes to the jihadist who engages in martyrdom.
580
Peter Mantello
Selfie as a martyr
Although commemoration of the fallen hero has traditionally been manifested
through visual imagery (wood-block prints to paintings), aural practices (prayers
and hymns) and social rituals (events and parades) (Leone 2015), the notion of
someone aspiring to be a martyr and then engineering their own hagiography a
priori is, in fact, a purely modern phenomenon closely tied to the selfie and microcelebritism. Originally purposed to promote and embellish hyper-individuated
lifestyles, applications, such as Instagram, TikTok, SnapChat and Tumblr, ensure
that the contemporary Islamic extremist’s metaphysical death wish is given visual
form and material substance. By following the norms and conventions of selfcapture culture, the contemporary martyr exercises control over their physical
body, freezing their corporeality within the flow of time. At the same time, the
post-production capability of self-capture applications grant the martyr “instant
nostalgia” to re-imagine themselves as if they were already dead (Jurgenson 2014).
Although the word ‘martyr’ has become synonymous with the protocols of jihad
until the crucifixion of Christ, the original meaning had nothing to do with dying
for a cause. Rather, the word ‘martyr’ was first used in a legal capacity to refer to a
person who bears witness (Bowerstock 1995), typically an onlooker or spectator
of an event or incident. The term later applied to anyone who suffered adversity
for their religious belief and then, finally, became limited to those persons who
were condemned to death for their convictions. Today, the word ‘martyr’ is most
commonly associated with people who kill and are killed for a religious or political
cause (Cook 2007). Curiously, however, the contemporary martyrs’ penchant for
snapping and sharing selfies of themselves and their victims’ violent demise circles
back to the term’s ocular-centric etymology, in other words: the visual act and
discursive practices of witnessing.
Early martyrdom videos of the 1980s and 1990s married the personal and
political. Convention dictated that a martyr would sit in front of a static camera
and recount his or her personal justifications and purposes for their sacrifice and
killing (Pantucci 2015). The video would then serve as a tool of indoctrination
for future recruits but also as a testament to the fallen hero’s achievements as they
benefitted the community. Critically, however, the martyr selfie aligns more closely
to the violent self-experiential and liminal part of martyrdom. There are two major
reasons for this development. First, the modern hagiography is no longer complete
without images of the martyr’s victims. Second, in its real-time dissemination, the
selfie reconfigures traditional notions of proximity and distance, making audiences
and community members surrogates to the celebratory event. In this moment,
the martyr’s embodiment and personal memory is externalized and replaced with
Fatal portraits: The selfie as agent of radicalization
581
mediated experience. Through this mediated perspective, the viewer is not only
witness to the martyr’s final demise but, increasingly, real time slaughter of the
vulnerable and helpless. For example, Amedy Coulibaly made several videos to
justify his future actions before filming his fatal attack on a Hyper-Marche in Paris.
Both Mohameed Mera and Mehdi Nemmouche strapped Go-Pro cameras to their
chests before embarking on fatal killing sprees against followers of the Jewish faith.
Not long afterwards, Larossi Abballa broadcasted his murder of a Magnanville
police couple on Facebook Live from a laptop at their home while sitting beside
their twelve-year-old child. Months later, Omar Mateen tweeted, took selfies and
constantly performed vanity searches on Twitter for news updates of his attack as
he gunned down dozens of innocent partygoers at a gay disco in Orlando, Florida.
Or, more recently, far-right extremists such as Norwegian Andres Breivik and New
Zealander Brenton Tarrant have made their video-recording an integral part of
their martyrdom. By shifting perspective away from their own corporeal body to
the destruction of ‘killable bodies’ (Agamben 1998), the martyr transfers their final
memories onto to the viewer10. This shift from corporeal self-reflection to liminal
consciousness completes the nascent construction of a purely modern hagiography,
accessible to the umma and adversary without spatial or temporal restraint. By assuming
the point of view of the martyr, the spectator (re)witnesses the “truth” of sacrifice.
The memoralization of digital hagiography unfolds through its communal imbrication
across porous boundaries between admirers and adversaries. Clearly, then, the importance
of the martyr selfie goes far beyond the formation and posterity of the jihadist’s
atomized self. Rather, it performs a wider influential role in galvanizing collective
identity, socializing community members and mobilizing participatory action of
members in their communities.
Instant nostalgia
Prior to Web 2.0, an extremist could not hope to gain recognition until after his
martyrdom. However, the post-production capability of self-capture applications
allows the aspiring martyr to change the colour and light values of their photos
based on analogue film stocks of previous eras. Nathaniel Jurgensen (2014) calls
this feature ‘instant nostalgia’. For the prospective martyrs, the selfie functions as
a mnemonic agent, bringing into the present a future that has yet to occur and
helping them to imagine themselves as if they were already deceased.
10
For these reasons, I consider the cell-phone recording an externalized form of selfie
that works (and in many cases is inspired) by the same semiotic principles and interpellative
practices as the first-person shooter videogame.
582
Peter Mantello
Figure 4. Early selfie by Dzhokhar Tsarnaev.
One of the most (in)famous examples of the martyr selfie and filter function
concerns Dzhokhar Tsarnaev, the surviving sibling of the Boston bombers. While
still on death row, the mediatization of Tsarnaev’s infamy and ubiquity of his image
circulating on the internet have already guaranteed him a prominent place in halls
of Islamic extremist martyrdom. Of the many digital self-portraits created by
Tsarnaev prior to his deadly attack, there are two which best illustrate the semiotics
of nostalgia elicited by the faux-vintage filter. The first was used in a LA Times
article on 27 June 2013 (Serrano 2013)11 in which Tsarnaev embellished his selfie
using a black-and-white filter effect (see Fig. 4). Besides generating an emotional
desire to connect the present with an era long past, for the prospective martyr the
iconicity of the black-and-white filter lies in its subliminal associations tied to
photographic representations of archetypal celebrity figures in the 20th century.
11
Serrano, Richard 2013. Boston bombing indictment: Dzhokhar Tsarnaev inspired by Al
Qaeda. Retrieved on 18 December 2020 from https://www.latimes.com/nation/nationnow/lana-nn-boston-marathon-bombing-suspect-indictment-20130627-story.html.
Fatal portraits: The selfie as agent of radicalization
583
Figure 5. Tsarnaev on the cover of Rolling Stone magazine.
The second, and perhaps best-known, selfie of the Boston Bomber is the one which
first appeared on front page of the New York Times on 5 May 2013, but only
aroused controversy when editors of the Rolling Stone magazine decided to put
the same self-portrait on the cover of their 17 July 2013 issue (See Fig. 5). Critics
world-wide condemned the iconic music entertainment magazine’s decision for
glamourizing terrorism and encouraging other terrorists, so its circulation was
banned from many commercial outlets (Gabbatt 2013)12. For this photo, Tsarnaev
chose a faux-vintage filter emulating the subdued yet warm colour spectrum of
Kodak’s reversal films of the 1970s. His choice of using this trendy retro-look filter
speaks of the bomber’s reputation as an ordinary and normal teenager. Indeed, the
contradictions of Tsarnaev’s character, so emblematic of the jihadist imaginary, are
12
Gabbatt, Adam 2013. Rolling Stone’s controversial Dzhokhar Tsarnaev cover ignites heated
debate. Retrieved on 20 January 2021 from https://www.theguardian.com/media/2013/jul/17/
rolling-stone-dzhokhar-tsarnaev-cover.
584
Peter Mantello
further illustrated in his Facebook profile where he wrote ‘Islam’ as his world view,
yet stated ‘career and money’ as his personal goal (Reitman 2013)13.
Conclusion
Clearly, then, the battlefield selfie is not simply a narcissistic tool for modern-day
jihadists to gain greater social capital. Rather, it plays a much wider ontological
role in galvanizing collective identity, socializing community members, and
mobilizing the participatory actions of members in their communities. This article
has mapped the various feedback loops and interplay between the highly
material world of self-capture culture and the jihadist imaginary. As a sociotechnical artifact, the digital self-portrait enables the contemporary jihadist
to traverse the boundaries between two seemingly opposing belief systems.
I have argued that the selfie can also be construed as an instructive marker
of the various stages of identity realignment that occurs in processes of a
jihadist’s radicalization. Whereas the hipster selfie promulgates jihad and the
battlefield as a productive lifestyle choice, the selfie as a war trophy reaffirms
age-old power relations of domination and subordination in war, while the
martyr selfie collapses the personal and political in a violent and unique form
of hagiography that preserves and externalizes the jihadist’s violent final
moments. This paper extends and nuances existing literature on the role of Web
2.0 in contemporary political violence as well as contributes to a nascent body of
literature on the phenomenon of self-capture culture and the battlefield.
Acknowledgements: This study is part of the project “Emotional AI in Cities: Cross Cultural
Lessons from UK and Japan on Designing for an Ethical Life” funded by JST-UKRI Joint
Call on Artificial Intelligence and Society (2019). (Grant No. JPMJRX19H6). Additionally,
special thanks go to the reviewers of this article and my research assistant Tung Manh Ho
for their valuable insights and comments.
13
Reitman, Janet 2013. Jahar’s world. Retrieved on 16 July 2021 from https://www.rollingstone.
com/culture/culture-news/jahars-world-83856/.
Fatal portraits: The selfie as agent of radicalization
585
References
Abidin, Crystal; Brockington, Dan; Goodman, Michael K.; Mostafanezhad, Mary; Richey,
Lisa A. 2020. The tropes of celebrity environmentalism. Annual Review of Environment
and Resources 45(1): 387–410. https://doi.org/10.1146/annurev-environ-012320-081703
Afsaruddin, Asma 2014. Martyrdom in Islamic thought and praxis. In: Dominic,
James; Houen, Alex (eds.), Martyrdom and Terrorism: Pre-Modern to Contemporary
Perspectives. Oxford: Oxford University Press, 40–58. https://doi.org/10.1093/acprof:
oso/9780199959853.003.0003
Agamben, Giorgio 1998. Homo Sacer: Sovereign Power and Bare Life. Palo Alto: Stanford
University Press.
Aidi, Hisham 2014. Rebel Music: Race, Empire, and the New Muslim Youth Culture. New
York: Vintage Books.
Andrews, William 2016. Dissenting Japan: A History of Japanese Radicalism and Counterculture from 1945 to Fukushima. Oxford: Oxford University Press.
Appadurai, Arjun 2006. Fear of Small Numbers. Durham: Duke University Press. https://
doi.org/10.2307/j.ctv11smfkm
Aolián, Fionnuala N. 2016. The complexity and challenges of addressing the conditions
conducive to terrorism. In: Nowak, Manfred; Charbord, Anne (eds.), Using Human
Rights to Combat Terror. Cheltenham: Edward Elgar Publishing Limited, 166–194.
Aslam, Maleeha 2012. Gender-Based Explosions: The Nexus Between Muslim Masculinities,
Jihadist Islamism and Terrorism. Tokyo: United Nations University Press.
Auerbach, Yehudit; Ben‐Yehuda, Hemda 1987. Attitudes towards an existence conflict:
Begin and Dayan on the Palestinian issue. International Interactions 13(4): 323–351.
https://doi.org/10.1080/03050628708434682
Bakhtin, Mikhail 1973. Mikhail Bakhtin: The Duvakin Interviews. Lewisburg: Bucknell
University Press.
Bandopadhyaya, Suvojit 2019. Branding the Islamic State of Iraq and Syria. Global Media
and Communication 15(3): 285–301. https://doi.org/10.1177/1742766519874380
Berger, J. M.; Perez, Heather 2016. The Islamic State’s diminishing returns on Twitter: How
suspensions are limiting of the social networks of English-speaking ISIS supporters.
Program on Extremism. Washington: George Washington University.
Boorstin, Daniel J. 1983. The Image: The Guide to Pseudo-Events in America. New York:
Routledge.
Bowerstock, Warren G. 1995. Martyrdom and Rome. Cambridge: Cambridge University
Press.
Carruthers, Susan 2011. Media at War. (2nd ed.) New Jersey: Globe Press. https://doi.
org/10.1007/978-0-230-34535-5
Cavarero, Adriana 2009. Horrorism: Naming of Political Violence. New York: Columbia
University Press.
Cook, David 2007. Martyrdom in Islam. (Themes in Islamic History 4.) Cambridge: Cambridge University Press.
Der Derian, James 2001. Virtuous War: Mapping the Military-Industrial Media-Entertainment Network. Milton Park: Routledge.
586
Peter Mantello
Djafarova, Elmira; Trofimenko, Oxana 2019. ‘Instafamous’ – credibility and self-presentation of micro-celebrities on social media. Information, Communication & Society 22(10):
1432–1446. https://doi.org/10.1080/1369118X.2018.1438491
Dobson, Amy Shields; Robards, Brady; Carah, Nicholas 2018. Digital intimate publics and
social media: Towards theorising public lives on private platforms. In: Dobson, Amy
Shields; Robards, Brady; Carah, Nicholas (eds.), Digital Intimate Publics and Social
Media. London: Palgrave Macmillan, 3–27. https://doi.org/10.1007/978-3-319-976075_1
Driessens, Olivier 2013. The celebritization of society and culture: Understanding the
structural dynamics of celebrity culture. International Journal of Cultural Studies 16(6):
641–657. https://doi.org/10.1177/1367877912459140
Droogan, Julian; Peattie, Shane 2018. Reading jihad: Mapping the shifting themes of Inspire
magazine. Terrorism and Political Violence 30(4): 684–717. https://doi.org/10.1080/09
546553.2016.1211527
Durkheim, Emile 2002[1897]. La Educación Moral [Moral education]. Madrid: Ediciones
Morata.
Fisher, Ali 2015. Swarmcast: How jihadist networks maintain a persistent online presence.
Perspectives on Terrorism 9(3): 3–20.
Galloway, Alex; Thacker, Eugene 2007. The Exploit: A Theory of Networks. Minneapolis:
University of Minnesota Press.
Harris, Angela P. 1999. Gender, violence, race, and criminal justice. Stanford Law Review
52(77): 777–807. https://doi.org/10.2307/1229430
Hoffman, Bruce 2017. Inside Terrorism. New York: Columbia University Press. https://doi.
org/10.7312/hoff17476
Horsch, Silivia 2014. Making salvation visible: Rhetorical and visual representations
of martyrs in Salafī jihadist media. In: Dehghani, Sasha; Horsch, Silvia (eds.),
Martyrdom in the Modern Middle East. Wurzburg: Ergon-Verlag, 141–166. https://doi.
org/10.5771/9783956505430-141
Inglis, Fred 2010. A Short History of Celebrity. Princeton: Princeton University Press.
https://doi.org/10.1515/9781400834396
Ingram, Haroro J.; Whiteside, Craig; Winter, Charlie 2020. The ISIS Reader: Milestone Texts
of the Islamic State Movement. Oxford: Oxford University Press. https://doi.org/10.1093/
oso/9780197501436.001.0001
Jakob, Brooke J. 2017. Beyond Abu Ghraib: War trophy photography and commemorative violence. Media, War & Conflict 10(1): 87–104. https://doi.org/10.1177/
1750635216636136
Jurgenson, Nathan 2019. The Social Photo: On Photography and Social Media. New York:
Verso.
Klausen, Jytte 2015. Tweeting the Jihad: Social media networks of Western foreign fighters
in Syria and Iraq. Studies in Conflict & Terrorism 38(1): 1–22. https://doi.org/10.1080
/1057610X.2014.974948
Larabee, Ann 2000. Decade of Disaster. Urbana: University of Illinois Press.
Larson, Francis 2014. Severed: A History of Heads Lost and Heads Found. New York: W. W.
Norton & Company.
Fatal portraits: The selfie as agent of radicalization
587
Leone, Massimo 2015. Propaganda mala fide: Towards a comparative semiotics of violent
religious persuasion. Semiotica 207: 631–655. https://doi.org/10.1515/sem-2015-0057
Nacos, Brigitte. 2016. Mass-Mediated Terrorism: Mainstream and Digital Media in Terrorism
and Counterterrorism. Lanham: Rowman & Littlefield. https://doi.org/10.4324/
9781315641270
Manovich, Lev 2019. The aesthetic society: Instagram as a life form. In: Mörtenboeck,
Peter; Mooshammer, Helge (eds.), Data Publics. Milton Park: Routledge.
Marshall, David P. 2015. The genealogy of celebrity. In: Marshall, David P.; Redmond,
Sean (eds.), Companion to Celebrity. Hoboken: John Wiley & Sons, 15–19. https://doi.
org/10.1002/9781118475089.part1
Marwick, Alice E. 2015. Status Update: Celebrity, Publicity, and Branding in the Social Media
Age. New Haven: Yale University Press.
Massumi, Brian 2015. Ontopower: War, Powers, and the State of Perception. Durham: Duke
University Press. https://doi.org/10.1515/9780822375197
McCants, William 2015. The ISIS Apocalypse: The History, Strategy, and Doomsday Vision
of the Islamic State. New York: Palgrave Macmillan.
McLuhan, Marshall 1994. Understanding Media: The Extensions of Man. Cambridge: The
MIT Press.
Michel, Lou; Herbeck, Dan 2001. American Terrorist: Timothy McVeigh and the Oklahoma
City Bombing. New York: Regan Books.
Morningstar, Cory 2019. The Manufacturing of Greta Thunberg. Norderstedt: Books on
Demand.
Nissen, Thomas E. 2015. #TheWeaponizationOfSocialMedia:@Characteristics of Contemporary Conflicts. Copenhagen: Royal Danish Defence College.
Ouzgane, Lahoucine 2006. Islamic Masculinities. London: Zed Books. https://doi.org/
10.5040/9781350220881
Pantucci, Raffaello 2015. "We Love Death As You Love Life": Britain’s Suburban Terrorists.
Oxford: Oxford University Press.
Papacharissi, Zizi (ed.) 2010. A Networked Self: Identity, Community, and Culture on Social
Network Sites. Milton Park: Routledge.
Porter, Roy 2004. Flesh in the Age of Reason. New York: W. W. Norton & Company.
Prucha, Nico 2016. IS and the jihadist information highway – Projecting influence and
religious identity via telegram. Perspectives on Terrorism 10(6): 48–58.
Schickel, Richard 2000. Intimate Strangers: The Culture of Celebrity in America. Chicago:
Ivan R. Dee.
Senft, Theresa M. 2008. Camgirls: Celebrity and Community in the Age of Social Networks.
(Digital Formations 4.) Bern: Peter Lang Publishing.
Shehabat, Ahmad; Mitew, Teodor 2017. Distributed swarming and stigmergic effects
on ISIS Networks: OODA Loop Model. Journal of Media and Information Warfare
10(2017): 79–109.
Simons, Greg 2018. Brand ISIS: Interactions of the tangible and intangible environments.
Journal of Political Marketing 17(4): 322–353. https://doi.org/10.1080/15377857.2018.
1501928
Tibi, Bassam 2012. Islam and Islamism. New Haven: Yale University Press.
588
Peter Mantello
Torres-Soriano, Manuel R. 2010. The road to media jihad: The propaganda actions of Al
Qaeda in the Islamic Maghreb. Terrorism and Political Violence 23(1): 72–88. https://
doi.org/10.1080/09546553.2010.512839
Thomas, Samuel 2013. Yours in revolution: Retrofitting Carlos the Jackal. Culture
Unbound: Journal of Current Cultural Research 5(3): 451–478. https://doi.org/10.3384/
cu.2000.1525.135451
Turner, Graeme 2013. Understanding Celebrity. Newcastle: Sage. https://doi.org/10.4135/
9781473957855
Veilleux-Lepage, Yannick 2016. Paradigmatic shifts in Jihadism in Cyberspace: The
emerging role of unaffiliated sympathizers in Islamic State’s social media strategy.
Journal of Terrorism Research 7(1): 36–51. https://doi.org/10.15664/jtr.1183
Weimann, Gabriel; Winn, Conrad 1994. The Theater of Terror: Mass Media and International Terrorism. London: Longman Publishing.
Winter, Charlie 2018. Apocalypse, later: A longitudinal study of the Islamic State brand.
Critical Studies in Media Communication 35(1): 103–121. https://doi.org/10.1080/152
95036.2017.1393094
Wood, Graeme 2019. The Way of the Strangers: Encounters with the Islamic State. New York:
Random House Trade Paperbacks.
Zelin, Aaron Y. 2014. The war between ISIS and al-Qaeda for supremacy of the global
jihadist movement. The Washington Institute for Near East Policy 20(1): 1–11.
Zuboff, Shosana 2019. The Age of Surveillance Capitalism: The Fight for a Human Future
at the New Frontier of Power. London: Profile Books.
Фатальные портреты: селфи как инструмент радикализации
Цифровой автопортрет, а, точнее, селфи на поле битвы, является популярным средством построения идентичности современного джихадиста. Подобно портретам,
которые создают не склонные к насилию участники культуры селфи, автопортрет
джихадиста представляет собой альтернативу картезианской идее единой и неделимой личности. Скорее это продукт социальных отношений и перформативных действий, созданный в диалоге с другими в контексте определенных социокультурных
рамок и ожиданий. Однако, в отличие от ненасильственного варианта, ожидания
от этого диалога строятся вокруг более широкой политической повестки, которая
активно стремится переписать коллективную память об имперских исламских завоеваниях и привлекает религию для морального оправдания насилия. Важно, что
селфи на поле боя позволяет джихадисту легко пересекать границы между двумя,
казалось бы, противоположными системами верований. Несмотря на изобилие исследований, посвященных культуре селфи, платформам по обмену изображениями
и микро-селебрити, до сих пор не изучено их всепроникающее влияние, касающееся
практик на поле боя. Истоки этого явления лежат в личных историях ключевых исламских экстремистов, которые благодаря своему поведению на поле боля оказались
одновременно обожествлены и демонизированы. При этом сегодня такие индивиды
могут снискать славу в интернете, участвуя в культуре селфи – публикуя автопортреты, видео или посты в блоге. Другими словами, никогда публичная коммуникация
солдата еще не была такой личной и при этом общественной.
Fatal portraits: The selfie as agent of radicalization
589
Saatuslikud portreed: selvefoto radikaliseerumise toimeviijana
Nüüdisaegsele džihadistile on digitaalne autoportree või täpsemalt lahinguvälja-selfi populaarne identiteediloome tööriist. Sarnaselt selfidega, mida teevad enesejäädvustamiskultuuri
vägivallatud praktiseerijad, esindab džihadistlik selvefoto alternatiivi kartesiaanlikule ühtse
ja jagamatu Ise formuleeringule. See on pigem ühiskondlike suhete ja performatiivsete
tegude tulemus, mida moodustatakse dialoogis teistega väga spetsiifiliste ühiskondlikkultuuriliste raamistuste ja ootuste kaudu. Ent erinevalt selle vägivallatust teisikust keskenduvad antud dialoogi ootused avarama poliitilise tegevuskava ümber, mis püüab aktiivselt
ümber kujundada kollektiivseid mälestusi islami imperialistlikest vallutustest ning kaasata
religiooni kui võimalust moraalse korra kehtestamiseks oma vägivalla üle. Oluline on see,
et lahinguväljaselfi võimaldab džihadistil hõlpsasti liikuda üle kahe näiliselt vastandliku
ususüsteemi vahelise piiri. Kuigi enesejäädvustamiskultuuri, pildijagamiskeskkondade ja
mikrokuulsusnähtuse kohta on olemas rikkalikult uurimusi, on nende kõikjaletungivat
mõju ja praktiseerimist lahinguväljal vähe uuritud. Nähtus tuleneb võtmetähtsusega islami
äärmuslaste isikulugudest, keda nende lahingus toimunud mikrosuunamudimistegevuse
tulemusena on nii ülistatud kui ka demoniseeritud. Tänapäeval aga võivad samasugused
isikud enesejäädvustamiskultuuris osalejatena saavutada kuulsuse internetis, postitades
selfisid ja videoid või blogides. Teisisõnu, sõjamehe avalik kommunikatsioon pole kunagi
varem olnud nii isiklik ja samas kollektiivne.
590
Elsa Soro
Sign Systems Studies 49(3/4), 2021, 590–607
Urban-human faces and the semiotic right to
the city: From the USSR propaganda machinery to
the participatory city
Elsa Soro1
Abstract. Now that the usage and meaning of urban spaces have been dramatically
challenged by the global pandemic, several debates and reflections are going on
around the manner in which cities – both as concerns the public and the private
spaces – have been designed. The article observes how “urban-human face”
representations have served different models of urbanity across times and cultures.
Using a framework deriving from semiotics of culture, according to which the
city represents a model of the world, the article attempts to interpret how portraits
of faces have been modelling the city through different urban faciality mechanisms.
The focus is on a sample of what we call ‘urban-human faces’, ranging from Soviet
propaganda posters to the digital #selfiecity project. The expression refers to series
of representations that bring together the city and the face. It can be argued that
both the city and the face, produced at a specific historical and cultural moment,
with their figurative and plastic elements, deploy the struggle for the city ownership
and authorship.
Nonetheless, the commensurability of the city and the face can be just based
on the fact that both semiotic configurations represent an excess with the help
of cartographic reproduction (the city) and the portrait (the face), respectively. A
city can be represented by a face to the extent to which it is also multifaceted as a
polylogue. On the basis of such instable commensurability, the article will ultimately
attempt to bring together the semiotics of the face and the semiotics of the city.
Keywords: faciality; urban semiotics; mediascape; selfies; selfiecities
1. Introduction: Cities and faces under (re)construction
Since the beginning of the global pandemic, the images of the cities circulating in
the media all over the world have been displaying dystopic scenarios made of vivid
contrasts between empty spaces, for instance the traditional tourist and leisure
1
Department of Philosophy and Education Science, University of Turin, Via Verdi 8 – 10124
Torino, Italy/UOC (Open University of Catalonia), Spain; e-mail: elsa.soro@unito.it.
https://doi.org/10.12697/SSS.2021.49.3-4.17
Urban-human faces and the semiotic right to the city
591
spaces deserted by visitors, and massive agglomerations of human and non-human
actors, namely the queues at the supermarkets or the gathering of patients in the
crowded hospital wards, together with the correspondent artefacts.
During the lockdown imposed by the different governments all over the world, the
public and commercial spaces, deprived of their dwellers and consumers, have been
invoked in public debate and media descriptions as “ghostly”, or “surreal”, and, thus,
such processes have emphasized the intrinsic social character of the urban texture.
While the elements of the city directly related with the urban lifestyle (i.e.
interaction, mobility, crowd activities, etc.) have been banned, a parallel process of
covering and concealment involves the human face these days: the sanitary masks,
recommended or imposed by the authorities cover the mouth and the nose, while
the face shields cover the face from the top of the head to below the chin.
On the basis of this, it can be argued that the current debate about the design
and planning of cities in an hypothetical postpandemic scenario could be
intermingled with a debate on the new aesthetic of the face: the visibility regimes
of both the city and the human face have been brought to question: What must
remain visible in a face in order to recognize a person? Which parts of the city and
which aspects of urban life could still remain when the risk of an infectious disease
threatens social life?
However, the overlapping between the discourses on the city and those on
the human face is not a novelty. The expression ‘the face of the city’, used both in
academic and public debates, refers to the appearance of the city and evokes the
fact that for a city, as a face, to be memorable it must leave a certain impression
on the visitors.
Sayings such as “the cities change their faces through times” or “cities with a
human face” attribute to the city the features of a visage, something that changes
over times, but, at the same time, refers to an identity that lasts and can be
recognized despite those changes.
There are studies that have already been delving into the ontological affinity
between the city and the face by analysing the extent to which a comparison and
an analogy can be found between the two notions (Deleuze, Guattari 1980; Black
2011; Mubi Brighenti 2019). Taking into account the existing research, this piece
aims at contributing to the debate on the relationship between the city and the
face from a different angle. Using a framework deriving from semiotics of culture,
according to which the city represents a model of the world, the study attempts
to interpret how the representations of human face and the conditions of their
production reveal different cities’ ideologies across times and cultures.
It is worth noting that the human face is a recurrent iconographic figure in
urban landscape: for instance, in the global metropolis street artists use the urban
592
Elsa Soro
walls as an informal and ephemeral canvas for portraits of the human face 2, while
digital billboards in the city centres have been advertising products through the
model human faces to catch the consumers’ attention.
With this in mind, the piece will focus on a sample of “human-urban face”
portraits, that is, a series of representations that bring together the city and the
face in a meaningful composition. Arguably, each of them, produced in a specific
historical moment, deploys the struggle for the city ownership and authorship with
its figurative and plastic elements.
The study proceeds from the assumption that the city is a contested object,
whose significance is produced by different forces and tensions that struggle for the
city ownership as shown by a vast literature3 in urban studies, geography and urban
sociology that has been focusing on political struggles (Davis 1990; Hardt, Negri
2009), economic asymmetries and social justice (Harvey 1973; Schneider, Susser
2003), privatization of urban space (Atkinson, Blandy 2006), residential and spatial
segregation (Massey 1996; Atkinson et al. 2004), social inequality (Skop 2006) and
creation of plural social movements (Castells 1983; Young 1990).
From a semiotic point of view, the work tries to embrace the urban conflict
as an overlapping between different writings and rewritings of the urban text
undertaken by different potential “authors” who aim to be the custodians of the
urban narratives and the urban gaze. Covering a trajectory of a series of “urbanhuman faces”, the text will aim to contribute to bringing together the semiotics of
the face with the semiotics of the city.
The first part of the article will frame the topic within a literary review of the
city as a semiotic object in a permanent tension between plurality and unity.
The trajectory covering “urban-human faces” starts with the representation of
the oversized face of the leader who overlooks the crowd in Soviet propaganda.
In opposition to socialist ideology, the representations of the capitalist global city
deploy the fragmentation of the power scattered on screen refractions. The face
of the “leader” pops up on television screens in the private space of the citizens’
houses. Later on, with the advent and diffusion of the personal computer and other
technological devices, the “real” city is recreated in online virtual worlds and “played”
by face avatars. The trajectory is “finalized” by illustrating the “urban-human face” in
the participatory rhetoric that reclaims a city co-owned by citizenship.
2
The best-known street artist who put the human face at the centre of his artistic production
is JR. The human face is also a recurrent topic in protest street art, a recent example being
the series dedicated to George Floyd, the black man killed by a police officer in the city of
Minneapolis, Minnesota, U.S.A. in 2020.
3
The list of references gathers those most quoted in the field. Nonetheless it cannot be
considered as exhaustive.
Urban-human faces and the semiotic right to the city
593
The series does not aspire to be exhaustive – rather, it aims at tracking figurative
and plastic patterns of the city iconographies through the face, by bringing together
these two unstable semiotic objects.
2. The semiotic city
2.1. The citysphere
The Tartu–Moscow School of Semiotics has extensively interrogated the structure
of the city, understood as a semiotically charged space. Initially,4 the spatial
concepts were only used metaphorically as an empirical description of culture.
Accordingly, the city and its organization was firstly considered by Ivanov (1986:
9) “as a model of universal space”.
Since the coining of the concept of the ‘semiosphere’ (Lotman 1984)5, as an
analogy to and extension of Vernadsky’s (1926) concept of the biosphere, Lotman
further interpreted the city as a territorial space for semiosis. In particular, in
Universe of the Mind, Lotman (1990: 191) clarified how “[i]n the system of a culture’s
symbols, the city has an important place. City symbolism can be divided into two
main areas: the city as symbolic space and the city as symbolic name”. The former
area refers to the relationship between the city and “the earth which surrounds it”
(Lotman 1990: 191). According to Lotman, a city can be either “isomorphous with
the state, and indeed personify it” (for instance Rome as the city and the world), or
“be an antithesis to the surrounding world” (Lotman 1990: 191).
Either way the city, according to the vision of the semiotics of culture, is
animated by elements of tension and struggle. In his description of St.Petersburg,
Lotman (1990: 193) underlined the dimension of “eternal struggle between
elements and culture”, epitomized by the antithesis between water and stone,
present in the myth of the Russian city.
Such tension can be considered as isomorphic to that of the semiosphere one
and the latter’s polylogue form in which different discourses, semiotic conflicts and
relations are woven together, escaping a unique or even a bijective logic of meaning
According to Lotman, the semiosphere is separated from the otherness by
its boundaries, the internal topographical organization being characterized by a
tension between continuity (with respect to the outside) and discontinuity. Such
4
See the Theses of the Tartu-Moscow School (Uspenskij et al. 2003)
Juri Lotman’s “On the semiosphere” was first published in 1984 and was an attempt to
describe the structural features of the semiosphere in broad terms. However, the concept of the
semiosphere was first formulated by Juri Lotman in 1982 inspired by the works of the Russian
geologist Vladimir Vernadsky on the noosphere and the biosphere of Earth.
5
594
Elsa Soro
internal discontinuity of the semiosphere is due, in the first place, to the presence
of certain saliences. Lotman and Uspensky (1977[1973]: 237) exemplify this with
the example of proper names in mythology, where “space is not conceived as a
sign continuum, but as a totality of separate objects bearing proper names. It is
as if space were interrupted by intervals between objects and thus absences”. Such
internal tensions can also be interpreted, at a semiosic level, as the combination
between explosive and gradual processes. Such processes are not just understood
as the alternation of asynchronous phases that mutually replace each other. Instead,
such a tension can give account of a simultaneous process.
The internal topography of the city can be described as highly discontinuous
in different respects, both synchronically and diachronically speaking, both
temporally and spatially speaking, considering both natural (for instance natural
catastrophes) and anthropogenic phenomena (for instance, military sieges,
gentrification processes), redesigning and reorganizing the internal urban forces
at the level of semiosis.
2.2 .The semiotic right to the city
The discontinuity in the city topography can also be attributed to the multiple
transformations of the collective action in the permanent struggle for achieving a
hegemonic position in the city governance.
In the frontispiece of Thomas Hobbes’ Leviathan by Abraham Bosse (1651), the
body of the Sovereign King is constituted by the blurring together of individual
faces of the citizenry, who face the Sovereign to represent his dominant position
over the city.
In contrast, the slogan “right to the city”, proposed in 1968 by Henri Lefebvre,
popularized by radical thinkers such David Harvey, and further recalled by social
movements against globalization, refers to the demand for a different distribution
of the city “ownership”. Harvey (2008: 23) defines the right to the city as the
common right “to make and remake our cities and ourselves”. The making of the
city implies the production of a significance that is always the result of a negotiation
among different actors. An urban semiotics approach makes it possible to interpret
such processes in terms of writing and co-writing of the urban text made by an
overlapping of signification processes.
According to Greimas’ seminal work on topological semiotics, the semiosis process
is triggered by a negotiation of the different meanings and interpretations given to the
city by its inhabitants (Greimas 1976: 138). The city geography is thus composed of a
variety of elements (for instance roads, neighbourhoods, squares, bridges, malls, parks,
etc.) and the “sum” of the manifold readings by its users. City dwellers or tourists can
Urban-human faces and the semiotic right to the city
595
follow the instructions given by the informative structure of the city or contravene it
through tactical actions (De Certeau 1990). The act of crossing the city and walking
through it can be considered as an enunciation act by which the citizens instantiate
their city and confront the dominant ideology of the city.
Nonetheless, such an informative gesture of rebellion gives account of the unity
of the city’s composition behind the plurality of elements that are intertwined
within it. The above-mentioned tensions among the different city actors for
ownership and authorship of the city are thematized by different textualities
which compose and articulate the urban fabric: for instance, a monument can be
considered as the inscription of a (new) memory intertwined with previous ones.
As a city sign, an urban-human face portrait literarily provides the city with eyes
that gaze and in turn are gazed upon. In the following, a trajectory along a series
of “urban-human faces” will be outlined, focusing on the figurative and plastic
elements of the face composition that unveil the underlying tensions concerning
the semiotic right to the city.
3. The urban face of the leader in the Socialist city
The overlapping between the face and the iconographies of power finds one of
the first cultural manifestations in face gigantism. The Mount Rushmore National
Memorial, with the colossal sculpture that features the 60-foot (18 m) faces of the US
Presidents George Washington, Thomas Jefferson, Theodore Roosevelt and Abraham
Lincoln is one of the most remarkable examples of representation of the power (in
this case over a nation) through the face. Yet face gigantism can be considered as
a cultural isotopy across historical ages and cultures: from the face of the Buddha
swallowed by a tree’s gnarled roots, to the face of King Antiochus scattered around
an empty plain in southeast Turkey around 60 BC, to the 180-foot high monument
to Decebalus, the last king of Dacia, on the banks of the Danube River. The character
of colossality entails the capacity of such faces to be seen at a distance, and by
doing so also to overview the natural landscape. The anthropomorphization of the
landscape creates a tension between the human presence and the natural elements
to which the latter is exposed with a domination aspiration.
When it comes to the city landscape, the domination is no longer projected to
natural elements, but to a landscape composed (mainly) by other human figures,
embodied by the notion of the urban crowd. 19th- and 20th-century literature,
philosophy and art have extensively and in different ways problematized the
dialectic between the individual and the crowd and their intersections in the
context of urban life. Figuratively, the alienation and reification in the modern city
596
Elsa Soro
have been represented by the accumulation and the repetition of indistinct faces.
According to Benjamin’s vision of the modern city, expressed in his narratives
in Das Passagen-Werk, the crowd is composed of an unremarkable repetition of
multiple faces (Benjamin 1990: 674). Such repetition of faces is similar to the
accumulation of items and goods stored in Émile Zola novel Bonheur des Dames
where the “gigantesque étalage”, composed of “des pièces de lainage et de draperie,
mérinos, cheviottes, molletons” overwhelm the protagonist Denise who has recently
arrived in Paris from the province (Zola 1883: 761). Similarly, in Benjamin’s
description of the ‘flâneur’, the latter is represented as someone abandoned in the
crowd and thus as a victim of a commodity intoxication.
In opposition to an indistinct agglomeration of miniaturized faces, the face of
the person standing out in the crowd expresses a potential difference, and entails
a sort of primus inter pares effect. Soviet propaganda often represented the act of
standing out by using the technique of photomontage. Both in the poster Stalinists!
Extend the front of the Stakhanovite movement! (1936) and in Varvara Stepanova’s
photomontage The Results of the First Five-Year Plan (1932), the oversized face of
the leader emerges from an undistinguished and anonymous crowd (Fig. 1). The
charismatic leader is responsible not only for the crowd, but also for designing the
new urban landscape, as depicted in the poster Glory to Great Stalin, the Architect
of Communism by N. Petrov and Konstantin Ivanov (1952) on which Moscow,
proclaimed by Stalin as the ideal model of the cosmos, appears as alive with
pedestrians, cars and river traffic in the background.
Figure 1. Stalinists! Extend the front of the Stakhanovite movement! Futerfas, 1936. (https://
digital.library.pitt.edu/) and The Results of the First Five-Year Plan. Varvara Stepanova,
1932. (AP Art History).
Urban-human faces and the semiotic right to the city
597
4. The urban face of the capital in the global city
If in the 20th-century Soviet regime the oversized face of the leader being opposed
to the miniaturization of the masses thematizes a relationship of domination
between the former who gazes over the latter, the images of the capital and the
city display a multiplication and expansion of point of views.
Already in the modern city, as described by Benjamin (1990: 537), the reflective
surface of the windows “brings the open expanse” and thus deceives the eye of
the flâneur. In the era of globalization, the illusion of the space expansion,
accompanied by the multiplication of goods and products, has been detonated by
capillary diffusion of the digital screens that reflect the augmented spaces in the
urban landscape. The geographer David Harvey (1989: 66) describes the urban
fabric, cultivated by the postmodern architecture movement6 as a fragmented
“palimpsest” of different temporalities.
The collage effect is also encompassed by the ‘media city’, a notion with a long
academic trajectory, that has also been defined as ‘information city’ (Castells 1989).
In the words of McQuire (2008: 22), it refers to “a heterogeneous spatial regime,
composed by a new conjunction of media and architecture”. In such a new urban
scenario, the augmentation of space is also caused by the overlapping between
different regimes of speed: the speed of the body, the speed of vehicles, and the
“absolute” light-speed of media and communication technologies (Virilio 1986).
The geographical and symbolic centre of the city, where the charismatic leader
in the 20th-century totalitarian regimes emanated his absolute power over the
crowd as depicted in the propaganda representations, has been delocalized and the
political influence is exercised by occupying the channels of communication. The
link between being omnipresent on the media, or even the ownership of media
groups, and the political influence can ben epitomized by the figure of Italy’s
former Prime Minister Silvio Berlusconi, whose face appearing on his various
channels of television became the symbol of the “media-ization” of power (Fig. 2).
At the same time, Milano 2, the residential area and headquarters of the first Italian
private television channel at the border of Milan, is the epitome of a decentred
utopic urban space.
6
Postmodern architecture became a movement in the late 1970s, in response to the formalism
of the International Style. Postmodernity is considered to be heralded by the reference and the
ornament and the Disneylandization of the contemporary urban environment, epitomized by
the American city of Las Vegas (Venturi, Scott Brown, Izenour 1977).
598
Elsa Soro
Figure 2. Silvio Berlusconi featured for a TV programme.
Even though symbolic urban spaces of consumption such as Times Square in New
York City, Piccadilly Circus in London or Shibuya Scramble Crossing in Tokyo, still
exhibit giant faces gazing upon the citizens (Fig. 3), the media city and its ideology
have entered citizens’ homes.
Figure 3. Piccadilly Circus billboards.
Urban-human faces and the semiotic right to the city
599
Derrida (2002: 79) describes the “desire” to be at home created in the context of
global city as follows:
We are witnessing such a radical expropriation, deterritorialization, delocalization,
dissociation of the political and the local, of the national, of the nation-state and the
local, that the response, or rather the reaction, becomes:
‘I want to be at home, I want finally to be at home, with my own, close to my friends
and family.’ […] The more powerful and violent the technological expropriation, the
delocalization, the more powerful, naturally, the recourse to the at-home, the return
toward home.
The human-urban face, reproduced on the TV screens of each and every citizen,
has been further augmented through its diffusion in the Internet. As another
relational space of consumption, alternative both to public and private space, the
cyberspace as an augmented substitute of the real sites increases the possibilities
of the urban space and its point of views. Projects based on virtual reality such as
the online virtual world Second Life, opened to public in 2003, have nurtured a
parallel city created by the articulation of new forms of social collective interactions
among facial avatars.
5. The urban face of the citizens in the participatory city
The digital transformation which has, technologically speaking, been feeding the
mediascapes described above is also considered to be the precondition for the rise
of the so-called ‘smart city’. This concept has become a major topic in the recent
discourse on city development (Crivello 2015) and has been widely discussed in
and out of academia, by pointing out its dimension of “global discourse network”
(Joss et al. 2019: 4). Strictly intertwined with sustainability and innovation, one
of the storylines that has informed the global smart city discourse has been
participation as a means to boost the democratization of the city management.
In an allegedly smart city, the citizens are provided with several platforms
and tools that supposedly are meant to enable citizens’ participation in different
domains such as environment, economy, mobility, education and politics.
ICT would thus empower the citizens to design and debate solutions about
city challenges and issues and eventually to have a voice in the public debate,
contributing to its governance (Simonofski et al. 2019).
The social innovation discourse has popularized the notion as co-creation
of urban living and promoted practices such as hackathons or data boot camps
by stressing the need to design and develop new smart city applications in
600
Elsa Soro
collaboration with citizens. On the company and business side, notions such as
‘co-production’, ‘prosumption’ and ‘peer-to-peer’ are some manifestations of the socalled sharing economy (or in its critical version, platform capitalism7) which refer
to the enrolling of the users on the demand side of the market through supposedly
disintermediated relationships.
The concept of participatory art has been recently galvanized by city administrations at a global level as a tool for providing the community with a new sense
of ownership of public space. Within this framework, the visualization of citizen’s
faces in artistic practices has become the figurative counterpart of the participative
smart city. In opposition to the photomontages of the totalitarian leader’s oversized
face that stood out from an indistinct mass, and in opposition to the dispersion and
fragmentation of power representations in the media city ideologies, the following
projects articulate a facial participative discourse and, by so doing, a collective
ownership of the city.
The French artivist JR has focused in his artistic research around faces through
a participative process. In fact, in several projects of his, faces of the citizens have
wallpapered different emblematic spaces, such as the Israeli West Bank Barrier,
the favelas of Rio de Janeiro, the Mexican border city Ciudad Juarez or the police
station in La Goulette, Tunisia, burnt down during the revolution. The art historian
Claire Bishop (2006: 1) defines the artist’s “interest in collectivity, collaboration,
and direct engagement with specific social constituencies” as “the social turn”
in contemporary art. In the case of JR, such a new form of participatory urban
storytelling has the face portrayal pasted on city surfaces at its core. Entire faces
can be pasted side by side, transforming a wall into a photo gallery (for instance
in Face 2 Face8), or appear as fragments, as in the case of the resistant vinyl photos
of women’s eyes in Women are Heroes9(Fig. 4).
7
The concept of ‘platform capitalism’ was introduced by the English scholar Nick Srnicek
(2017) to describe how the foundations of the economy after the crisis of 2008 were being
carved up among a small number of monopolistic platforms such as Airbnb, Uber, etc.
8
Face 2 Face, Israel and the West Bank. In the project JR paired up depicting Israelis and
Palestinians in locations on either side of the Separation Wall. Face 2 Face was accessed at
https://www.jr-art.net/projects/israel-palestine.
9
Women are Heroes, different locations. In the project JR portrayed women in their daily
lives and pasted them in relevant urban locations, such as Rio de Janeiro favelas and the Kibera
slum in Nairobi, Kenya. Women are Heroes was accessed at https://www.jr-art.net/projects/riode-janeiro.
Urban-human faces and the semiotic right to the city
601
Figure 4. Face2 Face; Women are Heroes. JR (https://www.jr-art.net/projects).
In the latter, the faces have been broken down and fragmented into their details
to create a proper facialization of the city. The informal and uneven surfaces of
the urban slums have been equipped with a gaze that looks at the spectators by
means of giving visibility to what is traditionally excluded by the city’s visibility
regimes. However, the sociologist Mubi Brighenti (2019: 16) argues that through
experiments such as JR’s “the city of faces may end up obscuring more than
clarifying what the face of the city is about [...]. If it exists, the face of the city
cannot be reduced to a selection of some of its dwellers’ faces”.
Although presuming that just the presence of citizens faces alone would enable
real democratization processes might be simplistic, the figurative mechanisms
make it possible to visualize the principle of the composition by creating a specific
meaningful mise-en-scène. In this regard, the collage used by JR can be interpreted
not just as a past technique, but, instead, as a composition method that is made
up of different-sized pieces: the collage (of faces) city shows thus a mélange of
different pieces, as a series of differences that have been brought together in a
unique composition.
Unlike the collage, the mosaic is a composition made up of pieces of the
same size. In this line, Selfiecities10 (Fig. 5) also defends the idea of the web,
and consequently a city, owned by users: “Contrary to earlier incarnations of
the web that were focused on content created by professionals, companies, and
organizations, we are now producing, sharing, or tagging massive amounts of our
own images and videos” (Hochman, Manovich 2013, s.p.)11. According to such
10
The project explores a dataset of 3200 Instagram selfie photos shared in 5 global cities, and
creates different visualizations of these photos with custom software. The website of the project
at http://selfiecity.net/ was accessed in November- December 2020.
11 Acceded online 2020 at https://firstmonday.org/article/view/4711/3698 in NovemberDecember.
602
Elsa Soro
rhetoric, Second Life avatars have been replaced by “real” citizens faces. The aim of
the project led by Lev Manovich and his team at CUNY is to question what visual
social media, and in particular Instagram, tells us about the relationship between
the city and individuals. As opposed to Manovich’s previous projects12, this inquiry
put the faces of city dwellers at the centre, by specifically investigating the style of
self-portraits (selfies).
Figure 5. Selfiecities, Bangkok. Manovich (http://selfiecity.net/).
12
Lev Manovich’s all projects can be browsed at http://manovich.net/index.php/projects/
tag:Project.
Urban-human faces and the semiotic right to the city
603
The project collected 656,000 images downloaded on Instagram, in five cities (New
York, Sao Paulo, Berlin, Bangkok, and Moscow) over a specific lapse of time (4–12
December 2013). The rudimentary automatic face analysis has been supplemented
by human judgment, through the inspection by Mechanical Turk workers, in
order to estimate the ages and genders of the people in the photos. The facial
expressions of the selfies in a city have also been studied for instance examining
which city smiles the most, with Bangkok smiling the most and Moscow the least.
The project’s conclusions focus, among other things, on how, despite Instagram
being a universal language, its usage changes according to the geography.
From a merely composition point of view Selfiecities appears as mosaics of
faces, which can be explored following visual patterns and rhythms across the
sets. The miniaturized faces are contained in a grid, side by side, in a series. The
pieces of the mosaic have the same, regular size and together they create a unified
series which does not present either discontinuity or rupture. Unlike the digital
maps that track communities’ clusters and mark distances and proximity among
them (as well as visualizing inclusion and exclusion processes), the Selfiescities,
apparently, advocates for denying hierarchies and categories, by displaying the
gesture of posting a composite self-portrait.
6. Conclusion
We have been observing the urban-human face(s) as configuration(s) that allow
us to unveil the tensions for the semiotic “ownership” and “authorship” of the
city. In the semiotics of culture, the city has been depicted as polyphonic and as a
polylogue in an isomorphic relation with the semiosphere as a whole. Following
this principle, the series of urban-human faces analysed diachronically in the piece
should be embraced in a synchronic way as well. Therefore, the assignation of a
face to a model of a city is always an incomplete operation as the city, from the
point of view of semiotics, is the result (always open) of the writing and rewriting
operations by different “authors” (for instance the “delegates” of a certain urban
ideology) who, together, orchestrate conflicting urban narratives.
Arguably, cities cannot have just a face unless we consider this as an unstable
semiotics mechanism. Therefore, the commensurability of the city and the face
can be based on the fact that both semiotic configurations represent an excess
with a cartographic reproduction (the city) and a portrait (the face), respectively.
A city can be represented by a face to the extent to which it is also multifaceted
as a polylogue. The aim to represent the struggle for the ownership of the city by
addressing the question to whom the city belongs is thus condemned to a fail, yet
this study has tried to give an account of such a failure.
604
Elsa Soro
To finalize the trajectory with a new start, a famous quote from Jorge Luis
Borges El Hacedor could be employed – an epilogue that refers to a man who
wanted to draw the world and populate the is world with all sorts of geographic
elements such as provinces, kingdoms, mountains, bays, ships, islands, fishes,
rooms, instruments, stars, horses, and individuals. Just before dying, the man
discovers that “the patient labyrinth of lines traces the lineaments of his own face”
(Borges 1960: 126). At present, cities and faces are both involved in processes of
resemantization of their respective geographies. In such crucial times, following
Borges’ image, we must not avoid the fact that, isomorphically speaking, the act of
drawing a space entails drawing a face.
Acknowledgements: This project has received funding from the European Research
Council (ERC) under the European Union’s Horizon 2020 research and innovation
programme (Grant agreement No 819649 – FACETS).
References
Atkinson, Rowland; Blandy, Sarah (eds.) 2006. Gated Communities: International Perspectives. London, New York: Routledge.
Atkinson, Rowland; Flint, John 2004. Fortress UK? Gated communities, the spatial revolt
of the elites and time–space trajectories of segregation. Housing Studies 19(6): 875–892.
https://doi.org/10.1080/0267303042000293982
Benjamin, Walter 1990. The Arcades Project. Cambridge: Harvard University Press.
Bishop, Claire 2006. The social turn: Collaboration and its discontents. Artforum International 44(9): 24–24.
Black, Daniel 2011. What is a face? Body & Society 17(4): 1–25. https://doi.org/10.1177/
1357034X11410450
Borges, Jorge Luis 1960. El Hacedor. Buenos Aires: Emecé Editores.
Castells, Manuel 1983. The City and the Grassroots: A Cross-Cultural Theory of Urban Social
Movements. London: Arnold.
Castells, Manuel 1989. The Informational City: Information Technology, Economic Restructuring, and the Urban-Regional Process. Oxford, Cambridge: Blackwell.
Crivello, Silvia 2015. Urban policy mobilities: The case of Turin as a Smart City. European
Planning Studies 23(5): 909–92. https://doi.org/10.1080/09654313.2014.891568
Davis, Mike 1990. City of Quartz: Excavating the Future in Los Angeles. London: Vintage.
De Certeau, Michel 1984. L’invention du quotidien. Paris: Gallimard.
Deleuze, Gilles; Guattari, Félix 1980. Mille Plateaux. Paris: Minuit.
Derrida, Jacques, 2002. Echographies of Television: Filmed Interviews. Cambridge: Polity
Press.
Farinelli, Franco 2009. La crisi della ragione cartografica. Torino: Einaudi.
Ferdman, Bertie 2012. Urban dramaturgy: The global art project of JR. PAJ. A Journal of
Performance and Art 34(3): 12–26. https://doi.org/10.1162/PAJJ_a_00107
Urban-human faces and the semiotic right to the city
605
Greimas, Algirdas 1976. Sémiotique et sciences sociales. Paris: Éditions du Seuil.
Hardt, Micheal; Negri, Antonio 2009. Commonwealth. Cambridge: Harvard University
Press. https://doi.org/10.2307/j.ctvjsf48h
Harvey, David 1973. Social Justice and the City. London: Arnold.
Harvey, David 2008. The right to the city. New Left Review 53: 23–40.
Hochman, Nadav; Manovich, Lev 2013. Zooming into an Instagram City: Reading the local
through social media. First Monday 18(7), s.p. https://doi.org/10.5210/fm.v18i7.4711
Joss, Simon; Sengers, Frans; Schraven, Daan; Caprotti, Federico; Dayot, Youri 2019. The
Smart City as global discourse: Storylines and critical junctures across 27 cities. Journal
of Urban Technology 26(1): 3–34. https://doi.org/10.1080/10630732.2018.1558387
Ivanov, Vyacheslav 1986. K semioticheskomu izucheniyu kul’turnoi istorii bol’shogo goroda’.
[On the semiotic research of the cultural history of the big city].Trudy po znakovym
sistemam [Sign Systems Studies] 19: 7–23.
Lefebvre, Henry 1991. The Production of Space. Oxford, Cambridge: Blackwell.
Lotman, Juri M. 1984. On the semiosphere. Sign Systems Studies 33(1): 205–226. https://
doi.org/10.12697/SSS.2005.33.1.09
Lotman, Yuri M. 1990. Universe of the Mind: A Semiotic Theory of Culture. London:
I. B. Tauris.
Lotman, Juri; Uspenskij, Boris 1977[1973]. Myth – name – culture. In: Lucid, Daniel (ed.),
Soviet Semiotics: An Anthology. Baltimore: Johns Hopkins University Press, 233–252.
Massey, Doreen 1996. The age of extremes: Concentrated affluence and poverty in the
twenty-first century. Demography 33(4): 395–412. https://doi.org/10.2307/2061773
McGrath, Brian 2006. Face city. Space and Culture 9(3): 235–244. https://doi.org/10.1177/
1206331206289363
Mubi Brighenti, Andrea 2019. Face and the city. Body & Society 17(4): 1–25. https://doi.
org/10.1177/1357034X19865941
Schneider, Jane; Susser, Ida (eds.) 2003. Wounded Cities: Destruction and Reconstruction in
a Globalized World. Oxford: Berg.
Simone, AbdouMaliq 2010. City Life from Jakarta to Dakar. New York: Routledge. https://
doi.org/10.4324/9780203892497
Simonofski, Anthony; Vallé, Troy; Serral, Estafania; Wautelet, Yves 2019. Investigating
context factors in citizen participation strategies: A comparative analysis of Swedish
and Belgian smart cities. International Journal of Information Management 56: 1–10.
https://doi.org/10.1016/j.ijinfomgt.2019.09.007
Skop, Emily 2006. Urban space: The shape of inequality. Urban Geography 27(5): 393–396.
https://doi.org/10.2747/0272-3638.27.5.393
Srnicek, Nick 2017. Platform Capitalism. Cambridge Polity Press.
Uspenskij, Boris A; Ivanov, Vjacheslav. V.; Toporov Vladimir N.; Pjatigorskij Alexander.
M.; Lotman, Juri M. 2003. Theses on the semiotic study of cultures (as applied to Slavic
texts). In: Eng, Jan van der; Grygar, Mojmír (eds.), Structure of Texts and Semiotics of
Culture. The Hague, Paris: Mouton, 1–28. https://doi.org/10.1515/9783110802962-002
Venturi, Robert; Scott Brown, Denise; Izenour, Steven 1977. Learning from Las Vegas.
Cambridge, MA: The MIT Press.
Virilio, Paul 1986. Speed and Politics. New York: Semiotext(e).
606
Elsa Soro
Young, Iris Marion 1990. Justice and the Politics of Difference. Princeton: Princeton University Press.
Zola, Emile 1883. Bonheur des Dames. Paris: Hatier.
Городские и человеческие лица и семиотическое право на город:
от механизма пропаганды СССР до города соучастия
Ситуация, в которой использование и значение городских пространств было
серьезно подорвано глобальной пандемией, породила множество дискуссий и
размышлений о том, каким образом устроены города – как общественные, так
и частные пространства. Статья рассматривает, как репрезентация «городского
человеческого лица» стала основой различных моделей урбанизма в разные эпохи
и культуры. Исходя из семиотики культуры, согласно которой город представляет
собой модель мира, в статье предпринята попытка интерпретировать, как портреты
лиц моделируют город с помощью различных механизмов градостроительства. В
центре внимания оказывается то, что мы называем «городским человеческим
лицом» – начиная от советских пропагандистских плакатов до цифрового проекта
#selfiecity. Это понятие относится к ряду образов, объединяющих город и лицо.
Можно утверждать, что и город, и лицо, произведенные в конкретный историкокультурный момент, с их образными и пластическими элементами, разворачивают
борьбу за право на город и на его авторство.
Тем не менее, соизмеримость города и лица может быть основана только на
том факте, что обе семиотические конфигурации представляют собой избыток с
помощью картографического воспроизведения (город) и портрета (лицо) соответственно. Город может представлять лицо в той степени, в которой он многогранен
как полилог. Исходя из такой неустойчивой соизмеримости, статья в конечном итоге
попытается свести воедино семиотику лица и семиотику города.
Linlik-inimlikud näod ning semiootiline õigus linnale:
NSVL propagandamasinast osaluslinnani
Nüüd, mil globaalne pandeemia on linnaruumi kasutamisele ja tähendusele dramaatilise
väljakute esitanud, on käimas mitmed debatid ja arutelud teemal, kuidas linnad – nii avaliku ja avalikkuse ees olevat privaatset ruumi osas kui ka privaatse ruumina – on kujundatud.
Artklis täheldatakse, et “linlik-inimliku näo” representatsioonid on erinevatel aegadel ja
kultuurides teeninud erinevaid linlikkuse mudeleid.
Kasutades kultuurisemiootikast tuletatud raamistikku, mille kohaselt linn esindab maailma mudelit, püüatakse artiklis tõlgendada seda, kuidas näoportreed on linna modelleerinud, erinevate linlike näolisusmehhanismide kaudu. Fookuses on kogum nähtusi, mida me
nimetame “linlik-inimlikeks nägudeks” ja mis ulatuvad Nõukogude propagandaplakatitest
digitaalse #selfiecity projektini. Väljend osutab reale representatsioonidele, mis toovad
kokku linna ja näo. Võib väita, et nii linn kui ka nägu, mis on produtseeritud konkreetsel
ajaloolisel ja kultuurilisel hetkel, oma figuratiivsete ja plastiliste elementidega, sisaldab
võitlust linna omamise ja autorsuse üle.
Urban-human faces and the semiotic right to the city
607
Linna ja näo ühismõõtmelisus võib tugineda lihtsalt tõigale, et mõlemad semiootilised
konfiguratsioonid esindavad liiasust, vastavalt kartograafilise reprodutseerimise (linn) ja
portree (nägu) abil. Nägu võib esindada linna sel määral, mil see on polüloogina paljutahuline. Sellise ebastabiilse ühismõõtmelisuse põhjal püüab artikkel viimaks kokku tuua
näo- ja linnasemiootika.
608
Ekaterina Velmezova, Kalevi Kull, Ene-Reet
SignSoovik
Systems Studies 49(3/4), 2021, 608–615
Jaan Kaplinski and his contacts with
the Tartu-Moscow School of Semiotics
Ekaterina Velmezova, Kalevi Kull, Ene-Reet Soovik
Как тихо все в Кяэрику каким молчанием молчат
окна лестницы и площадки молчит потрескавшийся асфальт
под муравьиными тропами как где-то молчат
все кто тут был говорил и читал наизусть стихи
как молчат Юрий Михайлович Александр Моисеевич
молчит Линнарт и Хальянд не скажет ни единого слова
и где-то куда я еще не вернулся я еще там
мартовское солнце режет-колет мои босые ноги
а я молодой и влюбленный убегаю дальше и дальше
по колючему снегу навстречу нашему соборному молчанию1
Kaplinski 2014: 63
Abstract. Jaan Kaplinski (1941–2021), Estonian poet, essayist and public intellectual, sadly passed away earlier this year. To commemorate him, we publish some
excerpts from a conversation with him that was recorded in 2018 and in which,
among other topics, we also talked about Kaplinski’s relationship with semiotics and
his personal contacts with eminent scholars of the Tartu-Moscow School.
Jaan Kaplinki (22 January 1941 – 8 August 2021), internationally acclaimed Estonian poet, essayist, and culture critic, recipient of the European Prize for Literature
(2015)2 was known for his focus on global issues, including academic concerns.
Kaplinski was born in Tartu and studied French language and literature at the
University of Tartu, graduating from the five-year programme that was called
1
How quiet everything in Kääriku is with what silence / the windows the stairs and the
grounds are silent the cracked asphalt keeps silent / under the ant paths how they are silent
somewhere /everyone who was here spoke and recited poetry / how silent Juri Mikhailovich
Alexander Moiseevich are / Linnart is silent and Haljand will not say a single word / and
somewhere where I have not yet returned I am still there / March sun pricks and cuts my bare
feet / while I young and in love run further and further / across the prickly snow towards our
conciliar silence
2
See http://www.prixeuropeendelitterature.eu/kaplinski.html.
https://doi.org/10.12697/SSS.2021.49.3-4.18
Jaan Kaplinski and his contacts with the Tartu-Moscow School of Semiotics
609
‘French Philology’ in 1964. Besides publishing poetry, he penned fiction, dramas,
texts of popular science and participated in discussions on linguistics, translation
studies, Oriental studies (particularly as concerns the Far East), sociology and
ecology.3 His notable role in mediating semiotic thinking in culture has prompted
us to observe his activities in the context of semiotics in general.
Canonical descriptions of the Tartu-Moscow School of Semiotics and its
activities in the 1960s and 1970s often tend to limit the list of its members on
the Estonian side to a single person – Juri Lotman. At a closer look, however, it
certainly appears that such an account is far from complete, as there were several
Estonian scholars representing different academic backgrounds who directly
contributed to the work of the school during its heyday in the period of 1964–1974:
Igor Černov, Linnart Mäll, Ivar Kull, Uku Masing, and others. There was also
a circle of local academics influenced by the school, who were instrumental in
transferring the ideas of the semiotic school into the Estonian academic discourse
and thus contributed to the local intellectual support to the school. The latter
included a group of young linguists who gravitated to the work of Huno Rätsep,
who taught structural linguistics at the University of Tartu in the 1960s, as well as
participants in the seminar called Generative Grammar Work Group. Listed among
these scholars were Haldur Õim, Mati Hint, Arvo Krikmann, Tiit-Rein Viitso, Pärt
Lias, Leo Võhandu and others – no to mention Jaan Kaplinski.
Kaplinski was not a semiotician by profession or formal training, but he
maintained important contacts with some of the members of the Tartu-Moscow
School. Even though he would later become a renowned poet and author of both
fiction and literary non-fiction, in the 1960s he entertained doubts about his
choice of career, feeling uncertain whether to opt for a poetic or an academic
vocation. He was certainly interested in linguistics, and there was much in his
linguistic interests – which also involved reflections on texts and signs – that would
bring him closer to the participants of the Tartu-Moscow School. He has also
expressed regret that, instead of French, he did not enroll at the university as a
student of Russian language and literature. Nevertheless, although Jaan Kaplinski
did not become a linguist or a semiotician, some episodes from the history of the
Tartu-Moscow School of Semiotics, as well as some semiotic concepts in general,
are reflected in his literary work. He also had an opinion on how the concepts
developed by the Tartu-Moscow semioticians could be applied to the study of
Estonian culture and cultural history – work that in his regard was still waiting to
be carried out.
3
On him and his writings, see e.g. Salumets 2014 in English; Soovik 2015a, 2015b, 2016 in
Estonian.
610
Ekaterina Velmezova, Kalevi Kull, Ene-Reet Soovik
In addition to the legendary Kääriku Summer Schools on secondary modelling
systems, Kaplinski also participated in the significant biosemiotic meeting “Biology
and Linguistics”, held in Tartu in February 1978. In 2000 he gave a talk at a seminar
on semiotics of nature, also arranged in Tartu, and in June 2007 he delivered a
keynote lecture “Two types of human communication” at the 9th World Congress
of Semiotics in Finland. In the autumn of 2012, he contributed to the vivid
academic scene of the Department of Semiotics at the University of Tartu by giving
a lecture series titled “Languages, cultures, and thought”.
Below, we reproduce, in English translation, some excerpts from the originally
Estonian-language conversation with Jaan Kaplinski that took place at his country
house at Mutiku, Estonia, on 7 January 2018 (for the full interview, see Kull,
Velmezova 2018).
When did your first come in contact with semiotics?
Jaan Kaplinski: I got acquainted with semiotics when I was a student. But my first
more significant contact with semiotics took place at a semiotics Summer School
at Kääriku in 1964.4 I think Igor Černov was an important link between me and
Lotman.
At the time you published several scholarly articles in the edited collections of the
generative grammar work group, didn’t you?
Jaan Kaplinski: I did that a little earlier, linguistics still interests me now,5 if I could,
I would do more… I have my own point of view here, something like a linguistic
typology; I wrote about Celtic and Finno-Ugric parallels and so on. I then read a lot
on Finno-Ugric and Celtic Studies ... But the orthodox structuralism that they were
engaged in at the time did not suit me. It’s boring and, in fact, has no perspective,
because it’s just plain wrong to approach language in this way. As far as semiotics is
concerned, it really gave people an opportunity to do things that would otherwise
have remained impossible.
Did you attend Lotman’s lectures?
Jaan Kaplinski: I did not regularly attend his lectures, but I have attended his talks
many times.
4
Jaan Kaplinski wrote a review of this event for the journal Keel ja Kirjandus [Language and
Literature] (Kaplinski 1964).
5
See Kaplinski 2020.
Jaan Kaplinski and his contacts with the Tartu-Moscow School of Semiotics
611
Did you only go to the already mentioned Summer School at Kääriku or did you also
visit some later schools?
Jaan Kaplinski: I visited others as well. One day, it was in spring, there was a
school at Kääriku that was devoted mainly to Oriental studies. But many of the
participants of the semiotic meetings were present as well6. It was at Kääriku that
I became in touch with them; then, when I worked at the university’s sociology
laboratory [in the early 1970s], I went to Moscow to work at the libraries there. In
Moscow, I visited Pyatigorskij, and after that I developed a close contact with him.7
Afterwards I visited him in London several times.
What do you think was the most important thing that Lotman and the Tartu-Moscow
School achieved?
Jaan Kaplinski: One of their undoubted achievements was that they opened up the
way to free thought, as those who were somehow connected with them in one way
or another – like Linnart Mäll, for example – mostly weren’t semioticians at all.
Pyatigorskij, in my opinion, was not a semiotician either... He was an Orientalist
and a philosopher – or rather, a philosopher and an Orientalist. And me, too,
I don’t really know semiotics, I’m a linguist. I am interested in the problems of
semantics – for example, how and how often the names of body parts are used
metaphorically in Eurasian languages. In the broadest sense of the word, this is,
of course, semiotics – it is meaning that is being investigated here. Studying a
language without studying meaning is no good.
Did you read Saussure at the university?
Jaan Kaplinski: Yes, it was not difficult for me: I spoke French well. ...
In many cases, the [language] sign is really arbitrary. When we say, for example,
‘saba’, the Estonian word itself is unlikely to be associated with the tail as such. Or,
for example, when we speak German: ‘das Auge’, ‘die Nase’, ‘der Mund’… Today,
why in one case we have ‘die’, in another – ‘der’, and in yet another – ‘das’, is not
determined by anything. There are many examples of arbitrariness.
There is a phenomenon in the Estonian language that particularly interested
me – this phenomenon is ideophony, it is my favorite topic in linguistics.8 If we
open the Estonian etymological dictionary, we will immediately see the expression
‘phonetically motivated’ (‘häälikuliselt motiveeritud’). And I counted them,
6
This may have been a reference to the third semiotic Summer School which took place in
Kääriku on 10–20 May 1968.
7
Thomas Salumets briefly evokes Jaan Kaplinski’s relationship with Lotman and Pyatigorskij
as well (Salumets 2014: 58, chapter “Erudite critic of culture – a walking contradiction”).
8
See Kaplinski 2010.
612
Ekaterina Velmezova, Kalevi Kull, Ene-Reet Soovik
these “phonetically motivated” stems comprise some 20 per cent or even more.
This phenomenon is typical of the Finno-Ugric languages, as well as of Korean
and Japanese languages, to some extent also of Chinese, and in the Dravidian
languages there is a lot of ideophony too. It is even more common in Finnish than
in Estonian – but in Finnish, ideophony is presented a little differently... You can
speak Finnish dryly – not in an official-sounding, but just neutral manner. And
you can speak brightly, using ideophonic words – as, for example, in a poem by
Ilmari Kianto about the northern lights: “Ne leimuaa, ne loimuaa ...”.
In what way, then, was Saussure important to you?
Jaan Kaplinski: His book organized a lot for me. In the end, it all became so
self-evident that I just got used to it – synchrony, diachrony... This even has a
connection with Buddhist philosophy: everything is connected with everything.
Each and every thing is a denial of all other things, so to speak…
I have the feeling that many things could be studied in the framework of
another paradigm, and I don’t know how much the semiotic paradigm allows to
explain things that other paradigms do not. Yet apparently it allows quite a lot.
More I cannot say. It is also interesting how semiotics makes it possible to combine
so different phenomena as Jakob von Uexküll and Ferdinand de Saussure. To build
bridges, interdisciplinary bridges – this alone is very important in today’s world
of over-specialization.
You have not been writing much about semiotics, but you have written a piece in
Russian, entitled “Действительные числа” (Real numbers),9 in which you speak
about the Tartu-Moscow School, and in addition, semioticians are mentioned in a
poem you wrote in Russian10…
Jaan Kaplinski: “Действительные числа” is a short story which is something
of an expression of sympathy for all these people on my part. I have a somewhat
Nabokovian aspiration in me to write about people who are a little crazy – ‘with
a birdie’ (‘linnukesega’), as they say in Estonian. The protagonist of “Действительные числа” is just that.
Why do you think there were so few Estonians who were drawn to Lotman?
Jaan Kaplinski: Estonians kept to their own bubble, their Estonian bubble. In
retrospect, this seems to have been wrong, and I was relatively internationally
minded on my part. After all, I grew up on Russian literature, and it was a stupid
9
10
Kaplinski 2000.
See the epigraph on p. 608 of this issue.
Jaan Kaplinski and his contacts with the Tartu-Moscow School of Semiotics
613
thing to do to go to study French. … I should have opted for Finno-Ugric studies –
I was quite energetic, I could have travelled across Russia visiting Finno-Ugric
peoples and collecting material, such things are of a great value for the posterity.
Or else I could have taken up Russian language and literature, in this case I would
naturally have reached Lotman. But what happened, happened. … There were two
places where one could seriously commit to studying philology – with Lotman and
with Ariste11...
And of course, Russian semioticians did not feel much noticeable interest
in what was so important for Estonians, in our own culture, history, desire for
independence – all this remained alien to them. Although “the semiotics of being
Estonian” could be a very interesting research topic.
What, in your opinion, was the most important or the most interesting thing about
Lotman all in all?
Jaan Kaplinski: His charisma! Many of his theoretical views were, perhaps, too
schematic, he could hover too far off from facts, yet at the same time he was a
wonderful specialist in the history of culture. This is the best thing he has done,
and this is what will live the longest. And how easy it was to get through to him...
And they should have addressed Estonian material – authors like Oskar Luts
or Tammsaare as far as I am concerned, and studied Estonian material from this
aspect... But none of them trusted themselves to do this… And they did not have
many Estonian students, so they could have just employed Lotman’s methods, as
it were, the way he treated cultural history, observe something in the evolution of
Estonian culture from a semiotic perspective, and this has not happened yet. For
instance, the opposition of and the relations between ‘one’s own’ and ‘the other’. Or
take the oppositions that have determined the Estonian culture: the orientation to
the West or the East, the self and the other and many other topics, Southern and
Northern Estonia, for example.
This work is not yet done. Lotman’s legacy has been travelling around the
world, but as far as I know, our own, Estonian culture has not been studied from
a semiotic perspective yet...
Instead of a postscript
“Lotman was right when he argued that prose is more complex than poetry,”
Kaplinski (2004: 361) said in connection with his literary work. The essay
“Kirjanduse tähendusest” (‘On the meaning of literature’) (Kaplinski 1968), one
11
Paul Ariste was a linguist, specialist in Finno-Ugric languages.
614
Ekaterina Velmezova, Kalevi Kull, Ene-Reet Soovik
of the earliest examples of Kaplinski’s ties to semiotics, can be considered a
publication in which the understanding of the role of literature and art is discussed
from Lotman’s perspective in many respects.
Later, Kaplinski would draw on his reflections on Ludwig Wittgenstein to write
a treatise entitled 621 (Kaplinski 1992). One of the few people to have provided indepth analysis of this text was the Estonian theoretical physicist and philosopher
Madis Kõiv (Kõiv 2001). In Kaplinski, he saw a potential disciple not so much of
Lotman, but of Viktor Shklovskij, placing the renowned Russian formalist in the
same – not just metaphorical, but also geographical – space with another mentor of
Kaplinski’s, theologian and poet Uku Masing who is behind the figure of the Teacher
in Kaplinski’s novel The Same River (Kaplinski 2009[2007]). This is what Kõiv wrote:
It would be better, for instance, to have Viktor Shklovskij settle in the post-war
Tartu, because in the case of Lotman, whose student he [Jaan Kaplinski] would
have liked to be, some issues of incompatibility [“õmblemisraskusi”] would arise.
Let there be Shklovskij in Lotman’s place, this would not mean any significant
difference in the primary tuition delivered; after his withdrawal from Moscow,
he could land in Hurda Street12 (“in exile”) right after the war, and, since in the
immediate post-war years there would not have been a sufficient, and sufficiently
well educated, Russian-speaking environment here, young K. as his student could
serve as a good consolation for him; as for language, I suppose no difficulties
would arise in case of two people so sensitive to it. (Kõiv 2001: 92)
References
Kaplinski, Jaan 1964. Semiootika suvekoolist Käärikul. [On a semiotics summer school in
Kääriku.] Keel ja Kirjandus 12: 764–765.
Kaplinski, Jaan 1968. Kirjanduse tähendusest. [On the meaning of literature.] Looming 3:
432–441.
Kaplinski, Jaan 1992. 621. Looming 11: 1499–1544.
Kaplinski, Jaan 2000 = Каплинский, Яан 2000. Действительные числа. [Real numbers.]
Звезда: Литературный и общественно-политический журнал 11: 88–100.
Kaplinski, Jaan 2004. Kõik on ime. [Everything is a miracle.] (Salumets, Thomas, ed.; Eesti
mõttelugu 55.) Tartu: Ilmamaa.
Kaplinski, Jaan 2009[2007]. The Same River. [Seesama jõgi.] (Wilson, Susan, trans.) London,
Chester Springs: Peter Owen.
Kaplinski, Jaan 2010. Ideofoonidest meil ja mujal. [On ideophones with us and elsewhere.]
Keel ja Kirjandus 2: 133–136.
Kaplinski, Jaan 2014 = Каплинский, Яан 2014. Белые бабочки ночи / Бѣлые бабочки
ночи. [White butterflies of night.] Таллинн: Kite.
12
This is the street in which Uku Masing lived.
Jaan Kaplinski and his contacts with the Tartu-Moscow School of Semiotics
615
Kaplinski, Jaan 2020. Eesti, estoranto ja teised keeled. [Estonian, Estoranto and other
languages.] (Bibliotheca controversiarum.) Tallinn: Tallinna Ülikooli Kirjastus.
Kull, Kalevi; Velmezova, Ekaterina 2018. Jaan Kaplinski ja semiootika. [Jaan Kaplinski and
semiotics.] Acta Semiotica Estica 15: 194–211.
Kõiv, Madis 2001. Nõnda lausus Jaan Kaplinski. [Thus said Jaan Kaplinski.] Looming 1:
73–101.
Salumets, Thomas 2014. Unforced Flourishing: Understanding Jaan Kaplinski. Montreal:
McGill-Queen’s University Press.
Soovik, Ene-Reet 2015a. Kodukoht ja teeleidmine: Kaplinski kirjutatust. [Home and
wayfinding: On Jaan Kaplinki’s writings.] Keel ja Kirjandus 58(2): 73–87.
Soovik, Ene-Reet 2015b. Kahest aiast ja mõnest mullast: Kaplinski aiad ja peenramaad.
[Two gardens and several soils: Kaplinskis gardens and vegetable plots.] Keel ja
Kirjandus 58(11): 765–776.
Soovik, Ene-Reet 2016. Kaplinski kaugused. [Kaplinski’s far-away spaces.] Keel ja Kirjandus
59(2): 114–126.