Figure 1: Lev Manovich; a comparison of brightness and saturation of a selection of about 130 paintings by Mondrian
and Rothko. Screenshot from: http://lab.culturalanalytics.info/2016/04/mondrianvsrothko.html
Digital Art History and the
Computational Imagination
Giacomo Mercuriali
Abstract: This essay explores the parallel development of computer vision technology and
digital art history, examining some of the current possibilities and limits of computational
techniques applied to the cultural and historical studies of images. A fracture emerges:
computer scientists seem to lack in the critical approach typical of the humanities, a
shortfall which sometimes condemns their attempts to remain technological curiosities.
For their part, humanists lack the technical knowledge that is needed to directly investigate
large archives of images, with the result that art historians often must limit digital research
to databases of text or metadata, a task that does not necessarily facilitate the study of
the images themselves. A future dialogue between the two areas is required to foster
this new branch of knowledge.
Keywords: Computer vision, digital art history, computational imagination.
Alternative
Futures
Let us think for a moment about
the futuristic world conceived by Isaac
Asimov in some novels and short stories.
In this narrative universe, the Multivac,
a supercomputer kept by the United
States in a secret location, is employed
by the public administrators to make
the most critical decisions about the
state of war, public health and scientiic
problems. Multivac acquires data thanks
to the work of a selected group of
engineers, who ill it with information
and pose questions in natural language.
he machine responds via text strings.
In some short novels, which preigure
the Internet, every citizen can employ
the Multivac in almost the same way,
posing questions through private ter-
minals and receiving personalized
answers. In he Last uestion (1956) the
most intriguing story among the series,
Multivac’s potentialities coincide with
all Earth’s computing power: it has
now acquired a kind of intellectual
supremacy over humans, who use it
to direct their interstellar expansion
towards the limit of the universe.1
In our reality, it was mostly the
work of individuals has provided the
world network with multitudes of data
and metadata, available in different
states of aggregation, the biggest of
which are known as big data. We then
ind ourselves in a specular position
compared to that devised by Asimov
as the initial episode of the Multivac
saga: an immense quantity of data is
available through the Internet, and yet
any artiicial intelligence technology
is nowadays able to coherently and
Computational Imagination
autonomously operate on the total
mass of information. In the field of
information technology, futurologists
multiply their cabalistic prophecies,
striving in atempts to determine the
“point of no return”, when the ultimate
self-improving artificial intelligence
will be born, inally merging with our
biological body.2
It is interesting to notice that in
Asimov’s iction the Multivac acquires
and hands out information only in the
form of text strings; his epoch didn’t
know about the graphical interfaces
that today mediate the interaction
between users and software. By
contrast, George Orwell’s 1984 (1949)
constitutes a milestone of modern
science-fiction precisely because it
stages the appearance of an iconotechnical knowledge based on the
continuous and pervasive analysis
of large amounts of images which
condemns the dim inhabitants of the
state of Oceania, transformed in an
enormous panopticon, to follow the
totalitarian form of life imposed by
the government’s Party. 3 Orwell’s
novel can, therefore, be inserted inside
a millennial line of thought that,
starting with Plato, has suspected the
social role of images. As a result, we
are accustomed to think that, on the
one hand massive computing based
on linguistic information seems to
naturally facilitate social devel opment; on the other hand, large-scale
elaboration of iconic data is primarily
thought as a form of danger for humankind.
142 DAH-Journal #3
Imagination and
algorithms
his presupposed dystopic scenario
is indeed already part of our reality:
we use facial-recognition sotware to
classify the images stored in our PCs
or social networks when they prompt
automatic tags for persons that recur
a certain number of times within our
digital photo albums. In 2016, a Russian firm developed a system that
identiies individual faces (morphology, gender, age, emotions) comparing
the images taken by public CCTVs and
photo albums uploaded in Vkontakte
(a Russian social media platform).4
If the police force implements this
technology in its surveillance system—
as it is already the case in China—it
will be almost impossible for citizens
to anonymously move in urban areas—at least without disguises or antirecognition camoulages, such as those
developed since 2010 by the artist Adam
Harvey.5 Automatic face-detection systems based on the computation of iconic
big data will be presumably added fast
(if they have not yet been implemented)
to the telecommunication systems employed by the USA for combat and forensic objectives, as recently revealed
by Edward Snowden.6
We are therefore crossing the threshold of an epoch in which the prosthetic
delocalization of the imaginative faculty, our capacity for thinking images
and operate with them, moves towards
the progressive demonstration of what
Computational Imagination
Charles Baudelaire afrmed in a leter
which atracted the atention of Walter
Benjamin while he was working on his
uninished essay on the 19th century:
“Imagination is the most scientific
of the faculties”.7 he economic and
intellectual eforts of the IT industry is
preparing a future in which the irreducibility of language and image, which
had seemed partitioned for a thousand
years, will be torn down by algorithms
which man i pu late pi xels: machine
vision is leading to self-driving vehicles,
identiication of tumors, bombing and
special efects in the visual arts. As we
await the oft-heralded bo dily reabsorption of technical prostheses through
biotechnologies, our current moment
is marked by the exponential growth
of automatic imaginative faculties that
are stemming from new methods of
automated calculus, statistical analysis
of enormous databases, and production
of novel hardware .
From the perspective of “artistic”
production, the frontier of the computational imagination is rapidly expanding: we need only to name a few
of the artistic applications, such as the
generators of actor-avatars employed
in cinema since the end of the ’90s or
the program designed by Robbie Barrat
which “paints” in diferent styles via
neural networks.8
What would happen if an ideal
Multivac were utilized by a group of
historians, rather than police states or
marketing irms? What would result if
this kind of artiicial intelligence would
direct its eforts not to the identiication
of potential terrorists or our tastes about
furniture and fashion,9 but rather to the
analysis of the history of visual culture?
his possibility is grounded in recent
acquisitions in information technology:
Google’s research of images through
images has been implemented just in
2011, and there is still a lot of space
for the improvement of the relative
algorithm.10
The development
of a new research
ield
The multidisciplinary field of
digital art history tries to integrate the
mathematical and statistical expertise
of information technology scientists
with art history and visual culture
studies. For the moment, the rit that
still separates the competences of those
who were trained in each of those
disciplines is quite large and the efects
of this situation can be perceived in the
distinctive features of the publications
and research projects that are currently
holding the label of digital art history.
As an emerging subield of digital
humanities, the discipline nowadays
is fostered by the recently born
International Journal for Digital Art
History. Among the authors who
published their researches in the
review, Lev Manovich is one of the most
representative. Manovich, professor
of theory and history of media at
the City University of New York, has
DAH-Journal #3 143
Computational Imagination
been processing iconic big data at the
“Cultural Analytics Lab” for the past
decade. His image sources come from
museums, movies, videogames, social
networks, and magazines.11
On some
epistemological
problems in
digital art history
In his paper Data Science and Digital
Art History,12 Manovich describes his
methodology, as part of a “quantitative
turn” that the humanities as a whole
have experienced in the 20th century:
the digital version of an image contains
certain kinds of information that can be
employed as a yardstick, allowing welldesigned algorithms to automatically
compare a vast number of documents,
a task unachievable by a human mind
with its limited memory. Big iconic data
sets—an artist’s oeuvre, the shots of a
movie, the covers of Time magazine—are
iltered through a computing process
that selects only certain features of the
source document; then each object gets
assigned coordinates that locate each
of them in an n-dimensional “feature
space”. his space of virtually ininite
dimensions is subsequently latened
into one or various bi-dimensional
graphics where the relative distances
of the objects (measures that stem from
the criteria chosen by the experimenter
at the beginning of the process) become
perceivable to our eye.
144 DAH-Journal #3
We can now grasp in a glimpse, for
example, the diferences in brightness
and saturation between the corpus of
Piet Mondrian and Mark Rothko, thus
evaluating general characteristics that
only well-experienced connoisseurs
of their work might appreciate.13 At
the same time, we ask ourselves if
Manovich’s conclusions (“Projecting
sets of paintings of these two artists
into the same coordinate space reveals
their comparative ‘foot prints’—the
parts of the space of visual possibilities
they explored. We can see the relative
distributions of their works—the denser
and the more sparse areas, the presence
or absence of clusters, the outliers,
etc. he visualizations also show how
Mark Rothko—the abstract artist of the
generation which followed Mondrian—
was exploring the parts of brightness/
hue space which Mondrian did not
reach») can give fundamental insights
to the art historian. Moreover, they contain some epistemological problems.
First of all, the features analyzed
are, strictly speaking, the photographic
reproductions of the paintings and
not to the artworks themselves. he
phenomenical atributes of paintings
strongly depend on the illumination
to which they are exposed (not to say
about the position—distance, nearness,
parallax, relative movement—of the
perceiver) and in many cases—such as
Rothko’s Seagram series—are relevant
to the conception of the artwork itself.
Secondly, dealing with numbers of reproductions, in the probable case of a
lack of a careful normalized process in
the shooting procedures that generate
Computational Imagination
the digital photographs of the study set,
a certain quantity of error will afect
the relative positions of the objects in
the feature space of optical values such
as brightness and saturation. his error
will not presumably be so discriminant
as to impede high-level considerations—
we could easily think of a fast and
efcient visualization of “color-periods”
inside the production of an artist (e.g.,
Picasso’s “pink” and “blue” periods)—
but, in the case of further employment
of this map, we must remember that
errors expand exponentially. Lastly, it
is questionable whether the inclusion
of a reduced number of documents and
not all the catalogue of the artists in
the calculus leads to a neutral scater
of the images on the table or, rather, to
a biased result (the “visual possibility”
insight being then compromised).
Manovich’s enthusiasm is also
shared by other research groups. In
2014, a team led by Babak Saleh at
Rutgers University published a paper
entitled Toward Automated Discovery
of Artistic Influence.14 The scientists,
commited, like Google, to the challenge of automatizing the semantic
description of images, have developed
an “inluence” algorithm that works on
certain formal similarities between the
images of the initial data set. he team
reported that the program they wrote
was able to spot a never-before-seen
connection between two paintings: one
from 1870 by Frédéric Bazille and the
other from 1950 by Norman Rockwell.
his result was harshly criticized by
the art historian Griselda Pollock,
that accused the computer engineers
of utilizing an anachronistic methodology: the reductionist paradigm
of connoisseurship.15 Saleh’s supervisor, Ahmed Elgammal, replied some
months later explaining that the new
research ield of “computer vision” is
only at its beginning and that its longterm objectives are the realization of a
program that could pass what he names
a “visual Turing test”.16
This statement is interesting because it seems to widen the classical
proof of computational intelligence
that computer engineers have been
trying to atain for more than half a
century. In the original version, the
test consists in a linguistic game in
which the computer is required to
mimick the communicative abilities of
a human being. Elgammal’s suggestion
indicates that nowadays the research
on AI is aware that language is only
half of the moon, the bright one. he
discovery of the dark side corresponds
to the project of providing the machine
with an imaginative capacity.17
Multivac’s paradigm remains the
foundation of computer sciences; as a
mater of fact, Elgammal continues with
a consideration on the digitalization of
archives: “Perhaps there will be a day
when the technology could evolve
to look at the historical, social, and
personal context of art—a day when
computers could mine these vast stores
of heterogeneous data to conduct an
analysis of artistic inluences that goes
beyond the connoisseurial approach”.18
To overcome such approach, with a
view on a Bildwissenshat 2.0, it would
DAH-Journal #3 145
Computational Imagination
however be necessary to automatize
the critical analysis carried out by human researchers, who comprehend typologies of resemblance (e.g. anthropomorphism, pseudomorphosis, the
informal) which can complicate the induction of relationships (of inluence)
on strictly mimetic similarities.19
Blending big
iconic data
Different approaches, which aim
instead to present large numbers of
images inside graphics or navigable 3D
virtual spaces in aesthetically pleasing
ways, are currently being explored
by Google. he big irm, compared to
other research teams, can avail itself
Figure 2: The “degrees of separations” that relate a symbolistic sculpture wich a drawing of a
glass jar for X Degrees of Separation. Screenshot from: https://artsexperiments.withgoogle.com/
xdegrees/8gHu5Z5RF4BsNg/BgHD_FxbV_K3A.
146 DAH-Journal #3
Computational Imagination
of the quality of the data gathered
via its Art Project, which brought the
cameras of Street View inside the major
museums of the world. The online
application X Degrees of Separation
is presented as such: “Using Machine
Learning techniques that analyze
the visual features of artworks, X
Degrees of Separation inds pathways
between any two artifacts, connecting
the two through a chain of artworks.
his network of connected artworks
allows X Degrees of Separation to
take us on the scenic route where serendipity is waiting at every step: surprising connections, masterful works
by unknown artists or the hidden
beauty of mundane objects”.20 It may
be superfluous that such paths are
limited by the initial set since, for the
moment, a universal catalog of (socalled) artistic objects does not yet
exist. It is nevertheless certain that
Google’s projects could be integrated,
Figure 3: The photography of a pet competition is related by Recognition to a XVIII century painting.
Screenshot from: http://recognition.tate.org.uk.
DAH-Journal #3 147
Computational Imagination
in the near future, with systems of
icon o graphic classification such as
Iconclass.21 What research possibilities
would be opened performing semantic
researches on big sets of images that
were not previously carefully cataloged
by human archivists—that is to say, the
vast majority of the cultural heritage
which is currently undergoing a process of digitalization around the world?
An essay is given, again, in Google’s
experiment Tags, which nonetheless
retains amusing censorship since it does
not allow one to search for “nudes”,
while other search terms such as “rile”,
“gun” or “guillotine” are currently allowed.22
An essay similar to Google’s was
that one performed by Recognition, a
program developed at the Italian innovation center Fabrica, winner of Tate
Gallery’s 2016 IK Prize.23 An algorithm
automatically compares photographs
coming from international press agencies with the artworks held by the
important English collection. he similarities are chosen through criteria of
formal and metadata resemblance; unfortunately, it remains unclear whether
any speciic knowledge could be gained
by such operations.
The quest for
interdisciplinarity
For the moment, traditional art historians can continue to sleep tight. As
long as the strong separation between
data sets and algorithms or AIs will
148 DAH-Journal #3
be maintained, it is impossible that
some computer will steal their job.
Nevertheless, some departments of art
history and architecture are developing
study programs and research centers
whose aim is to gather the competences
of humanists and computer scientists
under one roof. Institutions such as the
Gety Research Institute, the Courtauld
Institute of Art and the Frick Collection
are preparing for the future of digital
art history.24 hese initiatives relect the
slow reception of this new discipline
whose origins are to be found in the
late ‘80s.25
Nowadays, the digital art history
projects fostered by humanists can be
divided into three areas that, contrary to
the projects based on computer vision
and AIs, apply the new technological
possibilities to information that are external to the images themselves and,
interestingly, oten present their research in the form of another image.26
he irst class employs digitized text
databases to develop statistical approaches; one possible application is
the analysis of archival material related
to collections: such is the case of the
Medici archive recently digitized by
the Fondazione Memofonte.27 These
second kind of process facilitated by
digital technologies is the architectural
rendering of historical sites; such is
the case of Visualizing Venice, which
aims to build a virtual 3D model of the
Serenissima that should be navigable
at its diferent time periods.28 Finally,
the third type of research, an expansion
of social history of art, is the so-called
“network analysis” which, applied to
Computational Imagination
Figure 4: The relational network of Theodore Roussel and James Whistler: models, patrons, artists,
pupils and family members. Screenshot from: http://linkedvisions.artic.edu/network.php
art circles, galleries and the art market,
visualizes different kinds of social
realtions.
In this overview, I tried to trace
the borders of two areas of research
which still await coherent overlap.
For the moment, a fracture emerges:
those who study images with methods
of computer science seem to omit a
certain epistemological problems, with
results that, from the perspective of
the art historian, are more curiosities
than new knowledge. At the same time,
their work expands the awareness of
the need for imaginative capacities
for the future AIs, which should have
a high level of image comprehension
in order to interact with “intelligence”
with the world. On the other hand,
the humanists who try to update
their practices, tend not to possess
the technical programming skills that
would be necessary to apply a critical
approach to the study of images themselves, and, for the moment, they investigate information of another kind,
which reside in the contextual appearance of the data.
If in the future new scholars with
a double competence will be trained,
maybe we could progress a litle towards the goal of an intelligent computational imagination, that will let us
not only to drive cars, identify diseases
and monitor our neighbor but also to
glance with a new perspective towards
our past.29
DAH-Journal #3 149
Computational Imagination
Notes
1 Isaac Asimov, “he Last uestion,” Science
Fiction uarterly 4, no. 5 (November 1956), 6–15.
2 Ray Kurzweil, he Singularity Is Near: When
Humans Transcend Biology (New York: Viking,
2005).
3 George Orwell, Nineteen Eighty-Four: A
Novel (London: Secker & Warburg, 1949).
4 “FindFace Pro”, https://findface.pro; cfr.
Shaun Walker, “Face recognition app taking
Russia by storm may bring end to public anonymity”, he Guardian, May 17, 2016, htps://
www.theguardian.com/technology/2016/
may/17/findface-face-recognition-app-endpublic-anonymity-vkontakte.
5 Tom Phillips, “China testing facial-recognition surveillance system in Xinjiang”, he
Guardian, January 18, 2018, https://www.
theguardian.com/world/2018/jan/18/chinatesting-facial-recognition-surveillance-systemin-xinjiang-report. Adam Harvey’s “CV Dazzle”,
htps://cvdazzle.com.
6 Mateo Pasquinelli, “Arcana Mathematica
Imperii: he Evolution of Western Computational Norms”, Former West: Art and the Contemporary ater 1989, ed. M. Hlavajova and S.
Sheikh (Cambridge, MA: MIT University Press,
2017), 281–293; Grégoire Chamayou, héorie du
drone (Paris: La Fabrique, 2013), trans. A heory
of the Drone (New York: he New Press: 2015).
7 Charles Baudelaire, leter to A. Toussenel,
21st January 1856, cit. in Walter Benjamin,
Das Passagen-Werk, Gesammelte Schriften,
Band V, ed. R. Tiedemann (Frankfurt am Main:
Suhrkamp, 1982), trans. he Arcades Project
(Cambridge, MA, London: Harvard University
Press, 1999), 241.
8 “Massive”, htp://www.massivesotware.com:
a sotware speciically developed for ilming he
Lord of the Rings and now employed also to
simulate the efcacy of way outs in architectural
modelling. For the work of Barrat see: htps://
github.com/robbiebarrat. An extended list
of examples is given by Glenn W. Smith and
Frederic Fol Leymarie, “he Machine as Artist:
An Introduction”, Arts 7, no. 2 (2017), doi:
10.3390/arts6020005.
9 Andrew Zhai, “Introducing a New Way
150 DAH-Journal #3
to Visually Search on Pinterest”, Pinterest
Engineering, November 8, 2015, htps://medium.
com/@Pinterest_Engineering/introducinga-new-way-to-visually-search-on-pinterest67c8284b3684.
10 Since 2014, Google has given birth to an AI
project that outputs natural language captions
for images: Simon Shallue, “Show and Tell:
Image Captioning Open Sourced in TensorFlow”,
Google Research Blog, September 22, 2016;
htps://research.googleblog.com/2016/09/showand-tell-image-captioning-open.html.
11 “Cultural Analytics Lab”, http://lab.
culturalanalytics.info.
12 Lev Manovich, “Data Science and Digital
Art History”, International Journal for Digital Art History, no. 1, 2015, doi: 10.11588/
dah.2015.1.21631.
13 Lev Manovich, “Mondrian vs Rothko: Revealing the Comparative “Footprints” of the Modern
Painters”, Cultural Analytics Lab, 2016, htp://
lab.culturalanalytics.info/2016/04/mondrian-vsrothko.html.
14 Babak Saleh, Kanako Abe, Ravneet Singh
Arora, Ahmed Elgammal, “Toward Automated
Discovery of Artistic Inluence”, Multimedia
Tools and Applications 75, no. 7 (August 2014):
3565–3591.
15 Griselda Pollock, “Computers can find
similarities between paintings—but art history is
about so much more”, he Conversation, August
22, 2014, htp://theconversation.com/computerscan-ind-similarities-between-paintings-but-arthistory-is-about-so-much-more-30752.
16 Ahmed Elgammal, “Computer science
can only help — not hurt — art historians”,
he Conversation, December 4, 2014, htp://
theconversation.com/computer-science-canonly-help-not-hurt-art-historians-33780.
17 Alan Turing, “Computing Machinery and
Intelligence”, Mind 59, no. 236 (1950): 433–460;
Donald Geman, Stuart Geman, Neil Hallonquist,
Laurent Younes, “Visual Turing Test for Computer Vision Systems”, PNAS 112, no. 12 (2015):
3618–3623.
18 Elgammal 2014, cit.
19 On these themes, see: Andrea Pinoti, “Chi ha
Computational Imagination
paura dello pseudomorfo?”, Rivista di Estetica,
no. 62, 2016, 81–98; Georges Didi-Huberman, La
ressemblance informe ou le gai savoir visuelle
selon Georges Bataille (Paris: Macula, 1995).
20 “X Degrees of Separation”; https://
artsexperiments.withgoogle.com/#/x_degrees.
See also: “Google AutoDraw” and “Quick,
Draw!”, two ludic experiments which acquire
data for improving the performance of
Google’s image recognition algorithms (htps://
aiexperiments.withgoogle.com).
21 “Iconclass”, htp://www.iconclass.nl/home.
22 “Tags”; htps://artsexperiments.withgoogle.
com/tags.
23 “Recognition”; htp://recognition.tate.org.uk.
24 See: the Gety’s “Digital Art History Initiative”
of 2014 (http://getty.edu/research/scholars/
digital_art_history/index.html); the Courtauld’s
“Digital Art History Research Group”, active
since 2016; (http://courtauld.ac.uk/research/
research-forum/research-groups-and-projects/
digital-art-history-research-group); the Frick’s
“Digital Art History Lab” (http://www.frick.
org/research/DAHL). In Italy, to my knowledge,
there is only one introductory course ofered
by Elisabeta Molteni, Maria Chiara Piva and
Stefano Riccioni at the Università Ca’ Foscari of
Venezia since the academic year 2015/16, entitled
“Digital Art History” (htp://www.unive.it/data/
insegnamento/224320/programma).
25 Johanna Drucker, Anne Helmreich, Mathew
Lincoln, Francesca Rose, “Digital Art History:
he American Scene”, Perspective [Online], no.
2 (2015), htp://perspective.revues.org/6021.
26 Pamela Fletcher, “Relections on Digital Art
History”, caa.reviews, June 18, 2015, htp://www.
caareviews.org/reviews/2726#fnr6.
27 “Fondazione Memofonte”, http://www.
memofonte.it/ricerche/collezionismo-mediceoinventari.html.
28 “Visualizing Venice”, http://w w w.
visualizingvenice.org
29 Harald Klinke’s 2017 “Coding Dürer Workshop” goes in this direction; htp://codingdurer.
de. Another interesting essay is: Leonardo
Impet and Sabine Süsstrunk, “Pose and Pathosformel in Aby Warburg’s Bilderatlas”, Computer
Vision – ECCV 2016 Workshops, ed. G. Hua e H.
Jégou (Amsterdam: Springer, 2016), 888–902.
Giacomo Mercuriali is a PhD candidate in Aesthetics at the Università degli Studi
di Milano. He his working on political iconology and the relationships between
art and political philosophy. He is also part of a research team that organises a
seminar on image theory at the Department of Philosophy in the same university
(https://ilosoiadellimmagine.com).
Correspondence email: giacomo.mercuriali@unimi.it
DAH-Journal #3 151
Keep reading this paper — and 50 million others — with a free Academia account
Used by leading Academics
Fátima Sá
ISCTE - University Institute of Lisbon (ISCTE-IUL)
Maria Antónia Lopes
Universidade de Coimbra
Guilherme Moerbeck
UERJ - Universidade do Estado do Rio de Janeiro / Rio de Janeiro State University
Carolyn Strange
The Australian National University
