Beyond the Creative Species: Making Machines That Make Art and Music

Preface

This book is about the practice, the technology, and the deeper implications of the automation of creative tasks in artistic domains. It was written primarily to provide a broad introduction to the field—which is in fact really a cluster of different fields that come together to inform this fascinating topic. It is aimed first and foremost at professionals working in creative industries who want to understand how their field might be influenced by technological advances in the automation of cognitive tasks, some part of which involves cutting through the noise we find today around the impact of AI, whichever the area. As such it is also aimed more generally at anyone interested in learning more about this area of exciting and rapid development. This bringing together of different fields, from AI, design, social theory, the psychology of creativity, creative practice research, and elsewhere, also gives the book a secondary function, to collate the breadth of relevant ideas and their interconnections in a way that may be useful for experts in any specific one of these subdisciplines. It may be especially useful for engineers and data scientists unfamiliar with work in social, psychological, and creative domains, perhaps offering some common language and reference points that can be used to work across the richly multidisciplinary communities that must come together to work in this field.

As such, this book is unashamedly jack-of-all-trades-master-of-none. Most importantly, it is only partly about the technology and what it can do. To narrow the topic to just this question would be hopelessly lacking in context; every engineer respects the importance of understanding a problem clearly before you can solve it. The question of how to make convincing or effective creative algorithmic systems is a core theme. But to answer this question merely from the perspective of algorithms, or even combining algorithms and cognitive science, is insufficient. I am certainly not the first person to stress in this context that human creative artistic activities are not only social in important ways but primarily social. This is probably the majority view. But exactly how creative practice is best viewed through a social lens remains more than a little bit challenging. To understand in any depth what people are doing (and why) when they create, and hence understand what the technical challenges are, is at least as vexing as those technical challenges themselves. This book aims to provide informed discussion across this spectrum of topics and may seem to do so sometimes at the expense of depth or coherence. There is no subject covered in this book that couldn’t have been given more time and detail. The book’s reviewers were thorough in pointing out omissions of work that they themselves considered important in their respective fields of specialist interest. I addressed some of these omissions, but I did not labor each subject exhaustively where I felt that the underlying idea had been covered.

Having thought about this subject for many years, situated in different university departments and talking with people from very different walks of life, I feel certain that covering such broad ground can be done equally well from a wide number of starting points and, for each starting point, looking in many different directions. There is no obvious order of events, and my choice of structure is undoubtedly grounded in various biases and niches of interest I have accumulated from researching computational creativity in those contexts I have encountered it, from anthropology to the science of artificial life, to music psychology, design, and creative practice research. For practical reasons, this book has a broad arc, starting and ending in more accessible and broad discussion, while digging deeper into academic depth toward the middle of the book. Its progression through the psychology of creativity, social creativity, the arts, algorithms, design, evaluation, and social impacts follows the narrative that I feel is most relevant but is certainly not the only possibility.

I have also tried not to become trapped by the shackles of one academic filter bubble over another, so wherever possible—and it has not always been easy—I’ve tried to talk in plain language and maintain some skepticism about the limits of respective fields. I’d rather write something that jumps position and risks consistency than to write something too deeply entrenched in a certain framework. At the same time, I’m liberal in drawing in diverse theoretical ideas that may not be always be very well supported by evidence, and I hope that this liberty is understood as a way of getting ideas on the table, not as a naive, unfiltered acceptance of any particular body of theory. Likewise, this book may appear to omit my own view and have no overt thesis. This is largely by design; I try to speak through others as much as possible, and I don’t believe a subject as sprawling as this can be easily tamed by any one overarching thesis. But the choice and ordering of material is mine and outlines a thesis of sorts.

These considerations can be summed up in four principles that I followed for writing this book: what you leave out can be as important as what you include; order knowledge to help others; speak through other people; work in manageable units.

It is worth commenting on the timing of writing of this book. When I pitched the concept in 2016 to MIT Press there were no full single-authored books in this field and even as recently as then, there was some question as to whether people would be much interested in, or take seriously, the idea of AI-automated creative production as anything more than speculative. The small International Conference on Computational Creativity was, in my view, the only real place to be showing work in this area, and progress wasn’t occurring at lightning pace. To my simultaneous delight and horror, as I set out to write the book in 2017, and in the subsequent two years getting distracted and delayed by grant writing, teaching, children, and creative projects, the subject exploded in a big way in the wake of the deep learning revolution, with this book still sitting in draft status. Google were making music generation tools, mainstream artists were using machine learning to compose albums, ML-generated paintings were being sold for large sums at auction, and the number of academic conferences and journals and other public forums hosting special sessions on AI and creativity had multiplied many times. My social media feeds brimmed with a stream of articles asking with tedious repetition whether an AI might be the next Bach or Picasso, and every day a new world-first in AI art or music was announced. I was happy for (if a little overwhelmed by) the constant feed of new material to reference, but also a little anxious that this ship was sailing far more rapidly than I could keep up with.

At the time of writing, the burst of interest seems to have plateaued, causing me to reflect once again on the hype-cycle that a field such as this regularly passes through, as I discuss in chapter 1. There is no doubt that with such a fast-moving subject the material in this book will need refreshing frequently, but it is also hopefully possible to recognize the continuity of many themes, tracing back to earlier work in the field. On the one hand, we are already inhabitants of the fantasy world of someone else’s future, where machines routinely perform magic; our worldview has already changed significantly with respect to our relationship to AI and the computational built environment as we adapt to it. On the other hand, machines remain brittle and glaringly incapable except in the most structured of contexts. The primary takeaway for me, becoming another principle for this book, is to avoid committing to predictions! Another useful piece of advice, from a colleague with several books under her belt, is that everyone who writes a book on a subject will write a different book that appeals to a different audience, so not to worry about getting in there first, and to focus on clearly and patiently communicating the themes I’ve developed in my work over several years.

I wish to acknowledge my potential biases. Some of the academic fields related to this topic are historically male dominated, and consequently the literature and history of the field are too. This book reflects that history, but I acknowledge my potential complicity in reinforcing a biased view of the field. The contemporary field has a greater gender diversity but still has far to go. In writing this book, I have been inspired by the work of many women, including Anna Jordanous, Anna Kantosalo, Kate Compton, Gillan Smith, Amy Hoover, and Alice Eldridge, and by the significant work of creativity researchers such as Margaret Boden and Teresa Amabile and social researchers Georgina Born, Genevieve Bell, Kate Crawford, and Lucy Suchman. The book is also grounded in a predominantly European and North American community of research and practice, and may not evenly represent the world’s contributions to this field.

Writing this book was made possible first and foremost by the work of the computational creativity research community, whose work you will find discussed throughout. This includes several people whom I’ve had the pleasure of collaborating with, and who have taught me much in the process: Geraint Wiggins, Alice Eldridge, Jon McCormack, Tim Blackwell, Arne Eigenfeldt, Philippe Pasquier, Andrew Brown, Kaz Grace, Alan Dorin, Rob Saunders, and Petra Gemeinboeck. An extra special thanks goes to Dan Ventura for more detailed feedback and ideas, and to my partner Alana George not only for putting up with the emotional roller coaster of such a big project but for some very practical formatting and document management support in its final stages. My faculty management generously supported the sabbatical break to work intensively on the first draft on the manuscript and strategically dangled carrots in wait of the final publication. During that sabbatical break I had the opportunity to visit some of the most interesting people in the field in their home environments: Douglas Eck, Jesse Engel, Bill Hsu, Andy Elmsley, Pierre Barreau, Sebastian Risi, Michael Casey, Bob Sturm, Paul Brown, Georgina Born, Alan Blackwell, Julian Togelius, and Amy Hoover. In no particular order, I’d also like to extend my thanks to the following people for, variously, their engaging conversation, ideas, and collaborations: Sam Britton, Dan Wilson, Estelle Hoen, Margaret Boden, Benjamin Carey, Aengus Martin, Craig Vear, Lian Loke, Christian Guckelsberger, Christopher Redgate, François Houle, Petros Vrellis, Palle Dahlstedt, Lindsay Kelley, Roger Mills, Linda Candy, Caleb Kelly, Deb Turnbull, Dan McKinlay, Matt Yee-King, Prue Gibson, Anna Munster, Keir Winesmith, Toby Walsh, Caroline Pegram, Justin Shave, Charlton Hill, the Sydney Google Creative Lab team, Liam Bray, Steffan Ianigro, Sam Gillespie, Angelo Fraietta, and my family. I also thank Chloe McFadden for some help with the final proofs.

Needless to say, without MIT Press the book wouldn’t have received such extensive reach, and I thank Doug Sery and Noah Springer for their ongoing support and patient responses, as well as the book’s copyeditor, Lunaea Weatherstone, and Melody Negron at Westchester Publishing Services.

This book is dedicated to my fantastic dad, Bruce—who always encouraged me to pursue my interests and who declared that this is the first academic thing I’ve written that he can make sense of—and to the memory of Harold Cohen, an innovator and a most thoughtful individual.

I Questions

1 Inklings

A machine is a small part of the physical universe that has been arranged, after some thought by humans or animals, in such a way that when certain initial conditions are set up (by humans or animals) the deterministic laws of nature set to it that that small part of the physical universe automatically evolves in a way that humans or animals think is useful.

—Emanuel Derman¹

What’s a machine?—Till further notice, it is any system that operates according to the causal laws of physics. And what are we?

—Stevan Harnad²

Visions of Machines That Make Art

Intelligent machines that behave in various ways like humans are a critical ingredient of any vision of the future, increasingly so as our social media feeds become filled with robots that can play table tennis or cause financial crashes. In some of these visions, they are nearly indistinguishable from humans, as in the replicants of the movie Blade Runner, who only fail to pass as real humans under subtle emotional and biometric analyses. Sometimes they have glaring but specific differences, as with Star Trek’s android character, Data, who, besides his robotic eyes, seems human enough and yet almost as if from a different culture that doesn’t get some of the nuances of our behavior; he thinks we’re a bit odd, as we do him. Artificial intelligence (AI) characters in popular culture abound, each used to explore a different manifestation of machine intelligence and its relation to human intelligence, from 2001’s HAL to Marvin the Paranoid Android in A Hitchhiker’s Guide to the Galaxy. A common theme, as in those first two examples, is that there is something about humans that would be very hard to replicate, not so much to do with the domains of logic, memory, reasoning, and calculation, but to do with emotion and our appreciation of beauty, wit, the sublime, and the various forms of art that we engage with. This is the stuff of interpersonal relations, culture, individual conscious experience, and, some would say, souls.

Increasingly we recognize these speculations as bearing on a very real transformation taking place around us today. Machines are getting smarter, and as they do we tend to benchmark their progress against human behavior. At the same time, we come to see that the possible manifestations of machine intelligence are far more diverse than dictated by a mere likeness to humans. Spam filters and timetable schedulers are not particularly profound algorithms, but they are nevertheless very smart; they can perform cognitively challenging tasks with autonomy of a sort. It is hard to tell quite how these miscellaneous intelligences relate to ours. In mainstream media, machines have been described, often with a fair dose of metaphorical license, as dreaming, imagining, having new ideas, and working things out.³ But even when there is a clear semblance to human intelligence—as in neural networks, which are inspired by our own biological brains and achieve human-level competence at object recognition and other tasks—the context in which these algorithms operate is so different from ours that it is hard to see the big picture. What do they know and what can they really do on their own?

This inspires excitement and terror. "Let’s make all humans redundant, brilliant! Has everybody really lost their soul?!" exclaims an anonymous reader of the UK’s Daily Mail newspaper, in response to an article about a neural network⁴ that has been trained to generate folk melodies—a fairly commonplace approach to the automated generation of music based on machine learning and big data. It is a familiar reaction, alluding to the impact of such work on both meaningful employment and the wider meaning of life. The system’s co-creator, Bob Sturm, meanwhile, is actually quite well known for writing papers that stress the lack of understanding such a big-data approach has, compared to what we humans do when we create music. For Sturm, such systems are like the fabled Clever Hans, a real horse that was claimed to be capable of doing arithmetic and other feats of abstract thought but that turned out to have learned some much simpler cognitive tricks in order to create this illusion.⁵

Many pages have been written about this coming age of machine intelligence, including the automation of jobs, the dangers of killer robots, and the singularity—a hypothesized time when machine intelligence exceeds human intelligence and thus accelerates further innovation in machine intelligence. This book is about a seemingly more niche topic—the idea of making machines that are artistically competent, productive, and even creative—and yet one that is no less important in the story of the rise and potential impact of AI. For some, art is a profound part of human behavior that could be portrayed as the ultimate hurdle in the creation of artificial humanlike intelligence. Even if it is not seen to be the most sophisticated of human behaviors, it is still suitably mysterious and has a great deal to do with that other final frontier, emotion. Those stereotypical sci-fi robots who can do everything humans can do but lack that indistinguishable humanness typified by a sense of awe, or some such phenomenologically grounded sensation, set out a kind of grand challenge to AI researchers. Will AI always be out of reach of this goal?

That is the grander theme of this book: how is AI taking on this challenge, what might the results look like, and what will this tell us about ourselves and our love of art? But machines that make art and music can refer to much more down-to-earth stuff as well. The creative industries are no less susceptible to small incremental advances in automation than areas like manufacturing and piloting aircraft. The speedup, if not the full automation, of creative tasks by technological means is nothing new. Just as virtual software instruments have streamlined the production of film music by replacing entire orchestras, driven by massive cost savings (sometimes at the expense of quality), we see many areas where applied AI might transform creative work without the need to invoke Blade Runner’s replicants. Given a moment’s thought, it can be seen that such a gradualist creep toward automating creative tasks is no less impactful and deserves equally rich analysis.

Consider two views of the rise of machine-generated music. In Ray Kurzweil’s grand vision of the forward march of AI, written in 1999,⁶ he predicts that by 2019 it would be routine for algorithmic musicians to take to the stage to perform with live musicians, and it wouldn’t be long before celebrated machine artists were autonomously producing reputable work, possibly exceeding the quality of human artists. (In the same book he predicts the simulation of the human brain to be complete by 2099.) This is a vision of machines that would be perceived as the originators of creative work, far removed from our present notion of machines as tools in the hands of a human creator. They would fall short of human intelligence but would be like humans, fitting existing human roles such as musician and artist.

More modestly, but with equal impact, a couple of years earlier the composer and producer Brian Eno reflected on his experiments with computer generated music in the liner notes to a highly celebrated album. Eno asks whether, in the future, audiences might come to treat ever-changing compositions as the norm. You mean you used to listen to exactly the same thing over and over again? he imagines his grandchildren asking.⁷ Eno envisioned a world in which the music would come out of your headphones generatively, meaning that rather than being fixed in form, musical patterns would be generated on the fly according to an algorithm. Many others since (and some before) have explored the idea of listening to music that is dynamically generated based on where you are, who you are, what you are doing, or just to introduce new variants to a compositional theme, as performing musicians routinely do. Recent excitement around the impacts of AI have seen renewed interest in this vision. High-profile tech investor Vinod Khosla, for example, has claimed that we will stop listening to precomposed music altogether, preferring instead algorithmically generated music that responds to our mood.⁸ Unlike Kurzweil’s, this vision is more in keeping with a view of human creators using computers as smart tools. Such generative variation can be done in the simplest of ways: the random function on any standard CD player enabled albums of tiny fragments that could be reconfigured randomly in a new order, the most basic of generative processes spurring new ways to think about uniqueness and variation in the listening experience. The artist Eluvium is one to have recently continued this tradition into the world of Spotify, with his album Shuffle Drones, which he invites you to play in a random order to produce a new listening experience.

Of these two views, Kurzweil’s vision would seem to be a good deal further off the mark. We are beginning to see an increasing number of systems that at least claim some form of autonomous musical and artistic competency, but they are far from common and do not come near the level of sophistication his prophecy would appear to require. That said, how far off we are is a subject of some debate. One recent appraisal, far more positive than mine, is that the state of the art in mechatronics and computation is such that we can now begin to speak comfortably of the machine as artist.⁹

Both Kurzweil’s and Eno’s visions are sociotechnological, depending not only on some technological foundations but on adoption by listeners and creators (or metacreators, those people who author the systems that do the creation). Kurzweil’s prediction is more squarely focused on the capability of the technology, but this prediction is not just about whether machines could in principle create art, but whether, and how, a culture would emerge that embraces these metacreated artifacts. This in turn depends on the specifics of the technology, which itself progresses in part according to how much demand there is for it, a cycle of feedback we will also need to take into account.

Eno’s prediction is more overtly cultural, since he was already working with generative tools at the time, proclaiming that now, there are three alternatives: live music, recorded music, and generative music.¹⁰ The cultural question is, what will this technology do to music listening culture? But even for Eno, as for Kurzweil, technological unknowns still play a part in how this path unfolds. Eno was able to make beautiful ambient music with the generative systems he had to hand, but we would still struggle today to use a machine to generate a good variant on a Talking Heads or Roxy Music tune, let alone compose an original work in that style. Some styles of music are harder to create with generative means than others. The cultural and the technological play off each other intensively. Indeed, subgenres of algorithmic music have long existed that don’t really try to recreate existing musical styles, instead exploring the creative nature of the medium, just as synthesizers, jukeboxes, and powerful PAs have changed the nature of music in their own ways.

Between the cultural and the technological, we also have that question of the interface. How should the inheritors of Eno’s vision compose their generative music, and how should their audiences consume it? If there’s no compelling process to adopt—more compelling than picking up a guitar and playing it—then adoption of such technology may be a nonstarter, even if the technology is robust and there is a cultural demand. After making generative music myself with code, with all its bugs and opacity and uncertainty, I’m sometimes very glad to go back to making normal music—that is, for me at least, arranging sounds on a timeline—where I can control what is going to happen next. These are questions of human-computer interaction (HCI), interaction design (IxD), and user experience (UX) design, which may at first sight seem to have little bearing on the real hard challenges of automation but can turn out to be pivotal in many situations.

Such questions have become consolidated into a field of inquiry called computational creativity. This is the name given to the study of the automation of creative tasks by machines.¹¹ If its interdisciplinarity hasn’t already hit you, then consider that we’ve just skimmed fleetingly past algorithms, cognition, psychology, creative practice, theories of art, anthropology, philosophy, and human-computer interaction, and there are more disciplines to add to complete the picture, such as complex systems and evolutionary biology. Computational creativity is about machines that play some role in the creation of art or other creative outcomes. For the moment, I take the gross liberty of using art here as a shorthand or placeholder for the production of a wide variety of cultural artifacts: visual art, music, stories, poems, jokes, games, furniture designs, and so on.¹² There will be some wrangling to come about where the boundary lies around this vague cluster of activities and what the correct terminology is, as well as where neighboring activities such as science, innovation and scholarship fit in.

For now, take these prototypes, some of them already discussed above, as a guide to what the subject matter is: a robot who appreciates and can discuss fine art; a machine that can sit in on bass with an improvising jazz band, playing standards with flair; a generative algorithm that can be composed (or instructed) to produce variations on a style of art or music; or an architectural modeling program that fills out odd corners of a building in a baroque style of the designer’s choosing.

Amilcar Cardoso, Tony Veale, and Geraint Wiggins, three leaders in the field of computational creativity, put it that the study of creativity in AI is not new, but it is unusual.¹³ The interdisciplinary nature of this topic introduces several extraordinary challenges, but its foundation in a computational practice has the fantastic advantage that you can actually test out your ideas by building something:

Researchers … come [to conferences] … with laptops primed to give demos of what their systems can do, ready to show off features that have been added since their papers were first accepted. No abstract insight can compare with the ability to show a real creative system in full flow.¹⁴

In this chapter, I provide a cursory sketch of the work going on in this field and the wider perception of it, followed by a summary of the challenges and questions faced by those who aspire to automate aspects of creativity. This begins with a portrait of some early work in the field, followed by a wider look at how its practice has progressed. This sketch will show some of the ways in which computational creativity is progressing rapidly in certain areas but also illuminate some of the confusing hype surrounding the field, which muddies the waters.

The central message of this book is that understanding this field requires understanding what computationally creative algorithms are actually doing but also developing a strong, multidisciplinary understanding of the domain of creative artistic activity itself, in order to properly frame and evaluate the goals and achievements of computational creativity. This will show that some of the hype around this field is valid and some is misguided, but sometimes in a way that is not expected. It is perhaps less of a problem to make the bold claim that a computational system created an artwork or composed a piece of music all by itself than it is to attach to such activities grandiose mystique or to misrepresent exactly what that task entailed in its proper context. Such activities can be seen, on the one hand, as nothing more than mundane acts of arranging pixels on a screen or notes in a timeline. To create novel configurations that conform to expectations of a particular style can sometimes be a trivial task, and I will argue, indeed, that of course computers can do this and do so very well. On the other hand, these forms of production can, and should, also be seen as deeply social activities that are grounded in identities, group membership, and social competition, which have material outcomes for human beings. To arrange pixels or notes in such a way as to achieve individual social goals, as humans do through processes deeply ingrained in our biology and culture, is a different matter altogether, one that cannot be achieved merely by training a neural network to generate patterns, even if those resulting patterns may pass as something a human would have made. The analysis I offer is one in which artistic creativity is understood as occurring in the networked interaction between multiple people and objects, not within individual humans, and where value must be seen as an emergent property of this interactive process, not bound to individual goals and expectations. Many commentators have argued that it is the social nature of artistic activity that makes it difficult for computational systems to be employed in this space in any simple way. Exactly what is social about artistic activity, why we do it and care so much about it, is an important part of this book’s analysis.

Artistically creative machines, in this analysis, will be reframed as modules that perform aspects of tasks otherwise performed by humans, perhaps in complex combinations that achieve impressive feats. They already occupy a no-man’s-land of creative agency, changing the nature of creative production, not grounded in the same needs and goals as humans, yet creatively productive nonetheless.

Early Pioneers and World Firsts

The Low-Hanging Fruit

Between the conception of this book and the moment the first words were written, one of the pioneers of computational creativity passed away. Harold Cohen had become a successful visual artist in London by the time, at the age of 40 (in 1968), he set off to California to join the Visual Arts Department at the University of California, San Diego. Here he created AARON, arguably the world’s first generative art system and arguably still one of the best. Cohen worked on AARON and with AARON for the rest of his life. I met him in 2009 at the Dagstuhl Symposium on Computers and Creativity, and at the age of 81 he was still enthusiastically reporting on new developments in his work. AARON was what we call a classic rule-based system. It performed no learning, it did not evolve or develop in any way except when Cohen tinkered with its code, nor did it perform any evaluation of its output (although this was always a fascination of Cohen’s). All it did was generate paintings according to a series of rules of composition that Cohen had programmed, first rendered to the screen and then sometimes physically painted to canvas using a robotic mechanism. Furthermore, Cohen was often involved in curating the output, selecting some works and rejecting others. Despite these various reasons that might lead us to determine a lack of creativity in this description of AARON, the works were deemed impressive, Cohen himself reported being regularly surprised by the system’s output, and both the public and expert response was general enthusiasm about the possibilities it evidenced. It was significant work and generated a great deal of excitement about the potential of machines to act creatively or artistically.

Cohen died the most celebrated and written-about computational creativity practitioner of his time, and he seems set to remain so for some time to come. He was a hero of Ray Kurzweil’s, and his work was also discussed extensively by the philosopher of cognitive science, AI, and creativity Margaret Boden. He was also a competent commentator and theorist himself, and in today’s academic language we would describe him as an exemplary practice-based researcher in the field, reflecting and commenting on his practice, and developing his own conceptual and theoretical framework to work with.¹⁵

It is historically significant that Cohen was already an accomplished artist, who made the leap from this platform into the world of computer science. Firstly, in the production of rule-based systems, the most important thing you need is an expert’s knowledge of the activity in question to turn into rules. Although Cohen’s initial vision was of a more general artistically intelligent system,¹⁶ he arrived at an approach to AARON that embodied a complementary dualism. He was engaged on the one hand in understanding, by transcribing in code, his own painting practice. Formalizing a task in the strict language of computer code offers new insights into that activity and is a form of knowledge creation in its own right. In tandem with this, on the other hand, Cohen was interested in the more general technical achievement of building machines that could create original artworks. As many others have found, this fascination with a machine that makes art is as much about understanding oneself as an artist as about understanding machines. The former acts as a complement to the latter, for Cohen not only because it was the means for technical success but because it offered him a very different formulation of what he was achieving, one that was much less about the nature of mind and more about the simple goal of understanding the creative domain of painting. This began to feed into a picture of the place of the computer in his work, framed as a philosophical question. Here, for example, he foregrounds a conceptual ambiguity that arises whenever computers-as-tools start behaving like active agents:

If a photographer takes a picture, we do not say that the picture has been made by the camera. If, on the other hand, a man writes a chess-playing program for a computer, and then loses to it, we do not consider it unreasonable to say that he has been beaten by the computer. Both the camera and the computer may be regarded as tools, but it is clear that the range of functions of the computer is of a different order to that of the camera.¹⁷

The second reason that Cohen’s artistic background is important is much more pragmatic. He knew the art world and was already widely exhibited. He was in a good position to ensure that his work with AARON should make it into art galleries and receive the attention of critics and the public. This has proven to be a recurring theme in computational creativity, where some of the more successful work has taken place in the wild, leading to questions about how the rigor of the research lab can engage with the cultural contexts in which such work has meaning.

Two other celebrated artist-programmers followed very similar paths in the wake of Cohen. Both musicians, David Cope and George Lewis were, like Cohen, already recognized talents in their respective genres before embarking on (very different) studies of musical computational intelligence. Lewis’s Voyager system can be seen as the jazz-improvising equivalent of AARON’s painting, handcrafted from Lewis’s formal and intuitive musical experience without doing any learning or self-modifying or performing any sort of reflection. Although an improvising system, Voyager does have one responsibility that AARON is free of: it must listen and respond dynamically to other musicians.

As a jazz and improvised music lover, I delight at watching Voyager in action.¹⁸ Even knowing what Voyager is doing under the hood, knowing that its algorithms are conceptually relatively simple, watching it in action interacting with other improvising musicians still creates a sense of awe. There is a richness of output and a seeming conversation going on between musician and machine. Voyager appears to develop musical themes in interesting and appropriate ways and to respond meaningfully to performers. Like Cohen, Lewis has nurtured a kind of co-creative relationship with the system as part of his lifelong creative practice. It is a relationship that is very one-sided, that makes significant creative progress only when the human hacks away at the code, but a relationship in which the machine can act as a prolific mass-producer beyond the capacity of the artist, and in its own small way act as a fruitful originator of ideas.

David Cope’s work developed throughout the 1980s and 1990s and, like Cohen, was situated in California’s innovation-rich culture. He too used his compositional insight to develop an expert system for the automated generation of music¹⁹—sheet music this time, for performance by orchestral musicians—and with a focus on style imitation. He worked with a corpus of machine-readable annotations of music by various composers in order to extract these composers’ melodic signatures. He also created generative grammars by hand that captured higher-level aspects of their style, guided by his own musicological framework based loosely on a well-known theory of music called Schenkerian analysis. Thus while Cope followed the same trend as Cohen and Lewis of a co-creative relationship with his own software construct—hacking away, guided largely by expert intuition, reflecting, curating, tweaking—his work also took on a more musicological slant by aspiring to imitate great composers.

Possibly for this reason, Cope took on a slightly more confrontational role in the early philosophical discussion of art-making machines. Because his work was concerned with style imitation, it was positioned to pose the tricky philosophical provocation, for the first time, that a machine might create works of great composers as if they were still alive today. This was unsurprisingly not a palatable concept to many scholars and lovers of classical music. In addition to suggesting that computers could create music, an already challenging idea, Cope’s work could be read as suggesting that we could in effect bottle the essence of Bach and Beethoven. He also tended toward statements that heightened the provocative nature of his work:

Much of what happens in the universe results from recombination. The recombination of atoms, for instance, produces new molecules. … Music is no different. The recombinations of pitches and durations represent the basic building blocks of music. Recombination of larger groupings of pitches and durations, I believe, form the basis for musical composition and help establish the essence of both personal and cultural musical styles.²⁰

For some of his musicologically expert critics, such statements have read as oversimplification, and the limits of the system’s style-imitation were more visible to them than for general audiences, who were on the whole not expert enough to tell the difference between the system’s imitations and their original counterparts. Cope’s presentation of his work was also more dramatic overall. He tells the story of the creation of Emmy (the name derives from the more formal acronym EMI—experiments in musical intelligence) as one arising from a writer’s block. In the dramatic prose of one journalist:

In 1980, Cope was commissioned to write an opera. At the time, he and his wife, Mary … were supporting four children, and they’d quickly spent the commission money on household essentials like food and clothes. But no matter what he tried, the right notes just wouldn’t come. He felt he’d lost all ability to make aesthetic judgments. Terrified and desperate, Cope turned to computers.²¹

Accounts vary on how much of a role Cope and Emmy both played in the creation, curation, and manipulation of the resulting works. With the objective merely of breaking his writer’s block, there was no need for the system to compose complete works, only to generate workable material for Cope to play with; this is definitely how it was used in early works. At other times, the system is presented as one that operates autonomously in the creation of complete works, and Cope has produced a body of five thousand pieces composed by Emmy, quite the proof that his block was overcome.

In addition, various altercations with reviewers, record labels, and other gatekeepers led Cope to claim that there was a strong subconscious bias in people against the idea of computers being able to create meaningful music.²² He became a champion for the view of AI optimists like Kurzweil that what people think is the preserve of a human soul, too profound to be ever explained, is in fact perfectly modelable. He argued that people were prone to significantly overemphasize the mystery of music and reject Emmy’s work outright on essentialist grounds.

Another voice in this early conversation, Douglas Hofstadter, wrote the following in his earliest and most famous book, Gödel, Escher, Bach:

Question: Will a computer program ever write beautiful music?

Speculation: Yes, but not soon. Music is a language of emotions, and until programs have emotions as complex as ours, there is no way a program will ever write anything beautiful. There can be forgeries—shallow imitations of the syntax of earlier music—but despite what one might think at first, there is much more to musical expression than can be captured in syntactical rules.²³

But Hofstadter later claims that Cope’s work made him question this view. And what do I make of it now? Well, I am not quite sure, he mused, explaining his surprise as he examined Cope’s creations and started to concede that mere syntactical rules might go further than he had initially imagined.²⁴

It is fair to say, without any cause to dismiss their contribution and the brilliance of their work, that early figures like Cohen, Cope, and Lewis were at the right place at the right time to pluck the low-hanging fruit of computational creativity, looking at the new world of personal computers through the lens of their artistic expertise and asking a simple question: can I formalize some of my creative production, or that of others, in algorithms? In a certain light, the answer was a resounding yes; algorithms were producing novel outputs, and audiences were engaging.²⁵ But on closer inspection, these systems posed more questions than they answered, and the ways in which they were lacking began to outweigh their accomplishments. These initial experiments became the platform for further questions and considerable technical problems. Kurzweil, ever the optimist, saw systems such as AARON and Emmy as evidence of his vision of proliferating autonomous computer artists. But hindsight hasn’t been so kind. Looking at where Kurzweil predicted the evolution of the technology would be at this stage, we do not seem that much closer to building systems that have the ability to do things like iteratively evaluate what they are doing, let alone the full trappings of a culturally embedded social artist.

This is a familiar story in technology. Over the past few years virtual reality (VR) has been enjoying a second wave. I remember seeing the first incarnation as a teenager, a VR game with enormous head-mounted displays and jagged-line graphics, at the Trocadero Centre in London in the 1990s. The hype was ecstatic, but the VR bandwagon was not going anywhere. The technology just was not good enough to create that rich immersive experience that makes VR worth caring about. It took twenty years to be revived, during which time processors, graphics, and sensor technologies got cheaper, smaller, and more reliable, with more established ecosystems of creators and creative tools to jump into action when the market was ready. This process of expectation-based boom and bust in technology is widely documented and has been neatly expressed by the hype cycle, a model of technological maturity that predicts a period of initial hype, followed by a depression, a regrouping around more realistic technological expectations, and then a steady maturation of a workable technology. The hype cycle, developed by research firm Gartner, describes how some killer demo of a technology’s potential can create overexcitement and overinvestment in a field that still has a long way to go toward a practical product.

It is not really appropriate to locate computational creativity itself on this cycle, because it is a mélange of different technologies with different objectives, but there is nevertheless some sense that the field as a whole experienced its initial hype phase in these early days. The doors were pushed ajar in the late twentieth century to offer glimpses of a new world of possibilities, but we are still grappling with challenges spanning the technical, the methodological, the social, and not least the reaction to it in our culture. Now computational creativity is experiencing a second burst of activity driven by new advances in machine learning, which will be discussed at length in this book.

It is worth noting too that a similar such lull occurred in the wider field of AI research, dubbed the AI winter, an era of relative doubt about AI’s prospects, accompanied by reduced funding, at least in the US. AI chronicler Pamela McCorduck comments that all sciences move in rhythms, thriving, then lying fallow (or simply assimilating what has gone before), then growing once again. In the 1990s, shoots of green broke through the wintry AI soil.²⁶ Now the tables are turned. AI is booming, and funding is currently doubling every two years.²⁷ It has also become a buzzword in business, and