The Hidden Sense: Synesthesia in Art and Science

1

Introduction

How does it feel to hear music in color, or to see someone’s name in color? These are examples of synesthesia, a neurological phenomenon that occurs when a stimulus in one sense modality immediately evokes a sensation in another sense modality. Literally, synesthesia means to perceive (esthesia) together (syn).

When synesthesia is mentioned in the media, it is usually described as a neurological defect. Similarly, neurologists consulted by synesthetes often inform synesthetes that their synesthesia is probably a congenital brain defect. They explain that in synesthesia, regions of the brain that normally do not communicate, such as the visual and auditory cortexes, show signs of what is known as crosstalk. As a consequence, these synesthetes experience the world in a different way from the rest of us. For them, it is hard to imagine that others cannot hear music in color; they wonder what it must be like to see someone’s name without colors.

When I first read descriptions of synesthesia, I was immediately drawn to the phenomenon. How nice it must be to put on a CD, relax in an armchair, and see as well as hear beautiful music pass by in fascinating images. Or to go for dinner in a restaurant to try out not only new tastes but new colors as well.

I did not then, nor do I now, see music in colors. At that time, I thought of synesthesia as a skill or a trait possessed by a small and special group of individuals who perceived other dimensions of reality.

I wanted to meet these synesthetes. In some respects, they seemed almost like extraterrestrials to me. Did they perceive a different world from the one I saw? Was a cloudless sky not blue for them? Did they live in another reality, with colored music played on harps by cherubs? Did they taste wine in a bouquet of a thousand flowers that transported them to a vision of glimmering ballets and shimmering, rustling gossamer?

The synesthetes whom I met told me that other combinations of sensory impressions occurred for them as well. Some synesthetes perceive tactile forms and textures when tasting food, while others hear sounds from the smell of fragrances. Cases have been reported of synesthetes who feel colored pain, hear odors, hear tastes, taste sounds, feel sounds on their skin, hear images, and taste images. In addition, some synesthetes respond to symbols with their senses. In fact, the most commonly reported type of synesthesia is hearing or seeing letters in color. Vowels, and often consonants, too, have very specific—and fixed—colors for the synesthetes who see letters in color. For the synesthete Katinka Regtien, for example, the vowel E is not simply red but a specific translucent red with a hint of orange. In collaboration with the designer Beata Franso, she created a painting of her colored alphabet. Figure 1.1 (see plate 1) is an approximate representation of her letter colors. In reality, however, the letter colors of synesthetes are so specific that they often find it difficult to reproduce or describe them; this may be because the colors they see are colored light rather than the colored pigment.

For the synesthetes who see colored letters, the colors normally remain the same throughout their lifetime, though older synesthetes recount that the colors sometimes become paler in their later years; they remember the colors being brighter in their youth. The colors are so obvious to them that young synesthetes believe that everyone sees letters in color; many synesthetes only discover later that this is not the case. A common response when that occurs is: Gosh, I didn’t know this experience had a name, I always thought everyone has it.

Sylvia Roukens received an inkling after she handed in an exercise in elementary school. As a seven-year-old, she wrote the following little story in her school exercise book:

The Magic Butterfly and the Alphabet

Once upon a time, there was a butterfly. However, it was not a common butterfly, it was a magic butterfly. One day he thought it would be nice to fly around. Then he met the letter A. The A was yellow, but the butterfly didn’t like it, and so the butterfly changed the yellow A into a red A. Then he flew on. Then he came across a green B. The butterfly did not like that either and changed the green Bto a purple B. And so it went, until the butterfly had been round the whole alphabet. Then he was tired and when he got home, he fell asleep at once. In the middle of the night, the whole alphabet came to the butterfly’s house. Why have you changed our colors! they shouted angrily. All right, I will put your colors back, the butterfly said. The end.

Figure 1.1 Synesthete Katinka Regtien perceives the alphabet in colors and spatial forms. In collaboration with designer Beata Franso she created this visual representation. (Reproduced with permission.) See plate 1 for color version.

In the exercise book, the teacher wrote that she had found it a very original story, but Sylvia didn’t understand her teacher’s remarks, because to Sylvia it was not that strange. It was only later, when she discovered that not everyone sees the letter A in yellow and the letter B in green, that she understood her teacher’s remark. When she was about fifteen, she read the French poem A noir, E blanc, I rouge, U vert, O bleu (A Black, E White, I Red, U Green, O Blue) by Arthur Rimbaud. Did he have the same experience as she did? But if so, why did he see other colors for the letters? She decided to do some research on the subject for a school essay, and that was when she discovered that this phenomenon had a name: synesthesia.

Over the past few years, many synesthetes have told me about their colorful experiences. These people were not the extraterrestrial dreamers full of wild fantasies that I had originally pictured. Why did I imagine them that way? Is it because synesthesia comes dangerously close to fantasy? Or was I skeptical because there is no shortage of people who, yearning for attention, try to obtain it with spectacular stories?

It is probably for both these reasons that for a long time, synesthetes were not taken seriously by researchers. Synesthesia was considered a folly. When synesthetes insisted that letters have colors, researchers attributed it to their strong imagination—even when the person concerned showed little imagination in other areas of life. In other cases, it was felt to be a learned association: as children, these people had probably played with colored letter blocks and now they were remembering the colors of the blocks. This latter explanation does not stand up for long, though, when one considers that virtually every child has played with colored letter blocks, yet only a few become synesthetes. Also, synesthetic brothers and sisters who certainly played with the same letter blocks generally perceive the same letters differently, that is, they see them in different colors.

Another frequently heard explanation for synesthesia is that the colors of letters are not perceptions but are rather a type of associative metaphor. The word sea would thus be associated with a blue color because the word evokes an image of the sea for the inner eye. However, a synesthete may tell you that the word sea has red, yellow, and purple colors. The letter Smay be red for this person, the letter Eyellow, and the letter Apurple. The fact that for the synesthete there is a separation of the word into colored letters makes it clear that it is not the meaning but the physical appearance of the word that evokes the colors. Many synesthetes perceive the colors of words and letters only when they read them in written or printed form.

Brain scans of synesthetes have finally removed the doubts of the skeptics. They provide proof of the neurological existence of synesthesia. Experiments that compared the brain activity of synesthetes with that of nonsynesthetes reveal that there are neurological differences in their responses to the same stimulus. In one test, a synesthetic person was blindfolded and placed in a recording tunnel of the brain-scanning apparatus and wore headphones that produced spoken words at regular intervals. Figure 1.2 shows the results: activity in the areas of the brain responsible for hearing and color vision occur simultaneously when a blindfolded synesthete hears a word. Under the same conditions, the brains of nonsynesthetes generated activity only in the areas known to be responsible for hearing. These experiments represented a breakthrough in the study of synesthesia. Almost two centuries after the first scientific descriptions of the phenomenon, physiological evidence had been found that left little room for doubts: synesthesia exists!

Figure 1.2 This brain scan taken from the right side of the head of a blindfolded synesthete shows activity in the color vision center of the brain at the back of the head (left) when she hears words. This activity is absent in nonsynesthetes. (Aleman et al. 2001. Used with permission.)

How can we explain neurological synesthesia? After reading a number of neurological studies on the subject, I thought I knew the answer, but after interviewing synesthetes during the last ten years, I became unsure; I recognized that the scientific descriptions were not always compatible with the stories of the synesthetes I met.

For example, a well-known theory is that synesthesia is a brain impairment that causes a kind of short circuit between the areas of the brain that process colors and sounds. Yet calling it an impairment implies that it is a process against nature; an impairment suggests something that does not work properly. In the language of a mechanic, it refers to an electrical breakdown that needs to be repaired. But do synesthetes want or need to be repaired or helped? Do they suffer as a result of their brain breakdown? Do they experience synesthesia as a disability that impairs them in their daily lives?

Synesthetes often tell quite a different story, saying how useful they find their ability to see letters and numbers in colors, for example, when they have to remember names or telephone numbers. Some look pityingly at nonsynesthetes who have to live without those beautiful colors and patterns in music. Synesthesia actually offers them advantages in their daily lives. Now that scientists have provided the scientific proof of the existence of synesthesia, synesthetes have become a rich source of information on the advantages this perceptual ability offers them.

Does synesthesia have a function? Of what use is it? What are its benefits? We know that our five senses all have functions in our perception; hearing, for instance, is responsible for detecting relevant sound patterns in our surroundings. We also know that a loss of hearing can create danger. But what is the function of synesthesia in the perception of our environment? Similarly, does a lack of synesthesia cause harm?

What makes synesthesia such a fascinating phenomenon is that it raises questions that scientists cannot answer at present. Synesthesia is not an isolated phenomenon in human perception. It is not a fantasy, nor can it be marginalized as an unimportant by-product of a human brain process gone awry. The synesthetes to whom I have talked regard it as essential in their lives. And since the phenomenon really exists (as has been demonstrated), studying synesthesia just might turn our common image of the senses on its head. Reorganizing our concept of the sensory channels of the mind can change our view of the human mind, and possibly of the physical world.

Looking back, I would say that in my explorations of synesthesia, I allowed intuition to prevail over reason. Put simply, I could not reconcile myself with the idea of the five senses and synesthesia as merely an aberration created by a neurological breakdown. So I set out on a tour that might lead me into a murky, marshy area between the senses, an area that had as yet hardly been explored. For over two centuries, scientific disciplines have focused on just one of the sense organs at a time: the eyes, the ears, the tongue, the nose, and the skin. Rarely has scientific inquiry concerned itself with all the senses at the same time. The same can be said of the arts. Music and the visual arts have produced experts and critics for centuries, but the domain connecting music and the visual arts has been increasingly explored only in recent decades.

Driven by curiosity, I embarked on a quest to explore synesthesia. As I went along, the object of my study puzzled me and gave rise to all sorts of questions. How are my senses organized? Do they work in the way that I thought they worked? Looking back at my naïve ideas at the beginning of my journey, I now realize that it is not only my ideas about the senses that have changed, but also my senses themselves.

Like a nineteenth-century armchair scholar, I let my reading set my mind adrift, and searched for my answers like a classic naturalist for examples of this special species. I explored the world of synesthetes in the same way I enjoy roaming through a city center, venturing on a whim or intuition into alleys and stumbling across unexpected views of the city. When I search cities, I first read the guidebook word for word; but when I embark on the actual search, I leave it in my pocket, since it can only lead me to known locations. I have explored the phenomenon of synesthesia in the same way. Having first done extensive reading on the subject, I wanted to be surprised by what I might encounter. I relied on my instincts and intuition, gathering information by talking to people who referred me to other routes and other people I would talk with.

In addition to looking for an answer to the basic question of what is synesthesia? I was guided by a second question: What is the significance of synesthesia in people’s daily lives? In other words, is their synesthesia useful? Do synesthetes benefit from perceiving synesthetically, or does synesthesia only bring them confusion or harm? My explorations led me along many unexplored paths and brought me into contact with many different people, including synesthetes, children, educators, neurologists, psychologists, philosophers, artists, poets, dandies, and drug users. In the end, their hints brought me to an unexpected but well-known source: the hidden sense.

I

Perception

2

Music Video Clip without TV

This exploration starts with the experience of synesthesia. How does it feel to be a synesthete? How does a synesthete perceive the world? For instance, how does it feel to see both your visual surroundings and ambient music in color?

When it is silent, I see a black space, somewhere at an angle above me, but it looks different from the things that I perceive with my eyes. The forms that I perceive are often colored lines that disappear from the left and right of the image. Some sounds sculpt the lines into three-dimensional forms giving them depth, so to speak. Not all lines acquire depth. For instance, I see plop-sounds as circles, which is entirely logical, of course. That’s why I always thought that everyone perceived sounds in this way. All of them look so logical.

For most people, it is not logical at all to perceive images in front of their eyes when listening to music—unless they are watching a music video clip on TV. However, Patrick Heller does see images as soon as he hears music or sounds. He doesn’t need to watch MTV to have the most beautiful video clips presented in front of him.

The imagery made by Patrick’s brain differs from that of regular MTV video clips, in which singers, musicians and dancers star. Patrick’s mental video clips consist of moving colored lines and forms that look more like abstract art in motion. On a Dutch radio show, Patrick described his images live while listening to a fragment of ambient music played on a synthesizer:

I see dark red three-dimensional bars, flat and wide, in the distance: I see the bars or strips sideways and they run from left to right, and I don’t see their ends. In fact, the bars are dark red but in the distance almost transparent.

Also, in Patrick’s perception people’s voices have colors, ranging from brown to yellow. A low voice sounds brown while a high-pitched voice sounds yellow. The voice of his girlfriend is described by Patrick as orange-yellow when she talks, but when she laughs the sound turns to a dark blue color.

Patrick notices that the source of the sound mainly determines its color and form. Voices have brown-yellow nuances, laughter is blue, and in music, the instruments account for their colors. Sounds of synthesizers are often red-orange and guitar sounds have blue-white tones. He perceives the sounds of synthesizers as wide bars and that of electric guitars as many thin lines. He assumes that this difference has something to do with the fact that synthesizer sounds are constructed from one or only a few frequencies, whereas guitar sounds have background