Stanford_p18-25:E 8-15 Human vs Machine 10/2/06 2:24 PM Page 16

MICROARRAYS:
The Search for

In a Vast Sea of Data

BY KRISTIN COBB, PhD

16 BIOMEDICAL COMPUTATION REVIEW Fall 2006 www.biomedicalcomputationreview.org

Stanford_p18-25:E 8-15 Human vs Machine 10/2/06 2:24 PM Page 17

W hen DNA microarray technology emerged

more than a decade ago, it was met with unbri-
dled enthusiasm. By allowing scientists to look at
the expression of enormous numbers of genes in
the genome at once, microarrays promised to revolu-
tionize our understanding of complex diseases and
usher in an era of personalized medicine. Advocates
vowed that, someday, with just a finger prick, doctors
would instantly know whether patients were having
a heart attack, rejecting a transplant, or in the early
stages of cancer based on their mRNA patterns, and
would tailor treatment accordingly.
In the decade since their introduction, microar-
rays have permeated virtually all corners of bio-
medical research; have yielded some useful insights
into basic biology and cancer; and are being used,
in a preliminary way, to diagnose disease, guide
treatment, and streamline drug discovery. But
early enthusiasm has been tempered with a dose
of reality. Progress has been slower than predicted.
And some splashy results in high-profile journals
have proven difficult to reproduce, casting a shad-
ow over the real successes.
The shift in perception is palpable in the literature:
a 1999 Nature Genetics article was entitled “Array of
hope,” but a 2005 Nature Reviews article was entitled
“An array of problems.”1,2 One recent paper called
microarray studies a “methodological wasteland.”3

www.biomedicalcomputationreview.org Fall 2006 BIOMEDICAL COMPUTATION REVIEW 17

Stanford_p18-25:E 8-15 Human vs Machine 10/2/06 2:24 PM Page 18

All new technologies have growing multiple genetic mishaps. They may They said, ‘Wow, look at this—breast
pains, and early glitches with the technol- also yield improved tests for diagnosis cancer isn’t really breast cancer, it’s
ogy itself are partly to blame. But the big- and prognosis. many types of breast cancer,’” says
ger problem is more fundamental. The In a 1999 paper in Science, Golub Gilbert Chu, MD, PhD, professor of
huge promise of microarrays is that they automatically and accurately classified medicine and biochemistry at Stanford.
give information about every gene, but leukemia patients into the two main sub- “But if you talk to anyone who’s been a
this is also their huge curse—a crushing types of the disease using only gene clinician for many years, they already
onslaught of data. A decade ago, these expression patterns.5 Though these two knew this. They’ve seen breast cancers
data were a mismatch with existing statis- forms of leukemia were already well rec- that looked the same but in some cases
tical tools. Today, there is still no consen- ognized and characterized, in principle vanished with chemotherapy and in oth-
sus on how to analyze and interpret this strategy could uncover previously ers did not. So it’s not a surprise that the
them. A 2005 survey of microarray users unknown subtypes of cancer. gene expression profiles are proving that
concluded that, “Data interpretation and Indeed, in a 2001 Proceedings of the these cancers are different.”
bioinformatics remain the major hurdles National Academy of Sciences (PNAS) A natural extension of this work is to
in microarray technology.” 4 paper, researchers identified five isolate the particular genes and expres-
unique gene expression patterns in sion patterns that are linked to progno-
EXPONENTIAL ADVANCE breast cancer and showed that these sis. For example, Golub derived a 13-
Microarrays capture a snapshot of subtypes were five distinct diseases with gene expression signature that correlated
which genes are turned on—or different risks of progression.6 with survival in lymphoma patients; sev-
expressed—in a given cell at a given time. “This surprised the lab scientists. eral other groups have isolated gene sig-
Before 1995, scientists could
only explore the activity of a few
genes at a time. Then two groups
of researchers—at Stanford
University (led by Patrick
Brown, MD, PhD) and at
Affymetrix—scaled this up thou-
sands-fold with the invention of
the microarray. The Stanford
microarray is a glass slide coated
with a grid of thousands of
microscopic spots—each corre-
sponding to a gene—that light up
to show which genes are on,
which are off, and to what
degree they are being expressed.
“At the time it just blew away
the next best thing that you
could do. It was really a quan-
tum leap ahead,” says Todd
Golub, MD, director of the
Cancer Program at the Broad
Institute of Harvard and MIT.
Besides expression microar-
rays, genotyping microarrays
are becoming increasingly pop-
ular—these reveal variation in
the DNA code rather than in
gene activity. Scientists are also
working on microarrays that Above: This schematic portrays an experiment using a spotted or cDNA microarray, which consists
use antibodies to detect pro- of a grid of thousands of microscopic spots on a glass slide; each spot contains cDNA probes for
teins, but these have even more a gene. Here, researchers extract mRNA (made when genes are active) from two types of cells—
technical challenges. e.g., tumor cells and control cells—and then label the samples with different fluorescent dyes.
When washed over the microarray, these colored transcripts bind to their complementary probes,
ARRAY OF HOPE leaving a trail of informative spots: red for genes turned on in cancer cells, green for genes turned
Microarrays are ideally suit- on in normal cells, yellow for genes turned on in both types of cells—with more intense color indi-
ed to study cancer, a disease of cating higher gene activity. Courtesy of The Science Creative Quarterly, artist: Jiang Long.

18 BIOMEDICAL COMPUTATION REVIEW Fall 2006 www.biomedicalcomputationreview.org

Stanford_p18-25:E 8-15 Human vs Machine 10/2/06 2:24 PM Page 19

natures for breast cancer prognosis. gene codes for a protein that prevents of molecular and experimental medi-
These signatures can be used in prog- tumor cell death; blocking this protein cine at the Scripps Institute in San
nostic tests that gauge if a tumor should might boost chemotherapy response. Diego is working on developing a
be treated aggressively. Microarrays may also help tailor a microarray-based test to quickly tell him
Prognostic genes may also point to treatment to the person, not just the dis- if a kidney transplant patient is in acute
novel drug targets. For example, in a ease, Chu says. He has identified pre- rejection, chronic rejection, or good

The huge promise of microarrays is that they give information about

every gene, but this is also their huge curse—a crushing onslaught of data.
recent paper in the Journal of Clinical liminary gene signatures in the healthy condition. And he envisions an even
Oncology, Elaina Collie-Duguid, PhD, cells of cancer patients that predict more sophisticated personalized medi-
research fellow at the University of which patients will suffer serious side cine scenario: “What we ultimately
Aberdeen in Scotland, identified one effects from radiation therapy. want is where the doctor says to these
gene that had a 50-fold higher expres- In addition to oncology, microarrays patients, ‘I saw you on Thursday and
sion in lung cancers that were not are also being widely applied in heart you’re doing well but your gene expres-
responsive to chemotherapy compared disease and transplant research. Daniel sion analysis tells me you need more
with those that were responsive.7 The R. Salomon, MD, associate professor immunosuppression, so I’m increasing
your dose. Come back and see me in
four weeks and we’ll draw blood and
check your immunosuppression again.’”
Microarrays are also critical in basic
biological research that may ultimately
have a clinical payoff. For example, by
mapping the precise genetic program of
embryonic development, Wing Hung
Wong, PhD, professor of statistics and

Left: The oligonucleotide array (such as the GeneChip from Affymetrix) uses probes that are short strands of DNA synthesized directly on
the chip, and a single type of cell is examined at a time. Courtesy of The Science Creative Quarterly, artist: Jiang Long. Right: InkJet Array.
Agilent Technologies, Inc. uses inkjet printing technology to synthesize oligonucleotide probes on a chip. Courtesy of Agilent.

www.biomedicalcomputationreview.org Fall 2006 BIOMEDICAL COMPUTATION REVIEW 19

Stanford_p18-25:E 8-15 Human vs Machine 10/2/06 2:24 PM Page 20

of health research and policy at built to analyze data on a few variables reflect real biological changes, but many
Stanford, may be drawing a blueprint for measured on many samples. In microar- more will be false positives.
where and when to deliver genes for rays, the situation is reversed: tens of “The problem here is a deeply pro-
gene therapy. thousands of genes are measured on just found statistical one,” Chu says. “The
a few samples—a phenomenon statisti- very nature of microarrays is that they
ARRAY OF PROBLEMS cians are dubbing “p bigger than N” (p give you tons of data, and very unusu-
The initial successes in microarrays features on N samples). al patterns can emerge that are not
and their exhilarating promise set off a To illustrate the difficulty, imagine anything more than noise and statisti-
dizzying flood of microarray studies: that you randomly divide 50 people cal fluctuation.”
fewer than 100 publications in 1999 into two groups and start endlessly As Engel describes it, “What we’re all
measuring their characteristics: doing is we’re taking a statistical
age, hair color, favorite food, approach and we’re all trying it every
“The statistics and how you height, weight, and so forth. which way. And you’ll even get a pattern.
Eventually, you will find char- But is that pattern real? That’s the major
analyze the data are still a acteristics that are slightly issue for gene chips—is it real?”
imbalanced between the two Subtle statistical mistakes lead you to
quagmire,” says Greg Engel. groups just by chance. And the find patterns and get published in high-
more variables you consider, profile journals, Chu says. And it may
the more differences you will take years and several expensive follow-
grew to more than 6000 in 2004. find. But the pattern of characteristics up studies for anyone to realize that the
Suddenly investigators were identifying a that separates the two groups is an idio- finding is not reproducible, unless some-
molecular signature for every disease. syncrasy of the sample and has no larger one spots the error sooner.
But many publications have since meaning. The same thing happens Unfortunately, such sleuthing isn’t a job
been discredited or have simply fizzled when you compare 36,000 genes for the casual scientific reader, Chu says. As
out. Scientists say it’s hard to find stud- between two sets of 25 cellular samples— a perspective in Nature Genetics quips, this
ies that have led to anything concrete. some differences in expression may task requires “forensic statisticians.”8
“The thing that’s surprising to me is
that it’s taking so long to figure out
whether and when the technologies
work, and it’s taking so long in the face of
such enormous enthusiasm,” says David
Ransohoff, MD, professor of medicine
and epidemiology at the University of
North Carolina, Chapel Hill.
Of the many factors at work—includ-
ing initial snags with the technology—
scientists consistently point to data
analysis and interpretation as the critical
stumbling block.
“For the part of running the experi-
ment, microarrays seem to be working
pretty well,” says Stanford cardiologist
and medical fellow Greg Engel, MD.
“The informatics part is a whole other
area. The statistics and how you analyze
the data are still a quagmire.”
“I think the greatest challenge at this
time remains data interpretation,”
Golub agrees.
When the data amounted to whether
a single gene was on or off, biologists
had little need for statisticians. But find-
ing patterns in the activity of 36,000
genes is fundamentally a statistical prob-
lem. When microarrays were intro-
duced, even the statisticians were Grid of Lights. Picture of an expression microarray. Courtesy of: Colin Smith, Functional
stumped. Existing statistical tools were Genomics Laboratory, University of Surrey.

20 BIOMEDICAL COMPUTATION REVIEW Fall 2006 www.biomedicalcomputationreview.org

Stanford_p18-25:E 8-15 Human vs Machine 10/2/06 2:25 PM Page 21

“Again, the whole thing fell apart like a

house of cards,” he says. “I also had
other colleagues look over my analysis,
and they all agree with me: these data
look like noise.”
“On the broader issue, I think proba-
bly a good portion of microarray analy-
ses are wrong,” he says.
A 2005 Lancet paper confirms his sus-
picions.10 Stefan Michiels, PhD, and
colleagues at the Institute Gustave-
Roussy in France re-analyzed data from
the seven largest published studies to
report gene expression signatures for can-
cer prognosis. The papers were published
in top peer-reviewed journals, including:
Nature, PNAS, the Lancet, and the New
England Journal of Medicine.
For each of the seven datasets,
Michiels’ team randomly selected 500
training sets of different sizes; then they
built and tested 500 models. What they
found: The results were highly depend-
ent on the choice of training set. Every
different training set led to a different
molecular signature. Moreover, in the
Making of a microarray: A robot spots a glass slide. Courtesy of NIAID Microarray majority of trials the signatures selected
Research Facility. in the training set had poor or no dis-
criminatory ability in the validation set.
FORENSIC STATISTICIANS The authors of the lymphoma study Their conclusions: five of the seven stud-
At the request of a colleague, Robert had fit a model (a gene signature) using ies did not classify patients better than
Tibshirani, PhD, professor of statistics half the data and found that the model chance, and the remaining two did only
and health research and policy at performed well when tested on the slightly better than chance.
Stanford, set out to evaluate a 2004 remaining half. But when Tibshirani “The original investigators may have
paper in the New England Journal of simply swapped these training and test consciously or unconsciously reported
Medicine that reported a novel gene sig- sets and applied the model-fitting pro- the best performing pair of training-vali-
nature for predicting survival in follicu- gram to the new training set, unexpect- dation data,” explains John P. A.
lar lymphoma.9 edly the model did not pop out. In fact, Ioannidis, MD, PhD, professor and
Using the data the authors provided no models popped out, suggesting that chair of the department of hygiene and
online, Tibshirani spent two grueling their whole finding was spurious. epidemiology at the University of
weeks reconstructing the steps of their He also re-ran the computer program Ioannina in Greece, who wrote a com-
analysis and writing a computer program on the original training data with tiny mentary for the Lancet paper. “I suspect
that reproduced their results. Then he changes in the choice of parameters. that they probably had some source of
put their approach to the test. selection bias somewhere in the
To help determine whether a pattern process,” he says.
is real or just random noise, statisticians “On the broader To guard against bias, he recom-
use a trick called split-sample validation: mends using a repeated sampling
they fit a model only on a portion of the
dataset (called the training set) and then
issue, I think probably scheme like that in the Lancet paper or
having independent groups do the train-
test its discriminatory ability on the ing and validating steps.
untouched data (called the validation or
a good portion of
test set). If the model only fits noise in RE-INVENTING STATISTICS
the training set, it will usually fall apart microarray analyses The field of statistics moves more
when applied to the test set. But even slowly than biology, Tibshirani says.
this isn’t perfect, because bias can be are wrong,” says So, as the microarray technology
introduced in choosing the training set raced ahead of the analysis tools, non-
and specifying the model. Robert Tibshirani. statisticians made up their own statis-

www.biomedicalcomputationreview.org Fall 2006 BIOMEDICAL COMPUTATION REVIEW 21

Stanford_p18-25:E 8-15 Human vs Machine 10/2/06 2:25 PM Page 22

tics to fill in the gaps—which explains biologists wouldn’t even want to talk to basic problem: “Fancy math can’t undo
a lot of the statistical flaws. you if you were a statistician. But now biases that have been hard-wired into
“You don’t see people without train- the biologists all realize that statistics the data from fundamental errors in
ing going into labs and doing test-tube has something to offer. It’s really raised clinical study design,” he says.
experiments. Yet, anybody who has a PC the profile of our field.” “This is not fancy molecular stuff,
with Excel thinks they can invent statis- its basic study design that goes back to
tical methods,” he quips. BACK TO BASICS the 19th century. If case and control
Fortunately, statistics is beginning to Even if the statistical analyses are per- samples are not maintained the same
catch up to the technology. A whole new fect, however, this does not guarantee a way, then we might develop molecular
branch of statistics, “p bigger than N,” has reproducible finding, Ransohoff cau- signatures that simply tell us what
opened up to address the challenges of tions. Too often biologists and computa- refrigerator the samples were stored
analyzing microarray data. The resulting tional biologists overlook an even more in,” Ioannidis adds.
innovations will likely be applica-
ble across the burgeoning
“-omics” fields.
At the same time, journal edi-
tors are tightening standards and
requiring authors to follow the
MIAME (Minimum Information
About a Microarray Experiment)
guidelines and to make data
available online. They should
also encourage authors to pro-
vide a script of their analysis, like
a statistician’s lab book,
Tibshirani says.
“There’s implicit pressure
to find positive results. And
that’s not a good way to oper-
ate,” he says. “A script keeps
you honest. It forces you to
remember exactly what you
did, maybe six months ago.
Maybe you’ve forgotten that
you’ve actually tried 25 models
since last July.” A script also
makes it easier for others to
evaluate the approach.
Another solution is canned
software—such as the packages
that he’s developed and made
freely available online, SAM
and PAM (Significance Analysis
of Microarrays and Prediction
Analysis of Microarrays). These
programs constrain people
from simply making the choices
that make their data look best.
“You need a little bit of a
straight-jacket almost,” he says.
Biologists are also realizing
the importance of having statis-
ticians on their microarray
teams, Wong says.
“I can clearly detect a Commercial microarray products such as the Mammaprint and Oncotype DX (not shown) gene
changed perception about stat- expression tests determine whether breast cancer is likely to recur and thus should be treated
isticians,” he says. “Before, the aggressively. Courtesy of Agendia, Inc.

22 BIOMEDICAL COMPUTATION REVIEW Fall 2006 www.biomedicalcomputationreview.org

Stanford_p18-25:E 8-15 Human vs Machine 10/2/06 2:25 PM Page 23

For example, a 2002 Lancet paper of biostatistics and human genetics at the your current and future health, they
(by Lance Liotta and Emanuel University of California, Los Angeles. show that microarray data are not an
Petricoin) announced the develop- “Five years ago there was wide empty wasteland. Dismissing microarray
ment of a highly accurate blood test for enthusiasm about microarrays, so peo- technology now would be like stopping
early stage ovarian cancer.11 Ovarian ple were probably a little bit too naive flight travel because the first few planes
cancer is usually fatal because it is diag- about the challenges that lay ahead,” crashed, Horvath says. These early
nosed too late, so accurate early detec- he says. “Now the pendulum appears crashes led to strict and effective safety
tion would be a huge leap forward— to have swung back in the opposite procedures for flight, and, similarly,
exactly the incredible payoff that the direction, where people are much too early failures in the microarray field
“-omics” technologies have long prom- negative about the promise of have led to stricter standards to ensure
ised to deliver. The test was based on microarray data.” reproducibility, he says.
proteomics—patterns from mass spec- Indeed, the backlash has overshad- As a more mature field faces its sec-
trometry, rather than microarrays—but owed some exciting successes. In 2005, ond decade, it is also adopting a more

Dismissing microarray technology now would be like stopping

flight travel because the first few planes crashed, Horvath says.

the study design issue is the same. the FDA approved the first microarray- realistic outlook. Microarray users
The finding launched a commercial based clinical test, AmpliChip (from acknowledge that an all-inclusive fin-
test (OvaCheck, Correlogic Systems, Roche and Affymetrix). The test identi- ger-prick test is unlikely to materialize
Inc.); prompted an unprecedented con- fies genetic variations in the gene for anytime soon, but they have a more
gressional resolution granting more cytochrome P450—an enzyme that modest goal for their next decade: to
funding; and was deemed one of the top metabolizes common drugs—and allows streamline their search for meaning in
ten medical breakthroughs of 2002 by doctors to personalize drug choice and a vast sea of data.
Health magazine. dosing accordingly.
But soon after the initial paper, other A 21-gene expression test for breast FOOTNOTES
scientists began questioning the results. cancer, Oncotype DX (Genomic 1
Lander ES, Nat Genet, Jan 1999.
Many now believe that Liotta and Health), has been validated in large, 2
independent studies. By distinguishing Frantz S, Nat Rev Drug Discov,
Petricoin’s findings were actually an
lower and higher risk tumors, Oncotype May 2005.
unintentional artifact of differences in
the way the cancer and non-cancer sam- DX may spare up to half of women with 3
Ruschhaupt M, et al., Stat Appl Genet
ples were processed. The authors had a common type of early-stage breast can- Mol Biol, Jan 2004.
found a real statistical pattern that sepa- cer from unnecessary chemotherapy. A 4
2005 analysis showed the $3000 test to Knudtson KL, et al., J Biomol Tech,
rated the groups, but it wasn’t a signa-
be cost-effective because of the averted Apr 2006.
ture of the ovarian cancer.
To avoid such errors, Chu always chemotherapy.12 Oncotype DX is now 5
Golub TR, et al., Science, Oct 1999.
processes a patient sample at the same being tested in a major prospective clini- 6
Sorlie T, et al., Proc Natl Acad Sci,
time as its control. Some people might cal trial sponsored by the National
Sep 2001.
consider his attention to detail obsessive- Cancer Institute.
7
compulsive, he says. “But actually you A 70-gene breast cancer test devel- Petty RD, et al., J Clin Oncol,
almost have to be more obsessive with oped in the Netherlands, MammaPrint Apr 2006.
microarray data than with almost any (Agendia), is undergoing a second 8
Mehta T, et al., Nat Genet, Sep 2004.
conventional biological experiment.” round of validation studies. The jury is
9
Microarray teams should also still out, but it is already being used in Dave SS, et al., N Engl J Med,
include clinical epidemiologists to some clinical settings. A recent study in Nov 2004.
address these basic study design issues, the New England Journal of Medicine 10
Michiels S, et al., Lancet, Feb 2005.
Ransohoff concludes. found that though MammaPrint and
11
Oncotype DX only overlap in one gene, Petricoin EF, et al., Lancet, Feb 2002.
A MORE MATURE FIELD they give similar results—they agreed 12
Hornberger J, et al., Am J Manag
The result of high-profile failures has about whether tumors were “high” or
Care, May 2005.
been an unwarranted backlash against “low” risk in 81% of cases.13
13
microarray technology, reflects Steve While these examples fall far short of Fan C, et al., N Engl J Med,
Horvath, PhD, ScD, associate professor a finger-prick test that instantly sizes up Aug 2006. ■

www.biomedicalcomputationreview.org Fall 2006 BIOMEDICAL COMPUTATION REVIEW 23