Explore, Explain, Des ign

Andrew S. Gibbons
Brighmn You11g University, Provo, Utah, USA

C. Victor Bunderson
The EduMetrics Institute, Provo, Utah, USA

Glossary wruTmt for causal claims, nor do they fully rule out
altemate eÀ-planations.
design experimen ts Exhibit all aspects of a design siudy, invariant measurem ent A carefol and e>..'tended design,
except that, in seeldng explanatory and design theories, development, and validation process may obtain teclmo-
reliance on narrative methods is supplement ed with logical <levices (measureme nt instruments) that produce
invariant measuremen.t of the growth or change constructs measures approaching the ideal of invariance. The mea-
spanning the domain. The measureme nt instmments evolve sures produced are invadant to the sample of subjects or
over the cycles of design; they implement, evaluate, the set of tasks selected, have an agreed-upo n zero point
redesign, and come to embody an increasingly adequate and a standard unit used by many practitioners and
descriptive themy of the processes operative in the scientists across national boundaries, and have a widely
domain. ln addition, the technological <levices designed accepted interpretive framework based on a theory of the
to introduce and control the h·eahnent effects are forth- operative constructs of change or growth being measured in
rightly described using the emerging layers and languages the domain spanned by the inshument' s scales.
of technology in that domain. ln expeiimental termi- natural history A lmowledge-producing process in which
nology, design füq)eriments are quasi-experiments, but natural phenomena are observed, described, measured, and
may include mini-randomized mq)eriments within a larger collected to amass a body of facts from which patterns and
cycle. trends can be detected through study of the facts in the
design researcb Includes design studies and design eh.'Peri- presence of each other. This lype of lmowledge seeldng is
ments, both of whicl1 build domain-specific descriptive equated here with exploratory research. A benefit of
theories as well as design themi.es. Design research also attention to this type of knowledge seeking is the growth
includes research on design methods ar design theory, as of colleclions, which can be studied and resludied as
ªPlJlied across 1.wo or more domai.ns. evidence to support or question new models and hypoth-
design studies Seek 1.wo ldnds of theoretical lmowledge: first, eses as they are advanced.
a desmiptive eh.vlanation of the processes opera.tive in science A lmowledge-producing process in which questions
a domain, and second, technologica l or design lmowledge are asked, primarily of an eh-'Planatmy nature, m.1d research
about how to create and implement the tools-both is canied out to answer them. The questions asked seek to
measurement instruments and the treatment control desclibe authmitative ly the nature of underlying operations
technologies. These studies are attempts to discover new that lead to observed phenomena . This type of research
artifact- and intervention-related p1inciples or to improve altempts to rnscover the single best coherent description of
the effectiveness of existing artifacts or intervention plans. obseived phenomena that is consistent with all observa-
Design studies take place lll live settings, and are i.terntive, tions; here, explanatmy research and science are equated
cyclical applications of a process of principled design, technology A lmowledge-producing process in whicli ques-
implementa tion, evaluation, m.1d redesign. Design studies tions are asked, primarily to lean1 principles for connecting
often ai.d in explming a domain and possible h·ealments, human intentions with the fonn and function of human-
-and thus may be largely -qualitative, prnducing narrJ!.tiYe n1!1de arti-f~~ts,. m~d research is canied out to answer these
- - accounts. 1hese accounts may nõtp1t:Ni de-adequa t,,-----n--u esl:i0ns-;---'f l'l.e---quesâ0 ns-asked---s0illc--ways--0f:__gtr.uclu1:in.------
-,------

927
Encyclcpedia. of Social Measu.rnment, Volume 1 ©2005, Elsevier Inc. All Rights Reserved.
 928 Explore, Explain, Design

time and space with information, forces, and materials in synthesis requires rneasurement, and that measuremen t
such a way that a particular prespecified outcome is instruments are technological products-too ls both for
achieved. Here, this activity is placed under the heading research and forpractical purposes. These toais, however,
of design. Rather than a single explanation of obsetved
phenomena, this type of lmowledge seeking
have unique properties of scientific interest. Even prior to
attempts to
discover efficient structuring principies and processes that the existence of adequate theory, the development and
will produce a variety of solutions to a problem, from which use of new measuremen t instruments has repeatedly, in
the most suitable may be selected on the basis of problem- the history of science, led clirectly to discovery and then to
specific criteria. new theory. As instruments and theory evolve together,
the instruments are used to test, conRrm, or discreclit and
replace theory with better theory. Measurement instru-
ments are thus inextricably linked as products of-and as
The explore, explain, and design concept designates three precursors to-the advancement of theory.
synergisticaliy related lmowledge-producing enterprises. Social measurement requires extensive and disciplined
"Explain" denotes the familiar goal of scientiRc research, design and development to design, produce, validate, and
which is to explain why and explain how. ln this type of maintain scientiRcaliy, socialiy, and economicaliyvaluable
research, the single best explanatory principie is sought. measuremen t instruments. Examining the relation of ex-
"Design" denotes the goal of design research, which is to plore, explain, and design and their relationship to mea-
discover and apply structuring and synthesizingprinciples surement is important because valuable measuremen t
in arder to satisfy a set of criteria. ln this type of research, instruments produce social, economic, and scientific cap-
classes of increasingly desirable artifacts (or structuring ital. Producing this capital is the work of natural history
principies) are sought so that they may be used to satisfy and technology as much as it is the work of science,
target criteria. "Explore" denotes a type of research aimed a clistinction elaborated herein. Explore, explain, and de-
at producing observations that can lead to category for- sign activities are mutualiy sustaining, and no one of them
mation and formation of hypotheses of relationships rel- can proceed far without producing questions to be an-
evant to both of the other research enterprises. Within swered bythe other two. Figure 1 illustrates this relation-
emerging domains of human lmowledge, the questions ship. The two-way arrows in Fig. 1 indicate that influence
concern what is there and what are possible groupings between activities can actin either clirection. ln the graph-
and relationships among what is there. ical depiction, the explore activity is situated above the
explain and design activities, inclicating that exploration
often tends to provide a starting point, stimulating
the other two àctivities.
Distingui shing Explore, Explain,
New concepts about invariant and computer-
and Design administered instrumentation mal<e possible research
methods in the social sciences that offer new and prom-
The three lmowledge-producing enterprises---explore, ising opportunities for ali three lmowledge-producing
explain, and design-are necessary conclitions for each activities. Computers and networks can support introduc-
other, each producing results that become mutualiy tion of both treatments and metrics into live settings
and self-inputting, providing a continuous stream of re- where real phenomena of interest are found, though
search questions in ali three areas. The three enterprises it requires far greater engineering and scientiRc invest-
are cliscriminated on the basis of the kinds of questions ment. Extended efforts to achieve broad acceptance
they address and the types of lmowledge they produce, (efforts far beyond what is usualiy attempted in the social
but not deRnitively on the research tecbniques emp-
loyed. The focus here is to place these three lmowl-
edge-producing activities into context with each other,
and to highlight design research. Throughout this ency-
clopeclia, the word "design" appears in only a handful of Explore
titles, referring, for instance, to experimental design, sam-
pling design, survey design, and longitudinal cohort de-
sign. These are methods of setting up data- gathering
opportunities for exploration- and explanation- centered
observations, but design as a synthetic process by which
lmowledge can be gained is not featured. This is the do-
main here. Moreover, though measuremen t foundations
and models are cliscussed in the literature, little attention Figtrre I Mutually sustaining relati.onship between three
is paid to the fact that the practice of research through knowledge-creation activities: explore, explain, and design.
 E>..-plore, E>..-plain, Design 929

sciences) can then approach the ideal of invariant mea- became gnided by emply positions in collections: places
surement, providing comparability from occasion to oc- where something ought to be. The sense-making of nat-
casion and from group to group, so that unprecedented ural histmians was to find the palterns and gradations of
progress can be made. Hmvever, progress will continue to difference that might give hints about underlying
be limited in the human sciences and in design so longas relationships. These motives continue in today's natural
the metrics are incommensurable from study to study. histmy research. Researchers todayworkwith ali possible
speed to catalogue Brazilian tribes still untouched by
modern civilization, dying native la11guages, mutant
viruses, and evolving forms of computing machine:ry.
Explore Though the subjects of observation, collection, measure-
ment, and inventory have changed, as have the mea11s of
Early Scientific Exploration-Natural measuring, what has not changed is the necessity of mea-
Ifistory sming observed phenomena. The formation of categories
that result from this type of research depends on ordering
Before it is possible to create theories, we must have some
description of nature and the content to which themies along presumed dimensions, leading to measurement.
pertain. Natural histmy has been described as a type of The constant refinement of categories goes hand in
research involving the collection and cataloguing of spec-hand with the constant refinement of ali paris of mea-
imens as an inventmy of "what we've got." This becomes sures: const:ructs, dimensions, observable indi.cators, rub-
lics, scales, insh"uments, and inference/analysis methods.
a register of facts anda compilation of the contents of the
world. This way ofknowing is contrasted with e"-pe1imen-
tal inquiry into cause, which originated as natural philos- Many Exploratory Methods Exist and
ophy, i.e., an explanation of causes rather than an Often Lead to Explanation
inventory.
Natural histmy studies have historically embraced any- Natural histmy collections lead to tl1e formation of named
tlling tlrnt can be named and collected, though it is catego1ies, and subsequentlyto arder relationships. ln the
a common misperception today that naturalistic studies mathematical themy of measurement, this empilical
are confined to animals and plants. One of the mos! fa- achievementprovides tl1e basic definitions needed to con-
mous collectors, the founder of the B1itish Museum, Sir struct theorems of representation from the empüical
Hans Sloane (1660-1753), collected thousands of plant world into tl1e real number system. Theorems of repre-
and animal specimens, at the sarne time amassing collec- sentation must be anchored in the empüical world. First,
tions of coins, medals, spoons, goblets, rings, minerals, the categmies of objects are defined as seis, and arder
weaponry, combs, and a huge library of manusclipts. relations among them are defined as relations on those
Today, similar collections of human-made objects from sets. When these sets and relations meet ceitain axioms, it
around tl1e world compete in size with those of naturally can be shown that a mathematical strncture (usually the
occuning phenomena. Sloane's collections, donated to real number system with its many desirable properties)
government, formed an important core of the Blitish Mu- can be used to represent or model the real-world obser-
seum. Sloane considered his major conb·ibution to sei- vations. Matl1ematical models yield predictions of conse-
ence the "collection and accurate arrangement of these quences (often otherwise unexpected) that can be tested
cmiosities." using methods much stronger than natural history or
Natural histmy explorations have a long history of 1ich other largely exploratmy metl10ds. Useful explanatory
collectors assembling collections of oddities, gatl1ered themies can result, witl1 warrant to claims that the causal
mainly as conversation pieces and enhancements to social connections proposed in the themy operate ain:!laimed,
status. However, natural histmy research fü,.l)anded and warrant to claims that other competing explanations
greatly from approximately 1800 to 1900 and changed are not the causal agents. This path from e"-ploration
in character. More selious and systematic collectors through category formation through measurement,
emerged to gather and catalogue witl1 the pm-pose of which enables cmcial and rigorous testing, is lypical in
ananging specimens in orderly tableaux that emphasized science. Discussion of theorems of representation, based
gradation of similaiities, suggested regular underlying on the fundamental work ofDavid Krantz and co-autl1ors,
palterns, and idenlified gaps tlmt supplied targets for for- in three volumes published in 1971, 1989, ai1d 1990,
tl1er collection. The method of natural histmy research shows the fundamental contribution of exploration to fur-
becaine wide search, collection, description, cataloging, ther progress toward explanation. Exploration leads to the
preseivation, preparation for display, and archiving. Tllis, formation of the categmies (sets) of empirical obseiva-
of course, required that systems of measurement be de- tions, a11d relations among them. These lead in tum to
vised for describing the magnitude of the different qual- measurement, and tl1en to sb·ong tests of causal predic-
~-----m.esof-speci1nen1,--:--Tlre-search-of·--11atm·al-historia11s~---+<uuns----and----relations-among-thein·-.- - - - - - - - - - - - -
 930 Explore, Explain, Design

The intent of some methods is initially exploratory, but • How can practices that currently produce desirable
may lead to hypotheses testable by stronger methods. results be refined to meet ever higher, ever more specific
Exploratory narrative methods include case studies, criteria?
field and observational studies, naturalistic inquiry, • How can new measurement constructs, measure-
some of the cognitive methods, focus groups, participant ment instruments, and constructs be created that permit
observation, unobtrusive methods, and others. ln all these better measurements of artifact properties, intervention
cases, experts in these methods point out a linkage to points, intervention progress, and attainment of criterial
theories in their domains of interest, which increasingly end points?
gnide and shape the plans for their otherwise exploratory
observations. Thus, it may perhaps more properly be said As a method of research, design involves principies of
that a method is exploratory when it is used in an early search, analysis, problem re-representation, discovery
phase of observation in a relatively new area. ln a !ater of abstract operational principies, principled proliferation
phase, when constructs and relations have been observed, of configurations, means-end matching, optimizing, and
named, and documented, begtnning explanatory theories testing-all of which requires constant, continuous mea-
come into being and guide explanatory research. The surement. Herbert Simon has attributed recent rapid
example of the "empty space" on the shelf in a natural growth of interest in design research to "the fact that
history collection is a case in point. Even the begtnnings ali who use computers in complex ways are using com-
of relational constructs can direct the search for interest- puters to design or to participate in the process of design.
ing observations. A method used early to explore a new Consequently, we as designers or as designers of design
domain may be used !ater to investigate a theory-based processes, have had to be explicit as never before about
prediction. what is involved in creating a design and what talrns place
ln addition to narrative methods, quantitative methods while the creation is going on." The domain of design can
are also used in exploratory phases. Exploratory factor be illustrated as in Fig. 2. As showa, natural processes
analysis, cluster analysis, latent class analysis, and Q involving the transfer and translation of energy and infor-
methods produce sets of possible groupings-possible mation proceed independently of human agency. Even
constructs with varying explanatory value and potential matters as complexas human learning and forgetting are
natural processes, and either one may proceed in a natural
utility. After further development, the quantitative
methods give way to confirmatory analyses, structural but less desirable path than if instructional interventions
equation models, hierarchical models, and causal models. are introduced. Natural processes are the subject of much
Other more quantitative exploratory methods include exploratory (natural history) and explanatory (scientific)
data mining, computerized record linkage, and statistical research. Humans, in the exercise of technology designs,
matching. identify measured intervention points at which they im-
press energy (force) or information (structure) on natural
processes, using artifacts and intervention plans in a way
calculated to deflect the natural processes toward the
Exploratory Research Leads Also to achievement of desired, measured outcomes. Attaining
Design Research the desired outcomes may require multiple interventions,
multiple intervention artifacts, and multiple, often con-
Just as exploratory research leads naturally to research tinuous, measurements.
into causes, it leads also to research into outcomes and Intervening in different processes at different
the means of predictably producing them. Exploration measured points using d.ifferent artifacts is an important
leads to measurement instruments, which are technolog-
ical <levices, the products of design. "Design" is a term
that describes intentional structuring of artifacts and
intervention plans to bring about predictable outcomes.
1~
De_sign is both a subject of research and a method "?-
'1i,
of research and knowledge production. As a subject of 1\'i),
research, it poses many interesting questions:
Measured intervention point(s) { ~ Design/development process
• How can knowledge about how things work be and range of intervention
converted into principies for maldng things work in Artifact of
a particular way to reach specified goals? intervention
• What generalizable structuring principies guide syn-
thesis and dimensioning of artifacts? ~
• What generalizable principies guide the formation of E] Desired outcomes
intervention plans using artifacts? Figure 2 The domain of design.
 E>..1)lore, E>..'!)lain, Design 931

method oflmowledge production through design. It can For a long time, design has been deemphasized among
rely on random permutation, which results in occasional otherways oflmoviring. A1istotle, in Nichomachean Ethics,
success, or it can proceed deliberately, according to gen- described five kinds of"intellectual viltues," forwhich he
erative design principles tl1at promise a higher rate of used the term lmowledge. Conside,ing the first three of
success and predictability. Of interest here are tl1e these, scientific lmowledge is analytical, deductively prov-
many ways in which the technology of measurement is able, and teachable, whereas mt and design, or "maldng,"
not just ilwolved in, but makes possible, other teclmolo- were regarded as chancy, variable, and intuiiive, thus
gies. Measurement is therefore a central topic in design not teachable. Practical lmowledge, including ethics,
research. Design research produces ]mowledge that is was also vmiable, not deductively demonstrable, and
nonhivial, and according to VValter Vincenti, consists of a virtue, not an mt. Adstotle dealt much wi.th practical
severa! dislinct lypes: lmowledge or política! science, which he claimed was that
ldnd oflmowledge mos! related to tl1e mos! good for the
1. Fundamental design concepts-plinciples for
rnost people.
stlucludng designs. The axiomatic rnethod of deductive proofs developed
2. C1i.teda and specifications-means of e1.1)ressing and stressed by Aristotle became associated with
problems and designs. science, but modern writers such as Herbert Simon and
3. Theoretical tools-concepts that relate scientifio to Buckminster Fuller have reasserted it and other mathe-
teclmological lmowledge and that suppmt the formula- matical metl10ds in efforts to establish the foundations of
tion of design themies. a design science. It is tl1rough mathematical, logical, and
4. Quantitative data-<lata on properties and ranges, conceptual rigor that Simon hopes tliat design will be
constants, data for prediction. accepted within academic ci.rcles: "ln terms of the pre-
5. Practical considerations-how things work in real vailing norms, academic respectability calls for subject
and messy settings. matter that is intellectually tough, m1alytic, formalizable,
6. Design instrumentalities-design process reason- and teachab]e. ln the past, much, if not most, of what we
ing mid procedural lmowledge. ]mew about design and about the artificial sciences was
These categmies interact with lmowledge from otl1er intellectually soft, iniuitive, informal and cookbooky."
sources, especially from science, to enahle tl1e design The term "design" here tlms refers to lmowledge-
of new things, processes, and events. However, it is not producing siudies and expeiiments into (1) tl1e act of de-
always practical to separate out sources of lmowledge, signing, (2) the design processes and generative plinciples
because science and design opera.te hand in hand in for design, and (3) study of the sb·uctural properties of
eveiyday problem solving, Vincenti underscores the im- designed things. Design consists of the structming of time
portance of beginning to see design research as andlor space in arder to achieve specified pml_)oses within
a lmowledge-producing activity on a par with explanatmy the bounds of given constraints and to the levei of given
research. cdteda. This involves the arrangement of mateiials andlor
events to h"ansfer or translate energy or information with
the intention ofproducing aspecific, measured result. The
Design vaiieties of design lmowledge previously described are
produced through design research. Simon emphasizes
What Is Design? tl1e impmtance of giving design reseai·cl1 equal standing
Design research is historically the least well lmmvn and with explanatmy, or scientific, research: "The proper
least well understood of tlrn triad in Fig. 1, but interest study of those who are concerned witl1 the artificial is
in design research is increasing. Tough-minded e1.7?eri- thewayin which thatadaptationofmeans to environments
mental, sampling, and survey designs ai·e used, and other is brought about-and central to tliat is the process of
aspects of the design of measurement instruments, in- design itself. The professional schools will reassume
cluding new types of items or questions, and computer- tl1eir professional responsibilities just to the degree that
adminístration, are increasingly common, but the bulk of they can discover a science of design, a body of intellec-
these works take tl1eir perspective from tl1e framework of tually tough, analytic, partly formalizable, partly empili-
scientific research; the focus here is on discussing design cal, teachable dochine about the design process."
research in more detail to provi.de an alternative view- Some social scientists have puzzled over why rnany
point. Design research is emphasized here as a distinct graduate students and advisors prefer to engage in hy-
lmowledge-producing activiiy that discovers processes, pothesis testing rather than instrument design and dev-
plinciples, and sb·uctural concepts essential for the pro- elopment. Warren T,yon, in 1996, noted tl1at such
duction of the technological tools and devices used in investigators may recognize that defective instrumenta-
~~'PJm:_ational research, e"'-'Planatory research, and in tion limits their inferences, but that nevertheless, they
- - - - - - a . ,esigrnesea,,eJt.------------------"<Gntinue-to--eng,ige...in-11ypothesis__testi1,g...withouLpIJ·=·_ _ __
 932 Explore, Explain, Design

instrument development. Tryon explains tbis phenome- "universe" and its general principies and constraints on
non by pointing out that test development is not as sci- what can exist therein guides his designs, and has been
entiflcally respected as hypothesis testing is, even though arnplified in the Design Science Series of books on
test development entails construct validation, which is natural structural principies in design. Many other writers
a highly theoretical enterprise. The neglect of design as besides Simon and Fuller provide ample evidence that
aknowledge-producing activity has been changing slowly. design is not restricted to the chancy, the variable, and the
An Internet search on the tenn "design science" shows intuitive as Aristotle stated, nor is it unteachable.
that it has been incorporated forihrightly in fields as dis-
parate as engineeiing design, public administration, and
education. Relationships among
Explore-Explain-Design
Establishment of a Science of Design It is customary for scholarly writers to end their works
with the statement that more research is needed to answer
Simon lists the "tough-minded, analytic, formalizable, and questions generated by the results of the research being
teachable" subjects he thought might become a part of reported. Fignre 1 illustrates the relationship of the three
a design science curriculum. He encourages rigor and knowledge-producing activities-explore, explain, and
discipline by associating design research with techniques design-as a set of two-way arrows, to show that the
for utility theory and statistical decision theory for evalu- results of any one type ofresearch can lead to questions
ating designs; linear programming, control theory, and answerable by any of the other types. A few exarnples
dynamic programming for choosing optimal altematives; are given at tbis point to show that tbis relationship has
imperative and declarative logics for a formal logic of important implications for research of each type.
design; heuristic search for design altematives, theory
of structure, and organization of designs; and altemate Exploratory Research Leads to
representation of design problems. The term "design ExplanatortJ Research
science" has more often been used in disciplines influ- As the activities of natural history measure and catalog
enced by his work than has the term "artificial science." natural phenomena, pattems become evident, requiring
Simon emphasizes that natural science is lmowledge explanations of causal relationships, origins, and interde-
about natural objects and phenomena, so an "artificial" pendencies. For example, when paleontological research
science would contain knowledge about artificial objects on both sides of the Atlantic revealed the types of prehis-
and phenomena-and the disciplined activities by which toric animal and plant life that had once inhabited those
artifacts are designed and developed. Simon believed regions, a pattem ofrelationship became evident to the
that design is central, and should be at the core of scientist Alfred Wegener, and to others, that ran directly
every liberal education: "If I have made my case ... the contrary to the prevailing explanatory theories of the time
proper study of mankind is the science of design, not only regarding the origin and history of the continents. To
as a technical discipline but as a core discipline for every Wegener, the only explanation that lit ali of the observa-
liberally educated person." tions was that the separate continents had been joined at
Another influential 20th-century design scientist, one point in the past but had drifted apart. Though his
Buckminster Fuller, promulgated the concept of explanatory theory of continental drift was dismissed by
a "comprehensive anticipatmy design science" to seek opinion leaders, additional evidence that supported
solutions for global problems and to preserve life and Wegener' s theory appeared many years !ater when the
progress on "spaceship earih." As does Simon, Fuller Atlantic sea fioor was being mapped for the first time.
points to rigorous theory of structure and organization Clear signs of sea,floor spreading gave a new relevance
at the heart of design. Unlike Simon, he invented syner- to Wegener' s theory. What is important here is not that
getic geometry as the heart of his theory of structure. To Wegener' s theory triumphed, but that it was the descrip-
Fuller, the idea was not to set out to invent or design tion-the sea-fioor mapping-of natural phenomena that
something. As quoted by a colleague and interpreter, led to the ultimate reconsideration of Wegener' s theory.
Amy Edmonson, in 1987, he "did not set out to design
a geodesic dome." Rather, he sought to "tap into the ex- Exploratory Research Leads to
quisite workings of nature," to "discover the principies Design Research
operative in universe," and then to apply "generalized When humans find pattems in nature's mechanisms that
principies to produce artifacts useful to humankind." they can copy, they do so even ifthe details ofthe pattem' s
Fuller's observations of nature and explorations into internai operations are not fully understood. For
a geometry of space different from what he (and we) thousands of years of recorded history, humans have cre-
learned in school reveal a geometry of spatial forms ated and used artifacts and intervention processes that
that is practical and empírica!. Fuller's attention to mimic nahiral phenomena, with little orno prior research
 that could produce an e,l'lanation of the effect. The dis- possibilily. Though the ideal of science is often
coveiy of medicines and medical procedures through the e>.Fressed as seeking lmowledge for tlrn sal,e of lmowl-
ages has tended to follow this route. Even today, folk edge, it is also true tl1at the foundation of our technology-
remedies and natural healing concoctions derived from based economy relies on the abi1ity to turn p1inciples into
locally found substances supply the beginning point for products. Examples of this transfer are common. One of
the refinement of pharmaceutica]s. Until recently, seren- the most interesting and little recognized examples,
dipity, or dogged persistence in trial-and-error, ,vas however, is the basing of the modern technology of sur-
a major source of new drug discove1y. Interestingly, gery ou John I-Iunter's realization that the human body's
this is being replaced with plincipled anticipatmy design º""' healing powers could be relied on following surgery,
research. or what Sheiwin Nuland has desc1ibed as a "controlled
Early in the indusbial revolulion, some manufacturers injmy" to the body. By determining what injmy to infiict
proceeded by trial-and-error in the absence of direction on the self-healing system, surgeons today recmit tlrn
from e},._-planatory science. The excellence of early body's natural processes to carreei painful and deadly
'i\'edgwood china traces its 01igins to tireless design ex- conditions. The study of surgeiy can be seen in this
peiiments involving clay and va1ious minerals from ligbt as the design of injuries to infiict for given unhealtl1y
Wedgwood's own fann. Writer Jenny Uglow has ascer- body conditions.
talned that "ceramics has always been a mix of science, Tbe pattern of transfer from explanatory research to
design, and skill, and every good potter was in a sense an design research is so common that virl1at is being
e,qJeiirnental chemist, trying out new mixes and glazes, desclibed here as design research is often referred to
alert to the impact of temperatures and the plasticity of as "applied science." It is the unquestioning acceptance
clay." This, alongwith vmious patterns of aging, working, of tllis belief and a Jack of awareness of the complexities
and filing using a variety of glazes, ultimately produced of bridging the science-technology gap tliat has, nntil
formulas that looked attractive and survived in their en- recently, given technology an image subordinate to,
vironment of use belter than did the products of compet- and as simply a receiver of, tl1e benefits of science.
itors. ln some cases, then, e1.1)loratory research turns up One of Simon's main points is that the image of applied
useful structures and processes that become fundamental science has obscured the lmowledge-producing nature
technologies. For example, the transistor effect was dis- and complexities of design research and has masked
covered by chance because of studies on the impmities the full range of technological lmowledge tliat is required
in crystal substances. for tl1e design of artifacts m1d intervention plans. Histor-
ically, this has placed design resem·cl1 in the shadow of
Explanatory Research Leads to science, rather tlmn giving it equal regard and propor-
Explora.tory Research tionate study.
As explanatory themies gain suppmt through scientific
research, inferences using the tl1e01}' lead to tl1e e1.'Pec- Design Researcli Leads to
tation of finding as-yet undetected natural phenomena. Explorational Research
The early stages of success of a themy constmcted and Design research leads to e:>..1)loratory research when
initially supported through explanatmy research often a design study or a design expeliment produces results
that are unexpected and even undesclibed. The first
require additional e,l'loratmy research to develop further
support. Armed witl1 predictions of themy, researchers objective after such an event is to explore the region
around the undescribed phenomenon, in search of an
can go in search of specific phenomena that are predicted
but as yet not observed. Exploratory research motivatedentire class of similar new phenomena. This occurred
in the 1980s when chemists examined tl1e residue pro-
by the explanato1y theory of Mendeleyev's periodic table
duced by intense laser heating of small sections of
of the elements led to the discove1y of severa! elements,
most notably tl1e trans-urani.um elements, one of whichgraphite, or pure carbon. \iVhat was found was a family
was named after Mendeleyev, in honor of his discove1y. of pure carbon compounds containing up to 60 or more
Likewise, improved astronomical instruments developed atoms per cluster in the shape of a sphere-a form of
carbon never before recognized. Philip Bali has described
decades after Einstein's relativily themy allowed astron-
omers to observe the phenomenon of gravitational the process of discovering the compound's shape:
lensing, jus! as Einstein's themy had predicted. "Smalley [one of the chemists] located a book on
Buckminster Fuller's work, The Dymaxion World of
E~-planatory Research Leads to Bu.clcminster Fuller by Robeit W. Marks, which he took
Design Research home on the Monday evening to ponder. lt is remarkable
As descriptive tl1e01ies gain suppmt through scientific to find tliat, despi te what we like to think of as tl1e sophis-
resear.ch, using principles from the theories_ to e;i,;:e- tication of modern science, some of the most significant
~-----r=c1se conn·oho-pi'otlu-ce---sp-ecifrc-ontcomes-becomes-a------insights-can--stHl--he-ebt-ained--by--sitting-clewn--with-a-eart-ef----
 934 Explore, Explain, Design

beer and fidcllingwith cardboard or ball-and-stick models. toais that contrai the treatment effects. Design stuclies
Yet tbis is how the structure of C00 was deduced." Once take place in live settings, and are iterative, cyclical
the structure of sixty-carbon compouud_s had been recog- applications of a process of design, implement, evaluate,
nized and confirmed, and the C00 cage of hexagonal redesign. Design stuclies often aid in exploring a domain
and pentagonal sides with 60 vertices was named and possible treatments, and thus may be largely quali-
"Bucky Bali," in Fuller's honor, more exploratmyresearch tative, producing narrative accounts of intense, iterative,
was launched. An entire family ofpure carbon compounds often ideographic observations over each cycle. These
containing geometrically determined numbers of carbon accounts may perform the functions of exploration clis-
atoms had to be explored. This included the "stable cussed in this article, and may begin to use theory to guide
and cagelike" clusters of C32, C5 o, and C70 • This adclitional both the design interventions and the explanations of the
period of exploratmy research was initiated by the unex- processes operativein the domain. However, theymaynot
pected cliscovery of previously unrecognized design provide adequate warrant for causal claims, nor do they
research. fully rule out alternate explanations. For these research
goals, design experiments are needed.
Design Research Lead,i to
E,cplanatory Research Design Erperiments
Design research leads to explanatory research when phe- Ali aspects of a design study are shared by a design ex-
nomena result from design stuclies or design experiments periment, except that heavy reliance on narrative methods
that cannot be explained. As methods for producing the in seeking the explanatory anel design theories is supple-
buclaninsterfullerines (as the class of chemicals became mentecl with research using (and improving) the invariant
named) in quantity were perfected and published, the measurement instruments clesigned to assess and track
question of their chemical behavior became paramount, the learning or growth constructs spanning the domain.
particularly the manner in which they encapsulate other Because measurement in live settings has typically been
atoms witbin a spherical cage, and the arrangements they very costly in time and attention, design experiments
take on under clifferent conclitions of creation. Accorcling are becoming practical primarily because of new
to Bali, an entire subcliscipline of fullerene chemistry has adaptive measurement technologies that can be imple-
formed just around the study of one class of these forms, mented in live settings and can measure progress unob-
the naootubes. trusively, seemingly as a fully integrated part ofleaming
and instruction, not as separate events. Moreover, the
perception of measurement as a separate accountability
Design Stuclies and activity for gracling, sorting, rewarcling, and punishing can
Design Experiments change to a useful helping function owned by the partici-
pants in pursuing valued attainment goals. Ali the time,
Definitions and a Brief History the measurements are being used for research, and the
teachers and others are participant evaluators in this re-
The design study/design experiment movement has search, not unlmowing subjects. They are the ones who, in
been growing rapidly in educational research, and the concert with researchers and developers having adcli-
examples that follow will largely be drawn from that do- tional expertise, will participate in the redesigns, then
main, but surely Simon was correct in stating that design will implement them during the next cycle. The measure-
methods cut across the professional schools of maoy dis- ment instruments evolve over the cycles of design, imple-
ciplines. Education is but one of many social science and ment, evaluate, redesign, and come to embody an
engineering professions that can use design stuclies and increasingly adequate descriptive theory of the processes
design experiments, and it is the one closest to the expe- operative in tl1e domain. ln adclition, the technological
rience of the present authors. ln the glossary at the begin- <levices designed to introduce and contrai the treatment
ning of this article, brief definitions were given of the effects are forthrightly described using the emerging lay-
terms "design stuclies" and "design experiments." More ers and languages of technology in that domain. ln adcli-
extended definitions are given here. tion, both quasi-experimental designs and randomized
experiments may be used to warrant causal claims and
Design Studies rule out altemative explanations.
Two kinds of theoretical lmowledge are sought in a design
study; the first is a descriptive explanation of the processes Some History
operative in a domain (e.g., learning, growth, reversal of The roots of the idea of a design expeiiment in education
a disease state, improvement of an environmental condi- can be traced back to the time before the accelerating
tion), and the secondis teclmological or design lmowledge growth of interest in the previous decade; document-
about how to create and implement the technological ation of tbis was provided by John Dewey in 1916 and
Robert Glaser in 1976, for example. Despite this longer through design. Anything may be improved, inclucling
histmy, most proponents wiiling today cite Ann Brown's the measurement inshuments, the treatment, the imple-
1992 paper on design e,'Peiiments and Allan Collins' mentation methods, or the evaluation/research plan. If
1992 paper on a design science of education; the term measurement instruments are changed, equating must
"desigli. e},_1)eiiment" was introduced in the latter paper. be performed to assure comparability with measures
Both of these authors, as well as Glaser, cite Herbeit taken dming prev:ious cycles, or a new series of cycles
Simon's work. There has been a substantial grm:vtl1 of launched. Because new technologies are making possible
interest in, and publications dealing "óth, design stuclies continuous measurements ali along the way-that is,
and design expeiiments since the 1990s. An entire issue along the e,1,lanatory themy of progression of learning,
(Januaiy/Februaiy 2003) of the Educational Researcher disease progression or improvement, ar emp]oyee perfor-
has been devoted to the design research movement. mance (keeping to our three examples)-evaluation and
research data collection are also continuous, and their
How a Design Stu.dy or E~7,eriment Proceeds effects (good and bad) can be altlibuted to stages along
A study begins by selecting one or more live seltings in the patl1 of progression. The outcomes of a sedes of cycles
which füe phenomenon of interest may be observed. The investigated in this type of research are answers to the
setting may be one or more classrooms, or nonconven- lmowledge-seeldng queslions stated at the outset o[ each
tiona] lemning seltings, ar it may be a medical treatment study, whether e:>..1J]ore, e>..1?lain, or design. An outcome of
facility studying treatment options and ways to imple- substantial i.mpmtance is a documented argument for tl1e
ment them, ar a management initialive in a group setting, valiclity of a total treatment method, inclucling the tech-
repeated with clifferent groups at !ater times. A repeating nology used and "best practices," i.e., the manner ofim-
cycle is selected. A semester ar block schedule may be plementing it to get the best results. When design
use d for education, an end point in a medical treatment, or e>..veriments are used, this validity argument may include
natural assessment points for the management activity. evidence to back causal claims for the efficacy of the
The repeating cycles are essential, because the earlier treatment in leading to desirable outcomes.
cycles se1ve as controls for later ones. (There may also Terminology is still in flux. The term "design e1.."Peri-
be additional control ar treatment groups in rigorous de- ment'' has often been appropliated by those interested in
sign e,'Peiiments.) The goals of the research depend on intuitive and reflective types of action research in educa-
the stage of matuiity of the study in the domain of interest. tion, in which the teacher takes the lead role, rather than
The researchers may seek e>..1)loratmy, explanat01y, or in a partnership role witl1 researchers. The teacher
design lmowledge in any mixture. Good research design journals personal reflections on possible designs and out-
will, as usual, set forth the questions in advance. To be comes as a way to improve practice. Theory, measure-
called a design study or °''Peiiment, design knowledge ment, and the valiclity of causal inferences are not often
will be central among the research questions. ln explor- irnpmtant in such studies, but are very i111portant is sues in
atmy studies, the questions concern what design inter- research generalizable beyond single practitioners. For
ventions exist ah-eady and which seem to be working well. this reasou, the term "design studies" can be used to
As three examples of e,'Planatmy stuclies, the questions avoid the term .. e1..1?erimenf' so as not to confuse these
may concern. the nature of the descriptive progression of studies Viri.tl1 randomized hials in social and behavioral
learning or growth (in education), the progression of the science. This usage was suggested by Richard Shavelson
clisease state (in medicine), or the propagation of emplo- and co-authors, in tl1e previously mentioned issue of
yee behaviors and associated results (in management). ln Educational Researcher focusing on design research, to
all design research stuclies, design lmowledge is sought, distinguish single-practitioner studies from true eÀ1)eii.-
along with the implementation conditions that enable the ments. lnfluential ,vriters on e}.1Jerimental methods de-
designed treatments. to yield tlrnir R~§t re§nlts. fine and emphasize lvvo main types of experiments, i.e.,
The procedure is to design four aspects of the study: the randomi.zed experiment alid. the quisi-experiment.
(1) the measurement instnnnents to assess the progres- Both types of experiments are designed to test descliptive
sion (oflearning, clisease states, effective employee per- causal hypotheses about manipulable causes. Thus, when
formance, and results), (2) the treatment properties the definition of experiment or quasi-experiment holds
(technology), (3) the evaluation/research design for the in a design study, it may justly be called a design
next cycle, and (4) the implementation plan for the next e>..1)eriment.
cycle. Once the measurement instn1ments are in place,
even before a new treatment is introduced, a cycle can be Design Experiments and
run to collect baseline data. When all fom designs are in Experimental Design
place, subsequent cycles can be 1un, involving implemen-
tation of the latest version of the designs, then evaluation, It is useful to consider the nature of e>..verimental design
-!Ollowea-liy seelctng improvements in tlieneff cycle·- ·- to appreciate fully what can be accomplished with

----------------------- .,, __ ··--·--·--

 936 Explore, Explain, Design

a principled design experiment using invariant measure- subgroups of individuais recervmg the treatment
ment scales. ln publications extending back to 1963, differentially, depending on their individual profi!es.
Donald Carnpbell and Julian Stanley, and !ater Thomas To understand the sequential nature of a design exper-
Cook, William Shadish, and many others, have given iment, and how results from prior cycles can serve as
guidance much used over the years in how threats to contrais for !ater cycles, consider a simple pretest/
the validity of causal inferences can be reduced by post-test design, one of Carnpbell and Stanley's three
employing good experimental design. Classically, three true experimental designs. It can be diagrammed as
pre-experimental designs, three true experimental de- follows, where R means -random assignment to eithe::r
signs, and 10 quasi-experimental designs have been dis- group, and the sequence of events in a single row (e.g.,
cussed. There has been a shift away from significance R _, O _,X-> O for row one) is the temporal sequence of
testing in fairly recent times in favor of effect sizes, but events for the experimental group:
the need to consider threats to the validity of inference
has not abated. The simplest inference is that the intro- R O X O Experimental group
duction of treatment X did indeed cause the change in R O O Contrai group
observation (measure) O. The emphasis here is thatthere A gain score can be calculated as Opost-test - Opretest if
is a need for design disciplines to assure both that mea- the measurement scale O supports equal intervals, and
surement O and treatment X do indeed involve the other precautions in using gain scores have been
constructs oflearning progression along the pathway to followed. Then we may graph the gain for the
greater expertise. Design theories to guide the develop- experimental group and compare it to the gain for the
ment of instructional treatments can succeed far better to contrai group.
the extent that they have a descriptive account of the Considera series of such experiments, each with a well-
progress that learners typically follow in moving to higher documented change in the design specifications for treat-
leveis of lmowledge and expertise. Figure 2 depicts a de- ment X, or perhaps only changes in the implementation
sign intervention into a natural process. ln learning and procedures for the treatrnent.
instruction, the most useful explanatmy account would The design experiment in Table I assumes that all re-
describe the sequence of progressive attainments in the peating cycles of principled design intervention,
particular learning domain of interest. This descriptive X1 -X5 , take place in the sarne class in an educational
knowledge guides the design of both the measurement institution (or group of similar classes using the sarne
instruments, to determine outcome effects (O), and the treatrnents and outcome measures, X and O). The design-
instructional treatrnents (X). Instructional-design theo- ers of the experiment in Table I altered the treatrnent
ries (a term elaborated in a series of books by Charles condition X, each time, being careful to conform each
Reigeluth) are now available to guide the development of design change to a prescriptive theory and assuring
controlling technologies for the treatment. Designing X that each version of X was well documented. No R for
and O in a principled manner can help give this design the random assignment is shmvn in this table, because it is
six aspects of construct validity discussed in numerous neither possible nor necessru:y for random assignment to
publications by validity theorist Samuel Messick. The de- occur in these classes. Assume that both the registration
scriptive accouut (ortheory) ofprogressive attainments in procedures of the educational institution and the ethical
leaming the increasingly difficult tasks in a learning do- considerations bar such a practice. Thus we must depend
main, along with construct-linked measurement scales of on the sample-invariance and interpretive invariance
learning and growth, along with a validity argument, can properties of outcome measure O, and on repeated
assure that the O indeed measures valid leveis of progress near-replications with other groups in additional studies,
in the construct. For simplicity, designs that require presumably from the sarne or similar population to sub-
adaptation to individual difference measures are not stitute for random assignment. The construct-linked var-
discussed here. It is sufficient to note that good quasi- iable of interest, O, has the appropriate properties to
experimental and experimental designs exist for examin- reduce the risks to internai validity that randomization
ing hypotheses of how treatments might differ for was assumed. to reduce.

Table I Repeating Cycles of a Principled Design Experiment Using Invariant Scale O

Parameter Cycle l Cycle 2 Cycle 3 Cycle 4 Cycle 5
Baseline measure 0 0 X1 x, X3 X4 x,
o, o, o,
Control for cycles
ºº
Control for cycles
01
Control for cycles Control for cycles Control for Control for
1-5 2-5 3-5 4-5 cycle 5 furure cycles
 Explore, E:\.11lain, Design 937

Repeated measurements of subsequent groups over unprecedented contra! of a cornplex interplay of many
the cycles of a design e,q,erirnent give further basis for treabnent variables, possible in live settings through in-
ruling out the effects dueto peculiaiities of one group of teractive technology. Neither could they have envísioned
students dming one semester. But the siudents ,vho flow sophistication of online measurement, nor new measure-
through the classes of any one institution are unique and ment methods that can provide close approxirnations to
slantedin their o,vn way, so population representativeness the invaiiance prope1ties needed to realize tlie design
has not been obtained. For this function, randomization is e>qJeiiment scenario. Certain of Campbell and Stan]ey's
a poor too! as well. To obtain evídence of generalizability quasi-experimenta] designs are similar to parts of this de-
to other groups, with other language, gender, racial, and sign e)l..'Peiiment scenario, but scientists of their day could
special conditions, it is necessary that design e>,-JJeiiments not have anticipated the full extent of reliable replication
be set up in other locations ,:vherein samples of students (usingtechnology) \l~tl1 well-documented design changes.
,vitl1 subsets of these other characteiistics abound. Nev- Advances in the technology ofleanling, measurement, and
e1theless, it is no small benefit to causal inference to be management of complexity changes howweH we can con-
assured that each semester's group was either equivalent troleach ofthe 12 common threats tovalidity, of concern to
on the highly interpretable pretest/post-test measure O, these pioneering scientists. Moreover, tliese advances
or ofknown devíation. It is of no small benefit to lmow that both introduce new tl1reats to validity and help to find
tl1e use of a gain score is appropliate because we have the means to reduce their effects.
achieved a close approximation to equal interval proper- It is interesting that tlrn use of the term "validity" by
ties in the rneasurernent scale(s) O. Finally, the quest for e>..perimental design wliters and validity concepts from
the validity of causal inference is, in the framework of psychometiics, such as Messick's unified validity concept,
a design expeiirnent, set in a larger conteÃt similar to do not coincide. For example, tl1e idea of "externai
the quest for total quality management. After all, there validity" is used in entirely different ways. By biinging
are many aspects to treatment X and its procedures for these two views togetl1er through validity-centered design
implementation. To which specifically is the causal infer- and the metl1ods of principled design experiments,
ence to be made? An outstanding result is a tribute to the perhaps we are taldng a step toward bridging the gap
entire group of people a,1d the roles they assumed, the between the "lwo disciplines of scientific psychology''
mies they followed, and the toais they used to administer desc1ibed by Lee Cronbach.
the treatment. An outstanding result is evidence of the
possibility that such results can be obtained by managing
the implementation of the treatment well, and if in one Concl usion
group, why not in another?
ln addition to substituting lmown starting positions on The three lmowledge-producing acti\itie s-explo re, ex-
an invariant scale for random assignment, the design plain, and design- have been placed into a common con-
e,.-periment in Table I also replaces the contra! group, text in this article in hopes of distinguishing among the
using a series of comparisons to prevíous groups. Staiting different types of question each activily addresses; at the
with the baseline measurement on 0 0 , each succeeding sarne time, it has been shown tl1at it is not research meth-
cycle can compare its outcome not only to this baseline odology that allows them to be separated. Contributions
measure, but also to each of the preceding outcomes, of each form of lmowledge-seelctng to the other forrns
using inva1iant scales 0 0 -0 2 (the subscripts indicate have also been described, and examples have been
that, in this expe1iment, it was necessaiy to modify the given to show tl1at tl10ugh e>..JJloratmy research and ex-
outcome measure only twice, by adding or deleting tasks, planatory research have longer fonnal histories thai1 de-
whereas duiing tl1e sarne five cycles, the treatment X was sign research has, ali three fonns of research continue
modified fi_v~ R-.me~i aJter each cycle and before the next today to contribute to unanswered questions, and nane
one). By the specific objectivüy property of scale invari- has outlived its usefulness, especially not exploratory re-
ance, we can add or subtract questions ar tasks from the search, which continues today at what might be consid-
most recent version of the instrument each semester. ered an accelerated, rather than a diminished, rate.
Then, after assming that the versions are equated to The types of research and lmowledge-seeldng have
the same meaningful scale each semester, we can make been described in propmtion to the measurement theory
inferences that the version of X used that semester ,vas and technique concerns addressed in this volume. ln. par-
likely the main cause of whatever improvement in gain ticular, measurement has been related to tl1e health and
score was observed. progress of design research and e1.-plorat01y research to
New guidelines to update good expeiimental design correct what can be viewed as a current underemphasis
prescriptions must be developed for plincipled design in those areas caused by livíng in the shadow of science.
expeiiments. Scientists wiiting in Campbell and Stanley's The intent here has not been to isolate the three
rpri-ses-fr0m---eaeh--0tl::i.-ei~ut----to...showJ:heiLe.Ss~en~l~i~al~----
~------e -r"'a_c_ o_ül~·a 110CT1ave foreseen toâafs-opportunities-for---------ente
 938 Explore, Explain, Design

contribu tion to each otber: each produces results in tbe Edmonds on, A. C. (1987). A Fuller Explanlltion, The
fonn of answers that stimulate a continnous stream of Synergetic Geometry of Buckminster Fuller. Barkh.auser,
research questions iu ali three areas. The three are dis- Boston, NIA.
crimina ted on tbe basis of tbe kinds of questions tbey Glaser, R. (1976). Components of a psychology of instruction:
address and the types of lmowledge tbey produce , but Toward a science design. Rev. Edtt. &s. 46(1), 1-24.
not definitively on tbe research techniques employed. Hubka, V., and Eder, W. E. (1996). Design Sciencec
Progress will continue to be linúted in tbe human sci- lntroduction to Needs, Scope and 0-rganization of
ences and in design as longas metrics used in research are Engineering Design Knowledge. Springer, Berlin and
incomrr1ertsurable from study lo study. ln discussing de- New York.
Kelly, A. E. (ed.) (2003). Theme issue: The role of design in
sign studies and design experiments, tbe emphasis here is
educational research. Eclu. Res. 32(1}.
that rigorous design experiments are now possible using Krantz, D. H., Luce, R. D., Suppes, P., and Tversky, A. (1971).
well-designed instrum entation for administering botb ex- Foundat-tons of Mea.surement: Additive anel Polynomial,
perimen tal treatmen ts and measure ments in live settings. Representations. Vai. 1., Academic Press, New York.
This depends not only on development of commensurable Luce, R. D., Krantz, D. H., Suppes, P., and Tversky, A. (1990).
measure -buildin gtechniq ues, but also on use ofemerg ing Founclations of Mea.surement: Representatfon, A1.io-
technologies tbat give access to data collection and anal- matizat-ion, anel Inuariance. Vol. 3., Academic Press,
ysis toais previously not available. San Diego, CA.
Messick, S. (1989). Valiclity. ln Edttcat-ional Measurement
(R. L. Linn, ed.), pp. 13-103. Macmillan, New York.
See Also the Following Article Messick, S. (1995). Validity of Psychological Assessment. Am
Psychol. 50(9), 741-749 .
Research Designs Messick, S. (1998). Test validity: A matter of consequence.
Social Indicat. Res. 45, 35-44.
Pickstone, J. V. (2001). Ways of Knowing, A New HistonJ of
Furthe r Readin g Science, Technology, anel Medicine. University of Chicago
Press, Chicago, IL.
Brown, A. L. (1992). Design experiments: Theoretical and Reigeluth, C. M. (ed.) (1999). Instructional-Design Theories
methodological challenges in creating complex interven- And Models: A New Paradigm of Instructional Theory.
tions in classroom settings. J. Learn. Sei. 2(2), 141-178 .
Lawrence Erlbaum Assoe., Mahwah, NJ.
Campbell, D. T., and Stanley, J. (1963). Experimental and Shadish, W. R., Cook, T. D., and Campbell, D. T. (2002).
Quasi-Experimental Designs for Research. Hougton- Experimental and Quasi-Experimental Destgns for General-
Mifflin, Boston, MA. ized Causal Inference. Houghton-Mifflin, Boston, MA.
Collins, A. (1992). Toward a design science of education. ln Simon, H. (1969). The Sciences of the Artificial. MIT Press,
New Directions ín Eclucational Technology (E, Scanlon and
Cambridge, MA.
T. O"Shea, eds.), pp. 15-22. Springer-Verlag, New York. Suppes, P., Tversky, A., Krantz, D. M., and Luce, R. D. (1989).
Cronbac h, L. J. (1957). The two disciplines of scienttllc Founda:tions of lvíeasitrement: Geometrical, Threshold, and
psychology. Am. Psychol. 12, 671-684 . Probabilistic Rep-resentations. Vai. 2., Academic Press, San
Cronbach , L. (1975). Beyond the two disciplines of scientific Diego, CA.
psychology. Am. Psychol. 30, 116-127 . Tryon, W. W. (1996). Instrurnent-driven theory. ]. Minei
Dewey, J. (1916). Democracy and Eclucation. Free Press, New Behav. 17(1), 21-30.
York. [Cited in Tanner, L. N. (1997) Dewey's Laboraton1 Vincenti, W. G. (1990). What Engineers Know anel How They
School: Lessons for Today. Teachers College Press, New Know It: Analytical Studies from Aeronautical History.
York.] Johns Hopkins University Press, Baltimore, MD.