Research News

and Comment
Standards of Evidence in Qualitative Research:
An Incitement to Discourse
by Melissa Freeman, Kathleen deMarrais, Judith Preissle, Kathryn Roulston, and Elizabeth A. St. Pierre

In a climate of increased accountability, standardization, federal con- ensure the quality of qualitative work and even how to catalog
trol, and politicization of education research and scholarship, this and categorize the various kinds of qualitative research.
article briefly reviews various positions outlined by qualitative Qualitative research is open and supple, and one of its
researchers about quality in qualitative inquiry, showing how these
strengths is that it incorporates philosophies, theories, and
research designs and methods as diverse as postpositivist multi-
are implicated in the acquisition, conceptualization, and use of quali-
methods approaches and postmodernist social critiques.2 Rather
tative evidence. It concludes by identifying issues in and challenges to than being prescriptive about what qualitative research is and
setting standards of evidence for qualitative researchers in education. what makes it good (true, valid) across all instances of research,
qualitative researchers have tended to (a) study what researchers
Keywords: qualitative research; quality; standards of evidence; who say they are doing qualitative research are doing and then
validity
(b) encourage those who do that work to develop those practices
considered excellent. Although discussions of quality in qualita-
tive work have always been intense, they have usually been civil.
We believe that this respect for one another’s conventions is the
his article addresses standards of evidence in qualitative result of the heterogeneity in qualitative design and the absence

T research in education. Our premise is that it is neither

desirable nor possible to reach consensus about or pre-
scribe standards of evidence in this diverse field. Such prescriptions,
of an enduring hegemonic presence.
We situate our response to the call for standards of evidence
in qualitative research in a pervasive discourse that describes
we believe, amount to disciplinary action (Foucault, 1975/1979) educational research in general as historically and presently bro-
that constrain the generation of knowledge rather than improve it. ken and in need of repair (e.g., Lagemann, 2000; Kaestle,
We do argue, however, that qualitative researchers both accomplish 1993). Whether the situation is as dreary as it is made out to be
research of high quality and have a long tradition of demonstrating is beside the point; today’s fix is to make educational research
quality in reports of their investigations. scientific, and the federal government has taken the lead in this
We begin by discussing the importance of this conversation at project by mandating scientific method into law (for scientism,
this historical and political moment in the United States. We then see Lather, 2006; Ruccio & Amariglio, 2003). The fundamen-
review how validity, a preferred term for the overall merit of a study, tal idea is that rigorous science will make better schools, that
has been discussed by qualitative researchers. In this discussion, we quality science will enable us to finally reengineer schools so
consider commonalities in practice across qualitative research com- they work.
munities, describe how qualitative researchers have treated validity The National Research Council (NRC) has taken a leadership
in relation to data and evidence, and explain how they have justified role in this conversation. Having defined the nature of science in
their claims. We conclude our review by emphasizing the hetero- its 2002 and 2005 reports, the NRC has received funding to
geneity of qualitative research and cautioning against recent calls for begin a new “broad, long-term initiative related to the quality of
restrictive and disciplinary standards of evidence. evidence” (The National Academies, Division of Behavioral and
Social Sciences and Education, 2006) in the social and behavioral
The Context of the Discussion sciences. With the NRC prepared to define evidence and the
The culture of science and academic scholarship is heteroge- American Educational Research Association (AERA; 2006)
neous,1 with multiple, overlapping communities of practice and imposing standards for reporting on research methods in its pub-
knowledge that split into specialties or combine into interdisci- lications, qualitative researchers may feel under siege. Top-down
plinary and cross-disciplinary areas of study. Within disciplinary efforts such as these to legislate scientific practice and mandate
communities such as education or sociology and across commu- research design threaten to harden the boundaries of what counts
nities such as the anthropology of education, scholars have a long as science, to devalue many qualitative research endeavors, and to
history of disagreeing and challenging one another about how to limit creative research practice of all kinds.

Educational Researcher, Vol. 36, No. 1, pp. 25–32

DOI: 10.3102/0013189X06298009
JANUARY/FEBRUARY 2007 25
Commonalities in Standards of Practice Across the 1960s and 1970s to protest their absence from public policy,
Qualitative Research Communities academic scholarship, and positions of power and influence
All sciences emphasize innovation and boundary breaking as well throughout our culture. These as well as more mainstream epis-
as standardization and systemization. Conventions themselves temologies, such as critical and interpretive theories, vary both in
evolve and develop through trial and error and other kinds of what they consider standards of evidence and the priorities they
assessment. Self-correction among communities of scientific give to shared standards.
practitioners has been the strength of science across alternative In recognition of these variations, AERA (2002), in a com-
knowledge systems, and attempts to legislate standards have mentary on its own ethical standards for editing, reviewing, and
sometimes shut down scientific activity; the Soviet suppression of appraising research, has emphasized that it is important “to
genetics research in the 20th century is a case in point (Soyfer, ensure fair treatment of those who submit manuscripts for pub-
1994). The tension between innovation and conventionalizing lication and to promote the advancement of educational research
practice is, we believe, a sign of healthy inquiry. Overconvention- through attention to the quality of research and the preservation
alizing can lead to mindless recipes for research and limiting of the robust methodological pluralism of educational inquiry [ital-
scholarly orthodoxies. Underconventionalizing has its own pit- ics added]” (p. 103). Thus, the “methodological adequacy of a
falls: intellectual fads, a lack of continuity, fragmentation, and piece of work” must be judged in relation to the “methodologi-
intellectual alienation. One way to achieve a balance here cal requirements of its type . . . [and] the significance of its results
is to assess the status of conventions such as standards of evidence in the context of the problems internal to its own tradition, and
against what researchers actually do. not the requirements and aspirations of types to which it does not
A key source, then, of standards of evidence and quality belong” (p. 105). And, of course, philosophies and theories, like
throughout the history of the scientific method and its applica- scientific beliefs and research practices, change over time; thus,
tion in qualitative inquiries has been the systematic and careful AERA valorizes emergent traditions in its commentary.
documentation of all procedures—an account of practice—to pro- In qualitative inquiry, the last century’s “linguistic turn” (e.g.,
vide a record for a researcher’s ongoing contemplation as well as Rorty, 1967) that critiqued the stability and transparency of lan-
for peer review. This is descriptive work. The documentation of guage contributed to the proliferation of critical theories and the
procedure is crucial if we want to know what exemplary development of postmodern and poststructural approaches to sci-
researchers whose methodology is innovative and effective do and ence, research, and scholarship. These critiques question and
how they make their work convincing. Examining expert unsettle our notions of truth and knowledge, objectivity and sub-
researchers’ practices, then, contributes to establishing agreed-on, jectivity, and science and evidence and warn of the danger of calls
albeit ever changing, standards of quality and rigor. Of course, for norms and standards. As we have noted, such theories have
working with such experts is much to be desired (e.g., Wolcott’s their own logics that can in turn be interrogated for implicit and
[1994] version of how Louis Agassiz taught careful observation), explicit standards of practice.
and such conventionalizing not only lends confidence to results It is difficult to keep practice and theory from norming each
but also provides models for novices. As novices acquire experi- other and thus shutting down innovation in methods of inquiry.
ence and develop refined judgment, however, some come to pre- Communities of practice inevitably develop and enforce standards.
fer methodological uncertainty (e.g., Lather, 2004), and most But if these communities are to thrive, they must establish pro-
come to rely less frequently on routine protocols than they did as cedures for the ongoing interrogation of those standards.
newcomers (e.g., Benner, 2001; Flyvbjerg, 2001). Working in the tension of simultaneously doing science and trou-
Completely entangled with practice is a second source of bling it is not always easy, and this approach requires curiosity
guidelines for exemplary qualitative research: theory, the various and generosity as we encounter different and what may seem at
philosophical and theoretical traditions that both support com- the time incommensurable, threatening, and even dangerous the-
munities of practice and develop from them. This work can be ories and practices, with their accompanying standards. Yet this
prescriptive because a priori theory may impose a structure of balancing act enables conditions for good science. How then,
assumptions. Each community of practice using qualitative have qualitative researchers discussed what good science is and
research traditions adheres to its own particular theoretical how to assess the quality of their work?
assumptions. For example, one of the standards of the Chicago
school of qualitative sociology, begun in the 1920s, is to develop Validity
social science theory through research participants’ descriptions Qualitative researchers have always discussed how to evaluate their
of their own experiences, thus making explicit the “invisibility of science, the quality of their analyses and theoretical interpretations
everyday life” (Erickson, 1986, p. 121). In this tradition, the of data. They disagree, however, over the terms used in these dis-
practice of checking a researcher’s interpretations and represen- cussions: validity, reliability, rigor, and parallel terms such as trust-
tations with participants prior to publication is valued (e.g., worthiness, credibility, transferability, verisimilitude, relevance,
Duneier, 1999). plausibility, and confirmability. The literature contrasts qualitative
More recently, the interdisciplinary nature of qualitative quality standards to those used in the scientific method (Eisenhart
inquiry allowed it to welcome the epistemologies (e.g., queer the- & Howe, 1992; Howe & Eisenhart, 1990); discusses validity in
ories, feminist theories, race-based theories) and accompanying general (Kvale, 1995; Lincoln, 1995; Maxwell, 1996; Miles &
methodologies of groups who organized the social movements of Huberman, 1984; Seale, 2004; J. K. Smith & Deemer, 2000;

26 EDUCATIONAL RESEARCHER
J. K. Smith & Hodkinson, 2005); and considers it within specific evidence, claims, and warrants, terms derived from formal logic
qualitative traditions, such as ethnography (Altheide & Johnson, that are currently used in policy documents and reports (for a
1994; LeCompte & Goetz, 1982; Willis & Trondman, 2000), notable exception, see Morse, Swanson, & Kuzel, 2001).3
action research (Anderson & Herr, 1999; Melrose, 2001), alter- In methodological writing, the term qualitative data is gener-
native ethnographic designs (Denzin, 2000; Ellis, 2000; ally taken to encompass the “rough materials researchers collect
Richardson, 2000), mixed theory designs (Koro-Ljungberg, from the world they are studying” (Bogdan & Biklen, 2006,
2004), postmodern, feminist, critical, and poststructural designs p. 117), including field notes, documents, transcriptions of
(Lather, 2001; Lenzo, 1995; Scheurich, 1993), and self-study interviews and interactions, and artifacts. Data are produced
designs (Bullough & Pinnegar, 2001; Feldman, 2003). Lather from social interactions and are therefore constructions or inter-
(1993) has even claimed to be obsessed with validity. Validity has pretations. There are no “pure,” “raw” data, uncontaminated by
been defined and described in a variety of ways and for a variety human thought and action, and the significance of data depends
of purposes (LeCompte, Preissle, & Tesch, 1993; Maxwell, 1992; on how material fits into the architecture of corroborating data.
Talburt, 2004). Because validity as a term has been discussed, “Data analysis leads to a reconstruction of those constructions”
debated, contested, and redefined by qualitative researchers, we (Lincoln & Guba, 1985, p. 132). In other words, qualitative
focus on validity as characteristic of the standards of evidence dis- data and information are always already interpretations made by
course as it relates to qualitative inquiry. participants as they answer questions or by researchers as they
Validity is generally understood by educational researchers as write up their observations. Neither research participants nor
“the trustworthiness of inferences drawn from data” (Eisenhart researchers can be neutral, because, as emphasized earlier, they are
& Howe, 1992, p. 644). Indeed, Scheurich (1993) remarked that always positioned culturally, historically, and theoretically. There
validity is the boundary line (that differs among epistemologies) is no Archimedean standpoint outside human activity (Alexander,
for what is acceptable and not acceptable in research. Both Moss 2006; Hartsock, 1983) from which to claim neutrality and pro-
(1996) and Mishler (1990) echoed Scheurich’s point that valid- duce value-free data.
ity is an epistemological issue that methodological procedures can In a now classic consideration of qualitative methods in edu-
barely begin to address. Thus, Moss warned of the imposition of cation, Erickson (1986) asserted that
“a priori criteria abstracted from existing practice” (p. 26) because
of the very different theoretical work done under the umbrella of the corpus of materials collected in the field are not data themselves,
qualitative research. Lather (2001) summarized these cautions but resources for data. Fieldnotes, videotapes, and site documents
with her reminder that ever since the publication of “Cronbach are not data. Even interview transcripts are not data. All these are
documentary materials from which data must be constructed
and Meehl’s 1955 essay on the problems with construct validity
through some formal means of analysis. (p. 149)
in psychological testing, validity has been the problem, not the
solution” (p. 243). Driving these discussions is the need for Nevertheless, most qualitative textbooks now use a definition of
researchers to address a central question in any kind of research data similar to Bogdan and Biklen’s preceding formulation.
inquiry: “Why should I believe this?” (Wallace & Wray, 2006, Although Bogdan and Biklen claimed that “data are both the evi-
p. 28). How then do qualitative researchers deal with validity in dence and the clues” (p. 117), Lincoln (2002) cautioned that
their research? In the following sections, we show how qualitative “data and information are not evidence until two things happen:
researchers use data as evidence to warrant claims within different first, someone recognizes it as data, and second, an inquirer sub-
theoretical frameworks and specific communities of practice. jects it to some form of systematic analysis, which turns it into
These activities are separable only heuristically; in research prac- evidence directed toward some question or argument” (p. 6).
tice, they intertwine and implicate one another throughout the
course of any study. Claims and Interpretations
In research, claims are statements of meaning grounded in
Data and Evidence evidence and theory. Claims describe, interpret, deconstruct,
Qualitative researchers’ concerns about the quality of their work critique, predict, and explain lived experience. Claims are state-
are evident in discussions about formulating both research design ments that connect the world bounded by our data to our inter-
and questions within explicit theoretical and philosophical tradi- preted understanding of that data.
tions; accessing and entering settings; selecting, collecting, and In 1962, Raoul Naroll, an anthropologist, working from con-
analyzing data; and building a case for conclusions. Quality is cerns about the misuse of data, called for data quality control.
constructed and maintained continuously throughout the life of Naroll tried to assess information by the conditions under which
a research project and includes decisions that researchers make as it was generated and the extent to which it compared and con-
they interact with those they study and as they consider their trasted with other information. Because he studied patterns of
analyses, interpretations, and representations of data. Qualitative cross-cultural behavior and experience, he was concerned about
researchers in education have used, generated, and redefined var- the accuracy of reports describing cultures across the world and
ious terms that relate to the assessment of quality shared with whether they were biased by incomprehension, misapprehension,
other social, human, and professional science researchers: credi- or downright malice. Although in the 21st century, people are
bility, validity, triangulation, trustworthiness, truth, and verifica- better equipped to record information almost everywhere about
tion. Qualitative scholars have seldom used the terms standards, almost everyone, misunderstandings, alternative understandings,

JANUARY/FEBRUARY 2007 27
and conflicting standpoints continue to complicate the genera- knowledge and lore have long been exploited by researchers are
tion of knowledge, and most scholars recognize that the multiple well advised to consider the ethical and political implications of
layers of meaning in human experience can be inconsistent, inco- decisions concerning all aspects of doing research (e.g., L.T. Smith,
herent, and even incomprehensible as well as sometimes consis- 1999, 2005).
tent, coherent, and comprehensible. The material that constitutes Only recently have some qualitative scholars begun to explic-
data and the constituting of the data are limited and fallible, and itly address the issue of quality using the terminology of “stan-
the scholar’s task is to identify the limitations and faults of infor- dards of evidence.” For example, Wilson (1994) proposed five
mation along with its value. criteria that address the nature of the information, how it is
Justifying claims also depends on demonstrating and report- acquired, and how it is interpreted: “evidence should be consis-
ing appropriate and adequate methods of data generation. Naroll tent with a researcher’s chosen epistemology or perspective”
(1962) discussed sets of general criteria that might characterize (p. 26), “evidence should be observable” (p. 28), “evidence
competent work (for critiques of what is called criteriology, see, should be gathered through systematic procedures” (p. 29), “evi-
e.g., Schwandt, 1996; J. K. Smith & Deemer, 2000; J. K. Smith & dence should be shared and made public” (p. 30), and “evidence
Hodkinson, 2005). From expectations of careful documentation should be compelling” (p. 30).
of systematic fieldwork such as those developed by Naroll, suc- Lincoln (2002) offered another set of criteria: (a) “researchers
ceeding scholars have developed expectations for the conduct of should have been deeply involved and closely connected to the
quality research. These include attention to (a) thorough descrip- scene”; (b) “researchers should achieve enough distance from the
tion of design and methods in reports, (b) adequate demonstra- phenomenon to permit recording action and interpretations
tion of the relationship of claims to data, and (c) thoughtful relatively free of the researcher’s own stake”; (c) “claims should
consideration by the researcher of the strengths and limitations be based on an adequate selection of the total corpus of data”;
of the study. Next, we consider each of these issues in turn. (d) “data should come, at least partly, from publicly accessible
Thorough description of design and methods is the effort to observation records”; and (e) “data and analysis should include
represent decisions, procedures, and researcher thinking in ways consideration of inferences and interpretations, as well as con-
that audiences find recognizable and comprehensible. Practices crete phenomena” (p. 9). These are two possibilities for assessing
that support this effort can be contrasted with the mystique fos- standards of evidence that may fit some qualitative traditions.
tered among some artists who fear that demystifying or describ- The introductory texts scholars use to teach qualitative inquiry
ing their work in detail might threaten the product rather than also offer suggestions for how researchers use evidence to support
enhance it. Thus, researchers commonly provide detailed descrip- their claims. Because the relational aspects of qualitative work are
tions of how they went about their studies, the problems they so important, scholars value extended time in the field, what Wax
encountered, and the reasoning on which they based their deci- (1971) called immersion. “Being there” matters. Qualitative
sions (e.g., Fine, 1991; Fordham, 1996; Peshkin, 1986; methodologists also encourage member checks: going back to
Valenzuela, 1999). These methodological explorations often participants and asking them, “Have I got it right?” Working
appear in anthologies of researchers’ accounts of their studies with other researchers—peer debriefers and research groups—to
(e.g., Behar & Gordon, 1995; Lareau & Shultz, 1996; St. Pierre help think about the complexity and ethics of the work is also rec-
& Pillow, 2000). ommended (Lincoln & Guba, 1985). And because most qualita-
The relationship of data and claims is demonstrated in research tive research is grounded in descriptive claims about the work,
reports by offering adequate and appropriate information for not only are sufficient data to support claims crucial, but
readers to reexamine and assess a researcher’s assertions and inter- researchers must be able to, in Geertz’s (1973, p. 10) words, “con-
pretations. At a minimum, sufficient data are cited in reports to trive somehow first to grasp” what is going on before they can
support each claim (e.g., Lortie’s [1975] study of teachers demon- represent it for others. Writing of anthropological ethnography,
strates the patterns he claimed by extensive quotations from what Geertz added,
the teachers had to say); at a maximum, researchers make data
Doing ethnography is like trying to read (in the sense of “construct
available to others for secondary research (e.g., Goldman-Segall,
a reading of”) a manuscript—foreign, faded, full of ellipses, inco-
1998, has made video data from her digital ethnography available herencies, suspicious emendations, and tendentious commentaries,
to readers on the Web). How much material is provided and but written not in conventionalized graphs of sound but in tran-
whether entire collections of field notes, interviews, and docu- sient examples of shaped behavior. (p. 10).
ments are made available to the public are conventions that vary
by qualitative tradition. Oral historians and ethnographers often Geertz has explained elsewhere (as cited in an interview with
archive their data, for example. Olson, 1991) that description as a rhetorical marker of validity
Finally, how researchers address the strengths and limitations surfaces in representations of research with a “sense of circum-
of their studies includes issues about the relationships of stantiality and of power in reserve [italics added] (if an anecdote
researchers and participants, the roles of researchers in their stud- or an example doesn’t sound strained but sounds like you’ve got
ies, and the ethics and politics of representation. Of course, these fifty others and this is the best one you chose)” (p. 249).
decisions intersect with decisions about how much detail to dis- Using multiple researchers, multiple methods of data collec-
close about methods and design and how to select data to support tion, and multiple theoretical analyses to complicate rather than
assertions. For example, researchers may decide to limit access to simplify knowledge production also provide warrants for our
material from their research studies for ethical reasons. In particular, claims. And, of course, peer review both during the research
researchers working with indigenous groups whose traditional process and prior to publication has been standard practice. There

28 EDUCATIONAL RESEARCHER
is no single marker of validity in qualitative inquiry, and the best Ritchie (2003) distinguished inferential generalization from
qualitative research uses many of the strategies just described and theoretical generalization. The former applies a proposition to
invents others specific to the particular study. Thus, validity can- settings and people other than those studied, and the latter
not be defined in advance by a certain procedure but must be applies constructs developed in a study to the generation and
attended to at all times as the study shifts and turns. refinement of theory. The legitimacy of these claims depends on
In contrast to the emphasis on what qualitative researchers how qualitative researchers use the literature and other sources to
ought to do, Erickson (1986) offered “five major types of evi- build their arguments. Inferential generalization in qualitative
dentiary inadequacy [italics added]”: (a) “inadequate amounts work revolves around careful comparison across settings and
of evidence,” (b) “inadequate variety in kinds of evidence,” people. Theoretical generalization, in contrast, may depend on
(c) “faulty interpretive status of evidence,” (d) “inadequate dis- how convincingly a researcher accounts for diverse patterns with
confirming evidence,” and (e) “inadequate discrepant case analy- a compelling construct. The ultimate judge for some claims is, of
sis” (p. 140). Researchers, of course, must establish an evidentiary course, the readers of the research, who decide whether claims
warrant for assertions or claims they make. Erickson wrote that made apply to their situations.
assertions generated during fieldwork “are tested and retested Theory and practice are inseparable in doing qualitative
against the data base: the corpus of fieldnotes, interview proto- research. As we have shown, researchers generate theory from
cols, site documents” (p. 146), and so forth. Here, Erickson did their data through a complex process of warranting their claims.
use the language of warrants and evidence to discuss the impor- Geertz (1973) wrote that this kind of theory needs to “stay rather
tance of disconfirming cases: closer to the ground” (p. 24). Such “theoretical formulations
hover so low over the interpretations they govern that they don’t
To test the evidentiary warrant for an assertion the researcher con- make much sense or hold much interest apart from them” (p. 25).
ducts a systematic search of the entire data corpus, looking for dis- But researchers also produce evidence on the basis of a priori the-
confirming and confirming evidence. . . . If the discrepant cases ory. These include macro-level theory such as positivism, social
outnumbered those that fitted the assertion, the assertion would
constructionism, Marxism, and feminism, as well as midlevel the-
not be warranted by the data. Even if most of the cases fitted the
assertion, the discrepant instances would be noted for subsequent
ories such as cognitive and linguistic theories. Making use of data
analysis. (p. 146) and information as evidence means “relying on background
knowledge and auxiliary hypotheses, of ladening data with the-
Yet qualitative researchers working from other theoretical per- ory” (Wylie, 2002, p. 169).
spectives disagree that there is such a thing as a disconfirming case So the social, material, political, and theoretical contexts of
(e.g., Lather, 1993; Scheurich, 2001). Another way to think research are ever present, because all forms of evidence “presup-
about data or evidence that fail to fit emerging patterns is to pose a society within which they are symbolically meaningful”
rework the patterns to better represent the data. This work pro- (Alford, 1998, p. 36). The contingency of evidence may trouble
duces general statements, interpretations, and theories. those positivists, logical positivists, and scientific realists who pro-
duced an alternative to logical positivism in the 1960s and 1970s.
Generalizing and Theorizing Those scholars use an objectivist, realist, and foundationalist epis-
The way claims are warranted in qualitative research typically cen- temology and seem to believe that because qualitative research
ters on the data, but researchers also use the literature relevant to acknowledges that science, evidence, and truth are contingent, it
studies, information collected by other scholars, and a variety of is not valid, or, as the What Works Clearinghouse implies, it is
other sources to justify their claims. People unfamiliar with qualita- “weak.” Interpretivists, on the other hand, suspect that positivists
tive research assume that knowledge produced is not generalizable and scientific realists deny the role of theory, culture, and politics
in the sense that it does not make what Kaplan (1964), a philoso- in research when they assume, for instance, that “to describe the
pher of science, called nomological generalizations, assertions that physical and social world scientifically . . . multiple observers can
are “truly universal, unrestricted as to time and space . . . always and agree on what they see” (NRC, 2002, p. 25), with no considera-
everywhere the case, provided only that the appropriate conditions tion for how the knower shapes the known.
are satisfied” (p. 91). Kaplan himself, however, distinguished among
a variety of generalizations, assertions, and claims. Nomological or Conclusion
lawlike generalization rarely carries the weight in interpretive As we suggest in the title of this article and throughout, we intend
research that it does in positivist research, because the goal is not to our work here to be the thoughts of one group of qualitative
generalize to predict and control but rather to describe what people researchers, not an authoritative account of standards of evidence
do and say within local contexts. These particularistic generaliza- in qualitative inquiry. Although we have described in some depth
tions or assertions on the basis of the direct experience of observa- the many systematic and scientific ways qualitative researchers
tion and interviewing, called generic propositions by Kaplan and conduct their studies, we emphasize that we are not advocating
naturalistic generalizations by Stake and Trumbull (1982), are the for a set of standards of evidence that may be taken up by others
goal of much qualitative work. Lewis and Ritchie (2003) used the and used as a checklist to police our work—quite the contrary.
term representational generalization for similar assertions to empha- We call on other researchers, both qualitative and quantitative,
size that the particularistic generalization should be representative of to resist current political forces seeking to impose a set of restric-
the context and participants studied. tive standards on educational research that serve only to control
Qualitative researchers often make connections across studies what research gets funded and conducted and, at the same time,
to establish the applicability of their work. For these, Lewis and to inhibit the creation of new research methodologies.

JANUARY/FEBRUARY 2007 29
 Although not all qualitative researchers agree about the appli- incitement in the continuing conversations about truth and
cation or purpose of quality standards or even whether there validity that have always preoccupied qualitative researchers as
should be standards, several common concerns stand out as hav- we struggle to generate epistemologies and methodologies that
ing significantly shaped the dialogue and debate on this issue. enable us to grapple with the complex world in which we live
Because the strength of qualitative research is the close contact and do science.
and connection that qualitative researchers have with research
participants, this contact has played a central role in shaping the NOTES
principles and quality standards that guide them. A guiding ques- 1
The disciplinary sources for qualitative research are multiple and
tion has been, How can we best listen to, work with, and repre- diverse, including social and cultural anthropology; qualitative sociol-
sent the people our work is intended to serve? ogy; professional studies such as education, law, business, counseling,
Elliott (2006), for example, favors educational research as social work, library science, medicine, nursing, and the health profes-
a practical science and rejects current conceptions of “practi- sions; history; psychology, especially clinical, developmental, and cogni-
cal rationality” cast in terms of “the science of measurement” tive studies; case traditions in areas such as political science and
(pp. 180–181). Federal government agencies such as the Institute economics; communications and journalism, especially investigative
of Education Sciences, policy makers, funding agencies, and reporting and media studies; fieldwork in the natural sciences, especially
biology, geology, astronomy; and the arts and humanities.
sources external to the educational research community and their 2
Approaches to research design are likewise multiple and overlapping,
constituents expect educational research to produce generalizable, including (but not limited to) ethnography, field study, community
unambiguous, and immediately applicable solutions to complex study, case study, life history and biographical study, phenomenological
educational problems. In contrast to this engineering model, approaches, conversation analysis, oral history, the variety of kinds of
qualitative researchers have long been engaged in a more prag- interview studies, document analysis and other historical study, survey
matic conversation. How can research generate useful, infor- study, autoethnography, narrative inquiry, connoisseurship, portraiture,
mational, and thought-provoking feedback or knowledge to action research, insider research, collaborative research, observational
relevant and interested communities of scholars and practition- study, and multimethods research including unobtrusive research.
3We support this claim with our analysis of the qualitative literature
ers? Rather than focusing on eliminating the subjectivity of the
researcher in a fruitless effort to attain objective knowledge, (Preissle, Freeman, deMarrais, Roulsont, & St. Pierre, n.d.)
qualitative researchers pursue how best to work with the fruit-
REFERENCES
ful positionings that each researcher brings to a project (e.g.,
Peshkin, 1988). Alexander, H. A. (2006). A view from somewhere: Explaining the para-
Throughout this article, we have emphasized the heterogeneity digms of educational research. Journal of Philosophy of Education,
40(2), 205–221.
of qualitative research practice. We five authors do not all agree on
Alford, R. R. (1998). The craft of inquiry: Theories, methods, evidence.
or even view similarly what we have written here. What we have New York: Oxford University Press.
tried to do is identify common notions of qualitative evidence and Altheide, D. L., & Johnson, J. M. (1994). Criteria for assessing
the standards qualitative researchers use to judge whether the interpretive validity in qualitative research. In N. K. Denzin & Y. S.
claims they make are warranted or justified by that evidence. As Lincoln (Eds.), Handbook of qualitative research (pp. 485–499).
readers of qualitative work, we look for indications of researcher Thousand Oaks, CA: Sage.
skepticism: Do authors of research reports reveal themselves as their American Educational Research Association (2002). Ethical standards of
own best critics? Do they discuss the limits and uncertainties of the American Educational Research Association: Cases and commentary.
their work? How forthright are they about competing interpreta- Washington, DC: Author.
tions and explanations for the patterns they claim? American Educational Research Association. (2006). Standards for
Representing the multiple layers of human experience is reporting on empirical social science research in AERA publications.
Retrieved September 22, 2006, from http://www.aera.net/
fraught with challenge, alternative, and limitation. But every-
uploadedFiles/Opportunities/StandardsforReportingEmpiricalSocial
thing is not, as some argue, “just a matter of opinion,” nor is Science_PDF.pdf
what makes a qualitative study good a simple matter of meeting Anderson, G. L., & Herr, K. (1999). The new paradigm wars: Is there
a checklist of criteria. First, not all criteria are relevant to the room for rigorous practitioner knowledge in schools and universities?
range and variety of qualitative work. Second, pedestrian work Educational Researcher, 28, 12–21, 40.
can satisfy criteria without producing anything worth knowing. Behar, R., & Gordon, D. A. (Eds.). (1995). Women writing culture.
Third, applying even the limited standards we have discussed in Berkeley: University of California Press.
this article is no small endeavor: It requires expert judgment. Benner, P. (2001). From novice to expert: Excellence and power in clinical
Fourth, expectations for good qualitative work have historically nursing practice (Comm. ed). Upper Saddle River, NJ: Prentice Hall.
been tailored to the particular theoretical frameworks, method- Bogdan, R. C., & Biklen, S. K. (2006). Qualitative research for education:
ological traditions, and substantive issues addressed. We think An introduction to theories and methods (5th ed.). Boston: Pearson
Education Group.
that this is the level at which conversations about quality should
Bullough, R. V., Jr., & Pinnegar, S. (2001). Guidelines for quality in
remain. For example, standards to judge quality for life history autobiographical forms of self-study research. Educational Researcher,
research may be inappropriate for microethnography. Fifth, 30, 13–21.
what we have tried to do in this article is to indicate how research Cronbach, L. J., & Meehl, P. E. (1955). Construct validity. Psychological
practice and theory interact with relationships built with partic- Bulletin, 52, 281–302.
ipants such that decisions about quality are made as the research Denzin, N. K. (2000). Aesthetics and the practices of qualitative inquiry.
progresses. We hope that our discussion here will be another Qualitative Inquiry, 6, 256–265.

30 EDUCATIONAL RESEARCHER
Duneier, M. (1999). Sidewalk. New York: Farrar, Straus. and qualitative design in educational research (2nd ed., pp. 315–356).
Eisenhart, M. A., & Howe, K. R. (1992). Validity in educational San Diego, CA: Academic Press.
research. In M. D. LeCompte, W. L. Millroy, & J. Preissle (Eds.), The Lenzo, K. (1995). Validity and self-reflexivity meet poststructuralism:
handbook of qualitative research in education (pp. 644–680). San Scientific ethos and the transgressive self. Educational Researcher, 24,
Diego, CA: Academic Press. 17–23, 45.
Elliott, J. (2006). Educational research as a form of democratic rational- Lewis, J., & Ritchie, J. (2003). Generalising from qualitative research. In
ity. Journal of Philosophy of Education, 40(2), 169–185. J. Ritchie & J. Lewis (Eds.). Qualitative research practice: A guide for
Ellis, C. (2000). Creating criteria: An ethnographic short story. Qualitative social science students and researchers (pp. 263–286). London: Sage.
Inquiry, 6, 273–277. Lincoln, Y. S. (1995). Emerging criteria for quality in qualitative and
Erickson, F. (1986). Qualitative methods in research on teaching. In interpretive research. Qualitative Inquiry, 1, 275–289.
M. C. Wittrock (Ed.), Handbook of research on teaching (3rd ed., Lincoln, Y. S. (2002, November). On the nature of qualitative evidence.
pp. 119–161). New York: Macmillan. Paper presented at the annual meeting of the Association for the Study
Feldman, A. (2003). Validity and quality in self-study. Educational of Higher Education, Sacramento, CA.
Researcher, 32, 26–28. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury
Fine, M. (1991). Framing dropouts: Notes on the politics of an urban pub- Park, CA: Sage.
lic high school. Albany: State University of New York Press. Lortie, D. (1975). Schoolteacher: A sociological study. Chicago: University
Flyvbjerg, B. (2001). Making social science matter: Why social inquiry fails and of Chicago Press.
how it can succeed again. Cambridge, UK: Cambridge University Press. Maxwell, J. A. (1992). Understanding and validity in qualitative
Fordham, S. (1996). Blacked out: Dilemmas of race, identity, and success research. Harvard Educational Review, 62(3), 279–300.
at Capital High. Chicago: University of Chicago Press. Maxwell, J. A. (1996). Qualitative research design: An interactive
Foucault, M. (1979). Discipline and punish: The birth of the prison (Alan approach. Thousand Oaks, CA: Sage.
Sheridan, Trans.). New York: Vintage. (Original work published 1975) Melrose, M. J. (2001). Maximizing the rigor of action research: Why
Geertz, C. (1973). Thick description: Toward an interpretive theory of would you want to? How could you? Field Methods, 13(2), 160–180.
culture. In C. Geertz, The interpretation of cultures: Selected essays Miles, M. B., & Huberman, A. M. (1984). Drawing valid meaning
(pp. 3–30). New York: Basic Books. from qualitative data: Toward a shared craft. Educational Researcher,
Goldman-Segall, R. (1998). Points of viewing children’s thinking: A digi- 13, 20–30.
tal ethnographer’s journey. Mahwah, NJ: Lawrence Erlbaum. Mishler, E. G. (1990). Validation in inquiry-guided research: The role
Hartsock, N. C. M. (1983). The feminist standpoint: Developing the of exemplars in narrative studies. Harvard Educational Review, 60(4),
ground for a specifically feminist historical materialism. In S. Harding 415–442.
& M. B. Hintikka (Eds.), Discovering reality: Feminist perspectives on Morse, J. M, Swanson, J. M., & Kuzel, A. J. (Eds.). (2001). The nature
epistemology, metaphysics, methodology and philosophy of science (pp. of qualitative evidence. Thousand Oaks, CA: Sage.
283–310). Boston: D. Reidel. Moss, P. A. (1996). Enlarging the dialogue in educational measurement:
Howe, K., & Eisenhart, M. (1990). Standards for qualitative (and quan- Voices from interpretive research traditions. Educational Researcher,
titative) research: A prolegomenon. Educational Researcher, 19, 2–9. 25, 20–28.
Kaestle, C. F. (1993) The awful reputation of education research. Naroll, R. (1962). Data quality control—A new research technique:
Educational Researcher, 22, 23–31. Prolegomena to a cross-cultural study of culture stress. New York: Free
Kaplan, A. (1964). The conduct of inquiry: Methodology for behavioral sci- Press.
ence. Scranton, PA: Chandler. The National Academies, Division of Behavioral and Social Sciences and
Koro-Ljungberg, M. (2004). Impossibilities of reconciliation: Validity Education. (2006). Standards of evidence: Strategic planning initiative.
in mixed theory projects. Qualitative Inquiry, 10, 601–621. Retrieved June 30, 2006, from http://www7.nationalacademies.org/
Kvale, S. (1995). The social construction of validity. Qualitative Inquiry, dbasse/Standards%20of%20Evidence%20Description.html
1, 19–40. National Research Council. (2002). Scientific research in education. R. J.
Lagemann, E. C. (2000). An elusive science: The troubling history of edu- Shavelson & L. Towne (Eds.). Committee on Scientific Principles for
cation research. Chicago: University of Chicago Press. Education Research.Washington, DC: National Academy Press
Lareau, A., & Shultz, J. (Eds.). (1996). Journeys through ethnography: National Research Council. (2005). Advancing Scientific Research in
Realistic accounts of fieldwork. Boulder, CO: Westview. Education. L. Towne, L. L. Wise & T. M. Winters (Eds.). Committee
Lather, P. (1993). Fertile obsession: Validity after poststructuralism. on Research in Education. Washington, DC: National Academy
Sociological Quarterly, 34(4), 673–693. Press.
Lather, P. (2001). Validity as an incitement to discourse: Qualitative Olson, G. (1991). The social scientist as author: Clifford Geertz on
research and the crisis of legitimation. In V. Richardson (Ed.), ethnography and social construction. Journal of Advanced
Handbook of research on teaching (pp. 241–250). Washington, DC: Composition, 11(2), 245–268.
American Educational Research Association. Peshkin, A. (1986). God’s choice: The total world of a fundamentalist
Lather, P. (2004, April). Getting lost: Feminist efforts toward a double(d) Christian school. Chicago: University of Chicago Press.
science. Paper presented at the annual meeting of the American Peshkin, A. (1988). In search of subjectivity: One’s own. Educational
Educational Research Association, San Diego, CA. Researcher, 17, 17–22.
Lather, P. (2006). Foucauldian scientificity: Rethinking the nexus of Preissle, J., Freeman, M., deMarrais, K., Roulston, K, & St. Pierre, E. A.
qualitative research and educational policy analysis. International (n.d.). How qualitative researchers discuss quality work. Retrieved
Journal of Qualitative Studies in Education, 19, 783–791. September 23, 2006, from http://www.coe.uga.edu/leap/qual/
LeCompte, M. D., & Goetz, J. P. (1982). Problems of reliability and research/index.html
validity in ethnographic research. Review of Educational Research, Richardson, L. (2000). Evaluating ethnography. Qualitative Inquiry, 6,
52(1), 31–60. 253–255.
LeCompte, M. D., Preissle, J., & Tesch, R. (1993). Evaluating qualita- Rorty, R. M. (Ed). (1967). The linguistic turn: Essays in philosophical
tive design. In M. D. LeCompte, & J. Preissle, (Eds.). Ethnography method. Chicago: University of Chicago Press.

JANUARY/FEBRUARY 2007 31
Ruccio, D. F., & Amariglio, J. (2003). Postmodern moments in modern Wolcott, H. F. (1994). Transforming qualitative data: Description, analy-
economics. Princeton, NJ: Princeton University Press. sis, and interpretation. Thousand Oaks, CA: Sage.
Scheurich, J. J. (1993). The masks of validity: A deconstructive investigation. Wylie, A. (2002). Thinking from things: Essays in the philosophy of archae-
International Journal of Qualitative Studies in Education, 9(11), 49–60. ology. Berkeley: University of California Press.
Scheurich, J. J. (2001). Research method in the postmodern. Philadelphia,
PA: Routledge. AUTHORS
Schwandt, T. A. (1996). Farewell to criteriology. Qualitative Inquiry, 2,
58–72. MELISSA FREEMAN is an assistant professor in the Department of
Seale, C. (2004). Quality in qualitative research. In C. Seale, G. Gobo, Lifelong Education, Administration, and Policy in the College of
J. F. Gubrium, & D. Silverman (Eds.), Qualitative research practice Education at The University of Georgia, 329 River’s Crossing, Athens,
(pp. 409–419). London: Sage. GA 30602; freeman9@uga.edu. Her research interests include transfor-
Smith, J. K., & Deemer, D. K. (2000). The problem of criteria in the mational and relational theories and methodologies in qualitative
age of relativism. In N. K. Denzin & Y. S. Lincoln (Eds.), The hand- research and evaluation, parental perceptions of public schooling, and
book of qualitative research (2nd ed., pp. 877–896). Thousand Oaks, philosophical hermeneutics.
CA: Sage.
Smith, J. K., & Hodkinson, P. (2005). Relativism, criteria, and politics. KATHLEEN DEMARRAIS is a professor in the Department of Lifelong
In N. K. Denzin & Y. S. Lincoln (Eds.), The handbook of qualitative Education, Administration and Policy and associate dean in the College
research (3rd ed., pp. 915–932). Thousand Oaks, CA: Sage. of Education at The University of Georgia, G3 Aderhold Hall, Athens,
Smith, L. T. (1999). Decolonizing methodologies: Research and indigenous GA 30602; kpd@uga.edu. Her research interests include the political
peoples. London: Zed. context of qualitative research, qualitative pedagogy, and the influence
Smith, L. T. (2005). On tricky ground: Researching the native in the age of neoconservative philanthropy on educational policy.
of uncertainty. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage hand-
JUDITH PREISSLE is a professor in the Department of Lifelong
book of qualitative research (3rd ed., pp. 85–108). Thousand Oaks: Sage.
Education, Administration, and Policy in the College of Education at
Soyfer, V. N. (1994). Lysenko and the tragedy of Soviet science (L. Gruliow
The University of Georgia, 303 River’s Crossing, Athens, GA 30602,
& R. Gruliow, Trans.). New Brunswick, NJ: Rutgers University Press.
and an affiliated member of UGA’s Institute for Women’s Studies;
Stake, R., & Trumbull, D. (1982). Naturalistic generalizations. Review
jude@uga.edu. She teaches, researches, and writes in educational anthro-
Journal of Philosophy and Social Science, 7(1), 1–12.
pology, qualitative research, feminist studies, and ethics.
St. Pierre, E. A., & Pillow, W. S. (Eds.). (2000). Working the ruins: Feminist
poststructural theory and methods in education. New York: Routledge. KATHRYN ROULSTON is an associate professor in the Department of
Talburt, S. (2004). Ethnographic responsibilities without the “real.” Lifelong Education, Administration, and Policy in the College of
Journal of Higher Education, 75(1), 80–103. Education at The University of Georgia, 324 River’s Crossing, 850
Towne, L., Wise, L. L., & Winters, T. M. (Eds.). (2005). Advancing sci- College Station Road, Athens, GA 30602; roulston@uga.edu. Her research
entific research in education. Washington, DC: The National interests include qualitative research methodology, qualitative interview-
Academies Press. ing, analysis of talk-in-interaction, and topics in music education.
Valenzuela, A. (1999). Subtractive schooling: US-Mexican youth and the
politics of caring. Albany: State University of New York Press. ELIZABETH A. ST. PIERRE is a professor of language and literacy edu-
Wallace, M., & Wray, A. (2006). Critical reading and writing for post- cation and a qualitative methodologist at The University of Georgia, 125
graduates. Thousand Oaks, CA: Sage. Aderhold Hall, Athens, GA 30602; stpierre@uga.edu. Her research inter-
Wax, R. H. (1971). Doing fieldwork: Warnings and advice. Chicago: ests include poststructural theories of language and subjectivity and a cri-
University of Chicago Press. tique of conventional qualitative inquiry.
Willis, P., & Trondman, M. (2000). Manifesto for ethnography.
Ethnography, 1, 5–16.
Wilson, B. J. (1994). A challenge to communication empiricists: Let’s Manuscript received June 30, 2006
be more forthcoming about what we do. Western Journal of Revision received September 25, 2006
Communication, 58(1), 25–31. Accepted September 28, 2006

32 EDUCATIONAL RESEARCHER