See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/228340166

The Validity Issues in Mixed Research

Article · January 2006

CITATIONS READS
477 22,330

2 authors, including:

R. Burke Johnson
University of South Alabama
103 PUBLICATIONS 17,298 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Mixed Methods in RT-PE View project

Toward a theory of causation in mixed methods research. View project

All content following this page was uploaded by R. Burke Johnson on 28 July 2014.

The user has requested enhancement of the downloaded file.

Copyright 2006 by the RESEARCH IN THE SCHOOLS
Mid-South Educational Research Association 2006, Vol. 13, No. 1, 48-63

The Validity Issue in Mixed Research

Anthony J. Onwuegbuzie
University of South Florida

R. Burke Johnson
University of South Alabama

In quantitative research, the importance of validity has been long accepted. In qualitative research,
discussions of validity have been more contentious and different typologies and terms have been
produced. In mixed methods research, wherein quantitative and qualitative approaches are combined,
discussions about “validity” issues are in their infancy. We argue that because mixed research
involves combining complementary strengths and nonoverlapping weaknesses of quantitative and
qualitative research, assessing the validity of findings is particularly complex; we call this the
problem of integration. We recommend that validity in mixed research be termed legitimation in
order to use a bilingual nomenclature. Tashakkori and Teddlie’s (2003, 2006) evaluation criteria
frameworks involving the concept of inference quality are summarized. Although providing a
framework for assessing legitimation in mixed research always will be incomplete, it is important to
address several legitimation types that come to the fore as a result of combining inferences from the
quantitative and qualitative components of the study into the formation of meta-inferences. Nine types
of legitimation are described here in order to continue this emerging and important dialogue among
researchers and methodologists.

This paper is focused on validity in mixed methods quantitative research quality. The “validity” issue, at
research or what we refer to more broadly as mixed least as we use the term, is not about singular truths,
research. However, to understand the validity issue and it certainly is not limited to quantitative
(i.e., quality) in mixed research, a brief review of some measurement; rather, by validity we mean that a
related discussions in quantitative and qualitative research study, its parts, the conclusions drawn, and the
research will be helpful for orientation. Because these applications based on it can be of high or low quality,
issues have been discussed elsewhere in great detail, or somewhere in between. Research needs to be
we provide only brief summaries of those literatures, defensible to the research and practice communities for
but first we want to make a few introductory comments whom research is produced and used. The arbiters of
about our general approach to research validity or research quality will be the research stakeholders,
quality. which means that the quality or validity issue can have
We try to take a “middle of the road” position, subjective, intersubjective, and objective components
seeing some truth and insight to be gained from and influences. At the same time, research is something
multiple perspectives. Our approach is only one among about which we can “rationally” speak, and usually,
many, and we recommend that readers examine after considering our external and our internal or
additional perspectives as more work is carried out in epistemic standards, we can meaningfully assert that
this emerging area in mixed methods research as well some research is of higher quality for certain purposes
as in the more traditional areas of qualitative and than is other research (Longino, 1990). Anthropology,
sociology, and psychology teach us that communities,
cultures, and various kinds of groupings (including
Correspondence should be addressed to Anthony J. communities of researchers) have some shared norms,
Onwuegbuzie, Dept. of Educational Measurement practices, values, and beliefs.
and Research, College of Education, University of We aim our sense of justification at the research
South Florida, 4202 East Fowler Ave., EDU 162, community that sees many advantages to sometimes
Tampa, Florida 33620. using both qualitative and quantitative research in their
Email: tonyonwuegbuzie@aol.com single or highly related sets of research studies. One of
the exciting results of much mixed research is that in a

Spring 2006 48 RESEARCH IN THE SCHOOLS

 ANTHONY J. ONWUEGBUZIE AND R. BURKE JOHNSON

single study practical questions can be addressed, Validity in Qualitative Research

different perspectives can be examined, and if well
documented, practitioners can obtain some sense of In the qualitative research paradigm, a primary
what might be useful in their local situations. We do focus is for researchers to capture authentically the
not want to oversell mixed research, however; the lived experiences of people. As noted by Denzin and
evidence will be in the results. If mixed research Lincoln (2005), “Such experience, it is argued, is
produces useful results over time, as well as useful created in the social text written by the researcher. This
theory, then progress will have been made. We agree is the representational problem. It confronts the
with Kurt Lewin’s statement that “There is nothing so inescapable problem of representation, but does so
practical as a good theory” (Lewin, 1952, p. 169), and within a framework that makes the direct link between
we hope that all researchers, including mixed experience and text problematic” (p. 19). 2 Denzin and
researchers, will attempt to produce good theories and Lincoln (2005) also argue for “a serious rethinking of
other research works. such terms as validity, generalizability, and reliability,
terms already retheorized in postpositivist…,
Validity in Quantitative Research constructivist-naturalistic…, feminist…, interpretive…,
poststructural…, and critical…discourses. This
In quantitative research, discussions of “validity” problem asks, ‘How are qualitative studies to be
have been common and the importance of validity has evaluated in the contemporary, poststructural
been long accepted, and this is well documented in the moment?’” (pp. 19-20). 3, 4 Part of their solution to the
literature. Building on the seminal works of Campbell “validity issue” has been to reconceptualize traditional
and Stanley (Campbell, 1957; Campbell & Stanley, quantitative validity concepts and to use labels that are
1963), and many others, Onwuegbuzie (2003) more acceptable to qualitative researchers (Lincoln &
presented 50 different threats to internal and external Guba, 1985, 1990). One set of criteria (Lincoln &
validity that might occur at the research design/data Guba, 1985) includes the following types: credibility
collection, data analysis, and/or data interpretation (replacement for quantitative concept of internal
stages of the quantitative research process. These validity), transferability (replacement for quantitative
threats are presented in Figure 1, in what was later concept of external validity), dependability
called the Quantitative Legitimation Model. As (replacement for quantitative concept of reliability),
illustrated in Figure 1, Onwuegbuzie identified 22 and confirmability (replacement for quantitative
threats to internal validity and 12 threats to external concept of objectivity).
validity at the research design/data collection stage of Another useful classification for validity in
the quantitative research process. At the data analysis qualitative research was provided by Maxwell (1992),
stage, 21 and 5 threats to internal validity and external who identified the following five types of validity in
validity were conceptualized, respectively. Finally, at qualitative research: descriptive validity (i.e., factual
the data interpretation stage, 7 and 3 threats to internal accuracy of the account as documented by the
validity and external validity were identified, researcher), interpretive validity (i.e., the extent to
respectively. In Figure 2, Onwuegbuzie, Daniel, and which an interpretation of the account represents an
Collins’ (in press) have presented a schematic understanding of the perspective of the underlying
representation of instrument score validity, which also group and the meanings attached to the members’
is provided here for review by interested readers. words and actions), theoretical validity (i.e., the degree
Another very important work in validity in to which a theoretical explanation developed from
quantitative research is found in Shadish, Cook, and research findings is consistent with the data),
Campbell (2001). These authors continue to build on evaluative validity (i.e., the extent to which an
Campbell’s earlier work and classify research validity evaluation framework can be applied to the objects of
into four major types: statistical conclusion validity, study, as opposed to a descriptive, interpretive, or
internal validity, construct validity, and external explanatory one), and generalizability (i.e., the extent
validity. Other selected seminal works showing the to which a researcher can generalize the account of a
historical development of validity in quantitative particular situation, context, or population to other
research are summarized in the following references: individuals, times, settings, or context). With regard to
American Educational Research Association, American the last validity type, Maxwell differentiates internal
Psychological Association, and National Council on generalizability from external generalizability, with the
Measurement in Education (1999), Bracht and Glass former referring to the generalizability of a conclusion
(1968), Campbell (1957), Campbell and Stanley within the underlying setting or group, and the latter
(1963), Cook and Campbell (1979), Messick (1989, pertaining to generalizability beyond the group, setting,
1995), and Smith and Glass (1987). time, or context. According to Maxwell, internal

Spring 2006 49 RESEARCH IN THE SCHOOLS

 THE VALIDITY ISSUE IN MIXED RESEARCH

generalizability is typically more important to embodied) legitimation, descriptive validity, structural

qualitative researchers than is external generalizability corroboration, theoretical validity, observational bias,
(see also, Maxwell, 2005). researcher bias, reactivity, confirmation bias, illusory
Onwuegbuzie and Leech (in press-a) correlation, causal error, and effect size. Also in this
conceptualized what they called the Qualitative model, the following threats to external credibility
Legitimation Model, which contains 29 elements of were identified as being pertinent to qualitative
legitimation for qualitative research at the following research: catalytic validity, communicative validity,
three recursive stages of the research process: research action validity, investigation validity, interpretive
design/data collection, data analysis, and data validity, evaluative validity, consensual validity,
interpretation.1 As illustrated in Figure 3, the following population generalizability, ecological generalizability,
threats to internal credibility are viewed as pertinent to temporal generalizability, researcher bias, reactivity,
qualitative research: ironic legitimation, paralogical order bias, and effect size.
legitimation, rhizomatic legitimation, voluptuous (i.e.,

Figure 1. Threats to Internal and External Validity

Th reats to Ext ernal

Val idity/External
R ep licatio n

Popu lati on Valid ity

Ecolo gic al Valid ity
Tempo ral Valid ity
Mul tipl e-Trea tm en t In terfe rence
Popu lati on Valid ity
Rese arch er Bia s
Res e arch er Bia s Popu lati on Valid ity
Reactive Arrange ments
Sp eci fic ity of Vari able s Ecolo gical Valid ity
Ord er Bia s
Matchin g Bia s Tempo ral Valid ity
Matchin g Bia s
Mi s-Specifi cation Error
Sp eci ficity of Vari able s
Trea tmen t Diffusio n
Pre test x Trea tm en t In teractio n
Se lection x Trea tmen t In teractio n

Res e arch
Data Data
Desig n/Data
Analysi s Interpretatio n
Co llectio n

His tory
Matu ratio n Effe ct Size
Te s tin g Co nfirm atio n Bia s
Instru m entatio n Sta tistica l Regre ssio n
Sta tistica l Regre ss io n Sta tistica l Regre ssio n Di stort ed Grap hics
D ifferen tial Sele cti on o f Partici pant s Restri cte d Rang e Illu sory Correl atio n
Mortal ity Mortal ity Crud Factor
Selectio n In teractio n Effects Non-In teractio n See kin g Bia s Posi tive Man ifol d
Im ple mentatio n Bia s Type I - Typ e X Error Cau sal Error
Sa m ple Aug m entatio n Bia s Observationa l Bia s
Beha vi or Bia s Res e arch er Bia s
Ord er Bia s Matchin g Bia s
Observat iona l Bia s Treatment Repli cation Error
Rese arch er Bia s Vio late d As sum p tion s
Matchin g Bia s Mu lticoll inea rity
Treat ment Repli cation Error Mi s-Specifi cation Error
Evalu atio n An xi ety
Mul tipl e-Trea tm en t In terfe rence
Reactive Arrange ments
Trea tmen t Diffusio n
Time x Trea tm en t In teractio n
History x Trea tm en t In teractio n

Threats to Internal
Va lidi ty/Internal
Rep licatio n

Spring 2006 50 RESEARCH IN THE SCHOOLS

 ANTHONY J. ONWUEGBUZIE AND R. BURKE JOHNSON

Figure 2. Schematic Representation of Instrument Score Validity

LogicallyBased Empiricallybased

Content- Criterion- Construct-

Related Validity Related Validity Related Validity

Concurrent Predictive
Generalizability
Validity Validity

Sampling Substantive Outcome

Face Validity Item Validity
Validity Validity Validity

Structural Comparative
Validity Validity

Convergent Discriminant Divergent

Validity Validity Validity

Validity in Mixed Research

Because of the association with the quantitative Mixed research involves the mixing of quantitative
conceptualization of the research process, the term and qualitative methods or paradigm characteristics
validity has generally been replaced by the term into research studies (Johnson & Onwuegbuzie, 2004;
trustworthiness within qualitative research. The vast Onwuegbuzie & Johnson, 2004; Tashakkori & Teddlie,
and important literature on trustworthiness is 1998, 2003). According to the fundamental principle of
exemplified and discussed in the following references mixed research, it often should involve the combining
from the qualitative research literature: Creswell of quantitative and qualitative methods, approaches,
(1998), Glaser and Strauss (1967), Kvale (1995), and concepts that have complementary strengths and
Lather (1986, 1993), Lincoln and Guba (1985, 1990), nonoverlapping weaknesses (Brewer & Hunter, 1989;
Longino (1995), Maxwell (1992, 1996), Miles and Johnson & Turner, 2003). This principle is meant to be
Huberman (1984, 1994), Onwuegbuzie and Leech (in viewed broadly; it is not limited to triangulation or
press-a), Schwandt (2001), Strauss and Corbin (1998), corroboration. The words “complementary strengths”
and Wolcott (1990). are meant to include all of the strengths of qualitative

Spring 2006 51 RESEARCH IN THE SCHOOLS

 THE VALIDITY ISSUE IN MIXED RESEARCH

and quantitative research. Therefore, the principle can legitimation refers to the difficulty in obtaining
be used for the five traditional purposes of mixed findings and/or making inferences that are credible,
research identified by Greene, Caracelli, and Graham trustworthy, dependable, transferable, and/or
(1989). By “complementary strengths” we are implying confirmable. Indeed, in many instances, these problems
a putting together of different approaches, methods, are exacerbated in mixed research because both the
and strategies in multiple and creative ways. quantitative and qualitative components of studies
Mixed research still is plagued by the problems of bring into the setting their own problems of
representation, integration, and legitimation representation and legitimation, likely yielding either
(Onwuegbuzie, in press). The problem of an additive or a multiplicative threat—hence the
representation refers to the difficulty in capturing (i.e., problem of integration.
representing) lived experiences using text in general Mixed research can be conceptualized as
and words and numbers in particular. The problem of combining quantitative or qualitative research in a

Figure 3. Qualitative Legitimation Model

Threats to
External Credibility

Population Generalizability Catalytic Validity

Ecological Generalizability Communicative Validity
Temporal Generalizability Action Validity
Researcher Bias Investigation Validity
Reactivity Interpretative validity
Order Bias Evaluative Validity
Effect size Consensual Validity
Data
Interpretation

Research
Data Design/ Theoretical
Analysis Data Validity
Collection

Ironic Legitimation
Confirmation Bias
Paralogical Legitimation Observational Bias
Descriptive Observational Bias Illusory Correlation
Rhizomatic Legitimation Researcher Bias
Validity Researcher Bias Causal Error
Embodied Legitimation Reactivity
Effect Size
Structural Corroboration

Threats to
Internal Credibility

Spring 2006 52 RESEARCH IN THE SCHOOLS

 ANTHONY J. ONWUEGBUZIE AND R. BURKE JOHNSON

concurrent, sequential, conversion (Tashakkori & data from quantitative to qualitative or from qualitative
Teddlie, 2003; Teddlie & Tashakkori, 2006), parallel to quantitative. Moreover, conversion occurs via
(Onwuegbuzie & Leech, 2004a), or fully mixed techniques such as quantitizing data (i.e., transforming
(Leech & Onwuegbuzie, 2005; Teddlie & Tashakkori, the qualitative data to a numerical form; Tashakkori &
2006) manner. Quantitative and qualitative approaches Teddlie, 1998) or qualitizing data (i.e., converting
can be combined in these ways whether the study quantitative data into data that can be analyzed
represents primary research (Johnson & Onwuegbuzie, qualitatively; Tashakkori & Teddlie, 1998). Both data
2004; Teddlie & Tashakkori, 2006) or a mixed types are analyzed/re-analyzed, and inferences are
synthesis of the extant literature (i.e., integrating the made based on both sets of analyses.
findings from both quantitative and qualitative studies In parallel mixed designs, the data are collected
in a shared area of empirical research; Sandelowski, and analyzed separately. In this respect it is similar to
Voils, & Barroso, 2006). Further, quantitative and concurrent designs. However, while inferences are
qualitative approaches can be combined in these ways made in concurrent designs on both sources of data in
regardless of which approach has priority in the study an integrated manner, in parallel mixed designs, each
(cf. Creswell, Shope, Plano Clark, & Green, 2006). In data source leads to its own set of inferences, and no
basic concurrent mixed designs, the following three attempt is made to reach what Tashakkori and Teddlie
conditions hold: (a) both the quantitative and (2003) refer to as a “meta-inference” (p. 686), in which
qualitative data are collected separately at both sets of inferences are combined into a coherent
approximately the same point in time, (b) neither the whole. Such designs lead either to (a) two separate
quantitative nor qualitative data analysis builds on the reports that would be presented or published separately
other during the data analysis stage, and (c) the results or (b) two separate write-ups that are presented in two
from each type of analysis are not consolidated at the distinct sections of the same report. Whereas some
data interpretation stage, until both sets of data have researchers do not consider these designs as
been collected and analyzed separately, and (d) after representing mixed research (e.g., Yin, 2006) but rather
collection and interpretation of data from the quasi-mixed designs (e.g., Teddlie & Tashakkori,
quantitative and qualitative components, a meta- 2006), other researchers do (e.g., Onwuegbuzie &
inference is drawn which integrates the inferences Leech, 2004a). Given the formative stage of mixed
made from the separate quantitative and qualitative research, we see, as does Greene (2006), these current
data and findings. disagreements as being good for the field as it
In basic sequential mixed designs, data collected continues to develop through hard conceptual and
and analyzed from one phase of the study (i.e., empirical work.
quantitative/qualitative data) are used to inform the Fully mixed research designs (Leech &
other phase of the investigation (i.e., Onwuegbuzie, 2005), also known as fully integrated
qualitative/quantitative data). Here, the data analysis mixed research designs (Tashakkori & Teddlie, 2003;
begins before all the data are collected. At the highest Teddlie & Tashakkori, 2006), involve mixing
level of integration, referred to as sequential mixed quantitative and qualitative approaches in an
model studies, “multiple approaches to data collection, interactive way at all stages of the investigation (i.e.,
analysis, and inference are employed in a sequence of research objective, type of data/operations, type of
phases. Each phase, by itself, may use a mixed analysis/inference; Onwuegbuzie & Johnson, 2004)
approach and provide conceptual and/or such that at each stage, one approach (e.g.,
methodological grounds for the next one in the chain” quantitative) influences the formulation of the other
(Tashakkori & Teddlie, 1998, pp. 149-150). Sequential approach (e.g., qualitative). We agree with Teddlie and
mixed designs also can be applied when conducting Tashakkori (2006) that fully mixed (and nearly fully
what Chen (1990, 2006) conceptualizes as theory- mixed designs) are attractive because of the multiple
driven evaluations, via the following two strategies: (a) points of integration.
switch strategy (e.g., first applying qualitative methods Because of the complexity involved in combining
to illuminate program theory of stakeholders and then qualitative and quantitative studies either in a
use quantitative methods to assess the program theory) concurrent, sequential, conversion, parallel, or fully
and (b) contextual overlaying strategy (e.g., utilizing mixed manner, mixed research gives rise to what we
qualitative approaches to collect contextual information call the problem of integration. Surrounding this
for facilitating the interpretation of quantitative data or problem is the extent to which combining quantitative
reconciling findings). and qualitative approaches can address each of Greene
Conversion mixed designs involve data et al. (1989) five empirically derived, general purposes
transformation wherein one data form is converted into of mixed-methodological research studies: (a)
the other and then subsequently analyzed (Teddlie & triangulation (i.e., seeking convergence and
Tashakkori, 2006). That is, the other data type evolves corroboration of findings from different methods that
from the original data type either by converting the study the same phenomenon); (b) complementarity

Spring 2006 53 RESEARCH IN THE SCHOOLS

 THE VALIDITY ISSUE IN MIXED RESEARCH

(i.e., seeking elaboration, illustration, enhancement, pragmatism, we mean to search for workable solutions
and clarification of the findings from one method with through the practice of research (e.g., follow the
results from the other method); (c) development (i.e., fundamental principle of mixed research, including the
using the findings from one method to help inform the use of designs and criteria that are situation and context
other method); (d) initiation (i.e., discovering appropriate) to help answer questions that we value and
paradoxes and contradictions that lead to a re-framing to provide workable improvements in our world (i.e.,
of the research question); and (e) expansion (i.e., help in bringing about desired outcomes). Our
seeking to expand the breadth and range of inquiry by pragmatism includes a healthy dose of pluralism by
using different methods for different inquiry which we mean that it is not logically contradictory to
components). More generally, the problem of claim that quantitative and qualitative research are both
integration pertains to the extent to which combining useful, even if, at times, they appear to be
quantitative and qualitative research techniques contradictory; perhaps what is seen as contradictory are
addresses one or more of Collins, Onwuegbuzie, and different perspectives that are complementary and
Sutton’s (2006) four rationales for mixing or enable one to more fully to see his or her world.
combining qualitative and quantitative approaches: Further, different standards of quality will be useful for
participant enrichment (i.e., mixing quantitative and different people in different contexts (see Patton’s,
qualitative techniques to optimize the sample using 2002, five different sets of criteria for judging the
techniques that include recruiting participants, quality of qualitative research), which is reasonable as
engaging in activities such as Institutional Review long as one makes these standards clear to avoid
Board debriefings, and ensuring that each participant arguments based on equivocation (which can produce
selected is appropriate for inclusion), instrument misunderstandings based on nothing more than
fidelity (e.g., assessing the appropriateness and/or different meanings of terms being used by different
utility of existing instruments; creating new people because they “talk past” one another). Clarity of
instruments; performance of human instruments), language use is especially important when people from
treatment integrity (i.e., assessing fidelity of different communities are the interlocutors. Arguments
intervention), and significance enhancement (e.g., about values can be healthy, and at this time more
facilitating thickness and richness of data; augmenting discussion among qualitative, quantitative, and mixed
interpretation and usefulness of findings). researchers about values needs to take place in order to
The problem of integration motivates us to ask understand better each other. Different researchers
questions such as the following: Is it misleading to have different values and beliefs about research
triangulate, consolidate, or compare quantitative approaches for addressing important questions, and this
findings and inferences stemming from a large random diversity when put together is not a problem; we see it
sample on equal grounds with qualitative data arising as a potential strength of research and practice,
from a small purposive sample? How much weight especially when simple and clear solutions are not
should be placed on quantitative data compared to readily forthcoming. In short, diversity of this sort is
qualitative data? Are quantitatively confirmed findings not a problem needing to be fixed by someone. Our
more important than findings that emerge during a pragmatism also is eclectic, by which we are referring
qualitative study component? When findings conflict, to the inclusion of multiple quantitative and qualitative
what is one to conclude? techniques in one’s briefcase and then selecting
Before discussing the issue of integration more combinations of assumptions, methods, and designs
carefully, we will point out that we do not believe that that best fit one’s research questions of interest.
the goal of mixed research is to replace either In every mixed research study researchers must
quantitative or qualitative research. Rather, the goal of deal with the problems of representation, legitimation,
this third type of research is to utilize the strengths of and integration but discussions about validity issues
two or more approaches by combining them in one that characterize these problems are still in relative
study, and by attempting to minimize the weaknesses infancy. Developing justified inferences is at the center
of approaches in mixed designs. Philosophically, of many problems in mixed research. In fact, Teddlie
mixed research generally follows philosophical and and Tashakkori (2003) and Tashakkori and Teddlie
methodological pragmatism (with a very broad and (2003) identified drawing inferences as one of the six
inclusive ontological realism where virtually unresolved issues and controversies in mixed research.
everything a qualitative or quantitative researcher The purpose of the remainder of this paper is to
deems to be real can be considered, in some sense, to contribute to the present dialogue about validity
be real, including subjective realism, intersubjective (trustworthy or defensibility or quality) issues in mixed
realism, and objective realism). (See Johnson & research. We will discuss the following three
Onwuegbuzie, 2004 for more discussion on the role of unresolved and, at times, contentious issues in the
pragmatism in mixed research, and Sanders, 1997 for domain of validity in mixed research:
more discussion about inclusive ontology.) By

Spring 2006 54 RESEARCH IN THE SCHOOLS

 ANTHONY J. ONWUEGBUZIE AND R. BURKE JOHNSON

(a) Labels for criteria for assessing mixed research term validity or that qualitative researchers should
studies; cease using terms such as trustworthiness, credibility,
(b) conceptualization of legitimation in mixed plausibility, and dependability. It only is to suggest that
research studies; and in the context of discussing the overall criteria for
(c) identifying some types of legitimation for assessment of mixed research studies, we recommend
mixed research. that the term legitimation, or a similarly descriptive and
inclusive term, be used.
Label for Criteria for Assessing Mixed Research
Studies Conceptualization of Legitimation in Mixed Research
As noted by Teddlie and Tashakkori (2003), a Studies
primary decision that confronts the field of mixed In one of the very few essays written on the topic
research is what to call the concept of validity in mixed of validity or quality criteria in mixed research, Teddlie
research. Although the term “validity” is routinely used and Tashakkori (2003) stated that mixed methods
in quantitative research, this term is disliked by many researchers “should adopt a common nomenclature
qualitative researchers. In fact, as noted by Schwandt transcending the separate QUAL and QUAN
(2001), some qualitative researchers object to the orientations when the described processes (QUAL and
concept of validity based on their rejection of the QUAN) are highly similar and when appropriate
correspondence theory of truth. They argue that terminology exists” (p. 12). Because inferences are
because validity is the test of this correspondence, made in research studies regardless of whether the
validity does not exist because there simply is no single associated interpretation is inductive or deductive in
reality, with truth being partially arbitrary as nature, these authors contended that the concept of
individuals interact with their worlds. These “inference” transcends quantitative and qualitative
researchers also believe in fallabilism (i.e., all research and they recommended that inference quality
procedures for establishing legitimation represent be used as the mixed research term for validity. This
“fallible means of making a case for a plausible and use has much merit, and we attempt to build on it
credible account”; Schwandt, 2001, pp. 268-269). below.
Some qualitative researchers, although they believe Teddlie and Tashakkori (2003) conceptualized
that some validities are useful, contend that validity is inference quality as being associated with the following
always relative to a particular context, situation, two research components: design quality and
language system, or worldview (Schwandt, 2001). interpretive rigor. Design quality refers to the standards
These researchers refer to contextualization (i.e., used for the evaluation of the methodological rigor of
legitimation represents the standards set by a particular the mixed research study, whereas interpretive rigor
community at a specific time and place). Some pertains to the standards for evaluating the validity of
qualitative researchers refute any relationship between conclusions. Teddlie and Tashakkori also presented the
validity and objectivism, the latter of which is viewed term inference transferability to denote the
as foundational. These researchers are referred to as generalizability of the findings (for both quantitative
representing strong or radical relativism (i.e., no single and qualitative research), which comprises population
account can be judged as being superior to any other). transferability (i.e., transferability to other individuals,
Some qualitative researchers (i.e., postmodernists) groups, or entities), ecological transferability (i.e.,
view the concept of validity (and the word) as transferability to other contexts or settings), temporal
representing a debunked modernist perspective that transferability (i.e., transferability to other time
champions universal rationality, rules, order, logic, and periods), and operational transferability (i.e.,
the like. Thus, we conclude that use of the word transferability to other methods of measuring
validity in mixed research can be counterproductive. behaviors). Teddlie and Tashakkori appropriately
One attractive solution to this problem is for mixed differentiated data quality from inference quality. What
researchers to use an alternative word that is more also is appealing about Teddlie and Tashakkori’s
acceptable to both quantitative and qualitative conceptualization of inference quality is their
researchers. This solution involves what Teddlie and identification of the following four (non-exhaustive
Tashakkori (2003) refer to as “using a bilingual and not mutually exclusive) criteria for evaluation: (a)
nomenclature” (p. 12). In this respect, a possible term within-design consistency (i.e., “consistency of the
that might be acceptable to both quantitative and procedures/design of study and from which the
qualitative investigators is legitimation. This would be inference emerged”; p. 40); (b) conceptual consistency
consistent with its use in the Onwuegbuzie (2003) (i.e., “degree to which the inferences are consistent
Quantitative Legitimation Model and the Onwuegbuzie with each other and with the known state of knowledge
and Leech (in press-a) Qualitative Legitimation Model and theory”; “consistency of inferences with each other
presented in Figures 1 and 3. That is not to suggest that within a study [cross-inference consistency]”; and
quantitative researchers should refrain from using the “consistency of inference with current state of

Spring 2006 55 RESEARCH IN THE SCHOOLS

 THE VALIDITY ISSUE IN MIXED RESEARCH

knowledge and theory [theoretical consistency]”; p. should be seen as a continuous process rather than as a
40); (c) interpretive agreement (or consistency) (i.e., fixed attribute of a specific research study. Mixed
“consistency of interpretations across people”; p. 40); research tends to be iterative and interactive
and (d) interpretive distinctiveness (i.e., the “degree to (Onwuegbuzie & Johnson, 2004) such that, in a sense,
which the inferences are distinctively different from inference closure (i.e., being able to make definitive
other possible interpretations of the results and rival statements about the quality of inferences made) might
explanations are ruled out” p. 40). never be fully reached within a particular study or even
Building on the work of Teddlie and Tashakkori over a series of systematically linked studies. We look
(2003), Tashakkori and Teddlie (2006) proposed an forward to future dialogue about these issues as we all
integrative model of quality that also comprises design try to advance the field forward.
quality and interpretive rigor. According to their
model, design quality comprises (a) within-design Some Types of Legitimation for Mixed Research
consistency (as defined earlier), (b) design suitability As noted earlier, the problems of representation
(i.e., whether the methods of the study are appropriate and integration in mixed research suggest the need to
for addressing the research question(s); and the design identify specific legitimation issues that are not
is consistent with the research question), (c) design associated with monomethod designs. However, these
fidelity (i.e., whether the procedures are implemented legitimation issues are not addressed fully in Teddlie
with quality and rigor; the methods are capable of and Tashakkori’s (2003) and Tashakkori and Teddlie’s
capturing meaning, associations, or effects; and the (2006) inference quality frameworks, nor do these
components of the design such as sampling and data issues appear to have been addressed, to date, in any
collection procedures, are implemented adequately); other framework. Thus, we now will outline a new
and (d) analytic adequacy (i.e., whether the data typology of legitimation types in mixed research for
analysis techniques are appropriate for addressing the consideration, dialog, and refinement. Our typology
research question(s)). Interpretive rigor consists of (a) currently is in its infancy, and it contains nine
interpretive agreement (as defined earlier), (b) legitimation types. These legitimation types are
interpretive distinctiveness (as defined earlier), (c) summarized in Table 1. Each of these types of
interpretive consistency (i.e., whether the inferences legitimation is discussed next.
closely follow the relevant findings in terms of type, Sample integration legitimation. This legitimation
intensity, and scope; and the multiple inferences made type applies to situations in which a researcher wants to
on the basis of the findings are consistent with each make statistical generalizations from the sample
other), (c) theoretical consistency (i.e., whether the participants to a larger target population. Unless
inferences are consistent with theory and the state of exactly the same individuals or groups are involved in
knowledge in the field), and integrative efficacy (i.e., both the qualitative and quantitative components of a
whether the meta-inference adequately incorporates the study, constructing meta-inferences by pulling together
inferences stemming from quantitative and qualitative the inferences from the qualitative and quantitative
phases of the study). phases can be problematic. For example, a researcher
Teddlie and Tashakkori’s (2003) and Tashakkori might conduct a concurrent design in which inferences
and Teddlie’s (2006) conceptualizations present made from quantitative data yielded by a large random
inference as an outcome. However, as appealing and sample were integrated (i.e., into a meta-inference)
useful as their conceptualization is, we believe it needs with inferences made from qualitative data arising from
some elaboration and extension. We see useful a smaller subset of this sample or from an entirely
extensions of their model in two ways. First, we view different group of people. However, if this meta-
legitimation as a process, not just an outcome. Indeed, inference was generalized to the underlying population
we believe that legitimation checks should occur at from which the large random sample was selected, it
each stage of the mixed research process. Thus, the may not be justified for this inference to include
Quantitative Legitimation Model and Quantitative inferences from the qualitative component, especially if
Legitimation Model, shown earlier, can be used for the associated subsample is very small or if it forms a
assessing legitimation of the quantitative and separate group of people. That is, because of the
qualitative components of the study, respectively. unrepresentative sample from the qualitative phase, the
While, clearly, making inferences is a vital part of the ensuing meta-inference might be poor (statistically
research process, giving inference quality primary speaking), which, in turn, would affect statistical
emphasis could give the false impression that one does generalizability (i.e., population transferability). To the
not have to scrutinize as carefully some of the other degree to which the qualitative participants are similar
steps of the research process. Also, it is not clear yet a quantitative random sample, the problem will be
what role the validity types presented in this paper (and reduced.
in the selected references) will play in the evaluation Both the inference quality and generalizability are
process. Moreover, legitimation in mixed research even poorer if the quantitative sample is nonrandom, as

Spring 2006 56 RESEARCH IN THE SCHOOLS

 ANTHONY J. ONWUEGBUZIE AND R. BURKE JOHNSON

is the case in the vast majority of empirical research the combining of a strong inference (dominant phase)
studies (Onwuegbuzie & Leech, 2004b), and/or small. with a weak inference (less dominant design). If the
Even if the qualitative sample represented a random inferences stemming from the quantitative and
subset of the quantitative sample, as might be the case qualitative phases were consistent, then the meta-
in a sequential mixed design, the meta-inference inference quality likely would be higher. However, a
quality might still be poor. As such, when the mixed methods researcher should not assume that this
researcher’s goal is to make a statistical generalization will always be the case. Regardless, criteria are needed
we would urge caution in considering Teddlie and to be developed to identify the range of conditions
Tashakkori’s (2003) contention that “in evaluating the under which combining inferences from the qualitative
quality of inferences in mixed research, the issue of and quantitative components of a study leads to meta-
dominance or priority of one methodological approach inference quality. Indeed, as noted by Collins,
(e.g., QUAL-quan, qual-QUAN) over another is not Onwuegbuzie, and Jiao (in press) and Onwuegbuzie
very important” (p. 41). Specifically, the use of a and Collins (in press), the relationship between the
dominant-less dominant design is more likely to lead to quantitative and qualitative sampling designs (i.e.,

Table 1
Typology of Mixed Methods Legitimation Types

Legitimation Type Description

Sample Integration The extent to which the relationship between the

quantitative and qualitative sampling designs yields
quality meta-inferences.

Inside-Outside The extent to which the researcher accurately presents

and appropriately utilizes the insider’s view and the
observer’s views for purposes such as description and
explanation.

Weakness Minimization The extent to which the weakness from one approach
is compensated by the strengths from the other
approach.

Sequential The extent to which one has minimized the potential

problem wherein the meta-inferences could be affected
by reversing the sequence of the quantitative and
qualitative phases.

Conversion The extent to which the quantitizing or qualitizing

yields quality meta-inferences.

Paradigmatic mixing The extent to which the researcher’s epistemological,

ontological, axiological, methodological, and rhetorical
beliefs that underlie the quantitative and qualitative
approaches are successfully (a) combined or (b)
blended into a usable package.

Commensurability The extent to which the meta-inferences made reflect a

mixed worldview based on the cognitive process of
Gestalt switching and integration.

Multiple Validities The extent to which addressing legitimation of the

quantitative and qualitative components of the study
result from the use of quantitative, qualitative, and
mixed validity types, yielding high quality meta-
inferences.

Political The extent to which the consumers of mixed methods

research value the meta-inferences stemming from
both the quantitative and qualitative components of a
study.

Spring 2006 57 RESEARCH IN THE SCHOOLS

 THE VALIDITY ISSUE IN MIXED RESEARCH

sampling scheme, sample size) is crucial to assessing integrated (e.g., by maintaining a well informed and
meta-inference quality. Additional considerations balanced perspective when collecting, analyzing, and
regarding sample quality also should be considered interpreting what the whole set of qualitative and
when examining this type of legitimation. For example, quantitative data mean). One might be able to make the
in a qualitative sample, sometimes saturation is a useful case that quantitative research often seeks the objective
criterion with regard to the conclusions (Guest, Bunce, outsider view, that qualitative research often seeks the
& Johnson, 2006; Onwuegbuzie & Leech, 2005, in insider’s view, and that mixed research seeks to
press-b; Strauss & Corbin, 1998; Teddlie & Yu, 2006). balance fully these two viewpoints.
Sometimes theoretical generalizations can be made Weakness minimization legitimation. Mixed
even in the absence of statistical sampling methods; for research is in the optimal position for maximizing this
example, Yin (1994) has demonstrated this with some form of legitimation simply because the researcher is
of the classic sociological community studies able systematically to design a study that combines two
conducted in the twentieth century. In sum, it is or more methods. The key, however, is that the
essential that the way individuals and groups are researcher must consciously and carefully assess the
selected be considered, and that additional extent to which the weakness from one approach can
consideration be made on how to combine legitimately be compensated by the strengths from the other
different sets of people for use in making quality meta- approach and then plan and design the study to fulfill
inferences. this potential; the researcher also must use this
Inside-outside legitimation. As noted by Currall knowledge when combining, weighting, and
and Towler (2003), “etic refers to the trained interpreting the results. We refer to this process as
observer’s analysis of ‘raw’ data, whereas emic refers weakness minimization legitimation. The greater the
to how those data are interpreted by an ‘insider’ to the extent that the weakness from one approach is
system or organization (Pike, 1967)” (p. 522). In other compensated by the strengths from the other approach,
words, the emic viewpoint is the viewpoint of the group the more likely that combining a weak inference with a
member, the insider. The etic viewpoint is that of the strong inference will lead to a superior or high quality
“objective” outsider looking at and studying the group. meta-inference.
One can even speak of emic terms (language used by Sequential legitimation. When a sequential mixed
the group members) and etic terms (the language used research design is used, it is possible that the meta-
by the outsider researcher) (Johnson & Christensen, inference that arises is solely or largely the effect of the
2004). Along the same lines as sample integration sequencing itself. For example, if the results and
legitimation, when making meta-inferences by interpretations would have been different if the order
combining inferences from the qualitative and the quantitative and qualitative phases originally
quantitative phases of a study, there are times when presented had been reversed, then this would indicate
researchers should assess insider-outsider legitimation. that the sequencing itself was a threat to legitimation.
This refers to the degree to which the researcher One method of assessing this is by changing the
accurately presents and utilizes the insider’s view and sequential design to a multiple wave design, in which
the observer’s view. The ability to do this can be the quantitative and qualitative data collection and data
compromised when a researcher is ethnocentric or, on analysis phases oscillate multiple times (Sandelowski,
the other hand, when a researcher becomes so involved 2003).
with the group that he or she “goes native.” Conversion legitimation. All inferences or meta-
A strategy for obtaining a justified etic viewpoint inferences that are made after qualitizing and/or
is for the researcher to use peer review; that is, the quantitizing the data must be scrutinized. The extent to
research can have another (disinterested and trained in which these data conversion techniques lead to
social research) outsider/researcher examine the interpretable data and high inference quality is called
interpretations being made, the conceptualizations, and conversion legitimation. For example, a popularized
the relationship between the data and the conclusions. way of quantitizing data is by counting. Obtaining
An important strategy for obtaining a justified insider counts of the themes present in qualitative data can
viewpoint is member checking or participant review prevent researchers from over-weighting or under-
(i.e., have group members or participants assess the weighting emergent themes (Sandelowski, 2001). Also,
researcher’s interpretations). A strategy for obtaining a qualitative researchers can sometimes obtain more
justified meta-inference typically will be for everyone meaning by obtaining counts of observations in
on the research team as well as some researchers addition to their narrative descriptions (Johnson &
outside of the team and participants inside the group Christensen, 2004; Onwuegbuzie & Leech, 2004a;
under study to review the data and integration. In other Onwuegbuzie & Teddlie, 2003; Sandelowski, 2001)
words, the researcher should seek insider-outsider because counting can provide additional useful
legitimation for the qualitative part of a study, for the information about how often or how many or how
quantitative part, and when the parts are put together or much. However, counting is not appropriate for some

Spring 2006 58 RESEARCH IN THE SCHOOLS

 ANTHONY J. ONWUEGBUZIE AND R. BURKE JOHNSON

types of qualitative data and contexts. As noted by Commensurability legitimation. This type of
Sandelowski (2001), researchers should avoid the legitimation is based on a rejection of Kuhn’s and
problems associated with verbal counting, misleading Quine’s (and others’) concept of incommensurability
counting, over-counting, and acontextual counting. of findings, theories, language, and worldviews. In
Such problems would affect the meta-inference quality. order to meet this type of legitimation, the mixed
Similarly, a common method of qualitizing data is via researcher must learn to make Gestalt switches from
narrative profile formation (i.e., modal profiles, qualitative lens to a quantitative lens, going back and
average profiles, holistic profiles, comparative profiles, forth, again and again. We believe this is possible
normative profiles). Such profiles involve constructing through cognitive and empathy training. (If one
narrative descriptions from quantitative data. However, believes this is not possible, then one can ignore
these descriptions can represent an over-generalization “commensurability legitimation.”) Through an iterative
of the observed numeric data. Further, it is possible that process, a third viewpoint is created, a viewpoint that is
a profile that emerges from qualitizing (e.g., via informed by, is separate from, and goes beyond what is
average profiles) yields a representation of people that provided by either a pure qualitative viewpoint or a
is unrealistic. pure quantitative viewpoint. To the extent that the
Paradigmatic mixing legitimation. Combining researcher is able to negotiate cognitively this
quantitative and qualitative approaches is sometimes important Gestalt switch, the meta-inferences will
considered to be tenuous because of competing provide a more fully mixed worldview; it will go
dualisms: epistemological (e.g., objectivist vs. beyond the provision of both traditional viewpoints by
subjectivist), ontological (e.g., single reality vs. offering a third, well-informed viewpoint based on
multiple reality), axiological (e.g., value free vs. value- consideration of both qualitative and quantitative
bound), methodological (e.g., deductive logic vs. thinking. This argument takes seriously what has been
inductive logic), and rhetorical (e.g., formal vs. called the compatibility thesis (Howe, 1988; Reichardt
informal writing style) beliefs. One solution is to use & Rallis, 1994).
both viewpoints in a study (e.g., have a pure qualitative Multiple validities legitimation. This legitimation
part and a pure quantitative part each based on the pure type, which is pertinent in virtually every mixed
assumptions), and then attempt to make meaning from research study, refers to the extent to which all relevant
consideration of the two pure components of the study. research strategies are utilized and the research can be
Another solution is to think in terms of continua rather considered high on the multiple relevant “validities.”
than dualisms and then take more moderate positions For example, when addressing legitimation of the
on each continuum: ontological (recognition of quantitative component, the relevant quantitative
multiple affordances, levels of analysis, and validities are addressed and achieved; when addressing
disciplinary perspectives about what is studied; legitimation of the qualitative component, the relevant
recognizing subjective, intersubjective, and objective qualitative “validities” are addressed and achieved; and
types of reality; recognizing internal reality, external during integration and to allow strong meta-inferences,
reality, and most importantly the interaction between the relevant mixed legitimation types are addressed and
the two), epistemological (intersubjective approach to achieved. Relatedly, one should ask to what extent is
knowledge generation), axiological (distinguishing the whole (i.e., meta-inference quality) greater than the
between internal and external values, admitting and sum of its parts (i.e., inferences arising from each
describing the value ladeness of the research; stating component)?
one’s use of values in setting standards, determining Political legitimation. Onwuegbuzie (in press) has
what outcomes are to be valued, interpreting the data, identified four challenges that researchers face when
making recommendations, and making explicit how undertaking mixed methods research. One of these
one judged one’s own study), and rhetorical (e.g., use challenges is the challenge of politics. This challenge
of formal and informal writing styles using both refers to power and value tensions that come to the fore
impersonal and personal voices). When making meta- as a result of combining quantitative and qualitative
inferences, there are times when a researcher should approaches. These tensions include any value or
evaluate the extent to which her or his epistemological, ideologically based conflicts that occur when different
ontological, axiological, methodological, and rhetorical researchers are used for the quantitative and qualitative
beliefs that underlie the quantitative and qualitative phases of a study, as well as differences in perspectives
approaches are treated as separate but complementary about contradictions and paradoxes that arise when the
or are used in less extreme forms and treated as being quantitative and qualitative findings are compared and
compatible. Legitimation comes from the researcher contrasted. The challenge of politics also includes the
making the use of paradigm assumptions explicit and difficulty in persuading the consumers of mixed
conducting research that fits with the stated methods research, including stakeholders and
assumptions. policymakers, to value the meta-inferences stemming
from both the quantitative and qualitative components

Spring 2006 59 RESEARCH IN THE SCHOOLS

 THE VALIDITY ISSUE IN MIXED RESEARCH

of a study. In traditional quantitative research, decision quantitative and qualitative scholars alike who are
making and power over the research process is fully in committed to addressing the multiple problems that can
the hands of the centralized researcher in a top down occur in mixed research. This is the only way that the
manner. In postmodern qualitative research, much promise of mixed research can be realized in research
power is placed in the research participants themselves, practice.
and the researcher takes the role of collaborator and
facilitator. In mixed research, the researcher or research
team sometimes will take multiple roles; consequently, We are grateful to Dr. Abbas Tashakkori for his
mixed researchers will need to deal with issues constructive feedback on earlier versions of this
surrounding multiple or distributed power in the manuscript.
planning, conduct, and the use of research (Fetterman,
2000). A strategy for achieving this form of
legitimation is to advocate pluralism of perspectives References
and to strive to generate practical theory or results that
consumers naturally will value because the results American Educational Research Association, American
answer important questions and help provide workable Psychological Association, & National
solutions. Council on Measurement in Education (1999).
Standards for educational and psychological
Summary and Conclusions testing (rev. ed.). Washington: American
Educational Research Association.
The purpose of this paper has been to contribute to Bracht, G. H., & Glass, G. V. (1968). The external
the present dialogue about validity issues in mixed validity of experiments. American
research. We first overviewed the ways validity is Educational Research Journal, 5, 437-474.
viewed and defined in quantitative and qualitative Brewer, J., & Hunter, A. (1989). Multimethod
research, and we pointed out that there has been a research: A synthesis of styles. Newbury Park,
problem of legitimation in both of these paradigms. CA: Sage.
Second, we contended that there is also a problem of Campbell, D. T. (1957). Factors relevant to the validity
representation and problem of legitimation in mixed of experiments in social settings.
research. We argued that because mixed research Psychological Bulletin, 54, 297-312.
involves combining complementary strengths and Campbell, D. T., & Stanley, J. C. (1963). Experimental
nonoverlapping weaknesses of quantitative and and quasi-experimental designs for research.
qualitative research methods, assessing the validity of Chicago: Rand McNally.
findings can be particularly complex—yielding a Chen, H. T. (1990). Theory-driven evaluations.
problem of integration. We recommended that validity Newbury Park, CA: Sage.
in mixed research be termed legitimation in order to Chen, H. T. (2006). A theory-driven evaluation
use a bilingual nomenclature that can be used by both perspective on mixed methods research.
quantitative and qualitative researchers. We briefly Research in the Schools, 13(1), 75-83.
summarized Teddlie and Tashakkori’s (2003) and Collins, K. M. T., Onwuegbuzie, A. J., & Jiao, Q. G.
Tashakkori and Teddlie’s (2006) interesting, emerging (in press). Prevalence of mixed methods
evaluation criteria frameworks involving the concept of sampling designs in social science research.
inference quality. We identified nine new types of Evaluation and Research in Education.
legitimation that come to the fore as a result of Collins, K. M. T., Onwuegbuzie, A. J., & Sutton, I. L.
combining inferences from the quantitative and (2006). A model incorporating the rationale
qualitative components of a mixed research study to and purpose for conducting mixed methods
form meta-inferences. These nine types of legitimation research in special education and beyond.
were sample integration legitimation, insider-outsider Learning Disabilities: A Contemporary
legitimation, weakness minimization legitimation, Journal, 4, 67-100.
sequential legitimation, conversion legitimation, Cook, T. D., & Campbell, D. T. (1979). Quasi-
paradigmatic mixing legitimation, commensurability experimentation: Design and analysis issues
legitimation, multiple validities legitimation, and for field settings. Chicago: Rand McNally.
political legitimation. These types of legitimation need Creswell, J. W. (1998). Qualitative inquiry and
to be studied more closely in order to determine when research design: Choosing among five
and how they operate and how they can be maximized traditions. Thousand Oaks, CA: Sage.
or made to occur. Mixed methods researchers should Creswell, J. W., Shope, R., Plano Clark, V. L., &
keep in mind that legitimation represents a process that Green, D. O. (2006). How interpretive
is analytical, social, aesthetic, emic, etic, political, and qualitative research extends mixed methods
ethical, and which must involve the community of research. Research in the Schools, 13(1), 1-11.

Spring 2006 60 RESEARCH IN THE SCHOOLS

 ANTHONY J. ONWUEGBUZIE AND R. BURKE JOHNSON

Currall, S. C., & Towler, A. J. (2003). Research Invited James E. McLean Outstanding Paper
methods in management and organizational presented at the annual meeting of the
research: Toward integration of qualitative American Educational Research Association,
and quantitative techniques. In A. Tashakkori Montreal, Canada.
& C. Teddlie (Eds.), Handbook of mixed Lewin, K. (1952). Field theory in social science:
methods in social and behavioral research Selected theoretical papers by Kurt Lewin.
(pp. 513-526). Thousand Oaks, CA: Sage. London: Tavistock.
Denzin, N. K., & Lincoln, Y. S. (2005). The discipline Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic
and practice of qualitative research. In N. K. inquiry. Beverly Hills, CA: Sage.
Denzin & Y. S. Lincoln (Eds.), Handbook of Lincoln, Y. S., & Guba, E. G. (1990). Judging the
qualitative research (3rd ed; pp. 1-32). quality of case study reports. International
Thousand Oaks, CA: Sage. Journal of Qualitative Studies in Education, 3,
Fetterman, D. M. (2000). Empowerment evaluation. 53-59.
Newbury Park, CA: Sage. Longino, H. (1990). Science as social knowledge:
Glaser, B. G., & Strauss, A. L. (1967). The discovery of Values and objectivity in scientific inquiry.
grounded theory: Strategies for qualitative Princeton, NJ: Princeton University Press.
research. Chicago: Aldine. Longino, H. (1995). Gender, politics, and the
Grenne, J. C. (2006). Toward a methodology of mixed theoretical virtues. Synthese, 104, 383-397.
methods social inquiry. Research in the Maxwell, J. A. (1992). Understanding and validity in
Schools, 13(1), 93-98. qualitative research. Harvard Educational
Greene, J. C., Caracelli, V. J., & Graham, W. F. Review, 62, 279-299.
(1989). Toward a conceptual framework for Maxwell, J. A. (1996). Qualitative research design.
mixed-method evaluation designs. Newbury Park, CA: Sage.
Educational Evaluation and Policy Analysis, Maxwell, J. A. (2005). Qualitative research design: An
11, 255-274. interactive approach (2nd. ed.). Newbury
Guest, G., Bunce, A., & Johnson, L. (2006). How Park, CA: Sage.
many interviews are enough? An experiment Messick, S. (1989). Validity. In R. L. Linn (Ed.),
with data saturation and variability. Field Educational measurement (3rd ed., pp. 13-
Methods, 18(1), 59-82. 103). Old Tappan, N.J.: Macmillan.
Howe, K. R. (1988). Against the quantitative- Messick, S. (1995). Validity of psychological
qualitative incompatibility thesis or dogmas assessment: Validation of inferences from
die hard. Educational Researcher, 17(8), 10- persons' responses and performances as
16. scientific inquiry into score meaning.
Johnson, B., & Christensen, L. (2004). Educational American Psychologist, 50, 741-749.
research: Quantitative, qualitative, and mixed Miles, M. B., & Huberman, A. M. (1984). Qualitative
approaches (2nd ed.). Needham Heights, MA: data analysis: A sourcebook of new methods.
Allyn & Bacon. Beverly Hills, CA: Sage.
Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed Miles, M., & Huberman, A. M. (1994). Qualitative
methods research: A research paradigm whose data analysis: An expanded sourcebook (2nd
time has come. Educational Researcher, ed.). Thousand Oaks, CA: Sage.
33(7), 14-26. Onwuegbuzie, A. J. (2003). Expanding the framework
Johnson, B., & Turner, L. A. (2003). Data collection of internal and external validity in quantitative
strategies in mixed methods research. In A. research. Research in the Schools, 10(1), 71-
Tashakkori, and C. Teddlie (Eds.), Handbook 90.
of mixed methods in social and behavioral Onwuegbuzie, A. J. (in press). Mixed methods research
research (pp. 297-319). Thousand Oaks, CA: in sociology and beyond. In G. Ritzer (Ed.),
Sage. Encyclopedia of sociology. Cambridge, MA:
Kvale, S. (1995). The social construction of validity. Blackwell Publishers Ltd.
Qualitative Inquiry, 1, 19-40. Onwuegbuzie, A. J., & Collins, K. M. T. (in press). A
Lather, P. (1986). Issues of validity in openly typology of mixed methods sampling designs
ideological research: Between a rock and a in social science research. The Qualitative
soft place. Interchange, 17, 63-84. Report.
Lather, P. (1993). Fertile obsession: Validity after Onwuegbuzie, A. J., & Daniel, L. G., & Collins, K. M.
poststructuralism. Sociological Quarterly, 34, T. (in press). A meta-validation model for
673-693. assessing the score-validity of student
Leech, N. L., & Onwuegbuzie, A. J. (2005, April). A teaching evaluations. Quality & Quantity:
typology of mixed methods research designs. International Journal of Methodology.

Spring 2006 61 RESEARCH IN THE SCHOOLS

 THE VALIDITY ISSUE IN MIXED RESEARCH

Onwuegbuzie, A. J., & Johnson, R. B. (2004). Mixed Sandelowski, M., Voils, C. I., & Barroso, J. (2006).
method and mixed model research. In B. Defining and designing mixed research
Johnson & L. Christensen, Educational synthesis studies. Research in the Schools,
research: Quantitative, qualitative, and mixed 13(1), 29-40.
approaches (pp. 408-431). Boston, MA: Allyn Sanders, J. T. (1997). An ontology of affordances.
and Bacon. Ecological Psychology, 9(1), 97-112.
Onwuegbuzie, A. J., & Leech, N. L. (2004a). Schwandt, T. A. (2001). Dictionary of qualitative
Enhancing the interpretation of “significant” inquiry (2nd ed.). Thousand Oaks, CA: Sage.
findings: The role of mixed methods research. Shadish, W. R., Cook, T. D., & Campbell, D. T.
The Qualitative Report, 9(4), 770-792. (2001). Experimental and quasi-experimental
Retrieved April 19, 2005, from designs for generalized causal inference.
http://www.nova.edu/ssss/QR/QR9-4/ Boston: Houghton Mifflin.
Onwuegbuzie.pdf Smith, M. L., & Glass, G. V. (1987). Research and
Onwuegbuzie, A. J., & Leech, N. L. (2004b). Post-hoc evaluation in education and the social
power: A concept whose time has come. sciences. Englewood Cliffs, NJ: Prentice Hall.
Understanding Statistics, 3, 151-180. Strauss, A., & Corbin, J. (1998). Basics of qualitative
Onwuegbuzie, A. J., & Leech, N. L. (2005). The role research: Techniques and procedures for
of sampling in qualitative research. Academic developing grounded theory. Thousand Oaks,
Exchange Quarterly, 9, 280-284. CA: Sage.
Onwuegbuzie, A. J., & Leech, N. L. (in press-a). Tashakkori, A., & Teddlie, C. (1998). Mixed
Validity and qualitative research: An methodology: Combining qualitative and
oxymoron? Quality & Quantity: International quantitative approaches. Applied Social
Journal of Methodology. Research Methods Series (Vol. 46). Thousand
Onwuegbuzie, A. J., & Leech, N. L. (in press-b). A call Oaks, CA: Sage.
for qualitative power analyses: Considerations Tashakkori, A., & Teddlie, C. (2003). The past and
in qualitative research. Quality & Quantity: future of mixed methods research: From data
International Journal of Methodology. triangulation to mixed model designs. In A.
Onwuegbuzie, A. J., & Teddlie, C. (2003). A Tashakkori & C. Teddlie (Eds.), Handbook of
framework for analyzing data in mixed mixed methods in social and behavioral
methods research. In A. Tashakkori & C. research (pp. 671-701). Thousand Oaks, CA:
Teddlie (Eds.), Handbook of mixed methods in Sage.
social and behavioral research (pp. 351-383). Tashakkori, A., & Teddlie, C. (2006, April). Validity
Thousand Oaks, CA: Sage. issues in mixed methods research: Calling for
Patton, M. Q. (2002). Qualitative research and an integrative framework. Paper presented at
evaluation methods. Thousand Oaks, CA: the annual meeting of the American
Sage. Educational Research Association, San
Pike, K. L. (1967). Language in relation to a unified Francisco, CA.
theory of the structure of human behavior. Teddlie, C., & Tashakkori, A. (2003). Major issues and
The Hague, Netherlands: Mouton. controversies in the use of mixed methods in
Reichardt, S. S., & Rallis, S .F. (1994) Qualitative and the social and behavioral sciences. In A.
quantitative inquiries are not incompatible: a Tashakkori & C. Teddlie (Eds.), Handbook of
call for a new partnership. In C. S. Reichardt mixed methods in social and behavioral
& S. F. Rallis (Eds.), The qualitative- research (pp. 3-50). Thousand Oaks, CA:
quantitative debate: New perspectives (pp. 85- Sage.
91). San Francisco, CA: Jossey-Bass. Teddlie, C., & Tashakkori, A. (2006). A general
Sandelowski, M. (2001). Real qualitative researchers typology of research designs featuring mixed
don’t count: The use of numbers in qualitative methods. Research in the Schools, 13(1), 12-
research. Research in Nursing & Health, 24, 28.
230-240. Teddlie, C., & Yu, F. (2006, April). Mixed methods
Sandelowski, M. (2003). Tables or Tableaux? The sampling procedures: Some prototypes with
challenges of writing and reading mixed examples. Paper presented at the annual
methods studies. In A. Tashakkori & C. meeting of the American Educational
Teddlie (Eds.), Handbook of mixed methods in Research Association, San Francisco, CA.
social and behavioral research (pp. 321-350).
Thousand Oaks, CA: Sage.

Spring 2006 62 RESEARCH IN THE SCHOOLS

 ANTHONY J. ONWUEGBUZIE AND R. BURKE JOHNSON

Wolcott, H. F. (1990). On seeking--and rejecting--

validity in qualitative research. In E. W.
Eisner & A. Peshkin (Eds.), Qualitative
inquiry in education: The continuing debate
(pp. 121-152). New York: Columbia
University, Teachers College Press.
Yin, R. K. (1994). Case study research: Design and
methods. Thousand Oaks, CA: Sage.
Yin, R. K. (2006). Mixed methods research: Are the
methods genuinely integrated or merely
parallel? Research in the Schools, 13(1), 41-
47.

Notes
1
Onwuegbuzie and Leech (2006) note that, unlike the
case for quantitative research, in qualitative research,
the research design/data collection, data analysis, and
data interpretation stages are iterative. That is, in
qualitative studies, the research design/data collection,
data analysis, and data interpretation stages are
recursive, and, thus, non-linear in nature.
2
Denzin and Lincoln (2000) refer to this as the crisis of
representation.
3
Denzin and Lincoln (2000) refer to this as the crisis of
legitimation.
4
According to Denzin and Lincoln (2000), there is also
a crisis of praxis in qualitative research. This crisis
asks, “Is it possible to effect change in the world if
society is only and always a text?” (p. 17).

Spring 2006 63 RESEARCH IN THE SCHOOLS

View publication stats