Mapping Mixed-Methods Research: Theories, Models, and Measures

5
Mapping Mixed-Methods
Research
Theories, Models, and Measures 8
CHAPTER OVERVIEW AND OBJECTIVES
As we saw in Chapters 3 and 4, quantitative and qualitative researchers pursue differ-

ent approaches to gathering and analyzing data. For many years, these differences
have underscored broader political disagreements (Jick, 1979). For a new generation
of researchers, the either/or approaches of the past are incomplete and outdated.
Instead, the complexity of today’s research problems requires more comprehensive
and nuanced efforts (Wheeldon, 2010b). Indeed, past divisions among researchers
often failed to consider that, in many ways, qualitative and quantitative data are
inherently related. All quantitative data are based on qualitative judgments; all quali-
tative data can be described numerically. As presented in Chapter 1, all research is a
series of decisions (Palys, 1992). Mixed-methods research provides more choices,
options, and approaches to consider. For this reason, it has emerged as the “third
methodological movement” (Creswell & Plano Clark, 2007, p.13). As an important
new research community, it involves research in which both qualitative and quantita-
tive approaches to data gathering, analysis, interpretation, and presentation are used
(Teddlie & Tashakkori, 2009, p. 7).
Both concept maps and mind maps can be used as part of mixed-methods
research. This chapter will provide examples of how concept maps can be used as
113
311
114 VISUALIZING SOCIAL SCIENCE RESEARCH
part of pre/post mixed-methods designs and will offer a new mixed-methods mea-
sure based on the use of mind maps. To understand these examples, it is important to
understand the theoretical basis for this sort of integration and to know how different
data-collection procedures can be used together. Finally, through the use of a research
example, readers will be encouraged to consider how the use of mixed methods offers
another means to address activities presented in Chapters 2, 3, and 4. By the end of
this chapter, readers should be able to do the following:
· describe the potential of mixed-methods research and one theoretical basis

often associated with it;
· explain the different ways data, methods, and approaches can be mixed;
· provide examples of research designs to which different maps are best
suited; and
· define the salience score and explain its potential.
THEORETICAL JUSTIFICATION
As we have seen in previous chapters, the existing theoretical bases for quantitative
and qualitative research are rooted in postpositivism and constructivism. To understand
how mixed-methods research provides a different sort of theoretical understand-
ing of research, it may be useful to recall that earlier discussion. Postpositivists see
human knowledge as speculative and, therefore, not based on unchallengeable, rock-
solid foundations. They argue that the external world exists independently of an
individual’s experience of it, and thus knowledge is not hypothetical and foundation-
less. They acknowledge that all research will be incomplete in one way or another,
and they hold that approaches that can be tested and explored through the scientific
method should be favored. This often results in the application of deductive approaches
that rely on a series of steps to reach specific conclusions based on general premises.
In general, quantitative research seeks generalizability through controlled,
value-free (or value-neutral) processes that can test and validate theories through a
process of falsification. The emphasis on falsification often leads quantitative
researchers to focus on sample size and statistics to showcase broad generalizability.
At its most shortsighted, some quantitative research considers the role of setting
and context either irrelevant or unmanageable. A central critique is that some quan-
titative research models are statistics dependent, inflate the importance of mathe-
matical averages, and cannot capture the complexity associated with human
behavior (Goertzel & Fashing, 1981). By focusing solely on numeric information,
some approaches miss the depth and detail that are assigned to phenomena by
participants themselves.
Chapter 5 Mapping Mixed-Methods Research 115
Another view is one promoted by constructivists. Skeptical of the idea of one

universalistic notion of truth, they view meaningful understanding as contingent on
human practices and thus different people’s ability to socially construct reality in dif-
ferent ways. Although many qualitative researchers acknowledge the limitations
inherent in reporting individual understandings of complex ideas and concepts, in
their view research must do a better job in telling the stories of individuals. This often
results in inductive approaches to research that rely on a series of steps to reach gen-
eral conclusions based on specific premises. Qualitative research seeks to understand
or make sense of the world based on how individuals experience and perceive it.
Framed through social interaction and personal histories and narrative experiences
(Creswell & Plano Clark, 2007), knowledge is inherently localized, and the notion of
generalizability overly mythologized.
Unlike quantitative researchers, qualitative researchers focus on the develop-
ment of theories based on an interpretive or individualized process. Because there are
many possible interpretations of the same data, however, qualitative researchers
refuse to assign value to one interpretation of meaning without acknowledging the
role they themselves play within this construction (Guba & Lincoln, 1989). This
requires that researchers study the experiences, influences, and activities of research
participants while explicitly and reflexively acknowledging their own personal biases.
Yet the acceptance within qualitative research of the inherent bias of any researcher
challenges the tradition of objectivity and threatens the potential for nonpartisan
research. In addition, while privileging localized understanding through the inclusion
of depth and detail, qualitative research sometimes proudly presents findings that
would benefit from more rigorous analysis.
An emergent tradition based on a more pragmatic approach rejects either/or
approaches to understanding reality and developing knowledge. Through multiple
stages and methods of data collection and/or analysis, researchers can get a better
understanding of a phenomenon by combining the reliability of empirical counts with
the validity of lived experience. As discussed in Chapter 1, mixed-methods research is
understood as an abductive process that values the expertise, experience, and intuition
of researchers themselves. To understand the value of pragmatism and its connection
to abductive reasoning, it may be useful to recount our discussion of key issues in
social science research and reexamine a table presented in Chapter 1. Table 5.1 pro-
vides an important reminder about some of the key issues in social science research.
As we saw in Chapter 3, deductive reasoning is associated with quantitative
research and uses a top-down process that tests general premises though a series of
steps to reach specific conclusions. Researchers seek to be objective through the
research process and strive for generalizable findings by testing hypotheses through
a deliberate series of steps. In contrast, inductive reasoning is associated with qualitative
research and develops general conclusions based on the exploration of how individu-
als experience and perceive the world around them. Presented in Chapter 1, Figure 5.1
provides some differences between deductive and inductive reasoning.
Table 5.1 Key Issues in Social Science Research
Quantitative Approach Qualitative Approach Pragmatic Approach
Connection of Theory Deductive Inductive Abductive

and Data
Relationship to Objectivity Subjectivity Intersubjectivity

Research Process
Inference From Data Generality Context Transferability
Source: Morgan (2007, p. 71).
Figure 5.1 Comparing Deductive and Inductive Reasoning
Stated facts or general

begins Deductive
principles assumed
with Reasoning
to be TRUE
tested by
Developing More general

can involve conclusions
hypotheses
used to build
that are accepted,
or refine
rejected, or modified
Identification
Social Science
Based on of themes
Reasoning
tests/experiments
that leads to
that lead to
In-depth data
More specifc can involve collection
conclusions
explored through
Observations of
Inductive begins
specific cases assumed
Reasoning with
to be RELEVANT
Mixed-methods research represents an important departure from the either/or

assumptions of quantitative or qualitative approaches because it allows that both
methods may be valuable depending on the type of research question under investiga-
tion. A central assumption in mixed-methods research is that there are many social
science issues that can be better explored through the combination of different meth-
ods and techniques. Abductive reasoning can be understood as a process that values
both deductive and inductive approaches but relies principally on the expertise, expe-
rience, and intuition of researchers (see Figure 5.2). Associated with mixed-methods
research, through the intersubjectivity of researchers and their understanding based on
shared meaning, this approach to reasoning encourages testing intuitions theoretically
and empirically. Based on the best information at hand, tentative explanations and
hypotheses emerge through the research process and can be developed and/or tested
using methods that are either quantitative, qualitative, or a mix of both.
By relying on abductive reasoning, mixed-methods research offers an important
new way to conceive of research and can produce more robust measures of associa-
tion while allowing that multiple paths to meaning exist (Wheeldon, 2010b). In addi-
tion to escaping the trap of seeing research as an either/or choice between
quantitative or qualitative designs, mixed methods provide practical benefits as well.
Figure 5.2 One View of Abductive Reasoning
Abductive Reasoning
uses
both
based on Qualitative Quantitative based on

approaches approaches
Inductive to test/ Deductive

reasoning validate reasoning
Expertise/Intuition
of researchers
which both
Is based on best Acknowledges

information available understanding still may
at the time be incomplete
For example, students are often overcome by the nature of quantitative information
collected within some data sets and the view that, to be valid, quantitative research
requires a large number of cases to analyze. As discussed in Chapter 3, this is because
of the assumptions required by certain statistical tests often used in the analysis of
numeric information. On the other hand, whereas qualitative research can require
smaller samples and thus may be easier for students to engage in, many are uncertain
about how to identify a good group from which to gather data or are unclear about
the interview process and how to prepare. Mixed methods may require more work,
multiple analyses, and nuanced thinking; however, they also can provide flexibility for
researchers. Miles and Huberman (2002) urge all researchers to entertain mixed mod-
els. By avoiding polarization, polemics, and life at the extremes, they suggested that
both quantitative and qualitative inquiry can support and inform each other in
important ways. Narratives and variable-driven analyses need to interpenetrate
and inform each other. Realists, idealists and critical theorists can do better by
incorporating other ideas than remaining pure. (Miles & Huberman, 2002, p. 396)
Beyond these practical benefits, conceptually mixed-methods research and the

associated methodological concerns that may emerge can perhaps be addressed by
pragmatism (Morgan, 2007). John Dewey has been associated with both postpositiv-
ism and constructivism, but he is perhaps best understood as a pragmatic philoso-
pher who has influenced contemporary thinkers, including Richard Rorty. As a
philosophical movement, pragmatism holds that claims about the truth of one view
or another must be connected to the practical consequences of accepting that view.
Although Rorty rejects the idea of one truth, he does consider the value of consensus
or intersubjective agreement about various beliefs as a means to understanding pro-
visional or conditional truths. One means to obtain what he called “reflective equilib-
rium” is through research that can provide both realistic and socially useful outcomes
(Rorty, 1999). In this way, mixed-methods approaches may be valuable to new social
science research procedures because they provide “new ways to think about the
world—new questions to ask and new ways to pursue them” (Morgan, 2007, p. 73).
This kind of flexibility arises because instead of starting from theories or concep-
tual frameworks and testing them through deductive approaches or starting from
observations or facts, researchers can view both of these processes as part of the
broader research cycle (Teddlie & Tashakkori, 2009, pp. 87–89). For example, quantita-
tive approaches can be used to identify groups or individuals to interview and/or rele-
vant issues that make these people unique or interesting based on the analysis of
numeric data. In addition, qualitative techniques can lead researchers to discover exist-
ing data sets, develop survey questions, and/or weight data in different ways based on
narrative data (Wheeldon, 2010b). Maps may be especially valuable from a pragmatist’s
point of view because visualizing and imagining connections and relationships can be
creative, distinctive, and thus productive in ways other kinds of data collection may not
be. A broader understanding about how maps can be used in mixed-methods research
requires an understanding of current models, approaches, and techniques.
UNDERSTANDING, PLANNING, AND

DESCRIBING MIXED-METHODS RESEARCH
Mixed-methods research has been defined by Creswell and Plano Clark (2007, p. 5)
as a research design based on assumptions that guide the collection and analysis of
data and the mixture of qualitative and quantitative approaches. A central premise
is that the use of quantitative and qualitative approaches together can provide a
better understanding of research problems. Mixed methodologies can provide a
useful and novel way to communicate meaning and knowledge ( Johnson &
Onwuegbuzie, 2004) because they can combine the reliability of counts with the
validity of lived experience and perception. Mixed approaches to social science
research are increasingly popular. Tashakkori and Teddlie (1998) included 152 refer-
ences in their exploration of the growth of mixed methods in research areas such as
evaluation, health science and nursing, psychology, sociology, and education,
among others.
As mixed-methods research has grown during the past two decades, different
approaches to mixed-methods designs have been developed (Greene, Caracelli, &
Graham, 1989), revised (Creswell & Plano Clark, 2007), and reorganized (Teddlie &
Tashakkori, 2009). As discussed in Chapter 1, a variety of types and approaches of
mixed-methods research have been defined (Creswell & Plano Clark, 2007). One
approach is to use qualitative techniques to develop a theory that can then be tested
by establishing a conceptually connected hypothesis and quantitative means. Figure
5.3 provides an example.
Figure 5.3 Quantitatively Testing Qualitative Findings
Research Identification
Leads to Data collection that guides
question(s) of themes
used to
Analyzing Develop theory

and can lead to future
findings and generate hypothesis
and accepting,
rejecting, or modifying that influence the
by
Developed Development
Test hypothesis used to
theory of measures
Another approach is to develop a quantifiable means that can test a generated

hypothesis and then explore these findings using more qualitative techniques, as
presented in Figure 5.4.
With the use of these mixed approaches, research problems can benefit from
both qualitative and quantitative approaches to data analysis and the measurement
of meaning. There are a number of issues and considerations in both of the
approaches above, but for the sake of simplicity we describe three considerations
based on the useful overview provided by Creswell and Plano Clark (2007, pp. 79–85).
These include timing, weighting, and mixing.
The first surrounds the timing and ordering of methods within your study.
Sometimes these terms refer to when the data were collected and whether they were
collected at the same time (simultaneously) or during different periods (sequentially).
Some researchers interested in comparing how different tools capture perceptions col-
lect both qualitative and quantitative data at the same time (Gogolin & Swartz, 1992;
Jenkins, 2001). Others have collected and analyzed data sequentially and at different
times. For example, in a study on cross-national differences in classroom learning envi-
ronments in Taiwan and Australia by Aldridge, Fraser, and Huang (1999), qualitative
data were used to explain, in more detail, quantitative results. The authors used two
separate data-collection phases. The first was a quantitative instrument with multiple
subscales to assess aspects of the classroom environment. Some months later, they used
classroom observations and qualitative interviews with students and teachers to get a
more detailed picture of the differences in classroom environments in each country.
Figure 5.4 Qualitatively Validating Quantitative Findings
Research Testable Development Collected

leads to and the to apply to
question(s) hypothesis of measures data
allows
and development of new
New data collection or
Validation or reanalysis of existing data Hypothesis
skeptism of results testing
for the and can lead to based on

leads to
Accept or reject that allows
Data analysis
hypothesis one to
Development or
comparison of themes
Another example of interest is a study by Myers and Oetzel (2003) that used
qualitative data to create and validate a quantitative instrument. This study was
also organized through two phases of data collection. Based on qualitative inter-
views, the authors first gathered data through field notes and transcripts. Later
they engaged in analysis using techniques drawn from qualitative data including
coding, theme identification, and connection to existing literature. Based on this
analysis, the authors developed an instrument that could provide quantitative mea-
sures based on the qualitative interviews. They then administered this instrument,
and the quantitative data were analyzed to test correlations from the qualitative
interviews.
However, data collection and data analysis may not always be so closely inter-
twined. There may be times that data collected simultaneously are analyzed sepa-
rately, in different ways, and at various times. Other studies might collect data
through multiple data-collection phases over longer time periods. Although collect-
ing data in multiple settings may be useful, there may be research designs in which
data can be usefully compiled and analyzed together and at the same time. Thus,
there is an important difference between descriptive and analytic timing/ordering
considerations (Creswell & Plano Clark, 2007). Descriptive considerations focus on
whether data were collected at the same time or over a longer period of time.
Analytic considerations focus on whether the data were analyzed together, at the
same time, or separately, one after another. Whereas both may require some justifi-
cation, they ought not be confused. Figure 5.5 provides a visual overview of some of
these considerations.
The second question is related to how you weight different methods in your
study, or the relative importance of each approach. This is often indicated using
capital letters for the dominant approach (QUAN or QUAL) and lowercase letters
for the secondary, less dominant methodological approach (qual or quan). Of
course, you may choose to give equal weight to both traditions, in which case both
would be capitalized (QUAL/QUAN). More often one tradition is selected as domi-
nant. Whether your approach is primarily quantitative or qualitative in nature
depends to a large degree on the type of research question you are interested in.
Both approaches have strengths and weaknesses, of course, but thinking about
how and why some methods might work together better than others is important.
Some researchers have gathered data through quantitative surveys and qualitative
interviews (Baumann, 1999; Way, Stauber, Nakkula, & London, 1994). This allows
researchers to define beforehand the kind of data they seek by utilizing specific
data-collection tools. In essence this question boils down to whether you will
assign equal or unequal weight to the different sorts of data you have collected
and whether your analysis emphasizes quantitative or qualitative assumptions
about meaning. Your decision about how to weight data may also be related to the
Figure 5.5 Timing and Ordering of Data Collection/Analysis in Mixed Methods
Data Collection/Analysis
In which order will you collect/analyze data?
TIMING
Collect/analyze Collect/analyze through

at same time different stages
Concurrent Sequential Multiple
How many
instruments/operations How many stages?
Over what period?
will you use?
1 for QUAN
2 for QUAL
1 with 2 or more Months Years
different unique
sections instruments
How will you justify
your choice?
Note: QUAN = primarily quantitative; QUAL = primarily qualitative.
research question, your epistemological view, practical issues surrounding access

to data, data types, and additional issues associated with research—such as
deadlines and due dates.
To assist researchers in clearly presenting how they mixed methods within a
study, a series of useful notations has been developed. These can indicate not only
which approach was more dominant in a mixed-methods design but also whether
data collection and/or analysis was simultaneous or sequential (Morse, 2003, p. 198).
Table 5.2 provides some notation examples.
Table 5.2 Notions in Mixed-Methods Research
Symbol Meaning
QUAN Primarily a quantitative mixed-methods project
QUAL Primarily a qualitative mixed-methods project
Plus sign (+) Data collection/analysis conducted at the same time
Arrow (→) The sequence of data collection/analysis in mixed-methods

projects
quan Secondarily a quantitative mixed-methods project
qual Secondarily a qualitative mixed-methods project
EXERCISE 5.1
Think You Get It?
What kind of mixed-methods projects do the following notations indicate?

QUAN + qual ___________________________________________________
QUAL → quan __________________________________________________
quan + QUAL ___________________________________________________
QUAL → qual ___________________________________________________
These notations can help researchers present their approaches and think about
their designs. However, simply noting which design they have chosen, whether a
quantitative or qualitative approach will be dominant, or how their data will be mixed
is not enough. Central to any research, and perhaps especially to mixed-methods
research, is how researchers justify their approach. This is especially important with
regard to the question of mixing. There are at least three options available when
deciding how and why to mix your data. Data can be merged by transforming and/or
integrating two data types together, one data type can be embedded within another,
or they can be presented separately and then connected to answer different aspects
of the same or a similar research question. Creswell and Plano Clark (2007, p. 80) have
compiled a useful decision tree that provides an overview of a number of relevant
mixed-methods concerns. Building on their work, Figure 5.6 provides some examples
of how data might be mixed.
Figure 5.6 Mixing Strategies in Mixed-Methods Research
Mixing Data in Mixed-Methods Research
Combine
during Merge and Embed Connect
data integrate the data the data
analysis the data
Integrate QUAL QUAN QUAL QUAN

when within within leads to leads to
How? presenting QUAN QUAL QUAN QUAL
intepretation
of data
Focus
on QUAN
or QUAL? Must Must
Must
Always Justify Your Approach
Note: QUAL = primarily qualitative; QUAN = primarily quantitative.
But what about mixed-methods approaches that seek to integrate data analysis in
a more interactive way? Teddlie and Tashakkori (2009, pp. 280–281) presented a study
by Jang, McDougall, Pollon, Herbert, and Russell (2008) that analyzed both QUAN and
QUAL data independently and then attempted more integrative analysis by presenting
both QUAN and QUAL to participants for feedback. By transforming QUAN factors into
QUAL themes, and vice versa ( for comparison), they consolidated the themes and fac-
tors that emerged through both analyses and used QUAL data to provide nuance to the
consolidated themes/factors. This is perhaps more complex than is practical to con-
sider at this point; however, that example points to one of the major strengths of
mixed-methods data. By providing multiple options, researchers can experiment with
different analysis strategies and, provided they justify their approach, can offer valu-
able new approaches, methods, and even measures. The mind map research example
in this chapter provides perhaps a more simplistic example of how different sorts of
data can be integrated and combined in a novel and potentially useful way.
MAPS, DATA, AND INTEGRITY IN MIXED-METHODS RESEARCH
Before we turn to a couple of mixed-methods research examples, it may be useful to

reflect on our discussion in Chapter 2 about maps as data. Although mixed-methods
research has emerged as an important approach to social science research, it still
relies on data collection often associated with either quantitative or qualitative
research. As discussed in Chapter 2, quantitative data are often based on instruments
that measure individual performance and attitudes, based on clearly predefined cat-
egories. By contrast, qualitative data are generally based on themes that emerge
through open-ended interviews, observations, or the review of various documents. As
we have seen in Chapters 3 and 4, whereas both concept maps and perhaps mind
maps can be used to generate social science data, the kind of data elicited by each
approach to mapping requires some discussion.
This book presents the idea that knowledge and understanding are based on pat-
terns (Kaplan, 1964) and these patterns can be represented and analyzed in a variety
of ways. As Chapter 2 argued, and Chapters 3 and 4 explained, these patterns might
be better identified, recognized, and understood through more graphic representa-
tions of knowledge, experience, and perception (Wheeldon, 2010b). We have pre-
sented a number of examples of quantitative and qualitative research using concept
maps and mind maps; however, it may be that the mapping process is best suited to
mixed-methods researchers because as a data-collection technique, it can offer both
numeric and narrative data, provide a means to showcase analysis procedures, or
even be a means to present research findings. This flexibility is in line with mixed
methods as a pragmatic approach to research (Johnson & Onwuegbuzie, 2004), and
whereas researchers may choose to rely on traditional data-collection means and
ordering, combining, or embedding findings through existent models, other
approaches exist and should be explored.
Another issue is how to consider reliability and validity in mixed-methods
research. As you may recall, in Chapter 3 we discussed the idea that in quantitative
research, reliability is concerned with questions of stability and consistency and
whether the same measurement tool can yield stable and consistent results over time.
In contrast, validity considers how well we were able to design methods or measures
to investigate the broader constructs under investigation. In qualitative research, the
focus on these concepts is slightly different. As discussed in Chapter 4, these same
concepts mean different things within the context of the qualitative paradigm. This
requires that researchers focus on how they justify their approach, whether they con-
sider alternate explanations and approaches, and whether they address the research-
er’s reflexivity. We will return to these issues in Chapter 7. It is important to
acknowledge that depending on the mixed-methods design, each of these approaches
must be considered, either separately or together.
It is important to recognize that the quality of mixed-methods research is

based on the integrity of the process used to integrate or combine different meth-
ods within one project. For mixed-methods projects that emphasize quantitative
research, key questions surround the hypothesis under investigation, the size and
justification for the gathering of data from the samples selected, and the appropri-
ateness of the statistical tests and operations employed. For mixed-methods proj-
ects that emphasize qualitative research, key questions surround the nature of data
collection, the analytic process used to discover themes and commonalities and
differences, and how the data are presented. Although mixed methods involve both
quantitative and qualitative components that consider the elements described
above, they must do more than simply report the results of two separate projects
(Teddlie & Tashakkori, 2009). Meaningful mixed-methods research combines the
quantitative and qualitative results to offer more than the sum of each part. Quali
tative approaches might be used to contextualize numeric findings, or quantitative
methods might be used to assist readers to understand the generalizability of nar-
rative findings. New approaches to mixed methods can build on past designs that
aim to explore topics from more than one angle and use maps to collect data in a
variety of ways and for a variety of purposes. It may be useful to explore in practical
terms how concept maps and mind maps can be used through two mixed-methods
research examples.
RESEARCH EXAMPLES USING CONCEPT MAPS AND MIND MAPS
Based on research by Wheeldon (2010b), this example shows how maps can offer a
unique way for research participants to represent their experiences while assisting
researchers to make better sense of gathered data. Maps can be used both in established
pre/post designs and in the construction of unique and novel mixed-methods measures
constructed by assigning weights to different data-collection stages. Do you agree with
the notion that data can be weighted in this way? On what assumptions is it based?
Pre/Post Concept Maps and Validation

in Mixed-Methods Research
As discussed in Chapters 2 and 3, concept maps are most commonly used in quantita-
tive research. This may be because earlier versions of concept maps were used to explore
science education (Stewart, Van Kirk, & Rowell, 1979) and were often quantitatively
scored by an expert to assess how understanding was demonstrated through the struc-
ture of the map itself. A focus on structure remains an integral feature for many concept
map researchers (Novak & Cañas, 2008) because structured maps can be consistently
assessed, scored, and/or compared to assess an individual’s understanding of a topic.

Novak and Gowin (1984) described the utility of maps to assess understanding in educa-
tion. They argued that by having students complete concept maps on certain topics,
structured interview questions can be posed to a student to explore areas of misunder-
standing or confusion based on the student’s map. To score a concept map, Novak and
Gowin suggested that maps be assessed by a subject matter expert based on the number
of valid propositions, levels of hierarchy, and number of branchings, cross-links, and
specific examples provided in the maps. As presented in Chapter 2, there are a number
of ways to score a map, including based on the map’s structure.
By using concept maps as a pre/post data-collection tool, we can quantitatively
test if understanding, views, and/or perceptions change over time (Kilic, Kaya, &
Dogan, 2004). In mixed-methods designs, scoring pre/post concept maps can also be
used to test hypotheses that emerge from qualitative data analysis. Based on a pilot
study to assess different teaching strategies for internship students related to values
and ethics in criminal justice (Wheeldon, 2008), the example below provides one way
that concept maps might be used to test qualitative findings. As you read this exam-
ple, consider which qualitative findings were validated by the analysis of the pre/post
concept maps. Which questions remain?
Overview and Mixed Design
Forty-five students enrolled in the Administration of Justice internship program at

George Mason University were assigned unique identifier codes and tracked during 16
months between 2007 and 2009. This program involved the completion of a preintern-
ship course and a subsequent 4-month internship at a criminal justice agency. Of
interest was which methods of ethical instruction used in the preinternship class
students would identify as most useful. Based on a debate within the literature about
the best means to guide instruction on values and ethics (Cederblom & Spohn, 1991),
a variety of approaches were used. Through nine scenarios students were presented
with dilemmas and had to work together to identify the best course of action. An
equal number of scenarios were drawn from texts that used a more general philo-
sophic approach, a more practical criminal justice–focused approach, and a hybrid
approach that involved criminal justice examples and step-by-step deliberation.
Student perceptions were based on data collected in a variety of ways. Quantitative
data about personal ethics and their origins were collected before and after the pre-
internship class through concept maps. Some time later, qualitative data through
surveys and focus groups were collected before and after students’ criminal justice
internships.
As described above there are three central concerns related to mixed-methods
design. These include the timing, weighting, and mixing of data. In this example, the
timing aspect of the mixed-methods design might be described as multistage and

sequential. First, the quantitative data were collected through the pre/post concept
maps, and later, qualitative data were collected through surveys and focus groups.
Descriptively, this might be represented by the notation quan → QUAL. However, in
this case, the pre/post data were used to test whether the change in views suggested
by qualitative data collection through a survey and focus groups could be quantita-
tively validated. Thus, in analytic terms, it may be useful to describe the project as
QUAL → quan. The important thing to remember is that this was principally a quali-
tative project (QUAL). Quantitative data were collected first; however, they were
analyzed only later. The mixing strategy involved connecting some of the qualitative
findings to the quantitative pre/post analysis to corroborate key themes identified.
Collecting and Analyzing Qualitative Data
Data were collected during a 16-month interval from a student’s first preinternship
class to his or her final class following a criminal justice internship. The first stage of
data analysis was based on the qualitative data collected through the surveys and
focus groups. The open-ended survey and focus groups allowed students to provide
their views on the importance of ethics to their placements and the value of the dif-
ferent approaches, exercises, and scenarios used to teach ethical decision making
during the preinternship course (Wheeldon, 2008). This provided more nuance and
context to the quantified differences expressed in the maps. The survey questions of
interest are outlined in Table 5.3.
Table 5.3 Mapping Values Survey Questions
Number Question
1 How important are one’s ethics and values to a career in criminal
justice?
2 How well did ADJ 479 assist you to consider where your values
and ethics come from?
3 How useful were the exercises and discussions to assist you to
identify and address ethical dilemmas?
4 List any scenarios you recall from class that were useful in
exploring values, ethics, and criminal justice.
5 Anything you would like to add?
Note: ADJ = Administration of Justice.

Following the conclusion of their internships, these same students participated

in focus groups on values, ethics, and the criminal justice system in their last class,
Administration of Justice 480. Following these discussions, students were encouraged
to write to the researcher privately and/or anonymously to share their views about
their experiences.
The qualitative analysis strategy built on past approaches (Wheeldon & Faubert,
2009) and involved mapping the survey responses to identify common perceptions.
This included combining the presence and frequency of unique individual con
cepts into a color-coded Excel spreadsheet. Perhaps simplistic, this concept-counting
approach (Wheeldon, 2011) offered a useful way to present common sentiments
expressed by students. Another approach was to connect common sentiments to
illustrative quotations from the students. These quotations provided a means to iden-
tify thematic findings while rooting any conclusions in the language of those sur-
veyed. This approach was repeated in the focus groups held within class after
students had completed their internships. Wide-open discussion ensued, and stu-
dents offered insights into perceived strengths and weaknesses of the preinternship
course, teaching strategies, and the internship program overall. Both common con-
cepts and sentiments were again captured to provide additional and reflective data.
The qualitative findings provided key insights into student perceptions.
Based on the survey results, virtually all students identified values and ethics
as important or very important to a career in criminal justice, and most identified
the course and the exercises as important or very important to their ability to iden-
tify and address ethical dilemmas. One theme that emerged was the belief that the
course helped “students to understand their own values, and identify and address
ethical dilemmas.” When asked which scenarios were most useful, the majority
of students identified examples drawn from a text that combined specific real-work
situations with a step-by-step approach to identifying the dilemmas and possi-
ble solutions. Another important theme was that teaching ethics required that
real-life scenarios be used to “help students to evaluate how ethics are connected
to the criminal justice system.” These should not be “too easy,” because they can
provide a false sense of security and a limited understanding of the “real-world
complexity of ethics.”
The focus group results offered another view of the role of ethics. Although many
students acknowledged that the class “helped them identify ethical dilemmas in their
placements,” many more students saw ethics as “situational” and varied “depending
on the type of agency.” Some students wished that the course had “taught [them] what
the ethics in the criminal justice system were” and focused on the specific guidelines
required at the agencies where they did their internships. Other students shared more
personal accounts of their internship experience and some of the challenging or trau-
matic incidences they faced during their placements. These included seeing a dead
body, interviewing a victim of domestic violence, and accompanying a sheriff to a
home where a youth was to be taken to a juvenile facility jail. For these students the
value of ethics instruction was very personal. They suggested the experience of
thinking through the ethical dilemmas prepared them because they said they “knew
themselves a bit better” as a result.
Testing the Findings: Quantitative Pre/Post Concept Map

Analysis Strategy
To test the extent to which the preinternship class assisted students to consider and
reflect on their values, the pre/post concept maps were quantitatively assessed. As
you may recall, students were asked to complete concept maps during the first prein-
ternship class based on the general instructions to identify both important values and
ethics and their origin(s). These maps demonstrated how, beginning with themselves,
participants could provide what they believed to be core values and connect them
with lines to where they believed these values originated. They were provided an
exemplar map for how their maps should be constructed as well as basic instructions
about which sorts of concepts might be included (e.g., honest, hardworking) and
where these concepts may have originated (e.g., parents, religion, school). Each stu-
dent was asked to complete another concept map using the same instructions and
exemplars near the end of the course.
If the qualitative data are to be believed, we ought to be able to see a change
in student concept maps before and after the course. To test this idea the premaps
and postmaps were quantitatively assessed, and values and ethics identified in
the maps and their perceived origins before and after the preinternship class were
compared. In this case, the null hypothesis is that there would be no difference
between the means of the premaps and postmaps. The research hypothesis was
that the maps completed after the course would contain more concepts and would
be constructed in more complex ways. To test this hypothesis, all relevant data for
each student were compiled into an Excel table. Based on this process, a descriptive
analysis was made possible that included the values in the maps and data about
from whom, or from where, students suggested they had originated. Values in the
premaps and postmaps were first compared in a table, as presented in Figures 5.7
and 5.8 below.
As you can see, truth and loyalty remained important for these students through-
out the course, but compassion was identified more often in the postmaps, with
open-mindedness identified for the first time in the postmaps. The use of traditional
tables is common, but another approach is based on a computer program called
Wordle (Feinberg, 2010). This online program is free for all, is easy to use, and pro-
vides another means to visualize which values were important. To create Figures 5.5
and 5.6, one can simply copy the text into the Create box at www.wordle.net. The more
Figure 5.7 Most Common Premap Values
Respect
Compassion
Loyalty
Truth
0 5 10 15 20 25 30 35
Figure 5.8 Most Common Postmap Values
Open-Mindedness
Compassion
Loyalty
Truth
0 5 10 15 20 25 30 35
words you type, the more placement of the text changes, and the size of an individual
word depends on the number of times you enter the word into the Create box. The
resultant “wordle” is another way to visualize data. Figures 5.9 and 5.10 show the most
common values in the student pre- and postmaps.
Figure 5.9 Premap Values in Wordle
Figure 5.10 Postmap Values in Wordle

In addition, the student maps provided data about where these values origi-
nated. As Figure 5.11 presents, these changed pre- and postcourse.
As discussed above the value of using maps is that they can provide both narra-
tive and numeric data. Through a comparison of the pre- and postmaps, a number
of interesting narrative observations can be made. The values of honesty and loyalty
remained important for students both before and after the course; compassion as a
value of importance was identified more often postcourse, and open-mindedness
was identified for the first time postcourse. In terms of value origins, family, friends,
school, and religion all remain core sites of value origin. Postcourse, however, school
was identified more often. In addition to this descriptive information, the pre- and
postmaps also provided numeric data. The maps were scored based on the number
of concepts and the maps’ complexity, as outlined in Figures 5.12 and 5.13. In this
study, a complexity score was calculated based on one point for each unique con-
cept and five points for maps that included two or more connections between values
and origins.
To assess the significance of the changes in the pre- and postmaps, we can
return to our familiar friend: the dependent t test. As discussed in Chapter 3, this is
a very useful tool when we are comparing pre/post data from the same people. By
compiling the mean number of concepts in the premaps and the postmaps, and the
mean complexity of the pre- and postmaps, you might get something that looks like
Table 5.4.
Figure 5.11 Pre/Post Comparison of Value Origins
School/Teachers
Religion
Family
−5 5 15 25 35 45
Pre Post
Figure 5.12 Scoring Complexity in Pre- and Postmaps, Example 1
Parents
Scoring
Unique Concepts—6
Hierarchies/Levels—0
Hard work
Complexity Score—6
Honesty ME
Religion Generous Sister
Figure 5.13 Scoring Complexity in Pre- and Postmaps, Example 2
Work/Job Parents Friends
Responsibility Trust
Family Religion
Honesty ME Kindness Sister
Scoring
Religion Unique Concepts—11
Hierarchies/Levels—5
Complexity—16
Table 5.4 Pre-/Postmap Concept and Complexity Comparison
Mean Pre Mean Post Mean Pre Mean Post

Gender n Concepts Concepts Complexity Complexity
Male 18 8.05 13.87 9.72 17.94
Female 27 9.83 15.88 11.85 20.59
By using a one-tailed dependent t test, the mean difference on the number of

concepts is reported as 5.49 (with a standard error of .42) and a p value of less than
.001. The mean difference on the complexity score is reported as 8.53 (with a standard
error of .68) and a p value again less than .001. As you will recall, a p value less than
.05 is considered significant enough that we can reject the null hypothesis that there
were no differences between the pre and post means. Based on the scoring of pre- and
postmaps, maps completed postcourse contained more concepts and were con-
structed in more complex ways. The differences were statistically significant and
suggested that the course assisted students to provide a more detailed account and
understanding of their values.
Discussion and Limitations
In this example, of interest were the types of ethical instruction identified by students
based on the three approaches to this training provided during the preinternship
class. This involved a qualitative analysis of student surveys and focus groups that
suggested that approaches to ethical instruction should not be “too easy” and not shy
away from the “real-world complexity of ethics.” Some common themes were that
ethical instruction needed to provide (a) a means for students to understand their
own values and (b) opportunities to identify and address ethical dilemmas. Examples
drawn from a text that combined specific real-work situations with a step-by-step
approach to identifying the dilemmas and possible solutions were identified as useful
by students (Wheeldon, 2008). Yet not all students saw the preinternship course as
valuable, and as some suggested in the focus groups, ethics in the classroom and
ethics in the real world were two different things.
These qualitative findings led to the second, more general research question
designed to better understand the role of the preinternship class. The pre/post con-
cept maps were used to validate the hypothesis that exposure to ethical dilemmas
would influence how students represented their ethics and values and understood
their origins. Overall, the qualitative data suggested that students saw ethical deci-
sion making as very important in the justice system and that the instruction was most
useful when it provided them with an opportunity to work in groups to identify ethi-
cal dilemmas and analyze different approaches to resolving them. Although the pre/
post concept maps could not be used to corroborate all the qualitative data, they did
validate the general notion that the course was useful in assisting students to reflect
on their values and ethics and provided some additional hypotheses that could be
tested in subsequent studies. This analysis strategy is represented in Figure 5.14.
Although this pilot study has since been built on and more data have been col-
lected and analyzed from the sample, it provides a useful example to consider how
maps can be used in mixed-methods designs and how to think about the timing,
weighting, and mixing of the data. Nevertheless, a number of limitations should be
noted. These include the size of the sample, the limited geographic location of the
students, and the failure to capture other kinds of demographic data such as ethnicity,
income level, and previous criminal justice employment. Another issue refers to how
the data from the maps and data drawn from surveys were combined and compiled.
In this example the qualitative findings were tested quantitatively. Yet the quantita-
tive analysis did not consider all of the qualitative data that emerged from the surveys
Figure 5.14 Validating Qualitative Data on the Value of Ethical Instruction
focused Which methods of ethical instruction

Research Question(s)
on are most useful?
explored using
that may lead
to future
Survey responses
Qualitative analysis of
and focus groups
Value of classroom instruction
to influence student reflection
Hybrid which
philosophical/ including found
validated practical importance of
could not be approaches
tested through
Quantitative analysis of Value in
pre/post concept maps ethical instruction
tested through Value of instruction in the including a

classroom vs. the real world debate between
and focus groups. Thus, we can say the pre-/postmaps suggested the course assisted
students to provide a more detailed account and understanding of their values; how-
ever, they did not (and could not) validate the survey data that suggested which types
of ethical instruction were best. The choice to focus principally on qualitative data
collection might be seen as a limitation.
Another approach might have tried to find new ways to combine the map data
and survey results by individual students. In addition, by having students complete yet
another concept map on how best to teach ethics, these data might have suggested
how changes in values orientation were specifically connected to the style of ethical
instruction favored by each student. Another concern in this example might be the
assumption that concept count/complexity measures are useful proxies for knowl-
edge transfer. This has not yet been fully demonstrated. Although there is research on the
value of concept maps in education, science, and nursing, their application and the
validity of different approaches in criminal justice is still emerging (Wheeldon, 2010b).
Mind Maps and Constructing a Mixed-Methods Measure
Another approach to the use of maps in mixed-methods research attempts to locate

the strength of mind maps with the kind of research being undertaken. Using pre/post
concept maps as in the example above may be a useful way to measure how views
change over time, but quantitative comparisons may be less important than the ways
participants represent their individual understanding. Using less formal mind maps
to collect data may provide an important window into how participants understand
issues, events, or approaches. This technique was used in a study to assess training
approaches in the development of the first probation service in Latvia (Wheeldon,
2010a). Although this example also relied on sequential multistage data collection,
the ways in which the data were weighted and mixed is quite different from the pre/
the ways in which the data were weighted and mixed is quite different from the pre/
post concept map example presented above. Instead of comparing pre- and post-
maps, in this example the identification of themes within the maps led to another,
more complex analysis process that combined and quantified the frequency of indi-
vidual variables identified during a variety of data-collection stages.
Overview and Mixed Design
Through an innovative, exploratory mixed methodology involving a multistage data-

collection process, mind maps were used to gather evidence, capture experience, and
frame additional interviews among 14 research participants who during a 2-year period
were exposed to a variety of training methods. This project considered which train-
ing approaches were of most value to participants based on a dichotomy within the
organizational change literature between sharing specific organizational training tools

and the development of individual capacity to pursue reform through local innovation
(Wheeldon, 2010a). Building on past research, this study contributed to emerging
knowledge-transfer scholarship and considered the potential of legal technical assis-
tance projects to model democratic values in the former Soviet Union. In terms of the
timing, weighting, and mixing of data, this example provides yet another approach to
thinking about mixed-methods research. The timing once again involved sequential
data collection as the mind maps were collected first and the themes contained within
them informed the development of subsequent interviews. However, once the interview
data were collected, both the maps and the interviews were reanalyzed concurrently.
During this reanalysis concepts that emerged through more unsolicited data-collection
techniques were weighted higher than concepts identified in other stages. This allowed
for the construction of a novel mixed-methods measure, the salience score that was
used to identify the most common elements that emerged through data collection but
that explicitly privileged those captured in more unsolicited ways.
Once again, in this example, the sequence of data collection was less important
than the process by which the data were weighted and analyzed. As described below,
the salience score emerged from concurrent analyses that could be represented by the
notation QUAL + quan. On the other hand, although the sample was small, one could
argue that the quantitative measure developed through a series of numeric operations
is equally important as the qualitative assumptions from which it is drawn. If this view
is correct, the notation could also be described as QUAL + QUAN. As you read the
example, consider which notation you think is more appropriate. As we will see the
mixing strategy involved merging and integrating the data to develop a mixed measure
and then embedding the qualitative findings within the numeric salience score.
Data Collection and the Quantitative Salience Score
Like in the example above, the process of data collection and analysis here also
involved a number of steps and stages. In the first stage of data collection, partici-
pants were asked to complete mind maps about their experience of a legal technical
assistance project. Participants were provided with an exemplar map and encouraged
to make their own as reflective of their experiences during the project as possible.
One map adapted from the maps that were returned is presented in Figure 5.15.
In the second stage of data collection, participants were asked general interview
questions. Listed in Table 5.5, these general questions were open ended and probed
positive and negative experiences, perceived results and challenges, and previously
indentified concepts, gathered through a literature review.
In addition to the general questions, conclusionary and more reflective open-
ended questions followed the more directive data-collection stages. By providing
participants an opportunity to identify areas not previously addressed, the researcher
Figure 5.15 Example of Latvian Mind Map
Amended Laws Probation Draft

Law
Juvenile
Justice Programs Latvian Canadian
Experts
Prison
Reform Probation
Service
Key Latvian Supervision

Latvia-
Cities: Canadian Programs
Canada
Cesis Saldus Trainers Postprison
Project
Valmiera Release
Became
Friends
Probation Pilot
Directors Study Tour to
Shared Canada
Expertise
Table 5.5 General Interview Questions
Question
Number Question Text
1 Describe your most positive or memorable experience with Canadian trainers.
2 Describe your most negative or challenging experience with Canadian trainers.
3 What if anything did you learn through the mind map exercise?
4 How important was the role of the translator/translation within the training sessions?
5 Have you remained in touch with any of the Canadian trainers?
(Continued)
Table 5.5 (Continued)
Question
Number Question Text
6 What would you say was the biggest result of Latvian-Canadian cooperation?
7 What would you say was the biggest challenge of Latvian-Canadian cooperation?
8 Was working with Canadians different than working with other international experts?
Source: Wheeldon (2010b).
9 If you could change one thing about Canada’s involvement with Latvia, what would it be?
10 Anything else you’d like to add?
Source: Wheeldon, 2010b.
hoped they would reflect on their experience as whole, restating aspects of particular
significance, or provide additional clarifying commentary. By combining the maps
with the different stages of follow-up interviews, the frequency with which individual
variables were identified through the multiple data-collection stages was recorded.
To analyze the interview data in a more meaningful way, a mixed-methods mea-
sure called a “salience score” was developed (Wheeldon, 2010b). The construction of a
mixed-methods salience score may involve a number of separate yet rather simple
operations. In the first step, unique, individual concepts, elements, and activities iden-
tified by participants in different stages of data collection can be recorded as variables.
Individual variables might be identified in mind maps, through general or specific
interviews, or in summative and reflective statements. They also may be identified in
one, multiple, or all stages of data collection. These variables can then be quantified
through the use of a concept-counting technique that records the frequency or pres-
ence of individual variables throughout data collection. Table 5.6 lists some of the
variables identified through the study.
The number of times a variable was identified in total across the data-collection
stages and the number of times each participant identified a variable across multiple
data-collection stages were interesting, but these sorts of frequency measures can
provide only a sense of whether, and how often, these variables were identified. An
important assumption in this study was that the way in which the variables were
identified might more usefully demonstrate the relevance or legitimacy of a proposed
association (Cash et al., 2002).
For each variable identified in multiple stages of data collection, a salience score
or weighted measure was developed using a weighted count system (Stillwell,
Winterfeldt, & John, 1987). This strategy allows the researcher to assign participants
a score for each individual variable they identify depending on the stage(s) at which
these variables were recorded. For example, individual variables that emerge from
Table 5.6 Individual Variables Identified
Variables Identified Variables Identified
Presentence Report Job Shadowing

Risk/Needs Assessment Role-Plays
Prison Intake Assessment Working Groups
Reintegration Plan Canada Site Visits
Case Management Regional Coordination Councils
Canadian Program Manuals Networking
Probation Draft Personalities
Legislative Reform Pilot Projects
Police Reform Restorative Justice Exercises
user-generated, open-ended, and unsolicited data-collection procedures can be

treated as more valuable and given more weight in the overall measure. In this exam-
ple, user-generated concepts gathered through the maps were deemed worth four
points, and the responses to general, nonspecific questions were worth three.
Concepts identified following conclusionary questions asked at the end of both the
general question sets were worth two points. Given that participants came back to
these concepts after several other data-collection stages, they were felt to be less valu-
able than concepts generated without the priming of earlier data collection.
This approach to data transformation allowed a score to be tabulated for each
individual variable, the common unique variables identified in each mind map
(Turns, Atman, & Adams, 2000), and those that emerged through the qualitative
interviews (Sandelowski, 2001). These were combined for each individual by adding
the points assigned through each stage of the data-collection process. Salience scores
for identified variables can produce values ranging from 0 (not salient) to 9 (extremely
salient). Table 5.7 presents an example of how a salience score of 5 might be tabulated
for a concept identified in two out of four stages of data collection.
By repeating this process, a mixed-methods salience score was tabulated for
each variable. The individual variable salience scores (IVSSs) for each individual were
then combined to get an overall variable salience score (OVSS) for the total sample.
All participants’ IVSSs were added together, and the result was divided by the total
number of people in the sample (n). This operation is represented by the formula
OVSS = [(IVSS1 + IVSS2 . . . IVSSn) / n]. This weighted scoring scheme can incorpo-
rate both overall variable frequencies while accounting for variables identified
throughout multiple data-collection stages. When combined with the more nuanced
qualitative data gathered through interviews, this approach may provide a strength-
ened means to clarify and build on the results of one method with the perspective of
another (Greene & Caracelli, 1997). Top OVSSs are reported in Table 5.8.
Table 5.7 Example of Salience Scoring Procedure
Data-Collection Stage Frequency Weighted Measure Percentage

Mind map 1 4 50.0
General Interview 0 3 0.0
Reflective Statement 1 2 50.0
Total 2 100.0
Salience Score 6
Table 5.8 Top Overall Variable Salience Score for Sample
Individual Variable Salience Score

Personalities 5.64
Site Visits 4.86
Networking 4.71
Role-Plays 3.93
Probation Draft 3.64
Pilot Projects 3.42
A final step involved validating the salience score by considering whether differ-
ences between groups within the sample had skewed the findings. Differences
between groups can mean that what you thought were generalizable findings are
instead the results of strongly held views within one or more groups. In this example
there were three groupings of interest. These included male and female, participants
from Riga and outside Riga, and headquarters staff and probation officers. There were
mean differences between the groups within the sample; through t tests (adjusted for
undertaking multiple tests), these differences were found to be statistically insignifi-
cant in all instances. This means that the findings that made up the salience score can
be attributed to the group as a whole.
Qualitative Nuance and Embedding Data
As we saw above, the data were collected sequentially and weighted in such a way
as to privilege data collected through the mind maps and open-ended interview
questions. Although the quantification of qualitative data (Sandelowski, 2001) pro-

vided a means to develop a unique “mixed methods measure” (Wheeldon, 2010b),
this study relied on qualitative data gathered from the interviews to provide
another means to understand the value of the project from the participant’s point
of view. These data were mixed in such a way that compiled interview data were
embedded within the numeric findings to provide a more detailed means to under-
stand “why the concepts were identified as important, and how they might be inter-
related” (Wheeldon, 2010a, p. 519). Using this approach allowed the qualitative data
drawn from the interviews to provide some context to the numeric salience scores.
As depicted above, personalities were identified as the single most important
feature of the project. As such, interview results that spoke to the nature of the rela-
tionships should be presented first. These included statements about the trust par-
ticipants had in the “experience and expertise” of the trainers and how they saw them
as “friends and role models” who were willing to share both their successes and their
failures and “took time to learn about Latvia.” Embedding qualitative data based on
numeric salience also lends itself to the inclusion of interview data that considered
site visits to Canada. These were described as integral in allowing the participants a
chance to “see a variety of programs and services” and learn about “pre-sentence
reports, risk needs assessments, mediation programs, and post-penitentiary assis-
tance.” By seeing the “work in action” the tour provided important “practical experi-
ence.” Finally, the third most “salient” aspect of the training was networking.
Participants suggested project activities had assisted “team building between
Latvians” and helped to create a “common strategy” for Latvia (Wheeldon, 2010a).
Discussion and Limitations
This study developed an approach that allowed for the numeric salience score to help
present and organize qualitative findings about which elements of the training methods
and approaches were most useful. By mixing methods in this way, the research not only
presented a sense of what worked but provided some context and nuance about why
and how. The participants also noted the utility of the maps. Virtually all participants
identified the maps as a “useful way to see experience.” Some suggested this was
because making a map “helped them to remember events from years ago” and “organize
their thoughts about the experience systematically.” Others suggested that as visual
aids, maps helped put the experience in “context,” provided a “clearer view” by allowing
them to look at events again and realize how much had happened, and helped them to
“focus on the key experiences, concepts and connections.” For these participants, there
was value in visualizing their experiences and organizing their thoughts through maps.
Although the data collected in this study have been analyzed in a variety of ways
(Wheeldon, 2010a, 2011), they also provide a useful example to consider another way
maps can be used in terms of the timing, weighting, and mixing of data. Using mind
maps in this way allows researchers to embrace quantitative measures that use qualitative
assumptions about which sorts of data are valuable and how they might be privileged.
The mixed measure should be built on and revised, but it represents a unique way to
combine quantitative and qualitative data as presented in Figure 5.16.
Some limitations with this study include the sample size and the choices made
within the method and analysis strategy. The development of a mixed-methods mea-
sure called the salience score usefully combines elements from both the quantitative
and the qualitative traditions; however, it remains untested and only a first draft of
sorts. By privileging more user-generated data-collection stages by assigning more
weight to the variables that emerge through these stages, the mixed-methods
measure combined the “clarity of counts, with the nuance qualitative reflection can
provide” (Wheeldon, 2010b, p. 87). Yet its novelty is an inherent limitation. There are
few studies that have attempted to weight data in this way, and more study is needed
to understand the value of a mixed-methods measure. One useful approach for others
testing this measure would be to develop an additional validation process in which
focus groups made up of a study’s participants could validate the main findings. In
this way, one could test whether the main findings that emerged through the score
Figure 5.16 One View of a Mixed-Methods Measure
Mixed-Methods Measure
can combine
by
Quantitative Qualitative weighting
indicators assumptions concepts that
emerge from
estimated as
Structured/ Free form/
justified as closed format open ended
data collection data collection
Reliable LOWER HIGHER
and Valid
connected Salient and combined/

to assessment transformed into
must be
Statistically validated not the Specific group

as a general measure result of characteristic(s)
were seen as important by focus groups representative of the total sample. These
sorts of validation exercises can allow the findings to be reviewed by the participants
themselves through a more participatory approach toward the research process itself.
STUDENT ACTIVITY
Review the student activities in Chapters 2, 3, and 4. Consider how adding another
method to either of these activities can assist you to better understand the issue
under investigation. In Chapter 2, your class might have considered students’ ability
to recall key concepts and their relationships based on a lecture using concept maps.
In contrast, your class might have used mind maps to consider student perceptions of
the value of the material presented on that day. How might a mixed approach give you
more data from which to draw conclusions? Imagine each person in your class com-
pleted a mind map about the perceived value of that week’s lesson at the beginning of
class, based on that week’s readings. Now imagine that following the lesson, each
person completed a concept map in which he or she was to connect concepts and
propositions based on the lesson. Generate some hypotheses about what you might
see if you were to compare an individual’s prelesson interest level with his or her
postlesson understanding. What might this approach to student comparison miss?
How might you address this limitation?
Based on Chapter 2’s activity and the analysis presented in Chapter 3, how could
concept maps be used to explore how students learned concepts presented in a weekly
lesson? What additional information might be useful to gather? How could questions
to students about the most difficult concepts, propositions, or connections assist them
to reflect on their own learning and allow for teachers to better understand student
difficulties? How might you combine different sorts of data based on the timing,
weighting, and mixing considerations described above? Based on Chapter 4’s activity
and the analysis presented in Chapter 5, how could mind maps and interviews be
scored to assess their description of key people or events in their lives? How might the
different approaches to data gathering influence how you might score the data col-
lected in each? Are there common ideas that continually emerge? What additional
information might be useful to gather? How does this attempt to quantify qualitative
data assist your understanding, and to what extent do the numbers in your scoring
system connect to your experience interviewing your participant?
CONCLUSION
As mixed-methods research continues to grow, the use of maps as an alternative form

of data collection can be seen as part of a more pragmatic understanding of intuitive
and abductive connections between theory and data. Indeed by combining quantitative
and qualitative approaches alongside their associated data analysis strategies, mixed
methods provide a means to gain a better understanding of phenomena under inves-
tigation. As visual records of understanding, concept maps and mind maps may be
important tools in this regard because the data that are represented through their
construction can be assessed both quantitatively and qualitatively.
This chapter has provided both a theoretical justification for the use of concept
maps and mind maps in mixed-methods research and some examples of how maps
might be used in this way. Pre/post concept maps offer one way to investigate how
views have quantitatively changed over time and suggest a means to explore in more
detail some of the reasons why using qualitative techniques makes sense. The mixed-
methods measure is a unique way to consider how data gathered through multiple
stages of data collection can be compiled. This single measure explicitly values data
collected through more unsolicited means while at the same time ensuring the
reliability of counts is respected.
REVIEW
1. Define mixed-methods research, and explain the assumptions about knowledge on which it is
based. How is it different from quantitative and qualitative research?
2. What are three ways mixed-methods studies have been undertaken in the past?
3. Why might concept maps and mind maps be useful for mixed-methods research?
4. How can pre/post concept maps be used with other kinds of methods?
5. What is a mixed-methods measure? How was it first constructed, and how might it be improved?
SUGGESTED ADDITIONAL READINGS
Creswell, J., & Plano Clark, V. (2007). Designing and conducting mixed methods research. London:
Sage.
Morgan, D. L. (2007). Paradigms lost and pragmatism regained: Methodological implications of
combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1),
48–76.
Teddlie, C. B., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quan-
titative and qualitative approaches in the social and behavioral sciences. Thousand Oaks,
CA: Sage.
Wheeldon, J. P. (2010). Mapping mixed methods research: Methods, measures, and meaning.
Journal of Mixed Methods Research, 4(2), 87–102.
REFERENCES
Aldridge, J. M., Fraser, B. J., & Huang, T. (1999). Investigating classroom environments in Taiwan
and Australia with multiple research methods. Journal of Educational Research, 93, 48–57.
Baumann, C. (1999). Adoptive fathers and birthfathers: A study of attitudes. Child and
Adolescent Social Work Journal, 5(16), 373–391.
Cash, D., Clark, W. C., Alcock, F., Dickson, N., Eckley, N., & Jäger, J. (2002). Credibility, legitimacy
and boundaries: Linking, assessment and decision making. Cambridge, MA: Kennedy
School of Government.
Cederblom, J., & Spohn, C. (1991). A model for teaching criminal justice ethics. Journal of
Criminal Justice Education, 2, 201–218.
Creswell, J., & Plano Clark, V. (2007). Designing and conducting mixed methods research. London:
Sage.
Feinberg, J. (2010). Wordle. Retrieved July 28, 2010, from http://www.wordle.net/
Goertzel, T., & Fashing, J. (1981). The myth of the normal curve: A theoretical critique and
examination of its role in teaching and research. Humanity and Society, 5, 14–31.
Gogolin, L., & Swartz, F. (1992). A quantitative and qualitative inquiry into the attitudes toward
science of nonscience college students. Journal of Research in Science Teaching, 29, 487–504.
Greene, J. C., & Caracelli, V. J. (1997). Defining and describing the paradigm issue in mixed-
method evaluation. In J. C. Greene & V. J. Caracelli (Eds.), Advances in mixed-method
evaluation: The challenges and benefits of integrating diverse paradigms (New Directions
for Program Evaluation, No. 74, pp. 5–17). San Francisco: Jossey-Bass.
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-
method evaluation design. Educational Evaluation and Policy Analysis, 11(3), 255–274.
Guba, E., & Lincoln, Y. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.
Jang, E. E., McDougall, D. E., Pollon, D., Herbert, M., & Russell, P. (2008). Integrative mixed meth-
ods data analysis strategies in research on school success in challenging circumstances.
Jenkins, J. E. (2001). Rural adolescent perceptions of alcohol and other drug resistance. Child
Study Journal, 31(4), 211–224.
Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action.
Administrative Science Quarterly, 24, 602–611.
Johnson, R. B., & Onwuegbuzie, A. (2004). Mixed methods research: A research paradigm whose
time has come. Educational Researcher, 33(7), 14–26.
Kaplan, A. (1964). The conduct of inquiry: Methodology for behavioral science. San Francisco:
Chandler Press.
Kilic, Z., Kaya, O. N., & Dogan, A. (2004, August 3–8). Effects of students’ pre- and post-laboratory
concept maps on students’ attitudes toward chemistry laboratory in university general chem-
istry. Paper presented at the International Conference on Chemical Education, Istanbul,
Turkey.
Miles, M., & Huberman, M. (2002). Reflections and advice. In M. Huberman & M. Miles (Eds.),
The qualitative researcher’s companion (pp. 393–397). Thousand Oaks, CA: Sage.
Morgan, D. L. (2007). Paradigms lost and pragmatism regained: Methodological implications
of combining qualitative and quantitative methods. Journal of Mixed Methods Research,
1(1), 48–76.
Morse, J. M. (2003). Principles of mixed methods and multi-method research design. In

A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral
research (pp. 189-208). Thousand Oaks, CA: Sage.
Myers, K., & Oetzel, J. (2003). Exploring the dimensions of organizational assimilation: Creating
and validating a communication measure. Communication Quarterly, 51, 436–455.
Novak, J. D., & Cañas, A. J. (2008). The theory underlying concept maps and how to construct and
use them. Pensacola: Florida Institute for Human and Machine Cognition.
Novak, J. D., & Gowin, J. B. (1984). Learning how to learn. Cambridge, UK: Cambridge University
Press.
Palys, T. (1992). Research decisions: Quantitative and qualitative perspectives (3rd ed.). Toronto:
Thompson Canada.
Rorty, R. (1999). Philosophy and social hope. London: Penguin.
Sandelowski, M. (2001). Real qualitative researchers do not count: The use of numbers in
qualitative research. Research in Nursing and Health, 24(3), 230–240.
Stewart, J., Van Kirk, J., & Rowell, R. (1979). Concept maps: A tool for use in biology teaching.
American Biology Teacher, 41(3), 171–175.
Stillwell, W., Winterfeldt, D. V., & John, R. S. (1987). Comparing hierarchical and nonhierarchical
weighting methods for eliciting multiattribute value models. Management Science, 33(4),
937–943.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative
approaches. Thousand Oaks, CA: Sage.
Teddlie, C. B., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quan-
titative and qualitative approaches in the social and behavioral sciences. Thousand Oaks,
CA: Sage.
Turns, J., Atman, C., & Adams, R. (2000). Concept maps for engineering education: A cognitively
motivated tool supporting varied assessment functions. IEEE Transactions on Education,
43, 164–173.
Way, N., Stauber, H. Y., Nakkula, M. J., & London, P. (1994). Depression and substance use in two
divergent high school cultures: A quantitative and qualitative analysis. Journal of Youth
and Adolescence, 23(3), 331–335.
Wheeldon, J. P. (2008, November 12–15). Mapping ethics, values and problem solving among
criminal justice students. Paper presented at the American Society of Criminology Annual
Meeting, St. Louis, MO.
Wheeldon, J. P. (2010a). Learning from Latvia: Adoption, adaptation, and evidence based justice
reform. Journal of Baltic Studies, 41(4), 507–530.
Wheeldon, J. P. (2010b). Mapping mixed methods research: Methods, measures, and meaning.
Wheeldon, J. P. (2011). Is a picture worth a thousand words? Using mind maps to facilitate
participant recall in qualitative research. Qualitative Report, 16(2), 509–522. Retrieved
March 2, 2011, from http://www.nova.edu/ssss/QR/QR16–2/wheeldon.pdf
Wheeldon, J. P., & Faubert, J. (2009). Framing experience: Concept maps, mind maps, and data
collection in qualitative research. International Journal of Qualitative Methods, 8(3), 68–83.

Mapping Mixed-Methods Research: Theories, Models, and Measures

Uploaded by

Copyright:

Available Formats

Mapping Mixed-Methods Research: Theories, Models, and Measures

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Mapping Mixed-Methods Research: Theories, Models, and Measures

Uploaded by

Copyright:

Available Formats

5

As we saw in Chapters 3 and 4, quantitative and qualitative researchers pursue differ-

· describe the potential of mixed-methods research and one theoretical basis

Another view is one promoted by constructivists. Skeptical of the idea of one

Table 5.1 Key Issues in Social Science Research

Quantitative Approach Qualitative Approach Pragmatic Approach

Connection of Theory Deductive Inductive Abductive

Relationship to Objectivity Subjectivity Intersubjectivity

Inference From Data Generality Context Transferability

Source: Morgan (2007, p. 71).

Figure 5.1 Comparing Deductive and Inductive Reasoning

Stated facts or general

Developing More general

Mixed-methods research represents an important departure from the either/or

Figure 5.2 One View of Abductive Reasoning

based on Qualitative Quantitative based on

Inductive to test/ Deductive

Is based on best Acknowledges

Beyond these practical benefits, conceptually mixed-methods research and the

UNDERSTANDING, PLANNING, AND

Figure 5.3 Quantitatively Testing Qualitative Findings

Analyzing Develop theory

Another approach is to develop a quantifiable means that can test a generated

Figure 5.4 Qualitatively Validating Quantitative Findings

Research Testable Development Collected

for the and can lead to based on

Figure 5.5 Timing and Ordering of Data Collection/Analysis in Mixed Methods

In which order will you collect/analyze data?

Collect/analyze Collect/analyze through

Concurrent Sequential Multiple

Note: QUAN = primarily quantitative; QUAL = primarily qualitative.

research question, your epistemological view, practical issues surrounding access

Table 5.2 Notions in Mixed-Methods Research

QUAN Primarily a quantitative mixed-methods project

QUAL Primarily a qualitative mixed-methods project

Plus sign (+) Data collection/analysis conducted at the same time

Arrow (→) The sequence of data collection/analysis in mixed-methods

quan Secondarily a quantitative mixed-methods project

qual Secondarily a qualitative mixed-methods project

What kind of mixed-methods projects do the following notations indicate?

Figure 5.6 Mixing Strategies in Mixed-Methods Research

Mixing Data in Mixed-Methods Research

Integrate QUAL QUAN QUAL QUAN

Always Justify Your Approach

Note: QUAL = primarily qualitative; QUAN = primarily quantitative.

MAPS, DATA, AND INTEGRITY IN MIXED-METHODS RESEARCH

Before we turn to a couple of mixed-methods research examples, it may be useful to

It is important to recognize that the quality of mixed-methods research is

RESEARCH EXAMPLES USING CONCEPT MAPS AND MIND MAPS

Pre/Post Concept Maps and Validation

assessed, scored, and/or compared to assess an individual’s understanding of a topic.

Overview and Mixed Design

Forty-five students enrolled in the Administration of Justice internship program at

timing aspect of the mixed-methods design might be described as multistage and

Collecting and Analyzing Qualitative Data

Table 5.3 Mapping Values Survey Questions

Note: ADJ = Administration of Justice.

Following the conclusion of their internships, these same students participated

Testing the Findings: Quantitative Pre/Post Concept Map