Integrating and Advancing Policy & Program
Implementation Research
Jodi Sandfort
Associate Professor, Humphrey School of Public Affairs,
University of Minnesota
Stephen Roll
Doctoral candidate, Glenn School of Public Affairs,
The Ohio State University
Stephanie Moulton
Associate Professor, Glenn School of Public Affairs,
The Ohio State University
October 23, 2013
Paper presented at the Association for Public Policy Analysis and Management annual conference,
Washington, DC. November 7-9, 2013
Within our field, the topic of policy implementation has a complex and controversial intellectual
history (deLeon & deLeon, 2002; Klijn, 2005; O’Toole, 2004). On the one hand, the implementation of
public policies and programs can be viewed as a component of mainstream public administration and
management research. Topics explored such as human resources, budgeting practices, performance
measurement, or privatization strategies provide insights about how agencies contribute to (or deter
from) successful policy outcomes(Brown, Potoski, & Van Slyke, 2006; Fredrickson & Fredrickson, 2007;
Moynihan, 2008). Yet, the advent of a new generation of public policy schools in the 1970s proclaimed
implementation as a “new” topic of scholarly exploration; rather than starting from the organization as
the unit of analysis, this stream of scholarship started with a specific policy, isolating the managerial
dimensions of policy or program implementation in relation to other causal factors (Bardach 1977;
Mazmanian and Sabatier 1989; Pressman and Wildavsky 1973). During the 1980s-1990s, there was
significant attention to trying to develop a generalizable model of policy and program implementation
(DeLeon 1999; Goggin et al. 1990; O’Toole 1993). However, this scholarly attention has faded in the last
ten to fifteen years within the core disciplines of public affairs and political science.
At the same time, the growing demand by policymakers for adoption of evidence-based
practices in education, health care delivery, and children & youth services has caused research on
implementation in these services to grow. Saetren (2005) offered some evidence of the growth of this
trend; during the period 1985-2003, 72% of the nearly 2,500 scholarly publications referencing “policy
implementation” were published in journals outside of public administration, policy and political
science. In this paper, we embark upon a similar investigation, trying to learn more about the state of
research on policy and program implementation through a systematic analysis of scholarly articles
published over the last ten years. We deepen the analysis by applying an insight gleaned from the prior
public affairs scholarship (Lynn, Heinrich, and Hill 2001; Berman 1981; Hill and Hupe 2008; Robichau and
Lynn Jr. 2009; Kiser & Ostrom, 1982; Van Meter & Van Horn,1975 ); studying program and policy
1|Page
implementation requires a multi-level analysis because the field, organizational, and frontline settings
have unique impact on shaping the implementation process.
In this paper, we describe the scope, approach, and focus of policy and program implementation
research. In our analysis, we pay particular attention to trying to whether or not existing studies reflect
the reality that implementation takes place at these multiple levels. While acknowledged conceptually
in the literature, we know less about the extent to which implementation research reports findings
relevant to multiple levels. Promisingly, we find that one in four studies produces findings that cross
multiple levels of the implementation system, and nearly half contribute findings relevant to the specific
program under study. However, representation across levels is not equal and our analysis suggests that
a greater attention to and integration of organizational and frontline factors into studies of policy and
program implementation may be warranted.
The Study of Program and Policy Implementation
Reviews of previous literature on policy implementation often start with Pressman and
Wildavsky's book, Implementation, in 1973. However, scholarship informing implementation began
years before in fields such as public administration, political science, sociology and economics. While not
explicitly focused on a specific policy or program as the unit of analysis, these fields have provided
insights about public sector management, political systems, institutions and exchange systems critical to
an understanding of policy implementation. A classic example from sociology is Phillip Selznik’s (Selznick
1949) analysis of the Tennessee Valley Authority (TVA), where he observed that the goals and outcomes
of the federal economic development initiative were shaped substantially by the local implementation
context where cooptation by local leaders occurred.
While implementation questions had been previously explored in other fields, the Great Society
programs and subsequent growth of government interventions in the 1960s and 1970s spawned
2|Page
increasing attention to policy as the unit of analysis. Policy analyses were launched in an effort to
document the effectiveness (or ineffectiveness) of government interventions and thus secure (or
eliminate) continued funding. In the process of analyzing problems and putting forward policy solutions,
policy scholars realized that solutions needed to be implemented and, thus, investigated this process
(Allison 1972; Easton 1979; Pressman and Wildavsky 1973). This shift both altered the unit of analysis,
but also signified an important normative change. Conventional public administration was seen to focus
on bureau politics and process, without offering much relevance to public service delivery being carried
out by different instruments and through an array of institutions. Political scientists weathered their
share of critique from the emerging implementation scholars as well. Sequential linear models that
depicted implementation as merely a phase in the policy process was challenged (Sabatier and JenkinsSmith 1993). As traditional public administration and political science approaches were questions, new
schools of public policy and public affairs were launched in many major universities (Lynn 1996) with
new attention paid to policy and program implementation.1
The first generation of implementation research was predominately case studies taking a “topdown” approach to understanding and improving implementation. For example, Pressman and
Wildavsky’s (1973) book provided a case study of the local implementation of a federal economic
development program in Oakland, CA. One of their key insights was the ‘complexity of joint action’- that
the multitude of actors with different missions and timelines created implementation challenges, and
that common approaches to policy design were too indirect, requiring too much negotiation among
diverse actors. Their prescriptions for improvement, therefore, were tied to better policy design from
policymakers at the top to structure the local implementation context. Others offered similar top-down
advice for better policy design (Bardach 1977; Mazmanian and Sabatier 1989; VanMeter and VanHorn
1
Attention to this topic continues to be strong in this school, according to topics offered in the curriculum. In our
current scan of courses in fifty-two public policy and public affairs programs throughout the global (including the
Top 20 programs in the United States) the majority (60%) have courses directly related to policy and program
implementation.
3|Page
1975; Sabatier and Mazmanian 1980), stressing the importance of clear goals, limiting scope of change,
and restricting the number of actors to improve the likelihood of effectiveness.
As might be expected, other scholars pushed back against the top down approach, emphasizing
that it was not technically feasible nor politically viable for policymakers to comprehensively structure
implementation (Berman 1978, 1981; Elmore 1979-80; Lipsky 1980). For example, Berman (1978; 1981)
observed wide variation in the same policy in different local (micro-implementation) environments,
requiring policy designs that could be adapted to local conditions to prevent policy failure, particularly
for unclear technologies. These “bottom-up” scholars often called for a mapping of the local
implementation context and relationships between actors (Elmore 1979; 1982; Hjern and Porter, 1981)
that could then describe the incentive structures and behaviors on the ground. Rather than an either/or
approach, contingencies were offered where top down designs might be more applicable when
technology and goals are clear and the environment is tightly coupled; otherwise, bottom-up might be
most appropriate (Elmore 1982; Berman 1978; Sabatier 1986).
By the late 1980s, a plethora of variables were identified at the top (policy design) and bottom
(context) that might affect implementation outcomes; however, there was no comprehensive
framework or theoretical approach by which to make sense of the variables (Goggin 1986; O'Toole
1986). Thus, the next phase in policy implementation research was marked by frameworks and
techniques to integrate factors affecting implementation, and specifying when certain types of factors
would be more (or less) important (Goggin et al. 1990; Matland 1995; Rothstein 1998; Sabatier 1988;
Schneider and Ingram 1990). Perhaps the most comprehensive framework was proposed by Matland
(1995), who analyzed conditions under which implementation success should be driven by fidelity to a
policy design (which would then lead to intended policy outcomes), versus evolving from the
implementation process (e.g., through experimentation) in order to achieve positive outcomes.
4|Page
The focus on implementation within the core disciplines of public affairs began to dissipate in
the 1990s, leading some to conclude that interest in the subject had declined (DeLeon 1999), or that the
focus was no longer useful with more precise research questions, constructs and methods to analyze
complex systems (O’Toole 2000). However, research related to implementation continues to evolve.
Three trends are particularly worth mentioning. First, some scholars pushed to focus on the
coordinating mechanisms or policy tools (Schneider and Ingram 1990; Salamon 2002). These authors
focus on these coordinating mechanisms as the unit of analysis (e.g. contracts, grants, subsidies,
incentives) rather than a policy or program; they seek to identify systematic variation in these tools of
government, to inform their appropriate use in different situations (Salamon 2002). While heuristically
this approach has merit in explaining the different “levers” involved in implementation processes,
research has demonstrated that there continues to be much variation in outcomes that cannot be
explained by the tools (Blair 2002; Romzek and Johnston 2002; Sandfort, Selden, and Sowa 2008;
Twombly and Boris 1999). Further, the choice of a particular tool is endogenous to the implementation
environment. The mechanisms used coordinate actions are not only specified by the technology, but
also by power distributions (e.g. Matland 1995) and the institutional environment (e.g. Berman 1980;
Elmore 1982). As an endogenous variable, it is not sufficient to primarily focus on the coordinating
mechanisms to describe or predict implementation outcomes. Thus, while the tools approach helps
clarify one dimension of the implementation context (coordination), it does not sufficiently replace the
broader implementation research agenda.
Second, while many scholars stressed implementation systems are multi-level, multi-actor (e.g.
Berman 1978; Hall and O’Toole 2000), more mainstream public management moved from concern of
governmental agencies to consider multi-level governance and networks (Agranoff and McGuire 2003;
Frederickson 2005; Lynn et al. 2001; Milward and Provan 2003). The focus of this scholarship is broad
and less focused on specific policies or programs than original implementation research. However, the
5|Page
recognition of implementation taking place within multi-level systems complicates some of the prior
frameworks that have sought to make sense of the implementation environment. For example, although
conflict and ambiguity are critical dimensions affecting implementation outcomes (Matland, 1995), this
factors exist not only in policy formation but also within implementing organizations and at front-lines.
For implementation research to be useful, effort is needed to more appropriately integrate the multiple
levels within the system; our analysis of the existing scholarly research over the last ten years moves us
in that direction.
A final trend relevant is the blossoming of implementation studies in other research fields
(Nilsen et al. 2013; Saetren 2005). In areas such as medicine, community psychology, early childhood
development, youth and family programs, and education, researchers have made considerable inroads
in developing models and methods for studying implementation. These investigations have modest
scope. But a new section of the American Psychological Association, a new journal Implementation
Science, and biennial conference sponsored by the Global Implementation Initiative bespeaks the
growth of this area of research. In the introductory volume to Implementation Science the editors’
clarified their charge (Eccles & Mittman, 2006:1): “Implementation research is the scientific study of
methods to promote the systematic uptake of research findings and other evidence-based practices into
routine practice, and, hence, to improve the quality and effectiveness of … services and care.” Like the
original policy and program implementation studies, however, many models and theories have
developed; A recent analysis identified sixty-one different models being used to explore innovation
dissemination and implementation (Tabak et al. 2012). Recent reviews are seeking to develop
integrative frameworks or conceptual models (Durlak and DuPre 2008; Fixsen et al. 2005; Greenhalgh et
al. 2004; Meyers, Durlak, and Wandersman 2012). Yet, there is recognition that scant attention is paid
to the policy environment; while it is widely recognized as significant, only 13 percent of the models
Tabak and colleagues’ catalogue incorporate policy activities. In the widely used ‘Consolidated
6|Page
Framework for Implementation Research,’ it is merely referred to as the “outer setting” (Damschroder
et al. 2009). In spite of this fairly limited conception, a number of federal agencies in the Departments of
Education, Health and Human Services, and Veterans Affairs are investing in the implementation science
approach to scale evidence-based interventions because, for policy makers interested in affecting
outcomes, knowledge about implementation is essential.
In summary, issues of policy and program implementation are ever more pressing today than
they were in the 1970s. In this analysis, we are probing the paradox which Saetren (2005) articulates –
public affairs scholars believe interest in implementation research has faded and yet there is a thriving
research field exploring implementation questions important for public and nonprofit managers and
policy makers. Rather than a discrete body of literature within public affairs or political science, research
on implementation today is more heterogeneous and spread across a variety of fields of study. One way
to make sense of this diverse literature is to classify the findings across this literature according to the
level of analysis, probing the extent to which implementation research is focused on the frontlines,
organizations, and/or policy systems.
Multi-Level Framework
Our analysis is informed by this notion that significant policy and program implementation
activities occur in differentiated levels of a larger policy system (Berman 1981; Hill and Hupe 2008;
Robichau and Lynn Jr. 2009; Sabatier and Jenkins-Smith 1993). A multi-level framework is appropriate.
Implementation influences results at the frontlines, where the policy system interacts with the target
population, such as children, tax payers, employers, or business owners. At this micro-level, many
factors may be significant, such as target groups’ composition & attitudes, staff background and
experiences, and things which structure their interaction such paperwork or technology (Lipsky 1980;
Sandfort 2003; Watkins-Hayes 2009). These interactions are directly shaped by other factors at the
organizational level. Service organizations’ resources, structures, cultures, and competing
7|Page
programmatic responsibilities often determine the most prudent way the agency responds to policy and
program implementation pressures(Lin 2000; Spillane 1998). Authorizing agencies’ also shape
implementation through the administrative rules adopted, funding instruments selected, and
performance definitions (Berman 1978; Wilson 1991) . The organizational-level factors are found at the
mezzo-level of implementation systems.
Finally, at the policy field level, other macro-level factors come into focus (Milward and Wamsley
1984; Sandfort 2010; Stone and Sandfort 2009; Weible, Sabatier, and McQueen 2009).2 The institutions,
laws, tools used to address this type of issue in the past shape the way implementation problems and
solutions are understood. The networks among professionals, power concentration, and resources are
significant. To explore the significance a multi-level system approach throughout the existing research,
we categorize research findings as relevant to deepening understanding at the micro - meso, and/or
macro-levels in our analysis. By differentiating among these levels, we move closer to understanding
the state of research about policy and program implementing and pointing towards new avenues for
renewed investigations within public affairs.
Research Questions and Methodology
Our central research question is, “What is the scope, approach, and focus of the scientific field
of policy and program implementation research?” We are specifically interested in how the multi-level
framework suggested by policy and program implementation researchers is appearing in the extant
literature. To explore this question, we based our approach loosely on that undertaken by Saetren
(2005) who examined the development of policy implementation research from 1933-2003. We drew
our sample from 2004-2013 journals listed in the Expanded Social Science Citation Index in the Web of
Science, which covers over 8,500 major journals. From a potential population of over 14 million articles
over the years relevant to our study, we included any articles which included “policy implementation”
2
Other scholars have referred to this level as “policy subsystem.”
8|Page
and “program implementation” in the title, abstract, or author supplied key words using HistCite
software.3 In the course of our subsequent coding, we found 436 articles that – while including the key
terms in their title, abstract, or keywords – were not really focused on the topic. This is undoubtedly
because of the more conventional use of the term “implementation.” For example, such as articles
focused on implementation of computer programs or those mentioning “implementation” informally as
descriptor. We omitted these articles, as well as negligible number (4) of articles which did not include
an abstract, from our analysis for a total of 1,375 articles in our full sample.
We begin our review of this literature with a bibliometric analysis, exploring the concentration
of publication venues as an indicator of scholarly communication in a field. In Saetren’s (2005)
treatment of the scholarly implementation field, he includes policy, public administration, and political
science journals in his sample as “core” public affairs venues. We first adopted his approach in the
coding in our sample and found similar trends to what he reported ten years ago: in our analysis, 9
percent of the studies were published in his “core journals” for public affairs compared to 12 percent in
2003, 14 percent were in his “near-core” compared to 16 percent, and 77 percent were in non-core
journals compared to 72 percent.
Yet we wanted to strengthen this analysis on a number of key dimensions. First, we utilize a
more objective assessment of journals relying upon the Web of Knowledge (ISI) to identify journals
officially classified as “public administration” or “political science” to constitute our notion of “public
affairs core” journals. Second, because of our understanding of the larger trends of implementation
scholarship, we wanted to explore the growing attention to micro-implementation dynamics found in
implementation science. As a result, we identified a list of ten journals most frequently identified as
3
Initially, we sought to identify sources using a criteria that included the words “policy” or “program” and the word
“implementation” (as well as variations on these words) anywhere in the abstract, title, or keywords. This initial
scan yielded ~25,000 articles, a number too large to reasonably manage.
9|Page
significant outlets in numerous literature reviews (Durlak and DuPre 2008; Fixsen et al. 2005; Meyers et
al. 2012) designating them as “implementation science core” journals.4
Given the larger scholarly trends, we also want to do more detailed analysis of the full sample of
articles. To enable closer coding, we utilize Nvivo software to differentiate the abstracts across a
number of different key elements. We developed the descriptive coding described and adjusted it after
an initial review of 30 percent of the articles in our sample, to accurately and comprehensively describe
the sample. While we did all coding from the abstracts, occasionally the keywords and titles of the
articles were used to inform the coding when the abstract was unclear.
Consistent with our interest in multi-level framework, the findings in each article are coded as
relevant to “program-specific,” “front line,” “organizational,” “policy-field” levels, or “general
implementation” findings. Program findings include any findings which are directly related to the
evaluation of a specific program or policy in terms of its outcomes or impacts. These are likely the least
generalizable outside of the context of the specific study. Front-line factors are those relevant to
understanding frontline-staff, clients, or their interactions, as well as any findings relevant to
understanding how these dynamics impact policy. Organizational factors are those characteristics that
are significant in the implementation process, such as managerial characteristics, culture, capacity,
resources, or facilities. Policy field factors are those which have large-scale implications for policy,
networks, or the general structure of the policy system. General implementation factors are those
which focus on the improvement of implementation knowledge, either on a general level or within the
context of a nation or state. We determined our code of findings in relation to the focus and/or the
design of the study. By this standard, a study using a survey to assess teachers’ responses to the
implementation of a mentoring program would be coded as presenting “front-line” findings; if the
4
They are: American Journal of Community Psychology; Health Education and Behavior; American Journal of
Evaluation; Health Education Research; Journal of Primary Prevention; Prevention Science; Implementation Science;
Journal of Community Psychology; Children & Youth Services Review; American Journal of Public Health.
10 | P a g e
abstract also mentions potential broader implications for the education field it would not be coded
policy field-level as those claims would be largely speculative and beyond the scope of the actual
analysis within the study.
There were a number of other, descriptive dimensions of interest. We coded articles for their
geographic focus to be able to describe the sites of investigation. We coded specific policy and program
content areas by health, education, environment, social welfare, general implementation, crime,
agriculture, city and regional planning, energy, transportation, science and technology, food,
international development, monetary policy, international relations, private sector implementation, and
miscellaneous. While most of these areas are relatively clear, articles coded as “general
implementation” are those that did not focus on a specific content area, but rather the topic of
implementation itself. For example, a study which focused on how American federalism inhibits
efficient policy implementation would fit into the “general implementation” code. Furthermore, in
interdisciplinary content areas, we base our coding on the dependent variable or focus of the study. For
example, while a school vaccination program may happen in an educational setting, it is coded as
“health” rather than “education” since the outcome of interest is student vaccines. While we allowed
abstracts to be coded into multiple content areas, most fell within a single area.
We also are interested in number of target populations expected to have some substantial
presence in the implementation literature: children, disabled, elderly, medically vulnerable, parents and
families, racial and ethnic groups, and those in poverty. Not all articles in the sample are coded to a
specific target population and some were coded in more of these categories (most of the target
population definitions are not mutually exclusive).5 We also examine research type differentiating
5
The key words we searched for were often synonyms for the target population, such as “aged” or “senior for the
elderly target, or “youth” or “adolescent” for the child target. Where appropriate, we also included a number of
specific words associated with the targets based on an initial review of the abstract sample. For the medically
vulnerable target, these included references to specific diseases or ailments that are commonly referenced in the
11 | P a g e
“conceptual” and “empirical studies.” Conceptual studies focus on implementation without any
accompanying data or specific implementation case. Empirical studies were those that used some
source of data or specific real-world instances of policy or program implementation as their focus; we
further subdivide these based on their methods of analysis and research design. The methods include
“quantitative,” “qualitative,” and “mixed” studies that use both qualitative and quantitative approaches.
Qualitative methods were identified by an abstract referencing any research design that focused on the
collection of generally unstructured data: Open-ended surveys, interviews, ethnographies, and content
analyses were common approaches coded as qualitative research. Quantitative methods were
identified by references to surveys, positivist research designs or statistical analysis; typical quantitative
designs involved experiments, secondary data analysis, or closed-ended surveys. The criterion for a
study being included in one of these frames was an explicit reference to some qualitative or quantitative
data. A substantial number of the empirical abstracts (32%) do not include any explicit reference to
qualitative or quantitative methods, and are coded as unclear. From our review, this group is largely
comprised of descriptive case studies without any apparent rigorous study design; we drop these cases
for our exploration of the scientific focus of this literature. We also pulled mention of specific “research
designs” including case studies, experimental designs, content analyses, ethnographies, literature
reviews6, and “research methods,” including interviews, surveys, secondary data analyses, observations,
modeling or simulation. Studies could fit into multiple code frames if they included several different
research methods. Finally, we were also interested in the degree to which implementation articles were
focused on evaluating or analyzing specific public policies, and coded the abstracts based on whether
they made explicit reference to a particular public policy as a focus of their analysis.
literature (e.g. HIV, malaria, or malnourishment), while for the racial or ethnic minorities we searched for specific
groups (e.g. African-American or Latino/Latina).
6
Studies were only coded as having a literature review design if the literature review was the only focus.
12 | P a g e
Through this detailed review of the sample, we discovered that a significant portion (25%) of the
articles did not meet the basic standards of scientific rigor. They were neither conceptual nor empirical,
but rather most often provided general case description without any explicit methodological frame or
design. In our analysis, while we focus the first results on the full sample of 1,375 articles which include
these descriptive accounts, our more in-depth analysis of research approach and findings focus on 1033
articles that apply basic social science methodology.
Results & Discussion
To update our understanding of the major contours of policy and program implementation
research, we unpack our research question in stages, first describing the scope and focus of all published
research. We then limit the sample to only those studies using structured research methods to further
explore the findings across the multiple levels of policy and program implementation.
What is the scope and focus of published research on policy and program
implementation?
In Saetren’s (2005) article, he documents 3,523 research articles published in the field in the
seventy year period he studies, seventy-percent between 1985 and 2003. The final five year period
covered in his sample reveals significant growth in numbers of articles published. In the ten years since
2003, our sample includes 1,375 articles, further evidence of the expansion. Taken together, there is a
sizable body of research literature exploring policy and program implementation in scholarly journals.
Together with the other indicators noted earlier, such as the development of new journals and research
associations, this evidence suggests a robust field of inquiry.
To get a deeper understanding of the scope of the field, we sought to understand the
distribution of publications by outlet. Table 1 presents the comparison between publications outlets;
twelve percent of the articles are found in core public affairs journals and six percent in implementation
13 | P a g e
science core journals. It is notable
le th
that none of the identified journals we classify as implementation
imp
science core overlap with the ISI pub
public affairs core journals; this suggests that this trend
rend of
implementation scholarship is develo
veloping parallel to the extant public affairs research.
ch. When you adjust
for the much broader the pool off pub
public affairs journals (over 190 are included in the ISI classification), it
also is striking that the ten journals
als d
designated at as implementation science core are
e capturing
ca
a
comparably larger amount of articles
icles focused on policy and program implementation.
n.
Table 1
Geographic focus on the top
topics being studied is another dimension of research
rch scope.
s
Table 2
reports on the geographic focus of p
program and policy implementation research; while
hile the majority of
studies explore conditions in the Uni
United States and North America, this literature is certainly
cert
international. Twenty-one percent
nt o
of articles focus on European contexts (predominant
nantly Western
Europe), twelve percent focus in Asia
Asia, eight percent in Africa, and another eight percent
cent on international
policy or programs. In the last ten
n ye
years, then, this literature reflects concern for implem
plementation
dynamics around the globe.
Table 2:
14 | P a g e
As noted above, unlike more
ore general organizational management studies, imple
plementation is not
focused on changing the operations
ons of networks, organizations, and frontline operation
tions but upon doing
so in relation to particular problems
ms found in topical areas. It focuses in particular conte
ontent areas. Table
3 illustrates the distribution of our
ur sa
sample across 17 distinct content categories. Health
alth is the most
common, with nearly half of the sam
sample (48 percent) coming from this arena. Educatio
tion (19%),
environment (11%), and social welfa
elfare (9%) are the next most common focus areas. A handful
h
of papers
concentrate on the other areas. Wh
While Saetren’s assessment of 1933-2003 publications
ions (2005) also finds
health, education, environment, and social welfare as the most common topical areas,
as, health
h
comprised
only 24% of the sample during that
at p
period, with a comparable number of articles focuse
cused on education.
This suggests the growth of impleme
ementation studies focused in health care, again consis
nsistent with the
more general picture painted earlier
rlier about the focus of the implementation science research
rese
stream.
Table 3:
15 | P a g e
Another dimension of scope
ope is whether or not the research includes mention
n of a specific public
policy including legislation, executive
utive orders or agency mandates. Historically, conceptu
eptualizations of
policy implementation take as thee st
starting point enacted public policies. However, other
thers have rightly
argued that much of policy implemen
mentation takes place on the ground, with ongoing evolution
evo
of
programs. Thus, we explore is thee ex
extent to which published research on policy and program
prog
implementation includes explicit me
mention of a particularly public policy in the abstract
ct or
o keywords,
(suggesting that formal policy is cent
central to the analysis). Of those mentioning specific
ic policies,
po
we
further identify the level of governm
rnment for the policy (federal, state, local, or internatio
ational levels). Of
the articles in our sample, nearly a qu
quarter of the abstracts included mention a particula
icular policy. Those
articles which focused on agriculture
ture or the environment had the most focus on specific
cific public policies,
with over 40% of the abstracts coded
oded in each field making reference to specific policies.
ies. Abstracts in
areas with a relatively heavy program
gram-specific focus, such as health or education, have
ve a smaller
proportion of studies dedicated to ex
explicit policy evaluations, with only 23% of educatio
ation abstracts and
16 | P a g e
14% of health abstracts referencing explicit policies. Social welfare abstracts fall in the middle of the
aforementioned fields, with almost 30% of social welfare articles making explicit mention of public
policies.
Because policy and program implementation is fundamentally about making change in some
condition, we also were interested in research focused on target groups. While the possible targets
groups are diverse, in this description we focus on children, disabled, elderly, medically vulnerable,
parents/family, the poor, and racial or ethnic minority groups. Thirty-four percent of the articles on
program and policy implementation focused their attention on services reaching one of these groups.
Most common were medically vulnerable (15%), followed by children (13%) and parents/family (7%),
not surprising in research in which the majority of studies focus in either health or education.7
As the history of this scholarly field and recent trends suggests, there are apt to be a diverse
approach to conducting research about policy and program implementation. We wanted to describe
this diversity in our sample. We first differentiated between conceptual studies and those working with
some type of data or information. A small portion of the articles (6%) were conceptual; a typical
conceptual study might investigate the potential impact of implementing a policy tool, such as the
potential carbon tax schemes might have in influencing international trade dynamics.
We then explored in more detail the research approach among the empirical studies.
Dominated by the health care content area, the most prevalent research methods were quantitative
(35%), using surveys or secondary data and statistical analysis methods; in fact, experimental designs,
population surveys, and secondary data analysis are the most comment research methods used in the
whole sample. Qualitative research methods, interviews or focus groups and ethnographic methods,
are found in twenty percent of the articles, and are particularly use in education, environment, and
social welfare topics where they are deployed as frequently as quantitative methods. A small number of
7
Interesting, only four percent focus on programs/policy assisting the poor or three percent on racial or ethnic
minority groups. And even smaller number focuses on the elderly (2%) or disabled (1%).
17 | P a g e
articles, six percent in the whole sample, use a research approach which mixes qualitative and
quantitative approaches.
Significantly, a sizable group of program and policy implementation articles did not clearly
specify a standard research method or design in the article abstracts. The most sizable of this were one
in four articles (24.8%) that did not deploy systematic research methods at all but merely described a
particular case of policy and program implementation. This result raises concern about the rigor of a
sub-group of these publications; some topics with low concentration of policy and program
implementation studies– agriculture, city and regional planning, international development and
international relations, science and technology, transportation – had high prevalence of such descriptive
case-based articles. In contrast, health and education – where fifty-eight percent of all articles were
published – had low incidence of these types of these articles. This suggests that basic standards of
social science are expected in these areas in which larger numbers of scholars are investigating program
and policy implementation.
In the rest of our analysis and exploration of the multi-level framework, we drop out these
articles and focus on those applying conventional social science designs and methods (n=1,033).
Practically, this omission increases the overall significance of research concentrating on health, as over
half of the remaining rigorous articles focus on that content area. For that reason, we present this
analysis by content area, to illuminate where there are important distinctions that might get obscured.
As Table 4 illustrates, the approaches vary significantly across these areas, suggesting field-based
research practices. For example, it is interesting to see that while experimental designs are used
relatively frequently in health and crime, it is not common in many other topics. Although case studies
are a common approach, their relative proportion in any field varies between none to one in two studies
of private business. Because the codes are not exclusive (eg. each study could use multiple methods),
each cell of this table should be understood in reference to the total number of research articles focused
18 | P a g e
in that content area. For example,
le, 220 percent of the studies of policy and program implementation
impl
focused on agriculture are case studi
tudies, seven percent are content analysis, and seven
en percent
p
are
systematic literature reviews. Thirte
irteen percent involve interviews, twenty-seven percen
rcent analyze
secondary data, and none use system
stematic observation.
Table 4 (continued on next page)
19 | P a g e
These results provide context for scholars conducting implementation research in particular
content areas, helping to illuminate the dominant research design and method used in the existing
literature. They also highlight the significant variation in research approaches being used throughout
the field. As noted earlier, while experimental designs are the most common research design, they are
being used only in selected fields; in fact, none of the research methods we coded for are being utilized
across in all of the content areas of research. In response to this variation, some implementation
science researchers are creating shared collections of data and measures, such as that being developed
by the Seattle Implementation Research collaborative, to bring more alignment on these issues.8
What is the focus of the scientific field of policy and program implementation research?
To probe more deeply the manifestation of a multi-level framework for implementation
research, we explored study findings relevant to four different levels. Given that implementation occurs
at multiple levels, and that scholars have long recognized this phenomenon, it would be desirable for
research to reflect this multi-level reality in the report of their results to inspire informed
implementation practice. Overall, 24 percent of the studies in our sample of rigorous studies present
findings relevant to multiple-levels of implementation systems. Almost half of the studies, 49 percent
of the sample of rigorous studies, report program findings directly related to a particular program or
policy and its results. This finding challenges a criticism often made of implementation research that it
loses site of ultimate results, either in system change or target group conditions. Almost four hundred
of the studies (37 percent) also include findings related to the macro-policy field level, issues relevant
networks, large scale policy or general structure of the system. Smaller numbers of articles include
findings relevant to frontline conditions (19 percent) or organizational factors (16 percent) such as
8
See http://www.seattleimplementation.org/wp-content/uploads/2011/08/SIRC_IRP-Update_2013.pdf retrieved
on September 10, 2013.
20 | P a g e
managerial characteristics, culture, or capacity. Like all dimensions of this analysis, though, there is
important variation in these applications by content area.
Table 5 presents the level of findings by selected content areas (those with more than 100
publications). As is true throughout this analysis, the majority of studies cluster in health and education
where program-specific findings predominant. As noted earlier, considerable implementation research
is now focused on examinations of particular health or education-related interventions. Many social
welfare studies (39%) also report findings relevant to particular interventions. While such findings hold
substantive implications, the results are often not particularly generalizable to other implementation
questions. For example, relatively low proportions of implementation studies in education,
environment, health care, or social welfare report results in terms of organizational-level factors -managerial actions, organizational culture or capacity, or resources. However, this level of
implementation is substantively important in structuring the very terms of policy and program delivery.
We also are interested in whether or not findings are reported across the multi-level framework.
At least one in five studies in these content areas report multi-level findings in their abstracts. The
descriptive statistics also suggest that education and health fields are dominated by program specific
implementation studies, whereas environmental studies report policy-field, macro results (chi-squared
analysis reveals statistically different categories of comparison). Social Welfare implementation studies
appear to report more diverse array of findings (non-statistically significant different); furthermore, 38
percent of social welfare implementation studies report findings that cross the program, frontline,
organizational, or policy field level.
Table 5
21 | P a g e
As noted earlier, there are
re si
significant variations in the research approach used
ed across in this
literature. Tables 6 presents informa
rmation about these differences, again by considering
ing the results
presented in the articles in relation
on tto a multi-level framework. While some article abstracts
abst
were
ambiguous in relation to their resear
search approach, our analysis highlights that researcher
chers are deploying
particular methods to generate type
ypes of results in their studies of implementation. Among
Amo the studies
reporting program-specific findings,
gs, for example, quantitative methods clearly predomin
ominate. More
particularly 39 percent of program--specific results come from use of experimental desig
esigns, again likely
concentrated in the health area. In ccontrast, studies presenting findings relevant to the frontline or
organizational levels of practice are p
predominantly relying upon qualitative (or mixed)
d) research
re
approaches. Among studies reportin
orting findings relevant to frontline conditions, 29 perc
ercent use case
studies and 25 percent rely upon inte
interview methods. Among studies reporting organiza
nizationally-relevant
findings, 29 percent use either interv
terviews or surveys. Findings related to macro- policy
licy field
f
conditions
are split between quantitative and
dq
qualitative approaches, but also have reported their
eir research
approach more ambiguously in articl
rticle abstracts.
Table 6
22 | P a g e
Overall, this analysis highlights the diverse research being conducted about policy and program
implementation over the last ten years. While it seems a vibrant area of scholarship, it is notable by
how distinct content areas pursue unique methods of research with findings targeted at different levels
of analysis. This raises questions about the overall orientation towards building generalizable
knowledge about implementation processes useful for professionals operating at distinct levels within a
larger implementation system.
In our future analysis of these themes, we plan to explore in more detail articles seeking to
provide evidence of the multiple levels within implementation systems. Although this population level
analysis suggests some themes, we are interested in the degree to which the particular implementation
dynamics of resource adequacy, coordination, culture, service technology and change are addressed and
explored.
Conclusions
Our analysis clearly reinforces Saetren (2005)’s conclusion that scholarly research has moved
from the traditional disciplines of political science, public administration, and public policy and now is
more concentrated in content areas, such as health and education, concerned with examining
implementation dynamics in relation to program-level results. It is a research community exploring
policy and program implementation around the world using a wide array of research designs and
methods. Specifically, we find that in the 10 year period between 2003 and 2013, only 12 percent of
published articles on “policy implementation” or “program implementation” are within the 190 journals
classified as Public Affairs journals by the ISI Web of Science. Further, new journals focused specifically
on “implementation science” have been growing, with research relevant to policy and program
implementation more generally. And, scholarship is by no means limited to the U.S; while the U.S. is the
most common setting (representing 39 percent of articles), Europe is close behind (one in five studies),
with Asia also representing more than 12 percent of published studies.
23 | P a g e
The primary contribution of our analysis, however, is to describe the extent to which findings
from research on policy and program implementation inform a multi-level framework of governance.
Researchers in public affairs have acknowledged that implementation takes place within a multi-level
system including front lines, organizations, and policy fields (Berman 1978; Hill and Hupe 2008; Lynn et
al. 2001; Robichau and Lynn Jr. 2009). Criticisms of policy implementation research suggest that it is
either too broad, without policy specific relevance, or that it is too narrow, focused so singularly on a
particular program and, thus, not generalizable outside of the particular context. To the extent that
policy implementation research produces findings at various levels, particularly multiple levels within
the same analysis, such research may be able to provide policy specific relevance while still contributing
to generalizable knowledge. Promisingly, we find that one in four studies produces findings that cross
multiple levels of the governance framework, and nearly half contribute findings relevant to the specific
program under study. However, while more than one-third of studies contribute findings at the policy
field level of analysis, the organization and frontline levels comprise only 19 and 16 percent of findings
respectively. This suggests that a greater integration of organizational and frontline factors into studies
of policy and program implementation may be warranted.
Finally, our analysis also explores contributions to multi-level implementation findings by policy
fields and methodological approach. As might be expected, findings from environmental policy studies
are concentrated at the policy field level (76 percent), while findings from the health field are
concentrated at the program level (65 percent). This makes sense from a practical perspective; most
environmental policies involve numerous field level actors to bring about change, while health policies
and programs may be implemented on a smaller scale within a specific clinic or health facility. However,
cross-fertilization can occur when environmental policy researchers model some of the front-line and
organizational factors, and health policy researchers consider policy field specific factors. Our analysis
suggests that there is more room for growth in this regard. In terms of methodology, we find that while
24 | P a g e
program level findings are more likely to result from quantitative analyses (67 percent), organizational,
frontline and policy findings are more likely to employ qualitative methods. For example, nearly half (47
percent) of studies contributing organizational level findings employ qualitative methods. This suggests
that while quantitative analysis may be more feasible (and desirable) for program evaluations,
contributing to an understanding of organizational and front-line factors may require a qualitative or
mixed methods approach. Thus, while a push for quantitative, experimental research may be beneficial
to produce relevant program findings, multiple methods—including qualitative analyses-- are likely
needed to inform more generalizable findings across policy levels.
While our study offers exploratory insights, it is important to keep in mind the limitations. Our
study is based on a bibliometric and content analysis of published abstracts, employing the terms “policy
implementation” or “program implementation.” Our findings could reflect, as some suggest that the
term “implementation” has become less popular within public policy and management (Nilsen et al.
2013). There are likely numerous studies published in public affairs journals that contribute insights to
implementation, but do not employ the term when describing their central focus in published abstracts.
However, the purpose of our analysis is not to identify all relevant findings but rather to catalogue the
scope and focus of published research that is explicitly where implementation characterizes its core
focus. Further, our review only includes scholarly articles published in academic peer-review outlets
although valuable implementation studies are conducted by research firms under government
contracts. However, our approach provides a most systematic way to analyze the published and peerreviewed research.
The past decade has witnessed a steady flow of research engaging the topics of policy
implementation and program implementation. This research is not concentrated within public affairs or
a specific field of study, but is rather spread across different disciplines and policy fields. In this paper,
engage a multi-level framework provides to sort study findings across levels, identifying the extent to
25 | P a g e
which the research findings are program specific, or occur at the frontlines, organizational or field level.
We find evidence of findings across levels, with variation by methodology or program type. Future
research is needed to better unpack the substantive contributions of findings across levels.
References
Agranoff, Robert, and Michael McGuire. 2003. Collaborative Public Management: New Strategies for
Local Governments. Washington, DC: Georgetown University Press.
Allison, Graham. 1972. Essence of decision. Explaining the Cuban Missile Crisis. Boston, MA: Little Brown.
Bardach, E. 1977. The Implementation Game. Cambridge: MIT Press.
Berman, P. 1981. “Educational Change: An Implementation Paradigm.” Pp. 253–86 in Improving Schools:
Using what we know, edited by R Lehming and M Kane. Beverly Hills, CA: Sage.
Berman, Paul. 1978. The Study of Micro and Macro Implementation of Social Policy. Rand.
Blair, Robert. 2002. “Policy Tools Theory and Implementation Networks: Understanding State Enterprise
Zone Partnerships.” Journal of Public Administration Research and Theory 12(2):161–90.
Damschroder, Laura J. et al. 2009. “Fostering implementation of health services research findings into
practice: a consolidated framework for advancing implementation science.” Implementation
science 4:50.
DeLeon, Peter. 1999. “The missing link revisted: contemporary implementation research.” Policy Studies
Review1 16(3/4):311–38.
Durlak, Joseph a, and Emily P. DuPre. 2008. “Implementation matters: a review of research on the
influence of implementation on program outcomes and the factors affecting implementation.”
American journal of community psychology 41(3-4):327–50.
Easton, D. 1979. A systems analysis of political life. Chicago: University of Chicago Press.
Elmore, Richard F. n.d. “Backward Mapping : Implementation Research and Policy Decisions.” Political
Science Quarterly 94(4):601–16.
Fixsen, Dean L. et al. 2005. Implementation Research: A Synthesis of the Literature. Tampa, Fl.
Frederickson, George. 2005. “Whatever Happened to Public Administration? Governance, Governance
Everywhere.” Pp. 282–304 in The Oxford Handbook of Public Management.
Goggin, M. L. 1986. “The ‘Too Few Cases/Too Many Variables’ Problem in Implementation Research.”
Political Research Quarterly 39(2):328–47.
Goggin, M. L., A. O. M. Bowman, J. P. Lester, and L. J. O’Toole. 1990. “Studying the Dynamics of Public
Policy Implementation: A Third-Generation Approach.” Pp. 181–97 in, edited by D J Palumbo and D
J Calista. New York: Greenwood Press.
Greenhalgh, Trisha, Glenn Robert, Fraser MacFarlane, Paul Bate, and Olivia Kyriakidou. 2004. “Diffusion
of innovations in service organizations: systematic review and recommendations.” The Milbank
quarterly 82(4):581–629.
Hall, T. E., and Laurence O’Toole. 2000. “Structures for Policy Implementation: An Analysis of National
Legislation, 1965-1966 and 1993-1994.” Administration Society 31(6):667–86.
Hill, Michael, and Peter Hupe. 2008. Implementing Public Policy: An Introduction to the Study of
Operational Governance. 2nd ed. Sage Publications.
Lin, Ann Chih. 2000. Reform in the Making: The Implementation of Social Policy in Prison. Princeton, New
Jersey: Princeton University Press.
Lipsky, Michael. 1980. Street-Level Bureaucracy: Dilemmas of the Individual in Public Services. New York,
NY: Russel Sage Foundation.
26 | P a g e
Lynn, Laurence E. 1996. Public Management as Art, Science and Profession. Chatham, New Jersey:
Chatham House.
Lynn, Laurence E. Jr, Carolyn J. Heinrich, and Carolyn J. Hill. 2001. Improving Governance: A new logic
for empirical research. Washington, D.C.: Georgetown University Press.
Matland, Richard E. 1995. “Synthesizing the Implementation Literature : The Ambiguity-Conflict Model
of Policy Implementation.” Journal of Public Administration Research & Theory 5(2):145–74.
Mazmanian, D., and P. Sabatier. 1989. Implementation and Public Policy. Lanham: University Press of
America.
Meyers, Duncan C., Joseph a Durlak, and Abraham Wandersman. 2012. “The quality implementation
framework: a synthesis of critical steps in the implementation process.” American journal of
community psychology 50(3-4):462–80.
Milward, H. Brinton, and Keith G. Provan. 2003. “Managing Networks Effectively.”
Milward, H. Brinton, and Gary L. Wamsley. 1984. “Policy Subsystems, Networks, and the Tools of Public
Management.” Pp. 3–25 in Public Policy Formation, edited by Robert Eyestone. Boulder: Westview
Press.
Nilsen, Per, Christian Ståhl, Kerstin Roback, and Paul Cairney. 2013. “Never the twain shall meet?--a
comparison of implementation science and policy implementation research.” Implementation
science 8(63).
O’Toole, Laurence J. Jr. 1986. “Policy Recommendations for Multi-Actor Implementation: An Assessment
of the Field.” Journal of Public Policy 6(3):181–210.
O’Toole, Laurence J. Jr. 1993. “Interorganizational Policy Studies: Lessons Drawn from Implementation
Research.” Journal of Public Administration and Research Theory 3(22001 implementation course
pack 1043):232–51.
O’Toole, Laurence J. Jr. 2000. “Research on Policy Implementation : Assessment and Prospects.” Journal
of Public Administration Research & Theory 10:263–88.
O’Toole, Laurence J. Jr. 2004. "The Theory-Practice Issue in Policy Implementation Research. Public
Administration 82(2):309-329.
Pressman, J., and A. Wildavsky. 1973. Implementation: How Great Expectations in Washington are
Dashed in Oakland or Why it’s Amazing that Federal Programs Work at All. Berkeley: University of
California Press.
Robichau, Robbie Waters, and Laurence E. Lynn Jr. 2009. “The Implementation of Public Policy: Still the
Missing Link.” Policy Studies Journal 37(1):21–36.
Romzek, Barbara S., and Jocelyn M. Johnston. 2002. “Effective Contract Implementation and
Management: A Preliminary Model.” Journal of Public Administration Research and Theory
12(3):423–53.
Rothstein, B. 1998. Just institutions matter: the moral and political logic of the universal welfare state.
Sabatier, Paul a. 1988. “An advocacy coalition framework of policy change and the role of policyoriented learning therein.” Policy Sciences 21(2-3):129–68.
Sabatier, Paul A., and Hank C. Jenkins-Smith. 1993. Policy Change and Learning: An Advocacy Coalition
Approach. edited by Paul A Sabatier and Hank C Jenkins-Smith. Boulder: Westview Press.
Sabatier, Paul, and Daniel Mazmanian. 1980. “The Implementation of Public Policy: A Framework of
Analysis.” Policy Studies Journal 8(4):538–60.
Saetren, Harald. 2005. “Facts and Myths about Research on Public Policy Implementation: Out-ofFashion, Allegedly Dead, But Still Very Much Alive and Relevant.” Policy Studies Journal 33(4):559–
82.
Salamon, Lester, ed. 2002. The Tools of Government: A Guide to the New Governance. Oxford, United
Kingdom: Oxford University Press.
27 | P a g e
Sandfort, Jodi R. 2003. “Exploring the Structuration of Technology within Human Service Organizations.”
Administration & Society 34(6):605–31.
Sandfort, Jodi R. 2010. “Nonprofits within Policy Fields.” Journal of Policy Analysis & Management
29(3):637–44.
Sandfort, Jodi, Sally Coleman Selden, and Jessica Sowa. 2008. “Do the Tools Used by Government
Influence Organizational Performance?: An Examination of Early Childhood Education Policy
Implementation.” American Review of Public Administration 38(4):412–38.
Schneider, Anne, and Helen Ingram. 1990. “Behavioral Assumptions of Policy Tools.” The Journal of
Politics 52(2):510–29.
Selznick, Phillip. 1949. TVA and the grass roots: A study of politics and organization. Berkeley, California:
University of California Press.
Spillane, James P. 1998. “A Cognitive Perspective on the Role of the Local Educational Agency in
Implementing Instructional Policy: Accounting for Local Variability.” Educational Administration
Quarterly 34(1):31–57.
Stone, Melissa, and Jodi R. Sandfort. 2009. “Building a Policy Fields Framework to Inform Research in
Nonprofit Organizations.” Nonprofit and Voluntary Sector Quarterly 38(6):1054–75.
Tabak, Rachel G., Elaine C. Khoong, David a Chambers, and Ross C. Brownson. 2012. “Bridging research
and practice: models for dissemination and implementation research.” American journal of
preventive medicine 43(3):337–50.
Twombly, Eric, and Elizabeth Boris. 1999. “The Use of Vouchers in the Provision of Human Services and
the Potential Implications for Nonprofit Human Service Organizations.” Paper prep.
VanMeter, DS, and CE VanHorn. 1975. “The policy implementation process a conceptual framework.”
Administration & Society 6(4):445–88.
Watkins-Hayes, Celeste. 2009. The New Welfare Bureaucrats. Chicago, Illinois: University of Chicago.
Weible, Christopher M., Paul a. Sabatier, and Kelly McQueen. 2009. “Themes and Variations: Taking
Stock of the Advocacy Coalition Framework.” Policy Studies Journal 37(1):121–40.
Wilson, James Q. 1991. Bureaucracy: What Government Agencies Do and Why They Do It. New York:
Basic Books.
28 | P a g e