5,117
Using program logic model analysis to
evaluate & better deliver what works
Lori Megdal
Megdal & Associates
198 High Street
Acton, MA 01720-4218
USA
megdal@bellatlantic.net
Victoria Engle and Larry Pakenas with the New York State Energy Research and Development Authority
Scott Albert with GDS Associates, Inc.
Dr. Jane Peters with Research Into Acton, Inc.
Dr. Gretchen Jordan with Sandia National Laboratory
Keywords
Evaluation, logic models, program theory, market transformation, program redesign
Abstract
There is a rich history in using program theories and logic
models (PT/LM) for evaluation, monitoring, and program
refinement in a variety of fields, such as health care, social
and education programs. The use of these tools to evaluate
and improve energy efficiency programs has been growing
over the last 5-7 years. This paper provides an overview of
the state-of-the-art methods of logic model development,
with analysis that significantly contributed to:
1. Assessing the logic behind how the program expects to
be able to meets its ultimate goals, including the “who”,
the “how”, and through what mechanism. In doing so,
gaps and questions that still need to be addressed can be
identified.
2. Identifying and prioritize the indicators that should be
measured to evaluate the program and program theory.
3. Determining key researchable questions that need to be
answered by evaluation/research, to assess whether the
mechanism assumed to cause the changes in actions,
attitudes, behaviours, and business practices is workable
and efficient. Also will assess the validity in the program
logic and the likelihood that the program can accomplish
its ultimate goals.
4. Incorporating analysis of prior like programs and social
science theories in a framework to identify opportunities
for potential program refinements.
The paper provides an overview of the tools, techniques and
references, and uses as example the energy efficiency program analysis conducted for the New York State Energy Research and Development Authority’s (NYSERDA) New
York ENERGY $MARTSM programs.
Introduction and Background
There is a rich history in using program theories and logic
models (PT/LM) for evaluation, monitoring, and program
refinement in a variety of fields, such as health care, social
and education programs. Using program theory to drive
evaluation was promoted by Carol Weiss beginning in 1972
(Worthen, 1997, page 221). Program theory is a theory or
model that describes the underlying assumptions about how
a program is expected to work; how the program causes the
intended or observed outcomes. A logic model is a diagram
that describes the key logical (causal) relationships among
program elements and the problem to be solved, thus defining measurements of success. It helps portray the program
theory. The logic model can be used to help tell ”the story”
behind how the program expects to be able to meets its ultimate goals, including the “who”, the “how”, and through
what mechanism. In doing so, gaps and questions that still
need to be addressed can be identified.
The elements of the logic model describe and place the
causal sequence of program activities, outputs, immediate
outcomes, and longer-term outcomes. Often the logic model
is displayed with these elements in boxes and the causal
flow being shown by arrows. Indicators can be derived to
measure each of these elements and their tracking can be
used to assess program success.
ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS?
1045
5,117 MEGDAL ET AL
PANEL 5. EVALUATION AND MONITORING
Successful
Program
Program
set in
motion
Causal
process
which
led to
Desired
effect
Theory
Failure
Program
set in
motion
Causal
process
did not
lead to
Desired
effect
Program
Failure
Program
did not set
in motion
Causal
process
which would
have led to
Desired
effect
Figure 1. Theory Failure & Program Implementation Failure.
An evaluation design and program theory can work handin-hand to be able to have the evaluation differentiate between theory failure (incomplete or inaccurate theory), and
program failure (poorly designed or implemented operational procedures). In case of theory failure the assumptions
about the market or the causal mechanisms that create attitudes or behaviours are not valid, are only partially valid, or
key theoretical components are missing. The program implements as planned but the causal link to final outcome
does not occur. The program theory needs to change and the
program needs to be refined accordingly.
On the other hand, if the evaluation identifies program
failure, this means that the theory appears to be correct.
However, the program implementation had problems that
did not allow it to have the anticipated outputs and initial
subsequent outcomes. In other words, if the causal process
had occurred the outcome might have occurred but program
implementation problems did not create this process. Figure 1 shows the differences between theory failure and program failure1.
The use of these tools to evaluate and improve energy efficiency programs has been growing over the last 5-7 years.
In the past five years, it has become increasingly clear that
making a program’s theory and logic explicit is important for
effective program implementation and evaluation, especially when the program includes or is dominated by market
transformation goals and objectives (Erickson, Fagan &
Block 2003; Goldstone, Rufo & Wilson 2000). The recent
comprehensive energy efficiency evaluation reference manual, The California Evaluation Framework, states that “An important component of the evaluation effort is to draw upon
the program theory and logic model, to include its review (or
development if one is not available) and use as an evaluation
planning tool” (TecMarket Works (2003), page 30). It also
says that while important for all types of program evaluations it is especially important for complex programs and
programs with long-term market change goals. This document goes on to provide more in-depth descriptions, references, basics of how to develop PT/LM with an example in
an appendix. McLaughlin and Jordan (2004) write that Rogers et. al. (2000) and Birkmayer and Weiss (2000) present examples of theory-driven evaluations, but report that while
theory-driven evaluation is conceptually sound, it is rare to
find good examples in practice.
For all of these reasons, the New York Research and Development Authority (NYSERDA) contracted with an eval-
1. Weiss (1998) page 129.
1046
ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS?
uation consultant team in 2003 to conduct program analysis
with program theories and logic models, and developing
these where need be. The authors were part of this team
lead by GDS Associates. The task was to develop logic models for programs and to analyze each program relative to social science theory and other program experience. The
program analysis activities cover most of the over 30 residential, non-residential, research and development, and low-income programs in the New York Energy $martSM portfolio.
Over the 2003 and 2004 period, program analysis with PT/
LM was conducted for four residential, one low-income,
four research and development, and four non-residential
programs.
The program analysis approach has included two primary
stages: logic model development and program analysis. The
logic model development stage involved four steps by the
team: initial data collection, problem description, logic model definition, and logic model diagram construction. At the
end of the first stage, the program analysis team reviewed
the diagram with the program staff to refine the model and
assure that it articulated the program design and implementation.
The second stage involved three steps: relevant social science and business theory research, logic model assessment
against the social science and business theories, and recommendations for program refinement and further research.
The final activity in the comparative assessment with social
science theories involved exploring the logic model and program analysis findings with program staff to assure that the
findings articulated program design and implementation.
The last step on recommendations also included meeting
with staff to inform them of any findings, new information or
recommendations that arose from the program analysis activities.
Developing the Logic Model to Portray the
Program Theory
Based on current industry best practices, program logics include the following elements:
• Key program resources/inputs (program funding, internal
and contractor staffing, sources and magnitudes of leveraged funding/partnerships, etc.);
PANEL 5. EVALUATION AND MONITORING
5,117 MEGDAL ET AL
• Activities (internal and contractor program implementa-
is to start with the long-term outcomes and work backwards
to resources. Essentially, the process is one of repeatedly
asking the same question, if “Z” is a long-term outcome (or
short-term outcome, output, or activity), what is required to
produce “Z.” It is then a matter of writing the causal relation
in the form of a statement: “Y” will cause “Z.” One then
backs up and asks what will cause Y and continues until one
has described the required activities and resources. One can
then reverse the order and edit the statements until one has
a sequence of causal statements that describe how the program works.
Several things are likely to happen as this is being worked
upon. It is likely that gaps in the causal relationships between actions and expected effects will be found. Some steps
will be identified that require substantial leaps that suggest
that the theory needs further refinement. Some steps in the
theory will seem quite improbable, suggesting that the theory, and probably the program design, needs improvement.
Some steps will contradict what is known from the program,
marketing, and evaluation literature and other social science
and business theories. Sometimes an assumption, key to the
way the program is designed for one of its causal chains, can
be in conflict with an assumption in a different causal chain.
All of these instances identify places where the program theory and the program design may need improvement.
tion tasks, outreach/marketing and delivery mechanisms,
etc.);
• Customers and partners (who the program works for and
with – customers receive products and services directly
from the program and its partners, and change behaviour
or take action that translates into program outcomes);
• Outputs (internal and implementation contractor servic-
es, products, training/support being provided to target
customers or market actors, etc.);
• Outcomes (short, intermediate, and longer-term antici-
pated results/benefits/market changes from program activities – many of which come directly from the program’s
stated measurement indicators and appropriate/targeted
portfolio-level goals and objectives), including how these
contribute to overarching policy goals;
• Any perceived external influences (recognizing the influ-
ence that market actors, barriers, other New York Energy
$martSM programs, state, regional and national activities
or circumstances, etc., may have on a program’s logic);
and
• Drawn from the logic, measurable indicators and explicit,
researchable issues.
The basic logic model format is shown in Figure 2.
The steps to develop the logic model are as follows:
An Example Logic Model from NYSERDA’s
Programs
1. Collect information through documents and perhaps
establish a stakeholder workgroup.
The team collected all possible documentation describing
the program—its goals and objectives, its marketing plan,
and reports describing the program from different perspectives. Using these secondary data sources and the pre-existing logic model, we drafted a new graphic logic model and
prepared a summary of the secondary data in the form of a
description of the logic model.
The logic model and narrative were provided to the program staff for their review and comment; a two-hour conference was then held to share staff observations with the team.
In many ways, the development of the logic model was the
most straightforward of all the activities. The program staff
was able to review the narrative and determine where inevitable errors of fact had occurred. They also reviewed the
logic model and made recommendations regarding the diagram and how the program logic worked from their point of
view. The team then revised the model to reflect this input.
Figure 3 shows the final logic model for NYSERDA’s Keep
Cool Campaign and program to transform the market for
room air-conditioners (RAC) in New York.
2. Define the problem and context for the program.
3. Define elements of the logic in a table.
4. Develop a diagram of logical relationships.
5. Verify the program theory/logic with stakeholders, comparisons with implementation results
6. Then use the logic model to develop or confirm performance measures for program monitoring and performance contracts, and in planning and evaluation.
Often developing the logic model from gathered data can
begin from organizing the information into tables, or taking
it step-by-step from activity to output to outcomes.
Once a program has been systematically described in
terms of resources, outputs, outcomes, and long-term impacts, the procedure for developing a program theory is a
systematic one. Many PT/LM leaders and teachers have
found that one of the best ways to develop a program theory
Resources
(inputs)
Activities
Outputs
for
Customers
Reached
Short-term
Outcomes
Intermediate
Outcomes
(through
customers)
Long-Term
Outcomes
& Problem
Solution
External Influences and Related Programs (mediating factors)
Figure 2. Basic Logic Model Format.
ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS?
1047
5,117 MEGDAL ET AL
PANEL 5. EVALUATION AND MONITORING
Some of the key issues that emerged from the development of the Keep Cool logic model were that the program
had both resource acquisition and market transformation
strategies and associated tactics. Like many resource acquisition programs Keep Cool sought aggressively to achieve
kWh savings and kW peak reductions relatively quickly by
working within the market structure with significant incentives, so there are strong market elements in its resource acquisition strategy. At the same time, the program logic
included a strategy and associated tactics that, as the program staff anticipated, would lead to a long-term sustainable
transformation for RACs by encouraging significant changes
in stocking and consumer awareness.
INDICATORS DEVELOPED FOR LOGIC MODEL ELEMENTS
The next major step was to develop indicators for the output, and outcome elements of the program logic that could
be used for evaluation research design. Operationalizing
these indicators and then prioritizing them helps to define
the evaluation research goals needed to support tracking the
program effects. Several of the important indicators are
shown in Table 1.
RESEARCHABLE QUESTIONS
Each arrow in the logic model presents a researchable question in that it presupposes a causal mechanism. Each arrow
also implies the need for research on whether the action is
resulting from the prior action and how it occurs. This can be
part of the implementation theory, what is required to translate an activity into an output (the delivery of program services) or translating one outcome into another (the social,
psychological, and/or market processes that cause the
change to occur). Each item mentioned could be investigated as part of the evaluation.
Being able to identify the critical areas to be included in
the evaluation research design, apart from the many potential questions embedded within the causal linkages, is an
important skill to be honed. In a market transformation program, one way to identify key linkages is to assess which
boxes require behaviour change in market actors that are essential to the process. Also look for boxes where their outgoing arrows are expected to lead to several items in the causal
chain. Additionally, if there is only one mechanism that is to
change behaviour in a key market actor then that mechanism is critical and should be evaluated.
A few of the researchable questions derived from analysis
of the Keep Cool Program logic model provided above include the following:
• How well are retail staff trained?
• Do knowledgeable retail staff promote ENERGY STAR
products?
• How important is this/how great is the effect? (Are pro-
gram resources optimally targeted?)
• How well can the advertising messages target folks buy-
ing RACs to buy ENERGY STAR? Is there a spillover
where more people buy RACs then otherwise would? If
so, how much?
• How much recycling of RACs occurs without interven-
tion, post-bounty effort?
• Load-shifting from advertising has been measured. How
much is maintained behavior versus how much reinforcement messaging is needed?
One of the critical researchable questions, developed as
working through the Keep Cool Program logic model, was
derived from one of the design premises for the original
Keep Cool Program. This premise is that the replacement
(turn in) of old, operating AC units with new more efficient
units, will reduce the overall energy usage (especially during
summer peak periods in New York). It is implied within the
logic that the program’s recycling efforts will lead to fewer
RACs in the secondary market. The Keep Cool Program
thereby reduces energy and demand usage as more, new
RACs are purchased at higher efficiency levels than the efficiency levels in the secondary market. Without turn-in of
the old units, these less efficient RACs will likely find their
way into other rooms in the same house, in use at family or
friends homes, or for sale in the secondary market, thus increasing kWh usage and summer peak demands. It will be
important therefore, to confirm that the program’s recycling
efforts are in fact reducing the number of RACs in the secondary market. If it is found that, as a result of program ad-
Table 1. Some of the Indicators for the Keep Cool Program.
Program Outputs
Number of air conditioners
surrendered
Number and dollar value of
bounties (rebates for turn-in
of old working RACs)
Number of units
demanufactured and
amount of material diverted
from the waste stream
Number of ads placed,
impressions, and ad value
Number of retailers active
in the program
1048
Short-term Outcomes
Intermediate-term Outcomes Long-term Outcomes
Change in awareness of Perceived benefits of
NYSERDA program and ENERGY STAR® product
ENERGY STAR®
purchases
In conjunction with other
ENERGY STAR® efforts:
- Eliminated barriers
Degree of subsequent
- Reduced waste by
ENERGY STAR® product
recycling of old units
purchases given past ENERGY
STAR® experience
- Increasing market share
Knowledge and ability of
and penetration
Frequency and content of
retail staff to promote
- Sustained change in
ENERGY STAR® RACs communication to others
and efficient TTW units
market behavior
Retailers indicate that
Immediate Peak
ENERGY STAR® RACs and - Persistent energy savings
Reduction and KW and efficient TTW units are
- Emissions reductions
KWh savings
profitable to stock & sell
Effectiveness of TV
advertising versus other
advertising venues
ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS?
Activities
Develop marketing and
advertising strategies for
consumers, POP display,
educational and website
information for consumer
awareness campaign
Consensus
building
with NYPA
& LIPA
Outputs
Coordination with
ENERGY STAR
Products Program
Events, radio/TV ads,
newspaper ads, collateral,
event direct contact,
website hits
Recruit ENERGY STAR
(ES) retailers for
participation and training
Develop
capability for
demanufacturing
Develop availability of
facilities for turn-in
Participating retailers trained in
program processes
Install point of
purchase (POP)
displays
Process applications
Provide Bounty
Increased Awareness &
Knowledge of Keep
Cool Program
Short Term
Outcomes
1-3 Years
Develop bounty
application process
Increased ability of sales
personnel and stores to
promote the program
Sales of ES RAC and
TTW to bounty users
Old units turned in &
demanufactured, being
replaced by ES RAC or
high efficiency TTW
R ecognize benefits from selling up ES
product & word-of-mouth
Shift on-peak
to off-peak
usage
Recognize benefits from purchase of ES product
& increase likelihood of future ES purchase &
word-of-mouth
Jobs created through
continued
demanufacture and
recycling of AC units
Increased Awareness &
Knowledge Energy Star
Appliances (including
RACs)
Recycling centers
process without
bounty
kW and kWh savings
Intermediate and
Long Term
Outcomes
4-8 years (contributing to
MT along with other
residential ENERGY STAR
programs)
Long-term environmental
benefits
Increased demand of ES products based upon
awareness, knowledge and recognition of benefits.
Supplier provide greater ES RAC and TTW availability (without need for
incentives). Provide advertising, POP, and sales training.
Leads to less resistance to and support for increased codes and standards.
Legend
1049
Figure 3. Keep Cool Program Logic Model.
=
Program Activities
=
Program logic flow
=
Outputs and
outcomes
=
Critical logic flow to
meet long-term goals
=
Key logic elements
=
External influences
5,117 MEGDAL ET AL
External Influences: Weather, Codes and Standards; Positive - National ENERGY
STAR program, NYSERDA Home Performance with ES, NY Blackout, CA energy
crisis; Negative - Recession, Other competing attention draws (i.e., 9/11, War in Iraq)
and investments
Using Relevant Social Science Theory and
Prior Evaluations for Analyzing the Program
and It’s Logic
Outside of program sales of ES
Appliances (including RAC and
TTW)
This second stage of the process was exciting and innovative
territory to the team members. All had extensive experience
reading and pondering about social science theories and
ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS?
Peak Reduction
Increased stocking of
ES RAC and TTW
units
PANEL 5. EVALUATION AND MONITORING
1-29-2004
vertising, more air conditioners (albeit ENERGY STAR®
units) are being purchased than otherwise would have occurred. Then anticipated energy and peak period savings
benefits may be impacted and be less than originally anticipated by the program.
Keep Cool Program High Level Logic Diagram
Inputs: Funds, staff, allies,
market knowledge, synergistic
program management
5,117 MEGDAL ET AL
PANEL 5. EVALUATION AND MONITORING
COMMUNICATION CHANNELS
I. Knowledge
II. Persuasion
III. Decision
IV. Implementation
1. Adoption
V. Confirmation
Continued Adoption
Later Adoption
Discontinuance
2. Rejection
Continued Rejection
Figure 4. Innovation-Decision Process.
how they might apply to energy efficiency programs. However, this was the first time that team members were aware
of an energy efficiency client requesting a program logic
model be reviewed in the context of social science theory
and then tested against those theories within the experience
of an energy efficiency program. Hence, the team focus on
program analysis was a clear indication that a logic model, in
and of itself, was insufficient to meet the client’s needs and
interests. The most important issue was to test the model in
the context of social theory.
The first steps in the program analysis were to identify
what theories might be relevant to the program logic, what
anticipated outcomes should be explored through a review
of similar programs and verify the logic model relative to
those theories. The theories we identified to be of interest
were social marketing theory, diffusion of innovation, and
consumer economics. The next phase involved presenting
the program staff with the team’s findings and recommendations and then discussing their implications for the logic
model and the program itself.
Some of the program analysis findings are included below
as examples of this type of effort.
MARKETING, SOCIAL MARKETING, AND CONSUMER
SHOPPING
The PT/LM for Keep Cool is looking to change purchasing
behaviour of owners of old RACs. It is also looking to change
the behaviour of retailer stock and promote ENERGY
STAR RACs. This drew the social science/business research
to include marketing, social marketing, and consumer shopping.
There are many behaviour change models theorized. In
marketing, and most market transformation programs on energy efficiency, the most cited is the Rogers’ and Shoemaker’s innovation diffusion model (Rogers and Shoemaker,
1972). This field of work involves a few variations on the
theme of an awareness-adoption model. This model states
that the process for adopting an innovation (or buying a different product) moves through stages of awareness, knowledge, persuasion, decision, implementation, and confirmation, as displayed in Rogers’ diagram shown in Figure 4
(Rogers 1995, page 163). Evident from this is the importance of communication flows and interactions between
market participants (communication channels) in order to
move from one stage of adoption/diffusion to the next and to
do so with positive adoption, confirmation and continued
adoption. This is where education, advertising, marketing,
and selling influence the adoption process.
Marketing of the Keep Cool Program and the benefits of
ENERGY STAR qualifying air conditioners is a major component of the Keep Cool Program. One of the primary goals
of the market transformation programs is to move forward in
time the adoption of more efficient technology. The efficiency gain is then caused by the difference in the adoption
rate of the more efficient appliance, given program intervention, versus what the adoption would have been without the
Figure 5. Market Transformation Gains from Shifting S-Curve of Technology Adoption.
1050
ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS?
PANEL 5. EVALUATION AND MONITORING
program. The typical S-curve for technology adoption with
and without the program display the gain made with the intervention. This is shown in Figure 5.
Much of the marketing to purchase ENERGY STAR
room air-conditioners (RAC) focuses upon the energy and
utility bill savings that can be obtained by purchasing more
efficient RACs. This is a relatively straightforward economic
message to consumers.
A large part of the marketing campaign focuses on the
broader societal benefits of shifting energy use to off-peak
times to reduce load during peak periods throughout the
summer. Both the decision to purchase energy efficiency
and the shifting of clothes washing and dishwashing to offpeak periods used special marketing messages that fall
squarely in the realm of social marketing theory. Social marketing is the application of marketing theory to pro-social
activities—those activities that have a societal rather than a
private benefit (Andreasen 1995). Social marketing has been
directly applied to many products over the years: reduction
of litter, recycling, the Smokey Bear campaign, condom use
in countries plagued by HIV infection, and recently for international energy efficiency campaigns such as the Efficient Lighting Initiative (Vine et al. 2003). Indirectly, many
marketing campaigns for energy efficiency products and
services in the United States are also social marketing campaigns.
The types of promotions needed in these situations are
much more involved than marketing of more traditional consumer goods where little thought is required on the part of
the consumer. Rather, these decisions are defined in the literature as “high involvement” as they require greater effort
on the part of the customer and not all of the benefits of
these decisions are directly seen by the purchaser (i.e., the
benefits of reducing load during peak periods benefits all
utility customers.) As a consequence, the decision to purchase an ENERGY STAR room air conditioner will have
several stages of information gathering, contemplation, and
action. Awareness of the program and the benefits of
ENERGY STAR brands may not be enough; the customer
may spend time during a “contemplation” stage to carefully
assess the pros and cons of purchasing an energy efficient
model.
According to social marketing theory, the shift to purchase
high efficiency room air conditioners that are covered by the
Keep Cool Program will involve a change in values. This
change in values best occurs when the candidate understands that the change in behaviour (purchasing a more expensive ENERGY STAR model, moving their clothes and
dish washing to off-peak) is acceptable to others in their
peer group. Once this decision is made, the candidate also
needs to be convinced that the purchase will be beneficial
on a personal level. Given this decision structure, models for
social marketing provide useful maps of the decision processes as well as effective marketing techniques in these situations. A social marketing message is one that provides
information on how their action will affect societal attributes
that consumers will value, such as environmental quality.
For example, “If X number of people bought ENERGY
STAR rather than non-ENERGY STAR, then we would
take X lbs of CO2 out of the air”.
5,117 MEGDAL ET AL
The contemplation stage view of social marketing could
be used alongside the theory of social proof. The principle
of social proof is used in much advertising and operates in
many non-advertising ways. (Non-advertising examples of
the principle of social proof include copy cat suicide and
murder, and where strangers follow another’s lead on whether to ignore someone that might be injured or is asking for
donations, etc.) This is, that “we view a behaviour as correct
in a given situation to the degree that we see others performing it” (Influence: Science and Practice, 2001, pp. 100). In the
face of uncertainty and with others around, the principle of
social proof is strongest. Perhaps the point-of-purchase
(POP) and sales techniques can use this to sell ENERGY
STAR RACs where uncertainty is high, such as a customer
that has come to make an “emergency” replacement for a
broken RAC. These customers may not have gone through
the contemplation stage (unless they have done so for another ENERGY STAR product, a good reason for the branding and program coordination). They are also not eligible for
the bounty program (a consumer incentive which requires
both purchase of an ENERGY STAR RAC and turn-in of a
working old RAC). At the same time, they are still a good opportunity for the retailer to sell up to an ENERGY STAR
product. One method often used in this type of marketing is
advertising how many people have bought the product. This
program has had extremely large growth, from 17 000 in
2001 to over 233 000 in the first three years. This large
amount might be used as a sales tool to influence the purchasing decisions of future RAC consumers. The larger participating stores might be able to advertise how many
people have purchased ENERGY STAR RACs at their store
or ENERGY STAR products at their store. This type of
POP advertising would capitalize on social proof effects.
An example that was provided in one of the theory literature documents we reviewed in this project might provide
an interesting tidbit for store sales of RACs. In Why We Buy:
The Science of Shopping, Paco Underhill has a chapter on the
fact that many consumers buy as a sensual shopper: touch,
smell, feel of a product. He discusses that this can be difficult for some products and explicitly uses air conditioners as
an example. He points out that you can’t tell how an air conditioner cools by turning one on in the store, especially a
cool store. Customers can check with friends, Consumer Reports, or the salesclerk’s opinion. But at the store the thing
that could make a difference in choice of air-conditioner
would be “how does it sound?” “In the final analysis, it’s one
of the few things that distinguish one air conditioner from
another. The unit is going to be humming (or clattering)
away in your house for a number of years, after all. In a typical summer, I’ll bet I have three or four conversations about
air conditioner decibel levels. That’s what actual human beings care about when it comes to air conditioners, but you’d
never know it when you’re shopping for one.”
Selling retailers on using ENERGY STAR to distinguish
one air conditioner from another is a key benefit that can encourage a consumer to conclude their purchase. Marketing
should cover the fact that ENERGY STAR products offer all
the major benefits of non-ENERGY STAR plus added energy and money savings. Yet, drawing upon the work of Paco
Underhill marketing images that present ENERGY STAR
ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS?
1051
5,117 MEGDAL ET AL
as “environmentally friendly, but in a quiet cool way” might
be worth considering and testing.
FURTHER DIFFUSION OF INNOVATION EFFECTS
LEVERAGED
The primary goal of the program is to gain adoption of
ENERGY STAR RACs in place of older RACs, the older
technology. Diffusion of innovation theory was therefore
also closely examined relative to the program logic. Most
market transformation program planners and implementers
are familiar with the basic notion of diffusion promulgated
by Rogers (1983, 1995). This is where innovators become
the first to adopt a technology, followed by early adopters,
then the early majority, the middle, and finally the late majority and laggards. The late majority and laggards are rarely
of interest to market transformation programs, as these people are those who adopt after effective market transformation has occurred. Innovators are those who adopt even
without program efforts, though they often take advantage
of programs when they emerge and likely becoming freeriders. The focus of most market transformation programs
has been on early adopters and early majority adopters.
Consumer preferences for air conditioners are determined
in part by the choice of models available in addition to available rebates and incentives, which includes the secondary
market of used air conditioning units. The program’s rebate
effort requires the turn-in of old working RACs along with
the receipt of purchasing a new ENERGY STAR RAC. Often these old RACs will either be used in other rooms of the
house or sold on the secondary market. The turn-in and environmentally friendly disposal of these older units reduces
the number of older inefficient units in the secondary market. By helping reduce the option for choosing a used (and
less efficient) unit, the program can increase the market
share of ENERGY STAR models. This shift in the supply
of air conditioners is also consistent with diffusion of innovation theory. The stage of the market that is demanding used
air conditioning units likely falls in the “late majority” and
“laggard” portions of the innovation curve. That is, this segment of the market contains those customers that adopt
“new” technologies only after the equipment is well established in the market and considered more or less standard
(and thus has a relatively low price compared with other
available options.) The program’s pitch to properly dispose
of older RACs reduces the secondary market and can cause
a smaller and more efficient pool of RACs in this market. As
the secondary market shrinks causing prices to rise, there is
less incentive not to buy a more efficient new RAC (and all
new RACs are significantly more efficient than old ones given changes in technology and appliance standards).
BUSINESS ECONOMICS
Perhaps the most critical component of the Keep Cool Program is the recruitment of enough retailers to participate in
the program to have a significant influence on the market.
Once recruited, retailers receive sales training and other promotional support (advertising, POP materials, scanners to
facilitate intake of units) for ENERGY STAR air-conditioning units.
In order for recruitment to be successful, participation
must be economically beneficial for the retailers. That is,
1052
ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS?
PANEL 5. EVALUATION AND MONITORING
the support and tools provided by the program must be effective enough ultimately to increase profits. Although the
ENERGY STAR models are typically more expensive to
produce than standard efficiency models, the profit margin
also tends to be higher. Consequently, retailers may be able
to increase profits even if they sell fewer units.
It may be tempting to apply social marketing techniques
(discussed below) to appeal to retailers for participation.
Given that the non-monetary benefits to retailers from participation are small, these techniques are unlikely to be successful. That is, appealing only to the broader benefits of the
ENERGY STAR models (reduced bills for customers, peak
load reduction, and public relations benefit of retailer being
perceived as “green”) are unlikely to be effective as retailers
receive few of these benefits directly and must bear a disproportionate share of the costs. For this reason, recruitment
should focus on the potential for increased sales and the effectiveness of program support in terms of training and promotional materials in helping reach the increased sales
potential.
At the same time, providing evidence that offering
ENERGY STAR products and promoting environmentally
friendly products can help increase sales may be important.
“Selling” the product and concept to the retailers is different than providing them with alternative selling messages to
sell to consumers. It is here that the environmental messages can be useful for that niche of consumers that value environmentally friendly products.
The program, however, also indirectly provides another
benefit to retailers. The program strongly advocates disposing of working older inefficient units and replacing them
with new ENERGY STAR units. This is a definite message
that promotes greater purchases of ENERGY STAR RACs
(and other RAC purchases as well). This should have a positive effect on retailers and their perception of the benefits
of the program overall.
Diffusion is largely a communications issue in which information is transferred between individuals. When behaviour change is sought for pro-social reasons, such as adoption
of a new agricultural technology or new healthcare practice,
a change agent generally initiates interest in the new behaviour. Diffusion theory comments on both the characteristics
of the change agent, and the characteristics of those market
actors that the change agent targets for their outreach.
Research by Rogers (1995) suggested that change agents
needed to be technically knowledgeable, such that the target group respected them; however, those who had a commercial interest in the behaviour change might be less
accepted by the target group. Given this, retailers are not
true or perfectly designed change agents. Yet, to the extent
that sales staff do education, and promote the ENERGY
STAR RACs due to the retailer training and the benefits retailers perceive in supporting the program, the sales staff
may be able to leverage the consumer social marketing and
act as surrogate change agents from the diffusion of innovation perspective.
PANEL 5. EVALUATION AND MONITORING
Summary of the Advantages of this Approach
and Lessons Learned
Developing theory and logic models for programs, prior to
their implementation in the field, can be the most effective
way to influence program design and can allow the program
to build cost effective evaluation and performance monitoring into the program plan. Using these tools within program
design can help the program ensure that it is well designed
to meet both its short term and long term goals. Expecting
specific market responses to interventions that change the
way a market operates can be difficult. Being explicit about
all the steps required to make this happen can help provide
insight into whether the program logic appears to lead to the
ultimate goals without any “black boxes” where the creation
of the desired outcome is unknown and, therefore, less likely to occur.
Unfortunately, explicit logic modelling can be a low priority to some program designers and implementers. Though
these individuals had given a great deal of thought to the design of the programs studied and the theory behind the program, they did not have the time to invest in the process of
explicitly modelling the project or conducting the program
analysis steps. Designers and implementers are often faced
with political pressure to have programs fielded as quickly as
possible after funding is authorized. This normally does not
allow them to spend the time and resources for using these
more thorough program design tools. The mindset, backgrounds, and skills for those best suited to getting a program
into the field, making weekly and daily decisions about implementation and keeping the program progressing toward
its annual goals and objectives, are different from those that
best work in this almost abstract world. At the same time,
those best suited to this type of work are also probable less
likely to shine in program implementation.
It is necessary, though we often found some difficulty, to
obtain time and significant input from program staff. An important lesson learned is that logic modelling and program
analysis require not only technical skills and knowledge, but
also good listening and facilitation skills. Because logic modelling may be a low priority for program staff, it is necessary
for the logic modelling team to be able to facilitate discussions about the program, the program analysis and the logic
model, and to be able to take the time to listen to the response from program staff. It is not necessary for staff to articulate the model and the theories. But it is important to
recognize that they developed the program theory and thus,
if they do not buy into the logic model and analysis, any
work that flows out of the modelling process, such as evaluation research and reports, risks limited acceptance. In the
end, the involvement of all key program staff in the discussions, especially the program manager who makes final decisions, led to the program analysis being better understood
and the logic model being more consistent with the program. It would appear a cooperative team approach might
offer the best of both worlds.
The addition of examining evaluations of similar programs, and looking at social science theories, provided important perspectives for more in-depth analysis of the
program. The program analysis team was able to go deeper
into the program design logic and assess whether it was con-
5,117 MEGDAL ET AL
sistent with social science theory and research. Often this
provided information that might be able to be used for program refinements and lead to evaluation questions that can
help program implementers learn how to better facilitate the
causal mechanisms they are hoping to influence. These revelations in the logic modelling and program analysis process
have begun to provide the program staff with fodder for
thinking about opportunities to further refine the program.
Being able to identify the critical areas to be included in
the evaluation research design, apart from the many potential questions embedded within the causal linkages, is an
important skill to be honed. In a market transformation program, one way to identify key linkages is to assess which
boxes require behaviour change in market actors that are essential to the process. Also look for boxes where their outgoing arrows are expected to lead to several items in the causal
chain. Additionally, if there is only one mechanism that is to
change behaviour in a key market actor then that mechanism is critical and should be considered for evaluation efforts.
Evaluations conducted after these assessments adopted
many of the indicators for measurement and follow-up in future evaluations. The program performed well on these indicators. Enough progress was made in retailer stocking and
promotion that the program dropped the rebate component
and only focused on marketing. Indicators from the evaluation suggested that any increased sales of RACs due to
ENERGY STAR RAC promotion was minimal and the effect was quantified and included in total energy savings estimates. Finally, the program analysis with the social science
theories has led to changes in the advertising messages being tested in the current program.
Overall, NYSERDA’s assessment of the advantages and
reasons for developing program theories, logic models, and
conducting the in-depth program analysis discussed here
can be summarized as this tool’s and process’ ability to help :
• Promote critical thinking about programs
• Identify key indicators and researchable issues for evalu-
ation research design
• Map linkages (and disconnects) between activities, out-
puts, and outcomes
• Provide a basis for program changes or status quo
• Tell the program story in a short, precise format
• Management/stakeholder/policy maker comfort
• Best done during program planning and design but post-
implementation analysis can help provide a reality check
• Identify impacts related to changes in program design
ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS?
1053
5,117 MEGDAL ET AL
References
Albert, Scott, Victoria Engel, Gretchen Jordan, Lori Megdal,
and Jane Peters. 2004. “Using Program Theory And
Logic To Improve Design and Likelihood of Real Market Change- Experience With A State Public Benefits
Program,” Proceedings of the 2004 ACEEE Summer Study
on Buildings, 6.1 – 6.12.
Andreasen, Alan R. 1995. Marketing Social Change: Changing
Behavior to Promote Health, Social Development, and the
Environment. San Francisco, CA: Jossey-Bass.
Birkmayer, J. D. and Weiss, C. H. 2000. “Theory-Based
Evaluation in Practice: What do we learn?” Evaluation
Review., Vol.24, # 4, pp 407-431.
Cialdini, Robert B. 2001. Influence: Science and Practice. 4th
Edition. Boston, MA: Allyn and Bacon.
Erickson, Jeff, Jennifer Fagan, and Oscar Bloch. 2003. “The
Program Theory and Metrics Process.” Proceedings of the
2003 International Energy Program Evaluation Conference,
517-30. Madison, Wisc.: National Energy Program Evaluation Conference, Inc.
Goldstone, Sy, Michael Rufo, and John Wilson. 2000. “Applying a Theory-Based Approach to California’s Nonresidential Standard Performance Contract Program:
Lessons Learned.” Proceedings of the ACEEE 2000 Summer Study on Energy Efficiency in Buildings, 5:103-5.117.
Washington, D.C.: American Council for an Energy-Efficient Economy.
McLaughlin, John A., and Gretchen B. Jordan. 1999. “Logic
Models: A Tool for Telling Your Performance Story.”
Evaluation and Program Planning 22 (1): 65-72. Available
online: http://www.csubak.edu/~dgeorgi/pres_refs/proposals/McLaughlin's%20Logic%20Models.doc.
McLaughlin, John A., and Jordan, Gretchen B. 2004. “Chapter 2: Logic Models,” Handbook of Practical Program
Evaluation, 2nd Edition, Wholey, J., Hatry, H., and
Newcomer, K., Eds., Jossey-Bass.
Megdal, Lori, Allen Lee, Todd Board, Betsy Wilkins, and
Mary O’Drain. 1999. “Using Diffusion and Communications Theory to Expand Market Barrier Examination in
MT Measurement,” Proceedings from the 10th National
Energy Services Conference, Tucson, AZ: December, pp.
584-595.
Megdal, Lori, Dune Ives-Petersen, and Andy Ekman. 2000.
“Local Government Associations as Agents of Change,”
Proceedings from the 2000 ACEEE Summer Study, American Council for an Energy-Efficient Economy, Asilomar:
CA, pp. 9.287 – 9.298.
Megdal, Lori, Mark Coleman, Jennifer Ellefsen, Larry Pakenas, Helen Kim, and Scott Albert. 2001. “He did it!
He did it! – Providing Evidence for Causality”, Proceedings from the 12th National Energy Services Conference, Association of Energy Services Professionals, Jupiter: FL,
pp. 140-150
New York Energy State Energy Research and Development
Authority (NYSERDA). 2004. New York Energy $martSM
Program Evaluation and Status Report. New York, N.Y.:
New York Energy State Energy Research and Development Authority. Available online: http://www.nyserda.org/Energy_Information/04sbcreport.asp
1054
ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS?
PANEL 5. EVALUATION AND MONITORING
Peters, Jane, Scott Albert, Victoria Engel, and Lori Megdal.
2004. “Using Logic Models to Improve and Enhance
Nonresidential Programs,” Proceedings of the 2004
ACEEE Summer Study on Buildings, 4.275 – 4.285.
Rogers, Everett. 1983. Communication of Innovations: A CrossCultural Approach. New York, N.Y.: Free Press.
Rogers, Everett. 1995. Diffusion of Innovations, 4th Edition.
New York, N.Y.: Free Press.
Rogers, Everett with F. Floyd Shoemaker. 1971. Communication of Innovations: A Cross-Cultural Approach. New York,
N.Y.: Free Press.
Rogers, Patricia J., Petroscino, Anthony, Huebner, Tracy A,
Hacsi, Timothy A. 2000. “Program Theory Evaluation:
Practice, and Problems,” New Directions For Evaluation,
Number 87, Fall, Jossey-Bass, pp. 5-13.
TecMarket Works Framework Team 2004. The California
Evaluation Framework. Southern California Edison
Company. Study ID K2033910. Can obtain at: http://
www.calmac.org/search.asp Then enter in “California
Evaluation Framework” and can download the 500-page
reference document as an Adobe .pdf file.
Underhill, Paco. 1999. Why We Buy: The Science of Shopping.
New York, N.Y.: Touchstone, Simon & Schuster.
Vine, Edward, Luisa Freeman, Joseph Lopes, Martin Adelaar, Barbara Atkinson, Rafael Friedman, and Iris Sulyman. 2003. “Interim Process Evaluation of the Efficient
Lighting Initiative: 1999-2001.” Proceedings of the 2003
International Energy Program Evaluation Conference, 17785. Madison, Wisc.: National Energy Program Evaluation Conference, Inc.
Weiss, Carol H. 1998. Evaluation: Methods for Studying Programs and Policies. Prentice Hall: Upper Saddle River,
NJ.
Worthen, Blaine R, James R Sanders and Jody L Fitzpatrick.
1997. Program Evaluation: Alternative Approaches and
Practical Guidelines. 2nd Edition. Longman: New York,
NY.