5,117 Using program logic model analysis to evaluate & better deliver what works Lori Megdal Megdal & Associates 198 High Street Acton, MA 01720-4218 USA megdal@bellatlantic.net Victoria Engle and Larry Pakenas with the New York State Energy Research and Development Authority Scott Albert with GDS Associates, Inc. Dr. Jane Peters with Research Into Acton, Inc. Dr. Gretchen Jordan with Sandia National Laboratory Keywords Evaluation, logic models, program theory, market transformation, program redesign Abstract There is a rich history in using program theories and logic models (PT/LM) for evaluation, monitoring, and program refinement in a variety of fields, such as health care, social and education programs. The use of these tools to evaluate and improve energy efficiency programs has been growing over the last 5-7 years. This paper provides an overview of the state-of-the-art methods of logic model development, with analysis that significantly contributed to: 1. Assessing the logic behind how the program expects to be able to meets its ultimate goals, including the “who”, the “how”, and through what mechanism. In doing so, gaps and questions that still need to be addressed can be identified. 2. Identifying and prioritize the indicators that should be measured to evaluate the program and program theory. 3. Determining key researchable questions that need to be answered by evaluation/research, to assess whether the mechanism assumed to cause the changes in actions, attitudes, behaviours, and business practices is workable and efficient. Also will assess the validity in the program logic and the likelihood that the program can accomplish its ultimate goals. 4. Incorporating analysis of prior like programs and social science theories in a framework to identify opportunities for potential program refinements. The paper provides an overview of the tools, techniques and references, and uses as example the energy efficiency program analysis conducted for the New York State Energy Research and Development Authority’s (NYSERDA) New York ENERGY $MARTSM programs. Introduction and Background There is a rich history in using program theories and logic models (PT/LM) for evaluation, monitoring, and program refinement in a variety of fields, such as health care, social and education programs. Using program theory to drive evaluation was promoted by Carol Weiss beginning in 1972 (Worthen, 1997, page 221). Program theory is a theory or model that describes the underlying assumptions about how a program is expected to work; how the program causes the intended or observed outcomes. A logic model is a diagram that describes the key logical (causal) relationships among program elements and the problem to be solved, thus defining measurements of success. It helps portray the program theory. The logic model can be used to help tell ”the story” behind how the program expects to be able to meets its ultimate goals, including the “who”, the “how”, and through what mechanism. In doing so, gaps and questions that still need to be addressed can be identified. The elements of the logic model describe and place the causal sequence of program activities, outputs, immediate outcomes, and longer-term outcomes. Often the logic model is displayed with these elements in boxes and the causal flow being shown by arrows. Indicators can be derived to measure each of these elements and their tracking can be used to assess program success. ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS? 1045 5,117 MEGDAL ET AL PANEL 5. EVALUATION AND MONITORING Successful Program Program set in motion Causal process which led to Desired effect Theory Failure Program set in motion Causal process did not lead to Desired effect Program Failure Program did not set in motion Causal process which would have led to Desired effect Figure 1. Theory Failure & Program Implementation Failure. An evaluation design and program theory can work handin-hand to be able to have the evaluation differentiate between theory failure (incomplete or inaccurate theory), and program failure (poorly designed or implemented operational procedures). In case of theory failure the assumptions about the market or the causal mechanisms that create attitudes or behaviours are not valid, are only partially valid, or key theoretical components are missing. The program implements as planned but the causal link to final outcome does not occur. The program theory needs to change and the program needs to be refined accordingly. On the other hand, if the evaluation identifies program failure, this means that the theory appears to be correct. However, the program implementation had problems that did not allow it to have the anticipated outputs and initial subsequent outcomes. In other words, if the causal process had occurred the outcome might have occurred but program implementation problems did not create this process. Figure 1 shows the differences between theory failure and program failure1. The use of these tools to evaluate and improve energy efficiency programs has been growing over the last 5-7 years. In the past five years, it has become increasingly clear that making a program’s theory and logic explicit is important for effective program implementation and evaluation, especially when the program includes or is dominated by market transformation goals and objectives (Erickson, Fagan & Block 2003; Goldstone, Rufo & Wilson 2000). The recent comprehensive energy efficiency evaluation reference manual, The California Evaluation Framework, states that “An important component of the evaluation effort is to draw upon the program theory and logic model, to include its review (or development if one is not available) and use as an evaluation planning tool” (TecMarket Works (2003), page 30). It also says that while important for all types of program evaluations it is especially important for complex programs and programs with long-term market change goals. This document goes on to provide more in-depth descriptions, references, basics of how to develop PT/LM with an example in an appendix. McLaughlin and Jordan (2004) write that Rogers et. al. (2000) and Birkmayer and Weiss (2000) present examples of theory-driven evaluations, but report that while theory-driven evaluation is conceptually sound, it is rare to find good examples in practice. For all of these reasons, the New York Research and Development Authority (NYSERDA) contracted with an eval- 1. Weiss (1998) page 129. 1046 ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS? uation consultant team in 2003 to conduct program analysis with program theories and logic models, and developing these where need be. The authors were part of this team lead by GDS Associates. The task was to develop logic models for programs and to analyze each program relative to social science theory and other program experience. The program analysis activities cover most of the over 30 residential, non-residential, research and development, and low-income programs in the New York Energy $martSM portfolio. Over the 2003 and 2004 period, program analysis with PT/ LM was conducted for four residential, one low-income, four research and development, and four non-residential programs. The program analysis approach has included two primary stages: logic model development and program analysis. The logic model development stage involved four steps by the team: initial data collection, problem description, logic model definition, and logic model diagram construction. At the end of the first stage, the program analysis team reviewed the diagram with the program staff to refine the model and assure that it articulated the program design and implementation. The second stage involved three steps: relevant social science and business theory research, logic model assessment against the social science and business theories, and recommendations for program refinement and further research. The final activity in the comparative assessment with social science theories involved exploring the logic model and program analysis findings with program staff to assure that the findings articulated program design and implementation. The last step on recommendations also included meeting with staff to inform them of any findings, new information or recommendations that arose from the program analysis activities. Developing the Logic Model to Portray the Program Theory Based on current industry best practices, program logics include the following elements: • Key program resources/inputs (program funding, internal and contractor staffing, sources and magnitudes of leveraged funding/partnerships, etc.); PANEL 5. EVALUATION AND MONITORING 5,117 MEGDAL ET AL • Activities (internal and contractor program implementa- is to start with the long-term outcomes and work backwards to resources. Essentially, the process is one of repeatedly asking the same question, if “Z” is a long-term outcome (or short-term outcome, output, or activity), what is required to produce “Z.” It is then a matter of writing the causal relation in the form of a statement: “Y” will cause “Z.” One then backs up and asks what will cause Y and continues until one has described the required activities and resources. One can then reverse the order and edit the statements until one has a sequence of causal statements that describe how the program works. Several things are likely to happen as this is being worked upon. It is likely that gaps in the causal relationships between actions and expected effects will be found. Some steps will be identified that require substantial leaps that suggest that the theory needs further refinement. Some steps in the theory will seem quite improbable, suggesting that the theory, and probably the program design, needs improvement. Some steps will contradict what is known from the program, marketing, and evaluation literature and other social science and business theories. Sometimes an assumption, key to the way the program is designed for one of its causal chains, can be in conflict with an assumption in a different causal chain. All of these instances identify places where the program theory and the program design may need improvement. tion tasks, outreach/marketing and delivery mechanisms, etc.); • Customers and partners (who the program works for and with – customers receive products and services directly from the program and its partners, and change behaviour or take action that translates into program outcomes); • Outputs (internal and implementation contractor servic- es, products, training/support being provided to target customers or market actors, etc.); • Outcomes (short, intermediate, and longer-term antici- pated results/benefits/market changes from program activities – many of which come directly from the program’s stated measurement indicators and appropriate/targeted portfolio-level goals and objectives), including how these contribute to overarching policy goals; • Any perceived external influences (recognizing the influ- ence that market actors, barriers, other New York Energy $martSM programs, state, regional and national activities or circumstances, etc., may have on a program’s logic); and • Drawn from the logic, measurable indicators and explicit, researchable issues. The basic logic model format is shown in Figure 2. The steps to develop the logic model are as follows: An Example Logic Model from NYSERDA’s Programs 1. Collect information through documents and perhaps establish a stakeholder workgroup. The team collected all possible documentation describing the program—its goals and objectives, its marketing plan, and reports describing the program from different perspectives. Using these secondary data sources and the pre-existing logic model, we drafted a new graphic logic model and prepared a summary of the secondary data in the form of a description of the logic model. The logic model and narrative were provided to the program staff for their review and comment; a two-hour conference was then held to share staff observations with the team. In many ways, the development of the logic model was the most straightforward of all the activities. The program staff was able to review the narrative and determine where inevitable errors of fact had occurred. They also reviewed the logic model and made recommendations regarding the diagram and how the program logic worked from their point of view. The team then revised the model to reflect this input. Figure 3 shows the final logic model for NYSERDA’s Keep Cool Campaign and program to transform the market for room air-conditioners (RAC) in New York. 2. Define the problem and context for the program. 3. Define elements of the logic in a table. 4. Develop a diagram of logical relationships. 5. Verify the program theory/logic with stakeholders, comparisons with implementation results 6. Then use the logic model to develop or confirm performance measures for program monitoring and performance contracts, and in planning and evaluation. Often developing the logic model from gathered data can begin from organizing the information into tables, or taking it step-by-step from activity to output to outcomes. Once a program has been systematically described in terms of resources, outputs, outcomes, and long-term impacts, the procedure for developing a program theory is a systematic one. Many PT/LM leaders and teachers have found that one of the best ways to develop a program theory Resources (inputs) Activities Outputs for Customers Reached Short-term Outcomes Intermediate Outcomes (through customers) Long-Term Outcomes & Problem Solution External Influences and Related Programs (mediating factors) Figure 2. Basic Logic Model Format. ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS? 1047 5,117 MEGDAL ET AL PANEL 5. EVALUATION AND MONITORING Some of the key issues that emerged from the development of the Keep Cool logic model were that the program had both resource acquisition and market transformation strategies and associated tactics. Like many resource acquisition programs Keep Cool sought aggressively to achieve kWh savings and kW peak reductions relatively quickly by working within the market structure with significant incentives, so there are strong market elements in its resource acquisition strategy. At the same time, the program logic included a strategy and associated tactics that, as the program staff anticipated, would lead to a long-term sustainable transformation for RACs by encouraging significant changes in stocking and consumer awareness. INDICATORS DEVELOPED FOR LOGIC MODEL ELEMENTS The next major step was to develop indicators for the output, and outcome elements of the program logic that could be used for evaluation research design. Operationalizing these indicators and then prioritizing them helps to define the evaluation research goals needed to support tracking the program effects. Several of the important indicators are shown in Table 1. RESEARCHABLE QUESTIONS Each arrow in the logic model presents a researchable question in that it presupposes a causal mechanism. Each arrow also implies the need for research on whether the action is resulting from the prior action and how it occurs. This can be part of the implementation theory, what is required to translate an activity into an output (the delivery of program services) or translating one outcome into another (the social, psychological, and/or market processes that cause the change to occur). Each item mentioned could be investigated as part of the evaluation. Being able to identify the critical areas to be included in the evaluation research design, apart from the many potential questions embedded within the causal linkages, is an important skill to be honed. In a market transformation program, one way to identify key linkages is to assess which boxes require behaviour change in market actors that are essential to the process. Also look for boxes where their outgoing arrows are expected to lead to several items in the causal chain. Additionally, if there is only one mechanism that is to change behaviour in a key market actor then that mechanism is critical and should be evaluated. A few of the researchable questions derived from analysis of the Keep Cool Program logic model provided above include the following: • How well are retail staff trained? • Do knowledgeable retail staff promote ENERGY STAR products? • How important is this/how great is the effect? (Are pro- gram resources optimally targeted?) • How well can the advertising messages target folks buy- ing RACs to buy ENERGY STAR? Is there a spillover where more people buy RACs then otherwise would? If so, how much? • How much recycling of RACs occurs without interven- tion, post-bounty effort? • Load-shifting from advertising has been measured. How much is maintained behavior versus how much reinforcement messaging is needed? One of the critical researchable questions, developed as working through the Keep Cool Program logic model, was derived from one of the design premises for the original Keep Cool Program. This premise is that the replacement (turn in) of old, operating AC units with new more efficient units, will reduce the overall energy usage (especially during summer peak periods in New York). It is implied within the logic that the program’s recycling efforts will lead to fewer RACs in the secondary market. The Keep Cool Program thereby reduces energy and demand usage as more, new RACs are purchased at higher efficiency levels than the efficiency levels in the secondary market. Without turn-in of the old units, these less efficient RACs will likely find their way into other rooms in the same house, in use at family or friends homes, or for sale in the secondary market, thus increasing kWh usage and summer peak demands. It will be important therefore, to confirm that the program’s recycling efforts are in fact reducing the number of RACs in the secondary market. If it is found that, as a result of program ad- Table 1. Some of the Indicators for the Keep Cool Program. Program Outputs Number of air conditioners surrendered Number and dollar value of bounties (rebates for turn-in of old working RACs) Number of units demanufactured and amount of material diverted from the waste stream Number of ads placed, impressions, and ad value Number of retailers active in the program 1048 Short-term Outcomes Intermediate-term Outcomes Long-term Outcomes Change in awareness of Perceived benefits of NYSERDA program and ENERGY STAR® product ENERGY STAR® purchases In conjunction with other ENERGY STAR® efforts: - Eliminated barriers Degree of subsequent - Reduced waste by ENERGY STAR® product recycling of old units purchases given past ENERGY STAR® experience - Increasing market share Knowledge and ability of and penetration Frequency and content of retail staff to promote - Sustained change in ENERGY STAR® RACs communication to others and efficient TTW units market behavior Retailers indicate that Immediate Peak ENERGY STAR® RACs and - Persistent energy savings Reduction and KW and efficient TTW units are - Emissions reductions KWh savings profitable to stock & sell Effectiveness of TV advertising versus other advertising venues ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS? Activities Develop marketing and advertising strategies for consumers, POP display, educational and website information for consumer awareness campaign Consensus building with NYPA & LIPA Outputs Coordination with ENERGY STAR Products Program Events, radio/TV ads, newspaper ads, collateral, event direct contact, website hits Recruit ENERGY STAR (ES) retailers for participation and training Develop capability for demanufacturing Develop availability of facilities for turn-in Participating retailers trained in program processes Install point of purchase (POP) displays Process applications Provide Bounty Increased Awareness & Knowledge of Keep Cool Program Short Term Outcomes 1-3 Years Develop bounty application process Increased ability of sales personnel and stores to promote the program Sales of ES RAC and TTW to bounty users Old units turned in & demanufactured, being replaced by ES RAC or high efficiency TTW R ecognize benefits from selling up ES product & word-of-mouth Shift on-peak to off-peak usage Recognize benefits from purchase of ES product & increase likelihood of future ES purchase & word-of-mouth Jobs created through continued demanufacture and recycling of AC units Increased Awareness & Knowledge Energy Star Appliances (including RACs) Recycling centers process without bounty kW and kWh savings Intermediate and Long Term Outcomes 4-8 years (contributing to MT along with other residential ENERGY STAR programs) Long-term environmental benefits Increased demand of ES products based upon awareness, knowledge and recognition of benefits. Supplier provide greater ES RAC and TTW availability (without need for incentives). Provide advertising, POP, and sales training. Leads to less resistance to and support for increased codes and standards. Legend 1049 Figure 3. Keep Cool Program Logic Model. = Program Activities = Program logic flow = Outputs and outcomes = Critical logic flow to meet long-term goals = Key logic elements = External influences 5,117 MEGDAL ET AL External Influences: Weather, Codes and Standards; Positive - National ENERGY STAR program, NYSERDA Home Performance with ES, NY Blackout, CA energy crisis; Negative - Recession, Other competing attention draws (i.e., 9/11, War in Iraq) and investments Using Relevant Social Science Theory and Prior Evaluations for Analyzing the Program and It’s Logic Outside of program sales of ES Appliances (including RAC and TTW) This second stage of the process was exciting and innovative territory to the team members. All had extensive experience reading and pondering about social science theories and ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS? Peak Reduction Increased stocking of ES RAC and TTW units PANEL 5. EVALUATION AND MONITORING 1-29-2004 vertising, more air conditioners (albeit ENERGY STAR® units) are being purchased than otherwise would have occurred. Then anticipated energy and peak period savings benefits may be impacted and be less than originally anticipated by the program. Keep Cool Program High Level Logic Diagram Inputs: Funds, staff, allies, market knowledge, synergistic program management 5,117 MEGDAL ET AL PANEL 5. EVALUATION AND MONITORING COMMUNICATION CHANNELS I. Knowledge II. Persuasion III. Decision IV. Implementation 1. Adoption V. Confirmation Continued Adoption Later Adoption Discontinuance 2. Rejection Continued Rejection Figure 4. Innovation-Decision Process. how they might apply to energy efficiency programs. However, this was the first time that team members were aware of an energy efficiency client requesting a program logic model be reviewed in the context of social science theory and then tested against those theories within the experience of an energy efficiency program. Hence, the team focus on program analysis was a clear indication that a logic model, in and of itself, was insufficient to meet the client’s needs and interests. The most important issue was to test the model in the context of social theory. The first steps in the program analysis were to identify what theories might be relevant to the program logic, what anticipated outcomes should be explored through a review of similar programs and verify the logic model relative to those theories. The theories we identified to be of interest were social marketing theory, diffusion of innovation, and consumer economics. The next phase involved presenting the program staff with the team’s findings and recommendations and then discussing their implications for the logic model and the program itself. Some of the program analysis findings are included below as examples of this type of effort. MARKETING, SOCIAL MARKETING, AND CONSUMER SHOPPING The PT/LM for Keep Cool is looking to change purchasing behaviour of owners of old RACs. It is also looking to change the behaviour of retailer stock and promote ENERGY STAR RACs. This drew the social science/business research to include marketing, social marketing, and consumer shopping. There are many behaviour change models theorized. In marketing, and most market transformation programs on energy efficiency, the most cited is the Rogers’ and Shoemaker’s innovation diffusion model (Rogers and Shoemaker, 1972). This field of work involves a few variations on the theme of an awareness-adoption model. This model states that the process for adopting an innovation (or buying a different product) moves through stages of awareness, knowledge, persuasion, decision, implementation, and confirmation, as displayed in Rogers’ diagram shown in Figure 4 (Rogers 1995, page 163). Evident from this is the importance of communication flows and interactions between market participants (communication channels) in order to move from one stage of adoption/diffusion to the next and to do so with positive adoption, confirmation and continued adoption. This is where education, advertising, marketing, and selling influence the adoption process. Marketing of the Keep Cool Program and the benefits of ENERGY STAR qualifying air conditioners is a major component of the Keep Cool Program. One of the primary goals of the market transformation programs is to move forward in time the adoption of more efficient technology. The efficiency gain is then caused by the difference in the adoption rate of the more efficient appliance, given program intervention, versus what the adoption would have been without the Figure 5. Market Transformation Gains from Shifting S-Curve of Technology Adoption. 1050 ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS? PANEL 5. EVALUATION AND MONITORING program. The typical S-curve for technology adoption with and without the program display the gain made with the intervention. This is shown in Figure 5. Much of the marketing to purchase ENERGY STAR room air-conditioners (RAC) focuses upon the energy and utility bill savings that can be obtained by purchasing more efficient RACs. This is a relatively straightforward economic message to consumers. A large part of the marketing campaign focuses on the broader societal benefits of shifting energy use to off-peak times to reduce load during peak periods throughout the summer. Both the decision to purchase energy efficiency and the shifting of clothes washing and dishwashing to offpeak periods used special marketing messages that fall squarely in the realm of social marketing theory. Social marketing is the application of marketing theory to pro-social activities—those activities that have a societal rather than a private benefit (Andreasen 1995). Social marketing has been directly applied to many products over the years: reduction of litter, recycling, the Smokey Bear campaign, condom use in countries plagued by HIV infection, and recently for international energy efficiency campaigns such as the Efficient Lighting Initiative (Vine et al. 2003). Indirectly, many marketing campaigns for energy efficiency products and services in the United States are also social marketing campaigns. The types of promotions needed in these situations are much more involved than marketing of more traditional consumer goods where little thought is required on the part of the consumer. Rather, these decisions are defined in the literature as “high involvement” as they require greater effort on the part of the customer and not all of the benefits of these decisions are directly seen by the purchaser (i.e., the benefits of reducing load during peak periods benefits all utility customers.) As a consequence, the decision to purchase an ENERGY STAR room air conditioner will have several stages of information gathering, contemplation, and action. Awareness of the program and the benefits of ENERGY STAR brands may not be enough; the customer may spend time during a “contemplation” stage to carefully assess the pros and cons of purchasing an energy efficient model. According to social marketing theory, the shift to purchase high efficiency room air conditioners that are covered by the Keep Cool Program will involve a change in values. This change in values best occurs when the candidate understands that the change in behaviour (purchasing a more expensive ENERGY STAR model, moving their clothes and dish washing to off-peak) is acceptable to others in their peer group. Once this decision is made, the candidate also needs to be convinced that the purchase will be beneficial on a personal level. Given this decision structure, models for social marketing provide useful maps of the decision processes as well as effective marketing techniques in these situations. A social marketing message is one that provides information on how their action will affect societal attributes that consumers will value, such as environmental quality. For example, “If X number of people bought ENERGY STAR rather than non-ENERGY STAR, then we would take X lbs of CO2 out of the air”. 5,117 MEGDAL ET AL The contemplation stage view of social marketing could be used alongside the theory of social proof. The principle of social proof is used in much advertising and operates in many non-advertising ways. (Non-advertising examples of the principle of social proof include copy cat suicide and murder, and where strangers follow another’s lead on whether to ignore someone that might be injured or is asking for donations, etc.) This is, that “we view a behaviour as correct in a given situation to the degree that we see others performing it” (Influence: Science and Practice, 2001, pp. 100). In the face of uncertainty and with others around, the principle of social proof is strongest. Perhaps the point-of-purchase (POP) and sales techniques can use this to sell ENERGY STAR RACs where uncertainty is high, such as a customer that has come to make an “emergency” replacement for a broken RAC. These customers may not have gone through the contemplation stage (unless they have done so for another ENERGY STAR product, a good reason for the branding and program coordination). They are also not eligible for the bounty program (a consumer incentive which requires both purchase of an ENERGY STAR RAC and turn-in of a working old RAC). At the same time, they are still a good opportunity for the retailer to sell up to an ENERGY STAR product. One method often used in this type of marketing is advertising how many people have bought the product. This program has had extremely large growth, from 17 000 in 2001 to over 233 000 in the first three years. This large amount might be used as a sales tool to influence the purchasing decisions of future RAC consumers. The larger participating stores might be able to advertise how many people have purchased ENERGY STAR RACs at their store or ENERGY STAR products at their store. This type of POP advertising would capitalize on social proof effects. An example that was provided in one of the theory literature documents we reviewed in this project might provide an interesting tidbit for store sales of RACs. In Why We Buy: The Science of Shopping, Paco Underhill has a chapter on the fact that many consumers buy as a sensual shopper: touch, smell, feel of a product. He discusses that this can be difficult for some products and explicitly uses air conditioners as an example. He points out that you can’t tell how an air conditioner cools by turning one on in the store, especially a cool store. Customers can check with friends, Consumer Reports, or the salesclerk’s opinion. But at the store the thing that could make a difference in choice of air-conditioner would be “how does it sound?” “In the final analysis, it’s one of the few things that distinguish one air conditioner from another. The unit is going to be humming (or clattering) away in your house for a number of years, after all. In a typical summer, I’ll bet I have three or four conversations about air conditioner decibel levels. That’s what actual human beings care about when it comes to air conditioners, but you’d never know it when you’re shopping for one.” Selling retailers on using ENERGY STAR to distinguish one air conditioner from another is a key benefit that can encourage a consumer to conclude their purchase. Marketing should cover the fact that ENERGY STAR products offer all the major benefits of non-ENERGY STAR plus added energy and money savings. Yet, drawing upon the work of Paco Underhill marketing images that present ENERGY STAR ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS? 1051 5,117 MEGDAL ET AL as “environmentally friendly, but in a quiet cool way” might be worth considering and testing. FURTHER DIFFUSION OF INNOVATION EFFECTS LEVERAGED The primary goal of the program is to gain adoption of ENERGY STAR RACs in place of older RACs, the older technology. Diffusion of innovation theory was therefore also closely examined relative to the program logic. Most market transformation program planners and implementers are familiar with the basic notion of diffusion promulgated by Rogers (1983, 1995). This is where innovators become the first to adopt a technology, followed by early adopters, then the early majority, the middle, and finally the late majority and laggards. The late majority and laggards are rarely of interest to market transformation programs, as these people are those who adopt after effective market transformation has occurred. Innovators are those who adopt even without program efforts, though they often take advantage of programs when they emerge and likely becoming freeriders. The focus of most market transformation programs has been on early adopters and early majority adopters. Consumer preferences for air conditioners are determined in part by the choice of models available in addition to available rebates and incentives, which includes the secondary market of used air conditioning units. The program’s rebate effort requires the turn-in of old working RACs along with the receipt of purchasing a new ENERGY STAR RAC. Often these old RACs will either be used in other rooms of the house or sold on the secondary market. The turn-in and environmentally friendly disposal of these older units reduces the number of older inefficient units in the secondary market. By helping reduce the option for choosing a used (and less efficient) unit, the program can increase the market share of ENERGY STAR models. This shift in the supply of air conditioners is also consistent with diffusion of innovation theory. The stage of the market that is demanding used air conditioning units likely falls in the “late majority” and “laggard” portions of the innovation curve. That is, this segment of the market contains those customers that adopt “new” technologies only after the equipment is well established in the market and considered more or less standard (and thus has a relatively low price compared with other available options.) The program’s pitch to properly dispose of older RACs reduces the secondary market and can cause a smaller and more efficient pool of RACs in this market. As the secondary market shrinks causing prices to rise, there is less incentive not to buy a more efficient new RAC (and all new RACs are significantly more efficient than old ones given changes in technology and appliance standards). BUSINESS ECONOMICS Perhaps the most critical component of the Keep Cool Program is the recruitment of enough retailers to participate in the program to have a significant influence on the market. Once recruited, retailers receive sales training and other promotional support (advertising, POP materials, scanners to facilitate intake of units) for ENERGY STAR air-conditioning units. In order for recruitment to be successful, participation must be economically beneficial for the retailers. That is, 1052 ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS? PANEL 5. EVALUATION AND MONITORING the support and tools provided by the program must be effective enough ultimately to increase profits. Although the ENERGY STAR models are typically more expensive to produce than standard efficiency models, the profit margin also tends to be higher. Consequently, retailers may be able to increase profits even if they sell fewer units. It may be tempting to apply social marketing techniques (discussed below) to appeal to retailers for participation. Given that the non-monetary benefits to retailers from participation are small, these techniques are unlikely to be successful. That is, appealing only to the broader benefits of the ENERGY STAR models (reduced bills for customers, peak load reduction, and public relations benefit of retailer being perceived as “green”) are unlikely to be effective as retailers receive few of these benefits directly and must bear a disproportionate share of the costs. For this reason, recruitment should focus on the potential for increased sales and the effectiveness of program support in terms of training and promotional materials in helping reach the increased sales potential. At the same time, providing evidence that offering ENERGY STAR products and promoting environmentally friendly products can help increase sales may be important. “Selling” the product and concept to the retailers is different than providing them with alternative selling messages to sell to consumers. It is here that the environmental messages can be useful for that niche of consumers that value environmentally friendly products. The program, however, also indirectly provides another benefit to retailers. The program strongly advocates disposing of working older inefficient units and replacing them with new ENERGY STAR units. This is a definite message that promotes greater purchases of ENERGY STAR RACs (and other RAC purchases as well). This should have a positive effect on retailers and their perception of the benefits of the program overall. Diffusion is largely a communications issue in which information is transferred between individuals. When behaviour change is sought for pro-social reasons, such as adoption of a new agricultural technology or new healthcare practice, a change agent generally initiates interest in the new behaviour. Diffusion theory comments on both the characteristics of the change agent, and the characteristics of those market actors that the change agent targets for their outreach. Research by Rogers (1995) suggested that change agents needed to be technically knowledgeable, such that the target group respected them; however, those who had a commercial interest in the behaviour change might be less accepted by the target group. Given this, retailers are not true or perfectly designed change agents. Yet, to the extent that sales staff do education, and promote the ENERGY STAR RACs due to the retailer training and the benefits retailers perceive in supporting the program, the sales staff may be able to leverage the consumer social marketing and act as surrogate change agents from the diffusion of innovation perspective. PANEL 5. EVALUATION AND MONITORING Summary of the Advantages of this Approach and Lessons Learned Developing theory and logic models for programs, prior to their implementation in the field, can be the most effective way to influence program design and can allow the program to build cost effective evaluation and performance monitoring into the program plan. Using these tools within program design can help the program ensure that it is well designed to meet both its short term and long term goals. Expecting specific market responses to interventions that change the way a market operates can be difficult. Being explicit about all the steps required to make this happen can help provide insight into whether the program logic appears to lead to the ultimate goals without any “black boxes” where the creation of the desired outcome is unknown and, therefore, less likely to occur. Unfortunately, explicit logic modelling can be a low priority to some program designers and implementers. Though these individuals had given a great deal of thought to the design of the programs studied and the theory behind the program, they did not have the time to invest in the process of explicitly modelling the project or conducting the program analysis steps. Designers and implementers are often faced with political pressure to have programs fielded as quickly as possible after funding is authorized. This normally does not allow them to spend the time and resources for using these more thorough program design tools. The mindset, backgrounds, and skills for those best suited to getting a program into the field, making weekly and daily decisions about implementation and keeping the program progressing toward its annual goals and objectives, are different from those that best work in this almost abstract world. At the same time, those best suited to this type of work are also probable less likely to shine in program implementation. It is necessary, though we often found some difficulty, to obtain time and significant input from program staff. An important lesson learned is that logic modelling and program analysis require not only technical skills and knowledge, but also good listening and facilitation skills. Because logic modelling may be a low priority for program staff, it is necessary for the logic modelling team to be able to facilitate discussions about the program, the program analysis and the logic model, and to be able to take the time to listen to the response from program staff. It is not necessary for staff to articulate the model and the theories. But it is important to recognize that they developed the program theory and thus, if they do not buy into the logic model and analysis, any work that flows out of the modelling process, such as evaluation research and reports, risks limited acceptance. In the end, the involvement of all key program staff in the discussions, especially the program manager who makes final decisions, led to the program analysis being better understood and the logic model being more consistent with the program. It would appear a cooperative team approach might offer the best of both worlds. The addition of examining evaluations of similar programs, and looking at social science theories, provided important perspectives for more in-depth analysis of the program. The program analysis team was able to go deeper into the program design logic and assess whether it was con- 5,117 MEGDAL ET AL sistent with social science theory and research. Often this provided information that might be able to be used for program refinements and lead to evaluation questions that can help program implementers learn how to better facilitate the causal mechanisms they are hoping to influence. These revelations in the logic modelling and program analysis process have begun to provide the program staff with fodder for thinking about opportunities to further refine the program. Being able to identify the critical areas to be included in the evaluation research design, apart from the many potential questions embedded within the causal linkages, is an important skill to be honed. In a market transformation program, one way to identify key linkages is to assess which boxes require behaviour change in market actors that are essential to the process. Also look for boxes where their outgoing arrows are expected to lead to several items in the causal chain. Additionally, if there is only one mechanism that is to change behaviour in a key market actor then that mechanism is critical and should be considered for evaluation efforts. Evaluations conducted after these assessments adopted many of the indicators for measurement and follow-up in future evaluations. The program performed well on these indicators. Enough progress was made in retailer stocking and promotion that the program dropped the rebate component and only focused on marketing. Indicators from the evaluation suggested that any increased sales of RACs due to ENERGY STAR RAC promotion was minimal and the effect was quantified and included in total energy savings estimates. Finally, the program analysis with the social science theories has led to changes in the advertising messages being tested in the current program. Overall, NYSERDA’s assessment of the advantages and reasons for developing program theories, logic models, and conducting the in-depth program analysis discussed here can be summarized as this tool’s and process’ ability to help : • Promote critical thinking about programs • Identify key indicators and researchable issues for evalu- ation research design • Map linkages (and disconnects) between activities, out- puts, and outcomes • Provide a basis for program changes or status quo • Tell the program story in a short, precise format • Management/stakeholder/policy maker comfort • Best done during program planning and design but post- implementation analysis can help provide a reality check • Identify impacts related to changes in program design ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS? 1053 5,117 MEGDAL ET AL References Albert, Scott, Victoria Engel, Gretchen Jordan, Lori Megdal, and Jane Peters. 2004. “Using Program Theory And Logic To Improve Design and Likelihood of Real Market Change- Experience With A State Public Benefits Program,” Proceedings of the 2004 ACEEE Summer Study on Buildings, 6.1 – 6.12. Andreasen, Alan R. 1995. Marketing Social Change: Changing Behavior to Promote Health, Social Development, and the Environment. San Francisco, CA: Jossey-Bass. Birkmayer, J. D. and Weiss, C. H. 2000. “Theory-Based Evaluation in Practice: What do we learn?” Evaluation Review., Vol.24, # 4, pp 407-431. Cialdini, Robert B. 2001. Influence: Science and Practice. 4th Edition. Boston, MA: Allyn and Bacon. Erickson, Jeff, Jennifer Fagan, and Oscar Bloch. 2003. “The Program Theory and Metrics Process.” Proceedings of the 2003 International Energy Program Evaluation Conference, 517-30. Madison, Wisc.: National Energy Program Evaluation Conference, Inc. Goldstone, Sy, Michael Rufo, and John Wilson. 2000. “Applying a Theory-Based Approach to California’s Nonresidential Standard Performance Contract Program: Lessons Learned.” Proceedings of the ACEEE 2000 Summer Study on Energy Efficiency in Buildings, 5:103-5.117. Washington, D.C.: American Council for an Energy-Efficient Economy. McLaughlin, John A., and Gretchen B. Jordan. 1999. “Logic Models: A Tool for Telling Your Performance Story.” Evaluation and Program Planning 22 (1): 65-72. Available online: http://www.csubak.edu/~dgeorgi/pres_refs/proposals/McLaughlin's%20Logic%20Models.doc. McLaughlin, John A., and Jordan, Gretchen B. 2004. “Chapter 2: Logic Models,” Handbook of Practical Program Evaluation, 2nd Edition, Wholey, J., Hatry, H., and Newcomer, K., Eds., Jossey-Bass. Megdal, Lori, Allen Lee, Todd Board, Betsy Wilkins, and Mary O’Drain. 1999. “Using Diffusion and Communications Theory to Expand Market Barrier Examination in MT Measurement,” Proceedings from the 10th National Energy Services Conference, Tucson, AZ: December, pp. 584-595. Megdal, Lori, Dune Ives-Petersen, and Andy Ekman. 2000. “Local Government Associations as Agents of Change,” Proceedings from the 2000 ACEEE Summer Study, American Council for an Energy-Efficient Economy, Asilomar: CA, pp. 9.287 – 9.298. Megdal, Lori, Mark Coleman, Jennifer Ellefsen, Larry Pakenas, Helen Kim, and Scott Albert. 2001. “He did it! He did it! – Providing Evidence for Causality”, Proceedings from the 12th National Energy Services Conference, Association of Energy Services Professionals, Jupiter: FL, pp. 140-150 New York Energy State Energy Research and Development Authority (NYSERDA). 2004. New York Energy $martSM Program Evaluation and Status Report. New York, N.Y.: New York Energy State Energy Research and Development Authority. Available online: http://www.nyserda.org/Energy_Information/04sbcreport.asp 1054 ECEEE 2005 SUMMER STUDY – WHAT WORKS & WHO DELIVERS? PANEL 5. EVALUATION AND MONITORING Peters, Jane, Scott Albert, Victoria Engel, and Lori Megdal. 2004. “Using Logic Models to Improve and Enhance Nonresidential Programs,” Proceedings of the 2004 ACEEE Summer Study on Buildings, 4.275 – 4.285. Rogers, Everett. 1983. Communication of Innovations: A CrossCultural Approach. New York, N.Y.: Free Press. Rogers, Everett. 1995. Diffusion of Innovations, 4th Edition. New York, N.Y.: Free Press. Rogers, Everett with F. Floyd Shoemaker. 1971. Communication of Innovations: A Cross-Cultural Approach. New York, N.Y.: Free Press. Rogers, Patricia J., Petroscino, Anthony, Huebner, Tracy A, Hacsi, Timothy A. 2000. “Program Theory Evaluation: Practice, and Problems,” New Directions For Evaluation, Number 87, Fall, Jossey-Bass, pp. 5-13. TecMarket Works Framework Team 2004. The California Evaluation Framework. Southern California Edison Company. Study ID K2033910. Can obtain at: http:// www.calmac.org/search.asp Then enter in “California Evaluation Framework” and can download the 500-page reference document as an Adobe .pdf file. Underhill, Paco. 1999. Why We Buy: The Science of Shopping. New York, N.Y.: Touchstone, Simon & Schuster. Vine, Edward, Luisa Freeman, Joseph Lopes, Martin Adelaar, Barbara Atkinson, Rafael Friedman, and Iris Sulyman. 2003. “Interim Process Evaluation of the Efficient Lighting Initiative: 1999-2001.” Proceedings of the 2003 International Energy Program Evaluation Conference, 17785. Madison, Wisc.: National Energy Program Evaluation Conference, Inc. Weiss, Carol H. 1998. Evaluation: Methods for Studying Programs and Policies. Prentice Hall: Upper Saddle River, NJ. Worthen, Blaine R, James R Sanders and Jody L Fitzpatrick. 1997. Program Evaluation: Alternative Approaches and Practical Guidelines. 2nd Edition. Longman: New York, NY.