Jennifer Mulik - PACT, Kenya
•Download as PPTX, PDF•
0 likes•500 views
This document discusses measuring the performance of capacity development efforts. It presents a theory of change model showing how capacity development activities can strengthen organizations, improve their performance, and ultimately impact communities. The document then describes frameworks for defining and operationalizing performance, including effectiveness, efficiency, relevance, and sustainability. It shows results from applying the Organizational Performance Index tool globally and over time, finding acceptable levels of reliability. The purpose is to have a reliable way to measure the impact of capacity development work.
Report
Share
Jennifer Mulik - PACT, Kenya
- 1. Measuring Capacity Development Performance March 19, 2013 Jennifer Mulik, MA, MPH– Global Director Health jmulik@pactworld.org
- 2. The problem “The monitoring and evaluation of capacity building is as much a challenge now as it was two decades ago.” Simister & Smith, 2010, INTRAC
- 3. Theory of Change for Capacity Development (a)Stronger local organizations; (b)Improve their performance; and (c) Have greater impact in their target communities. Outputs Change in the internal Outcomes systems, skills and Improved external Impact policies of organizations, networks performance of and systems organizations, networks Improved and systems health, environment and/or livelihoods in the communities served by organizations, networks and systems • OCA • OPI • OD Roadmap • Outcome Mapping • Evaluations • ISP Tracking • Most Significant Change
- 4. Defining Performance - Framework IDRC/Universalia History CD Outcomes Mission Culture Framework Incentives Rewards Political Effectiveness Structure Efficiency Leadership Economic Relevance Financial Technological Sustainability Technology Administrative Infrastructure Social / cultural Human resources Stakeholder Program / services Linkages
- 5. Operationalizing Performance Effectiveness: Efficiency: Results Delivery Standards Reach Relevance: Sustainability: Target Population Resources Learning Social Capital
- 6. Benchmarks Level 4 Level 3 Level 2 Level 1 Level 1 Level 2 Level 3 Level 4 Effective The organization is in The organization has The organization has set The organization has Results the process of met over 75% of clearly defined outcome met over 50% of developing outcome outcome level targets level targets for its outcome level targets for level targets for its for its programs & programs & services. its programs & services. programs & services. services.
- 7. Global OPI Scores by Length of Partnership Results 4.0 Social Capital 3.0 Standards 2.0 Resources 1.0 Delivery Less than 1 year Learning Reach 1-3 Years Target Population More than 3 years
- 8. Global OPI Scores Time Series Results 4.0 Social Capital Standards 3.0 2.0 Resources 1.0 Delivery 2011 Learning Reach 2012 Target Population
- 9. Ethiopia OPI Scores by Length of Partnership 4.0 3.5 3.1 3.1 3.0 2.8 2.8 2.6 2.5 2.5 2.5 Less than 1 year 2.5 1-3 Years More than 3 years 2.0 1.8 1.8 1.5 1.5 1.3 1.0 Effectiveness Efficiency Relevance Sustainability
- 10. Organizational Performance Index (OPI) Testing Project The purpose of the OPI Testing Project funded by the Rockefeller Foundation is to check the OPI reliability, i.e. whether the Index generates the same set of scores when it’s reviewed by two people, separately.
- 11. Kappa statistic inter-score reliability Country Pact 1 v Pact 2 Pact 1 v Partner Pact 2 v Partner Vietnam .804 .498 .556 Ethiopia .706 .454 .576 South Sudan .605 .571 .437 Swaziland .504 .495 .583 Nigeria .415 .502 .299 Zimbabwe .357 .724 .587
- 12. Result Our OPI tool and methodology of application meets the reliability standard
Editor's Notes
- Operationalized performance: Effectivenessis about articulating results, measuring, achieving those results.Standards are identifying and ensuring that we hold to a certain qualification (national or international). Efficiency, delivery of programs according to workplan and budget. Reach is ensure that your program is delivered to all the beneficiaries that you promised to (target population). Relevance, ensuring that stakeholders are engaged in project design. Learning is about learning from our successes and challenges and then feeding those learnings back into the delivery of programs. Sustainabilityis about diversifying resource base and building social capital with other actors.
- One of the eightsubdomains.
- 328 local organizations (data points) in 17 countries that applied the OPI in 2012 and they are disaggregated by length of partnership with Pact. We can see that there is a strong correlation betw/ length of partnership and performance improvement. Improvement is faster in some subdomains than in other subdomains. In the delivery one, it improves quite quickly, partners are able to do their work on time and target. But in resource diversification—progress is much slower, so that is an area we might want to focus on more. Many of the organizations have had PEPFAR funding—so they were more focused on targets. Organizations are often so busy with their current projects they don’t have time or resources to pursue other funding. Might be a little data bias as poor performers have shorter relations. But given the size of the sample, there is a strong correlation between performance improvement and length of partnership with Pact. (Different set of orgs in each group. FBOs, CBOs, NGOs, informal networks across 17 countries.)
- Shows time series data. It is plotted for the 40 orgs that used the OPI in 2011 and 2012. This is the beginning of time series data. Shows a correlation between improved performance over time. These are the same organizations that have improved over time.
- Relevant at the country level, seeing increases in efficiency and relevance, but not in sustainability and effectiveness. There is value in having the data at the local level to be able to assess weakness and create QI plan.
- For all countries, and for all comparisons (Pact1 vs Pact 2 vs Partner), the Kappa values are statistically significant and are ranging from moderate (0.4 - 0.6) to substantial (0.6 and higher) inter rater agreement.For three countries (Vietnam, Ethiopia and South Sudan, the inter rater agreement between the Pact raters is higher than the agreement between Pact and PartnerSwaziland and Nigeria have comparable agreement levels among the different comparisonsZimbabwe has the lowest Pact 1 vs Pact 2 inter rater agreement, and has a higher Pact vs partner agreement.< 0 Less than chance agreement0.01–0.20 Slight agreement0.21– 0.40 Fair agreement0.41–0.60 Moderate agreement0.61–0.80 Substantial agreement0.81–0.99 Almost perfect agreement