Two Examples of Program Planning, Monitoring and Evaluation

Editor's Notes

1. Now, we will walk through two examples of program planning, monitoring and evaluation. As we present the examples, we will highlight the resources that are available within the toolkit.
2. The first example involves evaluating the impact of a family planning program in a rural village in West Africa.The goal is to increase knowledge of contraceptives, fertility preferences (preferences for limiting and spacing births, modern contraceptive use and parity) and modern contraceptive use.
3. We follow the sequence of questions identical to the ones described in the Toolkit. These questions describe the steps that are used when conducting an evaluation plan.For our particular question of interest, if there is a recent demographic and health survey from the country, data from the rural area could show that there is low contraceptive prevalence, knowledge, and attitudes (as well as limited exposure to media and other potential intervention strategies)The Toolkit outlines examples of other potential population based surveys and their strengths and limitations.For our example, Data from a pre-existing formative research would also show that there is low prevalence of contraceptive use.
4. OR you could conduct a needs assessment in the village of interest by surveying village leaders and community members.In this example, the results would show-A greater need for access to a large range of modern contraceptive methodsThe toolkit describes how to conduct a needs assessment for your question of interest
5. Design a community outreach program where health workers visit homes and address the contraceptive needs of familiesBefore carrying out the full-fledged intervention, the intervention is carried out in a neighborhood of the village as a pilot project. The lessons learned by getting feedback from the health workers and community members is used to improve upon the project design and is expanded to the entire village.
6. In order to track our activities, we can develop and use several tools.A logic model will outline all the steps between inputs and outcomes.The toolkit describes logic models and provides examples as well.We can also create an M&E plan that is helpful as it includes a timeline of program activities.The Toolkit also gives a good outline of an M&E plan and also has some program examples.Furthermore, we could produce a PMP which is useful as this document can be shared with donors and other stakeholders to give them an update on proposed activities. It also has a list of indicators that will be collected and analyzed.The toolkit has a great compendium of good FP indicators.In all these steps and processes, it is important to engage stakeholders in every step of the program planning and implementation as well as monitoring and evaluation.
7. Once our program activities are underway, we monitor the various components of the program.We monitor the use of inputs such as Finances, staff, training materials, contraceptives and transportationSome indicators that can be used to monitor our FP program activities includeThe number of health workers trained to work in the communityThe number of health workers who actually visit the community periodically and distribute contraceptivesThe frequency of meetings between program officers and village leadersMonitoring of inputs and processes ensures that the quality of the program activities are maintained.
8. In order to determine if we are implementing the program as planned, we collect indicators of output and outcome monitoring.Indicators of output monitoring for our FP program of interest include:Number of first visits madeNumber of follow-up visitsTypes and numbers of contraceptives distributedAn indicator for outcome monitoring is change in percentage of contraceptive users over time.The toolkit has some excellent indicator guides for FP programs.Indicators: fertility preferences (preferences for limiting and spacing births, modern contraceptive use and parity)
9. In order to determine if the FP intervention we conducted actually made a difference in contraceptive use, we conduct an outcome evaluation. One possible study design we use is the quasi-experimental design where we choose a control village early on. The villagers in this control village have similar sociodemographic characteristics than those in the intervention village. We collect data on contraceptive use, knowledge and attitudes before the intervention and after the intervention from both villages and determine if contraceptive use significantly went up in the intervention village. The toolkit also has examples of other evaluation designs and resources.Impact evaluationShould be conducted a few years after the end of the programNeed to measure attribution of change to the intervention taking into account all other changes/interventions that might have occurred during the same periodCan use existing data sources, such as large surveys
10. Once the evaluation results show that our FP program was successful, we can make the FP program sustainable by ensuring that the local government continues the community outreach program. With the aid of the same health workers, we can also build the community’s capacity to continue accessing voluntary family planning at local pharmacies and health care centers.Furthermore, the program can be scaled up and expanded to other villages and regions within the country.The toolkit describes how to make programs sustainable and scalable and provides several successful examples as well.
11. Once the program has been implemented and evaluated, it is important to share the findings and best practices with all the stakeholders that were involved including the village leaders and community members of the village, the local government and health department, the funding agency and other higher levels of government and health leadership. The findings can also be shared with others through mass media, research literature and the internet.
12. The second example we will discuss today involves the evaluation of a malaria prevention program in a district in East Africa.The main goal of the program is to ensure that all pregnant women and children <14 years are sleeping under ITNs in all the villages within a district in East Africa.
13. In order to determine the extent of the malaria problem in this district, we review recent population based surveys such as the DHS and MICS. The results will show that there is a high prevalence of malaria and low use of ITNs.We can also conduct a needs assessment by selecting a few homes in a village and interviewing some community members within the selected homes. The results of this needs assessment would show that there is a limited use of ITNs due to lack of knowledge and high cost of ITNs.The toolkit outlines examples of other potential data sources that can be used to identify a health problem within a specified region and also states the strengths and limitations of each of the data sources.
14. In order to determine what interventions could possibly work in this situation, we review other programs, collaborate with more experienced programs and staff, even maybe plan a pilot project.The toolkit has some great resources to learn about existing successful malaria interventions, such as on the Roll Back Malaria website referenced in our toolkit.In this example, we could design an intervention in our district of interest. With the help of handheld GPS devices, we create clusters of 200 households within the district. We visit all homes, identify pregnant women and children <14 years and provide them with vouchers for ITNs. Then, we have program staff do follow-up visits to ensure that the ITN is installed correctly and teach villagers on how to reapply the insecticide.
15. Once we have decided on the intervention we will roll out, we develop a logic model that outlines all the steps between inputs and outcomes. We can also develop a PMP to help guide the activities.It is important to engage stakeholders at every step of the program. These stakeholders includeCommunity members and village leadersLocal government and district officialsExperts in the fieldDonors and policy makersThe toolkit describes a logic model and has a checklist of the important components within a PMP plan for malaria program prevention.
16. In order to determine whether we are implementing the program as planned, we monitor the various components of the program. This also ensure that the quality of the program is maintained.The toolkit has several resources for monitoring malaria program, such as Indicator guides for malaria programs, References on how to conduct routine monitoring as well as impact evaluation.
17. In order to conduct an outcome evaluation for our malaria program, we can use a time series design. Generally, Time-series designs look for changes over time to determine trends. Evaluators observe the intervention group multiple times before and after the intervention and analyze trends before and after.Impact evaluationCan be conducted a few years after the end of the programNeed to measure attribution of change to the intervention taking into account all other changes/interventions that might have occurred during the same periodCan use existing data sources, such as large surveys
18. Once the program has been rolled out successfully, it is important to ensure that the program remains sustainable. This can be done by ensuring that the local government continues the voucher program. Also, the community members are empowered to access ITNs and use them effectively.The program can also be scaled up to other districts and regions within the country thus benefiting a larger audience.The toolkit has descriptions and examples of how programs can be sustained and scaled up to new areas.
19. Thank you for joining us today. We’ve appreciated your participation. We are about to begin a Q&A session for this webinar but I want to let you know about a 3-day online forum that is has begun today. It will1.       build capacity in monitoring and evaluation 2.       advocate the importance of investing in monitoring and evaluation (M&E)3.       and provide a more in-depth overview of MLE’s Measuring Success ToolkitOn your screen, please see the instructions to join the forum. Please join us in the Q&A being moderated by Gretchen.