WEEK 7: EVALUATION OF HEALTH IMPROVEMENT INITIATIVES

Learning objectives

• Appreciate the importance and the role of evaluation of health promotion initiatives.
• Understand the concepts and methods used in evaluation of health promotion
initiatives.
• Develop critical thinking for the appropriateness of each method depending on the
type and context of the planned intervention.

Contents
Learning objectives ................................................................................................................. 1
Introduction ............................................................................................................................ 1
The application of evaluation at all planning stages ............................................................... 3
Creating a plan for evaluation ................................................................................................. 5
Utilising various methods and tools for evaluation ................................................................. 6
The use of control groups in the evaluation of health promotion programmes ..................... 7
Problems arising regarding evaluation in health promotion ................................................... 8
The detailed plan .................................................................................................................... 9
References ............................................................................................................................ 10

Introduction
The evaluation is one of the most basic and decisive steps when planning any Health Promotion
activity. It offers the opportunity to identify any weaknesses and omissions in all stages of
planning stage, while answering the most critical question, that is if and to what extent the
activity designed and implemented was successful.

The term evaluation in the context of health promotion means the systematic analysis of all
parts of a programme and its results, in order to produce information that can be used by all
those who are interested in its effectiveness and improvement (WHO, 1998). The term
programme in the above definition refers to any activity designed to contribute to the promotion
of health and by extension to the improvement of health. From the above definition, it is clear
Page 1 of 10
that evaluation is not just about results but the whole programming process. In summary, some
of the key the benefits resulting from the implementation of evaluation in this context of each
health promotion activity, are as follows (Ewles & Simnett (1999); Ewles & Simnett (2010);
Tones & Green, (2019); Thorogood & Coombes, 2010):

• It contributes significantly to the knowledge of what is effective or not.

• It helps attune health promotion services to meet the health needs of their users.
• It helps identify areas that need improvement.
• It provides arguments for continuing to fund health promotion activities.
• It provides the ability to make comparisons between different health promotion
activities.
• It helps prevent the repetition of what has already being done.
• It helps to create or improve scientific evidence for health promotion.
• It helps disseminate knowledge about the cost and the merits of health promotion to
both the scientific community and the policymakers.

Evaluation must be a systematic, practical, clear and reliable process. However, it should not
be overlooked that it is not a breeding ground for intense controversy and disagreement, as
evidenced by the plethora of articles and publications regarding health care evaluation. It is a
fact that there are actually more than one hundred forms of evaluation (WHO,1998). It is also
pointed out that especially in the design of health promotion strategies at national, international,
or global level the evaluation is greatly influenced by the values that ultimately represent the
persons involved, but also of those who have funded the activity. Individuals working in this
field should always have as their guide, the values and principles that distinguish health
promotion when evaluating their programmes. Key criteria that are often used to determine the
evaluation are as follows:

Effectiveness - If the purpose and goals that had been set were achieved.

Adequacy - What was achieved and if in the end the activity that was implemented had the
desired results.

Economy - Management and allocation of available resources

Page 2 of 10
Appropriateness - To what extent did the planned activity meet the identified needs and
aspirations?

Acceptance - Whether the activity was implemented in acceptable ways.

Equality - To what extent were all the identified needs treated equally?

The application of evaluation at all planning stages

As it has been pointed out, the definition of evaluation is a methodological tool that concerns
all stages of programming. This is evident from the existence of different types of evaluation
that serve the different needs for evaluation that exist at each stage. There are four general types
of evaluation (Ewles & Simnett, 1999; Ewles & Simnett (2010); Coombes & Thorogood,
2010;Rootman et al., 2001; Hawe et al., 1990; Round et al., 2008; Thorogood & Coombes,
2010).

Process evaluation. This type of evaluation is used in order to evaluate all the stages until the
end of the implementation of the activity. It basically gives answers to the questions how and
why. The answers can arise from the reactions and perceptions of the recipients of the activities
regarding the quality, the appropriateness, and the extent of the implementation of all the
activities. It facilitates in setting the goals, in the flexibility of the activities and allows easier
modification of the interventions, the saving or the better management of the resources and the
immediate confrontation of any problems. For example, the suitability and quality of the
methodological tools used can be checked, as well as the extent to which everything that is
designed is implemented. A variety of methodologies can be used to apply this type of
evaluation, mainly qualitative, such as interviews, daily logs, observations, and content
decoding. Process evaluation can be applied to the pilot phase too, where an activity is still
being planned, and it such case it can be essentially a guide to design. In such cases it is often
called formative evaluation.

Formative evaluation. The main purpose here is to prevent any unforeseen results and to
maximise the outputs of the activity, by identifying problems that require modification. The
analysis of the needs, the testing of specific ideas, messages, even the location where the
intervention or the programme will take place are part of the formative evaluation. Formative

Page 3 of 10
evaluation essentially uses data that is collected throughout the duration of the programme,
giving feedback to the information at all the stages.

Impact evaluation. This type of evaluation concerns the measurement of the effects and impacts
of the implementation of a health promotion activity, namely its immediate effects. This
evaluation is carried out at the end of the implementation of the activity, being essentially the
last stage of its implementation. Through the impact assessment, the extent to which the set
goals have been achieved evaluated, and for this reason, when the targets are set, it is useful to
address the SMART principle. In this type of evaluation both quantitative and qualitative
methods can be used. For example, at the end of a health promotion programme, special
questionnaires may be distributed, or interviews may be conducted to identify changes that
occurred at the end of the programme.

Outcome evaluation. This is used to measure the extent to which the purpose of the planned
activity has been met and the long-term impact following implementation. The long-term
impacts of an intervention programme may include a reduction in the incidence or prevalence
of a disease, a disorder, mortality variations, a steady change in behaviour, improvements in
quality of life, and permanent environmental changes. This type of evaluation is probably more
difficult as it is more complex and therefore more expensive. Quantitative methods are mainly
used to carry out the measurement relate with outcome evaluation. In a health promotion
programme, the evaluation of the result could be implemented after one year in order to
determine to what extent the broader goal has been achieved. Table 1 presents the different
types of evaluation in relation to their possible use, in order to determine whether the purpose
and the objectives have been achieved as well as the success of the interventions that have been
planned.

Summative or final evaluation. Examines whether the intended objectives have been achieved
by answering whether the programme has succeeded. It concerns the formulation of final
conclusions and judgments in relation to the achievement of the objectives of the programme
and the recording of the factors that contributed to the formation of the result (Ewles & Simnett,
1995; Oosterhof, 2008).

Page 4 of 10
 Table 1: Application of the different types of evaluation

Purpose Outcome Evaluation

Goals / Objectives Impact Evaluation

Interventions / Strategies Process Evaluation

Source: Round et al., (2008)

Creating a plan for evaluation

The planning of the evaluation needs to be done in parallel with the design of the whole
programme. This parallel work is beneficial for the whole design, as through feedback and
interaction valuable information is not lost and there is constant updating. There are and have
been various methods and techniques that can help professionals design the evaluation plan.

The following steps when followed are among the most useful frameworks (Naidoo & Wills,
2000).

Description of the programme. In this step, the purpose and objectives of the programme, the
target population, the interventions that will take place as well as the indicators related to the
process and the impact of the interventions are identified and recorded.

Preview of the evaluation. It includes the identification and determining of all the bodies
involved, the purpose of the evaluation is clarified, the critical questions are identified as well
as the available resources required for the evaluation to take place.

Focus on evaluation planning. This is where the evaluation methodology is specified as well
as a detailed description of how the data for the evaluation will be collected, existing evaluation
tools are identified, or new ones are being developed.

Data collection. In this step, the coordinated and organised collection of data that will be used
to make the evaluation is carried out.

Analysis and interpretation of results. Once the necessary data have been collected, the data
are analysed, and specific results and conclusions are obtained.

Dissemination of results. After the analysis and the conclusions, the methodology for the
dissemination and management of the resulting information should be defined. Critical
Page 5 of 10
questions that need to be answered here are how the information collected will be disseminated,
in what form, what reports should be made and to whom they should address.

Utilising various methods and tools for evaluation

A basic condition for carrying out the evaluation is the collection of the necessary data in order
to reach to the conclusions regarding the successful or not achievement of the purpose, the
objectives and the planned interventions. The aim here is to identify and anticipate potential
methods and appropriate tools for data collection. The type of methods that will be used
depends on the purpose of the evaluation and can involve quantitative and qualitative methods
as well as their combination. The choice of methodology each time depends not only on the
purpose of the evaluation but also on factors such as the questions to be answered, the resources
available, the time available, but also on the skills and abilities of the people involved. For
example, if with the evaluation it is desirable to count the participants in an activity,
quantitative methods will be used, while if it is to investigate the experiences of the people who
participated in an activity, qualitative methods are more appropriate (Round et al., 2008).

Qualitative methods. The use of qualitative methods is improved and maximised when more
than one qualitative approach is used at the same time or even when there is a combination
with quantitative approaches. This parallel use has been termed 'triangulation' and is thought
to improve evaluation results (Patton, 2014). As is well known, data collection methods that
fall into this category are:

• In-depth interviews. They are usually structured or not structured and are performed
either in person or by telephone. Through them the views of important individuals or
groups participating can be explored.
• Focus groups. They are useful for exploring attitudes, perceptions, and experiences. At
the same time, they can be used to reveal issues that need further investigation with
quantitative methods but also for the better interpretation of quantitative data.
• Open-ended questionnaires. They include a certain number of questions which are
formulated in such a way that the respondents can answer freely without prior
restriction of choice between defined answers.

Page 6 of 10
 • Narratives. Provides the opportunity to explore the problems and the support factors
regarding the planned activities and interventions by offering some narratives.
• Participatory observation. It is a method that allows the development of relationships
of trust while providing the opportunity for a better understanding of behaviour and
reactions.
• Document analysis. It concerns the analysis of various documents in which the work of
the members of the team that designs and implements the health promotion programme
is recorded so that the degree to which the members have understood and implemented
everything that was planned can be ascertained.

Quantitative methods. They are very useful tools when measuring changes in behaviour,
knowledge, intention, or health levels. At the same time, they allow the generalisation of
conclusions arising from the sample to the entire population.

• Closed-ended questionnaires. These are part of the quantitative methods and consist of
closed questions in which participants choose the answer from a series of answers. They
allow the quantification of the answers in order to analyse the data in statistical terms,
allowing the generalisation of the conclusions.
• Observation data. The collection of this data is done by observing behaviours such as
washing hands after using the toilet in a personal hygiene improvement programme at
school.
• Other quantitative data. Such as the number of participants, indicators such as body
mass index and demographic data.

Regardless the method of data collection the way the sample is selected which will be used in
the evaluation is also important. Sample number, selection method and avoidance of bias are
important principles that should be considered.

The use of control groups in the evaluation of health promotion programmes

The population to which interventions of a health programme are applied is called an
experimental group. The evaluation is called to determine what were the effects and results of
the health promotion interventions for the specific group. For the evaluation to have reliable
results and to determine whether the results are due to the intervention that was planned, a
Page 7 of 10
control group is created at the same time. The experimental group and the control group should
have the same characteristics with the only difference being that in one group the health
promotion activity was applied (experimental) and in the other not (control). The use of control
groups often raises several ethical issues, especially when it comes to studies on new drugs and
vaccines. One way to deal with the issue is to carry out the intervention with a
relative/reasonable delay between the experimental and control group, or even to apply another
form of intervention with the same purpose in order to make a comparison between the two
health promotion interventions that were implemented (McKenzie, 2016). In some cases, it
may not be possible to have a control group at the same time. The next in this scenario is the
comparison group which consists of individuals who have as much in common as possible with
the experimental group, mainly in terms of factors that may potentially affect the outcomes
called confounding factors such as age, gender, education, and socioeconomic status.

Problems arising regarding evaluation in health promotion

At a theoretical level, the problem with evaluation starts from the fact that health is a relevant
concept for which there is not always a commonly accepted definition at the universal level, a
fact that leads to a similar range of definitions for health promotion. In cases where the
biomedical model continues to dominate, health is measured by mortality and morbidity
indices, while in cases where the social model is adopted, socio-economic indicators are
preferred. Respectively for those who follow the educational approach the desired evaluation
indicators are those relating to knowledge and behaviour.

On a practical level one of the most important issues is determining what needs to be measured.
Even though in the context of design the definition of purpose and objectives is necessary as a
means of the evaluation of their achievement, on a practical level this does not always happen.
There are cases where only the objectives are measured for which the process is easier while
others are overlooked that are more difficult to measure, something which violates the rules of
evaluation. A second important problem is the strong belief that the evaluated results are due
to the health promotion activity. Behaviour, attitudes, beliefs, and knowledge regarding health
are dynamic situations which are constantly changing, as is the wider social environment. On
the other hand, health promotion activities are interventions with long-term results. The

Page 8 of 10
combination of these two situations makes it difficult to draw safe conclusions that any change
observed is due solely to the intervention of health promotion, in order to isolate its effects
from the existence of many factors. Another important question is when the evaluation should
be done. Some activities or health promotion programmes have different results which become
apparent in different time periods. In such cases, when planning the evaluation, the need to
carry out the evaluation at different intervals should be recognised. But in order to do this, the
necessary financial adequacy and human resources must be available.

The detailed plan

All the information gathered from the previous stages should be formulated systematically and
in detail with the creation of the detailed plan. The detailed plan describes in writing and in
detail each activity, the required human resources, the cost, and the time of its implementation
(Naidoo & Wills, 2000). As in any other stage of planning a health promotion activity defining
the plan, is a dynamic process that can be feedback with new elements, changes, and
modifications according to the emerging needs. The advantages of developing a detailed plan
serve the following main purposes:

• It helps in the division of labour especially if the activity concerns many recipients or
is done at different levels. It is very important for each participating member to fully
understand what the work is about and within what time frame they need to operate.
• It adds to the reliability of the activity. Demonstrates that all effort is organised and that
there is a commitment to achieve the intended goals and objectives.
• Helps save money, time, and energy. While it may seem like a daunting and tedious
process, it does save money, effort, and time in the long run.
• It is a constant point of reference and progress. With the creation of the detailed plan,
the team of professionals remain firmly oriented towards their aspirations and obtain a
realistic and complete picture of the reality within which they are called to act.

Page 9 of 10
References
Ewles L & Simnett I. Promoting Health. (1995), A practice guide. 3rd Edition. Chapman &
Hall, London, 20-33, 106-112.

Ewles L, & Simnett I, (1999), Promoting Health, a practical guide. Bailliere Tindall.

Ewles L, & Simnett I, (2010), Promoting Health, a practical guide. Bailliere Tindall.

Hawe, P., Degeling, D. and Hall, J. (1990) Evaluating health promotion: A health workers
guide. MacLennan and Petty, Sydney.

McKenzie F.J., Neigel L.B., Smeiltzer L.J., (2016), 4th edition, Planning, Implementing and
Evaluating Health Promotion Programs, A Primer. Pearson Benjamin Cummings

Naidoo, J., & Wills, J., (2000), Health Promotion: foundations for practice. Bailliere Tindall.

Oosterhof A. (2008), Developing and using classroom assessments, 4th edition, Pearson

Patton MQ, (2014), Qualitative research and evaluation methods, 4th edition, Sage
Publications.

Rootman I., Goodstadt M., McQueen D., Potvin L., Springett J., Ziglio E., (eds) (2001),
Evaluating in health promotion: principles and perspectives, WHO(EURO), Copenhagen.

Round R, Marshall B and Horton K, (2008), Planning for effective health promotion
evaluation, Dept of Human Services, Deakin University, Melbourne, Vic.
https://www2.health.vic.gov.au/Api/downloadmedia/%7B7D38F506-FFA1-4728-A423-
1B7DDA901126%7D

Thorogood, M & Coombes, Y, (2010). Evaluating Health Promotion: Practice and Methods.

Tones K. & Green J., (2019), Health Promotion: Planning and Strategies. London, Sage
Publications.

WHO European Working Group on Health Promotion Evaluation & World Health
Organization. Regional Office for Europe. (1998). Health promotion evaluation :
recommendations to policy-makers : report of the WHO European Working Group on Health
Promotion Evaluation. Copenhagen : WHO Regional Office for
Europe. https://apps.who.int/iris/handle/10665/108116

Page 10 of 10