Evaluation – the 4 Ms: models, measures, monitoring and methods
•Download as PPTX, PDF•
1 like•426 views
Professor Nick Harding OBE, Chair of New Care Models evaluation oversight group
Day Two, Pop-up University 7, 09.00
1 of 19
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
More Related Content
Evaluation – the 4 Ms: models, measures, monitoring and methods
1. Evaluation - the 4 Ms: models,
measures, monitoring and methods
Chair- Professor Nick Harding OBE - Chair of new care
models evaluation oversight group
2. Reflections on evaluation and the New
Care Models programme
Fraser Battye, Strategy Unit
fraser.battye@nhs.net / 077364 71057
3. This presentation is brief and broad. It covers three inter-
related topics
1: The nature of the NCM programme
2: The implied role for evaluation (especially local evaluation) given 1
3: Our experience of doing 2 (focus on Dudley MCP)
4. The Vanguards are a set of themed experiments
Defines problems, outlines models
Sets models in
train
= multiple local tests of
broadly described care
models
Implied headline
questions for evaluation:
What are these models?
(How) do they work?
How might they be
replicated?
5. The design of the NCM programme has further implications for
evaluation
1: Local practice informs national model
codification / development
Aid the process of description and definition
(e.g. logic models); consider wider
application of local findings
2: The care models are not ‘a thing’; the
result can’t be ‘x works, do x’
Expose ‘active ingredients’ / combinations of
interventions; local context vital
(predecessor efforts / local problems etc)
3: There is a clear expectation of roll-out
(STP process / associated target)
Avoid ‘do Vs don’t do’ pronouncements;
focus on practical improvements and ‘things
to consider if adopting x / y / z’
6. The nature of the programme also creates pressures for local sites
and their evaluators
Significant
problems
High profile
response
Planned roll
out of
response
Demand for
evidence of success
7. But overall, there are multiple opportunities for local evaluators to
add real value to Vanguard sites
Clarify thinking and
programme design: how
will doing x address
problem y?
Aid local programme and
service implementation: a
‘live’ source of evidence
Support the development and
replication of the models: what are
they and how / when do they work?
Improve local evaluation capacity and
culture: healthy agnosticism and learning
(not audits and beatings)
(etc, etc). All framed
by NHSE striking a
sensible balance
between local and
national evaluation
8. We are trying to realise these benefits through our evaluation work
with Dudley’s MCP
Highly multi-disciplinary team:
o Strategy Unit (overall lead, quants expertise, NHS)
o Health Services Management Centre (academic rigour, broader lessons)
o ICF International (research expertise with consultancy focus)
Guided by overall local evaluation strategy and logic models
A close ‘no surprises’ relationship: mutual confidence and respect
9. All Vanguards are complex and multi-component…so where to
start? Dudley’s strategy defines three ‘levels’ of evaluation
Synthesised and
summarised to extract
lessons
(Helped by our wider
work – Modality
evaluation, NIHR
project on MCP)
10. We have (forthcoming) early findings…until these are out, here are
some general reflections on local evaluation in the programme
o Don’t rush to action, take time to understand and focus efforts to maximise value
o Use short outputs, focused on ‘what to do next’ (not method and caveat heavy tomes)
o Keep an eye on policy lessons as well as local practice (Site → Model → Policy)
o Be appropriately modest about what standard of evidence can be produced in this
context
o The NHS has an underdeveloped evaluation culture and an overdeveloped audit /
blame culture. Methodological and personal approaches fundamental to changing this
11. Evaluation of the new care model in
North East Hampshire and Farnham
Our reflections and learning
Paul Gray, Programme Director
North East Hampshire and Farnham Vanguard
12. ❶ Local people being
happier, healthier and
receiving more of the
care they need at
home or in the
community.
❷ Better value for money
for taxpayers,
contributing £23M
towards the £73M gap
we face between the
available resources
and the costs of
delivering care
The changes we are making are designed
to have three key impacts:
❸ Improved staff
satisfaction ability of
health and care
providers to recruit
and retain sufficient
numbers of skilled
staff to meet the needs
of local people
13. We gave considerable
emphasis to the
development of robust
logic models for the
programme and each of
its key elements
14. Eight outcomes
identified by which we
judge our success, and 14
metrics (some existing,
some new) to measure
progress against these
outcomes
16. Patients report a significant improvement in
their quality of life. Particularly feeling less
worried and low.
Their health confidence increases
significantly – feeling better able to manage
their health
Their reported wellbeing improves
significantly. In particular they are happier.
Their experience of using the service increases.
Example outputs
17. Quantitative
Qualitative
Dashboards for metrics relating
to logic models,
Continuous measurement
Meaningful visualisation
Working with the Universities,
R-outcomes, CSU, Local team
Including Patient experience,
patient perception, well being,
Staff experience, job
confidence, job satisfaction
Elements of our evaluation programme
Attribution
Economic
evaluation
Engagement
and learning
Healthy, Happy and at Home
Complex; understanding the
many contributing factors
Easy to value changes in
demand but difficult to
demonstrate cash released until
models replicated at scale.
With replication at pace in mind
Mainstream new sources of
data/dashboards
Quarterly Symposiums; share
evaluation
18. Our learning and reflections
We keep coming back to what we are trying to achieve – and the logic model is a key
foundation to the evaluation
Developing new measures as well as utilising existing – we have found the R-outcomes
family of measures hugely helpful
Mainstreaming new data collection – metrics and processes
The challenge of creating a culture of evaluation
Working out how to rapidly replicate things that work, at scale
The pressure is on now to determine which interventions to fund locally in 2017/18
19. Q&A
Interested in evaluation?
Charles Tallack- Head of Operational Research and Evaluation at NHS England
(charles.tallack@nhs.net)
Laura Freeman- New care models evaluation team at NHS England
(laura.freeman12@nhs.net)