Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Emerging findings from the impact and formative evaluations of the NCOP - CFE Research

•Download as PPTX, PDF•
•
Higher Education Funding Council for England
Higher Education Funding Council for England

This document provides an update on the formative and impact evaluations of the National Collaborative Outreach Programme (NCOP). It summarizes the following key points: 1) The formative evaluation has included capacity building activities for consortia and a survey of consortia that found new partnerships have been formed and activities would not have been delivered without NCOP support. 2) The impact evaluation participant survey has been finalized and rolled out to over 120,000 students across consortia. Two randomized controlled trials are also confirmed. 3) Work is underway to derive statistical controls for the evaluation through quasi-experimental methods and linking data to secondary sources. 4) Early observations from the consortia

1 of 26
DR. ABIGAIL DIAMOND Managing Director CFE Research Emerging findings from the Impact and Formative evaluations of the NCOP
Outline of the work undertaken in both strands of the NCOP evaluation
Formative evaluation – progress to date ◦ Capacity building activities (two webinars, best practice guidelines on surveying participants). ◦ Gathering together evaluation frameworks/logic chains ahead of the third Webinar. This will inform the meta-analysis of local evaluation evidence. ◦ Consortia survey some headline findings which I will be sharing with you shortly. ◦ Some capacity building and process evaluation work complete.
Impact evaluation – Participant survey update ◦ Participant Survey – baseline survey finalised (and optional Part B question bank) and consortia roll-out has commenced. Baseline surveying should be taking place when your participants engage and or in September. At least 12 consortia have their surveys underway now. The survey will be repeated every September and the baseline findings will inform our 12-month report to HEFCE. ◦ 10 consortia opted for Option A so will be administering the survey themselves, 19 opted for Option B that is the CFE administered survey. At least 17 consortia are either using a paper based version or a combination of paper and online.
Impact evaluation –update continued ◦ Collectively you have so far estimated that we can expect the participant survey to reach at least 120,000 young people. ◦ 9 consortia have requested or made use of a flyer that we have developed to explain the participant survey to schools. Let us know if you want it too. ◦ Quasi-experimental methods: good practice guide prepared for consortia to advise on how to derive comparison groups. Please get in touch with us if you are planning to derive comparison groups and undertake quasi-experimental activity.
Update on the quasi- experimental and experimental work
Update on quasi-experimental work ◦ Guidelines for constructing counterfactual groups have been shared with consortia now, please get in touch if you intend to do this. ◦ Quasi-experimental work isn’t planned across the whole national impact evaluation – our intention will be to derive a statistical control in the second phase of the NCOP to link with other secondary data in the form of HESA data, as it is unlikely that UCAS data will be available – this will only be possible for Year 12-13. ◦ Consortia quasi-experimental work will inform impact of the softer/proxy measures of impact via the participant survey.
Update on experimental work RCTs – two flagship RCTs confirmed. 1) Light-touch text-messaging NCOP outreach activity involving the NEACO consortium. Despite the intervention being light-touch, it will be sustained over time so therefore meets the NCOP requirements of intensive targeting. 2) Intensive outreach activity in the form of e-Mentoring (Brightside) with the SUN consortium. A third RCT will be planned for the next year in relation to a summer school.
National overview and early observations
Early observations – consortia survey findings • We received 325 full responses to the consortia survey. All consortia leads responded. The number of responses per consortium varied between 2 and 32. Respondent roles: 29 6 10 135 143 2 0 20 40 60 80 100 120 140 160 NCOP Lead/Director at Lead Higher Education Institution NCOP Chair Member of the consortium's governing board Employee of the Lead Higher Education Institution Employee of a partner institution or organisation Other
New posts created as part of NCOP in lead organisations Lead organisations • Consortia have created between 2 and 29 new full-time equivalent positions that have been specifically created and or funded by NCOP. • Most posts in lead organisations are new posts: Average is 10.1 FTE new posts, compared with 10.7 FTE posts overall (= 87% new posts) Partner organisations • Consortia have created between 1 and 35 new full-time equivalent positions. • Most posts in partner organisations are also new posts: Average 8.4 FTE new posts, compared with 9.2 FTE posts overall (= 91% new posts)
The roles of those within NCOP consortia include the following. 28 26 25 26 28 9 Management Delivery Evaluation Monitoring Administration Other role
Strategic priorities of NCOP consortia 6.3 6.2 5.7 5.7 5.5 5.4 5.4 5.4 5.4 1 2 3 4 5 6 7 Understand NCOP aims & objectives Understand consortium aims & objectives Consortium delivery plans outline how aims & objectives will be met Consortium aims & objectives align with local area needs NCOP will contribute to achievement of organisation's wider strategic priorities Clear targets have been set to measure progress Consortium aims & objectives align with those of organisation All partners support the consortium vision Other consortium members understand consortium aims & objectives
Ensuring partners understand NCOP strategic priorities Consortia report that better communication and collaboration is needed: “More guidance from HEFCE around where NCOP slots into the WP narrative and complements OFFA” “More support to promote the programme at a national level, particularly to schools and colleges” “Sharing of information and working more collaboratively.” “…continued open dialogue and communication are vitally important to ensure continued engagement both at a consortium and national level.” “Clearer communication between Operational and Strategic groups”
Perceptions of positive impact of NCOP 2.4 2.4 2.3 2.3 2.0 1 2 3 The quality of IAG for pupils in target wards Access to IAG for pupils in target wards The likelihood that able pupils will consider progressing to HE who would have otherwise only considered alternative options Teachers' knowledge and awareness of the options for students in HE Parents' knowledge and awareness of the options for students in HE
Perceptions of other impact of NCOP 6.0 5.7 5.6 5.4 5.4 3.3 1 2 3 4 5 6 7 NCOP adds value to existing activities within my organisation We have developed new partnerships with schools/colleges We have developed new partnerships with institutions in FE & HE We have developed new partnerships with local organisations We would have delivered WP activities without support from NCOP NCOP duplicates existing activities within my organisation
Consortia’s perceptions of the impact of NCOP 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% The consortium is on track to achieve its goals overall NCOP is on track to achieve its goals overall at a national level 1 2 3 4 5 6 7 I don't know It's too early to say
Relationships with consortia and work taking place locally
Relationships with consortia • We have a case management approach established now and hopefully you are all aware of who within the evaluation team is your case manager. The case managers are in regular contact with their consortia. If you are not aware then please do ask a member of the team today or email us at NCOP@cfe.org.uk • The NCOP Jiscmail NCOP@JISCMAIL.AC.UK also seems to be working effectively as a means to seek feedback. • We are also in the process of creating a password protected area of the website for consortia to share materials online.
Across all consortia, there is a total of 345 core members and 504 wider members 109 110 47 20 10 28 9 12 24 72 296 21 11 23 42 15 0 50 100 150 200 250 300 350 HEIs (including Lead Organisation) Further Education Colleges Schools Businesses Local Enterprise Partnerships Local Authorities Community or voluntary organisations Other Core members Wider members
To what extent do you agree with the following statements about partnership working in your consortium? 6.2 6.1 6.0 5.6 5.3 5.3 5.2 5.1 4.9 1 2 3 4 5 6 7 I understand how my role contributes to the work of the consortium I understand how my organisation contributes to the achievement of the consortium's aims and objectives The consortium is comprised of appropriate partner organisations I have a positive and trusting relationship with other partners within my consortium The roles and responsibilities of the partner organisations are clear Tasks are appropriately assigned to organisations within the consortium The workload is shared fairly between the partners, based on the level of funding provided Duplication between partners is avoided Partners effectively communicate with each other
To what extent do you agree with the following statements about the leadership of your consortium? 6.0 5.8 5.7 5.6 5.6 5.5 5.1 1 2 3 4 5 6 7 Senior leaders within my organisation are supportive of my role within NCOP Senior leaders within my organisation understand the goals of NCOP Effective measures are in place to evaluate the impact of the consortium's activities Other colleagues within my organisation are supportive of NCOP activities The Lead Organisation is providing effective leadership The Lead Organisation communicates effectively with consortium members Senior leaders within my organisation prioritise NCOP activities
Work taking place locally–consortia survey findings • All 29 consortia plan to use student interns/ambassadors (or the like) to deliver or support part of NCOP. • Numbers of consortia who envisage that student interns/ambassadors will get involved in: 10 11 5 6 1 Mentoring Delivering/supporting outreach Planning and administration support Evaluation support Other
Next Steps
Next steps • Six consortia case-study visits forthcoming in Oct-Nov 2017. We will be in touch shortly with participating consortia. • Third Webinar on supporting consortia in developing their evaluation frameworks/logic chains for their local evaluations will take place in November 2017. • Just a reminder that consortia need to share their data- sharing agreements for signing. • Please get in touch if you are planning to construct local control / comparison groups.
Any questions?

More Related Content

Emerging findings from the impact and formative evaluations of the NCOP - CFE Research

  • 1. DR. ABIGAIL DIAMOND Managing Director CFE Research Emerging findings from the Impact and Formative evaluations of the NCOP
  • 2. Outline of the work undertaken in both strands of the NCOP evaluation
  • 3. Formative evaluation – progress to date ◦ Capacity building activities (two webinars, best practice guidelines on surveying participants). ◦ Gathering together evaluation frameworks/logic chains ahead of the third Webinar. This will inform the meta-analysis of local evaluation evidence. ◦ Consortia survey some headline findings which I will be sharing with you shortly. ◦ Some capacity building and process evaluation work complete.
  • 4. Impact evaluation – Participant survey update ◦ Participant Survey – baseline survey finalised (and optional Part B question bank) and consortia roll-out has commenced. Baseline surveying should be taking place when your participants engage and or in September. At least 12 consortia have their surveys underway now. The survey will be repeated every September and the baseline findings will inform our 12-month report to HEFCE. ◦ 10 consortia opted for Option A so will be administering the survey themselves, 19 opted for Option B that is the CFE administered survey. At least 17 consortia are either using a paper based version or a combination of paper and online.
  • 5. Impact evaluation –update continued ◦ Collectively you have so far estimated that we can expect the participant survey to reach at least 120,000 young people. ◦ 9 consortia have requested or made use of a flyer that we have developed to explain the participant survey to schools. Let us know if you want it too. ◦ Quasi-experimental methods: good practice guide prepared for consortia to advise on how to derive comparison groups. Please get in touch with us if you are planning to derive comparison groups and undertake quasi-experimental activity.
  • 6. Update on the quasi- experimental and experimental work
  • 7. Update on quasi-experimental work ◦ Guidelines for constructing counterfactual groups have been shared with consortia now, please get in touch if you intend to do this. ◦ Quasi-experimental work isn’t planned across the whole national impact evaluation – our intention will be to derive a statistical control in the second phase of the NCOP to link with other secondary data in the form of HESA data, as it is unlikely that UCAS data will be available – this will only be possible for Year 12-13. ◦ Consortia quasi-experimental work will inform impact of the softer/proxy measures of impact via the participant survey.
  • 8. Update on experimental work RCTs – two flagship RCTs confirmed. 1) Light-touch text-messaging NCOP outreach activity involving the NEACO consortium. Despite the intervention being light-touch, it will be sustained over time so therefore meets the NCOP requirements of intensive targeting. 2) Intensive outreach activity in the form of e-Mentoring (Brightside) with the SUN consortium. A third RCT will be planned for the next year in relation to a summer school.
  • 9. National overview and early observations
  • 10. Early observations – consortia survey findings • We received 325 full responses to the consortia survey. All consortia leads responded. The number of responses per consortium varied between 2 and 32. Respondent roles: 29 6 10 135 143 2 0 20 40 60 80 100 120 140 160 NCOP Lead/Director at Lead Higher Education Institution NCOP Chair Member of the consortium's governing board Employee of the Lead Higher Education Institution Employee of a partner institution or organisation Other
  • 11. New posts created as part of NCOP in lead organisations Lead organisations • Consortia have created between 2 and 29 new full-time equivalent positions that have been specifically created and or funded by NCOP. • Most posts in lead organisations are new posts: Average is 10.1 FTE new posts, compared with 10.7 FTE posts overall (= 87% new posts) Partner organisations • Consortia have created between 1 and 35 new full-time equivalent positions. • Most posts in partner organisations are also new posts: Average 8.4 FTE new posts, compared with 9.2 FTE posts overall (= 91% new posts)
  • 12. The roles of those within NCOP consortia include the following. 28 26 25 26 28 9 Management Delivery Evaluation Monitoring Administration Other role
  • 13. Strategic priorities of NCOP consortia 6.3 6.2 5.7 5.7 5.5 5.4 5.4 5.4 5.4 1 2 3 4 5 6 7 Understand NCOP aims & objectives Understand consortium aims & objectives Consortium delivery plans outline how aims & objectives will be met Consortium aims & objectives align with local area needs NCOP will contribute to achievement of organisation's wider strategic priorities Clear targets have been set to measure progress Consortium aims & objectives align with those of organisation All partners support the consortium vision Other consortium members understand consortium aims & objectives
  • 14. Ensuring partners understand NCOP strategic priorities Consortia report that better communication and collaboration is needed: “More guidance from HEFCE around where NCOP slots into the WP narrative and complements OFFA” “More support to promote the programme at a national level, particularly to schools and colleges” “Sharing of information and working more collaboratively.” “…continued open dialogue and communication are vitally important to ensure continued engagement both at a consortium and national level.” “Clearer communication between Operational and Strategic groups”
  • 15. Perceptions of positive impact of NCOP 2.4 2.4 2.3 2.3 2.0 1 2 3 The quality of IAG for pupils in target wards Access to IAG for pupils in target wards The likelihood that able pupils will consider progressing to HE who would have otherwise only considered alternative options Teachers' knowledge and awareness of the options for students in HE Parents' knowledge and awareness of the options for students in HE
  • 16. Perceptions of other impact of NCOP 6.0 5.7 5.6 5.4 5.4 3.3 1 2 3 4 5 6 7 NCOP adds value to existing activities within my organisation We have developed new partnerships with schools/colleges We have developed new partnerships with institutions in FE & HE We have developed new partnerships with local organisations We would have delivered WP activities without support from NCOP NCOP duplicates existing activities within my organisation
  • 17. Consortia’s perceptions of the impact of NCOP 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% The consortium is on track to achieve its goals overall NCOP is on track to achieve its goals overall at a national level 1 2 3 4 5 6 7 I don't know It's too early to say
  • 18. Relationships with consortia and work taking place locally
  • 19. Relationships with consortia • We have a case management approach established now and hopefully you are all aware of who within the evaluation team is your case manager. The case managers are in regular contact with their consortia. If you are not aware then please do ask a member of the team today or email us at NCOP@cfe.org.uk • The NCOP Jiscmail NCOP@JISCMAIL.AC.UK also seems to be working effectively as a means to seek feedback. • We are also in the process of creating a password protected area of the website for consortia to share materials online.
  • 20. Across all consortia, there is a total of 345 core members and 504 wider members 109 110 47 20 10 28 9 12 24 72 296 21 11 23 42 15 0 50 100 150 200 250 300 350 HEIs (including Lead Organisation) Further Education Colleges Schools Businesses Local Enterprise Partnerships Local Authorities Community or voluntary organisations Other Core members Wider members
  • 21. To what extent do you agree with the following statements about partnership working in your consortium? 6.2 6.1 6.0 5.6 5.3 5.3 5.2 5.1 4.9 1 2 3 4 5 6 7 I understand how my role contributes to the work of the consortium I understand how my organisation contributes to the achievement of the consortium's aims and objectives The consortium is comprised of appropriate partner organisations I have a positive and trusting relationship with other partners within my consortium The roles and responsibilities of the partner organisations are clear Tasks are appropriately assigned to organisations within the consortium The workload is shared fairly between the partners, based on the level of funding provided Duplication between partners is avoided Partners effectively communicate with each other
  • 22. To what extent do you agree with the following statements about the leadership of your consortium? 6.0 5.8 5.7 5.6 5.6 5.5 5.1 1 2 3 4 5 6 7 Senior leaders within my organisation are supportive of my role within NCOP Senior leaders within my organisation understand the goals of NCOP Effective measures are in place to evaluate the impact of the consortium's activities Other colleagues within my organisation are supportive of NCOP activities The Lead Organisation is providing effective leadership The Lead Organisation communicates effectively with consortium members Senior leaders within my organisation prioritise NCOP activities
  • 23. Work taking place locally–consortia survey findings • All 29 consortia plan to use student interns/ambassadors (or the like) to deliver or support part of NCOP. • Numbers of consortia who envisage that student interns/ambassadors will get involved in: 10 11 5 6 1 Mentoring Delivering/supporting outreach Planning and administration support Evaluation support Other
  • 25. Next steps • Six consortia case-study visits forthcoming in Oct-Nov 2017. We will be in touch shortly with participating consortia. • Third Webinar on supporting consortia in developing their evaluation frameworks/logic chains for their local evaluations will take place in November 2017. • Just a reminder that consortia need to share their data- sharing agreements for signing. • Please get in touch if you are planning to construct local control / comparison groups.

Editor's Notes

  1. (unlikely that UCAS will be available following discussions with HEAT at the data tracking meeting)
  2. 11 data-sharing agreements have been signed with consortia. Reminder: CFE need to be included in DSAs, and these need to be in place before the baseline survey is complete.
  3. Q5 (posts in lead org) & Q7 (new posts in lead org). Questions only asked of NCOP leads (all 29 responded). Chart not much use for this average results. Analysis spreadsheet has chart of new posts per consortium.
  4. N.B. data for this Q is missing 1 respondent, so base = 28. Only 25 evaluation roles – an issue? Most common ‘other’ roles are finance (3) and communications (2). ‘Other’ comprises 6 single-responsibility roles and 3 multiple-responsibility roles (e.g. “Finance, Web, Marketing, Coordination of satellite staff“)
  5. We asked consortia to express the extent to which they agree with the following statements about the strategic priorities for their consortia. (Q28) Bases vary: 299 to 323 Not much difference between these priorities – all highly rated. Some variation in levels of agreement: Least variation in agreement for ‘Understanding NCOP aims & objectives’; greatest variation in agreement for ‘Clear targets have been set’
  6. Open question. (Q37) 141 responses. Common themes: Wide range of views, which reflects different capability and capacity of different consortia. More time to set up would have been better. Communication and clarity needed (for many but not all) Challenge is developing partnerships/relationships with consortia members and partners.
  7. Q33 (schools & colleges): Consortia suggest that schools and colleges have limited understanding of how NCOP outreach activity aligns with other WP activity (mean = 4.3 on a 7-point scale). Base = 278-302 This slide: Q34 (NCOP positive impact): Just under half of all responses say “too early to say” for all categories. 2nd most frequent is “Some impact”. Base = 107-196 Q36 (on track): Consortia perceive that local evaluation is more on track than national (5.9 compared with 5.1). Base = 109 and 228
  8. Q33 (schools & colleges): Consortia suggest that schools and colleges have limited understanding of how NCOP outreach activity aligns with other WP activity (mean = 4.3 on a 7-point scale). Base = 278-302 This slide: Q35 (Other impact on consortia): NCOP doesn’t duplicate much, but does add value. Base = 252-287 Q36 (on track): Consortia perceive that local evaluation is more on track than national (5.9 compared with 5.1). Base = 109 and 228
  9. Q36 (whether on track): Consortia perceive that local evaluation is more on track than national (mean = 5.9 compared with 5.1). 1/3 of consortia think “too early to say” about national evaluation being on track (base = 228). ¼ think “too early to say” for local eval (base = 109)
  10. N.B. Chart shows frequency, not %. Q4b. Only asked of consortium leads (29 responses).
  11. Q30. Bases = 286-324.
  12. Q31. Bases = 289-313.
  13. N.B. About min/max number of student interns: Survey limited numeric responses to 2 digits, so 99 was max response. Min = 1. Mean = 43.
  14. Describe what this will entail (we should be able to share who the first round of visits will be with following our case-study meeting on Tuesday and agreement from the consortia) The plan for this Webinar is to share a few examples of well-development evaluation frameworks and ask the relevant consortia to contribute to the Webinar. Date TBC before presentation