Introduction to the national collaborative outreach programme - Chris Millward, HEFCE

Editor's Notes

1. Bullets of bold headings in section 6, p21 of social mobility index : https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/496103/Social_Mobility_Index.pdf Map (figure 1) at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/496103/Social_Mobility_Index.pdf
2. The National Collaborative Outreach Programme is a four year £60m a year programme consisting of 29 consortia – comprising universities, colleges, schools, local agencies and third sector groups – delivering collaborative outreach in areas where the HE participation of young people is both low overall and lower than expected based on GCSE attainment. It aims to drive rapid progress towards increasing participation of disadvantaged learners from these areas, in line with government goals to: double the proportion of young people from disadvantaged backgrounds in higher education (HE) by 2020 increase by 20 per cent the number of students in HE from ethnic minority groups address the under-representation of young men from disadvantaged backgrounds in HE. It is a nationally designed, locally delivered programme and is sensitive to local context. Consortia will be targeting their work in specific wards within their local areas, with the aim of increasing the numbers of students from disadvantaged backgrounds entering HE. We have been clear that NCOP should be complementary to broader outreach work of institutions and will not replace it. NCOP will build an infrastructure for England and a platform for local, targeted and place based agency. It has collaboration at its core, recognising the need to work in partnership with a broad range of providers and organisations. The programme is evidence led and takes an innovative approach to monitoring and evaluation ensuring consortia are tightly managed and robust evidence of impact is secured.
3. September is nine months into the NCOP and through our monitoring and account management system we have a good overview of how consortia are getting on and what challenges and opportunities they face. Consortia spent some time setting up their partnership and scoping their activities at the start of the programme, however activity has been increasing across the last nine months. As we move into this academic year we expect a very significant increase in engagement with NCOP learners across all consortia Most schools have been receptive to engaging with NCOP and we recognize the hard work by consortia to build these relationships to ensure that as many as possible have been brought on board Consortia have also begun building their links with local partners and initiatives, including with the DfE Opportunity Areas and also other partners, such as the Careers and Enterprise Company There have of course been a few challenges emerging. Many consortia have really strong senior leadership from the host institution and other partners; however a small number are struggling to secure the support they need at the highest level of their institution.   We recognise that for many consortia legal and process issues have delayed recruitment. Agreeing on arrangements for data sharing and distributing finances across the consortia have also proved more time consuming than many anticipated. Where necessary we have worked with consortia to re-profile their finances and ensure that their activities get back on track as soon as possible.
4. This conference is entitled ‘achieving change through partnership’ which is what we want NCOP to do. Later today there will be a panel discussion on what success looks like and how we can maximise success for NCOP. So I wanted to close my presentation outlining what we think a successful NCOP looks like. It would be one that delivering on its targets at both a local and national level – for instance ensuring that sufficient learners are engaged with, and that we meet our goals around doubling the proportion of young people from disadvantaged backgrounds in higher education (HE) by 2020 It would also provide us with clear and robust evidence around the impact of various types of intervention on widening participation in HE And it would show the potential for the infrastructure we have developed to support local agency in the medium term, for example Opportunity areas University-schools collaboration Local skills needs and technical pathways, including degree apprenticeships
5. You will hear from Dr Abigail Diamond, Managing Director at CFE later today – so not looking to steal her thunder about the work they are doing, and key emerging issues and plans for both the formative and impact evaluation. We are hugely thankful to CFE and their partners for the work they have done to date, to you for the efforts put in to developing your activity locally, and to the tracking organisations for their work to support programme level monitoring and evaluation. But there is much more to do and developing a high quality evidence base to support future funding decisions is essential. So a few key messages: Engaging with CFE is a formal requirement of the programme. This means both the activity led by CFE at a national level, but also ensuring that your local plans align with the National Evaluation Framework which CFE circulated to you all earlier this year. See below. We appreciate the enthusiasm and commitment to date and are really pleased that every consortium responded to the survey of NCOP consortia staff and CFE will no doubt say more about this. We know you are talking to your CFE case managers and working hard. We appreciate that the evaluation activity is evolving and we are trialling and testing new approaches, not least the use of RCTS in a HEFCE funded outreach programme. At the moment we don’t have any flagship RCTs in the north and I might challenge those of you in the North to really consider getting involved. We welcome NEACO’s proactivity in supporting other consortia through the development of the NCOP data and evidence practitioner group – and are pleased CFE colleagues were able to join you in your discussions. Do continue to use each other and academic and professional evaluation colleagues in your institutions. CFE have allocated you a case manager but they are not resourced to provide support 24/7 [n.b. we are concerned that CFE are becoming v stretched]. A key issue will be about consistency of data collection amongst consortia, not least in the three tracking systems and what this might mean for the overall project. A really useful meeting with tracking organisations, CFE and HEFCE took place earlier this month and we continue to try and ensure alignment of plans. This is because tracking is compulsory and requires time and resource to enter learner data. We know that getting the legal agreements in place to share data takes time and effort. We know you can only enter data for learners who have supplied the consent to do so. But to actually track learners, it is your consortium that needs to be collecting and entering high quality data in a timely fashion. And it is important that the numbers you return to us in terms of monitoring information are based on data that are robust and auditable. I also know you are all working towards implementing the survey of NCOP beneficiaries which is an important baseline exercise for the impact evaluation. This will be linked to the data in the tracking systems which again highlights the importance of the exercise. GDPR is a challenge we acknowledge and we are very grateful to HEAT for their work to support consortia in navigating this- in particular through their work with the Information Commissioners Office and the workshops they put on, and the resources they have developed. We have a real chance here to do something robust, rigorous and makes a genuine ‘step change’ in the evidence base for WP. By utilising the gold standard of evaluation methodologies, and embedding high quality evaluation in programme design, implementation and review we really do have a chance to address the evidence gap in a meaningful way.   Rationale and background: We know from national data that the sector has made significant progress on access and participation, but that it is increasingly untenable not to be able to demonstrate which interventions (in which contexts, and to which learners) have been instrumental in delivering the genuine progress that has been made, and which could have the most impact. HEFCE is committed to creating a robust, sector-wide evidence base so that investment and support are directed to activity that delivers the greatest benefits to students, the economy and society. The evaluation for the NCOP is being undertaken in this context building on work from 2015 by CFE who developed an outcomes framework for evaluating WP. The overall aim of the national evaluation of NCOP is to assess how successfully the programme is driving rapid progress towards the overall programme objectives to increase participation of disadvantaged learners in the specific local areas where HE participation is low overall and lower than expected given GCSE attainment levels. As well as considering the programme objectives themselves, the evaluation will also need to consider the developing context of education, considering the impact and implications of broader sectoral changes such as: level 3 qualification reform, emerging schools policy, FE area reviews, growth in apprenticeship routes and broader political and structural changes that may occur over the course of the programme; alignment with, and impact on, the broader coverage of outreach activity and investment through access agreements. We use a ‘jigsaw’ to represent the different elements of the evaluation which will need to be brought together. Firstly in terms of the information the OfS Board will be given in make decisions about year 3 and 4 of the programme. But also in measuring longer term impact. The explanation of the elements is below. CFE Research have developed a national evaluation framework which includes a logic chain and indicator bank - designed in such a way that short-term evaluation methods such as the use of surveys will capture baseline and subsequent change in attitudes towards HE, aspirations for the future, knowledge about HE and intentions to apply to HE. Medium to longer-term outcomes that can be measured using data from UCAS and HESA can then be tracked via national administrative datasets once they are available in later years. EVALUATION: ORGANISATION(S) RESPONSIBLE  ELEMENT OF EVALUATION HEFCE Monitoring and account management National quantitative analysis of administrative data sets (e.g. HESA) that will be used to assess target area outcomes in terms of: rates of progression into pathways at Key Stage 5, and HE entry and progression rates. This includes econometric analysis on the return on investment. 2) Externally Commissioned CFE Research + The Behavioural Insights Team (plus LSE and University of Sheffield): Impact evaluation using experimental methods that includes the design of suitable experimental methodologies, including randomised control trials. This is a new and exciting approach for HEFCE working closely with behavioural science experts (who have a lot of kudos in this space) to: understand what aspects of NCOP activity is most effective – what works, how & in what context Cost-effectiveness of different approaches Evidence of differential impact for different individuals. CFE will be conducting baseline participant surveys in Sept/October and are currently finalizing plans for a number of flagship RCTs. CFE Research + Sheffield Hallam University: Formative evaluation and capacity building which will combine: in depth qualitative research (case studies, surveys of consortia etc), support for local consortia in developing and implementing high quality evaluation plans that utilise robust approaches to evaluation, and a systematic review of the evidence emerging from local consortia, in order to inform the formative evaluation itself, and other strands of the NCOP evaluation led by other bodies. CFE Research have run a number of webinars and are currently conducting a consortia survey with approximately 270 responses to date. This will be used to inform a series of case study visits in the autumn. We will publish a 12 month report covering both the CFE evaluations in March 2018. By spring/summer 2018 a number of areas of work will be underway, which will be brought together into a coherent narrative. These include the baseline NCOP participant survey, the design and early implementation of RCTs, work on the quasi-experimental methods and the qualitative work underpinning the trials. The impact evaluation would be complimented by the emerging evidence from the formative evaluation, including initial consortia survey, early field visits, and a synthesis of any local evaluation activity to date. 3) Local NCOP consortia + evaluators: Consortia evaluations (local evaluation) Higher Education Access Tracker, East Midlands Widening Participation Research and Evaluation Partnership, Aimhigher West Midlands: Longitudinal tracking – this is a compulsory element. The vast majority of consortia use HEAT.