Educational Measurement: Issues and Practice, 2020
The Standards for Educational and Psychological Testing identify several strands of validity evid... more The Standards for Educational and Psychological Testing identify several strands of validity evidence that may be needed as support for particular interpretations and uses of assessments. Yet assessment validation often does not seem guided by these Standards, with validations lacking a particular strand even when it appears relevant to an assessment. Consequently, the degree to which validity evidence supports the proposed interpretation and use of the assessment may be compromised. Guided by the Standards, this article presents an independent validation of OECD's PISA assessment of mathematical self‐efficacy (MSE) as an instructive example of this issue. OECD identifies MSE as one of a number of “factors” explaining student performance in mathematics, thereby serving the “policy orientation” of PISA. However, this independent validation identifies significant shortcomings in the strands of validity evidence available to support this interpretation and use of the assessment. The article therefore demonstrates how the Standards can guide the planning of a validation to ensure it generates the validity evidence relevant to an interpretive argument, particularly for an international large‐scale assessment such as PISA. The implication is that assessment validation could yet benefit from the Standards as what Zumbo calls “a global force for testing”.
Cognitive Interviewing (CI) provides a method of systematically collecting validity evidence of r... more Cognitive Interviewing (CI) provides a method of systematically collecting validity evidence of response processes for questionnaire items. CI involves a range of techniques for prompting individuals to verbalise their responses to items. One such technique is concurrent verbalisation, as developed in Think Aloud Protocol (TAP). This article investigates the value of the technique for validating questionnaire items administered to young people in international surveys. To date, the literature on TAP has focused on allaying concerns about reactivity – whether response processes are affected by thinking aloud. This article investigates another concern, namely the completeness of concurrent verbalisations – the extent to which respondents verbalise their response processes. An independent, exploratory validation of the PISA assessment of student self-efficacy in mathematics by a small international team of researchers using CI with concurrent verbalisation in four education systems (England, Estonia, Hong Kong and the Netherlands) provided the basis for this investigation. The researchers found that students generally thought aloud in response to each of the items, thereby providing validity evidence of responses processes varying within and between the education systems, but that practical steps could be taken to increase the completeness of concurrent verbalisations in future validations.
In a survey of 24 countries, England, Wales and Northern Ireland had the lowest levels of partici... more In a survey of 24 countries, England, Wales and Northern Ireland had the lowest levels of participation in upper secondary mathematics.
They were the only countries in which fewer than 20% of upper secondary students study maths. This includes all mathematics qualifications at this level, but excludes GCSE retakes.
Scotland does slightly better, with just under half of upper secondary students studying maths. This is still below average.
In the majority of countries surveyed (18 out of 24), at least 50% of upper secondary students study maths.
In eight countries, practically all students in this group study maths. Unsurprisingly, these are countries that make mathematics at this level compulsory for all students.
England, Wales and Northern Ireland also have comparatively low rates of participation in advanced mathematics (equivalent to AS Level). Between 11-15% of upper secondary students study mathematics A level. Countries with similar levels of participation are Germany, Ireland, Russia and Spain. All other countries for which data were available have higher levels of participation.
England, Scotland, Wales and Northern Ireland are four of only six countries that do not require compulsory participation in mathematics at upper secondary for any students. Mathematics is compulsory in all other countries surveyed for at least some students in general or vocational education.
This Commission Staff Working Document highlights the evidence base for the policy proposals outl... more This Commission Staff Working Document highlights the evidence base for the policy proposals outlined in the Commission's “Rethinking Education” Communication on key competences. Its findings are based on analyses, coordinated by the Commission in cooperation with Member States, of best policy practice across the EU and beyond, and on comparative data and current research.
In a previous European project, the assessment of learners’ key competences was identified as ess... more In a previous European project, the assessment of learners’ key competences was identified as essential for this implementation (Gordon et al., 2009). This particular literature review therefore focuses on assessment issues and responses, and complements the more general literature review on approaches to key competence development across Europe.
The development of key competences for lifelong learning has been an important policy imperative ... more The development of key competences for lifelong learning has been an important policy imperative for EU Member States. The European Reference Framework of key competences (2006) built on previous developments by the OECD, UNESCO and Member States themselves. It defined key competences as knowledge, skills and attitudes applied appropriately to contexts. Now most Member States have incorporated key competences, or similarly broad learning outcomes, into their school curriculum frameworks. This is a necessary but insufficient step towards implementation; for the effective development of learners' key competences, assessment must also change. This article focuses on the challenge of assessing cross-curricular key competences in primary and secondary education. It is based on a major study for the European Commission (Gordon, et al., 2009), which drew on information gathered and validated with the help of experts in each of the 27 EU Member States. The study's typology of assessment provides a basis for reviewing some recent developments in Member States. Present challenges and innovative responses are addressed, including ‘unpacking’ key competences, ‘mapping’ them to contexts and ‘accessment’ of their full scope and range. Policy developments are considered in the context of the author's work with the European Commission's Thematic Working Group on the assessment of key competences. The article concludes with considerations for policy and practice.
In England, schooling is compulsory from age 5 to 16. In the final phase of compulsory schooling ... more In England, schooling is compulsory from age 5 to 16. In the final phase of compulsory schooling (14–16 years of age) pupils follow the National Curriculum, with core subjects including English, mathematics, science and Information and Communication Technology and a range of optional subjects. At age 16 most pupils take the General Certificate of Secondary Education (GCSE; ISCED Level 2) in a range of single subjects; a certificate is issued listing the grade achieved in each subject. Optional studies are mainly in general subjects but schools may also offer GCSEs in vocational subjects or other vocationally related qualifications. In conjunction with local Colleges of Further Education (see below) some schools offer pupils aged 14–16 a more strongly vocational alternative to GCSE through the Increased Flexibility Programme.
Educational Measurement: Issues and Practice, 2020
The Standards for Educational and Psychological Testing identify several strands of validity evid... more The Standards for Educational and Psychological Testing identify several strands of validity evidence that may be needed as support for particular interpretations and uses of assessments. Yet assessment validation often does not seem guided by these Standards, with validations lacking a particular strand even when it appears relevant to an assessment. Consequently, the degree to which validity evidence supports the proposed interpretation and use of the assessment may be compromised. Guided by the Standards, this article presents an independent validation of OECD's PISA assessment of mathematical self‐efficacy (MSE) as an instructive example of this issue. OECD identifies MSE as one of a number of “factors” explaining student performance in mathematics, thereby serving the “policy orientation” of PISA. However, this independent validation identifies significant shortcomings in the strands of validity evidence available to support this interpretation and use of the assessment. The article therefore demonstrates how the Standards can guide the planning of a validation to ensure it generates the validity evidence relevant to an interpretive argument, particularly for an international large‐scale assessment such as PISA. The implication is that assessment validation could yet benefit from the Standards as what Zumbo calls “a global force for testing”.
Cognitive Interviewing (CI) provides a method of systematically collecting validity evidence of r... more Cognitive Interviewing (CI) provides a method of systematically collecting validity evidence of response processes for questionnaire items. CI involves a range of techniques for prompting individuals to verbalise their responses to items. One such technique is concurrent verbalisation, as developed in Think Aloud Protocol (TAP). This article investigates the value of the technique for validating questionnaire items administered to young people in international surveys. To date, the literature on TAP has focused on allaying concerns about reactivity – whether response processes are affected by thinking aloud. This article investigates another concern, namely the completeness of concurrent verbalisations – the extent to which respondents verbalise their response processes. An independent, exploratory validation of the PISA assessment of student self-efficacy in mathematics by a small international team of researchers using CI with concurrent verbalisation in four education systems (England, Estonia, Hong Kong and the Netherlands) provided the basis for this investigation. The researchers found that students generally thought aloud in response to each of the items, thereby providing validity evidence of responses processes varying within and between the education systems, but that practical steps could be taken to increase the completeness of concurrent verbalisations in future validations.
In a survey of 24 countries, England, Wales and Northern Ireland had the lowest levels of partici... more In a survey of 24 countries, England, Wales and Northern Ireland had the lowest levels of participation in upper secondary mathematics.
They were the only countries in which fewer than 20% of upper secondary students study maths. This includes all mathematics qualifications at this level, but excludes GCSE retakes.
Scotland does slightly better, with just under half of upper secondary students studying maths. This is still below average.
In the majority of countries surveyed (18 out of 24), at least 50% of upper secondary students study maths.
In eight countries, practically all students in this group study maths. Unsurprisingly, these are countries that make mathematics at this level compulsory for all students.
England, Wales and Northern Ireland also have comparatively low rates of participation in advanced mathematics (equivalent to AS Level). Between 11-15% of upper secondary students study mathematics A level. Countries with similar levels of participation are Germany, Ireland, Russia and Spain. All other countries for which data were available have higher levels of participation.
England, Scotland, Wales and Northern Ireland are four of only six countries that do not require compulsory participation in mathematics at upper secondary for any students. Mathematics is compulsory in all other countries surveyed for at least some students in general or vocational education.
This Commission Staff Working Document highlights the evidence base for the policy proposals outl... more This Commission Staff Working Document highlights the evidence base for the policy proposals outlined in the Commission's “Rethinking Education” Communication on key competences. Its findings are based on analyses, coordinated by the Commission in cooperation with Member States, of best policy practice across the EU and beyond, and on comparative data and current research.
In a previous European project, the assessment of learners’ key competences was identified as ess... more In a previous European project, the assessment of learners’ key competences was identified as essential for this implementation (Gordon et al., 2009). This particular literature review therefore focuses on assessment issues and responses, and complements the more general literature review on approaches to key competence development across Europe.
The development of key competences for lifelong learning has been an important policy imperative ... more The development of key competences for lifelong learning has been an important policy imperative for EU Member States. The European Reference Framework of key competences (2006) built on previous developments by the OECD, UNESCO and Member States themselves. It defined key competences as knowledge, skills and attitudes applied appropriately to contexts. Now most Member States have incorporated key competences, or similarly broad learning outcomes, into their school curriculum frameworks. This is a necessary but insufficient step towards implementation; for the effective development of learners' key competences, assessment must also change. This article focuses on the challenge of assessing cross-curricular key competences in primary and secondary education. It is based on a major study for the European Commission (Gordon, et al., 2009), which drew on information gathered and validated with the help of experts in each of the 27 EU Member States. The study's typology of assessment provides a basis for reviewing some recent developments in Member States. Present challenges and innovative responses are addressed, including ‘unpacking’ key competences, ‘mapping’ them to contexts and ‘accessment’ of their full scope and range. Policy developments are considered in the context of the author's work with the European Commission's Thematic Working Group on the assessment of key competences. The article concludes with considerations for policy and practice.
In England, schooling is compulsory from age 5 to 16. In the final phase of compulsory schooling ... more In England, schooling is compulsory from age 5 to 16. In the final phase of compulsory schooling (14–16 years of age) pupils follow the National Curriculum, with core subjects including English, mathematics, science and Information and Communication Technology and a range of optional subjects. At age 16 most pupils take the General Certificate of Secondary Education (GCSE; ISCED Level 2) in a range of single subjects; a certificate is issued listing the grade achieved in each subject. Optional studies are mainly in general subjects but schools may also offer GCSEs in vocational subjects or other vocationally related qualifications. In conjunction with local Colleges of Further Education (see below) some schools offer pupils aged 14–16 a more strongly vocational alternative to GCSE through the Increased Flexibility Programme.
Uploads
assessment of student self-efficacy in mathematics by a small international team of researchers using CI with concurrent verbalisation in four education systems (England, Estonia, Hong Kong and the Netherlands) provided the basis for this investigation. The researchers found that students generally thought aloud in response to each of the
items, thereby providing validity evidence of responses processes varying within and between the education systems, but that practical steps could be taken to increase the completeness of concurrent verbalisations in future validations.
They were the only countries in which fewer than 20% of upper secondary students study maths. This includes all mathematics qualifications at this level, but excludes GCSE retakes.
Scotland does slightly better, with just under half of upper secondary students studying maths. This is still below average.
In the majority of countries surveyed (18 out of 24), at least 50% of upper secondary students study maths.
In eight countries, practically all students in this group study maths. Unsurprisingly, these are countries that make mathematics at this level compulsory for all students.
England, Wales and Northern Ireland also have comparatively low rates of participation in advanced mathematics (equivalent to AS Level). Between 11-15% of upper secondary students study mathematics A level. Countries with similar levels of participation are Germany, Ireland, Russia and Spain. All other countries for which data were available have higher levels of participation.
England, Scotland, Wales and Northern Ireland are four of only six countries that do not require compulsory participation in mathematics at upper secondary for any students. Mathematics is compulsory in all other countries surveyed for at least some students in general or vocational education.
assessment of student self-efficacy in mathematics by a small international team of researchers using CI with concurrent verbalisation in four education systems (England, Estonia, Hong Kong and the Netherlands) provided the basis for this investigation. The researchers found that students generally thought aloud in response to each of the
items, thereby providing validity evidence of responses processes varying within and between the education systems, but that practical steps could be taken to increase the completeness of concurrent verbalisations in future validations.
They were the only countries in which fewer than 20% of upper secondary students study maths. This includes all mathematics qualifications at this level, but excludes GCSE retakes.
Scotland does slightly better, with just under half of upper secondary students studying maths. This is still below average.
In the majority of countries surveyed (18 out of 24), at least 50% of upper secondary students study maths.
In eight countries, practically all students in this group study maths. Unsurprisingly, these are countries that make mathematics at this level compulsory for all students.
England, Wales and Northern Ireland also have comparatively low rates of participation in advanced mathematics (equivalent to AS Level). Between 11-15% of upper secondary students study mathematics A level. Countries with similar levels of participation are Germany, Ireland, Russia and Spain. All other countries for which data were available have higher levels of participation.
England, Scotland, Wales and Northern Ireland are four of only six countries that do not require compulsory participation in mathematics at upper secondary for any students. Mathematics is compulsory in all other countries surveyed for at least some students in general or vocational education.