Programme for International Student A ssessment

PISA 2012 Results: Creative Problem Solving STUDENTS’ SKILLS IN TACKLING REAL-LIFE PROBLEMS VOLUME V P r ogr am m e f or Int er nat ional St udent As s es s m ent PISA 2012 Results: Creative Problem Solving StudentS’ SkillS in tackling real-life problemS (Volume V) this work is published on the responsibility of the Secretary-general of the oecd. the opinions expressed and arguments employed herein do not necessarily relect the oficial views of the organisation or of the governments of its member countries. this document and any map included herein are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. Please cite this publication as: oecd (2014), PISA 2012 Results: Creative Problem Solving: Students’ Skills in Tackling Real-Life Problems (Volume V), piSa, oecd publishing. http://dx.doi.org/10.1787/9789264208070-en iSbn 978-92-64-20806-3 (print) iSbn 978-92-64-20807-0 (pdf) note by turkey: the information in this document with reference to “cyprus” relates to the southern part of the island. there is no single authority representing both turkish and greek cypriot people on the island. turkey recognises the turkish republic of northern cyprus (trnc). until a lasting and equitable solution is found within the context of the united nations, turkey shall preserve its position concerning the “cyprus issue”. note by all the european union member States of the oecd and the european union: the republic of cyprus is recognised by all members of the united nations with the exception of turkey. the information in this document relates to the area under the effective control of the government of the republic of cyprus. the statistical data for israel are supplied by and under the responsibility of the relevant israeli authorities. the use of such data by the oecd is without prejudice to the status of the golan Heights, east Jerusalem and israeli settlements in the West bank under the terms of international law. Photo credits: © flying colours ltd /getty images © Jacobs Stock photography /kzenon © khoa vu /flickr /getty images © mel curtis /corbis © Shutterstock /kzenon © Simon Jarratt /corbis corrigenda to oecd publications may be found on line at: www.oecd.org/publishing/corrigenda. © oecd 2014 You can copy, download or print oecd content for your own use, and you can include excerpts from oecd publications, databases and multimedia products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable acknowledgement of oecd as source and copyright owner is given. all requests for public or commercial use and translation rights should be submitted to rights@oecd.org. requests for permission to photocopy portions of this material for public or commercial use shall be addressed directly to the copyright clearance center (ccc) at info@copyright.com or the centre français d’exploitation du droit de copie (cfc) at contact@cfcopies.com. Foreword Equipping citizens with the skills necessary to achieve their full potential, participate in an increasingly interconnected global economy, and ultimately convert better jobs into better lives is a central preoccupation of policy makers around the world. results from the oeCd’s recent Survey of adult Skills show that highly skilled adults are twice as likely to be employed and almost three times more likely to earn an above-median salary than poorly skilled adults. in other words, poor skills severely limit people’s access to better-paying and more rewarding jobs. Highly skilled people are also more likely to volunteer, see themselves as actors rather than as objects of political processes, and are more likely to trust others. fairness, integrity and inclusiveness in public policy thus all hinge on the skills of citizens. The ongoing economic crisis has only increased the urgency of investing in the acquisition and development of citizens’ skills – both through the education system and in the workplace. at a time when public budgets are tight and there is little room for further monetary and iscal stimulus, investing in structural reforms to boost productivity, such as education and skills development, is key to future growth. indeed, investment in these areas is essential to support the recovery, as well as to address long-standing issues such as youth unemployment and gender inequality. In this context, more and more countries are looking beyond their own borders for evidence of the most successful and eficient policies and practices. indeed, in a global economy, success is no longer measured against national standards alone, but against the best-performing and most rapidly improving education systems. over the past decade, the oeCd Programme for international Student assessment, PiSa, has become the world’s premier yardstick for evaluating the quality, equity and eficiency of school systems. but the evidence base that PiSa has produced goes well beyond statistical benchmarking. by identifying the characteristics of high-performing education systems PiSa allows governments and educators to identify effective policies that they can then adapt to their local contexts. The results from the PISA 2012 assessment, which was conducted at a time when many of the 65 participating countries and economies were grappling with the effects of the crisis, reveal wide differences in education outcomes, both within and across countries. using the data collected in previous PiSa rounds, we have been able to track the evolution of student performance over time and across subjects. of the 64 countries and economies with comparable data, 40 improved their average performance in at least one subject. top performers such as Shanghai in China or Singapore were able to further extend their lead, while countries like brazil, mexico, tunisia and turkey achieved major improvements from previously low levels of performance. Some education systems have demonstrated that it is possible to secure strong and equitable learning outcomes at the same time as achieving rapid improvements. of the 13 countries and economies that signiicantly improved their mathematics performance between 2003 and 2012, three also show improvements in equity in education during the same period, and another nine improved their performance while maintaining an already high level of equity – proving that countries do not have to sacriice high performance to achieve equity in education opportunities. Nonetheless, PISA 2012 results show wide differences between countries in mathematics performance. the equivalent of almost six years of schooling, 245 score points, separates the highest and lowest average performances Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 3 Foreword of the countries that took part in the PiSa 2012 mathematics assessment. the difference in mathematics performances within countries is even greater, with over 300 points – the equivalent of more than seven years of schooling – often separating the highest- and the lowest-achieving students in a country. Clearly, all countries and economies have excellent students, but few have enabled all students to excel. The report also reveals worrying gender differences in students’ attitudes towards mathematics: even when girls perform as well as boys in mathematics, they report less perseverance, less motivation to learn mathematics, less belief in their own mathematics skills, and higher levels of anxiety about mathematics. While the average girl underperforms in mathematics compared with the average boy, the gender gap in favour of boys is even wider among the highest-achieving students. these indings have serious implications not only for higher education, where young women are already underrepresented in the science, technology, engineering and mathematics ields of study, but also later on, when these young women enter the labour market. this conirms the indings of the oeCd gender Strategy, which identiies some of the factors that create – and widen – the gender gap in education, labour and entrepreneurship. Supporting girls’ positive attitudes towards and investment in learning mathematics will go a long way towards narrowing this gap. PISA 2012 also inds that the highest-performing school systems are those that allocate educational resources more equitably among advantaged and disadvantaged schools and that grant more autonomy over curricula and assessments to individual schools. a belief that all students can achieve at a high level and a willingness to engage all stakeholders in education – including students, through such channels as seeking student feedback on teaching practices – are hallmarks of successful school systems. PISA is not only an accurate indicator of students’ abilities to participate fully in society after compulsory school, but also a powerful tool that countries and economies can use to ine-tune their education policies. there is no single combination of policies and practices that will work for everyone, everywhere. every country has room for improvement, even the top performers. that’s why the oeCd produces this triennial report on the state of education across the globe: to share evidence of the best policies and practices and to offer our timely and targeted support to help countries provide the best education possible for all of their students. With high levels of youth unemployment, rising inequality, a signiicant gender gap, and an urgent need to boost growth in many countries, we have no time to lose. the oeCd stands ready to support policy makers in this challenging and crucial endeavour. Angel Gurría oeCd Secretary-general 4 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v Acknowledgements this report is the product of a collaborative effort between the countries participating in PiSa, the experts and institutions working within the framework of the PiSa Consortium, and the oeCd Secretariat. the report was drafted by andreas Schleicher, francesco avvisati, francesca borgonovi, miyako ikeda, Hiromichi katayama, flore-anne messy, Chiara monticone, guillermo montt, Sophie vayssettes and Pablo Zoido of the oeCd directorate for education and Skills and the directorate for financial affairs, with statistical support from Simone bloem and giannina rech and editorial oversight by marilyn achiron. additional analytical and editorial support was provided by adele atkinson, Jonas bertling, marika boiron, Célia braga-Schich, tracey burns, michael davidson, Cassandra davis, elizabeth del bourgo, John a. dossey, Joachim funke, Samuel greiff, tue Halgreen, ben Jensen, eckhard klieme, andré laboul, Henry levin, barry mcCrae, Juliette mendelovits, tadakazu miki, Christian monseur, Simon normandeau, lorena ortega, mathilde overduin, elodie Pools, dara ramalingam, William H. Schmidt (whose work was supported by the thomas J. alexander fellowship programme), kaye Stacey, lazar Stankov, ross turner, elisabeth villoutreix and allan Wigield. the system-level data collection was conducted by the oeCd neSli (ineS network for the Collection and adjudication of System-level descriptive information on educational Structures, Policies and Practices) team: bonifacio agapin, estelle Herbaut and Jean Yip. volume ii also draws on the analytic work undertaken by Jaap Scheerens and douglas Willms in the context of PiSa 2000. administrative support was provided by Claire Chetcuti, Juliet evans, Jennah Huxley and diana tramontano. the oeCd contracted the australian Council for educational research (aCer) to manage the development of the mathematics, problem solving and inancial literacy frameworks for PiSa 2012. achieve was also contracted by the oeCd to develop the mathematics framework with aCer. the expert group that guided the preparation of the mathematics assessment framework and instruments was chaired by kaye Stacey; Joachim funke chaired the expert group that guided the preparation of the problem-solving assessment framework and instruments; and annamaria lusardi led the expert group that guided the preparation of the inancial literacy assessment framework and instruments. the PiSa assessment instruments and the data underlying the report were prepared by the PiSa Consortium, under the direction of raymond adams at aCer. the development of the report was steered by the PiSa governing board, which is chaired by lorna bertrand (united kingdom), with benő Csapó (Hungary), daniel mcgrath (united States) and ryo Watanabe (Japan) as vice chairs. annex C of the volumes lists the members of the various PiSa bodies, as well as the individual experts and consultants who have contributed to this report and to PiSa in general. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 5 Table of Contents ExEcutivE Summary ..................................................................................................................................................................................................................... 13 rEadEr’S GuidE .................................................................................................................................................................................................................................. 17 What iS PiSa?........................................................................................................................................................................................................................................ 19 CHAPTER 1 aSSESSinG ProblEm-SolvinG SkillS in PiSa 2012 .............................................................................................................. 25 Why PiSa assesses problem-solving competence ........................................................................................................................................................... 26 the PiSa 2012 approach to assessing student performance in problem solving ...................................................................................... 29 • a focus on general cognitive processes involved in solving problems ................................................................................................... 29 • the centrality of interactive problem solving ................................................................................................................................................................ 29 • the PiSa deinition of problem-solving competence ............................................................................................................................................... 30 the PiSa 2012 framework for assessing problem-solving competence........................................................................................................... 31 the design and delivery of the PiSa 2012 computer-based assessment of problem solving ............................................................ 32 • the development of items for the assessment .......................................................................................................................................................... 32 • the structure and delivery of the assessment ................................................................................................................................................................ 32 • the opportunities afforded by computer delivery....................................................................................................................................................... 33 Problem-solving tasks.......................................................................................................................................................................................................................... 34 • general characteristics of static and interactive problem-solving tasks ................................................................................................. 34 • Sample tasks from the PiSa 2012 problem-solving assessment......................................................................................................................... 35 CHAPTER 2 StudEnt PErformancE in ProblEm SolvinG....................................................................................................................... 47 how the PiSa 2012 problem-solving results are reported ....................................................................................................................................... 48 • How the PiSa 2012 problem-solving tests were analysed and scaled ................................................................................................... 48 • How problem-solving proiciency levels are deined in PiSa 2012................................................................................................................ 49 • a proile of PiSa problem-solving questions................................................................................................................................................................. 49 What students can do in problem solving ............................................................................................................................................................................ 51 • average level of proiciency in problem solving .................................................................................................................................................... 52 • Students at the different levels of proiciency in problem solving .................................................................................................................... 56 variation in problem-solving proiciency.............................................................................................................................................................................. 61 • relationship between performance differences and school- and student-level factors............................................................... 64 • Comparing between-school variations ............................................................................................................................................................................. 66 Student performance in problem solving compared with performance in mathematics, reading and science .......................... 67 • Correlation between performance in mathematics, reading and science, and performance in problem solving .......... 67 • Students’ performance in problem solving relative to students with similar mathematics, reading and science skills ........................................................................................................................................................................................................................ 69 • Students’ performance in problem solving at different levels of performance in mathematics ............................................. 70 • the inluence of computer delivery on performance in problem solving ............................................................................................. 73 CHAPTER 3 StudEntS’ StrEnGthS and WEaknESSES in ProblEm SolvinG.......................................................................... 77 framework aspects and relative success of students in each area ...................................................................................................................... 79 • nature of the problem situation ........................................................................................................................................................................................ 79 • Problem-solving processes ...................................................................................................................................................................................................... 82 • Problem contexts and response formats .......................................................................................................................................................................... 88 a grouping of countries by their strengths and weaknesses in problem solving ....................................................................................... 90 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 7 TAble oF conTenTS CHAPTER 4 hoW ProblEm-SolvinG PErformancE variES Within countriES ................................................................. 93 Performance differences unique to problem solving ........................................................................................................................................................ 94 Performance differences across study programmes....................................................................................................................................................... 95 Gender differences in problem solving .................................................................................................................................................................................. 99 • How gender differences in problem-solving performance compare to differences in mathematics, reading and science performance ................................................................................................................................................................................................... 100 • differences in performance patterns across items ............................................................................................................................................. 102 the relationship between socio-economic status, immigrant background and problem-solving performance ............... 104 • Performance differences related to socio-economic status .......................................................................................................................... 104 • Performance patterns among advantaged and disadvantaged students ............................................................................................... 108 • immigrant background and student performance .............................................................................................................................................. 110 how students’ self-reported dispositions towards problem solving relate to performance ........................................................... 111 how problem-solving performance relates to differences in ict use across students ...................................................................... 111 CHAPTER 5 imPlicationS of thE ProblEm-SolvinG aSSESSmEnt for Policy and PracticE ....................... 117 improve assessments to make learning more relevant ............................................................................................................................................. 118 Empower students to solve problems.................................................................................................................................................................................... 120 revise school practices and education policies............................................................................................................................................................. 122 learn from curricular diversity and performance differences in problem solving ............................................................................... 125 reduce gender disparities among top performers........................................................................................................................................................... 126 reduce inequities in education related to socio-economic status ................................................................................................................... 126 ANNEX A PiSa 2012 tEchnical backGround ............................................................................................................................................... 129 annex a1 indices from the student context questionnaires ................................................................................................................................................. 130 annex a2 the PiSa target population, the PiSa samples and the deinition of schools ..................................................................................... 134 annex a3 technical notes on analyses in this volume............................................................................................................................................................ 145 annex a4 Quality assurance................................................................................................................................................................................................................... 149 annex a5 the problem-solving assessment design................................................................................................................................................................... 150 annex a6 technical note on brazil..................................................................................................................................................................................................... 152 ANNEX B PiSa 2012 data ...................................................................................................................................................................................................... 153 annex b1 results for countries and economies .......................................................................................................................................................................... 154 annex b2 results for regions within countries............................................................................................................................................................................. 224 annex b3 list of tables available on line......................................................................................................................................................................................... 243 ANNEX C thE dEvEloPmEnt and imPlEmEntation of PiSa – a collaborativE Effort ................................ 245 8 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v TAble oF conTenTS BOXES box v.1.1. long-term trends in the demand for problem-solving skills ............................................................................................................................... 27 Box V.2.1. How students progress in problem solving .............................................................................................................................................................. 51 box v.2.2. What is a statistically signiicant difference? ........................................................................................................................................................... 53 box v.2.3. interpreting differences in PiSa problem-solving scores: How large a gap? ................................................................................................. 55 box v.2.4. top performers in problem solving ............................................................................................................................................................................ 60 Box V.3.1. How item-level success is reported............................................................................................................................................................................ 78 Box V.5.1. When solutions are taught, problem solving is not learned ............................................................................................................................ 119 Box V.5.2. developing a curriculum for the 21st century in alberta (Canada) .............................................................................................................. 119 Box V.5.3. Problem-solving skills are best developed within meaningful contexts ...................................................................................................... 121 Box V.5.4. What is metacognitive instruction? ......................................................................................................................................................................... 121 Box V.5.5. teaching problem-solving skills through the visual arts ................................................................................................................................... 122 Box V.5.6. developing and assessing problem-solving skills in Singapore ..................................................................................................................... 123 Box V.5.7. developing and assessing problem-solving skills in Japan: Cross-curricular project-based learning ................................................ 124 FIGURES figure v.1.1 trends in the demand for skills: germany, united States and Japan ............................................................................................................... 27 figure v.1.2 main features of the PiSa problem-solving framework........................................................................................................................................ 31 figure v.1.3 the test interface .............................................................................................................................................................................................................. 33 figure v.1.4 mP3 PlaYer: Stimulus information ........................................................................................................................................................................... 35 figure v.1.5 mP3 PlaYer: item 1 ....................................................................................................................................................................................................... 35 figure v.1.6 mP3 PlaYer: item 2 ....................................................................................................................................................................................................... 36 figure v.1.7 mP3 PlaYer: item 3 ....................................................................................................................................................................................................... 36 figure v.1.8 mP3 PlaYer: item 4 ....................................................................................................................................................................................................... 37 figure v.1.9 Climate Control: Stimulus information............................................................................................................................................................ 37 figure v.1.10 Climate Control: item 1........................................................................................................................................................................................ 38 figure v.1.11 Climate Control: item 2........................................................................................................................................................................................ 38 figure v.1.12 tiCketS: Stimulus information .................................................................................................................................................................................... 39 figure v.1.13 tiCketS: item 1 ................................................................................................................................................................................................................ 39 figure v.1.14 tiCketS: item 2 ................................................................................................................................................................................................................ 40 figure v.1.15 tiCketS: item 3 ................................................................................................................................................................................................................ 40 figure v.1.16 traffiC: Stimulus information ................................................................................................................................................................................... 41 figure v.1.17 traffiC: item 1 ............................................................................................................................................................................................................... 41 figure v.1.18 traffiC: item 2 ............................................................................................................................................................................................................... 42 figure v.1.19 traffiC: item 3 ............................................................................................................................................................................................................... 42 figure v.1.20 robot Cleaner: Stimulus information................................................................................................................................................................. 42 figure v.1.21 robot Cleaner: item 1............................................................................................................................................................................................. 43 figure v.1.22 robot Cleaner: item 2............................................................................................................................................................................................. 43 figure v.1.23 robot Cleaner: item 3............................................................................................................................................................................................. 44 figure v.2.1 relationship between questions and student performance on a scale............................................................................................................ 49 figure v.2.2 map of selected problem-solving questions, illustrating the proiciency levels ........................................................................................... 50 figure v.2.3 Comparing countries’ and economies’ performance in problem solving ...................................................................................................... 52 figure v.2.4 Problem-solving performance among participating countries/economies..................................................................................................... 54 figure v.2.5 Summary descriptions of the six levels of proiciency in problem solving .................................................................................................... 57 figure v.2.6 Proiciency in problem solving .................................................................................................................................................................................... 58 figure v.2.7 top performers in problem solving ............................................................................................................................................................................ 61 figure v.2.8 variation in problem-solving performance within countries and economies ............................................................................................... 62 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 9 TAble oF conTenTS 10 figure v.2.9 Performance differences among high- and low-achieving students................................................................................................................. 63 figure v.2.10 average performance in problem solving and variation in performance ....................................................................................................... 64 figure v.2.11 total variation in problem-solving performance and variation between and within schools .................................................................. 65 figure v.2.12 between-school differences in problem-solving performance, mathematics performance and socio-economic status ................. 66 figure v.2.13 relationship among problem-solving, mathematics, reading and science performance ............................................................................. 68 figure v.2.14 variation in problem-solving performance associated with performance in mathematics, reading and science ............................. 68 figure v.2.15 relative performance in problem solving ................................................................................................................................................................ 69 figure v.2.16 expected performance in problem solving, by mathematics performance.................................................................................................... 71 figure v.2.17 Patterns of relative performance in problem solving............................................................................................................................................. 72 figure v.2.18 inluence of computer skills on the ranking of students within countries/economies................................................................................ 73 figure v.2.19 inluence of computer skills on relative performance in problem solving..................................................................................................... 74 figure v.3.1 number of tasks, by framework aspect ..................................................................................................................................................................... 79 figure v.3.2 examples of problem-solving tasks, by nature of the problem .......................................................................................................................... 80 figure v.3.3 differences in countries’/economies’ success on problem-solving tasks, by nature of the problem..................................................... 81 figure v.3.4 relative success on problem-solving tasks, by nature of the problem ............................................................................................................ 82 figure v.3.5 examples of problem-solving tasks, by process...................................................................................................................................................... 83 figure v.3.6 differences in countries’/economies’ success on problem-solving tasks, by process ................................................................................ 85 figure v.3.7 relative success on problem-solving tasks, by process ....................................................................................................................................... 86 figure v.3.8 relative strengths and weaknesses in problem-solving processes.................................................................................................................... 87 figure v.3.9 relative success on problem-solving tasks, by response format ....................................................................................................................... 89 figure v.3.10 Joint analysis of strengths and weaknesses, by nature of the problem and by process .............................................................................. 90 figure v.4.1 Performance variation unique to problem solving ................................................................................................................................................ 95 figure v.4.2 relative performance in problem solving among students in vocational and pre-vocational tracks .................................................... 96 figure v.4.3 relative performance in problem solving, by education track .......................................................................................................................... 97 figure v.4.4 gender differences in problem-solving performance ........................................................................................................................................... 99 figure v.4.5 Proiciency in problem solving among girls and boys ...................................................................................................................................... 100 figure v.4.6 difference between boys and girls in problem-solving, mathematics, reading and science performance ...................................... 101 figure v.4.7 relative performance in problem solving among girls...................................................................................................................................... 102 figure v.4.8 girls’ strengths and weaknesses, by problem-solving process ....................................................................................................................... 103 figure v.4.9a Strength of the relationship between socio-economic status and performance in problem solving, mathematics, reading and science ..................................................................................................................................................................................................... 105 figure v.4.9b Strength of the relationship between socio-economic status and performance in problem solving, between and within schools ...................................................................................................................................................................................... 106 figure v.4.10 difference related to parents’ occupational status in problem-solving, mathematics, reading and science performance.......... 107 figure v.4.11 relative performance in problem solving among students whose parents work in semi-skilled or elementary occupations ......... 108 figure v.4.12 Strengths and weaknesses in problem solving among students with at least one parent working in skilled occupations, by process........................................................................................................................................................................................................................ 109 figure v.4.13 relative performance in problem solving among immigrant students ......................................................................................................... 110 figure v.4.14 difference in problem-solving performance related to the use of computers at home .......................................................................... 112 figure v.4.15 difference in problem-solving performance related to the use of computers at school......................................................................... 113 figure v.4.16 difference in problem-solving, mathematics, reading and science performance related to computer use at home .................... 114 figure v.5.1 employment growth across occupations, grouped by workers’ level of problem-solving skills .......................................................... 118 figure a.3.1 labels used in a two-way table................................................................................................................................................................................. 145 figure a5.1 PiSa 2012 computer-based test design: Problem solving only....................................................................................................................... 150 figure a5.2 PiSa 2012 computer-based test design: Problem solving, mathematics and reading ............................................................................. 150 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v TAble oF conTenTS TABLES table v.a Snapshot of performance in problem solving ......................................................................................................................................................... 15 table a1.1 Student questionnaire rotation design .................................................................................................................................................................... 133 table a2.1 PiSa target populations and samples ...................................................................................................................................................................... 136 table a2.2 exclusions ........................................................................................................................................................................................................................ 138 table a2.3 response rates................................................................................................................................................................................................................ 140 table a2.4 Sample size for performance in mathematics and problem solving ............................................................................................................. 143 table a6.1 Percentage of brazilian students at each proiciency level on the problem-solving scale ..................................................................... 152 table a6.2 mean score, variation and gender differences in student performance in brazil ..................................................................................... 152 table v.2.1 Percentage of students at each proiciency level in problem solving ........................................................................................................... 154 table v.2.2 mean score and variation in student performance in problem solving ....................................................................................................... 156 table v.2.3 top performers in problem solving and other curricular subjects ................................................................................................................. 158 table v.2.4 between- and within-school variation in problem-solving performance .................................................................................................... 159 table v.2.5 Correlation of problem-solving performance with performance in mathematics, reading and science ........................................... 161 table v.2.6 relative performance in problem solving compared with performance in mathematics, reading and science ............................. 163 table v.3.1 Performance in problem solving, by nature of the problem situation .......................................................................................................... 166 table v.3.2 Performance in problem solving, by process ....................................................................................................................................................... 167 table v.3.3 Performance in problem solving, by technology setting................................................................................................................................... 169 table v.3.4 Performance in problem solving, by social focus ............................................................................................................................................... 170 table v.3.5 Performance in problem solving, by response format ....................................................................................................................................... 171 table v.3.6 relative performance on knowledge-acquisition and knowledge-utilisation tasks.................................................................................. 172 table v.4.1 Strength of the relationship between problem-solving and mathematics performance, between and within schools ................. 173 table v.4.2 Performance in problem solving and programme orientation ........................................................................................................................ 175 table v.4.3 differences in problem-solving, mathematics, reading and science performance related to programme orientation ................. 176 table v.4.4 relative performance in problem solving, by programme orientation......................................................................................................... 179 table v.4.6 Percentage of students at each proiciency level in problem solving, by gender ..................................................................................... 180 table v.4.7 mean score and variation in student performance in problem solving, by gender .................................................................................. 182 table v.4.8 differences in problem-solving, mathematics, reading and science performance related to gender ................................................ 185 table v.4.9 relative variation in performance in problem solving, mathematics, reading and science, by gender ............................................ 188 table v.4.10 relative performance in problem solving, by gender ........................................................................................................................................ 190 table v.4.11a Performance on problem-solving tasks, by nature of problem and by gender .......................................................................................... 191 table v.4.11b Performance on problem-solving tasks, by process and by gender .............................................................................................................. 192 table v.4.12 Performance in problem solving, by socio-economic status ........................................................................................................................... 194 table v.4.13 Strength of the relationship between socio-economic status and performance in problem solving, mathematics, reading and science ..................................................................................................................................................................................................... 196 table v.4.14 Strength of the relationship between socio-economic status and performance in problem solving, between and within schools ...................................................................................................................................................................................... 199 table v.4.15 Performance in problem solving and parents’ highest occupational status ................................................................................................ 200 table v.4.16 differences in problem-solving, mathematics, reading and science performance related to parents’ occupational status ........ 201 table v.4.17 relative performance in problem solving, by parents’ occupational status ............................................................................................... 204 table v.4.18a Performance on problem-solving tasks, by nature of problem and by parents’ occupational status.................................................. 205 table v.4.18b Performance on problem-solving tasks, by process and by parents’ occupational status ...................................................................... 206 table v.4.19 Performance in problem solving and immigrant background......................................................................................................................... 208 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 11 TAble oF conTenTS table v.4.20 differences in problem-solving, mathematics, reading and science performance related to immigrant background .................. 210 table v.4.21 relative performance in problem solving, by immigrant background ......................................................................................................... 213 table v.4.22a Performance on problem-solving tasks, by nature of problem and by immigrant background............................................................ 214 table v.4.22b Performance on problem-solving tasks, by process and by immigrant background ................................................................................ 215 table v.4.23 association between problem-solving performance and perseverance/openness to problem solving.............................................. 217 table v.4.24 Performance in problem solving and access to a computer at home ........................................................................................................... 218 table v.4.25 Performance in problem solving and use of a computer at home ................................................................................................................. 219 table v.4.26 Performance in problem solving and use of computers at school ................................................................................................................. 220 table v.4.27 differences in problem-solving, mathematics, reading and science performance related to computer use ................................... 221 table b2.v.1 Percentage of students at each proiciency level in problem solving, by region ...................................................................................... 224 table b2.v.2 mean score and variation in student performance in problem solving, by region ................................................................................... 226 table b2.v.3 relative performance in problem solving compared with performance in mathematics, reading and science, by region.............. 228 table b2.v.4 Percentage of students at each proiciency level in problem solving, by gender and by region.............................................................. 231 table b2.v.5 mean score and variation in student performance in problem solving, by gender and by region ...................................................... 233 table b2.v.6 Performance in problem solving, by socio-economic status and by region .................................................................................................. 236 table b2.v.7 Strength of the relationship between socio-economic status and performance in problem solving, mathematics, reading and science, by region................................................................................................................................................................................. 238 table b2.v.8 Performance in problem solving and use of a computer at home, by region ............................................................................................ 241 table b2.v.9 Performance in problem solving and use of computers at school, by region ............................................................................................ 242 This book has... StatLinks 2 ® A service that delivers Excel files from the printed page! Look for the StatLinks at the bottom left-hand corner of the tables or graphs in this book. To download the matching Excel® spreadsheet, just type the link into your Internet browser, starting with the http://dx.doi.org prefix. If you’re reading the PDF e-book edition, and your PC is connected to the Internet, simply click on the link. You’ll find StatLinks appearing in more OECD books. 12 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v Executive Summary in modern societies, all of life is problem solving. Changes in society, the environment, and in technology mean that the content of applicable knowledge evolves rapidly. adapting, learning, daring to try out new things and always being ready to learn from mistakes are among the keys to resilience and success in an unpredictable world. few workers today, whether in manual or knowledge-based occupations, use repetitive actions to perform their job tasks. What’s more, as the new Survey of adult Skills (PiaaC) inds, one in ten workers is confronted every day with more complex problems that require at least 30 minutes to solve. Complex problem-solving skills are particularly in demand in fast-growing, highly skilled managerial, professional and technical occupations. are today’s 15-year-olds acquiring the problem-solving skills needed in the 21st century? this volume reports the results from the PiSa 2012 assessment of problem solving, which was administered, on computer, to about 85 000 students in 44 countries and economies. Students in Singapore and Korea, followed by students in Japan, score higher in problem solving than students in all other participating countries and economies. four more east asian partner economies score between 530 and 540 points on the PiSa problem-solving scale: macao-China (with a mean score of 540 points), Hong kong-China (540 points), Shanghai-China (536 points) and Chinese taipei (534 points); and Canada, australia, finland, england (united kingdom), estonia, france, the netherlands, italy, the Czech republic, germany, the united States and belgium all score above the oeCd average, but below the former group of countries. Across OECD countries, 11.4% of 15-year-old students are top performers in problem solving. top performers attain proiciency level 5 or 6 in problem solving, meaning that they can systematically explore a complex problem scenario, devise multi-step solutions that take into account all constraints, and adjust their plans in light of the feedback received. in Singapore, korea and Japan, more than one in ive students achieve this level, while more than one in six students perform at level 5 or above in Hong kong-China (19.3%), Chinese taipei and Shanghai-China (18.3%), Canada (17.5%) and australia (16.7%). by contrast, in montenegro, malaysia, Colombia, uruguay, bulgaria and brazil, fewer than 2% of students perform at level 5 or 6; and all of these countries perform well below the oeCd average. On average across OECD countries, about one in ive students is able to solve only straightforward problems – if any – provided that they refer to familiar situations. by contrast, fewer than one in ten students in Japan, korea, macao-China and Singapore are low-achievers in problem solving. In Australia, Brazil, Italy, Japan, Korea, Macao-China, Serbia, England (United Kingdom) and the United States, students perform signiicantly better in problem solving, on average, than students in other countries who show similar performance in mathematics, reading and science. in australia, england (united kingdom) and the united States, this is particularly true among strong and top performers in mathematics; in italy, Japan and korea, this is particularly true among moderate and low performers in mathematics. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 13 execuTIve SummAry Students in Hong Kong-China, Korea, Macao-China, Shanghai-China, Singapore and Chinese Taipei perform strongest on problems that require understanding, formulating or representing new knowledge, compared to other types of problems. many of the best-performing countries and economies in problem solving are those with better-than-expected performance on tasks related to acquiring knowledge, such as “exploring and understanding” and “representing and formulating” tasks, and relatively weaker performance on tasks involving only the use of knowledge, such as “planning and executing” tasks that do not require substantial understanding or representation of the problem situation. meanwhile, students in brazil, ireland, korea and the united States perform strongest on interactive problems (those that require the student to uncover some of the information needed to solve the problem) compared to static problems (those that have all information disclosed at the outset). In Malaysia, Shanghai-China and Turkey, more than one in eight students attend a vocational study programme, and these students show signiicantly better performance in problem solving, on average, than students with comparable performance in mathematics, reading and science but who are in general study programmes. this inding can be interpreted in two ways. on the one hand, the curriculum and teaching practices in these vocational programmes may equip students better for tackling complex, real-life problems in contexts that they do not usually encounter at school. on the other hand, better-than-expected performance in problem solving may be an indication that in these programmes, students’ ability to solve problems is not nurtured within the core academic subjects. Boys outperform girls in problem solving in 23 countries/economies, girls outperform boys in ive countries/ economies, and in 16 countries/economies, there is no signiicant difference in average performance between boys and girls. gender differences are often larger among top performers. on average across oeCd countries, there are three topperforming boys for every two top-performing girls in problem solving. in Croatia, italy and the Slovak republic, boys are as likely as girls to be low-achievers, but are more than twice as likely to be top performers as girls. in no country or economy are there more girls than boys among the top performers in problem solving. girls appear to be stronger in performing the “planning and executing” tasks that measure how students use knowledge, compared to other tasks; and weaker in performing the more abstract “representing and formulating” tasks, which relate to how students acquire knowledge. The impact of socio-economic status on problem-solving performance is weaker than it is on performance in mathematics, reading or science. Students from disadvantaged backgrounds are more likely to score higher than expected in problem solving than in mathematics, perhaps because after-school opportunities to exercise their skills in problem solving arise in diverse social and cultural contexts. Still, the quality of schools matters: unequal access to high-quality schools means that, on average, disadvantaged students score below advantaged students in all subjects assessed, including problem solving. 14 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v execuTIve SummAry • Table V.A • SnAPShoT oF PerFormAnce In Problem SolvIng Countries/economies with mean score/share of top performers / relative performance /solution rate above the oeCd average Countries/economies with share of low achievers below the oeCd average Countries/economies with mean score/share of top performers /relative performance /share of low achievers/solution rate not statistically different from the oeCd average Countries/economies with mean score/share of top performers /relative performance /solution rate below the oeCd average Countries/economies with a share of low achievers above the oeCd average Performance in problem solving oEcd average relative performance in problem solving, Performance in problem solving, by process Solution Solution rate on tasks rate on tasks measuring measuring acquisition utilisation of knowledge of knowledge mean score in PiSa 2012 Share of low achievers (below level 2) Share of top performers (level 5 or 6) gender difference (boys - girls) compared with students around the world with similar performance in mathematics, reading and science mean score % % Score dif. Score dif. 500 21.4 11.4 7 -7 Performance in problem solving, by nature of the problem situation Solution rate on items referring to a static problem situation Solution rate on items referring to an interactive problem situation Percent correct Percent correct Percent correct Percent correct 45.5 46.4 47.1 43.8 Singapore 562 8.0 29.3 9 2 62.0 55.4 59.8 57.5 Korea 561 6.9 27.6 13 14 62.8 54.5 58.9 57.7 Japan 552 7.1 22.3 19 11 59.1 56.3 58.7 55.9 Macao-China 540 7.5 16.6 10 8 58.3 51.3 57.0 51.7 Hong Kong-China 540 10.4 19.3 13 -16 57.7 51.1 56.1 52.2 Shanghai-China 536 10.6 18.3 25 -51 56.9 49.8 56.7 50.3 Chinese Taipei 534 11.6 18.3 12 -9 56.9 50.1 56.3 50.1 Canada 526 14.7 17.5 5 0 52.6 52.1 52.7 50.5 Australia 523 15.5 16.7 2 7 52.3 51.5 52.8 49.9 Finland 523 14.3 15.0 -6 -8 50.2 51.0 52.1 47.7 England (United Kingdom) 517 16.4 14.3 6 8 49.6 49.1 49.5 47.9 Estonia 515 15.1 11.8 5 -15 46.8 49.5 49.7 45.6 France 511 16.5 12.0 5 5 49.6 49.4 50.3 47.6 Netherlands 511 18.5 13.6 5 -16 48.2 49.7 50.4 46.5 Italy 510 16.4 10.8 18 10 49.5 48.0 49.5 46.8 Czech Republic 509 18.4 11.9 8 1 45.0 46.9 46.2 44.4 Germany 509 19.2 12.8 7 -12 47.5 49.5 49.4 46.3 United States 508 18.2 11.6 3 10 46.5 47.1 46.6 45.9 Belgium 508 20.8 14.4 8 -10 47.0 47.5 48.3 45.4 Austria 506 18.4 10.9 12 -5 45.7 47.4 48.3 43.0 Norway 503 21.3 13.1 -3 1 47.7 48.1 49.4 44.5 Ireland 498 20.3 9.4 5 -18 44.6 45.5 44.4 44.6 Denmark 497 20.4 8.7 10 -11 44.2 48.1 47.9 42.3 Portugal 494 20.6 7.4 16 -3 41.6 45.7 44.0 42.0 Sweden 491 23.5 8.8 -4 -1 45.2 44.6 47.7 41.6 Russian Federation 489 22.1 7.3 8 -4 40.4 43.8 43.8 39.7 Slovak Republic 483 26.1 7.8 22 -5 40.5 43.2 44.2 38.8 Poland 481 25.7 6.9 0 -44 41.3 43.7 44.1 39.7 Spain 477 28.5 7.8 2 -20 40.0 42.3 42.3 39.8 Slovenia 476 28.5 6.6 -4 -34 37.8 42.3 42.9 36.7 Serbia 473 28.5 4.7 15 11 37.7 40.7 40.3 36.8 Croatia 466 32.3 4.7 15 -22 35.2 40.5 39.3 35.6 Hungary 459 35.0 5.6 3 -34 35.2 37.6 38.2 33.9 Turkey 454 35.8 2.2 15 -14 32.8 36.0 35.8 32.7 Israel 454 38.9 8.8 6 -28 38.7 37.0 39.7 35.6 Chile 448 38.3 2.1 13 1 30.9 35.2 34.9 31.8 Cyprus* 445 40.4 3.6 -9 -12 33.6 34.8 37.0 31.4 Brazil 428 47.3 1.8 22 7 28.0 32.0 29.8 29.1 Malaysia 422 50.5 0.9 8 -14 29.1 29.3 30.1 27.4 United Arab Emirates 411 54.8 2.5 -26 -43 28.4 29.0 29.9 27.1 Montenegro 407 56.8 0.8 -6 -24 25.6 30.0 30.3 25.1 Uruguay 403 57.9 1.2 11 -27 24.8 27.9 27.5 24.8 Bulgaria 402 56.7 1.6 -17 -54 23.7 26.7 28.4 22.3 Colombia 399 61.5 1.2 31 -7 21.8 27.7 26.3 23.7 Note: Countries/economies in which the performance difference between boys and girls is statistically signiicant are marked in bold. Countries and economies are ranked in descending order of the mean score in problem solving in PISA 2012. * See notes in the reader’s guide. Source: oeCd, PiSa 2012 database, tables v.2.1, v.2.2, v.2.6, v.3.1, v.3.6 and v.4.7. 12 http://dx.doi.org/10.1787/888933003649 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 15 Reader’s Guide Data underlying the igures the data referred to in this volume are presented in annex b and, in greater detail, including some additional tables, on the PiSa website (www.pisa.oecd.org). four symbols are used to denote missing data: a the category does not apply in the country concerned. data are therefore missing. c there are too few observations or no observation to provide reliable estimates (i.e. there are fewer than 30 students or fewer than 5 schools with valid data). m data are not available. these data were not submitted by the country or were collected but subsequently removed from the publication for technical reasons. w data have been withdrawn or have not been collected at the request of the country concerned. Country coverage the PiSa publications (PISA 2012 Results) feature data on 65 countries and economies, including all 34 oeCd countries and 31 partner countries and economies (see map in the section What is PISA?). this volume in particular contains data on 44 countries and economies that participated in the assessment of problem solving, including 28 oeCd countries and 16 partner countries and economies. the statistical data for israel are supplied by and under the responsibility of the relevant israeli authorities. the use of such data by the oeCd is without prejudice to the status of the golan Heights, east Jerusalem and israeli settlements in the West bank under the terms of international law. two notes were added to the statistical data related to Cyprus: 1. note by turkey: the information in this document with reference to “Cyprus” relates to the southern part of the island. there is no single authority representing both turkish and greek Cypriot people on the island. turkey recognises the turkish republic of northern Cyprus (trnC). until a lasting and equitable solution is found within the context of the united nations, turkey shall preserve its position concerning the “Cyprus issue”. 2. note by all the european union member States of the oeCd and the european union: the republic of Cyprus is recognised by all members of the united nations with the exception of turkey. the information in this document relates to the area under the effective control of the government of the republic of Cyprus. Calculating international averages an oeCd average corresponding to the arithmetic mean of the respective country estimates was calculated for most indicators presented in this report. the oeCd average is used to compare performance across school systems. in the case of some countries, data may not be available for speciic indicators, or speciic categories may not apply. readers should, therefore, keep in mind that the term “oeCd average” refers to the oeCd countries included in the respective comparisons. Rounding igures because of rounding, some igures in tables may not exactly add up to the totals. totals, differences and averages are always calculated on the basis of exact numbers and are rounded only after calculation. all standard errors in this publication have been rounded to one or two decimal places. Where the value 0.0 or 0.00 is shown, this does not imply that the standard error is zero, but that it is smaller than 0.05 or 0.005, respectively. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 17 reAder’S guIde Reporting student data the report uses “15-year-olds” as shorthand for the PiSa target population. PiSa covers students who are aged between 15 years 3 months and 16 years 2 months at the time of assessment and who are enrolled in school and have completed at least 6 years of formal schooling, regardless of the type of institution in which they are enrolled and of whether they are in full-time or part-time education, of whether they attend academic or vocational programmes, and of whether they attend public or private schools or foreign schools within the country. Focusing on statistically signiicant differences this volume discusses only statistically signiicant differences or changes. these are denoted in darker colours in igures and in bold font in tables. See annex a3 for further information. Categorising student performance this report uses a shorthand to describe students’ levels of proiciency in the subjects assessed by PiSa: top performers are those students proicient at level 5 or 6 of the assessment. Strong performers are those students proicient at level 4 of the assessment. moderate performers are those students proicient at level 2 or 3 of the assessment. lowest performers are those students proicient at or below level 1 of the assessment. Abbreviations used in this report eSCS PiSa index of economic, social and cultural status PPP Purchasing power parity gdP gross domestic product S.d. Standard deviation iSCed international Standard Classiication of education S.e. Standard error iSCo Stem Science, technology, engineering and mathematics international Standard Classiication of occupations Further documentation for further information on the PiSa assessment instruments and the methods used in PiSa, see the PISA 2012 Technical Report (oeCd, forthcoming). the reader should note that there are gaps in the numbering of tables because some tables appear on line only and are not included in this publication. to consult the set of web-only data tables, visit the PiSa website (www.pisa.oecd.org). this report uses the oeCd Statlinks service. below each table and chart is a url leading to a corresponding exceltm workbook containing the underlying data. these urls are stable and will remain unchanged over time. in addition, readers of the e-books will be able to click directly on these links and the workbook will open in a separate window, if their internet browser is open and running. 18 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v What is PISA? “What is important for citizens to know and be able to do?” that is the question that underlies the triennial survey of 15-year-old students around the world known as the Programme for international Student assessment (PiSa). PiSa assesses the extent to which students near the end of compulsory education have acquired key knowledge and skills that are essential for full participation in modern societies. the assessment, which focuses on mathematics, reading, science and problem solving, does not just ascertain whether students can reproduce knowledge; it also examines how well students can extrapolate from what they have learned and apply that knowledge in unfamiliar settings, both in and outside of school. this approach relects the fact that modern economies reward individuals not for what they know, but for what they can do with what they know. PiSa is an ongoing programme that offers insights for education policy and practice, and that helps monitor trends in students’ acquisition of knowledge and skills across countries and economies and in different demographic subgroups within each country. PiSa results reveal what is possible in education by showing what students in the highest-performing and most rapidly improving school systems can do. the indings allow policy makers around the world to gauge the knowledge and skills of students in their own countries in comparison with those in other countries, set policy targets against measurable goals achieved by other school systems, and learn from policies and practices applied elsewhere. While PiSa cannot identify cause-and-effect relationships between policies/practices and student outcomes, it can show educators, policy makers and the interested public how education systems are similar and different – and what that means for students. A test the whole world can take PiSa is now used as an assessment tool in many regions around the world. it was implemented in 43 countries and economies in the irst assessment (32 in 2000 and 11 in 2002), 41 in the second assessment (2003), 57 in the third assessment (2006) and 75 in the fourth assessment (65 in 2009 and 10 in 2010). So far, 65 countries and economies have participated in PiSa 2012. in addition to oeCd member countries, the survey has been or is being conducted in: East, South and Southeast Asia: Himachal Pradesh-india, Hong kong-China, indonesia, macao-China, malaysia, Shanghai-China, Singapore, Chinese taipei, tamil nadu-india, thailand and viet nam. Central, Mediterranean and Eastern Europe, and Central Asia: albania, azerbaijan, bulgaria, Croatia, georgia, kazakhstan, kyrgyzstan, latvia, liechtenstein, lithuania, the former Yugoslav republic of macedonia, malta, moldova, montenegro, romania, the russian federation and Serbia. The Middle East: Jordan, Qatar and the united arab emirates. Central and South America: argentina, brazil, Colombia, Costa rica, netherlands-antilles, Panama, Peru, trinidad and tobago, uruguay and miranda-venezuela. Africa: mauritius and tunisia. decisions about the scope and nature of the PiSa assessments and the background information to be collected are made by participating countries based on recommendations from leading experts. Considerable efforts and resources are devoted to achieving cultural and linguistic breadth and balance in assessment materials. Since the design and translation of the test, as well as sampling and data collection, are subject to strict quality controls, PiSa indings are considered to be highly valid and reliable. ... Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 19 whAT IS PISA? map of PISA countries and economies oEcd countries australia austria belgium Canada Chile Czech republic denmark estonia finland france germany greece Hungary iceland ireland israel italy Japan korea luxembourg mexico netherlands new Zealand norway Poland Portugal Slovak republic Slovenia Spain Sweden Switzerland turkey united kingdom united States Partner countries and economies in PiSa 2012 Partner countries and economies in previous cycles albania argentina brazil bulgaria Colombia Costa rica Croatia Cyprus1, 2 Hong kong-China indonesia Jordan kazakhstan latvia liechtenstein lithuania macao-China malaysia azerbaijan georgia Himachal Pradesh-india kyrgyzstan former Yugoslav republic of macedonia malta mauritius miranda-venezuela moldova Panama tamil nadu-india trinidad and tobago montenegro Peru Qatar romania russian federation Serbia Shanghai-China Singapore Chinese taipei thailand tunisia united arab emirates uruguay viet nam 1. note by turkey: the information in this document with reference to “Cyprus” relates to the southern part of the island. there is no single authority representing both turkish and greek Cypriot people on the island. turkey recognises the turkish republic of northern Cyprus (trnC). until a lasting and equitable solution is found within the context of the united nations, turkey shall preserve its position concerning the “Cyprus issue”. 2. note by all the european union member States of the oeCd and the european union: the republic of Cyprus is recognised by all members of the united nations with the exception of turkey. the information in this document relates to the area under the effective control of the government of the republic of Cyprus. PiSa’s unique features include its: • policy orientation, which links data on student learning outcomes with data on students’ backgrounds and attitudes towards learning and on key factors that shape their learning, in and outside of school, in order to highlight differences in performance and identify the characteristics of students, schools and school systems that perform well; • innovative concept of “literacy”, which refers to students’ capacity to apply knowledge and skills in key subjects, and to analyse, reason and communicate effectively as they identify, interpret and solve problems in a variety of situations; • relevance to lifelong learning, as PiSa asks students to report on their motivation to learn, their beliefs about themselves, and their learning strategies; • regularity, which enables countries and economies to monitor their progress in meeting key learning objectives; and • breadth of coverage, which, in PiSa 2012, encompasses the 34 oeCd member countries and 31 partner countries and economies. 20 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v whAT IS PISA? Key features of PISA 2012 The content • the PiSa 2012 survey focused on mathematics, with reading, science and problem solving as minor areas of assessment. for the first time, PiSa 2012 also included an assessment of the financial literacy of young people, which was optional for countries and economies. • PiSa assesses not only whether students can reproduce knowledge, but also whether they can extrapolate from what they have learned and apply their knowledge in new situations. it emphasises the mastery of processes, the understanding of concepts, and the ability to function in various types of situations. The students • around 510 000 students completed the assessment in 2012, representing about 28 million 15-year-olds in the schools of the 65 participating countries and economies. The assessment • Paper-based tests were used, with assessments lasting a total of two hours for each student. in a range of countries and economies, an additional 40 minutes were devoted to the computer-based assessment of mathematics, reading and problem solving. • test items were a mixture of multiple-choice items and questions requiring students to construct their own responses. the items were organised in groups based on a passage setting out a real-life situation. a total of about 390 minutes of test items were covered, with different students taking different combinations of test items. • Students answered a background questionnaire, which took 30 minutes to complete, that sought information about themselves, their homes and their school and learning experiences. School principals were given a questionnaire, to complete in 30 minutes, that covered the school system and the learning environment. in some countries and economies, optional questionnaires were distributed to parents, who were asked to provide information on their perceptions of and involvement in their child’s school, their support for learning in the home, and their child’s career expectations, particularly in mathematics. Countries and economies could choose two other optional questionnaires for students: one asked students about their familiarity with and use of information and communication technologies, and the second sought information about their education to date, including any interruptions in their schooling and whether and how they are preparing for a future career. who Are The PISA STudenTS? differences between countries in the nature and extent of pre-primary education and care, in the age of entry into formal schooling, in the structure of the school system, and in the prevalence of grade repetition mean that school grade levels are often not good indicators of where students are in their cognitive development. to better compare student performance internationally, PiSa targets a speciic age of students. PiSa students are aged between 15 years 3 months and 16 years 2 months at the time of the assessment, and have completed at least 6 years of formal schooling. they can be enrolled in any type of institution, participate in full-time or part-time education, in academic or vocational programmes, and attend public or private schools or foreign schools within the country or economy. (for an operational deinition of this target population, see annex a2.) using this age across countries and over time allows PiSa to compare consistently the knowledge and skills of individuals born in the same year who are still in school at age 15, despite the diversity of their education histories in and outside of school. the population of participating students is deined by strict technical standards, as are the students who are excluded from participating (see annex a2). the overall exclusion rate within a country was required to be below 5% to ensure that, under reasonable assumptions, any distortions in national mean scores would remain within plus or minus 5 score points, i.e. typically within the order of magnitude of 2 standard errors of sampling. exclusion could take place either through the schools that participated or the students who participated within schools (see annex a2, tables a2.1 and a2.2). there are several reasons why a school or a student could be excluded from PiSa. Schools might be excluded because they are situated in remote regions and are inaccessible, because they are very small, or because of organisational or operational factors that precluded participation. Students might be excluded because of intellectual disability or limited proiciency in the language of the assessment. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 21 whAT IS PISA? in 28 out of the 65 countries and economies participating in PiSa 2012, the percentage of school-level exclusions amounted to less than 1%; it was less than 4% in all countries and economies. When the exclusion of students who met the internationally established exclusion criteria is also taken into account, the exclusion rates increase slightly. However, the overall exclusion rate remains below 2% in 30 participating countries and economies, below 5% in 57 participating countries and economies, and below 7% in all countries except luxembourg (8.4%). in 11 out of the 34 oeCd countries, the percentage of school-level exclusions amounted to less than 1% and was less than 3% in 31 oeCd countries. When student exclusions within schools were also taken into account, there were 11 oeCd countries below 2% and 26 oeCd countries below 5%. (for more detailed information about the restrictions on the level of exclusions in PiSa 2012, see annex a2.) whAT KIndS oF reSulTS doeS The TeST ProvIde? the PiSa assessment provides three main types of outcomes: • basic indicators that provide a baseline profile of students’ knowledge and skills; • indicators that show how skills relate to important demographic, social, economic and educational variables; and • indicators on trends that show changes in student performance and in the relationships between student-level and school-level variables and outcomes. although indicators can highlight important issues, they do not provide direct answers to policy questions. to respond to this, PiSa also developed a policy-oriented analysis plan that uses the indicators as a basis for policy discussion. where cAn you FInd The reSulTS? this is the ifth of six volumes that presents the results from PiSa 2012. it begins by providing the rationale for assessing problem-solving competence in PiSa, and introduces the innovative features of the 2012 assessment. Chapter 2 introduces the problem-solving performance scale and proiciency levels, examines student performance in problem solving, and discusses the relationship between problem-solving performance and performance in mathematics, reading and science. Chapter 3 provides a nuanced look at student performance in problem solving by focusing on students’ strengths and weaknesses in performing certain types of tasks. Chapter 4 looks at differences in problem-solving performance related to education tracks and to students’ gender, socio-economic status and immigrant background. it also examines students’ behaviours and attitudes related to problem solving, and students’ familiarity with information and communication technology. the volume concludes with a chapter that discusses the implications of the PiSa problem-solving assessment for education policy and practice. the other ive volumes cover the following issues: Volume I, What Students Know and Can Do: Student Performance in Mathematics, Reading and Science, summarises the performance of students in PiSa 2012. it describes how performance is deined, measured and reported, and then provides results from the assessment, showing what students are able to do in mathematics. after a summary of mathematics performance, it examines the ways in which this performance varies on subscales representing different aspects of mathematics literacy. given that any comparison of the outcomes of education systems needs to take into consideration countries’ social and economic circumstances, and the resources they devote to education, the volume also presents the results within countries’ economic and social contexts. in addition, the volume examines the relationship between the frequency and intensity of students’ exposure to subject content in school, what is known as “opportunity to learn”, and student performance. the volume concludes with a description of student results in reading and science. trends in student performance in mathematics between 2003 and 2012, in reading between 2000 and 2012, and in science between 2006 and 2012 are examined when comparable data are available. throughout the volume, case studies examine in greater detail the policy reforms adopted by countries that have improved in PiSa. Volume II, Excellence through Equity: Giving Every Student the Chance to Succeed, deines and measures equity in education and analyses how equity in education has evolved across countries and economies between PiSa 2003 and PiSa 2012. the volume examines the relationship between student performance and socio-economic status, and describes how other individual student characteristics, such as immigrant background and family structure, and school characteristics, such as school location, are associated with socio-economic status and performance. the volume also reveals differences in how equitably countries allocate resources and opportunities to learn to schools with different 22 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v whAT IS PISA? socio-economic proiles. Case studies, examining the policy reforms adopted by countries that have improved in PiSa, are highlighted throughout the volume. Volume III, Ready to Learn: Students’ Engagement, Drive and Self-Beliefs, explores students’ engagement with and at school, their drive and motivation to succeed, and the beliefs they hold about themselves as mathematics learners. the volume identiies the students who are at particular risk of having low levels of engagement in, and holding negative dispositions towards, school in general and mathematics in particular, and how engagement, drive, motivation and self-beliefs are related to mathematics performance. the volume identiies the roles schools can play in shaping the well-being of students and the role parents can play in promoting their children’s engagement with and dispositions towards learning. Changes in students’ engagement, drive, motivation and self-beliefs between 2003 and 2012, and how those dispositions have changed during the period among particular subgroups of students, notably socio-economically advantaged and disadvantaged students, boys and girls, and students at different levels of mathematics proiciency, are examined when comparable data are available. throughout the volume, case studies examine in greater detail the policy reforms adopted by countries that have improved in PiSa. Volume IV, What Makes Schools Successful? Resources, Policies and Practices, examines how student performance is associated with various characteristics of individual schools and of concerned school systems. it discusses how 15-yearold students are selected and grouped into different schools, programmes, and education levels, and how human, inancial, educational and time resources are allocated to different schools. the volume also examines how school systems balance autonomy with collaboration, and how the learning environment in school shapes student performance. trends in these variables between 2003 and 2012 are examined when comparable data are available, and case studies, examining the policy reforms adopted by countries that have improved in PiSa, are presented throughout the volume. Volume VI, Students and Money: Financial Literacy Skills for the 21st Century, examines 15-year-old students’ performance in inancial literacy in the 18 countries and economies that participated in this optional assessment. it also discusses the relationship of inancial literacy to students’ and their families’ background and to students’ mathematics and reading skills. the volume also explores students’ access to money and their experience with inancial matters. in addition, it provides an overview of the current status of inancial education in schools and highlights relevant case studies. the frameworks for assessing mathematics, reading and science in 2012 are described in PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy (oeCd, 2013). they are also summarised in this volume. technical annexes at the end of this report describe how questionnaire indices were constructed and discuss sampling issues, quality-assurance procedures, the reliability of coding, and the process followed for developing the assessment instruments. many of the issues covered in the technical annexes are elaborated in greater detail in the PISA 2012 Technical Report (oeCd, forthcoming). all data tables referred to in the analysis are included at the end of the respective volume in annex b1, and a set of additional data tables is available on line (www.pisa.oecd.org). a reader’s guide is also provided in each volume to aid in interpreting the tables and igures that accompany the report. data from regions within the participating countries are included in annex b2. References OECD (forthcoming), PISA 2012 Technical Report, PiSa, oeCd Publishing. OECD (2013), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, PiSa, oeCd Publishing. http://dx.doi.org/10.1787/9789264190511-en Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 23 1 Assessing Problem-Solving Skills in PISA 2012 This chapter introduces the PISA 2012 assessment of problem solving. It provides the rationale for assessing problem-solving competence in PISA, and introduces the innovative features of the 2012 assessment. The framework for the assessment is presented, and sample items are discussed. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 25 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 Non vitae, sed scholae discimus [too often,] we don’t learn for life, but only for the lecture room Seneca, Ad Lucilium, c. 65 ad in daniel defoe’s novel, robinson Crusoe is stranded on a desert island. He first needs to secure food for himself. to solve this problem, he re-invents agriculture and tames a flock of wild goats. then, he returns to his true longing: “my desire to venture over for the main[land] increased, rather than decreased, as the means for it seemed impossible. this at length put me upon thinking whether it was not possible to make myself a canoe […], even without tools, […] of the trunk of a great tree. this i not only thought possible, but easy” (defoe, 1919). Problems are situations with no obvious solution, and solving problems requires thinking and learning in action. Problem solving “involves initiating, usually on the basis of hunches or feelings, experimental interactions with the environment to clarify the nature of a problem and potential solutions”, so that the problem-solver “can learn more […] about the nature of the problem and the effectiveness of their strategies”, “modify their behaviour and launch a further round of experimental interactions with the environment” (raven, 2000, p. 54). (robinson Crusoe’s first strategy to escape from his island in a canoe fails, for, as he explains, “my thoughts were so intent upon my voyage over the sea in [the canoe], that i never once considered how i should get it off the land”.) Just like robinson Crusoe, we solve small problems every day: “my mobile phone has stopped working; how do i tell my friends that i’m running late for our appointment?”; “this meeting room is so cold; are these the switches to control the air conditioning?”; “i don’t speak the local language, and my connecting flight leaves from a different airport in the same city. i just hope i can get there in time.” in modern societies, all of life is problem solving. Changes in society, the environment and in technology mean that the content of applicable knowledge evolves rapidly. today’s 15-year-olds are the robinson Crusoes of a future that remains largely unknown to us. adapting, learning, daring to try out new things, and always being ready to learn from mistakes are among the keys to resilience and success in an unpredictable world. this chapter begins with a discussion of the rationale for including a separate assessment of problem solving in PiSa. it then introduces what is new and distinctive about the PiSa 2012 approach to assessing problem solving, and describes the main dimensions covered in the problem-solving framework. the chapter concludes by presenting the test interface and sample items from the PiSa computer-based assessment of problem solving. why PISA ASSeSSeS Problem-SolvIng comPeTence today’s workplaces demand people who can solve non-routine problems. few workers, whether in manual or knowledgebased occupations, use repetitive actions to perform their job tasks. the Survey of adult Skills (PiaaC), for instance, measured how often workers are faced with a new or difficult situation in their jobs that requires some thinking before taking action (oeCd, 2013a). on average across countries, a large majority of workers are confronted at least once per week in their job with simple problems (those requiring less than 30 minutes to find a solution). meanwhile, one in ten workers is confronted every day with more complex problems that require at least 30 minutes to find a good solution. Complex problem-solving skills are particularly in demand in fast-growing, highly skilled managerial, professional and technical occupations. one possible explanation for this shift to non-routine tasks in the workplace is that, as computers and computerised machines were introduced in greater numbers, workers were needed less often to perform routine manual or analytical tasks. instead, they were required to deal with the unexpected and the unfamiliar, and to bring the best out of the machines and computers working alongside them (autor, levy and murnane, 2003). there is clear evidence of this change in the demand for skills in germany, Japan and the united States (box v.1.1 and figure v.1.1). acknowledging these changes, the emphasis in education is shifting too, from equipping students with highly codified, routine skills to empowering them to confront and overcome complex, non-routine cognitive challenges. indeed, the skills that are easiest to teach and test are also the skills that are easiest to digitise, automate and outsource. for students to be prepared for tomorrow’s world, they need more than mastery of a repertoire of facts and procedures; students need to become lifelong learners who can handle unfamiliar situations where the effect of their interventions is not predictable. When asked to solve problems for which they have no ready-made strategy, they need to be able to think flexibly and creatively about how to overcome the barriers that stand in the way of a solution. 26 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 • figure v.1.1 • Trends in the demand for skills: germany, united States and Japan non-routine analytic routine cognitive non-routine interactive routine manual non-routine manual Germany United States Japan average change in task inputs across education-industry cells, in percentiles of the 1960 task distribution Percentage-point change in mean task inputs across occupations relative to 1960 30 74 1.00 25 70 0.75 20 66 0.50 15 62 0.25 10 58 0.00 5 54 -0.25 0 50 -0.50 -5 46 -0.75 -10 42 -1.00 -15 38 -1.25 -20 34 -1.50 Percentage-point change in aggregate task inputs relative to 1979 1979 1986 1992 1999 1960 1970 1980 1990 2000 2009 1960 1970 1980 1990 2000 2005 Note: the scale of the vertical axis is not directly comparable across countries due to different methodologies. Sources: germany: based on Spitz-oener (2003), table 3; united States: based on autor and Price (2013), table 1; Japan: based on ikenaga and kambayashi (2010), figure 1. 1 2 http://dx.doi.org/10.1787/888933003554 box v.1.1. long-term trends in the demand for problem-solving skills trends in the demand for skills can be inferred from aggregate measures of workers’ job requirements, repeated over time. figure v.1.1 presents the observed evolution of job requirements in three major oeCd countries: germany, Japan and the united States. across all three countries, there has been a marked increase in the demand for problem-solving skills. according to autor, levy and murnane (2003), job requirements can be classiied into ive major skill categories. a irst distinction is between “routine” and “non-routine” tasks and skills. “routine” skills correspond to tasks that “require methodical repetition of an unwavering procedure” (p. 1283), i.e. those tasks in which machines and computers can fairly easily replace human beings. they can be cognitive (such as data entry) or manual (such as repetitive production). “non-routine” skills correspond to tasks that require tacit knowledge and are only imperfectly described in terms of a set of rules. a further distinction, within non-routine skills, is between “manual” and “abstract” skills. manual non-routine tasks, such as preparing a meal, demand situational adaptability, visual and language recognition, and interaction with other people. they are dificult to automate, but from the human perspective, they are straightforward, requiring primarily abilities that are hardwired into humans’ evolutionary endowments. abstract tasks are based on the processing of information and require problem-solving skills, intuition, persuasion and creativity. among abstract skills, there are “analytic” and “interpersonal” skills: “interpersonal” tasks (such as managing teams or persuading potential buyers) require complex interpersonal communication, while “analytic” tasks require the transformation of data and information. ... Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 27 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 Problem-solving competence is an essential component of the skills required to perform interpersonal and nonroutine analytic tasks successfully. in both kinds of tasks, workers need to think about how to engage with the situation, monitor the effect of their actions systematically, and adjust to feedback. in germany, a representative sample of workers has consistently reported on job requirements over more than 20 years, providing direct evidence of an increase in the use of non-routine analytic and interactive skills in the workplace during the 1980s and 1990s (Spitz-oener, 2006). this increase has been accompanied by declines in the importance of routine skills, both analytic (such as skills needed for bookkeeping) and manual (such as sorting). in the united States and Japan, the evolution of aggregate skill requirements has been estimated by matching job titles reported to the national population census with precise job descriptions in the dictionary of occupational titles, for the united States (autor, levy and murnane, 2003; autor and Price, 2013), or in the career matrix constructed by the institute for labour Policy and training in Japan (ikenaga and kambayashi, 2010). Changes in the occupational shares for precisely deined occupations can then be translated into changes in the economy’s skill requirements. this methodology has yielded strikingly similar results as found in germany, over a longer period of time, i.e. since 1960. While problem-solving skills are increasingly needed in today’s economies, the ability to adapt to new circumstances, learn throughout life, and turn knowledge into action has always been important for full participation in society. the best educators have always aimed to foster the skills needed to perform non-routine tasks, i.e. to teach for life, not for school. recent evidence confirms that the generic skills examined in a problem-solving assessment such as PiSa are strongly associated with academic success and are distinct from reasoning or intelligence, as traditionally measured (Wüstenberg et al., 2012; greiff et al., 2013a; funke and frensch, 2007). in addition, other research strongly supports the view that good teachers and schools can develop students’ overall problem-solving skills through and in addition to their competence in regular curricular subjects (Csapó and funke, forthcoming). Yet all too often teachers find that while their students may excel on routine exercises (those that they have already seen and practiced), they fail to solve problems that are unlike those they have previously encountered. Clearly, mastering the simple steps that are required to reach a solution is not enough. Students need to be able to know not only what to do, but also when to do it; and they need to feel motivated and interested. mayer (1998) summarises these three components of successful problem solving in all domains as “skill”, “metaskill” and “will”. the problem-solving assessment in PiSa 2012 focuses on students’ general reasoning skills, their ability to regulate problem-solving processes, and their willingness to do so, by confronting students with problems that do not require expert knowledge to solve. individual problem solving was assessed as a separate domain for the first time in 2003 (oeCd, 2005). the advances in our understanding of problem solving since then and the opportunities afforded by computers to improve the assessment of problem-solving skills led to the inclusion of problem solving as a core component of the PiSa 2012 assessment.1 the regular assessments of mathematics, reading and science in PiSa all include problem-solving tasks that assess students’ ability to use their curricular knowledge to meet real-life challenges. indeed, problem-solving competence need not be developed independently of expertise in curricular subjects; in fact, the literature on the development of general cognitive abilities suggests that content-based methods can be equally effective and may be preferable: “if you teach the specifics with abstraction in mind, the general is learned, but if you try to teach the general directly, the specifics are often not learned” (adey et al., 2007, p. 92). While schools are not the only environment in which problem-solving competence is nurtured, high-quality education, in a wide range of subjects, certainly helps to develop these skills. Progressive teaching methods, like problem-based learning, inquiry-based learning, and individual and group project work, can be used to foster deep understanding and prepare students to apply their knowledge in novel situations. good teaching promotes self-regulated learning and metacognition – particularly knowledge about when and how to use certain strategies for learning or for problem solving – and develops cognitive dispositions that underpin problem solving. it prepares students to reason effectively in unfamiliar situations, and to fill gaps in their knowledge by observing, exploring and interacting with unknown systems. 28 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 all teachers can create opportunities to develop problem-solving competence. for instance, thinking habits, such as careful observation, awareness about one’s working process, or critical self-evaluation, can be instilled in students as they learn techniques in the visual arts (Winner et al., 2013; see box v.5.5) – and indeed, in any other subject in the school curriculum. because the skills and dispositions that underpin successful problem solving in real life are not specific to particular subjects, students who learn to master them in several curricular contexts will be better equipped to use them outside of school as well. thus, by measuring 15-year-olds’ problem-solving skills, PiSa provides evidence about the comparative success of education systems in equipping students for success in life, evidence that can, in turn, inform education policies and practices. The PISA 2012 APProAch To ASSeSSIng STudenT PerFormAnce In Problem SolvIng the problem-solving assessment in PiSa 2012 focuses on general cognitive processes involved in problem solving, rather than on the ability to solve problems in particular school subjects. given the advances in understanding the cognitive processes involved in problem solving and the possibility of using computer-based simulated scenarios,2 the assessment also assigns a central place to so-called interactive problems. A focus on general cognitive processes involved in solving problems research findings suggest that outside of artificial laboratory conditions, the situation in which a problem is embedded influences the strategies used to solve it (kotovsky, Hayes and Simon, 1985; funke, 1992). in real life, highly proficient problem-solvers in one context may act as novices when confronted with problems outside of their field of expertise. in the context of a particular subject, trade or occupation, experts will use domain-specific knowledge and strategies to solve the problems. meanwhile, those who solve problems efficiently, even when they arise outside of their field of expertise, have mastered general reasoning skills, can apply those skills where appropriate, and are motivated to engage with unfamiliar problems. a glimpse at some of the names of problem-solving units included in the PiSa assessment reveals the typical contexts included in the assessment: technology devices (e.g. REMOTE CONTROL, CLOCK, LIGHTS), unfamiliar spaces (e.g. TRAFFIC, LOST), food or drink (e.g. VITAMINS, DRINK MACHINE), etc. these contexts refer to situations that students may encounter outside of school as part of their everyday experience. While including authentic scenarios related to real-life problems, the PiSa 2012 problem-solving assessment avoids the need for specific, curricular knowledge as much as possible. texts are short and use plain language. if arithmetic operations are required, calculators are embedded in the scenario. in contrast, when problem-solving tasks are incorporated in the assessment of the regular PiSa domains of mathematics, reading and science, expert knowledge in these areas is needed in order to reach a solution. by using authentic problem situations, the assessment also reduces the influence of affective factors related to school, or to specific subjects, on results. the student’s familiarity with the context may still influence how he or she approaches the problem. because the assessment tasks are embedded in real-life settings, in practice some students may be more familiar than others with the concrete contexts. However, since a wide range of contexts is included in the different assessment units, the degree of familiarity with the setting will vary, so that prior knowledge will not systematically influence performance. in addition, applying prior knowledge is never sufficient for solving new problems, even in familiar situations. The centrality of interactive problem solving in most problems that students practice in class or when studying for an exam, the information needed to solve the problem is provided at the outset. by contrast, solving real-life problems often requires identifying the pieces of information available in the environment/context that would be most useful for solving the problem. Problems that require students to uncover useful information by exploring the problem situation are called interactive problems. these kinds of problems are encountered when using unfamiliar everyday devices, such as a new mobile phone, home appliance or vending machine. outside of technological contexts, similar situations also arise in social interactions and in other settings as varied as cultivating plants or raising animals. a majority of PiSa 2012 problemsolving tasks correspond to interactive problems. the prevalence of interactive problems in the PiSa 2012 assessment reflects their importance in the real world. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 29 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 the inclusion of interactive tasks, made possible by computer delivery, represents the main innovation over the PiSa 2003 assessment of problem solving. PiSa 2012 therefore provides a broader measure of problem-solving competency than previous assessments of problem solving. The PISA deinition of problem-solving competence PiSa 2012 defines problem-solving competence as: …an individual’s capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. It includes the willingness to engage with such situations in order to achieve one’s potential as a constructive and reflective citizen. the PiSa 2012 framework publication (oeCd, 2013b) discusses the definition in full. among the key elements: … an individual’s capacity to engage in cognitive processing to understand and resolve problem situations… Problem solving begins with recognising that a problem situation exists and establishing an understanding of the nature of the situation. it requires the solver to identify the specific problem(s) to be solved, plan and carry out a solution, and monitor and evaluate progress throughout the activity. the verbs engage, understand and resolve underline that, in addition to the explicit responses to items, the assessment measures individuals’ progress towards solving a problem, including the strategies they employ. Where appropriate, these strategies are tracked through behavioural data captured by the computer. … where a method of solution is not immediately obvious… this part of the definition corresponds to the definition of “problem” as a situation in which the goal cannot be achieved by merely applying previously learned procedures (mayer, 1990). the PiSa assessment of problem solving is only concerned with such non-routine tasks. in many real-life situations, the same task may be considered a novel problem by some and a routine problem by others. With learning and practice, some activities that were initially experienced as problem solving may become routine activities. the problems included in the PiSa assessment of problem solving involve tasks that are non-routine for 15-year-old students. although some students may be familiar with the context or the goal of a problem situation that refers to a plausible real-world scenario, the particular problem faced is novel and the ways of achieving the goal are not immediately obvious. for example, consider the problem of determining whether a lamp is not working because a) the switch is malfunctioning, b) there is no power, or c) the light bulb needs to be changed. although the situation might be familiar to many 15-year-olds, few students, if any, have had the opportunity to develop expertise in this class of problems, and the unique design of a test unit around this problem situation makes sure that at least some adaptation of ready-made strategies is needed. even in non-routine problems, however, the knowledge of general strategies, including those learned at school, can be of help. the lamp problem described above is a case in point. as in many problems where the solver needs to develop an understanding of cause-effect relationships, an effective approach is to “vary one thing at a time”. this strategy is at the heart of the experimental method in the natural sciences and is taught as such in school curricula throughout the world. Several problem-solving units included in the PiSa assessment indirectly require students to apply a particular strategy in non-curricular contexts, without being prompted to do so. … it includes the willingness to engage with such situations… the last sentence of the definition underscores that the use of knowledge and skills to solve a problem depends on motivational and affective factors as well (mayer, 1998; funke, 2010). Students’ willingness to engage with novel situations is an integral part of problem-solving competence. motivational and affective factors are a distinct focus of the background questionnaire, which uses students’ answers to measure their perseverance (whether they agree or not with the statement “When confronted with a problem, i give up easily”, and other similar statements) and openness to problem solving (“i like to solve complex problems”). 30 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 The PISA 2012 FrAmeworK For ASSeSSIng Problem-SolvIng comPeTence the PiSa framework for assessing problem-solving competence guided the development of the assessment and sets the parameters for reporting results. the framework identifies three distinct aspects: the nature of the problem situation, the problem-solving processes involved in each task, and the problem context. the main elements of the problem-solving framework are summarised in figure v.1.2. • figure v.1.2• main features of the PISA problem-solving framework NATURE OF THE PROBLEM SITUATION Is all the information needed to solve the problem disclosed at the outset? • Interactive: not all information is disclosed; some information has to be uncovered PROBLEM-SOLVING PROCESS What are the main cognitive processes involved in the particular task? • Exploring and understanding the information provided with the problem. by exploring the problem situation. • Static: all relevant information for solving the problem is disclosed at the outset. • Representing and formulating: constructing graphical, tabular, symbolic or verbal representations of the problem situation and formulating hypotheses about the relevant factors and relationships between them. • Planning and executing: devising a plan by setting goals and sub-goals, and executing the sequential steps identified in the plan. • Monitoring and reflecting: monitoring progress, reacting to feedback, and reflecting on the solution, the information provided with the problem, or the strategy adopted. PROBLEM CONTEXT In what everyday scenario is the problem embedded? • Setting: does the scenario involve a technological device? • Focus: what environment does the problem relate to? – Technology (involves a technological device) – Non-technology – Personal (the student, family or close peers) – Social (the community or society in general) the nature of the problem situation is determined by whether the information disclosed to the student at the outset is sufficient to solve the problem (static problems), or whether interaction with the problem situation is a necessary part of the solving activity (interactive problems). examples of interactive problems include problems commonly faced when using unfamiliar devices, such as a new mobile phone or a ticket-vending machine. for the purpose of the PiSa assessment, the cognitive processes involved in problem solving are grouped into four problem-solving processes: • Exploring and understanding. this involves exploring the problem situation by observing it, interacting with it, searching for information and finding limitations or obstacles; and demonstrating understanding of the information given and the information discovered while interacting with the problem situation. • Representing and formulating. this involves using tables, graphs, symbols or words to represent aspects of the problem situation; and formulating hypotheses about the relevant factors in a problem and the relationships between them, to build a coherent mental representation of the problem situation. • Planning and executing. this involves devising a plan or strategy to solve the problem, and executing it. it may involve clarifying the overall goal, setting subgoals, etc. • Monitoring and reflecting. this involves monitoring progress, reacting to feedback, and reflecting on the solution, the information provided with the problem, or the strategy adopted. no assumption is made that the processes involved in solving a particular problem are sequential or that all of the processes listed are involved in solving a particular problem. as individuals confront, represent and solve problems, they may move to a solution in a way that transcends the boundaries of a linear, step-by-step model. nevertheless, single items were intended to have one of these processes as their main focus. although reasoning skills were not explicitly used to organise the domain, each of the problem-solving processes draws upon one or more of them. in understanding a problem situation, the solvers may need to distinguish between facts and Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 31 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 opinion; in formulating a solution, they may need to identify relationships between variables; in selecting a strategy, they may need to consider cause and effect; and, in reflecting on results, they may need to critically evaluate assumptions and alternative solutions. deductive, inductive, analogical, combinatorial, and other types of reasoning are embedded within problem-solving tasks in PiSa. it is important to note that these types of thinking can be taught and honed in classroom instruction (e.g. adey et al., 2007; klauer and Phye, 2008). the problem context is classified according to two dimensions: technology or non-technology, and personal or social. Problems in technology settings involve a technological device, such as a digital clock, an air conditioner, or a ticket machine; problems in non-technology settings do not, and include problems such as task scheduling or decision making. Problems with a personal focus refer to situations involving only the student, the student’s family or close peers; problems with a social focus relate to situations encountered more broadly in the community or society in general. items were developed to measure how well students perform when the various problem-solving processes are exercised within the two different types of problem situations across a range of contexts. each of these key aspects is discussed and illustrated in Chapter 3. The deSIgn And delIvery oF The PISA 2012 comPuTer-bASed ASSeSSmenT oF Problem SolvIng The development of items for the assessment as in all other domains, the items for the PiSa 2012 problem-solving assessment came from two sources: the PiSa Consortium and national submissions. the problem solving expert group that developed the PiSa 2012 framework reviewed all materials to ensure that they reflected the defined construct of problem-solving competence. the items were then reviewed by national centres and field tested. if the national review indicated significant concern that an item would advantage a particular country or language group, it was not considered for inclusion in the main assessment. the procedures to ensure that no group would be consistently advantaged (or disadvantaged) by a particular item are described in greater detail in the PISA 2012 Technical Report (oeCd, forthcoming). a variety of response formats were used, including many that were only possible because the assessment was delivered by computer, such as the use of drop-down menus for selected response formats, or constructed responses coded automatically. as usual in PiSa, items are arranged in units grouped around a common stimulus. the survey included 16 units, with a total of 42 items. Sample units from the PiSa assessment of problem solving are introduced and described at the end of this chapter. The structure and delivery of the assessment in the 28 oeCd countries and 16 partner countries and economies that participated in the assessment of problem solving, the survey was conducted after the paper-based assessment of mathematics, reading and science. in countries that also assessed mathematics and reading on computers, these computer-based tests were administered at the same time as the problem-solving assessment. the 16 units of the problem-solving assessment were grouped into four clusters, each of which was designed to be completed in 20 minutes. each student assessed was given either one or two clusters, depending on whether the student was also participating in the computer-based assessment of mathematics or reading. in all cases, the total time allocated to computer-based tests was 40 minutes. the appearance of the test interface was consistent across items (see figure v.1.3 for an example). for each item the stimulus material appeared in the top part of the screen. the item appeared in the lower part of the screen, and was separated visually from the stimulus by borders. the points at which the screen was divided varied from item to item so that scrolling was never required. test units within clusters and single items within units were delivered in a fixed order, with no possibility of returning to a previous item once students had begun the next item. each test item, with its associated stimulus material, occupied a single computer screen. Students were asked to confirm that they wanted to proceed to the next item when they pressed the next item icon (arrow) in the bottom right corner of the test interface. 32 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 • figure v.1.3 • The test interface TICKETS A train station has an automated ticketing machine. You use the touch screen on the right to buy a ticket. You must make three choices. • Choose the train network you want (subway or country). • Choose the type of fare (full or concession). • Choose a daily ticket or a ticket for a specified number of trips. Daily tickets give you unlimited travel on the day of purchase. If you buy a ticket with a specified number of trips, you can use the trips on different days. The BUY button appears when you have made these three choices. There is a CANCEL button that can be used at any time BEFORE you press the BUY button. Question 1: TIcKeTS CP038Q02 Buy a full fare, country train ticket with two individual trips. Once you have pressed BUY, you cannot return to the question. The opportunities afforded by computer delivery PiSa 2012 marks the second time that individual problem-solving competence was assessed in PiSa. in 2003, a paper and pencil test of cross-disciplinary problem solving was part of the assessment (oeCd, 2005). in PiSa 2012, computer delivery was fundamental to the conception of problem solving. a paper-and-pencil assessment of problem solving could not have measured the same construct. the inclusion of interactive problems, in which students need to explore the (simulated) environment and gather feedback on the effect of their interventions in order to obtain all the information needed to solve a problem, was only possible by asking students to use a computer to complete the assessment. in addition, information about how students interact with the material as they progressed through the assessment was stored on the computer. this information includes the types of actions a student completes (e.g. mouse click, drag and drop, keystrokes), the frequency of interaction between the student and the material, the sequence of actions, the state of the system at any given point, and the timing of specific interactions. the computer delivery made it possible to include authentic response formats, where the observed behaviour corresponds to the answer. this is a major step towards evaluating authentic problem-solving performance. for instance, Question 1 from the unit TICKETS asks students to use a machine that they have never seen before to buy a ticket (figure v.1.3); students earn credit if they succeed in buying the ticket. Students do not need to describe the process in a text or drawing field, or by ticking boxes. various selected response formats, such as drop-down menus, were also included that would not have been possible in a paper-based test. in several items the score reflects not only the explicit response given by students, but also the sequence of actions that they perform before giving the response. for example, in a hypothetical item that required students to troubleshoot a malfunctioning device, where students would need to explore the device in order to uncover information, students Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 33 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 would not get credit for selecting the broken element from a number of given possibilities unless the data logged by the computer indicated that the student had taken the necessary steps to rule out other plausible alternatives. one of the innovative features of the problem-solving assessment is that information contained in log files about the sequence of actions performed by students was used to inform scoring of items where appropriate. for example, when it could be established that students had guessed an answer, they received no credit for that answer. given that the assessment was delivered on computers, familiarity with information and communication technologies (iCt) may have influenced students’ performance. the iCt competence needed to navigate the test interface was limited to such basic skills as using a keyboard, a mouse or a touchpad, clicking radio buttons, dragging-and-dropping, scrolling and using pull-down menus and hyperlinks. in a further attempt to remove any advantage to students who were more familiar with computers, all students completed, before the assessment, a practice unit that contained examples of each of the response formats required. Problem-SolvIng TASKS General characteristics of static and interactive problem-solving tasks as in PiSa 2003, static tasks include decision-making problems, where the student has to choose among alternatives under constraints, and system-analysis problems, where the student needs to identify relationships between parts of a system. the unit TRAFFIC is an example of a decision-making problem, and the unit ROBOT CLEANER is an example of a system-analysis problem (see the section on sample tasks below for more details on each unit). in general, the five units with static items present analytical problems similar to those included in the PiSa 2003 assessment of problem solving. However, since these items were delivered on a computer in 2012, PiSa used new formats for the stimulus information (such as animations; see the unit ROBOT CLEANER) and new response formats (such as drag-and-drop). most interactive units included in the PiSa 2012 assessment of problem solving belong to one of two classes of problems studied in the literature, “microdYn” systems and “finite-state automata”. in both cases, exploration and control of an unknown system are the two main tasks for the student. the single exception is a resource-allocation problem, in which experimental interaction with the test scenario is needed to uncover important information about the available resources. four units are microdYn units, based on small dynamic systems of causal relationships (greiff et al., 2013b; Wüstenberg et al., 2012). the unit CLIMATE CONTROL provides an illustration. microdYn units share a common structure. they consist of a system of causal relations involving only a few variables that have to be explored and controlled in order to reach assigned goal states. in the first, “knowledge-generation” phase, the student has to control up to three input variables; a graph illustrates the effect of inputs on up to three output variables. Students typically have to demonstrate rule knowledge after this first phase. Students are then asked to control the system to reach a certain target by choosing the appropriate input levels. microdYn units vary in the way inputs and outputs are connected in a system, in the number of variables that the system comprises, and in the fictitious scenario in which interactions with the variables take place. Six interactive units are based on finite-state automata (buchner and funke, 1993; funke, 2001), including the unit TICKETS. the field trial unit MP3 PLAYER also belongs to this group. in contrast to mycrodYn units, the outcome of an intervention is not represented by a quantity, but by a new state of the system. many of these units are based on everyday technological devices, and the behaviour of the device depends on both the current state and on the input command received from the user. the context need not be technological, however; a simulated navigation task, where students need to orient themselves by exploring an unfamiliar neighbourhood, is similar in form. What students see in the next step depends both on where they are and what action they take. the distinctive characteristic of finite-state automata is that there are only a finite number of possible states (not all of which are known at the outset), and a limited number of input commands (whose effect may or may not be transparent at the outset).the effect of the interventions may, or may not, depend on the current state of the system. the amount of relevant information that needs to be discovered, the number of possible actions, and the number of possible states all contribute to the level of difficulty of the item. in these problems, students typically need to explore the system or device in order to understand the effect of their interventions, explain the functioning of the device, bring the device into some desired state, or propose improvements to the device. 34 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 Sample tasks from the PISA 2012 problem-solving assessment items from one unit included in the PiSa 2012 field trial, and from four units that were included in the PiSa 2012 main survey, are described below. for each unit, a screenshot of the stimulus information is provided, together with a brief description of the context of the unit. this is followed by a screenshot and description of each item from that unit. the test units described below are also available for viewing on the web at http://cbasq.acer.edu.au. the interactive nature of the units MP3 PLAYER, CLIMATE CONTROL and TICKET MACHINE can be best appreciated by trying to solve the items. Sample unit 1: MP3 PLAYER (ield trial) • figure v.1.4 • mP3 PlAyer: Stimulus information MP3 PLAYER A friend gives you an MP3 player that you can use for playing and storing music. You can change the type of music, and increase or decrease the volume and the bass level by clicking the three buttons on the player. ( , , ) Click RESET to return the player to its original state. in the unit MP3 PLAYER, students are told that they have been given an mP3 player by a friend. they do not know how it works and must interact with it to find out, so the nature of the problem situation for each item in this unit is interactive. Since the focus of the unit is on discovering the rules that govern a device intended for use by an individual, the context of each item in the unit is technology and personal. MP3 PLAYER: Item 1 • figure v.1.5 • mP3 PlAyer: Item 1 Question 1: mP3 PlAyer CP043Q03 The bottom row of the MP3 player shows the settings that you have chosen. Decide whether each of the following statements about the MP3 player is true or false. Select “True” or “False” for each statement to show your answer. Statement True You need to use the middle button ( False ) to change the type of music. You have to set the volume before you can set the bass level. Once you have increased the volume, you can only decrease it if you change the type of music you are listening to. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 35 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 in the first item in the unit, students are given a series of statements about how the system works and are asked to identify whether the statements are true or false. the statements offer scaffolding for students to explore the system. the problemsolving process for this item is exploring and understanding, and the exploration is guided but unrestricted. a “reset” button is available that allows students to return the player to its initial state at any time and start their exploration again if desired. there is no restriction on the number of times this can be done. in the field trial this was a somewhat harderthan-average item, with 38% of students gaining full credit (true, false, false), due probably to the requirement that all three answers must be correct and the degree to which information has to be uncovered (no information is known about the system at the outset and so all knowledge of the rules of the system must come from interacting with it). Partial credit was not available for this item. MP3 PLAYER: Item 2 • figure v.1.6 • mP3 PlAyer: Item 2 Question 2: mP3 PlAyer CP043Q02 Set the MP3 player to Rock, Volume 4, Bass 2. Do this using as few clicks as possible. There is no RESET button. the second item in the unit is classified as planning and executing. in this item, students must plan how to achieve a given goal and then execute this plan. of interest for this partial-credit item is that process information captured by the computer (in this case, how many steps the student takes to successfully reach the goal state) contributes to the score. the task is to be completed using as few clicks as possible and the option of returning the machine to its initial state by pressing the “reset” button is not available. if the number of clicks used (no more than 13) indicates that students have been efficient in reaching the goal they receive full credit; but if they reach the goal in a less-efficient manner (more than 13 clicks), they only receive partial credit. the requirement for efficiency made it more difficult to earn full credit for this item, though it was fairly easy to earn at least partial credit. in the field trial, about 39% of students received full credit and about 33% received partial credit. MP3 PLAYER: Item 3 • figure v.1.7 • mP3 PlAyer: Item 3 Question 3: mP3 PlAyer CP043Q01 Shown below are four pictures of the MP3 player’s screen. Three of the screens cannot happen if the MP3 player is working properly. The remaining screen shows the MP3 player when it is working properly. Which screen shows the MP3 player working properly? 36 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 the third item in the unit is classified as representing and formulating since it requires students to form a mental representation of the way the whole system works in order to identify which of four given pictures shows the mP3 player when it is working properly. returning the player to its initial state, which was possible in the first item, but absent in the second item of the unit, is again possible, so the student may interact with the system as much or as little as needed. Partial credit was not available for this item. in the field trial it was as difficult as the first item in the unit, with 39% of students selecting the correct response (the second option from the left). MP3 PLAYER: Item 4 • figure v.1.8 • mP3 PlAyer: Item 4 Question 4: mP3 PlAyer CP043Q04 Describe how you could change the way the MP3 player works so that there is no need to have the bottom button ( able to change the type of music, and increase or decrease the volume and the bass level. ). You must still be the final item in this unit is classified as monitoring and reflecting, and asks students to reconceptualise the way the device works. this item is a constructed-response item and requires expert scoring. full-credit answers are those that suggest how the mP3 player might operate with only two buttons instead of the original three. there is no single correct answer. Students may think creatively in devising a solution, but the most obvious solution is to suggest changing the way the top button works so that once you reach the right side of the display, one more click takes you back to the left of the display. in the field trial, this was by far the hardest item in the unit, likely because of the requirement of providing a constructed response and the item’s degree of abstraction: students must imagine a hypothetical scenario and link it to their mental representation of how the system currently works, in order to describe a possible alternative functioning. only 25% of students earned credit; partial credit was not available for this item. Sample unit 2: CLIMATE CONTROL • figure v.1.9 • clImATe conTrol: Stimulus information CLIMATE CONTROL You have no instructions for your new air conditioner. You need to work out how to use it. You can change the top, central and bottom controls on the left by using the sliders ( ). The initial setting for each control is indicated by . By clicking APPLY, you will see any changes in the temperature and humidity of the room in the temperature and humidity graphs. The box to the left of each graph shows the current level of temperature or humidity. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 37 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 in the unit CLIMATE CONTROL, students are told that they have a new air conditioner but no instructions for it. Students can use three controls (sliders) to vary temperature and humidity levels, but first they need to understand which control does what. a measure of temperature and humidity in the room appears in the top-right part of the screen, both in numeric and in graphical form. all items in this unit present an interactive problem situation, with context classified as personal and technological. the unit CLIMATE CONTROL is a typical microdYn unit, with a first “knowledge-generation” task and a second “knowledge-application” task. knowledge generation in the microdYn environment requires students to carefully monitor the effects of their interventions. the increase in the level of an input variable leads either to an increase, a decrease, a mixed effect (increase and decrease for different variables), or to no effect in one or more output variables. CLIMATE CONTROL: Item 1 • figure v.1.10 • clImATe conTrol: Item 1 Question 1: clImATe conTrol CP025Q01 Find whether each control influences temperature and humidity by changing the sliders. You can start again by clicking RESET. Draw lines in the diagram on the right to show what each control influences. To draw a line, click on a control and then click on either Temperature or Humidity. You can remove any line by clicking on it. Top control Temperature central control humidity bottom control in the first item in the unit, students are invited to change the sliders to find out whether each control influences the temperature or the humidity level. the problem-solving process for this item is representing and formulating: the student must experiment to determine which controls have an impact on temperature and which on humidity, then represent the causal relations by drawing arrows between the three controls and the two outputs (temperature and humidity). there is no restriction on the number of rounds of exploration that the student is allowed. full credit for this question requires that the causal diagram is correctly completed. Partial credit for this question is given if the student explores the relationships among variables efficiently, by varying only one input at a time, but fails to correctly represent them in a diagram. CLIMATE CONTROL: Item 2 • figure v.1.11 • clImATe conTrol: Item 2 Question 2: clImATe conTrol CP025Q02 The correct relationship between the three controls, Temperature and Humidity is shown on the right. Use the controls to set the temperature and humidity to the target levels. do this in a maximum of four steps. The target levels are shown by the red bands across the Temperature and Humidity graphs. The range of values for each target level is 18-20 and is shown to the left of each red band. you can only click APPly four times and there is no reSeT button. 38 Top control Temperature central control humidity bottom control © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 the second item in the unit asks students to apply their new knowledge of how the air conditioner works to set temperature and humidity at specified target levels (lower than the initial state). this is a planning and executing item. to ensure that no further exploration is needed beyond the one conducted in the previous item, a diagram shows how the controls are related to temperature and humidity levels (students could not return to any previous item during the test). because only four rounds of manipulation are permitted, students need to plan a few steps ahead and use a systematic, if simple, strategy to succeed in this task. nevertheless, the target levels of temperature and humidity provided can be reached in several ways within four steps – the minimum number of steps needed is two – and a mistake can often be corrected, if immediate remedial action is taken. a possible strategy, for instance, is to set separate subgoals and to focus on temperature and humidity in successive steps. if the student is able to bring temperature and humidity both closer to their target levels within the four rounds of manipulation permitted, but does not reach the target for both, partial credit is given. Sample unit 3: TICKETS in the unit TICKETS, students are invited to imagine that they have just arrived at a train station that has an automated ticketing machine. the context for the items in these units is classified as social and technological. • figure v.1.12 • TIcKeTS: Stimulus information TICKETS A train station has an automated ticketing machine. You use the touch screen on the right to buy a ticket. You must make three choices. • Choose the train network you want (subway or country). • Choose the type of fare (full or concession). • Choose a daily ticket or a ticket for a specified number of trips. Daily tickets give you unlimited travel on the day of purchase. If you buy a ticket with a specified number of trips, you can use the trips on different days. The BUY button appears when you have made these three choices. There is a CANCEL button that can be used at any time BEFORE you press the BUY button. at the machine, students can buy subway or country train tickets, with full or concession fares; they can choose daily tickets or a ticket for a specified number of trips. all items in this unit present an interactive problem situation: students are required to engage with the unfamiliar machine and to use the machine to satisfy their needs. TICKETS: Item 1 • figure v.1.13 • TIcKeTS: Item 1 Question 1: TIcKeTS CP038Q02 Buy a full fare, country train ticket with two individual trips. Once you have pressed BUY, you cannot return to the question. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 39 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 in the first item in the unit, students are invited to buy a full fare, country train ticket with two individual trips. this item measures the process of planning and executing. Students first have to select the network (“country trains”), then the fare type (“full fare”), then choose between a daily ticket and one for multiple individual trips, and finally indicate the number of trips (two). the solution requires multiple steps, and instructions are not given in the same order as they need to be applied. this is a relatively linear problem, compared to the following ones, but it is the first encounter with this new machine, which increases its level of difficulty relative to the following ones. TICKETS: Item 2 • figure v.1.14 • TIcKeTS: Item 2 Question 2: TIcKeTS CP038Q01 You plan to take four trips around the city on the subway today. You are a student, so you can use concession fares. Use the ticketing machine to find the cheapest ticket and press BUY. Once you have pressed BUY, you cannot return to the question. in the second item in the unit, students are asked to find and buy the cheapest ticket that allows them to take four trips around the city on the subway, within a single day. as students, they can use concession fares. this item is classified as exploring and understanding because this is the most crucial problem-solving process involved. indeed, to accomplish the task, students must use a targeted exploration strategy, first generating at least the two most obvious possible alternatives (a daily subway tickets with concession fares, or an individual concession fare ticket with four trips), then verifying which of these is the cheapest ticket. if students visit both screens before buying the cheapest ticket (which happens to be the individual ticket with four trips) they are given full credit. Students who buy one of the two tickets without comparing the prices for the two only earn partial credit. Solving this problem involves multiple steps. TICKETS: Item 3 • figure v.1.15 • TIcKeTS: Item 3 Question 3: TIcKeTS CP038Q03 You want to buy a ticket with two individual trips for the city subway. You are a student, so you can use concession fares. Use the ticketing machine to purchase the best ticket available. in the third item, students are asked to buy a ticket for two individual trips on the subway. they are told that they are eligible for concession fares. the third item in the unit is classified as monitoring and reflecting, since it requires them to modify their initial plan (to buy concession-fare tickets for the subway). When concession fares are selected, the machine says that “there are no tickets of this type available”. in this task, students must realise that it is not possible to carry through their initial plan, and so must adjust this plan by buying a full fare ticket for the subway instead. 40 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 Sample unit 4: TRAFFIC • figure v.1.16 • TrAFFIc: Stimulus information TRAFFIC Here is a map of a system of roads that links the suburbs within a city. The map shows the travel time in minutes at 7:00 am on each section of road. You can add a road to your route by clicking on it. Clicking on a road highlights the road and adds the time to the Total Time box. You can remove a road from your route by clicking on it again. You can use the RESET button to remove all roads from your route. in the unit TRAFFIC, students are given a map of a road network with travel times indicated. While this is a unit with static items, because all the information about travel times is provided at the outset, it still exploits the advantages of computer delivery. Students can click on the map to highlight a route, with a calculator in the bottom left corner adding up travel times for the selected route. the context for the items in this unit is classified as social and non-technological. TRAFFIC: Item 1 • figure v.1.17 • TrAFFIc: Item 1 Question 1: TrAFFIc CP007Q01 Pepe is at Sakharov and wants to travel to Emerald. He wants to complete his trip as quickly as possible. What is the shortest time for his trip? 20 minutes 21 minutes 24 minutes 28 minutes in the first item in the unit, a planning and executing item, students are asked about the shortest time to travel from “Sakharov” to “emerald”, two relatively close points shown on the map. four response options are provided. TRAFFIC: Item 2 the second item in the unit TRAFFIC is a similar planning and executing item. it asks students to find the quickest route between “diamond” and “einstein”, two distant points on the map. this time, students must provide their answer by highlighting this route. Students can use the indication that the quickest route takes 31 minutes to avoid generating all possible alternatives systematically; instead, they can explore the network in a targeted way to find the route that takes 31 minutes. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 41 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 • figure v.1.18 • TrAFFIc: Item 2 Question 2: TrAFFIc CP007Q02 Maria wants to travel from Diamond to Einstein. The quickest route takes 31 minutes. Highlight this route. TRAFFIC: Item 3 • figure v.1.19 • TrAFFIc: Item 3 Question 3: TrAFFIc CP007Q03 Julio lives in Silver, Maria lives in Lincoln and Don lives in Nobel. They want to meet in a suburb on the map. No-one wants to travel for more than 15 minutes. Where could they meet? in the third item, students have to use a drop-down menu to select the meeting point that satisfies a condition on travel times for all three participants in a meeting. the demand in this third item is classified as a monitoring and reflecting task, because students have to evaluate possible solutions against a given condition. Sample unit 5: ROBOT CLEANER • figure v.1.20 • roboT cleAner: Stimulus information ROBOT CLEANER The animation shows the movement of a new robotic vacuum cleaner. It is being tested. Click the START button to see what the vacuum cleaner does when it meets different types of objects. You can use the RESET button to place the vacuum cleaner back in its starting position at any time. 42 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 the unit ROBOT CLEANER presents students with an animation showing the behaviour of a robot cleaner in a room. the robotic vacuum cleaner moves forward until it meets an obstacle, then behaves according to a few, deterministic rules, depending on the kind of obstacle. Students can run the animation as many times as they wish to observe this behaviour. despite the animated task prompt, the problem situations in this unit are static, because the student cannot intervene to change the behaviour of the vacuum cleaner or aspects of the environment. the context for the items in these units is classified as social and non-technological. ROBOT CLEANER: Item 1 • figure v.1.21 • roboT cleAner: Item 1 Question 1: roboT cleAner CP002Q08 What does the vacuum cleaner do when it meets a red block? It immediately moves to another red block. It turns and moves to the nearest yellow block. It turns a quarter circle (90 degrees) and moves forward until it meets something else. It turns a half circle (180 degrees) and moves forward until it meets something else. in the first item, students must understand the behaviour of the vacuum cleaner when it meets a red block. the item is classified as exploring and understanding. to show their understanding, they are invited to select, among a list of four options and based on observation, the description that corresponds to the behaviour of the robot cleaner in this situation: “it turns a quarter circle (90 degrees) and moves forward until it meets something else.” ROBOT CLEANER: Item 2 • figure v.1.22 • roboT cleAner: Item 2 Question 2: roboT cleAner CP002Q07 At the beginning of the animation, the vacuum cleaner is facing the left wall. By the end of the animation it has pushed two yellow blocks. If, instead of facing the left wall at the beginning of the animation, the vacuum cleaner was facing the right wall, how many yellow blocks would it have pushed by the end of the animation? 0 1 2 3 in the second item in this unit, students must predict the behaviour of the vacuum cleaner using spatial reasoning. How many obstacles would the vacuum cleaner encounter if it started in a different position? this item is also an exploring and understanding item, because the correct prediction of the robot’s behaviour requires at least a partial understanding of the rules and careful observation of the animation to grasp the information needed. it is made easier if the student notes that the new starting position corresponds to an intermediate state of the robot’s trajectory in the animation. response options are provided. ROBOT CLEANER: Item 3 the final item in this unit is classified as representing and formulating, and asks students to describe the behaviour of the robot cleaner when it meets a yellow block. in contrast to the first task, students must formulate the answer themselves Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 43 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 by entering it in a text box. this item requires expert scoring for credit. full-credit answers are those that describe both of the rules that govern the robot’s behaviour (e.g. “it pushes the yellow block as far as it can and then turns around”). Partial credit was available for answers that only partially describe the behaviour, e.g. by listing only one of the two rules. only a small percentage of students across participating countries obtained full credit for this item. • figure v.1.23 • roboT cleAner: Item 3 Question 3: roboT cleAner CP002Q06 The vacuum cleaner’s behaviour follows a set of rules. Based on the animation, write a rule that describes what the vacuum cleaner does when it meets a yellow block. 44 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 Notes 1. an assessment of collaborative problem-solving skills, which will be included in PiSa 2015, will enrich the understanding of young people’s ability to solve problems. 2. ramalingam, mcCrae and Philpot (forthcoming) trace the history of how the PiSa assessment of problem solving was developed and discuss its relationship with the psychological literature on problem solving and how it is measured. References Adey, P. et al. (2007), “Can we be intelligent about intelligence? Why education needs the concept of plastic general ability”, Educational Research Review, vol. 2, pp. 75-97. Autor, D.H., F. Levy and R.J. Murnane (2003), “the Skill Content of recent technological Change: an empirical exploration”, The Quarterly Journal of Economics, vol. 118, pp. 1278-1333. Autor, D.H. and B. Price (2013), The Changing Task Composition of the US Labor Market: An Update of Autor, Levy and Murnane (2003), mimeo, June 21, 2013. Buchner, A. and J. Funke (1993), “finite-State automata: dynamic task environments in Problem-Solving research”, The Quarterly Journal of Experimental Psychology, vol. 46a, pp. 83-118. Csapó, B. and J. Funke (forthcoming), “developing and assessing Problem Solving”, Chapter 1 in Csapó, b. and J. funke (eds.), The Nature of Problem Solving, oeCd Publishing. Defoe, D. (1919), The Life and Adventures of Robinson Crusoe, Seeley, Service & Co., london (Chapter iX). Funke, J. (2010), “Complex problem solving: a case for complex cognition?”, Cognitive Processing, vol. 11, pp. 133-142. Funke, J. (2001), “dynamic systems as tools for analysing human judgement”, Thinking and Reasoning, vol. 7, pp. 69-79. Funke, J. (1992), “dealing with dynamic Systems: research Strategy, diagnostic approach and experimental results”, The German Journal of Psychology, vol. 16, pp. 24-43. Funke, J. and P.A. Frensch (2007), “Complex problem solving: the european perspective – 10 years after”, in d.H. Johannessen (ed.), Learning to Solve Complex Scientiic Problems, lawrence erlbaum, new York, pp. 25-47. Greiff, S. et al. (2013a), “Complex problem solving in educational settings – Something beyond g: Concept, assessment, measurement invariance, and construct validity”, Journal of Educational Psychology, vol. 105(2), pp. 364-379. Greiff, S. et al. (2013b), “Computer-based assessment of complex problem solving: Concept, implementation, and application”, Educational Technology Research & Development, vol. 61, pp. 407-421. Ikenaga, T. and R. Kambayashi (2010), Long-term Trends in the Polarization of the Japanese Labor Market: The Increase of Non-routine Task Input and Its Valuation in the Labor Market, Hitotsubashi university institute of economic research Working Paper. Klauer, K. and G. Phye (2008), “inductive reasoning: a training approach”, Review of Educational Research, vol. 78, no. 1, pp. 85-123. Kotovsky, K., J.R. Hayes and H.A. Simon (1985), “Why are some problems hard? evidence from tower of Hanoi”, Cognitive psychology, vol. 17, pp. 248-294. Mayer, R.E. (1998), “Cognitive, metacognitive, and motivational aspects of problem solving”, Instructional Science, vol. 26, pp. 49-63. Mayer, R.E. (1990), “Problem solving”, in m.W. eysenck (ed.), The Blackwell Dictionary of Cognitive Psychology, basil blackwell, oxford, pp. 284-288. OECD (forthcoming), PISA 2012 Technical Report, PiSa, oeCd Publishing. OECD (2013a), OECD Skills Outlook 2013: First Results from the Survey of Adult Skills, oeCd Publishing. http://dx.doi.org/10.1787/9789264204256-en OECD (2013b), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, PiSa, oeCd Publishing. http://dx.doi.org/10.1787/9789264190511-en Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 45 1 ASSeSSIng Problem-SolvIng SKIllS In PISA 2012 OECD (2005), Problem Solving for Tomorrow’s World: First Measures of Cross-Curricular Competencies from PISA 2003, PiSa, oeCd Publishing. http://dx.doi.org/10.1787/9789264006430-en Ramalingam, D., B. McCrae and R. Philpot (forthcoming), “the PiSa 2012 assessment of Problem Solving”, Chapter 7 in Csapó, b. and J. funke (eds.), The Nature of Problem Solving, oeCd Publishing. Raven, J. (2000), “Psychometrics, cognitive ability, and occupational performance”, Review of Psychology, vol. 7, pp. 51-74. Spitz-Oener, A. (2006), “technical Change, Job tasks, and rising educational demands: looking outside the Wage Structure”, Journal of Labor Economics, vol. 24, pp. 235-270. Winner, E., T. Goldstein and S. Vincent-Lancrin (2013), Art for Art’s Sake?: The Impact of Arts Education, educational research and innovation, oeCd Publishing. http://dx.doi.org/10.1787/9789264180789-en Wüstenberg, S., S. Greiff and J. Funke (2012), “Complex problem solving – more than reasoning?”, Intelligence, vol. 40, pp. 1-14. 46 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 Student Performance in Problem Solving This chapter examines student performance in problem solving. It introduces the problem-solving performance scale and proficiency levels, describes performance within and across countries and economies, and reports mean performance levels. It also discusses the relationship between problemsolving performance and performance in mathematics, reading and science. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 47 2 STudenT PerFormAnce In Problem SolvIng How well prepared are 15-year-olds to solve problems that they have never encountered before, for which a routine solution has not been learned? the PiSa 2012 computer-based assessment of problem solving uses scenarios that students may encounter in real life, outside of school, in order to measure the skills that students use to solve novel problems. as far as possible, these test problems do not require any expert knowledge to solve. as such, they offer a way of measuring the cognitive processes fundamental to problem solving in general. what the data tell us • Students in Singapore and korea, followed by students in Japan, score higher in problem solving than students in all other participating countries and economies. • on average across oeCd countries, about one in five students is only able to solve very straightforward problems – if any – provided that they refer to familiar situations. by contrast, fewer than one in ten students in Japan, korea, macao-China and Singapore are low-achievers in problem solving. • across oeCd countries, 11.4% of 15-year-old students are top performers in problem solving, meaning that they can systematically explore a complex problem scenario, devise multi-step solutions that take into account all constraints, and adjust their plans in light of the feedback received. • Problem-solving performance is positively related to performance in other assessed subjects, but the relationship is weaker than that observed between performance in mathematics and reading or between performance in mathematics and science. • in australia, brazil, italy, Japan, korea, macao-China, Serbia, england (united kingdom) and the united States, students perform significantly better in problem solving, on average, than students in other countries who show similar performance in mathematics, reading and science. in australia, england (united kingdom) and the united States, this is particularly true among strong and top performers in mathematics; in italy, Japan and korea, it is particularly true among moderate and low performers in mathematics. how The PISA 2012 Problem-SolvIng reSulTS Are rePorTed the previous chapter introduced the concept of problem-solving competence that underlies this assessment. this section discusses how an overall measure of problem-solving competence was derived from students’ answers to questions that measure different aspects of problem-solving competence, and how 15-year-olds were classiied into seven proiciency levels, one of which comprises only those students who perform below the irst, and lowest, described level of proiciency. How the PISA 2012 problem-solving tests were analysed and scaled the relative dificulty of each task included in the assessment of problem solving can be estimated based on student responses. tasks are ordered by increasing levels of dificulty along a single dimension. the dificulty of tasks is estimated by considering the proportion of students who answer each question correctly, with smaller proportions of correct answers indicating growing dificulty. by this measure, the 42 problem-solving tasks included in the PiSa 2012 assessment span a wide range of dificulties. Conversely, the relative proiciency of students taking a particular test can be estimated by considering the proportion of test questions they answer correctly. Students’ proiciency on the test can then be reported on the same scale that measures the dificulty of questions. estimates of student proiciency relect the kinds of tasks students would be expected to perform successfully. this means that students are likely to be able to complete questions successfully at or below the dificulty level associated with their own position on the scale, although they may not always do so.1 Conversely, they are unlikely to be able to complete questions above the dificulty level associated with their position on the scale, although they may sometimes do so. figure v.2.1 illustrates how this probabilistic model works. the further a student’s performance is located above a given question on the proiciency scale, the more likely he or she is to successfully complete the question, and other questions of similar dificulty; the further the student’s performance is located below a given question, the lower the probability that the student will be able to successfully complete the question, and other similarly dificult questions. 48 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng • figure v.2.1 • relationship between questions and student performance on a scale Problem-solving scale item vi items with relatively high dificulty item v item iv items with moderate dificulty item iii items with relatively low dificulty Student a, with We expect student A to successfully relatively high complete items I to V, and probably item VI as well. proiciency Student b, with moderate proiciency We expect student B to successfully complete items I and II, and probably item III as well; but not items V and VI, and probably not item IV either. item ii item i We expect student C to be unable to Student C, with relatively successfully complete any of items II to VI, low proiciency and probably not item I either. the location of student proiciency on this scale is set in relation to the particular group of questions included in the assessment; but just as the sample of students who participated in PiSa in 2012 is drawn to represent all 15-year-olds in the participating countries and economies, the individual questions used in the assessment are selected so that their solutions provide a broad representation of the PiSa 2012 deinition of problem-solving competence. How problem-solving proiciency levels are deined in PISA 2012 PiSa 2012 provides an overall problem-solving proiciency scale, drawing on all the questions in the problem-solving assessment. the problem-solving scale was constructed to have a mean score among oeCd countries of 500, with about two-thirds of students across oeCd countries scoring between 400 and 600.2 to help interpret what students’ scores mean in substantive terms, the scale is divided into seven proiciency levels. Six of these are described based on the skills needed to successfully complete the tasks that are located within them. the range of problem-solving tasks included in the PiSa 2012 assessment allows for describing six levels of problemsolving proiciency. level 1 is the lowest described level, and corresponds to an elementary level of problem-solving skills; level 6 corresponds to the highest level of problem-solving skills. Students with a proiciency score within the range of level 1 are expected to complete most level 1 tasks successfully, but are unlikely to be able to complete tasks at higher levels. Students with scores in the level 6 range are likely to be able to successfully complete all tasks included in the PiSa assessment of problem solving. A proile of PISA problem-solving questions Several questions from the PiSa 2012 assessment of problem solving were released to the public after the survey to illustrate the ways in which performance was measured. these items are presented at the end of Chapter 1. figure v.2.2 shows how these items map onto the described proiciency scale and presents a brief description of each task. tasks included in the same unit can represent a range of dificulties. the unit TICKETS, for example, comprises questions at all levels between 2 and 5. thus a single unit may cover a broad section of the PiSa problem-solving scale. a few tasks included in the test are associated with dificulty levels below level 1. among the released items, one task – Question 1 in unit TRAFFIC – is located below the lowest level of proiciency described. although the number of items that falls below level 1 is not suficient to adequately describe the skills that students who perform below level 1 possess, including tasks that most students, even in the lowest-performing countries, can complete is a way of ensuring that all countries can learn from the assessment results. this indicates that the PiSa 2012 assessment of problem solving can measure not only proiciency in problem solving at different levels, but can also capture some of the elementary components of problem-solving skills. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 49 2 STudenT PerFormAnce In Problem SolvIng • figure v.2.2 • map of selected problem-solving questions, illustrating the proiciency levels level 6 5 4 3 Score range Equal to or higher than 683 points tasks ROBOT CLEANER task 3 (CP002Q06) full credit 618 to less than 683 points CLIMATE CONTROL task 2 (CP025Q02) full credit 672 TICKETS task 2 (CP038Q01) full credit 638 CLIMATE CONTROL task 2 (CP025Q02) Partial credit 592 553 to less than 618 points 488 to less than 553 points 579 ROBOT CLEANER task 2 (CP002Q07) 559 TICKETS task 1 (CP038Q02) 526 task 1 (CP025Q01) Partial credit ROBOT CLEANER task 1 (CP002Q08) 423 to less than 488 points TICKETS task 2 (CP038Q01) Partial credit TRAFFIC task 2 (CP007Q02) 1 358 to less than 423 points below Below 358 points 1 50 701 TICKETS task 3 (CP038Q03) CLIMATE CONTROL task 1 (CP025Q01) full credit 2 task score nature of the task ROBOT CLEANER task 3 (CP002Q06) Partial credit TRAFFIC task 3 (CP007Q03) TRAFFIC task 1 (CP007Q01) 523 492 490 453 446 414 408 340 fully describe the logic governing an unfamiliar system. after observing the behaviour of a (simulated) robot cleaner, the student identifies and writes down the two rules that, together, completely describe what the robot cleaner does when it meets with a certain type of obstacle. efficiently control a system with multiple dependencies to achieve a given outcome. a diagram shows which controls of an air conditioner can be used to vary temperature and humidity levels. the student is only allowed four rounds of manipulation, but the target levels of temperature and humidity provided can be reached in several ways within these four steps and a mistake can often be corrected if immediate remedial action is taken. However, the student must use the information provided about causal dependencies to plan a few steps ahead, consistently monitor progress towards the target, and respond quickly to feedback. use targeted exploration to accomplish a task. buy tickets with a ticket machine, adjusting to feedback gathered over the course of the task to comply with all constraints: the ticket bought not only complies with three explicit instructions, but the student compared prices between the two possible options before making a selection, thus checking the constraint to buy the cheapest ticket. execution of the solution involves multiple steps. Control a system with multiple dependencies to achieve a given outcome. a diagram shows which controls of an air conditioner can be used to vary temperature and humidity levels. for partial credit, the student is able to bring the two outputs closer to their target levels, without actually reaching them for both, within the four rounds of manipulation permitted. execute a plan for working around an unexpected impasse: a malfunction of the ticket machine that is only discovered after multiple steps. the student wants to buy subway tickets at the ticket machine and is eligible to concession fares, but when concession fares are selected, the machine says that “there are no tickets of this type available”. the student instead buys a full fare ticket for the subway. Predict the behaviour of a simple unfamiliar system using spatial reasoning. the task prompt shows the behaviour of a robot cleaner in a room, and the student is asked to predict the behaviour of the robot cleaner if it were in a different starting position. the new starting position corresponds to an intermediate state of the robot’s trajectory shown to students: the correct prediction of the robot’s behaviour does not necessarily require a full understanding of the rules governing it. a partial understanding of the rules and careful observation are sufficient. use an unfamiliar ticketing machine to buy a ticket. the student follows explicit instructions to make the appropriate selection at each step. instructions, however, are not given in the order in which they must be used, and multiple steps are needed to execute the solution. explore and represent the relationships between variables in a system with multiple dependencies. an unfamiliar air conditioner has three controls that determine its effect on air temperature and humidity. the student must experiment with the controls to determine which controls have an impact on temperature and which on humidity, then represent the causal relations by drawing arrows between the three inputs (the controls) and the two outputs (temperature and humidity) (full credit). Partial credit for this question is given if the student explores the relationships between variables in an efficient way, by varying only one input at a time, but fails to correctly represent them in a diagram. understand behaviour of an unfamiliar system. Select, among a list of four options and based on observation, the description that corresponds to the behaviour of the robot cleaner in a specific situation: “What does the vacuum cleaner do when it meets a red block?” “it turns a quarter circle (90 degrees) and moves forward until it meets something else.” use a machine to buy tickets for a given situation, without checking that the solution satisfies a condition (cheapest ticket). to obtain partial credit, the student buys either a daily ticket or four single tickets for the subway, with concession fares, but does not compare the two options to determine the best choice as requested. the student had the opportunity to learn how to use the basic functions of the machine in the previous task (TICKETS, task 1). buying a ticket involves multiple steps. Highlight the shortest route between two distant points on a map. an indication in the task prompt can be used to verify that the solution found corresponds to the shortest route. Partially describe the logic governing an unfamiliar system after observing its behaviour in an animation: recognise and formulate, at least partially, a rule governing the behaviour of the robot cleaner in a specific situation (e.g. “it turns”). evaluate different possibilities using a network diagram to find a meeting point that satisfies a condition on travel times for all three participants in a meeting. read travel times on a simple network diagram to find the shortest route between two close points on a map. all necessary information is disclosed at the outset and response options are provided. the correct solution can be found with a few simple trial-and-error iterations. © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng box v.2.1 presents the major differences between dificult and easy tasks, and links them to students’ progress in problem solving. box v.2.1. how students progress in problem solving as students acquire proiciency in problem solving, they learn to handle increasingly complex demands. What these demands are, and what it means for students to become better problem-solvers, can be inferred by comparing the easier tasks at the bottom of figure v.2.2 to the harder tasks shown above them. an analysis of the entire problem set used in PiSa 2012 (Philpot et al., forthcoming) identiied several characteristics that are associated with task dificulty: 1) distance from goal and reasoning skills required: in problems at the bottom of the scale, there are generally few barriers to overcome in order to reach the solution; the goal is at most one or two steps away. in addition, overcoming the barriers does not require logical or combinatorial reasoning. in harder problems, the distance from the goal increases, and each step may require high levels of reasoning (such as combinatorial reasoning to identify all possible alternatives, deductive reasoning to eliminate possibilities, etc.). 2) number of constraints and conditions: the easiest tasks involve at most one condition to be satisied. in more dificult problems, the student often needs to monitor several conditions, and restrictions on actions, such as limits on the number of experimental rounds, are introduced. it thus becomes necessary to plan ahead, especially if the constraints cannot be addressed successively. 3) amount of information: to solve the easiest problems, all that is required is understanding a small amount of information that is explicitly provided in a simple format. as the problems become more dificult, the amount of information required increases. often, information has to be integrated from several sources and in several formats (e.g. graphs, tables and texts), including feedback received while solving the problem (as in the units TICKETS and CLIMATE CONTROL). 4) unfamiliarity and system complexity: the easiest tasks are cast in familiar settings, such as those involving a public transport map (e.g. TRAFFIC). tasks that use more abstract scenarios or that refer to less familiar objects (such as ROBOT CLEANER) are generally more dificult. in addition, the simplest problems have few possible actions, clear causal linkages, and no unexpected impasses. tasks that are harder to solve usually involve a larger number of possible actions and consequences to monitor; and the components of the problem form a more interrelated system. initially, students may be able to solve only problems cast in familiar settings that require one simple condition to be satisied and where the goal is only one or two steps away, as is the case in tasks 1 and 3 of the unit TRAFFIC. as students develop their problem-solving proiciency (i.e. their capacity to understand and resolve problems whose solution is not immediately obvious), the complexity of problems that they can solve grows. at level 3 on the problem-solving scale, students can handle information presented in several different formats, infer elementary relationships between the components of a simple system or device, and engage in experimental manipulation to conirm or refute a hypothesis. they are conident in solving problems such as task 1 in unit CLIMATE CONTROL and task 1 in unit ROBOT CLEANER. at level 5, students fully grasp the underlying structure of a moderately complex problem, which allows them to think ahead, detect unexpected dificulties or mistakes, and adjust their plans accordingly – all of which are required to achieve the goal in CLIMATE CONTROL (task 2) and TICKETS (task 2). whAT STudenTS cAn do In Problem SolvIng PiSa summarises student performance in problem solving on a single scale that provides an overall assessment of students’ problem-solving competence at age 15. results for this overall performance measure are presented below, covering both the average level of performance in problem solving in each country/economy and the distribution of problem-solving proiciency. Chapter 3 analyses these results in more detail, covering the various components of proiciency in problem solving. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 51 2 STudenT PerFormAnce In Problem SolvIng Average level of proiciency in problem solving this section uses students’ average scores to summarise the performance of countries and economies in problem solving, both relative to each other and to the oeCd mean. Since problem solving is a new domain in PiSa 2012, the oeCd average performance was set at 500 score points, and the standard deviation across oeCd countries at 100 score points. this establishes the benchmark against which each country’s problem-solving performance in PiSa 2012 is compared. • figure v.2.3 • comparing countries’ and economies’ performance in problem solving Statistically signiicantly above the oeCd average not statistically signiicantly different from the oeCd average Statistically signiicantly below the oeCd average mean score 562 561 552 540 540 536 534 526 523 523 517 515 511 511 510 509 509 508 508 506 503 498 497 494 491 489 483 481 477 476 473 466 459 454 454 448 445 428 422 411 407 403 402 399 comparison country/economy Singapore Korea Japan Macao-China Hong Kong-China Shanghai-China Chinese Taipei Canada Australia Finland England (UK) Estonia France Netherlands Italy Czech Republic Germany United States Belgium Austria Norway Ireland Denmark Portugal Sweden Russian Federation Slovak Republic Poland Spain Slovenia Serbia Croatia Hungary Turkey Israel Chile Cyprus1, 2 Brazil Malaysia United Arab Emirates Montenegro Uruguay Bulgaria Colombia countries and economies whose mean score is not statistically signiicantly different from the comparison country’s/economy’s score korea Singapore, Japan korea Hong kong-China, Shanghai-China macao-China, Shanghai-China, Chinese taipei macao-China, Hong kong-China, Chinese taipei Hong kong-China, Shanghai-China australia, finland, england (uk) Canada, finland, england (uk) Canada, australia, england (uk) Canada, australia, finland, estonia, france, netherlands, italy, Czech republic, germany, united States, belgium, austria england (uk), france, netherlands, italy, Czech republic, germany, united States england (uk), estonia, netherlands, italy, Czech republic, germany, united States, belgium, austria, norway england (uk), estonia, france, italy, Czech republic, germany, united States, belgium, austria, norway england (uk), estonia, france, netherlands, Czech republic, germany, united States, belgium, austria, norway england (uk), estonia, france, netherlands, italy, germany, united States, belgium, austria, norway england (uk), estonia, france, netherlands, italy, Czech republic, united States, belgium, austria, norway england (uk), estonia, france, netherlands, italy, Czech republic, germany, belgium, austria, norway, ireland england (uk), france, netherlands, italy, Czech republic, germany, united States, austria, norway england (uk), france, netherlands, italy, Czech republic, germany, united States, belgium, norway, ireland france, netherlands, italy, Czech republic, germany, united States, belgium, austria, ireland, denmark, Portugal united States, austria, norway, denmark, Portugal, Sweden norway, ireland, Portugal, Sweden, russian federation norway, ireland, denmark, Sweden, russian federation ireland, denmark, Portugal, russian federation, Slovak republic, Poland denmark, Portugal, Sweden, Slovak republic, Poland Sweden, russian federation, Poland, Spain, Slovenia Sweden, russian federation, Slovak republic, Spain, Slovenia, Serbia Slovak republic, Poland, Slovenia, Serbia, Croatia Slovak republic, Poland, Spain, Serbia Poland, Spain, Slovenia, Croatia Spain, Serbia, Hungary, israel Croatia, turkey, israel Hungary, israel, Chile Croatia, Hungary, turkey, Chile, Cyprus1, 2 turkey, israel, Cyprus1, 2 israel, Chile malaysia brazil montenegro, uruguay, bulgaria united arab emirates, uruguay, bulgaria united arab emirates, montenegro, bulgaria, Colombia united arab emirates, montenegro, uruguay, Colombia uruguay, bulgaria 1. footnote by turkey: the information in this document with reference to “Cyprus” relates to the southern part of the island. there is no single authority representing both turkish and greek Cypriot people on the island. turkey recognises the turkish republic of northern Cyprus (trnC). until a lasting and equitable solution is found within the context of the united nations, turkey shall preserve its position concerning the “Cyprus issue”. 2. footnote by all the european union member States of the oeCd and the european union: the republic of Cyprus is recognised by all members of the united nations with the exception of turkey. the information in this document relates to the area under the effective control of the government of the republic of Cyprus. Source: oeCd, PiSa 2012 database. 12 http://dx.doi.org/10.1787/888933003573 52 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng When interpreting mean performance, only those differences among countries and economies that are statistically signiicant should be taken into account (box v.2.2). figure v.2.3 shows each country’s/economy’s mean score, and allows readers to see for which pairs of countries/economies the differences between the means shown are statistically similar. the data on which figure v.2.3 is based are presented in annex b. for each country/economy shown in the middle column, the countries/economies listed in the column on the right are those whose mean scores are not suficiently different to be distinguished with conidence.3 for all other cases, Country a scores higher than Country b if Country a is above Country b in the list in the middle column, and scores lower if Country a is shown below Country b. for example, while finland clearly ranks above the united States, the performance of england (united kingdom) cannot be distinguished with conidence from either finland or the united States. box v.2.2. what is a statistically signiicant difference? a difference is called statistically signiicant if it is very unlikely that such a difference could be observed in the estimates based on samples, when in fact no true difference exists in the populations. the results of the PiSa assessments for countries and economies are estimates because they are obtained from samples of students, rather than a census of all students, and they are obtained using a limited set of assessment tasks, not the universe of all possible assessment tasks. When the sampling of students and assessment tasks are done with scientiic rigour, it is possible to determine the magnitude of the uncertainty associated with the estimate. this uncertainty needs to be taken into account when making comparisons so that differences that could reasonably arise simply due to the sampling of students and tasks are not interpreted as differences that actually hold for the populations. figure v.2.3 lists each participating country and economy in descending order of its mean problem-solving score (left column). the values range from a high of 562 points for the partner country Singapore to a low of 399 points for the partner country Colombia. Countries and economies are also divided into three broad groups: those whose mean scores are statistically around the oeCd mean (highlighted in dark blue), those whose mean scores are above the oeCd mean (highlighted in pale blue), and those whose mean scores are below the oeCd mean (highlighted in medium blue). box v.2.3 provides guidance to gauge the magnitude of score differences. because the igures are derived from samples, it is not possible to determine a country’s precise rank among the participating countries. However, it is possible to determine, with conidence, a range of ranks in which the country’s performance lies (figure v.2.4). Singapore and korea are the highest-performing countries in problem solving, with mean scores of 562 points and 561 points, respectively. fifteen-year-olds in these two countries perform about a full proiciency level above the level of students in other oeCd countries, on average. Japan ranks third among all participating countries, and second among oeCd countries, with a mean score of 552 points. four more east asian partner economies score between 530 and 540 points on the PiSa problem-solving scale: macao-China (with a mean score of 540 points), Hong kong-China (540 points), Shanghai-China (536 points) and Chinese taipei (534 points). twelve oeCd countries perform above the oeCd average, but below the former group of countries: Canada (526 points), australia (523 points), finland (523 points), england (united kingdom) (517 points), estonia (515 points), france (511 points), the netherlands (511 points), italy (510 points), the Czech republic (509 points), germany (509 points), the united States (508 points) and belgium (508 points). five countries, austria, norway, ireland, denmark and Portugal, score around the oeCd mean. there are clear and substantial differences in mean country performance on the problem-solving assessment. box v.2.3 illustrates how the differences in mean performance compare to differences in problem-solving proiciency within countries/economies. among oeCd countries, the lowest-performing country, Chile, has an average score of 448. this means that the gap between the highest- and lowest-performing oeCd country is 113 score points – well above one standard deviation. about 90% of students from korea perform above Chile’s mean score; conversely, only about 10% of students from Chile perform above korea’s mean score (table v.2.2). overall, more than two proiciency levels (163 score points) separate the highest-performing (Singapore) and lowest-performing (Colombia) countries in problem solving. only about one in 20 students in the four best-performing countries and economies performs at or below the mean of the lowest-performing country. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 53 2 STudenT PerFormAnce In Problem SolvIng • figure v.2.4 [Part 1/2] • Problem-solving performance among participating countries/economies Problem-solving scale range of ranks oEcd countries mean score Singapore 562 Korea 561 Japan 552 Macao-China 540 Hong Kong-China 540 Shanghai-China 536 Chinese Taipei 534 North West (Italy) 533 Western Australia (Australia) 528 North East (Italy) 527 Canada 526 Australian Capital Territory (Australia) 526 New South Wales (Australia) 525 Flemish Community (Belgium) 525 Victoria (Australia) 523 Australia 523 Finland 523 Queensland (Australia) 522 German-speaking Community (Belgium) 520 South Australia (Australia) 520 England (United Kingdom) 517 Estonia 515 Centre (Italy) 514 Northern Territory (Australia) 513 France 511 Netherlands 511 Italy 510 Czech Republic 509 Germany 509 United States 508 Belgium 508 Madrid (Spain) 507 Austria 506 Alentejo (Portugal) 506 Norway 503 Ireland 498 Denmark 497 Basque Country (Spain) 496 Portugal 494 Sweden 491 Tasmania (Australia) 490 Russian Federation 489 Catalonia (Spain) 488 South Islands (Italy) 486 French Community (Belgium) 485 Slovak Republic 483 Poland 481 Spain 477 Slovenia 476 S.E. (1.2) (4.3) (3.1) (1.0) (3.9) (3.3) (2.9) (8.6) (4.0) (6.4) (2.4) (3.7) (3.5) (3.3) (4.1) (1.9) (2.3) (3.4) (2.6) (4.1) (4.2) (2.5) (10.8) (7.9) (3.4) (4.4) (4.0) (3.1) (3.6) (3.9) (2.5) (13.0) (3.6) (13.4) (3.3) (3.2) (2.9) (3.9) (3.6) (2.9) (4.0) (3.4) (8.4) (8.5) (4.4) (3.6) (4.4) (4.1) (1.5) all countries/economies upper rank lower rank 1 2 1 2 3 5 8 10 3 3 6 6 8 8 11 11 4 6 11 10 9 11 16 15 6 6 7 7 7 7 9 14 16 16 15 16 16 16 11 11 12 12 12 12 14 19 21 21 20 21 21 21 8 17 13 22 11 15 16 18 19 20 16 20 21 23 24 25 17 18 20 21 22 23 26 27 23 27 25 26 27 28 29 31 31 31 20 21 21 22 23 24 24 24 upper rank 1 1 3 4 4 4 5 lower rank 2 2 3 6 7 7 7 Notes: oeCd countries are shown in bold black. Partner countries and economies are shown in bold blue. regions are shown in black italics (oeCd countries) or blue italics (partner countries). italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north east (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). 1. footnote by turkey: the information in this document with reference to “Cyprus” relates to the southern part of the island. there is no single authority representing both turkish and greek Cypriot people on the island. turkey recognises the turkish republic of northern Cyprus (trnC). until a lasting and equitable solution is found within the context of the united nations, turkey shall preserve its position concerning the “Cyprus issue”. 2. footnote by all the european union member States of the oeCd and the european union: the republic of Cyprus is recognised by all members of the united nations with the exception of turkey. the information in this document relates to the area under the effective control of the government of the republic of Cyprus. Countries, economies and subnational entities are ranked in descending order of mean problem-solving performance. Source: oeCd, PiSa 2012 database. 12 http://dx.doi.org/10.1787/888933003573 54 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng • figure v.2.4 [Part 2/2] • Problem-solving performance among participating countries/economies Problem-solving scale range of ranks oEcd countries mean score South (Italy) 474 Serbia 473 Croatia 466 Hungary 459 Dubai (United Arab Emirates) 457 Turkey 454 Israel 454 Chile 448 Southeast Region (Brazil) 447 Cyprus1, 2 445 Central-West Region (Brazil) 441 South Region (Brazil) 435 Brazil 428 Medellín (Colombia) 424 Manizales (Colombia) 423 Malaysia 422 Sharjah (United Arab Emirates) 416 United Arab Emirates 411 Bogotá (Colombia) 411 Montenegro 407 Uruguay 403 Bulgaria 402 Colombia 399 Cali (Colombia) 398 Fujairah (United Arab Emirates) 395 Northeast Region (Brazil) 393 Abu Dhabi (United Arab Emirates) 391 North Region (Brazil) 383 Ajman (United Arab Emirates) 375 Ras al-Khaimah (United Arab Emirates) 373 Umm al-Quwain (United Arab Emirates) 372 S.E. (8.4) (3.1) (3.9) (4.0) (1.3) (4.0) (5.5) (3.7) (6.3) (1.4) (11.9) (7.8) (4.7) (7.6) (5.3) (3.5) (8.6) (2.8) (5.7) (1.2) (3.5) (5.1) (3.5) (9.0) (4.0) (11.0) (5.3) (10.9) (8.0) (11.9) (3.5) upper rank all countries/economies lower rank upper rank lower rank 25 27 29 31 32 32 33 35 25 25 26 28 28 28 33 33 34 36 37 37 36 37 38 39 38 39 40 41 40 41 41 42 42 44 44 44 Notes: oeCd countries are shown in bold black. Partner countries and economies are shown in bold blue. regions are shown in black italics (oeCd countries) or blue italics (partner countries). italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north east (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). 1. footnote by turkey: the information in this document with reference to “Cyprus” relates to the southern part of the island. there is no single authority representing both turkish and greek Cypriot people on the island. turkey recognises the turkish republic of northern Cyprus (trnC). until a lasting and equitable solution is found within the context of the united nations, turkey shall preserve its position concerning the “Cyprus issue”. 2. footnote by all the european union member States of the oeCd and the european union: the republic of Cyprus is recognised by all members of the united nations with the exception of turkey. the information in this document relates to the area under the effective control of the government of the republic of Cyprus. Countries, economies and subnational entities are ranked in descending order of mean problem-solving performance. Source: oeCd, PiSa 2012 database. 12 http://dx.doi.org/10.1787/888933003573 box v.2.3. Interpreting differences in PISA problem-solving scores: how large a gap? in PiSa 2012, student performance in problem solving is described through six levels of proiciency, each of which represents 65 score points. thus, a difference in performance of one proiciency level represents a comparatively large disparity in performance. for example, students proicient at level 2 on the problem-solving scale are only starting to demonstrate problem-solving competence. they engage with unfamiliar problem situations, but need extensive guidance in order to progress towards a solution. they can perform only one task at a time, and can only test a simple hypothesis that is given to them. meanwhile, students proicient at level 3 are more self-directed in their problem solving. they can devise hypotheses to test themselves, and can handle multiple constraints by planning a few steps ahead, provided that the constraints can be addressed sequentially. ... Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 55 2 STudenT PerFormAnce In Problem SolvIng the difference in average performance between the highest- and lowest-performing countries is 163 score points. the difference between the highest- and lowest-performing oeCd countries is 113 score points. Within countries and economies, even larger gaps separate the highest- and lowest-performing students (table v.2.2). on average across oeCd countries, the distance between the highest-performing 10% of students and the lowest-performing 10% of students is equal to 245 score points; but half of all students in oeCd countries score within 129 points of each other. treating all oeCd countries as a single unit, one standard deviation in the distribution of student performance on the PiSa problem-solving scale corresponds to 100 points; this means that, on average within oeCd countries, two-thirds of the student population have scores within 100 points of the oeCd mean, set at 500 score points. Students at the different levels of proiciency in problem solving this section describes performance in terms of the six levels of proiciency that have been constructed for reporting the PiSa 2012 problem-solving assessment. a seventh proiciency level, below level 1, includes those students who cannot successfully complete many of the items of level 1 dificulty. figure v.2.5 shows what students can typically do at each of the six levels of proiciency in problem solving. these summary descriptions are based on the detailed analysis of task demands within each level. the task demands for released items are described in figure v.2.2. the distribution of student performance across proiciency levels is shown in figure v.2.6. Proiciency at Level 6 Students proicient at level 6 on the problem-solving scale are highly eficient problem-solvers. they can develop complete, coherent mental models of diverse problem scenarios, enabling them to solve complex problems eficiently. across oeCd countries, only one in 40 students (2.5%) performs at this level, but student proiciency varies among countries. in Singapore and korea, the proportion is more than three times as large (9.6% and 7.6%, respectively). in Singapore, almost one in ten students is a highly skilled problem-solver. these two countries also top the overall rankings in average performance (figure v.2.4). in contrast, some countries and economies with above-average overall performance do not have many students at the highest level of problem-solving proiciency. among these are italy (mean score of 510 points) and france (511 points), both with smaller-than-average proportions of students reaching level 6 (1.8% in italy, 2.1% in france) (figure v.2.6 and table v.2.1). the fact that such a small proportion of students performs at level 6 indicates that the PiSa scale can distinguish problem-solving proiciency up to the highest levels that 15-year-olds are capable of attaining. indeed, in two oeCd countries and seven partner countries and economies, fewer than one in 200 students perform at the top level. Proiciency at Level 5 Students proicient at level 5 on the problem-solving scale can systematically explore a complex problem scenario to gain an understanding of how relevant information is structured. When faced with a complex problem involving multiple constraints or unknowns, students whose highest level of proiciency is level 5 try to solve them through targeted exploration, methodical execution of multi-step plans, and attentive monitoring of progress. in contrast, level 6 problem-solvers are able to start by developing an overall strategic plan based on a complete mental model of the problem. Since students proicient at level 6 can also complete level 5 tasks, the following descriptions use “proicient at level 5” to mean those whose highest level of performance is either level 5 or level 6. the same terminology is used to refer to the cumulative proportions at lower levels. Students performing at level 5 or 6 are also referred to as “top performers” in the rest of this report. across oeCd countries, 11.4% of 15-year-old students are proicient at level 5 or higher. in Singapore, korea and Japan, more than one in ive students are capable of level 5 tasks. more than one in six students perform at level 5 or above in Hong kong-China (19.3%), Chinese taipei and Shanghai-China (18.3%), Canada (17.5%) and australia (16.7%). 56 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng all of these countries/economies also show relatively high mean proiciency. Conversely, countries with lower average performance also tend to have the smallest proportions of students who can complete level 5 tasks. in montenegro, malaysia, Colombia, uruguay, bulgaria and brazil, fewer than 2% of students perform at level 5 or 6. all of these countries perform well below the oeCd average. • figure v.2.5 • Summary descriptions of the six levels of proiciency in problem solving level Score range Percentage of students able to perform tasks at this level or above (oEcd average) What students can typically do 1 358 to less than 423 points 91.8% at level 1, students can explore a problem scenario only in a limited way, but tend to do so only when they have encountered very similar situations before. based on their observations of familiar scenarios, these students are able only to partially describe the behaviour of a simple, everyday device. in general, students at level 1 can solve straightforward problems provided there is a simple condition to be satisfied and there are only one or two steps to be performed to reach the goal. level 1 students tend not to be able to plan ahead or set subgoals. 2 423 to less than 488 points 78.6% at level 2, students can explore an unfamiliar problem scenario and understand a small part of it. they try, but only partially succeed, to understand and control digital devices with unfamiliar controls, such as home appliances and vending machines. level 2 problem-solvers can test a simple hypothesis that is given to them and can solve a problem that has a single, specific constraint. they can plan and carry out one step at a time to achieve a subgoal, and have some capacity to monitor overall progress towards a solution. 3 488 to less than 553 points 56.6% at level 3, students can handle information presented in several different formats. they can explore a problem scenario and infer simple relationships among its components. they can control simple digital devices, but have trouble with more complex devices. Problem-solvers at level 3 can fully deal with one condition, for example, by generating several solutions and checking to see whether these satisfy the condition. When there are multiple conditions or inter-related features, they can hold one variable constant to see the effect of change on the other variables. they can devise and execute tests to confirm or refute a given hypothesis. they understand the need to plan ahead and monitor progress, and are able to try a different option if necessary. 4 553 to less than 618 points 31.0% at level 4, students can explore a moderately complex problem scenario in a focused way. they grasp the links among the components of the scenario that are required to solve the problem. they can control moderately complex digital devices, such as unfamiliar vending machines or home appliances, but they don't always do so efficiently. these students can plan a few steps ahead and monitor the progress of their plans. they are usually able to adjust these plans or reformulate a goal in light of feedback. they can systematically try out different possibilities and check whether multiple conditions have been satisfied. they can form an hypothesis about why a system is malfunctioning and describe how to test it. 5 618 to less than 683 points 11.4% at level 5, students can systematically explore a complex problem scenario to gain an understanding of how relevant information is structured. When faced with unfamiliar, moderately complex devices, such as vending machines or home appliances, they respond quickly to feedback in order to control the device. in order to reach a solution, level 5 problem-solvers think ahead to find the best strategy that addresses all the given constraints. they can immediately adjust their plans or backtrack when they detect unexpected difficulties or when they make mistakes that take them off course. 6 Equal to or higher than 683 points 2.5% at level 6, students can develop complete, coherent mental models of diverse problem scenarios, enabling them to solve complex problems efficiently. they can explore a scenario in a highly strategic manner to understand all information pertaining to the problem. the information may be presented in different formats, requiring interpretation and integration of related parts. When confronted with very complex devices, such as home appliances that work in an unusual or unexpected manner, they quickly learn how to control the devices to achieve a goal in an optimal way. level 6 problem-solvers can set up general hypotheses about a system and thoroughly test them. they can follow a premise through to a logical conclusion or recognise when there is not enough information available to reach one. in order to reach a solution, these highly proficient problem-solvers can create complex, flexible, multi-step plans that they continually monitor during execution. Where necessary, they modify their strategies, taking all constraints into account, both explicit and implicit. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 57 2 STudenT PerFormAnce In Problem SolvIng • figure v.2.6 • Proiciency in problem solving Percentage of students at the different levels of problem-solving proiciency below level 1 Korea Japan Macao-China Singapore Hong Kong-China Shanghai-China Chinese Taipei Finland Canada Estonia Australia England (United Kingdom) Italy France United States Czech Republic Austria Netherlands Germany Ireland Denmark Portugal Belgium Norway OECD average Russian Federation Sweden Poland Slovak Republic Spain Slovenia Serbia Croatia Hungary Turkey Chile Israel Brazil Malaysia United Arab Emirates Bulgaria Montenegro Uruguay Colombia % 100 level 1 level 2 level 3 level 4 level 5 level 6 Students at level 1 or below Students at level 2 or above 80 60 40 20 0 20 40 60 80 Korea Japan Macao-China Singapore Hong Kong-China Shanghai-China Chinese Taipei Finland Canada Estonia Australia England (United Kingdom) Italy France United States Czech Republic Austria Netherlands Germany Ireland Denmark Portugal Belgium Norway OECD average Russian Federation Sweden Poland Slovak Republic Spain Slovenia Serbia Croatia Hungary Turkey Chile Israel Brazil Malaysia United Arab Emirates Bulgaria Montenegro Uruguay Colombia 100 % Countries and economies are ranked in descending order of the percentage of students at Levels 2, 3, 4, 5 and 6 in problem solving. Source: oeCd, PiSa 2012 database, table v.2.1. 1 2 http://dx.doi.org/10.1787/888933003573 in general, a ranking of countries and economies by the proportion of top-performing students (students at level 5 or above) matches the ranking of countries/economies by mean performance, but there are a number of exceptions (box v.2.4 and figure v.2.7). in belgium, the proportion of students proicient at level 5 (14.4%) is larger than that in estonia (11.8%), while overall, estonia has higher average performance (515 points) than belgium (508 points). Similarly, in israel the proportion of top performers is large (8.8%) compared with countries of similar average performance (454 points), such as turkey, where only 2.2% of students are top performers (figure v.2.6 and table v.2.1). Proiciency at Level 4 Students proicient at level 4 on the problem-solving scale can explore a problem scenario in a focused way, grasp the links among the components of the scenario that are required to solve the problem, plan a few steps ahead, and monitor 58 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng the progress of their plans. they can control moderately complex devices, such as unfamiliar vending machines or home appliances, but they don’t always do so eficiently. in the sample task CLIMATE CONTROL (task 2), for instance, they try to reach the target levels for humidity and temperature by addressing each of them in succession, rather than simultaneously. across oeCd countries, 31% of students are proicient at level 4 or higher. in korea, Singapore and Japan, most 15-yearold students can complete tasks at level 4; and in all of these countries, the highest proiciency attained by the largest proportion of students is level 4. the mean performance of Singapore (562 points) and korea (561 points) also falls within this level. by contrast, in Colombia, montenegro, malaysia, uruguay, bulgaria, brazil and the united arab emirates fewer than one in ten students reaches level 4. these are also the countries with the lowest mean scores in problem solving (figure v.2.6 and table v.2.1). Proiciency at Level 3 Students proicient at level 3 can handle information presented in several different formats. they can explore a problem scenario and infer simple relationships among its components. Problem-solvers at level 3 can fully deal with one condition, for example, by generating several solutions and checking to see whether these satisfy the condition. When there are multiple conditions or inter-related features, they can hold one variable constant to see the effect of change on the other variables. they can devise and execute tests to conirm or refute a given hypothesis. they understand the need to plan ahead and monitor progress. across oeCd countries, the majority (57%) of 15-year-old students are proicient at least at level 3. for about one in four students (26%), level 3 is the highest level reached. level 3 is the most common level of proiciency in problem solving attained by students in 26 of the 44 countries and economies that assessed problem-solving skills in PiSa 2012. three out of four students in korea, Japan and Singapore attain at least level 3 in problem solving. by contrast, in 18 countries, including eight oeCd countries, fewer than one in two students can complete tasks at level 3 successfully (figure v.2.6 and table v.2.1). Proiciency at Level 2 Students proicient at level 2 on the problem-solving scale can explore an unfamiliar problem scenario and understand a small part of it, can test a simple hypothesis that is given to them, and can solve a problem that has a single, speciic constraint. they can plan and carry out one step at a time to achieve a subgoal, and have some capacity to monitor overall progress towards a solution. level 2 can be considered a baseline level of proiciency, at which students begin to demonstrate the problem-solving competencies that will enable them to participate effectively and productively in 21st-century societies. at this level of proiciency, students engage with an everyday problem, make progress towards a goal, and sometimes achieve it. figure v.2.6 ranks countries and economies by the proportion of 15-year-olds who can complete tasks at least at level 2 dificulty. across oeCd countries, almost four in ive students (79%) are proicient at level 2 or higher. in korea, Japan, macao-China and Singapore, more than nine out of ten students perform at least at this level. by contrast, in six countries, only a minority of 15-year-old students reaches this baseline level of problem-solving performance. in eight countries/economies, level 2 is the most common level of proiciency among students (figure v.2.6 and table v.2.1). Proiciency at Level 1 Students proicient at level 1 can explore a problem scenario only in a limited way; but in contrast with level 2 problemsolvers, they tend to do so only when they have encountered very similar situations before. based on their observations of familiar scenarios, these students are able only to partially describe the behaviour of a simple, everyday device. in general, students at level 1 can solve straightforward problems provided there is only a simple condition to be satisied and there are only one or two steps to be performed to reach the goal. in contrast to students proicient at level 2, level 1 students tend not to be able to plan ahead or set subgoals. across oeCd countries, 92% of 15-year-olds are proicient at level 1 or higher. However, in bulgaria and Colombia, around one in three students does not reach this elementary level of problem-solving proiciency; and in uruguay, the united arab emirates, montenegro, malaysia, brazil and israel, more than one in ive students do not reach this level. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 59 2 STudenT PerFormAnce In Problem SolvIng Proiciency below Level 1 given that the PiSa 2012 problem-solving assessment was not designed to assess elementary problem-solving skills, there were insuficient items to fully describe performance that falls below level 1 on the problem-solving scale. However, it was observed that some students with proiciency below level 1 can use an unsystematic strategy to solve a simple problem set in a familiar context, such as task 1 in sample unit TRAFFIC. they may even ind the solution, provided there are a limited number of well-deined possibilities. on the whole, though, students who are below level 1 show limited problem-solving skills, at best. across oeCd countries, only 8% of students score below 358 points on the PiSa scale, below level 1. in bulgaria, Colombia, uruguay, the united arab emirates, montenegro and israel the proportion of students scoring below level 1 is larger than the proportion of students scoring at any higher level of proiciency – making below level 1 the most common level of proiciency in these six countries. interestingly, in israel, the proportion of students scoring at level 1 (but not higher) is smaller than both the proportion of students who score below level 1 and the proportion of students who score at level 2. this indicates a strong polarisation of results. While in most countries, measures aimed at raising the general level of proiciency will likely beneit students at all levels of the performance distribution, in israel, more targeted measures may be required for students who perform below level 1 (figure v.2.6 and table v.2.1). box v.2.4. Top performers in problem solving as machines and computers are increasingly replacing humans for performing routine tasks, highly skilled workers, who are capable of applying their unique skills lexibly in a variety of contexts, regulating their own learning, and handling novel situations, are more and more in demand. knowing the proportion of 15-year-old students who perform at the highest levels in problem solving allows countries to estimate how well they can respond to this demand. of particular interest is the proportion of students who, in addition to performing at the highest levels in problem solving, also show excellent mastery of speciic subjects. in analyses of PiSa data, the phrase “top performers” refers to students who attain level 5 or 6 in a domain. in problem solving, this corresponds to a performance above 618 score points. figure v.2.7 shows the proportion of top performers in problem solving in each country/economy, as well as the proportion of students who reach a comparable level of proiciency in at least one of the three assessment subjects: mathematics, reading and science. as noted earlier, the ranking of countries and economies by the percentage of top performers in problem solving substantially matches a ranking by mean performance levels. notable exceptions are belgium and israel, which have larger proportions of top performers than other countries of similar or higher mean performance in problem solving. in most countries and economies, most top performers in problem solving are also top performers in other domains. most frequently, top performers in problem solving are also top performers in mathematics. in fact, across oeCd countries, 64% of top performers in problem solving are also top performers in mathematics (table v.2.3). the proportion of students who reach the highest levels of proiciency in at least one domain (problem solving, mathematics, reading or science) can be considered a measure of the breadth of a country’s/economy’s pool of top performers. by this measure, the largest pool of top performers is found in Shanghai-China, where more than half of all students (56%) perform at the highest levels in at least one domain, followed by Singapore (46%), Hong kong-China (40%), korea and Chinese taipei (39%) (table v.2.3). only one oeCd country, korea, is found among the ive countries/economies with the largest proportion of top performers. on average across oeCd countries, 20% of students are top performers in at least one assessment domain. the proportion of students performing at the top in problem solving and in either mathematics, reading or science, too can be considered a measure of the depth of this pool. these are top performers who combine the mastery of a speciic domain of knowledge with the ability to apply their unique skills lexibly, in a variety of contexts. by this measure, the deepest pools of top performers can be found in Singapore (25% of students), korea (21%), Shanghai-China (18%) and Chinese taipei (17%). on average across oeCd countries, only 8% of students are top performers in both a core subject and in problem solving. 60 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng • figure v.2.7 • Top performers in problem solving Percentage of top performers in problem solving and at least one other subject Singapore Korea Japan Hong Kong-China Chinese Taipei Shanghai-China Canada Australia Macao-China Finland Belgium England (United Kingdom) Netherlands Norway Germany France Czech Republic Estonia United States OECD average Austria Italy Ireland Israel Sweden Denmark Slovak Republic Spain Portugal Russian Federation Poland Slovenia Hungary Serbia Croatia United Arab Emirates Turkey Chile Brazil Bulgaria Uruguay Colombia Malaysia Montenegro level 5 level 6 25.0 20.9 16.0 15.9 17.1 17.9 12.0 12.0 12.6 12.0 10.8 9.8 11.5 7.9 9.9 9.5 9.0 9.3 7.5 8.2 8.0 6.2 6.8 6.6 5.6 5.6 6.0 4.4 5.1 4.2 5.7 5.3 4.1 2.8 3.6 1.7 1.8 1.0 0.7 1.2 0.6 0.3 0.5 0.4 0 5 10 15 20 25 30 35 % Countries and economies are ranked in descending order of the percentage of top performers (Levels 5 and 6) in problem solving. Source: oeCd, PiSa 2012 database, tables v.2.1 and v.2.3. 1 2 http://dx.doi.org/10.1787/888933003573 vArIATIon In Problem-SolvIng ProFIcIency When looking at how performance within each country/economy is distributed across the proiciency levels (figure v.2.6), it becomes apparent that the variation observed between students from the same country/economy is, in general, much wider than the variation observed between countries/economies. the standard deviation summarises the distribution of performance among 15-year-olds within each country/economy in a single igure. by this measure, the smallest variation in problem-solving proiciency is found in turkey and macao-China, with standard deviations below 80 score points (figure v.2.8). among top-performing countries, Japan also has a narrow spread of performance (the standard deviation is 85 score points). at the other extreme, israel, bulgaria, belgium and the united arab emirates have the largest variations in problem-solving proiciency, with standard deviations well above 100 score points. the diversity in performance within israel, bulgaria, belgium and the united arab emirates is therefore larger than the diversity that one would expect to ind when sampling a diverse population of students across the 28 oeCd countries that participated in the assessment. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 61 2 STudenT PerFormAnce In Problem SolvIng • figure v.2.8 • variation in problem-solving performance within countries and economies Standard deviation and percentiles on the problem-solving scale Score-point difference between: the 25th and 10th Standard deviation 10th the 50th and 25th 25th the 75th and 50th 50th 45 52 the 90th and 75th 75th 56 90th Percentiles 53 Turkey Turkey 79 Macao-China 79 Malaysia 84 Japan 85 Chile 86 Estonia 88 Portugal 88 Russian Federation 88 Serbia 89 Shanghai-China 90 Italy 91 Chinese Taipei 91 Korea 91 Montenegro 92 Colombia 92 Hong Kong-China 92 Brazil 92 Croatia 92 Denmark 92 United States 93 Finland 93 Ireland 93 Austria 94 Singapore 95 Czech Republic 95 OECD average 96 France 96 Sweden 96 Poland 96 England (United Kingdom) 97 Slovenia 97 Uruguay 97 Australia 97 Slovak Republic 98 Germany 99 67 72 63 50 Netherlands 99 70 69 64 52 Canada 100 Norway 103 Spain 104 Hungary 104 United Arab Emirates 106 Belgium 106 Bulgaria 107 Israel 123 51 50 58 56 57 61 57 59 58 Brazil 68 63 63 66 67 63 67 71 300 350 400 Canada Norway Spain 56 Hungary United Arab Emirates 65 66 Belgium 53 Bulgaria 59 84 450 Netherlands 56 59 77 88 Germany 59 66 74 71 Slovak Republic 55 68 72 Australia 56 64 70 76 250 Uruguay 68 66 80 Slovenia 65 68 73 England (United Kingdom) 55 67 64 68 Poland 52 60 63 65 Sweden 54 62 66 France 55 61 67 OECD average 49 62 64 Czech Republic 53 59 64 Singapore 51 51 62 67 62 62 60 63 Austria 51 68 66 Ireland 53 61 68 Finland 53 60 63 United States 54 61 66 63 Denmark 51 62 63 63 Croatia 60 64 62 Hong Kong-China 56 64 60 53 55 61 69 Korea Colombia 62 57 65 Chinese Taipei 47 Montenegro 64 61 72 57 61 62 73 46 59 61 67 Italy 61 56 62 57 Shanghai-China 49 62 62 55 50 58 65 62 57 Serbia 58 62 60 Russian Federation 51 63 62 Portugal 55 62 61 Estonia 49 60 60 53 50 57 62 63 Japan Chile 59 58 57 47 50 61 57 55 54 60 54 Macao-China Malaysia 58 57 55 44 52 56 53 51 500 Israel 68 550 600 650 700 PiSa score in problem solving Countries and economies are ranked in ascending order of the standard deviation in problem solving. Source: oeCd, PiSa 2012 database, table v.2.2. 1 2 http://dx.doi.org/10.1787/888933003573 62 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng figure v.2.8 also shows how different parts of the performance distribution compare within and across countries and economies. the inter-quartile range – the gap between the top and bottom quarters of the performance distribution – provides another way of measuring differences in performance. on average across oeCd countries, the inter-quartile range is equal to 129 score points. in the countries with the largest variations in problem-solving proiciency (israel, bulgaria and belgium), the gap between the top and bottom quarters of students is more than 14 score points wider than the average gap in oeCd countries (table v.2.2). in many countries, the higher-performing students score closer to the median level of performance than do the lowerperforming students (figure v.2.9). this means that most of the variation is concentrated among low-performing students. in belgium, germany, the netherlands, Spain, france, the Czech republic and korea, the difference between the lowest-performing 10% of students and the median is more than 20 score points larger than the difference between the highest-performing 10% of students and the median. in these countries, many students perform well below the level achieved by a majority of students in the country and drag the mean performance down. • figure v.2.9 • Performance differences among high- and low-achieving students Gaps at the top and bottom end of the distribution of problem-solving performance Variation in performance among high-achieving students is larger than variation in performance among low-achieving students 160 OECD average variation in performance among high-achieving students: Score-point difference between the 90th percentile and the median student 170 150 Israel 140 United Arab Emirates Bulgaria 130 Uruguay Colombia 120 Brazil 100 Hungary Spain Australia Canada United Slovak Republic States Poland Sweden Netherlands Croatia Montenegro 110 Norway Slovenia Belgium OECD average Germany Ireland Russian Federation Denmark Singapore England (United Kingdom) Serbia Turkey Austria Estonia Malaysia Finland France Czech Republic Chile Chinese Taipei Italy Portugal Korea Hong Kong-China Shanghai-China Japan Variation in performance among low-achieving students is larger than variation in performance among high-achieving students Macao-China 90 90 100 110 120 130 140 150 160 170 variation in performance among low-achieving students: Score-point difference between the median student and the 10th percentile Source: oeCd, PiSa 2012 database, table v.2.2. 1 2 http://dx.doi.org/10.1787/888933003573 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 63 2 STudenT PerFormAnce In Problem SolvIng the performance variation in problem solving is not strongly related to mean performance (figure v.2.10). among countries and economies that perform above the oeCd average, Canada and belgium have a wider variation in performance than the oeCd average. by contrast, Japan and macao-China, among the top-performing countries and economies, show a narrow variation in student performance, as do turkey and malaysia, both of whose mean scores are well below the oeCd average. this shows that narrowing differences in performance and fostering excellence are not necessarily conlicting objectives. it is possible to combine high average levels of performance with small variations in performance. • figure v.2.10 • Average performance in problem solving and variation in performance average performance in problem solving is below the oeCd average average performance in problem solving is not statistically different from the oeCd average average performance in problem solving is above the oeCd average Above-average problem-solving performance Above-average variation in performance Above-average problem-solving performance Below-average variation in performance oEcd average average performance in problem solving (in score points) 600 575 550 England (United Kingdom) Australia 525 500 Norway Belgium oEcd average Canada Spain Korea Italy Ireland Denmark Poland Estonia Portugal Russian Federation Serbia Croatia Hungary 450 Macao-China United States Finland Austria Slovenia Israel Chinese Taipei Japan Hong KongChina Czech Shanghai-China Republic France Netherlands Germany Sweden Slovak Republic 475 Singapore Chile Turkey Brazil Malaysia 425 United Arab Emirates Montenegro Uruguay 400 Bulgaria Colombia 375 Below-average problem-solving performance Above-average variation in performance 125 120 115 110 105 100 Below-average problem-solving performance Below-average variation in performance 95 90 85 80 75 Standard deviation in problem-solving performance (in score points) Source: oeCd, PiSa 2012 database, table v.2.2. 1 2 http://dx.doi.org/10.1787/888933003573 Relationship between performance differences and school- and student-level factors the variation in performance within countries can be divided into a measure of performance differences between students from the same school, and a measure of performance differences between groups of students from different schools. figure v.2.11 shows the total variation in performance within each country/economy divided into its betweenschool and within-school components. the data show that there is substantial variation in problem-solving results across schools. on average across oeCd countries, the variation in student performance that is observed within schools amounts to 61% of the oeCd average variation in student performance. the remaining variation (38%) is due to differences in student performance between schools (table v.2.4). 64 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng • figure v.2.11 • Total variation in problem-solving performance and variation between and within schools Expressed as a percentage of the average variation in student performance across OECD countries Total variation as a proportion of the OECD variation 164 118 123 106 120 122 105 102 104 98 95 102 91 100 89 91 100 87 67 89 91 80 118 91 97 91 86 83 101 75 90 102 93 78 92 83 109 114 94 68 83 100 94 variation between schools (as a proportion of total) oEcd average 38% oEcd average 61% Israel Hungary Bulgaria Netherlands United Arab Emirates Belgium Germany Slovenia Slovak Republic Czech Republic Austria Uruguay Brazil OECD average Italy Croatia Poland Shanghai-China Turkey Chinese Taipei Montenegro Chile Spain Colombia Singapore Hong Kong-China Serbia Russian Federation England (United Kingdom) Malaysia Korea Australia United States Japan Denmark Portugal Canada Norway Ireland Macao-China Estonia Sweden Finland variation within schools (as a proportion of total) 100 80 60 40 20 0 20 40 60 80 100 Percentage of variation within and between schools Countries and economies are ranked in descending order of the between-school variation in problem-solving performance as a proportion of the betweenschool variation in performance across OECD countries. Source: oeCd, PiSa 2012 database, table v.2.4. 1 2 http://dx.doi.org/10.1787/888933003573 the variation in performance between schools is a measure of how big “school effects” are. these school effects may have three distinct explanations: irst, they may relect selection mechanisms that assign students to schools; in addition, they may be the result of differences in policies and practices across schools; inally, they may be the traces of local school cultures that originate from interactions among local communities. the between-school variation in student results is therefore not a direct measure of the importance of school policies and practices for student performance in problem solving. However, if the between-school variation is compared across different student characteristics – some sensitive to differences in education policy and practices, such as performance in mathematics, others not, such as socio-economic status – one may infer the extent to which problem-solving results are related to instructional policies and practices. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 65 2 STudenT PerFormAnce In Problem SolvIng Comparing between-school variations figure v.2.12 shows how much of the variation in student performance lies between schools in each country and economy. it shows that problem-solving proiciency, in general, is as closely related to school policies, practices, contextual factors (such as neighbourhood inluences) and peer inluences as is performance in the mathematics assessment. on average across oeCd countries, 38% of the overall variation in problem-solving performance is observed between schools (table v.2.4). this proportion is very similar across assessment domains: it ranges from 36% in science to 38% in reading.4 • figure v.2.12 • between-school differences in problem-solving performance, mathematics performance and socio-economic status Problem solving mathematics PiSa index of economic, social and cultural status (eSCS) Proportion of variation between schools as a percentage of the overall (within and between school) variation 80 70 60 50 40 30 20 10 Hungary Bulgaria Netherlands Slovenia Germany Israel Turkey Slovak Republic Czech Republic United Arab Emirates Austria Belgium Italy Brazil Chile Uruguay Croatia Shanghai-China Montenegro Chinese Taipei OECD average Serbia Malaysia Colombia Hong Kong-China Russian Federation Japan Poland Korea Singapore Portugal United States England (United Kingdom) Spain Denmark Macao-China Ireland Australia Estonia Canada Norway Finland Sweden 0 Countries and economies are ranked in ascending order of the proportion of variation in problem-solving performance that lies between schools. Source: oeCd, PiSa 2012 database, table v.2.4. 1 2 http://dx.doi.org/10.1787/888933003573 one might expect the proportion of variation in performance observed between schools to be smaller in problem solving than in mathematics, reading and science. first, the skills required in the PiSa assessment of problem solving are not taught as a speciic school subject in most countries, in contrast to those required in mathematics, reading and science. Second, assessments of problem solving are not explicitly used in high-stakes examinations that inluence decisions about selecting students for different classes or schools, where these exist. Yet the association between differences in instruction and selection mechanisms and performance in problem solving is as strong as the association between instruction and selection and performance in mathematics, reading and science. to compare the between-school variation across subjects and student characteristics the ratio of the between-school variation to the sum of the between- and within-school variation is computed. the within-school variation estimates how diverse students are within each school, on average. the between-school variation estimates how far the grouping of students across schools is from a random allocation of students to schools. low levels of between-school variation (relative to the overall within- and between-school variation) indicate inclusion: within the limits given by its size, each school’s diversity mirrors the level of diversity that exists in the country overall. large proportions of variation between schools signal segregation: students tend to be grouped together only with students who are similar to them in the characteristic being examined. 66 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng While, in general, the inluence of schools is as strong on performance in problem solving as for performance in curricular subjects, in some countries, the school seems to matter more for problem solving. in denmark, israel, norway, Poland, the russian federation and Spain, for instance, performance in problem solving is more strongly associated with schools than performance in mathematics. in these countries, strong performers and poor performers in problem solving are more clearly sorted across different schools than strong and poor performers in mathematics. Conversely, in Japan, the netherlands, Serbia and turkey, students tend to be sorted across schools according to their mathematics level, but less so according to their performance in problem solving. all four of these countries have below-average levels of academic inclusion (as indicated by large variations in mathematics performance between schools). in these countries, however, problem-solving results are more similar between schools than are results in mathematics. the between-school variation, on the other hand, is much larger in student outcome measures – such as reading, mathematics, or indeed problem solving – than in student background factors that inluence performance, such as the PISA index of economic, social and cultural status (eSCS). only 24% of the socio-economic variation lies between schools, on average across oeCd countries. this means that in most countries, students within the same school tend to be more diverse in their socio-economic status than in their performance (table v.2.4). by comparing the variation between schools in the socio-economic status of students with the between-school variation in performance, one can gauge the importance of classroom interactions between teachers and students, or among students themselves, in shaping performance. indeed, one could argue that the proportion of socio-economic variation between schools relects residential segregation and school selection practices, and is not inluenced by teacher-student or student-student relations. over the course of a school year, this proportion will remain ixed. Performance, in addition to being inluenced by these factors, will evolve over time. in particular, even if the allocation of pupils to schools remains the same, it is expected that over the course of schooling, differences in the quality of teaching create additional between-school variation in student performance. the fact that the proportion of variation between schools is, in most countries, larger in problem-solving performance than in socio-economic status, is evidence that school-level factors are as important in explaining problem-solving performance as they are in explaining performance in mathematics or reading. there is only one exception: in Chile, the between-school variation in student performance (in all subjects) is smaller than the between-school variation in socioeconomic status. this means that the school that a student attends says more about his or her socio-economic status than about his or her performance. in other countries and economies, such as finland, Portugal and the united States, the pattern is less clear: the observed between-school variation in problem-solving performance is similar to the betweenschool variation in students’ socio-economic status (figure v.2.12 and table v.2.4). STudenT PerFormAnce In Problem SolvIng comPAred wITh PerFormAnce In mAThemATIcS, reAdIng And ScIence a key distinction between the PiSa 2012 assessment of problem solving and the regular assessments of mathematics, reading and science is that the problem-solving assessment does not measure domain-speciic knowledge; rather, it focuses as much as possible on the cognitive processes fundamental to problem solving. However, these processes can also be used and taught in the other subjects assessed. for this reason, problem-solving tasks are also included among the test units for mathematics, reading and science, where their solution requires expert knowledge speciic to these domains, in addition to general problem-solving skills. it is therefore expected that student performance in problem solving is positively correlated with student performance in mathematics, reading and science. this correlation hinges mostly on generic skills, and should thus be about the same magnitude as between any two regular assessment subjects. the following sections examine the correlations between problem-solving performance and performance in mathematics, reading, and science. they then identify countries whose students’ performance in problem solving is better than that of students around the world who share their level of proiciency in mathematics, reading and science. the chapter concludes with a discussion of the effects of computer delivery of the assessment on performance differences within and between countries. Correlation between performance in mathematics, reading and science, and performance in problem solving Students who do well in problem solving are likely to do well in other areas as well, and students who have poor problemsolving skills are likely to do poorly in other subjects assessed. figure v.2.13 shows the strength of the relationship Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 67 2 STudenT PerFormAnce In Problem SolvIng between the three regular PiSa domains and student performance in problem solving. the largest correlation is between mathematics and problem solving (0.81); the smallest is between reading and problem solving (0.75). these correlations may appear large, but they are smaller than the correlation observed among mathematics, reading and science.5 • figure v.2.13 • relationship among problem-solving, mathematics, reading and science performance OECD average latent correlation, where 0.00 signiies no relationship and 1.00 signiies the strongest positive relationship latent correlation between: reading mathematics 0.81 0.75 0.85 Science and… 0.78 0.90 0.88 Problem solving mathematics reading Source: oeCd, PiSa 2012 database, table v.2.5. 12 http://dx.doi.org/10.1787/888933003573 • figure v.2.14 • variation in problem-solving performance associated with performance in mathematics, reading and science variation associated with more than one subject variation uniquely associated with mathematics performance variation uniquely associated with reading performance variation uniquely associated with science performance residual (unexplained) variation Colombia Russian Federation Spain Japan Italy Hong Kong-China Denmark Canada Poland Norway Macao-China Uruguay Portugal Ireland Austria Montenegro Chile Sweden Korea United Arab Emirates Belgium Bulgaria OECD average Slovenia Brazil Singapore Serbia France Malaysia Hungary Turkey Shanghai-China Australia Germany Finland Estonia Croatia Slovak Republic England (United Kingdom) United States Netherlands Israel Chinese Taipei Czech Republic 0 10 20 30 40 50 60 70 80 90 100 Percentage of variance explained Countries and economies are ranked in ascending order of the total percentage of variance explained in problem solving. Source: oeCd, PiSa 2012 database, table v.2.5. 1 2 http://dx.doi.org/10.1787/888933003573 68 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng Comparing the strength of the association among the skills measured in PiSa clearly proves that problem solving constitutes a separate domain from mathematics, reading and science. that the skills measured in the problem-solving assessment are those that are used in a wide range of contexts is conirmed by an analysis that relates the variation in problem-solving performance jointly to the variation in performance in mathematics, reading and science (figure v.2.14). on average, about 68% of the problem-solving score relects skills that are also measured in one of the three regular assessment domains.6 the remaining 32% relects skills that are uniquely captured by the assessment of problem solving. of the 68% of variation that problem-solving performance shares with other domains, the overwhelming part is shared with all three regular assessment domains (62% of the total variation); about 5% is uniquely shared between problem solving and mathematics only; and about 1% of the variation in problem solving performance hinges on skills that are speciically measured in the assessments of reading or science (table v.2.5). figure v.2.14 also shows that the association of problem-solving skills with performance in mathematics, reading and science is, in general, of similar strength across countries and economies. Comparatively weak associations between the skills measured in the problem-solving assessment and performance in mathematics, reading and science are found in Colombia, the russian federation, Spain, Japan, italy and Hong kong-China. in these countries and economies, more than in others, performance differences in problem solving do not necessarily match performance differences in core domains: some students who rank highly in, say, mathematics or reading, perform poorly in problem solving; conversely, some students who perform poorly in the core subjects still demonstrate high problem-solving proiciency. Students’ performance in problem solving relative to students with similar mathematics, reading and science skills the strong positive correlations across domains indicate that, in general, students who perform at higher levels in mathematics, reading or science also perform well in problem solving. there are, however, wide variations in problemsolving performance for any given level of performance in the core domains assessed by PiSa. this section uses this variation to assess country performance by comparing students from each country with students in other countries who have similar scores in mathematics, reading and science.7 • figure v.2.15 • relative performance in problem solving Score-point difference between actual and expected performance in problem solving 20 10 Students’ performance in problem solving is higher than their expected performance 0 -10 -20 -30 -40 -50 Students’ performance in problem solving is lower than their expected performance Korea Japan Serbia United States Italy England (UK) Macao-China Brazil Australia France Singapore Norway Chile Czech Republic Canada Sweden Portugal Russian Federation Slovak Republic Austria Colombia OECD average Finland Chinese Taipei Belgium Denmark Germany Malaysia Turkey Estonia Netherlands Hong Kong-China Ireland Spain Croatia Montenegro Uruguay Israel Slovenia Hungary United Arab Emirates Poland Shanghai-China Bulgaria -60 Notes: Signiicant differences are shown in a darker tone (see annex a3). each student’s expected performance is estimated, using a regression model, as the predicted performance in problem solving given his or her score in mathematics, reading and science. Countries and economies are ranked in descending order of the score-point difference between actual and expected performance. Source: oeCd, PiSa 2012 database, table v.2.6. 1 2 http://dx.doi.org/10.1787/888933003573 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 69 2 STudenT PerFormAnce In Problem SolvIng relative performance in problem solving is estimated by comparing students’ actual performance to the performance predicted by a regression model that estimates, for each student, the expected performance in problem solving depending on the performance in the three core domains. figure v.2.15 shows a ranking of countries/economies in relative performance. in nine countries and economies, students perform signiicantly better, on average, in problem solving than students in other countries with similar skills in mathematics, reading and science. of the 19 countries and economies whose mean performance is above the oeCd average, korea, Japan, the united States, italy, england (united kingdom), macao-China and australia have a speciic strength in problem solving. in brazil and in Serbia, students perform above the level attained by students of similar strength in the core assessment domains, on average; but this above-average relative performance in problem solving is not suficient to raise the countries’ mean absolute performance above the oeCd average. in korea, Japan, Serbia and the united States, the difference between students’ scores in problem solving and their expected performance given their scores in mathematics, reading and science, exceeds 10 score points. in korea, 61% of students outperform other students assessed in PiSa with similar performance in core subjects on the problem-solving assessment (figure v.2.15 and table v.2.6). in more than 20 countries and economies, students perform below par in problem solving, on average, when compared to students in the other participating countries and economies who display the same level of proiciency in mathematics, reading and science. in bulgaria, Shanghai-China, Poland and the united arab emirates, the difference exceeds 40 score points. in Shanghai-China, 86% of students perform below the expected level in problem solving, given their performance in mathematics, reading and science. Students in these countries/economies struggle to use all the skills that they demonstrate in the other domains when asked to perform problem-solving tasks. in six other countries/economies, problem-solving performance falls short of its expected level, given students’ performance in mathematics, reading and science, by between 20 and 40 score points: Hungary (34 score points), Slovenia (34 points), israel (28 points), uruguay (27 points), montenegro (24 points) and Croatia (22 points). Spain, ireland, Hong kong-China, the netherlands, estonia, turkey, malaysia, germany, denmark, belgium, Chinese taipei, finland and Colombia show smaller gaps. all these countries/ economies could improve their performance in problem solving if their students performed at the same level as students in other countries/economies who demonstrate similar skills in mathematics, reading and science (figure v.2.15 and table v.2.6). Students’ performance in problem solving at different levels of performance in mathematics figure v.2.16 shows the average problem-solving performance of students at different levels of mathematics proiciency. by comparing the performance of students from one country to the average performance observed across participating countries/economies at a given level of proiciency in mathematics, shown in figure v.2.16, one can infer whether these students perform the same as, above or below students with similar proiciency in mathematics. is the relatively strong performance in problem solving observed in some countries mainly due to the ability of some students at the bottom of the class to perform above expectations in problem solving, or to the good performance in problem solving among students who perform at or above level 4 in mathematics? the answer varies greatly by country. figure v.2.17 illustrates nine possible patterns and shows which pattern prevails in each of the participating countries and economies, based on results reported in table v.2.6. in italy, Japan and korea, the good performance in problem solving is, to a large extent, due to the fact that lowerperforming students score beyond expectations in the problem-solving assessment. in italy and Japan, students with strong mathematics skills perform on a par with students in other countries that share the same mathematics proiciency; but students who score at low or moderate levels in mathematics have signiicantly better problem-solving skills than students in other countries with similar levels of mathematics proiciency. this may indicate that some of these students perform below their potential in mathematics; it may also indicate, more positively, that students at the bottom of the class who struggle with some subjects in school are remarkably resilient when it comes to confronting real-life challenges in non-curricular contexts (figure v.2.17). in contrast, in australia, england (united kingdom) and the united States, the best students in mathematics also have excellent problem-solving skills. these countries’ good performance in problem solving is mainly due to strong performers in mathematics. this may suggest that in these countries, high performers in mathematics have access to – and take advantage of – the kinds of learning opportunities that are also useful for improving their problem-solving skills. 70 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng • figure v.2.16 • expected performance in problem solving, by mathematics performance expected performance in problem solving, at different levels of performance in mathematics Percentile correspondence between problem solving and mathematics Problem-solving performance (in score points) 800 99th percentile 700 Average performance in problem solving among students performing at the 95th percentile in mathematics (626 score points, or 92nd percentile in problem solving) 95th percentile 90th percentile 600 75th percentile 500 50th percentile 25th percentile 400 10th percentile 5th percentile 300 95th percentile in mathematics performance (649 score points) 1st percentile 200 200 300 400 500 600 700 800 mathematics performance (in score points) Notes: the blue line shows students’ expected problem-solving performance at each level of proiciency in mathematics. this conditional expectation line is estimated with local linear regression on the pooled international sample of students (see annex a3). the black line shows the correspondence between percentiles of performance in problem solving and percentiles of performance in mathematics. Percentiles are estimated on the pooled international sample of students. the comparison of the two lines indicates a certain amount of “mean reversion”. for instance, students performing at the 95th percentile in mathematics perform at the 92nd percentile in problem solving, on average, and thus closer to the international mean. this observed mean reversion is as expected for two partially independent skills. Source: oeCd, PiSa 2012 database. 1 2 http://dx.doi.org/10.1787/888933003573 there are similar differences among countries with overall weak performance in problem solving, relative to their students’ performance in mathematics. in several of these countries, speciic dificulties in problem solving are most apparent among students with poor mathematics skills, and students with strong mathematics skills often perform on or close to par with students in other countries/economies. these countries are shown in the top-right cell in figure v.2.17. in other countries, weak performance in problem solving, relative to mathematics performance, is mainly due to strong performers in mathematics who demonstrate lower proiciency in problem solving than do similarly proicient students in other countries/economies. this may indicate that in these countries and economies, high performers in mathematics are not exposed to the learning opportunities that could also help them to develop their problem-solving skills. they are shown in the bottom-right cell in figure v.2.17. Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 71 2 STudenT PerFormAnce In Problem SolvIng • figure v.2.17 • Patterns of relative performance in problem solving Average performance compared to students with similar scores in mathematics Stronger In line with 500 400 500 500 600 700 mathematics score 300 Australia, England (United Kingdom), United States 400 500 600 700 mathematics score Problem-solving score 500 300 400 500 600 700 mathematics score 300 500 600 700 mathematics score Italy, Japan, Korea 500 600 700 mathematics score Austria, Belgium, Malaysia, Montenegro, Poland, Shanghai-China, Singapore, Slovak Republic, Uruguay Problem-solving score 600 500 300 400 700 600 500 400 300 400 500 300 400 400 300 600 700 mathematics score 600 Chile, France, Sweden Problem-solving score 500 500 400 700 600 400 Bulgaria, Colombia, Croatia, Denmark, Estonia, Germany, Hungary, Ireland, Israel, Netherlands, Slovenia, Spain, United Arab Emirates 300 400 700 Problem-solving score 300 700 600 Brazil, Serbia lower among strong performers in mathematics 600 700 mathematics score 400 300 300 500 Canada, Czech Republic, Finland, Norway Problem-solving score Problem-solving score 500 500 300 400 700 600 600 400 300 400 700 Similar at all levels of mathematics performance 600 400 300 300 700 Problem-solving score 600 300 Weaker 700 Problem-solving score Problem-solving score higher among strong performers in mathematics 700 300 400 500 600 700 mathematics score Macao-China, Portugal 300 400 500 600 700 mathematics score Hong Kong-China, Russian Federation, Chinese Taipei, Turkey Notes: the dotted line is repeated across all graphs and shows the average performance in problem solving, across students from all participating countries/economies, at different levels of performance in mathematics (see figure v.2.16). the continuous line illustrates nine possible patterns of relative performance in problem solving. numbers on the axes refer to score points in the respective assessment domains. figures are for illustrative purposes only. Countries and economies are grouped according to the direction and signiicance of their relative performance in problem solving, compared with students around the world with similar scores in mathematics, and of their difference in relative performance between students performing at or above level 4 and students performing below level 4 in mathematics. Source: oeCd, PiSa 2012 database, table v.2.6. 1 2 http://dx.doi.org/10.1787/888933003573 72 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng The inluence of computer delivery on performance in problem solving the assessment of problem solving in PiSa 2012 was designed and delivered on a computer platform. as explained in Chapter 1, this allowed for a wider deinition of problem-solving competency – one that includes the willingness and capacity to explore an unknown environment to gather information about it. Students participating in the PiSa assessment of problem solving differ by how familiar they are with computers and with using computers as an assessment instrument. for some students, using computers may have increased test anxiety; for others, the use of computers may have had the opposite effect. for some, a lack of basic familiarity with a keyboard or mouse might have hindered their ability to complete the assessment in the time allotted. in part, variation in performance on the problem-solving test may result from differences in computer skills. these differences may have inluenced both the performance rankings within countries and the rankings among countries. How strong is this inluence? it can be gauged by comparing results in problem solving with results on the computer-based test of mathematics, on the one hand, and with results on the paper-based tests in mathematics, on the other hand. Students who perform below their expected level across all computer-based tests may have a generic dificulty with basic computer skills, rather than a particular weakness in problem solving. the proportion of variation in problem solving that is uniquely explained by performance differences in computer-based assessments, after accounting for differences in paper-based assessments, is a measure of the importance of the mode of delivery for rankings of students and schools within countries and economies. by this measure, the inluence of the computer delivery on within-country/economy rankings appears to vary markedly across countries and economies. in Japan, the russian federation, denmark, norway, france and Poland more than 5% of the variation in performance on the problem-solving test can be explained by the mode of delivery. in contrast, in Chile, ireland, Singapore, Chinese taipei and the united States, less than 1% of the variation in performance in problem solving across students is explained by differences in computer skills (figure v.2.18). • figure v.2.18 • Inluence of computer skills on the ranking of students within countries/economies Variation in problem-solving performance uniquely associated with performance on computer-based assessments, after accounting for performance on paper-based assessments variation in problem-solving performance explained by the mode of delivery, as a percentage of total variation 10 9 8 7 6 5 4 3 2 1 Japan Russian Federation Denmark France Norway Poland Spain Hong Kong-China Israel Sweden Slovenia OECD average Portugal Colombia Italy Australia Brazil Korea Hungary Macao-China Austria Germany Belgium Canada Shanghai-China United Arab Emirates Estonia United States Slovak Republic Chinese Taipei Singapore Chile Ireland 0 Note: only countries/economies that participated in the computer-based assessment of mathematics are included in this igure. Countries and economies are ranked in ascending order of the variation in problem-solving performance explained by computer skills. Source: oeCd, PiSa 2012 database, table v.2.5. 1 2 http://dx.doi.org/10.1787/888933003573 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v © OECD 2014 73 2 STudenT PerFormAnce In Problem SolvIng the mode of delivery also bears an inluence on between-country comparisons. figure v.2.19 shows that in most countries with a relative weakness in problem-solving performance, this weakness is compounded by a more general weakness on computer-based assessments, which can be ascribed to the mode of delivery. indeed, almost all of the country-level gaps between students’ actual performance and their expected performance shrink when the comparison accounts for scores on the computer-based assessment of mathematics, rather than on the paper-based assessment of mathematics. nevertheless, in most cases, whether the country shows a relative strength or weakness in problem solving after accounting for performance in mathematics does not depend on whether the comparison is with students’ performance on the paper-based test or on the computer-based test. this indicates that country-level computer mode effects are only part of the relative performance in problem solving discussed earlier in this chapter. one may even argue that the computer skills signalled by mode effects are related to actual problem-solving skills, such as the willingness and capacity to interact with unknown devices. • figure v.2.19 • Inluence of computer skills on relative performance in problem solving Average performance difference with students who have similar scores in computer-based mathematics Average performance difference with students who have similar scores in paper-based mathematics Score-point difference between actual and expected performance in problem solving 40 Students’ performance in problem solving is higher than their expected performance 20 0 -20 -40 -60 United Arab Emirates Hungary Shanghai-China Slovenia Slovak Republic Poland Colombia Brazil Hong Kong-China Belgium Israel Spain Russian Federation Austria Denmark Estonia Germany Chinese Taipei OECD average France Macao-China Ireland Portugal Norway Chile Canada Singapore Italy United States Korea Japan Australia Sweden Students’ performance in problem solving is lower than their expected performance -80 Notes: Statistically signiicant differences are shown in darker tones (see annex a3). only countries/economies that participated in the computer-based assessment of mathematics are included in this igure. The lines connecting diamonds and bars show the influence of computer skills on relative performance in problem solving. Countries are ranked in descending order of the score-point difference between actual and expected performance, given students’ scores on the computerbased assessment of mathematics. Source: OECD, PISA 2012 Database, Table V.2.6. 1 2 http://dx.doi.org/10.1787/888933003573 74 © OECD 2014 Creative Problem Solving: StudentS’ SkillS in taCkling real-life ProblemS – volume v 2 STudenT PerFormAnce In Problem SolvIng Notes 1. In particular, a student has a probability of 0.62 of correctly answering an item at the same point on the scale. The width of each proiciency level described below is set so that, for a test composed entirely of questions spread uniformly across a level, all students whose scores fall within that level would be expected to get at least 50% of the questions correct. In particular, students who are at the lower score limit for a level are expected to get exactly 50% of the questions of this level correct. 2. Technically, the mean score for student performance in problem solving across OECD countries was set at 500 score points and the standard deviation at 100 score points, with the data weighted so that each oeCd country contributed equally. the average standard deviation of the problem-solving scale across oeCd countries, reported in the appendix tables, is less than 100 score points, because it is computed as the arithmetic average of the countries’ individual standard deviations. this reported measure is based only on variation of performance within countries, and does not include the performance variation across countries. the standard deviation of 100 used for standardising scores, on the other hand, is a measure of overall variation within and between oeCd countries. 3. Conidence level of 95% for pairwise comparisons. 4. this proportion is known as the intra-class correlation coeficient in multi-level analyses and relates to the “index of inclusion” reported in table v.2.4. 5. note also that the correlations reported are latent correlations, which are not attenuated by measurement error. 6. Correlation and explained variance are strictly related concepts. a correlation of around 0.81 between problem solving and mathematics implies, for instance, that about two-thirds of the variation in problem-solving performance (0.81 × 0.81 = 0.66) is common across the two domains of mathematics and problem solving. 7. “Students in other countries” refers to all 15-year-old students in countries that participated in the PiSa assessment of problem solving. most (54%) of these students are in just ive countries: the united States (21%), brazil (14%), the russian federation (7%), Japan (7%) and turkey (5%). References Philpot, R. et al. (forthcoming), “factors that inluence the dificulty of problem solving items”, Chapter 8 in Csapó, b. and J. funke (eds.), The Nature of Problem Solving, oeCd Publishing. CrEATIVE PrOblEm SOlVIng: STuDEnTS’ SkIllS In TACklIng rEAl-lIfE PrOblEmS – VOlumE V © OECD 2014 75 3 Students’ Strengths and Weaknesses in Problem Solving This chapter provides a nuanced look at student performance in problem solving by focusing on students’ strengths and weaknesses in performing certain types of tasks. The items in the PISA problem-solving assessment are categorised by the nature of the problem (interactive or static items) and by the main cognitive processes involved in solving the problem (exploring and understanding; representing and formulating; planning and executing; monitoring and reflecting). The analysis in this chapter identifies the tasks and skills that students master better than students in other countries do, after taking into account overall differences in performance. CrEATIVE PrOblEm SOlVIng: STuDEnTS’ SkIllS In TACklIng rEAl-lIfE PrOblEmS – VOlumE V © OECD 2014 77 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng This chapter takes a more nuanced look at problem-solving performance by analysing how students interact with the test items. It focuses on performance profiles, rather than on performance levels, in order to identify each country’s/economy’s comparative strengths and weaknesses. The PISA problem-solving framework defines a broad construct. Problem-solving competence in PISA encompasses success with different types of problems and the mastery of several distinct cognitive processes. This chapter analyses strengths and weaknesses in problem-solving by breaking down overall performance into success rates according to broad types of tasks (box v.3.1).1 Why are students from certain countries particularly good at problem solving? the analysis in this chapter identifies the tasks and skills that these students master better than students in other countries. in doing so, it highlights, for each country/economy, the specific areas of problem solving with the greatest margin for improvement, thus suggesting priorities for improving curricula and teaching practices to foster students’ capacity to solve problems in real life. what the data tell us • Students in Hong kong-China, korea, macao-China, Shanghai-China, Singapore and Chinese taipei perform strongest on problems that require understanding, formulating or representing new knowledge, compared to other types of problems. • Students in brazil, ireland, korea and the united States perform strongest on interactive problems (those that require the student to uncover some of the information needed to solve the problem) compared to static problems (those that have all information disclosed at the outset). box v.3.1. how item-level success is reported PiSa reports the performance of all students on the problem-solving assessment on a common scale, despite the fact that different subsets of students are administered different items, depending on the test booklet they receive. the item-response model that underlies the scaling of students’ answers makes it possible to aggregate students’ answers into an overall score even if each student sees only a subset of the entire PiSa item pool (see annex a5 and oeCd, forthcoming). While this approach has many advantages, it can potentially hide interesting differences in patterns of performance at lower levels of aggregation, i.e. on single items or on subsets of items. to explore these patterns, one must use the unscaled responses of the students who answered each item. in this chapter, average percentages of correct responses are computed at the country/economy level. for each item, the percentage of correct responses is simply the number of correct (full credit) answers divided by the number of students who encountered the question (non-reached questions are counted as incorrect answers). the average percentage of correct responses on a particular group of items, or on the complete pool of problemsolving items, is then the simple average of item-by-country/economy percentages of correct responses. on average across countries, the percentage of correct responses is a measure of the dificulty of items. by comparing the percentage of correct responses across two distinct sets of items, one can identify the relative dificulty of each set. by further comparing the percentage of correct responses across two sets of items and across countries, one can identify where the relative strengths and weaknesses of each country lie. for each subset of items and for each country/economy, the result of this comparison is reported as an odds ratio. ratios equal to 1 for Country a, for instance, indicate that the pattern of performance across items is in line with the average oeCd pattern of performance. ratios above the value of 1 indicate that the items in this subset were easier for students in Country a than, on average, for students across oeCd countries, after accounting for overall differences in performance across the test. a ratio of 1.2, for instance, indicates that full-credit answers within this subset were 1.2 times more prevalent than on average across oeCd countries, after accounting for overall performance differences. ratios below the value of 1 indicate that the items in this subset were, on average, harder than expected for students in Country a: the pattern of performance corresponds to a country-speciic weakness on this subset of items. 78 © OECD 2014 CrEATIVE PrOblEm SOlVIng: STuDEnTS’ SkIllS In TACklIng rEAl-lIfE PrOblEmS – VOlumE V 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng The remainder of this chapter discusses in more detail the two main framework aspects (the nature of the problem situation, and problem-solving processes), and compares the performance profiles of countries within each aspect. It also links the framework aspects to skill demands and derives implications for teachers and curriculum developers. FrAmeworK ASPecTS And relATIve SucceSS oF STudenTS In eAch AreA The PISA problem-solving framework provides the basis for the analyses in this chapter. The framework was used to develop items that vary by the nature of the problem situation and by the particular problem-solving process targeted (see Chapter 1 and oeCd, 2013). together, the 42 items included in the test, which also vary by problem context, by difficulty and by response format, are representative of the problem-solving domain as defined in PiSa. the problemsolving proficiency scale summarises overall performance on the test. instead of focusing on the overall proficiency in problem solving, this chapter analyses performance on subsets of items in order to identify systematic differences, across countries, in students’ success in handling different families of tasks. the PiSa 2012 problem-solving framework organises the domain around two main aspects. a first important distinction among problem-solving items is between interactive and static items; this is referred to as the nature of the problem situation. a second important distinction between items is related to the main cognitive processes involved in problem solving. each process is defined by a pair of verbs: exploring and understanding; representing and formulating; planning and executing; monitoring and reflecting. figure v.3.1 presents an overview of the classification of items according to their characteristics. a statistical analysis2 confirms that the test was constructed so that there is no strong association between the main cognitive process involved in the task and the static or interactive nature of the problem situation. as a consequence, strengths and weaknesses in particular cognitive processes are unlikely to influence strengths and weaknesses that are found in interactive or static tasks. • figure v.3.1 • number of tasks, by framework aspect Problem-solving process Exploring and understanding (10 items) representing and formulating (9 items) Planning and executing (16 items) monitoring and relecting (7 items) Static (15 items) 5 2 6 2 Interactive (27 items) 5 7 10 5 nature of the problem situation Source: oeCd, PiSa 2012 database. in addition to these two aspects, each assessment unit is also characterised, on a more superficial level, by the particular context in which the problem situation occurs. the framework distinguishes problems with a social focus from problems with a personal focus, as well as problems cast in a technological setting from problems cast in a non-technological setting. items in the problem-solving test can also be classified according to their response format. a major distinction is between selected-response formats, which ask respondents to choose one or more answers from a closed list of possible responses, and constructed-response formats, where students produce a self-constructed response. Nature of the problem situation How a problem is presented has important consequences for how it can be solved. of crucial importance is whether the information about the problem disclosed at the outset is complete. these problem situations are considered static. Question 3 in the problem-solving unit TRAFFIC, described in the sample tasks section at the end of Chapter 1, is an example of a static unit: students are given all information about travel times and have to determine the best location for a meeting. by contrast, problem situations may be interactive, meaning that students can explore the situation to uncover additional relevant information. real-time navigation using a gPS system, where traffic congestion may be reported in response to a query, is an example of such a situation. CrEATIVE PrOblEm SOlVIng: STuDEnTS’ SkIllS In TACklIng rEAl-lIfE PrOblEmS – VOlumE V © OECD 2014 79 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng Interactive problem situations Interactive problem situations often arise when encountering technological devices, such as ticket-vending machines, air-conditioning systems or mobile telephones for the first time, especially if the instructions for using them are not clear or are not available. Individuals often confront these types of problems in daily life. In these situations, some relevant information is often not apparent at the outset. for example, the effect of performing an operation (say, pushing a button on a remote control) may not be known and cannot be deduced, but rather must be inferred by actually performing the operation (pushing the button) and forming a hypothesis about its function based on the outcome. In general, some exploration or experimentation is needed to acquire the knowledge necessary to control the device. Another common scenario is when a person must troubleshoot a fault or malfunction in a device. Here a certain amount of strategic experimentation – generating and testing hypotheses – must take place in order to collect data on the circumstances under which the device fails. Interactive problem situations can be simulated in a test setting by a computer. Including interactive problem situations in the computer-based PiSa 2012 problem-solving assessment allows for a wider range of authentic, real-life scenarios to be presented than would otherwise be possible using pen-and-paper tests. Problems where the student explores and controls a simulated environment are a distinctive feature of the assessment. Static problem situations in static problems all relevant information is disclosed at the outset and the problem situation is not dynamic, i.e. it does not change during the course of solving the problem. examples of static problems are traditional logic puzzles, such as the tower of Hanoi and the water jars problems (“How would you use three jars with the indicated capacities to measure out the desired amount of water?”); decision-making problems, where the student is required to understand a situation involving a number of well-defined alternatives and constraints so as to make a decision that satisfies the constraints (e.g. choosing the right pain killer given sufficient details about the patient, the complaint and the available pain killers); and scheduling problems for projects, such as building a house or generating a flight schedule for an airline, where a list of tasks with durations and relationships between tasks is given. figure v.3.2 illustrates how the nature of the problem situation varies across the PiSa 2012 problem-solving items that were made public. While all of the interactive units shown in figure v.3.2 are set in technology contexts, the assessment also included interactive problems in non-technology contexts; for instance, some items ask students to orient themselves in a maze. overall, a majority of items – 27 of 42 – are interactive. • figure v.3.2 • examples of problem-solving tasks, by nature of the problem nature of the problem situation Interactive Sample questions MP3 PLAYER – items 1, 2, 3 and 4 (field trial) CLIMATE CONTROL – items 1 and 2 TICKETS – items 1, 2 and 3 Static TRAFFIC – items 1, 2 and 3 ROBOT CLEANER – items 1, 2 and 3 Source: oeCd, PiSa 2012 database. What success on interactive tasks implies for education policy and practice the static or interactive nature of the problem situation is related to how information is presented. Static problems, where all relevant information is disclosed at the outset, are the typical textbook problems encountered in schools, whereas in most contexts outside of schools, the relevant information to solve the problem has to be obtained by interacting with the environment. Static problems can be regarded as a special case of interactive problems. this highlights the fact that the set of skills that are required to solve static tasks is a subset of the skills required for interactive tasks. to excel in interactive tasks, it is not sufficient to hold the problem-solving skills required by static, analytical problems; students must also be open to novelty, tolerate doubt and uncertainty, and dare to use intuitions (“hunches and feelings”) 80 © OECD 2014 CrEATIVE PrOblEm SOlVIng: STuDEnTS’ SkIllS In TACklIng rEAl-lIfE PrOblEmS – VOlumE V 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng to initiate a solution. A relatively weak performance on interactive items, compared to performance on static items, may indicate that students may benefit from greater opportunities to develop and exercise these traits, which are related to curiosity, perseverance and creativity. Success on interactive and static tasks figure v.3.3 plots average success rates for interactive items against average success rates for static items. the figure immediately reveals that, in general, country rankings are similar across the two types of items. Performance on interactive items is strongly related to performance on static items. However, as figure v.3.3 shows, performance is not always perfectly aligned. Countries that share similar levels of success on static items do not necessarily share the same performance on interactive items. often, when considering two countries with similar performance on static items, one country is significantly stronger on interactive items than the other. • figure v.3.3 • differences in countries’/economies’ success on problem-solving tasks, by nature of the problem interactive and static items Average percentage of full-credit responses for interactive items 70 60 50 Ireland 40 Sweden 30 20 10 0 0 10 20 30 40 50 60 70 Average percentage of full-credit responses for static items Note: Ireland and Sweden share similar levels of performance overall, but illustrate different patterns of performance across interactive and static items; this example is discussed in the text. Source: OECD, PISA 2012 Database, Table V.3.1. 1 2 http://dx.doi.org/10.1787/888933003592 in ireland, for instance, the percentage of full-credit answers was, on average, 44.6% across all items. this resulted from a 44.4% success rate on static items and a 44.6% success rate on interactive items. because interactive items were found to be slightly harder than static items, on average across oeCd countries, it can be deduced that performance on interactive items was stronger than expected in ireland. in comparison, the success rate of students in Sweden (43.8%) was similar to that of students in ireland overall, but this resulted from a higher success rate on static items (47.7%) and a lower success rate on interactive items (41.6%). While the former is in line with the oeCd average, the latter is significantly below the oeCd average (figure v.3.3 and table v.3.1). figure v.3.4 ranks countries and economies according to whether their students had greater success on interactive or on static tasks, after accounting for overall differences in performance. this analysis accounts for the relative difficulty of static and interactive tasks by comparing relative success in each country/economy to the average relative success across oeCd countries. it also adjusts for country/economy-specific response format effects (figure v.3.9). to continue with the same example used above, the measure of relative success on interactive items is 1.16 in ireland – and thus significantly above 1, indicating stronger-than-expected performance on interactive items. relative success is only 0.91 in Sweden (significantly below par), indicating weaker-than-expected performance on interactive items (table v.3.1). CrEATIVE PrOblEm SOlVIng: STuDEnTS’ SkIllS In TACklIng rEAl-lIfE PrOblEmS – VOlumE V © OECD 2014 81 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng • figure v.3.4 • relative success on problem-solving tasks, by nature of the problem Success on interactive items, relative to static items, compared to the OECD average, after accounting for booklet and country/economy-speciic response-format effects Odds ratio (OECD average = 1.00) 1.20 Better-than-expected performance on interactive tasks 1.15 1.10 1.05 1.00 0.95 0.90 0.85 Better-than-expected performance on static tasks Bulgaria Montenegro Sweden Slovenia Denmark Shanghai-China Finland Chinese Taipei Austria Slovak Republic Norway Netherlands Serbia Croatia Macao-China Turkey Hungary Poland Estonia Israel Uruguay Malaysia Russian Federation Chile Hong Kong-China Germany Colombia United Arab Emirates Belgium Australia Czech Republic England (United Kingdom) Italy Spain Japan Canada France Singapore Portugal Brazil United States Korea Ireland 0.80 Notes: Values that are statistically signiicant are marked in a darker tone (see Annex A3). This igure shows that students in Ireland are 1.16 times more likely than students across OECD countries, on average, to succeed on interactive items, given their success on static items. Countries and economies are ranked in descending order of the relative likelihood of success on interactive tasks, based on success in performing static tasks. Source: OECD, PISA 2012 Database, Table V.3.1. 1 2 http://dx.doi.org/10.1787/888933003592 Compared with students in other OECD countries, students in Ireland, korea, brazil, the united States, Portugal, Singapore, Canada and Japan were more successful on interactive tasks than expected, given their overall performance. in contrast, students in bulgaria, montenegro, Slovenia, Sweden, denmark, Shanghai-China, Chinese taipei, finland, the Slovak republic, austria, the netherlands, Croatia and Serbia had more facility with static tasks than with interactive tasks, as compared to the relative success of students in other oeCd countries. this may indicate a difficulty related to the specific skills used uniquely to solve interactive tasks. Problem-solving processes each item in the PiSa 2012 assessment of problem solving was designed to focus on measuring one distinct problemsolving process. for the purposes of the PiSa 2012 problem-solving assessment, the processes involved are: • exploring and understanding • representing and formulating • Planning and executing • monitoring and reflecting each of these broad processes applies to both static and interactive problems. Exploring and understanding. the objective is to build mental representations of each of the pieces of information presented in the problem. this involves: • exploring the problem situation: observing it, interacting with it, searching for information and finding limitations or obstacles; and • understanding given information and, in interactive problems, information discovered while interacting with the problem situation; and demonstrating understanding of relevant concepts. 82 © OECD 2014 CrEATIVE PrOblEm SOlVIng: STuDEnTS’ SkIllS In TACklIng rEAl-lIfE PrOblEmS – VOlumE V 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng representing and formulating. The objective is to build a coherent mental representation of the problem situation (i.e. a situation model or a problem model). to do this, relevant information must be selected, mentally organised and integrated with relevant prior knowledge. this may involve: • representing the problem by constructing tabular, graphic, symbolic or verbal representations, and shifting between representational formats; and • formulating hypotheses by identifying the relevant factors in the problem and their inter-relationships; and organising and critically evaluating information. Planning and executing. the objective is to use one’s knowledge about the problem situation to devise a plan and execute it. tasks where “planning and executing” is the main cognitive demand do not require any substantial prior understanding or representation of the problem situation, either because the situation is straightforward or because these aspects were previously solved. “Planning and executing” includes: • planning, which consists of goal setting, including clarifying the overall goal, and setting subgoals, where necessary; and devising a plan or strategy to reach the goal state, including the steps to be undertaken; and • executing, which consists of carrying out a plan. monitoring and reflecting. the objective is to regulate the distinct processes involved in problem solving, and to critically evaluate the solution, the information provided with the problem, or the strategy adopted. this includes: • monitoring progress towards the goal at each stage, including checking intermediate and final results, detecting unexpected events, and taking remedial action when required; and • reflecting on solutions from different perspectives, critically evaluating assumptions and alternative solutions, identifying the need for additional information or clarification and communicating progress in a suitable manner. figure v.3.5 uses the released items to illustrate how PiSa 2012 targeted the four problem-solving processes. in general, items were not equally spread across the processes (figure v.3.1). the assessment included a larger number of items tapping into planning and executing, and fewer items tapping into monitoring and reflecting, in recognition of the importance of being able to carry through a solution to a successful conclusion, and of the fact that monitoring progress is part of the three other processes as well. • figure v.3.5 • examples of problem-solving tasks, by process main problem-solving process Exploring and understanding Sample questions MP3 PLAYER – item 1 (field trial) ROBOT CLEANER – items 1 and 2 TICKETS – item 2 Representing and formulating MP3 PLAYER – item 3 (field trial) CLIMATE CONTROL – item 1 ROBOT CLEANER – item 3 Planning and executing MP3 PLAYER – item 2 (field trial) CLIMATE CONTROL – item 2 TICKETS – item 1 TRAFFIC – items 1 and 2 Monitoring and relecting MP3 PLAYER – item 4 (field trial) TICKETS – item 3 TRAFFIC – item 3 Source: oeCd, PiSa 2012 database. What success on different problem-solving processes implies for education policy and practice Strengths and weaknesses on items measuring particular problem-solving processes can be directly related to students’ skills. indeed, the classification by problem-solving process reflects the main demand of each item, although often several processes occur simultaneously, or in succession, while solving a particular item. CrEATIVE PrOblEm SOlVIng: STuDEnTS’ SkIllS In TACklIng rEAl-lIfE PrOblEmS – VOlumE V © OECD 2014 83 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng A major distinction among tasks is between acquisition and use of knowledge. In knowledge-acquisition tasks, the goal is for students to develop or refine their mental representation of the problem space. Students need to generate and manipulate the information in a mental representation. The movement is from concrete to abstract, from information to knowledge. In the context of the PISA assessment of problem solving, knowledge-acquisition tasks may be classified either as “exploring and understanding” tasks or as “representing and formulating” tasks. The distinction within knowledge-acquisition tasks between the two processes is sometimes small, and may relate to the amount of scaffolding provided for exploring and representing the problem space. “Exploring and understanding” items often come with response options provided (as in ROBOT CLEANER, Item 1), which can guide the exploration phase, while “representing and formulating” items more often require constructed responses (as in ROBOT CLEANER, Item 3). In knowledge-utilisation tasks, the goal is for students to solve a concrete problem. The movement is from abstract to concrete, from knowledge to action. knowledge-utilisation tasks correspond to the process of “planning and executing”. Within the PISA assessment of problem solving, tasks would only be classified as “planning and executing” if the execution of a plan is the dominant cognitive demand of the item (and likewise for other problem-solving processes). for instance, while all the items in unit TICKETS are introduced by a superficially similar demand (“buy a ticket”, “find the cheapest ticket and press buy”, “purchase the best ticket available”), only the first is classified as planning and executing. To ensure that no additional generation or refinement of knowledge about the problem is needed, items targeting “planning and executing” often had the results of “representing and formulating” tasks available, as is the case in item 2 of unit CLIMATE CONTROL. “monitoring and reflecting” tasks are intentionally left out of this distinction, because they often combine both knowledge-acquisition and knowledge-utilisation aspects. from an education perspective, the most insightful contrast is between performance on “planning and executing” tasks and performance on tasks requiring knowledge acquisition and abstract information processing. this contrast highlights a distinction that runs throughout school curricula. in the teaching of mathematics, for instance, there may be a tradeoff between a focus on higher-order activities, such as mathematical modelling (understanding real-world situations and transferring them into mathematical models), and a focus on the mastery of basic concepts, facts, procedures and reasoning. Students who are good at tasks whose main cognitive demand is “planning and executing” are good at using the knowledge they have; they can be characterised as goal-driven and persistent. Students who are strong on tasks measuring “exploring and understanding” or “representing and formulating” processes are good at generating new knowledge; they can be characterised as quick learners, who are highly inquisitive (questioning their own knowledge, challenging assumptions), generating and experimenting with alternatives, and good at abstract information processing. in practice, proficient problem-solvers are good at all sorts of tasks, and there is a strong positive relationship between success rates on any two sets of items. in the following sections, the focus is not on absolute levels of proficiency, but on areas of relative strength and weakness, compared with the skills observed among students with similar overall proficiency. Success on items by problem-solving process involved figures v.3.6 and v.3.7 present national performance by problem-solving process – first, using percent-correct figures to illustrate absolute strength, then, adjusting for country/economy-specific response-format effects and accounting for overall differences in performance, to show areas where performance is unexpectedly strong or weak. figure v.3.8 summarises countries’/economies’ relative strengths and weaknesses revealed by the comparison of performance on items measuring different problem-solving processes to the average performance of students across oeCd countries. “exploring and understanding” items, as a set, were found easier by students in Singapore, norway, Hong kong-China, korea, australia, austria, Chinese taipei, Japan, macao-China, Sweden and finland than by students in oeCd countries, on average. items with “representing and formulating” tasks, as a set, were easier than expected in macao-China, Chinese taipei, Shanghai-China, korea, Singapore, Hong kong-China, Canada, italy, Japan, france, australia and belgium. items assessing the process of “planning and executing”, as a set, were easier than expected in bulgaria, montenegro, Croatia, Colombia, uruguay, Serbia, turkey, Slovenia, brazil, malaysia, denmark, the Czech republic, the netherlands, Chile, Hungary, finland, the russian federation, Portugal and Poland. 84 © OECD 2014 CrEATIVE PrOblEm SOlVIng: STuDEnTS’ SkIllS In TACklIng rEAl-lIfE PrOblEmS – VOlumE V 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng • figure v.3.6 • differences in countries’/economies’ success on problem-solving tasks, by process Exploring and understanding 60 Shanghai-China Netherlands 50 representing and formulating 70 Average percentage of full-credit responses for items assessing the process of “representing and formulating” Average percentage of full-credit responses for items assessing the process of “exploring and understanding” 70 40 30 20 10 0 60 Shanghai-China 50 Netherlands 40 30 20 10 0 0 10 20 30 40 50 60 70 0 10 20 30 Planning and executing 60 Netherlands 50 Shanghai-China 40 30 20 10 50 60 70 Monitoring and reflecting 70 Average percentage of full-credit responses for items assessing the process of “monitoring and reflecting” Average percentage of full-credit responses for items assessing the process of “planning and executing” 70 40 Average percentage of full-credit responses on all items Average percentage of full-credit responses on all items 60 50 Shanghai-China 40 Netherlands 30 20 10 0 0 0 10 20 30 40 50 60 70 Average percentage of full-credit responses on all items 0 10 20 30 40 50 60 70 Average percentage of full-credit responses on all items Note: The netherlands and Shanghai-China share similar levels of performance on items assessing the process of “planning and executing”, but have different levels of performance on all remaining items; this example is discussed in the text. Source: OECD, PISA 2012 Database, Table V.3.2. 1 2 http://dx.doi.org/10.1787/888933003592 finally, “monitoring and reflecting” items, taken together, were easier than expected in Colombia, Chile, Turkey, Spain, uruguay, Ireland, brazil, Croatia, bulgaria, Singapore, the united States, the united arab emirates, montenegro, the Czech republic and england (united kingdom). to illustrate strengths and weaknesses on specific problem-solving processes, one can compare the performance of students in the netherlands and Shanghai-China. overall, students in Shanghai-China performed better on the problem-solving scale than students in the netherlands. the average success rate on all assessment items is 52.6% for Shanghai-China and 47.9% for the netherlands. However, student performance on planning and executing items in the netherlands, with a success rate of 49.7%, on average, was comparable to that of students in Shanghai-China on these same items (49.8%). CrEATIVE PrOblEm SOlVIng: STuDEnTS’ SkIllS In TACklIng rEAl-lIfE PrOblEmS – VOlumE V © OECD 2014 85 86 3 odds ratio (oeCd average = 1.00) Odds ratio (OECD average = 1.00) Norway Turkey Croatia Shanghai-China Hong Kong-China Spain Colombia Korea Korea Uruguay Uruguay Singapore Australia Montenegro Czech Republic England (UK) Belgium Korea Netherlands Israel Serbia France Netherlands Chile Hungary Finland Russian Federation Portugal Poland Slovak Republic Austria Estonia United Arab Emirates Germany Belgium United Arab Emirates Sweden Israel Denmark United States Malaysia Russian Federation Estonia Norway Poland England (UK) Ireland Japan Macao-China Sweden Finland Italy Ireland Israel Germany Shanghai-China France Netherlands Belgium Canada United States Slovak Republic England (UK) Denmark Austria Estonia Spain Hungary Slovenia England (UK) Germany Spain Italy Sweden Slovenia Hungary Germany Hong Kong-China Canada Slovak Republic Finland Poland Sweden Chinese Taipei Israel United States Belgium Canada Ireland Australia Italy Japan Macao-China Chinese Taipei Portugal Spain Slovak Republic Turkey Czech Republic Chile Serbia Brazil Finland Netherlands Croatia Portugal Bulgaria Czech Republic United Arab Emirates Russian Federation Serbia Slovenia Brazil Malaysia Uruguay Croatia Macao-China Shanghai-China Montenegro Chile Austria Hong Kong-China Uruguay Montenegro Norway Korea Colombia Colombia Denmark Singapore Bulgaria Turkey Weaker-than-expected performance Shanghai-China France Estonia Poland Weaker-than-expected performance Hungary Australia Weaker-than-expected performance Norway Weaker-than-expected performance Japan Exploring and understanding Russian Federation Czech Republic France Australia Planning and executing Portugal Malaysia Denmark Italy Japan Austria Chinese Taipei representing and formulating Malaysia Brazil Canada • figure v.3.7 • relative success on problem-solving tasks, by process After accounting for booklet and country/economy-speciic response-format effects United Arab Emirates Slovenia Hong Kong-China Stronger-than-expected performance Singapore United States Serbia Turkey Singapore Stronger-than-expected performance Croatia Bulgaria Stronger-than-expected performance Brazil Monitoring and reflecting Stronger-than-expected performance Ireland 1.40 Chinese Taipei 1.20 Macao-China Montenegro 1.00 Bulgaria Chile 0.80 0.60 1.40 1.20 1.00 0.80 0.60 1.40 1.20 1.00 0.80 0.60 Colombia STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng Odds ratio (OECD average = 1.00) 1.40 1.20 1.00 0.80 0.60 Note: Values that are statistically signiicant are marked in a darker tone (see Annex A3). Countries and economies are ranked in each chart in descending order of the relative success on tasks related to the respective problem-solving processes. Source: OECD, PISA 2012 Database, Table V.3.2. 1 2 http://dx.doi.org/10.1787/888933003592 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v Odds ratio (OECD average = 1.00) 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng thus, the main area for improving the performance of students in the netherlands so that it is closer to the performance of students in Shanghai-China appears to be in the remaining items, while students in Shanghai-China could have scored higher on the problem-solving scale if their performance on planning and executing items were not significantly weaker than their performance on the remaining items (figure v.3.6 and table v.3.2). • figure v.3.8 • relative strengths and weaknesses in problem-solving processes Stronger-than-expected performance on the problem-solving process non-signiicant strength or weakness Weaker-than-expected performance on the problem-solving process mean score in problem solving Singapore Korea Japan Macao-China Hong Kong-China Shanghai-China Chinese Taipei Canada Australia Finland England (United Kingdom) Estonia France Netherlands Italy Czech Republic Germany United States Belgium Austria Norway Ireland Denmark Portugal Sweden Russian Federation Slovak Republic Poland Spain Slovenia Serbia Croatia Hungary Turkey Israel Chile Brazil Malaysia United Arab Emirates Montenegro Uruguay Bulgaria Colombia difference between observed and expected performance, by problem-solving process Exploring representing Planning monitoring and understanding and formulating and executing and relecting 562 561 552 540 540 536 534 526 523 523 517 515 511 511 510 509 509 508 508 506 503 498 497 494 491 489 483 481 477 476 473 466 459 454 454 448 428 422 411 407 403 402 399 Note: Countries/economies with stronger-(weaker-)than-expected performance are countries/economies whose students’ relative likelihood of success in one group of tasks, based on their success in performing all other tasks, is signiicantly larger (smaller) than in the OECD average, after accounting for item dificulty and country/economy-speciic response-format effects. Countries and economies are ranked in descending order of the mean score in problem solving. Source: OECD, PISA 2012 Database, Tables V.2.2 and V.3.2. 1 2 http://dx.doi.org/10.1787/888933003592 figure V.3.8 summarises countries’ and economies’ strengths and weaknesses in problem-solving processes. Two patterns emerging from figure v.3.8 are worth noting. first, there is substantial overlap between the countries/economies that are strong on “exploring and understanding” items and the countries/economies that are strong on “representing and formulating” items. many of these same countries/economies, in turn, have weaker-than-expected performance on CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 87 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng “planning and executing” items. Conversely, there is also overlap between countries/economies that are strong on “planning and executing” items, but weak on “exploring and understanding” and “representing and formulating” items. this overlap confirms the assumption that, from the point of view of skills development, the main contrast is between “knowledge-acquisition” processes and “knowledge-utilisation” processes. the observed difference in students’ proficiency between these two major sets of skills may be traced back to differences in curricula and teaching practices. Second, many of the best-performing countries and economies in problem solving are those with better-than-expected performance on knowledge-acquisition tasks (“exploring and understanding”, “representing and formulating”), and relatively weaker performance on knowledge-utilisation tasks (“planning and executing” tasks that do not require substantial prior understanding or representation of the problem situation). this is observed despite the fact that the analysis adjusts for overall performance differences between countries and economies. this pattern reflects the fact that performance differences across countries/economies are much more pronounced on knowledge-acquisition tasks than on knowledge-utilisation tasks (figure v.3.6 and table v.3.2). Around 40 percentage points separate the country with the highest percentage of correct answers from the country with the lowest percentage of correct answers on “exploring and understanding” tasks (64.7% success in korea, 24.7% in Colombia) and on “representing and formulating” tasks (60.7% success in korea, 18.7% in Colombia). in contrast, only about 30 percentage points separate the top and bottom percent-correct on “planning and executing” tasks (56.3% in Japan, 26.7% in bulgaria). Similarly, there is a 30-percentage-point gap between the five best-performing systems and the five lowest-performing systems on knowledgeacquisition tasks, while the gap shrinks to about 20 percentage points on knowledge-utilisation tasks (table v.3.6). While in absolute terms, top-performing countries/economies perform above-average on all problem-solving processes, the difference with lower-performing countries/economies narrows on “planning and executing” tasks. this analysis shows that, in general, what differentiates high-performing systems, and particularly east Asian education systems, such as those in Hong kong-China, Japan, korea, macao-China, Shanghai-China, Singapore and Chinese taipei, from lower-performing ones, is their students’ high level of proficiency on “exploring and understanding” and “representing and formulating” tasks. Problem contexts and response formats the problems in the PiSA assessment can also be classified according to their context and response format. Solution rates and relative success on items by problem context are presented in Annex b (tables v.3.3 and v.3.4). figure v.3.9 shows the difference in relative success rates according to response formats. the classification of problems by their context refers to the fictional frame (scenario) of the assessment problems and has no implications in terms of task demands. in contrast to the classification by nature of the problem situation or by problem-solving process, all items within a given unit share the same context. Still, an individual’s familiarity with and understanding of the problem context will affect his or her ability to solve the problem. two dimensions were identified to ensure that assessment tasks reflect a range of contexts that are authentic and of interest to 15-year-olds: the setting (technology or not) and the focus (personal or social). Problems set in a technology context are based on the functionality of a technological device, such as a mobile phone, a remote control for appliances and a ticket-vending machine. knowledge of the inner workings of these devices is not required. typically, students are led to explore and understand the functionality of a device as preparation for controlling the device or for troubleshooting its malfunction. Problems set in a non-technology context include tasks such as route planning, task scheduling and decision making. Personal contexts include those relating primarily to the student, family and close peers. Social contexts typically do not involve the student directly and relate to situations encountered more broadly in the community or society in general. response formats also vary across items. one-third of the items (14 of 42 items) require students to select their response(s) by clicking a radio button or by selecting from a drop-down menu. this includes simple multiple-choice items, where there is one correct response to be selected, complex multiple-choice items, where two or three separate multiplechoice selections must be made, and variations of these (such as when there is more than one correct response to be selected). All of these items are automatically coded. 88 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng the remaining 28 items require students to construct their response, e.g. by entering text, dragging shapes, drawing lines between points, highlighting part of a diagram or interacting with the simulated device. most of these items were also automatically coded. However, where it was considered important to ask students to explain their method or justify a selected response, a trained expert coded correct and incorrect answers, giving partial credit where appropriate. Six constructed response items required expert coding (Question 3 in the unit ROBOT CLEANER provides an example). Students in many countries and economies, particularly in Asia, perform better, on average, on selected-response items than on constructed-response items. in the PiSA problem-solving test, a pattern of relatively strong performance on selected-response items (and weak performance on constructed-response items) was found in bulgaria, Shanghai-China, malaysia, korea, macao-China, uruguay, Hong kong-China and Chinese taipei. in these countries and economies, the success ratio on constructed-response items was at most 0.85 times as high as one could have expected, given performance on selected-response items and the relative difficulty of items as measured among oeCd students. Several other countries, namely israel, the united Arab emirates, Colombia, Japan, montenegro, brazil, turkey, Hungary and Croatia, had ratios of success significantly below one, also indicating unexpectedly weak performance on constructedresponse items (figure v.3.9 and table v.3.5). • figure v.3.9 • relative success on problem-solving tasks, by response format Success on constructed-response items, relative to selected-response items, compared to the OECD average, after accounting for booklet effects odds ratio (oeCd average = 1.00) 1.20 Better-than-expected performance on constructed-response items 1.15 1.10 1.05 1.00 0.95 0.90 0.85 0.80 Better-than-expected performance on selected-response items Bulgaria Shanghai-China Korea Malaysia Uruguay Macao-China Hong Kong-China Israel Chinese Taipei United Arab Emirates Japan Colombia Brazil Montenegro Turkey Chile Hungary Singapore Poland Croatia Sweden Slovenia Serbia Czech Republic Russian Federation Netherlands Slovak Republic Italy Finland Spain France Norway United States Canada Germany Portugal England (United Kingdom) Austria Estonia Denmark Ireland Belgium Australia 0.75 Note: Values that are statistically signiicant are marked in a darker tone (see Annex A3). Countries and economies are ranked in descending order of the relative likelihood of success on constructed-response items, based on success in performing selected-response items. Source: OECD, PISA 2012 Database, Table V.3.5. 1 2 http://dx.doi.org/10.1787/888933003592 the response format, however, is strongly associated with the particular process targeted by the item. items that focus on measuring students’ competence at “exploring and understanding” are mostly presented in a selected-response format. items that focus on measuring students’ competence at “planning and executing” are mostly presented in a constructedresponse format. nevertheless, within each set of items defined by a problem-solving process, there are both selectedand constructed-response items, so that one can control for the (country-specific) influence of the response format when comparing success ratios across item families involving different processes. CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 89 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng A grouPIng oF counTrIeS by TheIr STrengThS And weAKneSSeS In Problem SolvIng the analysis in this chapter identifies differences in the performance patterns of students across item types. the analysis has shown that two major dimensions along which performances of countries/economies differ are related to whether interaction with the problem situation is needed in order to uncover relevant information, and depending on whether the task primarily corresponds to knowledge-acquisition or to knowledge-utilisation processes. together, the differences in performance according to the nature of the problem situation and the major problem-solving process targeted identify several groups of countries/economies (figure v.3.10). interestingly, these groups often overlap with historical and geographical groupings. • figure v.3.10 • Joint analysis of strengths and weaknesses, by nature of the problem and by process Stronger-than-expected performance on interactive items and on knowledge-acquisition tasks oEcd average bEttEr PErformancE on intEractivE taSkS, rElativE to Static taSkS Stronger-than-expected performance on interactive items, weaker-than-expected performance on knowledge-acquisition tasks Ireland Brazil Germany United States Korea England (UK) Portugal United Arab Emirates Spain Czech Republic Colombia Chile Estonia Russian Federation Malaysia Turkey Uruguay Poland Serbia Croatia Hungary Netherlands Slovenia Finland Slovak Republic Denmark Montenegro France Canada Italy Belgium Singapore Japan Australia Israel Norway Austria oEcd average Hong Kong-China Sweden Macao-China Chinese Taipei Shanghai-China Bulgaria Weaker-than-expected performance on interactive items and on knowledge-acquisition tasks Weaker-than-expected performance on interactive items, stronger-than-expected performance on knowledge-acquisition tasks bEttEr PErformancE on knoWlEdGE-acQuiSition taSkS, rElativE to knoWlEdGE-utiliSation taSkS Note: This igure plots the odds ratios for success on interactive items, compared to static items, on the vertical axis, and the odds ratios for success on knowledge-acquisition tasks (“exploring and understanding” or “representing and formulating”), compared to knowledge-utilisation tasks (“planning and executing”), on the horizontal axis. both axes are in logarithmic scale. Source: OECD, PISA 2012 Database, Tables V.3.1 and V.3.6. 1 2 http://dx.doi.org/10.1787/888933003592 Six east Asian countries and economies, namely korea, Singapore, Hong kong-China, macao-China, Chinese taipei and Shanghai-China, stand out for their very high success rates on knowledge-acquisition tasks, compared to their success rates on planning and executing tasks. Within this group, however, there are relatively stark differences in their performance on interactive problems. Students in korea and Singapore are significantly more at ease with these 90 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng problems than students in Shanghai-China, Chinese taipei and macao-China. Students from Hong kong-China are in a middle position. While all of these countries and economies rank in the top positions for overall performance, this analysis suggests that in Shanghai-China, Chinese taipei and macao-China, a focus on students’ skills at dealing with interactive problem situations is required in order to improve further and close the performance gap with korea and Singapore. in reviewing their curricula, teachers and curriculum developers may want to introduce more opportunities for students to develop and exercise the traits that are linked to success on interactive items, such as curiosity, perseverance and creativity. they may find inspiration in the curricula and teaching practices of their regional neighbours. Among lower-performing countries and economies in problem solving, the low performance of latin American countries (brazil, Colombia, Chile and uruguay) appears to be mainly due to a large performance gap on knowledge-acquisition tasks. these countries have no particular difficulty with interactive tasks – and brazil even shows a relative strength on such tasks. in these countries, efforts to raise problem-solving competency should concentrate mainly on improving students’ performance on “exploring and understanding” and on “representing and formulating” tasks. these tasks require students to build mental representations of the problem situation from the pieces of information with which they are presented. moving from the concrete problem scenario to an abstract representation and understanding of it often demands inductive or deductive reasoning skills. teachers and curriculum experts may question whether current curricula include sufficient opportunities to model these abstract reasoning skills and whether these opportunities are offered in the classroom. in contrast, several countries in Southern and eastern europe, namely bulgaria, montenegro, Slovenia, Croatia and Serbia, show relatively weak performance both on knowledge-acquisition tasks and on interactive tasks, compared to their performance on “planning and executing” and on static tasks. in these countries, students seem to find it particularly difficult to understand, elaborate on, and integrate information that is not explicitly given to them (in a verbal or visual format), but has to be inferred from experimental manipulation of the environment and careful observation of the effects of that manipulation. Students in these countries may benefit from greater opportunities to learn from hands-on experience. the performance gap between oeCd countries in europe and north America and the top-performing countries in problem solving mainly originates from differences in students’ performance on knowledge-acquisition tasks. in general, the PiSA problem-solving assessment shows that there is significant room for improving students’ ability to turn information into useful knowledge, as measured by performance differences on the dimensions of “exploring and understanding” and “representing and formulating” problem situations. Within this group, ireland and the united States stand out for their strong performance on interactive items, compared, for instance, to the nordic countries (Sweden, finland, norway and denmark), the netherlands, and some countries in Central europe (in particular, Poland, Hungary and the Slovak republic). therefore, the analysis also identifies a strong potential for the nordic and Central european countries to improve on their students’ ability to cope with interactive problem situations. to do so, educators may need to foster such dispositions as being open to novelty, tolerating doubt and uncertainty, and daring to use intuition to initiate a solution. finally, several countries, while performing at different levels, show a similar balance of skill when compared to each other, and one that is close to the oeCd average pattern of performance. italy and Australia, for instance, have a very similar pattern of performance to that observed in Japan, although in terms of overall performance, Japan ranks significantly above Australia, which, in turn, performs better than italy. these three countries all perform close to their expected level on interactive items (based on the oeCd average pattern of performance), and slightly above their expected level on knowledge-acquisition tasks (although the example of korea and Singapore shows that significant gains are still possible for them). in other countries, such as Spain, england (united kingdom) and germany, performance across tasks reflects the balance observed across oeCd countries, on average. for students in this group of countries, as a whole, there are no clear indications as to which aspects of problem-solving competence deserve particular attention. nevertheless, the profile of performance may differ across particular groups of students. Such differences across groups of students will be analysed in the next chapter. CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 91 3 STudenTS’ STrengThS And weAKneSSeS In Problem SolvIng two notes of caution: first, throughout this chapter, patterns of performance within countries and economies have been compared to the oeCd average patterns in order to identify comparative strengths and weaknesses. implications drawn from this analysis tacitly assume that this international benchmark corresponds to a desirable balance between the various aspects of problem-solving competence. the oeCd average was selected for pragmatic reasons only. therefore, the normative interpretation of the benchmark can be challenged, and alternative comparisons (for instance, to the pattern observed in the top-performing country) are equally possible. Second, although this analysis can provide interesting indications, any conclusion that is drawn from subsets of the PiSA problem-solving test must be carefully checked against evidence collected independently in each system on the strengths of the respective curriculum and teaching practices. lacking supporting evidence, the conclusions should be interpreted with caution. indeed, the PiSA problem-solving assessment comprises a total of 42 items. When success is analysed on subsets of items that share common characteristics, the number of items inevitably drops. While the 42 items together reflect a consensus view of what problem-solving competence is, when this item set is split into smaller sets to analyse the individual components of problem-solving competence, the resulting picture is necessarily less sharp.3 the results of analyses based on small sets of items may sometimes be driven by idiosyncratic features of one or two items in the pool rather than by their common traits. Notes 1. A complementary analysis that can diagnose more detailed strengths and weaknesses will be made possible by the availability of behavioural sequences recorded by the computer interface (process data). After having identified the elementary task demands of each assessment item, the data recording students’ interactions with items can be used, for instance, to identify patterns in terms of frequent stumbling blocks that hinder students from reaching the solution. 2. fisher’s exact test of independence of rows and columns was performed. the null hypothesis of independence of rows and columns for the contingency tables pairing the cognitive processes with the nature of the problem situation cannot be rejected (p-value: 0.69). 3. this is a problem of external validity that is not reflected in the standard errors provided with the statistical analysis in this chapter. While the inference about strengths and weaknesses is internally valid for the particular test of problem solving analysed, the question of external validity is whether a different test, constructed according to the same definition and framework, would give exactly the same results: i.e. to what extent one can generalise from performance on a dozen items to competence on the unobserved construct underlying these items. References OECD (2013), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, oeCd Publishing. http://dx.doi.org/10.1787/9789264190511-en OECD (forthcoming), PISA 2012 Technical Report, oeCd Publishing. 92 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 How Problem-Solving Performance Varies within Countries This chapter looks at differences in problem-solving performance related to education tracks within countries and to students’ gender, socioeconomic status and immigrant background. It also examines students’ behaviours and attitudes related to problem solving, and their familiarity with information and communication technology. In addition, the chapter identifies particular groups of students who perform better in problem solving than expected, given their performance in mathematics, reading and science. CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 93 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS this chapter looks at performance differences across students and schools within countries. How does performance in problem solving relate to student characteristics, such as gender, socio-economic status, and immigrant background? do students in certain study programmes perform better in problem solving than in the core curricular subjects? the chapter also looks at student behaviours and attitudes related to problem solving, as well as at indicators of familiarity with information and communication technology (iCt), as they were measured through background questionnaires in PiSA 2012. the aim of this chapter is to understand how differences between countries and economies that are presented in Chapters 2 and 3 are related to differences in performance among various groups of students. the chapter focuses on identifying particular groups of students who perform better in problem solving than could be expected, given their performance in mathematics, reading and science; and on understanding whether the strengths and weaknesses of systems stem from the strengths and weaknesses of certain groups of students. what the data tell us • in malaysia, Shanghai-China and turkey, more than one in eight students attend a vocational study programme, and these students show significantly better performance in problem solving, on average, than students with comparable performance in mathematics, reading and science but who are in general study programmes. • on average across oeCd countries, there are three top-performing boys for every two top-performing girls in problem solving. in Croatia, italy and the Slovak republic, boys are as likely as girls to be low-achievers, but are more than twice as likely as girls to be top performers. in no country or economy are there more girls than boys among the top performers in problem solving. • girls appear to be stronger in performing the “planning and executing” tasks that measure how students use knowledge, compared to other types of problems; and weaker in performing the more abstract “representing and formulating” tasks, which relate to how students acquire knowledge. this is particularly true among girls in Hong kong-China, korea and Chinese taipei. • the impact of socio-economic status on problem-solving performance is weaker than it is on performance in mathematics, reading or science. • not using a computer at home is negatively related to problem-solving performance in 29 of 33 participating countries and economies, even after accounting for socio-economic status. A similarly strong relationship is observed between lack of computer use at home and performance on the paper-based assessments of mathematics and reading. PerFormAnce dIFFerenceS unIQue To Problem SolvIng the overall variation in problem-solving proficiency can be split into two components – one that is also observed in mathematics, reading and science (about two-thirds), and one that is unique to problem solving (about one-third) (see Chapter 2). this chapter will mainly explore the factors that are related to the unique aspects of problem-solving performance. How much of the variation in performance that is unique to problem solving lies between schools, and what part is related to differences between students attending the same school? figure v.4.1 shows that, on average, a similar proportion – about one-third – of the within-school and between-school variations in problem solving performance is not accounted for by differences in mathematics performance between and within schools, and can be considered unique to problem solving. therefore, not only do school policies and practices have a significant influence on the problem-solving performance of students (see Chapter 2, figure v.2.12), but a large proportion of the between-school variation in performance is unique to problem solving. this means that the differences in problem-solving performance between schools do not stem solely from differences in mathematics performance. School rankings based on problem solving will differ from school rankings based on mathematics. Among schools with similar results in mathematics, a significant proportion of the between-school differences in problem-solving performance likely reflects differences in schools’ emphases on and approaches towards fostering students’ problem-solving skills. 94 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS • figure v.4.1 • Performance variation unique to problem solving As a percentage of the total variation in performance 30 20 Variation unique to problem solving 10 0 20.6% 12.7% 10 25.6% Variation shared with performance in mathematics 20 40.4% 30 40 50 Variation between schools (38.3%) Variation within schools (61.0%) Note: The igure shows the components of the performance variation in problem solving for the OECD average. Source: OECD, PISA 2012 Database, Table V.4.1. 1 2 http://dx.doi.org/10.1787/888933003611 Similarly, the differences across students within schools only partly reflect general academic proficiency. to the extent that performance differences in problem solving are unique to problem solving, their origins also differ from those of performance variations in curricular subjects. PerFormAnce dIFFerenceS AcroSS STudy ProgrAmmeS Performance differences across schools can be at least partly related to differences in curricula. However, it is impossible to determine a causal impact of the curriculum on performance using only PiSA data. the comparison between two study programmes will always be confounded by differences between students, teachers and schools that cannot be captured by questionnaires; even figures that account for socio-economic background or gender cannot be interpreted causally. in most countries, there is a major distinction between vocational or pre-vocational study programmes and general study programmes. generally, only a minority of 15-year-olds in each country is enrolled in vocational study programmes; the exceptions are Serbia, Croatia, Austria, montenegro, Slovenia and italy, where a majority of 15-year-olds students is enrolled in such programmes (table v.4.2). How are study programmes related to the unique aspects of problem-solving performance? this “relative performance in problem solving” of each study programme can be estimated by comparing the performance of students in each study programme only to students who share their same proficiency in mathematics, reading and science. Such a comparison can show whether good or poor performance in a subject is reflected in equally good or poor performance in problem solving; or, conversely, whether there is a specific advantage in problem solving for students in a particular type of study programme. figure v.4.2 shows that, in 4 of 31 countries and economies, namely Shanghai-China, turkey, the united Arab emirates and malaysia, students in vocational study programmes have significantly better performance in problem solving than students with comparable performance in mathematics, reading and science who are in general study programmes. in all of these cases, the advantage of students in vocational programmes corresponds to at least 12 score points on the problem-solving scale. in all of these countries and economies, with the exception of the united Arab emirates, more than one in eight students (more than 12.5%) attend vocational study programmes. meanwhile, in the russian federation and germany, students in vocational study programmes have significantly lower performance in problem solving than students with comparable performance in mathematics, reading and science. the gap between the two groups of students exceeds 24 score points on the problem-solving scale. in both countries, however, fewer than 5% of students are enrolled in a vocational study programme (tables v.4.2 and v.4.4). CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 95 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS • figure v.4.2 • relative performance in problem solving among students in vocational and pre-vocational tracks Difference in problem-solving performance between students in vocational or pre-vocational programmes and students in general programmes with similar performance in mathematics, reading and science Score-point difference 30 Students in vocational/pre-vocational programmes perform above their expected level in problem solving 20 10 0 -10 -20 -30 Students in vocational/pre-vocational programmes perform below their expected level in problem solving -40 2.0 4.1 Germany Russian Federation 1.4 Hungary 14.3 0.8 Spain Uruguay Bulgaria 40.8 Slovenia 53.2 Czech Republic 31.0 0.8 Austria 69.3 Ireland Australia 10.9 1.2 OECD average 15.4 England (United Kingdom) 2.8 Chile Serbia 74.4 1.6 Netherlands 22.2 Croatia 70.1 Macao-China 8.2 Montenegro 66.0 Japan 24.2 Slovak Republic Belgium 44.0 Portugal 16.7 Colombia 25.2 Chinese Taipei 34.5 Italy 51.5 France 15.3 Malaysia 13.3 2.7 Korea 19.9 Turkey 38.1 United Arab Emirates Shanghai-China 21.2 -50 Percentage of students in pre-vocational or vocational programmes Note: Statistically signiicant differences are marked in a darker tone (see Annex A3). Countries and economies are ranked in descending order of the score-point difference between students in vocational/pre-vocational programmes and those in general programmes with similar performance in mathematics, reading and science. Source: OECD, PISA 2012 Database, Tables V.4.2 and V.4.4. 1 2 http://dx.doi.org/10.1787/888933003611 figure v.4.3 uses the national classification of study programmes to highlight education tracks where students have significantly better performance in problem solving than students with comparable performance in mathematics, reading and science in their country who are enrolled in different study programmes. many of the differences in relative performance across study programmes concern countries or economies with overall weaker-than-expected performance in problem solving (see figure v.2.15 and table v.2.6); in these cases, a “relatively strong” programme may constitute an exception to the overall weakness. Students enrolled in general study tracks that prepare for higher education in germany (Gymnasium) and in Hungary (Gimnázium), for instance, show stronger performance in problem solving, on average, than other german or Hungarian students with similar scores in mathematics, reading and science. While, overall, students in germany and Hungary perform below students from other countries with similar performance in core subjects, this finding suggest that students outside of these general study tracks account for most of this negative result. in other countries, students from specific vocational programmes score higher than other students in their country who are similarly proficient in mathematics, reading and science. Such is the case for students in the vocational upper secondary programmes in the flemish and german-speaking Communities of belgium: they tend to score 8 and 25 points, respectively, above their expected level when compared to all belgian students of similar proficiency in core subjects. Similarly, in Portugal, students in the professional upper secondary track score 17 points above their expected level. the performance gap in problem solving between students in the professional track and students in the general track is in this case smaller in problem solving than in mathematics, reading and science (table v.4.5). fewer significant differences can be observed among countries whose students, overall, are relatively strong in problem solving when compared with students in other countries with similar proficiency in mathematics, reading and science. 96 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS • figure v.4.3 [Part 1/2] • relative performance in problem solving, by education track Education tracks with a relative strength in problem solving Education tracks whose students’ performance in problem solving is in line with their performance in mathematics, reading and science Education tracks with a relative weakness in problem solving OECD Numbers in parentheses indicate the proportion of 15-year-olds in the study programme general lower secondary (75.4%); lower secondary with some vocational subjects (5.3%); general upper secondary (13.5%); upper secondary with some vocational subjects (4.1%); vocational upper secondary (1.5%) australia austria Charter schools (Statutschulen) (0.3%) Pre-vocational transition year (Polytechnische Schule) and lower secondary (Hauptschule) (14.6%); general lower and upper secondary leading to university entrance qualiications (AHS) (25.7%); Vocational school for apprentices (Berufsschule) (15.4%); Intermediate technical and vocational school (bmS) (11.7%); College for higher vocational education (bHS) (32.4%) belgium Vocational upper secondary (fl: TSO, kSO, bSO) (29.1%); lower secondary (ger.) (0.1%); Vocational upper secondary (ger.) (0.2%) lower secondary (fl.) (1.5%); general upper secondary (fl.: ASO) (24.3%); lower secondary (fr.) (5.3%); general upper secondary (fr.) (24.9%); Vocational upper secondary (fr.) (10.5%); general upper secondary (ger.) (0.4%); Vocational upper secondary, part-time programmes (fl.,fr.,ger.) (0.5%); Special education (fl.,fr.,ger.) (3.1%) lower secondary (5.5%); upper secondary, irst cycle (87.8%); general upper secondary, second cycle (3.9%); Vocational upper secondary, second cycle (2.8%) chile czech republic basic school (47.1%) general lower and upper secondary (gymnasium) (19.3%); vocational upper secondary with school-leaving exam (21.9%); vocational upper secondary without school-leaving exam (8.4%); Special education (2.8%) denmark upper secondary (0.5%) Primary and lower secondary (88.3%); Continuation school (11.2%) Estonia lower secondary (98.1%) general upper secondary (1.5%) lower secondary (27.3%); Special education (lower secondary) (2.5%); general upper secondary (57.4%); technical upper secondary (11.0%); Professional upper secondary (1.8%) france Germany general lower secondary with access to general upper secondary (Gymnasium) (36.1%) Special education (2.8%); general lower secondary without access to general upper secondary (Hauptschule) (15.5%); general lower secondary without access to general upper secondary (Realschule) (33.5%); general upper secondary (Gymnasium) (0.8%); Comprehensive lower secondary (Integrative Gesamtschule) (9.3%) Pre-vocational and vocational (Übergangsjahr, Berufsschule, Berufsfachschule) (2.0%) hungary general upper secondary (Gimnázium) (38.2%) vocational upper secondary with access to post-secondary and tertiary (36.2%); vocational upper secondary without access to post-secondary and tertiary (14.3%) Primary school (11.3%) ireland transition year programme (24.3%) Applied upper secondary (leaving certiicate applied) (0.8%); general upper secondary (leaving certiicate) (7.4%); Vocational upper secondary (leaving certiicate vocational) (5.1%) lower secondary (Junior certiicate) (62.4%) italy Scientiic, classical, social science, scientiic-technological, linguistic, artistic, music and performing arts high schools (45.9%); Technical institute (29.0%); Vocational institutes (service industry, industry, arts and crafts workers) (17.0%); Vocational training, vocational schools of bolzano and Trento provinces (5.5%) lower secondary (2.6%) Japan general upper secondary (74.4%); vocational upper secondary (24.2%) korea lower secondary (5.9%); general upper secondary (74.2%); vocational upper secondary (19.9%) netherlands Practical preparation for labour market (Pro) (2.5%); Pre-vocational secondary, years 1 and 2 (vmbo 1 & 2) (2.4%); Pre-vocational secondary, years 3 and 4, basic track (vmbo bb) (8.4%); Pre-vocational secondary, years 3 and 4, middle management track (vmbo kb) (11.4%); Pre-vocational secondary, years 3 and 4, theoretical and mixed track (vmbo gl/tl) (24.4%); Senior general secondary education (HAvo), leading to university of applied sciences (25.9%); Preuniversity (vWo) (25.1%) Portugal Professional upper secondary (7.2%) lower secondary (35.6%); general upper secondary (47.7%); vocational training (Cef - Curso de Educação e Formação) (9.3%) Slovak republic Specialised upper secondary with school-leaving exam (26.1%) general lower secondary (41.6%); Special education (1.2%); general lower and upper secondary (gymnasium) (22.9%); Specialised upper secondary without school-leaving exam (iSCed 3C) (8.2%) Note: numbers in parentheses indicate the proportion of 15-year-olds in the study programme; percentages may not add up to 100 within each country/economy because of rounding and of rare programmes for which results are not reported. only countries/economies with results reported for more than one study programme are included in this igure. The middle column includes all programmes for which relative performance in problem solving is not statistically different from 0 (see Annex A3). In belgium, the information about study programmes in variable PrOgn was combined with information about regions to identify education tracks: “fl.” refers to the flemish Community, “fr.” to the french Community, and “ger.” to the german-speaking Community; results for “Part-time vocational” programmes and “Special education” programmes are reported at the national level. in germany, students in schools with multiple study programmes are classiied according to their speciic education track. Source: OECD, PISA 2012 Database, Table V.4.5. 12 http://dx.doi.org/10.1787/888933003611 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 97 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS • figure v.4.3 [Part 2/2] • relative performance in problem solving, by education track Education tracks with a relative strength in problem solving Education tracks whose students’ performance in problem solving is in line with their performance in mathematics, reading and science Education tracks with a relative weakness in problem solving OECD Numbers in parentheses indicate the proportion of 15-year-olds in the study programme Slovenia technical upper secondary (38.3%) Spain lower secondary (99.2%); initial vocational qualiication programme (0.8%) Sweden general, compulsory basic (97.8%); general upper secondary (1.8%) Anatolian vocational high school (5.7%); technical high school (1.5%); Anatolian technical high school (2.5%) general upper secondary (general and classical gymnasiums) (33.8%); vocational programmes of medium duration (13.8%); vocational programmes of short duration (1.1%) Primary school (2.7%); general, science, and social sciences high school (32.2%); Anatolian high school (22.5%); vocational high school (24.7%); multi programme high school (3.7%) Anatolian teacher training high school (4.5%) general upper secondary, compulsory (Students studying mostly toward gCSe) (97.7%); vocational upper secondary, compulsory (Students studying mostly towards a level 1 diploma) (0.9%) general upper secondary, postcompulsory (Students studying mostly for AS or A levels) (1.1%) general upper secondary, non-specialised (6.7%); vocational upper secondary (40.8%) lower secondary (4.8%) colombia general upper secondary (35.7%); vocational upper secondary (25.2%) lower secondary (39.1%) croatia gymnasium (29.9%); four year vocational programmes (46.7%); vocational programmes for industry (6.5%); vocational programmes for crafts (15.2%); lower qualiication vocational programmes (0.8%) macao-china lower secondary (54.9%); general upper secondary (43.5%); Pre-vocational or vocational upper secondary (1.6%) turkey England (united kingdom) Partners general upper secondary (technical gymnasiums) (7.6%); basic (elementary) education (5.4%) bulgaria malaysia general upper secondary, specialised (47.6%) vocational upper secondary (13.3%) Arts upper secondary (44.8%); religious secondary (3.3%); lower secondary (4.0%) general upper secondary school or gymnasium (33.6%); four-year vocational secondary (60.0%); three-year vocational secondary (6.0%); montenegro russian federation general upper secondary (13.4%) lower secondary (82.5%); vocational upper secondary (technikum, college, etc.) (2.2%) Serbia Arts upper secondary (1.6%) general upper secondary (gymnasium) (24.0%); technical upper secondary (30.3%); vocational technical upper secondary (6.5%); medical upper secondary (9.3%); economic upper secondary (18.8%); vocational economic upper secondary (3.0%); Agricultural upper secondary (4.2%) Shanghai-china vocational upper secondary (19.8%) general upper secondary (34.3%) uruguay vocational upper secondary (professional schools, etc.) (1.9%) general lower secondary (44.4%) Junior high school (36.4%); Senior high school (29.1%); vocational senior high school (30.6%); five-year college (not including the last two years) (4.0%) chinese taipei united arab Emirates Science upper secondary (34.6%) vocational secondary (2.7%) general lower secondary (15.0%); general upper secondary (82.3%) general lower secondary (31.4%); lower secondary with a technological component (5.3%); lower Secondary with a very important technological component (2.9%); vocational lower secondary (1.3%); general upper secondary (50.2%); vocational upper secondary (more than one year) (1.3%) technical upper secondary (6.2%) Note: numbers in parentheses indicate the proportion of 15-year-olds in the study programme; percentages may not add up to 100 within each country/economy because of rounding and of rare programmes for which results are not reported. only countries/economies with results reported for more than one study programme are included in this igure. The middle column includes all programmes for which relative performance in problem solving is not statistically different from 0 (see Annex A3). In belgium, the information about study programmes in variable PrOgn was combined with information about regions to identify education tracks: “fl.” refers to the flemish Community, “fr.” to the french Community, and “ger.” to the german-speaking Community; results for “Part-time vocational” programmes and “Special education” programmes are reported at the national level. in germany, students in schools with multiple study programmes are classiied according to their speciic education track. Source: OECD, PISA 2012 Database, Table V.4.5. 12 http://dx.doi.org/10.1787/888933003611 Students in arts upper secondary programmes in Serbia seem to beat expectations by an even greater margin than other students in that country, but fewer than 2% of all 15-year-olds are in these programmes. In Italy, students who are held back in lower secondary education (about 2.6% of all 15-year-olds) are relatively weak in problem solving, even after accounting for differences in mathematics, reading and science performance. These students, therefore, do not seem to contribute to the overall (relative) strength of Italy’s students in problem solving. Strong performance in problem solving among students in certain education tracks, relative to their performance in the other subjects assessed by PISA, can be interpreted in two ways. On the one hand, the curriculum and teaching practices 98 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS in these programmes may promote authentic learning, and equip students for tackling complex, real-life problems in contexts that they do not usually encounter at school. on the other hand, better-than-expected performance in problem solving may be an indication that in these programmes, students’ potential is not nurtured as much as it could be within the core academic subjects. gender dIFFerenceS In Problem SolvIng differences between boys and girls can be analysed in terms of overall proficiency in problem solving, in relation to performance differences observed in other domains, and in terms of the distinct cognitive abilities that are emphasised by different families of assessment tasks. boys score seven points higher than girls in problem solving, on average across oeCd countries (figure v.4.4). the variation observed among boys is also larger than the variation observed among girls. the standard deviation among boys is 100 score points, while the standard deviation among girls is only 91 score points. Similarly, the distance between the top (95th percentile) and the bottom (5th percentile) of the performance distribution is significantly larger among boys than among girls (table v.4.7). • figure v.4.4 • gender differences in problem-solving performance boys All students girls Gender differences (boys – girls) mean score in problem solving United Arab Emirates Bulgaria Finland Montenegro Slovenia Sweden Norway Poland Spain Australia United States Hungary France Estonia Netherlands Ireland Canada England (United Kingdom) Israel Germany OECD average Czech Republic Malaysia Belgium Russian Federation Singapore Denmark Macao-China Uruguay Austria Chinese Taipei Korea Chile Hong Kong-China Serbia Turkey Croatia Portugal Italy Japan Slovak Republic Brazil Shanghai-China Colombia 350 400 450 500 550 600 Boys perform better Girls perform better oEcd average 7 score points -40 mean score -20 0 20 40 Score-point difference Note: Statistically signiicant gender differences are marked in a darker tone (see Annex A3). Countries and economies are ranked in ascending order of the score-point difference (boys - girls). Source: OECD, PISA 2012 Database, Tables V.2.2 and V.4.7. 1 2 http://dx.doi.org/10.1787/888933003611 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 99 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS on average across oeCd countries, boys are more likely than girls to perform at the highest levels in problem solving. the proportion of top-performing boys is 1.50 times larger than the proportion of top-performing girls. girls and boys are equally represented at the lowest levels of performance (below level 2) (figure v.4.5 and table v.4.6). in more than half of the countries and economies that participated in the assessment of problem solving, boys outperform girls, on average. the largest advantage in favour of boys is found in Colombia, Shanghai-China, brazil and the Slovak republic, where the difference exceeds 20 score points. Among the exceptions are the united Arab emirates, bulgaria, finland and montenegro, where girls outperform boys, on average. in 16 countries/economies, the difference in performance between boys and girls is not statistically significant (figure v.4.4 and table v.4.7). • figure v.4.5 • Proiciency in problem solving among girls and boys boys Girls 3.1 10.0 19.0 Level 4 24.5 26.8 Level 3 20.7 15 10 13.5 Level 1 7.8 Below Level 1 8.7 20 23.3 Level 2 12.8 25 7.7 Level 5 20.2 % 30 1.8 Level 6 5 0 0 5 10 15 20 25 30 % Source: oeCd, PiSA 2012 database, table v.4.6. 1 2 http://dx.doi.org/10.1787/888933003611 A greater variation in performance among boys than among girls is found in nearly every country/economy. the standard deviation for boys exceeds the standard deviation for girls by more than 15 score points in israel, the united Arab emirates and italy. there is no country or economy where the standard deviation for boys is smaller than the standard deviation for girls. in ten countries and economies, the standard deviation for boys and girls is about the same (table v.4.7). because the better performance of boys is accompanied by greater variation in performance, in several countries there are more boys at both the highest levels of performance – in line with higher average performance levels – and the lowest levels of performance – in line with the greater variation in performance. boys tend to be under-represented among students in the middle range of performance. in Croatia, italy and the Slovak republic, boys are as likely as girls to be low-achievers, but are more than twice as likely to be top performers as girls. in no single country/economy are there more girls than boys among the top performers in problem solving (table v.4.6). How gender differences in problem-solving performance compare to differences in mathematics, reading and science performance the greater variation in the results of boys, relative to the variation observed among girls, is not unique to problem solving. it is, in fact, a common finding across the PiSA assessments. the performance variation observed among boys is about 1.2 times larger than that observed among girls, on average across countries – similar to the ratio observed in mathematics, reading and science (table v.4.9). Across the subjects assessed by PiSA, gender differences in mean performance vary greatly. girls outperform boys in reading; but boys outperform girls in mathematics. the advantage of girls in reading is as large as 40% of a standard deviation, on average, across oeCd countries participating in the assessment of problem solving; while the advantage of boys in mathematics is equivalent to 11% of a standard deviation. in science, no clear advantage 100 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS for either boys or girls is found. boys’ advantage in problem solving (7% of a standard deviation on average across oeCd countries) is thus lower than the advantage for boys in mathematics, but larger than the gender gap observed in science (figure v.4.6). it is not clear whether one should expect there to be a gender gap in problem solving. on the one hand, the questions posed in the PiSA problem-solving assessment were not grounded in content knowledge, so boys’ or girls’ advantage in having mastered a particular subject area should not have influenced results. on the other hand, as shown in Chapter 2 (figure v.2.13), performance in problem solving is more closely related to performance in mathematics than to performance in reading. one could therefore expect the gender difference in performance to be closer to that observed in mathematics – a modest advantage for boys, in most countries – than to that observed in reading – a large advantage for girls. • figure v.4.6 • difference between boys and girls in problem-solving, mathematics, reading and science performance Expressed as a percentage of the overall variation in performance Problem solving mathematics reading Science Score difference as a percentage of the standard deviation 40 20 0 -20 -40 -60 Colombia Shanghai-China Brazil Slovak Republic Japan Italy Turkey Portugal Serbia Croatia Chile Hong Kong-China Korea Chinese Taipei Macao-China Austria Uruguay Denmark Singapore Malaysia Russian Federation Czech Republic Belgium OECD average Germany England (United Kingdom) Ireland Estonia Canada Netherlands Israel France United States Hungary Australia Spain Poland Norway Sweden Slovenia Montenegro Finland Bulgaria United Arab Emirates -80 Notes: gender differences that are statistically signiicant are marked in a darker tone (see Annex A3). All gender differences in reading performance are statistically signiicant. Countries and economies are ranked in descending order of the gender difference in problem solving (boys - girls). Source: OECD, PISA 2012 Database, Table V.4.8. 1 2 http://dx.doi.org/10.1787/888933003611 An analysis accounting for performance differences in curricular subjects shows that the gender gap in problem solving is largely the result of boys’ strengths in the skills that are uniquely measured by the problem-solving assessment. indeed, because the small disadvantage of girls in mathematics is counterbalanced by a large advantage in reading, when the analysis accounts for performance across all three subjects (mathematics, reading and science) – as shown in figure v.4.7 – the resulting gender gap in the relative performance in problem solving (8 score points, in favour of boys) is not much different from the actual gender gap in problem solving. there are few studies that focus on gender differences in problem solving (see Hyde, 2005; Wüstenberg et al., 2014). the results of the PiSA 2003 assessment of problem solving showed very few countries in which there were significant gender differences in performance (oeCd, 2005). However, the PiSA 2003 assessment was limited to static problem situations, and its results cannot be compared with those of the PiSA 2012 assessment. moreover, the PiSA 2003 assessment was a paper-based assessment, whereas the PiSA 2012 assessment of problem solving was delivered by computer. CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 101 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS • figure v.4.7 • relative performance in problem solving among girls Difference in problem-solving performance between girls and boys with similar performance in mathematics, reading and science Score-point difference 15 10 Girls perform above their expected level 5 0 -5 -10 -15 -20 -25 -30 Girls perform below their expected level Slovak Republic Shanghai-China Italy Turkey Poland Austria Malaysia Croatia Chinese Taipei Serbia Russian Federation Colombia Estonia Portugal Uruguay Macao-China Hong Kong-China Brazil Hungary Japan Korea Germany Singapore Czech Republic Israel OECD average Finland Slovenia Canada Belgium United States France Netherlands Denmark Montenegro Chile Norway Ireland Sweden Spain Bulgaria Australia United Arab Emirates England (United Kingdom) -35 Note: Statistically signiicant differences are marked in a darker tone (see Annex A3). Countries and economies are ranked in descending order of the score-point difference in problem solving between girls and boys with similar performance in mathematics, reading and science. Source: OECD, PISA 2012 Database, Table V.4.10. 1 2 http://dx.doi.org/10.1787/888933003611 in countries that also used computer-based instruments to assess mathematics and reading, boys perform better, relative to girls, in the computer test than in the paper test. in mathematics, the computer-based assessment shows a larger advantage for boys than girls; in reading, a smaller disadvantage for boys relative to girls (table v.4.8). one can therefore speculate that the computer delivery of the problem-solving assessment contributed to the better performance of boys over girls in the assessment. Differences in performance patterns across items Performance differences between boys and girls vary across the problem-solving assessment, depending on the type of task involved. for example, a comparison of success rates for boys and girls across items representing the four major problem-solving processes identified in the framework – “exploring and understanding”, “representing and formulating”, “planning and executing”, and “monitoring and reflecting” – reveals sharp contrasts. figure v.4.8 shows that girls perform better – and thus, in most cases, at similar levels as boys – on items measuring the “planning and executing” aspect. table v.4.11b shows that, on average across oeCd countries, the success ratio (i.e. the ratio of full-credit over no-credit and partial-credit answers) on these items for girls is 0.96 times the success ratio for boys – i.e. only slightly below that of boys. in contrast, girls’ performance is lower on items measuring the “representing and formulating” aspect. Here, the success ratio among girls is only 0.84 times as high as that among boys. After accounting for their lower overall success on the assessment, as in figure v.4.8, the “planning and executing” tasks that measure knowledge-utilisation processes appear to be a strong point for girls, while the more abstract “representing and formulating” tasks, which relate to knowledge-acquisition processes, appear to be a weak point for girls. based on the existing psychometric literature (see, for a review, Halpern and lamay, 2000), a difference, in favour of boys, on items that require a greater amount of abstract information processing could be expected. this literature finds consistent gender differences on some tests of cognitive abilities. the most frequently cited difference is in the ability 102 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS to transform a visual-spatial image in working memory. According to the literature, males often perform better than females on cognitive tasks requiring the ability to generate and manipulate the information in a mental representation. in the PiSA assessment of problem solving, this ability is particularly important for success on “representing and formulating” tasks. • figure v.4.8 • girls’ strengths and weaknesses, by problem-solving process Relative likelihood of success in favour of girls, accounting for overall performance differences on the test boys (= 1.00) girls’ success rate, relative to boys Exploring and understanding 0.99 girls have weaker-than-expected performance on “representing and formulating” tasks Monitoring and reflecting 1.06 0.89 representing and formulating girls have stronger-than-expected performance on “monitoring and reflecting” and “planning and executing” tasks 1.06 Planning and executing Notes: gender differences that are statistically signiicant are marked in bold (see Annex A3). This igure shows that girls’ success rate on items measuring the processes of “representing and formulating” is only 0.89 times as large as that of boys, after accounting for overall performance differences on the test and on average across OECD countries. Source: OECD, PISA 2012 Database, Table V.4.11b. 1 2 http://dx.doi.org/10.1787/888933003611 the profile of performance across problem-solving processes differs significantly between boys and girls in 27 of the 44 countries and economies participating in the assessment.1 in all but three of these countries/economies, girls perform below their expected level of performance in particular on items measuring “representing and formulating” processes (table v.4.11b). in korea, girls score lower than boys on the overall problem-solving scale. An analysis by families of items shows that girls’ performance is much weaker than boys’ on items measuring “exploring and understanding” and “representing and formulating” processes, but is close to boys’ performance (and thus stronger than expected) on “planning and executing” and “monitoring and reflecting” tasks. therefore, the good performance of korea on the problem-solving assessment, which is mainly attributed to the stronger-than-expected performance of its students on items measuring knowledge acquisition (see Chapter 3), results in part from boys’ strong performance on these items. A similar pattern applies to Hong kong-China and macao-China as well: in both, boys outperform girls overall, and on knowledge-acquisition tasks in particular, but not on knowledge-utilisation tasks (table v.4.11b). in contrast, in many european countries, including those with above-average performance in problem solving, such as france, the netherlands, italy and germany, the performance patterns for boys and girls are similar across the various problem-solving processes. in Spain, Hong kong-China, korea and macao-China, girls’ performance is weaker than boys’ performance on items measuring “exploring and understanding” processes, after accounting for overall differences in performance between boys and girls. in the remaining countries/economies, the evidence is not strong enough to identify different patterns for boys and girls. CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 103 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS on items measuring “representing and formulating” processes, girls’ performance is weaker than boys’ in 24 countries and economies, after accounting for overall differences in performance between boys and girls. the performance difference on these items, relative to the remaining test items, is largest in Shanghai-China, Colombia, korea and Hong kong-China, where girls perform only 0.8 times (at best) as well as expected. in the remaining 20 countries / economies, the evidence is not strong enough to identify different patterns for boys and girls (table v.4.11b). girls’ performance is stronger than boys’ on “planning and executing” items in Hong kong-China, korea, Chinese taipei, brazil, Japan, Portugal, Singapore, macao-China, england (united kingdom), Australia, Serbia and finland, after accounting for overall differences in performance between boys and girls. in all these countries and economies except finland, girls perform at lower levels than boys, on average (but not significantly so in Chinese taipei, england (united kingdom) and Australia). in contrast, in finland girls perform better than boys, on average; and this analysis shows that girls’ strong performance overall stems mainly from their better performance on tasks measuring the process of “planning and executing” compared to boys (table v.4.11b). finally, in Colombia, Shanghai-China, denmark, Chile, korea, malaysia, england (united kingdom) and Australia, girls perform better than boys on “monitoring and reflecting” items (table v.4.11b). the interactive or static nature of the problem situation is not associated with gender differences in performance, on average across oeCd countries (table v.4.11a): girls’ performance on interactive items is similar to their performance on static items. the relative success ratio (odds ratio) on interactive items for girls compared to boys (0.92) is about the same ratio as observed on static items (0.93). large differences in performance are found in Chile and Hungary, where girls perform more than 1.2 times worse on interactive items than on static items. Compared to boys in these two countries, girls seem to be particularly good at analysing and solving static problem situations – and weak at analysing and solving interactive problem situations. the opposite pattern is found in montenegro, where girls perform more than 1.2 times better on interactive items than on static items. because differences between girls’ performance on static and their performance on interactive items are not systematic, the inclusion of interactive items in the PiSA 2012 assessment cannot explain why the results of the PiSA 2012 assessment indicate larger gender differences in problem-solving skills than the results of the PiSA 2003 assessment, which found no difference, on average across oeCd countries. Similarly, in the PiSA assessment of problem solving, there are no large gender differences in the patterns of performance that are related to the context of the problem. on average, girls’ success rates are similar to those of boys – after accounting for overall differences across the test – on items situated in “personal” contexts, involving close relations, and on items that are cast in wider, impersonal contexts (“social” contexts). girls tend to have slightly better performance on items involving technology devices than on those in non-technology settings. the overwhelming use of problem contexts that come from male-dominated fields (such as sports, weapons or cars) has been proposed as one reason behind gender differences in assessments of mathematical problem solving (fennema, 2000), but does not seem to explain the performance differences found in PiSA 2012 (tables v.4.11c and v.4.11d). there are no differences in the pattern of performance according to the response format: success rates for boys and girls are, in general, similarly balanced on selected-response and constructed-response items (table v.4.11e). The relATIonShIP beTween SocIo-economIc STATuS, ImmIgrAnT bAcKground And Problem-SolvIng PerFormAnce Performance differences related to socio-economic status unsurprisingly, socio-economic status – as measured, for instance, by the PISA index of economic, social and cultural status (eSCS) – relates positively to performance in problem solving, as it does indeed to performance in all domains assessed in PiSA. but how do differences in performance by socio-economic status compare across domains? in general, the strength of the association between performance and socio-economic status, measured as the percentage of variation in performance explained by socio-economic disparities, is similar for mathematics (the oeCd average is 14.9%), reading (13.2%) and science (14.0%). interestingly, figure v.4.9a shows that this relationship is weaker in problem solving than in the three other domains. Still, even in problem solving, about 10.6% of the variation in performance can be explained by differences in socio-economic status; and on average, a one-unit increase in the eSCS index is associated with a score difference of 35 points in problem solving (table v.4.13). 104 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS • figure v.4.9a • Strength of the relationship between socio-economic status and performance in problem solving, mathematics, reading and science Percentage of variation in performance explained by socio-economic status Problem solving % mathematics reading Science 30 25 20 15 10 5 Slovak Republic Bulgaria Hungary Uruguay Chile Portugal Turkey Malaysia Brazil Czech Republic Shanghai-China Israel Belgium Serbia France Germany Slovenia Colombia Poland Russian Federation Austria Singapore Ireland OECD average Montenegro United States Chinese Taipei Croatia Netherlands Australia Spain Denmark England (United Kingdom) Finland United Arab Emirates Italy Sweden Korea Estonia Japan Norway Canada Hong Kong-China Macao-China 0 Note: All values are statistically signiicant (see Annex A3). Countries and economies are ranked in ascending order of the strength of the relationship between performance in problem solving and the PISA index of economic, social and cultural status (ESCS). Source: OECD, PISA 2012 Database, Table V.4.13. 1 2 http://dx.doi.org/10.1787/888933003611 As exceptions to this pattern, in the Czech republic and turkey, as well as in partner countries/economies brazil, malaysia, the russian federation, Serbia and Shanghai-China, the impact of socio-economic status on performance is as strong in problem solving as in mathematics. in no country, however, is the impact of socio-economic status stronger on problem solving than on mathematics performance (figure v.4.9a and table v.4.13). figure v.4.9b further explores the mechanisms through which socio-economic status is related to problem-solving performance. it shows that within the same school, students’ performance in problem solving is almost unrelated to their socio-economic status. However, at the school level, schools with more advantaged student populations often perform better in problem solving, while schools with more disadvantaged student populations often perform poorly in problem solving. this school-level association, however, is not distinct from the one observed in mathematics: the schools that have more disadvantaged student populations and poor results in mathematics tend to perform poorly in problem solving too. the variation in performance between schools that is unique to problem solving and can be accounted for by differences in students’ and schools’ socio-economic status represents only 0.2% of the total variation in performance in problem solving (table v.4.14). thus, the socio-economic status of students does not appear to have a direct association with their performance in problem solving. instead, socio-economic disparities in problem-solving performance reflect, to a large part, unequal access to good teachers and schools, not a domain-specific disadvantage. A simpler measure of socio-economic advantage yields the same conclusion: socio-economic differences have a weaker influence on problem-solving performance than on performance in curricular domains, and this influence is not due to a specific association between problem-solving performance and socio-economic disadvantage, but rather to the poorer performance, overall, observed among disadvantaged students. this simpler measure classifies students according to the highest occupational status of their father or mother. the higher-status group includes the children of managers, professionals, CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 105 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS technicians and associate professionals, such as teachers. on average across oeCd countries, 51% of students are in this higher-status group; 43% are in the lower-status group, with their parents in semi-skilled or elementary occupations; and 6% have missing or incomplete information on both parents’ occupation, and are therefore excluded from this analysis. • figure v.4.9b • Strength of the relationship between socio-economic status and performance in problem solving, between and within schools Percentage of variation in performance explained by socio-economic status of students and schools As a percentage of the total variation in performance 30 20 Variation unique to problem solving 10 0 10 Variation shared with performance in mathematics 12.4% 0.2% 20.4% 0.5% 1.6% 17.4% 20 7.8% 39.0% Variation between schools (37.8%) Variation within school (61.5%) 30 40 50 Notes: The igure shows the components of the performance variation in problem solving for the OECD average. The variation in performance accounted for by the PISA index of economic, social and cultural status (ESCS) of students and schools is marked in blue. Estimates shown in this igure exclude students with missing information on the ESCS. Source: OECD, PISA 2012 Database, Table V.4.14. 1 2 http://dx.doi.org/10.1787/888933003611 Students who have at least one parent in highly skilled occupation score 45 points higher than students whose parents work in semi-skilled occupations or in elementary occupations, on average across oeCd countries. the performance gap in problem solving related to parents’ highest occupational status amounts to almost half a standard deviation (48%) (figure v.4.10). However, this gap is smaller than that observed in performance in mathematics (57%) reading and science (both 56%). in norway, Hungary and the russian federation, the performance gap related to parents’ highest occupational status is of the same magnitude in problem solving as in mathematics, reading and science; in Shanghai-China, ireland and italy, the gap is as large as in mathematics, but smaller than in reading; and in Serbia, the united Arab emirates and malaysia, it is as large in problem solving as in mathematics and larger than in reading. in all other countries and economies, the performance gap in problem solving related to parents’ occupational status is smaller than that observed in mathematics, and often in the remaining domains as well. in france, Spain and Chinese taipei, the gap observed in mathematics performance exceeds that observed in problem solving by more than one-sixth of a standard deviation (table v.4.16). the differences in problem solving performance related to parents’ occupational status can be decomposed into two components. the first is poorer performance overall: students from lower-status families tend to perform less well in PiSA than high-status students, irrespective of the school subject. the second is specific to problem solving. it reflects differences, across groups, in how academic potential translates into performance in problem solving, as well as differences in the skills uniquely measured by problem solving. in Chapter 2, the overall variation in problem-solving proficiency was similarly split into two components – one that is common to mathematics, reading and science (68%), and a residual component that is unique to problem solving (32%) (table v.2.5). if the performance gap related to parents’ occupational status reflected only poorer performance overall, it would not affect this residual component, and the size of the gap in problem-solving proficiency would be smaller than that in curricular subjects assessed by PiSA.2 106 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS • figure v.4.10 • difference related to parents’ occupational status in problem-solving, mathematics, reading and science performance Score difference between students whose parents’ highest occupation is skilled and students whose parents’ highest occupation is semi-skilled or elementary expressed as a percentage of the overall variation in performance Problem solving mathematics reading Science Score difference as a percentage of the standard deviation 100 90 80 70 60 50 40 30 20 10 Bulgaria Uruguay Slovak Republic Israel Hungary Serbia Malaysia United Arab Emirates Czech Republic Chile Portugal Poland Russian Federation Belgium Germany Austria Croatia Slovenia Montenegro Brazil Netherlands Colombia Singapore France Turkey Ireland OECD average United States Shanghai-China Sweden Chinese Taipei Spain Denmark England (United Kingdom) Italy Australia Estonia Finland Norway Canada Hong Kong-China Japan Korea Macao-China 0 Notes: All values are statistically signiicant (see Annex A3). Semi-skilled or elementary occupations include major ISCO groups 4, 5, 6, 7, 8 and 9. Skilled occupations include major ISCO groups 1, 2 and 3. Countries and economies are ranked in ascending order of the difference in problem-solving performance between students whose parents’ highest occupation is skilled and students whose parents’ highest occupation is semi-skilled or elementary. Source: OECD, PISA 2012 Database, Table V.4.16. 1 2 http://dx.doi.org/10.1787/888933003611 to what extent does the performance gap related to parents’ occupational status reflect a specific difficulty with problem solving rather than poorer performance overall? to identify specific difficulties with problem solving, the performance of lower-status students is compared with that of higher-status students who share similar performance in mathematics, reading and science. on average across oeCd countries, students whose parents work in semi-skilled and elementary occupations perform close to their expected level in problem solving, given their performance in mathematics, reading and science. the analysis of PiSA data indicates that the poorer performance in problem solving observed among more disadvantaged students is not related to a specific difficulty with the skills assessed in this domain, but with poorer performance, in general, that is observed across the subjects assessed. in france, Chinese taipei, estonia and Canada, however, students whose parents work in occupations considered as semi-skilled or elementary tend to perform better in problem solving than students with the same mathematics, reading and science scores, but at least one of whose parents works in an occupation considered as skilled. one interpretation of this result is that, in these countries/economies, the potential of students from more disadvantaged families is not realised in curricular subjects. As a result, these students appear weaker in mathematics, reading and science than they do in problem solving. in contrast, in the russian federation, the united Arab emirates, malaysia, Serbia and the Slovak republic, more disadvantaged students score lower in problem solving than students of similar performance in core academic subjects. in these countries, poor proficiency in the skills specific to problem solving contributes to disadvantaged students’ low performance in problem solving (figure v.4.11 and table v.4.17). CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 107 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS • figure v.4.11 • relative performance in problem solving among students whose parents work in semi-skilled or elementary occupations Difference in problem-solving performance between lower-status students and higher-status students with similar performance in mathematics, reading and science Score-point difference 15 Students whose parents work in semi-skilled or elementary occupations perform above their expected level in problem solving 10 5 0 -5 -10 Russian Federation United Arab Emirates Serbia Malaysia Hungary Slovak Republic Bulgaria Uruguay Ireland Montenegro Italy Poland Norway Slovenia Shanghai-China Brazil Colombia Israel Singapore Turkey Czech Republic Croatia Hong Kong-China Korea Austria Netherlands OECD average Japan Australia Germany Finland Macao-China Sweden Denmark Chile Belgium United States Spain Canada Estonia England (United Kingdom) France Chinese Taipei -20 Portugal Students whose parents work in semi-skilled or elementary occupations perform below their expected level in problem solving -15 Notes: Statistically signiicant differences are marked in a darker tone (see Annex A3). lower-status students refers to students whose parents’ highest occupation is semi-skilled or elementary; semi-skilled or elementary occupations include major ISCO groups 4, 5, 6, 7, 8 and 9. Higher-status students refers to students whose parent’s highest occupation is skilled; skilled occupations include major ISCO groups 1, 2 and 3. Countries and economies are ranked in descending order of the score-point difference in problem solving between students whose parents’ highest occupation is semi-skilled or elementary and students with similar performance in mathematics, reading and science whose parents’ highest occupation is skilled. Source: OECD, PISA 2012 Database, Table V.4.17. 1 2 http://dx.doi.org/10.1787/888933003611 Performance patterns among advantaged and disadvantaged students do students from socio-economically disadvantaged backgrounds have different strengths and weaknesses in problem solving than students from more advantaged backgrounds, once their overall performance differences have been accounted for? in general, students with at least one parent who works in a skilled occupation have the same pattern of performance on static and interactive items as students with parents who work in semi-skilled or elementary occupations; and the pattern of performance, by problem context, is also similar across the two groups. there are slight differences according to response format, in that students from more advantaged backgrounds have relatively more ease with items requiring constructed responses, while more disadvantaged students perform better on selected-response items. All these analyses adjust for the difficulty of items (tables v.4.18a, v.4.18c, v.4.18d and v.4.18e). looking at the performance profile across items measuring the four problem-solving processes, the largest differences in performance related to parents’ occupational status are found in items measuring “exploring and understanding” and “representing and formulating” processes (figure v.4.12 and table v.4.18b). these are the tasks related to knowledge acquisition and abstract information-processing. in contrast, performance differences are narrower in “planning and executing” and “monitoring and reflecting” tasks. on “exploring and understanding” items, a larger-than-expected performance gap between higher- and lower-status students is observed, particularly in italy, Singapore, Austria, Canada and the united States. in these countries, the odds ratio for exploring and understanding items (a measure of the likelihood of success on these items, relative to all other items) 108 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS is more than 1.2 times larger for higher-status students than for lower-status students. by the same measure, in Chile, brazil, Sweden and uruguay, the performance gap in “representing and formulating” items is significantly wider than on other items, on average. on “planning and executing” items, the performance gap between higher- and lower-status students in Shanghai-China, turkey, Austria, Hong kong-China, Canada, Singapore, italy and Chile is between 1.15 and 1.20 times smaller than (or between 0.83 and 0.87 times as large as) on the remaining items. in these countries/economies, lower-status students reduce the performance gap substantially in items requiring them to set goals, devise a plan, and carry it out. these tasks are often introduced by concrete action verbs, such as “buy”, “go to”, and others that explicitly invite the student to interact with the system or device, in contrast to “representing and formulating” items, where the task is more abstract (e.g. “complete the diagram”). • figure v.4.12 • Strengths and weaknesses in problem solving among students with at least one parent working in skilled occupations, by process Relative likelihood of success in favour of students whose parents’ highest occupation is skilled, accounting for overall performance differences on the test Students whose parents’ highest occupation is semi-skilled or elementary (= 1.00) Success rate of students whose parents’ highest occupation is skilled, relative to students whose parents’ highest occupation is semi-skilled or elementary Exploring and understanding 1.09 Higher-status students have stronger-than-expected performance on knowledge-acquisition tasks Monitoring and reflecting representing and formulating 0.92 1.08 0.94 Planning and executing Notes: All differences between students with parents in skilled occupations and those with parents in semi-skilled or elementary occupations are statistically signiicant (see Annex A3). Higher-status students refers to students whose parents’ highest occupation is skilled. knowledge-acquisition tasks refers to tasks measuring the processes of “exploring and understanding” or “representing and formulating”. This igure shows that the success rate on items measuring the processes of “representing and formulating” is 1.08 times larger among students with at least one parent working in a skilled occupation, compared to students whose parents’ highest occupation is semi-skilled or elementary, after accounting for overall performance differences on the test and on average across OECD countries. Source: OECD, PISA 2012 Database, Table V.4.18b. 1 2 http://dx.doi.org/10.1787/888933003611 in Colombia and england (united kingdom), the performance gap is substantially narrower on “monitoring and reflecting” items (more than 1.2 times smaller, or less than 0.83 times as large). in contrast, in Shanghai-China, the gap in performance on “monitoring and reflecting” items is larger than that on all remaining items, on average (table v.4.18b). differences in performance profiles related to parents’ highest occupational status may stem from greater access to opportunities for developing problem-solving skills both in and outside of school. data from the oeCd Survey of Adult Skills (oeCd, 2013a) show that workers in occupations considered as skilled encounter abstract information-processing tasks and problems that require at least 30 minutes to solve much more frequently in their job than workers in semi-skilled or elementary occupations. these adults are more familiar with complex problem-solving tasks, and may be particularly good at them, thus they may value their children’s success on abstract problem-solving tasks to a greater extent. CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 109 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS Immigrant background and student performance in many countries and economies, children of immigrants are more at risk of low performance in education than the children of parents who were born in the country. A gap in problem-solving performance between immigrant and non-immigrant students is observed as well: children of immigrants tend to perform significantly below non-immigrant students (by 32 score points, on average across the oeCd), and immigrant students are 1.77 times more likely than non-immigrant students to score below level 2. this is not always the case, however: in the united Arab emirates, israel, montenegro, Singapore, Australia and macao-China, immigrant students score better than non-immigrant students in problem solving (table v.4.19). When performance differences between immigrant and non-immigrant students are compared across domains, the difference observed in problem-solving performance appears similar to that observed in mathematics and reading, but smaller than that observed in science, on average (table v.4.20). • figure v.4.13 • relative performance in problem solving among immigrant students Difference in problem-solving performance between immigrant students and non-immigrant students with similar performance in mathematics, reading, and science Score-point difference 60 Immigrant students perform above their expected level in problem solving 50 40 30 20 10 0 -10 -20 Immigrant students perform below their expected level in problem solving England (United Kingdom) Netherlands Shanghai-China Italy Denmark France Australia Belgium Ireland Colombia Serbia Canada Austria Norway Macao-China Hungary Hong Kong-China Sweden Singapore OECD average Estonia Czech Republic United States Chile Montenegro Finland Germany Portugal Slovenia Malaysia United Arab Emirates Turkey Russian Federation Israel Croatia Spain Chinese Taipei Brazil Slovak Republic -30 Note: Statistically signiicant differences are marked in a darker tone (see Annex A3). Countries and economies are ranked in descending order of the score-point difference in problem solving between immigrant students and non-immigrant students with similar performance in mathematics, reading and science. Source: OECD, PISA 2012 Database, Table V.4.21. 1 2 http://dx.doi.org/10.1787/888933003611 figure v.4.13 compares immigrant students’ performance in problem solving with the performance of non-immigrant students who perform similarly in mathematics, reading and science. on average across oeCd countries, there is no difference in the problem-solving performance between the two groups. Significant differences are found in 18 of the 39 countries/economies with sufficient data. this implies that, in many countries/economies, immigrant students’ poorer (or sometimes, better) performance in problem solving is related to differences that affect academic performance, in general, rather than problem-solving performance in particular. When it comes to problem solving, immigrant students in brazil, Spain, israel, Croatia, the russian federation and the united Arab emirates perform better than non-immigrant students with similar mathematics, reading and science scores. in these countries, immigrant students are either particularly good at problem solving – or perform below their potential in the assessments of curricular subjects. in contrast, in england (united kingdom), denmark, italy, Australia, 110 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS france, belgium, ireland, Canada, Serbia, macao-China, Hong kong-China and Singapore, immigrant students perform worse in problem solving than a comparison group of non-immigrant students who have similar scores in mathematics, reading and science. in these countries/economies, the poorer performance of immigrant students indicates a specific difficulty in the skills uniquely measured by the assessment of problem solving (figure v.4.13). how STudenTS’ SelF-rePorTed dISPoSITIonS TowArdS Problem SolvIng relATe To PerFormAnce A recurrent theme in the literature about problem solving is that problem solving is personal and directed; that is, the problem-solver’s processing of the problem situation is guided by his or her personal goals (mayer and Wittrock, 2006). motivational and affective factors at work in a specific problem situation may be influenced by the context (whether it is familiar or not), the constraints and resources available, the pay-offs attached to the eventual outcomes, and the incentives related to the possible actions. there is no doubt that performance on the PiSA test of problem solving is influenced by affective and motivational factors in addition to cognitive potential. the willingness to engage with the problems is perhaps influenced by the assessment situation (e.g. the assessment has low stakes for students and takes place at school) or its mode of delivery (the computerbased interface). to gauge differences in motivational and affective factors separately from differences in performance, the PiSA student questionnaire includes questions measuring students’ perseverance and openness to problem solving. Average levels of perseverance and openness to problem solving, and their relation to gender, socio-economic status and performance in mathematics, are presented in Chapter 3 of volume iii, Ready to Learn. table v.4.23 analyses the relationship between students’ perseverance and openness to problem solving and their performance in problem solving. one of the main results of analyses in Chapter 3 of volume iii is that high achievement in mathematics almost always corresponds to high levels on the index of openness to problem solving, a measure of general drive and motivation (not related to mathematics contexts) (oeCd, 2013b). High levels of openness to problem solving are no guarantee of high performance; in fact, the lowest-performing students among those with low levels of motivation show similar performance on the PiSA assessment as the lowest-performing students among those with high levels of motivation. but at the top of the performance distribution, openness to problem solving is associated with large performance differences. the association between perseverance and performance in mathematics is also stronger among high-achieving students than among low-achieving students, although the difference is less marked than that related to openness to problem solving. everything in the PiSA data indicates that high levels of perseverance and openness to problem solving work as a catalyst for ever-higher performance among the most talented students. When the same analyses are repeated using performance in problem solving instead of performance in mathematics, the same conclusion emerges: perseverance and, even more so, openness to problem solving are strongly associated with performance, particularly at the highest levels of proficiency. this shows that students’ ability to perform at high levels is not only a function of their aptitude and talent; if students do not cultivate their intelligence with hard work and perseverance, they will not achieve mastery in any field. moreover, general drive and motivation appear to spur high performance in all situations in which students encounter cognitive challenges, not just in an assessment of mathematics. how Problem-SolvIng PerFormAnce relATeS To dIFFerenceS In IcT uSe AcroSS STudenTS Since problem-solving skills were assessed with a computer-based test in PiSA 2012, familiarity with computers may have contributed to students’ performance on the test. PiSA data show that access to a home computer is now nearly universal for students in all countries and economies participating in PiSA. on average across oeCd countries that participated in the problem-solving assessment, 94% of students have at least one computer at home to use for schoolwork. only in Colombia, turkey, malaysia, Japan, brazil, Shanghai-China, Chile, uruguay and estonia is this proportion smaller than 90%. Accordingly, use of computers at home is also nearly universal (table v.4.24). Across the oeCd countries that distributed the optional questionnaire on familiarity with information and communication technology (iCt) and participated in the problem-solving assessment, 95% of students, on average, use a desktop, laptop or tablet computer at home. in all countries except turkey, Japan, CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 111 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS korea, uruguay, Shanghai-China and Chile, more than 90% of students do (table v.4.25). the few students who do not use a computer at home tend to come from socio-economically disadvantaged families. but even among disadvantaged students, some level of familiarity with computers is now universal in some countries. in germany, denmark, finland, the netherlands, norway, Sweden and Austria, more than 98% of students whose parents work in semi-skilled or elementary occupations have and use a home computer. in all of the 33 countries and economies that both distributed the optional questionnaire on iCt familiarity and administered the computer-based assessment of problem solving, students who use computers at home perform significantly better than students who do not (figure v.4.14). because socio-economically advantaged students are more likely than disadvantaged students to use a computer at home, the performance advantage among students who use a computer at home tends to be smaller after accounting for students’ socio-economic status, gender and immigrant background. Still, in all 33 countries and economies, students who use a computer at home perform better than those who do not, even after accounting for these characteristics (a similarly strong relationship is observed between lack of computer use at home and performance on the paper-based assessments of mathematics and reading, as discussed at the end of this section); only in ireland, finland, italy and germany is the difference not statistically significant, possibly because the small sample of non-users results in imprecise estimates of their performance. • figure v.4.14 • difference in problem-solving performance related to the use of computers at home Difference in problem-solving performance between students who use a desktop, laptop or tablet computer at home and those who don’t Difference in problem-solving performance between students who use a desktop, laptop or tablet computer at home and those who don’t, after accounting for socio-demographic characteristics of students Score-point difference 120 100 80 60 40 20 Ireland 97.0 Russian Federation 91.6 Chile 87.0 Uruguay 84.4 Slovenia 96.2 Japan 81.4 Turkey 68.3 Finland 99.1 Singapore 95.4 Chinese Taipei 94.7 Italy 97.4 Shanghai-China 85.5 Portugal 96.0 Germany 99.1 Korea 83.5 Macao-China 97.2 Spain 96.6 Estonia 98.6 Poland 96.1 Hungary 94.7 OECD average 94.5 Hong Kong-China 97.5 Israel 96.1 Denmark 99.2 Sweden 98.5 Austria 98.7 Australia 97.1 Slovak Republic 94.3 Serbia 91.1 Croatia 97.0 Norway 98.7 Belgium 98.2 Czech Republic 97.4 Netherlands 98.9 0 Percentage of students who use a desktop, laptop or tablet computer at home Notes: Statistically signiicant differences are marked in a darker tone (see Annex A3). Only countries/economies that participated in the questionnaire on ICT familiarity and in the assessment of problem solving are shown in this igure. Countries are ranked in descending order of the score-point difference in problem-solving performance between students who use a desktop, laptop or tablet computer at home and those who don’t, after accounting for socio-demographic characteristics of students. Source: OECD, PISA 2012 Database, Table V.4.25. 1 2 http://dx.doi.org/10.1787/888933003611 using computers at school (whether desktop, laptop or tablet computers) is part of the school experience for 15-year-olds in most countries, but is not nearly as common as the use of computers at home. on average across oeCd countries, 72% of students reported that they use computers at school. in Shanghai-China, korea, turkey and uruguay, fewer than 50% of students reported that they use computers at school (in uruguay, 15-year-olds were too old to benefit from the Plan Ceibal, an initiative that began in 2007 and equips all children in primary school with a laptop computer). by contrast, in the netherlands, Australia and norway, more than 90% of students use a computer at school (table v.4.26). 112 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS there is no consistent pattern across countries in the performance difference between students who reported that they use computers at school and students who reported that they do not use computers or had no access to computers at school. in the netherlands, Australia, norway, the Slovak republic, Sweden, Serbia, Shanghai-China, Chinese taipei, macao-China, Spain and belgium, students who use computers at school outperform those who do not, even after accounting for socio-demographic disparities across the two groups. in israel, uruguay, Singapore, Portugal, denmark and estonia, the opposite is true: students who do not use computers at school perform better in problem solving than students who do, after accounting for differences in socio-economic status, gender and immigrant background. in the remaining countries, there is no significant performance difference between these two groups of students (figure v.4.15). • figure v.4.15 • difference in problem-solving performance related to the use of computers at school Difference in problem-solving performance between students who use a desktop, laptop or tablet computer at school and those who don’t Difference in problem-solving performance between students who use a desktop, laptop or tablet computer at school and those who don’t, after accounting for socio-demographic characteristics of students Score-point difference 40 30 20 10 0 -10 -20 Israel 55.2 Uruguay 49.7 Portugal 69.4 Singapore 69.7 Estonia 61.3 Denmark 86.9 Croatia 78.5 Finland 89.4 Germany 68.2 Italy 66.5 Czech Republic 84.0 Japan 59.7 Chile 61.3 Hungary 75.4 Korea 42.7 Ireland 63.4 Poland 61.0 Austria 81.6 Russian Federation 80.4 Turkey 49.2 OECD average 71.7 Slovenia 57.1 Hong Kong-China 83.5 Spain 75.3 Belgium 65.3 Macao-China 87.9 Chinese Taipei 78.8 Serbia 82.4 Shanghai-China 38.7 Sweden 87.8 Norway 91.9 Slovak Republic 80.0 Australia 93.7 Netherlands 93.9 -30 Percentage of students who use a desktop, laptop or tablet computer at school Notes: Statistically signiicant differences are marked in a darker tone (see Annex A3). Only countries/economies that participated in the questionnaire on ICT familiarity and in the assessment of problem solving are shown in this igure. Countries are ranked in descending order of the score-point difference in problem-solving performance between students who use a desktop, laptop or tablet computer at school and those who don’t, after accounting for socio-demographic characteristics of students. Source: OECD, PISA 2012 Database, Table V.4.26. 1 2 http://dx.doi.org/10.1787/888933003611 in sum, using a computer at home is strongly related to problem-solving performance in 29 of 33 participating countries and economies; but in most countries only a small minority of students do not use a computer at home. in contrast, the relationship between using a computer at school and problem-solving performance varies across countries. it is positive in 11 countries and economies, negative in six countries, and makes no difference in 16 (figures v.4.14 and v.4.15). While it makes intuitive sense to link performance on a computer-based assessment with an indicator of computer familiarity, such as the use of computers at home, PiSA data show that differences in performance on computer-based assessments are not larger than differences in performance on paper-based assessments, across students of varying familiarity with computers (figure v.4.16). if students who do not use computers at home perform poorly, then, it is not because these students are at an unfair disadvantage; rather, the fact that these students lack familiarity with computers is indicative of a wider disadvantage in education that manifests itself on paper-and-pencil tests as well as on computerbased assessments. CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 113 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS • figure v.4.16 • difference in problem-solving, mathematics, reading and science performance related to computer use at home Score difference between students who use computers at home and students who don’t, after accounting for socio-demographic characteristics, expressed as a percentage of the overall variation in performance Problem solving mathematics reading Science Score difference as a percentage of the standard deviation 120 100 80 60 40 20 Serbia Netherlands Belgium Czech Republic Croatia Austria Norway Slovak Republic Sweden Australia Denmark Macao-China Hong Kong-China Poland OECD average Israel Hungary Estonia Spain Korea Italy Portugal Germany Shanghai-China Japan Turkey Finland Chinese Taipei Chile Singapore Slovenia Russian Federation Ireland Uruguay 0 Notes: Statistically signiicant differences are marked in a darker tone (see Annex A3). Only countries/economies that participated in the questionnaire on ICT familiarity and in the assessment of problem solving are shown in this igure. Countries and economies are ranked in ascending order of the difference in problem-solving performance associated with the use of computers at home, after accounting for socio-demographic characteristics of students. Source: OECD, PISA 2012 Database, Table V.4.27. 1 2 http://dx.doi.org/10.1787/888933003611 114 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 4 how Problem-SolvIng PerFormAnce vArIeS wIThIn counTrIeS Notes 1. based on pair-wise comparisons of national patterns to oeCd average patterns. note that p-values have not been adjusted for the joint testing of multiple hypotheses. 2. Specifically, the fact that problem-solving proficiency shares about 2/3 of its overall variation with mathematics, reading or science implies that one can expect, by virtue of this common variation alone, the socio-economic effect size in problem solving to be at least 82% as large as the socio-economic effect size in mathematics, reading or science (√2/3 = 0.82). References Fennema, E. (2000), Gender and Mathematics: What is Known and What Do I Wish Was Known?, paper presented at the fifth Annual forum of the national institute for Science education, 22-23 may, 2000, detroit michigan, http://www.wcer.wisc.edu/archive/nise/ news_Activities/Forums/Fennemapaper.htm. Halpern, D.F. and M.L LaMay (2000), “the Smarter Sex: A Critical review of Sex differences in intelligence”, Educational Psychology Review, vol. 12, no. 2, pp. 229-246. Hyde, J.S. (2005), “the gender Similarities Hypothesis”, American Psychologist, vol. 60, no. 6, pp. 581-592. http://dx.doi.org/10.1037/0003-066X.60.6.581 Mayer, R.E. and M.C. Wittrock (2006), “Problem Solving” in P.A. Alexander and P.H. Winne (eds.), Handbook of Educational Psychology, 2nd edition, lawrence erlbaum Associates, mahwah, new Jersey, Chapter 13. OECD (2005), Problem Solving for Tomorrow’s World: First Measures of Cross-Curricular Competencies from PISA 2003, PiSA, oeCd Publishing. http://dx.doi.org/10.1787/9789264006430-en OECD (2013a), OECD Skills Outlook 2013: First Results from the Survey of Adult Skills, oeCd Publishing. http://dx.doi.org/10.1787/9789264204256-en OECD (2013b), PISA 2012 Results: Ready to Learn: Students’ Engagement, Drive and Self-Beliefs (Volume III), PiSA, oeCd Publishing. http://dx.doi.org/10.1787/9789264201170-en Wüstenberg, S. et al. (2014), “Cross-national gender differences in complex problem solving and their determinants”, Learning and Individual Differences, vol. 29, pp. 18-29. CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 115 5 Implications of the Problem-Solving Assessment for Policy and Practice In order to succeed in life, students must be able to apply the problemsolving strategies that they learn at school beyond the curricular contexts in which they are usually cast. This chapter discusses the implications of the PISA problem-solving assessment for education policy and practice. CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 117 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce in a rapidly changing world, individuals are constantly faced with novel situations and unexpected problems that they had never encountered at school, and for which they cannot find specific guidance in prior experience. the ability to handle such situations and solve these problems as they arise is associated with greater opportunities for employment and with the ability to participate fully in society. recent evidence from the Survey of Adult Skills (PiAAC) shows that adults who reach the highest level of proficiency in problem solving have access to those occupations where most new jobs were created over the past 15 years (figure v.5.1).1 What’s more, this trend is related to shifts in the demand for skills that have been observed, over a longer period of time, across the most advanced economies (box v.1.1). this implies that today’s 15-year-olds who lack advanced problemsolving skills face high risks of economic disadvantage as adults. they must compete for jobs in occupations where opportunities are becoming rare; and if they are unable to adapt to new circumstances and learn in unfamiliar contexts, they may find it particularly difficult to move to better jobs as economic and technological conditions evolve. • figure v.5.1 • employment growth across occupations, grouped by workers’ level of problem-solving skills Percentage-point change in the share of employment relative to 1998 5 4 Occupations with high proportions of strong performers 3 2 1 Occupations with medium to high proportions of strong performers 0 Occupations with low proportions of strong performers -1 -2 -3 Occupations with medium to low proportions of strong performers -4 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Notes: results from the Survey of Adult Skills (PIAAC) are used to identify occupations associated with high levels of proiciency in problem solving (proiciency level 2 or 3 on the PIAAC scale), and then time-series data available from the labour force Survey (lfS) database are used to track changes in those occupations over time. Only the 24 OECD countries available in the 1998 lfS database are included in the analysis. Occupations with high proportions (more than 45%) of workers who are strong performers in problem solving include managers and professionals. Occupations with medium to high proportions (40-45%) of strong performers include technicians and associate professionals (excluding health associate professionals) as well as ofice clerks. Occupations with medium to low proportions (25-40%) of strong performers include health associate professionals, such as nurses, customer services clerks, sales workers, as well as craft and related trades workers (excluding building workers). Occupations with low proportions (less than 25%) of strong performers include building workers, plant and machine operators and assemblers, and elementary occupations. Source: Eurostat, lfS database; Survey of Adults Skills (PIAAC) (2012). 1 2 http://dx.doi.org/10.1787/888933003630 ImProve ASSeSSmenTS To mAKe leArnIng more relevAnT While it is notoriously difficult to teach and to assess skills that are not easily codified in a set of rules or procedures (box v.5.1), the importance of problem-solving skills in the 21st century is now widely recognised. in many regions of the world, such as Alberta (Canada) (box v.5.2), employers and parents ask schools and teachers to develop these skills in young people, in order to equip them for success in life. the PiSA 2012 assessment of problem-solving skills represents a major advance towards making learning more relevant. it helps to identify how students can learn better, teachers can teach better, and schools can operate more effectively in the 21st century. built on a deep understanding of what constitutes individual problem-solving competence, it provides educators around the world, as well as parents, employers and policy makers, with first-of-its-kind evidence on how well prepared today’s 15-year-olds are to solve complex, unfamiliar problems that they may encounter outside of curricular contexts. 118 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce box v.5.1. when solutions are taught, problem solving is not learned every teacher knows that rules and procedures to solve routine problems are relatively easy both to teach and to test. but skills that can be codiied in rules can also be performed by a computer. by their nature, the skills needed to solve complex, non-routine problems cannot be reduced to rules, and so they are relatively dificult to both teach and assess. While everyone agrees that children need problem-solving skills, in practice, these skills have largely been taught by focusing only on rules-based solutions, like the rules of algebra. The rules of algebra are important, but applying algebraic rules is just the second step of a two-step problem-solving process. The irst step – the step computers can’t do – involves examining the messy set of facts in a real-world problem to determine which set of algebraic rules to apply. for example, the labour market today values a mechanical engineer’s ability to formulate a problem as a particular mathematical model. Once the model is formulated, a computer – not the engineer – will apply rules to calculate the actual solution. How do engineers choose the correct mathematical model? They likely rely on analogies with problems they have solved in the past. It follows that to develop the expertise and lexibility required by non-routine problems, education in any subject, trade or occupation must include exposure to numerous real-world problems on which to draw. Source: levy (2010). box v.5.2. developing a curriculum for the 21st century in Alberta (canada) Canada is a relative latecomer to the top of the international education rankings. unlike Japan or Singapore, Canada found itself among the best-performing countries only after the release of the PiSA rankings in 2000. Since then, Canada has consistently performed above the oeCd average in PiSA, although performance declined in 2012 relative to the previous assessments. At the regional level, when compared to the other nine Canadian provinces, Alberta, along with british Columbia, stands outs for its strong performance. in PiSA 2012, Alberta students scored 517 points, on average, in mathematics and 539 points in science. With 531 points in problem solving, their performance is in line with Canada’s average performance. Canadian education is governed at a provincial level; thus education systems in each of the ten provinces and three territories have their own history, governance structure, and education strategy. the government of Alberta recently decided to develop a new vision for the future of teaching and learning, one that will inspire the curriculum for the 21st century. through a series of province-wide consultations starting in 2009, the government developed a curriculum redesign project (Alberta education, 2010). While Albertans expressed pride in their schools and universities, they also voiced the need for a transformation of the education system in order to help students engage in a rapidly changing knowledge-based society. these participatory dialogues inspired and informed the project, an ongoing initiative that involves revising the curriculum with the aim of developing engaged thinkers and ethical citizens with an entrepreneurial spirit. in this context, a framework for student learning was developed that identiies critical thinking, problem solving and decision making as key cross-curriculum competencies (Alberta Education, 2013a, 2013b). This involves, for example, developing the conidence and skills in students to solve different types of problems, including novel and ill-deined tasks and tasks related to their learning, work and personal lives; stimulating the use of multiple approaches to solving problems; and modelling students’ ability to transfer knowledge and experience gained in the past to solve problems and make decisions in the future. Proposals for further collaborative curriculum development are under review and the new curriculum is expected to be launched by 2016. ... CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 119 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce the open consultation leading to the formulation of the 21st Century Skills Curriculum in Alberta proves that problem-solving skills are valued by the economy and society at large. it also shows how curriculum reforms can provide opportunities to involve stakeholders – including parents, employers, and students themselves – in education, so that learning becomes a common goal and a shared responsibility. Sources: Alberta education (2010); Alberta education (2013a); Alberta education (2013b). the assessment of problem-solving skills in PiSA 2012 recognises that, in order to succeed in life, students must be able to apply the problem-solving strategies that they learn at school beyond the curricular contexts in which they are usually cast. While most problem-solving activities in schools are compartmentalised by subject, such as problem solving in mathematics or in science, success in the PiSA problem-solving assessment hinges on skills that are useful in a broad spectrum of contexts, in and out of school. Students who perform well in problem solving are able to examine the problem situation to collect useful information; build a coherent mental representation of the relevant parts involved and of the relationships between them, and communicate this representation; plan a strategy for overcoming the obstacles to resolving the problem and execute the plan while monitoring its progress; and critically review each step and reflect on possible alternatives and missing pieces. emPower STudenTS To Solve ProblemS the analysis of results from the problem-solving assessment shows that, on average across oeCd countries, about one in five students is only able to solve very straightforward problems – if any – provided they refer to familiar situations, such as choosing from a catalogue of furniture, showing different brands and prices, the cheapest models to furnish a room (level 1 tasks). in six partner countries, fewer than half the students are able to perform beyond this baseline level of problem-solving proficiency. in contrast, in korea, Japan, macao-China and Singapore, more than nine out of ten students can complete tasks at level 2 at least. these countries/economies are close to the goal of giving each student the basic tools needed to meet the challenges that arise in daily life. As in other assessment areas, there are wide differences between countries in the ability of 15-year-olds to fully engage with and solve non-routine problems in real-life contexts. over 160 score points separate the mean performance of the best- and lowest-performing countries – the equivalent of between two and three proficiency levels (on a scale going from “below level 1” to “level 6 and above”). in the best-performing countries – Singapore and korea – 15-year-old students, on average, are able to engage with moderately complex situations in a systematic way. for example, they can troubleshoot an unfamiliar device that is malfunctioning: they grasp the links among the elements of the problem situation, they can plan a few steps ahead and adjust their plans in light of feedback, and they can form a hypothesis about why a device is malfunctioning and describe how to test it (level 4 tasks). by contrast, in the lowest-performing countries, students, on average, are only able to solve very simple problems that do not require to think ahead and that are cast in familiar settings, such as determining which solution, among a limited set of alternatives, best meets a single constraint by using a “trial-and-error” strategy (level 1 tasks). mean performance differences between countries, however, represent only a fraction of overall variation in student performance. Within countries, about 245 score points (or four proficiency levels), on average, separate the highest-performing 10% of students from the lowest-performing 10% of students. thus, even within the best-performing countries, significant numbers of 15-year-olds do not possess the basic problem-solving skills considered necessary to succeed in today’s world, such as the ability to think just one step ahead or to engage with unfamiliar problem situations. but how can teachers and schools foster students’ competence in solving problems across domains? research shows that training problem-solving skills out of context is not the solution (box v.5.3). one promising approach is to encourage teachers and students to reflect on solution strategies when dealing with subject-specific problems in the classroom. this metacognitive reflection might support students’ own reflection, and expand their repertoire of generic principles applicable to different contexts (box v.5.4). in addition, such strategies can be applied within all areas of instruction – from reading and mathematics to biology, history, and the visual arts (box v.5.5). Students who recognise, for instance, a systematic exploration strategy when it occurs in history or science class may use it with more ease when confronted with unfamiliar problems. When teachers ask students to describe the steps they took to solve a problem, they encourage students’ metacognition, which, in turn, improves general problem-solving skills. 120 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce box v.5.3. Problem-solving skills are best developed within meaningful contexts decades of intense research have shown that direct training approaches for domain-general competencies (e.g. intelligence, working memory capacity, or brain eficiency) do not lead to greater capacity to solve problems independently of their domain. Domain-general competencies, such as intelligence, are extremely dificult and costly to train. They can be increased only within narrow limits, and the increases are usually not stable over time. Even more important, domain-general competencies do not help to solve a problem when a person lacks knowledge about the problem at hand and its solution. The highest intelligence, largest working memory capacity, or the most eficient brain cannot help to solve a problem if the person has no meaningful knowledge to process. A more effective alternative for broadening competencies is to teach concrete content knowledge in ways that aid subsequent transfer to new situations, problem types and content. This lexible kind of expertise, however, does not develop on its own. One important precondition for transfer is that students must focus on the common, deep structure underlying two problem situations rather than on their supericial differences. Only then will they apply the knowledge acquired in one situation to solve a problem in another. This can be accomplished by pointing out to students that two problem solutions require similar actions; by using diagrams to visualise the deep structures of different problems; by fostering comparisons between examples that highlight their structural similarities or differences; and by the use of analogies between phenomena arising in different domains. People are less likely to transfer isolated pieces of knowledge than they are to transfer parts of well-integrated hierarchical knowledge structures. The more connections a learner sees between the learning environment and the outside world, the easier the transfer will be. Source: Schneider and Stern (2010). box v.5.4. what is metacognitive instruction? An important component of the problem-solving skill of students is the ability to monitor and regulate their own thinking and learning. metacognition – thinking about and regulating thinking – is the “engine” that starts, regulates and evaluates the cognitive processes. the learning environments with the greatest potential to enhance these processes are those centred on metacognitive teaching methods. various models have been developed to help students regulate their behaviour during learning, in all kinds of disciplines. in general, metacognitive instruction relies on teachers’ ability to help students become aware and consciously relect on their own thought. it is characterised by frequent questioning by teachers or selfquestioning by students themselves (“Have i solved problems like this before? Am i on the right track? What information do i need?”). this questioning may take place in classroom dialogue and “thinking aloud” sequences that make the reasoning explicit and model the solution strategies of other students. metacognitive instruction can be successfully embedded in co-operative learning settings, where students work in small groups with assigned roles. the problems or inquiries that students work on must have room enough to allow students not only to learn routine procedures that are useful for their solution, but also to practice the questioning and dialogue and to experience some struggle before the goal is reached. in metacognitive instruction, students often work on challenging tasks that require them to think for an extended time. Such tasks also offer many opportunities for teachers to help students learn from their mistakes. by focusing attention on learning as a process, metacognitive instruction further conveys the message that success comes from hard work; it therefore positively inluences dispositions towards learning across the ability spectrum and reduces anxiety. ... CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 121 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce Studies have shown that metacognitive pedagogies can be effective across kindergartens, primary and secondary schools, and in higher education. in mathematics, students exposed to metacognitive pedagogies outperformed their counterparts in the control groups on routine textbook problems as well as on complex, unfamiliar and nonroutine mathematics tasks. Source: mevarech and kramarski (forthcoming). box v.5.5. Teaching problem-solving skills through the visual arts if you ask someone what students learn in visual arts classes, you are likely to hear that they learn how to paint, or draw, or throw a pot. of course students learn arts techniques in arts classes. but what else do they learn? Are there any kinds of general thinking dispositions that are instilled as students study arts techniques? An ethnographic study, based on video observations and interviews conducted in two prestigious art schools in the boston area (Hetland et al., 2013), identiied several habits of mind and working styles – all of which are applicable in contexts beyond the visual arts – taught in arts classes at the same time as students were learning the craft of painting and drawing. for example, through frequent dialogue with their teachers, all of whom are practicing artists, these highly motivated students are taught to envision what they cannot observe directly with their eyes, to observe carefully, to relect on their work process and product, to engage and persist in their efforts, and to stretch and explore creative possibilities: • Envision: Students in the visual arts classes observed in this study are constantly asked to envision what they cannot observe directly with their eyes – e.g. to detect the underlying structure of a form they were drawing and then envision how that structure could be shown in their work. • Observe: The skill of careful observation is taught all the time in visual arts classes and is not restricted to drawing classes where students draw from a model. Students are taught to look more closely than they ordinarily do and to see with “new” eyes. • reflect: Students are asked to become reflective about their art making. Teachers frequently ask open-ended questions that prompt students to reflect and explain, whether aloud or even silently to themselves. Students are thus stimulated to develop metacognitive awareness about their work and working process. Students are also asked to talk about what works and what does not work in their own pieces and in those by their peers. Thus students are trained to make critical judgements and to justify these judgements. • Engage and persist. Teachers in visual arts classes present their students with projects that engage them, and they teach their students to stick to a task for a sustained period of time. Thus they are teaching their students to focus and develop inner-directedness. As one of the teachers said, she teaches them to learn “how to work through frustration.” • Stretch and explore. Students are asked to try new things and thereby to extend beyond what they have done before – to explore and take risks. As one painting teacher said, “You ask kids to play, and then in one-on-one conversation you name what they’ve stumbled on.” Source: Hetland et al. (2013); Winner et al. (2013). revISe School PrAcTIceS And educATIon PolIcIeS Within all countries and economies, problem-solving results vary greatly between schools: differences in problemsolving performance between schools are as large as differences in mathematics performance, indicating that schools have an important role to play in building these skills. Several high-performing countries, such as Singapore, have recognised the importance of schools in developing problem-solving skills and have prioritised problem-solving skills throughout the curriculum (box v.5.6). 122 © OECD 2014 CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce Box V.5.6. developing and assessing problem-solving skills in Singapore Singapore ranks at the top in problem-solving performance, with students scoring on average 562 points on the PISA scale. The strong performance of Singapore students in problem solving may be related to several aspects of teaching and learning in Singapore. In addition to the country’s emphasis on providing a strong grounding in literacy and numeracy, a sharper focus on developing thinking skills in schools was launched in 1997 with the project “Thinking Schools, Learning Nation” (MOE, 1997). A fundamental review of the curriculum and assessment system was subsequently undertaken, and related revisions to subject syllabi were introduced (MOE, 2014a). National examinations were revised in tandem, giving greater importance to assessing higher-order thinking and problem-solving skills (SEAB, 2014a). In 2009, Singapore undertook another review that identified the 21st century competencies considered important: critical and inventive thinking; communication, collaboration and information skills; and civic literacy, global awareness and cross-cultural skills. The 21st century competencies framework (MOE, 2014b) now guides the development of the national curriculum as well as school-based programmes to nurture these competencies. Closely linked to the development of 21st century competencies is a wider effort across schools to harness information and communication technology (ICT) for teaching and learning. Provisions from three waves of the ICT Masterplan since 1997 have enabled teachers to use ICT tools that help students learn and work independently and collaboratively (MOE, 2011a; MOE, 2011b). At the subject level, the curriculum is reviewed in regular cycles to ensure alignment with developments in the discipline and national educational goals. The mathematics curriculum, for example, has an explicit focus on problem solving and details the teaching, learning and assessment of problem-solving skills. Students are guided to apply mathematical models and thinking to real-world contexts (MOE, 2014c). The science curriculum places scientific inquiry at the heart of teaching and learning science. Students are provided with opportunities to engage with a scientific phenomenon or problem, collect and interpret the evidence, reason, conduct investigations and make inferences or decisions (MOE, 2014d). Social studies reinforce the inquiry mindset, requiring students to examine evidence to support points of view (SEAB, 2014b). Collectively, these approaches help students become more adept at inquiring, culling relevant information to create new knowledge, experimenting with alternatives, and working with uncertainty when dealing with unfamiliar problems. Teachers are key to ensuring implementation, and there is strong support for teachers’ professional learning throughout their careers. The Academy of Singapore Teachers and the specialised teacher academies lead in developing teacher capacity across all schools. Professional learning activities include mentoring beginning teachers, in-service teacher training, and the establishment of teacher-learning communities to promote teacher collaboration (MOE, 2012). In addition, the Ministry’s curriculum officers and subject specialists work closely with Master Teachers in the academies to support teachers in developing classroom resources and teaching strategies. Sources: Ministry of Education, Academy of Singapore Teachers (2012); Ministry of Education, Educational Technology Division (2011a); Ministry of Education, Educational Technology Division (2011b); MOE (2014a); MOE (2014b); MOE (2014c); MOE (2014d); MOE (1997), Singapore Examinations and Assessment Board (2014a); Singapore Examinations and Assessment Board (2014b). The association between performance in problem solving and performance in the core PISA domains of mathematics, reading and science is strong and positive at the individual, the school and the country levels. In general among students, high performers in mathematics, reading or science also show the highest levels of problem-solving competence when confronted with unfamiliar problems in non-curricular contexts. They can develop coherent mental representations of the problem situation, plan ahead in a focused way, and show flexibility in incorporating feedback and in reflecting on the problem and its solution. Similarly, at the system level, the countries in which students are most prepared to use their mathematics, reading and science skills in real-life contexts are also those where students are most at ease with the cognitive processes that are required to solve everyday problems, such as interacting with unfamiliar technological devices. CreAtive Problem Solving: StudentS’ SkillS in tACkling reAl-life ProblemS – volume v © OECD 2014 123 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce Nevertheless, the strength of association between problem-solving skills and domain-specific skills that are explicitly taught in school subjects is weaker than the association between, say, mathematics and reading skills. And while better results in problem solving are associated with better results in mathematics, reading and science, the pattern is not without exceptions. Performance in problem solving, among both students and school systems, is not identical to that in other assessed subjects. In nine countries and economies (Australia, Brazil, Italy, Japan, korea, Macao-China, Serbia, England [united kingdom] and the united States), students perform significantly better in problem solving than students in other countries/economies who show similar performance in mathematics, reading and science. Countries where students perform worse in problem-solving than students with similar proficiency in curricular domains in other countries may look more closely at the features of the curricula and instructional styles in the more successful countries to determine how to equip students better for tackling complex, real-life problems in contexts that they do not usually encounter at school. A closer analysis reveals interesting differences within this set of nine countries. In some, such as the united States, England (united kingdom) and Australia, the good performance in problem solving at the system level stems mainly from the students with the strongest performance in mathematics. This alignment suggests that, in these countries, high performers in mathematics have greater access to the kinds of learning opportunities that build problem-solving skills. In others, such as Japan, korea and Italy, the good performance in problem solving at the system level can be attributed to the resilience of many low achievers in mathematics. These countries, more than others, seem to offer students who struggle to master the basic curriculum second chances to develop the problem-solving skills that are required to fully participate in today’s societies (Box V.5.7). Box V.5.7. developing and assessing problem-solving skills in Japan: cross-curricular project-based learning Japan ranks at or near the top in all subjects assessed in PISA 2012, and performance in problem solving is no exception. What’s more, Japanese students, who score 552 points, on average, show better performance in problem solving than students with similar performance in mathematics, reading and science in other countries and economies, particularly among moderate and low performers in core subjects. On the problem-solving scale, at least 20 points separate Japanese students who perform below Level 4 in mathematics, reading or science from similarly proficient students in other countries (Table V.2.6). One plausible explanation for this is Japan’s focus on developing every student’s problem-solving skills through his or her participation in cross-curricular, student-led projects, both within the subjects and through integrated learning activities. In the late 1990’s, the “zest for living” approach was introduced by the Japanese government through a reform to the Course of Study, Japan’s national curriculum standards. The aim of the approach was to strengthen students’ ability to think critically and creatively, and to identify and solve problems independently. This reform prompted substantial changes towards an inquiry-based, student-centred model of learning. The need for improving students’ engagement and motivation was at the heart of these transformations. The new approach led to a revision of subject-matter curricula. The new curricula reduced the content load by about 30%. for example, the number of English words that students had to memorise in junior high school was reduced from 1 000 to 900. The intention was to create space, within each subject, for deepening learning through classroom activities that cultivate introspection, the desire to learn and think, independent decision-making, and problem-solving skills. In 2007, new national assessments that focus on the ability of students to apply their knowledge in real-world contexts were introduced in sixth and ninth grades. The reform also allocated more time for elective offerings and introduced a new class period in all schools, called “Integrated Learning”. In these classes, students engage in cross-curricular projects related to international understanding, social welfare and health, or environmental issues, that provide opportunities to practice observation and experimentation and to discover multiple solutions to problems and draw connections to their own lives (MEXT, 2002; Aranil and fukaya, 2010). The homeroom teacher is responsible for this class period, and topics are often decided in collaboration with other teachers in the same school. The Ministry of Education, as well as local school boards, produce guidelines and scripted examples for the integrated study lesson, often in collaboration with other agencies and with private-sector employers (see www.mext.go.jp/a_menu/shotou/sougou/syokatsu.htm). ... 124 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce Students’ work is recorded in portfolios and qualitative feedback is provided to students and families, but the work is not formally assessed. The implementation of this reform sparked some controversy. In practice, the guidelines for teaching the “Integrated Learning” course gave a great deal of freedom to schools and teachers for deciding how to implement the programme, but not all teachers, particularly at the secondary level, felt that they were adequately prepared to do so. This resulted in changes to the curriculum standards, implemented in 2011 and 2012, involving a reduction of the time allocated to “Integrated Learning” in favour of teaching academic subjects (OECD, 2012). Nonetheless, the “zest for living” approach is still promoted throughout the curriculum and the national standards continue to recommend that schools increase the amount of learning activities, in all subjects, that involve the application of knowledge through observation and experimentation. Japan’s constant effort to improve the curriculum and instruction to promote more relevant learning has resulted not only in good results on the PISA test, but also in remarkable improvements, between 2003 and 2012, in students’ sense of belonging at school and in their dispositions towards learning (see Volume III, Ready to Learn: Students’ Engagement, Drive and Self-Beliefs) (OECD, 2013a). Sources: Aranil and fukaya (2010); MEXT (2002); OECD (2013a); OECD (2012). It seems that problem solving is a distinct skill with similar attributes as proficiency in specific school subjects. While influenced by differences in individuals’ cognitive abilities, its development depends on the opportunities offered by good teaching. Ensuring opportunities to develop problem-solving skills for all students and in all subjects, including those not assessed in PISA, in turn, depends on school- and system-level policies. leArn From currIculAr dIverSITy And PerFormAnce dIFFerenceS In Problem SolvIng Improving the curriculum and instruction to promote learning for life is a huge challenge. It is, to some extent, reassuring to know that students with good results in mathematics, reading and science also have, by and large, good results in problem solving. At the very least, this is consistent with the idea that better instruction in the core subjects corresponds to a greater capacity of students to meet the challenges they will encounter in life beyond school. further indications about how to improve the curriculum and instruction may come from the strengths and weaknesses in problem solving that are observed within and across countries. The analysis in Chapter 3, for instance, identifies interesting differences in performance across different types of problem-solving tasks. These differences are likely a reflection of how well students learn, through the content of the various school subjects and the way in which it is taught, to handle unexpected obstacles and deal with novelty. In some countries and economies, such as finland, Shanghai-China and Sweden, students master the skills needed to solve static, analytical problems similar to those that textbooks and exam sheets typically contain as well or better than 15-year-olds, on average, across OECD countries. But the same 15-year-olds are less successful when not all information that is needed to solve the problem is disclosed, and the information provided must be completed by interacting with the problem situation. A specific difficulty with items that require students to be open to novelty, tolerate doubt and uncertainty, and dare to use intuitions (“hunches and feelings”) to initiate a solution suggests that opportunities to develop and exercise these traits, which are related to curiosity, perseverance and creativity, need to be prioritised. In yet other countries and economies, such as Portugal and Slovenia, students are better at using their knowledge to plan and execute a solution than they are at acquiring such useful knowledge themselves, questioning their own knowledge, and generating and experimenting with alternatives. While these students appear to be goal-driven, motivated and persistent, their relatively weak performance on problems that require abstract information processing suggests that opportunities to develop the reasoning skills and habits of self-directed learners and effective problem-solvers need to be prioritised. The analysis in Chapter 4 also identifies, within many countries and economies, certain study programmes whose students perform significantly better in problem solving, on average, than students in the same country/economy with CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 125 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce similar proficiency in mathematics, reading and science. In Shanghai-China and Turkey, for instance, students in certain vocational study programmes have significantly better performance in problem solving than students with comparable performance in mathematics, reading and science in the remaining study programmes. By contrast, in germany, it is students in the education tracks with the strongest emphasis on academic learning (Gymnasium) who score higher than expected in problem solving, given their performance in core subjects. This may be because the instructional practices in the sciences and the arts in these programmes equip students for tackling complex, real-life problems in contexts that they do not usually encounter at school. If this is the case, students in these programmes not only learn the curriculum, they also learn how to enrich their knowledge and use that knowledge outside of school contexts. Alternatively, betterthan-expected performance in problem solving may have a less positive interpretation, particularly if it coincides with low performance overall: it may indicate that in these programmes, students’ cognitive potential is not realised within the core academic subjects. Whether it signals strong performance in problem solving or weak performance in the core subjects, the variation across programmes in their relative performance may have profound implications for policy, and invites further investigation. reducing this variation could involve revising the curriculum and instructional practices within each programme by borrowing the best elements of other programmes, while preserving the diversity in curricula needed to make the most of each student’s talents. Even within school systems that encourage diversity of curricula, the acquisition of critical reasoning and problem-solving skills can be promoted as a common aim, as these skills are applicable – and essential – in all pursuits. reduce gender dISPArITIeS Among ToP PerFormerS gender differences in school performance tend to vary across school subjects. In most countries and economies, boys perform better than girls in mathematics, while girls perform better than boys in reading. These gender differences, however, vary substantially across countries. This suggests that the observed differences are not inherent, but are largely the result of the opportunities provided by parents, schools and society in general for boys and girls to cultivate their individual talents. gender stereotypes about what boys and girls are good at, and what kind of occupations are suitable for them reinforce and crystallise performance differences between boys and girls, even if they initially reflect only the random variation among students. Because problem-solving skills are required in all kinds of occupations, and are not taught as such in school, but rather are nurtured by good instructional practices in every subject, performance in problem solving should not be strongly influenced by such gender-based stereotypes. Problem-solving performance could then be regarded as an overall indicator of gender biases in a country’s education system. The good news is that in most countries/economies, there are no large differences in boys’ and girls’ average performance in problem solving. However, countries that do show significant gender differences in problem-solving performance, such as the united Arab Emirates (where girls outperform boys), Colombia and Japan (where boys outperform girls), may not be offering boys and girls equitable opportunities in education, particularly if these differences are also apparent in other subjects. unless countries invest as much in the development of girls’ skills as they do in boys’ skills, they may lose out in the global competition for talent. While boys and girls do not differ markedly in their average performance, the variation in problem-solving performance is larger among boys than among girls. At lower levels of proficiency, there are, in general, equal proportions of boys and girls. But the highest-performing students in problem solving are largely boys – with a few notable exceptions, such as Australia, finland and Norway, where the proportion of top-performing girls is about the same as the proportion of top-performing boys. Similarly, among adults, top-performers in problem solving are mostly men (OECD, 2013b).2 Increasing the number of girls at the highest performance levels in problem solving, and improving their ability to handle complex, unfamiliar problems, may help more women attain leadership positions in the future. reduce IneQuITIeS In educATIon relATed To SocIo-economIc STATuS While large and significant, the impact of socio-economic disadvantage on problem-solving skills is weaker than it is on performance in mathematics, reading or science. At all levels of the socio-economic ladder, there is more variation in performance in problem solving than there is in mathematics, perhaps because after-school opportunities to develop problem-solving skills are more evenly distributed than opportunities to develop proficiency in mathematics or reading. 126 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce Still, unequal access to high-quality education means that the risk of not reaching the baseline level of performance in problem solving is about twice as large for disadvantaged students as it is for their more advantaged peers, on average. The fact that inequities in education opportunities extend beyond the boundaries of individual school subjects to performance in problem solving underscores the importance of promoting equal learning opportunities for all. Because current inequities have such significant consequences over the long term, the policies that aim to reduce socio-economic disparities in education can be expected to benefit the lives of students well beyond their school days. Notes 1. The Survey of Adult Skills (PIAAC) is based on a different assessment framework. PIAAC defines “problem solving in technologyrich environments” as the ability to use digital technology, communication tools and networks to acquire and evaluate information, communicate with others and perform practical tasks. The PIAAC assessment focuses on the abilities to solve problems for personal, work and civic purposes by setting up appropriate goals and plans, and accessing and making use of information through computers and computer networks (PIAAC Expert group in Problem Solving in Technology-rich Environments, 2009; OECD, 2013b). 2. The Survey of Adult Skills (PIAAC) similarly finds that there are about three men for every two women performing at the highest level of proficiency (Level 3) in “problem solving in technology-rich environments”. On average across countries, 6.9% of men perform at this level, but only 4.7% of all women aged 16-65 do. More equal shares of men and women performing at the top are found in Australia, Canada and finland (Table A3.5 in OECD, 2013b). References Alberta Education (2013a), Ministerial Order on Student Learning (#001/2013), http://education.alberta.ca/department/policy/ standards/goals.aspx. Alberta Education (2013b), Curriculum redesign, http://education.alberta.ca/department/ipr/curriculum.aspx. Alberta Education (2010), Inspiring Education: A Dialogue with Albertans, retrieved from http://www.inspiringeducation.alberta.ca/ LinkClick.aspx?ileticket=BjGiTVRiuD8%3d&tabid=37. Aranil, M. and Fukaya, K. (2010), “Japanese National Curriculum Standards reform: Integrated Study and Its Challenges”, in Joseph I. Zajda (ed.), Globalisation, Ideology and Education Policy Reforms, Globalisation, Comparative Education and Policy Research, Volume 11, pp. 63-77. Hetland, L. et al. (2013), Studio thinking 2: The real beneits of visual arts education, 2nd edition (irst edition: 2007), teachers College Press, new York. Levy, F. (2010), “How technology Changes demands for Human Skills”, OECD Education Working Papers, no. 45, oeCd Publishing. http://dx.doi.org/10.1787/5kmhds6czqzq-en Mevarech Z. and B. Kramarski (forthcoming), Critical Maths for Innovation: The Role of Metacognitive Pedagogies, oeCd Publishing. MEXT (Ministry of Education, Culture, Sports, Science and Technology) (2002), Japanese government Policies in education, Culture, Sports, Science and technology 2001: educational reform for the 21st Century, ministry of education, Culture, Sports, Science and technology, Japan. CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 127 5 ImPlIcATIonS oF The Problem-SolvIng ASSeSSmenT For PolIcy And PrAcTIce Ministry of Education, Academy of Singapore Teachers (2012), Professional Networks, http://www.academyofsingaporeteachers. moe.gov.sg/professional-networks (accessed 5 february 2014). Ministry of Education, Educational Technology Division (2011a), The ICT Connection, http://ictconnection.moe.edu.sg/our-ictmasterplan-journey/our-ict-in-education-journey (accessed 5 february 2014). Ministry of Education, Educational Technology Division (2011b), The ICT Connection, http://ictconnection.moe.edu.sg/masterplan-3/ mp3-towards-21cc (accessed 5 february 2014). MOE (Ministry of Education), Singapore (2014a), MOE Subject Syllabuses, http://www.moe.gov.sg/education/syllabuses/ (accessed 5 february 2014). MOE (Ministry of Education), Singapore (2014b), Singapore (2014b), 21st Century Competencies, http://www.moe.gov.sg/education/21cc/ (accessed 17 March 2014). MOE (Ministry of Education), Singapore (2014c), O- & N(A)-Level Mathematics Teaching and Learning syllabus, http://www.moe.gov. sg/education/syllabuses/sciences/iles/ordinary-and-normal-academic-level-maths-2013.pdf (accessed 5 february 2014). MOE (Ministry of Education), Singapore (2014d), Primary Science Syllabus 2014, http://www.moe.gov.sg/education/syllabuses/sciences/ iles/science-primary-2014.pdf (accessed 5 february 2014). MOE (Ministry of Education), Singapore (1997), Shaping our Future: Thinking Schools, Learning Nation, speech by Prime Minister goh Chok Tong at the 7th International Conference on Thinking on 2 June 1997, http://www.moe.gov.sg/media/speeches/1997/020697.htm (accessed 5 february 2014). Singapore Examinations and Assessment Board (SEAB), Singapore (2014a), Singapore-Cambridge gCE O-Level examination syllabuses, http://www.seab.gov.sg/oLevel/syllabusSchool.html (accessed 5 february 2014). Singapore Examinations and Assessment Board (SEAB), Singapore (2014b), Singapore-Cambridge gCE O-Level Combined Humanities (Social Studies Elective) examination syllabus, http://www.seab.gov.sg/oLevel/2015Syllabus/2204_2015.pdf (accessed 5 february 2014). OECD (2013a), PISA 2012 Results: Ready to Learn: Students’ Engagement, Drive and Self-Beliefs (Volume III), PISA, OECD Publishing, http://dx.doi.org/10.1787/9789264201170-en. OECD (2013b), OECD Skills Outlook 2013: First Results from the Survey of Adult Skills, OECD Publishing. http://dx.doi.org/10.1787/9789264204256-en OECD (2012), Lessons from PISA for Japan, Strong Performers and Successful Reformers in Education, OECD Publishing. http://dx.doi.org/10.1787/9789264118539-en PIAAC Expert Group in Problem Solving in Technology-Rich Environments (2009), “PIAAC Problem Solving in Technology-rich Environments: A Conceptual framework”, OECD Education Working Papers, No. 36, OECD Publishing. http://dx.doi.org/10.1787/220262483674 Schneider M. and E. Stern (2010), The cognitive perspective on learning: Ten cornerstone indings, Chapter 3 in H. Dumont, D. Istance and f. Benavides, The Nature of Learning: Using Research to Inspire Practice, OECD Publishing. http://dx.doi.org/10.1787/9789264086487-en Singapore Examinations and Assessment Board (SEAB), Singapore (2014a), Singapore-Cambridge gCE O-Level examination syllabuses, http://www.seab.gov.sg/oLevel/syllabusSchool.html (accessed 5 february 2014). Singapore Examinations and Assessment Board (SEAB), Singapore (2014b), Singapore-Cambridge gCE O-Level Combined Humanities (Social Studies Elective) examination syllabus, http://www.seab.gov.sg/oLevel/2015Syllabus/2204_2015.pdf (accessed 5 february 2014). Winner, E., T. Goldstein and S. Vincent-Lancrin (2013), Art for Art’s Sake?: The Impact of Arts Education, Educational research and Innovation, OECD Publishing. http://dx.doi.org/10.1787/9789264180789-en 128 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V Annex A PiSa 2012 tEchnical backGround All figures and tables in Annex A are available on line annex a1: Indices from the student context questionnaires annex a2: The PISA target population, the PISA samples and the deinition of schools http://dx.doi.org/10.1787/888933003725 annex a3: Technical notes on analyses in this volume annex a4: Quality assurance annex a5: The problem-solving assessment design annex a6: Technical note on brazil http://dx.doi.org/10.1787/888933003744 notes regarding cyprus Note by Turkey: The information in this document with reference to “Cyprus” relates to the southern part of the Island. There is no single authority representing both Turkish and greek Cypriot people on the Island. Turkey recognises the Turkish republic of northern Cyprus (TrnC). until a lasting and equitable solution is found within the context of the united nations, Turkey shall preserve its position concerning the “Cyprus issue”. Note by all the European Union Member States of the OECD and the European Union: The republic of Cyprus is recognised by all members of the united nations with the exception of Turkey. The information in this document relates to the area under the effective control of the government of the republic of Cyprus. a note regarding israel The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is without prejudice to the status of the golan Heights, East Jerusalem and Israeli settlements in the West bank under the terms of international law. CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 129 Annex A1: IndIceS From The STudenT, School And PArenT conTexT QueSTIonnAIreS Annex A1 IndIceS From The STudenT conTexT QueSTIonnAIreS Explanation of the indices This section explains the indices derived from the student context questionnaires used in PISA 2012. Several PISA measures relect indices that summarise responses from students, their parents or school representatives (typically principals) to a series of related questions. The questions were selected from a larger pool of questions on the basis of theoretical considerations and previous research. The PISA 2012 Assessment and Analytical Framework (OECD, 2013a) provides an in-depth description of this conceptual framework. Structural equation modelling was used to conirm the theoretically expected behaviour of the indices and to validate their comparability across countries. for this purpose, a model was estimated separately for each country and collectively for all OECD countries. for a detailed description of other PISA indices and details on the methods, see the PISA 2012 Technical Report (OECD, forthcoming). There are two types of indices: simple indices and scale indices. Simple indices are the variables that are constructed through the arithmetic transformation or recoding of one or more items, in exactly the same way across assessments. Here, item responses are used to calculate meaningful variables, such as the recoding of the four-digit ISCO-08 codes into “Highest parents’ socio-economic index (HISEI)” or, teacher-student ratio based on information from the school questionnaire. Scale indices are the variables constructed through the scaling of multiple items. unless otherwise indicated, the index was scaled using a weighted likelihood estimate (WlE) (Warm, 1989), using a one-parameter item response model (a partial credit model was used in the case of items with more than two categories). for details on how each scale index was constructed see the PISA 2012 Technical Report (OECD, forthcoming). In general, the scaling was done in three stages: • The item parameters were estimated from equal-sized subsamples of students from all participating countries and economies. • The estimates were computed for all students and all schools by anchoring the item parameters obtained in the preceding step. • The indices were then standardised so that the mean of the index value for the OECD student population was zero and the standard deviation was one (countries being given equal weight in the standardisation process). Sequential codes were assigned to the different response categories of the questions in the sequence in which the latter appeared in the student, school or parent questionnaires. Where indicated in this section, these codes were inverted for the purpose of constructing indices or scales. negative values for an index do not necessarily imply that students responded negatively to the underlying questions. A negative value merely indicates that the respondents answered less positively than all respondents did on average across OECD countries. Likewise, a positive value on an index indicates that the respondents answered more favourably, or more positively, than respondents did, on average, across OECD countries. Terms enclosed in brackets < > in the following descriptions were replaced in the national versions of the student, school and parent questionnaires by the appropriate national equivalent. for example, the term <qualiication at ISCED level 5A> was translated in the united States into “bachelor’s degree, post-graduate certiicate program, master’s degree program or irst professional degree program”. Similarly the term <classes in the language of assessment> in luxembourg was translated into “german classes” or “french classes” depending on whether students received the german or french version of the assessment instruments. In addition to simple and scaled indices described in this annex, there are a number of variables from the questionnaires that correspond to single items not used to construct indices. These non-recoded variables have preix of “ST” for the questionnaire items in the student background questionnaire, and “IC” for the items in the information and communication technology familiarity questionnaire. All the context questionnaires as well as the PISA international database, including all variables, are available through www.pisa.oecd.org. Student-level simple indices Study programme In PISA 2012, study programmes available to 15-year-old students in each country were collected both through the student tracking form and the student questionnaire. All study programmes were classiied using ISCED (OECD, 1999). In the PISA international database, all national programmes are indicated in a variable (PrOgn) where the irst six digits refer to the national centre code and the last two digits to the national study programme code. The following internationally comparable indices were derived from the data on study programmes: • Programme level (ISCEDl) indicates whether students are (1) primary education level (ISCED 1); (2) lower-secondary education level (ISCED 2); or (3) upper secondary education level (ISCED 3). • Programme designation (ISCEDD) indicates the designation of the study programme: (1) = “A” (general programmes designed to give access to the next programme level); (2) = “b” (programmes designed to give access to vocational studies at the next programme level); (3) = “C” (programmes designed to give direct access to the labour market); or (4) = “m” (modular programmes that combine any or all of these characteristics). 130 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V IndIceS From The STudenT, School And PArenT conTexT QueSTIonnAIreS: Annex A1 • Programme orientation (ISCEDO) indicates whether the programme’s curricular content is (1) general; (2) pre-vocational; (3) vocational; or (4) modular programmes that combine any or all of these characteristics. Occupational status of parents Occupational data for both a student’s father and a student’s mother were obtained by asking open-ended questions in the student questionnaire. The responses were coded to four-digit ISCO codes (ILO, 1990) and then mapped to the SEI index of ganzeboom et al. (1992). Higher scores of SEI indicate higher levels of occupational status. The following three indices are obtained: • Mother’s occupational status (OCOD1). • father’s occupational status (OCOD2). • The highest occupational level of parents (HISEI) corresponds to the higher SEI score of either parent or to the only available parent’s SEI score. Some of the analyses distinguish between four different categories of occupations by the major groups identified by the ISCO coding of the highest parental occupation: Elementary (ISCO 9), semi-skilled blue-collar (ISCO 6, 7 and 8), semi-skilled white-collar (ISCO 4 and 5), skilled (ISCO 1, 2 and 3). This classification follows the same methodology used in other OECD publications such as Education at a Glance (OECD, 2013b) and the OECD Skills Outlook (OECD, 2013c).1 Education level of parents The education level of parents is classiied using ISCED (OECD, 1999) based on students’ responses in the student questionnaire. As in PISA 2000, 2003, 2006 and 2009, indices were constructed by selecting the highest level for each parent and then assigning them to the following categories: (0) none, (1) ISCED 1 (primary education), (2) ISCED 2 (lower secondary), (3) ISCED 3b or 3C (vocational/pre-vocational upper secondary), (4) ISCED 3A (upper secondary) and/or ISCED 4 (non-tertiary post-secondary), (5) ISCED 5B (vocational tertiary), (6) ISCED 5A, 6 (theoretically oriented tertiary and post-graduate). The following three indices with these categories are developed: • Mother’s education level (MISCED). • father’s education level (fISCED). • Highest education level of parents (HISCED) corresponds to the higher ISCED level of either parent. Highest education level of parents was also converted into the number of years of schooling (PArED). for the conversion of level of education into years of schooling, see Table A1.1 in Volume I (OECD, 2013d). Immigration background Information on the country of birth of students and their parents is collected in a similar manner as in PISA 2000, PISA 2003 and PISA 2006 by using nationally speciic ISO coded variables. The ISO codes of the country of birth for students and their parents are available in the PISA international database (CObn_S, CObn_m, and CObn_f). The index on immigrant background (ImmIg) has the following categories: (1) native students (those students born in the country of assessment, or those with at least one parent born in that country; students who were born abroad with at least one parent born in the country of assessment are also classiied as native students), (2) second-generation students (those born in the country of assessment but whose parents were born in another country) and (3) irst-generation students (those born outside the country of assessment and whose parents were also born in another country). Students with missing responses for either the student or for both parents, or for all three questions have been given missing values for this variable. Use of computers at home An indicator about students’ use of desktop, laptop or tablet computers at home was derived using their responses to the questionnaire on students’ familiarity with information and communication. Three items in question IC01 (“Are any of these devices available for you to use at home?”) were used: Desktop computer; Portable laptop or notebook; <Tablet computer> (e.g. <iPad®>, <blackberry® PlaybookTm>). Students who answered “Yes, and I use it” to at least one of these questions have a value of 1 for this indicator. Use of computers at school An indicator about students’ use of desktop, laptop or tablet computers at school was derived using their responses to the questionnaire on students’ familiarity with information and communication technology (ICT). Three items in question IC02 (“Are any of these devices available for you to use at school?”) were used: Desktop computer; Portable laptop or notebook; <Tablet computer> (e.g. <iPad®>, <blackberry® PlaybookTm>). Students who answered “Yes, and I use it” to at least one of these questions have a value of 1 for this indicator. 1. note that for ISCO coding 0 “Arm forces”, the following recoding was followed: “Oficers” were coded as “managers” (ISCO 1), and “Other armed forces occupations” (drivers, gunners, seaman, generic armed forces) as “Plant and machine operators” (ISCO 8). In addition, all answers starting with “97” (housewives, students, and “vague occupations”) were coded into missing. CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 131 Annex A1: IndIceS From The STudenT, School And PArenT conTexT QueSTIonnAIreS Student-level scale indices In order to obtain trends for socio-economic scale indices from 2000 to 2012, the scaling of the indices WEALTH, HEDrES, CuLTPOSS, HOMEPOS and ESCS was based on data from all cycles from 2000 to 2012. Family wealth The index of family wealth (WEALTH) is based on students’ responses on whether they had the following at home: a room of their own, a link to the Internet, a dishwasher (treated as a country-speciic item), a DVD player, and three other country-speciic items; and their responses on the number of cellular phones, televisions, computers, cars and the number of rooms with a bath or shower. Home educational resources The index of home educational resources (HEDrES) is based on the items measuring the existence of educational resources at home including a desk and a quiet place to study, a computer that students can use for schoolwork, educational software, books to help with students’ school work, technical reference books and a dictionary. Cultural possessions The index of cultural possessions (CulTPOSS) is based on the students’ responses to whether they had the following at home: classic literature, books of poetry and works of art. Economic, social and cultural status The PISA index of economic, social and cultural status (ESCS) was derived from the following three indices: highest occupational status of parents (HISEI), highest education level of parents in years of education according to ISCED (PArED), and home possessions (HOmEPOS). The index of home possessions (HOmEPOS) comprises all items on the indices of WEAlTH, CulTPOSS and HEDrES, as well as books in the home recoded into a four-level categorical variable (0-10 books, 11-25 or 26-100 books, 101-200 or 201-500 books, more than 500 books). The PISA index of economic, social and cultural status (ESCS) was derived from a principal component analysis of standardised variables (each variable has an OECD mean of zero and a standard deviation of one), taking the factor scores for the irst principal component as measures of the PISA index of economic, social and cultural status. Principal component analysis was also performed for each participating country to determine to what extent the components of the index operate in similar ways across countries. The analysis revealed that patterns of factor loading were very similar across countries, with all three components contributing to a similar extent to the index (for details on reliability and factor loadings, see the PISA 2012 Technical Report (OECD, forthcoming). The imputation of components for students with missing data on one component was done on the basis of a regression on the other two variables, with an additional random error component. The inal values on the PISA index of economic, social and cultural status (ESCS) for PISA 2012 have an OECD mean of zero and a standard deviation of one. Perseverance The index of perseverance (PErSEV) was constructed using student responses (ST93) over whether they report that the following statements describe them very much, mostly, somewhat, not much, not at all: When confronted with a problem, I give up easily; I put off dificult problems; I remain interested in the tasks that I start; I continue working on tasks until everything is perfect; When confronted with a problem, I do more than what is expected of me. Openness to problem solving The index of openness to problem solving (OPEnPS) was constructed using student responses (ST94) over whether they report that the following statements describe them very much, mostly, somewhat, not much, not at all: I can handle a lot of information; I am quick to understand things; I seek explanations of things; I can easily link facts together; I like to solve complex problems. The rotated design of the student questionnaire A major innovation in PISA 2012 is the rotated design of the student questionnaire. One of the main reasons for a rotated design, which has previously been implemented for the cognitive assessment, was to extend the content coverage of the student questionnaire. Table A1.1 provides an overview of the rotation design and content of questionnaire forms for the main survey. The PISA 2012 Technical Report (OECD, forthcoming) provides all details regarding the rotated design of the student questionnaire in PISA 2012, including its implications in terms of (a) proiciency estimates, (b) international reports and trends, (c) further analyses, (d) structure and documentation of the international database, and (e) logistics have been discussed elsewhere. The rotated design has negligible implications for proiciency estimates and correlations of proiciency estimates with context constructs. The international database (available at www.pisa.oecd.org) includes all background variables for each student. The variables based on questions that students answered relect their responses; those that are based on questions that were not administered show a distinctive missing code. rotation allows the estimation of a full co-variance matrix which means that all variables can be correlated with all other variables. It does not affect conclusions in terms of whether or not an effect would be considered signiicant in multilevel models. 132 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V IndIceS From The STudenT, School And PArenT conTexT QueSTIonnAIreS: Annex A1 table a1.1 Student questionnaire rotation design form A Common Question Set (all forms) Question Set 1 – Mathematics Attitudes / Problem Solving Question Set 3 – Opportunity to Learn / Learning Strategies form B Common Question Set (all forms) Question Set 2 – School Climate / Attitudes towards School / Anxiety Question Set 1 – Mathematics Attitudes / Problem Solving form C Common Question Set (all forms) Question Set 3 – Opportunity to Learn / Learning Strategies Question Set 2 – School Climate / Attitudes towards School / Anxiety note: for details regarding the questions in each question set, please refer to the PISA 2012 Technical Report (OECD, forthcoming). References Ganzeboom, H.B.G., P. De Graaf, and D.J. Treiman (with J. De Leeuw) (1992), “A Standard International Socio-Economic Index of Occupational Status”, Social Science Research (21-1), pp. 1-56. ILO (1990), ISCO-88: International Standard Classiication of Occupations, International labour Ofice, geneva. OECD (forthcoming), PISA 2012 Technical Report, PISA, OECD Publishing. OECD (2013a), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264190511-en OECD (2013b), Education at a Glance 2013: OECD Indicators, OECD Publishing. http://dx.doi.org/10.1787/eag-2013-en OECD (2013c), OECD Skills Outlook 2013: First Results from the Survey of Adult Skills, OECD Publishing. http://dx.doi.org/10.1787/9789264204256-en OECD (2013d), PISA 2012 Results: What Students Know and Can Do: Student Performance in Mathematics, Reading and Science (Volume I), PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264201118-en OECD (2004), Learning for Tomorrow’s World: First Results from PISA 2003, PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264006416-en OECD (1999), Classifying Educational Programmes: Manual for ISCED-97 Implementation in OECD Countries. http://www.oecd.org/education/skills-beyond-school/1962350.pdf Warm, T.A. (1989), “Weighted likelihood estimation of ability in item response theory”, Psychometrika, Volume 54, Issue 3, pp 427-450. http://dx.doi.org/10.1007/BF02294627 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 133 Annex A2: The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS Annex A2 The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS Deinition of the PISA target population PISA 2012 provides an assessment of the cumulative yield of education and learning at a point at which most young adults are still enrolled in initial education. A major challenge for an international survey is to ensure that international comparability of national target populations is guaranteed in such a venture. Differences between countries in the nature and extent of pre-primary education and care, the age of entry into formal schooling and the institutional structure of education systems do not allow the deinition of internationally comparable grade levels of schooling. Consequently, international comparisons of education performance typically deine their populations with reference to a target age group. Some previous international assessments have deined their target population on the basis of the grade level that provides maximum coverage of a particular age cohort. A disadvantage of this approach is that slight variations in the age distribution of students across grade levels often lead to the selection of different target grades in different countries, or between education systems within countries, raising serious questions about the comparability of results across, and at times within, countries. In addition, because not all students of the desired age are usually represented in grade-based samples, there may be a more serious potential bias in the results if the unrepresented students are typically enrolled in the next higher grade in some countries and the next lower grade in others. This would exclude students with potentially higher levels of performance in the former countries and students with potentially lower levels of performance in the latter. In order to address this problem, PISA uses an age-based deinition for its target population, i.e. a deinition that is not tied to the institutional structures of national education systems. PISA assesses students who were aged between 15 years and 3 (complete) months and 16 years and 2 (complete) months at the beginning of the assessment period, plus or minus a 1 month allowable variation, and who were enrolled in an educational institution with grade 7 or higher, regardless of the grade levels or type of institution in which they were enrolled, and regardless of whether they were in full-time or part-time education. Educational institutions are generally referred to as schools in this publication, although some educational institutions (in particular, some types of vocational education establishments) may not be termed schools in certain countries. As expected from this deinition, the average age of students across OECD countries was 15 years and 9 months. The range in country means was 2 months and 5 days (0.18 years), from the minimum country mean of 15 years and 8 months to the maximum country mean of 15 years and 10 months. given this deinition of population, PISA makes statements about the knowledge and skills of a group of individuals who were born within a comparable reference period, but who may have undergone different educational experiences both in and outside of schools. In PISA, these knowledge and skills are referred to as the yield of education at an age that is common across countries. Depending on countries’ policies on school entry, selection and promotion, these students may be distributed over a narrower or a wider range of grades across different education systems, tracks or streams. It is important to consider these differences when comparing PISA results across countries, as observed differences between students at age 15 may no longer appear as students’ educational experiences converge later on. If a country’s scale scores in reading, scientiic or mathematical literacy are signiicantly higher than those in another country, it cannot automatically be inferred that the schools or particular parts of the education system in the irst country are more effective than those in the second. However, one can legitimately conclude that the cumulative impact of learning experiences in the irst country, starting in early childhood and up to the age of 15, and embracing experiences both in school, home and beyond, have resulted in higher outcomes in the literacy domains that PISA measures. The PISA target population did not include residents attending schools in a foreign country. It does, however, include foreign nationals attending schools in the country of assessment. To accommodate countries that desired grade-based results for the purpose of national analyses, PISA 2012 provided a sampling option to supplement age-based sampling with grade-based sampling. Population coverage All countries attempted to maximise the coverage of 15-year-olds enrolled in education in their national samples, including students enrolled in special educational institutions. As a result, PISA 2012 reached standards of population coverage that are unprecedented in international surveys of this kind. The sampling standards used in PISA permitted countries to exclude up to a total of 5% of the relevant population either by excluding schools or by excluding students within schools. All but eight countries, luxembourg (8.40%), Canada (6.38%), Denmark (6.18%), norway (6.11%), Estonia (5.80%), Sweden (5.44%), the united kingdom (5.43%) and the united States (5.35%), achieved this standard, and in 30 countries and economies, the overall exclusion rate was less than 2%. When language exclusions were accounted for (i.e. removed from the overall exclusion rate), Norway , Sweden, the united kingdom and the united States no longer had an exclusion rate greater than 5%. for details, see www.pisa.oecd.org. 134 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS: Annex A2 Exclusions within the above limits include: • At the school level: i) schools that were geographically inaccessible or where the administration of the PISA assessment was not considered feasible; and ii) schools that provided teaching only for students in the categories defined under “within-school exclusions”, such as schools for the blind. The percentage of 15-year-olds enrolled in such schools had to be less than 2.5% of the nationally desired target population [0.5% maximum for i) and 2% maximum for ii)]. The magnitude, nature and justification of school-level exclusions are documented in the PISA 2012 Technical Report (OECD, forthcoming). • At the student level: i) students with an intellectual disability; ii) students with a functional disability; iii) students with limited assessment language proficiency; iv) other – a category defined by the national centres and approved by the international centre; and v) students taught in a language of instruction for the main domain for which no materials were available. Students could not be excluded solely because of low proficiency or common discipline problems. The percentage of 15-year-olds excluded within schools had to be less than 2.5% of the nationally desired target population. Table A2.1 describes the target population of the countries participating in PISA 2012. further information on the target population and the implementation of PISA sampling standards can be found in the PISA 2012 Technical Report (OECD, forthcoming). • Column 1 shows the total number of 15-year-olds according to the most recent available information, which in most countries meant the year 2011 as the year before the assessment. • Column 2 shows the number of 15-year-olds enrolled in schools in grade 7 or above (as defined above), which is referred to as the eligible population. • Column 3 shows the national desired target population. Countries were allowed to exclude up to 0.5% of students a priori from the eligible population, essentially for practical reasons. The following a priori exclusions exceed this limit but were agreed with the PISA Consortium: Belgium excluded 0.23% of its population for a particular type of student educated while working; Canada excluded 1.14% of its population from Territories and Aboriginal reserves; Chile excluded 0.04% of its students who live in Easter Island, Juan fernandez Archipelago and Antarctica; Indonesia excluded 1.55% of its students from two provinces because of operational reasons; Ireland excluded 0.05% of its students in three island schools off the west coast; Latvia excluded 0.08% of its students in distance learning schools; and Serbia excluded 2.11% of its students taught in Serbian in kosovo. • Column 4 shows the number of students enrolled in schools that were excluded from the national desired target population either from the sampling frame or later in the field during data collection. • Column 5 shows the size of the national desired target population after subtracting the students enrolled in excluded schools. This is obtained by subtracting Column 4 from Column 3. • Column 6 shows the percentage of students enrolled in excluded schools. This is obtained by dividing Column 4 by Column 3 and multiplying by 100. • Column 7 shows the number of students participating in PISA 2012. Note that in some cases this number does not account for 15-year-olds assessed as part of additional national options. • Column 8 shows the weighted number of participating students, i.e. the number of students in the nationally defined target population that the PISA sample represents. • Each country attempted to maximise the coverage of the PISA target population within the sampled schools. In the case of each sampled school, all eligible students, namely those 15 years of age, regardless of grade, were first listed. Sampled students who were to be excluded had still to be included in the sampling documentation, and a list drawn up stating the reason for their exclusion. Column 9 indicates the total number of excluded students, which is further described and classified into specific categories in Table A2.2. • Column 10 indicates the weighted number of excluded students, i.e. the overall number of students in the nationally defined target population represented by the number of students excluded from the sample, which is also described and classified by exclusion categories in Table A2.2. Excluded students were excluded based on five categories: i) students with an intellectual disability – the student has a mental or emotional disability and is cognitively delayed such that he/she cannot perform in the PISA testing situation; ii) students with a functional disability – the student has a moderate to severe permanent physical disability such that he/she cannot perform in the PISA testing situation; iii) students with a limited assessment language proficiency – the student is unable to read or speak any of the languages of the assessment in the country and would be unable to overcome the language barrier in the testing situation (typically a student who has received less than one year of instruction in the languages of the assessment may be excluded); iv) other – a category defined by the national centres and approved by the international centre; and v) students taught in a language of instruction for the main domain for which no materials were available. • Column 11 shows the percentage of students excluded within schools. This is calculated as the weighted number of excluded students (Column 10), divided by the weighted number of excluded and participating students (Column 8 plus Column 10), then multiplied by 100. • Column 12 shows the overall exclusion rate, which represents the weighted percentage of the national desired target population excluded from PISA either through school-level exclusions or through the exclusion of students within schools. It is calculated as the school-level exclusion rate (Column 6 divided by 100) plus within-school exclusion rate (Column 11 divided by 100) multiplied by 1 minus the school-level exclusion rate (Column 6 divided by 100). This result is then multiplied by 100. Eight countries, Canada, Denmark, Estonia, Luxembourg, Norway, Sweden, the united kingdom and the united States, had exclusion rates higher than 5%. When language exclusions were accounted for (i.e. removed from the overall exclusion rate), Norway, Sweden, the united kingdom and the united States no longer had an exclusion rate greater than 5%”. CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 135 Annex A2: The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS table a2.1 [Part 1/2] PISA target populations and samples Population and sample information number of participating students Weighted number of participating students (3) 288 159 89 073 121 209 404 767 252 625 93 214 70 854 12 438 62 195 755 447 798 136 105 096 108 816 4 491 57 952 113 278 566 973 1 214 756 672 101 6 082 1 472 875 193 190 59 118 64 777 410 700 127 537 59 367 18 935 404 374 102 027 85 239 965 736 745 581 4 074 457 (4) 5 702 106 1 324 2 936 2 687 1 577 1 965 442 523 27 403 10 914 1 364 1 725 10 0 2 784 8 498 26 099 3 053 151 7 307 7 546 579 750 6 900 0 1 480 115 2 031 1 705 2 479 10 387 19 820 41 142 (5) 282 457 88 967 119 885 401 831 249 938 91 637 68 889 11 996 61 672 728 044 787 222 103 732 107 091 4 481 57 952 110 494 558 475 1 188 657 669 048 5 931 1 465 568 185 644 58 539 64 027 403 800 127 537 57 887 18 820 402 343 100 322 82 760 955 349 725 761 4 033 315 (6) 1.98 0.12 1.09 0.73 1.06 1.69 2.77 3.55 0.84 3.63 1.37 1.30 1.59 0.22 0.00 2.46 1.50 2.15 0.45 2.48 0.50 3.91 0.98 1.16 1.68 0.00 2.49 0.61 0.50 1.67 2.91 1.08 2.66 1.01 (7) 17 774 4 756 9 690 21 548 6 857 6 535 7 481 5 867 8 829 5 682 5 001 5 125 4 810 3 508 5 016 6 061 38 142 6 351 5 033 5 260 33 806 4 460 5 248 4 686 5 662 5 722 5 737 7 229 25 335 4 739 11 234 4 848 12 659 6 111 (8) 250 779 82 242 117 912 348 070 229 199 82 101 65 642 11 634 60 047 701 399 756 907 96 640 91 179 4 169 54 010 107 745 521 288 1 128 179 603 632 5 523 1 326 025 196 262 53 414 59 432 379 275 96 034 54 486 18 303 374 266 94 988 79 679 866 681 688 236 3 536 153 50 157 637 603 2 786 064 59 684 620 422 64 326 46 550 9 955 77 864 3 544 028 125 333 247 048 18 375 383 35 567 5 416 457 999 8 600 508 969 11 532 146 243 1 268 814 74 272 90 796 52 163 328 336 784 897 132 313 48 446 46 442 1 091 462 56 3 995 34 932 1 437 4 0 417 128 813 8 039 141 7 374 655 1 526 6 225 18 263 202 5 091 17 800 1 987 1 252 293 1 747 9 123 169 971 14 7 729 50 101 633 608 2 751 132 58 247 620 418 64 326 46 133 9 827 77 051 3 535 989 125 192 239 674 17 720 382 35 041 5 410 457 774 8 582 508 706 11 330 141 152 1 251 014 72 285 89 544 51 870 326 589 775 774 132 144 47 475 46 428 1 083 733 0.11 0.63 1.25 2.41 0.00 0.00 0.90 1.29 1.04 0.23 0.11 2.98 3.56 0.26 1.48 0.11 0.05 0.21 0.05 1.75 3.48 1.40 2.67 1.38 0.56 0.53 1.16 0.13 2.00 0.03 0.71 4 743 5 908 20 091 5 282 11 173 4 602 6 153 5 078 4 670 5 622 7 038 5 808 5 276 293 4 618 5 335 5 197 4 744 6 035 10 966 5 074 6 418 4 684 6 374 5 546 6 046 6 606 4 407 11 500 5 315 4 959 42 466 545 942 2 470 804 54 255 560 805 40 384 45 502 9 650 70 636 2 645 155 111 098 208 411 16 054 314 33 042 5 366 432 080 7 714 419 945 11 003 140 915 1 172 539 67 934 85 127 51 088 292 542 703 012 120 784 40 612 39 771 956 517 OECD School-level exclusion rate (%) total population of 15-year-olds total in national desired target population australia austria belgium canada chile czech republic denmark Estonia finland france Germany Greece hungary iceland ireland israel italy Japan korea luxembourg mexico netherlands new Zealand norway Poland Portugal Slovak republic Slovenia Spain Sweden Switzerland turkey united kingdom united States (1) 291 967 93 537 123 469 417 873 274 803 96 946 72 310 12 649 62 523 792 983 798 136 110 521 111 761 4 505 59 296 118 953 605 490 1 241 786 687 104 6 187 2 114 745 194 000 60 940 64 917 425 597 108 728 59 723 19 471 423 444 102 087 87 200 1 266 638 738 066 3 985 714 (2) 288 159 89 073 121 493 409 453 252 733 93 214 70 854 12 438 62 195 755 447 798 136 105 096 108 816 4 491 57 979 113 278 566 973 1 214 756 672 101 6 082 1 472 875 193 190 59 118 64 777 410 700 127 537 59 367 18 935 404 374 102 027 85 239 965 736 745 581 4 074 457 Partners total schoollevel exclusions total in national desired target population after all school exclusions and before within-school exclusions total enrolled population of 15-year-olds at Grade 7 or above albania argentina brazil bulgaria colombia costa rica croatia cyprus* hong kong-china indonesia Jordan kazakhstan latvia liechtenstein lithuania macao-china malaysia montenegro Peru Qatar romania russian federation Serbia Shanghai-china Singapore chinese taipei thailand tunisia united arab Emirates uruguay viet nam 76 910 684 879 3 574 928 70 188 889 729 81 489 48 155 9 956 84 200 4 174 217 129 492 258 716 18 789 417 38 524 6 600 544 302 8 600 584 294 11 667 146 243 1 272 632 80 089 108 056 53 637 328 356 982 080 132 313 48 824 54 638 1 717 996 50 157 637 603 2 786 064 59 684 620 422 64 326 46 550 9 956 77 864 3 599 844 125 333 247 048 18 389 383 35 567 5 416 457 999 8 600 508 969 11 532 146 243 1 268 814 75 870 90 796 52 163 328 336 784 897 132 313 48 446 46 442 1 091 462 Notes: for a full explanation of the details in this table please refer to the PISA 2012 Technical Report (OECD, forthcoming). The igure for total national population of 15-year-olds enrolled in Column 2 may occasionally be larger than the total number of 15-year-olds in Column 1 due to differing data sources. Information for the adjudicated regions is available on line. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003725 136 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS: Annex A2 table a2.1 [Part 2/2] PISA target populations and samples Population and sample information OECD australia austria belgium canada chile czech republic denmark Estonia finland france Germany Greece hungary iceland ireland israel italy Japan korea luxembourg mexico netherlands new Zealand norway Poland Portugal Slovak republic Slovenia Spain Sweden Switzerland turkey united kingdom united States Partners number of excluded students albania argentina brazil bulgaria colombia costa rica croatia cyprus* hong kong-china indonesia Jordan kazakhstan latvia liechtenstein lithuania macao-china malaysia montenegro Peru Qatar romania russian federation Serbia Shanghai-china Singapore chinese taipei thailand tunisia united arab Emirates uruguay viet nam Weighted number of excluded students coverage indices Within-school exclusion rate (%) overall exclusion rate (%) coverage index 1: coverage of national desired population coverage index 2: coverage of national enrolled population coverage index 3: coverage of 15-year-old population (9) 505 46 39 1 796 18 15 368 143 225 52 8 136 27 155 271 114 741 0 17 357 58 27 255 278 212 124 29 84 959 201 256 21 486 319 (10) 5 282 1 011 367 21 013 548 118 2 381 277 653 5 828 1 302 2 304 928 156 2 524 1 884 9 855 0 2 238 357 3 247 1 056 2 030 3 133 11 566 1 560 246 181 14 931 3 789 1 093 3 684 20 173 162 194 (11) 2.06 1.21 0.31 5.69 0.24 0.14 3.50 2.33 1.08 0.82 0.17 2.33 1.01 3.60 4.47 1.72 1.86 0.00 0.37 6.07 0.24 0.54 3.66 5.01 2.96 1.60 0.45 0.98 3.84 3.84 1.35 0.42 2.85 4.39 (12) 4.00 1.33 1.40 6.38 1.30 1.83 6.18 5.80 1.91 4.42 1.54 3.60 2.58 3.81 4.47 4.13 3.33 2.15 0.82 8.40 0.74 4.42 4.61 6.11 4.59 1.60 2.93 1.58 4.32 5.44 4.22 1.49 5.43 5.35 (13) 0.960 0.987 0.986 0.936 0.987 0.982 0.938 0.942 0.981 0.956 0.985 0.964 0.974 0.962 0.955 0.959 0.967 0.979 0.992 0.872 0.993 0.956 0.954 0.939 0.954 0.984 0.971 0.984 0.957 0.946 0.958 0.985 0.946 0.946 (14) 0.960 0.987 0.984 0.926 0.987 0.982 0.938 0.942 0.981 0.956 0.985 0.964 0.974 0.962 0.955 0.959 0.967 0.979 0.992 0.916 0.993 0.956 0.954 0.939 0.954 0.984 0.971 0.984 0.957 0.946 0.958 0.985 0.946 0.946 (15) 0.859 0.879 0.955 0.833 0.834 0.847 0.908 0.920 0.960 0.885 0.948 0.874 0.816 0.925 0.911 0.906 0.861 0.909 0.879 0.893 0.627 1.012 0.876 0.916 0.891 0.883 0.912 0.940 0.884 0.930 0.914 0.684 0.932 0.887 1 12 44 6 23 2 91 157 38 2 19 25 14 13 130 3 7 4 8 85 0 69 10 8 33 44 12 5 11 15 1 10 641 4 900 80 789 12 627 200 518 860 304 951 76 13 867 3 554 8 549 85 0 11 940 136 107 315 2 029 1 144 130 37 99 198 0.02 0.12 0.20 0.15 0.14 0.03 1.36 2.03 0.73 0.03 0.27 0.45 0.47 3.97 2.56 0.06 0.13 0.10 0.13 0.77 0.00 1.01 0.20 0.13 0.61 0.69 0.16 0.11 0.09 0.25 0.02 0.14 0.74 1.45 2.55 0.14 0.03 2.24 3.29 1.76 0.26 0.39 3.43 4.02 4.22 4.00 0.17 0.18 0.31 0.18 2.51 3.48 2.40 2.87 1.50 1.17 1.22 1.32 0.24 2.09 0.28 0.73 0.999 0.993 0.986 0.974 0.999 1.000 0.978 0.967 0.982 0.997 0.996 0.966 0.960 0.958 0.960 0.998 0.998 0.997 0.998 0.975 0.965 0.976 0.971 0.985 0.988 0.988 0.987 0.998 0.979 0.997 0.993 0.999 0.993 0.986 0.974 0.999 1.000 0.978 0.967 0.982 0.982 0.996 0.966 0.959 0.958 0.960 0.998 0.998 0.997 0.998 0.975 0.965 0.976 0.951 0.985 0.988 0.988 0.987 0.998 0.979 0.997 0.993 0.552 0.797 0.691 0.773 0.630 0.496 0.945 0.969 0.839 0.634 0.858 0.806 0.854 0.753 0.858 0.813 0.794 0.897 0.719 0.943 0.964 0.921 0.848 0.788 0.952 0.891 0.716 0.913 0.832 0.728 0.557 Notes: for a full explanation of the details in this table please refer to the PISA 2012 Technical Report (OECD, forthcoming). The igure for total national population of 15-year-olds enrolled in Column 2 may occasionally be larger than the total number of 15-year-olds in Column 1 due to differing data sources. Information for the adjudicated regions is available on line. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003725 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 137 Annex A2: The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS table a2.2 [Part 1/1] exclusions Student exclusions (unweighted) Student exclusions (weighted) OECD australia austria belgium canada chile czech republic denmark Estonia finland france Germany Greece hungary iceland ireland israel italy Japan luxembourg mexico netherlands new Zealand norway Poland Portugal korea Slovak republic Slovenia Spain Sweden Switzerland turkey united kingdom united States Partners Weighted number number Weighted Weighted of excluded of excluded number number number number students students number of number Weighted Weighted of of excluded of excluded number because of because of number of of excluded excluded students of excluded of excluded no materials students total total students excluded excluded no materials students with with weighted available in available in number students students with students students with functional intellectual because of for other the language number of of functional intellectual because of for other the language reasons of instruction excluded reasons of instruction excluded disability language disability disability language disability (code 1) students students (code 5) (code 5) (code 4) (code 3) (code 1) (code 2) (code 3) (code 4) (code 2) albania argentina brazil bulgaria colombia costa rica croatia cyprus* hong kong-china indonesia Jordan kazakhstan latvia liechtenstein lithuania macao-china malaysia montenegro Peru Qatar romania russian federation Serbia Shanghai-china Singapore chinese taipei thailand tunisia united arab Emirates uruguay viet nam (1) 39 11 5 82 3 1 10 7 5 52 0 3 1 5 13 9 64 0 6 21 5 27 11 23 69 2 2 13 56 120 7 5 40 37 (2) 395 24 22 1 593 15 8 204 134 80 0 4 18 15 105 159 91 566 0 261 36 21 118 192 89 48 15 14 27 679 0 99 14 405 219 (3) 71 11 12 121 0 6 112 2 101 0 4 4 2 27 33 14 111 0 90 1 1 99 75 6 7 0 0 44 224 81 150 2 41 63 (4) 0 0 0 0 0 0 42 0 15 0 0 111 9 18 66 0 0 0 0 0 0 0 0 88 0 0 13 0 0 0 0 0 0 0 (5) 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 11 0 6 0 0 0 0 0 0 0 0 0 0 (6) 505 46 39 1 796 18 15 368 143 225 52 8 136 27 155 271 114 741 0 357 58 27 255 278 212 124 17 29 84 959 201 256 21 486 319 (7) 471 332 24 981 74 1 44 14 43 5 828 0 49 36 5 121 133 596 0 6 812 188 235 120 1 470 860 223 22 23 618 2 218 41 757 1 468 18 399 (8) 3 925 438 154 18 682 474 84 1 469 260 363 0 705 348 568 105 1 521 1 492 7 899 0 261 2 390 819 926 2 180 5 187 605 2 015 135 76 11 330 0 346 2 556 15 514 113 965 (9) 886 241 189 1 350 0 34 559 3 166 0 597 91 27 27 283 260 1 361 0 90 45 50 813 832 177 94 0 0 81 2 984 1 571 706 371 3 191 29 830 (10) 0 0 0 0 0 0 310 0 47 0 0 1 816 296 18 599 0 0 0 0 0 0 0 0 4 644 0 0 89 0 0 0 0 0 0 0 0 1 17 6 12 0 10 8 4 1 8 9 3 1 10 0 3 3 3 23 0 25 4 1 5 6 2 4 3 9 0 0 11 27 0 10 2 78 54 33 0 6 16 7 7 120 1 4 1 5 43 0 40 4 6 17 36 10 1 7 6 1 1 0 0 0 1 0 3 60 1 1 5 0 4 5 0 2 0 0 0 19 0 4 2 1 11 2 0 0 1 0 0 0 0 0 0 0 0 0 35 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 12 44 6 23 2 91 157 38 2 19 25 14 13 130 3 7 4 8 85 0 69 10 8 33 44 12 5 11 15 1 0 84 1 792 80 397 0 69 9 57 426 109 317 8 1 66 0 274 7 269 23 0 4 345 53 14 50 296 13 104 26 66 0 0 557 3 108 0 378 12 539 64 446 0 72 634 45 7 801 1 279 1 280 43 0 6 934 55 80 157 1 664 1 131 26 9 33 198 10 0 0 0 14 0 19 72 15 434 122 0 24 5 0 2 0 0 0 19 0 660 28 14 109 70 0 0 2 0 0 0 0 0 0 0 0 0 55 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (11) 0 0 0 0 0 0 0 0 35 0 0 0 0 0 0 0 0 0 0 0 0 57 0 89 0 0 0 0 0 0 0 0 0 0 (12) 5 282 1 011 367 21 013 548 118 2 381 277 653 5 828 1 302 2 304 928 156 2 524 1 884 9 855 0 357 3 247 1 056 2 030 3 133 11 566 1 560 2 238 246 181 14 931 3 789 1 093 3 684 20 173 162 194 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 641 4 900 80 789 12 627 200 518 860 304 951 76 13 867 3 554 8 549 85 0 11 940 136 107 315 2 029 1 144 130 37 99 198 Exclusion codes: Code 1 functional disability – student has a moderate to severe permanent physical disability. Code 2 Intellectual disability – student has a mental or emotional disability and has either been tested as cognitively delayed or is considered in the professional opinion of qualiied staff to be cognitively delayed. Code 3 limited assessment language proiciency – student is not a native speaker of any of the languages of the assessment in the country and has been resident in the country for less than one year. Code 4 Other reasons deined by the national centres and approved by the international centre. Code 5 no materials available in the language of instruction. Note: for a full explanation of the details in this table please refer to the PISA 2012 Technical Report (OECD, forthcoming). Information for the adjudicated regions is available on line. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003725 138 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS: Annex A2 • Column 13 presents an index of the extent to which the national desired target population is covered by the PISA sample. Canada, Denmark, Estonia, Luxembourg, Norway, Sweden, the united kingdom and the united States were the only countries where the coverage is below 95%. • Column 14 presents an index of the extent to which 15-year-olds enrolled in schools are covered by the PISA sample. The index measures the overall proportion of the national enrolled population that is covered by the non-excluded portion of the student sample. The index takes into account both school-level and student-level exclusions. Values close to 100 indicate that the PISA sample represents the entire education system as defined for PISA 2012. The index is the weighted number of participating students (Column 8) divided by the weighted number of participating and excluded students (Column 8 plus Column 10), times the nationally defined target population (Column 5) divided by the eligible population (Column 2). • Column 15 presents an index of the coverage of the 15-year-old population. This index is the weighted number of participating students (Column 8) divided by the total population of 15-year-old students (Column 1). This high level of coverage contributes to the comparability of the assessment results. for example, even assuming that the excluded students would have systematically scored worse than those who participated, and that this relationship is moderately strong, an exclusion rate in the order of 5% would likely lead to an overestimation of national mean scores of less than 5 score points (on a scale with an international mean of 500 score points and a standard deviation of 100 score points). This assessment is based on the following calculations: if the correlation between the propensity of exclusions and student performance is 0.3, resulting mean scores would likely be overestimated by 1 score point if the exclusion rate is 1%, by 3 score points if the exclusion rate is 5%, and by 6 score points if the exclusion rate is 10%. If the correlation between the propensity of exclusions and student performance is 0.5, resulting mean scores would be overestimated by 1 score point if the exclusion rate is 1%, by 5 score points if the exclusion rate is 5%, and by 10 score points if the exclusion rate is 10%. for this calculation, a model was employed that assumes a bivariate normal distribution for performance and the propensity to participate. for details, see the PISA 2012 Technical Report (OECD, forthcoming). Sampling procedures and response rates The accuracy of any survey results depends on the quality of the information on which national samples are based as well as on the sampling procedures. Quality standards, procedures, instruments and veriication mechanisms were developed for PISA that ensured that national samples yielded comparable data and that the results could be compared with conidence. most PISA samples were designed as two-stage stratiied samples (where countries applied different sampling designs, these are documented in the PISA 2012 Technical Report [OECD, forthcoming]). The irst stage consisted of sampling individual schools in which 15-year-old students could be enrolled. Schools were sampled systematically with probabilities proportional to size, the measure of size being a function of the estimated number of eligible (15-year-old) students enrolled. A minimum of 150 schools were selected in each country (where this number existed), although the requirements for national analyses often required a somewhat larger sample. As the schools were sampled, replacement schools were simultaneously identiied, in case a sampled school chose not to participate in PISA 2012. In the case of Iceland, liechtenstein, luxembourg, macao-China and Qatar, all schools and all eligible students within schools were included in the sample. Experts from the PISA Consortium performed the sample selection process for most participating countries and monitored it closely in those countries that selected their own samples. The second stage of the selection process sampled students within sampled schools. Once schools were selected, a list of each sampled school’s 15-year-old students was prepared. from this list, 35 students were then selected with equal probability (all 15-year-old students were selected if fewer than 35 were enrolled). The number of students to be sampled per school could deviate from 35, but could not be less than 20. Data-quality standards in PISA required minimum participation rates for schools as well as for students. These standards were established to minimise the potential for response biases. In the case of countries meeting these standards, it was likely that any bias resulting from non-response would be negligible, i.e. typically smaller than the sampling error. A minimum response rate of 85% was required for the schools initially selected. Where the initial response rate of schools was between 65% and 85%, however, an acceptable school response rate could still be achieved through the use of replacement schools. This procedure brought with it a risk of increased response bias. Participating countries were, therefore, encouraged to persuade as many of the schools in the original sample as possible to participate. Schools with a student participation rate between 25% and 50% were not regarded as participating schools, but data from these schools were included in the database and contributed to the various estimations. Data from schools with a student participation rate of less than 25% were excluded from the database. PISA 2012 also required a minimum participation rate of 80% of students within participating schools. This minimum participation rate had to be met at the national level, not necessarily by each participating school. follow-up sessions were required in schools in which too few students had participated in the original assessment sessions. Student participation rates were calculated over all original schools, and also over all schools, whether original sample or replacement schools, and from the participation of students in both the original assessment and any follow-up sessions. A student who participated in the original or follow-up cognitive sessions was regarded as a participant. Those who attended only the questionnaire session were included in the international database and contributed to the statistics presented in this publication if they provided at least a description of their father’s or mother’s occupation. CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 139 Annex A2: The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS table a2.3 [Part 1/2] response rates Weighted school participation rate before replacement (%) Weighted number of responding schools (weighted also by enrolment) OECD final sample – after school replacement Weighted number of schools sampled (responding and non-responding) (weighted also by enrolment) australia austria belgium canada chile czech republic denmark Estonia finland france Germany Greece hungary iceland ireland israel italy Japan korea luxembourg mexico netherlands new Zealand norway Poland Portugal Slovak republic Slovenia Spain Sweden Switzerland turkey united kingdom united States (1) 98 100 84 91 92 98 87 100 99 97 98 93 98 99 99 91 89 86 100 100 92 75 81 85 85 95 87 98 100 99 94 97 80 67 (2) 268 631 88 967 100 482 362 178 220 009 87 238 61 749 12 046 59 740 703 458 735 944 95 107 99 317 4 395 56 962 99 543 478 317 1 015 198 661 575 5 931 1 323 816 139 709 47 441 54 201 343 344 122 238 50 182 18 329 402 604 98 645 78 825 921 643 564 438 2 647 253 (3) 274 432 88 967 119 019 396 757 239 429 88 884 71 015 12 046 60 323 728 401 753 179 102 087 101 751 4 424 57 711 109 326 536 921 1 175 794 662 510 5 931 1 442 242 185 468 58 676 63 653 402 116 128 129 57 353 18 680 403 999 99 726 83 450 945 357 705 011 3 945 575 (4) 757 191 246 828 200 292 311 206 310 223 227 176 198 133 182 166 1 104 173 156 42 1 431 148 156 177 159 186 202 335 902 207 397 165 477 139 (5) 790 191 294 907 224 297 366 206 313 231 233 192 208 140 185 186 1 232 200 157 42 1 562 199 197 208 188 195 236 353 904 211 422 170 550 207 (6) 98 100 97 93 99 100 96 100 99 97 98 99 99 99 99 94 97 96 100 100 95 89 89 95 98 96 99 98 100 100 98 100 89 77 (7) 268 631 88 967 115 004 368 600 236 576 88 447 67 709 12 046 59 912 703 458 737 778 100 892 101 187 4 395 57 316 103 075 522 686 1 123 211 661 575 5 931 1 374 615 165 635 52 360 60 270 393 872 122 713 57 599 18 329 402 604 99 536 82 032 944 807 624 499 3 040 661 (8) 274 432 88 967 119 006 396 757 239 370 88 797 70 892 12 046 60 323 728 401 753 179 102 053 101 751 4 424 57 711 109 895 536 821 1 175 794 662 510 5 931 1 442 234 185 320 58 616 63 642 402 116 128 050 58 201 18 680 403 999 99 767 83 424 945 357 699 839 3 938 077 Partners initial sample – before school replacement albania argentina brazil bulgaria colombia costa rica croatia cyprus* hong kong-china indonesia Jordan kazakhstan latvia liechtenstein lithuania macao-china malaysia montenegro Peru Qatar romania russian federation Serbia Shanghai-china Singapore chinese taipei thailand tunisia united arab Emirates uruguay viet nam 100 95 93 99 87 99 99 97 79 95 100 100 88 100 98 100 100 100 98 100 100 100 90 100 98 100 98 99 99 99 100 49 632 578 723 2 545 863 57 101 530 553 64 235 45 037 9 485 60 277 2 799 943 119 147 239 767 15 371 382 33 989 5 410 455 543 8 540 503 915 11 333 139 597 1 243 564 65 537 89 832 50 415 324 667 757 516 129 229 46 469 45 736 1 068 462 49 632 606 069 2 745 045 57 574 612 605 64 920 45 636 9 821 76 589 2 950 696 119 147 239 767 17 488 382 34 614 5 410 455 543 8 540 514 574 11 340 139 597 1 243 564 72 819 89 832 51 687 324 667 772 654 130 141 46 748 46 009 1 068 462 204 218 803 186 323 191 161 117 123 199 233 218 186 12 211 45 164 51 238 157 178 227 143 155 170 163 235 152 453 179 162 204 229 886 188 363 193 164 131 156 210 233 218 213 12 216 45 164 51 243 164 178 227 160 155 176 163 240 153 460 180 162 100 96 95 100 97 99 100 97 94 98 100 100 100 100 100 100 100 100 99 100 100 100 95 100 98 100 100 99 99 100 100 49 632 580 989 2 622 293 57 464 596 557 64 235 45 608 9 485 72 064 2 892 365 119 147 239 767 17 428 382 34 604 5 410 455 543 8 540 507 602 11 333 139 597 1 243 564 69 433 89 832 50 945 324 667 772 452 129 229 46 469 46 009 1 068 462 49 632 606 069 2 747 688 57 574 612 261 64 920 45 636 9 821 76 567 2 951 028 119 147 239 767 17 448 382 34 604 5 410 455 543 8 540 514 574 11 340 139 597 1 243 564 72 752 89 832 51 896 324 667 772 654 130 141 46 748 46 009 1 068 462 number of responding schools (unweighted) number of responding and non-responding schools (unweighted) Weighted school Weighted number of responding participation rate after replacement schools (weighted also by enrolment) (%) Information for the adjudicated regions is available on line. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003725 140 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V Weighted number of schools sampled (responding and non-responding) (weighted also by enrolment) The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS: Annex A2 table a2.3 [Part 2/2] response rates OECD australia austria belgium canada chile czech republic denmark Estonia finland france Germany Greece hungary iceland ireland israel italy Japan korea luxembourg mexico netherlands new Zealand norway Poland Portugal Slovak republic Slovenia Spain Sweden Switzerland turkey united kingdom united States Partners final sample – after school replacement albania argentina brazil bulgaria colombia costa rica croatia cyprus* hong kong-china indonesia Jordan kazakhstan latvia liechtenstein lithuania macao-china malaysia montenegro Peru Qatar romania russian federation Serbia Shanghai-china Singapore chinese taipei thailand tunisia united arab Emirates uruguay viet nam final sample – students within schools after school replacement number of students number of students sampled sampled (assessed number of students (assessed and absent) assessed and absent) (unweighted) (unweighted) (weighted) number of responding schools (unweighted) number of responding and non-responding schools (unweighted) Weighted student participation rate after replacement (%) number of students assessed (weighted) (9) 757 191 282 840 221 295 339 206 311 223 228 188 204 133 183 172 1 186 191 156 42 1 468 177 177 197 182 187 231 335 902 209 410 169 505 161 (10) 790 191 294 907 224 297 366 206 313 231 233 192 208 140 185 186 1 232 200 157 42 1 562 199 197 208 188 195 236 353 904 211 422 170 550 207 (11) 87 92 91 81 95 90 89 93 91 89 93 97 93 85 84 90 93 96 99 95 94 85 85 91 88 87 94 90 90 92 92 98 86 89 (12) 213 495 75 393 103 914 261 928 214 558 73 536 56 096 10 807 54 126 605 371 692 226 92 444 84 032 3 503 45 115 91 181 473 104 1 034 803 595 461 5 260 1 193 866 148 432 40 397 51 155 325 389 80 719 50 544 16 146 334 382 87 359 72 116 850 830 528 231 2 429 718 (13) 246 012 82 242 114 360 324 328 226 689 81 642 62 988 11 634 59 653 676 730 742 416 95 580 90 652 4 135 53 644 101 288 510 005 1 076 786 603 004 5 523 1 271 639 174 697 47 703 56 286 371 434 92 395 53 912 17 849 372 042 94 784 78 424 866 269 613 736 2 734 268 (14) 17 491 4 756 9 649 20 994 6 857 6 528 7 463 5 867 8 829 5 641 4 990 5 125 4 810 3 503 5 016 6 061 38 084 6 351 5 033 5 260 33 786 4 434 5 248 4 686 5 629 5 608 5 737 7 211 26 443 4 739 11 218 4 847 12 638 6 094 (15) 20 799 5 318 10 595 25 835 7 246 7 222 8 496 6 316 9 789 6 308 5 355 5 301 5 184 4 135 5 977 6 727 41 003 6 609 5 101 5 523 35 972 5 215 6 206 5 156 6 452 6 426 6 106 7 921 29 027 5 141 12 138 4 939 14 649 6 848 204 219 837 187 352 191 163 117 147 206 233 218 211 12 216 45 164 51 240 157 178 227 152 155 172 163 239 152 453 180 162 204 229 886 188 363 193 164 131 156 210 233 218 213 12 216 45 164 51 243 164 178 227 160 155 176 163 240 153 460 180 162 92 88 90 96 93 89 92 93 93 95 95 99 91 93 92 99 94 94 96 100 98 97 93 98 94 96 99 90 95 90 100 39 275 457 294 2 133 035 51 819 507 178 35 525 41 912 8 719 62 059 2 478 961 105 493 206 053 14 579 293 30 429 5 335 405 983 7 233 398 193 10 966 137 860 1 141 317 60 366 83 821 47 465 281 799 695 088 108 342 38 228 35 800 955 222 42 466 519 733 2 368 438 54 145 544 862 39 930 45 473 9 344 66 665 2 605 254 111 098 208 411 16 039 314 33 042 5 366 432 080 7 714 414 728 10 996 140 915 1 172 539 64 658 85 127 50 330 292 542 702 818 119 917 40 384 39 771 956 517 4 743 5 804 19 877 5 280 11 164 4 582 6 153 5 078 4 659 5 579 7 038 5 808 5 276 293 4 618 5 335 5 197 4 799 6 035 10 966 5 074 6 418 4 681 6 374 5 546 6 046 6 606 4 391 11 460 5 315 4 959 5 102 6 680 22 326 5 508 12 045 5 187 6 675 5 458 5 004 5 885 7 402 5 874 5 785 314 5 018 5 366 5 529 5 117 6 291 10 996 5 188 6 602 5 017 6 467 5 887 6 279 6 681 4 857 12 148 5 904 4 966 Information for the adjudicated regions is available on line. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003725 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 141 Annex A2: The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS Table A2.3 shows the response rates for students and schools, before and after replacement. • Column 1 shows the weighted participation rate of schools before replacement. This is obtained by dividing Column 2 by Column 3, multiply by 100. • Column 2 shows the weighted number of responding schools before school replacement (weighted by student enrolment). • Column 3 shows the weighted number of sampled schools before school replacement (including both responding and nonresponding schools, weighted by student enrolment). • Column 4 shows the unweighted number of responding schools before school replacement. • Column 5 shows the unweighted number of responding and non-responding schools before school replacement. • Column 6 shows the weighted participation rate of schools after replacement. This is obtained by dividing Column 7 by Column 8, multiply by 100. • Column 7 shows the weighted number of responding schools after school replacement (weighted by student enrolment). • Column 8 shows the weighted number of schools sampled after school replacement (including both responding and non-responding schools, weighted by student enrolment). • Column 9 shows the unweighted number of responding schools after school replacement. • Column 10 shows the unweighted number of responding and non-responding schools after school replacement. • Column 11 shows the weighted student participation rate after replacement. This is obtained by dividing Column 12 by Column 13, multiply by 100. • Column 12 shows the weighted number of students assessed. • Column 13 shows the weighted number of students sampled (including both students who were assessed and students who were absent on the day of the assessment). • Column 14 shows the unweighted number of students assessed. Note that any students in schools with student-response rates less than 50% were not included in these rates (both weighted and unweighted). • Column 15 shows the unweighted number of students sampled (including both students that were assessed and students who were absent on the day of the assessment). Note that any students in schools where fewer than half of the eligible students were assessed were not included in these rates (neither weighted nor unweighted). Differences between the problem-solving sample and the main PISA student sample Out of the 65 countries and economies that participated in PISA 2012, 44 also implemented the computer-based assessment (CBA) of problem solving. Of these, 12 countries and economies only assessed problem solving, while 32 also assessed mathematics and (digital) reading on computers. In all 44 countries/economies, only a random sub-sample of students who participated in the paper-based assessment (PBA) of mathematics were sampled to be administered the assessment of problem solving. However, as long as at least one student in a participating school was sampled for the computer-based assessment, all students in the PISA sample from that school received multiple imputations (plausible values) of performance in problem solving, This is similar to the procedure used to impute plausible values for minor domains in PISA (for instance, not all test booklets in 2012 included reading questions; but all students received imputed values for reading performance). Table A2.4 compares the inal samples (after school replacement) for mathematics and problem solving. • Column 1 shows the overall number of schools with valid data in the PISA 2012 database. • Column 2 shows the students with valid data in mathematics. This is the number of students with data included in the main database. All these students have imputed values for performance in mathematics, reading and science. Students are considered as participating in the assessment of mathematics if they were sampled to sit the paper-based assessment (all booklets included mathematics questions) and attended a test session. Those who only attended the questionnaire session but provided at least a description of their father’s or mother’s occupation are also regarded as participants. • Column 3 shows the number of schools with valid data in the PISA 2012 computer-based assessments database. • Column 4 shows the number of students with valid data in problem solving. This corresponds to all participating students (Column 2) within schools who were sampled for the computer-based assessments in PISA 2012 and were included in the database (Column 3). for all these students, performance in problem solving could be imputed. All these students contributed to the statistics presented in this publication (with the exception of statistics based on item-level performance). • Column 5 shows the number of students included in the database who were sampled for the assessment of problem solving. These are the students with valid data who were sampled to sit the computer-based assessment and assigned a form (the computer equivalent of a paper booklet) containing at least one cluster of problem-solving questions. • Column 6 shows the number of students who were actually assessed in problem solving. These are the students sampled for the assessment of problem solving who actually attended the computer-based assessment session and were administered the test. All these students contributed to statistics based on item-level performance in this volume. Differences between the number of students in Columns 5 and 6 can occur for several reasons: students who skipped the computer-based session; students who did not reach any of the problem-solving questions in their test form; technical problems with the computer; etc. 142 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS: Annex A2 table a2.4 [Part 1/1] Sample size for performance in mathematics and problem solving mathematics OECD Partners albania argentina brazil bulgaria colombia costa rica croatia cyprus* hong kong-china indonesia Jordan kazakhstan latvia liechtenstein lithuania macao-china malaysia montenegro Peru Qatar romania russian federation Serbia Shanghai-china Singapore chinese taipei thailand tunisia united arab Emirates uruguay viet nam number of students who were administered the assessment of problem solving (unweighted) number of students with valid data (unweighted) number of schools with valid data (unweighted) number of students with valid data (unweighted) number of students with valid data sampled for the assessment of problem solving (unweighted) (1) 775 191 287 885 221 297 341 206 311 226 230 188 204 134 183 172 1 194 191 156 42 1 471 179 177 197 184 195 231 338 902 209 411 170 507 162 (2) 14 481 4 755 8 597 21 544 6 856 5 327 7 481 4 779 8 829 4 613 5 001 5 125 4 810 3 508 5 016 5 055 31 073 6 351 5 033 5 258 33 806 4 460 4 291 4 686 4 607 5 722 4 678 5 911 25 313 4 736 11 229 4 848 12 659 4 978 (3) 775 191 287 885 221 297 341 206 311 226 230 0 204 0 183 172 208 191 156 0 0 179 0 197 184 195 231 338 368 209 0 170 170 162 (4) 14 481 4 755 8 597 21 544 6 856 5 327 7 481 4 779 8 829 4 613 5 001 0 4 810 0 5 016 5 055 5 495 6 351 5 033 0 0 4 460 0 4 686 4 607 5 722 4 678 5 911 10 175 4 736 0 4 848 4 185 4 978 (5) 5 922 1 376 2 309 5 415 1 674 3 229 2 104 1 412 3 685 1 509 1 426 0 1 355 0 1 303 1 445 1 554 3 178 1 351 0 0 2 258 0 1 463 1 256 1 631 1 589 2 179 2 866 1 337 0 2 022 1 963 1 300 (6) 5 612 1 331 2 147 4 602 1 578 3 076 1 948 1 367 3 531 1 345 1 350 0 1 300 0 1 190 1 346 1 371 3 014 1 336 0 0 1 752 0 1 240 1 227 1 446 1 465 2 065 2 709 1 258 0 1 995 1 458 1 273 204 226 839 188 352 193 163 117 148 209 233 218 211 12 216 45 164 51 240 157 178 227 153 155 172 163 239 153 458 180 162 4 743 5 908 19 204 5 282 9 073 4 602 5 008 5 078 4 670 5 622 7 038 5 808 4 306 293 4 618 5 335 5 197 4 744 6 035 10 966 5 074 5 231 4 684 5 177 5 546 6 046 6 606 4 407 11 500 5 315 4 959 0 0 241 188 352 0 163 117 148 0 0 0 0 0 0 45 164 51 0 0 0 227 153 155 172 163 0 0 458 180 0 0 0 5 506 5 282 9 073 0 5 008 5 078 4 670 0 0 0 0 0 0 5 335 5 197 4 744 0 0 0 5 231 4 684 5 177 5 546 6 046 0 0 11 500 5 315 0 0 0 1 590 2 333 2 595 0 2 016 2 630 1 367 0 0 0 0 0 0 1 577 2 072 2 101 0 0 0 1 574 1 930 1 213 1 438 1 512 0 0 3 418 2 048 0 0 0 1 463 2 145 2 307 0 1 924 2 503 1 325 0 0 0 0 0 0 1 565 1 929 1 845 0 0 0 1 543 1 777 1 203 1 394 1 484 0 0 3 262 2 013 0 number of schools with valid data (unweighted) australia austria belgium canada chile czech republic denmark Estonia finland france Germany Greece hungary iceland ireland israel italy Japan korea luxembourg mexico netherlands new Zealand norway Poland Portugal Slovak republic Slovenia Spain Sweden Switzerland turkey united kingdom united States Problem solving * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003725 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 143 Annex A2: The PISA TArgeT PoPulATIon, The PISA SAmPleS And The deFInITIon oF SchoolS In all but four of the 44 countries/economies that assessed problem solving, the school samples for CBA and PBA coincide. As a consequence, in 40 countries/economies the main student dataset, containing the results of paper-based assessments, and the CBA dataset have the same number of observations. In Brazil, Italy, Spain and the united kingdom, in contrast, the CBA school sample is smaller than the main sample. Brazil and Italy did not over-sample students for CBA to provide results at regional level. In Spain, students were over-sampled only in the Basque Country and in Catalonia, but not in the remaining adjudicated regions. In the united kingdom, only schools in England participated in the computer-based assessment of problem solving. Deinition of schools In some countries, sub-units within schools were sampled instead of schools and this may affect the estimation of the between-school variance components. In Austria, the Czech republic, germany, Hungary, Japan, romania and Slovenia, schools with more than one study programme were split into the units delivering these programmes. In the Netherlands, for schools with both lower and upper secondary programmes, schools were split into units delivering each programme level. In the flemish Community of Belgium, in the case of multi-campus schools, implantations (campuses) were sampled, whereas in the french Community, in the case of multi-campus schools, the larger administrative units were sampled. In Australia, for schools with more than one campus, the individual campuses were listed for sampling. In Argentina, Croatia and Dubai (united Arab Emirates), schools that had more than one campus had the locations listed for sampling. In Spain, the schools in the Basque region with multi-linguistic models were split into linguistic models for sampling. 144 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V TechnIcAl noTeS on AnAlySeS In ThIS volume: Annex A3 Annex A3 TechnIcAl noTeS on AnAlySeS In ThIS volume Methods and deinitions Relative performance in problem solving relative performance in problem solving is deined as the difference between a student’s actual performance in problem solving and his or her expected performance, based on performance in other domains: RPi ps = yips − E( yips yimrs ) where yips represents student i’s performance in problem solving, and (such as mathematics, reading and science). y imrs is a vector of student i’s performance in other domains A student’s (conditionally) expected performance is estimated using regression models; relative performance is therefore based on residuals from regression models. All analyses of relative performance in this volume derive residuals from parametric regression models that allow for curvilinear shapes and, when more than one domain enters the conditioning arguments, for interaction terms (secondor third-degree polynomials). However, different regression methods can be used, including non-parametric ones. figure V.2.16, for instance, graphically displays a non-parametric regression of problem-solving performance on mathematics performance. In some analyses, the regression model is calibrated only on a subsample of comparison students (e.g. on boys, when the relative performance of girls is analysed). In others, where the comparison group is less well deined and the focus is on comparisons to the national or international average, the regression model is calibrated on all students. In all cases, ive distinct regression models are estimated to compute ive plausible values of relative performance. Relative risk or increased likelihood The relative risk is a measure of the association between an antecedent factor and an outcome factor. The relative risk is simply the ratio of two risks, i.e. the risk of observing the outcome when the antecedent is present and the risk of observing the outcome when the antecedent is not present. figure A3.1 presents the notation that is used in the following. • figure A3.1 • labels used in a two-way table p11 p21 p.1 p12 p22 p.2 p1. p2. p.. n.. p. . is equal to n.. , with n. . the total number of students and p. . is therefore equal to 1, pi. , p.j respectively represent the marginal probabilities for each row and for each column. The marginal probabilities are equal to the marginal frequencies divided by the total number of students. finally, the pij represents the probabilities for each cell and are equal to the number of observations in a particular cell divided by the total number of observations. In PISA, the rows represent the antecedent factor, with the irst row for “having the antecedent” and the second row for “not having the antecedent”. The columns represent the outcome: the irst column for “having the outcome” and the second column for “not having the outcome”. The relative risk is then equal to: p p RR = ( 11 / 1. ) ( p21 / p2. ) Statistics based on multilevel models Statistics based on multilevel models include variance components (between- and within-school variance), the index of inclusion derived from these components, and regression coeficients where this has been indicated. multilevel models are generally speciied as two-level regression models (the student and school levels), with normally distributed residuals, and estimated with maximum likelihood estimation. Where the dependent variable is mathematics performance, the estimation uses ive plausible values for each student’s performance on the mathematics scale. models were estimated using mplus® software. In multilevel models, weights are used at both the student and school levels. The purpose of these weights is to account for differences in the probabilities of students being selected in the sample. Since PISA applies a two-stage sampling procedure, these differences are due to factors at both the school and the student levels. for the multilevel models, student inal weights (W_fSTuWT) were used. CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 145 Annex A3: TechnIcAl noTeS on AnAlySeS In ThIS volume Within-school-weights correspond to student inal weights, rescaled to sum up within each school to the school sample size. betweenschool weights correspond to the sum of student inal weights (W_fSTuWT) within each school. The deinition of between-school weights has changed with respect to PISA 2009. The index of inclusion is deined and estimated as: 100 * σ w2 σ + σ b2 2 w where σ w and 2 σ b2 , respectively, represent the within- and between-variance estimates. The results in multilevel models, and the between-school variance estimate in particular, depend on how schools are defined and organised within countries and by the units that were chosen for sampling purposes. for example, in some countries, some of the schools in the PISA sample were defined as administrative units (even if they spanned several geographically separate institutions, as in Italy); in others they were defined as those parts of larger educational institutions that serve 15-year-olds; in still others they were defined as physical school buildings; and in others they were defined from a management perspective (e.g. entities having a principal). The PISA 2012 Technical Report (OECD, forthcoming) and Annex A2 provide an overview of how schools were defined. In Slovenia, the primary sampling unit is deined as a group of students who follow the same study programme within a school (an educational track within a school). So in this particular case the between-school variance is actually the within-school, between-track variation. The use of stratiication variables in the selection of schools may also affect the estimate of the between-school variance, particularly if stratiication variables are associated with between-school differences. because of the manner in which students were sampled, the within-school variation includes variation between classes as well as between students. Effect sizes An effect size is a measure of the strength of the relationship between two variables. The term effect size is commonly used to refer to standardised differences. Standardising a difference is useful when a metric has no intrinsic meaning – as is the case with PISA performance scales or scale indices. Indeed, a standardised difference allows comparisons of the strength of between-group differences across measures that vary in their metric. A standardised difference is obtained by dividing the raw difference between two groups, such as boys and girls, by a measure of the variation in the underlying data. In this volume, the pooled standard deviation was used to standardise differences. The effect size between two subgroups is thus calculated as: m1 m2 σ 12,2 2 where m1 and m2, respectively, represent the mean values for the subgroups 1 and 2, and σ 1,2 represents the variance for the population pooling subgroups 1 and 2. Relative success ratios on subsets of items The relative likelihood of success on a subset of items is computed as follows. first, a country-speciic measure of success on each item is computed by converting the percentage of correct answers into the logit scale (the logarithm of odds is used instead of the percentage; odds are also referred to as success ratios, because they correspond to the number of full-credit answers over the number of no- and partial-credit answers). This success measure can also be interpreted as an item-dificulty parameter: lower success measures indicate more dificult items. next, a relative success measure for a given subset of items is derived as the difference between the average success on items in the subset and the average success on items outside of the subset. Again, this measure can also be interpreted as a relative dificulty of items in the two subsets. finally, a relative likelihood of success is derived that takes into account differences in item dificulty by subtracting the average relative success in OECD countries (i.e. the average dificulty of items) from country-speciic igures (or similarly, the relative success in a comparison group – e.g. boys – from the relative success in the focus group – e.g. girls). This difference is used as a basis for computing odds ratios (the difference of logits being the logarithm of the odds ratio). by design, each item carries the same weight in these analyses. However, the probability of success on a given item is also inluenced by its position within the test booklet. While ex ante, booklets are assigned so that they are present in equal proportions within any subsample, in practice given the inite number of students taking the test small differences remain. To control for these differences, booklet dummies are included in the model and generalised odds ratios are estimated with logistic regression. Similarly, in some analyses country- or group-speciic dummies are included for the response format to ensure that inferences about strengths and weaknesses on the items measuring the various framework aspects are not driven by the association of selected- and constructedresponse formats with speciic item families. 146 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V TechnIcAl noTeS on AnAlySeS In ThIS volume: Annex A3 Standard errors and signiicance tests The statistics in this report represent estimates of national performance based on samples of students, rather than values that could be calculated if every student in every country had answered every question. Consequently, it is important to measure the degree of uncertainty of the estimates. In PISA, each estimate has an associated degree of uncertainty, which is expressed through a standard error. The use of conidence intervals provides a way to make inferences about the population means and proportions in a manner that relects the uncertainty associated with the sample estimates. from an observed sample statistic and assuming a normal distribution, it can be inferred that the corresponding population result would lie within the conidence interval in 95 out of 100 replications of the measurement on different samples drawn from the same population. In many cases, readers are primarily interested in whether a given value in a particular country is different from a second value in the same or another country, e.g. whether girls in a country perform better than boys in the same country. In the tables and charts used in this report, differences are labelled as statistically signiicant when a difference of that size, smaller or larger, would be observed less than 5% of the time, if there were actually no difference in corresponding population values. Similarly, the risk of reporting a correlation as signiicant if there is, in fact, no correlation between two measures, is contained at 5%. Throughout the report, signiicance tests were undertaken to assess the statistical signiicance of the comparisons made. Gender differences and differences between subgroup means gender differences in student performance or other indices were tested for statistical signiicance. Positive differences indicate higher scores for boys while negative differences indicate higher scores for girls. generally, differences marked in bold in the tables in this volume are statistically signiicant at the 95% conidence level. Similarly, differences between other groups of students (e.g. native students and students with an immigrant background) were tested for statistical signiicance. The deinitions of the subgroups can in general be found in the tables and the text accompanying the analysis. All differences marked in bold in the tables presented in Annex b of this report are statistically signiicant at the 95% level. Differences between subgroup means, after accounting for other variables for many tables, subgroup comparisons were performed both on the observed difference (“before accounting for other variables”) and after accounting for other variables, such as the PISA index of economic, social and cultural status of students (ESCS). The adjusted differences were estimated using linear regression and tested for signiicance at the 95% conidence level. Signiicant differences are marked in bold. Performance differences between the top and bottom quartiles of PISA indices and scales Differences in average performance between the top and bottom quarters of the PISA indices and scales were tested for statistical signiicance. figures marked in bold indicate that performance between the top and bottom quarters of students on the respective index is statistically signiicantly different at the 95% conidence level. Change in the performance per unit of the index for many tables, the difference in student performance per unit of the index shown was calculated. figures in bold indicate that the differences are statistically signiicantly different from zero at the 95% conidence level. Relative risk or increased likelihood figures in bold in the data tables presented in Annex b of this report indicate that the relative risk is statistically signiicantly different from 1 at the 95% conidence level. To compute statistical signiicance around the value of 1 (the null hypothesis), the relative-risk statistic is assumed to follow a log-normal distribution, rather than a normal distribution, under the null hypothesis. Range of ranks To calculate the range of ranks for countries, data are simulated using the mean and standard error of the mean for each relevant country to generate a distribution of possible values. Some 10 000 simulations are implemented and, based on these values, 10 000 possible rankings for each country are produced. for each country, the counts for each rank are aggregated from largest to smallest until they equal 9 500 or more. Then the range of ranks per country is reported, including all the ranks that have been aggregated. This means that there is at least 95% conidence about the range of ranks, and it is safe to assume unimodality in this distribution of ranks. This method has been used in all cycles of PISA since 2003, including PISA 2012. The main difference between the range of ranks (e.g. figure V.2.4) and the comparison of countries’ mean performance (e.g. figure V.2.3) is that the former takes account of the multiple comparisons involved in determining ranks and the asymmetry of the distribution of rank estimates, while the latter does not. Therefore, sometimes there is a slight difference between the range of ranks and counting the number of countries above a given country, based on pairwise comparisons of the selected countries’ performance. for instance, the difference in average performance between England (united kingdom), which is listed in eleventh place in figure V.2.3, and Canada, which is listed in eighth place, is not statistically signiicant. However, because it is highly unlikely that all three countries/economies listed between eight and tenth place in reality have lower performance than England (united kingdom), the rank for England (united kingdom) CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 147 Annex A3: TechnIcAl noTeS on AnAlySeS In ThIS volume among all countries can be restricted to be, with 95% conidence, at best ninth (figure V.2.4). Since it is safe to assume that the distribution of rank estimates for each country has a single mode (unimodality), the results of range of ranks for countries should be used when examining countries’ rankings. Standard errors in statistics estimated from multilevel models for statistics based on multilevel models (such as the estimates of variance components and regression coeficients from two-level regression models) the standard errors are not estimated with the usual replication method which accounts for stratiication and sampling rates from inite populations. Instead, standard errors are “model-based”: their computation assumes that schools, and students within schools, are sampled at random (with sampling probabilities relected in school and student weights) from a theoretical, ininite population of schools and students which complies with the model’s parametric assumptions. The standard error for the estimated index of inclusion is calculated by deriving an approximate distribution for it from the (modelbased) standard errors for the variance components, using the delta-method. Differences between rankings based on proiciency scales and average percent-correct rankings PISA international results are based on a scaling of students’ item scores with an item response model (see the PISA 2012 Technical Report, OECD, forthcoming). This scaling is undertaken for a number of reasons. first, it supports the construction of described proiciency scales. Second, this approach summarises students’ responses to many items with few indices. In doing so, it ensures that the indices are comparable across students who respond to different test booklets that are composed of different subsets of items (Adams et al., 2010). The scaling of students’ scores relects the PISA approach, which consists in building internationally supported assessment frameworks and then developing items pools that sample widely from those frameworks in an agreed fashion. The average percent-correct approach used in Chapter 3 in this volume provides an alternative way of comparing country performance on the assessment. The advantage of the average percent-correct approach is that it can be easily replicated on arbitrary subsets of items. When rankings based on the percent-correct approach, using all items, are compared to rankings based on the usual scaling approach, small differences will occur for six reasons. first, the percent-correct methodology assigns an arbitrary value (typically, either 0 or 0.5) to all partial-credit answers; percent-correct igures are therefore based on a smaller set of information about students’ performance on the test than scaled results, where each partial credit value is scaled to its speciic dificulty. Second, the percent-correct methodology ignores students who did not answer any problem-solving item, despite being assigned to a problem-solving booklet and having answered, at least partially, the student questionnaire. because it is impossible to know why they did not answer problem-solving questions (e.g. a technical failure of the computer system or a deliberate absence from the test), their answers are coded as “not administered” rather than as incorrect, and treated as missing. The usual scaling approach, in contrast, corrects for possible self-selection in taking the test by imputing performance from the available information about these students, including their performance on other tests. Third, the percent-correct methodology weights all items equally, whereas in the scaling approach the items are weighted according to the number of booklets in which they were included. fourth, the percent-correct approach does not address the booklet effect that was observed in PISA. fifth, the scaling methodology transforms percentage values that are bounded at zero and 100 into the logit scale. This transformation has the effect of “stretching out” very low and very high percentages in comparison to percentages that are close to 50%. Sixth, when a problem such as a translation error affecting one item in one country is detected after the test has been administered, this item is coded as missing for all students in the country; the percent-correct rankings may therefore be based on fewer items than the scaled results. In the PISA 2012 assessment of problem solving, one item (CP018Q05) was withdrawn after the test in france, because by mistake a crucial direction to students had not been included in the national version. References Adams, R., A. Berezner and M. Jakubowski (2010), “Analysis of PISA 2006 Preferred Items ranking using the Percent-Correct Method”, OECD Education Working Papers, No. 46, OECD Publishing. http://dx.doi.org/10.1787/5km4psmntkq5-e OECD (forthcoming), PISA 2012 Technical Report, PISA, OECD Publishing. 148 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V QuAlITy ASSurAnce: Annex A4 Annex A4 QuAlITy ASSurAnce Quality assurance procedures were implemented in all parts of PISA 2012, as was done for all previous PISA surveys. The consistent quality and linguistic equivalence of the PISA 2012 assessment instruments were facilitated by providing countries with equivalent source versions of the assessment instruments in English and french and requiring countries (other than those assessing students in English and french) to prepare and consolidate two independent translations using both source versions. Precise translation and adaptation guidelines were supplied, also including instructions for selecting and training the translators. for each country, the translation and format of the assessment instruments (including test materials, marking guides, questionnaires and manuals) were veriied by expert translators appointed by the PISA Consortium before they were used in the PISA 2012 ield trial and main study. These translators’ mother tongue was the language of instruction in the country concerned and they were knowledgeable about education systems. for further information on the PISA translation procedures, see the PISA 2012 Technical Report (OECD, forthcoming). The survey was implemented through standardised procedures. The PISA Consortium provided comprehensive manuals that explained the implementation of the survey, including precise instructions for the work of School Co-ordinators and scripts for Test Administrators to use during the assessment sessions. Proposed adaptations to survey procedures, or proposed modiications to the assessment session script, were submitted to the PISA Consortium for approval prior to veriication. The PISA Consortium then veriied the national translation and adaptation of these manuals. To establish the credibility of PISA as valid and unbiased and to encourage uniformity in administering the assessment sessions, Test Administrators in participating countries were selected using the following criteria: it was required that the Test Administrator not be the mathematics, reading or science instructor of any students in the sessions he or she would administer for PISA; it was recommended that the Test Administrator not be a member of the staff of any school where he or she would administer for PISA; and it was considered preferable that the Test Administrator not be a member of the staff of any school in the PISA sample. Participating countries organised an in-person training session for Test Administrators. Participating countries and economies were required to ensure that: Test Administrators worked with the School Co-ordinator to prepare the assessment session, including updating student tracking forms and identifying excluded students; no extra time was given for the cognitive items (while it was permissible to give extra time for the student questionnaire); no instrument was administered before the two one-hour parts of the cognitive session; Test Administrators recorded the student participation status on the student tracking forms and illed in a Session report form; no cognitive instrument was permitted to be photocopied; no cognitive instrument could be viewed by school staff before the assessment session; and Test Administrators returned the material to the national centre immediately after the assessment sessions. national Project managers were encouraged to organise a follow-up session when more than 15% of the PISA sample was not able to attend the original assessment session. national Quality monitors from the PISA Consortium visited all national centres to review data-collection procedures. finally, School Quality monitors from the PISA Consortium visited a sample of seven schools during the assessment. for further information on the ield operations, see the PISA 2012 Technical Report (OECD, forthcoming). marking procedures were designed to ensure consistent and accurate application of the marking guides outlined in the PISA Operations manuals. national Project managers were required to submit proposed modiications to these procedures to the Consortium for approval. reliability studies to analyse the consistency of marking were implemented. Software specially designed for PISA facilitated data entry, detected common errors during data entry, and facilitated the process of data cleaning. Training sessions familiarised national Project managers with these procedures. for a description of the quality assurance procedures applied in PISA and in the results, see the PISA 2012 Technical Report (OECD, forthcoming). The results of adjudication showed that the PISA Technical Standards were fully met in all countries and economies that participated in PISA 2012, with the exception of Albania. Albania submitted parental occupation data that were incomplete and appeared inaccurate, since there was over-use of a narrow range of occupations. It was not possible to resolve these issues during the course of data cleaning, and as a result neither parental occupation data nor any indices which depend on this data are included in the international dataset. results for Albania are omitted from any analyses which depend on these indices. CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 149 Annex A5: The Problem-SolvIng ASSeSSmenT deSIgn Annex A5 The Problem-SolvIng ASSeSSmenT deSIgn How the PISA 2012 assessments of problem-solving was designed The development of the PISA 2012 problem-solving tasks was co-ordinated by an international consortium of educational research institutions contracted by the OECD, under the guidance of a group of problem-solving experts from participating countries (members of the problem solving expert group are listed in Annex C of this Volume). Participating countries contributed stimulus material and questions, which were reviewed, tried out and reined iteratively over the three years leading up to the administration of the assessment in 2012. The development process involved provisions for several rounds of commentary from participating countries, as well as small-scale piloting and a formal ield trial in which samples of 15-year-olds (about 1 000 students) from participating countries took part. The problem-solving expert group recommended the inal selection of tasks, which included material submitted by participating countries. The selection was made with regard to both their technical quality, assessed on the basis of their performance in the ield trial, and their cultural appropriateness and interest level for 15-year-olds, as judged by the participating countries. Another essential criterion for selecting the set of material as a whole was its it to the framework described in Chapter 1 of this volume, in order to maintain the balance across various aspect categories. finally, it was carefully ensured that the set of questions covered a range of dificulty, allowing good measurement and description of the problem-solving competence of all 15-year-old students, from the least proicient to the highly able. forty-two problem-solving questions arranged in 16 units were used in PISA 2012, but each student in the sample only saw a fraction of the total pool because different sets of questions were given to different students. The problem-solving questions selected for inclusion in PISA 2012 were organised into four 20-minutes clusters. In countries that also assessed mathematics and reading on computers, computer-based mathematics and digital reading questions were similarly arranged in 20-minutes clusters, and assembled together with problem-solving clusters to form test forms (the computer equivalent of paper booklets). In all cases, the total time allocated to computer-based tests was 40 minutes. In countries that assessed only problem-solving on computers, the four clusters of problem-solving units (CP1-CP4) were rotated so that each cluster appeared twice in each of the two possible positions in the form and every cluster formed two pairs with two other clusters. Eight test forms were built according to the scheme illustrated in figure A5.1: According to this scheme, each problem-solving item was administered to about one half of all students assessed in problem solving (see Table A2.4). In those countries that assessed problem solving, mathematics and reading on computers, the four clusters of problem-solving units, the four clusters of mathematics units (CM1-CM4) and the two clusters of reading units (Cr1, Cr2) were combined into 24 test forms as illustrated in figure A5.2. One form was chosen at random for administration to each student. • figure A5.1 • • figure A5.2 • PISA 2012 computer-based test design: Problem solving only PISA 2012 computer-based test design: Problem solving, mathematics and reading form id 31 32 33 34 35 36 37 38 150 © OECD 2014 cluster CP1 CP2 CP3 CP4 CP2 CP3 CP4 CP1 form id CP2 CP3 CP4 CP1 CP1 CP2 CP3 CP4 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 cluster CP1 Cr1 CM3 CP3 Cr2 CM1 Cr2 CM2 CP3 CM4 CP1 Cr1 CM1 CP4 Cr1 CP2 Cr2 CM2 CP2 CM4 Cr2 CM3 Cr1 CP4 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V CP2 Cr2 CM4 Cr1 CM2 CP4 Cr1 CM1 CP4 Cr2 CM3 CP2 CM3 CP1 Cr2 CM4 CP3 Cr1 CP3 CM2 Cr1 CP1 CM1 Cr2 The Problem-SolvIng ASSeSSmenT deSIgn: Annex A5 This scheme ensured that every cluster appeared twice in each position for problem solving and computer-based mathematics and four times for digital reading. Moreover, every cluster appeared twice with clusters from a different domain – once in the irst and once in the second position within the form. Each of the three domains got the same number of appearances within the 24 forms and therefore an equal proportion of the student sample was assessed in each domain. According to this scheme, each problem-solving item was administered to about one third of all students assessed in problem solving (see Table A2.4), or one sixth of all students assessed on computer. This design made it possible to construct a single scale of problem-solving proiciency, in which each question is associated with a particular point on the scale that indicates its dificulty, whereby each student’s performance is associated with a particular point on the same scale that indicates his or her estimated proiciency. A description of the modelling technique used to construct this scale can be found in the PISA 2012 Technical Report (OECD, forthcoming). References OECD (forthcoming), PISA 2012 Technical Report, PISA, OECD Publishing. CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 151 Annex A6: TechnIcAl noTe on brAzIl Annex A6 TechnIcAl noTe on brAzIl In 2006, the education system in Brazil was revised to include one more year at the beginning of primary school, with the compulsory school age being lowered from seven to six years old. This change has been implemented in stages and will be completed in 2016. At the time the PISA 2012 survey took place, many of the 15-year-olds in grade 7 had started their education under the previous system. They were therefore equivalent to grade 6 students in the previous system. Since students below grade 7 are not eligible for participation in PISA, the grade 7 students in the sample were not included in the database. Brazil also has many rural “multigrade” schools where it is dificult to identify the exact grade of each student, so not possible to identify students who are at least in grade 7. The results for brazil have therefore been analysed both with and without these rural schools. The results reported in the main chapters of this report are those of the brazilian sample without the rural schools, while this annex gives the results for brazil with the rural schools included. table a6.1 [Part 1/1] Percentage of brazilian students at each proiciency level on the problem-solving scale Percentage of students at each level below level 1 (below 358.49 score points) Problem-solving scale All level 1 (from 358.49 to less than 423.42 score points) level 2 (from 423.42 to less than 488.35 score points) level 3 (from 488.35 to less than 553.28 score points) level 4 (from 553.28 to less than 618.21 score points) level 5 (from 618.21 to less than 683.14 score points) level 6 (above 683.14 score points) % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. 23.5 (1.6) 25.5 (1.4) 26.1 (1.3) 16.8 (1.4) 6.3 (0.8) 1.4 (0.3) 0.4 (0.1) boys 20.8 (1.8) 23.8 (1.5) 25.9 (1.5) 18.3 (1.7) 8.5 (1.2) 2.0 (0.4) 0.6 (0.3) girls 26.0 (1.9) 27.1 (1.9) 26.2 (1.5) 15.3 (1.7) 4.3 (0.7) 0.9 (0.3) 0.1 (0.1) 1 2 http://dx.doi.org/10.1787/888933003744 table a6.2 [Part 1/1] mean score, variation and gender differences in student performance in brazil all students Gender differences Standard mean score deviation Problem-solving scale mean S.E. S.d. 425 (4.5) 92 boys Girls Percentiles difference (b - G) 5th 10th 25th 75th 90th 95th mean mean Score S.E. score S.E. score S.E. dif. S.E. Score S.E. Score S.E. Score S.E. Score S.E. Score S.E. Score S.E. Score S.E. (2.3) 436 (5.2) 415 (4.4) 21 (3.3) 273 (5.8) 307 (4.7) 363 (4.8) 426 (5.2) 487 (6.1) 543 (5.7) 573 (5.7) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1 2 http://dx.doi.org/10.1787/888933003744 152 50th (median) © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V Annex B PiSa 2012 data All tables in Annex B are available on line annex b1: results for countries and economies http://dx.doi.org/10.1787/888933003668 http://dx.doi.org/10.1787/888933003687 http://dx.doi.org/10.1787/888933003706 annex b2: results for regions within countries http://dx.doi.org/10.1787/888933003763 annex b3: List of tables available on line The reader should note that there are gaps in the numbering of tables because some tables appear on line only and are not included in this publication. notes regarding cyprus Note by Turkey: The information in this document with reference to “Cyprus” relates to the southern part of the Island. There is no single authority representing both Turkish and greek Cypriot people on the Island. Turkey recognises the Turkish republic of Northern Cyprus (TrNC). until a lasting and equitable solution is found within the context of the united Nations, Turkey shall preserve its position concerning the “Cyprus issue”. Note by all the European Union Member States of the OECD and the European Union: The republic of Cyprus is recognised by all members of the united Nations with the exception of Turkey. The information in this document relates to the area under the effective control of the government of the republic of Cyprus. a note regarding israel The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is without prejudice to the status of the golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law. CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 153 Annex b1: reSulTS For counTrIeS And economIeS Annex b1 reSulTS For counTrIeS And economIeS table v.2.1 [Part 1/2] Percentage of students at each proiciency level in problem solving Percentage of students at each level OECD below level 1 (below 358.49 score points) level 2 (from 423.42 to less than 488.35 score points) level 3 (from 488.35 to less than 553.28 score points) level 4 (from 553.28 to less than 618.21 score points) level 6 (above 683.14 score points) % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. 5.0 (0.3) 10.5 (0.5) 19.4 (0.5) 25.8 (0.7) 22.6 (0.5) 12.3 (0.5) 4.4 (0.3) austria 6.5 (0.9) 11.9 (0.8) 21.8 (1.1) 26.9 (1.2) 21.9 (1.0) 9.0 (0.8) 2.0 (0.4) belgium 9.2 (0.6) 11.6 (0.6) 18.3 (0.7) 24.5 (0.6) 22.0 (0.7) 11.4 (0.7) 3.0 (0.3) canada 5.1 (0.4) 9.6 (0.4) 19.0 (0.6) 25.8 (0.7) 22.9 (0.6) 12.4 (0.6) 5.1 (0.4) 15.1 (1.3) 23.1 (1.1) 28.6 (1.0) 22.2 (1.0) 8.8 (0.7) 1.9 (0.3) 0.2 (0.1) czech republic 6.5 (0.7) 11.9 (0.9) 20.7 (1.0) 27.2 (0.9) 21.8 (0.9) 9.5 (0.7) 2.4 (0.3) denmark 7.3 (0.7) 13.1 (0.7) 24.1 (0.8) 27.8 (0.9) 19.0 (1.1) 7.2 (0.7) 1.6 (0.3) Estonia 4.0 (0.5) 11.1 (0.8) 21.8 (0.7) 29.2 (1.0) 22.2 (0.8) 9.5 (0.7) 2.2 (0.3) finland 4.5 (0.4) 9.9 (0.5) 20.0 (0.9) 27.1 (1.1) 23.5 (0.8) 11.4 (0.6) 3.6 (0.5) france 6.6 (0.9) 9.8 (0.7) 20.5 (1.0) 28.4 (1.1) 22.6 (0.9) 9.9 (0.7) 2.1 (0.3) Germany 7.5 (0.8) 11.8 (0.9) 20.3 (0.9) 25.6 (1.0) 22.0 (1.0) 10.1 (1.0) 2.7 (0.4) hungary 17.2 (1.3) 17.8 (0.9) 23.9 (1.2) 22.4 (0.9) 13.0 (1.0) 4.6 (0.7) 1.0 (0.2) 7.0 (0.8) 13.3 (0.9) 23.8 (0.8) 27.8 (0.9) 18.8 (0.8) 7.3 (0.6) 2.1 (0.3) israel 21.9 (1.4) 17.0 (0.9) 20.1 (0.8) 18.5 (0.9) 13.7 (0.9) 6.7 (0.8) 2.1 (0.4) italy 5.2 (0.7) 11.2 (1.1) 22.5 (1.0) 28.0 (1.1) 22.3 (1.1) 8.9 (0.9) 1.8 (0.3) Japan 1.8 (0.4) 5.3 (0.6) 14.6 (0.9) 26.9 (1.1) 29.2 (1.0) 16.9 (1.0) 5.3 (0.7) korea 2.1 (0.3) 4.8 (0.6) 12.9 (0.9) 23.7 (1.0) 28.8 (0.9) 20.0 (1.2) 7.6 (0.9) netherlands 7.4 (1.0) 11.2 (1.0) 19.9 (1.2) 26.0 (1.3) 22.0 (1.2) 10.9 (1.0) 2.7 (0.5) norway 8.1 (0.7) 13.2 (0.7) 21.5 (0.9) 24.7 (0.8) 19.4 (0.8) 9.7 (0.7) 3.4 (0.4) Poland 10.0 (1.1) 15.7 (1.0) 25.7 (0.9) 26.0 (1.0) 15.7 (1.0) 5.8 (0.7) 1.1 (0.2) 6.5 (0.6) 14.1 (1.0) 25.5 (0.9) 28.1 (1.0) 18.4 (0.9) 6.2 (0.6) 1.2 (0.3) Slovak republic 10.7 (1.1) 15.4 (1.1) 24.3 (1.0) 25.6 (1.3) 16.2 (1.2) 6.3 (0.6) 1.6 (0.5) Slovenia 11.4 (0.6) 17.1 (1.0) 25.4 (1.2) 23.7 (0.8) 15.8 (0.8) 5.8 (0.5) 0.9 (0.2) Spain 13.1 (1.2) 15.3 (0.8) 23.6 (0.9) 24.2 (1.0) 15.9 (0.8) 6.2 (0.6) 1.6 (0.3) Sweden 8.8 (0.7) 14.6 (0.8) 23.9 (0.9) 26.3 (0.8) 17.6 (0.7) 7.0 (0.5) 1.8 (0.3) turkey 11.0 (1.1) 24.8 (1.3) 31.4 (1.4) 21.2 (1.2) 9.4 (1.1) 2.0 (0.5) 0.2 (0.1) England (united kingdom) 5.5 (0.8) 10.8 (0.8) 20.2 (1.3) 26.5 (0.9) 22.7 (1.1) 10.9 (0.8) 3.3 (0.6) united States 5.7 (0.8) 12.5 (0.9) 22.8 (1.0) 27.0 (1.0) 20.4 (0.9) 8.9 (0.7) 2.7 (0.5) oEcd average 8.2 (0.2) 13.2 (0.2) 22.0 (0.2) 25.6 (0.2) 19.6 (0.2) 8.9 (0.1) 2.5 (0.1) brazil 21.9 (1.6) 25.4 (1.4) 26.9 (1.3) 17.4 (1.4) 6.6 (0.8) 1.5 (0.3) 0.4 (0.2) bulgaria 33.3 (1.9) 23.3 (1.1) 22.1 (1.0) 14.1 (0.8) 5.6 (0.7) 1.4 (0.3) 0.2 (0.1) colombia 33.2 (1.7) 28.3 (1.1) 22.2 (0.9) 11.3 (0.8) 3.9 (0.5) 0.9 (0.2) 0.2 (0.1) croatia 12.0 (1.0) 20.2 (1.0) 26.8 (1.2) 22.9 (1.1) 13.2 (1.1) 4.0 (0.6) 0.8 (0.2) cyprus* 19.6 (0.6) 20.9 (0.6) 25.5 (0.8) 20.4 (0.9) 10.1 (0.6) 3.0 (0.3) 0.5 (0.2) hong kong-china 3.3 (0.5) 7.1 (0.7) 16.3 (1.0) 27.4 (1.4) 26.5 (1.0) 14.2 (1.1) 5.1 (0.6) macao-china 1.6 (0.2) 6.0 (0.4) 17.5 (0.6) 29.5 (0.8) 28.9 (0.9) 13.8 (0.6) 2.8 (0.3) malaysia 22.7 (1.5) 27.8 (1.2) 27.8 (1.2) 15.7 (0.9) 5.2 (0.6) 0.8 (0.2) 0.1 (0.0) montenegro 30.0 (0.8) 26.8 (0.8) 23.9 (1.0) 13.8 (0.7) 4.6 (0.4) 0.7 (0.2) 0.1 (0.1) 6.8 (0.7) 15.4 (1.1) 27.0 (0.9) 27.9 (1.2) 15.7 (0.9) 5.9 (0.7) 1.4 (0.3) 10.3 (1.0) 18.3 (0.8) 26.7 (1.4) 25.8 (1.1) 14.3 (0.8) 4.1 (0.4) 0.6 (0.2) Shanghai-china 3.1 (0.5) 7.5 (0.6) 17.5 (0.8) 27.4 (1.1) 26.2 (1.0) 14.1 (0.9) 4.1 (0.6) Singapore 2.0 (0.2) 6.0 (0.4) 13.8 (0.6) 21.9 (0.7) 27.0 (1.0) 19.7 (0.7) 9.6 (0.4) chinese taipei 3.4 (0.6) 8.2 (0.6) 17.8 (0.8) 26.3 (1.0) 25.9 (1.0) 14.6 (0.7) 3.8 (0.4) united arab Emirates 30.3 (1.2) 24.6 (0.8) 22.0 (0.7) 14.2 (0.6) 6.4 (0.4) 2.1 (0.2) 0.4 (0.1) uruguay 32.4 (1.6) 25.6 (1.0) 22.4 (1.0) 13.2 (0.7) 5.3 (0.5) 1.1 (0.2) 0.1 (0.1) ireland Portugal russian federation Serbia * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 154 level 5 (from 618.21 to less than 683.14 score points) australia chile Partners level 1 (from 358.49 to less than 423.42 score points) © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.2.1 [Part 2/2] Percentage of students at each proiciency level in problem solving Percentage of students at or above each proiciency level Partners OECD level 1 or above (above 358.49 score points) level 2 or above (above 423.42 score points) level 3 or above (above 488.35 score points) level 4 or above (above 553.28 score points) level 5 or above (above 618.21 score points) level 6 (above 683.14 score points) % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. australia 95.0 (0.3) 84.5 (0.6) 65.1 (0.8) 39.3 (0.8) 16.7 (0.6) 4.4 (0.3) austria 93.5 (0.9) 81.6 (1.3) 59.7 (1.6) 32.9 (1.5) 10.9 (1.0) 2.0 (0.4) belgium 90.8 (0.6) 79.2 (0.9) 60.9 (1.0) 36.4 (1.0) 14.4 (0.8) 3.0 (0.3) canada 94.9 (0.4) 85.3 (0.7) 66.3 (0.9) 40.5 (1.0) 17.5 (0.8) 5.1 (0.4) chile 84.9 (1.3) 61.7 (1.8) 33.1 (1.6) 10.9 (0.9) 2.1 (0.3) 0.2 (0.1) czech republic 93.5 (0.7) 81.6 (1.1) 60.9 (1.5) 33.7 (1.3) 11.9 (0.8) 2.4 (0.3) denmark 92.7 (0.7) 79.6 (1.1) 55.6 (1.3) 27.7 (1.2) 8.7 (0.8) 1.6 (0.3) Estonia 96.0 (0.5) 84.9 (1.0) 63.1 (1.2) 34.0 (1.1) 11.8 (0.8) 2.2 (0.3) finland 95.5 (0.4) 85.7 (0.7) 65.6 (1.1) 38.5 (1.1) 15.0 (0.8) 3.6 (0.5) france 93.4 (0.9) 83.5 (1.1) 63.1 (1.3) 34.6 (1.4) 12.0 (0.9) 2.1 (0.3) Germany 92.5 (0.8) 80.8 (1.4) 60.5 (1.5) 34.8 (1.4) 12.8 (1.1) 2.7 (0.4) hungary 82.8 (1.3) 65.0 (1.5) 41.1 (1.6) 18.6 (1.4) 5.6 (0.8) 1.0 (0.2) ireland 93.0 (0.8) 79.7 (1.1) 55.9 (1.4) 28.1 (1.2) 9.4 (0.7) 2.1 (0.3) israel 78.1 (1.4) 61.1 (1.8) 41.0 (1.9) 22.5 (1.6) 8.8 (1.0) 2.1 (0.4) italy 94.8 (0.7) 83.6 (1.5) 61.1 (1.9) 33.1 (1.8) 10.8 (1.1) 1.8 (0.3) Japan 98.2 (0.4) 92.9 (0.8) 78.3 (1.3) 51.5 (1.6) 22.3 (1.2) 5.3 (0.7) korea 97.9 (0.3) 93.1 (0.8) 80.2 (1.5) 56.5 (2.0) 27.6 (1.7) 7.6 (0.9) netherlands 92.6 (1.0) 81.5 (1.5) 61.6 (1.9) 35.6 (2.0) 13.6 (1.2) 2.7 (0.5) norway 91.9 (0.7) 78.7 (1.1) 57.2 (1.3) 32.5 (1.3) 13.1 (0.9) 3.4 (0.4) Poland 90.0 (1.1) 74.3 (1.7) 48.5 (1.9) 22.5 (1.5) 6.9 (0.8) 1.1 (0.2) Portugal 93.5 (0.6) 79.4 (1.3) 54.0 (1.8) 25.8 (1.4) 7.4 (0.8) 1.2 (0.3) Slovak republic 89.3 (1.1) 73.9 (1.6) 49.7 (1.6) 24.0 (1.4) 7.8 (0.9) 1.6 (0.5) Slovenia 88.6 (0.6) 71.5 (1.0) 46.1 (0.9) 22.4 (0.7) 6.6 (0.5) 0.9 (0.2) Spain 86.9 (1.2) 71.5 (1.4) 48.0 (1.5) 23.7 (1.3) 7.8 (0.7) 1.6 (0.3) Sweden 91.2 (0.7) 76.5 (1.1) 52.6 (1.3) 26.3 (1.0) 8.8 (0.6) 1.8 (0.3) turkey 89.0 (1.1) 64.2 (1.9) 32.8 (2.2) 11.6 (1.5) 2.2 (0.5) 0.2 (0.1) England (united kingdom) 94.5 (0.8) 83.6 (1.3) 63.5 (1.8) 37.0 (1.6) 14.3 (1.1) 3.3 (0.6) united States 94.3 (0.8) 81.8 (1.3) 59.0 (1.8) 32.0 (1.5) 11.6 (1.0) 2.7 (0.5) oEcd average 91.8 (0.2) 78.6 (0.2) 56.6 (0.3) 31.0 (0.3) 11.4 (0.2) 2.5 (0.1) brazil 78.1 (1.6) 52.7 (2.3) 25.8 (2.2) 8.4 (1.0) 1.8 (0.4) 0.4 (0.2) bulgaria 66.7 (1.9) 43.3 (1.9) 21.3 (1.5) 7.2 (1.0) 1.6 (0.4) 0.2 (0.1) colombia 66.8 (1.7) 38.5 (1.6) 16.4 (1.2) 5.0 (0.6) 1.2 (0.3) 0.2 (0.1) croatia 88.0 (1.0) 67.7 (1.6) 40.9 (1.9) 18.0 (1.5) 4.7 (0.7) 0.8 (0.2) cyprus* 80.4 (0.6) 59.6 (0.8) 34.1 (0.9) 13.7 (0.6) 3.6 (0.3) 0.5 (0.2) hong kong-china 96.7 (0.5) 89.6 (1.1) 73.2 (1.7) 45.8 (1.8) 19.3 (1.3) 5.1 (0.6) macao-china 98.4 (0.2) 92.5 (0.5) 75.0 (0.6) 45.5 (0.7) 16.6 (0.6) 2.8 (0.3) malaysia 77.3 (1.5) 49.5 (1.8) 21.8 (1.4) 6.1 (0.8) 0.9 (0.2) 0.1 (0.0) montenegro 70.0 (0.8) 43.2 (0.9) 19.3 (0.7) 5.5 (0.4) 0.8 (0.2) 0.1 (0.1) russian federation 93.2 (0.7) 77.9 (1.5) 50.9 (1.5) 23.0 (1.4) 7.3 (0.9) 1.4 (0.3) Serbia 89.7 (1.0) 71.5 (1.5) 44.8 (1.6) 19.0 (1.0) 4.7 (0.4) 0.6 (0.2) Shanghai-china 96.9 (0.5) 89.4 (0.9) 71.9 (1.4) 44.4 (1.6) 18.3 (1.3) 4.1 (0.6) Singapore 98.0 (0.2) 92.0 (0.4) 78.2 (0.6) 56.3 (0.8) 29.3 (0.8) 9.6 (0.4) chinese taipei 96.6 (0.6) 88.4 (0.9) 70.5 (1.3) 44.2 (1.3) 18.3 (0.9) 3.8 (0.4) united arab Emirates 69.7 (1.2) 45.2 (1.1) 23.2 (0.9) 9.0 (0.5) 2.5 (0.2) 0.4 (0.1) uruguay 67.6 (1.6) 42.1 (1.5) 19.7 (1.1) 6.5 (0.6) 1.2 (0.2) 0.1 (0.1) * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 155 Annex b1: reSulTS For counTrIeS And economIeS table v.2.2 [Part 1/2] mean score and variation in student performance in problem solving Percentiles Partners OECD mean score Standard deviation 5th 10th 25th 50th (median) 90th 95th S.E. S.d. S.E. Score S.E. Score S.E. Score S.E. Score S.E. Score S.E. Score S.E. Score S.E. australia 523 (1.9) 97 (1.0) 358 (3.5) 396 (2.7) 459 (2.4) 526 (2.3) 591 (2.2) 646 (2.3) 677 (2.8) austria 506 (3.6) 94 (2.9) 345 (8.7) 384 (6.8) 446 (4.6) 511 (3.8) 572 (3.7) 623 (4.4) 650 (4.9) belgium 508 (2.5) 106 (1.8) 317 (6.8) 364 (4.8) 441 (3.4) 518 (2.7) 583 (2.6) 637 (2.5) 665 (3.3) canada 526 (2.4) 100 (1.7) 357 (4.3) 398 (3.8) 462 (3.1) 530 (2.5) 594 (2.8) 649 (3.3) 684 (4.4) chile 448 (3.7) 86 (1.7) 304 (5.7) 337 (5.5) 390 (4.8) 450 (3.8) 507 (3.5) 557 (4.2) 587 (4.0) czech republic 509 (3.1) 95 (2.0) 344 (6.6) 384 (5.7) 447 (4.5) 515 (3.7) 575 (2.9) 626 (4.0) 656 (3.8) denmark 497 (2.9) 92 (1.9) 339 (5.7) 377 (5.2) 438 (3.8) 500 (3.3) 560 (3.3) 611 (4.5) 641 (4.9) Estonia 515 (2.5) 88 (1.5) 368 (4.2) 400 (4.6) 458 (3.4) 517 (2.8) 576 (3.1) 626 (3.7) 654 (4.0) finland 523 (2.3) 93 (1.2) 364 (4.8) 401 (3.1) 462 (3.5) 526 (2.6) 587 (3.1) 640 (3.6) 671 (3.9) france 511 (3.4) 96 (4.1) 340 (10.5) 387 (6.8) 455 (4.1) 518 (3.4) 577 (3.5) 626 (3.8) 653 (4.8) Germany 509 (3.6) 99 (2.5) 335 (7.0) 377 (6.9) 444 (5.3) 516 (3.6) 579 (4.0) 629 (4.3) 659 (5.8) hungary 459 (4.0) 104 (2.7) 277 (8.4) 319 (8.8) 391 (6.1) 465 (4.4) 532 (5.4) 591 (5.5) 622 (5.8) ireland 498 (3.2) 93 (2.0) 340 (6.5) 378 (5.0) 438 (4.0) 501 (3.1) 562 (3.5) 615 (3.8) 647 (4.6) israel 454 (5.5) 123 (3.2) 242 (10.6) 291 (7.8) 372 (6.2) 460 (6.4) 543 (6.2) 611 (6.7) 647 (7.5) italy 510 (4.0) 91 (2.1) 356 (7.2) 394 (5.8) 451 (5.2) 514 (4.9) 572 (4.5) 621 (4.6) 649 (5.5) Japan 552 (3.1) 85 (1.9) 405 (6.5) 441 (5.5) 498 (3.8) 556 (3.4) 610 (3.4) 658 (3.7) 685 (4.4) korea 561 (4.3) 91 (1.8) 406 (6.6) 443 (5.9) 505 (5.1) 568 (4.5) 625 (4.6) 672 (4.4) 698 (5.1) netherlands 511 (4.4) 99 (3.0) 336 (8.6) 378 (8.5) 448 (5.9) 517 (4.9) 581 (4.8) 633 (4.8) 662 (5.1) norway 503 (3.3) 103 (1.9) 328 (6.7) 370 (4.9) 436 (3.9) 507 (3.5) 574 (3.8) 633 (4.3) 665 (6.0) Poland 481 (4.4) 96 (3.4) 318 (8.9) 358 (6.3) 421 (5.4) 485 (4.3) 546 (4.6) 600 (4.8) 632 (6.0) Portugal 494 (3.6) 88 (1.6) 345 (5.5) 381 (4.3) 436 (4.2) 497 (4.3) 555 (3.7) 604 (4.2) 633 (5.4) Slovak republic 483 (3.6) 98 (2.7) 314 (7.1) 354 (6.2) 420 (4.8) 487 (3.9) 550 (4.2) 606 (5.2) 639 (6.9) Slovenia 476 (1.5) 97 (1.3) 310 (5.4) 350 (3.8) 413 (3.0) 479 (2.4) 545 (2.3) 599 (2.8) 628 (3.7) Spain 477 (4.1) 104 (2.9) 292 (10.4) 338 (7.8) 411 (5.3) 483 (3.8) 549 (3.9) 605 (4.3) 638 (5.0) Sweden 491 (2.9) 96 (1.8) 328 (7.6) 365 (4.0) 428 (3.7) 494 (3.2) 557 (2.9) 612 (3.7) 643 (4.4) turkey 454 (4.0) 79 (2.2) 328 (4.5) 354 (4.3) 399 (4.0) 451 (4.3) 508 (5.7) 560 (6.8) 590 (8.0) England (united kingdom) 517 (4.2) 97 (2.4) 352 (9.2) 391 (6.0) 455 (5.7) 522 (4.8) 584 (4.1) 636 (4.5) 667 (5.0) united States 508 (3.9) 93 (2.3) 352 (7.1) 388 (6.0) 446 (4.9) 510 (4.2) 571 (4.1) 626 (4.4) 658 (5.3) oEcd average 500 (0.7) 96 (0.4) 336 (1.4) 375 (1.1) 438 (0.9) 504 (0.7) 567 (0.7) 620 (0.8) 650 (1.0) brazil 428 (4.7) 92 (2.4) 276 (7.1) 311 (5.7) 368 (5.5) 429 (5.2) 490 (6.3) 545 (5.6) 575 (5.6) bulgaria 402 (5.1) 107 (3.5) 220 (10.2) 263 (8.6) 331 (6.1) 405 (5.5) 476 (5.3) 535 (7.1) 571 (7.6) colombia 399 (3.5) 92 (2.0) 253 (5.4) 284 (4.9) 337 (4.3) 397 (3.7) 459 (4.1) 517 (5.2) 553 (5.6) croatia 466 (3.9) 92 (2.0) 314 (5.6) 349 (4.9) 404 (4.0) 465 (4.2) 530 (4.6) 585 (5.1) 616 (6.2) cyprus* 445 (1.4) 99 (1.0) 278 (4.3) 315 (2.8) 378 (2.4) 447 (1.8) 513 (2.7) 571 (2.8) 604 (3.5) hong kong-china 540 (3.9) 92 (2.2) 379 (6.7) 421 (6.7) 483 (5.6) 544 (4.2) 601 (3.7) 654 (4.1) 684 (4.9) macao-china 540 (1.0) 79 (0.8) 405 (3.3) 437 (3.0) 488 (1.5) 544 (1.7) 595 (1.6) 640 (2.1) 664 (2.2) malaysia 422 (3.5) 84 (2.0) 287 (4.7) 315 (4.5) 364 (4.2) 422 (4.1) 479 (4.1) 531 (5.0) 561 (6.0) montenegro 407 (1.2) 92 (1.1) 256 (4.3) 289 (3.1) 344 (2.5) 407 (2.2) 470 (2.2) 526 (3.8) 556 (3.4) russian federation 489 (3.4) 88 (2.0) 345 (4.7) 377 (4.8) 431 (4.0) 490 (3.5) 547 (4.1) 602 (6.1) 635 (5.9) Serbia 473 (3.1) 89 (1.9) 322 (6.4) 357 (6.1) 414 (4.3) 476 (3.8) 535 (3.4) 586 (3.4) 616 (3.4) Shanghai-china 536 (3.3) 90 (2.2) 381 (7.0) 419 (5.7) 479 (3.9) 541 (3.5) 599 (3.9) 648 (4.7) 676 (4.9) Singapore 562 (1.2) 95 (1.0) 398 (3.0) 436 (2.9) 500 (2.0) 568 (2.1) 629 (1.9) 681 (2.1) 710 (3.4) chinese taipei 534 (2.9) 91 (1.9) 377 (6.7) 414 (5.1) 475 (4.1) 540 (3.3) 601 (2.9) 646 (3.2) 674 (3.2) united arab Emirates 411 (2.8) 106 (1.8) 237 (5.9) 277 (5.3) 342 (3.6) 411 (2.9) 482 (3.1) 546 (3.3) 584 (3.8) uruguay 403 (3.5) 97 (2.0) 244 (5.9) 279 (5.1) 337 (4.7) 403 (3.9) 470 (3.9) 530 (4.3) 566 (6.0) * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 156 75th mean © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.2.2 [Part 2/2] mean score and variation in student performance in problem solving range of performance Partners OECD inter-quartile range (75th minus 25th percentile) inter-decile range (90th minus 10th percentile) top range (90th minus 50th percentile) bottom range (50th minus 10th percentile) range S.E. range S.E. range S.E. range S.E. australia 132 (2.1) 251 (3.0) 121 (2.2) 130 (2.8) austria 126 (4.5) 239 (7.3) 111 (4.0) 128 (5.7) belgium 143 (3.2) 272 (5.3) 119 (2.5) 153 (4.5) canada 132 (3.0) 251 (4.1) 120 (2.4) 131 (3.1) chile 118 (3.8) 220 (5.7) 107 (3.4) 114 (4.2) czech republic 128 (4.0) 243 (6.6) 111 (3.8) 132 (5.0) denmark 122 (3.7) 234 (6.3) 111 (5.0) 123 (4.7) Estonia 118 (3.5) 225 (4.7) 109 (4.2) 117 (4.1) finland 125 (3.8) 239 (3.8) 114 (3.6) 125 (3.1) france 122 (4.4) 239 (7.4) 108 (3.4) 131 (6.6) Germany 135 (4.8) 252 (7.3) 113 (3.6) 139 (5.9) hungary 141 (7.1) 272 (9.5) 126 (4.7) 145 (8.2) ireland 124 (3.6) 237 (5.1) 113 (2.7) 123 (4.0) israel 172 (5.0) 320 (8.8) 151 (5.3) 168 (6.9) italy 121 (4.3) 227 (6.6) 107 (3.5) 121 (4.9) Japan 112 (3.2) 216 (5.7) 102 (3.1) 115 (4.2) korea 120 (3.6) 228 (5.6) 104 (3.5) 124 (4.5) netherlands 133 (6.0) 256 (9.0) 116 (4.0) 139 (7.6) norway 138 (3.5) 262 (5.8) 126 (3.3) 136 (4.8) Poland 125 (4.1) 242 (6.6) 115 (3.7) 126 (4.9) Portugal 119 (3.7) 223 (4.8) 107 (3.9) 116 (3.2) Slovak republic 131 (4.6) 251 (7.8) 118 (5.6) 133 (5.1) Slovenia 132 (3.5) 249 (4.5) 120 (3.4) 129 (4.0) Spain 138 (4.3) 267 (7.8) 122 (3.5) 145 (6.3) Sweden 129 (3.1) 247 (4.7) 117 (4.0) 130 (3.6) turkey 109 (4.7) 206 (7.0) 109 (4.9) 97 (3.8) England (united kingdom) 129 (4.8) 245 (6.2) 114 (4.1) 131 (4.3) united States 126 (4.2) 237 (6.3) 116 (3.6) 121 (5.0) oEcd average 129 (0.8) 245 (1.2) 115 (0.7) 129 (0.9) brazil 122 (4.1) 234 (6.1) 116 (4.0) 118 (5.0) bulgaria 145 (5.5) 272 (10.2) 131 (6.1) 142 (6.7) colombia 122 (3.8) 233 (6.3) 120 (4.4) 112 (3.9) croatia 126 (3.5) 237 (5.9) 120 (4.4) 117 (4.4) cyprus* 135 (3.1) 256 (4.0) 124 (3.1) 132 (3.3) hong kong-china 119 (4.4) 234 (6.7) 110 (4.2) 123 (5.2) macao-china 107 (2.1) 203 (3.1) 95 (2.5) 108 (3.2) malaysia 115 (3.8) 217 (5.6) 109 (3.9) 108 (3.4) montenegro 126 (3.3) 237 (4.4) 118 (4.6) 118 (3.8) russian federation 116 (3.8) 224 (6.6) 112 (4.6) 113 (4.0) Serbia 122 (4.0) 229 (6.4) 111 (3.4) 119 (5.4) Shanghai-china 120 (4.0) 229 (7.1) 107 (3.5) 121 (5.0) Singapore 130 (2.4) 244 (3.5) 113 (2.9) 131 (3.4) chinese taipei 126 (3.5) 232 (5.4) 107 (3.5) 125 (4.3) united arab Emirates 139 (3.5) 269 (5.7) 135 (3.4) 134 (4.4) uruguay 134 (4.3) 250 (6.3) 126 (4.0) 124 (4.1) * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 157 Annex b1: reSulTS For counTrIeS And economIeS table v.2.3 [Part 1/1] Top performers in problem solving and other curricular subjects 15-year-old students who are: Partners OECD top performers top performers in problem solving, top performers in at least in problem solving but not in any one subject, not top performers and in at least of the other but not in any of the four domains in problem solving subjects assessed one other subject Percentage of top performers in problem solving who are also top performers in mathematics Percentage of top performers in problem solving who are also top performers in science % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. australia 75.6 (0.8) 7.7 (0.4) 4.7 (0.4) 12.0 (0.5) 61.3 (2.0) 47.1 (2.0) 54.9 (1.8) austria 80.8 (1.1) 8.2 (0.7) 3.0 (0.4) 8.0 (0.7) 66.8 (2.9) 31.8 (3.5) 42.8 (3.3) belgium 74.1 (0.7) 11.5 (0.6) 3.5 (0.4) 10.8 (0.6) 70.8 (2.5) 47.4 (2.7) 43.3 (2.5) canada 72.6 (0.9) 9.9 (0.4) 5.5 (0.4) 12.0 (0.6) 57.7 (2.1) 44.5 (1.8) 43.9 (2.0) chile 96.7 (0.4) 1.2 (0.2) 1.1 (0.2) 1.0 (0.2) 40.0 (5.3) 12.8 (3.4) 22.9 (4.5) czech republic 81.9 (0.9) 6.2 (0.5) 2.9 (0.5) 9.0 (0.7) 70.3 (3.2) 34.9 (2.6) 45.0 (3.1) denmark 84.3 (0.9) 6.9 (0.7) 3.2 (0.5) 5.6 (0.6) 55.9 (4.7) 30.9 (3.1) 42.4 (4.3) Estonia 78.4 (0.8) 9.9 (0.7) 2.5 (0.4) 9.3 (0.7) 69.8 (2.5) 41.5 (3.9) 62.1 (3.2) finland 73.1 (0.8) 11.9 (0.8) 3.0 (0.4) 12.0 (0.7) 66.1 (2.5) 49.5 (2.0) 65.4 (2.4) france 78.8 (1.0) 9.2 (0.7) 2.5 (0.4) 9.5 (0.8) 67.4 (2.7) 55.3 (3.5) 44.9 (3.4) Germany 76.6 (1.2) 10.6 (0.8) 2.9 (0.5) 9.9 (0.8) 72.2 (2.9) 39.0 (2.7) 53.3 (3.6) hungary 86.9 (1.2) 7.5 (0.8) 1.5 (0.4) 4.1 (0.6) 67.8 (5.8) 42.0 (5.3) 50.7 (4.7) ireland 80.5 (0.8) 10.1 (0.6) 2.6 (0.4) 6.8 (0.5) 59.0 (3.5) 52.0 (3.1) 57.2 (3.5) israel 83.6 (1.3) 7.6 (0.7) 2.2 (0.4) 6.6 (0.8) 63.5 (3.0) 51.7 (3.8) 44.3 (3.4) italy 81.7 (1.2) 7.6 (0.7) 4.6 (0.6) 6.2 (0.7) 49.4 (3.7) 27.3 (3.7) 34.3 (4.2) Japan 63.7 (1.6) 14.1 (0.9) 6.3 (0.5) 16.0 (1.1) 62.9 (2.4) 47.0 (2.5) 50.7 (2.3) korea 61.0 (2.0) 11.3 (0.8) 6.7 (0.7) 20.9 (1.5) 73.5 (2.1) 40.3 (2.5) 34.1 (2.7) netherlands 75.4 (1.3) 11.0 (0.8) 2.1 (0.5) 11.5 (1.0) 79.1 (2.7) 45.1 (3.9) 57.3 (4.1) norway 79.9 (1.0) 7.0 (0.6) 5.2 (0.8) 7.9 (0.6) 46.9 (3.8) 42.5 (4.2) 36.9 (3.3) Poland 78.7 (1.4) 14.4 (1.0) 1.1 (0.3) 5.7 (0.7) 75.8 (4.0) 57.3 (4.2) 61.9 (5.1) Portugal 84.8 (1.0) 7.8 (0.6) 2.3 (0.5) 5.1 (0.6) 64.9 (4.5) 34.3 (4.8) 32.5 (4.0) Slovak republic 86.1 (1.0) 6.1 (0.7) 1.8 (0.4) 6.0 (0.8) 74.5 (4.8) 32.3 (5.4) 42.4 (6.4) Slovenia 82.6 (0.6) 10.8 (0.5) 1.4 (0.2) 5.3 (0.5) 74.4 (3.1) 34.9 (3.8) 60.1 (3.4) Spain 86.1 (0.8) 6.1 (0.6) 3.4 (0.4) 4.4 (0.4) 46.6 (3.3) 28.8 (3.3) 28.5 (2.8) Sweden 84.4 (0.9) 6.8 (0.8) 3.2 (0.4) 5.6 (0.5) 52.3 (3.3) 41.3 (3.8) 38.6 (3.2) turkey 91.7 (1.4) 6.1 (1.0) 0.3 (0.2) 1.8 (0.5) 76.2 (7.2) 49.3 (9.9) 30.1 (5.6) England (united kingdom) 78.9 (1.3) 6.8 (0.6) 4.4 (0.5) 9.8 (0.9) 59.0 (3.1) 41.7 (3.6) 52.8 (3.2) united States 83.9 (1.0) 4.5 (0.5) 4.1 (0.5) 7.5 (0.7) 54.6 (2.9) 45.1 (2.8) 46.9 (3.1) oEcd average 80.1 (0.2) 8.5 (0.1) 3.1 (0.1) 8.2 (0.1) 63.5 (0.7) 41.0 (0.7) 45.7 (0.7) brazil 97.6 (0.5) 0.6 (0.2) 1.1 (0.3) 0.7 (0.2) 34.1 (8.4) 14.5 (5.9) 12.0 (5.4) bulgaria 92.6 (0.9) 5.8 (0.7) 0.3 (0.2) 1.2 (0.3) 65.5 (8.2) 50.1 (8.8) 54.1 (12.0) colombia 98.6 (0.3) 0.3 (0.1) 0.9 (0.2) 0.3 (0.1) 17.6 (7.0) 9.3 (6.1) 6.8 (4.0) croatia 89.5 (1.3) 5.8 (0.7) 1.1 (0.2) 3.6 (0.6) 70.3 (5.5) 36.3 (4.8) 46.1 (6.7) cyprus* 92.4 (0.5) 4.0 (0.4) 1.4 (0.2) 2.2 (0.2) 49.4 (4.4) 36.4 (4.9) 28.5 (6.2) hong kong-china 60.2 (1.5) 20.5 (1.1) 3.4 (0.4) 15.9 (1.1) 79.8 (2.2) 48.9 (3.2) 49.4 (3.1) macao-china 70.8 (0.6) 12.6 (0.5) 4.0 (0.4) 12.6 (0.4) 74.9 (2.3) 26.5 (1.7) 28.3 (1.8) malaysia 98.1 (0.4) 1.0 (0.2) 0.4 (0.1) 0.5 (0.2) 50.7 (9.5) 4.4 (3.3) 20.8 (8.3) montenegro 97.8 (0.3) 1.4 (0.2) 0.4 (0.1) 0.4 (0.1) 39.4 (11.9) 21.3 (11.1) 18.4 (9.7) russian federation 86.8 (1.1) 5.9 (0.7) 3.0 (0.5) 4.2 (0.6) 50.0 (4.5) 32.1 (3.8) 31.3 (4.0) Serbia 92.5 (0.7) 2.7 (0.5) 1.9 (0.3) 2.8 (0.4) 53.0 (6.9) 24.9 (4.8) 23.8 (4.6) Shanghai-china 43.6 (1.4) 38.1 (1.5) 0.3 (0.1) 17.9 (1.3) 98.0 (0.7) 71.7 (2.3) 75.1 (2.0) Singapore 54.2 (0.7) 16.5 (0.6) 4.3 (0.4) 25.0 (0.7) 84.1 (1.2) 50.2 (1.5) 57.0 (1.7) chinese taipei 61.3 (1.3) 20.4 (1.0) 1.2 (0.2) 17.1 (0.9) 93.0 (1.2) 43.7 (2.6) 35.3 (2.2) united arab Emirates 94.3 (0.4) 3.2 (0.3) 0.8 (0.1) 1.7 (0.2) 54.9 (3.7) 36.8 (4.5) 46.6 (4.0) uruguay 97.2 (0.5) 1.6 (0.3) 0.5 (0.1) 0.6 (0.2) 44.7 (9.0) 23.8 (5.7) 28.0 (9.6) * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 158 Percentage of top performers in problem solving who are also top performers in reading © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.2.4 [Part 1/2] between- and within-school variation in problem-solving performance total variation in problem-solving performance1 OECD variance total variation between-school Within-school variation variation S.E. variance S.E. variance S.E. % % % 9 482 (209) 2 569 (178) 6 951 (106) 102.4 27.7 75.1 austria 8 801 (547) 4 183 (532) 4 505 (121) 95.1 45.2 48.7 belgium 11 314 (392) 5 412 (513) 5 804 (144) 122.2 58.4 62.7 canada 10 063 (343) 2 271 (236) 7 692 (168) 108.7 24.5 83.1 chile 7 382 (283) 3 153 (299) 4 123 (90) 79.7 34.1 44.5 czech republic 9 056 (371) 4 366 (473) 4 474 (174) 97.8 47.1 48.3 denmark 8 522 (363) 2 441 (326) 6 048 (164) 92.0 26.4 65.3 Estonia 7 658 (252) 1 826 (245) 5 868 (171) 82.7 19.7 63.4 finland 8 658 (218) 884 (120) 7 753 (183) 93.5 9.5 83.7 france 9 250 (812) w w w w 99.9 w w Germany 9 703 (475) 5 328 (471) 4 334 (111) 104.8 57.5 46.8 hungary 10 907 (573) 6 445 (683) 4 245 (113) 117.8 69.6 45.8 8 676 (338) 2 117 (272) 6 486 (162) 93.7 22.9 70.0 israel 15 230 (809) 7 751 (860) 7 429 (199) 164.5 83.7 80.2 italy 8 219 (363) 3 461 (360) 4 496 (131) 88.8 37.4 48.6 Japan 7 251 (320) 2 459 (280) 4 768 (124) 78.3 26.6 51.5 korea 8 311 (331) 2 604 (288) 5 575 (197) 89.8 28.1 60.2 netherlands 9 783 (597) 5 649 (634) 4 147 (146) 105.7 61.0 44.8 norway 10 600 (401) 2 264 (340) 8 270 (237) 114.5 24.4 89.3 Poland 9 303 (639) 3 357 (675) 5 930 (204) 100.5 36.3 64.0 Portugal 7 712 (280) 2 314 (240) 5 420 (157) 83.3 25.0 58.5 Slovak republic 9 597 (526) 4 761 (569) 4 625 (161) 103.7 51.4 50.0 Slovenia 9 428 (230) 5 114 (434) 4 272 (153) 101.8 55.2 46.1 10 890 (613) 3 121 (470) 7 776 (213) 117.6 33.7 84.0 Sweden 9 260 (348) 1 720 (321) 7 474 (182) 100.0 18.6 80.7 turkey 6 246 (367) 3 239 (385) 2 997 (89) 67.5 35.0 32.4 England (united kingdom) 9 342 (455) 2 735 (386) 6 606 (179) 100.9 29.5 71.3 united States 8 610 (398) 2 485 (410) 6 106 (165) 93.0 26.8 65.9 oEcd average 9 259 (85) 3 548 (87) 5 646 (30) 100.0 38.3 61.0 australia ireland Spain Partners variation variation in problem-solving in problem-solving performance between schools2 performance within schools3 as a percentage of the average total variation in problem-solving performance across oEcd countries brazil 8 421 (448) 3 988 (491) 4 435 (153) 90.9 43.1 47.9 11 347 (776) 6 294 (750) 4 994 (125) 122.5 68.0 53.9 colombia 8 397 (343) 3 092 (332) 5 262 (156) 90.7 33.4 56.8 croatia 8 472 (346) 3 426 (403) 5 042 (137) 91.5 37.0 54.5 cyprus* 9 781 (194) 3 448 (1 455) 6 641 (167) 105.6 37.2 71.7 hong kong-china 8 401 (397) 3 034 (365) 5 347 (160) 90.7 32.8 57.8 macao-china 6 269 (129) 1 871 (1 217) 5 035 (166) 67.7 20.2 54.4 malaysia 6 982 (320) 2 614 (306) 4 361 (162) 75.4 28.2 47.1 montenegro 8 390 (201) 3 212 (670) 5 178 (163) 90.6 34.7 55.9 russian federation 7 725 (360) 2 857 (393) 4 872 (145) 83.4 30.9 52.6 Serbia 7 942 (358) 2 935 (333) 4 949 (164) 85.8 31.7 53.4 Shanghai-china 8 082 (413) 3 333 (362) 4 723 (151) 87.3 36.0 51.0 Singapore 9 021 (181) 3 061 (362) 5 962 (159) 97.4 33.1 64.4 chinese taipei 8 266 (363) 3 214 (374) 5 010 (150) 89.3 34.7 54.1 11 134 (390) 5 607 (477) 5 504 (150) 120.2 60.6 59.4 9 457 (383) 4 000 (419) 5 446 (133) 102.1 43.2 58.8 bulgaria united arab Emirates uruguay 1. The total variation in student performance is calculated from the square of the standard deviation for all students. 2. In some countries/economies, sub-units within schools were sampled instead of schools; this may affect the estimation of between-school variation components (see Annex A3). 3. Due to the unbalanced clustered nature of the data, the sum of the between- and within-school variation components, as an estimate from a sample, does not necessarily add up to the total. 4. The index of academic inclusion is calculated as 100 × (1-rho), where rho stands for the intra-class correlation of performance, i.e. the variation in student performance between schools, divided by the sum of the variation in student performance between schools and the variation in student performance within schools. 5. The index of social inclusion is calculated as 100 × (1-rho), where rho stands for the intra-class correlation of socio-economic status, i.e. the between-school variation in the PISA index of economic, social and cultural status (ESCS) of students, divided by the sum of the between-school variation in students’ socio-economic status and the withinschool variation in students’ socio-economic status. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 159 Annex b1: reSulTS For counTrIeS And economIeS table v.2.4 [Part 2/2] between- and within-school variation in problem-solving performance index of academic inclusion: Proportion of performance variation within schools4 OECD Problem solving reading index of social inclusion: Proportion of EScS variation within schools5 Science % S.E. % S.E. % S.E. % S.E. % S.E. australia 73.0 (1.4) 72.1 (1.8) 73.1 (1.5) 75.6 (1.5) 76.5 (1.2) austria 51.9 (3.1) 51.6 (2.4) 46.7 (2.0) 52.0 (2.4) 71.2 (2.9) belgium 51.7 (2.5) 48.6 (2.3) 45.6 (2.6) 50.8 (2.4) 72.4 (2.1) canada 77.2 (1.8) 80.2 (1.4) 81.1 (1.3) 82.8 (1.4) 82.8 (1.3) chile 56.7 (2.4) 56.6 (2.2) 55.5 (2.3) 58.8 (2.2) 47.2 (2.4) czech republic 50.6 (3.0) 48.5 (2.8) 50.0 (2.8) 52.6 (3.1) 76.4 (2.3) denmark 71.2 (2.7) 83.5 (2.0) 79.0 (3.8) 82.4 (2.5) 82.3 (1.7) Estonia 76.3 (2.5) 82.7 (2.4) 78.8 (2.8) 81.1 (2.3) 81.5 (2.1) finland 89.8 (1.3) 92.5 (1.2) 90.9 (1.2) 92.3 (1.1) 91.1 (1.1) w w w w w w w w w w Germany 44.9 (2.3) 47.0 (2.1) 42.7 (2.1) 47.2 (2.5) 73.6 (2.0) hungary 39.7 (2.7) 38.1 (2.5) 35.3 (2.2) 42.8 (2.6) 62.6 (2.8) ireland 75.4 (2.4) 81.8 (2.3) 77.5 (2.6) 81.7 (2.4) 79.7 (2.3) israel 48.9 (2.9) 57.6 (2.8) 54.6 (3.6) 56.6 (3.1) 74.6 (1.9) italy 56.5 (2.6) 49.7 (2.9) 49.5 (2.9) 50.6 (2.8) 75.1 (2.4) Japan 66.0 (2.6) 47.0 (2.5) 55.3 (2.6) 56.6 (2.6) 77.8 (1.8) korea 68.2 (2.5) 60.4 (3.2) 63.7 (3.2) 63.7 (3.1) 78.3 (2.0) netherlands 42.3 (2.9) 34.1 (2.2) 34.4 (2.7) 38.8 (2.4) 81.8 (1.9) norway 78.5 (2.6) 87.1 (1.8) 86.2 (1.9) 86.9 (2.1) 91.0 (1.5) Poland 63.9 (4.8) 79.5 (3.4) 79.6 (2.6) 82.0 (2.9) 76.4 (2.3) Portugal 70.1 (2.3) 70.1 (2.5) 68.8 (2.4) 68.5 (2.6) 68.6 (3.6) Slovak republic 49.3 (3.1) 50.1 (2.9) 38.1 (2.7) 45.6 (3.0) 64.4 (3.0) Slovenia 45.5 (2.3) 41.3 (2.5) 39.9 (2.2) 43.9 (2.6) 74.6 (2.0) Spain 71.4 (3.1) 80.2 (1.8) 80.7 (2.1) 80.6 (2.2) 74.9 (2.3) Sweden 81.3 (2.9) 87.5 (1.8) 83.5 (2.0) 83.3 (2.0) 86.9 (1.5) turkey 48.1 (3.2) 38.2 (3.3) 44.4 (3.2) 43.6 (3.1) 72.3 (3.0) England (united kingdom) 70.7 (3.0) 71.1 (2.9) 69.2 (3.1) 70.7 (2.7) 78.7 (2.5) united States 71.1 (3.5) 76.3 (2.2) 76.3 (2.6) 76.0 (2.3) 73.8 (2.5) oEcd average 61.9 (0.5) 62.8 (0.5) 61.5 (0.5) 64.0 (0.5) 75.7 (0.4) brazil 52.7 (3.2) 55.3 (3.5) 58.7 (3.2) 57.2 (3.3) 61.2 (3.5) bulgaria 44.2 (3.1) 47.2 (2.7) 40.6 (2.4) 45.6 (2.6) 59.6 (2.9) colombia 63.0 (2.7) 64.9 (2.9) 61.2 (3.1) 67.0 (3.0) 63.2 (3.0) croatia 59.5 (3.0) 55.7 (3.9) 48.9 (2.9) 62.2 (3.3) 75.9 (2.2) cyprus* 66.1 (8.3) 67.6 (4.8) 65.5 (4.6) 60.1 (11.9) 76.6 (3.4) hong kong-china 63.8 (2.8) 57.6 (2.2) 58.4 (2.4) 63.5 (2.3) 67.7 (3.6) macao-china 72.9 (12.8) 65.6 (22.0) 64.7 (17.2) 66.5 (36.7) 73.7 (4.7) malaysia 62.5 (2.9) 67.6 (3.2) 74.9 (2.7) 73.5 (2.7) 71.5 (2.5) montenegro 61.7 (5.1) 63.5 (7.3) 62.4 (5.3) 65.3 (5.9) 80.6 (5.6) russian federation 63.0 (3.4) 73.2 (2.6) 67.3 (2.8) 70.5 (2.9) 75.0 (2.5) Serbia 62.8 (2.7) 54.0 (3.3) 54.5 (2.9) 58.5 (3.0) 78.0 (2.4) Shanghai-china 58.6 (2.7) 53.1 (2.7) 53.2 (2.7) 53.9 (2.6) 66.8 (2.6) Singapore 66.1 (2.8) 63.3 (3.2) 64.3 (3.1) 63.0 (3.2) 76.4 (2.7) chinese taipei 60.9 (3.0) 57.9 (3.2) 61.2 (2.9) 58.0 (3.3) 76.7 (2.1) united arab Emirates 49.5 (2.2) 55.6 (2.2) 51.0 (2.0) 56.6 (2.1) 73.9 (1.7) uruguay 57.7 (2.6) 58.0 (3.0) 54.7 (2.8) 60.8 (2.9) 60.2 (3.8) france Partners mathematics 1. The total variation in student performance is calculated from the square of the standard deviation for all students. 2. In some countries/economies, sub-units within schools were sampled instead of schools; this may affect the estimation of between-school variation components (see Annex A3). 3. Due to the unbalanced clustered nature of the data, the sum of the between- and within-school variation components, as an estimate from a sample, does not necessarily add up to the total. 4. The index of academic inclusion is calculated as 100 × (1-rho), where rho stands for the intra-class correlation of performance, i.e. the variation in student performance between schools, divided by the sum of the variation in student performance between schools and the variation in student performance within schools. 5. The index of social inclusion is calculated as 100 × (1-rho), where rho stands for the intra-class correlation of socio-economic status, i.e. the between-school variation in the PISA index of economic, social and cultural status (ESCS) of students, divided by the sum of the between-school variation in students’ socio-economic status and the withinschool variation in students’ socio-economic status. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 160 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.2.5 [Part 1/2] correlation of problem-solving performance with performance in mathematics, reading and science Partners OECD correlation1 between performance in problem solving and performance in curricular domains australia for comparison: correlation1 between performance in curricular domains mathematics and reading mathematics and science reading and science Problem solving and mathematics Problem solving and reading Problem solving and science corr. S.E. corr. S.E. corr. S.E. corr. S.E. corr. S.E. corr. S.E. 0.83 (0.00) 0.77 (0.01) 0.81 (0.01) 0.87 (0.00) 0.91 (0.00) 0.90 (0.00) austria 0.80 (0.01) 0.76 (0.01) 0.77 (0.02) 0.85 (0.01) 0.91 (0.00) 0.88 (0.01) belgium 0.81 (0.01) 0.76 (0.01) 0.79 (0.01) 0.88 (0.01) 0.92 (0.00) 0.90 (0.00) canada 0.76 (0.01) 0.71 (0.01) 0.75 (0.01) 0.82 (0.00) 0.87 (0.00) 0.87 (0.00) chile 0.80 (0.01) 0.72 (0.01) 0.75 (0.01) 0.80 (0.01) 0.86 (0.01) 0.84 (0.01) czech republic 0.88 (0.01) 0.79 (0.01) 0.83 (0.01) 0.84 (0.01) 0.88 (0.01) 0.84 (0.01) denmark 0.77 (0.01) 0.69 (0.02) 0.74 (0.02) 0.84 (0.01) 0.90 (0.00) 0.88 (0.01) Estonia 0.83 (0.01) 0.77 (0.01) 0.80 (0.01) 0.83 (0.01) 0.88 (0.00) 0.85 (0.01) finland 0.83 (0.01) 0.74 (0.01) 0.79 (0.01) 0.82 (0.01) 0.89 (0.00) 0.87 (0.00) france 0.83 (0.02) 0.76 (0.02) 0.80 (0.02) 0.86 (0.01) 0.90 (0.01) 0.88 (0.01) Germany 0.83 (0.01) 0.77 (0.01) 0.81 (0.01) 0.87 (0.01) 0.92 (0.00) 0.90 (0.00) hungary 0.83 (0.01) 0.79 (0.01) 0.81 (0.01) 0.87 (0.01) 0.93 (0.00) 0.88 (0.01) ireland 0.80 (0.01) 0.74 (0.01) 0.79 (0.01) 0.87 (0.01) 0.91 (0.00) 0.90 (0.00) israel 0.85 (0.01) 0.79 (0.01) 0.84 (0.01) 0.84 (0.01) 0.91 (0.00) 0.88 (0.01) italy 0.75 (0.01) 0.67 (0.02) 0.73 (0.02) 0.84 (0.01) 0.88 (0.01) 0.85 (0.01) Japan 0.75 (0.01) 0.68 (0.02) 0.72 (0.01) 0.86 (0.01) 0.89 (0.01) 0.89 (0.01) korea 0.80 (0.01) 0.76 (0.01) 0.77 (0.01) 0.88 (0.01) 0.90 (0.00) 0.88 (0.01) netherlands 0.84 (0.01) 0.80 (0.02) 0.85 (0.01) 0.88 (0.01) 0.92 (0.00) 0.89 (0.01) norway 0.79 (0.01) 0.71 (0.01) 0.75 (0.02) 0.84 (0.01) 0.90 (0.00) 0.86 (0.01) Poland 0.75 (0.02) 0.75 (0.02) 0.75 (0.02) 0.83 (0.01) 0.89 (0.00) 0.87 (0.01) Portugal 0.80 (0.01) 0.71 (0.02) 0.76 (0.01) 0.84 (0.01) 0.90 (0.00) 0.86 (0.01) Slovak republic 0.85 (0.01) 0.78 (0.01) 0.82 (0.01) 0.85 (0.01) 0.92 (0.01) 0.89 (0.01) Slovenia 0.81 (0.01) 0.75 (0.01) 0.80 (0.01) 0.83 (0.01) 0.90 (0.00) 0.90 (0.00) Spain 0.75 (0.01) 0.67 (0.02) 0.71 (0.01) 0.83 (0.01) 0.89 (0.00) 0.83 (0.01) Sweden 0.81 (0.01) 0.71 (0.01) 0.76 (0.01) 0.85 (0.00) 0.89 (0.00) 0.87 (0.01) turkey 0.84 (0.01) 0.73 (0.02) 0.77 (0.01) 0.81 (0.01) 0.87 (0.01) 0.84 (0.01) England (united kingdom) 0.86 (0.01) 0.79 (0.01) 0.83 (0.01) 0.90 (0.01) 0.93 (0.00) 0.91 (0.00) united States 0.86 (0.01) 0.80 (0.01) 0.83 (0.01) 0.89 (0.01) 0.93 (0.00) 0.91 (0.00) oEcd average 0.81 (0.00) 0.75 (0.00) 0.78 (0.00) 0.85 (0.00) 0.90 (0.00) 0.88 (0.00) brazil 0.83 (0.01) 0.70 (0.02) 0.75 (0.02) 0.80 (0.01) 0.86 (0.01) 0.82 (0.01) bulgaria 0.81 (0.01) 0.75 (0.01) 0.78 (0.01) 0.83 (0.01) 0.89 (0.01) 0.88 (0.01) colombia 0.74 (0.02) 0.65 (0.02) 0.67 (0.02) 0.81 (0.01) 0.86 (0.01) 0.81 (0.01) croatia 0.85 (0.01) 0.74 (0.02) 0.79 (0.01) 0.83 (0.01) 0.89 (0.01) 0.84 (0.01) cyprus* 0.80 (0.01) 0.71 (0.01) 0.76 (0.01) 0.82 (0.00) 0.89 (0.00) 0.85 (0.00) hong kong-china 0.76 (0.01) 0.72 (0.02) 0.71 (0.01) 0.86 (0.01) 0.89 (0.01) 0.90 (0.00) macao-china 0.80 (0.01) 0.69 (0.01) 0.74 (0.01) 0.82 (0.01) 0.87 (0.00) 0.86 (0.01) malaysia 0.83 (0.01) 0.70 (0.01) 0.78 (0.01) 0.80 (0.01) 0.88 (0.01) 0.85 (0.01) montenegro 0.81 (0.01) 0.68 (0.01) 0.75 (0.01) 0.80 (0.01) 0.89 (0.00) 0.84 (0.01) russian federation 0.74 (0.01) 0.65 (0.02) 0.65 (0.02) 0.78 (0.01) 0.85 (0.01) 0.84 (0.01) Serbia 0.83 (0.01) 0.72 (0.01) 0.77 (0.01) 0.82 (0.01) 0.88 (0.01) 0.83 (0.01) Shanghai-china 0.84 (0.01) 0.79 (0.01) 0.79 (0.01) 0.89 (0.01) 0.92 (0.00) 0.90 (0.01) Singapore 0.83 (0.00) 0.74 (0.01) 0.79 (0.01) 0.90 (0.00) 0.94 (0.00) 0.92 (0.00) chinese taipei 0.86 (0.01) 0.81 (0.01) 0.83 (0.01) 0.89 (0.00) 0.93 (0.00) 0.91 (0.00) united arab Emirates 0.80 (0.01) 0.75 (0.01) 0.78 (0.01) 0.85 (0.01) 0.89 (0.00) 0.89 (0.00) uruguay 0.79 (0.01) 0.71 (0.01) 0.73 (0.01) 0.81 (0.01) 0.84 (0.01) 0.83 (0.01) 1. The reported correlations are pairwise correlations between the corresponding latent constructs. 2. Total explained variance is the r-squared coeficient from a regression of problem-solving performance on mathematics, reading and science performance. Variation uniquely associated with each domain is measured as the difference between the r-squared of the full regression and the r-squared of a regression of problem solving on the two remaining domains only. The residual variation is computed as: 100 - total explained variation. 3. The variation explained by the mode of delivery is measured as the difference between the r-squared of regression of problem-solving performance on mathematics, reading and science performance and the r-squared of the same regression augmented with computer-based mathematics performance. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 161 Annex b1: reSulTS For counTrIeS And economIeS table v.2.5 [Part 2/2] correlation of problem-solving performance with performance in mathematics, reading and science variation in problem-solving performance associated with mathematics, reading and science performance Partners OECD total explained variation2 variation uniquely variation uniquely variation uniquely associated associated associated with mathematics with reading with science 2 2 performance performance performance2 variation associated with more than one domain2 residual (unexplained) variation2 variation in problem-solving performance explained by the mode of delivery, as a percentage of total variation3 % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. australia 71.1 (0.8) 4.5 (0.4) 0.1 (0.1) 0.7 (0.2) 65.7 (0.8) 28.9 (0.8) 2.1 (0.4) austria 65.9 (2.3) 4.1 (0.7) 1.6 (0.5) 0.1 (0.1) 60.1 (2.5) 34.1 (2.3) 1.7 (0.7) belgium 67.2 (1.4) 3.1 (0.5) 0.2 (0.1) 0.8 (0.2) 63.1 (1.5) 32.8 (1.4) 1.7 (0.4) canada 61.3 (1.2) 3.8 (0.5) 0.5 (0.2) 1.2 (0.3) 55.8 (1.3) 38.7 (1.2) 1.4 (0.4) chile 66.1 (1.5) 6.7 (0.6) 0.6 (0.2) 0.6 (0.2) 58.2 (1.6) 33.9 (1.5) 0.2 (0.2) czech republic 79.0 (1.2) 7.5 (0.7) 0.4 (0.2) 0.5 (0.2) 70.6 (1.4) 21.0 (1.2) m m denmark 60.0 (2.3) 4.7 (0.8) 0.1 (0.1) 0.6 (0.3) 54.6 (2.5) 40.0 (2.3) 5.8 (1.1) Estonia 72.0 (1.4) 4.8 (0.6) 0.8 (0.3) 0.8 (0.3) 65.6 (1.4) 28.0 (1.4) 1.2 (0.6) finland 71.3 (1.0) 7.1 (0.6) 0.1 (0.1) 0.6 (0.2) 63.4 (1.0) 28.7 (1.0) m m france 70.3 (3.3) 4.8 (0.6) 0.1 (0.1) 0.9 (0.3) 64.5 (3.1) 29.7 (3.3) 5.3 (2.8) Germany 71.2 (1.6) 3.9 (0.6) 0.2 (0.1) 0.7 (0.2) 66.4 (1.7) 28.8 (1.6) 1.7 (0.6) hungary 71.0 (1.6) 2.5 (0.4) 1.1 (0.4) 0.4 (0.2) 66.9 (1.6) 29.0 (1.6) 1.8 (0.4) ireland 65.8 (1.3) 3.1 (0.5) 0.1 (0.1) 1.2 (0.4) 61.4 (1.3) 34.2 (1.3) 0.3 (0.3) israel 75.4 (1.3) 4.2 (0.5) 0.6 (0.2) 0.6 (0.2) 69.9 (1.3) 24.6 (1.3) 3.2 (0.6) italy 58.4 (2.0) 4.5 (0.8) 0.0 (0.1) 1.4 (0.5) 52.5 (2.0) 41.6 (2.0) 2.0 (0.6) Japan 58.0 (1.9) 5.7 (0.8) 0.0 (0.0) 0.8 (0.3) 51.5 (1.9) 42.0 (1.9) 7.8 (0.9) korea 66.5 (1.6) 3.7 (0.6) 0.6 (0.2) 0.5 (0.2) 61.6 (1.6) 33.5 (1.6) 1.8 (0.4) netherlands 74.9 (2.0) 2.1 (0.4) 0.1 (0.1) 2.2 (0.5) 70.4 (2.1) 25.1 (2.0) m m norway 63.8 (2.1) 6.1 (0.8) 0.3 (0.2) 0.3 (0.2) 57.2 (2.2) 36.2 (2.1) 5.7 (1.0) Poland 62.4 (2.5) 1.8 (0.5) 2.5 (0.6) 0.6 (0.3) 57.5 (2.4) 37.6 (2.5) 5.2 (1.5) Portugal 65.5 (2.1) 6.8 (0.8) 0.2 (0.1) 0.2 (0.2) 58.2 (2.2) 34.5 (2.1) 2.2 (0.5) Slovak republic 74.1 (1.6) 5.8 (1.0) 0.5 (0.2) 0.1 (0.1) 67.6 (1.9) 25.9 (1.6) 1.0 (0.3) Slovenia 68.7 (1.1) 4.7 (0.6) 0.4 (0.2) 0.5 (0.2) 63.0 (0.9) 31.3 (1.1) 2.8 (0.4) Spain 57.1 (2.0) 4.3 (0.9) 0.2 (0.2) 0.8 (0.3) 51.7 (1.9) 42.9 (2.0) 4.4 (0.9) Sweden 66.4 (1.4) 6.9 (0.8) 0.0 (0.0) 0.6 (0.3) 58.8 (1.3) 33.6 (1.4) 3.2 (0.7) turkey 71.0 (1.6) 9.6 (0.8) 0.3 (0.1) 0.2 (0.1) 60.9 (1.9) 29.0 (1.6) m m England (united kingdom) 74.4 (1.3) 4.5 (0.6) 0.0 (0.0) 0.7 (0.3) 69.1 (1.4) 25.6 (1.3) m m united States 74.8 (1.5) 4.4 (0.6) 0.2 (0.2) 0.3 (0.2) 69.8 (1.6) 25.2 (1.5) 1.0 (0.4) oEcd average 68.0 (0.3) 4.9 (0.1) 0.4 (0.0) 0.7 (0.0) 62.0 (0.3) 32.0 (0.3) 2.8 (0.2) brazil 69.0 (2.1) 10.4 (1.2) 0.3 (0.2) 0.2 (0.2) 58.1 (2.4) 31.0 (2.1) 2.0 (0.7) bulgaria 67.6 (2.0) 5.2 (0.8) 0.7 (0.3) 0.4 (0.2) 61.2 (2.1) 32.4 (2.0) m m colombia 55.4 (2.5) 7.5 (0.9) 0.5 (0.2) 0.1 (0.1) 47.3 (2.4) 44.6 (2.5) 2.6 (0.7) croatia 72.7 (1.6) 8.2 (0.9) 0.2 (0.2) 0.3 (0.1) 64.0 (1.9) 27.3 (1.6) m m cyprus* 65.4 (1.1) 6.5 (0.5) 0.4 (0.1) 0.3 (0.1) 58.2 (1.1) 34.6 (1.1) m m hong kong-china 58.7 (2.1) 4.8 (0.7) 0.9 (0.4) 0.0 (0.1) 52.9 (2.1) 41.3 (2.1) 3.3 (0.7) macao-china 64.5 (1.0) 8.5 (0.6) 0.1 (0.1) 0.4 (0.1) 55.6 (1.0) 35.5 (1.0) 1.8 (0.3) malaysia 70.4 (1.4) 9.3 (0.9) 0.0 (0.1) 0.5 (0.2) 60.6 (1.6) 29.6 (1.4) m m montenegro 66.0 (1.3) 9.2 (0.8) 0.1 (0.1) 0.2 (0.1) 56.5 (1.1) 34.0 (1.3) m m russian federation 55.9 (2.0) 10.5 (1.1) 1.2 (0.3) 0.1 (0.1) 44.2 (2.6) 44.1 (2.0) 7.8 (1.4) Serbia 70.0 (1.2) 8.3 (0.9) 0.1 (0.1) 0.6 (0.3) 61.0 (1.5) 30.0 (1.2) m m Shanghai-china 71.1 (1.4) 5.8 (0.6) 0.4 (0.2) 0.0 (0.1) 64.8 (1.6) 28.9 (1.4) 1.6 (0.4) Singapore 69.7 (0.6) 6.8 (0.7) 0.2 (0.1) 0.3 (0.1) 62.4 (0.9) 30.3 (0.6) 0.5 (0.2) chinese taipei 75.5 (1.1) 4.7 (0.4) 0.4 (0.1) 0.1 (0.1) 70.3 (1.2) 24.5 (1.1) 0.9 (0.3) united arab Emirates 66.6 (1.2) 3.7 (0.5) 0.4 (0.2) 1.1 (0.2) 61.4 (1.2) 33.4 (1.2) 1.3 (0.4) uruguay 65.1 (1.7) 7.8 (0.8) 0.5 (0.3) 0.6 (0.2) 56.1 (1.8) 34.9 (1.7) m m 1. The reported correlations are pairwise correlations between the corresponding latent constructs. 2. Total explained variance is the r-squared coeficient from a regression of problem-solving performance on mathematics, reading and science performance. Variation uniquely associated with each domain is measured as the difference between the r-squared of the full regression and the r-squared of a regression of problem solving on the two remaining domains only. The residual variation is computed as: 100 - total explained variation. 3. The variation explained by the mode of delivery is measured as the difference between the r-squared of regression of problem-solving performance on mathematics, reading and science performance and the r-squared of the same regression augmented with computer-based mathematics performance. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 162 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.2.6 [Part 1/3] relative performance in problem solving compared with performance in mathematics, reading and science relative performance in problem solving compared with students around the world1 with similar scores in… OECD … mathematics, reading and science (expected performance) relative performance across all students2 Percentage of students who perform above relative performance (actual minus their expected score3 across all students4 expected score) relative performance among strong and top performers in mathematics (at or above level 4)4 relative performance among moderate and low performers in mathematics (at or below level 3)4 difference in relative performance: strong and top performers minus moderate and low performers Score dif. S.E. % S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. 7 (1.5) 56.0 (1.2) 10 (1.6) 14 (1.8) 8 (1.7) 6 (1.6) austria -5 (2.7) 46.4 (2.1) -8 (2.8) -8 (3.5) -9 (3.3) 1 (3.9) belgium -10 (2.1) 43.0 (1.5) -13 (2.1) -10 (2.6) -16 (2.7) 6 (3.2) canada 0 (1.9) 50.5 (1.2) 1 (2.0) 5 (2.1) -2 (2.3) 7 (2.3) chile 1 (2.7) 51.6 (2.3) 3 (2.7) -1 (3.8) 3 (2.8) -4 (3.3) czech republic 1 (2.4) 51.8 (2.3) 0 (2.5) 6 (2.7) -3 (2.9) 9 (3.0) denmark -11 (2.5) 41.7 (2.0) -14 (2.5) -8 (3.2) -16 (2.9) 8 (3.3) Estonia -15 (1.9) 38.2 (1.6) -13 (2.0) -5 (2.2) -17 (2.4) 12 (2.5) finland -8 (2.0) 43.8 (1.7) -3 (2.0) 7 (2.4) -9 (2.2) 16 (2.1) france 5 (2.7) 56.5 (1.8) 5 (2.8) 5 (2.8) 6 (3.4) -1 (3.6) Germany -12 (2.6) 41.0 (2.0) -12 (2.6) -6 (3.0) -16 (3.3) 10 (3.7) hungary -34 (2.6) 26.7 (1.7) -32 (2.8) -22 (3.5) -35 (3.2) 14 (4.1) ireland -18 (2.9) 36.2 (2.1) -14 (2.9) -7 (3.1) -17 (3.3) 10 (3.1) israel -28 (2.8) 33.9 (1.8) -28 (2.9) -2 (3.4) -35 (3.2) 33 (3.9) italy 10 (3.5) 56.8 (2.5) 9 (3.5) 0 (4.2) 13 (3.8) -12 (4.0) Japan 11 (2.0) 57.7 (1.6) 13 (2.1) 4 (2.4) 21 (2.6) -17 (2.9) korea 14 (2.6) 61.1 (2.1) 9 (2.6) 6 (2.7) 13 (3.3) -7 (2.9) -16 (3.5) 39.2 (2.4) -18 (3.8) -8 (3.8) -26 (5.0) 17 (5.0) 1 (3.1) 51.0 (2.1) 2 (3.1) 12 (3.1) -2 (3.4) 14 (2.7) -44 (3.5) 22.3 (1.8) -44 (3.5) -44 (3.4) -43 (4.2) -1 (3.5) Portugal -3 (2.7) 47.3 (2.1) -5 (2.7) -12 (3.4) -2 (2.8) -10 (3.1) Slovak republic -5 (2.4) 45.7 (2.2) -11 (2.5) -11 (4.6) -11 (2.7) 0 (4.8) Slovenia -34 (1.3) 27.4 (0.9) -35 (1.3) -30 (1.6) -38 (1.8) 8 (2.5) Spain -20 (3.8) 39.7 (2.0) -20 (3.8) -12 (4.4) -22 (4.1) 10 (3.8) Sweden -1 (2.8) 49.2 (2.1) -2 (2.8) 1 (3.1) -2 (3.0) 3 (2.7) turkey -14 (1.9) 37.1 (1.8) -12 (2.0) -28 (3.4) -9 (2.1) -19 (3.6) 8 (2.4) 57.0 (1.9) 11 (2.5) 15 (2.6) 9 (3.0) 6 (3.2) united States 10 (2.1) 59.4 (1.9) 13 (2.1) 20 (2.6) 11 (2.4) 9 (2.9) oEcd average -7 (0.5) 45.3 (0.4) -7 (0.5) -4 (0.6) -9 (0.6) 5 (0.6) australia netherlands norway Poland England (united kingdom) Partners … mathematics brazil 7 (2.9) 56.3 (2.4) 6 (3.0) 19 (7.9) 6 (3.0) 13 (7.4) -54 (3.0) 18.0 (1.2) -57 (3.1) -46 (4.4) -59 (3.4) 13 (5.2) -7 (2.8) 45.6 (2.1) -5 (2.8) 14 (7.4) -6 (2.8) 20 (7.2) croatia -22 (2.5) 32.3 (2.0) -20 (2.5) -13 (2.7) -22 (2.8) 9 (3.1) cyprus* -12 (1.3) 41.8 (1.2) -15 (1.3) -14 (2.9) -15 (1.4) 1 (2.9) hong kong-china -16 (2.7) 39.2 (1.8) -19 (2.7) -23 (3.0) -12 (3.8) -11 (3.8) 8 (1.1) 56.7 (1.0) 0 (1.1) -8 (1.3) 8 (1.8) -16 (2.2) malaysia -14 (2.2) 38.6 (2.0) -21 (2.3) -18 (3.9) -21 (2.5) 3 (4.3) montenegro -24 (1.4) 32.0 (1.0) -27 (1.4) -20 (5.9) -28 (1.4) 7 (5.9) russian federation -4 (2.4) 47.4 (1.9) -7 (2.6) -12 (4.2) -5 (2.5) -7 (3.5) Serbia 11 (2.4) 59.0 (2.2) 6 (2.4) 1 (2.9) 7 (2.5) -5 (3.2) -51 (2.5) 14.3 (1.3) -59 (2.5) -59 (2.6) -57 (3.7) -2 (3.4) 2 (1.0) 51.3 (1.0) -4 (1.0) -5 (1.4) -2 (1.3) -3 (1.8) -9 (1.8) 41.7 (1.6) -21 (1.9) -29 (2.0) -10 (2.5) -19 (2.3) united arab Emirates -43 (2.1) 24.2 (1.1) -44 (2.2) -28 (3.5) -46 (2.4) 17 (3.8) uruguay -27 (2.9) 32.6 (1.9) -30 (3.0) -24 (6.0) -30 (3.1) 6 (5.8) bulgaria colombia macao-china Shanghai-china Singapore chinese taipei Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. “Students around the world” refers to 15-year-old students in countries and economies that participated in the PISA 2012 assessment of problem solving. national samples are weighted according to the size of the target population using inal student weights. 2. This column reports the difference between actual performance and the itted value from a regression using a second-degree polynomial as regression function (math, math sq., read, read sq., scie, scie sq., math×read, math×scie, read×scie). 3. This column reports the percentage of students for whom the difference between actual performance and the itted value from a regression is positive. Values that are indicated in bold are signiicantly larger or smaller than 50%. 4. This column reports the difference between actual performance and the itted value from a regression using a cubic polynomial as regression function. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 163 Annex b1: reSulTS For counTrIeS And economIeS table v.2.6 [Part 2/3] relative performance in problem solving compared with performance in mathematics, reading and science relative performance in problem solving compared with students around the world1 with similar scores in… OECD ... reading australia relative performance across all students4 Score dif. S.E. 4 (1.7) relative relative difference performance performance in relative among strong among moderate performance: and top and low strong and top performers performers performers in mathematics in mathematics minus (at or above (at or below moderate and low performers level 4)4 level 3)4 Score Score Score dif. S.E. dif. S.E. dif. S.E. 2 (2.0) 6 (1.9) -4 (1.9) austria 11 (3.0) 11 (4.0) 11 (3.3) 0 (4.2) 0 (2.9) -2 (3.5) 1 (3.3) -3 (3.9) belgium -3 (2.3) -2 (3.1) -3 (2.7) 2 (3.4) 2 (2.3) 5 (2.6) 0 (2.8) 5 (3.1) canada 4 (1.9) 2 (2.6) 5 (2.4) -3 (3.2) 3 (1.9) 4 (2.3) 3 (2.1) 1 (2.4) chile -9 (2.7) -8 (4.3) -9 (2.8) 1 (4.4) -8 (2.8) -15 (4.1) -8 (2.9) -7 (3.8) czech republic 11 (2.8) 16 (2.9) 10 (3.2) 5 (3.3) 0 (2.6) 4 (3.3) -1 (3.0) 5 (3.6) denmark -3 (2.6) -6 (4.3) -2 (3.0) -3 (4.9) -3 (2.7) -10 (3.3) -1 (3.0) -8 (3.4) Estonia -1 (2.1) 3 (2.4) -3 (2.5) 6 (2.5) -21 (2.0) -16 (2.2) -24 (2.6) 8 (2.7) finland 1 (2.2) -5 (3.0) 4 (2.6) -9 (3.4) -16 (2.2) -17 (2.7) -15 (2.3) -2 (2.6) 3 (3.2) -9 (3.4) 9 (4.0) -18 (4.3) 10 (2.9) 5 (3.3) 12 (3.4) -7 (3.8) -1 (2.8) 4 (3.4) -3 (3.2) 7 (3.7) -13 (2.8) -9 (3.3) -15 (3.3) 7 (3.6) france Germany hungary -35 (2.8) -23 (4.5) -39 (3.1) 16 (4.8) -38 (2.6) -24 (3.7) -43 (2.9) 19 (4.2) ireland -23 (2.8) -22 (3.1) -24 (3.2) 2 (3.1) -21 (3.0) -22 (3.4) -21 (3.3) -2 (3.1) israel -39 (3.1) -26 (3.8) -45 (3.5) 19 (4.3) -23 (2.9) -1 (3.6) -30 (3.2) 29 (4.2) italy 16 (3.7) -2 (4.1) 22 (4.2) -24 (4.3) 11 (3.6) -4 (4.6) 16 (3.9) -20 (4.5) Japan 19 (1.9) 2 (2.5) 34 (2.5) -32 (3.4) 12 (2.2) -1 (2.3) 25 (2.9) -26 (3.0) korea 29 (2.8) 30 (3.0) 29 (3.5) 1 (3.3) 28 (2.9) 30 (3.2) 27 (3.5) 4 (3.4) netherlands -2 (3.4) 6 (3.5) -6 (4.4) 12 (4.9) -9 (3.1) -3 (3.3) -13 (4.0) 10 (4.5) norway -3 (3.2) -6 (3.7) -2 (3.5) -5 (3.3) 6 (3.2) 4 (3.5) 7 (3.4) -4 (3.1) Poland -37 (3.5) -35 (3.8) -38 (4.0) 4 (3.5) -42 (3.6) -41 (3.5) -43 (4.2) 2 (3.7) 1 (2.7) -11 (3.7) 4 (2.9) -15 (3.5) 2 (2.9) -5 (3.4) 4 (3.1) -9 (2.9) Portugal Slovak republic 8 (2.6) 3 (5.2) 10 (2.9) -6 (5.8) 5 (2.5) 2 (4.8) 6 (2.8) -4 (5.3) Slovenia -13 (1.6) -13 (2.3) -13 (1.9) 0 (2.8) -37 (1.5) -34 (2.0) -39 (2.1) 4 (2.9) Spain -15 (3.8) -19 (4.7) -14 (4.1) -5 (4.2) -21 (3.8) -16 (4.8) -22 (4.0) 6 (3.9) 0 (3.0) -16 (4.2) 6 (3.1) -22 (4.0) 1 (3.0) -8 (3.9) 4 (3.1) -13 (3.2) -29 (2.3) -37 (3.5) -27 (2.6) -10 (3.8) -17 (2.1) -22 (4.0) -16 (2.2) -6 (4.2) 13 (2.4) 13 (3.0) 14 (3.1) 0 (3.9) 2 (2.5) 0 (2.6) 4 (3.0) -4 (3.0) 7 (2.2) 9 (2.8) 6 (2.4) 3 (3.0) 9 (2.3) 9 (2.8) 9 (2.6) 0 (3.0) -3 (0.5) -5 (0.7) -2 (0.6) -3 (0.7) -6 (0.5) -7 (0.6) -6 (0.6) -1 (0.7) Sweden turkey England (united kingdom) united States oEcd average Partners relative performance across all students4 Score dif. S.E. 10 (1.7) ... Science relative relative difference performance performance in relative among strong among moderate performance: and top and low strong and top performers performers performers in mathematics in mathematics minus (at or above (at or below moderate and low performers level 4)4 level 3)4 Score Score Score dif. S.E. dif. S.E. dif. S.E. 10 (2.1) 10 (1.8) 0 (2.0) brazil bulgaria -7 (3.0) -7 (7.6) -7 (3.0) 0 (7.6) 2 (2.9) 12 (8.1) 1 (2.9) 10 (7.7) -54 (3.5) -68 (4.6) -51 (3.9) -16 (5.3) -56 (3.2) -56 (4.4) -56 (3.5) 0 (5.0) colombia -29 (3.2) -22 (6.8) -29 (3.2) 7 (6.1) -19 (3.0) -2 (9.1) -20 (3.0) 18 (8.7) croatia -25 (2.8) -21 (3.7) -26 (3.0) 4 (4.0) -28 (2.7) -23 (3.7) -30 (2.9) 7 (3.8) cyprus* -20 (1.4) -36 (3.0) -17 (1.4) -19 (3.0) -6 (1.4) -13 (2.9) -5 (1.4) -8 (3.0) 1 (3.2) -1 (3.7) 3 (4.0) -4 (4.3) -7 (2.9) -10 (3.1) -5 (3.7) -5 (3.8) hong kong-china macao-china 30 (1.2) 18 (1.7) 36 (1.4) -18 (2.1) 22 (1.2) 15 (1.7) 25 (1.6) -11 (2.4) malaysia -2 (2.6) -7 (7.9) -2 (2.6) -6 (7.4) -13 (2.6) -8 (5.2) -13 (2.6) 5 (5.1) montenegro russian federation Serbia Shanghai-china Singapore chinese taipei -36 (1.5) -50 (4.3) -35 (1.6) -15 (4.7) -21 (1.4) -22 (5.7) -21 (1.5) -1 (6.1) 6 (2.4) -10 (4.7) 9 (2.5) -19 (4.7) -1 (2.5) -16 (4.0) 2 (2.6) -18 (4.0) (4.5) 12 (2.7) 1 (3.8) 14 (2.9) -14 (4.4) 17 (2.9) 11 (4.0) 18 (3.0) -7 -22 (2.6) -17 (2.9) -29 (3.4) 12 (3.4) -31 (2.6) -28 (2.9) -36 (3.6) 8 (3.7) 26 (1.1) 18 (1.7) 33 (1.5) -15 (2.4) 19 (1.0) 12 (1.3) 27 (1.5) -14 (2.1) 13 (2.1) 14 (2.5) 12 (2.5) 2 (2.7) 13 (2.1) 20 (2.3) 10 (2.5) 11 (2.5) united arab Emirates -47 (2.0) -32 (3.7) -49 (2.1) 16 (3.9) -48 (2.1) -37 (3.4) -50 (2.3) 13 (3.5) uruguay -32 (3.0) -35 (7.2) -32 (3.1) -3 (7.7) -30 (2.9) -37 (6.2) -29 (3.0) -8 (6.4) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. “Students around the world” refers to 15-year-old students in countries and economies that participated in the PISA 2012 assessment of problem solving. national samples are weighted according to the size of the target population using inal student weights. 2. This column reports the difference between actual performance and the itted value from a regression using a second-degree polynomial as regression function (math, math sq., read, read sq., scie, scie sq., math×read, math×scie, read×scie). 3. This column reports the percentage of students for whom the difference between actual performance and the itted value from a regression is positive. Values that are indicated in bold are signiicantly larger or smaller than 50%. 4. This column reports the difference between actual performance and the itted value from a regression using a cubic polynomial as regression function. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 164 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.2.6 [Part 3/3] relative performance in problem solving compared with performance in mathematics, reading and science relative performance in problem solving compared with students in countries/economies that also assessed mathematics on computers who have similar scores in… ...Paper-based mathematics (a) ...computer-based mathematics (b) relative performance across all students4 relative performance across all students4 OECD Score dif. australia Score dif. S.E. Score dif. S.E. 8 (1.6) 12 (1.7) -4 (1.3) austria -10 (2.8) -4 (2.8) -6 (2.4) belgium -15 (2.2) -7 (2.4) -8 (1.6) canada -1 (1.9) 2 (2.0) -3 (1.4) 1 (2.8) 3 (3.6) -2 (2.3) -2 (2.5) m m m m denmark -15 (2.6) -4 (2.4) -12 (1.9) Estonia -14 (2.1) -3 (2.6) -11 (1.7) finland -5 (2.1) m m m m france 4 (2.7) -1 (2.3) 4 (2.3) Germany -14 (2.6) -3 (2.5) -10 (2.0) hungary -34 (2.8) -19 (2.7) -14 (2.2) ireland -15 (3.0) 0 (3.5) -15 (2.2) israel -29 (3.0) -6 (3.0) -23 (2.5) italy 8 (3.5) 7 (3.2) 1 (2.7) Japan 12 (2.1) 15 (2.0) -3 (1.7) korea 8 (2.6) 12 (2.7) -5 (2.0) -19 (3.9) m m m m 0 (3.2) 1 (3.0) -1 (2.2) -45 (3.5) -14 (3.1) -31 (2.2) -7 (2.7) 0 (2.9) -6 (2.1) Slovak republic -13 (2.5) -19 (2.8) 6 (1.8) Slovenia -37 (1.3) -17 (1.3) -20 (0.9) Spain -21 (3.8) -6 (3.6) -15 (2.6) Sweden -3 (2.8) -5 (3.0) 1 (2.3) turkey -14 (2.1) m m m m 9 (2.6) m m m m united States 11 (2.1) 6 (2.2) 6 (1.6) oEcd average -9 (0.5) -2 (0.6) -7 (0.4) (2.3) chile czech republic netherlands norway Poland Portugal England (united kingdom) Partners S.E. mode effects: Score-point difference attributed to computer delivery (a - b) brazil 5 (2.9) -7 (2.7) 12 -59 (3.2) m m m m -7 (2.8) -16 (3.0) 9 (2.3) croatia -22 (2.6) m m m m cyprus* -16 (1.4) m m m m hong kong-china -20 (2.8) -7 (3.1) -12 (2.1) bulgaria colombia macao-china -1 (1.2) -1 (1.4) -1 (1.0) malaysia -23 (2.5) m m m m montenegro -29 (1.5) m m m m -8 (2.6) -6 (2.4) -3 (1.9) russian federation Serbia Shanghai-china Singapore 4 (2.4) m m m m -59 (2.5) -20 (2.7) -39 (2.2) -5 (1.0) 3 (1.2) -8 (1.0) chinese taipei -22 (2.0) -2 (2.6) -20 (2.0) united arab Emirates -45 (2.2) -36 (1.9) -9 (1.7) uruguay -32 (3.0) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. “Students around the world” refers to 15-year-old students in countries and economies that participated in the PISA 2012 assessment of problem solving. national samples are weighted according to the size of the target population using inal student weights. 2. This column reports the difference between actual performance and the itted value from a regression using a second-degree polynomial as regression function (math, math sq., read, read sq., scie, scie sq., math×read, math×scie, read×scie). 3. This column reports the percentage of students for whom the difference between actual performance and the itted value from a regression is positive. Values that are indicated in bold are signiicantly larger or smaller than 50%. 4. This column reports the difference between actual performance and the itted value from a regression using a cubic polynomial as regression function. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003668 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 165 Annex b1: reSulTS For counTrIeS And economIeS table v.3.1 [Part 1/1] Performance in problem solving, by nature of the problem situation relative likelihood of success on interactive tasks, based on success in performing all other tasks (oEcd average = 1.00) average proportion of full-credit responses items referring to a static problem situation (15 items) Partners OECD all items (42 items) items referring to an interactive problem situation (27 items) accounting for booklet effects1 accounting for booklet and country/economyspeciic response-format effects2 % S.E. % S.E. % S.E. odds ratio S.E. odds ratio S.E. australia 50.9 (0.4) 52.8 (0.5) 49.9 (0.5) 1.03 (0.02) 1.02 (0.02) austria 44.9 (0.8) 48.3 (1.0) 43.0 (0.8) 0.93 (0.03) 0.93 (0.03) belgium 46.4 (0.5) 48.3 (0.6) 45.4 (0.6) 1.03 (0.02) 1.02 (0.02) canada 51.3 (0.6) 52.7 (0.7) 50.5 (0.7) 1.06 (0.02) 1.05 (0.02) chile 32.9 (0.8) 34.9 (0.9) 31.8 (0.8) 1.01 (0.03) 1.01 (0.03) czech republic 45.0 (0.7) 46.2 (0.7) 44.4 (0.7) 1.02 (0.02) 1.02 (0.02) denmark 44.3 (0.8) 47.9 (0.9) 42.3 (0.8) 0.92 (0.02) 0.91 (0.02) Estonia 47.1 (0.7) 49.7 (0.8) 45.6 (0.8) 0.98 (0.03) 0.97 (0.03) finland 49.3 (0.5) 52.1 (0.6) 47.7 (0.6) 0.92 (0.01) 0.92 (0.01) france 48.5 (0.7) 50.3 (0.8) 47.6 (0.7) 1.06 (0.03) 1.06 (0.03) Germany 47.4 (0.7) 49.4 (0.8) 46.3 (0.8) 1.02 (0.03) 1.02 (0.03) hungary 35.4 (0.9) 38.2 (1.1) 33.9 (0.9) 0.96 (0.03) 0.96 (0.03) ireland 44.6 (0.8) 44.4 (0.9) 44.6 (0.9) 1.17 (0.04) 1.16 (0.03) israel 37.1 (1.3) 39.7 (1.4) 35.6 (1.3) 0.96 (0.03) 0.98 (0.03) italy 47.8 (0.9) 49.5 (1.0) 46.8 (0.9) 1.05 (0.03) 1.04 (0.03) Japan 56.9 (0.7) 58.7 (0.8) 55.9 (0.7) 1.04 (0.02) 1.05 (0.02) korea 58.1 (0.9) 58.9 (1.0) 57.7 (1.0) 1.11 (0.03) 1.14 (0.03) netherlands 47.9 (1.1) 50.4 (1.2) 46.5 (1.2) 0.94 (0.02) 0.94 (0.02) norway 46.3 (0.9) 49.4 (1.0) 44.5 (0.9) 0.95 (0.03) 0.94 (0.03) Poland 41.3 (1.0) 44.1 (1.0) 39.7 (1.1) 0.96 (0.03) 0.97 (0.03) Portugal 42.7 (0.9) 44.0 (0.9) 42.0 (1.0) 1.07 (0.03) 1.07 (0.03) Slovak republic 40.7 (0.8) 44.2 (1.0) 38.8 (0.9) 0.92 (0.03) 0.92 (0.03) Slovenia 38.9 (0.7) 42.9 (0.8) 36.7 (0.8) 0.89 (0.03) 0.89 (0.03) Spain 40.7 (0.8) 42.3 (0.9) 39.8 (0.8) 1.05 (0.02) 1.04 (0.02) Sweden 43.8 (0.7) 47.7 (0.9) 41.6 (0.7) 0.90 (0.02) 0.91 (0.02) turkey 33.8 (0.9) 35.8 (0.9) 32.7 (0.9) 0.95 (0.02) 0.96 (0.02) England (united kingdom) 48.5 (1.1) 49.5 (1.0) 47.9 (1.1) 1.03 (0.02) 1.03 (0.02) united States 46.2 (1.0) 46.6 (1.1) 45.9 (1.0) 1.13 (0.04) 1.13 (0.04) oEcd average 45.0 (0.2) 47.1 (0.2) 43.8 (0.2) 1.00 (0.01) 1.00 (0.01) brazil 29.4 (0.9) 29.8 (1.0) 29.1 (1.0) 1.12 (0.04) 1.13 (0.04) bulgaria 24.5 (0.8) 28.4 (0.9) 22.3 (0.8) 0.79 (0.02) 0.82 (0.02) colombia 24.6 (0.7) 26.3 (0.8) 23.7 (0.7) 1.01 (0.03) 1.02 (0.03) croatia 36.9 (0.9) 39.3 (1.0) 35.6 (0.9) 0.94 (0.02) 0.94 (0.02) cyprus* 33.4 (0.4) 37.0 (0.5) 31.4 (0.5) 0.85 (0.02) 0.87 (0.02) hong kong-china 53.6 (0.8) 56.1 (0.9) 52.2 (0.8) 0.99 (0.02) 1.00 (0.02) macao-china 53.6 (0.5) 57.0 (0.6) 51.7 (0.6) 0.93 (0.02) 0.95 (0.03) malaysia 28.4 (0.8) 30.1 (0.8) 27.4 (0.8) 0.96 (0.02) 0.98 (0.02) montenegro 26.9 (0.4) 30.3 (0.5) 25.1 (0.4) 0.84 (0.02) 0.85 (0.02) russian federation 41.2 (0.8) 43.8 (0.9) 39.7 (0.8) 0.98 (0.02) 0.98 (0.02) Serbia 38.1 (0.8) 40.3 (0.8) 36.8 (0.8) 0.94 (0.02) 0.95 (0.02) Shanghai-china 52.6 (0.8) 56.7 (1.0) 50.3 (0.9) 0.89 (0.03) 0.92 (0.03) Singapore 58.3 (0.7) 59.8 (0.8) 57.5 (0.7) 1.05 (0.03) 1.06 (0.03) chinese taipei 52.3 (0.8) 56.3 (0.9) 50.1 (0.8) 0.90 (0.03) 0.92 (0.03) united arab Emirates 28.1 (0.5) 29.9 (0.6) 27.1 (0.6) 1.01 (0.03) 1.02 (0.03) uruguay 25.8 (0.6) 27.5 (0.7) 24.8 (0.6) 0.95 (0.02) 0.97 (0.02) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies are added to the estimation. 2. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies and response-format dummies interacted with country/economy dummies are added to the estimation. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003687 166 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.3.2 [Part 1/2] Performance in problem solving, by process average proportion of full-credit responses items assessing the process of “exploring and understanding” (10 items) Partners OECD all items (42 items) items assessing the process of “representing and formulating” (8 items) items assessing the process of “planning and executing” (17 items) items assessing the process of “monitoring and relecting” (7 items) % S.E. % S.E. % S.E. % S.E. % S.E. australia 50.9 (0.4) 54.9 (0.5) 49.3 (0.6) 51.5 (0.5) 45.9 (0.5) austria 44.9 (0.8) 49.2 (1.0) 41.8 (1.0) 47.4 (0.9) 37.2 (0.9) belgium 46.4 (0.5) 49.0 (0.7) 44.8 (0.8) 47.5 (0.6) 42.4 (0.7) canada 51.3 (0.6) 54.1 (0.7) 50.9 (0.9) 52.1 (0.6) 46.0 (0.8) chile 32.9 (0.8) 32.5 (1.0) 29.3 (0.9) 35.2 (0.8) 33.2 (0.8) czech republic 45.0 (0.7) 46.9 (0.9) 42.9 (0.9) 46.9 (0.6) 40.7 (0.7) denmark 44.3 (0.8) 46.1 (1.0) 42.1 (1.2) 48.1 (0.8) 36.1 (0.9) Estonia 47.1 (0.7) 48.9 (1.0) 44.4 (1.0) 49.5 (0.8) 42.5 (0.8) finland 49.3 (0.5) 53.7 (0.6) 46.3 (0.7) 51.0 (0.6) 42.7 (0.6) france 48.5 (0.7) 52.2 (1.0) 46.9 (0.9) 49.4 (0.8) 43.8 (0.8) Germany 47.4 (0.7) 50.6 (1.1) 44.1 (1.1) 49.5 (0.8) 42.2 (0.9) hungary 35.4 (0.9) 37.7 (1.1) 32.4 (1.1) 37.6 (0.9) 30.9 (1.1) ireland 44.6 (0.8) 47.5 (1.2) 41.4 (0.9) 45.5 (0.8) 42.2 (1.1) israel 37.1 (1.3) 41.9 (1.5) 35.2 (1.5) 37.0 (1.3) 32.7 (1.3) italy 47.8 (0.9) 51.5 (1.2) 47.2 (1.2) 48.0 (0.9) 42.8 (0.9) Japan 56.9 (0.7) 62.2 (0.9) 55.7 (0.9) 56.3 (0.7) 52.1 (0.7) korea 58.1 (0.9) 64.7 (1.1) 60.7 (1.3) 54.5 (0.9) 53.7 (1.1) netherlands 47.9 (1.1) 51.8 (1.2) 44.2 (1.3) 49.7 (1.1) 42.8 (1.2) norway 46.3 (0.9) 51.3 (1.0) 43.6 (1.2) 48.1 (1.0) 38.4 (1.1) Poland 41.3 (1.0) 43.8 (1.2) 38.5 (1.3) 43.7 (1.0) 35.6 (1.1) Portugal 42.7 (0.9) 43.5 (1.3) 39.4 (1.3) 45.7 (1.0) 39.0 (1.1) Slovak republic 40.7 (0.8) 43.6 (1.2) 37.1 (1.1) 43.2 (0.9) 35.7 (0.9) Slovenia 38.9 (0.7) 39.6 (1.0) 35.8 (1.0) 42.3 (0.7) 34.2 (0.8) Spain 40.7 (0.8) 42.5 (1.0) 37.3 (0.9) 42.3 (0.9) 39.0 (1.0) Sweden 43.8 (0.7) 48.3 (1.1) 41.9 (1.0) 44.6 (0.7) 38.0 (0.9) turkey 33.8 (0.9) 33.5 (1.0) 31.9 (1.1) 36.0 (0.9) 31.4 (1.0) England (united kingdom) 48.5 (1.1) 51.3 (1.3) 47.7 (1.3) 49.1 (1.0) 44.0 (1.0) united States 46.2 (1.0) 48.9 (1.2) 43.9 (1.3) 47.1 (1.0) 43.1 (1.2) oEcd average 45.0 (0.2) 47.9 (0.2) 42.7 (0.2) 46.4 (0.2) 40.3 (0.2) brazil 29.4 (0.9) 30.2 (1.1) 25.4 (1.2) 32.0 (1.1) 27.1 (0.9) bulgaria 24.5 (0.8) 27.8 (0.9) 19.1 (0.9) 26.7 (0.8) 21.6 (0.9) colombia 24.6 (0.7) 24.7 (0.9) 18.7 (0.8) 27.7 (0.8) 24.9 (0.8) croatia 36.9 (0.9) 37.2 (1.0) 33.0 (1.1) 40.5 (0.9) 33.5 (0.9) cyprus* 33.4 (0.4) 36.2 (0.5) 30.7 (0.6) 34.8 (0.5) 29.8 (0.5) hong kong-china 53.6 (0.8) 60.2 (1.2) 54.9 (1.0) 51.1 (0.8) 48.2 (1.1) macao-china 53.6 (0.5) 59.4 (0.9) 57.1 (0.9) 51.3 (0.5) 45.7 (0.8) malaysia 28.4 (0.8) 30.1 (0.9) 27.9 (1.0) 29.3 (0.7) 24.5 (0.8) montenegro 26.9 (0.4) 27.3 (0.6) 23.6 (0.5) 30.0 (0.5) 23.6 (0.5) russian federation 41.2 (0.8) 42.0 (1.0) 38.6 (1.1) 43.8 (0.8) 37.3 (0.9) Serbia 38.1 (0.8) 39.5 (0.9) 35.7 (0.9) 40.7 (0.8) 33.1 (0.9) Shanghai-china 52.6 (0.8) 58.3 (1.1) 55.3 (1.2) 49.8 (0.7) 47.2 (1.1) Singapore 58.3 (0.7) 64.1 (1.0) 59.7 (0.9) 55.4 (0.7) 55.2 (0.8) chinese taipei 52.3 (0.8) 58.1 (1.0) 55.5 (1.2) 50.1 (0.8) 44.7 (1.0) united arab Emirates 28.1 (0.5) 30.0 (0.6) 26.6 (0.8) 29.0 (0.6) 25.4 (0.7) uruguay 25.8 (0.6) 27.1 (0.7) 22.2 (0.7) 27.9 (0.7) 23.7 (0.7) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies are added to the estimation. 2. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies and response-format dummies interacted with country/economy dummies are added to the estimation. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003687 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 167 Annex b1: reSulTS For counTrIeS And economIeS table v.3.2 [Part 2/2] Performance in problem solving, by process relative likelihood of success, based on success in performing all other tasks (oEcd average = 1.00) on items assessing the process on items assessing the process of “exploring and understanding” of “representing and formulating” on items assessing the process of “planning and executing” on items assessing the process of “monitoring and relecting” Partners OECD accounting accounting accounting accounting for booklet for booklet for booklet for booklet and country/ and country/ and country/ and country/ economy-speciic economy-speciic economy-speciic economy-speciic accounting for response-format accounting for response-format accounting for response-format accounting for response-format effects2 effects2 effects2 effects2 booklet effects1 booklet effects1 booklet effects1 booklet effects1 odds ratio S.E. odds ratio S.E. odds ratio S.E. odds ratio S.E. odds ratio S.E. odds ratio S.E. odds ratio S.E. odds ratio S.E. australia 1.06 (0.02) 1.14 (0.02) 1.06 (0.02) 1.06 (0.02) 0.93 (0.02) 0.89 (0.02) 0.98 (0.02) 0.98 (0.02) austria 1.08 (0.03) 1.13 (0.04) 0.97 (0.04) 0.97 (0.04) 1.06 (0.03) 1.04 (0.03) 0.85 (0.03) 0.85 (0.03) belgium 0.98 (0.02) 1.03 (0.02) 1.05 (0.03) 1.05 (0.03) 0.96 (0.02) 0.93 (0.02) 1.03 (0.03) 1.03 (0.03) canada 0.99 (0.02) 1.02 (0.02) 1.12 (0.03) 1.12 (0.03) 0.95 (0.02) 0.92 (0.02) 0.97 (0.02) 0.97 (0.02) chile 0.83 (0.03) 0.77 (0.03) 0.92 (0.03) 0.92 (0.03) 1.06 (0.03) 1.09 (0.03) 1.27 (0.04) 1.28 (0.04) czech republic 0.92 (0.02) 0.89 (0.02) 0.92 (0.02) 0.92 (0.02) 1.09 (0.02) 1.11 (0.02) 1.05 (0.02) 1.06 (0.02) denmark 0.94 (0.03) 0.97 (0.03) 1.02 (0.04) 1.02 (0.04) 1.15 (0.03) 1.14 (0.04) 0.82 (0.03) 0.82 (0.03) Estonia 0.94 (0.03) 0.96 (0.03) 1.00 (0.03) 1.00 (0.03) 1.05 (0.03) 1.04 (0.03) 1.00 (0.03) 1.00 (0.03) finland 1.06 (0.02) 1.08 (0.02) 0.88 (0.02) 0.89 (0.02) 1.09 (0.02) 1.09 (0.02) 0.94 (0.02) 0.95 (0.02) france 1.02 (0.03) 1.03 (0.04) 1.07 (0.03) 1.07 (0.03) 0.95 (0.03) 0.94 (0.03) 1.00 (0.04) 1.00 (0.04) Germany 1.02 (0.03) 1.05 (0.04) 0.97 (0.03) 0.97 (0.03) 1.03 (0.03) 1.01 (0.03) 0.97 (0.03) 0.97 (0.03) hungary 0.98 (0.03) 0.93 (0.04) 0.97 (0.03) 0.97 (0.03) 1.05 (0.03) 1.09 (0.04) 0.98 (0.03) 0.98 (0.03) ireland 1.00 (0.04) 1.06 (0.04) 0.97 (0.03) 0.97 (0.03) 0.95 (0.03) 0.91 (0.03) 1.12 (0.04) 1.11 (0.04) israel 1.12 (0.03) 1.05 (0.03) 1.02 (0.03) 1.02 (0.03) 0.90 (0.02) 0.94 (0.03) 1.00 (0.03) 1.01 (0.03) italy 1.05 (0.03) 1.07 (0.04) 1.12 (0.03) 1.12 (0.03) 0.90 (0.02) 0.89 (0.03) 0.98 (0.03) 0.98 (0.03) Japan 1.15 (0.03) 1.11 (0.03) 1.08 (0.02) 1.08 (0.02) 0.86 (0.02) 0.88 (0.02) 0.99 (0.02) 1.00 (0.02) korea 1.25 (0.04) 1.16 (0.04) 1.33 (0.05) 1.32 (0.05) 0.69 (0.02) 0.71 (0.02) 1.00 (0.03) 1.02 (0.03) netherlands 1.02 (0.02) 1.03 (0.03) 0.85 (0.02) 0.85 (0.02) 1.09 (0.02) 1.10 (0.02) 1.02 (0.02) 1.02 (0.02) norway 1.12 (0.04) 1.19 (0.04) 1.00 (0.03) 1.00 (0.03) 1.01 (0.03) 0.99 (0.03) 0.84 (0.03) 0.84 (0.03) Poland 0.98 (0.03) 0.96 (0.03) 0.99 (0.03) 0.99 (0.03) 1.05 (0.03) 1.08 (0.03) 0.94 (0.03) 0.94 (0.03) Portugal 0.90 (0.03) 0.90 (0.03) 0.96 (0.04) 0.96 (0.04) 1.09 (0.04) 1.08 (0.04) 1.04 (0.05) 1.04 (0.05) Slovak republic 1.00 (0.03) 1.00 (0.04) 0.94 (0.03) 0.94 (0.03) 1.06 (0.03) 1.07 (0.04) 0.97 (0.03) 0.96 (0.03) Slovenia 0.89 (0.03) 0.85 (0.03) 0.97 (0.03) 0.97 (0.03) 1.13 (0.02) 1.16 (0.03) 0.98 (0.03) 0.98 (0.03) Spain 0.94 (0.03) 0.94 (0.03) 0.96 (0.03) 0.95 (0.03) 0.99 (0.03) 0.99 (0.03) 1.15 (0.03) 1.15 (0.03) Sweden 1.09 (0.04) 1.09 (0.04) 1.04 (0.03) 1.04 (0.03) 0.94 (0.03) 0.95 (0.04) 0.94 (0.03) 0.94 (0.03) turkey 0.82 (0.02) 0.75 (0.02) 0.92 (0.02) 0.93 (0.02) 1.14 (0.02) 1.19 (0.03) 1.15 (0.03) 1.15 (0.03) England (united kingdom) 0.97 (0.02) 0.99 (0.02) 0.98 (0.03) 0.99 (0.03) 1.01 (0.02) 0.99 (0.02) 1.05 (0.03) 1.05 (0.02) united States 0.99 (0.03) 1.01 (0.03) 1.02 (0.04) 1.02 (0.04) 0.95 (0.03) 0.94 (0.03) 1.08 (0.04) 1.08 (0.04) oEcd average 1.00 (0.01) 1.00 (0.01) 1.00 (0.01) 1.00 (0.01) 1.00 (0.00) 1.00 (0.01) 1.00 (0.01) 1.00 (0.01) brazil 0.90 (0.03) 0.84 (0.03) 0.89 (0.04) 0.89 (0.04) 1.10 (0.04) 1.16 (0.05) 1.10 (0.05) 1.10 (0.05) bulgaria 1.05 (0.03) 0.90 (0.02) 0.69 (0.02) 0.69 (0.02) 1.17 (0.03) 1.35 (0.04) 1.07 (0.03) 1.09 (0.03) colombia 0.86 (0.03) 0.77 (0.03) 0.74 (0.03) 0.74 (0.03) 1.18 (0.04) 1.29 (0.05) 1.28 (0.05) 1.29 (0.05) croatia 0.85 (0.02) 0.79 (0.02) 0.82 (0.02) 0.83 (0.02) 1.24 (0.03) 1.30 (0.03) 1.09 (0.03) 1.09 (0.03) cyprus* 0.98 (0.02) 0.90 (0.02) 0.88 (0.02) 0.88 (0.02) 1.07 (0.02) 1.14 (0.02) 1.06 (0.02) 1.07 (0.02) hong kong-china 1.23 (0.04) 1.17 (0.05) 1.23 (0.04) 1.23 (0.04) 0.76 (0.02) 0.78 (0.03) 0.96 (0.03) 0.97 (0.03) macao-china 1.18 (0.04) 1.09 (0.04) 1.38 (0.04) 1.38 (0.04) 0.77 (0.02) 0.80 (0.02) 0.85 (0.02) 0.86 (0.03) malaysia 0.93 (0.02) 0.80 (0.02) 1.00 (0.03) 1.00 (0.03) 1.04 (0.02) 1.15 (0.03) 1.03 (0.03) 1.04 (0.03) montenegro 0.86 (0.02) 0.77 (0.02) 0.82 (0.02) 0.82 (0.02) 1.24 (0.03) 1.35 (0.03) 1.05 (0.03) 1.06 (0.03) russian federation 0.90 (0.02) 0.87 (0.03) 1.00 (0.03) 1.00 (0.03) 1.07 (0.03) 1.08 (0.04) 1.03 (0.04) 1.03 (0.04) Serbia 0.90 (0.02) 0.87 (0.02) 0.90 (0.02) 0.90 (0.02) 1.16 (0.02) 1.19 (0.03) 1.00 (0.03) 1.01 (0.02) Shanghai-china 1.17 (0.04) 1.04 (0.03) 1.33 (0.05) 1.33 (0.05) 0.74 (0.02) 0.78 (0.03) 0.96 (0.03) 0.98 (0.03) Singapore 1.18 (0.04) 1.19 (0.04) 1.23 (0.04) 1.23 (0.04) 0.73 (0.02) 0.71 (0.02) 1.07 (0.03) 1.08 (0.03) chinese taipei 1.18 (0.03) 1.11 (0.04) 1.36 (0.04) 1.36 (0.04) 0.77 (0.02) 0.79 (0.02) 0.86 (0.03) 0.87 (0.03) united arab Emirates 0.97 (0.02) 0.88 (0.02) 1.04 (0.03) 1.04 (0.03) 0.96 (0.02) 1.02 (0.03) 1.07 (0.03) 1.07 (0.03) uruguay 0.91 (0.02) 0.80 (0.02) 0.80 (0.02) 0.80 (0.02) 1.15 (0.03) 1.28 (0.04) 1.14 (0.03) 1.15 (0.03) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies are added to the estimation. 2. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies and response-format dummies interacted with country/economy dummies are added to the estimation. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003687 168 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.3.3 [Part 1/1] Performance in problem solving, by technology setting relative likelihood of success on tasks set in a technology context, based on success in performing all other tasks (oEcd average = 1.00) average proportion of full-credit responses items not involving a technological device (24 items) Partners OECD all items (42 items) items involving a technological device (18 items) accounting for booklet effects1 accounting for booklet and country/economyspeciic response-format effects2 % S.E. % S.E. % S.E. odds ratio S.E. odds ratio australia 50.9 (0.4) 49.1 (0.4) 52.7 (0.5) 1.14 (0.02) 1.13 (0.02) S.E. austria 44.9 (0.8) 44.4 (0.9) 45.4 (0.8) 1.02 (0.03) 1.01 (0.03) belgium 46.4 (0.5) 45.6 (0.6) 47.3 (0.6) 1.05 (0.02) 1.04 (0.02) canada 51.3 (0.6) 50.3 (0.6) 52.3 (0.7) 1.06 (0.02) 1.05 (0.02) chile 32.9 (0.8) 32.3 (0.8) 33.5 (0.8) 1.04 (0.03) 1.04 (0.03) czech republic 45.0 (0.7) 43.5 (0.7) 46.6 (0.8) 0.96 (0.01) 0.97 (0.01) denmark 44.3 (0.8) 45.4 (0.9) 43.2 (0.8) 0.89 (0.02) 0.88 (0.02) Estonia 47.1 (0.7) 47.1 (0.8) 47.1 (0.8) 0.98 (0.03) 0.97 (0.03) finland 49.3 (0.5) 49.7 (0.6) 48.8 (0.6) 0.82 (0.01) 0.82 (0.01) france 48.5 (0.7) 47.8 (0.8) 49.2 (0.7) 1.06 (0.03) 1.06 (0.03) Germany 47.4 (0.7) 46.9 (0.8) 47.8 (0.8) 1.02 (0.02) 1.02 (0.02) hungary 35.4 (0.9) 35.3 (1.0) 35.5 (0.9) 0.98 (0.03) 0.99 (0.03) ireland 44.6 (0.8) 42.6 (0.9) 46.5 (0.9) 1.16 (0.04) 1.15 (0.04) israel 37.1 (1.3) 36.6 (1.4) 37.5 (1.3) 1.00 (0.04) 1.02 (0.04) italy 47.8 (0.9) 47.3 (1.0) 48.3 (0.9) 1.03 (0.03) 1.03 (0.03) Japan 56.9 (0.7) 56.0 (0.8) 57.8 (0.7) 1.05 (0.03) 1.07 (0.03) korea 58.1 (0.9) 57.8 (1.0) 58.4 (1.0) 1.01 (0.03) 1.03 (0.03) netherlands 47.9 (1.1) 47.1 (1.2) 48.7 (1.1) 0.90 (0.02) 0.91 (0.02) norway 46.3 (0.9) 46.4 (0.9) 46.2 (1.0) 0.97 (0.03) 0.97 (0.03) Poland 41.3 (1.0) 41.1 (1.1) 41.4 (1.1) 1.00 (0.03) 1.00 (0.03) Portugal 42.7 (0.9) 42.1 (0.9) 43.3 (1.0) 1.04 (0.03) 1.03 (0.03) Slovak republic 40.7 (0.8) 41.1 (0.9) 40.3 (1.0) 0.95 (0.03) 0.95 (0.03) Slovenia 38.9 (0.7) 39.0 (0.9) 38.8 (0.8) 0.96 (0.04) 0.96 (0.04) Spain 40.7 (0.8) 40.3 (0.9) 41.1 (0.8) 1.02 (0.03) 1.01 (0.03) Sweden 43.8 (0.7) 43.8 (0.8) 43.8 (0.8) 0.98 (0.03) 0.98 (0.03) turkey 33.8 (0.9) 34.0 (0.9) 33.6 (1.0) 0.83 (0.02) 0.85 (0.02) England (united kingdom) 48.5 (1.1) 46.1 (1.0) 50.9 (1.2) 1.03 (0.02) 1.03 (0.02) united States 46.2 (1.0) 44.6 (1.2) 47.8 (0.9) 1.12 (0.04) 1.11 (0.04) oEcd average 45.0 (0.2) 44.4 (0.2) 45.5 (0.2) 1.00 (0.01) 1.00 (0.01) brazil 29.4 (0.9) 28.9 (1.0) 29.8 (1.0) 1.03 (0.04) 1.03 (0.04) bulgaria 24.5 (0.8) 25.2 (0.8) 23.7 (0.9) 0.78 (0.02) 0.81 (0.02) colombia 24.6 (0.7) 24.6 (0.7) 24.5 (0.8) 0.98 (0.03) 0.99 (0.03) croatia 36.9 (0.9) 36.9 (0.9) 36.9 (0.9) 0.85 (0.02) 0.86 (0.02) cyprus* 33.4 (0.4) 33.0 (0.4) 33.9 (0.5) 0.88 (0.02) 0.90 (0.02) hong kong-china 53.6 (0.8) 52.2 (0.9) 55.0 (0.9) 1.10 (0.03) 1.12 (0.03) macao-china 53.6 (0.5) 54.7 (0.6) 52.4 (0.6) 0.89 (0.02) 0.90 (0.02) malaysia 28.4 (0.8) 28.8 (0.8) 28.0 (0.8) 0.82 (0.02) 0.84 (0.02) montenegro 26.9 (0.4) 27.7 (0.5) 26.2 (0.4) 0.79 (0.02) 0.80 (0.02) russian federation 41.2 (0.8) 40.6 (0.9) 41.7 (0.8) 1.03 (0.02) 1.03 (0.02) Serbia 38.1 (0.8) 38.4 (0.8) 37.7 (0.8) 0.82 (0.02) 0.83 (0.02) Shanghai-china 52.6 (0.8) 54.3 (0.9) 50.8 (1.0) 0.86 (0.02) 0.87 (0.03) Singapore 58.3 (0.7) 56.3 (0.7) 60.4 (0.8) 1.17 (0.04) 1.17 (0.04) chinese taipei 52.3 (0.8) 52.1 (0.9) 52.5 (0.9) 1.00 (0.02) 1.01 (0.03) united arab Emirates 28.1 (0.5) 27.4 (0.6) 28.8 (0.6) 1.06 (0.02) 1.07 (0.02) uruguay 25.8 (0.6) 25.9 (0.7) 25.6 (0.7) 0.83 (0.02) 0.86 (0.02) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies are added to the estimation. 2. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies and response-format dummies interacted with country/economy dummies are added to the estimation. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003687 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 169 Annex b1: reSulTS For counTrIeS And economIeS table v.3.4 [Part 1/1] Performance in problem solving, by social focus relative likelihood of success on tasks set in a social context, based on success in performing all other tasks (oEcd average = 1.00) average proportion of full-credit responses items relating primarily to the self, family, and peer groups (personal contexts) (29 items) Partners OECD all items (42 items) items relating to the community or society in general (social contexts) (13 items) accounting for booklet effects1 accounting for booklet and country/economyspeciic response-format effects2 % S.E. % S.E. % S.E. odds ratio S.E. odds ratio australia 50.9 (0.4) 47.1 (0.4) 55.6 (0.5) 1.02 (0.02) 1.07 (0.02) S.E. austria 44.9 (0.8) 41.4 (0.9) 49.2 (0.8) 1.00 (0.02) 1.03 (0.03) belgium 46.4 (0.5) 42.8 (0.6) 50.8 (0.6) 1.00 (0.02) 1.04 (0.02) canada 51.3 (0.6) 47.8 (0.6) 55.5 (0.8) 0.99 (0.02) 1.01 (0.03) chile 32.9 (0.8) 30.5 (0.8) 35.9 (0.9) 0.93 (0.03) 0.90 (0.03) czech republic 45.0 (0.7) 41.7 (0.7) 49.0 (0.8) 1.02 (0.01) 1.02 (0.02) denmark 44.3 (0.8) 41.9 (0.8) 47.3 (0.8) 0.90 (0.02) 0.92 (0.02) Estonia 47.1 (0.7) 44.3 (0.8) 50.4 (0.8) 0.93 (0.03) 0.94 (0.03) finland 49.3 (0.5) 46.2 (0.6) 53.0 (0.6) 1.00 (0.02) 1.01 (0.02) france 48.5 (0.7) 45.3 (0.6) 52.6 (0.9) 0.96 (0.03) 0.97 (0.03) Germany 47.4 (0.7) 44.1 (0.8) 51.4 (0.8) 0.98 (0.02) 0.99 (0.03) hungary 35.4 (0.9) 32.5 (0.9) 39.0 (1.0) 0.97 (0.03) 0.93 (0.03) ireland 44.6 (0.8) 40.4 (0.8) 49.6 (0.9) 1.06 (0.03) 1.11 (0.03) israel 37.1 (1.3) 34.3 (1.3) 40.4 (1.4) 0.95 (0.03) 0.89 (0.03) italy 47.8 (0.9) 44.1 (0.9) 52.2 (1.0) 1.01 (0.03) 1.02 (0.03) Japan 56.9 (0.7) 51.9 (0.7) 62.9 (0.8) 1.15 (0.02) 1.12 (0.02) korea 58.1 (0.9) 53.9 (0.9) 63.2 (1.1) 1.07 (0.03) 0.99 (0.03) netherlands 47.9 (1.1) 43.2 (1.2) 53.6 (1.1) 1.16 (0.02) 1.19 (0.03) norway 46.3 (0.9) 43.2 (0.9) 50.0 (0.9) 0.96 (0.03) 0.97 (0.03) Poland 41.3 (1.0) 37.7 (1.0) 45.6 (1.1) 1.01 (0.02) 0.99 (0.03) Portugal 42.7 (0.9) 38.5 (0.9) 47.8 (1.0) 1.06 (0.03) 1.10 (0.03) Slovak republic 40.7 (0.8) 37.9 (0.9) 44.1 (0.9) 0.94 (0.02) 0.93 (0.02) Slovenia 38.9 (0.7) 36.3 (0.8) 42.1 (0.8) 0.92 (0.02) 0.90 (0.03) Spain 40.7 (0.8) 37.6 (0.8) 44.4 (0.9) 0.96 (0.03) 0.96 (0.03) Sweden 43.8 (0.7) 40.0 (0.7) 48.4 (0.8) 1.02 (0.03) 1.01 (0.03) turkey 33.8 (0.9) 31.4 (0.9) 36.6 (1.0) 0.96 (0.02) 0.92 (0.02) England (united kingdom) 48.5 (1.1) 44.5 (1.1) 53.3 (1.1) 1.09 (0.02) 1.13 (0.03) united States 46.2 (1.0) 42.5 (1.0) 50.7 (1.1) 1.02 (0.02) 1.03 (0.03) oEcd average 45.0 (0.2) 41.5 (0.2) 49.1 (0.2) 1.00 (0.00) 1.00 (0.01) brazil 29.4 (0.9) 26.5 (0.9) 32.9 (1.0) 1.00 (0.03) 0.96 (0.04) bulgaria 24.5 (0.8) 21.1 (0.8) 28.6 (1.0) 1.15 (0.03) 1.05 (0.03) colombia 24.6 (0.7) 21.9 (0.7) 27.9 (0.8) 1.01 (0.04) 0.95 (0.05) croatia 36.9 (0.9) 33.6 (0.9) 41.0 (1.0) 1.05 (0.02) 1.04 (0.02) cyprus* 33.4 (0.4) 30.6 (0.5) 36.9 (0.5) 1.01 (0.02) 0.96 (0.02) hong kong-china 53.6 (0.8) 49.5 (0.9) 58.5 (0.8) 1.05 (0.03) 0.99 (0.03) macao-china 53.6 (0.5) 49.4 (0.5) 58.6 (0.7) 1.06 (0.02) 0.99 (0.03) malaysia 28.4 (0.8) 25.8 (0.8) 31.6 (0.8) 1.01 (0.02) 0.92 (0.02) montenegro 26.9 (0.4) 24.1 (0.4) 30.3 (0.5) 1.05 (0.02) 1.00 (0.03) russian federation 41.2 (0.8) 37.7 (0.8) 45.4 (0.9) 1.00 (0.04) 1.00 (0.04) Serbia 38.1 (0.8) 35.1 (0.8) 41.7 (0.8) 1.01 (0.02) 1.00 (0.02) Shanghai-china 52.6 (0.8) 48.3 (0.9) 57.7 (0.9) 1.06 (0.03) 0.97 (0.03) Singapore 58.3 (0.7) 53.8 (0.7) 63.8 (0.8) 1.10 (0.03) 1.10 (0.03) chinese taipei 52.3 (0.8) 47.1 (0.8) 58.5 (0.9) 1.16 (0.03) 1.11 (0.03) united arab Emirates 28.1 (0.5) 24.4 (0.5) 32.5 (0.7) 1.09 (0.03) 1.04 (0.03) uruguay 25.8 (0.6) 23.3 (0.6) 28.8 (0.7) 1.01 (0.02) 0.93 (0.02) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies are added to the estimation. 2. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies and response-format dummies interacted with country/economy dummies are added to the estimation. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003687 170 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.3.5 [Part 1/1] Performance in problem solving, by response format average proportion of full-credit responses items requiring simple or complex multiple-choice selections (14 items) Partners OECD all items (42 items) australia items requiring constructed responses (28 items) relative likelihood of success on constructed response items, based on success in performing all other tasks, accounting for booklet effects (oEcd average = 1.00)1, 2 % S.E. % S.E. % S.E. odds ratio S.E. 50.9 (0.4) 53.9 (0.5) 49.5 (0.5) 1.10 (0.02) austria 44.9 (0.8) 48.4 (0.9) 43.2 (0.9) 1.06 (0.03) belgium 46.4 (0.5) 49.5 (0.6) 44.9 (0.6) 1.09 (0.02) canada 51.3 (0.6) 54.9 (0.8) 49.5 (0.6) 1.05 (0.02) chile 32.9 (0.8) 37.7 (0.9) 30.5 (0.8) 0.95 (0.03) czech republic 45.0 (0.7) 48.9 (0.7) 43.1 (0.7) 0.98 (0.02) denmark 44.3 (0.8) 47.4 (1.0) 42.8 (0.8) 1.08 (0.03) Estonia 47.1 (0.7) 50.6 (0.8) 45.4 (0.8) 1.06 (0.03) finland 49.3 (0.5) 52.6 (0.6) 47.6 (0.6) 1.01 (0.02) france 48.5 (0.7) 52.4 (0.9) 46.5 (0.7) 1.02 (0.03) Germany 47.4 (0.7) 51.1 (0.8) 45.5 (0.8) 1.05 (0.03) hungary 35.4 (0.9) 40.6 (1.0) 32.8 (0.9) 0.93 (0.03) ireland 44.6 (0.8) 47.6 (1.0) 43.1 (0.8) 1.09 (0.04) israel 37.1 (1.3) 43.5 (1.3) 33.9 (1.4) 0.86 (0.03) italy 47.8 (0.9) 52.1 (1.1) 45.7 (0.9) 1.01 (0.03) Japan 56.9 (0.7) 63.1 (0.8) 53.8 (0.7) 0.89 (0.02) korea 58.1 (0.9) 65.6 (1.0) 54.4 (1.0) 0.81 (0.02) netherlands 47.9 (1.1) 51.3 (1.0) 46.2 (1.3) 1.00 (0.02) norway 46.3 (0.9) 49.9 (0.9) 44.5 (1.0) 1.05 (0.04) Poland 41.3 (1.0) 46.3 (1.1) 38.7 (1.1) 0.96 (0.03) Portugal 42.7 (0.9) 46.3 (1.0) 40.9 (1.0) 1.05 (0.03) Slovak republic 40.7 (0.8) 45.1 (0.9) 38.5 (0.9) 1.00 (0.03) Slovenia 38.9 (0.7) 43.5 (0.8) 36.6 (0.7) 0.98 (0.03) Spain 40.7 (0.8) 44.7 (0.8) 38.7 (0.9) 1.02 (0.03) Sweden 43.8 (0.7) 48.8 (0.9) 41.3 (0.7) 0.96 (0.03) turkey 33.8 (0.9) 38.1 (0.9) 31.6 (0.9) 0.93 (0.02) England (united kingdom) 48.5 (1.1) 51.1 (1.2) 47.2 (1.1) 1.06 (0.02) united States 46.2 (1.0) 50.1 (1.0) 44.2 (1.0) 1.03 (0.03) oEcd average 45.0 (0.2) 49.1 (0.2) 42.9 (0.2) 1.00 (0.01) brazil 29.4 (0.9) 34.3 (1.1) 26.9 (0.9) 0.92 (0.03) bulgaria 24.5 (0.8) 30.6 (0.9) 21.4 (0.8) 0.76 (0.02) colombia 24.6 (0.7) 29.8 (0.8) 22.0 (0.7) 0.87 (0.03) croatia 36.9 (0.9) 40.9 (0.8) 34.9 (0.9) 0.96 (0.02) cyprus* 33.4 (0.4) 38.6 (0.4) 30.9 (0.5) 0.88 (0.02) hong kong-china 53.6 (0.8) 60.7 (0.9) 50.0 (0.8) 0.84 (0.02) macao-china 53.6 (0.5) 61.0 (0.7) 49.8 (0.6) 0.82 (0.02) malaysia 28.4 (0.8) 34.4 (0.8) 25.4 (0.8) 0.81 (0.02) montenegro 26.9 (0.4) 31.3 (0.5) 24.7 (0.4) 0.89 (0.02) russian federation 41.2 (0.8) 45.8 (0.9) 38.9 (0.8) 0.98 (0.03) (0.02) Serbia 38.1 (0.8) 41.8 (0.8) 36.2 (0.8) 0.98 Shanghai-china 52.6 (0.8) 61.2 (0.9) 48.3 (0.9) 0.77 (0.02) Singapore 58.3 (0.7) 63.3 (0.8) 55.8 (0.7) 0.95 (0.03) chinese taipei 52.3 (0.8) 59.3 (0.8) 48.7 (0.9) 0.84 (0.02) united arab Emirates 28.1 (0.5) 33.8 (0.6) 25.2 (0.6) 0.86 (0.02) uruguay 25.8 (0.6) 31.1 (0.7) 23.1 (0.6) 0.82 (0.02) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. This classiication is not independent of the classiication of items by process or context (personal/social). Items measuring the process of “exploring and understanding” and items related to social contexts are under-represented among constructed-response items. 2. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies and response-format dummies interacted with country/economy dummies are added to the estimation. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003687 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 171 Annex b1: reSulTS For counTrIeS And economIeS table v.3.6 [Part 1/1] relative performance on knowledge-acquisition and knowledge-utilisation tasks relative likelihood of success on knowledge-acquisition tasks, based on success on knowledge-utilisation tasks (oEcd average = 1.00) average proportion of full-credit responses Partners OECD knowledge-acquisition tasks1 (18 items) australia knowledge-utilisation tasks2 (17 items) accounting for booklet effects3 accounting for booklet and country/economy-speciic response-format effects4 % S.E. % S.E. odds ratio S.E. odds ratio S.E. 52.3 (0.5) 51.5 (0.5) 1.11 (0.02) 1.16 (0.02) austria 45.7 (0.9) 47.4 (0.9) 0.99 (0.03) 1.03 (0.03) belgium 47.0 (0.6) 47.5 (0.6) 1.05 (0.02) 1.08 (0.03) canada 52.6 (0.8) 52.1 (0.6) 1.08 (0.03) 1.12 (0.03) chile 30.9 (0.9) 35.2 (0.8) 0.85 (0.03) 0.79 (0.03) czech republic 45.0 (0.8) 46.9 (0.6) 0.87 (0.02) 0.85 (0.02) denmark 44.2 (0.9) 48.1 (0.8) 0.94 (0.03) 0.98 (0.03) Estonia 46.8 (0.9) 49.5 (0.8) 0.94 (0.03) 0.96 (0.03) finland 50.2 (0.6) 51.0 (0.6) 0.91 (0.02) 0.91 (0.02) france 49.6 (0.8) 49.4 (0.8) 1.07 (0.03) 1.07 (0.04) Germany 47.5 (1.0) 49.5 (0.8) 0.97 (0.03) 1.00 (0.04) hungary 35.2 (1.0) 37.6 (0.9) 0.95 (0.03) 0.91 (0.03) ireland 44.6 (1.0) 45.5 (0.8) 1.04 (0.03) 1.06 (0.04) israel 38.7 (1.4) 37.0 (1.3) 1.13 (0.03) 1.09 (0.04) italy 49.5 (1.1) 48.0 (0.9) 1.15 (0.03) 1.17 (0.04) Japan 59.1 (0.8) 56.3 (0.7) 1.20 (0.03) 1.17 (0.03) korea 62.8 (1.1) 54.5 (0.9) 1.53 (0.05) 1.51 (0.05) netherlands 48.2 (1.2) 49.7 (1.1) 0.89 (0.02) 0.89 (0.02) norway 47.7 (1.0) 48.1 (1.0) 1.05 (0.03) 1.09 (0.04) Poland 41.3 (1.2) 43.7 (1.0) 0.96 (0.03) 0.94 (0.03) Portugal 41.6 (1.1) 45.7 (1.0) 0.91 (0.03) 0.90 (0.03) Slovak republic 40.5 (1.0) 43.2 (0.9) 0.94 (0.03) 0.94 (0.04) Slovenia 37.8 (0.9) 42.3 (0.7) 0.86 (0.02) 0.84 (0.03) Spain 40.0 (0.8) 42.3 (0.9) 0.96 (0.03) 0.95 (0.03) Sweden 45.2 (1.0) 44.6 (0.7) 1.08 (0.04) 1.08 (0.04) turkey 32.8 (1.0) 36.0 (0.9) 0.81 (0.02) 0.77 (0.02) England (united kingdom) 49.6 (1.2) 49.1 (1.0) 0.96 (0.02) 0.98 (0.02) united States 46.5 (1.1) 47.1 (1.0) 1.04 (0.03) 1.05 (0.04) oEcd average 45.5 (0.2) 46.4 (0.2) 1.00 (0.01) 1.00 (0.01) brazil 28.0 (1.1) 32.0 (1.1) 0.87 (0.03) 0.81 (0.04) bulgaria 23.7 (0.9) 26.7 (0.8) 0.80 (0.02) 0.68 (0.02) colombia 21.8 (0.8) 27.7 (0.8) 0.75 (0.03) 0.65 (0.03) croatia 35.2 (1.0) 40.5 (0.9) 0.75 (0.02) 0.71 (0.02) cyprus* 33.6 (0.5) 34.8 (0.5) 0.89 (0.02) 0.83 (0.02) hong kong-china 57.7 (1.0) 51.1 (0.8) 1.41 (0.04) 1.39 (0.05) macao-china 58.3 (0.7) 51.3 (0.5) 1.44 (0.05) 1.44 (0.05) malaysia 29.1 (0.9) 29.3 (0.7) 0.92 (0.02) 0.83 (0.02) montenegro 25.6 (0.5) 30.0 (0.5) 0.75 (0.02) 0.68 (0.02) russian federation 40.4 (1.0) 43.8 (0.8) 0.92 (0.03) 0.90 (0.04) (0.02) Serbia 37.7 (0.9) 40.7 (0.8) 0.84 (0.02) 0.82 Shanghai-china 56.9 (1.0) 49.8 (0.7) 1.45 (0.04) 1.43 (0.05) Singapore 62.0 (0.8) 55.4 (0.7) 1.42 (0.04) 1.46 (0.04) chinese taipei 56.9 (1.0) 50.1 (0.8) 1.43 (0.04) 1.43 (0.05) united arab Emirates 28.4 (0.6) 29.0 (0.6) 1.02 (0.03) 0.96 (0.03) uruguay 24.8 (0.7) 27.9 (0.7) 0.79 (0.02) 0.70 (0.02) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. “knowledge-acquisition tasks” are tasks measuring the processes of “exploring and understanding” or “representing and formulating”. 2. “knowledge-utilisation tasks” are tasks measuring the process of “planning and executing”. 3. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies are added to the estimation. 4. generalised odds ratios estimated with logistic regression on the pooled PISA sample. The average logit coeficient on country dummies for OECD countries is set at 0; booklet dummies and response-format dummies interacted with country/economy dummies are added to the estimation. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003687 172 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.1 [Part 1/2] Strength of the relationship between problem-solving and mathematics performance, between and within schools1 variation accounted for by students’ performance in mathematics4 variation in student performance in problem solving OECD total2 Within schools3 total between schools Within schools variance S.E. variance S.E. variance S.E. % % % 9 482 (198) 2 569 (178) 6 951 (106) 69.4 53.8 75.4 austria 8 801 (550) 4 183 (532) 4 505 (121) 63.2 71.3 59.1 belgium 11 314 (393) 5 412 (513) 5 804 (144) 65.3 73.8 57.7 canada 10 063 (333) 2 271 (236) 7 692 (168) 57.8 32.7 63.8 chile 7 382 (289) 3 153 (299) 4 123 (90) 63.7 69.4 59.3 czech republic 9 056 (389) 4 366 (473) 4 474 (174) 77.5 84.1 70.3 denmark 8 522 (354) 2 441 (326) 6 048 (164) 58.8 29.0 71.3 Estonia 7 658 (267) 1 826 (245) 5 868 (171) 69.1 48.8 75.5 finland 8 658 (225) 884 (120) 7 753 (183) 69.7 27.0 74.6 france 9 250 (786) w w w w 68.5 w w Germany 9 703 (486) 5 328 (471) 4 334 (111) 69.6 73.1 64.9 hungary 10 907 (568) 6 445 (683) 4 245 (113) 68.5 80.2 48.9 8 676 (365) 2 117 (272) 6 486 (162) 63.5 46.8 68.9 israel 15 230 (792) 7 751 (860) 7 429 (199) 72.9 77.9 66.2 italy 8 219 (376) 3 461 (360) 4 496 (131) 56.6 65.8 49.1 Japan 7 251 (325) 2 459 (280) 4 768 (124) 57.0 77.8 45.9 korea 8 311 (321) 2 604 (288) 5 575 (197) 64.4 75.1 59.1 netherlands 9 783 (592) 5 649 (634) 4 147 (146) 71.3 78.4 61.3 norway 10 600 (395) 2 264 (340) 8 270 (237) 62.8 22.9 73.7 Poland 9 303 (645) 3 357 (675) 5 930 (204) 56.5 41.9 64.8 Portugal 7 712 (281) 2 314 (240) 5 420 (157) 64.7 62.5 65.9 Slovak republic 9 597 (539) 4 761 (569) 4 625 (161) 72.9 76.6 68.9 Slovenia 9 428 (251) 5 114 (434) 4 272 (153) 66.2 73.5 58.7 10 890 (596) 3 121 (470) 7 776 (213) 55.6 32.8 64.7 Sweden 9 260 (349) 1 720 (321) 7 474 (182) 65.5 35.7 72.0 turkey 6 246 (349) 3 239 (385) 2 997 (89) 70.0 83.4 55.6 England (united kingdom) 9 342 (459) 2 735 (386) 6 606 (179) 73.4 65.6 76.8 united States 8 610 (419) 2 485 (410) 6 106 (165) 73.7 59.9 79.2 oEcd average 9 259 (85) 3 548 (87) 5 646 (30) 66.0 60.3 65.0 australia ireland Spain Partners between schools3 brazil 8 421 (434) 3 988 (491) 4 435 (153) 68.2 68.6 68.3 11 347 (752) 6 294 (750) 4 994 (125) 65.1 73.8 51.9 colombia 8 397 (358) 3 092 (332) 5 262 (156) 54.4 58.0 53.0 croatia 8 472 (361) 3 426 (403) 5 042 (137) 71.9 78.8 67.2 cyprus* 9 781 (195) 3 448 (1 455) 6 641 (167) 64.0 70.7 62.0 hong kong-china 8 401 (403) 3 034 (365) 5 347 (160) 57.0 70.8 49.2 macao-china 6 269 (129) 1 078 (237) 5 040 (167) 63.6 84.5 58.5 malaysia 6 982 (330) 2 614 (306) 4 361 (162) 69.4 72.0 67.6 montenegro 8 390 (200) 3 212 (670) 5 178 (163) 65.5 79.6 56.3 russian federation 7 725 (353) 2 857 (393) 4 872 (145) 54.6 42.8 62.1 Serbia 7 942 (342) 2 935 (333) 4 949 (164) 69.0 76.6 64.2 Shanghai-china 8 082 (404) 3 333 (362) 4 723 (151) 70.4 76.9 65.6 Singapore 9 021 (182) 3 061 (362) 5 962 (159) 69.3 65.9 71.0 chinese taipei 8 266 (350) 3 214 (374) 5 010 (150) 74.6 82.6 69.4 11 134 (385) 5 607 (477) 5 504 (150) 63.5 68.4 57.3 9 457 (388) 4 000 (419) 5 446 (133) 63.0 66.2 60.5 bulgaria united arab Emirates uruguay 1. The total variation in student performance is calculated from the square of the standard deviation for all students. 2. In some countries/economies, sub-units within schools were sampled instead of schools; this may affect the estimation of between-school variance components (see Annex A3). 3. Due to the unbalanced clustered nature of the data, the sum of the between- and within-school variation components, as an estimate from a sample, does not necessarily add up to the total. 4. Based on the residual variation in a model with student performance in mathematics. 5. Based on the residual variation in a model with student performance in mathematics and school average performance in mathematics. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 173 Annex b1: reSulTS For counTrIeS And economIeS table v.4.1 [Part 2/2] Strength of the relationship between problem-solving and mathematics performance, between and within schools1 Partners OECD variation accounted for by students’ and schools’ performance in mathematics5 variation in student performance unique to problem solving5 total between schools Within schools % % % variance S.E. variance S.E. variance S.E. australia 69.7 55.4 75.4 2 868 (77) 1 145 (81) 1 712 (33) austria 63.3 71.8 59.1 3 232 (230) 1 179 (221) 1 841 (63) belgium 65.4 74.1 57.7 3 915 (160) 1 404 (169) 2 454 (69) canada 58.0 34.2 63.8 4 223 (186) 1 494 (155) 2 781 (93) chile 63.8 69.7 59.3 2 669 (115) 955 (106) 1 679 (42) czech republic 77.6 84.2 70.4 2 030 (112) 688 (112) 1 327 (51) denmark 58.9 29.3 71.3 3 507 (221) 1 726 (226) 1 734 (61) Estonia 69.1 48.9 75.5 2 369 (132) 934 (141) 1 439 (37) finland 70.1 29.4 74.6 2 592 (79) 624 (73) 1 967 (60) france 68.5 w w 2 910 (494) w w w w Germany 69.8 73.7 65.0 2 932 (166) 1 400 (161) 1 519 (47) hungary 69.2 82.9 48.9 3 357 (155) 1 103 (129) 2 169 (80) ireland 63.5 46.8 68.9 3 164 (128) 1 127 (137) 2 017 (54) israel 74.3 82.4 66.2 3 914 (192) 1 367 (166) 2 510 (109) italy 56.6 65.8 49.1 3 568 (180) 1 183 (152) 2 290 (77) Japan 57.2 78.8 46.0 3 105 (99) 522 (75) 2 577 (64) korea 64.5 75.3 59.1 2 954 (127) 644 (81) 2 278 (94) netherlands 71.3 78.6 61.3 2 808 (227) 1 208 (215) 1 604 (47) norway 63.0 24.4 73.7 3 917 (246) 1 711 (238) 2 175 (66) Poland 56.8 42.9 64.8 4 019 (445) 1 917 (436) 2 088 (80) Portugal 64.7 62.6 65.9 2 722 (162) 865 (125) 1 847 (58) Slovak republic 73.0 76.9 68.9 2 593 (124) 1 098 (124) 1 437 (53) Slovenia 66.2 73.6 58.7 3 183 (97) 1 351 (140) 1 763 (69) Spain 55.6 33.0 64.7 4 835 (400) 2 092 (336) 2 743 (79) Sweden 65.6 36.2 72.0 3 186 (190) 1 098 (175) 2 092 (71) turkey 70.0 83.4 55.6 1 873 (72) 538 (69) 1 330 (33) England (united kingdom) 73.5 65.9 76.8 2 478 (132) 933 (126) 1 534 (41) united States 73.7 59.9 79.2 2 265 (173) 996 (181) 1 270 (38) oEcd average 66.2 61.0 65.0 3 114 (40) 1 177 (37) 1 907 (12) brazil 68.2 68.8 68.3 2 674 (158) 1 244 (172) 1 406 (44) bulgaria 66.1 77.2 51.9 3 845 (234) 1 432 (209) 2 400 (87) colombia 54.5 58.3 53.0 3 817 (229) 1 289 (146) 2 474 (147) croatia 71.9 78.8 67.2 2 384 (92) 727 (87) 1 653 (43) cyprus* 64.0 70.7 62.0 3 518 (124) 1 010 (212) 2 523 (89) hong kong-china 57.1 70.9 49.2 3 606 (160) 882 (114) 2 719 (89) macao-china 63.8 85.8 58.5 2 269 (60) 154 (51) 2 090 (69) total between schools Within schools malaysia 69.8 73.4 67.6 2 111 (93) 696 (78) 1 412 (50) montenegro 66.3 83.0 56.3 2 828 (109) 547 (114) 2 261 (84) russian federation 54.7 42.9 62.1 3 502 (199) 1 631 (193) 1 848 (63) Serbia 69.1 77.2 64.2 2 456 (111) 669 (99) 1 772 (50) Shanghai-china 70.4 77.0 65.6 2 395 (123) 766 (108) 1 626 (44) Singapore 69.4 66.5 71.0 2 756 (61) 1 026 (136) 1 729 (40) chinese taipei 74.6 82.6 69.4 2 101 (86) 558 (77) 1 534 (37) united arab Emirates 64.3 71.2 57.3 3 978 (151) 1 614 (161) 2 350 (83) uruguay 63.0 66.4 60.5 3 496 (176) 1 344 (169) 2 149 (58) 1. The total variation in student performance is calculated from the square of the standard deviation for all students. 2. In some countries/economies, sub-units within schools were sampled instead of schools; this may affect the estimation of between-school variance components (see Annex A3). 3. Due to the unbalanced clustered nature of the data, the sum of the between- and within-school variation components, as an estimate from a sample, does not necessarily add up to the total. 4. Based on the residual variation in a model with student performance in mathematics. 5. Based on the residual variation in a model with student performance in mathematics and school average performance in mathematics. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 174 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.2 [Part 1/1] Performance in problem solving and programme orientation Percentage of students OECD General programmes (G) Performance in problem solving modular programmes General programmes (G) S.E. mean score vocational (incl. prevocational) study programmes (v) S.E. mean score modular programmes S.E. mean score after accounting for sociodemographic characteristics of students1 observed S.E. Score dif. S.E. Score dif. S.E. % S.E. % S.E. % australia 89.1 (0.5) 10.9 (0.5) 0.0 c 526 (2.0) 497 (3.3) c c -29 (3.5) -22 (3.3) austria 30.7 (0.9) 69.3 (0.9) 0.0 c 534 (7.9) 494 (3.6) c c -40 (8.6) -28 (7.9) belgium 56.0 (1.1) 44.0 (1.1) 0.0 c 541 (3.3) 465 (3.5) c c -76 (5.0) -57 (4.8) canada 0.0 c 0.0 c 100.0 c c c c c 526 (2.4) c c c c 97.2 (0.2) 2.8 (0.2) 0.0 c 448 (3.7) 446 (9.4) c c -2 (8.7) 17 (8.2) chile czech republic 69.0 (1.2) 31.0 (1.2) 0.0 c 515 (3.9) 496 (4.9) c c -19 (6.1) -13 (5.7) 100.0 c 0.0 c 0.0 c 497 (2.9) c c c c c c c c Estonia 99.6 (0.2) 0.4 (0.2) 0.0 c 515 (2.5) c c c c c c c c finland 100.0 c 0.0 c 0.0 c 523 (2.3) c c c c c c c c france 84.7 (1.2) 15.3 (1.2) 0.0 c 518 (3.8) 474 (7.1) c c -44 (8.1) -31 (7.9) Germany 98.0 (0.9) 2.0 (0.9) 0.0 c 510 (3.6) 446 (13.4) c c -64 (14.1) -61 (13.0) hungary 85.7 (1.1) 14.3 (1.1) 0.0 c 475 (4.1) 361 (10.2) c c -114 (10.7) -83 (11.8) ireland 99.2 (0.2) 0.8 (0.2) 0.0 c 499 (3.2) 400 (13.7) c c -99 (13.7) -77 (13.9) israel 96.9 (0.2) 3.1 (0.2) 0.0 c w w w w c c w w w w italy 48.5 (1.6) 51.5 (1.6) 0.0 c 530 (5.4) 490 (5.8) c c -40 (8.2) -36 (8.2) Japan 75.8 (0.8) 24.2 (0.8) 0.0 c 560 (3.6) 529 (6.3) c c -31 (7.2) -22 (6.8) korea 80.1 (1.4) 19.9 (1.4) 0.0 c 572 (4.7) 518 (9.9) c c -54 (11.0) -42 (10.5) (8.6) denmark netherlands 77.8 (1.7) 22.2 (1.7) 0.0 c 538 (5.3) 417 (7.9) c c -121 (9.3) -108 norway 100.0 c 0.0 c 0.0 c 503 (3.3) c c c c c c c c Poland 99.9 (0.0) 0.1 (0.0) 0.0 c 481 (4.4) c c c c c c c c Portugal 83.3 (2.0) 16.7 (2.0) 0.0 c 504 (3.4) 446 (7.4) c c -58 (7.4) -38 (7.2) Slovak republic 65.7 (1.5) 8.2 (1.4) 26.1 (1.3) 488 (4.2) 407 (11.1) 496 (5.8) -81 (11.8) -60 (10.2) Slovenia 46.8 (0.5) 53.2 (0.5) 0.0 c 521 (2.7) 436 (1.7) c c -84 (3.2) -70 (3.8) Spain 99.2 (0.2) 0.8 (0.2) 0.0 c 478 (4.1) 361 (21.8) c c -116 (22.3) -100 (19.5) Sweden 99.6 (0.1) 0.4 (0.1) 0.0 c 491 (2.9) c c c c c c c c turkey 61.9 (0.5) 38.1 (0.5) 0.0 c 467 (5.8) 434 (4.1) c c -33 (6.9) -25 (5.9) (15.5) England (united kingdom) united States oEcd average Partners vocational (incl. prevocational) study programmes (v) difference in problem-solving performance: Students in vocational programmes minus students in general programmes (v - G) brazil 98.8 (0.2) 1.2 (0.2) 0.0 c 518 (4.2) 445 (14.8) c c -72 (15.0) -70 100.0 c 0.0 c 0.0 c 508 (3.9) c c c c c c c c 80.1 (0.2) 15.4 (0.2) 4.5 (0.0) 508 (0.8) 443 (2.3) 511 (3.1) -67 (2.4) -59 (2.3) 100.0 c 0.0 c 0.0 c 428 (4.7) c c c c c c c c bulgaria 59.2 (1.6) 40.8 (1.6) 0.0 c 420 (6.2) 375 (8.5) c c -45 (10.6) -26 (8.7) colombia 74.8 (2.3) 25.2 (2.3) 0.0 c 391 (3.8) 425 (5.5) c c 34 (6.1) 31 (5.2) croatia 29.9 (1.2) 70.1 (1.2) 0.0 c 531 (5.8) 439 (4.1) c c -93 (6.9) -89 (6.8) cyprus* 89.2 (0.1) 10.8 (0.1) 0.0 c 456 (1.5) 349 (3.1) c c -108 (3.2) -92 (4.0) 100.0 c 0.0 c 0.0 c 540 (3.9) c c c c c c c c macao-china 98.4 (0.1) 1.6 (0.1) 0.0 c 541 (1.0) 531 (7.6) c c -10 (7.6) -9 (7.5) malaysia 86.7 (1.2) 13.3 (1.2) 0.0 c 423 (3.9) 422 (6.6) c c -1 (7.6) 2 (6.7) montenegro 34.0 (0.2) 66.0 (0.2) 0.0 c 452 (2.5) 383 (1.4) c c -69 (2.9) -56 (3.3) russian federation 95.9 (1.1) 4.1 (1.1) 0.0 c 491 (3.3) 436 (14.1) c c -55 (13.9) -46 (11.5) Serbia 25.6 (1.0) 74.4 (1.0) 0.0 c 528 (6.2) 455 (3.8) c c -74 (7.4) -56 (8.0) Shanghai-china 78.8 (0.6) 21.2 (0.6) 0.0 c 548 (4.0) 493 (4.8) c c -56 (6.3) -42 (6.4) 100.0 c 0.0 c 0.0 c 562 (1.2) c c c c c c c c 65.5 (1.4) 34.5 (1.4) 0.0 c 551 (3.1) 503 (4.5) c c -47 (5.3) -35 (5.2) hong kong-china Singapore chinese taipei united arab Emirates 97.3 (0.0) 2.7 (0.0) 0.0 c 410 (2.8) 435 (5.2) c c 25 (5.8) 20 (6.3) uruguay 97.3 (0.4) 1.4 (0.4) 1.3 (0.3) 405 (3.4) 365 (25.3) 318 (16.0) -41 (25.0) -25 (21.7) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. The adjusted result corresponds to the coeficient from a regression where the PISA index of economic, social and cultural status (ESCS), ESCS squared, boy, and an immigrant (irst-generation) dummy are introduced as further independent variables. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 175 Annex b1: reSulTS For counTrIeS And economIeS table v.4.3 [Part 1/3] differences in problem-solving, mathematics, reading and science performance related to programme orientation Programme orientation effects: mean score difference between students in vocational programmes and students in general programmes OECD Problem solving reading computer-based mathematics Science S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. australia -29 (3.5) -34 (3.5) -34 (3.3) -31 (3.7) -29 (3.6) -34 (3.8) austria -40 (8.6) -38 (6.7) -55 (6.7) -42 (6.2) -32 (8.8) -27 (9.7) belgium -76 (5.0) -92 (4.4) -98 (4.3) -89 (4.2) -79 (4.5) -79 (5.5) canada c c c c c c c c c c c c -2 (8.7) -2 (7.2) -2 (7.8) -9 (7.5) -1 (6.9) -10 (8.1) czech republic -19 (6.1) -15 (5.5) -14 (5.2) -16 (5.6) m m m m denmark c c c c c c c c c c c c Estonia c c c c c c c c c c c c finland c c c c c c c c m m m m france -44 (8.1) -56 (7.2) -76 (8.6) -61 (9.5) -42 (6.5) -56 (9.9) Germany -64 (14.1) -37 (13.9) -59 (13.3) -57 (12.5) -25 (10.5) -38 (18.5) hungary -114 (10.7) -100 (5.8) -108 (7.8) -100 (6.7) -104 (11.3) -139 (12.4) -99 (13.7) -106 (11.6) -106 (14.4) -119 (13.5) -101 (13.6) -86 (14.4) w w w w w w w w w w w w italy -40 (8.2) -59 (7.1) -80 (7.4) -64 (7.6) -43 (8.0) -63 (8.2) Japan -31 (7.2) -52 (7.9) -51 (8.4) -43 (8.2) -41 (7.5) -31 (7.5) korea -54 (11.0) -88 (10.3) -67 (9.1) -67 (8.3) -73 (10.5) -50 (8.6) -121 (9.3) -132 (5.3) -132 (7.2) -133 (6.1) m m m m norway c c c c c c c c c c c c Poland c c c c c c c c c c c c -58 (7.4) -78 (6.1) -91 (6.0) -79 (5.8) -52 (5.7) -80 (6.3) Slovak republic -81 (11.8) -95 (10.2) -106 (14.1) -94 (13.1) -73 (11.4) -99 (12.7) Slovenia -84 (3.2) -94 (3.1) -99 (2.9) -93 (2.9) -86 (2.5) -105 (2.9) -116 (22.3) -114 (9.1) -134 (12.6) -114 (17.8) -88 (14.2) -150 (16.0) ireland israel netherlands Portugal Spain Sweden c c c c c c c c c c c c turkey -33 (6.9) -63 (7.8) -50 (7.1) -49 (6.4) m m m m England (united kingdom) -72 (15.0) -80 (13.0) -83 (13.9) -90 (12.7) m m m m c c c c c c c c c c c c -67 (2.4) -74 (2.0) -83 (2.2) -76 (2.0) -63 (2.4) -78 (3.1) united States oEcd average brazil bulgaria colombia c c c c c c c c c c c c -45 (10.6) -38 (7.6) -57 (11.1) -42 (8.7) m m m m 34 (6.1) 31 (5.5) 34 (5.8) 29 (5.2) 23 (5.8) 36 (7.1) croatia -93 (6.9) -105 (7.2) -105 (5.7) -93 (6.0) m m m m cyprus* -108 (3.2) -106 (3.0) -151 (4.3) -111 (3.4) m m m m c c c c c c c c c c c c -10 (7.6) -17 (8.0) 0 (7.6) -15 (7.5) -4 (7.0) 5 (9.3) m hong kong-china macao-china malaysia -1 (7.6) -16 (9.0) -9 (9.7) -12 (8.5) m m m montenegro -69 (2.9) -78 (2.7) -85 (3.0) -77 (2.5) m m m m russian federation -55 (13.9) -21 (8.9) -31 (12.8) -31 (11.2) -42 (13.1) -33 (19.8) Serbia -74 (7.4) -89 (9.3) -85 (9.4) -76 (8.8) m m m m Shanghai-china -56 (6.3) -92 (6.3) -69 (5.2) -76 (5.5) -75 (7.0) -63 (7.0) Singapore chinese taipei united arab Emirates uruguay c c c c c c c c c c c c -47 (5.3) -77 (5.4) -55 (5.3) -56 (4.1) -57 (5.1) -46 (5.8) 25 (5.8) 14 (5.4) 11 (5.5) 5 (6.3) 7 (5.3) 14 (6.0) -41 (25.0) -23 (17.2) -53 (21.2) -36 (22.2) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 176 digital reading Score dif. chile Partners mathematics © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.3 [Part 2/3] differences in problem-solving, mathematics, reading and science performance related to programme orientation Programme orientation effect size: Programme orientation effect divided by the variation in scores within each country/economy (standard deviation) OECD Problem solving reading computer-based mathematics Science digital reading Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. australia -0.30 (0.04) -0.35 (0.04) -0.35 (0.03) -0.31 (0.04) -0.31 (0.04) -0.35 (0.04) austria -0.43 (0.09) -0.41 (0.07) -0.59 (0.07) -0.46 (0.07) -0.36 (0.10) -0.26 (0.10) belgium -0.72 (0.04) -0.90 (0.04) -0.96 (0.04) -0.88 (0.04) -0.80 (0.04) -0.80 (0.05) canada c c c c c c c c c c c c chile -0.02 (0.10) -0.03 (0.09) -0.02 (0.10) -0.12 (0.09) -0.01 (0.08) -0.12 (0.10) czech republic -0.20 (0.06) -0.16 (0.06) -0.16 (0.06) -0.17 (0.06) m m m m denmark c c c c c c c c c c c c Estonia c c c c c c c c c c c c finland c c c c c c c c m m m m france -0.45 (0.09) -0.58 (0.07) -0.70 (0.07) -0.61 (0.09) -0.46 (0.07) -0.57 (0.10) Germany -0.65 (0.15) -0.38 (0.14) -0.65 (0.14) -0.59 (0.13) -0.27 (0.11) -0.38 (0.19) hungary -1.09 (0.09) -1.07 (0.06) -1.18 (0.08) -1.11 (0.07) -1.13 (0.11) -1.24 (0.09) ireland -1.07 (0.14) -1.26 (0.14) -1.23 (0.17) -1.31 (0.15) -1.26 (0.17) -1.05 (0.17) w w w w w w w w w w w w italy -0.44 (0.09) -0.63 (0.07) -0.81 (0.06) -0.67 (0.07) -0.52 (0.09) -0.66 (0.08) Japan -0.36 (0.08) -0.55 (0.08) -0.52 (0.08) -0.45 (0.08) -0.47 (0.08) -0.40 (0.09) korea -0.59 (0.12) -0.88 (0.09) -0.78 (0.10) -0.82 (0.09) -0.81 (0.11) -0.62 (0.10) netherlands -1.22 (0.08) -1.44 (0.05) -1.42 (0.06) -1.40 (0.06) m m m m norway c c c c c c c c c c c c Poland c c c c c c c c c c c c Portugal -0.65 (0.08) -0.83 (0.06) -0.97 (0.06) -0.89 (0.06) -0.61 (0.06) -0.90 (0.06) Slovak republic -0.78 (0.11) -0.88 (0.09) -0.95 (0.12) -0.86 (0.11) -0.80 (0.12) -0.97 (0.12) Slovenia -0.87 (0.03) -1.02 (0.03) -1.08 (0.03) -1.02 (0.03) -0.98 (0.03) -1.06 (0.03) Spain -1.12 (0.22) -1.31 (0.11) -1.45 (0.14) -1.32 (0.20) -1.07 (0.17) -1.53 (0.16) israel Sweden c c c c c c c c c c c c turkey -0.42 (0.08) -0.69 (0.07) -0.59 (0.08) -0.62 (0.07) m m m m England (united kingdom) -0.75 (0.16) -0.83 (0.14) -0.84 (0.15) -0.89 (0.13) m m m m c c c c c c c c c c c c -0.67 (0.02) -0.78 (0.02) -0.85 (0.02) -0.81 (0.02) -0.69 (0.03) -0.79 (0.03) united States oEcd average Partners mathematics brazil bulgaria colombia c c c c c c c c c c c c -0.43 (0.09) -0.40 (0.08) -0.48 (0.09) -0.41 (0.08) m m m m 0.37 (0.07) 0.41 (0.07) 0.41 (0.07) 0.39 (0.07) 0.31 (0.08) 0.39 (0.08) croatia -1.01 (0.06) -1.19 (0.06) -1.22 (0.05) -1.09 (0.05) m m m m cyprus* -1.09 (0.03) -1.14 (0.03) -1.36 (0.03) -1.15 (0.03) m m m m c c c c c c c c c c c c macao-china -0.13 (0.10) -0.18 (0.08) 0.00 (0.09) -0.20 (0.09) -0.04 (0.08) 0.06 (0.13) malaysia -0.01 (0.09) -0.19 (0.11) -0.10 (0.12) -0.15 (0.11) m m m m montenegro -0.76 (0.03) -0.94 (0.03) -0.92 (0.03) -0.91 (0.03) m m m m russian federation -0.63 (0.15) -0.24 (0.10) -0.34 (0.14) -0.36 (0.13) -0.52 (0.16) -0.39 (0.23) Serbia -0.83 (0.08) -0.98 (0.09) -0.92 (0.10) -0.87 (0.09) m m m m Shanghai-china -0.62 (0.07) -0.92 (0.06) -0.86 (0.07) -0.92 (0.07) -0.80 (0.07) -0.76 (0.08) hong kong-china Singapore chinese taipei united arab Emirates uruguay c c c c c c c c c c c c -0.52 (0.06) -0.66 (0.04) -0.60 (0.05) -0.67 (0.05) -0.65 (0.05) -0.52 (0.06) 0.23 (0.05) 0.15 (0.06) 0.12 (0.06) 0.05 (0.07) 0.08 (0.06) 0.13 (0.05) -0.42 (0.26) -0.27 (0.20) -0.56 (0.22) -0.38 (0.23) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 177 Annex b1: reSulTS For counTrIeS And economIeS table v.4.3 [Part 3/3] differences in problem-solving, mathematics, reading and science performance related to programme orientation difference in programme orientation effect sizes between problem solving (PS) and… … mathematics (PS - m) OECD Effect size dif. australia Effect size dif. S.E. … computer-based mathematics (PS - cbm) … Science (PS - S) Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. S.E. (0.03) 0.05 (0.03) 0.01 (0.03) 0.02 (0.03) 0.05 (0.03) austria -0.02 (0.07) 0.16 (0.07) 0.03 (0.07) -0.07 (0.06) -0.17 (0.09) belgium 0.19 (0.03) 0.25 (0.03) 0.17 (0.03) 0.08 (0.03) 0.08 (0.04) canada c c c c c c c c c c 0.01 (0.06) 0.00 (0.08) 0.10 (0.06) -0.01 (0.09) 0.10 (0.08) czech republic -0.04 (0.05) -0.03 (0.06) -0.02 (0.05) m m m m denmark c c c c c c c c c c Estonia c c c c c c c c c c finland c c c c c c m m m m france 0.12 (0.06) 0.25 (0.07) 0.16 (0.08) 0.01 (0.06) 0.12 (0.08) Germany -0.26 (0.10) 0.00 (0.12) -0.05 (0.11) -0.38 (0.12) -0.27 (0.16) hungary -0.03 (0.09) 0.08 (0.10) 0.01 (0.08) 0.03 (0.10) 0.15 (0.09) 0.19 (0.12) 0.16 (0.18) 0.24 (0.16) 0.19 (0.12) -0.02 (0.15) w w w w w w w w w w italy 0.20 (0.07) 0.37 (0.07) 0.24 (0.07) 0.08 (0.06) 0.23 (0.08) Japan 0.19 (0.06) 0.15 (0.06) 0.09 (0.06) 0.10 (0.05) 0.04 (0.05) korea 0.30 (0.07) 0.19 (0.10) 0.23 (0.10) 0.22 (0.09) 0.03 (0.09) netherlands 0.22 (0.06) 0.20 (0.06) 0.18 (0.06) m m m m norway c c c c c c c c c c Poland c c c c c c c c c c Portugal 0.18 (0.08) 0.32 (0.07) 0.24 (0.06) -0.04 (0.07) 0.24 (0.09) Slovak republic 0.10 (0.07) 0.17 (0.10) 0.08 (0.09) 0.02 (0.08) 0.19 (0.08) Slovenia 0.16 (0.03) 0.21 (0.03) 0.16 (0.02) 0.11 (0.02) 0.20 (0.02) Spain 0.20 (0.20) 0.33 (0.17) 0.20 (0.19) -0.04 (0.29) 0.42 (0.30) c c c c c c c c c c turkey 0.27 (0.05) 0.17 (0.07) 0.20 (0.06) m m m m England (united kingdom) 0.08 (0.11) 0.10 (0.12) 0.14 (0.10) m m m m c c c c c c c c c c 0.11 (0.02) 0.18 (0.02) 0.13 (0.02) 0.01 (0.03) 0.11 (0.03) ireland israel Sweden united States oEcd average brazil c c c c c c c c c c bulgaria -0.02 (0.06) 0.05 (0.07) -0.01 (0.07) m m m m colombia (0.06) -0.04 (0.05) -0.04 (0.06) -0.02 (0.06) 0.06 (0.06) -0.02 croatia 0.18 (0.03) 0.21 (0.04) 0.08 (0.05) m m m m cyprus* 0.05 (0.03) 0.27 (0.04) 0.06 (0.04) m m m m hong kong-china c c c c c c c c c c macao-china 0.05 (0.06) -0.13 (0.08) 0.07 (0.08) -0.08 (0.07) -0.19 (0.10) malaysia 0.18 (0.06) 0.09 (0.07) 0.14 (0.06) m m m m montenegro 0.18 (0.02) 0.16 (0.02) 0.15 (0.02) m m m m -0.38 (0.16) -0.29 (0.19) -0.27 (0.17) -0.10 (0.10) -0.24 (0.15) Serbia 0.16 (0.05) 0.10 (0.07) 0.05 (0.06) m m m m Shanghai-china 0.30 (0.06) 0.24 (0.06) 0.31 (0.07) 0.18 (0.07) 0.14 (0.08) russian federation Singapore chinese taipei united arab Emirates uruguay c c c c c c c c c c 0.14 (0.04) 0.08 (0.04) 0.15 (0.04) 0.13 (0.05) 0.00 (0.05) 0.08 (0.04) 0.12 (0.05) 0.18 (0.05) 0.15 (0.05) 0.10 (0.04) -0.15 (0.11) 0.15 (0.10) -0.04 (0.10) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 178 … digital reading (PS - dr) 0.06 chile Partners … reading (PS - r) © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.4 [Part 1/1] relative performance in problem solving, by programme orientation Problem-solving performance of students in vocational and pre-vocational programmes compared with that of students in general programmes with similar performance in mathematics, reading and science OECD Percentage of students average in vocational difference in Percentage Percentage Percentage programmes average problem solving of students of students of students average average who outperform difference in compared in vocational in vocational in vocational difference in difference in students problem solving with students programmes programmes programmes problem solving problem solving in general compared in general who outperform who outperform who outperform compared compared programmes with students programmes students students students with students with students with similar in general with similar in general in general in general in general in general performance programmes performance programmes programmes programmes programmes programmes with similar in mathematics, in mathematics, with similar with similar with similar with similar with similar reading performance reading performance performance performance performance performance and science2 in mathematics1 in mathematics2 and science3 in reading2 in science2 in reading1 in science1 australia austria belgium S.E. % S.E. Score dif. S.E. % S.E. Score dif. S.E. % S.E. Score dif. S.E. % S.E. -1 (2.4) 49.8 (1.9) -2 (2.4) 49.3 (1.9) -5 (2.5) 47.2 (2.0) -1 (2.3) 49.9 (2.0) -11 (7.8) 43.3 (6.1) 1 (8.4) 51.6 (5.8) -8 (8.1) 45.8 (5.9) -2 (8.2) 49.8 (6.4) 1 (4.0) 51.1 (2.7) 3 (4.6) 53.3 (2.7) -1 (4.3) 49.9 (2.9) 4 (4.1) 53.3 (2.9) canada c c c c c c c c c c c c c c c c chile 0 (4.9) 52.3 (5.2) 0 (6.5) 51.7 (6.1) 6 (5.3) 55.4 (5.3) 1 (4.7) 51.5 (5.1) czech republic -7 (4.6) 43.9 (4.4) -7 (5.6) 45.5 (4.3) -6 (4.8) 46.9 (3.9) -6 (4.5) 44.8 (4.6) denmark c c c c c c c c c c c c c c c c Estonia c c c c c c c c c c c c c c c c finland c c c c c c c c c c c c c c c c france 2 (6.0) 52.9 (4.9) 9 (6.9) 55.4 (4.9) 5 (7.1) 54.0 (5.4) 6 (6.0) 56.0 (5.3) (8.2) Germany -32 (9.1) 22.7 (7.3) -14 (10.7) 38.7 (11.3) -15 (10.5) 41.8 (8.8) -24 (8.8) 30.7 hungary -22 (11.1) 37.8 (6.9) -17 (11.1) 39.3 (6.2) -22 (9.4) 37.4 (6.2) -13 (11.0) 41.9 (6.9) ireland -6 (10.4) 44.3 (9.0) -14 (14.6) 42.8 (10.2) -4 (13.6) 45.6 (10.0) -1 (12.2) 48.0 (11.0) israel w w w w w w w w w w w w w w w w italy 5 (6.6) 54.4 (4.2) 14 (7.3) 58.5 (4.1) 8 (6.9) 56.4 (4.5) 12 (6.6) 58.6 (4.1) Japan 4 (4.8) 53.6 (3.3) -1 (4.7) 49.6 (3.2) -3 (5.1) 48.1 (3.3) 3 (4.8) 53.4 (3.3) korea 13 (7.1) 59.2 (5.2) 2 (8.9) 50.5 (5.9) 5 (8.8) 51.7 (6.0) 13 (7.8) 58.4 (5.6) netherlands -4 (9.6) 50.2 (5.9) -14 (9.9) 42.8 (5.9) -6 (10.0) 46.8 (6.7) 2 (10.5) 51.9 (7.6) norway c c c c c c c c c c c c c c c c Poland c c c c c c c c c c c c c c c c Portugal 0 (6.6) 50.5 (4.8) 0 (6.2) 50.9 (4.3) -1 (5.8) 50.4 (4.1) 4 (6.4) 53.1 (4.7) (8.1) Slovak republic -1 (7.9) 50.3 (7.6) 0 (9.9) 52.3 (7.2) -5 (9.5) 47.7 (7.2) 3 (8.6) 54.0 Slovenia -11 (5.3) 45.1 (3.3) -8 (6.1) 45.4 (3.6) -6 (4.5) 47.7 (2.8) -3 (5.0) 49.3 (3.3) Spain -14 (20.5) 43.9 (14.4) -18 (19.7) 38.0 (13.7) -23 (17.5) 37.1 (11.3) -9 (19.2) 43.3 (13.4) Sweden c c c c c c c c c c c c c c c c turkey 14 (4.1) 62.9 (3.8) 5 (5.7) 54.2 (4.5) 8 (5.2) 56.2 (4.2) 14 (4.2) 62.9 (4.1) England (united kingdom) -2 (10.1) 45.6 (10.7) -7 (10.9) 47.1 (11.5) 0 (9.5) 52.6 (10.0) 1 (9.5) 52.2 (11.6) c c c c c c c c c c c c c c c c -5 (2.4) 47.5 (1.6) -4 (2.2) 48.1 (1.7) -4 (2.1) 48.0 (1.5) 0 (2.2) 50.3 (1.7) united States oEcd average Partners Score dif. brazil bulgaria colombia c c c c c c c c c c c c c c c c -11 (7.5) 45.4 (4.4) -6 (8.0) 48.1 (4.4) -10 (7.9) 46.2 (4.4) -7 (7.5) 47.7 (4.6) 6 (4.4) 55.4 (3.7) 10 (5.2) 56.7 (3.8) 11 (5.0) 57.9 (3.5) 5 (4.5) 55.0 (3.9) croatia -2 (6.5) 49.3 (4.9) -16 (12.3) 39.8 (6.5) -22 (7.5) 35.5 (4.7) 3 (8.1) 52.5 (6.1) cyprus* (3.3) -19 (3.3) 38.4 (2.8) -20 (3.8) 40.6 (2.5) -27 (3.2) 35.8 (2.5) -14 (3.7) 42.6 hong kong-china c c c c c c c c c c c c c c c c macao-china 2 (4.9) 51.7 (6.3) -10 (5.7) 42.5 (7.6) 1 (5.6) 49.4 (7.2) 2 (4.8) 53.1 (5.3) malaysia montenegro russian federation Serbia 13 (4.4) 62.8 (4.4) 7 (4.9) 56.3 (4.2) 10 (4.1) 58.6 (3.7) 12 (4.3) 62.4 (4.5) 1 (2.4) 50.8 (1.8) -10 (3.1) 43.9 (2.3) -4 (2.6) 47.2 (1.8) 3 (2.7) 51.7 (2.0) -39 (13.4) 29.1 (6.9) -35 (14.4) 30.7 (7.3) -35 (13.1) 31.6 (7.1) -38 (13.6) 29.4 (8.1) 0 (6.3) 50.0 (4.7) -13 (8.5) 42.4 (5.2) -13 (8.3) 41.4 (5.7) 1 (6.4) 51.0 (5.0) (4.6) 17 (5.5) 64.0 (4.5) 9 (6.0) 57.3 (4.7) 14 (6.5) 60.7 (4.5) 18 (5.7) 65.0 Singapore c c c c c c c c c c c c c c c c chinese taipei 4 (3.8) 54.4 (3.5) -3 (4.1) 47.7 (3.3) 4 (4.0) 53.2 (3.2) 5 (3.7) 54.7 (3.3) 11 (4.1) 58.8 (4.2) 17 (4.4) 58.8 (5.6) 21 (4.7) 64.6 (4.3) 14 (4.1) 60.7 (4.6) -20 (13.1) 37.1 (9.6) -2 (13.3) 51.1 (9.6) -14 (12.4) 43.6 (9.3) -13 (11.1) 43.7 (10.6) Shanghai-china united arab Emirates uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. This column reports the difference between actual performance and the itted value from a regression using a cubic polynomial as regression function. 2. This column reports the percentage of students for whom the difference between actual performance and the itted value from a regression is positive. Values that are indicated in bold are signiicantly larger or smaller than 50%. 3. This column reports the difference between actual performance and the itted value from a regression using a second-degree polynomial as regression function (math, math sq., read, read sq., scie, scie sq., math×read, math×scie, read×scie). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 179 Annex b1: reSulTS For counTrIeS And economIeS table v.4.6 [Part 1/2] Percentage of students at each proiciency level in problem solving, by gender boys OECD below level 1 (below 358.49 score points) level 2 (from 423.42 to less than 488.35 score points) level 3 (from 488.35 to less than 553.28 score points) level 4 (from 553.28 to less than 618.21 score points) level 6 (above 683.14 score points) % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. 5.3 (0.4) 10.8 (0.7) 18.8 (0.6) 24.9 (0.9) 22.5 (0.8) 12.6 (0.7) 5.1 (0.5) austria 6.4 (1.1) 11.1 (1.1) 20.6 (1.3) 25.8 (1.4) 22.9 (1.3) 10.3 (1.0) 2.9 (0.6) belgium 9.4 (0.8) 11.6 (0.8) 17.0 (0.8) 23.2 (0.9) 22.3 (1.0) 12.7 (0.8) 3.8 (0.5) canada 5.3 (0.6) 9.6 (0.5) 18.1 (0.7) 25.1 (0.8) 23.0 (0.7) 13.1 (0.7) 5.9 (0.6) 14.4 (1.5) 21.2 (1.5) 27.2 (1.4) 23.9 (1.2) 10.5 (1.0) 2.6 (0.4) 0.3 (0.1) czech republic 7.2 (0.9) 10.6 (1.0) 19.7 (1.1) 26.3 (1.2) 22.8 (1.3) 10.6 (1.1) 2.8 (0.4) denmark 7.0 (0.9) 13.0 (1.0) 22.5 (1.0) 26.9 (1.3) 20.4 (1.5) 8.1 (1.0) 2.1 (0.4) Estonia 4.3 (0.6) 11.0 (1.0) 21.1 (1.0) 28.1 (1.2) 22.2 (1.2) 10.5 (0.8) 2.8 (0.4) finland 5.2 (0.6) 10.8 (0.7) 20.5 (1.0) 26.1 (1.3) 22.1 (1.0) 11.2 (0.7) 4.1 (0.6) france 7.1 (1.0) 9.6 (0.8) 20.0 (1.4) 26.6 (1.4) 23.0 (1.1) 11.3 (0.9) 2.6 (0.5) Germany 7.9 (0.9) 12.1 (1.1) 18.7 (1.2) 24.2 (1.1) 22.2 (1.2) 11.4 (1.3) 3.5 (0.6) hungary 19.0 (1.8) 16.5 (1.2) 22.0 (1.5) 21.5 (1.4) 13.9 (1.2) 5.5 (0.8) 1.5 (0.4) 7.5 (1.2) 13.1 (1.3) 22.7 (1.2) 27.2 (1.2) 18.6 (1.2) 8.0 (0.9) 3.0 (0.6) israel 24.0 (2.2) 15.2 (1.4) 17.0 (1.2) 17.1 (1.2) 14.9 (1.6) 8.6 (1.3) 3.2 (0.7) italy 5.6 (0.9) 10.7 (1.5) 19.4 (1.3) 25.7 (1.4) 24.0 (1.4) 11.9 (1.1) 2.7 (0.5) Japan 1.9 (0.5) 4.9 (0.6) 13.2 (1.0) 23.8 (1.3) 28.9 (1.4) 20.0 (1.5) 7.3 (0.9) korea 2.3 (0.4) 4.8 (0.7) 11.6 (1.1) 21.8 (1.3) 28.6 (1.5) 21.5 (1.4) 9.4 (1.1) netherlands 7.7 (1.2) 11.0 (1.2) 19.0 (1.3) 24.7 (1.6) 22.5 (1.7) 12.1 (1.4) 3.1 (0.6) norway 9.0 (0.9) 13.1 (0.9) 21.4 (1.2) 24.0 (1.0) 18.8 (1.1) 9.9 (1.0) 3.8 (0.5) Poland 11.8 (1.2) 15.5 (1.2) 23.4 (1.2) 24.2 (1.6) 16.9 (1.2) 6.6 (0.8) 1.5 (0.3) Portugal 6.3 (0.8) 12.8 (1.2) 23.2 (1.5) 27.7 (1.3) 20.6 (1.2) 7.7 (0.8) 1.7 (0.4) Slovak republic 9.4 (1.1) 14.9 (1.2) 23.2 (1.3) 23.7 (1.3) 18.1 (1.6) 8.3 (0.9) 2.4 (0.8) Slovenia 13.2 (0.8) 16.8 (1.3) 24.3 (1.6) 22.3 (1.2) 16.3 (1.0) 6.1 (0.7) 1.1 (0.4) Spain 14.1 (1.4) 15.6 (0.9) 21.5 (1.3) 23.5 (1.5) 16.2 (1.2) 7.0 (0.8) 2.2 (0.4) Sweden 10.2 (0.9) 14.8 (1.1) 23.1 (1.0) 24.8 (1.0) 17.6 (0.9) 7.3 (0.7) 2.2 (0.4) turkey 9.4 (1.2) 23.7 (1.6) 30.6 (1.8) 22.4 (1.4) 10.9 (1.3) 2.7 (0.6) 0.3 (0.1) England (united kingdom) 5.7 (1.1) 10.4 (1.0) 19.5 (1.3) 25.5 (1.3) 23.2 (1.3) 12.1 (1.3) 3.6 (0.9) united States 6.6 (1.0) 12.4 (1.1) 21.4 (1.3) 25.8 (1.2) 20.8 (1.2) 9.8 (0.9) 3.2 (0.5) oEcd average 8.7 (0.2) 12.8 (0.2) 20.7 (0.2) 24.5 (0.2) 20.2 (0.2) 10.0 (0.2) 3.1 (0.1) brazil 19.1 (1.8) 23.5 (1.5) 26.7 (1.5) 19.0 (1.8) 8.9 (1.3) 2.1 (0.5) 0.6 (0.3) bulgaria 36.7 (2.1) 22.7 (1.2) 20.9 (1.3) 12.9 (1.1) 5.3 (0.8) 1.4 (0.4) 0.2 (0.1) colombia 27.1 (1.9) 27.6 (1.4) 23.8 (1.3) 14.1 (1.1) 5.7 (0.7) 1.3 (0.4) 0.3 (0.1) croatia 12.2 (1.4) 18.7 (1.4) 24.6 (1.5) 22.4 (1.4) 15.3 (1.4) 5.6 (0.8) 1.2 (0.3) cyprus* 22.9 (0.8) 19.7 (1.1) 23.4 (1.1) 19.2 (1.1) 10.3 (1.0) 3.7 (0.4) 0.7 (0.3) hong kong-china 3.1 (0.6) 6.6 (0.8) 15.3 (1.0) 25.9 (1.5) 27.2 (1.2) 15.7 (1.3) 6.1 (0.8) macao-china 1.5 (0.3) 5.6 (0.7) 16.7 (0.9) 27.9 (1.2) 29.2 (1.1) 15.6 (0.8) 3.5 (0.5) malaysia 22.4 (1.7) 26.2 (1.5) 27.3 (1.5) 16.6 (1.2) 6.1 (0.9) 1.2 (0.4) 0.1 (0.1) montenegro 32.4 (1.0) 25.7 (1.1) 22.4 (1.0) 13.6 (0.8) 4.8 (0.7) 1.0 (0.3) 0.1 (0.1) russian federation 6.4 (0.7) 14.6 (1.1) 26.0 (1.2) 28.6 (1.8) 16.2 (1.0) 6.7 (1.0) 1.5 (0.4) Serbia 9.2 (1.2) 17.1 (1.2) 25.5 (2.0) 26.4 (1.6) 15.8 (1.1) 5.3 (0.6) 0.8 (0.3) Shanghai-china 2.6 (0.5) 6.2 (0.7) 15.0 (1.2) 25.6 (1.3) 27.8 (1.8) 17.0 (1.2) 5.7 (0.7) Singapore 2.3 (0.4) 6.3 (0.5) 13.0 (0.7) 20.1 (0.9) 25.8 (0.9) 20.4 (1.0) 12.0 (0.7) chinese taipei 4.2 (0.8) 7.9 (0.8) 15.8 (1.2) 23.9 (1.3) 25.9 (1.7) 17.3 (1.2) 5.0 (0.8) united arab Emirates 37.1 (2.0) 22.4 (1.5) 18.5 (1.0) 12.7 (0.9) 6.7 (0.7) 2.2 (0.3) 0.5 (0.1) uruguay 31.5 (1.8) 23.6 (1.3) 22.0 (1.3) 14.6 (1.1) 6.5 (0.8) 1.6 (0.4) 0.1 (0.1) ireland Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 180 level 5 (from 618.21 to less than 683.14 score points) australia chile Partners level 1 (from 358.49 to less than 423.42 score points) © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.6 [Part 2/2] Percentage of students at each proiciency level in problem solving, by gender increased increased likelihood of likelihood of boys scoring boys scoring level 5 level 4 level 3 level 2 level 1 below level 2 at or above level 6 below level 1 (from 358.49 to (from 423.42 to (from 488.35 to (from 553.28 to (from 618.21 to level 5 (less than (below 358.49 less than 423.42 less than 488.35 less than 553.28 less than 618.21 less than 683.14 (above 683.14 423.42 score (above 618.21 score points) score points) score points) score points) score points) score points) score points) score points) points) Girls OECD % % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. relative relative risk S.E. risk S.E. australia 4.7 (0.4) 10.1 (0.5) 20.0 (0.8) 26.7 (1.0) 22.7 (0.7) 12.0 (0.6) 3.7 (0.3) 1.09 (0.06) 1.13 (0.07) austria 6.5 (1.0) 12.8 (1.2) 23.1 (2.2) 28.0 (1.8) 20.9 (1.4) 7.6 (0.9) 1.1 (0.3) 0.91 (0.10) 1.52 (0.21) belgium 9.0 (0.8) 11.6 (0.8) 19.7 (1.2) 25.8 (0.9) 21.8 (0.9) 10.0 (0.8) 2.2 (0.3) 1.02 (0.08) 1.36 (0.11) canada 4.9 (0.4) 9.7 (0.6) 19.9 (1.0) 26.6 (1.0) 22.8 (0.8) 11.8 (0.7) 4.3 (0.4) 1.02 (0.05) 1.18 (0.06) chile czech republic 15.9 (1.5) 25.0 (1.2) 30.0 (1.3) 20.6 (1.3) 7.2 (0.8) 1.3 (0.3) 0.1 (0.0) 0.87 (0.05) 2.09 (0.57) 5.9 (0.8) 13.2 (1.2) 21.8 (1.4) 28.2 (1.3) 20.6 (1.2) 8.3 (0.8) 2.0 (0.4) 0.93 (0.09) 1.30 (0.14) denmark 7.6 (0.7) 13.1 (1.0) 25.6 (1.1) 28.8 (1.8) 17.7 (1.4) 6.2 (0.7) 1.0 (0.3) 0.96 (0.07) 1.41 (0.17) Estonia 3.8 (0.5) 11.1 (1.1) 22.4 (1.0) 30.2 (1.5) 22.2 (1.0) 8.6 (0.9) 1.6 (0.5) 1.03 (0.10) 1.30 (0.13) finland 3.7 (0.4) 8.9 (0.6) 19.5 (1.3) 28.2 (1.6) 25.1 (1.2) 11.6 (0.8) 3.0 (0.5) 1.27 (0.10) 1.05 (0.08) france 6.2 (1.0) 10.1 (0.9) 20.9 (1.2) 30.2 (1.4) 22.3 (1.2) 8.6 (0.9) 1.7 (0.4) 1.03 1.35 (0.13) (0.1) Germany 7.0 (0.9) 11.5 (1.0) 21.9 (1.1) 27.2 (1.4) 21.9 (1.2) 8.7 (0.9) 1.8 (0.4) 1.08 (0.07) 1.41 (0.13) hungary 15.6 (1.5) 18.9 (1.2) 25.7 (1.4) 23.3 (1.2) 12.3 (1.2) 3.7 (0.7) 0.5 (0.2) 1.03 (0.07) 1.67 (0.22) ireland israel 6.5 (0.7) 13.5 (1.0) 24.9 (1.2) 28.4 (1.1) 19.0 (1.0) 6.6 (0.7) 1.1 (0.3) 1.03 (0.10) 1.41 (0.20) 19.8 (1.3) 18.8 (1.0) 23.1 (1.0) 19.8 (1.0) 12.5 (0.9) 4.8 (0.6) 1.1 (0.3) 1.02 (0.07) 1.97 (0.31) italy 4.6 (0.8) 11.8 (1.2) 26.2 (1.6) 30.7 (1.5) 20.3 (1.6) 5.5 (1.0) 0.8 (0.3) 1.00 (0.14) 2.31 (0.37) Japan 1.7 (0.4) 5.8 (0.8) 16.1 (1.2) 30.3 (1.3) 29.5 (1.2) 13.6 (1.1) 3.2 (0.6) 0.92 (0.1) 1.63 (0.13) 1.30 (0.12) korea 2.0 (0.4) 4.7 (0.7) 14.5 (1.3) 25.9 (1.3) 29.1 (1.5) 18.3 (1.7) 5.5 (0.9) 1.06 (0.17) netherlands 7.0 (1.0) 11.4 (1.1) 20.8 (1.4) 27.4 (1.6) 21.5 (1.6) 9.8 (1.0) 2.2 (0.6) 1.02 (0.07) 1.26 (0.13) norway 7.2 (0.8) 13.3 (1.0) 21.5 (1.2) 25.4 (1.1) 20.1 (1.2) 9.5 (1.1) 3.0 (0.5) 1.08 (0.08) 1.09 (0.11) Poland 8.3 (1.2) 15.9 (1.4) 28.0 (1.4) 27.7 (1.3) 14.4 (1.2) 4.9 (0.8) 0.7 (0.3) 1.13 (0.1) 1.44 (0.20) Portugal 6.6 (0.7) 15.4 (1.1) 27.7 (1.2) 28.6 (1.6) 16.2 (1.0) 4.6 (0.6) 0.7 (0.3) 0.87 (0.05) 1.76 (0.21) Slovak republic Slovenia Spain 12.2 (1.5) 15.9 (1.6) 25.5 (1.5) 27.7 (1.8) 14.1 (1.3) 4.1 (0.6) 0.6 (0.3) 0.86 (0.07) 2.28 (0.30) 9.4 (0.8) 17.5 (1.0) 26.6 (1.6) 25.2 (1.3) 15.2 (1.1) 5.4 (0.9) 0.6 (0.2) 1.11 (0.06) 1.21 (0.24) 12.1 (1.2) 15.0 (1.0) 25.7 (1.1) 25.0 (1.2) 15.7 (1.0) 5.4 (0.6) 1.0 (0.3) 1.09 (0.06) 1.43 (0.16) Sweden 7.4 (0.8) 14.4 (0.9) 24.8 (1.3) 27.8 (1.2) 17.5 (0.9) 6.7 (0.8) 1.4 (0.3) 1.15 (0.08) 1.17 (0.14) turkey 12.6 (1.4) 25.9 (1.6) 32.3 (1.6) 20.0 (1.5) 7.9 (1.3) 1.3 (0.6) 0.0 (0.1) 0.86 (0.05) 2.36 (1.04) 5.4 (1.0) 11.2 (1.1) 20.8 (1.7) 27.5 (1.3) 22.2 (1.5) 9.9 (1.0) 3.0 (0.6) 0.97 (0.10) 1.22 (0.14) England (united kingdom) Partners S.E. united States 4.7 (0.7) 12.7 (1.2) 24.2 (1.3) 28.3 (1.3) 19.9 (1.2) 7.9 (0.8) 2.3 (0.5) 1.09 (0.1) 1.27 (0.12) oEcd average 7.8 (0.2) 13.5 (0.2) 23.3 (0.3) 26.8 (0.3) 19.0 (0.2) 7.7 (0.2) 1.8 (0.1) 1.02 (0.02) 1.50 (0.05) brazil 24.5 (1.9) 27.2 (1.9) 27.0 (1.6) 15.8 (1.7) 4.5 (0.7) 0.9 (0.3) 0.1 (0.1) 0.83 (0.03) 2.62 (0.67) bulgaria 29.8 (2.0) 24.0 (1.4) 23.3 (1.2) 15.3 (1.2) 6.0 (0.9) 1.4 (0.4) 0.2 (0.1) 1.10 (0.04) 1.00 (0.31) colombia 38.5 (1.9) 29.0 (1.3) 20.7 (1.3) 8.9 (0.9) 2.2 (0.5) 0.5 (0.2) 0.2 (0.1) 0.81 (0.03) 2.17 (0.82) croatia 11.9 (1.1) 21.9 (1.3) 29.2 (1.5) 23.4 (1.5) 11.1 (1.3) 2.2 (0.6) 0.3 (0.1) 0.92 (0.06) 2.71 (0.53) cyprus* 16.0 (0.8) 22.1 (0.9) 27.7 (1.4) 21.7 (1.5) 9.8 (0.7) 2.3 (0.4) 0.4 (0.2) 1.12 (0.05) 1.66 (0.35) 3.6 (0.6) 7.7 (1.2) 17.6 (1.4) 29.1 (2.0) 25.8 (1.3) 12.4 (1.5) 3.9 (1.0) 0.87 (0.11) 1.34 (0.16) hong kong-china 1.6 (0.3) 6.4 (0.6) 18.4 (0.8) 31.1 (1.1) 28.6 (1.2) 12.0 (0.8) 2.0 (0.3) 0.90 (0.11) 1.37 (0.10) malaysia macao-china 22.9 (1.7) 29.3 (1.4) 28.2 (1.3) 14.8 (1.2) 4.4 (0.6) 0.4 (0.3) 0.0 (0.0) 0.93 (0.04) 3.29 (2.44) montenegro 27.6 (1.1) 28.0 (1.2) 25.3 (1.5) 14.1 (1.0) 4.4 (0.6) 0.4 (0.2) 0.1 (0.1) 1.04 (0.03) 2.41 (1.50) 7.1 (0.9) 16.2 (1.5) 28.0 (1.2) 27.2 (1.5) 15.2 (1.2) 5.1 (0.7) 1.2 (0.4) 0.90 (0.06) 1.31 (0.16) 11.4 (1.1) 19.4 (1.1) 27.8 (2.0) 25.2 (1.5) 12.8 (0.9) 2.9 (0.5) 0.5 (0.2) 0.85 (0.06) 1.80 (0.35) russian federation Serbia Shanghai-china 3.5 (0.6) 8.8 (0.8) 19.9 (1.0) 29.2 (1.4) 24.6 (1.2) 11.4 (1.2) 2.6 (0.6) 0.72 (0.07) 1.63 (0.17) Singapore 1.7 (0.3) 5.5 (0.5) 14.6 (0.8) 23.8 (1.3) 28.3 (1.6) 19.0 (1.0) 7.1 (0.6) 1.20 (0.13) 1.24 (0.05) 2.7 (0.5) 8.5 (0.9) 19.8 (1.2) 28.6 (1.2) 25.9 (1.2) 12.0 (1.3) 2.5 (0.6) 1.07 (0.12) 1.54 (0.25) united arab Emirates chinese taipei 23.7 (1.4) 26.6 (1.3) 25.3 (1.0) 15.7 (0.8) 6.2 (0.6) 2.0 (0.3) 0.4 (0.1) 1.18 (0.05) 1.14 (0.20) uruguay 33.1 (1.9) 27.3 (1.6) 22.7 (1.2) 11.9 (0.9) 4.3 (0.6) 0.6 (0.2) 0.0 (0.0) 0.91 (0.03) 2.88 (0.99) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 181 Annex b1: reSulTS For counTrIeS And economIeS table v.4.7 [Part 1/3] mean score and variation in student performance in problem solving, by gender mean score Partners OECD boys australia Girls Standard deviation difference (b - G) Score dif. mean S.E. mean S.E. 524 (2.4) 522 (2.2) 2 boys Girls 5th percentile difference (b - G) Girls S.E. S.d. S.E. S.d. S.E. dif. S.E. Score S.E. Score S.E. (2.6) 100 (1.3) 95 (1.3) 5 (1.6) 355 (3.9) 361 (4.8) difference (b - G) Score dif. -5 S.E. (5.3) austria 512 (4.4) 500 (4.1) 12 (4.8) 98 (4.0) 90 (2.7) 8 (3.3) 345 (11.3) 344 (9.8) 0 (12.7) belgium 512 (3.1) 504 (3.1) 8 (3.7) 110 (2.4) 102 (2.1) 8 (2.6) 313 321 (8.7) -8 (10.6) (8.7) canada 528 (2.8) 523 (2.5) 5 (2.2) 104 (2.6) 96 (1.3) 8 (2.4) 355 (5.4) 359 (5.0) -5 (6.9) chile 455 (4.5) 441 (3.7) 13 (3.8) 89 (2.2) 82 (1.9) 7 (2.3) 303 (7.1) 304 (6.2) -1 (6.7) czech republic 513 (3.9) 505 (3.5) 8 (4.1) 98 (2.6) 92 (2.3) 6 (2.9) 334 (10.4) 351 (7.3) -17 (9.9) denmark 502 (3.7) 492 (2.9) 10 (3.1) 94 (2.3) 90 (2.1) 5 (2.2) 342 336 (6.4) 6 (8.4) (7.6) Estonia 517 (3.3) 513 (2.6) 5 (3.1) 91 (2.0) 84 (1.7) 6 (2.1) 366 (6.5) 369 (5.5) -3 (7.6) finland 520 (2.8) 526 (2.6) -6 (3.0) 96 (1.5) 89 (1.6) 7 (2.0) 355 (6.1) 373 (4.7) -18 (7.5) (13.8) france 513 (4.0) 509 (3.5) 5 (3.1) 100 (4.3) 93 (4.5) 7 (3.3) 335 (13.1) 344 (13.1) -8 Germany 512 (4.1) 505 (3.7) 7 (2.9) 103 (2.8) 94 (2.5) 9 (2.2) 333 338 (8.6) -5 (8.1) hungary 461 (5.0) 457 (4.3) 3 (4.8) 110 (3.3) 99 (3.3) 12 (3.8) 272 (10.1) 286 (14.2) -14 (16.8) (7.9) ireland 501 (4.8) 496 (3.2) 5 (5.0) 97 (3.1) 89 (1.8) 9 (3.4) 336 israel 457 (8.9) 451 (4.1) 6 (8.5) 134 (4.1) 112 (2.8) 22 (3.3) 227 (13.8) (9.7) (7.4) -8 (11.5) 259 (10.2) 343 -32 (13.3) (13.1) italy 518 (5.2) 500 (4.5) 18 (5.7) 97 (2.6) 82 (2.7) 15 (3.0) 351 (12.5) 362 (8.4) -11 Japan 561 (4.1) 542 (3.0) 19 (3.7) 89 (2.5) 79 (2.0) 10 (2.3) 406 (9.0) 405 (6.8) 1 (8.7) korea 567 (5.1) 554 (5.1) 13 (5.5) 95 (2.5) 87 (2.0) 8 (2.9) 403 (8.7) 408 (6.9) -6 (9.7) (9.7) netherlands 513 (4.9) 508 (4.5) 5 (3.3) 101 (3.5) 96 (3.3) 5 (3.2) 334 (10.4) 339 (9.6) -5 norway 502 (3.6) 505 (3.8) -3 (3.6) 106 (2.4) 99 (2.2) 7 (2.5) 318 340 (7.1) -22 (8.4) Poland 481 (4.9) 481 (4.6) 0 (3.3) 103 (3.7) 90 (3.4) 14 (2.6) 306 (10.7) 331 (10.2) -25 (9.5) (8.1) Portugal 502 (4.0) 486 (3.6) 16 (2.6) 91 (1.9) 84 (1.8) 7 (1.8) 345 (7.2) 346 (5.5) -1 (6.6) Slovak republic 494 (4.2) 472 (4.1) 22 (4.4) 100 (3.4) 94 (2.8) 6 (3.2) 327 (7.4) 302 (9.7) 24 (9.2) (4.3) 325 Slovenia 474 (2.1) 478 (2.2) -4 (3.0) 102 (1.6) 91 (2.0) 11 (2.6) 300 Spain 478 (4.8) 476 (4.1) 2 (3.4) 109 (3.3) 99 (3.1) 10 (2.7) 285 (12.9) (6.9) -25 (7.4) 301 (10.0) -16 (10.5) (10.3) Sweden 489 (3.7) 493 (3.1) -4 (3.6) 101 (2.4) 91 (2.0) 9 (2.7) 317 (7.4) 340 (8.1) -22 turkey 462 (4.3) 447 (4.6) 15 (4.0) 81 (2.4) 77 (2.6) 4 (2.3) 334 (6.4) 324 (4.4) 10 (6.9) England (united kingdom) 520 (5.4) 514 (4.6) 6 (5.5) 98 (3.0) 95 (2.9) 4 (3.4) 351 (11.8) 353 (10.5) -2 (14.5) united States 509 (4.2) 506 (4.2) 3 (3.1) 97 (3.0) 88 (2.0) 9 (2.5) 345 (9.4) 361 (7.4) -16 (8.8) oEcd average 503 (0.8) 497 (0.7) 7 (0.8) 100 (0.5) 91 (0.5) 8 (0.5) 332 (1.7) 340 (1.6) -8 (1.9) (9.7) 272 (6.7) 10 (8.6) 237 (10.7) -32 (10.8) brazil 440 (5.4) 418 (4.6) 22 (3.3) 95 (3.1) 87 (2.2) 8 (2.5) 282 bulgaria 394 (5.8) 410 (5.3) -17 (4.9) 110 (3.8) 102 (4.0) 8 (3.4) 205 (11.0) colombia 415 (4.1) 385 (3.9) 31 (3.8) 92 (2.3) 89 (2.3) 4 (2.5) 267 (6.6) 242 (6.3) 25 (6.2) croatia 474 (4.8) 459 (4.0) 15 (4.4) 98 (2.4) 85 (2.3) 13 (2.5) 311 (7.3) 318 (7.2) -7 (9.2) cyprus* 440 (1.8) 449 (2.0) -9 (2.5) 107 (1.5) 90 (1.3) 17 (1.9) 263 (6.4) 298 (5.8) -36 (6.5) hong kong-china 546 (4.6) 532 (4.8) 13 (5.2) 93 (2.3) 90 (3.1) 3 (3.1) 384 (9.1) 376 (7.2) 7 (8.1) macao-china 546 (1.5) 535 (1.3) 10 (2.0) 81 (1.3) 77 (1.3) 4 (2.0) 407 (4.6) 403 (4.5) 4 (5.6) malaysia 427 (3.9) 419 (4.0) 8 (3.7) 86 (2.5) 81 (1.9) 6 (2.1) 289 (5.6) 285 (6.3) 3 (6.5) montenegro 404 (1.8) 409 (1.8) -6 (2.8) 95 (1.8) 88 (1.4) 7 (2.5) 251 (5.7) 263 (4.7) -12 (6.9) russian federation 493 (3.9) 485 (3.7) 8 (3.1) 89 (2.2) 87 (2.5) 2 (2.6) 347 (6.0) 343 (6.0) 4 (7.9) Serbia 481 (3.8) 466 (3.2) 15 (3.5) 90 (2.5) 88 (2.2) 2 (2.6) 330 (7.5) 314 (6.5) 16 (6.8) Shanghai-china 549 (3.4) 524 (3.8) 25 (2.9) 90 (2.2) 88 (2.8) 3 (2.0) 390 (8.1) 373 (8.2) 17 (7.4) Singapore 567 (1.8) 558 (1.7) 9 (2.5) 100 (1.3) 89 (1.2) 11 (1.7) 394 (4.7) 402 (5.9) -8 (8.0) chinese taipei 540 (4.5) 528 (4.1) 12 (6.3) 96 (2.9) 85 (2.1) 11 (3.1) 369 (10.7) 384 (5.8) -16 (10.2) united arab Emirates 398 (4.6) 424 (3.2) -26 (5.6) 114 (2.9) 95 (2.2) 20 (3.8) 215 (9.0) 270 (6.3) -54 (10.6) uruguay 409 (4.0) 398 (3.8) 11 (3.4) 102 (2.2) 93 (2.2) 9 (1.8) 242 (7.6) 245 (6.7) -3 (7.1) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 182 boys © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.7 [Part 2/3] mean score and variation in student performance in problem solving, by gender 10th percentile Partners OECD boys australia Girls Score S.E. Score S.E. 392 (3.1) 400 (3.2) 25th percentile difference (b - G) Score dif. -8 boys Girls S.E. Score S.E. Score S.E. (3.6) 457 (3.3) 460 (2.9) 50th percentile (median) difference (b - G) Score dif. -3 boys Girls difference (b - G) Score dif. S.E. Score S.E. Score S.E. (3.6) 528 (3.0) 524 (2.6) 3 S.E. (3.3) austria 386 (8.9) 382 (7.1) 4 (8.4) 449 (5.6) 442 (5.7) 7 (6.9) 517 (4.7) 506 (4.9) 12 (6.1) belgium 363 (7.1) 365 (6.0) -3 (8.6) 441 (5.0) 440 (4.1) 1 (5.9) 522 (3.5) 513 (3.5) 9 (4.5) canada 397 (4.0) 400 (4.6) -3 (4.6) 464 (3.9) 460 (3.4) 3 (3.9) 533 (2.9) 526 (2.8) 7 (2.7) chile 338 (6.1) 335 (5.6) 4 (5.2) 395 (6.1) 385 (4.9) 9 (5.3) 458 (5.3) 443 (4.5) 15 (5.2) czech republic 381 (8.4) 386 (6.0) -5 (9.3) 452 (6.0) 443 (4.8) 9 (6.6) 521 (4.8) 510 (4.5) 11 (5.9) denmark 379 (6.6) 376 (5.7) 3 (6.0) 441 (5.0) 436 (3.7) 5 (4.8) 505 (4.5) 496 (3.4) 10 (4.9) Estonia 399 (5.9) 402 (5.4) -3 (6.8) 459 (4.6) 457 (4.0) 2 (5.2) 520 (4.1) 515 (3.1) 5 (4.5) finland 394 (4.9) 409 (4.7) -15 (7.1) 456 (3.7) 469 (3.8) -13 (4.0) 523 (3.3) 530 (3.2) -7 (3.8) france 384 (7.6) 391 (8.3) -7 (8.2) 454 (5.5) 456 (4.3) -2 (5.5) 521 (4.7) 516 (3.4) 5 (4.5) Germany 373 (7.5) 382 (6.9) -9 (5.5) 443 (6.8) 445 (5.1) -2 (5.6) 519 (4.3) 512 (4.2) 7 (4.0) hungary 308 (10.2) 328 (8.9) -20 (12.7) ireland 377 380 (5.4) -3 (9.6) (7.9) 384 (10.5) 396 (6.1) -12 (10.8) 467 (5.5) 463 (4.9) 5 (6.3) 438 438 (4.1) 0 (7.0) 504 (4.4) 499 (3.7) 5 (5.3) (11.3) (6.2) israel 277 (11.5) 304 (7.9) -28 (11.9) 362 (10.0) 379 (5.1) -17 (9.9) 464 (11.4) 456 (5.3) 8 italy 391 (7.4) 397 (7.3) -6 (9.2) 455 (8.1) 448 (5.3) 7 (8.5) 526 (5.6) 503 (4.6) 23 (6.1) Japan 445 (6.3) 438 (5.7) 6 (6.1) 504 (5.2) 492 (4.0) 12 (5.0) 567 (4.5) 546 (3.6) 22 (4.7) korea 444 (8.1) 443 (6.9) 1 (9.1) 510 (6.9) 501 (6.5) 10 (7.7) 575 (5.5) 559 (5.6) 16 (6.3) netherlands 377 (10.4) 379 (8.7) -2 (8.6) 449 (7.2) 447 (6.2) 1 (6.2) 521 (5.7) 514 (5.0) 7 (4.2) norway 365 (5.9) 376 (5.9) -11 (6.7) 433 (4.8) 439 (4.7) -6 (5.4) 505 (3.8) 508 (4.1) -3 (4.2) Poland 347 (7.5) 368 (7.0) -21 (6.8) 416 (5.7) 426 (6.1) -10 (6.0) 486 (5.6) 483 (4.7) 3 (5.6) Portugal 384 (5.7) 378 (5.1) 6 (5.0) 441 (5.3) 431 (4.3) 10 (3.8) 507 (5.2) 489 (4.4) 18 (4.7) Slovak republic 363 (7.3) 345 (8.7) 18 (7.5) 426 (5.8) 414 (6.4) 12 (6.6) 495 (5.4) 480 (4.9) 15 (6.1) Slovenia 341 (4.3) 361 (4.6) -21 (6.8) 408 (4.5) 417 (4.3) -9 (5.5) 477 (3.5) 480 (3.4) -4 (4.5) Spain 334 (9.7) 344 (8.0) -10 (8.0) 406 (6.0) 416 (5.1) -10 (5.6) 485 (5.1) 482 (3.9) 3 (4.9) Sweden 357 (6.4) 373 (5.7) -17 (7.3) 423 (5.3) 432 (3.6) -10 (5.2) 493 (4.1) 495 (3.9) -2 (4.7) turkey 361 (5.6) 349 (4.3) 12 (5.8) 404 (5.0) 394 (4.7) 11 (5.2) 459 (4.7) 444 (5.0) 14 (5.0) England (united kingdom) 391 (8.2) 391 (7.2) 0 (9.3) 457 (6.7) 453 (6.2) 3 (6.5) 525 (6.2) 518 (5.2) 7 (6.8) united States 383 (7.2) 394 (6.5) -11 (6.4) 443 (6.0) 447 (4.9) -4 (5.1) 513 (4.9) 507 (4.6) 6 (4.1) oEcd average 372 (1.4) 378 (1.2) -5 (1.5) 438 (1.2) 438 (0.9) 0 (1.2) 508 (1.0) 501 (0.8) 8 (1.0) brazil 319 (8.4) 305 (5.7) 14 (8.4) 377 (6.5) 360 (5.1) 17 (5.2) 440 (6.6) 419 (5.9) 21 (4.6) bulgaria 250 (10.0) 278 (8.3) -28 (9.5) 321 (7.2) 343 (6.4) -22 (6.7) 396 (6.8) 413 (5.8) -17 (6.4) colombia 300 (5.0) 273 (5.6) 27 (6.3) 353 (4.9) 326 (4.5) 27 (4.4) 413 (4.3) 384 (4.8) 29 (5.0) croatia 347 (6.8) 350 (5.0) -3 (6.8) 406 (5.6) 402 (4.6) 5 (6.0) 473 (6.1) 459 (4.7) 14 (6.0) cyprus* 299 (4.8) 333 (4.5) -34 (6.3) 366 (3.4) 388 (3.4) -22 (4.9) 444 (2.7) 451 (2.4) -7 (3.6) hong kong-china 425 (7.1) 416 (8.1) 9 (7.8) 488 (5.9) 477 (5.9) 11 (6.3) 551 (5.0) 537 (5.1) 14 (5.7) macao-china 439 (3.9) 434 (3.4) 5 (5.4) 492 (2.8) 485 (2.1) 7 (3.7) 550 (2.7) 539 (2.3) 11 (4.0) malaysia 315 (4.6) 314 (6.0) 1 (5.6) 365 (5.0) 364 (4.7) 2 (5.0) 426 (4.6) 418 (4.7) 8 (4.5) montenegro 282 (4.0) 296 (4.8) -14 (6.8) 338 (3.0) 351 (3.6) -13 (4.3) 403 (2.9) 411 (3.2) -9 (4.7) russian federation 380 (4.8) 374 (5.8) 6 (5.3) 435 (4.6) 428 (4.7) 7 (4.6) 495 (4.0) 485 (4.1) 10 (4.7) Serbia 363 (6.9) 350 (6.9) 13 (6.4) 420 (5.3) 408 (3.7) 11 (4.8) 484 (5.1) 469 (4.4) 15 (5.3) Shanghai-china 431 (6.7) 411 (7.1) 21 (7.2) 491 (4.8) 468 (5.0) 23 (4.7) 554 (4.9) 528 (4.4) 26 (4.6) Singapore 432 (4.1) 440 (4.2) -8 (5.4) 501 (2.9) 498 (2.8) 2 (4.1) 573 (2.5) 562 (3.0) 11 (3.6) chinese taipei 410 (8.0) 417 (4.4) -7 (7.4) 479 (5.7) 471 (4.7) 8 (6.3) 548 (5.0) 532 (4.0) 17 (6.6) united arab Emirates 253 (6.7) 307 (5.6) -53 (8.5) 319 (6.6) 362 (4.2) -43 (7.8) 396 (5.2) 422 (3.7) -26 (6.3) uruguay 279 (5.9) 280 (5.8) -1 (5.5) 338 (5.4) 335 (5.2) 4 (5.1) 410 (4.7) 398 (4.7) 12 (4.9) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 183 Annex b1: reSulTS For counTrIeS And economIeS table v.4.7 [Part 3/3] mean score and variation in student performance in problem solving, by gender 75th percentile Partners OECD boys australia Girls Score S.E. Score S.E. 594 (3.1) 588 (2.7) 90th percentile difference (b - G) Score dif. 6 boys Girls 95th percentile difference (b - G) Girls difference (b - G) S.E. Score S.E. Score S.E. Score dif. S.E. Score S.E. Score S.E. Score dif. S.E. (3.6) 651 (3.4) 641 (2.8) 10 (4.2) 684 (4.1) 671 (3.1) 12 (4.6) austria 581 (4.8) 564 (4.6) 17 (6.0) 631 (5.9) 612 (5.0) 19 (6.9) 661 (7.2) 639 (5.3) 22 (8.2) belgium 591 (2.9) 576 (3.0) 15 (3.3) 644 (3.3) 627 (3.9) 17 (4.3) 673 (4.0) 656 (4.6) 18 (5.2) canada 599 (3.4) 589 (2.9) 10 (3.1) 656 (3.9) 643 (3.4) 13 (4.0) 690 (5.0) 675 (4.1) 15 (5.2) chile 517 (4.5) 499 (4.0) 18 (4.5) 567 (5.3) 546 (4.4) 21 (5.6) 597 (5.9) 576 (5.7) 21 (7.2) czech republic 582 (4.3) 568 (4.2) 13 (5.4) 632 (5.1) 620 (4.5) 12 (5.9) 662 (5.5) 650 (5.2) 12 (7.5) denmark 568 (4.4) 553 (4.1) 16 (5.8) 619 (5.6) 604 (4.8) 15 (6.4) 650 (6.3) 631 (5.0) 19 (5.9) Estonia 580 (3.8) 571 (3.4) 9 (4.3) 632 (3.5) 619 (4.2) 12 (4.4) 661 (4.9) 647 (5.7) 14 (6.5) finland 586 (3.8) 588 (3.4) -2 (4.3) 642 (5.0) 638 (4.1) 3 (5.5) 675 (6.4) 667 (3.9) 8 (6.8) france 583 (4.1) 571 (3.9) 12 (4.3) 634 (4.3) 619 (4.8) 14 (4.8) 659 (5.6) 647 (4.9) 12 (5.8) Germany 586 (5.1) 572 (4.6) 14 (4.8) 637 (4.9) 620 (5.3) 17 (5.3) 669 (6.0) 649 (6.8) 20 (7.3) hungary 540 (5.8) 525 (6.0) 15 (5.6) 600 (7.5) 581 (6.1) 19 (5.3) 633 (7.2) 611 (5.8) 22 (6.2) ireland 566 (5.9) 558 (3.7) 8 (6.7) 622 (7.6) 607 (4.1) 15 (8.3) 660 (7.1) 635 (4.0) 25 (8.1) israel 560 (10.3) 529 (4.8) 31 (10.4) 628 (8.2) 591 (4.9) 37 (7.5) 664 (8.1) 626 (5.4) 38 (9.0) italy 587 557 (5.3) 30 (6.4) 635 (4.3) 599 (6.2) 36 (6.5) 662 (5.2) 627 (6.9) 36 (7.3) (5.1) Japan 623 (4.2) 596 (3.5) 27 (4.6) 670 (4.7) 641 (3.9) 29 (5.4) 697 (6.0) 667 (5.1) 30 (5.8) korea 633 (5.0) 615 (5.6) 18 (6.1) 680 (5.4) 661 (5.9) 19 (6.2) 709 (6.5) 686 (6.1) 23 (7.1) (6.9) netherlands 586 (5.3) 576 (5.8) 10 (4.6) 638 (5.2) 626 (6.3) 12 (5.6) 665 (5.0) 656 (7.4) 9 norway 575 (4.8) 574 (4.0) 1 (4.5) 636 (5.4) 630 (4.9) 7 (6.4) 669 (8.5) 662 (5.8) 7 (7.8) Poland 553 (5.2) 540 (4.8) 14 (4.8) 607 (5.1) 592 (6.0) 15 (5.6) 639 (6.5) 623 (6.6) 16 (6.7) Portugal 565 (4.4) 544 (3.7) 21 (3.5) 615 (4.6) 591 (5.2) 24 (4.5) 644 (6.0) 622 (6.4) 23 (6.4) Slovak republic 564 (4.8) 536 (4.4) 28 (5.7) 622 (7.1) 585 (5.5) 37 (7.0) 654 (7.9) 615 (5.6) 39 (6.9) Slovenia 548 (3.4) 542 (3.8) 6 (5.5) 602 (4.4) 596 (4.8) 6 (7.4) 631 (5.8) 624 (6.7) 6 (9.5) Spain 554 (4.5) 545 (4.5) 9 (4.9) 613 (5.6) 597 (5.8) 16 (7.2) 647 (6.3) 628 (7.1) 19 (9.2) Sweden 559 (3.6) 555 (3.7) 4 (4.3) 615 (5.3) 608 (4.2) 7 (6.4) 649 (6.3) 639 (4.2) 9 (6.6) turkey 517 (5.7) 498 (6.1) 18 (5.1) 570 (6.7) 549 (8.1) 21 (6.0) 599 (7.4) 579 (9.4) 21 (7.5) (10.4) England (united kingdom) 589 (5.4) 579 (5.3) 10 (6.2) 640 (5.3) 630 (5.8) 9 (7.2) 671 (8.3) 663 (7.2) 9 united States 577 (4.6) 566 (4.4) 11 (4.7) 632 (4.9) 619 (5.7) 13 (5.6) 666 (6.3) 650 (7.2) 17 (7.1) oEcd average 574 (0.9) 560 (0.8) 14 (1.0) 627 (1.0) 610 (1.0) 17 (1.1) 659 (1.2) 640 (1.1) 19 (1.4) brazil 505 (7.0) 478 (6.1) 27 (4.6) 560 (6.8) 529 (5.6) 31 (5.8) 589 (7.1) 557 (6.3) 31 (6.6) bulgaria 470 (6.1) 481 (6.2) -11 (6.5) 532 (8.5) 538 (7.9) -6 (8.1) 569 (8.3) 572 (8.9) -3 (8.0) colombia 477 (5.4) 443 (5.0) 33 (5.9) 537 (6.0) 497 (6.4) 40 (7.4) 569 (7.8) 531 (6.9) 38 (8.9) croatia 543 (6.2) 517 (5.1) 26 (6.5) 600 (6.4) 568 (5.8) 33 (5.9) 631 (6.8) 597 (6.6) 33 (7.0) cyprus* 516 (3.6) 510 (3.8) 6 (4.8) 576 (3.3) 565 (3.8) 11 (5.2) 613 (4.2) 596 (5.1) 17 (7.5) hong kong-china 609 (5.1) 592 (5.9) 17 (7.2) 661 (4.7) 644 (7.5) 17 (8.8) 690 (4.9) 673 (9.1) 17 (10.5) macao-china 602 (2.2) 589 (2.2) 14 (3.0) 647 (3.2) 631 (3.0) 16 (4.9) 672 (3.8) 656 (3.2) 16 (5.4) malaysia 485 (4.8) 474 (4.6) 12 (5.2) 540 (7.0) 524 (5.7) 15 (6.8) 571 (7.9) 551 (5.9) 20 (8.0) montenegro 469 (3.1) 470 (3.4) -1 (5.0) 528 (5.6) 524 (3.9) 4 (6.4) 561 (5.8) 552 (5.8) 9 (8.3) russian federation 551 (4.9) 542 (4.7) 9 (4.9) 608 (7.2) 596 (6.1) 12 (7.0) 640 (7.3) 628 (7.0) 12 (6.5) Serbia 544 (4.5) 528 (3.8) 16 (5.4) 596 (4.0) 576 (4.2) 20 (5.3) 626 (4.5) 604 (4.8) 23 (6.5) Shanghai-china 612 (4.4) 585 (5.1) 27 (5.8) 661 (4.2) 633 (6.9) 28 (5.5) 689 (4.6) 661 (6.4) 28 (5.4) Singapore 639 (2.6) 620 (2.5) 19 (3.5) 692 (3.2) 667 (3.8) 25 (4.6) 720 (3.8) 697 (4.7) 24 (6.3) chinese taipei 610 (4.5) 590 (5.3) 21 (7.6) 657 (4.7) 634 (6.0) 23 (8.8) 683 (4.7) 661 (7.1) 22 (9.8) united arab Emirates 476 (6.1) 486 (3.9) -10 (7.6) 548 (5.9) 545 (4.6) 4 (8.4) 589 (6.1) 580 (4.8) 9 (8.5) uruguay 481 (5.0) 461 (4.8) 20 (5.8) 543 (5.0) 519 (4.9) 24 (5.3) 580 (5.5) 552 (6.7) 28 (6.6) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 184 boys © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.8 [Part 1/3] differences in problem-solving, mathematics, reading and science performance related to gender Gender gap: mean score difference between boys and girls Problem solving (b - G) OECD Score dif. reading (b - G) computer-based mathematics (b - G) Science (b - G) S.E. Score dif. S.E. Score dif. S.E. S.E. Score dif. S.E. (2.6) 12 (3.1) -34 (2.9) 5 (3.0) 9 (2.8) -31 (2.9) austria 12 (4.8) 22 (4.9) -37 (5.0) 9 (5.0) 21 (4.9) -27 (6.1) belgium 8 (3.7) 11 (3.4) -32 (3.5) 4 (3.6) 14 (3.1) -25 (4.0) canada 5 (2.2) 10 (2.0) -35 (2.1) 3 (2.1) 17 (1.9) -21 (1.8) 13 (3.8) 25 (3.6) -23 (3.3) 7 (3.3) 19 (3.9) -9 (4.4) 8 (4.1) 12 (4.6) -39 (3.7) 1 (4.0) m m m m 10 (3.1) 14 (2.3) -31 (2.8) 10 (2.7) 20 (2.5) -23 (2.4) Estonia 5 (3.1) 5 (2.6) -44 (2.4) -2 (2.7) 9 (2.5) -37 (2.8) finland -6 (3.0) -3 (2.9) -62 (3.1) -16 (3.0) m m m m france 5 (3.1) 9 (3.4) -44 (4.2) -2 (3.7) 15 (3.0) -22 (3.6) Germany 7 (2.9) 14 (2.8) -44 (2.5) -1 (3.0) 10 (2.7) -30 (3.0) hungary 3 (4.8) 9 (3.7) -40 (3.6) 3 (3.3) 12 (3.8) -33 (4.9) ireland 5 (5.0) 15 (3.8) -29 (4.2) 4 (4.4) 19 (3.7) -25 (4.3) israel 6 (8.5) 12 (7.6) -44 (7.9) -1 (7.6) 3 (8.9) -27 (6.4) italy 18 (5.7) 10 (4.8) -45 (5.4) -7 (5.5) 18 (5.0) -21 (6.0) Japan 19 (3.7) 18 (4.3) -24 (4.1) 11 (4.3) 15 (3.8) -16 (3.8) korea 13 (5.5) 18 (6.2) -23 (5.4) 3 (5.1) 18 (6.7) -7 (5.1) 5 (3.3) 10 (2.8) -26 (3.1) 3 (2.9) m m m m -3 (3.6) 2 (3.0) -46 (3.3) -4 (3.2) 3 (2.8) -46 (3.1) czech republic denmark netherlands norway Poland Score dif. S.E. Score dif. digital reading (b - G) 2 australia chile 0 (3.3) 4 (3.4) -42 (2.9) -3 (3.0) 11 (3.2) -34 (3.4) Portugal 16 (2.6) 11 (2.5) -39 (2.7) -2 (2.6) 20 (2.3) -17 (3.0) Slovak republic 22 (4.4) 9 (4.5) -39 (4.6) 7 (4.5) 11 (3.9) -19 (4.3) Slovenia -4 (3.0) 3 (3.1) -56 (2.7) -9 (2.8) 3 (3.0) -39 (2.7) 2 (3.4) 13 (2.9) -32 (2.7) 3 (2.7) 12 (2.5) -27 (3.1) Sweden -4 (3.6) -3 (3.0) -51 (3.6) -7 (3.3) 13 (2.8) -33 (3.3) turkey 15 (4.0) 8 (4.7) -46 (4.0) -10 (4.2) m m m m England (united kingdom) 6 (5.5) 13 (5.5) -24 (5.4) 14 (5.5) m m m m united States 3 (3.1) 5 (2.8) -31 (2.6) -2 (2.7) 0 (3.0) -28 (2.6) oEcd average 7 (0.8) 10 (0.7) -38 (0.7) 1 (0.7) 13 (0.8) -26 (0.8) (3.2) Spain Partners mathematics (b - G) brazil 22 (3.3) 21 (2.4) -27 (2.9) 2 (2.9) 22 (2.4) -19 -17 (4.9) -2 (4.1) -70 (5.2) -20 (4.5) m m m m colombia 31 (3.8) 25 (3.2) -19 (3.5) 18 (3.4) 12 (3.3) -4 (4.3) croatia 15 (4.4) 12 (4.1) -48 (4.0) -2 (3.8) m m m m cyprus* -9 (2.5) 0 (2.2) -64 (3.0) -13 (2.5) m m m m hong kong-china 13 (5.2) 15 (5.7) -25 (4.7) 7 (4.2) 17 (4.3) -19 (5.0) macao-china 10 (2.0) 3 (1.9) -36 (1.7) -1 (1.7) 13 (2.0) -18 (1.7) 8 (3.7) -8 (3.8) -40 (3.1) -11 (3.5) m m m m -6 (2.8) 0 (2.4) -62 (3.1) -17 (2.4) m m m m 8 (3.1) -2 (3.0) -40 (3.0) -6 (2.9) 14 (2.8) -18 (3.0) Serbia 15 (3.5) 9 (3.9) -46 (3.8) -4 (3.9) m m m m Shanghai-china 25 (2.9) 6 (3.3) -24 (2.5) 5 (2.7) 18 (2.9) -10 (2.8) bulgaria malaysia montenegro russian federation Singapore chinese taipei united arab Emirates uruguay 9 (2.5) -3 (2.5) -32 (2.6) -1 (2.6) 1 (2.3) -18 (2.2) 12 (6.3) 5 (8.9) -32 (6.4) 1 (6.4) 15 (6.7) -17 (5.3) -26 (5.6) -5 (4.7) -55 (4.8) -28 (5.1) -13 (4.4) -50 (6.5) 11 (3.4) 11 (3.1) -35 (3.5) -1 (3.4) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 185 Annex b1: reSulTS For counTrIeS And economIeS table v.4.8 [Part 2/3] differences in problem-solving, mathematics, reading and science performance related to gender Gender effect size: Gender difference divided by the variation in scores within each country/economy (standard deviation) OECD Problem solving (b - G) reading (b - G) S.E. Effect size S.E. S.E. Effect size S.E. Effect size S.E. 0.03 (0.03) 0.13 (0.03) -0.35 (0.03) 0.05 (0.03) 0.10 (0.03) -0.32 (0.03) austria 0.13 (0.05) 0.24 (0.05) -0.40 (0.05) 0.09 (0.05) 0.23 (0.06) -0.26 (0.06) belgium 0.07 (0.03) 0.11 (0.03) -0.31 (0.03) 0.04 (0.04) 0.15 (0.03) -0.26 (0.04) canada 0.05 (0.02) 0.11 (0.02) -0.38 (0.02) 0.03 (0.02) 0.19 (0.02) -0.24 (0.02) chile 0.16 (0.04) 0.31 (0.04) -0.29 (0.04) 0.08 (0.04) 0.24 (0.05) -0.11 (0.05) czech republic 0.08 (0.04) 0.12 (0.05) -0.44 (0.04) 0.01 (0.04) m m m m denmark 0.11 (0.03) 0.17 (0.03) -0.36 (0.03) 0.11 (0.03) 0.23 (0.03) -0.27 (0.03) Estonia 0.06 (0.04) 0.07 (0.03) -0.54 (0.03) -0.03 (0.03) 0.11 (0.03) -0.39 (0.03) finland -0.07 (0.03) -0.03 (0.03) -0.65 (0.03) -0.18 (0.03) m m m m france 0.05 (0.03) 0.09 (0.03) -0.40 (0.04) -0.02 (0.04) 0.16 (0.03) -0.23 (0.04) Germany 0.07 (0.03) 0.14 (0.03) -0.48 (0.03) -0.01 (0.03) 0.10 (0.03) -0.30 (0.03) hungary 0.03 (0.05) 0.10 (0.04) -0.43 (0.04) 0.03 (0.04) 0.13 (0.04) -0.29 (0.04) ireland 0.06 (0.05) 0.18 (0.05) -0.33 (0.05) 0.04 (0.05) 0.23 (0.05) -0.31 (0.05) israel 0.05 (0.07) 0.11 (0.07) -0.38 (0.07) -0.01 (0.07) 0.02 (0.08) -0.24 (0.05) italy 0.20 (0.07) 0.11 (0.05) -0.46 (0.05) -0.08 (0.06) 0.22 (0.06) -0.22 (0.06) Japan 0.22 (0.04) 0.19 (0.05) -0.24 (0.04) 0.12 (0.04) 0.17 (0.04) -0.20 (0.05) korea 0.14 (0.06) 0.18 (0.06) -0.27 (0.06) 0.04 (0.06) 0.20 (0.07) -0.09 (0.06) netherlands 0.05 (0.03) 0.11 (0.03) -0.28 (0.03) 0.03 (0.03) m m m m norway -0.03 (0.03) 0.02 (0.03) -0.46 (0.03) -0.04 (0.03) 0.03 (0.03) -0.46 (0.03) Poland 0.00 (0.03) 0.04 (0.04) -0.48 (0.03) -0.03 (0.04) 0.13 (0.04) -0.35 (0.04) Portugal 0.18 (0.03) 0.12 (0.03) -0.42 (0.03) -0.02 (0.03) 0.24 (0.03) -0.19 (0.03) Slovak republic 0.22 (0.04) 0.09 (0.04) -0.38 (0.05) 0.07 (0.04) 0.13 (0.05) -0.20 (0.05) -0.04 (0.03) 0.04 (0.03) -0.61 (0.03) -0.10 (0.03) 0.03 (0.03) -0.40 (0.03) 0.01 (0.03) 0.15 (0.03) -0.34 (0.03) 0.04 (0.03) 0.15 (0.03) -0.28 (0.03) Sweden -0.04 (0.04) -0.03 (0.03) -0.48 (0.03) -0.07 (0.03) 0.16 (0.03) -0.35 (0.03) turkey 0.19 (0.05) 0.09 (0.05) -0.53 (0.04) -0.13 (0.05) m m m m England (united kingdom) 0.06 (0.06) 0.13 (0.06) -0.25 (0.05) 0.14 (0.05) m m m m united States 0.03 (0.03) 0.05 (0.03) -0.33 (0.03) -0.02 (0.03) 0.00 (0.03) -0.32 (0.03) oEcd average 0.07 (0.01) 0.11 (0.01) -0.40 (0.01) 0.01 (0.01) 0.15 (0.01) -0.27 (0.01) (0.03) brazil Effect size S.E. 0.24 (0.04) 0.27 (0.03) -0.32 (0.03) 0.02 (0.04) 0.26 (0.03) -0.21 -0.16 (0.05) -0.03 (0.04) -0.59 (0.04) -0.20 (0.04) m m m m 0.33 (0.04) 0.34 (0.04) -0.22 (0.04) 0.23 (0.05) 0.16 (0.04) -0.05 (0.05) croatia 0.16 (0.05) 0.13 (0.05) -0.56 (0.04) -0.03 (0.04) m m m m cyprus* -0.09 (0.02) 0.00 (0.02) -0.57 (0.02) -0.13 (0.03) m m m m hong kong-china 0.15 (0.06) 0.16 (0.06) -0.30 (0.05) 0.08 (0.05) 0.20 (0.05) -0.20 (0.05) macao-china 0.13 (0.02) 0.03 (0.02) -0.43 (0.02) -0.02 (0.02) 0.15 (0.02) -0.26 (0.02) malaysia 0.09 (0.04) -0.10 (0.05) -0.48 (0.04) -0.14 (0.05) m m m m -0.06 (0.03) 0.00 (0.03) -0.67 (0.03) -0.20 (0.03) m m m m russian federation 0.09 (0.04) -0.02 (0.04) -0.44 (0.03) -0.07 (0.03) 0.18 (0.03) -0.21 (0.04) Serbia 0.17 (0.04) 0.10 (0.04) -0.50 (0.04) -0.05 (0.04) m m m m Shanghai-china 0.28 (0.03) 0.06 (0.03) -0.30 (0.03) 0.06 (0.03) 0.20 (0.03) -0.12 (0.03) Singapore 0.10 (0.03) -0.03 (0.02) -0.32 (0.03) -0.01 (0.02) 0.01 (0.02) -0.20 (0.02) chinese taipei 0.13 (0.07) 0.05 (0.08) -0.35 (0.07) 0.01 (0.08) 0.17 (0.07) -0.19 (0.06) -0.25 (0.05) -0.05 (0.05) -0.58 (0.05) -0.30 (0.05) -0.15 (0.05) -0.45 (0.06) 0.12 (0.03) 0.13 (0.04) -0.37 (0.03) -0.01 (0.04) m m m m bulgaria colombia montenegro united arab Emirates uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 186 digital reading (b - G) Effect size Spain Effect size computer-based mathematics (b - G) Science (b - G) australia Slovenia Partners mathematics (b - G) © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.8 [Part 3/3] differences in problem-solving, mathematics, reading and science performance related to gender difference in gender effect sizes between problem solving (PS) and… Partners OECD … mathematics (PS - m) … reading (PS - r) … computer-based mathematics (PS - cbm) … Science (PS - S) … digital reading (PS - dr) Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. australia -0.10 (0.02) 0.38 (0.02) -0.02 (0.02) -0.07 (0.02) 0.34 (0.02) austria -0.11 (0.03) 0.53 (0.03) 0.03 (0.03) -0.11 (0.04) 0.38 (0.05) belgium -0.03 (0.02) 0.39 (0.02) 0.04 (0.02) -0.07 (0.02) 0.33 (0.03) canada -0.06 (0.02) 0.44 (0.02) 0.02 (0.02) -0.13 (0.02) 0.29 (0.02) chile -0.15 (0.03) 0.45 (0.03) 0.07 (0.03) -0.08 (0.04) 0.27 (0.04) czech republic -0.04 (0.03) 0.52 (0.03) 0.07 (0.03) m m m m denmark -0.06 (0.02) 0.47 (0.03) 0.00 (0.03) -0.12 (0.02) 0.38 (0.03) Estonia -0.01 (0.02) 0.60 (0.02) 0.09 (0.03) -0.05 (0.02) 0.45 (0.03) finland -0.03 (0.02) 0.58 (0.02) 0.11 (0.02) m m m m france -0.04 (0.03) 0.45 (0.03) 0.07 (0.03) -0.12 (0.02) 0.28 (0.03) Germany -0.07 (0.02) 0.55 (0.02) 0.07 (0.02) -0.03 (0.02) 0.37 (0.02) hungary -0.06 (0.03) 0.46 (0.03) 0.00 (0.03) -0.09 (0.03) 0.32 (0.03) ireland -0.12 (0.04) 0.39 (0.05) 0.01 (0.04) -0.17 (0.05) 0.36 (0.05) israel -0.06 (0.03) 0.43 (0.03) 0.06 (0.03) 0.03 (0.03) 0.28 (0.04) italy 0.08 (0.05) 0.65 (0.05) 0.27 (0.05) -0.03 (0.05) 0.42 (0.05) Japan 0.03 (0.03) 0.47 (0.03) 0.11 (0.03) 0.06 (0.03) 0.43 (0.03) korea -0.04 (0.04) 0.41 (0.04) 0.10 (0.04) -0.05 (0.05) 0.23 (0.05) netherlands -0.06 (0.02) 0.34 (0.02) 0.02 (0.02) m m m m norway -0.05 (0.02) 0.43 (0.03) 0.00 (0.02) -0.07 (0.02) 0.43 (0.02) Poland -0.04 (0.02) 0.48 (0.02) 0.03 (0.02) -0.12 (0.02) 0.35 (0.02) Portugal 0.06 (0.02) 0.60 (0.03) 0.20 (0.02) -0.06 (0.02) 0.37 (0.03) Slovak republic 0.13 (0.03) 0.60 (0.03) 0.15 (0.03) 0.09 (0.03) 0.42 (0.03) Slovenia -0.08 (0.02) 0.56 (0.02) 0.06 (0.02) -0.07 (0.02) 0.36 (0.02) Spain -0.13 (0.02) 0.36 (0.03) -0.02 (0.02) -0.14 (0.03) 0.29 (0.02) Sweden -0.01 (0.02) 0.44 (0.03) 0.04 (0.03) -0.19 (0.02) 0.31 (0.02) turkey 0.10 (0.03) 0.72 (0.03) 0.32 (0.04) m m m m England (united kingdom) -0.07 (0.03) 0.31 (0.04) -0.08 (0.03) m m m m united States -0.02 (0.02) 0.37 (0.02) 0.05 (0.02) 0.03 (0.02) 0.35 (0.02) oEcd average -0.04 (0.01) 0.48 (0.01) 0.07 (0.01) -0.07 (0.01) 0.35 (0.01) brazil -0.03 (0.03) 0.55 (0.03) 0.22 (0.03) -0.02 (0.03) 0.45 (0.03) bulgaria -0.13 (0.03) 0.43 (0.03) 0.04 (0.03) m m m m colombia -0.01 (0.03) 0.56 (0.03) 0.10 (0.03) 0.17 (0.04) 0.38 (0.04) croatia 0.03 (0.03) 0.72 (0.03) 0.19 (0.03) m m m m cyprus* -0.09 (0.02) 0.48 (0.02) 0.04 (0.02) m m m m hong kong-china -0.01 (0.03) 0.45 (0.04) 0.07 (0.04) -0.05 (0.04) 0.35 (0.04) macao-china 0.10 (0.02) 0.57 (0.02) 0.15 (0.02) -0.02 (0.02) 0.39 (0.02) malaysia 0.19 (0.02) 0.57 (0.03) 0.24 (0.02) m m m m -0.06 (0.02) 0.61 (0.02) 0.14 (0.02) m m m m russian federation 0.11 (0.02) 0.53 (0.03) 0.16 (0.04) -0.08 (0.02) 0.30 (0.02) Serbia 0.07 (0.03) 0.67 (0.03) 0.22 (0.03) m m m m Shanghai-china 0.22 (0.02) 0.58 (0.03) 0.22 (0.03) 0.08 (0.02) 0.40 (0.03) Singapore 0.13 (0.01) 0.42 (0.02) 0.11 (0.02) 0.09 (0.02) 0.30 (0.02) chinese taipei 0.09 (0.02) 0.49 (0.03) 0.12 (0.03) -0.04 (0.03) 0.32 (0.03) united arab Emirates -0.19 (0.04) 0.33 (0.04) 0.05 (0.04) -0.10 (0.04) 0.20 (0.04) uruguay -0.01 (0.02) 0.49 (0.02) 0.13 (0.02) m m m m montenegro Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 187 Annex b1: reSulTS For counTrIeS And economIeS table v.4.9 [Part 1/2] relative variation in performance in problem solving, mathematics, reading and science, by gender variation ratio: variation in performance among boys as a proportion of the variation in performance among girls Partners OECD Problem solving (b/G) australia mathematics (b/G) reading (b/G) computer-based mathematics (b/G) Science (b/G) ratio S.E. ratio S.E. ratio S.E. ratio S.E. ratio S.E. ratio S.E. 1.12 (0.04) 1.12 (0.05) 1.20 (0.05) 1.12 (0.04) 1.12 (0.05) 1.14 (0.04) austria 1.18 (0.08) 1.12 (0.07) 1.22 (0.07) 1.18 (0.07) 1.23 (0.08) 1.07 (0.09) belgium 1.16 (0.06) 1.14 (0.05) 1.22 (0.06) 1.22 (0.06) 1.16 (0.05) 1.21 (0.09) canada 1.17 (0.06) 1.15 (0.04) 1.21 (0.04) 1.17 (0.04) 1.15 (0.04) 1.13 (0.05) chile 1.18 (0.06) 1.11 (0.05) 1.16 (0.06) 1.14 (0.05) 1.11 (0.05) 1.14 (0.06) czech republic 1.14 (0.07) 1.08 (0.06) 1.13 (0.06) 1.12 (0.07) m m m m denmark 1.11 (0.05) 1.05 (0.05) 1.17 (0.06) 1.16 (0.06) 1.08 (0.04) 1.15 (0.04) (0.06) Estonia 1.15 (0.06) 1.14 (0.05) 1.19 (0.06) 1.15 (0.05) 1.19 (0.05) 1.14 finland 1.16 (0.05) 1.23 (0.05) 1.23 (0.06) 1.20 (0.05) m m m m france 1.16 (0.08) 1.21 (0.06) 1.28 (0.07) 1.25 (0.07) 1.18 (0.09) 1.16 (0.08) Germany 1.20 (0.05) 1.10 (0.05) 1.14 (0.04) 1.11 (0.05) 1.12 (0.05) 1.13 (0.05) hungary 1.25 (0.09) 1.18 (0.06) 1.21 (0.07) 1.12 (0.06) 1.28 (0.09) 1.21 (0.07) ireland 1.21 (0.09) 1.09 (0.06) 1.19 (0.07) 1.13 (0.07) 1.14 (0.06) 1.16 (0.07) israel 1.44 (0.07) 1.43 (0.06) 1.56 (0.09) 1.45 (0.06) 1.43 (0.09) 1.28 (0.08) italy 1.41 (0.10) 1.25 (0.06) 1.34 (0.08) 1.22 (0.07) 1.10 (0.07) 1.32 (0.10) Japan 1.25 (0.07) 1.22 (0.08) 1.31 (0.08) 1.23 (0.07) 1.27 (0.08) 1.24 (0.10) (0.10) korea 1.20 (0.08) 1.27 (0.08) 1.37 (0.10) 1.25 (0.08) 1.19 (0.10) 1.32 netherlands 1.11 (0.07) 1.05 (0.05) 1.18 (0.09) 1.06 (0.06) m m m m norway 1.14 (0.06) 1.10 (0.06) 1.22 (0.07) 1.12 (0.06) 1.08 (0.06) 1.20 (0.07) Poland 1.33 (0.07) 1.19 (0.06) 1.33 (0.08) 1.17 (0.05) 1.27 (0.06) 1.26 (0.07) Portugal 1.18 (0.05) 1.17 (0.04) 1.25 (0.06) 1.17 (0.06) 1.24 (0.05) 1.25 (0.06) Slovak republic 1.13 (0.07) 1.13 (0.06) 1.08 (0.06) 1.10 (0.06) 1.13 (0.06) 1.06 (0.07) Slovenia 1.25 (0.07) 1.07 (0.05) 1.23 (0.05) 1.14 (0.05) 1.14 (0.05) 1.25 (0.05) Spain 1.22 (0.06) 1.18 (0.05) 1.24 (0.06) 1.18 (0.05) 1.12 (0.05) 1.23 (0.05) Sweden 1.22 (0.07) 1.19 (0.06) 1.30 (0.07) 1.26 (0.07) 1.20 (0.06) 1.33 (0.07) turkey 1.11 (0.06) 1.11 (0.06) 1.20 (0.07) 1.18 (0.07) m m m m England (united kingdom) 1.08 (0.08) 1.00 (0.06) 1.04 (0.08) 1.02 (0.07) m m m m united States 1.22 (0.06) 1.13 (0.05) 1.23 (0.06) 1.20 (0.06) 1.26 (0.06) 1.29 (0.07) oEcd average 1.20 (0.01) 1.15 (0.01) 1.23 (0.01) 1.17 (0.01) 1.18 (0.01) 1.20 (0.02) (0.06) brazil 1.19 (0.06) 1.14 (0.05) 1.13 (0.06) 1.16 (0.06) 1.13 (0.05) 1.13 bulgaria 1.16 (0.07) 1.17 (0.05) 1.21 (0.06) 1.16 (0.06) m m m m colombia 1.08 (0.06) 1.20 (0.08) 1.19 (0.06) 1.16 (0.06) 1.18 (0.09) 1.13 (0.09) croatia 1.33 (0.07) 1.19 (0.06) 1.30 (0.07) 1.25 (0.06) m m m m cyprus* 1.41 (0.06) 1.52 (0.06) 1.56 (0.07) 1.48 (0.06) m m m m hong kong-china 1.08 (0.07) 1.23 (0.06) 1.25 (0.06) 1.22 (0.07) 1.27 (0.07) 1.21 (0.06) macao-china 1.10 (0.06) 1.12 (0.05) 1.27 (0.05) 1.20 (0.04) 1.23 (0.05) 1.22 (0.05) malaysia 1.14 (0.06) 1.08 (0.07) 1.18 (0.06) 1.13 (0.07) m m m m montenegro 1.16 (0.06) 1.13 (0.05) 1.23 (0.07) 1.16 (0.06) m m m m (0.06) russian federation 1.06 (0.06) 1.04 (0.04) 1.12 (0.05) 1.14 (0.04) 1.10 (0.05) 1.06 Serbia 1.05 (0.06) 1.05 (0.05) 1.16 (0.07) 1.09 (0.06) m m m m Shanghai-china 1.06 (0.05) 1.14 (0.04) 1.21 (0.05) 1.16 (0.05) 1.17 (0.05) 1.12 (0.05) Singapore 1.27 (0.05) 1.24 (0.04) 1.20 (0.05) 1.25 (0.05) 1.25 (0.05) 1.20 (0.05) chinese taipei 1.28 (0.09) 1.21 (0.09) 1.27 (0.09) 1.22 (0.10) 1.35 (0.10) 1.29 (0.07) united arab Emirates 1.46 (0.11) 1.31 (0.07) 1.42 (0.07) 1.30 (0.07) 1.42 (0.09) 1.41 (0.08) uruguay 1.21 (0.05) 1.21 (0.05) 1.26 (0.06) 1.22 (0.05) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 188 digital reading (b/G) © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.9 [Part 2/2] relative variation in performance in problem solving, mathematics, reading and science, by gender relative variation ratio: variation ratio in problem solving (PS), as a proportion of the variation ratio in... Partners OECD … mathematics (PS/m) … reading (PS/r) … computer-based mathematics (PS/cbm) … Science (PS/S) … digital reading (PS/dr) ratio S.E. ratio S.E. ratio S.E. ratio S.E. ratio S.E. australia 1.00 (0.04) 0.93 (0.04) 1.00 (0.04) 1.00 (0.04) 0.99 (0.04) austria 1.06 (0.06) 0.97 (0.05) 1.00 (0.06) 0.97 (0.06) 1.11 (0.11) belgium 1.02 (0.04) 0.96 (0.04) 0.96 (0.04) 1.00 (0.04) 0.96 (0.06) canada 1.01 (0.04) 0.97 (0.04) 0.99 (0.04) 1.01 (0.04) 1.03 (0.04) chile 1.07 (0.05) 1.02 (0.06) 1.04 (0.05) 1.06 (0.07) 1.04 (0.06) czech republic 1.06 (0.04) 1.01 (0.04) 1.02 (0.04) m m m m denmark 1.05 (0.06) 0.95 (0.05) 0.96 (0.05) 1.03 (0.05) 0.96 (0.05) Estonia 1.01 (0.03) 0.96 (0.05) 1.00 (0.04) 0.96 (0.05) 1.01 (0.05) finland 0.95 (0.04) 0.94 (0.04) 0.97 (0.04) m m m m france 0.96 (0.05) 0.91 (0.05) 0.93 (0.05) 0.99 (0.04) 1.00 (0.05) Germany 1.09 (0.04) 1.06 (0.04) 1.08 (0.04) 1.07 (0.04) 1.06 (0.05) hungary 1.06 (0.05) 1.04 (0.06) 1.12 (0.06) 0.98 (0.06) 1.04 (0.06) ireland 1.11 (0.07) 1.02 (0.07) 1.07 (0.07) 1.06 (0.08) 1.04 (0.08) israel 1.00 (0.05) 0.92 (0.06) 0.99 (0.05) 1.00 (0.06) 1.12 (0.06) italy 1.13 (0.06) 1.05 (0.07) 1.16 (0.07) 1.28 (0.10) 1.07 (0.08) Japan 1.03 (0.06) 0.96 (0.07) 1.02 (0.06) 0.99 (0.05) 1.01 (0.06) korea 0.94 (0.05) 0.87 (0.04) 0.96 (0.05) 1.01 (0.07) 0.90 (0.05) netherlands 1.06 (0.06) 0.94 (0.05) 1.05 (0.05) m m m m norway 1.04 (0.05) 0.94 (0.05) 1.02 (0.05) 1.06 (0.05) 0.95 (0.04) Poland 1.11 (0.07) 1.00 (0.04) 1.13 (0.06) 1.04 (0.05) 1.06 (0.04) Portugal 1.00 (0.04) 0.94 (0.05) 1.01 (0.05) 0.95 (0.04) 0.94 (0.04) Slovak republic 1.00 (0.05) 1.04 (0.05) 1.02 (0.05) 1.00 (0.05) 1.07 (0.07) Slovenia 1.16 (0.06) 1.01 (0.05) 1.10 (0.05) 1.10 (0.05) 0.99 (0.04) Spain 1.03 (0.05) 0.98 (0.05) 1.04 (0.05) 1.09 (0.05) 0.99 (0.04) Sweden 1.02 (0.04) 0.94 (0.05) 0.97 (0.04) 1.02 (0.05) 0.92 (0.05) turkey 1.00 (0.04) 0.93 (0.04) 0.94 (0.05) m m m m England (united kingdom) 1.07 (0.07) 1.04 (0.07) 1.06 (0.07) m m m m united States 1.08 (0.04) 0.99 (0.04) 1.01 (0.04) 0.97 (0.04) 0.94 (0.04) oEcd average 1.04 (0.01) 0.97 (0.01) 1.02 (0.01) 1.03 (0.01) 1.01 (0.01) brazil 1.04 (0.06) 1.05 (0.07) 1.02 (0.06) 1.05 (0.05) 1.05 (0.06) bulgaria 1.00 (0.05) 0.96 (0.05) 1.00 (0.06) m m m m colombia 0.91 (0.05) 0.91 (0.05) 0.94 (0.04) 0.92 (0.06) 0.96 (0.07) croatia 1.12 (0.04) 1.03 (0.05) 1.07 (0.06) m m m m cyprus* 0.93 (0.03) 0.90 (0.04) 0.95 (0.04) m m m m hong kong-china 0.87 (0.05) 0.86 (0.06) 0.88 (0.06) 0.85 (0.05) 0.89 (0.06) macao-china 0.98 (0.04) 0.87 (0.04) 0.91 (0.05) 0.90 (0.04) 0.90 (0.04) malaysia 1.05 (0.04) 0.97 (0.05) 1.01 (0.04) m m m m montenegro 1.03 (0.04) 0.95 (0.05) 1.00 (0.05) m m m m russian federation 1.02 (0.07) 0.94 (0.05) 0.93 (0.06) 0.97 (0.05) 1.00 (0.06) Serbia 1.00 (0.05) 0.91 (0.05) 0.97 (0.06) m m m m Shanghai-china 0.93 (0.03) 0.88 (0.03) 0.92 (0.04) 0.91 (0.04) 0.95 (0.05) Singapore 1.02 (0.03) 1.06 (0.04) 1.02 (0.04) 1.02 (0.04) 1.06 (0.04) chinese taipei 1.06 (0.05) 1.01 (0.05) 1.05 (0.05) 0.95 (0.05) 0.99 (0.05) united arab Emirates 1.12 (0.07) 1.03 (0.06) 1.12 (0.08) 1.03 (0.07) 1.03 (0.07) uruguay 1.01 (0.04) 0.96 (0.05) 0.99 (0.05) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 189 Annex b1: reSulTS For counTrIeS And economIeS table v.4.10 [Part 1/1] relative performance in problem solving, by gender Girls’ performance in problem solving, compared to boys with similar performance in mathematics, reading and science OECD average average average difference in difference in difference in Percentage Percentage Percentage problem solving problem solving problem solving of girls who of girls who of girls who compared with outperform boys compared with outperform boys compared with outperform boys boys with similar boys with similar boys with similar with similar with similar with similar performance performance performance performance performance performance 1 2 1 2 1 in science in reading in mathematics in mathematics in science2 in reading australia Score dif. S.E. % S.E. Score dif. S.E. % S.E. Score dif. S.E. % S.E. Score dif. S.E. % S.E. 8 (1.4) 56.8 (1.3) -30 (1.8) 30.6 (1.4) 1 (1.7) 51.5 (1.5) 7 (2.2) 56.0 (1.9) austria 7 (2.9) 55.1 (2.5) -42 (2.9) 20.4 (2.1) -5 (2.8) 46.7 (2.4) -20 (4.7) 34.1 (3.5) belgium 1 (2.2) 50.9 (1.7) -34 (2.0) 29.1 (1.3) -4 (2.2) 47.5 (1.5) -5 (2.5) 46.5 (1.8) canada 3 (1.4) 52.4 (1.3) -34 (1.5) 30.4 (0.9) -3 (1.4) 48.4 (0.9) -5 (1.8) 47.0 (1.5) chile 9 (2.3) 57.5 (2.1) -32 (2.1) 28.5 (1.7) -8 (2.4) 44.6 (2.1) 1 (2.6) 51.1 (2.5) czech republic 3 (2.4) 53.0 (2.5) -43 (2.3) 20.6 (1.9) -7 (2.4) 44.7 (2.0) -8 (3.7) 42.8 (3.2) denmark 2 (2.4) 52.0 (1.9) -34 (2.1) 28.9 (1.4) -2 (2.4) 48.3 (1.7) -2 (5.4) 48.7 (3.7) Estonia 0 (1.8) 50.9 (1.8) -45 (2.0) 18.4 (1.4) -7 (2.5) 44.7 (2.3) -14 (4.2) 37.7 (3.9) finland 4 (1.4) 53.6 (1.4) -44 (2.0) 22.2 (1.2) -7 (1.6) 45.2 (1.3) -6 (2.9) 45.5 (2.7) france 2 (2.3) 54.4 (2.4) -35 (2.2) 26.4 (2.1) -6 (2.3) 47.5 (2.2) -3 (3.9) 49.5 (4.2) Germany 5 (2.0) 54.3 (1.7) -47 (2.0) 19.5 (1.5) -7 (2.0) 44.5 (2.0) -8 (3.6) 43.9 (2.9) hungary 5 (2.8) 53.4 (2.3) -41 (3.1) 23.0 (1.6) 0 (3.2) 50.2 (2.3) -10 (4.3) 42.4 (3.6) (4.7) ireland 9 (3.9) 57.0 (3.2) -29 (4.3) 31.1 (2.6) -2 (4.0) 49.5 (3.2) 2 (5.3) 51.8 israel 6 (3.3) 54.2 (2.5) -46 (3.2) 24.9 (1.8) -6 (3.4) 46.3 (2.4) -7 (3.8) 45.2 (3.0) italy -10 (4.4) 42.1 (3.4) -49 (4.0) 19.3 (2.2) -23 (4.1) 34.2 (3.0) -21 (5.4) 34.3 (4.0) Japan -7 (2.6) 45.7 (2.0) -34 (2.4) 28.0 (1.7) -12 (2.6) 42.0 (2.0) -9 (3.0) 43.8 (2.7) korea -1 (3.1) 49.9 (2.7) -32 (3.2) 27.4 (2.3) -11 (3.3) 43.0 (2.5) -10 (3.7) 42.7 (3.3) netherlands 4 (2.1) 54.7 (2.0) -28 (1.9) 31.3 (1.8) -2 (2.0) 48.9 (1.8) -3 (2.3) 47.6 (2.4) norway 5 (2.3) 53.4 (1.7) -33 (2.7) 30.7 (1.6) 1 (2.3) 50.8 (1.6) -1 (3.2) 49.3 (2.2) 3 (2.2) 52.9 (1.8) -38 (1.9) 25.4 (1.7) -3 (1.9) 48.7 (2.0) -21 (3.5) 35.6 (2.7) -7 (1.8) 44.7 (1.8) -44 (2.1) 20.8 (1.6) -17 (1.7) 37.6 (1.5) -17 (2.8) 36.7 (2.9) -14 (2.6) 38.6 (2.4) -54 (2.6) 15.4 (1.5) -16 (2.8) 38.4 (2.2) -29 (3.5) 26.4 (2.6) 7 (2.0) 54.9 (2.0) -44 (2.3) 22.5 (1.9) -4 (2.1) 48.3 (2.5) -7 (4.3) 45.2 (3.2) 10 (2.2) 57.4 (1.9) -27 (2.5) 36.3 (1.7) 2 (2.2) 51.7 (1.8) 4 (3.3) 54.2 (2.4) Sweden 1 (2.3) 50.9 (2.3) -31 (2.4) 31.1 (1.5) -1 (2.5) 49.2 (2.1) 2 (3.4) 51.3 (2.8) turkey -9 (2.4) 41.2 (2.4) -49 (2.2) 15.6 (1.4) -23 (2.6) 32.6 (2.2) -24 (2.4) 27.9 (2.1) 5 (3.0) 55.1 (2.7) -26 (3.2) 32.4 (2.1) 5 (3.4) 54.5 (2.7) 7 (4.0) 56.4 (3.6) Poland Portugal Slovak republic Slovenia Spain England (united kingdom) Partners average difference in Percentage problem solving of girls who compared with outperform boys boys with similar with similar performance performance in mathematics, in mathematics, reading reading and science3 and science2 united States 1 (1.6) 51.4 (1.9) -29 (1.9) 28.9 (1.7) -5 (1.6) 46.6 (2.1) -5 (2.2) 46.5 (2.4) oEcd average 2 (0.5) 51.7 (0.4) -38 (0.5) 25.7 (0.3) -6 (0.5) 45.9 (0.4) -8 (0.7) 44.3 (0.6) brazil -1 (2.6) 49.4 (2.3) -44 (2.3) 23.3 (1.4) -20 (2.7) 37.8 (2.0) -10 (3.5) 42.1 (2.7) bulgaria 14 (2.9) 60.4 (2.2) -34 (3.2) 31.4 (2.4) 0 (3.1) 51.9 (2.1) 2 (3.8) 53.0 (2.7) colombia -8 (2.5) 44.8 (2.0) -44 (2.6) 23.6 (1.7) -16 (2.7) 39.7 (2.0) -18 (3.3) 37.6 (2.4) croatia -5 (2.5) 46.4 (2.4) -59 (2.6) 12.7 (1.3) -16 (2.8) 38.2 (2.2) -18 (3.1) 34.4 (2.8) cyprus* hong kong-china macao-china malaysia montenegro russian federation Serbia 9 (1.8) 56.6 (1.5) -36 (2.1) 29.0 (1.3) -1 (2.1) 49.8 (1.5) -1 (2.5) 48.7 (1.9) -3 (2.9) 48.9 (2.2) -33 (3.3) 29.5 (2.2) -9 (3.3) 44.8 (2.1) -12 (3.8) 42.2 (2.8) -9 (1.4) 43.7 (1.6) -35 (1.9) 25.5 (1.5) -11 (1.5) 42.5 (1.4) -13 (2.1) 39.6 (1.8) -15 (1.7) 37.3 (1.6) -39 (2.7) 24.9 (1.6) -17 (1.8) 37.3 (1.8) -19 (2.1) 33.5 (2.0) 6 (1.6) 54.3 (1.5) -42 (2.2) 24.8 (1.3) -8 (1.9) 45.1 (1.8) -1 (2.8) 49.0 (2.7) -9 (2.0) 44.0 (1.5) -35 (2.7) 27.8 (1.8) -11 (2.8) 43.3 (1.8) -18 (2.8) 37.0 (2.2) -8 (2.3) 44.1 (2.0) -50 (2.3) 19.2 (1.5) -18 (2.6) 37.5 (2.3) -18 (2.8) 35.1 (2.4) Shanghai-china -21 (1.9) 33.0 (1.9) -47 (2.0) 16.9 (1.5) -21 (2.1) 34.7 (1.8) -32 (2.6) 24.3 (2.2) Singapore -13 (1.3) 40.2 (1.2) -33 (1.8) 29.4 (1.4) -11 (1.5) 43.2 (1.4) -9 (1.8) 42.8 (1.6) chinese taipei -9 (1.9) 42.1 (1.9) -40 (2.3) 20.2 (1.8) -12 (2.2) 41.1 (2.4) -19 (2.5) 33.0 (2.5) united arab Emirates 22 (3.7) 64.5 (2.5) -23 (3.9) 36.5 (2.5) 1 (3.7) 51.6 (2.6) 13 (4.6) 59.5 (3.2) uruguay -1 (2.3) 50.9 (1.8) -39 (2.1) 27.3 (1.4) -12 (2.0) 43.3 (1.7) -13 (2.7) 42.0 (2.0) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. This column reports the difference between actual performance and the itted value from a regression using a cubic polynomial as regression function. 2. This column reports the percentage of students for whom the difference between actual performance and the itted value from a regression is positive. Values that are indicated in bold are signiicantly larger or smaller than 50%. 3. This column reports the difference between actual performance and the itted value from a regression using a second-degree polynomial as regression function (math, math sq., read, read sq., scie, scie sq., math×read, math×scie, read×scie). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 190 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.11a [Part 1/1] Performance on problem-solving tasks, by nature of problem and by gender items referring to a static problem situation average proportion of full-credit responses Partners OECD boys Girls Gender difference (b - G) % dif. S.E. items referring to an interactive problem situation relative likelihood of success, in favour of girls (boys = 1.00) average proportion of full-credit responses based accounting on success for booklet on remaining effects1 test items2 odds ratio % S.E. % S.E. S.E. australia 53.5 (0.8) 52.1 (0.6) -1.4 (1.1) 0.93 (0.04) austria 49.0 (1.4) 47.7 (1.4) -1.4 (1.9) belgium 50.0 (1.0) 46.6 (1.0) -3.4 (1.5) canada 54.4 (1.1) 51.0 (0.8) -3.4 chile 35.2 (1.3) 34.7 (1.1) czech republic 46.8 (1.0) 45.5 denmark 48.0 (1.6) Estonia 48.9 finland odds ratio S.E. boys Girls Gender difference (b - G) % dif. S.E. relative likelihood of success, in favour of girls (boys = 1.00) based accounting on success for booklet on remaining effects1 test items2 odds ratio S.E. odds ratio % S.E. % S.E. 0.97 (0.03) 50.2 (0.7) 49.5 (0.6) -0.6 (0.9) 0.96 (0.03) 1.03 (0.04) S.E. 0.95 (0.07) 1.02 (0.07) 43.9 (1.1) 42.3 (1.1) -1.6 (1.6) 0.93 (0.06) 0.98 (0.07) 0.86 (0.05) 0.94 (0.05) 46.4 (0.9) 44.4 (0.8) -2.0 (1.3) 0.91 (0.05) 1.06 (0.06) (1.4) 0.87 (0.05) 0.89 (0.05) 50.8 (0.8) 50.2 (0.8) -0.6 (1.0) 0.98 (0.04) 1.12 (0.07) -0.4 (1.7) 0.97 (0.07) 1.28 (0.09) 34.7 (1.2) 29.0 (0.9) -5.7 (1.5) 0.75 (0.05) 0.78 (0.06) (0.9) -1.3 (1.3) 0.95 (0.05) 1.00 (0.04) 45.0 (1.0) 43.8 (0.9) -1.2 (1.1) 0.96 (0.04) 1.00 (0.04) 47.9 (1.1) 0.0 (2.1) 1.01 (0.08) 1.16 (0.10) 44.3 (1.2) 40.5 (1.0) -3.7 (1.6) 0.87 (0.06) 0.87 (0.08) (1.5) 50.6 (1.0) 1.7 (2.1) 1.08 (0.09) 1.17 (0.10) 46.7 (1.1) 44.6 (1.2) -2.2 (1.6) 0.92 (0.06) 0.85 (0.07) 50.1 (0.8) 54.3 (0.9) 4.2 (1.2) 1.18 (0.06) 1.11 (0.05) 47.0 (0.8) 48.5 (0.8) 1.5 (1.1) 1.06 (0.04) 0.90 (0.04) france 51.6 (1.1) 49.0 (1.4) -2.6 (1.9) 0.91 (0.07) 0.98 (0.08) 48.8 (1.1) 46.4 (1.0) -2.5 (1.6) 0.93 (0.06) 1.02 (0.09) Germany 50.5 (1.2) 48.3 (1.1) -2.2 (1.7) 0.93 (0.06) 0.97 (0.06) 46.7 (1.1) 45.8 (1.1) -1.0 (1.4) 0.96 (0.05) 1.03 (0.07) hungary 36.8 (1.5) 39.6 (1.5) 2.8 (2.0) 1.13 (0.09) 1.20 (0.10) 34.5 (1.5) 33.2 (1.1) -1.3 (1.9) 0.94 (0.08) 0.83 (0.07) ireland 45.4 (1.5) 43.5 (1.1) -1.8 (1.9) 0.91 (0.07) 0.98 (0.07) 45.3 (1.5) 44.0 (1.0) -1.2 (1.8) 0.93 (0.07) 1.02 (0.07) israel 40.2 (2.5) 39.2 (1.3) -0.9 (2.7) 0.95 (0.11) 1.10 (0.08) 37.1 (2.4) 34.2 (1.1) -2.9 (2.6) 0.86 (0.10) 0.91 (0.07) italy 51.1 (1.5) 47.5 (1.5) -3.6 (2.2) 0.88 (0.07) 1.02 (0.09) 48.6 (1.3) 44.7 (1.2) -3.9 (1.8) 0.86 (0.06) 0.98 (0.08) Japan 60.1 (1.1) 57.1 (0.9) -3.1 (1.3) 0.87 (0.05) 1.05 (0.06) 57.9 (1.0) 53.8 (0.7) -4.1 (1.2) 0.83 (0.04) 0.96 (0.05) korea 60.9 (1.2) 56.6 (1.5) -4.3 (1.8) 0.83 (0.06) 0.95 (0.07) 59.1 (1.2) 56.1 (1.4) -3.0 (1.8) 0.87 (0.06) 1.05 (0.08) netherlands 51.4 (1.5) 49.4 (1.2) -2.0 (1.3) 0.92 (0.05) 0.93 (0.06) 46.6 (1.3) 46.4 (1.4) -0.2 (1.4) 0.99 (0.05) 1.07 (0.07) norway 49.6 (1.5) 49.2 (1.3) -0.4 (2.0) 0.95 (0.08) 1.01 (0.09) 44.9 (1.3) 44.1 (1.4) -0.8 (1.9) 0.93 (0.07) 0.99 (0.09) Poland 46.3 (1.5) 41.8 (1.2) -4.4 (1.7) 0.88 (0.06) 0.96 (0.08) 41.3 (1.5) 38.0 (1.3) -3.2 (1.7) 0.91 (0.07) 1.04 (0.08) Portugal 46.8 (1.4) 41.1 (1.3) -5.8 (2.0) 0.79 (0.06) 0.85 (0.07) 43.0 (1.3) 41.0 (1.1) -2.0 (1.3) 0.92 (0.05) 1.17 (0.10) Slovak republic 46.7 (1.2) 41.3 (1.4) -5.4 (1.9) 0.80 (0.06) 1.00 (0.08) 41.1 (1.2) 36.0 (1.3) -5.1 (1.9) 0.80 (0.06) 1.00 (0.08) Slovenia 42.1 (1.4) 43.8 (1.3) 1.7 (2.2) 1.08 (0.09) 1.12 (0.12) 37.2 (1.1) 36.2 (1.2) -1.0 (1.6) 0.96 (0.07) 0.89 (0.09) Spain 44.9 (1.4) 39.7 (1.0) -5.2 (1.8) 0.82 (0.06) 0.88 (0.06) 40.8 (1.0) 38.8 (1.0) -1.9 (1.4) 0.93 (0.05) 1.14 (0.08) Sweden 46.7 (1.5) 48.6 (1.2) 1.9 (2.1) 1.06 (0.08) 0.98 (0.08) 40.5 (1.1) 42.7 (0.9) 2.2 (1.4) 1.08 (0.06) 1.02 (0.08) turkey 37.5 (1.1) 33.9 (1.2) -3.6 (1.3) 0.86 (0.05) 0.98 (0.04) 34.1 (1.1) 31.2 (1.1) -2.9 (1.1) 0.88 (0.05) 1.03 (0.04) England (united kingdom) 50.4 (1.2) 48.6 (1.3) -1.8 (1.7) 0.93 (0.06) 0.98 (0.06) 48.6 (1.4) 47.4 (1.4) -1.2 (1.6) 0.95 (0.06) 1.03 (0.06) united States 48.3 (1.5) 44.9 (1.4) -3.4 (1.9) 0.86 (0.06) 0.86 (0.07) 45.9 (1.1) 46.0 (1.3) 0.1 (1.3) 1.00 (0.05) 1.16 (0.10) oEcd average 48.0 (0.3) 46.2 (0.2) -1.8 (0.3) 0.93 (0.01) 1.01 (0.01) 44.7 (0.2) 42.8 (0.2) -1.9 (0.3) 0.92 (0.01) 0.99 (0.01) brazil 31.8 (1.4) 27.9 (1.5) -3.8 (2.1) 0.84 (0.09) 1.02 (0.12) 31.1 (1.3) 27.2 (1.2) -3.8 (1.6) 0.83 (0.06) 0.98 (0.12) bulgaria 27.1 (1.1) 29.7 (1.1) 2.6 (1.2) 1.14 (0.07) 0.97 (0.05) 21.0 (0.9) 23.8 (1.1) 2.7 (1.1) 1.17 (0.08) 1.03 (0.06) colombia 28.8 (1.4) 24.0 (1.0) -4.8 (1.7) 0.78 (0.07) 1.08 (0.09) 26.8 (1.2) 20.9 (0.8) -5.9 (1.4) 0.72 (0.05) 0.92 (0.07) croatia 39.9 (1.3) 38.7 (1.1) -1.2 (1.4) 0.95 (0.05) 1.12 (0.06) 37.5 (1.1) 33.8 (1.0) -3.7 (1.3) 0.85 (0.05) 0.90 (0.05) cyprus* 36.8 (0.8) 37.2 (0.7) 0.4 (1.1) 1.02 (0.05) 1.00 (0.06) 31.2 (0.6) 31.6 (0.6) 0.4 (0.8) 1.02 (0.04) 1.00 (0.06) hong kong-china 58.2 (1.2) 53.9 (1.4) -4.3 (1.8) 0.81 (0.06) 0.98 (0.07) 53.9 (1.0) 50.1 (1.3) -3.9 (1.7) 0.83 (0.05) 1.02 (0.07) macao-china 59.2 (0.9) 54.7 (1.1) -4.5 (1.6) 0.84 (0.05) 0.95 (0.07) 53.3 (1.0) 50.1 (0.9) -3.2 (1.4) 0.88 (0.05) 1.06 (0.08) malaysia 31.2 (1.0) 29.1 (1.0) -2.1 (1.1) 0.91 (0.05) 0.97 (0.06) 28.1 (1.0) 26.8 (0.9) -1.3 (1.1) 0.94 (0.05) 1.03 (0.07) montenegro 30.7 (0.9) 29.9 (0.8) -0.8 (1.2) 0.98 (0.06) 0.83 (0.05) 23.6 (0.7) 26.4 (0.5) 2.9 (0.9) 1.19 (0.06) 1.21 (0.07) russian federation 44.4 (1.1) 43.1 (1.4) -1.3 (1.7) 0.95 (0.06) 0.94 (0.06) 39.7 (0.9) 39.8 (1.3) 0.2 (1.5) 1.01 (0.06) 1.06 (0.07) Serbia 42.1 (1.2) 38.6 (0.9) -3.6 (1.4) 0.85 (0.05) 1.05 (0.04) 39.1 (1.1) 34.5 (0.8) -4.6 (1.1) 0.81 (0.04) 0.95 (0.04) Shanghai-china 60.2 (1.3) 53.5 (1.4) -6.8 (1.8) 0.74 (0.05) 0.98 (0.07) 53.7 (1.0) 47.1 (1.3) -6.6 (1.4) 0.75 (0.04) 1.02 (0.07) Singapore 61.6 (1.1) 58.0 (1.1) -3.6 (1.6) 0.85 (0.06) 0.88 (0.06) 57.8 (0.9) 57.1 (1.0) -0.7 (1.4) 0.96 (0.05) 1.14 (0.07) chinese taipei 57.5 (1.4) 55.0 (1.3) -2.5 (2.0) 0.91 (0.08) 1.10 (0.08) 52.5 (1.6) 47.7 (1.2) -4.8 (2.2) 0.83 (0.07) 0.91 (0.06) united arab Emirates 28.4 (1.1) 31.3 (0.9) 3.0 (1.6) 1.16 (0.09) 0.91 (0.06) 24.8 (0.9) 29.2 (0.8) 4.5 (1.3) 1.27 (0.08) 1.09 (0.07) uruguay 27.8 (0.9) 27.2 (0.8) -0.6 (1.0) 0.97 (0.05) 1.07 (0.06) 25.8 (0.8) 23.9 (0.7) -1.8 (0.9) 0.91 (0.04) 0.94 (0.05) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, a female dummy, and an interaction term (female × item type). Booklet dummies are added to the estimation. This column presents the difference between the logit coeficient on the interaction term and the logit coeficient on the item type dummy in exponentiated form. 2. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, a female dummy, and an interaction term (female × item type). Booklet dummies are added to the estimation. This column presents the logit coeficient on the interaction term in exponentiated form. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 191 Annex b1: reSulTS For counTrIeS And economIeS table v.4.11b [Part 1/2] Performance on problem-solving tasks, by process and by gender items assessing the process of “exploring and understanding” average proportion of full-credit responses Partners OECD boys Girls Gender difference (b - G) % dif. S.E. items assessing the process of “representing and formulating” relative likelihood of success, in favour of girls (boys = 1.00) average proportion of full-credit responses based accounting on success for booklet on remaining effects1 test items2 odds ratio % S.E. % S.E. S.E. australia 56.0 (0.9) 53.9 (0.7) -2.1 (1.1) 0.91 (0.04) austria 49.6 (1.6) 48.8 (1.5) -0.9 (2.2) belgium 49.8 (1.1) 48.2 (1.1) -1.6 (1.8) canada 54.1 (1.0) 54.0 (1.0) -0.2 chile 34.4 (1.5) 30.6 (1.2) czech republic 47.4 (1.0) 46.4 denmark 47.7 (1.4) Estonia 48.0 finland odds ratio S.E. boys Girls Gender difference (b - G) % dif. S.E. relative likelihood of success, in favour of girls (boys = 1.00) based accounting on success for booklet on remaining effects1 test items2 odds ratio S.E. odds ratio % S.E. % S.E. 0.94 (0.03) 51.1 (0.9) 47.5 (0.8) -3.6 (1.2) 0.85 (0.04) 0.87 (0.03) S.E. 0.97 (0.09) 1.03 (0.07) 43.6 (1.4) 40.0 (1.5) -3.7 (2.0) 0.85 (0.07) 0.88 (0.06) 0.93 (0.07) 1.05 (0.06) 47.4 (1.2) 42.1 (1.1) -5.3 (1.7) 0.80 (0.06) 0.86 (0.05) (1.3) 0.99 (0.06) 1.08 (0.06) 52.7 (1.2) 48.9 (1.0) -3.8 (1.4) 0.86 (0.05) 0.90 (0.05) -3.8 (1.8) 0.83 (0.07) 1.00 (0.07) 32.3 (1.6) 26.3 (1.2) -6.1 (2.1) 0.73 (0.07) 0.86 (0.07) (1.2) -1.0 (1.4) 0.96 (0.05) 1.01 (0.04) 44.5 (1.2) 41.3 (1.0) -3.2 (1.3) 0.88 (0.05) 0.90 (0.03) 44.6 (1.3) -3.0 (1.8) 0.90 (0.06) 0.98 (0.06) 45.0 (1.6) 39.4 (1.4) -5.5 (2.0) 0.82 (0.06) 0.86 (0.06) (1.5) 49.7 (1.4) 1.7 (2.2) 1.08 (0.10) 1.14 (0.08) 46.5 (1.4) 42.5 (1.4) -4.0 (1.9) 0.86 (0.07) 0.85 (0.06) 52.8 (0.9) 54.7 (1.0) 1.9 (1.4) 1.08 (0.06) 0.97 (0.04) 46.0 (1.0) 46.6 (1.0) 0.6 (1.4) 1.02 (0.06) 0.91 (0.04) france 53.9 (1.3) 50.5 (1.4) -3.4 (2.0) 0.88 (0.07) 0.94 (0.07) 48.6 (1.4) 45.4 (1.2) -3.2 (1.9) 0.90 (0.07) 0.97 (0.05) Germany 51.7 (1.4) 49.5 (1.5) -2.2 (1.8) 0.91 (0.07) 0.96 (0.06) 45.3 (1.4) 42.8 (1.4) -2.5 (1.8) 0.89 (0.07) 0.92 (0.06) hungary 36.8 (1.5) 38.6 (1.4) 1.8 (1.8) 1.08 (0.08) 1.09 (0.08) 33.8 (1.7) 31.0 (1.4) -2.8 (2.3) 0.87 (0.09) 0.84 (0.06) ireland 48.2 (2.1) 46.9 (1.3) -1.3 (2.5) 0.93 (0.10) 1.01 (0.08) 42.8 (1.5) 40.1 (1.2) -2.7 (2.0) 0.87 (0.08) 0.92 (0.06) israel 43.1 (2.7) 40.9 (1.1) -2.2 (2.8) 0.90 (0.10) 1.01 (0.07) 37.5 (2.6) 33.0 (1.6) -4.5 (3.0) 0.80 (0.10) 0.87 (0.07) italy 53.4 (1.7) 49.2 (1.5) -4.2 (2.2) 0.85 (0.07) 0.98 (0.07) 49.4 (1.7) 44.6 (1.5) -4.8 (2.1) 0.83 (0.07) 0.95 (0.06) Japan 64.3 (1.3) 59.9 (1.1) -4.4 (1.5) 0.81 (0.05) 0.95 (0.05) 58.9 (1.2) 52.3 (1.0) -6.6 (1.4) 0.75 (0.04) 0.85 (0.03) korea 67.4 (1.4) 61.6 (1.5) -5.8 (1.9) 0.76 (0.06) 0.86 (0.06) 64.7 (1.6) 56.0 (1.9) -8.6 (2.3) 0.67 (0.06) 0.74 (0.05) netherlands 52.5 (1.4) 51.0 (1.4) -1.5 (1.4) 0.94 (0.05) 0.97 (0.05) 44.8 (1.6) 43.6 (1.6) -1.2 (1.7) 0.95 (0.06) 0.98 (0.04) norway 51.4 (1.4) 51.2 (1.5) -0.3 (2.0) 0.95 (0.08) 1.02 (0.07) 44.9 (1.5) 42.2 (1.7) -2.7 (2.2) 0.86 (0.08) 0.90 (0.07) Poland 44.7 (1.7) 42.8 (1.4) -1.9 (1.8) 0.97 (0.08) 1.10 (0.08) 42.2 (1.8) 34.8 (1.6) -7.3 (2.2) 0.76 (0.07) 0.81 (0.06) Portugal 46.4 (1.6) 40.5 (1.4) -5.9 (1.6) 0.78 (0.06) 0.87 (0.07) 42.3 (1.8) 36.4 (1.4) -5.9 (1.8) 0.78 (0.06) 0.87 (0.06) Slovak republic 46.0 (1.5) 40.6 (1.5) -5.4 (2.1) 0.80 (0.07) 1.00 (0.07) 40.9 (1.4) 32.5 (1.6) -8.4 (2.2) 0.69 (0.07) 0.83 (0.05) Slovenia 39.2 (1.3) 40.1 (1.6) 0.9 (2.0) 1.04 (0.09) 1.06 (0.09) 36.5 (1.5) 35.0 (1.3) -1.5 (2.0) 0.94 (0.08) 0.92 (0.06) Spain 45.7 (1.4) 39.2 (1.4) -6.5 (1.9) 0.77 (0.06) 0.83 (0.06) 39.2 (1.4) 35.4 (1.2) -3.8 (1.9) 0.85 (0.07) 0.95 (0.06) Sweden 47.9 (1.6) 48.6 (1.3) 0.7 (2.0) 1.01 (0.08) 0.92 (0.07) 41.7 (1.4) 42.0 (1.4) 0.3 (1.9) 0.99 (0.08) 0.90 (0.06) turkey 35.4 (1.0) 31.6 (1.3) -3.7 (1.3) 0.85 (0.05) 0.96 (0.04) 33.7 (1.4) 29.9 (1.3) -3.8 (1.5) 0.84 (0.06) 0.96 (0.04) England (united kingdom) 53.0 (1.5) 49.8 (1.7) -3.1 (2.1) 0.88 (0.07) 0.91 (0.06) 49.9 (1.6) 45.7 (1.6) -4.2 (1.8) 0.85 (0.06) 0.87 (0.04) united States 49.4 (1.5) 48.5 (1.3) -0.9 (1.6) 0.96 (0.07) 1.02 (0.07) 45.4 (1.5) 42.4 (1.7) -3.1 (1.9) 0.88 (0.07) 0.91 (0.07) oEcd average 48.9 (0.3) 46.9 (0.3) -2.1 (0.4) 0.91 (0.01) 0.99 (0.01) 44.7 (0.3) 40.7 (0.3) -4.0 (0.4) 0.84 (0.01) 0.89 (0.01) brazil 33.0 (1.6) 27.6 (1.4) -5.3 (1.8) 0.77 (0.07) 0.91 (0.08) 28.5 (1.5) 22.5 (1.5) -5.9 (1.8) 0.73 (0.07) 0.85 (0.06) bulgaria 26.7 (1.2) 29.0 (1.1) 2.3 (1.3) 1.13 (0.07) 0.96 (0.04) 18.3 (1.1) 20.0 (1.1) 1.7 (1.2) 1.12 (0.09) 0.96 (0.05) colombia 28.5 (1.5) 21.4 (1.1) -7.1 (1.8) 0.68 (0.07) 0.89 (0.09) 23.0 (1.4) 14.8 (0.8) -8.2 (1.6) 0.57 (0.06) 0.74 (0.06) croatia 38.8 (1.2) 35.7 (1.1) -3.1 (1.4) 0.88 (0.05) 0.98 (0.04) 35.1 (1.6) 30.9 (1.3) -4.1 (1.7) 0.83 (0.06) 0.92 (0.04) cyprus* 35.8 (0.8) 36.5 (0.7) 0.7 (0.9) 1.03 (0.04) 1.02 (0.04) 31.5 (0.8) 29.9 (0.8) -1.6 (1.1) 0.93 (0.05) 0.89 (0.04) hong kong-china 63.5 (1.6) 56.4 (1.5) -7.0 (1.9) 0.72 (0.06) 0.84 (0.06) 58.8 (1.3) 50.2 (1.5) -8.7 (2.0) 0.68 (0.05) 0.78 (0.05) macao-china 62.4 (1.3) 56.4 (1.0) -6.0 (1.5) 0.78 (0.06) 0.87 (0.06) 60.1 (1.2) 54.2 (1.2) -5.9 (1.7) 0.78 (0.05) 0.88 (0.05) malaysia 30.8 (1.1) 29.3 (1.0) -1.5 (1.2) 0.93 (0.05) 1.01 (0.05) 29.7 (1.4) 26.2 (1.2) -3.5 (1.5) 0.84 (0.06) 0.88 (0.05) montenegro 26.9 (0.9) 27.7 (0.8) 0.8 (1.3) 1.07 (0.07) 0.95 (0.06) 22.6 (0.9) 24.5 (0.7) 1.9 (1.2) 1.13 (0.07) 1.03 (0.06) russian federation 42.5 (1.3) 41.6 (1.6) -0.9 (2.1) 0.97 (0.08) 0.97 (0.07) 39.5 (1.4) 37.6 (1.6) -1.9 (2.1) 0.93 (0.08) 0.92 (0.06) Serbia 41.1 (1.4) 37.9 (0.9) -3.2 (1.5) 0.87 (0.05) 1.07 (0.05) 39.6 (1.3) 31.8 (0.9) -7.9 (1.4) 0.70 (0.04) 0.81 (0.04) Shanghai-china 60.2 (1.3) 56.6 (1.6) -3.6 (1.9) 0.84 (0.07) 1.17 (0.10) 61.8 (1.4) 49.3 (1.6) -12.5 (1.7) 0.58 (0.04) 0.73 (0.05) Singapore 65.5 (1.3) 62.5 (1.2) -3.0 (1.6) 0.87 (0.07) 0.92 (0.06) 62.2 (1.3) 57.1 (1.2) -5.1 (1.9) 0.80 (0.06) 0.83 (0.06) chinese taipei 61.1 (1.6) 55.3 (1.4) -5.9 (2.2) 0.79 (0.08) 0.90 (0.06) 59.1 (2.1) 52.1 (1.6) -7.0 (2.8) 0.75 (0.09) 0.84 (0.06) united arab Emirates 28.0 (1.0) 31.8 (0.9) 3.9 (1.4) 1.21 (0.08) 0.99 (0.05) 24.8 (1.1) 28.2 (1.0) 3.5 (1.5) 1.20 (0.10) 0.98 (0.06) uruguay 27.7 (1.1) 26.6 (0.8) -1.1 (1.2) 0.95 (0.06) 1.02 (0.06) 23.9 (1.0) 20.6 (0.9) -3.2 (1.1) 0.83 (0.05) 0.87 (0.05) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, a female dummy, and an interaction term (female × item type). Booklet dummies are added to the estimation. This column presents the difference between the logit coeficient on the interaction term and the logit coeficient on the item type dummy in exponentiated form. 2. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, a female dummy, and an interaction term (female × item type). Booklet dummies are added to the estimation. This column presents the logit coeficient on the interaction term in exponentiated form. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 192 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.11b [Part 2/2] Performance on problem-solving tasks, by process and by gender items assessing the process of “planning and executing” average proportion of full-credit responses Partners OECD boys Girls Gender difference (b - G) % dif. S.E. items assessing the process of “monitoring and relecting” relative likelihood of success, in favour of girls (boys = 1.00) average proportion of full-credit responses based accounting on success for booklet on remaining effects1 test items2 odds ratio % S.E. % S.E. S.E. australia 51.3 (0.7) 51.7 (0.6) 0.4 (0.9) 1.01 (0.04) austria 47.8 (1.2) 47.1 (1.2) -0.8 (1.5) belgium 48.5 (0.8) 46.6 (0.9) -1.9 (1.2) canada 53.1 (0.8) 51.1 (0.8) -2.0 chile 37.2 (1.1) 33.3 (0.9) czech republic 47.1 (0.9) 46.7 denmark 49.1 (1.4) Estonia 50.2 finland odds ratio S.E. boys Girls Gender difference (b - G) % dif. S.E. relative likelihood of success, in favour of girls (boys = 1.00) based accounting on success for booklet on remaining effects1 test items2 odds ratio S.E. odds ratio % S.E. % S.E. 1.10 (0.03) 45.5 (0.7) 46.4 (0.7) 0.9 (1.0) 1.03 (0.04) 1.09 (0.04) S.E. 0.98 (0.06) 1.07 (0.06) 38.0 (1.3) 36.5 (1.2) -1.5 (1.8) 0.94 (0.07) 1.00 (0.07) 0.92 (0.05) 1.04 (0.04) 43.2 (1.0) 41.6 (1.0) -1.6 (1.4) 0.93 (0.06) 1.05 (0.06) (1.1) 0.92 (0.04) 0.97 (0.04) 46.0 (1.0) 46.1 (1.0) 0.1 (1.3) 1.01 (0.05) 1.09 (0.05) -4.0 (1.4) 0.82 (0.05) 0.99 (0.05) 33.6 (1.3) 32.8 (1.0) -0.8 (1.7) 0.95 (0.07) 1.19 (0.09) (0.8) -0.5 (1.1) 0.98 (0.04) 1.05 (0.03) 41.0 (1.0) 40.4 (0.9) -0.6 (1.2) 0.98 (0.05) 1.03 (0.04) 47.1 (1.1) -2.0 (1.8) 0.93 (0.06) 1.02 (0.06) 35.3 (1.6) 36.8 (1.2) 1.5 (2.0) 1.08 (0.10) 1.21 (0.10) (1.3) 48.9 (1.0) -1.3 (1.7) 0.96 (0.07) 0.97 (0.06) 42.0 (1.2) 42.9 (1.3) 0.9 (1.9) 1.05 (0.09) 1.09 (0.08) 49.3 (0.7) 52.9 (0.7) 3.6 (1.0) 1.15 (0.04) 1.08 (0.04) 41.2 (0.8) 44.2 (0.8) 3.0 (1.1) 1.13 (0.05) 1.03 (0.04) france 50.2 (1.1) 48.6 (1.1) -1.6 (1.5) 0.95 (0.06) 1.06 (0.05) 44.9 (1.1) 42.7 (1.3) -2.2 (1.7) 0.94 (0.07) 1.02 (0.07) Germany 49.9 (1.1) 49.1 (1.0) -0.8 (1.4) 0.98 (0.06) 1.06 (0.06) 42.6 (1.2) 41.9 (1.2) -0.6 (1.6) 0.98 (0.06) 1.04 (0.07) hungary 36.8 (1.5) 38.4 (1.2) 1.6 (1.9) 1.07 (0.09) 1.11 (0.06) 31.6 (1.7) 30.2 (1.4) -1.4 (2.2) 0.93 (0.09) 0.91 (0.07) ireland 46.2 (1.4) 44.8 (1.0) -1.3 (1.9) 0.94 (0.07) 1.02 (0.06) 42.4 (1.6) 42.1 (1.3) -0.3 (2.0) 0.97 (0.08) 1.06 (0.08) israel 37.5 (2.4) 36.4 (1.3) -1.1 (2.8) 0.94 (0.11) 1.09 (0.07) 33.8 (2.1) 31.7 (1.4) -2.2 (2.4) 0.89 (0.10) 1.00 (0.07) italy 49.7 (1.4) 45.8 (1.3) -3.9 (2.0) 0.86 (0.06) 1.00 (0.07) 43.7 (1.4) 41.7 (1.4) -1.9 (2.2) 0.93 (0.09) 1.09 (0.10) Japan 57.1 (1.0) 55.5 (0.8) -1.6 (1.1) 0.93 (0.04) 1.16 (0.05) 54.0 (1.1) 50.1 (0.8) -3.9 (1.3) 0.84 (0.04) 1.00 (0.05) korea 54.7 (1.2) 54.2 (1.3) -0.5 (1.7) 0.98 (0.07) 1.24 (0.07) 53.8 (1.4) 53.5 (1.5) -0.3 (1.9) 0.99 (0.08) 1.19 (0.07) netherlands 50.1 (1.3) 49.3 (1.2) -0.9 (1.1) 0.97 (0.04) 1.00 (0.04) 42.5 (1.5) 43.1 (1.3) 0.6 (1.6) 1.02 (0.07) 1.07 (0.06) norway 48.1 (1.4) 48.1 (1.2) 0.0 (1.8) 0.96 (0.07) 1.04 (0.06) 38.5 (1.5) 38.3 (1.6) -0.2 (2.2) 0.96 (0.09) 1.03 (0.07) Poland 45.1 (1.4) 42.2 (1.2) -2.9 (1.7) 0.93 (0.06) 1.06 (0.06) 37.1 (1.4) 34.0 (1.3) -3.1 (1.8) 0.92 (0.07) 1.02 (0.07) Portugal 46.3 (1.3) 45.1 (1.3) -1.3 (1.6) 0.95 (0.06) 1.15 (0.08) 39.6 (1.6) 38.4 (1.3) -1.3 (1.9) 0.96 (0.08) 1.12 (0.08) Slovak republic 45.2 (1.1) 40.8 (1.2) -4.3 (1.7) 0.84 (0.06) 1.08 (0.05) 37.1 (1.1) 33.9 (1.4) -3.2 (1.8) 0.86 (0.07) 1.09 (0.06) Slovenia 41.7 (0.9) 42.9 (1.1) 1.1 (1.4) 1.05 (0.06) 1.09 (0.06) 35.3 (1.1) 33.0 (1.4) -2.3 (1.9) 0.90 (0.08) 0.88 (0.06) Spain 43.0 (1.4) 41.5 (1.0) -1.5 (1.6) 0.96 (0.06) 1.13 (0.07) 39.5 (1.1) 38.5 (1.3) -1.0 (1.4) 0.99 (0.06) 1.13 (0.08) Sweden 42.9 (1.2) 46.2 (0.9) 3.3 (1.5) 1.13 (0.07) 1.08 (0.06) 35.8 (1.4) 40.0 (1.2) 4.2 (1.9) 1.19 (0.09) 1.13 (0.07) turkey 37.3 (1.0) 34.6 (1.0) -2.7 (1.2) 0.89 (0.05) 1.04 (0.05) 32.6 (1.1) 30.2 (1.2) -2.5 (1.3) 0.89 (0.06) 1.03 (0.05) England (united kingdom) 49.1 (1.2) 49.2 (1.3) 0.1 (1.5) 1.00 (0.06) 1.10 (0.04) 43.5 (1.3) 44.5 (1.3) 1.1 (1.7) 1.05 (0.07) 1.13 (0.05) united States 47.7 (1.2) 46.6 (1.3) -1.1 (1.6) 0.95 (0.06) 1.00 (0.06) 42.7 (1.3) 43.4 (1.5) 0.7 (1.5) 1.02 (0.06) 1.09 (0.07) oEcd average 46.9 (0.2) 45.9 (0.2) -1.0 (0.3) 0.96 (0.01) 1.06 (0.01) 40.6 (0.2) 40.0 (0.2) -0.6 (0.3) 0.97 (0.01) 1.06 (0.01) brazil 33.0 (1.2) 31.1 (1.3) -2.0 (1.5) 0.92 (0.07) 1.19 (0.07) 28.8 (1.2) 25.6 (1.4) -3.3 (1.9) 0.85 (0.08) 1.02 (0.09) bulgaria 25.1 (0.9) 28.4 (1.1) 3.3 (1.2) 1.19 (0.07) 1.04 (0.05) 20.2 (1.0) 23.2 (1.2) 3.0 (1.2) 1.20 (0.09) 1.04 (0.05) colombia 30.2 (1.2) 25.6 (1.1) -4.6 (1.7) 0.79 (0.07) 1.12 (0.09) 25.7 (1.2) 24.1 (1.1) -1.5 (1.6) 0.92 (0.08) 1.31 (0.10) croatia 41.5 (1.1) 39.5 (1.0) -1.9 (1.2) 0.92 (0.05) 1.07 (0.04) 34.9 (1.2) 32.2 (0.9) -2.7 (1.2) 0.89 (0.05) 1.00 (0.04) cyprus* 34.7 (0.6) 34.9 (0.7) 0.2 (0.9) 1.01 (0.04) 0.98 (0.03) 28.3 (0.7) 31.3 (0.7) 3.0 (1.0) 1.15 (0.05) 1.16 (0.05) hong kong-china 51.2 (1.0) 51.0 (1.3) -0.2 (1.7) 0.96 (0.06) 1.28 (0.07) 48.9 (1.3) 47.3 (1.7) -1.6 (2.1) 0.91 (0.08) 1.13 (0.08) macao-china 52.2 (0.8) 50.4 (0.9) -1.8 (1.4) 0.94 (0.05) 1.14 (0.06) 46.5 (1.1) 44.9 (1.1) -1.7 (1.5) 0.94 (0.06) 1.11 (0.06) malaysia 30.1 (0.9) 28.5 (0.9) -1.6 (1.0) 0.93 (0.04) 1.00 (0.04) 24.2 (0.9) 24.9 (1.0) 0.7 (1.1) 1.04 (0.06) 1.15 (0.06) montenegro 29.3 (0.8) 30.6 (0.7) 1.3 (1.0) 1.09 (0.05) 0.97 (0.05) 22.1 (0.9) 24.9 (0.6) 2.8 (1.1) 1.19 (0.08) 1.10 (0.05) russian federation 44.0 (0.9) 43.6 (1.2) -0.4 (1.5) 0.99 (0.05) 1.00 (0.06) 36.2 (1.0) 38.5 (1.5) 2.3 (1.7) 1.12 (0.08) 1.15 (0.09) Serbia 42.4 (1.1) 39.1 (0.8) -3.3 (1.2) 0.87 (0.04) 1.08 (0.04) 34.9 (1.2) 31.3 (1.0) -3.5 (1.3) 0.84 (0.05) 1.02 (0.04) Shanghai-china 53.4 (1.0) 46.5 (1.3) -6.9 (1.7) 0.74 (0.04) 0.98 (0.06) 48.6 (1.5) 45.8 (1.5) -2.9 (2.0) 0.88 (0.08) 1.22 (0.09) Singapore 55.2 (1.0) 55.5 (1.2) 0.3 (1.6) 1.01 (0.06) 1.15 (0.07) 55.3 (1.1) 55.1 (1.1) -0.3 (1.6) 0.99 (0.07) 1.08 (0.07) chinese taipei 50.7 (1.3) 49.5 (1.1) -1.2 (1.9) 0.96 (0.07) 1.21 (0.07) 46.7 (1.6) 42.8 (1.4) -3.9 (2.2) 0.86 (0.08) 1.01 (0.07) united arab Emirates 26.4 (1.0) 31.4 (0.9) 5.0 (1.5) 1.28 (0.09) 1.08 (0.05) 24.2 (1.1) 26.4 (0.9) 2.2 (1.5) 1.13 (0.09) 0.91 (0.05) uruguay 28.4 (0.8) 27.4 (0.7) -1.0 (0.8) 0.95 (0.04) 1.04 (0.04) 23.9 (0.8) 23.5 (0.9) -0.4 (0.9) 0.98 (0.05) 1.06 (0.04) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, a female dummy, and an interaction term (female × item type). Booklet dummies are added to the estimation. This column presents the difference between the logit coeficient on the interaction term and the logit coeficient on the item type dummy in exponentiated form. 2. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, a female dummy, and an interaction term (female × item type). Booklet dummies are added to the estimation. This column presents the logit coeficient on the interaction term in exponentiated form. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 193 Annex b1: reSulTS For counTrIeS And economIeS table v.4.12 [Part 1/2] Performance in problem solving, by socio-economic status Results based on students’ self-reports PiSa index of economic, social and cultural status (EScS) OECD all students Second quarter third quarter top quarter S.E. mean index S.E. mean index S.E. mean index S.E. mean index S.E. australia 0.25 (0.01) -0.84 (0.02) 0.05 (0.02) 0.61 (0.01) 1.18 (0.01) austria 0.08 (0.02) -0.97 (0.03) -0.25 (0.02) 0.33 (0.03) 1.19 (0.03) belgium 0.15 (0.02) -1.05 (0.03) -0.19 (0.03) 0.55 (0.02) 1.27 (0.02) canada 0.41 (0.02) -0.75 (0.02) 0.16 (0.02) 0.79 (0.02) 1.44 (0.01) chile -0.58 (0.04) -1.97 (0.05) -1.02 (0.04) -0.27 (0.05) 0.95 (0.03) czech republic -0.07 (0.02) -0.98 (0.02) -0.37 (0.02) 0.16 (0.02) 0.93 (0.02) denmark 0.43 (0.02) -0.70 (0.03) 0.16 (0.04) 0.81 (0.03) 1.44 (0.02) Estonia 0.11 (0.01) -0.92 (0.02) -0.23 (0.02) 0.44 (0.02) 1.16 (0.01) finland 0.36 (0.02) -0.68 (0.02) 0.13 (0.02) 0.73 (0.02) 1.28 (0.01) france -0.04 (0.02) -1.10 (0.02) -0.30 (0.02) 0.29 (0.02) 0.95 (0.01) Germany 0.19 (0.02) -0.99 (0.03) -0.16 (0.03) 0.52 (0.04) 1.42 (0.02) hungary -0.25 (0.03) -1.46 (0.04) -0.65 (0.03) 0.09 (0.04) 1.01 (0.03) ireland 0.13 (0.02) -0.97 (0.02) -0.19 (0.03) 0.48 (0.03) 1.20 (0.02) israel 0.17 (0.03) -0.98 (0.04) -0.03 (0.04) 0.58 (0.03) 1.12 (0.02) italy -0.03 (0.03) -1.24 (0.03) -0.37 (0.03) 0.26 (0.03) 1.25 (0.04) Japan -0.07 (0.02) -0.99 (0.02) -0.35 (0.02) 0.20 (0.02) 0.85 (0.02) korea 0.01 (0.03) -0.97 (0.03) -0.23 (0.03) 0.33 (0.03) 0.92 (0.02) netherlands 0.23 (0.02) -0.82 (0.03) 0.02 (0.03) 0.58 (0.02) 1.15 (0.02) norway 0.46 (0.02) -0.56 (0.02) 0.27 (0.02) 0.79 (0.02) 1.35 (0.02) Poland -0.21 (0.03) -1.22 (0.02) -0.69 (0.02) -0.01 (0.05) 1.08 (0.03) Portugal -0.48 (0.05) -1.85 (0.03) -1.06 (0.04) -0.23 (0.07) 1.21 (0.07) Slovak republic -0.18 (0.03) -1.25 (0.04) -0.57 (0.02) 0.02 (0.04) 1.06 (0.03) 0.07 (0.01) -1.03 (0.01) -0.31 (0.02) 0.39 (0.02) 1.22 (0.02) -0.18 (0.03) -1.49 (0.03) -0.59 (0.03) 0.18 (0.05) 1.17 (0.03) Sweden 0.28 (0.02) -0.82 (0.02) 0.02 (0.02) 0.65 (0.02) 1.25 (0.01) turkey -1.46 (0.04) -2.74 (0.03) -1.96 (0.03) -1.21 (0.05) 0.07 (0.06) England (united kingdom) 0.29 (0.02) -0.76 (0.03) 0.02 (0.04) 0.62 (0.03) 1.27 (0.02) united States 0.17 (0.04) -1.14 (0.05) -0.11 (0.04) 0.60 (0.04) 1.35 (0.04) oEcd average 0.01 (0.00) -1.11 (0.01) -0.31 (0.01) 0.33 (0.01) 1.13 (0.01) brazil -1.11 (0.04) -2.60 (0.04) -1.56 (0.04) -0.74 (0.05) 0.47 (0.06) bulgaria -0.28 (0.04) -1.59 (0.06) -0.67 (0.03) 0.10 (0.04) 1.06 (0.03) colombia -1.26 (0.04) -2.82 (0.04) -1.65 (0.05) -0.83 (0.04) 0.24 (0.05) croatia -0.34 (0.02) -1.35 (0.02) -0.70 (0.02) -0.14 (0.03) 0.84 (0.02) cyprus* 0.09 (0.01) -1.06 (0.02) -0.28 (0.01) 0.43 (0.02) 1.25 (0.02) hong kong-china -0.79 (0.05) -2.00 (0.03) -1.20 (0.05) -0.46 (0.07) 0.50 (0.06) macao-china -0.89 (0.01) -1.91 (0.01) -1.23 (0.01) -0.68 (0.01) 0.28 (0.02) malaysia -0.72 (0.03) -1.99 (0.04) -1.07 (0.03) -0.38 (0.05) 0.54 (0.04) montenegro -0.25 (0.01) -1.40 (0.02) -0.57 (0.02) 0.09 (0.02) 0.89 (0.02) russian federation -0.11 (0.02) -1.10 (0.03) -0.37 (0.03) 0.22 (0.03) 0.82 (0.02) Serbia -0.30 (0.02) -1.37 (0.02) -0.70 (0.03) -0.05 (0.03) 0.95 (0.03) Shanghai-china -0.36 (0.04) -1.63 (0.05) -0.70 (0.04) 0.06 (0.04) 0.83 (0.03) Singapore -0.26 (0.01) -1.46 (0.02) -0.54 (0.02) 0.09 (0.02) 0.88 (0.02) chinese taipei -0.40 (0.02) -1.47 (0.03) -0.70 (0.03) -0.11 (0.03) 0.68 (0.03) 0.32 (0.02) -0.82 (0.03) 0.19 (0.02) 0.67 (0.01) 1.26 (0.01) -0.88 (0.03) -2.23 (0.02) -1.40 (0.03) -0.59 (0.04) 0.69 (0.05) Slovenia Spain Partners bottom quarter mean index united arab Emirates uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. Single-level bivariate regression of performance on the PISA index of economic, social and cultural status (ESCS). The slope of the gradient is the regression coeficient for ESCS; the strength of the relationship is the r-squared. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 194 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.12 [Part 2/2] Performance in problem solving, by socio-economic status Results based on students’ self-reports Partners OECD Performance in problem solving, by national quarters of this index bottom quarter Second quarter mean score mean score S.E. S.E. third quarter mean score S.E. top quarter mean score increased likelihood of students in the bottom quarter of the EScS index scoring in the bottom quarter of the problem-solving performance distribution S.E. relative risk S.E. Slope of the socio-economic gradient1 Strength of the relationship between student performance and EScS1 Score-point difference in problem solving associated with one-unit increase in the EScS Percentage of explained variation in student performance (r-squared x 100) S.E. S.E. australia 487 (2.6) 512 (2.4) 538 (3.1) 560 (2.5) 1.88 (0.07) 36 (1.3) 8.5 (0.6) austria 467 (4.7) 495 (5.4) 518 (4.8) 547 (4.9) 1.98 (0.13) 36 (2.6) 10.7 (1.4) belgium 458 (4.3) 495 (4.0) 529 (3.3) 557 (3.5) 2.22 (0.13) 43 (2.3) 14.0 (1.5) canada 503 (3.4) 518 (2.8) 534 (3.3) 555 (3.2) 1.52 (0.07) 23 (1.7) 4.0 (0.6) chile 405 (5.9) 439 (4.6) 454 (4.0) 493 (4.8) 2.12 (0.17) 30 (1.9) 15.8 (1.8) czech republic 460 (4.9) 500 (5.0) 519 (4.1) 557 (4.2) 2.25 (0.17) 49 (2.8) 14.9 (1.5) denmark 465 (5.2) 488 (4.0) 511 (3.7) 529 (3.5) 1.89 (0.14) 31 (2.3) 7.9 (1.2) Estonia 495 (3.8) 503 (3.8) 516 (4.1) 547 (3.4) 1.39 (0.11) 25 (2.0) 5.4 (0.8) finland 495 (3.7) 513 (3.0) 531 (3.7) 556 (3.0) 1.67 (0.10) 30 (2.2) 6.5 (0.9) france 472 (6.0) 497 (4.1) 521 (4.4) 559 (4.1) 2.01 (0.15) 43 (2.8) 12.7 (1.2) Germany 469 (5.6) 500 (4.5) 539 (4.4) 555 (4.2) 2.17 (0.15) 37 (2.4) 12.7 (1.4) hungary 397 (7.2) 445 (4.8) 474 (5.2) 520 (6.4) 2.74 (0.20) 49 (3.3) 20.5 (2.3) ireland 460 (4.7) 489 (4.2) 510 (3.5) 538 (4.8) 1.93 (0.14) 35 (2.2) 10.2 (1.1) israel 393 (5.7) 437 (6.9) 477 (7.1) 513 (7.1) 2.14 (0.14) 53 (3.0) 13.2 (1.4) italy 481 (5.6) 500 (4.4) 524 (5.3) 535 (5.6) 1.68 (0.15) 23 (2.5) 5.9 (1.2) Japan 526 (5.3) 547 (3.6) 562 (4.0) 576 (4.2) 1.73 (0.13) 27 (3.1) 5.2 (1.1) korea 534 (5.3) 552 (5.1) 571 (5.2) 588 (5.5) 1.60 (0.13) 28 (3.0) 5.4 (1.1) netherlands 473 (6.7) 502 (5.3) 523 (5.3) 549 (6.3) 1.84 (0.18) 38 (3.8) 9.1 (1.6) norway 473 (4.5) 495 (4.1) 518 (4.7) 533 (5.0) 1.66 (0.12) 31 (2.7) 5.2 (0.9) Poland 441 (5.5) 467 (5.2) 491 (5.8) 526 (6.3) 1.95 (0.18) 36 (2.7) 11.6 (1.7) Portugal 449 (4.7) 485 (4.5) 504 (4.7) 543 (5.8) 2.27 (0.15) 30 (1.9) 16.1 (2.0) Slovak republic 424 (7.5) 477 (4.2) 495 (4.2) 541 (5.5) 2.83 (0.27) 49 (3.3) 21.3 (2.0) Slovenia 434 (2.6) 463 (3.4) 488 (3.4) 522 (2.8) 1.91 (0.12) 40 (1.6) 12.6 (1.0) Spain 437 (7.2) 469 (4.3) 485 (4.9) 517 (6.6) 1.84 (0.13) 29 (3.0) 7.9 (1.5) Sweden 460 (3.7) 482 (4.1) 507 (4.7) 521 (4.5) 1.62 (0.11) 29 (2.3) 6.2 (1.0) turkey 419 (4.3) 443 (4.0) 459 (5.1) 497 (6.2) 1.95 (0.15) 28 (1.9) 15.1 (1.8) England (united kingdom) 486 (5.4) 505 (5.5) 531 (5.0) 555 (4.6) 1.74 (0.13) 33 (2.8) 7.8 (1.1) united States 473 (5.7) 493 (4.7) 518 (5.1) 549 (4.7) 1.87 (0.17) 30 (2.0) 10.1 (1.2) oEcd average 462 (1.0) 490 (0.8) 512 (0.9) 541 (0.9) 1.94 (0.03) 35 (0.5) 10.6 (0.3) brazil 385 (6.2) 420 (6.8) 436 (6.8) 477 (7.0) 2.13 (0.19) 30 (2.5) 14.6 (2.4) bulgaria 343 (8.3) 387 (5.9) 416 (6.6) 465 (6.8) 2.33 (0.19) 45 (3.6) 20.0 (2.5) colombia 359 (4.5) 388 (4.5) 406 (4.3) 442 (5.9) 1.97 (0.14) 27 (1.9) 12.6 (1.6) croatia 434 (5.1) 458 (4.4) 469 (4.9) 504 (5.5) 1.70 (0.12) 32 (2.6) 8.6 (1.2) cyprus* 406 (3.1) 438 (3.3) 450 (3.0) 488 (3.0) 1.84 (0.11) 34 (1.6) 9.5 (0.9) hong kong-china 517 (5.5) 533 (5.0) 546 (4.1) 567 (6.9) 1.58 (0.12) 21 (2.9) 4.9 (1.3) macao-china 530 (2.4) 540 (2.3) 545 (2.0) 548 (2.3) 1.27 (0.07) 9 (1.3) 1.0 (0.3) malaysia 385 (4.2) 409 (3.8) 427 (4.8) 469 (5.4) 1.99 (0.14) 33 (2.1) 14.9 (1.7) montenegro 371 (2.5) 400 (3.0) 410 (3.2) 447 (3.1) 1.92 (0.13) 32 (1.6) 9.8 (1.0) russian federation 450 (3.9) 472 (4.3) 502 (4.2) 531 (6.0) 1.96 (0.16) 41 (3.1) 12.3 (1.5) Serbia 437 (5.0) 461 (4.1) 476 (4.5) 519 (3.5) 1.90 (0.13) 35 (1.9) 12.8 (1.3) Shanghai-china 492 (6.5) 528 (3.8) 548 (3.6) 578 (5.1) 2.24 (0.17) 35 (2.6) 14.1 (1.9) Singapore 522 (2.6) 552 (2.9) 575 (2.8) 602 (2.5) 2.04 (0.13) 35 (1.3) 11.1 (0.9) chinese taipei 498 (4.9) 528 (4.0) 542 (3.2) 570 (3.8) 1.98 (0.13) 33 (2.3) 9.4 (1.2) united arab Emirates 367 (4.2) 403 (2.9) 432 (3.6) 445 (4.2) 1.90 (0.11) 35 (1.9) 7.7 (0.8) uruguay 358 (4.6) 384 (4.8) 410 (5.2) 463 (5.2) 2.07 (0.17) 36 (1.9) 17.8 (1.6) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. Single-level bivariate regression of performance on the PISA index of economic, social and cultural status (ESCS). The slope of the gradient is the regression coeficient for ESCS; the strength of the relationship is the r-squared. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 195 Annex b1: reSulTS For counTrIeS And economIeS table v.4.13 [Part 1/3] Strength of the relationship between socio-economic status and performance in problem solving, mathematics, reading and science Results based on students’ self-reports Slope of the socio-economic gradient:1 Score-point difference associated with a one-unit increase in EScS Partners OECD Problem solving australia mathematics Score dif. S.E. Score dif. S.E. 36 (1.3) 42 (1.3) reading Score dif. 42 computer-based mathematics Science digital reading S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. (1.3) 43 (1.3) 35 (1.5) 39 (1.4) austria 36 (2.6) 43 (2.2) 42 (2.3) 46 (2.2) 36 (2.5) 44 (3.1) belgium 43 (2.3) 49 (1.7) 47 (1.8) 48 (1.7) 43 (1.9) 41 (2.1) canada 23 (1.7) 31 (1.2) 30 (1.3) 29 (1.4) 26 (1.5) 25 (1.7) chile 30 (1.9) 34 (1.6) 31 (1.5) 32 (1.7) 28 (1.8) 31 (1.9) czech republic 49 (2.8) 51 (2.7) 46 (2.7) 46 (3.1) m m m m denmark 31 (2.3) 39 (1.7) 39 (1.9) 43 (2.2) 32 (1.8) 34 (1.6) (2.4) Estonia 25 (2.0) 29 (1.7) 26 (1.9) 27 (1.9) 28 (1.9) 26 finland 30 (2.2) 33 (1.8) 33 (2.2) 33 (2.1) m m m m france 43 (2.8) 57 (2.2) 58 (2.9) 58 (2.4) 47 (2.1) 50 (2.9) Germany 37 (2.4) 43 (2.0) 37 (2.0) 42 (2.2) 40 (2.3) 33 (2.5) hungary 49 (3.3) 47 (2.8) 42 (2.3) 44 (2.3) 41 (2.8) 52 (3.3) ireland 35 (2.2) 38 (1.8) 39 (1.9) 41 (2.0) 33 (2.0) 32 (1.8) israel 53 (3.0) 51 (2.6) 44 (2.9) 48 (2.9) 46 (2.9) 51 (2.8) italy 23 (2.5) 30 (2.3) 31 (2.5) 30 (2.3) 24 (2.3) 23 (2.5) Japan 27 (3.1) 41 (3.9) 38 (3.9) 36 (3.9) 34 (4.0) 29 (2.7) (2.4) korea 28 (3.0) 42 (3.3) 33 (2.8) 29 (2.6) 40 (3.0) 32 netherlands 38 (3.8) 40 (3.1) 39 (3.2) 43 (3.1) m m m m norway 31 (2.7) 32 (2.4) 33 (2.7) 34 (2.8) 28 (2.4) 34 (2.6) Poland 36 (2.7) 41 (2.4) 36 (2.2) 36 (2.4) 35 (2.4) 40 (2.6) Portugal 30 (1.9) 35 (1.6) 31 (1.8) 32 (1.6) 28 (1.7) 31 (1.9) Slovak republic 49 (3.3) 54 (2.9) 56 (3.3) 56 (2.9) 47 (2.7) 50 (2.7) Slovenia 40 (1.6) 42 (1.5) 40 (1.6) 39 (1.5) 35 (1.3) 39 (1.7) Spain 29 (3.0) 33 (1.7) 31 (1.9) 30 (1.9) 28 (1.8) 31 (2.6) Sweden 29 (2.3) 36 (1.9) 38 (2.5) 38 (2.4) 25 (2.1) 28 (2.2) turkey 28 (1.9) 32 (2.4) 30 (2.1) 24 (1.8) m m m m England (united kingdom) 33 (2.8) 41 (2.8) 41 (2.8) 46 (2.8) m m m m united States 30 (2.0) 35 (1.7) 33 (1.8) 36 (1.8) 31 (2.1) 33 (1.8) oEcd average 35 (0.5) 40 (0.4) 38 (0.4) 39 (0.4) 34 (0.5) 36 (0.5) (2.6) brazil 30 (2.5) 26 (2.7) 23 (2.4) 24 (2.4) 30 (2.7) 28 bulgaria 45 (3.6) 42 (2.7) 53 (2.9) 47 (2.8) m m m m colombia 27 (1.9) 25 (1.7) 28 (1.9) 23 (1.8) 18 (1.7) 29 (2.0) croatia 32 (2.6) 36 (2.6) 34 (2.5) 31 (2.3) m m m m cyprus* 34 (1.6) 38 (1.6) 35 (1.9) 39 (1.7) m m m m hong kong-china 21 (2.9) 27 (2.6) 20 (2.5) 21 (2.3) 19 (2.8) 19 (2.6) 9 (1.3) 17 (1.5) 11 (1.4) 13 (1.8) 13 (1.3) 13 (1.1) malaysia macao-china 33 (2.1) 30 (2.1) 23 (2.2) 25 (1.9) m m m m montenegro 32 (1.6) 33 (1.3) 34 (1.5) 32 (1.4) m m m m russian federation 41 (3.1) 38 (3.2) 43 (3.2) 43 (3.1) 33 (2.5) 37 (2.7) Serbia 35 (1.9) 34 (2.4) 30 (2.3) 29 (2.2) m m m m Shanghai-china 35 (2.6) 41 (2.7) 33 (2.0) 33 (2.1) 39 (2.4) 37 (2.7) Singapore 35 (1.3) 44 (1.4) 43 (1.4) 46 (1.6) 39 (1.4) 34 (1.2) chinese taipei 33 (2.3) 58 (2.5) 42 (2.2) 40 (1.8) 42 (1.9) 38 (2.4) united arab Emirates 35 (1.9) 33 (1.9) 30 (1.9) 33 (2.1) 30 (1.8) 44 (2.5) uruguay 36 (1.9) 37 (1.8) 35 (2.0) 37 (1.9) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. Single-level bivariate regression of performance on the PISA index of economic, social and cultural status (ESCS); the slope is the regression coeficient for ESCS. 2. r-squared from the regression coeficient of performance on the PISA index of economic, social and cultural status (ESCS). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 196 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.13 [Part 2/3] Strength of the relationship between socio-economic status and performance in problem solving, mathematics, reading and science Results based on students’ self-reports Strength of the relationship between performance and EScS:2 Percentage of explained variation in performance OECD Problem solving reading computer-based mathematics Science digital reading % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. 8.5 (0.6) 12.3 (0.8) 12.0 (0.8) 11.9 (0.7) 9.6 (0.8) 10.2 (0.7) austria 10.7 (1.4) 15.8 (1.5) 15.3 (1.6) 18.3 (1.7) 12.2 (1.6) 13.4 (1.5) belgium 14.0 (1.5) 19.6 (1.4) 18.2 (1.4) 19.2 (1.4) 15.8 (1.3) 14.4 (1.4) canada 4.0 (0.6) 9.4 (0.7) 8.1 (0.7) 7.8 (0.7) 6.1 (0.7) 6.0 (0.8) chile 15.8 (1.8) 23.1 (1.9) 20.4 (1.8) 20.2 (1.9) 15.4 (1.9) 17.9 (2.0) czech republic 14.9 (1.5) 16.2 (1.5) 14.8 (1.5) 14.3 (1.7) m m m m denmark 7.9 (1.2) 16.5 (1.4) 15.3 (1.3) 15.7 (1.5) 9.7 (1.1) 11.9 (1.2) Estonia 5.4 (0.8) 8.6 (0.9) 6.8 (1.0) 7.4 (0.9) 7.8 (1.0) 5.2 (0.9) finland 6.5 (0.9) 9.4 (0.9) 7.5 (0.9) 7.9 (0.9) m m m m france 12.7 (1.2) 22.5 (1.3) 18.7 (1.5) 21.5 (1.3) 16.9 (1.8) 17.2 (1.8) Germany 12.7 (1.4) 16.9 (1.4) 15.0 (1.4) 17.1 (1.4) 15.4 (1.4) 9.8 (1.2) hungary 20.5 (2.3) 23.1 (2.3) 20.0 (2.1) 22.4 (2.2) 18.3 (2.1) 19.8 (1.8) ireland 10.2 (1.1) 14.6 (1.2) 15.1 (1.2) 14.5 (1.2) 11.9 (1.3) 10.9 (1.1) israel 13.2 (1.4) 17.2 (1.5) 11.2 (1.4) 14.7 (1.4) 12.6 (1.5) 13.8 (1.5) italy 5.9 (1.2) 9.4 (1.2) 9.3 (1.3) 9.2 (1.3) 7.9 (1.3) 5.6 (1.1) Japan 5.2 (1.1) 9.8 (1.6) 7.9 (1.5) 7.3 (1.4) 7.8 (1.5) 6.9 (1.1) korea 5.4 (1.1) 10.1 (1.4) 7.9 (1.2) 6.7 (1.1) 10.6 (1.3) 8.6 (1.2) netherlands 9.1 (1.6) 11.5 (1.7) 10.8 (1.7) 12.5 (1.8) m m m m norway 5.2 (0.9) 7.4 (1.0) 6.3 (1.0) 6.9 (1.0) 6.0 (1.0) 6.8 (0.9) Poland 11.6 (1.7) 16.6 (1.7) 13.4 (1.6) 14.4 (1.7) 13.8 (1.7) 14.2 (1.7) Portugal 16.1 (2.0) 19.6 (1.8) 16.5 (1.7) 18.7 (1.7) 14.9 (1.8) 17.6 (1.8) Slovak republic 21.3 (2.0) 24.6 (2.1) 24.1 (2.1) 26.4 (2.0) 24.9 (2.1) 23.8 (1.9) Slovenia 12.6 (1.0) 15.6 (1.0) 14.2 (1.1) 14.1 (1.0) 11.9 (0.8) 11.9 (1.0) Spain 7.9 (1.5) 15.7 (1.6) 12.0 (1.5) 13.2 (1.6) 11.8 (1.4) 10.6 (1.6) Sweden 6.2 (1.0) 10.6 (1.1) 9.1 (1.1) 10.4 (1.2) 5.8 (0.9) 5.8 (0.9) turkey 15.1 (1.8) 14.5 (1.8) 14.5 (1.8) 11.0 (1.6) m m m m 7.8 (1.1) 12.4 (1.4) 11.8 (1.3) 13.7 (1.4) m m m m united States 10.1 (1.2) 14.8 (1.3) 12.6 (1.3) 14.2 (1.4) 11.9 (1.5) 13.5 (1.4) oEcd average 10.6 (0.3) 14.9 (0.3) 13.2 (0.3) 14.0 (0.3) 12.1 (0.3) 12.0 (0.3) (2.4) australia England (united kingdom) Partners mathematics brazil 14.6 (2.4) 15.5 (2.9) 10.3 (2.0) 13.2 (2.3) 17.6 (2.9) 12.9 bulgaria 20.0 (2.5) 22.3 (2.3) 21.9 (2.2) 23.8 (2.3) m m m m colombia 12.6 (1.6) 15.4 (1.8) 15.6 (1.9) 12.7 (1.8) 8.3 (1.5) 14.3 (1.8) m croatia 8.6 (1.2) 12.0 (1.4) 11.2 (1.4) 9.8 (1.2) m m m cyprus* 9.5 (0.9) 14.1 (1.1) 8.2 (0.8) 13.7 (1.0) m m m m hong kong-china 4.9 (1.3) 7.5 (1.5) 5.2 (1.2) 6.0 (1.3) 4.5 (1.3) 3.9 (1.1) macao-china 1.0 (0.3) 2.6 (0.4) 1.5 (0.4) 2.1 (0.6) 1.7 (0.4) 2.4 (0.4) 14.9 (1.7) 13.4 (1.6) 7.7 (1.4) 10.3 (1.4) m m m m 9.8 (1.0) 12.7 (0.9) 10.9 (1.0) 11.6 (0.9) m m m m russian federation 12.3 (1.5) 11.4 (1.7) 13.1 (1.6) 14.6 (1.9) 9.9 (1.4) 10.7 (1.4) Serbia 12.8 (1.3) 11.7 (1.4) 8.7 (1.2) 8.8 (1.2) m m m m Shanghai-china 14.1 (1.9) 15.1 (1.9) 15.6 (1.8) 15.3 (2.0) 15.9 (1.9) 17.6 (2.3) Singapore 11.1 (0.9) 14.4 (0.9) 15.2 (0.9) 16.5 (1.0) 13.0 (0.9) 12.2 (0.9) 9.4 (1.2) 17.9 (1.4) 15.1 (1.4) 16.7 (1.4) 15.7 (1.3) 13.0 (1.4) malaysia montenegro chinese taipei united arab Emirates uruguay 7.7 (0.8) 9.8 (1.0) 7.1 (0.9) 8.9 (1.0) 8.8 (1.0) 11.6 (1.1) 17.8 (1.6) 22.8 (1.9) 17.5 (1.8) 19.8 (1.8) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. Single-level bivariate regression of performance on the PISA index of economic, social and cultural status (ESCS); the slope is the regression coeficient for ESCS. 2. r-squared from the regression coeficient of performance on the PISA index of economic, social and cultural status (ESCS). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 197 Annex b1: reSulTS For counTrIeS And economIeS table v.4.13 [Part 3/3] Strength of the relationship between socio-economic status and performance in problem solving, mathematics, reading and science Results based on students’ self-reports Strength of the relationship between performance in problem solving (PS) and EScS,2 compared to… Partners OECD … mathematics (PS - m) … reading (PS - r) … computer-based mathematics (PS - cbm) … Science (PS - S) … digital reading (PS - dr) % dif. S.E. % dif. S.E. % dif. S.E. % dif. S.E. % dif. S.E. australia -3.9 (0.6) -3.5 (0.6) -3.4 (0.5) -1.1 (0.6) -1.7 (0.5) austria -5.1 (1.2) -4.6 (1.2) -7.5 (1.3) -1.5 (1.2) -2.7 (1.4) belgium -5.7 (0.8) -4.2 (1.0) -5.3 (1.0) -1.8 (0.9) -0.4 (1.0) canada -5.4 (0.5) -4.1 (0.5) -3.8 (0.5) -2.0 (0.5) -1.9 (0.6) chile -7.2 (1.4) -4.6 (1.4) -4.3 (1.5) 0.4 (1.9) -2.1 (1.7) czech republic -1.3 (0.7) 0.1 (1.0) 0.6 (0.9) m m m m denmark -8.6 (1.2) -7.4 (1.4) -7.8 (1.2) -1.7 (0.9) -3.9 (1.1) Estonia -3.2 (0.6) -1.4 (0.8) -1.9 (0.8) -2.4 (0.8) 0.2 (0.8) finland -2.9 (0.6) -1.0 (0.7) -1.4 (0.6) m m m m france -9.8 (1.0) -6.0 (1.2) -8.9 (1.0) -4.3 (1.5) -4.6 (1.4) Germany -4.2 (1.0) -2.3 (1.2) -4.4 (1.1) -2.7 (1.3) 2.9 (1.2) hungary -2.5 (1.1) 0.6 (1.2) -1.9 (1.0) 2.2 (1.2) 0.7 (1.5) ireland -4.5 (1.0) -4.9 (1.1) -4.3 (1.0) -1.7 (1.1) -0.7 (1.1) israel -3.9 (0.8) 2.0 (0.8) -1.5 (0.8) 0.7 (0.8) -0.5 (0.9) italy -3.5 (0.9) -3.4 (1.0) -3.3 (1.0) -2.0 (1.2) 0.3 (0.8) Japan -4.6 (1.0) -2.7 (0.8) -2.2 (0.9) -2.7 (0.8) -1.8 (0.6) korea -4.7 (0.7) -2.5 (0.8) -1.4 (0.7) -5.2 (0.9) -3.3 (0.9) netherlands -2.4 (1.0) -1.6 (1.1) -3.4 (1.1) m m m m norway -2.2 (0.7) -1.1 (0.8) -1.6 (0.7) -0.8 (0.6) -1.6 (0.6) Poland -5.1 (1.2) -1.8 (1.3) -2.8 (1.4) -2.2 (1.2) -2.6 (1.1) Portugal -3.6 (1.0) -0.4 (1.2) -2.7 (1.2) 1.1 (1.3) -1.5 (1.4) Slovak republic -3.3 (1.6) -2.8 (1.6) -5.1 (1.7) -3.6 (1.7) -2.5 (1.5) Slovenia -3.0 (0.9) -1.6 (1.1) -1.5 (0.7) 0.7 (0.7) 0.7 (0.8) Spain -7.8 (1.0) -4.1 (1.0) -5.3 (1.0) -3.9 (1.2) -2.7 (1.0) Sweden -4.5 (0.7) -2.9 (0.9) -4.3 (0.9) 0.3 (0.8) 0.4 (0.8) turkey 0.6 (0.8) 0.6 (1.1) 4.1 (0.9) m m m m England (united kingdom) -4.6 (0.9) -4.0 (1.0) -5.8 (0.9) m m m m united States -4.7 (0.9) -2.6 (1.0) -4.2 (1.0) -1.9 (1.1) -3.4 (1.0) oEcd average -4.3 (0.2) -2.6 (0.2) -3.4 (0.2) -1.6 (0.2) -1.4 (0.2) brazil -0.9 (1.4) 4.3 (1.4) 1.4 (1.5) -3.0 (1.6) 1.6 (1.3) bulgaria -2.3 (1.2) -1.9 (1.4) -3.7 (1.4) m m m m colombia -2.8 (1.2) -3.0 (1.5) 0.0 (1.4) 4.4 (1.1) -1.7 (1.3) croatia -3.4 (0.7) -2.6 (0.9) -1.2 (0.8) m m m m cyprus* -4.7 (0.7) 1.3 (0.7) -4.3 (0.7) m m m m hong kong-china -2.6 (0.9) -0.3 (0.9) -1.1 (0.9) 0.4 (1.1) 0.9 (0.9) macao-china -1.6 (0.3) -0.5 (0.3) -1.1 (0.5) -0.7 (0.2) -1.4 (0.4) 1.5 (1.0) 7.2 (1.1) 4.6 (1.1) m m m m -3.0 (0.6) -1.1 (0.9) -1.8 (0.9) m m m m 0.9 (1.4) -0.8 (1.2) -2.3 (1.4) 2.4 (1.0) 1.6 (1.2) malaysia montenegro russian federation Serbia 1.1 (0.8) 4.2 (0.9) 4.0 (1.0) m m m m Shanghai-china -1.0 (0.9) -1.6 (1.0) -1.2 (1.1) -1.9 (1.2) -3.5 (1.2) Singapore -3.3 (0.6) -4.1 (0.7) -5.4 (0.8) -1.8 (0.6) -1.1 (0.6) chinese taipei -8.5 (0.6) -5.6 (0.7) -7.3 (0.7) -6.3 (0.7) -3.6 (0.8) united arab Emirates -2.1 (0.7) 0.6 (0.6) -1.1 (0.8) -1.1 (0.6) -3.9 (0.7) uruguay -5.0 (1.6) 0.4 (1.7) -2.0 (1.6) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. Single-level bivariate regression of performance on the PISA index of economic, social and cultural status (ESCS); the slope is the regression coeficient for ESCS. 2. r-squared from the regression coeficient of performance on the PISA index of economic, social and cultural status (ESCS). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 198 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.14 [Part 1/1] Strength of the relationship between socio-economic status and performance in problem solving, between and within schools1 Results based on students’ self-reports variation components expressed as a percentage of total variation in student performance in problem solving2 variation in problem solving variation in problem solving accounted for by the socio-economic status of students and schools3 variation unique to problem solving4 variation unique to problem solving accounted for by the socio-economic status of students and schools5 OECD between schools Within schools between schools Within schools between schools Within schools between schools Within schools % % % % % % % australia 27.1 73.3 10.9 2.4 12.1 18.1 0.3 0.6 austria 47.5 51.2 18.8 1.2 13.4 20.9 0.0 0.4 belgium 47.8 51.3 24.3 1.7 12.4 21.7 0.0 0.2 canada 22.6 76.4 5.3 3.6 14.8 27.6 0.4 1.9 chile 42.7 55.9 23.6 0.1 12.9 22.7 0.1 0.0 czech republic 48.2 49.4 31.8 1.1 7.6 14.6 0.3 0.2 denmark 28.6 71.0 6.0 4.3 20.3 20.4 0.0 0.8 Estonia 23.8 76.6 8.0 1.4 12.2 18.8 0.2 0.5 finland 10.2 89.5 1.9 5.5 7.2 22.7 0.1 0.7 w w w w w w w w Germany 54.9 44.7 31.6 0.0 14.4 15.7 1.6 0.0 hungary 59.1 38.9 41.4 0.8 10.1 19.9 0.3 1.1 ireland 24.4 74.8 10.0 4.7 13.0 23.2 0.1 0.5 israel 50.9 48.8 25.9 1.0 9.0 16.5 0.1 0.8 italy 42.1 54.7 13.9 0.0 14.4 27.9 0.1 0.5 Japan 33.9 65.8 17.6 0.1 7.2 35.5 0.0 0.4 korea 31.3 67.1 13.1 0.5 7.7 27.4 0.0 0.2 netherlands 57.7 42.4 27.8 0.5 12.3 16.4 0.0 0.1 norway 21.4 78.0 4.6 3.1 16.1 20.5 0.5 0.0 Poland 36.1 63.7 10.3 4.8 20.6 22.4 0.2 0.7 Portugal 30.0 70.3 14.9 4.7 11.2 23.9 0.7 0.5 Slovak republic 49.6 48.2 31.2 2.0 11.4 15.0 0.2 0.1 Slovenia 54.2 45.3 30.5 0.5 14.3 18.7 0.0 0.2 Spain 28.7 71.4 5.9 3.0 19.2 25.2 0.3 0.6 Sweden 18.6 80.7 2.6 4.4 11.9 22.6 0.0 0.8 turkey 51.9 48.0 30.8 1.0 8.6 21.3 0.7 0.4 England (united kingdom) 29.3 70.7 12.9 2.4 10.0 16.4 0.2 0.4 united States 28.9 70.9 10.2 3.0 11.6 14.8 0.1 0.4 oEcd average 37.8 61.5 17.6 2.1 12.6 20.9 0.2 0.5 brazil 47.4 52.7 21.5 1.2 14.8 16.7 0.3 0.0 bulgaria 55.5 44.0 36.2 0.9 12.6 21.2 0.9 0.1 colombia 36.8 62.7 15.8 2.8 15.4 29.5 0.3 1.7 croatia 40.4 59.5 20.6 0.4 8.6 19.5 0.1 0.5 cyprus* 35.3 67.9 17.1 1.7 10.3 25.8 0.1 0.6 hong kong-china 36.1 63.7 12.1 0.0 10.5 32.4 0.2 0.0 macao-china 17.2 80.4 2.2 0.2 2.4 33.3 0.0 1.4 malaysia 37.4 62.5 20.4 2.9 10.0 20.2 0.7 0.6 montenegro 38.3 61.7 27.1 0.7 6.5 27.0 0.1 0.1 russian federation 37.0 63.1 15.2 3.0 21.1 23.9 2.7 0.3 Serbia 37.0 62.3 24.2 1.7 8.4 22.3 0.9 0.6 Shanghai-china 41.2 58.4 26.9 0.7 9.5 20.1 1.1 0.1 Singapore 33.9 66.1 16.2 2.3 11.4 19.2 0.2 0.0 chinese taipei 38.9 60.6 23.0 0.7 6.8 18.6 0.0 0.5 united arab Emirates 50.4 49.4 18.2 1.1 14.5 21.1 0.3 0.4 uruguay 42.3 57.6 23.8 1.8 14.2 22.7 0.4 0.1 Partners france % 1. In some countries/economies, sub-units within schools were sampled instead of schools; this may affect the estimation of between-school variance components (see Annex A3). 2. Due to the unbalanced clustered nature of the data, the sum of the between- and within-school variation components, as an estimate from a sample, does not necessarily add up to the total. All models were estimated on samples excluding students with missing information on the PISA index of economic, social and cultural status (ESCS). 3. Based on the residual variation in a model with student ESCS and school average ESCS. Negative estimates of explained variance values are reported as 0.0. 4. Based on the residual variation in a model with student performance in mathematics and school average performance in mathematics. 5. Based on the residual variation in a model with student performance in mathematics, student ESCS, school average performance in mathematics, and school average ESCS. Negative estimates of explained variance values are reported as 0.0. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 199 Annex b1: reSulTS For counTrIeS And economIeS table v.4.15 [Part 1/1] Performance in problem solving and parents’ highest occupational status Results based on students’ self-reports Skilled (iSco 1 to 3) missing data Semi-skilled or on father’s and mother’s elementary occupation (iSco 4 to 9) increased likelihood of students with at least one difference in problem- parent working in a skilled solving occupation performance: scoring below Skilled level 2 minus (less than semi-skilled 423.42 or elementary score points) occupations mean score mean score S.E. Score dif. S.E. (4.8) 40 (2.1) Percentage of students by parents’ highest occupation Performance in problem solving by parents’ highest occupation Partners OECD missing data Semi-skilled or on father’s elementary and mother’s Skilled (iSco 1 to 3) (iSco 4 to 9) occupation australia % S.E. % S.E. % S.E. 64.5 (0.6) 30.7 (0.5) 4.8 (0.2) 539 S.E. (2.0) 499 S.E. (2.5) mean score 462 relative risk S.E. 0.53 (0.03) increased likelihood of students with at least one parent working in a skilled occupation scoring at level 5 or above (above 618.21 score points) relative risk S.E. 1.92 (0.12) austria 48.7 (1.0) 47.2 (1.0) 4.1 (0.4) 532 (4.1) 482 (4.0) 488 (9.6) 50 (4.0) 0.48 (0.05) 2.60 (0.34) belgium 53.1 (0.9) 41.3 (0.9) 5.6 (0.4) 537 (2.7) 479 (3.3) 438 (9.3) 58 (3.6) 0.42 (0.03) 2.33 (0.23) canada 60.6 (0.7) 32.5 (0.6) 6.9 (0.3) 541 (2.5) 508 (2.6) 478 (8.5) 32 (2.4) 0.59 (0.03) 1.66 (0.10) chile 33.1 (1.2) 60.9 (1.1) 5.9 (0.4) 481 (4.2) 432 (3.9) 421 (7.7) 49 (4.5) 0.53 (0.04) 5.39 (2.38) czech republic 43.6 (1.0) 52.2 (1.0) 4.3 (0.4) 542 (3.0) 486 (3.8) 446 (14.5) 56 (3.6) 0.37 (0.04) 2.83 (0.32) denmark 58.6 (1.3) 37.4 (1.1) 4.0 (0.4) 516 (2.9) 475 (3.6) 431 (13.8) 40 (3.8) 0.52 (0.04) 2.32 (0.37) Estonia 54.2 (0.9) 42.9 (0.8) 2.9 (0.3) 531 (2.8) 497 (3.2) 471 (9.1) 34 (3.3) 0.53 (0.05) 1.95 (0.26) finland 64.1 (0.8) 33.5 (0.8) 2.4 (0.2) 536 (2.4) 503 (3.3) 457 (9.4) 33 (3.5) 0.55 (0.05) 1.88 (0.21) france 55.0 (1.0) 38.9 (1.0) 6.1 (0.4) 535 (3.4) 488 (4.5) 442 (8.9) 47 (4.1) 0.44 (0.04) 2.42 (0.27) Germany 43.0 (0.9) 37.5 (1.0) 19.5 (0.9) 542 (3.7) 488 (4.2) 476 (7.8) 54 (4.3) 0.40 (0.04) 2.47 (0.28) hungary 40.9 (1.2) 51.8 (1.2) 7.3 (0.6) 502 (4.8) 433 (4.6) 403 (10.5) 68 (6.0) 0.44 (0.04) 4.44 (0.81) ireland 55.7 (0.9) 40.8 (0.9) 3.5 (0.3) 520 (3.4) 476 (3.6) 416 44 (3.3) 0.52 (0.04) 2.64 (0.38) (8.2) israel 63.3 (1.5) 26.5 (1.1) 10.2 (0.9) 485 (6.0) 407 (5.9) 387 (10.2) 78 (6.8) 0.52 (0.04) 4.49 (0.93) italy 40.7 (1.3) 54.8 (1.3) 4.4 (0.6) 533 (4.7) 496 (4.4) 462 (9.3) 37 (4.3) 0.48 (0.06) 1.83 (0.24) (0.10) Japan 45.6 (0.7) 44.7 (0.8) 9.7 (0.6) 565 (3.5) 545 (3.5) 522 (5.7) 20 (3.5) 0.57 (0.08) 1.34 korea 55.7 (1.2) 42.5 (1.2) 1.8 (0.2) 572 (4.4) 548 (4.4) 514 (13.9) 24 (3.2) 0.66 (0.08) 1.41 (0.09) netherlands 66.0 (1.1) 29.0 (1.0) 5.0 (0.5) 530 (4.3) 481 (6.2) 422 (11.4) 49 (5.6) 0.48 (0.06) 2.72 (0.53) norway 68.0 (0.8) 27.1 (0.8) 4.9 (0.4) 517 (3.4) 479 (4.1) 450 (9.4) 37 (4.0) 0.63 (0.04) 1.87 (0.22) Poland 42.5 (1.4) 53.7 (1.3) 3.8 (0.3) 512 (4.9) 458 (4.6) 452 (9.7) 54 (4.5) 0.45 (0.05) 3.43 (0.67) Portugal 34.3 (1.7) 61.0 (1.6) 4.6 (0.5) 529 (4.1) 478 (3.6) 457 (7.9) 51 (4.3) 0.43 (0.05) 2.83 (0.41) Slovak republic 32.8 (1.2) 59.3 (1.1) 7.9 (0.7) 532 (4.4) 468 (3.6) 396 (8.5) 63 (5.2) 0.33 (0.03) 3.37 (0.62) Slovenia 53.9 (0.8) 42.4 (0.8) 3.7 (0.3) 501 (2.1) 450 (2.3) 413 (9.2) 51 (3.1) 0.52 (0.03) 3.24 (0.80) Spain 42.2 (1.3) 55.9 (1.3) 1.8 (0.3) 503 (4.6) 458 (4.5) 437 (12.9) 45 (4.6) 0.54 (0.04) 2.07 (0.30) Sweden 60.7 (0.9) 34.3 (0.8) 5.0 (0.5) 510 (3.3) 468 (3.2) 416 (10.7) 42 (3.4) 0.57 (0.04) 3.02 (0.45) turkey 18.6 (0.9) 69.2 (1.0) 12.2 (0.7) 488 (6.4) 448 (3.6) 438 40 (5.0) 0.62 (0.06) 4.31 (1.39) (6.0) England (united kingdom) 61.8 (1.4) 31.8 (1.1) 6.4 (0.6) 536 (3.8) 496 (4.8) 432 (10.8) 40 (4.6) 0.55 (0.06) 2.22 (0.32) united States 60.9 (1.4) 33.4 (1.2) 5.7 (0.5) 526 (3.8) 484 (4.2) 457 (8.2) 42 (3.9) 0.52 (0.05) 2.69 (0.34) oEcd average 50.8 (0.2) 43.3 (0.2) 5.9 (0.1) 525 (0.7) 479 (0.8) 446 (1.8) 46 (0.8) 0.51 (0.01) 2.70 (0.13) brazil 32.9 (1.4) 58.9 (1.4) 8.3 (0.6) 462 (5.5) 416 (5.4) 380 (7.2) 46 (5.9) 0.61 (0.05) 4.69 (2.02) bulgaria 41.1 (1.4) 49.2 (1.2) 9.7 (0.7) 448 (5.3) 378 (5.3) 328 (11.5) 70 (6.5) 0.58 (0.03) 9.42 (5.55) colombia 23.0 (1.0) 70.9 (0.9) 6.1 (0.5) 435 (5.6) 389 (3.5) 383 (6.8) 47 (5.0) 0.68 (0.04) 3.37 (1.32) croatia 39.2 (1.0) 56.0 (1.0) 4.8 (0.3) 498 (4.6) 448 (4.0) 428 (8.4) 50 (4.6) 0.52 (0.04) 3.43 (0.61) cyprus* 40.0 (0.8) 53.6 (0.8) 6.4 (0.4) 477 (2.2) 427 (2.1) 392 (5.0) 49 (3.1) 0.60 (0.03) 4.07 (0.92) hong kong-china 39.5 (1.9) 52.5 (1.8) 7.9 (0.6) 559 (5.0) 532 (4.0) 492 (6.7) 27 (5.3) 0.61 (0.09) 1.58 (0.16) 1.19 (0.11) macao-china 27.1 (0.6) 70.3 (0.6) 2.6 (0.2) 551 (2.2) 538 (1.2) 496 (9.3) 13 (2.6) 0.68 (0.09) malaysia 37.5 (1.3) 56.3 (1.3) 6.2 (0.5) 455 (4.6) 405 (3.1) 381 (7.2) 50 (4.3) 0.60 (0.04) montenegro 37.9 (0.7) 45.9 (0.8) 16.2 (0.6) 441 (2.3) 394 (1.8) 362 (3.6) 48 (3.2) 0.64 (0.03) 4.23 (3.01) russian federation 53.7 (1.1) 42.2 (1.1) 4.1 (0.4) 512 (3.9) 462 (3.3) 477 (8.4) 50 (3.4) 0.47 (0.04) 4.01 (0.68) 14.21 (12.86) Serbia 40.5 (1.1) 55.9 (1.1) 3.6 (0.3) 507 (2.9) 451 (3.5) 449 (10.0) 56 (3.7) 0.44 (0.03) 4.43 (0.88) Shanghai-china 56.5 (1.3) 41.9 (1.3) 1.6 (0.2) 555 (3.3) 514 (4.0) 461 (13.7) 41 (4.1) 0.46 (0.05) 2.09 (0.22) Singapore 67.5 (0.6) 29.8 (0.6) 2.7 (0.2) 579 (1.6) 532 (2.5) 477 (8.1) 47 (3.2) 0.43 (0.06) 1.93 (0.16) chinese taipei 41.6 (1.2) 53.4 (1.1) 4.9 (0.3) 561 (2.9) 521 (3.1) 448 (8.5) 40 (3.4) 0.39 (0.05) 1.82 (0.13) united arab Emirates 70.0 (0.8) 15.0 (0.5) 15.0 (0.6) 432 (2.6) 369 (4.3) 355 (5.0) 63 (3.7) 0.65 (0.02) 5.56 (2.45) uruguay 26.2 (0.9) 68.8 (0.9) 5.0 (0.3) 460 (4.4) 386 (3.6) 349 (7.5) 74 (5.0) 0.52 (0.03) 8.06 (3.05) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. Increased likelihood relative to students with parents in semi-skilled or elementary occupations. Students who did not report their parents’ occupation are excluded from this calculation. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 200 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.16 [Part 1/3] differences in problem-solving, mathematics, reading and science performance related to parents’ occupational status Results based on students’ self-reports difference in performance related to parents’ highest occupation: Skilled (iSco 1 to 3) minus semi-skilled or elementary (iSco 4 to 9) Partners OECD Problem solving mathematics Score dif. S.E. Score dif. S.E. australia 40 (2.1) 46 (2.2) austria 50 (4.0) 54 belgium 58 (3.6) 70 canada 32 (2.4) chile 49 czech republic reading Score dif. computer-based mathematics Science digital reading S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. 45 (2.3) 48 (2.3) 38 (2.2) 43 (2.2) (3.6) 55 (3.6) 58 (3.6) 47 (4.0) 54 (4.8) (3.3) 69 (3.2) 68 (3.1) 60 (3.4) 61 (3.6) 41 (2.0) 37 (2.2) 36 (2.1) 33 (2.2) 29 (2.6) (4.5) 59 (4.6) 53 (4.2) 55 (4.4) 51 (4.8) 52 (4.5) 56 (3.6) 61 (3.8) 54 (3.4) 55 (3.5) m m m m denmark 40 (3.8) 48 (3.1) 49 (3.2) 52 (3.7) 40 (3.4) 46 (3.2) Estonia 34 (3.3) 40 (3.0) 40 (3.3) 40 (3.2) 38 (3.3) 40 (3.8) finland 33 (3.5) 37 (2.8) 37 (3.3) 38 (3.1) m m m m france 47 (4.1) 67 (3.5) 71 (4.6) 66 (3.7) 53 (3.0) 59 (4.0) Germany 54 (4.3) 62 (4.3) 58 (4.1) 60 (4.5) 57 (4.1) 51 (4.5) hungary 68 (6.0) 65 (5.2) 60 (4.5) 62 (4.2) 60 (5.1) 73 (6.0) ireland 44 (3.3) 43 (2.9) 46 (3.3) 47 (3.0) 35 (3.2) 33 (3.3) israel 78 (6.8) 73 (5.9) 66 (6.5) 70 (6.2) 62 (6.4) 75 (6.6) italy 37 (4.3) 42 (4.3) 47 (4.6) 48 (4.4) 37 (4.2) 40 (4.7) Japan 20 (3.5) 32 (3.9) 30 (4.0) 28 (3.9) 25 (4.0) 22 (2.8) korea 24 (3.2) 34 (3.8) 26 (3.0) 24 (3.1) 35 (3.5) 30 (3.0) netherlands 49 (5.6) 52 (4.2) 53 (4.4) 54 (4.7) m m m m norway 37 (4.0) 37 (3.7) 38 (4.1) 39 (4.1) 34 (3.3) 40 (3.8) Poland 54 (4.5) 58 (4.6) 54 (3.9) 53 (4.3) 51 (4.3) 60 (4.4) Portugal 51 (4.3) 64 (4.2) 58 (4.6) 58 (4.3) 48 (4.3) 59 (4.7) Slovak republic 63 (5.2) 71 (5.1) 71 (5.3) 71 (5.2) 58 (4.6) 62 (4.7) Slovenia 51 (3.1) 53 (3.2) 54 (3.2) 53 (2.9) 46 (2.9) 54 (3.2) Spain 45 (4.6) 54 (3.2) 50 (3.3) 47 (3.3) 45 (3.7) 50 (4.1) Sweden 42 (3.4) 50 (3.3) 52 (3.9) 53 (3.9) 34 (3.5) 43 (3.6) turkey 40 (5.0) 51 (6.1) 50 (5.6) 40 (4.9) m m m m England (united kingdom) 40 (4.6) 49 (4.3) 51 (4.4) 55 (4.3) m m m m united States 42 (3.9) 50 (3.1) 49 (3.2) 51 (3.1) 45 (3.4) 49 (3.1) oEcd average 46 (0.8) 52 (0.7) 51 (0.8) 51 (0.7) 45 (0.8) 49 (0.9) (6.7) brazil 46 (5.9) 47 (6.6) 39 (6.1) 44 (5.9) 49 (6.3) 41 bulgaria 70 (6.5) 71 (5.1) 86 (6.2) 76 (5.5) m m m m colombia 47 (5.0) 44 (4.3) 50 (4.4) 43 (4.0) 35 (4.4) 53 (5.2) croatia 50 (4.6) 56 (4.8) 52 (4.6) 49 (4.2) m m m m cyprus* 49 (3.1) 58 (2.8) 51 (3.4) 60 (3.2) m m m m hong kong-china 27 (5.3) 36 (4.9) 24 (4.3) 27 (4.2) 24 (4.5) 25 (4.2) macao-china 13 (2.6) 22 (2.9) 14 (2.7) 19 (3.1) 16 (2.7) 16 (2.2) malaysia 50 (4.3) 46 (4.1) 37 (4.0) 38 (3.8) m m m m montenegro 48 (3.2) 48 (2.9) 51 (3.1) 48 (2.8) m m m m (3.5) russian federation 50 (3.4) 46 (4.2) 52 (4.3) 50 (4.2) 38 (3.4) 39 Serbia 56 (3.7) 58 (4.5) 53 (4.3) 50 (4.1) m m m m Shanghai-china 41 (4.1) 49 (4.5) 40 (3.5) 39 (3.8) 44 (3.9) 44 (4.5) Singapore 47 (3.2) 60 (3.3) 57 (3.3) 62 (3.5) 53 (3.3) 45 (3.1) chinese taipei 40 (3.4) 71 (4.3) 50 (3.6) 48 (3.0) 48 (3.0) 46 (3.5) united arab Emirates 63 (3.7) 53 (3.1) 49 (3.4) 51 (3.4) 48 (2.9) 69 (4.4) uruguay 74 (5.0) 76 (5.0) 73 (5.2) 75 (5.1) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 201 Annex b1: reSulTS For counTrIeS And economIeS table v.4.16 [Part 2/3] differences in problem-solving, mathematics, reading and science performance related to parents’ occupational status Results based on students’ self-reports occupational status effect size: difference in performance related to parents’ highest occupation divided by the variation in scores within each country/economy (standard deviation) Partners OECD Problem solving mathematics reading computer-based mathematics Science Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. australia 0.42 (0.02) 0.49 (0.02) 0.48 (0.02) 0.49 (0.02) 0.43 (0.02) 0.46 (0.02) austria 0.53 (0.04) 0.59 (0.04) 0.61 (0.03) 0.64 (0.04) 0.53 (0.04) 0.54 (0.05) belgium 0.55 (0.03) 0.70 (0.03) 0.70 (0.03) 0.70 (0.03) 0.62 (0.03) 0.62 (0.03) canada 0.33 (0.02) 0.48 (0.02) 0.42 (0.02) 0.41 (0.02) 0.37 (0.02) 0.34 (0.03) chile 0.58 (0.05) 0.73 (0.05) 0.69 (0.05) 0.69 (0.05) 0.63 (0.05) 0.64 (0.05) czech republic 0.60 (0.03) 0.65 (0.03) 0.63 (0.03) 0.62 (0.03) m m m m denmark 0.44 (0.04) 0.60 (0.04) 0.59 (0.04) 0.57 (0.04) 0.47 (0.04) 0.57 (0.04) (0.04) Estonia 0.39 (0.04) 0.50 (0.03) 0.50 (0.04) 0.51 (0.04) 0.46 (0.04) 0.44 finland 0.36 (0.04) 0.44 (0.03) 0.40 (0.03) 0.42 (0.03) m m m m france 0.50 (0.04) 0.70 (0.03) 0.66 (0.04) 0.67 (0.03) 0.59 (0.04) 0.62 (0.04) Germany 0.56 (0.04) 0.65 (0.04) 0.65 (0.04) 0.63 (0.04) 0.60 (0.04) 0.53 (0.04) hungary 0.67 (0.05) 0.71 (0.04) 0.68 (0.04) 0.71 (0.04) 0.66 (0.04) 0.67 (0.04) ireland 0.48 (0.03) 0.51 (0.03) 0.54 (0.03) 0.53 (0.03) 0.44 (0.04) 0.41 (0.04) israel 0.64 (0.05) 0.72 (0.05) 0.60 (0.06) 0.67 (0.05) 0.56 (0.06) 0.67 (0.06) italy 0.41 (0.05) 0.46 (0.04) 0.50 (0.04) 0.51 (0.04) 0.45 (0.04) 0.42 (0.04) Japan 0.24 (0.04) 0.35 (0.04) 0.32 (0.04) 0.30 (0.04) 0.29 (0.04) 0.29 (0.03) (0.03) korea 0.27 (0.03) 0.35 (0.04) 0.30 (0.03) 0.29 (0.04) 0.38 (0.03) 0.37 netherlands 0.51 (0.05) 0.58 (0.04) 0.59 (0.04) 0.59 (0.05) m m m m norway 0.37 (0.04) 0.41 (0.04) 0.40 (0.04) 0.41 (0.04) 0.40 (0.04) 0.42 (0.04) Poland 0.56 (0.04) 0.65 (0.04) 0.62 (0.04) 0.62 (0.04) 0.59 (0.04) 0.62 (0.04) Portugal 0.58 (0.05) 0.69 (0.04) 0.63 (0.04) 0.67 (0.04) 0.57 (0.05) 0.66 (0.04) Slovak republic 0.67 (0.04) 0.72 (0.04) 0.72 (0.04) 0.74 (0.04) 0.71 (0.04) 0.70 (0.04) Slovenia 0.53 (0.03) 0.58 (0.03) 0.59 (0.03) 0.59 (0.03) 0.53 (0.03) 0.55 (0.03) Spain 0.43 (0.04) 0.62 (0.04) 0.55 (0.03) 0.56 (0.04) 0.55 (0.04) 0.52 (0.04) Sweden 0.45 (0.04) 0.56 (0.03) 0.51 (0.04) 0.55 (0.04) 0.41 (0.04) 0.45 (0.04) turkey 0.50 (0.06) 0.56 (0.06) 0.59 (0.06) 0.51 (0.06) m m m m England (united kingdom) 0.43 (0.05) 0.53 (0.04) 0.54 (0.04) 0.57 (0.04) m m m m united States 0.46 (0.04) 0.56 (0.03) 0.54 (0.03) 0.55 (0.03) 0.52 (0.04) 0.56 (0.03) oEcd average 0.48 (0.01) 0.57 (0.01) 0.56 (0.01) 0.56 (0.01) 0.51 (0.01) 0.52 (0.01) (0.07) brazil 0.51 (0.06) 0.60 (0.07) 0.47 (0.06) 0.56 (0.06) 0.59 (0.07) 0.46 bulgaria 0.68 (0.05) 0.77 (0.04) 0.76 (0.04) 0.77 (0.04) m m m m colombia 0.51 (0.05) 0.59 (0.05) 0.60 (0.05) 0.56 (0.05) 0.47 (0.06) 0.58 (0.05) croatia 0.55 (0.04) 0.64 (0.04) 0.62 (0.04) 0.57 (0.04) m m m m cyprus* 0.51 (0.03) 0.63 (0.03) 0.48 (0.03) 0.64 (0.03) m m m m hong kong-china 0.29 (0.06) 0.38 (0.05) 0.29 (0.05) 0.33 (0.05) 0.29 (0.05) 0.27 (0.04) macao-china 0.17 (0.03) 0.24 (0.03) 0.18 (0.03) 0.24 (0.04) 0.19 (0.03) 0.23 (0.03) malaysia 0.60 (0.04) 0.57 (0.04) 0.45 (0.04) 0.49 (0.04) m m m m montenegro 0.53 (0.04) 0.59 (0.03) 0.56 (0.03) 0.58 (0.03) m m m m russian federation 0.57 (0.03) 0.53 (0.05) 0.58 (0.04) 0.59 (0.05) 0.48 (0.04) 0.46 (0.04) Serbia 0.63 (0.03) 0.64 (0.04) 0.58 (0.04) 0.58 (0.04) m m m m Shanghai-china 0.46 (0.04) 0.49 (0.04) 0.51 (0.04) 0.48 (0.04) 0.47 (0.04) 0.53 (0.04) Singapore 0.50 (0.03) 0.58 (0.03) 0.57 (0.03) 0.60 (0.03) 0.55 (0.03) 0.51 (0.03) chinese taipei 0.46 (0.03) 0.63 (0.03) 0.57 (0.03) 0.60 (0.03) 0.56 (0.03) 0.54 (0.03) united arab Emirates 0.61 (0.03) 0.60 (0.03) 0.53 (0.03) 0.55 (0.03) 0.58 (0.03) 0.64 (0.04) uruguay 0.76 (0.04) 0.86 (0.04) 0.77 (0.04) 0.80 (0.04) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 202 digital reading © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.16 [Part 3/3] differences in problem-solving, mathematics, reading and science performance related to parents’ occupational status Results based on students’ self-reports difference in occupational status effect sizes between problem solving (PS) and… Partners OECD … mathematics (PS - m) … reading (PS - r) … computer-based mathematics (PS - cbm) … Science (PS - S) … digital reading (PS - dr) Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. australia -0.07 (0.02) -0.06 (0.02) -0.08 (0.02) -0.01 (0.02) -0.04 (0.02) austria -0.06 (0.03) -0.07 (0.03) -0.10 (0.03) 0.00 (0.03) -0.01 (0.05) belgium -0.15 (0.02) -0.15 (0.02) -0.14 (0.02) -0.07 (0.02) -0.07 (0.02) canada -0.15 (0.02) -0.09 (0.02) -0.08 (0.02) -0.04 (0.02) -0.01 (0.02) chile -0.16 (0.02) -0.11 (0.03) -0.12 (0.03) -0.05 (0.04) -0.06 (0.03) czech republic -0.05 (0.02) -0.03 (0.03) -0.02 (0.02) m m m m denmark -0.15 (0.03) -0.15 (0.04) -0.13 (0.03) -0.02 (0.03) -0.12 (0.04) Estonia -0.11 (0.02) -0.11 (0.03) -0.12 (0.03) -0.07 (0.03) -0.05 (0.03) finland -0.09 (0.02) -0.04 (0.03) -0.06 (0.02) m m m m france -0.20 (0.03) -0.17 (0.03) -0.17 (0.03) -0.10 (0.03) -0.12 (0.03) Germany -0.09 (0.02) -0.09 (0.03) -0.07 (0.02) -0.04 (0.03) 0.03 (0.03) hungary -0.04 (0.03) -0.01 (0.03) -0.04 (0.03) 0.01 (0.02) 0.00 (0.03) ireland -0.03 (0.02) -0.06 (0.03) -0.05 (0.02) 0.04 (0.03) 0.07 (0.03) israel -0.08 (0.02) 0.04 (0.03) -0.03 (0.02) 0.08 (0.02) -0.02 (0.03) italy -0.05 (0.03) -0.09 (0.03) -0.10 (0.03) -0.04 (0.04) -0.02 (0.03) Japan -0.11 (0.03) -0.08 (0.03) -0.06 (0.03) -0.04 (0.03) -0.05 (0.03) korea -0.08 (0.02) -0.04 (0.02) -0.02 (0.02) -0.12 (0.03) -0.10 (0.03) netherlands -0.07 (0.03) -0.07 (0.03) -0.08 (0.03) m m m m norway -0.04 (0.03) -0.03 (0.03) -0.04 (0.03) -0.03 (0.03) -0.05 (0.03) Poland -0.08 (0.03) -0.06 (0.03) -0.05 (0.04) -0.03 (0.03) -0.06 (0.03) Portugal -0.10 (0.02) -0.05 (0.03) -0.09 (0.03) 0.01 (0.03) -0.08 (0.03) Slovak republic -0.05 (0.03) -0.04 (0.02) -0.07 (0.02) -0.04 (0.03) -0.02 (0.03) Slovenia -0.05 (0.02) -0.06 (0.03) -0.06 (0.02) 0.00 (0.02) -0.02 (0.02) Spain -0.19 (0.03) -0.12 (0.03) -0.12 (0.03) -0.12 (0.03) -0.08 (0.03) Sweden -0.11 (0.03) -0.07 (0.03) -0.11 (0.03) 0.04 (0.03) 0.00 (0.03) turkey -0.06 (0.03) -0.09 (0.04) 0.00 (0.03) m m m m England (united kingdom) -0.10 (0.03) -0.12 (0.03) -0.15 (0.03) m m m m united States -0.10 (0.03) -0.08 (0.03) -0.09 (0.03) -0.06 (0.03) -0.10 (0.03) oEcd average -0.09 (0.00) -0.07 (0.01) -0.08 (0.01) -0.03 (0.01) -0.04 (0.01) brazil -0.09 (0.03) 0.04 (0.04) -0.05 (0.03) -0.08 (0.04) 0.05 (0.03) bulgaria -0.09 (0.02) -0.07 (0.03) -0.08 (0.03) m m m m colombia -0.08 (0.03) -0.09 (0.04) -0.05 (0.04) 0.04 (0.04) -0.07 (0.04) m croatia -0.09 (0.02) -0.07 (0.03) -0.03 (0.03) m m m cyprus* -0.13 (0.03) 0.03 (0.03) -0.14 (0.03) m m m m hong kong-china -0.08 (0.03) 0.00 (0.04) -0.03 (0.04) 0.01 (0.04) 0.03 (0.04) macao-china -0.08 (0.03) -0.01 (0.04) -0.07 (0.04) -0.02 (0.03) -0.06 (0.03) 0.03 (0.03) 0.15 (0.03) 0.11 (0.03) m m m m -0.06 (0.03) -0.04 (0.03) -0.05 (0.03) m m m m 0.04 (0.03) 0.00 (0.03) -0.02 (0.03) 0.09 (0.03) 0.11 (0.03) Serbia -0.01 (0.02) 0.06 (0.02) 0.06 (0.03) m m m m Shanghai-china -0.03 (0.02) -0.06 (0.02) -0.02 (0.03) -0.02 (0.03) -0.08 (0.03) Singapore -0.07 (0.02) -0.06 (0.02) -0.10 (0.02) -0.05 (0.02) 0.00 (0.02) chinese taipei -0.17 (0.02) -0.11 (0.02) -0.14 (0.02) -0.10 (0.02) -0.08 (0.03) 0.01 (0.03) 0.08 (0.03) 0.06 (0.03) 0.03 (0.03) -0.03 (0.03) -0.10 (0.03) -0.01 (0.04) -0.04 (0.04) m m m m malaysia montenegro russian federation united arab Emirates uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 203 Annex b1: reSulTS For counTrIeS And economIeS table v.4.17 [Part 1/1] relative performance in problem solving, by parents’ occupational status Results based on students’ self-reports Problem-solving performance of students whose parents’ highest occupation is semi-skilled or elementary (iSco 4 to 9), compared to students with similar performance in mathematics, reading and science with at least one parent working in a skilled occupation (iSco 1 to 3) OECD Percentage of average students from difference in low-status problem solving families who compared outperform with students students from from highhigh-status status families families with similar with similar performance performance in mathematics1 in mathematics2 Score dif. S.E. % S.E. Score dif. S.E. % S.E. Score dif. S.E. % S.E. Score dif. S.E. % S.E. australia -1 (1.5) 50.4 (1.3) -5 (1.7) 47.7 (1.3) -2 (1.5) 49.5 (1.2) 1 (1.5) 51.5 (1.3) austria -4 (3.1) 46.7 (2.6) -6 (3.1) 46.0 (2.3) -2 (3.9) 48.6 (3.1) -1 (3.3) 49.5 (3.1) belgium 0 (2.4) 50.3 (1.7) -4 (2.6) 48.3 (1.7) -1 (2.3) 49.8 (1.5) 3 (2.2) 51.9 (1.7) canada 3 (1.7) 53.0 (1.2) -3 (1.9) 48.4 (1.5) -2 (1.8) 49.0 (1.3) 4 (1.6) 53.2 (1.3) chile 0 (2.6) 51.4 (2.0) -5 (3.0) 46.9 (2.1) -5 (3.0) 47.6 (2.2) 3 (2.7) 53.1 (2.2) czech republic denmark -3 (2.7) 48.2 (2.7) -12 (2.9) 42.2 (2.0) -8 (3.1) 44.4 (2.3) -2 (2.5) 49.2 (2.5) 1 (2.7) 51.0 (2.3) -3 (3.0) 48.2 (2.1) -2 (2.8) 47.6 (2.2) 2 (2.6) 51.4 (2.2) Estonia 2 (2.2) 54.2 (2.2) -1 (2.4) 50.0 (2.3) 1 (2.3) 51.7 (2.1) 5 (2.1) 55.6 (2.3) finland 1 (2.1) 51.4 (2.1) -6 (2.4) 46.0 (1.8) -3 (2.3) 47.9 (2.0) 2 (2.0) 51.8 (2.0) france 9 (2.7) 59.6 (2.2) 1 (2.9) 52.8 (2.2) 5 (2.5) 55.7 (2.0) 10 (2.6) 60.8 (2.2) Germany -2 (2.5) 50.5 (2.0) -5 (2.8) 48.3 (2.1) -3 (2.5) 48.1 (2.0) 1 (2.3) 51.4 (1.7) hungary -9 (3.8) 43.4 (2.7) -14 (3.9) 39.9 (2.7) -9 (3.7) 43.6 (2.8) -6 (3.7) 45.2 (2.7) ireland -6 (2.3) 44.4 (2.0) -7 (2.7) 44.6 (2.3) -6 (2.5) 46.0 (1.9) -4 (2.4) 46.3 (2.0) israel -2 (3.0) 50.0 (2.2) -20 (3.3) 39.0 (2.3) -9 (3.3) 44.0 (2.1) -2 (2.8) 49.2 (2.2) italy -7 (3.1) 47.0 (2.2) -7 (3.1) 46.6 (2.1) -5 (3.2) 49.0 (2.3) -4 (3.0) 48.9 (2.2) Japan 1 (2.5) 51.8 (1.6) -3 (2.7) 48.3 (1.7) -3 (2.5) 48.9 (1.7) 1 (2.5) 51.6 (1.5) korea 0 (1.7) 51.2 (1.5) -3 (2.1) 47.9 (1.8) -4 (2.1) 47.2 (1.7) 0 (1.8) 50.8 (1.6) netherlands -3 (3.4) 50.3 (2.8) -5 (3.2) 48.9 (2.5) -2 (3.5) 50.0 (2.8) 0 (3.3) 51.9 (2.8) norway -4 (2.7) 47.7 (2.3) -9 (3.1) 45.8 (1.9) -7 (3.0) 47.1 (2.1) -3 (2.6) 47.9 (2.1) Poland -8 (3.5) 46.4 (2.3) -10 (3.3) 44.8 (2.4) -10 (3.7) 44.5 (2.7) -4 (3.3) 48.2 (2.5) Portugal -3 (2.4) 48.0 (2.2) -12 (2.9) 41.8 (2.1) -8 (2.9) 44.7 (2.6) -2 (2.3) 48.8 (2.1) Slovak republic -9 (2.9) 43.6 (2.4) -16 (3.0) 39.7 (2.1) -12 (3.0) 42.6 (2.3) -7 (3.0) 44.9 (2.6) Slovenia (2.0) -6 (2.4) 46.8 (1.6) -10 (2.7) 44.4 (1.8) -7 (2.4) 46.9 (1.8) -4 (2.5) 48.8 Spain 4 (3.2) 54.0 (2.0) -6 (3.2) 47.3 (1.6) -3 (3.4) 49.6 (2.1) 4 (3.3) 54.1 (2.0) Sweden 1 (2.8) 50.8 (2.2) -8 (2.7) 46.2 (1.9) -2 (2.7) 48.9 (2.2) 2 (2.6) 51.7 (2.2) turkey -3 (2.4) 47.4 (2.5) -4 (3.6) 47.1 (2.9) -7 (2.5) 44.7 (2.0) -1 (2.5) 48.6 (2.5) 2 (3.0) 52.6 (2.6) 0 (3.4) 51.3 (2.7) 5 (2.8) 54.6 (2.4) 4 (2.9) 54.6 (2.6) England (united kingdom) united States Partners Percentage of students average from low-status difference in Percentage Percentage families who problem solving of students of students average average outperform compared from low-status from low-status difference in difference in students from with students families who families who problem solving problem solving high-status from highoutperform outperform compared compared families status families students from students from with students with students with similar with similar high-status high-status from highfrom highperformance performance families families status families status families in mathematics, in mathematics, with similar with similar with similar with similar reading reading performance performance performance performance and science2 and science3 in science2 in reading2 in science1 in reading1 3 (2.3) 52.8 (2.6) -2 (2.5) 48.1 (2.2) 0 (2.4) 50.5 (2.4) 4 (2.3) 53.7 (2.4) oEcd average -2 (0.5) 49.8 (0.4) -7 (0.5) 46.3 (0.4) -4 (0.5) 48.0 (0.4) 0 (0.5) 50.9 (0.4) brazil -3 (2.8) 48.1 (2.3) -16 (3.5) 39.9 (2.2) -8 (3.1) 45.4 (2.3) -3 (2.8) 48.2 (2.2) bulgaria -9 (3.3) 46.4 (2.2) -15 (3.6) 42.6 (2.0) -11 (3.6) 45.1 (2.2) -5 (3.2) 48.7 (2.0) colombia -5 (3.5) 47.5 (2.5) -9 (3.8) 44.9 (2.4) -11 (4.1) 43.5 (2.6) -3 (3.7) 48.5 (2.6) croatia -2 (2.4) 50.2 (2.2) -9 (2.8) 43.8 (1.8) -9 (2.9) 44.1 (2.1) -1 (2.3) 50.4 (2.1) cyprus* 0 (3.2) 49.9 (2.4) -17 (2.8) 40.5 (1.9) -2 (2.8) 49.0 (1.9) 1 (3.0) 50.5 (2.3) -1 (3.1) 50.7 (2.5) -7 (3.5) 46.2 (2.3) -5 (3.7) 47.0 (2.3) -1 (3.1) 50.0 (2.5) 1 (2.0) 51.8 (1.7) -5 (2.4) 47.7 (2.0) -1 (2.3) 50.6 (2.1) 1 (2.1) 52.0 (1.7) (2.2) hong kong-china macao-china malaysia -11 (2.3) 41.7 (2.4) -24 (2.6) 34.6 (1.9) -19 (2.3) 36.5 (1.9) -11 (2.2) 41.2 -5 (2.7) 46.2 (2.4) -13 (2.8) 41.9 (2.2) -9 (3.0) 44.2 (2.0) -4 (2.8) 46.6 (2.4) russian federation -16 (2.5) 39.5 (1.8) -18 (2.4) 38.9 (1.5) -17 (2.7) 40.1 (1.6) -15 (2.6) 40.4 (1.9) Serbia montenegro -10 (2.3) 42.9 (2.1) -20 (2.4) 37.8 (1.6) -18 (2.5) 38.3 (1.8) -9 (2.3) 42.8 (2.0) Shanghai-china -5 (2.1) 47.1 (2.0) -5 (2.2) 47.4 (1.7) -7 (2.4) 45.4 (1.9) -3 (2.1) 48.4 (1.9) Singapore -3 (2.0) 48.3 (1.9) -8 (2.2) 45.4 (1.7) -3 (2.1) 48.6 (2.4) -2 (2.0) 48.4 (1.8) 7 (1.8) 57.1 (2.1) -1 (2.2) 49.0 (1.8) 3 (2.0) 52.5 (2.0) 7 (1.9) 56.9 (1.9) -15 (3.2) 42.4 (2.2) -23 (3.2) 36.9 (1.9) -20 (3.0) 38.2 (2.1) -14 (3.0) 41.6 (2.1) -8 (4.3) 46.4 (2.9) -23 (4.2) 37.2 (2.3) -18 (3.8) 39.9 (2.3) -5 (3.7) 47.8 (2.7) chinese taipei united arab Emirates uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. This column reports the difference between actual performance and the itted value from a regression using a cubic polynomial as regression function. 2. This column reports the percentage of students for whom the difference between actual performance and the itted value from a regression is positive. Values that are indicated in bold are signiicantly larger or smaller than 50%. 3. This column reports the difference between actual performance and the itted value from a regression using a second-degree polynomial as regression function (math, math sq., read, read sq., scie, scie sq., math×read, math×scie, read×scie). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 204 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.18a [Part 1/1] Performance on problem-solving tasks, by nature of problem and by parents’ occupational status Results based on students’ self-reports items referring to a static problem situation items referring to an interactive problem situation relative likelihood of success, in favour of students with at least one parent working in a skilled occupation (semi-skilled or elementary = 1.00) average proportion of full-credit responses, by parents’ highest occupation average proportion of full-credit responses, by parents’ highest occupation relative likelihood of success, in favour of students with at least one parent working in a skilled occupation (semi-skilled or elementary = 1.00) Partners OECD difference difference related to related to parents’ parents’ occupational occupational status status based based accounting on success Semi-skilled (skilled Semi-skilled accounting on success (skilled semi-skilled or for booklet on remaining or elementary or elementary Skilled semi-skilled or for booklet on remaining Skilled effects1 (iSco 4 to 9) (iSco 1 to 3) elementary) effects1 test items2 test items2 (iSco 4 to 9) (iSco 1 to 3) elementary) % S.E. % S.E. australia 47.1 (0.8) 55.9 (0.5) austria 44.3 (1.5) 53.2 belgium 41.0 (1.2) 56.3 canada 49.2 (1.0) chile 31.5 czech republic % dif. S.E. odds ratio S.E. odds ratio S.E. % S.E. % S.E. % dif. S.E. odds ratio S.E. odds ratio S.E. 8.8 (0.9) 1.41 (0.05) 1.00 (0.04) 44.3 (0.8) 53.1 (0.5) 8.8 (0.8) 1.41 (0.04) 1.00 (0.04) (1.2) 8.9 (1.7) 1.47 (0.10) 0.94 (0.08) 38.1 (1.2) 48.4 (1.1) 10.3 (1.5) 1.57 (0.10) 1.07 (0.09) (0.9) 15.3 (1.6) 1.84 (0.12) 1.14 (0.07) 40.0 (0.9) 51.9 (0.8) 11.9 (1.3) 1.60 (0.08) 0.87 (0.05) 55.2 (0.9) 6.0 (1.4) 1.31 (0.07) 0.92 (0.06) 45.8 (0.9) 53.9 (0.8) 8.0 (1.2) 1.42 (0.07) 1.08 (0.07) (1.0) 41.7 (1.7) 10.2 (2.0) 1.55 (0.14) 0.91 (0.08) 27.8 (0.9) 39.8 (1.3) 12.0 (1.6) 1.70 (0.12) 1.10 (0.09) 41.3 (0.9) 53.8 (0.8) 12.5 (1.2) 1.66 (0.08) 0.98 (0.05) 39.5 (0.9) 52.5 (0.8) 13.0 (1.1) 1.70 (0.08) 1.02 (0.05) denmark 43.1 (1.6) 52.0 (1.1) 8.9 (2.0) 1.46 (0.11) 0.96 (0.07) 37.2 (1.3) 46.8 (0.8) 9.6 (1.5) 1.52 (0.10) 1.04 (0.08) Estonia 46.0 (1.3) 53.8 (1.3) 7.8 (2.1) 1.36 (0.11) 1.04 (0.09) 42.4 (1.1) 49.0 (1.3) 6.6 (1.7) 1.31 (0.09) 0.96 (0.08) finland 47.1 (1.0) 55.0 (0.7) 7.9 (1.2) 1.38 (0.07) 1.05 (0.05) 43.6 (1.0) 50.2 (0.7) 6.6 (1.2) 1.31 (0.06) 0.95 (0.04) france 45.1 (1.4) 55.1 (1.1) 10.0 (1.9) 1.52 (0.11) 0.92 (0.08) 41.5 (1.2) 53.2 (0.9) 11.7 (1.5) 1.65 (0.10) 1.08 (0.09) Germany 45.1 (1.6) 56.6 (1.1) 11.5 (2.1) 1.59 (0.12) 0.96 (0.08) 41.0 (1.4) 53.4 (1.0) 12.3 (1.5) 1.66 (0.11) 1.04 (0.08) hungary 32.0 (1.3) 48.5 (1.6) 16.6 (1.9) 2.09 (0.18) 1.03 (0.08) 28.1 (1.1) 43.1 (1.4) 14.9 (1.8) 2.02 (0.17) 0.97 (0.07) ireland 39.6 (1.7) 49.4 (1.1) 9.8 (2.1) 1.47 (0.12) 0.95 (0.08) 39.2 (1.3) 49.9 (1.1) 10.8 (1.7) 1.55 (0.11) 1.05 (0.09) israel 30.6 (1.6) 46.5 (1.7) 15.9 (2.2) 2.06 (0.20) 0.88 (0.07) 24.8 (1.3) 42.5 (1.6) 17.6 (1.7) 2.34 (0.19) 1.13 (0.09) italy 48.5 (1.4) 52.0 (1.4) 3.5 (2.0) 1.21 (0.09) 0.98 (0.08) 45.6 (1.3) 49.9 (1.3) 4.3 (1.8) 1.24 (0.09) 1.02 (0.09) Japan 57.4 (1.1) 60.7 (1.0) 3.3 (1.4) 1.14 (0.06) 0.94 (0.05) 53.9 (0.9) 58.9 (0.9) 5.0 (1.2) 1.21 (0.06) 1.06 (0.06) korea 56.0 (1.3) 61.4 (1.3) 5.5 (1.7) 1.31 (0.09) 1.10 (0.08) 56.1 (1.4) 59.1 (1.3) 3.1 (1.8) 1.18 (0.08) 0.91 (0.07) netherlands 42.2 (1.5) 54.9 (1.2) 12.7 (1.6) 1.67 (0.11) 0.97 (0.06) 38.0 (1.6) 51.3 (1.2) 13.3 (1.8) 1.73 (0.13) 1.03 (0.06) norway 44.2 (1.8) 52.4 (1.1) 8.2 (2.0) 1.44 (0.12) 0.98 (0.09) 38.8 (1.6) 47.4 (1.1) 8.6 (1.9) 1.47 (0.12) 1.02 (0.09) Poland 39.5 (1.4) 50.9 (1.4) 11.4 (2.1) 1.60 (0.13) 0.95 (0.08) 34.8 (1.2) 47.1 (1.6) 12.3 (1.8) 1.69 (0.13) 1.05 (0.09) Portugal 41.6 (1.3) 50.2 (1.6) 8.6 (2.2) 1.52 (0.14) 0.86 (0.06) 38.0 (1.1) 50.6 (1.3) 12.5 (1.5) 1.77 (0.11) 1.16 (0.09) Slovak republic 41.1 (1.2) 54.0 (1.3) 12.9 (1.6) 1.71 (0.11) 1.11 (0.08) 36.9 (1.1) 47.0 (1.3) 10.1 (1.9) 1.53 (0.12) 0.90 (0.07) Slovenia 36.2 (1.2) 49.8 (1.2) 13.6 (1.9) 1.81 (0.14) 1.06 (0.10) 31.0 (1.1) 42.5 (1.2) 11.5 (1.6) 1.72 (0.13) 0.95 (0.09) Spain 38.4 (1.1) 47.7 (1.2) 9.4 (1.7) 1.47 (0.10) 1.01 (0.07) 36.1 (0.9) 45.1 (1.1) 9.0 (1.3) 1.46 (0.08) 0.99 (0.07) Sweden 43.1 (1.5) 51.7 (1.2) 8.6 (2.2) 1.44 (0.12) 0.96 (0.09) 36.5 (1.2) 45.7 (0.9) 9.2 (1.5) 1.51 (0.09) 1.05 (0.10) turkey 34.5 (0.9) 42.4 (1.7) 7.9 (1.6) 1.41 (0.09) 0.93 (0.06) 31.3 (0.8) 40.5 (1.7) 9.2 (1.4) 1.51 (0.09) 1.08 (0.07) England (united kingdom) 45.9 (1.0) 52.9 (1.2) 7.0 (1.4) 1.32 (0.07) 1.00 (0.07) 44.7 (1.3) 51.7 (1.2) 7.0 (1.6) 1.31 (0.09) 1.00 (0.07) united States 39.3 (1.5) 51.2 (1.2) 11.8 (1.9) 1.67 (0.13) 1.08 (0.08) 40.0 (1.3) 50.2 (1.2) 10.1 (1.7) 1.55 (0.11) 0.93 (0.07) oEcd average 42.5 (0.2) 52.3 (0.2) 9.8 (0.3) 1.52 (0.02) 0.98 (0.01) 39.1 (0.2) 49.1 (0.2) 10.0 (0.3) 1.54 (0.02) 1.02 (0.01) brazil 27.8 (1.5) 35.3 (1.8) 7.4 (2.6) 1.45 (0.17) 0.92 (0.11) 26.3 (1.2) 35.5 (1.6) 9.2 (1.9) 1.57 (0.14) 1.08 (0.12) bulgaria 24.4 (1.0) 37.0 (1.3) 12.5 (1.6) 1.82 (0.13) 0.89 (0.05) 18.1 (0.7) 31.1 (1.1) 13.0 (1.2) 2.04 (0.14) 1.12 (0.07) colombia 24.2 (1.0) 33.0 (2.0) 8.8 (2.2) 1.55 (0.16) 0.99 (0.08) 21.9 (0.7) 30.4 (1.4) 8.5 (1.5) 1.57 (0.12) 1.01 (0.08) croatia 35.4 (1.0) 45.7 (1.3) 10.2 (1.4) 1.53 (0.09) 1.01 (0.06) 32.1 (0.9) 41.8 (1.2) 9.7 (1.3) 1.52 (0.08) 0.99 (0.06) cyprus* 33.3 (0.7) 43.7 (0.9) 10.3 (1.2) 1.55 (0.08) 0.97 (0.05) 27.9 (0.6) 38.2 (0.8) 10.3 (1.0) 1.60 (0.07) 1.03 (0.05) hong kong-china 56.4 (1.2) 58.0 (1.4) 1.6 (1.9) 1.06 (0.08) 0.83 (0.07) 50.7 (0.9) 56.7 (1.5) 6.0 (1.8) 1.28 (0.09) 1.21 (0.10) macao-china 56.6 (0.7) 59.4 (1.4) 2.8 (1.6) 1.13 (0.08) 1.00 (0.08) 51.0 (0.8) 54.0 (1.0) 3.0 (1.4) 1.13 (0.07) 1.00 (0.08) malaysia 26.8 (0.7) 36.0 (1.4) 9.2 (1.5) 1.54 (0.10) 0.86 (0.05) 23.0 (0.7) 34.9 (1.2) 11.9 (1.2) 1.79 (0.10) 1.16 (0.07) montenegro 28.3 (0.8) 34.8 (1.0) 6.6 (1.2) 1.35 (0.08) 0.94 (0.05) 23.1 (0.7) 30.2 (0.8) 7.1 (1.1) 1.44 (0.08) 1.07 (0.06) russian federation 38.6 (1.3) 48.4 (1.3) 9.8 (2.0) 1.51 (0.13) 0.99 (0.07) 34.3 (1.0) 43.9 (1.1) 9.6 (1.4) 1.53 (0.09) 1.01 (0.07) Serbia 34.3 (1.0) 48.6 (1.0) 14.2 (1.4) 1.80 (0.11) 1.08 (0.06) 31.9 (0.9) 43.8 (1.0) 11.9 (1.4) 1.66 (0.10) 0.92 (0.05) Shanghai-china 51.1 (1.4) 60.6 (1.3) 9.5 (1.6) 1.45 (0.10) 0.99 (0.07) 44.7 (1.3) 54.5 (1.0) 9.8 (1.4) 1.46 (0.09) 1.01 (0.07) Singapore 53.7 (1.5) 63.4 (0.9) 9.7 (1.8) 1.53 (0.11) 1.12 (0.09) 53.2 (1.4) 60.2 (1.0) 7.0 (1.9) 1.36 (0.10) 0.89 (0.07) chinese taipei 52.7 (1.3) 63.3 (1.3) 10.6 (1.9) 1.58 (0.13) 1.05 (0.07) 46.6 (1.2) 56.5 (1.1) 9.8 (1.7) 1.50 (0.09) 0.95 (0.07) united arab Emirates 22.9 (1.6) 33.1 (0.7) 10.1 (1.8) 1.74 (0.17) 0.87 (0.10) 18.9 (1.2) 31.1 (0.6) 12.2 (1.3) 2.00 (0.16) 1.15 (0.14) uruguay 24.4 (0.7) 37.3 (1.5) 12.9 (1.7) 1.84 (0.14) 0.97 (0.05) 21.8 (0.6) 34.7 (1.4) 12.9 (1.4) 1.91 (0.13) 1.03 (0.06) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an occupational status dummy, and an interaction term (occupational status × item type). Booklet dummies are added to the estimation. This column presents the difference between the logit coeficient on the interaction term and the logit coeficient on the item type dummy in exponentiated form. 2. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an occupational status dummy, and an interaction term (occupational status × item type). Booklet dummies are added to the estimation. This column presents the logit coeficient on the interaction term in exponentiated form. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 205 Annex b1: reSulTS For counTrIeS And economIeS table v.4.18b [Part 1/2] Performance on problem-solving tasks, by process and by parents’ occupational status Results based on students’ self-reports items assessing the process of “exploring and understanding” average proportion of full-credit responses, by parents’ highest occupation items assessing the process of “representing and formulating” relative likelihood of success, in favour of students with at least one parent working in a skilled occupation (semi-skilled or elementary = 1.00) average proportion of full-credit responses, by parents’ highest occupation relative likelihood of success, in favour of students with at least one parent working in a skilled occupation (semi-skilled or elementary = 1.00) Partners OECD difference difference related to related to parents’ parents’ occupational occupational status status based based accounting on success Semi-skilled (skilled Semi-skilled accounting on success (skilled semi-skilled or for booklet on remaining or elementary or elementary Skilled semi-skilled or for booklet on remaining Skilled effects1 (iSco 4 to 9) (iSco 1 to 3) elementary) effects1 test items2 test items2 (iSco 4 to 9) (iSco 1 to 3) elementary) % dif. S.E. odds ratio % S.E. % S.E. S.E. australia 48.8 (1.0) 58.5 (0.6) 9.7 (1.0) 1.47 (0.06) austria 42.4 (1.3) 56.8 (1.5) 14.4 (1.8) belgium 41.7 (1.2) 57.3 (1.0) 15.6 (1.6) canada 47.7 (1.2) 58.6 (0.8) 10.8 chile 28.8 (1.1) 39.8 (1.8) czech republic 41.3 (1.0) 55.9 denmark 40.9 (1.3) Estonia 44.5 finland odds ratio S.E. % dif. S.E. odds ratio S.E. odds ratio % S.E. % S.E. 1.05 (0.04) 43.9 (1.0) 52.5 (0.7) 8.6 (1.1) 1.40 (0.06) 0.99 (0.04) S.E. 1.85 (0.15) 1.28 (0.09) 36.0 (1.4) 47.7 (1.5) 11.7 (1.9) 1.68 (0.14) 1.12 (0.08) 1.87 (0.13) 1.15 (0.07) 38.5 (1.2) 52.3 (0.9) 13.7 (1.4) 1.74 (0.10) 1.05 (0.06) (1.4) 1.61 (0.10) 1.22 (0.06) 45.6 (1.1) 54.4 (1.1) 8.8 (1.4) 1.48 (0.08) 1.09 (0.05) 11.1 (2.0) 1.62 (0.15) 0.98 (0.07) 24.0 (1.1) 39.9 (1.7) 15.9 (2.1) 2.08 (0.21) 1.35 (0.10) (0.9) 14.7 (1.1) 1.81 (0.08) 1.10 (0.04) 37.9 (1.0) 50.9 (1.0) 12.9 (1.3) 1.70 (0.09) 1.01 (0.04) 50.5 (1.3) 9.6 (1.9) 1.51 (0.11) 1.01 (0.06) 36.1 (1.8) 47.2 (1.3) 11.1 (2.0) 1.63 (0.14) 1.11 (0.07) (1.6) 52.9 (1.7) 8.5 (2.5) 1.41 (0.15) 1.08 (0.09) 42.1 (1.4) 47.0 (1.5) 4.9 (2.1) 1.23 (0.10) 0.91 (0.05) 47.2 (1.1) 57.4 (0.8) 10.2 (1.3) 1.52 (0.08) 1.18 (0.05) 41.7 (1.3) 49.0 (0.9) 7.3 (1.7) 1.35 (0.09) 1.01 (0.05) france 46.1 (1.5) 57.9 (1.3) 11.8 (2.0) 1.64 (0.13) 1.03 (0.07) 40.8 (1.4) 52.9 (1.2) 12.1 (1.9) 1.67 (0.13) 1.06 (0.07) Germany 44.7 (1.9) 59.7 (1.2) 15.0 (2.2) 1.86 (0.16) 1.18 (0.08) 38.0 (1.6) 52.3 (1.4) 14.3 (2.2) 1.80 (0.17) 1.13 (0.08) hungary 31.1 (1.3) 48.1 (1.8) 17.0 (2.2) 2.17 (0.21) 1.08 (0.08) 26.3 (1.3) 42.1 (1.7) 15.9 (2.2) 2.16 (0.23) 1.07 (0.08) ireland 41.4 (2.0) 53.8 (1.2) 12.4 (2.2) 1.66 (0.16) 1.12 (0.09) 36.2 (1.5) 46.8 (1.3) 10.6 (2.0) 1.57 (0.13) 1.04 (0.08) israel 30.5 (1.8) 49.6 (1.7) 19.1 (2.2) 2.35 (0.24) 1.08 (0.09) 25.0 (1.7) 42.0 (1.9) 17.1 (2.3) 2.30 (0.26) 1.04 (0.09) italy 48.7 (1.6) 57.3 (1.8) 8.6 (2.3) 1.49 (0.13) 1.29 (0.10) 46.3 (1.4) 49.8 (1.6) 3.5 (1.8) 1.20 (0.10) 0.97 (0.07) Japan 60.1 (1.2) 65.2 (1.1) 5.1 (1.3) 1.23 (0.07) 1.05 (0.05) 53.8 (1.1) 58.6 (1.0) 4.8 (1.3) 1.20 (0.07) 1.02 (0.04) korea 61.5 (1.4) 67.6 (1.4) 6.1 (1.8) 1.38 (0.11) 1.16 (0.07) 58.1 (1.9) 62.8 (1.5) 4.7 (2.1) 1.28 (0.11) 1.06 (0.07) netherlands 42.6 (1.8) 56.8 (1.2) 14.2 (1.8) 1.79 (0.13) 1.06 (0.04) 34.2 (1.6) 49.7 (1.4) 15.4 (1.8) 1.91 (0.15) 1.15 (0.06) norway 45.5 (2.1) 54.5 (1.1) 8.9 (2.3) 1.49 (0.15) 1.03 (0.08) 37.4 (1.8) 46.7 (1.4) 9.3 (2.2) 1.52 (0.15) 1.06 (0.09) Poland 38.9 (1.4) 51.3 (1.8) 12.4 (2.2) 1.67 (0.15) 1.01 (0.07) 32.6 (1.4) 47.4 (1.8) 14.9 (2.0) 1.90 (0.15) 1.19 (0.08) Portugal 39.8 (1.6) 51.9 (1.7) 12.2 (2.3) 1.74 (0.18) 1.05 (0.10) 35.1 (1.6) 48.6 (1.8) 13.5 (2.4) 1.85 (0.19) 1.13 (0.11) Slovak republic 41.7 (1.4) 51.9 (1.9) 10.1 (2.2) 1.51 (0.14) 0.93 (0.07) 34.2 (1.3) 47.3 (1.6) 13.0 (2.0) 1.75 (0.15) 1.12 (0.07) Slovenia 32.4 (1.4) 46.7 (1.5) 14.4 (2.2) 1.91 (0.19) 1.12 (0.10) 29.1 (1.4) 42.3 (1.3) 13.2 (1.9) 1.87 (0.15) 1.09 (0.08) Spain 38.2 (1.0) 48.3 (1.5) 10.1 (1.6) 1.52 (0.11) 1.05 (0.06) 32.6 (1.1) 43.6 (1.4) 11.1 (1.8) 1.61 (0.13) 1.13 (0.08) Sweden 42.8 (1.6) 52.9 (1.2) 10.1 (1.9) 1.55 (0.12) 1.06 (0.08) 34.4 (1.5) 47.5 (1.3) 13.2 (2.0) 1.80 (0.15) 1.28 (0.09) turkey 31.7 (0.8) 43.5 (2.2) 11.7 (2.1) 1.67 (0.14) 1.18 (0.07) 30.2 (1.0) 40.3 (2.0) 10.1 (1.8) 1.58 (0.12) 1.09 (0.06) England (united kingdom) 48.7 (1.4) 54.6 (1.4) 5.8 (1.8) 1.26 (0.09) 0.94 (0.05) 42.6 (1.4) 52.4 (1.4) 9.8 (1.8) 1.48 (0.11) 1.16 (0.06) united States 40.5 (1.6) 54.6 (1.2) 14.1 (1.9) 1.83 (0.16) 1.20 (0.08) 39.1 (1.8) 47.6 (1.6) 8.5 (2.3) 1.45 (0.14) 0.89 (0.06) oEcd average 42.5 (0.3) 54.1 (0.3) 11.6 (0.4) 1.64 (0.03) 1.09 (0.01) 37.6 (0.3) 48.6 (0.3) 11.1 (0.4) 1.63 (0.03) 1.08 (0.01) brazil 28.3 (1.5) 35.6 (1.8) 7.2 (2.1) 1.41 (0.14) 0.90 (0.07) 21.4 (1.3) 33.7 (2.4) 12.3 (2.7) 1.88 (0.25) 1.30 (0.12) bulgaria 23.5 (1.0) 37.2 (1.4) 13.7 (1.7) 1.94 (0.15) 0.99 (0.05) 14.9 (0.9) 27.8 (1.3) 12.9 (1.5) 2.20 (0.20) 1.16 (0.08) colombia 22.3 (1.0) 32.9 (2.2) 10.6 (2.4) 1.71 (0.20) 1.13 (0.12) 16.5 (0.9) 26.3 (2.0) 9.7 (2.1) 1.81 (0.21) 1.20 (0.11) croatia 32.9 (1.0) 44.6 (1.4) 11.8 (1.5) 1.65 (0.10) 1.11 (0.05) 29.1 (1.1) 39.8 (1.7) 10.8 (1.6) 1.61 (0.12) 1.08 (0.06) cyprus* 32.9 (0.8) 42.3 (0.9) 9.4 (1.2) 1.50 (0.08) 0.93 (0.04) 26.6 (0.7) 38.3 (1.0) 11.7 (1.2) 1.72 (0.09) 1.11 (0.04) hong kong-china 58.8 (1.3) 65.0 (1.7) 6.2 (1.9) 1.31 (0.11) 1.12 (0.07) 53.7 (1.2) 59.2 (1.7) 5.5 (2.0) 1.27 (0.10) 1.08 (0.08) macao-china 58.2 (1.0) 63.6 (1.4) 5.4 (1.6) 1.28 (0.10) 1.17 (0.08) 56.9 (1.1) 58.6 (1.3) 1.7 (1.7) 1.08 (0.08) 0.94 (0.06) malaysia 26.2 (0.8) 37.0 (1.5) 10.7 (1.6) 1.65 (0.12) 0.96 (0.05) 23.4 (0.9) 35.9 (1.5) 12.5 (1.5) 1.83 (0.13) 1.11 (0.05) montenegro 24.6 (0.9) 32.6 (1.0) 8.0 (1.4) 1.47 (0.10) 1.07 (0.06) 21.6 (0.7) 28.8 (1.1) 7.3 (1.4) 1.47 (0.11) 1.06 (0.06) russian federation 35.5 (1.4) 47.6 (1.6) 12.1 (2.1) 1.68 (0.16) 1.14 (0.09) 32.3 (1.7) 43.2 (1.3) 11.0 (2.1) 1.62 (0.15) 1.09 (0.09) Serbia 33.5 (1.2) 47.8 (1.2) 14.3 (1.7) 1.81 (0.13) 1.08 (0.06) 30.1 (1.0) 43.5 (1.3) 13.4 (1.7) 1.79 (0.13) 1.06 (0.07) Shanghai-china 53.1 (1.6) 62.2 (1.4) 9.1 (2.0) 1.43 (0.12) 0.98 (0.08) 48.3 (1.9) 60.7 (1.5) 12.4 (2.2) 1.63 (0.15) 1.15 (0.10) Singapore 56.6 (1.7) 68.6 (1.1) 12.0 (2.0) 1.72 (0.15) 1.29 (0.10) 54.9 (1.7) 62.6 (1.2) 7.7 (2.3) 1.41 (0.13) 0.99 (0.08) chinese taipei 54.5 (1.4) 65.6 (1.4) 11.1 (1.9) 1.61 (0.13) 1.07 (0.07) 51.7 (1.6) 62.1 (1.5) 10.4 (2.1) 1.54 (0.13) 1.01 (0.06) united arab Emirates 21.5 (1.5) 33.6 (0.8) 12.1 (1.7) 1.91 (0.19) 1.01 (0.09) 17.4 (1.5) 30.3 (0.8) 13.0 (1.5) 2.11 (0.21) 1.14 (0.12) uruguay 23.8 (0.6) 37.9 (1.6) 14.1 (1.7) 1.95 (0.14) 1.05 (0.05) 18.4 (0.8) 34.1 (1.7) 15.7 (1.8) 2.30 (0.20) 1.28 (0.07) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an occupational status dummy, and an interaction term (occupational status × item type). Booklet dummies are added to the estimation. This column presents the difference between the logit coeficient on the interaction term and the logit coeficient on the item type dummy in exponentiated form. 2. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an occupational status dummy, and an interaction term (occupational status × item type). Booklet dummies are added to the estimation. This column presents the logit coeficient on the interaction term in exponentiated form. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 206 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.18b [Part 2/2] Performance on problem-solving tasks, by process and by parents’ occupational status Results based on students’ self-reports items assessing the process of “planning and executing” average proportion of full-credit responses, by parents’ highest occupation items assessing the process of “monitoring and relecting” relative likelihood of success, in favour of students with at least one parent working in a skilled occupation (semi-skilled or elementary = 1.00) average proportion of full-credit responses, by parents’ highest occupation relative likelihood of success, in favour of students with at least one parent working in a skilled occupation (semi-skilled or elementary = 1.00) Partners OECD difference difference related to related to parents’ parents’ occupational occupational status status based based accounting on success Semi-skilled (skilled Semi-skilled accounting on success (skilled semi-skilled or for booklet on remaining or elementary or elementary Skilled semi-skilled or for booklet on remaining Skilled effects1 (iSco 4 to 9) (iSco 1 to 3) elementary) effects1 test items2 test items2 (iSco 4 to 9) (iSco 1 to 3) elementary) % S.E. % S.E. australia 46.3 (0.8) 54.4 (0.5) austria 43.9 (1.4) 51.5 belgium 41.7 (0.9) 53.9 canada 49.3 (0.9) chile 31.9 czech republic % dif. S.E. odds ratio S.E. odds ratio S.E. % S.E. % S.E. % dif. S.E. odds ratio S.E. odds ratio S.E. 8.2 (0.9) 1.37 (0.05) 0.96 (0.03) 39.9 (0.8) 49.3 (0.5) 9.4 (0.9) 1.46 (0.06) 1.04 (0.04) (1.0) 7.5 (1.6) 1.39 (0.09) 0.85 (0.05) 34.5 (1.6) 40.4 (1.0) 5.9 (1.8) 1.32 (0.11) 0.83 (0.07) (0.9) 12.2 (1.5) 1.61 (0.09) 0.93 (0.05) 37.6 (1.0) 48.3 (1.1) 10.8 (1.6) 1.54 (0.10) 0.90 (0.05) 54.3 (0.8) 5.0 (1.2) 1.25 (0.06) 0.85 (0.04) 42.7 (1.1) 48.3 (1.0) 5.6 (1.4) 1.28 (0.08) 0.91 (0.06) (0.9) 41.5 (1.2) 9.6 (1.4) 1.50 (0.10) 0.87 (0.05) 29.6 (1.0) 39.5 (1.4) 9.9 (1.9) 1.54 (0.13) 0.92 (0.06) 42.3 (0.8) 54.3 (0.8) 12.0 (1.1) 1.62 (0.07) 0.94 (0.03) 36.3 (1.0) 48.2 (0.9) 12.0 (1.2) 1.64 (0.09) 0.97 (0.03) denmark 42.8 (1.5) 52.5 (0.9) 9.6 (1.7) 1.51 (0.10) 1.01 (0.06) 32.9 (1.5) 39.0 (1.1) 6.1 (1.9) 1.33 (0.11) 0.87 (0.08) Estonia 45.9 (1.1) 53.5 (1.2) 7.6 (1.7) 1.35 (0.09) 1.03 (0.07) 39.6 (1.2) 45.7 (1.2) 6.2 (1.7) 1.29 (0.10) 0.97 (0.07) finland 47.3 (0.9) 53.4 (0.7) 6.1 (1.1) 1.28 (0.05) 0.94 (0.04) 40.0 (1.0) 44.4 (0.7) 4.4 (1.2) 1.20 (0.06) 0.88 (0.04) france 43.7 (1.4) 54.3 (0.9) 10.6 (1.7) 1.57 (0.10) 0.96 (0.06) 38.7 (1.2) 49.0 (1.2) 10.3 (1.7) 1.56 (0.12) 0.96 (0.07) Germany 45.4 (1.3) 55.7 (1.0) 10.3 (1.6) 1.52 (0.10) 0.89 (0.05) 38.4 (1.4) 47.3 (1.2) 8.8 (1.6) 1.44 (0.10) 0.86 (0.06) hungary 31.6 (1.1) 47.6 (1.4) 16.1 (1.8) 2.05 (0.16) 1.00 (0.06) 26.6 (1.3) 38.3 (1.8) 11.7 (2.0) 1.76 (0.17) 0.84 (0.06) ireland 40.8 (1.3) 50.1 (1.0) 9.3 (1.7) 1.44 (0.10) 0.92 (0.05) 37.2 (1.7) 46.8 (1.4) 9.6 (2.1) 1.47 (0.13) 0.96 (0.08) israel 27.5 (1.5) 43.8 (1.7) 16.3 (1.9) 2.14 (0.19) 0.94 (0.07) 22.9 (1.3) 38.4 (1.5) 15.5 (1.7) 2.19 (0.19) 0.98 (0.07) italy 47.7 (1.2) 49.4 (1.4) 1.7 (1.8) 1.12 (0.08) 0.86 (0.05) 41.9 (1.3) 44.9 (1.4) 2.9 (2.0) 1.17 (0.10) 0.94 (0.07) Japan 54.8 (1.0) 58.5 (0.9) 3.7 (1.3) 1.15 (0.06) 0.96 (0.04) 50.6 (0.9) 54.8 (1.1) 4.2 (1.5) 1.18 (0.07) 0.99 (0.05) korea 53.1 (1.2) 55.7 (1.3) 2.6 (1.8) 1.15 (0.08) 0.90 (0.05) 52.2 (1.6) 55.1 (1.4) 2.9 (2.1) 1.16 (0.10) 0.93 (0.07) netherlands 42.2 (1.6) 54.2 (1.2) 12.0 (1.8) 1.63 (0.12) 0.92 (0.04) 35.7 (1.6) 46.8 (1.3) 11.1 (1.8) 1.59 (0.12) 0.92 (0.05) norway 42.4 (1.7) 51.1 (1.1) 8.6 (1.9) 1.47 (0.12) 1.01 (0.07) 34.6 (1.5) 40.4 (1.4) 5.8 (1.9) 1.32 (0.12) 0.89 (0.06) Poland 39.4 (1.3) 50.2 (1.4) 10.8 (1.9) 1.56 (0.12) 0.91 (0.06) 31.4 (1.3) 41.9 (1.6) 10.5 (2.1) 1.59 (0.15) 0.95 (0.07) Portugal 42.5 (1.0) 53.3 (1.6) 10.7 (1.8) 1.64 (0.12) 0.97 (0.07) 36.5 (1.3) 44.4 (1.9) 7.9 (2.2) 1.47 (0.14) 0.86 (0.07) Slovak republic 40.4 (1.1) 52.6 (1.3) 12.2 (1.7) 1.65 (0.11) 1.06 (0.06) 34.4 (1.2) 42.1 (1.4) 7.8 (2.0) 1.41 (0.13) 0.87 (0.07) Slovenia 36.6 (1.0) 48.2 (1.1) 11.6 (1.5) 1.66 (0.11) 0.92 (0.05) 29.6 (1.2) 39.4 (1.0) 9.9 (1.6) 1.60 (0.12) 0.90 (0.07) Spain 39.1 (1.1) 46.9 (1.0) 7.8 (1.4) 1.38 (0.08) 0.91 (0.06) 35.7 (1.2) 43.8 (1.4) 8.1 (1.8) 1.40 (0.11) 0.95 (0.07) Sweden 40.8 (1.2) 48.0 (0.9) 7.2 (1.6) 1.36 (0.08) 0.88 (0.05) 34.5 (1.2) 40.8 (1.3) 6.2 (1.8) 1.33 (0.10) 0.88 (0.06) turkey 35.0 (0.8) 41.5 (1.3) 6.5 (1.2) 1.33 (0.07) 0.84 (0.04) 30.5 (0.9) 38.4 (2.0) 7.9 (2.0) 1.43 (0.12) 0.97 (0.07) England (united kingdom) 45.1 (1.2) 53.0 (1.1) 7.9 (1.4) 1.37 (0.08) 1.06 (0.05) 43.4 (1.5) 46.3 (1.1) 2.9 (1.7) 1.12 (0.08) 0.82 (0.05) united States 40.8 (1.4) 51.3 (1.1) 10.4 (1.6) 1.56 (0.10) 0.98 (0.05) 37.4 (1.8) 46.9 (1.3) 9.6 (2.1) 1.51 (0.13) 0.94 (0.07) oEcd average 42.2 (0.2) 51.2 (0.2) 9.1 (0.3) 1.47 (0.02) 0.94 (0.01) 36.6 (0.2) 44.6 (0.2) 8.0 (0.3) 1.42 (0.02) 0.92 (0.01) brazil 29.9 (1.4) 37.7 (1.6) 7.8 (2.1) 1.45 (0.14) 0.92 (0.07) 24.5 (1.1) 32.4 (1.6) 8.0 (2.0) 1.52 (0.16) 0.99 (0.09) bulgaria 22.7 (0.8) 35.2 (1.1) 12.5 (1.2) 1.85 (0.11) 0.92 (0.04) 17.8 (0.8) 29.8 (1.3) 12.1 (1.4) 1.97 (0.15) 1.02 (0.05) colombia 26.1 (1.0) 34.2 (1.6) 8.1 (1.9) 1.48 (0.13) 0.92 (0.06) 23.7 (1.0) 28.8 (1.5) 5.2 (1.7) 1.32 (0.11) 0.82 (0.08) croatia 37.5 (0.9) 46.0 (1.2) 8.5 (1.3) 1.42 (0.07) 0.89 (0.04) 29.9 (0.9) 39.0 (1.2) 9.1 (1.2) 1.50 (0.08) 0.98 (0.05) cyprus* 31.3 (0.6) 41.3 (0.9) 9.9 (1.0) 1.54 (0.07) 0.96 (0.03) 26.2 (0.6) 36.8 (0.9) 10.6 (1.1) 1.64 (0.09) 1.05 (0.05) hong kong-china 51.0 (1.0) 53.1 (1.5) 2.2 (1.9) 1.08 (0.08) 0.85 (0.05) 47.2 (1.2) 52.4 (1.8) 5.2 (2.2) 1.24 (0.11) 1.04 (0.08) macao-china 50.8 (0.6) 53.4 (1.3) 2.5 (1.4) 1.11 (0.06) 0.98 (0.06) 45.8 (1.0) 47.1 (1.3) 1.2 (1.7) 1.05 (0.08) 0.91 (0.06) malaysia 25.4 (0.7) 35.7 (1.1) 10.3 (1.1) 1.63 (0.08) 0.94 (0.04) 20.5 (0.6) 31.1 (1.2) 10.6 (1.3) 1.75 (0.11) 1.04 (0.05) montenegro 28.1 (0.8) 35.0 (0.8) 6.8 (1.2) 1.37 (0.08) 0.96 (0.05) 22.4 (0.8) 27.7 (1.0) 5.2 (1.2) 1.32 (0.09) 0.93 (0.05) russian federation 39.5 (1.0) 47.5 (1.0) 8.0 (1.5) 1.40 (0.09) 0.87 (0.05) 32.5 (1.1) 41.2 (1.4) 8.6 (1.6) 1.48 (0.10) 0.97 (0.06) Serbia 35.7 (0.9) 47.8 (0.9) 12.1 (1.2) 1.65 (0.09) 0.94 (0.04) 28.4 (1.2) 39.6 (1.1) 11.2 (1.7) 1.65 (0.13) 0.96 (0.05) Shanghai-china 45.6 (1.2) 52.8 (1.0) 7.2 (1.4) 1.31 (0.08) 0.84 (0.06) 39.7 (1.6) 52.6 (1.3) 13.0 (1.8) 1.67 (0.13) 1.18 (0.08) Singapore 51.7 (1.4) 57.6 (1.0) 5.9 (1.8) 1.29 (0.09) 0.86 (0.05) 50.6 (1.6) 58.1 (1.1) 7.5 (2.1) 1.38 (0.11) 0.97 (0.07) chinese taipei 47.2 (1.2) 55.8 (1.1) 8.6 (1.8) 1.43 (0.10) 0.90 (0.05) 40.4 (1.3) 52.4 (1.5) 11.9 (2.0) 1.64 (0.13) 1.09 (0.08) united arab Emirates 21.8 (1.3) 32.7 (0.7) 10.9 (1.3) 1.83 (0.13) 0.94 (0.06) 19.2 (1.2) 29.0 (0.8) 9.8 (1.5) 1.80 (0.17) 0.94 (0.08) uruguay 25.0 (0.7) 37.1 (1.3) 12.1 (1.5) 1.77 (0.12) 0.90 (0.04) 21.5 (0.7) 31.0 (1.5) 9.5 (1.5) 1.64 (0.12) 0.85 (0.04) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an occupational status dummy, and an interaction term (occupational status × item type). Booklet dummies are added to the estimation. This column presents the difference between the logit coeficient on the interaction term and the logit coeficient on the item type dummy in exponentiated form. 2. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an occupational status dummy, and an interaction term (occupational status × item type). Booklet dummies are added to the estimation. This column presents the logit coeficient on the interaction term in exponentiated form. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 207 Annex b1: reSulTS For counTrIeS And economIeS table v.4.19 [Part 1/2] Performance in problem solving and immigrant background Results based on students’ self-reports non-immigrant students Partners OECD Percentage of students Second-generation immigrant students Performance in problem solving Percentage of students Performance in problem solving Students with an immigrant background (irst- or second-generation immigrant students) first-generation immigrant students Percentage of students Performance in problem solving Percentage of students Performance in problem solving % S.E. mean score S.E. % S.E. mean score S.E. % S.E. mean score S.E. % S.E. mean score S.E. australia 77.3 (0.7) 524 (1.9) 12.4 (0.6) 537 (4.8) 10.3 (0.4) 524 (4.0) 22.7 (0.7) 531 (3.4) austria 83.5 (1.1) 516 (3.6) 10.9 (0.7) 465 (6.0) 5.6 (0.6) 454 (8.6) 16.5 (1.1) 461 (5.7) belgium 84.7 (0.9) 522 (2.5) 8.0 (0.6) 438 (7.0) 7.3 (0.6) 455 (7.7) 15.3 (0.9) 446 (6.0) canada 70.4 (1.3) 532 (2.2) 16.6 (0.8) 519 (5.6) 13.0 (0.7) 521 (5.9) 29.6 (1.3) 520 (5.0) chile 99.1 (0.2) 448 (3.7) 0.2 (0.1) c c 0.7 (0.1) 454 (15.7) 0.9 (0.2) 448 (15.5) czech republic 96.7 (0.4) 510 (3.2) 1.4 (0.3) 477 (20.6) 1.9 (0.2) 482 (11.5) 3.3 (0.4) 480 (11.4) denmark 90.8 (0.6) 505 (2.9) 6.1 (0.5) 436 (7.6) 3.0 (0.2) 424 (7.6) 9.2 (0.6) 432 (6.0) Estonia 91.9 (0.5) 519 (2.5) 7.5 (0.5) 489 (7.3) 0.7 (0.2) c c 8.1 (0.5) 486 (7.3) finland 96.6 (0.2) 526 (2.3) 1.5 (0.1) 461 (5.7) 1.9 (0.2) 426 (8.2) 3.4 (0.2) 442 (5.2) france 85.0 (1.1) 523 (3.5) 10.0 (0.8) 464 (8.7) 5.0 (0.5) 432 (10.3) 15.0 (1.1) 454 (7.1) Germany 86.6 (0.8) 523 (3.4) 10.6 (0.7) 475 (6.8) 2.8 (0.3) 463 (10.6) 13.4 (0.8) 473 (6.1) hungary 98.3 (0.2) 459 (4.0) 1.0 (0.2) 482 (14.7) 0.8 (0.2) c c 1.7 (0.2) 479 (14.0) ireland 89.8 (0.7) 501 (3.4) 1.7 (0.2) 493 (14.1) 8.5 (0.7) 487 (5.6) 10.2 (0.7) 488 (5.1) israel 81.7 (1.2) 452 (5.7) 12.7 (0.8) 481 (9.4) 5.6 (0.6) 460 (10.7) 18.3 (1.2) 474 (8.4) italy 92.7 (0.6) 514 (4.1) 1.9 (0.3) 493 (10.1) 5.4 (0.5) 451 (10.5) 7.3 (0.6) 462 (9.2) Japan 99.7 (0.1) 553 (3.1) 0.2 (0.1) c c 0.1 (0.0) c c 0.3 (0.1) c c korea 100.0 (0.0) 562 (4.3) 0.0 (0.0) c c 0.0 (0.0) c c 0.0 (0.0) c c netherlands 89.1 (1.0) 520 (4.0) 8.1 (0.9) 450 (9.7) 2.7 (0.4) 440 (15.8) 10.9 (1.0) 448 (9.5) norway 90.5 (0.9) 510 (3.0) 4.7 (0.6) 467 (17.1) 4.8 (0.5) 446 (8.7) 9.5 (0.9) 457 (10.5) Poland 99.8 (0.1) 482 (4.4) 0.2 (0.1) c c 0.0 (0.0) c c 0.2 (0.1) c c Portugal 93.1 (0.6) 498 (3.6) 3.3 (0.4) 459 (10.5) 3.6 (0.5) 475 (8.0) 6.9 (0.6) 468 (7.7) Slovak republic 99.3 (0.2) 485 (3.5) 0.4 (0.1) c c 0.3 (0.1) c c 0.7 (0.2) 512 (29.8) Slovenia 91.3 (0.5) 481 (1.4) 6.5 (0.4) 453 (5.5) 2.2 (0.2) 383 (13.9) 8.7 (0.5) 435 (6.0) Spain 89.6 (0.8) 482 (4.0) 1.4 (0.2) 458 (15.2) 9.0 (0.7) 440 (6.9) 10.4 (0.8) 443 (7.1) Sweden 85.1 (0.9) 501 (3.2) 8.7 (0.6) 461 (5.8) 6.2 (0.5) 417 (9.1) 14.9 (0.9) 443 (5.1) turkey 99.1 (0.2) 455 (4.0) 0.7 (0.2) 489 (28.6) 0.2 (0.1) c c 0.9 (0.2) 466 (25.1) England (united kingdom) 85.7 (1.3) 523 (4.0) 6.4 (0.6) 474 (8.5) 7.9 (1.0) 503 (10.3) 14.3 (1.3) 490 (7.8) united States 78.4 (2.0) 512 (3.8) 14.8 (1.4) 503 (6.9) 6.8 (0.8) 487 (11.4) 21.6 (2.0) 498 (7.1) oEcd average 90.2 (0.2) 505 (0.7) 5.6 (0.1) 475 (2.4) 4.2 (0.1) 458 (2.2) 9.8 (0.2) 469 (2.2) brazil 99.3 (0.2) 431 (4.7) 0.4 (0.2) c c 0.3 (0.1) c c 0.7 (0.2) 409 (18.7) bulgaria 99.5 (0.2) 405 (5.0) 0.4 (0.2) c c 0.2 (0.1) c c 0.5 (0.2) c c colombia 99.7 (0.1) 400 (3.5) 0.2 (0.0) c c 0.1 (0.1) c c 0.3 (0.1) 322 (24.3) croatia 87.9 (0.8) 467 (4.0) 8.4 (0.5) 458 (6.0) 3.7 (0.4) 469 (8.5) 12.1 (0.8) 461 (5.4) cyprus* 91.5 (0.4) 447 (1.5) 1.8 (0.2) 457 (10.4) 6.7 (0.3) 429 (6.2) 8.5 (0.4) 435 (5.2) hong kong-china 65.3 (1.5) 545 (4.7) 20.5 (0.8) 544 (3.7) 14.2 (1.0) 519 (5.1) 34.7 (1.5) 534 (3.7) macao-china 34.9 (0.6) 538 (1.8) 49.7 (0.7) 545 (1.7) 15.4 (0.4) 535 (3.0) 65.1 (0.6) 543 (1.4) malaysia 98.3 (0.3) 424 (3.5) 1.7 (0.3) 417 (8.6) 0.1 (0.0) c c 1.7 (0.3) 415 (8.4) montenegro 94.2 (0.4) 406 (1.2) 2.7 (0.2) 439 (9.6) 3.1 (0.3) 412 (8.7) 5.8 (0.4) 425 (6.9) russian federation 89.1 (0.8) 490 (3.6) 7.7 (0.6) 485 (5.9) 3.2 (0.4) 476 (8.7) 10.9 (0.8) 482 (5.5) Serbia 91.5 (0.8) 474 (3.2) 6.6 (0.6) 480 (7.1) 1.9 (0.3) 473 (14.5) 8.5 (0.8) 478 (7.1) Shanghai-china 99.1 (0.2) 538 (3.2) 0.3 (0.1) c c 0.6 (0.1) 437 (13.8) 0.9 (0.2) 428 (12.7) Singapore 81.7 (0.8) 561 (1.4) 5.9 (0.3) 592 (5.4) 12.4 (0.7) 567 (4.3) 18.3 (0.8) 575 (3.2) chinese taipei 99.5 (0.1) 535 (2.9) 0.4 (0.1) c c 0.1 (0.0) c c 0.5 (0.1) 534 (15.4) united arab Emirates 45.2 (1.4) 376 (3.4) 23.2 (0.7) 424 (3.8) 31.6 (1.0) 459 (3.7) 54.8 (1.4) 444 (3.2) uruguay 99.5 (0.1) 405 (3.4) 0.2 (0.1) c c 0.3 (0.1) c c 0.5 (0.1) c c Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). This table was calculated considering all students with information on their immigrant status (students with missing data on the PISA index of economic, social and cultural status included). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 208 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.19 [Part 2/2] Performance in problem solving and immigrant background Results based on students’ self-reports difference in problem-solving performance OECD first-generation Second-generation first-generation immigrant students immigrant students immigrant students minus minus minus second-generation non-immigrant non-immigrant immigrant students students students increased likelihood increased likelihood of students with of students with an immigrant an immigrant background scoring background scoring at or above level 5 below level 2 (above 618.21 (less than 423.42 score points) score points) S.E. Score dif. S.E. relative risk S.E. relative risk S.E. 7 (3.1) 10 (3.0) 0.99 (0.07) 1.20 (0.09) (8.6) -55 (5.8) -32 (5.1) 2.24 (0.26) 0.30 (0.09) (8.7) -76 (5.8) -56 (4.9) 2.50 (0.20) 0.30 (0.05) 2 (5.6) -12 (5.1) -9 (4.8) 1.32 (0.12) 0.93 (0.09) (15.2) c c 0 (14.6) -9 (13.7) 1.00 (0.21) 1.31 (0.62) -28 (12.0) 6 (22.7) -30 (11.5) -22 (11.0) 1.51 (0.27) 0.70 (0.21) (8.5) -80 (7.5) -12 (9.7) -72 (6.7) -51 (5.8) 2.66 (0.24) 0.30 (0.07) -30 (7.2) c c c c -33 (7.1) -33 (6.8) 1.77 (0.22) 0.61 (0.15) finland -65 (6.1) -100 (7.9) -35 (10.4) -85 (5.1) -65 (4.4) 3.28 (0.24) 0.30 (0.07) france -59 (8.6) -91 (10.5) -32 (12.7) -69 (7.2) -48 (6.8) 2.81 (0.35) 0.27 (0.08) Germany -48 (6.7) -60 (10.4) -12 (11.9) -50 (5.9) -24 (5.4) 2.09 (0.21) 0.39 (0.08) hungary 23 (14.4) c c c c 19 (13.7) 0 (14.4) 0.73 (0.22) 1.24 (0.50) ireland -8 (14.2) -14 (6.0) -7 (15.5) -13 (5.5) -15 (5.2) 1.18 (0.13) 0.69 (0.16) israel 28 (8.3) 8 (11.4) -20 (10.8) 22 (7.8) 32 (6.9) 0.79 (0.08) 1.12 (0.17) italy -21 (9.5) -63 (9.8) -42 (12.3) -52 (8.4) -42 (8.4) 2.51 (0.31) 0.69 (0.16) Japan c c c c c c c c c c c c c c korea c c c c c c c c c c c c c c netherlands -70 (9.1) -80 (14.7) -10 (15.9) -73 (8.4) -52 (9.1) 2.62 (0.29) 0.31 (0.09) norway -43 (16.7) -63 (8.7) -20 (16.2) -53 (10.1) -37 (10.1) 2.02 (0.22) 0.59 (0.16) c c c c c c c c c c c c c c -38 (10.4) -23 (7.8) 16 (11.0) -30 (7.5) -25 (8.5) 1.62 (0.20) 0.71 (0.22) Score dif. S.E. S.E. Score dif. S.E. 13 (4.6) 0 (3.9) -14 (5.9) austria -51 belgium -84 (5.6) -62 (9.2) -12 (6.9) -67 (7.5) 17 canada -13 (5.7) -11 (5.9) c c 6 czech republic -34 (20.3) denmark -69 Estonia australia chile Poland Portugal Slovak republic Score dif. Score dif. c c c c c c 27 (29.7) 26 (24.0) 1.03 (0.37) 2.57 (1.16) Slovenia -28 (5.6) -98 (13.9) -69 (14.1) -46 (6.1) -21 (5.7) 1.75 (0.13) 0.60 (0.19) Spain -24 (14.7) -41 (6.3) -17 (13.6) -39 (6.4) -25 (6.2) 1.59 (0.13) 0.58 (0.15) Sweden -40 (5.8) -84 (9.6) -44 (11.0) -58 (5.4) -43 (5.4) 2.09 (0.18) 0.29 (0.08) turkey 34 (28.6) c c c c 11 (25.2) 4 (22.0) 1.06 (0.27) 4.28 (3.09) -49 (8.4) -20 (9.8) 29 (11.7) -33 (7.5) -28 (6.2) 1.80 (0.23) 0.63 (0.14) -9 (6.6) -25 (11.2) -16 (10.8) -14 (6.7) 9 (5.9) 1.32 (0.18) 0.86 (0.15) -30 (2.4) -47 (2.2) -15 (2.8) -32 (2.2) -22 (2.0) 1.77 (0.05) 0.87 (0.14) brazil c c c c c c -22 (18.1) -39 (19.1) 1.29 (0.23) 1.13 (2.50) bulgaria c c c c c c c c c c c c c c colombia c c c c c c -78 (24.2) -75 (22.1) 1.33 (0.18) 0.00 c croatia -9 (6.0) 2 (8.2) 11 (9.3) -6 (5.2) 3 (4.9) 1.04 (0.11) 0.56 (0.17) cyprus* 10 (10.6) -18 (6.1) -28 (12.3) -12 (5.3) -6 (5.0) 1.18 (0.08) 1.20 (0.32) hong kong-china -1 (4.2) -26 (5.7) -25 (4.7) -11 (4.3) 3 (3.8) 1.10 (0.15) 0.81 (0.07) 7 (2.7) -2 (3.6) -9 (3.5) 5 (2.5) 8 (2.6) 0.79 (0.09) 1.03 (0.08) malaysia -7 (8.6) c c c c -9 (8.4) 11 (8.9) 1.11 (0.14) 0.24 (0.81) montenegro 33 (9.7) 6 (8.9) -28 (11.8) 18 (7.1) 14 (6.7) 0.83 (0.08) 0.97 (1.11) russian federation -5 (5.3) -14 (8.6) -9 (9.1) -8 (5.0) -5 (4.5) 1.16 (0.13) 0.93 (0.18) Serbia 6 (7.0) -1 (14.1) -7 (14.4) 5 (6.9) 4 (6.3) 0.93 (0.11) 1.53 (0.31) Shanghai-china c c -101 (13.6) c c -110 (12.7) -86 (13.4) 4.12 (0.92) 0.06 (0.13) 31 (5.8) 6 (4.6) -25 (7.4) 14 (3.7) -1 (3.9) 0.68 (0.11) 1.20 (0.08) c c c c c c -1 (14.9) 16 (14.0) 0.80 (0.66) 0.66 (0.46) 48 (4.3) 84 (4.4) 36 (4.0) 69 (3.9) 65 (3.8) 0.60 (0.02) 9.30 (3.05) c c c c c c c c c c c c c c England (united kingdom) united States oEcd average Partners Students with an immigrant background minus non-immigrant students Students with an immigrant background minus non-immigrant students, after accounting for students’ socio-economic status macao-china Singapore chinese taipei united arab Emirates uruguay Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). This table was calculated considering all students with information on their immigrant status (students with missing data on the PISA index of economic, social and cultural status included). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 209 Annex b1: reSulTS For counTrIeS And economIeS table v.4.20 [Part 1/3] differences in problem-solving, mathematics, reading and science performance related to immigrant background Results based on students’ self-reports Score-point difference related to immigrant background: immigrant minus non-immigrant students Problem solving OECD Score dif. australia reading S.E. Score dif. S.E. (3.1) 26 (3.5) Score dif. 19 computer-based mathematics Science Score dif. S.E. Score dif. S.E. Score dif. S.E. (3.0) 11 (3.5) 22 (3.5) 18 (3.4) austria -55 (5.8) -60 (5.2) -51 (5.8) -70 (5.0) -48 (5.9) -62 (6.8) -76 (5.8) -76 (5.1) -66 (5.8) -76 (5.3) -57 (4.7) -71 (6.4) canada -12 (5.1) -2 (4.5) 3 (4.2) -10 (4.7) 7 (5.5) 1 (4.0) 0 (14.6) -1 (13.3) 9 (14.2) 2 (13.1) 17 (13.5) 32 (15.5) czech republic -30 (11.5) -28 (11.5) -20 (10.4) -38 (10.7) m m m m denmark -72 (6.7) -67 (3.5) -59 (3.5) -80 (3.8) -62 (5.3) -61 (3.6) (7.4) Estonia -33 (7.1) -30 (5.8) -35 (5.2) -32 (5.9) -41 (5.5) -45 finland -85 (5.1) -86 (4.9) -93 (5.1) -106 (5.4) m m m m france -69 (7.2) -67 (6.9) -67 (8.5) -77 (8.6) -58 (7.1) -54 (8.4) Germany -50 (5.9) -56 (5.9) -49 (5.7) -66 (6.1) -40 (6.2) -44 (6.0) hungary 19 (13.7) 32 (13.1) 16 (14.0) 24 (11.5) 4 (13.1) 16 (16.9) ireland -13 (5.5) -3 (4.7) -11 (4.9) -2 (5.0) 1 (5.1) -11 (5.8) israel 22 (7.8) 7 (5.7) 8 (6.2) 10 (6.4) 7 (6.4) 14 (6.7) italy -52 (8.4) -49 (7.4) -64 (8.9) -52 (7.6) -53 (5.9) -42 (8.4) Japan c c c c c c c c c c c c korea c c c c c c c c c c c c netherlands -73 (8.4) -58 (7.0) -56 (7.8) -68 (6.8) m m m m norway -53 (10.1) -47 (6.7) -50 (6.5) -69 (7.4) -35 (6.9) -65 (8.6) c c c c c c c c c c c c -30 (7.5) -44 (7.1) -38 (7.8) -44 (7.5) -35 (6.0) -45 (6.4) (25.4) Poland Portugal Slovak republic 27 (29.7) 6 (21.1) 7 (20.3) -10 (21.5) 31 (19.0) 2 Slovenia -46 (6.1) -52 (5.2) -46 (4.8) -58 (4.6) -40 (4.9) -43 (5.5) Spain -39 (6.4) -57 (5.1) -53 (4.9) -52 (5.7) -64 (4.8) -57 (6.7) Sweden -58 (5.4) -60 (5.1) -63 (5.8) -72 (5.6) -41 (4.3) -54 (5.6) turkey 11 (25.2) 3 (31.1) -12 (26.9) -17 (27.5) m m m m -33 (7.5) -15 (8.4) -13 (8.1) -26 (8.0) m m m m England (united kingdom) united States -14 (6.7) -13 (5.8) -7 (5.2) -26 (5.8) -16 (6.2) -19 (6.6) oEcd average -32 (2.2) -32 (2.0) -32 (2.0) -40 (1.9) -25 (1.8) -30 (2.2) brazil -22 (18.1) -78 (16.1) -84 (22.8) -78 (17.4) -99 (23.7) -86 (20.9) c c c c c c c c m m m m -78 (24.2) -69 (13.0) -92 (21.7) -81 (16.2) -89 (13.9) -122 (25.8) croatia -6 (5.2) -19 (5.2) -19 (6.4) -23 (5.7) m m m m cyprus* -12 (5.3) -21 (5.0) -10 (5.3) -16 (5.2) m m m m hong kong-china -11 (4.3) -7 (4.4) 0 (4.3) -6 (3.8) -7 (4.0) -6 (4.4) (2.1) bulgaria colombia macao-china 5 (2.5) 16 (2.8) 22 (2.2) 16 (2.3) 14 (2.8) 15 malaysia -9 (8.4) -21 (8.9) 2 (11.8) -15 (9.6) m m m m montenegro 18 (7.1) 21 (6.5) 3 (7.1) 24 (6.2) m m m m russian federation -8 (5.0) -22 (4.5) -29 (4.7) -30 (4.8) -20 (3.9) -8 (5.5) 5 (6.9) 15 (6.2) 24 (6.8) 13 (6.7) m m m m -110 (12.7) -126 (14.6) -90 (13.8) -109 (12.7) -92 (11.1) -123 (14.4) Serbia Shanghai-china Singapore 14 (3.7) 26 (4.3) 18 (4.1) 22 (3.9) 21 (4.3) -3 (3.2) chinese taipei -1 (14.9) -32 (23.1) -17 (15.3) -14 (14.6) -56 (15.3) -27 (17.1) united arab Emirates 69 (3.9) 66 (3.1) 63 (3.1) 66 (3.2) 54 (3.3) 79 (4.4) c c c c c c c c m m m m uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 210 digital reading S.E. belgium chile Partners 7 mathematics © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.20 [Part 2/3] differences in problem-solving, mathematics, reading and science performance related to immigrant background Results based on students’ self-reports immigrant effect size: Performance difference related to immigrant background divided by the variation in scores within each country/economy (standard deviation) OECD Problem solving australia reading computer-based mathematics Science digital reading Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. 0.07 (0.03) 0.27 (0.04) 0.20 (0.03) 0.11 (0.03) 0.24 (0.04) 0.19 (0.03) austria -0.59 (0.06) -0.65 (0.05) -0.55 (0.06) -0.76 (0.05) -0.55 (0.06) -0.60 (0.06) belgium -0.72 (0.05) -0.75 (0.05) -0.66 (0.06) -0.77 (0.05) -0.59 (0.05) -0.72 (0.06) canada -0.12 (0.05) -0.03 (0.05) 0.03 (0.05) -0.11 (0.05) 0.08 (0.06) 0.02 (0.05) 0.00 (0.17) -0.02 (0.16) 0.11 (0.18) 0.03 (0.16) 0.21 (0.16) 0.39 (0.19) chile czech republic -0.32 (0.12) -0.30 (0.12) -0.23 (0.12) -0.42 (0.12) m m m m denmark -0.79 (0.07) -0.83 (0.04) -0.70 (0.05) -0.88 (0.04) -0.72 (0.06) -0.74 (0.05) (0.08) Estonia -0.38 (0.08) -0.37 (0.07) -0.44 (0.07) -0.41 (0.07) -0.50 (0.07) -0.49 finland -0.91 (0.05) -1.02 (0.06) -1.00 (0.06) -1.16 (0.06) m m m m france -0.72 (0.08) -0.70 (0.07) -0.62 (0.08) -0.77 (0.08) -0.63 (0.07) -0.56 (0.09) Germany -0.52 (0.06) -0.58 (0.06) -0.54 (0.06) -0.69 (0.06) -0.42 (0.06) -0.45 (0.06) hungary 0.19 (0.13) 0.34 (0.14) 0.17 (0.15) 0.27 (0.13) 0.04 (0.14) 0.14 (0.15) ireland -0.14 (0.06) -0.04 (0.06) -0.13 (0.06) -0.03 (0.06) 0.01 (0.06) -0.14 (0.07) israel 0.18 (0.06) 0.07 (0.05) 0.07 (0.06) 0.09 (0.06) 0.07 (0.06) 0.12 (0.06) italy (0.09) -0.57 (0.09) -0.54 (0.08) -0.67 (0.09) -0.55 (0.08) -0.64 (0.07) -0.44 Japan c c c c c c c c c c c c korea c c c c c c c c c c c c netherlands -0.74 (0.08) -0.64 (0.07) -0.61 (0.08) -0.73 (0.07) m m m m norway -0.52 (0.10) -0.53 (0.07) -0.51 (0.07) -0.71 (0.07) -0.41 (0.08) -0.66 (0.08) Poland Portugal Slovak republic Slovenia c c c c c c c c c c c c -0.35 (0.08) -0.48 (0.08) -0.42 (0.09) -0.50 (0.08) -0.42 (0.07) -0.51 (0.07) 0.27 (0.31) 0.06 (0.21) 0.07 (0.20) -0.10 (0.21) 0.36 (0.22) 0.02 (0.27) -0.47 (0.06) -0.57 (0.06) -0.51 (0.05) -0.64 (0.05) -0.46 (0.06) -0.44 (0.06) Spain -0.37 (0.06) -0.66 (0.06) -0.59 (0.05) -0.62 (0.06) -0.78 (0.06) -0.58 (0.07) Sweden -0.61 (0.06) -0.66 (0.06) -0.61 (0.06) -0.74 (0.06) -0.48 (0.05) -0.56 (0.06) turkey England (united kingdom) Partners mathematics 0.14 (0.32) 0.04 (0.34) -0.14 (0.31) -0.21 (0.35) m m m m -0.34 (0.08) -0.16 (0.09) -0.14 (0.08) -0.27 (0.08) m m m m united States -0.15 (0.07) -0.15 (0.06) -0.08 (0.06) -0.27 (0.06) -0.19 (0.07) -0.21 (0.08) oEcd average -0.34 (0.02) -0.36 (0.02) -0.34 (0.02) -0.43 (0.02) -0.29 (0.02) -0.31 (0.02) brazil -0.24 (0.20) -0.98 (0.20) -0.99 (0.26) -1.00 (0.22) -1.18 (0.27) -0.94 (0.22) c c c c c c c c m m m m -0.86 (0.26) -0.93 (0.17) -1.10 (0.26) -1.06 (0.21) -1.22 (0.19) -1.34 (0.28) bulgaria colombia croatia -0.06 (0.06) -0.21 (0.06) -0.22 (0.07) -0.27 (0.07) m m m m cyprus* -0.12 (0.05) -0.23 (0.05) -0.09 (0.05) -0.17 (0.05) m m m m hong kong-china -0.12 (0.05) -0.08 (0.05) -0.01 (0.05) -0.07 (0.05) -0.09 (0.05) -0.07 (0.05) 0.06 (0.03) 0.17 (0.03) 0.27 (0.03) 0.21 (0.03) 0.17 (0.03) 0.21 (0.03) -0.11 (0.10) -0.26 (0.11) 0.03 (0.14) -0.19 (0.12) m m m m 0.20 (0.08) 0.26 (0.08) 0.03 (0.08) 0.29 (0.07) m m m m -0.09 (0.06) -0.25 (0.05) -0.32 (0.05) -0.36 (0.06) -0.26 (0.05) -0.09 (0.06) 0.05 (0.08) 0.16 (0.07) 0.26 (0.07) 0.15 (0.08) m m m m -1.23 (0.14) -1.25 (0.14) -1.13 (0.17) -1.34 (0.15) -0.99 (0.12) -1.48 (0.16) macao-china malaysia montenegro russian federation Serbia Shanghai-china Singapore chinese taipei united arab Emirates uruguay 0.15 (0.04) 0.25 (0.04) 0.18 (0.04) 0.22 (0.04) 0.22 (0.04) -0.03 (0.04) -0.01 (0.16) -0.28 (0.20) -0.19 (0.17) -0.17 (0.18) -0.64 (0.17) -0.31 (0.19) 0.66 (0.03) 0.74 (0.03) 0.67 (0.03) 0.71 (0.03) 0.65 (0.03) 0.72 (0.04) c c c c c c c c m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 211 Annex b1: reSulTS For counTrIeS And economIeS table v.4.20 [Part 3/3] differences in problem-solving, mathematics, reading and science performance related to immigrant background Results based on students’ self-reports difference in immigrant effect sizes between problem solving (PS) and… OECD … mathematics (PS - m) … computer-based mathematics (PS - cbm) … Science (PS - S) S.E. Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. -0.20 (0.02) -0.12 (0.02) -0.04 (0.02) -0.16 (0.02) -0.11 (0.02) austria 0.06 (0.04) -0.03 (0.05) 0.18 (0.04) -0.04 (0.05) 0.01 (0.06) belgium 0.03 (0.03) -0.06 (0.04) 0.05 (0.04) -0.14 (0.04) 0.00 (0.04) canada -0.09 (0.03) -0.15 (0.03) -0.01 (0.03) -0.20 (0.04) -0.14 (0.04) 0.02 (0.11) -0.11 (0.11) -0.03 (0.13) -0.21 (0.13) -0.39 (0.19) -0.02 (0.05) -0.09 (0.08) 0.10 (0.06) m m m m denmark 0.04 (0.05) -0.09 (0.08) 0.09 (0.06) -0.07 (0.04) -0.04 (0.06) Estonia 0.00 (0.05) 0.07 (0.06) 0.03 (0.06) 0.12 (0.06) 0.11 (0.07) finland 0.10 (0.04) 0.08 (0.04) 0.25 (0.04) m m m m france -0.03 (0.05) -0.10 (0.06) 0.05 (0.06) -0.10 (0.04) -0.17 (0.05) Germany 0.07 (0.04) 0.02 (0.04) 0.18 (0.04) -0.10 (0.04) -0.07 (0.04) hungary -0.15 (0.09) 0.01 (0.12) -0.08 (0.09) 0.14 (0.14) 0.05 (0.11) ireland -0.11 (0.04) -0.01 (0.05) -0.12 (0.04) -0.15 (0.05) -0.01 (0.05) israel 0.12 (0.04) 0.11 (0.04) 0.09 (0.04) 0.12 (0.05) 0.06 (0.05) italy -0.04 (0.06) 0.10 (0.07) -0.02 (0.05) 0.06 (0.06) -0.13 (0.07) Japan c c c c c c c c c c korea c c c c c c c c c c -0.11 (0.08) -0.13 (0.09) -0.01 (0.07) m m m m 0.01 (0.05) -0.01 (0.07) 0.19 (0.06) -0.11 (0.08) 0.14 (0.07) c c c c c c c c c c Portugal 0.13 (0.05) 0.08 (0.06) 0.16 (0.07) 0.07 (0.05) 0.17 (0.07) Slovak republic 0.22 (0.28) 0.21 (0.24) 0.38 (0.28) -0.08 (0.20) 0.26 (0.16) Slovenia 0.09 (0.05) 0.03 (0.05) 0.16 (0.04) -0.02 (0.04) -0.03 (0.04) Spain 0.29 (0.05) 0.22 (0.06) 0.25 (0.05) 0.40 (0.07) 0.21 (0.06) Sweden 0.05 (0.05) 0.00 (0.05) 0.13 (0.05) -0.13 (0.05) -0.05 (0.05) turkey 0.10 (0.09) 0.28 (0.14) 0.35 (0.14) m m m m England (united kingdom) -0.18 (0.05) -0.21 (0.05) -0.08 (0.05) m m m m united States -0.01 (0.04) -0.07 (0.05) 0.12 (0.05) 0.03 (0.05) 0.06 (0.06) oEcd average 0.02 (0.02) 0.00 (0.02) 0.09 (0.02) -0.03 (0.02) 0.00 (0.02) brazil 0.74 (0.15) 0.75 (0.20) 0.75 (0.15) 0.94 (0.23) 0.70 (0.22) c c c c c c m m m m colombia 0.08 (0.26) 0.24 (0.29) 0.20 (0.26) 0.36 (0.27) 0.48 (0.28) croatia 0.15 (0.04) 0.16 (0.05) 0.21 (0.04) m m m m cyprus* 0.10 (0.04) -0.03 (0.04) 0.05 (0.04) m m m m hong kong-china -0.04 (0.03) -0.11 (0.03) -0.05 (0.03) -0.03 (0.03) -0.05 (0.04) macao-china -0.10 (0.03) -0.21 (0.02) -0.14 (0.03) -0.11 (0.03) -0.15 (0.03) 0.15 (0.09) -0.14 (0.12) 0.08 (0.11) m m m m -0.06 (0.04) 0.17 (0.06) -0.09 (0.06) m m m m 0.16 (0.05) 0.23 (0.05) 0.26 (0.06) 0.16 (0.04) 0.00 (0.04) -0.11 (0.04) -0.21 (0.05) -0.10 (0.05) m m m m 0.02 (0.12) -0.10 (0.16) 0.10 (0.13) -0.25 (0.11) 0.24 (0.16) -0.10 (0.02) -0.02 (0.03) -0.07 (0.02) -0.07 (0.03) 0.18 (0.03) 0.26 (0.13) 0.18 (0.13) 0.15 (0.14) 0.63 (0.11) 0.29 (0.14) -0.08 (0.03) -0.01 (0.03) -0.05 (0.03) 0.01 (0.02) -0.06 (0.03) c c c c c c m m m m czech republic netherlands norway Poland bulgaria malaysia montenegro russian federation Serbia Shanghai-china Singapore chinese taipei united arab Emirates uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 212 … digital reading (PS - dr) Effect size dif. australia chile Partners … reading (PS - r) © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.21 [Part 1/1] relative performance in problem solving, by immigrant background Results based on students’ self-reports Problem-solving performance among immigrant students, compared to that of non-immigrant students with similar performance in mathematics, reading and science OECD average average Percentage difference in difference in of immigrant problem solving students who problem solving compared with compared with outperform non-immigrant non-immigrant non-immigrant students students students with similar with similar with similar performance performance performance in mathematics1 in mathematics2 in reading1 average difference in problem solving compared with non-immigrant students with similar performance in science1 Percentage of immigrant students who outperform non-immigrant students with similar performance in science2 Percentage average of immigrant difference in problem solving students who outperform compared with non-immigrant non-immigrant students students with similar with similar performance performance in mathematics, in mathematics, reading reading and science2 and science3 Score dif. S.E. % S.E. Score dif. S.E. % S.E. Score dif. S.E. % S.E. Score dif. S.E. % S.E. -14 (2.0) 40.1 (1.8) -8 (2.2) 45.7 (1.7) -1 (2.2) 49.7 (1.8) -12 (2.0) 41.7 (1.7) austria -7 (3.9) 44.3 (3.2) -16 (4.3) 38.9 (3.3) 0 (4.5) 50.1 (3.4) -7 (4.0) 43.3 (4.0) belgium -13 (4.1) 42.4 (2.9) -23 (4.3) 38.2 (2.5) -13 (4.2) 43.3 (2.6) -11 (4.0) 43.6 (2.6) canada -10 (2.9) 44.1 (2.2) -14 (3.2) 41.4 (2.3) -4 (3.0) 48.1 (2.1) -9 (2.8) 44.3 (2.2) 2 (9.1) 52.0 (10.3) -7 (8.5) 48.2 (8.1) -1 (10.3) 49.0 (9.8) 0 (8.5) 52.6 (11.0) australia chile czech republic -4 (4.7) 49.5 (5.6) -13 (7.1) 43.2 (6.9) 2 (6.0) 51.7 (7.5) -2 (4.8) 50.1 (7.5) -15 (5.5) 40.2 (3.6) -30 (7.3) 33.6 (3.4) -17 (6.3) 39.9 (3.8) -14 (5.9) 40.9 (3.9) Estonia -6 (4.4) 45.2 (4.0) -3 (4.9) 48.8 (4.4) -4 (4.8) 46.8 (4.9) -2 (4.3) 48.7 (4.6) finland -6 (4.4) 46.8 (4.4) -17 (3.5) 40.2 (3.6) 0 (3.9) 51.4 (3.6) 0 (4.1) 51.7 (3.8) france -15 (5.3) 42.2 (4.0) -26 (5.3) 34.5 (3.4) -11 (5.6) 45.0 (3.8) -11 (5.5) 44.6 (4.3) -3 (3.4) 49.1 (3.2) -10 (3.7) 46.1 (3.0) 5 (3.9) 55.2 (3.6) 1 (3.4) 52.7 (3.5) denmark Germany hungary -10 (8.5) 43.7 (8.1) 5 (10.6) 53.4 (8.1) -4 (8.8) 46.7 (10.0) -7 (8.4) 43.7 (9.9) ireland -11 (3.7) 40.5 (2.9) -4 (4.2) 47.0 (3.0) -11 (3.9) 41.3 (3.3) -10 (3.6) 41.3 (3.9) israel 15 (4.6) 61.1 (2.9) 15 (4.9) 59.0 (3.5) 12 (4.9) 58.4 (3.5) 14 (4.5) 60.0 (3.3) italy -16 (5.8) 42.7 (3.9) -12 (6.5) 43.1 (4.4) -17 (5.5) 43.0 (3.9) -13 (5.9) 44.4 (4.3) Japan c c c c c c c c c c c c c c c c korea c c c c c c c c c c c c c c c c netherlands -22 (8.2) 38.9 (4.6) -27 (8.7) 36.2 (4.4) -13 (7.7) 42.8 (5.6) -15 (7.7) 42.3 (5.0) norway (4.8) -11 (6.4) 42.5 (4.5) -17 (7.6) 41.9 (4.0) 1 (7.2) 51.4 (4.2) -7 (6.8) 45.3 Poland c c c c c c c c c c c c c c c c Portugal 4 (4.0) 55.5 (3.9) -4 (4.6) 48.5 (4.2) 3 (5.2) 54.4 (4.3) 4 (4.1) 56.6 (4.5) Slovak republic 22 (26.7) 52.7 (13.7) 22 (23.6) 59.1 (12.0) 35 (26.4) 62.6 (13.7) 23 (25.8) 52.8 (14.0) Slovenia -1 (4.5) 53.6 (3.7) -10 (4.6) 44.2 (4.8) 3 (4.2) 55.8 (3.8) 2 (4.2) 54.7 (3.8) Spain 15 (4.8) 60.9 (3.1) 4 (6.0) 54.4 (4.2) 10 (4.9) 55.3 (3.3) 16 (4.5) 61.6 (3.0) Sweden -7 (4.7) 44.6 (3.5) -18 (4.6) 39.5 (3.2) -5 (4.7) 46.6 (3.3) -4 (4.9) 46.0 (3.6) turkey 10 (7.1) 63.5 (9.3) 18 (11.5) 59.9 (10.2) 23 (10.6) 67.0 (11.0) 13 (7.4) 63.1 (9.1) -19 (4.0) 35.6 (4.8) -23 (4.4) 35.2 (3.0) -12 (4.5) 42.0 (4.7) -17 (3.8) 37.6 (5.0) -2 (4.0) 48.4 (3.7) -8 (4.6) 43.6 (3.5) 7 (4.4) 55.9 (3.7) -1 (4.2) 49.4 (3.4) England (united kingdom) united States Partners Percentage of immigrant students who outperform non-immigrant students with similar performance in reading2 oEcd average -5 (1.5) 47.2 (1.1) -9 (1.5) 45.0 (1.0) 0 (1.6) 50.1 (1.2) -3 (1.5) 48.5 (1.2) brazil 55 (14.4) 82.4 (13.9) 40 (14.8) 75.6 (11.2) 43 (11.5) 80.0 (10.2) 59 (13.3) 84.7 (12.4) bulgaria c c c c c c c c c c c c c c c c -17 (22.1) 42.3 (13.6) -16 (22.1) 42.7 (13.5) -17 (21.4) 43.0 (13.5) -10 (22.0) 45.5 (13.5) croatia 11 (3.1) 59.4 (3.2) 10 (3.7) 55.9 (3.2) 15 (3.0) 60.7 (2.7) 13 (3.1) 60.7 (3.0) cyprus* 6 (4.0) 55.2 (3.2) -6 (3.8) 46.7 (3.3) 1 (3.5) 51.5 (3.7) 4 (3.7) 53.0 (3.0) hong kong-china -6 (2.6) 47.9 (2.2) -10 (2.7) 44.6 (1.9) -6 (2.8) 46.8 (2.2) -7 (2.8) 46.8 (2.4) macao-china -6 (2.0) 45.7 (1.6) -10 (1.9) 43.6 (1.5) -7 (2.0) 45.6 (1.7) -7 (1.9) 44.6 (1.5) 9 (7.0) 57.4 (8.6) -11 (8.0) 41.0 (7.5) 3 (8.0) 50.5 (8.9) 7 (7.1) 54.9 (7.8) colombia malaysia montenegro russian federation Serbia Shanghai-china -1 (3.9) 49.9 (3.9) 17 (4.9) 62.1 (3.8) -2 (5.2) 48.8 (5.0) -1 (4.0) 49.1 (3.8) 8 (4.0) 55.5 (3.3) 10 (4.2) 56.2 (2.7) 12 (4.7) 58.2 (3.1) 9 (3.9) 56.6 (3.6) -7 (3.8) 43.4 (3.9) -12 (4.2) 43.8 (4.0) -6 (4.4) 46.7 (4.3) -8 (3.8) 42.3 (3.6) -16 (9.9) 39.5 (11.8) -30 (12.6) 30.6 (8.8) -15 (11.0) 40.6 (10.8) -15 (10.8) 40.0 (10.3) Singapore -5 (1.9) 46.0 (2.0) 2 (2.5) 51.1 (1.8) -2 (2.3) 48.9 (1.9) -6 (2.0) 45.6 (2.1) chinese taipei 20 (10.3) 66.4 (11.8) 13 (10.7) 60.6 (11.1) 11 (11.6) 58.7 (10.8) 18 (10.1) 66.7 (12.4) united arab Emirates 11 (3.5) 58.3 (2.4) 19 (3.2) 61.1 (2.0) 13 (2.8) 59.3 (1.9) 8 (3.0) 56.3 (2.3) c c c c c c c c c c c c c c c c uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. This column reports the difference between actual performance and the itted value from a regression using a cubic polynomial as regression function. 2. This column reports the percentage of students for whom the difference between actual performance and the itted value from a regression is positive. Values that are indicated in bold are signiicantly larger or smaller than 50%. 3. This column reports the difference between actual performance and the itted value from a regression using a second-degree polynomial as regression function (math, math sq., read, read sq., scie, scie sq., math×read, math×scie, read×scie). * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 213 Annex b1: reSulTS For counTrIeS And economIeS table v.4.22a [Part 1/1] Performance on problem-solving tasks, by nature of problem and by immigrant background Results based on students’ self-reports items referring to a static problem situation relative likelihood of success, in favour of immigrant students (non-immigrant students = 1.00) OECD average proportion of full-credit responses Partners items referring to an interactive problem situation relative likelihood of success, in favour of immigrant students (non-immigrant students = 1.00) average proportion of full-credit responses nonimmigrant students difference between based immigrant and accounting on success immigrant non-immigrant for booklet on remaining effects1 students students test items2 % S.E. % S.E. australia 53.1 (0.5) 54.1 (1.1) 1.0 (1.2) 1.05 (0.05) austria 50.2 (1.0) 41.0 (2.6) -9.2 (2.7) belgium 51.2 (0.8) 35.3 (1.9) -16.0 (2.3) canada 54.6 (0.7) 49.4 (1.8) -5.3 (2.1) chile 35.0 (0.9) c c c c czech republic 46.4 (0.7) 39.2 (4.0) -7.2 (4.2) denmark 49.0 (1.0) 37.8 (1.9) -11.2 Estonia 50.4 (0.8) 44.6 (3.2) -5.8 finland 52.7 (0.6) 37.3 france 51.9 (0.8) Germany 51.8 hungary nonimmigrant students difference between based immigrant and accounting on success immigrant non-immigrant for booklet on remaining effects1 students test items2 students % S.E. % S.E. 1.00 (0.04) 50.3 (0.5) 51.4 (1.0) 1.0 (1.0) 1.05 (0.04) 1.00 (0.04) 0.66 (0.07) 1.05 (0.10) 45.0 (0.8) 34.7 (1.8) -10.3 (1.9) 0.63 (0.05) 0.95 (0.09) 0.51 (0.05) 1.02 (0.09) 48.3 (0.6) 32.1 (1.7) -16.1 (1.9) 0.50 (0.04) 0.98 (0.09) 0.81 (0.07) 0.90 (0.06) 51.7 (0.7) 49.1 (1.6) -2.7 (1.8) 0.90 (0.07) 1.11 (0.08) c 31.8 (0.8) c c c c 0.74 (0.13) 0.96 (0.14) 44.7 (0.7) 38.6 (2.3) -6.1 (2.4) 0.77 (0.08) 1.04 (0.15) (2.0) 0.61 (0.05) 1.07 (0.09) 43.7 (0.8) 31.3 (1.5) -12.4 (1.6) 0.57 (0.04) 0.93 (0.08) (3.3) 0.79 (0.10) 0.94 (0.13) 45.9 (0.9) 41.8 (2.9) -4.1 (3.1) 0.84 (0.10) 1.07 (0.15) (2.2) -15.4 (2.3) 0.54 (0.05) 1.08 (0.09) 48.3 (0.6) 31.5 (2.3) -16.8 (2.3) 0.50 (0.05) 0.93 (0.08) 41.0 (2.9) -10.8 (3.0) 0.63 (0.08) 1.07 (0.12) 49.6 (0.8) 37.6 (2.3) -12.1 (2.4) 0.59 (0.06) 0.94 (0.11) (0.9) 47.9 (3.1) -4.0 (3.2) 0.80 (0.10) 1.32 (0.17) 49.1 (0.9) 38.2 (2.7) -10.9 (2.8) 0.60 (0.07) 0.76 (0.10) 38.2 (1.1) c c c c c 33.8 (0.9) c c c c ireland 44.8 (1.0) 43.2 (2.7) -1.7 (3.0) 0.93 (0.11) 1.11 (0.15) 45.2 (1.0) 40.8 (2.4) -4.4 (2.8) israel 40.3 (1.6) 39.9 (2.2) -0.4 (2.3) 1.01 (0.10) 0.88 (0.07) 35.6 (1.4) 38.3 (2.3) 2.7 (2.3) 1.15 (0.12) 1.14 (0.09) italy 51.0 (1.1) 34.7 (3.1) -16.3 (3.4) 0.51 (0.08) 0.70 (0.08) 47.6 (1.0) 39.9 (2.3) -7.7 (2.3) 0.72 (0.08) 1.43 (0.16) Japan 58.8 (0.8) c c c c c c c c 56.0 (0.7) c c c c c c c c korea 59.1 (1.0) c c c c c c c c 57.9 (1.0) c c c c c c c c netherlands 52.2 (1.0) 37.6 (3.2) -14.6 (2.8) 0.55 (0.07) 1.03 (0.10) 48.3 (1.0) 33.5 (3.3) -14.8 (3.0) 0.54 (0.07) 0.98 (0.09) norway 50.5 (1.0) 44.9 (3.2) -5.5 (3.5) 0.76 (0.10) 1.56 (0.21) (3.2) -15.4 (3.4) 0.49 (0.07) 0.64 (0.09) Poland 44.2 (1.0) c c c c Portugal 44.7 (1.0) 40.6 (3.2) -4.1 (3.3) Slovak republic 44.7 (1.0) c c c c Slovenia 44.3 (0.9) 29.0 (3.1) -15.3 (3.6) 0.55 (0.09) Spain 43.9 (0.8) 31.8 (2.8) -12.1 (2.9) 0.62 (0.08) Sweden 49.4 (1.0) 40.6 (2.0) -8.8 (2.3) 0.71 (0.06) turkey 36.0 (0.9) c c c c England (united kingdom) 50.6 (1.0) 43.5 (3.1) -7.1 (3.3) united States 48.3 (1.2) 41.0 (2.4) -7.3 oEcd average 48.1 (0.2) 40.7 (0.6) -8.4 brazil 30.5 (1.0) c c c c c c c bulgaria 28.8 (0.9) c c c c c c c colombia 26.5 (0.9) c c c c c c c croatia 39.3 (1.0) 39.2 (1.9) -0.1 (1.9) % dif. S.E. odds ratio c c S.E. c c odds ratio c c S.E. % dif. S.E. odds ratio c c S.E. c c odds ratio c c S.E. c c 0.84 (0.09) 0.90 (0.12) 46.2 (1.0) 30.8 c 39.8 (1.1) c c c c 1.07 (0.16) 42.8 (1.0) 37.5 (2.6) -5.3 (2.6) c 39.0 (0.8) c c c c 0.85 (0.15) 37.9 (0.9) 27.2 (2.0) -10.7 (2.2) 0.64 (0.07) 1.18 (0.21) 0.95 (0.10) 41.4 (0.8) 30.9 (2.1) -10.5 (2.4) 0.65 (0.07) 1.06 (0.11) 0.95 (0.10) 43.2 (0.9) 36.0 (1.8) -7.2 (2.1) 0.75 (0.07) 1.05 (0.11) c 32.8 (0.9) c c c c 0.74 (0.10) 0.98 (0.07) 49.0 (1.1) 42.4 (3.0) -6.6 (3.1) 0.76 (0.10) 1.02 (0.08) (2.7) 0.76 (0.08) 0.79 (0.10) 46.5 (1.2) 45.2 (2.3) -1.3 (2.6) 0.96 (0.10) 1.26 (0.15) (0.6) 0.70 (0.02) 1.00 (0.02) 44.7 (0.2) 37.6 (0.5) -8.2 (0.5) 0.70 (0.02) 1.00 (0.02) c 29.7 (1.0) c c c c c c c c c 22.8 (0.8) c c c c c c c c c 23.9 (0.7) c c c c c c c c 1.14 (0.09) 36.1 (0.9) 33.0 (1.4) -3.1 (1.5) 0.87 (0.06) 0.87 (0.07) c c 0.82 (0.11) c c c c 1.00 (0.08) c c c c c c c 0.77 (0.09) 0.94 (0.14) c c c c c c c c cyprus* 37.5 (0.5) 33.3 (1.5) -4.3 (1.6) 0.83 (0.06) 0.89 (0.06) 31.9 (0.5) 30.3 (1.5) -1.6 (1.6) 0.93 (0.07) 1.12 (0.07) hong kong-china 56.8 (1.3) 56.0 (1.3) -0.8 (2.0) 0.96 (0.08) 1.04 (0.10) 53.3 (1.1) 51.7 (1.0) -1.6 (1.6) 0.92 (0.06) 0.96 (0.09) macao-china 57.9 (1.3) 56.7 (0.9) -1.2 (1.8) 0.95 (0.07) 0.94 (0.07) 51.4 (1.0) 51.9 (0.8) 0.5 (1.4) 1.01 (0.05) 1.06 (0.07) malaysia 30.4 (0.8) c c c c c 27.7 (0.8) c c c c montenegro 30.2 (0.6) 32.4 (2.6) 2.2 (2.8) 1.11 (0.14) 0.93 (0.11) 25.0 (0.4) 28.5 (2.0) 3.5 (2.1) 1.19 (0.13) 1.08 (0.13) russian federation 44.1 (0.9) 41.2 (2.8) -2.9 (2.8) 0.91 (0.11) 1.05 (0.13) 40.1 (0.9) 36.2 (1.8) -3.8 (2.0) 0.87 (0.08) 0.95 (0.12) Serbia 40.4 (0.8) 41.5 (2.7) 1.1 (2.6) 1.05 (0.11) 1.06 (0.08) 37.0 (0.8) 36.6 (2.2) -0.4 (2.2) 0.99 (0.09) 0.94 (0.07) Shanghai-china 56.9 (1.0) c c c c c 50.7 (0.9) c c c c Singapore 59.8 (0.9) 62.8 (2.2) 3.1 (2.6) 0.97 (0.10) 57.1 (0.8) 60.7 (1.9) 3.7 (2.2) chinese taipei 56.6 (0.9) c c c c c 50.4 (0.8) c c c c united arab Emirates 23.8 (0.8) 35.6 (0.9) 11.8 (1.4) 0.81 (0.06) 19.0 (0.9) 34.1 (0.7) 15.1 (1.1) uruguay 27.8 (0.7) c c c c 25.2 (0.6) c c c c c 1.13 (0.13) c c 1.78 (0.12) c c c c c c c c c c c c c c c c c c 1.16 (0.10) 1.03 (0.10) c c c c 2.21 (0.14) 1.24 (0.09) c c c c Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an immigrant dummy, and an interaction term (immigrant × item type). Booklet dummies are added to the estimation. This column presents the difference between the logit coeficient on the interaction term and the logit coeficient on the item type dummy in exponentiated form. 2. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an immigrant dummy, and an interaction term (immigrant × item type). Booklet dummies are added to the estimation. This column presents the logit coeficient on the interaction term in exponentiated form. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 214 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.22b [Part 1/2] Performance on problem-solving tasks, by process and by immigrant background Results based on students’ self-reports items assessing the process of “exploring and understanding” relative likelihood of success, in favour of immigrant students (non-immigrant students = 1.00) OECD average proportion of full-credit responses Partners items assessing the process of “representing and formulating” relative likelihood of success, in favour of immigrant students (non-immigrant students = 1.00) average proportion of full-credit responses nonimmigrant students difference between based immigrant and accounting on success immigrant non-immigrant for booklet on remaining effects1 students students test items2 % S.E. % S.E. australia 54.9 (0.6) 57.6 (1.2) 2.7 (1.3) 1.14 (0.06) austria 51.1 (1.2) 41.5 (2.6) -9.6 (2.9) belgium 52.3 (0.8) 34.2 (2.0) -18.1 (2.3) canada 55.6 (0.8) 51.9 (1.8) -3.6 (2.1) chile 32.6 (1.0) c c c c czech republic 47.3 (0.9) 39.9 (3.8) -7.3 (4.1) denmark 47.4 (1.1) 34.3 (1.8) -13.1 Estonia 49.7 (1.1) 39.7 (3.6) -9.9 finland 54.4 (0.6) 36.9 france 54.0 (1.0) Germany 54.0 hungary nonimmigrant students difference between based immigrant and accounting on success immigrant non-immigrant for booklet on remaining effects1 students test items2 students % S.E. % S.E. 1.11 (0.04) 49.9 (0.7) 51.5 (1.2) 1.7 (1.3) 1.09 (0.06) 1.04 (0.04) 0.65 (0.08) 1.03 (0.09) 44.7 (1.0) 28.7 (2.5) -16.0 (2.5) 0.48 (0.06) 0.69 (0.07) 0.47 (0.05) 0.91 (0.06) 48.1 (0.9) 28.8 (1.8) -19.3 (2.1) 0.43 (0.04) 0.82 (0.06) 0.87 (0.08) 0.99 (0.06) 52.5 (1.0) 49.0 (1.8) -3.5 (2.1) 0.87 (0.08) 1.00 (0.05) c 29.3 (0.9) c c c c 0.73 (0.13) 0.96 (0.14) 43.2 (0.9) 36.7 (3.4) -6.4 (3.7) 0.76 (0.12) 0.99 (0.12) (2.1) 0.56 (0.05) 0.94 (0.07) 43.4 (1.2) 31.0 (2.1) -12.4 (2.3) 0.57 (0.06) 0.96 (0.09) (3.7) 0.67 (0.11) 0.75 (0.10) 44.5 (1.1) 44.9 (3.8) 0.5 (4.0) 1.02 (0.16) 1.31 (0.16) (2.4) -17.5 (2.4) 0.50 (0.05) 0.96 (0.09) 46.9 (0.7) 29.8 (3.1) -17.1 (3.2) 0.49 (0.07) 0.93 (0.10) 42.0 (2.9) -12.0 (3.1) 0.60 (0.08) 0.99 (0.10) 49.1 (0.9) 36.3 (2.9) -12.8 (3.0) 0.57 (0.08) 0.93 (0.10) (1.2) 43.5 (3.1) -10.5 (3.2) 0.61 (0.08) 0.89 (0.09) 46.7 (1.2) 36.9 (3.1) -9.8 (3.4) 0.63 (0.09) 0.93 (0.11) 37.7 (1.1) c c c c c 32.5 (1.1) c c c c ireland 48.5 (1.4) 41.1 (3.2) -7.5 (3.7) 0.73 (0.12) 0.80 (0.10) 41.4 (1.1) 42.9 (2.9) 1.5 (3.3) israel 42.1 (1.7) 43.9 (2.4) 1.8 (2.7) 1.11 (0.13) 1.02 (0.08) 35.4 (1.5) 37.0 (3.1) 1.5 (3.0) 1.10 (0.15) 1.00 (0.10) italy 52.9 (1.3) 38.4 (3.8) -14.5 (3.9) 0.54 (0.10) 0.81 (0.13) 48.0 (1.3) 39.4 (3.4) -8.6 (3.3) 0.70 (0.11) 1.12 (0.15) Japan 62.3 (0.9) c c c c c c c c 55.9 (0.9) c c c c c c c c korea 64.9 (1.1) c c c c c c c c 60.9 (1.3) c c c c c c c c netherlands 53.5 (1.1) 38.7 (3.6) -14.8 (3.2) 0.55 (0.08) 1.01 (0.09) 46.2 (1.2) 29.2 (3.8) -17.1 (3.5) 0.48 (0.08) 0.85 (0.08) norway 53.1 (1.2) 38.8 (3.2) -14.3 (3.7) 0.52 (0.08) 0.88 (0.12) (3.8) -13.3 (4.0) 0.53 (0.10) 0.90 (0.13) Poland 43.9 (1.2) c c c c Portugal 44.5 (1.4) 39.8 (3.2) -4.6 (3.5) Slovak republic 44.1 (1.1) c c c c Slovenia 41.1 (1.1) 26.9 (2.7) -14.1 (3.1) 0.55 (0.09) Spain 44.1 (1.1) 30.5 (3.1) -13.6 (3.2) 0.57 (0.09) Sweden 49.9 (1.1) 42.2 (2.5) -7.7 (2.7) 0.75 (0.09) turkey 33.7 (1.0) c c c c England (united kingdom) 52.5 (1.3) 45.1 (3.4) -7.3 (3.5) united States 50.6 (1.2) 44.0 (3.2) -6.6 oEcd average 49.0 (0.2) 40.5 (0.6) -9.6 brazil 31.0 (1.1) c c c c c c c bulgaria 28.3 (0.9) c c c c c c c colombia 24.9 (0.9) c c c c c c c croatia 37.6 (1.0) 35.4 (1.8) -2.2 (1.8) % dif. S.E. odds ratio c c S.E. c c odds ratio c c S.E. % dif. S.E. odds ratio c c S.E. c c odds ratio c c S.E. c c 1.08 (0.14) 1.31 (0.17) 45.0 (1.3) 31.8 c 38.7 (1.3) c c c c 1.02 (0.19) 40.3 (1.3) 35.6 (4.0) -4.7 (4.0) c 37.3 (1.1) c c c c 0.89 (0.14) 37.2 (1.0) 23.5 (2.7) -13.8 (2.9) 0.54 (0.09) 0.85 (0.13) 0.86 (0.11) 38.8 (1.0) 28.3 (2.5) -10.4 (2.7) 0.64 (0.08) 0.99 (0.11) 1.02 (0.12) 44.1 (1.2) 33.1 (2.4) -10.9 (2.7) 0.63 (0.08) 0.83 (0.08) c 32.0 (1.1) c c c c 0.73 (0.10) 0.97 (0.06) 48.7 (1.3) 41.5 (3.5) -7.2 (3.5) 0.74 (0.11) 0.97 (0.07) (3.4) 0.77 (0.10) 0.84 (0.10) 44.1 (1.6) 44.9 (2.4) 0.8 (2.9) 1.03 (0.12) 1.22 (0.09) (0.7) 0.67 (0.02) 0.93 (0.02) 43.7 (0.2) 36.2 (0.6) -8.4 (0.7) 0.69 (0.02) 0.97 (0.02) c 26.2 (1.2) c c c c c c c c c 19.6 (0.9) c c c c c c c c c 18.8 (0.8) c c c c c c c c 0.99 (0.07) 33.6 (1.2) 29.1 (1.8) -4.5 (1.6) 0.81 (0.06) 0.86 (0.06) c c 0.80 (0.13) c c c c 0.91 (0.07) c c c c c c c 0.78 (0.13) 0.98 (0.16) c c c c c c c c cyprus* 36.7 (0.5) 34.3 (1.9) -2.4 (1.9) 0.91 (0.08) 1.02 (0.06) 31.2 (0.6) 28.2 (1.9) -3.0 (2.0) 0.87 (0.09) 0.96 (0.07) hong kong-china 61.9 (1.6) 58.7 (1.3) -3.2 (2.0) 0.86 (0.08) 0.89 (0.07) 56.1 (1.3) 54.2 (1.5) -1.9 (2.0) 0.91 (0.08) 0.96 (0.07) macao-china 59.0 (1.3) 59.9 (1.2) 0.9 (1.8) 1.03 (0.07) 1.06 (0.07) 56.5 (1.5) 57.9 (1.1) 1.4 (1.9) 1.06 (0.08) 1.09 (0.07) malaysia 30.4 (0.9) c c c c c 28.2 (1.0) c c c c montenegro 27.2 (0.6) 30.5 (2.7) 3.3 (2.8) 1.18 (0.16) 1.02 (0.11) 23.7 (0.6) 24.2 (2.4) 0.5 (2.5) 1.03 (0.14) 0.86 (0.08) russian federation 42.3 (1.1) 39.1 (2.7) -3.2 (3.0) 0.90 (0.13) 1.02 (0.15) 38.6 (1.2) 37.0 (2.7) -1.5 (2.7) 0.95 (0.12) 1.10 (0.12) Serbia 39.5 (1.0) 41.5 (2.4) 2.0 (2.5) 1.09 (0.11) 1.11 (0.08) 35.9 (0.9) 35.9 (2.6) 0.0 (2.5) 1.00 (0.11) 1.00 (0.07) Shanghai-china 58.5 (1.1) c c c c c 55.8 (1.2) c c c c Singapore 64.2 (1.1) 66.7 (2.3) 2.6 (2.5) 0.95 (0.08) 58.8 (1.0) 65.5 (2.3) 6.7 (2.6) chinese taipei 58.6 (1.0) c c c c c 55.7 (1.2) c c c c united arab Emirates 22.4 (0.9) 36.7 (1.0) 14.2 (1.4) 0.99 (0.06) 20.1 (1.1) 32.6 (1.0) 12.5 (1.4) uruguay 27.4 (0.7) c c c c 22.5 (0.8) c c c c c 1.11 (0.13) c c 2.02 (0.15) c c c c c c c c c c c c c c c c c c 1.33 (0.15) 1.20 (0.10) c c c c 1.94 (0.15) 0.94 (0.06) c c c c Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an immigrant dummy, and an interaction term (immigrant × item type). Booklet dummies are added to the estimation. This column presents the difference between the logit coeficient on the interaction term and the logit coeficient on the item type dummy in exponentiated form. 2. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an immigrant dummy, and an interaction term (immigrant × item type). Booklet dummies are added to the estimation. This column presents the logit coeficient on the interaction term in exponentiated form. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 215 Annex b1: reSulTS For counTrIeS And economIeS table v.4.22b [Part 2/2] Performance on problem-solving tasks, by process and by immigrant background Results based on students’ self-reports items assessing the process of “planning and executing” relative likelihood of success, in favour of immigrant students (non-immigrant students = 1.00) OECD average proportion of full-credit responses Partners items assessing the process of “monitoring and relecting” relative likelihood of success, in favour of immigrant students (non-immigrant students = 1.00) average proportion of full-credit responses nonimmigrant students difference between based immigrant and accounting on success immigrant non-immigrant for booklet on remaining effects1 students students test items2 % S.E. % S.E. australia 52.1 (0.5) 51.7 (1.0) -0.4 (1.0) 1.00 (0.04) austria 49.2 (0.9) 40.1 (2.1) -9.1 (2.3) belgium 50.2 (0.6) 36.1 (1.8) -14.1 (2.0) canada 54.0 (0.6) 49.0 (1.7) -5.0 (1.8) chile 35.2 (0.8) c c c c czech republic 47.2 (0.7) 39.7 (2.8) -7.5 (2.9) denmark 49.5 (0.9) 36.4 (1.5) -13.1 Estonia 49.9 (0.9) 46.2 (2.4) -3.7 finland 51.7 (0.6) 35.9 france 51.1 (0.8) Germany 52.2 hungary nonimmigrant students difference between based immigrant and accounting on success immigrant non-immigrant for booklet on remaining effects1 students test items2 students % S.E. % S.E. 0.91 (0.03) 46.3 (0.5) 46.9 (1.1) 0.6 (1.2) 1.03 (0.05) 0.97 (0.04) 0.66 (0.06) 1.06 (0.09) 38.2 (0.9) 33.9 (2.4) -4.3 (2.4) 0.80 (0.09) 1.31 (0.16) 0.55 (0.05) 1.17 (0.07) 44.9 (0.8) 31.1 (1.8) -13.8 (2.0) 0.54 (0.05) 1.10 (0.09) 0.82 (0.06) 0.91 (0.05) 46.5 (0.8) 46.0 (1.9) -0.5 (2.1) 0.99 (0.08) 1.17 (0.07) c 33.0 (0.8) c c c c 0.73 (0.09) 0.94 (0.09) 40.8 (0.7) 37.9 (3.7) -2.9 (3.8) 0.88 (0.14) 1.19 (0.19) (1.6) 0.56 (0.04) 0.94 (0.06) 36.9 (1.0) 29.9 (1.6) -7.0 (1.9) 0.71 (0.07) 1.27 (0.11) (2.6) 0.86 (0.09) 1.07 (0.09) 43.0 (0.9) 36.8 (3.0) -6.2 (3.2) 0.77 (0.10) 0.92 (0.10) (2.0) -15.8 (2.0) 0.53 (0.04) 1.05 (0.07) 43.2 (0.6) 28.4 (2.9) -14.7 (2.9) 0.53 (0.07) 1.04 (0.10) 39.8 (2.4) -11.3 (2.6) 0.62 (0.06) 1.03 (0.08) 45.6 (0.9) 35.1 (2.4) -10.5 (2.6) 0.63 (0.07) 1.05 (0.10) (0.8) 44.5 (2.9) -7.6 (3.0) 0.69 (0.09) 1.04 (0.11) 44.0 (1.1) 38.8 (2.3) -5.3 (2.4) 0.75 (0.07) 1.16 (0.12) 37.6 (0.9) c c c c c 30.9 (1.1) c c c c ireland 46.0 (0.9) 43.1 (2.5) -2.9 (2.7) 0.89 (0.09) 1.03 (0.11) 42.8 (1.1) 37.6 (2.8) -5.2 (3.1) israel 37.4 (1.5) 38.5 (2.2) 1.1 (2.2) 1.07 (0.10) 0.96 (0.07) 32.5 (1.4) 34.9 (2.3) 2.4 (2.5) 1.13 (0.13) 1.04 (0.09) italy 49.3 (1.0) 35.1 (2.7) -14.2 (2.9) 0.55 (0.07) 0.80 (0.10) 43.0 (1.0) 42.3 (3.7) -0.6 (3.9) 0.98 (0.17) 1.68 (0.28) Japan 56.4 (0.7) c c c c c c c c 52.1 (0.7) c c c c c c c c korea 54.6 (0.9) c c c c c c c c 53.9 (1.1) c c c c c c c c netherlands 51.6 (1.0) 36.7 (2.8) -14.9 (2.5) 0.54 (0.06) 1.00 (0.09) 44.2 (1.0) 33.0 (3.6) -11.2 (3.3) 0.62 (0.09) 1.17 (0.10) norway 49.5 (1.0) 39.3 (3.3) -10.1 (3.4) 0.62 (0.09) 1.13 (0.12) 0.61 (0.11) 1.07 (0.13) Poland 43.8 (1.0) c c c c Portugal 46.4 (1.0) 40.3 (3.3) -6.2 (3.3) Slovak republic 43.5 (0.9) c c c c Slovenia 43.4 (0.8) 32.2 (2.2) -11.2 (2.5) 0.66 (0.07) Spain 43.8 (0.9) 33.9 (2.4) -9.9 (2.6) 0.68 (0.08) Sweden 46.4 (0.8) 37.5 (1.6) -9.0 (1.9) 0.70 (0.05) turkey 36.1 (0.9) c c c c England (united kingdom) 50.3 (1.1) 42.8 (2.8) -7.5 (2.9) united States 48.3 (1.2) 43.7 (1.8) -4.6 oEcd average 47.4 (0.2) 40.1 (0.5) -8.4 brazil 32.6 (1.1) c c c c c c c bulgaria 27.1 (0.8) c c c c c c c colombia 28.0 (0.8) c c c c c c c croatia 40.5 (0.9) 40.3 (1.7) -0.2 (1.6) % dif. S.E. odds ratio c c S.E. c c odds ratio c c S.E. % dif. S.E. odds ratio c c S.E. c c odds ratio c c S.E. c c 0.81 (0.11) 0.92 (0.13) 39.5 (1.2) 29.7 (3.6) -9.8 (3.7) c 35.6 (1.1) c c c c 0.93 (0.12) 39.4 (1.1) 36.9 (3.7) -2.5 (3.7) c 36.0 (0.9) c c c c 1.15 (0.12) 35.3 (0.8) 25.4 (2.4) -9.9 (2.6) 0.66 (0.09) 1.11 (0.11) 1.11 (0.09) 40.6 (1.0) 30.0 (3.1) -10.6 (3.3) 0.65 (0.10) 1.03 (0.12) 0.93 (0.08) 38.5 (1.0) 37.1 (2.4) -1.3 (2.8) 0.96 (0.12) 1.38 (0.14) c 31.6 (1.0) c c c c 0.73 (0.09) 0.95 (0.06) 44.8 (0.9) 41.3 (3.4) -3.5 (3.5) 0.85 (0.12) 1.17 (0.11) (2.1) 0.85 (0.07) 0.94 (0.07) 43.6 (1.3) 41.9 (2.4) -1.7 (2.5) 0.95 (0.10) 1.10 (0.08) (0.5) 0.70 (0.02) 1.00 (0.02) 40.9 (0.2) 35.9 (0.6) -5.6 (0.6) 0.78 (0.02) 1.13 (0.03) c 27.5 (0.9) c c c c c c c c c 22.1 (0.9) c c c c c c c c c 25.1 (0.8) c c c c c c c c 1.14 (0.05) 33.8 (0.9) 31.2 (1.6) -2.6 (1.8) 0.89 (0.07) 0.96 (0.06) c c 0.75 (0.09) c c c c 0.99 (0.07) c c c c c c c 0.88 (0.15) 1.13 (0.18) c c c c c c c c cyprus* 35.2 (0.6) 33.3 (1.4) -1.9 (1.5) 0.93 (0.06) 1.06 (0.06) 30.3 (0.6) 26.6 (1.4) -3.7 (1.6) 0.84 (0.07) 0.93 (0.05) hong kong-china 51.7 (1.1) 51.4 (1.0) -0.3 (1.5) 0.98 (0.06) 1.08 (0.06) 48.7 (1.4) 48.5 (1.4) -0.2 (1.9) 0.98 (0.08) 1.06 (0.08) macao-china 52.2 (1.1) 50.9 (0.7) -1.3 (1.4) 0.95 (0.05) 0.93 (0.05) 46.4 (1.3) 45.5 (1.2) -1.0 (1.9) 0.95 (0.07) 0.95 (0.06) malaysia 29.6 (0.8) c c c c c 24.9 (0.8) c c c c montenegro 29.9 (0.6) 33.4 (2.1) 3.4 (2.2) 1.17 (0.12) 1.02 (0.09) 23.4 (0.6) 28.2 (2.5) 4.8 (2.6) 1.29 (0.17) 1.13 (0.13) russian federation 44.3 (0.8) 40.4 (2.5) -3.9 (2.4) 0.87 (0.09) 0.98 (0.11) 37.8 (1.1) 32.2 (2.6) -5.6 (3.0) 0.80 (0.11) 0.89 (0.11) Serbia 41.0 (0.8) 40.4 (2.4) -0.6 (2.3) 0.98 (0.09) 0.96 (0.06) 33.2 (0.9) 31.9 (2.7) -1.3 (2.7) 0.95 (0.12) 0.93 (0.08) Shanghai-china 50.1 (0.7) c c c c c 47.6 (1.1) c c c c Singapore 55.1 (0.9) 57.9 (1.9) 2.8 (2.3) 0.95 (0.07) 55.2 (0.9) 57.4 (2.3) 2.3 (2.5) chinese taipei 50.5 (0.8) c c c c c 45.1 (1.0) c c c c united arab Emirates 21.3 (0.8) 35.6 (0.8) 14.3 (1.1) 1.01 (0.05) 18.1 (0.9) 32.2 (1.0) 14.1 (1.5) uruguay 28.2 (0.7) c c c c 24.1 (0.7) c c c c c 1.12 (0.10) c c 2.04 (0.12) c c c c c c c c c c c c c c c c c c 1.10 (0.12) 0.95 (0.09) c c c c 2.16 (0.19) 1.08 (0.07) c c c c Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an immigrant dummy, and an interaction term (immigrant × item type). Booklet dummies are added to the estimation. This column presents the difference between the logit coeficient on the interaction term and the logit coeficient on the item type dummy in exponentiated form. 2. generalised odds ratios estimated with logistic regression on national PISA samples. A success indicator for each item is regressed on an item type dummy, an immigrant dummy, and an interaction term (immigrant × item type). Booklet dummies are added to the estimation. This column presents the logit coeficient on the interaction term in exponentiated form. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 216 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.23 [Part 1/1] Association between problem-solving performance and perseverance/openness to problem solving Results based on students’ self-reports Score-point difference that is associated with students’ perseverance, by performance decile in problem solving 10th percentile1 mean OECD Score dif. Score dif. S.E. 23 (1.4) 20 (2.4) austria 10 (2.2) 9 belgium 13 (2.1) 9 canada 20 (1.3) chile 14 90th percentile1 Score dif. S.E. 10th percentile2 mean Score dif. S.E. Score dif. S.E. 90th percentile2 Score dif. S.E. 22 (2.9) 31 (1.3) 25 (3.1) 37 (2.3) (6.3) 9 (4.5) 26 (1.9) 19 (4.2) 30 (3.7) (4.7) 17 (3.2) 26 (2.0) 19 (4.6) 31 (2.4) 20 (2.5) 18 (2.4) 33 (1.2) 29 (2.9) 34 (2.6) (1.7) 15 (3.1) 13 (3.6) 19 (1.7) 13 (3.5) 24 (3.2) 9 (2.4) 8 (5.8) 9 (4.4) 31 (2.2) 23 (6.1) 36 (5.3) 17 (2.0) 13 (4.5) 18 (4.5) 26 (2.5) 20 (4.3) 29 (4.0) Estonia 1 (2.0) 0 (4.2) 0 (3.6) 27 (2.0) 17 (4.7) 34 (2.9) finland 30 (1.6) 28 (3.1) 31 (3.2) 37 (1.6) 32 (3.4) 41 (3.0) france 18 (2.0) 11 (4.6) 22 (2.3) 22 (1.9) 12 (4.5) 29 (2.8) Germany 13 (2.5) 4 (4.4) 16 (4.3) 19 (2.1) 9 (5.0) 24 (4.5) hungary 14 (2.7) 11 (7.8) 15 (4.2) 24 (3.3) 17 (7.7) 22 (4.8) ireland 23 (2.1) 21 (4.4) 27 (3.6) 30 (1.7) 20 (4.0) 38 (3.6) israel 1 (1.8) 8 (4.1) 0 (4.2) 12 (2.5) 5 (5.7) 24 (4.2) italy 0 (2.1) 0 (5.3) 1 (3.7) 13 (2.7) 8 (5.9) 18 (3.9) Japan 14 (2.5) 13 (3.7) 16 (3.2) 23 (2.3) 22 (3.9) 23 (2.6) korea 20 (2.9) 21 (5.1) 19 (5.4) 37 (2.3) 39 (4.0) 29 (4.1) 6 (2.5) 6 (4.2) 10 (5.6) 19 (2.3) 13 (4.5) 29 (5.7) norway 22 (1.9) 21 (4.7) 23 (2.8) 26 (1.8) 21 (3.1) 29 (3.0) Poland 20 (2.0) 19 (3.7) 19 (4.3) 20 (1.9) 18 (4.3) 20 (4.4) Portugal 21 (1.9) 20 (2.9) 20 (3.2) 25 (2.0) 15 (3.6) 33 (3.9) Slovak republic 12 (2.0) 1 (6.9) 16 (4.3) 19 (2.4) 9 (5.0) 26 (4.7) 7 (2.3) 7 (4.9) 7 (5.5) 25 (2.4) 18 (3.7) 35 (5.3) Spain 16 (2.3) 15 (5.3) 19 (2.9) 25 (2.0) 19 (5.0) 34 (4.3) Sweden 25 (2.1) 20 (5.3) 28 (3.5) 27 (1.9) 15 (4.2) 33 (2.9) turkey 10 (1.7) 9 (2.7) 11 (3.5) 14 (2.0) 9 (3.4) 25 (3.3) England (united kingdom) 20 (2.0) 19 (4.8) 19 (3.8) 34 (2.2) 30 (4.9) 39 (4.0) united States 19 (1.8) 15 (3.3) 23 (4.6) 26 (1.7) 15 (3.4) 35 (3.3) oEcd average 15 (0.4) 13 (0.9) 16 (0.7) 25 (0.4) 18 (0.9) 30 (0.7) brazil 18 (1.9) 16 (3.9) 17 (5.1) 16 (2.7) 5 (4.1) 22 (5.3) bulgaria 17 (2.1) 19 (3.4) 12 (3.7) 8 (2.3) 2 (4.2) 14 (4.3) colombia 9 (1.8) 7 (3.9) 11 (4.2) 8 (2.1) 3 (4.3) 17 (3.9) croatia 6 (1.6) 10 (2.7) 2 (2.9) 16 (2.2) 6 (4.1) 29 (5.0) cyprus* 20 (2.3) 20 (4.9) 20 (3.5) 23 (1.9) 17 (3.8) 28 (3.3) 7 (2.6) 12 (4.4) 3 (4.7) 22 (2.1) 21 (4.0) 23 (4.5) macao-china 13 (1.9) 14 (4.6) 11 (3.7) 22 (1.5) 23 (3.3) 19 (3.2) malaysia 13 (2.0) 12 (3.5) 14 (3.2) 8 (1.8) -1 (3.4) 19 (4.6) montenegro 13 (1.7) 13 (2.9) 14 (2.8) 1 (2.0) -5 (3.7) 8 (4.0) 6 (1.7) 6 (3.7) 6 (3.2) 20 (2.1) 12 (3.2) 28 (4.2) 10 (1.7) 11 (3.3) 6 (3.4) 12 (2.0) 5 (4.1) 20 (3.8) 9 (2.1) 8 (3.7) 7 (3.9) 26 (2.0) 26 (2.9) 23 (3.3) Singapore 13 (2.1) 13 (3.6) 11 (4.4) 18 (2.0) 12 (4.4) 20 (3.3) chinese taipei 13 (1.7) 10 (4.4) 11 (3.6) 21 (1.7) 17 (3.3) 22 (3.7) united arab Emirates 26 (1.5) 29 (2.4) 22 (3.6) 10 (1.8) 2 (3.3) 19 (3.6) uruguay 13 (2.3) 10 (4.7) 16 (2.7) 14 (2.1) 2 (2.9) 28 (3.6) czech republic denmark netherlands Slovenia Partners S.E. australia Score-point difference that is associated with students’ openness to problem solving, by performance decile in problem solving hong kong-china russian federation Serbia Shanghai-china Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. results based on quantile regression of problem-solving performance on the index of perseverance. 2. results based on quantile regression of problem-solving performance on the index of openness to problem solving. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 217 Annex b1: reSulTS For counTrIeS And economIeS table v.4.24 [Part 1/1] Performance in problem solving and access to a computer at home Results based on students’ self-reports Students who have at least one computer at home to use for school work difference in problem-solving performance Percentage of students Partners OECD all students boys Girls Gender difference (b - G) difference related to parents’ Parents’ highest highest Parents’ occupation: occupation: highest occupation: Semi-skilled or Skilled - semiskilled or elementary Skilled elementary (iSco 1 to 3) (iSco 4 to 9) observed after accounting for sociodemographic characteristics of students1 % S.E. % S.E. % S.E. % dif. S.E. % S.E. % S.E. % dif. S.E. Score dif. S.E. Score dif. australia 97.8 (0.1) 97.3 (0.1) 98.2 (0.1) -0.9 (0.1) 98.6 (0.1) 97.0 (0.1) 1.5 (0.2) 72 (7.2) 36 (6.7) austria 98.6 (0.2) 98.6 (0.3) 98.5 (0.2) 0.1 (0.3) 99.3 (0.1) 97.8 (0.3) 1.5 (0.3) 47 (13.3) 24 (13.9) S.E. belgium 97.0 (0.1) 96.7 (0.2) 97.2 (0.1) -0.5 (0.3) 98.2 (0.1) 96.3 (0.2) 1.9 (0.3) 86 (8.6) 46 (6.5) canada 97.2 (0.1) 97.1 (0.1) 97.4 (0.1) -0.4 (0.2) 98.3 (0.1) 95.9 (0.2) 2.4 (0.2) 48 (7.0) 26 (7.6) chile 86.3 (0.5) 86.2 (0.6) 86.3 (0.5) -0.1 (0.4) 95.5 (0.3) 81.3 (0.6) 14.2 (0.6) 59 (5.9) 24 (4.4) czech republic 97.3 (0.1) 96.9 (0.2) 97.8 (0.1) -0.9 (0.3) 99.4 (0.1) 96.5 (0.2) 2.8 (0.2) 89 (13.5) 31 (12.9) (18.8) denmark 99.0 (0.1) 98.8 (0.1) 99.2 (0.1) -0.4 (0.1) 99.5 (0.1) 98.6 (0.2) 0.9 (0.1) 43 (16.8) 12 Estonia 89.3 (0.3) 91.9 (0.3) 86.8 (0.4) 5.0 (0.4) 90.2 (0.3) 88.5 (0.4) 1.7 (0.4) -9 (5.0) -16 (4.9) finland 98.9 (0.1) 98.6 (0.1) 99.2 (0.1) -0.6 (0.1) 99.3 (0.1) 98.2 (0.2) 1.1 (0.2) 48 (11.4) 25 (11.1) france 96.8 (0.1) 96.6 (0.2) 97.0 (0.2) -0.4 (0.3) 98.2 (0.1) 95.2 (0.3) 3.0 (0.3) 64 (9.7) 33 (9.8) Germany 98.2 (0.1) 97.8 (0.2) 98.7 (0.1) -0.8 (0.2) 99.2 (0.1) 97.6 (0.2) 1.7 (0.3) 97 (13.7) 70 (16.1) (11.4) hungary 94.1 (0.3) 94.6 (0.3) 93.7 (0.4) 0.9 (0.4) 96.7 (0.2) 93.6 (0.3) 3.0 (0.4) 90 (11.6) 36 ireland 95.2 (0.2) 93.5 (0.2) 97.0 (0.2) -3.5 (0.3) 96.0 (0.2) 94.6 (0.3) 1.4 (0.3) 34 (8.5) 18 (8.1) israel 94.3 (0.3) 96.5 (0.3) 92.3 (0.4) 4.2 (0.5) 96.1 (0.3) 92.1 (0.5) 4.1 (0.6) 61 (9.5) 7 (8.5) italy 96.6 (0.1) 96.0 (0.2) 97.4 (0.2) -1.4 (0.3) 97.5 (0.2) 96.3 (0.2) 1.2 (0.3) 26 (8.3) 12 (8.3) Japan 70.1 (0.4) 67.1 (0.5) 73.4 (0.5) -6.3 (0.6) 74.6 (0.5) 66.5 (0.5) 8.1 (0.6) 27 (3.9) 17 (3.4) korea 94.6 (0.2) 93.9 (0.3) 95.5 (0.3) -1.6 (0.4) 95.2 (0.2) 93.9 (0.3) 1.3 (0.3) 31 (7.8) 17 (6.9) netherlands 98.3 (0.1) 98.1 (0.2) 98.5 (0.1) -0.4 (0.2) 98.7 (0.1) 97.5 (0.2) 1.2 (0.2) 74 (19.0) 55 (15.6) (13.5) norway 98.6 (0.1) 98.2 (0.1) 99.0 (0.1) -0.7 (0.2) 99.2 (0.1) 97.6 (0.2) 1.6 (0.2) 71 (15.1) 24 Poland 97.4 (0.2) 97.5 (0.3) 97.3 (0.2) 0.2 (0.3) 98.8 (0.1) 96.6 (0.3) 2.2 (0.4) 67 (7.3) 27 (8.1) Portugal 96.7 (0.2) 96.2 (0.2) 97.3 (0.2) -1.1 (0.3) 98.6 (0.2) 96.0 (0.3) 2.6 (0.3) 64 (9.8) 30 (9.5) Slovak republic 91.9 (0.3) 91.8 (0.4) 91.9 (0.4) -0.1 (0.5) 98.5 (0.2) 91.6 (0.4) 6.9 (0.4) 119 (8.1) 61 (7.0) Slovenia 98.6 (0.1) 98.3 (0.1) 98.9 (0.2) -0.6 (0.2) 98.9 (0.1) 98.6 (0.2) 0.3 (0.2) 70 (12.7) 40 (14.2) Spain 96.1 (0.2) 96.2 (0.3) 96.0 (0.2) 0.2 (0.3) 97.9 (0.1) 94.9 (0.3) 3.0 (0.3) 60 (8.6) 31 (8.1) Sweden 98.7 (0.1) 98.6 (0.1) 98.7 (0.1) -0.1 (0.2) 99.1 (0.1) 98.2 (0.1) 0.9 (0.2) 59 (17.0) 34 (16.6) turkey 68.3 (0.5) 68.5 (0.7) 68.0 (0.6) 0.5 (0.8) 86.7 (0.7) 65.7 (0.5) 21.0 (0.7) 53 (4.3) 28 (3.8) England (united kingdom) 96.8 (0.2) 96.6 (0.4) 97.0 (0.2) -0.4 (0.4) 97.9 (0.2) 96.1 (0.3) 1.8 (0.3) 65 (10.0) 30 (10.8) united States 91.1 (0.3) 89.8 (0.4) 92.5 (0.3) -2.8 (0.4) 95.0 (0.3) 85.6 (0.4) 9.4 (0.5) 42 (6.3) 9 (5.9) oEcd average 94.1 (0.0) 93.8 (0.1) 94.3 (0.1) -0.5 (0.1) 96.5 (0.0) 92.8 (0.1) 3.7 (0.1) 59 (2.0) 28 (2.0) brazil 73.2 (0.6) 74.9 (0.8) 71.6 (0.7) 3.4 (0.8) 90.7 (0.4) 64.9 (0.8) 25.8 (0.7) 66 (5.1) 37 (4.6) bulgaria 93.0 (0.3) 92.7 (0.4) 93.2 (0.6) -0.5 (0.9) 99.0 (0.1) 90.9 (0.3) 8.1 (0.4) 110 (11.6) 42 (10.3) colombia 62.9 (0.7) 62.9 (0.7) 62.9 (0.9) 0.0 (0.9) 84.5 (0.8) 56.5 (0.7) 28.0 (1.0) 53 (4.6) 27 (3.8) croatia 94.2 (0.2) 94.9 (0.2) 93.5 (0.3) 1.4 (0.4) 95.5 (0.2) 93.6 (0.2) 1.9 (0.3) 40 (6.5) 26 (6.1) cyprus* 96.7 (0.1) 95.2 (0.2) 98.2 (0.1) -3.0 (0.2) 98.4 (0.1) 96.1 (0.2) 2.3 (0.2) 73 (8.5) 39 (9.3) hong kong-china 98.8 (0.1) 98.7 (0.1) 98.9 (0.1) -0.2 (0.2) 98.9 (0.1) 98.8 (0.1) 0.1 (0.2) 33 (15.4) 20 (14.6) macao-china 97.1 (0.1) 96.5 (0.2) 97.9 (0.1) -1.4 (0.2) 97.7 (0.2) 97.2 (0.1) 0.6 (0.3) 36 (6.3) 32 (6.4) malaysia 69.6 (0.9) 68.8 (1.0) 70.4 (1.0) -1.6 (0.7) 84.6 (0.8) 59.8 (1.0) 24.8 (0.7) 50 (3.9) 24 (3.7) (5.5) montenegro 91.8 (0.2) 92.6 (0.3) 91.0 (0.3) 1.6 (0.4) 96.5 (0.2) 89.4 (0.3) 7.0 (0.4) 46 (5.0) 14 russian federation 93.0 (0.3) 92.9 (0.4) 93.2 (0.4) -0.3 (0.5) 96.9 (0.3) 89.0 (0.4) 7.9 (0.4) 44 (5.9) 7 (7.1) Serbia 95.4 (0.2) 95.9 (0.2) 94.9 (0.3) 1.0 (0.3) 98.8 (0.1) 93.2 (0.3) 5.6 (0.3) 74 (7.4) 39 (6.7) Shanghai-china 83.3 (0.5) 81.2 (0.7) 85.3 (0.5) -4.0 (0.7) 87.4 (0.3) 78.0 (1.1) 9.5 (1.2) 42 (6.6) 17 (4.1) Singapore 94.6 (0.2) 94.2 (0.2) 95.1 (0.2) -0.9 (0.3) 96.1 (0.2) 91.6 (0.4) 4.5 (0.4) 61 (6.2) 32 (6.4) chinese taipei 90.6 (0.2) 89.3 (0.4) 91.9 (0.3) -2.5 (0.6) 93.5 (0.2) 89.2 (0.4) 4.3 (0.5) 45 (6.4) 26 (5.9) united arab Emirates 92.9 (0.2) 91.7 (0.2) 94.1 (0.2) -2.5 (0.3) 94.5 (0.2) 91.2 (0.3) 3.4 (0.3) 59 (4.7) 28 (5.1) uruguay 88.9 (0.2) 89.8 (0.4) 88.2 (0.4) 1.6 (0.6) 97.2 (0.2) 86.3 (0.3) 10.9 (0.4) 51 (5.1) 12 (4.7) Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. The difference in problem-solving performance after accounting for socio-demographic characteristics of students corresponds to the coeficient from a regression where the PISA index of economic, social and cultural status (ESCS), ESCS squared, boy, and an immigrant (irst generation) dummy are introduced as further independent variables. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 218 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.25 [Part 1/1] Performance in problem solving and use of a computer at home Results based on students’ self-reports Students who use a desktop, laptop or tablet computer at home difference in problem-solving performance Percentage of students OECD all students Girls after accounting for sociodemographic characteristics of students1 observed % S.E. % S.E. % S.E. % dif. S.E. % S.E. % S.E. % dif. S.E. Score dif. australia 97.1 (0.1) 96.7 (0.1) 97.5 (0.1) -0.8 (0.2) 98.2 (0.1) 95.6 (0.2) 2.6 (0.2) 75 (5.9) 50 (6.4) austria 98.7 (0.1) 98.7 (0.2) 98.8 (0.1) -0.1 (0.2) 99.3 (0.1) 98.2 (0.2) 1.1 (0.2) 72 (18.8) 50 (20.0) belgium 98.2 (0.1) 98.1 (0.2) 98.4 (0.1) -0.3 (0.2) 98.9 (0.1) 97.6 (0.2) 1.2 (0.2) 85 (11.3) 60 (10.2) canada m m m m m m m m m m m m m m m m m m 87.0 (0.5) 86.8 (0.5) 87.2 (0.6) -0.4 (0.5) 96.1 (0.3) 82.1 (0.6) 14.1 (0.6) 55 (5.8) 21 (4.3) chile S.E. Score dif. S.E. czech republic 97.4 (0.2) 97.3 (0.2) 97.5 (0.2) -0.2 (0.2) 99.5 (0.1) 96.3 (0.2) 3.2 (0.2) 115 (12.6) 59 (13.1) denmark 99.2 (0.1) 99.0 (0.1) 99.4 (0.1) -0.4 (0.1) 99.5 (0.1) 98.9 (0.1) 0.6 (0.1) 71 (18.2) 44 (17.2) Estonia 98.6 (0.1) 98.6 (0.1) 98.6 (0.1) 0.0 (0.2) 99.2 (0.1) 97.9 (0.2) 1.3 (0.2) 47 (11.3) 33 (12.0) finland 99.1 (0.1) 99.0 (0.1) 99.2 (0.1) -0.2 (0.1) 99.3 (0.1) 98.9 (0.1) 0.5 (0.1) 43 (16.4) 24 (14.6) france Germany m m m m m m m m m m m m m m m m m m 99.1 (0.1) 99.0 (0.1) 99.2 (0.1) -0.2 (0.2) 99.4 (0.1) 99.1 (0.1) 0.2 (0.2) 59 (18.0) 32 (20.1) hungary 94.7 (0.2) 95.1 (0.3) 94.3 (0.4) 0.8 (0.6) 97.5 (0.2) 94.4 (0.3) 3.1 (0.4) 99 (10.7) 40 (10.0) ireland 97.0 (0.1) 96.7 (0.2) 97.3 (0.2) -0.6 (0.2) 97.8 (0.1) 96.2 (0.2) 1.6 (0.3) 31 (10.3) 11 (10.3) israel 96.1 (0.1) 96.4 (0.2) 95.7 (0.2) 0.7 (0.4) 97.9 (0.1) 93.7 (0.5) 4.2 (0.5) 94 (11.8) 47 (11.7) italy 97.4 (0.2) 96.9 (0.3) 98.0 (0.2) -1.2 (0.3) 98.6 (0.1) 96.9 (0.3) 1.7 (0.4) 52 (18.9) 30 (20.0) Japan 81.4 (0.4) 81.1 (0.4) 81.6 (0.5) -0.5 (0.5) 85.6 (0.5) 78.0 (0.4) 7.7 (0.5) 35 (4.3) 24 (3.9) korea 83.5 (0.5) 83.0 (0.5) 84.1 (0.7) -1.1 (0.8) 87.1 (0.4) 79.6 (0.7) 7.5 (0.6) 45 (4.6) 33 (4.2) netherlands 98.9 (0.1) 98.7 (0.1) 99.0 (0.1) -0.3 (0.2) 99.1 (0.1) 98.5 (0.2) 0.6 (0.2) 92 (14.7) 77 (13.0) norway 98.7 (0.1) 98.2 (0.1) 99.1 (0.1) -0.9 (0.1) 99.0 (0.1) 98.5 (0.2) 0.5 (0.2) 87 (15.6) 58 (15.6) Poland 96.1 (0.2) 96.5 (0.2) 95.6 (0.3) 0.9 (0.3) 98.5 (0.2) 94.5 (0.4) 4.0 (0.6) 74 (8.5) 38 (8.6) Portugal 96.0 (0.2) 95.6 (0.2) 96.4 (0.3) -0.8 (0.3) 98.4 (0.2) 94.7 (0.3) 3.7 (0.3) 63 (8.6) 31 (8.2) Slovak republic 94.3 (0.2) 94.4 (0.3) 94.1 (0.3) 0.4 (0.4) 98.3 (0.2) 94.1 (0.2) 4.2 (0.3) 107 (9.1) 51 (7.3) Slovenia 96.2 (0.2) 95.2 (0.3) 97.4 (0.2) -2.2 (0.3) 97.0 (0.2) 95.9 (0.2) 1.1 (0.3) 37 (8.6) 22 (7.9) Spain 96.6 (0.2) 96.6 (0.3) 96.5 (0.2) 0.2 (0.4) 98.3 (0.1) 95.5 (0.4) 2.8 (0.4) 63 (9.3) 37 (8.3) Sweden 98.5 (0.1) 98.4 (0.1) 98.7 (0.1) -0.3 (0.2) 98.9 (0.1) 98.4 (0.1) 0.4 (0.2) 65 (15.5) 47 (14.7) turkey (3.4) 68.3 (0.5) 69.9 (0.6) 66.7 (0.6) 3.1 (0.8) 85.9 (0.6) 65.7 (0.5) 20.2 (0.7) 48 (3.9) 24 England (united kingdom) m m m m m m m m m m m m m m m m m m united States m m m m m m m m m m m m m m m m m m 94.5 (0.0) 94.4 (0.1) 94.6 (0.1) -0.2 (0.1) 97.0 (0.0) 93.3 (0.1) 3.7 (0.1) 67 (2.5) 39 (2.5) oEcd average Partners boys Gender difference (b - G) difference related to parents’ Parents’ highest highest Parents’ occupation: occupation: highest occupation: Semi-skilled or Skilled - semiskilled or elementary Skilled elementary (iSco 1 to 3) (iSco 4 to 9) brazil m m m m m m m m m m m m m m m m m m bulgaria m m m m m m m m m m m m m m m m m m colombia croatia cyprus* m m m m m m m m m m m m m m m m m m 97.0 (0.1) 97.0 (0.1) 97.0 (0.2) 0.0 (0.2) 98.4 (0.1) 96.4 (0.2) 2.0 (0.2) 73 (11.3) 53 (11.1) m m m m m m m m m m m m m m m m m m hong kong-china 97.5 (0.1) 97.6 (0.2) 97.3 (0.2) 0.2 (0.3) 98.2 (0.2) 97.2 (0.1) 1.0 (0.2) 59 (9.9) 42 (10.8) macao-china (8.0) 97.2 (0.1) 96.4 (0.2) 97.9 (0.1) -1.5 (0.2) 98.9 (0.1) 96.9 (0.1) 2.0 (0.2) 36 (7.7) 33 malaysia m m m m m m m m m m m m m m m m m m montenegro m m m m m m m m m m m m m m m m m m russian federation 91.6 (0.4) 91.2 (0.4) 92.0 (0.6) -0.8 (0.7) 95.9 (0.3) 87.4 (0.5) 8.5 (0.4) 52 (4.7) 19 (4.3) Serbia 91.1 (0.3) 92.8 (0.4) 89.4 (0.3) 3.4 (0.5) 96.2 (0.2) 87.8 (0.4) 8.4 (0.5) 79 (5.2) 56 (5.8) Shanghai-china 85.5 (0.5) 84.0 (0.6) 87.0 (0.6) -3.0 (0.5) 90.7 (0.3) 79.0 (0.9) 11.7 (0.9) 56 (6.7) 28 (5.1) Singapore 95.4 (0.1) 95.6 (0.2) 95.1 (0.2) 0.5 (0.3) 96.7 (0.1) 92.7 (0.3) 4.0 (0.3) 50 (6.3) 24 (5.7) chinese taipei 94.7 (0.1) 94.6 (0.3) 94.8 (0.2) -0.2 (0.4) 96.9 (0.2) 93.5 (0.2) 3.4 (0.3) 50 (8.1) 25 (7.9) m m m m m m m m m m m m m m m m m m 84.4 (0.4) 85.9 (0.5) 83.2 (0.4) 2.7 (0.6) 96.3 (0.3) 80.6 (0.5) 15.8 (0.7) 59 (5.2) 21 (5.3) united arab Emirates uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. The difference in problem-solving performance after accounting for socio-demographic characteristics of students corresponds to the coeficient from a regression where the PISA index of economic, social and cultural status (ESCS), ESCS squared, boy, and an immigrant (irst generation) dummy are introduced as further independent variables. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 219 Annex b1: reSulTS For counTrIeS And economIeS table v.4.26 [Part 1/1] Performance in problem solving and use of computers at school Results based on students’ self-reports Students who use a desktop, laptop or tablet computer at school difference in problem-solving performance Percentage of students OECD all students Girls after accounting for sociodemographic characteristics of students1 observed % S.E. % S.E. % S.E. % dif. S.E. % S.E. % S.E. % dif. S.E. Score dif. australia 93.7 (0.1) 93.5 (0.1) 93.8 (0.2) -0.4 (0.2) 94.5 (0.1) 92.6 (0.2) 1.8 (0.2) 34 (4.3) 25 (4.0) austria 81.6 (0.5) 81.3 (0.6) 81.9 (0.6) -0.6 (0.8) 78.9 (0.6) 84.7 (0.6) -5.8 (0.6) -3 (5.2) 1 (4.7) belgium 65.3 (0.4) 65.6 (0.5) 65.1 (0.5) 0.4 (0.7) 65.0 (0.4) 65.7 (0.6) -0.8 (0.7) 13 (4.2) 10 (3.9) canada m m m m m m m m m m m m m m m m m m 61.3 (0.7) 59.8 (1.0) 62.8 (0.8) -3.0 (1.2) 61.6 (1.2) 60.9 (0.7) 0.7 (1.1) 1 (4.0) -3 (3.5) chile S.E. Score dif. S.E. czech republic 84.0 (0.6) 82.8 (0.9) 85.2 (0.7) -2.4 (0.8) 82.6 (0.8) 85.2 (0.6) -2.6 (0.7) -11 (5.8) -7 (5.1) denmark 86.9 (0.4) 86.4 (0.4) 87.4 (0.5) -1.1 (0.4) 85.6 (0.5) 89.3 (0.5) -3.6 (0.6) -16 (5.4) -14 (5.4) Estonia 61.3 (0.5) 59.4 (0.7) 63.2 (0.7) -3.8 (0.9) 60.2 (0.6) 62.7 (0.6) -2.5 (0.7) -8 (3.2) -8 (3.0) finland 89.4 (0.4) 87.5 (0.4) 91.5 (0.5) -4.0 (0.4) 89.4 (0.4) 89.8 (0.4) -0.5 (0.4) -5 (4.1) -7 (4.3) france Germany m m m m m m m m m m m m m m m m m m 68.2 (0.6) 69.1 (0.7) 67.4 (0.6) 1.7 (0.6) 66.3 (0.9) 71.8 (0.7) -5.6 (0.8) -9 (4.1) -7 (3.8) hungary 75.4 (0.6) 75.8 (0.8) 74.9 (0.6) 0.9 (0.7) 74.2 (0.7) 76.7 (0.8) -2.5 (0.7) -4 (4.2) -4 (3.8) ireland 63.4 (0.6) 62.0 (0.8) 64.9 (0.7) -2.8 (0.9) 62.8 (0.7) 64.0 (0.8) -1.2 (0.8) 0 (3.8) 1 (3.7) israel 55.2 (0.7) 56.3 (0.9) 53.9 (0.8) 2.4 (1.1) 53.8 (0.8) 56.8 (1.0) -2.9 (1.2) -25 (5.1) -24 (4.5) italy 66.5 (0.6) 70.6 (1.0) 61.9 (1.0) 8.7 (1.7) 60.6 (0.7) 70.3 (0.7) -9.6 (0.7) -10 (5.0) -6 (4.6) Japan 59.7 (0.9) 56.7 (0.9) 63.1 (1.2) -6.4 (1.0) 59.7 (1.0) 59.8 (0.9) -0.1 (0.8) -4 (3.8) -4 (3.5) korea 42.7 (0.9) 40.9 (1.0) 44.8 (1.2) -3.9 (1.2) 43.0 (0.9) 42.5 (1.2) 0.4 (1.0) 0 (5.3) 0 (4.7) netherlands 93.9 (0.3) 93.6 (0.4) 94.1 (0.3) -0.4 (0.4) 94.0 (0.4) 93.6 (0.4) 0.4 (0.4) 30 (9.9) 28 (9.0) norway 91.9 (0.3) 90.7 (0.4) 93.1 (0.3) -2.5 (0.4) 92.1 (0.4) 92.3 (0.5) -0.2 (0.6) 28 (7.8) 22 (7.5) Poland 61.0 (0.7) 60.8 (0.7) 61.1 (0.9) -0.4 (0.8) 58.2 (0.8) 63.3 (0.8) -5.1 (0.7) -1 (3.9) 1 (3.6) Portugal 69.4 (0.6) 71.5 (0.8) 67.3 (0.6) 4.2 (0.8) 66.8 (1.0) 71.3 (0.6) -4.5 (1.0) -21 (4.3) -16 (4.0) Slovak republic 80.0 (0.4) 77.5 (0.5) 82.8 (0.6) -5.2 (0.6) 79.6 (0.6) 81.5 (0.5) -1.9 (0.5) 26 (5.3) 21 (4.4) Slovenia 57.1 (0.4) 58.0 (0.5) 56.2 (0.6) 1.8 (0.8) 55.7 (0.5) 58.7 (0.6) -3.0 (0.8) 6 (3.4) 5 (3.1) Spain 75.3 (0.6) 75.8 (0.6) 74.7 (0.8) 1.1 (0.6) 75.0 (0.8) 75.5 (0.7) -0.5 (0.7) 12 (5.1) 11 (4.8) Sweden 87.8 (0.7) 87.0 (0.7) 88.6 (0.8) -1.6 (0.5) 88.7 (0.8) 86.8 (0.6) 1.9 (0.7) 21 (6.5) 17 (5.4) turkey (3.3) 49.2 (0.8) 50.7 (0.9) 47.8 (0.9) 3.0 (0.7) 48.5 (1.2) 50.1 (0.8) -1.7 (1.1) 8 (3.7) 4 England (united kingdom) m m m m m m m m m m m m m m m m m m united States m m m m m m m m m m m m m m m m m m 71.7 (0.1) 71.4 (0.1) 72.0 (0.1) -0.6 (0.2) 70.7 (0.2) 72.7 (0.1) -2.0 (0.2) 3 (1.0) 2 (1.0) oEcd average Partners boys Gender difference (b - G) difference related to parents’ Parents’ highest highest Parents’ occupation: occupation: highest occupation: Semi-skilled or Skilled - semiskilled or elementary Skilled elementary (iSco 1 to 3) (iSco 4 to 9) brazil m m m m m m m m m m m m m m m m m m bulgaria m m m m m m m m m m m m m m m m m m colombia croatia cyprus* m m m m m m m m m m m m m m m m m m 78.5 (0.5) 80.3 (0.5) 76.5 (0.7) 3.8 (0.7) 76.2 (0.9) 80.5 (0.5) -4.3 (0.8) -9 (5.2) -8 (4.5) m m m m m m m m m m m m m m m m m m hong kong-china 83.5 (0.4) 80.9 (0.5) 86.5 (0.4) -5.6 (0.5) 82.1 (0.8) 85.0 (0.4) -2.9 (0.9) 7 (6.4) 9 (5.9) macao-china (4.1) 87.9 (0.3) 86.2 (0.4) 89.6 (0.3) -3.4 (0.4) 88.9 (0.5) 87.9 (0.3) 1.1 (0.5) 11 (4.3) 11 malaysia m m m m m m m m m m m m m m m m m m montenegro m m m m m m m m m m m m m m m m m m russian federation 80.4 (0.4) 79.5 (0.5) 81.3 (0.5) -1.8 (0.6) 80.2 (0.4) 80.6 (0.6) -0.4 (0.6) 4 (5.1) 2 (4.5) Serbia 82.4 (0.5) 82.5 (0.4) 82.4 (0.7) 0.2 (0.7) 83.8 (0.7) 82.0 (0.5) 1.8 (0.7) 20 (5.2) 15 (4.5) Shanghai-china 38.7 (0.6) 38.1 (0.6) 39.3 (0.7) -1.3 (0.6) 39.6 (0.7) 37.9 (0.8) 1.7 (0.9) 16 (4.0) 12 (3.5) Singapore 69.7 (0.3) 67.2 (0.4) 72.2 (0.4) -5.0 (0.6) 69.0 (0.4) 70.9 (0.5) -2.0 (0.6) -18 (3.2) -16 (2.9) chinese taipei 78.8 (0.4) 75.6 (0.6) 82.0 (0.4) -6.3 (0.6) 80.2 (0.6) 78.4 (0.5) 1.9 (0.7) 13 (4.1) 11 (3.7) m m m m m m m m m m m m m m m m m m 49.7 (0.6) 55.4 (0.8) 44.8 (0.7) 10.5 (0.8) 51.7 (1.5) 48.9 (0.6) 2.8 (1.5) -16 (5.4) -23 (4.3) united arab Emirates uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. The difference in problem-solving performance after accounting for socio-demographic characteristics of students corresponds to the coeficient from a regression where the PISA index of economic, social and cultural status (ESCS), ESCS squared, boy, and an immigrant (irst-generation) dummy are introduced as further independent variables. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 220 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.27 [Part 1/3] differences in problem-solving, mathematics, reading and science performance related to computer use Results based on students’ self-reports difference in performance associated with the use of computers at home, after accounting for socio-demographic characteristics of students1 Partners OECD Problem solving (use - no use) mathematics (use - no use) reading (use - no use) Score dif. Science (use - no use) computer-based mathematics (use - no use) digital reading (use - no use) Score dif. S.E. Score dif. S.E. S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. australia 50 (6.4) 51 (5.7) 53 (6.5) 52 (6.4) 52 (4.7) 56 (6.6) austria 50 (20.0) 39 (15.6) 45 (16.2) 36 (14.7) 36 (13.8) 36 (13.3) belgium 60 (10.2) 43 (9.0) 48 (8.8) 42 (8.0) 51 (9.7) 67 (10.8) canada m m m m m m m m m m m m chile 21 (4.3) 16 (3.5) 20 (4.3) 18 (4.1) 14 (4.0) 20 (4.2) czech republic 59 (13.1) 45 (11.4) 50 (11.5) 51 (12.3) m m m m denmark 44 (17.2) 45 (12.6) 60 (15.7) 54 (15.8) 53 (15.8) 58 (13.7) Estonia 33 (12.0) 32 (10.8) 38 (12.3) 32 (12.6) 17 (12.1) 33 (13.8) finland 24 (14.6) 1 (12.0) 15 (13.2) 10 (11.9) m m m m france m m m m m m m m m m m m Germany 32 (20.1) 34 (16.0) 28 (15.1) 44 (17.1) 37 (15.0) 50 (20.2) hungary 40 (10.0) 30 (6.7) 41 (8.5) 36 (7.8) 23 (7.8) 32 (9.8) ireland 11 (10.3) 5 (7.9) 4 (7.7) 3 (7.9) 7 (7.1) 3 (7.3) israel 47 (11.7) 42 (8.6) 49 (9.4) 50 (8.0) 44 (9.9) 56 (10.4) italy 30 (20.0) 35 (14.7) 44 (14.7) 32 (13.8) 29 (11.8) 40 (14.6) Japan 24 (3.9) 24 (3.6) 23 (3.8) 24 (3.6) 26 (3.8) 23 (3.5) korea 33 (4.2) 45 (4.3) 39 (4.0) 36 (3.6) 35 (4.1) 29 (3.5) netherlands 77 (13.0) 78 (12.4) 88 (13.1) 76 (13.2) m m m m norway 58 (15.6) 55 (12.3) 70 (14.8) 58 (13.2) 42 (13.0) 81 (17.6) Poland 38 (8.6) 24 (8.3) 27 (7.9) 26 (8.0) 32 (7.5) 38 (9.0) Portugal 31 (8.2) 39 (8.0) 42 (8.3) 36 (8.7) 25 (6.6) 40 (8.1) Slovak republic 51 (7.3) 44 (7.3) 49 (7.0) 45 (7.0) 35 (6.1) 56 (8.0) Slovenia 22 (7.9) 12 (7.0) 24 (7.4) 23 (6.7) 14 (6.7) 32 (7.6) Spain 37 (8.3) 30 (6.3) 35 (7.0) 30 (7.0) 37 (6.9) 29 (8.1) Sweden 47 (14.7) 37 (12.6) 61 (14.7) 51 (14.5) 45 (11.9) 34 (14.5) turkey 24 (3.4) 19 (3.6) 18 (3.1) 17 (3.2) m m m m England (united kingdom) m m m m m m m m m m m m united States m m m m m m m m m m m m oEcd average 39 (2.5) 34 (2.0) 40 (2.2) 37 (2.1) 33 (2.2) 41 (2.5) brazil m m m m m m m m m m m m bulgaria m m m m m m m m m m m m colombia m m m m m m m m m m m m croatia 53 (11.1) 46 (7.6) 42 (7.8) 44 (7.1) m m m m cyprus* m m m m m m m m m m m m hong kong-china 42 (10.8) 44 (10.1) 36 (9.9) 41 (9.8) 44 (11.2) 37 (11.6) macao-china 33 (8.0) 35 (8.1) 31 (7.1) 31 (7.2) 31 (7.8) 26 (6.5) malaysia m m m m m m m m m m m m montenegro m m m m m m m m m m m m russian federation 19 (4.3) 24 (6.2) 25 (5.2) 20 (6.1) 18 (4.9) 36 (6.5) Serbia 56 (5.8) 52 (4.8) 53 (5.4) 45 (5.8) m m m m Shanghai-china 28 (5.1) 13 (4.5) 13 (3.2) 10 (3.8) 18 (5.1) 25 (4.1) Singapore 24 (5.7) 35 (6.2) 36 (5.6) 34 (6.1) 29 (5.6) 28 (4.9) chinese taipei 25 (7.9) 26 (8.0) 21 (5.6) 19 (5.2) 23 (5.7) 32 (5.8) united arab Emirates m m m m m m m m m m m m uruguay 21 (5.3) 23 (4.7) 26 (4.6) 21 (4.1) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. The adjusted effects correspond to the coeficient from a regression where the PISA index of economic, social and cultural status (ESCS), ESCS squared, boy, and an immigrant (irst generation) dummy are introduced as further independent variables. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 221 Annex b1: reSulTS For counTrIeS And economIeS table v.4.27 [Part 2/3] differences in problem-solving, mathematics, reading and science performance related to computer use Results based on students’ self-reports computer use effect size: difference in performance related to computer use, after accounting for socio-demographic characteristics of students,1 divided by the variation in scores within each country/economy (standard deviation) OECD Problem solving (use - no use) reading (use - no use) Science (use - no use) computer-based mathematics (use - no use) digital reading (use - no use) Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. Effect size S.E. australia 0.51 (0.07) 0.54 (0.06) 0.56 (0.07) 0.53 (0.06) 0.58 (0.05) 0.58 (0.07) austria 0.53 (0.21) 0.43 (0.17) 0.50 (0.18) 0.40 (0.16) 0.41 (0.16) 0.40 (0.15) belgium 0.59 (0.10) 0.43 (0.09) 0.51 (0.09) 0.44 (0.08) 0.54 (0.10) 0.70 (0.11) canada m m m m m m m m m m m m chile 0.24 (0.05) 0.19 (0.04) 0.25 (0.05) 0.22 (0.05) 0.18 (0.05) 0.24 (0.05) czech republic 0.63 (0.14) 0.47 (0.12) 0.57 (0.13) 0.57 (0.14) m m m m denmark 0.48 (0.19) 0.56 (0.15) 0.74 (0.19) 0.61 (0.18) 0.62 (0.18) 0.71 (0.17) Estonia 0.37 (0.14) 0.40 (0.13) 0.48 (0.15) 0.40 (0.16) 0.21 (0.15) 0.36 (0.15) finland 0.26 (0.16) 0.02 (0.15) 0.17 (0.15) 0.11 (0.13) m m m m m m m m m m m m m m m m 0.34 (0.21) 0.36 (0.17) 0.33 (0.17) 0.49 (0.19) 0.40 (0.16) 0.52 (0.21) france Germany hungary 0.39 (0.09) 0.33 (0.07) 0.46 (0.09) 0.41 (0.09) 0.25 (0.08) 0.29 (0.09) ireland 0.12 (0.11) 0.06 (0.09) 0.05 (0.09) 0.03 (0.09) 0.08 (0.09) 0.04 (0.09) israel 0.38 (0.09) 0.41 (0.08) 0.45 (0.09) 0.48 (0.08) 0.40 (0.09) 0.48 (0.09) italy 0.34 (0.22) 0.39 (0.16) 0.46 (0.15) 0.34 (0.14) 0.35 (0.14) 0.42 (0.15) Japan 0.28 (0.04) 0.26 (0.04) 0.23 (0.04) 0.26 (0.04) 0.30 (0.04) 0.30 (0.04) korea 0.36 (0.04) 0.46 (0.04) 0.45 (0.04) 0.44 (0.04) 0.38 (0.04) 0.36 (0.04) netherlands 0.82 (0.14) 0.88 (0.14) 1.01 (0.15) 0.85 (0.14) m m m m norway 0.56 (0.15) 0.61 (0.14) 0.72 (0.15) 0.59 (0.14) 0.49 (0.15) 0.82 (0.18) Poland 0.40 (0.09) 0.27 (0.09) 0.31 (0.09) 0.30 (0.09) 0.37 (0.09) 0.39 (0.09) Portugal 0.35 (0.09) 0.42 (0.09) 0.46 (0.09) 0.41 (0.10) 0.29 (0.08) 0.45 (0.09) Slovak republic 0.53 (0.07) 0.44 (0.07) 0.48 (0.07) 0.46 (0.07) 0.42 (0.07) 0.60 (0.08) Slovenia 0.23 (0.08) 0.13 (0.08) 0.27 (0.08) 0.26 (0.08) 0.16 (0.08) 0.32 (0.08) Spain 0.36 (0.08) 0.35 (0.07) 0.39 (0.08) 0.36 (0.08) 0.45 (0.08) 0.30 (0.08) Sweden 0.49 (0.15) 0.42 (0.14) 0.60 (0.14) 0.53 (0.15) 0.53 (0.14) 0.35 (0.15) turkey m 0.30 (0.04) 0.21 (0.04) 0.21 (0.04) 0.22 (0.04) m m m England (united kingdom) m m m m m m m m m m m m united States m m m m m m m m m m m m 0.41 (0.03) 0.38 (0.02) 0.44 (0.02) 0.40 (0.02) 0.37 (0.02) 0.43 (0.03) brazil m m m m m m m m m m m m bulgaria m m m m m m m m m m m m colombia m m m m m m m m m m m m croatia 0.57 (0.12) 0.52 (0.09) 0.49 (0.09) 0.51 (0.08) m m m m cyprus* m m m m m m m m m m m m hong kong-china 0.46 (0.12) 0.46 (0.10) 0.42 (0.11) 0.49 (0.12) 0.51 (0.13) 0.39 (0.12) macao-china oEcd average Partners mathematics (use - no use) 0.41 (0.10) 0.38 (0.09) 0.37 (0.09) 0.40 (0.09) 0.38 (0.09) 0.38 (0.09) malaysia m m m m m m m m m m m m montenegro m m m m m m m m m m m m russian federation 0.22 (0.05) 0.28 (0.07) 0.28 (0.06) 0.24 (0.07) 0.22 (0.06) 0.42 (0.07) Serbia 0.63 (0.06) 0.58 (0.05) 0.58 (0.06) 0.52 (0.06) m m m m Shanghai-china 0.31 (0.05) 0.13 (0.04) 0.16 (0.04) 0.12 (0.05) 0.19 (0.05) 0.30 (0.05) Singapore 0.25 (0.06) 0.33 (0.06) 0.36 (0.06) 0.33 (0.06) 0.30 (0.06) 0.31 (0.05) chinese taipei 0.27 (0.08) 0.22 (0.07) 0.23 (0.06) 0.22 (0.06) 0.26 (0.06) 0.36 (0.06) united arab Emirates uruguay m m m m m m m m m m m m 0.21 (0.05) 0.27 (0.05) 0.28 (0.05) 0.23 (0.04) m m m m Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. The adjusted effects correspond to the coeficient from a regression where the PISA index of economic, social and cultural status (ESCS), ESCS squared, boy, and an immigrant (irst generation) dummy are introduced as further independent variables. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 222 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For counTrIeS And economIeS: Annex b1 table v.4.27 [Part 3/3] differences in problem-solving, mathematics, reading and science performance related to computer use Results based on students’ self-reports difference in computer use effect sizes between problem solving (PS) and… OECD … mathematics (PS - m) … computer-based mathematics (PS - cbm) … Science (PS - S) … digital reading (PS - dr) Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. Effect size dif. S.E. -0.02 (0.04) -0.05 (0.05) -0.02 (0.04) -0.06 (0.05) -0.07 (0.06) austria 0.10 (0.09) 0.03 (0.09) 0.14 (0.09) 0.12 (0.12) 0.13 (0.14) belgium 0.15 (0.08) 0.08 (0.09) 0.15 (0.08) 0.05 (0.08) -0.12 (0.08) canada m m m m m m m m m m 0.05 (0.04) -0.01 (0.05) 0.02 (0.04) 0.07 (0.05) 0.00 (0.04) australia chile czech republic 0.15 (0.08) 0.05 (0.10) 0.06 (0.09) m m m m denmark -0.09 (0.15) -0.26 (0.20) -0.13 (0.14) -0.14 (0.18) -0.23 (0.18) Estonia -0.03 (0.10) -0.11 (0.10) -0.03 (0.11) 0.17 (0.10) 0.01 (0.10) finland 0.24 (0.08) 0.09 (0.11) 0.14 (0.10) m m m m m m m m m m m m m m Germany -0.03 (0.13) 0.01 (0.13) -0.16 (0.13) -0.07 (0.15) -0.18 (0.16) hungary 0.06 (0.06) -0.07 (0.07) -0.03 (0.07) 0.13 (0.09) 0.09 (0.07) ireland 0.06 (0.08) 0.08 (0.08) 0.09 (0.08) 0.04 (0.09) 0.08 (0.08) israel -0.03 (0.06) -0.08 (0.06) -0.11 (0.06) -0.02 (0.07) -0.10 (0.07) italy -0.05 (0.11) -0.12 (0.14) 0.00 (0.17) -0.01 (0.13) -0.09 (0.15) Japan 0.02 (0.04) 0.05 (0.04) 0.03 (0.04) -0.02 (0.04) -0.02 (0.03) korea -0.10 (0.03) -0.09 (0.03) -0.08 (0.03) -0.02 (0.03) 0.00 (0.04) netherlands -0.06 (0.14) -0.19 (0.13) -0.03 (0.11) m m m m norway -0.05 (0.11) -0.16 (0.12) -0.03 (0.12) 0.07 (0.10) -0.26 (0.11) france Poland 0.13 (0.07) 0.09 (0.06) 0.10 (0.06) 0.03 (0.06) 0.01 (0.07) -0.07 (0.05) -0.10 (0.06) -0.06 (0.07) 0.06 (0.07) -0.10 (0.07) Slovak republic 0.09 (0.06) 0.05 (0.06) 0.07 (0.06) 0.10 (0.06) -0.07 (0.07) Slovenia 0.10 (0.07) -0.04 (0.07) -0.03 (0.06) 0.06 (0.07) -0.10 (0.05) Spain 0.00 (0.07) -0.03 (0.09) 0.00 (0.07) -0.09 (0.08) 0.06 (0.08) Sweden 0.07 (0.09) -0.11 (0.11) -0.04 (0.09) -0.04 (0.12) 0.13 (0.11) turkey 0.09 (0.03) 0.09 (0.03) 0.08 (0.03) m m m m England (united kingdom) m m m m m m m m m m united States m m m m m m m m m m 0.03 (0.02) -0.03 (0.02) 0.01 (0.02) 0.02 (0.02) -0.04 (0.02) brazil m m m m m m m m m m bulgaria m m m m m m m m m m colombia m m m m m m m m m m 0.05 (0.07) 0.08 (0.07) 0.06 (0.10) m m m m Portugal oEcd average Partners … reading (PS - r) croatia cyprus* m m m m m m m m m m hong kong-china 0.00 (0.10) 0.04 (0.10) -0.04 (0.11) -0.05 (0.10) 0.06 (0.09) macao-china 0.03 (0.07) 0.04 (0.07) 0.01 (0.07) 0.03 (0.09) 0.04 (0.07) malaysia m m m m m m m m m m montenegro m m m m m m m m m m -0.06 (0.07) -0.06 (0.06) -0.02 (0.07) 0.00 (0.05) -0.20 (0.07) Serbia 0.05 (0.04) 0.05 (0.05) 0.12 (0.05) m m m m Shanghai-china 0.18 (0.04) 0.15 (0.04) 0.19 (0.05) 0.12 (0.04) 0.01 (0.05) -0.08 (0.04) -0.11 (0.05) -0.08 (0.04) -0.05 (0.04) -0.06 (0.05) 0.05 (0.04) 0.04 (0.05) 0.05 (0.05) 0.01 (0.06) -0.09 (0.06) m m m m m m m m m m -0.05 (0.04) -0.06 (0.05) -0.01 (0.04) m m m m russian federation Singapore chinese taipei united arab Emirates uruguay Note: Values that are statistically signiicant are indicated in bold (see Annex A3). 1. The adjusted effects correspond to the coeficient from a regression where the PISA index of economic, social and cultural status (ESCS), ESCS squared, boy, and an immigrant (irst generation) dummy are introduced as further independent variables. * See notes at the beginning of this Annex. 1 2 http://dx.doi.org/10.1787/888933003706 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 223 Annex b2: reSulTS For regIonS wIThIn counTrIeS Annex b2 reSulTS For regIonS wIThIn counTrIeS table b2.v.1 [Part 1/2] Percentage of students at each proiciency level in problem solving, by region Percentage of students at each level Partners OECD below level 1 (below 358.49 score points) australia Australian Capital Territory New South Wales Northern Territory Queensland South Australia Tasmania Victoria Western Australia belgium flemish Community• french Community german-speaking Community canada Alberta British Columbia Manitoba New Brunswick Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan italy Centre North East North West South South Islands Portugal Alentejo Spain Basque Country• Catalonia• Madrid brazil Central-West region Northeast region North region Southeast region South region colombia Bogotá Cali Manizales Medellín united arab Emirates Abu Dhabi• Ajman Dubai• fujairah ras al-khaimah Sharjah umm al-Quwain level 1 (from 358.49 to less than 423.42 score points) level 2 (from 423.42 to less than 488.35 score points) level 3 (from 488.35 to less than 553.28 score points) level 4 (from 553.28 to less than 618.21 score points) level 5 (from 618.21 to less than 683.14 score points) level 6 (above 683.14 score points) % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. 6.4 5.2 9.1 4.9 4.4 10.2 4.6 4.5 (1.2) (0.6) (1.5) (0.7) (0.7) (1.0) (0.8) (0.9) 9.5 10.3 12.4 10.7 10.7 16.5 10.5 9.3 (1.2) (0.8) (2.3) (1.0) (1.0) (1.9) (1.3) (1.1) 17.6 18.9 18.1 19.8 20.6 22.8 19.5 18.5 (1.5) (0.9) (2.5) (1.1) (1.4) (1.7) (1.2) (1.2) 24.1 25.6 21.7 25.8 27.2 22.8 26.3 25.9 (2.2) (1.0) (3.2) (1.1) (1.3) (1.5) (1.4) (1.7) 24.0 22.1 21.5 22.8 22.0 16.0 22.9 24.7 (2.1) (0.9) (3.0) (0.9) (1.4) (1.5) (1.2) (1.4) 13.5 12.7 12.2 11.7 11.8 8.5 12.4 12.8 (1.8) (0.9) (3.2) (0.9) (1.2) (1.1) (1.1) (1.2) 4.8 5.2 5.0 4.3 3.3 3.2 3.9 4.4 (1.1) (0.7) (2.7) (0.6) (0.6) (0.7) (0.6) (0.9) 6.7 12.6 5.8 (0.7) (1.1) (0.9) 9.5 14.4 9.1 (0.9) (0.8) (1.1) 16.8 20.3 19.5 (0.9) (1.2) (1.7) 24.9 24.0 26.3 (1.0) (1.1) (2.1) 24.2 19.1 24.7 (1.0) (1.1) (1.5) 13.9 8.1 11.1 (1.0) (0.9) (1.2) 4.1 1.5 3.6 (0.5) (0.4) (0.6) 4.6 3.1 7.3 5.4 7.6 5.1 5.1 7.0 5.8 5.2 (0.6) (0.7) (1.0) (0.7) (2.1) (1.4) (0.7) (0.7) (0.8) (0.7) 9.6 9.4 13.2 10.3 11.3 10.8 9.4 14.2 8.9 11.1 (1.0) (1.0) (1.2) (1.2) (1.6) (1.6) (1.0) (1.2) (0.7) (1.0) 16.8 18.2 21.6 20.8 21.6 22.6 19.4 25.7 18.0 21.1 (1.4) (1.3) (1.1) (1.6) (1.5) (3.2) (1.1) (1.5) (1.0) (1.6) 26.2 26.1 24.8 28.0 26.9 27.3 24.9 28.2 26.5 28.0 (1.6) (1.4) (1.6) (2.4) (1.7) (2.8) (1.2) (2.1) (1.2) (1.6) 23.9 24.0 21.2 23.4 21.0 22.6 22.5 17.7 23.4 20.7 (1.6) (1.4) (1.4) (1.7) (1.6) (2.4) (1.3) (1.2) (0.9) (1.3) 13.6 13.8 9.2 9.3 9.3 9.2 12.6 5.6 12.6 10.9 (1.2) (1.3) (1.2) (1.2) (1.1) (1.1) (1.0) (0.9) (1.1) (1.1) 5.3 5.3 2.7 2.8 2.3 2.5 6.0 1.6 4.7 2.9 (0.8) (0.7) (0.5) (0.6) (0.6) (0.8) (1.0) (0.5) (0.8) (0.6) 6.2 4.2 2.5 6.6 7.4 (1.6) (1.1) (0.8) (1.9) (2.0) 9.8 8.1 6.8 17.7 16.2 (2.7) (1.6) (1.8) (2.8) (2.5) 18.3 19.3 18.8 31.6 27.7 (2.4) (2.1) (2.1) (2.9) (2.0) 30.3 27.5 29.2 27.5 25.3 (3.6) (1.7) (3.1) (2.2) (2.3) 23.9 25.9 28.3 14.0 15.9 (2.2) (2.2) (3.2) (2.6) (2.1) 9.6 11.9 12.1 2.4 6.1 (2.2) (1.4) (2.5) (0.8) (1.6) 1.9 3.1 2.3 0.1 1.2 (1.0) (0.8) (0.8) (0.2) (0.6) 6.0 (2.0) 11.2 (2.1) 23.4 (2.4) 28.1 (2.7) 21.2 (2.7) 8.4 (2.6) 1.8 (1.3) 8.0 11.2 6.8 (0.8) (2.4) (2.0) 13.2 12.4 13.5 (0.8) (1.5) (2.6) 23.2 24.0 19.6 (0.9) (1.9) (3.0) 27.3 25.3 26.0 (0.9) (1.9) (2.2) 18.7 18.0 21.7 (1.1) (1.7) (3.0) 7.6 7.3 9.8 (0.6) (1.1) (3.0) 2.1 1.9 2.6 (0.3) (0.6) (1.2) 16.3 37.8 40.2 14.4 17.5 (4.5) (4.1) (5.6) (1.9) (3.1) 25.3 25.1 30.3 24.5 27.1 (3.5) (2.9) (3.6) (2.1) (2.7) 29.3 20.0 17.8 29.8 30.4 (2.7) (3.0) (3.5) (2.0) (2.8) 19.6 10.7 9.0 21.2 17.3 (4.0) (2.4) (2.8) (2.3) (2.9) 7.3 3.9 2.6 8.4 6.1 (2.1) (1.4) (1.2) (1.3) (1.3) 1.8 1.6 0.2 1.5 1.6 (0.9) (0.8) (0.2) (0.4) (0.7) 0.5 0.8 0.0 0.3 0.1 (0.4) (0.5) (0.0) (0.2) (0.1) 27.1 31.6 21.9 24.8 (2.4) (4.1) (2.1) (2.6) 28.3 28.1 27.0 26.9 (1.6) (2.0) (1.7) (2.8) 27.2 24.6 28.9 23.8 (1.8) (2.1) (2.0) (2.7) 13.2 12.4 15.6 15.0 (1.5) (1.7) (1.7) (1.7) 3.4 2.9 5.3 6.8 (0.8) (0.9) (0.9) (1.4) 0.8 0.4 1.0 2.2 (0.3) (0.2) (0.4) (0.9) 0.1 0.1 0.4 0.5 (0.1) (0.1) (0.2) (0.3) 37.7 42.6 18.1 32.4 40.6 24.2 44.8 (2.2) (4.5) (0.6) (2.8) (4.6) (4.0) (3.5) 23.0 29.1 19.6 32.6 31.4 29.4 28.8 (1.4) (2.8) (1.1) (2.8) (3.0) (3.2) (3.3) 20.5 19.5 22.6 22.4 18.3 26.1 18.5 (1.1) (2.5) (1.3) (2.7) (2.4) (3.0) (2.4) 12.2 8.1 20.6 9.6 7.3 14.8 6.2 (1.0) (2.5) (0.9) (1.0) (1.2) (2.2) (1.6) 4.9 0.7 12.7 2.6 1.9 4.8 1.5 (0.6) (0.6) (0.7) (0.8) (0.7) (1.5) (0.7) 1.5 0.0 5.1 0.3 0.5 0.6 0.1 (0.4) c (0.5) (0.3) (0.2) (0.5) (0.2) 0.2 0.0 1.4 0.0 0.0 0.1 0.0 (0.1) c (0.2) c c (0.2) c • PISA adjudicated region. Notes: Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), North East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), North West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). Brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), Northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), North region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.2.1 for national data. 1 2 http://dx.doi.org/10.1787/888933003763 224 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For regIonS wIThIn counTrIeS: Annex b2 table b2.v.1 [Part 2/2] Percentage of students at each proiciency level in problem solving, by region Percentage of students at or above each proiciency level OECD level 1 or above (above 358.49 score points) level 2 or above (above 423.42 score points) level 3 or above (above 488.35 score points) level 4 or above (above 553.28 score points) level 5 or above (above 618.21 score points) level 6 (above 683.14 score points) % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. Australian Capital Territory 93.6 (1.2) 84.1 (1.4) 66.5 (1.8) 42.4 (2.0) 18.4 (1.8) 4.8 (1.1) New South Wales 94.8 (0.6) 84.5 (1.1) 65.5 (1.4) 40.0 (1.5) 17.9 (1.3) 5.2 (0.7) Northern Territory 90.9 (1.5) 78.5 (2.4) 60.4 (3.0) 38.7 (4.1) 17.2 (3.9) 5.0 (2.7) Queensland 95.1 (0.7) 84.4 (1.3) 64.6 (1.6) 38.8 (1.4) 16.0 (1.0) 4.3 (0.6) South Australia 95.6 (0.7) 84.9 (1.3) 64.3 (1.8) 37.2 (2.0) 15.2 (1.5) 3.3 (0.6) Tasmania 89.8 (1.0) 73.2 (1.9) 50.5 (1.8) 27.7 (1.6) 11.7 (1.4) 3.2 (0.7) Victoria 95.4 (0.8) 85.0 (1.4) 65.4 (1.9) 39.2 (2.0) 16.3 (1.3) 3.9 (0.6) Western Australia 95.5 (0.9) 86.2 (1.4) 67.7 (1.7) 41.8 (2.0) 17.2 (1.5) 4.4 (0.9) australia belgium flemish Community• 93.3 (0.7) 83.8 (1.2) 67.0 (1.4) 42.2 (1.5) 18.0 (1.2) 4.1 (0.5) french Community 87.4 (1.1) 73.0 (1.5) 52.6 (1.9) 28.7 (1.6) 9.6 (1.0) 1.5 (0.4) german-speaking Community 94.2 (0.9) 85.1 (1.2) 65.6 (1.8) 39.3 (1.6) 14.7 (1.2) 3.6 (0.6) Alberta 95.4 (0.6) 85.8 (1.3) 69.1 (2.1) 42.9 (2.4) 19.0 (1.6) 5.3 (0.8) British Columbia 96.9 (0.7) 87.5 (1.2) 69.3 (1.6) 43.2 (1.7) 19.1 (1.4) 5.3 (0.7) Manitoba 92.7 (1.0) 79.5 (1.3) 57.9 (1.6) 33.1 (1.5) 11.9 (1.2) 2.7 (0.5) New Brunswick 94.6 (0.7) 84.3 (1.3) 63.5 (1.7) 35.5 (2.0) 12.1 (1.3) 2.8 (0.6) Newfoundland and Labrador 92.4 (2.1) 81.1 (2.8) 59.5 (2.6) 32.6 (2.0) 11.6 (1.2) 2.3 (0.6) Nova Scotia 94.9 (1.4) 84.1 (2.1) 61.5 (3.8) 34.2 (2.8) 11.6 (1.5) 2.5 (0.8) Ontario 94.9 (0.7) 85.5 (1.5) 66.1 (2.1) 41.2 (2.3) 18.7 (1.7) 6.0 (1.0) Prince Edward Island 93.0 (0.7) 78.8 (1.4) 53.2 (1.7) 25.0 (1.4) 7.3 (0.8) 1.6 (0.5) Quebec 94.2 (0.8) 85.3 (1.1) 67.2 (1.6) 40.8 (1.8) 17.3 (1.5) 4.7 (0.8) Saskatchewan 94.8 (0.7) 83.7 (1.1) 62.6 (1.6) 34.5 (1.7) 13.8 (1.1) 2.9 (0.6) Centre 93.8 (1.6) 84.0 (3.9) 65.8 (5.7) 35.5 (4.2) 11.6 (2.6) 1.9 (1.0) North East 95.8 (1.1) 87.7 (2.1) 68.4 (2.9) 40.9 (3.2) 15.0 (1.9) 3.1 (0.8) North West 97.5 (0.8) 90.7 (2.2) 71.9 (3.7) 42.7 (4.9) 14.4 (2.9) 2.3 (0.8) South 93.4 (1.9) 75.7 (4.1) 44.1 (4.2) 16.5 (3.0) 2.6 (0.8) 0.1 (0.2) South Islands 92.6 (2.0) 76.4 (3.5) 48.6 (3.9) 23.3 (3.0) 7.3 (2.0) 1.2 (0.6) 94.0 (2.0) 82.8 (3.9) 59.5 (5.5) 31.4 (5.3) 10.3 (3.9) 1.8 (1.3) canada italy Portugal Alentejo Partners Spain Basque Country• 92.0 (0.8) 78.8 (1.3) 55.6 (1.7) 28.4 (1.5) 9.6 (0.8) 2.1 (0.3) Catalonia• 88.8 (2.4) 76.4 (3.1) 52.4 (3.4) 27.1 (2.6) 9.2 (1.5) 1.9 (0.6) Madrid 93.2 (2.0) 79.7 (4.0) 60.1 (5.3) 34.1 (5.8) 12.4 (3.9) 2.6 (1.2) Central-West region 83.7 (4.5) 58.4 (5.7) 29.1 (5.1) 9.6 (2.8) 2.3 (1.0) 0.5 (0.4) Northeast region 62.2 (4.1) 37.1 (5.2) 17.0 (3.7) 6.3 (2.3) 2.4 (1.3) 0.8 (0.5) North region 59.8 (5.6) 29.5 (4.7) 11.8 (3.3) 2.8 (1.3) 0.2 (0.2) 0.0 (0.0) Southeast region 85.6 (1.9) 61.1 (3.2) 31.3 (3.4) 10.2 (1.6) 1.7 (0.5) 0.3 (0.2) South region 82.5 (3.1) 55.4 (4.3) 25.0 (3.5) 7.7 (1.4) 1.6 (0.7) 0.1 (0.1) Bogotá 72.9 (2.4) 44.7 (3.0) 17.5 (2.0) 4.3 (1.0) 0.9 (0.3) 0.1 (0.1) Cali 68.4 (4.1) 40.3 (3.6) 15.7 (2.2) 3.4 (0.9) 0.5 (0.2) 0.1 (0.1) Manizales 78.1 (2.1) 51.1 (2.6) 22.2 (1.9) 6.6 (1.1) 1.3 (0.4) 0.4 (0.2) Medellín 75.2 (2.6) 48.3 (3.9) 24.6 (3.3) 9.5 (2.2) 2.7 (1.1) 0.5 (0.3) (0.1) brazil colombia united arab Emirates Abu Dhabi• 62.3 (2.2) 39.3 (2.0) 18.8 (1.6) 6.6 (0.9) 1.7 (0.4) 0.2 Ajman 57.4 (4.5) 28.3 (4.2) 8.8 (2.5) 0.7 (0.6) 0.0 c 0.0 c Dubai• 81.9 (0.6) 62.3 (1.1) 39.6 (0.9) 19.1 (0.6) 6.4 (0.5) 1.4 (0.2) fujairah 67.6 (2.8) 35.0 (2.7) 12.6 (1.4) 3.0 (0.9) 0.3 (0.3) 0.0 c ras al-khaimah 59.4 (4.6) 28.0 (3.4) 9.6 (1.6) 2.4 (0.7) 0.5 (0.2) 0.0 c Sharjah 75.8 (4.0) 46.4 (4.0) 20.3 (3.0) 5.5 (1.8) 0.7 (0.4) 0.1 (0.2) umm al-Quwain 55.2 (3.5) 26.4 (2.4) 7.9 (1.6) 1.7 (0.6) 0.1 (0.2) 0.0 c • PISA adjudicated region. Notes: Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), North East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), North West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). Brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), Northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), North region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.2.1 for national data. 1 2 http://dx.doi.org/10.1787/888933003763 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 225 Annex b2: reSulTS For regIonS wIThIn counTrIeS table b2.v.2 [Part 1/2] mean score and variation in student performance in problem solving, by region Percentiles mean score OECD mean Standard deviation S.E. S.d. S.E. 5th Score 10th S.E. Score S.E. 25th Score S.E. 50th (median) Score S.E. 75th Score S.E. 90th Score S.E. 95th Score S.E. australia Australian Capital Territory 526 (3.7) 103 (3.3) 344 (13.8) 388 (10.6) 461 (6.2) 534 (4.7) 597 (5.0) 650 (6.2) 682 (8.4) New South Wales 525 (3.5) 99 (2.1) 356 394 459 (4.6) 527 (4.0) 593 (4.7) 652 (5.1) 684 (5.3) (6.0) (5.2) Northern Territory 513 (7.9) 112 (6.1) 313 (15.3) 364 (15.2) 438 (11.5) 524 (10.0) 593 (13.2) 653 (22.3) 676 (27.0) Queensland 522 (3.4) 97 (2.3) 359 (7.2) 396 (6.2) 457 (5.3) 525 (3.5) 589 (4.0) 644 (4.2) 677 (6.1) South Australia 520 (4.1) 93 (2.2) 364 (8.8) 400 (7.6) 458 (4.8) 522 (5.1) 584 (6.2) 639 (6.0) 669 (6.4) Tasmania 490 (4.0) 105 (2.6) 317 (7.5) 356 (6.9) 418 (6.7) 489 (5.7) 561 (5.5) 628 (8.5) 666 (9.7) Victoria 523 (4.1) 95 (2.1) 363 (9.1) 398 (6.0) 460 (5.5) 526 (4.7) 590 (4.9) 643 (4.9) 673 (5.7) Western Australia 528 (4.0) 96 (2.9) 363 (9.7) 402 (8.0) 465 (5.1) 533 (4.8) 595 (4.9) 647 (5.3) 677 (9.2) belgium flemish Community• 525 (3.3) 102 (2.3) 341 (7.4) 385 (6.1) 461 (5.2) 534 (4.0) 597 (3.5) 648 (3.5) 676 (4.4) french Community 485 (4.4) 108 (2.8) 288 (10.3) 340 (8.5) 415 (5.7) 495 (5.4) 564 (4.5) 616 (4.9) 645 (5.7) german-speaking Community 520 (2.6) 97 (2.4) 348 (11.9) 392 (7.5) 459 (6.9) 529 (4.4) 586 (4.3) 638 (5.7) 668 (7.1) canada Alberta 531 (5.1) 98 (2.3) 362 (7.1) 400 (7.8) 467 (8.1) 536 (6.3) 600 (5.6) 652 (6.5) 685 (6.4) British Columbia 535 (3.5) 94 (2.3) 379 (8.3) 409 (5.7) 471 (4.8) 538 (4.3) 599 (5.1) 653 (4.8) 685 (6.2) (5.0) Manitoba 504 (3.6) 102 (3.3) 332 (13.2) 375 (6.2) 440 (5.1) 507 (3.9) 576 (3.9) 627 (5.6) 659 New Brunswick 515 (3.1) 92 (2.2) 353 395 (6.2) 456 (5.0) 520 (3.7) 579 (5.4) 627 (6.0) 656 (10.9) (8.3) Newfoundland and Labrador 504 (7.3) 100 (6.2) 329 (17.9) 376 (19.2) 445 (9.2) 511 (6.5) 572 (4.5) 626 (5.9) 655 (7.2) Nova Scotia 512 (5.7) 92 (3.0) 359 392 452 (10.7) 515 (8.0) 575 (6.0) 625 (6.4) 656 (8.6) (8.3) (8.7) (9.7) Ontario 528 (5.7) 103 (3.1) 356 (7.9) 399 (8.4) 461 (6.3) 530 (6.0) 597 (5.8) 656 (7.5) 691 Prince Edward Island 493 (2.6) 90 (2.1) 342 (6.9) 376 (5.6) 435 (4.5) 495 (3.8) 553 (4.3) 605 (4.4) 636 (4.9) Quebec 525 (4.5) 102 (3.8) 349 (11.1) 397 (7.2) 465 (4.9) 531 (4.3) 593 (5.0) 648 (5.8) 680 (7.5) Saskatchewan 515 (2.8) 93 (1.9) 357 393 (5.9) 453 (4.2) 517 (4.0) 579 (5.2) 635 (5.1) 665 (5.5) (8.2) italy Centre 514 (10.8) 93 (5.5) 345 (17.4) 389 (16.3) 459 (18.0) 524 (11.2) 577 (9.1) 625 (11.6) 653 (12.9) North East 527 (6.4) 91 (3.7) 367 (17.3) 409 (12.9) 470 (8.7) 533 (7.9) 589 (6.7) 636 (7.5) 665 (9.1) North West 533 (8.6) 83 (3.4) 392 (13.0) 428 (11.4) 480 (10.3) 539 (9.3) 590 (9.1) 634 (9.6) 661 (9.4) (8.6) South 474 (8.4) 82 (4.5) 344 (23.2) 377 (13.3) 424 (9.7) 476 (8.2) 529 (8.6) 574 (10.6) 599 South Islands 486 (8.5) 90 (4.0) 339 (14.3) 374 (11.5) 428 (10.2) 485 (9.3) 548 (8.7) 600 (12.1) 634 (12.2) 506 (13.4) 90 (5.2) 348 (18.3) 388 (17.9) 447 (14.9) 511 (13.0) 569 (14.8) 619 (16.4) 645 (21.5) 371 436 Portugal Alentejo Partners Spain Basque Country• 496 (3.9) 97 (2.5) 330 (4.6) 501 (4.1) 562 (4.1) 616 (4.3) 648 (4.2) Catalonia• 488 (8.4) 103 (5.4) 302 (18.3) 350 (16.8) 428 (10.7) 495 (9.0) 559 (6.7) 614 (8.8) 645 (9.9) Madrid 507 (13.0) 97 (4.8) 345 (14.3) 378 (15.9) 439 (15.0) 513 (14.9) (7.7) (5.6) 575 (15.1) 627 (16.5) 660 (17.9) brazil Central-West region 441 (11.9) 87 (5.2) 297 (19.6) 331 (17.8) 384 (15.6) 441 (13.2) 498 (13.1) 552 (12.2) 582 (16.2) Northeast region 393 (11.0) 105 (8.2) 227 (18.0) 262 (13.9) 324 (11.2) 390 (12.3) 460 (15.3) 524 (18.7) 569 (25.9) North region 383 (10.9) 83 (5.0) 253 (19.6) 284 (12.5) 327 (11.1) 377 (13.2) 437 (14.9) 495 (14.7) 528 (16.1) Southeast region 447 (6.3) 83 (2.4) 309 (8.1) 341 (6.7) 390 (6.9) 447 (6.9) 504 (8.4) 554 (8.5) 578 South region 435 (7.8) 82 (2.6) 301 (9.9) 330 (13.0) 379 (9.3) 435 (8.9) 488 (8.9) 541 (9.6) 573 (11.8) (7.4) 352 (6.5) 411 (6.5) 467 (6.3) 518 (7.3) 549 (7.2) 339 (12.1) 402 (9.2) 460 (8.4) 512 (7.0) 537 (8.0) (5.3) 535 (6.5) 564 (8.1) (7.5) colombia Bogotá 411 (5.7) 84 (2.6) 272 Cali 398 (9.0) 90 (4.4) 245 (20.2) 302 (7.0) 278 (16.0) Manizales 425 (4.3) 86 (2.6) 284 (7.6) 314 (7.3) 367 (6.5) 426 (5.9) 481 Medellín 424 (7.6) 95 (5.1) 274 (9.8) 305 (7.4) 359 (7.8) 419 (9.7) 487 (11.9) (6.0) 550 (13.9) 589 (18.7) united arab Emirates Abu Dhabi• 391 (5.3) 109 (2.8) 212 (8.1) 250 (6.8) 319 (6.8) 394 Ajman 375 (8.0) 80 (3.6) 242 (11.8) 273 (8.5) 320 (9.0) 373 (10.0) 466 (6.1) 431 (10.2) 529 (5.7) 481 (12.1) 568 (6.9) 507 (14.4) Dubai• 457 (1.3) 108 (1.1) 274 (4.0) 316 (3.0) 383 (2.8) 458 (2.7) 533 (2.9) 595 (3.2) 630 (4.5) fujairah 395 (4.0) 81 (2.6) 262 (8.1) 290 (6.2) 340 (6.5) 394 (5.6) 448 (6.6) 501 (6.5) 531 (9.6) ras al-khaimah 373 (11.9) 95 (11.3) 205 (51.0) 253 (28.4) 318 (15.5) 379 (9.3) 433 (9.9) 486 (8.6) 516 (11.4) Sharjah 416 (8.6) 85 (6.2) 273 (19.7) 305 (16.0) 361 (11.2) 416 (8.8) 474 (9.2) 526 (11.0) 557 (12.5) umm al-Quwain 372 (3.5) 81 (2.9) 241 (11.1) 270 315 369 (6.9) 427 (7.9) 476 (10.0) 506 (12.0) (9.6) (5.7) • PISA adjudicated region. Notes: Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), North East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), North West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). Brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), Northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), North region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.2.2 for national data. 1 2 http://dx.doi.org/10.1787/888933003763 226 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For regIonS wIThIn counTrIeS: Annex b2 table b2.v.2 [Part 2/2] mean score and variation in student performance in problem solving, by region range of performance OECD inter-quartile range (75th minus 25th percentile) inter-decile range (90th minus 10th percentile) top range (90th minus 50th percentile) bottom range (50th minus 10th percentile) range S.E. range S.E. range S.E. range S.E. Australian Capital Territory 136 (7.2) 262 (12.9) 116 (7.2) 146 (10.6) New South Wales 134 (4.5) 258 (6.7) 126 (4.2) 133 (4.9) Northern Territory 155 (12.3) 289 (26.9) 129 (21.2) 160 (16.4) Queensland 132 (5.8) 248 (6.6) 120 (4.5) 128 (6.0) South Australia 126 (5.8) 239 (8.4) 116 (4.9) 123 (6.9) Tasmania 143 (7.4) 272 (10.7) 139 (8.8) 133 (7.8) Victoria 130 (4.8) 245 (6.5) 118 (4.5) 127 (5.5) Western Australia 129 (5.7) 245 (8.8) 114 (5.4) 131 (7.2) australia belgium flemish Community• 136 (5.2) 262 (6.8) 114 (3.2) 148 (5.5) french Community 148 (5.0) 276 (8.8) 121 (4.8) 155 (8.0) german-speaking Community 126 (8.9) 245 (9.6) 108 (6.7) 137 (8.5) Alberta 133 (7.2) 252 (7.6) 116 (6.4) 136 (7.0) British Columbia 128 (5.1) 244 (7.1) 115 (5.1) 128 (5.2) Manitoba 136 (4.8) 252 (8.0) 120 (5.1) 132 (6.2) New Brunswick 123 (7.2) 232 (8.4) 107 (6.0) 125 (6.0) Newfoundland and Labrador 127 (8.3) 250 (19.1) 115 (6.4) 134 (15.6) Nova Scotia 123 (8.9) 233 (10.6) 110 (8.9) 123 (7.5) Ontario 136 (4.8) 257 (8.5) 125 (5.2) 131 (6.8) Prince Edward Island 118 (5.3) 228 (7.1) 110 (5.3) 119 (6.6) Quebec 128 (4.3) 251 (8.3) 117 (4.5) 135 (6.6) Saskatchewan 126 (6.2) 242 (8.5) 117 (5.1) 125 (7.1) Centre 118 (14.5) 235 (17.9) 100 (10.0) 135 (12.2) North East 119 (7.4) 228 (14.2) 103 (7.2) 125 (12.8) North West 110 (7.6) 206 (10.9) 95 (6.4) 111 (8.7) South 106 (7.5) 197 (12.1) 98 (7.7) 99 (9.7) South Islands 121 (8.0) 226 (13.6) 115 (10.2) 111 (8.7) 122 (10.4) 231 (15.8) 108 (10.1) 123 (10.3) canada italy Portugal Alentejo Partners Spain Basque Country• 125 (3.7) 245 (5.8) 115 (3.8) 130 (4.4) Catalonia• 131 (8.2) 263 (16.1) 119 (7.4) 144 (13.3) Madrid 136 (13.2) 249 (17.2) 114 (11.0) 135 (14.6) Central-West region 115 (12.9) 221 (18.5) 111 (12.8) 110 (13.0) Northeast region 137 (12.9) 263 (22.8) 134 (14.9) 128 (16.6) North region 110 (10.6) 211 (16.7) 118 (13.9) 92 (10.9) Southeast region 114 (5.9) 214 (8.0) 107 (6.3) 106 (5.7) South region 108 (6.5) 211 (11.9) 107 (9.5) 105 (12.2) Bogotá 115 (5.7) 216 (7.9) 106 (6.0) 110 (6.0) Cali 121 (7.4) 234 (14.6) 110 (7.2) 123 (12.2) Manizales 113 (6.2) 221 (8.6) 109 (6.5) 112 (6.7) Medellín 128 (9.8) 244 (14.8) 131 (12.8) 114 (9.0) brazil colombia united arab Emirates Abu Dhabi• 147 (5.5) 279 (7.4) 136 (5.3) 143 (5.6) Ajman 111 (8.7) 208 (12.4) 108 (9.0) 100 (9.5) (4.1) Dubai• 150 (3.4) 279 (4.6) 137 (4.1) 142 fujairah 108 (7.3) 210 (9.1) 107 (8.6) 103 (7.7) ras al-khaimah 115 (12.9) 233 (27.5) 108 (8.0) 125 (24.0) Sharjah 113 (12.4) 220 (19.0) 110 (10.3) 110 (12.9) umm al-Quwain 112 (9.0) 206 (13.6) 107 (11.1) 99 (10.6) • PISA adjudicated region. Notes: Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), North East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), North West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). Brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), Northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), North region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.2.2 for national data. 1 2 http://dx.doi.org/10.1787/888933003763 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 227 Annex b2: reSulTS For regIonS wIThIn counTrIeS table b2.v.3 [Part 1/3] relative performance in problem solving compared with performance in mathematics, reading and science, by region relative performance in problem solving compared with students around the world1 with similar scores in… … mathematics, reading and science (expected performance) Partners OECD relative performance across all students2 (actual minus expected score) australia Australian Capital Territory New South Wales Northern Territory Queensland South Australia Tasmania Victoria Western Australia belgium flemish Community• french Community german-speaking Community canada Alberta British Columbia Manitoba New Brunswick Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan italy Centre North East North West South South Islands Portugal Alentejo Spain Basque Country• Catalonia• Madrid brazil Central-West region Northeast region North region Southeast region South region colombia Bogotá Cali Manizales Medellín united arab Emirates Abu Dhabi• Ajman Dubai• fujairah ras al-khaimah Sharjah umm al-Quwain Percentage of students who perform above their expected score3 … mathematics difference in relative relative performance relative performance performance: strong among strong and top among moderate and and top performers minus performers low performers relative moderate and low in mathematics in mathematics performance across performers (at or above level 4)4 (at or below level 3)4 all students4 Score dif. S.E. % S.E. Score dif. S.E. Score dif. -2 6 40 7 15 -5 9 2 (2.2) (2.6) (6.4) (3.1) (3.0) (2.2) (3.2) (3.9) 51.1 54.6 75.0 56.5 61.9 46.0 57.9 52.5 (1.7) (2.1) (4.5) (2.3) (2.6) (2.1) (2.7) (3.0) 2 8 44 9 18 -2 12 5 (2.2) (2.5) (6.0) (3.1) (3.2) (2.3) (3.2) (3.9) 8 13 48 13 20 12 19 7 -5 -16 5 (2.4) (3.9) (2.2) 45.9 39.0 50.0 (1.9) (2.4) (2.1) -9 -19 1 (2.4) (3.9) (2.2) 2 1 -1 4 -3 1 3 -1 -8 -1 (3.7) (3.5) (2.6) (1.9) (4.9) (4.1) (3.9) (2.8) (3.7) (2.5) 51.4 50.1 50.9 54.8 49.4 52.0 53.0 48.5 45.8 48.5 (2.9) (2.9) (1.9) (2.1) (3.5) (3.6) (2.3) (1.9) (2.2) (2.2) 7 6 -1 2 1 3 6 -1 -15 -1 11 4 15 10 9 (7.2) (4.9) (8.4) (7.5) (8.2) 57.0 53.3 61.3 55.8 55.5 (5.1) (4.1) (5.5) (5.4) (5.3) 7 (10.0) 55.7 -17 -15 -3 (3.0) (7.6) (9.1) 20 -9 -7 15 3 S.E. Score dif. S.E. Score dif. S.E. (4.6) (2.9) (13.8) (3.5) (4.4) (4.2) (4.0) (4.9) -3 6 43 7 18 -6 10 4 (3.5) (3.1) (6.3) (3.4) (3.6) (2.8) (3.5) (4.6) 11 7 4 5 2 19 9 3 (6.8) (3.3) (14.0) (3.0) (4.5) (5.2) (3.7) (5.3) -7 -15 0 (2.7) (4.8) (3.2) -11 -21 2 (3.2) (4.5) (3.1) 4 5 -2 (3.7) (5.3) (4.8) (3.6) (3.6) (2.7) (1.9) (4.8) (3.8) (4.1) (2.8) (3.8) (2.6) 14 13 5 10 8 8 12 -45 -13 7 (4.7) (4.6) (2.8) (3.4) (3.3) (6.0) (4.2) (5.0) (4.3) (3.9) 2 2 -3 -1 -1 2 2 12 -16 -5 (4.0) (4.0) (3.4) (2.3) (6.3) (4.7) (4.6) (3.4) (4.6) (2.9) 12 12 8 11 9 6 10 -57 3 12 (4.5) (4.4) (4.1) (4.2) (6.9) (7.9) (3.8) (6.4) (4.6) (4.0) 10 3 16 7 7 (7.2) (5.1) (8.5) (7.3) (8.3) 4 -1 4 -16 -3 (5.6) (7.3) (9.7) (10.1) (10.1) 12 6 21 11 10 (8.9) (6.2) (9.0) (7.7) (9.1) -8 -8 -17 -27 -12 (7.8) (8.7) (7.8) (11.1) (11.1) (7.0) 5 (10.0) 3 (14.6) 6 (9.4) -3 (10.5) 39.8 43.9 48.3 (1.9) (4.0) (6.9) -20 -17 -2 (3.0) (7.8) (8.9) -13 -16 5 (3.2) (8.6) (12.8) -23 -17 -5 (3.5) (8.4) (7.9) 9 2 9 (3.2) (7.2) (9.0) (8.9) (8.0) (11.1) (4.5) (6.8) 68.8 43.7 44.3 63.1 51.2 (7.8) (6.5) (9.4) (3.5) (6.1) 19 -10 -7 15 1 (9.8) (7.8) (10.7) (4.7) (7.3) 32 38 -16 17 0 (13.9) (20.6) (29.2) (9.1) (21.4) 18 -11 -7 15 1 (10.0) (7.6) (10.7) (4.8) (7.3) 14 49 -9 2 -1 (14.2) (19.6) (25.8) (8.6) (20.6) -10 -11 -6 3 (5.7) (7.4) (4.8) (5.2) 43.9 45.5 45.0 53.4 (4.1) (4.2) (4.2) (4.7) -9 -9 -4 5 (5.8) (7.4) (4.8) (5.3) 21 16 -15 23 (18.3) (22.4) (23.4) (5.8) -9 -10 -3 4 (5.8) (7.4) (4.4) (5.5) 30 26 -11 19 (17.3) (21.2) (21.1) (6.7) -53 -54 -23 -39 -65 -43 -51 (3.5) (5.7) (1.2) (6.5) (8.7) (6.9) (2.8) 20.5 16.4 35.1 26.2 12.9 22.0 15.7 (1.7) (3.4) (1.0) (4.4) (2.4) (3.7) (2.2) -52 -54 -23 -40 -67 -43 -52 (3.6) (6.1) (1.2) (6.3) (9.0) (7.3) (3.0) -43 -72 -5 -56 -49 -51 -64 (7.0) (15.8) (2.6) (11.7) (11.4) (11.2) (23.5) -53 -53 -28 -39 -68 -42 -52 (3.7) (6.1) (1.4) (6.2) (9.3) (7.9) (3.1) 11 -19 22 -17 18 -9 -12 (7.0) (15.4) (2.9) (10.8) (12.9) (12.5) (24.2) • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.2.6 for national data. 1. “Students around the world” refers to 15-year-old students in countries that participated in the PISA 2012 assessment of problem solving. national samples are weighted according to the size of the target population using inal student weights. 2. This column reports the difference between actual performance and the itted value from a regression using a second-degree polynomial as regression function (math, math sq., read, read sq., scie, scie sq., math×read, math×scie, read×scie). 3. This column reports the percentage of students for whom the difference between actual performance and the itted value from a regression is positive. Values that are indicated in bold are signiicantly larger or smaller than 50%. 4. This column reports the difference between actual performance and the itted value from a regression using a cubic polynomial as regression function. 1 2 http://dx.doi.org/10.1787/888933003763 228 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For regIonS wIThIn counTrIeS: Annex b2 table b2.v.3 [Part 2/3] relative performance in problem solving compared with performance in mathematics, reading and science, by region relative performance in problem solving compared with students around the world1 with similar scores in… ... reading Partners OECD relative performance across all students4 Score dif. S.E. australia Australian Capital Territory New South Wales Northern Territory Queensland South Australia Tasmania Victoria Western Australia belgium flemish Community• french Community german-speaking Community canada Alberta British Columbia Manitoba New Brunswick Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan italy Centre North East North West South South Islands Portugal Alentejo Spain Basque Country• Catalonia• Madrid brazil Central-West region Northeast region North region Southeast region South region colombia Bogotá Cali Manizales Medellín united arab Emirates Abu Dhabi• Ajman Dubai• fujairah ras al-khaimah Sharjah umm al-Quwain ... Science relative relative performance performance among strong among moderate and top and low performers performers in reading in reading (at or above (at or below level 4)4 level 3)4 Score Score dif. S.E. dif. S.E. difference in relative performance: strong and top performers minus moderate and low performers Score dif. S.E. relative performance across all students4 Score dif. S.E. relative relative performance performance among strong among moderate and top and low performers performers in science in science (at or above (at or below level 4)4 level 3)4 Score Score dif. S.E. dif. S.E. difference in relative performance: strong and top performers minus moderate and low performers Score dif. S.E. 2 12 34 12 16 -1 7 10 (2.7) (2.8) (6.4) (3.3) (3.2) (2.3) (3.6) (4.1) 6 13 32 9 18 4 7 8 (5.5) (3.3) (12.9) (4.1) (4.8) (8.0) (4.7) (5.7) -1 11 34 14 15 -4 6 11 (3.0) (3.4) (6.3) (3.7) (3.3) (2.8) (3.8) (4.7) 7 3 -2 -5 2 8 1 -3 (6.6) (4.0) (12.8) (4.0) (4.5) (9.5) (4.3) (6.3) -4 3 24 5 8 -12 7 -1 (2.3) (2.8) (6.7) (3.1) (3.3) (2.4) (3.5) (3.8) -5 1 16 4 1 -10 4 -5 (4.6) (3.4) (13.3) (3.7) (4.4) (5.2) (4.3) (4.2) -2 4 27 6 11 -12 9 2 (3.3) (3.3) (6.0) (3.3) (3.7) (2.7) (3.9) (4.8) -2 -3 -11 -2 -10 3 -4 -7 (6.4) (3.7) (12.6) (3.3) (4.8) (5.9) (4.3) (5.1) 7 -16 17 (2.6) (4.1) (2.2) 12 -23 -2 (3.1) (5.2) (4.5) 4 -13 26 (3.2) (4.6) (3.0) 7 -10 -28 (3.8) (5.7) (5.9) 8 -6 12 (2.5) (4.1) (2.3) 9 -2 11 (2.9) (4.8) (4.6) 8 -8 13 (3.0) (4.8) (2.9) 1 6 -2 (3.3) (5.5) (5.9) 8 4 4 15 -2 2 2 -2 6 8 (4.3) (3.9) (2.6) (2.5) (5.4) (5.1) (4.1) (2.8) (3.5) (2.7) 10 3 7 7 -9 -4 2 -48 -1 5 (5.3) (4.4) (2.9) (4.1) (5.0) (5.0) (5.3) (5.4) (4.4) (4.0) 6 4 2 18 1 4 2 14 10 9 (4.8) (5.0) (3.4) (2.8) (7.4) (6.3) (4.4) (3.4) (4.3) (3.1) 4 -1 5 -11 -10 -8 0 -62 -11 -4 (5.3) (5.5) (4.1) (4.5) (8.6) (6.4) (4.9) (6.6) (5.2) (4.8) -3 -3 0 8 -10 -3 5 -1 10 0 (4.2) (3.5) (2.8) (2.3) (5.8) (5.5) (3.7) (2.8) (3.8) (2.5) -2 -2 1 4 -11 -8 6 -45 13 -1 (4.9) (4.3) (3.1) (5.2) (3.5) (5.8) (4.8) (5.6) (4.3) (4.2) -3 -3 -1 10 -9 -1 4 12 9 0 (4.9) (4.2) (3.5) (2.7) (8.0) (7.1) (3.9) (3.3) (4.4) (2.8) 1 1 2 -6 -2 -7 2 -57 4 -2 (5.2) (4.8) (4.1) (6.2) (8.2) (8.1) (4.1) (6.9) (4.4) (4.7) 19 15 25 7 9 (8.6) (5.5) (8.4) (8.3) (7.6) 4 -3 6 -31 -5 (7.7) (5.3) (8.6) (12.6) (10.8) 25 25 34 13 12 (10.8) (6.8) (9.7) (8.3) (8.5) -22 -28 -29 -44 -17 (11.3) (6.5) (8.3) (13.8) (12.1) 10 4 15 13 11 (7.6) (5.2) (8.3) (7.7) (8.2) 1 -3 -2 -20 -8 (7.3) (6.1) (8.9) (11.4) (12.3) 13 8 24 17 15 (9.4) (6.6) (9.1) (7.6) (8.2) -12 -11 -26 -38 -23 (9.2) (8.0) (8.1) (11.4) (10.9) 11 (9.4) 8 (16.8) 12 (8.7) -5 (14.0) 10 (11.0) 9 (17.6) 10 (10.4) -1 (13.4) -6 -16 0 (3.2) (7.8) (8.7) -6 -25 2 (3.6) (8.4) (11.6) -6 -12 -1 (3.6) (8.5) (8.3) 0 -12 3 (3.8) (7.5) (7.9) -11 -7 -5 (3.1) (7.2) (10.4) -10 -2 2 (3.4) (7.0) (13.4) -11 -8 -9 (3.5) (7.8) (9.9) 1 7 11 (3.6) (6.7) (9.2) 6 -25 -29 3 -9 (7.9) (9.8) (12.9) (4.6) (5.8) 9 3 -34 -10 -13 (14.4) (21.5) (21.5) (10.9) (14.9) 6 -26 -29 4 -9 (8.0) (9.9) (13.0) (4.7) (5.8) 3 29 -5 -14 -5 (13.7) (20.6) (21.7) (11.3) (14.4) 15 -18 -18 11 3 (7.0) (8.4) (10.8) (4.2) (6.7) 38 24 -7 7 -2 (14.5) (22.7) (20.0) (9.9) (20.9) 14 -20 -18 11 3 (7.0) (8.3) (10.8) (4.2) (6.6) 24 44 11 -4 -5 (14.9) (21.4) (23.3) (9.5) (20.0) -31 -34 -25 -19 (5.3) (7.1) (5.1) (6.1) -18 -33 -39 -1 (14.0) (10.9) (11.9) (13.3) -32 -34 -24 -20 (5.3) (7.3) (5.0) (6.3) 14 1 -15 19 (13.1) (12.0) (10.8) (13.2) -17 -23 -17 -10 (6.1) (7.9) (5.1) (4.8) -3 -23 -25 21 (21.4) (16.5) (17.6) (8.8) -18 -23 -16 -11 (6.1) (8.0) (4.9) (5.0) 15 0 -9 32 (20.0) (16.8) (16.5) (9.2) -58 -62 -22 -43 -64 -49 -54 (3.9) (4.9) (1.4) (6.2) (9.2) (5.5) (3.2) -47 -90 -9 -76 -63 -53 -65 (7.7) (10.5) (2.8) (15.4) (21.4) (12.8) (15.4) -60 -60 -26 -42 -64 -48 -54 (4.0) (5.1) (1.6) (6.2) (9.4) (5.6) (3.4) 13 -30 16 -34 1 -5 -11 (7.9) (10.7) (3.1) (17.6) (22.1) (12.5) (16.1) -61 -62 -24 -46 -72 -44 -60 (3.4) (5.2) (1.2) (4.6) (8.9) (7.8) (2.6) -53 -86 -11 -53 -59 -60 -68 (6.3) (13.4) (2.8) (9.6) (14.4) (9.8) (15.1) -62 -60 -27 -45 -72 -42 -60 (3.6) (5.1) (1.5) (4.7) (9.1) (8.0) (2.8) 9 -26 16 -8 14 -17 -8 (6.3) (12.2) (3.4) (10.4) (15.0) (9.6) (15.9) • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.2.6 for national data. 1. “Students around the world” refers to 15-year-old students in countries that participated in the PISA 2012 assessment of problem solving. national samples are weighted according to the size of the target population using inal student weights. 2. This column reports the difference between actual performance and the itted value from a regression using a second-degree polynomial as regression function (math, math sq., read, read sq., scie, scie sq., math×read, math×scie, read×scie). 3. This column reports the percentage of students for whom the difference between actual performance and the itted value from a regression is positive. Values that are indicated in bold are signiicantly larger or smaller than 50%. 4. This column reports the difference between actual performance and the itted value from a regression using a cubic polynomial as regression function. 1 2 http://dx.doi.org/10.1787/888933003763 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 229 Annex b2: reSulTS For regIonS wIThIn counTrIeS table b2.v.3 [Part 3/3] relative performance in problem solving compared with performance in mathematics, reading and science, by region Partners OECD relative performance in problem solving compared with students in countries that also assessed mathematics on computers who have similar scores in… australia Australian Capital Territory New South Wales Northern Territory Queensland South Australia Tasmania Victoria Western Australia belgium flemish Community• french Community german-speaking Community canada Alberta British Columbia Manitoba New Brunswick Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan italy Centre North East North West South South Islands Portugal Alentejo Spain Basque Country• Catalonia• Madrid brazil Central-West region Northeast region North region Southeast region South region colombia Bogotá Cali Manizales Medellín united arab Emirates Abu Dhabi• Ajman Dubai• fujairah ras al-khaimah Sharjah umm al-Quwain ...Paper-based mathematics (a) ...computer-based mathematics (b) relative performance across all students4 relative performance across all students4 mode effects: Score-point difference attributed to computer delivery (a - b) Score dif. S.E. Score dif. S.E. Score dif. S.E. 0 7 43 8 17 -3 11 4 (2.3) (2.6) (6.1) (3.1) (3.2) (2.3) (3.3) (4.0) 11 15 34 13 17 4 9 12 (2.4) (2.8) (5.5) (3.4) (3.4) (2.3) (3.4) (4.1) -11 -8 9 -5 0 -7 2 -8 (1.4) (1.9) (3.3) (2.5) (3.5) (1.6) (2.2) (3.5) -11 -21 0 (2.4) (4.0) (2.2) -4 -11 6 (2.7) (4.0) (2.5) -7 -10 -6 (2.1) (2.7) (1.8) 5 5 -2 1 -1 2 4 -3 -16 -2 (3.7) (3.6) (2.7) (2.0) (4.8) (3.8) (4.1) (2.8) (3.8) (2.7) 13 4 5 14 -10 5 -2 -3 0 11 (5.0) (4.2) (3.1) (2.6) (5.1) (3.2) (3.9) (3.5) (4.4) (3.1) -8 1 -7 -13 10 -3 6 0 -16 -13 (3.2) (3.0) (1.6) (1.9) (1.4) (2.5) (3.2) (2.8) (2.5) (2.1) 8 2 14 6 6 (7.2) (5.2) (8.5) (7.3) (8.3) 10 12 8 -11 8 (8.3) (7.0) (7.3) (7.5) (6.5) -2 -11 6 16 -3 (5.3) (5.4) (5.3) (7.4) (5.8) 4 (10.1) 15 (9.2) -12 (6.2) -21 -19 -3 (3.0) (7.7) (9.0) 0 -1 9 (3.2) (8.3) (9.6) -21 -17 -13 (2.1) (5.2) (3.5) 17 -11 -9 13 -1 (9.8) (7.7) (10.7) (4.6) (7.2) 8 -22 -35 0 -6 (7.7) (6.6) (12.5) (4.9) (6.3) 9 11 26 13 5 (6.2) (6.0) (5.7) (3.9) (7.5) -11 -11 -6 3 (5.9) (7.5) (4.8) (5.3) -16 -17 1 -4 (6.2) (7.9) (5.0) (4.5) 5 6 -7 7 (3.4) (8.1) (2.4) (4.2) -54 -56 -25 -42 -69 -45 -54 (3.6) (6.1) (1.3) (6.4) (9.0) (7.3) (3.1) -46 -34 -13 -45 -58 -37 -37 (3.4) (3.3) (1.3) (4.4) (10.9) (5.3) (3.5) -8 -22 -12 3 -11 -8 -17 (3.0) (4.9) (1.0) (4.9) (4.8) (5.7) (2.8) • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.2.6 for national data. 1. “Students around the world” refers to 15-year-old students in countries that participated in the PISA 2012 assessment of problem solving. national samples are weighted according to the size of the target population using inal student weights. 2. This column reports the difference between actual performance and the itted value from a regression using a second-degree polynomial as regression function (math, math sq., read, read sq., scie, scie sq., math×read, math×scie, read×scie). 3. This column reports the percentage of students for whom the difference between actual performance and the itted value from a regression is positive. Values that are indicated in bold are signiicantly larger or smaller than 50%. 4. This column reports the difference between actual performance and the itted value from a regression using a cubic polynomial as regression function. 1 2 http://dx.doi.org/10.1787/888933003763 230 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For regIonS wIThIn counTrIeS: Annex b2 table b2.v.4 [Part 1/2] Percentage of students at each proiciency level in problem solving, by gender and by region boys OECD below level 1 (below 358.49 score points) level 1 (from 358.49 to less than 423.42 score points) level 2 (from 423.42 to less than 488.35 score points) level 3 (from 488.35 to less than 553.28 score points) level 4 (from 553.28 to less than 618.21 score points) level 5 (from 618.21 to less than 683.14 score points) level 6 (above 683.14 score points) % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. (1.3) australia Australian Capital Territory 7.7 (1.7) 10.5 (2.3) 17.1 (1.9) 23.8 (2.8) 21.7 (3.4) 13.8 (2.9) 5.5 New South Wales 6.0 (0.8) 10.8 (1.2) 18.6 (1.5) 24.3 (1.4) 21.2 (1.3) 12.7 (1.3) 6.5 (1.2) Northern Territory 10.3 (2.2) 12.9 (3.3) 16.5 (3.7) 17.2 (3.8) 21.1 (4.3) 14.3 (3.8) 7.8 (4.3) Queensland 4.9 (0.8) 11.4 (1.1) 19.4 (1.4) 24.6 (1.6) 23.3 (1.4) 11.6 (1.2) 4.9 (1.0) South Australia 5.1 (0.9) 11.1 (1.5) 20.5 (2.2) 25.8 (2.1) 22.0 (2.1) 11.6 (1.7) 3.8 (0.8) 12.1 (1.6) 16.8 (2.5) 20.9 (2.9) 20.9 (2.2) 17.1 (2.2) 8.4 (1.6) 3.9 (1.1) Victoria 4.6 (1.1) 10.7 (1.7) 18.9 (1.4) 26.5 (2.0) 22.4 (1.5) 12.8 (1.3) 4.1 (0.8) Western Australia 3.8 (0.9) 8.8 (1.3) 16.8 (1.6) 24.8 (2.0) 26.3 (2.1) 14.5 (1.6) 5.0 (1.5) Tasmania belgium flemish Community• 6.3 (0.8) 9.3 (1.2) 16.0 (1.2) 24.0 (1.1) 24.2 (1.4) 15.2 (1.2) 5.0 (0.6) 13.7 (1.5) 14.6 (1.1) 18.2 (1.3) 22.2 (1.5) 19.7 (1.3) 9.4 (1.1) 2.2 (0.6) 5.5 (1.2) 8.8 (1.6) 16.5 (2.4) 22.4 (2.7) 26.6 (2.9) 14.9 (1.9) 5.3 (1.2) Alberta 4.5 (0.8) 9.2 (1.3) 15.9 (1.5) 26.3 (2.0) 25.0 (2.1) 13.4 (1.5) 5.7 (1.1) British Columbia 2.9 (0.7) 9.1 (1.1) 17.5 (1.5) 26.3 (1.9) 23.1 (1.7) 14.7 (1.9) 6.5 (1.1) Manitoba 7.4 (1.4) 12.7 (1.6) 21.7 (2.4) 24.5 (2.6) 21.6 (1.9) 9.2 (1.6) 2.9 (0.7) New Brunswick 6.4 (1.3) 11.0 (1.6) 21.8 (2.1) 26.9 (3.3) 21.6 (1.9) 9.3 (2.1) 3.0 (0.8) Newfoundland and Labrador 9.9 (2.8) 12.2 (1.8) 19.8 (1.9) 25.5 (2.6) 21.5 (2.4) 9.0 (1.4) 2.1 (0.8) Nova Scotia 6.5 (2.1) 10.6 (1.8) 21.0 (3.2) 27.4 (3.8) 21.7 (2.6) 10.2 (1.8) 2.6 (1.2) Ontario 5.0 (1.1) 9.2 (1.3) 18.9 (1.7) 23.6 (1.4) 22.3 (1.6) 13.7 (1.3) 7.3 (1.3) Prince Edward Island 7.6 (1.2) 13.8 (1.6) 24.9 (2.5) 29.1 (3.2) 17.3 (2.2) 6.0 (1.0) 1.2 (0.5) Quebec 6.5 (1.2) 9.3 (1.0) 16.0 (1.4) 25.4 (1.4) 24.2 (1.3) 13.2 (1.4) 5.4 (1.0) Saskatchewan 6.2 (1.2) 11.7 (1.6) 21.4 (2.1) 27.2 (2.2) 21.0 (1.8) 10.1 (1.2) 2.5 (0.7) Centre 6.9 (2.4) 9.7 (3.6) 14.3 (2.3) 30.0 (4.9) 24.8 (2.8) 11.8 (2.6) 2.6 (1.4) North East 5.4 (1.8) 7.2 (2.4) 13.3 (2.3) 22.1 (2.5) 29.0 (2.4) 17.7 (2.1) 5.3 (1.5) North West 2.9 (1.1) 6.9 (2.4) 18.1 (2.4) 25.9 (2.9) 28.2 (3.9) 14.7 (2.6) 3.2 (1.0) South 5.9 (2.0) 17.4 (3.9) 28.2 (3.6) 27.7 (3.2) 16.7 (3.7) 3.8 (1.2) 0.2 (0.3) South Islands 7.7 (2.2) 14.5 (3.4) 25.1 (3.2) 22.9 (3.5) 18.9 (2.9) 9.3 (2.9) 1.6 (1.1) 5.6 (2.0) 9.8 (2.4) 21.0 (3.4) 26.8 (4.0) 22.8 (3.0) 10.8 (3.3) 3.2 (2.2) french Community german-speaking Community canada italy Portugal Alentejo Spain Basque Country• 8.3 (1.0) 13.1 (1.0) 22.2 (1.1) 26.1 (1.3) 19.5 (1.2) 8.5 (0.8) 2.4 (0.5) 13.3 (2.7) 12.2 (1.7) 21.8 (2.1) 24.0 (2.4) 17.7 (2.2) 8.4 (1.5) 2.6 (1.1) 6.9 (2.0) 14.1 (2.9) 18.0 (3.6) 25.4 (3.2) 22.0 (3.8) 10.3 (3.2) 3.2 (1.5) Central-West region 13.4 (5.2) 20.7 (5.5) 30.5 (5.0) 21.9 (4.6) 9.9 (3.1) 3.0 (1.4) 0.6 (0.8) Northeast region 32.8 (4.8) 25.3 (3.7) 20.3 (3.2) 12.1 (3.0) 5.7 (2.1) 2.2 (1.0) 1.5 (1.0) North region 37.0 (7.4) 29.8 (4.7) 20.0 (4.8) 10.0 (4.4) 2.8 (1.5) 0.4 (0.3) 0.0 (0.0) Southeast region 12.4 (2.4) 22.5 (2.2) 28.7 (2.4) 22.7 (3.0) 11.2 (2.2) 2.1 (0.7) 0.5 (0.3) South region 16.6 (3.2) 23.3 (3.5) 30.6 (4.1) 19.5 (4.1) 7.8 (1.8) 2.1 (1.1) 0.1 (0.2) Bogotá 21.1 (2.7) 26.0 (2.6) 29.5 (2.3) 16.8 (2.3) 5.2 (1.6) 1.2 (0.7) 0.2 (0.2) Cali 28.8 (3.6) 27.6 (2.3) 24.2 (2.3) 14.8 (2.2) 3.9 (1.5) 0.6 (0.4) 0.2 (0.2) Manizales 14.7 (1.8) 23.6 (2.4) 30.8 (2.3) 20.4 (2.6) 8.3 (1.6) 1.6 (0.9) 0.7 (0.5) Medellín 19.5 (2.9) 26.5 (3.2) 24.4 (3.2) 17.3 (2.4) 9.1 (2.2) 2.5 (1.3) 0.6 (0.5) Abu Dhabi• 46.3 (3.1) 19.7 (1.9) 16.8 (1.5) 10.8 (1.3) 4.9 (1.0) 1.5 (0.5) 0.2 (0.2) Ajman 56.4 (4.5) 26.6 (4.5) 12.9 (3.4) 3.6 (1.7) 0.4 (0.6) 0.0 c 0.0 c Dubai• 21.7 (1.0) 18.5 (1.2) 20.8 (1.7) 19.0 (1.3) 13.3 (1.1) 5.1 (0.7) 1.5 (0.4) fujairah 32.7 (3.6) 31.4 (3.4) 20.1 (3.4) 10.8 (1.7) 4.3 (1.3) 0.7 (0.7) 0.0 c ras al-khaimah 47.5 (7.1) 28.6 (4.8) 15.2 (3.1) 7.0 (1.7) 1.3 (0.7) 0.4 (0.3) 0.0 c Sharjah 31.8 (8.1) 29.1 (5.6) 21.4 (4.5) 12.1 (3.8) 4.6 (2.2) 0.8 (0.8) 0.2 (0.4) umm al-Quwain 61.3 (5.1) 25.1 (4.4) 10.6 (3.9) 1.9 (1.9) 0.9 (0.5) 0.3 (0.4) 0.0 c Catalonia• Partners Madrid brazil colombia united arab Emirates • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.6 for national data. 1 2 http://dx.doi.org/10.1787/888933003763 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 231 Annex b2: reSulTS For regIonS wIThIn counTrIeS table b2.v.4 [Part 2/2] Percentage of students at each proiciency level in problem solving, by gender and by region increased increased likelihood of likelihood of level 5 level 4 level 3 level 2 level 1 boys scoring boys scoring (from 358.49 (from 423.42 (from 488.35 (from 553.28 (from 618.21 at or above below level 2 level 6 to less than to less than to less than to less than below level 1 to less than level 5 (less than (above 683.14 683.14 618.21 553.28 488.35 423.42 (below 358.49 (above 618.21 423.42 score points) score points) score points) score points) score points) score points) score points) score points) score points) Girls OECD % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. relative risk S.E. relative risk S.E. australia Australian Capital Territory 5.0 (1.5) 8.6 (1.6) 18.2 (2.3) 24.4 (3.1) 26.4 (3.3) 13.2 (2.2) 4.2 (1.7) 1.35 (0.31) 1.11 (0.23) New South Wales 4.5 (0.8) 9.8 (1.0) 19.3 (1.6) 26.9 (1.3) 22.9 (1.3) 12.6 (1.2) 4.0 (0.6) 1.18 (0.15) 1.15 (0.14) Northern Territory 8.0 (1.8) 11.9 (3.0) 19.6 (4.2) 26.0 (5.1) 22.0 (3.6) 10.2 (4.8) 2.3 (2.1) 1.17 (0.25) 1.81 (0.89) Queensland 4.9 (0.9) 10.1 (1.3) 20.2 (2.0) 27.1 (1.8) 22.2 (1.7) 11.9 (1.4) 3.7 (0.7) 1.09 (0.10) 1.06 (0.15) South Australia 3.8 (0.9) 10.3 (1.4) 20.6 (2.2) 28.6 (2.1) 22.0 (2.2) 12.0 (2.0) 2.8 (0.9) 1.16 (0.18) 1.04 (0.18) Tasmania 8.2 (1.4) 16.2 (1.9) 24.8 (2.4) 24.8 (2.8) 14.8 (2.7) 8.5 (2.2) 2.6 (0.8) 1.18 (0.12) 1.13 (0.31) Victoria 4.6 (0.8) 10.2 (1.4) 20.2 (1.7) 26.0 (2.2) 23.5 (2.1) 11.9 (1.4) 3.6 (0.8) 1.04 (0.12) 1.10 (0.15) Western Australia 5.2 (1.3) 9.9 (1.6) 20.4 (1.8) 27.0 (2.4) 22.9 (1.9) 10.9 (1.7) 3.7 (1.0) 0.83 (0.11) 1.34 (0.24) belgium flemish Community• french Community german-speaking Community 7.1 (1.1) 9.7 (1.1) 17.5 (1.2) 25.7 (1.5) 24.2 (1.3) 12.6 (1.3) 3.2 (0.5) 0.93 (0.13) 1.28 (0.12) 11.5 (1.1) 14.2 (1.1) 22.5 (2.0) 25.8 (1.3) 18.5 (1.5) 6.7 (1.0) 0.9 (0.4) 1.10 (0.08) 1.55 (0.22) 6.1 (1.3) 9.4 (1.7) 22.6 (3.0) 30.5 (3.8) 22.6 (2.8) 6.9 (1.6) 1.8 (0.7) 0.92 (0.17) 2.36 (0.61) canada Alberta 4.7 (0.8) 9.9 (1.3) 17.7 (2.0) 26.0 (2.7) 22.8 (2.3) 13.9 (1.5) 4.9 (1.0) 0.94 (0.11) 1.01 (0.11) British Columbia 3.4 (1.0) 9.7 (1.6) 18.9 (1.8) 25.9 (2.3) 25.0 (2.5) 12.9 (1.5) 4.2 (0.9) 0.91 (0.13) 1.24 (0.16) Manitoba 7.2 (1.4) 13.7 (1.8) 21.5 (1.8) 25.2 (2.3) 20.8 (2.0) 9.1 (1.1) 2.5 (0.6) 0.97 (0.13) 1.04 (0.15) New Brunswick 4.4 (0.9) 9.6 (1.4) 19.8 (2.1) 29.1 (2.6) 25.2 (2.5) 9.4 (1.7) 2.5 (1.0) 1.25 (0.20) 1.04 (0.25) Newfoundland and Labrador 5.3 (2.1) 10.4 (2.1) 23.3 (2.0) 28.3 (2.1) 20.6 (2.0) 9.7 (1.5) 2.5 (0.9) 1.41 (0.22) 0.91 (0.16) Nova Scotia 3.6 (1.4) 11.1 (2.7) 24.2 (4.4) 27.2 (2.4) 23.5 (4.0) 8.1 (1.3) 2.3 (1.0) 1.18 (0.23) 1.24 (0.27) Ontario 5.2 (0.9) 9.6 (1.3) 19.9 (1.7) 26.2 (1.7) 22.6 (1.6) 11.7 (1.3) 4.8 (0.9) 0.96 (0.10) 1.27 (0.11) Prince Edward Island 6.4 (1.0) 14.6 (1.7) 26.4 (1.9) 27.2 (2.5) 18.1 (1.7) 5.2 (1.4) 2.0 (0.7) 1.02 (0.11) 1.01 (0.26) Quebec 5.2 (0.8) 8.5 (0.9) 20.1 (1.6) 27.5 (1.5) 22.7 (1.3) 11.9 (1.2) 4.1 (0.8) 1.15 (0.12) 1.17 (0.12) Saskatchewan 4.1 (1.0) 10.5 (1.4) 20.8 (1.9) 29.0 (2.1) 20.5 (1.8) 11.8 (1.6) 3.3 (0.8) 1.23 (0.18) 0.83 (0.11) italy Centre 5.2 (1.7) 9.9 (2.7) 23.7 (4.3) 30.7 (3.4) 22.7 (4.4) 6.7 (2.4) 1.1 (0.7) 1.10 (0.40) 1.87 (0.50) North East 2.8 (1.2) 9.1 (2.6) 25.8 (4.0) 33.5 (2.4) 22.6 (4.1) 5.6 (1.4) 0.6 (0.4) 1.06 (0.44) 3.75 (0.84) 1.69 (0.56) North West 2.1 (0.8) 6.7 (2.1) 19.4 (3.4) 32.7 (4.2) 28.4 (4.1) 9.3 (3.1) 1.3 (1.1) 1.13 (0.41) South 7.6 (3.0) 18.2 (3.9) 36.3 (3.5) 27.4 (3.0) 10.1 (2.4) 0.5 (0.6) 0.0 c 0.91 (0.25) South Islands 7.2 (2.3) 18.0 (2.6) 30.5 (2.9) 27.8 (2.8) 12.8 (2.4) 2.9 (1.0) 0.8 (0.3) 0.88 (0.14) 2.97 (1.09) 6.4 (2.2) 12.6 (2.8) 25.8 (2.7) 29.3 (2.8) 19.5 (3.9) 6.0 (2.2) 0.5 (0.6) 0.82 (0.14) 2.18 (0.52) (0.14) 13.30 (22.50) Portugal Alentejo Partners Spain Basque Country • 7.7 (0.9) 13.3 (1.1) 24.1 (1.3) 28.5 (1.1) 18.0 (1.3) 6.6 (0.7) 1.8 (0.4) 1.02 (0.07) 1.30 Catalonia• 8.9 (2.4) 12.7 (2.1) 26.3 (3.2) 26.7 (2.2) 18.2 (2.3) 6.1 (1.3) 1.1 (0.5) 1.19 (0.16) 1.54 (0.38) Madrid 6.6 (2.5) 12.9 (3.1) 21.3 (3.2) 26.5 (3.9) 21.4 (3.6) 9.3 (3.4) 1.9 (1.3) 1.07 (0.17) 1.22 (0.31) brazil Central-West region 18.8 (5.1) 29.2 (4.1) 28.3 (3.9) 17.5 (5.0) 5.0 (2.0) 0.9 (0.6) 0.3 (0.3) 0.71 (0.12) 3.27 (2.46) Northeast region 42.2 (4.4) 25.0 (2.9) 19.8 (3.8) 9.5 (2.3) 2.3 (1.0) 1.0 (0.7) 0.2 (0.3) 0.87 (0.05) 3.15 (1.15) North region 42.9 (6.0) 30.7 (4.5) 15.9 (3.7) 8.1 (2.9) 2.3 (1.5) 0.1 (0.2) 0.0 c 0.91 (0.09) 4.91 (10.69) Southeast region 16.3 (2.1) 26.4 (2.6) 30.8 (2.3) 19.7 (2.9) 5.8 (1.0) 0.9 (0.4) 0.1 (0.1) 0.82 (0.05) 2.54 (1.35) South region 18.3 (3.8) 30.8 (3.8) 30.3 (3.6) 15.1 (3.3) 4.4 (1.4) 1.1 (0.6) 0.0 c 0.81 (0.09) 2.08 (1.92) colombia Bogotá 32.5 (3.0) 30.3 (2.4) 25.0 (2.5) 10.0 (1.6) 1.8 (0.6) 0.3 (0.3) 0.0 c 0.75 (0.05) 4.72 (6.92) Cali 33.8 (5.2) 28.4 (3.3) 24.8 (2.9) 10.6 (2.0) 2.1 (0.8) 0.3 (0.2) 0.0 c 0.91 (0.05) 2.38 (2.58) Manizales 28.7 (3.2) 30.1 (2.3) 27.1 (3.2) 11.1 (2.1) 2.5 (0.9) 0.4 (0.3) 0.1 (0.1) 0.65 (0.05) 6.08 (8.66) Medellín 29.8 (3.4) 27.3 (3.1) 23.1 (3.4) 12.8 (2.1) 4.5 (1.3) 2.0 (0.8) 0.4 (0.2) 0.81 (0.07) 1.29 (0.65) (0.40) united arab Emirates Abu Dhabi• 29.3 (2.8) 26.2 (2.0) 24.1 (1.6) 13.6 (1.5) 5.0 (0.9) 1.6 (0.6) 0.1 (0.1) 1.19 (0.07) 0.97 Ajman 29.7 (6.5) 31.4 (3.2) 25.7 (3.6) 12.3 (3.7) 0.9 (0.9) 0.0 c 0.0 c 1.36 (0.14) c c Dubai• 14.4 (0.6) 20.7 (1.4) 24.6 (1.4) 22.1 (1.3) 12.0 (1.0) 5.0 (0.7) 1.2 (0.3) 1.15 (0.04) 1.08 (0.15) fujairah 32.1 (4.3) 33.8 (4.2) 24.7 (4.0) 8.4 (1.8) 1.0 (0.8) 0.0 c 0.0 c 0.97 (0.08) c c ras al-khaimah 34.0 (5.6) 34.2 (3.1) 21.2 (3.6) 7.5 (2.1) 2.6 (1.3) 0.5 (0.5) 0.0 c 1.12 (0.10) 1.20 (3.28) Sharjah 18.1 (3.6) 29.6 (4.4) 29.8 (3.1) 17.0 (3.1) 4.9 (2.4) 0.5 (0.5) 0.1 (0.2) 1.28 (0.22) 2.32 (5.44) umm al-Quwain 28.9 (3.9) 32.4 (4.8) 26.2 (3.5) 10.5 (2.6) 2.1 (1.2) 0.0 c 0.0 c 1.41 (0.11) c c • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.6 for national data. 1 2 http://dx.doi.org/10.1787/888933003763 232 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For regIonS wIThIn counTrIeS: Annex b2 table b2.v.5 [Part 1/3] mean score and variation in student performance in problem solving, by gender and by region mean score boys OECD mean S.E. Girls mean S.E. Standard deviation difference (b - G) Score dif. S.E. boys S.d. S.E. Girls S.d. 5th percentile difference (b - G) S.E. dif. S.E. boys Score S.E. Girls Score S.E. difference (b - G) Score dif. S.E. australia Australian Capital Territory 522 (5.9) 529 (4.9) -7 (8.0) 109 (4.6) 96 (5.0) 13.4 (7.0) 336 (15.9) 359 (21.7) -23 (27.0) New South Wales 525 (5.1) 525 (4.1) 0 (6.0) 104 (2.8) 95 (2.6) 9.2 (3.5) 349 364 (9.1) -14 (11.2) 12 (14.0) 121 (7.5) 101 (6.5) 20.2 (7.2) 304 (19.8) 323 (17.1) -18 (26.8) 1 (4.7) 99 (3.0) 95 (2.8) 4.3 (3.5) 359 (9.0) 359 (10.2) 0 (11.6) Northern Territory 519 (10.1) 507 (11.1) Queensland 523 521 (4.1) (4.2) (7.9) South Australia 519 (4.7) 521 (4.8) -2 (5.0) 96 (2.9) 90 (2.9) 5.9 (3.8) 358 (9.2) 376 (12.1) -18 (13.6) Tasmania 489 (5.4) 491 (5.5) -3 (7.5) 110 (3.6) 100 (3.4) 10.5 (4.8) 311 (14.5) 326 (16.3) -14 (23.9) Victoria 524 (4.9) 522 (4.5) 2 (4.6) 95 (2.4) 94 (3.0) 1.3 (3.4) 362 (9.9) 365 (11.4) -3 (11.4) Western Australia 537 (5.5) 519 (5.6) 17 (7.6) 96 (3.5) 95 (3.5) 1.2 (4.1) 373 (10.7) 356 (11.0) 17 (13.4) belgium flemish Community• 530 (4.0) 519 (4.6) 11 (5.5) 103 (2.7) 100 (3.4) 3.1 (3.9) 342 (9.7) 338 (9.9) 4 (12.9) french Community 487 (5.2) 483 (4.9) 4 (4.8) 114 (3.8) 101 (2.6) 13.1 (3.3) 282 (11.4) 296 (13.9) -14 (15.1) german-speaking Community 533 (4.3) 507 (3.8) 26 (6.2) 101 (3.6) 89 (3.8) 11.9 (5.7) 352 (20.8) 345 (14.4) 6 (23.9) canada Alberta 533 (5.1) 529 (5.6) 5 (3.7) 99 (3.3) 97 (2.5) 2.2 (3.7) 363 (10.1) 361 (8.1) 2 (11.6) British Columbia 540 (4.0) 530 (5.1) 9 (5.9) 96 (2.8) 92 (2.9) 3.9 (3.3) 381 378 (11.3) 3 (10.9) (8.3) Manitoba 504 (4.5) 503 (5.1) 1 (6.3) 103 (3.9) 100 (4.6) 3.1 (5.4) 325 (21.2) 336 (16.0) -11 (27.4) New Brunswick 511 (4.7) 520 (4.0) -9 (6.1) 94 (3.1) 89 (3.3) 4.9 (4.7) 344 (12.8) 366 (12.6) -22 (17.7) Newfoundland and Labrador 496 (10.6) 512 (5.4) -16 (8.3) 109 (9.3) 90 (3.7) 19.4 (8.2) 290 (35.3) 355 (19.1) -66 (36.9) Nova Scotia 512 (5.5) 512 (8.0) -1 (7.4) 96 (3.9) 88 (3.8) 8.7 (4.9) 348 (18.0) 371 (14.7) -23 (23.9) Ontario 533 (6.8) 523 (5.2) 9 (4.1) 107 (5.1) 98 (2.5) 9.2 (5.1) 358 (10.9) 355 (11.3) 2 (14.3) Prince Edward Island 492 (3.3) 494 (3.5) -2 (4.5) 90 (3.0) 90 (3.0) 0.7 (4.2) 337 347 (7.8) -9 (11.0) (8.9) Quebec 526 (5.5) 523 (4.7) 4 (4.8) 107 (5.6) 97 (3.0) 9.7 (4.6) 340 (15.4) 357 (9.7) -16 (15.4) Saskatchewan 510 (3.7) 520 (3.9) -10 (5.1) 94 (2.9) 91 (2.6) 3.4 (3.9) 347 (10.0) 369 (10.0) -22 (15.0) italy Centre 520 (13.2) 506 (11.4) 14 (13.2) 99 (8.5) 84 (5.4) 15.0 (9.0) 332 (35.5) 354 (20.8) -22 (42.1) North East 543 (10.2) 509 35 (13.5) 100 (6.0) 75 (3.8) 25.3 (7.3) 350 (32.0) 383 (18.2) -32 (39.5) (9.1) (8.7) North West 537 9 (12.1) 87 (4.7) 78 (4.1) 9.8 (5.4) 387 (18.3) 397 (14.7) -10 (20.8) South 481 (10.3) 464 528 (11.8) (9.2) 17 (10.8) 87 (5.8) 73 (4.7) 14.5 (6.0) 346 (27.4) 339 (28.1) 7 (33.8) South Islands 496 (10.8) 476 (7.5) 20 (8.5) 96 (4.6) 81 (4.5) 14.8 (4.8) 337 (12.8) 342 (19.2) -5 (18.5) 518 (15.4) 495 (12.3) 23 (8.2) 95 (6.8) 84 (4.6) 10.6 (5.0) 351 (23.4) 347 (16.3) 4 (16.5) Portugal Alentejo Partners Spain Basque Country• 498 (4.4) 494 (4.1) 4 (3.6) 100 (3.4) 94 (2.4) 6.0 (3.1) 326 (9.5) 334 (8.1) -7 (8.1) Catalonia• 487 (9.7) 489 (8.2) -2 (6.5) 110 (5.8) 94 (6.4) 16.2 (5.6) 284 (20.7) 321 (23.5) -37 (18.0) Madrid 509 (13.5) 506 (13.9) 4 (8.3) 99 (5.3) 94 (6.4) 5.4 (6.5) 346 (20.0) 346 (16.3) 0 (21.3) 89 (6.8) 83 (5.5) 6.0 (6.3) 305 (28.6) 291 (18.6) 14 (31.1) 111 (10.2) 97 (6.7) 14.2 (5.4) 231 (24.7) 225 (16.5) 6 (19.5) brazil Central-West region 457 (12.3) 429 (12.3) 28 (8.2) Northeast region 407 (13.2) 380 (10.0) 27 (7.6) North region 387 (15.5) 380 (10.8) 6 (14.8) 89 (7.5) 78 (6.3) 11.4 (9.6) 233 (51.2) 268 (16.0) -35 (53.6) Southeast region 457 (7.2) 437 (6.0) 20 (4.2) 85 (3.3) 79 (2.6) 5.5 (3.5) 316 (10.3) 303 (9.0) 13 (11.1) South region 444 (9.1) 426 (8.2) 18 (6.9) 85 (3.2) 77 (4.0) 7.8 (4.8) 303 (13.1) 299 (12.8) 4 (15.0) colombia Bogotá 428 (7.1) 395 (5.7) 33 (6.0) 85 (3.8) 81 (2.2) 4.3 (3.9) 290 (10.3) 261 (9.4) 29 (11.5) Cali 407 (8.2) 391 (10.4) 16 (6.3) 90 (4.5) 89 (5.0) 1.1 (4.0) 255 (16.7) 235 (30.5) 19 (24.5) Manizales 447 (5.6) 404 (5.5) 43 (6.5) 85 (3.9) 81 (2.5) 4.8 (4.4) 305 (9.1) 268 (10.9) 37 (12.8) Medellín 438 (9.6) 410 (8.6) 28 (9.9) 93 (6.2) 94 (5.5) -1.4 (6.3) 293 (10.9) 262 (10.0) 31 (13.2) united arab Emirates Abu Dhabi• 374 (7.7) 408 (6.4) -35 (9.6) 116 (4.2) 98 (3.5) 18.3 (5.6) 192 (11.2) 243 (11.2) -51 (15.5) Ajman 348 (8.1) 399 (11.2) -50 (13.8) 77 (5.6) 75 (4.3) 1.9 (6.9) 225 (19.3) 271 (22.1) -46 (28.6) Dubai• 450 (2.2) 463 (1.8) -13 (3.2) 116 (1.6) 99 (1.5) 16.8 (2.1) 254 (4.4) 302 (5.1) -48 (6.6) fujairah 398 (4.5) 391 (6.7) 8 (7.5) 86 (4.8) 76 (2.7) 10.0 (5.6) 263 (13.8) 260 (13.2) 3 (17.5) ras al-khaimah 356 (19.9) 388 (12.5) -32 (22.4) 103 (17.5) 84 (7.0) 19.0 (18.4) 165 (92.3) 244 (26.4) -80 (96.5) Sharjah 400 (18.2) 430 (9.2) -30 (20.9) 94 (10.7) 75 (4.9) 19.4 (12.1) 239 (29.2) 312 (10.3) -72 (30.6) umm al-Quwain 340 402 (4.9) -62 (8.1) 78 72 (3.9) 212 (15.7) 283 (10.6) -70 (19.0) (5.8) (4.8) 5.0 (6.4) • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.7 for national data. 1 2 http://dx.doi.org/10.1787/888933003763 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 233 Annex b2: reSulTS For regIonS wIThIn counTrIeS table b2.v.5 [Part 2/3] mean score and variation in student performance in problem solving, by gender and by region 10th percentile boys OECD Score S.E. Girls Score S.E. 25th percentile difference (b - G) Score dif. S.E. boys Score S.E. S.E. -17 (13.3) Girls Score 50th percentile (median) difference (b - G) Score dif. S.E. boys Score S.E. Girls Score S.E. difference (b - G) Score dif. S.E. australia Australian Capital Territory 376 (14.6) 403 (13.4) -27 (22.0) 451 (10.7) 468 (7.1) New South Wales 388 402 -15 456 463 (5.2) Northern Territory 357 (20.0) (7.0) (6.7) 372 (14.1) (9.2) -16 (21.8) -5 (6.5) 432 (15.7) 443 (14.8) -7 (7.4) -11 (20.0) 530 (9.2) 536 (7.1) -6 527 (5.5) 527 (5.1) 1 (12.9) (7.0) 532 (15.6) 519 (13.8) 13 (20.2) Queensland 394 (6.8) 399 (8.1) (7.9) 455 (7.0) 460 (5.5) -5 (6.9) 526 (5.3) 524 (4.4) 2 (6.8) South Australia 391 (9.6) 408 (7.1) -17 (10.8) 454 (6.3) 462 (6.0) -8 (6.9) 522 (6.0) 523 (6.4) -1 (7.7) -26 (14.9) (12.5) Tasmania 344 (10.6) 371 (9.2) 410 (8.8) 425 (7.5) -15 (9.7) 488 (8.5) 490 (8.1) -2 Victoria 397 (6.9) 401 (7.6) -5 (7.7) 461 (6.5) 460 (6.2) 1 (7.2) 526 (5.9) 525 (4.9) 1 (5.6) Western Australia 410 (9.2) 396 (9.6) 14 (11.2) 474 (7.2) 458 (6.8) 16 (8.7) 542 (6.2) 522 (6.4) 20 (8.7) flemish Community• 390 (8.5) 382 (8.7) french Community 332 (10.5) 347 (7.0) german-speaking Community 391 (14.4) 392 (12.3) belgium 8 (11.9) 465 (7.4) 458 (6.4) 7 (8.9) 539 (4.5) 528 (5.5) 11 (6.3) (9.5) 411 (7.1) 421 (6.3) -9 (7.2) 498 (5.7) 493 (6.3) 5 (6.2) -2 (20.5) 469 (9.9) 453 (7.0) 16 (11.7) 546 (6.3) 512 (5.2) 34 (8.4) -15 canada Alberta 400 (9.7) 400 (9.4) 0 (10.2) 472 (9.0) 461 (8.0) 11 (7.7) 540 (6.2) 532 (6.7) 8 (5.9) British Columbia 412 (5.5) 407 (8.2) 5 474 (5.7) 468 (6.7) 6 (7.7) 540 (5.2) 535 (6.4) 5 (7.3) Manitoba 376 (8.5) 375 (9.1) 1 (11.6) 441 (5.9) 439 (7.9) 2 (9.5) 507 (5.3) 506 (5.9) 1 (7.6) New Brunswick 386 (10.6) 406 (7.3) -19 (14.6) 452 (6.6) 460 (5.7) -9 (8.2) 512 (6.9) 527 (5.5) -14 (8.2) (8.1) Newfoundland and Labrador 358 (29.1) 394 (17.0) -36 (25.5) 434 (15.3) 453 (9.1) -18 (13.5) 508 (9.2) 513 (6.9) -5 (9.4) Nova Scotia 381 (17.5) 401 (11.7) -20 (17.4) 451 453 (14.8) -1 (14.8) 518 (7.7) 514 (11.1) 4 (12.2) (5.5) (9.0) Ontario 400 (9.0) 399 (10.4) 2 (10.3) 464 (7.2) 459 (7.3) 5 (7.2) 535 (7.6) 526 (5.9) 9 Prince Edward Island 373 (8.8) 378 (6.9) -5 (10.9) 435 (7.0) 435 (6.0) 0 (8.8) 496 (6.3) 493 (4.7) 3 (7.4) Quebec 392 (11.0) 401 (7.6) -9 (10.8) 465 (6.6) 465 (5.0) 0 (6.4) 536 (5.5) 527 (4.9) 9 (5.7) Saskatchewan 385 (10.4) 402 (8.8) -17 (13.0) 446 (6.5) 459 (6.1) -13 (9.0) 514 (5.2) 521 (5.1) -6 (6.6) italy Centre 385 (21.5) 398 (18.9) -13 (27.2) 466 (31.3) 454 (13.8) 12 (31.4) 533 (9.2) 510 (13.3) 23 (12.6) North East 402 (24.4) 414 (13.1) -12 (29.3) 484 (19.9) 463 (11.5) 21 (24.5) 558 (9.6) 513 (12.0) 45 (14.8) North West 425 (16.0) 430 (13.4) -5 (18.7) 478 (13.5) 481 (12.2) -2 (15.6) 545 (11.8) 534 (11.7) 11 (14.7) South 381 (14.9) 373 (18.8) 8 (20.5) 427 (15.0) 422 (9.9) 5 (16.0) 484 (12.0) 467 (8.5) 17 (13.1) South Islands 376 (15.9) 373 (11.2) 2 (14.9) 432 (12.5) 423 (10.2) 8 (11.1) 495 (14.7) 477 (8.5) 18 (12.4) 393 (17.5) 381 (22.6) 12 (15.5) 459 (18.3) 439 (13.0) 20 (15.3) 524 (13.3) 500 (12.8) 24 (10.6) Portugal Alentejo Partners Spain Basque Country• 370 (5.2) 437 (5.3) (5.1) 498 (4.4) 5 (4.9) Catalonia• 337 (19.7) 367 (18.9) -30 (15.1) 421 (14.9) 434 (9.9) -13 (12.1) 495 (10.3) 494 (10.1) 2 (9.8) Madrid 377 (14.4) 384 (20.3) -7 (20.1) 436 (16.7) 440 (15.7) -4 (13.7) 516 (16.0) 510 (15.5) 6 (12.9) (6.1) 374 (7.0) -4 (7.0) 435 -1 (5.3) 503 brazil Central-West region 343 (24.9) 323 (20.6) 20 (25.9) 398 (16.3) 374 (14.2) 24 (14.4) 455 (13.2) 428 (15.3) 27 (14.0) Northeast region 268 (21.2) 257 (11.0) 11 (15.8) 337 (13.3) 314 (11.0) 23 (12.5) 404 (12.7) 379 (13.0) 25 (11.6) -14 (27.1) (17.8) North region 274 (26.2) 288 (11.3) Southeast region 349 334 South region 333 (13.5) (9.1) (7.4) 327 (16.8) 14 (8.9) 6 (12.5) 329 (23.2) 326 (9.6) 388 (18.1) 371 (13.1) 17 396 383 (7.5) 14 (6.8) 457 (9.6) 438 (6.6) 19 (6.8) 374 (11.1) 11 (8.9) 444 (11.5) 426 (10.4) 18 (10.0) 28 (8.6) 429 (7.2) 396 (8.3) 33 (8.0) 15 (10.3) 408 (9.5) 397 (11.5) 11 (8.2) (8.2) 385 (10.8) 3 (21.7) colombia Bogotá 314 (12.3) 287 (11.4) 27 (13.7) 369 (8.6) 341 Cali 286 (20.3) 274 (16.3) 12 (17.1) 346 (9.9) 331 (14.7) Manizales 338 (10.0) 299 Medellín 324 (10.8) 292 (11.3) (6.5) 39 (7.3) (9.9) 392 (6.6) 349 (7.7) 43 (8.9) 446 (6.0) 406 (7.7) 41 (8.3) 32 (13.3) 373 (8.3) 346 (8.9) 27 (9.3) 433 (11.7) 407 (10.0) 27 (11.6) (8.9) 346 united arab Emirates Abu Dhabi• 229 (9.1) 282 (10.1) -53 (13.7) 294 Ajman 255 (14.9) 304 (15.0) -49 (22.3) 296 (10.9) -41 Dubai• 296 (5.0) 337 (4.0) fujairah 292 (10.8) 289 (8.7) (7.2) 3 (14.3) (8.4) -52 (12.5) 369 (9.8) 410 (6.0) -41 (11.4) 348 (14.0) -52 (17.8) 346 (8.5) 399 (14.6) -53 (17.0) (4.3) 394 (3.3) -22 340 (10.3) 340 (9.7) 372 (5.7) 452 (3.7) 464 (3.6) -12 (5.5) 0 (15.4) 393 (7.3) 394 (9.4) -2 (12.4) ras al-khaimah 228 (65.1) 282 (27.6) -54 (67.5) 298 (25.5) 337 (13.9) -39 (27.7) 363 (16.2) 390 (11.8) -27 (18.9) Sharjah 275 (27.9) 336 (10.9) -61 (28.9) 339 (27.9) 376 (9.3) -37 (28.7) 400 (16.5) 428 (11.0) -27 (19.8) umm al-Quwain 244 (12.3) 308 (14.4) -64 (20.3) 290 351 (7.1) -61 (11.0) 336 (10.4) 398 (8.7) -62 (14.2) (7.8) • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.7 for national data. 1 2 http://dx.doi.org/10.1787/888933003763 234 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For regIonS wIThIn counTrIeS: Annex b2 table b2.v.5 [Part 3/3] mean score and variation in student performance in problem solving, by gender and by region 75th percentile boys OECD Score S.E. Girls Score S.E. 90th percentile difference (b - G) Score dif. S.E. boys Score S.E. Girls Score S.E. 95th percentile difference (b - G) Score dif. S.E. boys Score S.E. Girls difference (b - G) Score dif. S.E. Score S.E. 687 (12.5) 677 (15.6) 10 (20.2) 693 675 (5.8) 18 (10.0) australia Australian Capital Territory 599 (8.9) 596 (5.5) 2 (10.3) 657 (8.9) 644 (8.1) 13 (11.6) New South Wales 597 (7.3) 591 (5.2) 6 (8.4) 659 (8.8) 646 (5.3) 14 (9.8) 30 (22.6) 38 (26.0) 4 (7.8) 6 (8.4) Northern Territory 608 (15.8) 578 (17.7) Queensland 591 587 (5.4) (5.7) 667 (23.1) 629 (20.6) 647 (6.7) 641 (5.6) (8.8) 636 (7.6) 5 624 (11.4) 6 South Australia 586 (7.8) 583 (8.4) 3 (10.6) 641 Tasmania 565 (8.8) 555 (8.6) 10 (12.3) 630 (11.4) (8.6) 699 (37.6) 660 (34.1) 39 (43.9) 682 (8.7) 672 (6.7) 10 (11.7) (10.0) 673 (8.7) 666 (7.8) 7 (11.3) (15.2) 671 (14.1) 659 (13.4) 12 (19.9) Victoria 592 (6.5) 589 (5.7) 3 (7.6) 645 (6.0) 640 (7.2) 5 (7.7) Western Australia 605 (7.0) 584 (7.5) 20 (10.4) 654 (9.3) 636 (9.6) 18 (14.9) (6.5) 669 (7.2) 8 (9.6) 683 (12.7) 677 669 (11.2) 14 (16.7) belgium flemish Community• 604 (4.5) 590 (4.6) 14 (5.0) 655 (4.1) 641 (4.8) 14 (5.3) 684 (4.9) 669 (4.4) 15 (5.3) french Community 572 (6.3) 556 (6.3) 16 (6.9) 624 (5.9) 605 (5.8) 19 (5.3) 653 (6.8) 633 (7.6) 20 (8.4) german-speaking Community 602 (6.8) 567 (6.8) 36 (9.7) 655 (8.2) 612 (8.6) 43 (12.5) 684 (10.4) 642 (12.1) 42 (17.5) canada Alberta 601 (5.9) 598 (6.9) 3 (6.9) 655 (8.0) 651 (8.1) 4 (9.4) British Columbia 605 (6.3) 594 (5.9) 10 (8.2) 661 (6.6) 644 (8.0) 17 (10.5) (7.6) 681 (8.7) 5 (11.2) 695 (12.1) 686 673 (10.4) 22 (16.1) Manitoba 578 (5.6) 573 (5.1) 5 (7.7) 627 (9.2) 627 (6.3) 0 (10.2) 662 (8.5) 658 (6.8) 5 (11.7) New Brunswick 576 (7.9) 581 (6.5) -5 (9.9) 628 (9.2) 626 (8.5) 1 (12.8) 661 (12.1) 654 (14.1) 7 (19.1) Newfoundland and Labrador 572 (6.3) 573 (5.2) -1 (7.0) 622 (8.1) 629 (8.0) -6 (11.4) 651 (9.5) 660 (8.8) -10 (11.5) Nova Scotia 577 (7.6) 573 (7.7) 4 (9.5) 630 (10.9) 620 (7.1) 9 (13.4) 659 (10.7) 653 (13.6) 5 (14.9) Ontario 605 (7.7) 590 (5.8) 16 (6.3) 664 (8.2) 645 (7.5) 19 (7.6) 701 (10.0) 681 (10.2) 20 (11.5) Prince Edward Island 553 (5.5) 554 (6.2) -1 (8.1) 604 (8.4) 605 (5.8) -1 (10.3) 635 (6.0) 638 (12.6) -3 (14.2) Quebec 598 (5.9) 588 (6.3) 10 (6.3) 653 (7.3) 644 (6.4) 8 (7.4) 686 (7.8) 673 (8.2) 13 (9.0) Saskatchewan 576 (6.5) 583 (7.0) -6 (8.6) 629 (8.5) 640 (6.8) -11 (10.8) 661 (8.4) 669 (8.2) -8 (12.1) italy Centre 587 (10.0) 564 (11.1) 23 (9.4) North East 614 560 54 (10.9) (7.8) (8.1) 634 (12.5) 607 (13.5) 27 (12.1) 660 (17.2) 638 (15.1) 22 (14.9) 657 (8.9) 599 58 (10.9) 686 (11.7) 624 (6.8) 62 (11.7) (7.7) (7.7) North West 600 (10.0) 581 (12.9) 20 (13.6) 644 619 (13.2) 24 (12.8) 667 (7.5) 646 (20.0) 22 (19.4) South 541 (11.4) 514 (11.4) 27 (12.8) 589 (12.0) 553 (9.2) 35 (12.6) 613 (10.7) 574 (10.8) 39 (14.8) South Islands 565 (11.7) 531 (7.8) 35 (11.2) 621 (13.5) 576 (9.6) 45 (13.8) 647 (15.8) 606 (12.9) 41 (16.3) 583 (18.0) 557 (16.8) 26 (14.9) 635 (24.5) 602 (14.6) 33 (17.0) 663 (25.7) 625 (14.9) 38 (19.2) Portugal Alentejo Partners Spain Basque Country• 567 (4.9) 556 (4.1) 11 (4.3) 622 (4.9) 609 (5.6) 13 (5.4) (5.0) 641 (4.7) 12 (5.9) Catalonia• 564 (8.7) 554 (7.7) 10 (8.5) 624 (11.1) 601 (9.4) 23 (13.6) 655 (14.4) 631 (10.1) 24 (17.9) Madrid 579 (18.7) 572 (14.5) 7 (15.1) 633 (18.4) 622 (17.8) 11 (16.8) 665 (18.1) 654 (24.6) 10 (23.8) 654 brazil Central-West region 515 (16.5) 485 (13.4) 31 (16.8) 568 (15.3) 533 (12.2) 34 (13.2) 604 (24.3) 561 (18.6) 43 (21.0) Northeast region 476 (18.0) 445 (16.8) 31 (11.6) 549 (26.7) 505 (16.0) 44 (21.2) 599 (32.3) 537 (16.1) 61 (26.2) 529 (19.3) 528 (21.2) 1 (20.9) 588 (7.6) 565 (7.2) 23 (8.2) 588 (15.8) 557 (13.2) 31 (20.3) North region 447 (18.7) 428 (20.3) 18 (24.4) Southeast region 518 491 (7.8) 26 (8.5) South region 500 (11.4) 477 (10.7) 23 (12.8) (9.3) 498 (14.0) 490 (23.7) 565 539 (6.9) 552 (10.2) 9 (22.4) (6.9) 26 (7.5) 526 (12.0) 26 (13.1) colombia Bogotá 484 (8.2) 452 (6.2) 33 (7.8) Cali 471 (9.5) 451 (9.3) 20 (10.0) 538 (11.4) 496 (8.1) 42 (12.8) 566 (16.3) 524 (8.9) 42 (17.8) 522 (7.3) 501 (10.3) 20 (10.7) 550 (13.4) 529 (9.8) 21 (13.3) Manizales 504 (9.1) 457 (6.6) 47 Medellín 501 (12.5) 470 (10.6) 31 (10.3) 556 (9.4) (13.3) 562 (15.4) (9.5) 52 (13.5) 582 (12.0) 538 (9.7) 45 (16.9) 532 (16.8) 505 30 (18.2) 600 (19.3) 573 (21.5) 26 (22.9) united arab Emirates Abu Dhabi• 455 (10.1) 474 (7.0) -19 (11.8) 527 Ajman 398 (11.7) 451 (11.9) -53 (15.6) 450 (16.8) Dubai• 534 (3.8) 531 (4.6) 3 (6.3) fujairah 455 (8.8) 442 (10.0) 13 (13.1) 598 (8.7) (7.5) -4 (11.0) 568 (12.8) 568 (7.8) 0 (14.6) 498 (12.6) 532 -48 (20.2) 480 (14.1) 520 (12.4) -40 (19.1) (4.7) 590 (5.7) 8 (7.6) 512 (10.3) 485 (9.1) 28 (12.6) (6.5) 627 (6.6) 4 (10.0) 553 (16.8) 631 515 (11.6) 38 (18.8) ras al-khaimah 419 (11.3) 441 (13.0) -22 (17.0) 481 (10.4) 490 (13.7) -9 (17.1) 508 (10.4) 529 (21.4) -22 (24.1) Sharjah 462 (22.8) 482 (12.9) -20 (28.2) 521 (24.0) 528 (14.8) -6 (30.7) 559 (27.2) 556 (14.6) 2 (31.8) umm al-Quwain 388 (10.2) 452 (12.2) -64 (16.8) 436 (14.2) 497 -61 (16.8) 470 (25.3) 519 (10.7) -49 (27.3) (9.7) • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.7 for national data. 1 2 http://dx.doi.org/10.1787/888933003763 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 235 Annex b2: reSulTS For regIonS wIThIn counTrIeS table b2.v.6 [Part 1/2] Performance in problem solving, by socio-economic status and by region Results based on students’ self-reports PiSa index of economic. social and cultural status (EScS) all students OECD mean index S.E. bottom quarter mean index S.E. Second quarter mean index S.E. third quarter mean index S.E. top quarter mean index S.E. australia Australian Capital Territory 0.62 (0.02) -0.23 (0.05) 0.49 (0.02) 0.87 (0.02) 1.33 (0.03) New South Wales 0.25 (0.02) -0.86 (0.03) 0.04 (0.03) 0.62 (0.02) 1.19 (0.02) Northern Territory 0.14 (0.06) -0.95 (0.09) -0.04 (0.07) 0.51 (0.06) 1.06 (0.07) Queensland 0.20 (0.02) -0.86 (0.03) -0.02 (0.04) 0.53 (0.03) 1.14 (0.02) South Australia 0.19 (0.02) -0.90 (0.05) 0.00 (0.03) 0.54 (0.02) 1.11 (0.03) Tasmania 0.02 (0.03) -1.05 (0.03) -0.25 (0.04) 0.35 (0.04) 1.05 (0.03) Victoria 0.30 (0.02) -0.76 (0.03) 0.11 (0.04) 0.66 (0.03) 1.20 (0.02) Western Australia 0.26 (0.03) -0.82 (0.04) 0.04 (0.04) 0.62 (0.03) 1.19 (0.03) belgium flemish Community• 0.16 (0.02) -1.04 (0.04) -0.18 (0.03) 0.58 (0.03) 1.28 (0.02) french Community 0.12 (0.03) -1.05 (0.04) -0.21 (0.04) 0.51 (0.04) 1.25 (0.03) german-speaking Community 0.29 (0.03) -0.81 (0.04) -0.05 (0.04) 0.66 (0.04) 1.35 (0.03) canada Alberta 0.51 (0.03) -0.58 (0.04) 0.27 (0.04) 0.87 (0.04) 1.51 (0.02) British Columbia 0.46 (0.04) -0.67 (0.04) 0.19 (0.05) 0.84 (0.04) 1.48 (0.03) Manitoba 0.26 (0.03) -0.94 (0.05) 0.00 (0.04) 0.66 (0.03) 1.34 (0.03) New Brunswick 0.37 (0.02) -0.72 (0.03) 0.10 (0.04) 0.73 (0.03) 1.37 (0.03) Newfoundland and Labrador 0.28 (0.04) -0.89 (0.06) -0.04 (0.05) 0.65 (0.05) 1.41 (0.04) Nova Scotia 0.31 (0.03) -0.78 (0.03) 0.04 (0.04) 0.63 (0.05) 1.33 (0.03) Ontario 0.44 (0.04) -0.76 (0.05) 0.20 (0.05) 0.83 (0.04) 1.49 (0.03) Prince Edward Island 0.33 (0.02) -0.77 (0.04) 0.09 (0.03) 0.72 (0.03) 1.31 (0.02) Quebec 0.34 (0.03) -0.80 (0.03) 0.09 (0.04) 0.73 (0.03) 1.34 (0.02) Saskatchewan 0.40 (0.02) -0.65 (0.03) 0.09 (0.03) 0.72 (0.03) 1.45 (0.03) italy Centre 0.17 (0.06) -1.00 (0.06) -0.15 (0.06) 0.47 (0.09) 1.35 (0.06) North East 0.00 (0.05) -1.16 (0.04) -0.32 (0.03) 0.24 (0.06) 1.24 (0.10) 0.00 (0.06) -1.16 (0.07) -0.32 (0.07) 0.28 (0.06) 1.20 (0.07) South North West -0.10 (0.07) -1.36 (0.05) -0.53 (0.08) 0.21 (0.09) 1.29 (0.09) South Islands -0.20 (0.07) -1.44 (0.05) -0.60 (0.08) 0.09 (0.09) 1.15 (0.08) -0.35 (0.14) -1.72 (0.07) -0.87 (0.15) -0.05 (0.19) 1.25 (0.16) (0.02) Portugal Alentejo Spain Basque Country • Catalonia• Partners Madrid 0.03 (0.03) -1.21 (0.03) -0.30 (0.03) 0.46 (0.04) 1.18 -0.14 (0.08) -1.45 (0.07) -0.53 (0.09) 0.27 (0.12) 1.15 (0.06) 0.03 (0.15) -1.28 (0.10) -0.36 (0.16) 0.43 (0.21) 1.36 (0.15) brazil Central-West region -1.03 (0.11) -2.46 (0.13) -1.47 (0.11) -0.73 (0.14) 0.58 (0.17) Northeast region -1.26 (0.11) -2.84 (0.13) -1.75 (0.14) -0.86 (0.12) 0.40 (0.12) (0.07) North region -0.91 (0.10) -2.28 (0.12) -1.26 (0.12) -0.58 (0.10) 0.48 Southeast region -1.01 (0.06) -2.49 (0.04) -1.46 (0.05) -0.64 (0.09) 0.54 (0.09) South region -1.32 (0.09) -2.70 (0.07) -1.76 (0.10) -1.01 (0.11) 0.22 (0.16) colombia Bogotá -1.09 (0.05) -2.34 (0.04) -1.42 (0.06) -0.75 (0.06) 0.14 (0.07) Cali -0.81 (0.08) -2.09 (0.07) -1.12 (0.09) -0.49 (0.08) 0.46 (0.10) Manizales -0.77 (0.07) -2.25 (0.09) -1.03 (0.10) -0.36 (0.07) 0.57 (0.05) Medellín -0.94 (0.10) -2.43 (0.10) -1.31 (0.09) -0.57 (0.11) 0.56 (0.15) united arab Emirates Abu Dhabi• Ajman 0.29 (0.03) -0.91 (0.06) 0.14 (0.04) 0.65 (0.03) 1.28 (0.02) -0.09 (0.06) -1.30 (0.12) -0.26 (0.06) 0.25 (0.06) 0.96 (0.06) Dubai• 0.50 (0.01) -0.46 (0.02) 0.37 (0.01) 0.77 (0.01) 1.32 (0.01) fujairah 0.01 (0.03) -1.17 (0.06) -0.19 (0.04) 0.36 (0.04) 1.03 (0.03) ras al-khaimah 0.06 (0.08) -1.19 (0.14) -0.12 (0.09) 0.43 (0.07) 1.11 (0.06) Sharjah 0.44 (0.04) -0.59 (0.09) 0.34 (0.05) 0.76 (0.03) 1.25 (0.03) -0.10 (0.04) -1.33 (0.09) -0.25 (0.05) 0.27 (0.05) 0.93 (0.05) umm al-Quwain • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.12 for national data. 1. Single-level bivariate regression of performance on ESCS. The slope of the gradient is the regression coeficient for ESCS; the strength of the relationship is the r-squared. 1 2 http://dx.doi.org/10.1787/888933003763 236 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For regIonS wIThIn counTrIeS: Annex b2 table b2.v.6 [Part 2/2] Performance in problem solving, by socio-economic status and by region Results based on students’ self-reports Partners OECD Performance in problem solving, by national quarters of this index australia Australian Capital Territory New South Wales Northern Territory Queensland South Australia Tasmania Victoria Western Australia belgium flemish Community• french Community german-speaking Community canada Alberta British Columbia Manitoba New Brunswick Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan italy Centre North East North West South South Islands Portugal Alentejo Spain Basque Country• Catalonia• Madrid brazil Central-West region Northeast region North region Southeast region South region colombia Bogotá Cali Manizales Medellín united arab Emirates Abu Dhabi• Ajman Dubai• fujairah ras al-khaimah Sharjah umm al-Quwain bottom quarter Second quarter mean score S.E. mean score 482 487 469 485 487 447 493 498 (8.5) (5.4) (16.5) (5.7) (7.8) (7.7) (5.9) (6.2) 473 437 495 third quarter S.E. mean score 523 515 512 507 514 478 508 519 (8.8) (4.5) (14.9) (4.3) (7.4) (7.0) (6.1) (7.2) (5.6) (6.7) (7.7) 511 474 514 503 507 477 491 461 497 510 463 500 493 (6.2) (5.6) (6.6) (6.7) (21.1) (7.0) (7.5) (6.4) (5.8) (5.7) 488 489 518 453 460 top quarter increased Strength of the Slope of likelihood of relationship between students in the the socio-economic student performance gradient1 bottom quarter and EScS1 of the EScS index scoring Score-point difference in the bottom Percentage in problem quarter of the of explained solving problem-solving variation associated performance in student with distribution performance one-unit (r-squared increase relative × 100) S.E. in the EScS S.E. risk S.E. S.E. mean score S.E. 554 537 530 534 529 504 544 541 (7.4) (5.3) (15.3) (5.8) (5.9) (7.9) (5.9) (6.9) 554 569 554 563 551 536 552 559 (8.1) (5.0) (17.8) (5.1) (6.7) (8.7) (5.4) (5.8) 2.19 1.92 1.90 1.90 1.79 2.13 1.73 1.66 (0.45) (0.16) (0.43) (0.18) (0.25) (0.37) (0.15) (0.18) 46 38 48 39 28 43 31 30 (6.6) (2.8) (9.0) (3.1) (4.2) (4.4) (2.5) (3.3) 8.4 10.0 11.6 10.0 5.9 10.9 6.6 6.2 (2.5) (1.4) (4.0) (1.5) (1.7) (2.1) (1.0) (1.2) (5.7) (6.8) (8.5) 548 506 530 (4.5) (6.3) (7.2) 574 531 541 (4.6) (6.1) (8.3) 2.42 2.25 1.57 (0.19) (0.21) (0.25) 44 41 21 (2.8) (3.6) (4.7) 16.1 12.2 3.4 (2.1) (1.9) (1.5) 521 521 499 516 477 500 521 483 522 506 (8.1) (5.5) (6.4) (5.2) (7.5) (13.2) (6.7) (5.2) (6.0) (5.4) 535 547 511 521 520 522 536 493 536 519 (7.1) (5.9) (6.0) (6.6) (7.8) (6.4) (7.8) (5.9) (5.5) (6.0) 566 567 535 536 557 538 554 529 551 544 (6.7) (6.0) (6.0) (7.2) (5.3) (6.4) (7.2) (4.8) (6.1) (4.8) 1.60 1.74 1.57 1.55 2.02 1.23 1.43 1.66 1.56 1.45 (0.20) (0.22) (0.18) (0.20) (0.47) (0.23) (0.14) (0.24) (0.14) (0.19) 30 27 25 19 41 19 19 31 23 25 (3.1) (2.9) (3.4) (4.7) (9.0) (3.5) (3.1) (3.7) (3.2) (2.8) 6.0 6.2 5.1 3.0 13.1 2.9 2.8 7.8 3.6 4.9 (1.2) (1.3) (1.2) (1.5) (4.8) (1.0) (0.9) (1.8) (0.9) (1.1) (14.4) (10.2) (11.3) (12.0) (13.4) 508 520 518 461 476 (13.9) (7.3) (10.6) (12.0) (10.7) 534 546 544 481 492 (11.9) (7.6) (11.3) (13.2) (10.1) 527 555 551 503 517 (17.9) (13.3) (7.7) (8.6) (12.3) 1.52 1.93 1.32 1.57 1.75 (0.29) (0.37) (0.22) (0.45) (0.39) 20 30 14 21 24 (4.9) (6.2) (4.8) (3.6) (4.9) 3.8 9.3 2.5 6.7 7.0 (2.0) (3.4) (1.6) (2.2) (2.7) 459 (15.8) 492 (17.9) 520 (11.6) 554 (18.0) 2.45 (0.53) 31 (4.9) 15.2 (3.8) 464 459 468 (6.5) (10.8) (16.9) 491 474 500 (5.2) (11.8) (9.0) 505 497 511 (5.0) (10.5) (17.3) 527 522 553 (4.5) (11.8) (26.9) 1.67 1.54 1.97 (0.14) (0.23) (0.56) 26 24 31 (2.9) (5.0) (10.1) 6.1 5.7 10.9 (1.2) (2.2) (6.9) 391 340 350 415 394 (16.2) (10.6) (11.6) (7.2) (8.7) 427 369 386 439 418 (19.0) (15.6) (15.9) (10.3) (13.4) 451 400 383 455 450 (15.7) (15.9) (13.4) (8.8) (9.4) 503 465 416 482 477 (17.3) (21.4) (16.9) (8.8) (11.0) 2.88 2.28 1.76 1.83 2.17 (1.09) (0.50) (0.59) (0.21) (0.39) 37 39 23 23 30 (4.4) (7.1) (5.7) (3.3) (3.5) 25.3 21.3 8.7 10.5 16.7 (6.0) (5.3) (4.1) (3.2) (4.4) 390 360 383 381 (6.6) (12.7) (9.9) (7.2) 405 386 419 401 (6.8) (11.3) (6.5) (9.4) 415 404 436 422 (7.1) (10.3) (9.4) (8.9) 434 441 454 495 (9.8) (10.2) (9.2) (21.7) 1.56 1.90 2.11 1.86 (0.20) (0.34) (0.40) (0.37) 19 30 26 39 (4.0) (4.7) (4.2) (5.2) 4.9 10.9 11.6 22.0 (2.0) (2.9) (3.4) (5.3) 355 353 406 371 350 389 349 (6.4) (11.2) (3.5) (5.4) (11.0) (12.1) (9.3) 381 366 447 384 346 415 377 (6.7) (8.6) (3.5) (6.8) (19.2) (7.6) (10.2) 412 383 480 403 389 440 370 (7.4) (13.2) (3.7) (6.3) (14.0) (13.1) (9.8) 424 397 495 420 405 421 395 (8.2) (9.8) (3.1) (10.0) (10.2) (12.5) (9.5) 1.65 1.64 2.10 1.56 1.31 1.48 1.81 (0.16) (0.36) (0.14) (0.25) (0.31) (0.29) (0.45) 28 19 48 23 25 18 22 (3.6) (4.2) (2.3) (4.3) (3.0) (5.4) (4.3) 5.5 4.5 10.5 6.5 6.1 2.7 5.9 (1.3) (2.1) (0.9) (2.4) (2.4) (1.4) (2.4) • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.12 for national data. 1. Single-level bivariate regression of performance on ESCS. The slope of the gradient is the regression coeficient for ESCS; the strength of the relationship is the r-squared. 1 2 http://dx.doi.org/10.1787/888933003763 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 237 Annex b2: reSulTS For regIonS wIThIn counTrIeS table b2.v.7 [Part 1/3] Strength of the relationship between socio-economic status and performance in problem solving, mathematics, reading and science, by region Results based on students’ self-reports Slope of the socio-economic gradient:1 Score-point difference associated with a one-unit increase in EScS Partners OECD Problem solving australia Australian Capital Territory New South Wales Northern Territory Queensland South Australia Tasmania Victoria Western Australia belgium flemish Community• french Community german-speaking Community canada Alberta British Columbia Manitoba New Brunswick Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan italy Centre North East North West South South Islands Portugal Alentejo Spain Basque Country• Catalonia• Madrid brazil Central-West region Northeast region North region Southeast region South region colombia Bogotá Cali Manizales Medellín united arab Emirates Abu Dhabi• Ajman Dubai• fujairah ras al-khaimah Sharjah umm al-Quwain mathematics reading computer-based mathematics Science digital reading Score dif. S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. Score dif. S.E. 46 38 48 39 28 43 31 30 (6.6) (2.8) (9.0) (3.1) (4.2) (4.4) (2.5) (3.3) 52 44 62 46 38 46 35 43 (5.3) (3.0) (7.8) (2.7) (3.7) (3.9) (2.5) (3.1) 54 45 66 45 35 44 34 41 (6.0) (2.5) (8.7) (3.2) (3.8) (4.1) (2.7) (3.0) 53 46 70 45 41 51 36 43 (5.4) (2.9) (6.6) (2.8) (3.7) (4.5) (2.6) (3.0) 48 37 56 38 33 44 28 39 (5.0) (2.9) (5.5) (2.9) (4.6) (3.9) (3.0) (3.9) 50 41 71 39 35 49 31 41 (5.5) (2.8) (8.6) (3.6) (4.5) (4.5) (2.7) (3.7) 44 41 21 (2.8) (3.6) (4.7) 50 48 22 (2.3) (2.6) (4.0) 44 50 24 (2.1) (3.4) (4.3) 48 47 26 (2.1) (2.9) (4.0) 44 41 9 (2.3) (2.7) (3.6) 42 39 9 (2.5) (3.3) (4.9) 30 27 25 19 41 19 19 31 23 25 (3.1) (2.9) (3.4) (4.7) (9.0) (3.5) (3.1) (3.7) (3.2) (2.8) 33 26 37 26 40 29 30 29 36 25 (2.4) (2.6) (3.0) (4.2) (4.6) (2.9) (2.4) (3.0) (2.7) (2.2) 32 24 35 24 36 23 28 28 33 24 (2.8) (3.2) (3.1) (4.1) (4.3) (3.8) (2.4) (3.4) (2.7) (3.0) 32 24 34 23 36 22 28 29 29 26 (3.0) (3.0) (3.1) (4.6) (3.8) (3.2) (2.5) (3.3) (2.5) (2.6) 32 23 29 23 37 29 24 15 26 27 (4.1) (3.0) (3.1) (4.3) (4.3) (2.6) (3.0) (3.7) (2.7) (2.6) 28 25 25 26 37 20 23 39 23 21 (3.1) (2.5) (3.0) (3.8) (5.2) (5.0) (3.3) (4.1) (2.7) (2.7) 20 30 14 21 24 (4.9) (6.2) (4.8) (3.6) (4.9) 25 37 21 27 30 (4.1) (5.5) (4.5) (3.8) (6.3) 30 40 20 31 31 (5.5) (5.1) (4.7) (4.3) (6.5) 27 35 22 29 30 (4.6) (4.9) (4.9) (4.1) (5.9) 19 29 19 19 27 (4.9) (6.9) (4.6) (5.1) (4.7) 24 27 16 20 23 (5.9) (6.7) (4.7) (4.9) (4.5) 31 (4.9) 33 (3.6) 27 (4.1) 27 (3.1) 28 (4.2) 27 (5.1) 26 24 31 (2.9) (5.0) (10.1) 28 35 35 (1.8) (3.1) (7.5) 28 31 28 (2.2) (3.0) (7.7) 26 31 27 (2.0) (2.9) (6.3) 25 24 26 (2.2) (3.5) (6.0) 27 31 31 (2.6) (4.7) (7.3) 37 39 23 23 30 (4.4) (7.1) (5.7) (3.3) (3.5) 38 32 23 23 25 (6.2) (4.8) (4.7) (4.5) (7.4) 33 28 21 19 25 (5.5) (5.6) (6.4) (3.7) (6.6) 36 29 16 21 24 (5.5) (4.9) (5.0) (3.8) (6.8) 38 31 25 28 28 (7.1) (4.4) (5.5) (4.3) (6.9) 40 32 26 23 27 (8.5) (5.6) (6.8) (3.9) (5.6) 19 30 24 39 (4.0) (4.7) (3.7) (5.2) 19 27 28 35 (3.6) (3.5) (3.2) (5.3) 18 29 26 32 (2.9) (4.1) (2.8) (4.9) 18 28 23 31 (3.6) (3.3) (3.5) (4.8) 17 19 15 30 (4.4) (3.6) (3.1) (5.1) 24 32 29 31 (4.3) (5.5) (2.6) (4.4) 28 19 48 23 25 18 22 (3.6) (4.2) (2.3) (4.3) (3.0) (5.4) (4.3) 29 21 43 20 22 28 22 (3.2) (3.8) (2.0) (7.2) (3.5) (6.6) (5.3) 24 22 43 15 17 22 20 (3.5) (4.9) (2.2) (8.0) (5.0) (6.6) (5.0) 29 23 47 15 19 27 21 (3.5) (3.8) (2.1) (6.0) (4.0) (7.8) (4.6) 28 13 43 12 13 16 17 (3.8) (3.4) (1.9) (5.2) (3.4) (3.4) (4.0) 38 24 57 23 23 34 23 (4.7) (6.5) (2.3) (7.1) (4.3) (7.3) (8.0) • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.13 for national data. 1. Single-level bivariate regression of performance on the PISA index of economic, social and cultural status (ESCS), the slope is the regression coeficient for ESCS. 2. r-squared from the regression coeficient of performance on the PISA index of economic, social and cultural status (ESCS). 1 2 http://dx.doi.org/10.1787/888933003763 238 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For regIonS wIThIn counTrIeS: Annex b2 table b2.v.7 [Part 2/3] Strength of the relationship between socio-economic status and performance in problem solving, mathematics, reading and science, by region Results based on students’ self-reports Strength of the relationship between performance and EScS:2 Percentage of explained variation in performance Partners OECD Problem solving australia Australian Capital Territory New South Wales Northern Territory Queensland South Australia Tasmania Victoria Western Australia belgium flemish Community• french Community german-speaking Community canada Alberta British Columbia Manitoba New Brunswick Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan italy Centre North East North West South South Islands Portugal Alentejo Spain Basque Country• Catalonia• Madrid brazil Central-West region Northeast region North region Southeast region South region colombia Bogotá Cali Manizales Medellín united arab Emirates Abu Dhabi• Ajman Dubai• fujairah ras al-khaimah Sharjah umm al-Quwain mathematics reading computer-based mathematics Science digital reading % S.E. % S.E. % S.E. % S.E. % S.E. % S.E. 8.4 10.0 11.6 10.0 5.9 10.9 6.6 6.2 (2.5) (1.4) (4.0) (1.5) (1.7) (2.1) (1.0) (1.2) 12.5 12.8 20.7 14.9 11.1 16.0 9.0 13.4 (2.7) (1.6) (5.0) (1.6) (2.0) (2.4) (1.1) (1.7) 12.7 13.5 18.9 13.5 9.4 14.1 8.4 12.0 (2.9) (1.5) (4.7) (1.7) (1.8) (2.3) (1.2) (1.7) 11.5 12.8 20.6 13.6 11.3 15.8 8.8 11.9 (2.3) (1.5) (4.5) (1.6) (2.0) (2.5) (1.1) (1.6) 11.7 9.9 18.3 11.3 8.7 13.8 6.4 10.7 (2.5) (1.4) (3.9) (1.6) (2.1) (2.2) (1.2) (1.8) 11.5 11.7 20.0 10.6 7.9 14.1 6.8 11.1 (2.5) (1.5) (4.7) (1.7) (1.9) (2.3) (1.1) (1.8) 16.1 12.2 3.4 (2.1) (1.9) (1.5) 19.9 20.6 4.4 (1.9) (2.0) (1.6) 17.6 19.4 4.5 (1.8) (2.2) (1.7) 19.5 19.6 5.9 (1.8) (2.0) (1.8) 16.0 16.7 0.8 (1.8) (2.0) (0.6) 15.5 13.6 0.5 (1.9) (1.9) (0.7) 6.0 6.2 5.1 3.0 13.1 2.9 2.8 7.8 3.6 4.9 (1.2) (1.3) (1.2) (1.5) (4.8) (1.0) (0.9) (1.8) (0.9) (1.1) 8.9 7.1 14.1 6.7 17.6 8.9 9.6 8.3 11.6 6.2 (1.3) (1.3) (2.2) (2.0) (4.0) (1.7) (1.3) (1.6) (1.5) (1.0) 8.8 5.7 11.6 4.9 11.5 4.5 7.9 6.2 9.2 5.1 (1.5) (1.5) (1.8) (1.6) (2.9) (1.4) (1.3) (1.5) (1.3) (1.2) 8.5 5.7 10.9 4.9 12.7 4.6 7.2 7.1 8.7 6.3 (1.5) (1.3) (1.8) (1.8) (3.0) (1.3) (1.2) (1.5) (1.4) (1.1) 7.1 4.8 8.6 4.9 15.8 7.8 5.7 1.8 5.9 5.8 (1.4) (1.2) (1.8) (1.9) (4.1) (1.4) (1.5) (0.9) (1.1) (1.1) 6.6 6.6 6.8 6.4 13.0 3.7 5.5 8.5 5.2 4.3 (1.3) (1.2) (1.6) (1.8) (3.6) (1.8) (1.5) (1.7) (1.0) (1.0) 3.8 9.3 2.5 6.7 7.0 (2.0) (3.4) (1.6) (2.2) (2.7) 6.3 14.3 5.4 10.2 10.7 (1.9) (3.2) (2.1) (2.6) (3.7) 8.7 15.5 3.6 11.2 9.6 (3.0) (3.0) (1.7) (2.5) (3.4) 7.5 13.1 5.1 10.2 10.2 (2.5) (2.9) (2.2) (3.0) (3.6) 4.8 9.8 4.9 7.5 11.4 (2.5) (4.0) (2.1) (3.2) (3.3) 5.9 7.1 3.0 5.3 5.4 (3.0) (2.9) (1.7) (2.4) (1.8) 15.2 (3.8) 17.9 (3.3) 12.9 (2.9) 14.9 (2.9) 14.0 (2.7) 14.1 (4.3) 6.1 5.7 10.9 (1.2) (2.2) (6.9) 10.4 17.9 17.0 (1.2) (2.9) (6.7) 10.2 12.5 11.5 (1.5) (2.2) (5.9) 9.2 15.1 11.2 (1.3) (2.6) (5.1) 8.2 9.6 12.6 (1.3) (2.3) (6.0) 7.8 9.5 12.8 (1.3) (2.3) (5.8) 25.3 21.3 8.7 10.5 16.7 (6.0) (5.3) (4.1) (3.2) (4.4) 28.5 22.6 11.9 12.2 13.9 (6.7) (4.6) (4.1) (4.3) (8.0) 22.4 15.0 6.9 7.3 10.6 (6.3) (4.7) (3.3) (2.7) (5.6) 25.7 18.4 6.1 10.3 12.7 (5.3) (4.6) (3.5) (3.5) (7.0) 28.3 19.4 15.3 15.8 15.9 (7.8) (5.0) (4.5) (4.3) (6.2) 26.7 15.0 9.0 10.8 12.3 (7.7) (3.9) (3.8) (3.7) (5.7) 4.9 10.9 8.9 22.0 (2.0) (2.9) (2.6) (5.3) 7.9 14.4 16.8 24.2 (2.8) (3.3) (2.9) (5.8) 5.5 12.6 14.1 19.3 (1.8) (3.3) (2.5) (5.5) 6.6 13.7 12.2 20.4 (2.5) (3.1) (3.1) (5.6) 5.2 5.6 5.9 18.1 (2.6) (2.2) (2.0) (5.6) 7.8 11.1 13.4 16.6 (2.8) (3.0) (2.2) (4.6) 5.5 4.5 10.5 6.5 6.1 2.7 5.9 (1.3) (2.1) (0.9) (2.4) (2.4) (1.4) (2.4) 8.4 6.6 11.1 4.8 7.6 6.6 6.9 (1.6) (2.2) (1.0) (3.4) (2.4) (2.6) (3.1) 5.1 5.3 9.8 2.2 3.6 4.3 4.2 (1.4) (2.2) (0.9) (2.6) (2.2) (2.3) (2.1) 7.2 6.2 12.3 2.6 5.3 5.9 5.3 (1.6) (2.0) (1.0) (2.3) (2.3) (3.2) (2.3) 8.5 3.0 12.1 1.9 2.6 3.3 4.8 (2.0) (1.4) (1.0) (1.6) (1.3) (1.2) (2.1) 10.1 4.6 13.8 4.6 5.9 6.7 3.2 (2.2) (2.3) (1.1) (2.6) (2.1) (2.4) (2.3) • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.13 for national data. 1. Single-level bivariate regression of performance on the PISA index of economic, social and cultural status (ESCS), the slope is the regression coeficient for ESCS. 2. r-squared from the regression coeficient of performance on the PISA index of economic, social and cultural status (ESCS). 1 2 http://dx.doi.org/10.1787/888933003763 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 239 Annex b2: reSulTS For regIonS wIThIn counTrIeS table b2.v.7 [Part 3/3] Strength of the relationship between socio-economic status and performance in problem solving, mathematics, reading and science, by region Results based on students’ self-reports Strength of the relationship between performance in problem solving (PS) and EScS,2 compared to… Partners OECD … mathematics (PS - m) australia Australian Capital Territory New South Wales Northern Territory Queensland South Australia Tasmania Victoria Western Australia belgium flemish Community• french Community german-speaking Community canada Alberta British Columbia Manitoba New Brunswick Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan italy Centre North East North West South South Islands Portugal Alentejo Spain Basque Country• Catalonia• Madrid brazil Central-West region Northeast region North region Southeast region South region colombia Bogotá Cali Manizales Medellín united arab Emirates Abu Dhabi• Ajman Dubai• fujairah ras al-khaimah Sharjah umm al-Quwain … reading (PS - r) … Science (PS - S) … computer-based mathematics (PS - cbm) … digital reading (PS - dr) % dif. S.E. % dif. S.E. % dif. S.E. % dif. S.E. % dif. S.E. -4.1 -2.8 -9.1 -4.9 -5.2 -5.1 -2.4 -7.2 (1.7) (1.1) (4.3) (1.1) (1.1) (1.6) (1.0) (1.5) -4.3 -3.5 -7.4 -3.5 -3.5 -3.1 -1.8 -5.8 (1.8) (1.1) (4.3) (1.3) (1.3) (1.6) (1.1) (1.5) -3.1 -2.8 -9.0 -3.7 -5.3 -4.9 -2.2 -5.7 (1.8) (1.0) (3.9) (1.1) (1.4) (1.7) (1.0) (1.4) -3.3 0.1 -6.7 -1.4 -2.7 -2.9 0.2 -4.5 (1.5) (1.2) (3.0) (1.2) (1.3) (1.2) (1.1) (1.6) -3.1 -1.7 -8.4 -0.6 -2.0 -3.2 -0.2 -4.9 (1.9) (1.2) (4.0) (1.3) (1.3) (1.4) (1.0) (1.4) -3.8 -8.4 -1.0 (0.9) (1.6) (1.0) -1.5 -7.2 -1.2 (1.1) (1.9) (1.3) -3.5 -7.4 -2.5 (1.0) (1.7) (1.3) 0.0 -4.5 2.6 (1.0) (1.7) (1.3) 0.6 -1.4 2.8 (1.1) (1.7) (1.2) -2.9 -0.9 -9.0 -3.7 -4.5 -6.0 -6.8 -0.5 -8.0 -1.3 (0.9) (0.9) (1.7) (1.1) (2.0) (1.5) (1.0) (1.7) (1.2) (0.7) -2.8 0.4 -6.6 -1.9 1.6 -1.6 -5.1 1.6 -5.6 -0.1 (1.1) (1.0) (1.3) (1.1) (2.8) (1.5) (1.0) (1.8) (1.0) (0.8) -2.5 0.5 -5.8 -1.9 0.4 -1.7 -4.4 0.7 -5.2 -1.4 (1.0) (1.0) (1.2) (1.0) (2.7) (1.1) (0.8) (1.6) (1.1) (0.8) -1.1 1.3 -3.5 -1.9 -2.7 -4.9 -2.9 6.0 -2.3 -0.9 (1.2) (1.1) (1.4) (1.2) (1.8) (1.5) (1.0) (1.9) (1.0) (1.0) -0.5 -0.4 -1.7 -3.5 0.1 -0.8 -2.7 -0.7 -1.6 0.6 (1.0) (1.1) (1.4) (1.1) (2.3) (2.0) (1.1) (2.1) (0.8) (0.8) -2.6 -5.1 -2.9 -3.5 -3.7 (1.2) (1.9) (1.4) (2.3) (2.7) -4.9 -6.2 -1.0 -4.6 -2.6 (1.9) (2.3) (1.3) (2.6) (2.4) -3.7 -3.8 -2.5 -3.5 -3.2 (1.6) (2.1) (1.5) (2.6) (2.7) -1.0 -0.5 -2.4 -0.9 -4.4 (3.0) (3.0) (1.4) (4.1) (2.8) -2.1 2.1 -0.4 1.4 1.6 (2.0) (1.8) (1.1) (2.1) (1.8) -2.7 (3.1) 2.2 (3.8) 0.3 (2.5) 1.2 (3.7) 1.1 (3.2) -4.3 -12.2 -6.1 (0.9) (2.1) (3.8) -4.1 -6.8 -0.7 (1.2) (1.9) (3.8) -3.1 -9.4 -0.4 (0.9) (1.7) (3.6) -2.1 -3.9 -1.8 (0.9) (1.8) (3.9) -1.7 -3.8 -1.9 (1.0) (1.9) (4.9) -3.2 -1.3 -3.2 -1.8 2.8 (3.2) (3.2) (2.8) (1.8) (4.4) 2.9 6.3 1.8 3.2 6.1 (3.4) (3.5) (3.5) (1.6) (2.1) -0.4 2.8 2.6 0.2 4.0 (3.5) (4.2) (2.6) (1.5) (3.8) -3.0 1.9 -6.6 -5.3 0.7 (4.8) (3.1) (4.2) (2.4) (3.4) -1.4 6.3 -0.3 -0.3 4.3 (3.7) (3.9) (4.1) (1.5) (2.3) -2.9 -3.5 -7.8 -2.2 (1.5) (2.5) (3.0) (2.2) -0.5 -1.7 -5.2 2.8 (1.4) (2.3) (2.9) (3.2) -1.6 -2.8 -3.2 1.6 (1.2) (2.1) (3.5) (2.3) -0.3 5.3 3.0 4.0 (1.2) (2.2) (2.7) (2.2) -2.9 -0.2 -4.4 5.5 (1.7) (2.2) (2.8) (4.2) -2.9 -2.0 -0.5 1.7 -1.5 -3.9 -1.0 (1.1) (2.0) (0.6) (2.2) (2.1) (2.1) (2.5) 0.4 -0.8 0.7 4.3 2.5 -1.6 1.7 (1.0) (2.0) (0.7) (2.0) (2.0) (1.5) (2.1) -1.7 -1.7 -1.8 3.8 0.9 -3.2 0.7 (1.0) (1.6) (0.6) (1.6) (1.9) (2.5) (2.1) -3.0 1.6 -1.6 4.6 3.5 -0.6 1.2 (1.3) (1.5) (0.6) (2.0) (2.1) (0.9) (2.6) -4.6 0.0 -3.3 1.9 0.2 -4.0 2.7 (1.5) (2.3) (0.6) (2.1) (2.8) (1.8) (2.1) • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.13 for national data. 1. Single-level bivariate regression of performance on the PISA index of economic, social and cultural status (ESCS), the slope is the regression coeficient for ESCS. 2. r-squared from the regression coeficient of performance on the PISA index of economic, social and cultural status (ESCS). 1 2 http://dx.doi.org/10.1787/888933003763 240 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V reSulTS For regIonS wIThIn counTrIeS: Annex b2 table b2.v.8 [Part 1/1] Performance in problem solving and use of a computer at home, by region Results based on students’ self-reports Students who use a desktop, laptop or tablet computer at home difference in problem-solving performance Percentage of students Partners OECD all students australia Australian Capital Territory New South Wales Northern Territory Queensland South Australia Tasmania Victoria Western Australia belgium flemish Community• french Community german-speaking Community canada Alberta British Columbia Manitoba New Brunswick Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan italy Centre North East North West South South Islands Portugal Alentejo Spain Basque Country• Catalonia• Madrid brazil Central-West region Northeast region North region Southeast region South region colombia Bogotá Cali Manizales Medellín united arab Emirates Abu Dhabi• Ajman Dubai• fujairah ras al-khaimah Sharjah umm al-Quwain boys Gender difference (b - G) difference related to parents’ Parents’ highest highest Parents’ occupation: occupation: highest Skilled occupation: Semi-skilled or elementary semi-skilled Skilled (iSco 1 to 3) (iSco 4 to 9) or elementary after accounting for sociodemographic characteristics of students1 Score dif. S.E. S.E. observed Score dif. S.E. 3.9 3.7 4.4 3.7 1.6 1.5 0.6 2.2 (0.8) (0.4) (1.8) (0.4) (0.4) (0.7) (0.2) (0.5) c 77 104 79 46 68 77 61 c (7.7) (33.1) (13.1) (18.2) (16.3) (17.5) (14.7) c 48 70 55 35 24 m 33 c (7.6) (30.7) (14.9) (18.0) (14.6) m (14.2) (0.2) (0.3) (0.5) 1.0 1.5 1.2 (0.2) (0.3) (0.6) 96 65 c (12.6) (15.3) c 69 42 c (14.7) (13.5) c m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m (0.2) (0.3) (0.4) (0.5) (0.1) 94.4 96.9 98.2 96.8 97.4 (1.2) (0.3) (0.5) (0.5) (0.4) 4.5 1.7 -0.5 1.5 2.4 (1.3) (0.3) (0.7) (0.8) (0.4) c 69 c c c c (28.8) c c c c 20 c c c c (25.8) c c c 99.5 (0.2) 97.4 (0.4) 2.1 (0.4) c c c c (0.4) (0.4) (0.7) 97.2 99.2 98.8 (0.2) (0.2) (0.4) 95.7 98.5 97.5 (0.3) (0.2) (0.9) 1.6 0.7 1.3 (0.3) (0.2) (0.7) 58 c c (17.5) c c 46 c c (13.5) c c m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m Girls % S.E. % S.E. % S.E. 99.0 96.8 92.8 95.9 97.9 93.5 98.8 96.6 (0.2) (0.2) (0.9) (0.2) (0.2) (0.4) (0.1) (0.3) 98.7 96.6 91.2 95.1 97.8 92.9 98.6 95.8 (0.3) (0.2) (1.6) (0.4) (0.3) (0.6) (0.2) (0.4) 99.3 97.1 94.3 96.7 98.0 94.2 99.0 97.6 (0.3) (0.2) (0.6) (0.2) (0.4) (0.5) (0.1) (0.5) 98.9 97.3 98.4 (0.1) (0.2) (0.2) 98.8 97.1 97.3 (0.2) (0.2) (0.3) 99.0 97.5 99.5 m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m 96.5 97.3 97.8 97.4 98.0 (0.7) (0.3) (0.3) (0.3) (0.3) 94.9 96.7 97.4 97.5 98.0 97.8 (0.3) 96.3 98.7 98.2 % dif. S.E. % S.E. % S.E. -0.6 -0.6 -3.2 -1.6 -0.2 -1.3 -0.4 -1.8 (0.5) (0.3) (1.7) (0.5) (0.5) (0.8) (0.3) (0.7) 99.7 98.3 95.5 97.2 98.5 95.3 99.2 97.6 (0.2) (0.1) (1.1) (0.2) (0.2) (0.5) (0.1) (0.3) 95.7 94.6 91.1 93.6 96.9 93.8 98.6 95.5 (0.8) (0.4) (1.4) (0.5) (0.4) (0.6) (0.2) (0.5) (0.1) (0.2) (0.2) -0.3 -0.4 -2.2 (0.2) (0.3) (0.4) 99.3 98.2 99.2 (0.1) (0.1) (0.2) 98.4 96.7 98.0 m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m (1.1) (0.5) (0.4) (0.4) (0.4) 98.5 98.0 98.2 97.3 98.0 (0.3) (0.4) (0.3) (0.4) (0.6) -3.6 -1.3 -0.9 0.2 0.0 (1.1) (0.7) (0.4) (0.6) (0.8) 98.9 98.6 97.7 98.3 99.8 97.8 (0.5) 97.9 (0.4) -0.1 (0.6) (0.2) (0.2) (0.7) 95.6 98.9 97.7 (0.3) (0.2) (0.8) 96.9 98.5 98.6 (0.2) (0.4) (0.6) -1.3 0.4 -0.9 m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m % dif. • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.25 for national data. 1. The adjusted result corresponds to the coeficient from a regression where the PISA index of economic, social and cultural status (ESCS), ESCS squared, boy, and an immigrant (irst generation) dummy are introduced as further independent variables. 1 2 http://dx.doi.org/10.1787/888933003763 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 241 Annex b2: reSulTS For regIonS wIThIn counTrIeS table b2.v.9 [Part 1/1] Performance in problem solving and use of computers at school, by region Results based on students’ self-reports Students who use a desktop, laptop or tablet computer at school difference in problem-solving performance Percentage of students Partners OECD all students australia Australian Capital Territory New South Wales Northern Territory Queensland South Australia Tasmania Victoria Western Australia belgium flemish Community• french Community german-speaking Community canada Alberta British Columbia Manitoba New Brunswick Newfoundland and Labrador Nova Scotia Ontario Prince Edward Island Quebec Saskatchewan italy Centre North East North West South South Islands Portugal Alentejo Spain Basque Country• Catalonia• Madrid brazil Central-West region Northeast region North region Southeast region South region colombia Bogotá Cali Manizales Medellín united arab Emirates Abu Dhabi• Ajman Dubai• fujairah ras al-khaimah Sharjah umm al-Quwain boys Gender difference (b - G) Girls % S.E. % S.E. % S.E. 93.5 89.8 89.3 94.4 97.5 97.4 96.5 94.2 (0.4) (0.3) (1.5) (0.2) (0.3) (0.3) (0.2) (0.3) 93.1 89.8 92.9 92.6 97.9 96.4 96.5 95.1 (0.6) (0.4) (1.5) (0.5) (0.5) (0.5) (0.3) (0.4) 93.9 89.7 86.1 96.2 97.2 98.4 96.4 93.2 (0.7) (0.4) (1.8) (0.3) (0.3) (0.3) (0.3) (0.6) 86.2 37.2 60.6 (0.4) (0.7) (0.6) 84.9 39.0 60.3 (0.5) (0.9) (0.9) 87.5 35.4 61.0 m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m 61.4 74.6 64.7 68.4 63.8 (1.5) (1.2) (1.3) (1.8) (1.5) 67.9 74.7 69.1 70.6 71.1 76.5 (0.9) 74.6 85.3 77.0 % dif. difference related to parents’ Parents’ highest highest Parents’ occupation: occupation: highest Skilled occupation: Semi-skilled or elementary semi-skilled Skilled (iSco 1 to 3) (iSco 4 to 9) or elementary % dif. S.E. observed Score dif. S.E. after accounting for sociodemographic characteristics of students1 Score dif. S.E. S.E. % S.E. % S.E. -0.8 0.1 6.8 -3.6 0.7 -2.0 0.1 1.9 (0.9) (0.5) (1.6) (0.6) (0.6) (0.6) (0.5) (0.8) 94.2 90.9 87.6 95.5 97.9 97.7 96.8 95.0 (0.5) (0.4) (2.1) (0.3) (0.3) (0.4) (0.2) (0.4) 93.7 88.6 93.2 93.0 97.1 97.0 95.7 92.9 (1.1) (0.5) (1.1) (0.5) (0.5) (0.5) (0.4) (0.6) 0.5 2.3 -5.6 2.4 0.8 0.7 1.1 2.1 (1.2) (0.5) (2.1) (0.5) (0.5) (0.7) (0.4) (0.7) 40 35 4 69 56 51 25 0 (17.3) (6.1) (29.2) (11.1) (18.8) (26.6) (11.6) (12.7) 32 23 -4 59 45 28 20 -8 (16.3) (6.3) (22.2) (11.0) (16.8) (20.6) (10.9) (11.5) (0.5) (0.8) (1.0) -2.6 3.6 -0.7 (0.5) (1.0) (1.5) 86.6 35.6 58.7 (0.5) (0.9) (1.0) 85.8 39.1 62.8 (0.8) (1.1) (1.4) 0.8 -3.5 -4.0 (1.0) (1.4) (1.9) 16 -27 -7 (5.7) (5.3) (7.1) 12 -25 -6 (4.5) (4.9) (7.1) m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m (2.4) (1.6) (1.6) (2.3) (1.9) 53.5 74.5 59.9 65.6 56.3 (2.1) (1.2) (1.8) (2.8) (2.4) 14.4 0.1 9.2 5.0 14.8 (3.4) (1.6) (1.9) (3.5) (3.1) 56.6 70.7 57.7 62.2 57.1 (1.7) (1.5) (1.7) (2.8) (1.9) 65.6 77.5 70.1 71.4 66.5 (2.0) (1.3) (1.2) (2.0) (1.5) -9.0 -6.8 -12.4 -9.1 -9.4 (2.2) (1.6) (0.9) (2.4) (1.5) -1 -5 -16 -3 -24 (12.6) (9.2) (6.8) (11.7) (11.7) 1 -2 -12 2 -22 (11.6) (9.3) (6.4) (10.3) (10.6) 74.7 (1.1) 78.2 (1.0) -3.5 (1.0) 76.9 (2.0) 76.8 (1.0) 0.2 (2.2) -20 (8.1) -15 (9.7) (0.8) (1.2) (1.3) 74.1 85.0 79.5 (0.9) (1.2) (1.4) 75.1 85.6 74.6 (0.8) (1.4) (1.7) -1.1 -0.6 4.9 (0.6) (1.1) (1.5) 71.9 84.5 75.8 (0.9) (1.7) (2.2) 77.8 86.3 78.2 (0.8) (0.9) (1.5) -5.9 -1.9 -2.4 (0.6) (1.5) (2.5) -2 26 11 (4.2) (11.2) (19.8) 1 26 4 (3.8) (10.1) (15.4) m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m • PISA adjudicated region. Notes: Values that are statistically signiicant are indicated in bold (see Annex A3). Italian administrative regions are grouped into larger geographical units: Centre (Lazio, Marche, Toscana, Umbria), north East (Bolzano, Emilia Romagna, Friuli Venezia Giulia, Trento, Veneto), north West (Liguria, Lombardia, Piemonte, Valle d’Aosta), South (Abruzzo, Campania, Molise, Puglia), South Islands (Basilicata, Calabria, Sardegna, Sicilia). brazilian states are grouped into larger geographical units: Central-West region (Federal District, Goiás, Mato Grosso, Mato Grosso do Sul), northeast region (Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte, Sergipe), north region (Acre, Amapá, Amazonas, Pará, Rondônia, Roraima, Tocantins), Southeast region (Espírito Santo, Minas Gerais, Rio de Janeiro, São Paulo), South region (Paraná, Rio Grande do Sul, Santa Catarina). See Table V.4.26 for national data. 1. The adjusted result corresponds to the coeficient from a regression where the PISA index of economic, social and cultural status (ESCS), ESCS squared, boy, and an immigrant (irst-generation) dummy are introduced as further independent variables. 1 2 http://dx.doi.org/10.1787/888933003763 242 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V lIST oF TAbleS AvAIlAble on lIne: Annex b3 Annex b3 lIST oF TAbleS AvAIlAble on lIne The following tables are available in electronic form only. chapter 4 How problem-solving performance varies within countries http://dx.doi.org/10.1787/888933003706 WEb Table V.4.5 Differences in problem-solving, mathematics, reading and science performance related to education tracks WEb Table V.4.11c Performance on problem-solving tasks, by technology setting and by gender WEb Table V.4.11d Performance on problem-solving tasks, by social focus and by gender WEb Table V.4.11e Performance on problem-solving tasks, by response format and by gender WEb Table V.4.18c Performance on problem-solving tasks, by technology setting and by parents’ occupational status WEb Table V.4.18d Performance on problem-solving tasks, by social focus and by parents’ occupational status WEb Table V.4.18e Performance on problem-solving tasks, by response format and by parents’ occupational status WEb Table V.4.22c Performance on problem-solving tasks, by technology setting and by immigrant background WEb Table V.4.22d Performance on problem-solving tasks, by social focus and by immigrant background WEb Table V.4.22e Performance on problem-solving tasks, by response format and by immigrant background annex b2 results for regions within countries http://dx.doi.org/10.1787/888933003763 WEb Table B2.V.10 Performance in problem solving, by nature of the problem situation and by region WEb Table B2.V.11 Performance in problem solving, by process and by region WEb Table B2.V.12 relative performance on knowledge-acquisition and knowledge-utilisation tasks, by region These tables, as well as additional material, may be found at: www.pisa.oecd.org. CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 243 Annex C thE dEvEloPmEnt and imPlEmEntation of PiSa – a collaborativE Effort CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 245 Annex c: The develoPmenT And ImPlemenTATIon oF PISA – A collAborATIve eFForT PISA is a collaborative effort, bringing together experts from the participating countries, steered jointly by their governments on the basis of shared, policy-driven interests. A PISA governing Board, on which each country is represented, determines the policy priorities for PISA, in the context of OECD objectives, and oversees adherence to these priorities during the implementation of the programme. This includes setting priorities for the development of indicators, for establishing the assessment instruments, and for reporting the results. Experts from participating countries also serve on working groups that are charged with linking policy objectives with the best internationally available technical expertise. By participating in these expert groups, countries ensure that the instruments are internationally valid and take into account the cultural and educational contexts in OECD member and partner countries and economies, that the assessment materials have strong measurement properties, and that the instruments place emphasise authenticity and educational validity. Through National Project Managers, participating countries and economies implement PISA at the national level subject to the agreed administration procedures. National Project Managers play a vital role in ensuring that the implementation of the survey is of high quality, and verify and evaluate the survey results, analyses, reports and publications. The design and implementation of the surveys, within the framework established by the PISA governing Board, is the responsibility of external contractors. for PISA 2012, the development and implementation of the cognitive assessment and questionnaires, and of the international options, was carried out by a consortium led by the Australian Council for Educational research (ACEr). Other partners in this Consortium include cApStAn Linguistic Quality Control in Belgium, the Centre de recherche Public Henri Tudor (CrP-HT) in Luxembourg, the Department of Teacher Education and School research (ILS) at the university of Oslo in Norway, the Deutsches Institut für Internationale Pädagogische forschung (DIPf) in germany, the Educational Testing Service (ETS) in the united States, the Leibniz Institute for Science and Mathematics Education (IPN) in germany, the National Institute for Educational Policy research in Japan (NIEr), the unité d’analyse des systèmes et des pratiques d’enseignement (aSPe) at the university of Liège in Belgium, and WESTAT in the united States, as well as individual consultants from several countries. ACEr also collaborated with Achieve, Inc. in the united States to develop the mathematics framework for PISA 2012. The OECD Secretariat has overall managerial responsibility for the programme, monitors its implementation daily, acts as the secretariat for the PISA governing Board, builds consensus among countries and serves as the interlocutor between the PISA governing Board and the international Consortium charged with implementing the activities. The OECD Secretariat also produces the indicators and analyses and prepares the international reports and publications in co-operation with the PISA Consortium and in close consultation with member and partner countries and economies both at the policy level (PISA governing Board) and at the level of implementation (National Project Managers). 246 PISA Governing Board mexico: francisco Ciscomani and Eduardo Backhoff Escudero Chair of the PISA Governing Board: Lorna Bertrand netherlands: Paul van Oijen OECD countries new Zealand: Lynne Whitney australia: Tony Zanderigo norway : Anne-Berit kavli and Alette Schreiner austria: Mark Német Poland: Stanislaw Drzazdzewski and Hania Bouacid belgium: Christiane Blondin and Isabelle Erauw Portugal: Luisa Canto and Castro Loura canada: Pierre Brochu, Patrick Bussiere and Tomasz gluszynski Slovak republic: romana kanovska and Paulina korsnakova chile: Leonor Cariola Huerta Slovenia: Andreja Barle Lakota czech republic: Jana Paleckova Spain: Ismael Sanz Labrador denmark: Tine Bak and Elsebeth Aller Sweden: Anita Wester Estonia: Maie kitsing Switzerland: Vera Husfeldt and Claudia Zahner rossier finland: Tommi karjalainen turkey: Nurcan Devici and Mustafa Nadir Çalis france: Bruno Trosseille united kingdom: Lorna Bertrand and Jonathan Wright Germany: Elfriede Ohrnberger and Susanne von Below united States: Jack Buckley, Dana kelly and Daniel Mcgrath Greece: Vassilia Hatzinikita and Chryssa Soianopoulou Observers hungary: Benõ Csapó albania: Ermal Elezi iceland: Júlíus Björnsson argentina: Liliana Pascual ireland: Jude Cosgrove and gerry Shiel brazil: Luiz Claudio Costa israel: Michal Beller and Hagit glickman bulgaria: Neda kristanova italy: Paolo Sestito chinese taipei: gwo-Dong Chen and Chih-Wei Hue Japan: ryo Watanabe colombia: Adriana Molina korea: Sungsook kim and keunwoo Lee costa rica: Leonardo garnier rimolo luxembourg: Amina kafai croatia: Michelle Bras roth © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V The develoPmenT And ImPlemenTATIon oF PISA – A collAborATIve eFForT: Annex c hong kong-china: Esther Sui-chu Ho korea: Ji-Min Cho and Mi-Young Song indonesia: khairil Anwar Notodiputro latvia: Andris kangro Jordan: khattab Mohammad Abulibdeh liechtenstein: Christian Nidegger kazakhstan: Almagul kultumanova lithuania: Mindaugas Stundza latvia: Andris kangro, Ennata kivrina and Dita Traidas luxembourg: Bettina Boehm lithuania: rita Dukynaite macao-china: kwok Cheung Cheung macao-china: Leong Lai malaysia: Ihsan Ismail and Muhamad Zaini Md Zain montenegro: Zeljko Jacimovic mexico: María Antonieta Díaz gutierrez Panama: Arturo rivera montenegro: Divna Paljevic Sturm Peru: Liliana Miranda Molina netherlands: Jesse koops Qatar: Hamda Al Sulaiti new Zealand: kate Lang and Steven May romania: roxana Mihail norway: Marit kjaernsli russian federation: Isak froumin and galina kovaleva Peru: Liliana Miranda Molina Poland: Michal federowicz Serbia: Dragica Pavlovic-Babic Portugal: Ana Sousa ferreira Shanghai-china: Minxuan Zhang Qatar: Aysha Al-Hashemi and Assad Tounakti Singapore: khah gek Low romania: Silviu Cristian Mirescu thailand: Precharn Dechsri russian federation: galina kovaleva united arab Emirates: Moza al ghuly and Ayesha g. khalfan Almerri Scotland: rebecca Wheater uruguay: Andrés Peri and Maria Helvecia Sanchez Nunez Serbia: Dragica Pavlovic-Babic viet nam: Le Thi My Ha Shanghai-china: Jing Lu and Minxuan Zhang Singapore: Chew Leng Poon and Sean Tan PISA 2012 National Project Managers Slovak republic: Julia Miklovicova and Jana ferencova albania: Alfonso Harizaj Slovenia: Mojca Straus argentina: Liliana Pascual Spain: Lis Cercadillo Pérez australia: Sue Thomson Sweden: Magnus Oskarsson austria: ursula Schwantner Switzerland: Christian Nidegger belgium: Inge De Meyer and Ariane Baye chinese taipei: Pi-Hsia Hung brazil: João galvão Bacchetto thailand: Sunee klainin bulgaria: Svetla Petrova tunisia: Mohamed kamel Essid canada: Pierre Brochu and Tamara knighton turkey: Serdar Aztekin chile: Ema Lagos Campos united arab Emirates: Moza al ghuly colombia: francisco reyes united kingdom: rebecca Wheater costa rica: Lilliam Mora united States: Dana kelly and Holly Xie croatia: Michelle Bras roth uruguay: Maria Helvecia Sánchez Nunez czech republic: Jana Paleckova viet nam: Thi My Ha Le denmark: Niels Egelund OECD Secretariat Estonia: gunda Tire Andreas Schleicher (Strategic development) finland: Jouni Välijärvi Marilyn Achiron (Editorial support) france: ginette Bourny francesco Avvisati (Analytic services) Germany: Christine Sälzer and Manfred Prenzel Brigitte Beyeler (Administrative support) Greece: Vassilia Hatzinikita Simone Bloem (Analytic services) hong kong-china: Esther Sui-chu Ho Marika Boiron (Translation support) hungary: Ildikó Balazsi francesca Borgonovi (Analytic services) iceland: Almar Midvik Halldorsson Jenny Bradshaw (Project management) indonesia: Yulia Wardhani Nugaan and Hari Setiadi Célia Braga-Schich (Production support) ireland: gerry Shiel and rachel Perkins Claire Chetcuti (Administrative support) israel: Joel rapp and Inbal ron-kaplan Michael Davidson (Project management and analytic services) italy: Carlo Di Chiacchio Cassandra Davis (Dissemination co-ordination) Japan: ryo Watanabe Elizabeth Del Bourgo (Production support) Jordan: khattab Mohammad Abulibdeh Juliet Evans (Administration and partner country/economy relations) kazakhstan: gulmira Berdibayeva and Zhannur Azmagambetova CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 247 Annex c: The develoPmenT And ImPlemenTATIon oF PISA – A collAborATIve eFForT Tue Halgreen (Project management) Jaap Scheerens (university of Twente, Netherlands) Miyako Ikeda (Analytic services) William Schmidt (Michigan State university, united States) Tadakazu Miki (Analytic services) fons van de Vijver (Tilburg university, Netherlands) guillermo Montt (Analytic services) giannina rech (Analytic services) Diana Tramontano (Administration) Sophie Vayssettes (Analytic services) Elisabeth Villoutreix (Production co-ordination) Pablo Zoido (Analytic services) keith rust (Chair) (Westat, united States) ray Adams (ACEr, Australia) Cees glas (university of Twente, Netherlands) John de Jong (Language Testing Services, Netherlands) David kaplan (university of Wisconsin – Madison, united States) PISA 2012 mathematics expert group Christian Monseur (university of Liège, Belgium) kaye Stacey (Chair) (university of Melbourne, Australia) Caroline Bardini (university of Melbourne, Australia) Sophia rabe-Hesketh (university of California – Berkeley, united States) Werner Blum (university of kassel, germany) Thierry rocher (Ministry of Education, france) Joan ferrini-Mundy (Michigan State university, united States) Norman Verhelst (CITO, Netherlands) Solomon garfunkel (COMAP, united States) kentaro Yamamoto (ETS, united States) Toshikazu Ikeda (Yokohama National university, Japan) rebecca Zwick (university of California, united States) Zbigniew Marciniak (Warsaw university, Poland) Mogens Niss (roskilde university, Denmark) Martin ripley (World Class Arena Limited, united kingdom) William Schmidt (Michigan State university, united States) PISA 2012 Consortium Australian Council for Educational Research ray Adams (International Project Director) Susan Bates (Project administration) PISA 2012 problem solving expert group Alla Berezner (Data management and analysis) Joachim funke (Chair) (university of Heidelberg, germany) Yan Bibby (Data processing and analysis) Benő Csapó (university of Szeged, Hungary) Phillipe Bickham (IT services) John Dossey (Illinois State university, united States) Esther Brakey (Administrative support) Arthur graesser (The university of Memphis united States) robin Buckley (IT services) Detlev Leutner (Duisburg-Essen university, germany) Mark Butler (financial literacy instruments and test development) romain Martin (université de Luxembourg fLSHASE, Luxembourg) Wei Buttress (Project administration and quality monitoring) richard Mayer (university of California, united States) renee Chow (Data processing and analysis) Ming Ming Tan (Ministry of Education, Singapore) John Cresswell (reporting and dissemination) PISA 2012 inancial literacy expert group Alex Daraganov (Data processing and analysis) Annamaria Lusardi (Chair) (The george Washington university School of Business, united States) Jorge fallas (Data processing and analysis) Jean-Pierre Boisivon (université de Paris II Panthéon-Assas, france) kim fitzgerald (IT Services) Diana Crossan (Commission for financial Literacy and retirement Income, New Zealand) Jennifer Hong (Data processing and sampling) Peter Cuzner (Australian Securities and Investments Commission, Australia) Winson Lam (IT services) Jeanne Hogarth (federal reserve System, united States) Dušan Hradil (Ministry of finance, Czech republic) Stan Jones (Consultant, Canada) Sue Lewis (Consultant, united kingdom) PISA 2012 questionnaire expert group Eckhard klieme (Chair) (Deutsches Institut für Internationale Pädagogische forschung (DIPf), germany) Eduardo Backhoff (university of Baja California at the Institute of Educational research and Development, Mexico) Ying-yi Hong (Nanyang Business School of Nanyang Technological university, Singapore) David kaplan (university of Wisconsin – Madison, united States) Henry Levin (Columbia university, united States) 248 Technical advisory group kate fitzgerald (Data processing and sampling) Paul golden (IT and helpdesk support) Nora kovarcikova (Survey operations) Petra Lietz (Questionnaire development) Tom Lumley (reading instruments and test development) greg Macaskill (Data management and processing and sampling) ron Martin (Science instruments and test development) Barry McCrae (Problem solving and science instruments and test development) Louise McDonald (graphic design) Juliette Mendelovits (reading and inancial literacy instruments and test development) martin murphy (field operations and sampling) Thoa nguyen (Data processing and analysis) Stephen Oakes (IT management and support) Elizabeth O’grady (Questionnaire development and project support) © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V The develoPmenT And ImPlemenTATIon oF PISA – A collAborATIve eFForT: Annex c Penny Pearson (Administrative support) management [Delivery System, Translation System]) ray Peck (mathematics and inancial literacy instruments and test development) brigitte Steinert (Questionnaire development) fei Peng (Quality monitoring and project support) Svenja Vieluf (Questionnaire development) ray Philpot (Problem Solving instruments and test development) Institutt for Lærerutdanning Og Skoleutvikling (ILS, NORWAY) Anna Plotka (graphic design) bjornar Alseth (mathematics instruments and test development) Dara ramalingam (reading instruments and test development) Sima rodrigues (Data processing and analysis) Ole kristian bergem (mathematics instruments and test development) Alla routitsky (Data management and processing) knut Skrindo (mathematics instruments and test development) James Spithill (mathematics instruments and test development) rolf V. Olsen (mathematics instruments and test development) rachel Stanyon (Project support) Arne Hole (mathematics instruments and test development) naoko Tabata (Survey operations) Therese Hopfenbeck (Problem-solving instruments and test development) Stephanie Templeton (Project administration and support) mollie Tobin (Questionnaire development and project support) David Tout (mathematics instruments and test development) Leibniz Institute for Science and Mathematics Education (IPN, GERMANY) ross Turner (management, mathematics instruments and test development) Christoph Duchhardt (mathematics instruments and test development) maryanne Van grunsven (Project support) Aiso Heinze (mathematics instruments and test development) Charlotte Waters (Project administration, data processing and analysis) Eva knopp (mathematics instruments and test development) martin Senkbeil (mathematics instruments and test development) maurice Walker (management, computer-based assessment) louise Wenn (Data processing and analysis) Yan Wiwecka (IT services) National Institute for Educational Policy Research (NIER, JAPAN) cApStAn Linguistic Quality Control (BELGIUM) keiichi nishimura (mathematics instruments and test development) raphael Choppinet (Computer-based veriication management) Yuji Surata (mathematics instruments and test development) Steve Dept (Translation and veriication operations) Andrea ferrari (linguistic quality assurance and quality control designs) musab Hayatli (right-to-left scripts, cultural adaptations) Elica krajceva (Questionnaire veriication co-ordinator) Shinoh lee (Cognitive test veriication co-ordinator) Irene liberati (manuals veriication co-ordinator) laura Wayrynen (Veriier training and veriication procedures) Educational Testing Service (ETS) The TAO Initiative: Henry Tudor Public Research Centre, University of Luxembourg (LUXEMBOURG) Joel billard (Software Engineer, School Questionnaire) marilyn binkley (Project Consultant, Assessment Expert) Jerome bogaerts (Software Engineer, TAO Platform) gilbert busana (Electronic Instruments, usability) Christophe Henry (System Engineer, School Questionnaire and Hosting) Jonas bertling (Questionnaire instruments and test development) raynald Jadoul (Technical Lead, School Questionnaire and Electronic Instruments) Irwin kirsch (reading Components) Isabelle Jars (Project Manager) Patricia klag (Problem-solving instruments and test development) Vincent koenig (Electronic Instruments, usability) Patrick kyllonen (Questionnaire instruments and test development) Thibaud Latour (Project Leader, TAO Platform) marylou lennon (Questionnaire instruments and test development) Primael Lorbat (Software Engineer, Electronic Instruments) richard roberts (Questionnaire instruments and test development) Matteo Melis (Software Engineer, School Questionnaire) matthias von Davier (Questionnaire instruments and test development) Lionel Lecaque (Software Engineer, Quality) romain Martin (Problem Solving Expert group Member) Patrick Plichart (Software Architect, TAO Platform) Vincent Porro (Software Engineer, Electronic Instruments) kentaro Yamamoto (member TAg, problem-solving instruments and test development) Igor ribassin (Software Engineer, Electronic Instruments) Deutches Institut für Internationale Pädagogische Forschung (DIPF, GERMANY ) Unité d’analyse des Systèmes et des Pratiques d’enseignement (ASPE, BELGIUM) frank goldhammer (Test developer, problem solving) Eckhard klieme (Chair of Questionnaire Expert group) Isabelle Demonty (Mathematics instruments and test development) Silke Hertel (Questionnaire development) Annick fagnant (Mathematics instruments and test development) Jean-Paul reeff (International Consultant) Anne Matoul (french source development) Heiko rolke (Software Design & Software Development Christian Monseur (Member of Technical Advisory group) Somsack Sipasseuth (Software Engineer, Electronic Instruments) CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V © OECD 2014 249 Annex c: The develoPmenT And ImPlemenTATIon oF PISA – A collAborATIve eFForT WESTAT Susan fuss (Sampling and weighting) Amita gopinath (Weighting) Jing kang (Sampling and weighting) Sheila krawchuk (Sampling, weighting and quality monitoring) Thanh Le (Sampling, weighting and quality monitoring) John Lopdell (Sampling and weighting) keith rust (Director of the PISA Consortium for sampling and weighting) Erin Willey (Sampling and weighting) Shawn Lu (Weighting) Teresa Strickler (Weighting) Yumiko Sugawara (Weighting) Joel Wakesberg (Sampling and weighting) Sergey Yagodin (Weighting) Achieve Inc. Michael Cohen (Mathematics framework development) kaye forgione (Mathematics framework development) Morgan Saxby (Mathematics framework development) Laura Slover (Mathematics framework development) Bonnie Verrico (Project support) HallStat SPRL Beatrice Halleux (Consultant, translation/veriication referee, french source development) University of Heidelberg Joachim funke (Chair, Problem Solving Expert group) Samuel greiff (Problem-solving instruments and test development) University of Melbourne Caroline bardini (member mathematics Expert group) John Dowsey (mathematics instruments and test development) Derek Holton (mathematics instruments and test development) kaye Stacey (Chair mathematics Expert group) Other experts michael besser (mathematics instruments and test development, university of kassel, germany) khurrem Jehangir (Data analysis for TAg, university of Twente, netherlands) kees lagerwaard (mathematics instruments and test development, Institute for Educational measurement of netherlands, netherlands) Dominik leiss (mathematics instruments and test development, university of kassel, germany) Anne-laure monnier (Consultant french source development, france) Hanako Senuma (mathematics instruments and test development, Tamagawa university, Japan) Publication layout fung kwan Tam 250 © OECD 2014 CrEATIVE PrOBLEM SOLVINg: STuDENTS’ SkILLS IN TACkLINg rEAL-LIfE PrOBLEMS – VOLuME V ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT the oecd is a unique forum where governments work together to address the economic, social and environmental challenges of globalisation. the oecd is also at the forefront of efforts to understand and to help governments respond to new developments and concerns, such as corporate governance, the information economy and the challenges of an ageing population. the organisation provides a setting where governments can compare policy experiences, seek answers to common problems, identify good practice and work to co-ordinate domestic and international policies. the oecd member countries are: australia, austria, belgium, canada, chile, the czech republic, denmark, estonia, finland, france, germany, greece, Hungary, iceland, ireland, israel, italy, Japan, korea, luxembourg, mexico, the netherlands, new Zealand, norway, poland, portugal, the Slovak republic, Slovenia, Spain, Sweden, Switzerland, turkey, the united kingdom and the united States. the european union takes part in the work of the oecd. oecd publishing disseminates widely the results of the organisation’s statistics gathering and research on economic, social and environmental issues, as well as the conventions, guidelines and standards agreed by its members. oecd publiSHing, 2, rue andré-pascal, 75775 pariS cedeX 16 (98 2014 01 1p) iSbn 978-92-64 20806-3 – 2014-04 PISA 2012 Results: Creative Problem Solving StudentS’ SkillS in tackling real-life problemS Volume V the oecd programme for international Student assessment (piSa) examines not just what students know in mathematics, reading and science, but what they can do with what they know. this is one of six volumes that present the results of the 2012 piSa survey, the ifth round of the triennial assessment. Volume i, What Students Know and Can Do: Student Performance in Mathematics, Reading and Science, summarises the performance of students in piSa 2012. Volume ii, Excellence through Equity: Giving Every Student the Chance to Succeed, deines and measures equity in education and analyses how equity in education has evolved across countries between piSa 2003 and piSa 2012. Volume iii, Ready to Learn: Students’ Engagement, Drive and Self-Beliefs, explores students’ engagement with and at school, their drive and motivation to succeed, and the beliefs they hold about themselves as mathematics learners. Volume iV, What Makes Schools Successful? Resources, Policies and Practices, examines how student performance is associated with various characteristics of individual schools and school systems. Volume V, Creative Problem Solving: Students’ Skills in Tackling Real-Life Problems, presents student performance in the piSa 2012 assessment of problem solving, which measures students’ capacity to respond to non-routine situations. Volume Vi, Students and Money: Financial Literacy Skills for the 21st Century, examines students’ experience with and knowledge about money. Contents of this volume chapter 1. assessing problem-solving skills in piSa 2012 chapter 2. Student performance in problem solving chapter 3. Students’ strengths and weaknesses in problem solving chapter 4. How problem-solving performance varies within countries chapter 5. implications of the problem-solving assessment for policy and practice Consult this publication on line at: http://dx.doi.org/10.1787/9789264208070-en This work is published on the OECD iLibrary, which gathers all OECD books, periodicals and statistical databases. Visit www.oecd-ilibrary.org and do not hesitate to contact us for more information. 2014 ISBN 978-92-64-20806-3 98 2014 01 1P