Until recently, most of the digital literacy frameworks have been based on assessment frameworks ... more Until recently, most of the digital literacy frameworks have been based on assessment frameworks used by commercial entities. The release of the DigComp framework has allowed the development of tailored implementations for the evaluation of digital competence. However, the majority of these digital literacy frameworks are based on self-assessments, measuring only low-order cognitive skills. This paper reports on a study to develop and validate an assessment instrument, including interactive simulations to assess citizens’ digital competence. These formats are particularly important for the evaluation of complex cognitive constructs such as digital competence. Additionally, we selected two different approaches for designing the tests based on their scope, at the competence or competence area level. Their overall and dimensional validity and reliability were analysed. We summarise the issues addressed in each phase and key points to consider in new implementations. For both approaches...
ABSTRACT Background and Context: Despite many initiatives to develop Computational Thinking (CT),... more ABSTRACT Background and Context: Despite many initiatives to develop Computational Thinking (CT), not much is known about how early programmers develop CT and how we can assess their learning. Objective: Determine if the analysis of students’ interactions with an online platform allows understanding the development of CT, how we can convert data collected into valuable insights, and the aspects that should be considered in platforms design. Method: We developed an online platform with a fine-grained log–recording system. We analysed the data collected from 1004 students (ages 8-14) to understand the difficulties they face. We explain our platform and the tools to process and filter the interaction logs. We calculate additional indicators that provide useful information about student’s behaviour. Findings: Age and gender have shown to influence on CT learning. Generating additional indicators from basic interaction data provide valuable insights. We provide a list of recommendations for developing more effective programming learning platforms.
Technical noise can compromise the precision and accuracy of the reaction times collected in psyc... more Technical noise can compromise the precision and accuracy of the reaction times collected in psychological experiments, especially in the case of Internet-based studies. Although this noise seems to have only a small impact on traditional statistical analyses, its effects on model fit to reaction-time distributions remains unexplored. Across four simulations we study the impact of technical noise on parameter recovery from data generated from an ex-Gaussian distribution and from a Ratcliff Diffusion Model. Our results suggest that the impact of noise-induced variance tends to be limited to specific parameters and conditions. Although we encourage researchers to adopt all measures to reduce the impact of noise on reaction-time experiments, we conclude that the typical amount of noise-induced variance found in these experiments does not pose substantial problems for statistical analyses based on model fitting.
Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality, 2020
This paper reports on an exploratory study with the aim to validate item response processes in di... more This paper reports on an exploratory study with the aim to validate item response processes in digital competence assessment through eye-tracking techniques. When measuring complex cognitive constructs, it is crucial to correctly design the evaluation items to trigger the intended knowledge and skills. Furthermore, to assess the validity of a test requires considering not only the content of the evaluation tasks involved in the test, but also whether examinees respond to the tasks by engaging construct-relevant response processes. The eye tracking observations helped to fill an ‘explanatory gap’ by providing data on variation in item response processes that are not captured by other sources of process data such as think aloud protocols or computer-generated log files. We proposed a set of metrics that could help test designers to validate the different item formats used in the evaluation of digital competence. The gaze data provided detailed information on test item response strategies, enabling profiling of examinee engagement and response processes associated with successful performance. There were notable differences between the participants who correctly solved the tasks and those who failed, in terms of the time spent on solving them, as well as the data on their gazes. Moreover, this included insights into response processes which contributed to the validation of the assessment criteria of each item.
Abstract In this work, we present two studies about changes in the creative thinking of people in... more Abstract In this work, we present two studies about changes in the creative thinking of people in prison when using Scratch (a programming language designed to be fun, educational, and easy to learn). Fifteen inmates from the prison of Alava (Spain) voluntarily participated in the first study. Thirteen inmates from the same prison voluntarily participated in the second study. Each of these workshops comprised 50 h, spread over 25 days at a rate of 2 h per day. We registered their improvements during their learning process using several instruments; we also evaluated the evolution in their creative intelligence quantitatively using the CREA test (Corbalan et al., 2014), and noticed a significant score increase among participants of both studies. In light of these results, we believe that the use of Scratch can enhance the creative abilities of people in difficult situations, including prisoners.
ABSTRACT The project “Serious Games for Education–Programming Skills” presents an innovative tech... more ABSTRACT The project “Serious Games for Education–Programming Skills” presents an innovative technology that integrates serious games techniques with remote laboratories and augmented reality. The flexible and scalable technology is designed with a three layer structure: (1) the physical layer - ROBOT (hardware and communications) remotely manipulated; (2) AR and instruction interface middleware; and (3) end-user game application including game interface. This design enables multiple pedagogical objectives and context of use. In the first prototype we have developed a serious game, the third end-user layer, to develop and assess programming skills, algorithmic thinking and debugging.
2012 15th International Conference on Interactive Collaborative Learning (ICL), 2012
ABSTRACT Within the 3rd International Digital Citizenship Conference celebrated in San Sebastian-... more ABSTRACT Within the 3rd International Digital Citizenship Conference celebrated in San Sebastian-Spain in May 8th, we took part in a round table debate about the use of Smartphone in education with teenagers. There is an open discussion about the benefits or threats of using such an innovative technology. Usually the main trend is to forbid the use of mobile phones in the classroom. We bet for considering the use of any technology if it can benefit somehow our learning/teaching process. Smartphones are very powerful devices that can be used in many innovative ways. We can profit their availability, since many students have one. In this paper we analyze how different authors understand mobile learning, we review previous experiences and their results in educational institutions using mobile learning, and we reflect on the threats and opportunities of using mobile devices in education. Finally we show an approach to a new educational paradigm that is being implemented thanks to the educational revolution derived from the use of mobile devices and we provide simple guidelines for its implementation.
Enthusiasm for research on the brain and its application in education is growing among teachers. ... more Enthusiasm for research on the brain and its application in education is growing among teachers. However, a lack of sufficient knowledge, poor communication between educators and scientists, and the effective marketing of dubious educational products has led to the proliferation of numerous 'neuromyths.' As a first step toward designing effective interventions to correct these misconceptions, previous studies have explored the prevalence of neuromyths in different countries. In the present study we extend this applied research by gathering data from a new sample of Spanish teachers and by meta-analyzing all the evidence available so far. Our results show that some of the most popular neuromyths identified in previous studies are also endorsed by Spanish teachers. The meta-analytic synthesis of these data and previous research confirms that the popularity of some neuromyths is remarkably consistent across countries, although we also note peculiarities and exceptions with impo...
Until recently, most of the digital literacy frameworks have been based on assessment frameworks ... more Until recently, most of the digital literacy frameworks have been based on assessment frameworks used by commercial entities. The release of the DigComp framework has allowed the development of tailored implementations for the evaluation of digital competence. However, the majority of these digital literacy frameworks are based on self-assessments, measuring only low-order cognitive skills. This paper reports on a study to develop and validate an assessment instrument, including interactive simulations to assess citizens’ digital competence. These formats are particularly important for the evaluation of complex cognitive constructs such as digital competence. Additionally, we selected two different approaches for designing the tests based on their scope, at the competence or competence area level. Their overall and dimensional validity and reliability were analysed. We summarise the issues addressed in each phase and key points to consider in new implementations. For both approaches...
ABSTRACT Background and Context: Despite many initiatives to develop Computational Thinking (CT),... more ABSTRACT Background and Context: Despite many initiatives to develop Computational Thinking (CT), not much is known about how early programmers develop CT and how we can assess their learning. Objective: Determine if the analysis of students’ interactions with an online platform allows understanding the development of CT, how we can convert data collected into valuable insights, and the aspects that should be considered in platforms design. Method: We developed an online platform with a fine-grained log–recording system. We analysed the data collected from 1004 students (ages 8-14) to understand the difficulties they face. We explain our platform and the tools to process and filter the interaction logs. We calculate additional indicators that provide useful information about student’s behaviour. Findings: Age and gender have shown to influence on CT learning. Generating additional indicators from basic interaction data provide valuable insights. We provide a list of recommendations for developing more effective programming learning platforms.
Technical noise can compromise the precision and accuracy of the reaction times collected in psyc... more Technical noise can compromise the precision and accuracy of the reaction times collected in psychological experiments, especially in the case of Internet-based studies. Although this noise seems to have only a small impact on traditional statistical analyses, its effects on model fit to reaction-time distributions remains unexplored. Across four simulations we study the impact of technical noise on parameter recovery from data generated from an ex-Gaussian distribution and from a Ratcliff Diffusion Model. Our results suggest that the impact of noise-induced variance tends to be limited to specific parameters and conditions. Although we encourage researchers to adopt all measures to reduce the impact of noise on reaction-time experiments, we conclude that the typical amount of noise-induced variance found in these experiments does not pose substantial problems for statistical analyses based on model fitting.
Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality, 2020
This paper reports on an exploratory study with the aim to validate item response processes in di... more This paper reports on an exploratory study with the aim to validate item response processes in digital competence assessment through eye-tracking techniques. When measuring complex cognitive constructs, it is crucial to correctly design the evaluation items to trigger the intended knowledge and skills. Furthermore, to assess the validity of a test requires considering not only the content of the evaluation tasks involved in the test, but also whether examinees respond to the tasks by engaging construct-relevant response processes. The eye tracking observations helped to fill an ‘explanatory gap’ by providing data on variation in item response processes that are not captured by other sources of process data such as think aloud protocols or computer-generated log files. We proposed a set of metrics that could help test designers to validate the different item formats used in the evaluation of digital competence. The gaze data provided detailed information on test item response strategies, enabling profiling of examinee engagement and response processes associated with successful performance. There were notable differences between the participants who correctly solved the tasks and those who failed, in terms of the time spent on solving them, as well as the data on their gazes. Moreover, this included insights into response processes which contributed to the validation of the assessment criteria of each item.
Abstract In this work, we present two studies about changes in the creative thinking of people in... more Abstract In this work, we present two studies about changes in the creative thinking of people in prison when using Scratch (a programming language designed to be fun, educational, and easy to learn). Fifteen inmates from the prison of Alava (Spain) voluntarily participated in the first study. Thirteen inmates from the same prison voluntarily participated in the second study. Each of these workshops comprised 50 h, spread over 25 days at a rate of 2 h per day. We registered their improvements during their learning process using several instruments; we also evaluated the evolution in their creative intelligence quantitatively using the CREA test (Corbalan et al., 2014), and noticed a significant score increase among participants of both studies. In light of these results, we believe that the use of Scratch can enhance the creative abilities of people in difficult situations, including prisoners.
ABSTRACT The project “Serious Games for Education–Programming Skills” presents an innovative tech... more ABSTRACT The project “Serious Games for Education–Programming Skills” presents an innovative technology that integrates serious games techniques with remote laboratories and augmented reality. The flexible and scalable technology is designed with a three layer structure: (1) the physical layer - ROBOT (hardware and communications) remotely manipulated; (2) AR and instruction interface middleware; and (3) end-user game application including game interface. This design enables multiple pedagogical objectives and context of use. In the first prototype we have developed a serious game, the third end-user layer, to develop and assess programming skills, algorithmic thinking and debugging.
2012 15th International Conference on Interactive Collaborative Learning (ICL), 2012
ABSTRACT Within the 3rd International Digital Citizenship Conference celebrated in San Sebastian-... more ABSTRACT Within the 3rd International Digital Citizenship Conference celebrated in San Sebastian-Spain in May 8th, we took part in a round table debate about the use of Smartphone in education with teenagers. There is an open discussion about the benefits or threats of using such an innovative technology. Usually the main trend is to forbid the use of mobile phones in the classroom. We bet for considering the use of any technology if it can benefit somehow our learning/teaching process. Smartphones are very powerful devices that can be used in many innovative ways. We can profit their availability, since many students have one. In this paper we analyze how different authors understand mobile learning, we review previous experiences and their results in educational institutions using mobile learning, and we reflect on the threats and opportunities of using mobile devices in education. Finally we show an approach to a new educational paradigm that is being implemented thanks to the educational revolution derived from the use of mobile devices and we provide simple guidelines for its implementation.
Enthusiasm for research on the brain and its application in education is growing among teachers. ... more Enthusiasm for research on the brain and its application in education is growing among teachers. However, a lack of sufficient knowledge, poor communication between educators and scientists, and the effective marketing of dubious educational products has led to the proliferation of numerous 'neuromyths.' As a first step toward designing effective interventions to correct these misconceptions, previous studies have explored the prevalence of neuromyths in different countries. In the present study we extend this applied research by gathering data from a new sample of Spanish teachers and by meta-analyzing all the evidence available so far. Our results show that some of the most popular neuromyths identified in previous studies are also endorsed by Spanish teachers. The meta-analytic synthesis of these data and previous research confirms that the popularity of some neuromyths is remarkably consistent across countries, although we also note peculiarities and exceptions with impo...
The popularization of digital educational devices
with low barriers to entry has encouraged the ... more The popularization of digital educational devices with low barriers to entry has encouraged the development of many training activities orient ed towards the incorporation of technology in schools. In some cases, the introduction of technology has led to the development of new educational practices that support the de velopment of computational thinking. However, the supposed benefits of these approaches have not been properly assessed . Moreover, having taught over 30 workshops with the Scratch progr amming tool to teachers at different educational levels (primary, secondary, university), we found no evidence of subsequent methodological changes in schools. This study tries to understand the dissonance between the alleged success of initiatives around digital literacy and the lack of continuity in the use of user-friendly programming environments like Scratch. For this purpose, we analyzed the evolution of the grades of Scra tch programming achieved by the students of the School of Educat ion of Vitoria-Gasteiz and their engagement with Scratch. We al so interviewed teachers from different schools who have participated in Scratch workshops with us. After this stud y, we identified some of the circumstances that facilitate and hinder the development of computational thinking. Since we consider Scratch as a resource that allows the development of new methodologica l approaches in the classroom as well as the acquisition of skills related to computational thinking, we propose a framework that will help to overcome the current status.
In this work, we present two studies about changes in the creative thinking of people in prison w... more In this work, we present two studies about changes in the creative thinking of people in prison when using Scratch (a programming language designed to be fun, educational, and easy to learn). Fifteen inmates from the prison of Alava (Spain) voluntarily participated in the first study. Thirteen inmates from the same prison voluntarily participated in the second study. Each of these workshops comprised 50 h, spread over 25 days at a rate of 2 h per day. We registered their improvements during their learning process using several instruments; we also evaluated the evolution in their creative intelligence quantitatively using the CREA test (Corbalán et al., 2014), and noticed a significant score increase among participants of both studies. In light of these results, we believe that the use of Scratch can enhance the creative abilities of people in difficult situations, including prisoners.
Uploads
Papers by Pablo Garaizar
with low barriers to entry has encouraged the development of
many training activities orient
ed towards the incorporation of
technology in schools. In some
cases, the introduction of
technology has led to the development of new educational
practices that support the de
velopment of computational
thinking. However, the supposed benefits of these approaches
have not been properly assessed
. Moreover, having taught over
30 workshops with the Scratch progr
amming tool to teachers at
different educational levels (primary, secondary, university), we
found no evidence of subsequent methodological changes in
schools. This study tries to understand the dissonance between
the alleged success of initiatives around digital literacy and the
lack of continuity in the use of user-friendly programming
environments like Scratch. For this purpose, we analyzed the
evolution of the grades of Scra
tch programming achieved by the
students of the School of Educat
ion of Vitoria-Gasteiz and their
engagement with Scratch. We al
so interviewed teachers from
different schools who have participated in Scratch workshops
with us. After this stud
y, we identified some of the circumstances
that facilitate and hinder the development of computational
thinking. Since we consider Scratch as a resource that allows the
development of new methodologica
l approaches in the classroom
as well as the acquisition of skills related to computational
thinking, we propose a framework
that will help to overcome the
current status.