PAPER ONLINE AND OTHER ICT-BASED ASSESSMENT TOOLS FOR PROBLEM-SOLVING SKILLS Online and other ICT-based Assessment Tools for Problem-solving Skills http://dx.doi.org/10.3991/ijet.v11i04.5339 Maria Karyotaki, Athanasios Drigas, N.C.S.R. ‘Demokritos’, Institute of Informatics and Telecommunications, Telecoms Lab - Net Media Lab Abstract—Problem-solving skills assessment starts with the imperative that the hypothesized construct of skills, is theoretically sound. Secondly, assessment refers to identifying and recording inductive and deductive types of reasoning as well as divergent and convergent thinking skills. The aforementioned procedures should be measured, independent of specific learning domains or knowledge backgrounds. Future research should be oriented towards the development of a prototype set of tasks that would embed problemsolving skills in different content and context areas and that would act as a model for implementation in online largescale assessments. Furthermore, the classroom provides an environment amenable to collaborative problem solving, in which capturing the progress of students’ both social and cognitive processes through identifying the exchange of implicit and explicit types of information, remains to be sufficiently addressed. Effective collaborative problemsolving assessment is related to well-defined, real-world problems as well as to the provision of respective scaffolds to group members for developing their cognitive and metacognitive skills. In other words, effective collaborative problem-solving assessment should bring about and measure group members’ cognitive and meta-cognitive skills, evolving from the social regulation of processes, such as goal setting, connecting information and patterns as well as testing hypotheses. Index Terms—problem-solving skills, cognitive assessment, collaborative problem-solving, metacognitive skills I. PROGRAMME FOR INTERNATIONAL STUDENT ASSESSMENT (PISA) Buchwald et al. investigated the effects of training students’ analytical problem solving competence on their cross-curricular and mathematical problem solving skills. Analytical problem solving incorporates several analytical reasoning sub-processes, such as structuring as well as integrating information and existing knowledge. High analytical reasoning ability is a common prerequisite for mathematics and cross-curricular problem solving in combination with planning, procedural knowledge and conditional knowledge as important components of both domains. Researchers designed a training experiment, encircling participants’ competency-based training in analytical problem solving and its effect on crosscurricular analytical problem solving. Moreover, the inferred competences developed and tested were, more or less, overlapping components of the mathematical competence, as well. Research results verified the common nature of cross-curricular problem solving and mathematical problem solving deriving from the importance of prior knowledge for both competences. Nevertheless, the training showed no significant main effect. However, the re- 56 sults showed that low-achieving problem-solvers made a small improvement in their mathematical problem solving competence. Future research on problem solving competence should thus, take into account motivation issues so as to induce the test reliability. Adaptive testing through open-source testing platforms accompanied by videotaping or by using computer-based assessments with log files could evoke changes in cognitive measurements, in general. II. STAND ON APPLICATIONS Greiff et al. delineate complex problem solving (CPS) in terms of its construct validity, effective measurement approach and solid research, as a whole. Researchers propose computer-based simulations of complex problems both for assessing and training CPS. According to current research, CPS measurement structure should be based on multiple complex systems (MCS), such as MicroDYN and MicroFIN, and multiple-item scales. The two aforementioned tests have their origin in the frameworks, Linear structural equations (LSE) for modelling linear relations between quantitative variables and Finite state automata (FSA) for describing relations between qualitative variables. Furthermore, the plasticity of the CPS construct and the influence of lifelong learning and training on its development as well as its relations to other facets of cognitive performance call for further theoretical and empirical work. Likewise, individual differences among participants, such as their personality or motivation, as well as participants’ encountering multiple tasks with different type of feedback, such as numerical, figural, or verbal, should be taken into consideration for improving the reliability of CPS measurement. Launching new forms of tests that can capture the complex processes occurring during problem solving corresponds to a contemporary daily life issue, the importance of which lays beyond its apparent educational purpose. Bühner et al. investigated the network of relationships between intelligence, working memory and problem solving abilities, using a dynamic computer-based simulation, called MultiFlux. This problem solving scenario is a domain-independent simulation, assessing the contribution of working memory above and beyond intelligence in predicting problem-solving performance. The study revealed that the individual differences in problem solving cannot be explained by differences in intelligence or working memory. Nevertheless, the importance of working memory for computer-simulated problem-solving scenarios was clearly shown. Schweizer et al. used computer-based scenarios, the MicroDYN test, to simulate dynamic problems for assessing students’ complex problem solving ability. This http://www.i-jet.org PAPER ONLINE AND OTHER ICT-BASED ASSESSMENT TOOLS FOR PROBLEM-SOLVING SKILLS complex problem solving measurement approach aims to examine the relation between complex problem solving ability and working memory capacity as well as it explores the concurrent and incremental validity of complex problem solving in three different subjects. Findings of the study confirm the relation between complex problem solving and working memory capacity, although complex problem solving encompasses specific processes beyond working memory capacity, such as using feedback to infer relations between variables and planning further actions. Although MicroDYN assessment and training tool was inconsistent as far as its validity across different scholastic domains, it revealed new insights into the relation between complex problem solving and school achievement. Concurrent validity was the highest for science followed by social studies. Future research should examine numeric, spatial-figural and verbal abilities as possible mediators in complex problem solving scenarios of different subjects. Greiff et al. performed an empirical study on the construct validity of the complex problem solving ability through assessing the convergence of three different complex problem solving assessment instruments, MicroDYN, the Genetics Lab and MicroFIN. Also, the research was centered on exploring the relation between complex problem solving and reasoning accompanied by the exploration of the relation between complex problem solving and students’ academic achievement. The results showed that complex problem solving assessment instruments converge sufficiently; the complex problem solving construct differed significantly from reasoning as well as complex problem solving predicted students’ natural science grades, thus indicating construct validity and convergent validity for complex problem solving. Therefore, the initiation of students’ complex problem solving training in combination with the development of alternative assessment instruments in the field of complex problem solving, are in due course. Nevertheless, the relations of different aspects of complex problem solving in terms of internal validity and in combination with other cognitive abilities such as reasoning were rather ambiguous and need further investigation. In addition, other cognitive and noncognitive constructs such as personality or motivation may be related to the incremental validity of complex problem solving, a research question, which can be addressed to, empirically, in future studies. Wüstenberg et al. investigated the predictive and incremental validity of all relevant complex problem solving facets, such as school grades in order to check construct validity of complex problem solving. Also, through multiple control rounds they aimed at the assessment of complex problem solving abilities like using and incorporating feedback in rule application. Thirdly, they looked further into the relationship between complex problem solving and reasoning. The study used computer-based assessment materials, the MicroDYN test and the Advanced Progressive Matrices (APM) so as to measure participants’ complex problem solving and reasoning ability, respectively. The findings suggest that complex problem solving is established as a valid construct that can predict cognitive performance and it is empirically separated from reasoning, as well. Moreover, even though rule identification and rule knowledge could not be empirically separated, rule knowledge and rule application were clearly distinguished. Furthermore, rule knowledge – the ability to draw conclusions in order to generate knowledge – was iJET ‒ Volume 11, Issue 4, 2016 more closely connected to school grade point average (GPA) than rule application — the ability to use knowledge in order to control a system. In future studies, a comparison of MicroDYN and tests including feedback should be conducted in order to provide more information on how closely complex problem solving and learning tests are related. Molnar et al. aimed to study the interrelations between the three constructs, inductive reasoning (IR), domain specific problem solving (DSPS), and complex problem solving (CPS), as they develop over school time. Three tests were deployed for measuring respective constructs between three age groups (5th, 7th and 11th graders). The fastest growth period for all three constructs was observed in Grade 7. Thus, this is the most effective time to enhance students’ IR, DSPS and CPS skills. Moreover, the mechanisms operated by the three correlated constructs IR, DSPS, and CPS are interrelated and specifically, DSPS and CPS strategies, knowledge acquisition and application skills, tend to become more similar over time. As a result, the role of knowledge and experience in specific content areas in order to solve problems decreases over time. Therefore, students’ complex problem solving (CPS) training during compulsory schooling may develop the other two thinking skills, implicitly, as well. Lai et al. explored the relationship among Mathematics Anxiety (MA), mathematical metacognition and Word Problem Solving (WPS) performance. In addition, they examined the role of Mathematics Anxiety and metacognitions of children both at four mathematical learning achievement levels, high achieving (HA), typical achieving (TA), low achieving (LA) and in case of them exhibiting a Mathematical Learning Difficulty (MLD). Mathematics Anxiety may both reflect the anxiety aroused in an assumed test situation and the anxiety of children’s ordinary life related to mathematics. In the present study, the concept of metacognition encompasses self-image and self-regulation. Researchers used two mathematical achievement measures, a mathematical metacognition questionnaire, a mathematics anxiety test, a standardized battery for measuring children’s mathematical abilities as well as two general intelligence tests. According to the models designed to illustrate the effects of mathematics anxiety and mathematical metacognition on word problem solving, metacognition had an indirect effect on word problem solving performance through mathematics anxiety. Moreover, the group differences in self-image and mathematics anxiety were significant and predicted mathematics achievement. More specifically, low achieving children exhibited lower levels of mathematics anxiety than children with a mathematical learning difficulty. The latter exhibited deficits in self-image, as well. Consequently, children with a mathematical learning difficulty should be supported in such a way that their self-image and specific metacognitive strategies are enhanced. Furthermore, children’s induced metacognitive processes can counter the negative or stressful perceptions in mathematical performance, although the relation between mathematics anxiety and children’s mathematical metacognition need to be further examined through dynamic, adaptable and large-scale assessment tools. Bryce et al. studied the naturally occurring metacognitive skills employed by five and seven year-olds while completing a problem-solving task, through the use of observational coding of their verbalizations and non- 57 PAPER ONLINE AND OTHER ICT-BASED ASSESSMENT TOOLS FOR PROBLEM-SOLVING SKILLS verbal behaviors. In this research, metacognitive skills are considered a construct comprising two basic categories of behaviors corresponding to respective cognitive processes, either coded as Monitoring or Control. All video coding was conducted by using Observer XT 9.0 software from Noldus. A Monitoring behavior is defined as one that serves to update the mental representation of the task, while a Control behavior asserts some action at the level of the task. The Children’s Independent Learning Development (CHILD) questionnaire was also an alternative validation method of the metacognitive skills coding scheme. In reference to the results of the study, older children showed overall higher rates of Monitoring and Control. Furthermore, both monitoring and control processes increased quantitatively and improved qualitatively in this age range. More specifically, verbally expressed planning in the younger groups was positively associated with high quality end products, possibly on account of their explicitly stated planning moves or approaches to the task. Additionally, task-specific ability or task difficulty appeared to affect children’s control processes more than their monitoring processes. Finally, the frequency rates of negative behaviors (PD) decrease as children’s taskspecific ability increases. Consequently, the design of naturalistic and challenging problem-solving tasks in correspondence to adopting sensitive and on-task assessment techniques can create meaningful and dynamic learning environments for children in an attempt to enhance their metacognitive skills. III. WEB Care et al. present the design process and its illustration in the field of collaborative problem solving skills assessment. Collaborative problem solving is one skillset requiring both social and cognitive competences. Therefore, by introducing cooperative problem solving tasks teachers are especially interested in students’ sharing of common resources and capacities, thus evoking their reasoning. Eleven collaborative problem solving tasks were designed and validated for measuring both students’ cognitive and collaborative skills, as part of the Assessment and Teaching of 21st Century Skills (ATC21S TM) project in Australia. The Laughing Clown online task assessed the collaborative problem solving construct and the five strands (Participation, Perspective Taking, Social Regulation, Task Regulation as well as students’ Learning and Knowledge Building skills). All the draft tasks were taken to research participants for paneling, piloting and trialing, along with coding, scoring and calibration activities. Moreover, such large scale assessments of collaborative problem solving or other 21st Century Skills provide the means for the concurrent development of adequate teaching strategies towards the promotion of the aforementioned thinking processes. Demiraslan Cevik et al. made an experimental study on peer assessment as a learning strategy for enhancing thirdgrade students’ problem solving skills. According to the researchers, peer assessment can scaffold students’ problem solving skills to the extent that they value and use the peer feedback. Groups with low task commitment or low involvement in the development of the problem solving outcomes, such as the reflection on the targets and solution processes, are less inclined to effective feedback use. Furthermore, peer feedback can be integrated in collaborative learning contexts as long as group members are mutu- 58 ally active contributors to the problem solving process and the feedback is perceived as much elaborative and concrete as possible. Vista et al. made a large-scale differential item functioning (DIF) analysis in order to validate an online problem solving ability test that minimizes the extraneous differential effects of students’ language background. Thus, this research insinuates that the implementation of such large scale interventions may have a positive impact on students’ problem solving and numeracy achievement as well as on the measurement of any other languageindependent skills. Furthermore, this problem solving assessment tool and the concurrent establishment of problem solving ability learning standards may lead to the advancement of the teaching techniques towards the development of students’ problem solving skills. Eventually, the differential item functioning (DIF) analysis verified of the fact that the test is appropriate for measuring students’ problem solving abilities, regardless of their English speaking background (ESB). Lubin et al. introduced the Learning Environments Approaching Professional Situations (LEAPS) model with the aim to enhance students’ problem-solving, metacognition and motivation in a non-linear, dynamic learning environment. This intervention incorporates elements from situated learning, human motivation and open learning environments literature in order to evoke students’ problem-solving ability as well as their meta-cognitive awareness regarding their perceptions of confidence for self-updating their knowledge and skills background. Above all, preservice teachers should be able to selfmonitor and self-regulate their problem-solving skills in addition to self-reflecting on their authentic, autonomous, life-long learning competence. Students worked on solving problems as members of small groups assisted by a holistic, open learning environment. The data originated from observation notes and videotaped data accompanied by interview sessions and student-generated materials. Researchers’ identification and classification of students’ problem-solving abilities showed that the ill-defined and shared-responsibility learning environment, induced students’ self-regulated and self-reflective thinking throughout the problem-solving process. Furthermore, teachers should encourage students to formulate heterogeneous prior knowledge-based groups, thus producing selfdirected, flexible in taking over new assignments and peer-supported problem-solvers. Notwithstanding, open learning environments should be ideally designed on the basis of creating and preserving as much a personalized and challenging learning experience as possible by implementing strategic scaffolds in combination to teachers’ supporting feedback. Rosenzweig et al. investigated students’ with and without Learning Disabilities (LD) metacognitive strategies, such as self-observation, self-instruction, self-monitoring, self-questioning, self-evaluation and self-control while being engaged in novel or challenging math problem solving tasks. The aforementioned set of metacognitive skills assessment was based on a think-aloud protocol, in which students were audio-taped as they processed their on-line textbook-type problems of increasing difficulty. Thus, such investigative method may be used in an attempt to verbalize and obtain information on the cognitive and metacognitive processes taking place in on-line activities. Productive verbalizations included self-monitoring, self- http://www.i-jet.org PAPER ONLINE AND OTHER ICT-BASED ASSESSMENT TOOLS FOR PROBLEM-SOLVING SKILLS instruction, self-questioning and self-correction statements directly related to solving the problem. Students with learning disabilities produced significantly more overall metacognitive verbalizations than their average achieving peers, with their non-productive verbalizations increasing as the problems became more difficult. Therefore, students with learning disabilities exhibited frustration and lack in the implementation of metacognitive strategies. Moreover, students’ problem-solving ability as well as their self-regulation and self-consciousness about their metacognitive skills are interrelated components of successful problem-solving processes, which should be taught and measured through complex, math problems. IV. REFERENCES [1] GAMES Hwang et al. integrated peer assessment into a digital game-based learning approach. More specifically, researchers introduced this game development approach into the learning process so as to improve students’ learning achievement, learning motivation and problem solving capability. Thus, they employed the peer assessment activity with the aim to enhance students’ formative qualitative assessment through engaging them in making selfreflections and sharing ideas among each other. The experimental group showed significantly improved learning achievements, learning motivation and problem solving conception in the natural science course, relative to the control group. As a whole, students thought that their experience in peer assessment was quite leveraging as it enhanced their ‘‘in-depth thinking’’, ‘‘creativity’’ and ‘‘learning motivation.’’ Future studies should take into account and analyze students’ learning patterns, cognitive processes and creative performances in addition to implementing collaborative game development strategies. V. scaffolding and immediate feedback supply towards learners. Notwithstanding, large-scale problem-solving assessment tools should be able to offer hybrid tasks, embedding the skillset in different content areas and in areas that vary in terms of how much content knowledge is necessary. Finally, the upgrade of cognitive and metacognitive skills assessment, such as in problem-solving tasks shall trigger changes in the teaching and learning techniques at schools. [2] [3] [4] [5] [6] RESEARCH HIGHLIGHTS Problem solving skills assessment verified of the construct validity of problem solving, the training of which may develop, implicitly, other thinking skills, as well. Future research on problem solving assessment should focus on adopting hybrid problem solving contexts, which counter with diverse cognitive mediators, such as verbal, numeric and spatial-figural abilities. Above all, problem solving skills assessment should be insusceptible to students’ prior knowledge background, whereas immersive, adaptable and interactive tasks set the grounds for the efficient assessment of learners’ cognitive and metacognitive sub-processes. Additionally, cooperative problem solving assessment provides a more holistic view of the problem solving construct on account of the simultaneous use of students’ social and cognitive competences while coordinating the group’s goals, task work and multiple perspectives. Thus, the benefits and challenges of designing effective cooperative problem solving tasks through training and assessment of synchronous and asynchronous social interaction features among group members are yet to come. VI. CONCLUSIONS Effective assessment of the problem-solving skills construct entails a dynamic environment, wherein the aforementioned set of cognitive and meta-cognitive skills can be captured. ICTs provide the means for the explicit design and implementation of individualized or collaborative problem-solving tasks through facilitators’ interactive iJET ‒ Volume 11, Issue 4, 2016 [7] [8] [9] [10] [11] [12] [13] [14] Buchwald, F. Fleischer, J. and Leutner, D. : “A field experimental study of analytical problem solving competence-Investigating effects of training and transfer”, Thinking Skills and Creativity, pp.1-29, (2015). Lai, Y. Zhu, X. Chen, Y. and Li, Y. : “Effects of Mathematics Anxiety and Mathematical Metacognition on Word problem Solving in Children with and without Mathematical Learning Difficulties”, PLoS ONE, vol. 10, number 6, pp. 1-19, (2015). http://dx.doi.org/10.1371/journal.pone.0130570 Bryce, D and Whitebread, D. : “The development of metacognitive skills: evidence from observational analysis of young children’s behavior during problem-solving”, Metacognition Learning, vol. 7, pp. 197-217, (2012). http://dx.doi.org/10.1007/s11409012-9091-2 Care, E. and Griffin, P. : “An approach to assessment of collaborative problem solving”, Research and Practice in Technology Enhanced Learning, vol. 9, number 3, pp. 367-388, (2014). Demiraslan Cevik, Y. Haslaman, T. and Celik, S. : “The effect of peer assessment on problem solving skills of prospective teachers supported by online learning activities”, Studies in Educational Evaluation, vol. 44, pp. 23-35, (2015). http://dx.doi.org/10.1016/ j.stueduc.2014.12.002 Vista, A. Care, E. and Griffin, P. : “Development of an online test of problem solving ability that minimizes the extraneous effects of language background and subsequent validation through a largescale DIF analysis”, International Journal of Educational Research, vol. 69, pp. 71-87, (2015). http://dx.doi.org/10.1016/ j.ijer.2014.10.003 Greiff, S. Fischer, A. Stadler, M. and Wüstenberg, S. : “Assessing complex problem- solving skills with multiple complex systems”, Thinking & Reasoning, vol. 21, number 3, pp. 356-382, (2015). http://dx.doi.org/10.1080/13546783.2014.989263 Bühner, M. Kröner, S. and Ziegler, M. : “Working memory, visual–spatial-intelligence and their relationship to problemsolving”, Intelligence, vol. 36, pp. 672-680, (2008). http://dx.doi.org/10.1016/j.intell.2008.03.008 Schweizer, F. Wüstenberg, S. and Greiff, S. : “Validity of the MicroDYN approach: Complex problem solving predicts school grades beyond working memory capacity”, Learning and Individual Differences, vol. 24, pp. 42-52, (2013). http://dx.doi.org/10.1016/j.lindif.2012.12.011 Wüstenberg, S. Greiff, S. and Funke, J. : “Complex problem solving — More than reasoning?”, Intelligence, vol. 40, pp. 1-14, (2012). http://dx.doi.org/10.1016/j.intell.2011.11.003 Greiff, S. Fischer, A. Wüstenberg, S. Sonnleitner, P. Brunner, M. and Martin, R. : “A multitrait–multimethod study of assessment instruments for complex problem solving”, Intelligence, vol. 41, pp. 579-596, (2013). http://dx.doi.org/10.1016/j.intell.2013.07.012 Molnar, G. Greiff, S. and Csapo, B. : “Inductive reasoning, domain specific and complex problem solving: Relations and development”, Thinking Skills and Creativity, vol. 9, pp. 35-45, (2013). http://dx.doi.org/10.1016/j.tsc.2013.03.002 Lubin, I. A. & Ge, X. : “Investigating the influences of a LEAPS model on preservice teachers’ problem solving, metacognition, and motivation in an educational technology course”, Education Tech Research Dev, vol. 60, pp. 239-270, (2012). http://dx.doi.org/10.1007/s11423-011-9224-3 Rosenzweig, C, Krawec, J. and Montague, M. : “Metacognitive strategy use of eigth-grade students with and without learning disabilities during mathematical problem solving: A think-aloud 59 PAPER ONLINE AND OTHER ICT-BASED ASSESSMENT TOOLS FOR PROBLEM-SOLVING SKILLS analysis”, Journal of Learning Disabilities, vol. 44, number 6, pp. 508-520, (2011). http://dx.doi.org/10.1177/0022219410378445 [15] Hwang, G.-J. Hung, C.-M. and Chen, N.-S.: “Improving learning achievements, motivations and problem-solving skills through a peer assessment-based game development approach”, Education Tech Research Dev, vol. 62, pp. 129–145, (2014). http://dx.doi.org/10.1007/s11423-013-9320-7 [16] Esteves, A. Bakker, S. Antle, A. N. May, A. Warren, J. and Oakley, I. : “The ATB Framework: Quantifying and Classifying Epistemic Strategies in Tangible Problem-Solving Tasks”, TEI 2015, January 15-19, 2015, Stanford, CA, USA. AUTHORS Athanasios Drigas is a Research Director at IITN.C.S.R. Demokritos. He is the Coordinator of Telecoms Lab and founder of Net Media Lab since 1996. From 1990 to 1999 he was the Operational manager of the Greek 60 Academic network. He has been the Coordinator of Several International Projects, in the fields of ICTs, and eservices (e-learning, e-psychology, e-government, einclusion, e-culture etc). He has published more than 270 articles, 7 books, 25 educational CD-ROMs and several patents. He has been a member of several International committees for the design and coordination of Network and ICT activities and of international conferences and journals. (e-mail: dr@iit.demokritos.gr). M. Karyotaki is with N.C.S.R. ‘Demokritos’, Institute of Informatics and Telecommunications, Telecoms Lab Net Media Lab, Agia Paraskevi, 153 10, Athens, Greece (e-mail: karyotakimaria@gmail.com) Submitted 07 December 2015. Published as rsubmitted by the authors 25 January 2016. http://www.i-jet.org