PAPER
ONLINE AND OTHER ICT-BASED ASSESSMENT TOOLS FOR PROBLEM-SOLVING SKILLS
Online and other ICT-based Assessment Tools
for Problem-solving Skills
http://dx.doi.org/10.3991/ijet.v11i04.5339
Maria Karyotaki, Athanasios Drigas,
N.C.S.R. ‘Demokritos’, Institute of Informatics and Telecommunications, Telecoms Lab - Net Media Lab
Abstract—Problem-solving skills assessment starts with the
imperative that the hypothesized construct of skills, is theoretically sound. Secondly, assessment refers to identifying
and recording inductive and deductive types of reasoning as
well as divergent and convergent thinking skills. The aforementioned procedures should be measured, independent of
specific learning domains or knowledge backgrounds. Future research should be oriented towards the development
of a prototype set of tasks that would embed problemsolving skills in different content and context areas and that
would act as a model for implementation in online largescale assessments. Furthermore, the classroom provides an
environment amenable to collaborative problem solving, in
which capturing the progress of students’ both social and
cognitive processes through identifying the exchange of
implicit and explicit types of information, remains to be
sufficiently addressed. Effective collaborative problemsolving assessment is related to well-defined, real-world
problems as well as to the provision of respective scaffolds
to group members for developing their cognitive and metacognitive skills. In other words, effective collaborative problem-solving assessment should bring about and measure
group members’ cognitive and meta-cognitive skills, evolving from the social regulation of processes, such as goal
setting, connecting information and patterns as well as testing hypotheses.
Index Terms—problem-solving skills, cognitive assessment,
collaborative problem-solving, metacognitive skills
I.
PROGRAMME FOR INTERNATIONAL STUDENT
ASSESSMENT (PISA)
Buchwald et al. investigated the effects of training students’ analytical problem solving competence on their
cross-curricular and mathematical problem solving skills.
Analytical problem solving incorporates several analytical
reasoning sub-processes, such as structuring as well as
integrating information and existing knowledge. High
analytical reasoning ability is a common prerequisite for
mathematics and cross-curricular problem solving in
combination with planning, procedural knowledge and
conditional knowledge as important components of both
domains. Researchers designed a training experiment,
encircling participants’ competency-based training in
analytical problem solving and its effect on crosscurricular analytical problem solving. Moreover, the inferred competences developed and tested were, more or
less, overlapping components of the mathematical competence, as well. Research results verified the common nature of cross-curricular problem solving and mathematical
problem solving deriving from the importance of prior
knowledge for both competences. Nevertheless, the training showed no significant main effect. However, the re-
56
sults showed that low-achieving problem-solvers made a
small improvement in their mathematical problem solving
competence. Future research on problem solving competence should thus, take into account motivation issues so
as to induce the test reliability. Adaptive testing through
open-source testing platforms accompanied by videotaping or by using computer-based assessments with log
files could evoke changes in cognitive measurements, in
general.
II.
STAND ON APPLICATIONS
Greiff et al. delineate complex problem solving (CPS)
in terms of its construct validity, effective measurement
approach and solid research, as a whole. Researchers
propose computer-based simulations of complex problems
both for assessing and training CPS. According to current
research, CPS measurement structure should be based on
multiple complex systems (MCS), such as MicroDYN and
MicroFIN, and multiple-item scales. The two aforementioned tests have their origin in the frameworks, Linear
structural equations (LSE) for modelling linear relations
between quantitative variables and Finite state automata
(FSA) for describing relations between qualitative variables. Furthermore, the plasticity of the CPS construct and
the influence of lifelong learning and training on its development as well as its relations to other facets of cognitive performance call for further theoretical and empirical
work. Likewise, individual differences among participants, such as their personality or motivation, as well as
participants’ encountering multiple tasks with different
type of feedback, such as numerical, figural, or verbal,
should be taken into consideration for improving the reliability of CPS measurement. Launching new forms of
tests that can capture the complex processes occurring
during problem solving corresponds to a contemporary
daily life issue, the importance of which lays beyond its
apparent educational purpose.
Bühner et al. investigated the network of relationships
between intelligence, working memory and problem solving abilities, using a dynamic computer-based simulation,
called MultiFlux. This problem solving scenario is a domain-independent simulation, assessing the contribution
of working memory above and beyond intelligence in
predicting problem-solving performance. The study revealed that the individual differences in problem solving
cannot be explained by differences in intelligence or
working memory. Nevertheless, the importance of working memory for computer-simulated problem-solving
scenarios was clearly shown.
Schweizer et al. used computer-based scenarios, the
MicroDYN test, to simulate dynamic problems for assessing students’ complex problem solving ability. This
http://www.i-jet.org
PAPER
ONLINE AND OTHER ICT-BASED ASSESSMENT TOOLS FOR PROBLEM-SOLVING SKILLS
complex problem solving measurement approach aims to
examine the relation between complex problem solving
ability and working memory capacity as well as it explores the concurrent and incremental validity of complex
problem solving in three different subjects. Findings of the
study confirm the relation between complex problem
solving and working memory capacity, although complex
problem solving encompasses specific processes beyond
working memory capacity, such as using feedback to infer
relations between variables and planning further actions.
Although MicroDYN assessment and training tool was
inconsistent as far as its validity across different scholastic
domains, it revealed new insights into the relation between
complex problem solving and school achievement. Concurrent validity was the highest for science followed by
social studies. Future research should examine numeric,
spatial-figural and verbal abilities as possible mediators in
complex problem solving scenarios of different subjects.
Greiff et al. performed an empirical study on the construct validity of the complex problem solving ability
through assessing the convergence of three different complex problem solving assessment instruments, MicroDYN,
the Genetics Lab and MicroFIN. Also, the research was
centered on exploring the relation between complex problem solving and reasoning accompanied by the exploration of the relation between complex problem solving and
students’ academic achievement. The results showed that
complex problem solving assessment instruments converge sufficiently; the complex problem solving construct
differed significantly from reasoning as well as complex
problem solving predicted students’ natural science
grades, thus indicating construct validity and convergent
validity for complex problem solving. Therefore, the initiation of students’ complex problem solving training in
combination with the development of alternative assessment instruments in the field of complex problem solving,
are in due course. Nevertheless, the relations of different
aspects of complex problem solving in terms of internal
validity and in combination with other cognitive abilities
such as reasoning were rather ambiguous and need further
investigation. In addition, other cognitive and noncognitive constructs such as personality or motivation may
be related to the incremental validity of complex problem
solving, a research question, which can be addressed to,
empirically, in future studies.
Wüstenberg et al. investigated the predictive and incremental validity of all relevant complex problem solving
facets, such as school grades in order to check construct
validity of complex problem solving. Also, through multiple control rounds they aimed at the assessment of complex problem solving abilities like using and incorporating
feedback in rule application. Thirdly, they looked further
into the relationship between complex problem solving
and reasoning. The study used computer-based assessment
materials, the MicroDYN test and the Advanced Progressive Matrices (APM) so as to measure participants’ complex problem solving and reasoning ability, respectively.
The findings suggest that complex problem solving is
established as a valid construct that can predict cognitive
performance and it is empirically separated from reasoning, as well. Moreover, even though rule identification and
rule knowledge could not be empirically separated, rule
knowledge and rule application were clearly distinguished. Furthermore, rule knowledge – the ability to
draw conclusions in order to generate knowledge – was
iJET ‒ Volume 11, Issue 4, 2016
more closely connected to school grade point average
(GPA) than rule application — the ability to use
knowledge in order to control a system. In future studies,
a comparison of MicroDYN and tests including feedback
should be conducted in order to provide more information
on how closely complex problem solving and learning
tests are related.
Molnar et al. aimed to study the interrelations between
the three constructs, inductive reasoning (IR), domain
specific problem solving (DSPS), and complex problem
solving (CPS), as they develop over school time. Three
tests were deployed for measuring respective constructs
between three age groups (5th, 7th and 11th graders). The
fastest growth period for all three constructs was observed
in Grade 7. Thus, this is the most effective time to enhance students’ IR, DSPS and CPS skills. Moreover, the
mechanisms operated by the three correlated constructs
IR, DSPS, and CPS are interrelated and specifically,
DSPS and CPS strategies, knowledge acquisition and
application skills, tend to become more similar over time.
As a result, the role of knowledge and experience in specific content areas in order to solve problems decreases
over time. Therefore, students’ complex problem solving
(CPS) training during compulsory schooling may develop
the other two thinking skills, implicitly, as well.
Lai et al. explored the relationship among Mathematics
Anxiety (MA), mathematical metacognition and Word
Problem Solving (WPS) performance. In addition, they
examined the role of Mathematics Anxiety and metacognitions of children both at four mathematical learning
achievement levels, high achieving (HA), typical achieving (TA), low achieving (LA) and in case of them exhibiting a Mathematical Learning Difficulty (MLD). Mathematics Anxiety may both reflect the anxiety aroused in an
assumed test situation and the anxiety of children’s ordinary life related to mathematics. In the present study, the
concept of metacognition encompasses self-image and
self-regulation. Researchers used two mathematical
achievement measures, a mathematical metacognition
questionnaire, a mathematics anxiety test, a standardized
battery for measuring children’s mathematical abilities as
well as two general intelligence tests. According to the
models designed to illustrate the effects of mathematics
anxiety and mathematical metacognition on word problem
solving, metacognition had an indirect effect on word
problem solving performance through mathematics anxiety. Moreover, the group differences in self-image and
mathematics anxiety were significant and predicted mathematics achievement. More specifically, low achieving
children exhibited lower levels of mathematics anxiety
than children with a mathematical learning difficulty. The
latter exhibited deficits in self-image, as well. Consequently, children with a mathematical learning difficulty
should be supported in such a way that their self-image
and specific metacognitive strategies are enhanced. Furthermore, children’s induced metacognitive processes can
counter the negative or stressful perceptions in mathematical performance, although the relation between mathematics anxiety and children’s mathematical metacognition
need to be further examined through dynamic, adaptable
and large-scale assessment tools.
Bryce et al. studied the naturally occurring metacognitive skills employed by five and seven year-olds while
completing a problem-solving task, through the use of
observational coding of their verbalizations and non-
57
PAPER
ONLINE AND OTHER ICT-BASED ASSESSMENT TOOLS FOR PROBLEM-SOLVING SKILLS
verbal behaviors. In this research, metacognitive skills are
considered a construct comprising two basic categories of
behaviors corresponding to respective cognitive processes,
either coded as Monitoring or Control. All video coding
was conducted by using Observer XT 9.0 software from
Noldus. A Monitoring behavior is defined as one that
serves to update the mental representation of the task,
while a Control behavior asserts some action at the level
of the task. The Children’s Independent Learning Development (CHILD) questionnaire was also an alternative
validation method of the metacognitive skills coding
scheme. In reference to the results of the study, older
children showed overall higher rates of Monitoring and
Control. Furthermore, both monitoring and control processes increased quantitatively and improved qualitatively
in this age range. More specifically, verbally expressed
planning in the younger groups was positively associated
with high quality end products, possibly on account of
their explicitly stated planning moves or approaches to the
task. Additionally, task-specific ability or task difficulty
appeared to affect children’s control processes more than
their monitoring processes. Finally, the frequency rates of
negative behaviors (PD) decrease as children’s taskspecific ability increases. Consequently, the design of
naturalistic and challenging problem-solving tasks in
correspondence to adopting sensitive and on-task assessment techniques can create meaningful and dynamic
learning environments for children in an attempt to enhance their metacognitive skills.
III.
WEB
Care et al. present the design process and its illustration
in the field of collaborative problem solving skills assessment. Collaborative problem solving is one skillset requiring both social and cognitive competences. Therefore, by
introducing cooperative problem solving tasks teachers
are especially interested in students’ sharing of common
resources and capacities, thus evoking their reasoning.
Eleven collaborative problem solving tasks were designed
and validated for measuring both students’ cognitive and
collaborative skills, as part of the Assessment and Teaching of 21st Century Skills (ATC21S TM) project in Australia. The Laughing Clown online task assessed the collaborative problem solving construct and the five strands (Participation, Perspective Taking, Social Regulation, Task
Regulation as well as students’ Learning and Knowledge
Building skills). All the draft tasks were taken to research
participants for paneling, piloting and trialing, along with
coding, scoring and calibration activities. Moreover, such
large scale assessments of collaborative problem solving
or other 21st Century Skills provide the means for the
concurrent development of adequate teaching strategies
towards the promotion of the aforementioned thinking
processes.
Demiraslan Cevik et al. made an experimental study on
peer assessment as a learning strategy for enhancing thirdgrade students’ problem solving skills. According to the
researchers, peer assessment can scaffold students’ problem solving skills to the extent that they value and use the
peer feedback. Groups with low task commitment or low
involvement in the development of the problem solving
outcomes, such as the reflection on the targets and solution processes, are less inclined to effective feedback use.
Furthermore, peer feedback can be integrated in collaborative learning contexts as long as group members are mutu-
58
ally active contributors to the problem solving process and
the feedback is perceived as much elaborative and concrete as possible.
Vista et al. made a large-scale differential item functioning (DIF) analysis in order to validate an online problem solving ability test that minimizes the extraneous
differential effects of students’ language background.
Thus, this research insinuates that the implementation of
such large scale interventions may have a positive impact
on students’ problem solving and numeracy achievement
as well as on the measurement of any other languageindependent skills. Furthermore, this problem solving
assessment tool and the concurrent establishment of problem solving ability learning standards may lead to the
advancement of the teaching techniques towards the development of students’ problem solving skills. Eventually,
the differential item functioning (DIF) analysis verified of
the fact that the test is appropriate for measuring students’
problem solving abilities, regardless of their English
speaking background (ESB).
Lubin et al. introduced the Learning Environments Approaching Professional Situations (LEAPS) model with
the aim to enhance students’ problem-solving, metacognition and motivation in a non-linear, dynamic learning
environment. This intervention incorporates elements
from situated learning, human motivation and open learning environments literature in order to evoke students’
problem-solving ability as well as their meta-cognitive
awareness regarding their perceptions of confidence for
self-updating their knowledge and skills background.
Above all, preservice teachers should be able to selfmonitor and self-regulate their problem-solving skills in
addition to self-reflecting on their authentic, autonomous,
life-long learning competence. Students worked on solving problems as members of small groups assisted by a
holistic, open learning environment. The data originated
from observation notes and videotaped data accompanied
by interview sessions and student-generated materials.
Researchers’ identification and classification of students’
problem-solving abilities showed that the ill-defined and
shared-responsibility learning environment, induced students’ self-regulated and self-reflective thinking throughout the problem-solving process. Furthermore, teachers
should encourage students to formulate heterogeneous
prior knowledge-based groups, thus producing selfdirected, flexible in taking over new assignments and
peer-supported problem-solvers. Notwithstanding, open
learning environments should be ideally designed on the
basis of creating and preserving as much a personalized
and challenging learning experience as possible by implementing strategic scaffolds in combination to teachers’
supporting feedback.
Rosenzweig et al. investigated students’ with and without Learning Disabilities (LD) metacognitive strategies,
such as self-observation, self-instruction, self-monitoring,
self-questioning, self-evaluation and self-control while
being engaged in novel or challenging math problem solving tasks. The aforementioned set of metacognitive skills
assessment was based on a think-aloud protocol, in which
students were audio-taped as they processed their on-line
textbook-type problems of increasing difficulty. Thus,
such investigative method may be used in an attempt to
verbalize and obtain information on the cognitive and
metacognitive processes taking place in on-line activities.
Productive verbalizations included self-monitoring, self-
http://www.i-jet.org
PAPER
ONLINE AND OTHER ICT-BASED ASSESSMENT TOOLS FOR PROBLEM-SOLVING SKILLS
instruction, self-questioning and self-correction statements
directly related to solving the problem. Students with
learning disabilities produced significantly more overall
metacognitive verbalizations than their average achieving
peers, with their non-productive verbalizations increasing
as the problems became more difficult. Therefore, students with learning disabilities exhibited frustration and
lack in the implementation of metacognitive strategies.
Moreover, students’ problem-solving ability as well as
their self-regulation and self-consciousness about their
metacognitive skills are interrelated components of successful problem-solving processes, which should be taught
and measured through complex, math problems.
IV.
REFERENCES
[1]
GAMES
Hwang et al. integrated peer assessment into a digital
game-based learning approach. More specifically, researchers introduced this game development approach into
the learning process so as to improve students’ learning
achievement, learning motivation and problem solving
capability. Thus, they employed the peer assessment activity with the aim to enhance students’ formative qualitative
assessment through engaging them in making selfreflections and sharing ideas among each other. The experimental group showed significantly improved learning
achievements, learning motivation and problem solving
conception in the natural science course, relative to the
control group. As a whole, students thought that their
experience in peer assessment was quite leveraging as it
enhanced their ‘‘in-depth thinking’’, ‘‘creativity’’ and
‘‘learning motivation.’’ Future studies should take into
account and analyze students’ learning patterns, cognitive
processes and creative performances in addition to implementing collaborative game development strategies.
V.
scaffolding and immediate feedback supply towards
learners. Notwithstanding, large-scale problem-solving
assessment tools should be able to offer hybrid tasks,
embedding the skillset in different content areas and in
areas that vary in terms of how much content knowledge
is necessary. Finally, the upgrade of cognitive and metacognitive skills assessment, such as in problem-solving
tasks shall trigger changes in the teaching and learning
techniques at schools.
[2]
[3]
[4]
[5]
[6]
RESEARCH HIGHLIGHTS
Problem solving skills assessment verified of the construct validity of problem solving, the training of which
may develop, implicitly, other thinking skills, as well.
Future research on problem solving assessment should
focus on adopting hybrid problem solving contexts, which
counter with diverse cognitive mediators, such as verbal,
numeric and spatial-figural abilities. Above all, problem
solving skills assessment should be insusceptible to students’ prior knowledge background, whereas immersive,
adaptable and interactive tasks set the grounds for the
efficient assessment of learners’ cognitive and metacognitive sub-processes. Additionally, cooperative problem solving assessment provides a more holistic view of
the problem solving construct on account of the simultaneous use of students’ social and cognitive competences
while coordinating the group’s goals, task work and multiple perspectives. Thus, the benefits and challenges of
designing effective cooperative problem solving tasks
through training and assessment of synchronous and asynchronous social interaction features among group members are yet to come.
VI.
CONCLUSIONS
Effective assessment of the problem-solving skills construct entails a dynamic environment, wherein the aforementioned set of cognitive and meta-cognitive skills can
be captured. ICTs provide the means for the explicit design and implementation of individualized or collaborative
problem-solving tasks through facilitators’ interactive
iJET ‒ Volume 11, Issue 4, 2016
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
Buchwald, F. Fleischer, J. and Leutner, D. : “A field experimental
study of analytical problem solving competence-Investigating effects of training and transfer”, Thinking Skills and Creativity,
pp.1-29, (2015).
Lai, Y. Zhu, X. Chen, Y. and Li, Y. : “Effects of Mathematics
Anxiety and Mathematical Metacognition on Word problem Solving in Children with and without Mathematical Learning Difficulties”, PLoS ONE, vol. 10, number 6, pp. 1-19, (2015).
http://dx.doi.org/10.1371/journal.pone.0130570
Bryce, D and Whitebread, D. : “The development of metacognitive skills: evidence from observational analysis of young children’s behavior during problem-solving”, Metacognition Learning, vol. 7, pp. 197-217, (2012). http://dx.doi.org/10.1007/s11409012-9091-2
Care, E. and Griffin, P. : “An approach to assessment of collaborative problem solving”, Research and Practice in Technology Enhanced Learning, vol. 9, number 3, pp. 367-388, (2014).
Demiraslan Cevik, Y. Haslaman, T. and Celik, S. : “The effect of
peer assessment on problem solving skills of prospective teachers
supported by online learning activities”, Studies in Educational
Evaluation, vol. 44, pp. 23-35, (2015). http://dx.doi.org/10.1016/
j.stueduc.2014.12.002
Vista, A. Care, E. and Griffin, P. : “Development of an online test
of problem solving ability that minimizes the extraneous effects of
language background and subsequent validation through a largescale DIF analysis”, International Journal of Educational Research, vol. 69, pp. 71-87, (2015). http://dx.doi.org/10.1016/
j.ijer.2014.10.003
Greiff, S. Fischer, A. Stadler, M. and Wüstenberg, S. : “Assessing
complex problem- solving skills with multiple complex systems”,
Thinking & Reasoning, vol. 21, number 3, pp. 356-382, (2015).
http://dx.doi.org/10.1080/13546783.2014.989263
Bühner, M. Kröner, S. and Ziegler, M. : “Working memory,
visual–spatial-intelligence and their relationship to problemsolving”, Intelligence, vol. 36, pp. 672-680, (2008).
http://dx.doi.org/10.1016/j.intell.2008.03.008
Schweizer, F. Wüstenberg, S. and Greiff, S. : “Validity of the
MicroDYN approach: Complex problem solving predicts school
grades beyond working memory capacity”, Learning and Individual
Differences,
vol.
24,
pp.
42-52,
(2013).
http://dx.doi.org/10.1016/j.lindif.2012.12.011
Wüstenberg, S. Greiff, S. and Funke, J. : “Complex problem
solving — More than reasoning?”, Intelligence, vol. 40, pp. 1-14,
(2012). http://dx.doi.org/10.1016/j.intell.2011.11.003
Greiff, S. Fischer, A. Wüstenberg, S. Sonnleitner, P. Brunner, M.
and Martin, R. : “A multitrait–multimethod study of assessment
instruments for complex problem solving”, Intelligence, vol. 41,
pp. 579-596, (2013). http://dx.doi.org/10.1016/j.intell.2013.07.012
Molnar, G. Greiff, S. and Csapo, B. : “Inductive reasoning, domain specific and complex problem solving: Relations and development”, Thinking Skills and Creativity, vol. 9, pp. 35-45, (2013).
http://dx.doi.org/10.1016/j.tsc.2013.03.002
Lubin, I. A. & Ge, X. : “Investigating the influences of a LEAPS
model on preservice teachers’ problem solving, metacognition,
and motivation in an educational technology course”, Education
Tech Research Dev, vol. 60, pp. 239-270, (2012).
http://dx.doi.org/10.1007/s11423-011-9224-3
Rosenzweig, C, Krawec, J. and Montague, M. : “Metacognitive
strategy use of eigth-grade students with and without learning disabilities during mathematical problem solving: A think-aloud
59
PAPER
ONLINE AND OTHER ICT-BASED ASSESSMENT TOOLS FOR PROBLEM-SOLVING SKILLS
analysis”, Journal of Learning Disabilities, vol. 44, number 6, pp.
508-520, (2011). http://dx.doi.org/10.1177/0022219410378445
[15] Hwang, G.-J. Hung, C.-M. and Chen, N.-S.: “Improving learning
achievements, motivations and problem-solving skills through a
peer assessment-based game development approach”, Education
Tech Research Dev, vol. 62, pp. 129–145, (2014).
http://dx.doi.org/10.1007/s11423-013-9320-7
[16] Esteves, A. Bakker, S. Antle, A. N. May, A. Warren, J. and Oakley, I. : “The ATB Framework: Quantifying and Classifying Epistemic Strategies in Tangible Problem-Solving Tasks”, TEI 2015,
January 15-19, 2015, Stanford, CA, USA.
AUTHORS
Athanasios Drigas is a Research Director at IITN.C.S.R. Demokritos. He is the Coordinator of Telecoms
Lab and founder of Net Media Lab since 1996. From 1990
to 1999 he was the Operational manager of the Greek
60
Academic network. He has been the Coordinator of Several International Projects, in the fields of ICTs, and eservices (e-learning, e-psychology, e-government, einclusion, e-culture etc). He has published more than 270
articles, 7 books, 25 educational CD-ROMs and several
patents. He has been a member of several International
committees for the design and coordination of Network
and ICT activities and of international conferences and
journals. (e-mail: dr@iit.demokritos.gr).
M. Karyotaki is with N.C.S.R. ‘Demokritos’, Institute
of Informatics and Telecommunications, Telecoms Lab Net Media Lab, Agia Paraskevi, 153 10, Athens, Greece
(e-mail: karyotakimaria@gmail.com)
Submitted 07 December 2015. Published as rsubmitted by the authors
25 January 2016.
http://www.i-jet.org