SMART ONLINE ASSESSMENTS FOR

TEACHING MATHEMATICS
Beth Price, Kaye Stacey, Vicki Steinle, and Eugene Gvozdenko outline the rationale,
design and potential impact of an online resource

ntroduction It is with all of this in mind that the smart test

I A Year 9 class has just had a lesson

introducing trigonometry. All seems to be
going well... or is it? The teacher asks students
system was designed. The brief was to develop
a way of making "assessment for learning"
a practical tool for teachers of students in
to do a quick online test and within minutes Years 7, 8 and 9 (Stacey, Price, Steinle, Chick
receives feedback revealing that four of the & Gvozdenko, 2009). This has since been
students cannot identify sides as adjacent, or expanded to include Years 5 and 6 as well.
opposite, in relation to given angles. She spends We set about designing "smati tests" that
a few minutes with this group conducting some could give teachers information about the
targeted teaching, and they are then able to understanding of their individual students in
catch up with the rest of the class. This group is key mathematics topics. A "smart test' is a
then asked to complete short tests on comparing "specific mathematics assessment that reveals
ratios and calculating with proportions to find out thinking" Most commonly, smart tests focus on
if they have a strong conceptual understanding fundamental understanding of essential ideas,
of these topics, which are intimately connected to like ratio and proportion. However, as illustrated
trigonometry. in the anecdote at the beginning of the paper,
some assessments also target prerequisite
Teaching mathematics is a wonderful career.
skills - identifying adjacent and opposite sides,
The subject area is fascinating and useful.
for example. Feedback to teachers consists of
Society values people with mathematical skills
a description of each student's 'developmental
and there are incentives for students to do well.
stage' and the diagnosis of many of the common
But, there are challenges. Every secondary
misconceptions. Teaching suggestions are
mathematics classroom contains students with
provided which are intended to assist with the
a range of attitudes, experiences, understanding
removal of these misconceptions and to move
and skills. Over 30 years ago, Kath Hart (1981)
students on to the next developmental stage.
reported that, in a typical class of early high
Smart tests were designed to supplement,
school students, there was a seven-year
rather than replace, the excellent assessments
range of achievement. There has been other
using rich tasks that teachers have developed
research which has confirmed this wide range.
and used over many years. Smart tests are
Teachers work with classes where some of
focused narrowly on precise topics to maximize
the high-achieving students are well ahead of
relevance to teaching, and do not give an
their classmates, whilst other students have
overall level of performance. The developmental
substantial difficulty and liftle enthusiasm to
stages reported are topic-specific, and not
improve the situation. These are the dilemmas
systematically linked to any overall system
that many of us face, and we are trying to do the
of development or government achievement
very best that we can for each of our students.
targets.
Understanding new mathematical concepts often
relies on having a good background knowledge We now have an extensive set of online tests
and so, to avoid presenting some ofthe class that has been developed to inform classroom
with tasks that they cannot do, we sometimes teachers about the understanding of students
excessively revise earlier material. We also make in their classes. These cover about 60 topics,
sure that the tasks set for most of fhe lesson with two versions of most tests. Teachers read
are straightforward enough to be tackled by all descriptions of the available smart tests, choose
students, hence limiting the more challenging one that is appropriate, and give students access
material. It can be time-consuming to keep track via a code. The computer marks the students'
of students' progress and difficult to identify responses and the response patterns are
exactly where some students are struggling as automatically analysed to diagnose the student's
well as the reasons for this. developmental stage in the topic as well as any
misconceptions or common errors.

10 Mattiematics Teaching 235 Joumal of the Association of Teachetsof Mathematics

 SMART ONLINE ASSESSMENTS FOR TEACHING MATHEMATICS

Each student's diagnosis is available to the can significantly hinder student progress.
teacher as soon as he or she accesses the Both types of tests help teachers to target their
teacher page of the smart test system. teaching to individual students' needs. Most
In the following sections we describe the tests come in matched pairs, which can be
educational and design philosophy behind used as pre-test and post-test, so teachers can
.the smart test system and illustrate this with track student progress.
an example. Further information about the An example: Readiness for operations with
project is given in Stacey, Price, Steinle, Chick, directed number
Gvozdenko, (2009). Access to the 'smart test' To illustrate the purpose, design, and
system is available on the project website. components of a typical smart test, we present
The smart test system - Educational and an item from a smart test that identifies both
design philosophy developmental stages and misconceptions
A smart test focuses on a single important involving students' understanding of integers.
concept or skill. The previous work of our team We will describe the background educational
members and many other researchers had issues and present the item, and then show
highlighted the importance of students needing the diagnosis that is provided to teachers after
to understand certain critical concepts in order students have completed the test, along with a
to make solid progress in new topics. Our aim discussion of the kinds of teaching suggestions
with the smart tests is to target some of these that are made. To view this and other smart
critical concepts, and design features; tests, follow the instructions on the smart test
home page, http://www.smartvic.com/
• short, and easy to administer, online
diagnostic assessment; The concept and an associated smart test
• immediate feedback to teachers about class item
and individual performance; Many teachers will be aware that for some
• targeted teaching suggestions that address students, operations with directed numbers
the conceptual hurdle. remain a difficulty throughout their study of
mathematics. We think this difficulty has two
These components, together, highlight the
sources. The first lies with the general concept
purpose of the smart tests as "assessment
of directed numbers. They are not numbers
for learning". Smart tests are not designed to
used to count normal objects, but numbers
give a score, but to identify what the student
which describe numerical information with
knows, to diagnose misconceptions, to provide
respect to a reference point: how many, or
teachers with information that will help them
how far above or below a baseline quantity,
meet students' needs, and to improve learning
to the left or the right of a position, or to the
outcomes. To do the diagnosis, complicated
north or the south of a location, for example.
algorithms have been designed to look for tell-
To understand operations with negative
tale patterns of systematic errors in student
numbers, initially addition and subtraction and
responses.
multiplication as repeated addition, students
There are several types of smart tests. Most need to be able to interpret operations as
assess underlying mathematical conceptual movements up and down a number line. The
understanding, and would generally be used basic ability to move up and down the number
before beginning to teach a topic that builds on line is the second aspect tested
these ideas. An example is given later Some
The intention is that the smart test "Readiness
other smart tests check students' knowledge of
for directed number" be used before secondary
facts and skills to report to the teacher whether
school teachers teach formally about
prerequisite understanding is in place prior to
operations on directed numbers. We have
teaching a new topic. An example is "Labelling
used the task of ordering positive and negative
hypotenuse, opposite, adjacent sides for
integers as a quick indicator of the underlying
trigonometry" discussed above. Although these
conceptual understanding. One such item is
do not have the conceptual emphasis of the
shown in Figure 1 (See page 12). The item
other smart tests, they were created because
requires students to drag the number cards into
we know that missing background knowledge
the correct order. The number cards snap into

July 2013 www.atm.cing.uk 11

 SMART ONLINE ASSESSMENTS FOR TEACHING MATHEMATICS

place. The precise choice ofthe numbers in this Students doing this smart test complete the
item is important, to enable us to identify relevant items, starting with that in Figure 1. By looking
misconceptions. We have used several items at the exact responses and the patterns in
to test students' ability to move up and down all their responses, it is possible to diagnose,
a number line, in preparation for operations. with reasonable confidence, what students
Students move the mercury level up and down can do, and whether or not they have any
on virtual thermometers to show temperature
misconceptions. The idea is that these tests are
increases and decreases. In these items, the
simple and quick for students to complete, and
visual labelled temperature scale is provided so
that it is concepts being targeted rather than the
that students can directly count the number of
steps that are being taken. Other items about ability to compute. Indeed, care has been taken
continuing number patterns across zero have to reduce having computational issues interfere
no such visual support. Because we know that with diagnosis of students' understandings as
students have other difficulties with non-integer they undertake the tests. In some smart tests
(decimal) negative numbers, there is a separate an electronic calculator is available. However,
test that reports on that topic. we find that the general format and students'
expectations of computer testing mitigates
Pm these temperatures in order (rom Unvea to highest.
Drag &D oT the cards below mto the grey spaces.
against items which require much 'working out':
in general, students are reluctant to go to pen-

••1 Ü mm
TtMniUTOMl
and-paper to complete an item on the screen, so
even with a calculator available, multi-step items
are not suitable. This is not a great restriction

1ÍBI 0
Figure 1. The first "Readiness for operations with
9 -tn|-2 for items targeting conceptual understanding.
However, it does highlight the need for teachers
to conduct their own assessments, including
multi-step items.
directed number (test B)" item probes understanding
of the order of negative integers and identifies some
misconceptions.

Developmental Stages and Teaching Suggestions

Readiness for operations with directed number

• Mathematical focus and overview of dtvtiopniental stages
• Detailed explanatiom of stages and common miseonceptions
• Teaching suggestions

Mathematical fltctts and overvieiv of developmental stages

Readiness for learning about operatiotis with directed ntuubers requires that students recognize the tneaning ofthe aegative sign ssd ore
able to coiTCCtly oirange signed numbers into ascending order, and also that the)' can count fonvard and back across zero using integers,
with imd without the support of a number line.

Students have sufiicient understanding of positi\-e and negati\-e integers to order them

and con solve simple problems requiring moxing up and down a number line by countitig and nñth Wsual
Stage 2
suppoil

Stage 3 and can consistently skip count in pattems that crass zeto.

Misconccpliotis, erran and gaps in understanditig

When arranging integers in ascending order, tltese students arrange in order of magnitude, ignoring the
IS negative signs.

W'hen arranging integer« in ascending order, tliese student! place the negative numbeis fírtt but re\-erse
their order.

W^en counting on or back with a number line, these students have a tendency to count the starting number
CS
as the fust ofthe steps on or back.

Figure 2. Diagnostic feedback for teachers from "Readiness for directed number" smart test.

12 Mattiematics Teaching 235 Joumal of the Assodation of Teachers of Mattiematics

 SMART ONLINE ASSESSMENTS FOR TEACHING MATHEMATICS

Because fhe assessmenf is aufomafed, smart These sfudents could move up and down the
tests require responses that are machine- number line (e.g. fhermomefer) when it was
inferprefable. The range of these response types shown, so that sfudenfs could answer correcfly
is growing rapidly (Stacey & Wiliam, 2013). by counfing along fhe scale by ones insfead
As well as single closed numerical answer and of calculating. The 29% of sfudenfs at Stage 1
traditional multiple choice, where we provide were able to place the number cards in order,
alfernafives derived from research resulfs as far but they were not able to move up and down the
as possible, fhere are drag-and-drop items of number line. They may have a generally vague
many variefies, and fhe use of sliders enables conception ofthe number line, or fhey may have
sfudenfs fo place numbers on number lines, one ofthe misconceptions below. Finally 14%
make bar graphs etc. We hope some students of fhis Year 8 sample was af Sfage 0, i.e. below
at least will find fhe ifems brighf, interesting and Sfage 1, nof yef ordering infegers correcfly, again
engaging through fheir different activity formats. possibly wifh idenfifiable misconcepfions.
The diagnosis As can be seen in Figure 2, three
As soon as students submit their responses misconceptions, or common errors, are reporfed
online, fhe resulfs are analysed elecfronically by fhis fesf. In fhe sample of 163 Year 8 sfudenfs,
using carefully designed algorithms fhaf 2% had misconcepfion IS, simply ignoring the
recognize patterns of responses corresponding negative sign in front of the numbers. This error
to different types of typical thinking. The would be more common in younger students.
diagnosis includes informafion for feachers Abouf 4% of sfudenfs placed all fhe negafive
abouf fhe different developmenfal sfages fhat are numbers on fhe left of zero, correcfly, buf ordered
revealed by the smart test. The developmental fhem as -2 < -7 < -11 (misconcepfion RN). This
stages are based on mathematics education misconcepfion has been observed elsewhere.
research results from around fhe world, wherever For example, Widjaja, Sfacey & Sfeinle (2007)
possible, (e.g. Küchemann, 1981) buf moderated reporf if with decimal numbers. A furfher 12% of
by an analysis of student resulfs on fhe smarf the sfudents were observed fo be systematically
fesfs, and by whaf we hope feachers will find 'one out' when they moved up and down the
practical and useful. Sfacey, Price and Sfeinle number line (misconception CS), including when
(2012) explains this process. The resulfs allow the number line is shown. For example, fhey say
feachers fo identify any global problems in the fhat 3 more than -2 is 0. Probably fhis is because
whole class, or fo group sfudenfs for fargefed fhey counf 3 places as -2, - 1 , 0, counfing the
feaching, and also to move sfudenfs on more starting point, insfead of - 1 , 0, 1, buf if is also
quickly if fhey show good understanding. possible fhaf some of fhem think there is a
The developmenfal sfages and misconceptions separate number -0, so that they count three
informafion for fhe "Readiness for directed places as-1,-0, 0.
number" smart test are shown in Figure 2 A compufer diagnosis from fhe smart fesf
(See page 12). Each student is allocated to sysfem gives good information, buf nof excellenf
a developmenfal stage. In some cases fhe informafion. For example, a sfudenf with good
paftern of incorrecf responses shows a particular understanding might accidenfally miss ouf some
misconception, and when fhis occurs if is also items and hence not jump fhe hurdles sef in fhe
included in fhe feedback for fhe student. In this programming for fhe early stages. We think fhis
example, the diagnosis is reported in 4 sfages is a quesfion of cosf-effecfiveness in educafional
(0 fo 3) and any of 3 misconcepfions will be design. If a feacher can gef good informafion
reporfed when fhey occur. These sfages relate on whole classes wifh a small amounf of eftort,
only to fhis fopic - fhey are nof linked fo an then they have time to investigate puzzling
exfernal framework, alfhough we believe fhey cases, for example fhrough a shorf interview
will relate well to mosf curriculum documenfs. with the student 'live'. We know that there
Anofher example is described in Price, Stacey, has been very good progress made through
Chick and Gvozdenko, (2009). teachers interviewing all their students to
At the beginning of 2013, fhis smart test was establish fheir sfages of learning see, Sfewarf,
used by 163 Year 8 students of volunfeer Wrighf & Gould, (1999). However, fhis is a very
teachers. In this sample, 49% had reached fime consuming pracfice, wifh the information
Stage 3 and so were ready for formal work on gleaned rapidly going ouf of date as children
operations. They can order infegers, and move learn. Interviewing gives more informafion to the
up and down a number line whefher it is shown teacher, but if comes at a high cost.
or not. There were 8% of students at Stage 2. To assisf feachers in cases where resulfs may

July 2013 wvvw.atm.otg.uk 13

 SMART ONLINE ASSESSMENTS FOR TEACHING MATHEMATICS

be questionable, we make student responses numbers, the student has to reason in ways
on all items available to teachers who choose to like this: "this number paftern is subtract 3 each
investigate, flag students who omit many items, time - I do this by counting down 3 on my mental
and flag students who have a high percentage number line - in the positive region I subtract
of items correct overall but get a low stage. The 3, but in the negative region it is like adding 3
latter students often have just made careless (e.g. I go from -5 to -8)". Stage 2 students will
errors on easy items, but sometimes the test benefit from skip counting across zero. A simple
reveals fundamental misunderstanding. activity is to use the constant addition or constant
Teaching strategies subtraction facility of simple calculators. The
beginning of a paftern is provided, such as
Included with the diagnostic information is a set
8, 6, 4,... Students are asked to predict the next
of ideas for teaching, generally differentiated
three numbers. Using a starting number of 8 and
by stage. In the case of "Readiness for directed
subtraction of the constant 2 students can see
numbers", the advice for Stage 0 students is
if their prediction is correct. Some 78% of the
to work on basic understanding of negative
students studied were able to do this correctly.
numbers. As noted, these are not counting
Of those who couldn't the major difficulty was
numbers, but give information about distance
with the third number, -2.
from a reference point and direction. These
students need to use directed numbers to Stage 0 students may have one of the two
describe common situations, perhaps beginning identifiable ordering misconceptions, IS
with the straightforward example of temperature. (ignore negative sign) and RN {reverse order
They should describe a profit of £100 as +100, of negatives). Like other Stage 0 students,
and a loss of £20 as -20, and order quantities they should work to understand negative
within this model. Students should also use other numbers through the common models of
common models such as distance north and integers. However, these students already have
south of a fixed point, as well as a number line. an incorrect idea in place. They need to be
The symmetry of the magnitudes of numbers convinced that there is an error in their thinking
on a number line with zero at the centre should before they realise that there is a need to do
be stressed, and also that positive numbers something different. Frequently students with
are shown on the right and negative on the misconceptions feel that they understand the
left. Students could be asked to order sets of situation and do not look for alternative ways
integers, using situations such as temperature, of thinking about it. Similarly Stage 1 students
or floors in a building above and below ground may exhibit the CS (count start) misconception,
level. They could also be asked to write all the which results in answers being one out - they
integers between, say, -6 and 3. count the (named) points on the number line,
not the intervals for measurement. For example,
Stage 1 students have a basic understanding,
a temperature change from -7 degrees to
but need to strengthen their mental image of
-2 degrees will be reported as 6 degrees
directed numbers on a number line, temperature
(-7, -6, -5, -4, -3, -2). Going back to consider
scale etc., and be sure about its details. The
the real situations is useful. It is also important
fact that 0 is not a signed number needs to be
for students to develop awareness that this
discussed with students, so that they appreciate
is an easy mistake to make and that good
that their intuitive ideas about -0 and +0 as
checking strategies are needed. For example,
two separate numbers are mistaken. Stage 1
they can test their method on a simple problem:
students would benefit from exploring situations
a temperature change from -6 degrees to -5
where they can count up and down a number
degrees is not 2 degrees.
line, such as finding the temperature after a
Feedback from teachers
change, or finding where a lift is, if it started at
floor 3 and went down 5 floors. Many teachers feel that they have learned
Stage 1 students benefit from visual support, something that has been useful for them through
for example labelling their own number line using the smart tests (Steinle & Stacey, 2012).
or thermometer scale, and may well count by Teachers complete a voluntary survey after they
ones, but by Stage 2 students are becoming access students' results from a test. It includes
able to use a mental number line and to make the multiple-choice question:
elementary links with operations and number line As a result of using this test, have you learned
movements. For example, to move on to Stage 3 and something useful for you as a teacher?
be ready to learn formal operations with negative The results show that nearly all teachers who

14 Mattiematics Teaching 235 Joumalofthe Association ofTeadiers of Mathematics

 SMART ONLINE ASSESSMENTS FOR TEACHING MATHEMATICS

responded reported that they learned something a test for homework, or a teacher might ask
useful and nearly half were very positive. just a few students to complete a test because
We had expected that many teachers would misconceptions are suspected among the
be alerted to students' inadequate preparation group. The diagnostic information and teaching
for learning a topic, and so have to start their suggestions are available almost immediately
teaching at a lower level than expected, after students complete the smart test. Results
and this was indeed the case. However, the can be easily downloaded to a spreadsheet for
opposite situation also occurred. Many teachers record keeping or more detailed analysis.
commented that they started their teaching Conclusion
of the topic at a higher level than they had The usability of fhe smart tests and the
previously planned after the diagnosis indicated opportunities they provide for quick diagnosis
that students had a befter than expected in a specific topic can be very helpful for busy
understanding. As well as whole class teaching, teachers. However the task is not yet complete.
many teachers reported using the diagnosis from In the future we would like to increase the range
the smart test system to form groups with similar of tests available to Year 5 and 6 students; to
needs. The most common strategy was to target refine the power of the diagnostic algorithms; to
specific difficulties by discussing misconceptions extend the teaching suggestions; and to make it
with groups of students who had the same possible for all tests to be used on tablets (most
diagnosis. already are) as well as desktop and notebook
Accessing the smart test system computers.
This year, 2013, access to the tests via the We know that the smart test system can be
www.smartvic.com server (Stacey, Price, Steinle, a powerful resource for diagnosing students'
Chick, Gvozdenko, 2009) is available to all thinking, easy for schools to use, informative for
teachers and their classes. Individual teachers teachers, and thus an important component of the
can obtain access to the tests relevant to their assessment for learning process, and we hope
classes, and are able to use them with their that they prove so for you too. We look forward to
students. On the website, teachers can see test working with any interested teachers and schools.
descriptions, choose three tests and try them
out with students. After this, before access to a
subsequent test is granted it will be necessary
to complete a 10-minute online feedback
questionnaire. In this fashion it is possible for a
teacher to build up a set of fests to which they
have access.
Beth Price, Kaye Stacey, Vicki Steinle, and
Each smart test is completed online and only
Eugene Gvozdenko, The University of Melbourne
takes a few minutes. As mentioned earlier, the
tests involve only minimal calculation; there is also
very little typing required. Responses are given Acknowledgements
by choosing from options in drop-down menus, This article is an updated version of Price, Stacey,
dragging-and-dropping images, typing short Steinle, Chick, Gvozdenko (2009), and includes new
numerical entries and moving sliders. examples. The project was initially supported by
Australian Research Council Linkage Projects grant
There are about sixty tests now available for key
LP0882176 in partnership with the Department of
topics in the Years 5 to 9 curricula, and most of
Education and Early Childhood Development, Victoria.
these have parallel tests available. Among the
Further funding has also come from the Catholic
topics currently addressed are number, algebra,
Education Office (Melbourne).
measurement, spatial visualisation, preparation
for Pythagoras' theorem and trigonometry,
basic understanding of decimals, fractions and
percentages, and statistics and probability. References to be found on page 26.
Teachers can choose how best to use the
smart test system in order to suit their needs
and facilities. The whole class could do a
test simultaneously if there are computers
available for everyone, students might do the
test a few at a time, students might complete

July 2013 www.atm.ang.uk 15

Copyright of Mathematics Teaching is the property of Association of Teachers of
Mathematics and its content may not be copied or emailed to multiple sites or posted to a
listserv without the copyright holder's express written permission. However, users may print,
download, or email articles for individual use.