Topics in Cognitive Science 9 (2017) 943–969
Copyright © 2017 Cognitive Science Society, Inc. All rights reserved.
ISSN:1756-8757 print / 1756-8765 online
DOI: 10.1111/tops.12291
This article is part of the topic “Sketching and Cognition,” Kenneth Forbus and Shaaron
Ainsworth (Topic Editors). For a full listing of topic papers, see: http://onlinelibrary.wiley.
com/doi/10.1111/tops.2017.9.issue-4/issuetoc
Promoting Sketching in Introductory Geoscience Courses:
CogSketch Geoscience Worksheets
Bridget Garnier,a Maria Chang,b Carol Ormand,a,c Bryan Matlen,d
Basil Tikoff,a Thomas F. Shipleye
a
Department of Geoscience, University of Wisconsin-Madison
b
T.J. Watson Research Center, IBM
c
Science and Education Resource Center, Carleton College
d
STEM Program, WestEd
e
Department of Psychology, Temple University
Received 27 April 2015; received in revised form 24 March 2016; accepted 26 September 2016
Abstract
Research from cognitive science and geoscience education has shown that sketching can improve
spatial thinking skills and facilitate solving spatially complex problems. Yet sketching is rarely implemented in introductory geosciences courses, due to time needed to grade sketches and lack of materials that incorporate cognitive science research. Here, we report a design-centered, collaborative effort,
between geoscientists, cognitive scientists, and artificial intelligence (AI) researchers, to characterize
spatial learning challenges in geoscience and to design sketch activities that use a sketch-understanding
program, CogSketch. We developed 26 CogSketch worksheets that use cognitive science–based principles to scaffold problem solving of spatially complex geoscience problems and report observations of
an implementation in an introductory geoscience course where students used CogSketch or humangraded paper worksheets. Overall, this research highlights the principles of interdisciplinary design
between cognitive scientists, geoscientists, and AI researchers that can inform the collaborative design
process for others aiming to develop effective educational materials.
Keywords: Geoscience; Sketching feedback; Science education; Spatial thinking; Sketching tutor;
Introductory courses; Science of design
Correspondence should be sent to Bridget Garnier, Department of Geoscience, University of WisconsinMadison, 1215 West Dayton St., Room 273, Madison, WI 53706. E-mail: bridget@wisc.edu
944
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
1. Introduction
Geoscience is a spatially demanding field as many disciplinary tasks involve complex spatial relations. Spatial reasoning challenges include inferring interior 3-D structures from surface patterns, inferring rigid and non-rigid changes over time, relating
features at different spatial scales, and reasoning about past events from current spatial
patterns (Kastens & Ishikawa, 2006; Manduca & Kastens, 2012). Therefore, beginning
geoscience students must develop the skills in their undergraduate careers to handle
these spatial reasoning challenges and educators are expected to facilitate this development despite a wide variability in their students’ spatial abilities (Ormand et al.,
2014).
Sketching has been shown to be an effective way to improve spatial thinking skills
(both general spatial thinking skills, as well as those needed for domain specific tasks)
and support the formation and communication of complex spatial concepts in STEM
(e.g., Ainsworth, Prain, & Tytler, 2011; Gagnier et al., 2012, 2016; Gobert, 2005;
Gobert & Clement, 1999; Johnson & Reynolds, 2005; Liben & Titus, 2012; Rapp,
Culpeppers, Kirkby, & Morin, 2007; Resnick, Atit, & Shipley, 2012; Resnick, Shipley,
Newcombe, Massey, & Wills, 2012; Sorby, 2009; Titus & Horsman, 2009). In the
geosciences, sketching is one of the most widely used tools among experts and has
been found to support the teaching of difficult concepts (e.g., Johnson & Reynolds,
2005). Thus, students in introductory courses have much to gain from sketching, both
in the form of increased mastery of disciplinary geoscience content and development
of skills that support more general spatial thinking (Gagnier et al., 2012; Kali &
Orion, 1996; Ormand et al., 2014; Titus & Horsman, 2009). However, we have noted
that sketching is rarely implemented in introductory courses. Below we summarize the
results of a small survey that suggests instructors do believe that sketching would support learning, but the primary obstacle to including sketching in introductory courses
appears to be the time it takes for instructors to grade and provide feedback on
sketches.
Providing opportunities for students to learn from sketching without excessive
demands on an instructor’s time requires designing a new approach—one that draws on
research in cognitive science on artificial intelligence (AI), spatial learning, and sketching. However, it is unclear how to best use cognitive science theory to specifically
address the practical problems faced by geoscience educators—a common problem in
educational research (Burkhardt & Schoenfeld, 2003; Magidson, 2002). Drawing from
recent work on design-based research, we report the development of a problem-centered
interdisciplinary research project that places the educator and her/his classroom at the
center of a rapid prototyping process with co-development of materials and concepts
among educators and researchers (Bryk, 2009; Bryk & Gomez, 2008; Bryk, Gomez, Grunow, & LeMahieu, 2015; Morris & Hiebert, 2011). This is a Design, Educational Engineering, and Development (D-EE-D) approach that combines cognitive science research
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
945
and sketching to support spatial skills and disciplinary concept development in geoscience students.
This paper reports on a collaborative effort to characterize the spatial learning
problem and design sketch activities that use a sketch-understanding program with a
built-in tutor (CogSketch). This work is in the early, yet research-and time-intensive
step, in the D-EE-D process. Our goals were to identify the challenges to using
sketching in a classroom, review the interdisciplinary research needed to address the
problem, develop a process for geoscience educators to generate sketch activities, and
test the design by implementing sketching in a large introductory geoscience course.
Thus, this initial design work should serve as a conceptual framework for other teams
to work on developing sketch-based support for learning in other spatially complex
fields.
2. Challenges of implementing sketching in the classroom
Although sketching helps students increase problem solving and spatial thinking skills,
no pedagogical materials based on cognitive science research, such as sketching-based
workbooks, are currently available from textbook publishers for introductory-level geoscience students. To better understand why this is the case, despite the long history of
sketching in the discipline, we conducted a survey of introductory geoscience instructors
to document the use of sketching and attitudes towards sketching in undergraduate geoscience classrooms. Seventy-two introductory geoscience instructors (49 from 2-year
institutions and 23 from 4-year institutions) from undergraduate institutions across the
United States were surveyed. The survey likely self-selected geoscience educators who
are specifically interested in geoscience education, and it is possible such instructors were
aware of the value of sketching as an effective learning activity. Thus, the results may
well overestimate the use of sketching in U.S. geosciences classes. Instructors were asked
a combination of Likert (using a scale from 1 to 5) and open-ended questions about
whether and how they implemented sketching in their classrooms, how feedback was
provided, time spent grading sketches, and their beliefs about the importance of
sketching.
We found that while over 80% of instructors believed sketching is important for
understanding geoscience concepts, 52% assign three or fewer sketching assignments
during a semester in their course. Of the instructors in the survey, 77% spent up to
2 min per sketch on grading, and the remaining instructors spent more than 3 min grading each sketch. At those rates, grading sketches for large-enrollment courses requires a
minimum of several hours per sketch assignment, even for the fastest graders. We
found that 90% of graded sketches are assigned for exams, quizzes, or lab assignments.
Thus, results suggest that while introductory geoscience instructors view sketching as
important, it can be a time-consuming activity for instructors and is often limited to
summative assessment.
946
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
3. Drawing from cognitive psychology and computer science
3.1. Spatial thinking skills in the geosciences
To constrain the design problem, we focused on developing cognitive science–based
support for four classes of spatial reasoning skills common to geoscience sketches. The
selection for these four classes for prioritization was based on (a) previous work on the
range of spatial skills applied to geoscience (Kastens & Ishikawa, 2006; Shipley, Tikoff,
Ormand, & Manduca, 2013); (b) identification of the spatial skills necessary for critical
disciplinary content included in an introductory geology course, guided by research group
members with expertise in geoscience and teaching introductory geoscience; and (c) spatial reasoning problems with some prior basic research to characterize the learning challenges, including the known range of spatial reasoning skills in geoscience courses
(Ormand et al., 2014).
Four broad spatial skills were identified that are important for success in the geosciences: disembedding, reasoning about dynamic processes, penetrative thinking, and
scaling. For each skill, we focus on developing sketching exercises to support the understanding of disciplinary content with complex spatial relations and perhaps support development of the underlying cognitive processes (Titus & Horsman, 2009; Uttal, Miller, &
Newcombe, 2013). We note that these four spatial skills are important throughout
science, technology, engineering, and math (STEM) disciplines, and they are not solely
applicable to geoscience (Newcombe & Shipley, 2015).
Disembedding is the ability to identify and focus on important information from a vast
array of information (Shipley et al., 2013). Expert geoscientists are able to walk up to an
outcrop, ignore irrelevant features such as vegetation and rubble, and identify key geologic features, while novices often focus on non-relevant objects (Canham & Hegarty,
2010; Coyan, Busch, & Reynolds, 2009; Goodwin, 1994; Jee, Gentner, Forbus, Sageman,
& Uttal, 2009; Jee, Sageman, Manduca, & Shipley, 2014; Manduca & Kastens, 2012;
Petcovic, Ormand, & Krantz, 2016; Reynolds et al., 2006). Expert geoscientists often use
disembedding during fieldwork, looking at diagrams, microscope work, and many other
activities.
Reasoning about dynamic processes includes all spatial reasoning that involves movement or change. Geologists are skilled in reasoning about a wide range of transformations
over time, including both brittle deformation (e.g., breaking) (Resnick & Shipley, 2013)
and plastic deformation (e.g., bending) (Atit, Shipley, & Tikoff, 2014), and recognizing
causal relationships (Jee et al., 2009, 2014). For the most part, changes caused by geological processes cannot be observed directly, because the changes that happen during a
human lifetime are insignificant, but over geologic time the cumulative effect of those
changes is profound. In most cases, therefore, the processes must be inferred from the
traces left behind, in the form of spatial patterns.
Penetrative thinking is the ability to use visible surface information to determine interior spatial structures (Kali & Orion, 1996). Expert geologists use this skill to create
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
947
models of Earth’s subsurface given sparse surface and subsurface data (Reynolds et al.,
2006), while medical professional need to create mental models of the human body
(Hegarty, Keehner, Cohen, Montello, & Lippa, 2007). Students’ skills can range from not
recognizing that patterns on the outside of a volume can be used to infer interior structures, to a sophisticated ability to integrate patterns on multiple surfaces to infer a complex 3D geometry (Kali & Orion, 1996). Introductory textbooks include many diagrams
that require this skill (Atit, Gagnier, & Shipley, 2015, reports an estimate of 17% of all
diagrams in the most widely adopted textbooks required this skill). Sketching with feedback has been found to improve reasoning with these diagrams (Gagnier et al., 2016).
Scaling is the skill required to understand and visualize very large and very small magnitudes, and to reason about relationships across different magnitudes. Geoscientists continually move between different spatial scales (e.g., moving from the atomic structure of
minerals in a hand sample to their implication for regional geology) to solve problems
(Kastens & Ishikawa, 2006; Manduca & Kastens, 2012). While students can generally
correctly order geological temporal and spatial magnitudes, they have difficulty reasoning
about these scales (Libarkin, Anderson, Dahl, Beilfuss, & Boone, 2005; Libarkin, Kurdziel, & Anderson, 2007; Trend, 1998, 2000). Recent research has shown that multiple
opportunities to sketch relationships across scales can improve scaling (Resnick,
Davatzes, Newcombe, & Shipley, 2016).
3.2. Sketch-based educational software
If a primary barrier to employing sketching in a classroom is the lack of instructor
time to provide effective feedback, then any solution must minimize instructor time and
any time requirements would ideally be independent of the number of students. One obvious class of such solutions is intelligent tutoring systems. AI-based tutoring systems have
been developed for many domains and, with the right tutoring strategies, can be as effective as human tutors (VanLehn, 2011). However, for a highly spatial domain like geoscience, one of the main technical challenges in building intelligent educational software
is understanding the spatial information in a student’s input. Sketching is a natural way
to input spatial information, but having the tutor interpret the spatial information accurately is a challenge.
Artificial intelligence researchers have explored techniques for interpreting sketches,
including the use of highly specialized sketch recognition algorithms (e.g., Taele, Barreto,
& Hammond, 2015; Valentine et al., 2012) and/or cognitive models of sketch understanding (Forbus, Usher, Lovett, Lockwood, & Wetzel, 2011). Sketch recognition works well
for domains where there is a well-defined mapping between symbols and concepts. However, many of the sketching exercises in geoscience use symbols that can represent different concepts in different contexts. For example, depending on where it is located in the
image, a straight line could represent a fault, a stratigraphic contact, the water table, or
some other geological feature. An algorithm that has perfect accuracy in identifying
straight lines would not be enough to understand the geoscientist’s sketch.
948
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
An alternative approach to sketch understanding has been implemented in CogSketch
(Forbus et al., 2011), which uses qualitative spatial representations to capture the content
of sketches. CogSketch circumvents the issue of having symbols with many potential
meanings by having users enter conceptual labels for the things they draw. This mimics
the way people often speak or provide text labels for things when they sketch (Schlaisich
& Egenhofer, 2001). When someone sketches something into CogSketch, he or she draws
glyphs, discrete parcels of ink and content, with a mouse or stylus, and labels the items
that he or she draws. Labels can indicate that a glyph represents something that belongs
to a particular category (e.g., a fault) or that a glyph represents something more abstract,
like an arrow to indicate movement or a relationship. CogSketch automatically computes
qualitative spatial relationships between the glyphs to support sketch understanding; for
example, the sandstone is above the shale.
Sketch worksheets (Yin, Forbus, Usher, Sageman, & Jee, 2010) are a domain-general education application built within CogSketch that we used to create geoscience
exercises. The general purpose of sketch worksheets is to provide a platform for
instructors to develop their own sketching exercises. It works by having an author
draw a solution to their exercise and identify important parts of the solution. The
author defines feedback prompts that are shown to the student if any of those important parts are missing or incorrect in the student’s sketch. The author can also define
grading rubrics so that the software can automatically produce an accuracy score for
any sketch. While working on a worksheet, students can click a Feedback button at
any time to receive advice about their sketch (Fig. 1). The Feedback button signals
CogSketch to compare the student’s sketch to the solution sketch created by the
author. The comparison process identifies any important aspects that are missing or
incorrect, and it provides students with the pre-written advice. Students can correct
their sketch while revising according to the advice, continuing to click the Feedback
button and correcting their sketch as many times as they want until they receive the
feedback message: “Your sketch looks good to me!”
As described above, sketch worksheets provide immediate feedback that is controlled by the user. Research focusing on the learning effects of immediate versus
delayed feedback is complex and does not always show that one method is superior
to the other (Kulik & Kulik, 1988). However, research that more closely applies to
CogSketch suggests that immediate feedback should be an effective strategy. A metaanalysis by Kulik and Kulik (1988), incorporating 53 studies, found that immediate
feedback was more effective in most included studies, particularly applied studies
focusing on classroom quizzes, as well as studies with computer presentation of feedback. Delayed feedback was only found to be more powerful in special experimental
situations. Another meta-analysis conducted by Azevedo and Bernard (1995) found that
computer-presented feedback was more effective than no feedback and that immediate
feedback provides the best instructional advantage. Additionally, a study using an
intelligent computer tutor to help students complete computer-programming exercises
found that instant, on-demand feedback helped students solve problems faster and
more efficiently than those without feedback (Corbett & Anderson, 2001). Overall, this
949
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
Worksheet Image
Worksheet Solution
Pressure/Temperature Diagram
0
0
Temperature °C
400
600
200
Pressure/Temperature Diagram
Temperature °C
800
1000
0
0
200
400
600
Hornfels
200
Zeolite
400
Pressure (MPa)
Greenschist
Pressure (MPa)
200
Amphibolite
Student attempt
800
1000
1200
Granulite
Blueschist
1000
Eclogite
Eclogite
1200
Cycle of
attempts and
tutor
Student Corrected Sketch
feedback
Pressure/Temperature Diagram
Pressure/Temperature Diagram
Temperature °C
Temperature °C
400
600
800
Tutor Suggestions
Pressure (MPa)
400
1000
Greenschist
200
400
600
800
Tutor Suggestions
1000
Hornfels
200
- Your drawing of the Subducting Plate Pathway isn’t quite
right. You just need to connect the dots! Start at 0,0, connect
the dots, and ontinue the line to the bottom.
- Review your X,Y graphing skills! You should try Subducting
Plate 4 again.
Zeolite
400
Amphibolite
1000
Eclogite
Hide missing entity suggestions?
Greenschist
Your sketch looks
good to me!
600
All feedback
corrected
800
Granulite
Blueschist
Update suggestions
1200
0
Suggestions for Sketch Meta_Facies
Selecting a suggestion will highlight the associated items on
the sketch.
Zeolite
600
800
0
Suggestions for Sketch Meta_Facies
Hornfels
200
1000
Pressure (MPa)
0
200
Amphibolite
800
Student Incorrect Attempt
0
Greenschist
400
600
Granulite
Blueschist
1000
Zeolite
Create solution
600
800
Hornfels
Granulite
Blueschist
Amphibolite
Eclogite
Update suggestions
1200
Hide missing entity suggestions?
Fig. 1. Flow chart of CogSketch worksheet completion process from the Metamorphic Facies Diagram worksheet. Top left: Author inserts a worksheet image that is used as a backdrop for a particular sketching task.
In this case, students are graphing Pressure/Temperature points onto the Metamorphic Facies diagram. Top
right: Author creates a solution sketch on top of the worksheet image. Lower left: The student sketches a
solution to the problem and can request feedback from the software at any time. When the student requests
feedback, the student’s sketch is compared to the solution sketch and advice (written by the worksheet
author) is provided for any discrepancies. Lower right: The student continues to edit his or her sketch and
press the Feedback button until there are zero discrepancies or until the student is satisfied with the sketch.
research supports that immediate feedback provided by CogSketch may help students
complete tasks more efficiently and help improve learning compared to students who
do not receive feedback.
3.3. Sketch understanding requirements in geoscience
A complete explanation of the representations and processes used by CogSketch and
Sketch Worksheets can be found elsewhere (Forbus, Chang, McLure, & Usher, 2017;
Forbus et al., 2011). Here, we focus on the capabilities that were required to allow CogSketch to effectively reason and provide feedback on geoscience sketches.
950
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
3.3.1. Quantitative ink constraints
Many of our planned CogSketch geoscience worksheets required image annotation.
When the absolute location of ink is important for drawing a sketch accurately, we use
quantitative ink constraints to define regions that should contain the ink. For example, in
a worksheet where a student must identify faults in an outcrop, his ink needs to coincide
with the actual location of the fault in the image. When the author draws the solution to
the worksheet, she would define a quantitative ink constraint around her correct version
of the fault line. The author sets a numerical tolerance, which is used to define a tolerance region for the student’s glyph. If the student’s glyph falls completely within the tolerance region, the constraint is satisfied. If not, the constraint is violated and the student
receives advice about it (e.g., “Your fault line is not in the correct location”). The author
may also define position-dependent advice prompts, to give the student more specific
advice if there is a particular positional relation between the student’s glyph and the solution glyph (e.g., “Your fault line is too low”).
3.3.2. Arrow interpretation
Arrows are used in CogSketch to convey both movement and processes. In order for
CogSketch to understand arrows in human-like ways, their qualitative direction must be
taken into account. Arrow interpretation identifies the head and tail of the arrow and
computes a qualitative vector between the two. The qualitative vector is oriented in one
of eight qualitative directions: up, down, left, right, and the diagonal quadrants (e.g., up
and to the right, down and to the right, etc.). Carving the visual space into these eight
qualitative directions, rather than using a quantitative representation like radians, is a cognitively plausible approach to arrow direction interpretation.
3.3.3. Grading
Grading in sketch worksheets uses analogical comparison (i.e., the structure-mapping
engine; Falkenhainer, Forbus, & Gentner, 1989) to detect important differences between
the student sketch and the solution sketch (for more information on this process, see Forbus et al., 2011). In the same way that differences are detected for generating feedback,
the differences found when students are finished with their worksheets are evaluated
against author-defined rubrics. Points are awarded based on these rubrics and scores are
automatically computed for each student worksheet.
4. Development of introductory geoscience worksheets
Geologists, psychologists, and computer scientists collaborated to develop a series of
26 Introductory Geoscience CogSketch worksheets that focus on key geoscience concepts
(Table 1). Our aim was to support students’ ability to learn complex spatial concepts and
make sketching assignments practical to grade. To this aim, we identified important spatial concepts from each week of an introductory course. To ensure a broad range of spatial problems, we selected concepts that represented at least one of the four classes of
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
951
Table 1
Relationship between Geoscience topics and CogSketch worksheets
Topics Covered in an Introductory Geoscience
Course
Plate tectonics
Minerals
Igneous rocks and processes
Sediments and sedimentary rocks
Metamorphic rocks and processes
Structural geology
Earthquakes
Geologic time/biogeography
Mass movement
Streams and floods
Oceans and coasts
Groundwater
Glaciers and glacial processes
CogSketch Geoscience Worksheet Concepts
Earth’s interior
Earth’s magnetic field
Magnetic reversals
Mineral grain boundaries
Mineral cleavage
Classifying igneous rocks
Effects of pressure and temperature on igneous rocks
Stratigraphy
Transgression and regression
Metamorphic facies
Metamorphic Rocks and fabrics in outcrops
Identifying geologic structures in outcrops
Strike/dip and block diagrams
Topographic maps
Fault identification
Stick-slip faulting and earthquakes
Geologic time scale
Fossil succession
Slope stability
Flood recurrence
Coastal processes
Ocean currents
Groundwater flow
Water table contours and contamination
Glacial movement
Glacial geomorphology
spatial thinking skills (disembedding, visualizing change, scaling, and penetrative thinking). Two to three target concepts were identified for thirteen topics commonly taught in
introductory geoscience courses. Topics were identified, by the geoscience experts in our
group, based on a review of commonly used Introductory Geoscience textbooks, particularly those textbooks where spatial concepts and diagrams are prevalent (i.e., Marshak,
2012; Reynolds, Johnson, Morin, & Carter, 2012). While each topic may not be present
in every course or textbook, we chose the most common topics to ensure that a range of
usable worksheets would be available for most introductory geoscience courses.
Table 1 lists the 26 key concepts (right column, Table 1) along with their corresponding topics commonly found in geoscience textbooks (left column, Table 1). Each worksheet contains instructions for the student and a background image or diagram that
students use to complete sketching tasks. In order to better understand the challenges and
opportunities of an intelligent tutor for sketching, we created analogous paper-based
952
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
worksheets. Instructions for paper and CogSketch worksheets were conceptually the same
(i.e., they always posed the same problem, described the same tasks, and the same goal),
differing only in how they described drawing and manipulation steps to reflect the different media of paper versus computer. Notably, some CogSketch worksheets are completed
by moving diagram elements, which is not possible to do with a single sheet of paper.
Therefore, instructions were written to be as similar and specific as possible to ensure that
student sketches would be consistent between the two formats.
4.1. Worksheet development
To illustrate how a CogSketch worksheet was conceived and developed, we will
review the Earth’s Magnetic Field worksheet in detail.
4.1.1. Identifying important geological concepts
Target worksheet concepts (right column, Table 1) were identified by reviewing introductory geology textbooks, from the authors’ classroom experience, and through discussions with other instructors, teaching assistants, and undergraduate students. Concepts
were selected as targets if they are challenging for students and sketching could be used
to support students’ learning. The Earth’s Magnetic Field worksheet (Fig. 2) was specifically requested by an instructor, who noted that students have difficulty visualizing
Earth’s magnetic field in three dimensions and understanding how the inclination of magnetic field lines project onto the surface of the Earth, despite the fact that the introductory
textbook contained a figure illustrating this concept.
4.1.2. Creating an exercise that utilizes interactive attributes of CogSketch
Once the concept was chosen, the authors determined the cognitive processes that were
involved when reasoning about the concept and how the concept correlated to the four
spatial reasoning problems. An exercise was then developed around activities that would
support the relevant spatial thinking skills and processes. A challenge in developing
worksheets using CogSketch was to move beyond paper and effectively utilize the interactive capabilities of CogSketch. Our intention in designing CogSketch worksheets was
to maximize sketching, interacting with diagrams, moving objects around, and reading
and responding to the worksheet tutor’s feedback. Thus, we expect students to play, interact, or use a trial-and-error approach to grasp the content and potentially correct common
mistakes or misconceptions through an iterative process with the tutor. We propose that
this interaction has the potential to enhance student understanding of the concepts and
lead students to self-evaluate mistakes, which has been shown to be a strategy that can
facilitate deeper learning (Bjork & Bjork, 2011).
The goals of the Earth’s Magnetic Field worksheet (Fig. 2) are for students to see how
magnetic inclination changes with latitude and to understand how geologists use that
information to make inferences about a rock’s historic location. Thus, this worksheet is
designed to support students who are having difficulty with the dynamic reasoning necessary to accurately infer movements over time.
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
Rotation Axis
953
Magnetic Pole
Geologic units with magnetic orientation:
Move and rotate to correct red dot
Example
N
S
Students move and
rotate each rock to the
correct place, matching
magnetic field lines
Fig. 2. Earth’s Magnetic Field worksheet. This worksheet consists of a diagram of the Earth’s magnetic
dipole and the associated magnetic field. Students complete the worksheet by moving and rotating the packages of rock to a point where the arrow in the rock, its magnetic inclination, matches the magnetic inclination
of the Earth. The red line on each “rock” indicates the top of the rock to help students correctly orient the
boxes. The red outlined example is given to students, while the black dashed lines illustrate two additional
examples for the reader.
Geologists use the orientation of magnetic minerals in a rock to understand the change
in latitude that has occurred since the rock was formed (Marshak, 2012). Different analyses can determine the three-dimensional orientation of magnetic minerals in a rock (like a
pencil in a particular orientation in space). A rock’s magnetic orientation can be represented numerically, but it is much more informative to visualize the orientation, as well
as the comparison to the Earth’s magnetic field (Gordin & Pea, 1995). The Earth’s Magnetic Field worksheet involves a simplified task that mimics the expert process by having
students match the magnetic inclination of a “rock” with the latitude that has the same
magnetic inclination. The worksheet consists of a diagram of the Earth and its magnetic
field, an explanation with the key features of the diagram, and a task designed to help
students understand how rocks can capture the orientation and inclination of Earth’s magnetic field as a rock cools, and that orientation can be used to deduce where the rock was
when it cooled. The diagram has “packages” of rock that contain magnetic orientation
(arrows). Students grab, rotate, and drag each package of rock to the location on the diagram where the magnetic arrow in the rock matches the inclination of Earth’s magnetic
field (action indicated with a red arrow in Fig. 2). Note, in this task, matching and
954
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
comparing objects to a diagram is a form of analogical reasoning; analogies are a powerful tool for teaching a new concept using what is known about a familiar concept (i.e.,
Gentner & Smith, 2013; Jee et al., 2010).
Students will succeed in this task if they notice how the magnetic inclination of an
individual rock can be related to a latitudinal position on the Earth and how inclination
changes with latitude. The process of moving and rotating objects in CogSketch allows
students to match the rocks to the Earth’s magnetic field (a task that geologists do mentally) without attempting to visualize the entire diagram while mentally rotating an object,
which is a difficult mental task (Xu & Franconeri, 2015). Lastly, instead of creating the
analogy of the magnetic inclination of the rock to the diagram in a student’s mind, students physically align the two objects. This is a literal analogy as rocks do contain magnetic particles that you can physically rotate back to the current magnetic field.
Therefore, the exercise incorporates both geologic concepts and analogical reasoning as
structured and supported by CogSketch, to increase the potential to learn from a homework exercise.
We should note that while the task of moving and rotating objects can make a spatial
task easier for students to visualize and complete, it also increases the chance of students
completing the worksheet using a strategy that is not connected to the geologic content,
such as just focusing on aligning the arrows and the magnetic field lines. In each worksheet, we try to discourage these types of strategies by focusing the directions and feedback messages on the content and spatial relationships rather than just the physical task.
However, future work should consider focusing on identifying when and if students are
using “gaming” strategies to complete the worksheets, rather than focusing on the content
(Baker, Corbett, & Koedinger, 2004).
4.1.3. Creating the final CogSketch worksheet
In the worksheet development process, an author uploads background images and
objects, inputs written directions for the student, creates the solution sketch, identifies
important quantitative and qualitative facts about the solution sketch, and constructs the
necessary feedback messages for each important fact. Once all the important features
about the solution sketch are identified, the author can set the grading rubric and protect
the solution with a password. The password prevents the student from looking at the solution, but it allows the solution to be used by the program and new instructor-users. The
worksheet editor interface was designed to be used by a non-expert. The first author created all worksheets for this project and is a geologist with no experience in computer programing. We estimate that <2 days were necessary to obtain sufficient proficiency to
construct a functional domain worksheet from a textbook diagram. Most of the time
required to complete the authoring process is spent testing the worksheet and exploring
the various possible solutions that a student might submit. This process often reveals situations where feedback could be more or less specific and gives the author the opportunity
to fine-tune the ideal solution and feedback messages.
Besides moving and rotating objects, other CogSketch geoscience worksheets made
use of three notable tools in CogSketch:
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
955
1. Drawing and labeling. The draw tool allows students to sketch by free hand and
then label each drawn object. This supports worksheet tasks that involve drawing
scale bars, annotating features on a photo or chart, graphing, completing diagrams,
drawing objects with certain spatial relations, and many other tasks. Drawing and
labeling may help facilitate disembedding by drawing students’ attentions to features as they trace and annotate features of interest in photos, as well as diagrams
and data graphs (Johnson & Reynolds, 2005). Penetrative thinking tasks also
require drawing and labeling since students are often sketching features that are not
directly visible.
2. Resizing. Students can resize (and move) any objects they draw in CogSketch. Any
objects provided by the worksheet author can also be resized and moved by the student, unless the worksheet author applies editing locks on them. Resizing is useful
when working on scaling tasks since students can directly scale objects instead of
re-drawing them.
3. Drawing arrows. CogSketch’s arrow interpretation identifies the qualitative direction of hand-sketched arrows (e.g., to the right, to the left, pointing from A to B).
Drawing arrows is useful for incorporating reasoning about motion into worksheets
and conveying movement, feedback loops, and interaction between objects.
5. Geoscience classroom implementation
To evaluate the prototype worksheets, CogSketch worksheets were used in the introductory Physical Geology course at the University of Wisconsin-Madison in the spring
semester of 2014. Two sections of Physical Geology participated in this implementation,
taught by professors who are colleagues of the authors, with a total of 262 participating
students. The goal of this activity was to better understand the challenges and requirements of implementing a sketch-based tutoring system in an authentic STEM learning
environment, as well as promote the understanding of the cognitive science principles that
are needed to support learning by a sketch-based tutoring system. The longer-term goal
was to develop a worksheet tutor that could provide instructor-quality feedback on
sketching assignments, and use the tutor to support student sketching in large-enrollment
courses. To better understand the design challenges and opportunities of the worksheet
tutor for sketching, we contrasted implementation of CogSketch worksheets with a
matched set of paper-based worksheets.
Over the course of the semester, students completed either all CogSketch worksheets
or all paper worksheets as homework assignments. Paper worksheets contained the same
images and tasks as the CogSketch worksheets, adapted as needed for the medium (e.g.,
redrawing a line on the paper worksheet instead of dragging a line across the screen).
Students were initially assigned to use either paper or CogSketch worksheets but were
allowed to self-select into the other group if they preferred to do so. Many students opted
out of the CogSketch group and into the paper group. Students typically opted out of the
CogSketch group because they did not own a Windows machine (which is currently the
956
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
only OS that can run CogSketch), they did not want to have to use an on-campus lab to
complete their assignments, or because they were reluctant to use a new computer program. In sum, 65 students completed CogSketch worksheets and 197 students completed
paper worksheets. CogSketch worksheets were downloaded from the course website and
students either downloaded CogSketch onto their personal computers or used public computers at UW-Madison that had had CogSketch preloaded.
Each section of the course was taught independently by one of two professors who
selected different sets of worksheets. Instructors chose worksheets from those presented
in Table 1 and assigned each worksheet to match lecture content. In total, instructors
chose 16 worksheets to assign in their classes. One professor assigned nine worksheets,
and the other professor assigned eleven. Of those, four worksheets overlapped between
the two sections (Table 2). Students completing CogSketch worksheets received immediate feedback and all completed worksheets were emailed to the first author. Paper worksheets were turned in during lecture, graded by the first author and provided with
feedback comments similar to those that would appear in CogSketch, and then returned
the following week in discussion sections.
5.1. Implementation observations
Students in the CogSketch group quickly learned how to use the CogSketch program
and completed their first worksheet during the first 2 weeks of class. Students completed
all CogSketch worksheets on their own time, without a CogSketch or geoscience expert
present, and about 85% of students turned in all assignments during the course of the
semester. Paper worksheets were turned in at a similar rate (87% of students turned in all
assignments). Overall, students had similar worksheet grades on CogSketch and paper
worksheets. CogSketch worksheet scores were not perfect, as some students did not
request feedback, and for others the feedback was not effective in correcting the error.
When comparing completed CogSketch and paper worksheets, we highlight four
important observations: (a) There were substantial individual differences in the ways students used the worksheet tutor; (b) Cogsketch worksheets helped identify and correct
common mistakes; and (c) almost no instructor time was needed to grade and provide
Table 2
Worksheets assigned to each instructor’s class and worksheets assigned to both classes
Assigned Worksheets
Assigned Only by Prof A
Assigned by Both
Assigned Only by Prof B
Mineral Grain Boundaries
Classifying Igneous Rocks
Fossil Succession
Groundwater Flow
Transgression/Regression
Geologic Time
Metamorphic Facies
Strike/Dip & Block Diagrams
Water Table Contours & Contamination
Earth’s Interior
Earth’s Magnetic Field
Mineral Cleavage
Effects of P&T on Igneous Rocks
Streams & Floods
Glacial Geomorphology
Stick-Slip Faulting & Earthquakes
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
957
individual feedback on CogSketch worksheets—the large majority of instructor time
investment in using CogSketch worksheets is in the design phase. Each observation is
discussed in greater detail below.
5.1.1. Individual differences in worksheet tutor interactions
Students requested feedback from the worksheet tutor on 89% of worksheets. For all
worksheets where the tutor was utilized, nearly 50% requested feedback 1–3 times per
worksheet with only 6% requesting feedback 20 or more times (Fig. 3). While a student
completes a worksheet, CogSketch records all actions that occur (e.g., creating an object,
rotating an object, requesting feedback, etc.). We compiled this information to analyze
how students completed each worksheet. We noticed that students used the worksheet
tutor in a variety of ways:
1. 11% of students never requested feedback and simply completed the worksheet and
turned it in, treating it the same way one would a normal homework assignment.
The average worksheet grade for this group was 52%.
2. 37% of students completed all or most of the worksheet (leaving no more than two
tasks incomplete), and then requested feedback, after which they corrected mistakes
or chose to do nothing. The average worksheet grade was 89%.
3. The remaining 52% of students requested feedback throughout the worksheet completion process (these students averaged eight feedback requests per worksheet).
For this strategy, the average worksheet grade was 82%.
Though descriptive, these findings suggest that the tutor feedback holds promise for
supporting students’ geoscience understanding, and that the program allows individuals to
complete the worksheet using different strategies to arrive at the correct answer. Future
work can experimentally examine the factors that influence student use of the worksheet
Fig. 3. Histogram of feedback requests for all worksheets used in our implementation. Students requested
feedback 1–3 times on ~50% of worksheets and requested feedback 20+ times on only 6% of worksheets.
958
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
tutor, the impact of tutor feedback on student learning, as well as the robustness of students’ knowledge when using these different strategies in CogSketch.
Requesting feedback generally helped students correct their sketches (an example of
incorrect and corrected sketches is shown in Fig. 4). The feedback could guide students to the feature of the sketch that was in error, so they could reconsider and correct the spatial location. After feedback, the student often redrew the lines, completing
a correct sketch. This is a type of desired difficulty (creating some difficulty for the
student to encourage deeper learning) that causes students to reevaluate the task or
process and use corrective feedback to address incorrect interpretations (Bjork &
Bjork, 2011; Hattie & Timperley, 2007). However, in some instances, students
requested feedback and redrew a feature multiple times without drawing it correctly.
Multiple factors may have attributed to this outcome, including: (a) student error in
labeling objects correctly, which resulted in non-helpful feedback from the tutor; (b)
feedback may have directed attention to the spatial relationship that was important, but
that alone did not help student correct their sketch; (c) feedback message did not
always provide information about the spatial nature of the error; and (d) student
needed more information rather than corrective feedback (Hattie & Timperley, 2007).
When the feedback did not help the students correct the object, then they kept redrawing the object until they happened upon the correct answer, or they chose to turn in
the assignment with an incorrect sketch. A record of these interactions provides
authors with an opportunity to revise the feedback to help future students who have
similar problems. One such example, from the Groundwater Flow worksheet, occurred
when many students had incorrect sketches that would have been graded as correct if
hand graded. In this case, the ink tolerance was too strict and any bumps or jogs in a
drawn line would count as out of bounds and thus wrong. The worksheet was revised
by increasing the ink tolerance and testing the revised version against the worksheets
originally graded as incorrect.
INCORRECT
222
Poorly maintained
chicken farm
219
216
219
214
216
208
209
213
leaking
septic tank
205
208
Poorly maintained
chicken farm
219
211
212
CORRECT
222
206
Feedback message: The boundaries aren’t quite right. The 2
lines need to be perpendicular to
the contour lines (parallel to your
flow arrow) and should extend
from the outermost boundaries of
the chicken farm. Continue the
lines to the edge of the image.
216
219
211
216
208
209
213
leaking
septic tank
205
212
208
201
207
214
206
201
207
204
202
199
204
202
199
Fig. 4. Corrected CogSketch sketches. A student’s Groundwater Contamination sketch before and after feedback. Student did not have lines in correct locations but redrew lines in the correct location once receiving
the feedback message.
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
959
5.1.2. CogSketch worksheets helped identify and correct common mistakes
Because students had analogous versions of worksheets for CogSketch and paper, it is
possible to compare how students complete the same homework tasks in the two formats.
First, we consider one of the most common and problematical spatial errors in reasoning
about geoscience block diagrams. In the Geologic Block Diagram paper worksheet, 35%
of students made a common mistake when completing a block diagram (n = 189, problem shown in Fig. 5). This specific mistake represents nearly 60% of all incorrect submitted sketches. Instead of connecting the same unit on the two sides of the block, students
continued the lines nearly parallel to how they were drawn on the sides. An expert would
immediately recognize that this was an error for two reasons: It is a geologically unlikely
configuration, and the orientation of the drawn planes would be inconsistent with the dip
symbols on the top face. This is similar to errors reported by Kali and Orion (1996), who
found students tended to continue lines from a known face to an unknown face.
For the Geologic Block Diagram CogSketch worksheets, 30% of students made the
common block diagram mistake (n = 19), which equates to 65% of all incorrect sketches
(19 out of 29 incorrect sketches). In contrast to the paper worksheets, the worksheet tutor
helped to correct 7 out of the 19 incorrect block diagram sketches (Fig. 5). Failure to correct all 19 incorrect sketches most likely resulted from inadequate feedback messages
from the worksheet tutor and students choosing not to request feedback on their incorrect
sketch. Thus, CogSketch is capable of helping students correct this common mistake
when the error is first made, but the feedback messages should be improved and students
should be prompted to use the tutor during each worksheet.
Although we anticipated that students would make errors, our CogSketch geoscience
worksheet was not specifically designed to detect the specific form of the error. For
Feedback: This layer contact isn’t quite right. The 2 sides show
that the layers are planar and you have 2 contact points to use, one
from the top face and one from the side face. Connect the 2 points
with a straight line, that is your layer contact!
Fig. 5. Corrected block diagram sketch. The image on the left is an incorrect sketch of the missing side of a
block diagram. The student received the Feedback message shown below the sketches and corrected the
sketch (right image). This is an example of how feedback was used to correct student sketches.
960
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
unanticipated errors such as this, instructors may inspect completed sketches to determine if
there is an error made by a large percentage of students. This can be done within minutes
and allows an instructor to address the common mistake and revise the feedback messages
to help future students correct the mistake early in their training. Such errors can be revealing about basic cognitive processes and thus serve as a powerful mechanism for increasing
the likelihood of serendipitous discovery (see e.g., Gagnier & Shipley, 2016).
An important difference between CogSketch and paper worksheets is the ability to
move and drag objects. This appeared to be very useful in the Earth’s Magnetic Field
worksheet, where for the paper version students had to redraw the packages of rock with
arrows onto the image of the Earth. Many students did not redraw the objects correctly
on the paper worksheets (Fig. 6). The CogSketch worksheet allows students to focus on
the geoscience task, matching the rocks to Earth’s magnetic field, rather than focusing on
how to redraw the objects in a different orientation. Here, CogSketch likely lessened the
mental burden of mentally rotating and moving objects, and allowed students to attend to
the critical variables of the geoscience concept. Future research should consider learning
outcomes in evaluating the value of moving components when understanding how to infer
location from magnetic information in rocks.
Despite the differences in labeling and moving objects, sketches between CogSketch
and paper worksheets were very similar and contained similar errors and variability.
Incorrect paper sketch
Correct CogSketch sketch
Rotation Axis
Magnetic Pole
Most arrows
redrawn
incorrrectly
N
Arrow pointing
the wrong way
S
Fig. 6. Difference between Earth’s Magnetic Field CogSketch and paper worksheets. Left image: Correct
completion of the CogSketch worksheet. Students moved and rotated boxes with arrows until the arrows
aligned with the magnetic field lines. Most students correctly completed this task. Right image: Incorrect
sketch from a paper worksheet. Students completing the paper Earth’s Magnetic Field worksheet had to
redraw the boxes with arrows instead of dragging and rotating (boxes were still off to the side as shown in
the CogSketch version in Fig. 2). Therefore, paper worksheet sketches contained boxes with arrows not in
alignment with Earth’s magnetic field lines and/or redrawn improperly.
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
961
Instructions were kept as consistent and specific as possible to ensure this would be the
outcome. Future research could involve analyzing how the steps and tools used in CogSketch affects sketching and how it might change what someone would sketch on paper.
5.1.3. Less time is needed to grade and provide individual feedback on CogSketch
worksheets
One obvious difference between a paper and computer worksheet was the role of the
worksheet author and grader. For this implementation, we aligned the grading to be as
similar as possible. The first author constructed CogSketch’s feedback suggestions, created the grading rubric used for the CogSketch worksheets, and developed and applied
the rubric for the human-graded paper worksheets.
For this implementation, it took 4–6 h per week to grade the paper worksheets for the
196 participants in the study, highlighting why most instructors do not assign sketching
in large classes. In contrast, it took 5–10 min per week—total—to upload the emailed
worksheets, apply the grading rubric, and analyze the sketches that had been completed
in CogSketch (66 participants). In this preliminary implementation, we monitored the
need for individualized feedback on the CogSketch worksheets by reviewing all worksheets after they were graded. If any unanticipated mistakes were found, the first author
either wrote an email explaining the correction, created a video of the author correcting
the sketch and verbally explaining the process, or drew the correct sketch and provided a
written explanation. In some instances, a common problem or mistake did not receive
detailed feedback from CogSketch and an email explanation was provided to all students.
Then, the worksheet was revised to eliminate the issue in future use. In a few cases, the
unique mistakes were corrected with individual feedback from the first author. Even in
this circumstance, it took less time than it took to hand grade and provide individual
feedback on paper worksheets. Therefore, CogSketch could free approximately 75 h of
instructor time for a 200-person class (assuming 15 sketch assignments). Future research
should push this comparison further and focus on determining whether CogSketch grading matches human grading.
6. Discussion
An interdisciplinary approach to science education that combines cognitive science and
disciplinary science offers a unique opportunity to craft learning tools that support student
understanding of cognitively challenging disciplinary content. AI systems that can support
sketching offer new opportunities. The advantages of CogSketch include (a) the ability to
implement sketching in the classroom where the instructor is not needed to view progress
on every student’s sketch; (b) feedback upon demand; and (c) a structure for cognitive
science theory to be implemented in the design of geoscience materials.
The introductory course in Physical Geology—in which CogSketch and graded paper
worksheets were used—offered students more sketch-based homework assignments than
offered by over 90% of surveyed instructors where the assignments must be hand graded.
962
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
From the perspective of a disciplinary instructor, the availability of well-crafted and cognitively robust worksheets, and the time savings offered by CogSketch, would make a
significant difference in the frequency of sketching opportunities in introductory courses.
A digital sketch provides opportunities that are not possible with a paper sketch. For
example, in CogSketch worksheets, students can “grab” lines or objects, rotate them, and
move them around on a base image. The ability to move or rotate objects supports students in making visual comparisons. The alternative, when lines cannot move (as with
images on paper), likely requires additional mental rotations and other mental transformations as a prerequisite for performing a mental comparison. It is likely that students with
low spatial working memory capacities may have difficulty with such a comparison (Baddeley, 2012). Thus, a digital tool may better support learning in such students. However,
abstracting physical relations is a key step in learning (Novack, Congdon, Hemani-Lopez,
& Goldin-Meadow, 2014), and it is possible that enabling some operations that had to be
completed mentally will undermine abstraction and, thus, generalization of learning. An
important line of research in the design of the CogSketch tutor for sketching is the net
learning value of manipulable elements.
The record of student activity while using CogSketch, such as seeking feedback from
the worksheet tutor, is a valuable resource for developing the science of learning. By
observing how students correct spatial mistakes in response to spatial feedback, a robust
theory of spatial learning may be developed. Such a theory is necessary for guiding the
construction of optimal spatial feedback.
CogSketch offers a crucible to combine the science of learning with the context of a
disciplinary science—geoscience. For cognitive psychologists, CogSketch offers a powerful research tool for gathering information about student learning, including the documentation of common mistakes, the role of feedback, and the individual use of feedback. For
geology instructors, CogSketch offers an opportunity to employ sketching in their courses
without the usual time commitment needed for grading. As a mechanism for collaboration
between cognitive science and geoscience educators, CogSketch offers a structure that
requires that (a) psychological theories are formulated in sufficient detail to be applied to
science tasks; and (b) geoscience educational goals are formulated in sufficient detail to
be evaluated. When the above are combined appropriately, worksheets will be engaging,
illustrative, and provide useful feedback when errors are made. Community-based projects
to develop cognitively appropriate and effective CogSketch worksheets for introductory
science classes are a potentially powerful engine for advancing science. Our observations
about this process suggest that it is important to have all sides at the table: the geologist
to provide legitimate goals, the psychologist to provide guidance on structuring learning,
and the AI researcher to provide tools that can evaluate goals and structure learning without real time human guidance.
Deploying the intelligent tutoring systems in classrooms still has some obvious limitations, as when no digital devices are available for students, or when the system is platform specific (currently CogSketch only runs on windows-based, or partitioned,
computers; however, an HTML, cloud-based version is in development). In the case of
the present study, the platform-dependency played a role in students self-selecting out of
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
963
the CogSketch group and into the paper group. Other technical challenges, mentioned
above, include the need for more detailed spatial representations and the trade-offs
between feedback specificity and generality. These challenges have implications for future
development of CogSketch and for the design of sketch-based tutoring systems in
general.
6.1. Lessons learned
Throughout the design and development of CogSketch geoscience worksheets, our
group has learned about an interdisciplinary collaborative approach to designing educational materials. The following points are lessons learned from our design process,
abstracted so that they might be applied to other interdisciplinary collaborations in
designing curricular materials.
1. The CogSketch program and worksheets are adaptable and able to change with use
and development. It is necessary throughout the design process to have continued
investment from developers and practitioners. Here we summarize key adjustments
in the design process that were only possible in the context of an integrated group.
•
Small-scale design changes necessary to enhance the tutoring system’s handling
of spatial concepts and tasks to improve the user’s experience:
Position-dependent quantitative ink feedback. This allowed spatial feedback. Where the worksheet tutor once provided only “This is not quite
right,” directional feedback now allows, “This needs to be higher.”
Arrows to indicate processes. Being able to interpret arrow orientation
o
allows tasks involving movement and change. Tasks, such as drawing an
arrow to show water flow, used to involve quantitative ink constraints
around a pre-drawn arrow. Arrow interpretation now allows drawn arrows
to convey directional meaning.
Feedback on progress. In the case of the present study, a side bar was
o
added to the CogSketch interface to summarize feedback with completion
gauges, a colored meter to show if a task is complete or not. This feedback allowed students to quickly see how much was correct and then drill
down into feedback categories for more detailed advice. This was
intended to prevent students from being overwhelmed by too many feedback items at once.
Large-scale changes:
o
•
o
We found that it was critical to have a disciplinary expert (with at least
graduate student level training) design the worksheets. In the beginning,
worksheets were developed by an AI expert with guidance from a geoscientist. Bridging the gap between domain expertise and worksheet implementation from both ends of the spectrum was laborious and error prone.
This approach was not an effective use of either expert’s time. In contrast,
964
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
a user-friendly authoring environment allows discipline experts to test and
develop activities without requiring deep programming knowledge. Having
an authoring environment that supports disciplinary experts extend the
system’s knowledge of particular domains also allows the AI program to
be used across different disciplines.
We also found that it was important to have a taxonomy of spatial skills
o
that were important to the geosciences, and cognitive principles that could
be directly incorporated into geoscience worksheets. The specified spatial
skills allowed the geoscience domain expert to classify geoscience concepts so that prior cognitive science research could be incorporated into a
worksheet design. This streamlined communication between the geoscientist and cognitive scientist by providing a shared vocabulary.
2. Creating effective feedback is not yet an exact process. With CogSketch, there is
an important trade-off between domain depth and breadth. In an ideal case, worksheets would provide rich, multi-step feedback, which can encourage self-evaluation and deeper learning by allowing students to correct faulty interpretations
(Bjork & Bjork, 2011; Hattie & Timperley, 2007). Most intelligent tutoring systems
achieve this by having detailed models of the domain (e.g., cognitive tutor that is
specific to geoscience concepts and processes), but CogSketch was designed for
domain-general tutoring, to handle spatial aspects of any defined concept. Therefore, feedback messages were designed to be written by the domain expert to provide the necessary detail, to direct student focus to particular features of interest
while addressing a range of possible student errors. We have found that worksheet
testing by novices is the best way to fine-tune feedback messages and increase their
effectiveness. Additionally, we have also found that position-dependent feedback,
which provides more detailed spatial feedback, allows for more specific and shorter
feedback messages. It may not be applicable to all worksheet tasks, but this feature
is very useful when it can be incorporated.
3. Despite CogSketch’s qualitative spatial reasoning understanding and feedback, there
were some instances where students were unable to understand the feedback and
correct their sketch. One suggestion for future work is to supplement textual feedback with optional spatiovisual feedback in the form of a diagram or comparison to
the solution sketch. This would integrate research that shows that visual comparison
is a quick and effective strategy to correct students’ mistakes and increase understanding (Gadgil, Nokes-Malach, & Chi, 2012; Gentner et al., 2015; Loewenstein,
Thompson, & Gentner, 1999).
4. During the development of CogSketch geoscience worksheets, geoscience educators have asked if CogSketch worksheets could work for fieldtrips or fieldwork,
since many courses incorporate sketching during these activities. Although we did
not do this, there are some scenarios where it should be possible. An instructor
could create a worksheet with outcrop photos from the outcrop the students will
visit or a topographic map of the region, and ask students to annotate the outcrop
photo or create a geologic map on top of the base map. However, these activities
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
965
would require tablets to allow easy use of CogSketch outdoors. For example, a
tablet computer with a camera could potentially take and import photos, which
would allow students to annotate particular features. A similar approach could be
applied to microscope work (e.g., taking a photo through the eyepiece of a
microscope and annotating features).
7. Conclusions
Our research illustrates the value of an interdisciplinary, D-EE-D approach to develop
educational materials that focus on a current teaching problem, integrates current research
from multiple disciplines, and tackles implementation problems simultaneously. Success
in this initial step of the design process to create a set of curricular materials that potentially reaches beyond the designer group required extensive work and multiple cycles of
theory refinement and tool adjustment.
CogSketch offers new ways to sketch in the geoscience classroom and provides students with a tool to aid in solving discipline-specific spatial problems, while providing
instructors with insights into student thinking and learning. Geoscience worksheets were
designed to address common geoscience concepts that students struggle with, and that are
important for continued learning in geoscience and STEM. Classroom implementation
showed that CogSketch provides an environment for students to physically interact with
diagrams and to receive spatially detailed feedback from the worksheet tutor. CogSketch
thus makes it possible and practical for instructors to incorporate sketching activities in
large-enrollment courses. Intelligent tutoring systems have the potential to enhance student learning in spatially rich domains, such as geoscience. Furthermore, there is nothing
in our study that limits the applicability of these findings to the geosciences. This technology and an interdisciplinary approach could be used to develop effective sketch worksheets in other STEM disciplines and beyond.
Acknowledgments
We sincerely thank the professors at University of Wisconsin-Madison for
accommodating this study in their introductory geology classrooms. The research was
funded through the NSF-sponsored SILC (Spatial Intelligence Learning Center: SBE0541957 and SBE-1041707). CogSketch and paper worksheets will be available on the
Science Education Resource Center (SERC) website, http://www.serc.carleton.edu, once
revised.
References
Ainsworth, S., Prain, V., & Tytler, R. (2011). Drawing to learn in science. Science, 333, 1096–1097.
966
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
Atit, K., Gagnier, K., & Shipley, T. F. (2015). Student gestures aid penetrative thinking. Journal of
Geoscience Education, 63(1), 66–72.
Atit, K., Shipley, T. F., & Tikoff, B. (2014). What do a geologist’s hands tell you? In D. R. Montello, K.
Grossner, & G. G. Janelle (Eds.), Space in mind: Concepts for spatial learning and education (pp. 173–
194). Cambridge, MA: The MIT Press.
Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based
instructions. Journal of Educational Computing Research, 13(2), 111–127.
Baddeley, A. (2012). Working memory: Theories, models, and controversies. Annual Review of Psychology,
63, 1–29.
Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004). Detecting student misuse of intelligent tutoring
systems. In J. C. Lester, R. M. Vicari, & F. Paraguacßua (Eds.), International conference on intelligent
tutoring systems (pp. 531–540). New York: Springer Berlin Heidelberg.
Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable
difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M. Hough, & J. R. Pomerantz
(Eds.), Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56–
64). London: Worth Publishers.
Bryk, A. S. (2009). Support a science of performance improvement. Phi Delta Kappan, 90(8), 597–600.
Bryk, A. S., & Gomez, L. M. (2008). Reinventing a research and development capacity. In F. M. Hess (Ed.),
The future of educational entrepreneurship: Possibilities for school reform (pp. 181–187). Cambridge,
MA: Harvard Educational Press.
Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America’s
schools can get better at getting better. Cambridge, MA: Harvard University Press.
Burkhardt, H., & Schoenfeld, A. H. (2003). Improving educational research: Toward a more useful, more
influential, and better-funded enterprise. Educational Researcher, 32(9), 3–14.
Canham, M., & Hegarty, M. (2010). Effects of knowledge and display design on comprehension of complex
graphics. Learning and Instruction, 20, 155–166.
Corbett, A. T., & Anderson, J. R. (2001). Locus of feedback control in computer-based tutoring: Impact on
learning rate, achievement, and attitudes. In J. Jacko, A. Sears, M. Beaudouin-Lafon, & R. Jacob (Eds.),
Proceedings of ACM CHI’ 2001 conference on human factors in computing systems (pp. 245–252). New
York: ACM Press.
Coyan, J. A., Busch, M. M., & Reynolds, S. J. (2009). Examining student interactions with distractors in
photographs using eye-tracking technology. Geological Society of America Abstracts With Programs, 41
(7), 196.
Falkenhainer, B., Forbus, K. D., & Gentner, D. (1989). The structure-mapping engine: Algorithm and
examples. Artificial Intelligence, 41, 1–63.
Forbus, K. D., Chang, M. D., McLure, M., & Usher, M. (2017). The cognitive science of sketch worksheets.
Topics in Cognitive Science, 9(4), 921–942.
Forbus, K., Usher, J., Lovett, A., Lockwood, K., & Wetzel, J. (2011). CogSketch: Sketch understanding for
cognitive science research and for education. Topics in Cognitive Science, 3(4), 648–666.
Gadgil, S., Nokes-Malach, T. J., & Chi, M. T. (2012). Effectiveness of holistic mental model confrontation in
driving conceptual change. Learning and Instruction, 22(1), 47–61.
Gagnier, K. M., Atit, K., Shipley, T. F., Ormand, C., Manduca, C., & Tikoff, B. (2012). Improving
penetrative thinking skills for geoscience education. San Diego, CA: Presented at iSLC.
Gagnier, K. M., & Shipley, T. F. (2016). Visual completion from 2D cross-sections: Implications for visual
theory and STEM education and practice. Cognitive Research: Principles and Implications, 1(1), 1–18.
Gagnier, K. M., Shipley, T. F., Tikoff, B., Garnier, B., Ormand, C. J., Atit, K., & Resnick, I. (2016).
Training spatial skills in geosciences: A review of tests and tools. In R. W. Krantz, C. J. Ormand, & B.
Freeman (Eds.), 3D structural interpretation: Earth, mind, and machine: AAPG Memoir 111 (pp. 7–23).
Boulder, CO: AAPG.
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
967
Gentner, D., Levine, S. C., Ping, R., Isaia, A., Dhillon, S., Bradley, C., & Honke, G. (2015). Rapid learning
in a children’s museum via analogical comparison. Cognitive Science, 40, 224–240.
Gentner, D., & Smith, L. A. (2013). Analogical learning and reasoning. In D. Reisburg (Ed.), The Oxford
handbook of cognitive psychology (pp. 10–11). Oxford: Oxford University Press.
Gobert, J. D. (2005). The effects of different learning tasks on model-building in plate tectonics:
Diagramming versus explaining. Journal of Geoscience Education, 53(4), 444–455.
Gobert, J. D., & Clement, J. J. (1999). Effects of student-generated diagrams versus student-generated
summaries on conceptual understanding of causal and dynamic knowledge in plate tectonics. Journal of
Research in Science Teaching, 36(1), 39–53.
Goodwin, C. (1994). Professional vision. American Anthropologist, 96(3), 606–633.
Gordin, D. N., & Pea, R. D. (1995). Prospects for scientific visualization as an educational technology. The
Journal of the Learning Sciences, 4(3), 249–279.
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.
Hegarty, M., Keehner, M., Cohen, C., Montello, D. R., & Lippa, Y. (2007). The role of spatial cognition in
medicine: Applications for selecting and training professionals. In G. L. Allen (Ed.), Applied spatial
cognition: From research to cognitive technology (pp. 285–315). Mahwah, NJ: Lawrence Erlbaum
Associates.
Jee, B. D., Gentner, D., Forbus, K., Sageman, B., & Uttal, D. H. (2009). Drawing on experience: Use of
sketching to evaluate knowledge of spatial scientific concepts. In N. A. Taatgen & van Rijn H. (Eds.),
Proceedings of the 31st Annual Conference of the Cognitive Science Society 2009 (pp. 2499–2504). New
York: Curran Associates.
Jee, B. D., Sageman, B., Manduca, C. A., & Shipley, T. F. (2014). Drawing on experience: How domain
knowledge is reflected in sketches of scientific structures and processes. Research for Science Education,
44, 859–883.
Jee, B. D., Uttal, D. H., Gentner, D., Manduca, C., Shipley, T. F., Tikoff, B., Ormand, C. J., & Sageman, B.
(2010). Commentary: Analogical thinking in geoscience education. Journal of Geoscience Education, 58
(1), 2–13.
Johnson, J., & Reynolds, S. J. (2005). Concept sketches- using student- and instructor-generated, annotated
sketches for learning, teach, and assessment in geology courses. Journal of Geoscience Education, 53(1),
85–95.
Kali, Y., & Orion, N. (1996). Spatial abilities of high-school students in the perception of geologic structures.
Journal of Research in Science Teaching, 33(4), 369–391.
Kastens, K. A., & Ishikawa, T. (2006). Spatial thinking in the geosciences and cognitive sciences: A crossdisciplinary look at the intersection of the two fields. In C. A. Manduca & D. Mogk (Eds.), Earth and
mind: How geologists think and learn about the earth: Geological Society of America Special Paper 413
(pp. 51–74). Boulder, CO: Geological Society of America.
Kulik, J. A., & Kulik, C. C. (1988). Timing of feedback and verbal learning. Review of Educational
Research, 58(1), 79–97.
Libarkin, J. C., Anderson, S. W., Dahl, J., Beilfuss, M., & Boone, W. (2005). Qualitative analysis of college
students’ ideas about the earth: Interviews and open-ended questionnaires. Journal of Geoscience
Education, 53(1), 17–26.
Libarkin, J. C., Kurdziel, J. P., & Anderson, S. W. (2007). College student conceptions of geological time
and the disconnect between ordering and scale. Journal of Geoscience Education, 55(5), 413–422.
Liben, L. S., & Titus, S. J. (2012). The importance of spatial thinking for geoscience education: Insights
from the crossroads of geoscience and cognitive science. In K. A. Kastens & C. A. Manduca (Eds.), Earth
and mind II: A Synthesis of research on thinking and learning in the geosciences: Geological Society of
America Special Paper 486 (pp. 51–70). Boulder, CO: Geological Society of America.
Loewenstein, J., Thompson, L., & Gentner, D. (1999). Analogical encoding facilitates knowledge transfer in
negotiation. Psychonomic Bulletin & Review, 6(4), 586–597.
968
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
Magidson, S. (2002). Teaching, research, and instructional design: Bridging communities in mathematics
education (doctoral dissertation). Retrieved from ProQuest database (Order No. 3063466).
Manduca, C. A., & Kastens, K. A. (2012). Geoscience and geoscientists: Uniquely equipped to study Earth.
In K. A. Kastens & C. A. Manduca (Eds.), Earth and mind II: A synthesis of research on thinking and
learning in the geosciences: Geological Society of America Special Paper 486 (pp. 1–12). Boulder, CO:
Geological Society of America.
Marshak, S. (2012). Essentials of geology. New York, NY: W. W. Norton.
Morris, A. K., & Hiebert, J. (2011). Creating shared instructional products: An alternative approach to
improving teaching. Educational Researcher, 40(1), 5–14.
Newcombe, N. S., & Shipley, T. F. (2015). Thinking about Spatial Thinking: New Typology, New
Assessments. In J. S. Gero (Ed.), Studying visual and spatial reasoning for design creativity (pp. 179–
192). Amsterdam: Springer.
Novack, M. A., Congdon, E. L., Hemani-Lopez, N., & Goldin-Meadow, S. (2014). From action to
abstraction: Using the hands to learn math. Psychological Science, 25(4), 903–910.
Ormand, C. J., Manduca, C., Shipley, T. F., Tikoff, B., Harwood, C. L., Atit, K., & Boone, A. P. (2014).
Evaluating geoscience students’ spatial thinking skills in a multi-institutional classroom study. Journal of
Geoscience Education, 62, 146–154.
Petcovic, H. L., Ormand, C. J., & Krantz, R. W. (2016). Earth, mind, and paper: Field sketches as expert
representations of the hat creek fault zone. In R. W. Krantz, C. J. Ormand, & B. Freeman (Eds.), 3D
structural interpretation: Earth, mind, and machine: AAPG Memoir 111 (pp. 173–189). Boulder, CO:
AAPG.
Rapp, D. N., Culpeppers, S. A., Kirkby, K., & Morin, P. (2007). Fostering students’ comprehension of
topographic maps. Journal of Geoscience Education, 55(1), 5–16.
Resnick, I., Atit, K., & Shipley, T. F. (2012). Teaching geologic events to understand geologic time. In K. A.
Kastens & C. A. Manduca (Eds.), Earth and mind II: A synthesis of research on thinking and learning in
the geosciences: Geological Society of America Special Paper 486 (pp. 41–43). Boulder, CO: Geological
Society of America.
Resnick, I., Davatzes, A., Newcombe, N. S., & Shipley, T. F. (2016). Using relational reasoning to learn
about scientific phenomena at unfamiliar scales (special issue). Educational Psychology Review, 9(1), 1–
15.
Resnick, I., & Shipley, T. F. (2013). Breaking new ground in the mind: An initial study of mental brittle
transformation and mental rigid rotation in science experts. Cognitive Processing, 14(2), 143–152.
Resnick, I., Shipley, T. F., Newcombe, N., Massey, C., & Wills, T. (2012). Examining the representation and
understanding of large magnitudes using the hierarchical alignment model of analogical reasoning. In N.
Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Cognitive Science Society
(pp. 917–922). Austin, TX: Cognitive Science Society.
Reynolds, S. J., Johnson, J., Morin, P., & Carter, C. (2012). Exploring geology. New York, NY: McGrawHill.
Reynolds, S. J., Piburn, M. D., Leedy, D. E., McAuliffe, C. M., Birk, J. P., & Johnson, J. K. (2006). The
hidden earth – Interactive, computer-based modules for geoscience learning. In C. A. Manduca & D.
Mogk (Eds.), Earth and mind: How geologists think and learn about the earth: Geological Society of
America Special Paper 413 (pp. 157–170). Boulder, CO: Geological Society of America.
Schlaisich, I., & Egenhofer, M. J. (2001). Multimodal spatial querying: What people sketch and talk about.
In 1st international conference on universal access in human-computer interaction (pp. 732–736).
Mahwah, NJ: Lawrence Erlbaum.
Shipley, T. F., Tikoff, B., Ormand, C., & Manduca, C. (2013). Structural geology practice and learning, from
the perspective of cognitive science. Journal of Structural Geology, 54, 72–84.
Sorby, S. (2009). Educational research in developing 3-D spatial skills for engineering students. International
Journal of Science Education, 31(3), 459–480.
B. Garnier et al. / Topics in Cognitive Science 9 (2017)
969
Taele, P., Barreto, L., & Hammond, T. (2015). Maestoso: An intelligent educational sketching tool for
learning music theory. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence
(AAAI) (pp. 3999–4005). Austin, TX: AAAI Press.
Titus, S., & Horsman, E. (2009). Characterizing and improving spatial visualization skills. Journal of
Geoscience Education, 57(4), 242–254.
Trend, R. (1998). An investigation into understanding of geological time among 10- and 11-year-old
children. International Journal of Science Education, 20, 973–988.
Trend, R. (2000). Conceptions of geological time among primary teacher trainees, with reference to their
engagement with geoscience, history, and science. International Journal of Science Education, 22, 539–
555.
Uttal, D. H., Miller, D. I., & Newcombe, N. S. (2013). Exploring and enhancing spatial thinking: Links to
achievement in science, technology, engineering, and mathematics? Current Directions in Psychological
Science, 22(5), 367–373.
Valentine, S., Vides, F., Lucchese, G., Turner, D., Kim, H.-H., Li, W., Linsey, J., & Hammond, T. (2012).
Mechanix: A sketch-based tutoring and grading system for free-body diagrams. AI Magazine, 34(1), 55–
66.
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other
tutoring systems. Educational Psychologist, 46(4), 197–221.
Xu, Y., & Franconeri, S. L. (2015). Capacity for visual features in mental rotation. Psychological Science, 26
(8), 1241–1251.
Yin, P., Forbus, K. D., Usher, J., Sageman, B., & Jee, B. D. (2010). Sketch worksheets: A sketch-based
educational software system. In Proceedings of the 22nd annual conference on Innovative Applications of
Artificial Intelligence (IAAI) (pp. 10–11). Austin, TX: AAAI Press.