Topics in Cognitive Science 9 (2017) 943–969 Copyright © 2017 Cognitive Science Society, Inc. All rights reserved. ISSN:1756-8757 print / 1756-8765 online DOI: 10.1111/tops.12291 This article is part of the topic “Sketching and Cognition,” Kenneth Forbus and Shaaron Ainsworth (Topic Editors). For a full listing of topic papers, see: http://onlinelibrary.wiley. com/doi/10.1111/tops.2017.9.issue-4/issuetoc Promoting Sketching in Introductory Geoscience Courses: CogSketch Geoscience Worksheets Bridget Garnier,a Maria Chang,b Carol Ormand,a,c Bryan Matlen,d Basil Tikoff,a Thomas F. Shipleye a Department of Geoscience, University of Wisconsin-Madison b T.J. Watson Research Center, IBM c Science and Education Resource Center, Carleton College d STEM Program, WestEd e Department of Psychology, Temple University Received 27 April 2015; received in revised form 24 March 2016; accepted 26 September 2016 Abstract Research from cognitive science and geoscience education has shown that sketching can improve spatial thinking skills and facilitate solving spatially complex problems. Yet sketching is rarely implemented in introductory geosciences courses, due to time needed to grade sketches and lack of materials that incorporate cognitive science research. Here, we report a design-centered, collaborative effort, between geoscientists, cognitive scientists, and artificial intelligence (AI) researchers, to characterize spatial learning challenges in geoscience and to design sketch activities that use a sketch-understanding program, CogSketch. We developed 26 CogSketch worksheets that use cognitive science–based principles to scaffold problem solving of spatially complex geoscience problems and report observations of an implementation in an introductory geoscience course where students used CogSketch or humangraded paper worksheets. Overall, this research highlights the principles of interdisciplinary design between cognitive scientists, geoscientists, and AI researchers that can inform the collaborative design process for others aiming to develop effective educational materials. Keywords: Geoscience; Sketching feedback; Science education; Spatial thinking; Sketching tutor; Introductory courses; Science of design Correspondence should be sent to Bridget Garnier, Department of Geoscience, University of WisconsinMadison, 1215 West Dayton St., Room 273, Madison, WI 53706. E-mail: bridget@wisc.edu 944 B. Garnier et al. / Topics in Cognitive Science 9 (2017) 1. Introduction Geoscience is a spatially demanding field as many disciplinary tasks involve complex spatial relations. Spatial reasoning challenges include inferring interior 3-D structures from surface patterns, inferring rigid and non-rigid changes over time, relating features at different spatial scales, and reasoning about past events from current spatial patterns (Kastens & Ishikawa, 2006; Manduca & Kastens, 2012). Therefore, beginning geoscience students must develop the skills in their undergraduate careers to handle these spatial reasoning challenges and educators are expected to facilitate this development despite a wide variability in their students’ spatial abilities (Ormand et al., 2014). Sketching has been shown to be an effective way to improve spatial thinking skills (both general spatial thinking skills, as well as those needed for domain specific tasks) and support the formation and communication of complex spatial concepts in STEM (e.g., Ainsworth, Prain, & Tytler, 2011; Gagnier et al., 2012, 2016; Gobert, 2005; Gobert & Clement, 1999; Johnson & Reynolds, 2005; Liben & Titus, 2012; Rapp, Culpeppers, Kirkby, & Morin, 2007; Resnick, Atit, & Shipley, 2012; Resnick, Shipley, Newcombe, Massey, & Wills, 2012; Sorby, 2009; Titus & Horsman, 2009). In the geosciences, sketching is one of the most widely used tools among experts and has been found to support the teaching of difficult concepts (e.g., Johnson & Reynolds, 2005). Thus, students in introductory courses have much to gain from sketching, both in the form of increased mastery of disciplinary geoscience content and development of skills that support more general spatial thinking (Gagnier et al., 2012; Kali & Orion, 1996; Ormand et al., 2014; Titus & Horsman, 2009). However, we have noted that sketching is rarely implemented in introductory courses. Below we summarize the results of a small survey that suggests instructors do believe that sketching would support learning, but the primary obstacle to including sketching in introductory courses appears to be the time it takes for instructors to grade and provide feedback on sketches. Providing opportunities for students to learn from sketching without excessive demands on an instructor’s time requires designing a new approach—one that draws on research in cognitive science on artificial intelligence (AI), spatial learning, and sketching. However, it is unclear how to best use cognitive science theory to specifically address the practical problems faced by geoscience educators—a common problem in educational research (Burkhardt & Schoenfeld, 2003; Magidson, 2002). Drawing from recent work on design-based research, we report the development of a problem-centered interdisciplinary research project that places the educator and her/his classroom at the center of a rapid prototyping process with co-development of materials and concepts among educators and researchers (Bryk, 2009; Bryk & Gomez, 2008; Bryk, Gomez, Grunow, & LeMahieu, 2015; Morris & Hiebert, 2011). This is a Design, Educational Engineering, and Development (D-EE-D) approach that combines cognitive science research B. Garnier et al. / Topics in Cognitive Science 9 (2017) 945 and sketching to support spatial skills and disciplinary concept development in geoscience students. This paper reports on a collaborative effort to characterize the spatial learning problem and design sketch activities that use a sketch-understanding program with a built-in tutor (CogSketch). This work is in the early, yet research-and time-intensive step, in the D-EE-D process. Our goals were to identify the challenges to using sketching in a classroom, review the interdisciplinary research needed to address the problem, develop a process for geoscience educators to generate sketch activities, and test the design by implementing sketching in a large introductory geoscience course. Thus, this initial design work should serve as a conceptual framework for other teams to work on developing sketch-based support for learning in other spatially complex fields. 2. Challenges of implementing sketching in the classroom Although sketching helps students increase problem solving and spatial thinking skills, no pedagogical materials based on cognitive science research, such as sketching-based workbooks, are currently available from textbook publishers for introductory-level geoscience students. To better understand why this is the case, despite the long history of sketching in the discipline, we conducted a survey of introductory geoscience instructors to document the use of sketching and attitudes towards sketching in undergraduate geoscience classrooms. Seventy-two introductory geoscience instructors (49 from 2-year institutions and 23 from 4-year institutions) from undergraduate institutions across the United States were surveyed. The survey likely self-selected geoscience educators who are specifically interested in geoscience education, and it is possible such instructors were aware of the value of sketching as an effective learning activity. Thus, the results may well overestimate the use of sketching in U.S. geosciences classes. Instructors were asked a combination of Likert (using a scale from 1 to 5) and open-ended questions about whether and how they implemented sketching in their classrooms, how feedback was provided, time spent grading sketches, and their beliefs about the importance of sketching. We found that while over 80% of instructors believed sketching is important for understanding geoscience concepts, 52% assign three or fewer sketching assignments during a semester in their course. Of the instructors in the survey, 77% spent up to 2 min per sketch on grading, and the remaining instructors spent more than 3 min grading each sketch. At those rates, grading sketches for large-enrollment courses requires a minimum of several hours per sketch assignment, even for the fastest graders. We found that 90% of graded sketches are assigned for exams, quizzes, or lab assignments. Thus, results suggest that while introductory geoscience instructors view sketching as important, it can be a time-consuming activity for instructors and is often limited to summative assessment. 946 B. Garnier et al. / Topics in Cognitive Science 9 (2017) 3. Drawing from cognitive psychology and computer science 3.1. Spatial thinking skills in the geosciences To constrain the design problem, we focused on developing cognitive science–based support for four classes of spatial reasoning skills common to geoscience sketches. The selection for these four classes for prioritization was based on (a) previous work on the range of spatial skills applied to geoscience (Kastens & Ishikawa, 2006; Shipley, Tikoff, Ormand, & Manduca, 2013); (b) identification of the spatial skills necessary for critical disciplinary content included in an introductory geology course, guided by research group members with expertise in geoscience and teaching introductory geoscience; and (c) spatial reasoning problems with some prior basic research to characterize the learning challenges, including the known range of spatial reasoning skills in geoscience courses (Ormand et al., 2014). Four broad spatial skills were identified that are important for success in the geosciences: disembedding, reasoning about dynamic processes, penetrative thinking, and scaling. For each skill, we focus on developing sketching exercises to support the understanding of disciplinary content with complex spatial relations and perhaps support development of the underlying cognitive processes (Titus & Horsman, 2009; Uttal, Miller, & Newcombe, 2013). We note that these four spatial skills are important throughout science, technology, engineering, and math (STEM) disciplines, and they are not solely applicable to geoscience (Newcombe & Shipley, 2015). Disembedding is the ability to identify and focus on important information from a vast array of information (Shipley et al., 2013). Expert geoscientists are able to walk up to an outcrop, ignore irrelevant features such as vegetation and rubble, and identify key geologic features, while novices often focus on non-relevant objects (Canham & Hegarty, 2010; Coyan, Busch, & Reynolds, 2009; Goodwin, 1994; Jee, Gentner, Forbus, Sageman, & Uttal, 2009; Jee, Sageman, Manduca, & Shipley, 2014; Manduca & Kastens, 2012; Petcovic, Ormand, & Krantz, 2016; Reynolds et al., 2006). Expert geoscientists often use disembedding during fieldwork, looking at diagrams, microscope work, and many other activities. Reasoning about dynamic processes includes all spatial reasoning that involves movement or change. Geologists are skilled in reasoning about a wide range of transformations over time, including both brittle deformation (e.g., breaking) (Resnick & Shipley, 2013) and plastic deformation (e.g., bending) (Atit, Shipley, & Tikoff, 2014), and recognizing causal relationships (Jee et al., 2009, 2014). For the most part, changes caused by geological processes cannot be observed directly, because the changes that happen during a human lifetime are insignificant, but over geologic time the cumulative effect of those changes is profound. In most cases, therefore, the processes must be inferred from the traces left behind, in the form of spatial patterns. Penetrative thinking is the ability to use visible surface information to determine interior spatial structures (Kali & Orion, 1996). Expert geologists use this skill to create B. Garnier et al. / Topics in Cognitive Science 9 (2017) 947 models of Earth’s subsurface given sparse surface and subsurface data (Reynolds et al., 2006), while medical professional need to create mental models of the human body (Hegarty, Keehner, Cohen, Montello, & Lippa, 2007). Students’ skills can range from not recognizing that patterns on the outside of a volume can be used to infer interior structures, to a sophisticated ability to integrate patterns on multiple surfaces to infer a complex 3D geometry (Kali & Orion, 1996). Introductory textbooks include many diagrams that require this skill (Atit, Gagnier, & Shipley, 2015, reports an estimate of 17% of all diagrams in the most widely adopted textbooks required this skill). Sketching with feedback has been found to improve reasoning with these diagrams (Gagnier et al., 2016). Scaling is the skill required to understand and visualize very large and very small magnitudes, and to reason about relationships across different magnitudes. Geoscientists continually move between different spatial scales (e.g., moving from the atomic structure of minerals in a hand sample to their implication for regional geology) to solve problems (Kastens & Ishikawa, 2006; Manduca & Kastens, 2012). While students can generally correctly order geological temporal and spatial magnitudes, they have difficulty reasoning about these scales (Libarkin, Anderson, Dahl, Beilfuss, & Boone, 2005; Libarkin, Kurdziel, & Anderson, 2007; Trend, 1998, 2000). Recent research has shown that multiple opportunities to sketch relationships across scales can improve scaling (Resnick, Davatzes, Newcombe, & Shipley, 2016). 3.2. Sketch-based educational software If a primary barrier to employing sketching in a classroom is the lack of instructor time to provide effective feedback, then any solution must minimize instructor time and any time requirements would ideally be independent of the number of students. One obvious class of such solutions is intelligent tutoring systems. AI-based tutoring systems have been developed for many domains and, with the right tutoring strategies, can be as effective as human tutors (VanLehn, 2011). However, for a highly spatial domain like geoscience, one of the main technical challenges in building intelligent educational software is understanding the spatial information in a student’s input. Sketching is a natural way to input spatial information, but having the tutor interpret the spatial information accurately is a challenge. Artificial intelligence researchers have explored techniques for interpreting sketches, including the use of highly specialized sketch recognition algorithms (e.g., Taele, Barreto, & Hammond, 2015; Valentine et al., 2012) and/or cognitive models of sketch understanding (Forbus, Usher, Lovett, Lockwood, & Wetzel, 2011). Sketch recognition works well for domains where there is a well-defined mapping between symbols and concepts. However, many of the sketching exercises in geoscience use symbols that can represent different concepts in different contexts. For example, depending on where it is located in the image, a straight line could represent a fault, a stratigraphic contact, the water table, or some other geological feature. An algorithm that has perfect accuracy in identifying straight lines would not be enough to understand the geoscientist’s sketch. 948 B. Garnier et al. / Topics in Cognitive Science 9 (2017) An alternative approach to sketch understanding has been implemented in CogSketch (Forbus et al., 2011), which uses qualitative spatial representations to capture the content of sketches. CogSketch circumvents the issue of having symbols with many potential meanings by having users enter conceptual labels for the things they draw. This mimics the way people often speak or provide text labels for things when they sketch (Schlaisich & Egenhofer, 2001). When someone sketches something into CogSketch, he or she draws glyphs, discrete parcels of ink and content, with a mouse or stylus, and labels the items that he or she draws. Labels can indicate that a glyph represents something that belongs to a particular category (e.g., a fault) or that a glyph represents something more abstract, like an arrow to indicate movement or a relationship. CogSketch automatically computes qualitative spatial relationships between the glyphs to support sketch understanding; for example, the sandstone is above the shale. Sketch worksheets (Yin, Forbus, Usher, Sageman, & Jee, 2010) are a domain-general education application built within CogSketch that we used to create geoscience exercises. The general purpose of sketch worksheets is to provide a platform for instructors to develop their own sketching exercises. It works by having an author draw a solution to their exercise and identify important parts of the solution. The author defines feedback prompts that are shown to the student if any of those important parts are missing or incorrect in the student’s sketch. The author can also define grading rubrics so that the software can automatically produce an accuracy score for any sketch. While working on a worksheet, students can click a Feedback button at any time to receive advice about their sketch (Fig. 1). The Feedback button signals CogSketch to compare the student’s sketch to the solution sketch created by the author. The comparison process identifies any important aspects that are missing or incorrect, and it provides students with the pre-written advice. Students can correct their sketch while revising according to the advice, continuing to click the Feedback button and correcting their sketch as many times as they want until they receive the feedback message: “Your sketch looks good to me!” As described above, sketch worksheets provide immediate feedback that is controlled by the user. Research focusing on the learning effects of immediate versus delayed feedback is complex and does not always show that one method is superior to the other (Kulik & Kulik, 1988). However, research that more closely applies to CogSketch suggests that immediate feedback should be an effective strategy. A metaanalysis by Kulik and Kulik (1988), incorporating 53 studies, found that immediate feedback was more effective in most included studies, particularly applied studies focusing on classroom quizzes, as well as studies with computer presentation of feedback. Delayed feedback was only found to be more powerful in special experimental situations. Another meta-analysis conducted by Azevedo and Bernard (1995) found that computer-presented feedback was more effective than no feedback and that immediate feedback provides the best instructional advantage. Additionally, a study using an intelligent computer tutor to help students complete computer-programming exercises found that instant, on-demand feedback helped students solve problems faster and more efficiently than those without feedback (Corbett & Anderson, 2001). Overall, this 949 B. Garnier et al. / Topics in Cognitive Science 9 (2017) Worksheet Image Worksheet Solution Pressure/Temperature Diagram 0 0 Temperature °C 400 600 200 Pressure/Temperature Diagram Temperature °C 800 1000 0 0 200 400 600 Hornfels 200 Zeolite 400 Pressure (MPa) Greenschist Pressure (MPa) 200 Amphibolite Student attempt 800 1000 1200 Granulite Blueschist 1000 Eclogite Eclogite 1200 Cycle of attempts and tutor Student Corrected Sketch feedback Pressure/Temperature Diagram Pressure/Temperature Diagram Temperature °C Temperature °C 400 600 800 Tutor Suggestions Pressure (MPa) 400 1000 Greenschist 200 400 600 800 Tutor Suggestions 1000 Hornfels 200 - Your drawing of the Subducting Plate Pathway isn’t quite right. You just need to connect the dots! Start at 0,0, connect the dots, and ontinue the line to the bottom. - Review your X,Y graphing skills! You should try Subducting Plate 4 again. Zeolite 400 Amphibolite 1000 Eclogite Hide missing entity suggestions? Greenschist Your sketch looks good to me! 600 All feedback corrected 800 Granulite Blueschist Update suggestions 1200 0 Suggestions for Sketch Meta_Facies Selecting a suggestion will highlight the associated items on the sketch. Zeolite 600 800 0 Suggestions for Sketch Meta_Facies Hornfels 200 1000 Pressure (MPa) 0 200 Amphibolite 800 Student Incorrect Attempt 0 Greenschist 400 600 Granulite Blueschist 1000 Zeolite Create solution 600 800 Hornfels Granulite Blueschist Amphibolite Eclogite Update suggestions 1200 Hide missing entity suggestions? Fig. 1. Flow chart of CogSketch worksheet completion process from the Metamorphic Facies Diagram worksheet. Top left: Author inserts a worksheet image that is used as a backdrop for a particular sketching task. In this case, students are graphing Pressure/Temperature points onto the Metamorphic Facies diagram. Top right: Author creates a solution sketch on top of the worksheet image. Lower left: The student sketches a solution to the problem and can request feedback from the software at any time. When the student requests feedback, the student’s sketch is compared to the solution sketch and advice (written by the worksheet author) is provided for any discrepancies. Lower right: The student continues to edit his or her sketch and press the Feedback button until there are zero discrepancies or until the student is satisfied with the sketch. research supports that immediate feedback provided by CogSketch may help students complete tasks more efficiently and help improve learning compared to students who do not receive feedback. 3.3. Sketch understanding requirements in geoscience A complete explanation of the representations and processes used by CogSketch and Sketch Worksheets can be found elsewhere (Forbus, Chang, McLure, & Usher, 2017; Forbus et al., 2011). Here, we focus on the capabilities that were required to allow CogSketch to effectively reason and provide feedback on geoscience sketches. 950 B. Garnier et al. / Topics in Cognitive Science 9 (2017) 3.3.1. Quantitative ink constraints Many of our planned CogSketch geoscience worksheets required image annotation. When the absolute location of ink is important for drawing a sketch accurately, we use quantitative ink constraints to define regions that should contain the ink. For example, in a worksheet where a student must identify faults in an outcrop, his ink needs to coincide with the actual location of the fault in the image. When the author draws the solution to the worksheet, she would define a quantitative ink constraint around her correct version of the fault line. The author sets a numerical tolerance, which is used to define a tolerance region for the student’s glyph. If the student’s glyph falls completely within the tolerance region, the constraint is satisfied. If not, the constraint is violated and the student receives advice about it (e.g., “Your fault line is not in the correct location”). The author may also define position-dependent advice prompts, to give the student more specific advice if there is a particular positional relation between the student’s glyph and the solution glyph (e.g., “Your fault line is too low”). 3.3.2. Arrow interpretation Arrows are used in CogSketch to convey both movement and processes. In order for CogSketch to understand arrows in human-like ways, their qualitative direction must be taken into account. Arrow interpretation identifies the head and tail of the arrow and computes a qualitative vector between the two. The qualitative vector is oriented in one of eight qualitative directions: up, down, left, right, and the diagonal quadrants (e.g., up and to the right, down and to the right, etc.). Carving the visual space into these eight qualitative directions, rather than using a quantitative representation like radians, is a cognitively plausible approach to arrow direction interpretation. 3.3.3. Grading Grading in sketch worksheets uses analogical comparison (i.e., the structure-mapping engine; Falkenhainer, Forbus, & Gentner, 1989) to detect important differences between the student sketch and the solution sketch (for more information on this process, see Forbus et al., 2011). In the same way that differences are detected for generating feedback, the differences found when students are finished with their worksheets are evaluated against author-defined rubrics. Points are awarded based on these rubrics and scores are automatically computed for each student worksheet. 4. Development of introductory geoscience worksheets Geologists, psychologists, and computer scientists collaborated to develop a series of 26 Introductory Geoscience CogSketch worksheets that focus on key geoscience concepts (Table 1). Our aim was to support students’ ability to learn complex spatial concepts and make sketching assignments practical to grade. To this aim, we identified important spatial concepts from each week of an introductory course. To ensure a broad range of spatial problems, we selected concepts that represented at least one of the four classes of B. Garnier et al. / Topics in Cognitive Science 9 (2017) 951 Table 1 Relationship between Geoscience topics and CogSketch worksheets Topics Covered in an Introductory Geoscience Course Plate tectonics Minerals Igneous rocks and processes Sediments and sedimentary rocks Metamorphic rocks and processes Structural geology Earthquakes Geologic time/biogeography Mass movement Streams and floods Oceans and coasts Groundwater Glaciers and glacial processes CogSketch Geoscience Worksheet Concepts Earth’s interior Earth’s magnetic field Magnetic reversals Mineral grain boundaries Mineral cleavage Classifying igneous rocks Effects of pressure and temperature on igneous rocks Stratigraphy Transgression and regression Metamorphic facies Metamorphic Rocks and fabrics in outcrops Identifying geologic structures in outcrops Strike/dip and block diagrams Topographic maps Fault identification Stick-slip faulting and earthquakes Geologic time scale Fossil succession Slope stability Flood recurrence Coastal processes Ocean currents Groundwater flow Water table contours and contamination Glacial movement Glacial geomorphology spatial thinking skills (disembedding, visualizing change, scaling, and penetrative thinking). Two to three target concepts were identified for thirteen topics commonly taught in introductory geoscience courses. Topics were identified, by the geoscience experts in our group, based on a review of commonly used Introductory Geoscience textbooks, particularly those textbooks where spatial concepts and diagrams are prevalent (i.e., Marshak, 2012; Reynolds, Johnson, Morin, & Carter, 2012). While each topic may not be present in every course or textbook, we chose the most common topics to ensure that a range of usable worksheets would be available for most introductory geoscience courses. Table 1 lists the 26 key concepts (right column, Table 1) along with their corresponding topics commonly found in geoscience textbooks (left column, Table 1). Each worksheet contains instructions for the student and a background image or diagram that students use to complete sketching tasks. In order to better understand the challenges and opportunities of an intelligent tutor for sketching, we created analogous paper-based 952 B. Garnier et al. / Topics in Cognitive Science 9 (2017) worksheets. Instructions for paper and CogSketch worksheets were conceptually the same (i.e., they always posed the same problem, described the same tasks, and the same goal), differing only in how they described drawing and manipulation steps to reflect the different media of paper versus computer. Notably, some CogSketch worksheets are completed by moving diagram elements, which is not possible to do with a single sheet of paper. Therefore, instructions were written to be as similar and specific as possible to ensure that student sketches would be consistent between the two formats. 4.1. Worksheet development To illustrate how a CogSketch worksheet was conceived and developed, we will review the Earth’s Magnetic Field worksheet in detail. 4.1.1. Identifying important geological concepts Target worksheet concepts (right column, Table 1) were identified by reviewing introductory geology textbooks, from the authors’ classroom experience, and through discussions with other instructors, teaching assistants, and undergraduate students. Concepts were selected as targets if they are challenging for students and sketching could be used to support students’ learning. The Earth’s Magnetic Field worksheet (Fig. 2) was specifically requested by an instructor, who noted that students have difficulty visualizing Earth’s magnetic field in three dimensions and understanding how the inclination of magnetic field lines project onto the surface of the Earth, despite the fact that the introductory textbook contained a figure illustrating this concept. 4.1.2. Creating an exercise that utilizes interactive attributes of CogSketch Once the concept was chosen, the authors determined the cognitive processes that were involved when reasoning about the concept and how the concept correlated to the four spatial reasoning problems. An exercise was then developed around activities that would support the relevant spatial thinking skills and processes. A challenge in developing worksheets using CogSketch was to move beyond paper and effectively utilize the interactive capabilities of CogSketch. Our intention in designing CogSketch worksheets was to maximize sketching, interacting with diagrams, moving objects around, and reading and responding to the worksheet tutor’s feedback. Thus, we expect students to play, interact, or use a trial-and-error approach to grasp the content and potentially correct common mistakes or misconceptions through an iterative process with the tutor. We propose that this interaction has the potential to enhance student understanding of the concepts and lead students to self-evaluate mistakes, which has been shown to be a strategy that can facilitate deeper learning (Bjork & Bjork, 2011). The goals of the Earth’s Magnetic Field worksheet (Fig. 2) are for students to see how magnetic inclination changes with latitude and to understand how geologists use that information to make inferences about a rock’s historic location. Thus, this worksheet is designed to support students who are having difficulty with the dynamic reasoning necessary to accurately infer movements over time. B. Garnier et al. / Topics in Cognitive Science 9 (2017) Rotation Axis 953 Magnetic Pole Geologic units with magnetic orientation: Move and rotate to correct red dot Example N S Students move and rotate each rock to the correct place, matching magnetic field lines Fig. 2. Earth’s Magnetic Field worksheet. This worksheet consists of a diagram of the Earth’s magnetic dipole and the associated magnetic field. Students complete the worksheet by moving and rotating the packages of rock to a point where the arrow in the rock, its magnetic inclination, matches the magnetic inclination of the Earth. The red line on each “rock” indicates the top of the rock to help students correctly orient the boxes. The red outlined example is given to students, while the black dashed lines illustrate two additional examples for the reader. Geologists use the orientation of magnetic minerals in a rock to understand the change in latitude that has occurred since the rock was formed (Marshak, 2012). Different analyses can determine the three-dimensional orientation of magnetic minerals in a rock (like a pencil in a particular orientation in space). A rock’s magnetic orientation can be represented numerically, but it is much more informative to visualize the orientation, as well as the comparison to the Earth’s magnetic field (Gordin & Pea, 1995). The Earth’s Magnetic Field worksheet involves a simplified task that mimics the expert process by having students match the magnetic inclination of a “rock” with the latitude that has the same magnetic inclination. The worksheet consists of a diagram of the Earth and its magnetic field, an explanation with the key features of the diagram, and a task designed to help students understand how rocks can capture the orientation and inclination of Earth’s magnetic field as a rock cools, and that orientation can be used to deduce where the rock was when it cooled. The diagram has “packages” of rock that contain magnetic orientation (arrows). Students grab, rotate, and drag each package of rock to the location on the diagram where the magnetic arrow in the rock matches the inclination of Earth’s magnetic field (action indicated with a red arrow in Fig. 2). Note, in this task, matching and 954 B. Garnier et al. / Topics in Cognitive Science 9 (2017) comparing objects to a diagram is a form of analogical reasoning; analogies are a powerful tool for teaching a new concept using what is known about a familiar concept (i.e., Gentner & Smith, 2013; Jee et al., 2010). Students will succeed in this task if they notice how the magnetic inclination of an individual rock can be related to a latitudinal position on the Earth and how inclination changes with latitude. The process of moving and rotating objects in CogSketch allows students to match the rocks to the Earth’s magnetic field (a task that geologists do mentally) without attempting to visualize the entire diagram while mentally rotating an object, which is a difficult mental task (Xu & Franconeri, 2015). Lastly, instead of creating the analogy of the magnetic inclination of the rock to the diagram in a student’s mind, students physically align the two objects. This is a literal analogy as rocks do contain magnetic particles that you can physically rotate back to the current magnetic field. Therefore, the exercise incorporates both geologic concepts and analogical reasoning as structured and supported by CogSketch, to increase the potential to learn from a homework exercise. We should note that while the task of moving and rotating objects can make a spatial task easier for students to visualize and complete, it also increases the chance of students completing the worksheet using a strategy that is not connected to the geologic content, such as just focusing on aligning the arrows and the magnetic field lines. In each worksheet, we try to discourage these types of strategies by focusing the directions and feedback messages on the content and spatial relationships rather than just the physical task. However, future work should consider focusing on identifying when and if students are using “gaming” strategies to complete the worksheets, rather than focusing on the content (Baker, Corbett, & Koedinger, 2004). 4.1.3. Creating the final CogSketch worksheet In the worksheet development process, an author uploads background images and objects, inputs written directions for the student, creates the solution sketch, identifies important quantitative and qualitative facts about the solution sketch, and constructs the necessary feedback messages for each important fact. Once all the important features about the solution sketch are identified, the author can set the grading rubric and protect the solution with a password. The password prevents the student from looking at the solution, but it allows the solution to be used by the program and new instructor-users. The worksheet editor interface was designed to be used by a non-expert. The first author created all worksheets for this project and is a geologist with no experience in computer programing. We estimate that <2 days were necessary to obtain sufficient proficiency to construct a functional domain worksheet from a textbook diagram. Most of the time required to complete the authoring process is spent testing the worksheet and exploring the various possible solutions that a student might submit. This process often reveals situations where feedback could be more or less specific and gives the author the opportunity to fine-tune the ideal solution and feedback messages. Besides moving and rotating objects, other CogSketch geoscience worksheets made use of three notable tools in CogSketch: B. Garnier et al. / Topics in Cognitive Science 9 (2017) 955 1. Drawing and labeling. The draw tool allows students to sketch by free hand and then label each drawn object. This supports worksheet tasks that involve drawing scale bars, annotating features on a photo or chart, graphing, completing diagrams, drawing objects with certain spatial relations, and many other tasks. Drawing and labeling may help facilitate disembedding by drawing students’ attentions to features as they trace and annotate features of interest in photos, as well as diagrams and data graphs (Johnson & Reynolds, 2005). Penetrative thinking tasks also require drawing and labeling since students are often sketching features that are not directly visible. 2. Resizing. Students can resize (and move) any objects they draw in CogSketch. Any objects provided by the worksheet author can also be resized and moved by the student, unless the worksheet author applies editing locks on them. Resizing is useful when working on scaling tasks since students can directly scale objects instead of re-drawing them. 3. Drawing arrows. CogSketch’s arrow interpretation identifies the qualitative direction of hand-sketched arrows (e.g., to the right, to the left, pointing from A to B). Drawing arrows is useful for incorporating reasoning about motion into worksheets and conveying movement, feedback loops, and interaction between objects. 5. Geoscience classroom implementation To evaluate the prototype worksheets, CogSketch worksheets were used in the introductory Physical Geology course at the University of Wisconsin-Madison in the spring semester of 2014. Two sections of Physical Geology participated in this implementation, taught by professors who are colleagues of the authors, with a total of 262 participating students. The goal of this activity was to better understand the challenges and requirements of implementing a sketch-based tutoring system in an authentic STEM learning environment, as well as promote the understanding of the cognitive science principles that are needed to support learning by a sketch-based tutoring system. The longer-term goal was to develop a worksheet tutor that could provide instructor-quality feedback on sketching assignments, and use the tutor to support student sketching in large-enrollment courses. To better understand the design challenges and opportunities of the worksheet tutor for sketching, we contrasted implementation of CogSketch worksheets with a matched set of paper-based worksheets. Over the course of the semester, students completed either all CogSketch worksheets or all paper worksheets as homework assignments. Paper worksheets contained the same images and tasks as the CogSketch worksheets, adapted as needed for the medium (e.g., redrawing a line on the paper worksheet instead of dragging a line across the screen). Students were initially assigned to use either paper or CogSketch worksheets but were allowed to self-select into the other group if they preferred to do so. Many students opted out of the CogSketch group and into the paper group. Students typically opted out of the CogSketch group because they did not own a Windows machine (which is currently the 956 B. Garnier et al. / Topics in Cognitive Science 9 (2017) only OS that can run CogSketch), they did not want to have to use an on-campus lab to complete their assignments, or because they were reluctant to use a new computer program. In sum, 65 students completed CogSketch worksheets and 197 students completed paper worksheets. CogSketch worksheets were downloaded from the course website and students either downloaded CogSketch onto their personal computers or used public computers at UW-Madison that had had CogSketch preloaded. Each section of the course was taught independently by one of two professors who selected different sets of worksheets. Instructors chose worksheets from those presented in Table 1 and assigned each worksheet to match lecture content. In total, instructors chose 16 worksheets to assign in their classes. One professor assigned nine worksheets, and the other professor assigned eleven. Of those, four worksheets overlapped between the two sections (Table 2). Students completing CogSketch worksheets received immediate feedback and all completed worksheets were emailed to the first author. Paper worksheets were turned in during lecture, graded by the first author and provided with feedback comments similar to those that would appear in CogSketch, and then returned the following week in discussion sections. 5.1. Implementation observations Students in the CogSketch group quickly learned how to use the CogSketch program and completed their first worksheet during the first 2 weeks of class. Students completed all CogSketch worksheets on their own time, without a CogSketch or geoscience expert present, and about 85% of students turned in all assignments during the course of the semester. Paper worksheets were turned in at a similar rate (87% of students turned in all assignments). Overall, students had similar worksheet grades on CogSketch and paper worksheets. CogSketch worksheet scores were not perfect, as some students did not request feedback, and for others the feedback was not effective in correcting the error. When comparing completed CogSketch and paper worksheets, we highlight four important observations: (a) There were substantial individual differences in the ways students used the worksheet tutor; (b) Cogsketch worksheets helped identify and correct common mistakes; and (c) almost no instructor time was needed to grade and provide Table 2 Worksheets assigned to each instructor’s class and worksheets assigned to both classes Assigned Worksheets Assigned Only by Prof A Assigned by Both Assigned Only by Prof B Mineral Grain Boundaries Classifying Igneous Rocks Fossil Succession Groundwater Flow Transgression/Regression Geologic Time Metamorphic Facies Strike/Dip & Block Diagrams Water Table Contours & Contamination Earth’s Interior Earth’s Magnetic Field Mineral Cleavage Effects of P&T on Igneous Rocks Streams & Floods Glacial Geomorphology Stick-Slip Faulting & Earthquakes B. Garnier et al. / Topics in Cognitive Science 9 (2017) 957 individual feedback on CogSketch worksheets—the large majority of instructor time investment in using CogSketch worksheets is in the design phase. Each observation is discussed in greater detail below. 5.1.1. Individual differences in worksheet tutor interactions Students requested feedback from the worksheet tutor on 89% of worksheets. For all worksheets where the tutor was utilized, nearly 50% requested feedback 1–3 times per worksheet with only 6% requesting feedback 20 or more times (Fig. 3). While a student completes a worksheet, CogSketch records all actions that occur (e.g., creating an object, rotating an object, requesting feedback, etc.). We compiled this information to analyze how students completed each worksheet. We noticed that students used the worksheet tutor in a variety of ways: 1. 11% of students never requested feedback and simply completed the worksheet and turned it in, treating it the same way one would a normal homework assignment. The average worksheet grade for this group was 52%. 2. 37% of students completed all or most of the worksheet (leaving no more than two tasks incomplete), and then requested feedback, after which they corrected mistakes or chose to do nothing. The average worksheet grade was 89%. 3. The remaining 52% of students requested feedback throughout the worksheet completion process (these students averaged eight feedback requests per worksheet). For this strategy, the average worksheet grade was 82%. Though descriptive, these findings suggest that the tutor feedback holds promise for supporting students’ geoscience understanding, and that the program allows individuals to complete the worksheet using different strategies to arrive at the correct answer. Future work can experimentally examine the factors that influence student use of the worksheet Fig. 3. Histogram of feedback requests for all worksheets used in our implementation. Students requested feedback 1–3 times on ~50% of worksheets and requested feedback 20+ times on only 6% of worksheets. 958 B. Garnier et al. / Topics in Cognitive Science 9 (2017) tutor, the impact of tutor feedback on student learning, as well as the robustness of students’ knowledge when using these different strategies in CogSketch. Requesting feedback generally helped students correct their sketches (an example of incorrect and corrected sketches is shown in Fig. 4). The feedback could guide students to the feature of the sketch that was in error, so they could reconsider and correct the spatial location. After feedback, the student often redrew the lines, completing a correct sketch. This is a type of desired difficulty (creating some difficulty for the student to encourage deeper learning) that causes students to reevaluate the task or process and use corrective feedback to address incorrect interpretations (Bjork & Bjork, 2011; Hattie & Timperley, 2007). However, in some instances, students requested feedback and redrew a feature multiple times without drawing it correctly. Multiple factors may have attributed to this outcome, including: (a) student error in labeling objects correctly, which resulted in non-helpful feedback from the tutor; (b) feedback may have directed attention to the spatial relationship that was important, but that alone did not help student correct their sketch; (c) feedback message did not always provide information about the spatial nature of the error; and (d) student needed more information rather than corrective feedback (Hattie & Timperley, 2007). When the feedback did not help the students correct the object, then they kept redrawing the object until they happened upon the correct answer, or they chose to turn in the assignment with an incorrect sketch. A record of these interactions provides authors with an opportunity to revise the feedback to help future students who have similar problems. One such example, from the Groundwater Flow worksheet, occurred when many students had incorrect sketches that would have been graded as correct if hand graded. In this case, the ink tolerance was too strict and any bumps or jogs in a drawn line would count as out of bounds and thus wrong. The worksheet was revised by increasing the ink tolerance and testing the revised version against the worksheets originally graded as incorrect. INCORRECT 222 Poorly maintained chicken farm 219 216 219 214 216 208 209 213 leaking septic tank 205 208 Poorly maintained chicken farm 219 211 212 CORRECT 222 206 Feedback message: The boundaries aren’t quite right. The 2 lines need to be perpendicular to the contour lines (parallel to your flow arrow) and should extend from the outermost boundaries of the chicken farm. Continue the lines to the edge of the image. 216 219 211 216 208 209 213 leaking septic tank 205 212 208 201 207 214 206 201 207 204 202 199 204 202 199 Fig. 4. Corrected CogSketch sketches. A student’s Groundwater Contamination sketch before and after feedback. Student did not have lines in correct locations but redrew lines in the correct location once receiving the feedback message. B. Garnier et al. / Topics in Cognitive Science 9 (2017) 959 5.1.2. CogSketch worksheets helped identify and correct common mistakes Because students had analogous versions of worksheets for CogSketch and paper, it is possible to compare how students complete the same homework tasks in the two formats. First, we consider one of the most common and problematical spatial errors in reasoning about geoscience block diagrams. In the Geologic Block Diagram paper worksheet, 35% of students made a common mistake when completing a block diagram (n = 189, problem shown in Fig. 5). This specific mistake represents nearly 60% of all incorrect submitted sketches. Instead of connecting the same unit on the two sides of the block, students continued the lines nearly parallel to how they were drawn on the sides. An expert would immediately recognize that this was an error for two reasons: It is a geologically unlikely configuration, and the orientation of the drawn planes would be inconsistent with the dip symbols on the top face. This is similar to errors reported by Kali and Orion (1996), who found students tended to continue lines from a known face to an unknown face. For the Geologic Block Diagram CogSketch worksheets, 30% of students made the common block diagram mistake (n = 19), which equates to 65% of all incorrect sketches (19 out of 29 incorrect sketches). In contrast to the paper worksheets, the worksheet tutor helped to correct 7 out of the 19 incorrect block diagram sketches (Fig. 5). Failure to correct all 19 incorrect sketches most likely resulted from inadequate feedback messages from the worksheet tutor and students choosing not to request feedback on their incorrect sketch. Thus, CogSketch is capable of helping students correct this common mistake when the error is first made, but the feedback messages should be improved and students should be prompted to use the tutor during each worksheet. Although we anticipated that students would make errors, our CogSketch geoscience worksheet was not specifically designed to detect the specific form of the error. For Feedback: This layer contact isn’t quite right. The 2 sides show that the layers are planar and you have 2 contact points to use, one from the top face and one from the side face. Connect the 2 points with a straight line, that is your layer contact! Fig. 5. Corrected block diagram sketch. The image on the left is an incorrect sketch of the missing side of a block diagram. The student received the Feedback message shown below the sketches and corrected the sketch (right image). This is an example of how feedback was used to correct student sketches. 960 B. Garnier et al. / Topics in Cognitive Science 9 (2017) unanticipated errors such as this, instructors may inspect completed sketches to determine if there is an error made by a large percentage of students. This can be done within minutes and allows an instructor to address the common mistake and revise the feedback messages to help future students correct the mistake early in their training. Such errors can be revealing about basic cognitive processes and thus serve as a powerful mechanism for increasing the likelihood of serendipitous discovery (see e.g., Gagnier & Shipley, 2016). An important difference between CogSketch and paper worksheets is the ability to move and drag objects. This appeared to be very useful in the Earth’s Magnetic Field worksheet, where for the paper version students had to redraw the packages of rock with arrows onto the image of the Earth. Many students did not redraw the objects correctly on the paper worksheets (Fig. 6). The CogSketch worksheet allows students to focus on the geoscience task, matching the rocks to Earth’s magnetic field, rather than focusing on how to redraw the objects in a different orientation. Here, CogSketch likely lessened the mental burden of mentally rotating and moving objects, and allowed students to attend to the critical variables of the geoscience concept. Future research should consider learning outcomes in evaluating the value of moving components when understanding how to infer location from magnetic information in rocks. Despite the differences in labeling and moving objects, sketches between CogSketch and paper worksheets were very similar and contained similar errors and variability. Incorrect paper sketch Correct CogSketch sketch Rotation Axis Magnetic Pole Most arrows redrawn incorrrectly N Arrow pointing the wrong way S Fig. 6. Difference between Earth’s Magnetic Field CogSketch and paper worksheets. Left image: Correct completion of the CogSketch worksheet. Students moved and rotated boxes with arrows until the arrows aligned with the magnetic field lines. Most students correctly completed this task. Right image: Incorrect sketch from a paper worksheet. Students completing the paper Earth’s Magnetic Field worksheet had to redraw the boxes with arrows instead of dragging and rotating (boxes were still off to the side as shown in the CogSketch version in Fig. 2). Therefore, paper worksheet sketches contained boxes with arrows not in alignment with Earth’s magnetic field lines and/or redrawn improperly. B. Garnier et al. / Topics in Cognitive Science 9 (2017) 961 Instructions were kept as consistent and specific as possible to ensure this would be the outcome. Future research could involve analyzing how the steps and tools used in CogSketch affects sketching and how it might change what someone would sketch on paper. 5.1.3. Less time is needed to grade and provide individual feedback on CogSketch worksheets One obvious difference between a paper and computer worksheet was the role of the worksheet author and grader. For this implementation, we aligned the grading to be as similar as possible. The first author constructed CogSketch’s feedback suggestions, created the grading rubric used for the CogSketch worksheets, and developed and applied the rubric for the human-graded paper worksheets. For this implementation, it took 4–6 h per week to grade the paper worksheets for the 196 participants in the study, highlighting why most instructors do not assign sketching in large classes. In contrast, it took 5–10 min per week—total—to upload the emailed worksheets, apply the grading rubric, and analyze the sketches that had been completed in CogSketch (66 participants). In this preliminary implementation, we monitored the need for individualized feedback on the CogSketch worksheets by reviewing all worksheets after they were graded. If any unanticipated mistakes were found, the first author either wrote an email explaining the correction, created a video of the author correcting the sketch and verbally explaining the process, or drew the correct sketch and provided a written explanation. In some instances, a common problem or mistake did not receive detailed feedback from CogSketch and an email explanation was provided to all students. Then, the worksheet was revised to eliminate the issue in future use. In a few cases, the unique mistakes were corrected with individual feedback from the first author. Even in this circumstance, it took less time than it took to hand grade and provide individual feedback on paper worksheets. Therefore, CogSketch could free approximately 75 h of instructor time for a 200-person class (assuming 15 sketch assignments). Future research should push this comparison further and focus on determining whether CogSketch grading matches human grading. 6. Discussion An interdisciplinary approach to science education that combines cognitive science and disciplinary science offers a unique opportunity to craft learning tools that support student understanding of cognitively challenging disciplinary content. AI systems that can support sketching offer new opportunities. The advantages of CogSketch include (a) the ability to implement sketching in the classroom where the instructor is not needed to view progress on every student’s sketch; (b) feedback upon demand; and (c) a structure for cognitive science theory to be implemented in the design of geoscience materials. The introductory course in Physical Geology—in which CogSketch and graded paper worksheets were used—offered students more sketch-based homework assignments than offered by over 90% of surveyed instructors where the assignments must be hand graded. 962 B. Garnier et al. / Topics in Cognitive Science 9 (2017) From the perspective of a disciplinary instructor, the availability of well-crafted and cognitively robust worksheets, and the time savings offered by CogSketch, would make a significant difference in the frequency of sketching opportunities in introductory courses. A digital sketch provides opportunities that are not possible with a paper sketch. For example, in CogSketch worksheets, students can “grab” lines or objects, rotate them, and move them around on a base image. The ability to move or rotate objects supports students in making visual comparisons. The alternative, when lines cannot move (as with images on paper), likely requires additional mental rotations and other mental transformations as a prerequisite for performing a mental comparison. It is likely that students with low spatial working memory capacities may have difficulty with such a comparison (Baddeley, 2012). Thus, a digital tool may better support learning in such students. However, abstracting physical relations is a key step in learning (Novack, Congdon, Hemani-Lopez, & Goldin-Meadow, 2014), and it is possible that enabling some operations that had to be completed mentally will undermine abstraction and, thus, generalization of learning. An important line of research in the design of the CogSketch tutor for sketching is the net learning value of manipulable elements. The record of student activity while using CogSketch, such as seeking feedback from the worksheet tutor, is a valuable resource for developing the science of learning. By observing how students correct spatial mistakes in response to spatial feedback, a robust theory of spatial learning may be developed. Such a theory is necessary for guiding the construction of optimal spatial feedback. CogSketch offers a crucible to combine the science of learning with the context of a disciplinary science—geoscience. For cognitive psychologists, CogSketch offers a powerful research tool for gathering information about student learning, including the documentation of common mistakes, the role of feedback, and the individual use of feedback. For geology instructors, CogSketch offers an opportunity to employ sketching in their courses without the usual time commitment needed for grading. As a mechanism for collaboration between cognitive science and geoscience educators, CogSketch offers a structure that requires that (a) psychological theories are formulated in sufficient detail to be applied to science tasks; and (b) geoscience educational goals are formulated in sufficient detail to be evaluated. When the above are combined appropriately, worksheets will be engaging, illustrative, and provide useful feedback when errors are made. Community-based projects to develop cognitively appropriate and effective CogSketch worksheets for introductory science classes are a potentially powerful engine for advancing science. Our observations about this process suggest that it is important to have all sides at the table: the geologist to provide legitimate goals, the psychologist to provide guidance on structuring learning, and the AI researcher to provide tools that can evaluate goals and structure learning without real time human guidance. Deploying the intelligent tutoring systems in classrooms still has some obvious limitations, as when no digital devices are available for students, or when the system is platform specific (currently CogSketch only runs on windows-based, or partitioned, computers; however, an HTML, cloud-based version is in development). In the case of the present study, the platform-dependency played a role in students self-selecting out of B. Garnier et al. / Topics in Cognitive Science 9 (2017) 963 the CogSketch group and into the paper group. Other technical challenges, mentioned above, include the need for more detailed spatial representations and the trade-offs between feedback specificity and generality. These challenges have implications for future development of CogSketch and for the design of sketch-based tutoring systems in general. 6.1. Lessons learned Throughout the design and development of CogSketch geoscience worksheets, our group has learned about an interdisciplinary collaborative approach to designing educational materials. The following points are lessons learned from our design process, abstracted so that they might be applied to other interdisciplinary collaborations in designing curricular materials. 1. The CogSketch program and worksheets are adaptable and able to change with use and development. It is necessary throughout the design process to have continued investment from developers and practitioners. Here we summarize key adjustments in the design process that were only possible in the context of an integrated group. • Small-scale design changes necessary to enhance the tutoring system’s handling of spatial concepts and tasks to improve the user’s experience: Position-dependent quantitative ink feedback. This allowed spatial feedback. Where the worksheet tutor once provided only “This is not quite right,” directional feedback now allows, “This needs to be higher.” Arrows to indicate processes. Being able to interpret arrow orientation o allows tasks involving movement and change. Tasks, such as drawing an arrow to show water flow, used to involve quantitative ink constraints around a pre-drawn arrow. Arrow interpretation now allows drawn arrows to convey directional meaning. Feedback on progress. In the case of the present study, a side bar was o added to the CogSketch interface to summarize feedback with completion gauges, a colored meter to show if a task is complete or not. This feedback allowed students to quickly see how much was correct and then drill down into feedback categories for more detailed advice. This was intended to prevent students from being overwhelmed by too many feedback items at once. Large-scale changes: o • o We found that it was critical to have a disciplinary expert (with at least graduate student level training) design the worksheets. In the beginning, worksheets were developed by an AI expert with guidance from a geoscientist. Bridging the gap between domain expertise and worksheet implementation from both ends of the spectrum was laborious and error prone. This approach was not an effective use of either expert’s time. In contrast, 964 B. Garnier et al. / Topics in Cognitive Science 9 (2017) a user-friendly authoring environment allows discipline experts to test and develop activities without requiring deep programming knowledge. Having an authoring environment that supports disciplinary experts extend the system’s knowledge of particular domains also allows the AI program to be used across different disciplines. We also found that it was important to have a taxonomy of spatial skills o that were important to the geosciences, and cognitive principles that could be directly incorporated into geoscience worksheets. The specified spatial skills allowed the geoscience domain expert to classify geoscience concepts so that prior cognitive science research could be incorporated into a worksheet design. This streamlined communication between the geoscientist and cognitive scientist by providing a shared vocabulary. 2. Creating effective feedback is not yet an exact process. With CogSketch, there is an important trade-off between domain depth and breadth. In an ideal case, worksheets would provide rich, multi-step feedback, which can encourage self-evaluation and deeper learning by allowing students to correct faulty interpretations (Bjork & Bjork, 2011; Hattie & Timperley, 2007). Most intelligent tutoring systems achieve this by having detailed models of the domain (e.g., cognitive tutor that is specific to geoscience concepts and processes), but CogSketch was designed for domain-general tutoring, to handle spatial aspects of any defined concept. Therefore, feedback messages were designed to be written by the domain expert to provide the necessary detail, to direct student focus to particular features of interest while addressing a range of possible student errors. We have found that worksheet testing by novices is the best way to fine-tune feedback messages and increase their effectiveness. Additionally, we have also found that position-dependent feedback, which provides more detailed spatial feedback, allows for more specific and shorter feedback messages. It may not be applicable to all worksheet tasks, but this feature is very useful when it can be incorporated. 3. Despite CogSketch’s qualitative spatial reasoning understanding and feedback, there were some instances where students were unable to understand the feedback and correct their sketch. One suggestion for future work is to supplement textual feedback with optional spatiovisual feedback in the form of a diagram or comparison to the solution sketch. This would integrate research that shows that visual comparison is a quick and effective strategy to correct students’ mistakes and increase understanding (Gadgil, Nokes-Malach, & Chi, 2012; Gentner et al., 2015; Loewenstein, Thompson, & Gentner, 1999). 4. During the development of CogSketch geoscience worksheets, geoscience educators have asked if CogSketch worksheets could work for fieldtrips or fieldwork, since many courses incorporate sketching during these activities. Although we did not do this, there are some scenarios where it should be possible. An instructor could create a worksheet with outcrop photos from the outcrop the students will visit or a topographic map of the region, and ask students to annotate the outcrop photo or create a geologic map on top of the base map. However, these activities B. Garnier et al. / Topics in Cognitive Science 9 (2017) 965 would require tablets to allow easy use of CogSketch outdoors. For example, a tablet computer with a camera could potentially take and import photos, which would allow students to annotate particular features. A similar approach could be applied to microscope work (e.g., taking a photo through the eyepiece of a microscope and annotating features). 7. Conclusions Our research illustrates the value of an interdisciplinary, D-EE-D approach to develop educational materials that focus on a current teaching problem, integrates current research from multiple disciplines, and tackles implementation problems simultaneously. Success in this initial step of the design process to create a set of curricular materials that potentially reaches beyond the designer group required extensive work and multiple cycles of theory refinement and tool adjustment. CogSketch offers new ways to sketch in the geoscience classroom and provides students with a tool to aid in solving discipline-specific spatial problems, while providing instructors with insights into student thinking and learning. Geoscience worksheets were designed to address common geoscience concepts that students struggle with, and that are important for continued learning in geoscience and STEM. Classroom implementation showed that CogSketch provides an environment for students to physically interact with diagrams and to receive spatially detailed feedback from the worksheet tutor. CogSketch thus makes it possible and practical for instructors to incorporate sketching activities in large-enrollment courses. Intelligent tutoring systems have the potential to enhance student learning in spatially rich domains, such as geoscience. Furthermore, there is nothing in our study that limits the applicability of these findings to the geosciences. This technology and an interdisciplinary approach could be used to develop effective sketch worksheets in other STEM disciplines and beyond. Acknowledgments We sincerely thank the professors at University of Wisconsin-Madison for accommodating this study in their introductory geology classrooms. The research was funded through the NSF-sponsored SILC (Spatial Intelligence Learning Center: SBE0541957 and SBE-1041707). CogSketch and paper worksheets will be available on the Science Education Resource Center (SERC) website, http://www.serc.carleton.edu, once revised. References Ainsworth, S., Prain, V., & Tytler, R. (2011). Drawing to learn in science. Science, 333, 1096–1097. 966 B. Garnier et al. / Topics in Cognitive Science 9 (2017) Atit, K., Gagnier, K., & Shipley, T. F. (2015). Student gestures aid penetrative thinking. Journal of Geoscience Education, 63(1), 66–72. Atit, K., Shipley, T. F., & Tikoff, B. (2014). What do a geologist’s hands tell you? In D. R. Montello, K. Grossner, & G. G. Janelle (Eds.), Space in mind: Concepts for spatial learning and education (pp. 173– 194). Cambridge, MA: The MIT Press. Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instructions. Journal of Educational Computing Research, 13(2), 111–127. Baddeley, A. (2012). Working memory: Theories, models, and controversies. Annual Review of Psychology, 63, 1–29. Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004). Detecting student misuse of intelligent tutoring systems. In J. C. Lester, R. M. Vicari, & F. Paraguacßua (Eds.), International conference on intelligent tutoring systems (pp. 531–540). New York: Springer Berlin Heidelberg. Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M. Hough, & J. R. Pomerantz (Eds.), Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56– 64). London: Worth Publishers. Bryk, A. S. (2009). Support a science of performance improvement. Phi Delta Kappan, 90(8), 597–600. Bryk, A. S., & Gomez, L. M. (2008). Reinventing a research and development capacity. In F. M. Hess (Ed.), The future of educational entrepreneurship: Possibilities for school reform (pp. 181–187). Cambridge, MA: Harvard Educational Press. Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America’s schools can get better at getting better. Cambridge, MA: Harvard University Press. Burkhardt, H., & Schoenfeld, A. H. (2003). Improving educational research: Toward a more useful, more influential, and better-funded enterprise. Educational Researcher, 32(9), 3–14. Canham, M., & Hegarty, M. (2010). Effects of knowledge and display design on comprehension of complex graphics. Learning and Instruction, 20, 155–166. Corbett, A. T., & Anderson, J. R. (2001). Locus of feedback control in computer-based tutoring: Impact on learning rate, achievement, and attitudes. In J. Jacko, A. Sears, M. Beaudouin-Lafon, & R. Jacob (Eds.), Proceedings of ACM CHI’ 2001 conference on human factors in computing systems (pp. 245–252). New York: ACM Press. Coyan, J. A., Busch, M. M., & Reynolds, S. J. (2009). Examining student interactions with distractors in photographs using eye-tracking technology. Geological Society of America Abstracts With Programs, 41 (7), 196. Falkenhainer, B., Forbus, K. D., & Gentner, D. (1989). The structure-mapping engine: Algorithm and examples. Artificial Intelligence, 41, 1–63. Forbus, K. D., Chang, M. D., McLure, M., & Usher, M. (2017). The cognitive science of sketch worksheets. Topics in Cognitive Science, 9(4), 921–942. Forbus, K., Usher, J., Lovett, A., Lockwood, K., & Wetzel, J. (2011). CogSketch: Sketch understanding for cognitive science research and for education. Topics in Cognitive Science, 3(4), 648–666. Gadgil, S., Nokes-Malach, T. J., & Chi, M. T. (2012). Effectiveness of holistic mental model confrontation in driving conceptual change. Learning and Instruction, 22(1), 47–61. Gagnier, K. M., Atit, K., Shipley, T. F., Ormand, C., Manduca, C., & Tikoff, B. (2012). Improving penetrative thinking skills for geoscience education. San Diego, CA: Presented at iSLC. Gagnier, K. M., & Shipley, T. F. (2016). Visual completion from 2D cross-sections: Implications for visual theory and STEM education and practice. Cognitive Research: Principles and Implications, 1(1), 1–18. Gagnier, K. M., Shipley, T. F., Tikoff, B., Garnier, B., Ormand, C. J., Atit, K., & Resnick, I. (2016). Training spatial skills in geosciences: A review of tests and tools. In R. W. Krantz, C. J. Ormand, & B. Freeman (Eds.), 3D structural interpretation: Earth, mind, and machine: AAPG Memoir 111 (pp. 7–23). Boulder, CO: AAPG. B. Garnier et al. / Topics in Cognitive Science 9 (2017) 967 Gentner, D., Levine, S. C., Ping, R., Isaia, A., Dhillon, S., Bradley, C., & Honke, G. (2015). Rapid learning in a children’s museum via analogical comparison. Cognitive Science, 40, 224–240. Gentner, D., & Smith, L. A. (2013). Analogical learning and reasoning. In D. Reisburg (Ed.), The Oxford handbook of cognitive psychology (pp. 10–11). Oxford: Oxford University Press. Gobert, J. D. (2005). The effects of different learning tasks on model-building in plate tectonics: Diagramming versus explaining. Journal of Geoscience Education, 53(4), 444–455. Gobert, J. D., & Clement, J. J. (1999). Effects of student-generated diagrams versus student-generated summaries on conceptual understanding of causal and dynamic knowledge in plate tectonics. Journal of Research in Science Teaching, 36(1), 39–53. Goodwin, C. (1994). Professional vision. American Anthropologist, 96(3), 606–633. Gordin, D. N., & Pea, R. D. (1995). Prospects for scientific visualization as an educational technology. The Journal of the Learning Sciences, 4(3), 249–279. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. Hegarty, M., Keehner, M., Cohen, C., Montello, D. R., & Lippa, Y. (2007). The role of spatial cognition in medicine: Applications for selecting and training professionals. In G. L. Allen (Ed.), Applied spatial cognition: From research to cognitive technology (pp. 285–315). Mahwah, NJ: Lawrence Erlbaum Associates. Jee, B. D., Gentner, D., Forbus, K., Sageman, B., & Uttal, D. H. (2009). Drawing on experience: Use of sketching to evaluate knowledge of spatial scientific concepts. In N. A. Taatgen & van Rijn H. (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society 2009 (pp. 2499–2504). New York: Curran Associates. Jee, B. D., Sageman, B., Manduca, C. A., & Shipley, T. F. (2014). Drawing on experience: How domain knowledge is reflected in sketches of scientific structures and processes. Research for Science Education, 44, 859–883. Jee, B. D., Uttal, D. H., Gentner, D., Manduca, C., Shipley, T. F., Tikoff, B., Ormand, C. J., & Sageman, B. (2010). Commentary: Analogical thinking in geoscience education. Journal of Geoscience Education, 58 (1), 2–13. Johnson, J., & Reynolds, S. J. (2005). Concept sketches- using student- and instructor-generated, annotated sketches for learning, teach, and assessment in geology courses. Journal of Geoscience Education, 53(1), 85–95. Kali, Y., & Orion, N. (1996). Spatial abilities of high-school students in the perception of geologic structures. Journal of Research in Science Teaching, 33(4), 369–391. Kastens, K. A., & Ishikawa, T. (2006). Spatial thinking in the geosciences and cognitive sciences: A crossdisciplinary look at the intersection of the two fields. In C. A. Manduca & D. Mogk (Eds.), Earth and mind: How geologists think and learn about the earth: Geological Society of America Special Paper 413 (pp. 51–74). Boulder, CO: Geological Society of America. Kulik, J. A., & Kulik, C. C. (1988). Timing of feedback and verbal learning. Review of Educational Research, 58(1), 79–97. Libarkin, J. C., Anderson, S. W., Dahl, J., Beilfuss, M., & Boone, W. (2005). Qualitative analysis of college students’ ideas about the earth: Interviews and open-ended questionnaires. Journal of Geoscience Education, 53(1), 17–26. Libarkin, J. C., Kurdziel, J. P., & Anderson, S. W. (2007). College student conceptions of geological time and the disconnect between ordering and scale. Journal of Geoscience Education, 55(5), 413–422. Liben, L. S., & Titus, S. J. (2012). The importance of spatial thinking for geoscience education: Insights from the crossroads of geoscience and cognitive science. In K. A. Kastens & C. A. Manduca (Eds.), Earth and mind II: A Synthesis of research on thinking and learning in the geosciences: Geological Society of America Special Paper 486 (pp. 51–70). Boulder, CO: Geological Society of America. Loewenstein, J., Thompson, L., & Gentner, D. (1999). Analogical encoding facilitates knowledge transfer in negotiation. Psychonomic Bulletin & Review, 6(4), 586–597. 968 B. Garnier et al. / Topics in Cognitive Science 9 (2017) Magidson, S. (2002). Teaching, research, and instructional design: Bridging communities in mathematics education (doctoral dissertation). Retrieved from ProQuest database (Order No. 3063466). Manduca, C. A., & Kastens, K. A. (2012). Geoscience and geoscientists: Uniquely equipped to study Earth. In K. A. Kastens & C. A. Manduca (Eds.), Earth and mind II: A synthesis of research on thinking and learning in the geosciences: Geological Society of America Special Paper 486 (pp. 1–12). Boulder, CO: Geological Society of America. Marshak, S. (2012). Essentials of geology. New York, NY: W. W. Norton. Morris, A. K., & Hiebert, J. (2011). Creating shared instructional products: An alternative approach to improving teaching. Educational Researcher, 40(1), 5–14. Newcombe, N. S., & Shipley, T. F. (2015). Thinking about Spatial Thinking: New Typology, New Assessments. In J. S. Gero (Ed.), Studying visual and spatial reasoning for design creativity (pp. 179– 192). Amsterdam: Springer. Novack, M. A., Congdon, E. L., Hemani-Lopez, N., & Goldin-Meadow, S. (2014). From action to abstraction: Using the hands to learn math. Psychological Science, 25(4), 903–910. Ormand, C. J., Manduca, C., Shipley, T. F., Tikoff, B., Harwood, C. L., Atit, K., & Boone, A. P. (2014). Evaluating geoscience students’ spatial thinking skills in a multi-institutional classroom study. Journal of Geoscience Education, 62, 146–154. Petcovic, H. L., Ormand, C. J., & Krantz, R. W. (2016). Earth, mind, and paper: Field sketches as expert representations of the hat creek fault zone. In R. W. Krantz, C. J. Ormand, & B. Freeman (Eds.), 3D structural interpretation: Earth, mind, and machine: AAPG Memoir 111 (pp. 173–189). Boulder, CO: AAPG. Rapp, D. N., Culpeppers, S. A., Kirkby, K., & Morin, P. (2007). Fostering students’ comprehension of topographic maps. Journal of Geoscience Education, 55(1), 5–16. Resnick, I., Atit, K., & Shipley, T. F. (2012). Teaching geologic events to understand geologic time. In K. A. Kastens & C. A. Manduca (Eds.), Earth and mind II: A synthesis of research on thinking and learning in the geosciences: Geological Society of America Special Paper 486 (pp. 41–43). Boulder, CO: Geological Society of America. Resnick, I., Davatzes, A., Newcombe, N. S., & Shipley, T. F. (2016). Using relational reasoning to learn about scientific phenomena at unfamiliar scales (special issue). Educational Psychology Review, 9(1), 1– 15. Resnick, I., & Shipley, T. F. (2013). Breaking new ground in the mind: An initial study of mental brittle transformation and mental rigid rotation in science experts. Cognitive Processing, 14(2), 143–152. Resnick, I., Shipley, T. F., Newcombe, N., Massey, C., & Wills, T. (2012). Examining the representation and understanding of large magnitudes using the hierarchical alignment model of analogical reasoning. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Cognitive Science Society (pp. 917–922). Austin, TX: Cognitive Science Society. Reynolds, S. J., Johnson, J., Morin, P., & Carter, C. (2012). Exploring geology. New York, NY: McGrawHill. Reynolds, S. J., Piburn, M. D., Leedy, D. E., McAuliffe, C. M., Birk, J. P., & Johnson, J. K. (2006). The hidden earth – Interactive, computer-based modules for geoscience learning. In C. A. Manduca & D. Mogk (Eds.), Earth and mind: How geologists think and learn about the earth: Geological Society of America Special Paper 413 (pp. 157–170). Boulder, CO: Geological Society of America. Schlaisich, I., & Egenhofer, M. J. (2001). Multimodal spatial querying: What people sketch and talk about. In 1st international conference on universal access in human-computer interaction (pp. 732–736). Mahwah, NJ: Lawrence Erlbaum. Shipley, T. F., Tikoff, B., Ormand, C., & Manduca, C. (2013). Structural geology practice and learning, from the perspective of cognitive science. Journal of Structural Geology, 54, 72–84. Sorby, S. (2009). Educational research in developing 3-D spatial skills for engineering students. International Journal of Science Education, 31(3), 459–480. B. Garnier et al. / Topics in Cognitive Science 9 (2017) 969 Taele, P., Barreto, L., & Hammond, T. (2015). Maestoso: An intelligent educational sketching tool for learning music theory. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI) (pp. 3999–4005). Austin, TX: AAAI Press. Titus, S., & Horsman, E. (2009). Characterizing and improving spatial visualization skills. Journal of Geoscience Education, 57(4), 242–254. Trend, R. (1998). An investigation into understanding of geological time among 10- and 11-year-old children. International Journal of Science Education, 20, 973–988. Trend, R. (2000). Conceptions of geological time among primary teacher trainees, with reference to their engagement with geoscience, history, and science. International Journal of Science Education, 22, 539– 555. Uttal, D. H., Miller, D. I., & Newcombe, N. S. (2013). Exploring and enhancing spatial thinking: Links to achievement in science, technology, engineering, and mathematics? Current Directions in Psychological Science, 22(5), 367–373. Valentine, S., Vides, F., Lucchese, G., Turner, D., Kim, H.-H., Li, W., Linsey, J., & Hammond, T. (2012). Mechanix: A sketch-based tutoring and grading system for free-body diagrams. AI Magazine, 34(1), 55– 66. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. Xu, Y., & Franconeri, S. L. (2015). Capacity for visual features in mental rotation. Psychological Science, 26 (8), 1241–1251. Yin, P., Forbus, K. D., Usher, J., Sageman, B., & Jee, B. D. (2010). Sketch worksheets: A sketch-based educational software system. In Proceedings of the 22nd annual conference on Innovative Applications of Artificial Intelligence (IAAI) (pp. 10–11). Austin, TX: AAAI Press.