Abstract
Popular movies such as Rain Man, A Beautiful Mind and The Imitation Game often depicting the leading characters with high-functioning autism spectrum disorder (ASD) with exceptional math skills (referred to as autistic savants) which is largely inconsistent with research; in fact, only 10% of individuals with ASD have such savant skills; instead, prior studies showed that during their middle school years, students with ASD usually under-perform an average of 5 years below their neuro-typically developing (NT) peers in mathematics. Plausible explanation of such a gap is attributed to the population’s impairments of memory and cognitive development which in turn might undermine their learning abilities in one or more mathematical domains. Our present study aims to compare the effects of three different presentation styles during the training for children with ASD’s mathematical skills (image-based, mathematical digit-based and audio-based ones) through the behavioral analysis and eye-tracking technology. Together with their actual performance in a small-scale pilot test, these data can provide clues to the questions we raised.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction and Background
Popular movies such as Rain Man, A Beautiful Mind and The Imitation Game often depicting the leading characters with high-functioning autism spectrum disorder (ASD) with exceptional math skills (referred to as autistic savants) which is largely inconsistent with research [1]; in fact, only 10% of individuals with ASD have such savant skills and is rare in the population [22]; instead, prior studies showed that during their middle school years, students with ASD usually under-perform an average of 5 years below their neuro-typically developing (NT) peers in mathematics [23]. Plausible explanations of such a gap have been attributed to the population’s impairments of working memory, cognitive development, visual-spatial which in turn might undermine their learning abilities in one or more mathematical domains [5, 6, 9, 13, 16, 20]. Whitby [24] also attributed such a gap appeared in their middle school years to the increasing difficulty of this population’s understanding of the abstract nature of mathematics and their impairments in generalizing acquired skills to the real-world settings.
In addition, quite a number of individuals with ASD have posited the fact that they tend to rely more on the visual mental representations instead of verbal ones in understanding abstract concepts in their daily life [7, 11]; a term referred to as ‘thinking in pictures’ [7, 15]. However, due to the wide variability of impairments each individual with ASD demonstrates, the bias towards the adoption of visual mental representations might be suitable to be interpreted at the group, not individual level [15], which thereby demanding avenues for further research. One of our present study goals aims to address this issue by recruiting a population of users that have not been included in previous studies: Chinese children with ASD [21].
Meanwhile, the overwhelming number of prior studies attempted to tap into the intertwined relationships among these factors, but little previous empirical evidence directly addresses the issue of how some fundamental math presentation strategies might affect the learning outcome which motivates our study. In particular, in present study, we aim to gain access to the additional information with regards to the complex interrelationship between presentation features, contexts and learning outcome afforded through the referential understanding of children’s gaze.
2 Related Work
Four indirect lines of prior research are relevant to our present study which will be briefly reviewed here.
2.1 Mathematics Learning of Individuals with ASD
Popular media has portrayed individuals with high-functioning ASD with exceptional math skills (referred to as autistic savants), which is largely inconsistent with research [1]. However, earlier studies showed that during their middle school years, students with ASD usually under-perform an average of 5 years below their neuro-typically developing (NT) peers in mathematics [23]. Plausible explanations of such a gap have been attributed to the population’s impairments of some cognitive impairment which in turn might undermine their learning abilities in one or more mathematical domains [5, 6, 9, 13, 16, 20]. Whitby [24] also attributed such a gap appeared in their middle school years to the increasing difficulty of this population’s understanding of the abstract nature of mathematics and their impairments in generalizing acquired skills to the real-world settings.
In addition, quite a number of individuals with ASD have posited the fact that they tend to rely more on the visual mental representations instead of verbal ones in understanding abstract concepts in their daily life [7, 11]; a term referred to as ‘thinking in pictures’ [7]. However, due to the wide variability of impairments each individual with ASD demonstrates, the bias towards the adoption of visual mental representations might be suitable to be interpreted at the group, not individual level [15], which thereby demanding avenues for further research.
2.2 Eye-Tracking for Behavioral Analysis During the Decision-Making Process
Eye-tracking technology has allowed the shift from traditional outcome-based measurement of decision making (as inferred by choices and preferences, etc.) to process-tracing based methodology [4]. The achievement of the latter has relied on the traditional mouse–tracking methods [17], and eye-tracking which could examine attention, information acquisition, arousal, etc. at the cognitive and neurological level [2,3,4, 19, 25]. For example, Fiedler et al. [3] analyzed the dynamics of risk preferences in two gambles and verified that the visual attention to the outcome of a gamble tends to increase when the probability of winning increases. In addition, the cognitive efforts and arousal indexed by pupil dilation had also been observed to increase. Zhou et al. [25] quantitatively measured the quality of the decision-making process via physiological sensors such as GSR and an eye tracker in an attempt to alter the user interface where human and computer can augment each other’s strengthens. Franco-Watkins and Johnson [4] observed increased pupil dilation during the decision making process and is influenced by presentation format.
2.3 Eye-Tracking for Behavioral Analysis in ASD Research
Eye-tracking technology has been commonly adopted in the autism research area both clinically and empirically due to a number of advantages it has provided [8, 18]: (1) it may be ideally suited to tap into many abnormalities this population are plagued with; (2) it is objective and easily implemented; (3) it is safe to be used from infancy to adulthood [10, 14, 18]. In our study, we will adopt the affordable version of eye-tracking device, Tobii EyeXTM Footnote 1which is marketed to develop eye-gazed based games; such an affordable eye-tracker will not comprise the quality of the collected eye-gaze data; instead, it offers us the same advantages as those of its more expensive peers (such as Tobii Eye GlassTM) by providing additional information with regards to the complex interrelationship between presentation features, contexts and learning outcome. As of present, the overwhelming number of prior works on eye-tracking attempt to probe into the unusual patterns of visual preferences among the population, among many, [12, 14, 18].
2.4 Preferences of Presentation Format Among Children with ASD
One of the notable studies is the one by Kunda and Goel [15] which examined the long-standing assumption of whether individuals with ASD exhibit a deposition towards visual mental representation over verbal mental representation, referred to as the ‘thinking in pictures’ hypothesis. In the context of a number of cognitive tasks, experiment results strongly support the hypothesis that individuals with ASD tend to tap the visual mental presentation for completing these tasks. However, Kunda and Goel urged researchers to take precautions in interpreting the data within the context of the tasks; and called for more in-depth studies at the behavioral and neurobiological levels.
3 Our Math Learning Application: Prototype Version
3.1 Application Design and Math Training Problem Presentation Styles
The application is running on a Windows Tablet (Win10), and consists of a “Training Material (TM)” module, a “Learning Materials (LM)” module and some audio files to (1) read out each question; (2) feedbacks on students’ performance (praise or encouragement). Instead of providing a ‘right’ or ‘wrong’ feedback on accurate and inaccurate answers, we use more encouraging feedback for the child to engage. Hence, three presentation strategies were included: mathematical digit, pictures, and audio file (see Figs. 1, 2 and 3).
Stage one—the training session.
At the beginning, the TM module consists of short questions on addition, subtraction and multiplication. Figures 1, 2 and 3 shows the presentation styles and performance record for future analysis.
Meanwhile, the audio instruction of this math problem: “yi jia yi” (one plus one in Chinese) will be played. The app will then animate the arithmetic practice, as shown in Fig. 2 as an example.
Again, the audio instruction of the answer to this math question: “er” (two in Chinese) will be played, followed by a rewarding praise (‘well done’) will be played if the answer is accurate; a redirection sentence will be played to encourage the child to try it again if the answer is inaccurate. The user may choose to go back, repeat, or to the next item (Fig. 2). When the training session is over, a log data showing both the child’s performance as well as the corresponding styles he/she had chosen will be recorded (Fig. 3).
The recorded data (as shown in Fig. 3) can then be saved and analyzed to offer valuable and objective information on the child’s preferences, which then can be fed into the system for personalization.
Stage two—the training session.
A learner can choose one of the presentation styles before each question is presented (as shown in Figs. 4, 5 and 6).
Again, all learners’ in-application interaction as well as the performance will be recorded (see Fig. 3).
Feed-back from Participatory Design.
The prototype of the application had been tested with three children with ASD in order to obtain feedbacks from both the children and their teachers who accompany them during the testing; interviews with teachers were also conducted. The results of their feedback had been incorporated into the design of next two versions of the application (v2.0 and v3.0).
4 Our Math Learning Application: Eye-Tracking Version (v2.0 and v3.0)
A second version of the application was designed in order to accommodate these feedbacks (participatory design). The application was developed using C# in Microsoft Visual Studio Enterprise 2015; an eye-tracking application (written in C# for Tobii EyeXTM) is running at the background of the Math app which can be utilized to collect the eye-gaze data. Figure 7 captures the screenshots of the system.
During the integration testing, we found out that since a working Tobii EyeX shows an array of three red lights right below the application (as highlighted in green in Fig. 7), it might pose a potential problem to the children who are not aware of an eye-tracking device before. Therefore, in the pilot testing, we adopted the third version of our application which uses a non-intrusive eye tracker Tobii Eye X2-60 (v3.0). Figure 8 shows a screenshot of the application while Fig. 9 shows the images used in the application (v3.0).
In order to assess our assumption of the possible link between presentation types and learning outcomes, we conducted a pilot study which will be described in details in the next section.
5 Study Results and Discussions
5.1 Participants, Apparatus and Study Procedure
Participants.
One nine-year old child with low functioning ASD (LFASD) and one six-year old child with high functioning ASD (AFASD) participated in the pilot study.
Apparatus.
A Dell workstation with 27 inches 4 K monitor with standard mouse and keyboard was used. A light and portable Tobii X2-60 eye-tracking device was mounted on the monitor.
Study procedure.
Due to the high usability of the learning application and conversation with the teachers, no training was provided. Each child was asked to sit in front of the computer and use mouse to complete a total of twelve math questions (simple addition). The researcher demonstrated how to move the mouse and click a correct answer and asked the child to answer the rests. Figure 10 showed the study environment where the dash-line highlighted device is the Tobii X2-60 which was mounted below the monitor.
Study measures.
Following those in previous eye-tracking studies [2,3,4, 18, 25], we adopted heat-maps and eye-gaze plots to measure the subjects’ preferences of the screen objects: image or mathematical symbol. Heat maps show where the subject concentrated their eye-gaze and the duration of the gaze at a given point. A red spot over an area suggest that the subject (or group of subjects) focused on the given point for a longer period of time which could be inferred to demonstrate their interest on the visual spot over other spots (e.g. Fig. 11). Eye-gaze plots, on the other hand, do not combine more than one user in the screen; it visually shows the temporal movement of the user’s fixation on the screen. The size of the dot denotes the duration of the eye-gaze time (Fig. 12).
5.2 Results and Discussions
Since each question has three optional answers, the chance of correctly answering it by random is 1/3, or 4 out of the total 12 questions. The child with LFASD failed to answer any question. In fact, he looked at the monitor for the first four seconds and then looking around (outside the monitor) for the next 70 s until he left. Figure 11 showed the heat-map of his eye-gaze data. The heat-map data showed no preferences towards either presentation styles.
Figure 12 present the gaze plots of the child where no preferences over either symbols were observed (size of the eye-gaze plot does not vary much). It was revealed that the child has not started to learn math, which provides clues to his eye-gaze patterns on the screen.
The child with HFASD solved 8 problems correctly (three wrongly). Out of 11 questions, he clicked pictures 5 times (once wrongly) and digits 6 times (twice wrongly). The selected pictures are clipart car (selected twice), and real doll, animated wheels, and animated book (once for each). His eye-gazing results are shown in Figs. 13 and 14.
The heat-map data suggested that the child focused more attention on the images (Fig. 13). The gaze plots data further supported it. Figures 14 and 15 showed two of them when he was solving two math question.
6 Concluding Remarks
In our present study, we aim to compare the effects of three different presentation styles during the training for children with ASD’s mathematical skills (image-based, mathematical digit-based and audio-based ones) through the behavioral analysis and eye-tracking technology. Except for one LFASD child who has not officially started the math education, eye-tracking data indicated that the HFASD child show more interest in images.
Although it is still too early to draw conclusions on the preferences of images over mathematical symbols and the influences of appropriate and personalized design elements in math learning application on children’s performance, our study nevertheless highlight the importance of tuning in such finer grained design elements so as to facilitate personalized learning.
In the future, more in-depth studies on viewing patterns could add to our understanding of the old question of whether individuals with ASD tend to ‘think’ in pictures. We are currently working on a prototype of adaptive math learning application that could react to the child’s preferences over the screen objects (via observation of their eye-gaze patterns).
Notes
References
Benaron, L.D.: Biographies of Disease: Autism. Greenwood Press, London (2009)
de Gee, J.W., Knapen, T., Donner, T.H.: Decision-related pupil dilation reflects upcoming choice and individual bias. Proc. Natl. Acad. Sci. 111(5), E618–E625 (2014)
Fiedler, S., Glockner, A., Johnson, J.G., Stewart, N., Krajbich, I.: The dynamics of decision making in risky choice: an eye-tracking analysis. Frontiers Psychol. 3, 335 (2012)
Franco-Watkins, A.M., Johnson, J.G.: Applying the decision moving window to risky choice: comparison of eye-tracking and mouse-tracing methods. Judgment Decis. Making 6(8), 740–749 (2011)
Geary, D.C.: Mathematical disorders: an overview for educators. Perspectives 26, 6–9 (2000)
Geary, D.C.: Mathematics and learning disabilities. J. Learn. Disabil. 37, 4–15 (2004)
Grandin, T.: Thinking in Pictures, Expanded Edition: My Life with Autism. Vintage Books, New York (2006)
Guillon, Q., Hadjikhani, N., Baduel, S., Roge, B.: Visual social attention in autism spectrum disorder: insights from eye tracking studies. Neurosci. Biobehav. Rev. 42, 279–297 (2014)
Hansen, N., Jordan, N.C., Fernandez, E., Siegler, R.S., Fuchs, L., Gersten, R., Micklos, D.: General and math-specific predictors of sixth-graders’ knowledge of fractions. Cogn. Dev. 35, 34–49 (2015)
Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., van de Weijer, J.: Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford University Press, Oxford (2011)
Hurlburt, R., Happé, F., Frith, U.: Sampling the form of inner experience in three adults with Asperger syndrome. Psychol. Med. 24, 385–396 (1994)
Jones, W., Klin, A.: Attention to eyes is present but in decline in 2–6 month old infants later diagnosed with autism. Nature 504, 427–431 (2013)
Kim, H., Cameron, C.E.: Implications of visuospatial skills and executive functions for learning mathematics: evidence from children with autism and williams syndrome. AERA Open 2(4), 1–16 (2016). SAGE Publications
Klin, A., Lin, D.J., Gorrindo, P., Ramsay, G., Jones, W.: Two-year-olds with autism orient to non-social contingencies rather than biological motion. Nature 459(7244), 257–261 (2009)
Kunda, M., Goel, A.K.: Thinking in pictures as a cognitive account of autism. J. Autism Dev. Disord. 41(9), 1157–1177 (2011)
Minshew, N.J., Goldstein, G., Taylor, H.G., Siegel, D.J.: Academic achievement in high functioning autistic individuals. J. Clin. Exp. Neuropsychol. 16(2), 261–270 (1994)
Payne, J.W., Bettman, J.R., Johnson, E.J.: The Adaptive Decision Maker. Cambridge University Press, New York (1993)
Pierce, K., Marinero, S., Hazin, R., McKenna, B., Barnes, C.C., Maligea, A.: Eye tracking reveals abnormal visual preference for geometric images as an early biomarker of an autism spectrum disorder subtype associated with increased symptom severity. Biol. Psychiatry 79(8), 657–666 (2016)
Preuschoff, K., ’t Hart, B.M., Einhäuser, W.: Pupil dilation signals surprise: evidence for noradrenaline’s role in decision making. Frontiers Neurosci. 5, 115 (2011)
Szucs, D., Devine, A., Soltesz, F., Nobes, A., Gabriel, F.: Developmental dyscalculia is related to visuo-spatial memory and inhibition impairment. Cortex 49(10), 2674–2688 (2013)
Tang, T., Flatla, D.: Autism awareness and technology-based intervention research in china: the good, the bad, and the challenging. In: Proceedings of Workshop on Autism and Technology - Beyond Assistance & Intervention, in conjunction with the 34th ACM International Conference on Human Factors in Computing Systems (CHI 2016), San Jose, CA, USA (2016)
Treffert, D.A.: The savant syndrome: an extraordinary condition. A synopsis: past, present, future. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364(1522), 1351–1357 (2009)
Wagner, M., Marder, C., Blackorby, J., Cameto, R., Newman, L., Levine, P., Davies-Mercier, E.: The Achievements of Youth with Disabilities During Secondary School: A Report from the National Longitudinal Transition Study-2. SRI International, Menlo Park (2003)
Whitby, P.J.S.: The effects of solve it! on the mathematical word problem solving ability of adolescents with autism spectrum disorders. Focus Autism Other Dev. Disabil. 28(2), 78–88 (2013)
Zhou, J.L., Sun, J.J., Chen, F., Wang, Y., Taib, R., Khawaji, A., Li, Z.: Measurable decision making with GSR and pupillary analysis for intelligent user interface. ACM Trans. Comput.-Hum. Interact. (TOCHI) 21(6), 33 (2015)
Acknowledgements
The authors acknowledge the financial support to this research by Wenzhou Municipal Science and Technology Bureau (温州市科技局, S20160012). Thanks also go to the children and their teachers at the Orange Wheat Field Children Support Center for participating in our study. Suggestion and constructive comments of the math questions by Dr. Lengleng Lim at Wenzhou-Kean University are also appreciated.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Winoto, P., Tang, T.Y., Huang, Z., Chen, P. (2017). “Thinking in Pictures?” Performance of Chinese Children with Autism on Math Learning Through Eye-Tracking Technology. In: Zaphiris, P., Ioannou, A. (eds) Learning and Collaboration Technologies. Technology in Education. LCT 2017. Lecture Notes in Computer Science(), vol 10296. Springer, Cham. https://doi.org/10.1007/978-3-319-58515-4_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-58515-4_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-58514-7
Online ISBN: 978-3-319-58515-4
eBook Packages: Computer ScienceComputer Science (R0)