Article

Off-task behavior in the cognitive tutor classroom: when students "game the system"

Authors:

Ryan Shaun Baker,

Albert T. Corbett,

Kenneth R. Koedinger,

Angela Z. WagnerAuthors Info & Claims

CHI '04: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 383 - 390

https://doi.org/10.1145/985692.985741

Published: 25 April 2004 Publication History

Abstract

We investigate the prevalence and learning impact of different types of off-task behavior in classrooms where students are using intelligent tutoring software. We find that within the classrooms studied, no other type of off-task behavior is associated nearly so strongly with reduced learning as "gaming the system": behavior aimed at obtaining correct answers and advancing within the tutoring curriculum by systematically taking advantage of regularities in the software's feedback and help. A student's frequency of gaming the system correlates as strongly to post-test score as the student's prior domain knowledge and general academic achievement. Controlling for prior domain knowledge, students who frequently game the system score substantially lower on a post-test than students who never game the system. Analysis of students who choose to game the system suggests that learned helplessness or performance orientation might be better accounts for why students choose this behavior than lack of interest in the material. This analysis will inform the future re-design of tutors to respond appropriately when students game the system.

References

[1]

Aleven, V. Helping Students To Become Better Help Seekers: Towards Supporting Metacognition in a Cognitive Tutor. Paper presented at German-USA Early Career Research Exchange Program: Research on Learning Technologies and Technology-Supported Education, Tubingen, Germany (October 2001).

[2]

Aleven, V., Koedinger, K.R. Limitations of Student Control: Do Students Know When They Need Help? Proceedings of the 5th International Conference on Intelligent Tutoring Systems (2000), 292--303.

Digital Library

[3]

Aleven, V., Koedinger, K.R. Investigations into Help Seeking and Learning with a Cognitive Tutor. In R. Luckin (Ed.), Papers of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001), 47--58.

[4]

Anderson, J.R. Rules of the Mind. Lawrence Erlbaum Associates, Hillsdale, NJ (1993).

[5]

Arbreton, A. Student Goal Orientation and Help-Seeking Strategy Use. In S.A. Karabenick (Ed.), Strategic Help Seeking: Implications For Learning And Teaching, Lawrence Erlbaum Associates, Mahwah, NJ (1998), 95--116.

[6]

Baker, R.S., Corbett, A.T., Koedinger, K.R. The Resilience of Overgeneralization of Knowledge about Data Representations. Presented at American Educational Research Association Conference (2002).

[7]

Baker, R.S., Corbett, A.T., Koedinger, K.R., Schneider, M.P. A Formative Evaluation of a Tutor for Scatterplot Generation: Evidence on Difficulty Factors, Proceedings of the Conference on Artificial Intelligence in Education (2003), 107--115.

[8]

Bloom, B.S. Human Characteristics and School Learning. McGraw-Hill, New York (1976). Ch. 7.

[9]

Bransford, J.D., Schwartz, D.L. Rethinking Transfer: A Simple Proposal With Interesting Implications. In Iron-Nejad, A., Pearson, P.D. (Eds.) Review of Research in Education, 24 (1998), 61--101.

[10]

Carroll, J. A Model For School Learning. Teachers College Record, 64 (1963), 723--733.

[11]

Corbett, A.T., Anderson, J.R. Student Modeling and Mastery Learning in a Computer-Based Programming Tutor. Second International Conference on Intelligent Tutoring Systems (1992), 413--420.

Digital Library

[12]

de Vicente, A., Pain, H. Informing the detection of the students' motivational state: an empirical study. Proceedings of the Sixth International Conference on Intelligent Tutoring Systems (2002), 933--943

Digital Library

[13]

Dweck, C.S., Leggett, E.L. A Social-Cognitive Approach to Motivation and Personality. Psychological Review, 95 (2) (1988), 256--271.

[14]

Hall, V.C. & Edmondson, B. Relative Importance of Aptitude and Prior Domain Knowledge on Immediate and Delayed Posttests. Journal of Educational Psychology, 84 (2) (1992), 219-223.

[15]

Heffernan, N.T., Koedinger, K.R. A developmental model for algebra symbolization: The results of a difficulty factors assessment. Proceedings of the Twentieth Annual Conference of the Cognitive Science Society (1998), 484--489.

[16]

Karweit, N., Slavin, R.E. Time-On-Task: Issues of Timing, Sampling, and Definition. Journal of Experimental Psychology, 74 (6) (1982), 844--851.

[17]

Koedinger, K.R., Anderson, J.R., Hadley, W.H., & Mark, M.A. Intelligent Tutoring Goes to School in the Big City. International Journal of Artificial Intelligence in Education, 8 (1997), 30-43.

[18]

Lahaderne, H.M. Attitudinal and Intellectual Correlates of Attention: A Study of Four Sixth-Grade Classrooms. Journal of Educational Psychology, 59 (5) (1968), 320--324.

[19]

Lee, S.W., Kelly, K.E., Nyre, J.E. Preliminary Report on the Relation of Students' On-Task Behavior With Completion of School Work. Psychological Reports, 84 (1999), 267--272.

[20]

Lloyd, J.W., Loper, A.B. Measurement and Evaluation of Task-Related Learning Behavior: Attention to Task and Metacognition. School Psychology Review, 15 (3) (1986), 336--345.

[21]

Mostow, J., Aist, G., Beck, J., Chalasani, R., Cuneo, A., Jia, P., Kadaru, K. A La Recherche du Temps Perdu, or As Time Goes By: Where Does the Time Go in a Reading Tutor that Listens? Proceedings of the Sixth International Conference on Intelligent Tutoring Systems (2002), 320--329.

Digital Library

[22]

Schofield, J.W. Computers and Classroom Culture. Cambridge University Press, Cambridge, UK (1995).

Digital Library

[23]

Wood, H., Wood, D. Help Seeking, Learning, and Contingent Tutoring. Computers and Education, 33 (1999), 153--169.

Digital Library

Cited By

Waleed Daweli TAbdulrahman Moqbel Mahoub R(2024)Exploring EFL Learners’ Perspectives on Using AI Tools and Their Impacts in Reading Instruction: An Exploratory StudyArab World English Journal10.24093/awej/call10.11(160-171)Online publication date: 28-Jul-2024
https://doi.org/10.24093/awej/call10.11
Sibia NBernuy ATran EXu JWilliams JPetersen ALiut M(2024)Exploring Self-Explanations in a Flipped Database CourseProceedings of the 3rd International Workshop on Data Systems Education: Bridging education practice with education research10.1145/3663649.3664374(20-26)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3663649.3664374
Wang KLiu HDeLiema DHaber NSalehi S(2024)Discovering Players’ Problem-Solving Behavioral Characteristics in a Puzzle Game through Sequence MiningProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636907(498-506)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636907
Show More Cited By

Index Terms

Off-task behavior in the cognitive tutor classroom: when students "game the system"

Recommendations

Modeling and understanding students' off-task behavior in intelligent tutoring systems
CHI '07: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

We present a machine-learned model that can automatically detect when a student using an intelligent tutoring system is off-task, i.e., engaged in behavior which does not involve the system or a learning task. This model was developed using only log ...
Comparing disengaged behavior within a cognitive tutor in the USA and philippines
ITS'10: Proceedings of the 10th international conference on Intelligent Tutoring Systems - Volume Part II

We study how student behaviors associated with engagement differ across different school settings We present a study to investigate the variation in gaming the system and off-task behavior in schools in the USA and Philippines, using quantitative field ...
Off-Task Behavior in Narrative-Centered Learning Environments
Proceedings of the 2009 conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling

Recent years have seen increasing interest in narrative-centered learning environments. However, the same qualities that make them engaging can also introduce seductive details that invite off-task behavior. This paper examines off-task behavior in the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '04: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2004

742 pages

ISBN:1581137028

DOI:10.1145/985692

Conference Chairs:
Elizabeth Dykstra-Erickson
Kinoma
,
Manfred Tscheligi
CURE, Austria

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CHI04

Sponsor:

CHI04: CHI 2004 Conference on Human Factors in Computing Systems

April 24 - 29, 2004

Vienna, Austria

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

247
Total Citations
View Citations
2,395
Total Downloads

Downloads (Last 12 months)107
Downloads (Last 6 weeks)18

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Waleed Daweli TAbdulrahman Moqbel Mahoub R(2024)Exploring EFL Learners’ Perspectives on Using AI Tools and Their Impacts in Reading Instruction: An Exploratory StudyArab World English Journal10.24093/awej/call10.11(160-171)Online publication date: 28-Jul-2024
https://doi.org/10.24093/awej/call10.11
Sibia NBernuy ATran EXu JWilliams JPetersen ALiut M(2024)Exploring Self-Explanations in a Flipped Database CourseProceedings of the 3rd International Workshop on Data Systems Education: Bridging education practice with education research10.1145/3663649.3664374(20-26)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3663649.3664374
Wang KLiu HDeLiema DHaber NSalehi S(2024)Discovering Players’ Problem-Solving Behavioral Characteristics in a Puzzle Game through Sequence MiningProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636907(498-506)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636907
Vanacore KGurung ASales AHeffernan N(2024)The Effect of Assistance on Gamers: Assessing The Impact of On-Demand Hints & Feedback Availability on Learning for Students Who Game the SystemProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636904(462-472)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636904
Belitz CLee HNasiar NFancsali SRitter SAlmoubayyed HBaker ROcumpaugh JBosch N(2024)Hierarchical Dependencies in Classroom Settings Influence Algorithmic Bias MetricsProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636869(210-218)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636869
Jiang LBelitz CBosch N(2024)Synthetic Dataset Generation for Fairer Unfairness ResearchProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636868(200-209)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636868
Hernández-Calderón JBenítez-Guerrero ERojano-Cáceres JMezura-Godoy C(2024)Mining User-Object Interaction Data for Student Modeling in Intelligent Learning EnvironmentsProgramming and Computer Software10.1134/S036176882308008X49:8(657-670)Online publication date: 24-Jan-2024
https://doi.org/10.1134/S036176882308008X
Figueroa EBatista EPalau RUnciti OFerre MMartínez-Ballesté A(2024)The Use of Artificial Intelligence Techniques in Smart Classrooms is in Its InfancyIEEE Access10.1109/ACCESS.2024.345437212(125179-125193)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3454372
Kristensen JTorkildsen JAndersson B(2024)Repeated mistakes in app-based language learningComputers & Education10.1016/j.compedu.2023.104966210:COnline publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1016/j.compedu.2023.104966
Hershkovitz ANoster NSiller HTabach M(2024)Learning analytics in mathematics education: the case of feedback use in a digital classification task on reflective symmetryZDM – Mathematics Education10.1007/s11858-024-01551-556:4(727-739)Online publication date: 3-Mar-2024
https://doi.org/10.1007/s11858-024-01551-5
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents