research-article

Seeing Beyond Expert Blind Spots: Online Learning Design for Scale and Quality

Authors:

Ken KoedingerAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 51, Pages 1 - 14

https://doi.org/10.1145/3411764.3445045

Published: 07 May 2021 Publication History

Abstract

Maximizing system scalability and quality are sometimes at odds. This work provides an example showing scalability and quality can be achieved at the same time in instructional design, contrary to what instructors may believe or expect. We situate our study in the education of HCI methods, and provide suggestions to improve active learning within the HCI education community. While designing learning and assessment activities, many instructors face the choice of using open-ended or close-ended activities. Close-ended activities such as multiple-choice questions (MCQs) enable automated feedback to students. However, a survey with 22 HCI professors revealed a belief that MCQs are less valuable than open-ended questions, and thus, using them entails making a quality sacrifice in order to achieve scalability. A study with 178 students produced no evidence to support the teacher belief. This paper indicates more promise than concern in using MCQs for scalable instruction and assessment in at least some HCI domains.

References

[1]

José Abdelnour-Nocera, Mario Michaelides, Ann Austin, and Sunila Modi. 2012. An intercultural study of HCI education experience and representation. In Proceedings of the 4th international conference on Intercultural Collaboration. 157–160.

Digital Library

[2]

Piotr D Adamczyk and Michael B Twidale. 2007. Supporting multidisciplinary collaboration: requirements from novel HCI education. In Proceedings of the SIGCHI conference on Human factors in computing systems. 1073–1076.

Digital Library

[3]

Tahani Alsubait, Bijan Parsia, and Ulrike Sattler. 2016. Ontology-based multiple choice question generation. KI-Künstliche Intelligenz 30, 2 (2016), 183–188.

[4]

Douglas Bates, Deepayan Sarkar, Maintainer Douglas Bates, and L Matrix. 2007. The lme4 package. R package version 2, 1 (2007), 74.

[5]

TAXONOMY MADE EASY BLOOM’S. 1965. Bloom’s taxonomy of educational objectives. Longman.

[6]

Clodis Boscarioli, Luciana AM Zaina, Sílvia Amélia Bim, Simone Diniz Junqueira Barbosa, and Milene S Silveira. 2016. HCI Education in Brazil from the Results of the Workshop on Teaching of HCI. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems. 1–4.

Digital Library

[7]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.

[8]

Neil CC Brown and Amjad Altadmri. 2014. Investigating novice programming mistakes: Educator beliefs vs. student data. In Proceedings of the tenth annual conference on International computing education research. 43–50.

Digital Library

[9]

Neil CC Brown and Amjad Altadmri. 2017. Novice Java programming mistakes: Large-scale data vs. educator beliefs. ACM Transactions on Computing Education (TOCE) 17, 2 (2017), 1–21.

Digital Library

[10]

Elizabeth Churchill, Jennifer Preece, and Anne Bowser. 2014. Developing a living HCI curriculum to support a global community. In CHI’14 Extended Abstracts on Human Factors in Computing Systems. 135–138.

[11]

Elizabeth F Churchill, Anne Bowser, and Jennifer Preece. 2016. The future of HCI education: a flexible, global, living curriculum. interactions 23, 2 (2016), 70–73.

Digital Library

[12]

Jason Day and Jim Foley. 2006. Evaluating web lectures: A case study from HCI. In CHI’06 Extended Abstracts on Human Factors in Computing Systems. 195–200.

[13]

Paul Denny, Andrew Luxton-Reilly, and John Hamer. 2008. The PeerWise system of student contributed assessment questions. In Proceedings of the tenth conference on Australasian computing education-Volume 78. Citeseer, 69–74.

Digital Library

[14]

Louis Deslauriers, Logan S McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin. 2019. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences (2019), 201821936.

[15]

Barbara J Ericson, Lauren E Margulieux, and Jochen Rick. 2017. Solving parsons problems versus fixing and writing code. In Proceedings of the 17th Koli Calling International Conference on Computing Education Research. 20–29.

Digital Library

[16]

Chase Felker, Radka Slamova, and Janet Davis. 2012. Integrating UX with scrum in an undergraduate software development project. In Proceedings of the 43rd ACM technical symposium on Computer Science Education. 301–306.

Digital Library

[17]

Steven C Funk and K Laurie Dickson. 2011. Multiple-choice and short-answer exam performance in a college classroom. Teaching of Psychology 38, 4 (2011), 273–277.

[18]

Colin M Gray. 2014. Evolution of design competence in UX practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1645–1654.

Digital Library

[19]

Catherine M Hicks, Vineet Pandey, C Ailie Fraser, and Scott Klemmer. 2016. Framing feedback: Choosing review environment features that support high quality peer assessment. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 458–469.

Digital Library

[20]

Instructure, Inc.2019. Canvas. https://www.canvaslms.com/research-education

[21]

David A Joyner, Wade Ashby, Liam Irish, Yeeling Lam, Jacob Langston, Isabel Lupiani, Mike Lustig, Paige Pettoruto, Dana Sheahen, Angela Smiley, 2016. Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 399–408.

Digital Library

[22]

Kenneth R Koedinger, John R Anderson, William H Hadley, and Mary A Mark. 1997. Intelligent tutoring goes to school in the big city. (1997).

[23]

Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science 36, 5 (2012), 757–798.

[24]

Kenneth R Koedinger, Jihee Kim, Julianna Zhuxin Jia, Elizabeth A McLaughlin, and Norman L Bier. 2015. Learning is not a spectator sport: Doing is better than watching for learning from a MOOC. In Proceedings of the second (2015) ACM conference on learning@ scale. ACM, 111–120.

Digital Library

[25]

Kenneth R Koedinger and Mitchell J Nathan. 2004. The real story behind story problems: Effects of representations on quantitative reasoning. The journal of the learning sciences 13, 2 (2004), 129–164.

[26]

Panayiotis Koutsabasis, Spyros Vosinakis, Modestos Stavrakis, and Panagiotis Kyriakoulakos. 2018. Teaching HCI with a studio approach: Lessons learnt. In Proceedings of the 22nd Pan-Hellenic Conference on Informatics. 282–287.

Digital Library

[27]

Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015. PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale. ACM, 75–84.

Digital Library

[28]

Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. 2020. A Systematic Review of Automatic Question Generation for Educational Purposes. International Journal of Artificial Intelligence in Education 30, 1(2020), 121–204.

[29]

Jeri L Little and Elizabeth Ligon Bjork. 2015. Optimizing multiple-choice tests as tools for learning. Memory & Cognition 43, 1 (2015), 14–26.

[30]

Jeri L Little, Elizabeth Ligon Bjork, Robert A Bjork, and Genna Angello. 2012. Multiple-choice tests exonerated, at least of some charges: Fostering test-induced learning and avoiding test-induced forgetting. Psychological science 23, 11 (2012), 1337–1344.

[31]

Mitchell J Nathan and Kenneth R Koedinger. 2000. An investigation of teachers’ beliefs of students’ algebra development. Cognition and Instruction 18, 2 (2000), 209–237.

[32]

Mitchell J Nathan and Kenneth R Koedinger. 2000. Teachers’ and researchers’ beliefs about the development of algebraic reasoning. Journal for Research in Mathematics Education (2000), 168–190.

[33]

Mitchell J Nathan and Anthony Petrosino. 2003. Expert blind spot among preservice teachers. American educational research journal 40, 4 (2003), 905–928.

[34]

Alannah Oleson, Meron Solomon, and Amy J Ko. 2020. Computing Students’ Learning Difficulties in HCI Education. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[35]

Leo Porter, Cynthia Taylor, and Kevin C Webb. 2014. Leveraging open source principles for flexible concept inventory development. In Proceedings of the 2014 conference on Innovation & technology in computer science education. 243–248.

Digital Library

[36]

Henry L Roediger III and Jeffrey D Karpicke. 2006. Test-enhanced learning: Taking memory tests improves long-term retention. Psychological science 17, 3 (2006), 249–255.

[37]

Wendy Roldan, Xin Gao, Allison Marie Hishikawa, Tiffany Ku, Ziyue Li, Echo Zhang, Jon E Froehlich, and Jason Yip. 2020. Opportunities and Challenges in Involving Users in Project-Based HCI Education. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–15.

Digital Library

[38]

James M Royer, Cheryl A Cisero, and Maria S Carlo. 1993. Techniques and procedures for assessing cognitive skills. Review of Educational Research 63, 2 (1993), 201–243.

[39]

Sherry Ruan, Liwei Jiang, Justin Xu, Bryce Joe-Kun Tham, Zhengneng Qiu, Yeshuang Zhu, Elizabeth L Murnane, Emma Brunskill, and James A Landay. 2019. Quizbot: A dialogue-based adaptive learning system for factual knowledge. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[40]

Mehdi SM Sajjadi, Morteza Alamgir, and Ulrike von Luxburg. 2016. Peer grading in a course on algorithms and data structures: Machine learning algorithms do not improve over simple baselines. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 369–378.

Digital Library

[41]

Eunice Sari and Bimlesh Wadhwa. 2015. Understanding HCI education across Asia-Pacific. In Proceedings of the International HCI and UX Conference in Indonesia. 65–68.

Digital Library

[42]

Richard A Schmidt and Robert A Bjork. 1992. New conceptualizations of practice: Common principles in three paradigms suggest new concepts for training. Psychological science 3, 4 (1992), 207–218.

[43]

Arjun Singh, Sergey Karayev, Kevin Gutowski, and Pieter Abbeel. 2017. Gradescope: a fast, flexible, and fair system for scalable assessment of handwritten work. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale. ACM, 81–88.

Digital Library

[44]

Megan A Smith and Jeffrey D Karpicke. 2014. Retrieval practice with short-answer, multiple-choice, and hybrid tests. Memory 22, 7 (2014), 784–802.

[45]

Olivier St-Cyr, Craig M MacDonald, Elizabeth F Churchill, Jenny J Preece, and Anna Bowser. 2018. Developing a community of practice to support global HCI education. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 1–7.

Digital Library

[46]

Brenda Sugrue. 1995. A Theory-Based Framework for Assessing Domainl-Specific Problem-Solving Ability. Educational Measurement: Issues and Practice 14, 3 (1995), 29–35.

[47]

Brenda Sugrue, Noreen Webb, and Jonah Schlackman. 1998. The Interchangeability of Assessment Methods in Science. CSE Technical Report 474.(1998).

[48]

Jennyfer Lawrence Taylor, Jessica Tsimeris, XuanYing Zhu, Duncan Stevenson, and Tom Gedeon. 2015. Observations from teaching HCI to Chinese students in Australia. In Proceedings of the ASEAN CHI Symposium’15. 31–35.

Digital Library

[49]

Gaurav Singh Tomar, Sreecharan Sankaranarayanan, Xu Wang, and Carolyn Penstein Rosé. 2017. Coordinating collaborative chat in massive open online courses. arXiv preprint arXiv:1704.05543(2017).

[50]

Anna Vasilchenko, Adriana Wilde, Stephen Snow, Madeline Balaam, and Marie Devlin. 2018. Video coursework: opportunity and challenge for HCI education. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces. 1–3.

Digital Library

[51]

Mihaela Vorvoreanu, Colin M Gray, Paul Parsons, and Nancy Rasche. 2017. Advancing UX education: A model for integrated studio pedagogy. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 1441–1446.

Digital Library

[52]

Xu Wang, Srinivasa Teja Talluri, Carolyn Rose, and Kenneth Koedinger. 2019. UpGrade: Sourcing Student Open-Ended Solutions to Create Scalable Learning Opportunities. In Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale (Chicago, IL, USA) (L@S ’19). ACM, New York, NY, USA, Article 17, 10 pages. https://doi.org/10.1145/3330430.3333614

Digital Library

[53]

Xu Wang, Miaomiao Wen, and Carolyn Rose. 2017. Contrasting explicit and implicit support for transactive exchange in team oriented project based learning. Philadelphia, PA: International Society of the Learning Sciences.

[54]

Noreen M Webb, Jonah Schlackman, and Brenda Sugrue. 2000. The dependability and interchangeability of assessment methods in science. Applied Measurement in Education 13, 3 (2000), 277–301.

[55]

Lauren Wilcox, Betsy DiSalvo, Dick Henneman, and Qiaosi Wang. 2019. Design in the hci classroom: Setting a research agenda. In Proceedings of the 2019 on Designing Interactive Systems Conference. 871–883.

Digital Library

[56]

Rainer Winkler, Sebastian Hobert, Antti Salovaara, Matthias Söllner, and Jan Marco Leimeister. 2020. Sara, the lecturer: Improving learning in online education with a scaffolding-based conversational agent. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[57]

Qian Yang, John Zimmerman, Aaron Steinfeld, Lisa Carey, and James F Antaki. 2016. Investigating the heart pump implant decision process: opportunities for decision support tools to help. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4477–4488.

Digital Library

[58]

Nesra Yannier, Kenneth R Koedinger, and Scott E Hudson. 2015. Learning from mixed-reality games: Is shaking a tablet as effective as physical observation?. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1045–1054.

Digital Library

Cited By

Cheng ZXu JJin H(2024)TreeQuestion: Assessing Conceptual Learning Outcomes with LLM-Generated Multiple-Choice QuestionsProceedings of the ACM on Human-Computer Interaction10.1145/36869708:CSCW2(1-29)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686970
Lu XWang XJoyner DKim MWang XXia M(2024)Generative Students: Using LLM-Simulated Student Profiles to Support Question Item EvaluationProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662031(16-27)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662031
Gurung AVanacore KMcreynolds AOstrow KWorden ESales AHeffernan N(2024)Multiple Choice vs. Fill-In Problems: The Trade-off Between Scalability and LearningProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636908(507-517)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636908
Show More Cited By

Index Terms

Seeing Beyond Expert Blind Spots: Online Learning Design for Scale and Quality

Index terms have been assigned to the content through auto-classification.

Recommendations

Evaluating web lectures: a case study from HCI
CHI EA '06: CHI '06 Extended Abstracts on Human Factors in Computing Systems

We present research using web lectures to enhance the classroom learning experience in an introductory HCI course. By using web lectures to present lecture material before class, more in-class time can be spent engaging students with hands-on learning ...
Metacognitive Strategies to Foster Interculturally-Aware Design Competency
EduCHI '24: Proceedings of the 6th Annual Symposium on HCI Education

Metacognition is vital for learning in general, and its value for HCI educational practices warrants investigation. In general, metacognitive awareness can help student designers consider how their thought processes might influence their design outputs. ...
e-Assessment in Mathematics Courses with Multiple-choice Questions Tests
CSEDU 2015: Proceedings of the 7th International Conference on Computer Supported Education - Volume 2

With the implementation of the Bologna Process several challenges have been posed to higher education

institution, particularly in Portugal. One of the main implications is related to the change of the paradigm of

a teacher centered education, to a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
795
Total Downloads

Downloads (Last 12 months)100
Downloads (Last 6 weeks)25

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cheng ZXu JJin H(2024)TreeQuestion: Assessing Conceptual Learning Outcomes with LLM-Generated Multiple-Choice QuestionsProceedings of the ACM on Human-Computer Interaction10.1145/36869708:CSCW2(1-29)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686970
Lu XWang XJoyner DKim MWang XXia M(2024)Generative Students: Using LLM-Simulated Student Profiles to Support Question Item EvaluationProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662031(16-27)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662031
Gurung AVanacore KMcreynolds AOstrow KWorden ESales AHeffernan N(2024)Multiple Choice vs. Fill-In Problems: The Trade-off Between Scalability and LearningProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636908(507-517)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636908
Choi SLee HLee YKim J(2024)VIVID: Human-AI Collaborative Authoring of Vicarious Dialogues from Lecture VideosProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642867(1-26)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642867
Wang JTang HKantor TSoltani TPopov VWang X(2024)Surgment: Segmentation-enabled Semantic Search and Creation of Visual Question and Feedback to Support Video-Based Surgery LearningProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642587(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642587
Fung KLee LHui PSong S(2024)Can Underprivileged Children Learn Effectively at Home? A Six-Month Study of Game-Based Traditional Chinese Learning During the Pandemic LockdownIEEE Transactions on Learning Technologies10.1109/TLT.2023.325588217(294-309)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TLT.2023.3255882
Zhou XKok CQuintana RDelahay AWang XSpikol DViberg OMartínez-Monés AGuo P(2023)How Learning Experience Designers Make Design Decisions: The Role of Data, the Reliance on Subject Matter Expertise, and the Opportunities for Data-Driven SupportProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3593388(132-143)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3593388
Brown NSouth KVenkatasubramanian SWiese E(2023)Designing Ethically-Integrated Assignments: It’s Harder Than it LooksProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600126(177-190)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600126
Pelánek REffenberger T(2022)Improving Learning Environments: Avoiding Stupidity PerspectiveIEEE Transactions on Learning Technologies10.1109/TLT.2022.315493615:1(64-77)Online publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1109/TLT.2022.3154936
Chine DChhabra PAdeniran AKopko JTipper CGupta SKoedinger K(2022)Scenario-Based Training and On-The-Job Support for Equitable MentoringInnovative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education10.1007/978-3-031-21569-8_54(581-592)Online publication date: 16-Dec-2022
https://doi.org/10.1007/978-3-031-21569-8_54
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents