Optimally Discriminative Choice Sets in Discrete Choice Models: Application to Data-Driven Test Design
Authors: 
Igor Labutov, Frans Schalekamp, Kelvin Luu, Hod Lipson, Christoph StuderAuthors Info & Claims
Igor Labutov, Frans Schalekamp, Kelvin Luu, Hod Lipson, Christoph StuderAuthors Info & ClaimsPages 1665 - 1674
Abstract
Difficult multiple-choice (MC) questions can be made easy by providing a set of answer options of which most are obviously wrong. In the education literature, a plethora of instructional guides exist for crafting a suitable set of wrong choices (distractors) that enable the assessment of the students' understanding. The art of MC question design thus hinges on the question-maker's experience and knowledge of the potential misconceptions. In contrast, we advocate a data-driven approach, where correct and incorrect options are assembled directly from the students' own past submissions. Large-scale online classroom settings, such as massively open online courses (MOOCs), provide an opportunity to design optimal and adaptive multiple-choice questions that are maximally informative about the students' level of understanding of the material. In this work, we (i) develop a multinomial-logit discrete choice model for the setting of MC testing, (ii) derive an optimization objective for selecting optimally discriminative option sets, (iii) propose an algorithm for finding a globally-optimal solution, and (iv) demonstrate the effectiveness of our approach via synthetic experiments and a user study. We finally showcase an application of our approach to crowd-sourcing tests from technical online forums.
References
[1]
Y. Bachrach, T. Graepel, T. Minka, and J. Guiver. How to grade a test without knowing the answers|a bayesian graphical model for adaptive crowdsourcing and aptitude testing. arXiv preprint arXiv:1206.6386, 2012.
[2]
I. Bejar. A sentence-based automated approach to the assessment of writing: A feasibility study. Machine-Mediated Learning, 2(4):321--332, 1987.
[3]
J. C. Brown, G. A. Frishko, and M. Eskenazi. Automatic question generation for vocabulary assessment. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 819--826. Association for Computational Linguistics, 2005.
[4]
C. H. Coombs. Psychological scaling without a unit of measurement. Psychological review, 57(3):145, 1950.
[5]
G. H. Fischer and I. W. Molenaar. Rasch models: Foundations, recent developments, and applications. Springer Science & Business Media, 2012.
[6]
R. Fortet. L'Algèbre de Boole et ses applications en recherche opérationnelle. Cahiers Centre Etudes Rech. Oper. no., 4:5--36, 1959.
[7]
T. M. Haladyna. Writing Test Items To Evaluate Higher Order Thinking. ERIC, 1997.
[8]
T. M. Haladyna and S. M. Downing. Validity of a taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1):51--78, 1989.
[9]
T. M. Haladyna and S. M. Downing. How many options is enough for a multiple-choice test item? Educational and Psychological Measurement, 53(4):999--1010, 1993.
[10]
T. M. Haladyna, S. M. Downing, and M. C. Rodriguez. A review of multiple-choice item-writing guidelines for classroom assessment. Applied measurement in education, 15(3):309--333, 2002.
[11]
F. M. Lord. Applications of item response theory to practical testing problems. Routledge, 1980.
[12]
T. Minka, J. Winn, J. Guiver, and D. Knowles. Infer.net 2.5. Microsoft Research Cambridge, 2012.
[13]
R. Mitkov, L. An Ha, and N. Karamanis. A computer-aided environment for generating multiple-choice test items. Natural Language Engineering, 12(02):177--194, 2006.
[14]
P. Ray. Independence of irrelevant alternatives. Econometrica: Journal of the Econometric Society, pages 987--991, 1973.
[15]
M. C. Rodriguez. Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2):3--13, 2005.
[16]
P. H. Schönemann. On metric multidimensional unfolding. Psychometrika, 35(3):349--366, 1970.
[17]
D. Thissen, L. Steinberg, and A. R. Fitzpatrick. Multiple-choice models: The distractors are also part of the item. Journal of Educational Measurement, 26(2):161--176, 1989.
[18]
D. Vats, C. Studer, A. S. Lan, L. Carin, and R. Baraniuk. Test-size reduction for concept estimation. In Educational Data Mining 2013, 2013.
[19]
A. E. Waters, A. Lan, C. Studer, and R. G. Baraniuk. Learning analytics via sparse factor analysis. In Personalizing education with machine learning, nips 2012 workshop, 2012.
[20]
D. J. Weiss. Improving measurement quality and efficiency with adaptive testing. Applied psychological measurement, 6(4):473--492, 1982.
Index Terms
- Optimally Discriminative Choice Sets in Discrete Choice Models: Application to Data-Driven Test Design
Recommendations
Optimally Discriminative Choice Sets in Discrete Choice Models: Application to Data-Driven Test Design
L@S '16: Proceedings of the Third (2016) ACM Conference on Learning @ ScaleDifficult test questions can be made easy by providing a set of possible answer options of which most are obviously wrong. In the education literature, a plethora of instructional guides exist for crafting a suitable set of wrong choices (distractors) ...
Designing Adaptive Assessments in MOOCs
L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ ScaleThere is an indisputable need for evidence-based instructional designs that create the optimal conditions for learners with different knowledge, skills and motivations to succeed in MOOCs. The study explores the technological feasibility and ...
Design of adaptive experiences in higher education through a learning management system
TEEM '15: Proceedings of the 3rd International Conference on Technological Ecosystems for Enhancing MulticulturalityIn this paper, several experiences on adaptive learning at higher education are shown. Different contexts and methodologies such as Problems and Projects Based Learning, final works of Grade and Massive Open Online Courses, present different needs ...
Comments
Information & Contributors
Information
Published In
August 2016
2176 pages
ISBN:9781450342322
DOI:10.1145/2939672
- General Chairs:
- Balaji Krishnapuram,
- Mohak Shah,
- Program Chairs:
- Alex Smola,
- Charu Aggarwal,
- Dou Shen,
- Rajeev Rastogi
Copyright © 2016 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 August 2016
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Templeton Foundation
- NSF
Conference
KDD '16: The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
August 13 - 17, 2016
California, San Francisco, USA
Acceptance Rates
KDD '16 Paper Acceptance Rate 66 of 1,115 submissions, 6%;
Overall Acceptance Rate 1,133 of 8,635 submissions, 13%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 238Total Downloads
- Downloads (Last 12 months)61
- Downloads (Last 6 weeks)10
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in