SC-Ques: A Sentence Completion Question Dataset for English as a Second Language Learners
Authors: 
Qiongqiong Liu, Yaying Huang, Zitao Liu, Shuyan Huang, Jiahao Chen, Xiangyu Zhao, Guimin Lin, Yuyu Zhou, Weiqi LuoAuthors Info & Claims
Qiongqiong Liu, Yaying Huang, Zitao Liu, Shuyan Huang, Jiahao Chen, Xiangyu Zhao, Guimin Lin, Yuyu Zhou, Weiqi LuoAuthors Info & ClaimsPages 678 - 690
Abstract
Sentence completion (SC) questions present a sentence with one or more blanks that need to be filled in, three to five possible words or phrases as options. SC questions are widely used for students learning English as a Second Language (ESL). In this paper, we present a large-scale SC dataset, SC-Ques, which is made up of 289,148 ESL SC questions from real-world standardized English examinations. Furthermore, we build a comprehensive benchmark of automatically solving the SC questions by training the large-scale pre-trained language models on the proposed SC-Ques dataset. We conduct detailed analysis of the baseline models performance, limitations and trade-offs. The data and our code are available for research purposes from: https://github.com/ai4ed/SC-Ques.
References
[1]
Argouarc’h, J.: Dependency, skip-grams, association tensors and machine learning for sentence completion. In: Proceedings of the 14th International Conference on Natural Language Processing (2018)
[2]
Banerjee, S., Bhaskar, P., Pakray, P., Bandyopadhyay, S., Gelbukh, A.: Multiple choice question (MCQ) answering system for entrance examination. In: Cross-Language Evaluation Forum (2013)
[3]
Beinborn, L., Zesch, T., Gurevych, I.: Candidate evaluation strategies for improved difficulty prediction of language tests. In: Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications (2015)
[4]
Beinborn, L.M.: Predicting and manipulating the difficulty of text-completion exercises for language learning. Ph.D. thesis, Technische Universität Darmstadt (2016)
[5]
Chen, J., Liu, Z., Huang, S., Liu, Q., Luo, W.: Improving interpretability of deep sequential knowledge tracing models with question-centric cognitive representations. In: AAAI Conference on Artificial Intelligence (2023)
[6]
Chen, M., D’Arcy, M., Liu, A., Fernandez, J., Downey, D.: CODAH: an adversarially-authored question answering dataset for common sense. In: Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP (2019)
[7]
Chen, S.F.: Performance prediction for exponential language models. In: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics (2009)
[8]
Chen, S.F.: Shrinking exponential language models. In: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics (2009)
[9]
Chesani, F., Mello, P., Milano, M.: Solving mathematical puzzles: a challenging competition for AI. AI Mag. 38(3) (2017)
[10]
Davey, G., De Lian, C., Higgins, L.: The university entrance examination system in china. J. Further High. Educ. 31(4) (2007)
[11]
Donahue, C., Lee, M., Liang, P.: Enabling language models to fill in the blanks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
[12]
Fedus, W., Goodfellow, I., Dai, A.M.: MaskGAN: better text generation via filling in the_. In: International Conference on Learning Representations (2018)
[13]
Franke, W.: The reform and abolition of the traditional Chinese examination system, vol. 10. Harvard University Asia Center (1960)
[14]
Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (2013)
[15]
Gubbins, J., Vlachos, A.: Dependency language models for sentence completion. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (2013)
[16]
He, P., Gao, J., Chen, W.: DeBERTaV3: improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In: The Eleventh International Conference on Learning Representations (2023)
[17]
He, P., Liu, X., Gao, J., Chen, W.: DeBERTa: decoding-enhanced BERT with disentangled attention. In: International Conference on Learning Representations (2021)
[18]
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2018)
[19]
Kenton, J.D.M.W.C., Toutanova, L.K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long and Short Papers) (2019)
[20]
Khot, T., Sabharwal, A., Clark, P.: Scitail: a textual entailment dataset from science question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
[21]
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
[22]
Lee, K., Lee, G.G.: Sentence completion task using web-scale data. In: 2014 International Conference on Big Data and Smart Computing (2014)
[23]
Lev, I.: Logic puzzles: a new test-suite for compositional semantics and reasoning (2006)
[24]
Lev, I., MacCartney, B., Manning, C.D., Levy, R.: Solving logic puzzles: from robust processing to precise semantics. In: Proceedings of the 2nd Workshop on Text Meaning and Interpretation (2004)
[25]
Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
[26]
Liu, D., Fu, J., Liu, P., Lv, J.: TIGS: an inference algorithm for text infilling with gradient search. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019)
[27]
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
[28]
Liu, Z., et al.: Enhancing deep knowledge tracing with auxiliary tasks. In: The International Conference of World Wide Web (2023)
[29]
Liu, Z., Liu, Q., Chen, J., Huang, S., Luo, W.: simpleKT: a simple but tough-to-beat baseline for knowledge tracing. In: International Conference on Learning Representations (2023)
[30]
Liu, Z., Liu, Q., Chen, J., Huang, S., Tang, J., Luo, W.: pyKT: a python library to benchmark deep learning based knowledge tracing models. In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022)
[31]
Madaus, G.F.: The effects of important tests on students: implications for a national examination system. Phi Delta Kappan 73(3) (1991)
[32]
Mikolov, T., Kombrink, S., Burget, L., Černockỳ, J., Khudanpur, S.: Extensions of recurrent neural network language model. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (2011)
[33]
Mohammad, S., Dorr, B., Hirst, G.: Computing word-pair antonymy. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (2008)
[34]
Ng, H.T., Teo, L.H., Kwan, J.L.P.: A machine learning approach to answering questions for reading comprehension tests. In: 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (2000)
[35]
Nozza, D., Bianchi, F., Hovy, D.: HONEST: measuring hurtful sentence completion in language models. In: The 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics (2021)
[36]
Park, H., Park, J.: Assessment of word-level neural language models for sentence completion. Appl. Sci. 10(4) (2020)
[37]
Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (2018)
[38]
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8) (2019)
[39]
Riloff, E., Thelen, M.: A rule-based question answering system for reading comprehension tests. In: ANLP-NAACL 2000 Workshop: Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Systems (2000)
[40]
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11) (1997)
[41]
Shen, T., Quach, V., Barzilay, R., Jaakkola, T.: Blank language models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (2020)
[42]
Tang, E.: Assessing the effectiveness of corpus-based methods in solving SAT sentence completion questions. J. Comput. 11(4) (2016)
[43]
Tran, K.M., Bisazza, A., Monz, C.: Recurrent memory networks for language modeling. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2016)
[44]
Turney, P.: A uniform approach to analogies, synonyms, antonyms, and associations. In: Proceedings of the 22nd International Conference on Computational Linguistics, Manchester, UK (2008)
[45]
Turney, P.D., Littman, M.L.: Corpus-based learning of analogies and semantic relations. Mach. Learn. 60(1–3) (2005)
[46]
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
[47]
Wang, D., et al.: CMU multiple-choice question answering system at NTCIR-11 QA-Lab. In: Proceedings of the 11th NTCIR Conference on Evaluation of Information Access Technologies (2014)
[48]
Woods, A.: Exploiting linguistic features for sentence completion. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (2016)
[49]
Yang, T., Deng, H.: Intelligent sentence completion based on global context dependent recurrent neural network LM. In: Proceedings of the International Conference on Artificial Intelligence, Information Processing and Cloud Computing (2019)
[50]
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
[51]
Zhu, W., Hu, Z., Xing, E.: Text infilling. arXiv preprint arXiv:1901.00158 (2019)
[52]
Zweig, G., Burges, C.J.: The Microsoft research sentence completion challenge. Technical report. Citeseer (2011)
[53]
Zweig, G., Platt, J.C., Meek, C., Burges, C.J., Yessenalina, A., Liu, Q.: Computational approaches to sentence completion. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2012)
Recommendations
Sentence Splitting for Vietnamese-English Machine Translation
KSE '12: Proceedings of the 2012 Fourth International Conference on Knowledge and Systems EngineeringTranslation quality is often disappointed when a phrase based machine translation system deals with long sentences. Because of syntactic structure discrepancy between two languages, the translation output will not preserve the same word order as the ...
Research on Element Sub-sentence in Chinese-English Patent Machine Translation
IALP '11: Proceedings of the 2011 International Conference on Asian Language ProcessingThis paper presents an approach to translate element sub-sentences which widely exist in Chinese patent documents. Element sub-sentence is a kind of language chunk in a sentence in which one part of sub-sentence is the headword and others are ...
Assessment of Multi-Engine Machine Translation for English to Hindi Language MEMTEHiL: Using F&A and iBLEU Metrics
English to Hindi translation of the computer-science related e-content, generated through an online freely available machine translation engine may not be technically correct. The expected target translation should be as fluent as intended for the ...
Comments
Information & Contributors
Information
Published In
Jun 2023
713 pages
ISBN:978-3-031-32882-4
DOI:10.1007/978-3-031-32883-1
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 02 June 2023
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 20 Dec 2024