Abstract
Large Language Models (LLMs) have experienced significant advancements across various contexts. However, their impact on vertical fields remains understudied and unsatisfactory due to the heightened requirement for domain-specific expertise in these fields. English Grammar Error Correction (GEC) is urgently needed in the current academic and educational fields, which are currently full of challenges regarding precision, adaptability, and complex grammatical mistakes. The release of the C4_200M Synthetic Dataset and advancements in LLaMA2’s QLoRA fine-tuning technology present an unprecedented opportunity to examine these issues more closely. This study aims to assess the performance of the LLaMA2 in the area of GEC. In this study, we implemented LLaMA2 augmented with QLoRA finetune model in Spark scalable cluster processing environment, and we investigated model performance under two methods, Zero-shot and Few-shot prompting, and configured the parameters for text generation, including Top-p, Top-k, and Beam search. We built an efficient and accurate scalable system, with BLEU from 12.33 to 14.8, ROUGE from 19.33% to 25.97% and the editing distance from 4.21 to 1.89, providing a solid foundation for future work. The code of this paper is available at LINK.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Touvron, H., Lavril, T., Izacard, G., et al.: LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
Touvron, H., Martin, L., Stone, K., et al.: Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
Pavlyshenko, B.M.: Financial news analytics using fine-tuned Llama 2 GPT model. arXiv preprint arXiv:2308.13032 (2023)
Zhao, H., et al.: Ophtha-LLaMA2: a large language model for ophthalmology. arXiv preprint arXiv:2312.04906 (2023)
Nguyen, T.T., et al.: Fine-tuning Llama 2 large language models for detecting online sexual predatory chats and abusive texts. arXiv preprint arXiv:2308.14683 (2023)
Stahlberg, F., Kumar, S.: Synthetic data generation for grammatical error correction with tagged corruption models. arXiv preprint arXiv:2105.13318 (2021)
Dettmers, T., et al.: QLoRA: efficient finetuning of quantized LLMs. arXiv preprint arXiv:2305.14314 (2024)
Fan, Y., et al.: GrammarGPT: exploring open-source LLMs for native Chinese grammatical error correction with supervised fine-tuning. arXiv preprint arXiv:2307.13923 (2023)
Deligiannis, P., et al.: Fixing rust compilation errors using LLMs. arXiv preprint arXiv:2308.05177 (2023)
Song, Y., et al.: GEE! Grammar error explanation with large language models. arXiv preprint arXiv:2311.09517 (2023)
Davis, C., et al.: Prompting open-source and commercial language models for grammatical error correction of English learner text. arXiv preprint arXiv:2401.07702 (2024)
Penteado, M.C., Perez, F.: Evaluating GPT-3.5 and GPT-4 on grammatical error correction for Brazilian Portuguese. arXiv preprint arXiv:2306.15788 (2023)
Zhang, Y., et al.: RobustGEC: robust grammatical error correction against subtle context perturbation. arXiv preprint arXiv:2310.07299 (2023)
Yang, C.-H.H., Gu, Y., Liu, Y.-C., Ghosh, S., Bulyko, I., Stolcke, A.: Generative speech recognition error correction with large language models and task-activating prompting. arXiv preprint arXiv:2309.15649 (2023)
Kaneko, M., Okazaki, N.: Controlled generation with prompt insertion for natural language explanations in grammatical error correction. arXiv preprint arXiv:2309.11439 (2023)
Kaddour, J., Liu, Q.: Text data augmentation in low-resource settings via fine-tuning of large language models. arXiv preprint arXiv:2310.01119 (2023)
Ji, Y., et al.: Exploring the impact of instruction data scaling on large language models: an empirical study on real-world use cases. arXiv preprint arXiv:2303.14742 (2023)
Chen, H., et al.: Maybe only 0.5% data is needed: a preliminary exploration of low training data instruction tuning. arXiv preprint arXiv:2305.09246 (2023)
Zhou, C., et al.: LIMA: less is more for alignment. In: Thirty-Seventh Conference on Neural Information Processing Systems (2023)
Acknowledgments
This work was jointly supported by National Natural Science Foundation of China (NSFC) under grants 62206301; Public Health & Disease Control and Prevention, Fund for Building World-Class Universities (Disciplines) of Renmin University of China. Project No. 2024PDPC; the Major Project of the MOE (China) National Key Research Bases for Humanities and Social Sciences (22JJD910003); Wine Group’s research grant No. 09202188. This work was supported by Public Computing Cloud, Renmin University of China. We sincerely thank the students at Renmin University of China for providing data processing and experiment support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
An, J. et al. (2024). Evaluating Performance of LLaMA2 Large Language Model Enhanced by QLoRA Fine-Tuning for English Grammatical Error Correction. In: Strauss, C., Amagasa, T., Manco, G., Kotsis, G., Tjoa, A.M., Khalil, I. (eds) Database and Expert Systems Applications. DEXA 2024. Lecture Notes in Computer Science, vol 14910. Springer, Cham. https://doi.org/10.1007/978-3-031-68309-1_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-68309-1_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-68308-4
Online ISBN: 978-3-031-68309-1
eBook Packages: Computer ScienceComputer Science (R0)