Abstract
With the recent rise of large language models (LLMs), in-context learning (ICL) has shown remarkable performance, eliminating the need for fine-tuning parameters and reducing the reliance on extensive labeled data. However, the intricacies of cross-lingual ICL remain underexplored. Prior studies on cross-lingual ICL overlooked the significance of language-specific nuances, neglecting the intrinsic linguistic properties of sentences and the interlingual connections between sentences in different languages. In this paper, we propose a novel cross-lingual prompt structure: Language-Emphasized cross-lingual In-context learning (LEI). LEI teaches LLMs how to adapt to language conversion by adding explicit language conversion examples in demonstrations. Specifically, LEI introduces a third language (example language) as an example of language conversion to adapt LLMs to language conversion in cross-lingual tasks. In addition, language alignment of demonstrations is achieved by adding language aligners and label aligners. Extensive experiments validate the state-of-the-art performance of LEI on 42 cross-lingual tasks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)
Chen, G., et al.: Zero-shot cross-lingual transfer of neural machine translation with multilingual pretrained encoders. arXiv preprint arXiv:2104.08757 (2021)
Chowdhery, A., et al.: Palm: scaling language modeling with pathways. J. Mach. Learn. Res. 24(240), 1–113 (2023)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Dong, Q., et al.: A survey for in-context learning. arXiv preprint arXiv:2301.00234 (2022)
Keung, P., Lu, Y., Szarvas, G., Smith, N.A.: The multilingual amazon reviews corpus. arXiv preprint arXiv:2010.02573 (2020)
Lin, X.V., et al.: Few-shot learning with multilingual generative language models. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9019–9052 (2022)
Lu, Y., Bartolo, M., Moore, A., Riedel, S., Stenetorp, P.: Fantastically ordered prompts and where to find them: overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786 (2021)
Mikolov, T., Le, Q.V., Sutskever, I.: Exploiting similarities among languages for machine translation. Comput. Sci. (2013)
Min, S., et al.: Rethinking the role of demonstrations: what makes in-context learning work? In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048–11064 (2022)
Prettenhofer, P., Stein, B.: Cross-language text classification using structural correspondence learning. In: Proceedings of the 48th Annual Meeting Of The Association For Computational Linguistics, pp. 1118–1127 (2010)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
Reimers, N., Gurevych, I.: Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813 (2020)
Tanwar, E., Borthakur, M., Dutta, S., Chakraborty, T.: Multilingual LLMs are better cross-lingual in-context learners with alignment. arXiv preprint arXiv:2305.05940 (2023)
Webson, A., Pavlick, E.: Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247 (2021)
Wei, J., et al.: Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural. Inf. Process. Syst. 35, 24824–24837 (2022)
Winata, G.I., Madotto, A., Lin, Z., Liu, R., Yosinski, J., Fung, P.: Language models are few-shot multilingual learners. arXiv preprint arXiv:2109.07684 (2021)
Zhang, N., et al.: Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2108.13161 (2021)
Zhao, Z., Wallace, E., Feng, S., Klein, D., Singh, S.: Calibrate before use: improving few-shot performance of language models. In: International Conference on Machine Learning, pp. 12697–12706. PMLR (2021)
Acknowledgements
This study was funded by National Natural Science Foundation of China (62302333) and the National Natural Science Foundation of China under Grant(U23B2053).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Appendix
The complete experimental data in the main experiment (including all the selected example languages) are shown in Table 4.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, J., Wei, X., Wang, X., Zhuang, N., Wang, L., Dang, J. (2025). Language-Emphasized Cross-Lingual In-Context Learning for Multilingual LLM. In: Wong, D.F., Wei, Z., Yang, M. (eds) Natural Language Processing and Chinese Computing. NLPCC 2024. Lecture Notes in Computer Science(), vol 15361. Springer, Singapore. https://doi.org/10.1007/978-981-97-9437-9_26
Download citation
DOI: https://doi.org/10.1007/978-981-97-9437-9_26
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-9436-2
Online ISBN: 978-981-97-9437-9
eBook Packages: Computer ScienceComputer Science (R0)
