Abstract
The escalating volume of cases in legal adjudication has amplified the complexity of citing relevant regulations and authoritative cases, posing an increasing challenge for legal professionals. Current legal citation prediction methods, which are predominantly reliant on keyword or interest-based retrieval, are proving insufficient. In particular, Collaborative Filtering (CF) based legal recommendation methods exhibited low accuracy. In response to these challenges, we propose the Instruction GPT with Low-Rank Adaptation architecture (IGPT-LoRA), aiming to enhance the performance of legal citation recommendations and reduce computational demands by tuning Pre-trained Language Models (PLMs). IGPT-LoRA leverages prompting and efficient tuning strategies, thus offering a significant improvement over previous context-aware legal citation prediction methods. We design effective domain-specific instruction templates to guide the adaptation of PLMs for recommendation purposes, shedding light on the potential of prompt-based learning in the legal domain. Furthermore, we optimize the learning process with an efficient tuning layer - the Low-Rank Adaptation (LoRA) architecture - to bolster applicability. Experimental results on a real-world legal data set (BVA) demonstrate that IGPT-LoRA outperforms state-of-the-art methods, delivering substantial improvements in accuracy and also in training time and computational efficiency.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
Overall, we observed a slight decrease in the range of 0.2% to 0.4%.
References
Ames, D., Handan-Nader, C., Ho, D.E., Marcus, D.: Due process and mass adjudication: crisis and reform. Stan. L. Rev. 72, 1 (2020)
Brin, S.: The PageRank citation ranking: bringing order to the web. In: Proceedings of ASIS, vol. 98, pp. 161–172 (1998)
Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)
Caragea, C., Silvescu, A., Mitra, P., Giles, C.L.: Can’t see the forest for the trees? A citation recommendation system. In: Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 111–114 (2013)
Chowdhery, A., et al.: PaLM: scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)
Dadgostari, F., Guim, M., Beling, P.A., Livermore, M.A., Rockmore, D.N.: Modeling law search as prediction. Artif. Intell. Law 29, 3–34 (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Fowler, J.H., Johnson, T.R., Spriggs, J.F., Jeon, S., Wahlbeck, P.J.: Network analysis and the law: measuring the legal importance of precedents at the us supreme court. Polit. Anal. 15(3), 324–346 (2007)
Ge, X., Chen, F., Jose, J.M., Ji, Z., Wu, Z., Liu, X.: Structured multi-modal feature embedding and alignment for image-sentence retrieval. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 5185–5193 (2021)
Ge, X., Chen, F., Xu, S., Tao, F., Jose, J.M.: Cross-modal semantic enhanced interaction for image-sentence retrieval. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pp. 1022–1031 (2023)
Gori, M., Pucci, A.: Research paper recommender systems: a random-walk based approach. In: 2006 IEEE/WIC/ACM International Conference on Web Intelligence (WI 2006 Main Conference Proceedings)(WI 2006), pp. 778–781. IEEE (2006)
Gunawardana, A., Shani, G.: Evaluating recommender systems. In: Ricci, F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook, pp. 265–308. Springer, Boston, MA (2015). https://doi.org/10.1007/978-1-4899-7637-6_8
He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., Neubig, G.: Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366 (2021)
He, Q., Pei, J., Kifer, D., Mitra, P., Giles, L.: Context-aware citation recommendation. In: Proceedings of the 19th International Conference on World Wide Web, pp. 421–430 (2010)
He, X., Liao, L., Zhang, H., Nie, L., Hu, X., Chua, T.S.: Neural collaborative filtering. In: Proceedings of the 26th International Conference on World Wide Web, pp. 173–182 (2017)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Houlsby, N., et al.: Parameter-efficient transfer learning for NLP. In: International Conference on Machine Learning, pp. 2790–2799. PMLR (2019)
Hu, E.J., et al.: LoRA: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
Hu, Z., et al.: LLM-adapters: an adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933 (2023)
Huang, Z., et al.: Context-aware legal citation recommendation using deep learning. In: Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pp. 79–88 (2021)
Jeh, G., Widom, J.: SimRank: a measure of structural-context similarity. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge dIscovery and Data Mining, pp. 538–543 (2002)
Kim, S., Gholami, A., Yao, Z., Mahoney, M.W., Keutzer, K.: I-BERT: integer-only BERT quantization. In: International Conference on Machine Learning, pp. 5506–5518. PMLR (2021)
Lai, G., Xie, Q., Liu, H., Yang, Y., Hovy, E.: RACE: large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683 (2017)
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: ALBERT: a lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019)
Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019)
Li, P.H., Fu, T.J., Ma, W.Y.: Why attention? Analyze BiLSTM deficiency and its remedies in the case of NER. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 8236–8244 (2020)
Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021)
Liben-Nowell, D., Kleinberg, J.: The link prediction problem for social networks. In: Proceedings of the Twelfth International Conference on Information and Knowledge Management, pp. 556–559 (2003)
Liu, X., et al.: P-Tuning v2: prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 (2021)
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Ma, J., Ganchev, K., Weiss, D.: State-of-the-art Chinese word segmentation with Bi-LSTMs. arXiv preprint arXiv:1808.06511 (2018)
McNee, S.M., et al.: On the recommending of citations for research papers. In: Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work, pp. 116–125 (2002)
Min, B., et al.: Recent advances in natural language processing via large pre-trained language models: a survey. ACM Comput. Surv. 56(2), 1–40 (2023)
Ouyang, L., et al.: Training language models to follow instructions with human feedback. Adv. Neural. Inf. Process. Syst. 35, 27730–27744 (2022)
Pfeiffer, J., et al.: AdapterHub: a framework for adapting transformers. arXiv preprint arXiv:2007.07779 (2020)
Pfeiffer, J., Vulić, I., Gurevych, I., Ruder, S.: MAD-X: an adapter-based framework for multi-task cross-lingual transfer. arXiv preprint arXiv:2005.00052 (2020)
Prasanna, S., Rogers, A., Rumshisky, A.: When BERT plays the lottery, all tickets are winning. arXiv preprint arXiv:2005.00561 (2020)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020)
Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: unanswerable questions for squad. arXiv preprint arXiv:1806.03822 (2018)
Ritchie, A., Robertson, S., Teufel, S.: Comparing citation contexts for information retrieval. In: Proceedings of the 17th ACM Conference on Information and Knowledge Management, pp. 213–222 (2008)
Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)
Stamenkovic, D., Karatzoglou, A., Arapakis, I., Xin, X., Katevas, K.: Choosing the best of both worlds: diverse and novel recommendations through multi-objective reinforcement learning. In: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 957–965 (2022)
Touvron, H., et al.: Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
Wang, J., et al.: TransRec: learning transferable recommendation from mixture-of-modality feedback. arXiv preprint arXiv:2206.06190 (2022)
Wang, J., Zhu, L., Dai, T., Wang, Y.: Deep memory network with Bi-LSTM for personalized context-aware citation recommendation. Neurocomputing 410, 103–113 (2020)
Wang, J., Zhu, L., Dai, T., Xu, Q., Gao, T.: Low-rank and sparse matrix factorization with prior relations for recommender systems. Appl. Intell. 51, 3435–3449 (2021)
Winkels, R., Boer, A., Vredebregt, B., Van Someren, A.: Towards a legal recommender system. In: JURIX, pp. 169–178 (2014)
Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020)
Zhang, P., Koppaka, L.: Semantics-based legal citation network. In: Proceedings of the 11th International Conference on Artificial Intelligence and Law, pp. 123–130 (2007)
Zhang, S., Yao, L., Sun, A., Tay, Y.: Deep learning based recommender system: a survey and new perspectives. ACM Comput. surv. (CSUR) 52(1), 1–38 (2019)
Zhang, T., Zhu, L., Wang, J.: Neighborhood constraints based bayesian personalized ranking for explainable recommendation. In: Li, B., Yue, L., Tao, C., Han, X., Calvanese, D., Amagasa, T. (eds.) Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data, pp. 166–173. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-25201-3_12
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, J., Bansal, K., Arapakis, I., Ge, X., Jose, J.M. (2024). Empowering Legal Citation Recommendation via Efficient Instruction-Tuning of Pre-trained Language Models. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14608. Springer, Cham. https://doi.org/10.1007/978-3-031-56027-9_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-56027-9_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-56026-2
Online ISBN: 978-3-031-56027-9
eBook Packages: Computer ScienceComputer Science (R0)