Abstract
The assessment of the relevance of legal documents and the application of legal rules embodied in legal documents are some of the key processes in the field of law. In this paper, we present our approach to the 2020 Competition on Legal Information Extraction/Entailment (COLIEE-2020), which provides researchers with the opportunity to find ways of accomplishing these complex tasks using computers. Here, we describe the methods used to build the models for the four tasks that are part of the competition and the results of their application. For Task 1, concerning the prediction of whether a base case cites a candidate case, we devise a method for evaluating the similarity between cases based on individual paragraph similarity. This method can be used to reduce the number of candidate cases by 85%, while maintaining over 80% of the cited cases. We then train a Support Vector Machines model to make the final prediction. The model is the best solution submitted for Task 1. We use a similar method for Task 2. For Task 3, we use an approach based on BM25 measure in combination with the identification of similar previously asked questions. For Task 4, we use a transformer model fine-tuned on existing entailment data sets as well as on the provided domain-specific statutory law data set.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326 (2015)
Cer D., et al.: Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018)
Chalkidis, I., Androutsopoulos, I., Aletras, N.: Neural legal judgment prediction in English. arXiv preprint arXiv:1906.02059 (2019)
Condevaux, C., Harispe, S., Mussard, S., Zambrano, G.: Weakly supervised one-shot classification using recurrent neural networks with attention: application to claim acceptance detection. In: JURIX, pp. 23–32 (2019)
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995). https://doi.org/10.1007/BF00994018
El Hamdani, R., Trousssel, A. Houvenagel, C.: COLIEE case law competition task 1: the legal case retrieval task. In: COLIEE 2019, pp. 10–15 (2019)
Gain, B., Bandyopadhyay, D., Saikh, T., Ekbal, A.: Iitp@coliee 2019: legal information retrieval using BM25 and BERT. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment, COLIEE 2019 (2019)
Harter, S.P.: A probabilistic approach to automatic keyword indexing. Part I. On the distribution of specialty words in a technical literature. J. Am. Soc. Inf. Sci. 26(4), 197–206 (1975)
Howe, J.S.T., Khang, L.H., Chai, I.E.: Legal area classification: a comparative study of text classifiers on Singapore supreme court judgments. arXiv preprint arXiv:1904.06470 (2019)
Hudzina, J., Vacek, T., Madan, K., Tonya, C., Schilder, F.: Statutory entailment using similarity features and decomposable attention models. In: Proceedings of the 6th Competition on Legal Information Extraction/Entailment, COLIEE 2019 (2019)
Kitaev, N., Kaiser, Ł., Levskaya, A.: Reformer: the efficient transformer. arXiv preprint arXiv:2001.04451 (2020)
Kotsiantis, S., Kanellopoulos, D., Pintelas, P.: Handling imbalanced datasets: a review. GESTS Int. Trans. Comput. Sci. Eng. 30(1), 25–36 (2016)
Liu, Y., et al.: Roberta: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Manning, C.D., Schütze, H., Raghavan, P.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)
Mitra, B., Nalisnick, E., Craswell, N., Caruana, R.: A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137 (2016)
Murdock, V.G.: Aspects of sentence retrieval. Department of Computer Science, Massachusetts University Amherst (2006)
Paulino-Passos, G., Toni, F.: Retrieving legal cases with vector representations of text. In: Proceedings of COLIEE 2019, pp. 50–55 (2019)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Platt, J.C.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classif. 10, 61–74 (1999)
Rabelo, J., Kim, M.Y., Goebel, R.: Combining similarity and transformer methods for case law entailment. In: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law, pp. 290–296 (2019)
Robertson, S., Zaragoza, H.: The Probabilistic Relevance Framework: BM25 and Beyond. Now Publishers Inc., Delft (2009)
Robertson, S.E., Walker, S.: Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In: Croft, B.W., van Rijsbergen, C.J. (eds.) SIGIR’94, pp. 232–241. Springer, London (1994). https://doi.org/10.1007/978-1-4471-2099-5_24
Rossi, J., Kanoulas, E.: Legal search in case law and statute law. In: JURIX, pp. 83–92 (2019)
Sanchez, L., He, J., Manotumruksa, J., Albakour, D., Martinez, M., Lipani, A.: Easing legal news monitoring with learning to Rank and BERT. In: Jose, J.M., et al. (eds.) ECIR 2020. LNCS, vol. 12036, pp. 336–343. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45442-5_42
Savelka, J.: Discovering sentences for argumentation about the meaning of statutory terms. Doctoral dissertation, University of Pittsburgh (2020)
Savelka, J., Walker, V.R., Grabmair, M., Ashley, K.D.: Sentence boundary detection in adjudicatory decisions in the United States. Traitement automatique des langues 58, 21 (2017)
Shao, Y., et al.: BERT-PLI: modeling paragraph-level interactions for legal case retrieval. In: Electronic Proceedings of IJCAI 2020, vol. 4, pp. 3501–3507 (2020)
Westermann, H., Savelka, J., Walker, V.R., Ashley, K.D., Benyekhlef, K.: Sentence embeddings and high-speed similarity search for fast computer assisted annotation of legal documents. In: Legal Knowledge and Information Systems: JURIX 2020, Brno, Czech Republic, 9–11 December 2020, vol. 334. IOS Press (2020)
Williams, A., Nangia, N., Bowman, S.R.: A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 (2017)
Zhang, Z., et al.: Semantics-aware BERT for language understanding. arXiv preprint arXiv:1909.02209 (2019)
Acknowledgements
We would like to thank the Cyberjustice Laboratory at Université de Montréal, the LexUM Chair on Legal Information, and the Autonomy through Cyberjustice Technologies (ACT) project for their support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Westermann, H., Savelka, J., Benyekhlef, K. (2021). Paragraph Similarity Scoring and Fine-Tuned BERT for Legal Information Retrieval and Entailment. In: Okazaki, N., Yada, K., Satoh, K., Mineshima, K. (eds) New Frontiers in Artificial Intelligence. JSAI-isAI 2020. Lecture Notes in Computer Science(), vol 12758. Springer, Cham. https://doi.org/10.1007/978-3-030-79942-7_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-79942-7_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-79941-0
Online ISBN: 978-3-030-79942-7
eBook Packages: Computer ScienceComputer Science (R0)