<inline-formula><tex-math notation="LaTeX">$\boldsymbol{R}^{2}$</tex-math></inline-formula>: A Novel Recall & Ranking Framework for Legal Judgment Prediction
Authors: 
Yuquan Le, Zhe Quan, Jiawei Wang, Da Cao, Kenli LiAuthors Info & Claims
Yuquan Le, Zhe Quan, Jiawei Wang, Da Cao, Kenli LiAuthors Info & ClaimsPages 1609 - 1622
Abstract
The legal judgment prediction (LJP) task is to automatically decide appropriate law articles, charges, and term of penalty for giving the fact description of a law case. It considerably influences many real legal applications and has thus attracted the attention of legal practitioners and AI researchers in recent years. In real scenarios, many confusing charges are encountered, which makes LJP challenging. Intuitively, for a controversial legal case, legal practitioners usually first obtain various possible judgment results as candidates based on the fact description of the case; then these candidates generally need to be carefully considered based on the facts and the rationality of the candidates. Inspired by this observation, this paper presents a novel <bold>R</bold>ecall & <bold>R</bold>anking framework, dubbed as <inline-formula><tex-math notation="LaTeX">$\boldsymbol{R}^{2}$</tex-math></inline-formula>, which attempts to formalize LJP as a two-stage problem. The recall stage is designed to collect high-likelihood judgment results for a given case; these results are regarded as candidates for the ranking stage. The ranking stage introduces a verification technique to learn the relationships between the fact description and the candidates. It treats the partially correct candidates as semi-negative samples, and thus has a certain ability to distinguish confusing candidates. Moreover, we devise a comprehensive judgment strategy to refine the final judgment results by comprehensively considering the rationality of multiple probable candidates. We carry out numerous experiments on two widely used benchmark datasets. The experimental results demonstrate our proposed approach's effectiveness compared to the other competitive baselines.
References
[1]
J. Cui, X. Shen, and S. Wen, “A survey on legal judgment prediction: Datasets, metrics, models and challenges,” IEEE Access, pp. 102050–102071, 2023.
[2]
H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, and M. Sun, “How does NLP benefit legal system: A summary of legal artificial intelligence,” in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics, 2020, pp. 5218–5230.
[3]
H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, and M. Sun, “JEC-QA: A legal-domain question answering dataset,” in Proc. AAAI Conf. Artif. Intell., 2020, pp. 9701–9708.
[4]
B. Luo, Y. Feng, J. Xu, X. Zhang, and D. Zhao, “Learning to predict charges for criminal cases with legal basis,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2017, pp. 2727–2736.
[5]
J. Ge, Y. Huang, X. Shen, C. Li, and W. Hu, “Learning fine-grained fact-article correspondence in legal cases,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 29, pp. 3694–3706, 2021.
[6]
L. Ma et al., “Legal judgment prediction with multi-stage case representation learning in the real court setting,” in Proc. 44th Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2021, pp. 993–1002.
[7]
H. Zhong, Z. Guo, C. Tu, C. Xiao, Z. Liu, and M. Sun, “Legal judgment prediction via topological learning,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2018, pp. 3540–3549.
[8]
W. Yang, W. Jia, X. Zhou, and Y. Luo, “Legal judgment prediction via multi-perspective BI-feedback network,” in Proc. 28th Int. Joint Conf. Artif. Intell., 2019, pp. 4085–4091.
[9]
N. Xu, P. Wang, L. Chen, L. Pan, X. Wang, and J. Zhao, “Distinguish confusing law articles for legal judgment prediction,” in Proc. 58th Annu. Meeting Assoc. Computat. Linguistics, 2020, pp. 3086–3095.
[10]
Q. Dong and S. Niu, “Legal judgment prediction via relational learning,” in Proc. 44th Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2021, pp. 983–992.
[11]
Y. Feng, C. Li, and V. Ng, “Legal judgment prediction via event extraction with constraints,” in Proc. 60th Annu. Meeting Assoc. Computat. Linguistics, 2022, pp. 648–664.
[12]
L. Yue et al., “Neurjudge: A circumstance-aware neural framework for legal judgment prediction,” in Proc. 44th Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2021, pp. 973–982.
[13]
Z. Hu, X. Li, C. Tu, Z. Liu, and M. Sun, “Few-shot charge prediction with discriminative legal attributes,” in Proc. 27th Int. Conf. Computat. Linguistics, 2018, pp. 487–498.
[14]
J. Shen et al., “Generate & rank: A multi-task framework for math word problems,” in Proc. Findings Assoc. Computat. Linguistics, 2021, pp. 2269–2279.
[15]
K. Cobbe et al., “Training verifiers to solve math word problems,” 2021, arXiv:2110.14168.
[16]
A. Vaswani et al., “Attention is all you need,” in Proc. 31st Int. Adv. Conf. Neural Inf. Process. Syst., 2017, pp. 5998–6008.
[17]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. Annu. Conf. North Amer. Chapter Assoc. Computat. Linguistics, 2019, pp. 4171–4186.
[18]
Y. Kim, “Convolutional neural networks for sentence classification,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2014, pp. 1746–1751.
[19]
K. Cho et al., “Learning phrase representations using RNN encoder–decoder for statistical machine translation,” in Proc. Conf. Empirical Methods Natural Lang. Process. 2014, pp. 1724–1734.
[20]
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[21]
H. Valizadegan, R. Jin, R. Zhang, and J. Mao, “Learning to rank by optimizing NDCG measure,” in Proc. 22nd Int. Adv. Conf. Neural Inf. Process. Syst., 2009, pp. 1883–1891.
[22]
J. Wang et al., “Learning fine-grained image similarity with deep ranking,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 1386–1393.
[23]
A. Rush, Y.-W. Chang, and M. Collins, “Optimal beam search for machine translation,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2013, pp. 210–221.
[24]
S. Wiseman and A. M. Rush, “Sequence-to-sequence learning as beam-search optimization,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2016, pp. 1296–1306.
[25]
Y. Wang, X. Liu, and S. Shi, “Deep neural solver for math word problems,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2017, pp. 845–854.
[26]
D. Huang, J. Liu, C.-Y. Lin, and J. Yin, “Neural math word problem solver with reinforcement learning,” in Proc. 27th Int. Conf. Computat. Linguistics, 2018, pp. 213–223.
[27]
L. Xiao, X. Huang, B. Chen, and L. Jing, “Label-specific document representation for multi-label text classification,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2019, pp. 466–475.
[28]
Z. Liu, C. Tu, and M. Sun, “Legal cause prediction with inner descriptions and outer hierarchies,” in China Nat. Conf. Chin. Computat. Linguistics, 2019, pp. 573–586.
[29]
C. Xiao et al., “CAIL2018: A large-scale legal dataset for judgment prediction,” 2018, arXiv:1807.02478.
[30]
Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, “Hierarchical attention networks for document classification,” in Proc. Conf. North Amer. Chapter Assoc. Computat. Linguistics: Hum. Lang. Technol., 2016, pp. 1480–1489.
[31]
Z. Wang, H. Mi, and A. Ittycheriah, “Sentence similarity learning by lexical decomposition and composition,” in Proc. 23rd Int. Conf. Computat. Linguistics, 2016, pp. 1340–1349.
[32]
Y.-H. Liu, Y.-L. Chen, and W.-L. Ho, “Predicting associated statutes for legal problems,” Inf. Process. Manage., vol. 51, no. 1, pp. 194–211, 2015.
[33]
F. Kort, “Predicting supreme court decisions mathematically: A quantitative analysis of the “right to counsel” cases.,” Amer. Political Sci. Rev., vol. 51, no. 1, pp. 1–12, 1957.
[34]
E. Mackaay and P. Robillard, “Predicting judicial decisions: The nearest neighbour rule and visual representation of case patterns,” Datenverarbeitung im Recht, vol. 3, no. 34, pp. 302–331, 1974.
[35]
S. S. Nagel, “Applying correlation analysis to case prediction,” Texas Law Rev., vol. 42, 1963, Art. no.
[36]
R. Keown, “Mathematical models for legal prediction,” Computer/LJ, vol. 2, 1980, Art. no.
[37]
C.-L. Liu, C.-T. Chang, and J.-H. Ho, “Case instance generation and refinement for case-based criminal summary judgments in chinese,” J. Inf. Sci. Eng., vol. 20, no. 4, pp. 783–800, 2004.
[38]
C.-L. Liu and C.-D. Hsieh, “Exploring phrase-based classification of judicial documents for criminal charges in chinese,” in Proc. Int. Symp. Methodologies Intell. Syst.. Springer, 2006, pp. 681–690.
[39]
W.-C. Lin, T.-T. Kuo, T.-J. Chang, C.-A. Yen, C.-J. Chen, and S.-d. Lin, “Exploiting machine learning models for Chinese legal documents labeling, case classification, and sentencing prediction,” Int. J. Computat. Linguistics Chin. Lang. Process., vol. 17, pp. 49–68, 2012.
[40]
D. M. Katz, M. J. Bommarito II, and J. Blackman, “A general approach for predicting the behavior of the supreme court of the United States,” PLoS One, vol. 12, no. 4, 2017, Art. no.
[41]
O.-M. Sulea, M. Zampieri, S. Malmasi, M. Vela, L. P. Dinu, and J. van Genabith, “Exploring the use of text classification in the legal domain,” in Proc. Automat. Semantic Anal. Inf. Legal Texts (ASAIL) Workshop, 2017, pp. 1–5.
[42]
P. Wang, Z. Yang, S. Niu, Y. Zhang, L. Zhang, and S. Niu, “Modeling dynamic pairwise attention for crime classification over legal articles,” in 41st Int. ACM SIGIR Conf. Res. & Develop. Inf. Retrieval, 2018, pp. 485–494.
[43]
C. He, L. Peng, Y. Le, J. He, and X. Zhu, “Secaps: A sequence enhanced capsule model for charge prediction,” in Int. Conf. Artif. Neural Networks, 2019, pp. 227–239.
[44]
Y. Le, C. He, M. Chen, Y. Wu, X. He, and B. Zhou, “Learning to predict charges for legal judgment via self-attentive capsule network,” in Proc. Eur. Conf. Artif. Intell., 2020, pp. 1802–1809.
[45]
P. Wang, Y. Fan, S. Niu, Z. Yang, Y. Zhang, and J. Guo, “Hierarchical matching network for crime classification,” in Proc. 42nd Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2019, pp. 325–334.
[46]
G. Wang et al., “Joint embedding of words and labels for text classification,” in Proc. 56th Annu. Meeting Assoc. Computat. Linguistics, 2018, pp. 2321–2331.
[47]
A. Aizawa, “An information-theoretic perspective of tf–idf measures,” Inf. Process. Manage., vol. 39, no. 1, pp. 45–65, 2003.
Index Terms
$\boldsymbol{R}^{2}$
Index terms have been assigned to the content through auto-classification.
Recommendations
Discovering the
$k$ <italic>A representative skyline</italic> contains <inline-formula><tex-math notation="LaTeX">$k$ </tex-math><alternatives><inline-graphic xlink:type="simple" xlink:href="bai-ieq2-2546242.gif"/></alternatives></inline-formula> skyline points that can ...
Confidence Interval for
${\bm {F_1}}$ $\bm {\times}$ In studies on the application of machine learning such as Information Retrieval (IR), the focus is typically on the estimation of the F<sub>1</sub> measure of algorithm performance. Approximate symmetrical confidence intervals constructed by the F<sub>1</...
$\tt{PoisonedGNN}$ Graph neural networks (GNNs) have shown great success in detecting intellectual property (IP) piracy and hardware Trojans (HTs). However, the machine learning community has demonstrated that GNNs are susceptible to data poisoning attacks, which result in ...
Comments
Information & Contributors
Information
Published In
2329-9290 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 19 February 2024
Published in TASLP Volume 32
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 4Total Downloads
- Downloads (Last 12 months)4
- Downloads (Last 6 weeks)1
Reflects downloads up to 23 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in