Predicting the Success of Mediation Requests Using Case Properties and Textual Information for Reducing the Burden on the Court

There are some works focusing on predicting legal judgment. Kort [6] applied a formula that combined pivotal factors to predict decisions by the Supreme Court. Nagel [25] predicted decisions by assigning correlation coefficients. Keown [12] discussed several mathematical models used in predicting judicial decisions, such as the scheme of nearest-neighbor rule and linear models. These studies focus on the analysis of non-textual information and judges’ votes. With the technical development of text mining, increasing numbers of works are focusing on textual information. Sim et al. [27] showed that the information extracted from texts is useful for predicting the votes of the Supreme Court judges. Lin et al. [15] introduced a framework to fetch 21 legal factor labels of robbery and intimidation cases and used the labels for case classification. Aletras et al. [1] used the N-gram model to predict the outcome of cases tried by the European Court of Human Rights. Sulea et al. [26] adopted support vector machines (SVMs) to predict the ruling of the French Supreme Court and the law discipline to which the case belongs. Hu et al. [10] proposed an attribute-attentive charge prediction model that had effective signals for distinguishing confusing charges. This research focuses on analyzing textual information. In addition, some articles discussed whether to use segmented words in Chinese. Meng et al. [19] conducted several natural language processing (NLP) experiments and found that word-based models in Chinese are more prone to overfitting. Song et al. [24] used character-level BERT and deep learning models to classify traditional Chinese medicine cases. In our work, we take the combination of case information and textual description as input and predict the results of mediation cases. To the best of our knowledge, there is no existing work dealing with mediation data, which is considered sensitive to the negotiation skill and personalities of the parties. For case information, we apply feature engineering methods to data related to mediators and parties. For textual content, we adopt state-of-the-art text mining methods and use a character-based model for extracting textual features. The experiments show that combining these two types of information can boost prediction performance.

Recently, multi-modal prediction and detection have become popular in NLP and text mining research fields. Zhang et al. [29] model emotion detection with modality and label dependence using multi-modal data, including textual, visual, and acoustic modalities. Cai et al. [3] focus on sarcasm detection in tweets consisting of texts and images in Twitter. The detection makes use of linguistic features, properties of the author, the audience, and the immediate communicative environment. Zhu et al. [31] predict product attributes by fusing textual and image data. Nojavanasghari et al. [22] predict persuasiveness using multi-modal information—including visual, acoustic, and textual features—with a deep fusion architecture. Shu et al. [23] claim that detecting fake news only from content is generally not satisfactory and suggest incorporating user social engagements as auxiliary information to improve fake news detection. Therefore, the study investigates how to jointly consider user profiles and post contents to detect fake news. Among these studies, they usually propose a comprehensive framework, which jointly models the multi-modal features such as content, social network, and user characteristics. However, it is not practical to directly apply a single machine learning model to combine heterogeneous features. For example, an image could be ideally dealt with using convolutional neural network (CNN)–based models [3, 29, 31]. In contrast, a textual data could be analyzed using embedding (e.g., BERT [7, 18, 21] or TextCNN) or recurrent neural network (RNN)–based modules [3, 22, 23, 29, 31]. To the best of our knowledge, few legal prediction or detection studies consider multi-modal prediction and detection; thus, this article is at the forefront of work on the issue.

The overview of our system is shown in Figure 1. First, we divide the original data into two parts. One is case information and the other is textual data. Then, these two kinds of data are fed into our feature engineering component. For the data from case information, we first do some preprocessing and feature extraction. After these manipulations, we use the processed data to train some powerful machine learning models, such as XGBoost and LightGBM, which will predict the mediation result as a probability distribution. This probability distribution will then be considered as features fed into the LSTM model. For the textual data, we perform a state-of-the-art embedding method, BERT, to extract the embedding vectors of contents. The output embedding vectors will then be utilized in two ways. One is directly feeding them into the LSTM model as feature vectors and the other acts as the features of TextCNN. The reason we perform this operation is to improve the quality of embedding by assembling two different text mining methods. The predictive judgment from TextCNN will also be another feature of LSTM. Lastly, we combine textual vectors, different classifiers’ results from CI features, and TextCNN's outcome as LSTM's inputs and obtain the mediation consequence as the output. We will explain each part in detail in the following sections.

We build a feature engineering framework that contains two major modules. The first is feature extraction of case information and the second is a text mining model for processing text information. We will also introduce the classifiers implemented in each part.

We find that real-world mediation cases have high temporal dependencies. Mediators can learn from experience and enhance their negotiating ability when involved in increasing numbers of cases as time goes on. We propose adopting LSTM to combine the results from all classifiers and utilize past cases’ information to predict the outcome of future cases. LSTM [9] is a powerful recurrent neural network. It can learn temporal dependency between mediation cases. In this work, we mainly use the past n cases’ information to predict the outcome of the next future case, where n ∈ N. According to our experiments, prediction accuracy is generally good from n = 1 to n = 10. The LSTM inference structure of this work is shown in Figure 4.

We should note that directly applying LSTM to make the prediction for the future case will encounter a limitation, in which the size of features must be the same for both past cases and future cases. However, we can see in Figure 5 that the feature size of the past case will be longer than that of the future case because the target case to predict will not have the genuine result. On the other hand, directly ignoring the ground truths of previous cases is also not applicable since these previous “answers” are highly correlated with the ground truth of the future case.

We claim that “Actual Results” must be considered in our LSTM-based approach, but we need to conquer the limitation of LSTM mentioned earlier. We propose utilizing XGBoost's (or LightGBM's) predictive outcome to fill into LSTM's future case to maintain the length. We name the updated LSTM as LSTM_AX (using Actual results for past cases and XGBoost's result for future cases) and LSTM_AL (using Actual results for past cases and LightGBM's result for future cases). These structures are shown in Figure 6 and we evaluate whether LSTM_AX and LSTM_AL can perform better than single classifiers in Section 5.2. We also verify that including “Actual Results” can help boost the effectiveness of prediction. Finally, we choose LSTM_AX as our main ensemble model of LSTMEnsembler because LSTM_AX has a slightly better performance than LSTM_AL in the experiments. LSTM_AX takes the combination of BERT's textual vectors, the predictive results of XGBoost and LightGBM classifiers, and TextCNN's outcome as input and makes the final prediction, indicating whether the mediation will succeed or not.

In the beginning, we aim to find out how many past days considered will benefit our prediction performance of LSTMEnsembler. We conduct the experiments by changing the number of cases in the past for each LSTM's prediction instance. The results are shown in Table 7. It shows that the best result is obtained when the number of past cases is five. Then, the effectiveness of increasing the number of past cases gets worse but not obvious. Therefore, in the following experiments, we take the past five cases into consideration and make a prediction for the next case.

In this section, we evaluate our proposed LSTMEnsembler compared with other baseline methods and other robust classifiers.

Baseline. We create a baseline to prove that machine learning works for the mediation classification problem. Since mediation cases can be sorted by application time and mediators can accumulate their experience by being involved in an increasing number of cases, we propose a reliable but intuitive baseline. For each new case and its sub-category (i.e., property, car accident, car accident compensation, injury, and referral), we consider the mediator's success rate of past cases in each sub-category. If the mediator's previous success rate exceeds 0.5 for one of the sub-categories, the baseline model guesses that the mediator will succeed for the next case corresponding to that sub-category. Therefore, we sort cases by application time and can generate the prediction for each case based on the mediator's dynamic success rate mentioned earlier. In the beginning, the success rate for each sub-category is initialized as zero. Then, the success rate increases when a successful case is encountered. An example of a mediator is shown in Table 8. In the mediator's first case, the baseline method will predict failure because the individual does not have experience serving as a mediator. However, after the individual successfully deals with the first case, we tend to predict that the mediator will succeed in the second case. For the mediator's fourth case of the car accident, the mediator's previous success rate is 66.67% since the individual has experienced two success cases and one failure case. On the basis of the result, we predict that the mediator will succeed again.

Comparative Methods. We compare LSTMEnsembler with two robust classifiers, XGBoost and LightGBM. The features used in XGBoost and LightGBM include embedding vectors from BERT, TextCNN's inferred probability, and all features of case information. XGBoost will also take LightGBM's inferred probability as its feature and vice versa. We propose that these two classifiers be compared because we would like to verify that our LSTMEnsembler is better than other single classifiers for handling temporal dependency and heterogeneous features. On the other hand, we also include another competitor, called Personal-LSTMEnsembler, which builds an LSTMEnsembler model for each mediator (not for all mediators) and considers only the past experiences of each one to make predictions.

Results. We show LSTMEnsembler and other competitors in Table 9. First, the baseline achieves 72% in F-score and 64% in accuracy. This baseline is utilized for modeling mediators’ experiences but does not consider the detailed properties of cases. The result of Personal-LSTMEnsembler shows that it is worse than the proposed LSTMEnsembler method. We believe that it is because there are not enough training samples for some mediators to train the models adequately. Among all of the results, our system achieves the best results and proves that when each classifier does what it is good at, LSTMEnsembler can achieve the best performance.

In this section, we would like to evaluate whether it is correct to involve “Actual Results” of Figure 5 in LSTMEnsembler with different feature combinations. To make the comparison more convincing, we include two more balanced input methods: LSTM_PX and LSTM_PL. LSTM_PX means that the “Actual Results” in the past cases are changed to the predictive probability distributions of XGBoost. The replacement is similar to LSTM_PL. We take XGBoost as an example in Figure 7, which is different from Figure 4; that is, all actual results in the past cases are replaced with XGBoost's inferred results.

We take seven different combinations of input features into the LSTM block in the experiments. The first three combinations simply use CI and text vectors (BERT or Word2Vec) to make predictions. The fourth and fifth consider TextCNN's predictive probability distribution to be a feature of LSTM to enhance the performance. The last two combinations include all classifiers’ results but do not consider CI due to the overfitting issue. The seventh includes the BERT vectors. Figures 8 and 9 show the results of these combinations evaluated by F-score and accuracy. For the first three results, we find that adding text information improves the result.

Furthermore, the BERT vectors can enhance more than Word2vec. Thus, we conclude that using characters as a token is more suitable for our work. In addition, when we combine more classifiers’ predictive probability distributions, the results improve more. We then add the BERT vectors and find that it improves the F-score, which turns out the best result (the seventh combination) in our framework. Among the four different ways to deal with the ground truth of the case, the LSTM_AX's performance is the best. It proves that using “Actual Results” is useful and better than using classifiers’ probability distributions. The experimental result also shows that BERT vectors allow our system to achieve 85.5% in F-score and 78.8% in accuracy. The experiments in this section show that our proposed LSTMEnsembler performs effectively for assembling the inferred results from other classifiers and the BERT's embedding vectors.

In this section, we further compare the performances of LSTM_AX, XGBoost, and LightGBM by varying different feature sets but do not consider the predictive probability distributions of each. We would like to verify that modeling temporal dependency between cases can improve performance.

The input feature is the same as the first five feature combinations in Section 5.3. Table 10 shows each performance in F-score and accuracy. XGBoost and LightGBM perform better when considering only CI for input. However, when the text vector is included, their results get worse. In contrast, by involving text information in our LSTM block, we can improve performance and obtain a better result.

We evaluate the performance of our proposed LSTMEnsembler by varying training data size from 50% to 90% compared with other methods. Figure 10 shows the F-score and Figure 11 shows the accuracy. The result shows that the performance of LSTMEnsembler can remain an 81% F-score and 73% accuracy when the training data size is dropped to 50%. However, Personal-LSTMEnsembler drops fast when the training data size is less than 60%. XGBoost and LightGBM also have stable performances but cannot gain the best effectiveness.