Semantic Segmentation of Legal Documents via Rhetorical Roles

Vijit Malik1∗ Rishabh Sanjay1∗ Shouvik Kumar Guha2

Angshuman Hazarika3 Shubham Nigam1 Arnab Bhattacharya1
Ashutosh Modi 1†
1
Indian Institute of Technology Kanpur (IIT-K)
2
West Bengal National University of Juridical Sciences (WBNUJS)
3
Indian Institute of Management Ranchi (IIM-R)
{vijitvm21,rishabh.lfs}@gmail.com shouvikkumarguha@nujs.edu
angshuman.hazarika@iimranchi.ac.in sknigam@cse.iitk.ac.in
arnabb@cse.iitk.ac.in ashutoshm@cse.iitk.ac.in

Abstract to clear the backlog of cases (Katju, 2019). Tech-

nology could come to the rescue in dealing with
Legal documents are unstructured, use legal the backlog, for example, if there were a technol-
jargon, and have considerable length, making ogy (based on NLP techniques) that could help a
them difficult to process automatically via con-
legal practitioner to extract relevant information
ventional text processing techniques. A legal
document processing system would benefit sub- from legal documents then it could make the legal
stantially if the documents could be segmented process more streamlined and efficient. However,
into coherent information units. This paper legal documents are quite different from conven-
proposes a new corpus of legal documents an- tional documents used to train NLP systems (e.g.,
notated (with the help of legal experts) with newspaper texts). Legal documents are typically
a set of 13 semantically coherent units labels long (tens of pages) (Malik et al., 2021), unstruc-
(referred to as Rhetorical Roles), e.g., facts, ar-
tured (Skylaki et al., 2021; Leitner et al., 2019),
guments, statute, issue, precedent, ruling, and
ratio. We perform a thorough analysis of the noisy (e.g., grammatical and spelling mistakes due
corpus and the annotations. For automatically to manual typing in courts) (Malik et al., 2021;
segmenting the legal documents, we experi- Kapoor et al., 2022), and use different lexicon (le-
ment with the task of rhetorical role prediction: gal jargon). The use of a specialized lexicon and
given a document, predict the text segments different semantics of words makes pre-trained neu-
corresponding to various roles. Using the cre- ral models (e.g., transformer-based models) inef-
ated corpus, we experiment extensively with fective (Chalkidis et al., 2020). The legal domain
various deep learning-based baseline models
has several sub-domains (corresponding to differ-
for the task. Further, we develop a multitask
learning (MTL) based deep model with docu- ent laws, e.g., criminal law, income tax law) within
ment rhetorical role label shift as an auxiliary it. Although some of the fundamental legal prin-
task for segmenting a legal document. The pro- ciples are common, the overlap between different
posed model shows superior performance over sub-domains is low; hence systems developed on
the existing models. We also experiment with one law (e.g., income tax law) may not directly
model performance in the case of domain trans- work for another law (e.g., criminal law), so there
fer and model distillation techniques to see the
is the problem of a domain shift (Bhattacharya
model performance in limited data conditions.
et al., 2019; Malik et al., 2021; Kalamkar et al.,
2022a; Kapoor et al., 2022).
1 Introduction
In this paper, we target legal case proceedings
The number of legal cases has been growing al- in the form of judgment documents. To aid the
most exponentially in populous countries like India. processing of long legal documents, we propose
For example, as per the India’s National Judicial a method of segmenting a legal document into co-
Data Grid, there are about 41 million cases pending herent information units referred to as Rhetorical
in India (National Judicial Data Grid, 2021). As Roles (Saravanan et al., 2008; Bhattacharya et al.,
per some of recent estimates by a retired Supreme 2019). We propose a corpus of legal documents
Court of India Judge, it will take about 450 years annotated with Rhetorical Roles (RRs). RRs could
∗
Equal Contributions be useful for various legal applications. Legal docu-
†
Corresponding Author ments are fairly long, and dividing these into rhetor-
153
Proceedings of the Natural Legal Language Processing Workshop 2022, pages 153 - 171
December 8, 2022 ©2022 Association for Computational Linguistics
ical role units can help summarize documents ef- cently, there has been a rapid growth in the de-
fectively. In the task of legal judgment prediction, velopment of NLP and ML technologies for the
for example, using RRs, one could extract the rele- Chinese legal system, inter alia, Chen et al. (2019);
vant portions of the case that contributes towards Hu et al. (2018); Jiang et al. (2018); Yang et al.
the final decision. RRs could be useful for legal in- (2019); Ye et al. (2018). Few works have focused
formation extraction, e.g., it can help extract cases on the creation of annotated corpora and the task of
with similar facts. Similarly, prior cases similar to automatic rhetorical role labeling. Venturi (2012)
a given case could be retrieved by comparing dif- developed a corpus, TEMIS of 504 sentences anno-
ferent rhetorical role units. In this work, we make tated both syntactically and semantically. The work
the following contributions: of Wyner et al. (2013) focuses on the process of
1. We create a new corpus of legal documents anno- annotation and conducting inter-annotator studies.
tated with rhetorical role labels. In contrast to pre- Savelka and Ashley (2018) conducted document
vious work (8 RRs) (Bhattacharya et al., 2019), we segmentation of U.S. court documents using Con-
create a more fine-grained set of 13 RRs. Further, ditional Random Fields (CRF) with handcrafted
we create the corpus on different legal domains features to segment the documents into functional
(§3). and issue-specific parts. Automatic labeling of
2. For automatically segmenting the legal doc- rhetorical roles was first conducted in Saravanan
uments, we experiment with the task of rhetori- et al. (2008), where CRFs were used to label seven
cal role prediction: given a document, predict the rhetorical roles. Nejadgholi et al. (2017) devel-
text segments corresponding to various roles. Us- oped a method for identification of factual and non-
ing the created corpus, we experiment with vari- factual sentences using fastText. The automatic
ous deep text classification and baseline models ML approaches and rule-based scripts for rhetor-
for the task. We propose new multi-task learn- ical role identification were compared in Walker
ing (MTL) based deep model with document level et al. (2019). Kalamkar et al. (2022b) create a large
rhetorical role shift as an auxiliary task for seg- corpus of RRs and propose transformer based base-
menting the document into rhetorical role units line models for RR prediction. Our work comes
(§4). The proposed model performs better than close to work by Bhattacharya et al. (2019), where
the existing models for RR prediction. We fur- they use the BiLSTM-CRF model with sent2vec
ther show that our method is robust against do- features to label rhetorical roles in Indian Supreme
main transfer to other legal sub-domains (§5). We Court documents. In contrast, we develop a multi-
release the corpus, model implementations and task learning (MTL) based model for RR prediction
experiments code: https://github.com/ that outperforms the system of Bhattacharya et al.
Exploration-Lab/Rhetorical-Roles (2019).
3. Given that annotating legal documents with RR
is a tedious process, we perform model distillation 3 Rhetorical Roles Corpus
experiments with the proposed MTL model and
attempt to leverage unlabeled data to enhance the Corpus Acquisition: We focus on Indian legal
performance (§5). We also show the use-case for documents in English; however, techniques we
RR prediction model. develop can be generalized to other legal sys-
tems. We consider legal judgments from the
2 Related Work Supreme Court of India, High Courts, and Tribunal
courts crawled from the website of IndianKanoon
Legal text processing has been an active area of (https://indiankanoon.org/). We also
research in recent times. A number of datasets, scrape Competition Law documents from Indian
applications, and tasks have been proposed. For Tribunal court cases (National Company Law Ap-
example, Argument Mining (Wyner et al., 2010), pellate Tribunal (NCLAT), COMPetition Appel-
Information Extraction and Retrieval (Tran et al., late Tribunal (COMPAT), Competition Commis-
2019), Event Extraction (Lagos et al., 2010), Prior sion of India (CCI)). We focus on two domains of
Case Retrieval (Jackson et al., 2003), Summariza- the Indian legal system: Competition Law (CL)
tion (Moens et al., 1999), and Case Prediction (Ma- (also called as Anti-Trust Law in the US and Anti-
lik et al., 2021; Chalkidis et al., 2019; Strickson Monopoly law in China) and Income Tax (IT). CL
and De La Iglesia, 2020; Kapoor et al., 2022). Re- deals with regulating the conduct of companies,
154
particularly concerning competition. With the help label. During the annotation, roles were further
of legal experts, we narrowed down the cases per- refined, and the documents were finally annotated
tinent to CL and IT from the crawled corpus (also with 13 fine-grained labels since some of the main
see Ethical Considerations in App. A). roles could be sub-divided into more fine-grained
Choice of CL and IT domains: India has a com- classes. The list of RRs is as follows (example sen-
mon law system where a decision may not be ex- tences for each role is in Table 15 in the Appendix
actly as per the statutes, but the judiciary may B.3):
come up with its interpretation and overrule exist-
• Fact (FAC): These are the facts specific to the
ing precedents. This introduces a bit of subjectivity.
case based on which the arguments have been
One of the biggest problems faced during the task
made and judgment has been issued. In addi-
of identifying the rhetorical roles in a judgment
tion to Fact, we also have the fine-grained la-
is that the element of subjectivity involved in the
bel Issues (ISS). The issues which have been
judicial perception and interpretation of different
framed/accepted by the present court for adju-
rhetorical roles, ranging from the factual matrix
dication.
(i.e., perception about facts, relevant facts and facts
• Argument (ARG): The arguments in the case
in an issue may vary) to the statutory applicabil-
were divided in two more fine-grained sub-
ity and interpretation to determine the fitness of a
labels: Argument Petitioner (ARG-P): Ar-
particular judicial precedent to the case at hand. In
guments which have been put forward by
order to overcome this particular obstacle, we focus
the petitioner/appellant in the case before the
on specific legal domains (CL and IT) that display
present court and by the same party in lower
a relatively greater degree of consistency and ob-
courts (where it may have been petitioner/re-
jectivity in terms of judicial reliance on statutory
spondent). Also, Argument Respondent
provisions to reach decisions (Taxmann, 2021).
(ARG-R): Arguments which have been put
Corpus Statistics: We randomly selected a set of forward by the respondent in the case before
50 documents each for CL and IT from the set of the present court and by the same party in
acquired documents (≈ 1.6k for IT and ≈ 0.8k for lower courts (where it may have been petition-
CL). These 100 documents were annotated with er/respondent)
13 fine-grained RR labels (vs. 8 by Bhattacharya • Statute (STA): The laws referred in the case.
et al. (2019)) by a team of legal experts. Our cor- • Dissent (DIS): Any dissenting opinion ex-
pus is double the size of the RR corpus of Bhat- pressed by a judge in the present judgment/de-
tacharya et al. (2019). The CL documents have cision.
13,328 sentences (avg. of 266 per document), and • Precedent (PRE): The precedents in the doc-
IT has a total of 7856 sentences (avg. of 157 per uments were divided into 3 finer labels, Prece-
document). Label-wise distribution for IT and CL dent Relied Upon (PRE-R): The precedents
documents are provided in Appendix B.3. Annotat- which have been relied upon by the present
ing legal documents with RRs is a tedious as well court for adjudication. These may or may not
as challenging task. Nevertheless, this is a grow- have been raised by the advocates of the par-
ing corpus, and we plan to add more annotated ties and amicus curiae. Precedent Not Relied
documents. However, given the complexity of an- Upon (PRE-NR): The precedents which have
notations, the RR labeling task also points towards not been relied upon by the present court for
looking for model distillation (§5) and zero-shot adjudication. These may have been raised by
learning-based methods. the advocates of the parties and amicus curiae.
Annotation Setup: The annotation team (legal Precedent Overruled (PRE-O): Any prece-
team) consisted of two law professors from presti- dents (past cases) on the same issue which
gious law schools and six graduate-level law stu- have been overruled through the current judg-
dent researchers. Annotating just 100 documents ment.
took almost three months. Based on detailed dis- • Ruling By Lower Court (RLC): Decisions
cussions with the legal team, we initially arrived of the lower courts which dealt with the same
at the eight main rhetorical roles (facts, arguments, case.
statues, dissent, precedent, ruling by lower court, • Ratio Of The Decision (ROD): The princi-
ratio and ruling by present court) plus one ‘none’ ple which has been established by the current
155
 judgment/decision which can be used in fu- Label IT CL
AR 0.80 0.93
ture cases. Does not include the obiter dicta FAC 0.80 0.89
which is based on observations applicable to PR 0.70 0.86
the specific case only. STA 0.78 0.89
RLC 0.58 0.74
• Ruling By Present Court (RPC): The de- RPC 0.78 0.79
cision of the court on the issues which have ROD 0.67 0.93
been framed/accepted by the present court for DIS _ 0.99
Macro F1 0.73 0.88
adjudication.
• None (NON): any other matter in the judg- Table 1: Label-wise Inter-Annotator agreement (F1
ment which does not fall in any of the above- Scores). Dissent label instance absent in IT.
mentioned categories.
The dataset was annotated by six legal experts FAC 94.5 2.8 0.3 1.6 0.0 0.4 0.0 0.5
(graduate law student researchers), 3 annotated 50 ARG 4.9 90.3 2.3 0.1 1.2 0.6 0.1 0.5 80
CL documents, and the remaining 3 annotated 50 PRE 0.4 0.8 96.0 0.4 0.0 0.0 2.4 0.0
60
IT documents. We used Webanno (de Castilho STA 3.0 0.3 0.2 94.2 0.1 1.9 0.0 0.2

et al., 2016) as the annotation framework. Each ROD 1.1 6.2 1.9 0.0 88.8 1.4 0.2 0.3 40
RLC 0.3 4.4 9.0 12.2 0.6 61.4 3.5 8.4
legal expert assigned one of the 13 Rhetorical roles
RPC 0.0 0.0 0.9 0.0 0.0 4.0 95.1 0.0 20
to each document sentence. Note that we initially
NON 0.1 0.1 1.0 0.0 0.0 0.6 0.0 98.1
experimented with different levels of granularity 0

A
G
E

D
C
C
N
(e.g., phrase level, paragraph level), and based on

FA

ST
PR

RL
RP
NO
RO
AR
the pilot study, we decided to go for sentence-level (a) IT
annotations as it maintains the balance (from the FAC 95.7 0.1 2.3 0.0 0.4 0.0 1.5 0.0 0.0
100

perspective of topical coherence) between too short ARG 0.0 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 80
(having no labels) and too long (having too many PRE 2.8 0.3 93.0 0.0 0.3 2.6 0.5 0.0 0.5

labels) texts. Legal experts pointed out that a single STA 2.5 1.7 47.0 22.0 4.2 3.0 8.1 10.6 0.8 60
ROD 0.1 0.0 0.0 0.0 99.6 0.2 0.2 0.0 0.0
sentence can sometimes represent multiple rhetori- RLC 0.0 0.0 17.2 0.0 0.0 82.8 0.0 0.0 0.0 40
cal roles (although this is not common). Each ex- RPC 3.6 0.0 1.9 0.0 2.6 0.1 89.4 1.0 1.4
pert could also assign secondary and tertiary rhetor- DIS 0.0 0.0 0.0 0.0 0.0 0.0 21.6 78.4 0.0 20

ical roles to a single sentence to handle such sce- NON 1.1 0.0 0.2 0.0 0.0 0.0 0.9 0.0 97.8
0
narios (also App. B.4). As an example, suppose a
C

A
G
E

D
C
C
DIS

N
FA

ST
PR

RL
RP

NO
RO
AR

sentence is a ‘Fact’ but could also be an ‘Argument’ (b) CL

according to the legal expert. In that case, the ex-
pert could assign the rhetorical roles ‘Primary Fact’ Figure 1: Confusion matrix between Annotators A1 and
and ‘Secondary Argument’ to that sentence. We A3 . Numbers represent % agreement. Dissent label
instance is absent in IT.
extended it to the tertiary level as well to handle
rare cases.
Our corpus is different from the existing cor- sentence. We asked the law professors to finalize
pus (Bhattacharya et al., 2019). Firstly, we use 13 the primary label in such cases. If the law profes-
fine-grained RR labels and the size of the corpus is sors decided to go with a label completely different
almost twice. Secondly, we focus on different legal from the three annotated labels, we went with their
sub-domains (IT and CL vs. Supreme Court Judg- verdict. However, such cases were not frequent (≈
ments). Lastly, we perform the primary, secondary, 4% of adjudicated cases). In this paper, for RR
and tertiary levels of annotations since, according prediction, we concentrate on the primary labels
to legal experts, it is sometimes possible that a and leave explorations of secondary and tertiary
sentence might have multiple RR labels. labels for future work.
Adjudication and Data compilation: Annotating Inter-annotator Agreements: The Fleiss kappa
RR is not a trivial task, and annotators can have dis- (Fleiss et al., 2013) between the annotators is 0.65
agreements. We followed a majority voting strategy for the IT domain and 0.87 for the CL domain,
over primary labels to determine the gold labels. indicating a substantial agreement between an-
There were a few cases (≈ 5%) where all the three notators. Additionally, as done in Bhattacharya
legal experts assigned a different role to the same et al. (2019) and Malik et al. (2021), we calculate
156
the pair-wise inter-annotator F1 scores. To deter- Model Dataset F1
SBERT-Shift IT 0.60
mine the agreement between the three annotators SBERT-Shift CL 0.49
A1 , A2 , A3 (each for IT and CL domain), we cal- SBERT-Shift IT+CL 0.47
culate the pairwise F1 scores (App. C) between BERT-SC IT 0.66
BERT-SC CL 0.64
annotators (A1 , A2 ), (A2 , A3 ) and (A3 , A1 ). We BERT-SC IT+CL 0.64
average these pairwise scores for each label and
further average them out. We report the label-wise Table 2: Results for the auxillary task LSP
F1 and Macro F1 in Table 1. The table shows
that the agreements between domains differ (0.73
(details and case studies in App. B.6).
for IT vs. 0.88 for CL). This is mainly due to (as
pointed by law professors) the presence of more 4 Rhetorical Roles Prediction
precedents and a greater number of statutory provi-
sions in IT laws. These factors combine to produce We would like to automate the process of segment-
more subjectivity (relative to CL) when it comes to ing a legal document, to develop ML models for
interpreting and retracing judicial decisions. The the automation, we experiment with the task of
confusion matrix between the annotators (A1 , A3 ) Rhetorical Roles prediction.
is shown in Figure 1 (more details in App. B.5). Task Definition: Given a legal document, D,
containing the sentences [s1 , s2 , ...sn ], the task of
Analysis: Annotation of judgments to identify RR rhetorical role prediction is to predict the label (or
is a challenging task even for legal experts. Sev- role) yi for each sentence si ∈ D.
eral factors contribute to this challenge. Annota- Baseline Models: For the first set of baseline mod-
tors need to glean and combine information non- els, the task is modeled as a single sentence predic-
trivially (e.g., facts and arguments presented, the tion task, where given the sentence s, the model pre-
implicit setting, and the context under which the dicts the rhetorical role of the sentence. In this case,
events described in the case happened) to arrive the context is ignored. We consider pre-trained
at the label. Moreover, the annotator only has ac- BERT (Devlin et al., 2019) and LEGAL-BERT
cess to the current document, which is a secondary (Chalkidis et al., 2020) models for this. As another
account of what actually happened in the court. set of baseline models, we consider the task as a
These limitations certainly make the task of the sequence labeling task, where the sequence of all
annotator more difficult and leave them with no the sentences in the document is given as input,
choice other than to make certain educated guesses and the model has to predict the RR label for each
when it comes to understanding the various nu- sentence. We used CRF with hand-crafted features
ances, both ostensible and probable, of certain RR. (Bhattacharya et al., 2019) and BiLSTM network.
It should, however, be noted that such variation Label Shift Prediction: Rhetorical role labels do
need not occur for every RR since not all the roles not change abruptly across sentences in a docu-
are equally susceptible to it. A cumulative effect of ment, and the text tends to maintain topical coher-
the aforementioned factors can be observed in the ence. Given the label y for a sentence si in the
results of the annotation. The analysis provided by document, we hypothesize that the chances of shift
the three annotators in the case of CL bears close (change) in the label for the next sentence si+1 are
resemblance with each other. On the other hand, low. We manually verified this using the training
in the case of IT, the analysis provided by Users 1 set and observed that on average in a document,
and 3 bears a greater resemblance with each other, if the label of sentence si is y, then 88% of the
compared to the resemblance between Users 1 and times the label of the next sentence si+1 is same
2, or between Users 2 and 3. On a different note, as y. Note that this is true only for consecutive
it is also observed that the rhetorical role where sentences, but in general, label shift inertia fades
the annotators have differed between themselves as we try to predict beyond the second consecu-
the most has been the point of Ruling made by the tive sentence. Since we are performing a sequence
Lower Court, followed by the Ratio. This also ties prediction task, this alone is not a good model for
in with the argument that all rhetorical roles are not label prediction. Nevertheless, we think that this
equally susceptible to the variation caused by the label shift inertia can provide a signal (via an auxil-
varying levels of success achieved by the different iary task) to the main sequence prediction model.
annotators in retracing the judicial thought pattern Based on this observation, we define an auxiliary
157
binary classification task: Label Shift Prediction y1 yi yn

(LSP), that aims to model the relationship between CRF CRF CRF

two sentences si and si+1 and predict whether the

labels yi for si and yi+1 for si+1 are different (shift LSTM LSTM LSTM

occurs) or not. In particular, for each sentence pair

S = {si , si+1 } ∈ D, we define the label of LSP ei-1,i ei,i+1

task, Y = 1 if yi ̸= yi+1 , otherwise Y = 0, here yi

bi
LSP Shift LSP Shift

is the rhetorical role for sentence si . Note that for BERT

the full model at the inference time, the true label si-1 si si+1

of a sentence is not provided; hence predicting a

shift in label makes more sense than performing Figure 2: LSP-BiLSTM-CRF Model
a binary prediction that the next sentence has the
same label or not. We model the LSP task via two
different models: MultiTask Learning (MTL): We use the frame-
work of Multitask learning, where rhetorical role
SBERT-Shift: We model the label shift via a prediction is the main task and label shift predic-
Siamese network. In particular, we use the pre- tion is the auxiliary task. Sharing representations
trained SBERT model (Reimers and Gurevych, between the main and related tasks helps in better
2019) to encode sentences si and si+1 to get repre- generalization on the main task (Crawshaw, 2020).
sentations ei and ei+1 . The combination of these The intuition is that a label shift would help the
representations (ei ⊕ ei+1 ⊕ (ei − ei+1 )) is passed rhetorical role component make the correct pre-
through a feed-forward network to predict the shift. diction based on the prospective shift. The MTL
BERT-SC: We use the pre-trained BERT model model (Figure 3) consists of two components: the
and fine-tune it for the task of LSP. We model the shift detection component and the rhetorical role
input in the form of sentence semantic coherence prediction component. The shift component pre-
task, [CLS]⊕si ⊕[SEP ]⊕si+1 ⊕[SEP ] to make dicts if a label shift occurs at ith position. The
the final prediction for shift. In general, the BERT- output of the BiLSTM layer of shift component is
SC model performs better than SBERT-Shift (Table concatenated with the BiLSTM output of the rhetor-
2). Due to the superior performance of BERT-SC, ical role component. The concatenated output is
we include it to provide label shift information to passed to a CRF layer for the final prediction of
the final MTL model. The aim of our work is to the rhetorical role. The loss for the model is given
predict RR, and we use label shift as auxiliary in- by: L = λLshif t + (1 − λ)LRR , where, Lshif t is
formation even if it may not be predicted correctly the loss corresponding to label shift prediction and
at all times. As shown in results later, this limited LRR is the loss corresponding to rhetorical role pre-
information improves the performance. diction, and hyperparameter λ balances the impor-
Proposed Models: We propose two main models tance of each of the task. If λ is set to zero, we are
for the rhetorical role prediction: Label Shift Pre- back with our baseline BiLSTM-CRF model. Since
diction based on BiLSTM-CRF and MTL models. there are two components, we experimented with
LSP-BiLSTM-CRF: Signal from label shift is sending the same encodings of sentences to both
used to aid the RR prediction in the LSP-BiLSTM- the components (E1 = E2 ), as well as sending dif-
CRF model. The model consists of (Figure 2) a ferent encodings of the same sentence to both com-
BiLSTM-CRF model with specialized input rep- ponents (E1 ̸= E2 ). The proposed model is very
resentation. Let the sentence embedding (from different from the previously proposed BiLSTM-
pre-trained BERT) corresponding to ith sentence CRF by Bhattacharya et al. (2019) that does not
be bi . Let, the representation of the label shift (the use any multitasking and label shift information.
layer before the softmax layer in LSP model) be-
5 Experiments, Results and Analysis
tween current sentence and previous sentence pair
{si−1 , si } be ei−1,i . Similarly for the next pair Due to the complexity of the task of RR prediction
({si , si+1 }) we get ei,i+1 . The sentence representa- and to be comparable with the existing baseline sys-
tion for ith sentence is given by ei−1,i ⊕ bi ⊕ ei,i+1 . tems, for experiments, we consider 7 main labels
This sentence representation goes as input to the (FAC, ARG, PRE, ROD, RPC, RLC, and STA). We
BiLSTM-CRF model for RR prediction. plan to explore all fine-grained RR label (13) pre-
158
 RR Component
y1 yi yn
0.66
Macro Average
Shift Component
y'1 y'i y'n CRF CRF CRF
0.64
CRF CRF CRF
0.62

F1
0.60
LSTM LSTM LSTM LSTM LSTM LSTM
0.58
IT
E1(s1) E1(si) E1(sn) E2(s1) .... E2(si) .... E2(sn) IT+CL
.... ....
0.56
E1 E2
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
s1 .... si .... sn

Figure 4: Variation of F1 score with λ on IT and IT+CL

Figure 3: MTL architecture for Rhetorical Role La- domain
belling and Shift Prediction.
Model IT (F1) CL (F1)
dictions in the future. Based on recommendations BERT 0.56 0.52
BERT-neighbor 0.53 0.51
by legal experts, we ignore sentences with NON LEGAL-BERT 0.55 0.53
(None) label (about 4% for IT and 0.5% for CL) CRF (Handcrafted) 0.55 0.52
(more details in App. D.1). Further, the IT domain BiLSTM (sent2vec) 0.55 0.54
BiLSTM-CRF (handcraft) 0.57 0.56
did not have any instance of dissent (DIS) label, BiLSTM-CRF (sent2vec) 0.59 0.61
and CL has only three documents with very few BiLSTM-CRF (BERT emb) 0.63 0.63
DIS instances. Based on consultations with law ex- BiLSTM-CRF (MLM emb) 0.58 0.60
perts, we discarded DIS sentences (more details in LSP (SBERT) 0.64 0.63
LSP (BERT-SC) • 0.65 0.68
App. D.1). We randomly split (at document level) MTL (MLM emb) 0.67 0.67
IT/CL into 80% train, 10% validation, and 10% test MTL (BERT-SC) ⋆ ⋄ 0.70±0.02 0.69±0.01
set. In contrast to Bhattacharya et al. (2019), we did
Table 3: Results of baseline and proposed models on
not perform cross-validation for better comparison
IT and CL. LSP and MTL refer to the LSP-BiLSTM-
across different models. We also experiment with a CRF and MTL-BiLSTM-CRF models respectively. •
combined dataset of IT and CL (IT+CL); the splits LSP result is significant with p ≤ 0.05 in comparison
are made by combining individual train/val/test to baseline (BiLSTM-CRF(sent2vec)). Similarly, MTL
split of IT and CL. We experimented with a num- (BERT-SC) has significant result in comparison to base-
ber of baseline models (Table 3, 4). In particular, line (⋄, p ≤ 0.05). MTL (BERT-SC) is significant w.r.t.
we considered BiLSTM with sent2vec embeddings LSP (⋆, p ≤ 0.05).
(Bhattacharya et al., 2019), non-contextual models
(single sentence) like BERT (Devlin et al., 2019), Results and Analysis: Among the baseline models
LegalBERT (Chalkidis et al., 2020) and BERT- (Table 3), we note that LEGAL-BERT performs
neighbour (we take both left and right neighboring slightly better on the CL domain but slightly worse
sentences in addition to the sentence of interest). on the IT domain when compared to pre-trained
We also considered sentence-level sequence predic- BERT. It might be attributed to that LEGAL-BERT
tion models (contextual models): CRF model us- (trained on EU legal documents, which also has
ing handcrafted features provided by Bhattacharya European competition law) is not trained on In-
et al. (2019), different variants of BiLSTM-CRF, dian IT law documents. Using BERT embeddings
one with handcrafted features, with sent2vec em- with BiLSTM-CRF provides better results. Both
beddings, with BERT embeddings, and with MLM the proposed approaches outperform the previous
embeddings. We finetuned BERT with Masked approaches by a substantial margin. The MTL ap-
Language Modeling (MLM) objective on the train proach (with λ = 0.6) provides the best results on
set to obtain MLM embeddings (CLS embedding) both datasets with an average (over six runs) F1
for each of the sentences (App. D has hyperparam- score of 0.70 (standard deviation of 0.02) on the
eters, training schedule, and compute settings). We IT domain, an average F1 of 0.69(±0.01) on CL
use the Macro F1 metric for evaluation (App. C). domain, and an average F1 score of 0.71(±0.01)
We tuned the hyperparameter λ of the MTL loss for the combined domain. The MTL model shows
function using the validation set. We trained the variance across runs; hence we average the results.
MTL model with λ ∈ [0.1, 0.9] with strides of 0.1 Other models were reasonably stable across runs.
(Figure 4). λ = 0.6 performs the best for the IT do- We use the LSP shift component with BERT-SC
main and performs competitively on the combined as the encoder E1 and the pre-trained BERT model
domains. as the encoder E2 in our MTL architecture. We
159
 Model IT+CL (F1) Train Test BiLSTM-CRF
MTL
BiLSTM-CRF (sent2vec) 0.65 Dataset Dataset (sent2vec)
BiLSTM-CRF (BERT embs) 0.63 Gtrain Gtest 0.55 0.59
LSP-BiLSTM-CRF (BERT-SC) 0.67 0.48 0.50
Gtrain CLtest
MTL-BiLSTM-CRF (BERT-SC) 0.70±0.01 (12.78%) (15.25%)
0.41 0.46
Gtrain IT test
Table 4: Results of baseline and proposed models on (25.45%) (22.03%)
combined dataset (IT+CL) 0.42 0.48
Gtrain (IT+CL)test
(23.64%) (18.64%)
Label IT CL (IT+CL)train Gtest 0.60 0.63
AR 0.67±0.010 0.78±0.005
FAC 0.78±0.020 0.75±0.010 Table 6: Domain transfer experiments to compare the
PR 0.69±0.005 0.62±0.005
performance of MTL-BiLSTM-CRF with the baseline
STA 0.79±0.020 0.82±0.020
RLC 0.62±0.005 0.53±0.005 BiLSTM-CRF. The number in parenthesis denotes ∆G :
RPC 0.70±0.010 0.71±0.010 the % difference between the performance on Gtest and
ROD 0.66±0.005 0.65±0.005 the new domain.
Macro F1 0.70±0.020 0.69±0.010
ization capabilities of the MTL model compared
Table 5: Label-wise average (across 6 runs) F1 scores to the baseline model, we conducted some domain
of MTL-BiLSTM-CRF (BERT-SC) model.
transfer experiments. We experimented with a RR
did not use SBERT since it was under-performing dataset of 50 documents (referred to as G) by Bhat-
when compared to BERT-SC. We provide the label- tacharya et al. (2019). G dataset comes from a dif-
wise F1 scores for the MTL model in Table 5. Note ferent legal sub-domain (criminal and civil cases)
the high performance on the FAC label and low with very less overlap with IT and CL. We tried
performance on the RLC label; this is similar to different combinations of train and test datasets of
what we observe for annotators (Table 1). Also, the IT, CL, and G. Note that G (criminal and civil
MTL model performs better on the AR label in the cases) has very less overlap with IT and CL cases,
CL domain than the IT domain. An opposite trend so practically, it is a different domain. The results
can be observed for the RLC label. The contribu- are in Table 6. We can observe that the MTL model
tion of the LSP task is evident from the superior generalizes better across the domains than the base-
performance. We conduct the ablation study of our line model. Both the models perform better on the
MTL architecture from multiple aspects. Instead Gtest when the combined (IT+CL)train set is used.
of using shift embeddings from BERT-SC as the This points towards better generalization.
encoder E1 , we use a BERT model fine-tuned upon Model distillation: RR annotation is a tedious
the MLM task on the IT and CL domain. However, process, however, there is an abundance of unla-
we obtain a comparatively lower score (see App. belled legal documents. We experimented with
D). This observation yet again points towards the semi-supervised techniques to leverage the unla-
significance of the LSP in the task of rhetorical belled data. In particular, we tried a self-training
role prediction (results on other encoders in App. based approach (Xie et al., 2020). The idea is
D). The results have two interesting observations: to learn a teacher model θtea on the labelled data
firstly, MTL model performance on IT cases comes DL . The teacher model is then used to gener-
close to the average inter-annotator agreement. In ate hard labels on unlabeled sentences su ∈ di :
the case of CL, there is a gap. Secondly, for the ŷi = fθtea (d̂i ) ∀d̂i ∈ DU . Next, a student
model, the performance on the IT domain is better model θstu is learned on labeled and unlabeled sen-
than the CL domain, but in the case of annotators tences, with the loss function for student training
P
opposite trend was observed. We do not know the given by: LST = |D1L | dj ∈DL L(fθstu (dj ), yj ) +
exact reason for this, but the legal experts pointed αU P
DU ˆ L(fθ (dˆi ), yˆi ). Here, αU is a weigh-
di ∈DU stu
out that this is possible because the selected doc- ing hyperparameter between the labelled and un-
uments might be restricted to specific sections of labelled data (details in App. D). The process can
the IT law and model learned solely from these be iterated and the final distilled model is used for
documents alone without any other external knowl- prediction. The results of model distillation are
edge. However, annotators, having knowledge of shown in Table 8 for two iterations (initializing the
the entire IT law, might have looked from a broader teacher model of the current iteration as the learned
perspective. student model of the previous iteration; further it-
Domain Transfer: In order to check the general- erations do not improve results). MTL model was
160
run just once, due to variance it shows F1 of 0.68. Model IT+CL docs F1
BERT-ILDC last 512 tokens 0.55
The results improve for majority of labels with an BERT-ILDC Gold ROD & RPC 0.58
increment of 0.11 F1 score for the RLC label in
the first iteration. Also, the variance of F1 scores Table 7: Judgment prediction using RR. The model
across labels decreases. using gold ROD and RPC is found to be statistically
significant (p ≤ 0.05).
5.1 Application of Rhetorical Role to Label Base MTL Dist. Iter 1 Dist. Iter 2
Judgment Prediction AR 0.62 0.70 0.70
To check the applicability of RR in downstream FAC 0.74 0.75 0.73
PR 0.68 0.72 0.74
applications, as a use-case, we experimented with STA 0.76 0.77 0.75
how RR could contribute towards judgment pre- RLC 0.59 0.70 0.70
diction (ethical concerns discussed later). We use RPC 0.67 0.63 0.73
the legal judgment corpus (ILDC) provided by Ma- ROD 0.68 0.66 0.68
Macro F1 0.68 0.71 0.72
lik et al. (2021) and fine-tune a pre-trained BERT
model on the train set of ILDC for the task of judg- Table 8: Model Distillation: F1 scores of MTL-
ment prediction on the last 512 tokens of the doc- BiLSTM-CRF (BERT-SC) model after two distillation
uments. Malik et al. (2021) observed that training iterations on the IT domain.
on the last 512 (also the max size of the input to the last 512 tokens of each document in the ILDC
BERT) tokens of a legal document give the best re- corpus (Malik et al., 2021) and use it to predict the
sults; we use the same setting. We use this trained judgment of the test set documents, given only the
model directly for predicting the outcome on 84 predicted ROD and RPC sentences. We compare
IT/CL cases. We removed text corresponding to the results by the MTL model and BiLSTM-CRF
the final decisions (and extracted gold decisions) baseline on performing judgment prediction with
from these documents with the help of legal ex- predicted rhetorical roles. Refer to Appendix Table
perts. In the first experiment, we use the last 512 14 for the results. Since RR prediction for ROD
tokens of IT/CL cases for prediction. To study the and RPC is not perfect, improving it would greatly
effect of RRs, in another experiment, we extract the enhance the results as shown in Table 7.
sentences corresponding to gold ratio (ROD) and
ruling (RPC) RR labels in IT/CL documents and 6 Conclusion
use this as input to the BERT model. We consider We introduce a new corpus annotated with rhetori-
these two RR only since, by definition, these sen- cal roles. We proposed a new MTL model that uses
tences denote the principles and the decision of the label shift information for predicting labels. We
court related to the issues in the proceedings. There further showed via domain transfer experiments the
were no ROD or RPC labels for some documents generalizability of the model. Since RR are tedious
(16 out of 100 for both IT and CL); we removed to annotate, we showed the possibility of using
these in both experiments. The results are shown in model distillation techniques to improve the sys-
Table 7. Using the gold RR gives a boost to the F1 tem. In the future, we plan to explore cross-domain
score. We also experimented with using predicted transfer techniques to perform RR identification in
RR, and the performance was comparable to that legal documents in other Indian languages. Never-
of the BERT model. theless, we plan to grow the corpus. We also plan
To explore how predicted rhetorical roles would to apply RR models for other legal tasks such as
perform on judgment prediction task, we perform summarization and information extraction.
the following experiment. We use our best perform-
ing model MTL (BERT-SC), trained on the com- Acknowledgements
bined IT+CL domain to check the applicability of We would like to thank anonymous reviewers for
rhetorical roles for the task of Judgment Prediction. their insightful comments. We would like to thank
In the first step, we obtain the predicted rhetorical student research assistants Tridib Mandal, Chirag
roles for each sentence in the documents. Next, Mittal, Shefali Deshmukh, Shailja Beria, and Sya-
we select the sentences labeled as ROD or RPC1 . mantak Sinha from West Bengal National Univer-
Third, we use a BERT base model fine-tuned on sity of Juridical Sciences (WBNUJS) for annotat-
1
We select only these two labels since by definition, these ing the documents. This work would not have been
sentences provide the necessary cues towards the judgment. possible without their help.
161
References Peter Jackson, Khalid Al-Kofahi, Alex Tyrrell, and Arun
Vachher. 2003. Information extraction from case law
Paheli Bhattacharya, Shounak Paul, Kripabandhu and retrieval of prior cases. Artificial Intelligence,
Ghosh, Saptarshi Ghosh, and Adam Wyner. 2019. 150(1-2):239–290.
Identification of rhetorical roles of sentences in in-
dian legal judgments. CoRR, abs/1911.05405. Xin Jiang, Hai Ye, Zhunchen Luo, WenHan Chao, and
Wenjia Ma. 2018. Interpretable rationale augmented
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Ale- charge prediction system. In Proceedings of the 27th
tras. 2019. Neural legal judgment prediction in En- International Conference on Computational Linguis-
glish. In Proceedings of the 57th Annual Meeting of tics: System Demonstrations, pages 146–151, Santa
the Association for Computational Linguistics, pages Fe, New Mexico. Association for Computational Lin-
4317–4323, Florence, Italy. Association for Compu- guistics.
tational Linguistics.
Prathamesh Kalamkar, Aman Tiwari, Astha Agarwal,
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka- Saurabh Karn, Smita Gupta, Vivek Raghavan, and
siotis, Nikolaos Aletras, and Ion Androutsopoulos. Ashutosh Modi. 2022a. Corpus for automatic struc-
2020. LEGAL-BERT: The muppets straight out of turing of legal documents. In Proceedings of the Lan-
law school. In Findings of the Association for Com- guage Resources and Evaluation Conference, pages
putational Linguistics: EMNLP 2020, pages 2898– 4420–4429, Marseille, France. European Language
2904, Online. Association for Computational Lin- Resources Association.
guistics.
Prathamesh Kalamkar, Aman Tiwari, Astha Agarwal,
Huajie Chen, Deng Cai, Wei Dai, Zehui Dai, and Saurabh Karn, Smita Gupta, Vivek Raghavan, and
Yadong Ding. 2019. Charge-based prison term pre- Ashutosh Modi. 2022b. Corpus for automatic struc-
diction with deep gating network. In Proceedings of turing of legal documents. In Proceedings of the
the 2019 Conference on Empirical Methods in Natu- Thirteenth Language Resources and Evaluation Con-
ral Language Processing and the 9th International ference, pages 4420–4429, Marseille, France. Euro-
Joint Conference on Natural Language Processing pean Language Resources Association.
(EMNLP-IJCNLP), pages 6362–6367, Hong Kong,
China. Association for Computational Linguistics. Arnav Kapoor, Mudit Dhawan, Anmol Goel, Arjun T H,
Akshala Bhatnagar, Vibhu Agrawal, Amul Agrawal,
Michael Crawshaw. 2020. Multi-task learning Arnab Bhattacharya, Ponnurangam Kumaraguru, and
with deep neural networks: A survey. CoRR, Ashutosh Modi. 2022. HLDC: Hindi legal docu-
abs/2009.09796. ments corpus. In Findings of the Association for
Computational Linguistics: ACL 2022, pages 3521–
Richard Eckart de Castilho, Eva Mujdricza-Maydt, 3536, Dublin, Ireland. Association for Computational
Seid Muhie Yimam, Silvana Hartmann, Iryna Linguistics.
Gurevych, Anette Frank, and Chris Biemann. 2016.
A web-based tool for the integrated annotation of Justice Markandey Katju. 2019. Backlog of
semantic and syntactic structures. In Proceedings of cases crippling judiciary. https://perma.cc/
the Workshop on Language Technology Resources D8V4-L566.
and Tools for Digital Humanities (LT4DH), pages
76–84. Nikolaos Lagos, Frederique Segond, Stefania Castellani,
and Jacki O’Neill. 2010. Event extraction for legal
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and case building and reasoning. In International Con-
Kristina Toutanova. 2019. BERT: Pre-training of ference on Intelligent Information Processing, pages
Deep Bidirectional Transformers for Language Un- 92–101. Springer.
derstanding. In Proceedings of the 2019 Conference
of the North American Chapter of the Association for Elena Leitner, Georg Rehm, and Julian Moreno-
Computational Linguistics: Human Language Tech- Schneider. 2019. Fine-grained named entity recogni-
nologies, Volume 1 (Long and Short Papers), pages tion in legal documents. In Semantic Systems. The
4171–4186, Minneapolis, Minnesota. Association for Power of AI and Knowledge Graphs, pages 272–287,
Computational Linguistics. Cham. Springer International Publishing.

Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam,
2013. Statistical methods for rates and proportions. Kripabandhu Ghosh, Shouvik Kumar Guha, Arnab
john wiley & sons. Bhattacharya, and Ashutosh Modi. 2021. ILDC for
CJPE: Indian legal documents corpus for court judg-
Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and ment prediction and explanation. In Proceedings
Maosong Sun. 2018. Few-shot charge prediction of the 59th Annual Meeting of the Association for
with discriminative legal attributes. In Proceedings of Computational Linguistics and the 11th International
the 27th International Conference on Computational Joint Conference on Natural Language Processing
Linguistics, pages 487–498, Santa Fe, New Mexico, (Volume 1: Long Papers), pages 4046–4062, Online.
USA. Association for Computational Linguistics. Association for Computational Linguistics.

162
Marie-Francine Moens, Caroline Uyttendaele, and Jos Adam Wyner, Raquel Mochales-Palau, Marie-Francine
Dumortier. 1999. Abstracting of legal cases: the Moens, and David Milward. 2010. Approaches to
potential of clustering based on the selection of rep- text mining arguments from legal cases. In Semantic
resentative objects. Journal of the American Society processing of legal texts, pages 60–79. Springer.
for Information Science, 50(2):151–161.
Adam Z Wyner, Wim Peters, and Daniel Katz. 2013. A
National Judicial Data Grid. 2021. National judi- case study on legal case annotation. In JURIX, pages
cial data grid statistics. https://www.njdg. 165–174.
ecourts.gov.in/njdgnew/index.php.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and
Quoc V Le. 2020. Self-training with noisy student
Isar Nejadgholi, Renaud Bougueng, and Samuel With-
improves imagenet classification. In Proceedings of
erspoon. 2017. A semi-supervised training method
the IEEE/CVF Conference on Computer Vision and
for semantic search of legal facts in canadian immi-
Pattern Recognition, pages 10687–10698.
gration cases. In JURIX, pages 125–134.
Wenmian Yang, Weijia Jia, Xiaojie Zhou, and Yutao
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Luo. 2019. Legal judgment prediction via multi-
Sentence embeddings using siamese bert-networks. perspective bi-feedback network. In Proceedings of
arXiv preprint arXiv:1908.10084. the Twenty-Eighth International Joint Conference on
Artificial Intelligence, IJCAI-19, pages 4085–4091.
M Saravanan, Balaraman Ravindran, and S Raman. International Joint Conferences on Artificial Intelli-
2008. Automatic identification of rhetorical roles gence Organization.
using conditional random fields for legal document
summarization. In Proceedings of the Third Interna- Hai Ye, Xin Jiang, Zhunchen Luo, and Wenhan Chao.
tional Joint Conference on Natural Language Pro- 2018. Interpretable charge predictions for criminal
cessing: Volume-I. cases: Learning to generate court views from fact
descriptions. In Proceedings of the 2018 Conference
Jaromir Savelka and Kevin D Ashley. 2018. Segmenting of the North American Chapter of the Association for
us court decisions into functional and issue specific Computational Linguistics: Human Language Tech-
parts. In JURIX, pages 111–120. nologies, Volume 1 (Long Papers), pages 1854–1864,
New Orleans, Louisiana. Association for Computa-
Stavroula Skylaki, Ali Oskooei, Omar Bari, Nadja tional Linguistics.
Herger, and Zac Kriegman. 2021. Legal entity ex-
traction using a pointer generator network. In 2021
International Conference on Data Mining Workshops
(ICDMW), pages 653–658.

Benjamin Strickson and Beatriz De La Iglesia. 2020.

Legal judgement prediction for uk courts. In Proceed-
ings of the 2020 The 3rd International Conference
on Information Science and System, pages 204–209.

Taxmann. 2021. Interpretation of statutes: Strict versus

liberal construction. https://tinyurl.com/
2p85h3xd.

Vu Tran, Minh Le Nguyen, and Ken Satoh. 2019. Build-

ing legal case retrieval systems with lexical matching
and summarization using a pre-trained phrase scoring
model. In Proceedings of the Seventeenth Interna-
tional Conference on Artificial Intelligence and Law,
pages 275–282.

Giulia Venturi. 2012. Design and development of temis:

a syntactically and semantically annotated corpus of
italian legislative texts. In Proceedings of the Work-
shop on Semantic Processing of Legal Texts (SPLeT
2012), pages 1–12.

Vern R Walker, Krishnan Pillaipakkamnatt, Alexan-

dra M Davidson, Marysa Linares, and Domenick J
Pesce. 2019. Automatic classification of rhetorical
roles for sentences: Comparing rule-based scripts
with machine learning. In ASAIL@ ICAIL.

163
Appendix of NCLAT (National Company Law Appellate
Tribunal)2 , CCI (Competition Commission of In-
A Ethical Considerations dia)3 , COMPAT (Competition Appellate Tribunal)4 .
The proposed corpus and methods do not have Since the IT laws are 50 years old and relatively
direct ethical consequences to the best of our dynamic, we stick to certain sections of IT domain
knowledge. The corpus is created from pub- only, whereas we use all the sections for CL do-
licly available data from a public resource: www. main. We restrict ourselves to the IT cases that
indiankanoon.org. The website allows free are based on Section 147, Section 92C and Sec-
downloads, and no copyrights were violated. With tion 14A only to limit the subjectivity in cases. We
the help of law professors, we designed a course randomly select 50 cases from IT and CL domain
project centered around RR annotations for the each to be annotated. We used regular expressions
student annotators. The students voluntarily par- in Python to remove the auxillary information in
ticipated in the annotations as a part of the course the documents (For example: date, appellant and
project. Moreover, annotators were curious about respondent names, judge names etc.) and filter out
learning about AI technologies and further con- the main judgment of the document. We use the
tributing towards its progress. There was no com- NLTK5 sentence tokenizer to split the document
pulsion to take part in the annotation activity. into sentences. The annotators were asked to anno-
The cases were selected randomly to avoid bias tate these sentences with the rhetorical roles.
towards any entity, situation, or laws. Any meta-
B.2 Annotators Details
information related to individuals, organizations,
and judges was removed so as to avoid any intro- With the help of law professors, we designed a
duction of bias. For the application of corpus to course project centered around RR annotations for
judgment prediction task, we are not the first ones the student annotators. The students voluntar-
to do the task of judgment prediction. For the task, ily participated in the annotations as a part of the
we took all the steps (names anonymization and course project. Moreover, annotators were curious
removal of meta-information) as outlined in the about learning about AI technologies and further
already published work of Malik et al. (2021). The contributing towards its progress. There was no
focus of this paper is rhetorical role prediction, and compulsion to take part in the annotation activity.
the task of judgment prediction is only a use-case. The 6 annotators come from an Indian Law Uni-
Moreover, in this paper we focus mainly on IT and versity. Three of them specialize in Income Tax
CL cases where facts and scenarios are more ob- domain and the other three specialize in Competi-
jective and there are less biases compared to other tion Law domain.
types of cases (e.g., criminal and civil cases). As
B.3 Rhetorical Roles
also described by Malik et al. (2021), we do not
believe that the task could be fully automated, but We provide the definition of each of the Rhetorical
rather it could augment the work of a judge or legal Role in the main paper. Examples for each of the
practitioner to expedite the legal process in highly RR are given in Table 15. Figure 5 provides the
populated countries. number of sentences for each label in the IT and
Legal-NLP is a relatively new area; we have CL dataset. Note that representation of both the
taken all the steps to avoid any direct and fore- domains is similar with the exception of DIS label.
seeable ethical implications; however, a lot more
B.4 Secondary and Tertiary Annotation
exploration is required by the research community
Labels
to understand implicit ethical implications. For this
to happen, resources need to be created, and we are Legal experts pointed out that a single sentence
making initial steps and efforts towards it. can sometimes represent multiple rhetorical roles
(although this is not common). Each expert could
B Dataset and Annotations also assign secondary and tertiary rhetorical roles
to a single sentence to handle such scenarios and
B.1 Data Collection and Preprocessing
2
https://nclat.nic.in/
The IT and CL cases come from the Supreme Court 3
https://www.cci.gov.in/
of India, Bombay and Kolkata High Courts. For 4
http://compatarchives.nclat.nic.in
CL cases, we use the cases from the tribunals 5
http://www.nltk.org/

164
 4500
IT
4000

3578
CL
3495
Number of sentences

3500

2753
3000
2500 FAC 770 184 42 66 12 108 3 69 1200

1912
1751
2000 ARG 50 1293 26 4 35 83 3 30
1561

1000
1310
1196

1500 PRE 8 102 162 5 2 165 26 14

800
909

1000 STA 28 131 44 696 25 450 10 61

581
517
511

363
600

298
500 ROD 21 195 42 19 159 109 8 33

250
143

56
0 RLC 25 156 54 46 4 869 23 112 400

0
FAC ARG STA PRE ROD RLC RPC NON DIS RPC 1 2 11 0 0 76 203 0
200
NON 4 41 12 4 0 144 14 762
0
Figure 5: Distribution of RR labels in IT and CL docu-

A
G
E

D
C
C
N
FA

ST
PR

RL
RP
NO
RO
AR
ments.
(a) Between annotators A1 and A2 for IT domain
FAC 759 58 6 35 22 23 1 3 1200
motivate future research. On an average annotators ARG 152 1310 72 132 211 190 2 35
1000
assigned secondary role in 5-7% cases and assigned PRE 42 29 110 44 52 93 9 14
tertiary roles in 0.5-1% cases. STA 54 3 3 622 19 137 0 2
800

ROD 6 30 0 26 171 4 0 0 600

B.5 Inter-annotator Agreement RLC 104 37 23 320 106 1310 28 76 400
Fleiss Kappa between all (fine-grained) labels is RPC 3 4 28 10 9 36 186 14
200
0.59 for IT and 0.87 for CL, indicating substantial NON 73 31 8 61 35 208 0 665
0
agreement. We provide the inter-annotator agree-
C

A
G
E

D
C
C
N
FA

ST
PR

RL
RP
NO
RO
AR
ment (averaged pairwise macro F1 between anno- (b) Between annotators A2 and A3 for IT domain
tators) upon 13 fine-grained labels in Table 9. Also,
FAC 2686 0 44 0 18 0 4 0 2
we provide the pairwise confusion matrices of an- ARG 0 361 0 0 2 0 0 0 0
3000

notators (A1 , A2 ) and (A2 , A3 ) for both IT and CL PRE 25 0 2929 1 411 46 51 0 40 2500

domain in Figure 6. STA 1 0 2 44 1 0 5 0 0 2000

ROD 71 0 0 0 1829 0 13 0 0
1500
Label IT CL RLC 0 0 36 1 118 358 0 0 0
RPC 55 0 18 0 139 5 3334 10 3 1000
ARG-P 0.74 0.90 0 0 0 0 0 0 4 135 0
DIS 500
ARG-R 0.73 0.97 NON 1 0 9 0 0 0 53 0 463
0
FAC 0.77 0.88
C

A
G
E

D
C
C
DIS

N
FA

ST
PR

RL
RP

NO
RO
AR

ISS 0.75 0.75

(c) Between annotators A1 and A2 for CL domain
PRE-RU 0.67 0.86
FAC 2485 0 94 7 70 0 178 0 5
PRE-NR 0.58 0.80 3000
ARG 2 345 10 4 0 0 0 0 0
PRE-O 0.43 _ PRE 75 0 2666 111 0 106 71 0 9 2500
STA 0.78 0.89 STA 0 0 0 46 0 0 0 0 0 2000
RLC 0.58 0.74 ROD 27 2 419 12 1702 116 240 0 0
1500
0 0 120 5 1 273 10 0 0
RPC 0.75 0.74 RLC
1000
RPC 29 0 47 24 16 0 3289 19 40
ROD 0.64 0.93 DIS 0 0 0 25 0 0 42 78 0 500
DIS _ 0.98 NON 2 0 55 2 0 0 40 0 409
0
NON 0.45 0.52
C

A
G
E

D
C
C
DIS

N
FA

ST
PR

RL
RP

NO
RO
AR

F1 0.73 0.88
(d) Between annotators A2 and A3 for IT domain
Table 9: Label-wise inter-annotator agreement for all 13
Figure 6: Confusion matrix between Annotators for IT
fine-grained labels.
and CL domains.

B.6 Annotation Analysis

Annotation of judgments in order to identify and
distinguish between the rhetorical roles played by
165
its various parts is in itself a challenging task even role where the annotators have differed between
for legal experts. We provide some qualitative ex- themselves the most has been the point of Ruling
amples of sentences and their corresponding rhetor- made by the Lower Court, followed by the Ratio.
ical roles in Table 15 There are several factors in- This also ties in with the aforesaid argument that
volved in the exercise that requires the annotator all rhetorical roles are not equally susceptible to the
to retrace the judicial decision making and recreate variation caused by the varying levels of success
the impact left by the inputs available to the judge achieved by the different annotators in retracing
such as certain specific facts of the case, a particular the judicial thought pattern.
piece of argument advanced by the lawyer repre-
senting one of the parties, or a judicial precedent B.7 Annotation Case Studies
from a higher court deemed applicable in the cur-
rent case by the lawyer(s) or by the judge or by both. Along with law professors, we analyzed some of
Moreover, the annotator only has access to the cur- the case documents. Please refer to data files for
rent document which is secondary account of what the actual judgment.
actually happened in the court. These limitations In the case of CL cases, the best resemblance that
certainly makes the task of the annotator further has been achieved is in the case of SC_Competition
difficult, and leaves them with no choice other than Commission of India vs Fast Way Transmission Pvt
to make certain educated guesses when it comes Ltd and Ors 24012018 SC.txt, one would find that
to understanding the various nuances, both osten- the judgment has been written in a manner as to
sible and probable, of certain rhetorical roles. It provide specific indicators before every rhetorical
should, however, be noted that such variation need role. For instance, before the Ruling by Lower
not occur for every rhetorical role, since not all Court starts, reference has been made that this is
the roles are equally susceptible to it –for instance, the opinion given the Competition Commission of
the facts of the case as laid down by the judge India (the lower court in the relevant domain). Sim-
are more readily and objectively ascertainable by ilarly, before Arguments made by Petitioner/Re-
more than one annotator, whereas the boundaries spondent, reference has been made that this is the
between the issues framed by the judge and those argument made by the lawyer representing the peti-
deemed relevant as per the arguments advanced by tioner/respondent. This judgment also provides a
the lawyers may blur more, especially because if nice, consistent flow following the arrangement of
the judge happens to agree with one of the lawyers the rhetorical roles in order. The relatively smaller
and adopts their argument as part of the judicial size of the judgment also indicates a lower level
reasoning itself. Similarly, it should also be noted of complexity (although there need not always be
that despite differing in their views of the nature a consistent correlation between the two). On the
and extent of rhetorical role played by a certain other hand, if one considers the least resemblance
part of the judgment, the annotators may still agree achieved in the competition law domain, in the case
with each other when it comes to identifying and of SC_Excel Crop Care Limited vs Competition
segregating the final ruling made by the judge in Commission of India and Ors 08052017 SC(1).txt,
that case –this phenomenon of having used two one would find that such specific indicators are usu-
different routes to arrive at the same destination ally absent, thus leaving scope for individual discre-
is not uncommon in the reenactment or ex-post- tion and interpretation, the judgment goes back and
facto analysis of a judicial hearing and decision forth between certain rhetorical roles (Issue, Ruling
making process. A cumulative effect of the afore- by Lower Court, Ratio by Present Court, Argument
mentioned factors can be observed in the results of by Petitioner/Respondent, Precedent Relied Upon),
the annotation. The analysis provided by the three and the relatively bigger size also involves addi-
annotators in case of competition law bear close tional complexity and analysis, which make room
resemblance with each other. On the other hand, for further nuances as described above.
in case of income tax law, the analysis provided Similarly, if one considers the best resemblance
by Users 1 and 3 bear greater resemblance with that has been achieved in the income tax domain,
each other, compared to the resemblance between in the case of SC_2014_17.txt, one would find
Users 1 and 2, or between Users 2 and 3. On a the case has involved fewer rhetorical roles, cut
different note, it is also observed that the rhetorical down on facts (mainly dealing with procedural is-
sues on an appellate stage), and even among the
166
rhetorical roles, it has focused on statutes and pro- were conducted on Google Colab7 and used the de-
visions thereof and the ratio and ruling. This has fault single GPU Tesla P100-PCIE-16GB, provided
significantly reduced the possibility of the afore- by Colab. Our models were trained upon a single
mentioned richer jurisprudence, greater range of 11GB GeForce RTX 2080 TI. We used the SBERT
precedents, and resulting greater degree of sub- model provided in the sentence-transformers li-
jective interpretation being at play. On the other brary8 . We use the Huggingface9 implementations
hand, if one considers the least resemblance that of BERT-base and LEGAL-BERT models. Refer
has been achieved in the income tax domain, in the to Table 10, 11 and 12 for dataset-wise results and
case of SC_2008_1597.txt, discusses Precedents to hyperparameters for each model. We also provide
a greater detail including facts thereof, goes back the training time and number of parameters of each
and forth between certain rhetorical roles instead model in Table 13.
of maintaining a consistent order, and is not very For SBERT-Shift, we kept the SBERT model as
clear about whether the judge is at times merely fixed and tuned the 3 linear layers on top. We used
reiterating the arguments made by the lawyers, or the Binary Crossentropy loss function with Adam
is demonstrating their own view of such arguments. Optimizer to tune the model upon the LSP task.
Collectively, these leave the scope for greater in- For BERT-SC, we fine-tuned the pre-trained
volvement of subjective interpretation of the afore- BERT-base model upon the LSP task. We used
said nuances. the maximum sequence length of 256 tokens, a
Yet on an overall basis, the elements of subjec- learning rate of 2e − 5 and kept the number of
tivity, personal discretionary interpretation, and ar- epochs as 5 during training. We used the same loss
bitrariness have been minimized by the selection of function and optimizer as the SBERT-Shift model.
the chosen domains, along with the methodology
adopted for annotation, thus leading to the present D.1 Reduced Label Set
success attained in identification of rhetorical roles
and using the same for prior relevant case identifi- Due to the complexity of the task of RR predic-
cation and prediction. tion, we consider seven main labels (FAC, ARG,
PRE, ROD, RPC, RLC, and STA) only. We plan
C Evaluation Metrics to explore developing predictive models using fine-
grained labels.
We use the Macro F1 metric to evaluate the per-
NON Label: We ignore sentences with NON
formance of models upon the task of Rhetorical
(None) labels (about 4% for IT and 0.5% for CL).
Role labelling. Macro F1 is the mean of the label-
We believe that this was necessary since the inter-
wise F1 scores for each label. Given the true posi-
annotator agreement for the NON label in both IT
tives (T P ), false positives (F P ) and false negatives
and CL domains, has an F1 score as low as 0.45,
(F N ), the F1 score for a single label is calculated
implying that even the legal experts themselves do
as:
TP not agree whether a particular sentence has a NON
F1 =   (1) label.
T P + F P +F N
2 Dissent Label: Analysis of the annotated dataset
The pairwise inter-annotator agreement F1 be- reveals that the IT domain does not have any in-
tween two annotators A and B is calculated by stance of dissent (DIS) label. There were only
considering the annotations by annotator A as the three documents (out of 50) in the CL domain hav-
true labels and the annotations by annotator B as ing few instances of dissent label. Moreover, the
the predicted labels. instances of dissent label were present as a contigu-
We also calculate Fleiss Kappa6 to measure the ous chunk of sentences at the end of the document.
inter-annotator agreement. Hence, we discarded the sentences with dissent
labels. Furthermore, law experts told us that the
D Model Training Details dissent phenomenon is rare; from a practical (appli-
cation) point of view, these labels can be discarded.
All of our baseline experiments and training of La-
bel shift prediction models (SBERT and BERT-SC) 7
https://colab.research.google.com/
8
https://pypi.org/project/sentence-transformers/
6 9
https://en.wikipedia.org/wiki/Fleiss%27_kappa https://huggingface.co/

167
D.2 Single Sentence Classification Baselines we used shift embeddings for the shift component
We train single sentence classification models for and BERT embeddings for the RR component. We
the task of rhetorical role labelling. We use BERT- used the NLL loss in both components of the MTL
base-uncased and Legal-BERT models and fine- model weighted by the hyperparameter λ. We
tune them upon the sentence classification task. We use the Adam Optimizer for training. We provide
also try a variant of using context sentences (left dataset-wise hyperparameters and results in Tables
sentence and the right sentence) along with the 10 , 12 and 11.
current sentence to make classification, we call this D.5 Hyperparameter λ
method BERT-neighbor. We use CrossEntropyLoss
We tuned the hyperparameter λ of the MTL loss
as the criterion and Adam as the optimizer. We
function upon the validation set. We trained the
use a batch size of 32 with a learning rate of 2e-5
MTL model with λ ∈ [0.1, 0.9] with strides of 0.1
and fine-tune for 5 epochs for all our experiments.
and show the performance of our method on IT and
Refer to Tables 10 , 12 and 11 and for results and
IT+CL datasets in Figure 4. λ = 0.6 performs the
more information about the hyperparameters.
best for the IT domain and also performs competi-
D.3 Sequence Classification Baselines tively on the combined domains.

We experiment with Sequence Classification Base- D.6 Model Distillation

lines like CRF with handcrafted features, BiLSTM For model distillation experiments we trained the
with sent2vec embeddings and different versions teacher model with same hyperparameters in Table
of BiLSTM-CRF in which we varied the input em- 10 on the IT dataset. For the next two iteration of
beddings. We experimented with sent2vec embed- learning a student model, we used 48 unlabelled
dings fine-tuned on Supreme Court Cases of India cases in each iteration. The weighing hyperparam-
(same as in (Bhattacharya et al., 2019)). We also eter, αU was kept as 0.3. In each iteration, the
tried with sentence embeddings obtained from the student model was trained with a batch size 16, a
BERT-base model. In another experiment, we fine- learning rate of 0.005 and for 300 epochs.
tuned a pre-trained BERT model upon the task of
Masked Language Modelling (MLM) on the unla-
belled documents of IT and CL domain, and used
this model to extract the sentence embeddings for
the BiLSTM-CRF model.
We used the same implementation of BiLSTM-
CRF from (Bhattacharya et al., 2019), with Adam
optimizer and NLL loss function. Refer to Tables
10 , 12 and 11 for experiment-wise hyperparame-
ters.

D.4 LSP-BiLSTM-CRF and

MTL-BiLSTM-CRF models
In our proposed approach of LSP-BiLSTM-CRF,
we experiment with two methods of generating
shift embeddings, namely BERT-SC and SBERT-
Shift. These embeddings were then used as input to
train a BiLSTM-CRF with similar training sched-
ules. Refer to Tables 10 , 12 and 11 for other
hyperparameters.
For MTL models, we experimented with differ-
ent encoders E1 and E2 . We experimented with
using Shift embeddings (or BERT embeddings of
sentences obtained from pre-trained BERT model)
from BERT-SC in both the components. However,
the best performing model was the one in which
168
 Hyperparameters(E=Epochs),
(LR=Learning rate),
(BS=Batch Size),
Model (Dim=Embedding dimension), IT (Macro F1)
(E1=Embedding dimension Shift),
(E2=Embedding dimension RR),
(H=Hidden dimension),
BERT LR=2e-5, BS=32, E=5 0.56
BERT-neighbor LR=2e-5, BS=32, E=5 0.53
Legal-BERT LR=2e-5, BS=32, E=5 0.55
CRF(handcrafted) LR=0.01, BS=40, Dim=172, E=300 0.55
BiLSTM(sent2vec) LR=0.01, BS=40, Dim=200, H=100, E=300 0.55
BiLSTM-CRF(handcrafted) LR=0.01, BS=40, Dim=172, H=86, E=300 0.57
BiLSTM-CRF(sent2vec) LR=0.01, BS=40, Dim=200, H=100, E=300 0.59
BiLSTM-CRF(BERT emb) LR=0.01, BS=40, Dim=768, H=384, E=300 0.63
BiLSTM-CRF(MLM emb) LR=0.01, BS=40, Dim=768, H=384, E=300 0.58
LSP(SBERT) LR=0.005, BS=40, Dim=2304, H=1152, E=300 0.64
LSP(BERT-SC) LR=0.005, BS=40, Dim=2304, H=1152, E=300 0.65
LR=0.005, BS=40, E1=2304, E2=768 , H=1152(Shift),
MTL(MLM emb) 0.67
H=384(RR), E=300
LR=0.005, BS=40, E1=2304, E2=768, H=1152(Shift),
MTL(BERT-SC) 0.70
H=384(RR), E=300
LR=0.005, BS=40, E1=2304, E2=2304, H=1152(Shift),
MTL(BERT-SC) 0.68
H=384(RR), E=300
LR=0.005, BS=40, E1=768, E2=768, H=1152(Shift),
MTL(BERT-SC) 0.64
H=384(RR), E=300

Table 10: Hyperparameters and results on the IT dataset

Hyperparameters(E=Epochs),
(LR=Learning rate),
(BS=Batch Size),
Model (Dim=Embedding dimension), IT+CL (Macro F1)
(E1=Embedding dimension Shift),
(E2=Embedding dimension RR),
(H=Hidden dimension),
BiLSTM-CRF(sent2vec) LR=0.01, BS=40, Dim=200, H=100, E=300 0.65
BiLSTM-CRF(BERT) LR=0.01, BS=40, Dim=768, H=384, E=300 0.63
LSP-BiLSTM-CRF(BERT-SC) LR=0.005, BS=20, Dim=2304, H=1152, E=300 0.67
LR=0.005, BS=20, E1=2304, E2=768,
MTL-BiLSTM-CRF(BERT-SC) 0.70
H=1152(Shift), H=384(RR), E=300
LR=0.005, BS=20, E1=2304, E2=2304,
MTL-BiLSTM-CRF(BERT-SC) 0.68
H=1152(Shift), H=384(RR), E=300
LR=0.005, BS=20, E1=768, E2=768,
MTL-BiLSTM-CRF(BERT-SC) 0.65
H=1152(Shift), H=384(RR), E=300

Table 11: Hyperparameters and results on the combined (IT+CL) dataset

169
 Hyperparameters(E=Epochs),
(LR=Learning rate),
(BS=Batch Size),
Model (Dim=Embedding dimension), CL (Macro F1)
(E1=Embedding dimension Shift),
(E2=Embedding dimension RR),
(H=Hidden dimension),
BERT LR=2e-5, BS=32, E=5 0.52
BERT-neighbor LR=2e-5, BS=32, E=5 0.51
Legal-BERT LR=2e-5, BS=32, E=5 0.53
CRF(handcrafted) LR=0.01, BS=40, Dim=172, E=300 0.52
BiLSTM(sent2vec) LR=0.01, BS=40, Dim=200, H=100, E=300 0.54
BiLSTM-CRF(handcrafted) LR=0.01, BS=40, Dim=172, H=86, E=300 0.56
BiLSTM-CRF(sent2vec) LR=0.01, BS=40, Dim=200, H=100, E=300 0.61
BiLSTM-CRF(BERT emb) LR=0.01, BS=40, Dim=768, H=384, E=300 0.63
BiLSTM-CRF(MLM emb) LR=0.01, BS=40, Dim=768, H=384, E=300 0.60
LSP(SBERT) LR=0.005, BS=40, Dim=2304, H=1152, E=300 0.63
LSP(BERT-SC) LR=0.005, BS=40, Dim=2304, H=1152, E=300 0.68
LR=0.005, BS=20, E1=2304, E2=768 , H=1152(Shift),
MTL(MLM emb) 0.67
H=384(RR), E=300
LR=0.005, BS=20, E1=2304, E2=768, H=1152(Shift),
MTL(BERT-SC) 0.69
H=384(RR), E=300
LR=0.005, BS=20, E1=2304, E2=2304, H=1152(Shift),
MTL(BERT-SC) 0.67
H=384(RR), E=300
LR=0.005, BS=20, E1=768, E2=768, H=1152(Shift),
MTL(BERT-SC) 0.64
H=384(RR), E=300

Table 12: Hyperparameters and results on the CL dataset

Model No of Parameters Training Time(min)

IT CL IT CL
BiLSTM(sent2vec) 240000 240000 15 30
BiLSTM-CRF(sent2vec) 240000 240000 15 30
BiLSTM-CRF(BERT emb) 3538944 3538944 30 50
BiLSTM-CRF(MLM emb) 3538944 3538944 30 50
LSP(SBERT) 31850496 31850496 90 250
LSP(BERT-SC) 31850496 31850496 90 250
MTL(MLM emb) 35411060 35411060 300 1200
MTL(BERT-SC) 35411060 35411060 300 1200

Table 13: Approx. number of parameters and computational budget of models.

Model IT+CL docs F1

BERT-ILDC Predicted ROD & RPC using BiLSTM-CRF(sent2vec) 0.55
BERT-ILDC Predicted ROD & RPC using MTL(BERT-SC) 0.56

Table 14: Judgment Prediction results using predicted ROD & RPC

170
Label Sentence
It has also been alleged that the copies of the notices were also sent,
inter alia, to the principal officer of the said company and also to the ladies
Fact
as mentioned herein before, who has sold the immovable property
in question.
For executing this contract, the assessee entered into various contracts
Fact
-Offshore Supply contract and Offshore Service Contracts.
But the words inland container depot were introduced in Section 2(12)
Ruling By Lower Court
of the Customs Act, 1962, which defines customs port.
We may also mention here that the cost of superstructure was
Ruling By Lower Court Rs. 2,22,000 as per the letter of the assessee dated 28-11-66 addressed
to the ITO during the course of assessment proceedings.
Such opportunity can only be had by the disclosure of the materials to
Argument the court as also to the aggrieved party when a challenge is thrown to the
very existence of the conditions precedent for initiation of the action.
In this connection, it was urged on behalf of the assessee(s) that, for the
relevant assessment years in question, the Assessing Officer was required
Argument
to obtain prior approval of the Joint Commissioner of Income Tax before
issuance of notice under Section 148 of the Act.
In the meantime, applicant has to pay the additional amount of tax with
Statute interest without which the application for settlement would not
be maintainable.
On the other hand, interest for defaults in payment of advance tax falls
Statute under section 234B, apart from sections 234A and 234C, in section
F of Chapter XVII.
The State having received the money without right, and having retained
Ratio of the Decision and used it, is bound to make the party good, just as an individual
would be under like circumstances.
Therefore, the Department is right in its contention that under the
Ratio of the Decision
above situation there exists a Service PE in India (MSAS).
For these reasons, we hold that the Tribunal was wrong in reducing the
Ruling by Present Court penalty imposed on the assessee below the minimum prescribed
under Section 271(1)(iii) of the Income-tax Act, 1961.
Hence, in the cases arising before 1.4.2002, losses pertaining to exempted
Ruling by Present Court
income cannot be disallowed.
Yet he none the less remains the owner of the thing, while all the
Precedent
others own nothing more than rights over it.
I understand the Division Bench decision in Commissioner of
Precedent
Income-tax v. Anwar Ali, only in that context.
None Leave granted.
None There is one more way of answering this point.
Dissent Therefore a constructive solution has to be found out.
In the light of the Supreme Court decision in the case of CCI vs SAIL
Dissent
(supra) t his issue has to be examined.

Table 15: Example sentences for each label.

171