Legal knowledge based systems
JURIX ’95
Telecommunication and AI & Law
The Foundation for Legal Knowledge Systems
Editors:
J.C. Hage
T.J.M. Bench-Capon
M.J. Cohen
H.J. van den Herik
F. Centinia, T. Routen, A. Hartmann and C. Hegarty, STATUTOR: Too intelligent by half?,
in: J.C. Hage, T.J.M. Bench-Capon, M.J. Cohen, H.J. van den Herik (eds.), Legal knowledge
based systems JURIX ’95: Telecommunication and AI & Law, Lelystad: Koninklijke
Vermande, 1995, 121-132, ISBN 90 5458 252 9.
More information about the JURIX foundation and its activities can be obtained by
contacting the JURIX secretariat:
Mr. C.N.J. de Vey Mestdagh
University of Groningen, Faculty of Law
Oude Kijk in 't Jatstraat 26
P.O. Box 716
9700 AS Groningen
Tel: +31 50 3635790/5433
Fax: +31 50 3635603
Email: sesam@rechten.rug.nl
© 1995 JURIX The Foundation for Legal Knowledge Systems
http://jurix.bsk.utwente.nl/
STATUTOR:
T OO INTELLIGENT BY HALF?
F. Centinia*, T. Routen*, A. Hartmann* and C. Hegarty‡
*Department of Computer Science, ‡School of Law
De Montfort University, The Gateway, Leicester LE1 9BH, United Kingdom
Tel: 0116 2551551 Ext. 8495, fax: 0116 2541891, email: twr@dmu.ac.uk
Abstract
This paper describes some of the issues that have arisen from the development of an
Intelligent Tutoring System to educate students in the statute-law domain. The
originality of the system consists of a graphical environment, in which the student can
represent valid legal arguments by constructing complex graphical structures. A brief
description of the system is given, mainly the graphical environment or tutorial module,
the authoring module and the expert-system module. The three modules share the same
knowledge base, that is the rule-based representation of statute law, providing an
interesting example of reusability of declarative knowledge. The rule-based
representation of the statute is discussed, with the main difficulties encountered in
catching the real meaning of the statute, while at the same time providing a suitable
representation format for the graphical display. The system provides assistance to the
student both during the process of constructing an argument and at the end of the
exercise; the latter consisting of the overall assessment of the student performance for the
given exercise, and the display of the correct answer as generated by the system. This
assessment is supported by a dynamic student-modelling approach, which is based on
the comparison of the student’s proof-tree and the correct proof-tree. The results of an
interim evaluation of the system are also provided in this paper.
1
Introduction
Intelligent Tutoring Systems (ITSs) are computer programs which are designed to provide
students with individualised, dedicated tutoring. This can only be achieved if ITSs know
who they teach, what they teach and how to teach it. These three main types of
knowledge are traditionally referred to in ITSs systems as the expert knowledge, the
student diagnostic knowledge and the instructional or curricular knowledge (Burns and
Capps, 1988). ITSs involve artificial-intelligence techniques, such as knowledge
representation, problem-solving approaches, dynamic student modelling, human
cognition, intelligent user interfaces (Frasson and Gauthier, 1990). ITSs need to be
generalised from applications too closely linked to a particular domain and moved
toward general purpose tools (Lawler and Yazdani, 1987). Those general purpose tools
for tutoring, or ITS shells, are intended to be applicable to teach different domains in the
way an expert system shell is used to implement expert systems in different domains. The
use of an ITS shell provides several advantages. A practical advantage is that the user of
the shell will have only to implement a knowledge base for the new domain and not a
complete new system (Sleeman, 1987). A second, theoretical advantage is that the main
knowledge of the shell can be oriented toward general theories and strategies of teaching,
at the same time providing a good means for testing their generality in different domains.
An important role in ITSs is played by the instructional environment. The instructional
environment defines the kinds of problems the student is to solve and the tools available
for solving them. The environment module may make explicit properties of the domain that
were previously hidden or implicit (Burton, 1988). For example, the graphical display in
the Geometry Tutor shows that geometry proofs are not linear, but tree-structured
(Anderson et al., 1985).
1995 JURIX
121
JURIX ’95: F. Centinia, T. Routen, A. Hartman and C. Hegarty
(Routen, 1992) described a prototype ITS shell called STATUTOR (Statute-Tutor) since
it was designed to support, amongst other things, the reuse of formalisations of statute
law in the presentation of exercises analogous to the familiar case analysis exercises of
traditional legal education. In 1994, the Joint Information Systems Committee of the UK
Higher Education Funding Councils, under its New Technologies Initiative, awarded a
support for a two-year project aimed at developing the prototype and producing a
finished system which could be made available to law schools. This paper reports on this
project, identifying some of the problems which arose and some of the issues raised by the
development.
2
Realistic knowledge base
attempts to educate students by requiring of them active learning in the
construction of complex structures, such as proof-trees representing a simple kind of legal
argument. Students are asked to demonstrate their understanding of the logical structure
of a piece of legislation by constructing an argument which shows how the legal
consequence of a number of case facts comes about. The argument consists of a number of
inferences, each inference represented by a graphical link from a number of facts to a
conclusion, which in turn may be then linked to establish further conclusions. The result
of this process is a graphical tree-like structure which serves as a proof-tree for the
conclusion.
STATUTOR was developed and demonstrated with only toy knowledge bases, and one
aim of this project was to provide a realistic knowledge base and associated exercises to
enable an effective evaluation of the potential of the system within the law curriculum. To
this end, in collaboration with a colleague in the school of law, the Data Protection Act
1984 (sections 1 and 21), was chosen to form the basis for the development of a
knowledge base which seeks to reflect the provisions of this law.
In developing the realistic knowledge base, several problems came to light. Firstly, it
became apparent that the system had to provide the flexibility for negated conditions to
appear in arguments. Secondly, it also became apparent that, although the main idea
behind STATUTOR was to treat the system’s ability to construct a proof-tree as analogous
to answering the case analysis exercise, students could not be expected to provide their
answer to the level of detail necessary for the system. That is, the system should not
require of students that they present ‘commonsense’ calculations such as ‘1992 > 1990’
as part of their arguments. Thirdly, we noted that it is possible that an argument may want
the same condition to play a role in establishing more than one intermediate conclusion,
which would necessitate the student duplicating part of his answer. This problem had to
be solved.
STATUTOR
2.1
Negative facts
In Figure 1 we can see a typical exercise as it is presented to the student. The conclusion
the student is required to prove is shown in a box at the centre top of the figure (text-box
in bold characters). The student needs to select from, and connect in the appropriate way,
the conditions shown in the set of boxes lined underneath the conclusion. These
conditions can be of three types: facts (text-boxes in underlined characters), negated
facts (as facts but with a NOT flag), and normal conditions which have to be proved by
using facts, negative facts or indeed other normal conditions. (The real system makes full
use of colours to distinguish among the different types of conditions). There are also red
herrings, that is conditions which are not required in the specific argument. The student
constructs his argument by using the tools on the left hand side of the figure. The open
book icon represents a source tool, this tool is used to label each inference in the prooftree with the portion of statute which warrants that particular inference.
122
STATUTOR:
Too intelligent by half?
Figure 1: The user interface of a typical exercise
In Figure 2, the student has drawn two inferences in order to prove the conclusion. In the
bottom-level inference the student states that three conditions are required to prove the
intermediate condition, and that this rule originates from the Data Protection Act, section
1, subsection (5), paragraph (h) and subparagraph (ii). In the top-level inference the
student believes that a single condition is required to prove the conclusion, and that
according to DPA 1(5).
Figure 2: Results of interaction with a student
2.2
Evaluable conditions
Here follows the representation of data holder as defined in DPA 1(5)(c):
123
JURIX ’95: F. Centinia, T. Routen, A. Hartman and C. Hegarty
srule(['DPA', '1', '(5)', '(c)'],
holds_data(USER, DATA),
[
/*rule conclusion*/
controls_content_and_or_use_of(USER, DATA),
/*normal condition*/
convertible(DATA, CONVERTED_DATA),
/*fact*/
different(DATA, CONVERTED_DATA),
/*evaluable condition*/
data(CONVERTED_DATA)]).
/*normal
condition*/
The above rule contains two normal conditions, which are conditions requiring to be
proved themselves, a fact, which can only be negated or asserted, and an evaluable
condition. Evaluable conditions are conditions that are evaluated internally by the
system. Students do not have to provide them in their construction of the legal argument,
though they must make sure that the remaining conditions are such that the evaluable
conditions are satisfied. The evaluable conditions are required in two main
circumstances. One is when the calculations involved are too complex, and we do not
want the student to perform them but only to concentrate on the conditions for their
successful termination. The other circumstance is when the facts stated in the evaluable
conditions are obvious from considering the other conditions, and therefore would be
quite futile questions to be asked to the student. Evaluable conditions are represented as
small circles in the correct proof-tree provided by the system (Figure 3).
Figure 3: A correct proof-tree with some evaluable condition
Students can click on the small circle to see the content of the evaluable condition ,
which in this case is clearly an obvious derivation of the other conditions (Figure 4).
In Figure 3 we can see the graphical representation of the rule just described. This
rule is represented between the second and third level in the proof-tree structure.
Students can also access the on-line source text of the statute by clicking, in the
proof-tree provided by the system, on the labels referring to the applied rules. Clicking
on the label DPA 1(5)(c) in Figure 3 for example, will present the following section of the
statute, with paragraph (c) highlighted, which represents the piece of legislation
formalised by the srule(['DPA', '1', '(5)', '(c)'], _) previously shown.
124
STATUTOR:
Too intelligent by half?
Figure 4: A sample evaluable condition
(5) "Data user" means a person who holds data, and a person "holds"
data if —
(a) the data form part of a collection of data processed or
intended to be processed by or on behalf of that person as
mentioned in subsection (2) above; and
(b) that person (either alone or jointly or in common with other
persons) controls the contents and use of the data comprised in
the collection; and
(c) the data are in the form in which they have been or are
intended to be processed as mentioned in paragraph (a) above or
(though not for the time being in that form) in a form into
which they have been converted after being so processed and
with a view to being further so processed on a subsequent
occasion.
2.3
Duplication of sub-trees
One of the problems caused by a fully modular rule-based representation is that it may
cause the same rule to be expanded several times in the same proof-tree. In fact, if each rule
is complete on its own, it means that almost each of them will require the basic definitions
to be proved. In the DPA this can be for example the Definition of data. Therefore if a
higher level rule calls a number of rules most of which contain the same Definition of
data, the student will be faced with the tedious task of repeating the same proof for
Definition of data several times. This problem can be solved by requiring the student to
supply the proof for the same statement only once, leaving the system to check that this
has been done at least once in the overall proof-tree. Another solution to this problem is
provided in the curriculum strategy (see dynamic student modelling and curriculum).
Negative conditions in the rule-based representation are explicitly represented in the
graphical display, so that the rule can be represented in the graphical display in its
completeness. At the moment we have dealt only with negated facts, which can be
interpreted either as negation by failure (we do not know whether they are true or false)
according to the closed world assumption, or as facts that have effectively been negated.
The system is to include the possibility of representing negative conditions when they
refer to a conclusion of another rule (normal conditions).
3
3.1
Interface developments
The forwards only exercise dialogue
125
JURIX ’95: F. Centinia, T. Routen, A. Hartman and C. Hegarty
Figure 5: A sample of a forwards reasoning dialogue
In the exercise dialogue seen in Figure 1, the student can proceed in both forwards and
backwards reasoning, as he is given all facts, conditions and conclusion, which he is
asked to link appropriately in the construction of the proof-tree. In the following
dialogue the student cannot reason backwards. The student is provided with only facts
and a set of rules (Figure 5).
The student links a set of facts and their appropriate rule to the unknown conclusion,
and then asks the system to produce the conclusion (Figure 6). If the student’s inference
is correct, the unknown conclusion will be changed into the conclusion derived by the
inference.
Once a correct inference is made, a new unknown conclusion is generated by the
system, so that the student can continue the forward-reasoning process until the final
conclusion is reached (Figure 7). When the final conclusion is reached the unknown
Figure 6: How the system may produce a conclusion
126
STATUTOR:
Too intelligent by half?
conclusion is not generated, meaning that the exercise was successfully completed.
Figure 7: Repeating the final conclusion
3.2
Node feedback
The node feedback tool provides students with immediate help when they find
themselves in difficulties. If students are not certain of their proof regarding the
intermediate node, they may ask for some help or hint on how to prove that particular
node. The feedback is based on each singular node or rules to solve that particular node
Feedback based on providing the premises, conclusion and applied-rule (the reference to
the portion of STATUTOR applied) for each rule appears to be the most effective for the
student (McKendree, 1990).
The variety of different kinds of feedback available by the student includes referencing
the primary source material (the statute); the rules which apply in the construction of the
particular node, in addition to an immediate, graphical diagnosis of their current answer.
The graphical feedback on the particular node would look as in Figure 8, where students
are hinted that two of the conditions they provided are correct, one condition is wrong
(double crossed box), and one condition is missing (shadowed box). A diagnosis is also
made on the label used to refer to the statute applied, by using symbolic characters in
place of Wrong, Missing or Surplus references.
Figure 8: Four conditions distinguished by class
The reference diagnosis can be seen more explicitly by clicking on the symbolised
diagnosis, as in Figure 9.
127
JURIX ’95: F. Centinia, T. Routen, A. Hartman and C. Hegarty
Figure 9: The references, answers and diagnosis
The feedback described so far is student-controlled feedback. The student decides if and
when feedback is required. The system also provides some kind of system-controlled
feedback. For example, the student is informed of this mistake when attempting to justify a
fact by pointing an arrow link toward it (a fact does not require to be justified).
In fact, by recording the various diagnoses made during the student’s problemsolving activity, the system can evaluate the student not only on the basis of the present
state of the exercise, but considering also the process through which the present state
was reached. The mechanism also provides a good means of controlling the type of
feedback we can provide to the student. In contrast with immediate feedback (Anderson et
al., 1990) where the system intervenes after each error made by the student, and final
feedback, where feedback is given only at the end of the exercise without considering the
intermediate steps, this mechanism gives us the possibility of intervening at any stage of
the student’s problem-solving activity, and allowing the student to make, for a given
situation, a more significant number of errors before intervening.
Students requiring a node feedback show some lack of knowledge on the rule used in
that particular node, and according to the degree of assistance they require we can make
assumptions about how much they know about the applicability of that particular rule
and therefore of the portion of statute associated with it. By requiring assistance students
have nevertheless shown some doubts about their knowledge of the domain, which
reduces the degree of error in the final proof-tree. This kind of student diagnosis can be
used in selecting the next exercises to be given to the student and indeed the format in
which these exercises should be presented. Students lack of knowledge of the use of a
particular rule will lead to exercises where that rule will be likely to reappear. On the
other hand when the student appears to have mastered the use of a particular rule, the
following exercises may not deal with the same rule, or, if they do, the rules will not be
required to be expanded (the conclusion of the rule may be provided as a fact). Collapsing
the expansion of a mastered rule into a simple fact can be done either dynamically or by
the human tutor when using the authoring mode. In fact the human tutor can decide before
constructing the new exercise which rules are to be expanded and which are not. Another
beneficial result of collapsing mastered rules into simple facts lies in avoiding the
128
STATUTOR:
Too intelligent by half?
repetition of the same rule expansion within a single proof-tree, as it is likely that the
most mastered rules are the ones at the lowest level.
4
Evaluation
A formative evaluation of the system has been recently performed whereby a small number
of subjects were guided in a rather formal manner through exercises with the system, and
subsequently questioned about their impressions and ideas concerning the applicability
of the system. Furthermore, the system was shown to a number of law tutors for the
purpose of eliciting in an informal manner perspectives of experienced educators.
The general impression of all the subject trials was fairly consistent. All test subjects
understood the presentation of the legal argument with the help of the graphical
representation used. They enjoyed using the system and developed new perspectives
how to structure arguments logically . Therefore it can be concluded that the graphical
presentation chosen is a useful and successful one. All the subjects considered the
system to be useful as an introduction to statute law and the Data Protection Act,
supported and supervised by a human tutor, but restricted to an introductory stage.
The subjects managed to complete the given exercises in a reasonable time. Subjects
with no previous exposure to the Data Protection Act concentrated on the graphical
feedback and solved the exercises mainly with graphical help, without consulting the
statute text. The students who had previous experience with the Data Protection Act
attempted to solve as much as possible of the exercises by relying solely on their
previous knowledge and the statute text. These students consulted the graphical
feedback only to verify and correct their solution. This supports the conclusion that the
students should be given an introduction into the statute text prior to the exercise. The
system would help them further to understand the statute law, rather than give them an
introduction into it. The graphical diagnosis as well as the graphical correct solution
were understood very well by the subjects and they enjoyed using them. They liked the
graphical presentation to such an extent that most of the students would like to refer to a
similar tool allowing them to freely construct their own exercises. In summary, students’
logical perception of the arguments actually improved, and students not previously
exposed to the domain managed to solve the paper-exercise given to them after the trials
surprisingly well.
The main criticism of the system, as expressed by both students and tutor, concerned
the rule-based representation of the statute law. Students did not make much use of, and
did not understand, the feedback providing them with the rule-format of the node they
were attempting to construct. Students and tutors would like to see the argument’s text
being phrased in a more articulate way, and not being restricted to the rigid format
required by the expert system. Furthermore, both students and teachers found the domain
too limited as it is restricted to only two sections of the DPA.
During these evaluations, we came upon a valuable but paradoxical insight: that by
making the system simpler, it could be made more useful. We are used to expecting that
injecting a little intelligence into the system will enhance its capabilities, but in the case
of the present system, for the domain of statute law at least, it may well be that it is
possible to decompose the elements of the system which law tutors have responded
positively to, from those elements of the system which seem to restrict its applicability. In
short, as long as we maintain the interface but dispense with the knowledge-based
aspects, then we may well be able to have more impact in presenting technological
assistance for law tutors.
As a result of this insight, we are investigating the possibility of a new system,
implemented in Visual Basic, which would apply the same principles as the current
system, but without requiring a rule-based representation of the law domain. In the new
system, the authoring tool permits the human tutor freely to edit both the argument’s
texts and structure. In brief, the tutor will construct his own graphical proof-tree, editing
129
JURIX ’95: F. Centinia, T. Routen, A. Hartman and C. Hegarty
the text boxes and structuring the argument as he wishes. The only drawback of this
approach is that the human tutor will not be provided with the argument structure, but
will have to construct it himself. Much of the functionality of the system as discussed in
this paper, such as feedback, tree-comparison and tutorial action, can then be applied by
using the graphical argument structure as constructed by the human tutor. With this
approach human tutors can build structured arguments related to any piece of law,
importantly including case law, without the requirements of the law having been
previously formalised.
5
Conclusion
This project was predicated on the idea that the ITS shell is of interest to law tutors, but
it was also predicated on the idea that it could be of interest in other subject domains too.
The reason why the project existed in the first place was because of positive responses
from law tutors to presentations of STATUTOR. However, the hypothesis now under
consideration is that those positive responses were not related to the fact that the system
could answer its own questions and had an expert system component too, but rather that
they were related to the interaction which the system provided. We hope to be in a
position in the near future, with two well-developed systems, one knowledge-based and
the other based on a much simpler technology, to be able to provide a conclusive answer
to this question.
References
Anderson, J.R., Boyle, C.F., and Yost, G. (1985). The Geometry Tutor. Proceedings of
International Joint Conference on Artificial Intelligence.
Anderson, J.R., Patterson, E.G., and Corbett, A.T. (1990). Student Modelling and
Tutoring Flexibility in the Lisp Intelligent Tutoring System. Intelligent Tutoring
Systems (At the Cross-roads of Artificial Intelligence and Education) (eds. C.
Frasson and G. Gauthier). Ablex Publishing Corporation, Norwood, New Jersey.
Burns, H.L., and Capps, C.G. (1988). Foundations of Intelligent Tutoring Systems: An
Introduction. Foundations of Intelligent Tutoring Systems (eds. M.C. Polson and J.J.
Richardson). Lawrence Erlbaum Associates, Hillsdale, N.J.
Burton, R.R. (1982). Diagnosing bugs in a simple procedural skill. Intelligent Tutoring
Systems (eds. D. Sleeman and J.S. Brown). Academic Press, London.
Burton, R.R. (1988). The Environment Module of Intelligent Tutoring Systems.
Foundations of Intelligent Tutoring Systems (eds. M.C. Polson and J.J. Richardson).
Lawrence Erlbaum Associates, Hillsdale, N.J.
Clancey, W.J. (1982). Tutoring rules for guiding a case method dialogue. Intelligent
Tutoring Systems (eds. D. Sleeman and J.S. Brown). Academic Press, London.
Frasson, C., and Gauthier, G. (1990). Introduction. Intelligent Tutoring Systems (At the
Cross-roads of Artificial Intelligence and Education) (eds. C. Frasson and G.
Gauthier). Ablex Publishing Corporation, Norwood, New Jersey.
Lawler, R.W., and Yazdani, M. (1987). Introduction. Artificial Intelligence and
Education. Volume One (eds. R.W. Lawler and M. Yazdani). Ablex Publishing,
Norwood, New Jersey.
McKendree, J. (1990). Effective Feedback Content for Tutoring Complex Skills. HumanComputer Interaction, Vol. 5, pp. 381-413. Lawrence Erlbaum Associates.
Routen, T. (1991). Complex Input: A Practical Way of Increasing the Bandwidth for
Feedback and Student Modelling in a Statute-Based Tutoring System. Proceeding of
the Third International Conference on Artificial Intelligence and Law. Oxford,
England.
Routen, T. (1992). Reusing formalisation of legislation in a tutoring system. Artificial
Intelligence Review 6, pp. 145-159. Kluwer Academic Publishers.
130
STATUTOR:
Too intelligent by half?
Sleeman, D. (1987). Pixie: a shell for developing intelligent tutoring systems. Artificial
Intelligence and Education. Volume One (eds. R.W. Lawler and M. Yazdani). Ablex
Publishing, Norwood, New Jersey.
131
132