Noise induced hearing loss: Building an application using the ANGELIC methodology

2.1.Noise induced hearing loss

Amongst other things, Weightmans acts for employers and their insurance companies and advises them when they face claims from claimants for Noise Induced Hearing Loss (NIHL) where it is alleged that the hearing loss is attributable to negligence on the part of the claimant’s employer(s), or former employer(s), during the period of the claimant’s employment. Weightmans receive approximately 5,000 new NIHL instructions annually, the majority of which are at the pre-litigation stage. Liability in law to compensate the claimant cannot be established in more than half of the claims in which Weightmans are asked to advise (clients will not normally involve lawyers unless there is some substantial issue in the claim). When damages are paid, the average figure is around £3,700. Industry wide, the Association of British Insurers (ABI) figures show that 2013 was a peak year for NIHL claims notifications with the number reaching over 85,000 with an estimated cost of over £400 million.1616

In each case Weightmans typically represents both the insurer and the insured employer (as their interests are aligned). Weightmans advises whether the claimant (the current or former employee) has a good claim in law and, if appropriate, the likely amount of any settlement (which the insurer will normally meet on behalf of the employer). Weightmans’ role is thus to identify potential arguments which the employers or their insurance companies might use to defend or mitigate the claim. Compensators are thus looking to use the explicit statement of domain knowledge produced in this project primarily to improve their ability to settle valid claims and, when appropriate, pay proper and fair compensation in a timely manner, whilst using the application to challenge cases which may have no basis in law or may otherwise be defendable. In such an application it is essential that the arguments be identified. Black box pronouncements are of no use: it is the reasons that are needed. Note that the idea is to identify usable arguments, not to replicate the argumentation process. As Branting remarks in [20], “[machine learning] techniques typically are opaque and unable to support the rule-governed discourse needed for persuasive argumentation and justification”.

2.2.Need for explicit arguments

It is this need to find the arguments that are available in the particular case under consideration which militates against currently fashionable systems which exploit the use of the large amounts of data now available, typically using some form of Machine Learning. Consider, for example, [6] which received a good deal of attention in the UK national media in 2017.1717 It was designed to predict whether cases heard before the European Court of Human Rights would result in violations or non-violations of three Articles of the European Convention on Human Rights. It was stated in that paper that “our models can predict the court’s decisions with a strong accuracy (79% on average)”. They used a dataset comprising 250 cases related to Article 3, 80 to Article 6 and 254 to Article 8, balanced between violation and non-violation cases. Although the resulting model was intended to provide a way of predicting outcomes in pending cases before they are decided, availability dictated that they used the published decisions, written after the decsion had been made, to produce their models. Since these decisions reflect the outcome as well as the facts, this may raise questions and potential concerns about the methodology,1818 in that the decisions may contain indications of the outcome not in the documents presented during the trial, but that is not pertinent here. The 79% success rate reported does not represent strong enough accuracy for our purposes, since it is doubtful whether the kind of application wanted by Weightmans could tolerate an error rate in excess of 20% for the decision support task we are focussing on. Again that is not really the issue here. What does matter is that the system offers only a binary violation or non-violation answer. This is not what is needed for our application: Weightmans’ clients will expect some justification for why the claimant’s arguments are valid, or, if they are worth challenging, the exact reason for the challenge needs to be identified. It should be possible to attempt to discover the features being used to make the predictions: over 25 years ago rules were being extracted from a neural net trained on extensive cases data [13]. The task in [13] was to identify the 12 relevant features from cases containing 64 features. The system achieved over 98% accuracy in classifying cases (note that the input data was structured rather than natural language, accounting for the high success rate), but this was achieved without considering 4 of the 12 relevant factors, demonstrating that successful prediction does not prove that the model discovered is correct or complete.

No extraction of rules is attempted in [6], but they do offer “the 20 most frequent words, listed in order of their SVM [Support Vector Machine] weight”. These do not, however, look immediately promising: the list for topic 23 of article 6 predicting violation, for example, is:

2.3.Problems with uncritical use of past decisions

A major problem with looking at the past decisions to build a model for use with future cases, is that the past decisions may contain all sorts of, perhaps unconscious, biases and prejudices. The need to look critically at the training data is shown by Microsoft AI chatbot, Tay, a machine learning project,1919 designed for human engagement. As it learns from previous examples, some of its responses are inappropriate and indicative of the types of interactions some people are having with it. Indeed, so racist and misogynist were its learned responses that it was necessary to withdraw it very quickly. Although less extreme than so-called social media, any corpus is likely to embody some human foibles. For example [25] states:

This also applies to legal decisions which very often reflect the attitudes and assumptions prevalent at the time they were written. Provided that one is aware of, or even trying to uncover, such biases, this can be useful (e.g. [28], reported in The Guardian newspaper as “Judges are more lenient after taking a break, study finds”2020). In the newspaper article, Jonathon Levav, one of the co-authors of [28], is reported as saying

The techniques might, however, be usefully applied to the task of discovering which base-level factors are present in a case, in the manner of SMILE [10]. Here prejudices are less likely to have an impact, and importantly changes in the law (see below) tend to affect the attribution (as opposed to the significance) of base level factors only to a limited extent. A success rate of 80% is, however, unlikely to be good enough for many practical applications. If 20% of base level factors are misattributed, there is likely to be a significant impact on the arguments available. Given, however, the specificity of the target classification, especially if, like [10], that a separate model is developed for each factor, and the relatively short period involved, the accuracy may well be capable of improvement to acceptable levels. Such a system could then serve as a front end to the system we describe in this paper.2222

2.4.Argument mining

Another currently popular form of AI based on large corpora is argument mining (e.g. [33]). The idea here is to extract arguments from the corpus. Again this can prove interesting, especially if the arguments so extracted are subject to an informed critique, but it is not appropriate to the current task. For one thing, the arguments that will be extracted are as likely to be bad, unsound, arguments, as good ones. But even more significant is that our experts already know all the arguments that can be used before the courts: what they want is a system that can help less expert users to apply them accurately and consistently to particular cases, and to assess which are likely to succeed in particular cases. The task is not about finding arguments in general, but determining which are applicable to particular cases.

2.5.Change and development in the law

Another serious problem, which faces every corpus-based legal application, is the fact that law changes frequently. This makes it different from other domains. The rules of chess have not changed significantly since 1886 when Steinitz became the first world champion, and so games from that era can still provide instruction for players today. Although fashions change, and theory improves understanding of various positions, several opening variations introduced by Steinitz remain playable today. Thus the games collected in a corpus remain relevant and usable. Similarly the human body does not change (although our understanding of it and the treatment of diseases does) and so we need not worry about, for example, x-rays becoming obsolete. Also such corpora can be built on an international basis whereas law is peculiar to particular jurisdictions, and so large corpora are more difficult to assemble in law. In practice, the number of landmark legal cases required to understand a domain using traditional analysis techniques is relatively small: HYPO [8] used fewer than thirty cases, and the wild animal possession cases have produced a good deal of insight from an investigation using fewer than ten [11]. In law, unlike data mining, rules get their authority from their source, not from the number of supporting instances.

A major problem with law, however, is that it does change, and AI systems directed towards law need to take this into account. This has long been recognized: the authors of [21] describe how very soon after their group had expended a good deal of time and effort in building a detailed model of a piece of legislation, there was a substantial change in the law, casting doubt on the whole of their model. Such a change also renders any cases in a corpus related to the legislation dubious: since one cannot tell what is and what is not affected, one cannot use any of them, and must rebuild the corpus from scratch. Even if such wholesale changes in legislation are not so frequent, there is a constant stream of small updates and modifications which can cast doubt on previous cases and their decisions. At least changes resulting from amendments of the legislation can be precisely dated, but changes can also occur through case law, without legislative amendment. Detecting such changes was the topic of [38] and [19]. Shifts in case law are required to adapt to changing social conditions and changing attitudes and values. Judges very often have the discretion to respond to these changes, so that their decisions may differ from those that would have been made, under the same legislation, in previous years. For example, UK Unemployment Benefit was originally conceived of as a six day benefit, since when it was introduced in the late forties Saturday was normally seen as a work day, although there was an exception for those whose customary working week was Monday–Friday. Since then, however, a five day week has become the norm, and the exception is now the standard case. There has been no change in the legislation, but the decisions now reflect this social change. Similarly as Justice Marshall remarked in the context of a 1972 capital punishment case2323 “stare decisis must give way to changing values”.2424 Sometimes it is a matter of actual change, overturning decisions, but often it is refinement: as [14] shows in a discussion of the automobile exception to the US 4th Amendment. While the original case from the Prohibition era2525 remains the basis of the exception, over time other issues have become to be seen as relevant, most notably privacy.

The recognition that law is always changing, so that new decisions may overturn old ones, and that past decisions may require reinterpretation, means that methodologies for building legal knowledge based systems have been designed with the maintainability of the knowledge as a chief concern [17,27]. The development of ontologies in AI and Law was, in part, motivated by this need for maintainability [41]. As discussed in [3], the ANGELIC methodology is no exception: the principled modularisation which is a key feature is designed to facilitate maintenance. Responding to change using ANGELIC is discussed in detail in [2].

2.6.Knowledge elicitation versus machine learning

Thus, given the difficulties with corpora containing dubious, misleading and superseded cases, the nature of the task and the availability of genuine experts, the knowledge for our application will need to be elicited using traditional techniques and deployed in a program not dissimilar from a “good old fashioned” expert system. This seems to meet the current task requirements, which are to support and improve decision making in terms of speed, quality and consistency. The novelty resides in the use of a newly developed methodology, which uses the target structure to drive the elicitation process, and provides an effective form in which the knowledge is captured and recorded: unlike Softlaw it does not restrict itself to encoding written rules, but draws on other forms of documentation and expert knowledge, which may include specific experiences such as previous dealings with a particular site and common sense knowledge, and it structures this knowledge into a set of highly cohesive modules. In this way, the knowledge acquired using ANGELIC is not restricted to specific items of law, or to particular precedents, but can capture the wider negligence principles that experts distill from the most pertinent decisions. Since the knowledge encapsulated is a superset of what is produced in the CATO system [7], it could, were the task teaching law students to distinguish cases, equally well be deployed in that style of program. Note too that the analysis producing the knowledge, as in CATO, is performed by a human analyst and then applied to cases: the knowledge is not derived from the cases, nor is it a machine learning system.

Machine Learning techniques may, however, be able to provide support for moving from natural language to the base level factors required as input to our model. Currently this task relies on the judgement of the users and, as noted in our concluding remarks, some support would be highly beneficial. Different tasks for automation support may well lend themselves to different techniques. A very recent discussion of the limitations of deep learning in general can be found in [34]. Marcus lists ten problems with deep learning, some of which are very applicable to law and related to the above discussion, including that it works best when there are “thousands, millions or even billions of training examples”, a situation unlikely to be found in a particular domain of law; its lack of transparency; its inability to distinguish between causation and correlation, which is linked to the inclusion of biases and prejudices; that it presumes a largely stable world, which is certainly not the case with law; that, although it gives generally good results, it cannot be trusted, especially where the examples show small but significant deviations from the standard case (as often happens in law); and that deep learning models are hard to reuse and to engineer with. Marcus, however, remains excited about deep learning provided its limitations are recognised and the current hype does not lead to it being seen as a panacea: it should be seen “not as a universal solvent, but simply as one tool among many, a power screwdriver in a world in which we also need hammers, wrenches, and pliers, not to mentions chisels and drills, voltmeters, logic probes, and oscilloscopes.” Marcus argues for hybrid models, where deep learning and good old fashioned AI are used to supply appropriate parts of a hybrid system, much as we have suggested above. Discussions of what kinds of legal task are most suitabe for various AI approaches can be found in [20] and [9].

5.1.Document analysis

The initial discussion with the domain experts provided an excellent orientation in the domain and the key issues. These issues included the fact that claims were time limited, and so had to be made within three years of the claimant becoming aware of the hearing loss. Both actual awareness, usually the date of an examination, and constructive awareness (the date on which the claimant should have been aware that there was a problem) need to be considered. Then there is a question of the nature of the hearing loss: there are many reasons why hearing deteriorates, and only some of them can be attributed to exposure to noise. Then there is the possibility of contributory negligence: there is a Code of Practice with which the employers should have complied, and it is also possible that the employee was in part to blame, by not wearing the ear defenders provided, for example.

Next, the Information Document was used to identify the components that would appear in the ADF, putting some flesh on the skeleton that emerged from the initial discussion. The Information Document provides summaries of the main definitions, the development of the legal domain rules, the assessment of general damages for noise induced hearing loss cases, and Judicial College Guidelines for the assessment of general damages. Other medical conditions related to hearing loss are listed and described.

At this stage components had been identified, and where these were elaborated in terms of the conditions that were associated with them, links between these components could be identified. For example hearing loss can be sensioneural, but can also be attributed to a number of other factors: natural loss through aging, loss accompanying cardio-vascular problems, infections, certain drugs, etc. Only sensioneural loss can be noise induced, and so hearing loss arising from the other factors cannot be compensated. The Information Document gives an indication of the various different kinds of hearing loss, and then further information of what may cause the various kinds of loss.

At the end of this phase we have a number of concepts, some of which are elaborated in terms of less abstract concepts, and some potential links. The next step is to organise these concepts in an appropriate structure.

The check list document (see the list of source documents in Section 4) was kept back to be used after the concepts had been organised into a hierarchy, to determine whether the hierarchy bottomed out in sensible base-level factors. The check list comprised a set of 18 questions and a “traffic light” system indicating their effect on the claim. The idea was to associate base level factors with the answers to these questions. For example, question 1 asks whether the exposure ceased more than three years before the letter of claim: if it did not, the claim is ipso facto within limitation and other kinds of defence must be considered.

Similarly the cases were not used to build the initial ADF but were held back to provide a means of working through the ADF to check that the arguments deployed in those cases could be recovered from the ADF.

5.2.Component organisation

The main goal now is to move from unstructured information gathered from the documentation to structured information. The main issues had been identified in the initial discussion and the document analysis. These were used to identify and cluster the relevant intermediate predicates from the documents. These nodes were further expanded as necessary to produce further intermediate predicates and possible base level factors. The checklist was then used to identify, and where necessary add, base level factors. The documents from the sample of particular cases were used to provide examples of possible facts, and the effect these facts had on decisions. The result was a factor hierarchy diagram where the root shows the question to be answered, while the leaves show facts obtainable from the sample cases. All this was recorded in a table that described the factors in the domain and their related children (immediate descendants).3030 Here we will consider one particular node from the NIHL application, breach of duty:

The six children of breach of duty listed above are the main things required of an employer under the code of practice, and so provide a list of the ways in which a breach of duty might have occurred. They may be further elaborated: for example noise reduction includes measures such as shielding the machinery and providing appropriate ear protection.

The final version of the ADF for NIHL contains 3 issues, 20 intermediate nodes and 14 base level factors, with 39 links. For comparison, the ADF equivalent of CATO given in [3] contained 5 issues, 11 intermediate nodes, and 26 base level factors with 48 links. Thus CATO is larger, but NIHL has more internal structure. The nodes in the tabular presentation of the ADF are annotated to show their provenance (which of the documents they originate from, and the particular section or sections in which they are defined or explained), and whether they relate to particular questions in the checklist document. Although the NIHL ADF contains commercially confidential material, which means that it cannot be shown in this paper, there are several examples of both the diagrammmatic and tabular presentations of domain ADFs in [3].

5.3.Defining acceptance conditions

Once the nodes had been identified, acceptance conditions providing sufficient conditions for acceptance and rejection of the non-leaf nodes in terms of their children were provided. These were then ordered by priority and a default provided. The particular example cases provided were used to confirm that the arguments used in them could be recovered from the ADF. Continuing the breach of duty example:

Factor: Breach of duty

Acceptance conditions:

Each of these is a sufficient condition to identify a breach of duty. If none of them apply to the case, we can assume, as a default, that there was no breach of duty, and so add rejection of the node as the default.

5.4.Review and refinement

After this stage, the analysts and domain experts met to discuss and revise the initial ADF. The final ADF produced using the ANGELIC methodology is acyclic and so can be interpreted using grounded semantics.

Once a final ADF had been agreed, a Prolog program was produced from the acceptance conditions. The program was designed to suggest whether, given a particular set of case facts, there might be a plausible defence against the claim.

5.5.Program implementation

The program is implemented using Prolog, following the process described in detail in [3]. The program was created by writing a Prolog procedure for each node, starting with the leaf nodes and following the links to their parents until the root is reached. The Prolog procedure rewrites each test in the acceptance conditions as a clause and so determines the acceptability of each node in terms of its children. As in [3], which provides detailed examples, this required re-stating the tests using the appropriate syntax. Some reporting was added to indicate whether or not the node is satisfied, and through which condition. Finally some control was added to call the procedure to determine the next node, and to maintain a list of accepted factors. We do not give the example rules or program output here for reasons of commercial sensitivity, but their form is identical to that produced from the Trade Secrets domain ADF in [3]. The ADF acts as a design document, greatly easing the task of the progammer when compared with earlier systems such as [40] and [18] which presented the programmer with raw legislation to rewrite as Prolog code. The closeness between Prolog procedures and expressions of the acceptance conditions, each condition mapping to a clause within the Prolog procedure, makes the implementation quick, easy and transparent. The process of moving from acceptance conditions to Prolog code is essentially a mechanical rewriting into a template (supplying the reporting and control) and so is highly amenable to automation. Automated generation of the Prolog program from the ADF is planned as part of the development of the ANGELIC environment [1]. The program operates by:

5.6.Program refinement

Both the initial ADF and the program were, again, shown to and discussed with the domain experts, who suggested corrections and enhancements. The corrections varied: some suggestions were made about considering missing information from the document, modifying the interpretation of existing acceptance conditions, or adding base factors or new parents to base facts. No changes were related to the main issues or intermediate predicates. As explained in [2], responding to changes in ANGELIC can be easily controlled since although the changes affect nodes individually, because of the modularisation achieved by the ADF, they can be made within a single node, and the programmer can be confident that they will not have ramifications in the rest of the structure. Refinement was an iterative process which was repeated until an ADF acceptable to the domain experts was obtained.

5.7.User interface

The ADF encapsulates knowledge of the domain to add support in handling the analysis of the cases in the legal domain. To fulfill this task, a forms-based interface was designed in conjunction with some of the case handlers who carry out the task and so are the target users of the implemented system.

The designed interface enables the ADF to act as a decision augmentation tool for the particular task. Where the information is already available, for example, from an on-line claim form, some parts of the form can be pre-completed. A sample screen is shown in Fig. 1: the gender is pre-completed, but can be changed if necessary from a drop down menu.

Fig. 1.

Screen for User Interface.