Article

Acknowledging insufficiency in the evaluation of legal knowledge-based systems: strategies towards a broadbased evaluation model

Authors:

Maria Jean J. Hall,

John ZeleznikowAuthors Info & Claims

ICAIL '01: Proceedings of the 8th international conference on Artificial intelligence and law

Pages 147 - 156

https://doi.org/10.1145/383535.383553

Published: 01 May 2001 Publication History

Abstract

This paper considers the need for evaluation of knowledge-based systems in general and legal knowledge-based systems in particular. Some special features of legal knowledge-based systems pertinent to their evaluation are presented. The expected benefits of such evaluations are discussed. and some of the difficulties likely to be encountered are outlined.

The proceedings of four International Conferences on Artificial Intelligence and Law are analysed to determine the rate of reporting evaluations in non-theoretical papers. These papers had a low rate of consideration of evaluation issues reflecting common practice in research biased development environments. These results confirm that more attention to evaluation is needed in the legal knowledge based systems domain.

This paper foreshadows the development of an evaluation methodology tailored specifically for legal knowledge-based systems. Evaluation strategies beyond verification and validation are drawn upon, both from the international ISO/IEC 14598 and 9126 standards and also from previous work on evaluation models for knowledge-based systems.

References

[1]

ACM Code of Ethics and Professional Conduct. http://www.acm.org/constitution/code.html.]]

[2]

ACS Australian Computer Society Code of Ethics. http://www.acs.org.au/national/pospaper/acs131.html.]]

[3]

Adelman, L. Integrating evaluation methods into the DSS development process. Information & Decision Technologies, 15(4), 1989, 227-41.]]

[4]

Adelman, L., Gualtieri, J., and Riedel, S.L. A multifaceted approach to evaluating expert systems. Artificial Intelligence for Engineering Design, Analysis & Manufacturing, 8(4), 1994, 289-306.]]

[5]

Avison, D., et al. Incorporating evaluation in the information systems development process. in Proceedings of 3rd Evaluation of Information Technology Conference. (Henley, UK, 1995).]]

[6]

Back, B. Validating an expert system for financial statement planning. Journal of Management Information Systems, 10(3), 1993, 157-77.]]

Digital Library

[7]

Ballantine, J., et al. An ethical perspective on information systems evaluation. International Journal of Agile Management Systems, 2(3), 2000, 233-41.]]

[8]

Boehm, B.W. Software Engineering Economics. Prentice Hall, 1981.]]

Digital Library

[9]

Boehm, B.W. A spiral model for software development and enhancement. Computer, 21(5), 1988, 61-72.]]

Digital Library

[10]

Boehm, B.W., et al. Characteristics of Software Quality. North Holland, New York, 1978.]]

[11]

Borenstein, D. Towards a practical method to validate decision support systems. Decision Support Systems, 23(3), 1998, 227-239.]]

Digital Library

[12]

Cairo, O. Tracking knowledge acquisition frameworks evaluation. Under review for presentation at DEXA 2001.]]

[13]

Caraca-Valente, J.P., et al. Knowledge-based systems' validation: when to stop running test cases. International Journal of Human-Computer Studies, 51(4), 1999, 757- 81.]]

Digital Library

[14]

Cohen, P.R. and Howe, A.E. Toward AI research methodology: three case studies in evaluation. IEEE Transactions on Systems, Man & Cybernetics, 19(3), 1989, 634-46.]]

[15]

Dijkstra, J.J. User interaction with legal knowledge-based systems. in Proceedings of Legal Knowledge and Information Systems, Jurix 2000. (Amsterdam, 2000), IOS Press, Amsterdam, 1-10.]]

[16]

ESSISCOPE Information technology - Software Product Evaluation. http://www.cse.dcu.ie/ essiscope/sm3/product/14598.html.]]

[17]

Farbey, B., Land, F., and Targett, D. A taxonomy of evaluation methods. in Proceedings of the First European Conference on IT Investment Evaluation. (Henley-on-Thames, UK, 13-14 Sept. 1994.), Oper. Res. Soc., Birmingham, UK., 44-70.]]

[18]

Farbey, B., Land, F., and Targett, D. Moving IS evaluation forward: learning themes and research issues. Journal of Strategic Information Systems, 8(2), 1999, 189-207.]]

[19]

Gonzalez, A.J. and Barr, V. Validation and verification of intelligent systems-what are they and how are they different? Journal of Experimental & Theoretical Artificial Intelligence, 12(4), 2000, 407-20.]]

[20]

Gonzalez, A.J., Gupta, A.G., and Chianese, C.B. Performance evaluation of a large diagnostic expert system using a heuristic test case generator. Engineering Applications of Artificial Intelligence, 9(3), 1996, 275- 84.]]

[21]

Gonzalez, A.J., Xu, L., and Gupta, U.M. Validation techniques for case-based reasoning systems. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 28(4), 1998, 465-467.]]

Digital Library

[22]

Grogono, P., et al. Expert system evaluation techniques: a selected bibliography. Expert Systems, 8(4), 1991, 227-40.]]

[23]

Groothuis, M.M. and Svensson, J.S. Expert system support and juridical quality. in Proceedings of Legal Knowledge and Information Systems, Jurix 2000. (Amsterdam, 2000), IOS Press, Amsterdam, 1-10.]]

[24]

Guida, G. and Mauri, G. Evaluating performance and quality of knowledge-based systems: foundation and methodology. IEEE Transactions on Knowledge & Data Engineering, 5(2), 1993, 204-24.]]

Digital Library

[25]

Gupta, U.G. Validation and verification of knowledgebased systems: a survey. Journal of Applied Intelligence, 3(4), 1993, 343-363.]]

[26]

Hamilton, D. and Kelley, K. State-of-the-practice in knowledge-based system verification and validation. Expert Systems with Applications, 3(4), 1991, 403-10.]]

[27]

Harter, D.E. and Slaughter, S.A. Process maturity and software quality: A field study. in Proceedings of Twenty-first International Conference on Information Systems. (Brisbane Australia, Dec 10-13, 2000), 407- 411.]]

Digital Library

[28]

Henderson, R., Podd, J., Smith, M., Varelaalvarez, H. An examination of four user-based software evaluation methods. Interacting with Computers, 7(4), 1995, 412- 432.]]

[29]

Juristo, N. and Morant, J.L. Common framework for the evaluation process of KBS and conventional software. Knowledge Based Systems, 11(2), 1998, 145-159.]]

Digital Library

[30]

Knauf, R., Gonzalez, A. J., Jantke, K. P. Validating rule-based systems: a complete methodology. in Proceedings of IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028). (Tokyo, Japan, 12-15 Oct. 1999.), IEEE. Piscataway, NJ, USA., 744-9.]]

[31]

Knauf, R., Philippow, I., and Gonzalez, A.J. Towards validation and refinement of rule-based systems. Journal of Experimental & Theoretical Artificial Intelligence, 12(4), 2000, 421-31.]]

[32]

Lee, S. and O'Keefe, R.M. Developing a strategy for expert system verification and validation. IEEE Transactions on Systems Man and Cybernetics, 24(4), 1994, 643-55.]]

[33]

Liebowitz, J. Issues in verification and validation of expert systems. in Proceedings of Fourth International Symposium on Knowledge Engineering. Technical Sessions. (Barcelona, Spain, 7-11 May 1990.), Univ. Politec. Madrid., pp.105-13.]]

[34]

Ligezinski, P. and Hall, M.J.J. Designing flexible software to accommodate dynamic user requirements: An alternative solution to a continuing IS problem. in Proceedings of World Conference on Systemics, Cybernetics and Informatics ISAS '97. (Caracas, Venezuela, 1997), International Institute of Informatics and Systemics.]]

[35]

Menzies, T. Critical Success metrics: evaluation at the business level. International Journal of Human- Computer Studies, 51(4), 1999, 783-799.]]

Digital Library

[36]

Meseguer, P. and Plaza, E. The VALID project: goals, development, and results. International Journal of Intelligent Systems, 9(9), 1994, 867-92.]]

[37]

Michels, J.E., Abel, T., Knauf, R., Gonzalez, A. J. Investigating the validity of a test case selection methodology for expert system validation. in Proceedings of FLAIRS-98. Proceedings of the Eleventh International Florida Artificial Intelligence Research Symposium Conference. (Sanibel Island, FL, USA. . 18-20 May 1998), AAAI Press. pp. Menlo Park, CA, USA., 462-6.]]

Digital Library

[38]

Mommers, L. and van den Herik, H.J. Knowledge criteria for the evaluation of legal beliefs. in Proceedings of Legal Knowledge Based Systems, Jurix 1999. (Nijmegan, 14-18 June 1999), Int. Assoc. Artificial Intelligence & Law, 73-83.]]

Digital Library

[39]

Murrell, S. and Plant, R.T. A survey of tools for the validation and verification of knowledge-based systems: 1985-1995. in Proceedings of Decision Support Systems. (Montreal, Que., Canada, Dec. 1997), Elsevier Netherlands, 307-23.]]

Digital Library

[40]

O'Keeffe, R.M. and O'Leary, D.E. Expert system verification and validation: a survey and tutorial. Artificial Intelligence Review, 7(1), 1993, 3-42.]]

[41]

Onoyama, T. and Tsuruta, S. Validation method for intelligent systems. Journal of Experimental & Theoretical Artificial Intelligence, 12(4), 2000, 461-72.]]

[42]

Oskamp, A. and Tragter, M.W. Automated legal decision systems in practice: the mirror of reality. Artificial Intelligence & Law, 5(4), 1997, 291-322.]]

Digital Library

[43]

Paulk, M.C., et al. Capability Maturity Model for Software version 1.1. 1993, Software Engineering Institute.]]

[44]

Raggad, B.G. and Gargano, M.L. Expert system defection and perfection. Logistics Information Management, 12( 5), 1999, 395-406.]]

[45]

Rees, P.L. User Evaluation of Expert Systems. Industrial Management & Data Systems, 1992(6), 1992, not available.]]

[46]

Reich, Y. Measuring the value of knowledge. International Journal of Human-Computer Studies, 42(1), 1995, 3-30.]]

Digital Library

[47]

Shadbolt, N., O'Hara, K., and Crow, L. The experimental evaluation of knowledge acquisition techniques and methods: history, problems and new directions. International Journal of Human-Computer Studies, 51(4), 1999, 729-55.]]

Digital Library

[48]

Sharma, R.S. and Conrath, D.W. Soft measures for performance analysis - the core dimensions of expert system quality. Microelectronics & Reliability, 36(6), 1996, 775-796.]]

[49]

Stranieri, A. and Zeleznikow, J. The evaluation of legal knowledge based systems. in Proceedings of Seventh International Conference on Artificial Intelligence and Law. (Oslo, Norway, 1999), ACM Press, New York, NY, 18-24.]]

Digital Library

[50]

Suen, C.Y., et al. Verifying, validating, and measuring the performance of expert systems. Expert Systems with Applications, 1(2), 1990, 93-102.]]

[51]

Susskind, R.E. Expert systems in law. Out of the research laboratory and into the marketplace. in Proceedings of ICAIL-1987. (Boston. MA, 1987), ACM Press, 1-8.]]

Digital Library

[52]

Terano, T. The JIPDEC check-list based guideline for expert system evaluation. International Journal of Intelligent Systems, 9, 1994, 893-925.]]

[53]

Vermesan, A.I. Software certification for industryverification and validation issues in expert systems. in Proceedings of Proceedings Ninth International Workshop on Database and Expert Systems Applications. (Vienna, Austria., 26-28 Aug. 1998.), IEEE Comput. Soc Los Alamitos, CA, USA., 3-14.]]

Digital Library

[54]

Vermesan, A.I. and Bench-Capon, T. Techniques for the verification and validation of knowledge-based systems: a survey based on the symbol/knowledge level distinction. Software Testing Verification & Reliability, 5(4), 1995, 233-71.]]

[55]

Zeleznikow, J. Building decision support systems in discretionary legal domains. International Review of Law, Computers and Technology, 14(3), 2000, 341-356.]]

[56]

Zeleznikow, J. and Hunter, D. Building Intelligent Legal Information Systems. Representation and Reasoning in Law. Kluwer Law and Taxation Publishers, Deventer, Netherlands, 1994.]]

Cited By

Novotná TLibal T(2022)An Evaluation of Methodologies for Legal FormalizationExplainable and Transparent AI and Multi-Agent Systems10.1007/978-3-031-15565-9_12(189-203)Online publication date: 23-Sep-2022
https://doi.org/10.1007/978-3-031-15565-9_12
Bartolini CLenzini GSantos C(2019)An Agile Approach to Validate a Formal Representation of the GDPRNew Frontiers in Artificial Intelligence10.1007/978-3-030-31605-1_13(160-176)Online publication date: 11-Oct-2019
https://doi.org/10.1007/978-3-030-31605-1_13
Conrad JAl-Kofahi KKeppens JGovernatori G(2017)Scenario analyticsProceedings of the 16th edition of the International Conference on Articial Intelligence and Law10.1145/3086512.3086516(29-37)Online publication date: 12-Jun-2017
https://dl.acm.org/doi/10.1145/3086512.3086516
Show More Cited By

Index Terms

Acknowledging insufficiency in the evaluation of legal knowledge-based systems: strategies towards a broadbased evaluation model

Recommendations

A process for evaluating legal knowledge-based systems based upon the context criteria contingency-guidelines framework
ICAIL '03: Proceedings of the 9th international conference on Artificial intelligence and law

In an ideal world, a legal knowledge-based system would be evaluated by an evaluator with expertise in both the legal domain and software engineering evaluation processes. However in the real world, this task is typically undertaken by legal ...
Verification and Validation of Knowledge-Based Systems

Knowledge-based systems (KBS) are being used in many applications areas where their failures can be costly because of the losses in services, property, or even life. To ensure their reliability and dependability, it is therefore important that these ...
The evaluation of legal knowledge based systems
ICAIL '99: Proceedings of the 7th international conference on Artificial intelligence and law

Evaluation strategies to assess the effectiveness of legal knowledge based systems enable strengths and limitations of systems to be accurately articulated. This facilitates efforts in the research community to develop systems and also promotes the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICAIL '01: Proceedings of the 8th international conference on Artificial intelligence and law

May 2001

234 pages

ISBN:1581133685

DOI:10.1145/383535

Editor:
Ronald P. Loui
Washington Univ., St. Louis, MO

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 May 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICAIL01

Sponsor:

SIGAI

ICAIL01: The 8th International Conference on Artificial Intelligence and Law

Missouri, St. Louis, USA

Acceptance Rates

Overall Acceptance Rate 69 of 169 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
918
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Novotná TLibal T(2022)An Evaluation of Methodologies for Legal FormalizationExplainable and Transparent AI and Multi-Agent Systems10.1007/978-3-031-15565-9_12(189-203)Online publication date: 23-Sep-2022
https://doi.org/10.1007/978-3-031-15565-9_12
Bartolini CLenzini GSantos C(2019)An Agile Approach to Validate a Formal Representation of the GDPRNew Frontiers in Artificial Intelligence10.1007/978-3-030-31605-1_13(160-176)Online publication date: 11-Oct-2019
https://doi.org/10.1007/978-3-030-31605-1_13
Conrad JAl-Kofahi KKeppens JGovernatori G(2017)Scenario analyticsProceedings of the 16th edition of the International Conference on Articial Intelligence and Law10.1145/3086512.3086516(29-37)Online publication date: 12-Jun-2017
https://dl.acm.org/doi/10.1145/3086512.3086516
Berry PDonneau-Golencer TDuong KGervasio MPeintner BYorke-Smith N(2017)Evaluating intelligent knowledge systemsKnowledge and Information Systems10.1007/s10115-016-1011-352:2(379-409)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.1007/s10115-016-1011-3
Conrad JZeleznikow JSichelman TAtkinson K(2015)The role of evaluation in AI and lawProceedings of the 15th International Conference on Artificial Intelligence and Law10.1145/2746090.2746116(181-186)Online publication date: 8-Jun-2015
https://dl.acm.org/doi/10.1145/2746090.2746116
Ronkainen AFrancesconi EVerheij B(2013)Intelligent trademark analysisProceedings of the Fourteenth International Conference on Artificial Intelligence and Law10.1145/2514601.2514632(227-231)Online publication date: 10-Jun-2013
https://dl.acm.org/doi/10.1145/2514601.2514632
Conrad JZeleznikow JFrancesconi EVerheij B(2013)The significance of evaluation in AI and lawProceedings of the Fourteenth International Conference on Artificial Intelligence and Law10.1145/2514601.2514624(186-191)Online publication date: 10-Jun-2013
https://dl.acm.org/doi/10.1145/2514601.2514624
Bench-Capon TAraszkiewicz MAshley KAtkinson KBex FBorges FBourcier DBourgine PConrad JFrancesconi EGordon TGovernatori GLeidner JLewis DLoui RMccarty LPrakken HSchilder FSchweighofer EThompson PTyrrell AVerheij BWalton DWyner A(2012)A history of AI and Law in 50 papersArtificial Intelligence and Law10.1007/s10506-012-9131-x20:3(215-319)Online publication date: 1-Sep-2012
https://dl.acm.org/doi/10.1007/s10506-012-9131-x
Hall MHall RZeleznikow JZeleznikow JSartor G(2003)A process for evaluating legal knowledge-based systems based upon the context criteria contingency-guidelines frameworkProceedings of the 9th international conference on Artificial intelligence and law10.1145/1047788.1047843(274-283)Online publication date: 24-Jun-2003
https://dl.acm.org/doi/10.1145/1047788.1047843

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents