Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Validation and verification of knowledge-based systems: A survey

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Validation and verification of expert systems or knowledge-based systems is a critical issue in the development and deployment of robust systems. This article is a comprehensive survey of the developments and trends in this field. More than 300 references are included in the References and Additional Readings at the end of article.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • U. G. Gupta,Validating and Verifying Knowledge-Based Systems IEEE Computer Society Press: Los Alamitos, CA, 1991.

    Google Scholar 

  • A. Bundy, “How to improve the reliability of expert systems,” Research and Development in Expert System IV, inProc. Expert Syst. '87, Seventh Annu. Tech. Conf. Comput. Soc. Specialist Group on Expert Syst., Brighton, UK, December 16–17, 1989, pp. 3–17.

  • C. Culbert, G. Riley, and R. T. Savely, “Verification issues of rule-based expert systems,” inProc. Third Conf. Artif. Intell. Space Appl. National Aeronautics and Space Administration—Scientific and Technical Information Office, Washington, DC, November 2–3, 1987.

    Google Scholar 

  • C. J. Green and M. M. Keyes, “Verification and validation of expert systems,” inProceedings of the Western Conference on Expert Systems, (WESTEX '87) June 2–4, 1987, IEEE Computer Society Press: Los Alamitos, CA, pp. 38–43, 1987.

    Google Scholar 

  • M. Shaw and J. Woodward, “Validation in a knowledge support system. Construing and consistency with multiple experts,”Int. J. Man Machine Studies vol. 29. no. 3, pp. 329–350, September 1988.

    Google Scholar 

  • D. E. Brown and J. Pomykalski, “Reliability estimation during prototyping of knowledge-based systems,” Institute for Parallel Computation, School of Engineering and Applied Science, University of Virginia, Charlottesville, VA, January 11, 1991, pp. 1–23.

    Google Scholar 

  • Lance A. Miller, “A realistic industrial strength life cycle model for knowledge-based system development and testing,”3rd Annu. Workshop on Verification and Validation of Knowledge-Based Syst., Boston, MA, 1990, pp. 1–13.

  • W. R. Adrion, M. A. Branstad, and J. C. Cherniavsky, “Validation, verification, and testing of computer software,”Computing Syst. vol. 14, no. 2, pp. 159–192, June 1982.

    Google Scholar 

  • W. E. Howden, “Introduction to software validation,” inTutorial: Software Testing and Validation Techniques edited by E. Miller and W. E. Howden, IEEE Computer Society Press: New York, pp. 1–2, 1978.

    Google Scholar 

  • E. Hollnagel, “Issues in the reliability of expert systems,” inThe Reliability of Expert Systems edited by E. Hollnagel. Wiley: New York, pp. 303–329, 1989.

    Google Scholar 

  • A. Preece, R. Shinghal, and A. Batarekh, “Verifying expert systems: A logical framework and a practical tool,”Expert Syst. Appl. vol. 5, no. 2/3, pp. 421–436, 1992.

    Google Scholar 

  • A. D. Preece, “Towards a methodology for evaluating expert system,”Expert Syst. vol. 7, no. 4, pp. 215–223, 1990.

    Google Scholar 

  • R. M. O'Keefe, O. Balci, and E. P. Smith, “Validating expert system performance,”IEEE Expert vol. 2, no. 4, pp. 81–89, Winter 1987.

    Google Scholar 

  • D. A. Cleaves, “Cognitive biases and corrective techniques: Proposals for improving elicitation procedures for knowledge-based systems,”Int. J. Man-Machine Studies vol. 27, pp. 155–166, 1987.

    Google Scholar 

  • Daniel E. O'Leary and N. A. Kandelin, “Validating the weights in rule-based expert systems: A statistical approach,”Int. J. Expert Syst. vol. 1, no. 3, pp. 253–279, 1988.

    Google Scholar 

  • J. Rushby, “Quality measures and assurance for AI software,”NASA Contractor SRI International, Menlo Park, CA, NASI-17067, October 1988.

    Google Scholar 

  • C. Culbert, G. Riley, and R. T. Savely, “An expert system development methodology which supports verification and validation,”ISA Trans. vol. 28, no. 1, pp. 15–18, September 1987.

    Google Scholar 

  • P. R. Cohen and A. E. Howe, “Toward AI research methodology: Three case studies in evaluation,”IEEE Trans. Syst., Man, Cybernet. vol. 19, no. 3, pp. 634–646, May/June 1989.

    Google Scholar 

  • J. R. Geissman and R. D. Schultz, “Verification and validation of expert systems,”AI Expert vol. 3, no. 2, pp. 26–33, February 1988.

    Google Scholar 

  • J. R. Geissman, “Verification and validation for expert systems: A practical methodology,” inProceedings of the Fourth Annual Artificial Intelligence and Advanced Computer Tech. Conference May 4–6, 1988, National Aeronautics and Space Administration—Scientific and Technical Information Branch: Washington, DC, 1988, pp. 344–351.

    Google Scholar 

  • D. E. O'Leary, “Validation of expert systems—with applications to auditing and accounting expert systems,”Decision Sci. vol. 18, no. 3, pp. 468–486, 1987.

    Google Scholar 

  • J. A. Naser, “Nuclear power plant: Expert system verification and validation,” inAAAI Workshop on Validation and Verification of Expert Syst American Association for Artificial Intelligence, Menlo Park, CA, August, 1988 (unpublished).

    Google Scholar 

  • I. Benbasat and J. Dhaliwal, “A framework for the validation knowledge acquisition,”Int. J. Man-Machine Studies vol. 1, no. 2, pp. 215–233, June 1989.

    Google Scholar 

  • A. Vandierendonck, “Inferential simulation: Hypothesis-testing by computer simulation,”Nederlands Tijdschrift voor de Psychologie, Amsterdam, vol. 30, pp. 677–700, 1975.

  • J. Rushby, “Validation and testing of knowledge-based systems: How bad can it get?” inAAAI Workshop on Validation and Verification of Expert Systems, December 1988, pp. 1–9.

  • K. G. Heisler, W. T. Tsai, and C. V. Ramamoorthy, “Integrating the role of requirements specification into the process of prototyping: The protospec,” inProc. 22nd Annu. Hawaii Int. Conf. Syst. Sci., vol. 2, Software Track, IEEE, pp. 348–357, 1989.

  • E. A. Sykes and C. C. White, “Specifications of a knowledge system for packet-switched data network topological design,”Expert Syst. Gov. Symp., IEEE, 1985, pp. 102–110.

  • A. Batarekh, A. D. Preece, A. Bennett, and P. Grogono, “Specifying an expert system,”Expert Syst. Appl. vol. 2, no. 4, pp. 285–303, 1991.

    Google Scholar 

  • J. Haugh, “The application of formal specification techniques to knowledge-based system development,” University College, Information Engineering Directorate, London,UK IT 88 Conference Publication pp. 95–98, 1988.

    Google Scholar 

  • B. Marcot, “Testing your knowledge base,”AI Expert vol. 2, no. 8, pp. 42–47, August 1987.

    Google Scholar 

  • T. A. Nguyen, W. A. Perkins, T. J. Laffey, and D. Pecora, “Knowledge base verification,”AI Mag. vol. 8, no. 2, pp. 69–75, Summer 1987.

    Google Scholar 

  • A. D. Preece, “Practical issues in specifying expert systems,”Intell. Syst. Rev. vol. 2, no. 3/4, pp. 3–26, 1991.

    Google Scholar 

  • D. L. Nazareth, “Issues in the verification of knowledge in rule-based systems,”Int. J. Man-Machine Studies (UK), vol. 30, no. 3, pp. 255–271, March 1989.

    Google Scholar 

  • R. D. Schultz and J. R. Geissman, “Bridging the gap between static and dynamic verification,” inValidation and Testing Knowledge Based Syst. Workshop, Minneapolis, MN, August 1988.

  • B. J. Cragun and H. J. Steudel, “A decision-table-based processor for checking completeness and consistency in rule-based expert systems,”Int. J. Man-Machine Studies vol. 26, no. 5, pp. 633–648, May 1987.

    Google Scholar 

  • J. Bachant and J. McDermott, “RI revisited: Four years in the trenches,”AI Mag. vol. 5, no. 3, pp. 21–32, Fall 1984.

    Google Scholar 

  • F. Hayes-Roth, D. Lenat, and D. Waterman,Building Expert Systems Addison-Wesley: Reading, MA, 1983.

    Google Scholar 

  • N. Botten, A. Kusiak, and T. Raz, “Knowledge bases: Integration, verification, and partitioning,”Eur. J. Oper. Res. vol. 42, pp. 111–128, 1989.

    Google Scholar 

  • A. E. Radwan, M. Goul, T. O'Leary, and K. Moffitt, “A verification approach for knowledge-based systems,”Transportation Res. vol. 23A, no. 4, pp. 287–300, July 1989.

    Google Scholar 

  • L. Morell, “Use of metaknowledge in the verification of knowledge-based systems,” College of William and Mary, N89-25627/5, 1989.

  • James D. Kiper, “Structural testing of rule-based expert systems,”ACM Trans. Software Eng. Methodol. vol. 1, no. 2, pp. 168–187, April 1992.

    Google Scholar 

  • P. E. Lehner, “Toward an empirical approach to evaluating the knowledge base of an expert system,”IEEE Trans. Syst. Man Cybernet. vol. 19, no. 3, pp. 658–662, May/June 1989.

    Google Scholar 

  • R. Davis,Applications of Meta-Level Knowledge to the Construction, Maintenance and Use of Large Knowledge Bases Doctoral dissertation, Computer Science Department, Stanford University, Stanford, CA, 1976.

    Google Scholar 

  • M. Suwa, C. A. Scott, and E. H. Shortliffe, “An approach to verifying completeness and consistency in a rule-based expert system,”AI Mag. vol. 3, no. 4, pp. 16–21, Fall 1982.

    Google Scholar 

  • T. A. Nguyen, W. A. Perkins, T. J. Laffey, and D. Pecora, “Checking an expert system's knowledge base for consistency and completeness,” inProc. Ninth Int. Joint Conf. Artif. Intell. (IJCAI '85), vol. 14, International Joint Conferences on Artificial Intelligence, Menlo Park, CA, 1985, pp. 375–378.

  • D. C. Wilkins and B. G. Buchanan, “On debugging rule sets when reasoning under uncertainty,” inProc. Fifth Natl. Conf. Artif. Intell. (AAAI-86), vol. 1, AAAI Press, Menlo Park, CA, pp. 448–454, 1986.

    Google Scholar 

  • R. A. Stachowitz and J. B. Combs, “Validation of expert systems,” inProceedings of the Twentieth Annual Hawaii Conference on System Sciences University of Hawaii Press: Honolulu, pp. 686–695, 1987.

    Google Scholar 

  • A. Ginsberg, “Knowledge-base reduction: A new approach to checking bases for inconsistency & redundancy,” inProceedings of the Seventh National Conference on Artificial Intell. (AAAI-88), vol. 2, AAAI Press: Menlo Park, CA, 1988, pp. 585–589.

    Google Scholar 

  • Johan de Kleer, “An assumption-based truth maintenance system,”Artificial Intelligence vol. 28, pp. 127–162, 1986.

    Google Scholar 

  • A. D. Preece, R. D. Bell, and C. Y. Suen, “Verifying knowledge-based systems using the COVER tool,” inPersonal Computers and Intelligent Systems (Information Processing 92, Volume III), edited by F. H. Vogt, Elsevier: Amsterdam, pp. 231–237, 1992.

    Google Scholar 

  • A. T. Bahill, M. Jafar, and R. F. Moller, “Tools for extracting knowledge and validation of expert systems,” inIEEE International Conference on Systems, Man and Cybernetics, vol. 2, IEEE Service Center: Piscataway, NJ, pp. 857–862, 1989.

    Google Scholar 

  • Y. Kang and T. A. Bahill, “A tool for detecting expert system errors,”AI Expert vol. 5, no. 2, pp. 42–51, February 1990.

    Google Scholar 

  • A. D. Preece and R. Shinghal, “A practical approach to knowledge base verification,” inProceedings Applications of Artificial Intelligence IX (1991, edited by M. Trivedi, SPIE: Bellingham, WA, pp. 608–619, 1991.

    Google Scholar 

  • D. C. Bochsler and M. A. Goodwin, “Software engineering techniques used to develop an expert system for automating space vehicle rendezvous,” inROBEXS '86, Second Annu. Workshop Robot. Expert Syst., Research Triangle Park, NC, pp. 87–95, June 1986.

  • K. L. Bellman, “Testing and correction of rule-based expert systems,” inProceedings of the Space Quality Conference, NSIA/ATA (Space Division) and NASA April 1988, NSIA: Washington, DC, 1988.

    Google Scholar 

  • B. W. Boehm, “A spiral model of software development and enhancement,”IEEE Comput. vol. 21, no. 5, pp. 61–72, May 1988.

    Google Scholar 

  • A. D. Preece and L. Mosely, “Empirical study of expert system development,”Knowledge-Based Syst. vol. 5, no. 2, pp. 137–148, 1991.

    Google Scholar 

  • K. J. Macleish and D. A. Vennergrund, “An expert system development life cycle model and its relevance to traditional software systems,” inProceedings of the Fifth Annual International Phoenix Conference on Computers and Communications IEEE Computer Society Press: Los Alamitos, CA, pp. 592–596, 1986.

    Google Scholar 

  • M. Bull, R. Duda, D. Port, and J. Reiter, “Applying software engineering principles to knowledge-based development,”Proc. First Annu. Conf. Expert Syst. Bus. Learned Information, Inc.: Medford, NJ, 1987, pp. 27–37.

    Google Scholar 

  • C. Culbert, G. Riley, and R. T. Savely, “Approaches to the verification of rule-based expert systems,” inProc. First Annu. Workshop on Space Operations Automation and Robotics SOAR Conference, Houston, TX, August 1987, National Aeronautics and Space Administration—Scientific and Technical Information Office, Washington, DC.

  • R. P. Bonasso, P. W. Jordan, K. S. Keller, R. O. Nugent, R. W. Tucker, and D. Vogel, “A software storming approach to rapid prototyping,” inProc. 22nd Annu. Hawaii Int. Conf. Syst. Sciences, vol. 2, Software Track, IEEE, 1989, pp. 368–376.

  • P. Siegel,Expert Systems: A Non-Programmer's Guide to Development and Application TAB: Blue Ridge Summit, PA, 1986.

    Google Scholar 

  • J. Liebowitz, “Useful approaches for evaluating expert systems,”Expert Syst. vol. 3, no. 2, pp. 86–96, April 1986.

    Google Scholar 

  • D. L. Hall, D. Heinze, and T. Lhinas, “Test and evaluation of expert systems,” inAI in Manufacturing: Theory and Practice edited by S. Kumora, R. L. Kashyap, A. L. Soyster, Institute of Industrial Engineering: Norcross, GA, 1988.

    Google Scholar 

  • P. R. Harrison, “Testing and evaluation of knowledge-based systems,” inStructuring Expert System: Domain, Design and Development edited by J. Liebowitz and D. A. DeSalvo, Yourdon Press: New York, pp. 303–329, 1989.

    Google Scholar 

  • Y. Ching Suen, P. D. Grogono, and R. Shinghal, “Verifying, validating, and measuring the performance of expert systems,”Expert Syst. Appl. vol. 1, no. 2, pp. 93–102, 1990.

    Google Scholar 

  • B. Chandrasekaran, “On evaluating AI systems for medical diagnosis,”AI Mag. vol. 4, no. 2, pp. 34–37, Summer 1983.

    Google Scholar 

  • N. E. Lane, “Global issues in evaluation of expert systems,”Proc. Int. Conf. Syst., Man Cybnernet. IEEE, Atlanta, GA, pp. 121–125, October 1986.

    Google Scholar 

  • G. A. Klein and J. A. King, “A test for the performance of knowledge-based systems: AIQ,” inProc. AAAI Workshop on Validation and Verification of Expert Syst. American Association for Artificial Intelligence, Menlo Park, CA, August 1988 (unpublished).

    Google Scholar 

  • D. W. Loveland and M. Valtorta, “Detecting ambiguity: An example in knowledge evaluation,” inProceedings Eighth International Joint Conference Artificial Intelligence ISCAI 1983, vol. 1, Morgan Kaufmann: San Mateo, CA, pp. 182–184, 1983.

    Google Scholar 

  • C. P. Langlotz, E. H. Shortliffe, and S. M. Fagan, “Using decision theory to justify heuristics,” inProceedings of the Fifth National Conference on Artificial Intelligence (AAAI '86), vol. 1, American Association for Artificial Intelligence: Menlo Park, CA, pp. 215–219, 1986.

    Google Scholar 

  • J. Gaschnig, “Preliminary performance analysis of the prospector consultant system for mineral exploration,” inProceedings Sixth International Joint Conference Artificial Intelligence, vol. 1, Morgan Kaufmann: San Mateo, CA, pp. 308–310, 1979.

    Google Scholar 

  • E. Soloway, J. Bachant, and K. Jensen, “Assessing the maintainability of XCON-IN-RIME: Coping with the problems of a very large rule-base,” inProc. Sixth Nat. Conf. Artif. Intell., vol. 2, Seattle, WA, July 1987, pp. 824–829.

  • T. Cochran and B. Hutchins, “Testing, verifying and releasing an expert system: The case history of Mentor,” inProc. Conf. Artif. Intell. Appl., 1987, pp. 163–167.

  • M. Valtorta, “Knowledge base refinement: A bibliography,”Appl. Intell. vol. 1, no. 1, pp. 87–94, July 1991.

    Google Scholar 

  • E. T. Scambos, “A scenario-based test tool for examining expert systems,” inProc. Int. Conf. Syst. Man Cybernet. IEEE, Atlanta, GA, October 1986, pp. 131–135.

    Google Scholar 

  • T. W. Finin, “Interactive classification: A technique for acquiring and maintaining knowledge bases,”Proc. IEEE vol. 74, no. 10, pp. 1414–1421, 1986.

    Google Scholar 

  • C. Krishnamurthy, S. Padalkar, J. Sztipznovits, and R. S. Purves, “Methodology for testing and validating knowledge bases,” inProceedings of the Third Conference on Artificial Intelligence for Space Applications, November 2–3, 1987, National Aeronautics and Space Administration—Scientific and Technical Branch: Washington, DC, pp. 21–32.

  • S. Marcus, “SALT: A knowledge acquisition tool that checks and helps test a knowledge base,” inProc. AAAI Workshop on Validation and Verification of Expert Syst. American Association for Artificial Intelligence, Menlo Park, CA, 1988 (unpublished).

    Google Scholar 

  • W. R. Franklin, R. Bansal, E. Gilbert, and G. Shroff, “Debugging and tracing expert systems,” inProceedings of the Twenty-First Annual Hawaii International Conference on System Sciences (HICSS), vol. 3, IEEE Computer Society Press: Los Alamitos, CA, pp. 159–167, 1988.

    Google Scholar 

  • D. L. Hall and D. T. Heinze, “The use of simulation techniques for expert system test and evaluation,”ISA Trans. vol. 28, no. 1, pp. 19–22, 1989.

    Google Scholar 

  • U. G. Gupta and J. E. Biegel, “RITCaG: Rule-Based Intelligent Test Case Generator,” inProc. AAAI Workshop on Validation and Verification of Expert Syst., Boston, MA, July 29, 1990 (unpublished).

Additional Readings

  • Adlassnig, K. P., and Scheithauer, W., “Performance evaluation of medical expert systems using ROC curves,”Comput. Biomed. Res. vol. 22, pp. 297–313, 1989.

    Google Scholar 

  • AICPA, “Statement of Position 82-1, Accounting and financial reporting for personal finance systems,” American Institute of Certified Public Accountants, New York, NY, 1982.

    Google Scholar 

  • Ayel, M., and Lourent, J. P., “Coherence testing of knowledge bases,” inProc. SPIE—Int. Soc. Optimization Eng. (USA), vol. 1095, Orlando, FL, March 28–30, 1989, pp. 996–1005.

  • Bachant, J., “Validating and testing XCON,” inAAAI Workshop on Validation and Verification of Expert Systems American Association on Artificial Intelligence, Menlo Park, CA, August 1988 (unpublished).

    Google Scholar 

  • Bahill, A. T.,Verifying and Validating Personal Computer-based Expert Systems Prentice-Hall: Englewood Cliffs, NJ, 1991.

    Google Scholar 

  • Balci, O., and Sargent, R., “A methodology for cost risk analysis in the statistical validation of simulation models,”Commun. ACM vol. 24, no. 4, pp. 190–197, 1981.

    Google Scholar 

  • Balci, O., and Sargent, R., “Validation of simulation models via simultaneous confidence intervals,”Am. J. Math. Manage. Sci. vol. 4, nos. 3 and 4, pp. 375–406, 1984.

    Google Scholar 

  • Bansal, R.,Debugging, Testing and Maintenance of Expert Systems, Master's thesis, Computer Science Department, Rensselaer Polytechnic Institute, 1987.

  • Barber, T. J., J. T. Boardman, and N. Brown, “Practical evaluation of an intelligent knowledge-based project control system,”IEEE Proc. vol. 137, pt. A., no. 1, pp. 35–51, January 1990.

    Google Scholar 

  • Barstow, D., “Artificial intelligence and software engineering,” inProceedings of the Ninth International Conference on Software Engineering IEEE Computer Society Press: Los Alamitos, CA, pp. 200–211, 1987.

    Google Scholar 

  • Barton, A., “Experiences in expert systems,”J. Operational Res. Soc. vol. 38, no. 10, pp. 965–966, October 1987.

    Google Scholar 

  • Beauvieux, A., “A method to check knowledge base consistency,”Foundations of Software Technology and Theoretical Computer Science Poona, India, December 21–23, 1988, Springer-Verlag: New York, pp. 455–468, 1988.

    Google Scholar 

  • Becker, L. A., Green, P. G., and Bhatnagar, H.,Evidence Flow Graph Methods for Validation and Verification of Expert Systems, Worchester Polytechnic Institute, N8923328/7, p. 46, 1989.

  • Behan, J., and Lecot, K., “Overview of financial applications of expert systems,” inProceedings of the Western Conference on Expert Systems (WESTEX-87) IEEE Computer Society Press: Washington, DC, pp. 223–229, 1987.

    Google Scholar 

  • Bell, M. Z., “Why expert systems fail,”J. Operational Res. Soc. vol. 36, no. 7, pp. 613–619, July 1985.

    Google Scholar 

  • Belli, F.; Camara, I.; and Schmidt, A., “A built-in test language for PROLOG to validate knowledge-based systems,” inProc. Third Int. Conf. Ind. Eng. Appl. Artif. Intell. Expert Syst., vol. 2, ACM Press: New York, NY, pp. 726–734, July 1990.

    Google Scholar 

  • Bellman, J. L., and Walter, D. O., “Analyzing and correcting knowledge-based systems requires explicit models,” inAAAI Workshop on Validation and Verification of Expert Syst. American Association on Artificial Intelligence, Menlo Park, CA, 1988 (unpublished).

    Google Scholar 

  • Berry, D. C., and Hart, A. E., “Evaluating expert systems,”Expert Syst. vol. 7, no. 4, pp. 199–208, 1990.

    Google Scholar 

  • Bezem, M., “Consistency of rule based systems,”CWI Report CS-R8736, Centrum voor Wiskunde en Informatica.

  • Biggs, S.; Messier, W.; and Hansen, J., “A descriptive analysis of computer audit specialists' decision making,”Auditing vol. 6, no. 2, pp. 1–21, 1987.

    Google Scholar 

  • Bird, S., and Buck, N., “New science's 1990 survey of AI in the insurance industry,”Artif. Intell. Res. Highlights, pp. 1443–1459, 1990.

  • Blanning, R. W., “Knowledge acquisition and system validation in expert systems for management,”Hum. Syst. Manage. vol. 4, pp. 280–285, 1984.

    Google Scholar 

  • Bobrow, D. G., “If Prolog is the answer, what is the question? Or what it takes to support AI programming paradigms,”IEEE Trans. Software Eng. vol. SE-11, no. 11, pp. 1401–1408, November 1985.

    Google Scholar 

  • Bobrow, D. G.; Mittal, S.; and Stefik, M. J., “Expert systems: perils and promises,”Commun. ACM vol. 29, no. 9, pp. 880–894, September 1986.

    Google Scholar 

  • Boehm, B. W., “Verifying and validating software requirements and design specifications,”IEEE Software vol. 1, no. 1, pp. 75–88, January 1984.

    Google Scholar 

  • Bologna, S.; Ness, E.; and Sivertsen, T., “Dependable knowledge-based systems development and verification: What we can learn from software engineering and what we need,”Proceedings 2nd International Conference on Tools for Artificial Intelligence (TAI-90) edited by A. Dollas, W. T. Tsai, and N. G. Bourbakis, IEEE Press: Los Alamitos, pp. 86–95, 1990.

    Google Scholar 

  • Bond, D.; Lane, D.; and Shikli, P., “Knowledge management in expert systems,” inExpert Systems and Intelligent Manufacturing edited by Michael D. Oliff, Elsevier Science Publishers: New York, 1988.

    Google Scholar 

  • Brandi, J. T., “Artificial intelligence, expert systems, and financial planning,”J. Am. Soc. CLU ChFC XLII, vol. 17, no. 2, pp. 64–74, 1988.

    Google Scholar 

  • Brown, C. E., “Expert systems in public accounting: Current practice and future directions,”Expert Syst. Appl. vol. 3, no. 1, pp. 3–18, 1991.

    Google Scholar 

  • Brown, C. E., “Tax expert systems in industry and accounting,”Expert Syst. Rev. Bus. Accounting vol. 1, no. 3, pp. 137–143, 1988.

    Google Scholar 

  • Brown, C. E.; Nielson, N. L.; and Phillips, M. E., “Expert systems for personal finance planning,”J. Fin. Planning vol. 3, no. 3, pp. 137–143, July 1990.

    Google Scholar 

  • Brown, C. E.; Nielson, N. L.; and Phillips, M. E., “A process for validating expert systems using hypothetical cases,” working paper, Oregon State University, Corvallis, OR, 1992.

    Google Scholar 

  • Buchanan, B., and Shortliffe, E.,Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Project Addison-Wesley: Reading, MA, 1985.

    Google Scholar 

  • Burton, A. M.; Shadbolt, N. R.; Hedgecock, A. P.; and Rugg, G., “A formal evaluation of knowledge elicitation techniques for expert systems,” inProc. First Eur. Workshop on Knowledge Acquisition for Knowledge-Based Syst., Reading, UK, 1987.

  • Camacho, R., “The liability exposure of financial planners and those sponsoring their services,” working paper, Oregon State University, Corvallis, OR, 1991.

    Google Scholar 

  • Castore, G., “Validation and verification for knowledge based control systems,”Soar Conf. NASA/JSC, Houston, TX, National Aeronautics and Space Administration—Scientific and Technical Information Office, Washington, DC, 1987.

    Google Scholar 

  • Chandrasekaran, B., “Generic tasks in knowledge-based reasoning: High-level building blocks for expert system design,”IEEE Expert vol. 1, no. 3, pp. 23–30, Fall 1986.

    Google Scholar 

  • Chang, C. L.; Combs, J. B.; and Stachowitz, R. A., “A Report on the Expert Systems Validation Associate (EVA),”Expert Syst. Appl. (US), vol. 1, no. 3, pp. 217–230, 1990.

    Google Scholar 

  • Chang, C. L.; Stachowitz, R. A.; and Combs, J. B., “Validation of nonmonotonic knowledge-based systems,” inProc. 2nd Int. Conf. Tools Artif. Intell. (TAI-90) IEEE, Washington, DC, 1990, pp. 776–782.

    Google Scholar 

  • Chang, C. L., and Stachowitz, R. A., “Testing expert systems,” inProc. SOAR-88 Workshop, Space Operations Automation and Robotics Workshop, Dayton, Ohio, National Aeronautics and Space Administration—Scientific and Technical Information Division, Washington, DC, pp. 1–5, 1988.

    Google Scholar 

  • Citrenbaum, R. L., and Geissnam, J. R., “A practical cost-conscious expert system development methodology,” inProc. AI-86, Long Beach, CA, April 1986.

  • Coats, P. K., “Why expert systems fail,”Fin. Manage. vol. 17, no. 3, pp. 77–86, Autumn 1988.

    Google Scholar 

  • Cohen, P. R., and Howe, A. E., “The invisible hand: How evaluation guides AI research,”AAAI Workshop on Validation and Verification of Expert Systems American Association on Artificial Intelligence: Menlo Park, CA, 1988 (unpublished).

    Google Scholar 

  • Courant, M., and Guevel, G., “Validation d'une base de connaissances hubride avec objets et regles de production,” Institut National de Recherche en Informatique et en Automatique, Le Chesnay, France, 1986.

    Google Scholar 

  • Culbert, C., “Special issue: Verification and validation of knowledge-based systems,”Expert Syst. Appl., vol. 1, no. 3, 1990.

  • Davis, L., and Liebowitz, J., “Testing and evaluation of expert system prototype: A case study,”Inf. Age vol. 12, no. 2, pp. 75–82, April 1990.

    Google Scholar 

  • Davis, R., “Interactive transfer of expertise: Acquisition of new inference rules,”Artif. Intell. vol. 12, pp. 121–157, 1979.

    Google Scholar 

  • Davis, R., “Reasoning from first principles in electronic trouble-shooting,”Int. J. Man-Machine Studies vol. 24, pp. 347–410, 1984.

    Google Scholar 

  • Doyle, J., “Expert systems and the ‘myth’ of symbolic reasoning,”IEEE Trans. Software Eng. vol. SE-11, no. 11, pp. 1386–1390, November 1985.

    Google Scholar 

  • Duke, E. L., “V&V of flight and mission-critical software,”IEEE Software vol. 6, no. 3, pp. 39–45, May 1989.

    Google Scholar 

  • Enand, R.; Kahn, G.; and Mills, R., “A methodology for validating large knowledge bases,”Int. J. Man-Machine Studies vol. 33, pp. 361–371, October 1990.

    Google Scholar 

  • Eschelman, L.; Ehret, D.; McDermott, J.; and Tan, M., “MOLE: A tenacious knowledge acquisition tool,”Int. J. Man-Machine Studies vol. 26, pp. 41–54, 1987.

    Google Scholar 

  • Everstz, R., “The automated analysis of rule-based systems, based on their procedural semantics,” inProc. IJCAI-91, 1991, pp. 22–27.

  • Ferguson, T. C., and Wagner, R. E., “Beyond rules: The next generation of expert systems,” inProc. First Annu. Workshop Space Operations Automation Robot. SOAR '87 Houston, TX, National Aeronautics and Space Administration, Scientific and Technical Information Office, Washington, DC, pp. 363–369, August 1987.

    Google Scholar 

  • Findlay, P.; Forsey, G.; and Wilson, J., “The validation of expert systems—contrasts with traditional methods,”J. Operational Res. vol. 39, no. 10, pp. 933–938, 1988.

    Google Scholar 

  • Forsyth, G.; Larkin, M.; and Wallace, G., “Verification of heuristic diagnostic knowledge by comparison with a causal/qualitative model,” inProc. Third Int. Conf. Ind. Eng. Appl. Artif. Intell. Expert Syst. vol. 2, pp. 751–756, July 1990.

    Google Scholar 

  • Fox, Mark S., “AI and expert system: Myths, legends, and facts,”IEEE Expert vol. 5, no. 1, pp. 8–20, February 1990.

    Google Scholar 

  • Franklin, W. R.; Bansal, R.; and Gilbert E., “Sensitivity analysis of expert systems,” inProc. AAAI Workshop on Validation and Verification of Expert Syst. American Association for Artificial Intelligence, Menlo Park, CA, 1988 (unpublished).

    Google Scholar 

  • Froscher, J. N., and Jacob, J. F. K., “Designing expert systems for ease of change,” inProceedings of the Expert Systems in Government Symposium IEEE Computer Society Press: Los Alamitos, CA, pp. 246–251, 1985.

    Google Scholar 

  • Fu, L., and Buchanan, B. G., “Enhancing performance of expert systems by automated discovery of meta-rules,” inProceedings of the First Conference on Artificial Intelligence Applications IEEE Computer Society Press: Los Alamitos, CA, pp. 107–115, 1984.

    Google Scholar 

  • Gaschnig, J., et al., “Evaluation of expert systems: Issues for case studies,” inBuilding Expert Systems edited by F. Hayes-Roth, D. Waterman, and D. B. Lenart, Addison-Wesley: Reading, MA, 1983.

    Google Scholar 

  • Gearhart, L. M., “Managing the development and deployment of expert systems,” inProceedings of IEEE National Aerospace and Electronics Conference, (NAECON 89), Vol. 3, Dayton, Ohio, May 22–26, 1989, IEEE Service Center: Piscataway, NJ, pp. 1004–1010, 1989.

    Google Scholar 

  • Gilbert, E.,Software Tools for the Maintenance of Expert Systems Master's thesis, Computer Science Department, Rensselaer Polytechnic Institute, Troy, NY, November 1987.

    Google Scholar 

  • Gilstrap, L., “Validation and verification of expert systems,”Telemation Inf. vol. 8, no. 4, pp. 439–448, 1991.

    Google Scholar 

  • Ginsberg, A., “A metalinguistic approach to the construction of knowledge base refinement systems,” inProc. 5th Nat. Conf. Artif. Intell., American Association for Artificial Intelligence, 1986, pp. 436–441.

  • Goldberg, A.; Weiss, S. M.; and Politakis, P., “Automatic knowledge base refinement for classification systems,”Artif. Intell. vol. 35, no. 2, pp. 197–226, June 1988.

    Google Scholar 

  • Goodwin, M. A., and Robertson, C. C., “A systematic view of validating and testing knowledge-based systems,” inProc. AAAI Workshop on Validation and Verification of Expert Syst. American Association for Artificial Intelligence, Menlo Park, CA, 1988 (unpublished).

    Google Scholar 

  • Goodwin, M. A., and Robertson, C. C., “Expert system verification concerns in an operations environment,” inProc. First Annu. Conf. Space Operations Automation and Robotics, 1987.

  • Green, C. J., “On the use of requirements in development of knowledge-based systems,” inProc. AAAI Workshop on Validation and Verification of Expert Syst. American Association for Artificial Intelligence, Menlo Park, CA, April 1987 (unpublished).

    Google Scholar 

  • Grogono, P.; Batarekh, A.; Preece, A.; Shinghal, R.; and Suen, C. Y., “Expert systems evaluation techniques: A selected biography,”Expert Systems vol. 8, no. 4, pp. 227–239, 1991.

    Google Scholar 

  • Grogono, P.; Preece, A.; Shinghal, R.; and Suen, C., “Evaluation of expert systems in telecommunications,”Expert Syst. Appl. vol. 5, no. 2/3, pp. 395–401, 1992.

    Google Scholar 

  • Groundwater, E. H., “Verification and validation plan for the Water Chemistry Expert Monitoring System (WCEMS),” Science Applications International Corporation, prepared for Electric Power Research Institute (J. Naser, Project Manager), 1989.

  • Groundwater, E. H.; Donnell, M. L.; and Archer, M. A., “Approaches to the verification and validation of expert systems for nuclear power plants,” Electric Power Research Institute Report, No. NP-5236, prepared by Science Applications International Corporation, Final Report, July 1987.

  • Gupta, U. G.,A Framework for Testing and Debugging Knowledge-Based Systems, Handbook of Systems Management: Development and Support, Auerabach Publishers, New York, NY, June 1993.

    Google Scholar 

  • Gupta, U. G., “Successful deployment of expert systems,”J. Inf. Syst. Manage. vol. 9, no. 1, pp. 21–27, Winter 1991.

    Google Scholar 

  • Gupta, U. G.Using Statistical Techniques to Validate Expert Systems, Decision Sciences Institute Conference, Miami, FL, Nov. 24–26, 1991.

  • Gupta, U. G., “Automatic test case generation for rule-based systems,”World Congress on Expert Systems, Orlando, FL, December 16–19, 1991.

  • Hamilton, D.; Kelley, K.; and Culbert, C., “State-of-the-practice knowledge-based system verification and validation,”Expert Syst. Appl. vol. 3, no. 4, pp. 403–410, 1991.

    Google Scholar 

  • Hansen, J., and Messier, W., “A preliminary investigation of EDP-XPERT,”Auditing vol. 6, no. 1, pp. 109–123, 1986.

    Google Scholar 

  • Hartley, Roger T., “How expert should an expert system be?”, inProceedings 7th International Joint Conference on Artificial Intelligence, vol. 2, Morgan Kaufman: San Mateo, CA, pp. 862–867, 1981.

    Google Scholar 

  • Hershauer, J.; Karim, A.; Ownes, H.; and Philippakis, A., “A field observation study of an expert system prototype development,”Inf. Manage. vol. 17, no. 2, pp. 107–116, September 1989.

    Google Scholar 

  • Hickam, D.; Shortliffe, E.; Bischoff, M.; Scoll, A.; and Jacobs, C., “The treatment advice of a computer-based cancer chemotherapy protocol advisor,”Ann. Intern. Med. vol. 103, pp. 928–936, 1985.

    Google Scholar 

  • Hollnagel, E., “Evaluation of expert systems,” inTopics in Expert Systems edited by G. Guida and C. Tasso, Elsevier/North Holland: Amsterdam, pp. 377–416, 1989.

    Google Scholar 

  • Hughes, C. E., “Issues in the reliability of expert systems,” inFundamental of Financial Planning edited by Robert M. Crowe, The American College: Bryn Mawr, PA, pp. 210–236, 1990.

    Google Scholar 

  • Humbert, B., and Holley, P., “Expert systems in finance planning,”Expert Syst. vol. 5, no. 2, pp. 78–100, 1988.

    Google Scholar 

  • Jacob, R. J. K., and Froscher, J. N., “Developing a software engineering methodology for knowledge-based systems,” Information Technology Division, Naval Research Laboratory, Washington, DC,Report Number 9019, 1988.

    Google Scholar 

  • Jacob, R. J. K., and Froscher, J. N., “Software engineering for rule-based systems,” inProceedings Fall Joint Computer Conference, Dallas, TX, Nov. 2–6, 1986, vol. 26, IEEE Computer Society Press: Washington, DC, pp. 185–189, 1986.

    Google Scholar 

  • Jacob, R. J. K., and Froscher, J. N., “A software engineering methodology for rule-based systems,”IEEE Trans. Knowledge Data Eng. (US) vol. 2, pp. 173–189, 1990.

    Google Scholar 

  • Jansen, B., “Applying software engineering concepts to rule-based expert systems,” Commonwealth Scientific & Industrial Research Organization. A Division of Info Techno, Australia, ISSN 0819-0011,Info Techno Technical Report TR-FC-87-02, June 1987.

  • Jansen, B., and Campton, F., “The knowledge dictionary: A relational tool for the maintenance of expert systems,” Commonwealth Scientific & Industrial Research Organization, A Division of Info Technology, Australia, ISSN 0819-0011,Info Techno Technical Report TR-FC-87-02, 1988.

  • Johanser, J. T., and Harbidge, R. M., “Validating of expert systems: Problems and solutions in practice,” inProceedings of the International Conference on Knowledge-Based Systems, London, England, Online Publications: Pinner, UK, pp. 215–229, 1986.

    Google Scholar 

  • Johnson, R.; Joly, G.; and King, P., “Validation and verification in expert system rule sets,”IEE Colloquium: Testing Expert Systems, Digest No. 110, IEE, pp. 3/1–3/3, 1987.

  • Kahn, G.; Nowlan, S.; and McDermott, J., “MORE: An Intelligent Knowledge Acquisition Tool,” inProceedings of the Ninth National Joint Conference on Artificial Intelligence, Los Angeles, CA, August 1985, Morgan Kaufman: Los Altos, CA, 1985.

    Google Scholar 

  • Kaiser, S. H., “Expert system metrics,” inProc. 1986 IEEE Int. Conf. Syst. Man Cybernet., Atlanta, GA, October 1986.

  • Kearsley, G., “Validation of an expert system: The CBT analyst,”J. Computer-Based Instruction vol. 15, no. 2, pp. 61–64, 1988.

    Google Scholar 

  • Kindle, K. W.; Cann, R. S.; Craig, M. R.; and Martin, T. J., “PFPS—Personal Financial Planning System,” presented at AAAI Innovative Intelligence Conference, March, 1989.

  • Kirk, D. B., and Murray, A. E., “Verification and validation systems for nuclear power plant applications,” Science Applications International Corporation, Dallas, TX,Report Number EPRI-NP 5978, August 1988.

    Google Scholar 

  • Kirk, D. B., and Murray, A. E., “Approaches to the verification and validation of expert systems for nuclear power plants,” Science Applications International Corporation, Dallas, TX,Report Number EPRI-NP 5236, August 1988.

    Google Scholar 

  • Lalo, A., “TIBRE: An expert system for testing inconsistencies in rule bases,” inEight Int. Workshop on Expert Syst. Appl., EC2, Nanterre, France, vol. 5, pp. 63–84, 30 May–3 June 1988 (French).

  • Landauer, C., “Correctness Principles for Rule-Based Expert Systems,”Expert Syst. Appl. (US) vol. 1, no. 3, pp. 291–316, 1990.

    Google Scholar 

  • Laurent, J. P., and Ayel, M., “Off-line coherence checking for knowledge based systems,”IJCAI-89 Workshop on Verification, Validation and Testing of Knowledge Based Systems, IJCAI, 1989.

  • Lenat, D.; Prakash, M.; and Shepherd, M., “CYC using common sense knowledge to overcome brittleness and knowledge acquisition bottlenecks,”AI Mag. vol. 6, no. 4, pp. 65–85, Winter 1986.

    Google Scholar 

  • Lethan, H., and Jacobsen, H., “ESKORT—An expert system for auditing VAT accounts,” inProc. Expert Syst. Appl., Avignon, France, 1987, pp. 131–146.

  • Levesque, H., “The logic of incomplete knowledge bases,” inConceptual Modelling edited by M. L. Brodie, J. Mypoilos, and J. W. Schmidt, Springer-Verlag: New York, pp. 165–189, 1984.

    Google Scholar 

  • Levi, K., “Expert systems should be more accurate than human experts: Evaluation procedures from human judgement and decision-making,”IEEE Trans. Syst. Man and Cybernet. vol. 19, no. 3, pp. 647–657, May/June 1989.

    Google Scholar 

  • Linden, T. A., “Alternative approaches to V&V for AI systems,” inProc. AAAI Workshop on Validation and Verification of Expert Syst. American Association for Artificial Intelligence, Menlo Park, CA, August 1988 (unpublished).

    Google Scholar 

  • Lopez, B.; Pedro, M.; and Enric, P., “Knowledge-based systems validation: a state of the art,”AI Commun. vol. 3, no. 2, June 1990, pp. 58–72.

    Google Scholar 

  • Ludvigsen, P. J., and Dupont, R., “Formal evaluation of the expert system DEMOTOX,”J. Comput. Civil Eng. vol. 2, no. 4, pp. 398–413, October 1988.

    Google Scholar 

  • March, Salvatore T., “Editorial policy,”ACM Comput. Surveys vol. 23, no. 2, June 1991, pp. 133–141.

    Google Scholar 

  • Marek, W., “Completeness and consistency in knowledge base systems,” inProceedings of 1st International Conference on Expert Database Systems, 1986, Benjamin/Cummings: Menlo Park, CA, 1987.

    Google Scholar 

  • Mars, N., and Miller, P., “Knowledge acquisition and verification tools for medical expert systems,”Med. Decision Making vol. 7, no. 1, pp. 23–48, 1987.

    Google Scholar 

  • Martinez, J.; Muro, P.; and Silva, M., “Modeling, validation, and software implementation of production systems using high level Petri nets,”IEEE International Conference on Robotics and Automation, vol. 2, IEEE Press: Los Alamitos, CA, pp. 1180–1185, 1987.

    Google Scholar 

  • Martinez, D. R., and Sobol, M. G., “Systems analysis techniques for the implementation of expert systems,”Inf. Software Technol., vol. 30, no. 2, 1988.

    Google Scholar 

  • Massey, J. G., “Expert system verification and validation—Implementing the concepts,”in Proceedings: Applications of Artificial Intelligence in Engineering, vol. 6, July 2–4, 1991, Elsevier Applied Science: London, UK, pp. 873–883, 1991.

    Google Scholar 

  • McAlindon, P. J., and Gupta, U. G., “A structured approach to verifying and validating expert systems,” edited by J. A. Wise, V. D. Hopkin, and P. Stager,Verification and Validation of Complex and Integrated Human-Machine Systems Springer-Verlag: Berlin, 1992 (in press).

    Google Scholar 

  • McGuire, J. G., “Uncovering redundancy and rule-inconsistency in knowledge bases via deduction,” inProc. 5th Annu. Conf. Comput. Assurance: Systems Integrity, Software Safety, and Process Safety, IEEE COMPASS-90, 1990.

  • McKell, L. J., and Jenkins, J. W., “PLANMAN structure—A strategy for compromise,”IEEE Trans. Syst. Man Cybernet. vol. 19, no. 3, pp. 648–662, 1988.

    Google Scholar 

  • McKinnion, J. M., “MODVEX: A model development and validation expert system,” inProc. 1986 Winter Meetings of Am. Soc. Agricultural Eng. The American Society of Agricultural Engineers, Chicago, IL, December 1986, paper number 86–4514.

    Google Scholar 

  • Medoff, S. M.; Register, M. S.; and Swartout, M. W., “A framework for design verification and evaluation systems,”Artif. Intell. Eng. Design Anal. Manuf. Managers vol. 3, no. 2, pp. 71–84, 1989.

    Google Scholar 

  • Meseguer, P., “Verification and validation of multi-level rule-based expert systems,” inProc. AAAI-91,

  • Meseguer, P., “Inconsistency checking in rule-based expert systems with uncertainty and control features,” inProc. Fifth Int. Symp. Methodol. Intell. Syst., ISMIS-90.

  • Meseguer, P., “A new method to checking rule bases for inconsistency: A Petri net approach,” inProc. ECAI-90.

  • Meseguer, P., and Plaza, E., “Three dimensions for rule-based systems validation,”Personal Computers and Intelligent Systems edited by E. H. Vogt, Information Processing 92, vol. 3, Elsevier Science Publishers, B. V. (North-Holland), IFIP: Amsterdam, 1992.

    Google Scholar 

  • Meservy, R.; Bailey, A.; and Johnson, P., “Internal control evaluation: A computational model of the review process,”Auditing vol. 6, no. 1, pp. 44–74, 1986.

    Google Scholar 

  • Morell, L. J., “Use of meta-knowledge in the verification of knowledge-based systems,” inProc. IEA-AIE IEEE, Tullahama, TN, June 1–3, 1988, pp. 847–857.

    Google Scholar 

  • Michalski, R. S., and Chilausky, R. L., “Knowledge acquisition by encoding expert rules versus computer induction from examples: A case study involving soybean pathology,” inFuzzy Reasoning and its Applications edited by E. H. Mamdani and B. R. Gaines, Academic Press: London, 1981.

    Google Scholar 

  • Miller, E. F., “Expert systems validation: Issues and approaches,”Expert Systems and their Applications, May 1987, pp. 233–236, Avignon, France, Nanterre, France.

  • Miller, L. A., “A comprehensive approach to the verification and validation of knowledge based systems,” Workshop on Verification, Validation, and Testing of Knowledge Based Systems,Int. Joint Conf. Artif. Intell., Detroit, MI, August, 1989.

  • Miller, Lance A., “Dynamic testing of knowledge bases using the heuristic testing approach,”Expert Syst. Appl. vol. 1, no. 3, pp. 249–269, 1990.

    Google Scholar 

  • Miller, Lance A., “Mapping a KBS life cycle onto 2167A,” in7th IEEE Conf. AI Appl., Miami Beach, FL, Feb. 25, 1991.

  • Miller, L. A., “New challenges for the validation and verification of knowledge based systems,” inAAAI Issues in Validation of Knowledge Based Systems, Reno, NV, January, 1989.

  • Miller, L. A., “Tutorial on validation and verification of knowledge based systems,” paper presented at the Conference on Expert Systems Applications for the Electric Power Industry, Orlando, FL, June 1989.

  • Mingers, John, “Expert systems-rule induction with statistical data,”J. Operational Res. Soc. vol. 38, no. 1, pp. 39–47, January 1987.

    Google Scholar 

  • Moninger, R.; Stewart, T.; and McIntosh, P., “Validation of knowledge-based systems for probabalistic reasoning,” inProc. 1988 AAAI Workshop on Validation and Testing of Knowledge-Based Syst. AAAI, Palo Alto, CA, 1988.

    Google Scholar 

  • Murray, K., and Porter, B., “Controlling search for the consequences of new information during knowledge integration,” inProc. Sixth Int. Workshop on Machine Learning, 1989, pp. 290–295.

  • Neches, R.; Swartout, W. R.; and Moore, J. D., “Enhanced maintenance and explanation of expert systems through explicit models and their development,”IEEE Trans. Software Eng. vol. SE-11, no. 11, pp. 1337–1351, November 1985.

    Google Scholar 

  • Neches, R.; Swartout, W. R.; and Moore, J. D., “Explainable (and maintainable) expert systems,”Proc. Ninth Int. Joint Conf. Artif. Ingell. (IJCAI '85), International Joint Conferences on Artificial Intelligence, Vol. 1. Los Angeles, CA, 1985, pp. 382–389.

  • Nguyen, T. A., “Verifiying consistency of production systems,” inProc. Third Conf. Artif. Intell. Appl. IEEE Computer Society Press: Orlando, FL, 1987, pp. 4–8.

    Google Scholar 

  • Nielson, N. L.; Phillips, M. E.; and Brown, C. E., “Insurance assumptions and recommendations in expert-systems-prepared personal financial plans,” working paper, Oregon State University, Corvallis, OR, 1992.

    Google Scholar 

  • Nielson, N. L.; Phillips, M. E.; and Brown, C. E., “Expert systems to provide financial planning benefits,”Benefits Q. vol. 7, no. 1, pp. 41–51, 1991.

    Google Scholar 

  • Oakley, B., “The problems of validating expert systems,”IEEE Colloq. Expert Syst. Liability, London UK, Vol. 23, pp. 2–4, October 1989.

  • O'Keefe, R. M., “A systems view of validating expert knowledge systems,”AAAI Workshop on Validation and Verification of Expert Syst. American Association for Artificial Intelligence, Menlo Park, CA, 1988.

    Google Scholar 

  • O'Keefe, R. M.; Belton, V.; and Ball, T., “Experiences with using expert systems in O.R.,”J. Operational Res. Soc. vol. 37, no. 7, pp. 657–668, July 1986.

    Google Scholar 

  • O'Keefe, R. M., “Expert systems and MS/OR methodology: Good news or bad,”Interfaces vol. 18, no. 6, pp. 105–113, 1988.

    Google Scholar 

  • O'Keefe, R. M., “The evaluation of decision-aiding systems: Guidelines and methods,”Inf. Manage. vol. 17, no. 4, pp. 217–226, November 1989.

    Google Scholar 

  • O'Leary, D., “Methods of validating expert systems,”Interfaces vol. 18, no. 6, pp. 72–79, December 1988.

    Google Scholar 

  • O'Leary, D., and O'Keefe, R. M., “Verifying and validating expert systems,” inProc. Eleventh Int. Joint Conf. Artif. Intell. (IJCAI '89), International Joint Conference on Artificial Intelligence, Detroit, MI, 1989.

  • O'Leary, T. J.; Goul, M.; Moffitt, K. E.; and Radwan, A. E., “Validating expert systems,”IEEE Expert vol. 5, no. 3, pp. 51–58, June 1990.

    Google Scholar 

  • O'Leary, D. E., “On the representation and the impact of reliability on expert systems weights,”Int. J. Man-Machine Studies vol. 1, no. 3, pp. 253–279, 1988.

    Google Scholar 

  • O'Leary, D., and Watkins, P.,Expert Systems in Internal Auditing, Institute of Internal Auditors, research monograph, 1989.

  • O'Neil, M., and Glowinski, A., “Evaluating and validating very large knowledge-based systems,”Med. Inf. vol. 15, no. 3, pp. 51–58, July–September 1990.

    Google Scholar 

  • Parsaye, K., “Acquiring and verifying knowledge automatically,”Al Expert vol. 3, no. 5, pp. 48–63, 1990.

    Google Scholar 

  • Pau, L. F., “Prototyping, validation, and maintenance of knowledge-based systems software,” inProceedings of the Third Annual Expert Systems in Government Symposium IEEE Computer Society Press: Washington, DC, pp. 248–253, 1987.

    Google Scholar 

  • Pearce, D., “KIC: A knowledge integrity checker,” Turing Institute, Research Memoranda TIRM-87-025, Glasgow, Scotland, 1987.

  • Pederson, K., “Well-structured knowledge bases—Part I,”Al Expert, pp. 44–45, April 1989.

  • Pederson, K., “Well-structured knowledge bases—Part II,”Al Expert, pp. 44–45, July 1989.

  • Phillips, M. E., and Brown, C. E., “Need an expert? Just press a computer key,”J. Accountancy vol. 172, no. 5, pp. 91–93, November 1991.

    Google Scholar 

  • Pipard, E., “Detection of contradictions in knowledge bases,” inFifth Int. Workshop Expert Syst. Appl., Avignon, France, May 1985, pp. 995–1010, Agence de l'Informatique, Paris, France.

  • Plant, R. T., “Towards the verification of knowledge based systems,”2nd Int. Conf. Ind. Eng. Appl. Artif. Intell. Expert Syst. vol. 2, p. 1106, June 1989.

    Google Scholar 

  • Politakis, P., and Weiss, S. M., “Using empirical analysis to refine expert system knowledge bases,”Artif. Intell. vol. 2, no. 1, pp. 23–48, January 1984.

    Google Scholar 

  • Politakis, P., “Empirical analysis for expert systems,”Research Notes in Artificial Intelligence vol. 6, Pitman: Boston, MA, 1984.

    Google Scholar 

  • Preece, A. D., “Verifying expert systems using conceptual models,” inProc. 5th Ann. Knowledge-Based Software Assistant (KBSA) Conf. Rome Air Development Center (RADC), Griffiss AFB, NY, 1990, pp. 280–289.

    Google Scholar 

  • Preece, A. D.; Suen, C. Y.; and Yu, C. L., “Performance assessment of a character recognition expert system,” inProc. 2nd Expert Syst. Appl. World Conf. (EXPERSYS-90), edited by A. Dollas, W. T. Tsai, and N. G. Bourbakis. Gournay-sur-Marne, France, 1990, pp. 103–109.

    Google Scholar 

  • Preece, A. D., and Shinghal, R., “Logical foundations of verifying expert systems rule bases”, inProc. ICO '9! Conf. Cognitive Sci. Tools for the Dev. Organ. GIRICO, Montreal, Canada, 1991, pp. 94–101.

    Google Scholar 

  • Preece, A. D., and Shinghal, R., “DARC: A procedure for verifying rule-based systems,” inProc. First World Congress Expert Syst. Pergamon Press: New York, pp. 971–979, 1991.

    Google Scholar 

  • Preece, A. D., and Shinghal, R., “Verifying knowledge bases by anomaly detection: An experience report,”1992 European Conference on Artificial Intelligence Feb. 7, 1992, Wiley Publishers: New York, NY, 1992.

    Google Scholar 

  • Preece, A. D., “Verification of rule-based expert systems in wide domains,” inResearch and Development in Expert Systems VI edited by N. Shadbolt, Cambridge University Press: Cambridge, England, pp. 66–77, 1990.

    Google Scholar 

  • Prerau, D. S.; Gunderson, A. S.; Reinke, .; and Goyal, S. K., “The COMPASS expert system: Verification, technology, transfer, and expansion,” inProc. 2nd Conf. Artif. Intell. Appl. Eng. Knowledge-Based Syst., IEEE Computer Society, 1985, pp. 597–602.

  • Puuronen, S., “A tabuloar rule-checking method,” inProc. Expert Syst. Appl. AGence Inf. La Defense, Paris, France, 1987, pp. 257–268.

    Google Scholar 

  • Ramamoorthy, C. V., and Shekhar, S., “Software development support for AI programs,”Computer, pp. 30–40, January 1987.

  • Ramsey, C. L., and Basili, V. R., “An evaluation of expert systems for software engineering management,”IEEE Trans. Software Eng., vol. SE-15, no. 4, December 1976.

  • Ravi, P.; Subrahmanian, G.; and Mahabala, H., “A methodology for systematic verification of OPS5-based AI applications,” inProc. IJCAI-91, pp. 3–8.

  • Reed, J., and Yeh, R. T., “Specification and verification of liveness properties of cyclic, concurrent processes,”ACM Trans. Programming Lang. Syst. vol. 10, no. 1, pp. 156–177, January 1988.

    Google Scholar 

  • Ribar, G., “Expert systems validation: A cause study,”Expert Syst. Rev., vol. 1, no. 3, 1988.

  • Richardson, K., and Wong, C., “KBS V&V as related to automation of space station subsystems: Rationale for a KBS lifecycle,” inProc. Third Conf. Artif. Intell. Space Appl. National Aeronautics and Space Administration—Scientific and Technical Branch, Washington, DC, p. 85. November 2–3, 1987 (abstract only).

    Google Scholar 

  • Richer, M. H., “An evaluation of expert system development tools,”Expert Syst. vol. 3, no. 3, pp. 166–177, July 1986.

    Google Scholar 

  • Ricks, W. R., and Abbott, K. H., “Evaluation of knowledge-based systems,” inProc. AAAI Workshop on Validation and Verification of Expert Syst. American Association for Artificial Intelligence, Menlo Park, CA, August 1988 (unpublished).

    Google Scholar 

  • Rousset, M. C., “On the consistency of knowledge bases: The Covadis system,”Comput. Intell. (Canada) vol. 4, no. 2, pp. 166–170, 1988.

    Google Scholar 

  • Rothenberg, J.; Paul, J.; Kameny, I.; Kipps, J. R.; and Swenson, M., “Evaluating expert system tools: A framework and methodology,” Rand Corporation,Technical Report R-3542-BARPA, July 1987.

  • Rushby, J., and Whitehurst, R. A.,Formal Verification of AI Software, SRI International, N89-2481116, 28 Feb. 1989, p. 156.

  • Sargent, R., “An expository on verification and validation of simulation models,” inProc. 1985 Winter Simulation Conf. IEEE Computer Society, Dallas, TX, 1986, pp. 15–22.

    Google Scholar 

  • Sargent, R., “The use of graphical models in model validation,” inProc. 1986 Winter Simulation Conf. IEEE Computer Society, Washington, DC, 1986, pp. 237–241.

    Google Scholar 

  • Satre, T. W., and Massey, J. G., “Expert systems verification and validation—Defining the concepts,” inProceedings: Applications of Artificial Intelligence in Engineering, Vol. 6, Computational Mechanics Publications: Boston, MA, July 2–4, 1991, pp. 859–872.

    Google Scholar 

  • Schaeffer, P. R.; Bozma, I. H.; and Beer, R. D., “Knowledge-based validity maintenance for production systems,” inProc. 5th Nat. Conf. Artif. Intell. (AAAI 86), Vol. 2, AAAI, pp. 918–922, 1986.

  • Schultz, R. D., and Geissman, J. R. “Bridging the gap between static and dynamic verification,” inValidation and Testing Knowledge Based Syst. Workshop, Minneapolis, MN, August 1988.

  • Shatz, H. R., and Campbell, L., “WxperTAX: The issue of long-term maintenance,” inProc. 3rd Int. Conf. Expert Syst., 1987, pp. 291–300.

  • Shaw, M. L. G., and Gaines, B. R., “Comparing conceptual structures: Consensus, conflict, correspondence and contrast,”Knowledge Acquisition vol. 1, pp. 341–363, 1989.

    Google Scholar 

  • Shpilberg, D.; DeSalvo, J.; and Muchalski, S., “Tomorrow's expert systems,”Best's Rev. (Prop/Casualty), 88, no. 1, pp. 28–29, 49, 1987.

    Google Scholar 

  • Shwe, M. A.; Tu, S. W.; and Fagan, L. M., “Validating the knowledge base of a therapy planning system,”Methods Inf. Med. vol. 28, no. 1, pp. 26–50, January 1989.

    Google Scholar 

  • Skingle, B., “The validation of knowledge-based systems,” inProceedings of the International Conference on Knowledge-Based Systems (KBS '87), Online Publishers: Pinner, pp. 27–36, 1987.

    Google Scholar 

  • Slagle, J., and Wick, M., “A metrhod for evaluating candidate expert system applications,”AI Mag. vol. 9, no. 4, pp. 44–53, 1989.

    Google Scholar 

  • Smith, R. G.; Winston, H. A.; Mitchele, T. M.; and Buchanan, B. G., “Representation and use of explicit justifications for knowledge base refinement,” inProc. Ninth Int. Joint Conf. Artif. Intell. IJCAT-PS, vol. 2, pp. 673–680, 1985.

    Google Scholar 

  • St. Clair, D. C.; Bond, W. E.; and Flachsbart, B. B., “Using output to evaluate and refine rules in rule-based expert systems,” inProc. 3rd Conf. Artif. Intelligence for Space Applications, University of Alabama, Huntsville and NASA, NASA Conference Publication 2492, pp. 9–13, 1987.

  • Stachowitz, R. A.; Chang, C. L.; and Combs, J. B., “Research on validation of knowledge-based systems,” inAAAI Workshop on Validation and Verification of Expert Syst. American Association for Artificial Intelligence, Menlo Park, CA, August, 1988.

    Google Scholar 

  • Stachowitz, R. A.; Chang, C. L.; Stock, T. S.; and Combs, J. B., “Building validation tools for knowledge-based systems,” inProc. Conf. Space Operations Automation Robot. (SOAR 87) SOAR Conference, NASA/JSC, Houston, TX, pp. 209–216, August 1987.

    Google Scholar 

  • Stanfield, J. L., and Greenfield, N. R., “Planpower: A comprehensive financial planner,”IEEE Expert vol. 2, no. 3, pp. 51–60, 1987.

    Google Scholar 

  • Stewart, D. A., and Surgenor, B. W., “Simulator validation of an expert system for process fault diagnosis,” inProc. 19th Summer Comput. Simulation Conf. Society for Computer Simulation, San Diego, CA, 1987, pp. 663–666.

    Google Scholar 

  • Suwa, M. A., and Shortliffe, E., “Completeness and consistency in rule-based expert systems,”AI Mag. vol. 3, no. 4, pp. 16–21, 1982.

    Google Scholar 

  • Swarout, W. R., and Snidiar, S. W., “On making expert systems more like experts,”Expert Syst. vol. 4, no. 3, pp. 196–207, August 1987.

    Google Scholar 

  • Sztipanovitz, J.; Padalkar, S.; Krishnamurthy, C.; and Purves, R. S., “Testing and validation in artificial intelligence programming,”SPIE Space Station Automation III, vol. 851, pp. 2–9, 1987.

    Google Scholar 

  • Terano, Takoa; Kongoji, H.; Kaji, K.; and Yamamoto, K., “Development of a guideline for expert system evaluation—A report on a three year project,”AAAI Workshop 1991, pp. pp1–19.

  • Terano, T., and Kongoji, S., “Problem analyses, tool evaluation, and verification & validation study: Three steps for knowledge-based systems development methodology, IJCAI-89 Workshop on VV&T of Knowledge-Based Systems, 1989.

  • Todd, P., and Benbasat, I., “Process tracing methods in decision support system research: Exploring the blackbox,”MiS Q. vol. 11, no. 4, pp. 493–512, 1987.

    Google Scholar 

  • Tong, R. M.; Newman, N. D.; Berg-Cross, G.; and Rook, F., “Performance valuation of artificial intelligence systems,” Advanced Decision Systems, Mountain View, CA,Technical Report ADS TR-3154, vol. 1, no. 3, 1988.

  • Tsai, T.; Heisler, K.; Volovik, D.; and Zualkerman, I., “Software engineering of expert systems,” Department of CS, University of Minneapolis, MN,TR-87-21, 1987.

  • Valid, “Validation methods and tools for knowledge-based systems,” Deliverable, Espirit II Valid Project 2148, 1988.

  • Van De Burg, A.; Bachant, J.; and McDermott, J., “The taming of AI,”IEEE Expert, pp. 33—39, Fall 1986.

  • Weiss, S. M., and Kulikowski, C. M., “Testing and evaluating expert systems,” inA Practical Guide to Designing Expert Systems Rowman & Allanheld: Totowa, NJ, pp. 138–156, 1983.

    Google Scholar 

  • Yen, John, “A principled approach to reasoning about the specificity of rules,” inProc. Natl. Conf. Artif. Intell. (AAAI-90), Boston, August 1990, pp. 701–707.

  • Yen, John, and Lee, J., “Designing verifiable expert systems,”Second Int. IEEE Conf. Tools for Artif. Intell., Nov. 6–9, 1990, pp. 878–884.

  • Yen, John, “Using terminological models to enhance the rule-based paradigm,” inProc. Second Int. Symp. Artif. Intell., Monterrey, Mexico, October 25–27, 1989.

  • Yoon, F. P., “Techniques for data and rule validation in knowledge-based systems,” inCompass 1989, Proc. Fourth Annu. Conf. Comput. Assurance Syst. Integrity Software Safety Process Security, Gaithersburg, MD, June 19–23, 1989, pp. 62–70.

  • Yu, V.; Buchanan, B.; Shortlife, E.; Wraith, S.; Davis, R.; Scott, A.; and Cohen, S., “Evaluating the performance of a computer-based consultant,”Comput. Programs Biomed. vol. 9, no. 1, pp. 95–102, 1979.

    Google Scholar 

  • Yu, V. L.; Fagan, L. M.; Bennett, S. W.; Clancey, W. J.; Scott, A. C.; Hannigan, J. F.; Blum, R. L.; Buchanan, B. G.; and Cohen, S. N., “An evaluation of MYCIN's advice,” inRule-Based Expert Systems edited by B. G. Buchanan, Addison-Wesley: Reading, MA, pp. 717–738, 1984.

    Google Scholar 

  • Yu, V.; Fagan, L.; Wraith, S.; Clancey, W.; Scott, A.; Hanigan, J.; Blum, R.; Bichanan, B.; and Cohen, S., “Antimicrobial selection by computer,”JAMA vol. 242, no. 12, pp. 1279–1282, 1979.

    Google Scholar 

  • Yu, X., and Biswas, G., “CHECKER: An Efficient Algorithm for Knowledge Base Verification,” inProc. Third Int. Conf. Ind. Eng. Appl. Artif. Intell. Expert Syst. vol. 2, pp. 735–744, July 1990.

    Google Scholar 

  • Zhang, Du, and Nguyen, D., “A technique for knowledge base verification,”IEEE Workshop on Tools for Artif. Intell., 1989, pp. 399–406.

  • Zhao, S., and Shen, S., “The consistency problem of knowledge base,”Proc. Third Int. Conf. Ind. Eng. Appl. Artif. Intell. Expert Syst. vol. 2, pp. 745–750, July 1990.

    Google Scholar 

  • Zlatareva, N., “A truth maintenance approach to knowledge-based systems verification,”Artif. Intell., Review, vol. 6, 1992.

Download references

Author information

Authors and Affiliations

  1. Department of Decision Sciences, East Carolina University, 27858-4353, Greenville, NC

    Uma G. Gupta (Assistant Professor)

Authors
  1. Uma G. Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gupta, U.G. Validation and verification of knowledge-based systems: A survey. Appl Intell 3, 343–363 (1993). https://doi.org/10.1007/BF00872136

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00872136

Key words

Search

Navigation