article

A methodology for certification of modeling and simulation applications

Author:

Osman BalciAuthors Info & Claims

ACM Transactions on Modeling and Computer Simulation (TOMACS), Volume 11, Issue 4

Pages 352 - 377

https://doi.org/10.1145/508366.508369

Published: 01 October 2001 Publication History

Abstract

Certification of modeling and simulation (M&S) applications poses significant technical challenges for M&S program managers, engineers, and practitioners. Certification is becoming increasingly more important as M&S applications are used more and more for military training, complex system design evaluation, M&S-based acquisition, problem solving, and critical decision making. Certification, a very complex process, involves the measurement and evaluation of hundreds of qualitative and quantitative elements, mandates subject matter expert evaluation, and requires the integration of different evaluations. Planning and managing such measurements and evaluations requires a unifying methodology and should not be performed in an ad hoc manner. This paper presents such a methodology. The methodology consists of the following body of methods, rules, and postulates: (a) employment of subject matter experts, (b) construction of a hierarchy of indicators, (c) relative criticality weighting of indicators using the analytic hierarchy process, (d) using a rule-based expert knowledge base with an object-oriented specification language, (e) assignment of crisp, fuzzy, and nominal scores for the indicators, (f) aggregation of indicator scores, (g) graphical representation of the indicator scores and weights, (h) hypertext certification report, and (i) interpretation of the results. The methodology can be used for certification of any kind of M&S application either throughout the M&S development life cycle or after the development is completed.

References

[1]

ABET. 2002. Accreditation Board for Engineering and Technology, Baltimore, MD. (http://www.abet.org)

[2]

Balci, O. 1998. Verification, validation, and testing. In The Handbook of Simulation, J. Banks, Ed. John Wiley & Sons, New York, NY, Aug., 335--393.

[3]

Balci, O. and Nance, R. E. 1985. Formulated problem verification as an explicit requirement of model credibility. Simulation 45, 2 (Aug.), 76--86.

[4]

Balci, O., Nance, R. E., Arthur, J. D., and Ormsby, W. F. 2002. Expanding our horizons in verification, validation, and accreditation research and practice. In Proceedings of the 2002 Winter Simulation Conference (San Diego, CA, Dec. 8--11). E. Yücesan, C.-H. Chen, J. L. Snowdon, and J. M. Charnes, Eds. IEEE, Piscataway, NJ.

[5]

Balci, O. and Ormsby, W. F. 2000. Well-defined intended uses: an explicit requirement for accreditation of modeling and simulation applications. In Proceedings of the 2000 Winter Simulation Conference (Orlando, FL, Dec. 10--13). J. A. Joines, R. R. Barton, K. Kang, and P. A. Fishwick, Eds. IEEE, Piscataway, NJ, 849--854.

[6]

Balci, O., Ormsby, W. F., Carr III, J. T., and Saadi, S. D. 2000. Planning for verification, validation, and accreditation of modeling and simulation applications. In Proceedings of the 2000 Winter Simulation Conference (Orlando, FL, Dec. 10--13). J. A. Joines, R. R. Barton, K. Kang, and P. A. Fishwick, Eds. IEEE, Piscataway, NJ, 829--839.

[7]

Binder, R. V. 2000. Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley, Reading, MA.

[8]

Brown, A. W., Ed. 1996. Component-Based Software Engineering. IEEE Computer Society Press, Los Alamitos, CA.

[9]

CMU SEI. 1994. The Capability Maturity Model: Guidelines for Improving the Software Process. Addison-Wesley, Reading, MA.

[10]

DoDI. 1996. DoD modeling and simulation verification, validation, and accreditation. Department of Defense Instruction 5000.61, Apr.

[11]

Glasow, P. A. 1998. Characteristics, selection, and use of subject matter experts. Tech. Rep. 99W0000048, POET WG2K IV&V Paper No. 25, The MITRE Corporation, McLean, VA, June.

[12]

Glasow, P. A. and Pace, D. K., Eds. 1999. Report on the simulation validation (SIMVAL) 1999---making VV&A effective and affordable---mini-symposium and workshop. Military Operations Research Society, Alexandria, VA, May.

[13]

IEEE. 1993. IEEE Guide for Software Verification and Validation Plans. IEEE Standard 1059--1993. IEEE, Piscataway, NJ.

[14]

ISACC. 1999. Proceedings of the 1st International Software Assurance Certification Conference. (Feb. 28--Mar. 2, Chantilly, VA). Reliable Software Technologies, Sterling, VA.

[15]

Loesh, R. E., Gosnell, A. B., Benoist, T. F., Wyskida, R. M., and Johannes, J. D. 1999. An engineering approach to critical software certification. In Proceedings of the 32nd Annual Hawaii International Conference on System Sciences (Jan. 5--8, Maui, Hawaii). IEEE Computer Society, Washington, DC.

[16]

NIST. 2002. Component-Based Software Advanced Technology Program. National Institute of Standards and Technology, Gaithersburg, MD. (http://www.atp.nist.gov/atp/focus/cbs.htm)

[17]

Orca Computer. 1999a. Evaluation Environment User's Guide. Orca Computer, Inc., Blacksburg, VA.

[18]

Orca Computer. 1999b. Evaluation Environment Subject Matter Expert User's Guide. Orca Computer, Inc., Blacksburg, VA.

[19]

Pace, D. K. 1998. Verification, validation, and accreditation. In Applied Modeling and Simulation: An Integrated Approach to Development and Operation. D. J. Cloud and L. B. Rainey, Eds. McGraw-Hill, New York, NY, 369--410.

[20]

Poore, J. H., Mills, H. D., and Mutchler, D. 1993. Planning and certifying software system reliability. IEEE Software 10, 1 (Jan./Feb.), 88--99.

[21]

Rae, A., Robert, P., and Hausen, H.-L. 1995. Software Evaluation for Certification: Principles, Practice, and Legal Liability. McGraw-Hill, London, UK.

[22]

Rodríguez-Dapena, P. 1999. Software safety certification: a multidomain problem. IEEE Software 16, 4 (July/Aug.), 31--38.

[23]

Saaty, T. L. 1980. The Analytic Hierarchy Process, McGraw-Hill, New York, NY.

[24]

Saaty, T. L. 1990. Multicriteria Decision Making: The Analytic Hierarchy Process, RWS Publications, Pittsburgh, PA.

[25]

Saaty, T. L. 1994. Fundamentals of Decision Making and Priority Theory with the Analytic Hierarchy Process. RWS Publications, Pittsburgh, PA.

[26]

Talbert, M. L. 1995. A methodology for the measurement and evaluation of complex system designs. Ph.D. Dissertation, Department of Computer Science, Virginia Tech, Blacksburg, VA, Dec.

[27]

Vermesan, A. I. 1997. Quality assessment of knowledge-based software: some certification considerations. In Proceedings of the 3rd International Software Engineering Standards Symposium (June 1--6, Walnut Creek, CA). IEEE Computer Society, Washington, DC, 144--154.

[28]

Vermesan, A. I. 1998. Software certification for industry---verification and validation issues in expert systems. In Proceedings of the 9th International Workshop on Database and Expert Systems Applications (Aug. 26--28, Vienna, Austria). IEEE Computer Society, Washington, DC.

[29]

Voas, J. M. 1998a. Certifying off-the-shelf software components. IEEE Computer 31, 6 (June), 53--59.

[30]

Voas, J. M. 1998b. Certifying high assurance software. In Proceedings of the 22nd International Computer Software and Application Conference (Aug. 17--21, Vienna, Austria). IEEE Computer Society, Washington, DC.

[31]

Voas, J. M. 1998c. The software quality certification triangle. Crosstalk 11, 11 (Nov.), 12--14.

[32]

Voas, J. M., Ed. 1999a. Special issue on software certification. IEEE Software 16, 4 (July/Aug.).

[33]

Voas, J. M. 1999b. Guest editor's introduction: certification: reducing the hidden costs of poor quality. IEEE Software 16, 4 (July/Aug.), 22--25.

[34]

Voas, J. M. 1999c. Certifying software for high-assurance environments. IEEE Software 16, 4 (July/Aug.), 48--54.

[35]

Voas, J. M. 1999d. Can generic software be assured? In Proceedings of the 23rd Annual International Computer Software and Applications Conference (Oct. 25--26, Phoenix, AZ). IEEE Computer Society, Washington, DC.

[36]

Voas, J. M. 2000a. Limited software warranties. In Proceedings of the 7th IEEE International Conference and Workshop on the Engineering of Computer Based Systems (Apr. 3--7, Edinburgh, Scotland). IEEE Computer Society, Washington, DC.

[37]

Voas, J. M. 2000b. Developing a usage-based software certification process. IEEE Computer 33, 8 (Aug), 32--37.

[38]

Wakid, S. A., Kuhn, D. R., and Wallace, D. R. 1999. Toward credible IT testing and certification. IEEE Software 16, 4 (July/Aug.), 39--47.

[39]

Wallace, D. R. 1999. Software verification & validation's role in product certification. In Proceedings of the First International Software Assurance Certification Conference (Feb. 28--Mar. 2, Chantilly, VA). Reliable Software Technologies, Sterling, VA.

[40]

Wohlin, C. and Regnell, B. 1998. Reliability certification of software components. In Proceedings of the 5th International Conference on Software Reuse (June 2--5, Victoria, B.C., Canada). IEEE Computer Society, Washington, DC.

[41]

Wohlin, C. and Runeson, P. 1994. Certification of software components. IEEE Transactions on Software Engineering 20, 6 (June), 494--499.

Cited By

Yang XXu ZXu J(2023)Large-scale group Delphi method with heterogeneous decision information and dynamic weightsExpert Systems with Applications10.1016/j.eswa.2022.118782213(118782)Online publication date: Mar-2023
https://doi.org/10.1016/j.eswa.2022.118782
Ören TGraciano Neto VDavis PZeigler B(2023)Reliability and Quality Assurance of M&SBody of Knowledge for Modeling and Simulation10.1007/978-3-031-11085-6_7(167-204)Online publication date: 28-Jan-2023
https://doi.org/10.1007/978-3-031-11085-6_7
Ören TLoper M(2023)Maturity and AccreditationBody of Knowledge for Modeling and Simulation10.1007/978-3-031-11085-6_10(249-253)Online publication date: 28-Jan-2023
https://doi.org/10.1007/978-3-031-11085-6_10
Show More Cited By

Index Terms

A methodology for certification of modeling and simulation applications

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation

ACM Transactions on Modeling and Computer Simulation Volume 11, Issue 4

October 2001

95 pages

ISSN:1049-3301

EISSN:1558-1195

DOI:10.1145/508366

Issue’s Table of Contents

Copyright © 2001 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2001

Published in TOMACS Volume 11, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

98
Total Citations
View Citations
1,907
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang XXu ZXu J(2023)Large-scale group Delphi method with heterogeneous decision information and dynamic weightsExpert Systems with Applications10.1016/j.eswa.2022.118782213(118782)Online publication date: Mar-2023
https://doi.org/10.1016/j.eswa.2022.118782
Ören TGraciano Neto VDavis PZeigler B(2023)Reliability and Quality Assurance of M&SBody of Knowledge for Modeling and Simulation10.1007/978-3-031-11085-6_7(167-204)Online publication date: 28-Jan-2023
https://doi.org/10.1007/978-3-031-11085-6_7
Ören TLoper M(2023)Maturity and AccreditationBody of Knowledge for Modeling and Simulation10.1007/978-3-031-11085-6_10(249-253)Online publication date: 28-Jan-2023
https://doi.org/10.1007/978-3-031-11085-6_10
Dolgov VNikishechkin PIvashin SDolgov N(2021)Simulation Modeling Features of Various Machine-Building Enterprise TypesMATEC Web of Conferences10.1051/matecconf/202134603079346(03079)Online publication date: 26-Oct-2021
https://doi.org/10.1051/matecconf/202134603079
Kim JMyoung JLim HSong D(2020)Efficiency Gap Caused by the Input Data in Evaluating Energy Efficiency of Low-Income Households’ Energy Retrofit ProgramSustainability10.3390/su1207277412:7(2774)Online publication date: 1-Apr-2020
https://doi.org/10.3390/su12072774
Birta LArbez GBirta LArbez G(2020)Modelling and Simulation FundamentalsModelling and Simulation10.1007/978-3-030-18869-6_2(19-53)Online publication date: 2-Jan-2020
https://doi.org/10.1007/978-3-030-18869-6_2
Jang RBae JLee HBak S(2019)Design and Implementation of Interworking System for Device Interaction in a Virtual Reality SettingJournal of Digital Contents Society10.9728/dcs.2019.20.2.28920:2(289-295)Online publication date: 28-Feb-2019
https://doi.org/10.9728/dcs.2019.20.2.289
Wall ESmith BVick SWatson NMcCall T(2019)Development of a weighting strategy for a manufacturability assessmentProcedia Computer Science10.1016/j.procs.2019.05.084153(309-316)Online publication date: 2019
https://doi.org/10.1016/j.procs.2019.05.084
Yang XXu ZHe RXue F(2019)Credibility Assessment of Complex Simulation Models Using Cloud Models to Represent and Aggregate Diverse Evaluation ResultsIntelligent Computing Methodologies10.1007/978-3-030-26766-7_28(306-317)Online publication date: 3-Aug-2019
https://dl.acm.org/doi/10.1007/978-3-030-26766-7_28
Galvão Scheidegger AFernandes Pereira TMoura de Oliveira MBanerjee ABarra Montevechi J(2018)An introductory guide for hybrid simulation modelers on the primary simulation methods in industrial engineering identified through a systematic review of the literatureComputers & Industrial Engineering10.1016/j.cie.2018.07.046124(474-492)Online publication date: Oct-2018
https://doi.org/10.1016/j.cie.2018.07.046
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents