Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
article
Free access

Concepts and criteria to assess acceptability of simulation studies: a frame of reference

Published: 01 April 1981 Publication History
  • Get Citation Alerts
  • First page of PDF

    References

    [1]
    Allen, B. and Clema, J.K. Independent Verification/Validation Support Software. Proc. Nat'l Aerospace and Electron. Conf., Dayton, Ohio, March 1976, pp. 276-281.
    [2]
    Annino, J.S. and Russell, E.C. The ten most frequent causes of simulation analysis failure-and how to avoid them. Simulation Today, 60, (June 1979), pp. 137-140.
    [3]
    B ahr, D. A note on figures of merit. A CM Performance Evaluations Rev., 4, 3, (July 1975), pp. 1-3.
    [4]
    Balci, O. and Sargent, R.G. Bibliography on Validation of Simulation Models. Newsletter-TIMS College on Simulation and Gaming, Spring 1980, pp. 11-15.
    [5]
    Beer, S. Platform for Change. Wiley, New York, 1975.
    [6]
    Belady, L.A. and Leavenworth, B. Program Modifiability. IBM, Thomas J. Watson Research Center, Res. Rept. RC 8147, 1980.
    [7]
    Boulton, J. and Gibson, A.R. First Ann. Rept. Canadian Nuclear Fuel Waste Management Program, Rept AECL-6443, Atomic Energy of Canada Ltd., Pinawa, Manitoba, 1979.
    [8]
    Britting, K.R. Backward integration of system dynamics models- A useful validation test? In Worm Modeling: A Dialogue. C.W. Churchman and R.O. Mason (Eds.) Elsevier, New York, 1976, pp. 141-149.
    [9]
    Cellier, F.E. Combined continuous/discrete system simulation by use of digital computers: Techniques and tools, Ph.D. Th.,Diss ETH No. 6483, Zurich, Switzerland. 1979, (Chapter 6-Software Robustness, pp. 121-159).
    [10]
    Cellier, F.E. How to enhance the robustness of simulation software. In Progress in Modelling and Simulation. F.E. CeUier (Ed.) Academic (London), England, (In press).
    [11]
    Cellier, F.E. and Moebius, P.J. Towards robust general purpose simulation software. ACM S1GNUM 14, 2, (June 1979).
    [12]
    Cutler, M.M. Proving properties of simulation programs for system verification and validation. Proc., Summer Comptr Simulation Conf. Toronto, Canada, 1979, pp. 610-616.
    [13]
    Cutler, M.M. A formal program model for discrete event simulations and its use in the mathematical verification and validation of system models and implementations. Ph.D. Th., UCLA, Calif., 1979.
    [14]
    Deeter, C.R. and Hoffman, A.A.J. An Annotated Bibliography of Mathematical Models Classified by Validation and Error Analysis Methods. Texas Christian University, Fort Worth, TX 1978.
    [15]
    De Millo, R.A., Lipton, R.J., Perlis, A.J. Social processes and proofs of theorems and programs. Comm. ACM, 22, 5, (May, 1979), pp. 271-280.
    [16]
    Dickhoven, S. Software requirements for an improvement of transfer and adaptability of models. In Utility and Use of Large-Scale Mathematical Models, S. Gass (Ed.). Nat'l Bur Stand. Special Publication No. 534, U.S. Dept. of Commerce, Washington, D.C., May 1979, pp. 195-205.
    [17]
    Elzas, M.S. Whither simulation? Proc., UKSC Conf. on Comptr Simulation, IPC Science and Technology Press, Chester, England, April 1978, pp. 494-501.
    [18]
    Elzas, M.S. Results of a simulation: Can they be "trusted"? Proc. Seminar on Modelling in Business of IFIP Applied Information Processing Group, Amsterdam, The Netherlands, Jan. 1978, pp. 170- 183.
    [19]
    Elzas, M.S. What is needed for robust simulation? In Methodology in Systems Modelling and Simulation, B.P. Zeigler et al. (Eds.) North-Holland, Amsterdam, 1979, pp. 57-91.
    [20]
    Emshoff, J.R. and Sisson, R.L. Design and Use of Computer Simulation Models, Collier-MacMillan, New York, 1970.
    [21]
    Gass, S.I. Evaluation of complex models. Comptrs and Oper. Res. 4. I (March, 1977), pp. 27-35.
    [22]
    Gass, S.I. A procedure for the evaluation of complex models. Proc. First Int. Conf. in Mathematical Models, Univ. of Missouri, 1977, pp. 247-258,
    [23]
    Gass, S.I. Computer model documentation: A review and an approach. Nat. Bur. Stand. Special Publication No. 500--39, U.S. Dept. of Commerce, Washington, D.C. Feb. 1979.
    [24]
    Gass, S.I. (Ed.). Utility and use of large-scale mathematical models. Proc. of a Workshop held at the Nat'l. Bur. Stand., Gaithersburg, Maryland, Apr. 28-29, 1977, National Bureau of Standards, Special Publication 534, Washington, D.C., U.S. Dept. of Commerce, May 1979.
    [25]
    Gass, S.I. Bibliography--Model evaluation and assessment. In Utility and Use of Large-Scale Mathematical Models, S.I. Gass (Ed.). Nat'l. Bur. Stand. Special Publication No. 534, U.S. Dept. of Commerce, Washington, D.C., May 1979, pp. 215-217.
    [26]
    Gass, S.I. Validation and Assessment Issues of Energy Models. Prepared for the NATO Advanced Research Institute, The Application of Systems Science to Energy Policy Planning. Nov. 12- 16, Brookhaven National Laboratory, Upton, New York, 1979.
    [27]
    Greenberger, M. and Richels, R. Assessing energy policy models: Current state and future directions. Ann. Rev. Energy, 4, (1979).
    [28]
    Henize, J. Evaluating policy models. Proc. Summer Comptr Simulation Conf., Toronto, Canada, 1979, pp. 550-556.
    [29]
    Holling, C.S. Adaptive Environmental Assessment and Management, John Wiley, New Yok, 1978.
    [30]
    House, P.W. and McLeod, J. Large-Scale Models for Policy Evaluation, Wiley-Interscience, New York, 1977.
    [31]
    Howden, W.E. Applicability of software validation techniques to scientific programs. A CM Trans. Programming Languages and Systems. 2, 3, (July 1980), pp. 307-320.
    [32]
    Hunt, R. Large-scale systems theories: Some paradigms from software systems engineering. In Cyberne.tics and Modelling and Simulation of Large-Scale Systems. T.I. Oren (Ed.) Int. Assoc. for Cybernetics, Namur, Belgium, 1978, pp. 141-150.
    [33]
    Kahne, S. Model credibility for large-scale systems. IEEE Trans. SMC, (Aug. 1976), pp. 586-590.
    [34]
    Karplus, W.J. The spectrum of mathematical modelling and systems simulation. In Simulation of Systems, L. Dekker (Ed.) North- Holland, Amsterdam, 1976, pp. 5-13.
    [35]
    Kheir, N.A. and W.M. Homes. On validating simulation models of missile systems. Simulation, 30, 4, (Apr. 1978), pp. 117-128.
    [36]
    Kleijnen, J.P.C. Statistical design and analysis of simulation experiments. Proc. Simulation '75, Ziirich, Switzerland Acta Press, 1975.
    [37]
    Kleijnen, J.P.C. The role of statistical methodology in simulation. In Methodology in Systems Modelling and Simulation, B.P. Zeigler, M.S. Elzas, G.J. Klir, and T.I. Oren (Eds.) North-Holland, Amsterdam, 1979, pp. 425-445.
    [38]
    Lee, D.B. Requiem for large-scale models. J. Am. Inst. of Planners, 39, 3 (May 1973), pp. 163-178.
    [39]
    Marmor-Squires, A.B. Software validation: An annotated bibliography. In Current Trends in Programming Methodology. Vol. II. Program Validation, R.T. Yeh (Ed.) Prentice-Hall, Englewood Cliffs, N.J., 1977, pp. 269-319.
    [40]
    McLeod, J. Terms and definitions-'Play it Again, Stew'. Simulation Today, 29, 4 (Oct. 1977) pp. vii-ix.
    [41]
    Mihram, G.A. Simulation--Statistical Foundations and Methodology. Academic Press, New York, 1972.
    [42]
    Mihram, G.A. Some practical aspects of the verification and validation of simulation models. Operational Res. Quart., 23, (1972) pp. 17-29.
    [43]
    Mihram, G.A. Four questions regarding the credibility of simulation. Proc. 7th Ann. Pittsburgh Conf., Part 2-Modeling and Simulation, Pittsburgh, PA, Instrum. Soc. Am. Apr 1976, pp. 1225- 1230.
    [44]
    Mihram, G.A., G. Innis, E.J. Mastascusa, S. Schlesinger, and T.J. Teorey. Bibliography on simular credibility. Simulation, 23, 5, (Dec. 1974) p. vi.
    [45]
    Miller, E., and Howden, W.E. Tutorial: Software Testing and Validation Techniques. IEEE Comptr Soc., New York, NY, 1978.
    [46]
    Myers, G.J. The Art of Software Testing. Wiley, New York, 1979.
    [47]
    Nance, R.E. The feasibility of and methodology for developing federal documentation standards for simulation models. Rept. Dept. of Comptr Sci. VP1 and State University, Blacksburg, VA, 1977.
    [48]
    Newton, O.L. and Weatherbee, J.E. Guidelines for documenting computer simulation models. Project Memorandum MV-068-060- USAF, Federal Computer Performance Evaluation and Simulation Center, Washington, DC, 1979.
    [49]
    Newton, O.L. and Whetherbee, J.E. Guidelines for documenting simulation models: A review and procedures. In Simulation with Discrete Models: A State-of-the-Art View, T.I. Oren, C.M. Shub, and P.F. Roth (Eds.). IEEE, New York, 1980 pp. 243-258.
    [50]
    Oren, T.I. Deductive general systems theories and simulation of large-scale systems. Proc., Summer Comptr Simulation Conf., Houston, Texas, July 1974 pp. 13-16.
    [51]
    Oren, T.I. Simulation-as it has been, is, and should be. Simulation, 29, 5 (Nov. 1977) pp. 182-183.
    [52]
    Oren, T.I. Modelling, model manipulation and programming concepts in simulation. In Modelling, Identification, and Control in Environmental Systems, G.C. van Steenkiste (Ed.) North-Holland, Amsterdam, 1978 pp. 833-84 Z
    [53]
    Oren, T.I. Rationale for large-scale systems simulation software based on cybernetics and general system., theories. In Cybernetics and Modelling of Large-Scale Systems, T.I. Oren (Ed.) Int. Assoc. for Cybe.rnetics, Namur, Belgium, 1978 pp. 151-179.
    [54]
    Oren, T.I. Reference manual of GEST 78-Level 1. (A modelling and simulation language for combined systems). Tech. Rept. 78-02, Univ. of Ottawa, Comptr Sci. Dept., Ottawa, Ontario, Canada, 1978.
    [55]
    Oren, T.I. Epilogue. In Cybernetics and Modelling of Large-Scale Systems. T.I. Oren (Ed.) Int. Assoc. for Cybernetics, Namur, Belgium, 1978, pp. 181-189.
    [56]
    Oren, T.I. Concepts for advanced computer-assisted modelling. In Methodology in Systems Mod.elling and Simulation, B.P. Zeigler, M.S. Elzas, G.J. Klir, and T.I. Oren (Eds.). North-Holland, Amsterdam, 1979, pp. 29-55.
    [57]
    Oren, T.I. Computer-aided modelling systems. In Progress in Modelling and Simulation F.E. Cellier (Ed.), Academic (London), Engl. and (In press).
    [58]
    Oren, T.I. and Collie, B. Design of SEMA: A software system for computer-aided modelling and simulation of.SEquential MAchines. In Proc. 1980 Winter Simulation Conf., T.I. Oren, C.M. Shub, and P.F. Roth (Eds.), 1980, pp 113-123.
    [59]
    Oren, T.I. and den Dulk, J. Ecological models expressed in GEST 78, Rept prepared for the Dept. of Theoretical Production Ecology, Dutch Agricultural Univ., Wageningen, the Netherlands, 1978.
    [60]
    Oren, T.I. and Zeigler, B.P. Concepts for advanced simulation methodologies. Simulation, 32, 3 (March 1979) pp. 69-82.
    [61]
    O'Shaugnessy, J. Inquiry and Decisions. Allen & Unwin, London, 1972.
    [62]
    Pugh, R.E. Evaluation of Policy Simulation Models. Information Resources Press, Washington, DC, 1977.
    [63]
    Roth, P. F., Gass, S.I., and Lemoine, A.J. Some considerations for improving federal modelling. Proc. 1978 Winter Simulation Conf., Dec. 1978 pp. 213-217.
    [64]
    Salaway, G. and Siler, K.F. Validation considerations during parameter development for simulation models. Proc., Summer Comptr Simulation Conf., Toronto, Canada, July 1979 pp. 624-628.
    [65]
    Sargent, R.G. Validation of discrete-event simulation models. Proc. NSF Workshop on Validation of Mathematical Models in Energy Related Research and Development. Forth Worth, Texas, 1978.
    [66]
    Sargent, R.G. Validation of simulation models. Proc., Winter Simulation Conf., Dec. 1979 pp. 497-503.
    [67]
    Sargent, R.G. Verification and validation of simulation models. In Progress in Modelling and Simulation. F.E. Cellier (Ed.), Academic (London), England, (In Press).
    [68]
    Saridis, G.N. Comparison of six on-line identification algorithms. Automatica, 10, 1 (Jan. 1974) pp. 69-79.
    [69]
    Schellenberger, R.E. Criteria for assessing model validity for managerial purposes. Decision Science, 5, 4 (Oct. 1974) pp. 644-653.
    [70]
    Schlesinger, S., J.R. Buyan, E.D. Collender, W.K. Clarkson, and F.M. Perkins. Developing standard procedures for simulation validation and verification. Proc. Summer Comptr Simulation Conf., July 1974, pp. 927-933.
    [71]
    Schlesinger, S. et al. Terminology for model credibility. Simulation, 32 3, (1979) pp. 103-104.
    [72]
    Schruben, L.W. Establishing the credibility of simulations. Simulation,.34, 3 (March 1980) pp. 101-105.
    [73]
    Teorey, T.J. Validation criteria for computer systems simulation. Proc. Symp. Simulation of Computer Systems, Boulder, Colorado, Nat'l Bur Stand, Aug. 1975, pp. 161-173.
    [74]
    Thissen, W. Investigations into the Club of Rome's World 3 Model - Lessons for understanding complicated models. Ph.D. Th., Dept. of Technical Physics, Eindhoven University of Technology, The Netherlands, 1978.
    [75]
    Thissen, W. Investigations into the World 3 Model: Lessons for understanding complicated models. IEEE Trans. on Systems, Man, and Cybernetics, SMC-8, 3 ( March 1978) pp. 183-193.
    [76]
    Toffler, A. The Eco-spasm Report. Bantam Books, New York, 1975.
    [77]
    U.S. Dep't Justice. Descriptions and assessments of the model evaluation program projects, Stock Number 27-000-00797-0, U.S. Govt Printing Office, Washington, DC., 1979.
    [78]
    Waller, J.D., Scanlon, J.W., Kemp, D.M., and Nalley, P.G. Developing useful evaluation capability: Lessons from the model evaluation program, Stock Number 27-000-00798-8, U.S. Govt Printing Office, Washington, DC., 1979.
    [79]
    Watt, K.F. Why won't anyone believe us? Simulation, 28 (Jan. 1977), pp. 1-3.
    [80]
    Wigan, M.R. The fitting, calibration, and validation of simulation models. Simulation, 18, 5, (May 1972) pp. 188-192.
    [81]
    Wong, C.S.Y. Criteria sensitivity analysis: A new approach to parameter sensitivity. Applied Mathematical Modelling. 4, (Feb. 1980) pp. 7-15.
    [82]
    Wright, R.D. Retrodictive tests of dynamic models. Proc. Summer Comptr Simulation Conf., Montreal, Canada, July 1973.
    [83]
    Wymore, A.W. Systems Engineering Methodology for Interdisciplinary Teams. Wiley, New York, 1974.
    [84]
    Yeh, R.T. (Ed.) Current Trends in Programming Methodoloy, Vol. ll--Program Validation. Prentice-Hall, Englewood Cliffs, N J, 1977.
    [85]
    Zeigler, B.P. Theory of Modelling and Simulation. Wileylnterscience, New York, 1976.
    [86]
    Zeigler, B.P. Structuring the Organization of Partial.Models. In Cybernetics and Modelling of Large-Scale Systems. T.1. Oren (Ed.). Int. Assoc. for Cybernetics, Namur, Belgium, 1978 pp. 127-139.

    Cited By

    View all
    • (2023)Reliability and Quality Assurance of M&SBody of Knowledge for Modeling and Simulation10.1007/978-3-031-11085-6_7(167-204)Online publication date: 28-Jan-2023
    • (2022)Calibrating Simulation Models with Sparse DataProceedings of the Winter Simulation Conference10.5555/3586210.3586251(496-507)Online publication date: 11-Dec-2022
    • (2022)Calibrating Simulation Models with Sparse Data: Counterfeit Supply Chains During Covid-192022 Winter Simulation Conference (WSC)10.1109/WSC57314.2022.10015241(496-507)Online publication date: 11-Dec-2022
    • Show More Cited By

    Index Terms

    1. Concepts and criteria to assess acceptability of simulation studies: a frame of reference

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image Communications of the ACM
      Communications of the ACM  Volume 24, Issue 4
      Special issue on simulation modeling and statistical computing
      April 1981
      102 pages
      ISSN:0001-0782
      EISSN:1557-7317
      DOI:10.1145/358598
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 01 April 1981
      Published in CACM Volume 24, Issue 4

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. assessment of simulation studies
      2. code verification
      3. figure of merit
      4. model calibration
      5. model certification
      6. model fitting
      7. model robustness
      8. model validation
      9. program efficiency
      10. program referability
      11. software certification
      12. software reliability
      13. software robustness

      Qualifiers

      • Article

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)68
      • Downloads (Last 6 weeks)9

      Other Metrics

      Citations

      Cited By

      View all
      • (2023)Reliability and Quality Assurance of M&SBody of Knowledge for Modeling and Simulation10.1007/978-3-031-11085-6_7(167-204)Online publication date: 28-Jan-2023
      • (2022)Calibrating Simulation Models with Sparse DataProceedings of the Winter Simulation Conference10.5555/3586210.3586251(496-507)Online publication date: 11-Dec-2022
      • (2022)Calibrating Simulation Models with Sparse Data: Counterfeit Supply Chains During Covid-192022 Winter Simulation Conference (WSC)10.1109/WSC57314.2022.10015241(496-507)Online publication date: 11-Dec-2022
      • (2022)Security of Cyber-Physical-Social Systems: Impact of Simulation-Based Systems Engineering, Artificial Intelligence, Human Involvement, and EthicsAdvances in Computing, Informatics, Networking and Cybersecurity10.1007/978-3-030-87049-2_26(711-732)Online publication date: 3-Mar-2022
      • (2021)Developing a calibrated discrete event simulation model of shops of a Dutch phone and subscription retailer during COVID-19 to evaluate shift plans to reduce waiting timesProceedings of the Winter Simulation Conference10.5555/3522802.3522894(1-12)Online publication date: 13-Dec-2021
      • (2021)Developing a Calibrated Discrete Event Simulation Model of Shops of a Dutch Phone and Subscription Retailer During COVID-19 to Evaluate Shift Plans to Reduce Waiting Times2021 Winter Simulation Conference (WSC)10.1109/WSC52266.2021.9715306(1-12)Online publication date: 12-Dec-2021
      • (2020)DEVS-scriptingProceedings of the Winter Simulation Conference10.5555/3466184.3466435(2196-2207)Online publication date: 14-Dec-2020
      • (2020)How do modelers code artificial societies?Proceedings of the 2020 Spring Simulation Conference10.5555/3408207.3408257(1-12)Online publication date: 19-May-2020
      • (2020)DEVS-scripting: A Black-Box Test Frame for Devs Models2020 Winter Simulation Conference (WSC)10.1109/WSC48552.2020.9384024(2196-2207)Online publication date: 14-Dec-2020
      • (2020)Computer Simulation as Evaluation Tool of Information Systems: Identifying Quality Factors of Simulation Modeling2020 IEEE 22nd Conference on Business Informatics (CBI)10.1109/CBI49978.2020.00030(211-220)Online publication date: Jun-2020
      • Show More Cited By

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Get Access

      Login options

      Full Access

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media