Article

Free access

Some examples of simulation model validation using hypothesis testing

Authors:

Robert G. SargentAuthors Info & Claims

WSC '82: Proceedings of the 14th conference on Winter Simulation - Volume 2

Pages 621 - 629

Published: 06 December 1982 Publication History

Abstract

The use of hypothesis testing with cost-risk analysis is illustrated for simulation model validation by two examples. In the first example, Hotelling's two-sample T<sup>2</sup> test with cost-risk analysis is used for illustrating the validation of a multivariate response self-driven steady-state simulation model representing a single server M/M/1 queueing system. In the second example, the validation of a multivariate response trace-driven terminating simulation model representing an M/M/1 system is illustrated by the use of Hotelling's one-sample T<sup>2</sup> test with cost-risk analysis.

References

[1]

Aigner, D. J. (1972), "A Note on Verification of Computer Simulation Models," Management Science, Vol. 18, No. 11, pp. 615--619.

[2]

Balci, O. (1981), "Statistical Validation of Multivariate Response Simulation Models," Ph.D. Dissertation, Syracuse University.

[3]

Balci, O. and R. G. Sargent (1980), "Bibliography on Validation of Simulation Models," Newsletter - TIMS College on Simulation and Gaming, Vol. 4, No. 2, pp. 11--15.

[4]

Balci, O. and R. G. Sargent (1981a), "A Methodology for Cost-Risk Analysis in the Statistical Validation of Simulation Models," Communications of ACM, Vol. 24, No. 4, pp. 190--197.

Digital Library

[5]

Balci, O. and R. G. Sargent (1981b), "A Methodology for Validating Multivariate Response Simulation Models by Using Simultaneous Confidence Intervals," Working Paper #81--009, Dept. of I.E.&O.R., Syracuse University.

[6]

Balci, O. and R. G. Sargent (1982a), "Validation of Multivariate Response Simulation Models Using Hotelling's Two-Sample T2 Test," to appear in Simulation.

[7]

Balci, O. and R. G. Sargent (1982b), "Validation of Multivariate Response Trace-Driven Simulation Models," Working Paper #82-002, Dept. of I. E.&O. R., Syracuse University.

[8]

Cohen, K. J. and R. H. Cyert (1961), "Computer Models in Dynamic Economics, "Economics Quarterly, Vol. 75, pp. 112--127.

[9]

Cyert, R. M. (1966), "A Description and Evaluation of Some Firm Simulation," Proc. IBM Scientific Computing Symposium on Simulation Models and Gaming, IBM, White Plains, NY.

[10]

Damborg, M. J. and L. F. Fuller (1976), "Model Validation Using Time and Frequency Domain Error Measures," ERDA Report 76--152.

[11]

Emshoff, J. R. and R. L. Sisson (1970), Design and Use of Computer Simulation Models, MacMillan, New York.

[12]

Fishman, G. S. (1978), Principles of Discrete Event Simulation, Wiley-Interscience, New York.

Digital Library

[13]

Fishman, G. S. and P. J. Kiviat (1967), "The Analysis of Simulation Generated Time Series," Management Science, Vol. 13, No. 7, pp. 525--557.

Digital Library

[14]

Forrester, J. W. (1961), Industrial Dynamics, MIT Press, Cambridge.

[15]

Gafarian, A. V. and C. J. Ancker (1966), "Mean Value Estimation from Digital Computer Simulation," Operations Research, Vol. 14, pp. 25--44.

Digital Library

[16]

Gafarian, A. V. and J. E. Walsh (1969), "Statistical Approach for Validating Simulation Models by Comparison with Operational Systems," Proc. Fourth International Conference on Operations Research, John Wiley & Sons, New York, pp. 702--705.

[17]

Gallant, A. R., T. M. Gerig, and J. W. Evans (1974), "Time Series Realizations Obtained According to an Experimental Design," J. American Statistical Association, Vol. 69, No. 347, Applications Section.

[18]

Garratt, M. (1974), "Statistical Validation of Simulation Models," Proc. Summer Computer Simulation Conference, Houston, TX, pp. 915--926.

[19]

General Accounting Office (1979), "Guidelines for Model Evaluation," PAD-79-17, U. S.GAO, Washington, DC.

[20]

Hermann, C. F. (1967), "Validation Problems in Games and Simulations with Special Reference to Models of International Politics," Behavioral Science, Vol. 12, pp. 216--231.

[21]

Howrey, P. and H. H. Kelejian (1969), "Simulation Versus Analytical Solutions," in The Design of Computer Simulation Experiments, T. H. Naylor, Ed., Duke University Press, Durham, NC.

[22]

Hunt, A. W. (1970), "Statistical Evaluation and Verification of Digital Simulation Models Through Spectral Analysis," Ph. D. Dissertation, The University of Texas at Austin.

[23]

Kheir, N. A. and W. M. Holmes (1978), "On Validating Simulation Models of Missle Systems," Simulation, pp. 117--128.

[24]

Kobayashi, H. (1978), Modeling and Analysis: An Introduction to System Performance Evaluation Methodology, Addison-Wesley, Reading, MA.

[25]

Law, A. M. (1980), "Statistical Analysis of the Output Data from Terminating Simulations," Naval Research Logistics Quarterly, Vol. 27, pp. 131--143.

[26]

Miller, D. K. (1975), "Validation of Computer Simulations in the Social Sciences," Proc. Sixth Annual Conference on Modelling and Simulation, Pittsburg, PA, pp. 743--746.

[27]

Miller, D. R. (1974a), "Sensitivity Analysis and Validation of Simulation Models," J. Theor. Biol., Vol. 48, pp. 345--360.

[28]

Miller, D. R. (1974b), "Model Validation Through Sensitivity Analysis," Proc. Summer Computer Simulation Conference, Houston, TX, pp. 911--914.

[29]

Mitroff, I. I. (1969), "Fundamental Issues in the Simulation of Human Behavior: A Case in Strategy of Behavioral Science," Management Science, Vol. 15, No. 12, pp. B635--B649.

Digital Library

[30]

Naylor, T. H. and J. M. Finger (19@@@), verification of Computer Simulation Models," Management Science, Vol. 14, No. 2, pp. B-92-101.

[31]

Rowland, J. R. and W. M. Holmes (1978), "Simulation Validation with Sparse Random Data," Computers and Electrical Engineering, Vol. 5, No. 1, pp. 37--49.

[32]

Rubinstein, R. Y. (1981), Simulation and the Monte Carlo Method, John Wiley & Sons, New York.

Digital Library

[33]

Sargent, R. G. (1979), "An Introduction to Statistical Analysis of Simulation Output Data," Proc. NATO AGARD Symposium on Modeling and Simulation of Avionics Systems and Command, Control, and Communication Systems, (No. 268), Paris, France, pp. 3-1/3-13.

[34]

Sargent, R. G. (1979), "Validation of Simulation Models," Proc. Winter Simulation Conference, San Diego, CA, pp. 497--503.

Digital Library

[35]

Sargent, R. G. (1982), "Verification and Validation of Simulation Models," in Progress in Modeling and Simulation, F. E. Cellier, Ed., Academic, London.

[36]

Schellenberger, R. E. (1974), "Criteria for Assessing Model Validity for Managerial Purposes," Decision Sciences, Vol. 5, No. 4, pp. 644--653.

[37]

Schlesinger, S. et al. (1979), "Terminology for Model Credibility," Simulation, pp. 103--104.

[38]

Schruben, L. W. (1980), "Establishing the Credibility of Simulations," Simulation, Vol. 34, No. 3, pp. 101--105.

[39]

Shannon, R. E. (1975). Systems Simulation: The Art and Science, Prentice-Hall, New Jersey.

[40]

Sherman, S. W. (1976), "Trace-Driven Modeling: An Update," Proc. Symposium on the Simulation of Computer Systems IV, pp. 87--91.

Digital Library

[41]

Teorey, T. J. (1975), "Validation Criteria for Computer System Simulations," Simuletter, Vol. 6, No. 4, pp. 9--20; also published in Proc. Symposium on the Simulation of Computer Systems III, August 1975, pp. 161--169.

Digital Library

[42]

Theil, H. (1961), Economic Forecasts and Policy, North-Holland, Amsterdam.

[43]

Turing, A. M. (1963), "Computing Machinery and Intelligence," in Computers and Thought, E. A. Feigenbaum and J. Feldman, Eds., McGraw-Hill, New York, pp. 11--15.

Digital Library

[44]

Tytula, T. P. (1978), "A Method for Validating Missile System Simulation Models," U. S. Army Missile R&D Command, Technical Report E-78--11.

[45]

Van Horn, R. L. (1971), "Validation of Simulation Results," Management Science, Vol. 17, No. 5, pp. 247--258.

Digital Library

[46]

Watts, D. (1969), "Time Series Analysis," in The Design of Computer Simulation Experiments, T. H. Naylor, Ed., Duke University Press, Durham, NC, pp. 165--179.

[47]

Wright, R. D. (1972), "Validating Dynamic Models: An Evaluation of Tests of Predictive Power," Proc. Summer Computer Simulation Conference, San Diego, CA, pp. 1286--1296.

Cited By

Bai YLam H(2020)Calibrating input parameters via eligibility setsProceedings of the Winter Simulation Conference10.5555/3466184.3466425(2114-2125)Online publication date: 14-Dec-2020
https://dl.acm.org/doi/10.5555/3466184.3466425
Lam HZhang XPlumlee M(2017)Improving prediction from stochastic simulation via model discrepancy learningProceedings of the 2017 Winter Simulation Conference10.5555/3242181.3242331(1-12)Online publication date: 3-Dec-2017
https://dl.acm.org/doi/10.5555/3242181.3242331
Sargent RBalci O(2017)History of verification and validation of simulation modelsProceedings of the 2017 Winter Simulation Conference10.5555/3242181.3242199(1-16)Online publication date: 3-Dec-2017
https://dl.acm.org/doi/10.5555/3242181.3242199
Show More Cited By

Some examples of simulation model validation using hypothesis testing
1. Computing methodologies
  1. Modeling and simulation
2. Mathematics of computing

Recommendations

Bootstrap hypothesis testing for some common statistical problems: A critical evaluation of size and power properties

The construction of bootstrap hypothesis tests can differ from that of bootstrap confidence intervals because of the need to generate the bootstrap distribution of test statistics under a specific null hypothesis. Similarly, bootstrap power calculations ...
Asymptotics for testing hypothesis in some multivariate variance components model under non-normality

We consider the problem of deriving the asymptotic distribution of the three commonly used multivariate test statistics, namely likelihood ratio, Lawley-Hotelling and Bartlett-Nanda-Pillai statistics, for testing hypotheses on the various effects (main, ...
A semiparametric hypothesis testing procedure for the ROC curve area under a density ratio model

We propose a semiparametric Wald statistic to test whether a diagnostic test is capable of discriminating between diseased and nondiseased subjects based on the receiver operating characteristic (ROC) curve area under a two-sample semiparametric density ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WSC '82: Proceedings of the 14th conference on Winter Simulation - Volume 2

December 1982

340 pages

General Chair:
Yen W. Chao
FEDSIM/MV, Washington, D.C.
,
Program Chair:
Orlando S. Madrigal
California State University, Chico, CA

Sponsors

IIE: Institute of Industrial Engineers
NBS: National Bureau of Standards
ORSA: Operations Research Society of America
TIMS/CSG: The Institute of Management Science College on Simulation and Gaming
SIGSIM: ACM Special Interest Group on Simulation and Modeling
IEEE-CS: Computer Society
IEEE-SMCS: Systems, Man & Cybernetics Society
SCS: The Society for Computer Simulation International

Publisher

Winter Simulation Conference

Publication History

Published: 06 December 1982

Check for updates

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 3,413 of 5,075 submissions, 67%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
1,075
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bai YLam H(2020)Calibrating input parameters via eligibility setsProceedings of the Winter Simulation Conference10.5555/3466184.3466425(2114-2125)Online publication date: 14-Dec-2020
https://dl.acm.org/doi/10.5555/3466184.3466425
Lam HZhang XPlumlee M(2017)Improving prediction from stochastic simulation via model discrepancy learningProceedings of the 2017 Winter Simulation Conference10.5555/3242181.3242331(1-12)Online publication date: 3-Dec-2017
https://dl.acm.org/doi/10.5555/3242181.3242331
Sargent RBalci O(2017)History of verification and validation of simulation modelsProceedings of the 2017 Winter Simulation Conference10.5555/3242181.3242199(1-16)Online publication date: 3-Dec-2017
https://dl.acm.org/doi/10.5555/3242181.3242199
Plumlee MLam HHuschka TChick S(2016)Learning stochastic model discrepancyProceedings of the 2016 Winter Simulation Conference10.5555/3042094.3042159(413-424)Online publication date: 11-Dec-2016
https://dl.acm.org/doi/10.5555/3042094.3042159
Sargent RGoldsman DYaacoub T(2015)Use of the interval statistical procedure for simulation model validationProceedings of the 2015 Winter Simulation Conference10.5555/2888619.2888625(60-72)Online publication date: 6-Dec-2015
https://dl.acm.org/doi/10.5555/2888619.2888625
Sargent RYücesan E(2010)Verification and validation of simulation modelsProceedings of the Winter Simulation Conference10.5555/2433508.2433528(166-183)Online publication date: 5-Dec-2010
https://dl.acm.org/doi/10.5555/2433508.2433528
Sargent RDunkin AIngalls R(2009)Verification and validation of simulation modelsWinter Simulation Conference10.5555/1995456.1995484(162-176)Online publication date: 13-Dec-2009
https://dl.acm.org/doi/10.5555/1995456.1995484
Sargent RJefferson TFowler JIngalls R(2008)Verification and validation of simulation modelsProceedings of the 40th Conference on Winter Simulation10.5555/1516744.1516780(157-169)Online publication date: 7-Dec-2008
https://dl.acm.org/doi/10.5555/1516744.1516780
Sargent RTew J(2007)A tutorial on verification and validation of simulation models (1984)Proceedings of the 39th conference on Winter simulation: 40 years! The best is yet to come10.5555/1351542.1352004(1-8)Online publication date: 9-Dec-2007
https://dl.acm.org/doi/10.5555/1351542.1352004
Sargent RTew J(2007)Verification and validation of simulation modelsProceedings of the 39th conference on Winter simulation: 40 years! The best is yet to come10.5555/1351542.1351570(124-137)Online publication date: 9-Dec-2007
https://dl.acm.org/doi/10.5555/1351542.1351570
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents