research-article

On the Marginal Standard Error Rule and the Testing of Initial Transient Deletion Methods

Authors:

Peter W. GlynnAuthors Info & Claims

ACM Transactions on Modeling and Computer Simulation (TOMACS), Volume 27, Issue 1

Article No.: 1, Pages 1 - 30

https://doi.org/10.1145/2961052

Published: 02 August 2016 Publication History

Abstract

In the planning of steady-state simulations, a central issue is the initial transient problem, in which an initial segment of the simulation output is adversely contaminated by initialization bias. Our article makes several contributions toward the analysis of this computational challenge. To begin, we introduce useful ways for measuring the magnitude of the initial transient effect in the single replication setting. We then analyze the marginal standard error rule (MSER) and prove that MSER’s deletion point is determined, as the simulation time horizon tends to infinity, by the minimizer of a certain random walk. We use this insight, together with fluid limit intuition associated with queueing models, to generate two nonpathological examples in which at least one variant of MSER fails to accurately predict the duration of the initial transient. Our results suggest that the efficacy of a deletion procedure is sensitive to the choice of performance measure, and that the set of standard test problems on which initial transient procedures are tested should be significantly broadened.

References

[1]

V. Anantharam. 1988. How large delays build up in a GI/G/1 queue. Queueing Systems 5, 345--368.

Digital Library

[2]

B. Avi-Itzhak, W. L. Maxwell, and L. W. Miller. 1965. Queueing with alternating priorities. Operations Research 13, 2, 306--318.

Digital Library

[3]

S. Axler. 1997. Linear Algebra Done Right (2nd ed.). Springer-Verlag, New York, NY.

[4]

R. Bhattacharya, M. Majumdar, and N. Hashimzade. 2010. Limit theorems for monotone Markov processes. Sankhya: The Indian Journal of Statistics 72, 1, 170--190.

[5]

M. Bramson. 2008. Stability of queueing networks. Probability Surveys 5, 169--345.

[6]

E. Cinlar. 1975. Introduction to Stochastic Processes. Dover Publications Inc., Mineola, New York.

[7]

J. G. Dai. 1996. A fluid limit model criterion for instability of multiclass queueing networks. Annals of Applied Probability 6, 3, 751--757.

[8]

A. DasGupta. 2011. Probability for Statistics and Machine Learning: Fundamentals and Advanced Topics. Springer, New York, NY.

Digital Library

[9]

W. W. Franklin and K. P. White Jr. 2008. Stationarity tests and MSER-5: Exploring the intuition behind mean-squared-error-reduction in detecting and correcting initialization bias. In Proceedings of the 2008 Winter Simulation Conference. 541--546.

Digital Library

[10]

W. W. Franklin and K. P. White Jr. 2010. Parametric expression for MSER with geometrically decaying bias. In Proceedings of the 2010 Winter Simulation Conference. 957--964.

Digital Library

[11]

P. W. Glynn. 1989. A GSMP formalism for discrete event systems. Proceedings of the IEEE 77, 1, 14--23.

[12]

Winfried K. Grassmann. 2011. Rethinking the initialization bias problem in steady-state discrete event simulation. In Proceedings of the 2011 Winter Simulation Conference. 593--599.

Digital Library

[13]

K. Hoad and S. Robinson. 2011. Implementing MSER-5 in commercial simulation software and its wider implications. In Proceedings of the 2011 Winter Simulation Conference. 495--503.

Digital Library

[14]

K. Hoad, S. Robinson, and R. Davies. 2010. Automating warm-up length estimation. Journal of the Operational Research Society 61, 1389--1403.

[15]

E. K. Lada, N. M. Steiger, and J. R. Wilson. 2006. Performance evaluation of recent procedures for steady-state simulation analysis. IIE Transactions 38, 9, 711--727.

[16]

A. M. Law. 2015. Simulation Modeling and Analysis (5th ed.). McGraw-Hill, New York, NY.

[17]

S. Meyn and R. L. Tweedie. 2009. Markov Chains and Stochastic Stability (2nd ed.). Cambridge University Press, Cambridge, UK.

Digital Library

[18]

A. C. Mokashi, J. J. Tejada, S. Yousefi, T. Xu, J. R. Wilson, A. Tafazzoli, and N.M. Steiger. 2010. Performance comparison of MSER-5 and N-Skart on the simulation start-up problem. In Proceedings of the 2010 Winter Simulation Conference. 971--982.

Digital Library

[19]

R. Pasupathy and B. Schmeiser. 2010. The initial transient in steady-state point estimation: Contexts, a bibliography, the MSE criterion, and the MSER statistic. In Proceedings of the 2010 Winter Simulation Conference. 184--197.

Digital Library

[20]

F. G. Rahimov and V. A. Abdurakhmanov. 2007. On limit behavior of linear first passage time of the Markov chain. Proceedings of the Institute of Mathematics and Mechanics of the National Academy of Sciences of Azerbaijan 27, 69--74.

[21]

S. Robinson. 2002. New simulation output analysis techniques: A statistical process control approach for estimating the warm-up period. In Proceedings of the 2002 Winter Simulation Conference. 439--446.

Digital Library

[22]

L. W. Schruben. 1982. Detecting initialization bias in simulation output. Operations Research 30, 3, 569--590.

Digital Library

[23]

L. W. Schruben, H. Singh, and L. Tierney. 1983. Optimal tests for initialization bias in simulation output. Operations Research 71, 6, 1167--1178.

Digital Library

[24]

A. Tafazzoli, N. M. Steiger, and J. R. Wilson. 2011. N-Skart: A nonsequential skewness- and autoregression-adjusted batch-means procedure for simulation analysis. IEEE Transactions on Automatic Control 56, 2, 254--264.

[25]

R. J. Wang and P. W. Glynn. 2014. Measuring the initial transient: Reflected Brownian motion. In Proceedings of the 2014 Winter Simulation Conference. 652--661.

Digital Library

[26]

R. J. Wang and P. W. Glynn. 2016. On the rate of convergence to equilibrium for reflected Brownian motion. In preparation.

[27]

K. P. White Jr. 1997. An effective truncation heuristic for bias reduction in simulation output. Simulation 69, 6, 323--334.

Digital Library

[28]

K. P. White Jr., M. J. Cobb, and S. C. Spratt. 2000. A comparison of five steady-state truncation heuristics for simulation. In Proceedings of the 2000 Winter Simulation Conference. 755--760.

Digital Library

[29]

W. Whitt. 2002. Stochastic-Process Limits. Springer-Verlag, New York, NY.

[30]

W. Whitt. 2006. Analysis for design. In Handbooks in Operations Research and Management Science, S. G. Henderson and B. L. Nelson (Eds.). North-Holland, Amsterdam, Netherlands.

[31]

J. R. Wilson and A. A. B. Pritsker. 1978. A survey of research on the simulation startup problem. Simulation 31, 2, 55--58.

[32]

E. M. M. Winands, I. J. B. F. Adan, and G. van Houtum. 2006. Mean value analysis for polling systems. Queueing Systems 54, 35--44.

Digital Library

Cited By

Lin SJia LZhang HWang Y(2021)A method for assessing resilience of high-speed EMUs considering a network-based system topology and performance dataProceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability10.1177/1748006X211004515(1748006X2110045)Online publication date: 19-Mar-2021
https://doi.org/10.1177/1748006X211004515
Alexopoulos CGoldsman DMokashi ATien KWilson J(2019)Sequest: A Sequential Procedure for Estimating Quantiles in Steady-State SimulationsOperations Research10.1287/opre.2018.1829Online publication date: 28-Jun-2019
https://doi.org/10.1287/opre.2018.1829
Glynn PWang R(2019)On the rate of convergence to equilibrium for two-sided reflected Brownian motion and for the Ornstein---Uhlenbeck processQueueing Systems: Theory and Applications10.1007/s11134-018-9591-091:1-2(1-14)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1007/s11134-018-9591-0
Show More Cited By

Index Terms

On the Marginal Standard Error Rule and the Testing of Initial Transient Deletion Methods
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation evaluation

Recommendations

Analysis of initial transient deletion for replicated steady-state simulations

Consider the method of independent replications with initial transient deletion for generating confidence intervals for 'steady-state' quantities. To produce intervals with good convergence characteristics, the relative growth rates of the number of ...
Fluid and diffusion limits for transient sojourn times of processor sharing queues with time varying rates

We provide an approximate analysis of the transient sojourn time for a processor sharing queue with time varying arrival and service rates, where the load can vary over time, including periods of overload. Using the same asymptotic technique as uniform ...
On the asymptotic optimality of the cμ/θ rule under ergodic cost

We consider an overloaded multi-server multi-class queueing model where customers may abandon while waiting to be served. For class i, service is provided at rate μ _i , and abandonment occurs at rate θ _i . In a many-server fluid regime, we show that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation

ACM Transactions on Modeling and Computer Simulation Volume 27, Issue 1

January 2017

150 pages

ISSN:1049-3301

EISSN:1558-1195

DOI:10.1145/2982568

Editor:
Adelinde M. Uhrmacher
University of Rostock, Germany

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 August 2016

Accepted: 01 February 2016

Revised: 01 September 2015

Received: 01 May 2013

Published in TOMACS Volume 27, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Arvanitidis Stanford Graduate Fellowship (SGF)
NSERC Postgraduate Scholarship (PGS D)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
222
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)2

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lin SJia LZhang HWang Y(2021)A method for assessing resilience of high-speed EMUs considering a network-based system topology and performance dataProceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability10.1177/1748006X211004515(1748006X2110045)Online publication date: 19-Mar-2021
https://doi.org/10.1177/1748006X211004515
Alexopoulos CGoldsman DMokashi ATien KWilson J(2019)Sequest: A Sequential Procedure for Estimating Quantiles in Steady-State SimulationsOperations Research10.1287/opre.2018.1829Online publication date: 28-Jun-2019
https://doi.org/10.1287/opre.2018.1829
Glynn PWang R(2019)On the rate of convergence to equilibrium for two-sided reflected Brownian motion and for the Ornstein---Uhlenbeck processQueueing Systems: Theory and Applications10.1007/s11134-018-9591-091:1-2(1-14)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1007/s11134-018-9591-0
Alexopoulos CGoldsman DMokashi AWilson JJohansson BJain S(2018)Sequential estimation of steady-state quantilesProceedings of the 2018 Winter Simulation Conference10.5555/3320516.3320736(1814-1825)Online publication date: 9-Dec-2018
https://dl.acm.org/doi/10.5555/3320516.3320736
Perez IHodge DKypraios T(2018)Auxiliary variables for Bayesian inference in multi-class queueing networksStatistics and Computing10.1007/s11222-017-9787-x28:6(1187-1200)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1007/s11222-017-9787-x
Whitt W(2018)A broad view of queueing theory through one issueQueueing Systems: Theory and Applications10.1007/s11134-018-9580-389:1-2(3-14)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s11134-018-9580-3
Glynn PWang R(2018)On the rate of convergence to equilibrium for reflected Brownian motionQueueing Systems: Theory and Applications10.1007/s11134-018-9574-189:1-2(165-197)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s11134-018-9574-1
Alexopoulos CKelton W(2017)A concise history of simulation output analysisProceedings of the 2017 Winter Simulation Conference10.5555/3242181.3242191(1-16)Online publication date: 3-Dec-2017
https://dl.acm.org/doi/10.5555/3242181.3242191
Alexopoulos CGoldsman DMokashi AWilson J(2017)Automated Estimation of Extreme Steady-State Quantiles via the Maximum TransformationACM Transactions on Modeling and Computer Simulation10.1145/312286427:4(1-29)Online publication date: 14-Nov-2017
https://dl.acm.org/doi/10.1145/3122864
Alexopoulos CDavid Kelton W(2017)A concise history of simulation output analysis2017 Winter Simulation Conference (WSC)10.1109/WSC.2017.8247786(115-130)Online publication date: Dec-2017
https://doi.org/10.1109/WSC.2017.8247786

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents