research-article

Input uncertainty and indifference-zone ranking & selection

Authors:

Eunhye Song,

Barry L. Nelson,

L. Jeff HongAuthors Info & Claims

WSC '15: Proceedings of the 2015 Winter Simulation Conference

Pages 414 - 424

Published: 06 December 2015 Publication History

Get Access

Abstract

The indifference-zone (IZ) formulation of ranking and selection (R&S) is the foundation of many procedures that have been useful for choosing the best among a finite number of simulated alternatives. Of course, simulation models are imperfect representations of reality, which means that a simulation-based decision, such as choosing the best alternative, is subject to model risk. In this paper we explore the impact of model risk due to input uncertainty on IZ R&S. "Input uncertainty" is the result of having estimated ("fit") the simulation input models to observed real-world data. We find that input uncertainty may force the user to revise, or even abandon, their objectives when employing a R&S procedure, or it may have very little effect on selecting the best system even when the marginal input uncertainty is substantial.

References

[1]

Barton, R. R. 2012. "Tutorial: Input Uncertainty in Output Analysis". In Proceedings of the 2012 Winter Simulation Conference, edited by C. Laroque, J. Himmelspach, R. Pasupathy, O. Rose, and A. M. Uhrmacher, 1--12. Piscataway, NJ, USA: IEEE.

Digital Library

Google Scholar

[2]

Bechhofer, R. E. 1954. "A Single-Sample Multiple Decision Procedure for Ranking Means of Normal Populations with Known Variances". The Annals of Mathematical Statistics 25 (1): 16--39.

Crossref

Google Scholar

[3]

Bechhofer, R. E., D. M. Goldsman, and T. J. Santner. 1995. Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. New York: Wiley.

Google Scholar

[4]

Corlu, C. G., and B. Biller. 2013. "A Subset Selection Procedure under Input Parameter Uncertainty". In Proceedings of the 2013 Winter Simulation Conference, edited by R. Pasupathy, S.-H. Kim, A. Tolk, R. Hill, and M. E. Kuhl, 463--473. Piscataway, NJ, USA: IEEE.

Digital Library

Google Scholar

[5]

Fan, W., L. J. Hong, and X. Zhang. 2013. "Robust Selection of the Best". In Proceedings of the 2013 Winter Simulation Conference, edited by R. Pasupathy, S.-H. Kim, A. Tolk, R. Hill, and M. E. Kuhl, 868--876. Piscataway, NJ, USA: IEEE.

Digital Library

Google Scholar

[6]

Hong, L. J. 2006. "Fully Sequential Indifference-Zone Selection Procedures with Variance-Dependent Sampling". Naval Research Logistics 53 (5): 464--476.

Crossref

Google Scholar

[7]

Kim, S.-H., and B. L. Nelson. 2001. "A Fully Sequential Procedure for Indifference-zone Selection in Simulation". ACM Transactions on Modeling and Computer Simulation 11 (3): 251--273.

Digital Library

Google Scholar

[8]

Kim, S.-H., and B. L. Nelson. 2006. "Selecting the Best System". In Handbook in Operations Research and Management Science: Simulation, edited by S. G. Henderson and B. L. Nelson. Amsterdam: Elsevier.

Google Scholar

[9]

Kutner, M., C. Nachtsheim, J. Neter, and W. Li. 2004. Applied Linear Statistical Models. 5 ed. McGraw-Hill/Irwin.

Google Scholar

[10]

Nelson, B. L., and S. Banerjee. 2001. "Selecting a Good System: Procedures and Inference". IIE Transactions 33 (3): 149--166.

Crossref

Google Scholar

[11]

Nelson, B. L., and F. J. Matejcik. 1995. "Using Common Random Numbers for Indifference-Zone Selection and Multiple Comparisons in Simulation". Management Science 41 (12): 1935--1945.

Digital Library

Google Scholar

[12]

Song, E., B. L. Nelson, and C. D. Pegden. 2014. "Advanced Tutorial: Input Uncertainty Quantification". In Proceedings of the 2014 Winter Simulation Conference, edited by A. Tolk, S. Y. Diallo, I. O. Ryzhov, L. Yilmaz, S. Buckley, and J. A. Miller, 162--176. Piscataway, NJ, USA: IEEE.

Digital Library

Google Scholar

Cited By

View all

Chen Y(2023)Data-Driven Optimal Allocation for Ranking and Selection under Unknown Sampling DistributionsProceedings of the Winter Simulation Conference10.5555/3643142.3643424(3376-3387)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3643142.3643424
He LShanbhag USong E(2023)Stochastic Approximation for Multi-period Simulation Optimization with Streaming Input DataACM Transactions on Modeling and Computer Simulation10.1145/3617595Online publication date: 29-Aug-2023
https://dl.acm.org/doi/10.1145/3617595
Kim KKim TSong E(2021)Selection of the most probable best under input uncertaintyProceedings of the Winter Simulation Conference10.5555/3522802.3522951(1-12)Online publication date: 13-Dec-2021
https://dl.acm.org/doi/10.5555/3522802.3522951
Show More Cited By

Recommendations

Using common random numbers for indifference-zone selection
WSC '01: Proceedings of the 33nd conference on Winter simulation

This paper discusses the validity of using common random numbers (CRNs) with two-stage selection procedures to improve the possibility of correct selection and discusses the intrinsic subset pre-selection of the Enhanced Two-Stage Selection (ETSS) ...
Indifference-Zone-Free Selection of the Best

Many procedures have been proposed in the literature to select the simulated alternative with the best mean performance from a finite set of alternatives. Among these procedures, frequentist procedures are typically designed under either the subset-...
Indifference zone selection procedures: inferences from indifference-zone selection procedures
WSC '03: Proceedings of the 35th conference on Winter simulation: driving innovation

Two-stage indifference-zone selection procedures have been widely studied and applied. It is known that most indifference-zone selection procedures also guarantee multiple comparisons with the best confidence intervals with half-width corresponding to ...

Comments

Information & Contributors

Information

Published In

WSC '15: Proceedings of the 2015 Winter Simulation Conference

December 2015

4051 pages

ISBN:9781467397414

Publisher

IEEE Press

Publication History

Published: 06 December 2015

Check for updates

Qualifiers

Research-article

Conference

WSC '15

Sponsor:

SIGSIM

WSC '15: Winter Simulation Conference

December 6 - 9, 2015

California, Huntington Beach

Acceptance Rates

WSC '15 Paper Acceptance Rate 202 of 296 submissions, 68%;

Overall Acceptance Rate 3,413 of 5,075 submissions, 67%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
51
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Chen Y(2023)Data-Driven Optimal Allocation for Ranking and Selection under Unknown Sampling DistributionsProceedings of the Winter Simulation Conference10.5555/3643142.3643424(3376-3387)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3643142.3643424
He LShanbhag USong E(2023)Stochastic Approximation for Multi-period Simulation Optimization with Streaming Input DataACM Transactions on Modeling and Computer Simulation10.1145/3617595Online publication date: 29-Aug-2023
https://dl.acm.org/doi/10.1145/3617595
Kim KKim TSong E(2021)Selection of the most probable best under input uncertaintyProceedings of the Winter Simulation Conference10.5555/3522802.3522951(1-12)Online publication date: 13-Dec-2021
https://dl.acm.org/doi/10.5555/3522802.3522951
Xu JZheng ZGlynn P(2020)Joint resource allocation for input data collection and simulationProceedings of the Winter Simulation Conference10.5555/3466184.3466427(2126-2137)Online publication date: 14-Dec-2020
https://dl.acm.org/doi/10.5555/3466184.3466427
Wu DZhou ESon YHaas P(2019)Fixed confidence ranking and selection under input uncertaintyProceedings of the Winter Simulation Conference10.5555/3400397.3400699(3717-3727)Online publication date: 8-Dec-2019
https://dl.acm.org/doi/10.5555/3400397.3400699
Song EShanbhag USon YHaas P(2019)Stochastic approximation for simulation optimization under input uncertainty with streaming dataProceedings of the Winter Simulation Conference10.5555/3400397.3400689(3597-3608)Online publication date: 8-Dec-2019
https://dl.acm.org/doi/10.5555/3400397.3400689
Pearce MBranke J(2017)Bayesian simulation optimization with input uncertaintyProceedings of the 2017 Winter Simulation Conference10.5555/3242181.3242372(1-11)Online publication date: 3-Dec-2017
https://dl.acm.org/doi/10.5555/3242181.3242372
Lakshmanan SVenkateswaran J(2017)Robust simulation based optimization with input uncertaintyProceedings of the 2017 Winter Simulation Conference10.5555/3242181.3242371(1-11)Online publication date: 3-Dec-2017
https://dl.acm.org/doi/10.5555/3242181.3242371
Wu DZhou E(2017)Ranking and selection under input uncertaintyProceedings of the 2017 Winter Simulation Conference10.5555/3242181.3242370(1-12)Online publication date: 3-Dec-2017
https://dl.acm.org/doi/10.5555/3242181.3242370
Gao SXiao HZhou EChen WHuschka TChick S(2016)Optimal computing budget allocation with input uncertaintyProceedings of the 2016 Winter Simulation Conference10.5555/3042094.3042210(839-846)Online publication date: 11-Dec-2016
https://dl.acm.org/doi/10.5555/3042094.3042210

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

Using common random numbers for indifference-zone selection

Indifference-Zone-Free Selection of the Best

Indifference zone selection procedures: inferences from indifference-zone selection procedures

Comments

Published In

Sponsors

Publisher

Publication History

Check for updates

Qualifiers

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Recommendations

Using common random numbers for indifference-zone selection

Indifference-Zone-Free Selection of the Best

Indifference zone selection procedures: inferences from indifference-zone selection procedures

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations