Cited By
View all- Currie CMonks T(2021)A Practical Approach to Subset Selection for Multi-objective Optimization via SimulationACM Transactions on Modeling and Computer Simulation10.1145/346218731:4(1-15)Online publication date: 16-Aug-2021
Practical considerations in selecting the best set of simulated systems
Pages 2191 - 2200
Abstract
In many practical problems, simulation models are used to support complex decision-making processes comparing hundreds or thousands of solutions. These problems typically have a key objective but the final decision may be dependent on other factors, which cannot be incorporated into the simulation model. In such cases, decision-makers may request a short list of 'good' solutions, which work well for the main objective and satisfy one or more chance constraints. While fully sequential ranking and selection procedures can be effective at solving these problems, surveys of experimentation practice suggest that they are under-utilized, potentially due to difficulties automating commercial software. We develop an approach with just two stages of replications. The approach, which has been designed to cope with the use of common random numbers, draws on ideas from indifference zones and makes use of bootstrapping to find a subset of high quality solutions. A Python implementation is freely available.
References
[1]
Andradóttir, S., and S.-H. Kim. 2010. "Fully Sequential Procedures for Comparing Constrained Systems via Simulation". Naval Research Logistics 57:403--421.
[2]
Bechhofer, R. 1954. "A Single-Sample Multiple Decision Procedure for Ranking Means of Normal Populations with Known Variances". Annals of Mathematical Statistics 25:16--39.
[3]
Bekki, J. M., B. L. Nelson, and J. W. Fowler. 2010. "Bootstrapping-Based Fixed-Width Confidence Intervals for Ranking and Selection". In Proceedings of the 2010 Winter Simulation Conference, edited by B. Johansson et al., 1024--1033. Piscataway, New Jersey: IEEE.
[4]
Branke, J., S. E. Chick, and C. Schmidt. 2007. "Selecting a Selection Procedure". Management Science 53:1916--1932.
[5]
Chen, C.-H. 1996. "A Lower Bound for the Correct Subset-Selection Probability and its Application to Discrete Event Simulations.". IEEE Transactions on Automatic Control 41:1227--1231.
[6]
Cheng, R. C. 2017. Non-Standard Parametric Statistical Inference. 1st ed. Oxford, UK: Oxford University Press.
[7]
Cheng, R. C., and C. S. Currie. 2009. "Resampling Methods of Analysis in Simulation Studies". In Proceedings of the 2009 Winter Simulation Conference, edited by M. D. Rossetti et al., 45--59. Piscataway, New Jersey: IEEE.
[8]
Chernick, M. R. 2007. Bootstrap Methods: A Guide for Practitioners and Researchers. New York: John Wiley & Sons, Inc.
[9]
Chick, S. E., and K. Inoue. 2001. "New Two-Stage and Sequential Procedures for Selecting the Best Simulated System". Operations Research 49:732--743.
[10]
Chingcuanco, F., and C. Osorio. 2013. "A Procedure to Select the Best Subset Among Simulated Systems using Economic Opportunity Cost". In Proceedings of the 2013 Winter Simulation Conference, edited by R. Pasupathy et al., 452--462. Piscataway, New Jersey: IEEE.
[11]
Fu, M. C. 2015. Handbook of Simulation Optimization. 1st ed. New York, USA: Springer.
[12]
Gao, S., and W. Chen. 2015. "A Note on the Subset Selection for Simulation Optimization". In Proceedings of the 2015 Winter Simulation Conference, edited by L. Yilmaz et al., 3768--3776. Piscataway, New Jersey: IEEE.
[13]
Gao, S., and W. Chen. 2016. "A New Budget Allocation Framework for Selecting Top Simulated Designs". IIE Transactions 48:855--863.
[14]
Gupta, S. 1965. "On some Multiple Decision (Selection and Ranking) Rules". Technometrics 7:225--245.
[15]
Hong, L. J., J. Luo, and B. L. Nelson. 2015. "Chance Constrained Selection of the Best". INFORMS Journal on Computing 27:317--334.
[16]
Hunter, S. R., and R. Pasupathy. 2013. "Optimal Sampling Laws for Stochastically Constrained Simulation Optimization on Finite Sets". INFORMS Journal on Computing 25:527--542.
[17]
Kim, S.-H., and B. L. Nelson. 2006a. "On the Asymptotic Validity of Fully Sequential Selection Procedures for Steady-State Simulation". Operations Research 54:475--488.
[18]
Kim, S.-H., and B. L. Nelson. 2006b. "Selecting the Best System". In Elsevier Handbooks in Operations Research and Management Science: Simulation, edited by S. G. Henderson and B. Nelson. Waltham, MA: Elsevier.
[19]
Koenig, L., and A. Law. 1985. "A Procedure for Selecting a Subset of Size m Containing the l Best of k Independent Normal Populations". Communications in Statistics - Simulation and Computation 14:719--734.
[20]
Law, A. M., and W. D. Kelton. 2000. Simulation Modeling & Analysis. 3rd ed. New York: McGraw-Hill, Inc.
[21]
Lee, L. H., E. P. Chew, S. Teng, and D. Goldsman. 2010. "Finding the Non-Dominated Pareto Set for Multi-Objective Simulation Models". IIE Transactions 42:656--674.
[22]
Lee, S., and B. L. Nelson. 2014. "Bootstrap Ranking and Selection Revisited". In Proceedings of the 2014 Winter Simulation Conference, edited by A. Tolk et al., 3857--3868. Piscataway, New Jersey: IEEE.
[23]
Lee, S., and B. L. Nelson. 2015. "Computational Improvements in Bootstrap Ranking and Selection Procedures via Multiple Comparison with the Best". In Proceedings of the 2015 Winter Simulation Conference, edited by L. Yilmaz et al., 3758--3767. Piscataway, New Jersey: IEEE.
[24]
Lee, S., and B. L. Nelson. 2016. "General-Purpose Ranking and Selection for Computer Simulation". IIE Transactions 48:555--564.
[25]
Nelson, B. L., and F. Matejcik. 1995. "Using Common Random Numbers for Indifference-Zone Selection and Multiple Comparisons in Simulation". Management Science 41:1935--1945.
[26]
Pasupathy, R., S. R. Hunter, N. A. Pujowidianto, L. H. Lee, and C.-H. Chen. 2014. "Stochastically Constrained Ranking and Selection via SCORE". ACM Transactions on Modeling and Computer Simulation 25:1--26.
[27]
Penn, M., T. Monks, A. Kazmierska, and M. Alkoheji. 2018. "Designing and Redeveloping Generic Models in Healthcare.". In Proceedings of the OR Society Simulation Workshop 2018, edited by A. Anagnostou et al. Stratford-Upon-Avon, UK: OR Society.
[28]
Wang, Y., L. Luangkesorn, and L. J. Shuman. 2011. "Best-Subset Selection Procedure". In Proceedings of the 2011 Winter Simulation Conference, edited by S. Jain et al., 4315--4323. Piscataway, New Jersey: IEEE.
Recommendations
On selecting the best of k systems: an expository survey of subset-selection multinomial procedures
WSC '88: Proceedings of the 20th conference on Winter simulationThis paper studies the subset-selection approach of the ranking and selection procedures of choosing among k arbitrary systems or alternatives. Ranking and selection problems have customarily been treated using two different approaches, namely, the ...
New Two-Stage and Sequential Procedures for Selecting the Best Simulated System
Standard “indifference-zone” procedures that allocate computer resources to infer the best of a finite set of simulated systems are designed with a statistically conservative, least favorable configuration assumption consider the probability of correct ...
Selecting the Best Alternative Based on Its Quantile
A value-at-risk, or quantile, is widely used as an appropriate investment selection measure for risk-conscious decision makers. We present two quantile-based sequential procedures—with and without consideration of equivalency between alternatives—for ...
Comments
Information & Contributors
Information
Published In
Sponsors
Publisher
IEEE Press
Publication History
Published: 09 December 2018
Check for updates
Qualifiers
- Research-article
Conference
WSC '18
Sponsor:
Acceptance Rates
WSC '18 Paper Acceptance Rate 183 of 260 submissions, 70%;
Overall Acceptance Rate 3,413 of 5,075 submissions, 67%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 19Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)0
Reflects downloads up to 16 Oct 2024
Other Metrics
Citations
Cited By
View all- Currie CMonks T(2021)A Practical Approach to Subset Selection for Multi-objective Optimization via SimulationACM Transactions on Modeling and Computer Simulation10.1145/346218731:4(1-15)Online publication date: 16-Aug-2021
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in