article

On Nonterminating Stochastic Games

Authors:

R. M. KarpAuthors Info & Claims

Management Science, Volume 12, Issue 5

Pages 359 - 370

https://doi.org/10.1287/mnsc.12.5.359

Published: 01 January 1966 Publication History

Abstract

A stochastic game is played in a sequence of steps; at each step the play is said to be in some state i, chosen from a finite collection of states. If the play is in state i, the first player chooses move k and the second player chooses move l, then the first player receives a reward a^kl_i, and, with probability p^kl_ij, the next state is j.

The concept of stochastic games was introduced by Shapley with the proviso that, with probability 1, play terminates. The authors consider the case when play never terminates, and show properties of such games and offer a convergent algorithm for their solution. In the special case when one of the players is a dummy, the nonterminating stochastic game reduces to a Markovian decision process, and the present work can be regarded as the extension to a game theoretic context of known results on Markovian decision processes.

References

[1]

AUMANN, R., "Mixed and Behavior Strategies in Infinite Extensive Games," Advances in Game Theory, Annals of Mathematics Studies Number 52, Princeton, 1964, pp. 627-650.

[2]

DERMAN, C. "On Sequential Decisions and Markov Chains," Management Science, 9, October 1962, pp. 16-24.

[3]

D'ÉPENOUX, F., "A Probabilistic Production and Inventory Problem," Management Science, 10, October 1963, pp. 98-108.

[4]

D'ÉPENOUX, F., "Sur un Probleme de Production de Stoekage dans l'Aleatoire," Revue Française de Recherche Opérationnelle, No. 14, 1960.

[5]

EVERETT, H., "Recursive Games," Contributions to the Theory of Games, Vol. III, 47-78, Princeton, 1957.

[6]

FREIMER, M., "On Solving a Markovian Decision Problem by Linear Programming," (Unpublished paper, Institute for Defense Analyses, Cambridge, Mass., 12 Dec. 1961).

[7]

DE GHELLINK, G., "Les Problèmes de Décisions Séquentielles," Cahiers de Centre d'Étude de Recherche Opérationnelle, 2, Brussels, 1960.

[8]

GILLETTE, D., "Stochastic Games with Zero Stop Probabilities," Contributions to the Theory of Games, Vol. III, 179-187, Princeton, 1957.

[9]

HOFFMAN, A. J., "On Approximate Solutions of Systems of Linear Inequalities," Journal of Research of the National Bureau of Standards, 49, October 1952, pp. 263-265.

[10]

HOWARD, R., Dynamic Programming and Markov Processes, Technology Press and Wiley, 1960.

[11]

KUHN, H., "Extensive Games and the Problem of Information," Annals of Mathematics Studies Number 28, Princeton, 1953, pp. 193-216.

[12]

MANNE, A., "Linear Programming and Sequential Decisions," Management Science, 6, April, 1960, pp. 259-267,

Digital Library

[13]

SHAPLEY, L., "Stochastic Games," Proc. Nat. Acad. Sci., U.S.A., 39, pp. 1095-1100.

[14]

SIMONNARD, M., Programmation Liniaire, Dunod, Paris, 1962.

[15]

WILLIAMS, A., "Marginal Values in Linear Programming," Journal of the Society for Industrial and Applied Mathematics, 11, March, 1963, pp. 82-94.

[16]

WOLFE, P. AND DANTZIG, G., "Linear Programming in a Markov Chain," Operations Research, 10, Sept.-Oct., 1962, pp. 702-710.

Digital Library

Cited By

Eisentraut JKelmendi EKřetínský JWeininger M(2022)Value iteration for simple stochastic gamesInformation and Computation10.1016/j.ic.2022.104886285:PBOnline publication date: 15-Jun-2022
https://dl.acm.org/doi/10.1016/j.ic.2022.104886
Křetínský JRamneantu ESlivinskiy AWeininger M(2022)Comparison of algorithms for simple stochastic gamesInformation and Computation10.1016/j.ic.2022.104885289:PBOnline publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1016/j.ic.2022.104885
Yu ZGuo XXia L(2022)Zero-sum semi-Markov games with state-action-dependent discount factorsDiscrete Event Dynamic Systems10.1007/s10626-022-00366-432:4(545-571)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1007/s10626-022-00366-4
Show More Cited By

Index Terms

On Nonterminating Stochastic Games

Index terms have been assigned to the content through auto-classification.

Recommendations

Perfect information stochastic priority games
ICALP'07: Proceedings of the 34th international conference on Automata, Languages and Programming

We introduce stochastic priority games -- a new class of perfect information stochastic games. These games can take two different, but equivalent, forms. In stopping priority games a play can be stopped by the environment after a finite number of stages,...
Stochastic o-regular games
The complexity of stochastic Müller games

The theory of graph games with @w-regular winning conditions is the foundation for modeling and synthesizing reactive processes. In the case of stochastic reactive processes, the corresponding stochastic graph games have three players, two of them (...

Comments

Information & Contributors

Information

Published In

cover image Management Science

Management Science Volume 12, Issue 5

January 1966

178 pages

ISSN:0025-1909

Issue’s Table of Contents

Publisher

INFORMS

Linthicum, MD, United States

Publication History

Published: 01 January 1966

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

55
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Eisentraut JKelmendi EKřetínský JWeininger M(2022)Value iteration for simple stochastic gamesInformation and Computation10.1016/j.ic.2022.104886285:PBOnline publication date: 15-Jun-2022
https://dl.acm.org/doi/10.1016/j.ic.2022.104886
Křetínský JRamneantu ESlivinskiy AWeininger M(2022)Comparison of algorithms for simple stochastic gamesInformation and Computation10.1016/j.ic.2022.104885289:PBOnline publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1016/j.ic.2022.104885
Yu ZGuo XXia L(2022)Zero-sum semi-Markov games with state-action-dependent discount factorsDiscrete Event Dynamic Systems10.1007/s10626-022-00366-432:4(545-571)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1007/s10626-022-00366-4
Cvetković AProtasov V(2022)The greedy strategy for optimizing the Perron eigenvalueMathematical Programming: Series A and B10.1007/s10107-020-01585-z193:1(1-31)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1007/s10107-020-01585-z
Azeem MEvangelidis AKřetínský JSlivinskiy AWeininger M(2022)Optimistic and Topological Value Iteration for Simple Stochastic GamesAutomated Technology for Verification and Analysis10.1007/978-3-031-19992-9_18(285-302)Online publication date: 25-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-19992-9_18
Sayin MZhang KLeslie DBaşar TOzdaglar ARanzato MBeygelzimer ADauphin YLiang PVaughan J(2021)Decentralized Q-learning in zero-sum Markov gamesProceedings of the 35th International Conference on Neural Information Processing Systems10.5555/3540261.3541662(18320-18334)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.5555/3540261.3541662
Chatterjee KKatoen JWeininger MWinkler T(2020)Stochastic Games with Lexicographic Reachability-Safety ObjectivesComputer Aided Verification10.1007/978-3-030-53291-8_21(398-420)Online publication date: 21-Jul-2020
https://dl.acm.org/doi/10.1007/978-3-030-53291-8_21
Phalakarn KTakisaka THaas THasuo I(2020)Widest Paths and Global Propagation in Bounded Value Iteration for Stochastic GamesComputer Aided Verification10.1007/978-3-030-53291-8_19(349-371)Online publication date: 21-Jul-2020
https://dl.acm.org/doi/10.1007/978-3-030-53291-8_19
Akian MGaubert SGrand-Clément JGuillaud J(2019)The Operator Approach to Entropy GamesTheory of Computing Systems10.1007/s00224-019-09925-z63:5(1089-1130)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1007/s00224-019-09925-z
Adjé AFrehse GMitra S(2017)Coupling Policy Iterations with Piecewise Quadratic Lyapunov FunctionsProceedings of the 20th International Conference on Hybrid Systems: Computation and Control10.1145/3049797.3049825(143-152)Online publication date: 13-Apr-2017
https://dl.acm.org/doi/10.1145/3049797.3049825
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents