research-article

Stochastic Shortest Paths and Weight-Bounded Properties in Markov Decision Processes

Authors:

Christel Baier,

Nathalie Bertrand,

Clemens Dubslaff,

Ocan SankurAuthors Info & Claims

LICS '18: Proceedings of the 33rd Annual ACM/IEEE Symposium on Logic in Computer Science

Pages 86 - 94

https://doi.org/10.1145/3209108.3209184

Published: 09 July 2018 Publication History

Abstract

The paper deals with finite-state Markov decision processes (MDPs) with integer weights assigned to each state-action pair. New algorithms are presented to classify end components according to their limiting behavior with respect to the accumulated weights. These algorithms are used to provide solutions for two types of fundamental problems for integer-weighted MDPs. First, a polynomial-time algorithm for the classical stochastic shortest path problem is presented, generalizing known results for special classes of weighted MDPs. Second, qualitative probability constraints for weight-bounded (repeated) reachability conditions are addressed. Among others, it is shown that the problem to decide whether a disjunction of weight-bounded reachability conditions holds almost surely under some scheduler belongs to NP ∩ coNP, is solvable in pseudo-polynomial time and is at least as hard as solving two-player mean-payoff games, while the corresponding problem for universal quantification over schedulers is solvable in polynomial time.

References

[1]

Parosh Aziz Abdulla, Radu Ciobanu, Richard Mayr, Arnaud Sangnier, and Jeremy Sproston. Qualitative analysis of VASS-induced MDPs. In FoSSaCS'16, LNCS 9634, p. 319--334. Springer, 2016.

[2]

Christel Baier, Nathalie Bertrand, Clemens Dubslaff, Daniel Gburek, and Ocan Sankur. Stochastic shortest paths and weight-bounded properties in markov decision processes (extended version). https://arxiv.org/abs/1804.11301, 2018.

[3]

Christel Baier, Marcus Daum, Clemens Dubslaff, Joachim Klein, and Sascha Klüppelholz. Energy-utility quantiles. In NFM'14, LNCS 8430, p. 285--299. Springer, 2014.

Digital Library

[4]

Christel Baier and Joost-Pieter Katoen. Principles of Model Checking. MIT Press, 2008.

Digital Library

[5]

Dimitri P. Bertsekas and John N. Tsitsiklis. An analysis of stochastic shortest path problems. Mathematics of Operations Research, 16(3):580--595, 1991.

Digital Library

[6]

Tomás Brázdil, Václav Brozek, Kousha Etessami, Antonín Kucera, and Dominik Wojtczak. One-counter Markov decision processes. In SODA'10, p. 863--874. SIAM, 2010.

Digital Library

[7]

Tomás Brázdil, Antonín Kucera, and Petr Novotný. Optimizing the expected mean payoff in energy Markov decision processes. In ATVA'16, LNCS 9938, p. 32--49, 2016.

[8]

Véronique Bruyère, Emmanuel Filiot, Mickael Randour, and Jean-François Raskin. Meet your expectations with guarantees: Beyond worst-case synthesis in quantitative games. Information and Computation, 254:259--295, 2017.

[9]

Krishnendu Chatterjee and Laurent Doyen. Energy and mean-payoff parity Markov decision processes. In MFCS'11, LNCS 6907, p. 206--218. Springer, 2011.

Digital Library

[10]

Krishnendu Chatterjee and Monika Henzinger. Faster and dynamic algorithms for maximal end-component decomposition and related graph problems in probabilistic verification. In SODA'11, p. 1318--1336. SIAM, 2011.

Digital Library

[11]

Luca de Alfaro. Formal Verification of Probabilistic Systems. PhD thesis, Stanford University, Department of Computer Science, 1997.

[12]

Luca de Alfaro. Computing minimum and maximum reachability times in probabilistic systems. In CONCUR'99, LNCS 1664, p. 66--81, 1999.

Digital Library

[13]

Kousha Etessami, Dominik Wojtczak, and Mihalis Yannakakis. Quasi-birth-death processes, tree-like qbds, probabilistic 1-counter automata, and pushdown systems. In QEST'08, p. 243--253. IEEE Computer Society, 2008.

Digital Library

[14]

Christoph Haase and Stefan Kiefer. The odds of staying on budget. In ICALP'15, LNCS 9135, p. 234--246. Springer, 2015.

Digital Library

[15]

Lodewijk Kallenberg. Markov Decision Processes. Lecture Notes. University of Leiden, 2011.

[16]

Daniel Krähmann, Jana Schubert, Christel Baier, and Clemens Dubslaff. Ratio and weight quantiles. In MFCS'15, LNCS 9234, p. 344--356. Springer, 2015.

[17]

Richard Mayr, Sven Schewe, Patrick Totzke, and Dominik Wojtczak. MDPs with energy-parity objectives. In LICS'17, IEEE Computer Society, IEEE Computer Society, p. 1--12, 2017.

[18]

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., New York, NY, 1994.

[19]

Michael Ummels and Christel Baier. Computing quantiles in Markov reward models. In FoSSaCS'13, LNCS 7794, p. 353--368. Springer, 2013.

Digital Library

Cited By

Dubslaff CKöhl M(2022)Configurable-by-Construction Runtime MonitoringLeveraging Applications of Formal Methods, Verification and Validation. Verification Principles10.1007/978-3-031-19849-6_14(220-241)Online publication date: 22-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-19849-6_14
Castro PD’Argenio PDemasi RPutruele L(2022)Playing Against Fair Adversaries in Stochastic Games with Total RewardsComputer Aided Verification10.1007/978-3-031-13188-2_3(48-69)Online publication date: 7-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-13188-2_3
Baier CFunke FPiribauer JZiemek R(2022)On probability-raising causality in Markov decision processesFoundations of Software Science and Computation Structures10.1007/978-3-030-99253-8_3(40-60)Online publication date: 4-Apr-2022
https://dl.acm.org/doi/10.1007/978-3-030-99253-8_3
Show More Cited By

Recommendations

The Complexity of Markov Decision Processes

We investigate the complexity of the classical problem of optimal policy computation in Markov decision processes. All three variants of the problem finite horizon, infinite horizon discounted, and infinite horizon average cost were known to be solvable ...
Continuous-Time Markov Decision Processes with Discounted Rewards: The Case of Polish Spaces

This paper deals with continuous-time Markov decision processes in Polish spaces, under an expected discounted reward criterion. The transition rates of underlying continuous-time jump Markov processes are allowed to be unbounded, and the reward rates ...
Markov Decision Processes with Sample Path Constraints: The Communicating Case

We consider time-average Markov Decision Processes MDPs, which accumulate a reward and cost at each decision epoch. A policy meets the sample-path constraint if the time-average cost is below a specified value with probability one. The optimization ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

LICS '18: Proceedings of the 33rd Annual ACM/IEEE Symposium on Logic in Computer Science

July 2018

960 pages

ISBN:9781450355834

DOI:10.1145/3209108

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGLOG: ACM Special Interest Group on Logic and Computation
EACSL: European Association for Computer Science Logic
IEEE-CS\DATC: IEEE Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 July 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

LICS '18

Sponsor:

SIGLOG
EACSL
IEEE-CS\DATC

LICS '18: 33rd Annual ACM/IEEE Symposium on Logic in Computer Science

July 9 - 12, 2018

Oxford, United Kingdom

Acceptance Rates

Overall Acceptance Rate 215 of 622 submissions, 35%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
175
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dubslaff CKöhl M(2022)Configurable-by-Construction Runtime MonitoringLeveraging Applications of Formal Methods, Verification and Validation. Verification Principles10.1007/978-3-031-19849-6_14(220-241)Online publication date: 22-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-19849-6_14
Castro PD’Argenio PDemasi RPutruele L(2022)Playing Against Fair Adversaries in Stochastic Games with Total RewardsComputer Aided Verification10.1007/978-3-031-13188-2_3(48-69)Online publication date: 7-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-13188-2_3
Baier CFunke FPiribauer JZiemek R(2022)On probability-raising causality in Markov decision processesFoundations of Software Science and Computation Structures10.1007/978-3-030-99253-8_3(40-60)Online publication date: 4-Apr-2022
https://dl.acm.org/doi/10.1007/978-3-030-99253-8_3
Quatmann TJunges SKatoen J(2021)Markov automata with multiple objectivesFormal Methods in System Design10.1007/s10703-021-00364-660:1(33-86)Online publication date: 29-Mar-2021
https://dl.acm.org/doi/10.1007/s10703-021-00364-6
Quatmann TKatoen J(2021)Multi-objective Optimization of Long-run Average and Total RewardsTools and Algorithms for the Construction and Analysis of Systems10.1007/978-3-030-72016-2_13(230-249)Online publication date: 27-Mar-2021
https://dl.acm.org/doi/10.1007/978-3-030-72016-2_13
Baier CBertrand NPiribauer JSankur OBouyer P(2019)Long-run satisfaction of path propertiesProceedings of the 34th Annual ACM/IEEE Symposium on Logic in Computer Science10.5555/3470152.3470170(1-14)Online publication date: 24-Jun-2019
https://dl.acm.org/doi/10.5555/3470152.3470170
Baier CBertrand NPiribauer JSankur O(2019)Long-run Satisfaction of Path Properties2019 34th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS)10.1109/LICS.2019.8785672(1-14)Online publication date: Jun-2019
https://doi.org/10.1109/LICS.2019.8785672
Piribauer JBaier C(2019)Partial and Conditional Expectations in Markov Decision Processes with Integer WeightsFoundations of Software Science and Computation Structures10.1007/978-3-030-17127-8_25(436-452)Online publication date: 5-Apr-2019
https://doi.org/10.1007/978-3-030-17127-8_25
Bouyer PGonzález MMarkey NRandour M(2018)Multi-weighted Markov Decision Processes with Reachability ObjectivesElectronic Proceedings in Theoretical Computer Science10.4204/EPTCS.277.18277(250-264)Online publication date: 7-Sep-2018
https://doi.org/10.4204/EPTCS.277.18
Baier CDubslaff C(2018)From verification to synthesis under cost-utility constraintsACM SIGLOG News10.1145/3292048.32920525:4(26-46)Online publication date: 12-Nov-2018
https://dl.acm.org/doi/10.1145/3292048.3292052

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents