Article

Maximizing the Conditional Expected Reward for Reaching the Goal

Authors:

Christel Baier,

Sascha Klüppelholz,

Sascha WunderlichAuthors Info & Claims

Proceedings, Part II, of the 23rd International Conference on Tools and Algorithms for the Construction and Analysis of Systems - Volume 10206

Pages 269 - 285

https://doi.org/10.1007/978-3-662-54580-5_16

Published: 22 April 2017 Publication History

Abstract

The paper addresses the problem of computing maximal conditional expected accumulated rewards until reaching a target state briefly called maximal conditional expectations in finite-state Markov decision processes where the condition is given as a reachability constraint. Conditional expectations of this type can, e.g., stand for the maximal expected termination time of probabilistic programs with non-determinism, under the condition that the program eventually terminates, or for the worst-case expected penalty to be paid, assuming that at least three deadlines are missed. The main results of the paper are i a polynomial-time algorithm to check the finiteness of maximal conditional expectations, ii PSPACE-completeness for the threshold problem in acyclic Markov decision processes where the task is to check whether the maximal conditional expectation exceeds a given threshold, iii a pseudo-polynomial-time algorithm for the threshold problem in the general cyclic case, and iv an exponential-time algorithm for computing the maximal conditional expectation and an optimal scheduler.

References

[1]

Abdulla, P.A., Henda, N.B., Mayr, R.: Decisive Markov chains. Logical Methods Comput. Sci. 34 2007

[2]

Acerbi, C., Tasche, D.: Expected shortfall: a natural coherent alternative to value at risk. Econ. notes 312, 379---388 2002

[3]

Alvim, M.S., Andrés, M.E., Chatzikokolakis, K., Degano, P., Palamidessi, C.: On the information leakage of differentially-private mechanisms. J. Comput. Secur. 234, 427---469 2015

[4]

Alvim, M.S., Chatzikokolakis, K., McIver, A., Morgan, C., Palamidessi, C., Smith, G.: Axioms for information leakage. In: Proceedings of Computer Security Foundations Symposium CSF, pp. 77---92. IEEE Computer Society 2016

[5]

Alvim, M.S., Chatzikokolakis, K., Palamidessi, C., Smith, G.: Measuring information leakage using generalized gain functions. In: Proceedings of Computer Security Foundations Symposium CSF, pp. 265---279. IEEE Computer Society 2012

Digital Library

[6]

Andrés, M.E.: Quantitative Analysis of Information Leakage in Probabilistic and Nondeterministic Systems. Ph.D. thesis, UB Nijmegen 2011

[7]

Andrés, M.E., Palamidessi, C., van Rossum, P., Sokolova, A.: Information hiding in probabilistic concurrent systems. Theoret. Comput. Sci. 41228, 3072---3089 2011

[8]

Andrés, M.E., van Rossum, P.: Conditional probabilities over probabilistic and nondeterministic systems. In: Ramakrishnan, C.R., Rehof, J. eds. TACAS 2008. LNCS, vol. 4963, pp. 157---172. Springer, Heidelberg 2008.

Digital Library

[9]

Baier, C., Dubslaff, C., Klein, J., Klüppelholz, S., Wunderlich, S.: Probabilistic model checking for energy-utility analysis. In: Breugel, F., Kashefi, E., Palamidessi, C., Rutten, J. eds. Horizons of the Mind. A Tribute to Prakash Panangaden. LNCS, vol. 8464, pp. 96---123. Springer, Heidelberg 2014.

[10]

Baier, C., Katoen, J.-P.: Principles of Model Checking. MIT Press, Cambridge 2008

Digital Library

[11]

Baier, C., Klein, J., Klüppelholz, S., Märcker, S.: Computing conditional probabilities in Markovian models efficiently. In: Ábrahám, E., Havelund, K. eds. TACAS 2014. LNCS, vol. 8413, pp. 515---530. Springer, Heidelberg 2014.

[12]

Baier, C., Klein, J., Klüppelholz, S., Wunderlich, S.: Weight monitoring with linear temporal logic: complexity and decidability. In: Proceedings of Computer Science Logic/Logic in Computer Science CSL-LICS, pp. 11:1---11:10. ACM 2014

Digital Library

[13]

Baier, C., Klein, J., Klüppelholz, S. Wunderlich, S.: Maximizing the conditional expected reward for reaching the goal extended version. arXiv:1701.05389 2017

Digital Library

[14]

Barthe, G., Espitau, T., Ferrer Fioriti, L.M., Hsu, J.: Synthesizing probabilistic invariants via Doob's decomposition. In: Chaudhuri, S., Farzan, A. eds. CAV 2016. LNCS, vol. 9779, pp. 43---61. Springer, Heidelberg 2016.

[15]

Bertsekas, D.P., Tsitsiklis, J.N.: An analysis of stochastic shortest path problems. Math. Oper. Res. 163, 580---595 1991

Digital Library

[16]

Bertsekas, D.P., Yu, H.: Stochastic path problems under weak conditions. Technical report, M.I.T. Cambridge, Report LIDS 2909 2016

[17]

Boker, U., Chatterjee, K., Henzinger, T.A., Kupferman, O.: Temporal specifications with accumulative values. In: Proceedings of Logic in Computer Science LICS, pp. 43---52. IEEE Computer Society 2011

Digital Library

[18]

Brázdil, T., Brozek, V., Chatterjee, K., Forejt, V., Kucera, A.: Two views on multiple mean-payoff objectives in Markov decision processes. Logical Methods Comput. Sci. 101 2014

[19]

Brázdil, T., Kuă era, A.: Computing the expected accumulated reward and gain for a subclass of infinite Markov Chains. In: Sarukkai, S., Sen, S. eds. FSTTCS 2005. LNCS, vol. 3821, pp. 372---383. Springer, Heidelberg 2005.

Digital Library

[20]

Chatterjee, K., Fu, H., Goharshady, A.K.: Termination analysis of probabilistic programs through Positivstellensatz's. In: Chaudhuri, S., Farzan, A. eds. CAV 2016. LNCS, vol. 9779, pp. 3---22. Springer, Heidelberg 2016.

[21]

Chatzikokolakis, K., Palamidessi, C., Braun, C.: Compositional methods for information-hiding. Math. Struct. Comput. Sci. 266, 908---932 2016

[22]

Alfaro, L.: Computing minimum and maximum reachability times in probabilistic systems. In: Baeten, J.C.M., Mauw, S. eds. CONCUR 1999. LNCS, vol. 1664, pp. 66---81. Springer, Heidelberg 1999.

Digital Library

[23]

Gretz, F., Katoen, J., McIver, A.: Operational versus weakest pre-expectation semantics for the probabilistic guarded command language. Perform. Eval. 73, 110---132 2014

Digital Library

[24]

Jansen, N., Kaminski, B.L., Katoen, J., Olmedo, F., Gretz, F., McIver, A.: Conditioning in probabilistic programming. In: Proceedings of Mathematical Foundations of Programming Semantics MFPS, Electronic Notes Theoretical Computer Science, vol. 319, pp. 199---216 2015

Digital Library

[25]

Kallenberg, L.: Markov Decision Processes. Lecture Notes. University of Leiden, Leiden 2011

[26]

Katoen, J.-P., Gretz, F., Jansen, N., Kaminski, B.L., Olmedo, F.: Understanding probabilistic programs. In: Meyer, R., Platzer, A., Wehrheim, H. eds. Correct System Design. LNCS, vol. 9360, pp. 15---32. Springer, Heidelberg 2015.

[27]

Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. eds. CAV 2011. LNCS, vol. 6806, pp. 585---591. Springer, Heidelberg 2011.

Digital Library

[28]

PRISM model checker. http://www.prismmodelchecker.org/

[29]

Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York 1994

Digital Library

[30]

Randour, M., Raskin, J.-F., Sankur, O.: Variations on the stochastic shortest path problem. In: D'Souza, D., Lal, A., Larsen, K.G. eds. VMCAI 2015. LNCS, vol. 8931, pp. 1---18. Springer, Heidelberg 2015.

Digital Library

[31]

Seber, G., Lee, A.: Linear Regression Analysis. Wiley Series in Probability and Statistics. Wiley, New York 2003

[32]

Uryasev, S.: Conditional value-at-risk: optimization algorithms and applications. In Proceedings of Computational Intelligence and Financial Engineering CIFEr, pp. 49---57. IEEE 2000

Cited By

Busatto-Gaston DChakraborty DMajumdar AMukherjee SPérez GRaskin J(2023)Bi-objective Lexicographic Optimization in Markov Decision Processes with Related ObjectivesAutomated Technology for Verification and Analysis10.1007/978-3-031-45329-8_10(203-223)Online publication date: 24-Oct-2023
https://dl.acm.org/doi/10.1007/978-3-031-45329-8_10
Watanabe KEberhart CAsada KHasuo I(2023)Compositional Probabilistic Model Checking with String Diagrams of MDPsComputer Aided Verification10.1007/978-3-031-37709-9_3(40-61)Online publication date: 17-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-37709-9_3
Piribauer JBaier C(2019)Partial and Conditional Expectations in Markov Decision Processes with Integer WeightsFoundations of Software Science and Computation Structures10.1007/978-3-030-17127-8_25(436-452)Online publication date: 8-Apr-2019
https://dl.acm.org/doi/10.1007/978-3-030-17127-8_25
Show More Cited By

Recommendations

Maximizing Edit Distance Accuracy with Hidden Conditional Random Fields
CAIP 2013: Proceedings, Part I, of the 15th International Conference on Computer Analysis of Images and Patterns - Volume 8047

Handwriting recognition aims at predicting a sequence of characters from an image of a handwritten text. Main approaches rely on learning statistical models such as Hidden Markov Models or Conditional Random Fields, whose quality is measured through ...
Hierarchical hidden conditional random fields for information extraction
LION'05: Proceedings of the 5th international conference on Learning and Intelligent Optimization

Hidden Markov Models (HMMs) are very popular generative models for time series data. Recent work, however, has shown that for many tasks Conditional Random Fields (CRFs), a type of discriminative model, perform better than HMMs. Information extraction ...
Optimally solving Markov decision processes with total expected discounted reward function

Compared computational performance of linear programming and the policy iteration.Considered only discrete-time infinite-horizon MDPs with discounted reward.Used randomly generated test problems and a real-life health-care problem.Showed that, unlike ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

Proceedings, Part II, of the 23rd International Conference on Tools and Algorithms for the Construction and Analysis of Systems - Volume 10206

April 2017

393 pages

ISBN:9783662545799

Editors:
Axel Legay
Inria, Rennes Cedex, France
,
Tiziana Margaria
University of Limerick and Lero - The Irish Software Research Center, Limerick, Ireland

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 22 April 2017

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Busatto-Gaston DChakraborty DMajumdar AMukherjee SPérez GRaskin J(2023)Bi-objective Lexicographic Optimization in Markov Decision Processes with Related ObjectivesAutomated Technology for Verification and Analysis10.1007/978-3-031-45329-8_10(203-223)Online publication date: 24-Oct-2023
https://dl.acm.org/doi/10.1007/978-3-031-45329-8_10
Watanabe KEberhart CAsada KHasuo I(2023)Compositional Probabilistic Model Checking with String Diagrams of MDPsComputer Aided Verification10.1007/978-3-031-37709-9_3(40-61)Online publication date: 17-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-37709-9_3
Piribauer JBaier C(2019)Partial and Conditional Expectations in Markov Decision Processes with Integer WeightsFoundations of Software Science and Computation Structures10.1007/978-3-030-17127-8_25(436-452)Online publication date: 8-Apr-2019
https://dl.acm.org/doi/10.1007/978-3-030-17127-8_25
Baier CDubslaff C(2018)From verification to synthesis under cost-utility constraintsACM SIGLOG News10.1145/3292048.32920525:4(26-46)Online publication date: 12-Nov-2018
https://dl.acm.org/doi/10.1145/3292048.3292052
Křetínský JMeggendorfer T(2018)Conditional Value-at-Risk for Reachability and Mean Payoff in Markov Decision ProcessesProceedings of the 33rd Annual ACM/IEEE Symposium on Logic in Computer Science10.1145/3209108.3209176(609-618)Online publication date: 9-Jul-2018
https://dl.acm.org/doi/10.1145/3209108.3209176

View Options

View options

Figures

Tables

Media

View Table of Conten