research-article

Conditional Value-at-Risk for Reachability and Mean Payoff in Markov Decision Processes

Authors:

Jan Křetínský,

Tobias MeggendorferAuthors Info & Claims

LICS '18: Proceedings of the 33rd Annual ACM/IEEE Symposium on Logic in Computer Science

Pages 609 - 618

https://doi.org/10.1145/3209108.3209176

Published: 09 July 2018 Publication History

Abstract

We present the conditional value-at-risk (CVaR) in the context of Markov chains and Markov decision processes with reachability and mean-payoff objectives. CVaR quantifies risk by means of the expectation of the worst p-quantile. As such it can be used to design risk-averse systems. We consider not only CVaR constraints, but also introduce their conjunction with expectation constraints and quantile constraints (value-at-risk, VaR). We derive lower and upper bounds on the computational complexity of the respective decision problems and characterize the structure of the strategies in terms of memory and randomization.

References

[1]

Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. 1999. Coherent Measures of Risk. Mathematical Finance 9, 3 (1999), 203--228.

[2]

Pranav Ashok, Krishnendu Chatterjee, Przemyslaw Daca, Jan Křetínský, and Tobias Meggendorfer. 2017. Value Iteration for Long-Run Average Reward in Markov Decision Processes. In CAV (LNCS), Vol. 10426. Springer, 201--221.

[3]

Christel Baier, Marcus Daum, Clemens Dubslaff, Joachim Klein, and Sascha Klüppelholz. 2014. Energy-Utility Quantiles. In NFM (LNCS), Vol. 8430. Springer, 285--299.

Digital Library

[4]

Christel Baier, Clemens Dubslaff, and Sascha Klüppelholz. 2014. Trade-off analysis meets probabilistic model checking. In CSL-LICS. ACM, 1:1--1:10.

Digital Library

[5]

Christel Baier, Clemens Dubslaff, Sascha Klüppelholz, Marcus Daum, Joachim Klein, Steffen Märcker, and Sascha Wunderlich. 2014. Probabilistic Model Checking and Non-standard Multi-objective Reasoning. In FASE (LNCS), Vol. 8411. Springer, 1--16.

Digital Library

[6]

Christel Baier, Joachim Klein, Sascha Klüppelholz, and Sascha Wunderlich. 2017. Maximizing the Conditional Expected Reward for Reaching the Goal. In TACAS (LNCS), Vol. 10206. 269--285.

Digital Library

[7]

Nicole Bäuerle and Jonathan Ott. 2011. Markov Decision Processes with Average-Value-at-Risk criteria. Math. Meth. of OR 74, 3 (2011), 361--379.

[8]

Tanya Styblo Beder. 1995. VAR: Seductive but dangerous. Financial Analysts Journal 51, 5 (1995), 12--24.

[9]

Tomás Brázdil, Václav Brozek, Krishnendu Chatterjee, Vojtech Forejt, and Antonín Kucera. 2014. Two Views on Multiple Mean-Payoff Objectives in Markov Decision Processes. LMCS 10, 1 (2014).

[10]

Tomás Brázdil, Krishnendu Chatterjee, Vojtech Forejt, and Antonín Kucera. 2013. Trading Performance for Stability in Markov Decision Processes. In LICS. IEEE Computer Society, 331--340.

Digital Library

[11]

Véronique Bruyère, Emmanuel Filiot, Mickael Randour, and Jean-François Raskin. 2017. Meet your expectations with guarantees: Beyond worst-case synthesis in quantitative games. Inf. Comput. 254 (2017), 259--295.

[12]

John F. Canny. 1988. Some Algebraic and Geometric Computations in PSPACE. In STOC. ACM, 460--467.

Digital Library

[13]

Stefano Carpin, Yinlam Chow, and Marco Pavone. 2016. Risk aversion in finite Markov Decision Processes using total cost criteria and average value at risk. In ICRA. IEEE, 335--342.

[14]

Krishnendu Chatterjee, Vojtech Forejt, and Dominik Wojtczak. 2013. Multi-objective Discounted Reward Verification in Graphs and MDPs. In LPAR (LNCS), Vol. 8312. Springer, 228--242.

[15]

Krishnendu Chatterjee, Zuzana Komárková, and Jan Křetínský. 2015. Unifying Two Views on Multiple Mean-Payoff Objectives in Markov Decision Processes. In LICS. IEEE Computer Society, 244--256.

Digital Library

[16]

Krishnendu Chatterjee, Zuzana Křetínská, and Jan Křetínský. 2017. Unifying Two Views on Multiple Mean-Payoff Objectives in Markov Decision Processes. LMCS 13, 2 (2017).

[17]

Lorenzo Clemente and Jean-François Raskin. 2015. Multidimensional beyond Worst-Case and Almost-Sure Problems for Mean-Payoff Objectives. In LICS. IEEE Computer Society, 257--268.

Digital Library

[18]

Costas Courcoubetis and Mihalis Yannakakis. 1995. The Complexity of Probabilistic Verification. J. ACM 42, 4 (1995), 857--907.

Digital Library

[19]

Kousha Etessami, Marta Z. Kwiatkowska, Moshe Y. Vardi, and Mihalis Yannakakis. 2008. Multi-Objective Model Checking of Markov Decision Processes. LMCS 4, 4 (2008).

[20]

J.A. Filar, D. Krass, and K.W Ross. 1995. Percentile performance criteria for limiting average Markov decision processes. IEEE Trans. Automat. Control 40, 1 (Jan 1995), 2--10.

[21]

Jerzy A. Filar, Dmitry Krass, and Keith W. Ross. 1995. Percentile performance criteria for limiting average Markov decision processes. IEEE Trans. Automat. Control 40 (1995), 2--10.

[22]

Vojtech Forejt, Marta Z. Kwiatkowska, Gethin Norman, David Parker, and Hongyang Qu. 2011. Quantitative Multi-objective Verification for Probabilistic Systems. In TACAS (LNCS), Vol. 6605. Springer, 112--127.

Digital Library

[23]

Hugo Gilbert, Paul Weng, and Yan Xu. 2017. Optimizing Quantiles in Preference-Based Markov Decision Processes. In AAAI. AAAI Press, 3569--3575.

[24]

Christoph Haase and Stefan Kiefer. 2015. The Odds of Staying on Budget. In ICALP (LNCS), Vol. 9135. Springer, 234--246.

Digital Library

[25]

Christoph Haase, Stefan Kiefer, and Markus Lohrey. 2017. Computing quantiles in Markov chains with multi-dimensional costs. In LICS. IEEE Computer Society, 1--12.

[26]

Yonghui Huang and Xianping Guo. 2016. Minimum Average Value-at-Risk for Finite Horizon Semi-Markov Decision Processes in Continuous Time. SIAM Journal on Optimization 26, 1 (2016), 1--28.

Digital Library

[27]

Masayuki Kageyama, Takayuki Fujii, Koji Kanefuji, and Hiroe Tsubaki. 2011. Conditional Value-at-Risk for Random Immediate Reward Variables in Markov Decision Processes. American J. Computational Mathematics 1, 3 (2011), 183--188.

[28]

Jan Křetínský and Tobias Meggendorfer. 2018. Conditional Value-at-Risk for Reachability and Mean Payoff in Markov Decision Processes. Technical Report abs/1805.xxxxx. arXiv.org.

[29]

Xiaocheng Li, Huaiyang Zhong, and Margaret L. Brandeau. 2017. Quantile Markov Decision Process. CoRR abs/1711.05788 (2017).

[30]

Christopher W. Miller and Insoon Yang. 2017. Optimal Control of Conditional Value-at-Risk in Continuous Time. SIAM J. Control and Optimization 55, 2 (2017), 856--884.

Digital Library

[31]

M. L. Puterman. 1994. Markov Decision Processes. J. Wiley and Sons.

[32]

Mickael Randour, Jean-François Raskin, and Ocan Sankur. 2017. Percentile queries in multi-dimensional Markov decision processes. FMSD 50, 2-3 (2017), 207--248.

Digital Library

[33]

R. Tyrrell Rockafellar and Stanislav Uryasev. 2000. Optimization of Conditional Value-at-Risk. Journal of Risk 2 (2000), 21--41.

[34]

R Tyrrell Rockafellar and Stanislav Uryasev. 2002. Conditional value-at-risk for general loss distributions. Journal of banking & finance 26, 7 (2002), 1443--1471.

[35]

Michael Ummels and Christel Baier. 2013. Computing Quantiles in Markov Reward Models. In FoSSaCS (LNCS), Vol. 7794. Springer, 353--368.

Digital Library

Cited By

Baier CChatterjee KMeggendorfer TPiribauer J(2024)Entropic risk for turn-based stochastic gamesInformation and Computation10.1016/j.ic.2024.105214(105214)Online publication date: Aug-2024
https://doi.org/10.1016/j.ic.2024.105214
Elsayed-Aly IParker DFeng L(2024)Distributional Probabilistic Model CheckingNASA Formal Methods10.1007/978-3-031-60698-4_4(57-75)Online publication date: 26-May-2024
https://doi.org/10.1007/978-3-031-60698-4_4
Akshay SChatterjee KMeggendorfer TŽikelić Đ(2023)MDPs as Distribution Transformers: Affine Invariant Synthesis for Safety ObjectivesComputer Aided Verification10.1007/978-3-031-37709-9_5(86-112)Online publication date: 17-Jul-2023
https://doi.org/10.1007/978-3-031-37709-9_5
Show More Cited By

Index Terms

Conditional Value-at-Risk for Reachability and Mean Payoff in Markov Decision Processes
1. Theory of computation
  1. Logic
    1. Verification by model checking

Recommendations

Monte Carlo Methods for Value-at-Risk and Conditional Value-at-Risk: A Review
Special Issue on Emerging Methodologies and Applications

Value-at-risk (VaR) and conditional value-at-risk (CVaR) are two widely used risk measures of large losses and are employed in the financial industry for risk management purposes. In practice, loss distributions typically do not have closed-form ...
Conditional value-at-risk model for hazardous materials transportation
WSC '11: Proceedings of the Winter Simulation Conference

This paper investigates how the conditional value-at-risk (CVaR) can be used to mitigate risk in hazardous materials (hazmat) transportation. Routing hazmat must consider accident probabilities and accident consequences that depend on the hazmat types ...
Risk-sensitive Markov Decision Processes with Risk Constraints of Coherent Risk Measures in Fuzzy and Stochastic Environment
IJCCI 2019: Proceedings of the 11th International Joint Conference on Computational Intelligence

Risk-sensitive decision making with constraints of coherent risk measures is discussed in Markov decision processes. Risk-sensitive expected rewards under utility functions are approximated by weighted average value-at-risks, and risk constraints are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

LICS '18: Proceedings of the 33rd Annual ACM/IEEE Symposium on Logic in Computer Science

July 2018

960 pages

ISBN:9781450355834

DOI:10.1145/3209108

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGLOG: ACM Special Interest Group on Logic and Computation
EACSL: European Association for Computer Science Logic
IEEE-CS\DATC: IEEE Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 July 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Deutsche Forschungsgemeinschaft
Grantová Agentura České Republiky

Conference

LICS '18

Sponsor:

SIGLOG
EACSL
IEEE-CS\DATC

LICS '18: 33rd Annual ACM/IEEE Symposium on Logic in Computer Science

July 9 - 12, 2018

Oxford, United Kingdom

Acceptance Rates

Overall Acceptance Rate 215 of 622 submissions, 35%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
183
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)5

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Baier CChatterjee KMeggendorfer TPiribauer J(2024)Entropic risk for turn-based stochastic gamesInformation and Computation10.1016/j.ic.2024.105214(105214)Online publication date: Aug-2024
https://doi.org/10.1016/j.ic.2024.105214
Elsayed-Aly IParker DFeng L(2024)Distributional Probabilistic Model CheckingNASA Formal Methods10.1007/978-3-031-60698-4_4(57-75)Online publication date: 26-May-2024
https://doi.org/10.1007/978-3-031-60698-4_4
Akshay SChatterjee KMeggendorfer TŽikelić Đ(2023)MDPs as Distribution Transformers: Affine Invariant Synthesis for Safety ObjectivesComputer Aided Verification10.1007/978-3-031-37709-9_5(86-112)Online publication date: 17-Jul-2023
https://doi.org/10.1007/978-3-031-37709-9_5
Piribauer JBaier C(2019)Partial and Conditional Expectations in Markov Decision Processes with Integer WeightsFoundations of Software Science and Computation Structures10.1007/978-3-030-17127-8_25(436-452)Online publication date: 5-Apr-2019
https://doi.org/10.1007/978-3-030-17127-8_25
Baier CDubslaff C(2018)From verification to synthesis under cost-utility constraintsACM SIGLOG News10.1145/3292048.32920525:4(26-46)Online publication date: 12-Nov-2018
https://dl.acm.org/doi/10.1145/3292048.3292052

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten