research-article

Approximating Markov Processes by Averaging

Authors:

Philippe Chaput,

Prakash Panangaden,

Gordon PlotkinAuthors Info & Claims

Journal of the ACM (JACM), Volume 61, Issue 1

Article No.: 5, Pages 1 - 45

https://doi.org/10.1145/2537948

Published: 01 January 2014 Publication History

Abstract

Normally, one thinks of probabilistic transition systems as taking an initial probability distribution over the state space into a new probability distribution representing the system after a transition. We, however, take a dual view of Markov processes as transformers of bounded measurable functions. This is very much in the same spirit as a “predicate-transformer” view, which is dual to the state-transformer view of transition systems. We redevelop the theory of labelled Markov processes from this viewpoint; in particular, we explore approximation theory. We obtain three main results.

(i) It is possible to define bisimulation on general measure spaces and show that it is an equivalence relation. The logical characterization of bisimulation can be done straightforwardly and generally.

(ii) A new and flexible approach to approximation based on averaging can be given. This vastly generalizes and streamlines the idea of using conditional expectations to compute approximations.

(iii) We show that there is a minimal process bisimulation-equivalent to a given process, and this minimal process is obtained as the limit of the finite approximants.

Supplemental Material

PDF File - a5-chaput_appdx.pdf

Unrefereed Appendix for Approximating Markov Processes by Averaging

Download
228.19 KB

References

[1]

Arendt, W., Grabosch, A., Greiner, G., Groh, U., Lotz, H., Moustakas, U., Nagel, R., Neubrander, F., and Schlotterbeck, U. 1986. One-Parameter Semigroups of Positive Operators. Springer-Verlag.

Digital Library

[2]

Baier, C. and Katoen, J.-P. 2008. Principles of Model Checking. MIT Press.

Digital Library

[3]

Bartels, F., Sokolova, A., and de Vink, E. 2004. A hierarchy of probabilistic system types. Theoret. Comput. Sci. 327, 3--22.

Digital Library

[4]

Billingsley, P. 1995. Probability and Measure. Wiley-Interscience.

[5]

Blute, R., Desharnais, J., Edalat, A., and Panangaden, P. 1997. Bisimulation for labelled Markov processes. In Proceedings of the 12th IEEE Symposium on Logic in Computer Science.

Digital Library

[6]

Bouchard-Côté, A., Ferns, N., Panangaden, P., and Precup, D. 2005. An approximation algorithm for labelled Markov processes: Towards realistic approximation. In Proceedings of the 2nd International Conference on the Quantitative Evaluation of Systems (QEST). 54--61.

Digital Library

[7]

Cattani, S., Segala, R., Kwiatkowska, M., and Norman, G. 2005. Stochastic transition systems for continuous state spaces and non-determinism. In Proceedings of the 8th International Conference on Foundations of Software Science and Computational Structures (FOSSACS). Lecture Notes in Computer Science, vol. 3441, 125--139.

Digital Library

[8]

Choksi, J. 1958. Inverse limits on measure spaces. Proc. London Math. Soc. 8, 3, 321--342.

[9]

Danos, V. and Desharnais, J. 2003. Labeled Markov Processes: Stronger and faster approximations. In Proceedings of the 18th Symposium on Logic in Computer Science. IEEE.

Digital Library

[10]

Danos, V., Desharnais, J., and Panangaden, P. 2003. Conditional expectation and the approximation of labelled Markov processes. In Proceedings of CONCUR’03 - Concurrency Theory. R. Amadio and D. Lugiez Eds., Lecture Notes in Computer Science, vol. 2761, Springer-Verlag, 477--491.

[11]

Danos, V., Desharnais, J., Laviolette, F., and Panangaden, P. 2006. Bisimulation and cocongruence for probabilistic systems. Inf. Computat. 204, 4, 503--523.

Digital Library

[12]

de Vink, E. and Rutten, J. J. M. M. 1997. Bisimulation for probabilistic transition systems: A coalgebraic approach. In Proceedings of the 24th International Colloquium on Automata Languages and Programming.

Digital Library

[13]

de Vink, E. and Rutten, J. J. M. M. 1999. Bisimulation for probabilistic transition systems: A coalgebraic approach. Theoret. Comput. Sci. 221, 1/2, 271--293.

Digital Library

[14]

Desharnais, J., Edalat, A., and Panangaden, P. 1998. A logical characterization of bisimulation for labelled Markov processes. In Proceedings of the 13th IEEE Symposium on Logic in Computer Science. IEEE, 478--489.

Digital Library

[15]

Desharnais, J., Gupta, V., Jagadeesan, R., and Panangaden, P. 2000. Approximation of labeled Markov processes. In Proceedings of the 15th Annual IEEE Symposium on Logic in Computer Science. IEEE, 95--106.

Digital Library

[16]

Desharnais, J., Edalat, A., and Panangaden, P. 2002. Bisimulation for labeled Markov processes. Inf. Computat. 179, 2, 163--193.

Digital Library

[17]

Desharnais, J., Gupta, V., Jagadeesan, R., and Panangaden, P. 2003. Approximating labeled Markov processes. Inf. Computat. 184, 1, 160--200.

Digital Library

[18]

Desharnais, J., Gupta, V., Jagadeesan, R., and Panangaden, P. 2004. A metric for labelled Markov processes. Theoret. Comput. Sci. 318, 3, 323--354.

Digital Library

[19]

Doberkat, E.-E. 2003. Semi-pullbacks and bisimulations in categories of stochastic relations. In Proceedings of the 27th International Colloquium on Automata Languages and Programming (ICALP’03). J. C. M. Baeten, J. K. Lenstra, J. Parrow, and G. J. Woeinger Eds., Lecture Notes in Computer Science, vol. 2719, Springer-Verlag, 996--1007.

Digital Library

[20]

Doberkat, E.-E. 2007. Stochastic Relations. Foundations for Markov Transition Systems. Chapman and Hall, New York.

Digital Library

[21]

Doberkat, E.-E. 2010. Stochastic Coalgebraic Logic. Springer-Verlag.

Digital Library

[22]

Dudley, R. M. 1989. Real Analysis and Probability. Wadsworth and Brookes/Cole.

[23]

Edalat, A. 1999. Semi-pullbacks and bisimulation in categories of Markov processes. Math. Struct. Comput. Sci. 9, 5, 523--543.

Digital Library

[24]

Ferns, N., Panangaden, P., and Precup, D. 2005. Metrics for Markov decision processes with infinite state spaces. In Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence. 201--208.

[25]

Foguel, S. R. 1980. Selected Topics in the Study of Markov Operators. Carolina Lecture Series, 9. University of North Carolina, Chapel Hill, NC.

[26]

Giry, M. 1981. A categorical approach to probability theory. In Categorical Aspects of Topology and Analysis, B. B. Ski Ed., Lecture Notes in Mathematics, vol. 915, Springer-Verlag, 68--85.

[27]

Goubault-Larrecq, J. 2007a. Continuous capacities on continuous state spaces. In Proceedings of the 34th International Colloquium on Automata, Languages and Programming (ICALP’07). Lecture Notes in Computer Science, vol. 4596, Springer-Verlag, 764--776.

Digital Library

[28]

Goubault-Larrecq, J. 2007b. Continuous previsions. In Proceedings of the 16th Annual EACSL Conference on Computer Science Logic (CSL’07). Lecture Notes in Computer Science, vol. 4646, Springer-Verlag, 542--557.

Digital Library

[29]

Goubault-Larrecq, J. 2007c. On Noetherian spaces. In Proceedings of the 22nd Annual IEEE Symposium on Logic in Computer Science (LICS’07). IEEE, 453--462.

Digital Library

[30]

Goubault-Larrecq, J. 2008a. Prevision domains and convex powercones. In Proceedings of the 11th International Conference on Foundations of Software Science and Computation Structures (FoSSaCS’08). Lecture Notes in Computer Science, vol. 4962, Springer-Verlag, 318--333.

Digital Library

[31]

Goubault-Larrecq, J. 2008b. Simulation hemi-metrics between infinite-state stochastic games. In Proceedings of the 11th International Conference on Foundations of Software Science and Computation Structures (FoSSaCS’08). Lecture Notes in Computer Science, vol. 4962, Springer-Verlag, 50--65.

Digital Library

[32]

Halmos, P. 1974. Measure Theory. Number 18 in Graduate Texts in Mathematics, Springer-Verlag. (Originally published in 1950.)

[33]

Hawke, P. 2006. Markov operators on Banach lattices. M.S. thesis, University of the Witwatersrand, Johannesburg.

[34]

Hennessy, M. and Milner, R. 1985. Algebraic laws for nondeterminism and concurrency. J. ACM 32, 1, 137--162.

Digital Library

[35]

Hopf, E. 1954. The general temporally discrete Markoff process. J. Ration. Math. Mech. Anal. 3, 13--45.

[36]

Joyal, A., Nielsen, M., and Winskel, G. 1993. Bisimulation and open maps. In Proceedings of 8th Annual IEEE Symposium on Logic in Computer Science. 418--427.

[37]

Kim, C.-W. 1972. Approximation theorems for Markov operators. Prob. Theory Relat. Fields 21, 3, 207--214.

[38]

Kingman, J. F. C. and Taylor, S. J. 1966. Introduction to Measure and Probability. Cambridge University Press.

[39]

Kozen, D. 1985. A probabilistic PDL. J. Comput. Syst. Sci. 30, 2, 162--178.

[40]

Larsen, K. G. and Skou, A. 1991. Bisimulation through probablistic testing. Inf. Computat. 94, 1--28.

Digital Library

[41]

Mislove, M., Ouaknine, J., Pavlovic, D., and Worrell, J. 2004. Duality for labelled Markov processes. In Foundations of Software Science and Computation Structures (FOSSACS). I. Walukiewicz Ed., Lecture Notes in Computer Science, vol. 2987, 393--407.

[42]

Panangaden, P. 2009. Labelled Markov Processes. Imperial College Press.

Digital Library

[43]

Pavlovic, D., Mislove, M., and Worrell, J. B. 2006. Testing semantics: Connecting processes and process logics. In Proceedings of the 11th International Conference on Algebraic Methodology and Software Technology (AMAST). Lecture Notes in Computer Science, vol. 4019, Springer-Verlag, 308--322.

Digital Library

[44]

Rudin, W. 1966. Real and Complex Analysis. McGraw-Hill.

[45]

Rutten, J. J. M. M. and de Vink, E. 1997. Bisimulation for probabilistic transition systems: a coalgebraic approach. In Proceedings of ICALP’97. P. Degano Ed., Lecture Notes in Computer Science, vol. 1256, Springer-Verlag, 460--470.

Digital Library

[46]

Schaefer, H. H. 1974. Banach Lattices and Positive Operators. Springer-Verlag.

[47]

Selinger, P. 2004. Towards a semantics for higher-order quantum computation. In Proceedings of the 2nd International Workshop on Quantum Programming Languages. 127--143. www.mathstat.dal.ca/~selinger/papers.htm.

[48]

van Benthem, J. 1976. Modal correspondence theory. Ph.D. thesis, University of Amsterdam.

[49]

van Breugel, F. and Worrell, J. 2001a. An algorithm for quantitative verification of probabilistic systems. In Proceedings of the 12th International Conference on Concurrency Theory (CONCUR’01). K. G. Larsen and M. Nielsen Eds., Lecture Notes in Computer Science, vol. 2154, Springer-Verlag, 336--350.

Digital Library

[50]

van Breugel, F. and Worrell, J. 2001b. Towards quantitative verification of probabilistic systems. In Proceedings of the 28th International Colloquium on Automata, Languages and Programming. Springer-Verlag.

Digital Library

[51]

van Breugel, F., Mislove, M., Ouaknine, J., and Worrell, J. 2003. An intrinsic characterization of approximate probabilistic bisimilarity. In Proceedings of FOSSACS 03. Lecture Notes in Computer Science, vol. 2620, Springer-Verlag.

Digital Library

[52]

Vardi, M. 1985. Automatic verification of probabilistic concurrent finite-state programs. In Proceedings of the 26th IEEE Symposium on Foundations of Computer Science. 327--338.

Digital Library

[53]

Williams, D. 1991. Probability with Martingales. CUP, Cambridge.

[54]

Yosida, K. and Kakutani, S. 1941. Operator-theoretical treatment of Markoff’s process and mean ergodic theorem. Ann. Math. 42, 1, 188--228.

Cited By

Dahlqvist FKozen D(2019)Semantics of higher-order probabilistic programs with conditioningProceedings of the ACM on Programming Languages10.1145/33711254:POPL(1-29)Online publication date: 20-Dec-2019
https://dl.acm.org/doi/10.1145/3371125
Chen LClerc FPanangaden P(2019)Bisimulation for Feller-Dynkin ProcessesElectronic Notes in Theoretical Computer Science (ENTCS)10.1016/j.entcs.2019.09.004347:C(45-63)Online publication date: 30-Nov-2019
https://dl.acm.org/doi/10.1016/j.entcs.2019.09.004
Dahlqvist FSilva ADanos VGarnier I(2018)Borel Kernels and their Approximation, CategoricallyElectronic Notes in Theoretical Computer Science10.1016/j.entcs.2018.11.006341(91-119)Online publication date: Dec-2018
https://doi.org/10.1016/j.entcs.2018.11.006
Show More Cited By

Index Terms

Approximating Markov Processes by Averaging
1. Mathematics of computing
  1. Probability and statistics
    1. Probabilistic representations
      1. Markov networks
    2. Stochastic processes
      1. Markov processes
2. Theory of computation

Recommendations

Approximating Bisimilarity for Markov Processes

In this paper we investigate bisimilarity for general Markov processes through the correspondence between sub-@s-algebras and equivalence relations. In particular, we study bisimulations from the perspective of fixed-point theory. Given a Markov process ...
Approximating Markov Processes by Averaging
ICALP '09: Proceedings of the 36th Internatilonal Collogquium on Automata, Languages and Programming: Part II

We take a dual view of Markov processes --- advocated by Kozen --- as transformers of bounded measurable functions. We redevelop the theory of labelled Markov processes from this view point, in particular we explore approximation theory. We obtain three ...
Optimal Stopping Games for Markov Processes

Let $X=(X_t)_{t \ge 0}$ be a strong Markov process, and let $G_1,\, G_2$, and $G_3$ be continuous functions satisfying $G_1 \le G_3 \le G_2$ and $\mathsf{E}_x\sup_t \vert G_i(X_t) \vert < \infty$ for $i=1,2,3$. Consider the optimal stopping game where ...

Comments

Information & Contributors

Information

Published In

cover image Journal of the ACM

Journal of the ACM Volume 61, Issue 1

January 2014

222 pages

ISSN:0004-5411

EISSN:1557-735X

DOI:10.1145/2578041

Editor:
Victor Vianu
University of California, San Diego

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 January 2014

Accepted: 01 October 2013

Revised: 01 April 2012

Received: 01 May 2010

Published in JACM Volume 61, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
494
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)2

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dahlqvist FKozen D(2019)Semantics of higher-order probabilistic programs with conditioningProceedings of the ACM on Programming Languages10.1145/33711254:POPL(1-29)Online publication date: 20-Dec-2019
https://dl.acm.org/doi/10.1145/3371125
Chen LClerc FPanangaden P(2019)Bisimulation for Feller-Dynkin ProcessesElectronic Notes in Theoretical Computer Science (ENTCS)10.1016/j.entcs.2019.09.004347:C(45-63)Online publication date: 30-Nov-2019
https://dl.acm.org/doi/10.1016/j.entcs.2019.09.004
Dahlqvist FSilva ADanos VGarnier I(2018)Borel Kernels and their Approximation, CategoricallyElectronic Notes in Theoretical Computer Science10.1016/j.entcs.2018.11.006341(91-119)Online publication date: Dec-2018
https://doi.org/10.1016/j.entcs.2018.11.006
Gagné NPanangaden P(2018)A Categorical Characterization of Relative Entropy on Standard Borel SpacesElectronic Notes in Theoretical Computer Science10.1016/j.entcs.2018.03.020336(135-153)Online publication date: Apr-2018
https://doi.org/10.1016/j.entcs.2018.03.020
Clerc FDanos VDahlqvist FGarnier I(2017)Pointless LearningProceedings of the 20th International Conference on Foundations of Software Science and Computation Structures - Volume 1020310.1007/978-3-662-54458-7_21(355-369)Online publication date: 22-Apr-2017
https://dl.acm.org/doi/10.1007/978-3-662-54458-7_21
Yang PJansen DZhang L(2017)Distribution-Based Bisimulation for Labelled Markov ProcessesFormal Modeling and Analysis of Timed Systems10.1007/978-3-319-65765-3_10(170-186)Online publication date: 3-Aug-2017
https://doi.org/10.1007/978-3-319-65765-3_10
Esmaeil Zadeh Soudjani SAdzkiya DAbate A(2016)Formal Verification of Stochastic Max-Plus-Linear SystemsIEEE Transactions on Automatic Control10.1109/TAC.2015.250278161:10(2861-2876)Online publication date: Oct-2016
https://doi.org/10.1109/TAC.2015.2502781
Dahlqvist FDanos VGarnier I(2016)Giry and the MachineElectronic Notes in Theoretical Computer Science10.1016/j.entcs.2016.09.033325(85-110)Online publication date: Oct-2016
https://doi.org/10.1016/j.entcs.2016.09.033
Panangaden P(2015)Probabilistic bisimulationACM SIGLOG News10.1145/2815493.28155012:3(72-84)Online publication date: 17-Aug-2015
https://dl.acm.org/doi/10.1145/2815493.2815501
McIver AMorgan CRabehaja T(2015)Abstract Hidden Markov ModelsProceedings of the 2015 30th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS)10.1109/LICS.2015.61(597-608)Online publication date: 6-Jul-2015
https://dl.acm.org/doi/10.1109/LICS.2015.61
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents