Abstract
Markov decision processes (MDPs) have developed as a standard for representing uncertainty in decision-theoretic planning. However, MDPs require an explicit representation of the state space and the probabilistic transition model which, in continuous or hybrid continuous-discrete domains, are not always easy to define. Even when this representation is available, the size of the state space and the number of state variables to consider in the transition function may be such that the resulting MDP cannot be solved using traditional techniques. In this paper a reward-based abstraction for solving hybrid MDPs is presented. In the proposed method, we gather information about the rewards and the dynamics of the system by exploring the environment. This information is used to build a decision tree (C4.5) representing a small set of abstract states with equivalent rewards, and then is used to learn a probabilistic transition function using a Bayesian networks learning algorithm (K2). The system output is a problem specification ready for its solution with traditional dynamic programming algorithms. We have tested our abstract MDP model approximation in real-world problem domains. We present the results in terms of the models learned and their solutions for different configurations showing that our approach produces fast solutions with satisfying policies.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Bellman, R.E.: Dynamic Programming. Princeton U. Press, Princeton (1957)
Boutilier, C., Dean, T., Hanks, S.: Decision-theoretic planning: structural assumptions and computational leverage. Journal of AI Research 11, 1–94 (1999)
Boutilier, C., Goldszmidt, M., Sabata, B.: Continuous value function approximation for sequential bidding policies. In: Laskey, K., Prade, H. (eds.) Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence (UAI 1999), Morgan Kaufmann Publishers, San Francisco (1999)
Cooper, G.F., Herskovits, E.: A Bayesian method for the induction of probabilistic networks from data. Machine Learning (1992)
Darwiche, A., Goldszmidt, M.: Action networks: A framework for reasoning about actions and change under understanding. In: Proceedings of the Tenth Conf. on Uncertainty in AI, UAI 1994, Seattle, WA, USA, pp. 136–144 (1994)
Dean, T., Givan, R.: Model minimization in Markov decision processes. In: Proc. of the 14th National Conf. on AI, pp. 106–111. AAAI Press, Menlo Park (1997)
Dean, T., Kanazawa, K.: A model for reasoning about persistence and causation. Computational Intelligence 5, 142–150 (1989)
Dearden, R., Boutillier, R.: Abstraction and approximate decision-theoretic planning. AI 89, 219–283 (1997)
Feng, Z., Dearden, R., Meuleau, N., Washington, R.: Dynamic programming for structured continuous Markov decision problems. In: Proc. of the 20th Conf. on Uncertainty in AI (UAI-2004), Banff, Canada (2004)
Kearns, M., Koller, D.: Efficient reinforcement learning in factored MDPs. In: Proc. of the Sixteenth International Joint on Artificial Intelligence, IJCAI 1999, Stockolm, Sweden (1999)
Li, L., Littman, M.L.: Lazy approximation for solving continuous finite-horizon MDPs. In: AAAI 2005, Pittsburgh, PA, pp. 1175–1180 (2005)
Munos, R., Moore, A.: Variable resolution discretization for high-accuracy solutions of optimal control problems. In: Dean, T. (ed.) Proceedings of the 16th International Joint Conference on Artificial Intelligence (IJCAI 1999), August 1999, pp. 1348–1355. Morgan Kaufmann Publishers, San Francisco (1999)
Pineau, J., Gordon, G., Thrun, S.: Policy-contingent abstraction for robust control. In: Proc. of the 19th Conf. on Uncertainty in AI, UAI 2003, pp. 477–484 (2003)
Puterman, M.L.: Markov Decision Processes. Wiley, New York (1994)
Quinlan, J.R.: C4.5: Programs for machine learning. Morgan Kaufmann, San Francisco (1993)
Reyes, A., Sucar, L.E., Morales, E., Ibarguengoytia, P.H.: Abstract MDPs using qualitative change predicates: An application in power generation. In: Planning under Uncertainty in Real-World Problems Workshop. Neural Information Processing Systems (NIPS-2003), Vancouver CA (winter, 2003)
Sallans, B., Hinton, G.E.: Reinforcement learning with factored states and actions. Journal of Machine Learning and Research, 1063–1088 (2004)
Suc, D., Bratko, I.: Qualitative reverse engineering. In: Proc. of the 19th International Conf. on Machine Learning (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Reyes, A., Sucar, L.E., Morales, E.F., Ibargüengoytia, P.H. (2006). Solving Hybrid Markov Decision Processes. In: Gelbukh, A., Reyes-Garcia, C.A. (eds) MICAI 2006: Advances in Artificial Intelligence. MICAI 2006. Lecture Notes in Computer Science(), vol 4293. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11925231_22
Download citation
DOI: https://doi.org/10.1007/11925231_22
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-49026-5
Online ISBN: 978-3-540-49058-6
eBook Packages: Computer ScienceComputer Science (R0)