The complexity of Markov decision processes

Journal of The ACM, 2000

2012

Starting from Zermelo's classical formal treatment of chess, we trace through history the analysis of two-player win/lose/draw games with perfect information and potentially infinite play. Such chess-like games have appeared in many different research communities, and methods for solving them, such as retrograde analysis, have been rediscovered independently. We then revisit Washburn's deterministic graphical games (DGGs), a natural generalization of chess-like games to arbitrary zero-sum payoffs.

2008

Journal of Artificial Intelligence Research, 2001

We show that for several variations of partially observable Markov decision processes, polynomial-time algorithms for nding control policies are unlikely to or simply don't have guarantees of nding policies within a constant factor or a constant summand of optimal. Here \unlikely" means \unless some complexity classes collapse," where the collapses considered are P = NP, P = PSPACE, or P = EXP. Until or unless these collapses are shown to hold, any control-policy designer must choose between such performance guarantees and e cient computation.

Abstract We consider a team of agents that are required to coordinate their actions in order to maximize a global objective. Our domains are characterized by uncertainty, dynamism, and distributed information. Determining appropriate actions becomes quite difficult, especially as the number of agents and the coupling between them increases.

Journal of Artificial Intelligence Research, 1998

We examine the computational complexity of testing and nding small plans in probabilistic planning domains with both at and propositional representations. The complexity of plan evaluation and existence varies with the plan type sought; we examine totally ordered plans, acyclic plans, and looping plans, and partially ordered plans under three natural de nitions of plan value. We show that problems of interest are complete for a variety of complexity classes: PL, P, NP, co-NP, PP, NP PP , co-NP PP , and PSPACE. In the process of proving that certain planning problems are complete for NP PP , we introduce a new basic NP PP -complete problem, E-Majsat, which generalizes the standard Boolean satis ability problem to computations involving probabilistic quantities; our results suggest that the development of good heuristics for E-Majsat could be important for the creation of e cient algorithms for a wide variety of problems.

2006

Abstract When an agent evolves in a partially observable environment, it has to deal with uncertainties when choosing its actions. An efficient model for such environments is to use partially observable Markov decision processes (POMDPs). Many algorithms have been developed for POMDPs. Some use an offline approach, learning a complete policy before the execution. Others use an online approach, constructing the policy online for the current belief state.

Journal of Artificial Intelligence Research, 1998

We examine the computational complexity of testing and nding small plans in probabilistic planning domains with both at and propositional representations. The complexity of plan evaluation and existence varies with the plan type sought; we examine totally ordered plans, acyclic plans, and looping plans, and partially ordered plans under three natural de nitions of plan value. We show that problems of interest are complete for a variety of complexity classes: PL, P, NP, co-NP, PP, NP PP , co-NP PP , and PSPACE. In the process of proving that certain planning problems are complete for NP PP , we introduce a new basic NP PP -complete problem, E-Majsat, which generalizes the standard Boolean satis ability problem to computations involving probabilistic quantities; our results suggest that the development of good heuristics for E-Majsat could be important for the creation of e cient algorithms for a wide variety of problems.

Automatica, 2000

The purpose of this paper is twofold: (a) to provide a tutorial introduction to some key concepts from the theory of computational complexity, highlighting their relevance to systems and control theory, and (b) to survey the relatively recent research activity lying at the interface between these "elds. We begin with a brief introduction to models of computation, the concepts of undecidability, polynomial-time algorithms, NP-completeness, and the implications of intractability results. We then survey a number of problems that arise in systems and control theory, some of them classical, some of them related to current research. We discuss them from the point of view of computational complexity and also point out many open problems. In particular, we consider problems related to stability or stabilizability of linear systems with parametric uncertainty, robust control, time-varying linear systems, nonlinear and hybrid systems, and stochastic optimal control. . Also, the second author has greatly bene"tted from collaborating with Christos Papadimitriou over the years.