Partially observable Markov decision process

ABSTRACT A path-planning algorithm to guide unmanned aerial vehicles (UAVs) for tracking multiple ground targets based on the theory of partially observable Markov decision processes (POMDPs) is presented. A variety of features of interest are shown to be easy to incorporate into the framework by plugging in the appropriate models, which demonstrates the power and flexibility of the POMDP framework. Specifically, it is shown how to incorporate the following features by appropriately formulating the POMDP action space, transition law, and objective function: 1) control UAVs with both forward acceleration and bank angle subject to constraints; 2) account for the effect of wind disturbance on UAVs; 3) avoid collisions between UAVs and obstacles and among UAVs; 4) track targets while evading threats; 5) track evasive targets; and 6) mitigate track swaps.

Inventory control is among the most
important topics in operations research
because of large investments
in inventory and their effect on the
profitability of the firms. A systematic
analysis of inventory problems began
with the development of the classical
EOQ formula of Ford W. Harris in 1913,
and a substantial amount of research
was reported in 1958 by Kenneth J.
Arrow, Samuel Karlin, and Herbert Scarf.
A critical assumption in this vast literature
has been that the level of inventory
at any given time is fully observed. Some
of the most celebrated results such as the
optimality of basestock or (s,S) policies
have been obtained under the assumption
of full observation. Yet the inventory
level is often not fully observed in
practice, as elaborated below. In such an
environment of incomplete information,
inventories are known to be partially
observed and most of the well-known
inventory policies are not even admissible,
let alone optimal.

We consider the problem of joint network coding and packet scheduling for multimedia transmission from the Access Point (AP) to multiple receivers in 802.11 networks. The state of receivers is described by a hidden Markov model and the AP acts as a decision maker which employs ...

Theoretical frameworks like the Probability Ranking Principle and its more recent Interactive Information Retrieval variant have guided the development of ranking and retrieval algorithms for decades, yet they are not capable of helping us model problems in Dynamic Information Retrieval which exhibit the following three properties; an observable user signal, retrieval over multiple stages and an overall search intent. In this paper a new theoretical framework for retrieval in these scenarios is proposed. We derive a general dynamic utility function for optimizing over these types of tasks, that takes into account the utility of each stage and the probability of observing user feedback. We apply our framework to experiments over TREC data in the dynamic multi page search scenario as a practical demonstration of its effectiveness and to frame the discussion of its use, its limitations and to compare it against the existing frameworks.

This work tackles the problem of robust zero-shot planning in non-stationary stochastic environments. We study Markov Decision Processes (MDPs) evolving over time and consider Model-Based Reinforcement Learning algorithms in this setting. We make two hypotheses: 1) the environment evolves continuously with a bounded evolution rate; 2) a current model is known at each decision epoch but not its evolution. Our contribution can be presented in four points. 1) we define a specific class of MDPs that we call Non-Stationary MDPs (NSMDPs). We introduce the notion of regular evolution by making an hypothesis of Lipschitz-Continuity on the transition and reward functions w.r.t. time; 2) we consider a planning agent using the current model of the environment but unaware of its future evolution. This leads us to consider a worst-case method where the environment is seen as an adversarial agent; 3) following this approach, we propose the Risk-Averse Tree-Search (RATS) algorithm, a zero-shot Mod...

Inventory control is among the most important topics in operations research because of large investments in inventory and their effect on the profitability of the firms. A systematic analysis of inventory problems began with the development of the classical EOQ formula of Ford W. Harris in 1913, and a substantial amount of research was reported in 1958 by Kenneth J. Arrow, Samuel Karlin, and Herbert Scarf. A critical assumption in this vast literature has been that the level of inventory at any given time is fully observed. Some of the most celebrated ...

A critical threat to organizations, and the Internet itself, is a class of automated network attacks referred to as Internet worms. This article examines the use of mathematical models and optimization algorithms—specifi-cally a partially-observable Markov decision process (PO-MDP) ...

ABSTRACT Dynamic Spectrum Access is a key capability of Cognitive Radio (CR) networks to increase the efficiency in the use of the available spectrum resources. In this respect, this paper focuses on the spectrum selection problem when a number of radio links has to be established in a CR network to support applications with different bit rate requirements. A novel strategy based on a Partially Observable Markov Decision Process (POMDP) is proposed, whose target is to maximize a reward function that reflects the suitability of the available spectrum blocks to the application requirements. The proposed strategy combines partial observations of the interference state in the different spectrum blocks together with a statistical characterization of the interference dynamics. Thanks to this feature, the performance comparison of the algorithm against different reference strategies reveals that it achieves a very similar performance than a strategy operating under full knowledge of the real interference state of all the spectrum blocks, while at the same time it has much less requirements in terms of measurement needs and associated signaling.