ABSTRACT A path-planning algorithm to guide unmanned aerial vehicles (UAVs) for tracking multiple ground targets based on the theory of partially observable Markov decision processes (POMDPs) is presented. A variety of features of... more
ABSTRACT A path-planning algorithm to guide unmanned aerial vehicles (UAVs) for tracking multiple ground targets based on the theory of partially observable Markov decision processes (POMDPs) is presented. A variety of features of interest are shown to be easy to incorporate into the framework by plugging in the appropriate models, which demonstrates the power and flexibility of the POMDP framework. Specifically, it is shown how to incorporate the following features by appropriately formulating the POMDP action space, transition law, and objective function: 1) control UAVs with both forward acceleration and bank angle subject to constraints; 2) account for the effect of wind disturbance on UAVs; 3) avoid collisions between UAVs and obstacles and among UAVs; 4) track targets while evading threats; 5) track evasive targets; and 6) mitigate track swaps.
Inventory control is among the most important topics in operations research because of large investments in inventory and their effect on the profitability of the firms. A systematic analysis of inventory problems began with the... more
Inventory control is among the most important topics in operations research because of large investments in inventory and their effect on the profitability of the firms. A systematic analysis of inventory problems began with the development of the classical EOQ formula of Ford W. Harris in 1913, and a substantial amount of research was reported in 1958 by Kenneth J. Arrow, Samuel Karlin, and Herbert Scarf. A critical assumption in this vast literature has been that the level of inventory at any given time is fully observed. Some of the most celebrated results such as the optimality of basestock or (s,S) policies have been obtained under the assumption of full observation. Yet the inventory level is often not fully observed in practice, as elaborated below. In such an environment of incomplete information, inventories are known to be partially observed and most of the well-known inventory policies are not even admissible, let alone optimal.
Partially Observable Markov Decision Processes (POMDPs) have been met with great success in planning domains where agents must balance actions that provide knowledge and actions that provide reward. Recently, nonparametric Bayesian... more
Partially Observable Markov Decision Processes (POMDPs) have been met with great success in planning domains where agents must balance actions that provide knowledge and actions that provide reward. Recently, nonparametric Bayesian methods have been success- fully applied to POMDPs to obviate the need of a priori knowledge of the size of the state space, allowing to assume that the number of visited states may grow as the agent explores its environment. These approaches rely on the assumption that the agent’s environment remains stationary; however, in real-world scenarios the environment may change over time. In this work, we aim to address this inadequacy by introducing a dynamic nonparametric Bayesian POMDP model that both allows for automatic inference of the (distributional) representations of POMDP states, and for capturing non-stationarity in the modeled environments. Formulation of our method is based on imposition of a suitable dynamic hierarchical Dirichlet process (dHDP) prior over state transitions. We derive efficient algorithms for model inference and action planning and evaluate it on several benchmark tasks.
We consider the problem of joint network coding and packet scheduling for multimedia transmission from the Access Point (AP) to multiple receivers in 802.11 networks. The state of receivers is described by a hidden Markov model and the AP... more
We consider the problem of joint network coding and packet scheduling for multimedia transmission from the Access Point (AP) to multiple receivers in 802.11 networks. The state of receivers is described by a hidden Markov model and the AP acts as a decision maker which employs ...
Theoretical frameworks like the Probability Ranking Principle and its more recent Interactive Information Retrieval variant have guided the development of ranking and retrieval algorithms for decades, yet they are not capable of helping... more
Theoretical frameworks like the Probability Ranking Principle and its more recent Interactive Information Retrieval variant have guided the development of ranking and retrieval algorithms for decades, yet they are not capable of helping us model problems in Dynamic Information Retrieval which exhibit the following three properties; an observable user signal, retrieval over multiple stages and an overall search intent. In this paper a new theoretical framework for retrieval in these scenarios is proposed. We derive a general dynamic utility function for optimizing over these types of tasks, that takes into account the utility of each stage and the probability of observing user feedback. We apply our framework to experiments over TREC data in the dynamic multi page search scenario as a practical demonstration of its effectiveness and to frame the discussion of its use, its limitations and to compare it against the existing frameworks.
This work tackles the problem of robust zero-shot planning in non-stationary stochastic environments. We study Markov Decision Processes (MDPs) evolving over time and consider Model-Based Reinforcement Learning algorithms in this setting.... more
This work tackles the problem of robust zero-shot planning in non-stationary stochastic environments. We study Markov Decision Processes (MDPs) evolving over time and consider Model-Based Reinforcement Learning algorithms in this setting. We make two hypotheses: 1) the environment evolves continuously with a bounded evolution rate; 2) a current model is known at each decision epoch but not its evolution. Our contribution can be presented in four points. 1) we define a specific class of MDPs that we call Non-Stationary MDPs (NSMDPs). We introduce the notion of regular evolution by making an hypothesis of Lipschitz-Continuity on the transition and reward functions w.r.t. time; 2) we consider a planning agent using the current model of the environment but unaware of its future evolution. This leads us to consider a worst-case method where the environment is seen as an adversarial agent; 3) following this approach, we propose the Risk-Averse Tree-Search (RATS) algorithm, a zero-shot Mod...
Inventory control is among the most important topics in operations research because of large investments in inventory and their effect on the profitability of the firms. A systematic analysis of inventory problems began with the... more
Inventory control is among the most important topics in operations research because of large investments in inventory and their effect on the profitability of the firms. A systematic analysis of inventory problems began with the development of the classical EOQ formula of Ford W. Harris in 1913, and a substantial amount of research was reported in 1958 by Kenneth J. Arrow, Samuel Karlin, and Herbert Scarf. A critical assumption in this vast literature has been that the level of inventory at any given time is fully observed. Some of the most celebrated ...
A critical threat to organizations, and the Internet itself, is a class of automated network attacks referred to as Internet worms. This article examines the use of mathematical models and optimization algorithmsspecifi-cally a... more
A critical threat to organizations, and the Internet itself, is a class of automated network attacks referred to as Internet worms. This article examines the use of mathematical models and optimization algorithmsspecifi-cally a partially-observable Markov decision process (PO-MDP) ...
ABSTRACT Dynamic Spectrum Access is a key capability of Cognitive Radio (CR) networks to increase the efficiency in the use of the available spectrum resources. In this respect, this paper focuses on the spectrum selection problem when a... more
ABSTRACT Dynamic Spectrum Access is a key capability of Cognitive Radio (CR) networks to increase the efficiency in the use of the available spectrum resources. In this respect, this paper focuses on the spectrum selection problem when a number of radio links has to be established in a CR network to support applications with different bit rate requirements. A novel strategy based on a Partially Observable Markov Decision Process (POMDP) is proposed, whose target is to maximize a reward function that reflects the suitability of the available spectrum blocks to the application requirements. The proposed strategy combines partial observations of the interference state in the different spectrum blocks together with a statistical characterization of the interference dynamics. Thanks to this feature, the performance comparison of the algorithm against different reference strategies reveals that it achieves a very similar performance than a strategy operating under full knowledge of the real interference state of all the spectrum blocks, while at the same time it has much less requirements in terms of measurement needs and associated signaling.