Markov Decision Process

We study a decision problem faced by an energy limited wireless device that operates in discrete time. There is some external arrival to the device's transmit buffer. The possible decisions are: a) to serve some of the buffer content; b) to order a new battery after serving the maximum possible amount that it can; or c) to remain idle so that the battery charge can increase owing to diffusion process (which is possible in some commercially available batteries). We look at open and closed-loop controls of the system. The closed-loop control problem is addressed using the framework of Markov Decision Processes. We consider both finite and infinite horizon discounted costs as well as average cost minimization problems. Without using any second order characteristics, we obtain results that include i) optimality of bang-bang control, ii) the optimality of threshold based policies, iii) parametric monotonicity of the threshold, and iv) uniqueness of the threshold. For the open-loop control setting we use recent advances in application of multimodular functions to establish optimality of bracket sequence based control.

— This paper presents a method for short-term electricity price forecasting based on combination of the Monte Carlo simulation and Markov chains. The method provides an estimation of the probabilities of various electricity price ranges, average prices, and probabilities of the highest price range, for each hour of the next 24 hours. The external variables have been implicitly accounted for through the Monte Carlo simulation. Using the market data of the European Power Exchange (EPEX) as a test case, the effectiveness of the proposed method has been verified by comparison with the best regression methods.

A major issue in supply chain inventory management is the coordination of inventory policies adopted by di!erent supply chain actors, such as suppliers, manufacturers, distributors, so as to smooth material #ow and minimize costs while responsively meeting customer demand. This paper presents an approach to manage inventory decisions at all stages of the supply chain in an integrated manner. It allows an inventory order policy to be determined, which is aimed at optimizing the performance of the whole supply chain. The approach consists of three techniques: (i) Markov decision processes (MDP) and (ii) an arti"cial intelligent algorithm to solve MDPs, which is based on (iii) simulation modeling. In particular, the inventory problem is modeled as an MDP and a reinforcement learning (RL) algorithm is used to determine a near optimal inventory policy under an average reward criterion. RL is a simulation-based stochastic technique that proves very e$cient particularly when the MDP size is large.

the physical phenomena. By this method, the students enhance their problem-solving abilities with minimal programming skills. By using examples, the paper presents an approach to computer-aided problem-solving methods for junior-level courses. The methods described in the paper have proven to be of value to students studying electric machines and power engineering at Arizona State University. Abstract: This paper describes modeling and simulation software developed specifically for teaching protective relaying application and design concepts. The emphasis was on implementing user-friendly and open-system solutions that will allow an easy use and straight-forward future expansion. This is achieved by introducing new libraries of signal sources and relay elements developed for the Simulink environment of MATLAB. Combined with the Power Block Set (PBS) toolbox of MATLAB, the mentioned libraries allow for a variety of studies aimed at a better understanding of protective relay design approaches and related applications.

Limited battery power at wireless video sensor nodes, along with the transmission quality requirements for video data, makes quality-of-service (QoS) provisioning in a wireless video sensor network a very challenging task. In this paper, a dynamic power-management framework is proposed for a wireless video sensor node to improve the energy-saving performance while satisfying video transmission quality requirements. This framework is developed based on a Markov decision process that considers the video traffic arrival process in the sensor node, the sleep and wakeup processes in the camera and wireless transceiver electronics, the queue status, and the wireless channel condition. A dynamic programming approach is used to find the optimum policy to achieve the desired performance measures in an energylimited sensor node.

Sepsis, the tenth-leading cause of death in the United States, accounts for more than $16.7 billion in annual health care costs. A significant factor in these costs is hospital length of stay. The lack of standardized hospital discharge policies and an inadequate understanding of sepsis progression have resulted in unnecessarily long hospital lengths of stay. In this paper, a general model of when to discharge a patient with pneumonia-related sepsis from the hospital is presented. The model is parameterized using patient-based disease progression data from a large clinical study in order to characterize optimal discharge policies for various problem instances. In the presented experiments, patient health is represented by SOFA scores, which are commonly used to assess sepsis severity. Control-limit policies for specific patient cohorts defined by age and race are demonstrated.

This book, written in Portuguese, deals with Operational Research topics, such as linear programming models and direct techniques for solving Ax = b systems, revised simplex method, Markov processes and queuing systems, in addition to Monte Carlo simulation.

Example 1. Consider the two-state, continuous-time Markov process with transition rate diagram for some positive constants A and B. The generator matrix is given by Q = −A A B −B. Solve the forward Kolmogorov equation for a given initial distribution π(0). Solution. The equation for π 1 (t) is ∂π 1 (t) ∂t = −Aπ 1 (t) + Bπ 2 (t); π 1 (0) given. But π 1 (t) = 1 − π 2 (t) so ∂π 1 (t) ∂t = −(A + B)π 1 (t) + B; π 1 (0) given. By differentiation we check that this equation has the solution π t (t) = π 1 (0)e −(A+B)t + B A + B (1 − e −(A+B)t) so that π(t) = π(0)e −(A+B)t + B A + B , A A + B (1 − e −(A+B)t). For any initial distribution π(0) lim t→∞ π(t) = B A + B , A A + B. The rate of convergence is exponential, with rate parameter A + B and the limiting distribution is the unique probability distribution satisfying πQ = 0. Example 2. Let (X t : t ≥ 0) be a time-homogeneous Markov process with the transition rate diagram

The theory of Markov Decision Processes is the theory of controlled Markov chains. Its origins can be traced back to R. Bellman and L. Shapley in the 1950's. During the decades of the last century this theory has grown dramatically. It has found applications in various areas like e.g. computer science, engineering, operations research, biology and economics. In this article we give a short introduction to parts of this theory. We treat Markov Decision Processes with finite and infinite time horizon where we will restrict the presentation to the so-called (generalized) negative case. Solution algorithms like Howard's policy improvement and linear programming are also explained. Various examples show the application of the theory. We treat stochastic linear-quadratic control problems, bandit problems and dividend pay-out problems. Keywords Markov decision process • Markov chain • Bellman equation • Policy improvement • Linear programming Mathematics Subject Classification (2010) 90C40 • 60J05 • 93E20 We dedicate this paper to Karl Hinderer who passed away on April 17th, 2010. He established the theory of Markov Decision Processes in Germany 40 years ago.

Semiconductor industry is very capital intensive in which capacity utilization significantly affect the capital effectiveness and profitability of semiconductor manufacturing companies. Due to constant technology advance driven by Moore's Law in semiconductor industry, multiple production technologies generally co-exist in a wafer fabrication facility with utilization of a pool of common tools for multiple technologies and critical tools dedicated for a specific technology. Because part of the equipment is common for products of different technologies, production managers have limited flexibility to dynamically allocate the capacity among the technologies via capacity migration. The possibility of capacity migration and interrelationship among different technologies make capacity planning difficult under demand and product-mix uncertainties.

The majority of learning algorithms available today focus on approximating the state (V ) or state-action (Q) value function and efficient action selection comes as an afterthought. On the other hand, real-world problems tend to have large action spaces, where evaluating every possible action becomes impractical. This mismatch presents a major obstacle in successfully applying reinforcement learning to real-world problems. In this paper we present an effective approach to learning and acting in domains with multidimensional and/or continuous control variables where efficient action selection is embedded in the learning process. Instead of learning and representing the state or stateaction value function of the MDP, we learn a value function over an implied augmented MDP, where states represent collections of actions in the original MDP and transitions represent choices eliminating parts of the action space at each step. Action selection in the original MDP is reduced to a binary search by the agent in the transformed MDP, with computational complexity logarithmic in the number of actions, or equivalently linear in the number of action dimensions. Our method can be combined with any discrete-action reinforcement learning algorithm for learning multidimensional continuous-action policies using a state value approximator in the transformed MDP. Our preliminary results with two well-known reinforcement learning algorithms (Least-Squares Policy Iteration and Fitted Q-Iteration) on two continuous action domains (1-dimensional inverted pendulum regulator, 2-dimensional bicycle balancing) demonstrate the viability and the potential of the proposed approach.

Recently, it has been recognized that revenue management of cruise ships is different from that of airlines or hotels. Among the main differences is the presence of multiple capacity constraints in cruise ships, i.e., the number of cabins in different categories and the number of lifeboat seats, versus a single constraint in airlines and hotels (i.e., number of seats or rooms). We develop a discrete-time dynamic capacity control model for a cruise ship characterized by multiple constraints on cabin and lifeboat capacities. Customers (families) arrive sequentially according to a stochastic process and request one cabin of a certain category and one or more lifeboat seats. The cruise ship revenue manager decides which requests to accept based on the remaining cabin and lifeboat capacities at the time of an arrival as well as the type of the arrival. We show that the opportunity cost of accepting a customer is not always monotone in the reservation levels or time. This non-monotone behavior implies that ''conventional" booking limits or critical time periods capacity control policies are not optimal. We provide analysis and insights justifying the non-monotone behavior in our cruise ship context. In the absence of monotonicity, and with the optimal solution requiring heavy storage for ''large" (industry-size) problems, we develop several heuristics and thoroughly test their performance, via simulation, against the optimal solution, well-crafted upper bounds, and a first-come first-served lower bound. Our heuristics are based on rolling-up the multidimensional state space into one or two dimensions and solving the resulting dynamic program (DP). This is a strength of our approach since our DP-based heuristics are easy to understand, solve and analyze. We find that single-dimensional heuristics based on decoupling the cabins and lifeboat problems perform quite well in most cases.

We formulate and analyze a Markov decision process (dynamic programming) model for airline seat allocation (yield management) on a single-leg flight with multiple fare classes. Unlike previous models, we allow cancellation, no-shows, and overbooking. Additionally, we make no assumptions on the arrival patterns for the various fare classes. Our model is also applicable to other problems of revenue management with perishable commodities, such as arise in the hotel and cruise industries. We show how to solve the problem exactly using dynamic programming. Under realistic conditions, we demonstrate that an optimal booking policy is characterized by state-and time-dependent booking limits for each fare class. Our approach exploits the equivalence to a problem in the optimal control of admission to a queueing system, which has been well studied in the queueing-control literature. Techniques for efficient implementation of the optimal policy and numerical examples are also given. In contrast to previous models, we show that 1) the booking limits need not be monotonic in the time remaining until departure; 2) it may be optimal to accept a lower-fare class and simultaneously reject a higher-fare class because of differing cancellation refunds, so that the optimal booking limits may not always be nested according to fare class; and 3) with the possibility of cancellations, an optimal policy depends on both the total capacity and the capacity remaining. Our numerical examples show that revenue gains of up to 9% are possible with our model, compared with an equivalent model omitting the effects of cancellations and no-shows. We also demonstrate the computational feasibility of our approach using data from a real-life airline application.

The focus of this work is the computation of efficient strategies for commodity trading in a multi-market environment. In today's "global economy" commodities are often bought in one location and then sold (right away, or after some storage period) in different markets. Thus, a trading decision in one location must be based on expectations about future price curves in all other relevant markets, and on current and future storage and transportation costs. Investors try to compute a strategy that maximizes expected return, usually with some limitations on assumed risk. With standard stochastic assumptions on commodity price fluctuations, computing an optimal strategy can be modeled as a Markov decision process (MDP). However, in general such a formulation does not lead to efficient algorithms. In this work we propose a model for representing the multi-market trading problem and show how to obtain efficient structured algorithms for computing optimal strategies for a number of commonly used trading objective functions (Expected NPV, Mean-Variance, and Value at Risk).

We introduce the concept of a Markov risk measure and we use it to formulate risk-averse control problems for two Markov decision models: a finite horizon model and a discounted infinite horizon model. For both models we derive risk-averse dynamic programming equations and a value iteration method. For the infinite horizon problem we develop a risk-averse policy iteration method and we prove its convergence. We also propose a version of the Newton method to solve a nonsmooth equation arising in the policy iteration method and we prove its global convergence. Finally, we discuss relations to min-max Markov decision models.

Material transportation is one of the most important aspects of open-pit mine operations. The problem usually involves a truck dispatching system in which decisions on truck assignments and destinations are taken in real-time. Due to its signicance, several
decision systems for this problem have been developed in the last few years, improving productivity and reducing operating costs. As in many other real-world applications, the assessment and correct modeling of uncertainty is a crucial requirement as the unpredictability originated from equipment faults, weather conditions, and human mistakes, can often result in truck queues or idle shovels. However, uncertainty is not considered in most commercial dispatching systems. In this thesis, we introduce novel truck dispatching systems as a starting point to modify the current practices with a statistically principled decision making methodology. First, we present a stochastic method using Time-
Dependent Markov Decision Process (TiMDP) applied to the truck dispatching problem. In the TiMDP model, travel times are represented as probabilistic density functions (pdfs), time-windows can be inserted for paths availability, and time-dependent utility can be used as a priority parameter. In order to minimize the well-known curse of dimensionality issue, to which multi-agent problems are subject when considering discrete state modelings, the system is modeled based on the introduced single-dependent-agents. Based also on the single-dependent-agents concept, we introduce the Genetic TiMDP (G-TiMDP) applied to the truck dispatching problem. This method is a hybridization of the TiMDP model and of a Genetic Algorithm (GA), which is also used to solve the truck dispatching problem. Finally, in order to evaluate and compare the results of the introduced methods, we execute Monte Carlo simulations in a example heterogeneous mine composed by 15 trucks, 3 shovels, and 1 crusher. The uncertain aspect of the problem is represented by the path selection through crusher and shovels, which is executed by the truck driver, being independent of the dispatching system. The results are compared to classical dispatching approaches (Greedy Heuristic and Minimization of Truck Cycle Times { MTCT) using Student's T-test, proving the eciency of the introduced truck dispatching methods.

We study the 'Creation of Pooling in Inventory and Queueing Models'. This research consists of the study of sharing a scarce resource (such as inventory, server capacity, or production capacity) between multiple customer classes. This is called pooling, where the goal is to achieve cost or waiting time reductions. For the inventory and queueing models studied, both theoretical, scientific insights are generated, as well as strategies which are applicable in practice.

This monograph consists of two parts: pooling and polling . In the first part, pooling is applied to multi-location inventory models. It is studied how cost reduction can be achieved by the use of stock transfers between local warehouses, so-called lateral transshipments. In this way, stock is pooled between the warehouses. The setting is motivated by a spare parts inventory network, where critical components of technically advanced machines are kept on stock, to reduce down time durations. We create insights into the question when lateral transshipments lead to cost reductions, by studying several models.

Firstly, a system with two stock points is studied, for which we completely characterize the structure of the optimal policy, using dynamic programming. For this, we formulate the model as a Markov decision process. We also derived conditions under which simple, easy to implement, policies are always optimal, such as a hold back policy and a complete pooling policy. Furthermore, we identified the parameter settings under which cost savings can be achieved. Secondly, we characterize the optimal policy structure for a multi-location model where only one stock point issues lateral transshipments, a so-called quick response warehouse. Thirdly, we apply the insights generated to the general multi-location model with lateral transshipments. We propose the use of a hold back policy, and construct a new approximation algorithm for deriving the performance characteristics. It is based on the use of interrupted Poisson processes. The algorithm is shown to be very accurate, and can be used for the optimization of the hold back levels, the parameters of this class of policies. Also, we study related inventory models, where a single stock point servers multiple customers classes.

Furthermore, in the first part, the pooling of server capacity is studied. For a two queue model where the head-of-line processor sharing discipline is applied, we derive the optimal control policy for dividing the servers attention, as well as for accepting customers. Also, a server farm with an infinite number of servers is studied, where servers can be turned off after a service completion in order to save costs. We characterize the optimal policy for this model.

In the second part of the thesis, polling models are studied, which are queueing systems where multiple queues are served by a single server. An application is the production of multiple types of products on a single machine. In this way, the production capacity is pooled between the product types. For the classical polling model, we derive a closed-form approximation for the mean waiting time at each of the queues. The approximation is based on the interpolation of light and heavy traffic results. Also, we study a system with so-called smart customers, where the arrival rate at a queue depends on the position of the server. Finally, we invent two new service disciplines (the gated/exhaustive and the k-gated discipline) for polling models, designed to yield 'fairness and efficiency' in the mean waiting times. That is, they result in almost equal mean waiting times at each of the queues, without increasing the weighted sum of the mean waiting times too much.


Slides of presentation during PhD defense: http://tue.academia.edu/SandraVanWijk/Talks/82570/Pooling_and_Polling_Creation_of_Pooling_in_Inventory_and_Queueing_Models

A major issue in supply chain inventory management is the coordination of inventory policies adopted by di!erent supply chain actors, such as suppliers, manufacturers, distributors, so as to smooth material #ow and minimize costs while responsively meeting customer demand. This paper presents an approach to manage inventory decisions at all stages of the supply chain in an integrated manner. It allows an inventory order policy to be determined, which is aimed at optimizing the performance of the whole supply chain. The approach consists of three techniques: (i) Markov decision processes (MDP) and (ii) an arti"cial intelligent algorithm to solve MDPs, which is based on (iii) simulation modeling. In particular, the inventory problem is modeled as an MDP and a reinforcement learning (RL) algorithm is used to determine a near optimal inventory policy under an average reward criterion. RL is a simulation-based stochastic technique that proves very e$cient particularly when the MDP size is large.

We provide a tutorial on the construction and evaluation of Markov decision processes (MDPs), which are powerful analytical tools used for sequential decision making under uncertainty that have been widely used in many industrial and manufacturing applications but are underutilized in medical decision making (MDM). We demonstrate the use of an MDP to solve a sequential clinical treatment problem under uncertainty. Markov decision processes generalize standard Markov models in that a decision process is embedded in the model and multiple decisions are made over time. Furthermore, they have significant advantages over standard decision analysis. We compare MDPs to standard Markov-based simulation models by solving the problem of the optimal timing of living-donor liver transplantation using both methods. Both models result in the same optimal transplantation policy and the same total life expectancies for the same patient and living donor. The computation time for solving the MDP model is significantly smaller than that for solving the Markov model. We briefly describe the growing literature of MDPs applied to medical decisions.

Q-learning is a simple, powerful algorithm for behavior learn- ing. It was derived in the context of single agent decision making in Markov decision process environments, but its applicability is much broader—in experiments in multia- gent environments, Q-learning has also performed well. Our preliminary analysis using dynamical systems finds that Q- learning's indirect control of behavior via estimates of value

Markov Chains provide support for problems involving decisionon uncertainties through a continuous period of time. The greateravailability and access to processing power through computersallow that these models can be used more often to represent clinicalstructures. Markov models consider the patients in a discrete stateof health, and the events represent the transition from one stateto another. The possibility of modeling repetitive events and timedependence of probabilities and utilities associated permits a moreaccurate representation of the evaluated clinical structure. Thesetemplates can be used for economic evaluation in health care takinginto account the evaluation of costs and clinical outcomes, especiallyfor evaluation of chronic diseases. This article provides a review ofthe use of modeling within the clinical context and the advantages ofthe possibility of including time for this type of study.

Board games are often taken as examples to teach decision-making algorithms in artificial intelligence (AI). These algorithms are generally presented with a strong focus on winning the game. Unfortunately, a few important aspects, such as the gaming experience of human players, are often missing from the equation. This paper presents a simple board game we use in an introductory course in AI to initiate students to the gaming experience issue. The Snakes and Ladders game has been modified to provide different levels of challenges for students. The game with such modifications offers theoretical, algorithmic and programming challenges. One of the most complex is the generation of an optimal policy to provide a fair challenge to an opponent. A solution based on Markov Decision Processes (MDPs) is presented. This approach relies on a simple model of the opponent's playing behaviour.

Cet article présente les résultats expérimentaux obtenus avec une architecture originale permettant un apprentissage générique dans le cadre de processus décisionnels de Markov factorisés observables dans le désordre (PDMFOD). L'article décrit tout d'abord le cadre formel des PDMFOD puis le fonctionnement de l'algorithme, notamment le principe de parallélisation et l'attribution dynamique des récompenses. L'architecture est ensuite appliquée à deux problèmes de navigation, l'un dans un labyrinthe et l'autre dans un trafic routier (New York Driving). Les tests montrent que l'architecture permet effectivement d'apprendre une politique de décisions performante et générique malgré le nombre élevé de dimensions des espaces d'états des deux systèmes.

An approach to real-time control of a network of signalized intersections is proposed based on a discrete time, stationary, Markov control model (also known as Markov decision process or Markov dynamic programming). The approach incorporates microscopic simulation of actuated controller output signals in response to probabilistic forecasts of individual vehicle actuations at downstream inductance loop detectors derived from a macroscopic link transfer function. An Artificial Neural Network representation of vehicle delay estimations is proposed and tested for approximate real-time evaluation of potential traffic signal transitions at three-second evaluation intervals. A series of off-line tests of the developed procedures are applied to a simplified network of five intersections: these tests provide promising indications of this approach.

Many market participants now employ algorithmic trading, commonly defined as the use of computer algorithms to automatically make certain trading decisions, submit orders, and manage those orders after submission. Identifying and understanding the impact of algorithmic trading on financial markets has become a critical issue for market operators and regulators. Advanced data feed and audit trail information from market operators now make the full observation of market participants' actions possible. A key question is the extent to which it is possible to understand and characterize the behavior of individual participants from observations of trading actions.

We provide a method, based on the theory of Markov decision processes, for e cient planning in stochastic domains. Goals are encoded as reward functions, expressing the desirability of each world state; the planner must nd a policy (mapping from states to actions) that maximizes future rewards. Standard goals of achievement, as well as goals of maintenance and prioritized combinations of goals, can be speci ed in this way. An optimal policy can be found using existing methods, but these methods require time at best polynomial in the number of states in the domain, where the number of states is exponential in the number of propositions (or state variables). By using information about the starting state, the reward function, and the transition probabilities of the domain, we restrict the planner's attention to a set of world states that are likely to be encountered in satisfying the goal. Using this restricted set of states, the planner can generate more or less complete plans depending on the time it has available.

2015): Gaussian process-based algorithmic trading strategy identification, Quantitative Finance, Many market participants now employ algorithmic trading, commonly defined as the use of computer algorithms, to automatically make certain trading decisions, submit orders and manage those orders after submission. Identifying and understanding the impact of algorithmic trading on financial markets has become a critical issue for market operators and regulators. Advanced data feeds and audit trail information from market operators now allow for the full observation of market participants' actions.

Decision-making in an environment of uncertainty and imprecision for real-world problems is a complex task. In this paper it is introduced general finite state fuzzy Markov chains that have a finite convergence to a stationary (may be periodic) solution. The Cesaro average and the -potential for fuzzy Markov chains are defined, then it is shown that the relationship between them corresponds to the Blackwell formula in the classical theory of Markov decision processes. Furthermore, it is pointed out that recurrency does not necessarily imply ergodicity. However, if a fuzzy Markov chain is ergodic, then the rows of its ergodic projection equal the greatest eigen fuzzy set of the transition matrix. Then, the fuzzy Markov chain is shown to be a robust system with respect to small perturbations of the transition matrix, which is not the case for the classical probabilistic Markov chains. Fuzzy Markov decision processes are finally introduced and discussed.

This paper¯rst explores the decision-making process in agile teams using scrum practices and second identi¯es factors that in°uence the decision-making process during the Sprint Planning and Daily Scrum Meetings. We conducted 34 semi-structured interviews and 18 observations across four agile teams. Our¯ndings show that a rational decision-making process is sometimes followed in the Sprint Planning and Daily Scrum Meetings and that three factors can in°uence the rational decision-making process: sprint duration, experience and resource availability. Additionally, decisions are not always made in a collaborative manner by team members. This research contributes to the decision-making literature and project management literature by highlighting di±culties pertinent to decision making in agile teams.

Recent trends in the commercial aviation industry have resulted in rapidly increasing complexity and decentralisation in service parts logistics systems. As a consequence, MRO service providers tend to adopt more flexible strategies, such as service parts sourcing and demand fulfilment for customers with different service-level requirements. The MRO service providers often enter into cooperative agreements with other service providers to pool inventories, enabling them to increase their flexibility in delivering services to multiple airlines with different contractual terms. Although using cooperative strategies, such as emergency resupply, is useful to increase flexibility, the inherent complexity of optimal mechanism is a critical issue that needs to be further investigated. To this aim, we consider a repairable service parts inventory system with multi-customer classes and develop an optimal emergency resupply policy. Following this, to overcome the intractability issue of finding the optimal policy, an efficient approximation method is proposed. Numerical results indicate that the proposed approximation method is highly accurate, and leads to a significant costs reduction. This paper sheds light on the effectiveness of emergency resupply policy that improves MRO service providers' flexibility and enables them to ensure responsive service parts inventory.

Android devices provide opportunities for users to install third-party applications through various online markets. This brings security and privacy concerns to the users since thirdparty applications may pose serious threats. The exponential growth and diversity of these applications render conventional defenses ineffective, thus, Android smartphones often remain unprotected from novel malware. In this work, we present XDroid, an Android app and resource risk assessment framework using hidden Markov model. In this framework, we first map the applications' behaviors into an observation set, and we introduce a novel approach to attach timestamp to some observations to improve the accuracy of the model. We show that our HMM can be utilized to generates risk alerts to users when suspicious behaviors are found. Furthermore, an online learning model is introduced to enable the integration of the input from users and provide adaptive risk assessment to meet user's preferences. We evaluate our model through a set of experiments on a benchmark malware dataset DREBIN. Our experimental results demonstrate that the proposed model can assess malicious apps risk-levels with high accuracy. It also provide adaptive risk assessment based on input from users.

The development of service robots has recently received considerable attention. Their deployment, however, normally involves a substantial programming effort to develop a particular application. With the incorporation of service robots to daily activities, it is expected that they will require to perform different tasks. Fortunately, many of such applications share common modules such as navigation, localization and human interaction, among others. In this chapter, a general framework to easily develop different applications for service robots is presented. In particular, we have developed a set of general purpose modules for common tasks that can be easily integrated into a distributed, layered architecture, and coordinated by a decision-theoretic planner to perform different tasks. The coordinator is based on a Markov decision process (MDP) whose reward is set according to the task's goal, the states are represented by a set of variables affected by the general modules, and the actions correspond to the execution of the different modules. In order to create a new application the user only needs to define a new MDP whose solution provides an optimal policy that coordinates the different behaviors for performing the task. The effectiveness of our approach is experimentally demonstrated in four different service robot tasks with very promising results. Additionally, several aspects include some novel ideas; in particular in navigation, localization and gesture recognition.

The problem of optimal policy formulation for teams of resource-limited agents in stochastic environments is composed of two strongly-coupled subproblems: a resource allocation problem and a policy optimization problem. We show how to combine the two problems into a single constrained optimization problem that yields optimal resource allocations and policies that are optimal under these allocations. We model the system as a multiagent Markov decision process (MDP), with social welfare of the group as the optimization criterion. The straightforward approach of modeling both the resource allocation and the actual operation of the agents as a multiagent MDP on the joint state and action spaces of all agents is not feasible, because of the exponential increase in the size of the state space. As an alternative, we describe a technique that exploits problem structure by recognizing that agents are only loosely-coupled via the shared resource constraints. This allows us to formulate a constrained policy optimization problem that yields optimal policies among the class of realizable ones given the shared resource limitations. Although our complexity analysis shows the constrained optimization problem to be NP-complete, our results demonstrate that, by exploiting problem structure and via a reduction to a mixed integer program, we are able to solve problems orders of magnitude larger than what is possible using a traditional multiagent MDP formulation.

We study the problem of learning near-optimal behavior in finite Markov Decision Processes (MDPs) with a polynomial number of samples. These "PAC-MDP" algorithms include the wellknown E 3 and R-MAX algorithms as well as the more recent Delayed Q-learning algorithm. We summarize the current state-of-the-art by presenting bounds for the problem in a unified theoretical framework. A more refined analysis for upper and lower bounds is presented to yield insight into the differences between the model-free Delayed Q-learning and the model-based R-MAX.

We consider the problem of optimally parking empty cars in an elevator group so as to anticipate and intercept the arrival of new passengers and minimize their waiting times. Two solutions are proposed, for the down-peak and up-peak traffic patterns. We demonstrate that matching the distribution of free cars to the arrival distribution of passengers is sufficient to produce savings of up to 80 % in down-peak traffic. Since this approach is not useful for the much harder case of up-peak traffic, we propose a solution based on the representation of the elevator system as a Markov decision process (MDP) model with relatively few aggregated states, and determination of the optimal parking policy by means of dynamic programming on the MDP model. Abstract-We consider the problem of optimally parking empty cars in an elevator group so as to anticipate and intercept the arrival of new passengers and minimize their waiting times. Two solutions are proposed, for the down-peak and up-peak traffic patterns. We demonstrate that matching the distribution of free cars to the arrival distribution of passengers is sufficient to produce savings of up to 80% in down-peak traffic. Since this approach is not useful for the much harder case of up-peak traffic, we propose a solution based on the representation of the elevator system as a Markov decision process (MDP) model with relatively few aggregated states, and determination of the optimal parking policy by means of dynamic programming on the MDP model.

With standard assumptions the routing and wavelength assignment problem (RWA) can be viewed as a Markov Decision Process (MDP). The problem, however, defies an exact solution because of the huge size of the state space. Only heuristic algorithms have been presented up till now. In this paper we propose an approach where, starting from a given heuristic algorithm, one obtains a better algorithm by the first policy iteration. In order to estimate the relative costs of states, we make a simulation on the fly studying, at each decision epoch, the consequences of all the alternatives actions. Being computationally intensive, this method can be used in real time only for systems with slow dynamics. Off-line it can be used to assess how close the heuristic algorithms come to the optimal policy. Numerical examples are given about the policy improvement.

Compared to the wave's intelligence we are equal to a goldfish. This I say not in jest but because it has taken us millions of years to finally arrive at this day where we find the simple truth it isn't even hidden, it never really was, just like children we were kept out of the pantry with good reason. In this paper, you will find out what the universe is exactly how it functions at every level and you will be able to communicate directly with the main system passing information into the creation process via a real code and very real system provided by the universe. Once I understood how the system worked I was able to understand a good part of the code. I don't know what the consequences will be of us moving out of our very limited reality. I think it is time we do something because the solution to every problem we face today and tomorrow is over there on the core systems processing side. I am the only one so far who has glimpsed the difference in our side and the main system side and well let's just say I now feel trapped and confined into a tiny domain with by comparison has extremely limited options. After glimpsing the other side it became clear that we are confined to a tiny part of reality and left in ignorance until we mature. The inability to capture or calculate the wave collapse, made us think that nothing important was happening in the wave collapse. It is unfathomable to think that so much information and energy could be made in the 1 −42 seconds it takes for the wave to collapse. It would be impossible unless there was a fundamental core process that was handling the wave collapse setting up the platform and delivering information during the collapse to the environment. The wave collapse is how the system creates events and information. The group wave intelligence of the universe acts as a central processing core that runs all the background processes that are needed for the software or user environment to work. Without these processes running in the background our reality would not function. Through these processes Infinite amounts of data are created in the fraction of a second it takes for the wave to collapse. It is not a collapse but a perfectly controlled system. A new breakthrough has allowed us into this hidden system.

In the Probabilistic I/O Automata (PIOA) framework, nondeterministic choices are resolved using perfect-information schedulers, which are similar to history-dependent policies for Markov decision processes (MDPs). These schedulers are too powerful in the setting of security analysis, leading to unrealistic adversarial behaviors. Therefore, we introduce in this paper a novel mechanism of task partitions for PIOAs. This allows us to define partial-information adversaries in a systematic manner, namely, via sequences of tasks. The resulting task-PIOA framework comes with simple notions of external behavior and implementation, and supports simple compositionality results. A new type of simulation relation is defined and proven sound with respect to our notion of implementation. To illustrate the potential of this framework, we summarize our verification of an Oblivious Transfer protocol, where we combine formal and computational analyses. Finally, we present an extension with extra expressive power, using local schedulers of individual components.

In supervised learning scenarios, feature selection has been studied widely in the literature. Here, feature selection is considered as an empirical strategy of restricting state space and lessen the complexity of hypothesis. In this work we introduce the environment as a one player game and improve a reinforcement learning method to traverse the state space and learn from experiments. In this way we exert a Monte Carlo based method, and adapt the problem with the requirements of such procedure. As we need to reward each action ...

In this paper we introduce a stochastic model for dialogue systems based on Markov decision process. Within this framework we show that the problem of dialogue strategy design can be stated as an optimization problem, and solved by a variety of methods, including the reinforcement learning approach. The advantages of this new paradigm include objective evaluation of dialogue systems and their automatic design and adaptation. We show some preliminary results on learning a dialogue strategy for an Air Travel Information System.