POMDPs

This research attempts to propose an approach to make decisions under uncertainty for designing a dialogue manager for Amharic spoken dialogue system, Amharic as under-resourced language. A prototype Amharic Spoken Dialogue System was implemented on hotel and restaurant address information domain for experimentation. Data for this research was collected through a method called Wizard of Oz and domain knowledge is prepared using address search websites. How to design a dialogue manager which is robust for languages especially with low performing Automatic Speech Recognition unit is a fundamental question of this research. Previous studies on designing of spoken dialogue system for under resourced languages focused mainly on Automatic Speech Recognizer. We reviewed methods and frameworks on dialogue management. Design of Partially Observable Markov Decision Processes (POMDP) based dialogue manager, which provides a principled framework to plan under uncertainty, yields robustness. Low performing Spoken Dialogue System (SDS) components, especially the Automatic Speech Recognizer (ASR) were considered the causes of uncertainty. Maintaining multiple hypotheses (evidences) improves the correctness of the dialogue manager. We conducted experiments to test correctness score, error rate and robustness. With a maximum of 6 N-best list and 20 partitions the correctness score grow by 14.19%. Increasing the number of n-best list of 6 reduced the error rate by 5.78% and with 6 n-best list. The belief updated below 0.12 seconds with 20 partitions and 6-nbest list. And the dialogue manager was able to complete a task with an average 8.75 turns by 50% Word Error Rate. The finding from the research illustrates that POMDP-based design approach to dialogue management is robust and possible to develop an improving spoken dialogue system for under-resourced languages.

A primary goal in robotics research is to provide means for mobile platforms to perform autonomously within their environment. Depending on the task at hand, autonomous performance can be defined as the execution by the robot, without human intervention, of certain navigational tasks. In mobile robotics literature, commonly addressed navigation tasks include the localization, mapping, path planning and obstacle avoidance tasks. Solving any of these tasks is a hard problem by itself. The reason stems from the inherent complexity associated with both the robot and its environment, each of them being an extremely complex dynamical system with uncertainty involved.

In this thesis, we propose a probabilistic framework for mobile robot navigation in dynamic environments based on the Partially Observable Markov Decision Process (POMDP) model. The proposed model is able to perform the navigation tasks of localization, path planning and obstacle avoidance. POMDPs are models for sequential decision making where the world in which the robot operates is partially observable, i.e. the true underlying robot's state is not known, and the outcome of actions it executes is modelled probabilistically. As such POMDPs perform localization and path planning in a probabilistic manner.

POMDPs have the major shortcoming of their extreme computational complexity and hence they have been mainly used in robotics as high level path planners only. In this thesis, we propose a novel hierarchical representation of POMDPs, specifically designed for the autonomous robot navigation problem and termed as the Robot Navigation-Hierarchical POMDP (RN-HPOMDP). The proposed hierarchical POMDP can efficiently model large real-world environments and is amenable to real time solution. This is achieved mainly due to the design choice of modelling the state transition and observation functions dependent only on the robot motion model and not on the environment as it is commonly used in the POMDP literature. Furthermore, the notion of the reference POMDP (rPOMDP) is introduced that infers the robot motion model in a very small POMDP and it transfers this information to the hierarchical structure while being solved. The environment specific information is modelled within the reward function of the RN-HPOMDP. The employed model is utilized as a unified probabilistic navigation framework that accommodates for localization, path planning and obstacle avoidance. Hence, real-time solution of the RN-HPOMDP is essential since no other external modules are utilized and paths have to be replanned at each time step.

The RN-HPOMDP has been developed for the application of robot navigation in dynamic real-world environments that are highly populated. Thus, it is desirable for the robot to perform obstacle avoidance in a manner that resembles the human motion for obstacle avoidance. That is, the robot should be able to decide the most suitable obstacle avoidance behavior based on the state of the environment. Therefore, the robot can decide to either perform a detour or follow a completely new path to the goal and also modify its speed of movement (increase it or decrease it) to bypass an obstacle or let it move away respectively. Any of the above four distinct behaviors for obstacle avoidance should be decided well before the robot comes too close to the obstacle. For that reason, future motion prediction of obstacles is employed. Two kinds of prediction are utilized: short-term and long-term prediction. Short term prediction refers to the one-step ahead prediction whereas long-term prediction refers to the prediction of the final destination point of the obstacle's movement. Both kinds of prediction are integrated into the reward function of the RN-HPOMDP and the speed decision is performed through a modified solution of the RN-HPOMDP. As a result, the RN-HPOMDP can decide the optimal obstacle avoidance behavior based on the current and the predicted state of the environment without the intervention of any other external module.

Experimental results have shown the applicability and effectiveness of the proposed framework for the navigation task. The robustness and the probabilistic nature of the RN-HPOMDP as well as the future motion prediction are required to be able to perform efficiently and effectively in dynamic real-world environments that are highly populated.

Decentralized partially observable Markov decision processes (Dec-POMDPs) are general models for decentralized decision making under uncertainty. However, they typically model a problem at a low level of granularity, where each agent's actions are primitive operations lasting exactly one time step.  We address the case where each agent has macro-actions:  temporally extended actions which may require different amounts of time to execute. We model macro-actions as \emph{options} in a factored Dec-POMDP model, focusing on options which depend only on information available to an individual agent while executing. This enables us to model systems where coordination decisions only occur at the level of deciding which macro-actions to execute, and the macro-actions themselves can then be executed to completion. The core technical difficulty when using options in a Dec-POMDP is that the options chosen by the agents no longer terminate at the same time. We present extensions of two leading Dec-POMDP algorithms for generating a policy with options and discuss the resulting form of optimality. Our results show that these algorithms retain agent coordination while allowing near-optimal solutions to be generated for significantly longer horizons and larger state-spaces than previous Dec-POMDP methods.

This paper introduces timeline trees, which are partial models of partially observable environments. Timeline trees are given some specific predictions to make and learn a decision tree over history. The main idea of timeline trees is to use temporally abstract features to identify and split on features of key events, spread arbitrarily far apart in the past (whereas previous decision-tree-based methods have been limited to a finite suffix of history). Experiments demonstrate that timeline trees can learn to make high quality predictions in complex, partially observable environments with high-dimensional observations (e.g. an arcade game).

Automatic Target Recognition (ATR) algorithm performance is highly dependent on the sensing conditions under which the input data is collected. Open-loop fly-bys often produce poor results due to less than ideal measurement conditions. In addition, ATR algorithms must be extremely complicated to handle the diverse range of inputs with a resulting reduction in overall performance and increase in complexity. Our approach, closed-loop ATR (CL-ATR), focuses on improving the quality of information input to the ATR algorithms by optimizing motion, sensor settings and team (vehicle-vehicle-human) collaboration to dramatically improve classification accuracy. By managing the data collection guided by predicted ATR performance gain, we increase the information content of the data and thus dramatically improve ATR performance with existing ATR algorithms. CL-ATR has two major functions; first, an ATR utility function, which represents the performance sensitivity of ATR-produced classification labels as a function of parameters that correlate to vehicle/sensor states. This utility function is developed off-line and is often available from the original ATR study as a confusion matrix, or it can be derived through simulation without direct access to the inner working of the ATR algorithm. The utility function is inserted into our CL-ATR framework to autonomously control the vehicle/sensor. Second, an on-board planner maps the utility function into vehicle position and sensor collection plans. Because we only require the utility function on-board, we can activate any ATR algorithm onto a unmanned aerial vehicle (UAV) platform no matter how complex. This pairing of ATR performance profiles with vehicle/sensor controls creates a unique and powerful active perception behavior.

Simulation is becoming more and more important for robotics research, especially for multi-robot system research. Simatch, originating from the Robot World Cup (RoboCup) Middle Size League (MSL) match, is proposed to simulate the highly dynamic confrontation between multi-robot systems and validate the adversarial multi-robot strategies. It consists of three independent subsystems: the simulation subsystem setting up the simulated environment, the strategy subsystem realizing the multi-robot strategies and the scene subsystem describing the specific scene. Due to the independence of these subsystems, it is easy to employ Simatch for various applications. Simatch is validated by a series of tests, including a test of the omnidirectional locomotion, a simulated MSL match among ten robots and a simulated encirclement between sixteen robots. Since 2016, Simatch has been the official software for the MSL simulation match in the Chinese Robotics Competition. Besides, an MSL simulation project based on Simatch has been proposed to promote the development of MSL in the 2017 MSL international workshop. The proposed simulation system facilitates the development of multi-robot cooperation algorithms and related research.

A long-standing goal of AI is to enable robots to plan in the face of uncertain and incomplete information, and to handle task failure intelligently. This paper shows how to achieve this. There are two central ideas. The first idea is to organize the robot's knowledge into three layers: instance knowledge at the bottom, commonsense knowledge above that, and diagnostic knowledge on top. Knowledge in a layer above can be used to modify knowledge in the layer(s) below. The second idea is that the robot should represent not just how its actions change the world, but also what it knows or believes. There are two types of knowledge effects the robot's actions can have: epistemic effects (I believe X because I saw it) and assumptions (I'll assume X to be true). By combining the knowledge layers with the models of knowledge effects, we can simultaneously solve several problems in robotics: (i) task planning and execution under uncertainty; (ii) task planning and execution in open worlds; (iii) explaining task failure; (iv) verifying those explanations. The paper describes how the ideas are implemented in a three-layer architecture on a mobile robot platform. The robot implementation was evaluated in five different experiments on object search, mapping, and room categorization.

The default mode network (DMN) is believed to subserve the baseline mental activity in humans. Its higher energy consumption compared to other brain networks and its intimate coupling with conscious awareness are both pointing to an unknown over-arching function. Many research streams speak in favor of an evolutionarily adaptive role in envisioning experience to anticipate the future. In the present work, we propose a process model that tries to explain how the DMN may implement continuous evaluation and prediction of the environment to guide behavior. The main purpose of DMN activity, we argue, may be described by Markov decision processes that optimize action policies via value estimates through vicarious trial and error. Our formal perspective on DMN function naturally accommodates as special cases previous interpretations based on (a) predictive coding, (b) semantic associations, and (c) a sen-tinel role. Moreover, this process model for the neural optimization of complex behavior in the DMN offers parsimonious explanations for recent experimental findings in animals and humans. K E Y W O R D S artificial intelligence, human intelligence, systems neuroscience

The Multi-Agent paradigm is becoming increasingly pop- ular as a way of capturing complex control processes with stochastic properties. Many existing modelling tools are not flexible enough for these purposes, possibly because many of the modelling frame- works available inherit their structure from single agent frameworks. This paper proposes a new family of modelling frameworks called FASP, which is based

Aberrant decision-making characterizes various pediatric psychopathologies; however, deliberative choice strategies have not been investigated. A transdiagnostic sample of 95 youths completed a child-friendly sequential sampling paradigm. Participants searched for the best offer by sampling a finite list of offers. Participants’ willingness to explore was measured as the number of offers sampled, and ideal task performance was modeled using a Markov decision-process model. As in previous findings in adults, youths explored more offers when lists were long compared with short, yet participants generally sampled fewer offers relative to model-estimated ideal performance. Searching deeper into the list was associated with choosing better price options. Analyses examining the main and interactive effects of transdiagnostic anxiety and irritability symptoms indicated a negative correlation between anxiety and task performance (p = .01, ηp2 = .08). Findings suggest the need for more research on exploratory decision impairments in youths with anxiety symptoms.

Peshkin et al.'s example is illustrated in Fig. 3. There are two team-mates, V1 and V2, and an opponent O, each agent has partial observ-ability and can only see if the 4 horizontally and vertically adjacent squares are occupied, or not. Also, players V1 and V2 have an extra ...

We describe a simple method for exact online coinfernece and decision making for partially observable and large Markov decision processes. This is based on a closed form Bayesian update procedure for certain classes of models exhibiting a special conditional independence structure, which can be used for prediction, and consequently for planning.

Partially-observable Markov decision processes provide a very general model for decision-theoretic planning problems, allowing the trade-offs between various courses of actions to be determined under conditions of uncertainty, and incorporating partial observations made by an agent. Dynamic programming algorithms based on the information or belief state of an agent can be used to construct optimal policies without explicit consideration of past history, but at high computational cost. In this paper, we discuss how structured representations of the system dynamics can be incorporated in classic POMDP solution algorithms. We use Bayesian networks with structured conditional probability matrices to represent POMDPs, and use this representation to structure the belief space for POMDP algorithms. This allows irrelevant distinctions to be ignored. Apart from speeding up optimal policy construction, we suggest that such representations can be exploited to great extent in the development of useful approximation methods.