Learning from Demonstration

The purpose of this tutorial is to help individuals use the \underline{FireCommander} game environment for research applications. The FireCommander is an interactive, probabilistic joint perception-action reconnaissance environment in which a composite team of agents (e.g., robots) cooperate to fight dynamic, propagating firespots (e.g., targets). In FireCommander game, a team of agents must be tasked to optimally deal with a wildfire situation in an environment with propagating fire areas and some facilities such as houses, hospitals, power stations, etc. The team of agents can accomplish their mission by first sensing (e.g., estimating fire states), communicating the sensed fire-information among each other and then taking action to put the firespots out based on the sensed information (e.g., dropping water on estimated fire locations). The FireCommander environment can be useful for research topics spanning a wide range of applications from Reinforcement Learning (RL) and Learning from Demonstration (LfD), to Coordination, Psychology, Human-Robot Interaction (HRI) and Teaming. There are four important facets of the FireCommander environment that overall, create a non-trivial game: (1) Complex Objectives: Multi-objective Stochastic Environment, (2)Probabilistic Environment: Agents' actions result in probabilistic performance, (3) Hidden Targets: Partially Observable Environment and, (4) Uni-task Robots: Perception-only and Action-only agents. The FireCommander environment is first-of-its-kind in terms of including Perception-only and Action-only agents for coordination. It is a general multi-purpose game that can be useful in a variety of combinatorial optimization problems and stochastic games, such as applications of Reinforcement Learning (RL), Learning from Demonstration (LfD) and Inverse RL (iRL).

In this work is developed an architecture which consist of one perception system, one motion planner, two control systems and one localization system that permit the autonomous generation of operations in backhoe machines. This set of systems are presented in two parts that are related with hierarchy levels, i.e. control and localization, and perception and planning. In this sense, in the first part is elaborated the architectures of control and localization, in which it exposes an analysis of stability of PID controller that permits the the tuning of controller, and a study and implementation of an Extended Kalman Filter  (EKF), in order to estimate the position of the control and localization system, respectively. Thereafter, in the second part, a set of techniques for collision mapping of the scene is implemented. Moreover, the development of motion planning system based on Learning from Demonstration using Dynamic Movement Primitives as control policy is proposed, which allow backhoe machines to perform operations in autonomous way. A statement of reasons is presented, wherein it is justified the implementation of such motion system versus planners like A*, Probabilistic RoadMap (PRM), Rapidly-exploring Random Tree (RRT) and Potential Fields. In addition, the results, contributions and conclusions are presented in each chapter. Finally, the tests were performed in a simulation environment called Gazebo over Robot Operating System (ROS).

The purpose of this tutorial is to help individuals use the \underline{FireCommander} game environment for research applications. The FireCommander is an interactive, probabilistic joint perception-action reconnaissance environment in which a composite team of agents (e.g., robots) cooperate to fight dynamic, propagating firespots (e.g., targets). In FireCommander game, a team of agents must be tasked to optimally deal with a wildfire situation in an environment with propagating fire areas and some facilities such as houses, hospitals, power stations, etc. The team of agents can accomplish their mission by first sensing (e.g., estimating fire states), communicating the sensed fire-information among each other and then taking action to put the firespots out based on the sensed information (e.g., dropping water on estimated fire locations). The FireCommander environment can be useful for research topics spanning a wide range of applications from Reinforcement Learning (RL) and Learning from Demonstration (LfD), to Coordination, Psychology, Human-Robot Interaction (HRI) and Teaming. There are four important facets of the FireCommander environment that overall, create a non-trivial game: (1) Complex Objectives: Multi-objective Stochastic Environment, (2)Probabilistic Environment: Agents' actions result in probabilistic performance, (3) Hidden Targets: Partially Observable Environment and, (4) Uni-task Robots: Perception-only and Action-only agents. The FireCommander environment is first-of-its-kind in terms of including Perception-only and Action-only agents for coordination. It is a general multi-purpose game that can be useful in a variety of combinatorial optimization problems and stochastic games, such as applications of Reinforcement Learning (RL), Learning from Demonstration (LfD) and Inverse RL (iRL).

There is neuroscientific evidence to suggest that imitation between humans is goal-directed. Therefore, when performing
multiple tasks, we internally define an unknown optimal policy to satisfy multiple goals. This work presents a method to
transfer a complex behavior composed by a sequence of multiple tasks from a human demonstrator to a humanoid robot.
We defined a multi-objective reward function as a measurement of the goal optimality for both human and robot, which
is defined in each subtask of the global behavior. We optimize a sequential policy to generate whole-body movements
for the robot that produces a reward profile which is compared and matched with the human reward profile, producing an
imitative behavior. Furthermore, we can search in the proximity of the solution space to improve the reward profile and
innovate a new solution, which is more beneficial for the humanoid. Experiments were carried out in a real humanoid robot.

The control described here is derived from motion and force-plate data captured at a gait laboratory. We have developed a 3-D, 12-degree-of-freedom model representing the legs and lower torso of a human. The joints and links of this model closely match those of the captured motion. Thus, the motion data can be applied to the model to obtain forces via

The current work presents a comprehensive methodological framework that facilitates robots to acquire human-like behavioral acts by observing human demonstrators. Accordingly, the introduced framework is established as a Learning from Demonstration (LfD) process that enables the reproduction of either learned or novel actions. Mapping of human actions to the respective robotic ones is achieved via an indeterminate depiction, termed latent space representation. The latter accomplishes a compact, yet precise abstraction of action trajectories, effectively representing high dimensional raw actions in a low dimensional space. Extensive experimentation with a real robotic arm demonstrates the robustness and applicability of the introduced framework.

Abstract—Learning by demonstration and learning by instruction offers a potentially more powerful paradigm than programming robots directly for specific tasks. Learning in humans or primates substantially benefits from demonstra-tion of actions or instruction by language in the ...

In this paper, we present Combined Learning from demonstration And Motion Planning (CLAMP) as an efficient approach to skill learning and gener-alizable skill reproduction. CLAMP combines the strengths of Learning from Demonstration (LfD) and motion planning into a unifying framework. We carry out probabilistic inference to find trajectories which are optimal with respect to a given skill and also feasible in different scenarios. We use factor graph optimization to speed up inference. To encode optimality, we provide a new probabilistic skill model based on a stochastic dynamical system. This skill model requires minimal parameter tuning to learn, is suitable to encode skill constraints, and allows efficient inference. Preliminary experimental results showing skill generalization over initial robot state and unforeseen obstacles are presented.

Execution of a manipulation after learning from demonstration many times requires intricate planning and control systems or some form of manual guidance for a robot. Here we present a framework for manipulation execution based on the so called “Semantic Event Chain” which is an abstract description of relations between the objects in the scene. It captures the change of those relations during a manipulation and thereby provides the decisive temporal anchor points by which a manipulation is critically defined. Using semantic event chains a model of a manipulation can be learned. We will show that it is possible to add the required control parameters (the spatial anchor points) to this model, which can then be executed by a robot in a fully autonomous way. The process of learning and execution of semantic event chains is explained using a box pushing example.