Purpose – The purpose of this paper is to describe the specification language TML for adaptive
mi... more Purpose – The purpose of this paper is to describe the specification language TML for adaptive mission plans that the authors designed and implemented for the open-source framework Aerostack for aerial robotics. Design/methodology/approach – The TML language combines a task-based hierarchical approach together with a more flexible representation, rule-based reactive planning, to facilitate adaptability. This approach includes additional notions that abstract programming details. The authors built an interpreter integrated in the software framework Aerostack. The interpreter was validated with flight experiments for multi-robot missions in dynamic environments. Findings – The experiments proved that the TML language is easy to use and expressive enough to formulate adaptive missions in dynamic environments. The experiments also showed that the TML interpreter is efficient to execute multi-robot aerial missions and reusable for different platforms. The TML interpreter is able to verify the mission plan before its execution, which increases robustness and safety, avoiding the execution of certain plans that are not feasible. Originality/value – One of the main contributions of this work is the availability of a reliable solution to specify aerial mission plans, integrated in an active open-source project with periodic releases. To the best knowledge of the authors, there are not solutions similar to this in other active open-source projects. As additional contributions, TML uses an original combination of representations for adaptive mission plans (i.e. task trees with original abstract notions and rule-based reactive planning) together with the demonstration of its adequacy for aerial robotics.
Revista Iberoamericana de Automática e Informática Industrial RIAI, 2013
ABSTRACT This paper describes and analyzes the processes carried out to obtain the model identifi... more ABSTRACT This paper describes and analyzes the processes carried out to obtain the model identification and the controller design, as well as its posterior implementation, for the quadrotor Parrot AR- Drone. The results are intended to control the autonomous tracking of that quadrotor on a path in the XY plane, flexible defined by the user. The identification phase is characterized by the division of the model into a linear and a non-linear part, resulting in a Hammerstein's model. The control is characterized by the use of a cascade control with an inner speed control loop and an outer position control loop. As characteristic features, it is highlighted that: the speed controller includes a static linearization of the model, and the position loop includes a nonlinear block to reduce the tracking error of the trajectory but using a simple position controller logic. The context of this study is the Control Engineering Competition 2012, “Autonomous trajectory tracking control of a quadrotor vehicle” organized by the Spanish Committee for Automation, which took place at the Automation Symposium in Vigo, last September 2012. The work presented was awarded in the first and second phase of the competition with the first and third prize, respectively.
Poaching is an illegal activity that remains out of control in many countries. Based on the 2014 ... more Poaching is an illegal activity that remains out of control in many countries. Based on the 2014 report of the United Nations and Interpol, the illegal trade of global wildlife and natural resources amounts to nearly $ 213 billion every year, which is even helping to fund armed conflicts. Poaching activities around the world are further pushing many animal species on the brink of extinction. Unfortunately, the traditional methods to fight against poachers are not enough, hence the new demands for more efficient approaches. In this context, the use of new technologies on sensors and algorithms, as well as aerial platforms is crucial to face the high increase of poaching activities in the last few years. Our work is focused on the use of vision sensors on UAVs for the detection and tracking of animals and poachers, as well as the use of such sensors to control quadrotors during autonomous vehicle following and autonomous landing.
In this paper an automated visual inspection system is described within a general architecture fo... more In this paper an automated visual inspection system is described within a general architecture for the online detection and analysis of surface defects in the production of continuous flat metallic products. Real-time performance requirements result in the development of a high-parallel architecture for high-speed image processing. Matrix cameras are used for image acquisition instead of linear ones because of the surface appearance. Inspection is achieved with 1 mm2 resolution. Similarity-based algorithms as well as texture algorithms have been developed and hardware-implemented for defect detection in a high-textured surface where, for the most of defects, segmentation can not be achieved by only means of threshold techniques. The system has been applied to continuous cast aluminum inspection, where up to fifteen different kinds of defects must be detected and classified. Online defect classification is attempted by means of the formal language theory
The present paper describes the objectives, structure, present stage, results and future mileston... more The present paper describes the objectives, structure, present stage, results and future milestones of the project ELEVA. This project is aimed to control an autonomous helicopter in order to follow an overhead power cable by means of a stereo computer vision system. The helicopter is aimed to have always in sight the overhead power cable, to follow it by using it as an external visual reference guide and to record it for its ulterior visual inspection. These objectives are achieved by using a 3D computer vision system to generate the reference trajectory to be followed and by using internal sensors to control its stability and its trajectory. The paper presents the results obtained so far: visual detection and tracking of the power cable, robust under changing environments, and robust stationary control of the helicopter, now linked to a safety mechanical platform. Finally this paper describes the future challenges of the project and its temporal milestones.
Computer vision is much more than a technique to sense and recover environmental information from... more Computer vision is much more than a technique to sense and recover environmental information from an UAV. It should play a main role regarding UAVs’ functionality because of the big amount of information that can be extracted, its possible uses and applications, and its natural connection to human driven tasks, taking into account that vision is our main interface to world understanding. Our current research’s focus lays on the development of techniques that allow UAVs to maneuver in spaces using visual information as their main input source. This task involves the creation of techniques that allow an UAV to maneuver towards features of interest whenever a GPS signal is not reliable or sufficient, e.g. when signal dropouts occur (which usually happens in urban areas, when flying through terrestrial urban canyons or when operating on remote planetary bodies), or when tracking or inspecting visual targets—including moving ones—without knowing their exact UMT coordinates. This paper also investigates visual servoing control techniques that use velocity and position of suitable image features to compute the references for flight control. This paper aims to give a global view of the main aspects related to the research field of computer vision for UAVs, clustered in four main active research lines: visual servoing and control, stereo-based visual navigation, image processing algorithms for detection and tracking, and visual SLAM. Finally, the results of applying these techniques in several applications are presented and discussed: this study will encompass power line inspection, mobile target tracking, stereo distance estimation, mapping and positioning.
In this paper we introduce a real-time trinocular system to control rotary wing Unmanned Aerial V... more In this paper we introduce a real-time trinocular system to control rotary wing Unmanned Aerial Vehicles based on the 3D information extracted by cameras located on the ground. The algorithm is based on key features onboard the UAV to estimate the vehicle's position and orientation. The algorithm is validated against onboard sensors and known 3D positions, showing that the proposed camera configuration robustly estimates the helicopter's position with an adequate resolution, improving the position estimation, especially the height estimation. The obtained results show that the proposed algorithm is suitable to complement or replace the GPS-based position estimation in situations where GPS information is unavailable or where its information is inaccurate, allowing the vehicle to develop tasks at low heights, such as autonomous landing, take-off, and positioning, using the extracted 3D information as a visual feedback to the flight controller.
In this paper, we seek to expand the use of direct methods in real-time applications by proposing... more In this paper, we seek to expand the use of direct methods in real-time applications by proposing a vision-based strategy for pose estimation of aerial vehicles. The vast majority of approaches make use of features to estimate motion. Conversely, the strategy we propose is based on a MR (Multi-Resolution) implementation of an image registration technique (Inverse Compositional Image Alignment ICIA) using direct methods. An on-board camera in a downwards-looking configuration, and the assumption of planar scenes, are the bases of the algorithm. The motion between frames (rotation and translation) is recovered by decomposing the frame-to-frame homography obtained by the ICIA algorithm applied to a patch that covers around the 80% of the image. When the visual estimation is required (e.g. GPS drop-out), this motion is integrated with the previous known estimation of the vehicles' state, obtained from the on-board sensors (GPS/IMU), and the subsequent estimations are based only on the vision-based motion estimations. The proposed strategy is tested with real flight data in representative stages of a flight: cruise, landing, and take-off, being two of those stages considered critical: take-off and landing. The performance of the pose estimation strategy is analyzed by comparing it with the GPS/IMU estimations. Results show correlation between the visual estimation obtained with the MR-ICIA and the GPS/IMU data, that demonstrate that the visual estimation can be used to provide a good approximation of the vehicle's state when it is required (e.g. GPS drop-outs). In terms of performance, the proposed strategy is able to maintain an estimation of the vehicle's state for more than one minute, at real-time frame rates based, only on visual information.
This article presents a real time Unmanned Aerial Vehicles UAVs 3D pose estimation method using p... more This article presents a real time Unmanned Aerial Vehicles UAVs 3D pose estimation method using planar object tracking, in order to be used on the control system of a UAV. The method explodes the rich information obtained by a projective transformation of planar objects on a calibrated camera. The algorithm obtains the metric and projective components of a reference object (landmark or helipad) with respect to the UAV camera coordinate system, using a robust real time object tracking based on homographies. The algorithm is validated on real flights that compare the estimated data against that obtained by the inertial measurement unit IMU, showing that the proposed method robustly estimates the helicopter's 3D position with respect to a reference landmark, with a high quality on the position and orientation estimation when the aircraft is flying at low altitudes, a situation in which the GPS information is often inaccurate. The obtained results indicate that the proposed algorithm is suitable for complex control tasks, such as autonomous landing, accurate low altitude positioning and dropping of payloads.
The aim of the paper is to present, test and discuss the implementation of Visual SLAM techniques... more The aim of the paper is to present, test and discuss the implementation of Visual SLAM techniques to images taken from Unmanned Aerial Vehicles (UAVs) outdoors, in partially structured environments. Every issue of the whole process is discussed in order to obtain more accurate localization and mapping from UAVs flights. Firstly, the issues related to the visual features of objects in the scene, their distance to the UAV, and the related image acquisition system and their calibration are evaluated for improving the whole process. Other important, considered issues are related to the image processing techniques, such as interest point detection, the matching procedure and the scaling factor. The whole system has been tested using the COLIBRI mini UAV in partially structured environments. The results that have been obtained for localization, tested against the GPS information of the flights, show that Visual SLAM delivers reliable localization and mapping that makes it suitable for some outdoors applications when flying UAVs.
Computer vision is much more than a technique to sense and recover environmental information from... more Computer vision is much more than a technique to sense and recover environmental information from an UAV. It should play a main role regarding UAVs’ functionality because of the big amount of information that can be extracted, its possible uses and applications, and its natural connection to human driven tasks, taking into account that vision is our main interface to world understanding. Our current research’s focus lays on the development of techniques that allow UAVs to maneuver in spaces using visual information as their main input source. This task involves the creation of techniques that allow an UAV to maneuver towards features of interest whenever a GPS signal is not reliable or sufficient, e.g. when signal dropouts occur (which usually happens in urban areas, when flying through terrestrial urban canyons or when operating on remote planetary bodies), or when tracking or inspecting visual targets—including moving ones—without knowing their exact UMT coordinates. This paper also investigates visual servoing control techniques that use velocity and position of suitable image features to compute the references for flight control. This paper aims to give a global view of the main aspects related to the research field of computer vision for UAVs, clustered in four main active research lines: visual servoing and control, stereo-based visual navigation, image processing algorithms for detection and tracking, and visual SLAM. Finally, the results of applying these techniques in several applications are presented and discussed: this study will encompass power line inspection, mobile target tracking, stereo distance estimation, mapping and positioning.
UAVs (unmanned aerial vehicles) are becoming more popular for civil task due to their unique flig... more UAVs (unmanned aerial vehicles) are becoming more popular for civil task due to their unique flight capabilities. The video shows several capabilities of a platform used as main research testbed of computer vision for unmanned aerial vehicles. Four major research areas are explored in the present work as visual servoing, trajectory planning, power line inspection and stereo-based visual navigation.
Purpose – The purpose of this paper is to describe the specification language TML for adaptive
mi... more Purpose – The purpose of this paper is to describe the specification language TML for adaptive mission plans that the authors designed and implemented for the open-source framework Aerostack for aerial robotics. Design/methodology/approach – The TML language combines a task-based hierarchical approach together with a more flexible representation, rule-based reactive planning, to facilitate adaptability. This approach includes additional notions that abstract programming details. The authors built an interpreter integrated in the software framework Aerostack. The interpreter was validated with flight experiments for multi-robot missions in dynamic environments. Findings – The experiments proved that the TML language is easy to use and expressive enough to formulate adaptive missions in dynamic environments. The experiments also showed that the TML interpreter is efficient to execute multi-robot aerial missions and reusable for different platforms. The TML interpreter is able to verify the mission plan before its execution, which increases robustness and safety, avoiding the execution of certain plans that are not feasible. Originality/value – One of the main contributions of this work is the availability of a reliable solution to specify aerial mission plans, integrated in an active open-source project with periodic releases. To the best knowledge of the authors, there are not solutions similar to this in other active open-source projects. As additional contributions, TML uses an original combination of representations for adaptive mission plans (i.e. task trees with original abstract notions and rule-based reactive planning) together with the demonstration of its adequacy for aerial robotics.
Revista Iberoamericana de Automática e Informática Industrial RIAI, 2013
ABSTRACT This paper describes and analyzes the processes carried out to obtain the model identifi... more ABSTRACT This paper describes and analyzes the processes carried out to obtain the model identification and the controller design, as well as its posterior implementation, for the quadrotor Parrot AR- Drone. The results are intended to control the autonomous tracking of that quadrotor on a path in the XY plane, flexible defined by the user. The identification phase is characterized by the division of the model into a linear and a non-linear part, resulting in a Hammerstein's model. The control is characterized by the use of a cascade control with an inner speed control loop and an outer position control loop. As characteristic features, it is highlighted that: the speed controller includes a static linearization of the model, and the position loop includes a nonlinear block to reduce the tracking error of the trajectory but using a simple position controller logic. The context of this study is the Control Engineering Competition 2012, “Autonomous trajectory tracking control of a quadrotor vehicle” organized by the Spanish Committee for Automation, which took place at the Automation Symposium in Vigo, last September 2012. The work presented was awarded in the first and second phase of the competition with the first and third prize, respectively.
Poaching is an illegal activity that remains out of control in many countries. Based on the 2014 ... more Poaching is an illegal activity that remains out of control in many countries. Based on the 2014 report of the United Nations and Interpol, the illegal trade of global wildlife and natural resources amounts to nearly $ 213 billion every year, which is even helping to fund armed conflicts. Poaching activities around the world are further pushing many animal species on the brink of extinction. Unfortunately, the traditional methods to fight against poachers are not enough, hence the new demands for more efficient approaches. In this context, the use of new technologies on sensors and algorithms, as well as aerial platforms is crucial to face the high increase of poaching activities in the last few years. Our work is focused on the use of vision sensors on UAVs for the detection and tracking of animals and poachers, as well as the use of such sensors to control quadrotors during autonomous vehicle following and autonomous landing.
In this paper an automated visual inspection system is described within a general architecture fo... more In this paper an automated visual inspection system is described within a general architecture for the online detection and analysis of surface defects in the production of continuous flat metallic products. Real-time performance requirements result in the development of a high-parallel architecture for high-speed image processing. Matrix cameras are used for image acquisition instead of linear ones because of the surface appearance. Inspection is achieved with 1 mm2 resolution. Similarity-based algorithms as well as texture algorithms have been developed and hardware-implemented for defect detection in a high-textured surface where, for the most of defects, segmentation can not be achieved by only means of threshold techniques. The system has been applied to continuous cast aluminum inspection, where up to fifteen different kinds of defects must be detected and classified. Online defect classification is attempted by means of the formal language theory
The present paper describes the objectives, structure, present stage, results and future mileston... more The present paper describes the objectives, structure, present stage, results and future milestones of the project ELEVA. This project is aimed to control an autonomous helicopter in order to follow an overhead power cable by means of a stereo computer vision system. The helicopter is aimed to have always in sight the overhead power cable, to follow it by using it as an external visual reference guide and to record it for its ulterior visual inspection. These objectives are achieved by using a 3D computer vision system to generate the reference trajectory to be followed and by using internal sensors to control its stability and its trajectory. The paper presents the results obtained so far: visual detection and tracking of the power cable, robust under changing environments, and robust stationary control of the helicopter, now linked to a safety mechanical platform. Finally this paper describes the future challenges of the project and its temporal milestones.
Computer vision is much more than a technique to sense and recover environmental information from... more Computer vision is much more than a technique to sense and recover environmental information from an UAV. It should play a main role regarding UAVs’ functionality because of the big amount of information that can be extracted, its possible uses and applications, and its natural connection to human driven tasks, taking into account that vision is our main interface to world understanding. Our current research’s focus lays on the development of techniques that allow UAVs to maneuver in spaces using visual information as their main input source. This task involves the creation of techniques that allow an UAV to maneuver towards features of interest whenever a GPS signal is not reliable or sufficient, e.g. when signal dropouts occur (which usually happens in urban areas, when flying through terrestrial urban canyons or when operating on remote planetary bodies), or when tracking or inspecting visual targets—including moving ones—without knowing their exact UMT coordinates. This paper also investigates visual servoing control techniques that use velocity and position of suitable image features to compute the references for flight control. This paper aims to give a global view of the main aspects related to the research field of computer vision for UAVs, clustered in four main active research lines: visual servoing and control, stereo-based visual navigation, image processing algorithms for detection and tracking, and visual SLAM. Finally, the results of applying these techniques in several applications are presented and discussed: this study will encompass power line inspection, mobile target tracking, stereo distance estimation, mapping and positioning.
In this paper we introduce a real-time trinocular system to control rotary wing Unmanned Aerial V... more In this paper we introduce a real-time trinocular system to control rotary wing Unmanned Aerial Vehicles based on the 3D information extracted by cameras located on the ground. The algorithm is based on key features onboard the UAV to estimate the vehicle's position and orientation. The algorithm is validated against onboard sensors and known 3D positions, showing that the proposed camera configuration robustly estimates the helicopter's position with an adequate resolution, improving the position estimation, especially the height estimation. The obtained results show that the proposed algorithm is suitable to complement or replace the GPS-based position estimation in situations where GPS information is unavailable or where its information is inaccurate, allowing the vehicle to develop tasks at low heights, such as autonomous landing, take-off, and positioning, using the extracted 3D information as a visual feedback to the flight controller.
In this paper, we seek to expand the use of direct methods in real-time applications by proposing... more In this paper, we seek to expand the use of direct methods in real-time applications by proposing a vision-based strategy for pose estimation of aerial vehicles. The vast majority of approaches make use of features to estimate motion. Conversely, the strategy we propose is based on a MR (Multi-Resolution) implementation of an image registration technique (Inverse Compositional Image Alignment ICIA) using direct methods. An on-board camera in a downwards-looking configuration, and the assumption of planar scenes, are the bases of the algorithm. The motion between frames (rotation and translation) is recovered by decomposing the frame-to-frame homography obtained by the ICIA algorithm applied to a patch that covers around the 80% of the image. When the visual estimation is required (e.g. GPS drop-out), this motion is integrated with the previous known estimation of the vehicles' state, obtained from the on-board sensors (GPS/IMU), and the subsequent estimations are based only on the vision-based motion estimations. The proposed strategy is tested with real flight data in representative stages of a flight: cruise, landing, and take-off, being two of those stages considered critical: take-off and landing. The performance of the pose estimation strategy is analyzed by comparing it with the GPS/IMU estimations. Results show correlation between the visual estimation obtained with the MR-ICIA and the GPS/IMU data, that demonstrate that the visual estimation can be used to provide a good approximation of the vehicle's state when it is required (e.g. GPS drop-outs). In terms of performance, the proposed strategy is able to maintain an estimation of the vehicle's state for more than one minute, at real-time frame rates based, only on visual information.
This article presents a real time Unmanned Aerial Vehicles UAVs 3D pose estimation method using p... more This article presents a real time Unmanned Aerial Vehicles UAVs 3D pose estimation method using planar object tracking, in order to be used on the control system of a UAV. The method explodes the rich information obtained by a projective transformation of planar objects on a calibrated camera. The algorithm obtains the metric and projective components of a reference object (landmark or helipad) with respect to the UAV camera coordinate system, using a robust real time object tracking based on homographies. The algorithm is validated on real flights that compare the estimated data against that obtained by the inertial measurement unit IMU, showing that the proposed method robustly estimates the helicopter's 3D position with respect to a reference landmark, with a high quality on the position and orientation estimation when the aircraft is flying at low altitudes, a situation in which the GPS information is often inaccurate. The obtained results indicate that the proposed algorithm is suitable for complex control tasks, such as autonomous landing, accurate low altitude positioning and dropping of payloads.
The aim of the paper is to present, test and discuss the implementation of Visual SLAM techniques... more The aim of the paper is to present, test and discuss the implementation of Visual SLAM techniques to images taken from Unmanned Aerial Vehicles (UAVs) outdoors, in partially structured environments. Every issue of the whole process is discussed in order to obtain more accurate localization and mapping from UAVs flights. Firstly, the issues related to the visual features of objects in the scene, their distance to the UAV, and the related image acquisition system and their calibration are evaluated for improving the whole process. Other important, considered issues are related to the image processing techniques, such as interest point detection, the matching procedure and the scaling factor. The whole system has been tested using the COLIBRI mini UAV in partially structured environments. The results that have been obtained for localization, tested against the GPS information of the flights, show that Visual SLAM delivers reliable localization and mapping that makes it suitable for some outdoors applications when flying UAVs.
Computer vision is much more than a technique to sense and recover environmental information from... more Computer vision is much more than a technique to sense and recover environmental information from an UAV. It should play a main role regarding UAVs’ functionality because of the big amount of information that can be extracted, its possible uses and applications, and its natural connection to human driven tasks, taking into account that vision is our main interface to world understanding. Our current research’s focus lays on the development of techniques that allow UAVs to maneuver in spaces using visual information as their main input source. This task involves the creation of techniques that allow an UAV to maneuver towards features of interest whenever a GPS signal is not reliable or sufficient, e.g. when signal dropouts occur (which usually happens in urban areas, when flying through terrestrial urban canyons or when operating on remote planetary bodies), or when tracking or inspecting visual targets—including moving ones—without knowing their exact UMT coordinates. This paper also investigates visual servoing control techniques that use velocity and position of suitable image features to compute the references for flight control. This paper aims to give a global view of the main aspects related to the research field of computer vision for UAVs, clustered in four main active research lines: visual servoing and control, stereo-based visual navigation, image processing algorithms for detection and tracking, and visual SLAM. Finally, the results of applying these techniques in several applications are presented and discussed: this study will encompass power line inspection, mobile target tracking, stereo distance estimation, mapping and positioning.
UAVs (unmanned aerial vehicles) are becoming more popular for civil task due to their unique flig... more UAVs (unmanned aerial vehicles) are becoming more popular for civil task due to their unique flight capabilities. The video shows several capabilities of a platform used as main research testbed of computer vision for unmanned aerial vehicles. Four major research areas are explored in the present work as visual servoing, trajectory planning, power line inspection and stereo-based visual navigation.
Uploads
Papers by Pascual Campoy
mission plans that the authors designed and implemented for the open-source framework Aerostack for
aerial robotics.
Design/methodology/approach – The TML language combines a task-based hierarchical approach
together with a more flexible representation, rule-based reactive planning, to facilitate adaptability. This
approach includes additional notions that abstract programming details. The authors built an interpreter
integrated in the software framework Aerostack. The interpreter was validated with flight experiments for
multi-robot missions in dynamic environments.
Findings – The experiments proved that the TML language is easy to use and expressive enough to
formulate adaptive missions in dynamic environments. The experiments also showed that the TML
interpreter is efficient to execute multi-robot aerial missions and reusable for different platforms. The TML
interpreter is able to verify the mission plan before its execution, which increases robustness and safety,
avoiding the execution of certain plans that are not feasible.
Originality/value – One of the main contributions of this work is the availability of a reliable solution to
specify aerial mission plans, integrated in an active open-source project with periodic releases. To the best
knowledge of the authors, there are not solutions similar to this in other active open-source projects.
As additional contributions, TML uses an original combination of representations for adaptive mission
plans (i.e. task trees with original abstract notions and rule-based reactive planning) together with the
demonstration of its adequacy for aerial robotics.
mission plans that the authors designed and implemented for the open-source framework Aerostack for
aerial robotics.
Design/methodology/approach – The TML language combines a task-based hierarchical approach
together with a more flexible representation, rule-based reactive planning, to facilitate adaptability. This
approach includes additional notions that abstract programming details. The authors built an interpreter
integrated in the software framework Aerostack. The interpreter was validated with flight experiments for
multi-robot missions in dynamic environments.
Findings – The experiments proved that the TML language is easy to use and expressive enough to
formulate adaptive missions in dynamic environments. The experiments also showed that the TML
interpreter is efficient to execute multi-robot aerial missions and reusable for different platforms. The TML
interpreter is able to verify the mission plan before its execution, which increases robustness and safety,
avoiding the execution of certain plans that are not feasible.
Originality/value – One of the main contributions of this work is the availability of a reliable solution to
specify aerial mission plans, integrated in an active open-source project with periodic releases. To the best
knowledge of the authors, there are not solutions similar to this in other active open-source projects.
As additional contributions, TML uses an original combination of representations for adaptive mission
plans (i.e. task trees with original abstract notions and rule-based reactive planning) together with the
demonstration of its adequacy for aerial robotics.