Automation of Taxiing: Jaroslav Burs Ik, Jakub Kraus, Marek Stumper
Automation of Taxiing: Jaroslav Burs Ik, Jakub Kraus, Marek Stumper
Automation of Taxiing: Jaroslav Burs Ik, Jakub Kraus, Marek Stumper
Automation of Taxiing
Jaroslav Bursı́k1, Jakub Kraus1*, Marek Štumper1
1 Department of Air Transport, Faculty of Transportation Sciences, Czech Technical University in Prague, Prague,
Czech Republic
*Corresponding author: Czech Technical University in Prague, Faculty of Transportation Sciences, Department of Air
Transport, Horská 3, 128 03 Prague, Czech Republic, Email: kraus@fd.cvut.cz
Abstract
The article focuses on the possibility of automation of taxiing, which is the part of a flight, which, under adverse
weather conditions, greatly reduces the operational usability of an airport, and is the only part of a flight that
has not been affected by automation, yet. Taxiing is currently handled manually by the pilot, who controls
the airplane based on information from visual perception. The article primarily deals with possible ways of
obtaining navigational information, and its automatic transfer to the controls. Analyzed wand assessed were
currently available technologies such as computer vision, Light Detection and Ranging and Global Navigation
Satellite System, which are useful for navigation and their general implementation into an airplane was designed.
Obstacles to the implementation were identified, too. The result is a proposed combination of systems along with
their installation into airplane’s systems so that it is possible to use the automated taxiing.
Keywords
Aircraft control — Automation — Safety — Taxiing — Taxiway
for sufficient amount of time [1, 2]. Following the taxiway or adjustment of the color spectrum. After the preprocessing
centerline is also very accurate, so the accuracy of the whole stage, the algorithms are applied and predefined objects, such
navigational system can be increased. Two systems were as straight lines, edges of objects or curves, are recognized.
chosen for this purpose: Computer vision and Lidar. Further algorithms are applied to these recognized elements
Both Computer vision and Lidar replace the pilots vision, of an image and the result is a description of the object of
allowing for very precise following of airport marking (taxi- our interest in the form of information about concrete points
way centerline) and also detection of obstacles in the field (location and brightness) of this object. With the knowledge
of vision. The Computer vision can also be used for detec- of physical location of the camera(s), it is possible to use this
tion and reading of airport signs and detection of active stop information for calculations of the actual values of distance,
bars. Combination of these systems with satellite navigation or to compare it with known objects (road signs, light color)
addresses the insufficient performance of satellite navigation and assign it a proper importance. [3]
and combines the advantages of accurate following of taxi- When considering the use of computer vision for taxiing,
way centerline with determination of position at the airport. it is appropriate to return to the analogy with a human vision.
The main disadvantage is the dependency on weather condi- The pilot determines the direction and speed of an airplane
tions (fog, rain, snow). The next paragraphs describe both based on what he/she sees outside. The basic monitored
Computer vision and Lidar. elements of surroundings are airport signs and markings, on
the basis of which the direction is controlled to ensure taxiing
2. Systems independent of GNSS according to a given clearance. Furthermore, obstacles in
the area are being observed, especially vehicles and other
2.1 Computer vision airplanes on the apron and taxiways, in order to provide for
Computer vision, in general, addresses digital image process- safe distance or to even prevent a collision. These activities
ing in order to obtain information that can be further processed. could be replaced by the computer vision and its principle is
It is quite common way of dealing with various issues in trans- shown in Fig. 1.
portation, such as recognizing of road signs, analysis of traffic
density or following of the lane. As a consequence of ever- Image from camera Computer vision Calculation
increasing computational power of today’s computers and of the required
improving of image quality of cameras, the range of appli- speed and direction
cations of the computer vision expands, and therefore it is Definition
desirable to consider the use of it in aviation. of
objects
The use of computer vision is essentially a direct substi-
tute of a human, who reacts to situations based on looking out Figure 1. Computer vision for taxiing.
of the cockpit. The function of eyes is replaced by a camera
mounted at a suitable location and the image evaluation is
Unlike the pilot’s eyes, the camera can be placed at a
performed digitally using a computer program, which uses
location, where it is least affected by the meteorological sit-
predefined algorithms. The analogy with human vision, how-
uation, or multiple cameras can be used to ensure sufficient
ever, introduces various setbacks to this solution (dependence
angle of view and redundancy. For ensuring movement along
on weather conditions, blind spots, etc.). The quality of evalu-
the taxiway centerline, it is necessary to identify the taxiway
ation of scanned situation depends on the quality of scanned
markings. The axis location on recorded image is clearly
image and on the sophistication of the algorithms.
defined by the computer vision and given the known loca-
Computer evaluation of an image is rather complex and
tion of the camera, i.e. known distance between the camera
demanding process, which can be divided into several steps:
and a taxiway, it is possible to, using the number of image
1. Recording of an image and its digitalization points, calculate distance difference between a defined middle
2. Preprocessing point and a taxiway centerline marking. This distance is con-
3. Detection of predefined elements sequently used to determine steering corrections in order to
4. Description of objects keep the nose wheel on the taxiway centerline. Intersections
can be detected and a required trajectory set by detecting in-
The image is first recorded by a camera (working in the ap- formation markings. The issue of speed of taxiing is usually
propriate region of the light spectrum) or a pair of cameras to dealt with based on the operational situation. The computer
ensure stereoscopic vision, which enables a relatively accurate vision enables detection of stop bars or runway holding points,
calculation of distance between objects in the captured image on the basis of which a command to stop the airplane can
and the camera. The image is further digitalized and as a re- be issued. When a pair of cameras is used for stereoscopic
sult, there is a matrix defining the light intensity of each of the vision, the distance between the airplane and various objects
recorded points. The preprocessing is used for adjusting the can be calculated, therefor an obstacle can be detected and a
whole image, right before the application of the algorithms. It collision avoided. This use is, however, limited by the meteo-
includes, for example, contrast enhancement, noise reduction rological conditions, because the obstacles are usually located
7
J. Bursı́k, J. Kraus and M. Štumper Automation of taxiing
in much greater distance then the markings and signs used for 2.2 Lidar
navigating. Light Detection and Ranging is an active system, using the
Visibility is a major limiting factor for the use of the reflection of laser beams to compute distances from surround-
computer vision for taxiing. This restriction is the most limit- ing objects by measuring the time between a beam emission
ing for detection airport signs, stop bars and runway holding and its return to the device. Historically, Lidar was developed
points, which need to be detected in advance in order to allow solely for determining distances from objects. But, as the
for a safe stopping of an aircraft. On the other hand, it is computational power increases, it finds use in much more
not a limiting factor in case of taxiway centerline detection, demanding applications, such as creating 3D maps of airport
because the location of the camera can be chosen in such obstacles, or detection of road and obstacles by an autonomous
way, that the ground is being recorded from relatively short car. Regarding the fact that Lidar is already successfully being
distance and the surrounding area can be lit up by the taxi used in driver’s assistant systems and in autonomous cars, it
lights. With such short distance, a good visibility of markings is possible to expect, that it can be used in the standardized
can be ensured even in a thick fog. However, in all cases of conditions of airports with similarly successful results.
the use of computer vision, it is necessary to ensure that all Lidar is usually composed of an evaluation unit and a
signs and markings are visible and not damaged. head, containing all the sensors and laser beams transmitters
Figure 2 shows an example of taxiway centerline detec- (see Fig. 3). The device continuously emits invisible laser
tion using only the basic algorithm probabilistic hough line beams, which reflect off objects back to receivers placed in
transform on a relatively poor quality image in grayscale (left) the head. Based on the information about intensity and an
without any filtering. On the right side of the figure is the re- angle of reflected beam, the evaluation unit processes a 3D
sulting matrix of data, containing information about detected map of surroundings, which is continuously updated [5, 6].
lines, converted to an image. The pink lines represent a de-
The Lidar principle suggests that the Lidar, although ca-
tected object, which is in this case taxiway centerline. It is
pable of creating an accurate model of objects, is not capable
noticeable that despite the absence of any filtering, there is
of reading airport signs. It is, however, possible to analyze
quite clear differentiation of the centerline from its surround-
the surface of a taxiway and detect markings based on their
ings. Using additional filters, such as color filter, the contrast
specific reflecting characteristics [7].
can be further increased and the resulting values compared
with the standard definition of the taxiway centerline. As well The principle of using Lidar technology for taxiing is
as the lines, curves can be detected and by their curvature, similar to the computer vision with the difference that Lidar
the turn radius can be determined in advance and optionally a detects objects and markings based on their reflecting char-
speed can be determined based on the radius. acteristics of laser beams. Figure 4 shows the principle of
To summarize the computer vision, its advantages are a taxiing using Lidar.
very accurate navigation along markings on a taxiway, abil- The Lidar system is, just as the computer vision, installed
ity to read signs, obstacle detection and a minimal Time to on an airplane and has similar advantages and disadvantages.
Alert [4]. Great potential for the use of computer vision on It detects obstacles and creates an accurate model of the air-
airports lies in the airport design philosophy itself, when great plane’s surroundings and is also capable of determining the
emphasis is given to contrast of signs and markings. However, distance from these obstacles.
there are disadvantages as well, such as the already mentioned
dependency on meteorological conditions, especially visibil-
ity and the state of taxiways (layer of snow prohibiting the
camera from scanning the markings). It is also impossible to
determine airplane’s location at the airport, but this could be
dealt with by combining the computer vision with the GNSS.
Figure 2. Example of the use of computer vision. Figure 3. Lidar sensor [8].
8
J. Bursı́k, J. Kraus and M. Štumper Automation of taxiing
Figure 4. Block diagram of taxiing using Lidar. Figure 5. Current process of taxiing.
The main disadvantage is the dependency on meteorolog- The need of automation in aviation was evident in the
ical conditions. Considerable limitation in comparison with relatively distant past, when the first autopilot systems were
computer vision is the inability to detect light emitting stop developed, using feedback loops for controlling the airplane
bars. On the other hand, Lidar does not require the illumi- in the air. However, deployment of automated systems for
nation of detected objects, therefor is capable of detecting taxiing could not take place, because there was not technology
obstacles even at night. capable of replacing the human. Currently, such systems
already exist and are successfully tested in transportation
industry.
3. Taxiing automation
The automation of motion control technology is in general
The previous sections discussed systems that could enable challenging and requires the integration of navigation systems
automation of taxiing. The main problem is the navigational and systems for actuators control (nose gear steering, braking
aspect of taxiing, when the objective is a reliable determi- intensity and engine power). The principle of automated
nation of airplane’s location anywhere at the airport and its taxiing is depicted in the following Fig. 6.
comparison with an ideal taxi route. Acquired deviations from Input parameters for automated taxiing (ideal taxi route,
ideal taxi route can be further used for airplane control on information on position, speed, acceleration and direction) are
the ground, regardless of manual or automated control. For identical to the case of manual control, as described in the
going forward, it is necessary to expect, that ideal navigation previous section. Taxi route can be defined by the crew based
data and trajectory definition are available and the only task on the received clearance or directly transmitted from ATC
is to determine the deviation and its subsequent transforma- through a datalink (for example CPDLC). The rest of input
tion to required corrections, which are used to control the parameters is, however, opposed to manual control, where
airplane systems responsible for taxiing (steering, accelerat- their values is subjectively determined by the crew, deter-
ing and decelerating). The goal is to point out the principles mined precisely via navigation systems, speedometers and
for realization of automated taxiing. accelerometers, which itself eliminates the factor of human
The current standardized way of controlling an airplane on error in the process of obtaining them. These input parameters
the ground does not, with the exception of automated landing are then processed by digital computing systems in order to
and braking, use any form of automation and its execution is determine deviations of real location and speed from a situa-
the full responsibility of a crew. Simplified process is depicted tion defined by the ideal taxi route. Based on the information
in Fig. 5. Ideal taxi route is defined by clearance in the form about these deviations, the value and an orientation (direc-
of taxiway letter labels, which is given to the crew by the tion) of corrections is calculated. The corrections are then
Air Traffic Control (ATC). The information about location transformed by actuators to an angle of a nose gear, braking
and direction is acquired by looking out of the window and intensity and engine power with regard to a model of taxiing
comparing the airport signs with taxi map. The speed is gen- dynamics.
erally determined using an on board device (usually GNSS)
and its rough value is especially important to avoid exceed-
ing limits specified by the manufacturer or an airline. The Ideal taxi route
of tiller (nose gear steering wheel), throttle lever and brake Computer
determination
pedals [9, 10]. of corrections
The described process has a number of disadvantages that
stem from the absolute dependence on human factor. Manual
Nose gear deviation Control of nose
taxiing requires considerable amount of experience and takes gear, brakes
up most of the mental capacity of the crew member controlling Engine power and engine power
the aircraft, similar to when the flight is done without the use
of autopilot. Figure 6. Block diagram of automated taxiing.
9
J. Bursı́k, J. Kraus and M. Štumper Automation of taxiing
The automated process of corrections determination also vacation is minimal. In conclusion, this arrangement is not
uses feedback from engines and a nose gear to ensure smooth suitable for automated taxiing and due to great weight and
and precise maneuver execution. The described process is tak- complexity, it will be completely replaced by electronic con-
ing place constantly and thanks to the current computational trol systems.
power of computers and high sampling rate of sensors, the The second, modern solution is shown in Fig. 8. It depicts
potential reaction time is several times lower than the reaction a system used on Airbus A380 that controls the steering of
time of the flight crew. nose gear and rear wheels of the main landing gear. The
The issue of implementing automated taxiing into airport control elements located in the cockpit and hydraulic valves
ground operations has to be assessed from various perspec- arranging the turning of wheels, are connected by electric lines
tives. One of them is the actual control of those airplane parts by means of two redundant computing units (SCS), which
responsible for direction and speed. The automation of cur- evaluate the requirements for an angle of nose gear generated
rent large passenger airplanes is quite significant a directly by autopilot (ECAM), brake pedals or the steering hand wheel.
controls engines, brakes and a nose gear (during automated Such solution is, understandably, more suitable as it does not
landing). There are two technical solutions in use to control require installation of additional equipment. However, it is
the nose gear: necessary to develop a software code, that generates proper
demands on steering during taxiing.
1. Control via cables, going from the steering wheel to hy-
draulic cylinders on the nose gear (for example Boeing Brake pedals Steering hand ECAM
737). wheels
2. Control of the hydraulic barrels via electric signals gen-
erated by computing unit. The control commands are Autopilot
generated based on the position of control elements
in the cockpit and autopilot requirements during au-
tomated landing. This solution is used at the newest
airplanes (for example A380).
SCS 2
Rudder pedal Y
steering rotary Steering wheel
actuator BWS
4. Conclusion
The summary of the current systems for the control of an
airplane on the ground shows that it is necessary to develop a
Steering metering
valve module new technological solution for automation of taxiing. Mod-
Nose wheel ern airplanes (generally the ones certified for CATIIIb and
steering collar Steering actuator (2)
IIIc) are suitable for implementation of automated taxiing as
Summing mechanism they already have automated systems for brake and nose gear
control. However, these functions are used only for slow-
ing down after landing, therefor it is necessary to develop a
FWD
new software solution. The older airplanes with mechanical
controls cannot be modified for automated taxiing without
Figure 7. The older solution [11]. installation of additional equipment, therefore it is reasonable
10
J. Bursı́k, J. Kraus and M. Štumper Automation of taxiing
to expect that costs would be disproportionate to the airplane’s 2347276. URL https://doi.org/10.1109%
remaining flight time. 2Fjstars.2014.2347276.
It is understandable that the automation of taxiing will be [8]
preceded by extensive studies, development and investments. Brad Davis, Casey Goodlett, and Pat Marion. Velodyne
However, it is a logical, and with increasing traffic, a needed lidar and kitware, inc. announce the release of veloview
step forward. 1.0!, 2013. URL https://blog.kitware.com/
blog/files/174_1551943598.jpg. [Online].
Acknowledgments [9] Jakub Hospodka. Electric taxiing – taxibot system. Maga-
This paper was supported by the Grant Agency of the Czech zine of Aviation Development (MAD), 2(10):17–20, 2014.
Technical University in Prague, grant No. SGS14/212/OHK2 [10] Jakub Hospodka. Cost-benefit analysis of electric taxi
/3T/16. systems for aircraft. Journal of Air Transport Manage-
ment, 39:81–88, 2014. doi: 10.1016/j.jairtraman.2014.
References 05.002. URL https://doi.org/10.1016%2Fj.
[1] Marek Regula, Vladimir Socha, Patrik Kutilek, Lubos jairtraman.2014.05.002.
Socha, Karel Hana, Lenka Hanakova, and Stanislav Sz- [11] Transportation Safety Board of Canada. Aviation
abo. Study of heart rate as the main stress indicator in investigation report A10Q0213, 2010. URL http:
aircraft pilots. In Proceedings of the 16th International //www.tsb.gc.ca/eng/rapports-reports/
Conference on Mechatronics - Mechatronika 2014, pages aviation/2010/a10q0213/a10q0213.asp.
1794–1799. Institute of Electrical and Electronics Engi- [Online].
neers (IEEE), dec 2014. doi: 10.1109/mechatronika.2014.
[12] Photobucket. A380 steering system, 2012. URL http:
7018334. URL https://doi.org/10.1109%
2Fmechatronika.2014.7018334. //i15.photobucket.com/albums/a357/
[2]
thezeke/A380%20systems/0168becd.png.
Ian Moir and Allan Seabridge. Aircraft systems: mechani- [Online].
cal, electrical, and avionics subsystems integration. John
Wiley & Sons, Hoboken, NJ, 2011. ISBN 978-11199-
652-06.
[3] Richard Szeliski. Computer vision: algorithms and ap-
plications. Springer, London, UK, 2011. ISBN 978-1-
84882-934-3.
[4] Jakub Hospodka, Stanislav Szabo, and Martin Novák.
Influence of autonomous vehicles on logistics. Interna-
tional Review of Aerospace Engineering (IREASE), 8(5):
179, 2015. doi: 10.15866/irease.v8i5.8339. URL https:
//doi.org/10.15866%2Firease.v8i5.8339.
[5] Eva Endrizalová, Vladimı́r Němec, Stanislav Szabo, and
Soušek Radovan. Dynamic mathematical model of
ground support equipment utilization in aircraft techni-
cal handling. In Proceedings of 20th International Con-
ference Transport Means 2016, pages 759–762. Kauno
technologijos universitetas (KTU), oct 2016.
[6] Worrawut Pananurak, Somphong Thanok, and Manukid
Parnichkun. Adaptive cruise control for an intelligent
vehicle. In Proceedings of International Conference on
Robotics and Biomimetics 2008. (ROBIO 2008), pages
1794–1799. IEEE, 2009.
[7] Yongtao Yu, Jonathan Li, Haiyan Guan, Fukai Jia, and
Cheng Wang. Learning hierarchical features for au-
tomated extraction of road markings from 3-d mobile
LiDAR point clouds. IEEE Journal of Selected Top-
ics in Applied Earth Observations and Remote Sens-
ing, 8(2):709–726, 2015. doi: 10.1109/jstars.2014.
11