COVER FEATURE GUEST EDITORS’ INTRODUCTION

Self-Driving Cars

Mike Daily, HRL Laboratories

Swarup Medasani, MathWorks
Reinhold Behringer, Daimler Protics GmbH
Mohan Trivedi, University of California, San Diego

18 CO M PUTE R P U B LISHED BY THE IEEE COMP UTER SOCIE T Y 0 0 1 8 - 9 1 6 2 / 1 7/ $ 3 3 .0 0 © 2 0 1 7 I E E E

 Many recent technological advances have helped to pave
the way forward for fully autonomous vehicles. This special
issue explores three aspects of the self-driving car revolution:
a historical perspective with a focus on perception for
autonomous vehicles, how government policy will impact
self-driving cars technically and commercially, and how
cloud-based infrastructure plays a role in the future.

S
ignificant improvements in countries are in various stages of adop- In China, web services company
the last decade have greatly tion with respect to connected and Baidu is leading the way among Asian
advanced self-driving car autonomous vehicles, more effort is companies in bringing innovative
technology. These new capa- needed before these technologies can deep-learning technologies to self-
bilities will have profound global be reliably deployed on a large scale. driving vehicles through its Project
impacts that could markedly change Customizing and improving the exist- Apollo. Baidu’s self-driving platform
society, not to mention the significant ing automated driving technologies to offers many capabilities including
improvements they bring to the overall traffic patterns and specific scenarios obstacle perception, route planning,
efficiency, convenience, and safety of our relevant to Asia remains a major focus cloud simulation, HD maps, and
roadways and transportation systems. of research in this region. end-to-end deep learning. Whether
Addressing self-driving technology– In addition to major Asian cities this approach is adopted by a wider
related concerns is important, par- like Singapore, Shanghai, and Tokyo audience remains to be seen. The Chi-
ticularly given these broad potential building deployment sites for testing nese government has plans to encour-
impacts. Worldwide, 10 trillion auto- and evaluating self-driving cars, sig- age adoption of autonomous vehicles
mobile miles are driven each year, with nificant activity in Asia-based corpo- by 2025, particularly in the trucking
complex and novel conditions gener- rations continues to expand. For exam- industry. Several other OEMs, such as
ating millions of situations in which ple, Samsung in South Korea recently Didi Chuxing, NIO (formerly NextEv),
autonomous vehicles could fail. Yet acquired US infotainment and audio Faraday Future, Geely through its own-
there are many challenges that remain company Harman, making it a Tier ership of Volvo, and Nvidia through
across all levels of system functionality. 1 company (Tier 1 companies supply its investments in JingChi are all in
To give readers some context for the parts to original equipment manufac- the race to reshape the transportation
work covered in this special issue, we turers [OEMs] like BMW). Analysts industry in China as well as the rest of
have provided a summary of ongoing expect Samsung will now play a role the world.
work in Asia, Europe, and the United in automotive telematics, infotain- Similarly, Tata and Mahindra in
States, as well as in academia. ment, and driver-safety technology. India are making progress in develop-
In addition, Hyundai Motors is work- ing market-viable automated driving
SELF-DRIVING CARS IN ASIA ing with HDmaps to field self-driving technologies that are relevant to local
AND INDIA cars in time for the 2018 Pyeongchang conditions. And, in Vietnam, FPT has
Several Asian countries including Winter Olympic Games. In Japan, auto- set up a 1,000-person team to develop
China, Japan, Korea, Singapore, and makers are working together to make automated driving technology.
India are making significant contri- self-driving cars a reality in time for Intel, which can now claim to be
butions to the field. Although these the 2020 Tokyo Olympics. a Tier 2 automotive supplier with

DECEMBER 2017 19
 GUEST EDITORS’ INTRODUCTION

its acquisition of collision-avoidance in such countries are high, with nearly initiatives to reduce pollution rates
system purveyor Mobileye, is making 150,000 deaths every year in India by reducing automobile congestion
significant progress in Asia through alone—that is almost 400 every day. in the narrow lanes would also help
its development of deep-learning Thus, Asian countries’ governments leapfrog to electric vehicles, thereby
capabilities. The company is putting are very focused on reducing the acci- easing the transition to autonomous
together integrated heterogeneous dent rates, as well as on promoting ini- vehicles. Of course, the cost of an
platforms with multicore proces- tiatives for reducing pollution caused autonomy sensor suite must come
sors, field-programmable gate arrays by cars clogging the roads. down, because the current lowest
(FPGAs), and chips that are much Beyond the latest wave of auto- price for LiDAR is equivalent to the
needed for automated driving. mated driver-safety solutions, such as price of two entry-level cars, and, for
Singapore, with its high popula- blind-spot warnings, lane-departure a fraction of that, many Indian car
tion density, small physical area, and warnings, lane-change assistance, owners can afford to pay a chauffeur
high adherence to traffic regulations, and adaptive cruise control, success- to drive them around.
has proved to be an ideal environment ful adoption and viability of auton- Several of the campaigns across
for adopting driverless cars. nuTon- omous vehicles in Asian countries Asia are looking to leverage open
omy (recently acquired by Delphi) and will require much more work in other source or curated software, lower
Drive.ai are using Singapore as a base areas. Because safety rules are often research costs, establish strict regu-
to develop technologies impacting violated in these regions, autonomous lations, employ a minimalistic sensor
mobility, and nuTonomy has begun vehicles with current technologies suite, and exploit learning from simu-
work on self-driving taxi services. would freeze or not function properly. lations to reduce accidents, make traf-
In many countries in the region, Systems deployed in these environ- fic flow smoothly, decrease pollution,
such as India, Thailand, Indonesia, ments will need to be more aggressive and improve safety and reliability
and Vietnam, the transportation sys- and take more risks while learning to to customers, especially as we move
tems and the experience of driving sense and interpret social cues, become toward complete autonomy.
are challenged by the lack of a regula- context-aware, and understand driver
tory environment, sound infrastruc- intent as they are scaled to negoti- AUTONOMOUS VEHICLES
ture, adherence to traffic laws, and ate a plethora of scenarios. Inclem- IN EUROPE
safe driving practices. The presence of ent weather presents another chal- On the European front, several signif-
heterogeneous classes of on-road vehi- lenge for autonomous vehicles in icant autonomous vehicle research
cles, the irregular compliance to traffic these regions, particularly given that efforts have been underway since the
rules, close proximity of surrounding the generally poor road infrastruc- 1980s which laid the foundations of
vehicles especially in highly congested ture also warrants the need for better autonomous vehicle technology. In
areas, unstructured road conditions detection of hazards such as potholes, particular, the Eureka PROMETHEUS
and traversal patterns, unexpected especially if recommended speeds Project (Programme for a European
scenarios, unpredictable driving hab- have to be maintained. Traffic of Highest Efficiency and
its, and poor signage all need to be In India, which is one of the larger Unprecedented Safety [1987–1995]),
tackled with novel technologies. Often markets, a gradual adoption of auton- funded by the European Commission,
traffic chaos overburdens the lanes— omous technologies is expected, with provided support for the development
four lanes of vehicles form and try to initial deployments and expansion of many groundbreaking technolo-
use two physical lanes—reducing the in farming and mining applications. gies that would eventually lead to the
safety envelopes around vehicles to Indian OEM Mahindra is working driver-assistance systems we know
a few centimeters. Existing auton- toward providing autonomous capa- today. Aimed at improving road
omous vehicle technology can only bilities to its farm equipment, exploit- traffic safety, the project attracted
serve as a baseline upon which custom ing the structured environments and many European car manufacturers
and affordable solutions will need to regular driving patterns. There is that became participants and devel-
be further developed. Accident rates also the expectation that government oped the early prototypes of these

20 COMPUTER  W W W.CO M P U T E R .O R G /CO M P U T E R

components and approaches. The driving vehicles on public roads (as offer initial capability at less than 40
most ambitious of the program’s long as a human driver is at the con- mph within the next year. Commer-
funded projects was the develop- trols, to take over in critical situa- cial autonomous ride-sharing ser-
ment of the first fully autonomous tions), helping to realize this technol- vices could be the enabler that helps
passenger road vehicle. Many of the ogy by providing clear legislation and build out the technology while limit-
resulting technological approaches legal responsibilities for its use. Work ing complexity and avoiding cost con-
(such as the 4D approach, recursive continues on these technologies such straints. For example, Cruise, Waymo,
estimation, and clothoid modeling that it is expected that by 2020 a vehi- Uber, and Lyft are all planning service
of road curvature) set the standard cle of Level 5 autonomy (fully autono- rollouts in the next year or so in lim-
for the suite of driver-assistance mous) could be delivered. However, we ited “geo-fenced” regions.
systems we use in modern vehicles. still need to address the legal frame- A huge influx of interest and
The 4D approach also inspired the work for situations in which vehicles funding are supporting advances
NIST 4D/RCS reference architecture have no human driver, no steering in self-driving systems. Waymo has
for unmanned vehicle systems. It wheel, and no pedals. spent over $1.1 billion so far to develop
is worth noting that in this special its own internal capability, with par-
issue, author Ernst D. Dickmanns, SELF-DRIVING VEHICLES IN ticularly large investments in simula-
the architect of this 4D approach, THE US tion and mapping, as well as real-world
describes the historic development In the US, the technology for self- driving, training, and testing. Many
and significance of this concept driving cars—not to mention the busi- suppliers of sensing and perception
throughout the past few decades. ness, policy, legal, ethical, and regula- systems are pursuing lower-cost alter-
Among other prizes, he was awarded tory challenges—have received enor- natives. A number of companies are
the IEEE ITS Lifetime Achievement mous attention in the popular media. developing LiDAR systems that can
Award in 2016 for his contributions Not a day goes by without additional function in much smaller, more eas-
to autonomous vehicles. coverage, new information, and pre- ily produced packages, even on a chip
After government funding for self- dictions of lasting changes in the US (for example, Strobe, Valeo, and Vel-
driving vehicle technologies dried up transportation and transit systems. odyne). The rapid development of
in the late 1990s, further development The business models of automotive higher-quality sensing in the opti-
in this area was conducted with less manufacturers operating in the US are cal, radar, and LiDAR domains has
public participation and less media in flux. Autonomous vehicles have the helped funnel data to a variety of
coverage. However, in 2005 autono- potential to remake not only the auto- deep-learning systems to recog-
mous driving was back in the public motive industry, but transportation nize objects with improved detection
spotlight when German car manu- services and infrastructure require- accuracy and fewer false-positives. In
facturer Volks­ wagen collaborated ments for cities as well. today’s systems, processing through-
with Stanford University, and their In the US, Level 2 systems (that is, put is predominantly determined
vehicle (named “Stanley”) became the partial automation in which humans by front-end perception capability,
winner of the 2005 US DARPA Grand are still responsible) are available from which has received much attention
Challenge, beating the assumed front- a number of automotive manufactur- from non-automotive companies,
runner vehicle by Carnegie Mellon ers. For example, Cadillac’s Super- including Intel (through its acquisi-
University. Since then we have seen a Cruise on the CT6 model enables lane tion of Mobileye) and Nvidia.
lot of growth in development of driver- following with adaptive cruise con- In the US, most technology develop-
assistance technology by many car trol, while monitoring the driver’s eye ers agree that more advanced decision-
manufacturers across the world and movements to determine vigilance. making capability is needed to enable
particularly in Europe, and much of Level 3 systems (conditional auto- functionality in the difficult “corner
this technology is available in our mation with some human involve- cases.” Exactly how this can be accom-
cars today. ment) are still under development. plished is still an open question: a next
In 2017, German lawmakers set For example, the Audi 8 aims to be generation of data-driven deep-learning
the legal framework for allowing self- one of the first production cars to methods, top-down knowledge–driven

DECEMBER 2017 21
GUEST EDITORS’ INTRODUCTION

reasoning, and everything in between. systems, especially those that activate most far-reaching technological and
Another challenge is the means to controls of the car to prevent accidents, societal revolutions since the advent
prove a system is safe; there is inter- requires an accurate understanding of of the transistor.
est in a variety of possible solutions, human behavior as well as modeling However, there remain many
ranging from formal methods to of human–vehicle interactions, driver important issues in need of deeper
accelerated testing in simulation to activities while behind the wheel, and examination so that the safety, reli-
driving lots of miles. The path to fully predictable human intent. ability, and robustness of these highly
autonomous vehicles will depend In a relatively short span of about complex systems can be assured. Mov-
greatly on the future developments in three decades, a handful of vehicle ing toward vehicles with higher auton-
the areas of improved decision-mak- prototypes that passed a “self-driving” omy opens new research avenues in
ing and demostrating safety. test in carefully orchestrated demon- the areas of learning, modeling, active
strations in fewer than a dozen labs control, perception of dynamic events,
ACADEMIA’S CONTRIBUTION have so far given rise to technology and novel architectures for distributed
As in most scientific and technological to support highly automated driving cognitive systems. Furthermore, these
breakthroughs, academic research- and advanced driver-assistance sys- challenges must be addressed safely
ers have played and continue to play tems in millions of vehicles currently and within very tight time constraints
a major role in contributing innova- on the road. Highly reliable and accu- to avoid collisions or unstable opera-
tive and novel ideas, proof-of-concept rate sensing, digital mapping, embed- tion. Globally, many academic insti-
systems, and a continuous pool of ded systems, powerful computing, tutions, in partnership with industry
talented engineers and entrepre- machine perception, sensor fusion, and government agencies, are now
neurs in advancing the technology machine learning, path planning, and committed and seriously engaged in
that makes autonomous driving and control have been crucial in making the field. Interested readers can fol-
highly automated vehicles a reality. these technological advances. low IEEE publications such as IEEE
These contributions come from mul- It is the noteworthy and sustained Transactions on Intelligent Vehicles,
tiple engineering and computational contributions of academic insti- IEEE Transactions on Vehicular Technol-
science disciplines. Given the nature tutions in both the US and Europe ogies, IEEE Transactions on Robotics and
of these vehicles, intelligent systems that have proved critical: for exam- Automation, and IEEE Transactions on
orientation and real-world experi- ple, technological advances in the Intelligent Transportation Systems for
mental programs have found a spe- areas of distributed dynamic con- important and exciting new develop-
cial niche in advancing the field of trols, computer vision, and machine ments in the field. For a current techni-
autonomous driving—not only in aca- learning from the University of Cal- cal overview of autonomous vehicles,
demic research groups but also, more ifornia, Berkeley; intelligent robotic see the new book Creating Autonomous
recently, in industry. systems, machine vision, and experi- Vehicle Systems.1 We are confident that
Self-driving and highly automated mental AI from Carnegie Mellon Uni- any limitations of the current breed of
vehicles must navigate smoothly versity; simultaneous localization autonomous vehicles will be success-
and avoid obstacles, while accurately and mapping (SLAM) and robotics at fully resolved to withstand the real-
understanding the highly complex Stanford and the University of Mich- world test of time.
semantic interpretation of scene and igan; and human-centered intelli-
dynamic activities. For achieving such gent vehicles and vision systems IN THIS ISSUE
goals, further developments in per- from the University of California, We received many high-quality sub-
ception, 3D scene understanding, and San Diego, have clearly moved the missions for consideration, and after
policy planning are needed. Designing research and development of these rigorous assessment by dedicated
fully autonomous robotic vehicles that technologies to the market in a big reviewers, we selected three articles
can drive on roads typically does not way. These advances offer a glimpse that represent the arc of research and
require models of drivers and how they of how key ideas from academic labs development in self-driving vehicles.
interact with vehicles. In contrast, the have played a foundational role in In “Developing the Sense of Vision
design of intelligent driver-assistance what is rapidly becoming one of the for Autonomous Road Vehicles at

22 COMPUTER  W W W.CO M P U T E R .O R G /CO M P U T E R

UniBwM,” pioneer Ernst D. Dickmanns
describes efforts under the European
PROMETHEUS Project, which began in
1987 to lead the charge in developing ABOUT THE AUTHORS
real-time vision capabilities for auton-
omous vehicles. He traces the develop- MIKE DAILY is manager of the Systems and Technology Centers at HRL Labo-
ment of technologies, including the 4D ratories. His research interests include augmented and virtual reality, computer
approach to dynamic real-time vision vision, autonomous systems and robotics, and machine learning. Daily received
that used an internal 3D represen- an MS in computer science from the University of New Mexico. He is a Senior
tation in space and time suitable for Member of IEEE and an area editor of Computer. Contact him at mjdaily@hrl.com.
driving vehicles at highway speeds. By
the end of 1994, the systems were capa- SWARUP MEDASANI is a director of engineering at MathWorks. His research
ble of a variety of highway driving, interests include autonomous vehicles, biologically inspired cognitive systems,
including lane-following and autono- computer vision, and machine learning. Medasani received a PhD in computer
mous lane changes. engineering and computer science from the University of Missouri–Columbia.
In “Current US Federal Policy He is a Senior Member of IEEE. Contact him at shanti.medasani@mathworks.com.
Framework for Self-Driving Vehi-
cles: Opportunities and Challenges,” REINHOLD BEHRINGER is a consultant at Daimler Protics GmbH, Leinfelden-
Mina J. Hanna and Shawn C. Kimmel Echterdingen. His research interests over the past 25 years include computer
offer an in-depth view of the US fed- vision, augmented reality, human–computer interaction, autonomous vehicles,
eral government’s current policies and intelligent systems. Behringer received a PhD in engineering from Bundes­
and trends in five major issue areas: wehr University, Munich. He is a member of IEEE. Contact him at reinhold
privacy, cybersecurity, safety regu- .behringer@ieee.org.
lation, energy and the environment,
and ethical issues. These authors give MOHAN TRIVEDI is a Distinguished Professor of Engineering and director of
a summary of many of the key impacts the Laboratory for Intelligent and Safe Automobiles (LISA) at the University
self-driving vehicles will have, as well of California, San Diego. His research interests include intelligent vehicles,
as the principal motivators driving machine vision, human–robot interactivity, and human-centered autonomous
both technology development and driving. Trivedi received a PhD from Utah State University. He is a Fellow of
government investment in this area. IEEE, SPIE, and IAPR. He served as chair of the Robotics Technical Committee
They also discuss the many chal- of the IEEE Computer Society and is on the Board of Governors of the IEEE
lenges and opportunities for autono- Intelligent Transportation Systems Society. Contact him at mtrivedi@ucsd.edu.
mous vehicle technology developers.
In “A Unified Cloud Platform for
Autonomous Driving,” Shaoshan Liu,
Jie Tang, Chao Wang, Quan Wang, and
Jean-Luc Gaudiot describe the features
and implementation of a future auton- autonomous driving services, includ- Computer’s editor in chief and staff
omous driving cloud, an essential ing data storage, simulation, high- for their hard work and assistance
part of the autonomous driving tech- definition (HD) map generation, and in preparing this special issue.
nology stack. This unified infrastruc- deep-learning model training, all built Self-driving cars are the future, and
ture provides distributed computing on a heterogeneous computing layer. we’re almost there!
and distributed storage capabilities,
as well as heterogeneous computing REFERENCE

W
for autonomous vehicles. Using this e hope you enjoy this spe- 1. S. Liu et al., Creating Autonomous
infrastructure, the authors describe cial issue. We thank the Vehicle Systems, Morgan Claypool
how to implement several essential authors, reviewers, and Publishers, 2017.

DECEMBER 2017 23