Galley Proof 15/12/2017; 14:54 File: idt–1-idt305.tex; BOKCTP/xhs p. 1 1 Intelligent Decision Technologies -1 (2017) 1–9 DOI 10.3233/IDT-170305 IOS Press Towards a cognitive warning system for safer hybrid traffic Ágoston Töröka,b,c,∗ , Krisztián Vargad , Jean-Marie Pergandie , Pierre Mallete , Ferenc Honbolygóa,c , Valéria Csépea and Daniel Mestree Brain Imaging Centre, Research Centre for Natural Sciences, Hungarian Academy of Sciences, Budapest, Hungary b Systems and Control Laboratory, Institute for Computer Science and Control, Hungarian Academy of Sciences, Budapest, Hungary c Department of Cognitive Psychology, Eötvös Loránd University, Budapest, Hungary d Nokia Bell Labs, Budapest, Hungary e Aix-Marseille University, Marseille Cedex 09, France a Abstract. Technological development brings increasingly closer the era of widely available self-driving cars. However, presumably there will be a time when human drivers and self-driving cars would share the same roads. In the current paper, we propose a cognitive warning system that utilizes information collected from the behaviour of the human driver and sends warning signals to self-driving cars in case of human related emergency. We demonstrate that such risk detection can identify danger earlier than an external sensor would, based on the behaviour of the human-driven vehicle. We used data from a simulator experiment, where 21 participants slalomed between road bumps in a virtual reality environment. Occasionally, they had to react to dangerous roadside stimuli by large steering movements. We used one-class SVM to detect emergency behaviour in both steering and vehicle trajectory data. We found earlier detection of emergency based on steering wheel data, than based on vehicle trajectory data. We conclude that tracking cognitive variables of the human driver means that we can utilize the outstanding power of the brain to evaluate external stimuli. Information about the result of this evaluation (be it steering action or saccade) could be the basis of a warning signal that is readily understood by the computer of a self-driving car. Keywords: Warning system, driver behaviour, one-class SVM, t-SNE 1 2 3 4 5 6 7 8 9 1. Introduction Since 2009, when Google started testing Google Chauffeur driven cars, they accomplished driving over 1.5 million miles with only 22 documented minor accidents [1]. Interestingly, human error was found underlying all but one of these [2]. This warns to the fact that in spite of self-driving cars being a safer mode of transportation [3], a hybrid traffic of humandriven and self-driving cars is still prone to human ∗ Corresponding author: Ágoston Török, Brain Imaging Centre, Research Centre for Natural Sciences, Hungarian Academy of Sciences, Magyar tudósok körútja 2. Budapest 1117, Hungary. Tel.: +36 1 382 6819; E-mail: torok.agoston@ttk.mta.hu. faults. Human drivers are object to biological limitations (e.g. drowsiness) and tend to do multitasking in the car, thus providing suboptimal response in emergency situations [4]. Several in-car warning system designs have been implemented in order to reduce the risk of fatal outcomes [5]. In the present paper, we propose that these warning systems should not only raise the driver’s attention, but could be also used to inform other participants of the traffic, namely selfdriving cars. Widespread availably of passenger cars in the middle of the 20th century raised attention to traffic safety [9]. Since then, several different kinds of accident risk evaluation systems have been proposed. Amongst these we can distinguish three main types ISSN 1872-4981/17/$35.00 c 2017 – IOS Press and the authors. All rights reserved 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Galley Proof 2 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 15/12/2017; 14:54 File: idt–1-idt305.tex; BOKCTP/xhs p. 2 Á. Török et al. / Towards a cognitive warning system for safer hybrid traffic based on the source of data they use for estimation. These are (1) traffic data based, (2) car position based, and (3) driver behaviour based approaches. Traffic data-based approaches are typically based on traffic surveillance data and use that to evaluate the risk of accident depending on timeslot, traffic frequency and area (highway, intersection) [10–14]. Not entirely different from these systems [15–17] are those that work on the single car basis and use sensors of the master vehicle to predict risks of the peers. Current self-driving concept cars rely mostly on this technology [18]. The third type of risk evaluation systems is the set of systems that collect information from the driver. Driver behaviour-based models use gaze [19,20], facial coding [19,21], EEG [22,23], and motion trajectories [24– 26] recorded with various sensors. These solutions give very good real-time estimates that can be used to warn the driver for a potential risk of falling asleep [24,27], driving through a red light [26], or for optimal lanechanging trajectory [28]. Here, we propose that these warnings could help the hybrid traffic of human-driven and self-driving cars in the future. This way they work more as a communication channel between two agents and not as a one-way sensor, hence the term cognitive in the title. While a human driver may not be able to evaluate a warning message from a lead car in a couple milliseconds, this is not a problem for the processor of a self-driving car. Automated vehicles constantly monitor their surroundings with several sensors to provide the safest transportation possible [29]. Nonetheless, information collected inside the car’s cockpit may forego the externally detectable risk with tens or, sometimes, hundreds of milliseconds. This is true even if we take the steering wheel, where there is a few millisecond delay between the steering action and the chassis response [30]. Thus, these warnings may be extremely helpful for self-driving cars. The proposed solution could be a good example of how biological and artificial cognitive agents could co-evolve [31,32], emerging in a safer traffic infrastructure. The current proposal is not the first that promote consideration of cognitive factors in traffic safety [9,33,34], or increased communication between traffic participants [35,36]. However, it is unique in its emphasis on human-to-machine information flow. Ongoing research [17,29,36,37] is focusing on the design of optimal wireless communication between vehicles (vehicle to vehicle, V2V) and between vehicles and road-side units (vehicle to infrastructure, V2I). These 122 communication links support efficiently the drivers’ situational awareness. Although situational awareness often refers to human situational awareness [38], it bears relevance also in human-machine (or possibly even in machine-machine) situations as a general concept of information availability and use in an interaction [39]. To demonstrate whether we can potentially facilitate situational awareness of a machine, we validate our idea by predicting abrupt steering wheel turn actions of a human driver in a virtual reality simulator paradigm. Here, from time to time the driver had to make emergency steering movements to roadside stimuli [40]. In the present analysis we used the car trajectory and the steering wheel angle data to investigate how early we can detect the initiation of an emergency steering behaviour only based on data from either external sensor. In the current proof-of-concept implementation we used a one-class support vector machine (OC-SVM). SVM [41–43] is a set of machine learning models that uses support vectors (i.e. hyperplanes) in high dimensional space for classification and regression problems. Our choice of model was motivated by three main reasons. First, SVM solutions are fast and are often used in real-time applications [44]. Second, such a model can be extended, for example, a recent study presented a hybrid model of an OC-SVM and a deep belief network that outperformed a deep autoencoder in terms of speed on an anomaly detection task in high dimensional data [45]. Third, SVM can be trained even on computers with modest processing power. This latter argument is important since the current ideas may later give birth to an actual product. Presumably, people who cannot afford buying new self-driving cars would adhere to using human-driven cars, and thus would be the target audience of such an instrument. This facilitates the design of an efficient, yet inexpensive device. We hypothesized that abrupt steering movements can be readily detected using both steering and car trajectory data. Moreover, we predicted that emergency events are detected earlier based on steering than on trajectory data. We aimed to propose a general anomaly detection system that could potentially use multidimensional data (e.g. EEG, eye-tracking etc.). These sensors could provide even earlier detection of an emergency [46]. Therefore we did not include any prior expectation of the dangerous events, only data of normal driving and hence the use of OC-SVM. 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 Galley Proof 15/12/2017; 14:54 File: idt–1-idt305.tex; BOKCTP/xhs p. 3 Á. Török et al. / Towards a cognitive warning system for safer hybrid traffic 3 Fig. 1. The experimental design. (a) Participants had to slalom through road bumps on a rural road. (b) From time to time, a deer raised up its head from the bushes. If the animal was facing to the road they had to steer to the other end of the road. If the deer looked the other direction they did not have to do anything. The red rectangle serves illustrative purposes. 123 2. Methods 124 2.1. Participants 146 Twenty-three participants took part in the virtual reality experiment. Two of them experienced simulator sickness, therefore their data was excluded. The training and test data were extracted from the steering and trajectory data of the remaining 21 participants (age M = 25.29, SD = 5.54 years; age range: 19–37 years; 10 men and 11 women). All of them reported normal hearing and normal or corrected-tonormal vision. They were also tested for stereo vision (Randot test) and stereo-projection was adjusted accordingly with the interpupillary distance. All participants were right-handed. Neither of the participants had a history of neurological disorder or epilepsy. All of them had valid driving license and frequently drove a car in the past months. As inclusion criteria they had at least 50,000 km driving experience prior to the experiment. Participants were recruited volunteers from students of the Aix-Marseille University. Written informed consent was collected prior to the experiment, and the experimental protocol was designed according to the Declaration of Helsinki and was approved by the local ethical committee. 147 2.2. Experiment 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 148 The experiment took place in a cave automatic virtual environment (CAVE [47]) at the Centre de la 180 149 Realité Virtuelle de la Mediterranean (CRVM), AixMarseille University. The CAVE consisted of three backprojected, 3 by 4 meter side screens and a fiberglass screen of 3 by 3 meter on the floor. Two Barco 5000 lumen projectors illuminated each screen. Participants sat in a custom built car simulator consisting of a car seat frame and a force feedback steering wheel (Logitech G27). Sounds were coming from two loudspeakers placed on both sides of the car frame. We designed a driving simulator game in Unity 3D, where participants were told to drive on a rural road bounded by bushes on both sides. The road was flat and the scene did not contain other landmarks that may have distracted the driver’s attention. The experiment contained two kinds of tasks. Most of the time they had to slalom between road bumps. The task required continuous left/right steering movements. The road bumps appeared on both sides of the road to guarantee that only small steering movements were used, and the trial was only successful if the participant passed between the two road bumps (see Fig. 1). A green disk placed between the road bumps indicated the ideal position of passing. Running over a road bump was signalled by a small vibration on the steering wheel. This task was sometimes interrupted by an emergency event. The emergency event was the appearance of a deer in the bushes, either on the left or on the right side of the road. The orientation of the deer’s jaw signalled whether a response was required or not (Go-NoGo task). If the deer was facing the road it signalled emergency (Go signal), if it turned away then no response 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 Galley Proof 4 15/12/2017; 14:54 Á. Török et al. / Towards a cognitive warning system for safer hybrid traffic 186 was required (NoGo signal). In case of emergency, participants were instructed to steer to the other side (i.e. large steering movement) in order to avoid a collision. If the orientation of the deer did not implicate emergency, the participants were instructed to execute the primary task and not to react to the deer. 187 2.3. Procedure 181 182 183 184 185 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 File: idt–1-idt305.tex; BOKCTP/xhs p. 4    X=  ~xn ~xi ∈ R       →    r4 ∇14 ∇24 ∇34 r5 ∇15 ∇25 ∇35   .. .. .. ..  , . . . .  rn ∇1n ∇2n ∇3n where X is the input data of ~x vectors for n time points. The dimensions of ~xi are r, which is either the raw measurement of steering wheel angle or vehicle position in the ith time point, and ∇1 , ∇2 , ∇3 , which are the first, second and third order finite backward differences in that time point i, respectively. The time points start at 4 because third order finite backward differences were defined only after 3 data points. Consequently, we had a four dimensional vector available for every time point, which was used as the input of the risk prediction model. This way the model was able to handle short range dependencies of the time-series data. In the following we will refer the normal driving data as no event and the emergency data as event. Thus data points were in theory either normal (S) or emergency (S̄) points labels, these were denoted as +1 or −1 such as ( +1 if ~x ∈ S y= −1 if ~x ∈ S̄ The experiment started with a practice phase where participants were familiarized with the task. We looked for signs of simulator sickness to avoid unwanted discomfort caused by performing the task for a prolonged period. The data used in the current analysis was collected from four 5 minute-long blocks. The participants were free to take a rest, stand up, walk and drink between the blocks. The total duration of the experiment was approximately one hour, including breaks. During the experiment, emergency events appeared with 20% chance. Time between road bumps varied between 300 and 1700 msec (distance: 5.9 m to 34 m at 70 km/h speed). Emergency events always followed a road bump with 650 to 700 msec and when they appeared they were the closest visual target stimuli. Emergency events were followed by road bump with 300 to 350 msec. This way the distance between the two road bumps bounding the emergency event was equal to the average distance of two road bumps. We used this configuration to avoid that participants could anticipate the emergency events. 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 where S = {no event} and S̄ = {event} This means that we could have used the S̄ data points and train a binary classifier. However, our aim was to design a model that could detect any anomalies outside the normal range. Hence, we trained separate one-class support vector machine models (OC-SVM) for the steering angle and for the trajectory data. The OC-SVM is finding a hyperplane that identifies the boundaries of the training pattern from the origin of the feature space F [52]. Because this is often difficult in the original feature space, we mapped them using function Φ and using a Gaussian (RBF) kernel space transformation [53]. The kernel function was formulated as 2.4. Data preprocessing Data preprocessing and modelling was done in Python [48] using Pandas [49], Scikit-learn [50], visualisation was done using Matplotlib [51] and Seaborn. Trajectory and steering angle data was logged in every 50 msec with high precision, according to the Unity environment internal physics. Normal driving data was extracted from the trajectories by selecting data points outside the emergency events. Emergency event onsets were defined as the moment when the deer become visible. We defined the time window of the emergency events from −100 msec 1900 msec, 0 msec being the onset of the emergency stimulus. Both for the trajectory and for the steering angle we calculated first (speed), second (acceleration) and third order (jerk) derivatives using finite difference approximation, formulated as ~x4 ~x5 .. . exp(−γk~x − ~x′ k2 ), 262 226 245 246 247 248 249 250 251 252 253 254 255 256 257 γ = 0.25, where γ is the kernel coefficient that defines how far the influence of a single training example reaches, where low values mean far and γ ∈ R|γ > 0, ~x′ are the centroids. During training, one needs to solve the quadratic programming problem of 258 259 260 261 Galley Proof 15/12/2017; 14:54 File: idt–1-idt305.tex; BOKCTP/xhs p. 5 Á. Török et al. / Towards a cognitive warning system for safer hybrid traffic of the time points to one and the other half to the other. Because our aim was to build a model that uses both general and personalized information, we did not split the data to two pools of participants. The model gave very small amount of false alarms on the validation set: 4.86% for the steering angle data and 4.06% for the trajectory data. After this, we used the support vectors of this model to detect the earliest anomaly point in the event data. We expected significantly high detection rate of the emergency events, and earlier detection of anomalies in the steering wheel data than in the trajectory data. Emergencies were detected 645.15 (± 219.67) msec after the onset of the event. In total 2735 emergency events were detected and 8 remained undetected. As can be seen in Fig. 2 this is in the beginning of the trajectory curvature in the emergency trials meaning that we detected emergency very early in time. On the trajectory data anomalies were detected 734.54 (± 269.44) msec after the onset of the event, significantly later than in the steering angle data (t(1530) = −17.24 p < 0.001). The detection rate was not different: 2736 emergency events were detected and 7 were undetected. The reason why steering angle made earlier detection possible is the non-linear relationship between steering angle and vehicle position (see Fig. 3). We visualized the anomaly detection thresholds based on the validation set and emergency event data points using the t-Distributed Stochastic Neighbour Embedding (t-SNE) method [55]. This method efficiently visualizes high-dimensional data by using joint probabilities of a low-dimensional embedding. The transformation was run using the Barnes-Hut approximation in order to perform calculation in quasi-linear time. The results of the t-SNE show that the no event and emergency event data points are easily differentiable (see Fig. 4). Summarizing the results, we found that emergency events were readily detected both in wheel angle and in trajectory data using a OC-SVM. Steering data made possible earlier detection of emergency events than trajectory data. Fig. 2. Detection time of Emergency from steering wheel and position data. We were able to predict emergency from steering data earlier than from lateral position because of the non-linear relation between steering angle and vehicle position. Whiskers show 95 % confidence intervals for the mean. n min(~ ω , ξ, ρ) 263 that is subject to 1 1 X ξi − ρ, k~ ω k2 + 2 νn i=4 (~ ω · Φ (~xi )) > ρ − ξi , 264 265 266 267 268 269 270 ν = 0.1 ξi > 0 here, n is the number of samples, ξi are the slack variables, ω ~ is the hyperplane weight vector, ρ is the bias term. ν ∈ (0, 1] and this regularization parameter adds an upper bound on the fraction of training errors and a lower bound on the fraction of resulting support vectors. If ω and ρ solved the problem the following decision function is yielded ŷ = sign ((~ ω · Φ(~x)) − ρ) 281 which yields positive values for S. Parameters were chosen to generate the least amount of false alarms. However because we cannot be certain that the training set does not include any accidental anomalies (i.e. quick/large steering movements), we set the ν parameter so that the false alarm rate was around 5% (i.e. this would mean 1 package/sec on average with the 20 Hz sampling rate). This was used a fair trade-off between earlier detection of emergency and more false alarms. Shrinking heuristic was used in the training to speed up optimization [54]. 282 3. Results 271 272 273 274 275 276 277 278 279 280 283 As a first step, we divided the whole no event data to training and validation sets by randomly assigning half 5 4. Discussion 332 284 In the current work we proposed an in-car risk detection and warning system that could inform automatic vehicles on the road about the cautious actions of the human driver (e.g. abrupt steering movement, falling asleep). We illustrated the benefits of the risk 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 Galley Proof 6 15/12/2017; 14:54 File: idt–1-idt305.tex; BOKCTP/xhs p. 6 Á. Török et al. / Towards a cognitive warning system for safer hybrid traffic Fig. 3. Relationship between steering angle and vehicle position. It can be on the two dimensional histogram, that the position of the vehicle changes in a rather curvilinear manner relative to the steering angle (nova from the centres). The two dense centres are results of the slaloming task, where the car was either going slightly left or slightly right, the smaller circular pattern around the centres also resulted from the slaloming task. The histogram uses jet colormapping, which goes from blue through green to red. Fig. 4. t-SNE embedding of no event and earliest detected emergency event data. The embedding method clearly visualizes the decision boundaries between event and no event data. Only a fraction of 30.000 data points are displayed. 333 334 335 336 337 338 339 340 detection component by predicting dangerous steering movements earlier from wheel angle data than from vehicle trajectory data, because of the non-linear relationship between steering angle and vehicle lateral position [56,57]. We used one class support vector machine for learning and prediction. These type of models are common in outlier detection scenarios for various problems [45,58,59]. Note, that by controlling the sparsity 371 341 parameter of the SVM we can limit the number of support vectors used for prediction [54], there are even solutions to find the optimal number of support vectors for a given problem [60]. Moreover, while training an SVM (and potentially multitude of SVMs for each car on the road) would be infeasible inside a master vehicle, our proposal leads to computational efficiency since training and prediction could run on the individual peer vehicles. This fact opens the door to highly individualized models. We found earlier detection of risk in wheel angle data than in trajectory data. Although this is in line with the expectations (i.e. because of steering backlash, vehicle inertia, tire stiffness), a limitation of the current study is that it was done in virtual reality. While reactions in virtual reality are comparable to those in the real-world [61], the physics of the virtual environment are simpler than reality. Not speaking of the large variance of normal driver behaviour in real world scenarios. While in our case there were only two tasks, outside of the simulator the driver faces all the challenges of traffic. This necessitates further exploration under more naturalistic circumstances. Nonetheless, our choice of virtual reality was motivated by the fact that only this way we were able to generate large amount of clean and labelled data for training and test without real risk of accident. Further studies should evaluate the effectiveness of such a system with more degrees of freedom. Here participants were only able to control the steering wheel angle but not the speed 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 Galley Proof 15/12/2017; 14:54 File: idt–1-idt305.tex; BOKCTP/xhs p. 7 Á. Török et al. / Towards a cognitive warning system for safer hybrid traffic 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 of the car, in reality steering wheel angle changes depends on the speed of the car too, also manufacturers apply speed steering solutions in today’s cars [56]. Worthy to note, that the change of the steering wheel angle is indicative of rather distant elements of the perception-action cycle. Hence, presumably more benefit we earn from such a model when more proximal cognitive variables are tracked. Eye and face tracking in the cockpit could help detecting drowsiness very early in time [21], but also – in situations like the current experiment – could also help identifying saccades to certain stimuli inside and outside the car [8]. Wearable sensors can monitor heart rate, and therefore can be used to inform traffic peers of medical emergency. Moreover, given the increasing availability of consumer EEG headsets, it is promising that research shows electrophysiological patterns can be extremely helpful as well [22,23]. Another interesting field of exploration is the study of information transmission and potentially further propagation of data in a vehicle network [17,62,63]. This way the risk information is not only locally useful but can change the state of the global network. For example, the network could start organizing detours even when an inevitable accident has not happened yet. On the one hand, creating such a one-directional inter-cognitive link between an artificial and a biological cognitive system is an important step forward from the perspective of the applied field of cognitive infocommunications [31]. On the other hand though, it raises important concerns regarding privacy and security. These systems would monitor the driver’s reactions and while communication is only intended in case of risk, it is still a potential data breach. Moreover, malicious attack is also possible against the automated car by sending large amount of risk notifications. The communication link therefore must be secured. Indeed, current research on intelligent automated traffic, smart cities and situation awareness of self-driving cars is aware of these challenges [17,35,64,65]. Researchers working on self-driving cars say that fully automated cars are still years or even decades ahead [29,66]. Meanwhile, semi-automatic solutions are increasingly available (automatic parking, highway autopilot) [67,68]. Thus, roads are becoming more and more a niche of biological and artificial drivers. In this situation we may want artificial cognitive agents to coevolve with our biological cognitive systems. In the present work we detailed one aspect of this endeavour, namely inter-cognitive warning systems. The core of arguments was the importance of communication of the human drivers’ cognitive and behavioural states to self-driving cars to increase road safety in the future. 7 Acknowledgments The research leading to these results has received funding from the European Community’s Research Infrastructure Action – grant agreement VISIONAIR 262044-under the 7th Framework Programme (FP7/ 2007-2013). Á.T. was additionally supported by a Young Researcher Fellowship from the Hungarian Academy of Sciences. The authors would like to thank László Kovács for his valuable comments on an earlier version of this manuscript. References [1] [2] [3] [4] [5] [6] [7] [8] [9] 479 424 Google. Google Self-Driving Car Project. https://static.google usercontent.com/media/, www.google.com/hu//selfdrivingcar /.2016. LaFrance A. When Google Self-Driving Cars Are in Accidents, Humans Are to Blame. Atl 2015. Blanco M, Atwood J, Russell S, Trimble T, McClafferty J, Perez M. Automated Vehicle Crash Rate Comparison Using Naturalistic Data [Internet]. Vtti. Virginia Tech Transportation Institute; 2016. Available from: http://www.apps. vtti.vt.edu/PDFs/Automated%5CnVehicle%5CnCrash%5Cn Rate%5CnComparison%5CnUsing%5CnNaturalistic%5CnD ata_Final%5CnReport_20160107.pdf%5Cn. http://www.vtti. vt.edu/featured/?p=422. Brumby DP, Salvucci DD, Howes A. Focus on driving. In: Proceedings of the 27th international conference on Human factors in computing systems – CHI 09 [Internet]. New York, New York, USA: ACM Press; 2009 [cited 2 Nov 2014], 1629. Available from: http://dl.acm.org/citation.cfm?id=1518701. 1518950. Ho C, Spence C, Gray R. Looming auditory and vibrotactile collision warning for safe driving. In: 7th International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design [Internet]; 2013 [cited 2 Nov 2014]. Available from: http://trid.trb.org/view.aspx?id= 1263140. Török Á, Tóth Z, Honbolygó F, Csépe V. Integration of warning signals and signaled objects to a multimodal object: A pilot study. In: 2013 IEEE 4th International Conference on Cognitive Infocommunications (CogInfoCom) [Internet]. IEEE 2013 [cited 18 Sep 2014], 653-8. Available from: http: //ieeexplore.ieee.org/articleDetails.jsp?arnumber=6719183. Ho C, Reed N, Spence C. Multisensory in-car warning signals for collision avoidance. Hum Factors [Internet]. Dec 2007 [cited 2 Oct 2013]; 49(6): 1107-14. Available from: http:// www.ncbi.nlm.nih.gov/pubmed/18074709. Colonius H, Diederich A. The Multisensory Driver: Contributions from the Time-Window-of-Integration Model. In: Cacciabue PC, Hjãlmdahl M, Luedtke A, Riccioli C, eds. Human Modelling in Assisted Transportation SE – 39 [Internet]. Springer Milan 2011; 363-71. Available from: http://dx. doi.org/10.1007/978-88-470-1821-1_39. Koren C, Borsos A. Is Smeed’s law still valid? A world-wide analysis of the trends in fatality rates. J Soc Transp Traffic Stud 2013; 1(1): 64-76. 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 Galley Proof 8 480 [10] 481 482 483 [11] 484 485 486 [12] 487 488 [13] 489 490 491 492 493 [14] 494 495 496 497 [15] 498 499 500 [16] 501 502 503 [17] 504 505 [18] 506 507 508 [19] 509 510 511 512 513 [20] 514 515 516 [21] 517 518 519 520 521 522 [22] 523 524 525 526 527 [23] 528 529 530 [24] 531 532 533 534 535 536 [25] 537 538 539 540 541 542 [26] 15/12/2017; 14:54 File: idt–1-idt305.tex; BOKCTP/xhs p. 8 Á. Török et al. / Towards a cognitive warning system for safer hybrid traffic Mountain L, Fawaz B, Jarrett D. Accident prediction models for roads with minor junctions. Accid Anal Prev Elsevier 1996; 28(6): 695-707. Abdel-Aty MA, Radwan AE. Modeling traffic accident occurrence and involvement. Accid Anal Prev Elsevier 2000; 32(5): 633-42. Greibe P. Accident prediction models for urban roads. Accid Anal Prev Elsevier 2003; 35(2): 273-85. Oh JT, Yun I, Hwang JW, Han E. A comparative study on accident prediction model using nonlinear regression and artificial neural network, structural equation for rural 4-legged intersection. J Korean Soc Transp Korean Society of Transportation 2014; 32(3): 266-79. Lu T, Dunyao Z, Lixin Y, Pan Z. The traffic accident hotspot prediction: Based on the logistic regression method. In: Transportation Information and Safety (ICTIS), International Conference on IEEE 2015; 107-10. Hu W, Xiao X, Xie D, Tan T. Traffic accident prediction using vehicle tracking and trajectory analysis. In: Intelligent Transportation Systems, Proceedings IEEE 2003; 220-5. Hu W, Xiao X, Xie D, Tan T, Maybank S. Traffic accident prediction using 3-D model-based vehicle tracking. Veh Technol IEEE Trans IEEE 2004; 53(3): 677-94. Jãmsã J, Sukuvaara T, Luimula M. Vehicle in a cognitive network. Intell Decis Technol IOS Press 2015; 9(1): 17-27. Berger C, Rumpe B. Autonomous Driving – 5 years after the urban challenge: The anticipatory vehicle as a cyber-physical system. Proc Inform (September) 2012; 789-98. Ji Q, Yang X. Real-Time Eye, Gaze, and Face Pose Tracking for Monitoring Driver Vigilance. Real-Time Imaging [Internet]. Oct 2002 [cited Sep 17 2014]; 8(5): 357-77. Available from: http://www.sciencedirect.com/science/article/pii/S1077 201402902792. Peng J, Guo Y, Fu R, Yuan W, Wang C. Multi-parameter prediction of drivers’ lane-changing behaviour with neural network model. Appl Ergon Elsevier 2015; 50: 207-17. Ueno H, Kaneda M, Tsukino M. Development of drowsiness detection system. In: Proceedings of VNIS’94 – 1994 Vehicle Navigation and Information Systems Conference [Internet]. IEEE 1994 [cited Sep 17 2014]; 15-20. Available from: http:// ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=3 96873. Huang K-C, Huang T-Y, Chuang C-H, King J-T, Wang Y-K, Lin C-T, et al. An EEG-Based Fatigue Detection and Mitigation System. Int J Neural Syst [Internet]. World Scientific 2016; 26(4): 1650018. Available from: http://www. worldscientific.com/doi/10.1142/S0129065716500180. Wang H, Zhang C, Shi T, Wang F, Ma S. Real-time EEGbased detection of fatigue driving danger for accident prediction. Int J Neural Syst World Scientific 2015; 25(2): 1550002. Suzuki K, Jansson H. An analysis of driver’s steering behaviour during auditory or haptic warnings for the designing of lane departure warning system. JSAE Rev [Internet]. Jan 2003 [cited Sep 17 2014]; 24(1): 65-70. Available from: http://www.sciencedirect.com/science/article/pii/S038943040 2002473. Engström J, Johansson E, Östlund J. Effects of visual and cognitive load in real and simulated motorway driving. Transp Res Part F Traffic Psychol Behav [Internet]. Mar 2005 [cited Jun 20 2015]; 8(2 SPEC ISS): 97-120. Available from: http: //www.sciencedirect.com/science/article/pii/S136984780500 0185. Hoehener D, Green PA, Del Vecchio D. Stochastic hybrid models for predicting the behavior of drivers facing [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] 607 543 [42] the yellow-light-dilemma. In: American Control Conference (ACC), IEEE 2015; 3348-54. Kozak K, Pohl J, Birk W, Greenberg J, Artz B, Blommer M, et al. Evaluation of Lane Departure Warnings for Drowsy Drivers. Proc Hum Factors Ergon Soc Annu Meet [Internet], SAGE Publications Oct 1 2006 [cited 18 Sep 2014]; 50(22): 2400-4. Available from: http://pro.sagepub.com/cont ent/50/22/2400.abstract. Xu G, Liu L, Ou Y, Song Z. Dynamic modeling of driver control strategy of lane-change behavior and trajectory planning for collision prediction dynamic modeling of driver control strategy of lane-change behavior and trajectory planning for collision prediction. Intell Transp Syst IEEE Trans IEEE 2012; 13(September): 1138-55. Waldrop MM. Autonomous vehicles: No drivers required. Nature [Internet], 2 Feb 2015 [cited 6 Feb 2015]; 518(7537): 203. Available from: http://www.nature.com/news/autonomousvehicles-no-drivers-required-1.16832?WT.ec_id=NATURE20150206. Abe M. Vehicle Handling Dynamics: Theory and Application [Internet]. Elsevier Science 2015 [cited May 28 2016]; 322. Available from: https://books.google.com/books?id=yOzHB QAAQBAJ&pgis=1. Baranyi P, Csapó Á. Definition and synergies of cognitive infocommunications. Acta Polytech Hungarica 2012; 9(1): 6783. Baranyi P, Csapo A, Sallai G. Cognitive infocommunications (CogInfoCom). Cognitive Infocommunications (CogInfoCom) Springer 2015; 1-219. Miletics D. Human decisions at irregular overtakings. In: Cognitive Infocommunications (CogInfoCom), 2015 6th IEEE International Conference on IEEE 2015; 145-9. Chen D, Ahn S, Laval J, Zheng Z. On the periodicity of traffic oscillations and capacity drop: The role of driver characteristics. Transp Res part B Methodol Pergamon 2014; 59: 117-36. Jãmsã J. Cognitive communication for traffic safety. In: 5th IEEE International Conference on Cognitive Infocommunications, CogInfoCom – Proceedings IEEE 2014; 103-8. Sepulcre M, Gozalvez J, Hernandez J. Cooperative vehicleto-vehicle active safety testing under challenging conditions. Transp Res Part C Emerg Technol [Internet], Jan 2013 [cited Sep 17 2014]; 26: 233-55. Available from: http://www.science direct.com/science/article/pii/S0968090X12001258. Heikkilã M, Kippola T, Jãmsã J, Nykãnen A, Matinmikko M, Keskimaula J. Active antenna system for cognitive network enhancement. 5th IEEE Int Conf Cogn Infocommunications, CogInfoCom – Proc IEEE 2014; 19-24. Politis I, Brewster SA, Pollick F. Evaluating multimodal driver displays under varying situational urgency. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems – CHI ’14 [Internet]. New York, New York, USA: ACM Press, 2014 [cited 15 Oct 2014]; 4067-76. Available from: http://dl.acm.org/citation.cfm?id=2611222.2 556988. Jãmsã J, Pieskã S, Luimula M. Situation awareness in cognitive transportation systems. Spec Issue Cogn Infocommunications Infocommun J 2013; 5(4): 10-6. Kling F, Török Á, Mestre D, Pergandi J-M, Mallet P, Honbolygó F, et al. Effectiveness of warning signals in dual-task driving scenarios. In: Cognitive Science Arena III 2015. Hearst MA, Dumais ST, Osman E, Platt J, Scholkopf B. Support vector machines. Intell Syst their Appl IEEE 1998; 13(4): 18-28. Aizerman A, Braverman EM, Rozoner LI. Theoretical foun- 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 Galley Proof 15/12/2017; 14:54 File: idt–1-idt305.tex; BOKCTP/xhs p. 9 Á. Török et al. / Towards a cognitive warning system for safer hybrid traffic 608 609 610 [43] 611 612 [44] 613 614 615 616 [45] 617 618 619 620 621 [46] 622 623 624 625 626 627 [47] 628 629 630 631 632 633 634 [48] 635 636 [49] 637 638 [50] 639 640 641 [51] 642 643 [52] 644 645 646 [53] 647 648 649 [54] 650 651 [55] dations of the potential function method in pattern recognition learning. Autom Remote Control 1964; 25: 821-37. Cortes C, Vapnik V. Support-vector networks. Mach Learn Springer 1995; 20(3): 273-97. Michel P, El Kaliouby R. Real time facial expression recognition in video using support vector machines. In: Proceedings of the 5th international conference on Multimodal interfaces ACM 2003; 258-64. Erfani SM, Rajasegarar S, Karunasekera S, Leckie C. Highdimensional and large-scale anomaly detection using a linear one-class SVM with deep learning. Pattern Recognit [Internet]. Oct 2016; 58: 121-34. Available from: http://www. sciencedirect.com/science/article/pii/S0031320316300267. Steenken R, Weber L, Colonius H, Diederich A. Designing driver assistance systems with crossmodal signals: Multisensory integration rules for saccadic reaction times apply. PLoS One [Internet]. Public Library of Science; May 6 2014; 9(5): e92666. Available from: http://dx.doi.org/10.1371%2Fjournal .pone.0092666. Cruz-Neira C, Sandin DJ, DeFanti TA. Surround-screen projection-based virtual reality. In: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques – SIGGRAPH ’93 [Internet]. New York, New York, USA: ACM Press 1993 [cited Mar 16 2015]; 135-42. Available from: http://dl.acm.org/citation.cfm?id=166117.16 6134. Van Rossum G. Python Programming Language. In: USENIX Annual Technical Conference 2007. McKinney W. Pandas: A Python data analysis library. 2012; 551. Online URL http://pandas. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine learning in Python. J Mach Learn Res JMLR.org 2011; 12: 2825-30. Hunter JD. Matplotlib: A 2D graphics environment. Comput Sci Eng 2007; 9(3): 90-5. Schölkopf B, Williamson RC, Smola AJ, Shawe-Taylor J, Platt JC. Support vector method for novelty detection. NIPS 1999; 12: 582-8. Schölkopf B, Smola AJ. Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT Press 2002. Joachims T. Making large scale SVM learning practical. Universitãt Dortmund, 1999. Van der Maaten L, Hinton G. Visualizing data using t-SNE. J Mach Learn Res 2008; 9(2579–2605): 85. [56] [57] [58] [59] [60] [61] [62] [63] [64] [65] [66] [67] [68] 652 9 Minh VT. Vehicle steering dynamic calculation and simulation. Proc 23rd Symp DAAAM Int Vienna 2012; 237-42. Andrzejewski R, Awrejcewicz J. Nonlinear dynamics of a wheeled vehicle. Vol. 10. Springer Science & Business Media, 2006. Huang YH, Erdogmus D, Pavel M, Mathan S, Hild KE. A framework for rapid visual image search using single-trial brain evoked responses. Neurocomputing 2011; 74(12–13): 2041-51. Hassan AH, Lambert-Lacroix S, Pasqualini F. Real-time fault detection in semiconductor using one-class support vector machines. Int J Comput Theory Eng IACSIT Press 2015; 7(3): 191. Cotter A, Shalev-Shwartz S, Srebro N. Learning optimally sparse support vector machines. In: ICML 2013; 266-74. Lloyd D. In Touch with the Future: The Sense of Touch from Cognitive Neuroscience to Virtual Reality. Presence Teleoperators Virtual Environ [Internet]. The MIT Press; Aug 4 2014 [cited Sep 10 2014]; 23(2): 226-7. Available from: http://ww w.mitpressjournals.org/doi/abs/10.1162/PRES_r_00182?jour nalCode=pres#.VBAm2vl_uMg. Karsai M, Kivelã M, Pan RK, Kaski K, Kertész J, Barabási A-L, et al. Small but slow world: How network topology and burstiness slow down spreading. Phys Rev E APS 2011; 83(2): 25102. Wang P, González MC, Hidalgo CA, Barabási A-L. Understanding the spreading patterns of mobile phone viruses. Science, American Association for the Advancement of Science 2009; 324(5930): 1071-6. Gerla M, Lee E-K, Pau G, Lee U. Internet of vehicles: From intelligent grid to autonomous cars and vehicular clouds. In: Internet of Things (WF-IoT), IEEE World Forum on 2014; 241-6. Hubaux J-P, Capkun S, Luo J. The security and privacy of smart vehicles. IEEE Secur Priv Mag 2004; 2(LCAARTICLE-2004-007): 49-55. Urmson C. Google Self-Driving Car Project. SXSW Interactive 2016. https://www.youtube.com/watch?v=Uj-rK8V-rik. Koo J, Kwac J, Ju W, Steinert M, Leifer L, Nass C. Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. Int J Interact Des Manuf Springer 2015; 9(4): 269-75. Mok BK-J, Johns M, Lee KJ, Ive HP, Miller D, Ju W. Timing of unstructured transitions of control in automated driving. In: Intelligent Vehicles Symposium (IV), IEEE 2015; 1167-72. 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697