Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

A multi-feature fusion algorithm for driver fatigue detection based on a lightweight convolutional neural network

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The majority of the current widely used algorithms for fatigue detection rely on shallow learning to extract fatigue characteristics and use a single feature to determine the level of fatigue. The accuracy of detection is greatly affected by individual and environmental differences, and there are certain limitations in complex scenes. To improve the accuracy and real-time performance of the fatigue detection algorithm, a new driver fatigue detection algorithm based on multi-feature fusion is proposed. This paper employs two cameras to capture photos of the driver and the road, respectively, and a lightweight convolutional neural network to extract features from the driver's face, including the eyes, mouth, and head, as well as lane departure features from the road images. The four fatigue features are analyzed and fused to comprehensively detect the driver's fatigue state. The experimental results show that the multi-feature fusion-based driver fatigue detection algorithm can not only detect the driver's fatigue condition accurately but also classify the fatigue state according to the degree of fatigue, which is useful for making effective pre-warning system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

All data and materials support their published claims and comply with field standards.

Code availability

Custom code.

References

  1. Chen Y, Xiang Z, Du, W. Improving lane detection with adaptive homography prediction. Vis Comput (2022).

  2. Li H T, Todd Z, Bielski N, et al. 3D lidar point-cloud projection operator and transfer machine learning for effective road surface features detection and segmentation. Vis Comput. 1–16 (2021)

  3. Khan, M.Q., Lee, S.: A comprehensive survey of driving monitoring and assistance systems. Sensors 19(11), 2574 (2019)

    Article  Google Scholar 

  4. Luo R C, Hsu C H, Wen Y C. Multi-model fusion on real-time drowsiness detection for telemetric robotics tracking applications. In: 2020 International Conference on Advanced Robotics and Intelligent Systems (ARIS). IEEE, 2020; pp. 1–6.

  5. Ma, D., Luo, X., Jin, S., et al.: Lane-based saturation degree estimation for signalized intersections using travel time data. IEEE Intell Transp Syst Mag 9(3), 136–148 (2017)

    Article  Google Scholar 

  6. Ma, D., Luo, X., Li, W., et al.: Traffic demand estimation for lane groups at signal-controlled intersections using travel times from video-imaging detectors. IET Intel Transport Syst 11(4), 222–229 (2017)

    Article  Google Scholar 

  7. Sikander, G., Anwar, S.: Driver fatigue detection systems: a review. IEEE Trans Intell Transp Syst 20(6), 2339–2352 (2018)

    Article  Google Scholar 

  8. Macchi, M.M., Boulos, Z., Ranney, T., et al.: Effects of an afternoon nap on nighttime alertness and performance in long-haul drivers. Accid Anal Prev 34(6), 825–834 (2002)

    Article  Google Scholar 

  9. Gao, Z., Wang, X., Yang, Y., et al.: EEG-based spatio–temporal convolutional neural network for driver fatigue evaluation. IEEE Trans Neural Netw Learn Syst 30(9), 2755–2763 (2019)

    Article  Google Scholar 

  10. Mulder, G.: Mulder—Hajonides van der Meulen W. Mental load and the measurement of heart rate variability. Ergonomics 16(1), 69–83 (1973)

    Article  Google Scholar 

  11. Abe E, Fujiwara K, Hiraoka T, et al. Development of drowsy driving accident prediction by heart rate variability analysis. In: Signal and information processing association annual summit and conference (APSIPA), 2014 Asia-Pacific. IEEE, 2014: 1-4.4

  12. Pandey N N, Muppalaneni N B. A novel drowsiness detection model using composite features of head, eye, and facial expression. Neural Comput Appl. 2022; 1–11

  13. Mandal, B., Li, L., Wang, G.S., et al.: Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Trans Intell Transp Syst 18(3), 545–557 (2016)

    Article  Google Scholar 

  14. Li L, Chen Y, Xin L. Driver fatigue detection based on mouth information. In: 2010 8th World Congress on Intelligent Control and Automation. IEEE, 2010; pp. 6058–6062.

  15. Ji, Y., Wang, S., Lu, Y., et al.: Eye and mouth state detection algorithm based on contour feature extraction. J Electron Imaging 27(5), 051205 (2018)

    Article  Google Scholar 

  16. Wan, et al.: Robust face alignment by multi-order high-precision hourglass network. IEEE Trans Image Process 30, 121–133 (2021)

    Article  Google Scholar 

  17. Wan, et al.: Robust facial landmark detection by multiorder multiconstraint deep networks. IEEE Trans Neural Netw Learn Syst 33(5), 2181–2194 (2021)

    Article  Google Scholar 

  18. Ma, et al.: Robust face alignment by dual-attentional spatial-aware capsule networks. Pattern Recogniton 122, 108297 (2022)

    Article  Google Scholar 

  19. Dehzangi O, Masilamani S. Unobtrusive driver drowsiness prediction using driving behavior from vehicular sensors. In: 2018 24th international conference on pattern recognition (ICPR). IEEE, 2018: pp. 3598–3603.

  20. Shi, S.Y., Tang, W.Z., Wang, Y.Y.: A review on fatigue driving detection. ITM Web of Conferences. EDP Sciences 12, 01019 (2017)

    Google Scholar 

  21. Yang S, Chen H, Xu F, et al. High-performance UAVs visual tracking based on siamese network. Vis Comput. (2021).

  22. Das D K, Shit S, Ray D N, et al. CGAN: closure-guided attention network for salient object detection. Vis Comput (2021).

  23. Hu, J., Xu, L., He, X., et al.: Abnormal driving detection based on normalized driving behavior. IEEE Trans. Veh. Technol. 66(8), 6645–6652 (2017)

    Article  Google Scholar 

  24. Baulk, S.D., Biggs, S.N., Reid, K.J., et al.: Chasing the silver bullet: measuring driver fatigue using simple and complex tasks. Accid Anal. Prev. 40(1), 396–402 (2008)

    Article  Google Scholar 

  25. An F-P, Liu J, Bai L. Object recognition algorithm based on optimized nonlinear activation function-global convolutional neural network. Vis Comput. 1–13 (2021)

  26. Malik H, Jin H, Xiaomin W. Lane line detection and departure estimation in a complex environment by using an asymmetric kernel convolution algorithm. Vis Comput. 519–538 (2022)

  27. Xie Y, Bian C, Murphey Y L, et al. An SVM parameter learning algorithm scalable on large data size for driver fatigue detection. In: 2017 IEEE symposium series on computational intelligence (SSCI). IEEE, 2017; pp. 1–8.

  28. Hou Q, Zhou D, Feng J. Coordinate attention for efficient mobile network design. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021; pp. 13713–13722.

  29. Biswas K, Kumar S, Banerjee S, et al. SMU: smooth activation function for deep networks using smoothing maximum technique. arXiv preprint arXiv:2111.04682, 2021.

  30. Ryou S, Jeong S G, Perona P. Anchor loss: modulating loss scale based on prediction difficulty. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019; pp. 5992–6001.

  31. Chaowei, M., Dean, L., He, H.: Lane line detection based on improved semantic segmentation. Sensors Mater 33(12), 4545–4560 (2021)

    Article  Google Scholar 

  32. Cheng, W.F., Wang, X.Y., Mao, B.G.: Research on lane line detection algorithm based on instance segmentation. Sensors 23(2), 789 (2023)

    Article  Google Scholar 

  33. Ding X, Zhang X, Ma N, et al. Repvgg: Making Vgg-style convnets great again. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021; pp. 13733–13742.

  34. Altomare, C., Bartolucci, C., Sala, L., et al.: I Kr impact on repolarization and its variability assessed by dynamic clamp. Circul Arrhythmia Electrophysiol 8(5), 1265–1275 (2015)

    Article  Google Scholar 

  35. Seifoory H, Taherkhani D, Arzhang B et al. An accurate morphological drowsy detection. lnt Proc Comput Sci Inf Technol. 2011; 21(2011):51–54

  36. Ma N, Zhang X, Zheng H T, et al. Shufflenet v2: Practical guidelines for efficient CNN architecture design. In: Proceedings of the European conference on computer vision (ECCV). 2018; pp. 116–131.

  37. Howard A, Sandler M, Chu G, et al. Searching for mobilenetv3[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019; pp. 1314–1324.

  38. Han K, Wang Y, Tian Q, et al. Ghostnet: More features from cheap operations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020; pp. 1580–1589.

Download references

Funding

This work was supported by the Anhui Provincial Natural Science Foundation under Grant (1908085ME159), and the Project funded by the Scientific Research Activities of Post-Doctoral Researchers in Anhui Province under Grant (2020B447), and Anhui University of Technology Research Institute of Environmentally Friendly Materials and Occupational Health (Wuhu) R&D special funding project (ALW2021YF05), and Anhui University of Science and Technology Postgraduate Innovation Fund Project (2022CX2068).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Anhui University of Science and Technology, Huainan, 232001, China

    Wangfeng Cheng, Xuanyao Wang & Bangguo Mao

  2. Institute of Environment-Friendly Materials and Occupational Health, Anhui University of Science and Technology, Wuhu, 241000, China

    Xuanyao Wang

  3. Shaanxi Automobile Holding Group Huainan Special Purpose Vehicle Co., Ltd, Huainan, China

    Xuanyao Wang

Authors
  1. Wangfeng Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuanyao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bangguo Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuanyao Wang.

Ethics declarations

Conflict of interest

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

This work is original and our paper has not been submitted to any other journals. The results are presented clearly, honestly and without fabrication, falsification, or inappropriate data manipulation (including image-based manipulation). We adhere to discipline-specific rules for acquiring, selecting, and processing data.

Consent to participate

Not applicable.

Consent for publication

All authors agreed with the content and gave explicit consent to publish the paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheng, W., Wang, X. & Mao, B. A multi-feature fusion algorithm for driver fatigue detection based on a lightweight convolutional neural network. Vis Comput 40, 2419–2441 (2024). https://doi.org/10.1007/s00371-023-02927-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02927-6

Keywords

Search

Navigation