Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Convolutional neural network with spatio-temporal-channel attention for remote heart rate estimation

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Remote photoplethysmography (rPPG), which measures human heart rate without physical contact with the skin, has become active research in recent years. Neural networks have been introduced into rPPG for accurate pulse measurement and have achieved overwhelming results. However, there is a lack of in-depth analysis of key components of neural networks exhibiting a crucial impact on pulse extraction from video. In this paper, we present a network with attention and spatio-temporal convolutional block (ASTNet), exploiting the impact of key factors including different spatio-temporal convolutions, attention mechanism, the number of convolutional layers, and receptive field sizes. The novel attention module named spatio-temporal-channel (STC) attention is designed to jointly learn weights in spatial, temporal, and channel dimensions in a more efficient way. Extensive experiments have been conducted on two uncompressed datasets and one compressed dataset. Results show that ASTNet outperforms state-of-the-art methods in accuracy and computational time. Specifically, networks with larger receptive field sizes and more spatio-temporal blocks generally achieve better performance. Networks with pseudo 3D convolution outperform those with convolutional 3D in static videos, and the opposite is true in motion videos. The results exhibit a similar tendency both on uncompressed and compressed datasets. The proposed method improves the performance of pulse signal compared to PhysNet (the second-best approach in the compared methods), with the signal-to-noise ratio increased by 7.03%, 10.19%, 4.79%, the mean absolute error decreased by 17.95%, 14.17%, 22.76%, and the root-mean-square error decreased by 21.43%, 2.73%, 25.43%, on the PURE, Self-rPPG, and COHFACE datasets, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of data and material

All data generated or analyzed during this study are included in this published article.

References

  1. Dawson, J., Kamlin, C., Wong, C., Te Pas, A., Vento, M., Cole, T., Donath, S., Hooper, S., Davis, P., Morley, C.: Changes in heart rate in the first minutes after birth. Arch Dis. Child. Fet. Neonatal Ed. 95(3), 177–181 (2010)

    Article  Google Scholar 

  2. Klaessens, J. H., Van Den Born, M., Van Der Veen, A., Sikkens-Van De Kraats, J., van den Dungen, F. A., Verdaasdonk, R. M.: Development of a baby friendly non-contact method for measuring vital signs: first results of clinical measurements in an open incubator at a neonatal intensive care unit. In: Proc. SPIE 8935, Advanced Biomedical and Clinical Diagnostic Systems XII, 89351P (2014)

  3. Zhang, Q., Wu, Q., Zhou, Y., Wu, X., Ou, Y., Zhou, H.: Webcam-based, non-contact, real-time measurement for the physiological parameters of drivers. Measurement 100, 311–321 (2017)

    Article  Google Scholar 

  4. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Robust heart rate from fitness videos. Physiol. Meas. 38(6), 1023 (2017)

    Article  Google Scholar 

  5. Liu, Y., Jourabloo, A., Liu, X.: Learning deep models for face anti-spoofing: Binary or auxiliary supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 389–398 (2018)

  6. Poh, M.-Z., McDuff, D.J., Picard, R.W.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762–10774 (2010)

    Article  Google Scholar 

  7. Lewandowska, M., Rumiński, J., Kocejko, T., Nowak, J.: Measuring pulse rate with a webcam a non-contact method for evaluating cardiac activity. In: 2011 Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 405–410 (2011)

  8. Tsouri, G.R., Kyal, S., Dianat, S.A., Mestha, L.K.: Constrained independent component analysis approach to nonobtrusive pulse rate measurements. J. Biomed. Opt. 17(7), 077011 (2012)

    Article  Google Scholar 

  9. De Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rppg. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013)

    Article  Google Scholar 

  10. Wang, W., Brinker, B.D., Stuijk, S., De Haan, G.: Algorithmic principles of remote-ppg. IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017)

    Article  Google Scholar 

  11. De Haan, G., Leest, A.V.: Improved motion robustness of remote-ppg by using the blood volume pulse signature. Physiol. Meas. 35(9), 1913–1926 (2014)

    Article  Google Scholar 

  12. Wang, W., Den Brinker, A.C., De Haan, G.: Single element remote-ppg. IEEE Trans. Biomed. Eng. 66(7), 2032–2043 (2019)

    Article  Google Scholar 

  13. Sharma, P.K., Basavaraju, S., Sur, A.: Deep learning-based image de-raining using discrete Fourier transformation. Vis. Comput. 37, 2083–2096 (2021)

    Article  Google Scholar 

  14. Špetlík, R., Franc, V., Čech, J., Matas J.: Visual heart rate estimation with convolutional neural network. In: Proceedings of the British Machine Vision Conference (BMVC), pp. 3–6 (2018)

  15. Huang, B., Lin, C.-L., Chen, W., Juang, C.-F., Wu, X.: A novel one-stage framework for visual pulse rate estimation using deep neural networks. Biomed. Signal Process. Control 66, 102387 (2021)

    Article  Google Scholar 

  16. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4489–4497 (2015)

  17. Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3d residual networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 5533–5541 (2017)

  18. Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6299–6308 (2017)

  19. Yu, Z., Li, X., Zhao, G.: Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. In: Proceedings of the British Machine Vision Conference (BMVC), pp. 277 (2019)

  20. Niu, X., Zhao, X., Han, H., Das, A., Dantcheva, A., Shan, S., Chen, X.: Robust remote heart rate estimation from face utilizing spatial-temporal attention. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (FG), pp. 1–8 (2019)

  21. Zhao, H., Yang, D., Yu, J.: 3D target detection using dual domain attention and sift operator in indoor scenes. Vis. Comput. (2021). https://doi.org/10.1007/s00371-021-02217-z

    Article  Google Scholar 

  22. Hu, J., Shen, L., Sun, G.: Squeeze-and-Excitation networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132–7141 (2018)

  23. Woo, S., Park, J., Lee, J.Y., So Kweon, I.: CBAM: Convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  24. Li, X., Wang, W., Hu, X., Yang, J.: Selective kernel networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 510–519 (2019)

  25. Xi, L., Wu, X., Chen, W., Wang, J., Zhao, C.: Weighted combination and singular spectrum analysis based remote photoplethysmography pulse extraction in low-light environments. Med. Eng. Phys. 105(7), 103822 (2022)

    Article  Google Scholar 

  26. Chen, W., McDuff, D.: Deepphys: Video-based physiological measurement using convolutional attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 349–365 (2018)

  27. Qiu, Y., Liu, Y., Arteaga-Falconi, J., Dong, H., Saddik, A.E.: Evm-cnn: real-time contactless heart rate estimation from facial video. IEEE Trans. Multimedia 21(7), 1778–1787 (2019)

    Article  Google Scholar 

  28. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31(4), 1–8 (2012)

    Article  Google Scholar 

  29. Niu, X., Shan, S., Han, H., Chen, X.: RhythmNet: end-to-end heart rate estimation from face via spatial-temporal representation. IEEE Trans. Image Process. 29, 2409–2423 (2020)

    Article  MATH  Google Scholar 

  30. Yu, Z., Li, X., Niu, X., Shi, J., Zhao, G.: Autohr: a strong end-to-end baseline for remote heart rate measurement with neural searching. IEEE Signal Process. Lett. 27, 1245–1249 (2020)

    Article  Google Scholar 

  31. Hu, M., Qian, F., Guo, D., Wang, X., He, L., Ren, F.: ETA-rPPGNet: effective time-domain attention network for remote heart rate measurement. IEEE Trans. Instrum. Meas. 70, 1–12 (2021)

    Google Scholar 

  32. Huang, B., Chen, W., Lin, C.-L., Juang, C.-F., Xing, Y., Wang, Y., Wang, J.: A neonatal dataset and benchmark for non-contact neonatal heart rate monitoring based on spatio-temporal neural networks. Eng. Appl. Artif. Intell. 106, 104447 (2021)

    Article  Google Scholar 

  33. Yue, Z., Ding, S., Yang, S., Yang, H., Li, Z., Zhang, Y., Li, Y.: Deep super-resolution network for rppg information recovery and noncontact heart rate estimation. IEEE Trans. Instrum. Meas. 70, 1–11 (2021)

    Article  Google Scholar 

  34. Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6450–6459 (2018)

  35. Cai, J., Hu, J.: 3D RANs: 3D residual attention networks for action recognition. Vis. Comput. 36, 1261–1270 (2020)

    Article  Google Scholar 

  36. Zhu, Y., Liu, G.: Fine-grained action recognition using multi-view attentions. Vis. Comput. 36, 1771–1781 (2020)

    Article  Google Scholar 

  37. Hu, M., Ge, P., Wang, X., Lin, H., Ren, F.: A spatio-temporal integrated model based on local and global features for video expression recognition. Vis. Comput. (2021). https://doi.org/10.1007/s00371-021-02136-z

    Article  Google Scholar 

  38. Liu, S., Yuen, P. C.: A general remote photoplethysmography estimator with spatiotemporal convolutional network. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 685–692 (2020)

  39. Hu, M., Guo, D., Wang, X., Ge, P., Chu, Q.: A novel spatial-temporal convolutional neural network for remote photoplethysmography. In: Proceedings of the International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 1–6 (2019)

  40. McDuff, D.: Deep super resolution for recovering physiological information from videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1367–1374 (2018)

  41. Zhao, C., Han, W., Chen, Z., Li, Y., Feng, Y.: Remote estimation of heart rate based on multi-scale facial rois. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 278–279 (2020)

  42. Zhao, C., Zhou, M., Han, W., Feng, Y.: Anti-motion remote measurement of heart rate based on region proposal generation and multi-scale ROI fusion. IEEE Trans. Instrum. Meas. (2022). https://doi.org/10.1109/TIM.2022.3169567

    Article  Google Scholar 

  43. Hu, J., Shen, L., Albanie, S., Sun, G.: Vedaldi, A.: Gather-excite: exploiting feature context in convolutional neural networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS), pp. 9423–9433 (2018)

  44. Stricker, R., Müller, S., Gross, H.-M.: Non-contact video-based pulse rate measurement on a mobile service robot. In: Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication, pp. 1056–1062 (2014)

  45. Bobbia, S., Macwan, R., Benezeth, Y., Mansouri, A., Dubois, J.: Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognit. Lett. 124, 82–90 (2019)

    Article  Google Scholar 

  46. Zhao, C., Lin, C.-L., Chen, W., Li, Z.: A novel framework for remote photoplethysmography pulse extraction on compressed videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1299–1308 (2018)

Download references

Funding

This research was supported by National Natural Science Foundation of China under Grant No.s 61903336, 61976190, 62073294, 62002327; Key Research and Development Program of Zhejiang Province under Grant No. 2020C03070. Natural Science Foundation of Zhejiang Province under Grant No.s LY21F030015, LZ21F030003.

Author information

Authors and Affiliations

  1. College of Information Engineering, Zhejiang University of Technology, Hangzhou, 310023, China

    Changchen Zhao, Meng Hu, Feng Ju, Zan Chen, Yongqiang Li & Yuanjing Feng

Authors
  1. Changchen Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meng Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Feng Ju
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yongqiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yuanjing Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the research idea and model design. Material preparation and analysis were performed by CZ, YL, and YF. The experiments were finished by FJ and ZC. The manuscript was written by MH. The idea of STC was proposed by MH and CZ. The experiments of the revised manuscript were conducted by MH. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Yuanjing Feng.

Ethics declarations

Ethical approval

Not applicable.

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Consent to participate

Not applicable.

Consent to publication

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, C., Hu, M., Ju, F. et al. Convolutional neural network with spatio-temporal-channel attention for remote heart rate estimation. Vis Comput 39, 4767–4785 (2023). https://doi.org/10.1007/s00371-022-02624-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02624-w

Keywords

Search

Navigation