Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Adversarial learning-based skeleton synthesis with spatial-channel attention for robust gait recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

    We’re sorry, something doesn't seem to be working properly.

    Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

Person re-identification (ReID) aims to identify the same person across multiple cameras. Gait recognition is the person ReID using human gait to identify a walking person, which is an effective identification technology with many advantages, such as remote identification and without invasion. State-of-the-art solutions solve the problem of extensive annotation skeleton information and labeling process by employing encoder-decoders to reconstruct skeleton data, which encodes gait feature with a fixed-length vector limiting the performance of this architecture. In this paper, we propose an end-to-end pipeline dubbed as SCA-GAN, which assembles spatial-channel attention with GAN-like framework to synthesize skeleton sequences reversely without labeled skeleton data. A disadvantage of traditional encoder-decoder architecture is that, because of the fixed-length latent vector encoded from the encoder, the decoder fails to learn the reasonable standard to generate imperfect samples. Therefore, we design a GAN-like framework for discriminative gait feature extraction via adversarial learning. In addition, for learning the rich global information of skeleton data, the information of skeleton is extracted via convolutional block embedding locality-aware attention mechanism. Specifically, a contrastive feature loss is constructed between the gait encoder and the gait decoder to minimize their pixel-wise distance explicitly. The proof-of-principle experiments and ablation study on several benchmarks prove that the proposed method significantly outperforms gait recognition counterparts in precision.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Andersson VO, Araujo RM (2015) Person identification using anthropometric and gait data from kinect sensor. In: association-for-the-advancement-of-artificial-intelligence (AAAI), pp 425–431

  2. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: international conference on learning representations (ICLR), pp 1–15

  3. Chao H, Wang K, He Y, Zhang J, Feng J (2021) Gaitset: Cross-view gait recognition through utilizing gait as a deep set. In: IEEE transactions on pattern analysis and machine intelligence, pp 1–12

  4. Chen C, Ramanan D (2017) 3d human pose estimation = 2d pose estimation + matching. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 5759–5767

  5. Feng Y, Li Y, Luo J (2016) Learning effective gait features using lstm. In: international conference on pattern recognition (ICPR), pp 325–330

  6. Goffredo M, Bouchrika I, Carter JN, Nixon MS (2010) Self-calibrating view-invariant gait biometrics. IEEE transactions on systems, man, and cybernetics, part B (cybernetics) 40(2):997–1008

    Article  Google Scholar 

  7. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Yoshua B (2014) Generative adversarial networks. In: In Neural Information Processing Systems, pp 2672–2680

  8. Gray D, Hai T (2008) Viewpoint invariant pedestrian recognition with an ensemble of localized features. In: european conference on computer vision (ECCV), pp 262–275

  9. Han J, Bir B (2006) Individual recognition using gait energy image. IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (2):316–322

    Article  Google Scholar 

  10. He Y, Zhang J, Shan H, Wang L (2019) Multi-task gans for view-specific feature learning in gait recognition. IEEE Transactions on Information Forensics and Security 14(1):102–113

    Article  Google Scholar 

  11. Jeana F, Bergevina R, Albu AB (2009) Computing and evaluating view normalized body part trajectories. Image Vis Comput 27(9):1272–1284

    Article  Google Scholar 

  12. Kusakunniran W (2014) Recognizing gaits on spatio-temporal feature domain. IEEE Transactions on Information Forensics and Security 9(9):1416–1423

    Article  Google Scholar 

  13. Kusakunniran W, Wu Q, Zhang J, Li H (2011) Pairwise shape configuration-based psa for gait recognition under small viewing angle change. In: IEEE Iiternational conference on advanced video and signal based surveillance (AVSS), pp 17–22

  14. Kusakunniran W, Wu Q, Zhang J, Li H, Wang L (2014) Recognizing gaits across views through correlated motion co-clustering. IEEE Trans Image Process 23(2):696–709

    Article  MathSciNet  MATH  Google Scholar 

  15. Kusakunniran W, Wu Q, Zhang J, Ma Y, Li H (2013) A new view invariant feature for cross-view gait recognition. IEEE transactions on information forensics and security 8(10):1642–1653

    Article  Google Scholar 

  16. Li J, Qi L, Zhao A, Chen X, Dong J (2017) Dynamic long short-term memory network for skeleton-based gait recognition. In: IEEE smartworld, ubiquitous intelligence and computing, advanced and trusted computed, scalable computing and communications, cloud and big data computing, internet of people and smart city innovation, pp 1–6

  17. Li N, Zhao X, Ma C (2020) Jointsgait: Gait recognition based on graph convolutional networks and joints relationship pyramid mapping. arXiv:2005.08625, 1–19

  18. Liao R, An W, Yu S, Li Z, Huang Y (2020) gait recognition based on dense-view gan. In: 2020 IEEE international joint conference on biometrics (IJCB), pp 1–9

  19. Liao R, Cao C, Garcia EB, Yu S, Huang Y (2017) Pose-based temporal-spatial network (ptsn) for gait recognition with carrying and clothing variations. In: chinese conference on biometric recognition, pp 474–483

  20. Liao R, Yu S, An W, Huang Y (2019) A model-based gait recognition method with body pose and human prior knowledge. Pattern Recogn 98:1–11

    Google Scholar 

  21. Liu Y, Jiang X, Sun T, Xu K (2019) 3d gait recognition based on a cnn-lstm network with the fusion of skegei and da features. In: IEEE international conference on advanced video and signal based surveillance (AVSS), pp 1–8

  22. Luong M-T, Pham H, Manning C D (2015) Effective approaches to attention-based neural machine translation. In: empirical methods in natural language processing, pp 1412–1421

  23. Matteo M, Fossati A, Basso A, Menegatti E, Gool LV (2014) One-shot person re-identification with a consumer depth camera. Springer, Berlin, pp 161–181

    Google Scholar 

  24. Munaro M, Ghidoni S, Dizmen DT, Menegatti E (2014) A feature based approach to people re-identification using skeleton keypoints. In: IEEE international conference on robotics and automation (ICRA), pp 5644–5651

  25. Nambiar A, Bernardino A, Nascimento JC, Fred A (2017) Context-aware person re-identification in the wild via fusion of gait and anthropometric features. In: IEEE international conference on automatic face & gesture recognition, pp 973–980

  26. Raúl M-F, Tao X (2014) Uncooperative gait recognition by learning to rank. Pattern Recognit 47(12):3793–3806

    Article  Google Scholar 

  27. Rao H, Wang S, Hu X, Tan M, Da H, Cheng J, Hu B (2020) Self-supervised gait encoding with locality-aware attention for person re-identification. In: international joint conference on artificial intelligence (IJCAI), pp 898–905

  28. Rao H, Wang S, Hu X, Tan M, Guo Y, Cheng J, Liu X, Hu B (2021) A self-supervised gait encoding approach with locality-awareness for 3d skeleton based person re-identification. IEEE Trans Pattern Anal Mach Intell PP:1–17

    Article  Google Scholar 

  29. Shiraga K, Makihara Y, Muramatsu D, Echigo T, Yagi Y (2016) Geinet: View-invariant gait recognition using a convolutional neural network. In: international conference on biometrics (ICB), pp 1–8

  30. Sun J, Wang Y, Li J, Wan W, Cheng D, Zhang H (2018) View-invariant gait recognition based on kinect skeleton feature. multimedia tools and applications 77(19):24909–24935

    Article  Google Scholar 

  31. Swets DL, Weng J (1996) Using discriminant eigenfeatures for image retrieval. IEEE transactions on pattern analysis and machine intelligence 18(8):831–836

    Article  Google Scholar 

  32. Tafazzoli F, Safabakhsh R (2010) Model-based human gait recognition using leg and arm movements. Eng Appl Artif Intell 23(8):1237–1246

    Article  Google Scholar 

  33. Takemura N, Makihara Y, Muramatsu D, Echigo T, Yagi Y (2019) On input/output architectures for convolutional neural network-based cross-view gait recognition. IEEE transactions on circuits and systems for video technology 29(9):2708–2719

    Article  Google Scholar 

  34. Teepe T, Khan A, Gilg J, Herzog F, Hörmann S, Gerhard R (2021) Gaitgraph: Graph convolutional network for skeleton-based gait recognition. arXiv:2101.11228, 1–5

  35. Wang Y, Song C, Huang Y, Wang Z, Wang L (2019) Learning view invariant gait features with two-stream gan. Neurocomputing 339:245–254

    Article  Google Scholar 

  36. Woo S, Park J, Lee J-Y, So KI (2018) Cbam: Convolutional block attention module. In: european conference on computer vision (ECCV), pp 3–19

  37. Wu C, Song Y (2019) A view-invariant gait recognition algorithm based on a joint-direct linear discriminant analysis. multimedia tools and applications 78(24):35789–35811

    Article  Google Scholar 

  38. Wu Z, Huang Y, Wang L (2015) Learning representative deep features for image set analysis. IEEE transactions on multimedia 17(11):1960–1968

    Article  Google Scholar 

  39. Wu Z, Huang Y, Wang L, Wang X, Tan T (2016) A comprehensive study on cross-view gait based human identification with deep cnns. IEEE transactions on pattern analysis and machine intelligence 39(2):209–226

    Article  Google Scholar 

  40. Wu Z, Huang Y, Wang L, Wang X, Tan T (2017) A comprehensive study on cross-view gait based human identification with deep cnns. IEEE transactions on pattern analysis and machine intelligence 39(2):209–226

    Article  Google Scholar 

  41. Yu S, Chen H, Reyes EBG, Poh N (2017) Gaitgan: Invariant gait feature extraction using generative adversarial networks. In: computer vision and pattern recognition workshops (CVPRW), pp 532–539

  42. Yu S, Chen H, Wang Q, Shen L, Huang Y (2017) Invariant feature extraction for gait recognition using only one uniform model. Neurocomputing 239 (24):81–93

    Article  Google Scholar 

  43. Yu S, Liao R, An H, Garcia EB, Huang Y, Poh N (2019) Gaitganv2: Invariant gait feature extraction using generative adversarial networks. Pattern Recogn 87:179–189

    Article  Google Scholar 

  44. Zhang P, Wu Q, Xu J (2019) Vn-gan: identity-preserved variation normalizing gan for gait recognition. In: 2019 international joint conference on neural networks (IJCNN), pp 1–8

  45. Zhang P, Wu Q, Xu J (2019) Vt-gan: View transformation gan for gait recognition across views. In: 2019 international joint conference on neural networks (IJCNN), pp 1–8

  46. Zhang Y, Huang Y, Yu S, Wang L (2020) Cross-view gait recognition by discriminative feature learning. IEEE Trans Image Process 29(99):1001–1015

    Article  MathSciNet  MATH  Google Scholar 

  47. Zhao G, Liu G, Li H, Pietikainen M (2006) 3d gait recognition using multiple cameras. In: automatic face and gesture recognition (FGR06), pp 529–534

  48. Zheng W, Li L, Zhang Z, Huang Y, Wang L (2019) Relational network for skeleton-based action recognition. In: IEEE international conference on multimedia and expo, pp 826–831

  49. Zou G, Fu G, Peng X, Liu Y, Gao M, Liu Z (2021) Person re-identification based on metric learning: a survey. multimedia tools and applications 80(17):26855–26888

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (No. U1610124, 61806206, 62172417), and the Natural Science Foundation of Jiangsu Province (No. BK20180639, BK20201346), the Six Talent Peaks Project in Jiangsu Province (No. 2015-DZXX-010, 2018-XYDXX-044), the Postgraduate Research and Practice Innovation Program of Jiangsu Province (NO. KYCX21_2263).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, 221116, Jiangsu, China

    Ying Chen, Shixiong Xia, Jiaqi Zhao, Yong Zhou, Qiang Niu, Rui Yao & Dongjun Zhu

  2. Engineering Research Center of Mine Digitization of the Ministry of Education, Xuzhou, 221116, Jiangsu, People’s Republic of China

    Ying Chen, Shixiong Xia, Jiaqi Zhao, Yong Zhou, Qiang Niu, Rui Yao & Dongjun Zhu

  3. Xuzhou Guanglian Technology Co., Ltd, Xuzhou, 221116, Jiangsu, China

    Hao Chen

Authors
  1. Ying Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shixiong Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiaqi Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yong Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qiang Niu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rui Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dongjun Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shixiong Xia.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of Interest regarding the manuscript preparation and submission.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Xia, S., Zhao, J. et al. Adversarial learning-based skeleton synthesis with spatial-channel attention for robust gait recognition. Multimed Tools Appl 82, 1489–1504 (2023). https://doi.org/10.1007/s11042-022-12665-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12665-x

Keywords

Search

Navigation