Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Semi-supervised anomaly detection in video surveillance by inpainting

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Video anomaly detection (VAD) task can be addressed as a semi-supervised learning problem as datasets are highly biased towards normal samples. The popular reconstruction methods train the network based on only normal images. These methods detect anomaly events by comparing the input with the reconstructed image, assuming that the network can not accurately reconstruct anomalous regions. However, these methods suffer from the serious flaw that the anomaly regions are generalized sufficiently well. This problem reduces the ability of anomaly detection by narrowing the gap between reconstructed and anomaly input images. By converting the reconstruction process into an inpainting process, we introduce Inpainting-GAN, a distinctive anomaly detection model that utilizes generative adversarial networks (GANs) as the backbone network. Each input image is partially occluded in random way by a series of disjoint masks before being fed into our model, any abnormal areas in the frame will definitely be covered. Our network is trained to restore those masked regions to normal parts, which overcomes the drawback of reconstruction models. To address the temporal continuity in videos, we have incorporated optical flow as a motion constraint during the training of generator. Both qualitative and quantitative experimental results show that our proposed method shows competitive performance in VAD compared to some reconstruction methods. The experimental results on the UCSD Ped1, Ped2 and CUHK Avenue datasets have obtained AUC values of 78.6%, 94.7% and 81.4%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availibility Statement

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM computing surveys (CSUR) 41(3):1–58

  2. Pang G, Shen C, Cao L et al (2021) Deep learning for anomaly detection: A review. ACM computing surveys (CSUR) 54(2):1–38

    Article  Google Scholar 

  3. Luo W, Liu W, Lian D et al (2021) Future frame prediction network for video anomaly detection. IEEE Trans Pattern Anal Mach Intell 44(11):7505–7520

    Article  Google Scholar 

  4. Huang C, Cao J, Ye F et al (2019) Inverse-transform autoencoder for anomaly detection. arXiv:1911.10676, 2(4)

  5. Abati D, Porrello A, Calderara S et al (2019) Latent space autoregression for novelty detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 481–490

  6. Dosovitskiy A, Fischer P, Ilg E et al (2015) Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766

  7. Zavrtanik V, Kristan M, Skočaj D (2021) Reconstruction by inpainting for visual anomaly detection. Pattern Recognit 112:107706

    Article  Google Scholar 

  8. Tung F, Zelek JS, Clausi DA (2011) Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance. Image Vis Comput 29(4):230–240

    Article  Google Scholar 

  9. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), IEEE, vol 1, pp 886–893

  10. Dalal N, Triggs B, Schmid C (2006) Human detection using oriented histograms of flow and appearance. In: Computer Vision–ECCV 2006: 9th european conference on computer vision, Graz, Austria, May 7-13, 2006. Proceedings, Part II 9, Springer, pp 428–441

  11. Kim J, Grauman K (2009) Observe locally, infer globally: a space-time mrf for detecting abnormal activities with incremental updates. In: 2009 IEEE conference on computer vision and pattern recognition, IEEE, pp 2921–2928

  12. Wang S, Miao Z (2010) Anomaly detection in crowd scene. In: IEEE 10th international conference on signal processing proceedings, IEEE, pp 1220–1223

  13. Colque RVHM, Caetano C, de Andrade MTL et al (2016) Histograms of optical flow orientation and magnitude and entropy to detect anomalous events in videos. IEEE Trans Circ Syst Vid Technol 27(3):673–682

    Article  Google Scholar 

  14. Agarwal S, Dušek O, Konstas I et al (2018) Improving context modelling in multimodal dialogue generation. In: Proceedings of the 11th international conference on natural language generation, pp 129–134

  15. Vincent P, Larochelle H, Lajoie I et al (2010) Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12)

  16. Doersch C (2016) Tutorial on variational autoencoders. arXiv:1606.05908

  17. Zhou C, Paffenroth RC (2017) Anomaly detection with robust deep autoencoders. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp 665–674

  18. Li S, Fang J, Xu H et al (2020) Video frame prediction by deep multi-branch mask network. IEEE Trans Circ Syst Vid Technol 31(4):1283–1295

    Article  Google Scholar 

  19. Hasan M, Choi J, Neumann J et al (2016) Learning temporal regularity in video sequences. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 733–742

  20. Malhotra P, Ramakrishnan A, Anand G et al (2016) Lstm-based encoder-decoder for multi-sensor anomaly detection. arXiv:1607.00148

  21. Luo W, Liu W, Gao S (2017) Remembering history with convolutional lstm for anomaly detection. In: 2017 IEEE international conference on multimedia and expo (ICME), IEEE, pp 439–444

  22. Luo W, Liu W, Lian D et al (2019) Video anomaly detection with sparse coding inspired deep neural networks. IEEE Trans Pattern Anal Mach Intell 43(3):1070–1084

    Article  Google Scholar 

  23. Gong D, Liu L, Le V et al (2019) Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1705–1714

  24. Luo W, Liu W, Gao S (2017) A revisit of sparse coding based anomaly detection in stacked rnn framework. In: Proceedings of the IEEE international conference on computer vision, pp 341–349

  25. Goodfellow I, Pouget-Abadie J, Mirza M et al (2020) Generative adversarial networks. Commun ACM 63(11):139–144

    Article  MathSciNet  Google Scholar 

  26. Schlegl T, Seeböck P, Waldstein SM et al (2017) Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: Information processing in medical imaging: 25th international conference, IPMI 2017, Boone, NC, USA, June 25-30, 2017, Proceedings, Springer, pp 146–157

  27. Akcay S, Atapour-Abarghouei A, Breckon TP (2019) Ganomaly: semi-supervised anomaly detection via adversarial training. In: Computer Vision–ACCV 2018: 14th asian conference on computer vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part III 14, Springer, pp 622–637

  28. Zenati H, Foo CS, Lecouat B et al (2018) Efficient gan-based anomaly detection. arXiv:1802.06222

  29. Akçay S, Atapour-Abarghouei A, and Breckon TP (2019) Skip-ganomaly: skip connected and adversarially trained encoder-decoder anomaly detection. In: 2019 international joint conference on neural networks (IJCNN), IEEE, pp 1–8

  30. Vu H, Phung D, Nguyen TD et al (2017) Energy-based models for video anomaly detection. arXiv:1708.05211

  31. Freund Y, Haussler D (1991) Unsupervised learning of distributions on binary vectors using two layer networks. Advances in neural information processing systems, p 4

  32. Hinton GE (2002) Training products of experts by minimizing contrastive divergence. Neural Comput 14(8):1771–1800

    Article  Google Scholar 

  33. Di Mattia F, Galeone P, De Simoni M et al (2019) A survey on gans for anomaly detection. arXiv:1906.11632

  34. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, Springer, pp 234–241

  35. Isola P, Zhu JY, Zhou T et al (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  36. Mahadevan V, Li W, Bhalodia V et al (2010) Anomaly detection in crowded scenes. In: 2010 IEEE computer society conference on computer vision and pattern recognition, IEEE, pp 1975–1981

  37. Lu C, Shi J, and Jia J (2013) Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE international conference on computer vision, pp 2720–2727

  38. Luo W, Liu W, Gao S (2017) A revisit of sparse coding based anomaly detection in stacked rnn framework. In: Proceedings of the IEEE international conference on computer vision, pp 341–349

  39. Chen T, Hou C, Wang Z et al (2018) Anomaly detection in crowded scenes using motion energy model. Multimed Tools Appl 77:14137–14152

    Article  Google Scholar 

  40. Nawaratne R, Alahakoon D, De Silva D et al (2019) Spatiotemporal anomaly detection using deep learning for real-time video surveillance. IEEE Trans Indus Inf 16(1):393–402

    Article  Google Scholar 

  41. Tudor Ionescu R, Smeureanu S, Alexe B et al (2017) Unmasking the abnormal events in video. In: Proceedings of the IEEE international conference on computer vision, pp 2895–2903

Download references

Author information

Authors and Affiliations

  1. School of Automation, China University of Geosciences, Wuhan, China

    Wei Liu, Mingqiang Gao, Shuaidong Duan & Longsheng Wei

Authors
  1. Wei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingqiang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuaidong Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Longsheng Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingqiang Gao.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Wei Liu and Mingqiang Gao are contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, W., Gao, M., Duan, S. et al. Semi-supervised anomaly detection in video surveillance by inpainting. Multimed Tools Appl 83, 47677–47698 (2024). https://doi.org/10.1007/s11042-023-17294-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-17294-6

Keywords

Search

Navigation