Abstract
Large amount of surveillance video data needs effective and interactive methods to record and utilize. Traditional methods mainly focus on low-level analysis of surveillance video data and lack user interaction. Therefore, there is a strong demand of easy-to-use interaction for efficiently analyzing surveillance video information. In this paper, we propose a multi-scale approach merged on the data flow, objects, and frames to achieve visualization of surveillance video data. Combined with the advantage of sketch interaction, the design of multi-scale structure makes the analysis of surveillance content natural and fluent with annotation of video contents. Extensive user studies demonstrate the effectiveness for facilitating users’ interactive visual analysis in surveillance video exploration and significantly reduce playback time to confirm available information.
Similar content being viewed by others
References
Babaguchi N, Fujimoto Y, Yamazawa K, Yokoya N (2002) A system for visualization and summarization of omnidirectional surveillance video. In: Multimedia Information Systems. Citeseer, pp 18–27
Bao X, Javanbakhti S, Zinger S, Wijnhoven R et al (2013) Context modeling combined with motion analysis for moving ship detection in port surveillance. J Electron Imaging 22(4):041114
Barnes C, Goldman DB, Shechtman E, Finkelstein A (2010) Video tapestries with continuous temporal zoom. In: ACM Transactions on Graphics (TOG), vol 29. ACM, p 89
Benfold B, Reid I (2011) Stable multi-target tracking in real-time surveillance video. In: Computer Vision & Pattern Recognition
Browne J, Lee B, Carpendale S, Riche N, Sherwood T (2011) Data analysis on interactive whiteboards through sketch-based interaction. In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. ACM, pp 154–157
Chen L, Shen J, Wang W, Ni B (2015) Video object segmentation via dense trajectories. IEEE Trans Multimed 17(12):2225–2234
Collomosse J, McNeill G, Qian Y (2009) Storyboard sketches for content based video retrieval. In: 2009 IEEE 12th International Conference on Computer Vision. IEEE, pp 245–252
Fang X, Xia Z, Su C, Xu T, Tian Y, Wang Y, Huang T (2013) A system based on sequence learning for event detection in surveillance video. In: 2013 IEEE International Conference on Image Processing. IEEE, pp 3587–3591
Frejlichowski D, Gościewska K, Forczmański P, Hofman R (2015) Application of foreground object patterns analysis for event detection in an innovative video surveillance system. Pattern Anal Applic 18(3):473–484
Frey BJ, Dueck D (2007) Clustering by passing messages between data points. Science 315(5814):972–976
Gantz J, Reinsel D (2012) The digital universe in 2020: big data, bigger digital shadows, and biggest growth in the far east. IDC iView: IDC Analyze the Future 2007(2012):1–16
Gelgon M, Bouthemy P (1998) Determining a structured spatio-temporal representation of video content for efficient visualization and indexing. In: European Conference on Computer Vision. Springer, pp 595–609
Goyette N, Jodoin PM, Porikli F, Konrad J, Ishwar P (2012) Changedetection. net: a new change detection benchmark dataset. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, pp 1–8
Hu R, James S, Wang T, Collomosse J (2013) Markov random fields for sketch based video retrieval. In: Proceedings of the 3rd ACM Conference on International Conference on Multimedia Retrieval. ACM, pp 279–286
Jang S, Elmqvist N, Ramani K (2015) Motionflow: visual abstraction and aggregation of sequential patterns in human motion tracking data. IEEE Trans Vis Comput Graph 22(1):21– 30
Javanbakhti S, Zinger S, de With P (2014) Context-based region labeling for event detection in surveillance video. In: 2014 International Conference on Information Science, Electronics and Electrical Engineering, vol 1. IEEE, pp 94–98
Jiang F, Wu Y, Katsaggelos AK (2007) Abnormal event detection from surveillance video by dynamic hierarchical clustering. In: 2007 IEEE International Conference on Image Processing, vol 5. IEEE, pp v–145
Liu YJ, Tang K, Joneja A (2005) Sketch-based free-form shape modelling with a fast and stable numerical engine. Comput Graph 29(5):771–786
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110
Meng J, Yuan J, Yang J, Wang G, Tan YP (2015) Object instance search in videos via spatio-temporal trajectory discovery. IEEE Trans Multimedia 18(1):116–127
Moxley E, Mei T, Manjunath BS (2010) Video annotation through search and graph reinforcement mining. IEEE Trans Multimed 12(3):184–193
Noh S, Jeon M (2012) A new framework for background subtraction using multiple cues. In: Asian Conference on Computer Vision. Springer, pp 493–506
Russell BC, Torralba A, Murphy KP, Freeman WT (2008) Labelme: a database and web-based tool for image annotation. Int J Comput Vis 77(1-3):157–173
Sabirin H, Kim J, Kim M (2011) Graph-based object detection and tracking in h. 264/avc bitstreams for surveillance video. In: 2011 IEEE International Conference on Multimedia and Expo. IEEE, pp 1–6
Sun Z, Liu J (2005) Informal user interfaces for graphical computing. lncs 3784. In: Jiang W, Sun Z (eds) LNCS 3784 (2005). Citeseer, pp 675–682
Verma KK, Kumar P, Tomar A (2015) Analysis of moving object detection and tracking in video surveillance system. In: 2015 2nd International Conference on Computing for Sustainable Global Development (INDIACom). IEEE, pp 1758–1762
Wang H, Ma C (2013) Interactive multi-scale structures for summarizing video content. Sci China Inform Sci 56(5):1–12
Wang J, Zhang G (2011) Video data mining based on k-means algorithm for surveillance video. In: 2011 International Conference on Image Analysis and Signal Processing. IEEE, pp 623–626
Wang M, Hong R, Li G, Zha ZJ, Yan S, Chua TS (2012) Event driven web video summarization by tag localization and key-shot identification. IEEE Trans Multimed 14(4):975–985
Whitaker RT, Mirzargar M, Kirby RM (2013) Contour boxplots: a method for characterizing uncertainty in feature sets from simulation ensembles. IEEE Trans Vis Comput Graph 19(12):2713– 2722
Xu W, Zhang Y, Lu J, Tian Y, Wang J (2011) A framework of simple event detection in surveillance video. In: International Conference on Intelligent Computing and Information Science. Springer, pp 556–561
Yuk JS, Wong KYK, Chung RH, Chow K, Chin FY, Tsang KS (2007) Object-based surveillance video retrieval system with real-time indexing methodology. In: International Conference Image Analysis and Recognition. Springer, pp 626–637
Zhang S, Yu X, Sui Y, Zhao S, Zhang L (2015) Object tracking with multi-view support vector machines. IEEE Trans Multimed 17(3):265–278
Zhang T, Chowdhery A, Bahl PV, Jamieson K, Banerjee S (2015) The design and implementation of a wireless video surveillance system. In: Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, pp 426–438
Zhang T, Liu S, Xu C, Lu H (2012) Mining semantic context information for intelligent video surveillance of traffic scenes. IEEE Trans Ind Inf 9(1):149–160
Funding
This work was supported by the National Key Research and Development Plan under Grant 2016YFB1001200, and the Natural Science Foundation of China under Grant 61872346.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhang, Z., Zuo, R., Guo, R. et al. Multi-scale visualization based on sketch interaction for massive surveillance video data. Pers Ubiquit Comput 25, 1027–1037 (2021). https://doi.org/10.1007/s00779-019-01281-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00779-019-01281-6