research-article

Asynchronous Tracking-by-Detection on Adaptive Time Surfaces for Event-based Object Tracking

Authors:

Hanzi WangAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 473 - 481

https://doi.org/10.1145/3343031.3350975

Published: 15 October 2019 Publication History

Abstract

Event cameras, which are asynchronous bio-inspired vision sensors, have shown great potential in a variety of situations, such as fast motion and low illumination scenes. However, most of the event-based object tracking methods are designed for scenarios with untextured objects and uncluttered backgrounds. There are few event-based object tracking methods that support bounding box-based object tracking. The main idea behind this work is to propose an asynchronous Event-based Tracking-by-Detection (ETD) method for generic bounding box-based object tracking. To achieve this goal, we present an Adaptive Time-Surface with Linear Time Decay (ATSLTD) event-to-frame conversion algorithm, which asynchronously and effectively warps the spatio-temporal information of asynchronous retinal events to a sequence of ATSLTD frames with clear object contours. We feed the sequence of ATSLTD frames to the proposed ETD method to perform accurate and efficient object tracking, which leverages the high temporal resolution property of event cameras. We compare the proposed ETD method with seven popular object tracking methods, that are based on conventional cameras or event cameras, and two variants of ETD. The experimental results show the superiority of the proposed ETD method in handling various challenging environments.

References

[1]

Dana H Ballard. 1981. Generalizing the Hough transform to detect arbitrary shapes. Pattern recognition, Vol. 13, 2 (1981), 111--122.

[2]

Patrick Bardow, Andrew J Davison, and Stefan Leutenegger. 2016. Simultaneous optical flow and intensity estimation from an event camera. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition. 884--892.

[3]

Francisco Barranco, Cornelia Fermuller, and Eduardo Ros. 2018. Real-time clustering and multi-target tracking using event-based sensors. In Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 5764--5769.

[4]

Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. 2016. Fully-convolutional siamese networks for object tracking. In Proc. of European Conference on Computer Vision. Springer, 850--865.

[5]

Erik Bochinski, Volker Eiselein, and Thomas Sikora. 2017. High-speed tracking-by-detection without using image information. In Proc. of IEEE International Conference on Advanced Video and Signal Based Surveillance . 1--6.

[6]

Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. 2014. A 240× 180 130 dB 3 μs latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, Vol. 49, 10 (2014), 2333--2341.

[7]

Luis A Camu nas-Mesa, Teresa Serrano-Gotarredona, Sio-Hoi Ieng, Ryad Benosman, and Bernabé Linares-Barranco. 2017. Event-driven stereo visual tracking algorithm to solve object occlusion. IEEE Transactions on Neural Networks and Learning Systems, Vol. 29, 9 (2017), 4223--4237.

[8]

Dorin Comaniciu and Peter Meer. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 5 (2002), 603--619.

Digital Library

[9]

Dorin Comaniciu and Visvanathan Ramesh. 2000. Mean shift and optimal prediction for efficient object tracking. In Proc. of IEEE International Conference on Image Processing, Vol. 3. 70--73.

[10]

Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, Michael Felsberg, et almbox. 2017. ECO: Efficient Convolution Operators for Tracking. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition. 6638--6646.

[11]

Guillermo Gallego, Jon EA Lund, Elias Mueggler, Henri Rebecq, Tobi Delbruck, and Davide Scaramuzza. 2017. Event-based, 6-DOF camera tracking from photometric depth maps. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 40, 10 (2017), 2402--2412.

Digital Library

[12]

Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza. 2018. A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition. 3867--3876.

[13]

Xu Gao and Tingting Jiang. 2018. OSMO: Online Specific Models for Occlusion in Multiple Object Tracking under Surveillance Scene. In Proc. of ACM on Multimedia Conference . 201--210.

Digital Library

[14]

Daniel Gehrig, Henri Rebecq, Guillermo Gallego, and Davide Scaramuzza. 2018. Asynchronous, photometric feature tracking using events and frames. In Proc. of European Conference on Computer Vision .

[15]

Arren Glover and Chiara Bartolozzi. 2017. Robust visual tracking with a freely-moving event camera. In Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems. 3769--3776.

[16]

Jo ao F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. 2015. High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 37, 3 (2015), 583--596.

Digital Library

[17]

Zdenek Kalal, Krystian Mikolajczyk, Jiri Matas, et almbox. 2012. Tracking-learning-detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, 7 (2012), 1409.

Digital Library

[18]

Hanme Kim, Stefan Leutenegger, and Andrew J Davison. 2016. Real-time 3D reconstruction and 6-DoF tracking with an event camera. In Proc. of European Conference on Computer Vision. 349--364.

[19]

Xavier Lagorce, Garrick Orchard, Francesco Galluppi, Bertram E Shi, and Ryad B Benosman. 2017. Hots: A hierarchy of event-based time-surfaces for pattern recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, 7 (2017), 1346--1359.

Digital Library

[20]

Yanjie Liang, Qiangqiang Wu, Yi Liu, Yan Yan, and Hanzi Wang. 2018. Robust Correlation Filter Tracking with Shepherded Instance-Aware Proposals. In Proc. of ACM on Multimedia Conference. 420--428.

Digital Library

[21]

Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. 2008. A 128×128 120 dB 15 μs latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, Vol. 43, 2 (2008), 566--576.

[22]

M Litzenberger, C Posch, D Bauer, AN Belbachir, P Schon, B Kohn, and H Garn. 2006. Embedded vision system for real-time object tracking using an asynchronous transient vision sensor. In Digital Signal Processing Workshop . 173--178.

[23]

H. Liu, D. P. Moeys, G. Das, D. Neil, S. Liu, and T. Delbrück. 2016. Combined frame- and event-based detection and tracking. In Proc. of IEEE International Symposium on Circuits and Systems. 2511--2514.

[24]

Min Liu and Tobi Delbruck. 2018. Adaptive time-slice block-matching optical flow algorithm for dynamic vision sensors. In British Machine Vision Conference .

[25]

Ana I Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso Garcia, and Davide Scaramuzza. 2018. Event-based vision meets deep learning on steering prediction for self-driving cars. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition . 5419--5427.

[26]

Anton Mitrokhin, Cornelia Fermüller, Chethan Parameshwara, and Yiannis Aloimonos. 2018. Event-based moving object detection and tracking. In Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems. 1--9.

[27]

Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. 2017. The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM . The International Journal of Robotics Research, Vol. 36, 2 (2017), 142--149.

Digital Library

[28]

Ben L Murphy-Baum and Gautam B Awatramani. 2018. An old neuron learns new tricks: redefining motion processing in the primate retina. Neuron, Vol. 97, 6 (2018), 1205--1207.

[29]

Bence P Ölveczky, Stephen A Baccus, and Markus Meister. 2003. Segregation of object and background motion in the retina. Nature, Vol. 423, 6938 (2003), 401.

[30]

Clyde W Oyster. 1968. The analysis of image motion by the rabbit retina. The Journal of Physiology, Vol. 199, 3 (1968), 613--635.

[31]

Ewa Piatkowska, Ahmed Nabil Belbachir, Stephan Schraml, and Margrit Gelautz. 2012. Spatiotemporal multiple persons tracking using dynamic vision sensor. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops . 35--40.

[32]

Christoph Posch, Daniel Matolin, and Rainer Wohlgenannt. 2011. A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS . IEEE Journal of Solid-State Circuits, Vol. 46, 1 (2011), 259--275.

[33]

Bharath Ramesh, Shihao Zhang, Zhi Wei Lee, Zhi Gao, Garrick Orchard, and Cheng Xiang. 2018. Long-term object tracking with a moving event camera. In British Machine Vision Conference .

[34]

Henri Rebecq, Guillermo Gallego, Elias Mueggler, and Davide Scaramuzza. 2018. EMVS: Event-based multi-view stereo-3D reconstruction with an event camera in real-time. International Journal of Computer Vision, Vol. 126, 12 (2018), 1394--1414.

Digital Library

[35]

Cedric Scheerlinck, Nick Barnes, and Robert Mahony. 2018. Continuous-time intensity estimation using event cameras. In Asian Conference on Computer Vision .

[36]

Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. 2018. Hats: histograms of averaged time surfaces for robust event-based object classification. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition . 1731--1740.

[37]

Benedict Wild. 2018. How does the brain tell self-motion from object motion? Journal of Neuroscience, Vol. 38, 16 (2018), 3875--3877.

[38]

Lingxiao Yang, Risheng Liu, David Zhang, and Lei Zhang. 2017. Deep location-specific tracking. In Proc. of ACM on Multimedia Conference . 1309--1317.

Digital Library

[39]

Mengdan Zhang, Jiashi Feng, and Weiming Hu. 2017. Robust Visual Object Tracking with Top-down Reasoning. In Proc. of ACM on Multimedia Conference. 226--234.

Digital Library

[40]

Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. 2018b. EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. In Proc. of Robotics: Science and Systems .

[41]

Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu. 2018a. Distractor-aware siamese networks for visual object tracking. In Proc. of European Conference on Computer Vision. 101--117.

[42]

Alex Zihao Zhu, Nikolay Atanasov, and Kostas Daniilidis. 2017. Event-based visual inertial odometry. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition. 5391--5399.

[43]

C Lawrence Zitnick and Piotr Dollár. 2014. Edge boxes: Locating object proposals from edges. In Proc. of European Conference on Computer Vision. 391--405.

Cited By

Tang SZhao YLv HSun MFeng YZhang Z(2024)Adaptive Optimization and Dynamic Representation Method for Asynchronous Data Based on Regional Correlation DegreeSensors10.3390/s2423743024:23(7430)Online publication date: 21-Nov-2024
https://doi.org/10.3390/s24237430
Wang ZMolloy Tvan Goor PMahony R(2024)Asynchronous Blob Tracker for Event CamerasIEEE Transactions on Robotics10.1109/TRO.2024.345441040(4750-4767)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TRO.2024.3454410
Banerjee SChopp HZhang JWang ZKang PCossairt OKatsaggelos A(2024)A Joint Intensity-Neuromorphic Event Imaging System With Bandwidth-Limited Communication ChannelIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.321477935:5(7216-7230)Online publication date: May-2024
https://doi.org/10.1109/TNNLS.2022.3214779
Show More Cited By

Index Terms

Asynchronous Tracking-by-Detection on Adaptive Time Surfaces for Event-based Object Tracking
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Tracking

Recommendations

Standard and Event Cameras Fusion for Feature Tracking
ICMVA '21: Proceedings of the 2021 International Conference on Machine Vision and Applications

Standard cameras are frame-based sensors that capture the scene at a fixed rate. They cannot provide information between two frames and suffer from the motion blur problem in high-speed robotic and vision applications. By contrast, event-based cameras ...
Object tracking on event cameras with offline–online learning

Compared with conventional image sensors, event cameras have been attracting attention thanks to their potential in environments under fast motion and high dynamic range (HDR). To tackle the lost‐track issue due to fast illumination changes under HDR ...
Robust object tracking via multi-cue fusion

A long-term object tracking method based on calibrated binocular cameras by fusing information of the two channels and binocular geometry constraints is proposed.The stereo filter which is built based on the epipolar geometry of the binocular cameras is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
660
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)4

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tang SZhao YLv HSun MFeng YZhang Z(2024)Adaptive Optimization and Dynamic Representation Method for Asynchronous Data Based on Regional Correlation DegreeSensors10.3390/s2423743024:23(7430)Online publication date: 21-Nov-2024
https://doi.org/10.3390/s24237430
Wang ZMolloy Tvan Goor PMahony R(2024)Asynchronous Blob Tracker for Event CamerasIEEE Transactions on Robotics10.1109/TRO.2024.345441040(4750-4767)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TRO.2024.3454410
Banerjee SChopp HZhang JWang ZKang PCossairt OKatsaggelos A(2024)A Joint Intensity-Neuromorphic Event Imaging System With Bandwidth-Limited Communication ChannelIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.321477935:5(7216-7230)Online publication date: May-2024
https://doi.org/10.1109/TNNLS.2022.3214779
Wang XLi JZhu LZhang ZChen ZLi XWang YTian YWu F(2024)VisEvent: Reliable Object Tracking via Collaboration of Frame and Event FlowsIEEE Transactions on Cybernetics10.1109/TCYB.2023.331860154:3(1997-2010)Online publication date: Mar-2024
https://doi.org/10.1109/TCYB.2023.3318601
Gossard TKrismer JZiegler ATebbe JZell A(2024)Table tennis ball spin estimation with an event camera2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00339(3347-3356)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00339
Wang XWang STang CZhu LJiang BTian YTang J(2024)Event Stream-Based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01821(19248-19257)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01821
Ji MWang ZYan RLiu QXu STang H(2023)SCTN: Event-based object tracking with energy-efficient deep convolutional spiking neural networksFrontiers in Neuroscience10.3389/fnins.2023.112369817Online publication date: 16-Feb-2023
https://doi.org/10.3389/fnins.2023.1123698
Gao YWang SSo H(2023)A Reconfigurable Architecture for Real-time Event-based Multi-Object TrackingACM Transactions on Reconfigurable Technology and Systems10.1145/359358716:4(1-26)Online publication date: 21-Apr-2023
https://dl.acm.org/doi/10.1145/3593587
El Shair ZRawashdeh S(2023)High-temporal-resolution event-based vehicle detection and trackingOptical Engineering10.1117/1.OE.62.3.03120962:03Online publication date: 1-Mar-2023
https://doi.org/10.1117/1.OE.62.3.031209
Sun SRen WLi JZhang KLiang MCao X(2023)Event-Aware Video Deraining via Multi-Patch Progressive LearningIEEE Transactions on Image Processing10.1109/TIP.2023.327228332(3040-3053)Online publication date: 2023
https://doi.org/10.1109/TIP.2023.3272283
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents