Gravity-Direction-Aware Joint Inter-Device Matching and Temporal Alignment between Camera and Wearable Sensors
Pages 433 - 441
Abstract
To analyze human interaction behavior in a group or crowd, identification and device time synchronization are essential but time demanding to be performed manually. To automate the two processes jointly without any calibration steps nor auxiliary sensor, this paper presents an acceleration-correlation-based method for multi-person interaction scenarios where each target person wears an accelerometer and a camera is stationed in the scene. A critical issue is how to remove the time-varying gravity direction component from wearable device acceleration, which degrades the correlation of body acceleration between the device and video, yet is hard to estimate accurately. Our basic idea is to estimate the gravity direction component in the camera coordinate system, which can be obtained analytically, and to add it to the vision-based data to compensate the degraded correlation. We got high accuracy results for 4 person-device matching with only 40 to 60 frames (4 to 6 seconds). The average timing offset estimation is about 5 frames (0.5 seconds). Experimental results suggest it is useful for analyzing individual trajectories and group dynamics at low frequencies.
References
[1]
Shervin Ardeshir and Ali Borji. 2018. Integrating egocentric videos in top-view surveillance videos: Joint identification and temporal alignment. In Proceedings of the European Conference on Computer Vision (ECCV). 285--300.
[2]
Chiraz BenAbdelkader, Ross Cutler, Harsh Nanda, and Larry Davis. 2001. Eigengait: Motion-based recognition of people using image self-similarity. In International conference on audio-and video-based biometric person authentication. Springer, 284--294.
[3]
Frederik Brudy, Christian Holz, Roman R"adle, Chi-Jui Wu, Steven Houben, Clemens Nylandsted Klokmose, and Nicolai Marquardt. 2019. Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM.
[4]
Rainer E Burkard, Mauro Dell'Amico, and Silvano Martello. 2009. Assignment problems. Springer.
[5]
Laura Cabrera-Quiros, Andrew Demetriou, Ekin Gedik, Leander van der Meij, and Hayley Hung. 2018. The MatchNMingle dataset: a novel multi-sensor resource for the analysis of social interactions and group dynamics in-the-wild during free-standing conversations and speed dates. IEEE Transactions on Affective Computing (2018).
[6]
Laura Cabrera-Quiros and Hayley Hung. 2016. Who is where?: Matching people in video to wearable acceleration during crowded mingling events. In Proceedings of the 24th ACM international conference on Multimedia (ACMMM). ACM, 267--271.
[7]
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 7291--7299.
[8]
Sarah D'Angelo and Andrew Begel. 2017. Improving communication between pair programmers using shared gaze awareness. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM, 6245--6290.
[9]
Chenyou Fan, Jangwon Lee, Mingze Xu, Krishna Kumar Singh, Yong Jae Lee, David J Crandall, and Michael S Ryoo. 2017. Identifying first-person camera wearers in third-person videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5125--5133.
[10]
Andreas Fender, Philipp Herholz, Marc Alexa, and Jörg Müller. 2018. OptiSpace: Automated Placement of Interactive 3D Projection Mapping Content. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM.
[11]
Martin A Fischler and Robert C Bolles. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, Vol. 24, 6 (1981), 381--395.
[12]
Shaogang Gong, Marco Cristani, Chen Change Loy, and Timothy M Hospedales. 2014. The re-identification challenge. In Person re-identification. Springer, 1--20.
[13]
Kensei Jo, Mohit Gupta, and Shree K Nayar. 2016. DisCo: Display-camera communication using rolling shutter sensors. ACM Transactions on Graphics (TOG), Vol. 35, 5 (2016), 150.
[14]
Brett R Jones, Hrvoje Benko, Eyal Ofek, and Andrew D Wilson. 2013. IllumiRoom: peripheral projected illusions for interactive experiences. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM, 869--878.
[15]
Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. 2015. Panoptic studio: A massively multiview system for social motion capture. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 3334--3342.
[16]
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. 2016. Deeper depth prediction with fully convolutional residual networks. In Proceedings of the Fourth international conference on 3D vision (3DV). IEEE, 239--248.
[17]
Ville M"akel"a, Mohamed Khamis, Lukas Mecke, Jobin James, Markku Turunen, and Florian Alt. 2018. Pocket transfers: Interaction techniques for transferring content from situated displays to mobile devices. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM.
[18]
Alessandro Masullo, Tilo Burghardt, Dima Damen, Toby Perrett, and Majid Mirmehdi. 2019. Who Goes There? Exploiting Silhouettes and Wearable Signals for Subject Identification in Multi-Person Environments. In Proceedings of the IEEE International Conference on Computer Vision Workshops.
[19]
Alaeddine Mihoub and Grégoire Lefebvre. 2017. Social Intelligence Modeling Using Wearable Devices. In Proceedings of the ACM International Conference on Intelligent User Interfaces (IUI). ACM.
[20]
Anton Milan, Stefan Roth, and Konrad Schindler. 2013. Continuous energy minimization for multitarget tracking. IEEE transactions on pattern analysis and machine intelligence, Vol. 36, 1 (2013), 58--72.
[21]
Daniel Olguin Olguin, Peter A Gloor, and Alex Pentland. 2009. Wearable sensors for pervasive healthcare management. In 2009 3rd International Conference on Pervasive Computing Technologies for Healthcare. IEEE, 1--4.
[22]
Thomas Plotz, Chen Chen, Nils Y Hammerla, and Gregory D Abowd. 2012. Automatic Synchronization of Wearable Sensors and Video-Cameras for Ground Truth Annotation--A Practical Approach. In Proceedings of the 16th International Symposium on Wearable Computers (ISWC). IEEE, 100--103.
[23]
Mahsan Rofouei, Andrew Wilson, AJ Brush, and Stewart Tansley. 2012. Your phone or mine?: fusing body, touch and device sensing for multi-user device-display interaction. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM, 1915--1918.
[24]
Osamu Shigeta, Shingo Kagami, and Koichi Hashimoto. 2008. Identifying a moving object with an accelerometer in a camera view. (2008), 3872--3877.
[25]
Frank Spitzer. 2013. Principles of random walk. Vol. 34. Springer Science & Business Media.
[26]
Thiago Teixeira, Deokwoo Jung, and Andreas Savvides. 2010. Tasking networked cctv cameras and mobile phones to identify and localize multiple people. In Proceedings of the 12th ACM international conference on Ubiquitous computing. ACM, 213--222.
[27]
Alessandro Vinciarelli, Maja Pantic, and Hervé Bourlard. 2009. Social signal processing: Survey of an emerging domain. Image and vision computing, Vol. 27, 12 (2009), 1743--1759.
[28]
Andrew D Wilson and Hrvoje Benko. 2014. Crossmotion: fusing device and image motion for user identification, tracking and device association. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI). ACM, 216--223.
[29]
Ryo Yonetani, Kris M Kitani, and Yoichi Sato. 2015. Ego-surfing first-person videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5445--5454.
[30]
Liang Zheng, Yi Yang, and Alexander G Hauptmann. 2016. Person re-identification: Past, present and future. arXiv preprint arXiv:1610.02984 (2016).
Index Terms
- Gravity-Direction-Aware Joint Inter-Device Matching and Temporal Alignment between Camera and Wearable Sensors
Index terms have been assigned to the content through auto-classification.
Recommendations
Camera–projector matching using unstructured video
This paper presents a novel approach for matching 2-D points between a video projector and a digital camera. Our method is motivated by camera–projector applications for which the projected image needs to be warped to prevent geometric distortion. Since ...
Joint Depth and Color Camera Calibration with Distortion Correction
We present an algorithm that simultaneously calibrates two color cameras, a depth camera, and the relative pose between them. The method is designed to have three key features: accurate, practical, and applicable to a wide range of sensors. The method ...
Camera-to-camera mapping for hybrid pan-tilt-zoom sensors calibration
SCIA'07: Proceedings of the 15th Scandinavian conference on Image analysisVideo surveillance becomes more and more extended in industry and often involves automatic calibration system to remain efficient. In this paper, a video-surveillance system that uses stationary-dynamic cameras devices is presented. The static camera is ...
Comments
Information & Contributors
Information
Published In
October 2020
548 pages
ISBN:9781450380027
DOI:10.1145/3395035
- General Chairs:
- Khiet Truong,
- Dirk Heylen,
- Mary Czerwinski,
- Program Chairs:
- Nadia Berthouze,
- Mohamed Chetouani,
- Mikio Nakano
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 December 2020
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ICMI '20
Sponsor:
ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
October 25 - 29, 2020
Virtual Event, Netherlands
Acceptance Rates
Overall Acceptance Rate 453 of 1,080 submissions, 42%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 57Total Downloads
- Downloads (Last 12 months)10
- Downloads (Last 6 weeks)1
Reflects downloads up to 01 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in