research-article

Finding Dory in the Crowd: Detecting Social Interactions using Multi-Modal Mobile Sensing

Authors:

Kleomenis Katevas,

Katrin Hänsel,

Ilias Leontiadis,

Laurissa TokarchukAuthors Info & Claims

SenSys-ML 2019: Proceedings of the 1st Workshop on Machine Learning on Edge in Sensor Systems

Pages 37 - 42

https://doi.org/10.1145/3362743.3362959

Published: 10 November 2019 Publication History

Abstract

Remembering our day-to-day social interactions is challenging even if you aren't a blue memory challenged fish. The ability to automatically detect and remember these types of interactions is not only beneficial for individuals interested in their behavior in crowded situations, but also of interest to those who analyze crowd behavior. Currently, detecting social interactions is often performed using ethnographic studies, computer vision techniques and manual annotation-based data analysis. However, mobile phones offer easier means for data collection that is easy to analyze and can preserve the user's privacy. In this work, we present a system for detecting stationary social interactions inside crowds, leveraging multi-modal mobile sensing data such as Bluetooth Smart (BLE), accelerometer and gyroscope. To inform the development of such system we conducted a study with 24 participants where we asked them to socialize with each other for 45 minutes. We built a machine learning system based on gradient-boosted trees that predicts both 1:1 and group interactions with a 30.2% performance increase compared to a proximity-based approach. By utilizing a community detection-based method, we further detected the various group formation that exist within the crowd. Using mobile phone sensors already carried by the majority of people in a crowd makes our approach particularly well suited to real-life analysis of crowd behavior and influence strategies.

References

[1]

Apple Inc. Getting Started with iBeacon v1.0. https://developer.apple.com/ibeacon/Getting-Started-with-iBeacon.pdf, 2014. [Online; accessed 07-April-2018].

[2]

L. Bazzani, M. Cristani, and V. Murino. Decentralized particle filter for joint individual-group tracking. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 1886--1893, June 2012.

[3]

V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre. Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008(10):P10008, 2008.

[4]

L. Breiman. Random forests. Mach. Learn., 45(1):5--32, Oct. 2001.

Digital Library

[5]

T. Chen and C. Guestrin. XGBoost: a scalable tree boosting system. In Proc. of the ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD), pages 785--794, 2016.

Digital Library

[6]

T. Choudhury and A. Pentland. Sensing and modeling human networks using the sociometer. In Proceedings of the 7th IEEE International Symposium on Wearable Computers, ISWC '03, pages 216--, Washington, DC, USA, 2003. IEEE Computer Society.

Digital Library

[7]

M. Cristani, L. Bazzani, G. Paggetti, A. Fossati, D. Tosato, A. Del Bue, G. Menegaz, and V. Murino. Social interaction discovery by statistical analysis of f-formations. In Proceedings of the British Machine Vision Conference, pages 23.1--23.12. BMVA Press, 2011. http://dx.doi.org/10.5244/C.25.23.

[8]

J. Davis and M. Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning, pages 233--240. ACM, 2006.

Digital Library

[9]

N. Deng, Y. Tian, and C. Zhang. Support Vector Machines: Optimization Based Theory, Algorithms, and Extensions. Chapman & Hall/CRC, 1st edition, 2012.

[10]

E. T. Hall. The hidden dimension. Doubleday & Co, 1966.

[11]

S. A. Hoseinitabatabaei, A. Gluhak, and R. Tafazolli. udirect: A novel approach for pervasive observation of user direction with mobile phones. In Pervasive Computing and Communications (PerCom), 2011 IEEE International Conference on, pages 74--83, March 2011.

[12]

W. Huang, Y.-S. Kuo, P. Pannuto, and P. Dutta. Opo: A wearable sensor for capturing high-fidelity face-to-face interactions. In Proceedings of the 12th ACM Conference on Embedded Network Sensor Systems, SenSys '14, pages 61--75, New York, NY, USA, 2014. ACM.

Digital Library

[13]

H. Hung and B. Kröse. Detecting f-formations as dominant sets. In Proceedings of the 13th International Conference on Multimodal Interfaces, ICMI '11, pages 231--238, New York, NY, USA, 2011. ACM.

Digital Library

[14]

F. Ichikawa, J. Chipchase, and R. Grignani. Where's the phone? a study of mobile phone location in public spaces. In 2005 2nd Asia Pacific Conference on Mobile Technology, Applications and Systems, pages 1--8, Nov 2005.

[15]

K. Katevas, H. Haddadi, and L. Tokarchuk. SensingKit: Evaluating the sensor power consumption in ios devices. In Intelligent Environments (IE), 2016 12th International Conference on, pages 222--225. IEEE, 2016.

[16]

K. Katevas, H. Haddadi, L. Tokarchuk, and R. G. Clegg. Detecting group formations using iBeacon technology. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, UbiComp '16, pages 742--752, New York, NY, USA, 2016. ACM.

Digital Library

[17]

A. Kendon. Conducting interaction: Patterns of behavior in focused encounters, volume 7. CUP Archive, 1990.

[18]

R. Lambiotte, J.-C. Delvenne, and M. Barahona. Laplacian dynamics and multi-scale modular structure in networks. arXiv preprint arXiv:0812.1770, 1, 12 2008.

[19]

A. Matic, V. Osmani, A. Maxhuni, and O. Mayora. Multi-modal mobile sensing of social interactions. In Pervasive Computing Technologies for Healthcare (PervasiveHealth), 2012 6th International Conference on, pages 105--114, May 2012.

[20]

A. Montanari, S. Nawaz, C. Mascolo, and K. Sailer. A study of bluetooth low energy performance for human proximity detection in the workplace. In 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 90--99, March 2017.

[21]

A. Montanari, Z. Tian, E. Francu, B. Lucas, B. Jones, X. Zhou, and C. Mascolo. Measuring interaction proxemics with wearable light tags. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(1):25, 2018.

[22]

J. A. Nelder and R. J. Baker. Generalized linear models. Wiley Online Library, 1972.

[23]

D. O. Olguín and A. S. Pentland. Social sensors for automatic data collection. AMCIS 2008 Proceedings, page 171, 2008.

[24]

N. Palaghias, S. A. Hoseinitabatabaei, M. Nati, A. Gluhak, and K. Moessner. Accurate detection of real-world social interactions with smartphones. In Communications (ICC), 2015 IEEE International Conference on, pages 579--585, June 2015.

[25]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825--2830, 2011.

Digital Library

[26]

T. Saito and M. Rehmsmeier. The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3):e0118432, 2015.

[27]

P. Wittenburg, H. Brugman, A. Russel, A. Klassmann, and H. Sloetjes. Elan: a professional framework for multimodality research. In Proceedings of LREC, volume 2006, page 5th, 2006.

[28]

H. Zhang, W. Du, P. Zhou, M. Li, and P. Mohapatra. Dopenc: Acoustic-based encounter profiling using smartphones. In Proceedings of the 22Nd Annual International Conference on Mobile Computing and Networking, MobiCom '16, pages 294--307, New York, NY, USA, 2016. ACM.

Digital Library

Cited By

Madan ATipper DPalanisamy BAbdelhakim MKrishnamurthy PChamola V(2025)BLE-based sensors for privacy-enabled contagious disease monitoring with zero trust architectureAd Hoc Networks10.1016/j.adhoc.2024.103693167(103693)Online publication date: Feb-2025
https://doi.org/10.1016/j.adhoc.2024.103693
Tocchetti ABrusadelli SFugini M(2024)Detecting Social Interaction2024 32nd International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE)10.1109/WETICE64632.2024.00018(50-55)Online publication date: 26-Jun-2024
https://doi.org/10.1109/WETICE64632.2024.00018
Bezerra LKouzayha NElsawy HBader AAl-Naffouri T(2024)CSI-Based Proximity Estimation: Data-Driven and Model-Based ApproachesIEEE Open Journal of the Communications Society10.1109/OJCOMS.2023.33397215(97-111)Online publication date: 2024
https://doi.org/10.1109/OJCOMS.2023.3339721
Show More Cited By

Recommendations

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web
WWW '15 Companion: Proceedings of the 24th International Conference on World Wide Web

Within the scope of this PhD proposal, we set out to investigate two pivotal aspects that influence the effectiveness of crowdsourcing: (i) microtask design, and (ii) workers behavior. Leveraging the dynamics of tasks that are crowdsourced on the one ...
Stock analysts vs. the crowd

Exploration of the drivers of crowd wisdom and their impact on prediction quality.Operationalization of measures based on constructs of crowd wisdom theory.Suggestion of future research avenues for crowd wisdom research.Recommendation of aggregation ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SenSys-ML 2019: Proceedings of the 1st Workshop on Machine Learning on Edge in Sensor Systems

November 2019

47 pages

ISBN:9781450370110

DOI:10.1145/3362743

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

SenSys '19

Sponsor:

SenSys '19: The 17th ACM Conference on Embedded Networked Sensor Systems

November 10, 2019

NY, New York, USA

Acceptance Rates

SenSys-ML 2019 Paper Acceptance Rate 7 of 14 submissions, 50%;

Overall Acceptance Rate 7 of 14 submissions, 50%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
284
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Madan ATipper DPalanisamy BAbdelhakim MKrishnamurthy PChamola V(2025)BLE-based sensors for privacy-enabled contagious disease monitoring with zero trust architectureAd Hoc Networks10.1016/j.adhoc.2024.103693167(103693)Online publication date: Feb-2025
https://doi.org/10.1016/j.adhoc.2024.103693
Tocchetti ABrusadelli SFugini M(2024)Detecting Social Interaction2024 32nd International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE)10.1109/WETICE64632.2024.00018(50-55)Online publication date: 26-Jun-2024
https://doi.org/10.1109/WETICE64632.2024.00018
Bezerra LKouzayha NElsawy HBader AAl-Naffouri T(2024)CSI-Based Proximity Estimation: Data-Driven and Model-Based ApproachesIEEE Open Journal of the Communications Society10.1109/OJCOMS.2023.33397215(97-111)Online publication date: 2024
https://doi.org/10.1109/OJCOMS.2023.3339721
Beyan CVinciarelli ABue A(2023)Co-Located Human–Human Interaction Analysis Using Nonverbal Cues: A SurveyACM Computing Surveys10.1145/362651656:5(1-41)Online publication date: 25-Nov-2023
https://dl.acm.org/doi/10.1145/3626516
Wu FLai SSou S(2023)ViWise: Fusing Visual and Wireless Sensing Data for Trajectory Relationship RecognitionACM Transactions on Internet of Things10.1145/36144414:4(1-29)Online publication date: 22-Nov-2023
https://dl.acm.org/doi/10.1145/3614441
Liang DZhang AThomaz E(2023)Automated Face-To-Face Conversation Detection on a Commodity Smartwatch with Acoustic SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108827:3(1-29)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610882
Tan STax DHung H(2022)Conversation Group Detection With Spatio-Temporal ContextProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556611(170-180)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536221.3556611
Madan ATipper DPalanisamy BAbdelhakim MKrishnamurthy P(2022)BECT: Beacon-based Contact Tracing for Detecting Direct & Indirect Contacts2022 IEEE 19th International Conference on Mobile Ad Hoc and Smart Systems (MASS)10.1109/MASS56207.2022.00038(236-242)Online publication date: Oct-2022
https://doi.org/10.1109/MASS56207.2022.00038
Semenov OAgu EPahlavan KSu Z(2022)COVID-19 Social Distance Proximity Estimation Using Machine Learning Analyses of Smartphone Sensor DataIEEE Sensors Journal10.1109/JSEN.2022.316260522:10(9568-9579)Online publication date: 15-May-2022
https://doi.org/10.1109/JSEN.2022.3162605
Davila-Montero SDana-Le JBente GHall AMason A(2021)Review and Challenges of Technologies for Real-Time Human Behavior MonitoringIEEE Transactions on Biomedical Circuits and Systems10.1109/TBCAS.2021.306061715:1(2-28)Online publication date: Feb-2021
https://doi.org/10.1109/TBCAS.2021.3060617
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten