research-article

Top in the Lab, Flop in the Field?: Evaluation of a Sensor-based Travel Activity Classifier with the SHL Dataset

Authors:

UbiComp '18: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers

Pages 1479 - 1487

https://doi.org/10.1145/3267305.3267514

Published: 08 October 2018 Publication History

Get Access

Abstract

We present a solution to the Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge (team "S304"). Our experiments reveal two potential pitfalls in the evaluation of activity recognition algorithms: 1) unnoticed overfitting due to autocorrelation (i.e. dependencies between temporally close samples), and 2) the accuracy/generality trade-off due to idealized conditions and lack of variation in the data. We show that evaluation with a random training/test split suggests highly accurate recognition of eight different travel activities with an average F1 score of 96% for single-participant/fixed-position data, whereas with proper backtesting the F1 score drops to 84%, for data of different participants in the SHL Dataset to 61%, and for different carrying positions to 54%. Our experiments demonstrate that results achieved 'in-the-lab' can easily become subject to an upward bias and cannot always serve as reliable indicators for the future performance 'in-the-field', where generality and robustness are essential.

References

[1]

Wendy Bohte, Kees Maat, and Wilko Quak. 2008. A method for deriving trip destinations and modes for GPS-based travel surveys. Research in Urbanism Series 1, 1 (2008), 127--143.

Google Scholar

[2]

Mathias Ciliberto, Francisco Javier Ordoñez Morales, Hristijan Gjoreski, Daniel Roggen, Sami Mekki, and Stefan Valentin. 2017. High reliability Android application for multidevice multimodal mobile data acquisition and annotation. In Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems. ACM, 62.

Digital Library

Google Scholar

[3]

Arindam Ghosh and Giuseppe Riccardi. 2014. Recognizing human activities from smartphone sensor signals. In Proceedings of the 22nd ACM international conference on Multimedia. ACM, 865--868.

Digital Library

Google Scholar

[4]

Hristijan Gjoreski, Mathias Ciliberto, Lin Wang, Francisco Javier Ordonez Morales, Sami Mekki, Stefan Valentin, and Daniel Roggen. 2018. The University of Sussex-Huawei Locomotion and Transportation Dataset for Multimodal Analytics with Mobile Devices. IEEE Access (2018).

Google Scholar

[5]

Samuli Hemminki, Petteri Nurmi, and Sasu Tarkoma. 2013. Accelerometer-based transportation mode detection on smartphones. In Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems. ACM, 13.

Digital Library

Google Scholar

[6]

Xiaoyuan Liang and Guiling Wang. 2017. A convolutional neural network for transportation mode detection based on smartphone platform. In Mobile Ad Hoc and Sensor Systems (MASS), 2017 IEEE 14th International Conference on. IEEE, 338--342.

Crossref

Google Scholar

[7]

Vincenzo Manzoni, Diego Maniloff, Kristian Kloeckl, and Carlo Ratti. 2010. Transportation mode identification and real-time CO2 emission estimation using smartphones. SENSEable City Lab, Massachusetts Institute of Technology, nd (2010).

Google Scholar

[8]

David Mizell. 2003. Using gravity to estimate accelerometer orientation. In null. IEEE, 252.

Digital Library

Google Scholar

[9]

Nelson Morgan and Herve A Bourlard. 1995. Neural networks for statistical recognition of continuous speech. Proc. IEEE 83, 5 (1995), 742--772.

Crossref

Google Scholar

[10]

Charissa Ann Ronao and Sung-Bae Cho. 2017. Recognizing human activities from smartphone sensors using hierarchical continuous hidden Markov models. International Journal of Distributed Sensor Networks 13, 1 (2017), 1550147716683687.

Crossref

Google Scholar

[11]

Pekka Siirtola and Juha Röning. 2012. Recognizing human activities user-independently on smartphones based on accelerometer data. IJIMAI 1, 5 (2012), 38--45.

Crossref

Google Scholar

[12]

Leon Stenneth, Ouri Wolfson, Philip S Yu, and Bo Xu. 2011. Transportation mode detection using mobile phones and GIS information. In Proceedings of the 19th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. ACM, 54--63.

Digital Library

Google Scholar

[13]

Peter Stopher, Camden FitzGerald, and Min Xu. 2007. Assessing the accuracy of the Sydney Household Travel Survey with GPS. Transportation 34, 6 (2007), 723--741.

Crossref

Google Scholar

[14]

Lin Wang, Hristijan Gjoreski, Kazuya Murao, Tsuyoshi Okita, and Daniel Roggen. 2018. Summary of the Sussex- Huawei Locomotion-Transportation Recognition Challenge. In Proceedings of the 6th International Workshop on Human Activity Sensing Corpus and Applications (HASCA2018). Singapore.

Digital Library

Google Scholar

Cited By

View all

Ge JXu GLu JXu XLi LMeng X(2024)SensorNet: An Adaptive Attention Convolutional Neural Network for Sensor Feature LearningSensors10.3390/s2411327424:11(3274)Online publication date: 21-May-2024
https://doi.org/10.3390/s24113274
Rehman SAli AKhan AOkpala C(2024)Human Activity Recognition: A Comparative Study of Validation Methods and Impact of Feature Extraction in Wearable SensorsAlgorithms10.3390/a1712055617:12(556)Online publication date: 5-Dec-2024
https://doi.org/10.3390/a17120556
Van Der Donckt JVan Der Donckt JVan Hoecke SKostakos VKay JHoang T(2024)Magnitude and Rotation Invariant Detection of Transportation Modes with Missing Data ModalitiesCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678463(597-602)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678463
Show More Cited By

Index Terms

Top in the Lab, Flop in the Field?: Evaluation of a Sensor-based Travel Activity Classifier with the SHL Dataset
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Signal processing systems
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Total Variation Models for Variable Lighting Face Recognition

In this paper, we present the logarithmic total variation (LTV) model for face recognition under varying illumination, including natural lighting conditions, where we rarely know the strength, direction, or number of light sources. The proposed LTV ...
Wireless non-invasive motion tracking of functional behavior
Abstract
The prevalence of a sedentary lifestyle is a major contributor to many chronic afflictions in modern society. Objective study and monitoring to gain an accurate understanding of situated sedentary behavior, for example when at home, ...
Nurse care activity recognition challenge: a comparative verification of multiple preprocessing approaches
UbiComp/ISWC '20 Adjunct: Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers

Although activity recognition has been studied considerably for the last two decades, it is still not so easy to handle complicated activity classes in a specific domain. The 2nd Nurse Care Activity Recognition Challenge Using Lab and Field Data aims to ...

Comments

Information & Contributors

Information

Published In

UbiComp '18: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers

October 2018

1881 pages

ISBN:9781450359665

DOI:10.1145/3267305

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGSPATIAL: ACM Special Interest Group on Spatial Information

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

UbiComp '18

Sponsor:

SIGMOBILE
University of Florida
SIGCHI

UbiComp '18: The 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing

October 8 - 12, 2018

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
173
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)2

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ge JXu GLu JXu XLi LMeng X(2024)SensorNet: An Adaptive Attention Convolutional Neural Network for Sensor Feature LearningSensors10.3390/s2411327424:11(3274)Online publication date: 21-May-2024
https://doi.org/10.3390/s24113274
Rehman SAli AKhan AOkpala C(2024)Human Activity Recognition: A Comparative Study of Validation Methods and Impact of Feature Extraction in Wearable SensorsAlgorithms10.3390/a1712055617:12(556)Online publication date: 5-Dec-2024
https://doi.org/10.3390/a17120556
Van Der Donckt JVan Der Donckt JVan Hoecke SKostakos VKay JHoang T(2024)Magnitude and Rotation Invariant Detection of Transportation Modes with Missing Data ModalitiesCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678463(597-602)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678463
Siargkas CPapapanagiotou VDelopoulos A(2024)Transportation Mode Recognition Based on Low-Rate Acceleration and Location Signals With an Attention-Based Multiple-Instance Learning NetworkIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2024.338783425:10(14376-14388)Online publication date: Oct-2024
https://doi.org/10.1109/TITS.2024.3387834
Demagh LGarda PGilbert CHachicha K(2024)SensOL: Memory-Efficient Online Learning for Tiny MCUs2024 IEEE SENSORS10.1109/SENSORS60989.2024.10784905(1-4)Online publication date: 20-Oct-2024
https://doi.org/10.1109/SENSORS60989.2024.10784905
Tang QCheng H(2024)Feature pyramid biLSTM: Using smartphone sensors for transportation mode detectionTransportation Research Interdisciplinary Perspectives10.1016/j.trip.2024.10118126(101181)Online publication date: Jul-2024
https://doi.org/10.1016/j.trip.2024.101181
Graser AJalali ALampert JWeißenfeld AJanowicz K(2024)MobilityDL: a review of deep learning from trajectory dataGeoInformatica10.1007/s10707-024-00518-8Online publication date: 28-May-2024
https://doi.org/10.1007/s10707-024-00518-8
Demagh LGarda PGilbert CHachicha K(2023)TinyMM: Multimodal-Multitask Machine Learning on Low-Power MCUs for Smart Glasses2023 IEEE SENSORS10.1109/SENSORS56945.2023.10325296(1-4)Online publication date: 29-Oct-2023
https://doi.org/10.1109/SENSORS56945.2023.10325296
Demagh LGarda PGilbert CHachicha K(2023)Transport mode recognition for smart eyewear using multimodal audio and accelerometer data2023 30th IEEE International Conference on Electronics, Circuits and Systems (ICECS)10.1109/ICECS58634.2023.10382908(1-4)Online publication date: 4-Dec-2023
https://doi.org/10.1109/ICECS58634.2023.10382908
Bragança HColonna JOliveira HSouto E(2022)How Validation Methodology Influences Human Activity Recognition Mobile SystemsSensors10.3390/s2206236022:6(2360)Online publication date: 18-Mar-2022
https://doi.org/10.3390/s22062360
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Total Variation Models for Variable Lighting Face Recognition

Wireless non-invasive motion tracking of functional behavior

Nurse care activity recognition challenge: a comparative verification of multiple preprocessing approaches

Comments

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Total Variation Models for Variable Lighting Face Recognition

Wireless non-invasive motion tracking of functional behavior

Nurse care activity recognition challenge: a comparative verification of multiple preprocessing approaches

Comments

Information

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations