research-article

Public Access

CrowdNAS: A Crowd-guided Neural Architecture Searching Approach to Disaster Damage Assessment

Authors:

Dong WangAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 6, Issue CSCW2

Article No.: 288, Pages 1 - 29

https://doi.org/10.1145/3555179

Published: 11 November 2022 Publication History

Abstract

Disaster damage assessment (DDA) has emerged as an important application in disaster response and management, which aims to assess the damage severity of an affected area by leveraging AI (e.g., deep learning) techniques to examine the imagery data posted on social media during a disaster event. In this paper, we focus on a crowd-guided neural architecture searching (NAS) problem in DDA applications. Our goal is to leverage human intelligence from crowdsourcing systems to guide the discovery of the optimal neural network architecture in the design space to achieve the desirable damage assessment performance. Our work is motivated by the limitation that the deep neural network architectures in current DDA solutions are mainly designed by AI experts, which is known to be both time-consuming and error-prone. Two critical technical challenges exist in solving our problem: i) it is challenging to design a manageable NAS space for crowd-based solutions; ii) it is non-trivial to transfer the imperfect crowd knowledge to effective decisions in identifying the optimal neural network architecture of a DDA application. To address the above challenges, we develop CrowdNAS, a crowd-guided NAS framework that develops novel techniques inspired by AI, crowdsourcing, and estimation theory to address the NAS problem. The evaluation results from two real-world DDA applications show that CrowdNAS consistently outperforms the state-of-the-art AI-only, crowd-AI, and NAS baselines by achieving the highest classification accuracy in the damage assessment while maintaining a low computational cost under various evaluation scenarios.

References

[1]

Firoj Alam, Ferda Ofli, Muhammad Imran, and Michael Aupetit. 2018. A twitter tale of three hurricanes: Harvey, irma, and maria. arXiv preprint arXiv:1805.05144 (2018).

[2]

Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, Vol. 34, 4 (2008).

[3]

Han Cai, Ligeng Zhu, and Song Han. 2018. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. In International Conference on Learning Representations.

[4]

Davide Chicco and Giuseppe Jurman. 2020. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC genomics, Vol. 21, 1 (2020), 6.

[5]

António Correia, Daniel Schneider, Hugo Paredes, and Benjamim Fonseca. 2018. SciCrowd: Towards a hybrid, crowd-computing system for supporting research groups in academic settings. In International Conference on Collaboration and Technology. Springer, 34--41.

[6]

Joy Dutta, Firoj Gazi, Sarbani Roy, and Chandreyee Chowdhury. 2016. AirSense: Opportunistic crowd-sensing based air quality monitoring system for smart city. In 2016 IEEE SENSORS. IEEE, 1--3.

[7]

Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. 2019. Neural architecture search: A survey. J. Mach. Learn. Res., Vol. 20, 55 (2019), 1--21.

[8]

Najmeh Fayyazifar. 2020. An Accurate CNN Architecture For Atrial Fibrillation Detection Using Neural Architecture Search. In 2020 28th European Signal Processing Conference (EUSIPCO). IEEE, 1135--1139.

[9]

Hector Garcia-Molina, Manas Joglekar, Adam Marcus, Aditya Parameswaran, and Vasilis Verroios. 2016. Challenges in data crowdsourcing. IEEE Transactions on Knowledge and Data Engineering, Vol. 28, 4 (2016), 901--911.

Digital Library

[10]

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely Connected Convolutional Networks. In CVPR, Vol. 1. 3.

[11]

Nguyen Quoc Viet Hung, Nguyen Thanh Tam, Lam Ngoc Tran, and Karl Aberer. 2013. An evaluation of aggregation techniques in crowdsourcing. In International Conference on Web Information Systems Engineering. Springer, 1--15.

[12]

Julian Jarrett, Iman Saleh, M Brian Blake, Rohan Malcolm, Sean Thorpe, and Tyrone Grandison. 2014. Combining human and machine computing elements for analysis via crowdsourcing. In 10th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing. IEEE, 312--321.

[13]

Giuseppe Jurman, Samantha Riccadonna, and Cesare Furlanello. 2012. A comparison of MCC and CEN error measures in multi-class prediction. PloS one, Vol. 7, 8 (2012), e41882.

[14]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[15]

Ziyi Kou, Lanyu Shang, Yang Zhang, and Dong Wang. 2022. HC-COVID: A Hierarchical Crowdsource Knowledge Graph Approach to Explainable COVID-19 Misinformation Detection. Proceedings of the ACM on Human-Computer Interaction, Vol. 6, GROUP (2022), 1--25.

Digital Library

[16]

Ziyi Kou, Daniel Zhang, Lanyu Shang, and Dong Wang. 2021. What and Why Towards Duo Explainable Fauxtography Detection under Constrained Supervision. IEEE Transactions on Big Data (2021).

[17]

Yury Kryvasheyeu, Haohui Chen, Nick Obradovich, Esteban Moro, Pascal Van Hentenryck, James Fowler, and Manuel Cebrian. 2016. Rapid assessment of disaster damage using social media activity. Science advances, Vol. 2, 3 (2016), e1500779.

[18]

Pakhee Kumar, Ferda Ofli, Muhammad Imran, and Carlos Castillo. 2020. Detection of Disaster-Affected Cultural Heritage Sites from Social Media Images Using Deep Learning Techniques. Journal on Computing and Cultural Heritage (JOCCH), Vol. 13, 3 (2020), 1--31.

Digital Library

[19]

Florian Laws, Christian Scheible, and Hinrich Schütze. 2011. Active learning with amazon mechanical turk. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. 1546--1556.

[20]

Xukun Li, Doina Caragea, Cornelia Caragea, Muhammad Imran, and Ferda Ofli. [n.,d.]. Identifying Disaster Damage Images Using a Domain Adaptation Approach.

[21]

Xukun Li, Doina Caragea, Huaiyu Zhang, and Muhammad Imran. 2018. Localizing and quantifying damage in social media images. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 194--201.

[22]

Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei. 2019. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 82--92.

[23]

Chenxi Liu, Piotr Dollár, Kaiming He, Ross Girshick, Alan Yuille, and Saining Xie. 2020. Are labels necessary for neural architecture search?. In European Conference on Computer Vision. Springer, 798--813.

Digital Library

[24]

Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. DARTS: Differentiable Architecture Search. In International Conference on Learning Representations.

[25]

Travis Mandel, Jahnu Best, Randall H Tanaka, Hiram Temple, Chansen Haili, Sebastian J Carter, Kayla Schlechtinger, and Roy Szeto. 2020. Using the crowd to prevent harmful AI behavior. Proceedings of the ACM on Human-Computer Interaction, Vol. 4, CSCW2 (2020), 1--25.

Digital Library

[26]

Danielle McDuffie. 2019. Using Amazon's mechanical Turk: benefits, drawbacks, and suggestions. APS Observer, Vol. 32, 2 (2019).

[27]

Yelena Mejova, Ingmar Weber, and Luis Fernandez-Luque. 2018. Online health monitoring using Facebook advertisement audience estimates in the United States: evaluation study. JMIR public health and surveillance, Vol. 4, 1 (2018), e30.

[28]

Hussein Mouzannar, Yara Rizk, and Mariette Awad. 2018. Damage Identification in Social Media Posts using Multimodal Deep Learning. In ISCRAM.

[29]

Dat T Nguyen, Ferda Ofli, Muhammad Imran, and Prasenjit Mitra. 2017. Damage assessment from social media imagery data during disasters. In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017. 569--576.

Digital Library

[30]

Ekin Ozer and Maria Q Feng. 2019. Structural reliability estimation with participatory sensing and mobile cyber-physical structural health monitoring systems. Applied Sciences, Vol. 9, 14 (2019), 2840.

[31]

Bratislav Predic and Dragan Stojanovic. 2015. Enhancing driver situational awareness through crowd intelligence. Expert Systems with Applications, Vol. 42, 11 (2015), 4892--4909.

Digital Library

[32]

Frerk Saxen, Philipp Werner, Sebastian Handrich, Ehsan Othman, Laslo Dinges, and Ayoub Al-Hamadi. 2019. Face Attribute Detection with MobileNetV2 and NasNet-Mobile. In 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA). IEEE, 176--180.

[33]

Ozan Sener and Silvio Savarese. 2018. Active Learning for Convolutional Neural Networks: A Core-Set Approach. In International Conference on Learning Representations.

[34]

Dong-Woo Seo and Soo-Yong Shin. 2017. Methods using social media and search queries to predict infectious disease outbreaks. Healthcare informatics research, Vol. 23, 4 (2017), 343.

[35]

Lanyu Shang, Ziyi Kou, Yang Zhang, and Dong Wang. 2021. A Multimodal Misinformation Detector for COVID-19 Short Videos on TikTok. In 2021 IEEE International Conference on Big Data (Big Data). IEEE, 899--908.

[36]

Lanyu Shang, Yang Zhang, Christina Youn, and Dong Wang. 2022. SAT-Geo: A social sensing based content-only approach to geolocating abnormal traffic events using syntax-based probabilistic learning. Information Processing & Management, Vol. 59, 2 (2022), 102807.

Digital Library

[37]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[38]

Max Sklar, Blake Shaw, and Andrew Hogue. 2012. Recommending interesting events in real-time with foursquare check-ins. In Proceedings of the sixth ACM conference on Recommender systems. 311--312.

Digital Library

[39]

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. 2019. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2820--2828.

[40]

Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. 2020. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12965--12974.

[41]

Cheng Wang, Delei Chen, Lin Hao, Xuebo Liu, Yu Zeng, Jianwei Chen, and Guokai Zhang. 2019a. Pulmonary Image Classification Based on Inception-v3 Transfer Learning Model. IEEE Access, Vol. 7 (2019), 146533--146541.

[42]

Dong Wang, Lance Kaplan, Tarek Abdelzaher, and Charu C Aggarwal. 2013. On credibility estimation tradeoffs in assured social sensing. IEEE Journal on Selected Areas in Communications, Vol. 31, 6 (2013), 1026--1037.

[43]

Dong Wang, Lance Kaplan, and Tarek F Abdelzaher. 2014. Maximum likelihood analysis of conflicting observations in social sensing. ACM Transactions on Sensor Networks (ToSN), Vol. 10, 2 (2014), 1--27.

Digital Library

[44]

Dong Wang, Boleslaw K Szymanski, Tarek Abdelzaher, Heng Ji, and Lance Kaplan. 2019b. The age of social sensing. Computer, Vol. 52, 1 (2019), 36--45.

Digital Library

[45]

Jie Yang, Judith Redi, Gianluca Demartini, and Alessandro Bozzon. 2016. Modeling task complexity in crowdsourcing. In Fourth AAAI Conference on human computation and crowdsourcing.

[46]

Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. 2019. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning. PMLR, 7105--7114.

[47]

Daniel Zhang, Yang Zhang, Qi Li, Thomas Plummer, and Dong Wang. 2019. Crowdlearn: A crowd-ai hybrid system for deep learning-based damage assessment applications. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE, 1221--1232.

[48]

Daniel Yue Zhang, Yifeng Huang, Yang Zhang, and Dong Wang. 2020a. Crowd-assisted disaster scene assessment with human-ai interactive attention. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 2717--2724.

[49]

Yang Zhang, Ruohan Zong, Ziyi Kou, Lanyu Shang, and Dong Wang. 2021a. CollabLearn: An Uncertainty-Aware Crowd-AI Collaboration System for Cultural Heritage Damage Assessment. IEEE Transactions on Computational Social Systems (2021).

[50]

Yang Zhang, Ruohan Zong, Ziyi Kou, Lanyu Shang, and Dong Wang. 2021b. On streaming disaster damage assessment in social sensing: A crowd-driven dynamic neural architecture searching approach. Knowledge-Based Systems (2021), 107984.

[51]

Yang Zhang, Ruohan Zong, and Dong Wang. 2020b. A Hybrid Transfer Learning Approach to Migratable Disaster Assessment in Social Media Sensing. In 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 131--138.

Digital Library

[52]

Yiyang Zhao, Linnan Wang, Yuandong Tian, Rodrigo Fonseca, and Tian Guo. 2020. Few-shot neural architecture search. arXiv preprint arXiv:2006.06863 (2020).

[53]

Barret Zoph and Quoc V Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016).

[54]

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. 2018. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8697--8710.

Cited By

Xu ZZhang JGreenberg JFrumkin MJaveed SZhang JBenedict BBotterbush KRodebaugh TRay WLu C(2024)Predicting Multi-dimensional Surgical Outcomes with Multi-modal Mobile SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596288:2(1-30)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659628
Li WGao RXiong JZhou JWang LMao XYi EZhang D(2024)WiFi-CSI Difference ParadigmProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596088:2(1-29)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659608
Jeong DHan K(2024)PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor DataProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595948:2(1-24)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659594
Show More Cited By

Index Terms

CrowdNAS: A Crowd-guided Neural Architecture Searching Approach to Disaster Damage Assessment
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing systems and tools

Recommendations

On streaming disaster damage assessment in social sensing: A crowd-driven dynamic neural architecture searching approach
Abstract
Motivated by the recent advances in Internet and communication techniques and the proliferation of online social media, social sensing has emerged as a new sensing paradigm to obtain timely observations of the physical world from “...
Promoting coordination for disaster relief: from crowdsourcing to coordination
SBP'11: Proceedings of the 4th international conference on Social computing, behavioral-cultural modeling and prediction

The efficiency at which governments and nongovernmental organizations (NGOs) are able to respond to a crisis and provide relief to victims has gained increased attention. This emphasis coincides with significant events such as tsunamis, hurricanes, ...
CNN Based Approach for Post Disaster Damage Assessment
ICDCN '20: Proceedings of the 21st International Conference on Distributed Computing and Networking

After any disaster, the Government rehabilitates the victims based on the severity of the damage caused to their properties. Since a huge number of rehabilitation claims flow in after the disaster, it takes up a lot of manual labor in inspecting and ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 6, Issue CSCW2

CSCW

November 2022

8205 pages

EISSN:2573-0142

DOI:10.1145/3571154

Editor:
Jeff Nichols
Google

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2022

Published in PACMHCI Volume 6, Issue CSCW2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
232
Total Downloads

Downloads (Last 12 months)129
Downloads (Last 6 weeks)21

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xu ZZhang JGreenberg JFrumkin MJaveed SZhang JBenedict BBotterbush KRodebaugh TRay WLu C(2024)Predicting Multi-dimensional Surgical Outcomes with Multi-modal Mobile SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596288:2(1-30)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659628
Li WGao RXiong JZhou JWang LMao XYi EZhang D(2024)WiFi-CSI Difference ParadigmProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596088:2(1-29)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659608
Jeong DHan K(2024)PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor DataProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595948:2(1-24)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659594
Zhou YZhao HHuang YRöddiger TKurnaz MRiedel TBeigl M(2024)AutoAugHAR: Automated Data Augmentation for Sensor-based Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595898:2(1-27)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659589
Yang LAmin OShihada B(2024)Intelligent Wearable Systems: Opportunities and Challenges in Health and SportsACM Computing Surveys10.1145/364846956:7(1-42)Online publication date: 9-Apr-2024
https://dl.acm.org/doi/10.1145/3648469
Sheng BHan RXiao FGuo ZGui L(2024)MetaFormerProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435508:1(1-27)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643550
Prinster GSmith CTan CKeegan B(2024)Community Archetypes: An Empirical Framework for Guiding Research Methodologies to Reflect User Experiences of Sense of Virtual Community on RedditProceedings of the ACM on Human-Computer Interaction10.1145/36373108:CSCW1(1-33)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637310
Gregersen SAguirre AHaselwarter PTassarotti JBirkedal L(2024)Asynchronous Probabilistic Couplings in Higher-Order Separation LogicProceedings of the ACM on Programming Languages10.1145/36328688:POPL(753-784)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3632868
Cousot P(2024)Calculational Design of [In]Correctness Transformational Program Logics by Abstract InterpretationProceedings of the ACM on Programming Languages10.1145/36328498:POPL(175-208)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3632849
Ge WMou GAgu ELee K(2024)Deep Heterogeneous Contrastive Hyper-Graph Learning for In-the-Wild Context-Aware Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314447:4(1-23)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631444
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents