research-article

Public Access

GroundTruth: Augmenting Expert Image Geolocation with Crowdsourcing and Shared Representations

Authors:

Sukrit Venkatagiri,

Jacob Thebault-Spieker,

John Purviance,

Rifat Sabbir Mansur,

Kurt LutherAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 3, Issue CSCW

Article No.: 107, Pages 1 - 30

https://doi.org/10.1145/3359209

Published: 07 November 2019 Publication History

Abstract

Expert investigators bring advanced skills and deep experience to analyze visual evidence, but they face limits on their time and attention. In contrast, crowds of novices can be highly scalable and parallelizable, but lack expertise. In this paper, we introduce the concept of shared representations for crowd--augmented expert work, focusing on the complex sensemaking task of image geolocation performed by professional journalists and human rights investigators. We built GroundTruth, an online system that uses three shared representations-a diagram, grid, and heatmap-to allow experts to work with crowds in real time to geolocate images. Our mixed-methods evaluation with 11 experts and 567 crowd workers found that GroundTruth helped experts geolocate images, and revealed challenges and success strategies for expert-crowd interaction. We also discuss designing shared representations for visual search, sensemaking, and beyond.

Supplementary Material

ZIP File (cscw107aux.zip)

This contains the subtitles file (GTCSCW_captions.sbv) for the video figure, as well as the video figure description (GT_CSCW_video_description).

Download
4.10 KB

MP4 File (cscw107.mp4)

Supplemental video

Download
92.37 MB

References

[1]

Tanja Aitamurto. 2015. Motivation Factors in Crowdsourced Journalism: Social Impact, Social Change, and Peer Learning. International Journal of Communication, Vol. 9, 0 (2015). https://ijoc.org/index.php/ijoc/article/view/3481

[2]

Salvatore Andolina, Hendrik Schneider, Joel Chan, Khalil Klouche, Giulio Jacucci, and Steven Dow. 2017. Crowdboard: Augmenting In-Person Idea Generation with Real-Time Crowds. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (C & C '17). ACM, New York, NY, USA, 106--118. https://doi.org/10.1145/3059454.3059477

Digital Library

[3]

Paul André, Aniket Kittur, and Steven P. Dow. 2014. Crowd Synthesis: Extracting Categories and Clusters from Complex Data. In Proceedings of CSCW 2014 .

[4]

Trushar Barot. 2014. Verifying Images. In Verification Handbook: A Definitive Guide to Verifying Digital Content for Emergency Coverage. http://verificationhandbook.com/book/chapter4.php

[5]

Michael S Bernstein, Joel Brandt, Robert C Miller, and David R Karger. 2011. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 33--42.

Digital Library

[6]

Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. 2010. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 313--322.

Digital Library

[7]

Jonathan Bragg, Mausam, and Daniel S. Weld. 2018. Sprout: Crowd-Powered Task Design for Crowdsourcing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 165--176. https://doi.org/10.1145/3242587.3242598

Digital Library

[8]

Petter Bae Brandtzaeg, Marika Lü ders, Jochen Spangenberg, Linda Rath-Wiggins, and Asbjørn Følstad. 2016. Emerging Journalistic Verification Practices Concerning Social Media. Journalism Practice, Vol. 10, 3 (2016), 323--342. https://doi.org/10.1080/17512786.2015.1020331 https://doi.org/10.1007/s13347-016-0216--4

[9]

Sukrit Venkatagiri and Amy X Zhang. 2018. Response to “Heuristics for the Online Curator”, Brian G Southwell and Vanessa Boudewyns (Eds.). Curbing the Spread of Misinformation: Insights, Innovations, and Interpretations From the Misinformation Solutions Forum, 9--10. https://doi.org/10.3768/rtipress.2018.cp.0008.1812

[10]

Nam Vo, Nathan Jacobs, and James Hays. 2017. Revisiting im2gps in the deep learning era. In Proceedings of the IEEE International Conference on Computer Vision. 2621--2630.

[11]

Nai-Ching Wang, David Hicks, and Kurt Luther. 2018. Exploring Trade-Offs Between Learning and Productivity in Crowdsourced History. Proc. ACM Hum.-Comput. Interact., Vol. 2, CSCW (Nov. 2018), 178:1--178:24. https://doi.org/10.1145/3274447

Digital Library

[12]

Tobias Weyand, Ilya Kostrikov, and James Philbin. 2016. Planet-photo geolocation with convolutional neural networks. In European Conference on Computer Vision. Springer, 37--55.

[13]

Joanne I White, Leysia Palen, and Kenneth M Anderson. 2014. Digital mobilization in disaster response: the work & self-organization of on-line pet advocates in response to hurricane sandy. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 866--876.

Digital Library

[14]

Mark E. Whiting, Dilrukshi Gamage, Snehalkumar (Neil) S. Gaikwad, Aaron Gilbee, Shirish Goyal, Alipta Ballav, Dinesh Majeti, Nalin Chhibber, Angela Richmond-Fuller, Freddie Vargus, Tejas Seshadri Sarma, Varshine Chandrakanthan, Teogenes Moura, Mohamed Hashim Salih, Gabriel Bayomi Tinoco Kalejaiye, Adam Ginzberg, Catherine A. Mullings, Yoni Dayan, Kristy Milland, Henrique Orefice, Jeff Regino, Sayna Parsi, Kunz Mainali, Vibhor Sehgal, Sekandar Matin, Akshansh Sinha, Rajan Vaish, and Michael S. Bernstein. 2017. Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 1902--1913. https://doi.org/10.1145/2998181.2998234 event-place: Portland, Oregon, USA.

[15]

Wesley Willett, Shiry Ginosar, Avital Steinitz, Björn Hartmann, and Maneesh Agrawala. 2013. Identifying Redundancy and Exposing Provenance in Crowdsourced Data Analysis. (2013).

[16]

Helen Wollan. 2004. Incorporating heuristically generated search patterns in search and rescue. University of Edinburgh (2004).

[17]

Anna Wu, Gregorio Convertino, Craig Ganoe, John M. Carroll, and Xiaolong (Luke) Zhang. 2013. Supporting collaborative sense-making in emergency management through geo-visualization., Vol. 71, 1 (2013), 4--23. https://doi.org/10.1016/j.ijhcs.2012.07.007

Digital Library

[18]

Elizabeth Yardley, Adam George Thomas Lynes, David Wilson, and Emma Kelly. 2018. What's the deal with `websleuthing'? News media representations of amateur detectives in networked spaces. Crime, Media, Culture, Vol. 14, 1 (2018), 81--109. https://doi.org/10.1177/1741659016674045 showeprinthttps://doi.org/10.1177/1741659016674045

[19]

Ling Yu, Sheryl Ball, Christine Blinn, Klaus Moeltner, Seth Peery, Valerie Thomas, Randolph Wynne, Ling Yu, Sheryl B. Ball, Christine E. Blinn, Klaus Moeltner, Seth Peery, Valerie A. Thomas, and Randolph H. Wynne. 2015. Cloud-Sourcing: Using an Online Labor Force to Detect Clouds and Cloud Shadows in Landsat Images, Vol. 7, 3 (2015), 2334--2351. https://doi.org/10.3390/rs70302334

Cited By

Mukhopadhyay AVenkatagiri SLuther K(2024)OSINT Research Studios: A Flexible Crowdsourcing Framework to Scale Up Open Source Intelligence InvestigationsProceedings of the ACM on Human-Computer Interaction10.1145/36373828:CSCW1(1-38)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637382
Wazzan AMacNeil SSouvenir R(2024)Comparing Traditional and LLM-based Search for Image GeolocationProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638305(291-302)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638305
Jahanbakhsh FKarger D(2024)A Browser Extension for in-place Signaling and Assessment of MisinformationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642473(1-21)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642473
Show More Cited By

Index Terms

GroundTruth: Augmenting Expert Image Geolocation with Crowdsourcing and Shared Representations
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing design and evaluation methods
    2. Collaborative and social computing systems and tools

Recommendations

CrowdIA: Solving Mysteries with Crowdsourced Sensemaking

The increasing volume of text data is challenging the cognitive capabilities of expert analysts. Machine learning and crowdsourcing present new opportunities for large-scale sensemaking, but we must overcome the challenge of modeling the overall process ...
Novice and Expert Sensemaking of Crowdsourced Design Feedback

Online feedback exchange (OFE) systems are an increasingly popular way to test concepts with millions of target users before going to market. Yet, we know little about how designers make sense of this abundant feedback. This empirical study investigates ...
Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria
CSCW

Misinformation about critical issues such as climate change and vaccine safety is oftentimes amplified on online social and search platforms. The crowdsourcing of content credibility assessment by laypeople has been proposed as one strategy to combat ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 3, Issue CSCW

November 2019

5026 pages

EISSN:2573-0142

DOI:10.1145/3371885

Editors:
Airi Lampinen
Stockholm University, Sweden
,
Darren Gergle
Northwestern University, USA
,
David A. Shamma
FXPAL, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2019

Published in PACMHCI Volume 3, Issue CSCW

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
920
Total Downloads

Downloads (Last 12 months)203
Downloads (Last 6 weeks)25

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mukhopadhyay AVenkatagiri SLuther K(2024)OSINT Research Studios: A Flexible Crowdsourcing Framework to Scale Up Open Source Intelligence InvestigationsProceedings of the ACM on Human-Computer Interaction10.1145/36373828:CSCW1(1-38)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637382
Wazzan AMacNeil SSouvenir R(2024)Comparing Traditional and LLM-based Search for Image GeolocationProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638305(291-302)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638305
Jahanbakhsh FKarger D(2024)A Browser Extension for in-place Signaling and Assessment of MisinformationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642473(1-21)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642473
Ballester RLabeyrie YMulayim MFernandez-Marquez JCerquides J(2024)Crowdsourced geolocation: Detailed exploration of mathematical and computational modeling approachesCognitive Systems Research10.1016/j.cogsys.2024.10126688(101266)Online publication date: Dec-2024
https://doi.org/10.1016/j.cogsys.2024.101266
Correia AGrover ASchneider DPimentel AChaves Rde Almeida MFonseca B(2023)Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine InteractionApplied Sciences10.3390/app1304219813:4(2198)Online publication date: 8-Feb-2023
https://doi.org/10.3390/app13042198
Heuer HGlassman E(2023)Accessible Text Tools for People with Cognitive Impairments and Non-Native Readers: Challenges and OpportunitiesProceedings of Mensch und Computer 202310.1145/3603555.3603569(250-266)Online publication date: 3-Sep-2023
https://dl.acm.org/doi/10.1145/3603555.3603569
Thebault-Spieker JVenkatagiri SMine NLuther K(2023)Diverse Perspectives Can Mitigate Political Bias in Crowdsourced Content ModerationProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594080(1280-1291)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594080
Venkatagiri S(2023)Supporting High-Stakes Investigations with Expert-Led CrowdsourcingCompanion Proceedings of the 2023 ACM International Conference on Supporting Group Work10.1145/3565967.3571764(79-81)Online publication date: 8-Jan-2023
https://dl.acm.org/doi/10.1145/3565967.3571764
Venkatagiri SMukhopadhyay AHicks DBrantly ALuther K(2023)CoSINT: Designing a Collaborative Capture the Flag Competition to Investigate MisinformationProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3595997(2551-2572)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563657.3595997
Flores-Saviaga CFeng SSavage S(2022)Datavoidant: An AI System for Addressing Political Data Voids on Social MediaProceedings of the ACM on Human-Computer Interaction10.1145/35556166:CSCW2(1-29)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555616
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents