research-article

Open access

Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development

Authors:

Serena BoothAuthors Info & Claims

AIES '21: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society

Pages 134 - 145

https://doi.org/10.1145/3461702.3462527

Published: 30 July 2021 Publication History

Abstract

Practitioners from diverse occupations and backgrounds are increasingly using machine learning (ML) methods. Nonetheless, studies on ML Practitioners typically draw populations from Big Tech and academia, as researchers have easier access to these communities. Through this selection bias, past research often excludes the broader, lesser-resourced ML community---for example, practitioners working at startups, at non-tech companies, and in the public sector. These practitioners share many of the same ML development difficulties and ethical conundrums as their Big Tech counterparts; however, their experiences are subject to additional under-studied challenges stemming from deploying ML with limited resources, increased existential risk, and absent access to in-house research teams. We contribute a qualitative analysis of 17 interviews with stakeholders from organizations which are less represented in prior studies. We uncover a number of tensions which are introduced or exacerbated by these organizations' resource constraints---tensions between privacy and ubiquity, resource management and performance optimization, and access and monopolization. Increased academic focus on these practitioners can facilitate a more holistic understanding of ML limitations, and so is useful for prescribing a research agenda to facilitate responsible ML development for all.

Supplementary Material

PDF File (aiesfp219aux.pdf)

Download
374.79 KB

References

[1]

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems. 1--13.

Digital Library

[2]

Cecilia Aragon, Clayton Hutto, Andy Echenique, Brittany Fiore-Gartland, Yun Huang, Jinyoung Kim, Gina Neff, Wanli Xing, and Joseph Bayer. 2016. Developing a Research Agenda for Human-Centered Data Science. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion (San Francisco, California, USA) (CSCW '16 Companion). Association for Computing Machinery, New York, NY, USA, 529--535.

Digital Library

[3]

Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019. Beyond accuracy: The role of mental models in human-ai team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7. 2--11.

[4]

Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, Vol. 6 (2018), 587--604.

[5]

Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Allison Woodruff, Christine Luu, Pierre Kreitmann, Jonathan Bischof, and Ed H Chi. 2019. Putting fairness principles into practice: Challenges, metrics, and improvements. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 453--459.

Digital Library

[6]

Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Gauthier Melancc on, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, et al. 2020. Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty. arXiv preprint arXiv:2011.07586 (2020).

[7]

Serena Booth, Yilun Zhou, Ankit Shah, and Julie Shah. 2021. Bayes-TrEx: a Bayesian Sampling Approach to Model Transparency by Example. Proceedings of the 35th AAAI Conference on Artificial Intelligence (2021).

[8]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology, Vol. 3, 2 (2006), 77--101.

[9]

Andrea Brennen. 2020. What Do People Really Want When They Say They Want" Explainable AI?" We Asked 60 Stakeholders. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1--7.

Digital Library

[10]

Carrie J Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019. “Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making. Proceedings of the ACM on Human-computer Interaction, Vol. 3, CSCW (2019), 1--24.

Digital Library

[11]

M. Correll and M. Gleicher. 2014. Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error. IEEE Transactions on Visualization and Computer Graphics, Vol. 20, 12 (2014), 2142--2151. https://doi.org/10.1109/TVCG.2014.2346298

[12]

Kate Crawford and Ryan Calo. 2016. There is a blind spot in AI research. Nature, Vol. 538, 7625 (2016), 311--313.

[13]

Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. 2020. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03395 (2020).

[14]

Stephanie Deitrick and Robert Edsall. 2006. The influence of uncertainty visualization on decision making: An empirical evaluation. In Progress in spatial data handling. Springer, 719--738.

[15]

Mary T Dzindolet, Scott A Peterson, Regina A Pomranky, Linda G Pierce, and Hall P Beck. 2003. The role of trust in automation reliance. International journal of human-computer studies, Vol. 58, 6 (2003), 697--718.

Digital Library

[16]

Robert Wall Emerson. 2015. Convenience sampling, random sampling, and snowball sampling: How does sampling affect the validity of research? Journal of Visual Impairment & Blindness, Vol. 109, 2 (2015), 164--168.

[17]

Ilker Etikan, Sulaiman Abubakar Musa, and Rukayya Sunusi Alkassim. 2016. Comparison of convenience sampling and purposive sampling. American journal of theoretical and applied statistics, Vol. 5, 1 (2016), 1--4.

[18]

Antonio Ginart, Melody Y Guan, Gregory Valiant, and James Zou. 2019. Making AI forget you: Data deletion in machine learning. arXiv preprint arXiv:1907.05012 (2019).

[19]

Bryce Goodman and Seth Flaxman. 2017. European Union regulations on algorithmic decision-making and a ?right to explanation". AI magazine, Vol. 38, 3 (2017), 50--57.

[20]

Alicja Gosiewska and Przemyslaw Biecek. 2019. iBreakDown: Uncertainty of Model Explanations for Nonadditive Predictive Models. arXiv preprint arXiv:1903.11420 (2019).

[21]

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning. PMLR, 1321--1330.

[22]

Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M. Drucker. 2019. Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--13.

Digital Library

[23]

Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677 (2018).

[24]

Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--16.

Digital Library

[25]

Sungsoo Ray Hong, Jessica Hullman, and Enrico Bertini. 2020. Human factors in model interpretability: Industry practices, challenges, and needs. Proceedings of the ACM on Human-Computer Interaction, Vol. 4, CSCW1 (2020), 1--26.

Digital Library

[26]

Aspen K Hopkins, Michael Correll, and Arvind Satyanarayan. 2020. VisuaLint: Sketchy In Situ Annotations of Chart Construction Errors. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 219--228.

[27]

C. R. Johnson and A. R. Sanderson. 2003. A Next Step: Visualizing Errors and Uncertainty. IEEE Computer Graphics and Applications, Vol. 23, 5 (2003), 6--10. https://doi.org/10.1109/MCG.2003.1231171

Digital Library

[28]

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. 2020. Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--14.

Digital Library

[29]

Miryung Kim, Thomas Zimmermann, Robert DeLine, and Andrew Begel. 2016. The emerging role of data scientists on software development teams. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, 96--107.

Digital Library

[30]

M. Kim, T. Zimmermann, R. DeLine, and A. Begel. 2018. Data Scientists in Software Teams: State of the Art and Challenges. IEEE Transactions on Software Engineering, Vol. 44, 11 (2018), 1024--1038.

[31]

Zachary C Lipton. 2018. The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, Vol. 16, 3 (2018), 31--57.

Digital Library

[32]

Nina McCurdy, Julie Gerdes, and Miriah Meyer. 2018. A framework for externalizing implicit error using visualization. IEEE transactions on visualization and computer graphics, Vol. 25, 1 (2018), 925--935.

[33]

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 (2019).

[34]

Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency. 220--229.

Digital Library

[35]

Michael Muller, Melanie Feinberg, Timothy George, Steven J. Jackson, Bonnie E. John, Mary Beth Kery, and Samir Passi. 2019 a. Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA '19). Association for Computing Machinery, New York, NY, USA, 1--8.

[36]

Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q Vera Liao, Casey Dugan, and Thomas Erickson. 2019 b. How data science workers work with data: Discovery, capture, curation, design, creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--15.

Digital Library

[37]

Michael Muller, Christine Wolf, Josh Andres, Michael Desmond, Narendra Nath Joshi, Zahra Ashktorab, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Evelyn Duesterwald, et al. 2021. Designing Ground Truth and the Social Life of Labels. Submitted for publication to CHI (2021).

[38]

Michael J Muller and Sarah Kuhn. 1993. Participatory design. Commun. ACM, Vol. 36, 6 (1993), 24--28.

Digital Library

[39]

Lace Padilla, Matthew Kay, and Jessica Hullman. 2020. Uncertainty visualization. (2020).

[40]

David Lorge Parnas. 1979. Designing software for ease of extension and contraction. IEEE transactions on software engineering 2 (1979), 128--138.

Digital Library

[41]

Samir Passi and Steven J. Jackson. 2018. Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects. Proc. ACM Hum.-Comput. Interact., Vol. 2, CSCW, Article 136 (Nov. 2018), 28 pages.

Digital Library

[42]

Paula Pereira, Jácome Cunha, and Jo ao Paulo Fernandes. 2020. On understanding data scientists. In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 1--5.

[43]

Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-Francc ois Bonnefon, Cynthia Breazeal, Jacob W Crandall, Nicholas A Christakis, Iain D Couzin, Matthew O Jackson, et al. 2019. Machine behaviour. Nature, Vol. 568, 7753 (2019), 477--486.

[44]

Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 33--44.

Digital Library

[45]

Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2020. Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices. arXiv preprint arXiv:2006.12358 (2020).

[46]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135--1144.

Digital Library

[47]

Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. arXiv preprint arXiv:2005.04118 (2020).

[48]

Rwamahe Rutakumwa, Joseph Okello Mugisha, Sarah Bernays, Elizabeth Kabunga, Grace Tumwekwase, Martin Mbonye, and Janet Seeley. 2020. Conducting in-depth interviews with and without voice recorders: a comparative analysis. Qualitative Research, Vol. 20, 5 (2020), 565--581.

[49]

Lloyd S Shapley. 1953. A value for n-person games. Contributions to the Theory of Games, Vol. 2, 28 (1953), 307--317.

[50]

Tom Simonite. 2017. AI and 'Enormous Data' Could Make Tech Giants Harder to Topple. Wired (2017).

[51]

Svenja C Sommer, Christoph H Loch, and Jing Dong. 2009. Managing complexity and unforeseeable uncertainty in startup companies: An empirical study. Organization Science, Vol. 20, 1 (2009), 118--133.

Digital Library

[52]

William Stallings. 2020. Handling of Personal Information and Deidentified, Aggregated, and Pseudonymized Information Under the California Consumer Privacy Act. IEEE Security & Privacy, Vol. 18, 1 (2020), 61--64.

[53]

Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.02243 (2019).

[54]

Harini Suresh, Natalie Lao, and Ilaria Liccardi. 2020. Misplaced Trust: Measuring the Interference of Machine Learning in Human Decision-Making. In 12th ACM Conference on Web Science. 315--324.

[55]

Paul Vanhaesebrouck, Aurélien Bellet, and Marc Tommasi. 2017. Decentralized collaborative learning of personalized models over networks. In Artificial Intelligence and Statistics. PMLR, 509--517.

[56]

S. Viaene. 2013. Data Scientists Aren't Domain Experts. IT Professional, Vol. 15, 6 (2013), 12--17.

[57]

Eduard Fosch Villaronga, Peter Kieseberg, and Tiffany Li. 2018. Humans forget, machines remember: Artificial intelligence and the right to be forgotten. Computer Law & Security Review, Vol. 34, 2 (2018), 304--313.

[58]

Dakuo Wang, Justin D. Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019. Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI. Proc. ACM Hum.-Comput. Interact., Vol. 3, CSCW, Article 211 (Nov. 2019), 24 pages.

Digital Library

[59]

Jenna Wiens, Suchi Saria, Mark Sendak, Marzyeh Ghassemi, Vincent X Liu, Finale Doshi-Velez, Kenneth Jung, Katherine Heller, David Kale, Mohammed Saeed, et al. 2019. Do no harm: a roadmap for responsible machine learning for health care. Nature medicine, Vol. 25, 9 (2019), 1337--1340.

[60]

Amy X. Zhang, Michael Muller, and Dakuo Wang. 2020. How do Data Science Workers Collaborate? Roles, Workflows, and Tools. Proc. ACM Hum.-Comput. Interact. CSCW (Oct. 2020).

Digital Library

Cited By

Hanschke VRees DAlanyali MHopkinson DMarshall P(2024)Data Ethics Emergency Drill: A Toolbox for Discussing Responsible AI for Industry TeamsProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642402(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642402
Hohman FKery MRen DMoritz D(2024)Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning ExperiencesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642109(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642109
Suh AAppleby GAnderson EFinelli LChang RCashman D(2024)Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter ExpertsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.325934130:7(4137-4153)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2023.3259341
Show More Cited By

Index Terms

Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development
1. Social and professional topics
  1. Professional topics
    1. Computing and business
      1. Socio-technical systems
    2. Computing profession
      1. Codes of ethics
      2. Computing organizations

Recommendations

Learning Contextual Inquiry and Distributed Cognition: a case study on technology use in anaesthesia

There have been few studies on how analysts learn or use frameworks to support gathering and analysis of field data. Distributed Cognition for Teamwork (DiCoT) is a framework that has been developed to facilitate the learning of Distributed Cognition (...
Software development as mediated activity: applying three analytical frameworks for studying compound mediation
SIGDOC '01: Proceedings of the 19th annual international conference on Computer documentation

Field research in software documentation has a tradition of investigating how artifacts (from documentation to online help to interfaces to mundane equipment such as Post-It? notes) mediate or enable workers to perform complex tasks (see for instance [...
Powerful Futures: How a Big Tech Company Envisions Humans and Technologies in the Workplace of the Future
CSCW

Big tech companies have had increasing control over how we work with technologies and how technologies define the work we do. In this paper, I identify the sociotechnical futures that Amazon-one of the big tech companies-envisions and the future of work ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AIES '21: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society

July 2021

1077 pages

ISBN:9781450384735

DOI:10.1145/3461702

Program Chairs:
Marion Fourcade
University of California Berkeley, USA
,
Benjamin Kuipers
University of Michigan, USA
,
Seth Lazar
Australian National University, Australia
,
Deirdre Mulligan
University of California Berkeley, USA

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 July 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Siebel Foundation
NSF

Conference

AIES '21

Sponsor:

SIGAI

AIES '21: AAAI/ACM Conference on AI, Ethics, and Society

May 19 - 21, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 61 of 162 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
985
Total Downloads

Downloads (Last 12 months)378
Downloads (Last 6 weeks)28

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hanschke VRees DAlanyali MHopkinson DMarshall P(2024)Data Ethics Emergency Drill: A Toolbox for Discussing Responsible AI for Industry TeamsProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642402(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642402
Hohman FKery MRen DMoritz D(2024)Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning ExperiencesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642109(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642109
Suh AAppleby GAnderson EFinelli LChang RCashman D(2024)Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter ExpertsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.325934130:7(4137-4153)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2023.3259341
Samsonova EMorić ZGvozden GHlupić T(2024)Data Enrichment Pipeline Model for Web Classification Based on Web Scraping and Machine Learning2024 47th MIPRO ICT and Electronics Convention (MIPRO)10.1109/MIPRO60963.2024.10569560(199-204)Online publication date: 20-May-2024
https://doi.org/10.1109/MIPRO60963.2024.10569560
Brower-Sinning RLewis GEcheverría SOzkaya I(2024)Using Quality Attribute Scenarios for ML Model Test Case Generation2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C)10.1109/ICSA-C63560.2024.00058(307-310)Online publication date: 4-Jun-2024
https://doi.org/10.1109/ICSA-C63560.2024.00058
Serban Avan der Blom KHoos HVisser J(2024)Software engineering practices for machine learning — Adoption, effects, and team assessmentJournal of Systems and Software10.1016/j.jss.2023.111907209:COnline publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111907
Li YGoel S(2024)Making It Possible for the Auditing of AI: A Systematic Review of AI Audits and AI AuditabilityInformation Systems Frontiers10.1007/s10796-024-10508-8Online publication date: 2-Jul-2024
https://doi.org/10.1007/s10796-024-10508-8
Bondi-Kelly EHartvigsen TSanneman LSankaranarayanan SHarned ZWickerson GGichoya JOakden-Rayner LCeli LLungren MShah JGhassemi M(2023)Taking Off with AI: Lessons from Aviation for HealthcareProceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization10.1145/3617694.3623224(1-14)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3617694.3623224
Berman G(2023)Machine Learning practices and infrastructuresProceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3600211.3604689(466-481)Online publication date: 8-Aug-2023
https://dl.acm.org/doi/10.1145/3600211.3604689
Cabrera ÁPerer AHong J(2023)Improving Human-AI Collaboration With Descriptions of AI BehaviorProceedings of the ACM on Human-Computer Interaction10.1145/35796127:CSCW1(1-21)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579612
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents