research-article

CollabCoder: A GPT-Powered WorkFlow for Collaborative Qualitative Analysis

Authors:

Toby Jia-Jun Li,

Simon Tangi PerraultAuthors Info & Claims

CSCW '23 Companion: Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing

Pages 354 - 357

https://doi.org/10.1145/3584931.3607500

Published: 14 October 2023 Publication History

Abstract

Collaborative Qualitative Analysis (CQA) process can be time-consuming and resource-intensive, requiring multiple discussions among team members to refine codes and ideas before reaching a consensus. We introduce CollabCoder, a system leveraging Large Language Models (LLMs) to support three CQA stages: independent open coding, iterative discussions, and the development of a final codebook. In the independent open coding phase, CollabCoder provides AI-generated code suggestions on demand and allows users to record coding decision-making information (e.g. keywords and certainty) as support for the process. During the discussion phase, CollabCoder helps to build mutual understanding and productive discussion by sharing coding decision-making information within the team. It also helps to quickly identify agreements and disagreements through quantitative metrics, in order to build a final consensus. During the code grouping phase, CollabCoder employs a top-down approach for primary code group recommendations, reducing the cognitive burden of generating the final codebook. The source code for CollabCoder can be accessed via GitHub at https://github.com/gaojie058/CollabCoder.

References

[1]

Jessica Díaz, Jorge Enrique Pérez-Martínez, Carolina Gallardo, and Ángel González-Prieto. 2021. Applying Inter-rater Reliability and Agreement in Grounded Theory Studies in Software Engineering. CoRR abs/2107.11449 (2021). arXiv:2107.11449https://arxiv.org/abs/2107.11449

[2]

Margaret Drouhard, Nan-Chen Chen, Jina Suh, Rafal Kocielnik, Vanessa Peña-Araya, Keting Cen, Xiangyi Zheng, and Cecilia R. Aragon. 2017. Aeonium: Visual analytics to support collaborative qualitative coding. In 2017 IEEE Pacific Visualization Symposium (PacificVis). 220–229. https://doi.org/10.1109/PACIFICVIS.2017.8031598

[3]

Jessica L. Feuston and Jed R. Brubaker. 2021. Putting Tools in Their Place: The Role of Time and Perspective in Human-AI Collaboration for Qualitative Analysis. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 469 (oct 2021), 25 pages. https://doi.org/10.1145/3479856

Digital Library

[4]

Abbas Ganji, Mania Orand, and David W McDonald. 2018. Ease on Down the Code: Complex Collaborative Qualitative Coding Simplified with’Code Wizard’. Proceedings of the ACM on human-computer interaction 2, CSCW (2018), 1–24.

Digital Library

[5]

Jie Gao, Kenny Tsu Wei Choo, Junming Cao, Roy Ka-Wei Lee, and Simon Perrault. 2023. CoAIcoder: Examining the Effectiveness of AI-Assisted Human-to-Human Collaboration in Qualitative Analysis. ACM Trans. Comput.-Hum. Interact. (aug 2023). https://doi.org/10.1145/3617362 Just Accepted.

Digital Library

[6]

Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Tangi Perrault. 2023. CollabCoder: A GPT-Powered Workflow for Collaborative Qualitative Analysis. arxiv:2304.07366 [cs.HC]

[7]

Simret Araya Gebreegziabher, Zheng Zhang, Xiaohang Tang, Yihao Meng, Elena L. Glassman, and Toby Jia-Jun Li. 2023. PaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 362, 19 pages. https://doi.org/10.1145/3544548.3581352

Digital Library

[8]

Matt-Heun Hong, Lauren A. Marsh, Jessica L. Feuston, Janet Ruppert, Jed R. Brubaker, and Danielle Albers Szafir. 2022. Scholastic: Graphical Human-AI Collaboration for Inductive and Interpretive Text Analysis. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 30, 12 pages. https://doi.org/10.1145/3526113.3545681

Digital Library

[9]

Jialun Aaron Jiang, Kandrea Wade, Casey Fiesler, and Jed R. Brubaker. 2021. Supporting Serendipity: Opportunities and Challenges for Human-AI Collaboration in Qualitative Analysis. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 94 (apr 2021), 23 pages. https://doi.org/10.1145/3449168

Digital Library

[10]

Alireza Nili, Mary Tate, Alistair Barros, and David Johnstone. 2020. An approach for selecting and using a method of inter-coder reliability in information management research. International Journal of Information Management 54 (2020), 102154. https://doi.org/10.1016/j.ijinfomgt.2020.102154

Digital Library

[11]

OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774 [cs.CL]

[12]

K Andrew R Richards and Michael A Hemphill. 2018. A practical guide to collaborative qualitative data analysis. Journal of Teaching in Physical Education 37, 2 (2018), 225–231. https://doi.org/10.1123/jtpe.2017-0084

[13]

Tim Rietz and Alexander Maedche. 2021. Cody: An AI-Based System to Semi-Automate Coding for Qualitative Research. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 394, 14 pages. https://doi.org/10.1145/3411764.3445591

Digital Library

[14]

Ziang Xiao, Xingdi Yuan, Q. Vera Liao, Rania Abdelghani, and Pierre-Yves Oudeyer. 2023. Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23 Companion). Association for Computing Machinery, New York, NY, USA, 75–78. https://doi.org/10.1145/3581754.3584136

Digital Library

[15]

Himanshu Zade, Margaret Drouhard, Bonnie Chinh, Lu Gan, and Cecilia Aragon. 2018. Conceptualizing Disagreement in Qualitative Coding. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3173574.3173733

Digital Library

[16]

Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li. 2023. VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping. arxiv:2304.07810 [cs.HC]

Cited By

Cardoso Sampaio RChagas VSinimbu Sanchez CGonçalves JBorges TBrum Alison MSchiavon Tigrinho CRibeiro de Souza JSchwarzer Paz F(2024)Uma revisão de escopo assistida por inteligência artificial (IA) sobre usos emergentes de ia na pesquisa qualitativa e suas considerações éticasRevista Pesquisa Qualitativa10.33361/RPQ.2024.v.12.n.30.72912:30(01-28)Online publication date: 1-May-2024
https://doi.org/10.33361/RPQ.2024.v.12.n.30.729
Lee Vvan der Lubbe SGoh LValderas J(2024)Harnessing ChatGPT for Thematic Analysis: Are We Ready?Journal of Medical Internet Research10.2196/5497426(e54974)Online publication date: 31-May-2024
https://doi.org/10.2196/54974
Bland TGuo MDousay T(2024)Multimedia design for learner interest and achievement: a visual guide to pharmacologyBMC Medical Education10.1186/s12909-024-05077-y24:1Online publication date: 5-Feb-2024
https://doi.org/10.1186/s12909-024-05077-y
Show More Cited By

Index Terms

CollabCoder: A GPT-Powered WorkFlow for Collaborative Qualitative Analysis
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing systems and tools

Recommendations

CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Collaborative Qualitative Analysis (CQA) can enhance qualitative analysis rigor and depth by incorporating varied viewpoints. Nevertheless, ensuring a rigorous CQA procedure itself can be both complex and costly. To lower this bar, we take a theoretical ...
Ways of Qualitative Coding: A Case Study of Four Strategies for Resolving Disagreements
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

The process of qualitative coding often involves multiple coders coding the same data to ensure reliable codes and a consistent understanding of the codebook. One aspect of qualitative coding includes resolving disagreements, where coders discuss ...
Ease on Down the Code: Complex Collaborative Qualitative Coding Simplified with 'Code Wizard'

This paper describes the design and development of a preliminary qualitative coding tool as well as a method to improve the process of achieving inter-coder reliability (ICR) in small teams. Software applications that support qualitative coding do not ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '23 Companion: Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing

October 2023

596 pages

ISBN:9798400701290

DOI:10.1145/3584931

Editors:
Casey Fiesler
University of Colorado Boulder, USA
,
Loren Terveen
University of Minnesota, USA
,
Morgan Ames
University of California, Berkeley, USA
,
Susan Fussell
Cornell University, USA
,
Eric Gilbert
University of Michigan, USA
,
Vera Liao
Microsoft Research Montreal, Canada
,
Xiaojuan Ma
Hong Kong University of Science and Technology, Hong Kong
,
Xinru Page
Brigham Young University, USA
,
Mark Rouncefield
Lancaster University, UK
,
Vivek Singh
Rutgers University, USA
,
Pamela Wisniewski
Vanderbilt University, USA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CSCW '23

Sponsor:

SIGCHI

CSCW '23: Computer Supported Cooperative Work and Social Computing

October 14 - 18, 2023

MN, Minneapolis, USA

Acceptance Rates

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
439
Total Downloads

Downloads (Last 12 months)325
Downloads (Last 6 weeks)12

Reflects downloads up to 23 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cardoso Sampaio RChagas VSinimbu Sanchez CGonçalves JBorges TBrum Alison MSchiavon Tigrinho CRibeiro de Souza JSchwarzer Paz F(2024)Uma revisão de escopo assistida por inteligência artificial (IA) sobre usos emergentes de ia na pesquisa qualitativa e suas considerações éticasRevista Pesquisa Qualitativa10.33361/RPQ.2024.v.12.n.30.72912:30(01-28)Online publication date: 1-May-2024
https://doi.org/10.33361/RPQ.2024.v.12.n.30.729
Lee Vvan der Lubbe SGoh LValderas J(2024)Harnessing ChatGPT for Thematic Analysis: Are We Ready?Journal of Medical Internet Research10.2196/5497426(e54974)Online publication date: 31-May-2024
https://doi.org/10.2196/54974
Bland TGuo MDousay T(2024)Multimedia design for learner interest and achievement: a visual guide to pharmacologyBMC Medical Education10.1186/s12909-024-05077-y24:1Online publication date: 5-Feb-2024
https://doi.org/10.1186/s12909-024-05077-y
Friedman COwen AVanPuymbrouck L(2024)Should ChatGPT help with my research? A caution against artificial intelligence in qualitative analysisQualitative Research10.1177/14687941241297375Online publication date: 5-Dec-2024
https://doi.org/10.1177/14687941241297375
Liu JWang YLyu YSu YNiu SXu XZhang YFarzan RLópez CCardoso Llach DQuercia DMustafa MNiu SWong-Villacrés M(2024)Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on DepressionCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681850(190-196)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3678884.3681850
Sinha RSolola INguyen HSwanson HLawrence L(2024)The Role of Generative AI in Qualitative Research: GPT-4's Contributions to a Grounded Theory AnalysisProceedings of the 2024 Symposium on Learning, Design and Technology10.1145/3663433.3663456(17-25)Online publication date: 21-Jun-2024
https://dl.acm.org/doi/10.1145/3663433.3663456
Oksanen J(2024)Bridging the Integrity Gap: Towards AI-assisted Design ResearchExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3647962(1-5)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3647962
Choksi MAubin Le Quéré MLloyd TTao RGrimmelmann JNaaman M(2024)Under the (neighbor)hood: Hyperlocal Surveillance on NextdoorProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641967(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641967
Zhang ZHu MZhang BJin X(2024)Application Scenario Analysis of Large Language Models Education Based on Activity Theory2024 IEEE/ACIS 24th International Conference on Computer and Information Science (ICIS)10.1109/ICIS61260.2024.10778383(30-34)Online publication date: 20-Sep-2024
https://doi.org/10.1109/ICIS61260.2024.10778383
De Paoli SMathis W(2024)Reflections on inductive thematic saturation as a potential metric for measuring the validity of an inductive thematic analysis with LLMsQuality & Quantity10.1007/s11135-024-01950-6Online publication date: 22-Aug-2024
https://doi.org/10.1007/s11135-024-01950-6
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten