Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

Estimating Conversational Styles in Conversational Microtask Crowdsourcing

Published: 29 May 2020 Publication History

Abstract

Crowdsourcing marketplaces have provided a large number of opportunities for online workers to earn a living. To improve satisfaction and engagement of such workers, who are vital for the sustainability of the marketplaces, recent works have used conversational interfaces to support the execution of a variety of crowdsourcing tasks. The rationale behind using conversational interfaces stems from the potential engagement that conversation can stimulate. Prior works in psychology have also shown that conversational styles can play an important role in communication. There are unexplored opportunities to estimate a worker's conversational style with an end goal of improving worker satisfaction, engagement and quality. Addressing this knowledge gap, we investigate the role of conversational styles in conversational microtask crowdsourcing. To this end, we design a conversational interface which supports task execution, and we propose methods to estimate the conversational style of a worker. Our experimental setup was designed to empirically observe how conversational styles of workers relate with quality-related outcomes. Results show that even a naive supervised classifier can predict the conversation style with high accuracy (80%), and crowd workers with an Involvement conversational style provided a significantly higher output quality, exhibited a higher user engagement and perceived less cognitive task load in comparison to their counterparts. Our findings have important implications on task design with respect to improving worker performance and their engagement in microtask crowdsourcing.

Supplementary Material

ZIP File (v4cscw032aux.zip)

References

[1]
Ahmed Abbasi, Hsinchun Chen, and Arab Salem. 2008. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM Transactions on Information Systems (TOIS), Vol. 26, 3 (2008), 1--34.
[2]
Sandeep Avula, Gordon Chadwick, Jaime Arguello, and Robert Capra. 2018. SearchBots: User Engagement with ChatBots During Collaborative Search. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval. ACM, 52--61.
[3]
Alessandro Bozzon, Piero Fraternali, Luca Galli, and Roula Karam. 2014. Modeling CrowdSourcing Scenarios in Socially-Enabled Human Computation Applications. Journal on Data Semantics, Vol. 3, 3 (2014), 169--188. https://doi.org/10.1007/s13740-013-0032--2
[4]
Luka Bradevs ko, Michael Witbrock, Janez Starc, Zala Herga, Marko Grobelnik, and Dunja Mladenić. 2017. Curious Cat--Mobile, Context-Aware Conversational Crowdsourcing Knowledge Acquisition. ACM Transactions on Information Systems (TOIS), Vol. 35, 4 (2017), 33.
[5]
Erin Brady, Meredith Ringel Morris, and Jeffrey P Bigham. 2015. Gauging receptiveness to social microvolunteering. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1055--1064.
[6]
John D Burger, John Henderson, George Kim, and Guido Zarrella. 2011. Discriminating gender on Twitter. In Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, 1301--1309.
[7]
Zoey Chen and Jonah Berger. 2013. When, why, and how controversy causes conversation. Journal of Consumer Research, Vol. 40, 3 (2013), 580--593.
[8]
Phil Cohen, Adam Cheyer, Eric Horvitz, Rana El Kaliouby, and Steve Whittaker. 2016. On the future of personal assistants. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 1032--1037.
[9]
Benjamin R Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. What can i help you with?: infrequent users' experiences of intelligent personal assistants. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 43.
[10]
Gianluca Demartini, Djellel Eddine Difallah, Ujwal Gadiraju, Michele Catasta, et al. 2017. An introduction to hybrid human-machine information systems. Foundations and Trends® in Web Science, Vol. 7, 1 (2017), 1--87.
[11]
Carsten Eickhoff and Arjen P de Vries. 2013. Increasing cheat robustness of crowdsourcing tasks. Information retrieval, Vol. 16, 2 (2013), 121--137.
[12]
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, Vol. 76, 5 (1971), 378.
[13]
Ujwal Gadiraju, Alessandro Checco, Neha Gupta, and Gianluca Demartini. 2017. Modus operandi of crowd workers: The invisible role of microtask work environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 1, 3 (2017), 49.
[14]
Ujwal Gadiraju and Stefan Dietze. 2017. Improving learning through achievement priming in crowdsourced information finding microtasks. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference. 105--114.
[15]
Ujwal Gadiraju, Ricardo Kawase, and Stefan Dietze. 2014. A taxonomy of microtasks on the web. In Proceedings of the 25th ACM conference on Hypertext and social media. ACM, 218--223.
[16]
Amy L Gonzales, Jeffrey T Hancock, and James W Pennebaker. 2010. Language style matching as a predictor of social dynamics in small groups. Communication Research, Vol. 37, 1 (2010), 3--19.
[17]
Bettina Graf, Maike Krüger, Felix Müller, Alexander Ruhland, and Andrea Zech. 2015. Nombot: simplify food tracking. In Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia. ACM, 360--363.
[18]
Nathan Hahn, Shamsi T Iqbal, and Jaime Teevan. 2019. Casual Microtasking: Embedding Microtasks in Facebook. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 19.
[19]
Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 2019 a. All those wasted hours: On task abandonment in crowdsourcing. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. ACM, 321--329.
[20]
Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 2019 b. The impact of task abandonment in crowdsourcing. IEEE Transactions on Knowledge and Data Engineering (2019).
[21]
Ting-Hao Kenneth Huang, Joseph Chee Chang, and Jeffrey P Bigham. 2018. Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself Over Time. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 295.
[22]
Ting-Hao Kenneth Huang, Walter S Lasecki, and Jeffrey P Bigham. 2015. Guardian: A crowd-powered spoken dialog system for web apis. In Third AAAI conference on human computation and crowdsourcing.
[23]
Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--12.
[24]
Kevin Jepson. 2005. Conversations-and negotiated interaction-in text and voice chat rooms. Language Learning & Technology, Vol. 9, 3 (2005), 79--98.
[25]
Patrik Jonell, Mattias Bystedt, Fethiye Irmak Dogan, Per Fallgren, Jonas Ivarsson, Marketa Slukova, José Lopes Ulme Wennberg, Johan Boye, and Gabriel Skantze. 2018. Fantom: A Crowdsourced Social Chatbot using an Evolving Dialog Graph. Proc. Alexa Prize (2018).
[26]
Joy Kim, Sarah Sterman, Allegra Argent Beal Cohen, and Michael S Bernstein. 2017. Mechanical novel: Crowdsourcing complex work through reflection and revision. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 233--245.
[27]
Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Article 86, 12 pages. https://doi.org/10.1145/3290605.3300316
[28]
Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah, Aidan C Crook, Imed Zitouni, and Tasos Anastasakos. 2016. Understanding user satisfaction with intelligent assistants. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval. ACM, 121--130.
[29]
Aniket Kittur, Susheel Khamkar, Paul André, and Robert Kraut. 2012. CrowdWeaver: visually managing complex crowd work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1033--1036.
[30]
Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43--52.
[31]
Ari Kobren, Chun How Tan, Panagiotis Ipeirotis, and Evgeniy Gabrilovich. 2015. Getting More for Less: Optimized Crowdsourcing with Dynamic Tasks and Goals. In Proceedings of the 24th International Conference on World Wide Web (WWW '15). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 592--602. https://doi.org/10.1145/2736277.2741681
[32]
Pavel Kucherbaev, Alessandro Bozzon, and Geert-Jan Houben. 2018. Human-Aided Bots. IEEE Internet Computing, Vol. 22, 6 (2018), 36--43.
[33]
Robin Tolmach Lakoff. 1979. Stylistic strategies within a grammar of style. Annals of the New York Academy of Sciences, Vol. 327, 1 (1979), 53--78.
[34]
Walter S Lasecki, Rachel Wesley, Jeffrey Nichols, Anand Kulkarni, James F Allen, and Jeffrey P Bigham. 2013. Chorus: a crowd-powered conversational assistant. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 151--162.
[35]
Ewa Luger and Abigail Sellen. 2016. Like having a really bad PA: the gulf between user expectation and experience of conversational agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5286--5297.
[36]
Andrew Mao, Ece Kamar, and Eric Horvitz. 2013. Why stop now? predicting worker engagement in online crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing.
[37]
Panagiotis Mavridis, Owen Huang, Sihang Qiu, Ujwal Gadiraju, and Alessandro Bozzon. 2019. Chatterbox: Conversational Interfaces for Microtask Crowdsourcing. In Proceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization. ACM, 243--251.
[38]
Robert J Moore, Raphael Arar, Guang-Jie Ren, and Margaret H Szymanski. 2017. Conversational UX design. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 492--497.
[39]
Sean A Munson, Karina Kervin, and Lionel P Robert Jr. 2014. Monitoring email to indicate project team performance and mutual attraction. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 542--549.
[40]
Dong Nguyen, Noah A Smith, and Carolyn P Rosé. 2011. Author age prediction from text using linear regression. In Proceedings of the 5th ACL-HLT workshop on language technology for cultural heritage, social sciences, and humanities. Association for Computational Linguistics, 115--123.
[41]
Heather O'Brien. 2016. Theoretical perspectives on user engagement. In Why Engagement Matters. Springer, 1--26.
[42]
Heather L O'Brien, Paul Cairns, and Mark Hall. 2018. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies, Vol. 112 (2018), 28--39.
[43]
Nilma Perera, Gregor Kennedy, and Jon Pearce. 2008. Are You Bored?: Maybe an Interface Agent Can Help!. In Proceedings of the 20th Australasian Conference on Computer-Human Interaction: Designing for Habitus and Habitat (OZCHI '08). ACM, New York, NY, USA, 49--56. https://doi.org/10.1145/1517744.1517760
[44]
Sihang Qiu, Ujwal Gadiraju, and Alessandro Bozzon. 2020. Improving Worker Engagement Through Conversational Microtask Crowdsourcing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1--12.
[45]
M Rafael Salaberry. 2000. L2 morphosyntactic development in text-based computer-mediated communication. Computer Assisted Language Learning, Vol. 13, 1 (2000), 5--27.
[46]
Saiph Savage, Andres Monroy-Hernandez, and Tobias Höllerer. 2016. Botivist: Calling volunteers to action using online bots. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 813--822.
[47]
Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. 2006. Effects of age and gender on blogging. In AAAI spring symposium: Computational approaches to analyzing weblogs. 199--205.
[48]
Ameneh Shamekhi, Mary Czerwinski, Gloria Mark, Margeigh Novotny, and Gregory A Bennett. 2016. An exploratory study toward the preferred conversational style for compatible virtual agents. In International Conference on Intelligent Virtual Agents. Springer, 40--50.
[49]
Deborah Tannen. 1987. Conversational style. Psycholinguistic models of production (1987), 251--267.
[50]
Deborah Tannen. 2005. Conversational style: Analyzing talk among friends .Oxford University Press.
[51]
Yla R Tausczik and James W Pennebaker. 2013. Improving teamwork using real-time language feedback. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 459--468.
[52]
Paul Thomas, Mary Czerwinski, Daniel McDuff, Nick Craswell, and Gloria Mark. 2018. Style and alignment in information-seeking conversation. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval. ACM, 42--51.
[53]
Paul Thomas, Daniel McDuff, Mary Czerwinski, and Nick Craswell. 2017. MISC: A data set of information-seeking conversations. In SIGIR 1st International Workshop on Conversational Approaches to Information Retrieval (CAIR'17), Vol. 5.
[54]
Carlos Toxtli, Joel Chan, Walter S Lasecki, and Saiph Savage. 2018. Enabling Expert Critique with Chatbots and Micro Guidance. In Collective Intelligence 2018. ACM, 4.
[55]
Melissa A Valentine, Daniela Retelny, Alexandra To, Negar Rahmati, Tulsee Doshi, and Michael S Bernstein. 2017. Flash organizations: Crowdsourcing complex work by structuring crowds as organizations. In Proceedings of the 2017 CHI conference on human factors in computing systems. ACM, 3523--3537.
[56]
Bert Vandenberghe. 2017. Bot personas as off-the-shelf users. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 782--789.
[57]
Alexandra Vtyurina, Denis Savenkov, Eugene Agichtein, and Charles LA Clarke. 2017. Exploring conversational search with humans, assistants, and wizards. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2187--2193.
[58]
Jie Yang, Judith Redi, Gianluca Demartini, and Alessandro Bozzon. 2016. Modeling task complexity in crowdsourcing. In Fourth AAAI Conference on Human Computation and Crowdsourcing.
[59]
Xi Yang, Marco Aurisicchio, and Weston Baxter. 2019. Understanding Affective Experiences With Conversational Agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 542.
[60]
Ying Zhang, Xianghua Ding, and Ning Gu. 2018. Understanding Fatigue and its Impact in Crowdsourcing. In 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design ((CSCWD)). IEEE, 57--62.
[61]
Zijian Zhang, Jaspreet Singh, Ujwal Gadiraju, and Avishek Anand. 2019. Dissonance Between Human and Machine Understanding. Proc. ACM Hum.-Comput. Interact. 3 CSCW, Vol. 56 (2019), 26.
[62]
Mengdie Zhuang and Ujwal Gadiraju. 2019. In What Mood Are You Today? An Analysis of Crowd Workers' Mood, Performance and Engagement. In Proceedings of the 10th ACM Conference on Web Science. 373--382.

Cited By

View all
  • (2024)DECI: The 2nd Tutorial on Designing Effective Conversational InterfacesAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3658531(5-8)Online publication date: 27-Jun-2024
  • (2024)"Are we all in the same boat?" Customizable and Evolving Avatars to Improve Worker Engagement and Foster a Sense of Community in Online Crowd WorkProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642429(1-26)Online publication date: 11-May-2024
  • (2023)ContextBotProceedings of the 34th ACM Conference on Hypertext and Social Media10.1145/3603163.3609031(1-14)Online publication date: 4-Sep-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction
Proceedings of the ACM on Human-Computer Interaction  Volume 4, Issue CSCW1
CSCW
May 2020
1285 pages
EISSN:2573-0142
DOI:10.1145/3403424
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 May 2020
Published in PACMHCI Volume 4, Issue CSCW1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. cognitive task load.
  2. conversational style
  3. microtask crowdsourcing
  4. user engagement
  5. work outcomes

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)157
  • Downloads (Last 6 weeks)30
Reflects downloads up to 12 Sep 2024

Other Metrics

Citations

Cited By

View all
  • (2024)DECI: The 2nd Tutorial on Designing Effective Conversational InterfacesAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3658531(5-8)Online publication date: 27-Jun-2024
  • (2024)"Are we all in the same boat?" Customizable and Evolving Avatars to Improve Worker Engagement and Foster a Sense of Community in Online Crowd WorkProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642429(1-26)Online publication date: 11-May-2024
  • (2023)ContextBotProceedings of the 34th ACM Conference on Hypertext and Social Media10.1145/3603163.3609031(1-14)Online publication date: 4-Sep-2023
  • (2023)DECI: A Tutorial on Designing Effective Conversational InterfacesCompanion Proceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581754.3584165(187-189)Online publication date: 27-Mar-2023
  • (2023)CommunityBots: Creating and Evaluating A Multi-Agent Chatbot Platform for Public Input ElicitationProceedings of the ACM on Human-Computer Interaction10.1145/35794697:CSCW1(1-32)Online publication date: 16-Apr-2023
  • (2023)Crowdsourcing as a Future Collaborative Computing ParadigmMobile Crowdsourcing10.1007/978-3-031-32397-3_1(3-32)Online publication date: 21-Apr-2023
  • (2022)Designing PairBuddy—A Conversational Agent for Pair ProgrammingACM Transactions on Computer-Human Interaction10.1145/349832629:4(1-44)Online publication date: 5-May-2022
  • (2022)Great Chain of Agents: The Role of Metaphorical Representation of Agents in Conversational CrowdsourcingProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517653(1-22)Online publication date: 29-Apr-2022
  • (2022)To Trust or Not To Trust: How a Conversational Interface Affects Trust in a Decision Support SystemProceedings of the ACM Web Conference 202210.1145/3485447.3512248(3531-3540)Online publication date: 25-Apr-2022

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media