research-article

"How May I Help You?": Modeling Twitter Customer ServiceConversations Using Fine-Grained Dialogue Acts

Authors:

Pritam Gundecha,

Mansurul Bhuiyan,

Rama AkkirajuAuthors Info & Claims

IUI '17: Proceedings of the 22nd International Conference on Intelligent User Interfaces

Pages 343 - 355

https://doi.org/10.1145/3025171.3025191

Published: 07 March 2017 Publication History

Abstract

Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understand trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop a novel taxonomy of fine-grained "dialogue acts" frequently observed in customer service, showcasing acts that are more suited to the domain than the more generic existing taxonomies. Using a sequential SVM-HMM model, we model conversation flow, predicting the dialogue act of a given turn in real-time. We characterize differences between customer and agent behavior in Twitter customer service conversations, and investigate the effect of testing our system on different customer service industries. Finally, we use a data-driven approach to predict important conversation outcomes: customer satisfaction, customer frustration, and overall problem resolution. We show that the type and location of certain dialogue acts in a conversation have a significant effect on the probability of desirable and undesirable outcomes, and present actionable rules based on our findings. The patterns and rules we derive can be used as guidelines for outcome-driven automated customer service platforms.

References

[1]

Allen, J., Ferguson, G., and Stent, A. An architecture for more realistic conversational systems. In Proceedings of the 6th international conference on Intelligent user interfaces, ACM (2001), 1--8.

Digital Library

[2]

Altun, Y., Tsochantaridis, I., and Hofmann, T. Hidden markov support vector machines, 2003.

[3]

Austin, J. L. How to do things with words. The William James lectures. Harvard University Press, Cambridge, 1962.

[4]

Bird, S., Klein, E., and Loper, E. Natural Language Processing with Python, 1st ed. O'Reilly Media, Inc., 2009.

Digital Library

[5]

Bunt, H., Alexandersson, J., Carletta, J., Choe, J.-W., Fang, A. C., Hasida, K., Lee, K., Petukhova, V., Popescu-Belis, A., Romary, L., Soria, C., and Traum, D. Towards an ISO standard for dialogue act annotation. Seventh conference on International Language Resources and Evaluation (LREC'10) (2010), 2548--2555.

[6]

Core, M. G., and Allen, J. Coding dialogs with the damsl annotation scheme. In AAAI fall symposium on communicative action in humans and machines, vol. 56, Boston, MA (1997).

[7]

Fleiss, J., et al. Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 5 (1971), 378--382.

[8]

Gasic, M., Breslin, C., Henderson, M., Kim, D., Szummer, M., Thomson, B., Tsiakoulis, P., and Young, S. Pomdp-based dialogue manager adaptation to extended domains. In Proceedings of the SIGDIAL 2013 Conference, Association for Computational Linguistics, Association for Computational Linguistics (Metz, France, August 2013), 214--222.

[9]

Herzig, J., Feigenblat, G., Shmueli-Scheuer, M., Konopnicki, D., Rafaeli, A., Altman, D., and Spivak, D. Classifying emotions in customer support dialogues in social media. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics (Los Angeles, September 2016), 64--73.

[10]

Hochreiter, S., and Schmidhuber, J. Long short-term memory. Neural Comput. 9, 8 (Nov. 1997), 1735--1780.

Digital Library

[11]

Honeycutt, C., and Herring, S. C. Beyond microblogging: Conversation and collaboration via twitter. In Proceedings of the Forty-Second Hawai'i International Conference on System Sciences (HICSS-42). Los Alamitos, CA., IEEE Computer Society (Los Alamitos, CA, USA, 2009), 1--10.

Digital Library

[12]

Ivanovic, E. Dialogue act tagging for instant messaging chat sessions. Proceedings of the ACL Student Research Workshop on - ACL '05, June (2005), 79.

Digital Library

[13]

Ivanovic, E. Using Dialogue Acts to Suggest Responses in Support Services via Instant Messaging. Technology (2006), 159--160.

[14]

Ivanovic, E. Automatic instant messaging dialogue using statistical models and dialogue acts. Masters Research thesis (2008).

[15]

Jurafsky, D., Shriberg, E., and Biasca, D. Switchboard-damsl labeling project coder's manual, draft 13. technical report 97-02, university of colorado, institute of cognitive science. boulder, co., 1997.

[16]

Kim, S. N., Cavedon, L., and Baldwin, T. Classifying Dialogue Acts in Multi-party Live Chats. 26th Pacific Asia Conference on Language, Information and Computation (2012), 463--472.

[17]

Kim, S. N., Cavedon, L., and Baldwin, T. Classifying Dialogue Acts in One-on-one Live Chats. 26th Pacific Asia Conference on Language, Information and Computation, October (2012), 463--472.

[18]

Klüwer, T., Uszkoreit, H., and Xu, F. Using syntactic and semantic based relations for dialogue act recognition. Proceedings of COLING 2010, August (2010), 570--578.

Digital Library

[19]

Lafferty, J. D., McCallum, A., and Pereira, F. C. N. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, Morgan Kaufmann Publishers Inc. (San Francisco, CA, USA, 2001), 282--289.

Digital Library

[20]

Landis, J. R., and Koch, G. G. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977).

[21]

Manuvinakurike, R., Paetzel, M., Qu, C., Schlangen, D., and DeVault, D. Toward incremental dialogue act segmentation in fast-paced interactive dialogue systems. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics (Los Angeles, September 2016), 252--262.

[22]

Mohammad, S. M., and Turney, P. D. Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, CAAGET '10, Association for Computational Linguistics (Stroudsburg, PA, USA, 2010), 26--34.

Digital Library

[23]

Morelli, R. A., Bronzino, J. D., and Goethe, J. W. A computational speech-act model of human-computer conversations. In Bioengineering Conference, 1991., Proceedings of the 1991 IEEE Seventeenth Annual Northeast (Apr 1991), 263--264.

[24]

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.

Digital Library

[25]

Puterman, M. L. Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st ed. John Wiley & Sons, Inc., NY, NY, USA, 1994.

Digital Library

[26]

Ritter, A., Cherry, C., and Dolan, B. Unsupervised modeling of twitter conversations. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Association for Computational Linguistics (Los Angeles, CA, June 2010), 172--180.

Digital Library

[27]

Sacks, H., and Jefferson, G. Lectures on Conversation. No. v. 1 in Harvey Sacks : Lectures on Conversation. Blackwell, 1992.

[28]

Schiffrin, A. Modelling Speech Acts in Conversational Discourse.

[29]

Searle, J. R. Language Mind and Knowledge Minnesota Studies. in the Philosophy of Science chapter A Taxonomy of Illocutionary Acts. University of Minnesota Press, 1975.

[30]

Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Ess-Dykema, C., and Meteer, M. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics 26, 3 (2000), 339--373.

Digital Library

[31]

Sun, M., Chen, Y.-N., and Rudnicky, A. I. An intelligent assistant for high-level task understanding. In Proceedings of the 21st International Conference on Intelligent User Interfaces, ACM (2016), 169--174.

Digital Library

[32]

Tur, G., Guz, U., and Hakkani-Tür, D. Model adaptation for dialog act tagging. 2006 IEEE ACL Spoken Language Technology Workshop, SLT 2006, Proceedings (2006), 94--97.

[33]

Vosoughi, S., and Roy, D. A Semi-automatic Method for Efficient Detection of Stories on Social Media. 2013--2016.

[34]

Vosoughi, S., and Roy, D. Tweet acts: A speech act classifier for twitter. arXiv preprint arXiv:1605.05156 (2016).

[35]

Zarisheva, E., and Scheffler, T. Dialog Act Annotation for Twitter Conversations. 114--123.

[36]

Zhang, R., Gao, D., and Li, W. What Are Tweeters Doing: Recognizing Speech Acts in Twitter. Analyzing Microtext (2011), 86--91.

Digital Library

Cited By

Huang SWu JWu Y(2024)The Social Stage of Responses: Social Intent Detection in Discussion Threads Using Deep Learning Model2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650297(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650297
Jia QLiu YRen SZhu K(2023)Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches, and Future DirectionsACM Computing Surveys10.1145/362293356:3(1-38)Online publication date: 5-Oct-2023
https://dl.acm.org/doi/10.1145/3622933
Hellman JChen JUddin MCheng JGuo JBaldassarre MPrechelt LRodríguez-Pérez G(2022)Characterizing user behaviors in open-source software user forumsProceedings of the 15th International Conference on Cooperative and Human Aspects of Software Engineering10.1145/3528579.3529178(46-55)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3528579.3529178
Show More Cited By

Index Terms

"How May I Help You?": Modeling Twitter Customer ServiceConversations Using Fine-Grained Dialogue Acts
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces

Recommendations

Modeling and Computational Characterization of Twitter Customer Service Conversations
Special Issue on Highlights of ACM IUI 2017

Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understanding trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we ...
Haru He's Here to Help!: A Demonstration of Implementing Comedic Rapid Turn-taking for a Social Robot
HRI '23: Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction

Current robot dialog systems are predominantly implemented using a sequential, utterance based, two-party, speak-wait/speak-wait approach. This is inadequate for any interaction where rapid timed turn-taking is required. In this demo we show how a ...
(Semi-)Automatic Analysis of Dialogues
ICAART 2014: Proceedings of the 6th International Conference on Agents and Artificial Intelligence - Volume 1

We study human-human dialogues and human-computer dialogues with the aim to determine which dialogue acts and communicative strategies do the participants of interaction use, and which structural parts does a dialogue include. In order to simplify the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '17: Proceedings of the 22nd International Conference on Intelligent User Interfaces

March 2017

654 pages

ISBN:9781450343480

DOI:10.1145/3025171

General Chairs:
George A. Papadopoulos
The University of Cyprus, Cyprus
,
Tsvi Kuflik
The University of Haifa, Israel
,
Program Chairs:
Fang Chen
Data61, CSIRO Australia
,
Carlos Duarte
The University of Lisbon, Portugal
,
Wai-Tat Fu
The University of Illinois at Urbana-Champaign, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 March 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IUI'17

Sponsor:

IUI'17: 22nd International Conference on Intelligent User Interfaces

March 13 - 16, 2017

Limassol, Cyprus

Acceptance Rates

IUI '17 Paper Acceptance Rate 63 of 272 submissions, 23%;

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
586
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)9

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Huang SWu JWu Y(2024)The Social Stage of Responses: Social Intent Detection in Discussion Threads Using Deep Learning Model2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650297(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650297
Jia QLiu YRen SZhu K(2023)Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches, and Future DirectionsACM Computing Surveys10.1145/362293356:3(1-38)Online publication date: 5-Oct-2023
https://dl.acm.org/doi/10.1145/3622933
Hellman JChen JUddin MCheng JGuo JBaldassarre MPrechelt LRodríguez-Pérez G(2022)Characterizing user behaviors in open-source software user forumsProceedings of the 15th International Conference on Cooperative and Human Aspects of Software Engineering10.1145/3528579.3529178(46-55)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3528579.3529178
Basu SDas SKolya A(2022)An End-to-End Topic-Based Sentiment Analysis Framework from Twitter Using Feature Set CumulationProceedings of International Conference on Industrial Instrumentation and Control10.1007/978-981-16-7011-4_27(267-276)Online publication date: 15-Feb-2022
https://doi.org/10.1007/978-981-16-7011-4_27
Singh TOlivares SCohen TCobb NWang JFranklin AMyneni S(2021)Pragmatics to Reveal Intent in Social Media Peer Interactions: A Mixed-Methods Study (Preprint)Journal of Medical Internet Research10.2196/32167Online publication date: 16-Jul-2021
https://doi.org/10.2196/32167
Elliott-Maksymowicz KNikolaev APorpora D(2021)How much can you say in a tweet? An approach to political argumentation on TwitterHumanities and Social Sciences Communications10.1057/s41599-021-00794-x8:1Online publication date: 14-May-2021
https://doi.org/10.1057/s41599-021-00794-x
Barz MSonntag D(2021)Incremental Improvement of a Question Answering System by Re-ranking Answer Candidates Using Machine LearningIncreasing Naturalness and Flexibility in Spoken Dialogue Interaction10.1007/978-981-15-9323-9_34(367-379)Online publication date: 11-Mar-2021
https://doi.org/10.1007/978-981-15-9323-9_34
Ray ABala PJain R(2020)Utilizing emotion scores for improving classifier performance for predicting customer's intended ratings from social media postsBenchmarking: An International Journal10.1108/BIJ-01-2020-0004ahead-of-print:ahead-of-printOnline publication date: 22-Sep-2020
https://doi.org/10.1108/BIJ-01-2020-0004
Akhyani FKomeili Birjandi ASheikh RSana S(2020)New approach based on proximity/remoteness measurement for customer classificationElectronic Commerce Research10.1007/s10660-020-09402-722:2(267-298)Online publication date: 14-Feb-2020
https://doi.org/10.1007/s10660-020-09402-7
Shah KMare SAnderson RChandwani RSingh PKumar NVeeraraghavan R(2019)Understanding mobile money grievances from tweetsProceedings of the Tenth International Conference on Information and Communication Technologies and Development10.1145/3287098.3287123(1-6)Online publication date: 4-Jan-2019
https://dl.acm.org/doi/10.1145/3287098.3287123
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents