research-article

Sentiment and politeness analysis tools on developer discussions are unreliable, but so are people

Authors:

Justin Middleton,

Peter Girouard,

Emerson Murphy-HillAuthors Info & Claims

SEmotion '18: Proceedings of the 3rd International Workshop on Emotion Awareness in Software Engineering

Pages 55 - 61

https://doi.org/10.1145/3194932.3194938

Published: 02 June 2018 Publication History

Abstract

Many software engineering researchers use sentiment and politeness analysis tools to study the emotional environment within collaborative software development. However, papers that use these tools rarely establish their reliability. In this paper, we evaluate popular existing tools for sentiment and politeness detection over a dataset of 589 manually rated GitHub comments that represent developer discussions. We also develop a coding scheme on how to quantify politeness for conversational texts found on collaborative platforms. We find that not only do the tools have a low agreement with human ratings on sentiment and politeness, human raters also have a low agreement among themselves.

References

[1]

Toufique Ahmed, Amiangshu Bosu, Anindya Iqbal, and Shahram Rahimi. 2017. SentiCR: a customized sentiment analysis tool for code review interactions. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 106--111.

Digital Library

[2]

Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc.".

Digital Library

[3]

Cássio Castaldi Araujo Blaz and Karin Becker. 2016. Sentiment analysis in tickets for it support. In Mining Software Repositories (MSR), 2016 IEEE/ACM 13th Working Conference on. IEEE, 235--246.

Digital Library

[4]

Chris Brown, Justin Middleton, Esha Sharma, and Emerson Murphy-Hill. {n. d.}. How Software Users Recommend Tools to Each Other. ({n. d.}).

[5]

Penelope Brown and Stephen C Levinson. 1987. Politeness: Some universals in language usage. Vol. 4. Cambridge university press.

[6]

Fabio Calefato, Filippo Lanubile, Federico Maiorano, and Nicole Novielli. 2017. Sentiment Polarity Detection for Software Development. Empirical Software Engineering (2017), 1--31.

Digital Library

[7]

Wikipedia contributors. 2018. Politeness --- Wikipedia, The Free Encyclopedia. (2018). https://en.wikipedia.org/wiki/Politeness {Online; accessed 18 Jan, 2018}.

[8]

Jonathan Culpeper. 1996. Towards an anatomy of impoliteness. Journal of pragmatics 25, 3 (1996), 349--367.

[9]

Bill Curtis, Herb Krasner, and Neil Iscoe. 1988. A field study of the software design process for large systems. Commun. ACM 31, 11 (1988), 1268--1287.

Digital Library

[10]

Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. arXiv preprint arXiv:1306.6078 (2013).

[11]

Prasun Dewan. 2015. Towards emotion-based collaborative software engineering. In Proceedings of the Eighth International Workshop on Cooperative and Human Aspects of Software Engineering. IEEE Press, 109--112.

Digital Library

[12]

Felipe Ebert, Fernando Castor, Nicole Novielli, and Alexander Serebrenik. 2017. Confusion detection in code reviews. In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on. IEEE, 549--553.

[13]

Daviti Gachechiladze, Filippo Lanubile, Nicole Novielli, and Alexander Serebrenik. 2017. Anger and its direction in collaborative software development. In Proceedings of the 39th International Conference on Software Engineering: New Ideas and Emerging Results Track. IEEE Press, 11--14.

Digital Library

[14]

Michael Gamon, Anthony Aue, Simon Corston-Oliver, and Eric Ringger. 2005. Pulse: Mining customer opinions from free text. Lecture notes in computer science 3646 (2005), 121--132.

Digital Library

[15]

David Garcia, Marcelo Serrano Zanetti, and Frank Schweitzer. 2013. The role of emotions in contributors activity: A case study on the gentoo community. In Cloud and green computing (CGC), 2013 third international conference on. IEEE, 410--417.

Digital Library

[16]

Georgios Gousios, Bogdan Vasilescu, Alexander Serebrenik, and Andy Zaidman. 2014. Lean GHTorrent: GitHub data on demand. In Proceedings of the 11th working conference on mining software repositories. ACM, 384--387.

Digital Library

[17]

Daniel Graziotin, Xiaofeng Wang, and Pekka Abrahamsson. 2014. Happy software developers solve problems better: psychological measurements in empirical software engineering. PeerJ 2 (2014), e289.

[18]

Emitza Guzman, David Azócar, and Yang Li. 2014. Sentiment analysis of commit comments in GitHub: an empirical study. In Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, 352--355.

Digital Library

[19]

Emitza Guzman and Bernd Bruegge. 2013. Towards emotional awareness in software development teams. In Proceedings of the 2013 9th joint meeting on foundations of software engineering. ACM, 671--674.

Digital Library

[20]

Md Rakibul Islam and Minhaz F Zibran. 2016. Towards understanding and exploiting developers' emotional variations in software engineering. In Software Engineering Research, Management and Applications (SERA), 2016 IEEE 14th International Conference on. IEEE, 185--192.

[21]

Robbert Jongeling, Proshanta Sarkar, Subhajit Datta, and Alexander Serebrenik. 2017. On negative results when using sentiment analysis tools for software engineering research. Empirical Software Engineering (2017), 1--42.

Digital Library

[22]

Iftikhar Ahmed Khan, Willem-Paul Brinkman, and Robert M Hierons. 2011. Do moods affect programmersâÃ&Zacute; debug performance? Cognition, Technology & Work 13, 4 (2011), 245--258.

Digital Library

[23]

J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159--174.

[24]

Bin Lin, Fiorella Zampetti, Gabriele Bavota, Massimiliano Di Penta, Michele Lanza, and Rocco Oliveto. 2018. Sentiment Analysis for So ware Engineering: How Far Can We Go? (2018).

[25]

Saif Mohammad. 2016. A Practical Guide to Sentiment Annotation: Challenges and Solutions. In WASSA@ NAACL-HLT. 174--179.

[26]

Alessandro Murgia, Parastou Tourani, Bram Adams, and Marco Ortu. 2014. Do developers feel emotions? an exploratory analysis of emotions in software artifacts. In Proceedings of the 11th working conference on mining software repositories. ACM, 262--271.

Digital Library

[27]

Nicole Novielli, Fabio Calefato, and Filippo Lanubile. 2014. Towards discovering the role of emotions in stack overflow. In Proceedings of the 6th international workshop on social software engineering. ACM, 33--36.

Digital Library

[28]

Nicole Novielli, Fabio Calefato, and Filippo Lanubile. 2015. The challenges of sentiment detection in the social programmer ecosystem. In Proceedings of the 7th International Workshop on Social Software Engineering. ACM, 33--40.

Digital Library

[29]

Marco Ortu, Bram Adams, Giuseppe Destefanis, Parastou Tourani, Michele Marchesi, and Roberto Tonelli. 2015. Are bullies more productive?: empirical study of affectiveness vs. issue fixing time. In Proceedings of the 12th Working Conference on Mining Software Repositories. IEEE Press, 303--313.

Digital Library

[30]

Marco Ortu, Giuseppe Destefanis, Mohamad Kassab, Steve Counsell, Michele Marchesi, and Roberto Tonelli. 2015. Would you mind fixing this issue?. In International Conference on Agile Software Development. Springer, 129--140.

[31]

Marco Ortu, Alessandro Murgia, Giuseppe Destefanis, Parastou Tourani, Roberto Tonelli, Michele Marchesi, and Bram Adams. 2016. The emotional side of software developers in JIRA. In 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR). IEEE, 480--483.

Digital Library

[32]

Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends® in Information Retrieval 2, 1--2 (2008), 1--135.

Digital Library

[33]

Brian Parkinson, Peter Totterdell, Rob B Briner, and SA Reynolds. 1996. Changing moods: The psychology of mood and mood regulation. Longman.

[34]

Daniel Pletea, Bogdan Vasilescu, and Alexander Serebrenik. 2014. Security and emotion: sentiment analysis of security discussions on GitHub. In Proceedings of the 11th working conference on mining software repositories. ACM, 348--351.

Digital Library

[35]

Athanasios-Ilias Rousinopoulos, Gregorio Robles, and Jesús M González-Barahona. 2014. SENTIMENT ANALYSIS OF FREE/OPEN SOURCE DEVELOPERS: PRELIMINARY FINDINGS FROM A CASE STUDY/ANÁLISE DE SENTIMENTOS DE DESENVOLVEDORES DE SOFTWARE LIVRE: ACHADOS PRELIMINARES DE UM ESTUDO DE CASO. Revista Electronica de Sistemas de Informaçao 13, 2 (2014), 1.

[36]

Vinayak Sinha, Alina Lazar, and Bonita Sharif. 2016. Analyzing developer sentiment in commit logs. In Proceedings of the 13th International Conference on Mining Software Repositories. ACM, 520--523.

Digital Library

[37]

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing. 1631--1642.

[38]

Igor Steinmacher, Tayana Conte, Marco Aurélio Gerosa, and David Redmiles. 2015. Social barriers faced by newcomers placing their first contribution in open source software projects. In Proceedings of the 18th ACM conference on Computer supported cooperative work & social computing. ACM, 1379--1392.

Digital Library

[39]

Margaret-Anne Storey, Christoph Treude, Arie van Deursen, and Li-Te Cheng. 2010. The impact of social media on software engineering practices and tools. In Proceedings of the FSE/SDP workshop on Future of software engineering research. ACM, 359--364.

Digital Library

[40]

Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai, and Arvid Kappas. 2010. Sentiment strength detection in short informal text. Journal of the Association for Information Science and Technology 61, 12 (2010), 2544--2558.

Digital Library

[41]

Jason Tsay, Laura Dabbish, and James Herbsleb. 2014. Let's talk about it: evaluating contributions through discussion in GitHub. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering. ACM, 144--154.

Digital Library

[42]

Ning Wang, W Lewis Johnson, Richard E Mayer, Paola Rizzo, Erin Shaw, and Heather Collins. 2008. The politeness effect: Pedagogical agents and learning outcomes. International Journal of Human-Computer Studies 66, 2 (2008), 98--112.

Digital Library

Cited By

Mott TFanganello AWilliams TGrollman DBroadbent EJu WSoh HWilliams T(2024)What a Thing to Say! Which Linguistic Politeness Strategies Should Robots Use in Noncompliance Interactions?Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634943(501-510)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634943
Specht AObaidi MNagel LStess MKlünder J(2024)What is Needed to Apply Sentiment Analysis in Real Software Projects: A Feasibility Study in IndustryHuman-Centered Software Engineering10.1007/978-3-031-64576-1_6(105-129)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1007/978-3-031-64576-1_6
Mott TWilliams T(2023)Confrontation and Cultivation: Understanding Perspectives on Robot Responses to Norm Violations2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309577(2336-2343)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309577
Show More Cited By

Recommendations

A query-based multi-document sentiment summarizer
CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge management

Review websites, such as Epinions.com, which offer users a platform to share their opinions on diverse products and services, provide a valuable source of opinion-rich information. Browsing through archived reviews to locate different opinions on a ...
Exploring how politeness impacts the user experience of chatbots for mental health support
Abstract
Politeness is important in human–human interaction when asking people to engage in sensitive conversations. If politeness manifests similarly in human–chatbot interaction, it may play an important role in the design of sensitive chatbot ...
Highlights
- Politeness can both positively and negatively impact the chatbot user experience.
- The Personal politeness chatbot was experienced as caring and encouraging.
- The Passive politeness chatbot was experienced as too apologetic and ...
Effect of politeness strategies in dialogue on negotiation outcomes
IVA '21: Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents

Negotiation is a social interaction aimed at reaching a mutually beneficial agreement among all participants in a conflict situation. Unfortunately, parties often find negotiations threatening or aversive, undermining the chances of reaching good ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SEmotion '18: Proceedings of the 3rd International Workshop on Emotion Awareness in Software Engineering

June 2018

76 pages

ISBN:9781450357517

DOI:10.1145/3194932

Program Chairs:
Andrew Begel
Microsoft Research
,
Alexander Serebrenik
Eindhoven University of Technology, The Netherlands
,
Daniel Graziotin
University of Stuttgart, Germany

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

June 2, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
436
Total Downloads

Downloads (Last 12 months)155
Downloads (Last 6 weeks)38

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mott TFanganello AWilliams TGrollman DBroadbent EJu WSoh HWilliams T(2024)What a Thing to Say! Which Linguistic Politeness Strategies Should Robots Use in Noncompliance Interactions?Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634943(501-510)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634943
Specht AObaidi MNagel LStess MKlünder J(2024)What is Needed to Apply Sentiment Analysis in Real Software Projects: A Feasibility Study in IndustryHuman-Centered Software Engineering10.1007/978-3-031-64576-1_6(105-129)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1007/978-3-031-64576-1_6
Mott TWilliams T(2023)Confrontation and Cultivation: Understanding Perspectives on Robot Responses to Norm Violations2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309577(2336-2343)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309577
Han YWang ZFeng YZhao ZWang Y(2023)Cross-status communication and project outcomes in OSS developmentEmpirical Software Engineering10.1007/s10664-023-10298-828:3Online publication date: 12-May-2023
https://dl.acm.org/doi/10.1007/s10664-023-10298-8
Novielli NSerebrenik A(2023)Emotion Analysis in Software EcosystemsSoftware Ecosystems10.1007/978-3-031-36060-2_5(105-127)Online publication date: 26-May-2023
https://doi.org/10.1007/978-3-031-36060-2_5
Lin BCassee NSerebrenik ABavota GNovielli NLanza M(2022)Opinion Mining for Software Development: A Systematic Literature ReviewACM Transactions on Software Engineering and Methodology10.1145/349038831:3(1-41)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.1145/3490388
Herrmann MObaidi MChazette LKlünder J(2022)On the subjectivity of emotions in software projectsJournal of Systems and Software10.1016/j.jss.2022.111448193:COnline publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1016/j.jss.2022.111448
Obaidi MNagel LSpecht AKlünder J(2022)Sentiment analysis tools in software engineeringInformation and Software Technology10.1016/j.infsof.2022.107018151:COnline publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1016/j.infsof.2022.107018
Schroth LObaidi MSpecht AKlünder J(2022)On the Potentials of Realtime Sentiment Analysis on Text-Based Communication in Software ProjectsHuman-Centered Software Engineering10.1007/978-3-031-14785-2_6(90-109)Online publication date: 24-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-14785-2_6
Obaidi MKlünder J(2021)Development and Application of Sentiment Analysis Tools in Software Engineering: A Systematic Literature ReviewProceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering10.1145/3463274.3463328(80-89)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3463274.3463328
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents