research-article

Supporting Comment Moderators in Identifying High Quality Online News Comments

Authors:

Simranjit Sachar,

Nicholas Diakopoulos, and

Niklas ElmqvistAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

Pages 1114 - 1125

https://doi.org/10.1145/2858036.2858389

Published: 07 May 2016 Publication History

Abstract

Online comments submitted by readers of news articles can provide valuable feedback and critique, personal views and perspectives, and opportunities for discussion. The varying quality of these comments necessitates that publishers remove the low quality ones, but there is also a growing awareness that by identifying and highlighting high quality contributions this can promote the general quality of the community. In this paper we take a user-centered design approach towards developing a system, CommentIQ, which supports comment moderators in interactively identifying high quality comments using a combination of comment analytic scores as well as visualizations and flexible UI components. We evaluated this system with professional comment moderators working at local and national news outlets and provide insights into the utility and appropriateness of features for journalistic tasks, as well as how the system may enable or transform journalistic practices around online comments.

Supplementary Material

ZIP File (pn1744-file4.zip)

Download
.98 KB

References

[1]

Ashley A. Anderson, Dominique Brossard, Dietram A. Scheufele, Michael A. Xenos, and Peter Ladwig. 2014. The "Nasty Effect:" Online Incivility and Risk Perceptions of Emerging Technologies. Journal of Computer-Mediated Communication 19, 3, 373--387. http://doi.org/10.1111/jcc4.12009

Digital Library

[2]

Saeideh Bakhshi, Partha Kanuparthy, and David A. Shamma. 2015. Understanding Online Reviews: Funny, Cool or Useful? Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, 1270--1276. http://doi.org/10.1145/2675133.2675275

Digital Library

[3]

Dirk Brand and Brink Van Der Merwe. 2014. Comment classification for an online news domain. Retrieved January 18, 2015 from https://scholar.sun.ac.za/handle/10019.1/96148

[4]

M. Brehmer, S. Ingram, J. Stray, and T. Munzner. 2014. Overview: The Design, Adoption, and Analysis of a Visual Document Mining Tool for Investigative Journalists. IEEE Transactions on Visualization and Computer Graphics 20, 12, 2271--2280. http://doi.org/10.1109/TVCG.2014.2346431

[5]

Dan Brown. 2006. Communicating Design: Developing Web Site Documentation for Design and Planning. New Riders Publishing, Thousand Oaks, CA, USA.

Digital Library

[6]

Kevin Coe, Kate Kenski, and Stephen A. Rains. 2014. Online and Uncivil Patterns and Determinants of Incivility in Newspaper Website Comments. Journal of Communication 64, 4, 658--679. http://doi.org/10.1111/jcom.12104

[7]

Kushal Dave. 2004. Flash Forums and ForumReader: Navigating a New Kind of Large-Scale Online. Discussion, Proceedings of the ACM Conference on Computer Supported Cooperative Work, 232--241.

Digital Library

[8]

Nicholas Diakopoulos. 2015. The Editor's Eye: Curation and Comment Relevance on the NY Times. Proc. Conference on Computer Supported Cooperative Work (CSCW).

Digital Library

[9]

Nicholas Diakopoulos. 2015. Picking the NYT picks: Editorial criteria and automation in the curation of online news comments. The official research journal of international symposium on online journalism (ISOJ) 5, 1.

[10]

Nicholas Diakopoulos, Stephen Cass, and Joshua Romero. 2014. Data-Driven Rankings: The Design and Development of the IEEE Top Programming Language News App. In Proceedings of the Symposium on Computation + Journalism.

[11]

Nicholas Diakopoulos, Munmun De Choudhury, and Mor Naaman. 2012. Finding and Assessing Social Media Information Sources in the Context of Journalism. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2451--2460. http://doi.org/10.1145/2207676.2208409

Digital Library

[12]

Nicholas Diakopoulos and Mor Naaman. 2011. Towards Quality Discourse in Online News Comments. Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, ACM, 133--142. http://doi.org/10.1145/1958824.1958844

Digital Library

[13]

Bassey Etim. 2014. A Comment's Path to Publication. The NY Times. Retrieved August 8, 2015 from http://www.nytimes.com/times-insider/2014/04/17/acomments-path-to-publication/

[14]

Siamak Faridani, Ephrat Bitton, Kimiko Ryokai, and Ken Goldberg. 2010. Opinion Space: A Scalable Tool for Browsing Online Comments. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 1175--1184. http://doi.org/10.1145/1753326.1753502

Digital Library

[15]

Samuel Gratzl, Alexander Lex, Nils Gehlenborg, Hanspeter Pfister, and Marc Streit. 2013. LineUp: Visual Analysis of Multi-Attribute Rankings. IEEE Transactions on Visualization and Computer Graphics 19, 12, 2277--2286. http://doi.org/10.1109/TVCG.2013.173

Digital Library

[16]

Enamul Hoque and Giuseppe Carenini. 2015. ConVisIT: Interactive Topic Modeling for Exploring Asynchronous Online Conversations. Proceedings of the 20th International Conference on Intelligent User Interfaces, ACM, 169--180. http://doi.org/10.1145/2678025.2701370

Digital Library

[17]

Chiao-Fang Hsu, Elham Khabiri, and James Caverlee. 2009. Ranking Comments on the Social Web. Proceedings of the 2009 International Conference on Computational Science and Engineering - Volume 04, IEEE Computer Society, 90--97. http://doi.org/10.1109/CSE.2009.109

Digital Library

[18]

Karin Wahl Jorgensen. 2002. Understanding the Conditions for Public Discourse: four rules for selecting letters to the editor. Journalism Studies 3, 1. http://doi.org/10.1080/14616700120107347

[19]

Cliff A.C. Lampe, Erik Johnston, and Paul Resnick. 2007. Follow the Reader: Filtering Comments on Slashdot. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 1253--1262. http://doi.org/10.1145/1240624.1240815

Digital Library

[20]

Cliff Lampe and Paul Resnick. 2004. Slash(Dot) and Burn: Distributed Moderation in a Large Online Conversation Space. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 543--550. http://doi.org/10.1145/985692.985761

Digital Library

[21]

G. Harry Mc Laughlin. 1969. SMOG Grading-a New Readability Formula. Journal of Reading 12, 8, 639--646.

[22]

Annie Louis and Ani Nenkova. What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain.

[23]

Kathleen McElroy. 2013. Where Old (Gatekeepers) Meets New (Media). Journalism Practice 7, 6, 755--771. http://doi.org/10.1080/17512786.2013.774117

[24]

Kevin K. Nam and Mark S. Ackerman. 2007. Arkose: Reusing Informal Information from Online Discussions. Proceedings of the 2007 International ACM Conference on Supporting Group Work, ACM, 137--146. http://doi.org/10.1145/1316624.1316644

Digital Library

[25]

Emily Pitler and Ani Nenkova. 2008. Revisiting Readability: A Unified Framework for Predicting Text Quality. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 186--195.

Digital Library

[26]

Vladimir V. Putin. 2013. What Putin Has to Say to Americans About Syria. The NY Times. Retrieved August 8, 2015 from http://www.nytimes.com/2013/09/12/opinion/putin-pleafor-caution-from-russia-on-syria.html

[27]

Bill Reader. 2007. Air Mail: NPR Sees "Community" in Letters From Listeners. Journal of Broadcasting & Electronic Media 51, 4, 651--669. http://doi.org/10.1080/08838150701626529

[28]

Raz Schwartz, Mor Naaman, and Rannie Teodoro. 2015. Editorial Algorithms: Using Social Media to Discover and Report Local News. Ninth International AAAI Conference on Web and Social Media.

[29]

M. Sedlmair, M. Meyer, and T. Munzner. 2012. Design Study Methodology: Reflections from the Trenches and the Stacks. IEEE Transactions on Visualization and Computer Graphics 18, 12, 2431--2440. http://doi.org/10.1109/TVCG.2012.213

Digital Library

[30]

Jane B. Singer. 2010. Quality Control. Journalism Practice 4, 2, 127--142. http://doi.org/10.1080/17512780903391979

[31]

Sara Owsley Sood, Elizabeth F. Churchill, and Judd Antin. 2012. Automatic identification of personal insults on social news sites. Journal of the American Society for Information Science and Technology 63, 2, 270--285. http://doi.org/10.1002/asi.21690

Digital Library

[32]

Natalie Jomini Stroud, Joshua M. Scacco, Ashley Muddiman, and Alexander L. Curry. 2015. Changing Deliberative Norms on News Organizations? Facebook Sites. Journal of Computer-Mediated Communication 20, 2, 188--203. http://doi.org/10.1111/jcc4.12104

[33]

Abhay Sukumaran, Stephanie Vezich, Melanie McHugh, and Clifford Nass. 2011. Normative Influences on Thoughtful Online Participation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 3401--3410. http://doi.org/10.1145/1978942.1979450

Digital Library

[34]

Nayer Wanas, Motaz El-Saban, Heba Ashour, and Waleed Ammar. 2008. Automatic Scoring of Online Discussion Posts. Proceedings of the 2Nd ACM Workshop on Information Credibility on the Web, ACM, 19--26. http://doi.org/10.1145/1458527.1458534

Digital Library

[35]

Kevin Wise, Brian Hamman, and Kjerstin Thorson. 2006. Moderation, Response Rate, and Message Interactivity: Features of Online Communities and Their Effects on Intent to Participate. Journal of ComputerMediated Communication 12, 1, 24--41. http://doi.org/10.1111/j.1083--6101.2006.00313.x

Cited By

Nishal SSinchai JDiakopoulos N(2024)Understanding Practices around Computational News Discovery Tools in the Domain of Science JournalismProceedings of the ACM on Human-Computer Interaction10.1145/36374198:CSCW1(1-36)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637419
Wentzel ALevine LDhariwal VFatemi ZBhattacharya AEugenio BRojecki AZheleva EMarai G(2024)MOTIV: Visual Exploration of Moral Framing in Social MediaComputer Graphics Forum10.1111/cgf.15072Online publication date: 28-Mar-2024
https://doi.org/10.1111/cgf.15072
Wu QSemaan B(2023)"How Do You Quantify How Racist Something Is?": Color-Blind Moderation in Decentralized GovernanceProceedings of the ACM on Human-Computer Interaction10.1145/36100307:CSCW2(1-27)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610030
Show More Cited By

Index Terms

Supporting Comment Moderators in Identifying High Quality Online News Comments
1. Information systems
  1. Information retrieval

Recommendations

Towards quality discourse in online news comments
CSCW '11: Proceedings of the ACM 2011 conference on Computer supported cooperative work

With the growth in sociality and interaction around online news media, news sites are increasingly becoming places for communities to discuss and address common issues spurred by news articles. The quality of online news comments is of importance to ...
Read More
Highlighting High-quality Content as a Moderation Strategy: The Role of New York Times Picks in Comment Quality and Engagement
News commenting is a prevalent form of online interaction, but it is fraught with issues, such as a low quality of discussion that often takes place. While various moderation methods can be used to maintain online discussion quality, one moderation ...
Read More
Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator

What one may say on the internet is increasingly controlled by a mix of automated programs, and decisions made by paid and volunteer human moderators. On the popular social media site Reddit, moderators heavily rely on a configurable, automated program ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI PLAY '24

Sponsor:
sigchi

The Annual Symposium on Computer-Human Interaction in Play

October 14 - 17, 2024

Tampere , Finland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

66
Total Citations
View Citations
1,087
Total Downloads

Downloads (Last 12 months)100
Downloads (Last 6 weeks)7

Other Metrics

View Author Metrics

Citations

Cited By

Nishal SSinchai JDiakopoulos N(2024)Understanding Practices around Computational News Discovery Tools in the Domain of Science JournalismProceedings of the ACM on Human-Computer Interaction10.1145/36374198:CSCW1(1-36)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637419
Wentzel ALevine LDhariwal VFatemi ZBhattacharya AEugenio BRojecki AZheleva EMarai G(2024)MOTIV: Visual Exploration of Moral Framing in Social MediaComputer Graphics Forum10.1111/cgf.15072Online publication date: 28-Mar-2024
https://doi.org/10.1111/cgf.15072
Wu QSemaan B(2023)"How Do You Quantify How Racist Something Is?": Color-Blind Moderation in Decentralized GovernanceProceedings of the ACM on Human-Computer Interaction10.1145/36100307:CSCW2(1-27)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610030
Shahid FVashistha A(2023)Decolonizing Content Moderation: Does Uniform Global Community Standard Resemble Utopian Equality or Western Power Hegemony?Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581538(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581538
Song JLee SLee JKim MKim J(2023)ModSandbox: Facilitating Online Community Moderation Through Error Prediction and Improvement of Automated RulesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581057(1-20)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581057
Liu CZhou SLiu DLi JHuang ZMa X(2023)CoArgue : Fostering Lurkers’ Contribution to Collective Arguments in Community-based QA PlatformsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580932(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580932
Jiang JNie PBrubaker JFiesler C(2023)A Trade-off-centered Framework of Content ModerationACM Transactions on Computer-Human Interaction10.1145/353492930:1(1-34)Online publication date: 7-Mar-2023
https://dl.acm.org/doi/10.1145/3534929
Aitamurto TBorges-Rey EDiakopoulos N(2023)The Future of Design + Journalism: A Manifesto for Bridging Digital Journalism and DesignDigital Journalism10.1080/21670811.2023.218564911:3(399-410)Online publication date: 18-Apr-2023
https://doi.org/10.1080/21670811.2023.2185649
Prem EKrenn B(2023)On Algorithmic Content ModerationIntroduction to Digital Humanism10.1007/978-3-031-45304-5_30(481-493)Online publication date: 21-Dec-2023
https://doi.org/10.1007/978-3-031-45304-5_30
Liu CHuang ZLiu DZhou SPeng ZMa X(2022)PlanHelperProceedings of the ACM on Human-Computer Interaction10.1145/35555556:CSCW2(1-26)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555555
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents