research-article

Protecting chatbots from toxic content

Authors:

Guillaume Baudart,

Evelyn Duesterwald,

Avraham ShinnarAuthors Info & Claims

Onward! 2018: Proceedings of the 2018 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software

Pages 99 - 110

https://doi.org/10.1145/3276954.3276958

Published: 24 October 2018 Publication History

Abstract

There is a paradigm shift in web-based services towards conversational user interfaces. Companies increasingly offer conversational interfaces, or chatbots, to let their customers or employees interact with their services in a more flexible and mobile manner. Unfortunately, this new paradigm faces a major problem, namely toxic content in user inputs. Toxic content in user inputs to chatbots may cause privacy concerns, may be adversarial or malicious, and can cause the chatbot provider substantial economic, reputational, or legal harm. We address this problem with an interdisciplinary approach, drawing upon programming languages, cloud computing, and other disciplines to build protections for chatbots. Our solution, called BotShield, is non-intrusive in that it does not require changes to existing chatbots or underlying conversational platforms. This paper introduces novel security mechanisms, articulates their security guarantees, and illustrates them via case studies.

References

[1]

Amazon. 2014. Lambda. (2014). https://aws.amazon.com/lambda/ (Retrieved June 2018).

[2]

Ioana Baldini, Perry Cheng, Stephen J. Fink, Nick Mitchell, Vinod Muthusamy, Rodric Rabbah, Philippe Suter, and Olivier Tardieu. 2017. The Serverless Trilemma: Function Composition for Serverless Computing. In Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!). 89–103.

Digital Library

[3]

Emery D. Berger, Ting Yang, Tongping Liu, and Gene Novark. 2009. Grace: Safe Multithreaded Programming for C/C++. In Conference on ObjectOriented Programming, Systems, Languages, and Applications (OOPSLA). 81–96.

Digital Library

[4]

Gavin Bierman, Martín Abadi, and Mads Torgersen. 2014. Understanding TypeScript. In European Conference for Object-Oriented Programming (ECOOP). 257–281.

Digital Library

[5]

Daniel G. Bobrow, Ronald M. Kaplan, Martin Kay, Donald A. Norman, Henry Thompson, and Terry Winograd. 1977. GUS, a Frame-Driven Dialog System. Artificial Intelligence 8, 2 (1977), 155–173.

Digital Library

[6]

Herbert H. Clark and Susan E. Brennan. 1991. Grounding in Communication. Perspectives on Socially Shared Cognition 13 (1991), 127–149.

[7]

C. Cowan, F. Waggle, and Calton Pu. 2000. Buffer Overflows: Attacks and Defenses for the Vulnerability of the Decade. In DARPA Information Survivability Conference and Exposition (DISCEX). 119–129.

[8]

Yao Dong, Ana Milanova, and Julian Dolby. 2016. JCrypt: Towards Computation over Encrypted Data. In Conference on Principles and Practices of Programming in Java (PPPJ). 8:1–8:12.

Digital Library

[9]

Saurabh Dutta, Ger Joyce, and Jay Brewer. 2017. Utilizing Chatbots to Increase the Efficacy of Information Security Practitioners. In Conference on Advances in Human Factors in Cybersecurity (AHFE). 237–243.

[10]

Facebook. 2011. Messenger Platform. (2011). https://developers.facebook. com/docs/messenger-platform/ (Retrieved June 2018).

[11]

Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. 2014. Privacy in Pharmacogenetics: An End-toEnd Case Study of Personalized Warfarin Dosing. In USENIX Security Symposium. 17–32.

Digital Library

[12]

Paul T. Graunke, Robert Bruce Findler, Shriram Krishnamurthi, and Matthias Felleisen. 2001. Automatically Restructuring Programs for the Web. In Conference on Automated Software Engineering (ASE). 211–222.

Digital Library

[13]

Salvatore Guarnieri, Marco Pistoia, Omer Tripp, Julian Dolby, Stephen Teilhet, and Ryan Berg. 2011. Saving the World Wide Web from Vulnerable JavaScript. In International Symposium on Software Testing and Analysis (ISSTA). 177–187.

Digital Library

[14]

Philipp Haller, Heather Miller, and Normen Müller. 2018. A Programming Model and Foundation for Lineage-Based Distributed Computation. Journal on Functional Programming (JFP) 28, e7 (2018).

[15]

Hamza Harkous, Kassem Fawaz, Kang G. Shin, and Karl Aberer. 2016. PriBots: Conversational Privacy with Chatbots. In Workshop on the Future of Privacy Indicators (WSF@SOUPS).

[16]

Robert L. Herting Jr and Michael R. Barnes. 1998. Large Scale Database Scrubbing Using Object Oriented Software Components. In American Medical Informatics Association Annual Symposium (AMIA). 508–512.

[17]

Martin Hirzel, Louis Mandel, Avraham Shinnar, Jérôme Siméon, and Mandana Vaziri. 2017. I Can Parse You: Grammars for Dialogs. In Summit oN Advances in Programming Languages (SNAPL). 6:1–6:15.

[18]

IBM. 2008. Cloudant NoSQL Database Service. (2008). https://www.ibm. com/cloud/cloudant (Retrieved June 2018).

[19]

IBM. 2016a. Cloud Functions. (2016). https://www.ibm.com/cloud/functions (Retrieved June 2018).

[20]

IBM. 2016b. Watson Assistant. (2016). https://www.ibm.com/watson/ services/conversation/ (Retrieved June 2018).

[21]

IBM. 2016c. Watson Tone Analyzer Service. (2016). https://www.ibm.com/ watson/services/tone-analyzer/ (Retrieved June 2018).

[22]

Donald Byron Johnson, Stephen M. Matyas, An V. Le, and John D. Wilkins. 1994. The Commercial Data Masking Facility (CDMF) Data Privacy Algorithm. IBM Journal of Research and Development 38, 2 (1994), 217– 226.

Digital Library

[23]

Gregor Kiczales, Erik Hilsdale, Jim Hugunin, Mik Kersten, Jeffrey Palm, and William G. Griswold. 2001. An Overview of AspectJ. In European Conference for Object-Oriented Programming (ECOOP). 327–354.

Digital Library

[24]

Balachander Krishnamurthy and Craig E. Willis. 2009. On the Leakage of Personally Identifiable Information via Online Social Networks. In Workshop on Online Social Networks (WOSN).

Digital Library

[25]

Byeongcheol Lee, Martin Hirzel, Robert Grimm, and Kathryn McKinley. 2009. Debug All Your Code: Portable Mixed-Environment Debugging. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). 207–225.

Digital Library

[26]

Du Li and Witawas Srisa-an. 2011. Quarantine: A Framework to Mitigate Memory Errors for JNI Applications. In Conference on Principles and Practice of Programming in Java (PPPJ). 1–10.

Digital Library

[27]

Bruce Lucas. 2000. VoiceXML for Web-based Distributed Conversational Applications. Communications of the ACM (CACM) 43, 9 (2000), 53–57.

Digital Library

[28]

Erika McCallister, Timothy Grance, and Karen A. Scarfone. 2010. Guide to Protecting the Confidentiality of Personally Identifiable Information (PII). National Institute of Standards and Technology Special Publication (NIST-SP) 800-122 (2010).

Digital Library

[29]

Scott McGlashan, Daniel C. Burnett, Jerry Carter, Peter Danielsen, Jim Ferrans, Andrew Hunt, Bruce Lucas, Brad Porter, Ken Rehor, and Steph Tryphonas. 2004. Voice Extensible Markup Language (VoiceXML) Version 2.0. (2004). https://www.w3.org/TR/voicexml20/ (Retrieved June 2018).

[30]

Michael F. McTear. 2002. Spoken Dialogue Technology: Enabling the Conversational Interface. ACM Computing Surveys (CSUR) 34, 1 (2002), 90–169.

Digital Library

[31]

Microsoft. 2015. Bot Framework Documentation. (2015). https://azure. microsoft.com/en-us/services/bot-service/ (Retrieved June 2018).

[32]

Microsoft. 2016. Azure Functions. (2016). https://functions.azure.com/ (Retrieved June 2018).

[33]

George C. Necula, Scott McPeak, and Westley Weimer. 2002. CCured: Type-Safe Retrofitting of Legacy Code. In Symposium on Principles of Programming Languages (POPL). 128–139.

Digital Library

[34]

Nicholas Nethercote and Julian Seward. 2007. Valgrind: A Framework for Heavyweight Dynamic Binary Instrumentation. In Conference on Programming Language Design and Implementation (PLDI). 89–100.

Digital Library

[35]

Sam Newman. 2015. Building Microservices: Designing Fine Grained Systems. O’Reilly.

Digital Library

[36]

Amit Patil, K. Marimuthu, and R. Niranchana. 2017. Comparative Study of Cloud Platforms to Develop a Chatbot. International Journal of Engineering & Technology 6, 3 (2017), 57–61.

[37]

Jason Reed and Benjamin C. Pierce. 2010. Distance Makes the Types Grow Stronger: A Calculus for Differential Privacy. In International Conference on Functional Programming (ICFP). 157–168.

Digital Library

[38]

Salesforce. 2016. Block Sensitive Data in Chats. (2016). https: //releasenotes.docs.salesforce.com/en-us/winter16/release-notes/rn_ live_agent_block_sensitive_data.htm (Retrieved June 2018).

[39]

Joseph Siefers, Gang Tan, and Greg Morrisett. 2010. Robusta: Taming the Native Beast of the JVM. In Conference on Computer and Communication Security (CCS). 201–211.

Digital Library

[40]

Peri Tarr, Harold Ossher, William Harrison, and Stanley M. Sutton, Jr. 1999. N Degrees of Separation: Multi-dimensional Separation of Concerns. In International Conference on Software Engineering (ICSE). 107–119.

Digital Library

[41]

Sai Deep Tetali, Mohsen Lesani, Rupak Majumdar, and Todd Millstein. 2013. MrCrypt: Static Analysis for Secure Cloud Computations. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). 271–286.

Digital Library

[42]

Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing Machine Learning Models via Prediction APIs. In USENIX Security Symposium. 601–618.

Digital Library

[43]

Mandana Vaziri, Louis Mandel, Avraham Shinnar, Jérôme Siméon, and Martin Hirzel. 2017. Generating Chat Bots from Web API Specifications. In Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!). 44–57.

Digital Library

[44]

Larry Wall, Tom Christiansen, and Jon Orwant. 2000. Programming Perl (third ed.). O’Reilly.

Digital Library

[45]

Wikipedia. 2016. Tay (bot). (2016). https://en.wikipedia.org/wiki/Tay_(bot) (Retrieved June 2018).

[46]

Saman T. Zargar, James Joshi, and David Tipper. 2013. A Survey of Defense Mechanisms Against Distributed Denial of Service (DDoS) Flooding Attacks. IEEE Communications Surveys and Tutorials 15, 4 (2013), 2046– 2069.

Cited By

Giordani J(2024)Mitigating Chatbots AI Data Privacy Violations in the Banking Sector: A Qualitative Grounded Theory StudyEuropean Journal of Applied Science, Engineering and Technology10.59324/ejaset.2024.2(4).022:4(14-65)Online publication date: 1-Jul-2024
https://doi.org/10.59324/ejaset.2024.2(4).02
Szmurlo HAkhtar Z(2024)Digital Sentinels and Antagonists: The Dual Nature of Chatbots in CybersecurityInformation10.3390/info1508044315:8(443)Online publication date: 29-Jul-2024
https://doi.org/10.3390/info15080443
Zainabi BMaleh YBelaissaoui M(2024)Security risks of chatbots in customer service: a comprehensive literature reviewEDPACS10.1080/07366981.2024.237893469:7(51-66)Online publication date: 15-Jul-2024
https://doi.org/10.1080/07366981.2024.2378934
Show More Cited By

Index Terms

Protecting chatbots from toxic content
1. Security and privacy
  1. Human and societal aspects of security and privacy
    1. Privacy protections
2. Software and its engineering
  1. Software notations and tools
    1. Context specific languages
      1. Domain specific languages

Recommendations

Evaluating and Informing the Design of Chatbots
DIS '18: Proceedings of the 2018 Designing Interactive Systems Conference

Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not ...
Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings – Experiences from a Field Study
Chatbot Research and Design
Abstract
In this paper, we analyze the use of small talk conversations based on a dialogue analysis of a long-term field study in which university students regularly interacted with a chatbot during a 3-month period of time in an educational setting. In ...
Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots
RAID '23: Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses

Recent advances in natural language processing and machine learning have led to the development of chatbot models, such as ChatGPT, that can engage in conversational dialogue with human users. However, understanding the ability of these models to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

Onward! 2018: Proceedings of the 2018 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software

October 2018

146 pages

ISBN:9781450360319

DOI:10.1145/3276954

General Chairs:
Elisa Gonzalez Boix
Vrije Universiteit Brussel, Belgium
,
Richard P. Gabriel
Dream Songs, USA / HPI, Germany

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SPLASH '18

Sponsor:

SIGPLAN

SPLASH '18: Conference on Systems, Programming, Languages, and Applications: Software for Humanity

November 7 - 8, 2018

MA, Boston, USA

Acceptance Rates

Overall Acceptance Rate 40 of 105 submissions, 38%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
440
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)5

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Giordani J(2024)Mitigating Chatbots AI Data Privacy Violations in the Banking Sector: A Qualitative Grounded Theory StudyEuropean Journal of Applied Science, Engineering and Technology10.59324/ejaset.2024.2(4).022:4(14-65)Online publication date: 1-Jul-2024
https://doi.org/10.59324/ejaset.2024.2(4).02
Szmurlo HAkhtar Z(2024)Digital Sentinels and Antagonists: The Dual Nature of Chatbots in CybersecurityInformation10.3390/info1508044315:8(443)Online publication date: 29-Jul-2024
https://doi.org/10.3390/info15080443
Zainabi BMaleh YBelaissaoui M(2024)Security risks of chatbots in customer service: a comprehensive literature reviewEDPACS10.1080/07366981.2024.237893469:7(51-66)Online publication date: 15-Jul-2024
https://doi.org/10.1080/07366981.2024.2378934
Bhutada SUppala SNagavajyula SPottigari T(2023) C 2 -PILS: A Chatbot using Chat-GPT for Pharma Industry and Life Sciences 2023 International Conference on Advanced Computing Technologies and Applications (ICACTA)10.1109/ICACTA58201.2023.10392641(1-6)Online publication date: 6-Oct-2023
https://doi.org/10.1109/ICACTA58201.2023.10392641
Marjerison RZhang YZheng H(2022)AI in E-Commerce: Application of the Use and Gratification Model to The Acceptance of ChatbotsSustainability10.3390/su14211427014:21(14270)Online publication date: 1-Nov-2022
https://doi.org/10.3390/su142114270
Scott EPanda SLoukas GPanaousis E(2022)Optimising user security recommendations for AI-powered smart-homes2022 IEEE Conference on Dependable and Secure Computing (DSC)10.1109/DSC54232.2022.9888829(1-8)Online publication date: 22-Jun-2022
https://doi.org/10.1109/DSC54232.2022.9888829
Jangda APinckney DBrun YGuha A(2019)Formal foundations of serverless computingProceedings of the ACM on Programming Languages10.1145/33605753:OOPSLA(1-26)Online publication date: 10-Oct-2019
https://dl.acm.org/doi/10.1145/3360575

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents