Just Accepted
Hybrid Prompt Learning for Generating Justifications of Security Risks in Automation Rules
Accepted on 29 May 2024
Abstract
Trigger-Action Platforms (TAPs) enable users without programming experience to personalize the behavior of Internet of Things applications and services through IF-THEN rules. Unfortunately, the arbitrary connection of smart devices and online services, even with simple rules such as “IF the entrance Netatmo Wheather Station detects a temperature above 30 \({}^{\circ}C\) ( \(86^{\circ}F\) ) THEN open the shutters in the living room”, might expose users to potential security and privacy risks (e.g., the execution of the previous rule might provide an easy entry point for thieves, especially during the summer vacation period). The goal of our research is to make the users capable of understanding and mitigating the threats and risks associated with the execution of IF-THEN rules. To this end, we define a new challenging task, namely generating post-hoc justifications of privacy and security risks associated with automation rules, and propose a novel natural language generation strategy based on hybrid prompt learning producing justifications in the form of real-life threat scenarios. The proposed strategy allows for prompt customization with task-specific information, providing contextual details enabling to grasp the nuances and subtleties of the domain language, resulting in more coherent justifications. The experiments conducted on the If-This-Then-That (IFTTT) platform show that our method produces effective justifications, improving the explainability of discrete and hybrid prompting methods up to 27% in BLEURT score. The code of the software is publicly available on GitHub.
References
[1]
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. 2019. A convergence theory for deep learning via over-parameterization. In ICML ’19. 242–252.
[2]
Luigi Atzori, Antonio Iera, and Giacomo Morabito. 2010. The internet of things: A survey. Comput. Netw. 54, 15 (2010), 2787–2805.
[3]
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In ACL StatMT ’07. 65–72.
[4]
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. In ICML ’23. 2397–2430.
[5]
Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In XAI ’17 Workshop, Vol. 8. 8–13.
[6]
Bernardo Breve, Gaetano Cimino, and Vincenzo Deufemia. 2022. Towards Explainable Security for ECA Rules. In EMPATHY ’21 Workshop.
[7]
Bernardo Breve, Gaetano Cimino, and Vincenzo Deufemia. 2023. Identifying security and privacy violation rules in trigger-action IoT platforms with NLP models. IEEE IoT J 10, 6 (2023), 5607–5622.
[8]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
[9]
Z Berkay Celik, Patrick McDaniel, and Gang Tan. 2018. Soteria: Automated IoT safety and security analysis. In ATC ’18. 147–158.
[10]
Miruna-Adriana Clinciu, Arash Eshghi, and Helen F. Hastie. 2021. A study of automatic metrics for the evaluation of natural language explanations. In EACL ’21. 2376–2387.
[11]
Camille Cobb, Milijana Surbatovich, Anna Kawakami, Mahmood Sharif, Lujo Bauer, Anupam Das, and Limin Jia. 2020. How risky are real users’ IFTTT applets?. In SOUPS ’20. 505–529.
[12]
Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. J. Appl. Soc. Psychol. 60, 2 (1975), 283.
[13]
Fulvio Corno, Luigi De Russis, and Alberto Monge Roffarello. 2019. RecRules: Recommending IF-THEN rules for end-user development. ACM Trans. Intell. Syst. Technol. 10, 5 (2019).
[14]
Felipe Costa, Sixun Ouyang, Peter Dolog, and Aonghus Lawlor. 2018. Automatic generation of natural language explanations. In IUI ’18. 1–2.
[15]
Giuseppe Desolda, Carmelo Ardito, and Maristella Matera. 2017. Empowering end users to customize their smart environments: model, composition paradigms, and domain-specific tools. ACM Trans. on Comput.-Hum. Interact. 24, 2 (2017), 1–52.
[16]
Desmond Elliott and Frank Keller. 2014. Comparing automatic evaluation measures for image description. In ACL ’14. 452–457.
[17]
Chenglong Fu, Qiang Zeng, and Xiaojiang Du. 2021. HAWatcher:Semantics-aware anomaly detection for appified smart homes. In USENIX Security ’21. 4223–4240.
[18]
Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. 2019. Question answering is a format; When is it useful? arXiv:1909.11291 (2019). Retrieved from https://arxiv.org/abs/1909.11291.
[19]
Giuseppe Ghiani, Marco Manca, Fabio Paternò, and Carmen Santoro. 2017. Personalization of context-dependent applications through trigger-action rules. ACM Trans. on Comput.-Hum. Interact. 24, 2 (2017), 1–33.
[20]
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022. PTR: Prompt tuning with rules for text classification. AI Open 3 (2022), 182–192.
[21]
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP ’21. 3045–3059.
[22]
Lei Li, Yongfeng Zhang, and Li Chen. 2023. Personalized prompt learning for explainable recommendation. ACM Trans. Inf. Syst. 41, 4 (2023), 1–26.
[23]
Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing continuous prompts for generation. In ACL/IJCNLP ’21. 4582–4597.
[24]
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need II: phi-1.5 technical report. arXiv:2309.05463 (2023). Retrieved from https://arxiv.org/abs/2309.05463.
[25]
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1–35.
[26]
Xuanyu Liu, Xiao Fu, Xiaojiang Du, Bin Luo, and Mohsen Guizani. 2022. Machine learning based non-intrusive digital forensic service for smart homes. IEEE Trans. Netw. Serv. Manag. 20, 2 (2022), 945–960.
[27]
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023. GPT understands, too. AI Open (2023).
[28]
Ana Marasovic, Iz Beltagy, Doug Downey, and Matthew E. Peters. 2022. Few-shot self-rationalization with natural language prompts. In NAACL ’22. 410–424.
[29]
Xianghang Mi, Feng Qian, Ying Zhang, and XiaoFeng Wang. 2017. An empirical characterization of IFTTT: ecosystem, usage, and performance. In IMC ’17. 398–404.
[30]
Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoît Sagot, et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP. arXiv:2112.10508 (2021). Retrieved from https://arxiv.org/abs/2112.10508.
[31]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv:1301.3781 (2013). Retrieved from https://arxiv.org/abs/1301.3781.
[32]
Cataldo Musto, Marco de Gemmis, Pasquale Lops, and Giovanni Semeraro. 2021. Generating post hoc review-based natural language justifications for recommender systems. User Model. User-Adapt. Interact. 31, 3 (2021), 629–673.
[33]
Dang Tu Nguyen, Chengyu Song, Zhiyun Qian, Srikanth V Krishnamurthy, Edward JM Colbert, and Patrick McDaniel. 2018. IotSan: Fortifying the safety of IoT systems. In CoNEXT ’18. 191–203.
[34]
Steven Ovadia. 2014. Automate the internet with “if this then that”(IFTTT). Behav Soc Sci Librar 33, 4 (2014), 208–211.
[35]
Federica Paci, Davide Bianchin, Elisa Quintarelli, and Nicola Zannone. 2020. IFTTT privacy checker. In ETAA ’20. 90–107.
[36]
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018. Multimodal explanations: Justifying decisions and pointing to the evidence. In CVPR ’18. 8779–8788.
[37]
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL ’18. 2227–2237.
[38]
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases?. In EMNLP-IJCNLP ’19. 2463–2473.
[39]
Atharva Phatak, David W Savage, Robert Ohle, Jonathan Smith, and Vijay Mago. 2022. Medical text simplification using reinforcement learning (TESLEA): Deep learning–based text simplification approach. JMIR Medical Informatics 10, 11 (2022), e38095.
[40]
Wolter Pieters. 2011. Explanation and trust: what to tell the user in security and AI? Ethics Inf Tech 13, 1 (2011), 53–64.
[41]
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In NAACL-HLT ’21. 5203–5212.
[42]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
[43]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In EMNLP-IJCNLP ’19. 3982–3992.
[44]
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL ’21. 255–269.
[45]
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In ACL ’20. 7881–7892.
[46]
Richard Shin, Christopher H. Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In EMNLP ’21. 7699–7715.
[47]
Milijana Surbatovich, Jassim Aljuraidan, Lujo Bauer, Anupam Das, and Limin Jia. 2017. Some recipes can do more than spoil your appetite: Analyzing the security and privacy risks of IFTTT recipes. In WWW ’17. 1501–1510.
[48]
Blase Ur, Melwyn Pak Yong Ho, Stephen Brawner, Jiyun Lee, Sarah Mennicken, Noah Picard, Diane Schulze, and Michael L Littman. 2016. Trigger-action programming in the wild: An analysis of 200,000 IFTTT recipes. In CHI ’16. 3227–3231.
[49]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS ’17. 6000–6010.
[50]
Luca Viganò and Daniele Magazzeni. 2020. Explainable security. In EuroS&P Workshops ’20. 293–300.
[51]
Guangjing Wang, Hanqing Guo, Anran Li, Xiaorui Liu, and Qiben Yan. 2023. Federated IoT interaction vulnerability analysis. In ICDE ’23. 1517–1530.
[52]
Guangjing Wang, Nikolay Ivanov, Bocheng Chen, Qi Wang, ThanhVu Nguyen, and Qiben Yan. 2023. Graph Learning for Interactive Threat Detection in Heterogeneous Smart Home Rule Data. Proc. ACM Manag. Data 1, 1 (2023), 1–27.
[53]
Qi Wang, Pubali Datta, Wei Yang, Si Liu, Adam Bates, and Carl A. Gunter. 2019. Charting the attack surface of trigger-action IoT platforms. In CCS ’19. 1439–1453.
[54]
Qi Wang, Wajih Ul Hassan, Adam Bates, and Carl Gunter. 2018. Fear and logging in the internet of things. In NDSS Symposium. 1–16.
[55]
Karl Weiss, Taghi Khoshgoftaar, and DingDing Wang. 2016. A survey of transfer learning. J Big Data 3, 1 (2016), 1–40.
[56]
Ding Xiao, Qianyu Wang, Ming Cai, Zhaohui Zhu, and Weiming Zhao. 2019. A3ID: an automatic and interpretable implicit interference detection method for smart home via knowledge graph. IEEE IoT J 7, 3 (2019), 2197–2211.
[57]
Zhouhang Xie, Sameer Singh, Julian J. McAuley, and Bodhisattwa Prasad Majumder. 2023. Factual and informative review generation for explainable recommendation. In AAAI ’23, Brian Williams, Yiling Chen, and Jennifer Neville (Eds.). 13816–13824.
[58]
Yuhang Yao, Mohammad Mahdi Kamani, Zhongwei Cheng, Lin Chen, Carlee Joe-Wong, and Tianqiang Liu. 2023. FedRule: Federated Rule Recommendation System with Graph Neural Networks. In IoTDI ’23. 197–208.
[59]
Raciel Yera, Ahmad A Alzahrani, and Luis Martinez. 2022. Exploring post-hoc agnostic models for explainable cooking recipe recommendations. Knowl. Based Syst. 251 (2022), 109216.
[60]
Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji. 2021. On explainability of graph neural networks via subgraph explorations. In ICML ’21. 12241–12252.
[61]
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. In NeurIPS ’21. 27263–27277.
[62]
Imam Nur Bani Yusuf, Lingxiao Jiang, and David Lo. 2022. Accurate generation of trigger-action programs with domain-adapted sequence-to-sequence learning. In ICPC ’22. 99–110.
[63]
Yongfeng Zhang, Xu Chen, et al. 2020. Explainable recommendation: A survey and new perspectives. Found. Trends Inf. Retr. 14, 1 (2020), 1–101.
[64]
Guoshuai Zhao, Hao Fu, Ruihua Song, Tetsuya Sakai, Zhongxia Chen, Xing Xie, and Xueming Qian. 2019. Personalized reason generation for explainable song recommendation. ACM Trans. Intell. Syst. Technol. 10, 4 (2019), 1–21.
[65]
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and Earth mover distance. In EMNLP-IJCNLP ’19. 563–578.
[66]
Serena Zheng, Noah Apthorpe, Marshini Chetty, and Nick Feamster. 2018. User perceptions of smart home IoT privacy. Proc. ACM on Human-Computer Interaction 2 (2018), 1–20.
Index Terms
- Hybrid Prompt Learning for Generating Justifications of Security Risks in Automation Rules
Recommendations
On the User Perception of Security Risks of TAP Rules: A User Study
End-User DevelopmentAbstractTrigger-Action Platforms (TAPs) provide users with enhanced control to automate interactions between IoT devices using rules that consist of trigger conditions and actions that get executed when the triggers are fired. To better describe the ...
Security Risks: Management and Mitigation in the Software Life Cycle
WETICE '04: Proceedings of the 13th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative EnterprisesA formal approach to managing and mitigating security risks in the software life cycle is requisite to developing software that has a higher degree of assurance that it is free of security defects which pose risk to the computing environment and the ...
TAGen: Generating Trigger-Action Rules for Smart Homes by Mining Event Traces
Service-Oriented ComputingAbstractA smart home facilities human daily lives by orchestrating IoT devices through trigger-action (TA) rules. However, creating TA rules is challenging for novice users as (1) it requires comprehensive domain knowledge, (2) the created rules often ...
Comments
Information & Contributors
Information
Published In
EISSN:2157-6912
Table of ContentsCopyright © 2024 Copyright held by the owner/author(s).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Online AM: 29 June 2024
Accepted: 29 May 2024
Revised: 13 February 2024
Received: 09 March 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 223Total Downloads
- Downloads (Last 12 months)223
- Downloads (Last 6 weeks)49
Reflects downloads up to 14 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in