Modeling and Analysis of Explanation for Secure Industrial Control Systems
Authors: 
Sridhar Adepu, Nianyu Li, Eunsuk Kang, and 
David GarlanAuthors Info & Claims
Sridhar Adepu, Nianyu Li, Eunsuk Kang, and David GarlanAuthors Info & ClaimsArticle No.: 5, Pages 1 - 26
Abstract
Many self-adaptive systems benefit from human involvement and oversight, where a human operator can provide expertise not available to the system and detect problems that the system is unaware of. One way of achieving this synergy is by placing the human operator on the loop—i.e., providing supervisory oversight and intervening in the case of questionable adaptation decisions. To make such interaction effective, an explanation can play an important role in allowing the human operator to understand why the system is making certain decisions and improve the level of knowledge that the operator has about the system. This, in turn, may improve the operator’s capability to intervene and, if necessary, override the decisions being made by the system. However, explanations may incur costs, in terms of delay in actions and the possibility that a human may make a bad judgment. Hence, it is not always obvious whether an explanation will improve overall utility and, if so, then what kind of explanation should be provided to the operator. In this work, we define a formal framework for reasoning about explanations of adaptive system behaviors and the conditions under which they are warranted. Specifically, we characterize explanations in terms of explanation content, effect, and cost. We then present a dynamic system adaptation approach that leverages a probabilistic reasoning technique to determine when an explanation should be used to improve overall system utility. We evaluate our explanation framework in the context of a realistic industrial control system with adaptive behaviors.
References
[1]
Betty H. C. Cheng and et al.2009. Software engineering for self-adaptive systems: A research roadmap. In International Symposium on Software Engineering for Self-adaptive Systems. 1–26.
[2]
Rogério de Lemos and et al.2010. Software engineering for self-adaptive systems: A second research roadmap. In International Symposium on Software Engineering for Self-adaptive Systems. 1–32.
[3]
Roykrong Sukkerd, David Garlan, and Reid G. Simmons. 2015. Task planning of cyber-human systems. In 13th International Conference on Software Engineering and Formal Methods. 293–309.
[4]
Javier Cámara, Gabriel A. Moreno, and David Garlan. 2015. Reasoning about human participation in self-adaptive systems. In 10th IEEE/ACM International Symposium on Software Engineering for Adaptive and Self-managing Systems. 146–156.
[5]
Eric Lloyd, Shihong Huang, and Emmanuelle Tognoli. 2017. Improving human-in-the-loop adaptive systems using brain-computer interaction. In 12th IEEE/ACM International Symposium on Software Engineering for Adaptive and Self-managing Systems. 163–174.
[6]
Joel E. Fischer, Chris Greenhalgh, Wenchao Jiang, Sarvapali D. Ramchurn, Feng Wu, and Tom Rodden. 2021. In-the-loop or on-the-loop? Interactional arrangements to support team coordination with a planning agent. Concurr. Computat.: Pract. Exper. 33, 8 (2021), e4082.
[7]
Department of Homeland Security. 2022. ICS-CERT Advisories. Retrieved from https://ics-cert.us-cert.gov/advisories.
[8]
Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 Workshop on Explainable AI (XAI), Vol. 8. 8–13.
[9]
Tatsuya Nomura and Kayoko Kawakami. 2011. Relationships between robot’s self-disclosures and human’s anxiety toward robots. In IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology. IEEE Computer Society, 66–69.
[10]
Nianyu Li, Sridhar Adepu, Eunsuk Kang, and David Garlan. Explanations for human-on-the-loop: A probabilistic model checking approach. In 15th International Symposium on Software Engineering for Adaptive and Self-managing Systems (SEAMS).
[11]
Marta Kwiatkowska, Gethin Norman, and David Parker. 2018. Probabilistic Model Checking: Advances and Applications. Springer International Publishing, Cham, 73–121.
[12]
Singapore University of Technology and Design. 2022. Secure Water Treatment (SWaT). Retrieved from https://itrust.sutd.edu.sg/itrust-labs-home/itrust-labs_swat/.
[13]
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell. 267 (2019), 1–38.
[14]
B. Chandrasekaran, Michael C. Tanner, and John R. Josephson. 1989. Explaining control strategies in problem solving. IEEE Expert 4, 1 (1989), 9–24.
[15]
T. R. Fennel and James D. Johannes. 1990. An architecture for rule based system explanation. In Fifth Conference on Artificial Intelligence for Space Applications, vol. 3073. NASA, 113.
[16]
Cécile L. Paris. 1991. Generation and explanation: Building an explanation facility for the explainable expert systems framework. In Natural Language Generation in Artificial Intelligence and Computational Linguistics. Springer, 49–82.
[17]
Bradley Hayes and Julie A. Shah. 2017. Improving robot controller transparency through autonomous policy explanation. In 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 303–312.
[18]
Ryan W. Wohleber, Kimberly Stowers, Jessie Y. C. Chen, and Michael Barnes. 2017. Effects of agent transparency and communication framing on human-agent teaming. In IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3427–3432.
[19]
Matthew Molineaux, Dustin Dannenhauer, and David W. Aha. 2018. Towards explainable NPCs: A relational exploration learning agent. In Workshops at the 32nd AAAI Conference on Artificial Intelligence.
[20]
Maaike Harbers, Karel Van Den Bosch, and John-Jules Meyer. 2009. A methodology for developing self-explaining agents for virtual training. In International Workshop on Languages, Methodologies and Development Tools for Multi-Agent Systems. Springer, 168–182.
[21]
Frank Kaptein, Joost Broekens, Koen Hindriks, and Mark Neerincx. 2017. The role of emotion in self-explanations by cognitive agents. In 7th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). IEEE, 88–93.
[22]
Todd Kulesza, Simone Stumpf, Margaret Burnett, Sherry Yang, Irwin Kwan, and Weng-Keen Wong. 2013. Too much, too little, or just right? Ways explanations impact end users’ mental models. In IEEE Symposium on Visual Languages and Human Centric Computing. IEEE, 3–10.
[23]
Peter Carey.2018. Data Protection: A Practical Guide to UK and EU Law.Oxford University Press, Inc.
[24]
Mark A. Neerincx, Jasper van der Waa, Frank Kaptein, and Jurriaan van Diggelen. 2018. Using perceptual and cognitive explanations for enhanced human-agent team performance. In 15th International Conference on Engineering Psychology and Cognitive Ergonomics. 204–214.
[25]
Roykrong Sukkerd. 2018. Improving Transparency and Understandability of Multi-objective Probabilistic Planning. Thesis Proposal. School of Computer Science Institute for Software Research Software Engineering, Carnegie Mellon University.
[26]
Thomas Hellström and Suna Bensch. 2018. Understandable robotswhat, why, and how. Paladyn, J. Behav. Robot. 9, 1 (2018), 110–123.
[27]
Cindy L. Bethel. 2009. Robots without faces: Non-verbal social human-robot Interaction. Doctoral dissertation, dissertation/Ph. D.’s thesis. University of South Florida.
[28]
Joost Broekens, Maaike Harbers, Koen Hindriks, Karel Van Den Bosch, Catholijn Jonker, and John-Jules Meyer. 2010. Do you get it? User-evaluated explainable BDI agents. In German Conference on Multiagent System Technologies. Springer, 28–39.
[29]
Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In 27th International Conference on Human Factors in Computing Systems. 2119–2128.
[30]
Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, and Subbarao Kambhampati. 2017. Plan explanations as model reconciliation: Moving beyond explanation as soliloquy. In 26th International Joint Conference on Artificial Intelligence. 156–163.
[31]
Francisco Elizalde, L. Enrique Sucar, Manuel Luque, J. Diez, and Alberto Reyes. 2008. Policy explanation in factored Markov decision processes. In European Workshop on Probabilistic Graphical Models (PGM). 97–104.
[32]
Omar Zia Khan, Pascal Poupart, and James P. Black. 2009. Minimal sufficient explanations for factored Markov decision processes. In 19th International Conference on Automated Planning and Scheduling.
[33]
Roykrong Sukkerd, Reid G. Simmons, and David Garlan. 2018. Towards explainable multi-objective probabilistic planning. In 4th International Workshop on Software Engineering for Smart Cyber-physical Systems. 19–25.
[34]
Alvaro A. Cárdenas, Saurabh Amin, Zong-Syun Lin, Yu-Lun Huang, Chi-Yen Huang, and Shankar Sastry. 2011. Attacks against process control systems: Risk assessment, detection, and response. In 6th ACM Symposium on Information, Computer and Communications Security. 355–366.
[35]
Sridhar Adepu and Aditya Mathur. 2018. Assessing the effectiveness of attack detection at a hackfest on industrial control systems. IEEE Trans. Sustain. Comput. 6, 2 (2018), 231–244.
[36]
Yao Liu, Peng Ning, and Michael K. Reiter. 2011. False data injection attacks against state estimation in electric power grids. ACM Trans. Inf. Syst. Secur. 14, 1 (2011), 1–33.
[37]
Nikola Trcka, Mark Moulin, Shaunak Bopardikar, and Alberto Speranzon. 2014. A formal verification approach to revealing stealth attacks on networked control systems. In 3rd International Conference on High Confidence Networked Systems. 67–76.
[38]
Sridhar Adepu, Nandha Kumar Kandasamy, and Aditya Mathur. 2018. EPIC: An electric power testbed for research and training in cyber physical systems security. In Computer Security. Springer, 37–52.
[39]
Yuriy Zacchia Lun, Alessandro D’Innocenzo, Francesco Smarra, Ivano Malavolta, and Maria Domenica Di Benedetto. 2019. State of the art of cyber-physical systems security: An automatic control perspective. J. Syst. Softw. 149 (2019), 174–216.
[40]
Giedre Sabaliauskaite and Sridhar Adepu. 2017. Integrating six-step model with information flow diagrams for comprehensive analysis of cyber-physical system safety and security. In IEEE 18th International Symposium on High Assurance Systems Engineering (HASE). IEEE, 41–48.
[41]
Sridhar Adepu, Eunsuk Kang, and Aditya P. Mathur. 2019. Challenges in secure engineering of critical infrastructure systems. In 34th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW). IEEE, 61–64.
[42]
Konstantinos Demertzis, Lazaros S. Iliadis, and Vardis-Dimitrios Anezakis. 2018. An innovative soft computing system for smart energy grids cybersecurity. Adv. Build. Energ. Res. 12, 1 (2018), 3–24.
[43]
Muhammad Taimoor Khan, Dimitrios Serpanos, and Howard Shrobe. 2017. ARMET: Behavior-based secure and resilient industrial control systems. Proc. IEEE 106, 1 (2017), 129–143.
[44]
Xin Zhang, Xiaobo Cai, Chaogang Wang, Ke Han, and Shujuan Zhang. 2019. A dynamic security control architecture for industrial cyber-physical system. In IEEE International Conference on Industrial Internet (ICII). IEEE, 148–151.
[45]
Joel Greenyer, Malte Lochau, and Thomas Vogel. 2019. Explainable software for cyber-physical systems (ES4CPS): Report from the GI Dagstuhl Seminar 19023, January 6-11 2019, Schloss Dagstuhl. arXiv preprint arXiv:1904.11851.
[46]
Tesla’s Trouble with Semi Trucks & Another Shakeup of the Autopilot Team – Is There a Connection? Retrieved 10 Oct., 2021 from https://cleantechnica.com/2019/05/21/teslas-trouble-with-trucks-and-another-shakeup-of-the-autopilot-team-is-there-a-connection/.
[47]
Yao Deng, Tiehua Zhang, Guannan Lou, Xi Zheng, Jiong Jin, and Qing-Long Han. 2021. Deep learning-based autonomous driving systems: A survey of attacks and defenses. IEEE Trans. Industr. Inf. 17, 12 (2021), 7897–7912.
[48]
Sébastien Combéfis, Dimitra Giannakopoulou, Charles Pecheur, and Michael Feary. 2011. Learning system abstractions for human operators. In International Workshop on Machine Learning Technologies in Software Engineering. 3–10.
[49]
E. Palmer. 1996. Oops, it didn’t arm.—A case study of two automation surprises. In 8th International Symposium on Aviation Psychology. 227–232.
[50]
Gabriel A. Moreno, Javier Cámara, David Garlan, and Bradley R. Schmerl. 2015. Proactive self-adaptation under uncertainty: A probabilistic model checking approach. In 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE. 1–12.
[51]
Andrea Bianco and Luca de Alfaro. 1995. Model checking of probabilistic and nondeterministic systems. In Foundations of Software Technology and Theoretical Computer Science, P. S. Thiagarajan (Ed.). Springer Berlin.
[52]
Marta Z. Kwiatkowska, Gethin Norman, and David Parker. 2011. PRISM 4.0: Verification of probabilistic real-time systems. In 23rd International Conference on Computer Aided Verification. 585–591.
[53]
Martin L. Puterman. 1994. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley.
[54]
Gethin Norman, David Parker, and Jeremy Sproston. 2013. Model checking for probabilistic timed automata. Form. Meth. Syst. Des. 43, 2 (2013), 164–190.
[55]
Karline Soetaert. 2018. plot3D: Tools for Plotting 3-D and 2-D Data. Retrieved from https://cran.r-project.org/web/packages/plot3D/vignettes/plot3D.pdf.
[56]
A. P. Mathur and N. O. Tippenhauer. 2016. SWaT: A water treatment testbed for research and training on ICS security. In International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater). 31–36.
[57]
Eunsuk Kang, Sridhar Adepu, Daniel Jackson, and Aditya P. Mathur. 2016. Model-based security analysis of a water treatment system. In IEEE/ACM 2nd International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS). IEEE, 22–28.
[58]
Sridhar Adepu and Aditya Mathur. 2016. An investigation into the response of a water treatment system to cyber attacks. In IEEE 17th International Symposium on High Assurance Systems Engineering (HASE). IEEE, 141–148.
[59]
Aung Maw, Sridhar Adepu, and Aditya Mathur. 2019. ICS-BlockOpS: Blockchain for operational data security in industrial control system. Pervas. Mob. Comput. 59 (2019), 101048.
[60]
Yuqi Chen, Christopher M. Poskitt, Jun Sun, Sridhar Adepu, and Fan Zhang. 2019. Learning-guided network fuzzing for testing cyber-physical system defences. In 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 962–973.
[61]
Sridhar Adepu, Ferdinand Brasser, Luis Garcia, Michael Rodler, Lucas Davi, Ahmad-Reza Sadeghi, and Saman Zonouz. 2020. Control behavior integrity for distributed cyber-physical systems. In ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS). IEEE, 30–40.
[62]
Tanmoy Kanti Das, Sridhar Adepu, and Jianying Zhou. 2020. Anomaly detection in industrial control systems using logical analysis of data. Comput. Secur. 96 (2020), 101935.
[63]
Muhammad Azmi Umer, Aditya Mathur, Khurum Nazir Junejo, and Sridhar Adepu. 2017. Integrating design and data centric approaches to generate invariants for distributed attack detection. In Workshop on Cyber-Physical Systems Security and PrivaCy. 131–136.
[64]
Sara Mahdavi-Hezavehi, Vinicius H. S. Durelli, Danny Weyns, and Paris Avgeriou. 2017. A systematic literature review on methods that handle multiple quality attributes in architecture-based self-adaptive systems. Inf. Softw. Technol. 90 (2017), 1–26.
Index Terms
- Modeling and Analysis of Explanation for Secure Industrial Control Systems
Recommendations
Explanation in Scientists and Children
In this paper we provide a psychological account of the nature and development of explanation. We propose that an explanation is an account that provides a conceptual framework for a phenomenon that leads to a feeling of understanding in the reader/hearer. ...
The Shadows and Shallows of Explanation
We introduce two notions–the shadows and the shallows of explanation–in opening up explanation to broader, interdisciplinary investigation. The “shadows of explanation” refer to past philosophical efforts to provide either a conceptual analysis of ...
Complexity results for explanations in the structural-model approach
We analyze the computational complexity of Halpern and Pearl's (causal) explanations in the structural-model approach, which are based on their notions of weak and actual cause. In particular, we give a precise picture of the complexity of deciding ...
Comments
Information & Contributors
Information
Published In
December 2022
49 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 December 2022
Online AM: 17 August 2022
Accepted: 29 June 2022
Revised: 03 May 2022
Received: 11 February 2021
Published in TAAS Volume 17, Issue 3-4
Check for updates
Author Tags
Qualifiers
- Research-article
- Refereed
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 303Total Downloads
- Downloads (Last 12 months)112
- Downloads (Last 6 weeks)8
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format