CrystalBox: future-based explanations for input-driven deep RL systems
Article No.: 1624, Pages 14563 - 14571
Abstract
We present CrystalBox, a novel, model-agnostic, posthoc explainability framework for Deep Reinforcement Learning (DRL) controllers in the large family of input-driven environments which includes computer systems. We combine the natural decomposability of reward functions in input-driven environments with the explanatory power of decomposed returns. We propose an efficient algorithm to generate future-based explanations across both discrete and continuous control environments. Using applications such as adaptive bitrate streaming and congestion control, we demonstrate Crystal-Box's capability to generate high-fidelity explanations. We further illustrate its higher utility across three practical use cases: contrastive explanations, network observability, and guided reward design, as opposed to prior explainability techniques that identify salient features.
References
[1]
Anderson, A.; Dodge, J.; Sadarangani, A.; Juozapaitis, Z.; Newman, E.; Irvine, J.; Chattopadhyay, S.; Fern, A.; and Burnett, M. 2019. Explaining reinforcement learning to mere mortals: An empirical study. arXiv preprint arXiv:1903.09708.
[2]
Bastani, O.; Pu, Y.; and Solar-Lezama, A. 2018. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31.
[3]
Beattie, C.; Leibo, J. Z.; Teplyashin, D.; Ward, T.; Wainwright, M.; Kuttler, H.; Lefrancq, A.; Green, S.; Valdés, V.; Sadik, A.; et al. 2016. Deepmind lab. arXiv preprint arXiv:1612.03801.
[4]
Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016. Openai gym. arXiv preprint arXiv:1606.01540.
[5]
Browne, C. B.; Powley, E.; Whitehouse, D.; Lucas, S. M.; Cowling, P. I.; Rohlfshagen, P.; Tavener, S.; Perez, D.; Samothrakis, S.; and Colton, S. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1): 1-43.
[6]
Burkart, N.; and Huber, M. F. 2021. A survey on the explainability of supervised machine learning. Journal of Artificial Intelligence Research, 70: 245-317.
[7]
Chen, L.; Lingys, J.; Chen, K.; and Liu, F. 2018. Auto: Scaling deep reinforcement learning for datacenter-scale automatic traffic optimization. In Proceedings of the 2018 conference of the ACM special interest group on data communication, 191-205.
[8]
Cruz, F.; Dazeley, R.; Vamplew, P.; and Moreira, I. 2021. Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario. Neural Computing and Applications, 1-18.
[9]
Doshi-Velez, F.; Kortz, M.; Budish, R.; Bavitz, C.; Gershman, S.; O'Brien, D.; Scott, K.; Schieber, S.; Waldo, J.; Weinberger, D.; et al. 2017. Accountability of AI under the law: The role of explanation. arXiv preprint arXiv:1711.01134.
[10]
Greydanus, S.; Koul, A.; Dodge, J.; and Fern, A. 2018. Visualizing and understanding atari agents. In International conference on machine learning, 1792-1801. PMLR.
[11]
Holkar, K.; and Waghmare, L. M. 2010. An overview of model predictive control. International Journal of control and automation, 3(4): 47-63.
[12]
Iyer, R.; Li, Y.; Li, H.; Lewis, M.; Sundar, R.; and Sycara, K. 2018. Transparency and explanation in deep reinforcement learning neural networks. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, 144-150.
[13]
Jay, N.; Rotman, N.; Godfrey, B.; Schapira, M.; and Tamar, A. 2019. A deep reinforcement learning perspective on internet congestion control. In International conference on machine learning, 3050-3059. PMLR.
[14]
Juozapaitis, Z.; Koul, A.; Fern, A.; Erwig, M.; and Doshi-Velez, F. 2019. Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence.
[15]
Krishnan, S.; Yang, Z.; Goldberg, K.; Hellerstein, J.; and Stoica, I. 2018. Learning to optimize join queries with deep reinforcement learning. arXiv preprint arXiv:1808.03196.
[16]
Lyle, C.; Rowland, M.; Ostrovski, G.; and Dabney, W. 2021. On the effect of auxiliary tasks on representation dynamics. In International Conference on Artificial Intelligence and Statistics, 1-9. PMLR.
[17]
Mao, H.; Negi, P.; Narayan, A.; Wang, H.; Yang, J.; Wang, H.; Marcus, R.; Khani Shirkoohi, M.; He, S.; Nathan, V.; et al. 2019. Park: An open platform for learning-augmented computer systems. Advances in Neural Information Processing Systems, 32.
[18]
Mao, H.; Netravali, R.; and Alizadeh, M. 2017. Neural adaptive video streaming with pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, 197-210.
[19]
Mao, H.; Venkatakrishnan, S. B.; Schwarzkopf, M.; and Alizadeh, M. 2018. Variance reduction for reinforcement learning in input-driven environments. arXiv preprint arXiv:1807.02264.
[20]
Meng, Z.; Wang, M.; Bai, J.; Xu, M.; Mao, H.; and Hu, H. 2020. Interpreting deep learning-based networking systems. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, 154-171.
[21]
Miller, T. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267: 1-38.
[22]
Mittelstadt, B.; Russell, C.; and Wachter, S. 2019. Explaining explanations in AI. In Proceedings of the conference on fairness, accountability, and transparency, 279-288.
[23]
Mok, R. K.; Chan, E. W.; and Chang, R. K. 2011. Measuring the quality of experience of HTTP video streaming. In 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops, 485-492. IEEE.
[24]
Patel, S.; Jyothi, S. A.; and Narodytska, N. 2023. CrystalBox: Future-Based Explanations for DRL Network Controllers. arXiv:2302.13483.
[25]
Patel, S.; Zhang, J.; Jyothi, S. A.; and Narodytska, N. 2023. Plume: A Framework for High Performance Deep RL Network Controllers via Prioritized Trace Sampling. arXiv:2302.12403.
[26]
Puri, N.; Verma, S.; Gupta, P.; Kayastha, D.; Deshmukh, S.; Krishnamurthy, B.; and Singh, S. 2019. Explain your move: Understanding agent actions using specific and relevant feature attribution. arXiv preprint arXiv:1912.12191.
[27]
Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016. " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 1135-1144.
[28]
Rotman, N. H.; Schapira, M.; and Tamar, A. 2020. Online safety assurance for learning-augmented systems. In Proceedings of the 19th ACM Workshop on Hot Topics in Networks, 88-95.
[29]
Silver, D. 2015. Lectures on Reinforcement Learning. https://www.davidsilver.uk/teaching/. Accessed: 2022-10-12.
[30]
Sutton, R. S.; and Barto, A. G. 2018. Reinforcement learning: An introduction. MIT press.
[31]
van der Waa, J.; van Diggelen, J.; Bosch, K. v. d.; and Neerincx, M. 2018. Contrastive explanations for reinforcement learning in terms of expected consequences. arXiv preprint arXiv:1807.08706.
[32]
Verma, A.; Murali, V.; Singh, R.; Kohli, P.; and Chaudhuri, S. 2018. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, 5045-5054. PMLR.
[33]
Yan, F. Y.; Ayers, H.; Zhu, C.; Fouladi, S.; Hong, J.; Zhang, K.; Levis, P.; and Winstein, K. 2020. Learning in situ: a randomized experiment in video streaming. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI20), 495-511.
[34]
Yau, H.; Russell, C.; and Hadfield, S. 2020. What did you think would happen? explaining agent behaviour through intended outcomes. Advances in Neural Information Processing Systems, 33: 18375-18386.
[35]
Zahavy, T.; Ben-Zrihem, N.; and Mannor, S. 2016. Graying the black box: Understanding dqns. In International conference on machine learning, 1899-1908. PMLR.
[36]
Zhang, H.; Zhou, A.; and Lin, X. 2020. Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis. Complex & Intelligent Systems, 6(3): 741-753.
Index Terms
- CrystalBox: future-based explanations for input-driven deep RL systems
Index terms have been assigned to the content through auto-classification.
Recommendations
Complexity results for explanations in the structural-model approach
We analyze the computational complexity of Halpern and Pearl's (causal) explanations in the structural-model approach, which are based on their notions of weak and actual cause. In particular, we give a precise picture of the complexity of deciding ...
Complexity results for structure-based causality
We give a precise picture of the computational complexity of causal relationships in Pearl's structural models, where we focus on causality between variables, event causality, and probabilistic causality. As for causality between variables, we consider ...
Comments
Information & Contributors
Information
Published In
February 2024
23861 pages
ISBN:978-1-57735-887-9
Copyright © 2024 Association for the Advancement of Artificial Intelligence.
Sponsors
- Association for the Advancement of Artificial Intelligence
Publisher
AAAI Press
Publication History
Published: 20 February 2024
Qualifiers
- Research-article
- Research
- Refereed limited
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 31 Jan 2025