Abstract
Web applications are constantly under attack as the public-facing components of information systems. One defense mechanism is deception, which introduces deceptive components into the application to detect the attacks with high fidelity, while distracting attackers from the successful attack path.
One important challenge that hinders the widespread adoption of deception is the difficulty to assess its effectiveness. This often requires conducting human experiments, which can be both costly and impractical for every individual web application scenario. A recent solution proposed to address this issue for network-layer deception has been to use a Reinforcement Learning (RL) based framework to simulate an attacker in a network with deceptive elements.
In this paper, we extend this framework to simulate the different components of web applications and related deceptive strategies. We then conduct several experiments to understand how the different quantities and types of deceptive elements impact the time to detect the attacker. Our empirical findings reveal that a larger number of honeytokens impede the agent’s learning, and allows for earlier attack detection. We also demonstrate the impact of each honeytoken on the success rate of attack detection, and how the implementation of deceptive elements can affect the performance of the agent.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
References
Betarte, G., Pardo, A., Martínez, R.: Web application attacks detection using machine learning techniques. In: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 1065–1072 (Dec 2018). https://doi.org/10.1109/ICMLA.2018.00174
Brockman, G., et al.: Openai gym. CoRR (2016). http://arxiv.org/abs/1606.01540
Caminero Fernández, G., Lopez-Martin, M., Carro, B.: Adversarial environment reinforcement learning algorithm for intrusion detection. Comput. Netw. 159, 96–109 (2019). https://doi.org/10.1016/j.comnet.2019.05.013
Charpentier, A., Boulahia Cuppens, N., Cuppens, F., Yaich, R.: Deep Reinforcement Learning-Based Defense Strategy Selection. In: Proceedings of the 17th International Conference on Availability, Reliability and Security, pp. 1–11. ACM, Vienna Austria (Aug 2022). https://doi.org/10.1145/3538969.3543789
El-Kosairy, A., Azer, M.A.: A New web deception system framework. In: 2018 1st International Conference on Computer Applications & Information Security (ICCAIS), pp. 1–10 (Apr 2018). https://doi.org/10.1109/CAIS.2018.8442027
Elderman, R., Pater, J.J., L., S. Thie, A., M. Drugan, M., M. Wiering, M.: Adversarial Reinforcement Learning in a Cyber Security Simulation. In: Proceedings of the 9th International Conference on Agents and Artificial Intelligence, pp. 559–566. SCITEPRESS - Science and Technology Publications, Porto, Portugal (2017). https://doi.org/10.5220/0006197105590566, http://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0006197105590566
Erdődi, L., Sommervoll, A.A., Zennaro, F.M.: Simulating SQL injection vulnerability exploitation using Q-learning reinforcement learning agents. Journal of Information Security and Applications 61(C) (Sep 2021). https://doi.org/10.1016/j.jisa.2021.102903
Even-Dar, E., Mansour, Y.: Learning Rates for Q-Learning. In: Goos, G., Hartmanis, J., Van Leeuwen, J., Helmbold, D., Williamson, B. (eds.) Computational Learning Theory, vol. 2111, pp. 589–604. Springer, Berlin Heidelberg, Berlin, Heidelberg (2001). https://doi.org/10.1007/3-540-44581-1_39, http://link.springer.com/10.1007/3-540-44581-1_39, series Title: Lecture Notes in Computer Science
Gan, Y., et al.: An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems. In: Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 3–18. ACM, Providence RI USA (Apr 2019). https://doi.org/10.1145/3297858.3304013, https://dl.acm.org/doi/10.1145/3297858.3304013
Han, X., Kheir, N., Balzarotti, D.: Evaluation of Deception-Based Web Attacks Detection. In: Proceedings of the 2017 Workshop on Moving Target Defense, pp. 65–73. ACM, Dallas Texas USA (Oct 2017). https://doi.org/10.1145/3140549.3140555, https://dl.acm.org/doi/10.1145/3140549.3140555
Han, X., Kheir, N., Balzarotti, D.: Deception techniques in computer security: a research perspective. ACM Comput. Surv. 51(4), 80 (2018). https://doi.org/10.1145/3214305
van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. CoRR (2015). http://arxiv.org/abs/1509.06461
Kunz, T., Fisher, C., La Novara-Gsell, J., Nguyen, C., Li, L.: A Multiagent CyberBattleSim for RL Cyber Operation Agents (Apr 2023). 10.48550/arXiv. 2304.11052, http://arxiv.org/abs/2304.11052, arXiv:2304.11052 [cs]
Li, H., Guo, Y., Huo, S., Hu, H., Sun, P.: Defensive deception framework against reconnaissance attacks in the cloud with deep reinforcement learning. Sci. China Inf. Sci. 65(7), 170305 (Jul 2022). https://doi.org/10.1007/s11432-021-3462-4, https://link.springer.com/10.1007/s11432-021-3462-4
Li, L., Fayad, R., Taylor, A.: CyGIL: A Cyber Gym for Training Autonomous Agents over Emulated Network Systems (Sep 2021). https://doi.org/10.48550/arXiv.2109.03331
Li, Q., et al.: A hierarchical deep reinforcement learning model with expert prior knowledge for intelligent penetration testing. Computers & Security 132, 103358 (Sep 2023). https://doi.org/10.1016/j.cose.2023.103358, https://www.sciencedirect.com/science/article/pii/S0167404823002687
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., et al.: Playing Atari with Deep Reinforcement Learning. NIPS Deep Learning Workshop 2013 (Dec 2013), http://arxiv.org/abs/1312.5602,arXiv: 1312.5602
Reti, D., Elzer, K., Schotten, H.D.: SCANTRAP: Protecting Content Management Systems from Vulnerability Scanners with Cyber Deception and Obfuscation (Jan 2023). http://arxiv.org/abs/2301.10502arXiv:2301.10502 [cs]
Sahin, M., Hebert, C., De Oliveira, A.S.: Lessons Learned from SunDEW: A Self Defense Environment for Web Applications. In: Proceedings 2020 Workshop on Measurements, Attacks, and Defenses for the Web. Internet Society, San Diego, CA (2020). https://doi.org/10.14722/madweb.2020.23005, https://www.ndss-symposium.org/wp-content/uploads/2020/02/23005.pdf
Sahin, M., Hébert, C., Cabrera Lozoya, R.: An Approach to Generate Realistic HTTP Parameters for Application Layer Deception. In: Ateniese, G., Venturi, D. (eds.) Applied Cryptography and Network Security. vol. 13269, pp. 337–355. Springer International Publishing, Cham (2022). https://doi.org/10.1007/978-3-031-09234-3-17, https://link.springer.com/10.1007/978-3-031-09234-3_17, series Title: Lecture Notes in Computer Science
Shashkov, A., Hemberg, E., Tulla, M., O’Reilly, U.M.: Adversarial agent-learning for cybersecurity: a comparison of algorithms. The Knowledge Engineering Review 38, e3 (Jan 2023). https://doi.org/10.1017/S0269888923000012, publisher: Cambridge University Press
Standen, M., Lucas, M., Bowman, D., Richer, T.J., Kim, J., Marriott, D.: CybORG: A Gym for the Development of Autonomous Cyber Agents (Aug 2021). https://doi.org/10.48550/arXiv.2108.09118
van der Stock, A., Glas, B., Smithline, N., Gigler, T.: Owasp Web Security Testing Guide v4.2. https://github.com/OWASP/wstg/releases/download/v4.2/wstg-v4.2.pdf (2014)
van der Stock, A., Glas, B., Smithline, N., Gigler, T.: Owasp Appsensor project guide v2. https://owasp.org/www-pdf-archive/Owasp-appsensor-guide-v2.pdf (2015)
van der Stock, A., Glas, B., Smithline, N., Gigler, T.: OWASP Top 10 project (2021). https://owasp.org/Top10/
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press (2018)
Team., M.D.R.: Cyberbattlesim. https://github.com/microsoft/cyberbattlesim (2021)
Walter, E., Ferguson-Walter, K., Ridley, A.: Incorporating Deception into CyberBattleSim for Autonomous Defense. IJCAI-21 1st International Workshop on Adaptive Cyber Defense (Aug 2021), http://arxiv.org/abs/2108.13980arXiv:2108.13980 [cs]
Wang, S., Pei, Q., Wang, J., Tang, G., Zhang, Y., Liu, X.: An Intelligent Deployment Policy for Deception Resources Based on Reinforcement Learning. IEEE Access 8, 35792–35804 (2020). https://doi.org/10.1109/ACCESS.2020.2974786, conference Name: IEEE Access
Xin, W., Gengyu, W., Yixian, Y.: Web application vulnerability detection based on reinforcement learning. Int. J. Digital Content Technol. Appl. 6, 12–20 (2012). https://doi.org/10.4156/jdcta.vol6.issue10.2
Yao, Q., Wang, Y., Xiong, X., Wang, P., Li, Y.: Adversarial decision-making for moving target defense: a multi-agent markov game and reinforcement learning approach. Entropy 25(4), 605 (Apr 2023). https://doi.org/10.3390/e25040605, https://www.mdpi.com/1099-4300/25/4/605, number: 4 Publisher: Multidisciplinary Digital Publishing Institute
Zhang, L., Thing, V.L.L.: Three Decades of Deception Techniques in Active Cyber Defense - Retrospect and Outlook. Computers & Security 106, 102288 (Jul 2021). https://doi.org/10.1016/j.cose.2021.102288, http://arxiv.org/abs/2104.03594,arXiv:2104.03594 [cs]
Zhu, M., Anwar, A.H., Wan, Z., Cho, J.H., Kamhoua, C., Singh, M.P.: Game-theoretic and machine learning-based approaches for defensive deception: a survey (May 2021). http://arxiv.org/abs/2101.10121arXiv:2101.10121 [cs]
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kvasov, A., Sahin, M., Hebert, C., De Oliveira, A.S. (2024). Simulating Deception for Web Applications Using Reinforcement Learning. In: Katsikas, S., et al. Computer Security. ESORICS 2023 International Workshops. ESORICS 2023. Lecture Notes in Computer Science, vol 14399. Springer, Cham. https://doi.org/10.1007/978-3-031-54129-2_42
Download citation
DOI: https://doi.org/10.1007/978-3-031-54129-2_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54128-5
Online ISBN: 978-3-031-54129-2
eBook Packages: Computer ScienceComputer Science (R0)