Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Adversarial robustness of deep reinforcement learning-based intrusion detection

  • Regular Contribution
  • Published:
International Journal of Information Security Aims and scope Submit manuscript

Abstract

Machine learning techniques, including Deep Reinforcement Learning (DRL), enhance intrusion detection systems by adapting to new threats. However, DRL’s reliance on vulnerable deep neural networks leads to susceptibility to adversarial examples-perturbations designed to evade detection. While adversarial examples are well-studied in deep learning, their impact on DRL-based intrusion detection remains underexplored, particularly in critical domains. This article conducts a thorough analysis of DRL-based intrusion detection’s vulnerability to adversarial examples. It systematically evaluates key hyperparameters such as DRL algorithms, neural network depth, and width, impacting agents’ robustness. The study extends to black-box attacks, demonstrating adversarial transferability across DRL algorithms. Findings emphasize neural network architecture’s critical role in DRL agent robustness, addressing underfitting and overfitting challenges. Practical implications include insights for optimizing DRL-based intrusion detection agents to enhance performance and resilience. Experiments encompass multiple DRL algorithms tested on three datasets: NSL-KDD, UNSW-NB15, and CICIoV2024, against gradient-based adversarial attacks, with publicly available implementation code.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. https://github.com/mamerzouk/robust_drl_ids.

References

  1. Abou El Houda, Z., Moudoud, H., Brik, B.: Federated deep reinforcement learning for efficient jamming attack mitigation in o-ran. In: IEEE Transactions on Vehicular Technology (2024)

  2. Ajay, J.: Why Cybersecurity Should Be Top Of Mind In 2023 (2023). https://www.forbes.com/sites/forbestechcouncil/2023/01/06/why-cybersecurity-should-be-top-of-mind-in-2023/?sh=1654d131235c

  3. Annachhatre, C., Austin, T.H., Stamp, M.: Hidden Markov models for malware classification. J. Comput. Virol. Hack. Tech. 11, 59–73 (2015)

    Article  Google Scholar 

  4. Bajaj, K., Arora, A.: Improving the intrusion detection using discriminative machine learning approach and improve the time complexity by data mining feature selection methods. Int. J. Comput. Appl. 76(1), 5–11 (2013)

    Google Scholar 

  5. Behzadan, V., Munir, A.: Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks. In: Machine Learning and Data Mining in Pattern Recognition (2017)

  6. Buczak, A.L., Guven, E.: A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Commun. Surv. Tutor. 18(2), 1153–1176 (2016)

    Article  Google Scholar 

  7. Caminero, G., Lopez-Martin, M., Carro, B.: Adversarial environment reinforcement learning algorithm for intrusion detection. Comput. Netw. 159, 96–109 (2019)

    Article  Google Scholar 

  8. Cevallos M., J.F., Rizzardi, A., Sicari, S., Coen Porisini, A.: Deep Reinforcement Learning for intrusion detection in Internet of Things: Best practices, lessons learnt, and open challenges. Computer Networks (2023)

  9. Chebrolu, S., Abraham, A., Thomas, J.P.: Feature deduction and ensemble design of intrusion detection systems. Computers & Security (2005)

  10. Chen, T., Liu, J., Xiang, Y., Niu, W., Tong, E., Han, Z.: Adversarial attack and defense in reinforcement learning-from ai security view. Cybersecurity (2019)

  11. Dabney, W., Rowland, M., Bellemare, M., Munos, R.: Distributional reinforcement learning with quantile regression. In: Proceedings of the AAAI Conference on Artificial Intelligence (2018)

  12. Debar, H., Becker, M., Siboni, D.: A neural network component for an intrusion detection system. In: Proceedings 1992 IEEE Computer Society Symposium on Research in Security and Privacy (1992)

  13. Farebrother, J., Machado, M.C., Bowling, M.: Generalization and Regularization in DQN. In: NeurIPS 2018: Deep Reinforcement Learning Workshop (2018)

  14. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: 3rd International Conference on Learning Representations (2015)

  15. Hasselt, H.v., Guez, A., Silver, D.: Deep reinforcement learning with double Q-Learning. In: AAAI Conference on Artificial Intelligence (2016)

  16. He, K., Kim, D.D., Asghar, M.R.: Adversarial machine learning for network intrusion detection systems: a comprehensive survey. IEEE Commun. Surv. Tutor. 25(1), 538–566 (2023)

    Article  Google Scholar 

  17. Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., Silver, D.: Rainbow: combining improvements in deep reinforcement learning. In: AAAI Conference on Artificial Intelligence (2018)

  18. Hickling, T., Aouf, N., Spencer, P.: Robust adversarial attacks detection based on explainable deep reinforcement learning for uav guidance and planning. In: IEEE Transactions on Intelligent Vehicles (2023)

  19. Huang, S., Papernot, N., Goodfellow, I., Duan, Y., Abbeel, P.: Adversarial Attacks on Neural Network Policies (2017). http://arxiv.org/abs/1702.02284

  20. Ilahi, I., Usama, M., Qadir, J., Janjua, M.U., Al-Fuqaha, A., Hoang, D.T., Niyato, D.: Challenges and countermeasures for adversarial attacks on deep reinforcement learning. IEEE Trans. Artif. Intell. 3(2), 90–109 (2021)

    Article  Google Scholar 

  21. Khraisat, A., Gondal, I., Vamplew, P., Kamruzzaman, J.: Survey of intrusion detection systems: techniques, datasets and challenges. Cybersecurity 2(1), 1–22 (2019)

    Article  Google Scholar 

  22. Kiran, B.R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A.A.A., Yogamani, S., Pérez, P.: Deep Reinforcement Learning for Autonomous Driving: A Survey (2021). http://arxiv.org/abs/2002.00444

  23. Kos, J., Song, D.: Delving into adversarial attacks on deep policies. In: 5th International Conference on Learning Representations (2019)

  24. Kreibich, C., Crowcroft, J.: Honeycomb: creating intrusion detection signatures using honeypots. ACM SIGCOMM Comput. Commun. Rev. 34(1), 51–56 (2004)

    Article  Google Scholar 

  25. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems. Curran Associates, Inc. (2012)

  26. Kurakin, A., Goodfellow, I.J., Bengio, S.: Adversarial examples in the physical world. In: 5th International Conference on Learning Representations (2017)

  27. Levine, S., Finn, C., Darrell, T., Abbeel, P.: End-to-End Training of Deep Visuomotor Policies (2016). http://arxiv.org/abs/1504.00702

  28. Liao, H.J., Richard Lin, C.H., Lin, Y.C., Tung, K.Y.: Intrusion detection system: a comprehensive review. J. Netw. Comput. Appl. 36(1), 16–24 (2013)

    Article  Google Scholar 

  29. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning (2016). http://arxiv.org/abs/1509.02971

  30. Lin, Y.C., Hong, Z.W., Liao, Y.H., Shih, M.L., Liu, M.Y., Sun, M.: Tactics of Adversarial Attack on Deep Reinforcement Learning Agents. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence (2017)

  31. Lopez-Martin, M., Carro, B., Sanchez-Esguevillas, A.: Application of deep reinforcement learning to intrusion detection for supervised problems. Expert Syst. Appl. 141, 112963 (2020)

    Article  Google Scholar 

  32. Meng, M.H., Bai, G., Teo, S.G., Hou, Z., Xiao, Y., Lin, Y., Dong, J.S.: Adversarial Robustness of deep neural networks: a survey from a formal verification perspective. IEEE Transactions on Dependable and Secure Computing (2022)

  33. Merzouk, M.A., Cuppens, F., Boulahia-Cuppens, N., Yaich, R.: A Deeper analysis of adversarial examples in intrusion detection. In: 15th International Conference on Risks and Security of Internet and Systems (2021)

  34. Merzouk, M.A., Cuppens, F., Boulahia-Cuppens, N., Yaich, R.: Investigating the practicality of adversarial evasion attacks on network intrusion detection. Anna. Telecommun. 77(11), 763–775 (2022)

    Article  Google Scholar 

  35. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous Methods for Deep Reinforcement Learning. In: Proceedings of The 33rd International Conference on Machine Learning (2016)

  36. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. http://arxiv.org/abs/1312.5602 (2013)

  37. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  38. Mohamed, S., Ejbali, R.: Deep SARSA-based reinforcement learning approach for anomaly network intrusion detection system. Int. J. Inf. Security 22(1), 235–247 (2023)

    Article  Google Scholar 

  39. Moudoud, H., Cherkaoui, S.: Empowering security and trust in 5g and beyond: A deep reinforcement learning approach. IEEE Open Journal of the Communications Society (2023)

  40. Moustafa, N., Slay, J.: UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). In: Military Communications and Information Systems Conference (MilCIS) (2015)

  41. Neto, E.C.P., Taslimasa, H., Dadkhah, S., Iqbal, S., Xiong, P., Rahman, T., Ghorbani, A.A.: CICIoV2024: advancing realistic IDS approaches against DoS and spoofing attack in IoV CAN bus. Internet Things 26, 101209 (2024)

    Article  Google Scholar 

  42. Nicolae, M.I., Sinn, M., Tran, M.N., Buesser, B., Rawat, A., Wistuba, M., Zantedeschi, V., Baracaldo, N., Chen, B., Ludwig, H., Molloy, I.M., Edwards, B.: Adversarial Robustness Toolbox v1.0.0. http://arxiv.org/abs/1807.01069 (2019)

  43. Papernot, N., McDaniel, P., Goodfellow, I.: Transferability in machine learning: from phenomena to black-box attacks using adversarial samples (2016). http://arxiv.org/abs/1605.07277

  44. Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the ACM on Asia Conference on Computer and Communications Security (2017)

  45. Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: IEEE European Symposium on Security and Privacy (2016)

  46. Pasikhani, A.M., Clark, J.A., Gope, P.: Adversarial RL-based IDS for evolving data environment in 6LoWPAN. IEEE Trans. Inf. Forens. Security 17, 3831–3846 (2022)

    Article  Google Scholar 

  47. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems (2019)

  48. Pierazzi, F., Pendlebury, F., Cortellazzi, J., Cavallaro, L.: Intriguing Properties of Adversarial ML Attacks in the Problem Space. In: IEEE Symposium on Security and Privacy (SP) (2020)

  49. Priya, S., Kumar, K.P.M.: Binary bat algorithm based feature selection with deep reinforcement learning technique for intrusion detection system. Soft Comput. 27(15), 10777–10788 (2023)

    Article  Google Scholar 

  50. Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: Stable-baselines3: reliable reinforcement learning implementations. J. Mach. Learn. Res. 22(268), 1–8 (2021)

    Google Scholar 

  51. Rice, L., Wong, E., Kolter, Z.: Overfitting in adversarially robust deep learning. In: Proceedings of the 37th International Conference on Machine Learning (2020)

  52. Ring, M., Wunderlich, S., Scheuring, D., Landes, D., Hotho, A.: A survey of network-based intrusion detection data sets. Comput. Security 86, 147–167 (2019)

    Article  Google Scholar 

  53. Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: International conference on machine learning (2015)

  54. Schulman, J., Moritz, P., Levine, S., Jordan, M.I., Abbeel, P.: High-dimensional continuous control using generalized advantage estimation (2016). http://arxiv.org/abs/1506.02438

  55. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. http://arxiv.org/abs/1707.06347 (2017)

  56. Servin, A., Kudenko, D.: Multi-agent Reinforcement Learning for Intrusion Detection. In: Adaptive Agents and Multi-Agent Systems III. Adaptation and Multi-Agent Learning (2008)

  57. Sewak, M., Sahay, S.K., Rathore, H.: Deep reinforcement learning in the advanced cybersecurity threat detection and protection. Inf. Syst. Front. 25(2), 589–611 (2023)

    Google Scholar 

  58. Sheatsley, R., Papernot, N., Weisman, M.J., Verma, G., McDaniel, P.: Adversarial examples for network intrusion detection systems. J. Comput. Security 30, 727 (2022)

    Article  Google Scholar 

  59. Sun, J., Zhang, T., Xie, X., Ma, L., Zheng, Y., Chen, K., Liu, Y.: Stealthy and efficient adversarial attacks against deep reinforcement learning. In: AAI Conference on Artificial Intelligence (2020)

  60. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems. Curran Associates, Inc. (2014)

  61. Sutton, R.S.: Learning to predict by the methods of temporal differences. Mach. Learn. 3, 9–44 (1988)

    Article  Google Scholar 

  62. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. In: 2nd International Conference on Learning Representations, ICLR (2014)

  63. Tavallaee, M., Bagheri, E., Lu, W., Ghorbani, A.A.: A detailed analysis of the kdd cup 99 data set. In: IEEE symposium on computational intelligence for security and defense applications (2009)

  64. Umer, M.A., Junejo, K.N., Jilani, M.T., Mathur, A.P.: Machine learning for intrusion detection in industrial control systems: applications, challenges, and recommendations. Int. J. Critic. Infrastruct. Protect. 38, 100516 (2022)

    Article  Google Scholar 

  65. Wang, Z.: Deep learning-based intrusion detection with adversaries. IEEE Access 6, 38367–38384 (2018)

    Article  Google Scholar 

  66. Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., De Freitas, N.: Dueling network architectures for deep reinforcement learning. In: 33rd International Conference on International Conference on Machine Learning - Volume 48 (2016)

  67. Xiong, Z., Eappen, J., Zhu, H., Jagannathan, S.: Defending observation attacks in deep reinforcement learning via detection and denoising. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 235–250 (2022)

  68. Xu, X., Xie, T.: A Reinforcement Learning Approach for Host-Based Intrusion Detection Using Sequences of System Calls. In: Advances in Intelligent Computing (2005)

  69. Yilmaz, I., Masum, R., Siraj, A.: Addressing imbalanced data problem with generative adversarial network for intrusion detection. In: IEEE 21st International Conference on Information Reuse and Integration for Data Science (2020)

  70. Yuan, X., He, P., Zhu, Q., Li, X.: Adversarial examples: attacks and defenses for deep learning. IEEE Trans. Neural Netw. Learn. Syst. 30(9), 2805–2824 (2019)

Download references

Acknowledgements

This work was supported by Mitacs through the Mitacs Accelerate International program and the CRITiCAL chair. It was enabled in part by support provided by Calcul Québec, Compute Ontario, the BC DRI Group, and the Digital Research Alliance of Canada.

Author information

Authors and Affiliations

  1. Polytechnique Montréal, Montreal, Canada

    Mohamed Amine Merzouk, Christopher Neal, Joséphine Delas, Nora Boulahia-Cuppens & Frédéric Cuppens

  2. IRT SystemX, Palaiseau, France

    Mohamed Amine Merzouk, Christopher Neal, Joséphine Delas & Reda Yaich

Authors
  1. Mohamed Amine Merzouk
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christopher Neal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joséphine Delas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Reda Yaich
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nora Boulahia-Cuppens
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Frédéric Cuppens
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohamed Amine Merzouk.

Ethics declarations

Conflict of interest

The authors have no conflict of interest to declare.

Ethical approval

The authors confirm their compliance with ethical standards.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

IoV case study (CICIoV2024)

IoV case study (CICIoV2024)

In this section, we apply our method to the CICIoV2024 dataset. In contrast with the two previous datasets, NSL-KDD and UNSW-NB15, which contain data from TCP/IP networks, CICIoV2024 consists of IoV data captured from real car components. Here we demonstrate the applicability of our method to a more recent dataset and a different type of data. More precisely, we focus on the impact of the DRL algorithm used by the detection agent on the robustness. We use all 5 algorithms described in Sect.  3.1 with 4 hidden layer sizes (1, 2, 4, 8) and 4 hidden unit sizes (32, 64, 128, 256), with a total of 80 combinations.

Figure 6 shows a line plot of the FNR for each network width and depth combination. The lines correspond to different DRL algorithms (FGSM and BIM are represented with continuous and dashed lines, respectively). On the other hand, Fig. 5 shows the FNR mean and standard deviation averaged together over all neural network architectures, it is the summarized version of Fig. 6.

First, we observe the performance of DRL agents before adversarial perturbations (\(\epsilon =0\)). Figure 6 shows an FNR equal to 0 for almost all DRL algorithms on all architectures. Our results also show an F1 score equal to 1, demonstrating a perfect classification. These results are due to the relatively simple structure of the CICIoV2024 dataset compared to network datasets, the former consists of 8 values each encoded in one byte and an ID. IoV components are designed to be lightweight and energy-efficient, limiting the size of data communication to the minimum.

Similarly to Fig. 3, Figure  5 shows some variance because the values are averaged over 16 different architectures. We can nonetheless observe the behavioral patterns of DRL algorithms on CICIoV2024. First, we notice that the properties of the datasets disadvantage A2C which ends up with the highest average FNR, reaching an average around 0.8 at \(\epsilon =0.1\). Similarly to previous datasets, DQN, QRDQN, and PPO show comparable performance in terms of robustness. Their FNR evolves similarly and reaches 0.5 to 0.6 when \(\epsilon =0.1\). Finally, TRPO maintains the best robustness to adversarial examples, especially against FGSM, with a low FNR below \(\epsilon =0.6\) and an average FNR of 0.37.

Furthermore, the detailed Fig. 6 shows the specific behavior of the algorithms with different neural network architectures. It shows the relative stability of A2C across architectures, with an FNR increasing starting from around \(\epsilon =0.04\). PPO and TRPO show a similar behavior overall, in addition to a decrease in FNR with larger neural networks (especially 8 hidden layers). Moreover, DQN and QRDQN show a higher sensitivity to adversarial examples on smaller neural networks (up to 4 hidden layers and 64 units), with high FNR values starting from \(\epsilon =0.02\). The two algorithms are also the only ones that reach an FNR equal to 1 on some architectures. Finally, Regardless of the DRL algorithm used, agents with the largest neural network (8 hidden layers of 256 units) achieved the best robustness, with an FNR below 0.4 for all algorithms (except QRDQN against BIM).

In conclusion, the performance of DRL-based intrusion detection is not limited to the network domain, it extends to other critical tasks, and so do their vulnerabilities. In this section, we have shown the efficiency of DRL detection agents in detecting attacks against IoV devices using the CICIoV2024 dataset. Across different architectures, our agents present a perfect detection across almost all DRL algorithms and neural network architectures. These performances are explained by the relative simplicity and lightheartedness of IoV data. However, DRL agents also present similar vulnerabilities to adversarial attacks in IoV. We have shown how gradient-based adversarial examples with a perturbation amplitude \(\epsilon \le 0.1\) could considerably increase the FNR (attacks labeled as benign). This case study on IoV data demonstrates the seriousness of the adversarial examples threat in critical infrastructures and the need for further investigation on the robustness of DRL agents in IoV environments.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Merzouk, M.A., Neal, C., Delas, J. et al. Adversarial robustness of deep reinforcement learning-based intrusion detection. Int. J. Inf. Secur. 23, 3625–3651 (2024). https://doi.org/10.1007/s10207-024-00903-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10207-024-00903-2

Keywords

Search

Navigation