research-article

Leveraging Reinforcement Learning and Generative Adversarial Networks to Craft Mutants of Windows Malware against Black-box Malware Detectors

Authors:

Tran Duc Luong,

Nguyen Hoang Quoc An,

Quyen Nguyen Huu,

Hoang Khoa Nghi,

Van-Hau PhamAuthors Info & Claims

SoICT '22: Proceedings of the 11th International Symposium on Information and Communication Technology

Pages 31 - 38

https://doi.org/10.1145/3568562.3568636

Published: 01 December 2022 Publication History

Abstract

To build an effective malware detector, it is required to collect a diversity of malware samples and their evolution, since malware authors always try to evade detectors through strategies of malware mutation. So, this paper explores the ability to craft mutants of malware for gathering numerous mutated samples in training a machine learning (ML)-based malware detector. Specifically, we leverage Reinforcement Learning (RL) and Generative Adversarial Networks (GAN) to generate adversarial malware samples against ML-based detectors. The more we use this approach with different targeted antivirus and malware samples in training the RL agent as a malware mutator, the more it learns how to avoid black box malware detectors. The experimental results in real-world dataset indicate that RL can help GAN in crafting variants of malware with executability preservation to evade ML-based detectors and VirusTotal. Finally, this approach can be used as an automated tool for benchmarking the robustness of malware detectors against the metamorphic malwares.

References

[1]

Hyrum S. Anderson and Phil Roth. 2018. EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models. https://doi.org/10.48550/ARXIV.1804.04637

[2]

Ömer Aslan Aslan and Refik Samet. 2020. A Comprehensive Review on Malware Detection Approaches. IEEE Access 8(2020), 6249–6271. https://doi.org/10.1109/ACCESS.2019.2963724

[3]

Ben Athiwaratkun and Jack W. Stokes. 2017. Malware classification with LSTM and GRU language models and a character-level CNN. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2482–2486. https://doi.org/10.1109/ICASSP.2017.7952603

Digital Library

[4]

BackupGGCode. 2015. PE Bliss - Cross-Platform Portable Executable C++ Library. https://github.com/BackupGGCode/portable-executable-library.

[5]

Ho Bae, Younghan Lee, Yohan Kim, Uiwon Hwang, Sungroh Yoon, and Yunheung Paek. 2021. Learn2Evade: Learning-Based Generative Model for Evading PDF Malware Classifiers. IEEE Transactions on Artificial Intelligence 2, 4(2021), 299–313. https://doi.org/10.1109/TAI.2021.3103139

[6]

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. https://doi.org/10.48550/ARXIV.1606.01540

[7]

Lingwei Chen, Yanfang Ye, and Thirimachos Bourlai. 2017. Adversarial Machine Learning in Malware Detection: Arms Race between Evasion Attack and Defense. In 2017 European Intelligence and Security Informatics Conference (EISIC). 99–106. https://doi.org/10.1109/EISIC.2017.21

[8]

Fred Cohen. 1987. Computer viruses: Theory and experiments. Computers & Security 6, 1 (1987), 22–35. https://doi.org/10.1016/0167-4048(87)90122-2

Digital Library

[9]

CyberForce. 2015. PEsidious - Malware Mutation Using Reinforcement Learning and Generative Adversarial Networks. https://github.com/CyberForce/Pesidious.

[10]

George E. Dahl, Jack W. Stokes, Li Deng, and Dong Yu. 2013. Large-scale malware classification using random projections and neural networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 3422–3426. https://doi.org/10.1109/ICASSP.2013.6638293

[11]

Luca Demetrio, Scott E. Coull, Battista Biggio, Giovanni Lagorio, Alessandro Armando, and Fabio Roli. 2021. Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection. ACM Trans. Priv. Secur. 24, 4, Article 27 (9 2021), 31 pages. https://doi.org/10.1145/3473039

Digital Library

[12]

Zhiyang Fang, Junfeng Wang, Boya Li, Siqi Wu, Yingjie Zhou, and Haiying Huang. 2019. Evading Anti-Malware Engines With Deep Reinforcement Learning. IEEE Access 7(2019), 48867–48879. https://doi.org/10.1109/ACCESS.2019.2908033

[13]

Zhiyang Fang, Junfeng Wang, Boya Li, Siqi Wu, Yingjie Zhou, and Haiying Huang. 2019. Evading Anti-Malware Engines With Deep Reinforcement Learning. IEEE Access 7(2019), 48867–48879. https://doi.org/10.1109/ACCESS.2019.2908033

[14]

Daniel Gibert, Carles Mateu, and Jordi Planes. 2020. The Rise of Machine Learning for Detection and Classification of Malware: Research Developments, Trends and Challenges. J. Netw. Comput. Appl. 153, C (3 2020), 22 pages. https://doi.org/10.1016/j.jnca.2019.102526

Digital Library

[15]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems 27 (2014).

[16]

Weiwei Hu and Ying Tan. 2017. Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN. CoRR abs/1702.05983(2017). arXiv:1702.05983http://arxiv.org/abs/1702.05983

[17]

Raphael Labaca-Castro, Sebastian Franz, and Gabi Dreo Rodosek. 2021. AIMED-RL: Exploring Adversarial Malware Examples with Reinforcement Learning. In Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Proceedings, Part IV (Bilbao, Spain). Springer-Verlag, Berlin, Heidelberg, 37–52. https://doi.org/10.1007/978-3-030-86514-6_3

Digital Library

[18]

Deqiang Li, Qianmu Li, Yanfang (Fanny) Ye, and Shouhuai Xu. 2021. Arms Race in Adversarial Malware Detection: A Survey. ACM Comput. Surv. 55, 1, Article 15 (11 2021), 35 pages. https://doi.org/10.1145/3484491

Digital Library

[19]

Davide Maiorca, Battista Biggio, and Giorgio Giacinto. 2019. Towards Adversarial Malware Detection: Lessons Learned from PDF-Based Attacks. ACM Comput. Surv. 52, 4, Article 78 (8 2019), 36 pages. https://doi.org/10.1145/3332184

Digital Library

[20]

Nuno Martins, José Magalhães Cruz, Tiago Cruz, and Pedro Henriques Abreu. 2020. Adversarial Machine Learning Applied to Intrusion and Malware Scenarios: A Systematic Review. IEEE Access 8(2020), 35403–35419. https://doi.org/10.1109/ACCESS.2020.2974752

[21]

Athiq Mohammed, G. Viswanath, K. babu, and T. Anuradha. 2020. Malware Detection in Executable Files Using Machine Learning. 277–284. https://doi.org/10.1007/978-3-030-24322-7_36

[22]

Ori Or-Meir, Nir Nissim, Yuval Elovici, and Lior Rokach. 2019. Dynamic Malware Analysis in the Modern Era—A State of the Art Survey. ACM Comput. Surv. 52, 5, Article 88 (9 2019), 48 pages. https://doi.org/10.1145/3329786

Digital Library

[23]

Razvan Pascanu, Jack W. Stokes, Hermineh Sanossian, Mady Marinescu, and Anil Thomas. 2015. Malware classification with recurrent networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1916–1920. https://doi.org/10.1109/ICASSP.2015.7178304

[24]

Quarkslab. 2017. LIEF - Library to instrument executable formats. https://github.com/lief-project.

[25]

Umara Urooj, Bander Ali Saleh Al-rimy, Anazida Zainal, Fuad A. Ghaleb, and Murad A. Rassam. 2022. Ransomware Detection Using the Dynamic Analysis and Machine Learning: A Survey and Research Directions. Applied Sciences 12, 1 (2022). https://doi.org/10.3390/app12010172

Cited By

Khalifa MAlmomani IEl-Shafai W(2024)SAMA: A Comprehensive Smart Automated Malware Analyzer Empowered by ChatGPT Integration2024 IEEE 30th International Conference on Telecommunications (ICT)10.1109/ICT62760.2024.10606026(1-6)Online publication date: 24-Jun-2024
https://doi.org/10.1109/ICT62760.2024.10606026
Ilca LLucian OBalan T(2023)Enhancing Cyber-Resilience for Small and Medium-Sized Organizations with Prescriptive Malware Analysis, Detection and ResponseSensors10.3390/s2315675723:15(6757)Online publication date: 28-Jul-2023
https://doi.org/10.3390/s23156757
Arif RAslam MAl-Otaibi SMartinez-Enriquez ASaba TBahaj SRehman A(2023)A Deep Reinforcement Learning Framework to Evade Black-Box Machine Learning Based IoT Malware Detectors Using GAN-Generated Influential FeaturesIEEE Access10.1109/ACCESS.2023.333464511(133717-133729)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3334645

Index Terms

Leveraging Reinforcement Learning and Generative Adversarial Networks to Craft Mutants of Windows Malware against Black-box Malware Detectors
1. Security and privacy
  1. Intrusion/anomaly detection and malware mitigation
    1. Malware and its mitigation

Recommendations

Enhancing Machine Learning Based Malware Detection Model by Reinforcement Learning
ICCNS '18: Proceedings of the 8th International Conference on Communication and Network Security

Malware detection is getting more and more attention due to the rapid growth of new malware. As a result, machine learning (ML) has become a popular way to detect malware variants. However, machine learning models can also be cheated. Through ...
An Evolutionary based Generative Adversarial Network Inspired Approach to Defeating Metamorphic Malware
GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary Computation

Defeating dangerous families of malware like polymorphic and metamorphic malware have become well studied due to their increased attacks on computer systems and network. Traditional Machine Learning (ML) models have been used in detecting this malware,...
Metamorphic malware detection using base malware identification approach

Malware is a malicious program that is intentionally developed to harm computer systems. Because the metamorphic malwares are advanced in nature, they mutate their code in each generation by employing code obfuscation techniques to thwart detection. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SoICT '22: Proceedings of the 11th International Symposium on Information and Communication Technology

December 2022

474 pages

ISBN:9781450397254

DOI:10.1145/3568562

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SoICT 2022

SoICT 2022: The 11th International Symposium on Information and Communication Technology

December 1 - 3, 2022

Hanoi, Vietnam

Acceptance Rates

Overall Acceptance Rate 147 of 318 submissions, 46%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
173
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)5

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Khalifa MAlmomani IEl-Shafai W(2024)SAMA: A Comprehensive Smart Automated Malware Analyzer Empowered by ChatGPT Integration2024 IEEE 30th International Conference on Telecommunications (ICT)10.1109/ICT62760.2024.10606026(1-6)Online publication date: 24-Jun-2024
https://doi.org/10.1109/ICT62760.2024.10606026
Ilca LLucian OBalan T(2023)Enhancing Cyber-Resilience for Small and Medium-Sized Organizations with Prescriptive Malware Analysis, Detection and ResponseSensors10.3390/s2315675723:15(6757)Online publication date: 28-Jul-2023
https://doi.org/10.3390/s23156757
Arif RAslam MAl-Otaibi SMartinez-Enriquez ASaba TBahaj SRehman A(2023)A Deep Reinforcement Learning Framework to Evade Black-Box Machine Learning Based IoT Malware Detectors Using GAN-Generated Influential FeaturesIEEE Access10.1109/ACCESS.2023.333464511(133717-133729)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3334645
Rigaki MGarcia S(2023)The Power of MEME: Adversarial Malware Creation with Model-Based Reinforcement LearningComputer Security – ESORICS 202310.1007/978-3-031-51482-1_3(44-64)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-51482-1_3

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents