research-article

Enhancing Machine Learning Based Malware Detection Model by Reinforcement Learning

Authors:

Wenhua LiAuthors Info & Claims

ICCNS '18: Proceedings of the 8th International Conference on Communication and Network Security

Pages 74 - 78

https://doi.org/10.1145/3290480.3290494

Published: 02 November 2018 Publication History

Abstract

Malware detection is getting more and more attention due to the rapid growth of new malware. As a result, machine learning (ML) has become a popular way to detect malware variants. However, machine learning models can also be cheated. Through reinforcement learning (RL), we can generate new malware samples which can bypass the detection of machine learning. In this paper, a RL model on malware generation named gym-plus is designed. Gym-plus is built based on gym-malware with some improvements. As a result, the probability of evading machine learning based static PE malware detection models is increased by 30%. Based on these newly generated samples, we retrain our detecting model to detect unknown threats. In our test, the detection accuracy of malware increased from 15.75% to 93.5%.

References

[1]

Anderson HS, Kharkar A, Filar B, Evans D, Roth P et al. Learning to evade static PE machine learning malware models via reinforcement learning{J}. arXiv preprint arXiv:1801.08917v2, 2018.

[2]

Anderson HS, Roth P, et al. EMBER: an open dataset for training static PE maiware machine learning models{J}. arXiv preprint arXiv:1804.04637v2, 2018.

[3]

Schultz M G, Eskin E, Zadok F, Stolfo S J, et al. Data mining methods for detection of new malicious executables{C}. In Security and Privacy, 2001. S&P 2001. Proceedings. 2001 IEEE Symposium on, pages 38--49. IEEE, 2001.

Digital Library

[4]

Huang A, Al-Dujaili A, Hemberg E, O'Reilly U M, et al. Adversarial deep learning for robust detection of binary encoded malware{J}. arXiv preprint arXiv:1801.02950, 2018.

[5]

Shafq M Z, Tabish S M, Mirza F, Farooq M, et al. A framework for efcient mining of structural information to detect zero-day malicious portable executables. Technical report, TR-nexGINRC-2009-21, January, 2009, available at http://www. nexginrc. org/papers/tr21-zubair. pdf, 2009.

[6]

Nataraj L, Karthikeyan S, Jacob G, Manjunath B S, et al. Malware images: visualization and automatic classification{J}. ISBN 987-1-4503-0679-9.

[7]

Saxe J and Berlin K. Deep neural network based malware detection using two dimensional binary program features{C}. In Malicious and Unwanted Software, 2015 10th International Conference on, pages 11--20. IEEE, 2015.

Digital Library

[8]

Raff E, Barker J, Sylvester J, Brandon R, Catanzaro B Nicholas C, et al. Malware detection by eating a whole exe{J}. arXiv preprint arXiv:1710.09435, 2017.

[9]

Masabo E, Kaawaase K S, Sansa O J. Big Data: Deep Learning for detecting Malware{C}. 2018 ACM/IEEE.

Digital Library

[10]

Dang H, Yue H, Chang E C, et al. Evading classifer in the dark: Guiding unpredictable morphing using binary output blackboxes{J}. arXiv preprint arXiv:1705.07535, 2017.

[11]

Hu W and Tan Y. Generating adversarial malware examples for black-box attacks based on GAN{J}. arXiv preprint arXiv:1702.05983, 2017.

[12]

Sutton R S and Barto A G. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.

Digital Library

[13]

Quarkslab. LIEF: library for instrumenting executable fles. https://lief.quarkslab.com/, 2017-2018.

[14]

Bradley A P. The use of the area under the ROC curve in the evaluation of machine learning algorithms{J}.

Digital Library

[15]

Dulac-Arnold G, Evans R, Hasselt H, Sunehag P, Lillicrap T, Hunt J, Mann T, Weber T, Degris T, Coppin B et al. Deep reinforcement learning in large discrete action spaces{J}. arXiv preprint arXiv:1512.07679, 2015.

[16]

Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, Ye Q, Liu T Y, et al. Lightgbm: A highly effcient gradient boosting decision tree{J}. In Advances in Neural Information Processing Systems, pages 3149--3157, 2017.

Digital Library

[17]

Virustotal-free online virus, malware and url scanner. https://www.virustotal.com/en. Accessed: 2018-03-09.

Cited By

Koch LBegoli E(2025)Adversarial Binaries: AI-guided Instrumentation Methods for Malware Detection EvasionACM Computing Surveys10.1145/370657357:5(1-36)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3706573
Qu ZLing XWang TChen XJi SWu C(2024)AdvSQLi: Generating Adversarial SQL Injections Against Real-World WAF-as-a-ServiceIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.335091119(2623-2638)Online publication date: 2024
https://doi.org/10.1109/TIFS.2024.3350911
Mwangi MCheng S(2024)An Adversarial Attack on ML-Based IoT Malware Detection Using Binary Diversification TechniquesIEEE Access10.1109/ACCESS.2024.3513713(1-1)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3513713
Show More Cited By

Index Terms

Enhancing Machine Learning Based Malware Detection Model by Reinforcement Learning
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Markov decision processes
2. Security and privacy
  1. Intrusion/anomaly detection and malware mitigation
    1. Intrusion detection systems
    2. Malware and its mitigation

Recommendations

Self-propagating Malware Containment via Reinforcement Learning
Machine Learning and Knowledge Extraction
Abstract
We introduce a reinforcement learning based containment system for self-propagating malware in local networks. The system is trained with real-world software and malware and leverages a network of virtual machines for execution and propagation. ...
A Malware Detection Framework Based on Forensic and Unsupervised Machine Learning Methodologies
ICSCA '20: Proceedings of the 2020 9th International Conference on Software and Computer Applications

The detection of malware intrusion requires the identification of its signature. However, it is a complex task due to the malware sophisticated ability to evade security mechanisms deployed by cybersecurity practitioners. Evasion is possible due to ...
Web Bot Detection Evasion Using Deep Reinforcement Learning
ARES '22: Proceedings of the 17th International Conference on Availability, Reliability and Security

Web bots are vital for the web as they can be used to automate several actions, some of which would have otherwise been impossible or very time consuming. These actions can be benign, such as website testing and web indexing, or malicious, such as ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICCNS '18: Proceedings of the 8th International Conference on Communication and Network Security

November 2018

166 pages

ISBN:9781450365673

DOI:10.1145/3290480

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICCNS 2018

ICCNS 2018: 2018 the 8th International Conference on Communication and Network Security

November 2 - 4, 2018

Qingdao, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
874
Total Downloads

Downloads (Last 12 months)76
Downloads (Last 6 weeks)8

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Koch LBegoli E(2025)Adversarial Binaries: AI-guided Instrumentation Methods for Malware Detection EvasionACM Computing Surveys10.1145/370657357:5(1-36)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3706573
Qu ZLing XWang TChen XJi SWu C(2024)AdvSQLi: Generating Adversarial SQL Injections Against Real-World WAF-as-a-ServiceIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.335091119(2623-2638)Online publication date: 2024
https://doi.org/10.1109/TIFS.2024.3350911
Mwangi MCheng S(2024)An Adversarial Attack on ML-Based IoT Malware Detection Using Binary Diversification TechniquesIEEE Access10.1109/ACCESS.2024.3513713(1-1)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3513713
Shahin MMaghanaki MHosseinzadeh AChen F(2024)Advancing Network Security in Industrial IoT: A Deep Dive into AI-Enabled Intrusion Detection SystemsAdvanced Engineering Informatics10.1016/j.aei.2024.10268562(102685)Online publication date: Oct-2024
https://doi.org/10.1016/j.aei.2024.102685
Louati FKtata FAmous I(2024)Enhancing Intrusion Detection Systems with Reinforcement Learning: A Comprehensive Survey of RL-based Approaches and TechniquesSN Computer Science10.1007/s42979-024-03001-15:6Online publication date: 21-Jun-2024
https://doi.org/10.1007/s42979-024-03001-1
Van Ouytsel CLegay ALucca SWauters D(2024)Assessing Static and Dynamic Features for Packing DetectionThe Combined Power of Research, Education, and Dissemination10.1007/978-3-031-73887-6_12(146-166)Online publication date: 23-Oct-2024
https://doi.org/10.1007/978-3-031-73887-6_12
D’Hondt ABertrand Van Ouytsel CLegay A(2024)Extended Abstract: Evading Packing Detection: Breaking Heuristic-Based Static DetectorsDetection of Intrusions and Malware, and Vulnerability Assessment10.1007/978-3-031-64171-8_9(174-183)Online publication date: 9-Jul-2024
https://doi.org/10.1007/978-3-031-64171-8_9
Corlătescu DDinu AGăman MSumedrea POh ANaumann TGloberson ASaenko KHardt MLevine S(2023)EMBERSimProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3667283(26722-26743)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3667283
Vasani VBairwa AJoshi SPljonkin AKaur MAmoon M(2023)Comprehensive Analysis of Advanced Techniques and Vital Tools for Detecting Malware IntrusionElectronics10.3390/electronics1220429912:20(4299)Online publication date: 17-Oct-2023
https://doi.org/10.3390/electronics12204299
Zhan DBai WLiu XHu YZhang LGuo SPan Z(2023)PSP-Mal: Evading Malware Detection via Prioritized Experience-based Reinforcement Learning with Shapley PriorProceedings of the 39th Annual Computer Security Applications Conference10.1145/3627106.3627178(580-593)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3627106.3627178
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents