Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems
Authors: 
Ahmed Abusnaina, Afsah Anwar, Sultan Alshamrani, Abdulrahman Alabduljabbar, RhongHo Jang, DaeHun Nyang, David MohaisenAuthors Info & Claims
Ahmed Abusnaina, Afsah Anwar, Sultan Alshamrani, Abdulrahman Alabduljabbar, RhongHo Jang, DaeHun Nyang, David MohaisenAuthors Info & ClaimsPages 308 - 320
Abstract
The rapid growth of the Internet of Things (IoT) devices is paralleled by them being on the front-line of malicious attacks. This has led to an explosion in the number of IoT malware, with continued mutations, evolution, and sophistication. Malware samples are detected using machine learning (ML) algorithms alongside the traditional signature-based methods. Although ML-based detectors improve the detection performance, they are susceptible to malware evolution and sophistication, making them limited to the patterns that they have been trained upon. This continuous trend motivates large body of literature on malware analysis and detection research, with many systems emerging constantly, outperforming their predecessors. In this paper, we systematically examine the state-of-the-art malware detection approaches, that utilize various representation and learning techniques, under a range of adversarial settings. Our analyses highlight the instability of the proposed detectors in learning patterns that distinguish the benign from the malicious software. The results exhibit that software mutations with functionality-preserving operations, such as stripping and padding, significantly deteriorate the accuracy of such detectors. Additionally, our analysis of the industry-standard malware detectors shows their instability to the malware mutations. Through extensive experiments, we highlight the gap between the capabilities of the adversary and that of the existing malware detectors. The evaluations and analyses show that the optimal malware detection system is nowhere near and calls for the community to streamline their efforts towards testing the robustness of malware detectors to different manipulation techniques.
References
[1]
2019. CyberIOCs. Available at [Online]: https://freeiocs.cyberiocs.pro/.
[2]
2019. Radare2. Available at [Online]: https://rada.re/r/.
[3]
2019. VirusTotal. Available at [Online]: https://www.virustotal.com.
[4]
2022. Smart Yet Flawed: IoT Device Vulnerabilities Explained. Available at [Online]: https://bit.ly/2MBykDx.
[5]
2022. Strip: GNU binary Utility. Available at [Online]: https://sourceware.org/binutils/docs/binutils/strip.html.
[6]
2022. UCL Data Compression Library. Available at [Online]: http://www.oberhumer.com/opensource/ucl/.
[7]
2022. UPX: the Ultimate Packer for eXecutables. Available at [Online]: https://upx.github.io/.
[8]
2022. VirusShare. Available at [Online]: https://virusshare.com/.
[9]
Ahmed Abusnaina, Hisham Alasmary, Mohammed Abuhamad, Saeed Salem, DaeHun Nyang, and Aziz Mohaisen. 2019. Subgraph-Based Adversarial Examples Against Graph-Based IoT Malware Detection Systems. In International Conference on Computational Data and Social Networks. 268–281.
[10]
Ahmed Abusnaina, Aminollah Khormali, Hisham Alasmary, Jeman Park, Afsah Anwar, and Aziz Mohaisen. 2019. Adversarial Learning Attacks on Graph-based IoT Malware Detection Systems. In IEEE International Conference on Distributed Computing Systems, ICDCS.
[11]
Hojjat Aghakhani, Fabio Gritti, Francesco Mecca, Martina Lindorfer, Stefano Ortolani, Davide Balzarotti, Giovanni Vigna, and Christopher Kruegel. 2020. When Malware is Packin’Heat; Limits of Machine Learning Classifiers Based on Static Analysis Features. In Network and Distributed Systems Security (NDSS) Symposium.
[12]
Mansour Ahmadi, Dmitry Ulyanov, Stanislav Semenov, Mikhail Trofimov, and Giorgio Giacinto. 2016. Novel feature extraction, selection and fusion for effective malware family classification. In Proceedings of ACM conference on data and application security and privacy. 183–194.
[13]
Hisham Alasmary, Aminollah Khormali, Afsah Anwar, Jeman Park, Jinchun Choi, Ahmed Abusnaina, Amro Awad, DaeHun Nyang, and Aziz Mohaisen. 2019. Analyzing and Detecting Emerging Internet of Things Malware: A Graph-based Approach. IEEE Internet of Things Journal(2019).
[14]
Omar Alrawi, Charles Lever, Kevin Valakuzhy, Kevin Snow, Fabian Monrose, Manos Antonakakis, 2021. The Circle Of Life: A {Large-Scale} Study of The {IoT} Malware Lifecycle. In 30th USENIX Security Symposium (USENIX Security 21). 3505–3522.
[15]
Manos Antonakakis, Tim April, Michael Bailey, Matt Bernhard, Elie Bursztein, Jaime Cochran, Zakir Durumeric, J Alex Halderman, Luca Invernizzi, Michalis Kallitsis, 2017. Understanding the mirai botnet. In USENIX security symposium (USENIX Security). 1093–1110.
[16]
Afsah Anwar, Hisham Alasmary, Jeman Park, An Wang, Songqing Chen, and David Mohaisen. 2020. Statically Dissecting Internet of Things Malware: Analysis, Characterization, and Detection. In International Conference on Information and Communications Security. Springer, 443–461.
[17]
Danilo Bruschi, Lorenzo Martignoni, and Mattia Monga. 2006. Detecting self-mutating malware using control-flow graph matching. In International conference on detection of intrusions and malware, and vulnerability assessment. Springer, 129–143.
[18]
Nicholas Carlini and David A. Wagner. 2017. Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy, SP. 39–57.
[19]
Nicholas Carlini and David A. Wagner. 2017. Towards Evaluating the Robustness of Neural Networks. In Proceedings of the IEEE Symposium on Security and Privacy. 39–57.
[20]
Zhenxiang Chen, Qiben Yan, Hongbo Han, Shanshan Wang, Lizhi Peng, Lin Wang, and Bo Yang. 2018. Machine learning based mobile malware detection using highly imbalanced network traffic. Inf. Sci. 433-434(2018), 346–364.
[21]
Emanuele Cozzi, Mariano Graziano, Yanick Fratantonio, and Davide Balzarotti. 2018. Understanding Linux Malware. In IEEE Symposium on Security & Privacy.
[22]
Emanuele Cozzi, Pierre-Antoine Vervier, Matteo Dell’Amico, Yun Shen, Leyla Bilge, and Davide Balzarotti. 2020. The tangled genealogy of IoT malware. In Annual Computer Security Applications Conference. 1–16.
[23]
Developers. 2019. GitHub. Available at [Online]: https://github.com/.
[24]
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations, ICLR.
[25]
Mariano Graziano, Davide Canali, Leyla Bilge, Andrea Lanzi, and Davide Balzarotti. 2015. Needles in a Haystack: Mining Information from Public Dynamic Analysis Sandboxes for Malware Intelligence. In 24th USENIX Security Symposium (USENIX Security 15).
[26]
Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick McDaniel. 2017. Adversarial examples for malware detection. In European Symposium on Research in Computer Security. Springer, 62–79.
[27]
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. 2018. Countering Adversarial Images using Input Transformations. In International Conference on Learning Representations, ICLR.
[28]
Shengyuan Hu, Tao Yu, Chuan Guo, Wei-Lun Chao, and Kilian Q Weinberger. 2019. A new defense against adversarial images: Turning a weakness into a strength. In Neural Information Processing Systems, NeurIPS.
[29]
Weiwei Hu and Ying Tan. 2017. Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN. arXiv preprint arXiv:1702.05983 abs/1702.05983 (2017).
[30]
Pankaj Jalote. 2012. An integrated approach to software engineering. Springer Science & Business Media.
[31]
Kesav Kancherla and Srinivas Mukkamala. 2013. Image visualization based malware detection. In IEEE Symposium on Computational Intelligence in Cyber Security (CICS). IEEE, 40–44.
[32]
Bojan Kolosnjaji, Ambra Demontis, Battista Biggio, Davide Maiorca, Giorgio Giacinto, Claudia Eckert, and Fabio Roli. 2018. Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables. In The European Signal Processing Conference, EUSIPCO. 533–537.
[33]
Felix Kreuk, Assi Barak, Shir Aviv-Reuven, Moran Baruch, Benny Pinkas, and Joseph Keshet. 2018. Deceiving end-to-end deep learning malware detectors using adversarial examples. In Workshop on Security in Machine Learning (NIPS).
[34]
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2017. Adversarial examples in the physical world. In the 5th International Conference on Learning Representations, ICLR.
[35]
Jin Li, Lichao Sun, Qiben Yan, Zhiqiang Li, Witawas Srisa-an, and Heng Ye. 2018. Significant Permission Identification for Machine-Learning-Based Android Malware Detection. IEEE Trans. Ind. Informatics 14, 7 (2018), 3216–3225.
[36]
Zhiqiang Li, Jun Sun, Qiben Yan, Witawas Srisa-an, and Yutaka Tsutano. 2019. Obfusifier: Obfuscation-Resistant Android Malware Detection System. In Security and Privacy in Communication Networks - 15th EAI International Conference, SecureComm 2019, Orlando, FL, USA, October 23-25, 2019, Proceedings, Part I(Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, Vol. 304), Songqing Chen, Kim-Kwang Raymond Choo, Xinwen Fu, Wenjing Lou, and Aziz Mohaisen (Eds.). Springer, 214–234.
[37]
Francesco Mercaldo and Antonella Santone. 2020. Deep learning for image-based mobile malware detection. Journal of Computer Virology and Hacking Techniques (2020), 1–15.
[38]
Aziz Mohaisen, Omar Alrawi, and Manar Mohaisen. 2015. AMAL: High-fidelity, behavior-based automated malware analysis and classification. Computers & Security 52 (2015), 251–266.
[39]
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks. In IEEE Conference on Computer Vision and Pattern Recognition. 2574–2582.
[40]
Ryan Elfmaster O’Neill. 2016. Learning Linux Binary Analysis. Packt Publishing.
[41]
Hamed Haddad Pajouh, Ali Dehghantanha, Raouf Khayami, and Kim-Kwang Raymond Choo. 2018. A deep Recurrent Neural Network based approach for Internet of Things malware threat hunting. Future Gener. Comput. Syst. 85 (2018), 88–96.
[42]
Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. 2017. Practical Black-Box Attacks against Machine Learning. In Proceedings of the ACM on Asia Conference on Computer and Communications Security, AsiaCCS. 506–519.
[43]
Nicolas Papernot, Patrick D. McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. 2016. The Limitations of Deep Learning in Adversarial Settings. In IEEE European Symposium on Security and Privacy. 372–387.
[44]
Giorgio Severi, Jim Meyer, Scott Coull, and Alina Oprea. 2021. Exploring Backdoor Poisoning Attacks Against Malware Classifiers. In USENIX security symposium (USENIX Security). 1093–1110.
[45]
Shigen Shen, Longjun Huang, Haiping Zhou, Shui Yu, En Fan, and Qiying Cao. 2018. Multistage Signaling Game-Based Optimal Detection Strategies for Suppressing Malware Diffusion in Fog-Cloud-Based IoT Networks. IEEE Internet of Things Journal 5, 2 (2018), 1043–1054.
[46]
Jiawei Su, Danilo Vasconcellos Vargas, Sanjiva Prasad, Daniele Sgandurra, Yaokai Feng, and Kouichi Sakurai. 2018. Lightweight Classification of IoT Malware Based on Image Recognition. In IEEE Annual Computer Software and Applications Conference, COMPSAC. IEEE Computer Society, 664–669.
[47]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations, ICLR.
[48]
Danish Vasan, Mamoun Alazab, Sobia Wassan, Babak Safaei, and Qin Zheng. 2020. Image-based malware classification using ensemble of CNN architectures (IMCEC). Computers & Security(2020), 101748.
[49]
Bolun Wang, Yuanshun Yao, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. 2018. With Great Training Comes Great Vulnerability: Practical Attacks against Transfer Learning. In Proceedings of the USENIX Security Symposium, USENIX Security. 1281–1297.
[50]
Shanshan Wang, Zhenxiang Chen, Qiben Yan, Ke Ji, Lizhi Peng, Bo Yang, and Mauro Conti. 2020. Deep and broad URL feature mining for android malware detection. Inf. Sci. 513(2020), 600–613.
[51]
Carsten Willems, Thorsten Holz, and Felix Freiling. 2007. Toward automated dynamic malware analysis using cwsandbox. IEEE Security & Privacy 5, 2 (2007), 32–39.
[52]
Teng Xu, James Wendt, and Miodrag Potkonjak. 2014. Security of IoT systems: Design challenges and opportunities. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD). IEEE, 417–423.
[53]
Zhiwu Xu, Kerong Ren, Shengchao Qin, and Florin Craciun. 2018. CDGDroid: Android malware detection based on deep learning using CFG and DFG. In International Conference on Formal Engineering Methods. 177–193.
[54]
Sravani Yajamanam, Vikash Raja Samuel Selvin, Fabio Di Troia, and Mark Stamp. 2018. Deep Learning versus Gist Descriptors for Image-based Malware Classification. In Icissp. 553–561.
[55]
Anli Yan, Zhenxiang Chen, Haibo Zhang, Lizhi Peng, Qiben Yan, Muhammad Umair Hassan, Chuan Zhao, and Bo Yang. 2021. Effective detection of mobile malware behavior based on explainable deep neural network. Neurocomputing 453(2021), 482–492.
Index Terms
- Systematically Evaluating the Robustness of ML-based IoT Malware Detection Systems
Recommendations
Enhancing malware detection: clients deserve more protection
Sophisticated malware is designed to spread over the network and infect as many connected client machines as possible before being detected. Network security engineers have always been challenged to detect and track down such malware before infecting ...
Evading API Call Sequence Based Malware Classifiers
Information and Communications SecurityAbstractIn this paper, we present a mimicry attack to transform malware binary, which can evade detection by API call sequence based malware classifiers. While original malware was detectable by malware classifiers, transformed malware, when run, with ...
Malware detection using adaptive data compression
AISec '08: Proceedings of the 1st ACM workshop on Workshop on AISecA popular approach in current commercial anti-malware software detects malicious programs by searching in the code of programs for scan strings that are byte sequences indicative of malicious code. The scan strings, also known as the signatures of ...
Comments
Information & Contributors
Information
Published In
October 2022
536 pages
ISBN:9781450397049
DOI:10.1145/3545948
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 October 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
RAID 2022
RAID 2022: 25th International Symposium on Research in Attacks, Intrusions and Defenses
October 26 - 28, 2022
Limassol, Cyprus
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 315Total Downloads
- Downloads (Last 12 months)98
- Downloads (Last 6 weeks)4
Reflects downloads up to 30 Aug 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format