research-article

Open access

We cannot trust in you: a study about the dissonance among anti-malware engines.

Authors:

Antonio Pirozzi,

Corrado Aaron VisaggioAuthors Info & Claims

ARES '22: Proceedings of the 17th International Conference on Availability, Reliability and Security

Article No.: 65, Pages 1 - 13

https://doi.org/10.1145/3538969.3544411

Published: 23 August 2022 Publication History

All formats PDF

Abstract

The impressive volume of malware circulating today, the increase in its sophistication, its unpredictability and evasiveness pose new problems to face in the deployment of effective defence systems. It is proven today that no single anti-malware solution could be universally effective in protecting from all threats due to its inherent inaccuracy. Different anti-malware solutions could produce analyses that may significantly vary, in terms of both detection rate (not all are able to recognize the same malware) and classification (not all assign the same family name to the same malware). In this study we realize a quantitative analysis of the dissonance among anti-malware solutions commonly used in the marketplace and widely deployed in real world with the aim of evaluating the classification inaccuracies. We carried out this evaluation on two datasets: one comprising 103,073 malware updated to 2020/10/03 from the MalwareBazaar repository; the other composed of 100k malware extracted from EMBER dataset. Our results show that there is a large disagreement in classification and the uncertainty about type and family identification is still high.

References

[1]

Abdullah Al-Dujaili, Alex Huang, Erik Hemberg, and Una-May O’Reilly. 2018. Adversarial deep learning for robust detection of binary encoded malware. In 2018 IEEE Security and Privacy Workshops (SPW). IEEE, 76–82.

[2]

Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. 2014. Drebin: Effective and explainable detection of android malware in your pocket. In Ndss, Vol. 14. 23–26.

[3]

Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. 2013. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases. Springer, 387–402.

Digital Library

[4]

Pengbin Feng, Jianfeng Ma, Cong Sun, Xinpeng Xu, and Yuwan Ma. 2018. A novel dynamic Android malware detection system with ensemble learning. IEEE Access 6(2018), 30996–31011.

[5]

Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick McDaniel. 2017. Adversarial examples for malware detection. In European symposium on research in computer security. Springer, 62–79.

[6]

Médéric Hurier, Kevin Allix, Tegawendé F Bissyandé, Jacques Klein, and Yves Le Traon. 2016. On the lack of consensus in anti-virus decisions: Metrics and insights on building ground truths of android malware. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 142–162.

Digital Library

[7]

Médéric Hurier, Guillermo Suarez-Tangil, Santanu Kumar Dash, Tegawendé F Bissyandé, Yves Le Traon, Jacques Klein, and Lorenzo Cavallaro. 2017. Euphony: Harmonious unification of cacophonous anti-virus vendor labels for android malware. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 425–435.

Digital Library

[8]

Roberto Jordaney, Kumar Sharad, Santanu K Dash, Zhi Wang, Davide Papini, Ilia Nouretdinov, and Lorenzo Cavallaro. 2017. Transcend: Detecting concept drift in malware classification models. In 26th {USENIX} Security Symposium ({USENIX} Security 17). 625–642.

[9]

Alex Kantchelian, Michael Carl Tschantz, Sadia Afroz, Brad Miller, Vaishaal Shankar, Rekha Bachwani, Anthony D Joseph, and J Doug Tygar. 2015. Better malware ground truth: Techniques for weighting anti-virus vendor labels. In Proceedings of the 8th ACM Workshop on Artificial Intelligence and Security. 45–56.

Digital Library

[10]

Yunus Kucuk and Guanhua Yan. 2020. Deceiving Portable Executable Malware Classifiers into Targeted Misclassification with Practical Adversarial Examples. In Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy. 341–352.

Digital Library

[11]

Aziz Mohaisen and Omar Alrawi. 2014. Av-meter: An evaluation of antivirus scans and labels. In International conference on detection of intrusions and malware, and vulnerability assessment. Springer, 112–131.

[12]

Cyberint Research. 2020. Ryuk Crypto-Ransomware. Technical Report.

[13]

Aleieldin Salem, Sebastian Banescu, and Alexander Pretschner. 2020. Maat: Automatically Analyzing VirusTotal for Accurate Labeling and Effective Malware Detection. arXiv preprint arXiv:2007.00510(2020).

[14]

Marcos Sebastián, Richard Rivera, Platon Kotzias, and Juan Caballero. 2016. Avclass: A tool for massive malware labeling. In International symposium on research in attacks, intrusions, and defenses. Springer, 230–253.

[15]

Suleiman Y Yerima and Sarmadullah Khan. 2019. Longitudinal performance analysis of machine learning based Android malware detectors. In 2019 International Conference on Cyber Security and Protection of Digital Services (Cyber Security). IEEE, 1–8.

[16]

Yanxin Zhang, Yulei Sui, Shirui Pan, Zheng Zheng, Baodi Ning, Ivor Tsang, and Wanlei Zhou. 2019. Familial clustering for weakly-labeled android malware using hybrid representation learning. IEEE Transactions on Information Forensics and Security 15 (2019), 3401–3414.

[17]

Yajin Zhou and Xuxian Jiang. 2012. Dissecting android malware: Characterization and evolution. In 2012 IEEE symposium on security and privacy. IEEE, 95–109.

Digital Library

[18]

Shuofei Zhu, Jianjun Shi, Limin Yang, Boqin Qin, Ziyi Zhang, Linhai Song, and Gang Wang. 2020. Measuring and modeling the label dynamics of online anti-malware engines. In 29th {USENIX} Security Symposium ({USENIX} Security 20). 2361–2378.

[19]

Shuofei Zhu, Ziyi Zhang, Limin Yang, Linhai Song, and Gang Wang. 2020. Benchmarking Label Dynamics of VirusTotal Engines. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security. 2081–2083.

Digital Library

Cited By

Okazaki NUsuzaki SWaki TKawagoe HPark MYamaba HAburada K(2024)Optimal Weighted Voting-Based Collaborated Malware Detection for Zero-Day Malware: A Case Study on VirusTotal and MalwareBazaarFuture Internet10.3390/fi1608025916:8(259)Online publication date: 23-Jul-2024
https://doi.org/10.3390/fi16080259
G SFaloutsos M(2024)MalFusion: Simple String Manipulations Confuse Malware Detection2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619782(113-121)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619782
Bernardinetti GCaporaso PDi Cristofaro DQuaglia FBianchi G(2023)PHOENIX: A Cloud-based Framework for Ensemble Malware Detection2023 21st Mediterranean Communication and Computer Networking Conference (MedComNet)10.1109/MedComNet58619.2023.10168868(11-14)Online publication date: 13-Jun-2023
https://doi.org/10.1109/MedComNet58619.2023.10168868
Show More Cited By

Index Terms

We cannot trust in you: a study about the dissonance among anti-malware engines.

Index terms have been assigned to the content through auto-classification.

Recommendations

Malware Detection Method Focusing on Anti-debugging Functions
CANDAR '14: Proceedings of the 2014 Second International Symposium on Computing and Networking

Malware has received much attention in recent years. Antivirus software is widely used as a countermeasure against malware. However, some kinds of malware can evade detection by antivirus software, hence, a new detection method is required. In this ...
Effectiveness of Android Obfuscation on Evading Anti-malware
CODASPY '18: Proceedings of the Eighth ACM Conference on Data and Application Security and Privacy

Obfuscation techniques have been conventionally used for legitimate applications, including preventing application reverse engineering, tampering and protecting intellectual property. A malware author could also leverage these benign techniques to hide ...
A Clustering Method for Improving Performance of Anomaly-Based Intrusion Detection System

Intrusion detection system (IDS) has played a central role as an appliance to effectively defend our crucial computer systems or networks against attackers on the Internet. The most widely deployed and commercially available methods for intrusion ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ARES '22: Proceedings of the 17th International Conference on Availability, Reliability and Security

August 2022

1371 pages

ISBN:9781450396707

DOI:10.1145/3538969

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ARES 2022

ARES 2022: The 17th International Conference on Availability, Reliability and Security

August 23 - 26, 2022

Vienna, Austria

Acceptance Rates

Overall Acceptance Rate 228 of 451 submissions, 51%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
250
Total Downloads

Downloads (Last 12 months)147
Downloads (Last 6 weeks)14

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Okazaki NUsuzaki SWaki TKawagoe HPark MYamaba HAburada K(2024)Optimal Weighted Voting-Based Collaborated Malware Detection for Zero-Day Malware: A Case Study on VirusTotal and MalwareBazaarFuture Internet10.3390/fi1608025916:8(259)Online publication date: 23-Jul-2024
https://doi.org/10.3390/fi16080259
G SFaloutsos M(2024)MalFusion: Simple String Manipulations Confuse Malware Detection2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619782(113-121)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619782
Bernardinetti GCaporaso PDi Cristofaro DQuaglia FBianchi G(2023)PHOENIX: A Cloud-based Framework for Ensemble Malware Detection2023 21st Mediterranean Communication and Computer Networking Conference (MedComNet)10.1109/MedComNet58619.2023.10168868(11-14)Online publication date: 13-Jun-2023
https://doi.org/10.1109/MedComNet58619.2023.10168868
González-Manzano Lde Fuentes JLombardi FRamos C(2023)A technical characterization of APTs by leveraging public resourcesInternational Journal of Information Security10.1007/s10207-023-00706-x22:6(1567-1584)Online publication date: 15-Jun-2023
https://dl.acm.org/doi/10.1007/s10207-023-00706-x

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents