Re-measuring the Label Dynamics of Online Anti-Malware Engines from Millions of Samples
Pages 253 - 267
Abstract
VirusTotal is the most widely used online scanning service in both academia and industry. However, it is known that the results returned by antivirus engines are often inconsistent and changing over time. The intrinsic dynamics of VirusTotal labeling have prompted researchers to investigate the characteristics of label dynamics for more effective use. However, they are generally limited in terms of the size and diversity of the datasets used in the measurements. This poses threats to many of their conclusions. In this paper, we perform an extraordinary large-scale study to re-measure the label dynamics of VirusTotal. Our dataset involves all the scan data in VirusTotal over a 14-month period, including over 571 million samples and 847 million reports in total. With this large dataset, we are able to revisit many issues related to the label dynamics of VirusTotal, including the prevalence of label dynamics/silence, the characteristics across file types, the impact of label dynamics on common label aggregation methods, the stabilization patterns of labels, etc. Our measurement reveals some observations that are unknown to the research community and even inconsistent with previous research. We believe that our findings could help researchers advance the understanding of the VirusTotal ecosystem.
References
[1]
2022. Statistics - VirusTotal. https://www.virustotal.com/en/statistics/.
[2]
2022. VirusTotal. https://www.virustotal.com/.
[3]
Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. 2014. Drebin: Effective and explainable detection of android malware in your pocket. In Ndss, Vol. 14. 23--26.
[4]
Ulrich Bayer, Paolo Milani Comparetti, Clemens Hlauschek, Christopher Kruegel, and Engin Kirda. 2009. Scalable, behavior-based malware clustering. In NDSS, Vol. 9. 8--11.
[5]
Onur Catakoglu, Marco Balduzzi, and Davide Balzarotti. 2016. Automatic extraction of indicators of compromise for web applications. In Proceedings of the 25th international conference on world wide web. 333--343.
[6]
Kai Chen, Peng Wang, Yeonjoon Lee, XiaoFeng Wang, Nan Zhang, Heqing Huang, Wei Zou, and Peng Liu. 2015. Finding unknown malice in 10 seconds: Mass vetting for new threats at the google-play scale. In 24th {USENIX} security symposium ({USENIX} security 15). 659--674.
[7]
Euijin Choo, Mohamed Nabeel, Ravindu De Silva, Ting Yu, and Issa Khalil. 2022. A Large Scale Study and Classification of VirusTotal Reports on Phishing and Malware URLs. arXiv preprint arXiv:2205.13155 (2022).
[8]
Fady Copty, Matan Danos, Orit Edelstein, Cindy Eisner, Dov Murik, and Benjamin Zeltser. 2018. Accurate malware detection by extreme abstraction. In Proceedings of the 34th Annual Computer Security Applications Conference. 101--111.
[9]
Wayne W Daniel et al. 1978. Applied nonparametric statistics. Houghton Mifflin.
[10]
Yue Duan, Mu Zhang, Abhishek Vasisht Bhaskar, Heng Yin, Xiaorui Pan, Tongxin Li, Xueqiang Wang, and XiaoFeng Wang. 2018. Things You May Not Know About Android (Un) Packers: A Systematic Study based on Whole-System Emulation. In NDSS.
[11]
Yu Feng, Saswat Anand, Isil Dillig, and Alex Aiken. 2014. Apposcopy: Semantics-based detection of android malware through static analysis. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering. 576--587.
[12]
Mariano Graziano, Davide Canali, Leyla Bilge, Andrea Lanzi, and Davide Balzarotti. 2015. Needles in a haystack: Mining information from public dynamic analysis sandboxes for malware intelligence. In 24th {USENIX} Security Symposium ({USENIX} Security 15). 1057--1072.
[13]
Mahmoud Hammad, Joshua Garcia, and Sam Malek. 2018. A large-scale empirical study on the effects of code obfuscations on Android apps and anti-malware products. In Proceedings of the 40th international conference on software engineering. 421--431.
[14]
Heqing Huang, Cong Zheng, Junyuan Zeng, Wu Zhou, Sencun Zhu, Peng Liu, Suresh Chari, and Ce Zhang. 2016. Android malware development on public malware scanning platforms: A large-scale data-driven study. In 2016 IEEE International Conference on Big Data (Big Data). IEEE, 1090--1099.
[15]
Alex Kantchelian, Michael Carl Tschantz, Sadia Afroz, Brad Miller, Vaishaal Shankar, Rekha Bachwani, Anthony D Joseph, and J Doug Tygar. 2015. Better malware ground truth: Techniques for weighting anti-virus vendor labels. In Proceedings of the 8th ACM Workshop on Artificial Intelligence and Security. 45--56.
[16]
Doowon Kim, Bum Jun Kwon, and Tudor Dumitracs. 2017. Certified malware: Measuring breaches of trust in the windows code-signing pki. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 1435--1448.
[17]
Bo Li, Phani Vadrevu, Kyu Hyung Lee, Roberto Perdisci, Jienan Liu, Babak Rahbarinia, Kang Li, and Manos Antonakakis. 2018. JSgraph: Enabling Reconstruction of Web Attacks via Efficient Tracking of Live In-Browser JavaScript Executions. In NDSS.
[18]
Vector Guo Li, Matthew Dunn, Paul Pearce, Damon McCoy, Geoffrey M Voelker, Stefan Savage, and Kirill Levchenko. 2019. Reading the tea leaves: A comparative analysis of threat intelligence. In USENIX Security Symposium.
[19]
Hui Peng, Yan Zhang, Weiping Shi, and Xiaofeng Chen. 2019b. Understanding the label process of virus total: A phishing URL perspective. IEEE Transactions on Information Forensics and Security, Vol. 15 (2019), 2221--2234.
[20]
Peng Peng, Limin Yang, Linhai Song, and Gang Wang. 2019a. Opening the blackbox of virustotal: Analyzing online phishing scan engines. In Proceedings of the Internet Measurement Conference. 478--485.
[21]
Moheeb Abu Rajab, Lucas Ballard, Noé Lutz, Panayiotis Mavrommatis, and Niels Provos. 2013. CAMP: Content-agnostic malware protection. (2013).
[22]
Christian Rossow, Christian J Dietrich, Chris Grier, Christian Kreibich, Vern Paxson, Norbert Pohlmann, Herbert Bos, and Maarten Van Steen. 2012. Prudent practices for designing malware experiments: Status quo and outlook. In 2012 IEEE Symposium on Security and Privacy. IEEE, 65--79.
[23]
Marcos Sebastián, Richard Rivera, Platon Kotzias, and Juan Caballero. 2016. Avclass: A tool for massive malware labeling. In International symposium on research in attacks, intrusions, and defenses. Springer, 230--253.
[24]
Mahmood Sharif, Jumpei Urakawa, Nicolas Christin, Ayumu Kubota, and Akira Yamada. 2018. Predicting impending exposure to malicious content from user behavior. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security. 1487--1501.
[25]
Linhai Song, Heqing Huang, Wu Zhou, Wenfei Wu, and Yiying Zhang. 2016. Learning from big malwares. In Proceedings of the 7th ACM SIGOPS Asia-Pacific Workshop on Systems. 1--8.
[26]
Nedim vS rndic and Pavel Laskov. 2013. Detection of malicious pdf files based on hierarchical document structure. In Proceedings of the 20th Annual Network & Distributed System Security Symposium. Citeseer, 1--16.
[27]
Bo Sun, Akinori Fujino, and Tatsuya Mori. 2016. Poster: Toward automating the generation of malware analysis reports using the sandbox logs. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. 1814--1816.
[28]
Saravanan Thirumuruganathan, Mohamed Nabeel, Euijin Choo, Issa Khalil, and Ting Yu. 2022. SIRAJ: A Unified Framework for Aggregation of Malicious Entity Detectors. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 507--521.
[29]
Ke Tian, Steve TK Jan, Hang Hu, Danfeng Yao, and Gang Wang. 2018. Needle in a haystack: Tracking down elite phishing domains in the wild. In Proceedings of the Internet Measurement Conference 2018. 429--442.
[30]
Haoyu Wang, Zhe Liu, Jingyue Liang, Narseo Vallina-Rodriguez, Yao Guo, Li Li, Juan Tapiador, Jingcun Cao, and Guoai Xu. 2018. Beyond google play: A large-scale comparative study of chinese android app markets. In Proceedings of the Internet Measurement Conference 2018. 293--307.
[31]
Liang Wang, Antonio Nappa, Juan Caballero, Thomas Ristenpart, and Aditya Akella. 2014. Whowas: A platform for measuring web deployments on iaas clouds. In Proceedings of the 2014 Conference on Internet Measurement Conference. 101--114.
[32]
Liu Wang, Haoyu Wang, Xiapu Luo, and Yulei Sui. 2022. MalWhiteout: Reducing Label Errors in Android Malware Detection. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1--13.
[33]
Mingyuan Xia, Lu Gong, Yuanhao Lyu, Zhengwei Qi, and Xue Liu. 2015. Effective real-time android application auditing. In 2015 IEEE Symposium on Security and Privacy. IEEE, 899--914.
[34]
Zhaoyan Xu, Antonio Nappa, Robert Baykov, Guangliang Yang, Juan Caballero, and Guofei Gu. 2014. Autoprobe: Towards automatic active malicious server probing using dynamic binary analysis. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security. 179--190.
[35]
Wei Yang, Deguang Kong, Tao Xie, and Carl A Gunter. 2017. Malware detection in adversarial settings: Exploiting feature evolutions and confusions in android apps. In Proceedings of the 33rd Annual Computer Security Applications Conference. 288--302.
[36]
Shuofei Zhu, Jianjun Shi, Limin Yang, Boqin Qin, Ziyi Zhang, Linhai Song, and Gang Wang. 2020. Measuring and Modeling the Label Dynamics of Online {Anti-Malware} Engines. In 29th USENIX Security Symposium (USENIX Security 20). 2361--2378.
[37]
Chaoshun Zuo and Zhiqiang Lin. 2017. Smartgen: Exposing server urls of mobile apps with selective symbolic execution. In Proceedings of the 26th International Conference on World Wide Web. 867--876.
Index Terms
- Re-measuring the Label Dynamics of Online Anti-Malware Engines from Millions of Samples
Recommendations
Measuring and modeling the label dynamics of online anti-malware engines
SEC'20: Proceedings of the 29th USENIX Conference on Security SymposiumVirusTotal provides malware labels from a large set of anti-malware engines, and is heavily used by researchers for malware annotation and system evaluation. Since different engines often disagree with each other, researchers have used various methods to ...
Malware Detection Method Focusing on Anti-debugging Functions
CANDAR '14: Proceedings of the 2014 Second International Symposium on Computing and NetworkingMalware has received much attention in recent years. Antivirus software is widely used as a countermeasure against malware. However, some kinds of malware can evade detection by antivirus software, hence, a new detection method is required. In this ...
Towards Accurate Labeling of Android Apps for Reliable Malware Detection
CODASPY '21: Proceedings of the Eleventh ACM Conference on Data and Application Security and PrivacyIn training their newly-developed malware detection methods, researchers rely on threshold-based labeling strategies that interpret the scan reports provided by online platforms, such as VirusTotal. The dynamicity of this platform renders those labeling ...
Comments
Information & Contributors
Information
Published In
October 2023
746 pages
ISBN:9798400703829
DOI:10.1145/3618257
- General Chairs:
- Marie-José Montpetit,
- Aris Leivadeas,
- Program Chairs:
- Steve Uhlig,
- Mobin Javed
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
IMC '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 277 of 1,083 submissions, 26%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 258Total Downloads
- Downloads (Last 12 months)258
- Downloads (Last 6 weeks)16
Reflects downloads up to
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in