research-article

Probing the Limits of Virtualized Software Protection

Authors:

Joshua Cazalas,

J. Todd McDonald,

Natalia StakhanovaAuthors Info & Claims

PPREW-4: Proceedings of the 4th Program Protection and Reverse Engineering Workshop

Article No.: 5, Pages 1 - 11

https://doi.org/10.1145/2689702.2689707

Published: 09 December 2014 Publication History

Abstract

Virtualization is becoming a prominent field of research not only in distributed systems, but also in software protection and obfuscation. Software virtualization has given rise to advanced techniques that may provide intellectual property protection and anti-cloning resilience. We present results of an empirical study that answers whether integrity of execution can be preserved for process-level virtualization protection schemes in the face of adversarial analysis. Our particular approach considers exploits that target the virtual execution environment itself and how it interacts with the underlying host operating system and hardware. We give initial results that indicate such protection mechanisms may be vulnerable at the level where the virtualized code interacts with the underlying operating system. The resolution of whether such attacks can undermine security will help create better detection and analysis methods for malware that also employ software virtualization. Our findings help frame research for additional mitigation techniques using hardware-based integration or hybrid virtualization techniques that can better defend legitimate uses of virtualized software protection.

References

[1]

B. Anckaert, M. Jakubowski, and R. Venkatesan (2006). Proteus: virtualization for diversified tamper-resistance. In DRM '06: Proc. of the ACM Workshop on Digital Rights Management, pp. 47--58, ACM Press.

Digital Library

[2]

F. Cohen (1993). Operating system protection through program evolution. Jour. Computers and Security 12(6), Oct. 1993, pp. 565--584, Elsevier Advanced Technology Publications.

Digital Library

[3]

C. Collberg and J. Nagra (2010). Surreptitious software: obfuscation, watermarking, and tamperproofing for software protection. Upper Saddle River, NJ: Addison-Wesley.

Digital Library

[4]

K. Coogan, G. Lu, and S. Debray (2011). Deobfuscation of virtualization-obfuscated software: A Semantics-Based Approach. CCS '11: Proc. of the 18th ACM conference on Computer and communications security, pp. 275--284, ACM Press.

Digital Library

[5]

A. Dinaburg, P. Royal, M. Sharif and W. Lee (2008). Ether: Malware Analysis via Hardware Virtualization Extensions In CCS '08: Proc. of the 15th ACM conference on Computer and Communications Security, pp. 51--62, ACM Press.

Digital Library

[6]

S. Ghosh, J. Hiser, and J.W. Davidson (2012). Replacement attacks against VM-protected applications. In VEE '12: Proc. of the 8th ACM SIGPLAN/SIGOPS conference on Virtual Execution Environments, pp. 203--214, ACM Press.

Digital Library

[7]

S. Ghosh, J. Hiser, and J.W. Davidson (2013). Software protection for dynamically generated code. In PPREW '13: Proc. of the 2nd ACM SIGPLAN Program Protection and Reverse Engineering Workshop, ACM Press.

Digital Library

[8]

W. Hu, J. Hiser, D. Williams, A. Filipi, J.W. Davidson, D. Evans, J.C. Knight, et al. (2006). Secure and practical defense against code-injection attacks using software dynamic translation. In VEE '06: Proc. of the 2nd international conference on Virtual execution environments, pp. 2--12, ACM Press.

Digital Library

[9]

R. Paleariy, L. Martignoniz, G.F. Rogliay, and D. Bruschiy (2009). A fistful of red-pills: How to automatically generate procedures to detect CPU emulators. In WOOT'09: Proc. of the 3rd USENIX conference on Offensive technologies, pp. 2--2. USENIX Association, URL http://dl.acm.org/citation.cfm?id=1855876.1855878

Digital Library

[10]

R. Rolles (2009). Unpacking virtualization obfuscators Proc. of the 3rd USENIX conference on Offensive technologies, p.1--1, August 10, 2009, Montreal, Canada.

Digital Library

[11]

J. Kamunyori (2007). Handling self-modifying code using software dynamic translation. In TAPIA '07: Proc. of the 2007 Conference on Diversity in Computing, pp. 32--32, ACM Press. doi.acm.org/10.1145/1347787.1347807

Digital Library

[12]

J. Rutkowska (2007). Virtualization detection vs. blue pill detection. The Invisible Things Labs blog. URL http://theinvisiblethings.blogspot.com/2007/08/virtualization-detection-vs-blue-pill.html.

[13]

J. Rutkowska (2006). Blue Pill Detection. URL http://theinvisiblethings.blogspot.com/2006/08/blue-pill-detection.html

[14]

J. Rutkowska (2008). Virtualization: Red Pill or Blue Pill. URL http://teckinfo.blogspot.com/2008/04/virtualisation-red-pill-or-blue.html

[15]

J. Rutkowska (2008). Red Pill. URL http://www.hackerzvoice.net/ouah/Red_Pill.html

[16]

K. Scott, N. Kumar, S. Velusamy, B. Childers, J.W. Davidson and M.L. Soffa (2003). Retargetable and reconfigurable software dynamic translation. In CGO'03: Proc. of the International Symposium on Code Generation and Optimization, pp. 36--47.

Digital Library

[17]

M. Sharif, A. Lanzi, J. Gifn and W. Lee (2009). Automatic reverse engineering of malware emulators. In SP '09: Proc. of the 30th IEEE Symposium on Security and Privacy, pp. 94--109, IEEE Computer Society.

Digital Library

[18]

Z. Qi, B. Li, Q. Lin, M. Yu, M. Xia, and H. Guan (2012). SPAD: Software Protection Through Anti-Debugging using hardware-assisted virtualization. Jour. Information Science and Engineering, vol. 28, pp. 813--827.

[19]

A. Vasudevan and R. Yerraballi (2006). Cobra: Fine-grained malware analysis using stealth localized-executions. In SP '06: Proc. of 2006 IEEE Symposium on Security and Privacy, pp. 264--279, IEEE Computer Society.

Digital Library

[20]

M. Webster and G. Malcolm (2009). Detection of metamorphic and virtualization-based malware using algebraic specification. Journal in Computer Virology 5(3), August 2009, pp. 221--245, Springer-Verlag.

[21]

J. Smith and R. Nair (2005). Virtual Machines: Versatile platforms for systems and processes (The Morgan Kaufmann Series in Computer Architecture and Design), Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

Digital Library

[22]

G. Kc, A. Keromytis and V. Prevelakis (2003). Countering code injection attacks with instruction set randomization. In CCS '03: Proc. of the 10th ACM conference on Computer and communications security, pp. 272--280, ACM Press.

Digital Library

[23]

N. Sovarel, D. Evans, and N. Paul (2005). Where's the FEEB? The Effectiveness of Instruction Set Randomization. 14th USENIX Security Symposium, Baltimore, MD, 4 August 2005.

Digital Library

[24]

E.G. Barrantes, D. H. Ackley, T.S. Palmer, D. Stefanovi and D.D. Zovi (2003). Randomized instruction set emulation to disrupt binary code injection attacks. In CCS '03: Proc. of the 10th ACM conference on Computer and communications security, pp. 281--289, ACM Press.

Digital Library

[25]

E.G. Barrantes, D. H. Ackley, S. Forrest and D. Stefanovi (2005). Randomized instruction set emulation. ACM Transactions on Information and System Security (TISSEC) 8(1), pp. 3--40, February 2005, ACM Press.

Digital Library

[26]

C. Helmstetter, V. Joloboff, Z. Xinlei and G. Xiaopeng (2011). Fast Instruction Set Simulation Using LLVM-based Dynamic Translation. In IMECS'11: Proc. of the International MultiConference of Engineers and Computer Scientists, March 16-18, Hong Kong.

[27]

H. Kim and J.E. Smith (2003). Dynamic binary translation for accumulator-oriented architectures. In CGO'03: International Symposium on Code Generation and Optimization, pp. 25--35, March 23-26, 2003.

Digital Library

[28]

K. Scott, N. Kumar, B. R. Childers, J. W. Davidson, and M. L. Soffa (2004). Overhead reduction techniques for software dynamic translation. In Proc. of the 18th International Parallel and Distributed Processing Symposium, April 26-30, 2004.

[29]

K. Scott, N. Kumar, S. Velusamy, B. Childers, J. W. Davidson, M. L. Soffa (2003). Retargetable and reconfigurable software dynamic translation. In CGO '03: Proc. of the International Symposium on Code Generation and Optimization, pp. 36--47, IEEE Computer Society, Washington, DC, USA.

Digital Library

[30]

J. Pincus and B. Baker (2004). Beyond Stack Smashing: Recent Advances in Exploiting Buffer Overruns. In IEEE Security and Privacy, July/August 2004, pp. 20--27, IEEE Computer Society, Washington, DC, USA.

Digital Library

[31]

J. Seward and N. Nethercote (2005). Using Valgrind to detect undefined value errors with bit-precision. In ATEC '05: Proc. of the Annual Conference on USENIX Annual Technical Conference, USENIX Association, Berkeley, CA, USA.

Digital Library

[32]

C. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V.J. Reddi, and K. Hazelwood (2005). Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation. In PLDI '05: Proc. of the 2005 ACM SIGPLAN conference on Programming Language Design and Implementation, 40(6), pp. 190--200, Chicago, IL, USA.

Digital Library

[33]

M. Payer and T.R. Gross (2011). Fine-grained user-space security through virtualization. In VEE '11: Proc. of the 7th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, 46(7), pp. 157--168, July 2011, ACM Publishing.

Digital Library

[34]

J. Kinder (2012). Towards Static Analysis of Virtualization-Obfuscated Binaries. In WCRE '12: Proc. of 19th Working Conf. Reverse Engineering, Kingston, Ontario, Canada, October 15-18, 2012. IEEE Publishing.

Digital Library

[35]

E. Schwartz (2011). The Danger of Unrandomized Code. ;login: The Usenix Magazine, 36(6), pp. 7--12, December 2011.

[36]

K. Li and X. Li (2011). Defeating the Transparency Feature of DBI. Presented talk at Blackhat USA 2014, August 06, 2014. Online: https://www.youtube.com/watch?v=9oKZx6Cr3X8.

[37]

Escaping DynamoRIO and Pin - or why it's a worse-than-you-think idea to run untrusted code or to input untrusted data. Online: https://github.com/lgeek/dynamorio_pin_escape.

[38]

DynamoRIO: Dynamic Instrumentation Tool Platform. Online: http://dynamorio.org/docs/overview.html.

Cited By

Ahmadvand MBelow DBanescu SPretschner AFalcarin PZunke M(2019)VirtSCProceedings of the 3rd ACM Workshop on Software Protection10.1145/3338503.3357723(53-63)Online publication date: 15-Nov-2019
https://dl.acm.org/doi/10.1145/3338503.3357723
Xu DMing JFu YWu DLie DMannan MBackes MWang X(2018)VMHuntProceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security10.1145/3243734.3243827(442-458)Online publication date: 15-Oct-2018
https://dl.acm.org/doi/10.1145/3243734.3243827
Di Pietro RLombardi F(2018)Virtualization Technologies and Cloud Security: Advantages, Issues, and PerspectivesFrom Database to Cyber Security10.1007/978-3-030-04834-1_9(166-185)Online publication date: 30-Nov-2018
https://doi.org/10.1007/978-3-030-04834-1_9
Show More Cited By

Probing the Limits of Virtualized Software Protection
1. Security and privacy
  1. Systems security
2. Social and professional topics
  1. Computing / technology policy

Recommendations

Replacement attacks against VM-protected applications
VEE '12: Proceedings of the 8th ACM SIGPLAN/SIGOPS conference on Virtual Execution Environments

Process-level virtualization is increasingly being used to enhance the security of software applications from reverse engineering and unauthorized modification (called software protection). Process-level virtual machines (PVMs) can safeguard the ...
Replacement attacks against VM-protected applications
VEE '12

Process-level virtualization is increasingly being used to enhance the security of software applications from reverse engineering and unauthorized modification (called software protection). Process-level virtual machines (PVMs) can safeguard the ...
Hybrid static-dynamic attacks against software protection mechanisms
DRM '05: Proceedings of the 5th ACM workshop on Digital rights management

Advances in reverse engineering and program analyses have made software extremely vulnerable to malicious host attacks. These attacks typically take the form of intellectual property violations, against which the software needs to be protected. The ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PPREW-4: Proceedings of the 4th Program Protection and Reverse Engineering Workshop

December 2014

77 pages

ISBN:9781605586373

DOI:10.1145/2689702

Program Chairs:
Mila Dalla Preda
University of Verona, Italy
,
J. Todd McDonald
University of South Alabama, USA

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 December 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

PPREW-4

PPREW-4: 4th Program Protection and Reverse Engineering Workshop

December 9, 2014

LA, New Orleans, USA

Acceptance Rates

PPREW-4 Paper Acceptance Rate 7 of 14 submissions, 50%;

Overall Acceptance Rate 21 of 36 submissions, 58%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
228
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)2

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ahmadvand MBelow DBanescu SPretschner AFalcarin PZunke M(2019)VirtSCProceedings of the 3rd ACM Workshop on Software Protection10.1145/3338503.3357723(53-63)Online publication date: 15-Nov-2019
https://dl.acm.org/doi/10.1145/3338503.3357723
Xu DMing JFu YWu DLie DMannan MBackes MWang X(2018)VMHuntProceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security10.1145/3243734.3243827(442-458)Online publication date: 15-Oct-2018
https://dl.acm.org/doi/10.1145/3243734.3243827
Di Pietro RLombardi F(2018)Virtualization Technologies and Cloud Security: Advantages, Issues, and PerspectivesFrom Database to Cyber Security10.1007/978-3-030-04834-1_9(166-185)Online publication date: 30-Nov-2018
https://doi.org/10.1007/978-3-030-04834-1_9
Holder WMcDonald JAndel TMcDonald JPreda MStakhanova N(2017)Evaluating Optimal Phase Ordering in Obfuscation ExecutivesProceedings of the 7th Software Security, Protection, and Reverse Engineering / Software Security and Protection Workshop10.1145/3151137.3151140(1-12)Online publication date: 5-Dec-2017
https://dl.acm.org/doi/10.1145/3151137.3151140
Manikyam RMcDonald JMahoney WAndel TRuss SMcDonald JPreda MStakhanova N(2016)Comparing the effectiveness of commercial obfuscators against MATE attacksProceedings of the 6th Workshop on Software Security, Protection, and Reverse Engineering10.1145/3015135.3015143(1-11)Online publication date: 5-Dec-2016
https://dl.acm.org/doi/10.1145/3015135.3015143
Banescu SLucaci CKrämer BPretschner AWyseur BDe Sutter B(2016)VOT4CSProceedings of the 2016 ACM Workshop on Software PROtection10.1145/2995306.2995312(39-49)Online publication date: 28-Oct-2016
https://dl.acm.org/doi/10.1145/2995306.2995312

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents