research-article

Testing system virtual machines

Authors:

Lorenzo Martignoni,

Roberto Paleari,

Giampaolo Fresi Roglia,

Danilo BruschiAuthors Info & Claims

ISSTA '10: Proceedings of the 19th international symposium on Software testing and analysis

Pages 171 - 182

https://doi.org/10.1145/1831708.1831730

Published: 12 July 2010 Publication History

Abstract

Virtual machines offer the ability to partition the resources of a physical system and to create isolated execution environments. The development of virtual machines is a very challenging task. This is particularly true for system virtual machines, since they run an operating system and must replicate in every detail the incredibly complex environment it requires. Nowadays, system virtual machines are the key component of many critical architectures. However, only little effort has been invested to test if the environment they provide is semantically equivalent to the environment found on real machines. In this paper we present a methodology specific for testing system virtual machines. This methodology is based on protocol-specific fuzzing and differential analysis, and consists in forcing a virtual machine and the corresponding physical machine to execute specially crafted snippets of user- and system-mode code and in comparing their behaviors. We have developed a prototype, codenamed KEmuFuzzer, that implements our methodology for the Intel x86 architecture and used it to test four state-of-the-art virtual machines: BOCHS, QEMU, VirtualBox and VMware. We discovered defects in all of them.

References

[1]

Amazon elastic compute cloud (Amazon EC2). http://aws.amazon.com/ec2/.

[2]

Anubis. http://anubis.iseclab.org/.

[3]

F. Bellard. QEMU, a fast and portable dynamic translator. In Proceedings of the annual conference on USENIX Annual Technical Conference (ATEC), Berkeley, CA, USA, 2005.

Digital Library

[4]

D. Brumley, J. Caballero, Z. Liang, J. Newsome, and D. Song. Towards automatic discovery of deviations in binary implementations with applications to error detection and fingerprint generation. In Proceedings of the 16th USENIX Security Symposium, 2007.

Digital Library

[5]

C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE: Automatically Generating Inputs of Death. In Proceedings of the 13th ACM conference on Computer and communications security, 2006.

Digital Library

[6]

B. Daniel, D. Dig, K. Garcia, and D. Marinov. Automated testing of refactoring engines. In Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, Sept. 2007.

Digital Library

[7]

P. Ferrie. Attacks on Virtual Machine Emulators. Technical report, Symantec, 2006.

[8]

GNU Binutils. http://gnu.org/software/binutils/.

[9]

P. Godefroid, M. Y. Levin, and D. Molnar. Automated Whitebox Fuzz Testing. In Proceedings of the Network and Distributed System Security Symposium, 2008.

[10]

Google Inc. Android emulator. http://code.google.com/android/reference/emulator.html.

[11]

A. Groce, G. Holzmann, and R. Joshi. Randomized differential testing as a prelude to formal verification. In Proceedings of the 29th International Conference on Software Engineering (ICSE), pages 621--631, 2007.

Digital Library

[12]

Intel. Intel 64 and IA-32 Architectures Software Developer's Manual, Sept. 2009. System Programming Guide, Part 1.

[13]

R. Kaksonen. A Functional Method for Assessing Protocol Implementation Security. Technical report, VTT Electronics, 2001.

[14]

Kernel-based Virtual Machine (KVM). http://linux-kvm.org/.

[15]

R. Lämmel and W. Schulte. Controllable combinatorial coverage in grammar-based testing. In 18th IFIP International Conference on Testing Communicating Systems (TestCom 2006), 2006.

Digital Library

[16]

K. P. Lawton. Bochs: A Portable PC Emulator for Unix/X. Linux Journal, Sept. 1996.

Digital Library

[17]

R. Majumdar and K. Sen. Hybrid Concolic Testing. In Proceedings of the 29th International Conference on Software Engineering (ICSE'07), 2007.

Digital Library

[18]

L. Martignoni, R. Paleari, G. Fresi Roglia, and D. Bruschi. Testing CPU emulators. In Proceedings of the 2009 International Conference on Software Testing and Analysis (ISSTA). ACM, July 2009.

Digital Library

[19]

W. M. McKeeman. Differential Testing for Software. Digital Technical Journal, 10(1), 1998.

[20]

Windows XP Mode Homepage. http://www.microsoft.com/windows/virtual-pc/.

[21]

B. P. Miller, L. Fredrikson, and B. So. An Empirical Study of the Reliability of UNIX Utilities. Communications of the ACM, 33(12), December 1990.

Digital Library

[22]

G. Neiger, A. Santoni, F. Leung, D. Rodgers, and R. Uhlig. Intel Virtualization Technology: Hardware support for efficient processor virtualization. Intel Technology Journal, 10(3):167--177, Aug. 2006.

[23]

Oracle. VirtualBox. http://virtualbox.org.

[24]

T. Ormandy. An Empirical Study into the Security Exposure to Host of Hostile Virtualized Environments. In Proceedings of CanSecWest Applied Security Conference, 2007.

[25]

R. Paleari, L. Martignoni, G. Fresi Roglia, and D. Bruschi. A fistful of red-pills: How to automatically generate procedures to detect CPU emulators. In Proceedings of the 3rd USENIX Workshop on Offensive Technologies (WOOT). ACM, Aug. 2009.

Digital Library

[26]

G. J. Popek and R. P. Goldberg. Formal requirements for virtualizable third generation architectures. Communications of the ACM, 17(7):412--421, 1974.

Digital Library

[27]

T. Raffetseder, C. Kruegel, and E. Kirda. Detecting System Emulators. In Proceedings of Information Security Conference (ISC 2007), 2007.

Digital Library

[28]

J. S. Robin and C. E. Irvine. Analysis of the Intel Pentium's Ability to Support a Secure Virtual Machine Monitor. In Proceedings of the 9th conference on USENIX Security Symposium, 2000.

Digital Library

[29]

J. Rutkowska. Red Pill... or how to detect VMM using (almost) one CPU instruction. http://invisiblethings.org/papers/redpill.html.

[30]

K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for c. In Proceedings of the 10th European Software Engineering Conference, 2005.

Digital Library

[31]

E. G. Sirer and B. N. Bershad. Testing java virtual machines. In Proceedings of the International Conference on Software Testing And Review, nov 1999.

[32]

J. E. Smith and R. Nair. Virtual Machines: Versatile Platforms for Systems and Processes. Morgan Kaufmann, June 2005.

Digital Library

[33]

M. Sutton, A. Greene, and P. Amini. Fuzzing: Brute Force Vulnerability Discovery. Addison-Wesley Professional, 2007.

Digital Library

[34]

VMware, Inc. http://vmware.com/.

[35]

VMware security advisor. http://www.vmware.com/security/advisories/VMSA-2009-0015.html.

Cited By

Lombardi FRecchia A(2024)On Abstract Machines Security and PerformanceProcedia Computer Science10.1016/j.procs.2023.12.182231:C(111-118)Online publication date: 12-Apr-2024
https://dl.acm.org/doi/10.1016/j.procs.2023.12.182
Chesser MNepal SRanasinghe DJust RFraser G(2023)Icicle: A Re-designed Emulator for Grey-Box Firmware FuzzingProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598039(76-88)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598039
Wang GZhu ZCheng XMeng D(2023)A High-Coverage and Efficient Instruction-Level Testing Approach for x86 ProcessorsIEEE Transactions on Computers10.1109/TC.2023.328876272:11(3203-3217)Online publication date: Nov-2023
https://doi.org/10.1109/TC.2023.3288762
Show More Cited By

Index Terms

Testing system virtual machines
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Virtual Battery: A testing tool for power-aware software

Virtualization is an inexpensive and convenient method for setting up software test environments. Thus it is being widely used as a test tool for software products requiring high reliability such as mission critical cyber-physical systems. However, ...
Enabling Instantaneous Relocation of Virtual Machines with a Lightweight VMM Extension
CCGRID '10: Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing

We are developing an efficient resource management system with aggressive virtual machine (VM) relocation among physical nodes in a data center. Existing live migration technology, however, requires a long time to change the execution host of a VM, it ...
A high performance inter-domain communication approach for virtual machines

Highlights Presented a novel inter-domain communication approach (IVCOM) for virtual machine. Using bypassing protocol stacks, shunning page flipping methods. IVCOM applies a direct communication channel between domain 0 and U. IVOCM can improve the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA '10: Proceedings of the 19th international symposium on Software testing and analysis

July 2010

294 pages

ISBN:9781605588230

DOI:10.1145/1831708

General Chair:
Paolo Tonella
Fondazione Bruno Kessler -- IRST, Italy
,
Program Chair:
Alessandro Orso
Georgia Institute of Technology, USA

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '10

Sponsor:

SIGSOFT

ISSTA '10: International Symposium on Software Testing and Analysis

July 12 - 16, 2010

Trento, Italy

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '24

Sponsor:
sigsoft

33rd ACM SIGSOFT International Symposium on Software Testing and Analysis

September 16 - 20, 2024

Vienna , Austria

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

48
Total Citations
View Citations
870
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)2

Reflects downloads up to 02 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lombardi FRecchia A(2024)On Abstract Machines Security and PerformanceProcedia Computer Science10.1016/j.procs.2023.12.182231:C(111-118)Online publication date: 12-Apr-2024
https://dl.acm.org/doi/10.1016/j.procs.2023.12.182
Chesser MNepal SRanasinghe DJust RFraser G(2023)Icicle: A Re-designed Emulator for Grey-Box Firmware FuzzingProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598039(76-88)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598039
Wang GZhu ZCheng XMeng D(2023)A High-Coverage and Efficient Instruction-Level Testing Approach for x86 ProcessorsIEEE Transactions on Computers10.1109/TC.2023.328876272:11(3203-3217)Online publication date: Nov-2023
https://doi.org/10.1109/TC.2023.3288762
Nappa AÚbeda-Portugués APapadopoulos PVarvello MTapiador JLanzi A(2022)Scramblesuit: An effective timing side-channels framework for malware sandbox evasion1Journal of Computer Security10.3233/JCS-22000530:6(851-876)Online publication date: 23-Nov-2022
https://doi.org/10.3233/JCS-220005
Zhong H(2022)Enriching Compiler Testing with Real Program from Bug ReportProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556894(1-12)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3556894
Jiang MXu TZhou YHu YZhong MWu LLuo XRen KFalsafi BFerdman MLu SWenisch T(2022)EXAMINER: automatically locating inconsistent instructions between real devices and CPU emulators for ARMProceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3503222.3507736(846-858)Online publication date: 28-Feb-2022
https://dl.acm.org/doi/10.1145/3503222.3507736
Weaver V(2022)Improving HPC Security with Targeted Syscall Fuzzing2022 IEEE/ACM First International Workshop on Cyber Security in High Performance Computing (S-HPC)10.1109/S-HPC56715.2022.00006(1-8)Online publication date: Nov-2022
https://doi.org/10.1109/S-HPC56715.2022.00006
Nunes MBurnap PReinecke PLloyd K(2022)Bane or BoonJournal of Information Security and Applications10.1016/j.jisa.2022.10320267:COnline publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1016/j.jisa.2022.103202
Nappa APapadopoulos PVarvello MGomez DTapiador JLanzi A(2021)PoW-How: An Enduring Timing Side-Channel to Evade Online Malware SandboxesComputer Security – ESORICS 202110.1007/978-3-030-88418-5_5(86-109)Online publication date: 30-Sep-2021
https://doi.org/10.1007/978-3-030-88418-5_5
Moghimi DLipp MSunar BSchwarz MCapkun SRoesner F(2020)MedusaProceedings of the 29th USENIX Conference on Security Symposium10.5555/3489212.3489293(1427-1444)Online publication date: 12-Aug-2020
https://dl.acm.org/doi/10.5555/3489212.3489293
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents