research-article

Verification of Cyber Emulation Experiments Through Virtual Machine and Host Metrics

Authors:

Laura P. Swiler,

Trevor Rollins,

Bert J. DebusschereAuthors Info & Claims

CSET '22: Proceedings of the 15th Workshop on Cyber Security Experimentation and Test

Pages 71 - 80

https://doi.org/10.1145/3546096.3546115

Published: 08 August 2022 Publication History

Abstract

Virtual machine emulation environments provide ideal testbeds for cybersecurity evaluations because they run real software binaries in a scalable, offline test setting that is suitable for assessing the impacts of software security flaws on the system. Verification of such emulations determines whether the environment is working as intended. Verification can focus on various aspects such as timing realism, traffic realism, and resource realism. In this paper, we study resource realism and issues associated with virtual machine resource utilization. We examine telemetry metrics gathered from a series of structured experiments which involve large numbers of parallel emulations meant to oversubscribe resources at some point. We present an approach to use telemetry metrics for emulation verification, and we demonstrate this approach on two cyber scenarios. Descriptions of the experimental configurations are provided along with a detailed discussion of statistical tests used to compare telemetry metrics. Results demonstrate the potential for a structured experimental framework, combined with statistical analysis of telemetry metrics, to support emulation verification. We conclude with comments on generalizability and potential future work.

References

[1]

2012. IEEE Standard for System and Software Verification and Validation. IEEE Std 1012-2012 (Revision of IEEE Std 1012-2004) (2012), 1–223. https://doi.org/10.1109/IEEESTD.2012.6204026

[2]

2018. Alert (TA18-201A) Emotet Malware. Techncial Report. US CERT. https://us-cert.cisa.gov/ncas/alerts/TA18-201A

[3]

2021. Azure Monitor. (2021). https://docs.microsoft.com/en-us/azure/azure-monitor/autoscale/autoscale-common-metrics.

[4]

2021. Cloud Telemetry: Advancing Your IT Strategy. (2021). https://www.intel.com/content/www/us/en/cloud-computing/telemetry.html.

[5]

2021. Network Telemetry. (2021). https://cloud.google.com/network-telemetry.

[6]

2021. What is Telemetry? The Guide to Application Monitoring. (2021). https://www.sumologic.com/insight/what-is-telemetry/.

[7]

Terry Benzel, Bob Braden, Ted Faber, Jelena Mirkovic, Steve Schwab, Karen Sollins, and John Wroclawski. 2009. Current developments in DETER cybersecurity testbed technology. In 2009 Cybersecurity Applications & Technology Conference for Homeland Security. IEEE, 57–70.

Digital Library

[8]

Scott Brown, Brian Henz, Harold Brown, Michael Edwards, Michael Russell, and Jonathan Mercurio. 2015. Validation of network simulation model with emulation using example malware. In 2015 IEEE Military Communications Conference (MILCOM). IEEE.

Digital Library

[9]

Prasad Calyam, Sudharsan Rajagopalan, Sripriya Seetharam, Arunprasath Selvadhurai, Khaled Salah, and Rajiv Ramnath. 2014. VDC-Analyst: Design and verification of virtual desktop cloud resource allocations. Computer Networks 68(2014), 110–122. https://doi.org/10.1016/j.comnet.2014.02.022 Communications and Networking in the Cloud.

[10]

Cisco. 2019. Snort intrusion detection and prevention system. https://www.snort.org/.

[11]

Jonathan Crussell, Thomas M Kroeger, Aaron Brown, and Cynthia Phillips. 2019. Virtually the Same: Comparing Physical and Virtual Testbeds. In 2019 International Conference on Computing, Networking and Communications (ICNC). IEEE.

[12]

Jon Davis and Shane Magrath. 2013. A survey of cyber ranges and testbeds. Technical Report. Cyber and Electronic Warfare Division, Defence Science and Technology Organization, Australian Government.

[13]

Bernard Ferguson, Anne Tall, and Denise Olsen. 2014. National cyber range overview. In 2014 IEEE Military Communications Conference. IEEE, 123–128.

Digital Library

[14]

Seth Hanson, Jerry Cruzy, and Casey Glatter. 2021. SCORCH User Guide. Technical Report SAND2021-11504 O. Sandia National Laboratories.

[15]

Brandon David Heller. 2013. Reproducible Network Research with High-Fidelity Emulation. (2013).

[16]

Mike Hibler, Robert Ricci, Leigh Stoller, Jonathon Duerig, Shashi Guruprasad, Tim Stack, Kirk Webb, and Jay Lepreau. 2008. Large-scale Virtualization in the Emulab Network Testbed. In 2008 USENIX Annual Technical Conference (USENIX ATC 08). USENIX Association.

[17]

Alefiya Hussain, David DeAngelis, Erik Kline, and Stephen Schwab. 2020. Replicated Testbed Experiments for the Evaluation of a Wide-range of DDoS Defenses. In 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 46–55.

[18]

Quan Jia, Zhaohui Wang, and Angelos Stavrou. 2009. The Heisenberg Measuring Uncertainty in Lightweight Virtualization Testbeds. In CSET.

[19]

Junyu Lai, Jiaqi Tian, Ke Zhang, Zheng Yang, and Dingde Jiang. 2021. Network Emulation as a Service (NEaaS): Towards a Cloud-Based Network Emulation Platform. Mobile Networks and Applications 26 (2021), 766–780. Issue 2. https://doi.org/10.1007/s11036-019-01426-0

Digital Library

[20]

Aleksander Maricq, Dmitry Duplyakin, Ivo Jimenez, Carlos Maltzahn, Ryan Stutsman, and Robert Ricci. 2018. Taming Performance Variability. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18). USENIX Association.

[21]

Stephen Midway, Matthew Robertson, Shane Flinn, and Michael Kaller. 2020. Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test. PeerJ 8(2020). https://doi.org/10.7717/peerj.10387

[22]

minimega developers. 2019. minimega: a distributed VM management tool. http://minimega.org/

[23]

Jelena Mirkovic, Genevieve Bartlett, and Jim Blythe. 2018. DEW: Distributed Experiment Workflows. In 11th USENIX Workshop on Cyber Security Experimentation and Test CSET 18).

[24]

Jelena Mirkovic, Terry V Benzel, Ted Faber, Robert Braden, John T Wroclawski, and Stephen Schwab. 2010. The DETER project: Advancing the science of cyber security experimentation and test. In 2010 IEEE International Conference on Technologies for Homeland Security (HST). IEEE, 1–7.

[25]

William L Oberkampf and Christopher J Roy. 2010. Verification and validation in scientific computing. Cambridge University Press.

[26]

Lee M Rossey, Robert K Cunningham, David J Fried, Jesse C Rabek, Richard P Lippmann, Joshua W Haines, and Marc A Zissman. 2002. Lariat: Lincoln adaptable real-time information assurance testbed. In Proceedings, IEEE Aerospace Conference, Vol. 6. IEEE, 6–6.

[27]

Stephen Schwab and Erik Kline. 2019. Cybersecurity Experimentation at Program Scale: Guidelines and Principles for Future Testbeds. In 2019 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 94–102.

[28]

Christos Siaterlis, Andres Perez Garcia, and Béla Genge. 2012. On the use of Emulab testbeds for scientifically rigorous experiments. IEEE Communications Surveys & Tutorials 15, 2 (2012), 929–942.

[29]

Thomas Tarman, Trevor Rollins, Laura Swiler, Jerry Cruz, Eric Vugrin, Hao Huang, Abhijeet Sahu, Patrick Wlazlo, Ana Goulart, and Kate Davis. 2021. Comparing reproduced cyber experimentation studies across different emulation testbeds. Proceedings, 14th Workshop on Cyber Security Experimentation and Test (CSET21), ACM (2021).

Digital Library

[30]

Eric Vugrin, Jerry Cruz, Christian Reedy, Thomas Tarman, and Ali Pinar. 2020. Cyber threat modeling and validation: port scanning and detection. In Proceedings of the 7th Symposium on Hot Topics in the Science of Security. Association for Computing Machinery.

Digital Library

[31]

Eric Vugrin, Seth Hanson, Jerry Cruz, Casey Glatter, Thomas Tarman, and Ali Pinar. 2021. Detection of command and control traffic: model development and experimental validation. in preparation (2021).

[32]

Xi Zheng and Christine Julien. 2015. Verification and Validation in Cyber Physical Systems: Research Challenges and a Way Forward. In 2015 IEEE/ACM 1st International Workshop on Software Engineering for Smart Cyber-Physical Systems. IEEE. https://doi.org/10.1109/SEsCPS.2015.11

Digital Library

Cited By

Yulianto SSoewito BGaol FKurniawan A(2023)Metrics and Red Teaming in Cyber Resilience and Effectiveness: A Systematic Literature Review2023 29th International Conference on Telecommunications (ICT)10.1109/ICT60153.2023.10374053(1-7)Online publication date: 8-Nov-2023
https://doi.org/10.1109/ICT60153.2023.10374053

Index Terms

Verification of Cyber Emulation Experiments Through Virtual Machine and Host Metrics
1. Security and privacy
  1. Formal methods and theory of security

Recommendations

Comparing reproduced cyber experimentation studies across different emulation testbeds
CSET '21: Proceedings of the 14th Cyber Security Experimentation and Test Workshop

Cyber testbeds provide an important mechanism for experimentally evaluating cyber security performance. However, as an experimental discipline, reproducible cyber experimentation is essential to assure valid, unbiased results. Even minor differences in ...
Virtual Time Machine for Reproducible Network Emulation
SIGSIM-PADS '19: Proceedings of the 2019 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation

Reproducing network emulation experiments on diverse physical platforms with varying computation and communication resources is non-trivial. Many state-of-the-art network emulation testbeds do not guarantee timing fidelity. Consequently, results ...
Performance Metrics of Virtual Machine Live Migration
CLOUD '15: Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing

Live virtual machine migration allows resources from one physical server to be moved to another with little or no interruption in the processes of the guest operating system. The process involved in performing a live migration includes copying the guest ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

CSET '22: Proceedings of the 15th Workshop on Cyber Security Experimentation and Test

August 2022

150 pages

ISBN:9781450396844

DOI:10.1145/3546096

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CSET 2022

CSET 2022: Cyber Security Experimentation and Test Workshop

August 8, 2022

CA, Virtual, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
105
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)6

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yulianto SSoewito BGaol FKurniawan A(2023)Metrics and Red Teaming in Cyber Resilience and Effectiveness: A Systematic Literature Review2023 29th International Conference on Telecommunications (ICT)10.1109/ICT60153.2023.10374053(1-7)Online publication date: 8-Nov-2023
https://doi.org/10.1109/ICT60153.2023.10374053

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten