research-article

Open access

SAIBERSOC: Synthetic Attack Injection to Benchmark and Evaluate the Performance of Security Operation Centers

Authors:

Michele Campobasso,

Ganduulga Gankhuyag,

Luca AllodiAuthors Info & Claims

ACSAC '20: Proceedings of the 36th Annual Computer Security Applications Conference

Pages 141 - 153

https://doi.org/10.1145/3427228.3427233

Published: 08 December 2020 Publication History

All formats PDF

Abstract

In this paper we introduce SAIBERSOC, a tool and methodology enabling security researchers and operators to evaluate the performance of deployed and operational Security Operation Centers (SOCs) (or any other security monitoring infrastructure). The methodology relies on the MITRE ATT&CK Framework to define a procedure to generate and automatically inject synthetic attacks in an operational SOC to evaluate any output metric of interest (e.g., detection accuracy, time-to-investigation, etc.). To evaluate the effectiveness of the proposed methodology, we devise an experiment with n = 124 students playing the role of SOC analysts. The experiment relies on a real SOC infrastructure and assigns students to either a BADSOC or a GOODSOC experimental condition. Our results show that the proposed methodology is effective in identifying variations in SOC performance caused by (minimal) changes in SOC configuration. We release the SAIBERSOC tool implementation as free and open source software.

Supplementary Material

p141-rosso-supplement (p141-rosso-supplement.pdf)

Artifact repository

Download
146.47 KB

References

[1]

J. Abadie, B. P. Abbott, R. Abbott, T. Accadia, F. Acernese, R. Adhikari, P. Ajith, B. Allen, G. Allen, E. Amador Ceron, R. S. Amin, S. B. Anderson, W. G. Anderson, F. Antonucci, M. A. Arain, 2010. All-sky search for gravitational-wave bursts in the first joint LIGO-GEO-Virgo run. Phys. Rev. D 81 (May 2010), 102001. Issue 10. https://doi.org/10.1103/PhysRevD.81.102001

[2]

Anne Adams and Ann Blandford. 2005. Bridging the gap between organizational and user perspectives of security in the clinical domain. International Journal of Human-Computer Studies 63, 1-2 (2005), 175–202. https://doi.org/10.1016/j.ijhcs.2005.04.022

Digital Library

[3]

Luca Allodi, Marco Cremonini, Fabio Massacci, and Woohyun Shim. 2020. Measuring the accuracy of software vulnerability assessments: experiments with students and professionals. Empirical Software Engineering 25, 2 (2020), 1063–1094. https://doi.org/10.1007/s10664-019-09797-4

[4]

Manos Antonakakis, Tim April, Michael Bailey, Matt Bernhard, Elie Bursztein, Jaime Cochran, Zakir Durumeric, J. Alex Halderman, Luca Invernizzi, Michalis Kallitsis, Deepak Kumar, Chaz Lever, Zane Ma, Joshua Mason, Damian Menscher, Chad Seaman, Nick Sullivan, Kurt Thomas, and Yi Zhou. 2017. Understanding the Mirai Botnet. In 26th USENIX Security Symposium (USENIX Security 17). USENIX Association, Vancouver, BC, USA, 1093–1110. Retrieved 2020-06-13 from https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/antonakakis

Digital Library

[5]

Philippe Biondi. [n.d.]. scapy. Retrieved 2020-06-13 from https://scapy.net/

[6]

Sonia Chiasson and P. C. van Oorschot. 2015. Quantifying the security advantage of password expiration policies. Designs, Codes and Cryptography 77, 2 (2015), 401–408. https://doi.org/10.1007/s10623-015-0071-9

Digital Library

[7]

Common Vulnerabilities and Exposures (CVE) Database 2018. CVE-2018-6789. Available from MITRE, CVE-ID CVE-2018-6789. Retrieved 2020-06-13 from https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-6789

[8]

The MITRE Corporation. [n.d.]. MITRE PRE-ATT&CK Introduction. Retrieved 2020-06-13 from https://attack.mitre.org/resources/pre-introduction/

[9]

Elasticsearch B.V.[n.d.]. The Elastic Stack. Retrieved 2020-06-13 from https://www.elastic.co/elastic-stack (Elasticseach and Kibana).

[10]

Flask-RESTX [n.d.]. Flask-RESTX. Retrieved 2020-06-13 from https://github.com/python-restx/flask-restx

[11]

Rajesh Ganesan, Sushil Jajodia, and Hasan Cam. 2017. Optimal scheduling of cybersecurity analysts for minimizing risk. ACM Transactions on Intelligent Systems and Technology (TIST) 8, 4(2017), 1–32. https://doi.org/10.1145/2914795

Digital Library

[12]

Cormac Herley and Paul C. van Oorschot. 2017. SoK: Science, Security and the Elusive Goal of Security as a Scientific Pursuit. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017. IEEE Computer Society, 99–120. https://doi.org/10.1109/SP.2017.38

[13]

ISO. 2013. Information technology — Security techniques — Information security management systems — Requirements. ISO/IEC 27001:2013. International Organization for Standardization, Geneva, Switzerland.

[14]

Pierre Jacobs, Alapan Arnab, and Barry Irwin. 2013. Classification of Security Operation Centers. In 2013 Information Security for South Africa, Johannesburg, South Africa, August 14-16, 2013 (2013-08). IEEE, 1–7. https://doi.org/10.1109/ISSA.2013.6641054

[15]

Pierre Conrad Jacobs. 2014. Towards a Framework for Building Security Operation Centers. Master Thesis. Rhodes University. Retrieved 2020-06-13 from https://research.ict.ru.ac.za/SNRG/Theses/Jacobs%202014%20Msc.pdf

[16]

Faris Bugra Kokulu, Ananta Soneji, Tiffany Bao, Yan Shoshitaishvili, Ziming Zhao, Adam Doupé, and Gail-Joon Ahn. 2019. Matched and Mismatched SOCs: A Qualitative Study on Security Operations Center Issues. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS 2019, London, UK, November 11-15, 2019. ACM, 1955–1970. https://doi.org/10.1145/3319535.3354239

Digital Library

[17]

Lockheed Martin Corporation. [n.d.]. The Cyber Kill Chain. Retrieved 2020-06-13 from https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html

[18]

Gordon “Fyodor” Lyon. [n.d.]. Nmap Security Scanner. Retrieved 2020-06-13 from https://nmap.org/

[19]

Doug Miller, Ron Alford, Andy Applebaum, Henry Foster, Caleb Little, and Blake E. Strom. 2018. Automated Adversary Emulation: A Case for Planning and Acting with Unknowns. Retrieved 2020-06-13 from https://www.mitre.org/publications/technical-papers/automated-adversary-emulation-a-case-for-planning-and-acting-with

[20]

Joseph Muniz, Gary McIntyre, and Nadhem AlFardan. 2015. Security Operations Center: Building, Operating, and Maintaining Your SOC. Cisco Press, Hoboken, NJ, USA.

[21]

OffSec Services Limited. [n.d.]. Offensive Security’s Exploit Database. Retrieved 2020-06-13 from https://www.exploit-db.com/

[22]

Open Security Foundation (OISF). [n.d.]. Suricata. Retrieved 2020-06-13 from https://suricata-ids.org/

[23]

Vern Paxson. 1999. Bro: a System for Detecting Network Intruders in Real-Time. Computer Networks 31, 23-24 (1999), 2435–2463. https://doi.org/10.1016/S1389-1286(99)00112-7 Zeek – An Open Source Network Security Monitoring Tool, (formerly known as bro), retrieved 2020-06-13 from https://zeek.org.

Digital Library

[24]

Shari Pfleeger and Robert Cunningham. 2010. Why measuring security is hard. IEEE Security & Privacy 8, 4 (2010), 46–54. https://doi.org/10.1109/MSP.2010.60

Digital Library

[25]

Rapid7, Inc.[n.d.]. Metasploit – The world’s most used penetration testing framework. Retrieved 2020-06-13 from https://www.metasploit.com/

[26]

Red Canary. [n.d.]. Atomic Red Team. Retrieved 2020-06-13 from https://atomicredteam.io/

[27]

Armin Ronacher. [n.d.]. Flask. Retrieved 2020-06-13 from https://palletsprojects.com/p/flask/

[28]

Security Onion Solutions, LLC. [n.d.]. Security Onion. Retrieved 2020-06-13 from https://securityonion.net/

[29]

Ankit Shah, Rajesh Ganesan, Sushil Jajodia, and Hasan Cam. 2018. A methodology to measure and monitor level of operational effectiveness of a CSOC. International Journal of Information Security 17, 2 (2018), 121–134. https://doi.org/10.1007/s10207-017-0365-1

Digital Library

[30]

Ankit Shah, Rajesh Ganesan, Sushil Jajodia, and Hasan Cam. 2018. Understanding Tradeoffs Between Throughput, Quality, and Cost of Alert Analysis in a CSOC. IEEE Transactions on Information Forensics and Security 14, 5(2018), 1155–1170. https://doi.org/10.1109/TIFS.2018.2871744

Digital Library

[31]

Sathya Chandran Sundaramurthy, Jacob Case, Tony Truong, Loai Zomlot, and Marcel Hoffmann. 2014. A Tale of Three Security Operation Centers. In Proceedings of the 2014 ACM Workshop on Security Information Workers, SIW ’14, Scottsdale, Arizona, USA, November 7, 2014. ACM, 43–50. https://doi.org/10.1145/2663887.2663904

Digital Library

[32]

Sathya Chandran Sundaramurthy, John McHugh, Xinming Ou, Michael Wesch, Alexandru G. Bardas, and S. Raj Rajagopalan. 2016. Turning Contradictions into Innovations or: How We Learned to Stop Whining and Improve Security Operations. In Twelfth Symposium on Usable Privacy and Security, SOUPS 2016, Denver, CO, USA, June 22-24, 2016. USENIX Association, 237–251. Retrieved 2020-06-13 from https://www.usenix.org/conference/soups2016/technical-sessions/presentation/sundaramurthy

[33]

tcpreplay [n.d.]. tcpreplay. Retrieved 2020-06-13 from https://tcpreplay.appneta.com/

[34]

The MITRE Corporation. [n.d.]. MITRE ATT&CK. Retrieved 2020-06-13 from https://attack.mitre.org/

[35]

The Tor Project, Inc.[n.d.]. Tor. Retrieved 2020-06-13 from https://www.torproject.org/

[36]

Verizon Enterprise Solutions. 2018. 2018 Data Breach Investigation Report. Technical Report 11th edition. Verizon. Retrieved 2020-06-13 from https://enterprise.verizon.com/resources/reports/DBIR_2018_Report.pdf

[37]

Evan You. [n.d.]. vue.js. Retrieved 2020-06-13 from https://vuejs.org/

[38]

Carson Zimmerman. 2014. Ten strategies of a world-class cybersecurity operations center. The MITRE Corporation.

Cited By

Kersten LMulders TZambon ESnijders CAllodi LKelley PKapadia A(2023)'Give me structure 'Proceedings of the Nineteenth USENIX Conference on Usable Privacy and Security10.5555/3632186.3632192(97-111)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.5555/3632186.3632192
Shon HLee YYoon M(2023)Semi-Supervised Alert Filtering for Network SecurityElectronics10.3390/electronics1223475512:23(4755)Online publication date: 23-Nov-2023
https://doi.org/10.3390/electronics12234755
Almashor MAhmed EPick BXue JAbuadbba SGaire RWang SCamtepe SNepal S(2023)Unraveling Threat Intelligence Through the Lens of Malicious URL CampaignsProceedings of the 18th Asian Internet Engineering Conference10.1145/3630590.3630600(78-86)Online publication date: 12-Dec-2023
https://dl.acm.org/doi/10.1145/3630590.3630600
Show More Cited By

Index Terms

SAIBERSOC: Synthetic Attack Injection to Benchmark and Evaluate the Performance of Security Operation Centers
1. Security and privacy
  1. Intrusion/anomaly detection and malware mitigation
2. Social and professional topics
  1. Computing / technology policy
    1. Computer crime

Index terms have been assigned to the content through auto-classification.

Recommendations

SAIBERSOC: A Methodology and Tool for Experimenting with Security Operation Centers
In this article, we introduce SAIBERSOC (Synthetic Attack Injection to Benchmark and Evaluate the Performance of Security Operation Centers), a tool and methodology enabling security researchers and operators to evaluate the performance of deployed and ...
Technical performance metrics of a security operations center
Abstract
This research introduces a novel framework for creating metrics intended for security operations centers (SOCs). The framework is developed using the design science research methodology and has been validated by generating four novel metrics to ...
Methodology to verify, debug and evaluate performances of NoC based interconnects
NoCArc '15: Proceedings of the 8th International Workshop on Network on Chip Architectures

Latest developments in electronic systems lead hardware architects to completely rethink the traffic exchanges inside Systems on Chip. Constraints on more parallelism, less power, more coherency, less order while designing system fabrics and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACSAC '20: Proceedings of the 36th Annual Computer Security Applications Conference

December 2020

962 pages

ISBN:9781450388580

DOI:10.1145/3427228

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conference

ACSAC '20

ACSAC '20: Annual Computer Security Applications Conference

December 7 - 11, 2020

Austin, USA

Acceptance Rates

Overall Acceptance Rate 104 of 497 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
794
Total Downloads

Downloads (Last 12 months)276
Downloads (Last 6 weeks)98

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kersten LMulders TZambon ESnijders CAllodi LKelley PKapadia A(2023)'Give me structure 'Proceedings of the Nineteenth USENIX Conference on Usable Privacy and Security10.5555/3632186.3632192(97-111)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.5555/3632186.3632192
Shon HLee YYoon M(2023)Semi-Supervised Alert Filtering for Network SecurityElectronics10.3390/electronics1223475512:23(4755)Online publication date: 23-Nov-2023
https://doi.org/10.3390/electronics12234755
Almashor MAhmed EPick BXue JAbuadbba SGaire RWang SCamtepe SNepal S(2023)Unraveling Threat Intelligence Through the Lens of Malicious URL CampaignsProceedings of the 18th Asian Internet Engineering Conference10.1145/3630590.3630600(78-86)Online publication date: 12-Dec-2023
https://dl.acm.org/doi/10.1145/3630590.3630600
Nafees MSaxena NCardenas AGrijalva SBurnap P(2023)Smart Grid Cyber-Physical Situational Awareness of Complex Operational Technology Attacks: A ReviewACM Computing Surveys10.1145/356557055:10(1-36)Online publication date: 2-Feb-2023
https://dl.acm.org/doi/10.1145/3565570
Luthfi MLubis MSaedudin R(2023)Development of Security Operation Center (SOC) Governance Blueprint Based on Consideration of Process Maturity Level Parameters2023 8th International Conference on Information Technology and Digital Applications (ICITDA)10.1109/ICITDA60835.2023.10427358(1-8)Online publication date: 17-Nov-2023
https://doi.org/10.1109/ICITDA60835.2023.10427358
de Céspedes RDimitoglou G(2022)Development of a virtualized security operations centerJournal of Computing Sciences in Colleges10.5555/3512489.351250137:3(108-119)Online publication date: 19-Jan-2022
https://dl.acm.org/doi/10.5555/3512489.3512501
Pružinec JNguyen QBaldwin AGriffin JLiu YKiss ÁMarín BSaadatmand M(2022)KUBO: a framework for automated efficacy testing of anti-virus behavioral detection with procedure-based malware emulationProceedings of the 13th International Workshop on Automating Test Case Design, Selection and Evaluation10.1145/3548659.3561307(37-44)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3548659.3561307
Collins MHussain ASchwab S(2022)Towards an Operations-Aware Experimentation Methodology2022 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW)10.1109/EuroSPW55150.2022.00046(384-393)Online publication date: Jun-2022
https://doi.org/10.1109/EuroSPW55150.2022.00046
Harris DMiknis MSmith CWilson I(2020)Metrics for Evaluating Cyber Security Data Visualizations in Virtual RealityPRESENCE: Virtual and Augmented Reality10.1162/pres_a_0036329(223-240)Online publication date: 1-Dec-2020
https://doi.org/10.1162/pres_a_00363

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents