research-article

Open access

Assessing a Decision Support Tool for SOC Analysts

Authors:

Ioannis Agrafiotis,

Martin Helmhout,

Thomas Bashford-Rogers,

Michael Goldsmith,

Sadie CreeseAuthors Info & Claims

Digital Threats: Research and Practice , Volume 2, Issue 3

Article No.: 22, Pages 1 - 35

https://doi.org/10.1145/3430753

Published: 08 June 2021 Publication History

All formats PDF

Abstract

It is difficult to discern real-world consequences of attacks on an enterprise when investigating network-centric data alone. In recent years, many tools have been developed to help understand attacks using visualisation, but few aim to predict real-world consequences. We have developed a visualisation tool that aims to improve decision support during attacks in Security Operation Centres (SOCs). Our tool visualises propagation of risks from sensor alert data to Business Process (BP) tasks. This is an important capability gap present in many SOCs today, as most threat detection tools are technology-centric. In this article, we present a user study that assesses our tool’s usability and ability to support the analyst. Ten analysts from seven SOCs performed carefully designed tasks related to understanding risks and recovery decision-making. The study was conducted in laboratory conditions with simulated attacks and used a mixed-method approach to collect data from questionnaires, eye tracking, and semi-structured interviews. Our findings suggest that relating business tasks to network asset in visualisations can help analysts prioritise response strategies. Finally, our article also provides an in-depth discussion on user studies conducted with SOC analysts more generally, including lessons learned, recommendations and a critique of our own study.

References

[1]

R. Jordan Crouser, Erina Fukudy, and Subashini Sridhar. 2017. Retrospective on a decade of research in visualization for cybersecurity. In Proceedings of the IEEE International Symposium on Technologies for Homeland Security (HST’17). IEEE, 1–5.

[2]

Sushil Jajodia, Peng Liu, Vipin Swarup, and Cliff Wang. 2009. Cyber Situational Awareness. Springer.

[3]

Sadie Creese, Michael Goldsmith, Nick Moffat, Jassim Happa, and Ioannis Agrafiotis. 2013. Cybervis: Visualizing the potential impact of cyber attacks on the wider enterprise. In Proceedings of the IEEE International Conference on Technologies for Homeland Security (HST’13). IEEE, 73–79.

[4]

BPMN. Business Process Modelling and Notation. 1989. Retrieved April 2021 from http://www.omg.org.

[5]

Martin Roesch. 1999. Snort-lightweight intrusion detection for networks. In Proceedings of the 13th USENIX Conference on System Administration. 229–238.

Digital Library

[6]

NAGIOS. NAGIOS Network Monitoring Software Application. 1996. Retrieved from http://www.nagios.org/.

[7]

ClamAV. Clam Anti-Virus. 2004. Retrieved from http://www.clamav.net.

[8]

B. Shneiderman and C. Plaisant. 2004. Designing the User Interface: Strategies for Effective Human-computer Interaction (4th ed.). Addison-Wesley, Boston, MA.

Digital Library

[9]

Jacques Bertin. 1983. Semiology of Graphics: Diagrams, Networks, Maps. University of Wisconsin Press.

Digital Library

[10]

L. Itti, C. Koch, and E. Niebur. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20, 11 (1998), 1254–1259.

Digital Library

[11]

Greg Conti. 2007. Security Data Visualization: Graphical Techniques for Network Analysis. No Starch Press.

[12]

Raffael Marty. 2009. Applied Security Visualization. Addison-Wesley.

Digital Library

[13]

SECVIZ. Security Visualization. 2006. Retrieved from http://www.secviz.org.

[14]

Daniel Best, Shawn Bohn, Douglas Love, Adam Wynne, and William Pike. 2010. Real-time visualization of network behaviors for situational awareness. In Proceedings of the IEEE Symposium on Visualization for Cyber Security (VizSec’10). ACM.

[15]

Greg Conti. Rumint. 2006. Retrieved from http://www.rumint.org.

[16]

Hyogon Kim, Inhye Kang, and Saewoong Bahk. 2004. Real-time visualizaton of network attacks on high-speed links. IEEE Netw. 18, 5 (2004), 30–39.

Digital Library

[17]

Stephen Lau. 2004. The spinning cube of potential doom. Commun. ACM 47, 6 (2004), 25–26.

Digital Library

[18]

Anatoly Yelizarov and Dennis Gamayunov. 2009. Visualization of complex attacks and state of attacked network. In Proceedings of the IEEE Symposium on Visualization for Cyber Security (VizSec’09). ACM.

[19]

Qi Liao, Aaron Striegel, and Nitesh Chawla. 2010. Visualizing graph dynamics and similarity for enterprise network security and management. In Proceedings of the IEEE Symposium on Visualization for Cyber Security (VizSec’10). ACM.

[20]

Matthew Chu, Kyle Ingols, Richard Lippmann, Seth Webster, and Stephen Boyer. 2010. Visualizing attack graphs, reachability, and trust relationships with NAVIGATOR. In Proceedings of the IEEE Symposium on Visualization for Cyber Security (VizSec’10). ACM.

[21]

Jamie Rasmussen, Kate Ehrlich, Steven Ross, Susanna Kirk, Daniel Gruen, and John Patterson. 2010. Nimble cybersecurity incident management through visualization and defensible recommendations. In Proceedings of the IEEE Symposium on Visualization for Cyber Security (VizSec’10). ACM.

[22]

Raffael Marty. 2009. Applied Security Visualization. Addison-Wesley.

Digital Library

[23]

Greg Conti. 2007. Security Data Visualization: Graphical Techniques for Network Analysis. No Starch Press.

[24]

R. Marty. 2007. SecViz.org. Retrieved from: http://secviz.org/.

[25]

Kyle Gancarz and Kenneth Prole. 2012. Visual techniques for analyzing wireless communication patterns. In Proceedings of the IEEE Conference on Technologies for Homeland Security (HST’12). IEEE, 341–347.

[26]

The Wireshark team. Wireshark. 1998. Retrieved from http://www.wireshark.org.

[27]

Kulsoom Abdullah, Christopher P. Lee, Gregory J. Conti, John A. Copeland, and John T. Stasko. 2005. IDS RainStorm: Visualizing IDS alarms. In Proceedings of the IEEE Symposium on Visualization for Cyber Security (VizSec’05).

[28]

Arcsight. Arcsight Enterprise Security Manager. 2000. Retrieved from http://www.arcsight.com.

[29]

Tenable. Tenable 3D Tool. 2011. Retrieved from http://www.tenable.com.

[30]

SecureDecisions. SecureScope. 2012. Retrieved from https://securedecisions.com.

[31]

Enrico Bertini. Background Literature on Evaluation for Information Visualization. Retrieved from: http://enrico.bertini.io/publications.

[32]

Catherine Plaisant. 2004. The challenge of information visualization evaluation. In Proceedings of the Working Conference on Advanced Visual Interfaces. ACM, 109–116.

Digital Library

[33]

Christopher Ahlberg and Ben Shneiderman. 1994. The alphaslider: A compact and rapid selector. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 365–371.

[34]

Pourang Irani and Colin Ware. 2003. Diagramming information structures using 3D perceptual primitives. ACM Trans. Comput.-hum. Interact. 10, 1 (2003), 1–19.

Digital Library

[35]

Alistair G. Sutcliffe, Mark Ennis, and J. Hu. 2000. Evaluating the effectiveness of visual user interfaces for information retrieval. Int. J. Hum.-comput. Stud. 53, 5 (2000), 741–763.

Digital Library

[36]

Donald Byrd. 1999. A scrollbar-based visualization for document navigation. In Proceedings of the 4th ACM Conference on Digital Libraries. ACM, 122–129.

[37]

Catherine Plaisant, Jesse Grosjean, and Benjamin B. Bederson. 2002. Spacetree: Supporting exploration in large node link tree, design evolution and empirical evaluation. In Proceedings of the IEEE Symposium on Information Visualization (INFOVIS’02). IEEE, 57–64.

[38]

J. Gregory Trafton, Susan S. Kirschenbaum, Ted L. Tsui, Robert T. Miyamoto, James A. Ballas, and Paula D. Raymond. 2000. Turning pictures into numbers: Extracting and generating information from complex visualizations. Int. J. Hum.-comput. Stud. 53, 5 (2000), 827–850.

[39]

Diane Staheli, Tamara Yu, R. Jordan Crouser, Suresh Damodaran, Kevin Nam, David O’Gwynn, Sean McKenna, and Lane Harrison. 2014. Visualization evaluation for cyber security: Trends and future directions. In Proceedings of the 11th Workshop on Visualization for Cyber Security. ACM, 49–56.

[40]

R. Burke Johnson and Anthony J. Onwuegbuzie. 2004. Mixed methods research: A research paradigm whose time has come. Educ. Res. 33, 7 (2004), 14–26.

[41]

James Gosling. 1995. Java Programming Language.

[42]

Oracle. MySQL. 1995. Retrieved from http://www.mysql.com/.

[43]

JogAmp Community. Java OpenGL (JOGL). 2005. Retrieved from http://jogamp.org/.

[44]

Seeing Machines. FaceLAB. 2011. Retrieved from https://www.seeingmachines.com/.

[45]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualit. Res. Psychol. 3, 2 (2006), 77–101.

[46]

Jan M. Ahrend, Marina Jirotka, and Kevin Jones. 2016. On the collaborative practices of cyber threat intelligence analysts to develop and utilize tacit threat and defence knowledge. In Proceedings of the International Conference on Cyber Situational Awareness, Data Analytics and Assessment (CyberSA’16). IEEE, 1–10.

Cited By

Reeves AAshenden D(2023)Understanding decision making in security operations centres: building the case for cyber deception technologyFrontiers in Psychology10.3389/fpsyg.2023.116570514Online publication date: 23-May-2023
https://doi.org/10.3389/fpsyg.2023.1165705
Husák MJavorník M(2023)Lightweight Impact Assessment and Projection of Lateral Movement and Malware Infection2023 IEEE Conference on Communications and Network Security (CNS)10.1109/CNS59707.2023.10288665(1-6)Online publication date: 2-Oct-2023
https://doi.org/10.1109/CNS59707.2023.10288665
Husák MSadlek LŠpaček SLaštovička MJavorník MKomárková J(2022)CRUSOEComputers and Security10.1016/j.cose.2022.102609115:COnline publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1016/j.cose.2022.102609

Index Terms

Assessing a Decision Support Tool for SOC Analysts
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI
  2. Visualization
    1. Visualization application domains
      1. Visual analytics
2. Security and privacy
  1. Intrusion/anomaly detection and malware mitigation
    1. Intrusion detection systems

Recommendations

Exploring the Use of an Information Visualization Tool for Decision Support under Uncertainty and Risk
ICEMIS '15: Proceedings of the The International Conference on Engineering & MIS 2015

In this paper, we present an evaluation study as part of the research effort toward the development of RiDeViz. RiDeViz is an information visualization tool designed to support decision-making under uncertainty and risk. We studied how people use the ...
Developing a tool to support collaborative decision making via visualizations
ECCE '08: Proceedings of the 15th European conference on Cognitive ergonomics: the ergonomics of cool interaction

Motivation -- To provide visualizations that will support high-level decision makers in leveraging findings from analyses conducted using modelling and simulation techniques.

Research approach -- Cognitive systems engineering methods were used to ...
Geographic Visualization of Risk as Decision Support in Emergency Situations
HSI '12: Proceedings of the 2012 5th International Conference on Human System Interactions

We investigate how geographic visualization can be used as a means to facilitate understanding of risk in emergency situations. Specifically, we identify the need of decision makers regarding access to risk information; introduce a visualization ...

Comments

Information & Contributors

Information

Published In

cover image Digital Threats: Research and Practice

Digital Threats: Research and Practice Volume 2, Issue 3

September 2021

143 pages

EISSN:2576-5337

DOI:10.1145/3470118

Editors:
Arun Lakhotia
University of Louisiana at Lafayette and Cythereal, USA
,
Leigh Metcalf
CERT, USA

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 June 2021

Online AM: 15 April 2021

Accepted: 01 October 2020

Revised: 01 August 2020

Received: 01 September 2019

Published in DTRAP Volume 2, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Defence Science and Technology Laboratory

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
778
Total Downloads

Downloads (Last 12 months)257
Downloads (Last 6 weeks)31

Reflects downloads up to 03 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Reeves AAshenden D(2023)Understanding decision making in security operations centres: building the case for cyber deception technologyFrontiers in Psychology10.3389/fpsyg.2023.116570514Online publication date: 23-May-2023
https://doi.org/10.3389/fpsyg.2023.1165705
Husák MJavorník M(2023)Lightweight Impact Assessment and Projection of Lateral Movement and Malware Infection2023 IEEE Conference on Communications and Network Security (CNS)10.1109/CNS59707.2023.10288665(1-6)Online publication date: 2-Oct-2023
https://doi.org/10.1109/CNS59707.2023.10288665
Husák MSadlek LŠpaček SLaštovička MJavorník MKomárková J(2022)CRUSOEComputers and Security10.1016/j.cose.2022.102609115:COnline publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1016/j.cose.2022.102609

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents