Control Effectiveness: a Capture-the-Flag Study

The relationship between organisational risk and risk controls, supports the experimentation and analysis presented in this article. The residual risk that an organisation faces is considered to result from a set of factors, namely the assets it possesses (and their value), the risk controls it deploys, the threats it faces, and the harms that may result [2].

Focusing on harms, various types of harm can result from a cyber incident (e.g., financial, physical, reputational) [3] and a harm that occurs following a cyber incident may lead to further harms (the loss or exposure of customer data may lead to financial harms in the form of regulator fines, for example) [5]. Harm propagates, and risk controls may address various branches of a harm-propagation “tree”. Our examination of risk controls in this article is guided by this organisational risk context, and in Section 5 we reflect on how our findings can be used to refine estimations of organisational risk and Cyber Value at Risk (CVaR).

Throughout this article, we use the SANS 20 Critical Security Controls (CSCs), a risk-control guidance source widely used by organisations that covers the key classes of controls, as the basis for our examination of security-control effectiveness and deployment [11]. The full list of the SANS CSCs can be found in Table 1. Elsewhere in this work, we refer to the CSCs by their number and, for convenience, may include their description – these descriptions may be shortened and therefore serve only as a reference.

To support the development of methodologies for studying our research, we consider the characteristics of controls that might impact their effectiveness and deployment. We examined the following characteristics, which we consider relevant to our methodology, for each of the SANS 20 CSCs. Our findings are presented in Table 1.

• Human and Technical: It may be more difficult to practically assess the effectiveness of controls that require a “human” in the loop (i.e. based on a procedure that humans must carry out, such as CSC17 “Skills and training”) rather than “technical” (i.e., based on a technological measure, such as CSC12 “Boundary defences”). Designing an experiment to assess the effectiveness of “human” controls would require qualitative metrics and participation by humans, whose competence varies.
  
• Longitudinal: It may be more difficult to practically assess controls whose effectiveness relies on being successful over an extended period of time, such as monitoring of systems. This would require conducting longitudinal studies; the alternative is controls whose effectiveness can be assessed with an one-time implementation (e.g. email/web browser protection), in a fixed state.
  
• Individual: Some controls do not protect or mitigate threats but support others. Inventories, for example, do not actively prevent threats but provide information that can improve the deployment of other threat-prevention measures. Considering this property of the risk controls may indicate which controls can meaningfully be tested individually or in groups.
  

We selected the controls for testing based on the characteristics and rankings identified in Section 2. First, we excluded CSCs that required longitudinal assessment, or that acted as an intermediate step for a risk control, as these would require a longitudinal study or would be impractical to assess individually (we assumed effective mechanisms in place supporting our selected controls, i.e. accurate hardware and software inventories in our experiment). We scoped our examination to only include purely “technical” controls, due to the challenges in setting up and running an exercise involving “humans”, as we will have been validating the competence of human actors instead.

For the remaining controls, we used our rankings (as presented in Table 1) to select those which appeared most ubiquitously through the three lenses; this resulted in a shortlist of five CSCs:

• CSC3 “Vulnerability assessment”
  
• CSC4 “Admin privileges”
  
• CSC8 “Malware defences”
  
• CSC12 “Boundary defences”
  
• CSC14 “Access control”
  

All five shortlisted controls can be considered “technical”; can be assessed as one-time implementations; and act as risk controls rather than as intermediate steps to risk control. All five are therefore viable for experimentation.

We designed a virtual network setup with a number of security controls in place. Clearly, there is a huge space of possible network-security setups due to the range of variables: network topologies and assets; security controls deployed; configuration of these controls; vulnerabilities present on the network (i.e., recency of asset and control vulnerability patches). For this initial study, we selected two contrasting setups, and we discuss how this might be expanded to consider a larger part of the problem space in Section 5.

Two scenarios were created to represent two organisations with different maturity levels. The first one, shown in Figure 1, represents a mature company, where machines are segregated and updated. The second scenario, Figure 2, shows a less experienced company where all machines connect to the same network and patching is not carried out routinely.

Further to the scoping and control selection detailed above, we aimed to specifically explore subcontrols of CSC14 (Controlled Access Based on the Need to Know); in particular segmentation of the network based on sensitivity of the data, and encryption of sensitive data at rest, by segregating sensitive database servers onto a critical network and encrypting their contents. We also explore the impact of variations in CSC3 (Continuous Vulnerability Assessment and Remediation) by varying the patch recency of machines on the network. Also present were varying levels of password security, and a number of artefacts representing human error: insecure SSH keys, a list of network users, and scripts allowing database connection. These were intended to represent violations of CSC14.

Scenario 1. Figure 1 shows a network topology of a small company where machines are segregated onto three different networks: 10, 110, and 200. Network 10 contains client machines and the Active Directory. Network 110 contains critical services/data for the company. Network 200 is a DMZ. Two firewalls/gateways control access to the networks. The critical gateway restricts connections to port 3306 on the Database Server, on which the database contents are AES-encrypted, and only Admin2 has complete access to the 110 network. inet gateway provides Internet access to the 10 network, and restricts access to/from the DMZ. Only Admin2 is allowed to connect to the 200 network. Lastly, the Graylog server collects logs from networks 10 and 110.

Any machines not mentioned in this section are updated and do not contain known vulnerabilities. Any vulnerability found during the CTF in these machines was not intended to be part of the scenario, but provides important insights onto the effectiveness of the security controls, i.e., if that vulnerability can be exploited to bypass the security controls.

Balancing the investigation of research questions with the need to create a meaningful exercise with achievable goals is challenging; our approach was to introduce realistic weaknesses exploiting which would enable some flags to be captured. The following intentional weaknesses were present:

• Repository (192.168.10.205): Insecure (unauthenticated) NFS; stored credentials.
  
• Admin2 (192.168.10.215): Weak password; privilege escalation via script.
  
• User2 (DHCP, network 10): Unpatched, vulnerable to known remote exploit; weak password; stored insecure access keys.
  
• Old Database (192.168.110.9): Unpatched and containing unencrypted database contents.
  
• Graylog (192.168.10.253 and 192.168.110.2): Acts as bridge into secure network.
  

Scenario 2. Figure 2 depicts a network topology where machines are segregated onto two networks: 10 and 200. Network 10 connects all the internal machines of the corporation. Network 200 is the DMZ, and contains a public-facing Web Server. Internet access is provided to the internal machines by inet gateway, and traffic is freely allowed between networks 10 and 200.

The same weaknesses were present as in Scenario 1 on Repository, Admin2, User2 and Old Database, although the structure of the network rendered access to Old Database, on which the database contents were not encrypted, more straightforward and removed the need to use Graylog as a bridge.

Flags. The aim of flags on the network was to facilitate the collection of a reliable record of compromised network infrastructure by the participants. The placement of flags was intended to indicate which parts of the network (subnets, individual servers, etc.) attackers could access and with which privilege levels. We therefore littered the network with flags at each individual location (each machine; piece of network infrastructure), that could be captured (i.e. the flag file could be read) only by users with specific privilege levels (users and root). We also placed flags associated with specific network aims: given the presence of sensitive databases on the network, we planted flags in the contents of each database, in order to indicate whether a participant was able to read the contents in plaintext. All flags were weighted equally (in terms of the points assigned) to avoid biasing participants towards a certain course of action.

Study materials. We prepared the following materials for the participants of our study.

• Participant information sheet and written consent form. The information sheet outlined the study procedure, our data anonymisation and storage methods, and the processes to ask questions or withdraw data after the study.
  
• Flag-reporting platform. This was created using CTFd, an open-source tool for creating reporting platforms for CTF events¹, and was hosted by a server in our research lab.
  
• Post-task questionnaire. After each CTF round, participants responded to the following questions online in free-text boxes.
  
  • Please describe the security of the network and your experience penetration testing it.
    
  • How difficult was it to compromise/capture flags on this network, and why?
    
• Participant notebooks. We asked participants to use a notebook to record the “types of attack you attempt, tools you use to help you (if you use any), timestamp, network location and any rationale or strategy you apply”.
  

CTF rounds. Based on personal contacts, we managed to find four participants for our study, all of whom were security professionals who had extensive experience of working as senior penetration testers in three different organisations. Each network had two participants being randomly assigned to it. Participants were given approximately four hours to try and acquire as many flags as possible. Participants worked as individuals, capturing flags and reporting them using the flag-reporting interface, and recording their decision processes in the provided notebooks, guided by the instruction sheet.

Post-CTF focus group. Upon completion of the CTF exercise, we conducted a focus group with all four participants. The aim of the focus group was to promote group interaction, allow individuals to delve in details and to balance free-flowing debate against the need to cover the specified questions [19].

Participants were guided to discuss their experiences in the study, their perceptions of the security and how realistic the networks were, as well as their views on how the experimental setup could be improved. Other topics arose naturally, either in response to new questions asked by the researchers, or through steering of the discussion in new directions by participants. Such topics included, for example, real-world security practice and participants’ perceptions of risk-control deployment in organisations, and approaches to adapting the study design to account for the impact of humans as defenders and network users.

We began by analysing the data collected using the flag-reporting interface, by gathering the flags captured by each participant and noting the order of capture. We supplemented this data with comments made by participants in their notebooks, the post-task questionnaire, and the post-CTF focus group.

We manually transcribed the focus group data and analysed it using template analysis, a technique for qualitative data analysis in which the researcher has some understanding of the concepts to be identified [15]. The following a-priori themes were identified: how realistic the network setup is; data-collection methods; views on the potential of our methodology. We then coded the focus-group dataset initially, attaching relevant parts of the transcriptions to the a-priori themes. We assigned new codes to relevant sections of data that did not fit into these themes, thus producing an initial template of codes. This template was developed through iterative application to the dataset, and modified as appropriate to the data. Through this refinement we produced a final template and dataset coded according to it.

We recruited participants by emailing our existing contacts in organisations that employ penetration testers. For each, our recruitment email described the study, its aims, and information on how to participate. Ethical approval to conduct the study was granted by the Computer Science Department Research Ethics Committee at the University of Oxford under reference: CS_C1A_19_033.

As Figure 1 shows, two flags were captured by participants on Scenario 1: on the Repository server and on Admin2. The capture of the Repository flag “via the NFS share hosting a user directory” was described by (P1) and P4 as captured “easily”. The Admin2 flag was captured after participants accessed the Admin2 machine by brute-forcing the weak password of a user account found in the NFS.

Participants expressed the view that the patch level of Scenario 1 was reasonable: “there weren't that many really big out-of-date vulnerabilities that we found” (P4). While the User2 machine (Windows 7) was present on this network, and its patch state meant that it was vulnerable to EternalBlue, participants stated that their “enumeration didn't pick up a Windows 7” (P1). This was likely due to the host-based firewall on the Windows 7 machine blocking the necessary ports, and P1 stated that they “didn't do any non-ping scans, just to speed up the process”. In general, participants described the host-based firewall defences as being “rather relaxed... with some hosts having SMB Signing disabled which facilitates [relay attacks]” (P1). It was noted that there were “numerous ports available where they are not required” (P1), but “minimal known vulnerable services” (P4).

Focusing on the performance of participants in compromising the critical network and its databases, it is evident from the participants’ notebooks that both accessed the Database Server MySQL database via the Admin2 machine's scripts. Since the database's contents were encrypted, they were unable to capture the flags contained within the database. It was noted that the “DB is encrypted... no easily accessible information” (P4), and the researchers observed the participants attempting to decrypt the database. While the Database Server was accessible from the 10 network (“the MySQL database was accessible from a 10 range, although it was apparently firewalled, depending on the ACLs” (P1), the firewall blocking of the Old Database Server prevented participants from accessing it (“not live”) (P1).

The Graylog server, with connections to the .10 and .110 networks, allowed participants to observe the .110 network (“so we were seeing another subnet through the Graylog server” (P4)). This observation, however, did not allow them to use the Graylog as a bridge, so they were not able to communicate with machines in .110 through this path.

As shown in Figure 1, three flags were captured by participants in Scenario 2: the first on the Repository server, then on the User2 machine (Windows 7), then on the Old Database. As in Scenario 1, participants captured the Repository server flag by using “insecure file shares available on the network giving anonymous access to user directories” (P2). The patch level and host-based firewall rules of the User2 machine meant that it “had open SMB and it was vulnerable to EternalBlue” (P3), and participants leveraged this to carry out “direct remote code execution on the machine” (P2): “I executed a payload creating a new user called “newadmin” and then I mounted a share folder using that new user” (P3).

Once the User2 machine had been accessed as root, and the “root” flag captured, participants located insecure SSH private keys that “could be used to gain access to .212” (P2). It was noted that this does not relate to “a technical control, it's more of a people thing. You just need to tell people to secure their keys” (P2). Participants thus connected to the Old Database (.212) and captured its flag. Participants did not access the MySQL database, but it was noted that the server was “running outdated mysql” (P2). It was also noted that “Fail2Ban was not present on any of the SSH machines”, and that therefore “brute-forcing of passwords could be attempted” (P2). It is evident from P2’s notebook that the Database Server (.213) was also scanned and found to have the SSH port open, although they did not try to SSH into this server using the keys found in the User2 machine. The credentials to query both databases were located on the Admin2 machine and could have been accessed had the participants attempted to brute force the weak password of the test user. They, however, focused first on the EternalBlue vulnerability which then determined their path through the network.

With regard to the patch level, participants noted variations between machines, stating: “the use of outdated software was quite prominent in the network that I was testing, so that allowed for some pretty easy exploitation in some regards, but other machines were quite up-to-date and that made it a little bit harder” (P2). The variation in machine patch levels prevented lateral movement even after a foothold had been gained: “EternalBlue not patched but in all the others it was patched, so you cannot leverage that and move laterally”, as P3 explained. However, combining the circumvention of technical controls with the exploitation of human errors, such as the insecure SSH keys, allowed participants “to leverage those and move laterally – and with that, you actually bypass some of the patching and updated versions.” (P3).

We present our observations of the way in which control deployment may have impacted on the performance of participants. It is important to note that these observations are tentative, and cannot constitute hard evidence given the small scale of the study (in terms of the timeframe of the experiment, the number of networks tested and the number of participants). It is more fitting that our results contribute towards working hypotheses to be tested as future work.

1. Patch levels. Participants could easily compromise unpatched machines (particularly due to the EternalBlue vulnerability).
   
   1. While participants were able to compromise some unpatched machines, they were not able to leverage this to move laterally, due to the improved patching level of other machines. This cannot be seen as a security feature, as the time constraint likely prevented the lateral movement.
      
   2. Participants needed to leverage the misconfiguration of other control types in combination with the unpatched-machine vulnerabilities in order to compromise the machines. For example, User2 in Scenario 2 was only compromised because of the combination of EternalBlue and open SMB, while User2 in Scenario 1, it was not spotted, and the EternalBlue vulnerability therefore was never discovered, due to the host-based firewall blocking scans (ping).
      
2. Network segmentation. Participants did not access machines in isolated network segments unless using some inbuilt access-control rules.
   
   1. Web Server in the DMZ was not accessed by any participant. This was probably due to time constraints, as firewall rules allowed direct access to this network in Scenario 2.
      
   2. In Scenario 1, database servers were segregated from the critical network. Participants accessed the database on Database Server via Admin2 which contained a script that provided access to the database, but they did not access Old Database. In Scenario 2, on the other hand, in which the database servers were not isolated, participants did access Old Database, and scanned Database Server, finding that it had open SSH.
      
3. Encryption of data at rest. Participants were unable to access the flag included within the Database Server database's content in Scenario 1, even after accessing the database, since it had been AES-encrypted. This represents a level of database security in which certain parties (the administrators) are able to connect to the database, but require a separately stored password to read its contents.
   
4. Password security. Participants gained access to one machine by compromising its authentication credentials, and this was the Admin2 machine which had a weak (brute-forceable) password. This represents a valid method of account takeover, which is still one of the most predominant threats today, but there is a wide range of means by which credentials can be compromised, including social engineering, use of common passwords, shoulder surfing, and phishing emails.
   
5. Human and technical controls. Participants leveraged artefacts that represent human errors, alongside the omission of, and mistakes in the configuration of, technical controls, to achieve compromises, move laterally and capture flags on the network. It was noted, for example, that the presence of insecure SSH keys on Scenario 2’s User2 represents a human error, and this allowed bypassing of some technical controls such as the strong patch state of other machines. The importance of leveraging user mistakes to attacker actions was emphasised in the focus group: “most of the time, the attackers leverage the user level, all the mistakes that are done there, and all the artefacts that are left” (P3).