Computers & Security 88 (2020) 101607

Contents lists available at ScienceDirect

Computers & Security

journal homepage: www.elsevier.com/locate/cose

A framework for competence development and assessment in hybrid

cybersecurity exercises
Agnė Brilingaitė a, Linas Bukauskas a,∗, Aušrius Juozapavičius b
a
Institute of Computer Science, Vilnius University, Vilnius, Lithuania
b
General Jonas Žemaitis Military Academy of Lithuania, Vilnius, Lithuania

a r t i c l e i n f o a b s t r a c t

Article history: Rising numbers and sophistication of security threats in the digital domain cause an increase in the de-
Received 13 June 2019 mand for skilled cybersecurity professionals. In response, cybersecurity exercises, and in particular—cyber
Revised 3 September 2019
defence exercises (CDX) are becoming ever more popular. They provide a training platform to simulate
Accepted 5 September 2019
real-life situations. CDX are significant events involving months of preparation, and previous studies show
Available online 6 September 2019
a lack of objective evidence of their relevance regarding the learning impact. Skills of exercise participants
Keywords: are usually different and vary from tech-savvy to beginners. Also, trainees are diverse when considering
Cybersecurity skills their background, current work profile (position and institution), and experience. Assessment of their
Cyber defence exercises competencies is essential to ensure quality in training. The complexity and multi-dimensionality of the
Competence assessment usual CDX make it challenging. Additionally, the costly event usually focuses on just a subset of partic-
Hybrid exercises ipants, and non-technical members of an organisation are not included. The goal of our research is to
Competence development framework
provide a proper methodology to optimise the exercises so that every team and each participant, in-
Cybersecurity trainer’s questionnaire
cluding a non-technical trainee, are adequately evaluated and trained using the allocated resources most
effectively.
This paper presents a framework to aid in the development and assessment of cybersecurity compe-
tences of all teams during hybrid CDX. The framework aims towards raised cybersecurity awareness—a
state when every user of digital technologies understands the associated risks. The framework consists
of a sequence of steps including stages of formative assessment, team construction, determination of ob-
jectives for different types of teams, and the exercise flow. It complements standard methodologies for
cybersecurity training programs. The framework was developed based on data collected using question-
naires, interviews, and direct observation in a case study carried out during international cybersecurity
exercises. The framework would help organise hybrid exercises for a diverse community of trainees, in-
cluding non-technical members of an organisation.
© 2019 Elsevier Ltd. All rights reserved.

1. Introduction fails to ensure cybersecurity awareness and readiness at all levels

of user involvement in the digital world.
An increasing number of cybersecurity incidents and threat ac- Cybersecurity assurance in an organisation relies on both hard
tors affects businesses all over the world (ENISA, 2019; Morgan, and soft skills of all its employees. Technological and socio-
2019). Capabilities to protect against or discourage cyberattacks organisational solutions and skills should be combined as a hu-
are strongly related to readiness and competence of professionals man is a critical player in each organisation (White et al., 2004).
at the scene as well as cyber-literacy and awareness of the gen- Formal cybersecurity education systems in high-tech fields tend to
eral population. Cybersecurity exercises are used to alleviate the stamp a cliché on peoples’ behaviour and prioritise technical skills.
situation and train specialists. Most often, the exercises are or- Soft skills usually remain neglected, although they may be essen-
ganised for a dedicated and homogeneous group of professionals, tial in achieving a successful cybersecurity career (Joint Task Force
and traditional methods to evaluate participants and compare their on Cybersecurity Education, 2017). In real life, technical and secu-
achievements can be used. A narrow target audience allows the or- rity staff is surrounded by people having different levels of educa-
ganisers to streamline exercise objectives. However, this strategy tion in diverse fields. General competencies for cross-disciplinary
communication are necessary. Also, technical solutions have to be
communicated to different organisations at different levels of hier-
∗
Corresponding author.
archy and different levels of complexity. Aoyama et al. (2017) map
E-mail address: linas.bukauskas@mif.vu.lt (L. Bukauskas).

https://doi.org/10.1016/j.cose.2019.101607
0167-4048/© 2019 Elsevier Ltd. All rights reserved.
2 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607

types of exercises to the degree of organisation cybersecurity pre- Education. Cyber attacks should be treated as a precondition to
paredness to reduce misuse of resources. Cyber Defence Exercises all significant events (Ohta et al., 2018), but despite advances
(CDX) is the most common and expensive approach to train, test, in automating cyber attack detection, the primary defence el-
and verify the professional skills of organisation workforce at the ement is a trained human cyber defence specialist. Joint Task
highest preparedness tier. Force on Cybersecurity Education (2017) emphasises the urg-
European Network and Information Security Agency (ENISA) ing necessity and promotion of education in cybersecurity. Many
recommends to include the CDX as a part of the national cy- student-oriented competitions and exercises are organised for cy-
bersecurity strategy (ENISA, 2016). Exercises on a regular ba- bersecurity training and education. On the one hand, competi-
sis (ENISA, 2018) should test standard operating procedures of tions are used as sources to recruit talented people into the cy-
a state. However, specialist-oriented exercises are not enough. ber warfare forces (Andress and Winterfeld, 2014). On the other
Cybersecurity awareness and skills of a larger audience can be hand, CDX are being integrated into formal education as a part
improved if decision-makers and the public, in general, are in- of courses of cybersecurity study programmes (Mauer et al.,
cluded (Ogee et al., 2015), because IT and cybersecurity skills will 2012). Hoffman et al. (2005) list four types of cybersecurity ex-
be required in all future jobs. ercises organised for students in educational institutions to de-
A growing number of technical cybersecurity exercises and var- velop technical skills in combination with ethical behaviour and
ious hackathons attract many technically skilled participants. The teamwork. Also, implementation of the CDX method has proved
events also contribute to increased cybersecurity awareness and its worth as a proper competence evaluation and a motivat-
welcome beginners to the field. Due to the nature of CDX exercises, ing tool in cybersecurity-related military education (Schepens and
participants with different background are involved. They form op- James, 2003).
posing teams with dedicated responsibilities in complex simulated It is a challenge to create cyber exercises equally stimulating for
real-world situations. every participant. The passiveness of trainees who are either over-
We raise the research question whether CDX is an appropri- whelmed or insufficiently stimulated is a known problem (Henshel
ate tool to develop and assess cybersecurity-related competencies et al., 2016; Kick, 2014). Students may be involved in preparing
of all participants and if the concept of CDX can be improved to the CDX game itself to overcome this problem (Svábenský et al.,
address the learning needs of all trainees including non-technical 2018). The involvement usually gives plenty of positive motivation
persons. Based on our observations and findings in a case study, to learn about new vulnerabilities and attack methods.
we developed an improved organisational framework of hybrid Furtună et al. (2010) distinguish seven steps to create cyberse-
CDX to maximise the learning effectiveness. The framework com- curity exercises for training purposes. The steps start with objec-
plements the usual CDX life cycle and adds new phases to facilitate tives and end with evaluation and lessons learned.
competence assessment. We present the steps to apply the frame-
work to quantify the training result of all participating training en- Objectives. Tobey et al. (2014) point out that existing competitions
tities. The framework encourages CDX organisers to involve non-IT are more attractive to experienced participants than to novices. Es-
specialists to increase the resilience of an organisation against cy- tablishment of a preparatory environment to practice in advance of
ber threats. competitions might be a solution. Thus, the role and effectiveness
The structure of the paper is as follows. Section 2 elucidates of cybersecurity exercises should be discussed to find new cyber-
CDX-related challenges found by other researchers. Section 3 de- security talents and to encourage them to stay in the cybersecu-
scribes the methods we used to analyse the learning experience rity field. Wei et al. (2016) emphasise the need to educate general
of participants during international CDX. We present our find- masses because they are the weakest link in cybersecurity defence
ings in Section 4 followed by a discussion in Section 5. We fi- during their everyday business activities. Therefore, CDX should
nalise our paper with conclusions and possible future directions address the needs of a broader audience than technical cyberse-
in Sections 6 and 7, respectively. curity specialists.
Vykopal et al. (2017) identified the general life cycle of a cy-
2. Related work ber defence exercise consisting of five phases with an emphasis
on the preparation phase. The key success factor to the exercises
We group related work into several categories based on chal- is exercise difficulty matching the level of participants. Thus, dur-
lenges related to competence development and assessment in cy- ing the preparation phase, the participants should be assessed or
bersecurity. pretested to get their profile. This step could be implemented us-
A lack of cybersecurity workforce is identified globally, and re- ing surveys that include details about the employment, education,
gional initiatives try to address the problem. In Europe, ENISA a self-assessment of cyber defence, security knowledge, and other
leads information sharing and development of guidelines, conven- exercise-related information (Henshel et al., 2016).
tional approaches and procedures related to cybersecurity. In the An ideal person for cyber warfare operations is creative, hav-
United States, NIST developed NICE cybersecurity education frame- ing problem-solving skills, intelligent, and independent, but typi-
work (Newhouse et al., 2017). The widely accepted framework cally possessing such skills means a person does not tend to fol-
specifies knowledge, skills, and abilities to perform specific work low the rules well (Andress and Winterfeld, 2014). Team effec-
roles in cybersecurity. Paulsen et al. (2012) distinguish cybersecu- tiveness is a critical element that determines success during cy-
rity awareness as one of the vital NICE components. An increased ber exercises (Buchler et al., 2018b). Steinke et al. (2015) present a
cybersecurity awareness can be achieved through high(-er) educa- multitude of practical recommendations for cyber response team
tion and training of professionals. performance improvement based on training experience of mili-
Parrish et al. (2018) provide the framework for the cybersecu- tary, medical, and nuclear power plant operating teams. Adaptabil-
rity integration into the existing computing programmes defined in ity, problem-solving, sharing team knowledge, trust building, and
the ACM Computing Curricula series. They present cybersecurity as communication skills can be developed during pre-exercise train-
a meta-discipline that goes beyond the computing and engineering ing of participating teams. For example, adaptability could be im-
education and points out that cybersecurity is a cross-domain is- proved by applying perturbation training. In addition to psycholog-
sue. Therefore, cybersecurity-related components are defined, and ical challenges, each team also faces an environment overloaded
models of exposure to cybersecurity for all students regardless of with information. Team members use a variety of inter-team com-
their study field are provided. munication methods, information gathering and sharing tools, and
 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607 3

incident triage methods. These methods of team collaboration sig- back stimulates learning and improves participant satisfaction dur-
nificantly impact the overall performance of the team (Rajivan and ing the exercises. Also, educators and organisers can use the feed-
Cooke, 2017). Therefore, the methods should also be learned and back to tune the scenario to improve future exercises.
trained. Henshel et al. (2016) evaluated team dynamics using survey-
Buchler et al. (2018b) tried to find performance indicators by ing method. The survey was completed by an embedded observer
analysing collaboration and communication aspects. Greater face- of the Blue team to address skills like collaboration, communica-
to-face communication led to less effective team performance in tion, leadership, and task distribution. The main parameters to as-
tasks of service maintenance and incident response, although it sess technical skills were time to detect an incident and time to
was beneficial during scenario injects. Therefore, objectives of the report it. Surveying could be used together with observation and
exercises could be reached easier if functional role-specialisation analysis of data logs collected during the event (Granåsen and An-
was applied to construct teams (Buchler et al., 2018a) as the team dersson, 2016), although objective scoring was the main challenge.
results are influenced by proficiency, not by team size. Maennel et al. (2017) created a 5-timestamp model enabling ob-
Dawson and Thomson (2018) emphasise the importance of servers to assess group and individual skills within the Blue team,
blending technical skills with social and cognitive skills to develop e.g. time management, task distribution, leadership.
cybersecurity workforce. Thus, the scope of training and evaluation The assessment and evaluation are critical in cybersecurity edu-
should be broadened to incorporate metrics of organisational and cation that is influenced by new technological advancements. Thus,
social fit. The objectives of the exercises could be defined using the Dark and Mirkovic (2015) present five-step recommendations to
existing security training and education standards (Dodge et al., design evaluation in cybersecurity education to meet the needs of
2009) supported by organisations such as ACM, ISO, NIST. society. The evaluation must be framed by antecedents, transac-
tions, and outcomes that are defined based on the underlying be-
Team types. Typically, cybersecurity exercises are oriented to- liefs, assumptions, and theories. Cybersecurity society is interested
wards two teams: attackers and defenders (Furtună et al., 2010). in knowledge transfer, but education experts are rarely involved
Team types are distinguished based on an assigned colour. Blue in cybersecurity training activities. Therefore, teaming of the two
teams work as cybersecurity response teams, and Red teams work groups could enable the development of a reliable training sys-
as attackers. Other teams have their roles, too. In some inter- tem with an appropriate evaluation model to objectively track the
national exercises, the White team represents exercise managers, progress of learners (Mirkovic et al., 2015).
referees, organisers, and instructors, and the Green team consists
of operators and system administrators (Granåsen and Andersson, 3. Methodology
2016; Vykopal et al., 2018). In Cyber Shield (related to US Na-
tional Guard Bureau) class exercises (Henshel et al., 2016), EX- We performed a case study of joint military-civilian cyber-
CON team is distinguished to have a separate exercise control security exercises. The exercises aimed to improve international
group, and White team consists of three members with different military-civilian cooperation in defending critical IT infrastructure.
roles: Embedded Observer (EO), Training Analyst (TA), and Team Training of the cybersecurity incident response team was the main
Controller (TC) that are located in the space of Blue team, Red focus as specified in MITRE guidelines (Kick, 2014). Planning and
team, and EXCON, respectively. Sometimes additional team types preparation of the scenario and the cyber-range lasted just under
are introduced, or they are assigned different roles. For example, a year, and the execution took five full days.
a Grey team consists of White and Black in exercises conducted
by students (Mauer et al., 2012) where the White team represents 3.1. Team setup and game rules
business operations, whereas the Black team supports the techni-
cal infrastructure. In National Collegiate Cyber Defence Competi- The overall format of the exercise was of the Red-Blue (attack—
tion (CCDC, 2019), Gold, White, and Black teams represent organ- defend) type. A separate identical network of over 30 virtual and
isers, observers, and technical support, respectively. Consequently, physical machines was created for each participating team. The ex-
there is no commonly accepted team naming standard apart from ercises were hybrid: defenders had to prevent attacks on their live
the Blue and Red teams. infrastructure in real time, and they had to follow legal regulations
while reporting the attacks to simulated authorities. Four colours
Assessment. The European Network and Information Security were used to distinguish the participants. In Fig. 1 the colour-
Agency reports that about half of cybersecurity exercises organ- wheel of different teams and their relationships to services and re-
ised globally focus on training of participants and provide an op- sponsibilities is presented. White Team (WT) represents organisers,
portunity to gain knowledge, understanding and skills. However, Red Team (RT) — attackers, Blue Team (BT) — defenders, and Purple
evaluation of individual or organisation capabilities and measur- Team (PT) simulates business owners.
ing of knowledge are not common (Ogee et al., 2015). Therefore, The White team was responsible for physical and virtual infras-
Seker and Ozbenli (2018) argue the importance of evaluation and tructure management, exercise coordination (EXCON), and evalu-
scoring as a motivational and competition enabling instrument. ation. Observers were a part of this team and had access to all
Many researchers point out that the goal of most CDX is learn- information. Infrastructure as a service (IaaS) enabled the activ-
ing, but to assess individual participants or teams is a real chal- ities of every team. It was the responsibility of the White team
lenge as objective tools and methodologies are lacking (Henshel to control the IaaS and provide technical support during the CDX.
et al., 2016; Maennel et al., 2017; Vykopal et al., 2018). In par- Several members of WT simulated the state Computer Emergency
ticular, the standard assessment based on voluntarily filled out Response Team (CERT). They analysed team reports and acted ac-
questionnaires might be misleading and insufficient to identify the cording to the country’s legislation.
achieved proficiency during the exercises (Henshel et al., 2016). For The Red colour denoted skilled professionals who used auto-
some questions, self-assessment is more accurate than findings by mated and manual tools to perform attacks according to a pre-
observers, but it is still not a good predictor of performance during designed exercise plan. One Red team was responsible for attacking
CDX (Granåsen and Andersson, 2016). Thus, surveys are treated as every defending team.
an unreliable performance measure. The Purple team represented employees of a simulated organi-
Vykopal et al. (2018) emphasise the need for a timely and in- sation and business end-users. They had to use networked services,
dividual group-oriented feedback during the exercises. The feed- IoT, ICS, and perform routine daily business operations, including
4 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607

Fig. 1. An Overview of Teams and Their Responsibilities.

communication with external clients, and purchases and sales in a

simulated marketplace. The team members also had all the neces-
sary permissions to modify and change the services of their organ-
isation as long as they would not hinder the business. At the same
time, one Purple team acted as external business clients and per-
formed actual daily business activities creating an additional bur-
den for the internal PTs. This external PT was given a list of as-
signments they had to accomplish each day—orders to place, ser-
vices to obtain. In other cyber exercises, these assignments would
be called ”injects”, and they would be handed down to Blue teams
to satisfy specific business needs in addition to defending the busi-
Fig. 2. CDX Timeline.
ness. In our case, business operations and cyber defence operations
were assigned to different teams. The business activities also gen-
erated data flows obscuring some of the attacks and making the
scenario more realistic.
Blue team acted as a cybersecurity incident response team able, an annoying alarm would start beeping to increase the stress
(CSIRT) called to defend a simulated organisation. The formation level of the participants. The monitoring software was also directly
of Blue teams was left for their appointed team leaders to decide. connected to the scoring system.
However, each of them fell under one of the following categories: The CDX started with detailed setup instruction and a short
general introduction to participants. Each day the exercises started
• Ad hoc team members had no prior acquaintance and had little
with a joint briefing where the teams had to recap the last day’s
knowledge of personal or professional skills of each other.
events and were presented an intelligence report about an ever-
• Single organisation team members were from the same organi-
escalating situation in the region and country. In Fig. 2, the time-
sation and had prior knowledge of each other’s skills as well as
line of an exercise routine is shown. The exercise execution con-
previous working experience together. Teams of this type would
tained well recognisable stages balancing short breaks and actual
usually have a leader with strong technical knowledge.
work as well as information sessions. Daily briefings included in-
• Hierarchical local team was a hierarchically organised and man-
telligence information about future cyber-attacks and their types.
aged team with a formal leader, well-defined roles, under-
WT organised the coordination meetings, and the defending teams
standing of each other’s responsibilities, personal and profes-
were encouraged to have additional self-organised status check-
sional competencies from local homogeneous working places.
points and share knowledge during the break and internal brief-
The real-life working relationships among team members were
ings. All blue teams had to decide on the set of elements of the in-
cross-departmental with firm leadership.
frastructure to focus their defensive attention on while maintaining
• Hierarchical international team was a hierarchically organised
prescribed business activities. Each simulated organisation had one
team with a good understanding of each other’s professional
PT and one BT working together and sharing the working environ-
skills, but with an added international and multicultural aspect.
ment. In Fig. 3, a general scheme of interaction of all participating
Each BT had a separate room and working environment. The teams is presented. A single external PT shown on the right of the
workplace had several TV screens and various devices to emulate figure served all simulated businesses. Other PTs cooperated with
real-life SCADA systems used in the critical infrastructure of the their corresponding BTs. The responsibility of BT and PT went be-
organisation with visible elements, e.g. railway traffic lights and yond the deflection of the attacks. During an active defence phase,
alarm buzzers. The TVs were always showing output from differ- the joint team also had to analyse the attacks and correctly report
ent video cameras, SCADA systems, and the service and infrastruc- them via specialised ticketing system to a simulated CERT within
ture monitoring tools. If any of the machines or services on the the WT. Well-constructed reports of one or other Blue/Purple team
range became non-responsive, it was immediately apparent to the would sometimes induce the CERT to send warnings about the on-
teams. In some cases, if specific critical services become unavail- going attacks to other Blue/Purple teams as in real-life situations.
 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607 5

Table 1
Pre-event questions for blue teams.

No Question for the Blue Team and Answer Type

1. Have you ever participated in similar exercises? Please, provide some details about the exercises, your role, your lessons learned. Open ended.
2. How would you rate your skills, knowledge, and abilities in cybersecurity? Scale 1–5 (from Beginner to Professional)
3. What are your expectations for the event? Check all that apply. Show off yourself. Have a good time. Establish contacts. Get a certificate. Other (please, specify).
4. What are your strengths? Check all that apply. Firewall management. Usage of specialised software (to specify). Network analysis. Forensics. MS Windows OS.
Linux OS. Database administration. Soft skills. Any other (please, specify).
5. Any other comments. Open ended.

The post-event questionnaires included a set of identical ques-

tions for all teams and a set of specific questions for different team
types. The post-event questions are presented in Table 2. The first
column of the table shows the question number and team(s) that
received the question. Notation BPR means that the question was
given to all teams. A dash was used if the question was not pre-
sented to the particular team. For example, notation B–R means
that the Purple team did not receive the question.
Questions 1–6 in Table 2 were designed to obtain general feed-
back regarding the exercises and the learning experience from all
the participants. Blue and Purple teams had additionally to specify
Fig. 3. An Overview of CDX Team Interaction.
the technologies they used (7), indicate their knowledge gaps (8)
and challenges they encountered during the exercises (9).
Blue and Red teams reflected on their teamwork (10). They also
3.2. Observation and assessment
could suggest technological solutions and cyber attack types for fu-
ture exercises (11). Purple teams described the collaboration with
Previous research of cyber exercises had pointed out a lack
Blue teams (12) and expressed their expectations about participa-
of objective evidence of their usefulness, especially regarding the
tion in future exercises (13). The Red team identified technologies,
learning experience of the participants. Therefore, we had placed
roles, and skills used during the exercises to develop their attacks
observers in each team to monitor team collaboration and informa-
(14). Note, that some Red team members had worked with the
tion sharing, identify the learning curves of the participants, note
White team to prepare the infrastructure of the cyber exercises.
obstacles in maximising their learning experience as individuals or
Thus, some listed items are closely related to the preparation and
as a group.
support of the environment, e.g. hardware equipment, user sup-
The observers had several tasks. Firstly, they needed to assess
port. Red team members could also reflect on their challenges dur-
the participants and their expectations before the exercise. Sec-
ing the preparation of attacks (15).
ondly, they had to keep track of all individual activities from the
On the third day of the exercises, the Observers performed an
perspective of teamwork during the five days noting all the ob-
additional survey to measure the stress level of the Blue teams.
servations in timed handwritten notes. Finally, they had to survey
This particular day was chosen because it represented normal
each participant at the end of the event. Additionally, they had
working day conditions – during the first two days the teams had
an objective to try and map knowledge, skills, and abilities of ev-
yet to get used to the environment and each other, and the last
ery team member according to NICE roles based upon tools, com-
two days had planned interruptions because of visits by media and
mands and methods the observed participants used. Apart from
representatives of non-participating organizations. The third day
the survey data and observation logs, observers also had in-depth
also had a challenging scenario where the Red team had to hack
interviews with representatives of White and Red teams. Blue team
into railway systems and derail a simulated train. Therefore both
members were surveyed as the main focus was on the identifica-
before, during, and after the main attack, each BT had to determine
tion of the participant’s profile to know the training audience be-
their stress level and fill the questionnaire 3 every hour. Later, the
fore the event. After the event, members of Blue, Red, and Pur-
answers were supplemented with data from the observers and the
ple teams were surveyed. We made a hypothesis that members of
timing of the attacks.
all teams achieved some learning progress. Thus, the survey was
based on self-reflection on the usefulness of the event, challenges,
used technologies. The participants were also asked to provide sug- 4. Results
gestions for improvement. Questionnaires were provided to partic-
ipants to fill anonymously. Their relationships (team membership) Data gathered during the exercise enabled us to measure the
among each other were not recorded. learning effect on the participants. The CDX event had more than
The pre-event questionnaire included five questions (see 70 participants in total, including the organisers. There were four
Table 1). The questions were related to prior experience (ques- BTs (24 trainees), five PTs (14 trainees), and one RT (9 members).
tion 1), skill level (2 and 4), and expectations (3). The respondents We obtained feedback from all of the participants, except 3 PT
could provide details about their previous roles in other cybersecu- members. Results obtained from different teams cover several im-
rity exercises, specify personal strengths in the usage of technolo- portant areas related to exercise objectives.
gies, and give some comments to organisers (5). A set of technical BT Profile. Typically, defence oriented CDX focus on training of
fields, e.g. firewall management, Linux OS, as well as the possibil- BTs. The BT participant profile was constructed using collected data
ity to select soft skills, were listed in question 4 to narrow down from the questionnaire presented in Table 1. Each participant self-
the profile of Blue team members. There was a possibility to add evaluated proficiency in cybersecurity (five levels) and described
and specify a specific professional skill. The list of technical fields the previous experience in CDX (number of times, roles, and CDX
was chosen after discussions with the WT to match the prepared types). The constructed BT profile is presented in Fig. 4. The expe-
attack plan. rience was classified into four colour-coded groups depending on
6 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607

Table 2
Post-event questions.

No.Type Question and Answer Type

1. BPR Was it worth participating in the exercises? Yes or No

2. BPR How did the exercises meet your expectations? Scale 1–5 (from Poorly to Perfectly)
3. BPR What did you learn during the exercises? Open ended.
4. BPR What did you like in the exercises (atmosphere, particular incidents, etc.)? Why? Open ended.
5. BPR What did you dislike in the exercises? Why? Open ended.
6. BPR Any other comments, suggestions, feedback, ideas, thoughts that would help to organise and manage such an event (organisation, setup,
etc.). Open ended.
7. BP-- Which of the following have you used during the exercises? Check all that apply. Firewall management. Usage of specialised software (to
specify). Network analysis. Forensics. MS Windows OS. Linux OS. Database administration. Soft skills. Any other (please, specify).
8. BP-- Which of the following would you like to learn/improve? Check all that apply. Firewall management. Usage of specialised software (to
specify). Network analysis. Forensics. MS Windows OS. Linux OS. Database administration. Soft skills. Any other (please, specify).
9.BP-- Did you find any attack, task, or disruption challenging? If yes — which and why? Open ended.
10. B--R How would you describe your teamwork? Did you have any problems? If yes — what? Open ended.
11. B--R What technological solutions or types of attacks might be considered next year? Open ended.
12. --P-- How would you describe your collaboration with a blue team? Did you have any problems? If yes — what? Open ended.
13. --P-- Would you like to participate in the event next year? If yes—what team (red, blue, white, purple) would you like to choose? Open ended.
14. ----R Which of the following have you used during the exercises? Check all that apply. Network configuration, network maintenance, network
monitoring and analysis, virtualisation, software installation, resource allocation, scripting, hardware/equipment installation, MS Windows OS,
Linux OS, Database administration, firewall management, data analytics, forensics, programming, testing, server administration,
scenario/algorithm design, soft skills, user support, usage of specialised software (specify), other (specify)
15. ----R Did you find the preparation of an attack, task, or disruption challenging? If yes — which and why? Open ended.

Table 3
Stress management self-evaluation form.

Time:

What is the stress level of the team Scale 1–5 (low to high)
How confident do you feel handling Scale 1–5 (not confident to
the situation? very confident)
Explanation of the evaluation Open ended

Fig. 5. Blue Team Participant Skills.

Skills of Participants. After the exercises, BTs and PTs answered

questions concerning cybersecurity-related skills they used dur-
ing exercises and skills they would like to learn afterwards (see
Table 2). Additionally, BT members indicated skills they possessed
before the exercises (see Table 1). The trends of skills are pre-
sented for BTs and PTs in Figs. 5 and 6, respectively. More than
50% of BT participants would like to improve their skills in fire-
Fig. 4. Blue Team Participant Profile. wall management, network analysis, forensics, and Linux OS. Only
the development of skills in firewall management is relevant for
more than 50% of PT members (approx. 70%). Approx. 35% of PT
the number of previous exercises: no experience, 1-time experi- participants would be interested to learn network analysis, Linux
ence, experienced (2–10 times), and very experienced (more than OS, and database administration. Firewall, Linux OS, and forensics
10 times). Most of the participants self-assigned their proficiency seem attractive to PT members even if these particular skills were
to medium level and below (1–3). not needed for Purple teams during the exercises.
Similarly, over half of the participants had no previous expe- BT members were skilled in MS Windows OS and possessed
rience in CDX. Only a few participants self-evaluated proficiency soft skills. Also, more than 45% of BT and PT members indicated
as high and very high (levels 4–5), and they were very experi- that soft skills and MS Windows OS skills were required, but they
enced. Moderately experienced participants evaluated themselves are not as attractive as other cybersecurity-related skills for further
at a medium proficiency (level 3). Among medium level partic- development.
ipants, there were only a few with no experience, and the rest Fig. 7 presents the number of skills per individual for BTs and
had participated in at least one CDX event. Thus, the audience was PTs for different time points of the exercises. Box plots show mini-
composed of novices in CDX with medium or lower proficiency in mum and maximum numbers per individual, a median, and a stan-
cybersecurity. dard deviation. BT participants had a small number of skills (with
 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607 7

Out of all PT participants, 45% would like to be a part of BT only,

and 18% would only select PT. Other respondents indicated that
they would like to belong to any of the three teams. Out of 72% of
participants that chose BT, 27% indicated BT or any other team. Out
of 36% of participants selecting PT, 18% responded that the prefer-
ence could be PT or some other team(s). No one answered that RT
would be the only option for consideration. The results show that
PT members are rather ambitious and consider themselves eligi-
ble for BT and even RT, but they do not single out a broad set of
skills for further personal development as presented in Fig. 7. Some
PT members experienced the exercises as not challenging enough.
Also, several PT members think they could be more involved in BT
work to learn more.
Proper Start. Most of BT members emphasised the need
for some pre-training before the exercises. The critical im-
portance of preparation was also pointed out by other re-
Fig. 6. Purple Team Participant Skills. searchers (Maennel et al., 2017). The novices would expect some
learning material and courses or lectures to have better orienta-
tion at the beginning of the exercises. A lack of preparations made
a start confusing, e.g. missing information, unclear rules, as par-
ticipants made assumptions based on their prior experience. The
participants characterised the start as fast without proper analysis
of the setting (network). Thus, participants treated exercises more
as a training and learning event than a competition.
Team Building and Collaboration. BT members highlighted
that the exercises enabled the development of team collabora-
tion and coordination (internally and externally) and the ability to
work as a team. Task distribution was mentioned as a challenge
and as an ability developed during the exercises. Participants men-
tioned a decision to divide themselves into subgroups to solve the
tasks. Most BT respondents noted that their teams were perfect,
team members were patient, helpful, and understanding. However,
some BTs experienced miscommunication and problems with in-
formation sharing (insubordination) during the first days. A short-
age of time allocated for team building was a negative aspect.
Fig. 7. Number of Skills BT and PT Participants Claimed Before, Used During, and Several respondents suggested prior team building exercises, espe-
Expressed an Interest to Develop After the Exercises. cially for teams that contained members not knowing each other,
e.g. ad-hoc and hierarchical international teams. The pre-training
week could be used for team building as well as for ”touching the
range”. Based on questionnaires, teaming was also important for
RT and allowed expressing oneself. RT team was built on active
core members who knew each other, but the members struggled
with various difficulties and challenges, e.g. coordination, cooper-
ation, a lack of human resources (e.g. for writing documentation),
not enough time for attack testing, and keeping up with deadlines.
However, they liked the stress, challenges, incidents, and the level
of attack sophistication.
Cross-Team Cooperation. PT members underlined they under-
stood the importance of communication with team members and
other organisations. Some PT participants would expect more ac-
cessible communication with the White team. BT members liked
the exercise emphasis on communication, meeting new people
during breaks, and access to other BTs/PTs. PTs highlighted an
excellent atmosphere, challenges, and communication as positive
aspects of the exercises. PTs mentioned some miscommunication
Fig. 8. PT Participants Expectations for the Next Year. with BTs at the beginning of exercises, but later their collaboration
was excellent, friendly, and even fun. BT participants desired to ob-
tain the technical details of the attack implementation. They liked
a couple of outliers), but they used more skills than they listed be- briefings and missed more thorough and tech-styled explanations
fore the exercises. After the exercises, they created larger sets of about the how and the what from the RT.
skills to be developed. PT participants had to use a limited set of Learning Curve. PT members emphasised they learnt a lot of
skills during the exercises. Afterwards, they expressed an interest new things about attacks and their impact on the system. They
to develop only a few, although completely new competencies. found it interesting to observe the effects of an infected file and
PT Ambitions. PT members answered the question if they to notice service perturbations. It was ”fascinating and yet scary
would like to participate in CDX next year and what team they how simple it is to break a network and how hard it is to restore
would choose. The answers to this question are presented in Fig. 8. it.” BT members marked that they improved technical skills, e.g.
8 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607

Fig. 9. Red Team Participant Skills.

usage and configuration of monitoring tools, network analysis, and increased the awareness of individuals regarding their knowledge
incident analysis. The attacks were described as challenging, not and skill gaps.
obvious, exciting, fun, and difficult at the same time. Stress man-
agement self-evaluation form (see Table 3) revealed that all teams 5. Discussion
except the ad hoc team experienced low-stress level throughout
the exercises. The confidence level for most teams increased dur- We did a case study on a hybrid CDX to find out if this type
ing the period of intensive attacks, even when they were struggling of event is optimal for cybersecurity competence development
with unavailable services. The ad hoc team had decreased confi- and assessment. CDX observers performed surveys, interviews, and
dence during the attack, and one team had a high confidence level continuous observation of participants focusing on learning activ-
(4 out of 5) throughout the stress observation. Half of the teams ities, collaboration, and self-assessment. The methodology encour-
failed to protect their systems, but their reported stress level re- aged self-reflection and helped to answer our research question.
mained very low. Thus, their self-confidence did not correspond to We present several CDX improvements based on our findings.
the performance. Granåsen and Andersson (2016) had reached a Proper arrangement of CDX could enable the development
similar conclusion about self-assessed expertise being a poor pre- of competencies of all involved participants. CDX is a resource-
dictor of performance. The BT participants liked the variety of at- intensive event, and it could be used more effectively than just for
tacks. They also suggested several improvements for future exer- a specific group of individuals. Our results show that Purple Team
cises: more noise in the range, higher attack speed (frequency), a members also gain knowledge about cybersecurity, and they would
larger number of attacks, tasks in forensic and social engineering, like to be involved more actively than usual. PT represents business
a spy in a blue team that would try to sabotage the team, and co- users, decision makers, managers, and fresh or non-technical users
operation with other BTs as a mandatory element to solve some whose actions may have a considerable impact on cyber defence
tasks. PTs enjoyed the element of unexpectedness, a possibility to in any organisation. Thus, organisers could include specific tasks
learn web system administration, and similarly to BTs, they would for PT in the scenario to increase their cybersecurity awareness.
like more social engineering aspects. RT members mentioned they Objectives of CDX should cover all participating groups. Spe-
gained experience in organising the event, planning the overall pic- cific learning outcomes should be defined for the defenders (Blue
ture, and understanding psychology of the blue team. They made Team), attackers (Red Team), business users (Purple team), and
new connections and improved critical thinking. Fig. 9 presents even infrastructure support specialists (White Team). The outcomes
skills that were needed by RT members. The majority of RT mem- should not necessarily be related to technical skills. During our
bers performed tasks requiring network maintenance, software in- case study, Red Team members pointed out their soft skill devel-
stallation, virtualisation, scripting, Linux, and server administra- opment, e.g., the psychology of Blue Teams, collaboration, and time
tion. They had challenges with the scenario and attack coordina- planning, even though they were not the main target group of the
tion, virtualisation and configuration. In questionnaires, RT mem- exercises.
bers mentioned they learnt new management tools and new attack List of competencies to be trained should be derived directly
methods. Also, they identified areas for future personal develop- from CDX objectives and not from the assigned roles of team mem-
ment. They used soft skills and had to provide user support. There- bers. During the analysed CDX, a specific shortened list of the rele-
fore, participation in RT provides opportunities for self-assessment vant NICE framework roles had been made before the exercise. The
and gaining experience in the application of technical and soft Observers were given a task to match the roles to each of the par-
skills. ticipants, but despite a considerable effort, they failed. Many par-
According to the majority of respondents, regardless of the ticipants exhibited skills and knowledge assigned to many different
team type, the general atmosphere and environment of the exer- roles with a blurred overlap. The reason was a broad spectrum of
cises enabled learning, self-reflection, and improvement in techni- attacks and a variety of used technologies. An overabundance of
cal and soft skills. The results are consistent with Observer reports possible safeguarded targets forced members of limited-size teams
and insights from an interview with RT members. The exercise to assume different roles during different attacks. Therefore, to
 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607 9

Fig. 10. Detailed Layout of Training Phases.

train all competencies of particular roles, the scenario of attacks

should be adapted correspondingly. For example, the organisers
should choose only particular attack types and then observe and
evaluate specific steps performed by a participant playing a partic-
ular role. Alternatively, a specific subset of competences of chosen
roles can be selected and developed according to the objectives of
the CDX.
CDX time-flow should include dedicated points to assess learn-
ing outcomes derived from the objectives for different types of
teams. According to scientific literature, attempts to measure the
performance of participants had considerable challenges and un-
certainties. During CDX, team performance can be based on many
criteria. Measured data points represent a single moment result,
but not the impact of the whole CDX on competence development.
Tracking of an individual learning curve is even more challenging
as it requires substantial resources. Also, a team performs success-
fully if at least one member has the necessary skills to solve an
incident or prevent an attack. Personal learning value of the CDX
mostly comes from anecdotal evidence. Thus, time should be allo-
cated for assessment separately from the training activities.
Pre-training should become an integral part of the exercise
to fill the competency gap for all individual participants. First Fig. 11. General Incident Flow During CDX.
of all, each participant needs skills outside the specialisation or
previous experience. Secondly, CDX is a team based event. Thus,
even professionals should participate in team building activities performance under stress, then experience gathered from medi-
before the main competition. Pre-training would also serve as a cal or other emergency response teams can be applied by design-
platform for initial assessment of the audience. It would aid as ing special pre-training exercises. Also, pre-training should specifi-
an intrinsic motivation tool and as a baseline for learning curve cally cover the legal base to emphasise the importance of compre-
measurements. hensive reports. Instructions on how to use the reporting system
To sum up our findings, we propose an extended competence should be provided, as participants tend to give low priority to the
development and assessment (CDA) framework. The CDA frame- reporting task. Pre-exercise training should include an assessment
work combines methods, tools, and procedures to develop and as- strategy to follow the learning progress of participants.
sess competences of all CDX participants including non-technical Phase 3 is a standard main component of CDX where pre-
players. The framework supplements a typical CDX life cycle to en- trained teams play against each other as well as train to respond
able competence assessment at an individual level to reach learn- to incidents by facing unpredictable non-routine challenges. Typi-
ing objectives. It consists of four phases detailed in Fig. 10. cal exercises are competitive, and training activities are underused.
Phase 1 is dedicated to Pre-exercise assessment of the training Phase 3 should include hot wash-up covering more details than
audience and design of learning objectives. The learning objectives usual to cover technical details about attack implementation and
of the CDX depend on the audience type (proficiency, experience, defence strategies. As a rule, each team operates in an isolated
specialisation), and competence map might be different for each environment. During hot wash-up, one team can be chosen to
team type (Red, Blue, Purple, White). During Phase 1, the learning present their defence strategy against a specific attack. Thus, teams
objectives are adjusted based on the CDX goals and the identified can share their best practices and learn from each other. Some at-
participant audience profile. tacks can be periodically repeated several times to objectively as-
Phase 2 covers pre-exercise training of teams and individu- sess performance improvement and help the teams achieve learn-
als. An individual participant may be trained based on personal ing outcomes.
competence matrix and task specifics for the role assigned in a Simulation of a real-world environment serves as a motivational
team. The basis for efficient teamwork is built by organising var- tool. Some time of CDX should be allocated for ”competitive exer-
ious group activities. For example, team building would be ben- cises” even when the primary goal of CDX is training. A scoring
eficial to ad-hoc teams. If CDX objectives include evaluation of system encourages competition atmosphere. We suggest a scoring
10 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607

Table 4 Table 5
A description of timestamps from Fig. 11. An example mapping of exercise objectives to exercise phases.

Action Objective P1 P2 P3 P4

[T1] WT orders RT to initiate a cyber attack. Apply tool X for network monitoring    
[T2] RT launches the attack and records the result. Ensure web-server availability    
[T3] PT notifies BT of service failure, or BT detects the attack. Recover service after crash    
[T4] BT evaluates the threat level and sends a short report. 
[T5] BT defends and recovers the system. Perform incident triage    
[T6] BT submits detailed report of the incident with indicators
of compromise and possible attribution if possible.
[T7] WT orders RT to validate the previous attack.
[T8] RT reports the final status of the attack.
phase where the participants are trained and assessed to achieve a
corresponding learning outcome.
The number of sub-phases could be easily adapted to support
system covering both attack-defend actions and reports by extend- individual and team competence learning objectives.
ing the 5-timestamp model (Maennel et al., 2017) by adding three The CDA framework complements the traditional CDX life cycle
additional points (T6–T8). The schema of the typical workflow of having four stages: Identify, Plan, Conduct, and Evaluate. In Fig. 12,
an incident from activation to resolution is depicted in Fig. 11. WT the mapping of the CDX life cycle to the CDA phases is shown, and
initiates incidents according to the CDX scenario. RT launches the key results are presented in the corresponding stage and phase. As
attack, and then PT together with BT should detect it, take de- depicted in the figure, CDA phases are spanning over the whole
fensive actions, investigate the incident, and report back to WT CDX life cycle. Usually, training and assessment are concentrated
(see Table 4). Timestamps of each step of the incident flow can in the Conduct stage. However, CDA activities should start in the
be recorded either automatically or by Observers. Interval lengths Identify stage and continue past LiveEx to benefit most from the
can be scored to measure team (and sometimes—individual) per- CDX. The presented layout of CDA activities enables a learner-
formance as described by Maennel et al. (2017), e.g., T3–T2 mea- oriented approach. Learning objectives and assessment strategies
sures Time to Detect, and T5–T2 corresponds to Time to Restore. are adapted to the training audience based on the participant pro-
Validation of the recovered service (steps T7, T8) could even be file. Also, the learning objectives can be differentiated for technical
performed during another incident independently of the current and non-technical participants. Results of the post-exercise assess-
status of the system as well as multiple times. During hybrid CDX, ment phase can be integrated into CDX first impression and final
the game does not end with an incident resolution and requires reports as a measurable indicator of the CDX value.
additional efforts from the BT to perform incident triage and de- A detailed timeline of CDA framework activities is described in
scribe the process in detail (T6). WT can additionally score the Table 6. The first and last columns indicate periods of CDX life cy-
quality of the reports as well as use them during hot wash-up ses- cle stages and CDA phases, respectively (corresponding to Fig. 12).
sions to educate less successful teams. The other two columns describe and explain suggested steps, ac-
Phase 4 of CDX in our framework is dedicated to the post- tions, responsibilities, and tools with a clear assignment to the
exercise assessment of competencies of individual participants be- phases. The table specifies only steps related to competence devel-
cause Phase 3 can mostly be used to determine team performance. opment and assessment, while other standard CDX life cycle activ-
If CDX is implemented as a part of formal education, then the most ities are omitted (see MITRE Playbook for their list (Kick, 2014)).
common assessment strategy with assessment criteria can be ap- The CDA activities require extra resources, and the White Team
plied to grade the students using tests and practical assignments takes on additional responsibilities. Therefore, an Educational Team
as assessment methods. Alternatively, if CDX is organised to train (EduTeam) is created as a part of the White Team. The EduTeam
professionals, then less formal assessment methods such as ques- mainly consists of observers and academic personnel, and it closely
tionnaires would support self-reflection about the development of collaborates with the EXCON. The EduTeam selects a suitable com-
skills. Assessment results from all phases are compiled into the fi- petence framework, prepares a pre-training plan for all partic-
nal assessment report giving a comprehensive view on the impact ipants, and creates a competence development and assessment
of the event both to individual participants and to each team. schema corresponding to the defined CDX concept and objectives.
Learning objectives are defined in Phase 1 of CDX with many Competence frameworks are extensive. Thus, the EduTeam should
assessment points during the four phases of the exercise. Assess- identify key competences to be developed during the CDX. For ex-
ment methods should match the assessment strategy and criteria ample, according to the NICE framework (Newhouse et al., 2017)
based on learning objectives. We suggest using a competence plan- a person in the database administrator role should be able to per-
ner to keep track of the objectives during all the phases as pre- form more than ten tasks and possess knowledge and skills in over
sented in Table 5. Each objective should be mapped to at least one twenty topics and areas. Some of them cannot be easily developed

Fig. 12. A High-level Overview of the CDA Framework.

 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607 11

Table 6
A detailed timeline of the CDA framework.

Step Description, examples, notes

IDENTIFY Concept development meeting. Key stakeholders define CDX The White team determines size and type of CDX, possible
concept, objectives, and participant profile. training audience, and other essential parameters.
EduTeam selects reference competence frameworks for The NICE framework might be used for IT persons, ACM/IEEE—for Phase 1
technical and non-technical participants. students in a higher education institution, and information or
cybersecurity curricula—for non-IT persons.
EduTeam identifies roles, tasks, and competencies to be A subset of competencies to be developed during the CDX is
trained and assessed based on the CDX objectives and creates chosen from the selected frameworks. Then, the subset is
a competence map (CM). narrowed for detailed assessment. Different parts of CM may
apply to different teams and individual roles.
PLAN Initial planning meeting. White Team supplies an initial Note, the list of participants might be incomplete at this stage.
participant list. Stakeholders provide feedback on CM.
EduTeam prepares and uses remote tests and/or Assessment tools and methods (e.g., virtual learning
questionnaires to profile participants. environment, test cyber ranges, or online questionnaires) are
used to determine the current level of participants according to
CM.
EduTeam updates CM and defines learning objectives (LOs). Based on the obtained results, EduTeam defines LOs using Bloom,
SOLO, or other taxonomy.
Main planning conference. EduTeam and White Team CM should be fully covered at this point. Specifically, the
coordinate requirements for the environment, range, attack scenario should address the training of the PT. Tools are selected
vectors, and scenario to enable development and assessment to facilitate automated and semi-automated scoring of teams.
of competences according to the set LOs. Appropriate legislative documents, rules of engagement, and
technological solutions applied during the CDX are determined
during this step.
EduTeam makes a plan for pre-training, selects assessment Pre-training schema is selected. EduTeam prepares tools for the
methods and criteria for Pre-exercise training, LiveEx, and assessment, e.g. questions and tasks for the pre-exercise
Post-exercise assessment phases. activities of all teams (and roles, if applicable) with assessment
criteria in the virtual learning environment.
Final planning conference. EduTeam finalises assessment EduTeam and White Team define what group and individual
methods. White Team provides participant lists. tasks of PT, BT, and RT will be used in the assessment process.
Specifically, RT might get additional tasks during LiveEx.
EduTeam initiates pre-training (moderate intensity, The learning/training material (e.g. documentation, instructions, Phase 2
self-driven, off-site). information on remote access to test range, first tasks) is
provided to participants based on the pre-training schema.
CONDUCT EduTeam continues intense pre-training on-site. Pre-training schema is implemented. Different teams might get
Participant/team learning progress is assessed using selected specific training that corresponds to the CDX scenario. E.g. PTs
tools. Team profiles are created. study cases on how to avoid cybersecurity incidents by
implementing security policies and how to report detected
incidents in the context of business processes and social issues of
the organisation.
LiveEx. CDX progresses according to Fig. 2 timeline. White The assessment and scoring schema is implemented. Observers Phase 3
Team monitors, scores, observes and assesses teams, gathers make notes and log participant activities/behaviour based on the
factual observation data. assessment criteria, e.g. PT ability to follow the organisation
security policy described in the playbook, and BT ability to
re-define roles or take leadership after removal of a member.
Repeated attacks could be implemented to observe learning
progress.
White Team organises the collection of participant A self-reflection session should be organised right before the Phase 4
self-reflection on the CDX. EndEx to collect fresh impressions.
EVALUATE EduTeam prepares the final assessment report and issues Analysis of the log and observation data is performed to make
competence certificates. White Team integrates the assessment the final assessment and grading. Additional scoring points might
results into the first impression and final reports. be given for tasks completed by teams and/or competences
shown. In formal education, an exam might be organised.

and assessed during the CDX (e.g. Provide recommendations on new ber range implementation), and post-exercise assessment type (e.g.
database technologies and architectures), others are off-topic regard- exam). The EduTeam collaborates with all other White Team sub-
ing the CDX objectives (e.g. a skill in optimising database perfor- groups and influences preparation of the scenario to cover the
mance). Usually, LiveEx requires a lot of different competences to competence development needs of all team types, especially the
deal with challenges during the attacks. However, it would require non-technical participants. Therefore, CDA framework enables to
huge resources to develop and assess each of them individually. develop, objectively assess, and certify competences of all CDX par-
Hence, the EduTeam selects key competences during the Identify ticipants.
stage and narrows down the list during the Planning stage. The
short-listed competences result in learning objectives and compe- 6. Conclusions
tence development and assessment schema (tools and methods):
pre-training tasks and type, material, lectures, case studies, test Acquisition of cybersecurity skills and abilities requires commit-
questions, questionnaires, assessment criteria for level identifica- ment and time. Exercises aimed at developing cyber skills are also
tion, attack sequence and implementation (e.g. re-launch), learn- time-consuming and costly events that require dedication and col-
ing/training environment (e.g. virtual learning environment, cy- laboration of trainers and trainees. Our analysis of related works
12 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607

and our observations showed that purely competitive exercises do bersecurity exercise ”Amber Mist2018” for the opportunity to ob-
not satisfy the learning needs of every participant and fail to mea- serve and gather data. The authors are also thankful for the com-
sure their learning progress. To optimise the training output, we puting resources provided by the IT Research Center of Vilnius
presented a framework addressing the competence development University.
needs of a broader exercise audience.
The framework partially sacrifices the competitive nature of
References
CDX. We recommend all teams to spend more time analysing, re-
porting and reflecting on attacks. Several assessments and focused Andress, J., Winterfeld, S., 2014. Cyber Warfare—Techniques, Tactics and Tools
training stages embedded within the exercise would enable the for Security Practitioners, 2nd Syngress, an imprint of Elsevier doi:10.1016/
C2013- 0- 0 0 059-X.
measurement of the learning curve and add additional motivation
Aoyama, T., Nakano, T., Koshijima, I., Hashimoto, Y., Watanabe, K., 2017. On the com-
for participants. The framework makes it possible to use CDX ef- plexity of cybersecurity exercises proportional to preparedness. J. Disaster Res.
fectively both in professional training and formal education. Also, 12 (5), 1081–1090. doi:10.20965/jdr.2017.p1081.
the initial evaluation of participants before the exercise allows the Buchler, N., La Fleur, C.G., Hoffman, B., Rajivan, P., Marusich, L., Lightner, L., 2018a.
Cyber teaming and role specialization in a cyber security defense competi-
organisers to know the training audience better and suggest possi- tion. Front. Psychol. 9 (2133), 17. doi:10.3389/fpsyg.2018.02133. https://www.
ble ways to fill in knowledge gaps. With such a framework, novices frontiersin.org/article/10.3389/fpsyg.2018.02133
would not be overwhelmed with complexity and would be moti- Buchler, N., Rajivan, P., Marusich, L.R., Lightner, L., Gonzalez, C., 2018b. Sociomet-
rics and observational assessment of teaming and leadership in a cyber security
vated to stay in the field of the cybersecurity. defense competition. Comput. Secur. 73, 114–136. doi:10.1016/j.cose.2017.10.013.
http://www.sciencedirect.com/science/article/pii/S0167404817302298
7. Future work Dark, M., Mirkovic, J., 2015. Evaluation theory and practice applied to cybersecurity
education. IEEE Secur. Privacy 13 (2), 75–80. doi:10.1109/MSP.2015.27.
Dawson, J., Thomson, R., 2018. The future cybersecurity workforce: going beyond
Future work can have several research directions related to technical skills for successful cyber performance. Front Psychol 9, 744. doi:10.
competence development of cybersecurity specialists and organi- 3389/fpsyg.2018.00744.
Dodge, R.C., Hay, B., Nance, K.L., 2009. Standards-based cyber exercises. In: Proceed-
sation of CDX. ings of the The Forth International Conference on Availability, Reliability and
Firstly, future studies could explore the application of the CDA Security, ARES 2009, March 16–19, 2009, Fukuoka, Japan. IEEE Computer Soci-
framework in different settings and a broad audience of partici- ety, pp. 738–743. doi:10.1109/ARES.2009.72.
European Union Agency for Network and Information Security (ENISA), 2016. NCSS
pants. For example, the timeline points (steps of EduTeam), domi-
Good Practice Guide Designing and Implementing National Cyber Security
nant competences, pre-training duration, and proportion of the in- Strategies. Technical Report. Publications Office of the European Union doi:10.
dividual and group training in formal education and professional 2824/48036. TP-05-16-002-EN-N, ISBN: 978-92-9204-179-3
training might be determined, compared, and optimised. European Union Agency for Network and Information Security (ENISA). Cyber Eu-
rope 2018—After Action Report Findings from a cyber crisis exercise in Europe;
Further investigation in the design of teams is necessary to 2018. TP-06-18-410-EN-N, ISBN: 978-92-9204-287-5; 10.2824/369640.
identify an optimal team composition to perform assigned tasks European Union Agency for Network and Information Security (ENISA). ENISA
effectively. We envision early mappings of individual competences Threat Landscape Report 2018; 2019.. 10.2824/622757.
Furtună, A., Patriciu, V.V., Bica, I., 2010. A structured approach for implementing
would aid in the team composition, but the criteria should be cyber security exercises. In: 8th International Conference on Communications.
analysed and identified based on the organisational environment, IEEE, pp. 415–418. doi:10.1109/ICCOMM.2010.5509123.
attack types, and other relevant parameters. Also, an exciting re- Granåsen, M., Andersson, D., 2016. Measuring team effectiveness in cyber-defense
exercises: a cross-disciplinary case study. Cognition, Technology & Work 18 (1),
search topic would be to determine the influence of the number 121–143. doi:10.1007/s10111- 015- 0350- 2.
of high-level professionals on the learning curve and motivation of Henshel, D.S., Deckard, G.M., Lufkin, B., Buchler, N., Hoffman, B., Rajivan, P., Coll-
other team members (e.g. ad-hoc novices). Alternatively, future re- man, S., 2016. Predicting proficiency in cyber defense team exercises. In: MIL-
COM 2016 - 2016 IEEE Military Communications Conference, pp. 776–781.
search could consider the effect of inclusion of non-technical peo- doi:10.1109/MILCOM.2016.7795423.
ple in BTs, e.g. a scenario could provide these participants with ex- Hoffman, L.J., Rosenberg, T., Dodge, R., Ragsdale, D., 2005. Exploring a national
tra facts to simulate an increased situational awareness. cybersecurity exercise for universities. IEEE Security & Privacy 3 (5), 27–33.
doi:10.1109/MSP.2005.120.
The CDA framework would benefit from a set of semi-
Joint Task Force on Cybersecurity Education, 2017. Cybersecurity Curricula 2017:
automated tools designed to generate a skeleton of the scenario, Curriculum Guidelines for Post-Secondary Degree Programs in Cybersecurity.
attack vectors, tasks, and assessment criteria based on the com- Technical Report. ACM/IEEE/AIS-SIGSEC/IFIP WG 11.8, New York, NY, USA doi:10.
petence map chosen by the CDX organisers. A generator algorithm 1145/3184594.
Kick, J., 2014. Cyber exercise playbook. Technical Report. MITRE Corp Bedford, MA,
could be created after specific case studies investigating combina- USA. https://apps.dtic.mil/dtic/tr/fulltext/u2/a624910.pdf
tions of roles, attacks, participant competences, and rate of involve- Maennel, K., Ottis, R., Maennel, O., 2017. Improving and measuring learning ef-
ment of non-technical trainees. fectiveness at cyber defense exercises. In: Lipmaa, H., Mitrokotsa, A., Mat-
ulevičius, R. (Eds.), Secure IT Systems - 22nd Nordic Conference, NordSec 2017,
Finally, future research could cover the analysis and develop- Tartu, Estonia, November 8–10, 2017, Proceedings, volume 10674. Springer,
ment of tools to facilitate the assessment of the trainee perfor- pp. 123–138. doi:10.1007/978- 3- 319- 70290- 2_8. Lecture Notes in Computer Sci-
mance during exercises and objectively evaluate the learning curve. ence
Mauer, B., Stackpole, B., Johnson, D., 2012. Developing small team-based cyber secu-
Our exercise competence planner can be tuned according to the rity exercises. In: Proceeding of the 2012 International Conference on Security
team composition types. It could be extended with assessment cri- and Management (SAM’12), pp. 213–217. Las Vegas, NV, USA
teria for each trained competence to follow the learning progress Mirkovic, J., Dark, M., Du, W., Vigna, G., Denning, T., 2015. Evaluating cybersecu-
rity education interventions: three case studies. IEEE Secur. Priv. 13 (3), 63–69.
of an individual or a team. doi:10.1109/MSP.2015.57.
Morgan S. Top 5 cybersecurity facts, figures, predictions, and statistics for 2019 to
Declaration of Competing Interest 2021. Cybersecurity Ventures, Cybercrime magazine; 2019.
National CCDC. Collegiate cyber defense competition. 2019. http://www.
nationalccdc.org/index.php/competition/competitors/rules.
The authors declare that they have no known competing finan- Newhouse, W., Keith, S., Scribner, B., Witte, G., 2017. National initiative for cyber-
cial interests or personal relationships that could have appeared to security education (NICE) cybersecurity workforce framework. NIST Spec. Publ.
800-181, 144. doi:10.6028/NIST.SP.800-181.
influence the work reported in this paper.
Ogee, A., Gavrila, R., Trimintzios, P., Stavropoulos, V., Zacharis, A., 2015. The 2015
Report on National and International Cyber Security Exercises. Technical Report.
Acknowledgements European Network and Information Security Agency doi:10.2824/627469. ISBN:
978-92-9204-158-8
Ohta, T., Takenaka, M., Katou, M., Masuoka, R., Kayama, K., Fukushima, N., Imai, H.,
The authors of the paper would like to express their grati- 2018. Cybersecurity solutions for major international events. Fujitsu Sci. Tech. J.
tude to the organisers and participants of the international cy- 54 (4), 57–65.
 A. Brilingaitė, L. Bukauskas and A. Juozapavičius / Computers & Security 88 (2020) 101607 13

Parrish, A., Impagliazzo, J., Raj, R.K., Santos, H.M.D., Asghar, M.R., Jøsang, A., Vykopal, J., Vizváry, M., Oslejsek, R., Celeda, P., Tovarnák, D., 2017. Lessons learned
Pereira, T., Stavrou, E., 2018. Global perspectives on cybersecurity education for from complex hands-on defence exercises in a cyber range. In: 2017 IEEE Fron-
2030: a case for a meta-discipline. In: Rößling, G., Scharlau, B. (Eds.), Proceed- tiers in Education Conference, FIE 2017, Indianapolis, IN, USA, October 18–21,
ings Companion of the 23rd Annual ACM Conference on Innovation and Tech- 2017. IEEE Computer Society, pp. 1–8. doi:10.1109/FIE.2017.8190713.
nology in Computer Science Education, ITiCSE 2018, Larnaca, Cyprus, July 02–04, Wei, W., Mann, A., Sha, K., Yang, T.A., 2016. Design and implementation of a multi-
2018. ACM, pp. 36–54. doi:10.1145/3293881.3295778. facet hierarchical cybersecurity education framework. In: Proceedings of IEEE
Paulsen, C., McDuffie, E., Newhouse, W., Toth, P., 2012. NICE: Creating a cyber- Conference on Intelligence and Security Informatics (ISI). IEEE, pp. 273–278.
security workforce and aware public. IEEE Security & Privacy 10 (3), 76–79. doi:10.1109/ISI.2016.7745488.
doi:10.1109/MSP.2012.73. White, G.B., Dietrich, G.B., Goles, T., 2004. Cyber security exercises: testing an or-
Rajivan, P., Cooke, N.J., 2017. Impact of team collaboration on cybersecurity situa- ganizations ability to prevent, detect, and respond to cyber security events. In:
tional awareness. In: Liu, P., Jajodia, S., Wang, C. (Eds.), Theory and Models for Proceedings of the 37th Hawaii International Conference on System Sciences
Cyber Situation Awareness, volume 10030. Springer, pp. 203–226. doi:10.1007/ (HICSS-37 2004), CD-ROM / Abstracts Proceedings, 5–8 January 2004, Big Island,
978- 3- 319- 61152- 5_8. Lecture Notes in Computer Science HI, USA. IEEE Computer Society, pp. 1–10. doi:10.1109/HICSS.2004.1265411.
Schepens, W., James, J.R., 2003. Architecture of a cyber defense competition. In:
Proceedings of IEEE International Conference on Systems, Man and Cybernetics. Agnė Brilingaitė. Brilingaitė holds a PhD in computer science from Aalborg Univer-
Conference Theme - System Security and Assurance, volume 5, pp. 4300–4305. sity, Denmark. She is an associate professor at Vilnius University in the Institute of
doi:10.1109/ICSMC.2003.1245660. Computer Science. Her research interests focus on spatial data modelling, location-
Seker, E., Ozbenli, H.H., 2018. The concept of cyber defence exercises (CDX): plan- based services, cybersecurity training, and education in computer science. She is in-
ning, execution, evaluation. In: 2018 International Conference on Cyber Secu- volved in the process of quality assurance in studies at the university. She has been
rity and Protection of Digital Services, Cyber Security 2018, Glasgow, Scotland, taking part in EU-funded projects related to the development of student-centred
United Kingdom, June 11–12, 2018. IEEE, pp. 1–9. doi:10.1109/CyberSecPODS. learning, teaching, assessment, and internationalisation.
2018.8560673.
Steinke, J., Bolunmez, B., Fletcher, L., Wang, V., Tomassetti, A.J., Repchick, K.M., Za-
ccaro, S.J., Dalal, R.S., Tetrick, L.E., 2015. Improving cybersecurity incident re- Linas Bukauskas. Bukauskas holds a PhD in computer science from Aalborg Univer-
sponse team effectiveness using teams-based research. IEEE Secur. Privacy 13 sity, Denmark. He is an associate professor and head of Cybersecurity Laboratory in
(4), 20–29. doi:10.1109/MSP.2015.71. the Institute of Computer Science at Vilnius University. He was one of the organ-
Svábenský, V., Vykopal, J., Cermák, M., Lastovicka, M., 2018. Enhancing cybersecurity isers of National Cybersecurity Training “Cyber Shield” and “Amber Mist” (2016–
skills by creating serious games. In: Polycarpou, I., Read, J.C., Andreou, P., Ar- 2018). His research interests include Cybersecurity, Data Mining, and Natural Lan-
moni, M. (Eds.), Proceedings of the 23rd Annual ACM Conference on Innovation guage Processing.
and Technology in Computer Science Education, ITiCSE 2018, Larnaca, Cyprus,
July 02–04, 2018. ACM, pp. 194–199. doi:10.1145/3197091.3197123. Aušrius Juozapavičius. Juozapavičius holds a PhD in theoretical physics from KTH
Tobey, D.H., Pusey, P., Burley, D.L., 2014. Engaging learners in cybersecurity careers: Royal Institute of Technology, Sweden. He is a professor and the head of the De-
lessons from the launch of the national cyber league. ACM Inroads 5 (1), 53–56. partment of Defence Technologies at General Jonas Žemaitis Military Academy of
doi:10.1145/2568195.2568213. Lithuania. His research interests are cybersecurity and computer modelling and op-
Vykopal, J., Oslejsek, R., Burská, K., Zákopcanová, K., 2018. Timely feedback in un- timisation of various systems, including semiconductor antennas and road traffic.
structured cybersecurity exercises. In: Barnes, T., Garcia, D.D., Hawthorne, E.K., He participates in EU-funded cybersecurity-related projects, and he is responsible
Pérez-Quiñones, M.A. (Eds.), Proceedings of the 49th ACM Technical Symposium for the cybersecurity specialisation of the study programs at the Military Academy.
on Computer Science Education, SIGCSE 2018, Baltimore, MD, USA, February 21–
24, 2018. ACM, pp. 173–178. doi:10.1145/3159450.3159561.