1000048042
1000048042
1000048042
SEPTEMBER 2021
GP
JR
21-023
ABBREVIATIONS
i
TABLE OF CONTENTS
1. Summary ......................................................................................................................................... 1
2. Implementation method and progress of the work .......................................................................... 3
2.1 Policies for achieving the target............................................................................................ 3
2.2 Contents of the work and implementation steps (plan and actual) ....................................... 4
2.3 Overall work schedule and the result .................................................................................... 7
2.4 Experts .................................................................................................................................. 7
3. Results of the work.......................................................................................................................... 8
3.1 Preliminary surveys .............................................................................................................. 8
3.2 Making course materials ....................................................................................................... 8
3.3 Performing TTT .................................................................................................................. 14
3.4 Evaluation of TTT participants ........................................................................................... 15
3.5 Evaluation of course materials and experts ........................................................................ 18
4. Suggestions ................................................................................................................................... 34
5. Conclusion .................................................................................................................................... 34
Appendix
Appendix A Photo................................................................................................................ A-1
Appendix B Overall work schedule (Plan and Actual) ........................................................ A-3
Appendix C Results of preliminary survey (Supply chain) .................................................... A-5
Appendix D Results of preliminary survey (Forensic) ........................................................ A-17
Appendix E Rating score sheet for trial lesson .................................................................... A-23
ii
LIST OF FIGURES AND TABLES
< Figures >
Figure 1 Structure of Supply Chain course, Supply Chain course text ....................................... 9
Figure 2 Virtual computing / network environment for IoC creation ....................................... 12
iii
1. SUMMARY
The “Project for Human Resources Development for Cyber Security Professionals” was started in May
2019 as a five-year project. The objective of the Project is to establish the cybersecurity education
system at Universitas Indonesia (University of Indonesia, hereafter referred to as UI). As part of this
Project activity, we have been working to develop two cybersecurity professional courses named “Case
Study & Practice: Supply chain cyber risk” (hereafter referred to as Supply Chain course) and “Case
Study & Practice: How to make IT systems forensic-enabled” (hereafter referred to as Forensic course).
1. Supposed participants
The courses target full-time lecturers and guest lecturers at UI. Also, the targets are assumed to
be senior lecturers who can communicate in English and have experience of teaching IT-related
subjects at the university.
2. Target course trainees
Senior IT engineers (with 3-5 years of experience) belonging to government, financial
institutions, power companies and other critical infrastructure operators
3. Other important points
(1) The courses will be part of future master courses in cybersecurity for working adults.
(2) It is planned to publicly disclose the courses as open courseware.
(3) It will be essential to subcontract assistance for the site surveys, course development and
technology transfer to local consultants.
(4) Trial lessons having the persons targeted for technology transfer as lecturers will be
implemented.
(5) Evaluation of the ability of the persons targeted for technology transfer will be
implemented after the technology transfer.
Table 2 Summary of requirements for “Case Study & Practice: Supply Chain cyber risk”
1. Course outline
The course should include the following contents:
・ Examples of incidents occurring in the supply chain
・ Standards and technologies (e.g. secure coding) that need to be known for mitigating supply
chain cyber risk
・ Sample contract documents for procuring IT devices and services
2. Goal for attainment after taking the course
The trainees will understand supply chain cyber risk and be able to take countermeasures in
their respective organizations.
3. Number of hours in the course
14 hours (7 hours x 2 days)
However, in the case of remote lectures, it will be 3.5 hours x 4 days, considering the limits of
sustained concentration of trainees.
4. Important points to consider
(1) Since this will be a stand-alone course having no other associated courses, it shall be
designed to provide broad coverage allowing the trainees to take a general view of supply
chain cyber risk.
– 1 –
(2) Primarily classroom learning is anticipated, however, it shall be designed as a practical
course that includes case studies (e.g. examples of disputes between customers and suppliers
due to contractual issues) and practical exercises (e.g. how to state information security
requirements in contract documents).
Table 3 Summary of requirements for “Case Study & Practice: Forensic enablement”
The target of the work is to make the course materials and to perform “Train the Trainers” (hereafter
referred to as TTT) so that the counterparts have capability to teach these courses in the university.
The work started from October 2020 and ended in August 2021, achieving the target.
Following sections describe the detail of the activities.
1
Contents equivalent to the Legal Rules of Evidence and Court Procedure defined as K0156 in NIST.SP800-
181 (National Institute of Standards and Technology)
2
CFHI: EC-Council Computer Hacking Forensic Investigator
3
ECIH: EC Council Certified Incident Handler
– 2 –
2. IMPLEMENTATION METHOD AND PROGRESS OF THE WORK
2.1 POLICIES FOR ACHIEVING THE TARGET
At the beginning of the work, we set the following policies to ensure the development of the desired
short-term course.
Considering that the intended trainees are not students but rather cybersecurity professionals who
work in corporations and government agencies, the course contents will be designed to leverage
the experience and knowledge of the trainees. Specifically, the ratio of classroom learning will be
reduced while the ratio of case studies and practical exercises will be increased to ensure that the
trainees are compelled to make full use of their own knowledge and experience. Doing so will
enable the trainees to gain authentic experiences in real workplace environments and acquire the
practical skills required in the “Goals for attainment after taking the course”.
Policy 2: Experts
The following three experts will be assigned in consideration of the workload and aptitude.
Expert 1: Work chief / Course development (also in charge of Supply chain course)
This expert has experience of CSIRT work and handling incidents in real work situations. He
also has experience of not only forensic but also designing and installing Cyber Range and
developing and implementing Cyber Range practical exercises.
This expert has experience of system development, operation and maintenance and is
endowed with sufficient knowledge and experience concerning network, server and PC
management and settings.
– 3 –
Policy 3: Utilization of local consultants
It will be essential to subcontract work to local consultants in the Project. Specifically, a contract
will be signed with a local cybersecurity company to consign assistance for the surveys, course
development and technology transfer necessary for implementing the work. Considering that
Japanese experts cannot travel to Indonesia due to the impact of COVID-19, it is possible that these
local consultants will act as classroom facilitators in remote lessons, so it will be necessary to
recruit human resources who are endowed with a certain degree of skills in the specialist fields.
The contents to be consigned to the subcontracted local consultants are summarized below.
2.2 CONTENTS OF THE WORK AND IMPLEMENTATION STEPS (PLAN AND ACTUAL)
The next table shows the planned contents of the work and implementation steps. The actual results are
indicated with a right arrow symbol () followed by highlighted result (Yellow=Done, Grey=Not
done). Note that the term “Counterpart” is abbreviated as “C/P” in the table.
– 4 –
Division Work Implementation Contents and Methods
・ Confirm the quality of the network necessary for remote lessons.
Not done because no gathering session was planned due
to COVID-19
Preparation of course ・ Prepare the following course materials (all English language)
materials (supply chain for the 2 courses:
and forensic) - Course concept (Removed because not specified in TOR)
- Syllabus
- Texts (text for trainees and text for teachers)
The texts for teachers should state the number of hours
and important points to consider for each topic).
- Auxiliary teaching materials (e.g. slides)
Done by 29 Jan 2021
・ Prepare questionnaires for evaluating ability before and
after the technology transfer.
Done by 29 Jan 2021
Recruitment of the local ・ Select the local consultants and sign the contract.
consultants and consignment Done by 28 Dec. 2020
of the start of work ・ Consign survey related to supply chain and forensic.
Done on 28 Dec. 2020
・ Obtain the findings of the supply chain survey.
Survey for supply chain cyber risk was conducted from
4 Jan 2021 until 31 Mar 2021.
First TTT Explanation of course ・ Explain the course materials to the C/Ps and the Project side.
(Supply Chain materials to the C/Ps, and Done on 3 Feb 2021
course) evaluation of the C/Ps’ ・ Have the C/Ps fill out the ability evaluation questionnaire.
ability Done on 8 Feb 2021
・ Evaluate the ability of the C/Ps.
Done on 8 Feb 2021
Implementation of TTT ・ Using the course materials, implement technology transfer
in the form of lessons with the C/Ps.
Done from 9 Feb to 11 Feb 2021
Implementation of trial ・ Have the C/Ps implement trial lessons (partial)
lessons and guidance If possible, implement the trial lessons upon inviting the
actual corporate cybersecurity staff targeted for the training.
Done on 12 and 15 Feb 2021
・ Appropriately offer guidance on the implementation methods.
Done on 12 and 15 Feb 2021
Post-technology transfer ・ Have the C/Ps fill out the ability evaluation questionnaire.
ability evaluation Done on 12 and 15 Feb 2021
・ Evaluate the ability of the C/Ps.
Done on 15 and 16 Feb 2021
Discussions about ・ In light of the technology transfer results, discuss making
correcting the course corrections to the course materials with the C/Ps and reach
materials conclusions.
Done on 16 Feb 2021
Meetings with the Project ・ In light of the technology transfer results, exchange
opinions on the future approach to work.
Done on 16 Feb 2021
Second Correction and revision of ・ Based on the results of discussing making corrections to the
preparation the course materials course materials for the Supply chain course, correct and
work in Japan (Supply Chain course) revise the materials.
1st: Done from 17 Feb to 22 Feb 2021
2nd: Done from 5 Jul to 11 Aug 2021
・ Share the results with the C/Ps and the Project side via TV
conference, etc.
Done on 13 Aug 2021
– 5 –
Division Work Implementation Contents and Methods
Acquisition of survey findings ・ Obtain the survey findings concerning forensic.
from the local consultants Survey was conducted from 15 Apr 2021 until 29 Jun
(Forensic) 2021
Preparation of the course ・ Prepare the following course materials (all English language)
materials for the Forensic course:
(Forensic course) - Course concept (Removed because not specified in TOR)
- Syllabus
- Texts (text for trainees and text for teachers)
The texts for teachers should state the number of hours
and important points to consider for each topic).
- Auxiliary teaching materials (e.g. slides)
Done by 09 Jul 2021
・ Prepare questionnaires for evaluating ability before and
after the technology transfer.
Done by 09 Jul 2021
Implementation of trial ・ Conduct remote trial lessons to deepen the understanding of
lessons for the local local consultants who undertake local lecture support.
consultants (forensic) Briefing of the contents: Done on 12 Jul 2021
・ After the trial lessons, reflect any bugs or improvements
points in the course materials.
Not done because no suggestion was given
Second TTT Explanation of course ・ Explain the course materials to the C/Ps and the Project side.
(Forensic materials to the C/Ps, and Done by 21 Jul 2021
course) evaluation of the C/Ps’ ・ Have the C/Ps fill out the ability evaluation questionnaire.
ability Done on 26 Jul 2021
・ Evaluate the ability of the C/Ps.
Done on 26 Jul 2021
Meeting with the local ・ Hold discussions with the local consultants concerning the
consultants work implementation.
Done on 12 Jul 2021
Implementation of TTT ・ Using the course materials, implement technology transfer
in the form of lessons with the C/Ps.
Done on 26, 28, 29 Jul and 02, 04, 05, 06 Aug 2021
Implementation of trial ・ Have the C/Ps implement trial lessons (partial)
lessons and guidance If possible, implement the trial lessons upon inviting the
actual corporate cybersecurity staff targeted for the training.
Done on 10 and 12 Aug 2021
・ Appropriately offer guidance on the implementation methods.
Done on 10 and 12 Aug 2021
Post-technology transfer ・ Have the C/Ps fill out the ability evaluation questionnaire.
ability evaluation ・ Evaluate the ability of the C/Ps.
Done on 10 and 12 Aug 2021
Discussions about ・ In light of the technology transfer results, discuss making
correcting the course corrections to the course materials with the C/Ps and reach
materials conclusions.
Done on 12 Aug 2021
Meetings with the Project ・ In light of the technology transfer results, exchange
opinions on the future approach to work.
Done on 10 and 12 Aug 2021
Wrap-up work Finalization of the course ・ If the C/Ps and the Project side have any opinions for
in Japan materials improving the course materials, reflect them and finalize the
course materials.
Done on 13 and 16 Aug 2021
・ Share the results with the C/Ps and the Project side via TV
conference, etc.
Done on 24 Aug 2021
Preparation of the work ・ Prepare the work completion report.
completion report, and ・ Report to JICA headquarters.
reporting
– 6 –
2.3 OVERALL WORK SCHEDULE AND THE RESULT
The overall work schedule is attached as Appendix B. It shows both plan and actual results.
The initial plan included two field works for TTT implementation, but due to the unpredictable COVID-
19 situation, discussions with the Project staff and C/P was conducted at an early stage of this work, it
was decided that all operations would be conducted in Japan. In this case, the TTT will be conducted
online remotely, but since the Forensic course TTT is focused on practical exercises it was decided to
conduct it in a group remote style that means participants gather in a physical classroom at UI. The
timing of the TTT was postponed to June or later in consideration of the UI semester break. On the other
hand, the Supply Chain course does not have any practical exercise, so it was held in February during
the lockdown as originally planned, with participants participating remotely from their homes.
However, in June, because the situation of COVID-19 was not improved, it was judged that the gathering
session is impossible, so the TTT of the Forensic course was also conducted remotely by letting
participants join from their homes from 26th of July to 12th of August. The presence of local consultants
was helpful in this implementation. We asked them to prepare USB memory sticks with copying the
huge data for exercises and send them to the participants' homes, and also, they provided detailed follow-
up services in Indonesian during the TTT. As a result, we were able to complete the Forensic course
TTT without any trouble.
2.4 EXPERTS
– 7 –
3. RESULTS OF THE WORK
3.1 PRELIMINARY SURVEYS
Preliminary surveys were conducted for both the Supply Chain course and Forensic course. The results
are compiled in Appendix C and Appendix D accordingly. The purpose of the surveys was to know the
actual situation in Indonesia of each field (supply chain and forensic) and the results are introduced in
the course texts. If the course content needs to be adapted to the Indonesian situation, the text will need
to be modified. The next table summarizes the result of the surveys.
The courses was designed and implemented to fulfill the requirements described in Table 1, Table 2 and
Table 3. Below are indicated the points for making the course materials.
Although there is a lot of literature and guidelines on supply chain cyber risk management in the
world, the concept is relatively new and there is no standard that companies can adopt without
– 8 –
hesitation. Therefore, the following guidelines were set in the development of this course to ensure
consistency.
Clarify the relevance of referenced documents based on the standards, guidelines, and
frameworks published by NIST 4 in the United States, which can be said to be the global
standard for cyber security.
Introduce the history and latest trends in supply chain cyber risk management standards. This
makes it possible to ride the tide of the field.
Introducing supply chain information models that can be applied in recent years to the future,
which are necessary for discussing supply chain cyber risks.
NIST Referring
Cyber Security Framework
Derived Referring
Derived
NIST METI
CMMC
SP800-171 CPSF
4
NIST: National Institute of Standards and Technology
– 9 –
The next table lists the created course materials of Supply Chain course.
Every slide in the Power Point documents has notes for guiding the lecturer on how to explain the slide.
– 10 –
2. Cybersecurity risks in the supply chain
Supply chain
- What is supply chain?
- Characteristics and examples of supply chain in each industrial sector
- Cyber Physical Security Framework (CPSF) by METI Japan
Trend of cybersecurity incidents in the supply chain
- Global trend
- Situation in Indonesia
Exercise 1:
Identification of cybersecurity risks in the supply chain in each industrial
sector.
Techniques and examples of cyberattacks targeting the supply chain
3. NIST Cyber Security Framework (CSF) and SP800-171
Overview of standards, frameworks and guidelines regarding supply chain cybersecurity
NIST Cyber Security Framework 1.1
How to apply CSF to the organization?
Exercise 2:
Applying CSF to your organization.
Make profile for your organization.
Summary of SP800-171
[Day 2]
1. Cybersecurity Maturity Model Certification (CMMC)
Summary of CMMC
How to comply with CMMC
Exercise 3:
Discussion on implementing CMMC in your organization.
2. Contracts and cybersecurity risk management
Cybersecurity risk management in work outsourcing
Exercise 4:
Practice in preparing a work outsourcing contract document.
Cybersecurity risk management in procurement of products and services
Exercise 5:
Practice in preparing a specification document for ordering products (or services)
Consideration in contract negotiation (from both the acquirer’s and supplier’s point
of view)
3. Wrap-up
Scheme of Lecture 60 %, Hands-on training 40 %
Instructions (Hands-on training includes exercises and case studies)
Keywords Cybersecurity, Supply chain, Risk management, ISO 28000, NIST Cybersecurity
Framework, Contract, Subcontractor
Tools (software) N. A.
required for
hands-on training
Reference books ISO 28000 A Complete Guide - 2020 Edition [ISBN 0655916679]
Supply Chain Risk Management (Internal Audit and IT Audit) 1st Edition [ISBN 978-
1138197336]
NIST Cyber Security Framework
https://www.nist.gov/cyberframework/framework
NIST SP800 documents
https://csrc.nist.gov/publications/sp800
CMMC portal
https://www.acq.osd.mil/cmmc/
– 11 –
Forensic course
The Forensic course consists of 31 exercises including 6 scenario-based digital forensics practices.
The scenarios contain Website defacement, Unauthorized access, DDoS attack, Ransomware
attack and APT attack. The IoC (Indicator of Compromise = Evidence on devices that points out
to a security breach) was created for each scenario using virtual computing / network environment
shown in next diagram.
Information about the configuration of servers and network devices, as well as some log files and
dump files, can be given to the participants to analyze, making the exercise very realistic.
Every slide in the Power Point documents has note which guides the lecturer how to explain the
slide.
– 12 –
The next table is the course syllabus of Forensic course.
– 13 –
- CDIR-A (Data parser for CDIR-Collector)
- WinPrefetchView (Viewer for prefetch)
- Event Log Explorer (Viewer for Windows Event Log)
- Autoruns (Viewer for auto-starting programs)
- RegRipper (Registry investigation tool)
- Registry Explorer (Viewer for registry)
- The Volatility Framework (Memory dump analysis tool)
Reference books - Incident Response & Computer Forensics, McGraw-Hill Education, ISBN 978-
0071798686.
- Practical Packet Analysis, No Starch Press, ISBN 1593278020.
- Intelligence-Driven Incident Response, O’Reilly Media, ISBN 978-149134944
TTTs for 2 courses were performed in February 2021 for Supply Chain course and July to August 2021
for Forensic course. The Supply Chain course had an additional supplemental TTT on 13 August 2021
to explain modified content. The participants in TTT for the 2 courses are listed in following tables.
Table 11 List of participants (Supply Chain course TTT/ Forensic course TTT)
– 14 –
3.4 EVALUATION OF TTT PARTICIPANTS
Each participant’s ability as a teacher was measured using multiple factors such as attendance rate,
evaluation of questionnaires and evaluation of trial lesson. In the Forensic course, submitted worksheets,
which record the progress and result of exercises, are also be used for the evaluation. The following
sections describe the method of ability measurement for each course.
(1) Calculate the score from 0 to 5 according to the attendance result. [A]
(2) Calculate the score from 1 to 5 based on the answers in the questionnaire. [B]
i.e.) For the question “Are you confident to teach chapter 1?”, the score is assigned according to
the answer such as “Not confident”=1, “OK but need assistance”=2, “OK but need further
review”=3, “OK with little review”=4, “OK no problem”=5
(3) Rate the performance of trial lesson for each participant (0 - 5). The rating score sheet which contains
the rating criteria is attached as Appendix E [C]
(4) Calculate the overall score from 0 to 10 by compiling [A] [B] and [C] with giving weight. The formula
is as below.
The next table is the actual result of evaluation of Supply Chain course.
– 15 –
Table 12 Evaluation result of TTT participants (Supply chain)
Supply Chain Risk course eval < 1.0 < 7.5
Attendance Mock class Total
Questionnaire
No. Name score score score Mock class comments
(weight=2)
(weight=3) (weight=5) (10.0)
- He has very good presentation skill. He added some slides to complement
1 A 3.0 1.9 4.8 9.7 the difficult content.
- Excellent lecturer
- He totally changed the material, and presented different theory of incident
response.
2 B 3.0 1.6 3.1 7.7
- Should not deviate from the original purpose.
- But his effort to improve the quality can be evaluated.
- She just read the material.
3 C 3.0 1.5 3.3 7.8
- Need to review the contents
- He just read the material.
4 D 3.0 1.6 3.6 8.2
- Need review before teaching
- He tried to let student understand by explaining details for each item.
5 E 3.0 1.6 4.4 9.0
- Can be a good teacher.
- He just read the material.
6 F 3.0 0.7 3.6 7.3
- Need review before teaching
- He has very good presentation skill.
7 G 3.0 1.3 4.5 8.8
- Can be a good teacher. Students will like him.
- He just read the material and skipped few important items.
8 H 3.0 1.0 3.3 7.3
- Need support to teach
- He just read the material.
9 I 3.0 1.4 3.5 7.9
- Need to improve his teaching skill
- He just read the material.
10 J 3.0 1.2 3.5 7.7
- Need review before teaching
- He has very good presentation skill. He reviewed the contents very well.
11 K 3.0 1.5 4.6 9.1
- Can be a good teacher.
- She may need review of the material so that she can explain the content well.
12 L 3.0 1.0 3.5 7.5
- Need support to teach
- He has good presentation skill.
13 M 3.0 1.8 4.4 9.2
- Can be a good teacher.
- He has very good presentation skill. He prepared well for this mock class.
14 N 3.0 1.4 4.6 9.0
- Can be a good teacher. Students will like him.
Forensic course
(1) Calculate the score from 0 to 5 according to the attendance result. [A]
(2) Calculate the score from 1 to 5 based on the answers in the questionnaire. [B]
i.e.) For the question “Are you confident to teach chapter 1?”, the score is assigned according to
the answer such as “Not confident”=1, “OK but need assistance”=2, “OK but need further
review”=3, “OK with little review”=4, “OK no problem”=5
(3) Rate the performance of trial lesson for each participant (0 - 5). The rating score sheet which contains
the rating criteria is attached as Appendix E. [C]
(4) Evaluate the performance of exercise based on the worksheets submitted by participants (0 - 5). The
worksheet contains the record of the progress and result of participant’s exercise. [D]
(5) Calculate the overall score from 0 to 10 by compiling [A] [B] [C]and [D] with giving weight. The
formula is as below.
– 16 –
The next table is the actual result of evaluation of Forensic course.
– 17 –
3.5 EVALUATION OF COURSE MATERIALS AND EXPERTS
The design of the courses, course materials and experts who conducted the TTTs are evaluated by
participants using online questionnaire. The results are shown as follows.
0
Negative ( < 50%)
5
Not much ( = 50%)
[Question] Do you think the course goals can be achieved with this design? Please select the
respective answer for each goal.
9 Yes, I think so
– 18 –
There are 2 negative answers “No I don’t think so” in Goal 2. The reasons for the answers are
unknown because the respondents said “Why i chose the answer”. It might be a sinple mistake.
0
3
Not enough
Just nice
Too much
12
0
3
Not enough
Just nice
Too much
12
4
Poor
Not much as expected
Acceptable
Good
Very good
11
– 19 –
How was the lecturer's teaching quality and attitude?
4
Poor
Not much as expected
Acceptable
Good
Very good
11
Chapter 1 Introduction
0
2
Poor
Not much as expected
Accceptable
8
Good
5
Very good
Poor
6 Not much as expected
Accceptable
Good
9
Very good
– 20 –
Chapter 3-1 NIST Cyber Security Framework
Poor
6 Not much as expected
Accceptable
Good
9
Very good
Poor
Not much as expected
7 Accceptable
8
Good
Very good
Poor
6
Not much as expected
Accceptable
9 Good
Very good
0
2
4
Poor
Not much as expected
Accceptable
Good
Very good
– 21 –
The quality of each content is OK.
Chapter 1 Introduction
1 1
Not enough
Just nice
Too much
13
Not enough
Just nice
Too much
15
1 0
Not enough
Just nice
Too much
14
– 22 –
Chapter 3-2 NIST SP 800-171
Not enough
Just nice
Too much
15
Not enough
Just nice
Too much
13
1
3
Not enough
Just nice
Too much
11
– 23 –
[Question] Are you confident in teaching the topic?
Chapter 1 Introduction
0 1
Not confident
2
OK but need assistance
6
OK, no problem
6
0 1
Not confident
3
OK but need assistance
6
OK, no problem
5
0
2
Not confident
5
OK but need assistance
OK, no problem
4
– 24 –
Chapter 3-2 NIST SP 800-171
1
3 1 Not confident
OK, no problem
6
OK, no problem
6
1
3 Not confident
3
OK but need assistance
OK, no problem
5
One participant answered “not confident” on important topics (SP 800-171, CMMC and contracts).
This is considered to be a problem of the participants' comprehension. As for the topic
“Consideration for cybersecurity in contracts”, it seems relatively difficult because it contains a lot
of legal jargon.
– 25 –
Do you recommend to your subordinates, colleague or
students to take this course?
0 1
No
Maybe
Yes
14
It is good to be recommended.
Forensic course
0
Negative ( < 50%)
[Question] Do you think the course goals can be achieved with this design? Please select the
respective answer for each goal.
Yes, I think so
10
– 26 –
Able to explain how to design an IT infrastructure that
can record and collect logs for digital forensics.
0
2
Yes, I think so
0
2
Yes, I think so
3 3
Not enough
Just nice
Too much
– 27 –
How was the quality of the course materials?
3
Poor
Not much as expected
Acceptable
Good
Very good
Poor
Not much as expected
5 Acceptable
6 Good
Very good
Module 0 - Introduction
Poor
Not much as expected
5
Acceptable
6 Good
Very good
– 28 –
Module 1 - DFIR: Digital Forensics and Incident Response
3
Poor
Not much as expected
Acceptable
Good
Very good
3
Poor
Not much as expected
Acceptable
Good
Very good
8
3
Poor
Not much as expected
Acceptable
Good
Very good
8
– 29 –
Module 4 - Conclusions
Poor
4
Not much as expected
Acceptable
Good
7 Very good
Module 0 - Introduction
1 0
Not enough
Just nice
Too much
10
1 1
Not enough
Just nice
Too much
– 30 –
Module 2 - How to design secure IT infrastructure
1 0
Not enough
Just nice
Too much
10
1
2
Not enough
Just nice
Too much
Module 4 - Conclusions
1 0
Not enough
Just nice
Too much
10
– 31 –
[Question] Are you confident in teaching the topic?
Module 0 - Introduction
0 1
Not confident
3 1
OK but need assistance
OK, no problem
6
0 1
Not confident
1
4 OK but need assistance
OK, no problem
5
1
2 Q21 Module 2 - How to design
secure IT infrastructure
OK but need assistance
OK, no problem
5
– 32 –
Module 3 - Scenario-based DFIR training
1 0 1
Not confident
OK, no problem
Module 4 - Conclusions
0 1
2 Not confident
1
OK but need assistance
OK, no problem
7
No
Maybe
Yes
11
It is good to be recommended.
– 33 –
4. SUGGESTIONS
(1) Since the course materials contain a certain amount of information on today's state and trends of
cybersecurity, it is necessary to constantly update such information. It is advised to review those
parts at least once a year and keep the contents of the course materials up to date.
(2) It is recommended to consider developing another practical training course such as “How to build
Cyber Range for cyber-attack and defense exercises”. Because having and operating a Cyber Range
will be essential for future Cybersecurity organizations. For the UI, Cyber Range will also be
needed to update the exercises in this Forensic course.
(3) The course materials are not specific to Indonesia except few parts (i.e., Summary of Supply Chain
Survey) and can be used in other countries. For this reason, it is recommended to use it for similar
educational purposes in other countries.
(4) When planning similar TTT in the future, it will be necessary to take care that it is not performed
in the semester. Otherwise, sufficient attendance of the counterparts cannot be expected.
(5) It is not clear whether this is a problem peculiar to Indonesia, but it seems necessary to prepare
reward to increase the response rate and quality level of the questionnaire. This is a piece of advice
from one of the counterparts and it would be useful.
5. CONCLUSION
We have successfully completed making the materials and performed TTT for the cybersecurity courses
“Case Study & Practice: Supply chain cybersecurity risks” and “Case Study & Practice: How to make
IT systems forensic-enabled”. We hope that these achievements will contribute to the cybersecurity
human resource development in Indonesia, which is the major purpose of the Project.
– 34 –
APPENDIX
APPENDIX A PHOTO
TTT for Supply Chain course (from 9 Feb to 11 Feb 2021)
TTT for Forensic course (Done on 26, 28, 29 Jul and 02, 04, 05, 06 Aug 2021)
A-1
APPENDIX B OVERALL WORK SCHEDULE (PLAN AND ACTUAL)
November 2020 – March 2021
Plan : □=Execution △=Completion (i.e. Submit the documents) Result: ■=Execution ▲=Completion
November 2020 December 2020 January 2021 February 2021 March 2021
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Japan/ S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W
No Action item u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e
On-site
n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d
1 Japan ◆Common work
(1)Make Work plan (Japanese) □ □ □□□□□□□□□△
2 Japan
■ ■ ■
(2)Make Work plan (English) □ □ □□□□□□□□□△
3 Japan
■ ■ ▲
4 Japan (3)Explain Work plan to JICA HQ △
(4)Engage contract with local consultant □□ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ △
5 Japan
■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■■■■■ ▲
(4)Procure equipment and books □□□□□□□□□ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □□□ □ □ □ □ □□□□□□□△
6 Japan
■■■■■■■■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■■■ ■ ■ ■ ■ ▲
7 Japan ◆Develop Supply chain risk course
(1)Collect information and conduct research □□□□□□□□□□□□□□□□□□□□□□□□□△
8 Japan
■■■■■■■■■■■■■■■■■■■■■■■■■▲
(2)Survey conducted by local consultant □□□□□□□□□□□□□□□□ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ △
9 Japan
■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■▲
(3)Make course materials □□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □ □□□□□□□△
10 Japan
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■■■■■■■▲
11 Japan (4)Material review with local consultant □□□□△
12 Japan ◆1st TTT (Individual remote lecture)
(1)Explain plan & course contents to C/P △
13 Japan
and project staff ▲
(2)Evaluate C/P's capacity (Pre) □
14 Japan
■
(3)Conduct TTT (3.5 hours/day) □□□□
15 Japan
■■■
(4)Perform trial lesson (partial) □
16 Japan
■ ■
(5)Evaluate C/P's capacity (Post) □
17 Japan
■■
(6)Discuss for material correction with C/Ps □
18 Japan
■
19 Japan (7)Meeting with project staff □
20 Japan ◆Modify Supply chain risk course materials
(1)Do modification □□□□□□
21 Japan
■■■■■■
(2)Share and approval □□△
22 Japan
23 Japan ◆Develop Forensics exercise course materials
(1)Collect information and conduct research □□□□□□□□□□□□□□□□□□□□□□□□□△
24 Japan
■■■■■■■■■■■■■■■■■■■■■■■■■▲
(2)Survey conducted by local consultant □□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□△
25 Japan
(3)Make course materials □□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□△
26 Japan
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
(4)Material review with local consultant
27 Japan
28 Japan ◆2nd TTT (Group remote lecture)
(1)Explain plan & course contents to C/P
29 Japan
and project staff
30 Japan (2)Evaluate C/P's capacity (Pre)
(3)Conduct TTT (7 hours/day)
31 Japan
(4)Perform trial lesson (partial)
32 Japan
(5)Evaluate C/P's capacity (Post)
33 Japan
(6)Discuss for material correction with C/Ps
34 Japan
35 Japan (7)Meeting with project staff
36 Japan ◆Wrap-up work
(1)Finalize course materials
37 Japan
(2)Share and approval
38 Japan
(3)Make work completion report
39 Japan
(4)Explain the result & conclusion to JICA
40 Japan
HQ
A-3
June 2021 – September 2021
Japan/ T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T
No Action item u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h r a u o u e h
On-site
e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u i t n n e d u
1 Japan ◆Common work
(1)Make Work plan (Japanese)
2 Japan
(2)Make Work plan (English)
3 Japan
4 Japan (3)Explain Work plan to JICA HQ
(4)Engage contract with local consultant
5 Japan
(4)Procure equipment and books
6 Japan
A-4
APPENDIX C RESULTS OF PRELIMINARY SURVEY (SUPPLY CHAIN)
Q1 First Name, Last Name, Company / Organization, Company Address, City, Zip Code,
Country, State, Phone, Email
<This response result is not disclosed because the responses include privacy information.>
General employee
Other (no title)
37% 17%
Director / Executive
Class
Profession
3%
Section
Section chief class manager class
3% 13%
Department Business
manager class manager class
7% 7%
Other
23%
Procurement
department
0%
Information system
Business unit department or
13% Information security
department
64%
A-5
Q4 What industry is your company categorized to?
Other
19%
Information
Technology
Gas 32%
3%
Construction
3%
Government or
Public Service
5%
Telecommunication
5% Manufacturing
14%
Consulting
8% Education
11%
Q5 Please select the total number of employees at your company (including full-time and part-
time employees).
Over 50,000
5,001 – 10,000 3%
10%
0 – 50
27%
1,001 – 5,000
7%
501 – 1,000
13%
301 – 500
3% 51 – 100
20%
101 – 300
17%
A-6
Q7 What kind of Company / Organization that you are working on, in the IT Supply Chain above ?
Entruster
(User Company)
Contractor
(Prime vendor)
Subcontractor
(Sub-vendor)
Sub-subcontractor
(Sub-sub-vendor)
Sub-subcontractor
(Sub-sub-vendor)
Subcontractor
0%
(Sub-vendor)
10%
Contractor (Prime
Vendor)
32%
Entruster (End
User Company)
58%
0.0% No outsourcing
Response percent
A-7
Q9 What basis does your company decide whether or not to outsource information security ?
No internal rules,
but decisions are
made as needed. The decision is
31% made based on
internal rules.
61%
Q10 How concerned are you about the information security risks associated with outsourced assets?
100%
7.7% 7.7% 7.7% 7.7% 7.7% 7.7%
90% 7.7%
15.4%
80% 23.1% 23.1%
30.8%
70% 38.5%
60%
50%
40% 84.6%
76.9%
69.2% 69.2%
30% 61.5%
53.9%
20%
10%
0%
Internal fraud External attack Human error System failure, Disaster Information
(Virus, (Misoperation, etc.) interruption, Leakage
Cybersecurity, etc.) outage
Q11 What do you consider to be the issues in managing the information security of contractors?
Choose up to 3 ONLY
100%
Lack of awareness of information
90% security
A-8
Q12 Please select which Information Security control that you already have in place during the
process o f managing contractors.
0 2 4 6 8
Response count
Q13 What do you consider to be the most important information security issues that needs to be
described in the contracts with contractors? (Multiple choice up to 3)
80.0% It is not clear what to agree on as
69.2% 69.2% information security requirements
70.0%
A-9
Q14 What points do you place importance on when selecting contractors? Please choose the four
most important items in order of priority.
9.0 Quality, price, and delivery of work
8.0 7.7
Past orders placed with contractors and
evaluation of past work
7.0
6.1 Implementation status of information security
5.8
6.0 measures and acquired certification (ISMS, PCI
5.3 DSS, GDPR compliant, etc.)
Management and financial status of the
5.0 4.5 contractor
Score
2.9
3.0 Being an affiliated company / group company
2.1
2.0 1.6
Being a domestic company
1.0
Being a global company
0.0
Q15 From the perspective of information security, what are the key issues when selecting
outsourcing partners? (Multiple choice up to 3)
100% Lack of personnel with sufficient knowledge
92.3% and skills within the company
90%
A-10
Q16 What kind of information security requirements do you include in your contracts? (Select
all that apply)
100% 100.0% Confidentiality
0%
Handling of information assets after termination of
Response percent
contract (return, deletion, disposal, etc.)
Q17 Please indicate the implementation status of information security measures for each type of
information you handle. (Select all that apply)
0
Establishment of the incident response system 4
6
6
0
Response to confirmation of the implementation status of 4
information security measures by the entruster 5
7
0
Confirmation of contractual details regarding information 4
management and information security 7
7
1
Explanation of information security measures at the time of 3
proposal 6
6
0 1 2 3 4 5 6 7 8
Response count
Q18 Please feel free to describe anything you would like to say about information security in the
supply chain.
Cybersecurity is never just a technology problem, it’s a people, processes and knowledge problem.
Information security must be handled properly from the beginning until the end of the whole process
A-11
From Q19 to Q30 were responded by Contractor companies
Q19 Do you have sub-contractors or / and sub-sub contractors for your Company ?
0.0%
No subcontractor or sub-sub
16.7% 16.7% contractors
Q20 Please select all that apply to your Company current existing IT system services.
8
PMO (Project management office)
7
7
Requirement definition and design
6
Development, implementation and test
5
Response count
Q21 What basis does your company decide whether or not to outsource information security ?
4.6%
The decision is made based on
internal rules.
22.7%
No internal rules, but
decisions are made as needed.
72.7%
Not much concern about
information security.
A-12
Q22 How concerned are you about the information security risks associated with outsourced assets?
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Internal fraud External attack Human error System failure, Disaster Information
(Cyber Attack, interruption, Leakage
Unauthorized outage
Access, etc.)
Q23 What do you consider to be the most important information security issues in contracts with
outsourcers? (Multiple choice up to 3)
80% 76.0%
Many contractors have low awareness
70% 68.0% of ensuring information security.
60.0%
60% Lack of personnel with sufficient
knowledge and skills within the
50% company
Difficult to implement information
40% 36.0% security measures for various types and
levels of contract work
30% High burden on related departments for
information security measures in
20% contracted operations
Other
10%
0.0%
0%
Q24 What do you emphasize about your business proposals to the outsourcer? Please choose the
four most important items in order of priority
A-13
Q25 What kind of information security measures do you take to prevent internal fraud in your
contracted business? (Select all that apply)
20
Acquisition and storage of operation
18 logs, etc.
16
16 15 Controlling the carrying in and out of
14
personal mobile devices and storage
14
media.
12
Response count
2 Other
0
0
Q26 From the perspective of information security, what are the key issues when proposing to the
outsourcers? (Multiple choice up to 3)
60% 56.0% 56.0% 56.0%
The content of the proposal is not
checked internally from an information
50% security perspective.
44.0%
Due to the high level of information
40% security requirements of the outsourcer,
the cost burden is large.
Since information security requirements
30% differ depending on the contractor, the
24.0% burden of responding to each is high.
The outsourcer does not understand the
20%
cost of information security measures.
0%
A-14
Q27 What kind of information security requirements are included in the contracts with
entrusters? (Select all that apply)
100% Confidentiality
95.8%
70%
62.5% 62.5% Measures to be taken when violation of the terms of the
58.3% information security agreement occurs
60%
54.2% 54.2% Scope of responsibility of entruster and contractor for
information security
50% 45.8%
41.7% Responses when information security incident occurs
40%
Q28 Have you ever had an incident in the past three years of outsourced work in your company
or subcontractor? (Select one for each row)
100%
12.5% 16.7%
90%
80%
70%
60%
50% 79.2% 70.8%
40%
30%
20%
10%
8.3% 12.5%
0%
In My Company In subcontractors or sub-subcontractors
Yes No Unknown
A-15
Q29 Please answer if you chose “Yes” in above question. What kind of incident occurred?
90%
83.3%
80%
10%
0%
Q30 Please feel free to describe anything you would like to say about information security in the
supply chain.
IT Security Regulation
Information Security is a must since the begining day of information technology being implemented
Cybersecurity is never just a technology problem, it’s a people, processes and knowledge problem.
Very important due to data privacy
A-16
APPENDIX D RESULTS OF PRELIMINARY SURVEY (FORENSIC)
Q1 First Name, Last Name, Company / Organization, Company Address, City, Zip Code,
Country, State, Phone, Email
<This response result is not disclosed because the responses include privacy information.>
Other
Director / 20%
Executive Class General employee
0% (no title)
28%
Section manager
class
4%
Business manager
class
8% Profession
Department 20%
manager class
12%
Business unit
25%
Other
54% Information system
department or Information
security department
21%
Procurement
department
0%
A-17
Q4 What industry is your company categorized to?
Other
25% Information
Technology
32%
Internet Service
Provider
4%
Electricity
3%
Government or Education
Public Service 18%
7% Consulting
11%
Q5 Please select the total number of employees at your company (including full-time and part-
time employees).
Over 50,000
0 – 50
5,001 – 10,000 4%
12%
8%
501 – 1,000
4%
301 – 500
8%
101 – 300
38%
1 million USD – 5
Over 1 billion USD
million USD
9%
22%
A-18
Q7 Please select logging and monitoring status in your company. (Select one)
Q8 Please select the occurrence of security incidents (cyberattacks, malware infections, internal
fraud, etc.) in your company in the past. (Select one)
Not occurred;
constant monitoring
ensures it
35%
Q9 For those who selected “Occurred” (1)(2) in Previous Question, what kind of cyberattack
occurred? (Multiple choice)
10 Unauthorized access due to spoofing or hacking
9 account
Unauthorized access due to vulnerability exploit
8
Website defacement
7
Response count
6 DoS/DDoS attack
5
APT (Advanced Persistent Threats)
4
3 Malware infection, e.g., ransomware, trojan, worm
and another computer virus
2 Internal fraud, e.g., data theft and confidential
1 1
1 information leakage
0 0 0 0 0 0 Other
0
A-19
Q10 Has your company performed digital forensics in the past regardless of using internal or
external resources? And how often? (Select one)
Q11 For those who selected “Yes” in Q10 which forensics process did your company's employees
perform? (Multiple choice)
10
Collect the evidences, e.g., devices related to the
9 incident, log files
Q12 For those who selected “Yes” in Q10, which forensics process did your company's employees
perform? (Multiple choice)
10 Autopsy
9 Autoruns (Microsoft)
AXIOM/IEF (Magnet Forensics)
8 BrowsingHistoryView (Nirsoft)
7 FTK (AccessData)
FTK Imager/FTK Imager Lite (AccessData)
Response count
6 log2timeline/plaso
5 Registry Viewer (AccessData)
5
4 RegRipper
4 The Sleuth Kit
3 3 3 3 3 The Volatility Framework
3
2 2 2 2 2 2 2 2 Winpmem
2 WinPrefetchView (Nirsoft)
1 Wireshark
1
X-Ways Forensics (X-Ways Software Technology AG)
0 Other
A-20
Q13 How do you train employees to perform digital forensics? (Multiple choice)
14
Get them into professional training courses
12 11
10 Get them into on-the-job training
10 9
Response count
6
Other
4
2
2
Q14 Please select the security challenges your company faces. (Multiple choice)
14
14 Insufficient knowledge and experience in
security
2 1
Q15 What do you think of the necessity of digital forensics ? (Multiple choice)
17
It must be done within own organization to
16 prevent the disclosure of corporate sensitive
information.
14
It is too costly to have forensics function in the
12 organization because it requires additional head
count, lab, tools, and equipment
Response count
10
The necessity of forensics function is
understandable, but the managements do not
8
understand it.
2 1 Other
A-21
Q16 Please feel free to describe anything you would like to say about digital forensics.
As a digital forensics expert, I see that digital forensics become more important and urgent to continuously
develop not only to investigate the security incident, but also to fight against any computer/technology-based
crimes and any fraud occurring in any organization. Please contact me for further discussion ->
izazi.mubarok@afdi.or.id
Digital Forensic must be learned and developed constantly following the development of Information
Technology. Never stop to explore digital forensic in various digital evidence. There are 4 pillars to strengthen
Digital Forensic, namely:
1. Qualified Examiners, according to ISO/IEC 27035, 27037, 27042
2. Reliable Tools, according to NIST, Interpol Digital Forensic Experts Group, ISO/IEC 27037, 27042
3. Validated Methods and Standards, according to ISO/IEC 27035, 27037, 27042
4. Accredited Laboratory, according to ISO/IEC 17025
Digital forensic is very expensive but important to implement in every organization with centralized
monitored regularly by advanced specialists team in security.
Hal ini penting namun, masih butuh banyak system & sdm yg perlu di perbaiki & di latih
It's becoming more and more important especially in nowadays since everything is connected in digital
information world
It is very important to look for digital traces that can indeed be done to look for errors or fraud in a company
I'm not understand about digital forensics
System and data are company assets that need to be manage professionally
Very necessary
Company need digital forensic to investigate employee violation, ethic violation, corruption and other crime
done.
Important like insurance, to make sure everything has tracking
A-22
APPENDIX E RATING SCORE SHEET FOR TRIAL LESSON
Category No. Evaluation point Score (*1)
Are there any deficiencies in essential basic knowledge such
A Basic knowledge of the field 1
as operating systems and networks?
Are there any ambiguous explanations of the content that
2
may indicate a lack of understanding?
3 Are there any incorrect explanations?
Are the purposes and cautions explained in the explanation of
4
Understanding of class contents tools and techniques?
B
and appropriateness of explanation 5 Are the answers to questions appropriate?
6 Are the purpose and goal of the exercise explained?
Are the positioning of this course among the cybersecurity
7 courses and the learning path (what they need to learn before
and after) explained?
Sufficiency of course content Does the lecture cover all the content of the section?
C 8
(no omissions)
Does he / she try to improve students' understanding by
9
giving concrete examples?
Does he / she try to keep students' concentration by asking
10
D Teaching Techniques questions?
Is the time allocated for classes, lectures, and exercises
11
appropriate?
12 Is there any follow-up for students who do not understand well?
E Appropriateness of materials 13 Have the materials been deleted/changed/added?
A-23