Effective January 2024

Evidence-Based Training
Implementation Guide
Edition 2
 DISCLAIMER
The content, data and information (the “Content”) contained in this publication (“Publication”), is provided
for information purposes only and is made available to you on an “AS IS” and “AS AVAILABLE” basis.

IATA has used reasonable efforts to ensure the Content of this Publication is accurate and reliable. We,
however, do not warrant, validate, or express any opinions whatsoever as to the accuracy, genuineness,
origin, tracing, suitability, availability or reliability of the sources, completeness, or timeliness of such
Content. IATA makes no representations, warranties, or other assurances, express or implied, about
the accuracy, sufficiency, relevance, and validity of the Content. IATA’s observations are made on a best
efforts and non-binding basis, and shall not be deemed to replace, interpret, or amend, in whole or in part,
your own assessment and evaluation or independent expert advice. Nothing contained in this Publication
constitutes a recommendation, endorsement, opinion, or preference by IATA.

IATA has no obligation or responsibility for updating information previously furnished or for assuring that
the most up-to-date Content is furnished. IATA reserves the right to remove, add or change any Content at
any time. Links to third-party websites or information directories are offered as a courtesy. IATA expresses
no opinion on the content of the websites of third parties and does not accept any responsibility for third-
party information. Opinions expressed in advertisements appearing in this publication are the advertiser’s
opinions and do not necessarily reflect those of IATA. The mention of specific companies or products in
advertisements does not imply that they are endorsed or recommended by IATA in preference to others
of a similar nature which are not mentioned or advertised.

This Publication is not intended to serve as the sole and exclusive basis for assessment and decision
making and is only one of many means of information gathering at your disposal. You are informed to
make your own determination and make your own inquiries as you may deem necessary and suitable.
You shall independently and without solely relying on the information reported in this Publication, perform
your own analysis and evaluation regarding the nature and level of information you may require, based
upon such information, analyses, and expert advice as you may deem appropriate and sufficient, and
make your own determination and decisions pertaining to the subject matter under consideration.

This Publication is the property of IATA and is protected under copyright. The Content of this Publication is
either owned by or reproduced with consent or under license to IATA. This Publication and its Content are
made available to you by permission by IATA, and may not be copied, published, shared, disassembled,
reassembled, used in whole or in part, or quoted without the prior written consent of IATA. You shall not
without the prior written permission of IATA: re-sell or otherwise commercialize, make mass, automated
or systematic extractions from, or otherwise transfer to any other person or organization, any part of
this Publication and its Content in whole or in part; store any part of this Publication, or any Content, in
such a manner that enables such stored Content to be retrieved, manually, mechanically, electronically or
systematically by any subscriber, user or third-party; or include it within, or merge it with, or permit such
inclusion in or merge with, another archival or searchable system.

TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IATA DISCLAIMS ANY REPRESENTATION
OR WARRANTY (I) AS TO THE CONDITION, QUALITY, PERFORMANCE, SECURITY, NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE OF THIS PUBLICATION AND CONTENT;
OR (II) THAT THE ACCESS TO OR USE OF THIS PUBLICATION (INCLUDING ANY AUTOMATED FEEDS OR
OTHER DELIVERY MODES) OR ANY CONTENT SUPPLIED OR CONTRIBUTED TO THIS PUBLICATION
BY THIRD PARTIES, WILL BE UNINTERRUPTED, ACCURATE, THE MOST UP TO DATE, COMPLETE OR
ERROR-FREE. IATA EXCLUDES ALL LIABILITY (TO THE EXTENT PERMITTED BY APPLICABLE LAW) FOR
ANY COSTS, LOSSES, CLAIMS, DAMAGES, EXPENSES OR PROCEEDINGS OF WHATEVER NATURE
INCURRED OR SUFFERED BY YOU OR ANY OTHER PARTY ARISING DIRECTLY OR INDIRECTLY IN
CONNECTION WITH THE USE OF THIS PUBLICATION OR ANY CONTENT CONTAINED OR ACCESSED
THEREFROM, OR DUE TO ANY UNAVAILABILITY OF THIS PUBLICATION IN WHOLE OR IN PART.

Montreal - Geneva
Cataloguing in Publication data can be obtained from Library and Archives Canada.
© International Air Transport Association. All rights reserved.
 Table of Contents

Definitions������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������ iii

Acronyms��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������vi

Executive Letter����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������viii

Acknowledgements�����������������������������������������������������������������������������������������������������������������������������������������������������������������������������ix

Disclaimer��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������x

Publications���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������xi

Section 1 - Introduction����������������������������������������������������������������������������������������������������������������������������������������������������������������������1
1.1 Background����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������1
1.2 EBT paradigm shift��������������������������������������������������������������������������������������������������������������������������������������������������������������������1
1.3 Objective of this guide�������������������������������������������������������������������������������������������������������������������������������������������������������������1
1.4 Guide structure���������������������������������������������������������������������������������������������������������������������������������������������������������������������������2
1.5 ICAO provisions for EBT����������������������������������������������������������������������������������������������������������������������������������������������������������2

Section 2 - EBT Fundamentals���������������������������������������������������������������������������������������������������������������������������������������������������������3

2.1 Introduction���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������3
2.2 Program overview����������������������������������������������������������������������������������������������������������������������������������������������������������������������3
2.3 Phases within a module�����������������������������������������������������������������������������������������������������������������������������������������������������������3
2.4 Program content������������������������������������������������������������������������������������������������������������������������������������������������������������������������4
2.5 Pilot and instructor/evaluator competency frameworks��������������������������������������������������������������������������������������������6
2.6 Adapted Competency Model������������������������������������������������������������������������������������������������������������������������������������������������7
2.7 Relationship between CRM, TEM and Pilot/IE competency frameworks����������������������������������������������������������11
2.8 Competency assessment process and method��������������������������������������������������������������������������������������������������������12
2.9 Concept of resilience������������������������������������������������������������������������������������������������������������������������������������������������������������14
2.10 EBT IE standardization�������������������������������������������������������������������������������������������������������������������������������������������������������15
2.11 Data collection and analysis��������������������������������������������������������������������������������������������������������������������������������������������16
2.12 Feedback loop�����������������������������������������������������������������������������������������������������������������������������������������������������������������������20

Section 3 - Implementation Overview���������������������������������������������������������������������������������������������������������������������������������������21

3.1 Baseline EBT requirements�������������������������������������������������������������������������������������������������������������������������������������������������21
3.2 Staged implementation��������������������������������������������������������������������������������������������������������������������������������������������������������21
3.3 Enhanced EBT��������������������������������������������������������������������������������������������������������������������������������������������������������������������������26

Section 4 - Implementation Process�����������������������������������������������������������������������������������������������������������������������������������������28

Step 0: Information session(s) of the stakeholders�����������������������������������������������������������������������������������������������������������28

 i
 Evidence-Based Training Implementation Guide

Step 1: Implementation plan and safety risk assessment�����������������������������������������������������������������������������������������������29

Step 2: Pre-EBT training effectiveness review���������������������������������������������������������������������������������������������������������������������32
Step 3: Development of an adapted competency model and grading system�������������������������������������������������������33
Step 4: Communication plan�������������������������������������������������������������������������������������������������������������������������������������������������������35
Step 5: EBT IE initial standardization����������������������������������������������������������������������������������������������������������������������������������������36
Step 6: EBT IE’s competency assessment����������������������������������������������������������������������������������������������������������������������������38
Step 7: Equivalency of malfunctions����������������������������������������������������������������������������������������������������������������������������������������39
Step 8: Equivalency of approaches������������������������������������������������������������������������������������������������������������������������������������������42
Preamble to Step 9 and Step 10 – EBT program design general����������������������������������������������������������������������������������44
Step 9: Assessment and training topics distribution��������������������������������������������������������������������������������������������������������45
Step 10: Module design����������������������������������������������������������������������������������������������������������������������������������������������������������������46
Step 11: Implementation of the EBT program����������������������������������������������������������������������������������������������������������������������52
Step 12: Review of training effectiveness�����������������������������������������������������������������������������������������������������������������������������60
Step 13: Measurement of training system performance�������������������������������������������������������������������������������������������������61
Step 14: Continuous improvement of the training program�������������������������������������������������������������������������������������������64
Step 15: EBT IE recurrent standardization and ICAP���������������������������������������������������������������������������������������������������������66

Section 5 - IE Conduct of EBT�������������������������������������������������������������������������������������������������������������������������������������������������������68

5.1 Briefing����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������68
5.2 FSTD training����������������������������������������������������������������������������������������������������������������������������������������������������������������������������69
5.3 Observable Behavior (OB) collection������������������������������������������������������������������������������������������������������������������������������71
5.4 Debriefing����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������74
5.5 Competency Assessment and associated Grading��������������������������������������������������������������������������������������������������76
5.6 Instructional methods�����������������������������������������������������������������������������������������������������������������������������������������������������������81

Appendix A - Safety Risk Assessment (SRA) examples���������������������������������������������������������������������������������������������������86

Appendix B - Competencies and Observable Behaviors�������������������������������������������������������������������������������������������������97

Appendix C - Communication plan template���������������������������������������������������������������������������������������������������������������������� 102

Appendix D - EBT IE initial standardization guidance����������������������������������������������������������������������������������������������������� 105

Appendix E - Equivalency of Malfunction examples������������������������������������������������������������������������������������������������������� 111

Appendix F - Equivalency of Approaches example���������������������������������������������������������������������������������������������������������� 115

Appendix G - EBT program development guidance�������������������������������������������������������������������������������������������������������� 116

Appendix H - Training program design for pilots after long absence��������������������������������������������������������������������� 148

Appendix I - ICAP implementation guidance���������������������������������������������������������������������������������������������������������������������� 157

ii
 Definitions
Adapted competency model—A group of competencies with their associated description and perfor-
mance criteria adapted from an ICAO competency framework that an organization uses to develop com-
petency-based training and assessment for a given role.
Air operator certificate (AOC)—A certificate authorizing an operator to carry out specified commercial
air transport operations.
Approved training organization (ATO)—An organization approved by and operating under the supervi-
sion of a Contracting State in accordance with the requirements of Annex 1 to perform approved training.
Assessment—The determination by an instructor, assessor or evaluator as to whether a candidate meets
a required competency standard under given conditions, by collecting evidence from observable behav-
iors. Assessment takes place during instruction and evaluation.
ATA Chapters—A common industry referencing standard for aircraft technical documentation.
Competency—A dimension of human performance that is used to reliably predict successful perfor-
mance on the job. A competency is manifested and observed through behaviors that mobilize the relevant
knowledge, skills and attitudes to carry out activities or tasks under specified conditions.
Competency-based training and assessment—Training and assessment that are characterized by a
performance orientation, emphasis on standards of performance and their measurement, and the devel-
opment of training to the specified performance standards.
Competency standard—A level of performance that is defined as acceptable when assessing whether or
not competency has been achieved.
Conditions—Anything that may qualify a specific environment in which performance will be
demonstrated.
Critical flight maneuvers—Maneuvers that place significant demand on a proficient crew.
Critical system malfunctions—Aircraft system malfunctions that place significant demand on a profi-
cient crew. These malfunctions should be determined in isolation from any environmental or operational
context.
Evidence-based training (EBT)—Training and assessment that is characterized by developing and as-
sessing the overall capability of a trainee across a range of competencies rather than by measuring the
performance of individual events or maneuvers.
EBT instructor—A person, who has undergone a screening and selection process, successfully complet-
ed an approved course in delivering competency-based training and is subsequently authorized to con-
duct recurrent assessment and training within an approved EBT program.
Note: ICAO Doc 9995 and the European EBT regulations regularly use the term “EBT Instructor”, to
highlight the fact that under any baseline EBT program, the instructor should have the privileges to
conduct evaluations that includes the summative assessment of the pilot performance and the evaluation
of the training system.
For clarity and to be in line with the definitions of ICAO Doc 9868 (PANS-TRG), this guide consistently uses
the term “EBT Instructor/Evaluator” (EBT IE) to refer to a person authorized to provide training and conduct
evaluations during an EBT module.
EBT module—A session or combination of sessions in a qualified FSTD as part of the 3-year program of
recurrent assessment and training.
EBT scenario element—Part of an EBT session designed to address a specific training topic.
EBT session—A single defined period of training in a qualified FSTD that normally forms part of an EBT
module.

 iii
 Evidence-Based Training Implementation Guide

Error—An action or inaction by an operational person that leads to deviations from organizational or the
operational person’s intentions or expectations.
Error management—The process of detecting errors and responding to them with countermeasures that
reduce or eliminate the consequences of errors and mitigate the probability of further errors or undesired
states.
Evaluator—A person authorized to conduct the formal and final summative assessment of a trainee’s
performance.
Event—A combination of a task or a sub-task and the conditions under which the task or sub-task is to be
performed.
Facilitation technique—An active training method, which uses effective questioning, listening and a
non-judgmental approach and is particularly effective in developing skills and attitudes, assisting train-
ees to develop insight and their own solutions and resulting in better understanding, retention and
commitment.
Flight crew member—A licensed crew member charged with duties essential to the operation of an air-
craft during a flight duty period.
Grading—A relation of the results of the competency assessment to a defined numerical scale by an in-
structor or evaluator (in order to facilitate a harmonized and consistent training data collection).
Human factors principles—Principles which apply to aeronautical design, certification, training, opera-
tions and maintenance and which seek safe interface between the human and other system components
by proper consideration to human performance.
Human performance—Human capabilities and limitations which have an impact on the safety and effi-
ciency of aeronautical operations.
ICAO competency framework—A competency framework, developed by ICAO, is a selected group of
competencies for a given aviation discipline. Each competency has an associated description and observ-
able behaviors.
Instructional systems design (ISD)—A formal process for designing training which includes analysis,
design and production, and evaluation.
Inter-rater reliability—The consistency or stability of scores between different raters.
Line orientated flight scenario (LOFS)—LOFS refers to training and assessment involving a realistic, “real
time”, full mission simulation of scenarios that are representative of line operations.
Note: Special emphasis should be given to scenarios involving a broad set of competencies that simulate
the total line operational environment, for the purpose of training and assessing flight crew members.
Maneuvers—A sequence of deliberate actions to achieve a desired flight path. Flight path control may be
accomplished by a variety of means including manual aircraft control and the use of auto flight systems.
Monitoring—A cognitive process to compare an actual to an expected state.
Observable behavior (OB)—A single role-related behavior that can be observed and may or may not be
measurable.
Operations manual—A manual containing procedures, instructions and guidance for use by operational
personnel in the execution of their duties.
Phase of flight—A defined period within a flight.
Performance criteria—Statements used to assess whether the required levels of performance have been
achieved for a competency. A performance criterion consists of an observable behavior, condition(s) and a
competency standard.

iv
 Definitions

Pilot flying (PF)—The pilot whose primary task is to control and manage the flight path. The secondary
tasks of the PF are to perform non–flight path related actions (radio communications, aircraft systems,
other operational activities, etc.) and to monitor other crewmembers.
Pilot monitoring (PM)—The pilot whose primary task is to monitor the flight path and its management by
the PF. The secondary tasks of the PM are to perform non–flight path related actions (radio communica-
tions, aircraft systems, other operational activities, etc.) and to monitor other crewmembers.
Rating—An authorization entered on or associated with a license and forming part thereof, stating special
conditions, privileges or limitations pertaining to such license.
Scenario (event-set)—Relatively independent segment of training made up of several events.
Standardizers:—For the purpose of this guide, the standardizers are the instructors/evaluators in charge
of the training and assessment of other instructors/evaluators.
Threat—Events or errors that occur beyond the influence of the flight crew, increase operational complex-
ity and must be managed to maintain the margin of safety.
Threat management—The process of detecting threats and responding to them with countermeasures
that reduce or eliminate the consequences of threats and mitigate the probability of errors or undesired
states.
Trainee—For the purpose of this guide, “trainee” means a pilot or an instructor/evaluator receiving training
or evaluation.
Trainer—For the purpose of this guide, “trainer” means an instructor/evaluator conducting training or
evaluation.
Training objective—A clear statement that is comprised of three parts, i.e., the desired performance or
what the trainee is expected to be able to do at the end of training (or at the end of particular stages of
training), the performance standard that must be attained to confirm the trainee’s level of competence and
the conditions under which the trainee will demonstrate competence.
Unsafe situation—A situation which has led to an unacceptable reduction in safety margin.

 v
Acronyms
A/C Aircraft
ACAS Airborne Collision Avoidance System
ADDIE Analyze, design, develop, implement and evaluate
APP Approach
AQP Advanced Qualification Program
ATA Air Transport Association
ATC Air Traffic Control
ATO Approved Training Organization
ATQP Alternative Training and Qualification Program
CAA Civil Aviation Authority
CBT Computer-based training
CBTA Competency-Based Training and Assessment
CLB Climb
COM Competency Communication
CPDLC Controller–pilot data link communication
CRM Crew Resource Management
CRZ Cruise
DA Decision altitude
DES Descent
ETOPS Extended-range Twin-engine Operations Performance Standards
EASA European Aviation Safety Agency
EBT Evidence-Based Training
EGPWS Enhanced Ground Proximity Warning Systems
EVAL Evaluation Phase
FAA Federal Aviation Administration (United States of America)
FAF Final Approach Fix
FCOM Flight Crew Operating Manual
FFS Full Flight Simulator
FL Flight level
FDA Flight data analysis
FMS Flight Management System
FOQA Flight Operations Quality Assurance
FPA Competency Aeroplane Flight Path Management, automation
FPM Competency Aeroplane Flight Path Management, manual control
FSTD Flight Simulation Training Device
GA, G-A Go-around
GND Ground
GPS Global Positioning System
ICAO International Civil Aviation Organization
ICAP Instructor Concordance Assurance Program
IE Instructor/Evaluator
IFALPA International Federation of Air Line Pilots’ Associations
IOSA IATA Operational Safety Audit
IRR Inter-rater reliability
ISI In-seat instruction
KNO Competency Application of Knowledge
KSA Knowledge, skills and attitudes
LDG Landing
LIFUS Line Flying under Supervision
LOFS Line Orientated Flight Scenario
LOFT Line Oriented Flight Training

vi
 Acronyms

LOSA Line Operations Safety Audit

LTW Competency Leadership and Teamwork
MEL Minimum equipment list
MT Maneuvers Training Phase
MV Maneuvers Validation Phase
ND Navigation Display
NPCT Nominated Person Crew Training
OB Observable Behavior
OEM Original equipment manufacturer
ORCA Observe, Record, Classify and Assess
PANS-TRG Procedures for Air Navigation Services - Training, ICAO Doc 9868
PBN Performance-based navigation
PF Pilot Flying
PFD Primary Flight Display
PM Pilot Monitoring
PRO Competency Application of Procedures and Compliance with Regulations
PSD Competency Problem Solving and Decision Making
QNH Mean sea level pressure (MSLP)
R/T Radio Telephony
RVSM Reduced Vertical Separation Minima
SAW Competency Situation Awareness and Management of the Information
SMS Safety management system
SBT Scenario-Based Training (Phase)
SOP Standard Operating Procedure
SPA Special Approval
TEM Threat and Error Management
TO Take-off
UPRT Upset prevention and recovery training
WLM Competency Workload Management

 vii
Executive Letter
The Evidence-Based Training project was one of the major achievements of the IATA Training and
Qualification Initiative (ITQI) launched in 2007. EBT was endorsed by ICAO in 2013, with the publication of
Doc 9995, Manual of Evidence-based Training. To support operators with EBT implementation, a co-brand-
ed IATA/ICAO/IFALPA “Evidence-Based Training Implementation Guide” first Edition was also published in
2013.
Since then, many regulators and operators have implemented EBT and, more globally, the competen-
cy-based training and assessment (CBTA) approach of training has expanded. Hence, this guide has been
authored synchronously with the first IATA CBTA Guide for Flight Crew Training in order to ensure consis-
tency with the latest international provisions related to CBTA and the latest regulatory requirements relat-
ed to EBT, while providing the best practices gained during EBT implementation during the last decade.
Moreover, EBT as any CBTA program generates a significant volume of training data that relates directly to
human performance and that complements the safety data already collected in operations. Therefore, this
guide also clarifies the role of the training data into the safety management system to enhance operation-
al safety, with the goal to cope with the increased complexity of the aviation system, the introduction of
advanced technologies, the design of new procedures and the enforcement of environmental constraints.
It is our belief that the shared efforts put into updating this guide will support the enhancement of the
global implementation of EBT and will contribute to achieving our common goal of improving aviation safe-
ty worldwide.

Nick Careen Captain Uwe Harter

Senior Vice President Executive Vice-President
Operations, Safety and Security Technical & Safety Standards
IATA IFALPA

viii
Acknowledgements
The revision of the EBT Implementation Guide, 1st Edition, and the integration of experience gained with
EBT program implementation in this 2nd Edition, required a substantial amount of work and the support
from various industry stakeholders, without which the publication would not have been possible.
Therefore, IATA wants to acknowledge the critical and valuable contribution of the following EBT Subgroup
(EBT SG) members of the IATA Pilot Training and Licensing Task Force (PTL TF), and the representatives of
the following international organizations, in the development of this guide.

Capt. Arnar Agnarsson Capt. Stephen Mercer

B767/757 Captain TRE Standards Training Manager
MSc Human Factors, SME CRM Flight Operations, Training
Icelandair Emirates

Capt. Robert Gräf Capt. Thomas Robertson

Type Rating Instructor/Examiner (TRI/TRE) Director Global Flight Training
Flight Operations Support & Training Standards Boeing Test & Evaluation
Airbus Boeing

Capt. Glen Finch Capt. Sanjay Sapra

IFALPA Human Performance Committee Head of Crew Training
Line Training Captain Etihad Airways
Jazz Aviation
Frank Steiner
Capt. Hiroshi Horikawa Head of EBT Standards Lufthansa Group
777 Captain Lufthansa
Vice President Flight Training
Flight Operations Capt. David Swarbrick
Japan Airlines Training Manager Boeing
Vice Chairman of the PTLTF Flight Operations
Emirates
Dr Patrizia Knabl-Schmitz Chairman of the PTLTF
Human Factors Specialist
Emirates Flight Operations, Flight Training Capt. (ret) Holger Hoffmann
Emirates Type Rating Instructor/Examiner (TRI/TRE)
Senior Consultant
Capt. Fabien Laignel IATA
Compliance and Regulatory Affairs
EBT Manager - Crew Training Safety Manager Capt. (ret) Yann Renier
Air France Head Training and Licensing
OSS-Flight and Technical Operations
Capt. Richard Lenz IATA
Head of Conversion Training Standards
Chief Training CPT Airbus Long-range Victoria Romero
Lufthansa Senior Manager Training and Licensing
OSS-Flight and Technical Operations
Capt. Olivier Mazzoleni IATA
Flight Training International Regulatory Affairs
Airbus

 ix
Disclaimer
Although every effort has been made to ensure accuracy, the International Air Transport Association (IATA)
and the International Federation of Air Line Pilots’ Associations (IFALPA), and other contributors to this
publication shall not be held responsible for any loss or damage caused by errors, omissions, misprints
or misinterpretation of the contents hereof. Furthermore, the International Air Transport Association, the
International Federation of Air Line Pilots’ Associations, and the contributors to this publication expressly
disclaim any and all liability to any person or entity in respect of anything done or omitted, and the con-
sequences of anything done or omitted, by any such person or entity in reliance on the contents of this
publication.
Opinions expressed in this publication do not necessarily reflect the opinion of the International Air
Transport Association, nor the International Federation of Air Line Pilots’ Associations. The mention of
specific companies, products in this publication does not imply that they are endorsed or recommended
by any of the above in preference to others of a similar nature which are not mentioned.
No part of this publication may be reproduced, recast, reformatted or transmitted in any form by any
means, electronic or mechanical, including photocopying, recording or any information storage and re-
trieval system, without the prior written permission of the authors.

x
Publications
The material in this guide is based upon the following source documents:

Publication Date
ICAO Annex 1 Fourteenth Edition, July 2022
ICAO Annex 6, Part I Twelfth Edition, July 2022
ICAO Doc 9625, Manual of Criteria for the Qualification of Flight Fourth Edition, 2015
Simulation Training Devices, Volume 1 – Aeroplanes
ICAO Doc 9868, Procedures for Air Navigation Services – Training Third Edition, 2020 (Amendment 7)
(PANS-TRG)
ICAO Doc 9841 Manual on the Approval of Training Organizations Third Edition, 2018
ICAO Doc 9859 Safety Management Manual (SMM) Fourth Edition, 2018
ICAO Doc 9995, Manual of Evidence-based Training First Edition, 2013
Second Edition, draft September 2023
ICAO Doc 10011 Manual on Aeroplane Upset Prevention and First Edition, 2014
Recovery Training
IATA CBTA Guide for Flight Crew Training First Edition, 2023
IATA Guidance Material for Competency Assessment and Evaluation Edition 1, 2021
for Pilots, Instructors and Evaluators
IATA Guidance Material and Best Practices for Instructor and Edition 2, 2021
Evaluator Training
IATA Guidance Material for Improving Flight Crew Monitoring Edition 1, 2016
IATA EBT Data Report Edition 1, 2014 and Amendment 2021
IATA white paper “Competency-Based Training and Assessment 2021
(CBTA) Expansion within the Aviation System”
EASA Aircrew regulation Revision state October 2022
EASA Air Operations regulation Revision state February 2023
EASA EBT Manual Version 2.0, 2023

 xi
Section 1 - Introduction
1.1 Background
Evidence-Based Training (EBT) is a major safety initiative that arose from an industry-wide consensus
that, in order to reduce the airline accident rate, a strategic review of recurrent and type-rating training for
airline pilots was necessary. A review of available data sources, their scope, and relative reliability was un-
dertaken. This was followed by comprehensive analyses of the data sources chosen, with the objective of
determining the relevance of existing pilot training and to identify the most critical areas of training focus.
The aim of EBT is to train and assess the competencies required to operate safely and efficiently in a
commercial air transport environment, while addressing the most relevant threats according to evidence
collected in accidents, incidents, flight operations and training. The EBT program seeks to drive pilots to
master competencies, to enhance their confidence and to develop resilience managing unforeseen situa-
tions in stark contrast to traditional recurrent training and checking programs that train and evaluate “set
piece” events, often found to lack relevance to modern aircraft.
Since the publication of ICAO Doc 9995, Manual of Evidence-based Training in 2013, many States and
regulators have adopted EBT into their regulations and have encouraged their operators to implement
EBT. As an example, as of 2015 EASA permitted the introduction of EBT provisions for operator flight crew
training via the publication of a guidance material addressing the implementation of a mixed EBT program,
in advance to the baseline EBT European regulation applicability in 2020.
The experience gained by operators having implemented EBT during the last decade combined with the
work performed by IATA to review the EBT data report and produce its first amendment in 2021, led the
IATA Training and Licensing, Flight and Technical Department, to regroup EBT training experts from differ-
ent parts of the world to propose a second edition of the “Evidence-Based Training Implementation Guide”,
which integrates the latest Competency-Based Training and Assessment (CBTA) best practices.

1.2 EBT paradigm shift

Recurrent training and assessment is a process of revalidation and affirmation that the pilot continues to
demonstrate an adequate level of performance, in addition to presenting valuable opportunities for con-
tinuous improvement.
For many decades the industry has used the satisfactory completion of maneuvers as performance mea-
surements, such as rejected take-off, engine failure between V1 and V2, etc. A pilot able to demonstrate
the ability to fly these often-repetitive maneuvers within prescribed quantitative performance measure-
ments indicating an acceptable level of deviation from ideal criteria was deemed to be “proficient”.
The paradigm shift developed by CBTA, including EBT, is that pilot performance should be determined
across a wider range of competencies, and not simply by the achievement of a pre-determined outcome in
a specific task or maneuver.
Tasks and maneuvers remain important, as they are reflected in specific training topics which serve as
vehicles to develop and maintain the pilot competencies in an EBT recurrent training program.
EBT emphasizes training versus checking and promotes learning from positive performance. Therefore,
EBT recognizes the need that all pilots should be challenged with unrehearsed events that continually
build resilience and confidence throughout a pilot’s career. An essential component of the EBT ethos is
that pilots enrolled in an EBT program are more confident in their ability to perform their role.

1.3 Objective of this guide

The objective of this guide is to consolidate EBT guidance material and industry best practices to sup-
port the effective implementation of EBT. The contents of this guide are intended primarily to provide
guidance to the States and the industry on how to apply the provisions of ICAO Doc 9995, the Manual of

 1
 Evidence-Based Training Implementation Guide

Evidence-based Training, and the relevant provisions of ICAO Doc 9868 (PANS-TRG). This guide also refers
to published EBT regulations when they enhance effectiveness of the EBT implementation.
The contents of this guide provide awareness to and support EBT implementation by the following entities:
• Civil Aviation Authorities (CAAs) and Regulators
• Operators (AOCs)
• Approved Training Organizations (ATOs)
• Aircraft manufacturers (OEMs)
• Course developers
• Pilot representative bodies
It is expected that as EBT further matures there may be changes or improvements to the program. IATA,
together with industry stakeholders, is committed to review this guide on a regular basis and to incorpo-
rate any significant changes that may occur considering “in-service” experience.

1.4 Guide structure

The guide is structured as follows:
• Section 1–provides a general introduction
• Section 2–addresses the fundamentals of an EBT program
• Section 3–contains an overview for the (staged) implementation of an EBT program
• Section 4–provides specific guidance on the implementation process for the operator
• Section 5–provides specific guidance for the conduct of EBT from an instructor/evaluator (IE)
perspective
• Appendices–provide more details on the program's aspects and its implementation, in addition to oth-
er supporting reference material

1.5 ICAO provisions for EBT

EBT was developed as a means of compliance with ICAO Annex 6 to the Convention on International Civil
Aviation, Operation of Aircraft, Part I, International Commercial Air Transport – Aeroplanes, paragraphs 9.3,
Flight crew member training programs, and 9.4.4, Pilot proficiency checks.
Reference should also be made to ICAO Annex 1 to the Convention on International Civil Aviation,
Personnel Licensing, paragraph 1.2.5, Validity of licenses. Further guidance material is provided in ICAO
Doc 9868 (PANS-TRG), Chapter 3, which details the purpose and intent of EBT, and determines that States
wishing to implement EBT shall do this in accordance with ICAO Doc 9995, the Manual of Evidence-based
Training. This manual contains detailed guidance material for States and operators.
ICAO Doc 9868 (PANS-TRG), Chapter 7, contains the framework of qualification and competencies for
instructors, including those conducting EBT. This is further amplified in the Manual of Evidence-based
Training, Doc 9995.
Of particular interest are the appendices in ICAO Doc 9841, the Manual on the Approval of Training
Organizations, which detail the unique differences inherent in competency-based training programs, such
as EBT, and the need for specialized training for those Licensing Authorities that will regulate and oversee
such activities.
It is important to realize that the EBT program specifically, and only, addresses training to be conducted
in a qualified FSTD. Guidance on the qualification of FSTDs is contained in ICAO Doc 9625, the Manual of
Criteria for the Qualification of Flight Simulation Training Devices, Volume 1 – Aeroplanes.

2
Section 2 - EBT Fundamentals
2.1 Introduction
This section elaborates on the essential principles, components, and associated definitions of the EBT
program.
Since the publication of ICAO Doc 9868 (PANS-TRG) Amendment 7 in December 2020, the principles of
CBTA are applicable to all licensing and operator training with the goal to provide a competent workforce
for a safe and efficient air transport.
CBTA is defined by ICAO as “training and assessment that are characterized by a performance orientation,
emphasis on standards of performance and their measurement, and the development of training to the
specified performance standards".
EBT is defined by EASA as “assessment and training based on operational data that is characterized by the
development and assessment of the overall capability of a pilot across a range of competencies, rather
than by measuring the performance in individual events or maneuvers”.
Hence, EBT is a CBTA program that uses specific training topics as vehicles to develop the pilot com-
petencies. The training topics and their associated frequency were defined during the inception of
EBT, through the analysis of both safety and training data from a worldwide perspective.
EBT is a CBTA program applicable as an alternative to traditional operator recurrent training and checking.

2.2 Program overview

The EBT program should consist of:
• EBT modules composed of FSTD sessions
• Ground training
• Line evaluation(s) of competence
Within EBT, recurrent training and assessment of the competencies is considered over a three-year
period.
The EBT program includes a notional exemplar 48 FSTD hours for each pilot in a suitably qualified FSTD,
distributed over six EBT modules across the 3-year period. Each EBT module consists of an evaluation
phase and a training phase.
The training items that do not necessitate an FSTD as a training platform should be addressed during
ground training.
Section 4 provides guidance on how to implement the EBT modules, the ground training and a line evalua-
tion of competence which may supplement line checks conducted within traditional training programs.

2.3 Phases within a module

Each module should normally contain the following three phases (in certain circumstances the CAA may
accept that the evaluation phase is conducted in a different sequence than the one presented in this
guide; this is intended to enable coherence with certain existing AQP or ATQP programs).
(a) Evaluation phase (EVAL). This phase consists of one or more line-oriented flight scenarios. The
assessment should be realistic, and the scenario(s) should be representative of the operator’s
environment.
(b) Maneuvers training phase (MT). This phase consists of maneuvers that place a significant demand on
a proficient crew. Maneuvers in this context mean a sequence of deliberate actions to achieve a pre-
scribed flight path or to perform a prescribed event to a prescribed outcome.

 3
 Evidence-Based Training Implementation Guide

(c) Scenario-based training phase (SBT). This phase forms the largest phase in the EBT program and is
designed to focus on the development of competencies, whilst training to mitigate the most critical
risks. The phase will include the management of specific threats in a line orientated environment. A
portion of the phase will also be directed towards the management of critical system malfunctions.
Lists of maneuvers and the most critical risks are provided in Appendix G.IV, according to aircraft gen-
eration with indications of the recommended frequency in the EBT program. The scenarios shall include
critical operational and environmental threats, in addition to building effective crew interaction to identify
and manage errors. Scenario elements to be used for EBT module design and their related competencies
are provided in Appendix G.V.
For the EBT program to be fully effective, it is important to recognize that the predetermined scenarios are
simply a means to develop competency, and not an end or “tick box” exercise in themselves.
Industry best practice shows that the modules are usually scheduled as two consecutive FSTD sessions
of 4 hours each. The following distribution of the EBT module phases is recommended: EVAL and MT con-
stituting Session 1 and SBT constituting Session 2.

2.4 Program content

2.4.1 Assessment and training topics
The purpose of the EBT program is to use training topics defined to be most critical, as a means of as-
sessing and developing competencies. It is important to note that, when adapting material in Appendix G
to specific operator’s needs, it is never the intention that all possible events or scenarios be programed
within the EBT recurrent cycle. Operators should select the scenarios most useful to their needs while
ensuring that the frequency of defined topics is maintained, to minimize competence decay over time.
Prioritization of the training topics is probably the most important result from the EBT data analysis (de-
tailed information is provided in the EBT Data Report). Three levels of priority A, B and C are used to deter-
mine the frequency of pilot exposure to the defined training topics within a 3-year rolling recurrent training
program.
The assessment and training guidance matrix is an extraction from the Data Report and its amendment
(2021) which correlate specific training scenarios to competencies considered critical to the management
of a specified threat or error.

4
 EBT Fundamentals

The usage of the matrix during EBT program design is described in Section 4, Step 9: Assessment and
training topics distribution, and Appendix G.II.
The following template provides the training topics and their associated frequency for the most common
Jet aircraft generations 3 and 4.

EBT/CBTA Overarching Principles (A frequency)

• Competencies non-technical (CRM) • Surprise
Generations 3 & 4 Jet Training Topics

• Compliance • Workload, distraction, pressure

• Monitoring and cross-checking • Aircraft system management
Adverse weather Adverse wind ATC
Automation management Aircraft system malfunction Engine failure
Go-Around management Approach, visibility close to Fire and smoke
minimum management
Manual aircraft control Landing Managing loading, fuel,
performance errors
A B C
Error management, mis- Runway or taxiway condition Navigation
managed aircraft state
Unstable approach Terrain Pilot incapacitation
UPRT Traffic
Windshear recovery 1
Operations or type
specific

2.4.2 Equivalency of aircraft system malfunctions

The philosophy of EBT is that the qualified FSTD should be used to the maximum extent possible for as-
sessing and developing pilots’ competence. Pilots should be exposed to a wide variety of situations that
may be faced in line operations. Aircraft system malfunctions to be considered for the EVAL and SBT
phases are those that place a significant demand on a proficient crew.
The EBT philosophy for the equivalency of aircraft system malfunctions is the following:
• Each pilot is trained and assessed in the management of aircraft system malfunctions
• Aircraft system malfunctions that place a significant demand on a proficient crew are organized by
reference to the following characteristics:
– Immediacy
– Complexity
– Degradation of aircraft control
– Loss of instrumentation
– Management of consequences
• Each pilot is exposed to at least one malfunction for each characteristic, at the frequency determined
by the table of assessment and training topics
• Demonstrated proficiency in the management of one malfunction is considered equivalent to demon-
strated proficiency in the management of other malfunctions with the same characteristics
All malfunctions not covered by these characteristics continue to require review and appropriate proce-
dural knowledge but with different means. For instance, to repeat simple malfunctions and use the FSTD

1 Should be distributed across the EBT modules of the triannual plan.

 5
 Evidence-Based Training Implementation Guide

environment as a “procedure training device” is to deny the full benefit of the FSTD as a learning tool. It is
precisely for this reason that the malfunction clustering system has been developed, to reduce unneces-
sary requirements to “tick boxes” against each listed malfunction of the ATA chapters.
Procedures and guidelines on how to perform the equivalency of malfunctions process can be found in
Section 4, Step 7: Equivalency of malfunctions.

2.4.3 Equivalency of approaches

The development of an EBT program requires the determination of critical training events, the develop-
ment of training scenarios and the definition of appropriate pilot performance criteria when managing
these events and scenarios.
Selection of approaches for scenario-based training should be based on the underlying elements of flight
crew performance to conduct them. Equivalent groups of approaches can be determined by reference
to these elements. Demonstrated competence in the conduct of one approach type can be considered
equivalent to demonstrated competence for the other approach types in the same group.
Procedures and guidelines on how to perform the Equivalency of Approaches process can be found in
Section 4, Step 8: Equivalency of approaches.

2.5 Pilot and instructor/evaluator competency frameworks

One of the essential components of the EBT program is the implementation of a pilot competency
framework.
IATA and several CAAs recommend using an adapted competency model, consisting of nine competen-
cies, the eight ICAO aeroplane pilot competencies, plus the competency “Application of Knowledge”. The
competency “Application of Knowledge” refers to the ability of the pilot to recall and proactively update
relevant knowledge and to apply acquired knowledge to the operational environment.
The following are the 9 IATA pilot competencies and their associated acronyms.

Acronyms Pilot Competencies

KNO Application of Knowledge
PRO Application of Procedures and Compliance with Regulation
COM Communication
FPA Aeroplane Flight Path Management, automation
FPM Aeroplane Flight Path Management, manual control
LTW Leadership and Teamwork
PSD Problem Solving and Decision Making
SAW Situation Awareness and Management of Information
WLM Workload Management

IATA also recommends implementing an instructor/evaluator (IE) competency framework for the initial and
recurrent EBT IE standardization. In 2017, IATA developed such a framework, which includes the pilot com-
petencies and four additional competencies for instructor/evaluator (IEC).
ICAO endorsed this IATA competency framework under the name of “ICAO pilot instructor and evaluator
competency framework” which is described in ICAO Doc 9868 (PANS-TRG).

Pilot Instructor/Evaluator (IE) Competencies

– Pilot Competencies – Interaction with the trainees
– Management of the learning environment – Assessment and Evaluation
– Instruction

6
 EBT Fundamentals

The IATA pilot and IE competency frameworks are available in Appendix B to this Section.
The competency frameworks are structured in such a way that each competency and associated descrip-
tion and Observable Behaviors (OB) are clearly identifiable and properly determined.
Since 2018, the OBs receive a number to facilitate CBTA data collection and analysis.
Example of the IATA pilot competency framework for the competency Communication:

COMMUNICATION
Description Observable behaviors (OB)
Communicates OB 2.1 Determines that the recipient is ready and able to receive information
through appropri- OB 2.2 Selects appropriately what, when, how and with whom to communicate
ate means in the
operational environ- OB 2.3 Conveys messages clearly, accurately and concisely
ment, in both normal OB 2.4 Confirms that the recipient demonstrates understanding of important
and non-normal information
situations.
OB 2.5 Listens actively and demonstrates understanding when receiving information
OB 2.6 Asks relevant and effective questions
OB 2.7 Uses appropriate escalation in communication to resolve identified deviations
OB 2.8 Uses and interprets non-verbal communication in a manner appropriate to the
organizational and social culture
OB 2.9 Adheres to standard radiotelephone phraseology and procedures
OB 2.10 Accurately reads, interprets, constructs, and responds to datalink messages in
English

Operators and ATOs should implement the IATA and ICAO pilot and IE competency frameworks in order to
facilitate the analysis and benchmarking of the training and safety data.

2.6 Adapted Competency Model

The Adapted Competency Model introduces the performance criteria to complement the competency
framework. These performance criteria include the competency standards and the conditions under
which the competencies are trained and assessed.

 7
 Evidence-Based Training Implementation Guide

2.6.1 Structure of an adapted competency model

Performance criteria. Statements used to assess whether the required levels of performance have been
achieved for a competency. A performance criterion consists of an observable behavior, condition(s) and a
competency standard.

Performance criteria
Competency Description Observable
Competency Assessment
behavior (OB)
OB 1 Competency standard Conditions:
Competency 1 Description 1 OB 2 (Final or interim) • Context com-
plexity (op-
OB n
erational and
OB 1 environmental)
Competency 2 Description 2 OB 2 • Level of sup-
OB n port of the
instructor/
OB 1 evaluator
Competency n Description n OB 2 • Aircraft, FSTD,
tool, system, or
OB n equipment

Competency Standard. A level of performance that is defined as

acceptable when assessing whether or not competency has been
achieved

Conditions. Anything that may qualify a specific environment

in which performance will be demonstrated.

Notes:
1. Final and interim competency standards are defined by the organization.
2. There are different types of conditions to be considered:
• Conditions related to context (nature and complexity of the operational and environmental
context)
• Conditions related to the amount of support or assistance a trainee can expect from the instructor
• Conditions related to tools and systems/equipment airplane, FSTD

2.6.2 Characterization of the conditions

IATA recommends that AOCs and ATOs characterize the conditions under which the competencies are
trained and assessed during any CBTA program, including EBT, as follows:

Conditions related to tools and systems/equipment, airplane, FSTD

These conditions relate to the training platform that is used to train and assess the competencies. The
training platform may be a classroom, a part task trainer, an FSTD, an airplane, etc.

Conditions related to the amount of support a trainee can expect from the instructor
These conditions should be correlated to recognized instructional methods as described in Section 5.6.
Three levels of instructor support (low, medium and high) have been defined and correlated to the recog-
nized instructional methods as follows.
Note: Facilitation may be relevant for low to medium levels of support, depending on the trainees’ needs.

8
 EBT Fundamentals

Level of support from the

Low Medium High
instructor
Instructional method – Discover without – Discover with – Explain (Tell)
assistance assistance – Demonstrate (Show)
– Facilitation – Facilitation

Conditions related to context (nature and complexity of operational and environmental

context)
The characterization of the context complexity is of upmost importance for an effective and efficient EBT
implementation since:
• Most of the competencies necessitate the insertion of operational and/or environment threats to en-
sure regular demonstration of the associated OBs during the EBT module, and
• The EBT approach to the competency development is based on three phases within the modules
where the complexity of the context is variable across the phases.
IATA addresses the context complexity characterization by the creation of a bi-dimensional matrix that
combines different levels of complexity for both the operational and the environmental context.
To provide examples with three levels of complexity, in a training event containing the threat E01.01
Thunderstorm, the environmental context complexity would be:
• “Low” for an isolated cumulonimbus (CB-ISOL)
• “Medium” for an occasional cumulonimbus (CB-OCNL)
• “High” for frequent cumulonimbus (CB-FREQ)
To provide examples with three levels of complexity, in a training event containing the threat A01 Aircraft
malfunction, the operational context complexity would be:
• “Low” for aircraft malfunction placing limited demand on the crew
• “Medium” for aircraft malfunction placing moderate demand on the crew
• “High” for aircraft malfunction placing significant demand on the crew
Note: The criteria to determine the levels of demand (limited, moderate or significant) that a malfunction
places on a crew can be found in ICAO Doc 9995, in the European Air Operation regulation and in the OEM
documentation.
The bi-dimensional matrix presents the zones (Zone XX) resulting from the combination of the operational
and environmental context complexity. These zones are proportionate to the level of resources to be de-
ployed by the pilot(s) to safely operate the aircraft while mitigating the risks associated to the operational
and environmental threats present in the training event.
The matrix below presents an illustration of the zones resulting from 3 levels of operational and environ-
mental context complexity.

 9
 Evidence-Based Training Implementation Guide

2.6.3 IATA general recommendation for adaptation

The criteria for adaptation are related to the following elements:
1. The competency itself, including its name
2. The description of the competency
3. The observable behaviors
4. The interim or final competency standard
5. The conditions of the demonstration of competency
The recommended IATA Adapted Competency Model for EBT is described in Section 4, Step 3:
Development of an adapted competency model and grading system.
The notion of interim and final competency standards in the context of EBT is elaborated in Section 5,
Grading.
IATA considers that elements 1 to 3 are optional for adaption and the ATO/AOC should modify only a
few elements for high-level training strategies or for tactical reasons due to a specific type of operation.
An example of strategic adaptation is the inclusion of the competency “Application of Knowledge” in the
IATA pilot competencies framework. It is worth noting that “Application of Knowledge” has been adopted
by most operators and by regulators such as EASA.
An example of tactical adaptation could be the removal of Observable Behaviors, OB 2.10 “Accurately
reads, interprets, constructs and responds to datalink messages in English” from the competency
Communication for the ab initio training centers that do not have data link equipment for their operations.
IATA considers that elements 4 and 5 necessitate adaptation, depending on the purpose of the training. An
example for different final competency standards can be illustrated during the selection of pilot instruc-
tors who need to achieve a higher final competency standard for specific competencies, compared to the
regular pilots. For example, for the competency Workload Management the final competency standard
should be “effective” (corresponding to grade 4) for pilot instructors, while the final competency standard
should be “adequate” (corresponding to grade 3) for regular pilots.
The example for different conditions can be illustrated during a type rating course. The competency stan-
dard is identical throughout the course but during the early stages, trainees can expect active coaching
and teaching from the instructor. As the trainee progresses towards the final competency standard and
gains more confidence in performing independently, the instructor takes on a more passive role.

10
 EBT Fundamentals

2.7 Relationship between CRM, TEM and Pilot/IE competency

frameworks
In 2017, the IATA Pilot Training and Licensing experts conducted an analysis among several international
organizations representing airlines and training organizations. The aim of the analysis was to assess the
potential differences in the implementation of the TEM principles and to harmonize the role of the pilot and
IE competency frameworks within the TEM model.
The IATA Pilot Training and Licensing experts agreed on the following conclusions:
• TEM is the overarching concept in training and operations
• Pilot and IE competencies represent the set of individual and team countermeasures
• CRM skills are embedded in the pilot and IE competencies
• CRM training supports the development of the pilot and IE competencies as countermeasures in the
TEM concept
The output of this work was adopted by ICAO through Amendment 5 to ICAO Doc 9868 (PANS-TRG), which
states that: “from a CBTA (including EBT) perspective, the competencies of the approved adapted com-
petency model provide individual and team countermeasures to threats and errors and undesired aircraft
states. CRM skills are embedded in the approved adapted competency model. Therefore, the CRM training
supports the development of the competencies as countermeasures in the TEM concept.”
and
“In competency-based training, including EBT, TEM is naturally and fully embedded in the training curricu-
lum. The competencies of the approved adapted competency model provide individual and team counter-
measures to threats and errors to avoid undesired aircraft states.”
The following schematic is used as a pedagogical tool to illustrate the above concept.

 11
 Evidence-Based Training Implementation Guide

Note:
• “Limited and Momentary Reduction of Safety Margin” describes an outcome of TEM where the pilot or
the IE demonstrated OBs that did not allow, on few occasions, a timely management of the threats or
errors. This led to a limited and momentary reduction of the safety margin.
• “Unacceptable Reduction of Safety Margin” describes an outcome of TEM where the pilot or the IE
demonstrated OBs that did not allow a timely management of the threats or errors. This led to an unac-
ceptable reduction of the safety margin.
Under EBT, Threat and Error Management is naturally and fully embedded in the training program. The Pilot
and IE competencies provide individual and team countermeasures to threats and errors to avoid a reduc-
tion of safety margins during training and operations:
• The more OBs are timely demonstrated, the better the TEM should be. This should lead to the mainte-
nance of the safety margins.
• Per opposition, the OBs that have not been demonstrated when they were required could result in the
mismanagement of the threats and errors. This could lead to a reduction of the safety margins.

2.8 Competency assessment process and method

2.8.1 Definitions
Assessment. The determination by an IE as to whether a candidate meets a required competency stan-
dard under given conditions, by collecting evidence from observable behaviors.
The assessment takes place during instruction and evaluation and can be either formative or summative.
Formative Assessment. The formative assessments are a part of the learning process. IEs provide feed-
back to the trainees on how they are progressing toward the interim or final competency standard. This
type of assessment enables the trainees to progressively build on competencies already acquired and
should aid learning by identifying gaps as learning opportunities.
The formative assessment should serve to motivate trainees, identify strengths and weaknesses, and
promote learning.
Summative Assessment. The summative assessments provide a method that enables the IE to work
with a trainee to collect evidence of the competencies and performance criteria to be demonstrated with
respect to the interim or final competency standard(s).
Summative assessments are carried out at defined points during the training and/or at the end of the
training. During summative assessments, the decision is either ‎ “competent” ‎ or ‎ “not ‎ competent”
with respect to the interim or final competency standard(s). However, this can be further developed into
a more refined grading system with a scale of judgment to improve feedback for the trainee and training
personnel.

2.8.2 Process to assess performance

To assess the trainee’s performance, the trainer should apply the following process (ORCA):
• Observe performance (behaviors) during the training or evaluation.
• Record details of effective and ineffective performance (behaviors) observed during the training or
evaluation (‘record’ in this context refers to instructors/evaluators taking notes).
• Classify observations against the Observable Behaviors (OBs) and allocate the OBs to each
competency.
• Assess the performance by determining the root cause(s) for low or high performance according to
the competency framework. Low performance would normally indicate the area of performance to be
remediated in subsequent training.
Note: Depending on the training objective, the trainer guidance may indicate competencies which may
be irrelevant to be assessed or recorded. In that case, the trainer will record “N/O” (NOT OBSERVABLE).

12
 EBT Fundamentals

2.8.3 Competency assessment method

IATA recommends applying the following methodology to ensure the maximum level of consistency and
objectivity to assessments performed in a CBTA program.
To assess how well the trainee demonstrated the competency during training or evaluation, the trainer
should assess the associated OBs of each competency against the following dimensions by determining:
• How many OBs the trainee demonstrated when they were required,
• How often the trainee demonstrated the OB(s) when they were required, and
• What was the outcome of the threat management and error management relating specifically to the
competency being assessed?
The competency assessment (HOW WELL) is the combination of the number of OBs demonstrated and
their frequency of demonstration and the consequential outcome of the TEM relating specifically to the
competency being assessed.
The “HOW MANY” dimension provides evidence related to the acquisition of the competency.
The “HOW OFTEN” dimension provides evidence related to the robustness of the competency.
The “Outcome of TEM” dimension provides evidence related to the effectiveness of the competency as
individual and team countermeasures against threats and errors.
The following template provides an illustration of the competency assessment method.

regularly*: When they were required, the trainee has regularly* (very often) demonstrated most of the OBs
during training or evaluation. This led to an improvement of the safety margin.
safe* : Illustrates a more pro-active safety level. (Please see 5.5.3, Word Pictures of “OUTCOME OF TEM”
dimension.)
The IATA “Competency Assessment and Evaluation for Pilots, Instructors and Evaluators” is intended to
provide guidance to Civil Aviation Authorities (CAA), Operators (AOC) and Approved Training Organizations
(ATO) for the competency assessment and evaluation of pilots and IEs in the context of the global expan-
sion of CBTA programs.
Excerpts from this guide and EBT-specific guidance are contained in Section 4 and Section 5.

 13
 Evidence-Based Training Implementation Guide

2.9 Concept of resilience

The EBT program seeks to drive pilots to master competencies, to enhance their confidence and to devel-
op resilience managing unforeseen situations. Consequently, this paragraph elaborates on the concept of
resilience and on the how an EBT program should ensure its development.
Resilience is defined by the EU regulation as “the ability of a flight crew member to recognize, absorb and
adapt to disruptions.”
Note: Science and academics provide additional definitions of the concept. For example, resilience is
also defined as “the ability to prepare and plan for, absorb, recover from, and more successfully adapt to
adverse events” (US National Academies of science, engineering and medicine). Scientific research in
psychology often describes resilience as the ability to bounce back from adversity as well as to grow from
challenges.
From a practical perspective, pilots demonstrate resilience when they are able to maintain or to restore
the safety margins following one or more unexpected events, which may result in a deterioration of
performance.
Managing operational disruptions is a core business for pilots. Many of these disruptions may be antici-
pated and consequently managed effectively. However, in some cases operational disruptions may not be
anticipated and thus occur unexpectedly, while triggering surprise, or inducing a startle response.
Note: Definitions from ICAO Doc 10011:
• Surprise. The emotion-based recognition of a difference in what was expected and what is actual.
• Startle. The initial short-term, involuntary physiological and cognitive reactions to an unexpected
event that commence the normal human stress response.
Startle and surprise are terms that are often used interchangeably, although they have different origins.
• In prevalent aviation literature, surprise is regarded as a cognitive-emotional response to an unexpect-
ed event, resulting from a mismatch between expectation and perception. Surprise is triggered either
through the occurrence of an unexpected event, or the unexpected absence of an expected event.
• In contrast, startle is a physiological reflex which occurs after sudden exposure to an intense external
stimulus (auditory, visual, tactile). Immediate consequences of the startle response include physical,
cognitive, emotional, and behavioral impairments, whereby their magnitude and duration depend on
the severity of the stimulus and the perceived seriousness of the threat. In life-threating situations,
these reactions can be substantial and long lasting, making it difficult for pilots to recover from the
critical event.
Startle and surprise can be perceived simultaneously. However, it is also possible to be surprised without
being startled, and to be startled without being surprised. In the latter case, despite the onset of the star-
tling stimulus being known or anticipated, the startle reflex will nevertheless occur, and its psychophysio-
logical consequences have to be managed.
A clear understanding of the concepts of startle and surprise, and to train the recovery from both is essen-
tial for resilience development in managing critical events.
• A robust application of TEM can reduce surprise through the anticipation and effective subsequent
management of events. In addition, Situation Awareness is a critical competency to avoid surprises.
• The training of startle may comprise the exposure to startling events in the simulator but should also
include education of pilots about the nature of the startle reflex, the consequences they need to
expect, and the development of resilience strategies to overcome a deterioration of performance.
Extensive evidence from literature suggests that a pilot’s immediate actions following a startle are
often wrong or inadequate to recover from the situation. Since startle may severely impair cognitive
functioning, easy and basic rules are often taught, such as “sit on your hands”, or “take a deep breath
before starting any action”. In any case, it is essential to establish a brief but deliberate cognitive gap
between the startling event and taking the first action to regain control. Mnemonics such as “push, roll,
power, stabilize” in UPRT may support recovery from startle.

14
 EBT Fundamentals

• After overcoming the initial effects of surprise and/or startle, pilots should apply the required compe-
tencies to maintain or restore the safety margin, including the competency PRO. To manage disrup-
tions not covered by any normal or abnormal procedures or checklists, pilots need to adapt by applying
appropriate OBs (e.g., OB 6.8 Adapts when faced with situation where no guidance or procedure exists,
or OB 5.2 Demonstrates initiative and provides direction when required), and OB 8.6 Seeks and accepts
assistance, when appropriate).
The objective of resilience development is to:
• Strengthen the competency situation awareness, to minimize surprise
• Understand the concepts of surprise and startle, as well as their differences
• Understand and train resilience strategies to recover from surprise and startle
• React in a controlled and structured manner to an unexpected event
• Maintain or restore the safety margin by applying appropriate competencies
Resilience development should be integrated in EBT module design and the conduct of its training phase,
as indicated in Sections 4 and 5. Emphasis on resilience development should also be given in Human
Factors and CRM training. Awareness and training on protective resilience factors such as mental agility,
self-regulation, emotional control, and self-efficacy, etc., can further support resilience on the flight deck.

2.10 EBT IE standardization

The EBT IEs should hold or have held a license and be authorized to carry out instruction on the basis of
their expertise, and/or qualifications, and/or ratings, and should also meet additional EBT related qualifica-
tion requirements.
CBTA, including EBT programs, are highly dependent upon the IEs’ abilities and commitment to develop
and assess the trainees’ competence and confidence. Therefore, it is important that only those individuals
who have the potential to support the trainees’ professional and personal development are considered for
IE positions. Hence prospective IEs should be selected, trained and qualified in accordance with the provi-
sions in Section 4 of this guide to ensure safe operations and training effectiveness.
Therefore, the IE applicant should:
• Undergo a selection process designed to ensure that the individual’s motivation and disposition are
suitable for the IE role.
• Receive initial theoretical and practical training to develop the required IE competencies (as described
in Appendix B.II). The IE training should also focus on:
– Determining root causes for deviations below the standards of performance; and
– Identifying situations that could result in unacceptable reductions in safety margins.
• Successfully complete an initial formal IE competency assessment during the conduct of practical EBT
FSTD training. During the assessment, the applicant should demonstrate the IE competencies accord-
ing to the level of performance defined in the adapted competency model, as approved by the appro-
priate authority. The competency assessment should be conducted during a practical training session
supervised by a person nominated by the AOC or the ATO.
All the EBT IEs should receive refresher training and complete a formal recurrent IE competency assess-
ment during the conduct of practical EBT FSTD training, at an interval not greater than three years. The
competency assessment should be conducted during a practical training session supervised by a person
nominated by the operator or the ATO.
For more details regarding EBT IE standardization refer to Section 4, Step 5: EBT IE initial standardization,
and Step 15: EBT IE recurrent standardization and ICAP.

 15
 Evidence-Based Training Implementation Guide

2.11 Data collection and analysis

2.11.1 Importance of data within the EBT program
Data collection and analysis are essential components to inform, develop, monitor, and enhance training
programs. An EBT program requires the determination of critical training events, the development of train-
ing scenarios, and the definition of appropriate flight crew OBs when managing these events and scenari-
os. It is necessary to collect real world data from accidents, incidents, flight operations and training to feed
and validate course development. Data collection should provide for a detailed analysis of existing threats
and identify potential weaknesses regarding the airline’s operational safety.
The following data sources are relevant:
Inner Loop Data, which refers to data available within the operator. It comprises two main components:
1. Training data
• Pilot performance data, with an analysis of recent trends across all the operator’s fleets, to identify
and quantify differences and specific areas of threat or interest; this requires the development of a
training measurement system
• IE data, within the framework of a robust Instructor Concordance Assurance Program (ICAP)
• Subjective feedback data, to complement objective data and derive insights into the quality and
effectiveness of the EBT program from a trainee and IE perspective
2. Operational data
• Flight data with an analysis of recent trends across the operator’s own or similar fleets, to identify
and quantify differences and specific areas of threat or interest
• Operator’s SMS data, including safety reports with an analysis of the operator’s safety data from all
sources with specific identification of those risks that can be mitigated by pilot training
• Operator’s specific operational challenges that relate to route network, aerodromes used, weather,
etc.
Outer Loop Data, which refers to data from multiple external sources outside of the operator. World fleet
data can provide an analysis of safety data from operations with similar aircraft types and operations.
Outer loop data also includes OEM data. The Data Report for Evidence-Based Training allows to create a
training framework that could mitigate the operational risks raised in the report.
The development of the operator’s training program starts with the generic training program by genera-
tion of aircraft, provided by the regulator. With this program, the operator distributes the assessment and
training topics over a period of 3 years and adds other requirements such as specific approvals (e.g., LVO),
further operational or training requirements (e.g., either seat qualification).
Evidence gained from previous training and the operator’s SMS has to be integrated at this point, as well. It
is advantageous to develop, train and assess competencies utilizing scenarios that are relevant to opera-
tions. In some cases, the data may highlight certain competencies considered critical to the management
of a specified threat or error in the operation, which may lead to a focus on specific areas as part of the
training program.

2.11.2 Collection and analysis of training data

Training metrics introduction
Training metrics, sustaining the monitoring and enhancement of the EBT program performance, constitute
the core of the EBT training data that should be collected and analyzed by the CAAs, operators and train-
ing organizations. Taking full advantage of such data requires robust and well-calibrated metrics.
Metrics derived from EBT training data can be classified into the following categories:
• Pilot performance
• Instructor/Evaluator performance
• Trainee feedback
16
 EBT Fundamentals

Pilot performance metrics refer to data derived directly from the EBT grading system. The following grad-
ing metrics were introduced within the European regulatory framework in the context of baseline EBT
implementation and recommended that they be captured.

Example of grading metrics mandated by the Evidence-Based Training European regulation

Level 0 (competent metrics): The information whether the pilot(s) is (are) competent or not.
Level 1 (competency metrics): Level of performance reflected by numeric grade of the competencies (e.g., 1 to 5).
Level 2 (observable behavior metrics): The instructors record OBs predetermined or required by the organization
(Regulatory or Policy requirements).
Level 3 (TEM metrics): The instructor records Threats, Errors or Reduction of Safety Margin predetermined or
required by the organization.

These metrics should then be compared to independent variables such as different aircraft types, pilot
roles, pilot qualifications, experience level, or training topics.
Vital evidence can be gained from assessing OBs not just on a quantitative but also on a qualitative level.
These insights can provide a fundamental source for deriving training needs and feeding the respective
EBT program design.
Instructor performance metrics are important to measure the effectiveness of the instructor calibration
process. However, it is essential that the purpose of this system is not to spy on instructors or to pressure
individuals to change their grading. Instructor performance metrics are further discussed in Section 4 as
part of the Instructor Concordance Assurance Program (ICAP).
Trainee feedback should be captured to provide a different perspective on the quality and effectiveness
of the training program and induce subsequent program developments. It is recommended to capture this
feedback through an electronic feedback survey, which should be reviewed regularly through an estab-
lished feedback review process. Operators should be attentive to response rates and actively encourage
feedback from trainees and instructors.
Data review processes and procedures should be established within the training department to ensure the
quality and proper conduct of the data review and analysis. Descriptive statistics should be used to summa-
rize and describe the metrics captured. Inferential statistics should be used when comparing results be-
tween different groups or trying to reach conclusions which extend beyond the immediate sample observed.
It needs to be recognized that data analytics requires a specific skill set and expertise. Appropriate busi-
ness analytics capabilities should be established within training departments to ensure that data is ana-
lyzed correctly, and interpretations of the results are valid. If such expertise is not available, consideration
should be given to outsourcing data analytics and reporting to either a dedicated analytics department
within the business or seeking external consultation.
Although training metrics are an invaluable component in supporting an EBT training program, they must
be placed in the context of operational data, because only the latter can justify the importance of a specif-
ic competency within the real operation.

Diverging needs of assessment and grading data for the various stakeholders
The training data provided as output of assessment and grading in EBT serve three different stakeholders,
who have specific and diverging demands. See the following table.

Stakeholder Demand on the assessment and grading system

Trainee Knows his/her own strength and weakness. Understands how to improve.
Operator Monitors and ensures the development of the competencies of the individual pilot.
Manages training system performance.
Obtains data for instructor standardization and concordance.
Monitors and compares performance trend by pilot groups (e.g., fleet, rank)
Validates and refines the EBT program.
CAA Conducts EBT oversight (revalidation or renewal of licenses).

 17
 Evidence-Based Training Implementation Guide

The specific demands shown in the above table consequentially require that the IEs focus on various feed-
back messages and data when conducting the assessment and grading:

Stakeholder Feedback following an assessment (at the end of ORCA)

Trainee Facilitated guidance based on the demonstrated (or missing) OBs; potentially with com-
menting text (Level 2 and Level 3 grading metrics).
Operator Numerical competency grades + identification of effective and ineffective performance
(demonstrated and missing OBs) for the short and mid-term training needs of the in-
dividual Pilot (Level 1 and Level 2 grading metrics).Numerical competency grades for
statistical trend analysis (training system performance, instructor standardization and
concordance, trend-analysis of pilot group performance (e.g., per fleet, rank …) (Level 0
and Level 1 grading metrics).
CAA EBT oversight - effectiveness of the operator’s EBT program using primarily data related
to Level 0 grading metrics.

The collection and analysis of these EBT training metrics within the global SMS should first enhance a pro-
active hazard identification, and second support a more predictive approach to hazard identification
by providing visibility on the individual and the team countermeasures (the competencies) to efficiently
manage the threats encountered and errors committed in both training and operational contexts.

2.11.3 Collection and analysis of operational data

The purpose of data collection and analysis is to inform the training program by deriving insights into per-
formance deficiencies. This should then drive adjustments to the program and allow to determine whether
it has been effective or not. The following section will deal with the safety reporting programs forming the
most classic source of safety information.

Flight data analysis (FDA)

Flight data analysis is a powerful data collection tool that allows quick access to the results. A limitation is
that FDA can only detect pre-defined events based on predetermined technical cues.
FDA information is useful for examining what has occurred in the operation, but not why an event occurred
or how the situation was managed after the event did occur. However, FDA can be very powerful in high-
lighting important operational trends, for example:
• The rate of unstable approaches and corresponding rate of resultant go-arounds versus landings
• The frequency of some threats and events, e.g., ACAS alerts, rejected take-offs
• Operation and route specificities, including those of destinations, and other operational factors
• Issues that relate directly to training, e.g., hard landings or rotation technique
FDA is most effective as a trending tool to measure improvement or degradation in operational perfor-
mance in terms of the risk events defined in the specific FDA program. From the trends, adjustments can
be made in the training program to mitigate the risk shown by the FDA analysis. As the trend continues, the
effectiveness of the adjustments can be measured and validated in a quality loop process.

Flight deck observation

Flight deck observation is intended to monitor normal operations by trained observers, in programs such
as Line Oriented Safety Audit (LOSA) and other similar methods. The philosophy is a non-intrusive obser-
vation of the flight crew activity. The focus is on threats and errors and on the effectiveness of compe-
tencies in their management. Results are not correlated to the individual pilots but are interpreted at the
level of the whole operation. LOSA is performed on a time-limited (snapshot) basis, but other variations of
normal operations monitoring can occur on a more continuous basis.
The benefit of flight deck observation is in its capability to capture data through anonymous and struc-
tured recording of normal operations. The processes and competencies used by pilots to manage threats,
errors and undesired aircraft states may be accurately captured. All threats and errors seen by the

18
 EBT Fundamentals

observer are captured in addition to the competencies used to manage them. Also, and very importantly,
all contextual factors (e.g., weather, time pressure, etc.) are recorded, and the “Whys” and “Hows” missed
by FDA, are also observed. The principal disadvantage is a relatively high human resource requirement.
For the purpose of training enhancement and review of effectiveness, flight deck observation may pro-
duce a valuable source of information. Of course, a significant number of observations needs to be made
to derive any meaningful results.

Participation in data sharing groups

There are opportunities to share relevant operational and training data between operators. The relevance
of data from other operators depends on the similarity of aircraft types, destinations, training programs
and other factors. While some of such data may be valuable, care must be taken not to drive the training
program too extensively based on such external data. Furthermore, aircraft manufacturers share infor-
mation on fleet-wide trends and individual events of concern. Such information may be very useful to the
operators of the aircraft type/family in question.
As an over-arching report, the EBT Data Report (1st Edition, and its amendment 2021) might primarily be
used to consider outer loop data for the EBT program and secondly establish a common, worldwide EBT
framework.

2.11.4 Summary
Data collection and analysis generally needs to cover various types of data, both from within the training
activity and from the flight operations and SMS. Data analysis can be as simple as analyzing the opera-
tor’s mission and making sure that operator-specific threats are accounted for in the training program.
Alternatively, the analysis may be carried out using sophisticated flight data analysis software.

Procedures and guidelines on how to analyze safety and training data can be found in Section 4, Step 13:
Measurement of training system performance.

 19
 Evidence-Based Training Implementation Guide

2.12 Feedback loop

EBT implicitly provides dynamic, effective, and efficient programs because it respects the instructional
system design (ISD) principles. In particular, the ADDIE (Analyze, Design, Develop, Implement and Evaluate)
model ensures that the training program is adapted to the organization and trainees’ needs while making
best usage of training media and devices.
The EBT program targets to deliver competent trainees (Workflow 4) that have demonstrated the achieve-
ment of the organization’s competency standard.
The achievement of the organization’s competency standard should permit to reliably predict that the
trainees conduct safe and efficient operations after the course delivery. Consequently, the consistent
demonstration by a cohort of trainees of an adequate level of performance across all the competencies of
the adapted competency model should be the basic criteria to evaluate the effectiveness of the course.
Specifically, Workflow 5 generates a “feedback loop” that sustains a continuous enhancement of the
course.

In particular, the Workflow 5 process mandates the analysis of the courses results, which includes the
trainees’ performance data based on training metrics. Therefore, at the end of each EBT module, feed-
back on training performance is gathered to determine the effectiveness in supporting the progression of
learning towards competence in the operational environment.
Furthermore, EBT programs use operational and safety data to adapt the training program. For details,
refer to Section 4, Step 14: Continuous improvement of the training program.

20
Section 3 - Implementation Overview
This section provides an overview of a possible pathway to implement an EBT program. The proposed
pathway is a step-by-step approach for an AOC/ATO to transition from traditional recurrent training and
checking to EBT. This pathway is recommended because EBT represents an important shift in terms of
training philosophy, also because EBT involves a significant amount of resources for its successful im-
plementation. Therefore, the following paragraphs describe the steps that should permit the AOC/ATO to
progressively introduce EBT within the organization, ensuring a planned allocation of resources and facili-
tating the adoption by the workforce (pilots, instructors, courses developers, etc.).
Note: The Steps ensuring the progressive implementation of EBT are presented in a template at the end
of this section and Section 4 provides the details on each implementation step.

3.1 Baseline EBT requirements

This paragraph illustrates the magnitude of the transition from traditional recurrent training and checking
to EBT, by presenting the requirements for the baseline EBT implementation and the actions to be con-
ducted by the organization for the program to be approved by the Authority.
The minimum requirements considered necessary prior to any implementation of EBT are as follows:
(a) Implementation of an adapted competency model
(b) Development of the EBT program, by distributing the training and assessment topics over a period of
36 months
(c) Training and assessment of the EBT IEs, including initial and recurrent standardization and inter-rater
reliability assurance
(d) Providing information to pilots regarding EBT principles, methodology and the level of performance
expected to be demonstrated across the pilot competencies
(e) Ensuring the availability of a measurement of training system performance
Before implementing EBT, the AOC/ATO should ensure that it meets the requirements specified by the
CAA for the implementation of EBT.
There are various mechanisms for the implementation of an EBT program, which should be conducted in
close consultation with the CAA, which include:
• The definition of an implementation and operations plan
• Review of the training effectiveness before and after the EBT implementation
• The adaptation and approval of the EBT program, in compliance with Appendix G, according to the gen-
eration of aircraft (fleet) and type of operation for the operator
• The EBT program implementation
• The adjustment and continuous improvement of the training program according to the training system
feedback
Note: Appendices D and E to the Manual on the Approval of Training Organizations (Doc 9841) also
provide guidance to ATOs and CAAs on managing risk while planning for and implementing a proof-of-
concept trial.

3.2 Staged implementation

Stage 0 – Traditional recurrent training and checking programs
In traditional recurrent training and checking FSTD programs, operators conduct mainly task-based
training and assessment. As most of the regulations (based on Annex 1 and Annex 6 provisions) mandate
the completion of two operator proficiency checks per year and a license proficiency check per year, the

 21
 Evidence-Based Training Implementation Guide

yearly operator recurrent training and checking patterns are often composed of two FSTD sessions every
six months. These training and checking FSTD sessions are often scheduled as consecutive days, with the
first session being a training session mainly dedicated to preparing the second session, which is dedicat-
ed to the evaluation (License and/or Operator Proficiency check).
The following schematic presents traditional recurrent training and checking FSTD sessions that are con-
ducted approximately every six months.

Stage 1 - Introducing EBT principles within traditional recurrent training and

checking
The introduction of the EBT principles means that the operator starts to implement some elements of EBT
within its current traditional recurrent training and checking program. This means a reversion of the se-
quence of the FSTD training and checking sessions.
The introduction of the EBT principles is ensured by the following:
• FSTD Session 1 as proficiency check, and FSTD Session 2 dedicated to training, to develop the pilot’s
performance, based on individual strengths and weaknesses identified during Session 1
• The implementation of an adapted competency model to enable a more effective outcome even when
using existing program elements
• The expansion of the facilitation technic utilization by the instructor to increase the involvement of the
pilots in their learning process
The introduction of the EBT principles does not require any regulatory amendment nor specific approval
as the operator continues to comply with the traditional training and checking requirements. However, the
operator starts to implement competency-based pilot assessment in addition to the regulatory require-
ments for license/operator proficiency checks.
For the introduction of the EBT principles, the operator should conduct the following steps of the imple-
mentation process detailed in Section 4:
• Step 3: Development of an adapted competency model
• Step 5: Standardize the IEs regarding the adapted competency model content, the competency-based
assessment process and the methodology, and the facilitation as preferred instructional method
• Step 4: Inform the pilots about the adapted competency model and associated performance criteria
During the introduction of the EBT principles, the operator may also consider the following to facilitate the
steps towards baseline EBT implementation:

22
 Implementation Overview

• Step 0: Step 0: Information session(s) of the stakeholders

• Step 1: Safety Risk Assessment regarding the introduction of the adapted competency model
• Step 2: Step 12: Review of training effectiveness
The conduct of the traditional training program should integrate the following EBT program elements (see
Chapter 5 for more details):
• Competency-based assessment
• Briefing and debriefing using the facilitation technique
In accordance with EBT principles, after the FSTD Evaluation Session 1 (dedicated to proficiency check),
the operator should implement a procedure to tailor the content of the FSTD Session 2 dedicated to
training. The content of the FSTD Session 2 should be based on the performance demonstrated during
Session 1. This procedure should support the IE in further developing the pilot competencies.
The following schematic illustrates the introduction of EBT principles within traditional recurrent training
and checking FSTD sessions, which are conducted approximately every six months.

The change of sequence between the sessions dedicated to training and evaluation should be commu-
nicated carefully to the pilot community to ensure they understand that the intent is to move away from
“training for the check” to “evaluation for competency development”. Experience has demonstrated that
this change provides a good opportunity to build trust among the pilots’ community.

Stage 2 - Mixed EBT implementation

Mixed implementation of an EBT program means that EBT program elements are progressively applied, to
the maximum extent permissible, under existing rules. The introduction of the EBT principles, as described
in the previous paragraph, may be a preamble (or a part) of the mixed EBT implementation. EBT mixed im-
plementation is a way to transition from traditional training and checking to the baseline EBT program.
Mixed implementation also recognizes the potential for an EBT program to be developed and implemented
in advance of any future enabling regulatory changes, which may then permit full implementation.
The revalidation or renewal of pilot licenses is performed in accordance with the traditional licensing and
operator checking regulations.

 23
 Evidence-Based Training Implementation Guide

To start the mixed EBT implementation, the operator should conduct the following steps of the implemen-
tation process detailed in Section 4:
• Step 0: Information session(s) of the stakeholders
• Step 1: Implementation plan and safety risk assessment
• Step 2: Pre-EBT training effectiveness review
• Step 3: Development of an adapted competency model and grading system
• Step 4: Communication plan
• Step 5: EBT IE initial standardization
• Step 6: EBT IE’s competency assessment
• Step 9: Assessment and training topics distribution over a 3-year period
• Step 10: Module design
• Step 11:Implementation of the EBT program
During mixed EBT implementation, the operator should progressively implement the following remaining
steps necessary to reach baseline EBT:
• Step 7: Definition of aircraft system malfunctions equivalency
• Step 8: Definition of approaches equivalency
• Step 12:Review of training effectiveness
• Step 13:Measurement of training system performance
• Step 14:Continuous improvement of the training program
• Step 15:Implementation of the recurrent IE standardization
Under mixed EBT, one of the challenges is to set up EBT modules with the three phases (EVAL, MT, SBT) in-
cluding their associated assessment and training topics, and to combine them with the significant number
of mandatory items to be performed during traditional license and operator proficiency checks.
This challenge may be overcome by considering that a certain amount of the assessment and training top-
ics during the MT are identical with some proficiency check items. In this case, the phase may be consid-
ered as MV, which maintains the same purpose, that is, to train and assess the handling skills necessary to
fly critical flight maneuvers so that they are maintained to a defined level of proficiency. These assessment
and training topics should place a significant demand on a proficient pilot. Handling skills within this defini-
tion refer to the competencies FPA, FPM and PRO.
This method allows a smooth transition from traditional proficiency checks to the baseline EBT program,
as the definition and training topics of the MV and MT are almost identical.
Note: European EBT regulations require that operators implement mixed EBT for three years before
moving to baseline EBT. EASA also provides a detailed checklist for mixed EBT implementation on its
EBT-related website.
The following schematic illustrates the mixed EBT implementation of FSTD sessions that are conducted
approximately every six months.

24
 Implementation Overview

Stage 3 - Baseline EBT

Implementation of a baseline EBT program means that the recurrent training and assessment is fully
dedicated to the application of EBT, and the revalidation or renewal of the pilot’s licenses is accomplished
in accordance with EBT regulations described by the CAA. The EBT module should be the means by which
Licensing Authorities continue to ensure that competence is maintained to hold a professional license and
type rating as applicable.
The baseline EBT program is a generation-specific, ready-made program. It does not require detailed anal-
ysis or program design by the operator or the ATO. It only needs the necessary adaptation to aircraft type
and operation, and the development of an adapted competency model, and a grading system. The opera-
tor should address and comply with all the requirements indicated/listed in Chapter 3.1.
In baseline EBT, the training phase starts already after the evaluation phase. MT and SBT may be delivered
in any order; however, IATA recommends the sequence EVAL – MT – SBT to allow the instructor a deeper
insight into the manual and automatic flying skills before adapting the training for the SBT phase.
The following schematic illustrates the scheduling option chosen by most operators for Baseline EBT im-
plementation of FSTD sessions, which are conducted approximately every six months.

 25
 Evidence-Based Training Implementation Guide

The following template shows the progressive implementation of the EBT elements to achieve the baseline
EBT status.

Introducing Start
Start
Step Scope EBT Mixed
Baseline EBT
principles EBT
0 Information session(s) X X
1 Implementation plan and safety risk assessment X X
2 Pre-EBT review of training effectiveness X X
3 Development of an adapted competency model and asso- X X X
ciated grading system
4 Communication plan X X X
5 EBT Instructor/Evaluator initial standardization X X X
6 EBT Instructor/Evaluator competency assessment X X
7 Equivalency of Malfunctions X
8 Equivalency of Approaches X
9 Distribution of assessment and training topics over a X X
3-year period
10 Module design X X
11 Implementation of the EBT program X X
12 Review of training effectiveness X
13 Measurement of training system performance X
14 Continuous improvement of training program X
15 EBT IE recurrent standardization X

3.3 Enhanced EBT

An enhanced EBT program is an EBT program where the assessment and training topics matrix contained
in Appendix G.IV is adapted. Adaptation may be necessary to:
• Mitigate a specific operational risk identified by the Authority, operator or OEM
• Maintain the suitability of the EBT program
Any adaptation must be approved by the CAA, and duly sustained by a safety case.

26
 Implementation Overview

The baseline EBT program is based on a robust methodology and reliable worldwide data acquisition and
analysis. An enhanced EBT program requires the same level of operator data robustness as described in
the EBT Data Report and its amendment(s). Due to the lack of reliable data for enhanced EBT, presently no
guidance for enhanced EBT programs can be given to operators and CAAs. However, this does not pre-
clude operators from collaborating with their CAA to enhance EBT.
When enhancing an EBT program, it is very important to focus on the most critical operational risks, pro-
vided that training can demonstrably mitigate these. There should be a close correlation between training
and operations.
An identical adapted competency model is recommended for both baseline and enhanced EBT programs.

 27
Section 4 - Implementation Process
The content of this section provides practical guidance for an organization to transition from traditional
recurrent training and checking to EBT.
The transition to EBT has a significant impact on the training policy and procedures, the program design,
the qualification of the personnel and potentially the training tools. Therefore, EBT concepts and principles
should be understood by the operational staff of the organization (training managers, safety officers, etc.),
and the high-level management of the organization (e.g., Accountable Manager) should commit to provide
significant resources for the transition to EBT.
The transition to EBT represents opportunities in terms of safety benefit and training efficiency but also
internal challenges (expertise, resources, communication…), and challenges for external stakeholders
(oversight resources, service providers availability…), which should be addressed by a specific EBT proj-
ect team nominated by the Accountable Manager (or the nominated post holder flight crew training) of the
organization.
The following template provides the essential steps to be followed during the transition process towards
EBT. Some steps are sequential, and some can run in parallel enabling the most efficient implementation of
EBT. Each step is further detailed in this section.

Step Scope Parties involved

0 Information session(s) AOC/ATO, pilot representatives, CAA
1 Implementation plan and safety risk assessment AOC/ATO, CAA
2 Pre-EBT review of training effectiveness AOC/ATO
3 Development of an adapted competency model AOC/ATO, pilot representatives, CAA as
and associated grading system appropriate
4 Communication plan AOC/ATO, pilot representatives
5 EBT IE initial standardization AOC/ATO
6 EBT IE competency assessment AOC/ATO, CAA
7 Equivalency of Malfunctions AOC/ATO, OEM, CAA as appropriate
8 Equivalency of Approaches AOC/ATO
9 Distribution of assessment and training topics AOC/ATO Course developers
over a 3-year period
10 Module design AOC/ATO Course developers
11 Implementation of the EBT program AOC/ATO, CAA
12 Review of training effectiveness AOC/ATO, CAA as appropriate
13 Measurement of training system performance AOC/ATO, CAA
14 Continuous improvement of training program AOC/ATO, CAA
15 EBT IE recurrent standardization AOC/ATO, CAA

Note: If an organization transitioning to EBT elects to be supported by a service provider for any of the
following steps, it should consider recognized and accredited entities that demonstrate the compliance of
their methodology to international standards.

Step 0: Information session(s) of the stakeholders

Description: The EBT project team provides awareness about the transition to EBT.

Before starting any implementation measures, the operator/ATO shall ensure that all relevant stakeholders
are informed about the intent to introduce EBT in pilot recurrent training and assessment.

28
 Implementation Process

The objective of the information session(s) is to ensure proper awareness of internal and external stake-
holders about the operator plans for the transition from legacy recurrent training and checking to EBT.
Information session(s) are necessary because successful implementation of EBT depends on an effective
partnership between:
• Operator-internal stakeholders such as safety, operations and compliance managers, and pilot repre-
sentatives in addition to the staff involved in the development and delivery of training and of the pilot
population
• Operator-external stakeholders such as the Competent Authority in charge of the operator oversight
The information session(s) content should permit to share the operator’s vision, including the challenges
and the benefits of EBT implementation, with internal and external stakeholders.
It is also recommended that the EBT project team sets up regular information sessions with the internal
and external stakeholders to inform them on the project’s status. The duration and the frequency of these
sessions should be adapted to the audience. They should be consistently aligned with the communication
plan and the EBT IE standardization elaborated under Steps 4 and 5.

Step 1: Implementation plan and safety risk assessment

Description: The EBT project team formalizes the EBT transition plans and addresses the management of change.

Implementation plan
Once a decision has been taken by the AOC or ATO to implement EBT, a consultative document should be
created in cooperation with the CAA, defining the objectives, timelines and any limitations based upon ex-
isting rules and risk management processes. This can be agreed according to the options described within
the guide, for staged or total implementation, fleet-wide or operation-wide, or as a program that runs in
parallel to components of existing training. Agreement about the program principles should be reached
before detailed planning commences.
Before implementing the Baseline EBT program, a minimum experience with the application of the CBTA
principles should be assured. This experience may include Mixed EBT. A staged implementation, as de-
scribed in Section 3, should be considered.
Note: European Baseline EBT regulations require a minimum experience of:
• 3 years of a mixed EBT program
• 2 years of an instructor concordance assurance program
• 1 year of a valid equivalency of malfunctions
• 1 year of integration of training data in the customization of the EBT program and SMS data for the
contextualization of the example scenario elements
The EBT project team should provide the implementation plan, including milestones and timeframes, to
the CAA in charge of the approval. The implementation plan should be the result of an organizational gap
analysis to move from traditional recurrent training to EBT in the following main areas:
• Awareness of management
• Qualification of personnel
• Documentation including training policy and procedures
• Training tools, equipment, and material necessary to deliver the training and collect the training data

Safety Risk Assessment

The gap analysis should be complemented by a safety risk assessment (SRA) conducted in accordance
with the operator management system process. The SRA entails that the organization develops and main-
tains a process that ensures analysis, assessment and control of the safety risks associated with identi-
fied hazards.

 29
 Evidence-Based Training Implementation Guide

Examples of considerations that would form part of a risk profiling exercise are:
• Selection and training of management and staff
• Training program development, application for approval, validation, and continued review
• Development and maintenance of training courseware
• Administrative staff duties in support of the training program, the instructors/evaluators and pilots
• Delivery of training
• Record-keeping
• Grading and assessment processes
• Licensing authority feedback
• Customer feedback, where applicable
As result of the SRA, the implementation plan may include the option to return to traditional recurrent
training and checking if the implementation of EBT is cancelled.
Note: The risk assessment is a living document and may be amended throughout the project.

Generic SRA template

The following example uses the recommendations of ICAO Doc 9859 (Safety Management Manual) to per-
form the risk assessment and specifically to rate the risk.
Note: Appendix A provides further SRA examples.

Example of a safety risk matrix

30
 Implementation Process

Practical illustration of how to perform an SRA

This illustration is based on the following assumptions:
• The organization is an operator that has already implemented a competency framework which is used
by the IE to support the pilot performance assessment in addition to the regulation requirements
based on pass/fail criteria.
• The operator plans to implement CBTA for its recurrent training under the umbrella of EBT.
• The operator has standardized the IEs to perform pilot competency assessment using the opera-
tor competency framework. The operator has not established any inter-rater reliability assurance
program.
A) SRA regarding the competency framework
Step 1 is to perform the gap analysis based on the operator’s current status regarding the competency
framework.

Gap analysis
Component ICAO provisions Current Status Compliance
Implementation of pilot ICAO Doc 9868, The competency frame- Partial
competency framework • Part I, Chapter 2 para- work used by the operator
graph 2.4 is partially aligned with the
ICAO one
• Part II, section
1, Chapter 1 and
Appendix 2 to
Chapter 1
ICAO Doc 9995

Step 2 is to perform the Safety Risk Assessment based on the gap analysis.

Risk assessment PRIOR mitigations

Hazard Consequence Existing Probability Severity Risk
controls
The training data Tailored and remedial NIL 4-Occasional D-Minor 4D Moderate
is not relevant to training inadequately risk
ensure appropriate implemented due to
inner loop data mismatch between
the ICAO compe-
tency framework
(training topics/com-
petency mapping)
and operator compe-
tency framework

Step 3 is to define mitigation measures to reduce the risk level to acceptable.

Risk assessment WITH mitigations

Mitigation Ownership New Control Probability Severity Residual Risk
Implementation of Training Manager Consistency 1- Extremely D-Minor 1D Low risk
the ICAO competen- of the Training improbable
cy framework data collection
and the ICAO
training topic
mapping

 31
 Evidence-Based Training Implementation Guide

B) SRA regarding the instructor standardization

Step 1 is to perform the Gap Assessment based on the Operator current status regarding the instructor
standardization.

Gap analysis
Component ICAO provisions Current Status Compliance
Instructor selection, initial ICAO Doc 9868, The instructors have Partial
training, recurrent training • Part I, Chapter 3 received initial training to
and inter-rater reliability perform pilot’s compe-
• Part II, section 1,
(IRR) requirements tency assessment using
Chapter 7
operator competency
ICAO Doc 9995 framework

Step 2 is to perform the Safety Risk Assessment based on the gap assessment.

Risk assessment PRIOR mitigations

Hazard Consequence Existing Probability Severity Risk
controls
Inappropriate initial Non-competent Traditional 4-Occasional B- Hazardous 4B High risk
and recurrent train- pilot conducting line Operator profi-
ing, lack of inter-rater operations ciency checks
reliability assurance
program

Step 3 is to define mitigation measures to reduce the risk level to acceptable.

Risk assessment WITH mitigations

Mitigation Ownership New Control Probability Severity Residual Risk
Progressive imple- Training Manager Initial and re- 1-Extremely B-Hazardous 1B Low risk
mentation of training current IE stan- improbable
data collection and dardization and
analysis including IRR assurance
instructor perfor- program based
mance data on evidence

Step 2: Pre-EBT training effectiveness review

Description: The EBT project team commits the Nominated Person Crew Training/Head of Training to report on the
training effectiveness of the organization before EBT implementation.

The Nominated Person Crew Training/Head of Training, supported by the training managers, analyses the
existing training data to report the current training effectiveness of the organization.
The review of the current training effectiveness is necessary for both internal and external stakeholders
and specifically the Competent Authority.
Providing visibility on the training effectiveness permits the AOCs/ATOs to:
• Raise awareness of the internal stakeholders on the overall performance of the training organization
and, where applicable, identify the strengths and weaknesses of the current training system
• Provide the accurate status about the organization’s current training system performance to the
Competent Authority and consequently set the bottom-line in terms of training system performance to
be enhanced during the EBT implementation
The main challenge for the training effectiveness review under traditional recurrent training and checking
programs relates to the quality of the training metrics. From a regulatory perspective the requirements,

32
 Implementation Process

applicable to both AOCs and ATOs, mandate to collect and keep records for a limited period, such as the
pilots’ training, checking and qualification (e.g., AOC - 3 years).
Generally, these records provide a macroscopic view with limited insights into the pilot’s performance, as
the traditional training metrics are based on Pass/Fail criteria for each task and maneuver realization.
To assess the training effectiveness under traditional recurrent training and checking program, the follow-
ing indicators should be considered:
• Failure rates during proficiency checks
• Rate of successful training objective achievement
• Volume of remedial training, etc.
These indicators have been considered sufficient to set the bottom-line before the implementation of an
EBT program.
Organizations having implemented Advanced Qualification Program (AQP), or Alternative Training and
Qualification Program (ATQP) should get a better visibility on their training effectiveness, as the training
metrics contain behavioral markers that provide more human performance insights, also because AQP/
ATQP contains quality assurance components sustaining the monitoring of the program’s effectiveness.

Step 3: Development of an adapted competency model and grading

system
Description: The EBT project team defines the content of the adapted competency model in collaboration with
training, operation and safety managers.

Adapted Competency Model for EBT

IATA recommends that the AOC/ATO use the IATA competency frameworks for aeroplane pilots and for
pilot instructor/evaluator described in Appendices B.I and B.II for the training and assessment of the pilot
receiving EBT and the IE delivering EBT.
IATA recommends that the AOC/ATO characterize the conditions under which the competencies are
trained and assessed depending on the phases of the EBT module.

Conditions related to tools and systems/equipment, airplane, FSTD

During an EBT module, the training platform is an ICAO FSTD level 7 (Full Flight Simulator).

Conditions related to the amount of support or assistance a trainee can expect

from the IE
Instructor support should vary between the evaluation and the training phases of the module:
• During the EVAL phase, the IE should not normally give any instruction to the pilots or interrupt
• During the MT phase, the IE should provide support, where needed, to ensure that the pilot demon-
strates the motor skills necessary for the successful realization of the maneuvers
• During the SBT phase, the IE should apply all the instructional methods to ensure that the pilot effi-
ciently develops the competencies
The template below proposes the instructional methods to be applied by the IE during an EBT module
depending on the phase.

 33
 Evidence-Based Training Implementation Guide

EBT Module Phase Evaluation Phase (EVAL) Training Phases (MT/SBT)

Level of support from the N/A Low to High
EBT IE
Instructional Method The IE provides guidance for the conduct All instructional methods depending on
of the flight profile of the EVAL the pilot’s needs

Conditions related to context (nature and complexity of operational and

environmental context)
According to the training objectives of the evaluation phase and the scenario-based training phase, IATA
recommends applying the following:
• The context under which the competency is demonstrated during the EVAL phase of an EBT Module
should be defined and limited to zone III, and
• The context under which the competency is demonstrated during SBT of an EBT Module should be
extended but limited to zone IV

Note: The grey zones are not used because either not challenging enough (zone I and II) or too
demanding (Zone V).

Example of conditions applied to all IATA pilot competencies for an EBT EVAL phase
The pilot should demonstrate an integrated performance of all the competencies under the following conditions:
Conditions related to the context:
Context consistent with the operator’s network and sustaining a realistic application of both normal and abnormal
procedures under the following combination of complexity of operational and environmental context:
• High operational context complexity (e.g., malfunction placing significant demand on the flight crew) and low
environmental complexity (e.g., VMC conditions with maximum cross wind 15 Knots), or
• Medium operational context complexity (e.g., Cabin event: O2 mask deployed) in a medium environmental com-
plexity (frequent CB during the phase of flight), or
• Low operational context complexity (e.g., MEL item with no impact on performance nor procedures); and
high environmental complexity (severe icing conditions, runway contamination having significant impact on
performance)
Conditions related to tools and systems/equipment:
FSTD ICAO level 7
Conditions related to the amount of support from the IE:
Without assistance from the Instructor/Evaluator

34
 Implementation Process

Competency assessment and grading

Prior to the introduction of EBT in the operator’s training program, a detailed description of the training and
assessment methodology (including standard terminology to be used) acceptable to the CAA should be
published in the Operations Manual. This methodology should include procedures to be applied if pilots do
not achieve the required level of performance over the EBT modules (refer to Step 11 for more details).
IATA’s recommended competency assessment methods are described in Section 2 and Section 5.
IATA recommends that the AOC/ATO require an “adequate” level of performance (corresponding to a grade
3) to be demonstrated by the pilot for each pilot competency at the end of the EBT module.
This recommendation aims at maintaining a reasonable margin against the minimal acceptable level of
performance to cover the case of a potential pilot competency decay between EBT modules and/or the
estimated probable instructor error of alignment with the operator standard during the pilot competency
assessment.

Step 4: Communication plan

Description: The EBT project team produces and implements a communication plan including specific information
to the IEs and the pilots.

As with any change management process, timely and accurate communication with the internal and exter-
nal stakeholders affected by the change is critical.
The general communication content should elaborate on the What is CBTA, the Why, the How and the
When of the transition to EBT. This communication should be adapted to the audience (the Who) that is
affected by the EBT implementation.
Consequently, the communication plan should cover the following items with different levels of granularity:
• EBT principles and objectives
• Relationship between Threat and Error Management, competencies, and crew resource management
• Description of the EBT modules and associated phases
• Expected value in terms of safety and efficiency of EBT versus the organization’s current training
programs
• Description of the Adapted Competency Model and associated grading system
• Performance assessment process and methodology
• EBT data collection and protection
Concerning the specific information towards the IEs, the general communication items listed above,
should be communicated in advance of the IE initial standardization to enhance its effectiveness and
efficiency.

 35
 Evidence-Based Training Implementation Guide

Concerning the specific information towards the pilots, the general communication items listed above,
should be communicated in advance. Pilots generally accept that training processes evolve over time,
however, a lack of information or uncertainty as to what to expect can lead to a lack of acceptance. It is
essential that pilots who will be trained and assessed according to CBTA principles understand all the
processes involved and are given time to adjust to the new performance requirements. The CBTA project
team should coordinate the communication with pilot representatives to enhance the understanding and
acceptance of this fundamental change in the training system.
Refer to Appendix C for an example of a communication plan template.

Step 5: EBT IE initial standardization

Description: The Standardizers* deliver the IE initial standardization. The EBT project team monitors this activity.

*Note: For the purpose of this guide, the standardizers are the IEs in charge of the training and assess-
ment of other IEs.
IEs play a critical role in the delivery of EBT. They are essentially a link between the organization and the
trainees in the delivery of the training program. As such, it is critical that suitably qualified IEs are selected
and trained to deliver EBT.
As EBT programs expand, many already qualified IEs under the traditional training system should transition
to EBT program delivery. These IEs should complete initial and recurrent standardization to ensure safe
operations and effective application of the EBT methodology.
The IATA Guidance Material and Best Practices for IE Training contains practical details for the design of
the IEs initial and recurrent standardizations.

Selection
The selection process should consider the following:
• The level of performance as a pilot
• The potential to support the trainee’s learning development
• The potential to become a role model
• The potential to successfully complete the IE training

Standardization
The competency-based training and assessment for IEs should consist of an integrated program of theo-
retical and practical instruction.
The training objectives of the theoretical and practical training should be to develop the IE competencies
(as described in Appendix B) and to acquire knowledge of EBT principles, with special emphasis on the
following:
• Threat and Error Management
• Learning from positive performance
• Building resilience
• Data driven training
• Principles of adult learning and how they relate to CBTA
The IE standardization should place special emphasis on the instructor competencies “Instruction” and
“Assessment and Evaluation”. It should include training covering the importance of using Observable
Behaviors (OBs) for effective training and assessment, the ability of the IE to analyze the root-cause
for high and low trainee performance and the application of facilitation during the conduct of the EBT
modules.

36
 Implementation Process

IATA experts have applied the Instructional System Design methodology to design the IE initial standard-
ization with the assumption that the candidates are already qualified IEs under the traditional training sys-
tem. According to the training need analysis, it has been identified that the training objectives of the EBT IE
initial standardization program, for IEs transitioning from traditional training to CBTA, are sustained by the
OBs related to the competencies “Instruction” and “Assessment and Evaluation”.

“Special Emphasis” for Traditional IE moving to CBTA (including EBT)

IEC1 IEC2 IEC3 IEC4 IEC5
Pilot competencies Management of the Instruction Interaction with the Assessment and
learning environment trainees Evaluation
TA TA TA-SE TA TA-SE
Note: TA: Means competencies trained and assessed
SE: Means competencies requiring special emphasis during training

The table below proposes, as a guidance for the course developers, the OBs in bold that permit to sustain
and adjust the training objectives for the EBT IE initial standardization.
Placing emphasis on the competency assessment procedures and methods should also ensure inter-rater
reliability.

Competency Description Instructor Observable Behaviors (IOB)

OB 3.1 References approved sources (operations and technical sources, train-
ing manuals and regulations)
OB 3.2 States clearly the objectives and clarifies roles for the training
OB 3.3 Follows the approved training program
OB 3.4 Applies instructional methods as appropriate, (e.g., explanation,
demonstration, learning by discovery, facilitation, in-seat instruction)
Conducts train- OB 3.5 Sustains operational relevance and realism
IEC3: ing to develop OB 3.6 Adapts the amount of instructor inputs to ensure that the training
Instruction the trainee’s objectives are met
competencies
OB 3.7 Adapts to situations that might disrupt a planned sequence of events
OB 3.8 Continuously assesses trainee’s competencies
OB 3.9 Encourages the trainee to self-assess
OB 3.10 Allows trainee to self-correct in a timely manner
OB 3.11 Applies trainee-centered feedback techniques (e.g.: facilitation, …)
OB 3.12 Provides positive reinforcement
OB 5.1 Complies with Operator/ATOs and authority requirements
OB 5.2 Ensures that the trainee understands the assessment process
OB 5.3 Applies the competency standards and conditions
OB 5.4 Assesses trainee’s competencies
Assesses the
competencies OB 5.5 Performs grading
IEC5: of the trainee- OB 5.6 Provides recommendations based on the outcome of the assessment
Assessment Contributes OB 5.7 Makes decisions based on the outcome of the summative assessment
and Evaluation to continuous
training system OB 5.8 Provides clear feedback to the trainees
improvement OB 5.9 Reports strengths and weaknesses of the training system (training
environment, curriculum, assessment/evaluation) including feedback from
trainees
OB 5.10 Suggests improvements for the training system
OB 5.11 Produces reports using provided appropriate forms and media

 37
 Evidence-Based Training Implementation Guide

Course delivery
Detailed EBT IE initial standardization course requirements are described in Appendix D.I. Sample foot-
prints are shown in Appendix D.II.
AOCs/ATOs should ensure that all IEs conducting EBT are qualified in accordance with all applicable state
regulations. This includes IEs who are external to the operator, e.g., subcontracted IEs.
For ATOs/AOCs delivering an EBT program with subcontracted IEs, the subcontracted IEs should com-
plete the initial standardization, and achieve the same performance standard and inter-rater reliability as
the in-house IEs. Recurrent standardization and regular feedback loops between the operator and training
the organization are required to maintain the proper application of specific contents and training methods
of the operator.

Step 6: EBT IE’s competency assessment

Description: The Standardizer conducts the IE’s competency assessment before delivering EBT.

Prior to delivering mixed or baseline EBT, the IE should undergo an assessment of competence, conducted
during a practical CBTA session by a person nominated by the AOC/ATO and acceptable to the CAA.
Note: European EBT regulations require a specific EBT competency assessment prior to baseline
implementation. For mixed EBT, the competency assessment should be performed according to the
applicable regulations, prior to EBT implementation.
The IE should demonstrate the competency standard approved by the CAA, for all the competencies of
the adapted competency model.
For the “entry into service” of EBT, it is recognized that the first Standardizer(s) nominated by the op-
erator may not have had the opportunity to complete their assessments. Under these conditions, the
Standardizer(s) nominated by the operator should receive the EBT IE initial standardization (Step 5) and
have experience in the role of a Standardizer undertaking instructor competency assessments under
traditional training regulations.
The performance assessment process and methodology, as described in Section 2 and Section 5, are fully
applicable to the IE’s competency assessment.
In the context of the IE’s competency assessment, the OUTCOME of TEM dimension is applicable for
IE performance assessment.
In the context of training and licensing, the OUTCOME of TEM dimension integrates the specific threats,
errors and potential reductions of safety margins that could happen or result from the conduct of training
or evaluation activities.
In the context of training and licensing, potential threats could be:
• Event requiring an evacuation of the facilities or of the device
• Facilities, training device or equipment not appropriate for the training objective (Actual Malfunctions,
MEL, Device certification, ...)
• Training interruption or disruption (FFS down, ATC constraint, phone call, …)
• Any disruption that generates time pressure (late arrival of the trainee, trainee does not show up, train-
ing time reduced, …)
• Last minute change of training rostering (session content, trainees, …)
• Inappropriate official documentation (FCOM not up to date, training programs deficiencies, …)
• …
In the context of training and licensing, potential errors could be:
The IE:
• Does not prepare sufficiently the training session (not familiar enough with the training facilities ac-
cess, with the training device functionalities, with the IT procedures, …)

38
 Implementation Process

• Has hobby horses

• Does not manage time appropriately
• Does not manage priorities appropriately (e.g., during flight instruction focuses on instruction instead
of safety of the flight, …)
• Omits safety briefings elements or training tool limitations
• Intervenes inappropriately (too early or too late)
• Refers to personal customized documentation
• Uses inappropriate teaching method (does not facilitate, …)
• Generates unrealistic or inappropriate conditions for the training
• Demonstrates negative attitude towards the trainees (is careless, harsh, has bias, is lacking
empathy, …)
• Does not allocate enough time for trainee feedback
• Is not familiar with training policy and procedures
• Omits to provide necessary advice to improve performance
• Cuts corners with the training program
• Does not apply organizational performance standards
• …
In the context of training and licensing, potential reduction of safety margins could be:
• Limited and momentary reduction of the safety margin:
– Temporarily involuntary reduction of safety margin during training (e.g., mismanagement of a stall
exercise) recognized and timely recovered by the instructor
– Temporarily negative transfer of training, recognized and timely recovered by the instructor
– Temporarily negative training, recognized and timely recovered by the instructor
– …
• Unacceptable reduction of safety margins:
– Involuntary reduction of safety margins during training (e.g., mismanagement of a stall exercise) not
recognized or recovered too late by the instructor
– Negative transfer of training not recognized or not recovered by the instructor
– Negative training not recognized or not recovered by the instructor
– Incident or accident during training
– …
Note: When the training and assessment are conducted during flight operations (for example IE as a
trainee delivering instruction in an aircraft as an FI), the IE as trainer observes, as well, the IE as trainee
managing the threats, the errors, and the potential reductions of safety margins in the operational context.

Step 7: Equivalency of malfunctions

Description: The EBT project team commits a group of Subject Matter Experts (SMEs) to define equivalency be-
tween malfunctions according to EBT principles.

For the practical training in the management of aircraft system malfunctions, EBT moves away from group-
ing malfunctions according to ATA chapters, to organizing them according to the crew performance re-
quired to manage the malfunction.
The aircraft system malfunctions that place a significant demand on a proficient crew are organized by
reference to the following characteristics:
• Immediacy
• Complexity

 39
 Evidence-Based Training Implementation Guide

• Degradation of aircraft control

• Loss of instrumentation
• Management of the consequences
The following table shows the malfunction characteristics and the required crew performance to manage
the malfunctions.

Characteristic Description of required crew performance Examples

Immediacy System malfunctions requiring immediate and Memory items:
urgent crew intervention or decision. fire, smoke, loss of pressurization at high
altitude, failures during take-off, brake failure
during landing
Complexity System malfunctions requiring complex proce- Multiple hydraulic system failures, smoke
dures, i.e., that require recovery procedures with and fumes procedures
multiple options to analyze and/or multiple deci-
sion paths to apply.
Degradation System malfunctions resulting in significant degra- Jammed flight controls, certain degrada-
of aircraft dation of flight controls in combination with abnor- tion of fly by wire (FBW) control, flaps/slats
control mal handling characteristics, such as modification inoperative
of the normal pitch attitude during approach and
landing or reconfiguration of the flight control laws
or modes.
Loss of instru- System malfunctions that require monitoring and Unreliable primary flight path information,
mentation management of the flight path using degraded or unreliable airspeed, flaps indication, loss of
alternative displays such as temporary or per- fuel indications, etc.
manent loss of any flight-path-related parameter
displayed on the primary flight display (PFD), head-
up display (HUD) or navigation display (ND), includ-
ing loss of any setting capability of one of these
indications. It includes primary instrumentation to
monitor and manage primary aircraft systems.
Management System malfunctions that require extensive Fuel leak or fuel not usable, altitude/speed
of management of their consequences, i.e., that limitations, malfunctions with “deferred”
consequences significantly affect the flight crew standard task items in later flight phases.
sharing and/or the workload management and/or
the decision-making process during an extensive
period after managing the malfunction itself.

Process
The equivalency of malfunctions process should be undertaken by SMEs who hold or have held a type
rating on the aircraft type and who are standardized on the equivalency of malfunction process.
The process steps during the EBT program design can be summarized as follows:

Review aircraft system malfunctions

Normally the Abnormal Procedure Section in the OM-B/FCOM will provide sufficient information. All abnor-
mal/emergency procedures related to non-technical conditions (e.g., overweight landing, bomb on board)
may be discarded. For most OEMs these procedures are listed in the MISC section of the OM-B/FCOM.
The remaining procedures will be provided to the group of SMEs selected for the process.

40
 Implementation Process

The list of malfunctions provided by the FSTD may provide malfunctions not listed in the OEM Abnormal
Procedures. Nevertheless, the operator may decide to add these malfunctions. This will allow for the train-
ing of system malfunctions where no clear procedure guideline exists (e.g., an un-annunciated IR drift is
provided in the FSTD, but not in the OEM documentation).

Determine significant demand and characteristics

The SME should determine:
• The malfunctions that place a significant demand on a proficient crew, in isolation from an environmen-
tal or operational context
• The applicable characteristic or characteristics for the malfunctions that place a significant demand on
a proficient crew

Note: Based on ICAO and the European EBT regulations, as well as best practices developed by operators and
OEMs, the following guidance should be applied:
• Isolation from an environmental or operational context. When assessing significant demand on a proficient
crew, SMEs may consider that there are no significant environmental and operational threats. For example, the
aircraft is close to a suitable aerodrome with environmental conditions permitting all published approaches to
be made, no ATC restrictions, with no pre-existing malfunctions and sufficient fuel for several hours.
E.g., European EBT regulations require that whenever the possibility of icing is specified in the abnormal/emer-
gency procedure, it shall be assumed that this meteorological condition is present (e.g., in case of pitot heating
failures).
• Persistence of malfunction. Consider that the abnormal/emergency condition will remain when steps to fix the
malfunction are included in the malfunction procedure (e.g., the failed engine will not restart, or the fuel pump
remains failed after the reset, or the electric generator is not fixed after the reset, etc.). When a reset puts a
significant demand on a proficient crew, then both options should be included in the malfunction clustering and
therefore the same malfunction should be evaluated for both cases: successful reset/restart and unsuccessful
reset/restart.
The operator should reproduce the malfunction in the FSTD program in the same way (no restart of the engine,
or successful reset of the fuel pump or electric generator) in order to meet the characteristics’ assumptions.
The operator may include successful resets or restart in addition to the malfunctions considered for the
characteristics.
• Worst case consideration. If the worst result of a system malfunction is addressed in a different (second) proce-
dure, this worst case shall be assessed in that second procedure and not in the originating malfunction (e.g.: if
the unreliable airspeed procedure is referenced in the blocked pitot procedure, unreliable airspeed shall NOT be
considered for the pitot blockage).

Incorporate malfunctions
Develop the EBT FSTD program to incorporate malfunctions at the frequency specified in the table of as-
sessment and training topics (Appendix G.IV).

Practical training in the management of aircraft system malfunctions

Each pilot shall receive assessment and training in the management of aircraft system malfunctions. The
aircraft system malfunctions to be considered for the EVAL and SBT phases are those that place a signif-
icant demand on a proficient crew. At least one malfunction with each characteristic should be included
every year. Combining characteristics should not reduce the number of malfunctions below four, for each
crewmember, every year according to the EBT module cycle.

Procedural training in the management of aircraft system malfunctions

Some malfunctions meeting the significant demand criteria may not be included in the EBT FSTD program.
These malfunctions require procedural knowledge training to ensure that the pilot demonstrates adequate
knowledge of the relevant procedures, systems and their interaction. Training may be conducted in a less
qualified immersive but suitable alternative environment (classroom, flight procedures training device,
advanced computer-based training, etc.).

 41
 Evidence-Based Training Implementation Guide

Note: The operator should establish procedures to determine what malfunctions should be included in
the FSTD in accordance with the equivalency of malfunctions criteria. This may include a different
malfunction difficulty between the EVAL and the SBT.

Methodologies
The SMEs should apply one of the two following methods to ensure consistent determination of the mal-
function characteristics and of an appropriate significant level of demand of the malfunctions.

Characteristics-oriented method
1. The SMEs review aircraft system malfunctions provided in the official documentation of the OEM.
2. A group of minimum three SMEs rates individually each of the five characteristics in each malfunction
listed in point 1:
– The rate will be 0 when the characteristic is irrelevant for the malfunction.
– The rate will be 1 to 5 when the characteristic is relevant for the malfunction. Rate 1 applies when
the characteristic is hardly relevant for the malfunction and rate 5 when the characteristic is highly
relevant.
3. An average rate of the group of SMEs is calculated for each characteristic of each malfunction.
4. The average calculated in point 3 is communicated to the group of SMEs. The group of SMEs agrees on
a final rate for each characteristic of each malfunction.
5. The SMEs select the rate of the characteristics (e.g., rate 2 or 3) at which or above which the charac-
teristic is considered to be relevant in the malfunction, i.e. placing a significant demand on a proficient
crew.
6. The SMEs may use the rates of the characteristics to determine the level of demand placed on a profi-
cient crew to manage the malfunction.
7. The SMEs may refer to an aircraft OEM malfunction analysis to support all the steps of the session.

OEM-based method
1. The OEM provides equivalency of malfunction documentation, based on aircraft system malfunctions
provided in the official OEM documentation, with lists of:
– The malfunction characteristics, and
– The malfunctions placing a significant demand on a proficient crew
2. A group of minimum three SMEs performs an analysis of the OEM equivalency of malfunction
documentation.
The group of SMEs agrees on an operator list of the malfunctions that combine a significant demand on a
proficient crew and the characteristic.

Step 8: Equivalency of approaches

Description: The EBT project team commits a group of Subject Matter Experts (SMEs) to define equivalency be-
tween approaches according to EBT principles.

A focus of EBT is to remove extraneous training for which there is little safety benefit or evidence of need,
and in particular those approaches that are regularly performed in line operations. Equivalency of ap-
proaches relevant to operations shall ensure training of approaches that place an additional demand on a
proficient crew.
Selection of approaches for scenario-based training should be based on the underlying elements of flight
crew performance to conduct them. Equivalent groups of approaches can be determined by reference
to these elements. Demonstrated competence in the conduct of one approach type can be considered
equivalent to demonstrated competence for the other approach types in the same group.

42
 Implementation Process

Process
The equivalency of approaches process should be undertaken by SMEs who hold or have held a type rat-
ing on the relevant aircraft generation and are standardized on the equivalency of approaches process.
The required process steps during the EBT Program Design can be summarized as follows:

Note: All steps should be performed for each aircraft type to allow type-specific training.

Review operational network

To identify approaches that place an additional demand on a proficient crew, the SMEs should review the
operator’s operational network.
Note: This should be implemented as a continuous process to assure an adaptation of the EBT program
to type-specific destination changes.

Determine additional demand and characteristics

The SMEs should select approaches with one or more of the following characteristics:

Characteristic Description Examples

Unusual approach and/or Non-standard approach aids and/or Offset final approach track
aerodrome design approach patterns Steep approach
Unusual DH/DA
Unusual aerodrome characteristics Non-standard lighting
or performance limitations or any Non-standard marking
other relevant attributes, including Optical illusions due to slope, unusual rwy
obstructions, physical layout, light- width
ing, etc.
Low frequency of Infrequently operated aerodrome Destination alternate aerodrome
exposure En-route alternate aerodrome
Infrequently conducted approaches Circling approach, NDB approach,
at commonly operated aerodromes CAT II operations, SA CAT I approach
Non-standard approach Non-standard internal guidance or Head-up display (HUD) disconnected
guidance aircraft equipment Flight directors off
Non-standard external guidance or
ground equipment

Incorporate approaches
Each pilot shall receive assessment and training in the management of approaches that place an addi-
tional demand on a proficient crew. The approaches selected via the equivalency of approaches process
should be embedded in the training topic of “Operation and type specific” at the frequency required by the
local CAA. Approach training may be performed in all phases of an EBT module.
Any approach that is required to be flown in the PF role specifically should be classified as “skills retention”
and may be trained in the MT.
Note: Aerodrome qualification requirements according to ICAO Annex 6 may be met by this training
topic.

 43
 Evidence-Based Training Implementation Guide

The operator’s policy generally defines which flight method should be used during line operations to con-
duct approaches. These recommendations should be followed by crews during EVAL. During SBT or MT,
it should be considered interesting to adapt the conduct of the selected approaches in order to develop
specific competencies.

Preamble to Step 9 and Step 10 – EBT program design general

The purpose of the EBT program is to use events defined to be most critical, as a means of assessing and
developing competencies. The Baseline EBT program should be constructed to contain all topics for the
aircraft generation listed according to following provisions:
Appendix G forms the basis for the construction of the EBT recurrent assessment and training program.
• G.III CBTA/EBT program overarching principles: describes topics applicable to all airplane generations
• G.IV Generation-specific assessment and training matrix: lists topics for each specific generation of
airplanes
• G.V Scenario elements and competency map: supports the module-specific scenario design, based on
given training topics and competencies to be assessed and/or trained
In order to address all assessment and training topics at the defined frequency, a training program of 48
FSTD hours over a three-year cycle for each pilot has been assumed. This EBT recurrent assessment and
training should be conducted in an FSTD qualified for the purpose.
When adapting material in Appendix G to specific operator’s needs, it is important to note that it is never
the intention that all possible events or scenarios be programed within the EBT recurrent cycle. Operators
should select the scenarios that are most useful to their needs, while ensuring that the frequency of de-
fined topics is maintained, to minimize competence decay over time.
The EBT program customization should be based on evidence sustaining the following three different
layers:
1. The first layer of customization is based on the evidence collected in operations (e.g., safety manage-
ment system, operator occurrences, state safety plan, international operational information, occur-
rences, manufacturer data, etc.) and concerns specifically the adaptation of the example scenario
elements attached to the training topics. Therefore, the EBT program is similar across the generations
for the entire pilot population, but the volume and the relevance of the scenario elements is enhanced.
This should permit to effectively contextualize the training topics and to better expose the pilots to
surprise and unexpected events.
2. The second layer of customization is based on the evidence collected in operations (see first layer
above) and on the evidence collected in the training context (training metrics collected and analyzed
by pilot population, per fleet, per aircraft type, per network, etc.). Therefore, the EBT module should be
adapted to the pilot population defined by the operator. This should permit to develop the pilot com-
petencies according to the needs identified within a specific pilot population on a particular fleet (e.g.,
long haul vs short haul), and/or depending on the role (e.g., Captains vs First Officers). This may include
different example scenario elements for each population.
3. The third layer of customization is based on the evidence collected in the training context (evidence
collected during the EVAL or historical training metrics of the pilot). Therefore, the training phases
(MT and SBT) of the module should be adapted to the individual pilot needs. This is the goal of EBT; to
develop the individual pilot competencies depending on the needs identified during the EVAL or from
previous modules. The tailorization to the individual’s needs is elaborated in Step 10.

44
 Implementation Process

Step 9: Assessment and training topics distribution

Description: The EBT project team commits the course developers to distribute the assessment and training
topics.

The EBT course developers should select scenarios and priorities according to the principles described
in the previous paragraph “EBT program design general” (Preamble to Step 9 and Step 10), combined with
any additional requirements, while respecting the assessment and training topic frequency.
The additional requirements may include, but are not limited to:
• Legal requirements not covered by EBT regulations (e.g., requirements for Low Visibility and other
Special Approval Operations)
• Requirements based on the Equivalency of Malfunctions process for the aircraft type
• Airport-specific requirements not covered by the Equivalency of Approaches process (e.g., require-
ment to regularly practice approaches at this specific airport)
• IOSA Standards and Recommended Practices mandated by the IOSA Standards Manual (ISM)
• Operational Suitability Data (OSD) mandated by aircraft manufacturers
• Training Areas of Specific Emphasis (TASE) and other training recommendations of aircraft manufac-
turers (e.g., documented within the Airbus FCTS)
• Operator-specific requirements derived from the SMS processes (e.g., temporary or permanent Safety
Review Board decisions)

Guidance to manage the modules for pilots qualified to operate in either seat
The following recommendations apply to module design for pilots designated to perform their duties from
either pilot seat, such as:
• Type Rating Instructors/Examiners (TRI/TRE)
• Line check/evaluation airmen
• Pilots who are authorized to conduct take-offs and landings from either control seat
• Cruise relief pilots
State regulations as well as IOSA standards require a seat-specific (right or left seat, as applicable) recur-
rent program for pilots who perform duties from either pilot station.
Under traditional training programs, the additional training and checking items required to operate in ei-
ther pilot seat include a selection of the following events:
• An engine failure during take-off
• A one-engine-inoperative approach and go-around
• A one-engine-inoperative landing
• Relevant UPRT exercises
IATA recommends that under EBT, the above-mentioned events be considered as training topics with a
B frequency (see 2.4.1) and be positioned during the MT phase of the EBT module. The pilots qualified to
operate in either seat should perform the maneuver training phase by alternatively operating the aircraft
from the left and the right seat as per the following example:

Module 1 Module 2 Module 3 Module 4 Module 5 Module 6

EVAL / MT EVAL / MT EVAL / MT EVAL / MT EVAL / MT EVAL / MT
SBT SBT SBT SBT SBT SBT
Left-Hand Seat LHS RHS Left-Hand Seat LHS RHS Left-Hand Seat LHS RHS

 45
 Evidence-Based Training Implementation Guide

This approach ensures that the pilot operates the aircraft from both operating seats each year and allows
for a wider exposure to aircraft drills and procedures that would not normally be the pilot’s responsibility.
This approach complies with the EU regulation for EBT.
Developing EBT MT to accommodate operations from either pilot seat includes:
• Assessment of the training need through review of the respective aircraft generation table of assess-
ment and training topics for the maneuver training phase as well as any additional elements required by
the CAA for operating in both pilot seats.
Items required by traditional training programs are already incorporated in the EBT maneuvers training
table.
• Design the composition and structure of the MT phase across the whole EBT program to include the
training events, their delivery sequence, scenario elements and the schedule using the frequency guid-
ance published for the MT phase.
• Develop the training and assessment materials based on the operator’s adapted competency model,
including training notes, exercise briefings, etc.
• Implement the module and conduct the training while ensuring a robust tracking system is in place to
ensure compliance with the training design.

Guidance to manage the modules for augmented flight crews

For long haul operators with a network requiring augmented flight crews, during the controlled rest of the
Commander only two First Officers may be present as active pilots on the flight deck (i.e., around 30% of
the cruise flight time). At least one of the F/Os must be qualified to operate from the left-hand seat during
flight in cruise (generally above flight level 200). For the pilot competency development of the F/O operat-
ing regularly during augmented flight crew operations, it should be possible to occasionally conduct EBT
modules with two F/Os occupying the left and right pilot seats.
The CAA approval may require that the operator considers the following elements:
• At least one of the pilots is qualified and approved to relieve the Commander as per applicable
regulations
• The pilot acting on the left-hand seat is assessed only as Pilot Monitoring during the departure and
arrival phases of the flight
• All training topics required for the MT phase are performed as Pilot Flying from the right-hand seat,
except for maneuvers required in abnormal or emergency situations (e.g., manual emergency descent
or UPRT elements) which may be trained from both seats
• The scenarios should be adapted to the specific crew composition
• At least every second module should be conducted in standard crew composition (i.e., with a
Commander) to enable training and assessment as Pilot Flying during the departure and arrival phases
in the EVAL and/or SBT

Step 10: Module design

Description: The EBT project team commits the course developers to design the EBT modules.

The EBT course developers should design the modules according to the principles described in the previ-
ous paragraph “EBT program design general” (Preamble to Step 9 and Step 10).
All modules and lesson plans should be evaluated (dry run) by a group of IEs before the delivery of the
training, to ensure that anticipated timings and FSTD fidelity permit to achieve the training objectives.
Special attention should be given to the guidance and training materials for IEs and pilots. Specifically, for
the EVAL, both, IEs and pilots, should have access to the operator’s standard flight data file (weather data,
NOTAMs, etc.) in order to prepare and to conduct the EVAL flight profile in a realistic way.

46
 Implementation Process

Evaluation Phase (EVAL)

The purpose of the EVAL is to assess competence, determine training system effectiveness and identify
individual training needs. The EVAL should be designed so that a flight crew, operating in a non over-de-
manding operational and environmental context (see Step 3, characterization of the context complexity), is
not overly challenged. On completion of the EVAL, any areas that do not meet the required level of perfor-
mance, should become the focus of the MT and SBT phases. If all competencies are assessed at or above
the required level of performance, the focus of subsequent training should be to further enhance the pilot
competencies.
IATA recommends that all competencies be assessed during the EVAL in order to identify individual train-
ing needs. This recommendation is aligned with the EU regulation for EBT.
The ideal balance in a three-year EBT program cycle is to balance the training and assessment of the com-
petencies, to ensure that the training topics and associated scenarios permit to regularly challenge the
9 competencies, over the 3-year period. The course developers should balance the focus of evaluations
on two to three competencies by systematically ensuring that all competencies have been addressed
over the 3-year period. However, the assessment of all competencies should be possible in any module by
choosing appropriate scenario elements, system malfunctions and approaches.
Once the operator has sufficient data, targeted competencies may be selected to ensure that the pro-
gram design addresses the most cited weaker competencies and observable behaviors. Course develop-
ers should ensure that each scenario and option is weighed to a similar difficulty level for the entire pilot
population.
The development of real-time realistic scenarios, which enable the pilots to utilize all resources at their
disposal, is a key element to the evaluation. The evaluation phase should consist of a line-oriented flight
scenario (or scenarios) during which there are one or more occurrences to assess all competencies and
identify individual training needs.
The assessment of all the competencies requires that both pilots act as Pilot Flying (PF) and Pilot
Monitoring (PM). A change of the PF can be achieved either by:
• Creating two independent scenarios, or
• A reposition within a flight (this reposition may also include a clearance of system malfunctions experi-
enced during the first part), or
• A single flight scenario with a change of PF and PM duties at a predetermined point of the flight
The first scenario in the EVAL should commence with a normal aircraft pre-flight set up with full opera-
tional flight plan information provided to the crew. This helps build realism and allows the pilots time to
assimilate their environment. There are other possible commencement points for the EVAL, but great care
should be exercised to ensure that the crew is given time to fully prepare and assimilate the environment
before the evaluation begins. The pilots should be regularly exposed to all phases of flight across the EVAL
phases of the EBT triennial approved program.
It is intended that only a limited number of training topics be selected as the assessment vehicle, and that
the scenario devised for each pilot should be conducted in real time. Consideration should also be given
to the benefit of scenarios that are time-constrained, especially when the focus is on competencies WLM
and LTW. Where aircraft malfunctions are considered for this phase, they should be drawn from a tradition-
al, determined list from the aircraft manufacturer and not from unforeseen scenarios.

Maneuvers training phase (MT)

The critical elements of each maneuver are described in the matrix of Appendix G.V. This phase has a fo-
cus on skill retention and addresses mainly the competencies FPA, FPM and PRO.
It is important to maintain the focus on skill, and not to turn this into a LOFT-style training. Once the pilot
has completed the critical part of the maneuver successfully, the aim has been achieved. Repositioning of
the flight simulation, to focus training on the intended maneuvers will be a commonly used FSTD feature

 47
 Evidence-Based Training Implementation Guide

for this phase. Every effort should be made to provide a relaxed environment free from the normal LOFT
style considerations, wherein the crew can practice skills with coaching where necessary.
Note: When moving from traditional training programs without repositions to EBT, frequent repositions
during the MT phase are likely to be criticized by IEs and pilots. Therefore, to ensure the acceptance of the
EBT program:
• Repositions should be thoughtfully planned during module design
• The initial EBT IE standardization should emphasize the pilot’s need for sufficient time and
standardized processes to accommodate with the new situation (e.g., responsibilities for aircraft
system/configuration changes, requirements to brief the new position/condition and next
maneuver)
• The communication plan to the pilots should emphasize the focus of the MT compared to the other
EBT module phases

Scenario-based training phase (SBT)

This training should consist of line-oriented flight scenarios during which one or more threats may be in-
troduced. Details are specified in Appendix G.V. The contents of this training should be adapted to develop
the competencies identified during the EVAL.
The goal of SBT is to develop competencies and the confidence for effective management of threats and
errors to enhance the pilot’s ability to cope with both predictable and unforeseen situations. To enable
competency-specific training, the module design for SBT should allow the IE to choose scenarios, or sce-
nario elements, that have specific competencies in focus, while fulfilling all required training topics for this
phase. The IEs may be given some flexibility to adapt the module within the required training topics, as long
as they remain within the framework of the competencies and system malfunctions, e.g., they can slightly
change the conditions.
The underlying philosophy of individually tailored training within this phase is the identification of the
pilot’s individual training needs during the EVAL phase. Tailored training enhances competency develop-
ment even for those pilots who have not demonstrated deficits during the EVAL phase.
Concerning the competency development at the individual pilot level, the course design should use the
following solutions for the SBT scenarios.
Solution 1:
The content of the scenario is totally prescriptive, but the IE can choose different options within the sce-
nario to develop different competencies. Each option places special emphasis on specific pilot competen-
cies while all options realize the same assessment and training topics. The IE selects the option according
to the pilot’s performance, demonstrated during the EVAL phase.
Example

Scenario options within a totally prescriptive scenario

Option 1: Special emphasis on SAW and LTW Option 2: Special emphasis on COM and WLM

The IE selects the option according to the pilot’s performance demonstrated during the EVAL phase.

48
 Implementation Process

Solution 2:
The content of the scenario is partially prescriptive, and the scenario allows the IE to develop the pilot’s
competencies with the instructor “toolbox”. This toolbox provides the IE with a set of events-exercises
that permits to place special emphasis on the competencies that have been identified as “to be enhanced”
during Session 1 of the module. The IE selects the exercises that place special emphasis on specific pilot
competencies.
Example

Instructor “toolbox” within a partially prescriptive scenario

Scenario event Time KNO PRO COM FPA FPM PSD LTW SAW WLM
Uncontained engine 10 min X
failure
Smoke event 10 min X X X X X
Pilot incapacitation 5 min X X
during take-off roll
The IE selects the exercises according to the pilot’s performance demonstrated during day 1.

A more detailed example is shown in Appendix G.VII.

The scenarios of the SBT phase may be more demanding compared to the EVAL, in order to challenge the
pilot’s competence and to build confidence within a line operations simulated environment. They should
maximize exposure to a variety of situations according to the priorities determined through analysis, for
the purpose of enabling learning and developing competence and resilience. Special emphasis should be
given to scenarios that involve a broad set of competencies.

Development and assessment of the Pilot Monitoring (PM) role

The PM plays a vital role in operational safety. One of the objectives of the EBT program is to devote spe-
cial attention to the development and enhancement of this role.
IATA recommends that ATOs/AOCs place special emphasis on the PMs’ role in general. Further details are
available in the IATA Guidance Material (GM) for Improving Flight Crew Monitoring.
This guidance material indicates that monitoring is embedded within all competencies of the Pilot
Competency Framework, including, but not limited to the following observable behaviors:

Monitoring within the Pilot Competency Framework

Application of Procedures and Compliance with Regulations
– OB 1.5 Monitors aircraft system status
Communication
– OB 2.3 Conveys messages clearly, accurately and concisely
– OB 2.4 Confirms that the recipient demonstrates understanding of important information
– OB 2.5 listens actively and demonstrates understanding when receiving information
– OB 2.8 Uses and interprets non-verbal communication
Aeroplane Flight Path Management, automation and manual control
– OB 3.2 & 4.2 Monitors and detects deviations from the intended flight path and takes appropriate action
– OB 3.6 & 4.7 Effectively monitors automation / flight guidance systems, including engagement and automatic
mode transitions
Leadership and Teamwork
– OB 5.3 Engages others in planning
– OB 5.6 Addresses and resolves conflicts and disagreements in a constructive manner
– OB 5.7 Exercises decisive leadership when required
– OB 5.10 Applies effective intervention strategies to resolve identified deviations

 49
 Evidence-Based Training Implementation Guide

Monitoring within the Pilot Competency Framework

Problem Solving and Decision Making
– OB 6.2 Seeks accurate and adequate information from appropriate sources
– OB 6.3 Identifies and verifies what and why things have gone wrong, if appropriate
– OB 6.7 Monitors, reviews and adapts decisions as required
Situation Awareness and Management of Information
– OB 7.1 Monitors and assesses the state of the aeroplane and its systems
– OB 7.2 Monitors and assesses the aeroplane’s energy state, and its anticipated flight path
– OB 7.3 Monitors and assesses the general environment as it may affect the operation
– OB 7.4 Validates the accuracy of information and checks for gross errors
– OB 7.5 Maintains awareness of the people involved in or affected by the operation and their capacity to perform
as expected
Workload Management
– OB 8.7 Monitors, reviews and cross-checks actions conscientiously

The following excerpt from the IATA GM for Improving Flight Crew Monitoring refers to course design:
• Predictive monitoring (expected threats) and reactive monitoring (unexpected/pop-up and latent
threats) are needed to manage threats in a safe, effective, and efficient way. Typical examples are
threats resulting from dispatch, weather, crew, technical, etc., challenges.
• Scenarios should be designed around effective monitoring in a complex/dynamic environment.
Consideration should be given to how and when information will be presented to the flight crew and in
what form (i.e., aural, visual, color coded or a pre-flight text). Other things to clearly define and take into
account during module design include common and possible conditions that can lead to monitoring
lapses, and available tools and strategies (i.e., CRM skills, SOPs, etc.) that can help flight crew cope with
possible monitoring weaknesses.
• Line-oriented scenarios are useful to evaluate predictive monitoring. Introducing threats (such as
weather, technical deviation and dispatch) in the flight preparation papers should allow the IE to assess
the crew’s ability to identify major threats in the flight profile at this stage. In this situation, monitoring
activities will be centered on planning and reviewing countermeasures such as planning, inquiring,
briefings, and contingency management of the potential threats. A typical example is the management
of a bad weather situation, which should be identified during flight preparation.
• For pop-up threats, the flight crew should identify the pop-up threat in a timely manner and the moni-
toring will focus on reviewing and modifying countermeasures such as evaluation of plans.
• Simple training and evaluation exercises are useful to assess reactive monitoring because error detec-
tion is centered on execution of countermeasures such as cross-check, use of checklist, announce-
ment, corrections, take over, etc. A common example of this is the unsuccessful localizer engagement
scenario during ILS approach.

In-Seat Instruction (instructor-induced instruction)

Effective monitoring and error detection are increasingly important when operating highly reliable auto-
mated aircraft. During the MT and SBT phases of an EBT module, In-Seat Instruction (ISI) may be used as a
method to train monitoring and cross-checking, and specifically for upset recovery training.
Note: As the acronym ISI is used worldwide in EBT regulations, this guide continues to use ISI. However,
as pointed out below, ISI does not always require the instructor to occupy the pilot’s seat. Therefore, IATA
proposes the wording “instructor-induced instruction”.
ISI should follow a predetermined scripted scenario, which can be achieved by:
• The response of one pilot to simple instructions provided confidentially by the instructor, for example
to simulate pilot incapacitation, or
• By the instructor occupying a pilot seat and performing pre-determined exercises acting as PF or PM
for the purpose of demonstration and of intervention by the other pilot, or

50
 Implementation Process

• Positioning the aircraft outside the normal operating envelope (e.g., for upset recovery training) and
providing appropriate support for CM1 or CM2 actions (e.g., RTO). This is in line with avoiding negative
training.
ISI is a major enhancement to traditional FSTD operation where instructors normally remain at the in-
structor operating station, far removed from the trainees, operating seats and flight controls. ISI creates a
better training platform for learning and allows timely intervention and repetition when required.
To avoid mistrusts between the pilots, ISI should be performed only during specific scenarios, clearly
identifiable to the pilots. The exercises should be carefully scripted in a manner that does not cause any
negative training transfer.
Where a pilot is instructed to play a role, there should be no assessment of this function. Where a pilot is
expected to respond to an error induced by ISI, there should be no negative consequences to any assess-
ment of performance for the duration of ISI. Once ISI has ceased and/or control is transferred, subsequent
performance may be assessed in the normal way.
Examples are given in Appendix G.V and should be confined to simple acts or omissions for the purpose of
eliciting active and effective monitoring and, where necessary, intervention by the pilot monitoring. Topics
marked “ISI” are intended to be the focus of instructor-induced instruction. In these cases, topics should
be combined to create an ISI scenario to be used at the determined frequency. The scenarios should be
realistic and relevant for the purpose of demonstrating and reinforcing effective flight path monitoring.
Demonstrated role-play should contain realistic, not gross, errors leading at times to a mismanaged air-
craft state, which can also be combined with upset management training.
Note: The crew’s experience during ISI is highly dependent on the instructor delivering the program. A
motivated and competent instructor will be able to effectively train aircraft monitoring and error detection
during ISI. These are increasingly important in risk mitigation tools while operating highly reliable
automated aircraft. ISI enables pilots to not only practice psychomotor skill-based exercises under the
supervision of an instructor, but it also enables the crew to practice verbalizing effective flight path
monitoring and error detection.

Resilience development
The EBT program as a whole, but in particular the SBT phase, should be designed observing the following
principles:
• Resilience, surprise, and unexpected events: The simulator session should allow variations so that
the pilots are not familiar with the scenarios presented in the simulator session. To preserve program
integrity and fairness, variations should be the focus of EBT program design. They should not be left to
the discretion of individual instructors.
• Resilience and decision-making (dilemma): The EBT program should be designed in such a way that at
least in every second module the crews are exposed to a scenario where multiple non-optimal solu-
tions are possible, with some unfavorable conditions attached to each solution. This situation challeng-
es the competency Problem Solving and Decision Making (PSD).

Alternative scenario elements

It is not mandatory to implement the example scenario elements from ICAO Doc 9995, those scenarios are
proposed to facilitate the design of the EBT program by the operators, who may develop alternative sce-
nario elements that are more relevant to their operations, based on their own data. However, the course
developers should ensure that when implementing alternative scenarios, the desired outcome of the
training topic is achievable.
The IATA CBTA Guide for Flight Crew Training provides a mapping from task to competencies, and from
threats to competencies, which supports the design of alternative scenario elements by ensuring the
identification of the competencies that are supposed to be demonstrated during the realization of a
scenario.

 51
 Evidence-Based Training Implementation Guide

Step 11: Implementation of the EBT program

Description: The EBT project team commits the training department and the flight standards department to define
the policy and procedures applicable to the EBT program.

The following paragraphs provide recommendations to address the EBT program procedures applicable
for the implementation and the conduct of EBT. The precise scope and limitation of the procedures should
be agreed in partnership with the CAA. The training and logistical difficulties of only fleet-wide versus op-
erator-wide trials should be considered.
Note: Section 5 describes in detail how the IEs deliver the EBT program. The CBTA project team monitors
this activity.

Enrolment of pilots in the EBT program

General concept
The concept of enrolment in EBT originates from the need to formalize the pilots’ licensing revalidation and
renewal aspects under an EBT program.
For the purpose of this guide:
• “Enrolment” means the administrative action carried out by the operator where a pilot participates in
the operator’s EBT program
• “Enrolled pilot” means the pilot that participates in the EBT recurrent training program
These definitions comply with the EU regulation for EBT.
Generally, enrolment occurs when a pilot commences the first EBT module after the formal EBT program
start.

Enrolment for pilots at the end of a conversion course

Enrolment of a pilot may be based on the following options:
1. A pilot performing the EBT module at the end of a non-CBTA type rating course, of which Session 1
meets all the authority requirements for the associated type rating skill test
2. A pilot performing the EBT module latest six months after passing the type rating skill test/operator
proficiency check of a non-CBTA type rating/operator conversion course
3. A pilot completing a competency-based type rating course including all the type rating skill test
requirements
Option 1. This option is a challenge as the integration of the type rating skill test requirements into Session
1 of an EBT module generates a high volume of events combining skill test items and training topics to be
performed within the time slot allocated to FSTD. This option may necessitate to extend the FSTD time
slot.
Option 2. This option is recommended for operators having already implemented an EBT program, as this
option permits to avoid the challenge of option 1.
Option 3. This option applies to CBTA-based type rating and operator conversion courses. It represents
the optimum and consistent implementation of the competency-based principles. Ideally, the type rat-
ing course/operator conversion course is designed in such a way that the pilot demonstrates the level of
performance required by the operator at the end of the course, during the EBT module sessions. The type
rating skill test items should be performed during the course in accordance with the approval of the CAA.
Note: For option 3, the CBTA-based type rating and operator conversion course should be designed in
accordance with the IATA CBTA Guide for Flight Crew Training.

52
 Implementation Process

De-enrolment
De-enrolment occurs when a pilot leaves the operator’s EBT program, and the operator is no longer re-
sponsible for the administrative action for license revalidation under the EBT program.
De-enrolment may occur due to:
• A pilot leaving the operator
• A pilot not demonstrating an acceptable level of competence
• A deliberate action from the operator to de-enroll a pilot, e.g., due to termination of the EBT program,
pilot conversion to another aircraft type, administrative reasons after prolonged absence of a pilot, etc.
• Prolonged absence of a pilot, exceeding the contingency procedures stated below
If a pilot fails to demonstrate an acceptable level of competence and leaves the operator’s EBT program,
the pilot should be informed not to exercise the privileges of that type rating.

EBT Module Scheduling

Pilot scheduling
Planning Sessions 1 and 2 of an EBT module with the same pilots and on consecutive days is recommend-
ed by IATA, based on the following considerations:

PRO CON
Training perspective
• Tailored training is proactively delivered during – Less frequent training, compared to four events
Session 2 on observed deficiencies during evenly distributed over 12 months. This aspect is
Session 1. particularly relevant for long haul pilots.
• Higher pilot satisfaction and confidence due to – Training effectiveness during Session 2 may be low if
the proactive tailorization of the training during pilots had a negative experience during Session 1.
Session 2.
• Performance improvements from Session 1 to
Session 2 are easier to manage during two consecu-
tive days.
• Support of “learning from the positive”, as the
tailored training is also applicable for pilot already
demonstrating high performance.
• Better training quality and enhanced resilience de-
velopment during Session 2, as the assessment has
been done during Session 1.
Operational perspective
• Schedule planning merits due to less man days when – Probable scheduling constraints (scheduler needs
training is not delivered at the homebase. to consider rest time after Session 1, less flexible as
• Lower travel costs when training is not delivered at two consecutive days are blocked).
homebase.
• No schedule change required when performance
demonstrated during Session 1 is below adequate
and can be recovered during Session 2.

IE Scheduling
The IATA training committee members discussed extensively the pros and cons of the conduct of an EBT
module by the same IE during Session 1 and Session 2. The following template provides the training and
operational perspective aspects that have been assessed by the IATA experts. Most members of the IATA
training committee have elected to conduct the EBT module with the same IE during the two sessions on
consecutive days. Nevertheless, there may be reasons to start the mixed EBT implementation with differ-
ent IEs.

 53
 Evidence-Based Training Implementation Guide

When the module is planned with different IEs, clear handover procedures between Session 1 and Session
2 should be implemented to ensure effective pilot competency development during Session 2.
Planning for the same IE to conduct Sessions 1 and 2 of an EBT module is recommended by IATA, based on
the following considerations:

PRO CON
Training perspective
• EBT is designed as a complete training module ded- – An IE conducting the entire module may be biased
icated to the pilot competency development during during Session 2.
the training phases (MT and SBT). – One IE may generate a less diverse learning experi-
• EBT’s objective is to achieve “more training, and ence compared to two different IEs.
less checking”; it is important that the trainees, even
when being evaluated, are not in the “check mind”.
Having the same IE during both sessions facilitates
the achievement of the objective.
• A consistent training delivery from assessment and
facilitated debriefing during Session 1 to a tailored
training during Session 2 enhances the overall learn-
ing experience during the module.
• Content of Session 2 can be prepared in advance, by
the IE as the same coach for the entire module.
• During Session 2, no second initial assessment is
required, as the instructor already knows the pilots’
training needs.
• It is easier for the IE to grasp the entire Session 2
training objectives, asses the performance and give
integrated feedback across the module.
• The trainees have a better appreciation of their com-
petencies’ development across the phases of the
module.
Operational perspective
• Handover procedures from Session 1 are only re- – Cannot be implemented, if IE requirements de-
quired in case of unforeseen circumstances. fined by the CAA are different during Session 1 and
• IE’s expertise with the entire EBT module content Session 2.
develops faster and is broader. – More examiners needed during mixed implementa-
• Training efficiency is improved as less EBT IEs are tion, as both days are performed by an examiner.
needed. – Risk of higher costs due to reduced scheduling
• The acceptance of the EBT program by the pilots and flexibility.
the IEs is facilitated because the learning experience – Potential scheduling constraints using same instruc-
is reinforced by having the same IE coaching the tor two consecutive days.
pilot during the entire module.

Handover procedures
When the EBT module is conducted by two different IEs, information regarding the pilot’s performance and
needs in terms of competency development from Session 1 should be transmitted to the IE conducting
Session 2 to allow for tailored training. The ATO/AOC should implement “handover” procedures and asso-
ciated tools to ensure the transmission of the relevant information.
Example

A summary of Session 1 performance assessment done by IE 1 is available in the Session 2 electronic template for
IE 2.

The pilots’ performance data is generally accessible to the IEs through electronic records. Otherwise, the
training department should provide the information in another format such as paper records or by commu-
nicating specific training objectives tailored to the pilot’s needs.

54
 Implementation Process

It is important to consider that the second IE has got enough time to interpret the information received
and, if necessary, adapt the training program to the individual needs of the pilots.

Contingency procedures
The EBT program should include contingency procedures for unforeseen circumstances that could affect
the delivery of the EBT modules (e.g., simulator breakdown, sick pilot or IE) and prolonged pilots’ absence.
The contingency procedures should also cover the situation where the pilot demonstrates a performance
below the performance level defined by the operator.

Module interruption
In case of unforeseen interruption of a module at any point, the missing parts of the module should be re-
scheduled. The pilot may continue line flying until the expiry of the validity period unless the performance
observed was below the performance level defined by the operator. If the interruption results in an IE
change, the operator should ensure that the IE completing the module is provided with details of the pilots’
performance demonstrated before the interruption of the module.
Operator sample 1

Interruption Simulator breakdown / Pilot absence Instructor/Evaluator (IE) absence

Before/during Try to schedule another simulator/pilot Try to schedule another IE.
Session 1 If not possible: If not possible:
– Use the slot of Session 2 to conduct Session – Use the slot of Session 2 to conduct
1 with the same pilots and IE. Session 1 with another IE.
– Reschedule Session 2 as a single event as – Reschedule Session 2 as a single event as
close as possible to Session 1. close as possible to Session 1.
After Session 1 Reschedule Session 2 as a single event as close Schedule another IE to complete the EBT
as possible to Session 1. module.
If not possible:
– Reschedule Session 2 as a single event as
close as possible to Session 1.
During Session Reschedule Session 2 as a single event as close Reschedule Session 2 as a single event.
2 as possible to Session 1
Rescheduling of Session 2 should preferably take place within 30 days, otherwise a complete EBT module shall be
rescheduled.
Generally, no single cockpit-crew is allowed.
If only one pilot is present: The training event may be conducted according to individual wishes but must never be
counted as OPC or EBT-module day.

Operator sample 2

Contingency procedures for unforeseen circumstances that may affect the delivery of the module
(a) In case of missed EBT modules and recency training, training will be managed in accordance with the operator
policy.
(b) In case of unforeseen interruption of a module at any point (e.g., sim breakdown or IE illness), the missing parts
of the module should be rescheduled prior to the pilot-specific EBT module expiry date.
The pilot may continue line flying until the expiry of the validity period, unless the performance observed was below
the minimum acceptable level.
A schedule interruption will most likely result in a different IE conducting Session 2 of the EBT module. IEs should
familiarize themselves with the Session 1 report prior to delivering Session 2 in order to ensure an appropriate
training focus.

 55
 Evidence-Based Training Implementation Guide

Recency requirements missing

If a pilot is non-compliant with regards to operational elements required by Annex 6 (i.e., recent experience
and/or aerodrome, route, area recency requirements), a CBTA refresher session should be scheduled
which has to include the necessary means to fulfil the requirements (e.g., 3 take-offs and landings).
This session is optional for longer absences below 90 days.

EBT modules missed by the pilot

The following paragraphs propose CBTA training solutions depending on the number of EBT modules
missed by the pilot and on the validity of the pilot’s license. They provide an effective competency-based
training path to maintain the pilot’s enrollment in the EBT program.
The CBTA training solutions below were developed originally during the COVID period and were endorsed
by ICAO to facilitate the management of training and licensing to ensure safe operations.
The following template presents the CBTA training solutions recommend by IATA. Appendix H gives fur-
ther guidance regarding the different components of the proposed training solutions.

CBTA Training solutions for missed EBT modules

Topic Situation 1 Situation 2 Situation 3 Situation 4
Missed 1 module 1 module 1 to 2 modules 3 to 5 modules
modules
License validity Valid Expired < 90 days 90 days ≤ Expired 1 year ≤ Expired
< 1 year < 3 years
Ground Optional X X
refresher
FSTD refresher Optional1 X2 X
FSTD Optional3 X3
consolidation
FSTD EBT EVAL phase only 0 X X X
module
LIFUS X4 X

0
To conduct line operation, the pilot should perform the EVAL phase of the module and complete the module be-
fore the expiration date of the license validity (this solution originates from EU regulations)
1
Recommended when last flight was performed >60 days
2
Should include the recent experience and (as applicable aerodrome/route/area recency requirements)
3
Based on the performance demonstrated during the FSTD CBTA refresher session
4
Recommended when last flight has been performed beyond 6 months

Note: Refer to Appendix H for a detailed description of the different components. Local CAA regulations
may require more specific procedures (e.g., current European EBT regulations are more detailed regarding
license revalidation requirements in case of missing modules).

One module is missed but the license or rating is still valid

Situation 1: When a pilot has missed one module, at least the missing EVAL should be rescheduled before
the pilot resumes line operations. The MT and SBT of the missed module should be completed within a giv-
en timeframe defined in the operator’s procedures (e.g., 30 days), but at the latest before the expiry date of
the pilot’s license or rating. This solution originates from the EU regulation for EBT.

One or more module(s) missed, and the license or rating validity has expired
IATA recommends that the AOC/ATO approved for renewal of the license or type ratings conducts compe-
tency-based refresher training for the pilots to renew their license or rating while ensuring their enrollment

56
 Implementation Process

within the EBT program is maintained. The CBTA refresher training proposes a progressive and tailored
training solution depending on the pilots’ competencies needs. (Refer to the template above.)
Situation 2: When a pilot has missed one module and the license or type rating has expired by less than 90
days, the renewal of the license or rating should be performed during the operator’s current EBT module. A
CBTA refresher session is recommended when the last flight was performed more than 60 days ago.
Situation 3: When a pilot has missed one or two modules, and when the license or type rating has expired
by less than one year but more than 90 days:
• The content of the CBTA refresher training should be the following:
– A ground training refresher consisting of the missed ground training elements or a specific
refresher program
– A CBTA refresher session including a competency assessment
• The renewal of the license or rating should be performed during the operator’s current EBT module
Situation 4: When a pilot has missed three to five modules and when the license or type rating has expired
by more than one year but less than three years:
• The content of the CBTA refresher training should be the following:
– A ground training refresher consisting of the missed ground training elements or a specific
refresher program
– A CBTA refresher session
– A CBTA consolidation session G.VII
• The renewal of the license or rating should be performed during the operator current EBT module
• IATA recommends as well that the pilot performs line flying under supervision (LIFUS) to demonstrate
competence and confidence during normal line operations

Six modules missed

Situation 5 (not shown on the template): A pilot who has missed six consecutive EBT modules should be
de-enrolled from the EBT program, and the relevant requalification requirements for traditional training
programs should apply (i.e., the pilot has to repeat parts or all sessions of the type rating/operator conver-
sion course).

Consequences of competency assessment

The operator’s training policy should describe the process to monitor and to ensure that the pilot’s per-
formance is consistently demonstrated at or above an adequate level of performance. The training should
be adapted to individual needs in order to continuously develop the pilot’s competencies and ensure the
maintenance of an adequate (or better) level of performance.
The operator’s training policy should describe the outcome of the competency assessment based on the
different levels of performance as defined in their adapted competency model (e.g., ineffectively/minimal
acceptable/adequately/effectively/exemplary manner). The training policy should describe at least the two
following situations:
• The pilot has demonstrated an adequate (or better) level of performance across all competencies (e.g.,
all competencies are graded 3 or above on a 5-point scale). The training objectives of the EBT module
are achieved.
• The pilot has demonstrated a level of performance below an adequate level in one or more competen-
cies (e.g., one or more competencies are graded below 3 on a 5-point scale). The training objectives of
the EBT module are partially achieved (or not achieved). The training policy should describe the remedi-
al actions required to address deficiencies in performance.

 57
 Evidence-Based Training Implementation Guide

Examples of training policies

Example of policy for the management of performance below adequate (grade 3)

A tailored training (that may include remedial training) is required for:
– Any competency graded 1, or
– Two successive grades 2 in a same competency, or
– Any competency graded 2 if the trainer evaluates that the trainee will not be able to demonstrate an adequate
performance (grade 3) during the next training or evaluation session

Example of an Operator Policy for EBT training objectives achievement

EBT IEs shall ensure minimum competency standard has been met and objectives for MT and SBT are accom-
plished by the end of the module.
Although EVAL, MT and SBT can be taken in their individual parts, if there is a combination of training to proficiency
for EVAL and OAWAP for MT and SBT, or excessive instructor input has been required to meet the exit standard, the
module shall be recorded as being unsuccessful.
Note: OAWAP means Objectives Achieved with Additional Practice. This would categorize a performance where the
standard had been met but with more instructional input than normally expected – i.e., some instructor input re-
quired above and beyond that expected for the set objective.

Example of policy for pilot performance below adequate in an EBT module

Situation Not competent Not released Grade 2 and
for line operation released for line ops
Competency – At least one GRADE 1 any – More than four compe- – Four GRADE 2 in EVAL,
grading phase (EVAL/MT/SBT) tencies with GRADE 2 in – One GRADE 2 in SBT or
any SIM session, or prior interruption of the
– More than one GRADE 2 EBT Module, or
in SBT or prior interrup- – At least one GRADE 2 in
tion of the EBT Module two out of three consecu-
tive EBT Modules
Consequence of – Pilot not released for line – Pilot not released for line – Pilot released for line
grading operations operations operations
Responsibility Training Manager/Chief Flight Training Manager/Chief Chief Training Captain of
Instructor Flight Instructor respective type
Flow of information – Instructor/Evaluator – Instructor/Evaluator – Instructor/Evaluator
○ Chief Training Capt. ○ Chief Training Capt. ○ Chief Training Capt.
○ Crew scheduling ○ Crew scheduling – Chief Training Captain
– Chief Training Captain – Chief Training Captain ○ Training Manager
○ Training Manager ○ Training Manager
○ Head of Fleet ○ Head of Fleet
Actions – Competency analysis – Competency analysis – Competency analysis
– Training Committee – Training Committee – Training Committee
– Assistance & Extra train- – Assistance & Extra train- – Assistance & Extra train-
ing prior next flight duty ing prior next flight duty ing in a timely manner
– Repetition of entire EBT – Released for line ops
Module after successful after successful Extra
Extra training training

Recurrent ground training

Technical ground training
The philosophy of EBT is that the qualified FSTD should be used to the maximum extent possible for as-
sessing and developing crew competence. To repeat simple malfunctions and use the FSTD environment

58
 Implementation Process

as a procedure training device is to deny the benefit of the FSTD as a learning tool, and for this reason the
equivalency of malfunctions system was developed (see Step 7), to reduce unnecessary requirements to
“tick boxes” against each listed malfunction of the ATA chapters.
The following recommendations have been inserted from the European EBT regulations which introduced
a performance-based ground training concept.
The objective of the technical ground training program is to ensure that pilots have adequate:
• Knowledge of aircraft systems and the operational procedures and requirements; and
• Awareness of (a) the most significant accidents or incidents that could affect their operations following
the TEM model or an alternative risk model agreed with the authority; and (b) the occurrences in the
airline or occurrences from other airlines that may be relevant for their operations, accident/incident
and occurrence review.
The technical ground training should:
• Be conducted as part of a 3-year program
• Allow a customization of syllabi, based on evidence both internal and external to the operator
• As a minimum, allow the pilot to receive technical ground training every 12 months
The technical ground training syllabi should be delivered using different methods and tools:
• The selection of the method and tools results from a combination of the learning objectives and the
target group receiving the training (WHAT needs to be trained and WHO needs to be trained).
• The selection of the appropriate method and tool should be driven by the desired outcome in terms of
adequate knowledge.
• The delivery of the technical ground training syllabi should include the methods or tools to verify if
the pilot has acquired the objective of the technical ground training program. This may be achieved
by means of a questionnaire, assessment of application of the competency Application of Knowledge
(KNO) or other suitable methods.
The measurement and evaluation of the training system performance through the feedback process
should include the performance of the technical ground training.

Integration of CRM training

IATA Operational Safety Audit (IOSA) Standards presently specify that operators shall ensure that:
• Flight crew members complete training and, when applicable, an evaluation in crew resource manage-
ment (CRM), including Threat and Error Management, using facilitators that have been trained in human
performance and human factors principles
• Flight crew members complete training in CRM skills, which may be accomplished as part of simulator,
aircraft and/or line training, as applicable
Operators implementing EBT may demonstrate compliance with CRM requirements by showing how recur-
rent CRM is integrated within the operator’s EBT program. An operator example, based on European CRM
and EBT regulations, is provided in Appendix G.VI.

Line evaluation of competence

The provision on the line evaluation of competence is intended to have the same scope as the line check
currently has. Obviously, this implies successful demonstration of competence in the management of any
abnormal or emergency situations that may occur during the flight.
The following recommendations have been inserted from the European EBT regulations:
• Each pilot shall periodically undertake a line evaluation of competence in an aircraft to demonstrate
the safe, effective and efficient conduct of normal line operations described in the operations manual.
The line evaluation of competence substitutes the line check. For successful completion of the line

 59
 Evidence-Based Training Implementation Guide

evaluation of competence, the pilot shall demonstrate an acceptable level of performance in all ob-
served competencies.
• The line evaluation of competence shall be conducted by a suitably qualified commander nominated by
the operator that is standardized in EBT concepts and the assessment of competencies (line evalua-
tor). The line evaluator should have a valid line evaluation of competence and should receive an accept-
able training based on the EBT IE training. The EBT assessment of competence is not required. The line
evaluator training may be included in the EBT ICAP.
Note: The wording “normal line operations” does not refer to malfunctions; it refers to a normal flight (no
test flight, no maintenance flight, etc.).

Step 12: Review of training effectiveness

Description: The EBT project team commits the IEs and course developers to evaluate the EBT program. Once im-
plemented, training metrics are analyzed at a predetermined frequency, to establish training system effectiveness
and, where necessary, make corrections to the program.

Note: Effectiveness, according to ISO 9000, is defined as the “extent to which planned activities are
realized and planned results are achieved”.
EBT programs target to deliver competent pilots that have demonstrated the achievement and mainte-
nance of the organization’s competency standard.
The achievement of the organization’s competency standard should permit to reliably predict that the
pilots conduct safe and efficient operations. Consequently, the consistent demonstration by a cohort of
pilots of an adequate level of performance across all competencies of the adapted competency model
should be the basic criteria to evaluate the effectiveness of the program.
The training metrics should be composed of the competency metrics, the observable behaviors metrics
and the TEM metrics. The analysis of these metrics should ensure the continuous enhancement of the
program by identifying the trends regarding trainees’ performance during the course delivery.
Practically, the analysis of these metrics should provide evidence to the organization about the competen-
cies that are globally effective countermeasures regarding threats and errors management and about the
competencies that globally contributed to the reduction of safety margins due to the mismanagement of
specific threats and errors.
The measurement of training effectiveness should be performed by comparing the progress and the
levels of performance achieved by the different pilot populations. Beyond training modules and individual
training needs, data analysis offers a wide range of possibilities, e.g., in specific states and comparisons:
• Against safety performance levels
• Within/between fleets
• Within/between ranks
• Within/between airlines
• With similar experience (in total and on type)
• Clustering and movement between clusters
• With other or among IEs
Further areas of application are evolutions and trends, e.g.:
• After training program changes
• Within training campaigns
• Periodic summaries
• Performance changes after time
The measurement of effectiveness is complex, as the performance outcomes
(a) between the legacy system and the EBT program, and/or

60
 Implementation Process

(b) between different EBT modules

have to be compared by common training metrics, however the “difficulty levels” of the training programs/
EBT modules may differ.
This measurement can be done by comparing the progress and final competency results of the pilots.
Data validity of this method presumes that the difficulty level of the different modules is comparable and
that the IEs are well standardized (for details see Step 15).
Another option would be a practical evaluation at the end of an EBT module which is performed by a stan-
dardizer who has not participated in the module. This method can be compared with LOSA and its benefits.
With this method the factors of improper standardization and possible IE bias are minimized and only the
outcome of the practical evaluation serves as an indicator of how effective the training has been.
Data from the line evaluation of competence are important to measure the effectiveness of the EBT pro-
gram in operations. It may include data from the line operations monitoring process.
It is also vital that a subjective feedback system be established, enabling both pilots under assessment
and training and the IEs to provide feedback. This process is part of the buy-in considered essential for
safety improvement and the partnership between all parties.
Note: To compare the EBT system with the legacy system, a measurement at the “event level” can form a
suitable approach: Level 3 metrics may be used, considering the first look attempt during the EVAL phase
and the initial exercises during the MT phase, instead of the final competency level. Therefore, simply
comparing the grading results of EBT modules having items of the legacy system integrated with modules
without such items of the legacy system is not recommended.

Step 13: Measurement of training system performance

Description: The EBT project team commits the Nominated Person Crew Training (NPCT)/Director of Training to
report on the training system performance post EBT program implementation.

The NPCT/Director of Training, supported by the training managers, analyzes EBT data to evaluate the
effectiveness and efficiency of the EBT program.
Additionally, the NPCT/Director of Training, supported by the training managers and in collaboration with
the Head of Safety, analyzes the training data and the safety data to evaluate the value of the training sys-
tem to maintain or enhance safety in operation.
Finally, the NPCT/Director of Training defines the training system performance by integrating the measure-
ment of the training effectiveness, the measurement of the training efficiency, and the collaborative analy-
sis of the operational and safety data.

Training system performance

The purpose of any licensing and operator training is to ensure an adequate level of safety in operations. In
the context of EBT program implementation, the characterization of training system performance should
integrate the following definitions:
• The competency is a dimension of human performance that is used to reliably predict successful per-
formance on the job, and
• EBT is characterized by a performance orientation, emphasis on standards of performance and their
measurement, and the development of training to the specified performance standards.
Therefore, the training system performance should determine whether the training permits to achieve the
specified performance standards and to maintain safe operations.
While the course evaluation in Step 12 relates to effectiveness within the training system, training system
performance should also integrate the measurement of training efficiency and should be complemented
by the analysis of the data collected in operations. The operational and safety data analysis allows the

 61
 Evidence-Based Training Implementation Guide

operator and or training organization to challenge the value of the training system to maintain or to en-
hance safety in operations.
Note: Efficiency, according to ISO 9000, is defined as the “relationship between the result achieved and
the resources used”. Efficiency complements effectiveness by integrating the amount of resources that
were necessary to achieve or to improve the effectiveness of the course.
Specifically, the EBT training metrics that are used to measure the training effectiveness should also be
used to benchmark the operational and safety data analysis. This data collected in the training context
should be interpreted in conjunction with the operational data, providing evidence on the pilots’ perfor-
mance to manage threats and errors in operations.
Practically, the training system’s performance definition can be supported using TEM metrics collected
during a training session and comparing them with safety data on the prevalence of those events in line
operations. For example, TCAS events in the training context compared to the ones gathered by the safety
department.
The training data has become more and more integrated into each organization’s SMS. Therefore, all the
relevant data streams collected by an organization should be analyzed under the umbrella of the ATO’s or
AOC’s SMS. Hence, the “inner loop” relates to the organization’s own training, operational and safety data
collection and analysis, while the “outer loop” relates to the relevant data streams that are available out-
side the organization, which include international organizations’ data, state safety recommendations, OEM
guidance, etc.
Note: When an AOC subcontracts its operator training to an ATO, the data collected in the training
context belongs to the AOC and is part of the inner loop.
The figure below illustrates the relationship between training effectiveness, training efficiency and training
system performance. For further guidance on training effectiveness and efficiency refer to the IATA CBTA
Guide for Flight Crew Training, Chapter 5.

Analysis of the training and safety data

The data collection and analysis generally need to cover various types of data, both from within the train-
ing activity and from the flight operations and the SMS.
Operational data may serve as an indicator of the effectiveness of a training system. However, the simple
idea of evaluating and comparing the number of events is not a suitable way, as other variables (such as

62
 Implementation Process

changes of aircraft types and destinations, environmental threats, culture changes, etc., within the opera-
tor) have to be considered and reviewed before reaching conclusions about training effectiveness.
The collaborative safety and training data analysis imposes a strong connection between the training
department and the safety department. This link is often difficult to establish as the two departments do
not use the same taxonomy: training is competency-based, whereas safety is event-based/event-driven.
Specifically, the alignment of training and safety taxonomies should be implemented to facilitate data ex-
change and data interpretation from both, the safety and training departments.
For example, several operators have already adopted the pilot competencies as flight crew countermea-
sures in their safety taxonomy (accident/incident analysis, occurrence reporting, etc.), while training
organizations have adopted the IATA safety taxonomy to encode the CBTA TEM metrics. These initiatives
should permit to save resources during the analysis within the “inner loop” and facilitate global internation-
al data analysis within the “outer loop”.
Where this alignment in taxonomies has not yet been achieved, the safety and training departments
should consider the following steps for collaborative analysis:
• Use operational events (e.g., flight data recorder deviations) to identify the most relevant operational
risks and associated geographical areas of occurrence.
• “Translate” these operational risks into competency-based training needs, (e.g., by applying root cause
analysis) to provide CBTA inputs for the training program improvement.
Furthermore, evidence from operational events can be mirrored by specific training events that may be
integrated into future EBT modules, to allow a comparison of training and operational data.
To integrate safety event or report data into competency-based training, the classification and assess-
ment steps of the ORCA process described in Chapter 2.8 should be used when analyzing these data.
A strategy to mitigate the gap between competency-based training and event-driven operations/safety
management could be the establishment of regular data sharing meetings (“evidence meetings”) among
all relevant operational and training stakeholders. These meetings are intended to perform Step 2 and
Step 3 as mentioned above.
Operator sample for responsibilities regarding training system performance

The EBT system performance is measured and evaluated. Training System Performance is monitored
during Evidence Meetings.
The objectives of Evidence Meetings are to:
• Identify trends and ensure corrective action where necessary
• Identify collective training needs
• Review, adjust and continuously improve the training program
• Gain feedback from technical ground training (including Safety and Emergency Procedures [SEP])
• Further develop the training system

Evidence Nominated Person “Crew Training”, Training Manager (TM), Head of Cabin 2/year
Meeting Crew Training, Safety, Security & CRM Training, Representative of Flight Safety
Department, Fleet Representatives, EBT Managers, Data Analyst

Training effectiveness may also be deducted from examining first look (first attempt) results of specific
training events (using level 3 metrics, as specified in Step 12 and comparing them with the prevalence
of those events in line operations. TCAS events during the EBT session may serve as an example. These
events are well comparable with the ones gathered by the safety department.

 63
 Evidence-Based Training Implementation Guide

An “event-evaluation” during the EVAL, or at the beginning of the MT phase, shows whether the effective-
ness of the training system is robust enough to keep the level of performance from one training session to
the next. Thus, it can be concluded that the line operation (in between the training sessions) is performed
at the same level of performance.
Additionally, data gained from the outer loop might be used to benchmark the training system’s perfor-
mance. Many opportunities to share relevant operational and training data between organizations exist
but the relevance of data from other operators depends on the similarity of aircraft types, destinations,
training programs and other factors. While other organizations’ data may be valuable, a cautious approach
should be taken not to drive the training program too extensively based on such external data.

Data protection
Whatever data concerned, handling of any data, its protection, transfer, and storage needs to be regulat-
ed, as well as data ownership, data identification, and de–identification. Moreover, as training data is part
of the safety data within the SMS, the relevant protection requirements apply to both safety and training
data.
The objective of protecting the EBT data is to avoid its inappropriate usage, in order to ensure the contin-
ued availability of such data to maintain and improve pilot competencies.
Identifying specific sets of data considered suitable, as well as their source and purpose for collection,
requires considering existing agreements, or may lead to the development of particular agreements be-
tween aircraft operators, pilot representatives, ATOs and regulators or other stakeholders.
European EBT regulations require, and IATA recommends, an operator data access and security policy
(including a procedure to prevent disclosure of crew identity), which should:
(a) Restrict information access to authorized persons
(b) Include the measures to ensure the security of the data (e.g., information security standard)
(c) Be agreed by all parties involved (airline management and flight crew member representatives nomi-
nated either by the union or the flight crew themselves)
(d) Be in line with the organization’s safety policy in order not to make available, or not to make use, of the
EBT data to attribute blame or liability
The operator may integrate the security policy within other management systems already in place (e.g.,
information security management).

Step 14: Continuous improvement of the training program

Description: The Nominated Person Crew Training, supported by the training managers, ensures that the training
system performance measurement results in a continuous improvement of the EBT program.

As indicated in the previous steps, the EBT program should be reviewed periodically and potential changes
considered, based on internal or external safety and/or training data. EBT modules are built on data gath-
ered in previous modules, authority approval-required content from a predefined compliance matrix, and
inputs from stakeholders such as operations and safety departments.
The training system feedback process may highlight areas for particular focus during the adaptation of the
EBT program.
In this context, triggers for a change of elements are set based on the analysis of data. An authority-ap-
proved compliance matrix and training data collected thus lead from one training module to another.
Grades achieved by trainees propose a performance-corresponding adaption of training elements within
the compliance matrix, leading to an increase in competency and safety.
If overall and individual grades expose areas in need of improvement, EBT offers the possibility for these
areas to be put in focus in the next program module. If certain competencies, e.g., Workload Management
(WLM) show a decay after phases of inactivity, such as during the COVID pandemic, these competencies
can be specifically addressed in the next session or the next training module.

64
 Implementation Process

While overall trends lead to changes in general program content, individual results will lead to tailored
assistance, both backed by data.
Data reflecting each individual trainee’s progress helps identify individual training needs, which can then
be addressed in a further training event. The evaluation of a trainee’s individual level of competency leads
to facilitated debriefings, which then lay the foundations for tailored briefings for the consecutive lesson
and consequently lead to the development of the competencies concerned. Within the context of EBT, this
can even incorporate previous experience made on any type at any time. It does not even require for the IE
to have any personal knowledge of the trainee.

Mixed and baseline EBT programs

While mixed and baseline EBT programs do not require operator-specific modification of the EBT training
topics and competencies, they do not relieve the operator from making use of inner loop data for the EBT
program and from considering state safety recommendations, OEM guidance, etc.
Operator example for responsibilities regarding training program improvement

The nominated person “Crew Training” and his/her subdivisions ensure continual improvement of the training and
evaluation program.
To achieve this goal, at least the following measures should be taken:
• Monitoring, recording and evaluation of results of successful and unsuccessful flight crew evaluations
• Monitoring of training deficiencies and evaluation trends (simulator, aircraft and line operations)
• Monitoring of internal reports containing feedback, either received by flight reports or other means
Furthermore, the nominated person “Crew Training” shall ensure, that formal and regular communication occurs
between and among management, instructors, examiners and pilots to achieve continual improvement of ground,
simulator and aircraft training.
When information on necessary improvements is received, the revision process should be initiated.

Care should be exercised if deviations from the recommended priorities or frequency are made. Data an-
alyzed during the creation of EBT was considerable and encompassed a wide range of types of operation.
The priorities indicated in the assessment and training topic matrix were created with a careful analysis
and should only be adjusted when there is compelling data indicating the need for a deviation.

Development of an enhanced EBT program

An enhanced EBT program is an EBT program where the assessment and training topics matrix contained
in Appendix G.IV is adapted. Adaptation may be necessary to:
• Mitigate a specific operational risk identified by the Authority, operator or OEM
• Maintain the suitability of the EBT program
Any adaptation must be approved by the CAA, and duly sustained by a safety case.
Enhanced EBT program development requires the determination of results from the analysis of data in
order to develop training scenarios and the identification of the competencies of special emphasis for the
management of these scenarios.
In addition to the baseline EBT program, the information and data used to create the training scenarios
should be derived according to the principles laid out in the previous steps of this Section.
One method to perform the development of the enhanced EBT program may be:
(a) Completion of the steps required in 3.1.
(b) Identification of threats and errors to be considered in a risk and training analysis.
(c) Execution of a risk and training analysis, as described in the Data Report for Evidence-Based Training.
(d) Development of the training guidance: This can include all threats and errors listed in the risk and
training analysis; for each training maneuver or scenario, measurable performance criteria for the

 65
 Evidence-Based Training Implementation Guide

competencies should be defined; each training maneuver and scenario should have appended to it the
competencies of special emphasis to its management; and
(e) Definition of the enhanced EBT program: This includes the outline guidance for the training programs
and the assessment of trainee performance, in addition to information for instructors conducting
the training; it is assumed that the training and assessment described will be conducted in an FSTD
qualified to ICAO Level VII according to the Manual of Criteria for the Qualification of Flight Simulation
Training Devices (Doc 9625), Volume I – Aeroplanes.
On completion of program implementation, all available measurement and tracking tools should continue
to be utilized to chart enhancements and degradations in pilot performance and the deployment of com-
petence. These tools also can be utilized to facilitate further program development and customization.

Step 15: EBT IE recurrent standardization and ICAP

Description: The IEs complete annual recurrent standardization to continuously enhance the IEs competence to
conduct EBT and to ensure the consistency of instructor assessments.

IE recurrent standardization
The EBT IE recurrent standardization should be a combination of the following:
• Essential CBTA components that should be stable over time (e.g., role of competencies within the
TEM model, competency assessment process, etc.), and that are already available in the initial
standardization
• Updated components that are not stable over time because they are the result of previous course de-
livery evaluation and/or training data analysis
Specifically, the evaluation of the EBT IE standardization and the training metrics related to the IE’s as-
sessment should be analyzed to place emphasis on the relevant IEs competencies during the recurrent
standardization.
At regular intervals, not to exceed three years, the IE should undergo a recurrent assessment of compe-
tence, conducted during the delivery of a practical EBT session.
The EBT IE recurrent standardization should also ensure the consistency of the IE’s assessment. This
relates to following concepts.

Instructor-rater reliability
Instructor inter-rater reliability (IRR) describes the degree of agreement, consensus, or homogeneity be-
tween raters. The term inter-rater reliability can be used interchangeably with the term “concordance”.
Note: While inter-rater reliability is the common terminology in statistics and academia, concordance
has recently become more prevalent in European EBT literature.
The importance of instructor inter-rater reliability has been widely emphasized in the industry. Strong
inter-rater reliability is a prerequisite for high training data quality and further safeguards the accuracy of
assessments, which allows for training data to be meaningfully interpreted and performance trends to be
derived over time.
Poor inter-rater reliability is an indicator for the training management to:
• Initiate remedial actions in order to increase training data quality
• Interpret any current EBT findings with caution
To ensure the data integrity for IE concordance, the operator should maintain data traceability for each
group of IEs (airline and subcontracted), as the root cause for the good or bad performance of each group
may be different given that the background and environment of each group is different. It may be neces-
sary to apply the same principles to other groups (e.g., mature versus young IEs).

66
 Implementation Process

Assessment consistency
Consistency of assessments necessitates:
• An agreement between raters, and
• An alignment of the raters towards the same standard
While the former ensures that a group of several IEs reaches similar conclusions on an observed specific
level of performance, the latter ensures that the IE group agreement is valid and aligned with the operator
standard of reference defined for this specific level of performance.
The examples below provide an illustration of IEs’ agreement and alignment.
Note: The center of the target (yellow dot) represents the operator’s standard of reference defined by
the organization. The black dots represent the IEs’ individual assessment regarding the observed level of
performance.

• The example on the left indicates that IEs do not agree on the level of performance observed. In this
case, the IEs’ assessments are scattered across several different levels of performance which are not
aligned with the standard defined by the organization.
• The example in the center shows high agreement, with individual IEs assessing the specific level of
performance that they observed similarly. They are, however, not aligned with the operator’s stan-
dard of reference (the assessment reported by the IEs group is either higher or lower compared to the
standard).
• The example on the right illustrates the IEs group agreement on the observed specific level of perfor-
mance and alignment with the operator’s standard of reference for the specific level of performance.
This situation illustrates high IRR and represents one of the aims of the EBT IE standardization.
Additional training to the entire IE community or individual IEs should be provided if either agreement or
alignment are found to be weak for one or more competencies.

Instructor Concordance Assurance Program (ICAP)

In the context of the global expansion of CBTA, including EBT, training data has become an essential
source of information for the SMS. As the accuracy and correctness of the training data are fundamental
aspects for an effective SMS, and as today the training data is mainly collected by an instructor, several
regulators moving to EBT recommend that AOCs/ATOs delivering EBT measure and assure instructor as-
sessment consistency, which in Europe is called “Instructor Concordance Assurance Program” (ICAP).
The role of an ICAP is to ensure that all EBT IEs are properly qualified to perform their tasks by using ap-
propriate metrics. In particular, the ICAP training data analysis should indicate the IE-group assessment
homogeneity (agreement) and the IE assessment accuracy (alignment).
The ICAP should be able to identify areas of weak inter-rater reliability to drive improvement in the reliabili-
ty and validity of the assessments performed by the IEs.
Appendix I provides details on how to implement an ICAP.

 67
Section 5 - IE Conduct of EBT
This section provides guidance for IEs on how to deliver an EBT module in an FSTD, and on facilitation as
the preferred instructional method under this program.
Moreover, this section proposes a method to facilitate the OBs data collection, supporting the competen-
cy assessment and associated grading.

5.1 Briefing
5.1.1 Briefing objectives
In order to achieve the training objectives of each phase of the EBT module session, the IE should conduct
a briefing to:
• Ensure that the pilots are in a conducive learning environment
• Understand the pilots’ individual needs
• Ensure that the pilots have a clear mental picture about the training objective
The briefing contributes to the global training efficiencies by optimizing the resources (media, FSTD, …).
Specifically, the IE should ensure that the pilots have a clear understanding of:
• The phase of the EBT module that is the purpose of the session: EVAL, MT, or SBT
• The training objectives related to each phase in terms of competency standard

5.1.2 Instructor Observable Behaviors (IOBs) demonstration during briefing

To sustain the briefing, the IE should demonstrate, among others, the following OBs:
• IOB 2.1 Applies TEM in the context of instruction/evaluation
• IOB 2.2 Briefs on safety procedures for situations that are likely to develop during instruction
• IOB 2.6 Briefs on training devices or aircraft limitations that may influence training, when applicable
• IOB 2.9 Manages time, training media and equipment to ensure that training objectives are met
• IOB 3.2 States clearly the objectives and clarifies roles for the training
• IOB 3.4 Applies instructional methods as appropriate
• IOB 3.6 Adapts the amount of instructor inputs to ensure that the training objectives are met
• IOB 3.8 Continuously assesses trainees’ competencies
• IOB 3.12 Provides positive reinforcement
• IOB 4.3 Manages trainees’ barriers to learning
• IOB 4.4 Encourages engagement and mutual support
• IOB 4.5 Coaches the trainees
• IOB 5.2 Ensures that the trainees understand the assessment process
• IOB 5.4 Assesses trainees’ competencies

5.1.3 Recommended briefing sequence

The following guideline for CBTA training events was developed by IATA experts, which may be adopted
and used for EBT training events.

68
 IE Conduct of EBT

Recommended BRIEFING sequence for CBTA including EBT training events

Introduction – Health and safety
– If applicable, open questions from previous training sessions
– General overview of the training event
Training Objectives – Which competencies are to be trained?
– Which training topics/tasks/events are chosen to train the competencies, why are they
suitable?
– Which level of performance is expected (or what the trainee should be able to do at the
end of the training event)?
– Which conditions apply for the training?
○ Operational and environmental context
○ Tools, systems, equipment (Aircraft/FSTD, Manuals, e-tools, etc.)
○ Level of support provided by the instructor
TEM focus – Which threats can be anticipated, based on the operational/environmental context?
in accordance with – Which common errors could happen during the training?
training objectives
– Which potential reductions of safety margins could develop from the mismanagement
(selection of task/
of threats and errors?
event)
– Which competencies and OBs are critical to manage the expected threats and common
errors and to recover from potential reduction of safety margins?
Human factors/ – Where necessary, discuss strategies to absorb and adapt to disruptions.
resilience
Detailed tasks – Discuss, if needed: flight profile, maneuvers/air exercises, procedures, technical as-
analysis pects, conditions, flight parameters, …
Summary – Summarize the key aspects with confirmation of trainees’ understanding
Feedback – Ask for feedback regarding the briefing content

5.2 FSTD training

Once EBT is implemented, the pilots are provided with focused opportunities to practice procedures
under normal, abnormal, and training conditions. Pilots, through normal procedures, are provided with the
opportunity to practice specific behaviors every time they fly. This promotes the development of compe-
tencies that an operator has identified as essential for good performance within its operational environ-
ment. In addition, pilots are provided with the opportunity to practice good behaviors under emergency
and abnormal conditions when undergoing EBT training in an FSTD.
IEs should balance their attention between both PF and PM roles and maximize learning opportunities,
which are often revealed when both pilots are busy with particular tasks, sometimes to the exclusion of
effective flight path monitoring.
As a reminder from Section 4, Step 3, the template below proposes the potential instructional methods to
be applied by the IE during an EBT module.

EBT Module Phase Evaluation Phase (EVAL) Training Phases (MT / SBT)
Level of support from the IE N/A Low to High
Instructional Method The IE provides guidance for the con- All instructional methods depending
duct of the flight profile of the EVAL on the pilot’s needs

5.2.1 Evaluation (EVAL) phase

The purpose of the EVAL phase is to:
• Observe and assess pilot competencies
• Collect data to further develop and validate the effectiveness of the training system, and
• Identify individual training needs

 69
 Evidence-Based Training Implementation Guide

IEs should be assigned to carry out assessments to determine that all required competency standards
have been achieved. On completion of the EVAL any areas that do not meet the required competency
standard should become the primary focus of subsequent training. When the required competency stan-
dard has been demonstrated across all the competencies, the IE should select the competency(ies) to be
further developed to achieve the highest level of performance.
During the EVAL of the session, the IE should not give any instruction to the pilots or interrupt. Instead, the
IE should focus on observation, running the scenario and playing the role of external parties (ATC, cabin
crew, etc.) where necessary.
If the IE must intervene, the effect of this intervention on the pilot’s performance should be taken into
account.

5.2.2 Maneuvers Training (MT) Phase

The purpose of the MT is to practice and develop the handling skills necessary to fly critical flight maneu-
vers and associated procedures, according to predetermined performance criteria established by the
operator or training organization. Flight path control may be accomplished by a variety of means, including
manual aircraft control and the use of auto flight systems.
The MT, as a separate phase, allows training to be accomplished with greater efficiency. It does not con-
sist of LOFT-style training but allows the time to practice and improve performance in motor skill-based
exercises.
The IE should behave as an active trainer, utilizing learning opportunities whilst ensuring that the desired
level of competency is achieved.
Experience has demonstrated that frequent repositions during MT are often criticized by IEs and pilots.
Therefore, sufficient time and standardized processes (e.g., responsibilities for aircraft system/configura-
tion changes, requirements to brief the new position/condition and next maneuver) should be promulgated
for the IEs and pilots to accommodate with the frequent repositioning imposed by the MT phase.

5.2.3 Scenario-Based Training (SBT) Phase

The purpose of the SBT is to develop the competencies for effective management of threats and errors to
enhance the pilots’ ability to cope with both predictable and unforeseen situations.
The IE should choose scenarios or scenario elements that focus on specific competencies, based on the
competency-specific performance observed during the EVAL phase. These options should be integrated
by module design, as indicated in Section 4, Step 10: Module design.

70
 IE Conduct of EBT

Operator example for competency-specific scenario options

Focus competency map SBT 1 SBT 2 SBT 3 SBT 4

a b c a b c a b c all options
0 KNO Application of Knowledge X X X X X X X X X X
1 PRO Procedure Application and X X X
Compliance
2 COM Communication X X X
3 FPA Flight Path Management X X X
– Automation
4 FPM Flight Path Management – Manual X X X X X
5 LTW Leadership and Teamwork X X
6 PSD Problem Solving and Decision X X
Making
7 SAW Situation Awareness & X X
Information Mgmt.
8 WLM Workload Management X X X
Note: Application of Knowledge (KNO) as underlying competency is required for all scenarios and will be consid-
ered especially during briefing/debriefing.
Note: The scenario outlines contain references to OBs. These references are created from a course design per-
spective and are not intended for use as checklist items. Observation and assessment of other OBs of the same
or other competencies are specifically not excluded.

In contrast to the EVAL, during the SBT the IE should intervene or interrupt where necessary to enable the
development of the pilots’ competencies or enhance the learning experience.

5.3 Observable Behavior (OB) collection

The following paragraphs elaborate on the value of the collection and analysis of the level 2 grading met-
rics for any CBTA program, including EBT, as many organizations are considering different options for the
IE to effectively collect the OBs.
The importance of the OB collection relates directly to:
• The competency assessment process, which mandates that the instructor observes, records, and
classifies the OBs
• The competency assessment which is the combination of the number of OBs demonstrated and their
frequency of demonstration and the consequential outcome of the TEM
Therefore, many organizations are discussing specifically the two following options:
1. Systematically collecting the OBs demonstrated when required
2. Systematically collecting the OBs that have not been demonstrated when required
Both options, 1) and 2), are relevant as they sustain:
• The goal of any CBTA program, including EBT, and the general competency assessment principles
• The achievement of CBTA/EBT benefits by encouraging and enabling individual aviation professionals
to reach their highest level of operational capability
• The assessment method by motivating the trainees and identifying strengths and weakness (identi-
fying gaps as learning opportunities)

5.3.1 OB collection options and volume of data

Options 1) and 2) must also be challenged by assessing the volume of OBs to be collected by the IE con-
ducting the training.

 71
 Evidence-Based Training Implementation Guide

It is reasonable to assume that the majority (# 80%) of the competency assessments would conclude to
the demonstration of an adequate or an effective level of performance, while a minority (# 20%) of the
assessments would conclude to a low (minimum acceptable or ineffective) or high (exemplary) level of
performance.
Option 1) systematically collecting the OBs demonstrated when required, imposes the collection of many,
most or all the OBs for any performance level above adequate. This situation concerning approximately
80% or more of the competency assessment is not manageable by the IE and the value of further analysis
taking into consideration such volumes remains doubtful.
Option 2) systematically collecting the OBs that have not been demonstrated when required (the so called
“missing” OBs), imposes the collection of many or most of the OBs for any performance below adequate.
This situation, which represents approximately 20% or less of the competency assessment is manageable
by the IE, and the value of the analysis is obvious as it supports the implementation of remedial actions to
recover an adequate level of performance.
The following template illustrates the volume of OBs to be collected for options 1) [“demonstrated” OBs
column] and option 2) [“missing” OBs column] based on a realistic assumption. The blue and red rectangles
indicate the level of performance mandating numerous OBs collection depending on the option.

5.3.2 Third option for OB collection

This paragraph elaborates on a third option, which applies the concept of special emphasis by identifying
the critical OBs that sustain the different levels of performance.
As a reminder, the competency assessment is the combination of the number (how many dimension) of
OBs demonstrated and their frequency of demonstration (how often dimension) and the consequential
outcome of the Threat and Error Management (outcome of TEM dimension).
Experience has shown that the demonstration of:
• Adequate and above performance is always the result of the regular (or more) demonstration of the
required OBs, and
• Below adequate performance is often the result of the regular non demonstration of the required
OBs, and/or the punctual non-demonstration of specific OBs that have a direct impact on safety (e.g.,
non-demonstration of OB 5.10 Applies effective intervention strategies to resolve identified deviations,
when required)
In this context, the third option for OB collection would be that the IE identifies the OBs that have been
critical to justify the level of performance.
In the case of the adequate and above performance, the IE would select the OBs that have been demon-
strated the most frequently.
In the case of below adequate performance, the IE would select the OBs that have been “missing” most
frequently and/or the specific “missing” OBs that have led to any safety margin reduction.

72
 IE Conduct of EBT

The IE should identify those critical OBs during the classification stage of the competency assessment
process.
The value of the identification of the critical OBs sustaining the level of performance is the following:
• For the training managers, in the case of weak performance, it permits to implement effective and
efficient remedial actions by having visibility on the “missing” OBs that must be trained to recover an
adequate level of performance.
• For the training managers, in the case of adequate and above performance, it permits to identify the
trainees that have the prerequisites for specific upgrades or duties (e.g., the training department look-
ing for line instructors would favor the captains that demonstrate regularly and most frequently certain
OBs from the competency Workload Management).
• For the instructor and for any performance, it permits to sustain the debriefing of the session and to
provide actionable, potential subsequent solutions. Additionally, it permits to justify a consistent ratio-
nal for any level of performance and subsequent grading.
• For the trainees and for any performance, it permits to have clear visibility on their strengths and
weaknesses.
The following template illustrates the proposed third option for OB collection, which ensures a consistent
application of the CBTA principles and provides most relevant data to the organization, while placing a
reasonable level of effort on the instructor population.

5.3.3 Conclusion
As the training data that sustains the competency assessment and further enhancement of any CBTA
program, including EBT, is essentially based on the OBs metrics, each organization should define a policy
regarding the management of the CBTA training data and associated procedures for the collection and
recording of the OBs.
Both, options 1) and 2), which propose a systematic demonstrated or missing OB collection, have limita-
tions. Option 1) to systematically collect the OBs demonstrated when they were required is not manage-
able by the IE, due to the volume of data to manipulate. Option 2), to systematically collect the OBs that
have not been demonstrated when they were required, is manageable by the IE in terms of volume of data
but could discourage the trainee on the long run by illustrating exclusively what has been missing.
The third option, which applies the concept of special emphasis to record OBs that have been critical for
the achievement of the different levels of performance, could be a way forward as it motivates the trainee
by identifying both strengths and weakness, while encouraging and enabling pilots and IEs to reach their
highest level of operational capability.

 73
 Evidence-Based Training Implementation Guide

5.4 Debriefing
5.4.1 Debriefing objectives
The objectives of the debriefing are:
• To state whether the trainees have achieved the training objectives
• To ensure that the trainees are aware of their own performance
• To formalize the level of performance achieved by the trainees
• To ensure that the trainees have the means to improve their performance
The debriefing is also the opportunity for the IE to get direct feedback from the trainees about their
performance.
Reflecting on performance throughout an EBT module is key to the development of the pilots. Reflection
during the debrief empowers and encourages the trainees to reflect on the lessons learned once away
from the training facility.
During a debrief, the pilots are encouraged to practice competency-based self-assessment (root cause
analysis) and to deconstruct their actions to better understand their performance. It is important to em-
phasize why good performance has been demonstrated, so that the behaviors may be engrained and
repeated. Too often the IE tends to take positive outcomes for granted and concentrate on the negative
outcomes or on any area that needs improvement. In case of weak performance, the IE should commit the
trainee to identify:
• The contributing factors to the poor performance
• The solution to better perform in the future
The debriefing should be competency-based, objective/factual, comprehensive (leaving no doubt to the
outcome or what happened), prioritized (importance of what occurred, not when it occurred). There can be
no doubt that the real effectiveness of training is in the reflection phase afterwards.
EBT sessions should provide for adequate time to be directed towards assimilation of the lessons learned,
and adjustment of strategies, actions or processes prior to Session 2, and at the conclusion of the module.
The overall goal of the debrief is that pilots understand their performance and gain confidence in their
abilities.

5.4.2 Debriefing elements

The debriefing should comprise a fair and unbiased review based on observed behaviors and facts. A
debriefing is successful if the pilots have a clear understanding of their performance, particularly in areas
that can be improved.
The debriefing should be a facilitated discussion where pilots should be encouraged to assess
themselves.
The IE provides feedback to the flight crew to encourage the changes needed and provides specific rec-
ommendations to improve individual pilot’s performance.
In CBTA programs, the focus during debriefings changes from analyzing tasks/events to competencies.
The IE should facilitate the trainees’ performance self-assessment. IEs should adapt their facilitation ap-
proach to focus on questions related to the competencies, such as:
• “How good was your manual flying (FPM)?” - “How good was your Situation Awareness (SAW)?” Which,
for example, would be followed by “How did this affect your XYZ-recovery?”
• “How well did you apply your procedures (PRO)?” - “Which non-normal maneuver was especially diffi-
cult?” - “Why?” - “What could you have done to make it less difficult?”
• “How effective was your Workload Management (WLM)?” - “Which event was most challenging?”

74
 IE Conduct of EBT

The TEM model is a very effective tool during the debriefing for the pilots to better understand their per-
formance in the operational context because the pilots apply their competencies to perform their tasks,
and to manage the threats and errors in order to avoid undesired aircraft states.
Therefore, IEs are encouraged to use the TEM model to enhance the quality of the briefing and debriefing.
During the briefing, IEs should ask the pilots which threats and errors they would expect, which undesired
aircraft states could arise and, most important, which competencies they plan to apply/use, in order to
maintain the safety margins in the given scenario. During the facilitated debriefing, pilots should reflect on
how successful they were in applying their competencies as countermeasures and explore possibilities for
optimization in future scenarios.
With the consent of the pilots, animated playback systems and videos can be used to target and develop
competencies and understand individual and flight crew performance. Once the debriefing is completed,
the videos or playback systems data should be erased unless the participants agree to the contrary.

5.4.3 Instructor Observable Behaviors (IOBs) demonstration during debriefing

To sustain the debriefing, the IE should demonstrate, among others, the following OBs:
• IOB 3.8 Continuously assesses trainee’s competencies
• IOB 3.9 Encourages the trainee to self-assess
• IOB 3.11 Applies trainee-centered feedback techniques (e.g., facilitation, etc.)
• IOB 3.12 Provides positive reinforcement
• IOB 4.5 Coaches the trainees
• IOB 5.3 Applies the competency standards and conditions
• IOB 5.4 Assesses trainee’s competencies
• IOB 5.5 Performs grading
• IOB 5.6 Provides recommendations based on the outcome of the assessment
• IOB 5.7 Makes decisions based on the outcome of the summative assessment
• IOB 5.8 Provides clear feedback to the trainee

5.4.4 Debriefing sequence

The debriefing should commence with a statement as to whether the phase has been successfully com-
pleted, or if additional training is required. The IE should state the reason for additional training required
and the effect on their pilot privileges.
Note: The grading should be done after the facilitated debriefing.
The following guideline for CBTA training events was developed by IATA experts, which may be adopted
and used for EBT training events:

 75
 Evidence-Based Training Implementation Guide

Recommended DEBRIEFING sequence for CBTA training events

Introduction – State if the training objectives were achieved or not
– State the debriefing agenda and set time limits
– Solicit topics the trainees would like to cover during the de-briefing
Trainee’s – Review the competencies stated in the training objectives and ask the trainees to
self-assessment self-assess their performance (you may begin with LTW); focus on the positive
Facilitated – Replay: Let the trainees tell how the training event went and whether there were any
debriefing problems:
○ Select suitable tasks/events/occurrences of the lesson to illustrate how well the
trainees performed (OBs) in the relevant competencies
– Reconstruct: Let the trainees tell why they have done something in a certain way:
○ Use the experienced threats, potential errors and reduction of safety margins to
discuss which competencies and OBs have been applied for the TEM
○ Discuss resilience aspects, ask which behavior helped maintain or restore the safety
margins after disruptions during the lesson
– Reflect: Involve the trainees in the learning process by having them reflect on what they
have learned:
○ Ask: What did you learn? What could you have done differently? …)
– Redirect: Commit the trainees to consider the future application of their training
experience:
○ Ask: What can you do with the experience gained during this training, what would
you do differently in the future? ...)
○ Where necessary, provide recommendations for improvement
○ Summarize by highlighting “lessons learned” and emphasizing the positive
Note:
• Avoid detailed discussions about all competencies and events, focus on key behaviors
• Ensure that all trainees actively participate and receive specific feedback on their
performance
• Exercise patience, and do not be reluctant to probe into key areas where individual and/
or trainee improvement is needed
Summary – Summarize the key learning points and recommendations to the trainees
– Look ahead, what are the training objectives for the next few sessions, assign any study
items in preparation
Feedback – Ask for feedback regarding lesson design and performance of the instructor

5.5 Competency Assessment and associated Grading

The content of this chapter is applicable to the pilot and to the IE competency assessment.
IATA provides more details in the guidance material: Competency Assessment and Evaluation for Pilots,
Instructors and Evaluators.

5.5.1 Principles
The following principles should be implemented and monitored by the AOC or the ATO to ensure the ef-
fectiveness of the EBT program. The statements in bold are key elements for the IE’s initial and recurrent
standardization and for the continuous monitoring of the IE’s performance.
• Clear performance criteria are used to assess competence. The adapted competency model estab-
lishes these performance criteria.
• An integrated performance of the competencies is observed. The pilot undergoing assessment must
demonstrate all competencies and their seamless interaction with each other.
• Multiple observations are undertaken. To determine whether or not a pilot has achieved the interim
and/or final competency standard, multiple observations must be carried out.

76
 IE Conduct of EBT

• Assessments are valid. All competencies that comprise the adapted competency model must be
assessed. There must be sufficient evidence to ensure that the pilot achieves the competency and
meets the interim and/or final competency standard.
• Assessments are reliable. All IEs should reach the same conclusion when performing an assess-
ment. All IEs should be trained and monitored to achieve and maintain an acceptable level of inter-rater
reliability.

5.5.2 Process
To assess the pilot’s performance, the IE should apply the following process:
• Observe performance (behaviors) during the training or evaluation.
• Record details of effective and ineffective performance (behaviors) observed during the training or
evaluation (“record” in this context refers to the IE taking notes).
• Classify observations against the OBs and allocate the OBs to each competency (or competencies).
• Assess the performance by determining the root cause(s) for low and high performance according to
the competency framework. Low performance would normally indicate the area of performance to be
remediated in subsequent training.
The assessment should be accomplished by relating the observed pilot’s behavior to the competencies
outlined in Appendix B. The determination of the pilots’ performance should be made solely with reference
to defined standards established by the operator or training organization.
Note: Depending on the training objective, the IE guidance may indicate competencies which may be
irrelevant to be assessed or recorded. In that case, the IE will record "N/O" (not observable).
When transitioning to CBTA/EBT, IEs need to recognize that there is no continuous summative assessment
during an EBT module. The focus needs to shift away from constant evaluation. EBT involves observing,
recording, classifying, and assessing the pilot’s performance over the complete session and not evaluat-
ing each event; presenting the ORCA process as a video rather than a snapshot. This approach allows for
the collection of evidence to develop the trainee’s learning curve throughout and across EBT modules.

5.5.3 Method
IATA recommends applying the following methodology to ensure the maximum level of consistency and
objectivity to assessments performed in a CBTA program.
To assess how well the pilot demonstrated the competency during training or evaluation, the IE should
assess the associated OBs of each competency against the following dimensions, by determining:
• How many OBs the trainee demonstrated when they were required
• How often the trainee demonstrated the OB(s) when they were required
• What was the outcome of the threat management and error management relating specifically to the
competency being assessed
The competency assessment (HOW WELL) is the combination of the number of OBs demonstrated and
their frequency of demonstration and the consequential outcome of the TEM relating specifically to the
competency being assessed.

 77
 Evidence-Based Training Implementation Guide

• The “HOW MANY” dimension provides evidence related to the acquisition of the competency
• The “HOW OFTEN” dimension provides evidence related to the robustness of the competency
• The “Outcome of TEM” dimension provides evidence related to the effectiveness of the competency
as individual and team countermeasure against the threats and errors

Word Pictures of “HOW MANY” dimension Word Pictures of “HOW OFTEN” dimension
The following word pictures support the compe- The following word pictures support the compe-
tency assessment methodology by providing a tency assessment methodology by providing a
scale for the “HOW MANY” dimension regarding a scale for the “HOW OFTEN” dimension regarding a
number of OBs demonstrated when required: frequency of OBs demonstrated when required:

HOW MANY HOW OFTEN

few, hardly any rarely
some occasionally
many regularly
most regularly*
all, almost all Always, almost always
regularly* This regularly* is interpret-
ed as very often

Word Pictures of “OUTCOME of TEM” dimension

The following word pictures support the competency assessment methodology by providing a scale for
the “Outcome of TEM” dimension relating specifically to the competency being assessed:

Note:
• safe*: This word picture (safe*) illustrates a more pro-active safety level.
• or could have led to** must be used to:
– Integrate the outcome of TEM dimension when the conditions of training are significantly limited,
e.g., classroom, part task trainer, …

78
 IE Conduct of EBT

– Ensure that the OUTCOME of TEM dimension relates specifically to the competency being
assessed
During the competency assessment, the TEM model assists the IE understand the interrelationship be-
tween safety and the trainee’s performance in dynamic and challenging operational contexts.

5.5.4 Competency Assessment: Abbreviated word pictures

The competency assessment is illustrated by the “HOW WELL” terminology, which reflects the lowest
level of each dimension (“HOW MANY” - “HOW OFTEN” - “OUTCOME of TEM”).

5.5.5 Grading
Definition and Process
Grading means that the IEs allocate a number to the performance achieved by the pilot during the com-
petency assessment, relating specifically to the competency being assessed. The aim of the grading is to
facilitate a harmonized and consistent training data analysis.
Therefore, IEs should consider grading as a method to collect training data, not as a means to provide
feedback to the trainees.

When and which level to grade

In an EBT module, formative and summative assessments are performed. However, there is a distinction
between assessments within mixed EBT implementation and baseline EBT:
• In mixed EBT, the license is revalidated after the EVAL and MV phases of the module. The outcome of
Session 1 is therefore a PASS/FAIL decision of the IE regarding the proficiency check. Generally, flight
tolerances are still the regulatory benchmark. However, the pilot’s competency grades also serve as a
formative assessment because they form the basis for the subsequent training during the SBT phase.

 79
 Evidence-Based Training Implementation Guide

• In baseline EBT, formative assessments are performed during the entire module, with summative
assessment(s) as defined by the operator and/or the authority. IATA recommends a summative assess-
ment at the end of the EBT module.
The following templates provide recommendations regarding the training data (metrics) to be collected
and the level of performance to be achieved. The templates indicate the type of assessment (formative or
summative, as described in Chapter 2.8.1) depending on the organization of the EBT modules.
• Option 1 illustrates the IATA recommendation where the EBT module is conducted during two consec-
utive days with the same IE.
• Option 2 illustrates an EBT module conducted during two non-consecutive days, which allows line op-
erations between Session 1 and Session 2.
Option 1: EBT module conducted during two consecutive days by the same IE (IATA recommendation)

OPTION 1 – valid during mixed implementation

EVAL MV SBT Line operations
Type of assessment Formative Formative Summative
Traditional profi- PASS/FAIL
ciency check
Metrics to be Level 0 Level 0
collected Level 1 Level 1
Level 2 Level 2 Level 2
Level 3 Level 3 Level 3
Required level of Interim competency standard Final competency
performance standard

OPTION 1 – valid during baseline EBT

EVAL MT SBT Line operations
Type of assessment Formative Formative Summative
Metrics to be Level 0
collected Level 1 Level 1
Level 2 Level 2 Level 2
Level 3 Level 3 Level 3
Required level of Interim competency standard Final competency
performance standard

Option 2: EBT module conducted during two non-consecutive days allowing line operations between
Session 1 (generally dedicated to EVAL and MT) and Session 2 (dedicated to SBT)

OPTION 2 – valid for all EBT stages

EVAL MT/MV Line operations SBT
Type of assessment Summative Summative Summative
Traditional profi- PASS/FAIL
ciency check
Metrics to be Level 0 Level 0 Level 0
collected Level 1 Level 1
Level 2 Level 2 Level 2
Level 3 Level 3 Level 3
Required level of Final competency standard Final competency
performance standard

80
 IE Conduct of EBT

The essential differences between option 1 and option 2 are the following:
• Option 1 permits to define an interim competency standard that may be lower compared to the recom-
mended final competency standard which is “adequate” (corresponding to a grade 3).
Example: Under option 1, it is possible that the interim competency standard would be “minimal
acceptable” (corresponding to a grade 2) for few competencies as the IE should further develop
pilot competencies during the SBT phase up to the achievement of the final competency standard
recommended to an “adequate level of performance” (corresponding to grade 3).
• Operators electing option 2 should define the interim competency standard to “adequate” (corre-
sponding to grade 3), as the pilots are supposed to conduct line operations after the first session com-
posed by the EVAL and the MT phases.
• Consequently, the type of assessment is also different depending on the chosen option: For option 1,
a formative assessment is possible after the EVAL and MT phases as the same IE conducts the entire
EBT module, while a summative assessment is mandatory for option 2 as the formalization of the pilot
competence is necessary for the return to line operation.
Notes:
1. During the MT/MV phase, only a limited number of competencies may be observed and graded.
Therefore, no level 1 metrics are considered to be collected.
2. European EBT regulations for baseline EBT presently require that grades be determined during each
EBT module as follows:
– EVAL – overall performance of the phase for each competency at level 1 grading metrics.
– MT – overall performance of the phase at level 0 grading metrics. When the phase is graded “not
competent”, it requires level 2 grading metrics.
– SBT – overall performance of the phase for each competency at level 1 grading metrics, unless just
culture and the necessary non-jeopardy environment during training may be compromised. In that
case, level 0 grading metrics.
Where any competency is graded below the minimum acceptable level of performance, additional level 2
grading metrics must be recorded.

5.6 Instructional methods

5.6.1 Instruction and level of support by the IE
In order to develop the pilots’ competencies effectively, IEs use a variety of instructional meth-
ods. Curricula should contain guidance to match the training objectives with the optimum method of
instruction.
IEs should master all available methods and should have the flexibility to select the most appropriate one
to achieve the training objectives.
As indicated in 2.6.2, the conditions related to the amount of support or assistance a trainee can expect
from the IE shall be defined for each training event as per ICAO provisions. Therefore, IATA recommends
correlating the IE’s level of support to recognized instructional technics.
The template below illustrates the allocation of three different levels of IE support (Low, Medium and High)
to five instructional technics.

 81
 Evidence-Based Training Implementation Guide

Instructional
Description Level of IE support
method
Demonstrate (Show) The instructor or the training media performs or directs the execu- High
tion of a task, procedure, or maneuver to the trainees. In addition,
facilitation is used to verify knowledge and to check understanding.
Trainees will demonstrate the acquisition of the competencies.
Explain (Tell) The instructor, or the training media, provides information verbally to High
the trainees or (whenever facilitation seems not suitable in particular
circumstances) recalls/reminds them of key points already acquired
during the course. Questions are used to either establish current
knowledge or to check understanding. Trainees will demonstrate the
acquisition of the competencies.
Facilitate Facilitation technique refers to an active training method, which uses Variable, depending
effective questioning, listening and a non-judgmental approach, and on the trainees needs
is particularly effective in developing skills and attitudes, assisting
trainees in developing insight and their own solutions, resulting in
better understanding, retention and commitment.
Discover with The instructor, or the training media, provides trainees with objec- Medium
assistance tives and conditions. Using their existing competencies, trainees
“figure out” appropriate solutions and means to achieve the ob-
jectives. The instructor intervenes only when necessary to ensure
achievement of the objectives and to minimize inefficiency.
Discover without The instructor or the training media provides trainees with objec- Low
assistance tives and conditions. Using their existing competencies, trainees
“figure out” appropriate solutions and means to achieve the objec-
tives. The instructor or training media verifies the outcomes.

During the training phases of an EBT module, the IE may assist the pilots in their competency develop-
ment; however, during the EVAL phase and at the end of the module, the pilots should show the required
level of performance without assistance from the IE.

5.6.2 Demonstrate/explain versus facilitate/discover

The following elaborates on the different instructional methods, why they are needed and guidance to
apply them.
Facilitation means that trainees are given the opportunity to discover what they are doing and the effect
it has on others and on the task, so that they can make the decision to alter their behavior or reinforce any
positive behavior.
While the techniques with a high level of instructor support target a direct execution by the trainee, facili-
tation and discover trigger an indirect execution; the trainees are encouraged to finding their own solution
and then to execute it. The better the instructional questioning and listening, the more effective the train-
ees’ understanding, execution and retention.
There are three main reasons why the facilitation method is instrumental in CBTA/EBT:
• As an educational principle, any training that aims at attitude or behavioral change will be less effective
through telling and explaining, than by enabling pilots to find out an adequate behavior themselves.
CBTA extended the traditional training of aviation personnel beyond a pure task-based focus towards
developing a set of technical and non-technical competencies. While technical competencies can be
effectively trained by traditional techniques (show/tell/explain, etc.), the non-technical competencies
(soft skills) are much better developed when the IE triggers the trainees’ capacity for self-analysis,
which allows the trainees to find the answers by themselves.
• Any training event should prepare and enable the trainees to self-assess their performance. This
enables them to develop in day-to-day operations outside the learning environment. The aim of train-
ing by facilitation is to trigger an intuitive practice by the trainees, so that they are able to assess and
“grade” themselves spontaneously, even without an IE being present.

82
 IE Conduct of EBT

• Facilitation during debriefing of a training event may support the instructor in the root cause analysis of
certain outcomes of crew activity or behavior and to address the correct competencies. Such facilita-
tion is particular useful if the instructor considers a variety of possible triggers for an observed crew
behavior.
The facilitation technique is more effective than the showing and telling technique because the pilot’s
involvement and experiences are actually part of the learning process. Showing and telling complement
facilitation if the trainees need more IE support to achieve the training objective(s).
The facilitation technique is not just for the poor performer or for the development of attitude but can be
equally used to reinforce effective behavior because it gives trainees an understanding of why they are
good, which encourages their continued development.
The first crucial prerequisite for effective facilitation is the IE’s capability to model a desired output into
questions and a dialogue to stimulate the trainees understanding and own commitment. The second
prerequisite is to apply facilitation only with enough time to allow mental processing by the trainees.
Therefore, facilitation should be the preferred technique during briefing and debriefing of a training event.
During a training event the IE should control the session progress in a way that allows facilitation when
feasible
The differences between the different instructional methods are highlighted in the following table.

Demonstrate/Explain Facilitate/Discover
How does the instructor Telling, showing (demonstrating) Enabling the pilots to find the solution
communicate? by themselves
What is the aim? Transfer knowledge and develop skills Develop insight/self-analysis to find a
solution or behavioral change
Who knows the subject? Instructor Both instructor and pilot
Who has the experience? Instructor Both instructor and pilot
What is the relationship? Authoritarian Equal
Who sets the agenda? Instructor Both instructor and pilot
Who talks the most? Instructor Pilot
What is the timescale? Finite Infinite
Where is the focus? Instructor – task Pilot – performance and behavior
What is the workload? Moderate High
What are instructors’ thoughts? Judgmental Non-judgmental
How is progress evaluated? Observation Guided self-assessment

5.6.3 Facilitation skills

The various skills required to use facilitation as an effective instructional technique are as follows.

Questioning
Asking the right questions at the right time is a fundamental skill of facilitation and these are examples:

Type Purpose Response Example

Open To get a more accurate and Unknown but they will say “What, when, why, where, who,
fuller response. more than a few words. how”
Closed To check understanding and to Can be “Yes”, “No” or specific “Did you, were you, had you”
control the discussion. data.
Probing/ To obtain further information. More in depth response. “Tell me more, why was that,
building explain”
Summarizing To confirm agreement. Yes “Is what you mean, have you
agreed?”

 83
 Evidence-Based Training Implementation Guide

Things to avoid in questioning:

• Leading, e.g., “Would you not agree that…?”
• Multiple, e.g., “What did you say next and what was displayed on the radar...?”
• Rhetorical, e.g., “You will remember what I am saying, won’t you...?”
• Ambiguous, e.g., “How long was it before the leak started...?”

Active listening
It has often been said that hearing is done with the ears whereas listening is done with the mind. The term
active listening means that a person concentrates carefully on what is being said, in order to fully under-
stand the other person. LISTEN, the following mnemonic, helps to capture key points:
1. Look interested
2. Inquire with questions
3. Stay on target
4. Test understanding
5. Evaluate the message
6. Neutralize your thoughts, feelings and opinions

Body language
Reading body language and managing one’s own body language are essential skills when facilitating.
An instructor should know when a trainee is uncomfortable, confused, interested, distracted or bored.
Furthermore, it is important that an instructor is able to manage his/her own body language so that it
is congruent with what they are saying and so that the messages being transmitted are accurate and
consistent.

Observation of behavior
The ability to observe and discuss behavior and attitudes rather than technical issues is an important skill
that trainers need to develop to become effective at facilitation. It is not unusual for instructors who are
not used to facilitation to feel a certain amount of embarrassment when first attempting this. As attitudes
(as exhibited by behaviors) are a less precise competency to measure, there is no better way of demon-
strating appropriate behavior than role-modeling. This is because the trainee can observe what this be-
havior is and experience the positive effects on themselves. Furthermore, in order to maintain credibility
as an instructor, it is important that appropriate behavior is demonstrated as a form of role modeling.

Giving and receiving criticism

There may be occasions when it is appropriate and constructive to give trainees direct criticism, and this
should be carefully handled. Similarly, an instructor should be able to receive criticism well, in order to
develop.

Continuous development
In order to ensure continuous improvement in facilitation skills, the recommended method is to seek feed-
back from trainees. This must be done genuinely otherwise nothing useful may be gained. A measure of
whether an instructor is doing this well is whether in fact any feedback is given.

84
 IE Conduct of EBT

Some advice for effective facilitation in a debriefing

Dos Don’ts
– Give an introduction – Miss the introduction – it is the most common way to
– Encourage self-analysis (research indicates that it is spoil facilitative training
the best form of learning) – Lecture
– State that participation from the trainees is needed – Use your chronological agenda
– Allow pilots to set the agenda order by asking: – Short-change high performing crews with a quick
• Which parts of the session they want to discuss? debrief
• What went well? – Interrupt
– Use open questions (who, where, when, what, why, – Answer your own questions (if they don’t reply, in-
how) stead reword the question)
– Deepen the discussion with supplementary ques- – Just use question and answer
tions – let them analyze – Do the thinking for them
– Ask what happened / why it happened / what could
we improve on
– listen and encourage
– Use names, nods, smiles, eye contact
– Sit forward to show interest
– Use silence/pauses (sit back and allow them time to
think for several seconds)
– Mix instruction with facilitation for issues on which
the trainees do not have the knowledge
– Summarize the discussion to meet training aims
Self-check
– Who is talking most – you or them?
– Have you used at least two questions per issue (to deepen discussion)?
– Are the pilots doing the analysis themselves?
– Are the training points being covered?
– Have the pilots spoken to each other?
– Has positive behavior been reinforced?

 85
 Evidence-Based Training Implementation Guide

Appendix A - Safety Risk Assessment (SRA) examples

This Appendix is intended to support operators implement EBT, Step 1: Implementation plan and safety risk assessment.

A.I Operator sample before start of mixed implementation (excerpt)

Evaluate Acceptance and
Evaluate Enhanced
Existing Existing Risk Enhanced Enhanced Risk Further action reqd Implementation
Task Frequency People/ Existing Enhanced Safety Severity
Severity Category Index Probability Category Index by whom & when of enhanced Documentation of
Ref Identified Task (select from Hazard Consequence Equipment at Probability Existing controls Risk Control & (select from
(select from (auto-calculates (select from (auto-calculates (see options list Safety Risk Change Requirements
drop-down list) risk (select from Mitigations drop-down
drop-down from sev & prob) drop-down list) from sev & prob) below) Control &
drop-down list) list)
list) Mitigations

Hazard 1. Errors Hazard 1- Course construc- Hazard 1-

or ommissions Consequence tion achieved Consequence 1-
occur during 1. Mandatory drawing on Mitigation 1.
move from pre- OEM or Regulator existing course For each ‘revised’
viously ‘Original specified training footprints. course, a ‘Delta’
Equipment event/item Mandatory between course
Manufacturer inadvertently om- training events/ compared
(OEM)’/Regulator mitted/removed items specified training items
constructed from the new EBT by either OEM or was constructed.
courses to an course. Regulator remain These ‘Deltas’
Evidence Based in place. were based on
training (EBT)/ one of three
Emirates Airline parameters:
constructed 1. Any training
course. event reduced in
Transition from frequency from Review of Course
‘OEM/Legacy’ ‘legacy’ course. structure and con-
Periodic (Six
to Evidence 2C 2. Any training Extremely 1D tent due February
1 Low frequency (L) Course students Major (C) Improbable (2) Minor (D) Monthly) Review
Based Training Review event moved Improbable (1) Acceptable 2018, in line with
required.
(EBT) Transition to reduced EBT Implementation
Courses fidelity Synthetic Review.
Training Device
(STD) (ie. FFS
to FBS).
3. Any training
event potentially
removed from
the course. For all
negative ‘Delta’
training events
alling into any
of the above
categories, an
individual Risk
Assessment (RA)
was conducted.

Hazard 1. Hazard 1- Current Hazard 1-

Training Consequence Examiner/ Consequence 1-
Department 1. Benefits and Instructor Mitigation
examiners and advantages of ‘Standardisation’ 1. ‘Trainer
instructors fail EBT ‘competency meetings Refresher
to recognise based training’ take place Seminar (TRS)’
and engage not realised. every ‘Phase’ contributes
with Evidence (6mths). Training to further
Ongoing (ie. annually)
Based Training concepts and ‘standardisation’ Periodic (Six
Instructor High frequnecy Operating Flight 3C 2D review of instructor
2a concept and Major (C) Remote (3) methodologies of instructors and Minor (D) Improbable (2) Monthly) Review
Engagement (H) Crews Review Acceptable training guidance
methodology. reviewed and allows further required.
and best practise.
discussed. development
and training of
instructors in
respect of EBT
programme.
TRS occurs
annually for all
instructors.

86
 Safety Risk Assessment (SRA) examples

Evaluate Acceptance and

Evaluate Enhanced
Existing Existing Risk Enhanced Enhanced Risk Further action reqd Implementation
Task Frequency People/ Existing Enhanced Safety Severity
Severity Category Index Probability Category Index by whom & when of enhanced Documentation of
Ref Identified Task (select from Hazard Consequence Equipment at Probability Existing controls Risk Control & (select from
(select from (auto-calculates (select from (auto-calculates (see options list Safety Risk Change Requirements
drop-down list) risk (select from Mitigations drop-down
drop-down from sev & prob) drop-down list) from sev & prob) below) Control &
drop-down list) list)
list) Mitigations

Hazard 1-
Consequence 1-
Mitigation 2.
Publication of
the Emirates
Flight Training
- Evidence
Based Training
Instructors
Handbook
Ongoing (ie. annually)
(Conversion and
3C 2D review of instructor Periodic (Annual)
2b As above As above As above As above As above Major (C) Remote (3) As above NaC Courses) is Minor (D) Improbable (2)
Review Acceptable training guidance Review required.
a comprehensive
and best practise.
guide and refer-
ence document
that provides
a detailed
approach to EBT
and competency
based training. It
will be reviewed
and revised as
applicable.

Hazard 1. Hazard 1- Current transition Hazard 1-

Training deliv- Consequence course referenc- Consequence 1-
ered on Synthetic 1. Flight Crews es OEM sourced Mitigation
Training Device faced with unfa- technical data. 1. Difference
will be at a dif- miliar or unseen Revisions and between
ferent (previous systems or fault updates to STD modification
or earlier) modi- resolution issues, are provided by states of STD
fication state to to that trained STD OEM. and aircraft have
that on in service during transition been identified
aircraft. course. and significance
understood. Review of current
Medium frequen- Flight Crews/In Reasonably 4D Some STD up- 3D STD standards and
3a Technology Minor (D) Minor (D) Remote (3) Review Required
cy (M) service aircraft. Probable (4) Review dates will negate Review equivalent aircraft
current STD/ modification states
aircraft delta.
Alternative &
additional
training modules
available to
bridge gap where
continued differ-
ence in STD and
line aircraft may
be experienced.

Hazard 1- Current transition Hazard 1-

Consequence course referenc- Consequence 2-
2. Instructor es OEM sourced Mitigation 1.
reference technical data. Where differenc-
material unable Revisions and es in STD and
to identify differ- updates to STD aircraft modifi-
Review of current in-
ences between are provided by cation states are
structor provisioned
STD and aircraft Flight Crews/In Reasonably 4D STD OEM. identified in "1" 3D
3b As above As above As above Minor (D) Minor (D) Remote (3) training material and Review Required
modification service aircraft. Probable (4) Review above, additional Review
equivalent aircraft
states. training material
modification states
required for deliv-
ery by instructor
to trainees that
details and
highlights these
differences.

 87
 Evidence-Based Training Implementation Guide

Evaluate Acceptance and

Evaluate Enhanced
Existing Existing Risk Enhanced Enhanced Risk Further action reqd Implementation
Task Frequency People/ Existing Enhanced Safety Severity
Severity Category Index Probability Category Index by whom & when of enhanced Documentation of
Ref Identified Task (select from Hazard Consequence Equipment at Probability Existing controls Risk Control & (select from
(select from (auto-calculates (select from (auto-calculates (see options list Safety Risk Change Requirements
drop-down list) risk (select from Mitigations drop-down
drop-down from sev & prob) drop-down list) from sev & prob) below) Control &
drop-down list) list)
list) Mitigations

Hazard 2. EBT Hazard 2- Existing training Hazard 2-

Concept identi- Consequence 1. profile and re- Consequence 2-
fies a Knowledge, Additional equip- sourcing capable Mitigation 1.
Skill or Attitude ment required. of adapting to Where revisions
(KSA) currently Additional train- training need. and develop-
not supported ing resources ments to training
by existing required. programmes
equipment. are identified,
none have been
the subject of
‘immediate’ Continuous review
implementation. and assessment of
As such, a period new and innovative
of transition and training techniques Periodic (Six
Flight Crews/In 3C 2D
3c As above As above Major (C) Remote (3) introduction Minor (D) Improbable (2) and equipment will Monthly - TRC)
service aircraft. Review Acceptable
is available be required. This Review required.
to resource should be available
equipment, within the Training
establish training Department.
expertese and
roll out the re-
vised/enhanced
programme.
(UPRT is a
relevant example
of such training
equipment and
training resource
introduction).

Hazard 1. Hazard 1- Internal Audit Hazard 1-

Emirates Airline is Consequence Programme pro- Consequence 1-
the industry ‘lead’ 1. External (and vides a Quality Mitigation 1.
with respect to Internal) Audit or- Compliance re- Relevant and
introduction and ganisations may view and remains focused training
implementation be unfamiliar with in place. can be provided
of EBT in both audit processes to both external
Review of current
Recurrent and when measuring and internal au-
STD internal and Internal and external
Conversion against an EBT ditors, to ensure Periodic (Six
Medium frequen- 3D 2D external audit audit standards
4a Organisation Course meth- profile. Negligible Minor (D) Remote (3) the elements of Minor (D) Improbable (2) Monthly) Review
cy (M) Review Acceptable standards and rec- reviewed against current
odology. Other an EBT training required.
ommended practises processes
organisations in programme are
would be required.
the wider ‘exter- understood.
nal environment’
may not recog-
nise and have
the expertese to
understand EBT
methodology.

Hazard 1-
Consequence 1-
Mitigation 2.
IOSA Standards
Review of current
Manual (ISM)
STD internal and Internal and external
Edition 11, pub- Periodic (Six
3D 2D external audit audit standards
4b As above As above As above As above As above Minor (D) Remote (3) As above lished September Minor (D) Improbable (2) Monthly) Review
Review Acceptable standards and rec- reviewed against current
2017 includes required.
ommended practises processes.
an Evidence
would be required.
Based Training
(EBT) element
and guidance
material.

Hazard 2. Hazard 2- Organisation Hazard 1-

Emirates Airline Consequence previously Consequence 1-
‘corporate 1. Corporate demonstrated Mitigation 1.
culture’ is resis- Culture restrains ability for change Understanding
tant to change and negates during rapid cor- and acceptance Review and
with respect to introduction of porate growth. that Emirates understanding of
introduction and EBT, preventing Airline can be other departments
implementation full benefits being Individuals and 3D a ‘Learning 2D understanding of For review by
4c As above As above Minor (D) Remote (3) Minor (D) Improbable (2) N/A
of EBT. realised. Organisation. Review Organisation’, Acceptable the EBT Project, its Project Manager.
specifically with methodology and
respect to EBT positive benefits for
implementation, the airline.
will contribute
to a successful
programme
rollout.

88
 Safety Risk Assessment (SRA) examples

Evaluate Acceptance and

Evaluate Enhanced
Existing Existing Risk Enhanced Enhanced Risk Further action reqd Implementation
Task Frequency People/ Existing Enhanced Safety Severity
Severity Category Index Probability Category Index by whom & when of enhanced Documentation of
Ref Identified Task (select from Hazard Consequence Equipment at Probability Existing controls Risk Control & (select from
(select from (auto-calculates (select from (auto-calculates (see options list Safety Risk Change Requirements
drop-down list) risk (select from Mitigations drop-down
drop-down from sev & prob) drop-down list) from sev & prob) below) Control &
drop-down list) list)
list) Mitigations

Hazard 1. Hazard 1- Existing Training Hazard 1-

Interaction Consequence programme Consequence 1-
between internal 1. Commercial remains in place. Mitigation 1.
departments and/or Training Switchover date Whilst change
(Flight Programme can be altered. from current
Operations, disrupted or conversion
Network Control compromised. course to EBT
Centre (NCC), planned, this date
Course plan and
Safety and Training Courses can be moved
Medium frequen- 3E 2E schedule under Depends on Risk out-
5 Interfaces Regulatory) delayed, training Negligible (E) Remote (3) and is not con- Negligible (E) Improbable (2) None required.
cy (M) Acceptable Acceptable review by Project come. Probably nil.
prevents or output reduced. tingent on other
Manager.
hinders Project departments or
completion on their input with
schedule or pro- respect to EBT
vides significant rollout. There
delay. is no impact
anticipated with
respect to de-
layed courses.

Hazard 1. Hazard 1- Project oversight Hazard 2-

Politically sen- Consequence via SVP Trg, dela- Consequence 1-
sitive to internal 1. Reputational gated to Training Mitigation 1.
stakeholders. damage, in Standards Project team
Oversight
event Project Management drawn from
Political Visibility Training Courses management and
Medium frequen- not delivered 3E team to existing fleets 5E
6a and Stakeholder delayed, training Negligible (E) Remote (3) Negligible (E) Frequent (5) review available via None required. None identified.
cy (M) or significantly Acceptable implement. and Training Acceptable
Involvement output reduced. Standards Training
delayed. Standards to
Management.
ensure day to
day management
and project
progression.

Hazard 2. Hazard 2- Current contin- Hazard 2-

Independent Consequence 1. uous dialogue Consequence 1-
Oversight or Project hindered with Regulatory Mitigation 1.
Significant out- or delayed due Authority Project Manager
reach/external independent/reg- precludes (PM) will provide
During project progres-
input required. ulator oversight requirement for additional Project Manager
sion, any changes or
Medium frequen- restrictions and 2D further and ad- updates and pro- Extremely 1E (PM) responsible for
6b As above Negligable Minor (D) Improbable (2) Negligible (E) None requried. Regulator influenced
cy (M) review. Acceptable ditional dialogue gression reports Improbable (1) Acceptable Regulator comunica-
inputs will be introduced
with respect to to the Regulator, tions and dialogue.
and addressed by PM.
this Project. ensuring project
slowdown or
curtailment
unlikely from this
source.

Hazard 2-
Consequence 1-
Mitigation 2.
With respect to
During project progres-
course construc- Project Manager
sion, any changes or
Medium frequen- 2D tion; courses Extremely 1E (PM) responsible for
6c As above As above As above Negligible Minor (D) Improbable (2) As above Negligible (E) None required. Regulator influenced
cy (M) Acceptable are based on Improbable (1) Acceptable Regulator comunica-
inputs will be introduced
industry data and tions and dialogue.
and addressed by PM.
no Regulatory
or OEM require-
ments have been
removed.

Hazard 3. Hazard 3- Relevant Aviation Hazard 3-

Additional Consequence Authorities in Consequence 1-
External 1. Project this respect are Mitigation 1.
Authority (for considered IATA, EASA, FAA Project Manager
example: IOSA) ‘Non-Compliant’ and Australian(?). (PM), with sup-
Impact Review due change/mod- Continuous port of Training
required. ification of ‘Third Regulatory Department
Party’, but not Compliance Regulatory and During project progres-
Project Manager
immediate over- and Regulator Compliance sion, any changes or
Medium frequen- 2D Extremely 1E (PM) responsible for
6d As above sight, Aviation Negligable Minor (D) Improbable (2) Directives review Officer (RACO) Negligible (E) None required. Regulator influenced
cy (M) Acceptable Improbable (1) Acceptable Regulator comunica-
Regulatory body on publication should review inputs will be introduced
tions and dialogue.
(ie IATA, FAA). ensures compli- input of Aviation and addressed by PM.
ance issues are Authority
recognised and directives and
implemented. adopt/implement
as required. This
ensures continu-
ous compliance
is retained.

 89
 Evidence-Based Training Implementation Guide

Evaluate Acceptance and

Evaluate Enhanced
Existing Existing Risk Enhanced Enhanced Risk Further action reqd Implementation
Task Frequency People/ Existing Enhanced Safety Severity
Severity Category Index Probability Category Index by whom & when of enhanced Documentation of
Ref Identified Task (select from Hazard Consequence Equipment at Probability Existing controls Risk Control & (select from
(select from (auto-calculates (select from (auto-calculates (see options list Safety Risk Change Requirements
drop-down list) risk (select from Mitigations drop-down
drop-down from sev & prob) drop-down list) from sev & prob) below) Control &
drop-down list) list)
list) Mitigations

Hazard 4. Hazard 4 - Hazard 4 -

Aircraft Original Consequence 1. Consequence 1-
Equipment Project hindered Mitigation 1.
Manufacturer or delayed due With one OEM
(OEM) do not OEM resistance (Airbus) already
recognise or en- to modification/ moving toward an
One OEM already During project progres-
dorse the Project amendment EBT perspective Project Manager
Training Courses moving to an sion, any changes or
Medium frequen- Objectives. of their 2D and the other Extremely 1D (PM) responsible for
6e As above delayed, training Minor (D) Improbable (2) EBT perspective Minor (D) None required. Regulator influenced
cy (M) manufacturers Acceptable (Boeing) fully Improbable (1) Acceptable Regulator comunica-
output reduced. in their course inputs will be introduced
supportive and tions and dialogue.
construction. and addressed by PM.
actively involved
in supporting the
EK EBT Project,
likelihood of OEM
course rejection
limited.

Hazard 1. Hazard 1- Current data Hazard 1-

Reporting Consequence gathering refer- Consequence 1-
Culture risks data 1. Insufficient ences primarily Mitigation 1.
submission en- or inaccurate ASR data. Some Inclusion and
gagement from data applied to Quality Audits review of greater
flight crew. providing EBT available detail of available
‘databank’. data sources.
Enhanced data
PM Review, in
sources will
conjunction with
require accurate
Training Department
Medium frequen- Flight Crews/In Reasonably 4C and detailed 3C
7 Doctrine Major (C) Major (C) Remote (3) and Fleet. Structure None required. TBA
cy (M) service aircraft. Probable (4) Review review of FDM, Review
and essence of
Quality and
Reporting Culture
Engineering
should be reviewed
Reviews, as
well as existing
ASR data input.
All elements
will require to
be classified
into Risk Based
assessments.

Hazard 1. Hazard 1- Previous detailed Hazard 1-

Funding Consequence 1. estimate and Consequence 1-
Assessment Project delayed project resource Mitigation 1.
insufficient or or significantly assessment Project Team
further appraisal slowed in prog- conducted. participants
required. ress due funding form majority of
constraints. funding required
resource that
Reviewed by EBT
will have been
PM, with respect
applied to project
2D 2D to request for and
8 Funding Low frequency (L) Negligible Minor (D) Improbable (2) cost. Commercial Minor (D) Improbable (2) None required. None required.
Acceptable Acceptable release of resource
programme will
on month-by-month
dictate, with de-
basis.
gree of certainty
and advance
notice, whether
refinement and/
or reduction of
this available
resource will be
required.

Hazard 1. EBT rollout plan Project Plan Hazard 1-

Implementation and course is 36 months. Consequence 1-
date not commencement Implementation Mitigation 1.
achieved. not achieved. is 24 months Regular (monthly) Reviewed by SVP
from Project reviews on Trg and EBT PM on
start. Scope Project progress month-by-month
within existing in place. Time timeline. Any project
Time Scale/ 2D plan to move line continuously 2D slip should, be iden- None required unless
9 Low frequency (L) Trainees Minor (D) Improbable (2) Minor (D) Improbable (2) None required.
Schedule Acceptable implementation under review Acceptable tified early and with project timeline alters
date without and scope within sufficient scheduled
compromis- project to move time to implement
ing Project implementation appropriate change
Objectives. date remains if required.
available up to
Project End Date
(August 2017).

90
 Safety Risk Assessment (SRA) examples

Evaluate Acceptance and

Evaluate Enhanced
Existing Existing Risk Enhanced Enhanced Risk Further action reqd Implementation
Task Frequency People/ Existing Enhanced Safety Severity
Severity Category Index Probability Category Index by whom & when of enhanced Documentation of
Ref Identified Task (select from Hazard Consequence Equipment at Probability Existing controls Risk Control & (select from
(select from (auto-calculates (select from (auto-calculates (see options list Safety Risk Change Requirements
drop-down list) risk (select from Mitigations drop-down
drop-down from sev & prob) drop-down list) from sev & prob) below) Control &
drop-down list) list)
list) Mitigations

Hazard 1. Project Hazard 1- Project Manager Hazard 1-

participants con- Consequence possesses proj- Consequence 1-
tain insufficient 1. Complexity ect management Mitigation 1.
individual and of Project skill sets, as Whilst moderate
collective skill requires Project does Training to high level
sets to progress Management Standards project manage-
project. Skill sets that are Department ment skill sets
PM retains discretion
unavailable. leadership, with required, within
to alter team partic-
overall responsi- respective proj-
Project Team Medium frequen- 2D 2D ipants. Additional Only applicable if partici-
10a Negligeable Minor (D) Improbable (2) bility for project ect sub-teams Minor (D) Improbable (2) None required.
Participants cy (M) Acceptable Acceptable internal resource can pants change.
management. (aircraft type
be added, if consid-
and standards
ered appropriate.
department
representation),
sufficient skill
sets exist,
under guidance,
to progress
appropriately.

Hazard 1- Project partici- Hazard 1-

Consequence 2. pants identified Consequence 2-
Insufficent avail- and defined as Mitigation
ability of project required on a 1. Advance
participants, on a monthly basis. scheduling and
month-by-month identification of
scheduling, project partici-
PM will be advised
restricts project pants forwarded
of any restrictions
progress and to crew sched-
Reasonably 4D 3D to individual partici-
10b As above As above As above continuity. Negligable Minor (D) uling ‘planners’, Minor (D) Remote (3) None required. None
Probable (4) Review Review pants availability on
providing visibili-
a month-by-month
ty of commercial
basis.
programme im-
pact. Protection
of the project
team members
confirmed by
‘planners’ in
advance.

Hazard 1. High Hazard 1- Internal Quality Hazard 1-

level of ‘Quality Consequence Control require- Consequence 2-
Control’ detail 1. Project output ments already Mitigation
not achievable. requires further understood and 1. Internal
review and planned to. Regulation &
amendment Compliance
prior to continuously PM review and
Quality Medium frequen- 2D 2D
11 implementation, Negligable Minor (D) Improbable (2) monitored within Minor (D) Improbable (2) necessary None required. None
Requirements cy (M) Acceptable Acceptable
compromising Emirates Training implementation.
delivery Department.
Achieved by ded-
icated Regulatory
& Compliance
Officer (RACO)
position.

Hazard 1. Hazard 1- Regulatory con- Hazard 1-

Precedence set- Consequence tact and dialogue Consequence 2-
ting methodology 1. Project does already in place Mitigation
(EBT) change not achieve reg- and frequent. 1. Regulator
unacceptable ulatory approval, fully appraised
to Regulatory preventing EBT of project, its’ PM Review and
Authority. implementation. objectives and appropriate action
Regulatory Organisation 2D Extremely 1D
12 Low frequency (L) Minor (D) Improbable (2) timeline prior to Minor (D) in event of Regulator None required. None
Involvement (EK)/Trainees Acceptable Improbable (1) Acceptable
project launch. challenge to project
No objections output.
were raised prior
to project launch,
or its’ principle
aims and
objectives.

 91
 Evidence-Based Training Implementation Guide

Evaluate Acceptance and

Evaluate Enhanced
Existing Existing Risk Enhanced Enhanced Risk Further action reqd Implementation
Task Frequency People/ Existing Enhanced Safety Severity
Severity Category Index Probability Category Index by whom & when of enhanced Documentation of
Ref Identified Task (select from Hazard Consequence Equipment at Probability Existing controls Risk Control & (select from
(select from (auto-calculates (select from (auto-calculates (see options list Safety Risk Change Requirements
drop-down list) risk (select from Mitigations drop-down
drop-down from sev & prob) drop-down list) from sev & prob) below) Control &
drop-down list) list)
list) Mitigations

Hazard 1. EBT Hazard 1- Knowledge, Skills Hazard 1-

adoption identi- Consequence and Attitude Consequence 1-
fies a previously 1. Identified currently Mitigation
unrecognised/ Training Need re- continuously 1. Training
unknown quires additional monitored. EBT Department can
Additional ‘Knowledge, training and/or designed to train amend/revise or PM Review, in
Training Need/ Skill or Attitude’ methodology Organisation 2C and develop adopt required Extremely 1C conjunction with
13a Low frequency (L) Major (C) Improbable (2) Major (C) None required. None
Concept (KSA). change. (EK)/Trainees Review Knowledge and training methods Improbable (1) Acceptable Training Depatment
Identified Skills. and concepts. and Fleet.
Specific training
objectives can be
altered to reflect
any new training
needs.

Hazard 1- ’Attitude’ to Hazard 1-

Consequence 2. training and Consequence 2-
‘Attitude’ skillset pilot role covered Mitigation
requires further within HF Training 1. Training
and additional during conver- Department can
training/as- sion and subse- identify "attitude
sessment that quent recurrent to training" by an
PM Review, in
remains outside training. individual trainee.
Organisation 2C Extremely 1C conjunction with
13b As above As above As above current scope Major (C) Improbable (2) This can be ad- Major (C) None required. None
(EK)/Trainees Review Improbable (1) Acceptable Training Depatment
of EK Training dressed directly
and Fleet.
Department. within Training
Department.
Feedback avail-
able to Fleet who
could amend/re-
vise recruitment
process.

92
 Safety Risk Assessment (SRA) examples

A.II Operator sample when transitioning from mixed implementation to baseline EBT

 93
Evidence-Based Training Implementation Guide

94
 Safety Risk Assessment (SRA) examples

 95
Evidence-Based Training Implementation Guide

96
Appendix B - Competencies and Observable
Behaviors
This Appendix is intended to support operators implement EBT, Step 3: Development of an adapted com-
petency model and grading system.
The following tables contain an example of an adapted competency model for EBT, without conditions
and competency standards, which can be used for the recurrent training of pilots in an FSTD. The models
are based on the ICAO competency framework for aeroplane pilots and IEs contained in ICAO Doc 9868
(PANS-TRG).
Note 1: Demonstration of the competencies can be assessed using the OBs, which should meet the re-
quired level of performance, described in the performance criteria established by the AOC/ATO for its
specific operation. OBs may include but are not limited to the OBs listed in the table below.
Note 2: OBs are performed to a criterion, e.g., accurately or correctly, generally not stated.
Note 3: The competencies and OBs in the table are not listed according to any pre-defined priority. OBs
may include but are not limited to the OBs listed in the tables below.

Application of Knowledge (KNO)

Underpinning the pilot competencies is the “application of knowledge” which collectively refers to the
ability of the pilot to:
• Recall and proactively update relevant knowledge; and
• Apply acquired knowledge to the operational environment, including TEM
Throughout all phases of EBT, an operator or ATO must progressively develop, assess and debrief the
pilots on their KNO, recording at least strengths, weaknesses and any remedial action.
Note: IATA has adapted the ICAO competency framework for aeroplane pilots by adding the competency
KNO. This adaptation was considered necessary by both operators and training organizations to ensure
that the pilots demonstrate their knowledge and understanding of relevant information, operating
instructions, aircraft systems and the operating environment. This adaptation by IATA of the ICAO
aeroplane pilot competency framework has been endorsed by all regulators applying CBTA principles and
procedures.

 97
 Evidence-Based Training Implementation Guide

B.I Pilot Competencies

Competency Description Observable behavior (OB)
OB 0.1 Demonstrates practical and applicable knowledge of limita-
tions and systems and their interaction
Demonstrates
OB 0.2 Demonstrates required knowledge of published operating
knowledge and
instructions
understanding
OB 0.3 Demonstrates knowledge of the physical environment, the
Application of of relevant infor-
air traffic environment including routings, weather, airports and the
knowledge mation, operat-
operational infrastructure
(KNO) ing instructions,
OB 0.4 Demonstrates appropriate knowledge of applicable legisla-
aircraft systems
tion
and the operating
OB 0.5 Knows where to source required information
environment.
OB 0.6 Demonstrates a positive interest in acquiring knowledge
OB 0.7 Is able to apply knowledge effectively
OB 1.1 Identifies where to find procedures and regulations
Identifies and OB 1.2 Applies relevant operating instructions, procedures and
applies appropri- techniques in a timely manner
Application of proce- ate procedures in OB 1.3 Follows SOPs unless a higher degree of safety dictates an
dures and compliance accordance with appropriate deviation
with regulations published oper- OB 1.4 Operates aeroplane systems and associated equipment
(PRO) ating instructions correctly
and applicable OB 1.5 Monitors aircraft systems status
regulations. OB 1.6 Complies with applicable regulations
OB 1.7 Applies relevant procedural knowledge
OB 2.1 Determines that the recipient is ready and able to receive
information
OB 2.2 Selects appropriately what, when, how and with whom to
communicate
OB 2.3 Conveys messages clearly, accurately and concisely
Communicates OB 2.4 Confirms that the recipient demonstrates understanding of
through appro- important information
priate means in OB 2.5 Listens actively and demonstrates understanding when
Communication the operational receiving information
(COM) environment, OB 2.6 Asks relevant and effective questions
in both normal OB 2.7 Uses appropriate escalation in communication to resolve
and non-normal identified deviations
situations. OB 2.8 Uses and interprets non-verbal communication in a manner
appropriate to the organizational and social culture
OB 2.9 Adheres to standard radiotelephone phraseology and pro-
cedures
OB 2.10 Accurately reads, interprets, constructs and responds to
datalink messages in English
OB 3.1 Uses appropriate flight management, guidance systems and
automation, as installed and applicable to the conditions
OB 3.2 Monitors and detects deviations from the intended flight
path and takes appropriate action
Aeroplane Flight Path OB 3.3 Manages the flight path safely to achieve optimum opera-
Controls the flight
Management, tional performance
path through
automation OB 3.4 Maintains the intended flight path during flight using auto-
automation.
(FPA) mation while managing other tasks and distractions
OB 3.5 Selects appropriate level and mode of automation in a time-
ly manner considering phase of flight and workload
OB 3.6 Effectively monitors automation, including engagement and
automatic mode transitions

98
 Competencies and Observable Behaviors

Competency Description Observable behavior (OB)

OB 4.1 Controls the aircraft manually with accuracy and smooth-
ness as appropriate to the situation
OB 4.2 Monitors and detects deviations from the intended flight
path and takes appropriate action
OB 4.3 Manually controls the aeroplane using the relationship be-
tween aeroplane attitude, speed and thrust, and navigation signals
Aeroplane Flight Path
Controls the flight or visual information
Management, manual
path through man- OB 4.4 Manages the flight path safely to achieve optimum opera-
control
ual control. tional performance
(FPM)
OB 4.5 Maintains the intended flight path during manual flight while
managing other tasks and distractions
OB 4.6 Uses appropriate flight management and guidance systems,
as installed and applicable to the conditions (See Glossary)
OB 4.7 Effectively monitors flight guidance systems including en-
gagement and automatic mode transitions
OB 5.1 Encourages team participation and open communication
OB 5.2 Demonstrates initiative and provides direction when re-
quired
OB 5.3 Engages others in planning
Influences others OB 5.4 Considers inputs from others
to contribute to a OB 5.5 Gives and receives feedback constructively
Leadership and
shared purpose. OB 5.6 Addresses and resolves conflicts and disagreements in a
Teamwork
Collaborates to ac- constructive manner
(LTW)
complish the goals OB 5.7 Exercises decisive leadership when required
of the team. OB 5.8 Accepts responsibility for decisions and actions
OB 5.9 Carries out instructions when directed
OB 5.10 Applies effective intervention strategies to resolve identi-
fied deviations
OB 5.11 Manages cultural and language challenges, as applicable
OB 6.1 Identifies, assesses and manages threats and errors in a
timely manner
OB 6.2 Seeks accurate and adequate information from appropriate
sources
OB 6.3 Identifies and verifies what and why things have gone
wrong, if appropriate
Identifies precur-
Problem Solving and OB 6.4 Perseveres in working through problems while prioritizing
sors, mitigates
Decision Making safety
problems; and
(PSD) OB 6.5 Identifies and considers appropriate options
makes decisions.
OB 6.6 Applies appropriate and timely decision-making techniques
OB 6.7 Monitors, reviews and adapts decisions as required
OB 6.8 Adapts when faced with situations where no guidance or
procedure exists
OB 6.9 Demonstrates resilience when encountering an unexpected
event

 99
 Evidence-Based Training Implementation Guide

Competency Description Observable behavior (OB)

OB 7.1 Monitors and assesses the state of the aeroplane and its
systems
OB 7.2 Monitors and assesses the aeroplane’s energy state, and its
anticipated flight path
Perceives, com-
OB 7.3 Monitors and assesses the general environment as it may
Situation Awareness prehends and
affect the operation
and Management of manages infor-
OB 7.4 Validates the accuracy of information and checks for gross
Information mation and antici-
errors
(SAW) pates its effect on
OB 7.5 Maintains awareness of the people involved in or affected by
the operation.
the operation and their capacity to perform as expected
OB 7.6 Develops effective contingency plans based upon potential
risks associated with threats and errors
OB 7.7 Responds to indications of reduced situation awareness
OB 8.1 Exercises self-control in all situations
OB 8.2 Plans, prioritizes and schedules appropriate tasks
effectively
Maintains available OB 8.3 Manages time efficiently when carrying out tasks
workload capacity OB 8.4 Offers and gives assistance
Workload Management by prioritizing and OB 8.5 Delegates tasks
(WLM) distributing tasks OB 8.6 Seeks and accepts assistance, when appropriate
using appropriate OB 8.7 Monitors, reviews and cross-checks actions
resources. conscientiously
OB 8.8 Verifies that tasks are completed to the expected outcome
OB 8.9 Manages and recovers from interruptions, distractions,
variations and failures effectively while performing tasks

B.II Instructor/Evaluator (IE) competencies

IE Competency Description Observable behavior (IOB)
Refer to Appendix
Pilot Competencies Refer to observable behaviors in the Pilot Competencies section
B.I
IOB 2.1 Applies TEM in the context of instruction/evaluation
IOB 2.2 Briefs on safety procedures for situations that are likely to
develop during instruction/evaluation
IOB 2.3 Intervenes appropriately, at the correct time and level (e.g.,
progresses from verbal assistance to taking over control)
Ensures that the IOB 2.4 Resumes instruction/evaluation as practicable after any
instruction, as- intervention
sessment and IOB 2.5 Plans and prepares training media, equipment and
Management of the
evaluation are resources
learning environment
conducted in a IOB 2.6 Briefs on training devices or aircraft limitations that may
suitable and safe influence training, when applicable
environment IOB 2.7 Creates and manages conditions (e.g., airspace, ATC,
weather, time, etc.) to be suitable for the training objectives
IOB 2.8 Adapts to changes in the environment whilst minimizing
training disruptions
IOB 2.9 Manages time, training media and equipment to ensure that
training objectives are met

100
 Competencies and Observable Behaviors

IE Competency Description Observable behavior (IOB)

IOB 3.1 References approved sources (operations, technical, and
training manuals, standards and regulations)
IOB 3.2 States clearly the objectives and clarifies roles for the
training
IOB 3.3 Follows the approved training program
IOB 3.4 Applies instructional methods as appropriate (e.g., explana-
tion, demonstration, facilitation, discover with assistance, discover
without assistance)
Conducts train-
IOB 3.5 Sustains operational relevance and realism
ing to develop
Instruction IOB 3.6 Adapts the amount of instructor inputs to ensure that the
the trainee’s
training objectives are met
competencies
IOB 3.7 Adapts to situations that might disrupt a planned sequence
of events
IOB 3.8 Continuously assesses trainee’s competencies
IOB 3.9 Encourages the trainee to self-assess
IOB 3.10 Allows trainee to self-correct in a timely manner
IOB 3.11 Applies trainee-centered feedback techniques (e.g., facili-
tation, etc.)
IOB 3.12 Provides positive reinforcement
IOB 4.1 Shows respect for the trainees (e.g., for culture, language,
experience)
IOB 4.2 Shows patience and empathy (e.g., by actively listening,
reading non-verbal messages and encouraging dialogue)
Supports the train- IOB 4.3 Manages trainees’ barriers to learning
ees’ learning and IOB 4.4 Encourages engagement and mutual support
development IOB 4.5 Coaches the trainees
Interaction with the
and IOB 4.6 Supports the goal and training policies of the operator/ATO
trainees
Demonstrates ex- and Authority
emplary behavior IOB 4.7 Shows integrity (e.g., honesty and professional principles)
(role model) IOB 4.8 Demonstrates acceptable personal conduct, acceptable
social practices, content expertise, a model for professional and
interpersonal behavior
IOB 4.9 Actively seeks and accepts feedback to improve own
performance
IOB 5.1 Complies with Operator/ATOs and Authority requirements
IOB 5.2 Ensures that the trainee understands the assessment
process
IOB 5.3 Applies the competency standards and conditions
Assesses the IOB 5.4 Assesses trainee’s competencies
competencies of IOB 5.5 Performs grading
the trainee IOB 5.6 Provides recommendations based on the outcome of the
Assessment and and assessment
Evaluation Contributes IOB 5.7 Makes decisions based on the outcome of the summative
to continuous assessment
training system IOB 5.8 Provides clear feedback to the trainee
improvement IOB 5.9 Reports strengths and weaknesses of the training system
(e.g., training environment, curriculum, assessment/evaluation)
including feedback from trainees
IOB 5.10 Suggests improvements for the training system
IOB 5.11 Produces reports using appropriate forms and media

 101
 Evidence-Based Training Implementation Guide

Appendix C - Communication plan template

This Appendix is intended to support operators implement EBT, Step 4: Communication plan (courtesy of Airbus).

OPERATOR: XXXXX

Schedule Duration
Ref Communication Audience Objectives When Frequency (if Format Responsibility
relevant)
0 FLIGHT TRAINING ANALYSIS [FTA] = Pre- EBT implementation analysis
Proactive Accountable
Visionconference
Communication with Manager or
Information relat- Before the start of or face-to-face
0,1 Authority Authority in order Once - Operator
ed to EBT project the FTA meeting and
to faciliate the EBT Training
Presentations
implementation Manager
Operator Introduce the man-
Management agement of the EBT Visionconference
(Safety-CMM-Flight implementation Project At the start of the or face-to-face
0,2 Kick Off Meeting Once 3 hours EBT Manager
OPS-etc) and Operator specifif- FTA meeting and
Other Project ties and organization Presentations
Stakeholders understanding.
EBT Impact on Flight
Face-to-face
Operations and Training During Flight
EBT Awareness for Operator meeting or visio-
0,3 and required resource Training Analysis Once 2 hours EBT Manager
manager Management conference and
(Human and Materials). Phase
Presentations
EBT project management
Operator Training During the Flight Face-to-face
Report status, identify Once every 2
0,4 FTA Status Report Manager Other Training Analysis 1 hour meeting or EBT Manager
issues. weeks
Project Stakeholders Phase visioconference
Meeting & associ-
Operator ated documents
Management (Implementation
(Safety-CMM-Flight Final Report related to At the end of the Plan, Gap
0,5 FTA Final Report OPS-etc) Operator the pre-EBT implementa- Flight Traiing Once 2 hours Assessment, and EBT Manager
Training Manager tion analysis Analysis eventually asso-
Other Project ciated documents
Stakeholders as required by
regulators)

102
 Communication plan template

Schedule Duration
Ref Communication Audience Objectives When Frequency (if Format Responsibility
relevant)
0 FLIGHT TRAINING ANALYSIS [FTA] = Pre- EBT implementation analysis
At the end of the Operator
Formal application for
0,6 Application Letter Authority Flight Traiing Once - Letter Training
Mixed EBT
Analysis Manager
Authority Introduction of the
Operator Implementation Plan
Management and its assocaited risk At the end of the Face-to-face Operator
Formal Application
0,6 (Safety-CMM-Flight assessment, and define Flight Traiing Once 3 hours meeting or Training
with Authority
OPS) where neces- the required meeting and Analysis visioconference Manager
sary Operator EBT interactions betweeen
Manager operaotr and auhority
At the end of the
Initial Pilot Operator’s pilot (in- Awareness related to
0,7 Flight Traiing Once - Bulletin-Leaflet EBT Manager
Communication cluding instructors) EBT
Analysis
At then end of the
Initial Management Operators staff other Awareness related to
0,8 Flight Training Once - Bulletin-Leaflet EBT Manager
Communication than Pilots EBT project
Analysis

 103
 Evidence-Based Training Implementation Guide

Schedule Duration
Ref Communication Audience Objectives When Frequency (if Format Responsibility
relevant)
1 Project Monitoring
Report status, present
deliverables, feedback,
Operator
identify issues and devel- Face-to-face
Management Throughout all the
EBT Project - op strategy to solve, next Once a meeting or
1,1 (Safety-CMM-Flight EBT implementa- 4 hours EBT Manager
Comittee Sterring step. Quarter visioconfer-
OPS) tion phase
Updated implementation ence
Training Manager
Plan and its associated
risk assessment
Report status, present
deliverables, feedback,
identify issues and devel- Face-to-face
Training Manager Throughout all the
EBT Project status op strategy to solve, next Once every 2 meeting or
1,2 Instructors EBT implementa- 1 hour EBT Manager
report step. weeks visioconfer-
SMEs tion phase
Updated implementation ence
Plan and its associated
risk assessment
Report Status, present
Authority EBT delivrables, Authority Face-to-face
Throughout all the
EBT Project status Manager feedback. Once a meeting or Operator
1,3 EBT implementa- 2 hours
report with Authority SMEs where Updated implementation quarter visioconfer- Training Manager
tion phase
necessary Plan and its associated ence
risk assessment
EBT Principles Throughout all the
EBT project Status Once a Bulletin-
1,4 Pilots Communicate current EBT implementa- - EBT Manager
for Pilot semester Leaflet
Status tion phase
Operator Satff EBT principles and
Throughout all the
EBT Project Status impacted by EBT Training Effectiveness Once a Bulletin-
1,5 EBT implementa- - EBT Manager
for Operator Staff project (Safety - Communicate current semester Leaflet
tion phase
CMM - Flight Ops) Status
Formal application for At the end of Mixed
Application Letter Operator
1,6 Authority baseline EBT (from Mixed EBT implementa- Once - Letter
for EBT Training Manager
EBT) tion taks

104
Appendix D - EBT IE initial standardization guidance
This Appendix is intended to support operators implement EBT, Step 5: EBT IE initial standardization.

D.I Initial standardization requirements

The following requirements are based on the IATA Instructor and Evaluator Training Guidance Material
and Best Practices, Chapter 4.5, and are consistent with European EBT regulations for an EBT instructor
course.
Note: According to the European EBT guidance material, the training course may be a minimum of 14
hours (training alone) and the recommended length is between 21 to 24 hours (training plus assessment of
competence).
The EBT IE course should be delivered by a qualified EBT IE and comprise both theoretical and practical
training. At the completion of the training course, the applicant instructor should:
1. Have knowledge of EBT, including the following underlying principles:
– Threat and error management
– Competency-based training
– Learning from positive performance
– Building resilience, and
– Data-driven training
2. Demonstrate knowledge of the structure of an EBT module
3. Demonstrate knowledge of the method of training delivery for each phase of an EBT module
4. Demonstrate knowledge of the principles of adult learning and how they relate to EBT
5. Conduct objective observations based on the adapted competency model, and document evidence of
observed performance
6. Relate specific performance observations of competencies
7. Analyze trainee performance to determine competency-based training needs and recognize strengths
8. Evaluate performance using the competency-based assessment and grading system
9. Apply appropriate teaching styles for the different EBT phases to accommodate trainee learning needs
10. Facilitate trainee’s learning, focusing on specific competency-based training needs
11. Conduct a debrief using facilitation techniques
An IE may be given credit for parts of the above if the IE has previously demonstrated competence in
those topics.

D.II Initial EBT IE course footprints

Generic example
General
The following is intended as an example of the integration of EBT focused training described in D.I above,
within an ab initio instructor training program. The example describes only the relevant objectives and
focus areas. It is assumed that any ab-initio training course will develop the IE competencies described in
Appendix B.II.
The objective is to develop the ability to train aviation-based knowledge and skills and attitudes, including
human factors. By the end of the course instructor trainees will be able to:
• Make adequate preparation for the conduct of training
• Use different training techniques

 105
 Evidence-Based Training Implementation Guide

• Develop effective relationships with trainees

• Clearly define the objectives of a training session
• Understand trainees needs and how they want to be trained
• Transfer information and key messages effectively and efficiently
• Manage a training session appropriately
• Understand the subject of human factors and behavior
• Manage CRM issues on training sessions
• Manage difficult situations and people effectively
• Assess a trainee’s performance against a defined standard
• Recognize the importance of making accurate and timely progress reports
• Know how to continuously develop their own training skills
• Understand the knowledge, skills and attitudes of an effective instructor, and
• Understand learning processes and styles
The course is intended to develop a clear but practical understanding of the learning process and the
critical importance of the role of an IE in providing an effective environment, in addition to root cause
analysis, the use of facilitation techniques, and the assessment of outcomes using video role play and live
debriefing.

Ab-initio IEs training program

Day 1 Day 2
– Learning processes and styles: Theory on how – Understanding behavior: Understanding princi-
people learn, the process they go through and the ples of behavior and how people behave in various
different ways that people prefer to learn. situations.
– Training experiences: Identifying knowledge, skills – Managing behavior: Developing skills for managing
and attitudes required of an IE. difficult situations and people.
– Elements of effective training: Identifying and priori- – Training preparation: Learning what to consider and
tizing what is important for effective training. how to prepare for training.
– Communication: Recognizing the difficulty – Body language: Understanding the importance of
of communications and developing effective body language and techniques for improving your
communications. capability in this subject.
Day 3 Day 4
– Human factors: Understanding what human factors – Facilitation: Understanding the importance of facili-
are and how they can be categorized. tation and when the technique should be used.
– Crew resources management: Introduction to EBT – Questioning: Understanding different types of ques-
competencies and learning how to assess a crew. tions and when to use them.
– Instruction: Developing instructional competencies/ – Developing facilitation skills: Exercises in facilitation.
techniques. – Receiving and giving feedback: Learning how to
receive and give feedback.
Day 5
– Issues specific to flight instruction: Identifying the issues specific to flight instruction and how to manage them.
– Common errors and problems: Being prepared for common errors and problems associated with trainees and
how to manage them.
– Report writing and data capture: Learning how to write adequate training reports.
– Practical training exercises: Utilizing all instructional techniques learnt during the course including briefing,
demonstration and training, and debriefing.

IEs transitioning from traditional training to EBT

The following abbreviated course is intended as an example for instructors already qualified, prior to the
introduction of EBT. It is assumed that the instructor is already qualified and has been assessed as compe-
tent according to ICAO Doc 9868 (PANS-TRG).

106
 EBT IE initial standardization guidance

Day 1 Day 2
– Review of instructional techniques course – Review of training techniques
Measure of understanding and refresher on learning Review of the differences between directed instruc-
from initial course. tion and facilitation.
– Trainee learning styles – Situational training
Identifying the different trainee learning styles and Understanding how to adapt training to different
how to adapt. training situations.
– Instructional skills – Competency assessment and debriefing.
Developing instructional and briefing skills. Practice in competency assessment and debriefing
– EBT Competencies using facilitation of real crew activities.
Full understanding of competencies, how they are
assessed and the grading process.
– Competency assessment
Practice in assessing competencies.
Day 3
– Competency assessment and debriefing.
Continued practice in competency assessment and
debriefing using facilitation.
– Final exercises
Briefing, instruction and developing debriefing skills
including assessment of competencies.

Operator Example 1
Day 1
Content Learning Objectives Method

Unit 00 Introduction 45 min

– Welcome Participants get to know each other and the – Lecture
– Self-introduction seminar schedule. – Conversation
– Organizational matters
– Daily schedule

Unit 01 EBT Instructor/Evaluator 75 min

– EBT Instructor/Evaluator (EBT-IE) Participants know the term EBT-IE, know – Lecture
Training how to become and remain an EBT-IE, and – Conversation
– Extension of the EBT-IE know their responsibilities. They know the
– Group work
IE Competencies against which they are as-
– Responsibilities of the EBT-IE – Presentation by
sessed during the Competency Assessment.
– Instructor Competencies participants
– Exchange

Unit 02 EBT Basics 45 min

– EBT Principles Participants know the difference between – Lecture
– EBT Implementation task- and competency based. They know – Conversation
the four principles of EBT and how EBT was
introduced.

 107
 Evidence-Based Training Implementation Guide

Unit 03 Structure of EBT Modules 90 min

– Training requirements – a puzzle Participants learn about the multidimen- – Lecture
with many dimensions sional nature of regulatory requirements for – Conversation
– Levels of documentation training and understand the importance of
the Master Compliance list and other docu-
– EVAL and MT with relevant
mentation. They know the different phases
competencies
of the EBT module, their objectives, and the
– SBT with relevant competencies appropriate teaching methods for each.

Unit 04 Full EBT 45 min

– Competency Report Participants know the components of the – Lecture
– Low Grade Policy Competency Report, how to proceed with – Conversation
Low Grades, and how pilots become and
– Enrolment in the EBT System
remain enrolled in EBT.

Unit 05 Line check / Line Evaluation 45 min

– Line check Participants know their role (with possi- – Lecture
– Line Evaluation bilities of influence and impact) in Line – Conversation
Evaluation and how it is documented and
– Understanding of roles – Exchange and
about the consequences of Low Grades.
– Documentation discussion

Day 2
Content Learning Objectives Method

Unit 06 Targeted observation 135 min

– Competencies and Observable Participants deal with the competencies and – Lecture
Behaviors (OBs) OBs and learn to observe them in a differen- – Group work with
– Conduct of Assessment (ORCA) tiated manner. presentation and
discussion

Unit 07 Differentiated assessment 60 min

– Assessment Method Participants know the assessment process, – Lecture
– Competency Assessment and grading scale, the meaning of each grade – Discussion
Grading and methodology to arrive at a grade.
– Video-based evalu-
– Grading Scale ation exercise with
discussion

Unit 08 Didactics 60 min

– Didactic principles Participants refresh the didactic basics – Lecture
– Tasks of an instructor of TRI Part 1, learn about different ways of – Conversation
transcribing and know that the chronology
– Learning styles and activation of – Exchange and
of observations has to be translated to the
trainees discussion
competencies structure.
– Possibilities of transcripts

108
 EBT IE initial standardization guidance

Unit 09 Competency Assessment during Conversion Training 75 min

– Phase-specific assessment Participants – Lecture
– Final completion standards – Learn completion standards and know – Conversation
– Interim completion standards where to find them – Video-based evalu-
– Evaluate individual competencies for ation exercise with
crew members based on the videos they discussion
have seen and discuss them
– Consider and discuss the benchmark
regarding the training status

Day 3
Content Learning Objectives Method

Unit 10 Competency Assessment – Overall Grading 75 min

– Recap For the first time, the participants create an – Lecture
– Competency Assessment at the overall assessment for both crew members – Assessment exercise
end of conversion training as a result of 3 video samples. in individual work
– Exchange and
discussion
– Online poll

Unit 11 Competency Grading 105 min

– Grading exercise EVAL Participants learn to create competency – Individual work
– Concordance gradings. They can compare their bench- – Exchange and
marks with the other participants and also discussion
– Excursus ICA
get to know the training management opin-
– Grading Exercise MT – 2 video-based
ion as a benchmark.
exercises
Participants learn about Concordance and
ICAP. – Online poll

Unit 12 Competency Based Debriefing 180 min

– Goals, structure and components Participants know the requirements for a – Lecture
of a debriefing competency-based debriefing. They know – Conversation
– Increase activity level what belongs to a good preparation of the
– Individual work
debriefing and know the way to a good
– Facilitation-techniques (DOs and – Exchange and
debriefing result. They conduct a debriefing,
DON’Ts) discussion
grade and receive structured feedback.
– Creating the competency report They are also able to debrief competen- – Video-based evalu-
– Options for conducting cy-based in a short form. ation exercise with
competency-based debriefings discussion
– Selection of debriefing contents – Debriefing exercise in
groups
– Short debriefing exer-
cise in plenum

 109
 Evidence-Based Training Implementation Guide

Operator Example 2
Initial standardization training
Prior to conducting assessment and training within the Emirates’ EBT program, the EBT instructor shall
undergo EBT Initial Standardization training consisting of the below two components:
• EBT Instructor Training; and
• EBT Assessment of Competence

Course entry requirements

In order to complete the initial EBT Standardization program IEs shall:
• Hold a CAR-FCL instructor or examiner certificate
• Be considered suitable by the relevant Fleet Training Manager following the selection process docu-
mented in the Flight Training Process and Procedures Manual

Overview
EBT INSTRUCTOR COURSE
Day Specific Hours Event Details Resource Trainer Qualification
DAY 1 8 EBT INSTRUCTOR CORE DAY 1 CLASSROOM STC
DAY 2 8 EBT INSTRUCTOR CORE DAY 1 CLASSROOM STC
DAY 3 6 EBT SESSION 1 OBSERVATION FFS STC
DAY 4 6 EBT SESSION 2 OBSERVATION FFS STC
EBT SESSION 1 CONDUCT
DAY 5 6 FFS STC
UNDER SUPERVISION
EBT SESSION 2 CONDUCT
DAY 6 6 FFS STC
UNDER SUPERVISION
EBT INSTRUCTOR –
DAY 7 6 UNDERGOING ASSESSMENT of FFS STC
COMPETENCE

110
Appendix E - Equivalency of Malfunction examples
This Appendix is intended to support operators implement EBT, Step 7: Equivalency of malfunctions.

E.I OEM example

The following table is an excerpt from the Airbus A320 FLIGHT CREW TRAINING STANDARDS Manual, ver-
sion 1.4. It lists all of the Airbus abnormal and emergency procedures of the FCOM used as the reference
for this FCTS version.
Note: Operators implementing EBT shall refer to the original document!
The “PROCEDURES” column indicates the procedure title as it appears on the Engine/Warning Display (E/
WD) and in the QRH.
For each procedure, 5 columns indicate if an EBT characteristic is considered applicable based on the
Airbus analysis:
• A column with the letter “I” identifies the “Immediacy” characteristic
• A column with the letter “L” identifies the “Loss of instrumentation” characteristic
• A column with the letter “D” identifies the “Degradation of aircraft control” characteristic
• A column with the letter “M” identifies the “Management of consequences” characteristic
• A column with the letter “C” identifies the “Complexity” characteristic
A letter “Y” meaning “YES” indicates when a characteristic is considered applicable to a procedure.
A letter “N” meaning “NO” indicates when a characteristic is considered not applicable to a procedure.

 111
Evidence-Based Training Implementation Guide

112
 Equivalency of Malfunction examples

 113
 Evidence-Based Training Implementation Guide

E.II Operator example with/without OEM information

The following tables provide excerpts of the Equivalency of Malfunctions process for one aircraft type with
existing OEM information (A330) and for an aircraft type without existing OEM information (B747). The two
methodologies described in Section 4, Step 7, were used. A malfunction difficulty level was determined
in following ways: OEM-based rating (A330, “AI Equivalency class”) / SME-based rating (B747, “Delphi
Rating”), and the number of malfunction characteristics regarded as significant (“EBT equivalency class”).

114
Appendix F - Equivalency of Approaches example
This Appendix is intended to support operators implement EBT, Step 8: Equivalency of approaches.

 115
Appendix G - EBT program development guidance
This Appendix is intended to support operators implement EBT:
• Step 9: Assessment and training topics distribution, and
• Step 10: Module design
Using the data in this Appendix, operators can develop recurrent training programs based on the EBT con-
cept. It is imperative that the recommendations provided in this guide are well understood by developers
of an EBT program.
Note: According to European EBT regulations, program design during mixed implementation should
demonstrate as a minimum:
• Traditional proficiency check elements are included in the mixed EBT program
• EBT training topics and frequencies are included
• There should be a reasonable contextualization of the example scenarios based on the real operation
performed by the operator and feedback from the SMS
• Data provided by the EBT system is used to design the EBT program (e.g., if deficiencies are found
in one competency across the fleet/pilot rank/airline, the future EBT modules should reinforce this
competency)

Background – Prioritization
Prioritization of the training topics is probably the most important result from the EBT data analysis. It is
a key part in the process for translating data into useful events and scenarios to assess and develop pilot
performance in recurrent training programs. This result is the first rigorous attempt to rank parameters
such as threats, errors and competencies, along with factors affecting accidents and serious incidents,
from multiple data sources systematically to formulate a recurrent training program.
The EBT Data Report and its amendment (2021) show the feasibility of collecting an adequate set of op-
erational and training data; developing the necessary methods to analyze that data, while corroborating
results to produce a criticality ranking of training topics. The prioritization process occurs for each aircraft
generation by ordering critical parameters to highlight differences and commonality. There is sufficient
flexibility in the process to allow enhancement according to mission, culture and type of aircraft. The data
are also used to build scenarios for use in recurrent assessment and training conducted in an FSTD qual-
ified for the purpose according to the Manual of Criteria for the Qualification of Flight Simulation Training
Devices (Doc 9625), Volume I – Aeroplanes.
The process used is transparent and repeatable and results in a unique prioritization, according to aircraft
generation. Three levels of priority A, B and C were used to determine the frequency of pilot exposure to
the defined training topics within a 3-year rolling recurrent training program.
For the creation of the EBT recurrent training program, a cautious approach was taken, and the suggested
frequency of training is higher than the results indicate unless the corroborating data is very strong. If the
data are quite strong in the generation that demands more training, the training category in the adjacent
generation is upgraded.
Note: Operational and training data from multiple sources indicate that pilots operating the more modern
generation aircraft need less time to perform certain maneuvers. However, modern generation aircraft are
more complex, and therefore pilots need more time to achieve a defined level of competency operating the
aircraft in all levels of automation/laws. While the number of assessment and training topics is slightly less
in early aircraft generations, the training time in the FTSD should be largely the same.

116
 EBT program development guidance

G.I Aircraft Generations

The EBT program considers the differences between aircraft generations by tailoring the recurrent train-
ing program to the particular aircraft generation. The following is representative of the generations of
aircraft considered within this guide.

Generation Definition Types

Generation 4 – Jet From 1988 A318/A319/A320/A321 (including neo),
EFIS cockpit – FMS equipped A330, A340-200/300, A340-500/600,
FADEC A350, A380,
Fly-by-wire control systems Bombardier C Series (Airbus A220),
Advanced flight envelope protection B777, B787,
Integrated auto flight control system – nav- COMAC C919,
igation performance, and terrain avoidance Embraer E170/E175/E190/E195
systems
Generation 3 – Jet From 1969 A310/A300-600,
EFIS cockpit – FMS equipped B717, B737-300/400/500, B737-600/700/
FADEC 800 (NG), B737 MAX, B747-400, B747-8,
Integrated auto flight control system – nav- B757, B767, BAE 146,
igation performance, and terrain avoidance MD11, MD80, MD90, Fokker F70, F100,
systems Bombardier CRJ Series,
Basic flight envelope protection – stick COMAC ARJ21,
shaker/pusher Embraer ERJ 135/145
Generation 3 – From 1992 ATR 42-600, ATR 72-600,
Turboprop EFIS cockpit – FMS equipped Bombardier Dash 8-400,
EEC/ECU or higher engine control BAE ATP, Saab 2000
Integrated auto flight control system – nav-
igation performance and terrain avoidance
systems
Basic flight envelope protection – stick
shaker/pusher
Generation 2 – Jet From 1964 A300 (except A300-600), BAC111,
Integrated auto flight system. B727, B737-100/200, B747-100/200/300,
EEC/ECU or higher engine control DC9, DC10,
Analogue/CRT instrument display Fokker F28, L1011
Basic flight envelope protection – stick
shaker/pusher
Generation 2 – From 1964 ATR 42, ATR 72 (all series except-600),
Turboprop Analogue/CRT instrument display BAE J-41, Embraer 120, Fokker
EEC/ECU F27/50, Bombardier Dash 7 and Dash
Basic flight envelope protection – stick shak- 8-100/200/300 Series,
er/pusher Convair 580-600 Series, Saab 340,
Integrated auto flight control system Shorts 330 and 360
Generation 1 – Jet From 1952 DC8, B707
First commercial jets.
Manual engine control
Analogue instrument display
Not integrated auto flight control system
Basic flight envelope protection – stick
shaker/pusher, attitude warning

 117
 Evidence-Based Training Implementation Guide

G.II Using the assessment and training matrices

The following table describes the component elements in the column headings of the matrices as follows:

Component Description
Assessment and Training A topic or grouping derived from threats, errors or findings from data analysis, to be considered for assessment and mitigation by
Topic training. Topics marked with “ISI” are those considered only as part of a defined in-seat instruction or demonstration exercise.
The priority of the topic to be considered in an EBT program, according to evidence is linked to a recommended frequency. There
are three levels of frequency:
A – assessment and training topic to be included with defined scenario elements during every EBT module
Frequency B – assessment and training topic to be included with defined scenario elements during alternate EBT modules (i.e., at least three
times in the three-year cycle and at least once within each three consecutive modules); and
C – assessment and training topic to be included with defined scenario elements at least once in the three-year cycle of the EBT
program
Flight phase for activation The flight phase for the realization of the critical threat or error in the assessment and training scenario as set out in Appendix G.V.
Description A description of the training topic (includes type of topic, being threat, error or focus).
Desired outcome Simple evaluative statements on the desired outcome (includes performance criteria or training outcome).
Example scenario elements A list of example scenarios addressing the training topic. This list contains only key elements of sample scenarios.
Competency map Competencies marked are those considered critical in managing the scenario. They were determined according to the following:
a) those competencies considered most critical to the successful management of the defined threat or error; or
b) those competencies most likely to be linked to the root cause of poor performance, in the case of unsuccessful management of
a defined threat or error.

Note: The competency map can also be used to determine which scenarios or combinations of scenarios may be used in developing particular
competencies.
The following table describes the flight phases for the realization of the critical threat or error in the assessment and training scenarios.

Abbreviation Flight Phase Description

ALL All Any or all phases of flight
GND Flight planning, pre-flight, engine Ground phases up to when the crew increases thrust for the purpose of taking-off.
start and taxi-out
Taxi-in, engine shut-down, post- From the speed that permits the aircraft to be maneuvered by means of taxiing for the purpose of arriving
flight and flight closing at a parking area until the crew completes post-flight and flight closing duties.
TO Take-off This phase begins when the crew increases the thrust for the purpose of taking-off.
It ends after the speed and configuration are established at a defined maneuvering altitude or to continue
the climb for the purpose of cruise.

118
 EBT program development guidance

Abbreviation Flight Phase Description

CLB Climb This phase begins when the crew establishes the aircraft at a defined speed and configuration enabling
the aircraft to increase altitude for the purpose of cruise. It ends with the aircraft established at a prede-
termined constant initial cruise altitude at a defined speed.
CRZ Cruise The cruise phase begins when the crew establishes the aircraft at a defined speed and predetermined
constant initial cruise altitude and proceeds in the direction of a destination. It ends with the beginning of
descent for the purpose of an approach.
DES Descent This phase begins when the crew departs the cruise altitude for the purpose of an approach at a particular
destination. It ends when the crew initiates change in aircraft configuration and/or speed to facilitate a
landing on a particular runway.
APP Approach This phase begins when the crew initiates changes in aircraft configuration and/or speeds, enabling the
aircraft to maneuver for the purpose of landing on a particular runway. It ends when the aircraft is in the
landing configuration and the crew is dedicated to land on a specific runway. It also includes go-around
where the crew aborts the descent to the planned landing runway during the approach phase. Go-around
ends after speed and configuration are established at a defined maneuvering altitude or to continue the
climb for the purpose of cruise.
LDG Landing This phase begins when the aircraft is in the landing configuration and the crew is dedicated to touchdown
on a specific runway. It ends when the speed permits the aircraft to be maneuvered by means of taxiing for
the purpose of arriving at a parking area.

 119
 Evidence-Based Training Implementation Guide

G.III EBT program overarching principles

EBT training program design overarching principles should be applied for the development of any EBT module.
These overarching principles regroup several training topics from the 1st Edition of the EBT Data Report (2014) and are foundational components of
any EBT program as they relate directly to the pilot competencies training and assessment.
In particular, the following training topics are no longer trained in isolation since Amendment 7 to ICAO Doc 9868 (PANS-TRG) clarified the relation
between CRM training, pilot competencies and Threat and Error Management. Amendment 7 to ICAO Doc 9868 (PANS-TRG) also provided guidance
for CBTA (including EBT) course development regarding the integration of TEM and surprise elements with the goal to enhance pilot resilience.
• Competencies non-technical (CRM)
• Compliance
• Workload, distraction, pressure
• Monitoring and cross-checking
• Surprise
• Aircraft system management
The following simplified assessment and training matrix is applicable to the airplane generation described in G.I for the design and delivery of all EBT
modules (equivalent to frequency A) and for all flight phases.

120
 EBT program development guidance

Topic Description Desired outcome Remark

Pilot Competencies ICAO competency framework for aero- The pilot demonstrates the competencies The competencies of the approved adapted
Note: This encompass- plane pilot licenses, ratings and recurrent to the final competency standard defined competency model provide individual and
es training topic com- training (ICAO Doc 9868 PANS-TRG Part II, by the operator’s adapted competency team countermeasures to threats and errors
petency non-technical section 1, Chapter 1) model approved by the authority. and undesired aircraft states. CRM skills are
(CRM) embedded in the approved adapted compe-
tency model.
Compliance Compliance means conforming to rules, The desired outcome from an intentional EBT training program designers should
procedures, limitations, and clearances. or unintentional compliance event would consider scenarios where the pilots’ cogni-
An operator will typically require that flight be that the pilots tive load (resources required by the mental
crews comply with all operational proce- – recognize that a non-compliance has processes of perception, memory, judgement,
dures and instructions unless there is an occurred and reasoning) significantly increases to chal-
urgent and compelling reason not to do so. lenge the pilot’s ability to detect and correct
– make a verbal announcement
the potential noncompliance errors.
– take appropriate action Note: The cognitive load is considered to be
– restore a safe flight path if necessary significantly increased when it is well above
– manage the consequences the cognitive load induced by the application
of the normal standard operating procedures.
EBT Instructors should use noncompliance
errors as a learning opportunity during the fa-
cilitated debrief to identify the root cause and
to provide the means to improve performance.
Workload, distraction, High workload, distraction and pressure The pilot demonstrates the competency EBT training program designers should con-
pressure can arise through various means, from a Workload Management to the final compe- sider scenarios where the pilots’ workload
system malfunction, adverse weather or tency standard defined by the operator’s significantly increases in order to challenge
low performing ATC controller, or a combi- adapted competency model approved by the pilot’s demonstration of the competency
nation of these. the authority. Workload Management.
Note: The workload is considered to be sig-
nificantly increased when it is well above the
workload induced by the application of the
normal standard operating procedures.

 121
 Evidence-Based Training Implementation Guide

Topic Description Desired outcome Remark

Monitoring and cross Monitoring is a cognitive process to com- The pilots In-seat instruction (ISI) may be used as a
checking pare an actual to an expected state. – recognize mismanaged aircraft state. method to train monitoring and cross check-
Monitoring and cross checking are embed- ing.
– take appropriate action if necessary
ded in the ICAO competency framework Scripted role-play scenarios encompassing
for aeroplane pilot licenses, ratings and – restore desired aircraft state the need to monitor flight path excursions
recurrent training (ICAO Doc 9868 PANS- – identify and manage consequences from the instructor pilot (PF), detect errors
TRG Part II, section 1, Chapter 1) Note: Special emphasis is placed on the and make appropriate interventions, either
Monitoring and cross checking require demonstration by the pilot of the following verbally or by taking control as applicable.
knowledge, skills and attitudes to create Observable Behaviors (OB): OB1.3, OB1.4, The scenarios should be realistic and relevant
a mental model and to take appropriate OB1.5, OB2.1, OB2.4, OB2.8, OB3.2, OB3.6, and are for the purpose of demonstration and
action when deviations are recognized. OB4.2, OB4.7, OB6.7, OB7.1, OB7.2, OB7.3, reinforcement of effective flight path monitor-
OB8.7, OB8.8 ing.
Demonstrated role-play should contain real-
istic and not gross errors, leading at times to a
mismanaged aircraft state, which can also be
combined with upset management training
Surprise “Surprise” means the emotionally based Pilots demonstrate their ability to prepare EBT training program designers should ensure
Note: Surprise should recognition of a difference in what was and plan for, absorb, recover from, and that pilots are exposed to unexpected events
be distinguished from expected and what is actual. more successfully adapt to adverse event or an unexpected sequence of events during
“Startle” which is The data analyzed during the development and disruptions. every EBT module.
defined as the initial of this guide and of the EBT concept indi- Special emphasis is placed on the demon- Wherever possible, consideration should
short-term, involun- cated substantial difficulties encountered stration of OB 6.9 Demonstrates resilience be given towards variations in the types of
tary physiological and by crews when faced with a threat or error, when encountering an unexpected event scenarios, times of occurrences and types
cognitive reactions to an which was a surprise, or an unexpected of occurrences, so that pilots do not become
unexpected event that event. overly familiar with repetitions of the same
commence the normal scenarios. Variations should be the focus
human stress response. of EBT program design, and not left to the
discretion of individual instructors, in order to
preserve program integrity and fairness
Aircraft system Aircraft system management describes The pilot demonstrates the competency This is not considered as a stand-alone topic.
management the normal and abnormal system oper- “Application of procedures and compli- It links with the topic “compliance”.
ations, based on defined instructions ance with regulations” to the final compe- Where a system is not managed according to
and standard operating procedures. The tency standard defined by the operator’s normal or defined procedures, this is deter-
underlying foundation for successfully adapted competency model approved by mined as a non-compliance.
managing aircraft systems is knowledge of the authority.
systems and understanding of the interac- The pilot demonstrates his ability to:
tions between systems. – recall and proactively update relevant
Aircraft system management relates knowledge; and
directly to the competency “Application of
– apply acquired knowledge to the oper-
procedures and compliance with regula-
ational environment, including TEM.
tions” and the application of knowledge.

122
 EBT program development guidance

G.IV Generation-specific assessment and training matrix

This section provides the recurrent assessment and training matrix according to the particular airplane generation. A list of such airplanes is in G.I.
The matrix supports the EBT implementation process Step 9: Assessment and training topics distribution over a 3-year period.
Notes:
1. Given the very small number of turbo-jet aeroplanes of the first generation in current use in commercial air transport operations and the lack of
appropriate FSTD for recurrent training, it has not been deemed possible to provide an assessment and training matrix for those aeroplanes.
2. This Appendix is based on ICAO Doc 9995 Manual for Evidence-based Training, Edition 2 (draft Sep 2023). ICAO Doc 9995, Edition 1 and European
EBT regulations currently use a slightly different Assessment and Training Topic table. Operators should ensure that they implement the
assessment and training topic matrices required by the relevant CAA for the development of their EBT program.

Assessment and training topic matrix (MT/MV phase) Frequency according aircraft generation
Assessment and training topic Description (includes type of topic, being threat, error or focus) / remarks 4 Jet 3 Jet 3 Prop 2 Jet 2 Prop
Rejected take-off Engine failure after the application of take-off thrust and before reaching V1. A A A A A
Failure of critical engine between V1 & Failure of a critical engine from V1 and before reaching V2 in lowest CAT I
A A A A A
V2 (until stabilized) visibility conditions.
Failure of critical engine between V1 & Failure of a critical engine from V1 and before reaching V2 in lowest CAT I
V2 (until engine-out procedures visibility conditions. B B B B B
complete)
Emergency Descent Initiation of emergency descent from normal cruise altitude. C C C C C
Go-around Go-around, all engines operative. A A A A A
Engine-out approach & go-around With a critical engine failed, manually flown normal precision approach to DA,
followed by manually flown go-around, the whole maneuver to be flown without A A A A A
visual reference.
Engine-out landing With a critical engine failed, normal landing. A A A A A
Adverse weather Thunderstorm, heavy rain, turbulence, ice build up to include de-icing issues, as
well as high temperature conditions.
A A A A A
The proper use of use of anti-ice and de-icing systems should be included
generally in appropriate scenarios.
Automation management The purpose of this topic is to encourage and develop effective flight path
management through proficient and appropriate use of flight management
system(s), guidance and automation including transitions between modes,
monitoring, mode awareness, vigilance and flexibility needed to change from A A A A A
one mode to another. Included in this topic is the means of mitigating errors
described as: mishandled auto flight systems, inappropriate mode selection,
flight management system(s) and autopilot usage.

 123
 Evidence-Based Training Implementation Guide

Assessment and training topic matrix (MT/MV phase) Frequency according aircraft generation
Assessment and training topic Description (includes type of topic, being threat, error or focus) / remarks 4 Jet 3 Jet 3 Prop 2 Jet 2 Prop
Error management and mismanaged By scenario design, pilots should have the opportunity to practice the right
aircraft state techniques and attitudes related to error management through pilot
performance, and that instructors have the opportunity to assess and train this
topic in a realistic environment.
In addition, the operator may also develop scripted role-play scenarios in the
form of ISI. These scenarios cater for the need to monitor flight path excursions
from the instructor pilot (PF), detect errors and make appropriate interventions,
A A A A A
either verbally or by taking control as applicable. The scenarios should be
realistic and relevant, and are for the purpose of demonstration and
reinforcement of effective flight path monitoring. Demonstration scenarios may
also be used.
Demonstrated role-play should contain realistic and not gross errors, leading at
times to a mismanaged aircraft state, which can also be combined with upset
management training.
Go around management Any threat or error that can result in circumstances that require a decision to go-
around, in addition to the execution of the go-around.
Go-around scenarios should be fully developed to encourage effective
leadership and teamwork, in addition to problem solving and decision making,
plus execution using manual aircraft control or flight management system(s) and
A A A A A
automation as applicable. Design should include the element of surprise and
scenario-based go-arounds should not be predictable and anticipated.
This topic is completely distinct from the go-around maneuver listed in the MT
section that is intended only to practice psychomotor skill and a simple
application of the procedures.
Manual aircraft control Controls the flight path through manual control. A A A A A
Unstable approach Reinforce stabilized approach philosophy and adherence to defined parameters.
Encourage go-arounds when crews are outside these parameters. Develop and A A A A A
sustain competencies related to the management of high energy situations.
Adverse wind Adverse wind/crosswind. This includes tailwind but not ATC misreporting of the
B B C B C
actual wind.
Aircraft system malfunctions, including Any internal failure(s) apparent or not apparent to the crew.
operations under MEL Any item cleared by the MEL but having an impact upon flight operations. E.g.,
thrust reverser locked.
Malfunctions to be considered should have one or more of the following
characteristics:
– Immediacy B B B B B
– Complexity
– Degradation of aircraft control
– Loss of primary instrumentation
– Management of consequences
Approach, visibility close to minimum Any situation where visibility becomes a threat. B B B ? C
Landing Pilots should have opportunities to practice landings in demanding situations at
the defined frequency. Data indicates that landing problems have their roots in a
variety of factors, including appropriate decision making, in addition to manual B B B B B
aircraft control skills if difficult environmental conditions exist. The purpose of
this item is to ensure that pilots are exposed to this during the program.

124
 EBT program development guidance

Assessment and training topic matrix (MT/MV phase) Frequency according aircraft generation
Assessment and training topic Description (includes type of topic, being threat, error or focus) / remarks 4 Jet 3 Jet 3 Prop 2 Jet 2 Prop
Runway or taxiway condition Contamination or surface quality of the runway, taxiway, or tarmac including
B B C C C
foreign objects.
Terrain Alert, warning, or conflict. B B C C B
Upset Prevention and Recovery (UPRT) An airplane upset is defined as an undesired airplane state in flight characterized
by unintentional divergences from parameters normally experienced during line
operations or training. An airplane upset may involve pitch and/or bank angle
divergences as well as inappropriate airspeeds for the conditions.
Note. – Required UPRT elements and respective components are listed in ICAO
Doc 10011, table 2-1. The elements are numbered with letters from A to K. Each B B B B B
element is made up of several numbered components. To comply with the
requirements, include elements in a B frequency, such that all elements are
covered over a period not exceeding 3 years. According to the principles of EBT,
covering one component should satisfy the requirement to cover the whole
element of recognizing and preventing the development of upset conditions.
Windshear recovery With or without warnings including predictive. A wind shear scenario is ideally
B B B B C
combined into an adverse weather scenario containing other elements.
ATC ATC error. Omission, miscommunication, garbled, poor quality transmission, all
of these act as distractions to be managed by the crew. The scenarios should be
C C - - -
combined where possible with others of the same or higher weighting, the
principal reason being to create distractions.
Engine failure Any engine failure or malfunction that causes loss or degradation of thrust that
impacts performance. This is distinct from the engine-out maneuvers described
in the maneuvers training section above, which are intended only for the C C C B B
practice of psychomotor skill and reinforcement of procedures in managing
engine failures.
Fire and smoke management This includes engine, electric, pneumatic, cargo fire, smoke or fumes. C C C B C
Loss of communications Lost or difficult communications. Either through pilot mis-selection or a failure
- - C C C
external to the aircraft. This could be for a few seconds or a total loss.
Managing loading, fuel, performance A calculation error by one or more pilots, or someone involved with the process,
C C C C C
errors or the process itself, e.g., incorrect information on the load sheet.
Navigation External NAV failure.
C C C C C
Loss of GPS satellite, ANP exceeding RNP, loss of external NAV source(s)
Operations and type specific Topics covered by the equivalency of approaches process, or additional topics
C C C C C
mandated by the OEM or the Authority.
Pilot Incapacitation Consequences for the non-incapacitated pilot. C C C C C
Traffic Traffic conflict. ACAS RA or TA, or visual observation of conflict, which requires
C C C C C
evasive maneuvering.

 125
 Evidence-Based Training Implementation Guide

G.V Scenario elements and competency map

This section assists operators perform the EBT implementation process, Step 10: Module design by providing example scenario elements and a
related competency map for the assessment and training topics listed in section G.IV.
EBT module design should be based on the assessment and training topics required for the specific module. The combination of scenario elements
allows operators to design scenarios which involve the required set of competencies.
Note: Not all scenario elements and related competencies may be fully applicable for each aeroplane generation and each aeroplane type within a
specific generation. Examples are:
• Predictive wind shear warning: The required system may not be available for all aeroplanes of an operator, even of the same aeroplane type.
• Wind shear encounter: Depending on available aeroplane systems and OEM/Operator-specific procedures, this maneuver may be flown manually
or with the autopilot.
• ACAS warning requiring crew intervention: Depending on the aircraft type, a Resolution Advisory may be flown manually or with the autopilot, even
for aeroplanes within the same aeroplane type (e.g., automatic mode available for newer, but not for older A320).
• Emergency Descent: Depending on the aircraft type, this maneuver may be flown manually or with the autopilot, even for aeroplanes within the
same generation (e.g., automatic mode only available for A350/A380).
As not all possible variations can be considered in the following table, the operator may:
• Select scenario elements during the module design process which are applicable for the operator-specific aeroplane types, and
• Modify the competency map with regards to FPM vs. FPA, as mandated by the operator-specific procedures

ICAO-defined scenario elements

The following table is an excerpt from ICAO Doc 9995 Manual for Evidence-based Training, Edition 2 (draft Sep 2023).
Note: ICAO Doc 9995, Edition 1, and the European EBT regulations currently use a slightly different Assessment and Training Topic table with
integrated scenario elements and competency maps (as referenced in this section G.V). Operators should ensure that they implement the
assessment and training topic matrices required by the relevant CAA for the development of their EBT program.

126
 EBT program development guidance

Competency Map

activation
phase for
Description Desired outcome

Flight
Assessment and training topic (includes type of topic, being threat, error or (includes performance criteria OR training Example scenario elements

WLM
COM

SAW
focus) outcome)

LTW
PRO

FPM

PSD
FPA
SCENARIO ELEMENTS AND COMPETENCY MAP (Manoeuvres training phase)

Engine failure after the application of take-off

Rejected take-off From initiation of take-off to complete stop (or as applicable to procedure) TO x x
thrust and before reaching V1

The manoeuvre is considered to be complete at a point when aircraft is stabilised at normal engine-out
Failure of a critical engine from V1 and before
Failure of critical engine between V1 & V2 climb speed with the correct pitch and lateral control, in trim condition and, as applicable, autopilot TO x x
reaching V2 in lowest CAT I visibility conditions
engagement

Failure of a critical engine from V1 and before The manoeuvre is considered to be complete at a point when aircraft is stabilised in a clean configuration
Failure of critical engine between V1 & V2 TO x x
reaching V2 in lowest CAT I visibility conditions with engine-out procedures completed
Demonstrate manual aircraft control skills
Initiation of emergency descent from normal with smoothness and accuracy as appropriate The manoeuvre is considered to be completed once the aircraft is stabilised in emergency descent
Emergency descent to the situation CRZ x x x
cruise altitude configuration (and profile)
Detect deviations through instrument
scanning
With a critical engine failed, manually flown This manoeuvre should be flown from intercept to centreline until acceleration after go-around. The
Maintain spare mental capacity during manual
normal precision approach to DA, followed by manoeuvre is considered to be complete at a point when aircraft is stabilised at normal engine-out climb
Engine-out approach & go-around aircraft control APP x x
manually flown go-around, the whole manoeu- speed with the correct pitch and lateral control, in trim condition and, as applicable, autopilot engage-
Maintain the aircraft within the flight envelope
vre to be flown without visual reference ment* (describe generally critical part of manoeuvre)
Apply knowledge of the relationship between
aircraft attitude, speed and thrust
High energy, initiation during the approach at 150 to 300 m (500 to 1000 ft) below the missed approach
APP x x x
level off altitude

Go-around Go-around, all engines operative

Initiation of go-around from DA followed by visual circuit and landing APP x x x

During flare/rejected landing APP x x x

Initiation in a stabilized engine-out configuration from not less than 3 NM final approach, until completion
Engine-out landing With a critical engine failed, normal landing LDG x x
of roll-out

Competency Map

Flight phase for

activation
Description Desired outcome
Assessment and training topic (includes type of topic, being threat, error or (includes performance criteria OR training Example scenario elements
focus) outcome)

WLM
COM

SAW
LTW
PRO

FPM

PSD
FPA
EXAMPLE SCENARIO ELEMENTS AND COMPETENCY MAP (Evaluation and scenario-based training phases)

Predictive wind shear warning before take-off, as applicable GND x x x

Adverse weather scenario, e.g., thunderstorm activity, precipitation, icing TO x x x x

Wind shear encounter during take-off, not predictive TO x x x

Predictive wind shear warning during take-off TO x x x x

Crosswinds with or without strong gusts on take-off TO x x

Turbulence that increases to severe turbulence CRZ x x x x

Wind shear encounter scenario during cruise CRZ x x x x x

Reactive wind shear warning during approach or go-around APP x x x x

Thunderstorm, heavy rain, turbulence, ice Anticipate adverse weather
build up to include de-icing issues, as well as Prepare for suspected adverse weather Predictive wind shear warning during approach or go-around APP x x x x
high temperature conditions. Recognize adverse weather
Adverse Weather
The proper use of use of anti-ice and de-icing Take appropriate action Thunderstorm encounter during approach or on missed approach APP x x x
systems should be included generally in Apply appropriate procedure correctly
appropriate scenarios. Assure aircraft control
Increasing tailwind on final (not reported) APP x x x x

Approach and landing in demanding weather conditions, e.g., turbulence, up and downdrafts, gusts and
APP x x x
crosswinds including shifting wind directions

Non-precision approach in cold temperature conditions, requiring altitude compensation for tempera-
APP x x x
ture, as applicable to type

APP
Crosswinds with or without strong gusts on approach, final and landing (within and beyond limits) x x x
LDG

In approach, unexpected braking action ‘good to medium’ reported by the preceding aircraft APP x x x x

Moderate to severe icing conditions during approach effecting aircraft performance APP x x x x

Reduced visibility even after acquiring the necessary visual reference during approach, due to rain or fog APP x x x

 127
 Evidence-Based Training Implementation Guide

Competency Map

Flight phase for

activation
Description Desired outcome
Assessment and training topic (includes type of topic, being threat, error or (includes performance criteria OR training Example scenario elements
focus) outcome)

WLM
COM

SAW
LTW
PRO

FPM

PSD
FPA
EXAMPLE SCENARIO ELEMENTS AND COMPETENCY MAP (Evaluation and scenario-based training phases)

ACAS warning, recovery and subsequent engagement of automation ALL x x

FMS tactical programming issues, e.g., step climb, runway changes, late clearances, destination re-pro-
ALL x x
gramming, executing diversion

Recoveries from TAWS, management of energy state to restore automated flight ALL x x x

Amendments to ATC cleared levels during altitude capture modes, to force mode awareness and
ALL x x x
intervention

Late ATC clearance to an altitude below acceleration altitude TO x x x

Know how and when to use flight management
system(s), guidance and automation TO
Demonstrate correct methods for engage- Engine-out special terrain procedures x x x
APP
ment and disengagement of auto flight
The purpose of this topic is to encourage and
system(s) Forcing AP disconnect followed by re-engagement, recovery from low or high speed events in cruise CRZ x x x x
develop effective flight path management
Demonstrate appropriate use of flight
through proficient and appropriate use of flight
guidance, auto thrust and other automation
management system(s), guidance and auto- Engine failure during or after initial climb using automation CLB x x
systems
mation including transitions between modes,
Maintain mode awareness of auto flight sys-
monitoring, mode awareness, vigilance and
tem(s), including engagement and automatic Engine failure in cruise to onset of descent using automation CRZ x x
Automation management flexibility needed to change
transitions
from one mode to another. Included in this
Revert to different modes when appropriate Emergency descent CRZ x x
topic is the means of mitigating errors
Detect deviations from the desired aircraft
described as:
state (flight path, speed, attitude, thrust, etc.)
mishandled auto flight systems, inappropriate Managing high energy descent capturing descent path from above (correlation with unstable approach DES
and take appropriate action. x x x
mode selection, flight management system(s) training) APP
Anticipate mishandled auto flight system
and autopilot usage.
Recognize mishandled auto flight system.
Take appropriate action if necessary No ATC clearance received prior to commencement of approach or final descent APP x x x
Restore correct auto flight state
Identify and manage consequences Reactive wind shear and recovery from the consequent high-energy state APP x x x

Automation fails to capture the approach altitude in descent (e.g., last altitude before the FAP). Ideally,
APP x x x x
the failure occurs when the workload is high (e.g., configuration of the aircraft for final approach).

Non precision or infrequently flown approaches using the maximum available level of automation APP x x

Gear malfunction during approach. Competency FPA may or may not be included depending on the im-
APP x (x) x x
pact of such malfunction on the automation (e.g., approach planned with Autoland, including autobrake)

ATC clearances to waypoints beyond programmed descent point for a coded final descent point during
APP x x x
an approach utilising a final descent that is commanded by the flight management system.

Adverse weather scenario leading to a reactive wind shear warning during approach APP x x x x

Adverse weather scenario leading to a predictive wind shear warning during approach or go-around APP x x x x

Any threat or error which can result in Adverse weather scenario, e.g., thunderstorm activity, heavy precipitation or icing forcing decision at
circumstances which require a decision to go- APP x x x x
or close to DA/MDA
around, in addition to the execution of the go-
around. Go-around scenarios should be fully
developed to encourage effective leadership DA with visual reference in heavy precipitation with doubt about runway surface braking capability APP x x x x
and teamwork, in addition to problem solving
and decision making, plus execution using Adverse wind scenario resulting in increasing tailwind below DA (not reported) APP x x x
manual aircraft control or flight management
Go-around management
system(s) and automation as applicable. Adverse wind scenario including strong gusts and/or crosswind out of limits below DA (not reported) APP x x x
Design should include the element of surprise
and scenario-based go-arounds should not be Adverse wind scenario including strong gusts and/or crosswind out of limits below 15 m (50 ft) (not
predictable and anticipated. This topic is com- APP x x x
reported)
pletely distinct from the go-around manoeuvre
listed in the manoeuvres training section that
is intended only to practice psychomotor skill Lost or difficult communications resulting in no approach clearance prior to commencement of approach
APP x x x
and a simple application of the procedures or final descent

Birds: large flocks of birds below DA once visual reference has been established APP x x x

System malfunction, landing gear malfunction during the approach APP x x x x

128
 EBT program development guidance

Competency Map

Flight phase for

activation
Description Desired outcome
Assessment and training topic (includes type of topic, being threat, error or (includes performance criteria OR training Example scenario elements
focus) outcome)

WLM
COM

SAW
LTW
PRO

FPM

PSD
FPA
EXAMPLE SCENARIO ELEMENTS AND COMPETENCY MAP (Evaluation and scenario-based training phases)

Flight with unreliable airspeed, which may be recoverable or not recoverable ALL x x x

Alternate flight control modes according to malfunction characteristics ALL x x x

ACAS RA to descend or ATC calls for immediate descent (preferably during climb which requires a
ALL x x x
significant change in aircraft attitude).

Desired competency outcome: ACAS RA to climb or ATC calls for immediate climb (preferably during descent which requires a signifi-
Demonstrates manual aircraft control skills ALL x x x
cant change in aircraft attitude).
with smoothness and accuracy as appropriate
to the situation
Detects deviations through instrument TAWS warning when deviating from planned descent routing, requiring immediate response DES x x x
Controls the flight path through manual scanning
Manual aircraft control
control Maintains spare mental capacity during Scenario immediately after take-off which requires an immediate and overweight landing TO x x x x x x
manual aircraft control
Maintains the aircraft within the normal flight Adverse wind, crosswinds with or without strong gusts on take-off TO x x
envelope
Applies knowledge of the relationship be-
tween aircraft attitude, speed and thrust Adverse weather, wind shear, wind shear encounter during take-off, with or without reactive warnings TO x x x

Engine failure during initial climb, typically 30-60 m (100-200 ft) TO x x x x

Wind shear encounter scenario during cruise, significant and rapid change in windspeed or down/
CRZ x x x x x
updrafts, without wind shear warning

Adverse weather, wind shear, wind shear encounter with or without warning during approach APP x x x x

Adverse weather, deterioration in visibility or cloud base, or adverse wind, requiring a go-around from
APP x x x x x x x
visual circling approach, during the visual segment

Interception of the glide slope from above (correlation with unstable approach). FPA may be required if
APP x x x x
intercept is possible with automation

Adverse wind, crosswinds with or without strong gusts on approach, final and landing (within and beyond APP
x x x
limits) LDG

Adverse weather, adverse wind, approach and landing in demanding weather conditions, e.g., turbulence, APP
x x x
up and downdrafts, gusts and crosswinds including shifting wind directions LDG

Circling approach at night in minimum in-flight visibility to ensure ground reference, minimum environ- APP
mental lighting and no glide slope guidance lights LDG

Runway incursion during approach, which can be triggered by ATC at various altitudes or by visual APP
x x x
contact during the landing phase LDG

Adverse wind, visibility, type specific, special consideration for long bodied aircraft, landing in minimum
LDG x x x x
visibility for visual reference, with crosswind

System malfunction, auto flight failure at DA during a low visibility approach requiring a go-around flown
LDG x x x x
manually

Approach planned with autoland, failure below 1 000 ft requiring a manual go-around and an immediate APP
x x x x x
landing due to fuel shortage LDG

In-seat instruction:
TO x x x x
Insufficient engine failure recovery, forcing the pilot monitoring to take over the flight controls

In-seat instruction:
APP
Unstable approach on short final or long landing, forcing the pilot monitoring to take over the flight x x x x
LDG
controls

By scenario design, pilots should have the In-seat instruction:

ALL x x x
opportunity to practice the right techniques Deviations from the flight path, in pitch attitude, speed, altitude, bank angle
and attitudes related to error management
Observe the pilot’s behaviour: how the pilot is
through pilot performance, and that instruc- Deviations from the flight path, in pitch attitude, speed, altitude, bank angle ALL x x x
mitigating errors, performing cross-checking,
tors have the opportunity to assess and train
monitoring performance and
this topic in a realistic environment.
dealing with a mismanaged aircraft state, in In-seat instruction:
In addition, the operator may also develop
order to ensure that observed deviations, Simple automation errors (e.g., incorrect mode selection, attempted engagement without the necessary
scripted role-play scenarios in the form of ISI. ALL x x
errors and mistakes are taken as learning conditions, entering wrong altitude or speed, failure to execute the desired mode) culminating in a need
These scenarios cater for the need to monitor
opportunities throughout the programme. for direct intervention from the PM, and where necessary taking control.
flight path excursions from the instructor
Error management and mismanaged Monitor flight path excursions.
pilot (PF), detect errors and make appropriate
aircraft state Detect errors and threats through proper In-seat instruction:
interventions, either verbally or by taking
cross-checking performance. Unstable approach or speed/path/vertical rate not congruent with required state for given flight APP x x x x
control as applicable. The scenarios should be
Make appropriate interventions either verbally condition
realistic and relevant, and are for the purpose
or by taking control if applicable.
of demonstration and reinforcement of ef-
Recognize mismanaged aircraft state.
fective flight path monitoring. Demonstration
Take appropriate action, if necessary
scenarios may also be used.
Restore desired aircraft state In-seat instruction:
Demonstrated role-play should contain real-
Identify and manage consequences Demonstration exercise – recovery from bounced landing, adverse wind, strong gusts during landing LDG x x x
istic and not gross errors, leading at times to a
phase, resulting in a bounce and necessitating recovery action from the PM
mismanaged aircraft state, which can also be
combined with upset management training

 129
 Evidence-Based Training Implementation Guide

Competency Map

Flight phase for

activation
Description Desired outcome
Assessment and training topic (includes type of topic, being threat, error or (includes performance criteria OR training Example scenario elements
focus) outcome)

WLM
COM

SAW
LTW
PRO

FPM

PSD
FPA
EXAMPLE SCENARIO ELEMENTS AND COMPETENCY MAP (Evaluation and scenario-based training phases)

ATC or terrain related environment creating a high energy descent with the need to capture the optimum DES
x x x
profile to complete the approach in a stabilised configuration APP

ATC or terrain related environment creating a high energy descent leading to unstable conditions and DES
Reinforce stabilised approach philosophy and x x x
requiring a go-around APP
adherence to defined parameters. Encourage
Unstable go-arounds when crews are outside these
Approach and landing in demanding weather conditions, e.g., turbulence, up and downdrafts, gusts and
approach parameters. Develop and sustain compe- APP x x x
crosswinds including shifting wind directions
tencies related to the management of high
energy situations
Increasing tailwind on final (not reported) APP x x x x

APP
Crosswinds with or without strong gusts on approach, final and landing (within and beyond limits) x x x
LDG

Take-off with different crosswind/tailwind/gust conditions TO x x

Take-off with unreported tailwind TO x x

Crosswinds with or without strong gusts on take-off TO x x

Wind exceeding limits on final approach (not reported) APP x x x x

Wind exceeding limits on final approach (reported) in manual aircraft control APP x x x x

Recognize adverse wind conditions

Increasing tailwind on final (not reported) APP x x x x
Adverse wind/crosswind. This includes Observe limitations
Adverse wind tailwind but not ATC mis-reporting of the Apply appropriate procedures
actual wind Maintain directional control and safe flight Approach and landing in demanding weather conditions, e.g., turbulence, up and downdrafts, gusts and
APP x x x
path crosswind including shifting wind directions

Adverse wind scenario resulting in increasing tailwind below DA (not reported) APP x x x

Adverse wind scenario including strong gusts and/or crosswind out of limits below DA (not reported) APP x x x

Adverse wind scenario including strong gusts and/or crosswind out of limits below 15 m (50 ft) (not
APP x x x
reported)

APP
Crosswind with or without strong gusts on approach, final and landing (within and beyond limits) x x x
LDG

For full details see the Equivalency of Malfunction methodology and results. ALL Intentionally blank

MEL items with crew operating procedures applicable during take-off TO x x

Response to an additional factor that is affected by MEL item (e.g., system failure, runway state) TO x x x

Malfunction during pre-flight preparation and prior to departure GND x x x

Malfunction after departure CLB x x x

Any internal failure(s) apparent or not apparent
to the crew Malfunctions requiring immediate attention (e.g., bleed fault during engine start, hydraulic failure during
Any item cleared by the MEL but having an ALL x x x
Recognize system malfunction taxi)
impact upon flight operations. E.g., thrust Take appropriate action including correct
reverser locked stop/go decision
Malfunctions to be considered should have Fuel leak (management of consequences) ALL x x x
Apply appropriate procedure correctly
Aircraft system malfunctions, including one or more of the following characteristics: Maintain aircraft control
Take-off high speed below V1 TO x x x
operations under MEL – Immediacy Manage consequences
Apply crew operating procedure where
– Complexity necessary. Take-off high speed above V1 TO x x
– Degradation of aircraft control Respond appropriately to additional system
abnormals associated with MEL dispatch During taxi to the runway, a spurious brake temperature announcement. Correct brake temperature
– Loss of primary instrumentation GND x x x
before the failure.
– Management of consequences
Tire failure during take-off TO x x x

Malfunction during initial climb TO x x

Malfunction on approach APP x x x

Malfunction during go-around APP x x x

Malfunction during landing LDG x x x x x

Recognize actual conditions Approach in poor visibility APP x x x x

Observe aircraft and/or procedural limitations
Approach, visibility close to minimum Any situation where visibility becomes a threat Apply appropriate procedure if applicable Approach in poor visibility with deteriorations necessitating a decision to go-around APP x x x
Maintain directional control and safe flight
path Landing in poor visibility LDG x x x

130
 EBT program development guidance

Competency Map

Flight phase for

activation
Description Desired outcome
Assessment and training topic (includes type of topic, being threat, error or (includes performance criteria OR training Example scenario elements
focus) outcome)

WLM
COM

SAW
LTW
PRO

FPM

PSD
FPA
EXAMPLE SCENARIO ELEMENTS AND COMPETENCY MAP (Evaluation and scenario-based training phases)

Pilots should have opportunities to practice

landings in demanding situations at the
defined frequency. Data indicates that landing
problems have their roots in a variety of fac-
Landing in demanding environmental condi- This topic should be combined with the adverse weather topic, aircraft system malfunctions topic or any
Landing tors, including appropriate decision making, LDG Intentionally blank
tions, with malfunctions as appropriate topic that can provide exposure to a landing in demanding conditions
in addition to manual aircraft control skills if
difficult environmental conditions exist. The
purpose of this item is to ensure that pilots are
exposed to this during the programme

GND
Planned anticipated hazardous conditions with dispatch information provided to facilitate planning and
TO x
execution of appropriate procedures
LDG
Recognize hazardous runway condition
Contamination or surface quality of the Observe limitations GND
Runway or taxiway condition runway, taxiway, or tarmac including foreign Take appropriate action Unanticipated hazardous conditions, e.g., unexpected heavy rain resulting in flooded runway surface TO x x x
objects Apply appropriate procedure correctly LDG
Assure aircraft control
Take-off on runway with reduced cleared width due to snow TO x x x x

Stop / go decision in hazardous conditions TO x x x

ATC clearance giving insufficient terrain clearance ALL x x x

Anticipate terrain threats Demonstration of terrain avoidance warning systems ALL x x x

Prepare for terrain threats
Recognize unsafe terrain clearance TO
Engine failure where performance is marginal leading to TAWS warning x x x
Take appropriate action CLB
Terrain Alert, warning, or conflict
Apply appropriate procedure correctly
Maintain aircraft control ATC provides a wrong QNH DES x x
Restore safe flight path
Manage consequences
“Virtual mountain” meaning the surprise element of an unexpected warning. Care should be exercised in
creating a level of realism, so this can best be achieved by an unusual and unexpected change of route DES x x x
during the descent

Note 1. – Upset prevention exercises may be trained in EVAL and SBT, upset recovery may be trained in
MT and SBT phases
Note 2. – The example scenario elements may be done in ISI, as non-ISI or a combination of both. If done
Intentionally blank
in ISI: The instructor should position the aircraft within but close to the edge of the validated training
envelope before handing control to the trainee to demonstrate the restoration of normal flight. Careful
consideration should be given to flying within the validated training envelope.

Upset recognition: Demonstration of the defined normal flight envelope and any associated changes
in flight instruments, flight director systems, and protection systems. This should take the form of an ALL x x x x
instructor-led exercise to show the crew the points beyond which an upset condition could exist
An aeroplane upset is defined as an undesired
aeroplane state in flight characterized by TO
unintentional divergences from parameters Upset recognition and recovery – Severe wind shear or wake turbulence during take-off or approach x x x x
APP
normally experienced during line operations
or training. An aeroplane upset may involve Early recognition and prevention of upset
pitch and/or bank angle divergences as well as Upset recognition and recovery – as applicable and relevant to aircraft type, demonstration at a suitable
conditions: CLB
inappropriate airspeeds for the conditions. intermediate level, with turbulence as appropriate; practice steep turns and note the relationship x x
DES
Note: – Required UPRT elements and respec- – Recognize upset condition between bank angle, pitch and stalling speed
tive components are listed in ICAO Doc 10011, – Take appropriate action Upset recognition and recovery – at the maximum cruise flight level for current aircraft weight, turbu-
Upset Prevention and Recovery Training table 2-1. The elements are numbered with
(UPRT) letters from A to K. Each element is made up – Assure aircraft control lence to trigger overspeed conditions (if FSTD capability exists, consider use of vertical wind component CRZ x x x x
of several numbered components. To comply to add realism)
– Maintain or restore a safe flight path
with the requirements, include elements in a B
frequency, such that all elements are covered – Assess consequential issues Upset recognition and recovery – high-altitude ACAS RA (where the RA is required to be flown in manual
over a period not exceeding 3 years. CRZ x x x x
flight)
According to the principles of EBT, covering – Manage outcomes
one component should satisfy the requirement Upset recognition and recovery – at the maximum cruise flight level for current aircraft weight,
to cover the whole element of recognizing turbulence and/or significant temperature rise to trigger low speed conditions (if FSTD capability exists, CRZ x x x
and preventing the development of upset consider use of vertical wind component to add realism)
conditions.
Upset recognition and recovery – demonstration at a normal cruising altitude, set conditions and
disable aircraft systems as necessary to enable trainee to complete stall recovery according to OEM CRZ x x x
instructions

Upset recognition and recovery – demonstration at an intermediate altitude during early stages of the
approach, set conditions and disable aircraft systems as necessary to enable trainee to complete stall APP x x x
recovery according to OEM instructions

Recovery from a wake turbulence position with high-bank angle APP x x x x

 131
 Evidence-Based Training Implementation Guide

Competency Map

Flight phase for

activation
Description Desired outcome
Assessment and training topic (includes type of topic, being threat, error or (includes performance criteria OR training Example scenario elements
focus) outcome)

WLM
COM

SAW
LTW
PRO

FPM

PSD
FPA
EXAMPLE SCENARIO ELEMENTS AND COMPETENCY MAP (Evaluation and scenario-based training phases)

Predictive wind shear warning during take-off TO x x

Anticipate potential for wind shear
Avoid known wind shear or prepare for
suspected wind shear Wind shear encounter during take-off TO x x x
Recognize wind shear encounter
With or without warnings including predictive. Wind shear encounter after rotation TO x x
Take appropriate action
A wind shear scenario is ideally combined
Wind shear recovery Apply appropriate procedure correctly
into an adverse weather scenario containing
Assure aircraft control Predictive wind shear after rotation TO x x
other elements.
Recognize out of wind shear condition
Maintain or restore a safe flight path Predictive wind shear during approach APP x x x
Assess consequential issues and manage
outcomes
Wind shear encounter during approach APP x x x

ATC role-play: the instructor provides scripted instructions, as a distraction to the crew ALL x x x

Controller error, provided by the instructor according to a defined scripted scenario ALL x x x x

Frequency congestion, with multiple aircraft using the same frequency ALL x
ATC error. Omission, miscommunication,
garbled, poor quality transmission. All of these Destination temporarily closed APP x x x x
Respond to communications appropriately
act as distractions to be managed by the crew.
Recognize, clarify and resolve any ambiguities.
ATC The scenarios should be combined where
Refuse or question unsafe instructions. Rescue and firefighting services (RFFS) level reduction at destination CRZ x x x
possible with others of the same or higher
Use standard phraseology whenever possible
weighting, the principal reason being to create
distractions. Runway change before the interception of the localiser or similar navigation aid in azimuth APP x x x x

GND
Stray dogs at the opposite threshold runway x x x
TO

Poor quality transmissions ALL x

Take-off low speed TO x x x x

Take-off high speed below V1 TO x x x x

Any engine failure or malfunction, which Take-off above V1 TO x x x x

causes loss or degradation of thrust that
Recognize engine failure Initial climb TO x x x
impacts performance. This is distinct from
Take appropriate action
the engine-out manoeuvres described in the
Engine failure Apply appropriate procedure correctly
manoeuvres training section above, which are Engine malfunction APP x x x
Maintain aircraft control
intended only for the practice of psychomotor
Manage consequences
skill and reinforcement of procedures in Engine failure in during cruise (FPA or FPM depending on use of autopilot) CRZ x (x) (x) x
managing engine failures
Multiple engine failure during CRZ (volcanic ash, recoverable). FPM may or may not be included depend-
CRZ (x) x x x x
ing on the impact on automation.

On landing LDG x

Fire in cargo or cabin/cockpit at gate GND x x x x

Fire during taxi GND x x x x

Fire with no cockpit indication GND x x x x

Take-off low speed TO x x x x

Take-off high speed below V1 TO x x x x

Take-off high speed above V1 TO x x x

Recognize fire, smoke or fumes
Take appropriate action Initial climb TO x x x
This includes engine, electric, pneumatic,
Fire and smoke management Apply appropriate procedure correctly
cargo fire, smoke or fumes
Maintain aircraft control Cargo or avionics compartment fire CRZ x x x
Manage consequences
Engine fire in during approach (extinguishable) APP x x

Engine fire in during approach (non-extinguishable) APP x x x

CLB
Lithium battery fire in the cockpit or cabin compartment CRZ x x x x x
DES

Flight deck or cabin fire APP x x x

Any of the example scenario elements above ending in an evacuation GND x x x x

132
 EBT program development guidance

Competency Map

Flight phase for

activation
Description Desired outcome
Assessment and training topic (includes type of topic, being threat, error or (includes performance criteria OR training Example scenario elements
focus) outcome)

WLM
COM

SAW
LTW
PRO

FPM

PSD
FPA
EXAMPLE SCENARIO ELEMENTS AND COMPETENCY MAP (Evaluation and scenario-based training phases)

Recognize loss of communications Loss of communications during ground manoeuvring GND x x

Lost or difficult communications. Either
Take appropriate action
through pilot mis-selection or a failure external
Loss of communications Execute appropriate procedure as applicable Loss of communications after take-off TO x x
to the aircraft. This could be for a few seconds
Use alternative ways of communications
or a total loss
Manage consequences Loss of communications during approach phase, including go-around APP x x x x

This can be a demonstrated error, in that the crew may be instructed to deliberately insert incorrect
data, for example to take-off from an intersection with full length performance information. The crew will
ALL x x x
be asked to intervene when acceleration is sensed to be lower than normal, and this may be part of the
operator procedures, especially when operating mixed fleets with considerable variations in MTOM

Wind report with take-off clearance not consistent with prior performance calculation. ATC, cabin crew
TO x x x x
Anticipate the potential for errors in load/fuel/ or other people are pushing crew to take-off quickly.
performance data
A calculation error by one or more pilots, or Environmental change during taxi (e.g., heavy rain) not consistent with prior take-off performance
Recognize inconsistencies GND x x
Managing loading, fuel, performance someone involved with the process, or the calculation
Manage/avoid distractions
errors process itself, e.g., incorrect information on
Make changes to paperwork/aircraft system(s)
the load sheet
to eliminate error Fuel ground staff on industrial action. Only limited amount of fuel available, which is below the calculated
Identify and manage consequences GND x x x x
fuel for the flight.

Crew advise during taxi out that there is a change of the load sheet figures. The crew may have limited
GND x x
time due to a calculated take-off time (CTOT) – ATC Slot.

Braking action reported ‘medium’. The information is transmitted just before take-off. The flight is
GND x x x
subject to a CTOT – ATC slot.

External failure or a combination of external failures degrading aircraft navigation performance GND x x x x

TO
CLB
External failure or a combination of external failures degrading aircraft navigation performance x x x x
APP
LDG
Recognize a NAV degradation.
External NAV failure. Take appropriate action Standard initial departure change during taxi. The flight may be subject to a CTOT – ATC slot. GND x x x
Navigation Loss of GPS satellite, ANP exceeding RNP, loss Execute appropriate procedure as applicable
of external NAV source(s) Use alternative NAV guidance Loss of runway lighting below decision height APP x x x
Manage consequences
No fly zone: when the crew changes control frequency, the new ATCO informs the crew that they are
flying over an unannounced ‘no fly zone’ that is not included in the NOTAMs. (To trigger such an event, the
context may be as follows: an unexpected military conflict in the territory the aircraft is flying over or the
CRZ x x x
crew is forced to re-route in flight and the new route flies over a city that has an important event such the
Olympic games, a G20/G7 submit, or the route is flying near a space rocket launch close to the time of the
launch, like the Guiana Space Centre, Cape Canaveral, etc.).

Topics covered by the equivalency of

Operations or
approaches process, or additional topics Intentionally blank Intentionally blank Intentionally blank
type specific
mandated by the OEM or the Authority

Recognize incapacitation During take-off TO x x x x

Take appropriate action including correct
stop/go decision
Pilot incapacitation Consequences for the non-incapacitated pilot
Apply appropriate procedure correctly
Maintain aircraft control During approach APP x x x
Manage consequences

CLB
ACAS warning requiring crew intervention CRZ x x x x
DES
Anticipate potential loss of separation
Recognize loss of separation
Traffic conflict. ACAS RA or TA, or visual CLB
Take appropriate action Dilemma: Visual acquisition of conflicting traffic followed by an ACAS warning (resolution advisory)
Traffic observation of conflict, which requires evasive CRZ x x x
Apply appropriate procedure correctly triggered by the same or other traffic. Even if the traffic is in sight, the pilot should follow the RA.
manoeuvring DES
Maintain aircraft control
Manage consequences
While in descent, ACAS traffic advisory of an aircraft below. The crew should not initiate an avoidance CLB
manoeuvre based on a TA (except decreasing the rate of descent unless otherwise instructed by ATC, CRZ x x x
etc.). This example scenario can be done during climb with conflicting traffic above. DES

 133
 Evidence-Based Training Implementation Guide

Additional scenario elements

This table contains additional scenario elements developed by IATA member airlines which are not included in the current Assessment and Training
Topic tables provided by ICAO and the European EBT regulations (as of status September 2023).

Training topic Example scenario elements Flight phase PRO COM FPA FPM LTW PSD SAW WLM Source
Adverse weather Weather deterioration (winter), need to re-calculate take-off performance,
GND x x x x x x JL
holdover time
RWY change due to wind / rwy condition, requiring new take-off data, FMS inputs,
GND x x x x x JL
etc.
Alternate airport closure DES, APP x x x JL
Diversion with changing alternate weather scenario after go-around APP x x x x x LH
Visibility change (below minimum) before FAF. APP x x x x SC
Increasing tailwind on approach without ATC notification APP x x x x EK
the autopilot disengaged due to turbulence, during the turn while intercepting
APP x x CZ
final course at (LOC*) mode
gusty wind, make the approach unstable APP x x CZ
icing condition, left and right engine vibration high, then one engine
CRZ x x x CZ
stall(unrecoverable)
Automation Low altitude level off after liftoff CLB x x x JL
management
Autopilot mode failure (i.e., ALT CAP or ALT*) ALL x x x x EK
Incorrect QNH supplied by ATC on RNP approach APP x x x x x EK
Non-precision approach in hot/cold temperature conditions APP x x x x JL
Go-around Runway incursion (vehicle, aircraft) APP x x x JL
management
Baulked landing after touchdown APP x x x x x x
ATC instruction during simultaneous parallel approach. APP x x x JL
Go-around call above go-around altitude (discontinued approach) APP x x x x OZ
Go-around with TOGA switch failure (Type specific) APP x x x x EK
Go-around call at or right below go-around altitude (go-around or discontinued
APP x x x x OZ
approach)
No landing clearance due to frequency congestion or loss of communication APP x x x LH
Loss of adequate visibility below DA LDG x x x LH
System malfunction during LV APCH below DA/Alert Hight, RWY in sight LDG x x x x LH
cloud ceiling change (below DA/minimum descent altitude [MDA]) after FAF APP x x x SC
visibility change (below minimum) after FAF APP x x x x SC

134
 EBT program development guidance

Training topic Example scenario elements Flight phase PRO COM FPA FPM LTW PSD SAW WLM Source
Manual aircraft Single engine approach, landing or go-around APP x x x JL
control
System malfunction, causing approach and landing with degraded aircraft
ALL x x LH
controls
Breakout from precision runway monitor (PRM) approach APP x x x LH
Baulked landing LDG
Control wheel malfunction during taxi/landing. GND, LDG x x x SC
Error In-seat instruction:
CLB, APP JL
management, Flap/slat mis-operation (by PM instructor)
mismanaged
ISI: Scripted errors made by instructor during take-off and departure that EBT Data
aircraft state TO, CLB x x
requires effective monitoring and intervention by trainee in the PM role. Report
ISI: Subtle incapacitation: e.g., instructor is PM and doesn’t respond to flap
EBT Data
retraction call during climb; instructor is PF and doesn’t initiate deceleration to TO, CLB, APP, LDG x x x x
Report
final approach speed on approach.
Upset prevention: Instructor led exercise to practice high altitude manual flight, EBT Data
CRZ x
including in a degraded flight control law. Report
ISI: Scripted errors or omissions made by the instructor during malfunctions EBT Data
CRZ x x x x x
during the cruise, for example engine failure Report
ISI: Demo of a runway switch followed by at least one further switch with each
crew member as PM. EBT Data
APP x x x
Introduce some scripted errors during the second approach that requires Report
effective monitoring and intervention.
ISI: Demo of a mishandled glideslope intercept from above; further demo to EBT Data
APP x x
highlight appropriate prevention and recovery technique. Report
ISI: Altimeter setting by mistake. TO, CLB, APP, LDG x x x SC
Unstable Increasing tailwind on final (reported) APP x x LH
approach
In-seat instruction (ISI): High speed approach, landing configuration too low, or
APP x x x LH
other parameters out of limits
Visual Approach from an unusual (high) position APP x x x x LH
An unexpected lasting tailwind on the long final APP x x CZ
later approach clearance, leading to high energy in final profile APP x x x x CZ
Visual Approach at night without PAPI intermittently APP x x x CZ

 135
 Evidence-Based Training Implementation Guide

Training topic Example scenario elements Flight phase PRO COM FPA FPM LTW PSD SAW WLM Source
Adverse wind Unable to follow ALT constraint due to wind change CRZ x x x JL
Wind change, approaching VMO/MMO CRZ x x x JL
Wind change, approaching APP, LDG
Approach and landing with Flight Management Computer (FMC) generated winds EBT Data
APP, LDG x x x x
exceeding tailwind limit with tower reported winds within tailwind limit. Report
EBT Data
Increasing turbulence with increasing wind gusts. APP, LDG x x
Report
Wind conditions requiring Circling Approach DES, APP x x x x LH
headwind increase just passing 200 AGL rapidly in typhoon condition in heavy
TO x x x x
weight
gusty wind, wind change from tailwind 20 knots to headwind 20kts APP x x x CZ
The head wind increases rapidly during flaps retraction CLB x x x CZ
suddenly extreme wind change from tailwind to crosswind CRZ x x CZ
head wind increases while setting landing flap APP x x x CZ
Wind change, approaching flaps limit/stall speed. APP, LDG x x SC
Aircraft system Malfunctions requiring immediate attention (e.g., abnormal engine start, brake/
ALL x x x x JL
malfunctions, wheel fault during taxi or near V1, unable to retract gear)
including
TO - incorrect airspeed indication below V1 TO x x x x x JL
operations under
MEL TO - inadequate thrust (thrust lever stucking, TOGA mode failure) TO x x x x x JL
TO - airspeed unreliable after liftoff TO x x x x x x x JL
Unclear system malfunction in cabin, with (IP induced) different opinions about
CLB, CRZ x x x LH
consequences for flight
Malfunction - Flap Slat (drive, asymmetry, skew) CBL, APP x x x x JL
Slow decompression (bleed, pack) CRZ x x x x JL
Partial gear up landing required APP x x x x LH
Fuel leak suspected ALL x x x x x JL
System malfunctions which are not annunciated or covered by an abnormal
ALL x x x x LH
procedure
Malfunction before approach, affecting approach capability CRZ, DES x x x LH
Unreliable Airspeed indications until landing ALL x x x x x x LH
GA - Thrust Lever in idle position due to A/T failure APP x x x x JL
Approach, Visual path information lost below 200ft (JAL issue) LDG x x x JL
visibility close to
Low visibility snow covered runway loss of runway lights at landing LDG x x EK
minimum
Low visibility crosswind Landing APP x x JL
Landing No flap/slat landing LDG x x x JL
Engine failure just prior to the flare (Bird strike) LDG x EK
Overweight landing following immediate air turn back x x x x x x x JL

136
 EBT program development guidance

Training topic Example scenario elements Flight phase PRO COM FPA FPM LTW PSD SAW WLM Source
Runway or taxiway Ground vehicle during take-off roll
TO x x JL
condition
Terrain Pressurization problem or bomb warning creating the need to descent below
CRZ x x x x x LH
minimum altitude (e.g., L888)
EBT Data
Incorrect baro setting on 3D RNAV approach leading to EGPWS warning in IMC. APP x x x x x x
Report
‘Soft’ go around into ‘glass mountain’ requiring change to TOGA for escape EBT Data
APP x x x x
manoeuvre. Report
EBT Data
Curved approach/RNPAR radius to fix ‘glass mountain’ causing EGPWS. APP x x x x x
Report
EBT Data
Spurious EGPWS in VMC. APP x x x x x x x
Report
ATC Wrong instruction by ATC (e. g taxi instruction for opposite side of runway or
GND x x OZ
already passed taxiway)
Similar callsign error (e. g take-off clearance to other aircraft call sign, when A/C
ALL x x x OZ
hold short of runway)
Sudden level-off instruction (after take-off, high altitude) CLB x x x x JL
Loss of communication during departure/arrival into mountainous terrain CLB, DES x x x x x LH
Complicated RT during configuration change (e.g., flap operation) CLB, APP x x x x JL
DEScent re-clearance (QNH-STD-QNH) CLB, APP x x LH
Late approach clearance requiring G/S intercept from above DES, APP x x x x x AF
ATC instructs that the approach was cancelled at high altitude due to insufficient
APP x x x SC
spacing
Loss of comms at an FIR boundary CRZ x x x EK
ATC no reply when crew request deviation due to weather when is in critical CLB
x x x SC
situation, or relay crew to other ATC with different frequency CRZ
Engine failure Engine failure in cruise (high altitude, driftdown) CRZ x x x x x JL
Engine failure in cruise over critically high terrain (Himalayas) CRZ x x x x EK
Engine failure during descent or approach, affecting landing capability DES, APP x x x x LH
Engine failure during Go-around. APP x x x SC
Fire and smoke During boarding: Auxiliary power unit (APU) trouble, smoke into the cabin GND x x x JL
management
Avionics smoke in polar/NATS/HLA region CRZ x x x x EK

 137
 Evidence-Based Training Implementation Guide

Training topic Example scenario elements Flight phase PRO COM FPA FPM LTW PSD SAW WLM Source
Managing loading, Fuel loading error (difference to load sheet or wrong fuel distribution in tanks)
GND x x x LH
fuel, The flight is subject to a calculated take-off time (CTOT) – ATC slot.
performance
ATC advise crew that there is a change in environmental conditions or runway EBT Data
errors GND x x
state during taxi out that requires take-off performance to be re-calculated Report
ATIS report of degraded braking action or contaminated conditions (un-forecast) EBT Data
GND x x x
at destination onto a performance limited runway Report
Fuel supply shortage such that fuel quantity available to uplift is slightly below EBT Data
GND x x x
flight plan fuel Report
EBT Data
Informed of load sheet error during taxi out GND x x x x
Report
EBT Data
Fuelling figures passed which leads to discrepancy that requires resolving GND x x x x
Report
Performance calculation error, for example incorrect runway or incorrect runway EBT Data
GND x x x
starting point Report
EBT Data
Stabilizer/trim out of take-off limitation GND x x x x
Report
Informed by company excessive aft C of G due incorrect loading CRZ x x x x EK
Navigation Internal (aircraft) failure which affect navigation performance ALL x x x JL
Weather radar failure ALL x x x x EK
GPS jamming resulting in loss of GPS signal and navigation accuracy downgrade
CLB, CRZ, EBT Data
below minimum required for flight in current airspace or to fly SID/STAR/ x x x x x
DES, APP Report
approach
Aircraft navigation equipment failure below minimum required for flight in current CLB, CRZ, EBT Data
x x x x x x
airspace or to fly SID/STAR/approach/oceanic airspace DES, APP Report
SCATANA rules activated that require aircraft to comply with ATC instructions to EBT Data
CRZ x x x x x
change course, altitude or land. Report
5G interference effects on approach APP x x x x x EK
Glideslope signal fluctuation; APP x x x x x JL
CLB, CRZ,
Flight Management Computer (FMC) fail during flight. x x x x x SC
DES, APP
CLB, CRZ,
ND fail during flight. x x x x x SC
DES, APP
CLB, CRZ,
MAP fail during flight. x x x x x SC
DES, APP
CLB, CRZ,
IRS fail during flight. SC
DES, APP
Operations or
RJTT approaches (LDA, VOR A circle 16L) in various conditions APP x x x x x x JL
type specific

138
 EBT program development guidance

Training topic Example scenario elements Flight phase PRO COM FPA FPM LTW PSD SAW WLM Source
Pilot Incapacitation during rejected take-off requiring role change for evacuation EBT Data
TO x x x x x x
incapacitation management. Report
EBT Data
Incapacitation of Pilot Flying at rotation. TO x x x x x
Report
EBT Data
Incapacitation of PF during LVO approach. APP x x x x
Report
Incapacitation of PF during Go-around. APP x x x x x SC
EBT Data
ISI: Somatogravic illusion on go around, PF pitches nose down. APP x x x x
Report
EBT Data
ISI: Subtle incapacitation sensed by errors in automation. TO, APP x x x x
Report
ISI: Subtle incapacitation (no call or reply standard callouts) ALL x x x JL
Pilot loses consciousness, leans on the control wheel ALL x x x JL
Traffic ACAS warning with approaching terrain DES, APP x x x JL
Engine out traffic conflict with ACAS set in TA only APP x x x EK
ACAS caution with single engine operation ALL x x x x x JL
Dilemma: Visual acquisition of conflicting traffic followed by an ACAS warning
ALL x x x AF
(resolution advisory) triggered by the same or other traffic.
While in climb/descent, ACAS caution (traffic advisory) of an aircraft above/below CLB, DES x x x AF
ACAS warning during parallel departure/approach. CLB, APP x x x SC
Upset prevention Excess bank due to wake turbulence following heavy jet APP x x x JL
and recovery
ENG failure during holding at high altitude CRZ x x x x x JL
ACAS warning at maximum operating altitude CRZ x x x x JL
Aircraft entering volcanic ash, several system malfunctions CRZ x x x x x x LH
In-seat instruction (ISI): Dual Sidestick Input TO, LDG x x x x x LH
Multiple engine failure on short final APP x x x JL
Windshear Windshear conditions reported by ATC during taxi out or during intermediate
EBT Data
recovery approach that require the crew to avoid or prepare for a possible windshear GND, APP x x x x
Report
encounter.
EBT Data
Predictive windshear alert during take-off roll, initial climb or approach. TO, CLB, APP x x x x x
Report
EBT Data
Reactive windshear alert during take-off roll, initial climb or approach. TO, CLB, APP x x x x
Report

 139
 Evidence-Based Training Implementation Guide

G.VI Integration of CRM requirements

The following table provides an operator’s example for demonstration of compliance with CRM requirements, based on European CRM and EBT regu-
lations. The example was slightly modified to adapt to changes in training topics suggested by the EBT Data report amendment 2021. Further explan-
atory notes regarding the CRM compliance checklist are provided in the EASA Checklists for mixed and baseline EBT.
Note: According to the European CRM requirements, CRM training should be conducted in:
– Non-operational environment (NON OP), e.g., classroom and computer-based), and
– Operational environment (OP), e.g., flight simulation training device (FSTD) and aircraft

Environment
CRM training element Means of compliance
OP NON OP
Automation and philosophy Covered by the “automation management” training topic mandated at frequency “A” by the assess-
on the use of automation ment and training topic. Additionally, one module in every 3-year program is dedicated to automa-
tion (together with other competencies or alone).
In the non-operational environment, the pilot will additionally review by means of Booklet or
x x
Computer-Based Training (CBT): the philosophy on the use of automation (e.g., the concept of auto-
mation, charts/statistics of the different generation of automation, case studies, technical knowl-
edge, etc.)
Note: This training element must be trained in-depth.
Case studies The requirement will be covered:
– in the non-operational environment, within the combined CRM training for flight crew and cabin
crew where an accident/incident is reviewed and
– in the simulator’s briefing and when appropriate in the actual simulator
x x Throughout the 3-year program, the operator has different case studies (accident or incident) where
all the competencies are covered. The pilot learns the importance of a competency or group of com-
petencies in each case study. For example, by studying an incident/accident where a competency
was missing and or by studying cases where the competency was exercised to the right level, and
this allows the crew to “save the day”.
Some of the case studies may be provided by the SMS flight safety department.
Human factors in aviation EBT training topic A “competencies non-technical (CRM)”
In the non-operational environment it is included in the combined CRM training for flight crew and
x x
cabin crew or CBT, at least once in the 3-year cycle. Interaction human-machine with the focus on
the HUMAN element in the model (e.g., SHELL).
General instructions on CRM EBT training topic A “competencies non-technical (CRM)”
principles and objectives Facilitation technique is used as a means of debriefing in every simulator session.
x x
In the non-operational environment it is included in the combined CRM training for flight crew and
cabin crew or CBT at least once in the 3-year cycle.

140
 EBT program development guidance

Environment
CRM training element Means of compliance
OP NON OP
Human performance and EBT training topic A “competencies non-technical (CRM)”
limitations In the non-operational environment it is included in the combined CRM training for flight crew and
x x
cabin crew or CBT at least once in the 3-year cycle. Interaction human-machine with the focus on
the HUMAN element in the model (e.g., SHELL).
Personality awareness, – Personality awareness: it is trained in the combined CRM training for flight crew and cabin crew.
human error and reliabil- This element is trained in a non-operational environment. This could be additionally comple-
ity, attitudes and behav- mented by a CBT.
iors, self-assessment and – Human error and reliability, attitudes and behaviors, self-assessment and self-critique: EBT fully
self-critique x x
endorses the facilitated de-briefing because it provides opportunities to the pilots for self-as-
sessment and self-critique exercise. The grading system also provides observable behaviors
that will foster this CRM training element. This element is, therefore trained in the operational
environment.
Fatigue and vigilance x This requirement is trained by CBT and in the FRMS training.
Stress and stress Training topic (workload distraction pressure) frequency B. The Competency WLM is assessed every
management 6 months. Additionally, at least, one module in every 3-year program is dedicated to WLM (together
with other competencies or alone). The briefing and de-briefing script of this module is planned in
such a way that IES have opportunities to further develop this CRM element. Note: stress and stress
management are not always related to Workload; that is why the non-operational environment train-
x x ing is additionally proposed.
In the non-operational environment it is included in:
– The combined CRM training for flight crew and cabin crew.
– Case study/s that includes “stress and stress management”. This element maybe delivered by
advanced CBT (e.g., Virtual reality).
The CBT explaining the theory of this element, once every 3 years.
Cultural differences x combined CRM training for flight crew and cabin crew
Operator’s safety culture and Operator’s safety culture and company culture training are not specifically addressed in the EBT
company culture, standard program. To meet the objectives of this requirement operators should provide training in the
operating procedures (SOPs), non-operational environment e.g., in the form of a group discussion.
organisational factors, Operator’s safety culture, company culture, organisational factors and factors linked to the type of
factors linked to the type of operations are included in the non-operational environment in the combined CRM training for flight
x x
operations crew and cabin crew.
Standard operating procedures (SOPs) and factors linked to the type of operations are trained in the
EBT simulator program, specifically, during the SBT. Furthermore, the competency PRO is graded in
every simulator session. Additionally, the 3-year EBT program has one specific module dedicated to
the development of this competency.

 141
 Evidence-Based Training Implementation Guide

Environment
CRM training element Means of compliance
OP NON OP
Threat and error Annual CBT or classroom training once every 3 years.
management Additionally, the briefing and de-briefing in the simulator session may be guided through the TEM
x x
model (e.g., during the briefing when the IE is explaining a malfunction, he/she should identify with
the pilots the hazards and possible errors; this helps a facilitated briefing).
Assertiveness, situation The competency SAW is assessed in every simulator session. Additionally, one module in every
awareness, information ac- 3-year program is dedicated to SAW (together with other competencies or alone).
quisition and processing Assertiveness is partially covered through the competency COM through OB 2.1 and OB 2.4. This
competency is assessed in every simulator session. Additionally, one module in every 3-year pro-
gram is dedicated to COM (together with other competencies or alone).
x “Information acquisition and processing” is additionally trained as follows:
– briefly discussed during the briefing/de-briefing by means of a presentation delivered by the
instructor and
– in CBT (or advance CBT such VR, chat boot, interactive scenario trainer)
Note: When possible, the training actions above may be done together with “human performance
and limitation”.
Specific type-related Mandated at frequency “C” by the training and assessment matrix.
differences Additionally, the operator that provided this example includes when appropriate:
x x – a reminder during the simulator briefing sessions by means of a presentation delivered by the IE,
or
– in the non-operational environment, the pilot will additionally review by means of Booklet or CBT
Monitoring and intervention x Mandated at frequency “A” by the “table of assessment and training topics”.
Shared situation awareness, The competency SAW is assessed in every simulator. Additionally, one module in every 3-year pro-
shared information acquisi- gram is dedicated to SAW (together with other competencies or alone).
x
tion and processing SAW is additionally discussed during the briefing/de-briefing by means of a presentation delivered
by the IE.
Workload management Workload Management is one of the pilot competencies that must be trained throughout the EBT
program.
x
Workload is also a specific training topic in the “table of assessment and training topics” (overarch-
ing principles).
Effective communication COM is one of the competencies that must be trained throughout the EBT program.
and coordination inside In the non-operational environment in the combined CRM training involving both flight and cabin
x x
and outside the flight crew crew.
compartment Note: this element may be credited during the emergency and safety equipment training.

142
 EBT program development guidance

Environment
CRM training element Means of compliance
OP NON OP
Leadership, cooperation, LTW is one of the competencies that must be trained throughout the EBT program. Additionally,
synergy, delegation, deci- leadership, cooperation, synergy, delegation is complemented in the combined CRM training for
sion-making, actions flight crew and cabin crew.
Cooperation, synergy and delegation are additionally covered in the non-operational environment
by means of CBT once in a 3-year period.
Decision-making:
x x
– When the EBT program is designed in such a way that crews are exposed to a dilemma. This
means crews are presented with situations where more than one possible less than ideal solu-
tions exist, with some unfavorable conditions attached, then this element is covered in the
simulator session. This training action also relates to the competency PSD.
– Alternatively, the operator may develop an advanced CBT where the pilot faces a scenario/s
where the pilot must take a decision/s.
Resilience development This element is fulfilled in the simulator session by:
– Training topic “surprise” (see EBT overarching principles).
– Scenario-based training phase, as per definition, the SBT should develop resilience.
x x
Additionally, it will be trained in the non-operational environment during:
– CBT that explains the theory of this element. Once every 3 years.
– Advanced EBT (e.g., VR) for the element of surprise and startle effect that relates to resilience.
Surprise and startle effect This element is fulfilled in the simulator session by the training topic ‘surprise and startle effect’ (see
the table of assessment and training topics).
Additional it will be trained in the non-operational environment during:
x x – the combined CRM training for flight crew and cabin crew once in the 3 years, or the legacy CBT
explaining the theory of this element. Once every 3 years, or
– advanced CBT (e.g., Virtual Reality) where the pilot faces a scenario/s where the pilot experience
this element.
Effective communication and COM is one of the competencies that must be trained throughout the EBT program. Additionally,
coordination with other oper- there are 2 training topics (ATC and Loss of Communication) frequency C and 1 training topic
ational personnel and ground (Competency Non-Technical CRM) frequency A.
x x
services In the non-operational environment in the combined CRM training involving both flight and cabin
crew, or the operator may develop an advanced CBT (e.g., chatbot) where the pilot faces a scenari-
o(s) where the pilot experiences this element.

 143
 Evidence-Based Training Implementation Guide

G.VII EBT module design examples

IE “toolbox”
Scenario elements per topics and per competencies
Scenario event Time KNO PRO COM FPA FPM PSD LTW SAW WLM
Manual Flying
“Base training” pattern with
10 min x
crosswind
ENG OUT ILS AP OFF, FD ON,
10 min x
followed by GA
Aborted landing 5 min x x x
Engine Failure during TO
10 min x x
heavy weight aircraft
Engine Failure during TO
10 min x
light weight aircraft
Automation management
Glideslope intercept from above 5 min x x x x
NPA 2D Approach 10 min x x x x
Circling RNAV visual approach 10 min x x x
GA all engines with AP 5 min x
GA all engines with AP in clean
5 min x
flaps configuration
Level bust management 5 min x
Cancellation of Approach mode 5 min x
Non-complex system failures management
One Hydraulic circuit low level 15 min x
One electric generator failure 15 min x
Engine failure on final 10 min (x) (x) x x x
Complex or multiple system failures management
Major hydraulic failures (CRZ) 35 min x x x
Major electrical failures (CRZ) 35 min x x x
All engines failure (CRZ) 15 min x x x x
All engines failure (initial CLB) 5 min x x x x
Landing with abnormal landing
25 min x x
gear configuration
Priority Management
Uncontained engine failure 10 min x
Smoke event 10 min x x x x x
Pilot incapacitation during
5 min x x
take-off
Memory Items
RTO, engine fire V1-20Kt 5 min x
RTO, followed by request of
10 min x x x
evacuation by chief purser
TCAS event 5 min x (x) (x)
Windshear during take-off 5 min x x x
Windshear during final approach 5 min x (x) (x)
GPWS warning 5 min x x

144
 EBT program development guidance

Scenario elements per topics and per competencies

Scenario event Time KNO PRO COM FPA FPM PSD LTW SAW WLM
Unreliable airspeed 10 min x x x (x)
Loss of braking 5 min x x
Emergency descent 10 min x x x

EVAL/SBT examples

 145
Evidence-Based Training Implementation Guide

146
 EBT program development guidance

 147
Appendix H - Training program design for pilots after
long absence
Section 4, Step 11 “Contingency procedures”, provides a template for required training elements after
long absence within an EBT program. This Appendix gives further information regarding their content and
possible course design, following the TA-SE (Training Areas of Special Emphasis) concept.

Background
The IATA “Guidance for Post-COVID Restart of Operations: CBTA Training Solutions”, Edition 2, provided
training solutions for operators and their pilot population to recover competence and confidence after a
long absence. The proposed training solutions:
• Present incremental training options that are adapted to each pilot’s individual situation in regard to
ICAO’s Annex 6 and Annex 1 standards compliance
• Should be assessed by the safety and compliance management system of the operator to en-
sure that the training options are suitable to the operator’s pilot population and, where necessary,
complemented
• Respect the competency-based training and assessment (CBTA) principles and are applicable to both
traditional training and advanced training methodologies

H.I Ground training refresher

Principles
The ground training content should always be adapted and proportionate to the duration of the absence
of a pilot (refer to the template pilot situation / training solutions on page 72). The proposed solution origi-
nates from the EU regulation with reference to the license expiry period:
• In regard to situations 1 and 2 described in the Section 4, Step 11 “EBT modules missed by the pilot”
template, the ground training refresher is not required or optional because the license has expired by
less than 90 days. Therefore, there is no formal ground training to be recovered as the pilot received
the expected ground training elements.
• In regard to situations 3 and 4, the ground training refresher is mandatory because the license has ex-
pired by 90 days or more. Consequently, there is formal ground training to be recovered as the pilot did
not receive the expected ground training elements.

Components
Ground training includes several components. Some components are related to the delivery and acqui-
sition of specific knowledge elements (e.g., aircraft system, operator policy and procedures, UPRT aca-
demic topic, etc.). Other components integrate on the job training by adding skills and attitude training to
the knowledge elements, such as Safety and Emergency Procedures (SEP) training, and Crew Resources
Management (CRM) training.

Knowledge components
When designing the ground training refresher, the operator should consider reviewing and reinforcing
following topics:
• Aircraft systems, limitations and procedures (normal, abnormal and emergency)
• Operational policy and associated procedures
• Safety events

148
 Training program design for pilots after long absence

• UPRT academics, if necessary

• Special Approval (SPA) related knowledge, if necessary
Note: SPA concerns the following operations: performance-based navigation (PBN), minimum
operational performance specifications (MNPS), reduced vertical separation minima (RVSM) airspace, low
visibility operations (LVO), extended range operations with two-engine aeroplanes (ETOPS), and transport
of dangerous goods (DGR)
The Observable Behaviors marked bold below should be used to define the training objectives of the
knowledge part of the ground training refresher.

OB 0.1 Demonstrates practical and applicable knowledge of limitations and systems and their interaction
OB 0.2 Demonstrates required knowledge of published operating instructions
OB 0.3 Demonstrates knowledge of the physical environment, the air traffic environment including rout-
ings, weather, airports and the operational infrastructure
OB 0.4 Demonstrates appropriate knowledge of applicable legislation
OB 0.5 Knows where to source required information
OB 0.6 Demonstrates a positive interest in acquiring knowledge
OB 0.7 Is able to apply knowledge effectively

Safety and Emergency Procedures (SEP) component

When designing the ground training refresher, the operator should consider reviewing and reinforcing
following topics:
• Actual donning of a lifejacket, where fitted
• Actual donning of protective breathing equipment, where fitted
• Actual handling of fire extinguishers of the type used
• Instruction on the location and use of all emergency and safety equipment carried on the aircraft
• Instruction on the location and use of all types of exits, security procedures

Crew Resource Management (CRM) component

When designing the ground training refresher, the operator should consider the CRM training elements
that should be reviewed and reinforced in accordance with the different pilot situations. The magnitude of
the CRM refresher training may vary significantly depending on the time elapsed since the pilot’s last CRM
training.
In the EBT program context, CRM training is largely embedded in the FSTD training and should be charac-
terized by a performance orientation, with emphasis on standards of performance and their measurement,
and the development of training to the specified performance standards.

H.II FSTD training and assessment

Principles
The FSTD training content should always be adapted and proportionate to the duration of the absence of a
pilot (refer to the template pilot situation/training solutions in Section 4, Step 11, paragraph “EBT modules
missed by the pilot”).
For a pilot being absent for more than 90 days, it is expected that a competency assessment and consoli-
dation is necessary to recover and perform successfully during a regular EBT module.
The session design, especially for the consolidations session(s) should take into account the EBT mod-
ule design principles and specifically the two design solutions for the SBT phase described in Section 4,
Step 10.

 149
 Evidence-Based Training Implementation Guide

Components
The FSTD training is composed by the CBTA refresher session followed, when necessary, by one or more
CBTA consolidation sessions.
• The CBTA refresher session is foundational, as this session has been designed to train and assess
the pilot competencies in the context of pilots with limited training and operational exposure, and who
need to recover the recent experience requirements (3 take-offs and landings during the last 90 days).
Therefore, the CBTA refresher session is optional for situation 2, but recommended when the pilot’s
last flight was performed several weeks ago (e.g., beyond 60 days). In this case the goal is to develop
the pilot’s flight path management competencies and reinforce maximum pilot confidence.
• The CBTA consolidation session(s) is a complement to the CBTA refresher session when the pilot’s
performance needs to be further enhanced. The number of CBTA consolidation sessions is determined
by the operator based on the pilot’s training needs.
Therefore, the CBTA consolidation session(s) is proposed for situation 4 because it is expected that
the CBTA refresher session will not be enough to recover competence and confidence. Moreover, the
CBTA consolidation session(s) is always an option after any CBTA refresher session where remedial
training is needed.
• Pilots in situations 3 and 4, having missed one or more EBT modules, should participate in the regular
EBT module in place at the operator after their absence, to be fully requalified within the EBT program.
If the pilots’ license has to be renewed, the EBT module should be performed by an EBT instructor with
examiner privileges.
For all sessions, it is assumed that:
• the pilots’ documentation is up to date and the operator has provided the pilots with the necessary
policy and operational procedure knowledge refresher
• the pilots have received the training program sufficiently in advance to prepare the session, and
• the training sessions are conducted in an FSTD approved by the CAA to conduct the EBT program

FSTD sessions overview

The following tables provide an overview of the FSTD CBTA Refresher and CBTA Consolidation sessions
mentioned above.

FSTD refresher session

Assumptions – The flight crew composition is two pilots. If there is only one pilot to be trained, the IE
may act as pilot from one of the pilot seats.
– The pilots may have lost partially or totally their recent experience (3 takes-off and
3 landings during the last 90 days).
– The pilots had limited training or operational experience in the last 6 months or more.
– The AOC and the ATO have criteria based on previous global competence and ex-
perience to define the pilot population eligible to this session (e.g., pilot is not under
remedial training).
– The AOC and the ATO design the session with some spare time to allow for the IE to
adapt to the pilot’s needs. For example, the IE further develops pilot competencies
or the pilot requests to train specific events to gain confidence.
Special emphasis Training and assessment of all competencies, focus on PRO, FPM, SAW, WLM
Note: The special competency emphasis for FSTD is the result of an analysis of the pilot
competencies and their associated OBs, combined with the training need, of an average
already qualified pilot population lacking both training and operational experience.
Global training The pilot must be able to conduct normal and abnormal operations in accordance with
objective the performance standard defined by the AOC, in an FSTD approved by the CAA for EBT
programs, under realistic conditions that are representative of the nature and the com-
plexity of their operational and environmental context.

150
 Training program design for pilots after long absence

FSTD refresher session

Detailed training – The pilot must be able to conduct normal SOPs in an operational context in accor-
objectives dance with the performance standard under real time operational conditions.
– The pilot must be able to perform abnormal SOPs including emergency procedures
in operational context, in accordance with the performance standard under opera-
tional conditions. At least one of the abnormal procedures should have an impact on
aircraft performance.
– The pilot must be able to manage the flight path manually and with automation, in
accordance with the performance standard under operational conditions, including
a combination of precision, non-precision and visual approaches.
Conditions For all exercises mentioned above, the IE’s support and intervention are acceptable
at the beginning of the session, but pilots must be fully autonomous at the end of the
session.
Assessment and Missed B/C topics
training topics The CBTA refresher and consolidation sessions should include as many, if not all, B and
C topics of the EBT modules the pilot has missed when being absent.

FSTD consolidation session(s)

Assumptions – The CBTA consolidation session(s) is delivered after the CBTA refresher session.
– The flight crew composition is two pilots.
– The pilots may need to recover some assessment and training topics that are related
to the EBT program or operator-specific Special Approvals (SPA).
– The pilots had limited training or operational experience for 12 months or more.
Special emphasis Training and assessment of all competencies, competency focus (tailored training) is
based on assessment results of the CBTA refresher session.
Global training The pilots must be able to conduct normal and abnormal operations, including SPA, in
objective accordance with the performance standard defined by the AOC, in an FSTD approved
by the CAA for EBT programs, under realistic conditions that are representative of the
nature and the complexity of their operational and environmental context.
Detailed training – The pilot’s performance demonstrated during the CBTA refresher session must be
objectives enhanced (or consolidated) to demonstrate consistently the performance standards
defined by the AOC across all pilot competencies.
– The pilot must demonstrate a level of performance that exceeds the minimum
license proficiency checks standards.
– The pilot must be able to perform normal SOPs, in accordance with the performance
standard, under real time operational conditions using a representative operator
network (including network that necessitate an area, route and aerodrome (ARA)
qualification).
– The pilot must be able to perform abnormal SOPs including emergency procedures
that place a significant demand on a proficient crew, in operational context and in
accordance with the performance standard.
– The pilot must be able to manage the flight path manually and with automation, in
accordance with the performance standard under operational conditions, using a
combination of precision, non-precision and visual approaches (including approach-
es that require special approval).

 151
 Evidence-Based Training Implementation Guide

FSTD consolidation session(s)

Conditions For all exercises mentioned above, the IE’s support and intervention are acceptable at
the beginning of the session, but pilots must be fully autonomous at the session end.
Assessment and train- Missed B/C topics
ing topics The CBTA refresher and consolidation sessions should include as many, if not all, B and
C topics of the EBT modules the pilot has missed when being absent.
Absence-related emphasis
It is expected that the CBTA consolidation session(s) may be used to recover some SPA
and UPRT training elements that must be performed in an FSTD. When designing the
FSTD training, the operator should consider the UPRT elements and the following topics,
which must be trained or checked to maintain the associated SPA:
– Performance-based navigation (PBN)
– Minimum operational performance specifications (MNPS)
– Reduced vertical separation minima (RVSM) airspace
– Low visibility operations (LVO)
– Extended range operations with two-engine aero planes (ETOPS)
– Transport of dangerous goods (DGR)

H.III Line Flying Under Supervision (LIFUS)

Principles
LIFUS or Supervised Line Flying (SLF) training is an integral part of the EBT program as it provides a vehicle
for pilots to consolidate competence and confidence in line operations. The aim of LIFUS is to allow the
trainee to regain practical experience and exposure to the operator’s route structure. By the end of the
LIFUS, the trainee will have demonstrated the ability to undertake effectively normal rostered duties.
The number of sectors should be tailored to the needs of the pilot as an individual, taking into account
their experience with the operation, the complexity of the network and the period of time since the last op-
erational duties. The training designer should ensure sufficient exposure to the network to meet the global
training objective of the LIFUS element. LIFUS is also a method to recover Area, Route and Aerodrome
qualification and associated recency.

LIFUS overview
LIFUS overview
Special emphasis Training and assessment of all competencies, focus on competencies SAW and WLM
Note: The special competency emphasis for LIFUS is the result of an analysis of the pilot
competencies and their associated OBs, combined with the training need of an average
already qualified pilot population lacking both training and operational experience for
more than 12 months and having received ground and FSTD refresher training.
Global training The pilots must be able to conduct normal operations in accordance with the per-
objective formance standard defined by the AOC or the ATO, in an aircraft, during normal line
operations.

152
 Training program design for pilots after long absence

LIFUS overview
Detailed training The pilot must be able to:
objectives – Perform normal SOPs in all phases of flight, in accordance with the performance
standard
– Manage the flight path manually and with automation, in accordance with the perfor-
mance standard under operational conditions
– Demonstrate a competent standard of operational and regional knowledge, includ-
ing published operating instructions, and where to source required information
– Appropriately manage workload in order to maintain proper situational awareness
during operations
– Achieve safe and efficient line operations by demonstrating an effective
decision-making process
– Perform effectively monitoring duties and use appropriate intervention strategy
(when necessary) to maintain safe aircraft operations throughout all phases of flight
– Demonstrate effective communication by conducting interactive threat-based brief-
ings and using correct R/T procedures and phraseology at all times
– Demonstrate commercial awareness
LIFUS syllabus The LIFUS syllabus should consist of ground training and line training to ensure that
flight crews are qualified to operate in areas, on routes or route segments and into the
airports used by the operator. This is achieved by means of the following:
– For the less complex routes, familiarization by self-briefing with route documenta-
tion, or by means of programed instruction; and
– For more complex routes, in-flight familiarization as a commander, co-pilot or ob-
servers under supervision, or familiarization in a Synthetic Training Device using a
database appropriate to the route concerned.
The achievement of LIFUS competency requires Knowledge, Skills and Attitude ele-
ments. Knowledge is built from previous experience and throughout the ground training
phase. LIFUS provides an opportunity to expand this knowledge base and to apply it to
develop and reinforce practical line skills.

Example topics to be considered for inclusion in a LIFUS syllabus

Cold Weather Operations
• Identify when de-icing and/or anti-icing is required
• Demonstrate knowledge of the de-icing and anti-icing procedure and its application
• Show knowledge of how to determine runway surface conditions
• Demonstrate application of cold temperature altitude corrections

Adverse Runway Conditions

• Determine runway condition
• Apply runway condition to performance calculations (Take-off and Landing)
• Demonstrate knowledge of contaminated runway procedures and limitations (aircraft and company)

Continuous Descent Approach (CDA)

• Apply CDA techniques
• Demonstrate knowledge of stabilization requirements
• Locate airport-specific requirements (e.g., minimum RoD)

 153
 Evidence-Based Training Implementation Guide

DARD
• Demonstrate proficiency in determining 1EO and 2EO ceilings
• Interpret flight plan DARD output and be able to apply an escape strategy in a practical manner
• Interpret ND and VD terrain information and en-route charts to determine safe altitudes
• Understand aircraft oxygen system requirements and demonstrate correct application of the related
pre-flight checks

FANS/CPDLC/ADS
• Determine CPDLC capability of aircraft (FANS 1/A, ATN B1)
• Demonstrate proficiency in the use of CPDLC functions in normal situations
• Locate any FIR-specific Datalink requirements
• Determine the primary and secondary communication methods in Datalink environments and establish
contact with them as appropriate
• Demonstrate a basic understanding of performance-based communication and surveillance (PBCS),
including areas of use and required equipment
• Demonstrate knowledge of CPDLC/ADS use in emergency or abnormal situations

High Frequency (HF) Comms

• Identify areas where HF is used
• Demonstrate proficiency in HF communication and the use of SELCAL

High Elevation Airports (> 5,000’ AMSL)

• Demonstrate knowledge of the effects of operating into high elevation airports, including density alti-
tude consideration
• Demonstrate an ability to apply strategies to mitigate the effect of high elevation on aircraft
performance

IATA In-Flight Broadcast Procedure (IFBP)

• Demonstrate ability to locate where IFBP is applicable and proficiency in its application
• Demonstrate understanding of additional operating procedures/contingencies associated with IFBP
areas

Adverse Weather (Adverse Runway conditions covered as separate topic)

• Demonstrate knowledge and application of departure/arrival hazardous weather guidance
• Demonstrate knowledge of turbulence procedures relating to aircraft (overspeed, severe turbulence)
and cabin management

North Atlantic High-Level Airspace (NAT HLA)

• Demonstrate awareness of required aircraft capability/equipment for a NAT flight
• Demonstrate ability to locate and apply FIR communication and navigation procedures in the various
NAT FIRs, including methods of obtaining an oceanic clearance and HF comms
• Demonstrate proficiency in entry and cross-checking of FMS waypoints, including in the event of a
re-route
• Demonstrate knowledge on the application of Oceanic procedural requirements during the various
phases of flight
• Demonstrate understanding of NAT contingency procedures

154
 Training program design for pilots after long absence

Polar Operations
• Demonstrate an understanding of the impact of space weather on polar operations (at pre-flight plan-
ning stage and in-flight)
• Identify communication options and their potential limitations within the polar region
• Identify navigational specificities to polar flight
• Identify areas where cold fuel may be expected and demonstrate an understanding of the fuel system
in such a case
• Demonstrate an awareness of nearest alternate airports during a polar flight

PBN
• Determine required equipment for a given RNP/RNAV capability
• Interpret the aircraft’s monitoring of navigation performance
• Identify when amendment of the RNP field in the FMS may be required
• Identify actions to be taken in the event of a downgrade in navigation capability

RVSM
• Demonstrate knowledge of required equipment and actions in the event of any failure
• Identify allowable tolerances of altitude indications
• Identify any considerations to operating in RVSM airspace

Thunderstorms and Weather Radar

• Demonstrate knowledge of company policy regarding thunderstorm avoidance
• Demonstrate knowledge of turbulence procedures relating to aircraft (overspeed, severe turbulence)
and cabin management
• Demonstrate effective use of the WXR functions
• Demonstrate proficiency in weather analysis and avoidance based on the weather radar display (ND
and VD)
• Understand the limitations of the weather radar

UPRT
• Demonstrate an understanding of the relevant environmental hazards, such as:
– Clear Air Turbulence (CAT)
– Intertropical Convergence Zone (ITCZ)
– Thunderstorms
– Microbursts
– Wind shear
– Icing
– Mountain waves
– Wake turbulence, and
– Temperature changes at high altitude

 155
 Evidence-Based Training Implementation Guide

• Be familiar with the evaluation and management of the associated risks of the relevant hazards above;
and the available mitigating procedures related to the specific route, route area, or aerodrome
• Identify an aircraft upset
• Demonstrate knowledge of upset prevention and recovery techniques
• Demonstrate knowledge of the intervention model

Metric Altimetry
• Determine appropriate PFD altimeter display above and below transition
• Apply conversions correctly (above and below transition as appropriate)

156
Appendix I - ICAP implementation guidance
This Appendix is intended to support operators implement EBT, Step 15: EBT IE recurrent standardization
and ICAP.

Human Factors Considerations

Human Factors should be considered in instructor assessments. Two components which frequently
contribute to errors in judgment are bias and noise. While bias refers to systematic deviations, noise re-
fers to random scatter or unwanted variability in expert judgments. Nobel laureate Daniel Kahneman has
contributed considerably to this field and found that the extent of disagreement among experts is usually
much greater than expected, posing a serious challenge to organizations. These findings are widespread
across various industries. They are equally applicable to aviation training and the way instructors form
assessments.
In addition to common biases, there are different components of noise which affect the judgments of
skilled experts. Variability needs to be expected in the average level of assessments by different instruc-
tors, with some being generally stricter and others more lenient (“level noise”). Further, variability needs to
be expected in attitudes and responses of instructors to a particular case. An overall strict instructor may
be exceptionally lenient with trainees exhibiting particular traits (“pattern noise”). Finally, unstable, with-
in-subject variability is to be expected (“occasion noise”). Thereby, factors such as mood, sleep, fatigue, or
even the weather, were found to contribute to this intra-person variability. When provided with the same
data on different occasions, experts are therefore often found to contradict their own prior judgment.
Emphasis on these human factors aspects should be given in training. For instructors, this will enhance
understanding and self-awareness about influences on their assessments. For training managers, this
will allow for more realistic expectations towards the magnitude of variability that needs to be expected
among the instructor community. The latter also directly relates to understanding human factors consid-
erations within the instructor concordance assurance program (ICAP).

Implementation
To ensure a comprehensive understanding of inter-rater reliability, data from two sources should be
derived:

Operator EBT data

Data collected from EBT sessions needs to be interpreted with limited scope. It is neither recommended
to use this data for measures of alignment, nor for agreement, as the grades awarded by IEs are based on
different content observed and reflect different performance of crews.
However, real EBT data can be a valuable source of information to detect obvious trends in IE gradings,
such as IEs generally grading very high, very low, very centered in the middle with little spread, or grading
many extremes. This data might serve as an indication of possible poor alignment or grading bias of an IE.
To verify these observations, they should then be correlated with the findings from the standardized mate-
rial assessment. For instance, if an IE was found to grade relatively low in real EBT data, but was always well
calibrated in the standardized assessments, he/she might have coincidentally just been conducting EBT
sessions with trainees whose performance required a lower grade.

Standardized material
IE agreement and alignment may only be inferred from assessments using standardized material. It is
important that all IEs observe and grade the same content and receive identical instructions on the ex-
pected conduct of the assessment. The standardized assessment process is regarded as the foundation

 157
 Evidence-Based Training Implementation Guide

for determining inter-rater reliability levels. It should be conducted once in every 12-month period, or if
needed more frequently.

Implementation process
The figure below illustrates a process on how the ICAP could be implemented in practice.

1. Content Development
An important factor in the ICAP is the observation of the same content for all IEs. The usage of videos is
the preferred method because they best transfer the environment, stress level, non-verbal communica-
tion, etc. Videos with real crews undertaking EBT may be used if available, but this may be inefficient and
time consuming if certain levels of performance for specific competencies need to be targeted. Using
scripted videos instead serves the advantage of addressing different performance levels and compe-
tencies in focus, thereby using the designers’ time efficiently. The length of the video should ensure that
sufficient evidence can be observed to grade the competencies of interest.
An operator may decide to use the same videos for all IEs, or separate ones for each fleet. In the first case,
the video should be designed in a way that type-related procedures are reduced to a minimum and must
be explained in advance if relevant to IEs who do not operate on the fleet featured in the video. Fleet-
specific videos may allow for better procedural assessments but make it more difficult to establish con-
cordance across the entire IE community in all other (particularly the non-technical) competencies. This
approach is also time consuming as many videos need to be created and analyzed individually.
The videos should target different competencies and comply with the guidance on showing three different
levels of performance for each competency within a 72-month period. Thus, areas of weak agreement in
one ICAP cycle may be assessed again after having performed the concordance training with the affected
IEs to determine whether standards have improved within the same competency.

2. IE Assessment
All IEs, airline and subcontracted, should have the same EBT IE initial training, and be required to partici-
pate in the recurrent concordance assessment. During the assessment, IEs should be presented with the
standardized material and required to grade the observed performance. They should also receive a brief

158
 ICAP implementation guidance

introduction about the ICAP objective along with instructions about the expected conduct of the assess-
ment. The operator’s learning management system might serve as a suitable platform to distribute the
assessment to the IE community.
The grading itself should be as close as possible to the IEs’ daily EBT environment. Therefore, the usage of
the same IT grading system is preferred to avoid data entry errors resulting from unfamiliarity with a differ-
ent grading system or survey format. If the operator’s systems permit, the videos should be restricted to a
“one time only” view, as during regular EBT sessions IEs are also limited to observing a scenario only once.
Neither the targeted competencies nor the grade levels may be communicated in advance to the IEs, as
possible areas of weak concordance might not be revealed in this case. IEs should mark competencies as
“not observed” if insufficient evidence for grading was present in the standardized material.

3. Operator Standard of Reference

To measure alignment, first an operator’s standard of reference (also known as “Gold Standard”) needs
to be established to determine the expected grading level of the observed performance recorded in the
standardized material. The standard should be agreed upon by a team of experts which may consist of
training managers and highly experienced EBT IEs. This should be performed in an in-person “standard
meeting”. Ample time of at least several hours up to a full day should be allocated to this meeting. A few
recommendations apply:
• As an academic rule of thumb, the definition process should comprise at least ten experts. If the ex-
perts are EBT IEs as well, the discussion should take place after they have participated in the ICAP
assessment.
• The standard should be defined by every operator on its own, even if material of other operators
(with a standard already available) is used, in order to cope with the operators’ cultural and grading
philosophies.
• The agreement on the standard should be unanimous across all experts involved. The performance
standard should be set for each video, each competency, and each crew (CM1/CM2).
• The justifications for the experts’ decision should form the basis for the subsequent concordance
training. Arguments must be available to explain to IEs why a particular grade had been decided on by
the group. Thus, it is essential to record detailed minutes of the standard meeting.
• It is essential to understand that the standard has no direct influence on the measured agreement of
the EBT IEs. However, the outcome of the standard meeting drives the subsequent concordance train-
ing. Therefore, the application of the ORCA process should facilitate the determination of the reference
standard. It inevitably supports improvements in alignment as well as higher agreement over time.
• Friction is to be anticipated during the meeting when attempting to form an agreement among the
group of experts. An impartial moderator may therefore be considered to chair the meeting. It is ad-
visable to set the tone and establish ground rules upfront on how to address tension and conflict. The
moderator should support the team in maintaining a healthy debate and ensures that concerns and
disagreements are productively addressed. A representative of the human factors / CRM team may be
well suited for the moderator role.

4. Statistical Analysis
Once the assessment phase and the standard meeting are completed, data analysis can commence.
While appropriate methods and metrics must be established to assess concordance, the analytical com-
plexity may vary depending on the size and complexity of the operator. In any case, assessing concor-
dance should be a data-driven process and preferably involve statistical analysis.
Various different methods exist for calculating inter-rater reliability coefficients. Generally, the type of
analysis required primarily depends on the number of raters and the scale level of ratings, among others.
Suitable metrics for measuring IRR are discussed by EASA. Closer analysis showed that not all of these
metrics are suitable to assess IRR across 9 competencies on a 5-point grading scale (including “not

 159
 Evidence-Based Training Implementation Guide

observed”). Therefore, in addition to these metrics, the agreement coefficient by Gwet should be consid-
ered as a feasible method.
Prior to commencing any analysis, it should be ensured that a suitable method is chosen for the respec-
tive data available. Industry experience has shown that usually the expertise to conduct the ICAP-specific
data analysis is not available within an operator. Therefore, it is recommended that the analysis be per-
formed by external experts, unless an operator has applicable in-house expertise in research methods and
statistics.
Concordance assessment should be conducted for all instructors combined. In addition, if applicable, it
should also be investigated for different groups of instructors such as fleet and type-specific instructors,
groups of different experience levels, or subcontracted instructors. Results might then reveal different
levels of concordance between groups of instructors, which allows for more targeted intervention and
training.
It is further recommended to use a benchmarking system when presenting concordance results, rather
than purely numeric values. Various benchmark systems exist, and further reading is recommended. As an
example, a common system is the Landis-Koch benchmark system (1977). It provides a conservative clas-
sification by taking into account, among others, errors of instructors who may not have participated in the
assessment. This benchmark system classifies the different size of agreement with the following levels:

Agreement Interpretation
0.8 – 1 Almost Perfect
0.6 – 0.8 Substantial
0.4 – 0.6 Moderate
0.2 – 0.4 Fair
0 – 0.2 Slight
-1.0 – 0 Poor

5. Insights Consolidation
After the analysis has been completed, a meeting is recommended to derive insights, recommendations,
and a subsequent action plan from the findings. This should be a collaborative effort between training
management and data analysts.
Special attention should be paid to reporting the insights in a way that caters for end-users who may not
have statistical training. Specific terminology should be well explained and transparent to training manag-
ers. Classifications in word pictures and graphical representations may be more suited to aid interpreta-
tion compared to numbers and statistical values alone.
Depending on which benchmark system is used, thresholds are usually recommended on which result
should warrant intervention. If the above-mentioned Landis-Koch benchmark system is used, classifica-
tions of “Almost Perfect” and “Substantial” indicate no severe problems in the agreement between the
IEs. For a class of “Moderate”, corrective actions should be taken in concordance training. Landis-Koch
classes that are below “Moderate” shall be addressed in concordance training, as they indicate that IEs are
either not coherent among themselves or apply the grading scale inconsistently between the pilots to be
rated.
The table below provides an example on how concordance assessment results could be presented, in this
case for CM1 across a 5-point grading scale. Inter-rater reliability was calculated using the Gwet coeffi-
cient and the Landis-Koch benchmarking system.

160
 ICAP implementation guidance

Modified operator sample

CM1 IRR % of grades assigned
Competencies (Gwet, Landis-Koch) 1 2 3 4 5 NO
KNO Substantial 6% 50% 40% 4%
PRO Substantial 5% 55% 36% 4%
COM Almost Perfect 2% 3% 95%
FPA Substantial 1% 1% 6% 24% 8% 60%
FPM Moderate 8% 8% 38% 38% 8%
LTW Moderate 1% 16% 25% 29% 4% 25%
PSD Almost Perfect 3% 90% 7%
SAW Substantial 30% 50% 15% 5%
WLM Almost Perfect 21% 56% 23%

• The results on agreement are presented in the “IRR” column. The word pictures indicate the results
derived from the assessment based on the Landis-Koch benchmark system. Most competencies ob-
tained rather high agreement levels, classified as “Substantial” or “Almost Perfect”. However, FPM and
LTW highlight the need for corrective action in subsequent concordance training.
• The results on alignment are displayed in the grading columns. Yellow fields indicate the agreed stan-
dard of reference for the respective competency. The values indicate the percentage of raters that
have assigned the respective grade. The example presents some notable cases:
– The COM competency not only shows almost perfect inter-rater reliability, but also 95% alignment
with the operator standard of reference. IEs are in large agreement among themselves as well as
calibrated with the expected standard.
– The PSD competency also shows almost perfect inter-rater reliability, but only 3% alignment with
the operator standard of reference. IEs are high in their agreement, but the group appears to be “off
target”.
– FPA and LTW competencies show very large heterogeneity among IEs, as every possible grading
option was marked.
The following reasons could contribute to the case of high agreement but poor alignment:
• IEs assessed the correct competency but wrongly assessed the grading level
• IEs did not assess the correct competency
• The operator standard of reference itself might be incorrect and should be reviewed
All the above need to be addressed during concordance training. Insights from real EBT data can be taken
into account as practicable.

6. IE Training
Training is one of the elements to ensure reliability within the EBT IE community and should address espe-
cially IEs not meeting the required reliability and validity.
Nevertheless, to keep the level of reliability high, it is recommended to perform training with all IEs. The
previously assessed content may be a good working base for discussion, training and aligning the IEs to a
consistent grading.
Using the known principles of the competency assessment process stated in Section 5, and especially
root-cause analysis, enhances the training effect as IEs can understand the way and reasons for a specific
assessment result. Remembering the way to this result provides them with the capability to transfer it to
other assessment situations.

 161
Evidence-Based Training Implementation Guide
ISBN 978-92-9264-877-0
Customer service: www.iata.org/cs
+1 800 716 6326

iata.org/publishing

Printed in Canada