LINE PILOT PERFORMANCE OF MEMORY ITEMS

Howard Au
Boeing Commercial Airplane Group*
P.O. Box 3707 MC 67-TC
Seattle, WA 98124, USA

* This research was paid for through out-of-pocket expenses by the author. It was conducted while the author was a
student at Embry-Riddle Aeronautical University in partial fulfillment for the degree of M.S. Aeronautics.

An evaluation of Boeing 737 line pilot performance of memory items in 5 abnormal checklists was performed in a
single-blind experiment using tabletop exercises at the crew base of a major U.S. airline. A study of 16 pilots shows
that performance of memory items results in errors in identifying the failure, selecting the proper checklist to be
completed, and checklist step errors.

Introduction commission (adding steps or performing steps

incorrectly), omission (missing steps), or order
Some system failures that can occur on commercial (completing steps in the wrong sequence).
airliners require flight crews to perform checklist steps
from memory prior to referring to the checklist. These Baddeley [1] presented a review of studies that included
steps, called memory items or recall steps, are for time performance of deep-sea divers, combat aviators in
critical actions crucial to the safe continuation of the actual combat, soldiers in simulated emergencies, and
flight (e.g., preventing severe aircraft damage or crew skydivers. These studies evaluated the performance of
incapacitation). Typically, line pilots do not study manual dexterity tasks, tracking tasks, and attention to
memory items except in preparation for a proficiency peripheral cues. They showed that danger manifests
check (PC), usually every 6 or 12 months. They arrive itself in human performance through a narrowing of
for their evaluation prepared to be tested on the recall attention or through an increase in time to complete a
of the memory items. Their performance in these manual dexterity task. The narrowing of attention can
evaluations may not reflect their performance on the potentially lead to increased performance only if the
line, months after a PC. task being performed is understood to be important.
However, performance on tasks made to seem
This study examines whether line pilots are familiar peripheral during an emergency can deteriorate [3].
enough with the memory items to perform all of them Similarly, if the task is so complex as to require
reliably, without prior knowledge that they will be attention to numerous cues, the narrowing of attention
evaluated. It was predicted that the performance of the will result in an inability to integrate relevant task
memory items would show errors of commission, information and an inability to conduct a proper
omission, and order due to the pilots’ infrequent assessment of the situation [6].
review of the memory items. This impromptu method
of evaluation more closely resembles an unanticipated It is possible that training can mitigate some of these
inflight emergency. This paper reviews some of the effects. However, even though pilots receive regular
literature on performance under stress and then training in emergency procedures in simulators, that
discusses the results pertaining to errors in does not mean they are unaffected by the stress of an
identification of failures and errors in checklist actual emergency. An emergency in a simulator is not
selection. Although checklist step omission and order perceived as life-threatening. If the pilot fails, the
errors were observed, this paper will focus on the simulator can be reset for another attempt. Unless a
commission errors in the completion of checklist steps. pilot has had repeated experience in dealing with a
truly dangerous emergency, performance in a real
Human Performance Under Stress emergency could be similar to a novice. It has been
shown that subjects are able to inhibit fear and prevent
An inflight emergency requiring timely action imposes it from affecting their performance only if they are
a great deal of stress on the flight crew. Previous studies repeatedly exposed to a dangerous situation [1]. Due to
have shown that recall under high-stress conditions is the reliability of today’s airliners, it is unlikely for the
more prone to errors than recall under low-stress average airline pilot to have this kind of exposure in an
conditions [8]. These errors, as they relate to checklist airplane.
use, may include errors in identifying the abnormal
condition, selecting the correct checklist, and errors of

20
Stress Effects on Problem Analysis time resulted from omission errors by crews
performing memory items. They occasionally omitted
It is possible that performance on infrequent tasks, deploying the speedbrake, causing the airplane to
such as identifying the root cause of multiple failures descend slower. On the other hand, crews that
or shutting down an engine inflight, is affected more performed the procedure by reference to the checklist
by stress than are common tasks. This is “an effect that did not make these errors, but took longer to complete
has profound implications for the design of procedures the checklist. Regardless of the time required to read
to be used under the stressful conditions of through the checklist, the crews performing the
emergency” [9]. procedure by reference descended to a safe altitude in
less time because of the use of the speedbrake.
This effect can sometimes be observed when people
continue with a planned series of actions they are The perceived requirement to perform checklist steps
familiar with even when the actions appear quickly from memory during high-stress situations is at
unsuccessful or inappropriate. By acting before odds with the need to perform those checklist steps
analyzing the situation, the operator may exacerbate accurately. There is a potential for loss of accuracy as
the situation, which may induce more stress, and make the performance speed increases. Attempting 100%
it increasingly difficult to identify the original cause of accuracy would require so much time to complete a
the failure. This is related to an effect referred to as checklist that other flying tasks would be disrupted.
confirmation bias, where a person attends to cues that There is a tradeoff between getting the procedure done
support a belief, and discounts cues that contradict the quickly, and getting it done while minimizing the
belief. Confirmation bias has been demonstrated in the possibility of error.
use of automation and even in the diagnosis of
everyday situations [4, 5, 7]. Other studies have shown The following methodology seeks to identify examples
that under stress, subjects are less effective and more of these errors in the flight operations domain. Even
disorganized at considering alternative solutions and though inducing a level of stress similar to that of a
incorporate less data in decision-making [6]. real emergency was not possible in this study, it was
hypothesized that errors of commission, omission, and
Stress Effects on Completion of Checklist Steps order would still be observed.

Discussions with pilot participants in this study suggest Methodology

that the requirement to perform certain actions from
memory implies a sense of urgency in the performance Participants
of those actions. This introduces another potential
source of error due to the loss of accuracy as speed is Sixteen 737 line pilots at a crew base of a major U.S.
increased, an effect that is best described by the speed- airline volunteered for the study. These pilots were
accuracy operating characteristic (SAOC). The SAOC already at the crew base either in preparation for a
is a function that represents the inverse relationship flight or returning from one. Participants were
between accuracy and speed. As the performance of a accepted without regard to experience level and
task requires more speed, accuracy is reduced until it participated in the study individually and not as a
approaches chance. If accuracy is excessively member of a two person crew. Pilots reported being
emphasized, then the time required to complete a task trained in both the 737 Classic and 737 NG.
increases greatly with little improvement in accuracy.
Procedure
Wickens & Hollands [9] summarize studies that
demonstrate the effects of stress, induced by speed or In order to avoid any priming effects in the recall of
by threat of bodily harm, on performance accuracy. their emergency procedures, subjects were not
For example, bomb-disposal experts performing under informed of the purpose of the research. They were
stress made more errors while working faster, and instead briefed that:
subjects who were threatened with the potential for
electric shock gave up on problem-solving activities the research was on the suitability of the 737
early. alerting system,
they would be asked to talk through five
Using an emergency descent as an example, an earlier procedures, and
study [2] showed that crews performing an emergency the results from this study may be relevant to the
descent from memory took longer to descend than design of a new alerting system in future airplanes.
crews using the checklist. The difference in descent

21
A brief survey of experience was collected. This who did not have military experience came from
included data on total number of hours flown, their various corporate jets, commuter planes, other large
time in airplane type, flying time since last PC, and commercial airlines, and corporate turboprops.
their crew position.
Checklist Selection Errors
Subjects were seated in front of a poster of the flight
deck. For consistency, a color poster of the 737 Classic When pilots were given an engine start condition with no
flight deck was used. Five non-alerted abnormal oil pressure indications, four pilots initially chose the
procedures that contain memory items were used. They Engine Low Oil Pressure checklist. Upon reading that
included aborted engine start, engine limit/surge/stall, checklist, two of those pilots realized it was not
rapid depressurization, runaway stabilizer trim, and appropriate for the situation, and correctly selected the
dual engine failure. Aborted Engine Start checklist. One pilot reported that
there was no checklist needed, and that a maintenance call
The experimenter began each scenario by describing a would be the only action required after completing the
normal flight situation, and then interjecting cues that engine shutdown. The remaining 10 pilots correctly
suggest a particular failure. Subjects were asked to referenced the Aborted Engine Start checklist (Table 2).
react to the cues as they would inflight, performing any
procedures they felt were necessary. When responses # of
to the scenarios seemed vague, the researcher probed Checklists selected pilots
the participants to encourage them to elaborate. The Aborted Engine Start 10
participants were provided with their airline Quick
Reference Handbook (QRH), and were allowed to Engine Low Oil Pressure 2
select the checklist they felt was most appropriate for Engine Low Oil Pressure > Aborted Engine
the situation. Each session lasted approximately 30 Start 2
minutes. None 1

Results Table 2. Aborted Engine Start Checklist Selection

Demographics The Engine Limit/Surge/Stall scenario had the lowest

identification rate (Table 3). Only two pilots
The participants in this study were 16 current line referenced the correct checklist. One of those two
pilots at a major U.S. airline. Of those pilots, one was selected the Engine Fire/Severe Damage/Separation
eliminated from the final analysis because he checklist first. The remaining pilots referenced various
determined during the interview that an evaluation of checklists, including Engine Fire/Severe
the performance of memory items was the goal of the Damage/Separation, Engine Failure/Shutdown, and
research. Engine Overheat.
Weeks
Since
Total Months Time in QRH # of
Time Since PC Type Used Checklists selected pilots
13,40 Engine Limit/Surge/Stall (Correct) 1
Mean 4 6 6,614 13
Standard
Engine Fire > Engine Limit / Surge / Stall
Deviation 6,829 4 6,535 17 (Experimenter prompted the correct checklist 1
by saying the engine was “surging”)
Minimum 4,500 0.5 400 1
Engine Failure 6
25,00
Maximum 0 11 20,000 52 Engine Fire 4
Engine Overheat > Engine Fire 1
Table 1. Demographics
Engine Overheat 1
Data from the experience survey is presented in Table Engine Overheat > Engine Failure 1
1. Nine First Officers and six Captains participated.
Two pilots incorrectly reported their total time and Table 3. Engine Limit / Surge / Stall Checklist
time in type, and their numbers were excluded. Seven Selection
pilots had prior military experience ranging from land
and carrier-based fighters to large transports. Pilots The remaining three scenarios had few checklist

22
selection errors. One pilot selected the Auto
Fail/Unscheduled Pressurization Change checklist
Engine bleed switches..........................On
during a rapid depressurization. Another pilot
performed the Stabilizer Out Of Trim checklist in the Isolation valve...................................Close
runaway stabilizer scenario. Oxygen masks & regulators....On/100%
Crew communications............Establish
Checklist Step Errors
Pressurization mode selector.........Man
The majority of checklist step errors occurred during Outflow valve.................................Close
the completion of the dual engine failure memory
items. Many of those were commission errors. These Figure 2. Rapid Depressurization Commission
included: Errors.

• bringing the thrust levers back to idle before Four pilots made commission errors in the completion
attempting to restart the engine, of the runaway stabilizer trim checklist by attempting
to activate the electric trim switches in the direction
• advancing the thrust levers as the engines failed in
opposite the runaway. One of those four pilots stated
an attempt to get them to restart,
that he would also attempt to engage a different
• starting the APU to try an assisted start,
autopilot in the hopes that it would not experience the
• waiting three seconds to attempt a restart after same malfunction (Figure 3).
shutting off the fuel,
• placing the ignition selector to both, and
• using engine anti-ice (Figure 1). Control column.....................Hold firmly
Autopilot (if engaged)...........Disengage
Ignition Selector.................................Both
Thrust Levers...............................Advance Electric trim in opposite direction

Engine Start Levers.............................Idle Engage other autopilot

Engine Start Switches........................Flt If runaway stabilizer continues:
Turn around Trim cutout switches...............Cutout
Thrust Levers...................................Close Trim wheel.....................Grasp & hold
Engine Anti-ice......................................On
Figure 3. Runaway Stabilizer Commission Errors.
Engine Start Levers......................Cutoff
EGT decreasing: Discussion
Wait three seconds:
Checklist Selection Errors
Engine Start Levers...........................Idle
APU...................................................Start When presented with cues to an abnormal situation,
If EGT exceeds 950°C: pilots sometimes omit a thorough analysis of the
Allow engines to overheat
situation. This became evident through previous
observations of pilots performing abnormal procedures
Repeat above steps in simulators and anecdotal evidence. The pilots in this
Attempt restart one at a time study demonstrated a tendency to fixate on the most
prominent cue and perform the checklist appropriate to
Figure 1. Dual Engine Failure Commission Errors. that cue. However, a thorough analysis of the situation
Bold items indicate the correct steps. Arrows indicate can reveal that the single most prominent cue does not
all additional steps performed by the 15 pilots. always lead the pilot to the correct checklist.

In the rapid depressurization scenario, two pilots There were 23 checklist selection errors. With the
included additional steps: following three exceptions, the errors appear to be
caused by the pilots’ fixation on a single cue.
• verifying the engine bleeds were on, and Experimenter error in describing the rapid
• closing the bleed air isolation valve (Figure 2). depressurization failure to one pilot gave the
impression that the cabin altitude began to stabilize at
approximately 12,000 feet, which led him to the Auto

23
Fail/Unscheduled Pressurization Change checklist. Checklist Step Errors
Another error was due to a pilot’s belief that no
checklist was required for an aborted engine start. There appear to be consistent patterns in the observed
Finally, one pilot referred to the Dual Engine Failure checklist step errors. Many of the commission errors
checklist as the Engine Inflight Start checklist, but appear to result from the pilots’ creativity in dealing
performed the correct memory items. with an abnormal situation. It was observed that many
pilots perform steps in addition to what was required
The remaining 20 checklist selection errors appear to based on their understanding of how the airplane
be caused by pilots fixating on a single cue, and systems functioned, even though their understanding of
performing the checklist that appears most related to the systems may be incorrect. Some pilots explained
that cue. For example, in the aborted engine start, the that the performance of some additional steps occurs
cues given to the pilots were the continued because of knowledge of the intricacies of a complex
illumination of the LOW OIL PRESSURE light and no system gained over years of experience or knowledge
oil pressure indication. Four pilots stated that, given of common and simple failure modes, which are not
those cues, they would complete the Low Oil Pressure addressed in the checklist. This may resolve the
checklist. situation without the need for a checklist. In other
cases, an incorrect or incomplete understanding of the
Two of those pilots realized the Low Oil Pressure system may lead pilots to perform additional steps that
checklist was inappropriate by considering the delay the completion of steps necessary to resolve the
reasonableness of the checklist steps they were situation, or that may exacerbate the condition.
reading. The checklist directed the pilots to the Engine
Failure/Shutdown checklist, which is meant for an The pilots’ creativity in dealing with certain situations
inflight engine shutdown. A shutdown of an engine on was most evident in the dual engine failure scenario,
the ground is simpler than a shutdown inflight and which had the highest number of commission errors. A
these pilots determined that irrelevant steps such as: possible explanation was apparent in the pilots’
starting the APU, maintaining fuel balance, and response to this scenario: a desire to “do whatever it
preparing for a single-engine landing, indicated they takes” to resolve a serious situation. Their perception
were in the wrong checklist. However, one pilot who was that this failure was so severe that they would
entered the Engine Failure checklist from the Low Oil exercise their authority as pilots, beyond what is
Pressure checklist did not consider the appropriateness written in the checklist, in an attempt to get an engine
of the checklist steps he was reading, and showed a running, regardless of the consequences. Some pilots’
tendency for perseveration. He went so far as to willingness to allow the engines to exceed EGT and
complete the Engine Failure checklist, reading aloud overheat, contrary to the guidance in the checklist,
and bypassing irrelevant steps to complete the only demonstrated this belief.
step required to actually shutdown the engine while on
the ground. Most errors of commission were intended to
troubleshoot the failures, such as: advance the thrust
In the engine limit scenario, the 14 subjects who did levers, verify the start levers are at idle, turn around to
not select the correct checklist instead performed the exit the heavy rain that caused the failure, and
checklist that most closely reflected the cue they said manually select both igniters. This last step
was the most important. One pilot initially selected the demonstrates a misunderstanding of the ignition
Engine Fire/Severe Damage/Separation checklist, but system. By correctly completing the recall item in the
turned to the Engine Limit/Surge/Stall checklist only checklist, both igniters were automatically energized.
after the experimenter said the engine was “surging”.
The term “surging” was not used as a cue in any other When the situation called for a shutdown of both
scenarios. Pilots who were primarily concerned by the engines, two pilots performed the additional step of
abnormal “popping” or “banging” noises referenced delaying 3 seconds between restart attempts. They
the Engine Fire/Severe Damage/Separation checklist, explained that this stemmed from a folk belief carried
stating that they believed the noises suggested severe over from their military background that additional
engine damage. Pilots who considered excessive time was needed for excess fuel to clear the engine
exhaust gas temperature (EGT) to be more important before attempting a restart.
completed checklists related to overheat conditions.
The pilot who referenced the Stabilizer Out Of Trim This disposition towards creative troubleshooting was
checklist in the runaway stabilizer scenario did so also seen in the Runaway Stabilizer Trim and Rapid
because he believed the STAB OUT OF TRIM light Depressurization checklists. Errors of commission
would be illuminated. included moving the electric trim switches in the

24
opposite direction and engaging the other autopilot. demonstrated a misunderstanding of how the systems
One pilot reported that he had experienced a runaway in the 737 functioned. Other errors were a result of
stabilizer in the past, and activating the electric trim either knowledge gained during a real experience in
switches stopped the runaway. This is an example of a the past, or a belief carried over from previous
pilot’s knowledge of the failure modes of a complex organizations and airplanes, which may no longer be
system that could resolve the situation without using a applicable.
checklist.
Implications
The rapid depressurization scenario showed that some
commission errors, such as closing the isolation valve Even though the method used in this study did not
and ensuring the engine bleeds are on, would not induce stress, it allowed for an evaluation of the pilots’
exacerbate the situation, but would not be beneficial knowledge of the memory items without prior
either. They would simply delay the completion of the preparation. Pilots generally perform well during their
necessary steps. Moreover, the manual closing of the PCs, and possibly better than inflight, because they
isolation valve demonstrates a lack of understanding of expect an evaluation and can prepare for it. Pilot
the bleed air system. This step is not required because performance observed in this study may be closer to
the valve is already closed during its normal operation. that in an inflight emergency, in which the pilots are
unprepared to perform their memory items.
On the other hand, some commission errors aggravated
the situation. An example was seen in some pilots’ Clearly, an inflight emergency places a pilot under a
willingness to allow the engines to overheat while great deal of stress. Based on the literature review, it
restarting after a dual engine failure. The consequence can be inferred that errors similar to those observed
of the overheating could be engine damage and a true here may occur inflight during an actual emergency,
engine failure, instead of the original problem of a and may even occur more frequently due to increased
temporary flameout due to an environmental condition stress. Conducting a similar study in a full-flight
such as heavy rain, resulting in no engine damage. simulator may provide a level of stress similar to what
is experienced in a real emergency. The results
Conclusion obtained from a simulator could be a more realistic
representation of the results obtained inflight.
The results demonstrate that pilots have difficulty
identifying the cause of the failure and selecting the Acknowledgments
correct procedure. After identifying the situation,
knowledge of the appropriate memory items is such The author wishes to thank Barbara Holder and Randy
that pilots commit errors in recall even during Mumaw for their valuable guidance throughout the
unstressed conditions with a poster of the flight deck course of this project. The author also wishes to thank
for context. reviewers of drafts and the pilots who assisted by
reviewing and commenting on research methodology,
None of the five failure scenarios in this study had a which greatly influenced the quality of data collected.
distinct indicator light that would annunciate the The author also wishes to thank those who gave their
condition. Pilots were forced to analyze the cues and time to allow us to gain some insight into the
determine the appropriate procedure. This is an challenges they face in the cockpit.
uncommon and involved task, and not performing it
may force pilots to complete only those tasks they are References
familiar with, such as following an illuminated LOW
OIL PRESSURE light to the Low Oil Pressure 1. Baddeley, A. D. (2000). Selective attention and
checklist during an aborted engine start, or fixating on performance in dangerous environments. Human
abnormal engine noises and performing the Engine Performance in Extreme Environments, 5(1), 86-91.
Fire/Severe Damage/Separation checklist, instead of (Reprinted from Baddeley, A. D. 1972. The British
the more appropriate Engine Limit/Surge/Stall Journal of Psychology, 63(4), 537-546.)
checklist. 2. Hamman, W. R. (n.d.). Quick Reference
Checklist (QRC). United Airlines validation study.
The observed checklist step errors showed that pilots Colorado: Author.
commit a number of errors. The majority of the
commission errors were steps performed by pilots to
resolve a failure based on their knowledge of the
airplane systems. Some of these commission errors

25
 3. Idzikowski, C., & Baddeley, A. D. (1983). Fear Driskell & E. Salas (Vol. Eds.), Stress and human
and dangerous environments. In D. H. Holding (Series performance (pp. 1-45). New Jersey: Lawrence
Ed.) & G. R. J. Hockey (Vol. Ed.), Stress and fatigue Erlbaum.
in human performance (pp. 123-144). New York: John 7. Skitka, L. J., Mosier, K. L., Burdick, M.,
Wiley & Sons. Rosenblatt, B. (2000). Automation bias and errors: Are
4. Mosier, K. L., Skitka, L. J., Dunbar, M., crews better than individuals? International Journal of
McDonnell, L. (2001). Aircrews and automation bias: Aviation Psychology, 10(1), 85-97.
The advantages of teamwork? International Journal of 8. Thompson, L. A., Williams, K. L., L’Esperance,
Aviation Psychology, 11(1), 1-14. P. R., & Cornelius, J. (2001). Context-dependent
5. Reason, James. (1990). Human Error. New York: memory under stressful conditions: The case of
Cambridge University Press. skydiving. Human Factors, 43(4), 611-619.
6. Salas, E., Driskell, J. E., & Hughes, S. (1996). 9. Wickens, C. D., & Hollands, J. G. (2000).
Introduction: The study of stress and human Engineering psychology and human performance. New
performance. In E. A. Fleishman (Series Ed.) & J. E. Jersey: Prentice Hall

26