Fidelity and validity of simulator training

Theoretical Issues in Ergonomics Science Vol. 10, No. 4, July–August 2009, 305–314 Fidelity and validity of simulator training N. Dahlstroma*, S. Dekkera, R. van Winsena and J. Nyceb a Lund University School of Aviation, Ljungbyhed, Sweden; bDepartment of Anthropology, Ball State University, Muncie, IN, USA Downloaded By: [Dahlstrom, Nicklas] At: 10:51 16 May 2009 (Received 20 February 2008; final version received 25 July 2008) Through a case study, this article explores a number of theoretical issues related to the often taken for granted relationship between simulator fidelity and the quality and transferability of training in complex, dynamic, safety-critical settings. A counterexample based on mid-fidelity simulation is presented and the assumed coincidence of fidelity and validity is tested, that is the study tests the equation of constructed photorealism (built to mimic reality) and effective development of the competence that operators require to manage situations that involve underspecified problems, time pressure constraints and complex group interaction. The article concludes that such competence development cannot rely only on highly context-specific (photorealistic) environments. Further, it will be argued that lower-fidelity simulation, when appropriately designed, can provide competence development with pedagogical and economic advantages. Keywords: safety; simulation; fidelity; training; aviation 1. Introduction Much of the training in one operational world today – aviation – focuses primarily on ‘technical skills’ (see Dahlström et al. 2006), i.e. the build-up of an inventory of proceduralised interaction with the technical systems of an aircraft. From the start, this training is quite context-specific: it is set in, and tightly anchored to, the local technical environment (of an aircraft cockpit or a simulator with as high a fidelity as economically feasible) in which problem-solving activities are to be carried out. This is analogous to other domains that assert the primacy of learning-by-doing (for example surgery; see Bosk 2003). This means that individual and group generic competencies such as problem solving, communication, coordination or management of unanticipated and escalating situations are likely to arise and emerge only during the exercise of context-specific work (Klein 1998). Such exercise is valorised by the increasing technical sophistication of simulated training environments, in worlds ranging from aviation to shipping to healthcare, which are now using ‘proceduralisation’ of interaction as a means of increasing safety. This article explores a number of theoretical issues and systemic aspects related to the often taken for granted relationship between simulator fidelity and the quality and transferability of training in complex, dynamic, safety-critical settings. A counterexample is presented based on mid-fidelity simulation and the assumed coincidence of fidelity and validity is tested, that is the study tests the equation of constructed photorealism (built to mimic reality) and effective development of the *Corresponding author. Email: nicklas.dahlstrom@tfhs.lu.se ISSN 1463–922X print/ISSN 1464–536X online ß 2009 Taylor & Francis DOI: 10.1080/14639220802368864 http://www.informaworld.com 306 N. Dahlstrom et al. operator competence required to manage situations that involve underspecified problems, time pressure and complex group interaction. Can such competence development rely on highly context-specific (photorealistic) environments alone? Can lower-fidelity simulation, when appropriately designed, provide competence development with pedagogical and economic advantages? Downloaded By: [Dahlstrom, Nicklas] At: 10:51 16 May 2009 1.1. Fundamental surprises Operational activities in any industry contain situations whose subtle and infinite variations may mismatch the exact circumstances of training. This includes ‘fundamental surprises’ (Lanir 1986), i.e. situations that fall outside the procedures constructed for normal and emergency operations. On these occasions, operators must be able to apply skills and knowledge that no training department was able to foresee or deliver. This leaves a residue of potential problems that crews are not prepared for (i.e. they are not in their inventory; see Dismukes et al. 2007). The same is true for other industries and operators as well. Formal mechanisms of safety regulation (through, for example, design requirements, policies, procedures, training programmes, checks of operator competence) will always fall short in foreseeing and meeting the shifting demands posed by uncertainty, limited resources and multiple conflicting goals. For this residue one has to count on crews’ generic competencies to add to the ability of a system to respond to unexpected and escalating situations. These surprises at the margins of an otherwise very safe system stem from limits in the industry’s knowledge or, more often, limits on its ability to put together diverse pieces of knowledge, as well as from limits on the understanding of operational environments (Lanir 1986). In other words, the knowledge base for creating safety in complex systems is inherently imperfect (Rochlin 1999). Often the problem is not that the industry lacks the data. After all, the electronic footprint left by any commercial flight today is huge. The problem is that this accumulation of noise and signals can muddle both the perception and conception of ‘risk’ (Amalberti 2001, Dekker 2005). Pockets of expertise that may have predicted what could go wrong often exist in some corner of the industry long before any accident. An example is the Swissair 111 accident in 1998, where the crew responded promptly to the presence of smoke in the cockpit by adhering to the relevant checklist. However, while following the established and trained procedures, fire engulfed the aircraft (Transportation Safety Board of Canada 2003). This accident reminded the industry of the immense difficulty that crews face in making trade-offs when adapting plans and procedures under duress and uncertainty and of industry shortcomings in preparing them for incidents like these (Dekker 2001). In this case, following the procedures turned out to be the problem rather than the solution and this tragedy initiated further research on the use of checklists and procedures (Burian and Barshi 2003). Regarding training for these types of situations, Burian et al. (2005, p. 3) concluded that: ‘the degree to which training truly reflects real life emergency and abnormal situations, with all of their real-world demands, is often limited’. More recent is the accident of Pinnacle Airlines flight 3701 in 2004, where pilots ferrying an empty 50-seat aircraft carried out several non-standard manoeuvres that made the engines shut down in flight and then failed in their attempts to restart them. This accident exposed lacks in crew knowledge of high-altitude operations, handling of low-speed and stall situations, recovery from double-engine failures and on how the Theoretical Issues in Ergonomics Science 307 absence of passengers erodes operational margins (National Transportation Safety Board 2007). The type of engine on the Pinnacle type aircraft had a history of problems with in-flight restarts during flight tests. But few or no operational crews would have been aware of any of this, in part because of structural industry arrangements that regulate who gets or needs to know what and in what depth. Similar restrictions of access to knowledge have played a role in accidents in other industries (e.g. nuclear industry, the Three Mile Island accident and Chernobyl). Downloaded By: [Dahlstrom, Nicklas] At: 10:51 16 May 2009 1.2. Resilience and the limits of expertise As a result, some crews will, at some point or other, be left to ‘fend for themselves’ at the edges of an extremely safe industry. It is at these edges that the skills trained for meeting standard threats need transpositioning to counter threats no one has foreseen. The flight of United Airlines 232 in 1989 is an extreme example (National Transportation Safety Board 1990). The triple engine DC-10 lost total hydraulic power and became seemingly uncontrollable as a result of a mid-flight tail engine rupture, with debris ripping through all hydraulic lines that ran through the tail-plane and subsequent loss of hydraulic fluid. The crew figured out how to use differential power on the two remaining engines and steered the craft toward an extremely difficult high-speed landing at Sioux City, Iowa. Despite their efforts, the plane broke up on the runway, but the majority of the passengers and crew subsequently survived the landing. In simulator re-enactments of this scenario, none of 42 crews managed to get the aircraft down on the runway. Both the crew and the investigation concluded that the relatively successful outcome of this impossible situation could largely be attributed to the training of general competencies in the carrier’s crew resource management (CRM) training programme. Thinking outside the box, taking a system way beyond what it was designed to do (even making use of an adverse design quality such as pitching moments with power changes) are hallmarks of resilience. Resilience is the ability of a system to recognise, absorb and adapt to disruptions that fall outside a system’s design base (Hollnagel et al. 2006), where the design base incorporates soft and hard aspects that went into putting the system together (e.g. equipment, people, training, procedures). Resilience is about enhancing people’s adaptive capacity so that they can recognise and counter unanticipated threats. Adaptive capacity with regard to a narrow set of challenges can grow when an organisation courts exposure to smaller dangers (Rochlin et al. 1987). This allows it to keep learning about the changing nature of the risk it faces – ultimately forestalling larger dangers. Such adaptation could be one explanation behind recent data that suggest that the passenger mortality risk on major airlines that have suffered non-fatal accidents is lower than on airliners that have been accident-free (Barnett and Wang 2000). United Airlines 232 at Sioux City is such a case. Crews can effectively counter many threats by replicating or slightly varying the technical skills learned during their training. Most non-normal situations in commercial aviation, after all, are quite ordinary or at least recognisable: they fit within the anticipated scenarios and, accordingly, crew behaviour can stay within the prescribed sets of procedures. Then there is a huge middle ground. It consists of daily safety threats that feature (or occur because of) subtle variations that call for some adaptive capacity and response. For example, these threats can demand extra work (e.g. gathering and processing of more data, increased communication and coordination), recruitment of additional expertise (e.g. dispatch, air traffic control) and the deployment of new strategies. Resilience, however, means effectively meeting threats 308 N. Dahlstrom et al. Downloaded By: [Dahlstrom, Nicklas] At: 10:51 16 May 2009 that represent infinite reconfigurations of – or ones that may lie entirely beyond – what the industry could anticipate. 2. Simulation fidelity and development of resilience The aviation industry has relied on simulation perhaps more than any other safetycritical industry. While simulators are still used for stick-and-rudder and instrument training, today they are also part of practically all aspects of aviation training (Salas et al. 1998) and the new multi-crew pilot licence (MPL) rests almost entirely on simulated flight training. This investment in simulation reflects an industry-wide confidence that it can save time, money and lives (Bürki-Cohen et al. 1998), in addition to provide effective training – i.e. developing skills and knowledge that are transferable to any target situation. The evolution of simulation in aviation has mainly been technology-driven, from the introduction of visual systems and computer graphics (Dennis and Harris 1998, Lee 2005), to recent additions of satellite imagery to represent the visual scene of the ground below and current moves to further integrate into simulations the role of air traffic control (Longridge et al. 2001). Although this increase of face-validity to improve training quality has been questioned (Roscoe 1991, Salas et al. 1998, Dahlström and Nählinder 2007), there seems to be a taken for granted assumption within the aviation community that incremental quantitative progress (e.g. more computing power, higher resolution, greater visual angles) adds up to a positive qualitative difference. In other words, as a simulated environment becomes ever more ‘photorealistic’, so does the yield that simulations have for crews and staff. This link between maximum fidelity to maximum training transfer is taken on faith: if it looks real it will provide good training. However, over time the continually increased demand for higher levels of fidelity to make simulations look ‘real’ increase cost and lower availability of training simulators (one has to keep in mind the huge capital investment a commitment to this style of simulation requires). Even though there have been few studies of transfer of training from photorealistic simulators to aircraft (Carretta and Dunlap 1998, Dahlström and Nählinder 2007) and the problems of performing such studies have been documented (Hays et al. 1992, Bell and Waag 1998), the assumed relation between fidelity and transfer of training seems to prevail in the aviation industry (in other industries too, e.g. maritime transport, nuclear power, medicine, as well as in the military). Some have suggested that lower-fidelity simulations (ones that do not attempt to mimic directly the target technical environment) are a cost-effective alternative and may actually improve many aspects of learning that help people deal with unanticipated situations (Rouse 1981, Roscoe 1991, Caird 1996). As Caird (1996, p. 127) put it: ‘For decades, the naı̈ve but persistent theory of fidelity has guided the fit of simulation systems to training’. In addition, Heeter (1992) concluded that the environmental presence experienced in simulated environments is determined more by the extent to which it acknowledges and reacts to the participant than by the simulation’s physical fidelity. In other words, high levels of technologically driven fidelity can simply be wasteful in terms of costs and time relative to the pedagogical undertaking at hand. (The cost of these simulations may also inadvertently limit access to what the industry considers to be ‘proper’, ‘legitimate’ training opportunities.) As well, ‘featurism’ can be distracting (Jackson 1993), both for the trainer and the trainee, especially when the features argued for, promoted and designed in are skewed in the direction of realism. Downloaded By: [Dahlstrom, Nicklas] At: 10:51 16 May 2009 Theoretical Issues in Ergonomics Science 309 The emphasis on photorealism in visual and task contexts may retard or limit the development of skill sets critical for creating safety in domains where not all combinations of technical and operational failure can be foreseen or formalised (and for which failure strategies then cannot be proceduralised and simulated). The assumption that photorealism can capture all possible naturalistic cues in addition to the skills necessary to act competently in these domains may be overly optimistic. Competencies the aviation community recognises as important and significant (e.g. communication, coordination, problem solving, management of unanticipated and escalating situations) are thought to emerge directly from context-fixed simulator training. It is assumed that photorealism can achieve these ends. The focus on face-validity has muted perspectives on simulation styles and use that could allow a more subtle analysis of cognitive and group interaction aspects to form the base of training. This is particularly true for training of skills related to CRM (Baker et al. 1993). It is in unusual, unanticipated and escalating situations where such skills are most needed. Highly dynamic situations involving underspecified problems, time pressure constraints and complex group interaction are situations that cannot be resolved through procedural guidance. Pertraglia (1998) observed that experience in the world can be neither ‘predetermined nor preordained’ and that this, together with the willing suspension of disbelief, is what should make a simulated activity seem authentic as experience. However, the quarter-century-long aviation industry focus on CRM has resulted in few attempts to provide training that addresses situations with underspecified problems and time pressure in the context of group interaction. The commitment to highly realistic simulation has meant that crews have not been well trained when it comes to situations that are neither ‘predetermined nor preordained’. The recently introduced MPL, which aims to qualify candidate airline pilots as part of a crew from the very beginning of their training by increased use of simulated flight training, is an opportunity to review the relationship between simulator fidelity, quality and transferability of training and the underlying assumptions upon which this training is based. 3. Fidelity and validity: an example of a more complex relationship than assumed Lund University School of Aviation has been experimenting with lower-fidelity simulations over the last two years to assess the relationship between simulation fidelity and validity – that is, the connection between the faithfulness of the constructed world in which operators are trained on the one hand, and the extent to which this actually supported the development of skills useful in target environment. These experiments are presented here as an example of the complex relation between fidelity and validity of simulator training. These studies have used a simulation of a ship’s bridge (essentially consisting of a laptop computer, printer and table top) that is simple in regard to the participant interface (printouts) but with time-pressure and various event-driven scenarios built in. These range from minor problems to serious threats to the safety of the ship M/S Antwerpen (Strohschneider and Gerdes 2004). This ‘mid-fidelity’ (far from ‘real’ but capturing sufficient and salient aspects of reality) simulation has been run twice during a two-day M/S Antwerpen training programme, which also included debriefings, discussions and lectures. The participating groups were made up of student pilots, maritime students and operators and the safety and security group of Lund University. Observational notes and video recordings were taken to allow the simulation sessions to be Downloaded By: [Dahlstrom, Nicklas] At: 10:51 16 May 2009 310 N. Dahlstrom et al. used as a series of case studies on information management, group coordination, leadership and decision making as well as on management of unusual, unanticipated and escalating situations (Dahlström 2006). During debriefings, participants have consistently stated that this simulation provides them with relevant and valuable training for their actual work and explicitly wished for more training of this type. In particular, participants with no maritime background find the ‘cross-domain’ elements of the training beneficial. In spite, or perhaps because, of its lack of fidelity to photorealistic representation and feedback, the engagement and level of intensity of communication, cooperation and decision making observed in groups normally surprise the participants themselves as well as instructors. Participants typically bring this up and note that this surprising ability to engage has had an important influence on the training effect. There have been very few requests from participants for increased fidelity to improve the simulation; in fact, contrary opinions have been more often recorded. These comments, together with observations of the over-emphasis (in particular on the first simulation run) on technical ‘real world’ ship parameters (e.g. engine rpm, course and roll angle) and lack of process-oriented discussion, indicates that potential ‘improvements’ to fidelity could in fact have a detrimental effect on the validity of the training. For example, if higher-fidelity features were to be ‘engineered in’, e.g. knobs, levers and buttons, participants are of the opinion that this would shift their focus from generic to procedural competencies. Also, since building in such higher-fidelity items could never bridge the qualitative gap to the ‘real’ thing in any case, much participant attention and commentary would be directed to their insufficiency or still unconvincing nature. The lack of such features leaves groups with no option but to focus on use of general competencies as tools to manage the situations they encounter. Participant concerns did not focus on the relevance of the simulation (most participants, it needs to be kept in mind, were non-maritime operators) nor on issues during simulation exercises related to maritime technical or procedural knowledge. Instead the researchers heard much from participants about how training of this kind could support development of competencies and understanding applicable to problems that an operator in any industry might encounter. Participants recognised the importance of experiencing and learning to address problems such as unusual, unanticipated and escalating situations in combination with underspecified problems, time pressure constraints and complex group interaction, issues seldom encountered in regular operator training, but key features of the simulation. These observations support the idea that a shift of domain seemed to recreate and emphasise the types of uncertainties reported in incident and accident investigations. High-fidelity flight simulator training is normally focused on removing, rather than enabling, participant understanding of such uncertainties. In addition, the non-domain-specific environment seemed to encourage participants to step out of their normal roles and explore aspects of general group interaction competencies – ones not covered by standard procedures and instructional theory. An example of this was seen during a fire on board the M/S Antwerpen, where one operator in a group of maritime operators insisted on relieving the captain of handling passenger evacuation since he was getting overwhelmed trying to control the spread of the fire. Also, an analysis of M/S Antwerpen group processes and outcomes (in terms of saved lives and damage to ship) show that groups that built up and relied on generic competences performed better than those who relied more heavily on established roles and procedures (van Winsen et al. 2008). This is most clearly seen when maritime students on M/S Antwerpen exercises cooperated far beyond the formal responsibilities of their respective roles compared to maritime operators, who established Downloaded By: [Dahlstrom, Nicklas] At: 10:51 16 May 2009 Theoretical Issues in Ergonomics Science 311 standard roles and explicit procedures that they then were unable to break out of as an emergency escalated. The observations from the M/S Antwerpen simulation support that inherent in problem-solving exercises carried out in high-fidelity (realistic), highly context-specific environments is the risk that they can impede people’s imaginative, creative involvement and the resilience it may eventually deliver to the workplace. Training in high-fidelity settings alone valorises the internalisation of a series of highly contextualised instrumental stimulus–response relationships – putatively stress-resistant procedural responses that may be insensitive to, or even make actors unprepared for, contingencies outside of rehearsed routines. If the desire is to have operators successfully extrapolate and improvise beyond a set of fixed, learned responses, this issue of what is ‘carried away’ from context-specific simulation exercises needs to be looked at more carefully than it has been in the past. As Roscoe (1991, p. 1) notes: ‘Research has shown that innovations in training strategies, in some cases involving intentional departures from reality, can have stronger effects than high simulator fidelity on the resulting quality of pilot performance’. Indeed, as Caird (1996, p. 128) adds: . . . there is some evidence from flight simulation that higher levels of fidelity have little or no effect on skill transfer and reductions in fidelity actually improve training. Reductions of complexity may aid working memory and attention as skills and knowledge are initially acquired. It may be that the lack of physical fidelity in the lower-fidelity simulation can enhance the focus on training of general principles of communication, coordination and problem solving in a workgroup. These principles are ones that actors can use to understand and resolve situations beyond those covered by procedural guidance (Dörner 1989). Locking training to context-specific environments affects more than the exportability of instrumentally rehearsed skills. It can also amplify and reify role socialisation. Effective management of escalating or otherwise novel situations has been associated with the breaking out of roles and power structures that were formally designed into the system. It is not clear whether naı̈ve (built to mimic reality) simulation can ‘train’ in this direction at all. When roles are involved, Weitz and Adler (1973, p. 224) concluded that: ‘. . . it might be wise to stress the principles, not the roles’ to ensure that participants do not ‘become wedded to particular performances’. Roles and power structures often go hand-in-glove (e.g. Captain and First Officer) and various programmes (e.g. CRM training in aviation) aim to soften role boundaries and flatten hierarchies in order to increase opportunities for coordinating viewpoints and sharing information. Operational success in the face of extreme or rapidly shifting demands can hinge on individuals going beyond the formal roles assigned to them – as illustrated by various near-accidents or accidents that could have been worse, such as United 232 at Sioux City (see also Dekker 2005). 4. Conclusion If no training opportunities exist in which individuals can disconnect from the constant reification and elaboration of their normal operational activities, the ability to respond effectively may remain inextricably anchored to (and fundamentally limited by) known and rehearsed roles, duties and procedures (as shown powerfully in Weick 1993). High-fidelity simulation is normally a costly and restricted resource primarily used to train and emphasise such roles, duties and procedures. Lower-fidelity simulation can complement high-fidelity simulation and serve as an important resource in the creation Downloaded By: [Dahlstrom, Nicklas] At: 10:51 16 May 2009 312 N. Dahlstrom et al. of resilient operators. In the research that has been conducted, it seems to force participants to confront the interpersonal and goal-oriented demands of managing unanticipated and escalating situations. These are exactly the work elements that seem to be lost or hidden behind the procedural specifics fostered by high-fidelity training and operational experience. In addition, the limited costs connected to using lower-fidelity simulation could increase availability and frequency of training sessions. Both the studies of aviation accidents and the use of lower-fidelity simulation reveal a disconnect between the fidelity (or photorealistic faithfulness) of a simulation and its validity (how the skills it develops map on to situations in the target environment). Lowerfidelity simulation allows the development of generic problem-solving skills, such as sharing knowledge, making and following up on plans, dividing work, stepping back for broader evaluation, borrowing time from the future by current task investments and maximally exploiting a group’s available expertise. These skills (and the confidence that comes from successfully deploying them in settings other than the target environment) could contribute significantly to the development of resilient crews in ways that reliance on considerably more costly and more high-fidelity (photorealism) training cannot. Traditional assumptions about simulation tend to portray both role and context as though they are natural, unalterable facts. This message seems to be implicit in almost all attempts at ‘realistic’ simulation. The current authors would, however, argue that this conveys exactly the wrong message if one wants individuals and workgroups to be adaptive and capable of creative, appropriate improvisation – skills that can be practised and learnt effectively in lower-fidelity simulations to complement the procedural skills gained from high-fidelity simulation. These simulations, by design, can lead participants to rethink their normal roles and behaviour, which, in turn, can help develop more adaptive and flexible competencies, strengthening operator and system resilience in the face of unanticipated and escalating situations. Acknowledgements We gratefully acknowledge our cooperation with Professor Stefan Strohschneider at the University of Jena in Germany, who, together with Jürgen Gerdes, originally developed the M/S Antwerpen mid-fidelity simulation and training programme. The research presented in this article was funded by the Swedish Emergency Management Agency and the Swedish Governmental Agency for Innovation Systems. References Amalberti, R., 2001. The paradoxes of almost totally safe transportation systems. Safety Science, 37 (2–3), 109–126. Baker, D., et al., 1993. Aviation computer games for crew resource management training. The International Journal of Aviation Psychology, 3 (2), 143–156. Barnett, A. and Wang, A., 2000. Passenger-mortality risk estimates provide perspectives about airline safety. Flight Safety Digest, 19 (4), 1–12. Bell, H.H. and Waag, W.L., 1998. Evaluating the effectiveness of flight simulators for training combat skills: A review. The International Journal of Aviation Psychology, 8 (3), 223–242. Bosk, C., 2003. Forgive and remember: managing medical failure. Chicago, IL: University of Chicago Press. Burian, B.K. and Barshi, I., 2003. Emergency and abnormal situations: A review of ASRS reports, Proceedings of the 12th international symposium on aviation psychology. Dayton, OH: Wright State University Press. Downloaded By: [Dahlstrom, Nicklas] At: 10:51 16 May 2009 Theoretical Issues in Ergonomics Science 313 Burian, B.K., Barshi, I., and Dismukes, R.K., 2005. NASA Technical Memorandum 213462, The challenges of aviation emergency and abnormal situations. Moffett Field, CA: NASA Ames Research Center. Bürki-Cohen, J., Soja, N.N., and Longridge, T., 1998. Simulator fidelity requirements: the case of platform motion. Proceedings of the 9th international training & education conference (ITEC), Lausanne, Switzerland, 216–231. Caird, J.K., 1996. Persistent issues in the application of virtual environment systems to training. Proceedings of HICS’96: Third annual symposium on human interaction with complex systems. Los Alamitos, CA: IEEE Computer Society Press, 124–132. Carretta, T.R. and Dunlap, R.D., 1998. Transfer of training effectiveness in flight simulation: 1986 to 1997. AFRL-AZ-TR-1998–0078. Mesa, AZ: Air Force Research Laboratory. Dahlström, N., 2006. Training of collaborative skills with mid-fidelity simulation, Proceedings of human factors and economic aspects on safety. Linköping, Sweden: The Institute of Technology at Linköping University. Dahlström, N., Dekker, S.W.A., and Nählinder, S., 2006. Introduction of technically advanced aircraft in ab-initio flight training. International Journal of Applied Aviation Studies, 6 (1), 131–144. Dahlström, N. and Nählinder, S., 2007. Mental workload in aircraft and simulator during basic civil aviation training. The International Journal of Aviation Psychology (accepted for publication). Dekker, S.W.A., 2001. Follow the procedure or survive. Human Factors and Aerospace Safety, 1 (4), 381–385. Dekker, S.W.A., 2005. Ten questions about human error: a new view of human factors and system safety. Mahwah, NJ: Lawrence Erlbaum Associates. Dennis, K.A. and Harris, D., 1998. Computer-based simulation as an adjunct to ab initio flight training. The International Journal of Aviation Psychology, 8 (3), 261–276. Dismukes, R.K., Berman, B.A., and Loukopoulos, L.D., 2007. The limits of expertise: rethinking pilot error and the causes of airline accidents. Aldershot, UK: Ashgate Publishing Co. Dörner, D., 1989. The logic of failure: recognizing and avoiding error in complex situations. Cambridge, MA: Perseus Books. Hays, R.T., et al., 1992. Requirements for future research in flight simulation training: Guidance based on a meta-analytic review. The International Journal of Aviation Psychology, 2 (2), 143–158. Heeter, C., 1992. Being there: The subjective experience of presence. Presence: teleoperators and Virtual Environments, 1 (2), 262–271. Hollnagel, E., Woods, D.D., and Leveson, N.G., 2006. Resilience engineering: Concepts and precepts. Aldershot, UK: Ashgate Publishing Co. Jackson, P., 1993. Applications of virtual reality in traning simulation. In: K.Warwick, J. Gray, and D. Roberts, eds. Virtual reality in engineering. London: The Institution of Electrical Engineers. Klein, G.A., 1998. Sources of power: how people make decisions. Cambridge, MA: MIT Press. Lanir, Z., 1986. Fundamental surprise. Eugene, OR: Decision Research. Lee, A.T., 2005. Flight simulation: virtual environments in aviation. Aldershot, UK: Ashgate Publishing Co. Longridge, T., et al., 2001. Simulator fidelity considerations for training and evaluation of today’s airline pilots, Proceedings of the 11th international symposium on aviation psychology. Columbus, OH: Ohio State University Press. National Transportation Safety Board, (1990). Accident report: United Airlines Flight 232, McDonnell Douglas DC-10–10, Sioux Gateway Airport, Sioux City, Iowa, July 19, 1989. NTSB/AAR-90/06. Washington, DC: NTSB. National Transportation Safety Board, (2007). Accident report: Crash of Pinnacle Airlines Flight 3701, Bombardier CL600–2-B19, N8396A, Jefferson City, Missouri, October 14, 2004. NTSB/ AAR-07/01. Washington, DC: NTSB. Pertraglia, J., 1998. Reality by design: the rhetoric and technology of authenticity in education. Mahwah NJ: Lawrence Erlbaum. Downloaded By: [Dahlstrom, Nicklas] At: 10:51 16 May 2009 314 N. Dahlstrom et al. Rochlin, G.I., 1999. Safe operation as a social construct. Ergonomics, 42 (11), 1549–1560. Rochlin, G.I., LaPorte, T.R., and Roberts, K.H., 1987. The self-designing high reliability organization: aircraft carrier flight operations at sea. Naval War College Review, 40 (4), 76–190. Roscoe, S.N., 1991. Simulator qualification: Just as phony as it can be. The International Journal of Aviation Psychology, 1 (4), 335–339. Rouse, W.B., 1981. Human–computer interaction in the control of dynamic systems. IEEE ACM Computing surveys, 13 (1), 71–99. Salas, E., Bowers, C.A., and Rhodenizer, L., 1998. It is not how much you have but how you use it: Toward a rational use of simulation to support aviation training. The International Journal of Aviation Psychology, 8 (3), 197–208. Strohschneider, S. and Gerdes, J., 2004. M/S ANTWERPEN: emergency management training for low risk environments. Simulation and Gaming, 35 (3), 394–413. Transportation Safety Board of Canada, 2003. Aviation investigation report: In-flight fire leading to collision with water, Swissair Transport Limited, McDonnell Douglas MD-11 HB-IWF, Peggy’s Cove, Nova Scotia 5 nm SW, 2 September 1998. Report number A98H0003. Gatineau, QC: Transportation Safety Board of Canada. van Winsen, R., et al., 2008. Rule- and role-retreat: an empirical study of procedures and resilience. Submitted to Cognition, Technology and Work. Weick, K.E., 1993. The collapse of sensemaking in organizations: the Mann Gulch disaster. Administrative Science Quarterly, 38 (4), 628–652. Weitz, J. and Adler, S., 1973. The optimal use of simulation. Journal of Applied Psychology, 58 (2), 219–224. About the authors Nicklas Dahlstrom is a Human Factors researcher and Crew Resource Management instructor at Lund University School of Aviation, Sweden. He has been an officer in the Swedish Air Force, has a BSc in Meteorology from Stockholm University and a PhD in Technology from Lund University. His main research areas are mental workload and he has worked primarily with aviation but also in maritime and railway transportation, nuclear and chemical industry, medicine and rescue services. Sidney Dekker is Professor of Human Factors and Flight Safety, and Director of Research at Lund University, School of Aviation, Sweden. With two Dutch Master’s Degrees in Psychology, he gained his Ph.D. in Cognitive Systems Engineering from Ohio State University, USA. He is currently scientific advisor on healthcare system safety to the Winnipeg Regional Health Authority and Professor of Community Health Science at the Faculty of Medicine, University of Manitoba, Canada. He is also a commercial pilot. Roel van Winsen is a Human Factors researcher at Lund University School of Aviation, Sweden, and is currently enrolled in a programme to become an Air Traffic Controller at Air Traffic Control The Netherlands. He received a Master’s Degree in Cognitive Psychology from the University of Leiden, The Netherlands, and whilst working with Professor Sidney Dekker he received another Master’s Degree in Human Factors and System Safety from Lund University. James M. Nyce, a cultural anthropologist at Ball State University, studies how information technologies emerge and are used in workplaces and organisations. A docent at Linköping University, Sweden, he is an adjunct associate professor in the Department of Health and Environment, Faculty of Health Sciences there. He is also an adjunct associate professor in the Department of Radiology, Indiana University School of Medicine, Indianapolis. Nyce has been visiting professor in military technology and war science at the National Defence College, Stockholm, 1998–2000 and 2005 to present. He is also a visiting professor in Lund University’s Master’s Programme in Human Factors and System Safety. View publication stats