Keywords

1 Introduction

Within the nuclear industry, a significant amount of effort has been dedicated to the study of human reliability, with important implications for training, human-machine interfaces, procedural support, and safety culture [1, 2]. Those studies have primarily focused on operations in the main control room, where human actions are considered to be particularly critical for the safety of the plant.

The human factors and ergonomics (HFES) community has questioned whether traditional Human Reliability Analysis (HRA) methods derived from operation of analogue control rooms can successfully accommodate the modified and new types of tasks and actions that characterize digital and/or hybrid control rooms. Human reliability has traditionally focused on obtaining empirical data from complex, full scope studies where realistic simulations of control room operations are conducted. As a consequence of the growing digitalization in control rooms, it is possible to observe a renewed claim for empirical data on operator actions and especially the likelihood of errors in the new settings [3]. This has led to a growing body of data regarding control room actions, generated in simulator experiments, where the difficulty of the tasks is experimentally manipulated to increase the chances of observing human performance errors.

Simultaneously, operations outside the control room have come to attract more attention, as a more integrated view on safety surfaced in the HFES literature [4]. The contextual aspects of work in the field such as physical demands or the impact of fatigue or noise levels, have been vastly described in the classical ergonomics literature [5]. On the other hand, the cognitive aspects of operator performance outside the control room and cooperation with the main control room have deserved a lot less consideration. With this work we intend to explore these aspects and attempt to both describe and empirically explore the performance shaping factors more relevant for field operations. These are factors that can affect human performance and can be behavioral and/or contextual [6].

A central aim of our ongoing work on operator reliability for actions outside the control room, consists on the mapping of the most relevant performance shaping factors for field work. The literature on this topic is very scarce and there are very few instances where empirical data on field operator actions was collected [7], making it difficult to understand and estimate the impact of actions outside the control room for overall plant safety.

One of the core challenges regarding empirical data collections outside the control room relates to the available methods [8]. Quite often the methods used inside the control room are not directly applicable to actions outside the control room. The nature of the tasks inside and outside the control room can be clearly distinguished: the space where field operators work is a lot more vast than the control room, allowing free movement of the operators and limiting observation possibilities; environmental conditions such as noise, temperature, humidity or radiation are more likely to vary, affecting tasks planning and execution; the physical demands on the field operators are significantly different which can impact their performance; and planning of the tasks outside the control room might be of particular relevance, since the field operators often walk long distances in the plant and need to make sure they have the correct procedures and tools before they leave the job briefings and head to the rounds. As such, collecting data inside and outside the control room presents significantly different challenges that require different methods and strategies.

1.1 Framework

The framework that supports the work on this paper can be found in [9]. This framework proposes a classification and methodology to collect empirical data for actions outside the control room and covers: 1) task identification and characterization; 2) performance shaping factors; 3) scenario characteristics; 4) data collection environments; and 5) methods for empirical data collection. Table 1 summarizes the components of this framework’s first draft. We intend to elaborate and improve the framework based on iterations resulting from empirical data collections and its findings.

Table 1. Components of the framework supporting this work
Full size table

Outside the control room, there is a considerable diversification of tasks which often reflects the specificity of the plants. This can lead to very sparse data, that cannot be generalized. To overcome these aspects we suggest grouping tasks, by classifying them based on their complexity and cognitive demand rather than single steps or other plant specific details. Contextual circumstances and time pressure are also taken into consideration.

The second item in the framework – performance shaping factors – is the focus of the current work. Some performance shaping factors cannot be easily controlled in studies outside the control room. For instance, training and experience, or existing procedures and administrative control. Those are as such discarded in our work (attending to the final aim of manipulating variables in data collections). Likewise, contextual factors and fitness to work are also out of scope since their implications are well described in the literature. We will then focus on ergonomics and human machine interfaces, complexity, time pressure, and the work processes surrounding the task execution.

The scenarios refer to the context provided to the operators. They involve a description of the status of the plant and/or of the task the operators are asked to fulfil. The scenarios can cover when a task is executed in the expected conditions without additional disturbance, or it can involve situations where unexpected events occur or where significant incidents/accidents are presented. In the presented field study normal operative conditions has been considered to verify the hypothesized relevance of the selected performance factors at least in this condition.

Regarding data collection environments we present three alternatives: i) real-world context, where the data can be collected directly at the plant; ii) training centers, where simulations of events and scenarios can be used with the operators performing actions on mock-ups or decommissioned components; and iii) virtual reality simulations, where we are able to present all levels of scenarios, but where realism and generalizability are compromised. In this study we used the real-world approach.

The final item refers to methods to collect data. After screening existing methods within the nuclear context, we have identified as more promising: interviews, debriefings, task analysis, questionnaires, biometric data, and eye tracking. Other methods that can support the data collection are observation, audio/video recording, walkthrough, talk-through, response accuracy, response time, role playing, report analysis.

1.2 Study Objectives

This is an initial exploratory study in a real-world setting, where a routine task is being performed. The focus of the study will be on the identification of performance shaping factors. The objectives for this study are:

  1. 1.

    Assess whether communication, situation awareness, and background knowledge are relevant factors shaping field operations, this will be measured through the feedback of the operators through interviews and questionnaires

  2. 2.

    Evaluate the efficacy of using talk-through, walkthrough, and task analysis as tools to prepare data collection in the field, measured through qualitative assessment by the research team, considering the obtained information before the data collection and its usefulness to understand the work processes

  3. 3.

    Explore the applicability of direct observation as a data collection method in real world settings, measured by the research team, assessing the value of observations compared with other possible methods (process expert assessment, biometric data, etc.)

1.3 Halden Research Nuclear Reactor Operational Characteristics

The Halden Boiling Water Reactor (HBWR) is a small (25 MW) boiling water reactor used for research on fuel properties and material corrosion. The reactor was initiated in 1958 and was operated by the Institute for Energy Technology until 2018. The reactor is currently on permanent shutdown, and daily operations are reduced, consisting of maintaining reactor stability and safety.

The control room is operated by a crew of two operators in three shifts (morning, evening, and night). In the control room, besides the operators there is a reactor engineer that supports operations and is involved in task planning. In the control room the operators have workstations with digital interfaces, as well as analogue wall panels, and a large screen display to support process overview.

Unlike commercial nuclear power plants where the roles of control room and field operators are separate, at the Halden Reactor the same operators perform actions and monitoring both inside and outside the control room. This implies that all tasks are planned and executed by the same two operators, one being at the control room at all times and the other acting as field operator as necessary, leaving the main control room when required. Contact with the control room when the operator is on the field is maintained through telephone (most common) or noise cancelling headphones with transmission to the control room.

2 Method

The study was conducted together with another experiment where the effects of large screen displays on operator performance were being tested. As such, for half the crews, the large screen display was not available at the control room for the tested scenario. This was not deemed relevant for the purposes of the study, since we were mostly interested on the observations performed outside the control room. Nonetheless, in the debrief interviews, some of the control room operators mentioned that the large screen display was missed either during planning or execution stages for process overview inside the control room.

2.1 Task Classification

To prepare the data collection, we contacted a process expert with several years of experience at the Halden Reactor. The preparation was done in three consecutive phases over a period of two weeks: 1) initial interview: the process expert was initially interviewed by the research team to gather information about work processes and operational characteristics at the plant; 2) walkthrough: the process expert guided the research team through a walkthrough at the plant, focusing on the target scenario tasks; 3) talk-through: was conducted after the visit with the support of Piping & Instrumentation Diagrams (P&IDs) and procedures to clarify aspects of task performance that came up to the research team while performing task analysis.

A result of the walkthrough with the process expert was a task analysis of the targeted scenario which identified the main characteristics of the actions outside the control room and allowed us to better understand the nature of the task and explore possible paths for solving it.

2.2 Participants

Six crews of two operators, took part in the study, with six operators acting as field operators and six as control room operators. The participants were all male, Norwegian, had an average age of 47.6 years (SD = 7.9) and had on average 13.9 years of experience has nuclear operators (SD = 8.1; Median = 14.5).

2.3 Tools

Observation Checklist.

The observation checklist was used by two researchers – one at the control room and one shadowing the field operator outside the control room. The observation checklist was constructed based on the task analysis that preceded the data collection and was discussed with a process expert. It included four main moments:

  1. 1.

    Preparation activities: such as retrieving documentation, discussing, and checking pre-conditions for the test;

  2. 2.

    Going to target components: included getting necessary equipment (protection clothing, hat, telephone, documentation), leaving control room and reaching the target components for the test;

  3. 3.

    Performing test: preparing local conditions, executing accorded plan and perform test, restore local conditions;

  4. 4.

    Return and wrap-up: getting back to main control room and closing task.

Operator Assessment.

At the end of the scenario the operators were asked to individually respond to a questionnaire evaluating the relevance of specific factors in the task they had just performed. The questionnaire included 14 items to be rated in a Likert scale from 1 (completely disagree) to 5 (completely agree) and a final section for open comments. The items included the following factors: importance of communication with control room; there as a lot to do; the noise level on the telephone was high; there was a time pressure; external factors such as noise, temperature, humidity and vibrations are relevant; you need to be an experienced operator; you are dependent on existing procedures; you had a predefined plan in the current task; there was a lot to think about; it was a complicated task; you need to have a lot of knowledge; it was a difficult task to perform; communication between the inside and outside control room was appropriate; planning was important; overall it was a difficult task.

Interviews.

The operators were interviewed jointly at the end of the scenario. The interview was initiated by a comparison of the answers the two operators provided to the questionnaire, and then exploring any comments they added to it. A semi-structured interview was used that focused on the relevance of performance shaping factors (as formulated in the questionnaire), the specificities of task execution, and feedback on the study experience.

2.4 Scenario

A process expert was responsible for designing the test scenarios and providing the instructions to the crews. The task used for the data collection consisted on the test of a pump, running it for 5 min. The pump is part of the tertiary cooling system at the reactor. The operators needed to be aware of the need to close a parallel circuit used for chemical samples before the test is initiated. At the end the operators need to restore the initial conditions. The scenario did not foresee additional disturbances and was presented as a work order in standard daily activity.

2.5 Procedure

Before the data collection started the operators were briefed on the focus and goals of the data collection, as well as the plan for the data collection. They were then presented with an informed consent form. After this, the operators and researchers went into the control room were the task would take place.

The operators were briefed on the core task to be performed in a normal day scenario and then could initiate the work. Two researchers were at the back of the main control room observing the planning stage. When the field operator left the room, one researcher would shadow him, while another stayed with the control room operator.

The researchers used pen and paper during the data collection and recorded notes in a pre-defined observation checklist, taking note of time stamps, communication characteristics and any other information on the duration of the task.

At the end of the scenario the operators were invited to a room adjacent to the main control room, where they filled in the questionnaire individually and then were interviewed together.

3 Results and Discussion

3.1 Planning Phase

In this study we were able to have access to a process expert who could guide us through the initial preparations for the data collection. We performed interviews and a tour to the power plant facilities. The process expert was questioned about possible alternatives to perform the tasks, potential errors, possible implications, required background knowledge on the actions, and requirements on communication.

Even though we were analyzing a relatively simple and short task, that is close to routine tasks at the plant, the preparation required several resources in order to be able to map all the steps and understand the nature of the actions outside the control room. This emphasizes the challenges regarding data collections in the field, in a space that is less familiar to researchers and analysts.

3.2 Observations

The scenario had a duration between 16 and 37 min across the six crews (Median = 29 min). We organized the findings according to the different stages in the scenario: planning, moving to the field, execution, and debriefing.

Planning.

All the crews planned the task together. To understand the current status of the plant all crews used the workstations, in most cases also the analog panel. When the Large Screen Display was available it was used significantly only by one crew.

In all cases the P&IDs were the main support on which to reason cooperatively to formulate a plan. Four crews retrieved also a procedure describing how to operate the pump object of the test.

The communication style varied, in two cases, the crew members were sitting close to each other from the beginning of the planning, other crews started with separately checking the status of the loops at different workstations, with limited verbal communication. The details of the plan had some variations, but all operators agreed to operate the main tasks (open/closing pumps) under continuous communication. The time spent on planning varied from three to 21 min.

Moving into the Field.

In five out of six cases, the field operator brought the P&ID to the field, only one case he picked the procedure for operating the pump. In all cases they chose the mobile phone as communication device. This choice was justified by the small size of the task and the fact that the noise level in the hall was low, especially compared to when the reactor was active and the cooling loops functioning at regime. The distance between the control room and the operating area was very short (200 m or a 1-min walk).

During a scenario an extra loop, the chemical analysis circuit (CAC) was supposed to be isolated, in order to avoid spurious results influenced by the increased pressure generated by the tested pump. No crews considered this step in their plans, but all recognized immediately the need as reminded by the process expert and/or the reactor engineer. Most crews isolated the CAC from the control room, two crews let the operation to the field operator, in one case the field operator checked the status of CAC even if the control room was responsible to operate it. It is important to mention that the operating crew is usually not responsible for task planning and as such this can be seen as an extra requirement on them for the current study. The engineering team would normally prepare and plan the tasks that would be delivered to the operators for execution.

Execution.

All crews communicated to the control room while operating the pumps. Some operators relied mainly on sensorial input like noise, vibrations rather than on local instrumentation for monitoring the effects of the actions, others used also the displayed values of pressure gauges. Only in two cases the field operator had to operate in addition valves in adjacent circuits, on request from the control room. During the five minutes wait, while PB2 was operating, two field operators went back to the control room, one performed a short inspection of other areas, and the others waited by the pumps. In average, the complete operation took 16 min, with small variance among the crews.

Debriefing.

The scenario did not explicitly require a debrief after the field operator returned to the control rooms, but three crews did it, checking if the situation was correctly restored in the control room interfaces and with the field operator shortly reporting technical details.

3.3 Questionnaire

The operators were very congruent in the ratings they attributed for each item in the questionnaire. A central difference relates to the rating regarding the noise levels during the phone calls, which showed a tendentially higher rating by the field operation in comparison with the control room operation. A similar trend was visible for the item “there was a lot to do” that was often rated higher by the field operators.

Three factors were consistently signaled by all participants as very relevant in field work: communication with control room, its clarity, and planning of the work before leaving to the field (Fig. 1).

Fig. 1.
figure 1

Ratings of agreement on importance of performance shaping factors

Full size image

The task was not seen as difficult or complicated, and the operators did not think they were under time pressure to complete it.

A common comment from the operators referred to the fact that noise canceling headphones with communication capabilities were available, but still not taken into the field.

3.4 Debriefing Interview

The interviews with the crew focused on the actions performed outside the control room and were initiated with an analysis of the responses to the questionnaire items.

The fact that there was not a specific procedure for this scenario contributed to increase the difficulty level of a task that was considered otherwise common. The lack of procedure required more planning and reasoning before performing the necessary steps to complete the pump test. Another factor highlighted by some of the operators relates to the differences in operation now that the reactor is in permanent shutdown, mentioning that some tasks are now more difficult because it is harder to “know what reference values are expected for the current reactor status”, however, other operators mentioned that they felt “less stress” performing this task since the reactor is in shutdown.

In the interview the operators stressed the importance of communication in the current task. They mentioned that maintaining a phone conversion where they share the ongoing actions inside and outside the control room helped them to maintain good collaboration and a shared understanding of the task. The noise levels outside the control room were often noted by the operators, but it did not seem to disturb the conversation. On the other hand, holding the phone and operating some of the valves in the field was not compatible and required the operators to put the phone down while manipulating valves with two hands – and alternative is available with the use of noise cancelling headphones with a transmitter, but none of the crews took them to the field.

Another recurrent topic in the interview was the need for planning in the current task. Being able to set up a plan for task execution was crucial in this scenario since there was not a specific procedure that could be implemented. For planning, aspects such as background knowledge and experience were mentioned as relevant by the operators. Likewise, planning using the P&IDs drawing and/or partially applicable procedures contributed to the creation of a shared understanding of the task goals and steps that was reckoned important by the operators.

3.5 Methodological Insights

Observing operation in the field requires a good understanding of the task and knowledge of the components, even if superficial. An important aspect missing from the current data collection is process expert expertise assessing the crew performance in the field. This could be achieved through post-hoc analysis, collecting video and/or audio recordings, but ideally could be done online, with an expert on site.

Observation has anyway an impact on task performance, not being the observer behind the scene, as it could be in a simulator set up, but rather close in normally small area with restricted possibility of movement. The intrusiveness of the observer has been commented in the interviews.

Regardless of the alternative data collection methods, interviews and debriefs are crucial qualitative data necessary to understand the observations and interpret other types of data collected (e.g. errors performing the tasks, response time, biometric data).

4 Conclusions and Further Work

4.1 Findings and Contributions

We were able to successfully conduct a data collection in a real world setting and obtain relevant input and information towards the verification of the anticipated framework. We mapped typical ways of working, main tasks characteristics, developed a study design, selected tools, and collected data based on the theoretical hypothesis and classifications proposed beforehand.

The opportunity to prepare the data collection with support from a process expert was valuable in the current setting. The quality of the observations and the understanding of the events during the scenario would not have been possible without some degree of guidance from a process expert.

The pre-identified performance shaping factors communication, situation awareness, and background knowledge were verified as relevant in the current study. Moreover, the operators highlighted the relevance of trust as well due to the need of close collaboration. In the current study context, awareness and understanding of the task and plant status was achieved through joint planning of actions and frequent communication during task execution. It was facilitated by the fact that both operators need to work closely together and have the same background (both are control room and field operators at this plant). Considering this, we believe that situation awareness can be a key relevant performance shaping factor affecting operator’s performance in commercial plants, where the field operators is often less involved in the planning stages and has a different background from the control room operator.

In the current study we performed close direct observations while the operators performed the tasks. However, the application of this technique in real world data collections will be limited due to safety/security reasons at the plants. It can be useful in simulations at training centers, especially if the observer is a trained operator who can recognize components and better understand the actions being performed. Although useful for the research purposes, this type of observation was considered intrusive and might generate additional stress to the operators, especially if they are not used to being observed (for instance in training context).

The questionnaires were a quick way to register the operators’ individual assessment of the task and served well their purpose as a discussion topic in the debrief interview. The interviews were a central aspect of the data collection and are essential to obtain an overview of the tasks as well as understand how and why the crew implemented the work.

4.2 Limitations

We would like to present three core limitations with the current work that we plan to mitigate in future data collections:

  • Generalization of findings

The setting of the HBWR is quite different from the commercial power plant settings. The reactor is currently in shutdown mode, is small, and was used exclusively for research purposes. It has a different organization, team composition and conduct of operations when compared to commercial power plants. As we seen before, only two operators composed the shift and both have an equivalent role and can perform control actions inside and outside the control room. This implies that they always plan and discuss all tasks together and are both aware of the specificities of both types of work – this is not necessarily the case in commercial power plants where the roles of field operator and control room operator are well demarked.

  • Selected task

In the current work we were unable to control variables in the study and restricted the data collection to observations and descriptions of the events as performed in a routine task. This was useful as an initial technique to explore performance outside the control room. However, we expect that incident/accident scenarios will be the most relevant since they present extreme situations where the performance shaping factors can become more visible/observable. Also, controlling variables within the data collections will enable us to compare performance in situations where specific parameters are stressed. This will not be possible, of course, in real world data collections, and as such simulations (e.g. in training centers), become more attractive as data collection environments.

  • Assessing performance – process expertise

Regarding the methods and tools, we consider that the process expertise when evaluating performance (online or after the scenarios) will be an important component in the understanding and accurate evaluation of the operators’ performance, especially when we focus on aspects such as situation awareness and background knowledge. In the current study, the human factors researcher were the only ones assessing performance based on the checklist developed with the support of the process expert. In a simple task such as the one presented here this is doable, but in more complex scenarios, we expect that process knowledge will be decisive in understanding the crews’ actions and decisions online.

4.3 Further Work

We are planning further studies for verification of the presented framework. These will take place in training centers and/or real-world contexts (focusing on commercial power plants). We will further explore the role and relevance of different performance shaping factors for different scenarios (e.g. in routine versus incident situations) and emphasize the study of other aspects of the framework, namely the methods for data collection. In the near future we also aim at testing the feasibility and usefulness of new tools for data collection in training centers, involving quick responses to simplified tasks and use of eye-tracking in realistic contexts.