Keywords

1 Introduction

Training is an essential component for the successful deployment of any complex socio-technical system, especially those that may involve various types of interactions, specifically between human-to-human, human-to-machine, machine-to-human, and machine-to-machine as well as the interaction among the multiple engaged teams. Emergency management and response is one such type of complex environment, requiring first responders to efficiently and effectively carry out an incident response with multiple teams (e.g. fire and rescue teams, EMS teams, dispatch, etc.) to conduct life-saving operations while leveraging a suite of technologies to assist in their missions. Simulation training provides a key component and process for emergency management and response personnel to help address the inherent uncertainty of a simulated incident toward more efficient task coordination and effective planning and communication (Bannan et al. 2019).

Despite the obvious importance of training, the simulation training process may not always be optimal if the training tools are inefficient to assist the training simulation instructors and the trainees (Buck et al. 2006; Feese et al. 2013). In current practice, the training simulation instructors often primarily rely on direct observation and radio-based audio communication for documenting trainee behaviors and interactions. Such direct observation-based methods can sometimes lose the valuation of information in providing an effective learning feedback to an individual trainee and team during the debrief due to a variety of data-centric limitations (Dubrow and Bannan 2019). For instance, the incompleteness of the observed data, inefficient extraction of behavioral knowledge from the observed data for learning that could be interpretable to instructors, etc.

Therefore, in this paper, we ask the following research questions to help improve the training in the complex environment of emergency management:

  • (RQ1). How is a shared understanding of situation established across personnel within and between teams during an incident response or a training exercise?

  • (RQ2). What challenges do current practices of human observation and audio-based information collection and processing present when attempting to measure and interpret shared situational awareness and team coordination?

  • (RQ3). How can we address the identified challenges from the current practices of information processing using the state-of-the-art technologies?

This research takes a user-centered design approach to address the above questions. In particular, for research questions RQ1 we conducted several iterations of focus groups with our collaborating emergency management trainers at a regional fire and rescue training academy. For answering research questions RQ2 and RQ3 we analyzed the focus group observations in light of the existing literature on data analytics and learning technologies. We found the recent advancement in the multimodal data streams as a unique, unprecedented opportunity to address our challenges in the conceptualization of a human-computer interface for emergency management training.

Recent advancements in video, wearable, and social & web stream technologies present unconventional opportunities to capture and improve multiple streams of information that may be accessible to instructors during simulations and enhance debriefing sessions (Dubrow et al. 2017; Feese et al. 2013; Kranzfelder et al. 2011). The debriefing session concluding a live simulation training event is where prior research indicates the most learning occurs as the first responders view and reflect on their behavioral activities post-event. The capability to mine behaviors from the integration and combination of such non-traditional data streams present a promising method to enrich the information collected, processed, and analyzed in-situ (Bannan et al. 2019). These new combinations of information streams may assist incident commanders and instructors in facilitating relevant pieces of information to provide enhanced situation awareness through replaying targeted episodes that are triangulated and visualized through redundant data sources (e.g. video segments and proximity sensors both recording the proximity of personnel to the patient as the event unfolds). Multiple views of varied streaming information about the specific events during the training exercises provide complementary situational awareness. The improved, integrated data streams optimally visualized to incident command during the live simulation and post-event during the simulation debriefing session could possibly lead to improved awareness for reflection and experiential learning for fire and rescue personnel, as they engage in this complex socio-technical system environment.

We propose a multimodal system design, which provides real-time collection, visualization, and analysis of heterogeneous streams including location sensors, audio communication, video observations, and social media and 911 call streams by building on the redundancy and avoiding split attention effects in the learning science literature.

In the remainder of this paper, we first provide a background on the emergency response and management training with consideration of designing a data streaming and learning analytics system for the first responders audience, followed by presenting the conceptual system design as well as the results from iterative cycles of prototyping in a user-centered design process toward that goal.

2 Background: Emergency Response Training and Stream Data

This section describes the context of our research through the two following related areas.

2.1 Emergency Management and Response Training

There are several types of training exercises utilized in the emergency management domain, ranging from tabletop discussions to high-fidelity in-situ simulations centered around incident management (Buck et al. 2006). There are disparate training forms and each serves unique purposes for team learning, situation awareness, coordination, and improved performance.

Tabletop exercises (TTX) are often used by leaders in emergency management systems when planning future response techniques, such as future assessments of emergency-based damage and potential community recovery (Volunteer Fairfax). Researchers and trainers may want to collect data streams such as audio and video to record discussions during such meetings. Additionally, sociometric and dyadic measures of interactions between leaders may be examined to assess coordination and effort in TTX.

Drill exercises occur when training for a single function, such as applying a tourniquet to a patient. In such exercises, trainers may care more about specific body placement, speed, and proximity to the victim to assess performance (California Hospital Association). Functional exercises are less focused on planning and specific skills, and more focused on the interactions between individuals, teams, and agencies when responding to an event. Functional exercises allow for training and assessment of responders’ skills without the costs and risks of in-situ simulations. Thus, data collection in functional exercises should likely be more focused on video, audio, and sociometric sources than on physical sources (e.g., GPS, proximity).

Finally, full-scale in-situ exercises are high-fidelity and as close to real life events as possible (California Hospital Association). Full-scale exercises are the most realistic for first responders, and are better able to create the stress, sense of urgency, and obstacles that come with real life events. Researchers and trainers will want the most information from full-scale exercises, but full-scale exercises are the most challenging to collect data from, due to environmental factors such as noise, vision, and heavy equipment. Thus, data redundancy is especially important in full-scale exercises. Physical, sociometric, audio, and video data are all critical for after-action reviews following full-scale exercises.

Currently, most of the existing approaches to the learning analytics during these variety of training exercises do not exploit the multimodal data streams.

2.2 Streaming Analytics Systems

Researchers and practitioners have designed streaming data analytics systems in a variety of domains. Safety-critical systems in industries have long employed the variety of sensing methods, video surveillance, and audio communication technologies using cyber-physical systems (Rajkumar et al. 2010) for operational and training purposes, which require streaming analytics including signal processing. Likewise, military domain use such technologies, primarily using the simulation based modeling systems that analyze the video data streams for operational training (Buller et al. 2010). Similarly, aviation domain extensively use audio communication and sensor data streams for analyzing team behavior in the training processes. Modern information systems for surveillance and intelligence have extensively mined a variety of open data streams including multiple modalities (Glassman and Kang 2012). These applications of streaming data analytics across different domains suggest the important of studying the diverse data streams, rather than relying on single data stream for any analytical tasks.

However, there is a lack of investigation for designing such streaming analytics system with multimodal data streams in the context of emergency management and response, which provides a strong motivation to our research.

3 System Design: User-Centered Approach

In the current study, we utilize a user-centered design approach (Wallach and Scholz 2012) to ideate, refine, and execute a multimodal analytics software system for supporting the operational training of emergency services. We designed a system to meet the trainer needs, summarized in Fig. 1, that captures multimodal data streams and processes them to identify common behavioral events (e.g., proximity of two personnel from different teams for a common task).

Fig. 1.
figure 1

Overview of the multimodal streaming analytics system for supporting training of emergency personnel.

Full size image

The system design aims to (a) save time, by allowing trainers and incident commanders to browse content from multiple streams on the same visual dashboard, (b) reduce effort, by seamlessly aligning common information streams and eliminating the need for trainers and incident commanders to check several different dashboards, and (c) facilitate scalable monitoring of trainee learning by allowing more instructor roles to access the same dashboard views. In addition, the system design will also ultimately increase situation awareness to facilitate improved team coordination and experiential learning from the exercise.

3.1 Design Challenges

Our discussions with fire and rescue trainers have highlighted that the lack of access to redundant information is a key limitation in current trainee learning practices that leads to missing information and limited situational awareness, which impact learning and team coordination during highly dynamic training exercise environments. To enhance awareness, learning, and team coordination by leveraging streaming analytics, we must align the demands of context and behavioral activity with the identified research constructs and the information processing requirements and challenges in the emergency response situation.

Situation awareness as a complex construct has been defined in multiple ways (Stanton et al. 2017). Fundamentally, individual situation awareness is described as “knowing what is going on around you or having the big picture” (Jones 2015, p. 98). However, other definitions include information from humans as well as devices in describing situation awareness as:

activated knowledge for a specific task within a system….[and] the use of appropriate knowledge (held by individuals, captured by devices, etc.) which relates to the state of the environment and the changes as the situation develops (Stanton et al. 2006 , p. 1291).

This expanded view of situational awareness is extendable to the wider emergency management and response system to include multiple factors and collective agents (e.g. both human and non-human entities) who “…also have to track and/or represent the situation in complex (and not so complex) ways in order to function as they should, or at the very least, help human agents to achieve acceptable levels of performance (Stanton 2017, p. 452).” We have adopted this definition of situation awareness to include multiple factors represented by multiple data streams to inform the incident commanders and emergency response personnel about their own behaviors and performance during the simulation through both human and non-human agent interaction. Our vision is to ultimately represent various facets of behavior in-situ while the first responders are engaged in the simulation for the benefit of incident command decision making during the event as well as concluding the simulation for personnel reflection during the debriefing session. This would create a common operating picture formed through the captured human-to-human interaction as well as human-to-machine (or device) and then potentially mined for meaningful information through behavior analytics for the observed information by machine (or device) or human. We envision these devices or sensors capturing the dynamic response by multiple teams (e.g., fire and rescue, EMS) and visualizing their different patterns of coordination (e.g. who is near who at what point in time) through a machine learning approach for behavior mining of relevant information in near real-time. This would hopefully, provide the incident commanders, their personnel and observers of the simulation, a common operating picture to form a shared mental model of the event and increase their experiential learning from that event.

However, we identify three key challenges in the training process centered around information processing systems of sensors and other data streams are the following:

  1. 1.

    Incompleteness: missing observational data streams.

  2. 2.

    Miscommunication: noisy or lossy radio communication channel.

  3. 3.

    Inconsistency: across the observational and continuous audio data streams.

During the focus group discussion with our collaborators at the firefighting training academy, we validated the need for solving the above challenges in the data streams across modalities, and the need to minimize uncertainty of information as well as improve the perceived validity of the information using multimodal streaming analytics approach.

3.2 Theory of Redundancy and Split Attention Effect

Our system design is based on the ideas of presenting redundant information through multimodal data streams and purposefully integrating different data streams to avoid the split attention effect to potentially improve first responder awareness, learning and team coordination. The split attention effect occurs when learners are required to divide their awareness between multiple sources of information that have been separated either spatially or temporally and therefore, need to be integrated to avoid cognitive overload. Studies have shown that integrated information is more effective for learning especially if the individual components of information are found to be less intelligible in isolation (Ayers and Cierniak 2012). However, when strategically combined to avoid split attention, an integration of data streams may move toward more intuitive processing of information (Chen et al. 2017). If the same information is physically or spatially (digitally) integrated, this may obviate the need to mentally integrate it, then, potentially reduce the number of interacting elements to minimize extraneous cognitive load.

The split attention effect has been well-documented in multimedia learning. For example, Kalyuga et al. (1999) tested the effect on computer-based instructional materials consisting of diagrams and texts. They found that physically integrated information enhanced learning compared with information split across multiple sources. Al-Shehri and Gitsaki (2010) compared a split attention design with an integrated design for learners’ online reading performance. Results supported the view that online integrated materials enhanced students’ learning. Similarly, Liu et al. (2012) investigated the split attention effect on mobile learning in the domain of physics. Therefore, we anticipate similar behavior when using a multimodal analytics system design for assisting trainers in complex training environments.

According to Ayers and Cierniak (2012) “…the split-attention principle says that several separated sources of information should be replaced with a single integrated source of information.” In our work, we envision multiple modes and streams of data and information aligned in a single, meaningful display for the first responder audience that moves toward optimal information processing in a complex dashboard for simulation training. The incident commanders as well as the fire and rescue trainers must hold a significant amount of information in their working memory during simulation exercise or an actual emergency response incident typically obtained through radio communication. The tendency for information overload is palpable with firefighters during an emergency response with stated concerns related to avoiding cognitive load or adding to their workload with additional, unnecessary information and radio traffic. Therefore, attempting to reduce the number of information streams in temporal and spatial contiguity is the goal and continuing identified challenge for our system.

Prior research has indicated the first responders desire an overall picture or mental model of the incident at hand typically processed through input gained from various information sources. For example, in a fireground incident, gaining information about the structure of the building, number of occupants, any hazardous materials on-site, etc. would be important information to know immediately upon initial assessment (Xiaodong et al. 2004). Important factors from the Xiaodong et al., study revealed the following design issues that emerged from the field:

  1. 1.

    Accountability of resources and personnel is crucial and should be as simple and accurate as possible.

  2. 2.

    Assessment of the situation through multiple sources of information while avoiding information overload is key.

  3. 3.

    Resource allocation is a primary task for incident commanders and should be a primary focus in designs.

  4. 4.

    Communication support should add reliability and/or redundancy to existing communication channels to ensure that important messages reach the right people (p. 683).

Our study attempts to intersect these core findings with learning science to advance the design of an ubiquitous computing system interface, which adheres to appropriate instructional design principles as well as prior research findings. For example in the same study, tracking individual firefighters in a particular building was perceived as useful, however, incident commanders preferred less granular information and desired broad-levels of information initially such as warnings of imminent dangers (e.g. low levels of oxygen) for their teams. In visualizing their respective team tasks, only presenting information that was necessary or explicitly queried or providing redundancy of important information to validate incoming contextual data was perceived as useful by the firefighters (Xiaodong et al. 2004).

The multimodal learning analytics system is designed to visualize information about the simulation event in-situ but also enhance the debriefing session to improve experiential learning from the live simulation. Learning through reflection on action in a simulation experience provides a rich source of objective data that can contribute to effective feedback from the situation. Jenvald and Morin (2004) found that discussion alone in the debrief did not appear to facilitate learning as well as the opportunity to visualize targeted replay information to promote rich reflection in providing relevant details in a calmer environment immediately following the simulation event. Our goal is to provide relevant streams of data to incident commanders and first responder teams that are visualized in the debriefing session with appropriate redundancy and attention to relevant integrated streams of information for this type of enhanced feedback.

3.3 Design Constructs

The specific constructs in our multimodal analytics system are the following:

Data Collection.

The rising adoption of mobile technology, Internet of Things (IoT), and social & Web data sources provide emergency management and response organizations a novel opportunity to collect information about incidents from diverse channels, and reduce uncertainty in information for better situational awareness. Social media has enabled citizens to act as “human sensors”, who observe, share and update situational awareness information for an unfolding emergency event. Ubiquitous computing and IoT sensors have opened up opportunities to collect field observations from the deployed sensors on resources and personnel in the field, complementing conventional channels of audio communication.

Streaming Information and Behavior Mining.

Given, a collection of streaming media sources (e.g., IoT sensor stream, location sensors, wearable biometric sensors, social network streams, news rss feeds, 911 call records), the technical objective is to develop time-series data mining and machine learning approaches with human-machine collaboration to identify the occurrence of a behavioral event of interest that has potential implication for emergency management operations (Pandey and Purohit 2018). We also seek to identify the granular information i.e., specific information from the multitude of streaming outlets that can be considered as supporting evidence for integration within the common operating picture for decision making, such as for personnel management and resource coordination analysis.

Team Coordination Analytics.

Our proposed approach of incorporating redundant and complementary information sources, and their unified organization, provides a form of intelligence through selected “smart” capture, integration of data sources with computational intelligence, and timely visualization of these varied information sources for emergency response personnel to consider in extending their situation awareness, decision-making and learning (Dubrow and Bannan 2019). This system design specifically targets team activity by displaying complementary sources of information to improve confidence in, and efficiently support situational awareness analysis for decision making, explicitly addressing the who-what-where of coordination and experiential learning at important inflection points between teams.

Visualization Dashboards.

Modeling and visualizing information flow across multiple constituent groups in an emergency scenario to potentially improve situational awareness, coordination, decision-making, and experiential learning requires multidisciplinary expertise and an integrative research approach with attention to the redundancy principle and split-attention effects. Complex interdependencies exist with a focus on actors, processes, and patterns that may be highlighted through the actions of individuals over time responding dynamically in-situ (Zaccaro et al. 2012). We plan custom visualizations such as temporal charts, dynamic networks, and geographical maps that leverage real-time behavioral data for immediate display (and later analysis) from multiple sources, including unconventional data (e.g. physical and biometric sensors on emergency responders, environmental sensors, social media, news feeds) and conventional data (e.g., human-observer checklist, audio communication streams). These visualizations help establish a common operating picture for incident commanders and for reflection by the emergency response teams in the debriefing session. Our primary hypothesis for this design is that viewing these multiple data stream provides important redundant information to reduce uncertainty related to conventional and unconventional data sources for assisting decision-making, coordination, and learning.

4 Preliminary Prototyping, Lessons Learned, and Conclusion

The proposed system dashboard, shown in Fig. 2, was prototyped for visualizing the movements of fire and rescue exercise participants at our collaborating regional fire and rescue academy. The emergent system continues to evolve testing and integrating multiple sources of information to visualize relevant information about resources, events and actions by location and time. The identified information layers will continue to be designed to deliberately address critical points of team-based interaction related to crisis response decision-making, situation awareness, coordination, and learning during identified interactions or inflection points (such as the effective handoff of the patient from one team to another that involves decisions for routing, awareness of patient condition and environment, coordination and communication between teams, etc.).

Fig. 2.
figure 2

Snapshot of a dashboard with two widgets from our preliminary prototype for the proposed system design, where the time series data visualization is shown for the movement pattern of diverse trainees during an exercise, which shows the movement pattern of a potential coordinated behavior between multiple actors of different teams (c.f. details: Dubrow et al. 2017). The visual dashboard provides the ability to browse the data over time and location, which enables an instructor during debriefing to go back in time and filter all the information across the set of behavioral analytics widgets, which correspond to mining each type of multimodal data streams.

Full size image

The next step is for the system to visualize these selected multimodal information streams into a common operating picture (commonly referred COP in the emergency domain), or dashboard, for use in the simulation by the emergency operations center as well as in after-action review, or debrief. Providing visualization of additional unconventional information channels from social media and sensors may offer new information for decision-making and learning to enhance situation awareness, and coordination for participants. Providing additional information channels that are not centralized or accessible in Emergency Operations Centers has been linked to improving understanding of a crisis situation and supporting decision-making of emergency response teams (Van de Walle et al. 2016). However, ensuring that the emergency response professionals are provided the right information at the right time that does not add to their cognitive load and provides validity and improved comprehension with selective redundant information remains a challenge that we plan to address in our ongoing work.

In conclusion, this paper presented a multimodal streaming system design and preliminary findings and challenges from prototyping in a user-centered design process with professionals at a suburban fire and rescue training facility to inform future research.