Analysis and Requirement Generation for Defense Intelligence Search: Addressing Data Overload through Human–AI Agent System Design for Ambient Awareness

Abstract

This research addresses the data overload faced by intelligence searchers in government and defense agencies. The study leverages methods from the Cognitive Systems Engineering (CSE) literature to generate insights into the intelligence search work domain. These insights are applied to a supporting concept and requirements for designing and evaluating a human-AI agent team specifically for intelligence search tasks. Domain analysis reveals the dynamic nature of the ‘value structure’, a term that describes the evolving set of criteria governing the intelligence search process. Additionally, domain insight provides details for search aggregation and conceptual spaces from which the value structure could be efficiently applied for intelligence search. Support system designs that leverage these findings may enable an intelligence searcher to interact with and understand data at more abstract levels to improve task efficiency. Additionally, new system designs can support the searcher by facilitating an ‘Ambient Awareness’ of non-selected objects in a large data field through relevant system cues. Ambient Awareness achieved through the supporting concept and AI teaming has the potential to address the data overload problem while increasing the breadth and depth of search coverage.

1. Introduction

Intelligence searchers across government and defense agencies face an increasing volume of data and documents from which to locate or ‘discover’ information that yields insights to support informed decisions. This situation is characterized as the problem of data overload, wherein an individual, aided by systems or other agents, encounters difficulty in selecting, assembling, or synthesizing the required data subset to address a task requiring situational assessment across a much larger data field [1]. Here, the intelligence searcher is supported with search tools along with co-workers in a collaborative effort to locate, gather, and assess documents that address the task of informing an intelligence objective from the nearly limitless supply of documents that could be used for the task. Specifically, data overload hinders the searcher’s ability to identify a subset of data that provides sufficient detail to satisfy an intelligence objective, which is critical to operational task accomplishment.

The systems employed for the general task of intelligence search may contribute to data overload symptoms. Specifically, tools for intelligence search exhibit flaws related to the idea of ‘keyhole brittleness’ [2]. Keyhole in this context refers to the narrowing of presented data, allocating the remaining data to additional hidden screens. This presentation requires the researcher to manually navigate and synthesize the information from multiple screens of data to understand the utility of a search. Concurrently, the brittleness of these existing tools is tied to their potential to support the extraction of meaning from data present in the intelligence search work domain. Therefore, brittleness arises from the narrow presentation of information in search tools. This limitation leads to a breakdown in effectiveness for complex tasks requiring the extraction of meaning from large quantities of data extending beyond the given system presentation. Additionally, the failure to adequately capture and convey the complex structure of the intelligence search work and information domain impedes the understanding and management of information necessary to support operations. This results in inefficiencies where a searcher often misses valuable insights and data pertinent to the objective while expending additional time navigating across screens to complete the task.

These broad challenges facing search within the intelligence work domain form the research problem for investigation in this study, which is summarized at the top of Figure 1. Terminology and acronyms commonly used to address such challenges are also structured in Figure 1 as a roadmap for this research. This overview with defined acronyms may also be helpful for readers unfamiliar with the background material covered in Section 1.1 and Section 1.2 as it organizes and relates the concepts into a visual workflow as described in Section 2.3. This figure will also be referred to directly from Section 2 and Section 4, as it bridges the methodology to research questions and ultimately to the related findings. Three general steps from Cognitive Systems Engineering (CSE) methods define our approach below our research problem area. Lines indicate input/output relationships within and across the steps as a workflow, where applicable.

The following subsections address the background, related work, and detailed research objectives being pursued. The next subsection describes the background for approaches to modeling and understanding work as it is performed. Section 1.2 provides background on applying domain-specific insights to the development of effective work-supporting systems. Section 1.3 introduces related work that investigates the analysis of work domains with similar concepts or challenges and the subsequent design of supporting systems. Section 1.4 provides additional background for intelligence search work with the reported data overload and keyhole brittleness problems. Finally, Section 1.5 concludes with the precise research objective, aims, and research questions addressed in this research.

1.1. Work Domain Analysis

Understanding work, particularly work with complex technological systems, as it is performed requires understanding physical items or functions as well as more conceptual or abstract concepts [3]. Rasmussen’s Functional Abstraction Hierarchy (FAH) provides the prototypical representation of this multi-layered approach. The FAH links high-level human goals to the physical entities that are manipulated to achieve the goal through a multi-layered hierarchy of physical functions, abstract functions, and measures that are monitored to assess goal achievement [4,5]. While the representation provided by the FAH illuminates the association of physical items with abstract cognitive functions and user goal achievement, it is often difficult to represent the fact that humans pursue multiple goals in an FAH, where these goals are often at different levels of abstraction [6]. As a result, alternative representations, such as goal hierarchies that are tied to decision requirements and the information needed to facilitate these decisions, have been applied [7,8]. However, these frameworks alone do not capture the functions and knowledge necessary to model the organization of abstract concepts to support the monitoring of higher-level goals. In fact, Humphrey and colleagues combined the goal hierarchy approach with decision ladders, where the decision ladders aided in the identification of shunts, which represented decisions made with more abstract concepts [7].

Another representation of the work domain that captures goals and processes to permit the derivation of information and decision requirements is the Functional Abstraction Network (FAN) [9]. This approach has been applied to support the development of decision aids [10,11]. An apparent advantage of the FAN is its ability to represent multiple goals at different levels of abstraction while detailing the cognitive process associated with each goal. The FAN process model associated with each goal includes the transformation of resources in the system into a commodity [10,11]. These commodities are influenced by physical quantities or information flowing through the system and are the focus of each goal being modeled in the FAN.

Work domains for analysis have been classified by the environmental structure within which work is being conducted. This structure influences the transformation of system inputs into goal states. Goal achievement within a physical system is generally constrained by the laws of nature, with conceptual systems being constrained by user intent [12]. Rasmussen characterizes these extremes as “Causal systems” and “Intentional systems”, respectively [13]. Bennett and Flach also describe the nature of work environments in similar terms and refer to the extremes as “Law-driven” and “Intent-driven” work domains [14]. The work domain surrounding general research for intelligence is intent-driven, as the goal states are primarily determined by human intent in the form of value structures related to the resources and commodities found in work processes. The character of a work environment has a strong influence on successful approaches to designing effective systems and interfaces for humans to work with [12]. While the EID methodology has been applied to solutions for work systems operating in both law-driven and intent-driven domains, work focusing on the application of EID to intent-driven domains is limited, as pointed out by Bennett and Flach [14].

1.2. Cognitive Systems Engineering

The Applied Cognitive Work Analysis (ACWA) method, proposed by Potter, Elm, and colleagues, provides a comprehensive Cognitive Systems Engineering (CSE) approach to develop and apply a new understanding of a work domain to effective system designs [9]. The CSE workflow of the ACWA method consists of a process to define fundamental elements of the domain and design that relate to a functional goal requiring support through the system design effort. Once a functional goal is selected from the domain model as the basis of an ecological system design, it is further analyzed into elements that support the design process. This analysis is performed in the context of the domain model to determine requirements for the functional goal with corresponding elements for representation in an effective design.

The artifacts resulting from the analysis of a functional goal with a corresponding design effort using the ACWA method are illustrated in the context of the intelligence search domain in Figure 1. First, the cognitive work requirements (CWRs) of the selected functional goal are identified using an understanding of the associated goal process and related insights from the wider domain analysis. From these CWRs, the Information and Relationship Requirements (IRRs) of the cognitive work are identified to shape representation requirements. Representations for the search data based on IRRs are described as Representation Design Requirements (RDRs) in conjunction with a supporting concept for the DSS. While the ACWA method is shown as a linear workflow, it is expected to be executed in a recursive and iterative manner [9].

The constraints within a FAN domain model are documented using the abstraction modeling frameworks originally proposed by Marr [15] and Rasmussen [13], which have been recently updated by Elm and colleagues [9]. The FAN model within ACWA is used to identify requirements for a system that allows the user to visualize and control the domain at a more abstract level, directed at goal achievement. The alternatives are representations that are misaligned with respect to the true nature of the identified goal, which are therefore more likely to be inefficient representations [9].

Goal-linked requirements and representations are generated by identifying the system variable and processes that influence it. These system variables, also described as commodities due to their production through system processes, are measured to determine the status or state of a functional goal. Relationships across the FAN domain model are then used to determine the CWR necessary to achieve the functional goal.

The concept of a CWR is also often referred to simply as a decision requirement, as many instances of cognitive work take the form of decisions. CWRs also represent any comparison or inference based on information or relationships found in the work domain, as further explained by Potter et al. [10]. The inputs to each CWR must be analyzed to produce a list of IRRs necessary to accomplish the cognitive work as it relates to the functional goal of interest.

IRRs, which correspond to resources and states identified in the FAN, are linked to goals through the processes associated with their CWRs. The IRRs identified also describe relationships between objects that can be derived from the work domain, as described by Elm and colleagues [9]. System design requirements proceed in the ACWA method with the development of RDRs from the CWR and IRR details. These RDRs account for the functional information and representation structure necessary to support human cognition and decision making within the system. These RDRs are expected to be consistent with the work domain ecology as they are based on the domain modeled in the FAN.

EID extends the CSE framework for analyzing and developing sociotechnical systems with the primary goal of providing affordances to perception and action capabilities that reduce the cognitive demand placed on humans [14]. Affordances are features in the interface design that signal what is possible and how the user can interact with the system to achieve their goals. EID includes collecting data regarding the work domain and the decision processes employed by system users. Analysis techniques are employed to identify the salient characteristics of the work domain, including the cognitive and physical activities being performed along with the decisions and information needed to execute the activities to achieve the user’s goals [16]. During design, provisions are made to present goal-relevant information, thereby providing the user with knowledge of affordances related to goal accomplishment [17]. Without these design affordances, a highly desirable action or outcome may be possible given the environmental state, but the user will not perceive it as being available within the system. In EID, the user is then given the ability to directly manipulate the system state through these affordances, coupling perception and action [17]. In summary, the EID approach enables the development of interface designs that are tailored to the characteristics of the supported work domain, ideally permitting the user to perceive and manipulate objects to achieve their goals.

1.3. Related Work

The work domain surrounding intelligence production has been addressed in the CSE/EID literature, with a primary focus on the analyst role. Additionally, applications of research in this problem domain often seek improved support for analysts and the production of intelligence assessments. Recent investigations have addressed coordination of the sensemaking process [18] and improving the accuracy of assessments [19].

Elm and colleagues introduced a “support function model” that enumerates broad system requirements for effective human-machine teaming in support of intelligence analysis [20]. This model is structured with three abstract functions: “Down-Collect”, “Conflict & Corroboration”, and “Hypothesis Exploration”, and it incorporates functional requirements for analysis with requirements for effective teaming. Recent work by Voshell et al., extending the work of Elm and others, emphasizes the challenges analysts face in managing data overload and the limitations of existing automated decision aids [21]. Their research suggests that analytic support tools have historically failed because they too often try to replace analyst reasoning instead of supporting the iterative process. One key finding of Voshell et al. for realizing effective support was to facilitate the development, maintenance, and representation of the analyst’s “problem domain model” used to carry out assigned tasks.

A desirable outcome of EID is to reduce overload through the application of a goal-centered model of the information domain in a supporting system [22]. Subsequently, Patterson et al. suggest model-based visualizations in these systems that serve as conceptual spaces for organizing data based on the relationships found between task-related objects to address data overload [23]. This conceptual space is part of a set of constraints on effective solutions to the general data overload problem [1]. Systems leveraging relevant conceptual spaces could augment the analyst’s ability to perceive, analyze, and manipulate the relationships among complex information elements in the work domain [24]. These designs could be coupled with AI support to address Elm and colleagues’ calls for new automation to assess document relevance in intelligence analysis support tools [20]. Finally, AI support could also be leveraged to promote the observability of document relevance or more abstract criteria related to goal accomplishment through familiar social interactions [25].

The recent literature investigating the potential for human-agent teaming for support emphasizes the importance of understanding the nature of the work involved and the capabilities of both human and AI agents for the work being considered [26]. Similarly, concepts for implementing coordination schemes between humans and AI agents have been proposed for collaborative work [27]. Also, to design effective human-agent teams, frameworks have been developed and refined to specify the level of automation expected from artificial agents [28].

While research has been published on addressing challenges within the intelligence work domain as described, the recommendations have been in the context of the intelligence analysis process, specifically toward abductive inferential analysis. This research seeks to understand and support the work of intelligence searchers, which could be considered one of many specializations within the work domain [29]. Individuals serving in the duty positions titled ‘general researcher’ and ‘intelligence analyst’, referred to as ‘intelligence searchers’ here, execute a search workflow to accomplish intelligence objectives. While our primary focus is on intelligence search, our model of cognitive activities and supporting concepts may generalize to similar abstract functional processes found in other work domains. These similarities may exist with work requiring detailed search tasks to discover new information, such as academic research, legal, financial, or medical professions.

1.4. Intelligence Search

Intelligence search work involves the systematic retrieval and analysis of information to address specific intelligence objectives. This process begins with managing and organizing existing knowledge as a baseline for understanding the intelligence objective and informational domain. Based on this foundation, strategies are devised to search for and curate relevant data. While executing queries within a search, vast amounts of data are sifted through to extract and describe the available information to satisfy an intelligence objective. Throughout this workflow, the objectives and search parameters may evolve in response to new insights, thus requiring dynamic and adaptive support tools.

Within the context of intelligence search, all possible documents and the information they contain are considered the document universe that serves as the instantiation of the information domain the users are operating in. Therefore, the document universe includes the entire collection of documents, data, and sources that could be used to address a specific research question or provide information needed during the Intelligence Search Workflow (ISW). Thus, within this work domain, a significant challenge in solving the data overload problem is the ability to create an effective representation of the document universe that highlights task-relevant relationships.

The general researcher role within some defense intelligence organizations fulfills a specialized function in the intelligence workforce. This role focuses on retrieving new or previously collected documents that are relevant to a tasked intelligence objective. Organizations adopting these roles enable worker specialization by focusing primary duties on either retrieval or analysis tasks. The general researcher specializes in crafting and executing queries in search of information related to current intelligence objectives. Intelligence analysts are specialized in the function of formulating a defense intelligence hypothesis with respect to currently available information. Both roles may also specialize in maintaining knowledge of a particular mission area, such as a regional or technological-based assignment. While there is specialization among these two roles, both are challenged by properties of the intelligence work domain related to the retrieval and evaluation of task-relevant information.

1.5. Research Objective and Structure

As previously described, intelligence searchers are facing the issue of data overload, where they are challenged in identifying task-relevant documents and selecting content within those documents to satisfy an intelligence objective. Additionally, tools for intelligence search generally offer poor support due to their design and susceptibility to keyhole brittleness, which uniquely impacts each functional goal within the ISW.

Given these challenges, the foundational aim of this effort is the development of a work domain model that represents the abstract functional nature of the ISW with sufficient detail to facilitate further investigation and the development of design requirements for work-supporting systems. The research questions associated with this first aim, addressing Work Domain Analysis are grouped by scope to address the following:

• Modeling the broad structure of the work domain to facilitate a general understanding of the functional nature of intelligence search (RQ 1a, 1b) and narrowing from the domain structure to a specific goal that may currently be challenged (RQ 1c);
• Modeling the detailed cognitive work and information requirements associated with a selected goal (RQ 2a, 2b) and identifying insights related to the domain structure that may relate to the development of effective decision-supporting representations (RQ 3).

The second aim of this research is to investigate and describe system design requirements that are consistent with a domain model representation of intelligence search. The research questions associated with this aim address domain-driven system design requirements with the following:

• Visualizations corresponding to conceptual spaces relevant to a selected functional goal in the ISW (RQ 4a);
• The linking of proposed visualizations back to the functional goal intended for support through specifically identified requirements (RQ 4b).

Finally, the third aim of this research is to develop a concept and theory for achieving effective support for decision making through the proposed system design requirements. The research questions tied to this aim describe the envisioned solution for achieving support for intelligence search through the following:

• Addressing the coordination of visualizations within a DSS incorporating AI agents to support the identified requirements (RQ 5a);
• Describing the theory for how the provided information representations and concept for teaming with AI agents facilitate effective support for the selected functional goal (RQ 5b).

The alignment of aims and associated research questions to fulfill the stated objective is illustrated in Figure 2. The research aims and associated questions are described in conjunction with the research methodology in Section 2. Section 3 presents the results as artifacts resulting from the application of the research methodology. Section 4 provides the subsequent analysis based on further investigation of the primary artifacts representing the domain model, system design requirements, and the envisioned solution. Section 5 follows with a discussion summarizing the aims, limitations, and future work related to this study. Section 6 provides conclusions for this work.

2. Materials and Methods

The current research employs a CSE method that seeks insights into the properties of the ISW and work domain through modeling the networked goal-related processes within. Additionally, this methodology is applied to the generation of Representation Design Requirements (RDRs) for the design of future search DSS systems to team with humans during intelligence search. This method includes a description of participants, data collection procedures, a data analysis approach for modeling the domain, and developing design requirements to support a subset of the intelligence searcher’s goals.

2.1. Study Participants

The primary author relied on seven years of prior work experience in the intelligence community, ranging from collection to analysis and production operations, to motivate this research and provide an initial understanding of the work domain. Additionally, a pair of participants with a combined 40 years of experience in the intelligence community provided domain and system design feedback. These Subject Matter Experts (SMEs) provided details related to the ISW during the domain analysis and design steps of the study illustrated in Figure 1.

The SME participants routinely rely on the ISW in the execution of their primary duties. These participants were selected due to their knowledge of the work domain and availability for this study. The SMEs were assigned to different organizations with unique roles in the intelligence community while participating. Participants from organizations with dissimilar mission areas were selected to provide insight into the potential variation in the ISW with respect to the goals, tools, processes, and policies being applied. The diversity of viewpoints and experiences from operations in multiple organizations were prioritized while modeling the work domain from which to develop the DSS concept.

2.2. Data Collection

Data collection consisted of gathering work domain data with traditional knowledge elicitation techniques and soliciting feedback on the resulting domain models and system design artifacts. Within the first round of data collection, interviews were conducted with the SMEs utilizing the Critical Decision Method (CDM) as outlined by [30]. This approach includes reviewing existing documentation and eliciting expert knowledge, as originally summarized by Hoffman and colleagues to support cognitive task analysis [31]. A generic intelligence research scenario was generated from the raw SME data to guide further knowledge elicitation sessions and to support system engineering efforts.

The CDM interviews generated the initial data for analysis to develop a preliminary FAN to capture the intelligence searcher’s decision-making process and the goal structure of the work being conducted. Draft domain models and subsequent artifacts were reviewed with the SMEs to gather feedback on the goal structure, processes, and design requirement suitability. These steps were iterated through work scenarios until a design assessment with SMEs confirmed that the model accurately represented the work domain and that the system requirements addressed the identified challenges. The domain model was further analyzed to identify the CWRs, IRRs, and RDRs associated with goal-process nodes of the FAN [9].

2.3. Analysis and Design Approach

An analysis of the domain and design feedback data was performed using the ACWA method proposed by Potter, Elm, and colleagues, as summarized in Section 1.2 and illustrated with the CSE workflow in Figure 1. This CSE workflow seeks to address the research objective, aims, and research questions depicted in Figure 2. Research questions are related to the methodology steps across the figures through the aligned research aims, as described below.

The Work Domain Analysis step of the CSE method was undertaken to develop a new understanding of the intelligence search work domain, answering RQs 1a–1c. RQ 1a and 1b are addressed through the development of a FAN model of the intelligence search work domain. RQ 1c is addressed through analysis with the domain model and yields a functional goal that is challenged with the current approaches and tools intended to support intelligence searchers. This functional goal continues as the basis for further analysis and is the focus of the envisioned solution. The goal is further analyzed within the Work Domain Analysis step to address RQs 2a and 2b, which outline requirements specific to goal accomplishment. Finally, the functional goal is considered within the context of the intelligence search FAN to address RQ 3, which provides additional insights into the nature of intelligence search. These insights are identified based on their potential to inform the effective representation of functional information in future system designs.

The system design requirements step was performed to outline design requirements consistent with the challenged functional goal by considering the CWRs and IRRs and the domain insights as identified with RQs 1c, 2a, 2b, and 3. The FAN was used to identify requirements for a system to allow the intelligence searcher to visualize and control the domain at a more abstract level, directed at goal achievement. The resulting design requirements take the form of RDRs organized into visualizations in a DTD, thus addressing RQs 4a and 4b. The RDR development approach incorporated the EID principles for intent-driven domains by developing requirements that reflect the work domain structure that governs goal outcomes [16]. The design requirements outlined in this step were then paired with a supporting concept for the challenged goal to form the envisioned solution. This solution, built upon domain insight, incorporates an AI teaming concept to address the current challenges in the final step of the CSE method.

The envisioned solution step integrates the general domain insights from RQ 3 with the functional goal-supporting design requirements from RQs 4a and 4b. This synthesis yields the concept of coordination and integrating visualizations into a proposed AI teaming DSS for effective support of the functional goal. This concept includes implementation considerations for elements of the proposed DSS, which in total addresses RQ 5a. Finally, further analysis was performed to formulate a theory for how the proposed design, which incorporates insights from the domain ecology, may achieve support for the functional goal and address the specific challenges identified. The resulting theory describing the process of achieving decision support through a desirable system property thus addresses RQ 5b.

The iteration of these steps provided insight into the abstract nature of the work being performed across multiple cycles of analysis and design. Recursion through all layers of goal and process decomposition and back through these cycles ensured consistency and coherence among process input-output and goal-influencing commodities. This allowed for the development of representation requirements through multiple rounds of feedback to better address the structure and content of information in the work domain. Iteration was halted when there was a clear link between FAN, CWR, IRR, and RDR elements that addressed support for the identified goal.

3. Results

The domain modeling results from the ACWA methodology are expressed in a FAN domain model for intelligence search with the functional structure described in Section 3.1. Further detail on the functional goal of information mapping modeled within the FAN is provided in Section 3.2. Insights from further analysis of both the structure and detail modeled are presented in Section 4.1 as the observed challenges and nature of the work domain. Section 3.3 applies insight from the domain structure and functional goal details to RDRs that address the challenging aspects of information mapping. A Display Task Description (DTD), which organizes the RDRs with descriptions for corresponding visualizations, is also covered. Finally, Section 3.4 introduces the desired system property to facilitate support for information mapping with a DSS operating as a human-AI agent team. Further analysis of the system design requirements, Ambient Awareness system property, potential benefits, and additional considerations is covered in Section 4.2.

3.1. Intelligence Search Work Domain (Aim 1 Structure)

The domain modeling process reveals that intelligence searchers perform their work within the context of the larger intelligence mission presented in Figure 3. As shown, the information searcher strives to achieve the functional goals of intelligence search as part of the intelligence production process while balancing support for these wider organizational goals. The information searcher’s primary functional goal is to ‘apply information discovery to the intelligence objective’ in support of organizational goals.

The organizational goals of the supported intelligence analysis and reporting mission establish the overall outcome and quality measures that influence the ISW. During this research, it was observed that while the organizational goals shown in Figure 3 are generally beyond the scope of work performed with the information searcher alone, they shape the work of the information searcher. Therefore, analyzing these goals helped to form a better understanding of how organizational goals influence the information searcher’s work.

Information discovery is the process of uncovering new information or knowledge from existing data. It generally involves a variety of activities, such as gathering and analyzing data, making observations, forming hypotheses, and testing theories. These activities help to identify patterns and relationships in the data that can be used to draw conclusions or make predictions. Information discovery also involves creative thinking and problem-solving skills to uncover new insights from existing information. By engaging in these activities, individuals can gain a deeper understanding of their environment and develop new ideas for further exploration. We further define information discovery in the context of intelligence operations as an extensive search activity designed to locate documents and describe how they address specific facets of an overarching research project (intelligence objective).

Intelligence searchers approach their work by considering the abstract concept of ‘information discovery’ within the document universe. They achieve this by applying search and information mapping processes to intermediate representations of the intelligence objective, mission area, operational policy, and data repositories to deliver new information related to the intelligence objective. This necessitates assessing search requirements, formulating a search strategy, executing queries to produce relevant results, and mapping information from all searches back to the intelligence objective. As depicted in Figure 4, these search and information mapping processes comprise the information discovery (ID) function, which is integrated within the broader intelligence production process.

Figure 4 presents the Functional Analysis Network (FAN) of the intelligence search work domain, expanding from the focus on Goal F1 within the intelligence mission in Figure 3. The FAN identifies three core abstract functions in support of this goal:

• Knowledge Management: represented by Goal F2, this function focuses on retaining expert knowledge representing critical mission area information and process execution details;
• Collection Planning: manages the subjects and their associated Required Information Elements (RIEs) to be searched and later described;
• Information Discovery: Comprises six functionally related goals, each with an associated process flow model yielding a goal-linked commodity. These commodities are consumed as indicated by relationships to supported goals. Cycles formed by relationships within the ID function are driven by the multi-search information mapping goal labeled F4.2, which provides direct support back to Goal F1.

The abstraction functions are interlinked by cycles driven by Goal F1, like those driven by Goal F4.2 within the information discovery function. The component functional goal-process nodes within these abstract functions are grouped with background shading to show the functional relationship. The ‘F’ prefix for the numbers within the named functional goal-process node specifies the functional nature of the goal within the ISW. A fully detailed figure representing the FAN model for the ISW is available in the supplemental materials, Figure S1.

As shown, the primary goal of the intelligence searcher within the intelligence production process is the successful application of the ID function. Feedback from the Analysis and Reporting (A&R) function is the commodity being monitored to assess this goal state. A&R feedback is produced from the Annotated Source Documents delivered using the ID function for a tasked intelligence objective. The goal of applying ID within the intelligence production process to yield A&R feedback is illustrated in the goal-process node labeled Goal F1 in Figure 4. Sub-goals within the ID function are evaluated by monitoring commodities that represent search requirements, strategy, search query and document quality, and search history. All these commodities and associated goals ultimately contribute to the development of mission-aligned information, which feeds the A&R function of the intelligence production process.

Organizational goals were considered when modeling and analyzing the execution of all supporting goal-process nodes in the FAN. For example, the need for timely and up-to-date information products leads to the need to maintain and understand changes in search results to reduce the time and effort needed to perform this evaluation on an ongoing basis, which are captured as goals F4.2–F4.6 in the FAN provided in Figure 4. Finally, it was noted that these organizational goals can compete during process execution, requiring searchers and supporting systems to adapt to achieve acceptable compromises among them.

The observation that the work processes and outputs of the intelligence searcher are embedded within the wider A&R function of the intelligence process highlights a significant challenge in evaluating human work. While traditional systems engineering analyses seek to understand, decompose, or construct systems with known, typically closed boundaries, there is often no clear boundary where human work begins and ends [32]. For the intelligence searcher, the work they are solely responsible for can be analyzed, but this work is embedded in joint work with others in a team, which breaks the system boundary. Thus, analyzing human systems with open system boundaries presents a challenge to traditional systems engineering approaches.

One recurring observation throughout the domain modeling effort was that expert intelligence searchers are challenged by current tools, which do not effectively aggregate search results. This situation can be generally described as a mismatch between the functional representation present in the current system and the mental model or work domain-structured model employed by intelligence searchers. This specific mismatch shifts significant workload toward manual workarounds to aggregate and make sense of the results acquired across multiple queries. Therefore, we suggest that supporting concepts facilitate efficient aggregation of searches comprising numerous queries into a single pool of candidate documents for further inspection.

The supporting concept should also allow intelligence searchers to visualize and interact with the information domain at more abstract levels, which supports ID through information mapping. This type of support would therefore expand the ‘keyhole’ while more directly addressing the true functional goal within the ISW (map documents to intelligence objectives). Systems supporting work at this level may aid searchers when mapping documents retrieved from multiple queries back to the RIEs of an intelligence objective, thus providing some relief to the ‘brittleness’ problem.

Finally, supporting systems should provide representations of the value structure that are observable and accessible to the searcher and cooperating agents. This is critical since this value structure holds the criteria for determining relevance and other important relationships among retrieved documents and the intelligence objective. By facilitating access to and applying this value structure, the supporting system can more directly support the mapping of retrieved documents to RIEs.

3.2. Information Mapping (Aim 1 Goal Detail)

As the central functional goal within the ID function directly supporting the intelligence mission, the information mapping goal is further described in detail. These details further the aim of identifying challenges facing intelligence search and pinpointing the specific means by which to address them. While Figure 4 offers a high-level view of the ISW, Figure 5 zooms into the details of the goal-process node for Goal F4.2, which is elaborated upon in this section.

In addition to goal-process node details, Figure 5 contains added context for process decomposition and intra-goal relationships. The process block depicting ‘multi-search information mapping’ reveals a cyclical process that yields mission-aligned information consisting of source documents annotated with insights related to the intelligence objective. The process expands a pool of candidate documents through search and the execution of successive searches to address objective-related requirements and the dynamics inherent in information discovery. Information mapping within this process generates insights from search results by applying decision-making criteria from a dynamic value structure to produce mission-aligned information. The current state of mission-aligned information is assessed for search requirements and halting criteria. Additional search requirements may be generated based on new insights from the information mapping process. The status of all identified requirements forms halting criteria for the cycle, where mission-aligned information is returned to the intelligence production process. The resulting mission-aligned information is structured by the subjects and RIEs of the tasked intelligence objective. This structure is populated with detailed annotations or excerpts from documents retrieved from searches that are referenced as sources that are believed to address the task objective.

The intelligence searcher seeking to satisfy the information mapping goal and return mission-aligned information to the intelligence production process is faced with specific cognitive work requirements in the effort. Furthermore, these cognitive work requirements, which may take the form of specific decisions, require access to information from which to make an informed decision or execute the specified cognitive work. The CWRs and IRRs enumerated for the information mapping goal F.4.2 are shown in

Table 1. Supporting IRRs for the CWRs inherent to Goal F4.2.

Goal-Related CWRs and IRRs
F4.2 Manage Information Mapping Across Searches
❖ CWR 4.2.1—Develop and capture insights with respect to document content and relationships defined by the current value structure in a revised value structure. ⮚ IRR 4.2.1.1—Document relationships to the project value structure. ⮚ IRR 4.2.1.2—Relationships among documents based upon qualities established in the value structure. ⮚ IRR 4.2.1.3—Project value structure. ⮚ IRR 4.2.1.4—Document content.
❖ CWR 4.2.2—Determine which segments from candidate documents address the RIEs. ⮚ IRR 4.2.2.1—Elements of the project value structure derived from sources including the intelligence objective and insights gained through the information discovery process. ⮚ IRR 4.2.2.2—The semantic relationship between segments of candidate documents and RIEs in terms of the project value structure (e.g., Relevant to the RIE, Subject, or additional topic/concept of interest. ⮚ IRR 4.2.2.3—The data contained in candidate documents.
❖ CWR 4.2.2A—Establish a confidence level for information quality based upon document content and source. ⮚ IRR 4.2.2A.1—The general quality of the originating source of the candidate document. ⮚ IRR 4.2.2A.2—The general quality of the candidate document based upon consistency between the information contained in the candidate document and other available information. ⮚ IRR 4.2.2A.3—The data contained in the candidate document
❖ CWR 4.2.2B—Determine the utility of candidate documents to addressing RIEs. ⮚ IRR 4.2.2B.1—The semantic relationship between the candidate document and RIE. ⮚ IRR 4.2.2B.2—The information quality of each candidate document. ⮚ IRR 4.2.2B.3—The data contained in the candidate document
❖ CWR 4.2.3—Choose a hypothesis which links selected candidate documents to an RIE. The hypothesis is a statement which addresses the RIE by referencing the retrieved document and highlights consistency or contradiction among document segments. ⮚ IRR 4.2.3.1—The mutually supporting document segments aligned to the RIE. ⮚ IRR 4.2.3.2—The contradicting document segments aligned to the RIE. ⮚ IRR 4.2.3.3—A summary of supporting and contradicting document segments aligned to the RIE. ⮚ IRR 4.2.3.4—The confidence level for information contributing to the RIE from each document. ⮚ IRR 4.2.3.5—The data contained in the candidate documents

Note. The goal, CWRs, and IRRs listed here are enumerated to form the basis for developing precise system requirements consistent with providing support for goal accomplishment.

. Here, the cognitive demands represented in the goal-process block in the context of influence across the FAN domain model were documented as CWRs. Additionally, the IRRs associated with each CWR are specified to be consistent with the process flow model within the goal and influence the connections represented in the FAN model.

The insights from the domain model we have identified to shape the proposed support for decision making in the ISW are generalized as improving the functional alignment between the system, user, and work domain. The insights applied to achieve this alignment are as follows:

• Aggregate Query Results (foundation for functional alignment);
• Coordinate multiple perspectives addressing the cognitive work within information mapping;
• Visualize and facilitate interaction with the proposed coordinated representations;
• Support dynamic coordination of the value structure representing the search objective;
• Leverage the value structure of automation to facilitate decision-aiding concepts.

These insights are summarized from the detail provided in Section 3.1 and further analyzed in Section 4.1. They effectively answer RQ 3 and form the basis for system design requirements and the supporting concepts addressed within the second and third aims of this research. These insights are applied with the functional goal details from Table 1 to produce the representation requirements for coordinated visualizations to address the specified functional alignment and support covered in Section 3.3.

3.3. Supporting Representations—Visualization Descriptions and DTD (Aim 2)

With precise descriptions of the cognitive work and the associated information requirements to perform information mapping, corresponding representations were generated. This effort continues as part of the investigation into the development of system requirements that may provide effective support to the CWRs associated with intelligence search, specifically with respect to the information mapping goal F4.2. This support is expected to be generated through the effective representation of the IRRs identified that are necessary to execute the process associated with information mapping. RDRs were developed from the information mapping requirements specified in Table 1 with the intent of satisfying the requirements behind the goal. In conjunction with developing the necessary information representations for interaction, a set of coordinated visualizations was also developed. These visualizations were developed to represent the information in a form that supports the identified cognitive work or decision. The RDR and associated visualizations are presented as a DTD in

Table 2. DTD for the ‘Information Mapping’ display.

Information Mapping DTD Context
The information mapping display described by the requirements in this DTD is intended to support intelligence searchers by providing the IRRs necessary to achieve the goal of mapping information from candidate documents to intelligence RIEs. It should assist the intelligence searcher in choosing a hypothesis based upon document segments that address the RIE, which considers the quality of supporting information.
RDR and Coordinated View	CWRs, IRRs, and Functional Support Description
D1: Provide representations of the Information Environment within context of the value structure.	CWR 4.2.1/IRR 4.2.1.1,2,3,4; CWR 4.2.1/IRR 4.2.2.1,2,3—Support the observability of semantic relationships corresponding to the value structure among documents, and between documents and the value structure for all documents in the document universe.
D1a: (Tagged Document View) Represent document content with relationships to specified aspects of the value structure within the context of the original document.	CWR 4.2.1/IRR 4.2.1.4—Represent the development of insights into the information domain from the content of a selected document and relationships between that content and the value structure. CWR 4.2.2/IRR 4.2.2.3—Represent the determination or validation of document segment relationships to RIEs from document content and segment relationships to the value structure. CWR 4.2.2A/IRR 4.2.2A.3—Represent the assessment of a confidence level for information quality from document content and relationships between that content and the value structure. CWR 4.2.2B/IRR 4.2.2B.3—Represent the assessment of document utility to RIEs from document content and relationships between that content and the value structure.
D1b: (Detailed RIE View) Represent document content corresponding to the subjects and RIEs identified in the value structure in context of the information domain representations corresponding to subjects and RIEs identified in the value structure.	CWR 4.2.1/IRR 4.2.1.1,2,3,4—Represent the development of insights into the information domain from the content of grouped documents with shared relationships to an element of the value structure. CWR 4.2.2/ IRR 4.2.2.1,2,3—Represent the determination or validation of document segment relationships to RIEs from document content and relationships between that content and the value structure. CWR 4.2.2B/IRR 4.2.2B.1,3—Represent the assessment of document utility to RIEs from document content and relationships between that content and the value structure. Provides document segments with relationships to subjects and RIEs from the value structure in the context of the intelligence objective as described by the identified subjects and RIEs.
D1c: (Summary RIE View) Represent information aggregated from document segments corresponding to the subjects and RIEs identified in the value structure in context of the subjects and RIEs identified in the value structure.	CWR 4.2.3/IRR4.2.3.3—Represent the selection of a hypothesis statement addressing an RIE with candidate documents with a summary of the supporting and contradicting document segments. Provides a representation of the information domain addressing the subjects and RIEs from the value structure in the context of the intelligence objective as described by the identified subjects and RIEs.
D1d: (Tabletop View, Value Structure View) Represent relationships between document representations across the information domain based upon specified aspects of the value structure and provide direct access to the document contents corresponding to points and relationships in the representation.	CWR 4.2.1/IRR 4.2.1.1,2,3,4—Represent the development of insights into the information domain from the representation of documents and relationships to each other or an element of the value structure. Provides a representation of documents and relationships within the information domain in the context of features of interest from the value structure.

Note: Domain value structure is composed of key terms and concepts from multiple sources, including the tasked intelligence objective. The value structure defines the abstractions and instances of subjects, RIEs, etc., that form weighted attributes used in criteria to make decisions to determine document relevance, quality, etc. Document utility, for example, is based on the weighted attributes of source quality, document quality, and suitability of contents to address RIEs.

in conjunction with the CWRs and IRRs they seek to address.

The DTD describes representation requirements for multiple conceptual spaces from which to view and interact with the document universe through the information mapping process. These conceptual spaces are unique perspectives from which to view the information contained in the document universe with respect to weighted attributes and decision criteria in the value structure. Each view provides a unique perspective on the information domain. They should be linked through common information elements in the interface and used in combination to support information mapping through the coordinated representation of IRRs that are associated with the process.

The Tagged Document View provides a detailed look at individual documents, each tagged with elements of the value structure. This tagging allows users to see how the content of a specific document relates to the value structure, supporting the development of insights into the information domain, the determination or validation of document segment relationships to RIEs, and the assessment of a confidence level for information quality and document utility to RIEs. This view serves as a crucial tool for users to navigate the information domain, make informed decisions based on the relevance and utility of individual documents within the context of the individual document, and interact with found instances of objects represented in the value structure.

The Detailed RIE View offers a focused perspective on the subjects and RIEs identified in the value structure. It presents document content corresponding to these subjects and RIEs in the context of the information domain, allowing users to develop insights from grouped documents sharing relationships to an element of the value structure. This view would be particularly useful for determining or validating document segment relationships to RIEs and assessing document utility to RIEs.

The Summary RIE View provides an aggregated representation of information from document segments corresponding to the subjects and RIEs identified in the value structure. It offers a high-level overview of the information domain, summarizing the supporting and contradicting document segments for a hypothesis statement addressing an RIE. This view would be instrumental for quickly understanding the overall state of the information domain with respect to the intelligence objective.

Examples clarifying the concepts behind the Detailed RIE View and Summary RIE View are found in modern search engine designs. These are the interface features highlighting sentence or paragraph excerpts corresponding to a query and synthesized summaries on the search results page, respectively. The Detailed RIE View in essence, provides aggregation of the highlighted sentences or paragraphs related to the RIE along with semantic relationships to one another. Similarly, The Summary RIE View provides a summary of all related content to form a hypothesis addressing the RIE with information sourced from available documents. By synchronizing these two views, a system design may allow the searcher to easily develop and assess working hypotheses using both document excerpts and summaries in the context of intelligence objective requirements. Further synchronization to the Detailed RIE View would allow the searcher to refine their understanding of interesting content contributing to a hypothesis in the context of the original document.

The Tabletop View provides a broad perspective of the information domain, representing relationships between documents based on user-specified aspects of the value structure. They offer direct access to document contents corresponding to points and relationships in the representation, supporting the development of insights into the information domain from the representation of documents and relationships to each other or an element of the value structure. These views are essential for understanding the overall structure of the information domain and how documents relate to each other and to elements of the value structure.

The Value Structure View focuses on the value structure itself, providing a consistent representation of the elements of interest found in the current value structure. It allows users to understand the degree to which key terms and concepts apply to the RIEs and “Subjects of the Intelligence Objective”, which drive the determination of relevance and relationships between documents. The view provides a means to navigate the information domain based on the value structure and interpret the automation behind relevance assessment and document alignment to RIEs.

3.4. Envisioned Solution—Intelligence Search Teaming DSS (Aim 3)

The concept of facilitating Ambient Awareness was synthesized from insights into the work domain to describe the desired system property with the potential to support information mapping in intelligence search. We define Ambient Awareness, in this context, as the state of being cognizant of non-selected objects in a data field through relevant system cues corresponding to newly revealed relationships tied to a shared understanding of the value structure governing work domain processes. Insights underpinning this concept are the challenges identified in the work domain described in Section 4.1, along with the system design requirements for representations within the supporting concept for information mapping. The concept of Ambient Awareness is uniquely suited to guide the design of AI to benefit the human-agent team for intelligence search.

It is worth noting that the term “ambient awareness” has been used in various contexts, including social media and communication technologies, where it refers to a more general awareness of other activities or presence in social settings. A notable example is the article titled “Social Business Applications Promote Observability via Ambient Awareness”, where the concept is depicted as a form of unobtrusive observability in working environments and is related to social media tools like Facebook [25]. The authors further describe the potential for ambient awareness to address the general issue of ‘information overload’ in the business environment by leveraging the concept in social business applications. The motivation for addressing information overload in business parallels the motivation and challenges presented by the specific problem of data overload defined in Section 1. However, in this context, Ambient Awareness is meant to describe how a user can maintain awareness of non-selected objects in a data field through relevant cues from the system.

To illustrate, consider the way social media platforms like Facebook function. Generally, people do not have the time for one-on-one conversations daily with hundreds of friends to stay apprised of interesting life events. Similarly, searchers do not have the time to initially review and re-review every document in the pool of available documents every time their understanding of a term or concept changes. Facebook and similar platforms allow individuals to cultivate an awareness of their friends through a consistent stream of curated cues to content indicating events of shared interest. This awareness not only provides a feeling of connectedness but also contributes to the efficient identification and exchange of relevant information from a large pool of available information. Analogously, this concept is envisioned to allow an intelligence searcher to maintain an effective awareness of the information domain represented by a collection of documents. Ambient Awareness in this context is achieved through curated cues to available information based on shared insights with respect to dynamics in the value structure. This is expected to enhance the depth and effectiveness of data exploration and interpretation by leveraging a stable structure from which to visualize the information domain and select objects related to goal accomplishment.

Ambient Awareness supports search by allowing the user to maintain passive awareness of non-selected objects or information from a wide data field. This awareness originates at the system level and cues the user as new goal-relevant insights emerge within the system. It then becomes a general sense for the system user in that the feeling of awareness of dynamics across the entire information domain is maintained regardless of focus, zoom level, or subsequent selection of objects from the data field.

The idea behind Ambient Awareness, analogous to the awareness cultivated through social media tools, is to create an environment where the user has a sense of the broader context of the work and information domains. This is achieved through mechanisms that cue the discovery of important information, even if the source is not currently within the immediately selected focus, allowing the user to maintain an effective awareness of the information domain. By maintaining a level of attention and sensitivity to non-selected parts of the data, the system can provide cues or notifications to the user when important information becomes apparent through shifts in the governing value structure. This helps users stay informed and make more informed decisions while participating in the information mapping process.

In a teaming DSS facilitating Ambient Awareness, as defined, the AI agent is continuously monitoring and assessing the dynamics of the information environment and its relationships to the value structure. The AI agent is responsible for sensing shifts in the value structure that governs the relevance and utility of documents retrieved through search, akin to the curated cues in social media platforms that keep users informed of relevant activities.

As the value structure changes, the AI agent cues the human operator to potentially valuable insights, such as newly identified relevant concepts within the current document pool or contradictions to newly accepted evidence. The AI agent maintains an awareness of the value structure and its dynamics in relation to human review activity, thereby providing a form of “Ambient Awareness”. As seen in the context of social media, this may help an intelligence searcher manage large volumes of data and navigate through this data to find the most relevant and useful information.

The human operator within the team provides insight into the current value structure based on their understanding of the information domain with respect to the ongoing search task and intelligence objective. The human operator interacts with the AI agent, responding to cues and making decisions based on information provided in the cue, observed relationships, and their own understanding of the task at hand. The human operator is also responsible for defining and adjusting the value structure that the AI agent uses to assess important relationships, relevance, and utility for documents. In this way, the human operator and the AI agent work together in a collaborative manner, leveraging the strengths of each to achieve the goal of effective information mapping in support of information discovery.

4. Analysis

Here, an analysis of the results produced in the CSE project is reported, which project the modeled or developed artifacts into an assessed potential in support of the ISW. First, insights from the analysis of the ISW domain model are presented. These insights are outlined as abstract challenges represented in the domain, with specific challenges presented by the “as-implemented” system. Finally, an analysis of the proposed design requirements and envisioned solution is provided, along with an analytical assessment of the proposed support for information mapping with general implementation and ethical considerations for the AI-enabled DSS solution.

4.1. Domain Insights (Aim 1)

Further analysis of the intelligence search work domain in conjunction with the development of a representative FAN reveals insights into how the nature of intelligence search work presents challenges given the current situation and opportunities to produce more effective systems. Considering the organizational goals that influence the intelligence searcher’s work through the A&R process yields some insight into innovative or influential aspects of the search process. One important insight is that while some of the high-level goals can be complementary, some can conflict. For example, an extremely thorough intelligence product may be less timely than another, less thorough report, given the extra time required to consider additional source material.

By modeling the organizational goals and analyzing supporting connections to goal-process nodes, opportunities to support user decisions that optimize the satisfaction of multiple goals can be identified. This depiction also permits one to understand any innovative work processes that have been developed to reduce the conflict between competing goals. For example, the goal of “Maintaining a Knowledge Base” presents an opportunity to reduce the competition between the thoroughness and timeliness of a report. Current tools in the work domain do not robustly support the work processes needed to achieve this goal effectively and address this competition. However, manual workarounds have been developed by the information searchers themselves to organize and maintain mission knowledge. These workarounds allow for more timely and thorough reports, thus relieving some competition between the high-level goals.

4.1.1. Current Challenge—System Representation to Functional Goal Mismatch (Aim 1)

Expert intelligence searchers apply abstract concepts like ‘multi-search information mapping’, modeled as Goal F4.2, when determining the relationships between documents and RIEs. This functional goal employs decisions about which documents are relevant and the determination of relationships, demonstrating how each document supports or contradicts a given hypothesis statement for a RIE. However, the current tools available to intelligence searchers (e.g., general-purpose internet search engines and website search interfaces) provide a limited view of the document universe (e.g., single query results) and present a more concrete representation of the problem space (e.g., raw document text, although some search engines provide highlighted document excerpts in the ranked result list). This mismatch between user cognitive requirements and system affordances challenges the information searcher, leading to “keyhole brittleness” [2]. The keyhole brittleness in current systems for intelligence search makes estimating ‘information mapping’ between documents and RIEs challenging and susceptible to error.

Given this keyhole brittleness, the searcher must manually navigate through various screens and independent systems to comprehensively synthesize the information addressing the RIEs of an intelligence objective. Essentially, the ‘keyhole’ restricts the scope of information that can be directly accessed at any given moment, requiring active navigation to explore different aspects of the available data. The “brittleness” is the lack of flexibility and adaptability of these tools to support the extraction of meaningful insights from the available data. Tools for intelligence search exhibit a degree of inflexibility when it comes to accommodating complex tasks that require deriving meaning from substantial volumes of data. This brittleness is tied to the inability to effectively present and manipulate data in a manner that facilitates nuanced analysis and understanding. In essence, brittleness makes these tools less resilient and versatile in addressing the dynamic nature of analysis over the large quantities of data required to meet intelligence objectives. The tools’ brittleness, characterized by their inability to adapt to diverse and complex tasks, can lead to a situation where they break or fail to adequately support the extraction of meaningful insights from the available data. Here, this breakdown manifests as an inability for searchers to comprehend the nuances and relationships within the data, hindering their ability to make informed decisions or derive valuable insights from the available information.

The propensity to break when stressed, or brittleness, can be influenced by several factors. One key factor for search tools is the narrow presentation of information, which restricts the searcher’s access to comprehensive and interconnected data. When information is presented in a fragmented manner across multiple screens or partitions, the information display can impede the searcher’s ability to see the bigger picture and discern meaningful patterns or relationships. Furthermore, the failure to capture and convey the complex structure of intelligence search work and information domains can also contribute to brittleness. Inadequate representation of the intricate connections between data points and the inability to accommodate diverse analytical needs can undermine the tool’s effectiveness. Ultimately, the brittleness of intelligence search tools arises from inconsistencies between the data representation and the nature of the search tasks being supported.

4.1.2. Current Challenge—Dynamic Objective (Aim 1)

The challenge of intelligence search is underscored by the potential for dynamic changes in human understanding that influence information needs throughout the search process. These changes also modify the various criteria guiding the notion of success for the search, creating a ‘moving goalpost’ situation. In this analogy, the criteria for evaluating whether search results meet an information need represent a goalpost that moves as new information is discovered. This metaphor illuminates how current systems may frustrate both the searcher and the search system designer, akin to a soccer game where the goalposts are invisibly shifted, with current locations known only to the referee.

Within the context of a single query search effort, a document is deemed relevant when it matches the criteria for relevance to the information needed. If the criteria are altered after each search attempt, akin to moving goalposts, all participants must adapt to the new success standards. This situation becomes vexing if the new criteria are not easily observable or applicable to the problem, as is often the case with single query systems used for intelligence searches.

From the intelligence searcher’s perspective, search systems that do not dynamically model the search goal present a significant impediment to effective task completion. These systems fail to account for the evolving nature of information needs, leading to inefficient representations that increase cognitive load. System designers face parallel challenges, as they must model this dynamic interaction and create flexible designs that accommodate evolving objectives. This shift from static to dynamic criteria complicates many aspects of system design and implementation, from evaluation metrics to user interaction specifications.

The overall design of systems and supporting structures must therefore transition from query-level interactions to search- or aggregated search-level structures that employ a dynamic representation of the search objective. This search objective therefore must model dynamic user understanding, which contributes to elements in a ‘value structure’ that specifies information needs and corresponding document evaluation criteria for search. One important aspect of the value structure is that it essentially defines the movement of the goalpost, or ‘search goal’. This search goal is the expectation for the final output of search work with respect to the information domain and selected documents. If the value structure is successfully implemented in intelligence search tools, the metaphor will no longer apply to the system, as the ‘movement of goalposts’ is expected and supported. By addressing this challenge, the design and implementation of search systems can better align with the fluid nature of information discovery for intelligence search, enhancing both user experience and system performance.

4.1.3. Opportunity—Domain Regulating Structure (Aim 1)

We find the nature of intelligence search from expert observation and development of the FAN to be rooted in the curation and application of a ‘value structure’. The value structure concept entails the weighted importance of attributes within a set of decision-making criteria that a searcher employs when executing all supporting processes of the ISW. Elements of the value structure comprise objects, relationships, and their relative importance to processes that govern the transition of system inputs into goal states. The value structure governs the entire search process, representing decision criteria that, when applied to documents during the search, yield valuable documents and insights with respect to the intelligence objective. The expert generates an initial baseline for this value structure when tasked with searching and adjusts it as insights are gained through the search to ultimately locate and explain mission-relevant documents. The value structure serves as a model for decision-making within the ISW, independent of specific tools or system implementations.

The value structure can be seen as a comprehensive framework that not only guides the search and analysis process with insight into the nature of the information needed but also provides expert knowledge of the search process itself. Thus, elements of the value structure include attributes of criteria for assessing the importance of information and those for assessing steps within the search process. Search process assessment relates to goals for determining where relevant documents may be found (database or search engine) and the means to assess the reliability of information found (external context for the information source). Elements define the information needed, specify information and its related importance for query development, and support the analysis of results. Each piece of information forming a proposition in search results carries an implied value to the searcher. For assessing results, these elements guide the logical structuring of document-related statements supporting hypothesis development for each RIE.

To address this insight, systems seeking to support functional goals of intelligence search must precisely model the goal-process relevant aspects of the governing value structure dynamically. The fluid nature of the value structure contributes to variability during the search process to address a single objective and between assignments of new intelligence objectives. This observation leads to the requirement for systems to dynamically generate, represent, and apply the value structure governing the current search problem to provide effective support in the intelligence search work domain.

The value structure is dynamic and focused for use in mapping information from located documents back to the RIEs of a tasked intelligence objective. This process resolves abstractions from the task with objects identified in candidate documents, thereby refining the set of subjects and RIEs and contributing to the dynamic nature of the value structure. It is applied and revised through the ISW goal processes to discover information related to RIEs across search sessions.

The value structure goes beyond simply defining the intelligence need; it also includes a broader set of elements that describe how or where information may be found in the environment. This includes the type and likely location of documents that generally contain important intelligence information, as well as related keywords and concepts that may lead to the discovery of important intelligence information. This effectively broadens the search and provides a deliberately wide document set from which to identify the true context of interest, containing the documents and information that address the tasked intelligence objective.

4.2. Teaming DSS/Ambient Awareness (Aims 2, 3)

Considering the insights gained from our examination of the intelligence search work domain and the corresponding FAN, this subsequent analysis will focus specifically on system requirements that enable efficient aggregation of multi-query searches into a cohesive pool of candidate documents. This emphasis aligns with the identified need to enhance the searcher’s ability to visualize and interact with the information domain at more abstract levels, thereby bolstering the wider ID function and component information mapping processes. Concretely, we concentrate on features that aid searchers in mapping documents sourced from multiple queries back to the RIEs of an intelligence objective, a process detailed in Figure 5 within Section 3.2 as part of the ID functional model.

4.2.1. Support for Information Mapping

A defining goal within intelligence search is managing information mapping across searches. Its importance in providing effective support to the ISW overall is indicated by connections across the FAN, specifically numerous connections to the ‘Search History’ commodity. Additionally, we consider it a challenged goal in response to RQ 1c due to the observation that current tools lack effective representations for the required information, leading to ineffective support for the necessary cognitive work.

We have focused attention on specifying system design requirements that effectively consolidate redundant results across search sessions and provide an intuitive measure of a search effort’s comprehensiveness. Moreover, see the potential for these systems to save searchers time by eliminating repetitive information review and facilitating the identification of conflicting and corroborating information for documents of interest. The strategy for this focus is to allow a shift in human effort from simply identifying document relevance toward the goal of determining or validating document relationships to RIEs. Knowing if a document is relevant to the task is helpful to be sure, but knowing why a document is relevant is even better as it addresses the higher-level goal of matching a specific document segment to the value structure, or RIE, directly. This shift is how we envision “improving the functional alignment between System, User, and Work Domain”.

The system design requirements outlined in Section 3.3 and Section 3.4 address specific work domain requirements that emerged from our initial findings to form the basis of a supporting concept. A summary of the concepts described and the envisioned workflow with the proposed teaming DSS is shown in Figure 6. The key findings incorporated into the proposed design requirements are as follows: (1) support the compilation and management of a comprehensive and dynamic value structure from which to analyze search results; (2) aggregate searches consisting of many queries to form a document pool for analysis; and (3) form conceptual spaces for the application of the value structure to map information within a pool of candidate documents to address RIEs.

The results established a unique set of system requirements organized into a DTD based on desired system properties and goal-oriented functionality. The DTD and related concept descriptions are intended to establish the foundation for understanding and generating early-cycle requirements in an iterative development process. Our DTD outlines requirements to support information mapping during searches, which might enable effective human-agent interaction to keep critical data in focus using AI’s data processing abilities. The theory achieving this support was described as “Ambient Awareness”, which is facilitated by AI agents implemented in the teaming DSS. This blend of general design solutions from CSE and the specific needs of intelligence search creates the foundation for a more streamlined and efficient intelligence search process. Our approach clarifies the link between raw data produced by search engines and intelligence objectives, which, if visualized properly, should reduce data overload and pave the way for tools that combine human skills with AI capabilities to boost efficiency in defense intelligence production.

The methods employed to generate system design requirements that may effectively support the ISW are based on the theories of EID calling for synchronization between domain structures and system representations. To validate this aspect of the effort, the proposed system design requirements can be traced back to artifacts from the domain modeling results. In this case, specific cognitive demands were identified by analyzing the FAN and documented as CWRs that align with the goal-process node requiring support. Additionally, IRRs for the challenged goal intended for support were described based on further analysis of the CWRs within the context of the FAN. The CWRs and IRRs covered in Section 3.2 provide the basis for supporting the information mapping goal. Finally, the RDRs presented in the DTD covered in Section 3.3 are associated with the corresponding CWRs and IRRs they seek to address. Thus, the linkage from the proposed visualization solution back to the functional goal of interest is presented. A visual outline for this methodology is found in Figure 1, with a collapsed summary in relation to the findings shown in Figure 6.

4.2.2. Implementation Considerations

AI Agent Training Data: One of the most significant challenges in implementing a system with the proposed capabilities is the gathering and management of extensive datasets for training the AI agent. AI agent training may ultimately require a large amount of diverse and high-quality data to accurately model the complex dynamics and relationships inherent to the intelligence work domain—specifically, interacting with the value structure governing search and applying it to generating valuable insights for the intelligence searcher. Managing such datasets involves complexities related to data privacy, data integrity, and data storage. Moreover, regular updates to the dataset would be essential to keep the AI agent aligned with the dynamic nature of intelligence work. Potential mitigations to this challenge may be found in recent successes with the application of pre-trained large language models to domain-specific problems, where little task-specific training data is available.

Integration with Existing Tools and Processes: Another layer of complexity arises when integrating specialized systems with existing tools and processes. This impacts not only current tools supporting intelligence search but also those tools and systems integrated more broadly across the intelligence production process. Many organizations already have established workflows and systems in place, and integrating a new component may require substantial changes to these existing structures or impose additional compatibility requirements for the teaming DSS. These requirements may include compliance with existing data structures, interfaces, and data handling policies. Compatibility issues, data migration challenges, and retraining staff to adapt to the new system are some of the hurdles that must be overcome to realize any potential benefit from the proposed system design. Opportunities may exist to leverage common frameworks or APIs available within many organizations to integrate new tools into an enterprise ecosystem, which can alleviate many of these challenges.

Value Structure Dynamics: The value structure in intelligence search is highly dynamic, and any system aiming to provide Ambient Awareness through its application to search data must be adaptable. While the AI agent is proposed to monitor shifts in this structure, it must also be able to handle the noise and potential for false positives that may arise from the ever-changing information environment. This must all be accomplished in coordination with the intelligence searcher, who curates the value structure by specifying decision criteria within the current intelligence objective.

Human-AI Collaboration: The effectiveness of the proposed system hinges on the symbiotic relationship between the human operator and the AI agent. The AI agent is responsible for sensing shifts in the value structure and cueing the human operator. Conversely, the human operator is responsible for defining and adjusting this value structure. This interdependence necessitates a finely tuned collaboration, whose success is dependent on both the AI’s performance and the human operator’s willingness and ability to understand and act upon the cues provided by the AI.

4.2.3. Ethical Implications

Bias in Search Results: One of the most pressing ethical considerations for the proposed system design is the potential for the AI agent to introduce bias into the analysis of search results. Despite advancements in machine learning, algorithms can inadvertently incorporate biases present in training data or, additionally in this case, the value structures being curated by human operators. These biases could distort the system’s ability to map information accurately and ethically, potentially skewing intelligence efforts and leading to harmful actions. It is imperative to incorporate methods for auditing and correcting biases in the AI’s understanding of the value structure and the resulting cues for human operators.

Privacy Concerns: Another ethical dimension to consider is privacy. The proposed DSS facilitating Ambient Awareness is intended to sift through extensive datasets, some of which may contain sensitive information. The system’s capability to bring non-selected objects into awareness based on shifts in the value structure can potentially infringe upon individuals’ privacy. It is crucial to integrate privacy-preserving techniques into the system’s design and ensure compliance with applicable privacy laws and regulations.

Transparency and Accountability: The proposed shared effort between humans and AI in decision making raises questions about accountability. The human operator depends on the cues provided by the AI agent to make informed decisions, while the AI operates based on the value structure effectively defined by the human. This shared responsibility complicates the attribution of accountability, particularly when errors occur. A transparent system design that allows for the tracing of decisions back to either the human operator or AI agent may mitigate this concern.

Ethical Oversight: Given these ethical complexities, a governance mechanism to oversee the system’s ethical performance may address these and additional unforeseen concerns. An oversight body could conduct periodic reviews to ensure that the AI-enabled DSS aligns with ethical standards, identifies and corrects biases, and respects policy constraints, which may also concern privacy.

5. Discussion

This study pursued three core aims: First, to develop a comprehensive work domain model for intelligence search, focusing both on its broad functional structure and on enumerating the details for a challenged functional goal. Second, to generate system design requirements guided by the domain model, emphasizing visualizations tied to functional goals and linking these visualizations to specific requirements. Third, to formulate a concept and theory for an envisioned solution that incorporates AI agents within a Decision Support System (DSS), detailing how these agents and visualizations collectively facilitate effective decision-making support for intelligence search. These aims were pursued through a series of research questions to develop and apply an in-depth understanding of the intelligence search work domain to the identified challenges. Here, the results of the CSE project and the subsequent analysis are discussed in relation to the aims and research questions that were addressed.

5.1. Work Domain Analysis (Aim 1)

Findings related to the domain model for information discovery, the nature of intelligence search, and supporting system requirements have the potential to generalize to other problem domains or research areas. Specifically, the intelligence search problem has parallels to the problem domain of academic search and involves concepts that are addressed in the field of information retrieval and relate to the wider concept of human information-seeking behavior. The domain modeling methodology and the CWA results could potentially be applied to new research in these areas, providing insights into how to support users in managing and navigating large volumes of text or multimedia data.

It is important to note that our study was conducted with a limited sample of two expert participants. These SMEs supplied process-specific data as a model for the intelligence searcher role in developing the ISW model and providing design feedback. While the findings provide valuable insights into the nature of intelligence search and supporting concepts that address the scope of this study, the small sample size necessitates caution in generalizing these results for direct application across all intelligence organizations. The specificity and in-depth nature of our study with these participants, however, do provide a foundational basis for understanding the complexities involved with intelligence search. Thus, this foundation may be useful when seeking to apply these findings within other intelligence organizations as a baseline model for validation through the application of a similar CSE methodology for the specific workforce of interest.

The FAN described in Section 3.1 answers RQ 1a by outlining the functional goals and processes for the ISW domain and RQ 2a through the relationships among those goals. Section 3.2 identifies and describes the challenged goal of information mapping, thus addressing RQ 1c along with the corresponding CWRs, IRRs, and a summary of domain insights that may influence effective supporting systems, which answer RQ 2a, RQ 2b, and RQ 3, respectively. Further analysis related to these questions provided in Section 4.1 addresses the implications of these results toward the overall objective of the research.

5.2. System Design Requirements (Aim 2)

The RDRs for information mapping provide the initial design specification for a work-supporting system. These requirements tailored to information mapping aim to assist searchers in identifying task-relevant documents and relationships that support conclusions with respect to a tasked research objective. The requirements generated to support the information mapping process are consistent with many elements of Elm and colleagues’ support function model for intelligence analysis. Interestingly, the information mapping process spans activities modeled across all three functions (down collect, conflict and corroboration, and hypothesis exploration) described by Elm and colleagues in the support function model for abductive intelligence analysis. By shifting our focus to intelligence search independent of the analysis and reporting processes, the primary work output or commodity is mission-aligned information. The functional goal of producing mission-aligned information is then to deliver task-relevant documents with an associated summary of search results, thus feeding the wider intelligence production process. This is in contrast with the challenging cognitive work dominating the intelligence analysis focus, where a more detailed synthesis of available information is performed in the effort to describe a single or set of required subjects represented in the information domain.

The visualizations described in Section 3.3 answer RQ 4a with the features required to realize conceptual spaces consistent with the ISW model. The DTD, also within Section 3.3, answers RQ 4b by aligning the CWRs and IRRs of information mapping to each proposed visualization. An analysis of the results related to system design in relation to the overall research objective is presented in Section 4.2.

5.3. Envisioned Solution (Aim 3)

The abstract concept of information mapping between the value structure stemming from the intelligence objective and the available document universe was selected as the basis from which to generate domain-specific support system requirements. The process of information mapping was analyzed to identify the information and relationships required within the cognitive work used to perform this mapping. Additional system requirements were developed in the form of representation requirements using ACWA. This resulted in a unique set of system requirements that support the goal of information mapping in intelligence search. Additionally, the concept of enabling Ambient Awareness of the information domain for the intelligence searcher can help tackle the problem of data overload.

Ambient Awareness is expected to tackle the dynamic nature of the value structure in intelligence search by allowing the searcher to identify and assess informative relationships in ‘missed’ documents. These informative relationships are based on the value structure, which governs the relevance and utility of the documents retrieved through search. Ambient Awareness is achieved by monitoring unseen documents for ‘hits’ as the value structure shifts due to insight gained by the team during the search. Modern AI methods have demonstrated strong natural language processing capabilities, and therefore, these AI methods should be capable of realizing the Ambient Awareness function in a search DSS team.

Future work should explore the potential for these findings to generalize as a means to leverage the intelligence domain perspective on the challenges and potential support for executing a rigorous search for information discovery purposes. Furthermore, an investigation into how these findings may align with general-purpose tools for information discovery across a wide range of information domains could add a unique perspective on this general problem. One approach could involve examining how the principles of domain modeling, search goal structuring, and information representation presented here can be adapted to an alternative or generalized work domain structure. Specifically, new research comparing information seeking behaviors and validating that the supporting concept addresses equivalent goals as indicated by the associated CWRs and IRRs would be able to suggest where these results and conclusions address more general aspects of search or are limited to intelligence search as reported.

The results and analysis Section 3.3, Section 3.4 and Section 4.2, respectively, address the envisioned solution and theory describing how proposed system design requirements may support decision making in the ISW. The proposed interaction between individual visualizations is addressed in Section 3.3, which answers RQ 5a. The desired system property described as “Ambient Awareness” in Section 3.4 answers RQ 5b by describing how the proposed system could address the identified challenges and support the information mapping goal. Finally, Section 4.2.1 provides an analysis of these results and summarizes the coordination proposed in the teaming DSS concept.

6. Conclusions

This CSE project identified a need to support intelligence searchers performing information mapping to locate and describe mission-relevant documents. The ecology of the intelligence search work domain is defined by the fundamental ways in which documents can be related to RIEs. The challenge is that these relationships have a semantic structure that varies with the intelligence objective as opposed to the static physical causal structure often found in law-driven domains. The expert intelligence searcher aims to gain an understanding of the ideal ‘value structure’ that can then be used to relate candidate documents to RIEs, or more generally, the search objective for every new objective. A static example of this was presented by Vicente [33] as the domain analysis and system design (the BookHouse) performed by Pejtersen for recommending books of fiction in a library setting. In this case, expert librarians provided details that revealed a static value structure (i.e., author intent, bibliographical elements, physical elements, etc.) that could be applied repeatedly to index and retrieve books that meet a customer’s needs. In the case of intelligence search, the value structure that governs success is often dynamic and is based on factors unique to each intelligence objective.

To better convey the potential benefits and functionalities of the proposed DSS, consider its application in conducting intelligence-objective-related searches, such as those required for in-depth assessments of an adversary’s capabilities. In the context of military intelligence, the value structure is intended to dynamically model various factors of interest, such as geopolitical developments, technological advancements, and shifts in alliances. The AI agent could therefore be programmed to recognize these factors and their impact on the intelligence objective within a vast collection of potential source documents. For example, if a foreign military develops a new type of missile system, the AI agent could automatically cue the intelligence searcher to documents describing this technology. This allows for real-time adaptation to additional intelligence objectives or newly emerging threats while increasing the coverage of available documents.

In such scenarios, the DSS can support the human operator in efficiently mapping out documents that relate to the new missile technology to address RIEs, such as evaluating its capabilities, identifying deployment locations, assessing potential countermeasures, etc. This supports the dual goals of producing intelligence that is both timely and thorough, thereby making the system a powerful enabler in the field of military intelligence.

The presented system design requirements and enabling concept can guide the development of more effective AI-agent-enabled search tools, leading to more efficient, reliable, and adaptive information search workflows. It can further serve as a basis for critically evaluating the effectiveness of these new tools, ensuring they truly support the analyst in executing intelligence searches by supporting the information mapping process. Finally, the realization of Ambient Awareness could be a powerful enabler of intelligence search by relieving some of the inherent conflict between the organizational goals of producing timely intelligence and producing thorough intelligence.