MacMillan, J., Paley, M.J., Levchuk, Y.N., Entin, E.E., Serfaty, D., and Freeman, J.T. (2002).

Designing the Best Team for the Task: Optimal Organizational Structures for Military Missions. In Mike McNeese, Ed Salas, and Mica Endsley (editors), New Trends in Cooperative Activities: System Dynamics in Complex Settings. San Diego, CA: Human Factors and Ergonomics Society Press.

Designing the Best Team for the Task: Optimal Organizational Structures for Military Missions
Jean MacMillan, Michael J. Paley, Yuri N. Levchuk, Elliot E. Entin, Daniel Serfaty, Jared T. Freeman Aptima, Inc. 1. Introduction Teams are not usually designed in a formal sense. Instead, team structures and individual roles for team members evolve over time, based on previous structures and roles, through an ad hoc process of trial, error, and adjustment. For military teams in recent years, however, a combination of rapidly evolving technology and frequently changing missions have created the need for more rapid and efficient ways to create team structures that take maximum advantage of the capabilities of technology for accomplishing mission goals. The influx of technology on the battlefield has altered the nature of military missions. Todays military missions are complex processes executed by networked individuals, supported by highly sophisticated hardware, all functioning in dynamic and uncertain environments. They require extensive communications, coordination, synchronization, and information management. This rapid development of advanced information technology and the resulting concepts of information-centric warfare demand changes to communication and collaboration at both the individual and organizational levels within the military. This changing environment has created the need for innovative methods for designing effective military teams. This chapter describes a breakthrough team design methoda systematic, formal, quantitative approach to designing a team that best fits the mission to be accomplished. The Team Integrated Design Environment (TIDE) is a tool set designed to support this method, enabling the quantitative definition of requirements for command teams operating in complex
2

mission environments. The TIDE methods and tools represent a powerful methodology to create novel organizational structures, based on operational mission variables, using quantitative methods. We know of no other methods that provide a similar formal framework for this type of design. This chapter explains what it means to design a team and describes the TIDE method for team design. Then it presents some initial empirical results that indicate that optimally designed teams can outperform teams that use more traditional organizational structures, and discusses how the team design process must be altered to focus on different concerns, depending on the nature of the team being designed and the environment in which that team must function. 2. What Does It Mean to Design a Team? The militarys need for effective teams has led to considerable progress in the last decade on methods for improving the performance of teams (see Serfaty, Entin, Deckert, and Volpe, 1993; Brannick, Salas, and Prince, 1997; Salas, Bowers, and Cannon-Bowers, 1995; Salas, Dickinson, Converse, and Tannenbaum, 1992; Swezey and Salas 1992). A useful product of this research has been the development of a shared definition of what constitutes a team. Salas, Dickinson, Converse, and Tannenbaum (1992) define a team as having the following characteristics: There is dynamic, interdependent, and adaptive interaction. There is a common goal, mission, or objective. There is some organizational structure of the team members. Each individual team member has specific tasks or functions. Task completion requires the dynamic interchange of information, the coordination of task activities, and constant adjustment to task demands.

The TIDE design approach produces a team in the sense that it is defined above. Based on the mission objectives for the team, we specify the specialized roles and functions of each team member, the information exchange and coordination interactions that must take place among the team members based on those roles and functions, and the organizational structure for the team. The focus of much prior team research has been on improving team performance through training to improve team competencies and through collaborative tool technology. However, as shown in Figure 1, there is a third major facet in understanding and improving team performancethe team structure. Research focused on collaborative support technology or team training and assessment usually takes the team structure as a given. Our work, in contrast, focuses on designing the best team structure for given set of goals and tasks (the teams mission), based on the application of optimization algorithms to a model that relates team structure to team performance.

Acquisition Support Technologies Assessment & Training Competencies Structures Modeling

Figure 1. Three Facets of Team Performance The process of designing a team structure is far more complex than simply specifying an organization chart or wiring diagram. The team structure specifies both the structure and the strategy of team, including who owns which resources, who takes which actions, who uses what

information, who coordinates with whom and the tasks about which they coordinate, and who communicates with whom. It includes role definitions for each of the team members as well as a specification of a command structure for the team. 3. The TIDE Approach to Team Design The TIDE approach to team design is model-based in the sense that it represents the mission, tasks, and functions to be accomplished by the team, the demands of those tasks and the resources required to accomplish them, the constraints on the team structure, and the performance goals for the team in a mathematical structure. This mathematical structure can then be manipulated to create a team design that is optimized for specified criteria. As illustrated in Figure 2, such a mathematical representation of a complex problem such as the accomplishment of a military mission, is, by necessity, a simplification of a complicated, messy, and uncertain world. As the saying goes, the map is not the territoryit is only a representation of that territory. The ultimate criteria for the usefulness of such a simplification is whether it produces answers to questions that are useful when they are fed back into the real world and put to use. A map is useful if it helps you get to your destination. A mathematical model of a mission is useful if it can be manipulated to produce a team structure that functions effectively for the mission for which it was designed.

Questions

Models Represent the World

Be cA e h ac B ah A

h ld o ba A e ch N rthro d o a g u dm e ro n in

md l e ica e cu tion va a

h ill se -m s a ine

H ill H ill

h old h ill

P rt o P rt o

Complex, messy, uncertain, dynamic

d ct & ete e in lim ate le dve icle a h

sup re p rt p ss o SM s A -site B g rid e B g rid e su re a o pp ss irp rt SM s A -site Be cB e ch a B a hB ho ld be chB a S u ro d o th a gu m ro nd ine A o irp rt A o irp rt

ba B e ch sea in -m es

*N vy S CJ in sce a (N rthA circa20 a /U M o t n rio o frica 05)

The World

Answers

The map is not the territory.

Manipulations to generate designs

Figure 2. The Model-Based Design Problem Three-Part Allocation for Team Design Team design requires, in essence, the specification of who does what when. The central thesis of our team-design method is that a set of interdependent, interrelated tasks that must be completed under time constraints has an underlying quantitative structure that can be exploited to design the best team for accomplishing those tasks. At the core of our method is a systems engineering approach that describes organizational performance criteria as a multi-variable objective function to be optimized. This approach is based on a three part allocation model, presented in Figure 3, that considers: 1) the tasks that must be accomplished and their interrelationships (the mission); 2) the external resources needed to accomplish those tasks (e.g., information, raw materials, or equipment), and 3) the human decision makers who will constitute the team. The team design process is, in simplest terms, an algorithm-based allocation between these three parts.

Mission

Tasks What it takes to complete Substitutability Resources Who owns what & Who knows what Who does what

Decision Makers

Who talks to whom

Figure 3. Three Part Allocation Model for Team Design First, a quantitative model describing the mission and the existing organizational constraints is built. Then, one or more objective functions for the design are specified. Finally, an organization is designed to optimize the objective function(s). When the objective function includes several non-commensurate criteria, the organizational design problem is treated as a multi-objective optimization problem. The power of quantitative modeling lies in describing a great variety of phenomena underlying the structure of a mission and of an organization by a relatively limited set of fundamental elements, parameters, variables, laws, and principles. These laws and principles, especially, specify the functional interdependencies among the structural elements and the dynamics of system parameters and variables. The algorithms that are fundamental to this team design method (Levchuk, Pattipati, and Kleinman, 1998) were originally developed under the sponsorship of the Office of Naval Research for the Adaptive Architectures for Command and Control (A2C2) program (Serfaty, 1996).

Inputs from Subject Matter Experts Our team design method is algorithm-based, but it relies on heuristics and on the judgment of subject matter experts to frame the design problem in a meaningful way, including decomposing an overall mission (or goal) into specific tasks, specifying the relationships between tasks, specifying the resources needed to complete the tasks, and specifying the criteria to be optimized for the team. Subject matter experts in the area of application are also needed to review and revise the organization and structures suggested by the model. The design method is iterative. Typically, review of the team designs suggested by the algorithms reveals adjustments and corrections to be made in the task structure, the organizational constraints, or the optimization criteria. The team-design methodology is goal- or mission-driven. That is, the model uses a detailed scenario that specifies the tasks required to accomplish a goal and the resources available to accomplish those tasks, and uses algorithms to optimally allocate these tasks and resources to team members to create an organizational structure for best accomplishing the goal. To capture the operational elements in a scenario, we rely on expert insight from subject-matter experts who develop scenarios. The interaction between operational experts and modeling specialists at this stage is essential for the design process. In addition to the selection or development of a scenario (or multiple scenarios), it is necessary to create a detailed model of the mission that serves as the input for the method. An essential question that underlies all organizational design processes is Who does what? This requires that a mission be described in terms of its tasks (the what independent of the who). There are multiple ways to decompose a mission, and this process relies on interaction between

the designer and domain experts. Mission analysis, functional decomposition, and subsequent function allocation must be driven by design goals. After multi-dimensional task decomposition is used to identify mission elements, specific modeling techniques are applied to capture the internal structure of the mission. The mission decompositions are used to define parallelism, sequence, and structure for the mission tasks. These task interdependencies are used to create a hierarchical structure among mission tasks which is represented by a mission task dependency graph. There are two major inputs for the team design method, the quantitative mission structure just described, and a set of organizational constraints. Organizational constraints include the specific resources and technologies available for accomplishing the tasks as well as any restrictions on how tasks are assigned to team members, based on specifications by subject matter experts who understand the domain of application. Team size may be set as an organizational constraint, or allowed to vary as part of the optimization. 4. Steps in the Design Process Figure 4 shows the steps followed in a typical team design process. The first stage is mission representation, which depends heavily on inputs from subject matter experts. At this stage, we define the tasks that must be completed in order to accomplish the mission and specify their interdependencies. Tasks may be triggered by events (e.g., the appearance of a new air track triggers the task of identifying that track) or they may be triggered by other tasks (e.g., once a track is identified, its intent must be evaluated). Still other tasks are on-going, independent of events or other tasks (e.g., the need to continually monitor for new tracks). Typically we work with one or many mission scenarios in designing the team. If possible, we develop a stochastic

mission model, which specifies the scenario in terms of the probabilities of various events occurring, rather working from a single deterministic scenario.
Event to Task Mapping & Stochastic Mission Model Optimized Task Schedules Definition of Individual Roles & Info Requirements. Definition of Team Comms & Coordination Team Design (Structures + Processes)

Mission Representation

Task Scheduling Clustering Tasks into Roles

Design Objectives & Constraints

Decomposition of Role Overlap

Organizational Structuring

Figure 4. Steps in Designing a Team At the mission representation stage we define attributes for the tasks to be accomplished. The task attributes of greatest interest will vary depending on the nature of the team design problem (as discussed in more detail below), but typical attributes that are considered include the workload associated with the task, the time needed to complete the task, the information needed to accomplish the task, and the communication/coordination links that exist among tasks due to the nature of the work being performed (e.g., the planning of air-toground strikes requires coordination with the planning of ground troop movements). At the mission representation stage, we also specify the resources that could be used to accomplish the task, if the problem is resource constrained. Resources include, for some types of teams, assets such as sensor or weapons systems. Some types of assets can only be used at one place at one time (e.g., an artillery unit) while other types of assets can be used simultaneously

10

by many people in many locations (e.g., information). Depending on the domain of application, there may be multiple ways to accomplish the same task with different combinations of assets (e.g., ships, amphibious units, and aircraft may all be involved in a mission task such as take the beach). If there are multiple ways to accomplish a task, we specify (based on subject matter expert input) the relative effectiveness of each of the possible combinations of assets for accomplishing the task. The nest step in team design is task scheduling. This step is accomplished by algorithms that determine the optimal way to use the available assets to accomplish the tasks given an overall objective, e.g., to minimize the time needed to accomplish the mission or to maximize mission effectiveness. The importance of the task scheduling step in team design depends on the nature of the mission domain. If there are a number of assets that can only be used in one place at one time, and a number of different ways that assets can be combined to accomplish tasks, this step may be extremely important in team design. In contrast, if there is relatively little competition for assets, or only one way to accomplish a task with those assets, then task scheduling is not a major factor in the design of the team. The output of this step in the design process is an optimized task schedule for using the available assets to accomplish the mission. At this stage, human roles have not yet been considered. The next step, and the central one for team design, is to create roles for individuals by clustering tasks (and the resources needed to accomplish them) in such a way as to optimize an objective function. Task clustering is often done on the basis of two (potentially competing) criteria: the goal of equalizing workload across the team members, and the goal of minimizing the amount of communication/coordination required between team members. The tension between these two criteria can been seen from a simplified example: the best way to minimize

11

the need for coordination is to assign all of the tasks to one individual, but this obviously directly contradicts the goal of equalizing the workload across the team. While the goal of equalizing workload (or keeping workload below a tolerable threshold) is a relatively intuitive one, the goal of minimizing the need for coordination requires further explanation. It is not that coordination is, in itself, bad. However, if communication is required in order to achieve that coordination, then that communication takes up the time and attention of team members. Therefore, the need to coordinate through communication can have a negative effect on performance in conditions where there is a high task load (i.e., workload imposed from outside the team). While it is always good to have information about what other members of the team are doing, there may be a cost to acquiring that information. Communication can be good or bad for team performance, depending on when it occurs and what else is going on at that time. Team theory suggests that if individuals on a team have a good mental model of what each of the other team members is doing and a good shared mental model of the situation, then this mental model allows them to anticipate the needs of the other team members (MacIntyre, Morgan, Salas, and Glickman, 1988; Cannon-Bowers, Salas and Converse, 1990; Kleinman and Serfaty, 1989; Orasanu, 1990). This mental model can be acquired through communication and planning during periods of low workload (heres how we are going to handle it when) (Orasanu, 1990) or through cross training (each team member receives training in the others job) (Travillian, Volpe, Cannon-Bowers, and Salas, 1993; Baker, Salas, Cannon-Bowers, and Spector, 1992) or simply through experience. In periods of high workload, these mental models allow members of the team to anticipate the needs of other team members so that they can coordinate implicitly (with less

12

need for communication) rather than coordinating explicitly (requiring communication of the form send me this or do this now). Implicit coordination reduces the need for communication under high task load, freeing team members up to do other things, and causing the team to perform better (Serfaty, Entin, and Volpe, 1993; Serfaty, Entin, and Johnston, 1998). So, it is not that either coordination or communication is bad, it is just that, especially under stress, teams often perform better if they can coordinate without the need for frequent communication. For team design, assigning tasks to minimize the need for coordination (to the extent possible, without overloading any of the team members) reduces the amount of knowledge the team members need to have about each others roles, and the amount they need to communicate. This is most critical, and probably will have the most effect on performance, when the team is in high stress conditions. The product of the clustering step in team design is to define roles for individuals in terms of the tasks for which they will be responsible. Associated with those roles, based on the attributes of the tasks, is a specification of the information that will be used by each team member, the resources that each individual will control in order to accomplish the tasks, and the need for coordination among team members (based on the interdependencies of tasks). Another product of the clustering is a prediction of each individuals workload over time, based on the tasks assigned to that individual and the timing of the tasks in the mission scenario. Note that if workload is a major concern for the team design, we also include an estimate of the overhead workload associated with managing multiple tasks simultaneously. The results of the clustering step must be fed back into the optimized task schedule to determine if that schedule is feasible given the assignment of tasks to individuals. We might

13

discover, for example, that the optimal schedule requires an individual to accomplish too many tasks simultaneously, and will therefore need to delay tasks or to change the task assignments as a result. For some team designs, it will be possible to assign tasks to individual team members in such a way that no one team member is overloaded. For other teams, such an assignment may not be possible, and it may be necessary to assign the same task to multiple individuals, creating an overlap in task responsibilities. If so, this creates a need for communication and coordination among the individuals with overlapping responsibilities, which must then be factored back into calculations of the workload for each of the affected team members. The final step in the design process, once individual roles have been defined, is the specification of an organizational structure (e.g., a command hierarchy) for the team. For military teams, this is usually straightforward, driven primarily by the need to designate a team commander. The workload associated with being the team commander must also be fed back into the workload calculations, however, to ensure that command responsibility has not been placed on an individual who is already at a maximum workload ceiling. The final output of the team design process is a specification of both a team structure and a team process associated with that structure. The team design specifies which team member (or members) accomplish each task, what resources are controlled by each team member, what information is used by each team member, and who needs to coordinate with whom (and about what). Depending on the criteria used to optimize the team and the attributes defined for the tasks, the final design can also produce predictions about the teams performance and the workload that will be experienced by each of the individuals on the team.

14

5.

Experimental Evaluation of Team Designs The ultimate test of the model-based optimal team design method is the performance of

the teams that have been designed using this method. Initial empirical evidence is available from the Adaptive Architectures for Command and Control (A2C2) program (Serfaty, 1996) on the effectiveness of model-based team design. In the A2C2 program, innovative mission-based Joint Task Force (JTF) team structures were designed using the model-based optimization method. As a comparison, a group of subject matter experts also generated team structures for the same JTF mission. The two team structures were played out in a simulation-based experiment, with 10 sixperson teams of military officers from the Naval Postgraduate School in Monterey (Entin, 1999). Each team participated under both architectures, with the order counterbalanced to control for learning effects. Figure 5 shows the results of the experiment. Two types of summary performance measures are shown: simulation-based measures, which come directly from the simulation testbed, and observer-based measures, which were prepared by subject matter expert observers rating team behavior during the experiment sessions. For both types of performance measures, the performance of the six-person team designed using the model-based method was superior to the performance of the six-person team using a more traditional team structure developed by subject matter experts. The model-based method was also used to design a four person team, shown as model reduced in Figure 5. The performance of this four-person model-based team was at the same level as (not significantly different from) the performance of the six-person team designed by the experts.

15

Design
6-person

79.7 85.1 76.2

Expert Modelbased Modelreduced 100 50 60

59.7 78.1 68.5

6-person

4-person

50

60

70

100

Figure 5. Performance in optimized versus traditional (designed by subject matter experts) team structures. The optimized team was designed to reduce the need for communication and coordination among team members, and the results in Figure 6 show that it was successful in this objective. The six-person optimized team achieved higher performance levels with fewer coordination actions and a lower communication rate. The six-person optimized team also had a higher anticipation ratio. This anticipation ratio measures the ratio of information transfers over requests for information. Higher values of the anticipation ratio indicate that team members were pushing information without having to be asked, also indicating more effective coordination (i.e., coordination with less communication). The innovative team structures developed using the optimal design method resulted in superior performance only if the teams were thoroughly trained in the new team structure prior to using that structure in the experiment, however. Earlier experiments (Entin, Serfaty, and Kerrigan, 1998) in which subjects received less training on the innovative team structures failed to find significant differences between model-based and traditional team designs.

16

Observers Overall Rating

Overall Mission Outcome

-- Model-based architectures required less team communications -- Engineered capabilities at each command node economize need for wasteful inter-node coordination -- Better and more timely use of communication channels supports the teams anticipatory behavior (performance predictor)
Coordination Actions [/min]
3

Comm Rate [msgs/min]

Findings

8 7 6 5 4 3 2 1 0

Ad-hoc (6)

Model-based (6) Model-based (4)

Anticipation Ratio
Ad-hoc (6) Model-based (6) Model-based (4)

2.5 2 1.5 1 0.5 0

4 3 2 1 0

Ad-hoc (6)

Model-based (6) Model-based (4)

Figure 6. Communication and coordination measures for model-based and expertdesigned team structures An interesting feature of the JTF team designs produced by the model-based method was that the algorithms tended to push the jointness of the control of resources down to much lower levels in the command structure than is current military practice (e.g., a lower-level commander might control both Navy and Air Force resources). Although the military domain experts working on the project commented that the expertise to handle this combination of resources does not currently exist at lower levels of command, they admitted that such an organization would probably be more efficient than current practice. Overall, the results of the A2C2 experiments indicate that the optimized, model-based team design method can produce innovative team structures in which teams can perform t a higher level than they do under more traditional structures. This improved performance is observed only if teams receive sufficient training in how to function in the new structures, however.

17

6.

Design Focus by Domain of Application and Type of Team We are currently engaged in applying the TIDE team design approach described in this

chapter in a number of different military domains. Each domain presents different challenges for team design, and requires adaptation of the method and emphasis on different aspects of the design process. Joint Task Force (JTF) Command Team. A primary issue for the design of JTF command teams (see results above) is the control of resources. In the mission being analyzed, the JTF teams orchestrated the use of Navy, Air Force, Marine, and Army resources (ships, planes, infantry units, satellite sensors, etc.) to recapture a port that was being occupied by the enemy. Many of the tasks depended on the success of the previous task (e.g., advance to the airport could not be initiated until take the beach was accomplished). There were often a number of ways in which a particular task could be accomplished with the available resources, but a resource being used in one geographical area could not be used immediately in another. For this application, the optimal (requiring least coordination) control of resources was a driving factor for the design, leading to the creation of team structures in which each team member directly controlled many if not all of the resources needed to accomplish his or her tasks. Note that the model-based approach produced team designs that are quite different from traditional JTF designs, with joint control of Navy/Army/Air Force assets at much lower levels of the command hierarchy than is currently the case. Next Generation Navy Surface Ships Command Team. For this application, the goal is to design the next generation of Navy ships to take advantage of automation and to operate with a much smaller crew than is currently required. The goal is to reduce the number of individuals needed in the shipboard command center by half, from 20 or more to approximately 10. In this

18

application, the control of scarce and geographically dispersed resources is not the driving issue for team design, as it was for JTF team design. The major resource needed by the shipboard command team is information, which can be made available to everyone simultaneously with the planned technology. The primary concern for this team is balancing workload in order to keep workload below a manageable threshold for all team members. The fundamental question is: Can 10 people, aided by technology, handle a mission that previously required 20? For this design effort, we are working with more detailed workload data and developing new methods for modeling workload, including methods for calculating the workload effects of multi-tasking. AWACS Command and Control Team. Teams on board Air Force AWACS planes direct air traffic and monitor for hostile aircraft from an airborne command center. Because the team is airborne, and must fit into limited space, the number of crew positions needed is a critical concern. With the introduction of new sensor technology, some of the tasks previously performed by the crew will be automated. The primary issue for this team redesign problem is how the responsibilities of the team members should be reallocated now that some tasks have been eliminated, and whether it may be possible to reduce the number of positions needed on board the aircraft. Uninhabited Air Vehicle Combat Operations Center Team. Current uninhabited air vehicles (UAVs) require a team of multiple operators on the ground to control one UAV in the air. Future concepts call for a reversal of this ratio, with a small team of operators on the ground controlling many UAVs in the air. Our focus is the design of roles for the ground controller team. Preliminary analysis shows that the major problem for designing this team involves the sequencing of waves of aircraft and the patterns in which the aircraft will be flown. The workload associated with the control of the UAVs varies enormously at various stages in the

19

UAVs flight. The challenge will be to develop deployment patterns for the UAVs that do not result in the creation of infeasible workload peaks for the team in the control center. Air Operations for Time Critical Targets (JFACC) Team. A theater-level air campaign such as the one just conducted in the Balkans requires the generation and execution of Air Tasking Orders (ATOs), typically on a daily basis. These ATOs specify targets as well as the aircraft and weapons to be used to strike these targets. Difficulties arise when new target information is received, however, or when some aspect of the plan proves unworkable (e.g., a tanker that was scheduled to perform airborne refueling has mechanical problems and must return to base). In these situations, the speed with which the JFACC air operations organization can respond to new information, modify plans, and execute those new plans, becomes critical. In previous operations, the time needed to strike a time critical target (e.g., a SCUD launcher not likely to remain in position for very long) was too long for effective action. A critical concern in developing new architectures for the JFACC is therefore the speed of response of the organization. Workload is not a primary concern. Instead, the focus is on optimizing the organization for quick reaction to changing information. 7. Conclusions The TIDE model-based method for optimal team design has shown promise for generating innovative team structures that can provide insight into how military organizations can best take advantage of changes in technology. With the enormous increases in network capability, many tasks in an organization can be done in almost any location, although some are still geographically constrained. The TIDE approach provides tools for working with subject matter experts in a domain to specify the tasks that must be accomplished, then producing

20

optimized structures for accomplishing those tasks. A major advantage of the approach is that it is not necessarily constrained by how things are done now, and can generate new ideas and new approaches. While these ideas may not be workable for a variety of practical reasons (e.g., the training costs for a new position may be too great), they provide a innovative starting point for rethinking military team structures. Initial empirical evidence indicates that the model-based approach has value for the Joint Task Force domain. Considerably more research and empirical testing is needed in other domains. Also, the applicability of the approach to the redesign of organizational structures in nonmilitary environments should be explored. Commercial organizations face many of the same problems as the military in adapting their organizational structures to take advantage of new technologies. The less-structured nature of many commercial missions and tasks may present a challenge for the method, however. References Baker, C.V., Salas, E., Cannon-Bowers, J.A., & Spector, P. (1992). The effects of interpositional uncertainty and workload on team coordination skills and task performance. Presented at the annual meeting of the Society for Industrial and Organizational Psychology, Montreal, Canada. Brannick, M.T., Salas, E., & Prince, C. (Eds.) (1997). Team Measurement and Performance: Theory, Methods, and Applications, Mahway, NJ: Lawrence Erlbaum Associates. Cannon-Bowers, J.A., Salas, E., & Converse, S. (1990). Cognitive psychology and team training: Training shared mental models of complex systems. Human Factors Bulletin, 33 (12), 1-4.

21

Entin, E.E. (1999). Optimized Command and Control Architectures for Improved Process and Performance. Proceedings of the 1999 Command and Control Research and Technology Symposium, Newport, RI. Entin, E.E., Serfaty, D. & Kerrigan, C.K. (1998). Choice and performance under three command and control architectures. Proceedings of the 1998 Command and Control Research and Technology Symposium, Monterey, CA. Kleinman, D. L. and Serfaty, Daniel (1989). Team Performance assessment in distributed decision-making. Proceedings of the Symposium on Interactive Networked Simulation for Training, pp. 22-27, Orlando, FL. Levchuk, Y., Pattipati, C., and Kleinman, D. (1998). Designing Adaptive Organizations to Process a Complex Mission: Algorithms and Applications. Proceedings of the 1998 Command and Control Research and Technology Symposium (11-32) Naval Postgraduate School, Monterey, CA. McIntyre, R. M., & Salas, E. (1995). Team performance in complex environments: What we have learned so far. In R. Guzzo & E. Salas (Eds.), Team effectiveness and decision making in organizations (9-45). San Francisco: Jossey-Bass. Orasanu, J. M. (1990). Shared Mental Models and Crew Decision Making, CSL Report 46. Princeton, NJ: Cognitive Science Laboratory, Princeton University. Salas, E., Bowers, C.A., & Cannon-Bowers, J.A. (1995). Military team research: 10 years of progress. Military Psychology, 7, 5575.

22

Salas, E., Dickinson, T. L., Converse, S. A. and Tannenbaum, S. I. (1992) Toward and Understanding of Team Performance and Training in Teams: Their Training and Performance, Eds. Robert W. Swezey and Eduardo Salas, Ablex Publishing Company, Norwood, NJ. Serfaty, D. (1996). Adaptive Architectures for Command and Control (A2C2): An Overview. Proceedings of the 1996 International Command and Control Research and Technology Symposium (272-276) NDU, Washington, DC. Serfaty, D., Entin, E. E., & Johnston, J. H. (1998). Team coordination training. In J. A. CannonBowers & E. Salas (Eds.), Making decisions under stress: Implications for individual and team training (pp. 221-245). Washington, D. C.: American Psychological Association. Serfaty, D., Entin, E.E., & Deckert, J.C., & Volpe, C (1993). Implicit coordination in command teams. In Proceedings of the 1993 Symposium on Command and Control Research, NDU, Washington, D.C, 53-57. Swezey, R.W. & Salas, E. (Eds.) (1992) Teams: Their Training and Performance, Ablex Publishing Company, Norwood, NJ. Travillian, K.K., Volpe, C.E., Cannon-Bowers, J.A., & Salas, E. (1993). Cross training highly interdependent teams: Effects on team processes and team performance. Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting, 1243-1247.

23