CRITICAL SUCCESS FACTORS

for Crowdsourcing

with Virtual Environments

TO UNLOCK INNOVATION

Glenn E. Romanczuk, Christopher Willy, and John E. Bischoff

Senior defense acquisition leadership is increasingly advocating new

approaches that can enhance defense acquisition. Their constant refrain is
increased innovation, collaboration, and experimentation. The then Under
Secretary of Defense for Acquisition, Technology, and Logistics Frank Kendall,
in his 2014 Better Buying Power 3.0 White Paper, called to “Incentivize inno­
vation … Increase the use of prototyping and experimentation.” This article
explores a confluence of technologies holding the key to faster development
time linked to real warfighter evaluations. Innovations in Model Based
Systems Engineering (MBSE), crowdsourcing, and virtual environments
can enhance collaboration. This study focused on finding critical success
factors, using the Delphi method, allowing virtual environments and MBSE
to produce needed feedback and enhance the process. The Department of
Defense can use the emerging findings to ensure that systems developed
reflect stakeholders’ requirements. Innovative use of virtual environments
and crowdsourcing can decrease cycle time required to produce advanced
innovative systems tailored to meet warfighter needs.

DOI: https://doi.org/10.22594/dau.16 -758.24.02 (Online only)

Keywords: Delphi method; collaboration; innovation; expert judgment

 Image designed by Diane Fleischer

Crowdsourcing with Virtual Environments http://www.dau.mil

A host of technologies and concepts holds the key for reducing devel­
opment time linked to real warfighter evaluation and need. Innovations in
MBSE, networking, and virtual environment technology can enable collabo­
ration among the designers, developers, and end users, and can increasingly
be utilized for warfighter crowdsourcing (Smith & Vogt, 2014). The inno­
vative process can link ideas generated by warfighters, using game-based
virtual environments in combination with the ideas, ranking, and filtering
of the greater engineering staff. The DoD, following industry’s lead in crowd-
sourcing, can utilize the critical success factors and methods developed in
this research to reduce the time needed to develop and field critical defense
systems. Innovative use of virtual environments and crowdsourcing can
increase the usefulness of weapon systems to meet the real needs of the true
stakeholders—the warfighters.

The DoD, as a whole, has begun looking for efficiency by employing inno­
vation, crowdsourcing, MBSE, and virtual environments (Zimmerman,
2015). Industry has led the way with innovative use of crowdsourcing for
design and idea generation. Many of these methods utilize the public at
large. However, this study will focus on crowdsourcing that uses warfight­
ers and the larger DoD engineering staff, along with MBSE methodologies.
This study focuses on finding the critical success factors, or key elements,
and developing a process (framework) to allow virtual environments and
MBSE to continually produce feedback from key stakeholders throughout
the design cycle, not just at the beginning and end of the process. The pro­
posed process has been developed based on feedback from a panel of experts
using the Delphi method. The Delphi method, created by RAND in the 1950s,
allows for exploration of solutions based on expert opinion (Dalkey, 1967).
This study utilized a panel of 20 experts in modeling and simulation (M&S).
The panel was a cross section of Senior Executive Service, senior Army,
Navy, and DoD engineering staff, and academics with experience across the
range of virtual environments, M&S, MBSE, and human systems integra­
tion (HSI). The panel developed critical success factors in each of the five
areas explored: MBSE, HSI, virtual environments, crowdsourcing, and the
overall process. HSI is an important part of the study because virtual envi­
ronments can enable earlier detailed evaluation of warfighter integration
in the system design.

Many researchers have conducted studies that looked for methods to make
military systems design and acquisition more fruitful. A multitude of stud­
ies conducted by the U.S. Government Accountability Office (GAO) has also
investigated the failures of the DoD to move defense systems from the early
stages of conceptualization to finished designs useful to warfighters. The

336 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

GAO offered this observation: “Systems engineering expertise is essential

throughout the acquisition cycle, but especially early when the feasibility
of requirements are [sic] being determined” (GAO, 2015, p. 8). The DoD
process is linked to the systems engineering process through the mandated
use of the DoD 5000-series documents (Ferrara, 1996). However, for many
reasons, major defense systems design and development cycles continue
to fail, major programs are canceled, systems take too long to finish, or
costs are significantly expanded (Gould, 2015). The list of DoD acquisition
projects either canceled or requiring significantly more money or time to
complete is long. Numerous attempts to redefine the process have fallen
short. The DoD has, however, learned valuable lessons as a result of past
failures such as the Future Combat System, Comanche, Next Generation
Cruiser CG(X), and the Crusader (Rodriguez, 2014). A partial list of those
lessons includes: the need for enhanced requirements generation, detailed
collaboration with stakeholders, and better systems engineering utilizing
enhanced tradespace tools.

Innovative use of virtual environments and

crowdsourcing can increase the usefulness of
weapon systems to meet the real needs of the true
stakeholders—the warfighters.

The DoD is now looking to follow the innovation process emerging in indus­
try to kick-start the innovation cycle and utilize emerging technologies to
minimize the time from initial concept to fielded system (Hagel, 2014). This
is a challenging goal that may require significant review and restructuring
of many aspects of the current process. In his article “Digital Pentagon,”
Modigliani (2013) recommended a variety of changes, including changes
to enhance collaboration and innovation. Process changes and initiatives
have been a constant in DoD acquisition for the last 25 years. As weapons
have become more complex, software-intensive, and interconnected, DoD
has struggled to find the correct mix of process and innovation. The DoD
acquisition policy encourages and mandates the utilization of systems
engineering methods to design and develop complex defense systems. It is
hoped that the emergence of MBSE concepts may provide a solid foundation
and useful techniques that can be applied to harness and focus the fruits of
the rapidly expanding innovation pipeline.

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 337

Crowdsourcing with Virtual Environments http://www.dau.mil

The goal and desire to include more M&S into defense system design and
development has continually increased as computer power and software
tools have become more powerful. Over the past 25 years, many new efforts
have been launched to focus the utilization of advanced M&S. The advances
in M&S have led to success in small pockets and in selected design efforts,
but have not diffused fully across the entire enterprise. Several different
process initiatives have been attempted over the last 30 years. The acquisi­
tion enterprise is responsible for the process, which takes ideas for defense
systems; initiates programs to design, develop, and test a system; and then
manages the program until the defense system is in the warfighters’ hands.
A few examples of noteworthy process initiatives are Simulation Based
Acquisition (SBA); Simulation and Modeling for Acquisition, Requirements,
and Training (SMART); Integrated Product and Process Development
(IPPD); and now, Model Based Systems Engineering (MBSE) and Digital
Engineering Design (DED) (Bianca, 2000; Murray, 2014; Sanders, 1997;
Zimmerman, 2015). These process initiatives (SBA, SMART, and IPPD)
helped create some great successes in DoD weapon systems; however, the
record of defense acquisition and the amount of time required to develop
more advanced and increasingly complex interoperable weapon systems
has been mixed at best. The emerging MBSE and DED efforts are too new
to fully evaluate their contribution.

As weapons have become more complex, software-

intensive, and interconnected, DoD has struggled to
find the correct mix of process and innovation

The Army’s development of the Javelin (AAWS-M) missile system is an

interesting case study of a successful program that demonstrated the abil­
ity to overcome significant cost, technical, and schedule risks. Building
on design and trade studies conducted by the Defense Advanced Research
Projects Agency (DARPA) during the 1970s and utilizing a competitive
prototype approach, the Army selected an emerging (imaging infrared
seeker) technology from the three technology choices proposed. The inno­
vative Integrated Flight Simulation, originally developed by the Raytheon/
Lockheed joint venture, also played a key role in Javelin’s success. The final
selection was heavily weighted toward “fire-and-forget” technology that,
although costly and immature at the time, provided a significant benefit

338


Defense



ARJ, April 2017, Vol. 24 No. 2 : 334–367
 April 2017

to the warfighter (David, 1995; Lyons, Long, & Chait, 2006). This is a rare
example of warfighter input and unique M&S efforts leading to a successful
program. In contrast to Javelin’s successful use of innovative modeling and
simulation is the Army’s development of Military Operations on Urbanized
Terrain (MOUT) weapons. In design for 20 years, and still under develop­
ment, is a new urban shoulder-launched munition for MOUT application
now called the Individual Assault Munition (IAM). The MOUT weapon
acquisition failure was in part due to challenging requirements. However,
the complex competing technical system requirements might benefit from
the use of detailed virtual prototypes and innovative game-based war-

The record of defense acquisition and the amount

of time required to develop more advanced and
increasingly complex interoperable weapon systems
has been mixed at best.

fighter and engineer collaboration. IAM follows development of the Army’s

Multipurpose Individual Munition (MPIM), a program started by the
Army around 1994 and canceled in 2001. Army Colonel Richard Hornstein
indicates that currently, after many program changes and requirements
updates, system development of IAM will now begin again in the 2018
timeframe. However, continuous science and technology efforts at both
U.S. Army Armament Research, Development, and Engineering Center
(ARDEC) and U.S. Army Aviation and Missile Research, Development,
and Engineering Center (AMRDEC) have been maintained for this type
of system. Many of our allies and other countries in the world are actively
developing MOUT weapons (Gourley, 2015; Jane’s, 2014). It is hoped that by
using the framework and success factors described in this article, DoD will
accelerate bringing needed capabilities to the warfighter, using innovative
ideas and constant soldier, sailor, and airman input. With the changing
threat environment in the world, the U.S. military can no longer allow
capability gaps to be unfilled for 20 years or just wait to purchase similar
systems from our allies. The development of MOUT weapons is an applica­
tion area that is ripe for the methods discussed in this article. This study
and enhanced utilization of virtual environments cannot correct all of the
problems in defense acquisition. However, it is hoped that enhanced utili­
zation of virtual environments and crowdsourcing, as a part of the larger

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 339

Crowdsourcing with Virtual Environments http://www.dau.mil

effort into Engineered Resilient Systems (ERS) and expanded tradespace

tools, can provide acquisition professionals innovative ways to accelerate
successful systems development.

BACKGROUND
Literature Review
This article builds upon detailed research by Murray (2014); Smith and
Vogt (2014); London (2012); Korfiatis, Cloutier, and Zigh (2015); Corns and
Kande (2011); and Madni (2015) that covered elements of crowdsourcing,
virtual environments, gaming, early systems engineering, and MBSE. The
research study described in this article was intended to expand the work
discussed in this section and determine the critical success factors for using
MBSE and virtual environments to harvest crowdsourcing data from war-
fighters and stakeholders, and then provide that data to the overall Digital
System Model (DSM). The works reviewed in this section address virtual
environments and prototyping, MBSE, and crowdsourcing. The majority
of these are focused on the conceptualization phase of product design.
However, these tools can be used for early product design and integrated
into the detailed development phase up to Milestone C, the production and
deployment decision.

Many commercial firms and some government agencies have studied the
use of virtual environments and gaming to create “serious games” that
have a purpose beyond entertainment (National Research Council [NRC],
2010). Commercial firms and DARPA have produced studies and programs
to utilize an open innovation paradigm. General Electric, for one, is com­
mitted to “crowdsourcing innovation—both internally and externally …
[b]y sourcing and supporting innovative ideas, wherever they might come
from…” (General Electric, 2017, p. 1).

Researchers from many academic institutions are also working with open
innovation concepts and leveraging input from large groups for concept
creation and research into specific topics. Dr. Stephen Mitroff of The George
Washington University created a popular game while at Duke University
that was artfully crafted not only to be entertaining, but also to provide
researchers access to a large pool of research subjects. Figure 1 shows a
sample game screen. The game allows players to detect dangerous items
from images created to look like a modern airport X-ray scan. The research
utilized the game results to test hypotheses related to how the human brain
detects multiple items after finding similar items. In addition, the game
allowed testing on how humans detect very rare and dangerous items. The

340 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

game platform allowed for a large cross section of the population to interact
and assist in the research, all while having fun. One of the keys to the use­
fulness of this game as a research platform is the ability to “phone home”
or telemeter the details of the player-game interactions (Drucker, 2014;
Sheridan, 2015). This research showed the promise of generating design
and evaluation data from a diverse crowd of participants using game-based
methods.

FIGURE 1. AIRPORT SCANNER SCREENSHOT

Note. (Drucker, 2014). Used by permission Kedlin Company.

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 341

Crowdsourcing with Virtual Environments http://www.dau.mil

Process
Several examples of process-related research that illustrates begin­
ning inquiry into the use of virtual environments and MBSE to enhance
systems development are reviewed in this section. Marine Corps Major
Kate Murray (2014) explored the data that can be gained by the use of a
conceptual Early Synthetic Prototype (ESP) environment. The envisioned
environment used game-based tools to explore requirements early in the
design process. The focus of her study was “What feedback can be gleaned,
and is it useful to decision makers?” (Murray, 2014, p. 4). This innovative
thesis ties together major concepts needed to create an exploration of design
within a game-based framework. The study concludes that ESP should be
utilized for Pre-Milestone A efforts. The Pre-Milestone A efforts are domi­
nated by concept development and materiel solutions analysis. Murray also
discussed many of the barriers to fully enabling the conceptual vision that
she described. Such an ambitious project would require the warfighters to be
able to craft their own scenarios and add novel capabilities. An interesting
viewpoint discussed in this research is that the environment must be able to
interest the warfighters enough to have them volunteer their game-playing
time to assist in the design efforts. The practical translation of this is that
the environment created must look and feel like similar games played by the
warfighters, both in graphic detail and in terms of game challenges to “keep
… players engaged” (Murray, 2014, p. 25).

Corns and Kande (2011) describe a virtual engineering tool from the
University of Iowa, VE-Suite. This tool utilizes a novel architecture, includ­
ing a virtual environment. Three main engines interact: an Xplorer, a
Conductor, and a Computational engine. In this effort, Systems Modeling
Language (SysML) and Unified Modeling Language (UML) diagrams are
integrated into the overall process. A sample environment is depicted sim­
ulating a fermentor and displaying a virtual prototype of the fermentation
process controlled by a user interface (Corns & Kande, 2011). The extent
and timing of the creation of detailed MBSE artifacts, and the amount of
integration achievable or even desirable among specific types of modeling
languages—e.g., SysML and UML—are important areas of study.

In his 2012 thesis, Brian London described an approach to concept creation

and evaluation. The framework described utilizes MBSE principles to assist
in concept creation and review. The benefits of the approach are explored
through examples of a notional Unmanned Aerial Vehicle design project.
Various SysML diagrams are developed and discussed. This approach advo­
cates utilization of use-case diagrams to support the Concept of Operations
(CONOPS) review (London, 2012).

342 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

Carlini (2010), in the Director, Defense Research and Engineering Rapid

Toolbox Study, called for accelerated concept engineering with an expanded
use of both virtual and physical prototypes, and support for more innovative
interdisciplinary red teams. In this article, the terms “virtual environment”
and “virtual prototype” can be used interchangeably. Korfiatis, Cloutier,
and Zigh (2015) authored a series of articles between 2011 and 2015 related
to CONOPS development and early systems engineering design methods.
The Integrated Concept Engineering Framework evolved out of numerous
research projects and articles looking at the combination of gaming and
MBSE methods related to the task of CONOPS creation. This innovative
work shows promise for the early system design and ideation stages of the
acquisition cycle. There is recognition in this work that the warfighter will
need an easy and intuitive way to add content to the game and modify the
parameters that control objects in the game environment (Korfiatis et al.,
2015).

Madni (2015) explored the use of storytelling and a nontechnical narrative

along with MBSE elements to enable more stakeholder interaction in the
design process. He studied the conjunction of stakeholder inputs, nontradi­
tional methods, and the innovative interaction between the game engine, the
virtual world, and the creation of systems engineering artifacts. The virtual
worlds created in this research also allowed the players to share common
views of their evolving environment (Madni, 2015; Madni, Nance, Richey,
Hubbard, & Hanneman, 2014). This section has shown that researchers
are exploring virtual environments with game-based elements, sometimes
mixed with MBSE to enhance the defense acquisition process.

Crowdsourcing
Wired magazine editors Jeff Howe and Mark Robinson coined the
term “crowdsourcing” in 2005. In his Wired article titled “The Rise of
Crowdsourcing,” Howe (2006) described several types of crowdsourcing.
The working definition for this effort is "… the practice of obtaining needed
services, ideas, design, or content by soliciting contributions from a large
group of people and especially from the system stakeholders and users
rather than only from traditional employees, designers, or management"
(Crowdsourcing, n.d.).

The best fit for crowdsourcing, conceptually, for this current research proj­
ect is the description of research and development (R&D) firms utilizing the
InnoCentive Website to gain insights from beyond their in-house R&D team.
A vital feature in all of the approaches is the use of the Internet and modern
computational environments to find needed solutions or content, using the

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 343

Crowdsourcing with Virtual Environments http://www.dau.mil

diversity and capability of “the crowd” at significant cost or time savings.

The DoD, following this lead, is attempting to explore the capabilities and
solutions provided by the utilization of crowdsourcing concepts. The DoD
has numerous restrictions that can hinder a full utilization, but an artfully
crafted application and a focus on components or larger strategic concepts
can help to overcome these barriers (Howe, 2006).

In a Harvard Business Review article, “Using the Crowd as an Innovation

Partner,” Boudreau and Lahkani (2013) discussed the approaches to crowd-
sourcing that have been utilized in very diverse areas. They wrote: “Over the
past decade, we’ve studied dozens of company interactions with crowds on
innovation projects, in areas as diverse as genomics, engineering, operations

research, predictive analytics, enterprise software development, video

games, mobile apps, and marketing” (Boudreau & Lahkani, 2013, p. 60).

Boudreau and Lahkani discussed four types of crowdsourcing: contests,

collaborative communities, complementors, and crowd labor. A key enabler
of the collaborative communities’ concept is the utilization of intrinsic
motivational factors such as the desire to contribute, learn, or achieve. As
evidenced in their article, many organizations are clearly taking note of, and
are beginning to leverage the power of, diverse, geographically separated
ad hoc groups to provide innovative concepts, engineering support, and a
variety of inputs that traditional employees normally would have provided
(Boudreau & Lahkani, 2013).

In 2015, the U.S. Navy launched “Hatch.” The Navy calls this portal a
“crowdsourced, ideation platform” (Department of the Navy, 2015). Hatch
is part of a broader concept called the Navy Innovation Network (Forrester,
2015; Roberts, 2015). With this effort, the Navy hopes to build a continuous

344 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

process of innovation and minimize the barriers for information flow to help
overcome future challenges. Novel wargaming and innovation pathways are
to become the norm, not the exception. The final tools that will fall under
this portal are still being developed. However, it appears that the Navy has
taken a significant step foward to establish structural changes that will
simplify the ideation and innovation pipeline, and ensure that the Navy uses
all of the strengths of the total workforce. “Crowdsourcing, in all of its forms,
is emerging as a powerful tool…. Organizational leaders should take every
opportunity to examine and use the various methods for crowdsourcing at
every phase of their thinking” (Secretary of the Navy, 2015, p. 7).

The U.S. Air Force has also been exploring various crowdsourcing concepts.
They have introduced the Air Force Collaboratory Website and held a num­
ber of challenges and projects centered around three different technology
areas. Recently, the U.S. Air Force opened a challenge prize on its new
Website, http://www.airforceprize.com, with the goal of crowdsourcing
a design concept for novel turbine engines that meet established design
requirements and can pass the validation tests designed by the Air Force
(U.S. Air Force, n.d.; U.S. Air Force, 2015).

Model Based Systems Engineering

MBSE tools have emerged and are supported by many commercial firms.
The path outlined by the International Council on Systems Engineering
(INCOSE), in their Systems Engineering Vision 2020 document (INCOSE,
2007) shows that INCOSE expects the MBSE environment to evolve into
a robust, interconnected development environment that can serve all sys­
tems engineering design and development functions. It remains to be seen
if MBSE can transcend the past transformation initiatives of SMART, SBA,
and others on the DoD side. The intent of the MBSE section of questions is
to identify the key or critical success factors needed for MBSE to integrate
into, or encompass within, a crowdsourcing process in order to provide the
benefits that proponents of MBSE promise (Bianca, 2000; Sanders, 1997).

The Air Force Institute of Technology discussed MBSE and platform-based

engineering as it discussed collaborative design in relation to rapid/expe­
dited systems engineering (Freeman, 2011). The process outlined is very
similar to the INCOSE view of the future, with MBSE included in the design
process. Freeman covered the creation of a virtual collaborative environ­
ment that utilizes “tools, methods, processes, and environments that allow
engineers, warfighters, and other stakeholders to share and discuss choices.
This spans human-system interaction, collaboration technology, visual­
ization, virtual environments, and decision support” (Freeman, 2011, p. 8).

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 345

Crowdsourcing with Virtual Environments http://www.dau.mil

As the DoD looks to use MBSE concepts, new versions of the DoD Instruction
5000.02 and new definitions have emerged. These concepts and definitions
can assist in developing and providing the policy language to fully utilize
an MBSE-based process. The Office of the Deputy Secretary of Defense,
Systems Engineering is working to advance several new approaches related
to MBSE. New definitions have been proposed for Digital Threads and DED,
using a DSM. The challenges of training the workforce and finding the cor­
rect proof-of-principle programs are being addressed (Zimmerman, 2015).
These emerging concepts can help enable evolutionary change in the way
DoD systems are developed and designed.

The director of the AMRDEC is looking to MBSE as the “ultimate cool

way” to capture the excitement and interest of emerging researchers and
scientists to collaborate and think holistically to capture “a single evolving
computer model” (Haduch, 2015, p. 28). This approach is seen as a unique
method to capture the passion of a new generation of government engineers
(Haduch, 2015).

Other agencies of the federal government are also working on pro­

grams based on MBSE. David Miller, National Aeronautics and Space
Administration (NASA) chief technologist, indicates that NASA is trying
to use the techniques to modernize and focus future engineering efforts
across the system life cycle, and to enable young engineers to value MBSE
as a primary method to accomplish system design (Miller, 2015).

The level of interaction required and utilization of MBSE artifacts, methods,

and tools to create, control, and interact with future virtual environments
and simulations is a fundamental challenge.

SELECTED VIRTUAL

ENVIRONMENT ACTIVITIES

Army
Within the Army, several efforts are underway to work on various
aspects of virtual environments/synthetic environments that are import­
ant to the Army and to this research. Currently, efforts are being funded
by the DoD at Army Capability Integration Center (ARCIC), Institute for
Creative Technologies (ICT) at University of Southern California, Naval
Postgraduate School (NPS), and at the AMRDEC. The ESP efforts managed
by Army Lieutenant Colonel Vogt continue to look at building a persistent,
game-based virtual environment that can involve warfighters voluntarily in
design and ideation (Tadjdeh, 2014). Several prototype efforts are underway

346 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

at ICT and NPS to help evolve a system that can provide feedback from the
warfighters, playing game-based virtual environments that answer real
design and strategy questions. Key questions being looked at include: what
metrics to utilize, how to distribute the games, and whether the needed
data can be saved and transmitted to the design team. Initial prototype
environments have been built and tested. The ongoing work also looks at
technologies that could enable more insight into the HSI issues by attempt­
ing to gather warfighter intent from sensors or camera data relayed to the
ICT team (Spicer et al., 2015).

The “Always ON-ON Demand” efforts being managed by Dr. Nancy Bucher
(AMRDEC) and Dr. Christina Bouwens are a larger effort looking to tie
together multiple simulations and produce an “ON-Demand” enterprise
repository. The persistent nature of the testbed and the utilization of vir­
tual environment tools, including the Navy-developed Simulation Display
System (SIMDIS™) tool, which utilizes the OpenSceneGraph capability,
offers exploration of many needed elements required to utilize virtual envi­
ronments in the acquisition process (Bucher & Bouwens, 2013; U.S. Naval
Research Laboratory, n.d.).

Navy
Massive Multiplayer Online War Game Leveraging the Internet
(MMOWGLI) is an online strategy and innovation game employed by the
U.S. Navy to tap the power of the “crowd.” It was jointly developed by the
NPS and the Institute for the Future. Navy researchers developed the mes­
sage-based game in 2011 to explore issues critical to the U.S. Navy of the
future. The game is played based on specific topics and scenarios. Some of
the games are open to the public and some are more restrictive. The way to
score points and “win” the game is to offer ideas that other players comment
upon, build new ideas upon, or modify. Part of the premise of the approach
is based on this statement: “The combined intelligence of our people is an
unharnessed pool of potential, waiting to be tapped” (Moore, 2014, p. 3).
Utilizing nontraditional sources of information and leveraging the rapidly
expanding network and visualization environment are key elements that
can transform the current traditional pace of design and acquisition. In
the future, it might be possible to tie this tool to more highly detailed vir­
tual environments and models that could expand the impact of the overall
scenarios explored and the ideas generated.

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 347

Crowdsourcing with Virtual Environments http://www.dau.mil

RESEARCH QUESTIONS
The literature review demonstrates that active research is ongoing into
crowdsourcing, MBSE, and virtual environments. However, there is not a
fully developed process model and an understanding of the key elements that
will provide the DoD a method to fully apply these innovations to successful
system design and development. The primary research questions that this
study examined to meet this need are:
• What are the critical success factors that enable game-based
virtual environments to crowdsource design and requirements
information from warfighters (stakeholders)?

• What process and process elements should be created to inject war

fighter-developed ideas, metrics, and feedback from game-based
virtual environment data and use cases?

• What is the role of MBSE in this process?

METHODOLOGY AND DATA COLLECTION

The Delphi technique was selected for this study to identify the critical
success factors for the utilization of virtual environments to enable crowd-
sourced information in the system design and acquisition process. Delphi
is an appropriate research technique to elicit expert judgment where com­
plexity, uncertainty, and only limited information available on a topic area
prevail (Gallop, 2015; Skutsch & Hall, 1973). A panel of M&S experts was
selected based on a snowball sampling technique. Finding experts across
DoD and academia was an important step in this research. Expertise in
M&S, as well as virtual environment use in design or acquisition, was the
primary expertise sought. Panel members that met the primary requirement
areas, but also had expertise in MBSE, crowdsourcing, or HSI were asked to
participate. The sampling started with experts identified from the literature
search as well as Army experts with appropriate experience known by the
researcher. Table 1 shows a simplified description of the panel members as
well as their years of experience and degree attainment. Numerous, addi­
tional academic, Air Force, and Navy experts were contacted; however, the
acceptance rate was very low.

348 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

TABLE 1. EXPERT PANEL EXPERTISE

DESCRIPTION EDUCATION EXPERIENCE

Academic Researcher—Alabama PhD 20-30 years
Navy—Academic Researcher—California PhD 20-30 years
Army Officer—Requirements/Game Based Masters 15-20 years
Enviroments—Virginia
Army SES—M&S—Retired—Maryland PhD 30 + years
Navy M&S Expert—Virgina Masters 10-15 years
M&S Expert—Army SES—Retired Masters 30 + years
M&S Expert—Army—Virtual Environments Masters 10-15 years
M&S Expert—Army—V&V PhD 20-30 years
M&S Expert—Army—Virtual Environments PhD 15-20 years
M&S Expert—Army—Simulation Masters 20-30 years
M&S Expert—Virtual Environments/Gaming BS 15-20 years
M&S Expert—Army—Serious Games— PhD 10-15 years
Colorado
Academic Researcher—Virtual PhD <10 years
Environments—Conops—New Jersey
M&S Expert—Army—Visualization Masters 20-30 years
M&S Expert—Army/MDA—System of BS 20-30 years
Systems Simulation (SoS)
Academic Researcher—Florida PhD 20-30 years
M&S Expert—Army Virtual Environments— PhD 15-20 years
Michigan
M&S Expert—Army—Simulation PhD 10-15 years
Army M&S—Simulation/SoS Masters 20-30 years
Army—Simulation—SES—Maryland PhD 30 + years

Note. CONOPS = Concept of Operations; M&S = Modeling and Simulation; MDA =

Missile Defense Agency; SES = Senior Executive Services; SoS = System of Systems;
V&V = Verification and Validation

An exploratory “interview-style” survey was conducted using SurveyMonkey

to collect demographic data and answers to a set of 38 questions. This sur­
vey took the place of the more traditional semistructured interview due
to numerous scheduling conflicts. In addition, each member of the expert
panel was asked to provide three possible critical success factors in the
primary research areas. Follow-up phone conversations were utilized to

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 349

Crowdsourcing with Virtual Environments http://www.dau.mil

seek additional input from members of the panel. A large number of possi­
ble critical success factors emerged for each focus area. Figure 2 shows the
demographics of the expert panel (n=20). More than half (55 percent) of the
panel have Doctoral degrees, and an additional 35 percent hold Master’s
degrees. Figure 2 also shows the self-ranked expertise of the panel. All have
interacted with the defense acquisition community. The panel has the most
experience in M&S, followed by expertise in virtual environments, MBSE,
HSI, and crowdsourcing. Figure 3 depicts a word cloud; this figure was
created from the content provided by the experts in the interview survey.
The large text items show the factors that were mentioned most often in
the interview survey. The initial list of 181 possible critical success factors
was collected from the survey, with redundant content grouped or restated
for each major topic area when developing the Delphi Round 1 survey. The
expert panel was asked to rank the factors using a 5-element Likert scale
from Strongly Oppose to Strongly Agree. The experts were also asked to
rank their or their groups’ status in that research area, ranging from “inno­
vators” to “laggards” for later statistical analysis.

FIGURE 2. EXPERT PANEL DEMOGRAPHICS AND EXPERTISE

Degrees M & S VE
Bachelors Medium
10%  5%
Medium
25%
Masters PhD
35% 55% High High
95% 75%

HSI Crowdsource MBSE

Low
High High
10%
20% 20%
Medium
40% Low
Low
50%
60% Medium
Medium
35% 30%

350


Defense



ARJ, April 2017, Vol. 24 No. 2 : 334–367
 April 2017

FIGURE 3. WORDCLOUD FROM INTERVIEW SURVEY

Fifteen experts participated in the Round 1 Delphi study. The data generated
were coded and statistical data were also computed. Figure 4 shows the top
10 factors in each of four areas developed in Round 1—virtual environments,
crowdsourcing, MBSE, and HSI. The mean, Interquartile Range (IQR), and
percent agreement are shown for 10 factors developed in Round 1.

The Round 2 survey included bar graphs with the statistics summarizing
Round 1. The Round 2 survey contained the top 10 critical success factors
in the five areas—with the exception of the overall process model, which
contained a few additional possible critical success factors due to survey
software error. The Round 2 survey shows an expanded Likert scale with
seven levels, ranging from Strongly Disagree to Strongly Agree. The addi­
tional choices were intended to minimize ties and to help show where the
experts strongly ranked the factors.

Fifteen experts responded to the Round 2 survey, rating the critical success
factors determined from Round 1. The Round 2 survey critical success
factors continued to receive a large percentage of experts choosing survey
values ranging from “Somewhat Agree” to “Strongly Agree,” which con­
firmed the Round 1 top selections. But Round 2 data also suffered from an
increase in “Neither Agree nor Disagree” responses for success factors past
the middle of the survey.

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 351

Crowdsourcing with Virtual Environments http://www.dau.mil

FIGURE 4. CRITICAL SUCESS FACTOR RESULTS ROUND 1

VIRTUAL ENVIRONMENTS
CRITICAL SUCCESS FACTOR MEAN IQR % AGREE
Real Time Operation 4.67 1 93%
Utility to Stakeholders 4.47 1 93%
Fidelity of Modeling/Accuracy of Representation 4.40 1 87%
Usability/Ease of Use 4.40 1 93%
Data Recording 4.27 1 87%
Verification, Validation and Accreditation 4.20 1 87%
Realistic Physics 4.20 1 80%
Virtual Environment Link to Problem Space 4.20 1 80%
Flexibility/Customization/Modularity 4.07 1 80%
Return On Investment/Cost Savings 4.07 1 87%

CROWDSOURCING
CRITICAL SUCCESS FACTOR MEAN IQR % AGREE
Accessibility/Availability 4.53 1 93%
Leadership Support/Commitment 4.53 1 80%
Ability to Measure Design Improvement 4.47 1 93%
Results Analysis by Class of Stakeholder 4.33 1 93%
Data Pedigree 4.20 1 87%
Timely Feedback 4.20 1 93%
Configuration Control 4.13 1 87%
Engaging 4.13 1 80%
Mission Space Characterization 4.13 1 87%
Portal/Web site/Collaboration Area 4.07 1 87%

MBSE
CRITICAL SUCCESS FACTOR MEAN IQR % AGREE
Conceptual Model of the Systems 4.60 1 87%
Tied to Mission Tasks 4.43 1 93%
Leadership Commitment 4.40 1 80%
Reliability/Repeatability 4.33 1 93%
Senior Engineer Commitment 4.33 1 80%
Fidelity/Representation of True Systems 4.27 1 93%
Tied To Measures of Performance 4.27 1 87%
Validation 4.27 1 93%
Well Defined Metrics 4.27 1 80%
Adequate Funding of Tools 4.20 2 73%

352 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

FIGURE 4. CRITICAL SUCESS FACTOR RESULTS ROUND 1— CONTINUED

HSI
CRITICAL SUCCESS FACTOR MEAN IQR % AGREE
Ability to Capture Human Performance Behavior 4.64 1 100%
Adequate Funding 4.57 1 100%
Ability to Measure Design Improvement 4.43 1 93%
Ability to Analyze Mental Tasks 4.36 1 100%
Integration with Systems Engineering Process 4.33 1 87%
Leadership Support/Commitment 4.29 1.25 79%
Intuitive Interfaces 4.29 1.25 79%
Consistency with Operational Requirements 4.27 1 93%
Data Capture into Metrics 4.21 1 86%
Fidelity 4.14 1 86%

Note. IQR = Interquartile Range.

The Round 3 survey included the summary statistics from Round 2 and
charts showing the experts’ agreement from Round 2. The Round 3 ques­
tions presented the top 10 critical success factors in each area and asked
the experts to rank these factors. The objective of the Round 3 survey was
to determine if the experts had achieved a level of consensus regarding the
ranking of the top 10 factors from the previous round.

PROCESS AND EMERGING CRITICAL

SUCCESS FACTOR THEMES

In the early concept phase of the acquisition process, more game-like

elements can be utilized, and the choices of technologies can be very wide.
The graphical details can be minimized in favor of the overall application
area. However, as this process is applied later in the design cycle, more
detailed virtual prototypes can be utilized, and there can be a greater focus
on detailed and subtle design differences that are of concern to the war-
fighter. The next sections present the overall process model and the critical
success factors developed.

Process (Framework)
“For any crowdsourcing endeavor to be successful, there has to be a
good feedback loop,” said Maura Sullivan, chief of Strategy and Innovation,
U.S. Navy (Versprille, 2015, p. 12). Figure 5 illustrates a top-level view
of the framework generated by this research. Comments and discussion

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 353

Crowdsourcing with Virtual Environments http://www.dau.mil

from the interview phase have been combined with the literature review
data and information to create this process. Key elements from the Delphi
study and the critical success factors have been utilized to shape this pro­
cess. The fidelity of the models utilized would need to be controlled by the
visualization/modeling/prototyping centers. These centers would provide
key services to the warfighters and engineers to artfully create new game
elements representing future systems and concepts, and to pull information
from the enterprise repositories to add customizable game elements.

FIGURE 5. CROWDSOURCE INNOVATION FRAMEWORK

S&T Projects
MBSE Warfighter
& Ideas Ideation

Voting/Ranking/Filter Use Case in Graphical Scenario Vote/Rank/Comment

Feedback MBSE SysML/UML Development Feedback
Artifacts

Engineers & Scientists Visualization/Modeling/ Warfighters

Prototype Centers
Engage Modeling
Deploy/Capture & Develop Game Play
Telemeter Metrics Models & Physics Team to Add  Game/Compete
Game Features
Collaborative
MBSE UML/SysML MBSE Artifacts
Artifacts Innovation
Crowdsource Game Engines
Autogenerated
Portal Innovation Ranking/Polling
Environment Engines

nts Let
Decisio

me
MBS

ha
iron lity
els

Physics

E A

Env
n Engi
Mod

rtifa
cts
nes

Enterprise Repository/Digital System Models

Note. MBSE = Model Based Systems Engineering; S&T = Science and Technology;
SysML/UML = Systems Modeling Language/Unified Modeling Language.

The expert panel was asked: “Is Model Based Systems Engineering neces­
sary in this approach?” The breakdown of responses revealed that 63 percent
responded “Strongly Agree,” another 18.5 percent selected “Somewhat
Agree,” and the remaining 18.5 percent answered “Neutral.” These results
show strong agreement with using MBSE methodologies and concepts as
an essential backbone, using MBSE as the “glue” to manage the use cases,
and subsequently providing the feedback loop to the DSM.

In the virtual environment results from Round 1, real time operation and
realistic physics were agreed upon by the panel as critical success factors.
The appropriate selection of simulation tools would be required to sup­
port these factors. Scenegraphs and open-source game engines have been
evolving and maturing over the past 10 years. Many of these tools were
commercial products that had proprietary architectures or were expensive.

354


Defense



ARJ, April 2017, Vol. 24 No. 2 : 334–367
 April 2017

However, as the trend toward more open-source tools continues, game

engines have followed the trend. Past research conducted by Romanczuk
(2012) linked scenegraph tools such as Prospect, Panda3D, and Delta3D to
high-fidelity human injury modeling and lethality application programming
interfaces. Currently, the DoD has tools like VBS2 and VBS3 available, but
newer commercial-level engines are also becoming free for use by DoD and
the public at large. Premier game engines such as Source, Unity, and Unreal
are now open-source engines (Heggen, 2015). The trend continues as WebGL
and other novel architectures allow rapid development of high-end complex
games and simulations.

In the MBSE results from Round 1, the panel indicated that both ties to
mission tasks and to measures of performance were critical. The selection
of metrics and the mechanisms to tie these factors into the process are very
important. Game-based metrics are appropriate, but these should be tied
to elemental capabilities. Army researchers have explored an area called
Degraded States for use in armor lethality (Comstock, 1991). The early work
in this area has not found wide application in the Army. However, the ele­
mental capability methodology, which is used for personnel analysis, should
be explored for this application. Data can be presented to the warfighter that
aid gameplay by using basic physics. In later life-cycle stages, by capturing
and recording detailed data points, engineering-level simulations can be
run after the fact, rather than in real time, with more detailed high-fidelity
simulations by the engineering staff. This allows a detailed design based on
feedback telemetered from the warfighter. The combination of telemetry
from the gameplay and follow-up ranking by warfighters and engineering
staff can allow in-depth, high-fidelity information flow into the emerging
systems model. Figure 6 shows the authors’ views of the interactions and
fidelity changes over the system life cycle.

FIGURE 6. LIFE CYCLE

Early Con
cept
Warfighter Prototype 
s Evaluatio
n
Warfighter EMD
Open Innovation s
Collaboration Competitive Warfighters
IDEATION

Strategic Medium Fidelity High Fidelity

S&T

Design Features
Trade Study Evolving Representations Eng/Sci
Analysis of Alternatives Focused
Low Fidelity Eng/Sci
Comparative
Eng/Sci
Broad

Note. EMD = Engineering and Manufacturing Development; Eng/Sci = Engineers/

Scientists; S&T = Science and Technology.

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 355

Crowdsourcing with Virtual Environments http://www.dau.mil

Collaboration and Filtering

A discussion on collaboration and filtering arose during the interviews.
The feedback process from a crowd using virtual environments needs voting
and filtering. The voting techniques used in social media or on Reddit are
reasonable and well-studied. Utilizing techniques familiar to the young
warfighters will help simplify the overall process. The ranking and filtering
needs to be done by both engineers and warfighters so the decisions can
take both viewpoints into consideration. Table 2 shows the top 10 critical
success factors from Round 2 for the overall process. The Table includes
the mean, IQR, and the percent agreement for each of the top 10 factors. A
collaboration area, ranking and filtering by scientists and engineers, and
collaboration between the warfighters and the engineering staff are critical
success factors—with a large amount of agreement from the expert panel.

TABLE 2. TOP 10 CRITICAL SUCCESS FACTORS

OVERALL PROCESS—ROUND 2

CRITICAL SUCCESS FACTOR MEAN IQR % AGREE

Filtering by Scientists/Engineers 5.56 1 81%
Portal/Website/Collaboration Area 5.56 1 81%
Leadership Support 6 2.5 75%
Feedback of Game Data into Process 5.56 2.75 75%
Timely Feedback 5.75 2.75 75%
Recognition 5.13 1.75 75%
Data Security 5.5 2.75 75%
Collaboration between Eng/Scientist 6.06 2.5 75%
and Warfighters
Engagement (Warfighters) 5.94 3 69%
Engagement (Scientists & Engineers) 5.75 3 69%

Fidelity
Fidelity was ranked high in virtual environments, MBSE, and HSI.
Fidelity and accuracy of the modeling and representations to the true
system are critical success factors. For the virtual environment, early
work would be done with low facet count models featuring texture maps
for realism. However, as the system moves through the life cycle, higher
fidelity models and models that feed into detailed design simulations will
be required. There must also be verification, validation, and accreditation
of these models as they enter the modeling repository or the DSM.

356 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

Leadership Commitment
Leadership commitment was ranked near the top in the MBSE, crowd-
sourcing, and HSI areas. Clearly, in these emerging areas the enterprise
needs strong leadership and training to enable MBSE and crowdsourcing
initiatives. The newness of MBSE and crowdsourcing may be related to the
experts’ high ranking of the need for leadership and senior engineer commit­
ment. Leadership support is also a critical success factor in Table 2—with
75 percent agreement from the panel. Leadership commitment and support,
although somewhat obvious as a success factor, may have been lacking in
previous initiatives. Leadership commitment needs to be reflected in both
policy and funding commitments from both DoD and Service leadership to
encourage and spur these innovative approaches.

Critical Success Factors

Figure 7 details the critical success factors generated from the Delphi
study, which visualizes the top 10 factors in each by using a mind-map­
ping diagram. The main areas of study in this article are shown as major
branches, with the critical success factors generated appearing on the limbs
of the diagram. The previous sections have discussed some of the emerging
themes and how some of the recurring critical success factors in each area
can be utilized in the framework developed. The Round 3 ranking of the
critical success factors was analyzed by computing the Kendall’s W, coef­
ficient of concordance. Kendall’s W is a nonparametric statistics tool that
measures the agreement of a group of raters. The experts’ rankings of the
success factors showed moderate, but statistically significant agreement
or consensus.

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 357

358
Real Time Operation
ability
Utility to Stakeholders Accessibilty/Avail
mitment
Fidelity of Modeling/Accuracy of R d e rs h i p Support/Com
epresentat
io n Lea
Usability/Ease of Use ign Improvement
o Measure Des
Ability t
Data Recording
s   A na l ys i s by Class of Stakeholder
Verification, Validation and Accreditation Result
Data Pedigree
Realistic Physics
Timely Feedback
Virtual Environment Link to Problem Space             
  Virtu Configuration Control
Flexibility/Customization/Modularity al En g Engaging
rcin
Crowdsourcing with Virtual Environments

viro s ou
Return on Investment/Cost Savings n d Mission Space Characterization
me n
t w Portal/Websit
 Cro e/Collaboration Area
      
Conceptual Model of the Systems          
Tied to Mission Tasks
Leadership Commitment

Reliability/Repeatability

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

Critical Success Factors
Senior Engineer Commitment
            MBSE Filtering by Scientists/Engineers
Fidelity/Representation of True Systems
       
Portal/Website Collaboration Area
      O
Tied to Measure of Performance
veral
Leadership Support
l Pro
Validation
ces s
Feedback of Game Data Into Process
Well Defined Metrics SI Timely Feedback
ols
     H
Adequate Funding of To          Recognition

    
Ability to Capture Human Performance Behavior
Data Security
Adequate Funding
Collabo
Ability to Measure Design Improvement ration Betwee
n Eng/Scientist & Warfighters 

Ability to Analyze Mental Tasks Engagement (Warfighters)
Integration with Systems Engineering Process Engagemen
t (Scientists & Engineers)
Leadership Support/Commitment

Intuitive Interfaces
ments
FIGURE 7. CRITICAL SUCCESS FACTOR IN FIVE KEY AREAS

rational Require
Consistency with Ope
Data Capture In to  M et rics

Fidelity
http://www.dau.mil
 April 2017

LIMITATIONS TO THE RESEARCH

The ideas presented here and the critical success factors have been
developed by a team of experts who have, on average, 20 to 30 years of expe­
rience in the primary area of inquiry and advanced degrees. However, the
panel was more heavily weighted by Army experts than individuals from
the rest of the DoD. Neither time nor resources allowed for study of other
important groups of experts, including warfighters, industry experts, and
program managers. The Delphi method was selected for this study to gen­
erate the critical success factors based on the perceived ease of use of the
method and the controlled feedback gathered. The critical success factors
developed are ranked judgment, but based on years of expertise. This study
considered five important areas and identified critical success factors in
those areas. This research study is based on the viewpoint of experts in
M&S. Nonetheless, other types of expert viewpoints might possibly gen­
erate additional factors. Several factor areas could not be covered by M&S
experts, including security and information technology.

The surveys were constructed with 5- and 7- element Likert scales that
allowed the experts to choose “Neutral” or “Neither Agree nor Disagree.”
Not utilizing a forced-choice scale or a nonordinal data type in later Delphi
rounds can limit data aggregation and statistical analysis approaches.

RECOMMENDATIONS AND

CONCLUSIONS

In conclusion, innovation tied to virtual environments and linked to

MBSE artifacts can help the DoD meet the significant challenges it faces in
creating new complex interconnected designs much faster than in the past
decade. This study has explored key questions and has developed critical
success factors in five areas. A general framework has also been developed.
The DoD must look for equally innovative ways to meet numerous informa­
tion technology (IT), security, and workforce challenges to enable the DoD to
implement the process successfully in the acquisition enterprise. The DoD
should also explore interdisciplinary teams by hiring and funding teams of
programmers and content creators to be co-located with systems engineers
and subject matter experts.
Artfully crafted game-based scenarios that help explore design and usability
issues can be crafted and provided to warfighters as a part of the process
and help focus on needed system information. The challenge remains for
the methods to harvest, filter, and convert the information gathered to

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 359

Crowdsourcing with Virtual Environments http://www.dau.mil

MBSE artifacts that result from this process. An overall process can be
enacted that takes ideas, design alternatives, and data harvested—and
then provides a path to feed back this data at many stages in the acquisition
cycle. The extent to which MBSE tools such as SysML, UML, and emerging
new standards are adopted or utilized in the process may depend upon the
emerging training of acquisition professionals in MBSE and the leadership
commitment to this approach.

Artfully crafted game-based scenarios that help

explore design and usability issues can be crafted
and provided to warfighters as a part of the process
and help focus on needed system information.

This article has answered the three research questions posed in earlier
discussion. Utilizing the expert panel, critical success factors have been
developed using the Delphi method. An emerging process model has been
described. Finally, the experts in this Delphi study have affirmed an essen­
tial role of MBSE in this process.

FUTURE RESEARCH
The DoD is actively conducting research into the remaining challenges
to bring many of the concepts discussed in this article into the acquisition
process. The critical success factors developed here can be utilized to focus
some of the efforts.

Key challenges in DoD remain, as the current IT environment attempts

to study larger virtual environments and prototypes. The question of
how to utilize the Secret Defense Engineering Research Network, High
Performance Supercomputing, and Secret Internet Protocol Router
Network, while simultaneously making the process continually available
to warfighters, will need to be answered. The ability of deployed warfighters
to engage in future system design efforts is also a risk item that needs to be
investigated. Research is essential to identify the limitations and inertia
associated with the DoD IT environment in relation to virtual environments
and crowdsourcing. An expanded future research study that uses additional

360


Defense



ARJ, April 2017, Vol. 24 No. 2 : 334–367
 April 2017

inputs, including a warfighter expert panel and an industry expert panel,

would provide useful data to compare and contrast with the results of this
study.

An exploration of how to combine the process described in this research

with tradespace methodologies and ERS approaches could be explored.
MBSE methods to link and provide feedback should also be studied.

The DoD should support studies that select systems in the early stages of
development in each Service to apply the proposed framework and process.
The studies should use real gaps and requirements, and real warfighters. In
support of ARCIC, several studies are proposed at the ICT and the NPS that
explore various aspects of the challenges involved in testing tools needed
to advance key concepts discussed in this article. The Navy, Air Force, and
Army have active programs under various names to determine how M&S
can support future systems development as systems and designs become
more complex, distributed, and interconnected (Spicer et al., 2015).

The extent to which MBSE tools such as SysML, UML,

and emerging new standards are adopted or utilized
in the process may depend upon the emerging
training of acquisition professionals in MBSE and
the leadership commitment to this approach.

When fully developed, MBSE and DSM methods can leverage the emerging
connected DoD enterprise and bring about a continuous-feedback design
environment. Applying the concepts developed in this article to assessments
conducted by developing concepts, Analysis of Alternatives, and trade
studies conducted during early development through Milestone C can lead
to more robust, resilient systems continuously reviewed and evaluated by
the stakeholders who truly matter: the warfighters.

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 361

Crowdsourcing with Virtual Environments http://www.dau.mil

References
Bianca, D. P. (2000). Simulation and modeling for acquisition, requirements, and
training (SMART) (Report No. ADA376362). Retrieved from http://oai.dtic.mil/
oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA376362
Boudreau, K. J., & Lakhani, K. R. (2013). Using the crowd as an innovation partner.
Harvard Business Review, 91(4), 60–69.
Bucher, N., & Bouwens, C. (2013). Always on–on demand: Supporting the
development, test, and training of operational networks & net-centric systems.
Presentation to National Defense Industrial Association 16th Annual Systems
Engineering Conference, October 28-31, Crystal City, VA. Retrieved from http://
www.dtic.mil/ndia/2013system/W16126_Bucher.pdf
Carlini, J. (2010). Rapid capability fielding toolbox study (Report No. ADA528118).
Retrieved from http://www.dtic.mil/dtic/tr/fulltext/u2/a528118.pdf
Comstock, G. R. (1991). The degraded states weapons research simulation: An
investigation of the degraded states vulnerability methodology in a combat
simulation (Report No. AMSAA-TR-495). Aberdeen Proving Ground, MD: U.S.
Army Materiel Systems Analysis Activity.
Corns, S., & Kande, A. (2011). Applying virtual engineering to model-based systems
engineering. Systems Research Forum, 5(2), 163–180.
Crowdsourcing (n.d.). In Merriam-Webster’s online dictionary. Retrieved from http://
www.merriam-webster.com/dictionary/crowdsourcing
Dalkey, N. C. (1967). Delphi (Report No. P-3704). Santa Monica, CA: The RAND
Corporation.
David, J. W. (1995). A comparative analysis of the acquisition strategies of Army
Tactical Missile System (ATACMS) and Javelin Medium Anti-armor Weapon
System (Master’s thesis), Naval Postgraduate School, Monterey, CA.
Department of the Navy. (2015, May 20). The Department of the Navy launches the
“Hatch.” Navy News Service. Retrieved from http://www.navy.mil/submit/display.
asp?story_id=87209
Drucker, C. (2014). Why airport scanners catch the water bottle but miss the
dynamite [Duke Research Blog]. Retrieved from https://sites.duke.edu/
dukeresearch/2014/11/24/why-airport-scanners-catch-the-water-bottle-but­
miss-the-dynamite/
Ferrara, J. (1996). DoD's 5000 documents: Evolution and change in defense
acquisition policy (Report No. ADA487769). Retrieved from http://oai.dtic.mil/
oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA487769
Forrester, A. (2015). Ray Mabus: Navy’s ‘Hatch’ platform opens collaboration on
innovation. Retrieved from http://www.executivegov.com/2015/05/ray-mabus­
navys-hatch-platform-opens-collaboration-on-innovation/
Freeman, G. R. (2011). Rapid/expedited systems engineering (Report No.
ADA589017). Wright-Patterson AFB. OH: Air Force Institute of Technology,
Center for Systems Engineering.
Gallop, D. (2015). Delphi, dice, and dominos. Defense AT&L, 44(6), 32–35. Retrieved
from http://dau.dodlive.mil/files/2015/10/Gallop.pdf
GAO. (2015). Defense acquisitions: Joint action needed by DOD and Congress to
improve outcomes (Report No. GAO-16-187T). Retrieved from http://www.gao.
gov/assets/680/673358.pdf

362 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

General Electric. (2017). GE open innovation. Retrieved from http://www.ge.com/

about-us/openinnovation
Gould, J. (2015, March 19). McHugh: Army acquisitions 'tale of failure'. DefenseNews.
Retrieved from http://www.defensenews.com/story/defense/land/
army/2015/03/19/mchugh-army-acquisitions-failure-underperforming­
canceled-/25036605/
Gourley, S. (2015). US Army looks to "full spectrum" shoulder-fired weapon. Retrieved
from https://www.military1.com/army-training/article/572557-us-army-looks-to­
full-spectrum-shoulder-fired-weapon/
Haduch, T. (2015). Model based systems engineering: The use of modeling enhances
our analytical capabilities. Retrieved from http://www.army.mil/e2/c/
downloads/401529.pdf
Hagel, C. (2014). Defense innovation days. Keynote presentation to Southeastern New
England Defense Industry Alliance. Retrieved from http://www.defense.gov/
News/Speeches/Speech-View/Article/605602
Heggen, E. S. (2015). In the age of free AAA game engines, are we still relevant?
Retrieved from http://jmonkeyengine.org/301602/in-the-age-of-free-aaa-game­
engines-are-we-still-relevant/
Howe, J. (2006). The rise of crowdsourcing. Wired, 14(6), 1–4. Retrieved from http://
www.wired.com/2006/06/crowds/
INCOSE. (2007). Systems engineering vision 2020 (Report No. INCOSE­
TP-2004-004-02). Retrieved from http://www.incose.org/ProductsPubs/pdf/
SEVision2020_20071003_v2_03.pdf
Jane’s International Defence Review (2015). Lighten up: Shoulder-launched weapons
come of age. Retrieved from http://www.janes360.com/images/assets
/442/49442/ shoulder-launched weapon _systems_come_of_age.pdf
Kendall, F. (2014). Better buying power 3.0 [White Paper]. Retrieved from Office
of the Under Secretary of Defense (Acquisition, Technology & Logistics)
Website: http://www.defenseinnovationmarketplace.mil/resources/
BetterBuyingPower3(19September2014).pdf
Korfiatis, P., Cloutier, R., & Zigh, T. (2015). Model-based concept of operations
development using gaming simulation: Preliminary findings. Simulation & Gaming.
Thousand Oaks, CA: Sage Publications. https://doi.org/1046878115571290
London, B. (2012). A model-based systems engineering framework for concept
development (Master’s thesis), Massachusetts Institute of Technology,
Cambridge, MA. Retrieved from http://hdl.handle.net/1721.1/70822
Lyons, J. W., Long, D., & Chait, R. (2006). Critical technology events in the
development of the Stinger and Javelin Missile Systems: Project hindsight
revisited. Washington, DC: Center for Technology and National Security Policy.
Madni, A. M. (2015). Expanding stakeholder participation in upfront system
engineering through storytelling in virtual worlds. Systems Engineering, 18(1),
16–27. https://doi.org/10.1002/sys.21284
Madni, A. M., Nance, M., Richey, M., Hubbard, W., & Hanneman, L. (2014). Toward an
experiential design language: Augmenting model-based systems engineering
with technical storytelling in virtual worlds. Procedia Computer Science,
28(2014), 848–856.
Miller, D. (2015). Update on OCT activities. Presentation to NASA Advisory Council,
Technology, Innovation, and Engineering Committee. Retrieved from https://
www.nasa.gov/sites/default/files/atoms/files/dmiller_oct.pdf

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 363

Crowdsourcing with Virtual Environments http://www.dau.mil

Modigliani, P. (2013, November–December). Digital Pentagon. Defense AT&L, 42(6),

40–43. Retrieved from http://dau.dodlive.mil/files/2013/11/Modigliani.pdf
Moore, D. (2014). NAWCAD 2030 strategic MMOWGLI data summary. Presentation
to Naval Air Systems Command. Retrieved from https://portal.mmowgli.nps.
edu/documents/10156/108601/COMMS+1_nscMMOWGLIOverview_post.
pdf/4a937c44-68b8-4581-afd2-8965c02705cc
Murray, K. L. (2014). Early synthetic prototyping: Exploring designs and concepts
within games (Master’s thesis), Naval Postgraduate School, Monterey, CA.
Retrieved from http://calhoun.nps.edu/handle/10945/44627
NRC. (2010). The rise of games and high-performance computing for modeling and
simulation. Committee on Modeling, Simulation, and Games. Washington, DC:
National Academies Press. https://doi.org/10.17226/12816
Roberts, J. (2015). Building the Naval Innovation Network. Retrieved from http://www.
secnav.navy.mil/innovation/Pages/2015/08/NIN.aspx
Rodriguez S. (2014). Top 10 failed defense programs of the RMA era. War on the
Rocks. Retrieved from http://warontherocks.com/2014/12/top-10-failed-defense­
programs-of-the-rma-era/
Romanczuk, G. E. (2012). Visualization and analysis of arena data, wound ballistics
data, and vulnerability/lethality (V/L) data (Report No. TR-RDMR-SS-11-35).
Redstone Arsenal, AL: U.S. Army Armament Research, Development and
Engineering Center.
Sanders, P. (1997). Simulation-based acquisition. Program Manager, 26(140), 72–76.
Secretary of the Navy (2015). Characteristics of an innovative Department of the Navy.
Retrieved from http://www.secnav.navy.mil/innovation/Documents/2015/07/
Module_4.pdf
Sheridan, V. (2015). From former NASA researchers to LGBT activists – meet some
faces new to GW. The GW Hatchet. Retrieved from http://www.gwhatchet.
com/2015/08/31/from-former-nasa-researchers-to-lgbt-activists-meet-some­
faces-new-to-gw/
Skutsch, M., & Hall, D. (1973). Delphi: Potential uses in educational panning.
Project Simu-School: Chicago Component. Retrieved from https://eric.
ed.gov/?id=ED084659
Smith, R. E., & Vogt, B. D. (2014, July). A proposed 2025 ground systems “Systems
Engineering” process. Defense Acquisition Research Journal, 21(3), 752–774.
Retrieved from http://www.dau.mil/publications/DefenseARJ/ARJ/ARJ70/ARJ­
70_Smith.pdf
Spicer, R., Evangelista, E., Yahata, R., New, R., Campbell, J., Richmond, T., Vogt, B., &
McGroarty, C. (2015). Innovation and rapid evolutionary design by virtual doing:
Understanding early synthetic prototyping (ESP). Retrieved from http://ict.usc.
edu/pubs/Innovation%20and%20Rapid%20Evolutionary%20Design%20by%20
Virtual%20Doing-Understanding%20Early%20Synthetic.pdf
Tadjdeh, Y. (2014). New video game could speed up acquisition timelines. National
Defense. Retrieved from http://www.nationaldefensemagazine.org/blog/lists/
posts/post.aspx?ID=1687
U.S. Air Force. (n.d.). The Air Force collaboratory. Retrieved from https://
collaboratory.airforce.com/
U.S. Air Force. (2015). Air Force prize. Retrieved from https://airforceprize.com/about
U.S. Naval Research Laboratory. (n.d.). SIMDIS™ presentation. Retrieved from https://
simdis.nrl.navy.mil/SimdisPresentation.aspx

364 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

Versprille, A. (2015). Crowdsourcing to solve tough Navy problems. National Defense.

Retrieved from http://www.nationaldefensemagazine.org/archive/2015/June/
Pages/CrowdsourcingtoSolveToughNavyProblems.aspx
Zimmerman, P. (2015). MBSE in the Department of Defense. Seminar presentation
to Goddard Space Flight Center. Retrieved from https://ses.gsfc.nasa.gov/ses_
data_2015/150512_Zimmerman.pdf

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 365

Crowdsourcing with Virtual Environments http://www.dau.mil

Author Biographies

Mr. Glenn E. Romanczuk is a PhD candi­

date at The George Washington University. He
is a member of the Defense Acquisition Corps
matrixed to the Operational Test Agency (OTA),
evaluating the Ballistic Missile Defense System.
He holds a BA in Political Science from DePauw
University, a BSE from the University of Alabama
in Huntsville (UAH), and an MSE from UAH in
Engineering Management. His research includes
systems engineering, lethality, visualization, and
virtual environments.

(E-mail address: gromanczuk@gwmail.gwu.edu)

Dr. Christopher Willy is currently a senior

systems engineer and program manager with
J. F. Taylor, Inc. Prior to joining J. F. Taylor in
1999, he completed a career in the U.S. Navy.
Since 2009, he has taught courses as a professo­
rial lecturer for the Engineering Management
and Systems Engineering Department at The
George Washington University (GWU). Dr. Willy
holds a DSc degree in Systems Engineering from
GWU. His research interests are in stochastic
processes and systems engineering.

(E-mail address: cwilly@gwmail.gwu.edu)

366 Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367

 April 2017

Dr. John E Bischoff is a professorial lecturer

of Engineering Management at The George
Washington University (GWU). He has held exec­
utive positions in several firms including AOL/
Time Warner and IBM Watson Research Labs.
Dr. Bischoff holds a BBA from Pace University,
an MBA in Finance from Long Island University,
an MS in Telecommunications Management
from the Polytechnic University, and a Doctor of
Science in Engineering Management from GWU.

(E-mail address: jeb@email.gwu.edu)

Defense ARJ, April 2017, Vol. 24 No. 2 : 334–367 367

Reproduced with permission of copyright
owner. Further reproduction prohibited
without permission.