Hindawi

Discrete Dynamics in Nature and Society

Volume 2020, Article ID 6243085, 7 pages
https://doi.org/10.1155/2020/6243085

Review Article
A Virtual Reality Platform for Safety Training in Coal Mines with
AI and Cloud Computing

Mei Li ,1 Zhenming Sun ,2 Zhan Jiang ,1 Zheng Tan ,3 and Jinchuan Chen 3

1
Institute of Remote Sensing and Geographical Information System, Peking University, Beijing 100871, China
2
School of Energy and Mining Engineering, China University of Mining and Technology (Beijing), Beijing 100083, China
3
Beijing Longruan Technologies, Beijing 100190, China

Correspondence should be addressed to Zhenming Sun; sun@cumtb.edu.cn

Received 25 June 2020; Revised 20 September 2020; Accepted 26 September 2020; Published 23 October 2020

Academic Editor: Chi-Hua Chen

Copyright © 2020 Mei Li et al. This is an open access article distributed under the Creative Commons Attribution License, which
permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Coal mining, regarded as a high-risk industry, has a strong demand for virtual reality (VR) to fulfill safety and emergency rescue
training. In the past ten years, VR technology has significantly improved miner training on both the hardware and software side.
However, it still has some drawbacks, such as expensive and unsuitable hardware, lack of satisfactory user experience, without
direct browser access, and lack of humanized and intelligent design. To solve these problems, a cloud-based VR system is designed
for the training of coal miners in this paper. The system, with browser/client architecture, includes eight modules demonstrating
the full procedure of an underground coal mine. The online cloud-rendered video streaming is adopted to provide enough
computing and rendering power and hence a better browser-based user experience. Furthermore, game artificial intelligence (AI)
is also introduced into the system to increase the emotional exchange between the system and users. Unlike traditional VR training
software, this system designs two virtual miners to enhance the experience of trainees. The first virtual miner is a task-oriented
non-player-character (NPC) which conveys general knowledge about the mine and guides the users in visiting the underground
work sites. The second virtual miner is a disaster-oriented character which prepares the users for typical disasters. The system has
been successfully implemented in a laboratory environment, and its performance has been validated. Yet, further practices are
needed to stimulate more innovative applications of VR-based miner training and disaster drilling.

1. Introduction occupational health and safety situation, especially for the

young miners [2].
Coal mines, with dangerous working environments and The United States, Australia, and the UK have used VR
complex production systems, have a strong demand for as a training environment for mining simulation, accident
virtual reality (VR). VR technology has its unique advan- reconstruction and investigation, education, and safety
tages in several fields related to coal mining, such as miner training in the past two decades. The UK, in particular, has a
safety training and emergency rescue drilling, besides long history of developing and using VR technology in coal
drifting-producing process simulation, disaster scenario mine safety training. The VR products, such as SafeVR and
simulation, and mechanical operation training [1]. The fifth- Vroom, are very famous for open-pit truck operator training
generation (5G) cellular, artificial intelligence (AI), data [3]. The researchers of the National Institute for Occupa-
analytics, Internet of Things (IoT), and cloud computing will tional Safety and Health (NIOSH) explored how the mining
provide the information infrastructure required for the industry could effectively use “serious games” for fire-escape
intelligent mine construction in the next three to five years. training. Spokane Research Laboratory (SRL) developed
With these emerging technologies, the next-generation VR fire-escape training software for a mine safety training
systems for underground coal mining will be achieved, course. The software allowed four trainees to work together
which will improve the professional adaptation process and in a virtual environment through a computer network. The
2 Discrete Dynamics in Nature and Society

trainees responded favorably and showed a marked im- 2. Methodology

provement in locating the proper evacuation routes on
subsequent trials [4]. The University of New South Wales To address the abovementioned problems and train the coal
developed iCinema (an advanced visualization and inter- miners, a VR-based system using the Unreal Engine is
action environment), which included a 360° immersive designed and developed in this paper. Contrary to the
theater environment with 18 modules for improving traditional standalone or client/server VR software, this
teaching and learning activities in mining engineering [5]. system adopts browser/client architecture and uses cloud
The University of Queensland Experimental Mine (UQEM) rendering and Game AI technologies. It aims to enhance
laboratory has been used by students for experiments since users’ learning experience, strengthen their memory, and
1956. This institution developed several VR applications, improve safety awareness. The system uses the Unreal En-
including VR-based teaching modules, drill rig, Instron UCS gine from Epic Games, which includes a rendering engine
rock testing simulations, and VR-based ventilation simu- (renderer), Blueprint scripting language, animation, artifi-
lations [6]. cial intelligence (AI), networking, terrain, streaming, and
In China, universities, large mining companies, and local scene graphs. Figure 1 illustrates the system interface, which
governments utilize the VR hardware and software for safety has eight modules as follows:
and emergency rescue training [7]. Those VR systems (i) Geological environments, such as procedure of
provide the desktop versions of character “Q and A” ses- coal formation, geology condition, coal seam, and
sions. The VR hardware is composed of a projection-based geology structures
panoramic display system, the infrared tracking stereoscopic
glasses, the VR headset display, a pad, and the other devices. (ii) Geographical environment and surface plants
Furthermore, the 3D visualization system is very popular in (iii) Underground roadway arrangement
coal mines with the functions of the 3D geological model, (iv) Mining area and working face design
ventilation simulation, real-time data monitoring, and (v) Coal mine production system, which includes
emergency response. ventilation, transportation, belt, lifting, power
In recent ten years, many coal mine enterprises estab- supply, water supply, and drainage
lished underground industrial networks and tried to apply
cloud computing, big data, Internet of Things, and even AI (vi) Mining processes, including coal mine blasting
technology into mining, ventilation, safety, and education and longwall mining processes
projects. There has been a shift from traditional mining to (vii) Coal mine disaster (gas leakages, roof falls, and fire
intelligent mining. A three-tier ITframework with automatic events) simulation
control system, production execution system, and enterprise (viii) Underground safety monitoring system (methane,
management system is established. Based on cloud com- wind speed, temperature, and carbon monoxide),
puting techniques, big data related to underground coal self-rescue, and emergency response
mining can be analyzed quickly, and critical data associated
with user requirements can be extracted precisely. With the In this paper, we use three technologies, namely, cloud
established platform, the danger-spotting and subsequent rendering architecture, 3D modeling, and AI behavior de-
decision-making capabilities of users can be greatly en- sign to achieve multiuser online operation and improve the
hanced to ensure underground mining safety [8]. interaction and immersion of VR devices.
Although VR is an important part of intelligent mine The browser/client architecture of cloud rendering
construction, there are still some problems with these ap- means that each user connects to the server and accesses 3D
plications that need to be tackled. Firstly, most of the existing streaming over the cloud. The cloud servers handle all events
VR-based training systems lack a good learning experience. and the state of the virtual world and transfer the VR render
Although these systems adopt teaching and learning func- streaming to the client. These servers can be virtual servers or
tions via a single-player mode and an ask-answer pattern, server clusters. They take full advantage of virtually infinite
there is no complex AI man-machine interaction. VR computing power to create, capture, and compress high-
training systems always focus on professional computation resolution rendered streaming, which is sent to the client
or disaster 3D simulation or text and voice presentation. The with low performance and limited computing power [11].
learning process needs to be improved. Furthermore, the Compared with other popular Web3D technologies,
advantages of immersion, interaction, and imagination of such as HTML5 and WebGL, cloud rendering has incom-
VR technology have not been fully utilized yet [9]. Secondly, parable advantages mainly because it has no special or ex-
traditional VR-based training systems are standalone or pensive requirements on the client side. It allows users to
client/server versions, without direct browser access. Cloud choose different platforms, including PCs, laptops, tablets,
rendering will become the mainstream to support various and smartphones. Furthermore, since Wi-Fi 6 and 5G will
light terminals, such as PCs, mobile phones, tablets, and provide sufficient network bandwidth, online cloud ren-
headsets. These technologies will enable real-time trans- dering will deliver adequate computing and rendering
mission and enhance cyber-physical and social experiences power. With the advancements in the Internet infrastruc-
in a fully immersive environment [10]. Hence, VR training ture, the users can interact with VR programs through the
software should adopt gamification and online learning, network on the client with high-quality videos and low-
which have become a tendency in education. response delays [12].
Discrete Dynamics in Nature and Society 3

Figure 1: The interface of the virtual reality system for underground coal mine.

In this B/S architecture of the cloud rendering system, models. We use the Blueprint scripting language to program
the system is divided into three parts: portal server, render the characters’ animation, the AI behavior tree, and the user
server, and client, as shown in Figure 2. interface (UI).
The portal server is responsible for user login, finding the AI in games or “Game AI” is a module of a 3D engine.
appropriate server according to the user’s choice, and Traditional AI typically demonstrates some of the following
controlling the server to start the virtual machine, i.e., render behaviors associated with human intelligence: planning,
server. The client and the render server establish a con- learning, reasoning, problem-solving, knowledge represen-
nection when the portal server notifies the client with the tation, perception, motion, and manipulation. The emerging
address of the virtual machine. The user can run and interact Game AI is becoming a narrow branch differentiated from
with the 3D program in the remote server through the the traditional AI. Game AI refers to the virtual man, also
network. called non-player characters (NPCs), controlled by the
The render server is responsible for processing the input computer, which simulates the intelligent behaviors of
events of the client sourced by mouse or keyboard, as well as human beings or other creatures, providing users with a
translating and sending them to the interaction module of reasonable challenge to overcome [13].
the 3D program for further parsing. The 3D scene images are Figure 3 shows the behaviors of the two NPCs we
captured, encoded, and sent to the client as a video designed. These NPCs can simulate the behaviors of miners
streaming. The render server program includes three and guide young miners and/or students to understand the
modules, namely, basic function, Game AI, and advanced whole mining production, experience the underground
function. The VR basic function module provides 3D scene scene, and improve the relationship between users and
management, sound, particle effect, physical simulation, and computer programs during the training, conversation, or
other basic functions. The Game AI module uses a scripting other tasks. Besides, these AI characters are designed to
language to design a behavior tree for virtual miners. The VR automatically find the local navigation grid path.
advanced module builds complex computation models and The first NPC is a task-oriented character. When the user
simulates disasters, such as fire numerical simulation and meets it somewhere on the ground, he will provide general
escaping route computation. knowledge about the coal mines and guide the user to de-
The client is responsible for displaying the received scend a well from a cage and visit the underground work
streaming of the 3D scene and sending user inputs, such as sites, such as longwalls or heading faces. Once the user
keyboard and mouse events, to the server. completes the corresponding task and provides feedback, the
We use 3ds Max to build 3D scenes of the underground NPC releases the next task. Those tasks are performed before
and the surface. Then, we import the 3D models into the entering the underground mine, preparing the miners to
Unreal Engine editor. We adopt a series of material tools to look for the entrance, enter the cage, and visit the under-
apply 3D models, such as a normal map, diffusion map, ground working face. The second NPC is a disaster-oriented
metal map, and texture map. These materials are like the character which guides the user to experience typical di-
“paint” applied to the model. Then, we set the lights to il- sasters, such as gas explosions, roof falls, and fire accidents.
luminate our scenes. From the computer-aided design To realize the NPC’s behavior, the blueprint is used to
(CAD) or geographic information system (GIS) files, we build a behavior tree graph, which adds and connects a series
automatically build 3D geological and underground laneway of nodes. A behavior tree, mainly composed of a leaf node
4 Discrete Dynamics in Nature and Society

Portal server

Selection

Server selection

VR streaming
Client/user
User database
Interaction

Render database
Render server

Environment

Emergency response
Disaster simulation
Scene management

Basic database
Trigger
Particle effect

AI guidance
AI scripting
Animation

Self-rescue
TPC/NPC
Sound

VR basic functions Game AI VR advance module

3D data
representation
3D simulation of mine training process
Basic data

Figure 2: The architecture of the cloud rendering system.

and composite node, is essentially an acyclic graph [14, 15]. actions. The sequence executes the disaster animation in
We use composite tasks and design nodes of selectors, se- turn and tests whether the user’s reaction is correct.
quences, parallels, and decorators. Figure 4 shows an ex-
ample of the second NPC’s behavior tree, where selector, 3. Results and Discussion
sequence, and parallel are used to trigger accidents. A se-
lector is a branch task that runs the behavior of each of its 3.1. System Implementation. The cloud rendering cluster is
children. It will immediately return a success status code composed of two virtual rendering servers with a configu-
when one of its children runs successfully. Once one of the ration of 4-core 8G M60 CPU, 32 GB RAM, and 40 Mbps
child nodes is successful, it returns to the selector, and the BGP bandwidth. The experiments have shown that the
rest of the nodes are not executed. A sequence executes the hardware can support up to seven players at the same time. A
node task sequentially instead. A sequence node tries to client PC should have at least Intel i7 CPU and 16 GB RAM.
execute each child in turn until all children have reported The system has been tested by undergraduate students in the
that they have executed successfully. Once a task fails, the VR Teaching and Experiment Laboratory, China University
sequence will return to the upper node. In this case, the of Mining & Technology, Beijing. The laboratory also has a
remaining tasks are not executed. A parallel is a composite 120-degree 2-channel VR projector and an infrared laser
task that handles concurrent behaviors. It is a special branch tracking system as shown in Figures 5 and 6, respectively.
task that runs all children during execution time. The be- The system can run either on the immersive VR hardware or
havior tree allows parallel node tasks, which might be state the PC headset.
machines. Multiple state machines can be executed in
parallel. For example, when the second NPC’s behavior tree
is triggered, it first judges whether it is dangerous or not and 3.2. AI Guidance. The virtual roaming function of the NPC
then runs the user and sends alerts. Following these, it gives will help the user to walk around buildings, enter the mine
the on-site emergency solution by a series of animation through a cage, and visit the longwall mining face or the
Discrete Dynamics in Nature and Society 5

First person main

perspective Warning of Simulate gas
User control
roaming on the the NPC explosion scenes
ground

evacuation
Report the
Handling A class on
Task-oriented Gas explosion hazards disasters
NPC

Complete
feedback
Preparing before Responses Mission failed
entering the
Guide

End
underground Ignore

Looking for the The entrance to

Warning of Simulate roof
entrance to the the underground
the NPC fall accident
underground mine mine

Support
Task-oriented Support A class on
NPC operation disasters

Roof fall
accident
Responses Mission failed
Complete
feedback

No Is OK for entering End

the mine? Ignore
Guide

Yes Warning of Simulate

the NPC Fire accident
Entering cage in
the mine

Forward
Avoid, repair, A class on
Visiting key sites etc. disasters
Fire accident
of underground
mine

Responses Mission failed

Free movement Disaster-oriented End
operation NPC Ignore

Mission
End simulation
accomplished

General knowledge learning Safety training

Figure 3: VR mission design of Game AI NPCs.

blasting place. It will also introduce the ventilation system, the source, such as footsteps when walking on the grass and
transportation system, water supply, drainage system, and the roadway, wheezing when running, and the mechanical
power supply system. This function enables the new miners sound of an opening cage. By analyzing the disaster/accident
or students to have an overall understanding of the mine. reasons, features, and occurrence conditions, a disaster
When a user is wandering in the roadway, the accident simulation knowledge library is constructed. This library
guidance function of the NPC will start a dialogue with them contains a brief description of prevention and control
to detect their behavior and provide information on illegal measures, emergency plans, site disposal programs, miti-
operations. The system will then trigger disaster animation gation, and postdisaster evaluation. It is combined with
depending on the location, and the NPC will demonstrate system guidance to show the potential consequences of
the escape routes for self-rescue. disasters.

3.3. Disaster Simulation. In our system, mining accidents are 3.3.1. Gas and Coal Dust Explosion. This simulation is to
limited to gas explosions, roof falls, and fire events. The show the boundary conditions of gas and coal dust explosion
disaster simulation module integrates various 3D animation, as illustrated in Figure 7. It helps users to perceive the di-
audio, particle effect, and physical-mechanical models of the saster from the vision, auditory, and other aspects and
3D engine. The 3D model is to establish the models of mines, understand the movement and destruction of the wind that
terrain, tunnels, and NPCs. The audio function is to create is mixed with dust. Besides, some emergency measures, such
sounds as realistic as possible considering the dimension of as collaborative gas extraction, are fully displayed.
6 Discrete Dynamics in Nature and Society

Root

Is danger
Selector

Y N
Sequence Selector

Find player Attached player Disaster types Talk Run Idle

Parallel Selector

Run to player Alert Roof fall Gas leakage explosion Mechanical accident Seepage
Sequence Sequence Sequence Sequence

Give issue Demonstrati Guide Check operation Give a mark

solutions on scheme operation
Selector

Right Wrong

Figure 4: The behavior tree of the disaster-oriented NPC.

Figure 5: The professor introducing the system to the visitors on

Figure 7: Gas explosion animation.
the 120-degree 2-channel VR projector.

Figure 8: Roof fall animation.

Figure 6: The overview of VR Teaching and Experiment

flames, blasting, and short circuits, as well as the internal
Laboratory.
ones caused by the combustion of coal or other flammable
substances due to their oxidation or burn. The VR system
3.3.2. Roof Fall and Wall Collapse. Through this simulation, can simulate the conditions of various fires, illustrate their
the user will understand the main factors of roof fall and wall severity, and therefore inform decision making for disaster
collapse as shown in Figure 8. It can simulate various rock prevention. The system also shows the ways of treatment,
burst and ground pressure disasters, introducing the form of including the closed grouting and nitrogen injection
disasters, their causes, and the potential methods of disaster methods.
prevention and control.
4. Discussion
3.3.3. Fire Event. The user will see the main forms of fires, The VR system uses cloud rendering and AI behavior trees to
such as the external fires that are mostly caused by open enhance training performance. It can realistically simulate
Discrete Dynamics in Nature and Society 7

the complex underground coal mine environment and Research and Development Program of Inner Mongolia
provide the function of login, roaming, virtual simulation, titled “Technology and application on digital mine resource
and operation evaluation. However, due to intellectual management and ecological environment monitoring,
property rights, only a small part of the functions is made 2015–2019.”
accessible to the public through the website. To date (August
1, 2020), the system has been visited 20, 853 times and the References
training system drew perfect user reviews (5.0/5.0) on the
National Virtual Simulation Experiment Teaching Project [1] J. Tichon and R. Burgess-Limerick, “A review of virtual reality
as a medium for safety related training in the minerals in-
Sharing Platform [16]. According to the experiment score
dustry,” Journal of Health & Safety Research & Practice, vol. 1,
data, 1355 people have finished the experiment and the no. 3, pp. 33–40, 2011.
excellent score rate was 46.0%. The students commented that [2] A. Grabowski and J. Jankowski, “Virtual reality-based pilot
the system could give them a more intuitive understanding training for underground coal miners,” Safety Science, vol. 72,
of the underground environment and disasters without no. 72, pp. 310–314, 2015.
visiting the real underground coal mines. Despite satisfac- [3] D. Schofield, B. Denby, and R. Hollands, “Mine safety in the
tory performance and remarkable feedback, the current twenty-first century: the application of computer graphics and
cloud render hardware needs to be upgraded to a larger virtual reality,” in Mine Health and Safety Management,
cloud rendering cluster to provide a quicker response and M. Karmis, Ed., pp. 153–174, Society of Mining, Metallurgy,
hence a better experience. Furthermore, a performance test and Exploration, Inc., Englewood, CO, USA, 2001.
is needed to estimate the amount of concurrency and [4] T. J. Orr, L. G. Mallet, and K. A. Margolis, “Enhanced fire
escape training for mine workers using virtual reality simu-
runtime efficiency. The system functions also need more
lation,” Mining Engineering, vol. 61, no. 11, pp. 41–44, 2009.
research, where a more detailed behavior tree design can [5] T. Barker, “Images and eventfulness: expanded cinema and
improve the user experience. Combined with the numerical experimental research at the University of New South Wales,”
computation, the system can be turned into a testbed to Studies in Australasian Cinema, vol. 6, no. 2, pp. 111–123,
support the ventilation design, emergency drilling, and other 2012.
systems and fields. We expect more innovative applications [6] M. S. Kizil, A. P. Kerridge, and M. G. Hancock, “Use of virtual
of VR-based miner training and disaster drilling in the near reality in mining education and training,” in Proceedings of the
future. 2004 CRC Mining Research and Effective Technology Transfer
Conference, pp. 1–7, Noosa Heads, Australia, June 2004.
5. Conclusions [7] M. Li, Z. M. Sun, P. Y. Lyu, J. Chen, and S. Mao, “Study on key
technology of multiplayer virtual reality training platform for
VR, as a part of intelligence mine, provides the new tool for fully-mechanized coal mining face,” Coal Science and Tech-
work procedure management, occupational health and di- nology, vol. 46, no. 1, pp. 156–161, 2018.
[8] Y. Wu, M. Chen, K. Wang, and G. Fu, “A dynamic infor-
saster simulation, and visualization. The VR system provides
mation platform for underground coal mine safety based on
macroscopic and microscopic underground 3D scenes for internet of things,” Safety Science, vol. 113, pp. 9–18, 2019.
coal mine and displays information such as personnel po- [9] R. Mitra and S. Saydam, “Can artificial intelligence and fuzzy
sitioning, safety monitoring, and equipment management. logic be integrated into virtual reality applications in mining?”
Combined with cloud rendering, it can achieve more real- Journal of the Southern African Institute of Mining and
istic and more detailed 3D visualization effects. Considering Metallurgy, vol. 114, no. 12, pp. 1009–1016, 2014.
the advantages of cloud rendering and AI, this paper designs [10] E. Bastug, M. Bennis, M. Medard, and M. Debbah, “Toward
a cloud-based VR system for mine safety training. AI interconnected virtual reality: opportunities, challenges, and
guidance is adopted to roam underground and simulate enablers,” IEEE Communications Magazine, vol. 55, no. 6,
accidents, such as gas explosions, roof falls, and fire events. pp. 110–117, 2017.
[11] R. Shea, J. Jiangchuan Liu, E. C.-H. Ngai, and Y. Cui, “Cloud
We believe that the results of this study will provide a useful
gaming: architecture and performance,” IEEE Network,
tool for coal mine safety training, avoiding the potential vol. 27, no. 4, pp. 16–21, 2013.
hazards of the extreme underground environment. [12] C. Y. Huang, C. H. Hsu, Y. C. Chang, and K.-T. Chen,
“GamingAnywhere: an open cloud gaming system,” in Pro-
Data Availability ceedings of the 4th ACM Multimedia Systems Conference,
pp. 36–47, New York, NY, USA, February 2013.
No data were used to support the findings of this study. The [13] G. N. Yannakakis and J. Togelius, “A panorama of artificial
system will be considered to be openly accessed afterward. and computational intelligence in games,” IEEE Transactions
on Computational Intelligence and AI in Games, vol. 7, no. 4,
Conflicts of Interest pp. 317–335, 2015.
[14] I. Millington and J. Funge, Artificial Intelligence for Games,
The authors declare that they have no conflicts of interest. CRC Press, Boca Raton, FL, USA, 2009.
[15] W. J. He, Design and Implementation of Game AI Based on
Acknowledgments Behavior Tree, Chengdu University of Technology, Chengdu,
China, 2018.
This study was financially supported by the National Key [16] National Virtual Simulation Experiment Teaching Project
Research and Development Program of China (grant no. Sharing Platform, September 2020, http://www.ilab-x.com/
2016YFC0803108). It was also supported by the Key details/v5?id�5022&isView�true.