A Three-Dimensional Visualization and Optimization Method of Landslide Disaster Scenes Guided by Knowledge

Abstract

The rapid acquisition of deposit volume information and dynamic modeling, as well as the visualization of disaster scenes, have great significance for the sharing of landslide information and the management of emergency rescue. However, existing methods have shortcomings, such as a long and costly deposit volume acquisition cycle, lack of knowledge and guidance, complex operations for scene modeling expression, and low scene rendering efficiency. Therefore, this paper focuses on the study of a three-dimensional visualization and optimization method for landslide disaster scenes guided by knowledge, and discusses key technologies such as the rapid acquisition of landslide deposit volume information based on three-dimensional reconstruction, the knowledge-guided dynamic modeling visualization of disaster scenes, and scene optimization considering visual significance. The prototype systems are developed and used in a case experiment and analysis. The experimental results show that the proposed method can quickly obtain the deposit volume, and the results are equivalent to ContextCapture, Metashape, and Pix4Dmapper software. The method realizes the dynamic visualization of the whole disaster process, provides rich information, achieves high readability, and improves the efficiency of scene rendering, with a stable average rendering frame rate of more than 80 frames/second.

1. Introduction

Landslide disasters are widely distributed, sudden, destructive, and can easily cause secondary disasters, which may result in serious casualties and economic loss [1,2,3,4,5,6,7,8,9]. According to a report from the Center for Disaster Epidemiology Research (CRED) and the United Nations Strategies for Disaster Reduction (UNISDR), from 1998 to 2017, the number of people affected by landslides worldwide reached 480 million, with 18,414 people dying as a result [10]. In 2014, the Journal of Science published a paper related to natural disasters. It pointed out that the construction of a virtual geographic disaster environment can be valuable for the simulation, display, and analysis of spatiotemporal processes. Moreover, trend prediction is improved, decision making is supported, the expression of landslide scenes is optimized, and knowledge regarding landslides is shared [11]. The Sendai Framework for Disaster Risk Reduction 2015–2030 placed clear emphasis on the research and development of new technologies to develop new products and services that contribute to disaster risk reduction [12]. Therefore, research on the scientific problems of landslide disasters is of great significance to the scientific emergency disposal.

During a landslide, due to the urgency of the emergency response, it is necessary to rapidly obtain disaster information such as the deposit volume and to construct three-dimensional scenes quickly, so that a reference for rescue and management can be provided. With the development of integrated space–air–ground monitoring technology, landslide deposit volume acquisition methods are becoming increasingly diversified, including field investigation [13,14], multitemporal digital elevation model (DEM) comparison [15,16,17], spaceborne interferometric synthetic aperture radar inversion [18,19], empirical power-law relationships [20,21,22], and three-dimensional (3D) reconstruction [23,24,25]. However, these methods have some shortcomings, such as the complexity of data processing, poor timeliness, inaccurate pre-disaster information acquisition, and large differences between the DEM before and after the landslide. It is necessary to develop an efficient and convenient method of acquiring landslide disaster information.

A virtual scene should be constructed to express the whole process of the landslide disaster. In this way, users can obtain details about the surroundings, process, and trends of the landslide. Many studies have been carried out in this regard; for example, multi-source data can be fused to construct a 3D disaster scene, the spatiotemporal process is simulated based on particle flow, smoke and dust are added to optimize the virtual effect, and various disaster information used in the modeling is processed by interpolation [26,27,28,29,30,31]. However, most studies focus on the simulation of the sliding process and the provision of post-disaster information, ignoring pre-disaster information. Landslides are triggered by earthquakes, geological conditions, heavy rainfall, and other factors. Therefore, pre-disaster information is also a key part of disaster scene expression and cannot be ignored [32,33,34]. Moreover, the dynamic change of the disaster process is complicated, the differences in the scene visualization content are large, there is a lack of modeling expression knowledge and guidance, and the operation of scene modeling expression is complex. It is difficult to solidify and expand the visualization content.

To carry out the emergency disposal and management of a disaster scene, in addition to a comprehensive understanding of the landslide disaster, it is also necessary to obtain accurate landslide disaster information, which requires higher visualization and rendering efficiency of the landslide disaster scene. The existing research mainly focuses on scene optimization through model simplification [35], LOD dynamic scheduling [36,37], view clipping [38,39], tunnel vision [40], etc. However, most of these methods are direct animation simulations. Meanwhile, the characteristics of users and the visualization characteristics of disaster scenes are not fully considered. As a result, the efficiency of disaster scene rendering is low, which makes it difficult to meet the needs of those responding to landslide disasters and who require disaster information sharing. Therefore, to improve the rendering efficiency of landslide scenes, optimization is necessary.

To address the above problems, this paper proposes a three-dimensional visualization and optimization method for landslide disaster scenes guided by knowledge. We focus on key technologies such as the rapid acquisition of landslide deposit volume information based on 3D reconstruction, the knowledge-guided dynamic visualization of landslide scenes, and scene optimization that considers visual saliency. Finally, a prototype system is developed, and the effectiveness and feasibility of the method are verified through a case experiment and analysis.

The rest of this paper is organized as follows. Section 2 introduces the key technologies and methods of this research, including rapid deposit volume acquisition, dynamic scene modeling and visualization, and scene optimization. Section 3 uses an example to describe the development of a prototype system and a corresponding experimental analysis. Section 4 presents the conclusions of this study and provides a brief discussion of future work.

2. Methodology

2.1. Overall Framework

The overall research idea of this paper, which includes the acquisition of disaster deposit volume information, the dynamic visualization of landslide scenes, and the optimization of landslide scenes, is shown in Figure 1. First, through the 3D reconstruction of images and the interpolation of the bottom surface of the deposit considering the steepest gradient, the DEM before and after the disaster is used to perform the difference calculation to quickly obtain the volumetric information of the landslide deposit. Second, the scene objects are extracted and modeled from the disaster-pregnant environment, triggering factors, disaster process, and disaster information of the landslide. Then, the multi-level semantic constraint rules are constructed. Next, the landslide scene modeling knowledge template is established, and the scene objects are instantiated to realize the knowledge-guided landslide scene modeling and dynamic expression. Finally, according to the visual characteristics of human eyes and the principle of visual saliency, the region of interest (ROI) in the scene is calculated. Meanwhile, the Gaussian blur algorithm and the high-pass algorithm are used to realize the step-by-step optimization of the landslide disaster scene.

2.2. Rapid Acquisition of Deposit Volume Based on 3D Reconstruction

First, according to the acquired landslide deposit images, the feature points of the landslide deposit are quickly extracted and matched. Under the boundary constraints of the landslide deposit, the matching feature points are screened, and the feature points within the deposit range are retained. Combined with the feature points of the picture, the camera parameters (including the focal length of the camera f, and the focal lengths f_x and f_y of the camera in the X-axis and Y-axis directions) are calculated, and the 2D image feature points are converted into 3D feature points using Formulas (1)–(3).

$$\{\begin{array}{c}{x}_c=\left(p.x-u_0\right)\times f/f_x\\ y_c=\left(p.y-v_0\right)\times f/f_y\\ z_c=f \end{array}$$

(1)

$$R_M=Rot\left(Y,\theta \right)\times Rot\left(X,\theta \right)\times Rot\left(Z,\theta \right)$$

(2)

$${\left[\begin{array}{ccc}{x}_c& y_c& z_c\end{array}\right]}^T{R}_M={\left[\begin{array}{ccc}{X}_W^{\prime }& Y_W^{\prime }& Z_W^{\prime }\end{array}\right]}^T$$

(3)

where ($p.x$, $p.y$) are the pixel coordinates of the feature points, ($u_0$, $v_0$) are the coordinates of the picture center point, and ($x_c$, $y_c$, $z_c$) are the coordinates of the feature points in the camera coordinate system. Formula (2) is the rotation matrix formed by rotating the camera around the corresponding coordinate axis with a certain angle. ($X_W^{\prime }$, $Y_W^{\prime }$, $Z_W^{\prime }$) are the three-dimensional coordinates of the feature point P calculated under a certain camera rotation matrix. Finally, according to the principle of binocular vision, with the optical center of the left and right cameras as the coordinate origin, two straight lines are constructed, and the two straight line equations are combined. The real-world coordinates $P\left(X_W,Y_W,Z_W\right)$, corresponding to the feature points, can be calculated by the least square method, and the sparse reconstruction of the 3D landslide deposit can scene be realized.

Three-dimensional reconstruction can only reconstruct the surface of the landslide deposit. It is difficult to accurately obtain the pre-disaster topography when the bottom surface is covered. Based on the inverse distance weight, a interpolation for the bottom surface considering the steepest gradient is proposed, which controls the Z value contribution rate in the direction of the steepest slope drop. The landslide deposit bottom-surface DEM is interpolated by Formula (4). The difference between the reconstructed post-disaster and interpolated bottom-surface DEM is calculated, and a regular triangular network is constructed. The idea of the integral is used to calculate the volume of the triangular network, and the volume of the landslide deposit can be quickly obtained.

$$\hat{Z}={\mathsf{\Sigma }}_{i=1}^N\frac{1}{D_i{}^{p+\frac{\alpha \left(90°-\theta \right)}{90°}}}{Z}_i/{\mathsf{\Sigma }}_{i=1}^N\frac{1}{D_i{}^{p+\frac{\alpha \left(90°-\theta \right)}{90°}}} \theta \in \left[0,90\right]$$

(4)

where $\hat{Z}$ is the elevation of the point to be interpolated, and N is the number of known points in the neighborhood to be interpolated (generally the number is 12). $\theta$ is the angle between the line connecting the point to be interpolated and the known point in the steepest direction. The smaller the angle of θ, the greater the distance weight, and the less the contribution to the point to be interpolated. p is the distance weight of point i, and α is the contribution rate of the Z-value in the direction of the steepest gradient, where p = 15 and α = 0.8 [41].

2.3. Knowledge-Guided Dynamic Visualization of Landslide Disaster Scene

Landslide scene visualization involves many scene objects. First, the scene visualization objects and their characteristics, such as the disaster-pregnant environment, trigger factors, disaster process, and disaster information are analyzed. The terrain and ground objects are modeled according to the characteristics of different scene objects. The objects represented in different landslide scenes are varied. To solve the problems of solidified visualization content, inflexible configuration, and the difficult extension of visualization forms in existing disaster scene modeling, there is a need to focus on the content completeness, spatial semantics, information readability, and logical relationships of the disaster scene modeling expression. Starting from this, the multi-level semantic constraints for modeling and expression of the landslide scene are constructed, so that the expression of the landslide disaster scenarios is more accurate and easier to understand.

To simplify the modeling operation, improve the modeling efficiency, and enhance the readability of the represented information, multi-level semantic constraint rules and expert empirical knowledge are integrated to build the knowledge template for landslide scene modeling and visualization, as shown in Figure 2. Knowledge templates are managed in the form of nodes, including root nodes, parent nodes, and child nodes. Each landslide disaster scene corresponds to a root node. The parent node represents the scene object information that needs to be represented, the parent and child nodes of the knowledge template are created through information completeness constraints, and the parent node creation order is constrained by the logical relationship. Each child node contains information about specific scene objects, spatial positions, spatial poses, and expression methods.

The root node and parent node in the knowledge template are read in turn, and the content of the knowledge template is parsed through the keyword information in the knowledge template, such as “path”, “pos”, “attitude”, “text”, and “color”, where “path” is the path of the scene object data storage, which can be displayed by directly reading the data under the path, “pos” is the spatial position of the scene object, “attitude” is the orientation and attitude of the scene object in the 3D space, “text” expresses the text description, and its corresponding value is the text content to be expressed, and “color” is the color expressed by the scene object, such as “White”, “Red”, “Gray”, “Blue”, “Green”. Finally, the dynamic visualization of the landslide disaster scene is realized by instantiation according to the content characteristics of the scene object. Users can adjust the content of each node in the knowledge template according to the actual needs and configure knowledge templates for different landslide disaster scenes for easy visualization.

2.4. Optimization of Landslide Disaster Scenes Considering Visual Salience

Although the visual field of the human eye can reach 188 degrees, the information can only be correctly identified within 20 degrees of the fixation point. The resolution ability gradually decreases with the expansion of the visual field. Visual saliency detection refers to the human eye focusing on the ROI according to their own needs and interests. The bottom-up visual attention mechanism is the visual attention driven by perceptual data such as color, texture, and shape. The greater the difference, the more attention it attracts. The direct impact range of landslide disasters is generally several square kilometers. Other information with weak correlation is outside of the range. To allow users to understand the landslide disaster more clearly, the disaster scene is generally displayed from a fixed perspective. Therefore, based on the dynamic visualization of landslide disaster scenes, a scene optimization method considering visual saliency is proposed.

The camera frustum parameters of the scene include the left and right camera space coordinates $C_L\left(x,y,z\right)\mathrm{and}{C}_R\left(x,y,z\right)$, the horizontal field of view angle β, the vertical field of view angle α, and the distance from the center of the camera to the near and far section (represented by f and n). The offset $f_y$ between the upper and lower sides of the viewing cone and the XZ plane, as well as the offset $f_x$ between the left and right sides and the YZ plane, are calculated by Formula (5).

$$\{\begin{array}{c}{f}_y=tan\mathsf{\beta }/2\\ f_x=f_y\times \tan \beta /\tan \alpha \end{array}$$

(5)

The model transformation matrix $R_M$ is obtained using Formula (2), and Formulas (6) and (7) are used to calculate the coordinates of the eight vertices ($V_{Lf1}$, $V_{Lf2}$, $V_{Lf3}$, $V_{Lf4}$, $V_{Ln1}$, $V_{Ln2}$, $V_{Ln3}$ and $V_{Ln4}$) of the quadrangular prism formed by the far and near sections of the visual cone. The range of the frustum is divided into a mixed horizon (the overlapping range of the frustum) and the peripheral horizon (monocular range of vision), and the scene outside the scope of the frustum is regarded as a non-horizon scene.

$$\left[\begin{array}{llll}\overrightarrow{f_1}& \overrightarrow{f_2}& \overrightarrow{f_3}& \overrightarrow{f_4}\end{array}\right]=R_M\times \left[\begin{array}{cccc}-f_x& -f_x& f_x& f_x\\ -f_y& f_y& -f_y& f_y\\ 1& 1& 1& 1\end{array}\right]$$

(6)

$$\{\begin{array}{l}{V}_{Lfi}=C_L(x,y,z)+f\times {\overrightarrow{f}}_l\\ V_{Lni}=C_L(x,y,z)+n\times {\overrightarrow{f}}_l\end{array}\hspace{1em}\hspace{1em}(i=1,2,3,4)$$

(7)

where $\overrightarrow{f_1}$, $\overrightarrow{f_2}$, $\overrightarrow{f_3}$, and $\overrightarrow{f_4}$ represent the direction vectors of the four lateral sides of the optic vertebra.

Based on the visual characteristics of the human eye and the landslide scene, the fixed-fovea rendering method is used to further subdivide the mixed visual field of the landslide into areas of interest and areas of non-interest, as shown in Figure 3.

The landslide range is set as the ROI. The radius is R, and the center point of the landslide disaster range is set by default to the gaze point $P_{foveated}$. The distance between the scene object $P_{object}$ and the gaze point in the mixed-view range is determined using Formula (8).

$$d=\left|P_{object}-P_{foveated}\right|$$

(8)

Comparing the sizes of d and R, when d > R, the current scene object is located in the non-interested area; when d ≤ R, the current scene object is located in the area of interest. Whether the scene objects in the mixed viewport are within the scope is determined pixel-by-pixel, and the mixed viewport is divided into a region of interest (foveal viewport) and a region of non-interest.

Using the field-of-view clipping technology, the non-view field scene range is cropped and culled, and the peripheral field of view scene objects are simplified. The Gaussian blur algorithm in Formula (9) is used to optimize the peripheral vision scene. For the non-interested area in the mixed view area, the high-pass algorithm is integrated on the basis of Gaussian blur for blur optimization by Formula (10). The ROI is rendered with high resolution. Thus, the disaster scene considering the visual saliency can be optimized step-by-step, which is more in line with the visual characteristics of the human eye.

$$GB\left(x,y\right)={\displaystyle \sum }_{i=-n}^n{\displaystyle \sum }_{j=-n}^{n}S\left(x+i,y+j\right)\frac{1}{2\pi {\sigma }^2}{e}^{-\left(i^2+j^2\right)/2{\sigma }^2}$$

(9)

$$NB\left(x,y\right)=\omega \left(S\left(x,y\right)-GB\left(x,y\right)+P\right)+GB\left(x,y\right) \omega \in \left[0,1\right]$$

(10)

where S(x,y) is the original image pixel value, GB(x,y) is the Gaussian blurred image pixel value, (x,y) is the row and column position of the image pixel, and (2n + 1) × (2n + 1) is the size of the convolution kernel. If the size of the convolution kernel is 3 × 3, then n = 1. P is the hyperparameter constant, usually 127, so the high contrast keeps the image from being too dark and losing information. NB(x,y) is the pixel value at the (x,y), which is processed by Gaussian blur and the high-pass algorithm. The weight value of the high-pass algorithm is ω.

3. Prototype System Implementation and Experimental Analysis

3.1. Prototype System Implementation and Study Area

In this paper, the prototype system presented is supported by a hardware environment consisting of a Dell T7610 graphics workstation and Windows 10 X64 operating system. The Android Studio 4.0 development platform was selected, combined with the Android SDK of the computer vision open-source library OpenCV 3.4 to develop a prototype system for the rapid acquisition of the volume of landslide deposits based on 3D reconstruction. The operating test environment was the VIVO X9S smartphone. Then, an Alienware M17 R4 laptop with a Core i7-10870H CPU @ 2.20 GHz octa-core, a memory of 32 GB, and an NVIDIA GeForce RTX 3080 Laptop GPU graphics card was used as hardware support, using Microsoft Visual Studio 2019. The R&D environment was Net, Unity 3D, and SteamVR, and a prototype system for the dynamic expression and optimization of landslide scenes was developed and displayed in combination with VR interactive devices such as Oculus Quest2. The interface of the prototype system is shown in Figure 4.

To verify the feasibility and effectiveness of the proposed method, a landslide (coordinates: 103.479° E, 31.097° N) near Lanmo Line, Yingxiu Town, Wenchuan County, Aba Tibetan and Qiang Autonomous Prefecture, Sichuan Province, was taken as a case to conduct an experiment on the rapid volume acquisition of landslide deposits. Taking the Jinsha River catastrophic landslide disaster in Jomda County, Changdu, Tibet Autonomous Region as a case, the modeling expression and optimization of the disaster scene were carried out. The specific case experiment distribution is shown in Figure 5.

The experimental data used in the rapid acquisition of deposit information mainly came from public data release platforms such as the National Geographic information resource directory service system and on-site photo acquisition. The acquired data included remote sensing images, on-site images, and deposit feature points. Meanwhile, the experimental data for the landslide disaster scene visualization and optimization were mainly from the Sichuan Provincial Bureau of Surveying, Mapping and Geographic Information, and included high spatial resolution remote-sensing image and terrain data, triggering factors, geological/rainfall, landslide disaster range, drop, deposit thickness, and width, as well as various multi-source disaster data such as barrier lake capacity and damaged roads, rivers, and buildings. The simulation data for the landslide spatiotemporal process was provided by He [31].

3.2. Experimental Analysis

3.2.1. Rapid Acquisition of Landslide Deposit Volume

The data processing was carried out, that is, camera calibration, the acquisition of landslide deposit images, screening, correction, feature-point selection, and the deposit boundary determination of the original images. The method proposed in this paper was used to perform the rapid 3D reconstruction and bottom-surface interpolation on the 3D landslide deposit scene, and the result is shown in Figure 6. At the same time, to verify the effectiveness of this method, referring to Mora et al. [42], the efficiency and accuracy of this method to obtain the landslide deposit volume were compared with ContextCapture, Metashape, and Pix4Dmapper commercial software. The comparative analysis results are shown in

Table 1. Comparative analysis of estimation accuracy and efficiency.

Estimation Method	Estimation Result (m3)	Estimation Time (s)
Proposed method	9102.79	182
ContextCapture	8644.23	500
Metashape	10,769.54	352
Pix4Dmapper	8710.59 ± 228.665	366

.

From Table 1, the proposed method can be used to quickly obtain the landslide deposit volume using the three-dimensional reconstruction based on smartphone imaging, and interpolate the bottom surface considering the steepest gradient. The accuracy of the volume estimation results of this method is comparable to that of commercial software, while the estimation efficiency is slightly better. Compared with other landslide deposit volume estimation methods, such as field investigation and multi-temporal DEM comparison analysis, the proposed method has the advantages of rapidity and high efficiency.

3.2.2. Dynamic Visualization of Disaster Scene

With the guidance of the knowledge template of landslide scene modeling, the landslide scene was constructed in OpenSceneGraph 3.4. The dynamic modeling and visualization of the whole disaster process, from the disaster-pregnant environment, triggering factors, disaster process, and disaster information, are realized. The modeling and visualization effects of the landslide scenes are shown in Figure 7. The knowledge-guided dynamic modeling and visualization of the landslide scenes can not only simplify complex modeling operations, but also improve the efficiency of the scene modeling and expression. In addition, it also has the advantages of rich expression content, good expression effect, complete and clear disaster information presentation, and strong information readability. Users can adjust the knowledge template according to different landslide disaster conditions and requirements to realize dynamic modeling and expression in different types of landslide scenes, which has good applicability and reliability and can provide support for landslide disaster emergency rescue and management.

3.2.3. Optimization of Disaster Scene

The landslide scene is imported into the disaster scene visualization and optimization prototype system for optimization considering visual saliency. Figure 8a is the original landslide expression scene, and Figure 8b is the landslide disaster scene optimized by the method proposed in this paper. The ROI within the landslide disaster area has a good rendering effect and high resolution. The user can clearly identify the landslide information. The landslide disaster gradually spreads outside of the boundary; the image gradually becomes blurred, and the blurred boundary is not obvious. Such a landslide scene expression is more in line with the visual characteristics of the human eye so that the user can focus on the ROI and the user’s information perception ability is improved.

To quantitatively analyze the performance of the optimization method proposed in this paper, we also compared the scene-rendering frame rate of the original and optimized scenes. As shown in Figure 9, the average scene-rendering frame rate reaches 80 frames per second (fps), which is obviously better than the rendering efficiency of the original scene. This shows that the scene optimization method proposed in this paper can significantly improve the efficiency of scene rendering and meet the higher requirements of scene visualization.

4. Conclusions and Future Work

In terms of the existing methods, there are shortcomings such as a long and costly deposit volume acquisition cycle, lack of knowledge guidance, complex operations for scene modeling visualization, and low scene-rendering efficiency. A three-dimensional visualization and optimization method for use in landslide disasters guided by knowledge is proposed. A rapid landslide deposit volume acquisition algorithm based on three-dimensional reconstruction is designed, and a landslide disaster scene modeling and visualization knowledge template is constructed to guide the dynamic modeling and visualization. Moreover, the scene optimization method, considering visual salience and human visual characteristics, is studied. This study realizes the whole research process from landslide deposit volume data acquisition to dynamic disaster scene visualization and optimization. The experimental case study analysis shows that the method proposed in this paper can achieve the rapid acquisition of deposit volume, and the results are equivalent to ContextCapture, Metashape, and Pix4Dmapper software; the approach has the advantages of high efficiency and convenience, and is capable of dynamically visualizing the whole disaster process with rich content expression and strong readability. Finally, the scene-rendering efficiency is improved, and the average rendering frame rate is stable at more than 80 frames/second. It can promote the expression and sharing of disaster information, and provide support for landslide emergency rescue management.

Despite the abovementioned achievements of the research, the rapid acquisition of deposit volume information method proposed in this paper is limited by the shooting scope and computing power of smartphones, and it is only suitable for small-scale, single landslide disasters. A more universal deposit information acquisition method will be developed in the future to provide data for the modeling and visualization of large-scale, more complex landslide scenes. Meanwhile, the above disaster data acquisition and scene visualization approaches are mainly for post-disaster emergency use. In the future, we will pay more attention to pre-disaster landslide information. Through the modeling and analysis of pre-disaster information, “passive disaster avoidance and relief” is changed to “active disaster prevention and management”, providing support for disaster prevention and mitigation. In addition, eye movement experiments can be conducted to improve the scene visualization and optimization method.