Sensor Fusion Based Pipeline Inspection for the Augmented Reality System

Abstract

Augmented reality (AR) systems are becoming next-generation technologies to intelligently visualize the real world in 3D. This research proposes a sensor fusion based pipeline inspection and retrofitting for the AR system, which can be used in pipeline inspection and retrofitting processes in industrial plants. The proposed methodology utilizes a prebuilt 3D point cloud data of the environment, real-time Light Detection and Ranging (LiDAR) scan and image sequence from the camera. First, we estimate the current pose of the sensors platform by matching the LiDAR scan and the prebuilt point cloud data from the current pose prebuilt point cloud data augmented on to the camera image by utilizing the LiDAR and camera calibration parameters. Next, based on the user selection in the augmented view, geometric parameters of a pipe are estimated. In addition to pipe parameter estimation, retrofitting in the existing plant using augmented scene are illustrated. Finally, step-by-step procedure of the proposed method was experimentally verified at a water treatment plant. Result shows that the integration of AR with building information modelling (BIM) greatly benefits the post-occupancy evaluation process or pre-retrofitting and renovation process for identifying, evaluating, and updating the geometric specifications of a construction environment.

1. Introduction

Building information modelling (BIM) is an intelligent process that provides information about the construction and management of building infrastructures. BIM includes a complex multiphase process that gathers information from multiple sources to model the components and tools used during the construction process [1,2,3]. BIM adoption in the construction process can help achieve greater building construction efficiency [4]. One of the greatest benefits of BIM technology is the three-dimensional (3D) visualization of the construction environment, which makes it easier for professional engineers to analyze the actual details of architectural, structural, and geometrical information [5]. It also enables a more intuitive understanding, facilitates timely communication, and improves error checking in designs. Furthermore, mapping 3D models with intelligent systems can provide immersive visualization as a virtual experience through computer-aided simulation.

Although 3D visualization using BIM technology allows engineers to efficiently visualize a construction environment, the overall effectiveness of real-time communication within the BIM environment is restrained due to the limited sense of immersion in virtual environments [6]. A previous research review showed that improving visualization and communication in the construction environment greatly helps engineers in terms of productivity and safety of construction processes [7]. Improving safety in the construction phase is described by Chantawit et al. [8] and Li et al. [9]. In the context of easy access and utilization of information in a BIM environment, augmented reality (AR) has been identified as an optimal solution due to its technology that superimposes the virtual objects onto the physical world. Also, BIM + AR can provide effective visualization of BIM data with a close connection in a physical context [10]. AR systems have already been successfully explored for various applications in the fields of architecture, engineering, and construction/facilities management (AEC/FM) [11,12,13,14,15,16]. There are also examples of using VR for BIM applications, as introduced by Sanz et al. [17], to provide training for assembly processes.

Few investigations have been carried out with respect to the utilization of BIM + AR at construction sites to guide professional workers in real-time. There is a limited level of realism due to a lack of sensor feedback, including the dimensions, textures, and spatial locations of the environment, as well as the confined utilization of real-time sensor information. Therefore, there is little perceptual and cognitive benefit to onsite workers. Also, digital biding of information in BIM is limited due to a lack of interactions between the real world and the virtual world. However, with efficient communication and onsite information sharing, AR has the potential to assist site engineers and construction workers in the BIM environment. Although the conventional role of AR is visualization, providing geometrical information and allowing for interactions with the data provided on site (virtual model) will also greatly benefit users (as shown in Figure 1).

This paper presents Light Detection and Ranging (LiDAR) and Camera fusion system for the pipeline virtual retrofitting process in BIM. The pose of the prebuilt 3D point cloud is aligned according to the pose of the real environment scene using real-time LiDAR scan point cloud by employing the registration algorithm. Next, a Red-Green-Blue (RGB) image of the prebuilt 3D point cloud is generated using the external camera parameters and merged with frames of a real-time video. To establish correspondence between the data from these two different sensing devices (i.e., the LiDAR and camera fusion), calibration between the camera and LiDAR performed prior to correspondence establishment. Additionally, identification of the geometric specifications of pipes using real-time LiDAR scan data helps workers choose specific targets for retrofitting. Finally, the proposed approach can be utilized in pipeline pre-retrofitting applications via the integration of pre-generated 3D models for upgradation or modification in an existing environment.

The rest of this paper is organized into the following sections: Section 2 presents a literature review of the research relevant to AR in BIM engineering and point cloud registration methods. In Section 3, the proposed methods, 3D point cloud alignment, 3D point cloud to 2D image extraction, and video alignment are presented. Virtual retrofitting is presented in Section 4, and Section 5 presents the experimental verification and results.

2. Literature Review

Augmented reality systems for BIM applications have been gaining increased attention from researchers of late. In recent years, many methodologies have been proposed for BIM + AR technologies. However, there is still a lack of sufficient investigations and insight into how real-time information can be utilized in BIM with an AR system. In this section, we discuss some of the AR systems proposed for BIM applications.

A recent study presented by Tabrizi et al. [18] concludes that BIM + AR can enhance an information retrieval, task efficiency, and productivity in building construction. The augmented reality proposed by Wang et al. [6] highlights the need for a structured methodology to integrate AR into BIM applications. In their paper, the authors concentrated more on assembling a new pipe plant. The proposed system helped workers find locations to place components at the construction site. Identification of object specifications in the model were displayed based on a QR code attached to an object. Each component in the model had a unique QR code with an associated list of attributes. The authors also investigated how BIM can be extended onsite by using AR technology.

Based on a post occupancy evaluation (POE) study, an AR system for BIM design applications was introduced by Freits and Regina [19]. In their paper, the geolocation was identified to augment a virtual object in the scene by considering the global positioning system (GPS) and digital compass data. Based on user movement, the position of the virtual object was maintained according to the point of view of the user. The accuracy of the method proposed in this paper depends on the position data obtained via GPS.

In Gupta et al. [20] proposed an AR system using LiDAR point cloud data to display dimensional information of objects on mobile phones. In their paper, the authors described an overall system based on two main steps. In the first step, the correspondence between the LiDAR point cloud data and the camera image was established using interior and exterior orientation parameters of the mobile camera and the coordinates of the LiDAR data points. A pseudo intensity image was generated from the 3D point cloud registered with the camera image by using an image registration method (SIFT) to ensure the correspondence between the point in the point cloud and the point (pixel) in the image. Their method utilizes random sample consensus (RANSAC) to estimate the homography for the projective transformation matrix. Estimation of the transformation matrix using RANSAC is an iterative process that requires more computational time. The authors also mentioned that false matches resulting from the SIFT findings should be removed prior to the RANSAC process. Estimating the orientation based on image feature analysis directly affects the computational time. Thus, adopting this method in real-time applications is difficult because of the computational time. The authors did not discuss the estimation of dimensional information of the objects.

Bae et al. [11] proposed a mobile AR system for context-aware construction and facility management applications. The system presented in their paper does not require any RF-based location tracking (e.g., GPS or WLAN) or optical fiducial markers for tracking a user’s position. Instead, a user’s location and orientation are automatically derived by comparing photographs from the phone with the 3D point cloud model created from a set of site photographs. Also, the authors described a detailed procedure for 3D reconstruction based on the structure-from-motion (SfM) algorithm, which estimates the 3D position of image features through feature extraction, matching, initial triangulation, and an optimization process.

LiDAR scanning systems acquire point clouds with enhanced reliability in three dimensions. When a similar scene or the environment of an area of interest is acquired multiple times from multiple views, alignment problems can arise depending on the application. In the last two to three decades, many efficient registration algorithms [21] have been introduced to address difficulties related to scenes or object registration; these problems can be classified into two categories depending on the use of the registration targets, i.e., rigid or non-rigid scenes or objects. In the case of rigid targets, a homogeneous transformation is applied to rigid surfaces, which can be modelled using six degrees of freedom. In the other case, non-rigid targets require the application of a coarse transformation to shapes or scenes, which vary with time.

BIM-based AR maintenance system (BARMS) [22] a building information modeling-based augmented reality maintenance system is a smartphone-based platform for an illumination function in darkness, showing a safe maintenance route and projecting BIM models. The BIM models are projected into a real environment for maintenance workers to analyze the correct locations of pipes. The proposed approach has the drifting problem which affects the positioning accuracy of application-anchoring locations.

The scene or environment in our proposed method comes under the first category, i.e., rigid targets. Currently, most state-of-the-art applications use the standard iterative closest point (ICP) algorithm introduced by Besl et al. [23] (or one of its variant algorithms) for rigid target alignment; these algorithms are superior in terms of their precision, performance, and robustness [21] under a set of restrictions. Initially, we utilized the standard ICP algorithm in the proposed method for the alignment of the point cloud [23]. ICP considers pairs of nearest points in the source and target point cloud data sets as correspondences and treats every point as having a corresponding point. ICP tries to find the optimal transformation between two datasets by iteratively minimizing a distance error metric.

A main disadvantage of the traditional standard ICP algorithm is that it assumes that the source cloud is acquired from a known geometric scene instead of being acquired through noisy measurements. However, it is generally impractical to obtain a perfect point-to-point alignment result, even after complete merging of the target and source data, due to discretization errors. Chen et al. [24] proposed the point-to-plane registration algorithm by considering that most of the range measurements are commonly sampled from a locally planar surface. The point-to-surface ICP algorithm proposed by Low et al. [25] tried to manage the discretization differences by modifying its constraint by permitting points to aggregate along the surface. Alshawa et al. [26] proposed a line-based matching ICP variant called the iterative closest line (ICL). In ICL, line features are obtained from range scans that are aligned to achieve the rigid body transformation.

Segal et al. [27] proposed a Generalized-ICP (G-ICP) algorithm that improves ICP by using the underlying surface structure of the point cloud to reject poorly corresponding points. G-ICP performs plane-to-plane matching by a probabilistic interpretation of the minimization process such that structural information from both the source cloud and the target cloud can be incorporated easily into the optimization algorithm.

In our previous work [28], we proposed a prototype framework for pipeline virtual retrofitting with a 4-in-1 alignment approach. In previous work, Velodyne VLP-16 LiDAR was used for prebuilt data acquisition as well as for real-time scan data and an external GoPro camera was used for the video capture. The basic setup and preliminary results demonstrated in an indoor environment provide the 4-in-1 alignment approach that can be utilized for the pipeline retrofitting applications. The current proposed work extends the previous work [28] by optimizing point cloud alignment algorithm for better time efficiency and accuracy. Further, LiDAR and camera calibration steps are enhanced with sensor fusion technique for pipeline inspection and retrofitting for the AR system in BIM application.

3. Proposed Method

This section describes the proposed AR system for pipeline retrofitting in BIM using LiDAR point cloud data and a camera scene fusion, as shown in Figure 2. This section begins with a description of the 3D prebuilt point cloud alignment using a real-time scan point cloud. Next, a prebuilt 3D point cloud was linked to the camera scene using the proposed approach. Step-by-step processes of aligning the prebuilt 3D point cloud information in the camera scene are also described in detail.

The proposed method includes two different scanning setups for the acquisition of the 3D prebuilt and real-time point cloud, as shown in Figure 3. The commercial Trimble TX5 3D laser scanner [29] was utilized to acquire the detailed 3D point cloud of the water treatment environment. As shown in Figure 3a, the processed 3D point cloud of the pipeline plant includes detailed information related to the environment. However, in the proposed method, more emphasis is given to pipeline parts than in the complete point cloud model, wherein only pipeline parts are segmented from the complete point cloud model for further processing. Figure 3b shows the segmented pipeline point cloud data highlighted in blue.

The Velodyne LiDAR PUCK (VLP-16) [30] 3D scanning platform is used to acquire the scan point cloud. Data from the Velodyne platform are only used for aligning the prebuilt cloud because a real-time scan point cloud of the environment (as shown in Figure 3c) is sufficient.

3.1. Real-Time Point Cloud Alignment

The accuracy and time efficiency of any AR system greatly relies on the performance of the localization and real-time alignment process used for mixing real scenes with virtual scenes. In the proposed AR framework, predefined positions are used to localize the platform. Therefore, the accuracy and time efficiency of the AR system directly depends on the performance of the real-time alignment process.

Many techniques have been implemented for pairwise 3D point cloud registration, including well-known ICP-based methods. In the proposed approach, the G-ICP [27] algorithm is used with an initial optimization for alignment of a prebuilt point cloud data set from real-time scan point cloud data (Figure 4).

Consistently aligning pairwise 3D point cloud data views into a complete model is referred to as registration. We mainly focused on the alignment of prebuilt point cloud data with the physical world in a global coordinate framework, regardless of pairwise registration. Aligning pairwise point cloud data from different sensors is complicated because each sensor has a different coordinate system and scale. Before the refined alignment, the prebuilt point cloud data from its local coordinate $(P_{local})$ is initially shifted and scaled offline to map with the real-time scan point cloud data’s coordinate system $(P_{global})$, as given in Equation (1).

$$P_{global}=\left(\frac{P_{local}}{S}\right)-T$$

(1)

Here, $T(x,y,z)$ is the global shift (translation matrix), $S$ is the global scale matrix, and $P$ is a 3D point.

However, the goal of our approach is to align the prebuilt point cloud data with respect to the relative position and orientation of a physical model in a global coordinate framework using a real-time scanned point cloud from a LiDAR sensor. Furthermore, if there are any changes in the plant then respective changes are updated (i.e. acquiring prebuilt point cloud again). As shown in the block diagram (Figure 4), prebuilt data is optimized via subsampling to minimize the point density for the correspondence grouping (the selection of key points). Using the prebuilt data for the alignment directly is computationally expensive for real-time alignment. This is because the correspondence grouping process requires a comparison between every prebuilt point and every point in the real-time scan data. Figure 5a,b show the complete prebuilt data set and the subsampled point cloud data, respectively, and Figure 3c shows the real-time scan point cloud data from a LiDAR sensor.

To align the prebuilt point cloud with the physical environment in the proposed method, the previously mentioned prebuilt point cloud is optimized via random sampling to improve the key point detector performance in the correspondence grouping. The random sampling method [31], which is implemented in the PCL library [32], is used to achieve the required optimal points.

$$P_o=randomsample(P_p)$$

(2)

Here, $P_o$ are the optimal sampled prebuilt points and $P_p$ is the original prebuilt point.

After obtaining the optimal points by sampling, the G-ICP technique is used for alignment by keeping the rest of the algorithm unchanged. First, the transformation matrix is computed by considering optimal prebuilt point cloud data ($P_o$) and real-time scan point cloud data ($P_r$) using Equation (3):

$$T=\arg {\min }_T{\displaystyle {\sum }_i{d}_i(}{P}_{r_i}+T{P}_{o_i}{T}^T)d_i^T$$

(3)

Here, $d_i=a_i-T{b}_i$ and $a_i$ and $b_i$ are point clouds corresponding to $P_o$ and $P_r$, respectively and $T$ is iteratively computed until the $P_o$ and $P_r$ aligned.

Once the transformation matrix from G-ICP is obtained, the optimal rotation and translation are applied to get the aligned data. A homogeneous transformation matrix $T_h$ is multiplied by the prebuilt point cloud $P_p$ to get the aligned prebuilt point cloud $P_t$.

$$T_h=\left[\begin{array}{cccc}\mathrm{c}\varphi \mathrm{c}\theta & \mathrm{c}\varphi \mathrm{s}\psi \mathrm{s}\theta -\mathrm{c}\psi \mathrm{s}\varphi & \mathrm{s}\varphi \mathrm{s}\psi -\mathrm{c}\varphi \mathrm{s}\psi \mathrm{s}\theta & t_x\\ \mathrm{s}\varphi \mathrm{c}\theta & \mathrm{c}\varphi \mathrm{c}\psi +\mathrm{s}\varphi \mathrm{s}\psi \mathrm{s}\theta & \mathrm{c}\psi \mathrm{s}\varphi \mathrm{s}\theta -\mathrm{c}\varphi \mathrm{s}\psi & t_y\\ -\mathrm{s}\theta & \mathrm{s}\psi \mathrm{c}\theta & \mathrm{c}\psi \mathrm{c}\theta & t_z\\ 0& 0& 0& 1\end{array}\right]$$

(4)

Here, $\psi$, $\varphi$, and $\theta$ represent the three rotations (roll, yaw, and pitch, respectively) between $P_o$ and $P_r$, and the column vector $\overrightarrow{t}={({\overrightarrow{t}}_x,{\overrightarrow{t}}_y,{\overrightarrow{t}}_z)}^T$ represents a translation.

$$P_t=P_p.T_h$$

(5)

Here, $P_t$ is the aligned original prebuilt point cloud, $P_p$ is the original prebuilt point cloud, and $T_h$ is the homogeneous transformation matrix obtained from the alignment of $P_o$ and $P_r$.

The result of prebuilt point cloud (in blue color) alignment with respect to the relative position and orientation of a physical model in the global coordinate framework using a real-time scan point cloud (in red color) from LiDAR is shown in Figure 6.

3.2. 3D Point-to-Image Pixel Correspondence

The geo-referenced prebuilt 3D point cloud data generated using the total station were acquired based on the world coordinate system; these were used to generate the corresponding 2D image representation. These points were projected onto a plane defined by the camera parameters. Furthermore, the generated 2D image representation of the 3D point cloud was superimposed over the camera image, as shown in Figure 7.

3.2.1. Sensor Calibration and Initial Alignment of Prebuilt Data

Calibration between the LiDAR sensor and camera is required to accurately align the point cloud information in an image. This is because an image captured using the camera is an RGB image and its characteristics are different from an image created from the point cloud acquired from the LiDAR sensor. Fusion of the camera and LiDAR sensor was recently used in many computer vision applications to enhance performance. Many researchers have proposed methods to calibrate RGB cameras with LiDAR sensors. For our proposed method, we used the calibration approach presented by Martin et al. [33] to calibrate the RGB camera with the LiDAR. The sensor setup in our method includes a Velodyne’s VLP-16 LiDAR sensor and an RGB camera integrated with the MTw XSens IMU [34].

Calibration of the RGB camera with LiDAR was mainly used to fuse different data types, such as images and point clouds, to establish correspondence between pixels in the camera image and points in the LiDAR sensor. A marker-based calibration method provides efficient alignment of the camera and LiDAR data. Geometrical displacement in the data from the two sensors was adjusted with respect to the marker position detected in both sensors.

Orientation of the scanning platform was obtained from the MTw IMU sensor, which outputs 3D orientations using different representations, such as unit quaternions, Euler angles, and a rotation matrix. We have considered a unit quaternion $Q_{mat}$ output from the IMU sensor for the initial alignment of a prebuilt point cloud and projected the 3D point cloud using camera intrinsic parameters $I$ by using Equation (6).

$$\begin{array}{l}{P}_{proj} = I \times P_{input} \\ C_{p_{align}} = Q_{mat} . P_{proj}\end{array}$$

(6)

where $I=\left[\begin{array}{ccc}1& 0& c_x\\ 0& 1& c_y\\ 0& 0& 1\end{array}\right] \times \left[\begin{array}{ccc}{f}_x& 0& 0\\ 0& f_y& 0\\ 0& 0& 1\end{array}\right]\times \left[\begin{array}{ccc}1& s/f_x& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$ is the known camera intrinsic parameters $c_x$ and $c_y$ are image center, $f_x$ and $f_y$ are focal lengths in pixel units and $s$ is the shearing factor, $P_{input}$ is the input from previous aligned data $P_t$ and $P_{proj}$ is the projected point cloud. $C_{Palign}$ is the aligned point cloud with respect to scanning platform using IMU sensor.

3.2.2. 2D RGB Image from a 3D Point Cloud

The prebuilt 3D point cloud data obtained from the LiDAR sensor are defined in the world coordinate system. To generate the corresponding RGB image, 3D points were projected onto an image plane defined by the intrinsic camera parameters and LiDAR and camera calibrated extrinsic calibration parameters as described in Equation (7).

$$P_{2d}\left[\begin{array}{c}x\\ y\\ 1\end{array}\right] = I\cdot E \cdot P_{3d}\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right]$$

(7)

where $E = \left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_x\\ r_{21}& r_{22}& r_{23}& t_y\\ r_{31}& r_{32}& r_{33}& t_z\end{array}\right]$ is extrinsic camera & LiDAR calibration matrix, with translation parameters $t_x$, $t_y$ and $t_z$, rotational parameters $r_{11}$ to $r_{33}$. $I$ is the camera intrinsic parameters, $P_{3d}$ is 3D point cloud and $P_{2d}$ is the converted 2D image coordinates.

The center of the camera lens is considered to be the origin of the camera coordinate system, as shown in Figure 8. The $Z$ direction is perpendicular to the $XY$ plane of the camera coordinate system, with the positive direction towards the objects to be imaged. The 3D LiDAR point cloud is projected onto the image plane defined by the camera sensor. The positional difference between the LiDAR and camera is obtained by the parameters $t_x$, $t_y$ and $t_z$. These parameters estimated by performing LiDAR and the camera calibration using the calibration method presented in Section 3.2.1.

As shown in Figure 8, the projection origin of the camera coordinate system occurs when the capturing image is defined at the center $(X,Y,f)$. The camera focal length (f), width (w), and height (h) of the generated image were used to define the image plane for the LiDAR point cloud projection. As shown in Figure 9a, the prebuilt 3D point cloud was generated from the LiDAR sensor and projected onto the image plane defined by the camera parameters using Equations (8) and (9).

$$x=-f\frac{a_1(X-X_0)+a_2(Y-Y_0)+a_3(Z-Z_0)}{a_7(X-X_0)+a_8(Y-Y_0)+a_9(Z-Z_0)}$$

(8)

$$y=-f\frac{a_4(X-X_0)+a_5(Y-Y_0)+a_6(Z-Z_0)}{a_7(X-X_0)+a_8(Y-Y_0)+a_9(Z-Z_0)}$$

(9)

Here, $\left(\begin{array}{ccc}{a}_1& a_2& a_3\\ a_4& a_5& a_6\\ a_7& a_8& a_9\end{array}\right)=\left(\begin{array}{ccc}1& 0& 0\\ 0& \cos \omega & \sin \omega \\ 0& -\sin \omega & \cos \omega \end{array}\right)\left(\begin{array}{ccc}\cos \phi & 0& -\sin \phi \\ 0& 1& 0\\ \sin \phi & 0& \cos \phi \end{array}\right)\left(\begin{array}{ccc}\cos \kappa & -\sin \kappa & 0\\ \sin \kappa & \cos \kappa & 0\\ 0& 0& 1\end{array}\right)$.

3.2.3. Point Cloud Augmentation in Video

After aligning the prebuilt point cloud according to the real-time scanner view using a 3D point cloud-based registration algorithm, a 2D RGB image of the 3D point cloud was extracted, as described in Section 3.2.2. Furthermore, the extracted point cloud image was aligned onto the camera image, as shown in Figure 10.

The intensity value in the point cloud image was updated to the camera frame using the following Equation (10):

$$if {\displaystyle \sum _{i=0}^{i=w}{\displaystyle \sum _{j=0}^{j=h}{C}_I}(i,j)}[r,g,b]=I[r:0, g:0, b:255]$$

(10)

Here, $w and h$ represent the width and height of the image, respectively.

Then, we replaced the intensity of the $F_I{(i,j)}^{th}$ pixel with the corresponding intensity of the $C_I{(i,j)}^{th}$ pixel, as shown in the equation below. Here, $F_I and C_I$ are the camera video frame and point cloud image, respectively.

$$F_I[r,g,b]=C_I[r,g,b]$$

(11)

As shown in Figure 10, the first row represents the frame of the video captured from the camera and image, while the second row represents the RGB image extracted from the prebuilt point cloud. Finally, the point cloud image alignment in the camera image is presented in the third row.

4. Pipeline Retrofitting

The proposed AR framework for the BIM technology enables retrofitting of pipeline plants. The proposed method was used to demonstrate real-world retrofitting of a water supply plant facility at KICT (Korea Institute of Construction Technology), as shown in Figure 11. Technological advancements in the equipment used in water supply plant facilities force owners to periodically retrofit treatment plants. There are some essential considerations that need to be kept in mind to ensure that the modification or extension of existing water plants utilizes fewer resources and is more cost effective. Retrofitting existing plants occurs for many reasons, but common reasons include pipeline reinforcement upgrades, the need to increase water requirements, and the integration of new technologies into existing facilities.

Virtual Pre-Retrofitting

As AR systems become more promising, tools for complex structure monitoring tasks in construction engineering, their application increasingly benefits onsite engineers by allowing for virtual retrofitting processes. Engineers can efficiently investigate the effects of rectifying defects, retrofitting or replacing defective parts, and extending existing portions of the plant. Also, virtual retrofitting allows for the feasibility of an overall rehabilitation process to be analyzed before any actual retrofitting, thereby reducing the possibility of uncertain challenges during the process.

Virtual pre-retrofitting is a process that provides an affordable computational platform for data analysis and integration to support decision-making in retrofitting applications. Figure 12 provides an overview of the retrofitting process. In this work, retrofitting of a pipeline plant at a water treatment site was considered as a special scenario for experimental evaluation. The virtual retrofit model presented in this article gathered and integrated pipeline data information utilizing LiDAR and camera sensors. The virtual model of the pipeline was acquired using the LiDAR sensor and a real environment scene obtained using a camera.

The aim of pipeline virtual pre-retrofitting is to allow users to evaluate tasks, such as pipeline geometry inspection, in real-time by using dimensional information (shown in Figure 13) presented in our previous work [35]. Additionally, engineers can use this method to modify existing pipeline structures and extend the pipeline with additional equipment in a virtual environment. As show in Figure 14, the proposed AR framework can be integrated into the pipeline virtual retrofitting process to modify existing equipment and add new parts to an existing plant.

5. Experimental Verification

To verify the proposed AR framework for pipeline retrofitting applications in real-time, tests and statistical analyses were conducted separately for each of the core processes involved in the proposed system. First, the robustness of the real-time registration process was verified by comparing existing registration methods. Next, the efficiency of the retrofitting process was evaluated by considering water treatment plant pipeline data.

5.1. 3D Point Cloud-Based Registration

To clarify the behavior of the alignment approach, the proposed method was evaluated by considering the time efficiency and its precision score. For validation, the point cloud with the original prebuilt data and optimal prebuilt data were compared utilizing G-ICP. Two different experimental real-time LiDAR scan point cloud data (in red color) sets were used for alignment of prebuilt data (in blue color), as shown in Figure 15 (i.e., View-1 and View-2).

Table 1. Time efficiency analysis for the prebuilt point cloud.

Method	Iterations		Run Time (s)		Alignment Score 1
	View-1	View-2	View-1	View-2	View-1	View-2
GICP	6	5	1.04	0.630	0.023	0.037
OGICP	8	8	0.18	0.097	0.021	0.032

¹ Alignment Score is the squared Euclidean distance between two sets of correspondence (Key points as shown in Figure 4).

shows the validation results for achieving final alignment of the prebuilt point cloud.

The time efficiency (which decreased by more than 80%) and the precision score (Alignment) of (which improved by ~10%) of the optimal prebuilt data are better than those of the directly used, original prebuilt data for our alignment method. The alignment of the original prebuilt point cloud was achieved with fewer iterations, whereas the number of optimal point cloud iterations increased by an average of three iterations.

5.2. Comparison with Image-Based Registration

Experiments were conducted for a water treatment pipeline plant to verify the efficiency of the image-based registration with 3D point cloud registration for the AR framework. Therein, four different views of the point cloud image, each with a different orientation, were considered to verify the accuracy of the image-based registration method, and the obtained results were compared with the point cloud-based registration method results with similarly oriented views of the prebuilt point cloud. In Figure 16, the left-side column shows four different views of the point cloud image with respect to the camera image; the detection of key points in the 3D point cloud registration is also shown.

In

Table 2. Run time and rotation accuracy for 2D image-based registration for scenarios shown in Figure 16.

Scenario	Run Time (s)	Translation/mm			Rotation/°
a	1.913	−0.012	0.040	0.00	0.000	0.000	−1.344
b	2.036	0.145	0.196	0.00	0.000	0.000	−1.414
c	1.996	0.045	0.029	0.00	0.000	0.000	−0.925
d	2.035	0.105	0.329	0.00	0.000	0.000	−1.098

and

Table 3. Run time and rotation accuracy for 3D point cloud-based registration for scenarios shown in Figure 16.

Scenario	Run Time (s)	Translation/mm			Rotation/°
a	0.189	−0.045	0.329	0.324	0.055	0.057	−0.177
b	0.189	−0.064	0.353	0.321	0.041	0.068	−0.337
c	0.153	−0.082	0.305	0.323	0.074	0.018	0.382
d	0.183	−0.039	0.319	0.324	0.078	−0.017	0.812

, the computational run time requirement of the proposed point cloud-based registration and image-based registration method were analyzed. Based on this analysis, we can draw the conclusion that the time requirement of the point cloud-based registration method is better than image-based registration.

The results obtained from the proposed point cloud registration method were compared with those of a typical image-based registration method to verify the accuracy of the proposed method. In

Table 4. Orientation accuracy comparison between the image and point cloud registration methods for scenarios shown in Figure 16.

Scenario	Actual Orientation (°)	Image Registration (°)	Point Cloud Registration (°)
a	+5	6.344	4.956
b	+10	11.414	9.871
c	−5	−5.925	−4.866
d	−10	−11.098	−9.789

, the orientation estimated from the two-registration approach was compared with predefined orientation scenarios. An experiment was conducted by using the input point cloud and image with predefined orientations of 5° and 10° in a clockwise direction as well as 5° and 10° in an anti-clockwise direction. The orientations estimated from the four different scenarios using these two approaches are presented in Table 4; this shows that our method estimates almost the same orientation as the predefined orientation, indicating that this procedure is reliable.

5.3. Point Cloud Alignment in the Video Frames

The results of the alignment of point clouds in the images extracted from video are shown in Figure 17. The first column in the figure contains frames of the real environment scene extracted from the video at 30 FPS, while the second column presents an RGB image of a 3D point cloud with a view corresponding to the camera image. The third column in the figure shows the final alignment in real-time. Computational time for alignment of points on each camera frame is 60 ms. This provides smooth visualizing experience to the user without any delay. Average time requirement for alignment varies depending on the Field of View (FOV) and resolution of camera. In this work, Samsung USB camera (Model: SPC-A1200MB) with 60° horizontal and 45° vertical FOV, and 640 × 480 resolution image is used.

5.4. Pipeline Virtual Retrofitting

The AR framework proposed in this paper is efficiently integrated with the pipeline retrofitting process. Since the proposed AR framework utilizes a 2D camera for actual scene acquisition, user interactions during the virtual retrofitting process with the 2D data are difficult and unnatural. Hence, the virtual retrofitting process was carried out in the 3D environment with the prebuilt point cloud. User modifications performed on the prebuilt cloud in the 3D environment were regularly updated onto the 2D camera scene. Figure 18 shows the experimental results of the proposed AR framework applied to pipeline retrofitting.

6. Conclusions and Future Work

This paper proposes a method for an augmented reality framework for pipeline pre-retrofitting in BIM engineering. The method presented in this study utilizes prebuilt 3D point cloud data and real-time 3D point cloud data generated from a LiDAR sensor along with a real environment scene captured by a 2D camera. Our method integrates RGB images extracted from the 3D prebuilt point cloud with frames of a video in real-time, which involves alignment of prebuilt point cloud data from the user viewpoint and establishing 3D point clouds in the camera image. This approach introduces the proposed AR framework, which can be utilized in industrial pipeline pre-retrofitting.

Experimental verification was conducted at water treatment plant to demonstrate the significance of the proposed AR framework for use in retrofitting applications. The resulting run time differences for four scenarios, as well as the orientation estimation results of two approaches (i.e., 2D image-based registration and 3D point cloud registration), show that 3D point cloud-based alignment is optimal for the AR framework.

This work can continue in several directions. Mainly, future work will involve integrating the localization of a scanning platform to enhance the system’s real-time capabilities. Also, efficient interaction techniques can be developed to benefit operators in BIM applications.