Automation in Construction 129 (2021) 103829

Contents lists available at ScienceDirect

Automation in Construction
journal homepage: www.elsevier.com/locate/autcon

Automated subsurface defects' detection using point cloud reconstruction

from infrared images
Marco Puliti a, b, Giovanni Montaggioli a, c, Alessandro Sabato a, *
a
Department of Mechanical Engineering, University of Massachusetts Lowell, 1 University Avenue, Lowell, MA 01854, USA
b
Department of Mechanical and Aerospace Engineering, Polytechnic of Turin, Corso Duca degli Abruzzi 24, 10121 Torino, Italy
c
Department of Computer Engineering, University of Modena and Reggio Emilia, Via Pietro Vivarelli 10, 41125 Modena, Italy

A R T I C L E I N F O A B S T R A C T

Keywords: Non-destructive evaluation (NDE) has become a reliable inspection tool to detect structural flaws in many en­
Energy efficiency gineering domains. Similarly, advancements in computer-vision made it possible to assess structural conditions
Infrared thermography from a set of digital images. This research introduces a novel inspection technique for detecting subsurface
Non-destructive evaluation
defects that cause heat loss. A combination of Structure from Motion (SfM) algorithms and infrared (IR) imaging
Structure from motion
Photogrammetry
has been developed for quantifying the severity of subsurface damages inducing energy loss on a scaled building
Automatic detection structure. Furthermore, an automated detection algorithm is proposed to segment the contours of the damages.
Results of experiments performed using different IR cameras prove that this approach can identify subsurface
defects and quantify their dimensions with an error below 5% when compared to the actual size of the damages.
The proposed approach can also be easily integrated with unmanned aerial vehicles for remote inspection and
damage detection on large-scale systems.

1. Introduction visual inspection because they often occur underneath the structure's
surface. For these reasons, inspection methods should highlight anom­
According to the Energy Information Administration (EIA), the en­ alies even when those defects are embedded in the structure's envelope.
ergy consumed by the building sector in the United States accounts for Non-destructive evaluation (NDE) techniques can provide significant
20% of the total energy consumed worldwide [1] and contributes be­ benefits for inspecting large areas and have shown versatility in several
tween 40 and 48% to the nation's CO2 emissions [2]. Advancing novel engineering domains by enhancing structural safety while reducing in­
techniques for increasing energy resilience in buildings is critical to spection costs and downtime [7]. Among the many available NDE
preserve their efficiency. Materials and design standards are continu­ techniques, infrared thermography (IRT) has proved capable to detect
ously evolving to reduce the construction sector's carbon footprint. For temperature distributions that could represent defects [8,9]. This paper
example, the concept of zero-emission buildings has contributed to a introduces a method to identify areas of heat loss using an integrated
significant decrease in energy consumption [3]. While these approaches computer vision algorithm. The proposed approach allows automated
mainly target new construction, existing buildings suffer from various quantification of subsurface defects in a building's envelope through the
factors that reduce their overall energy performance. As shown in Fig. 1, reconstruction of a three-dimensional (3D) model from a cloud of
heat loss in a residential building is related to air leaking through doors infrared (IR) images. The main novelty introduced here is that the vir­
and windows, moisture infiltrations, and poor wall insulation [4]. In tual 3D model is obtained directly from IR images used as input for a
addition to environmental, design, and materials factors, structural Structure from Motion (SfM) algorithm. It represents an attempt to
ageing plays a critical role in reducing a building's energy efficiency [5]. combine IR images and SfM to generate high-resolution, 3D thermal
Structural ageing leads to cracks, water infiltrations, unstable founda­ renderings that can be used to assess structural and energy attributes of
tions, joint corrosion, and roof defects that deteriorate the building's the systems being inspected. This research aims to i) develop an
energy rating over time [6]. Structural monitoring can help identify approach to generate point cloud reconstruction from IR images and ii)
defects at an early stage and mitigate their effects. advance a virtual mapping method for automatic heat loss detection and
Defects causing heat loss cannot always be detected using traditional related damage quantification. Because the resolution of IR imaging is

* Corresponding author.
E-mail addresses: marco.puliti@polito.it (M. Puliti), 217074@studenti.unimore.it (G. Montaggioli), Alessandro_Sabato@uml.edu (A. Sabato).

https://doi.org/10.1016/j.autcon.2021.103829
Received 18 February 2021; Received in revised form 18 June 2021; Accepted 13 July 2021
Available online 18 July 2021
0926-5805/© 2021 Elsevier B.V. All rights reserved.
M. Puliti et al. Automation in Construction 129 (2021) 103829

lower than their visible-spectrum counterpart, a pre-processing phase is have been proposed for damage identification. Some advantages of these
required to allow a proper reconstruction of the model using SfM approaches include noise resilience while extracting features repre­
[10,11]. If further developed, this technique can find applications senting damages during the training process without pre-processing and
beyond just energy audits. This research would lay the foundation for post-processing steps [22]. For example, deep learning and neural
monitoring the status of energy systems over time and reduce the risk network in IRT have been used to classify damages by analysing the
and cost of significant service disruptions [12]. cooling curves of two stainless steel plates having air, oil, and water
This paper is organised as follows. A summary of relevant IRT and infiltrations [23,24]. For engineering applications, a deep learning-
SfM applications is presented in Section 2. Section 3 describes the based method to detect subsurface damages in a steel truss bridge
methodology used to reconstruct the virtual model from the cloud of IR from IR images have been proposed and showed an overall accuracy of
images. It also discusses the segmentation algorithm used for defect ~98% in detecting defects when compared to ultrasonic pulse velocity
detection. The experiments conducted to validate the proposed IR-SfM tests [25]. The difficulty of detecting subsurface defects in real-world
method are analysed in Section 4, together with a validation of the scenarios is caused by the variability of the shapes of the damages.
automated damage detection algorithm. Finally, conclusions and sug­ Often, the use of just one of the approaches listed above makes the
gestion for future work are discussed in Section 5. detection difficult since the same defect can have multiple shapes and
sizes, so feature recognition methods used for damages such as concrete
2. Background and related studies cracks are not adequate to identify subsurface defects [26,27]. More­
over, the temperature difference between damaged and undamaged
IRT and SfM have been widely used for inspecting and assessing the parts in civil structures can be lower than 1.0 ◦ C, making the detection
structural conditions of many engineering systems. This section sum­ even more challenging [28]. Small temperature differences do not
marizes the studies focused on IR imaging for NDE and the working necessarily represent damages. They can indicate objects attached to a
principles and applications of SfM. surface such as peeling plaster, irregularities in the shape of structure, or
outliers due to environmental conditions such as different exposure to
light and shadows [29]. Finally, IR images have lower resolution
2.1. Infrared thermography for automated thermal assessment and
compared to visible images. Thus, a single approach may perform
damage detection
poorly, produce inaccurate identification, and make locating subsurface
defects a challenge [30]. For all these reasons, the algorithm presented
IRT is a non-contact NDE technique that allows measuring the energy
in this paper to detect automatically subsurface defects features a
emitted by the object under test (OUT) in the infrared spectrum. The
cascade of image processing techniques and approaches. As a result, the
OUT's surface emissivity is used to generate an IR image called ther­
contours of the detected damages obtained from the thermal data
mogram, which highlights anomalies in the temperature distribution
analysis are superimposed to the corresponding visible images to quickly
characterising faulty parts and damaged structural components [13].
locate the location and severity of flaws on the OUT.
Damages such as cracks, delamination, and water infiltration have a
different thermal transmittance than the surrounding undamaged areas
and are distinguishable as spots of different colours in the thermogram. 2.2. Principle of operation and application of structure from motion
IRT has been used in several domains such as environmental monitoring
[14], building diagnostics [8], and NDE of engineering structures [15] Advancements in computer-vision made point cloud reconstruction
and it is capable of detecting defects with accuracy comparable to other methods like SfM useful tools to assess the conditions of large-scale
commonly used techniques [16]. IRT can be categorised as active or structures [31]. SfM is a photogrammetric technique that returns 3D
passive. In the former, an external heat source warms up the OUT and renderings of an object from several retinal images. It relies on the
enhance the detection, while the latter relies on the object's own tem­ overlap between adjacent views of the OUT and does not require prior
perature. Although active IRT allows for better detection of the defects knowledge of the location, pose, or control points of the cameras to
by enhancing the temperature differential between the OUT and the reconstruct a 3D virtual model of the OUT. The camera's intrinsic and
environment, warming up the structure is unpractical in large-scale extrinsic parameters are simultaneously estimated directly from images'
applications [17]. The inspection is frequently performed once the descriptors, which are vectors representing distinct features of the scene,
external environmental conditions guarantee the largest temperature such as shapes, edges, colours, or textures. Those features are used to
difference (e.g. winter, cloudy and unwind day, and at dusk or dawn) identify specific points in the image(s) needed to reconstruct the object's
[9]. 3D coordinates and generate its digital model. Whenever two or more
Automated detection from IR images has also been the objective of descriptors are similar in different views of the OUT, a “keypoint” is
several research. Applications of IRT coupled to k-means clustering al­ generated to i) estimate the camera's spatial location by triangulation
gorithm showed promising results for automated detection and classi­ and ii) form a point cloud. Among different types of descriptors, Scale-
fication of delamination defects in concrete bridge decks [18]. Invariant Feature Transform (SIFT) is the most commonly used for
Following this example, energy minimization [19], wavelet transforms SfM due to its reliability and accuracy in both spatial and frequency
[20], deep learning [21], and convolutional neural network approaches domains [32]. The SIFT transforms the feature data from each image

Fig. 1. Contribution of the different parts of a building to the overall heat and energy loss.

2
M. Puliti et al. Automation in Construction 129 (2021) 103829

into scale-invariant coordinates and allows matching points among the instance, a low-power thermal sensor and a UAV are presented in [45] to
OUT's views to create a keypoint. The matching phase is optimised using detect heat loss in buildings' roofs and high-rise facades. Other examples
algorithms based on cost-weighing functions or neighbourhood re­ are available in [46], showing that IRT is widely used for energy audits.
lationships. An example of those algorithms is the approximate nearest However, these studies focus on the analysis of retinal IR images only.
neighbour that identifies similarities in the descriptors [33]. The goal is The proposed IR-SfM approach generates a point cloud model from IRT;
to recognise similar features in each set of descriptors to enable the it can increase the social acceptability of energy audit where consumers
reconstruction of 3D points. Finally, the Random Sample Consensus receive an intuitive 3D rendering of their building that shows precisely
(RANSAC) algorithm is implemented to discard outliers with a proba­ where energy is being lost.
bility threshold criterion [34]. The outcome is a sparse cloud of 3D data Some hybrid approaches are available in literature using remote
called “tie points” that act as nodes of the mesh used to reconstruct the sensing to generate the point cloud and IR images are superimposed on
rendering of the OUT. Fig. 2 summarizes the main steps in the SfM the reconstructed 3D model. Some researchers reconstructed the 3D
workflow used for reconstructing a 3D virtual model. point cloud model of building's facade using laser scanning [47] or high-
Once the 3D model is reconstructed, the coordinate system in which resolution RGB-images [48] while 2D IR images were used to assign
tie points are generated can be converted into a global reference system thermal intensities to the pre-oriented 3D point cloud rendering. Other
to obtain a virtual model that is a true representation of the physical works focused on the superimposition of IRT with radar-generated im­
OUT. Three possible options are available to scale the reconstructed 3D ages for assessing the conditions of a bridge deck [29] or the use of an IR
object to its physical dimensions. The first requires a minimum of three projector to generate a light pattern onto the scene and measure the
non-aligned Ground Control Points (GCPs). GCPs are distinct points in reflected signal [49]. More recently, studies were performed by maxi­
the scene with known coordinates that define the transformation matrix mizing the correspondence between RGB images used to reconstruct the
(i.e., three translations, three rotations, and a scaling factor) between point cloud and the IR images for heritage building analyses and large
the reconstructed and the global reference coordinate systems [35]. The area mapping [50–52]. Works related to the combination of thermal
second method relies on the knowledge of one known dimension of the images from eddy current pulsed thermography and features charac­
structure to create a scaling constraint for the 3D virtual model. By terization have been presented and introduced a novel descriptor
knowing one dimension on the physical OUT (e.g., width or height of the starting from the SIFT operator by adding global shape information to
facade), it is possible to compare that dimension with the same one detect thermal defects with complex geometry [53]. The main novelty
measured in the digital model and create a scaling constraint. The third introduced in this research is that the digital 3D model is obtained
option uses the camera's georeferenced data to ensure the 3D model has directly from IR images used as input for the SfM algorithm. By con­
the proper dimensions. When the georeferenced data of the cameras (i. verting IR-spectrum images into RGB images, and because only one type
e., images' GPS coordinates) are used, the accuracy of the 3D model is of spectral image is used, the robustness of the SIFT operator [54] is
increased because tie points are matched using the features of the scene preserved without the need to extract contours to register IR and visible
and the object's physical positions [36]. images to generate high-resolution virtual models of the OUT [55,56].
SfM has been applied in a variety of domains including heritage This procedure allows to directly identifying subsurface damages
buildings surveying [37] and civil infrastructures inspection such as without superimposing different datasets. Fig. 3 summarizes the work­
railroad tracks [38], bridges [39,40], and road pavements [41]. Point flow introduced in this paper to generate point cloud and virtual ren­
cloud reconstruction techniques are gaining relevance in many engi­ derings directly from IR images and the possibility of using image
neering domains due to the possibility to generate 3D representations of segmentation and edge detection algorithms to automatically extract the
objects, mesh and geometric models, and quantify surface deteriorations shape of subsurface defects that result in heat loss.
[42]. The majority of the proposed approaches focus on the recon­
struction of the OUT's surface only. In order to assess the subsurface 3. Methodology
defects of the OUT, different sensing techniques other than LIDAR and
digital images should be employed [43]. Because not all the damages This section introduces the methodology used to validate the possi­
can be detected when visible spectrum images are used to generate 3D bility of generating 3D virtual models with a SfM algorithm using IR
models, it motivates this research. In addition, the possibility to combine images directly and detecting subsurface defects that cause heat loss.
heat loss detection with unmanned aerial vehicles (UAVs) simplifies the This section provides details about the experimental setup and seg­
inspection of areas that are difficult or dangerous to access [44]. For mentation algorithm used for detecting damages.

Fig. 2. Structure from Motion (SfM) workflow diagram.

3
M. Puliti et al. Automation in Construction 129 (2021) 103829

Fig. 3. Workflow of the proposed IR-SfM methodology: Phase I - 3D digital reconstruction using SfM; Phase II - Automated damages' contours segmentation
and detection.

3.1. Experimental setup for point cloud acquisition and reconstruction still satisfactory for detecting subsurface defects that cause heat loss. The
second test was conducted outside, and the effects of external factors
The OUT analysed in this paper is a three-story scaled building, increase the possibility of having biased results compared to the indoor
shown in Fig. 4a. It has been coated with a 2.5⋅10− 2 m thick insulation test. Therefore, a Zenmuse XT2 IR imager fitting a 13 mm focal length
material with thermal conductivity k equal to 0.025 Wm− 1 K− 1. Defects lens was selected [58]. The camera has a 45◦ ×37◦ FOV, 640 × 512 pixels
simulating water infiltration (top story), lack of insulation (middle thermal resolution, NETD of 50 mK, and 4000 × 3000 pixels visible
story), and reduced thickness of the insulation material (bottom story) resolution. Another interesting feature is that this IR camera can be
were included in the building's coat to facilitate heat transfer to the embedded on a UAV to inspect large-scale structures. It is important to
external environment (see Fig. 4b). The OUT has uninsulated windows note that there is no direct comparison between the two experiments
and doors, gaps underneath the roof, and thermal bridges that change presented in this research and that the rationale for using two cameras is
the thermal behaviour of the structure locally. Two different tests were to prove the robustness of the proposed IR-SfM approach, even when
performed to validate the robustness of the IR-SfM method to create a 3D inexpensive sensors are used.
virtual model of the OUT that can be used for subsurface defect detec­ The image acquisition plan for the two cameras used to acquire the
tion. The first one used a relatively inexpensive hand-held IR camera, IR images is shown in Fig. 5a. For each test, the cameras were moved
while the second one was performed using a higher-resolution IR imager around the OUT on a spiral trajectory having ~1-m radius at ~0.02 m
that can be also embedded on an UAV (see Fig. 4c and d). increments to guarantee an overlap of 80% between adjacent images
For the first test that was performed indoor, a FLIR One Pro was used required by the SfM algorithm to work. Before acquiring the IR images,
[57]. This device is a hand-held camera with a 55◦ ×43◦ field of view the OUT was heated from the inside using an electric heater to simulate
(FOV), 160 × 120 pixels thermal resolution, a Noise Equivalent Tem­ the behaviour of a residential building's HVAC system and guarantee an
perature Difference (NETD) of 100 mK, and 1440 × 1080 pixels visible adequate temperature differential with the outside. This procedure
resolution that can be attached to a cell phone. The purpose of the first replicates what is listed in [59] and relies on the heat flux through the
test is to validate the proposed IR-SfM method and prove that the ac­ area of least thermal resistance (e.g., damages in the coat, windows,
curacy of the results obtained when low-resolution cameras are used is thermal bridges, etc.) to facilitate the detection of defects. Fig. 5b

Fig. 4. Three-story laboratory scale building with embedded defects and infrared cameras used during the test. a) Outside view with dimensions of the structure; b)
Inside view highlighting geometry and dimensions of damages; c) FLIR One Pro hand-held IR camera used for the first test; d) DJI Matrice 200 equipped with DJI
Zenmuse XT2 thermal camera used for the second test.

4
M. Puliti et al. Automation in Construction 129 (2021) 103829

Fig. 5. a) IR image acquisition plan with cameras trajectory to guarantee side overlap between adjacent images; b) Example of an IR-MSX image of the OUT with
highlighted sub surface defects.

presents an example of an IR image of the structure's facades recorded

using the Multi-Spectral Dynamic (MSX) feature of the IR camera. MSX
images contain the infrared spectrum components of an image and some
relevant features of the visible spectrum to enhance lines and edges'
sharpness and reduce blurriness.
Fig. 5b also shows that areas characterized by higher thermal
transmissibility (i.e., defects or absence of insulating material) are easily
identified in the IR image. Undamaged areas have a green colour (e.g.,
~30 ◦ C in the example shown in Fig. 5b). Thermal bridges are observed
in correspondence of the wooden junctions between stories in the form
of yellow strips (~35 ◦ C). Other portions of the structure with reduced
thermal resistance are shown with an orange-to-red colour (~45 ◦ C).
This include defects, gaps underneath the roof, and sections without
insulation material. Finally, the shades of blue correspond to back­
ground or components that are not relevant to the analysis. The red spot
in the bottom right part of Fig. 5b is not an intended defect, but the
thermal shadow of one of the heaters used to heat up the OUT. This
unwanted flaw is not considered for the analysis.

3.2. Image segmentation for automated detection

Automatic detection is a valuable tool to expedite the identification

of subsurface defects producing heat loss. Because the shape and
severity of the damages is case-specific, the detection algorithm has to
be customized to fit different types of flaws. Fig. 6 summarizes the main
steps of the developed segmentation algorithm. Because the algorithm
works by considering temperature values, image pre-processing is
necessary to extract temperature values from the RGB images and
Fig. 6. Algorithm pseudo-code.
generate an equivalent greyscale image (see Fig. 6, Steps 1 and 2). The
RGB format alone is not suitable for detection purposes because the
Fig. 7b). By normalizing the temperature obtained so that the minimum
intensity of each pixel is correlated to a colour scale. When different
temperature value corresponds to zero (i.e., black) and the maximum to
images of the same object are considered, each image may have its own
255 (i.e., white), the greyscale image shown in Fig. 7c is generated.
colour scale that results in pixels representing the same region of the
Subsequently, a neighbourhood operator is applied to enhance the
OUT to have different RGB values in different photos. The issue can be
image's contrast. A local intensity weighting (LIW) operator is used
overcome by considering a constant temperature scale for the whole set
before applying the Otsu thresholding method used to divide the image
of IR images analysed. By exploiting the emissivity of the OUT, tem­
in background and foreground [60] (see Fig. 6, Step 4). The LIW oper­
perature values are extracted from each thermogram to obtain (i) an
ator enhances the outcomes of the Otsu thresholding method that pro­
intensity temperature histogram and (ii) a temperature matrix having
vides adequate results only if the statistical distribution of the analysed
the exact size of the original image (see Fig. 6, Step 3).
image is unimodal [61]. When the ratio of the OUT-to-background pixel
An example of the operations performed in step 3 is shown in Fig. 7
population is high, the Otsu method does not perform well. This is the
for an RGB image having dimensions H x W. When the emissivity of the
case of IR images where the background has a significantly lower tem­
object's surface, the ambient temperature, and the specifications of the
perature than the OUT. The LIW operator is then used to mitigate the
thermal sensor are known, it is possible to pass from 3-channel RGB
variability of pixel distribution in the image [62]. Once the image is pre-
images to temperature values. Step 3 takes into considerations these
processed with the LIW, the Otsu thresholding method allows i) defining
parameters and converts the RGB intensity of each pixel into a tem­
a region of interest that corresponds to the OUT only (i.e., foreground)
perature value providing a conversion matrix with size H x W (see

5
M. Puliti et al. Automation in Construction 129 (2021) 103829

Fig. 7. Thermal images pre-processing steps for image segmentation: a) RGB thermogram acquired using an IR camera; b) Temperature matrix extraction using
ambient temperature and OUT emissivity; c) Greyscale image conversion.

and ii) reducing the computational time by a factor proportional to the 3D model of the OUT starting from a set of IR images. The methodology
number of pixels of the excluded background. The result is a single value presented in Section 3 is applied in both tests to study the accuracy of the
of intensity that works as a threshold and divides the intensity histogram SfM reconstruction and the detection capabilities of the image-
in two regions by separating the image into foreground (class 1, i) and processing algorithm.
background (class 2, j) (see Fig. 6, Step 5). Pixels in the background class
are all set to zero intensity (i.e., black) and only foreground pixels are 4.1. Hand-held IR camera test
considered.
Once the background of the image is discarded, a sliding temperature Following the workflow presented in Fig. 3 - Phase I and the acqui­
window is defined so that the distribution of temperature of the fore­ sition protocol shown in Fig. 5a, 140 IR images were acquired using the
ground can be inspected in smaller intervals [22]. This process helps to hand-held IR camera. Before reconstructing the digital model, it is worth
compare variation of temperature among adjacent pixels that can indi­ investigating some intermediate results. Considering the size of the
cate defects. The sliding temperature window is an essential feature of OUT, the possibility of using the camera's GPS for geotagging the images
the proposed algorithm because subsurface damages may have different or any GCP on the structure has been ruled out. These two options are
temperature values function of their typology and severity. The user can the preferable choice for applications of this technique on larger struc­
select an appropriate value based on the specific structure that is tures and guarantee increased resolution. In this research, the 3D model
inspected. Three parameters define the temperature window: size, ab­ is reconstructed using features in the images only. Fig. 8 presents the
solute minimum, and absolute maximum (see Fig. 6, Step 6). Absolute camera's locations estimations and details about the reconstruction of a
minimum and absolute maximum represent the window starting and 3D point. From Fig. 8a, it is possible to observe how an adequate lateral
ending points respectively. As discussed in Section 2.1, the temperature overlap is obtained. Moreover, Fig. 8b, shows an example of 3D point
difference between undamaged and damaged areas of a structure can be reconstruction (i.e., the green dot in the figure), where the same 2D
as low as 1.0 ◦ C. In the example shown in this paper, the size of the point (i.e., the blue dots in the three different views of the OUT) is
sliding temperature window was set to 2.5 ◦ C because of the severity of triangulated to compute its 3D coordinates. It is also worth noticing that
the defects embedded in the OUT. It is worth mentioning that defects before generating the 3D virtual model, the other intermediate result is a
with various severity can be identified and segmented by selecting cloud of data points in space (i.e., tie points) representing the shape of
different values of the temperature window. As an alternative, it is also the OUT.
possible to select smaller-size temperature sliding windows and iterating The minimum number of matches to generate a 3D point was set
step 6 for each of the sub-interval. The absolute minimum and maximum equal to three to obtain a satisfactory result. In this way, the point cloud
temperatures are the maximum and minimum values of the greyscale is densified enough, and the 3D model can be generated smoothly. Fig. 9
image or can be selected by the user based on the image's temperature shows two examples of point cloud and 3D virtual model reconstruction
distribution. By using this second option, parts that are not associated from two different views of the OUT obtained from the IR images
with defects are discarded and the probability of having outliers de­ collected using the hand-held camera.
creases. In this research, a Gaussian probability distribution was Fig. 9a and b show the point cloud (left) and the reconstructed 3D
assumed for the histogram of intensity. Because damages in active IRT model (right) obtained by fitting the tie points with a triangular mesh.
applications have higher temperature compared to their surroundings, it On average, ~5500 keypoints and 1500 matches per image were found,
is more appropriate to investigate high-temperature ranges. Therefore, providing a mean reprojection error of 0.81 pixels. Considering the
the absolute minimum and maximum temperatures were selected in the specifications of the hand-held camera, the camera-OUT working dis­
[− 1 s; +2 s] range, with s being the standard deviation of the Gaussian tance, and basic trigonometric formulations it is possible to compute the
distribution. Overall, 82% of the intensity's values are considered while spatial resolution of the reconstructed scene (i.e., distance between
the computational time is significantly decreased. Then, the Canny pixels centre in the actual scene). This value is equal to 0.6 cm⋅pixel− 1
method is applied to the IR images to detect the edge of regions of pixels and results in a reprojection error of 0.4 cm. This value represents a
characterized by sudden changes of temperature (Fig. 6, Step 7) [63]. To benchmark that is used to quantify the accuracy of the reconstructed 3D
finish, the contours of these regions representing damages are identified model. In Fig. 9, areas characterized by subsurface defects are identified
(Fig. 6, Step 8) and superimposed to the visible-spectrum images to with boxes and highlight heat loss through the envelope using the same
highlight their location on the structure (Fig. 6, Steps 9). colour's scale of the original IR images. In particular, two defects are
visible on the first story; the one corresponding reduced insulation
4. Results material as yellow rectangle in the middle and the heater's thermal
shadow on the right red spot. An orange rectangular-shaped spot iden­
Two tests are performed to validate the possibility to reconstruct a tify the water infiltration on the third story. Even if water infiltration is a

6
M. Puliti et al. Automation in Construction 129 (2021) 103829

Fig. 8. 3D model reconstruction from IR images using SfM: a) Estimation of camera's locations; b) 3D point reconstruction (green dot) based on the features (blue
dots) identified in different views of the OUT. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of
this article.)

Fig. 9. Examples of point cloud and model reconstruction's results for two different views of the OUT and damage location.

significantly large defect, it is partially covered by the thermal shadow of the defects in the physical world and those computed using the 3D
caused by the gap underneath the roof. Finally, thermal bridges between digital model.
stories and heat loss through doors and windows are visible. As observed, the average error in the measurements is 8.30%. This
The approach presented in this paper can also be used to quantita­ can be mainly attributed to the low resolution of the IR camera
tively assess the severity of damages by measuring distances, areas, employed, leading to errors in the digital reconstruction or clicking error
surfaces, and volumes directly from the reconstructed 3D model. As in selecting the exact point in the cloud to define the polygon enclosing
discussed in Section 2.2, the digital and global reference coordinate the defect. The accuracy is function of the defects' size, where higher
systems must be correlated to analyse the results. Therefore, before errors characterise small defects whose edges are more difficult to be
proceeding with a quantitative assessment of the severity of the defects, selected when clicking on the cloud and more difficult to be divided
the 3D model is scaled to the physical dimension of the OUT by defying a from the rest of the structure in the 3D model. Considering the spatial
scaling constraint. The dimension “D” shown in Fig. 4a has been used as resolution of the generated 3D model (i.e., 0.6 cm⋅pixel− 1), the clicking
scaling constraint. Then portions of the scaled 3D model that correspond error would result in an average error that range from 18.33% to 6.89%
to defects are manually selected to evaluate their area. The corners of a for the four defects considered. Nonetheless, results shown in Table 1
defect are selected using a closed curve, and the resulting polygon area confirm that the proposed approach provides a way to quantify the size
can be measured. Table 1 shows the comparison between the dimension of subsurface defects that result in energy loss.

Table 1 4.2. UAV-embedded IR camera test

Comparison between defects measured on the physical OUT and those measured
on the 3D model, hand-held camera test. In the second test, the method's accuracy is evaluated as the set of IR
Defect type Physical area 3D model area Error images are collected using a UAV platform. Following the procedure
(cm2) (cm2) (%) described in Section 3.1, 140 IR images were acquired during the
Water infiltration 300.00 278.00 − 7.30 experiment. The pre-processing phase is the same as the first test. It
Lack of insulation (Large) 100.00 101.10 +1.10 should be pointed out that the model reconstruction was performed
Lack of insulation (Small) 9.00 7.85 − 12.80 using IR images only and not the hybrid MSX format. This is due to the
Reduced-thickness 100.00 112.00 +12.00 fact that the Zenmuse XT2 camera does not have the MXS option
insulation
available. The purpose of the second test is to investigate the

7
M. Puliti et al. Automation in Construction 129 (2021) 103829

reconstruction's ability of the IR-SfM method in an outdoor application The data in Table 2 shows that the average error obtained is equal to
in which environmental factors such as wind, surface cooling, and 3.5%. It is worth pointing out that the most significant uncertainty
sunlight can bias the results. The variability of the outside environment characterizes the water infiltration defect because of the difficulty of
and the lower temperature differential justifies the use of a camera with defining a clear shape and boundaries for that defect due to the shadow
higher resolution. Fig. 10 shows the IR images acquisition, an example of generated by the gap underneath the roof. More geometrical regular
the IR images collected using the Zenmuse XT2 with defects highlighted, defects can be detected with better accuracy, and the overall average
and the estimation of the camera locations computed using the features error is below 3%. Considering the spatial resolution of the generated 3D
extracted from the thermograms. As it is possible to observe from model (i.e., 0.135 cm⋅pixel− 1), the clicking error would result in an
Fig. 10b, defects are mainly visualized by yellow spots highlighted average error that range from 11.22% to 2.63%.
within boxes. Relatively low-temperature values identify the majority of
the areas compared to those of the previous test. For instance, the spot
temperature measure of the water infiltration is 25.3 ◦ C, whereas its 4.3. Automatic detection of the damages
temperature was ~45 ◦ C during the first test (see Fig. 5b). During the
model reconstruction phase, ~4000 keypoints and 1000 matches per The analysis performed in this study relates to the automatic detec­
image were found on average. This corresponds to a reprojection error of tion of damages from IRT images. The algorithm described in Section 3.2
0.6 cm⋅pixel− 1, similar to the one that characterise the reconstruction has been used to segment areas with an increase in the heat transmitted
performed using the hand-held camera even though its thermal resolu­ through the OUT's surface and detect the contours of the damaged areas.
tion is three times lower. Two percentiles must be selected to initiate the analysis, apply the LIW-
This decrease in performance is caused by i) the use of IR images only Otsu thresholding methods, and discard outlier held in the Gaussian
rather than MSX images and ii) the effect of the environmental factors distribution tails. The best results are achieved considering − 1 s and + 2
that reduce the temperature difference between adjacent areas of the s as percentiles for the minimum and maximum temperature, respec­
OUT faster than in the previous experiment that was performed indoor. tively. Another parameter that needs to be set to complete the analysis is
Nonetheless, the accuracy of the measurements can be increased the sliding window step-size. Different damages can be detected
considering the Ground Sample Distance (GSD) associated with the test considering different temperature ranges. Fig. 12 shows an example of
or the GPS coordinates of the images. GSD represents the spatial reso­ temperature's spot measure for different defects to understand the
lution of the 3D model that is the distance between the centres of pixels importance of using a sliding temperature window, along with its tem­
in the actual scene. During the second test, GSD helped to increase the perature histogram. The algorithm runs in each temperature interval,
spatial resolution to 0.135 cm⋅pixel− 1 providing a reprojection error of searching for damages just in those clusters of pixels characterized by a
0.1 cm. Fig. 11 shows two examples of point cloud and 3D virtual model temperature within the specified range. By repeating this process for
reconstruction of two different views of the OUT obtained from the IR each interval, the relevant defects' contours are detected and super­
images collected using the Zenmuse XT2 camera. From a qualitative imposed to the original visible image. It is essential to notice that the
standpoint, the relevant defects areas are visible and highlighted by a results are affected by the choice of the parameters used in the algo­
change of colour with respect to adjacent undamaged areas. The water rithm; especially by the temperature step-size. Large temperature
infiltration in the third story is more visible than in the first test, and the window-steps decrease the computational time, but only some defects
shadow created by the gap underneath the roof is not as invasive as in are detected reducing the likelihood of identifying the smaller or less
the previous example. Moreover, the 3D reconstruction is less affected severe ones.
by image distortion, and the reprojection error is smaller than those On the other hand, small temperature window-steps increase the
calculated for the hand-held camera's experiment. Finally, thermal computational time significantly because the algorithm will run several
bridges are identified by yellow strips surrounded by green areas at a times. The probability of having outliers and false-positive increases too
lower temperature. since any temperature gradient could be confused for a defect. As shown
Similarly to what was done for the hand-held test, a scaling in Fig. 13, the trade-off and more accurate results are obtained with a
constraint was obtained by comparing the measure of one dimension of temperature window step-size of 2.5 ◦ C.
the OUT with the corresponding dimension on the digital model. In this Fig. 14 presents the results of the automated damage detection al­
way, a reference system can be generated, and the shape of the actual gorithm when different IR images are considered. In Fig. 14, only two
damage can be measured. Further validation of the proposed IR-SfM views of the OUT are shown but similar results were obtained for other
method is provided with a quantitative analysis of the defects' IR images not shown here for the sake of brevity. By looking at the re­
severity, as shown in Table 2. sults, it is observed that the detection appears to be more successful in
the hand-held IR camera test (i.e., Fig. 14a). The reason is because of the

Fig. 10. Validation of the proposed IR-SfM approach during the second test: a) IR images acquisition using the camera embedded on a UAV; b) example of thermal
image collected; c) Estimation of camera's locations in the digital model and 3D point cloud reconstruction of the OUT.

8
M. Puliti et al. Automation in Construction 129 (2021) 103829

Fig. 11. Examples of point cloud and model reconstruction's results for two different views of the OUT.

approach to conduct an energy audit. Higher temperature difference

Table 2
to facilitate the flux of thermal energy and external environmental
Comparison between defects measured on the physical OUT and those measured
conditions should be considered when planning the inspection.
on the 3D model, UAV-embedded camera test.
Defect type Physical area 3D model area Error 4.4. Quantitative analysis of the damages
(cm2) (cm2) (%)

Water infiltration 300.00 315.15 +5.05 The accuracy of the proposed algorithm is quantified by developing a
Lack of insulation (Large) 100.00 98.80 − 1.20
metric. The algorithm is assessed in terms of i) damage detection and ii)
Lack of insulation (Small) 9.00 9.29 +3.22
Reduced-thickness 100.00 104.76 +4.76 damage localisation capabilities. For the damage detection part, the
insulation aptitude of the algorithm to predict damage is tested considering the F1-
score. The F1-score summarizes the damage detection capabilities of the
algorithm and represents the harmonic mean of Precision (Pre) and
higher temperature differential between damaged and undamaged areas Recall (Rec) as defined by Eq. (1).
that characterized that test. From Fig. 14a, it is also possible to notice
that the algorithm identifies the shape of the defects from their tem­ F1 = 2⋅
Pre⋅Rec
(1)
perature distribution. All defects are within closed lines, and each one is Pre + Rec
distinct and separate. From the analysis of Fig. 14b, it is observed that
Where precision is defined as the number of times damages have been
there is not a clear separation in adjacent defects. For example, the
detected correctly (TP) compared to the number of times they have been
defect representing the water infiltration on the third story merges with
observed by the algorithm (TP + FP) and recall is defined as ratio be­
the thermal bridge between the second and the third story and the lack
tween correct detections (TP) and the number of detections the algo­
of insulation defect on the second story. Because of the small tempera­
rithm is supposed to spot (TP + FN) as shown in eqs. (2) and (3).
ture difference between undamaged and damaged parts in the second
test, damages' contours in the processed images are not identified as TP
Pre = (2)
clearly as in the first test. TP + FP
The results shown in Fig. 14 also provide useful insights about the
best time and environmental conditions to apply the proposed IR-SfM

Fig. 12. Analysis of temperature difference between damaged and undamaged areas to properly select the proper parameters of the algorithm: a) Temperature's spot
measure on the OUT; b) Temperature distribution in the IR image shown in Fig. 5a.

9
M. Puliti et al. Automation in Construction 129 (2021) 103829

due to failure in detecting minor flaws in the OUT. The F1-score is

similar for both tests, while the presence of damages is predicted with
94% accuracy for the first test and 93% for the second one. This slight
variation can be due to the different sample sizes used for this analysis
and the temperature difference between the inside and outside of the
OUT that was larger in the hand-held IR camera test than the UAV IR
camera test. For what concerns the Jaccard similarity index, the UAV IR
camera test, carried out with a more performing thermal imaging
camera, shows slightly better performance, as the increased number of
pixel in the images can generate more pronounced thermal gradients.

5. Conclusion and future works

The presented work aims to develop a novel method to assess heat

loss and subsurface structural defects in a remote digital environment
that can change the way thermal loss in buildings is assessed and
quantified. The goal is to obtain an automated, reliable, and fast
Fig. 13. Data accuracy as a function of the temperature window's size. approach to evaluate defects that can reduce the energy efficiency of the
system being inspected. Those defects are usually sub-superficial, thus
not detectable with traditional visual inspection or other procedures that
TP
Rec = (3) rely on visible spectrum imaging. In this work, two techniques that are
T P + FN
used as stand-alone methods are combined: infrared thermography
Precision provides information about false positive (FP) predictions (IRT) and Structure from Motion (SfM). Infrared (IR) images collected
made by the algorithm. Recall represents information about false using a handheld camera and an unmanned aerial vehicle (UAV)
negative (FN) detections produced by the algorithm. equipped camera are used to generate three-dimensional (3D) models
To quantify the accuracy of the damage localisation inside the IR using a SfM algorithm. The possibility of generating point cloud virtual
images, the Intersection over Union (IoU) approach has been imple­ models would allow homeowners, consumers, and contractors to receive
mented. This approach, also known as the Jaccard similarity index, an intuitive 3D rendering of their building that shows precisely where
considers two elements: ground-truth and damage detected. The first energy is being lost. In addition, the generated 3D point cloud can be
represents the coordinates of the damage's perimeter. It relies on labels implemented into Building Information Modelling (BIM) approaches,
to define regions of interest in the images to save the information in advancing off-site construction and analysis techniques. This approach
JavaScript Object Notation (JSON) format using Labelbox [64]. The guarantees faster energy audits with detection accuracy compared to
second one is the perimeter's coordinates of the detected damage and traditional methods. Using UAV-based inspection, it would be possible
represents the algorithm's output. Jaccard similarity index is calculated to inspect inaccessible structure locations as well as reconstruct large
as the ratio between the intersection of ground-truth and observed structures thermal models and inspect them remotely with the result of
damage and their union. Its formula is described in eq. (4), where A increasing inspectors' efficiency while reducing costs and risks associ­
represents the area of the ground-truth and B represents the area of the ated with the inspection. The method has been validated by employing a
damage detected by the algorithm. scaled building model that embeds different types of subsurface defects.
A∩B Both tests simulated an energy audit inspection process where the
IoU = (4)
A∪B
IoU ranges between 0 and 1; the more significant the overlap be­ Table 3
tween these two parameters, the closer IoU will be to 1, representing a Damage detection's results with different metrics.
more accurate detection. For the tests shown in this paper, the algo­
Test Sample Precision Recall F1 - Accuracy IoU
rithm's accuracy was validated considering a total of 80 IR images. The size score
outcomes of such validation obtained using the metrics described above
Hand-held IR 44 0.91 0.84 0.87 0.94 0.78
are summarized in Table 3. camera
As observed, for both tests performed, Precision and Recall have UAV IR 36 0.90 0.85 0.87 0.93 0.83
almost identical values, with Recall being slightly lower than precision camera

Fig. 14. Damage detection's results obtained from the automated damage detection algorithm: a) Hand-held camera test; b) UAV-embedded camera test.

10
M. Puliti et al. Automation in Construction 129 (2021) 103829

structure is heated from the inside and IR images are captured from the [8] A. Kylili, P.A. Fokaides, P. Christou, S.A. Kalogirou, Infrared thermography (IRT)
applications for building diagnostics: a review, Appl. Energy 134 (2014) 531–549,
outside to detect discontinuities in the thermal flux emitted through the
https://doi.org/10.1016/j.apenergy.2014.08.005.
walls of the structure. During the first test, an accuracy of 8.3% is ach­ [9] F.N.C. Shuhei Hiasa, R. Birgul, A. Watase, M. Matsumoto, K. Mitani, A review of
ieved when the size of the defects measured in the 3D model is compared field implementation of infrared thermography as a non-destructive evaluation
with their actual sizes. In the second test, results showed that the pro­ technology, Comput. Civil Build. Eng. (2014) 1715–1722, https://doi.org/
10.1061/9780784413616.213.
posed approach identifies subsurface defects and quantify their di­ [10] A. Sabato, M. Puliti, C. Niezrecki, Combined Infrared Imaging and Structure from
mensions with accuracy of 3.5%. The accuracy of these results is also Motion Approach for Building Thermal Energy Efficiency and Damage Assessment
function of the spatial resolution of the generated 3D model, affected by vol. 86, SPIE Proceedings, 2020, https://doi.org/10.1117/12.2558796.
[11] A. Sabato, C. Niezrecki, Use of Infrared Imaging for Structure from Motion
the clicking error, resulting in an inaccurate selection of point in the Assessment of Heat Loss in Buildings, SPIE Proceedings, 2019, https://doi.org/
cloud that defines the polygon enclosing the defect. The research also 10.1117/12.2515077.
shows the possibility of using automatic detection via a segmentation [12] K. Willis, H.H. Loa, Measuring the Resilience of Energy Distribution Systems,
RAND Corporation, 2015. https://www.rand.org/pubs/research_reports/RR883.
algorithm to highlight the location of the damages from IR images. The html.
proposed method uses the difference of temperature between clusters of [13] C.A. Balaras, A.A. Argiriou, Infrared thermography for building diagnostics, Energy
neighbouring pixels to determine the contours of the defect. It also su­ Build. 34 (2002) 171–183, https://doi.org/10.1016/S0378-7788(01)00105-0.
[14] M. Lega, J. Kosmatka, C. Ferrara, F. Russo, R.M.A. Napoli, G. Persechino, Using
perimposes the detected contours on the corresponding visible image to advanced aerial platforms and infrared thermography to track environmental
locate the position of the flaws on the structure. The proposed algorithm contamination, Environ. Forensic 13 (4) (2012) 332–338, https://doi.org/
shows to be robust and able to detect defects with an accuracy above 10.1080/15275922.2012.729002.
[15] T. Omar, M.L. Nehdi, Remote sensing of concrete bridge decks using unmanned
90% for both tests considered.
aerial vehicle infrared thermography, Autom. Constr. 83 (2017) 360–371, https://
If further developed, the use of the segmentation algorithm can in­ doi.org/10.1016/j.autcon.2017.06.024.
crease the accuracy of the damage quantification because the contours [16] R.W. Martin, A. Sabato, A. Schoenberg, R.H. Giles, C. Niezrecki, Comparison of
of the defect can be automatically selected based on the temperature nondestructive testing techniques for the inspection of wind turbine blades’ spar
caps, Wind Energy 21 (11) (2018) 980–996, https://doi.org/10.1002/we.2208.
difference between adjacent points of the cloud instead of being selected [17] N.P. Avdelidis, A Look on Thermography: From Passive to Active NDT & E Surveys,
by the user manually. Further testing should be done to assess the ac­ SPIE Proceedings: Thermosense XXIX, 2007, https://doi.org/10.1117/12.719205.
curacy of the proposed methodology on real-world structures and [18] T. Omar, M.L. Nehdi, T. Zayed, Infrared thermography model for automated
detection of delamination in RC bridge decks, Constr. Build. Mater. 168 (2018)
infrastructure. In this scenario, UAVs can be used to acquire images and 313–327, https://doi.org/10.1016/j.conbuildmat.2018.02.126.
guarantee enough overlap between adjacent images and allow using a [19] Z. Wenjing, W. Xueqiang, Design of Infrared Small Target Detection Software Based
SfM algorithm. For these reasons, additional considerations about the on Energy Minimization Constraints, IEEE Proceedings: International Conference
on Intelligent Systems Design and Engineering Applications, 2014, https://doi.org/
planning of the photogrammetric mission (e.g., drone's speed, sampling 10.1109/ISDEA.2014.88.
frequency, working distance, etc.) should also be considered. Future [20] W. Ting, S. Yang, Weak and Small Infrared Target Automatic Detection Based on
works will also focus on the automated detection of damages using Wavelet Transform, IEEE Proceedings International Symposium on Intelligent
Information Technology Application, 3, 2008, https://doi.org/10.1109/
machine learning or deep learning techniques to improve the results of IITA.2008.537.
the automatic damage detection procedure and evaluate the state of [21] N. Sang, T. Zhang, Segmentation of FLIR images by Hopfield neural network with
target objects by processing information from the set of images acquired edge constraint, Pattern Recogn. 34 (2001) 811–821, https://doi.org/10.1016/
S0031-3203(00)00041-8.
with the thermal camera. Some examples include the use of larger
[22] Y.J. Cha, W. Choi, O. Büyüköztürk, Deep learning-based crack damage detection
datasets of subsurface damages to train a neural network, refine the use using convolutional neural networks, Comput. Aided Civil Infrast. Eng. 32 (5)
of the sliding window to facilitate the recognition of defects having (2017) 361–378, https://doi.org/10.1111/mice.12263.
different temperature [65], or add information about the materials' [23] Y. Duan, S. Liu, C. Hu, J. Hu, H. Zhang, Y. Yan, J. Meng, Automated defect
classification in infrared thermography based on a neural network, NDT & E Int.
emissivity to distinguish between different parts of the structure (e.g., 107 (2019) 102147, https://doi.org/10.1016/j.ndteint.2019.102147.
windows or masonry/concrete). [24] Q. Fang, B.D. Nguyen, C.I. Castanedo, Y. Duan, X. Maldague II, Automatic defect
detection in infrared thermography by deep learning algorithm, Thermosense:
Thermal Infrared Appl. XLII 11409 (2020), https://doi.org/10.1117/12.2555553
(114090T pp.).
Declaration of Competing Interest [25] R. Ali, Y. Cha, Subsurface damage detection of a steel bridge using deep learning
and uncooled micro-bolometer, Constr. Build. Mater. 226 (2019) 376–387, https://
doi.org/10.1016/j.conbuildmat.2019.07.293.
The authors declare that they have no known competing financial [26] R. Ali, D. Gopal, Y. Cha, Vision-based concrete crack detection technique using
interests or personal relationships that could have appeared to influence cascade features, in: SPIE Smart Structures and Materials + Nondestructive
the work reported in this paper. Evaluation and Health Monitoring vol. 10598, 2018, https://doi.org/10.1117/
12.2295962 (105980L pp.).
[27] S. Dorafshan, R.J. Thomas, M. Maguire, Comparison of deep convolutional neural
References networks and edge detectors for image-based crack detection in concrete, Constr.
Build. Mater. 186 (2018) 1031–1045, https://doi.org/10.1016/j.
conbuildmat.2018.08.011.
[1] EIA, International Energy Outlook 2016-Buildings Sector Energy Consumption,
[28] M. Clark, D. McCann, M. Forde, Application of infrared thermography to the non-
Energy Information Administration, 2016. https://www.eia.gov/forecasts/ieo/buil
destructive testing of concrete and masonry bridges, NDT & E Int. 36 (4) (2003)
dings.cfm.
265–275, https://doi.org/10.1016/S0963-8695(02)00060-9.
[2] P.H. Shaikh, N.B.M. Nor, P. Nallagownden, I. Elamvazuthi, T. Ibrahim, A review on
[29] S. Hiasa, R. Birgul, M. Matsumoto, F. Necati Catbas, Experimental and numerical
optimized control systems for building energy and comfort management of smart
studies for suitable infrared thermography implementation on concrete bridge
sustainable buildings, Renew. Sust. Energ. Rev. 34 (2014) 409–429, https://doi.
decks, Measurement 121 (2018) 144–159, https://doi.org/10.1016/j.
org/10.1016/j.rser.2014.03.027.
measurement.2018.02.019.
[3] E. Pujadas-Gispert, C.C. Korevaar, M. Alsailani, S.P.G. Moonen, Linking
[30] S. Abu, S. Yaghi, S. Alkass, O. Moselhi, Concrete bridge deck condition assessment
constructive and energy innovations for a net zero-energy building, J. Greenbuild.
using IR thermography and ground penetrating radar technologies, Autom. Constr.
15 (2020) 153–184, https://doi.org/10.3992/1943-4618.15.1.153.
81 (2017) 340–354, https://doi.org/10.1016/j.autcon.2017.04.006.
[4] S. Yacob, A.S. Ali, C.P. Au-Yong, Establishing relationship between factors affecting
[31] A. Khaloo, D. Lattanzi, A. Jachimowicz, C. Devaney, Utilizing UAV and 3D
building defects and building condition, J. Survey. Const. Property 10 (2019)
computer vision for visual inspection of a large gravity dam, Front. Built Environ. 4
31–41, https://doi.org/10.22452/jscp.vol10no1.3.
(2018) 1–16, https://doi.org/10.3389/fbuil.2018.00031.
[5] N.L. Othman, M. Jaafar, W.M.W. Harun, F. Ibrahim, A case study on moisture
[32] D.G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J.
problems and building defects, Procedia Soc. Behav. Sci. 170 (2015) 27–36,
Comput. Vis. 60 (2004) 91–110, https://doi.org/10.1023/B:
https://doi.org/10.1016/j.sbspro.2015.01.011.
VISI.0000029664.99615.94.
[6] N.N.O. Bakri, M.A.O. Mydin, General building defects: causes, symptoms and
[33] S. Dhanabal, S. Chandramathi, A review of various k-nearest neighbor query
remedial work, Europ. J. Technol. Design 3 (2014) 4–17, https://doi.org/
processing techniques, Int. J. Comput. Appl. 31 (2011) 14–22. https://www.
10.13187/ejtd.2014.3.4.
ijcaonline.org/archives/volume31/number7/3836-5332.
[7] D.M. McCann, M.C. Forde, Review of NDT methods in the assessment of concrete
[34] M.A. Fischler, R.C. Bolles, Random sample consensus: a paradigm for model fitting
and masonry structures, NDT E Int. 34 (2001) 71–84, https://doi.org/10.1016/
with applications to image analysis and automated cartography, Commun. Assoc.
S0963-8695(00)00032-3.

11
M. Puliti et al. Automation in Construction 129 (2021) 103829

Comput. Machinery 24 (6) (1981) 381–395, https://doi.org/10.1145/ [50] D. Lin, M. Jarzabek-Rychard, X. Tong, H.G. Maas, Fusion of thermal imagery with
358669.358692. point clouds for building Façade thermal attribute mapping, ISPRS J. Photogramm.
[35] Pix4D, Offline Getting Started and Manual. https://support.pix4d.com/h Remote Sens. 151 (2019) 162–175, https://doi.org/10.1016/j.
c/en-us/articles/204272989-Offline-Getting-Started-and-Manual-pdf-#gsc.tab=0, isprsjprs.2019.03.010.
2021 (Accessed on April 2021). [51] G. Patrucco, G. Cortese, F.G. Tonolo, A. Spanò, Thermal and optical data fusion
[36] S. Liu, H. Zhang, S. Kawai, H. Nobuhara, 3D point cloud reconstruction based on supporting built heritage analyses, Int. Arch. Photogram. Remote Sensing Spatial
stereo camera and RTK-GPS with high accuracy under sequential correction, in: Inform. Sci. 43 (2020) 619–626, https://doi.org/10.5194/isprs-archives-XLIII-B3-
IEEE 9th Global Conference on Consumer Electronics, 2020, pp. 90–93, https:// 2020-619-2020.
doi.org/10.1109/GCCE50665.2020.9291803. [52] Y. Hou, R. Volk, M. Chen, L. Soibelman, Fusing tie Points’ RGB and thermal
[37] E. Ippoliti, A. Meschini, F. Sicuranza, Structure from motion systems for information for mapping large areas based on aerial images: a study of fusion
architectural heritage. A Survey of the Internal Loggia Courtyard of Palazzo dei performance under different flight configurations and experimental conditions,
Capitani, Ascoli Piceno, Italy, Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. Autom. Constr. 124 (2021) 103554, https://doi.org/10.1016/j.
40 (2015) 53–60, https://doi.org/10.5194/isprsarchives-XL-5-W4-53-2015. autcon.2021.103554.
[38] A. Sabato, C. Niezrecki, Evaluation of using structure from motion optical [53] X. Chen, G. Tian, J. Wu, C. Tang, K. Li, Feature-based registration for 3D Eddy
techniques for structural health monitoring of railroad tracks, Struct. Health Monit. current pulsed thermography, IEEE Sensors J. 19 (2019) 6998–7004, https://doi.
(2017), https://doi.org/10.12783/shm2017/14228. org/10.1109/JSEN.2019.2911699.
[39] A. Khaloo, D. Lattanzi, K. Cunningham, R. Dell’Andrea, M. Riley, Unmanned aerial [54] D.G. Lowe, Object recognition from local scale-invariant features, in: 7th IEEE
vehicle inspection of the placer river trail bridge through image-based 3D International Conference on Computer Vision, 1999, pp. 1150–1157, https://doi.
modelling, Struct. Infrastruct. Eng. 14 (1) (2018) 124–136, https://doi.org/ org/10.1109/ICCV.1999.790410.
10.1080/15732479.2017.1330891. [55] C. Aguilera, F. Barrera, F. Lumbreras, A.D. Sappa, R. Toledo, Multispectral image
[40] G. Morgenthal, N. Hallermann, J. Kersten, J. Taraben, P. Debus, M. Helmrich, feature points, Sensors 12 (9) (2012) 12661–12672, https://doi.org/10.3390/
V. Rodehorst, Framework for automated UAS-based structural condition s120912661.
assessment of bridges, Autom. Constr. 97 (2019) 77–95, https://doi.org/10.1016/ [56] Y. Li, J. Zou, J. Jing, H. Jin, H. Yu, Establish keypoint matches on multispectral
j.autcon.2018.10.006. images utilizing descriptor and global information over entire image, Infrared
[41] L. Inzerillo, G. Di Mino, R. Roberts, Image-based 3D reconstruction using Phys. Technol. 76 (2016) 1–10, https://doi.org/10.1016/j.infrared.2016.01.011.
traditional and UAV datasets for analysis of road pavement distress, Autom. Constr. [57] FLIR, FLIR One Pro-Series Pro and Pro LT Models. https://www.flir.it/products/flir
96 (2018) 457–469, https://doi.org/10.1016/j.autcon.2018.10.010. -one-pro/, 2021 (Accessed on April 2021).
[42] Z. Ma, S. Liu, A review of 3D reconstruction techniques in civil engineering and [58] DJI, Zenmuse XT 2 - User manual. https://dl.djicdn.com/downloads/Zenmuse%20
their applications, Adv. Eng. Inform. 37 (2018) 163–174, https://doi.org/10.1016/ XT%202/Zenmuse%20XT%202%20User%20Manual%20v1.0.pdf, 2018 (Accessed
j.aei.2018.05.005. on April 2021).
[43] R. Wang, J. Peethambaran, D. Chen, LiDAR point clouds to 3-D urban models: a [59] ISO 6781:1983, Thermal Insulation - Qualitative Detection of Thermal
review, IEEE J. Selected Topics Appl. Earth Observations Remote Sens. 11 (2) Irregularities in Building Envelopes - Infrared Method. https://www.iso.org/
(2018) 606–627, https://doi.org/10.1109/JSTARS.2017.2781132. standard/13277.html, 2021 (Accessed on April 2021).
[44] S. Siebert, J. Teizer, Mobile 3D mapping for surveying earthwork projects using an [60] N. Otsu, Threshold selection method from gray-level histogramhs, in: IEEE
unmanned aerial vehicle (UAV) system, Autom. Constr. 41 (2014) 1–14, https:// Transactions on Systems, Man, and Cybernetics 9, 1979, pp. 62–66, https://doi.
doi.org/10.1016/j.autcon.2014.01.004. org/10.1109/tsmc.1979.4310076.
[45] R. Ali, J. Zeng, M. Kavgic, Y. Cha, Heat Loss Detection Using Thermal Imaging by a [61] J. Kittler, J. Illingworth, On threshold selection using clustering criteria, in: IEEE
Small UAV Prototype, SPIE Smart Structures + Nondestructive Evaluation, 2020, Transactions on Systems, Man, and Cybernetics, SMC-15, 1985, pp. 652–655,
https://doi.org/10.1117/12.2557902. https://doi.org/10.1109/TSMC.1985.6313443.
[46] E. Lucchi, Applications of the infrared thermography in the energy audit of [62] R. Heriansyah, S.A.R. Abu Bakar, Defect detection in thermal image using
buildings: a review, Renew. Sust. Energ. Rev. 82 (2018) 3077–3090, https://doi. thresholding technique, in: Conference on Circuits, Systems, Electronics, Control &
org/10.1016/j.rser.2017.10.031. Signal Processing, 2007, pp. 341–346. http://www.wseas.us/e-library/conferences
[47] Y. Huang, C.H. Chiang, K.T. Hsu, Combining the 3D Model Generated from Point /2007egypt/papers/568-736.pdf.
Clouds and Thermography to Identify the Defects Presented on the Facades of a [63] J. Canny, A computational approach to edge detection, IEEE Trans. Pattern Anal.
Building, SPIE Smart Structures and Materials + Nondestructive Evaluation and Machine Intelligence PAMI-8 (1986) 679–698, https://doi.org/10.1109/
Health Monitoring, 2018, https://doi.org/10.1117/12.2297656. TPAMI.1986.4767851.
[48] L. Hoegner, U. Stilla, Mobile thermal mapping for matching of infrared images with [64] Labelbox Inc. https://labelbox.com/, 2021 (Accessed on June 2021).
3D building models and 3D point clouds, Quantitative Infrared Thermograp. J. 15 [65] D. Kang, Y.J. Cha, Autonomous UAVs for structural health monitoring using deep
(2) (2018) 252–270, https://doi.org/10.1080/17686733.2018.1455129. learning and an ultrasonic beacon system with geo-tagging, Comput. Aided Civil
[49] C. Popescu, B. Täljsten, T. Blanksvärd, L. Elfgren, 3D reconstruction of existing Infrast. Eng. 33 (10) (2018) 885–902, https://doi.org/10.1111/mice.12375.
concrete bridges using optical methods, Struct. Infrastruct. Eng. 15 (7) (2019)
912–924, https://doi.org/10.1080/15732479.2019.1594315.

12