Keywords

1 Introduction

The management of existing bridges is a widespread problem throughout the world. In Italy and many other countries, bridges are nearing the end of their useful life and show significant deterioration conditions. During its life cycle, each infrastructure will face deterioration depending on several factors such as the natural aging, the execution of works, the material quality, the planned maintenance, and the environmental condition. In recent years, many Italian bridges have been affected by severe failures or collapses that have caused significant economic and human lives losses. The main defects are related to phenomena of cracking, spalling and delamination, which significantly shorten the useful life of the bridge and make it more fragile against the traffic on wheels with a consequent increase of safety‐related risks.

The current situation would require in-depth analyses to be carried out on all infrastructures in order to assess the actual level of safety, according to current regulations, and to take appropriate measures. It is a costly procedure in terms of time and money and the number of infrastructures on which it should be applied is very high, about one million bridges and viaducts just in Italy. Many of these are of mixed competence, about 30 thousand are managed by local authorities with reduced financial capacity, and as many as 1,425 viaducts are without an identified owner and manager [1]. The strategic approach suggested by the decision-making bodies is defining and developing simple and efficient methods for assessing the structural risk associated with infrastructures throughout the national territory. These assessments must be carried out following the classification and identification of deterioration and damage.

To this end, Superior Council of Public Works of the Ministry of Infrastructure and Transport has issued the “Guidelines for the Classification and Risk Management, Safety Assessment and Monitoring of Existing Bridges” [2]. They are intended to provide a procedure for managing the safety of existing bridges in order to prevent inadequate levels of damage by making the risk acceptable. One of the various aspects described by the guidelines is the classification of risk by identifying defects following a visual inspection. The guidelines explain how to identify these issues through photographic identification, measurement and geometric survey. This lends itself to automation as it is a well-structured methodology.

This automation has started with the advent of computer vision and machine-learning techniques, which provides effective tools for automatic analysis and detection of defects in bridge pictures. Image enhancement, morphological operators, object segmentation algorithms have been used on images for several years [3,4,5]. With the technological development of imaging sensors and the increased amount of data acquired, machine-learning techniques [6,7,8] and artificial neural networks [9, 10] have become of great interest in this field.

A further step forward has been the spread of remotely piloted aircraft, which allowed georeferenced images of large parts of the infrastructure to be acquired. In the specific case of bridges, these are generally complicated structures to detect due to limited access to the structure itself. This often means increased risk for the operators involved in the inspection procedures, as well as the need to stop traffic, which increases costs. The use of drones could be a solution to these problems and could also provide an effective tool for cost saving with respect to traditional techniques [11]. Unmanned Aerial Vehicles (UAV) have already been used in bridge inspection and also developed in order to acquire those non-conventional data [12, 13] for documentation. In this scenario, photogrammetric techniques are effective tools to perform more in-depth analyses: i.e. georeferencing, measuring and quantifying defects in bridges. In fact, digital photogrammetry provides three-dimensional models and metrically accurate solid orthophotos, which could be used as a rich dataset for machine learning classification techniques [14].

This work aims to automate the procedure by means of geomatic and machine-learning techniques for the identification and automatic classification of structural defects in bridges. A UAV was modified for carrying a low-cost camera facing upward in order to acquire images of the intrados of a bridge. The automatic identification of the defects was made thorough a semiautomatic object-oriented (OBIA) supervised machine learning classification. The input data used as training sets were an orthoimage and a digital surface model results of the photogrammetric processing. All the steps of the work were made by means of free and open-source software (FOSS).

2 Case Study

The procedure proposed in this contribution was tested on the Stura bridge. The Stura river is crossed by this 150 m length double bridge, located on the motorway connection RA10, which allows the communication between Turin and the international airport Sandro Pertini (Fig. 1). The Stura Bridge is realized with two separated structures, one next to the other, each one of them corresponding to a direction of travel and 11 m wide. Each structure has five spans of variable lengths formed by a three main pre-stressed concrete girder beams system and complemented with squared sections transversal beams.

Fig. 1.
figure 1

Location of the Stura bridge on the RA10, its top and bottom view.

Full size image

Given its strategic importance, the Stura bridge has been subject to various monitoring activities in recent years. Regardless of the nature of the inspection technique, monitoring the bridge's intrados is arduous because most of its length crosses the Stura river, and therefore it is hard to reach. This is especially true for geomatics approaches, which rely on different instruments. To this purpose, a novelty procedure for realising a photogrammetric survey with an upward camera installed on a UAV and the automatic classification of the defects was realized and applied on the second span of the bridge, 35 m long and 22 m wide.

3 The Methodology

The methodology consists of three macro-steps: data acquisition, data processing and the visual restitution of the results. It was entirely realized using FOSS (Fig. 2).

Fig. 2.
figure 2

FOSS workflow for the detection of structural damages.

Full size image

3.1 Data Acquisition

The input data used in this work for the automatic classification of the structural defects of the bridge was the orthophoto of its intrados. The realization of this product required a novelty approach, which involved the use of an upward camera installed on a UAV platform. Typically, the commercial UAV solution integrates a camera gimbal that can be oriented at an angle of 90°, from zenith to nadiral view. In some cases, where it is necessary to inspect particular structures that cannot be observed and analysed effectively with other instruments, such as the intrados of a bridge, the possibility of exploiting an upward tilt can be decisive.

For the survey, a common Dji Phantom 4 aircraft was used. The quadrotor has a flight autonomy of about 20 min and a maximum upward orientation of the camera of 30°, which is insufficient for the complete photogrammetric reconstruction of the intrados. For this purpose, the UAV was equipped with a Raspberry Pi Camera Module 2 connected and managed by a Raspberry Pi (RPi) 3 system (Fig. 3). The advantage of this solution is the limited cost (<60 euro), their light weight and their simplicity to be used and embedded. Moreover, the RPi system can be exploited to control and process data collected from other sensors, such as inertial platforms and GNSS receivers, in order to perform the direct photogrammetry, where environmental conditions allow it [15].

Fig. 3.
figure 3

The photogrammetric system integration composed of the Raspberry Pi Camera Module V2 (in green), the Raspberry Pi 3 (in blue) and the power-bank (in red). (Color figure online)

Full size image

A power-bank with 5000 mAh capacity was connected to the RPi, guaranteeing the autonomy of about 7 h. The devices were fixed with some cable ties to the camera gimble clamp of the drone and a support for a particular optical sensor (not used in this study) so as not to unbalance the platform. The total weight of the devices and the mounting components amounted to about 220 g. The RPi RGB camera module was installed upwards on the UAV to capture images with an orientation of 45° with respect to the zenith view. This camera sensor has an 8 MP resolution, but, for this study, a video recording at a lower resolution was set (1029 × 702 pixel) in order to automatically acquire images with a higher frame rate (2 fps) and ensure sufficient overlap between consecutive frames. To this purpose, a specific application in Python was developed to set the images acquisition with RPi camera and to store them automatically on an SD card inside the RPi. The application can be launched before the start of the flight. Due to the lack of GNSS signal, the flight was manually performed. In order to cover the entire study area, a flight path, at a distance of about five meters from the intrados, was followed, consisting of five survey lines corresponding to the spans of the bridge, each travelled in both directions. Table 1 summarizes the flight plan parameters and the characteristics of the acquired data.

Table 1. Flight plan parameters
Full size table

As the external orientation of the images was unknown, external markers and photogrammetric points, easily identifiable on the images, were used as Ground Control Points (GCPs) to reference the model and to predict its accuracy. Therefore, the first step of the survey involved the construction of a topographic network composed of six vertices. Two vertices, identifying the primary network, were surveyed through a GNSS Leica GS-18 double frequency and multi-constellation receiver in static modality, standing on each point for about 1 h. The coordinates have been estimated considering a single-base solution (through the Leica Geo Office® software v.8.4) with the Torino permanent station of CORSs (Continuous Operating Reference Stations) network by Piedmont district, with an accuracy of about 3 mm.

Starting from this reference network, it was possible to acquire the position of four other points materialized on the ground under the bridge using a Leica MS-50 total station and a prism. The position of three artificial markers and of ten photogrammetric points spread on the intrados was measured from the station points of the topographic network and adjusted with the MicroSurvey StarNet v.7.0 software in order to obtain the final coordinates: the root mean square (RMS) of the estimated coordinates was lower than 1 cm.

4 Data Processing

The data processing regards the reconstruction of the orthophoto of the bridge's intrados and the supervised machine learning classification of the damages of the structure.

4.1 Photogrammetric Data Processing

Photogrammetric processing allows reconstructing of different scenarios from a set of images acquired from different points of view in the form of 3D digital model and ortho-imagery of the surveyed area. In order to maintain the FOSS approach, the free open-source photogrammetric suite MicMac (Multi-Images Correspondances, Methodes Automatiques de Correlation) from the French National Geographic Institute (IGN) [16, 17] has been used. The first step of the point cloud reconstruction is the identification of relevant features visible in multiple images, which is performed with the well-known SIFT algorithm [18]. Thanks to these common features, the position of the camera centre and the calibration parameters are computed. The MicMac libraries use the Structure from Motion procedure in order to accurately reconstruct the scene, generating the dense point cloud from which the digital elevation model (DEM) and the orthomosaic map are extracted. All these steps are applied by running sequentially five libraries: Tapioca, Tapas, Malt, Nuage2Ply and Tawny. During the process, some known reference points measured with a Total Station has been used to scale and georeference the model (GPSBascule tool). An optimization procedure which minimizes the coordinate errors is performed by Campari tool and the overall discrepancies between the reference coordinates and the estimated ones are reported in Table 2. From this procedure a DEM with resolution of 2.5 cm and an orthophoto with resolution of 0.6 cm have been extracted.

Table 2. Overall RMSE of the photogrammetric points on the intrados measured with the total station.
Full size table

4.2 Damage Classification

The damages were identified thorough a semiautomatic object-oriented (OBIA) supervised machine learning classification.

In the Very High Resolution (VHR) image classification realm, the OBIA approach dominancy is undiscussed. Ontologically, OBIA allows the identification and extraction of real-world features more accurately than pixel-based solutions and reliably from remotely sensed data only on more appropriate scales [19].

The OBIA classification is generally organized in six steps [20]:

  1. a.

    Segmentation;

  2. b.

    Features extraction and data preparation;

  3. c.

    Training and validation datasets generation;

  4. d.

    Classification algorithm;

  5. e.

    Feature selection;

  6. f.

    Validation of the classification.

The segmentation is the core process of OBIA, and several steps generally characterize it. Starting with individual pixels, OBIA algorithms merge contiguous pixels into groups (i.e., objects) based on three parameters: scale, shape, and compactness [20,21,22]. The objects should represent the classification features of interest (i.e. structural damages and non-damaged areas) [19]. The scale represents the degree of spectral heterogeneity allowed in each object. Generally, the higher the scale value (that is unitless), and larger the object will be because it is more heterogeneous [19, 22]. The compactness can be defined as the degree of similarity of a polygon to a circumference. The compactness parameter optimizes the resulting objects regarding the compactness.

The segmentation was realised on three-dimensional information of visible spectral information imagery using ORFEO toolbox [23], an open-source library for remote sensing application. The ORFEO project was funded in 2006 by the French space agency and it has rapidly evolved into a reliable toolbox for remote sensing practitioners. The OTB Large-Scale Mean-Shift segmentation workflow (LSMS) was applied [24]. First, the Red-Green-Blue image was smoothed with a range radius of 10 and a spatial radius of 2. Then the smoothed dataset was segmented using LSMS segmentation algorithm, with 2 as spatial radius and 5 as range radius. The LSMS Small region merging algorithm allowed the merging the smallest segments by setting the minimum segment size (i.e. 100). Finally, the segmentation was vectorised using the LSMS vectorization algorithm.

Effective use of features as input data for a classification procedure can improve classification accuracy [20, 25, 26]. Thus, the classification dataset was enriched with derivative features, namely, spectral-, textural-, and statistical-based features. The extraction of features was based on the literature classification research [27,28,29,30,31,32] and the authors’ personal experience. Since the OBIA approach allows the analyst to introduce into the classification model features regarding the segmented objects’ geometric characteristics and their relationship with the neighbourhood's objects, the shape, the extent and the compactness of the segments were computed too. 156 features were computed for each object. Table 3 reports the used measures for segmentation and classification. For each segment were computed the mean, the standard deviation, the median, the variance, the skewness and the kurtosis of spectral, histogram and textural features of Table 3.

Table 3. Features selected for the classification input dataset. They were computed using ORFEO toolbox.
Full size table

Five classes describe the existing damages over the study area: Drainage; uncovered metal bar (UMB); Oxidized rebar (OR); Non-damaged intrados; and Non-damaged beam (Table 4). For each class were visually identified 100 segments to be used as training and test (70% training – 30% test). Reaching 100 samples was not always possible since some classes are more frequent (in terms of area coverage) than others, and some other classes, although covering large areas, are spectrally homogeneous and thus described by large segments. This lead to small and unbalanced training datasets that can negatively affect the classification results. Thus, the training dataset was oversampled in Python environment. The oversampling [34] creates synthetic samples by interpolation of the feature values of the nearest neighbours. The Synthetic Minority Oversampling Technique (SMOTE) was applied considering the 6 nearest neighbour samples (n) to reach the largest available class (100 samples of Drainage). The final training dataset was composed of 500 samples. Then, data were scaled according to each feature minimum-maximum values.

Table 4. Classes of the model.
Full size table

The classification was performed in Python environment using Pandas, NumPy and Sklearn libraries. Random Forest classification [35] model was trained using the smoted dataset. One-thousand trees with Gini criterion for the node splitting were used. The Random Forest algorithm can compute the Gini gain of each feature [36]. A simplified description of the GINI gain defines it as the sum of impurity decreases from two nodes and the parent node. The GINI is calculated for each variable of the classifier. This parameter is a proxy of the importance of each feature within the model: the variables that have high GINI gain (so they have less impurity) are more important. Features that have GINI importance less than the median values of the feature importance values were excluded from the classification.

The classification model was assessed using cross-validation approach. The cross-validation splits the test dataset in k folds of equal size and test the k-model on each fold. Since the test dataset is small (i.e. only 30 samples per class), the Leave-One-Out cross-validation (LOOCV) algorithm was applied [37, 38]. The LOOCV is a k-fold method where the number of folds is equal to the number of test samples. Thus, each fold's has only one test sample. In the case of correct classification, the fold validation is 1, and it is 0 in the case of wrong classified test samples. The assessment of the results is the mean of the assessment measures calculated on each fold. The precision, the Recall, and the F1-score were computed. It is important mentioning that the SMOTE was applied inside the LOOCV cycle.

5 Results and Discussion

The segmentation resulted in 8068 segments characterised by 156 feature each. Figure 4 provides the visual restitution of the classification. The feature selection reduces the input classification variables from 156 to 74 without losing in accuracy. Indeed the overall accuracy of the model moved from 0.816 (without feature selection) to 0.819 (with feature selection). Table 5 reports the F1-score, Precision and Recall values of each class.

Table 5. Accuracy metrics calculated with LOOCV.
Full size table

The classification is well-performant in almost all the classes. Indeed, the F1-score is never below 0.750, except for the Oxidized rebar class. The latter provides the lowest accuracy; the F1-score is only 0.602. This aspect is quite interesting since there should be a clear difference over the Red band of the orthophoto. Nevertheless, within the damages classification, OR class less affects the infrastructure's stability. Precision and recall measures mirror the F1-score and are quite well balanced. The Recall value of class OR relates to a high false-negative rate. The non-damaged areas class shows opposite behaviour with respect to OR. The Recall is high, indicating a low number of false positive, but the precision is relatively low relating to high false positive.

Fig. 4.
figure 4

Result of the classification.

Full size image

It should be noted that in the specific case of the bridge intrados, the final result of the classification could be influenced by environmental lighting conditions. The limited illumination that characterizes this area of the infrastructure, influences both the reconstruction of the photogrammetric block and therefore the accuracy of the final model, as well as the radiometric values of the orthophoto, causing a loss of accuracy in the result of the classification procedure.

6 Conclusion

This study has proposed an automatic approach for the detection of bridge defects as support for managing the safety of existing bridges in order to prevent inadequate levels of damage. The approach is intended to follow the directives of the Italian Ministry of Infrastructure and Transport, expressed in the “Guidelines for the Classification and Risk Management, Safety Assessment and Monitoring of Existing Bridges”. Nevertheless, the procedure can be extended for defect classification of bridges worldwide. In particular, the intent of this work is to check the potential of FOSS-focused unconventional photogrammetry for the classification of infrastructures’ damages. A commercial UAV has been modified with a low-cost imaging sensor managed by an opensource environment to acquire images of the intrados of a bridge. Those images have been processed in an open-source photogrammetric suite to extract an orthomosaic map and a digital surface model. Again, open source libraries for remote sensing image processing has been used to perform supervised classification. Uncovered metal bar, Drainage, oxidized rebar, non-damaged intrados and non-damaged beam has been classified with good results and metrics.

Although the application resulted in accurate results, they can be further improved by enlarging the input dataset by using sensors sensitive to infrared light. Additional studies should explore the possibility of spectral calibration to ensure the model's replicability.