Structure and Infrastructure Engineering

Maintenance, Management, Life-Cycle Design and Performance

ISSN: 1573-2479 (Print) 1744-8980 (Online) Journal homepage: https://www.tandfonline.com/loi/nsie20

Bridge condition rating data modeling using deep

learning algorithm

Heng Liu & Yunfeng Zhang

To cite this article: Heng Liu & Yunfeng Zhang (2020) Bridge condition rating data modeling
using deep learning algorithm, Structure and Infrastructure Engineering, 16:10, 1447-1460, DOI:
10.1080/15732479.2020.1712610

To link to this article: https://doi.org/10.1080/15732479.2020.1712610

Published online: 21 Jan 2020.

Submit your article to this journal

Article views: 167

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=nsie20
STRUCTURE AND INFRASTRUCTURE ENGINEERING
2020, VOL. 16, NO. 10, 1447–1460
https://doi.org/10.1080/15732479.2020.1712610

Bridge condition rating data modeling using deep learning algorithm

Heng Liu and Yunfeng Zhang
Department of Civil & Environmental Engineering, University of Maryland, College Park, MD, USA

ABSTRACT ARTICLE HISTORY

This paper presents a deep learning-based bridge condition rating data modeling approach using Received 23 May 2019
selected data from the National Bridge Inventory (NBI) database. The objective of this research is to Revised 6 August 2019
develop a data-driven approach that enables prediction of future conditions of highway bridge com- Accepted 17 September 2019
ponents from historical inspection data. The problem is solved by training a Convolutional Neural
KEYWORDS
Network (CNN) model with online available NBI data. One prominent feature of the CNN model is that Bridge condition rating;
if well-trained it can represent the high dimensional data in the dataset abstractions for which conven- machine learning;
tional mathematical models may be difficult to describe. A case study of Maryland and Delaware high- convolutional neural
way bridges using historical data (1992–2017) sourced from the NBI database has been performed to network; condition
demonstrate the proposed method. CNN models for three primary components of these highway forecasting; national
bridges including the deck, superstructure, and substructure have been established. Optimization of bridge inspection
model parameters is achieved through a parametric study. Research findings suggest that the deep
learning model offers a promising tool as a data-driven condition forecasting approach for bridge
components with a demonstrated prediction accuracy over 85%.

1. Introduction deterministic models, probabilistic models, and artificial

intelligence (AI). Deterministic models usually describe the
Bridge condition assessment is crucial to a successful Bridge
deterioration of bridge component conditions as a determin-
Management System (BMS) as a proper measurement of
istic relationship in relation to selected independent (or
bridge performance in communication with bridge engineers
explanatory) variables such as age, average daily traffic, and
and administrations. Since 1972, the National Bridge
climate factors. The selection of these variables is usually
Inventory (NBI) initiated by the Federal Highway
based on engineering judgment and factor analysis (e.g.,
Administration (FHWA) provides the primary source of
Morcous, Lounis, & Mirza, 2003; Goyal, Whelan, &
data for bridge management in the United States (Chase,
Small, & Nutakor, 1999; Frangopol, Kong, & Gharaibeh, Cavalline, 2017). The relationship may be formulated in the
2001). According to the National Bridge Inspection form of the polynomial or exponential/logistic (Bolukbasi,
Standards (NBIS), the highway bridges within the NBI shall Mohammadi, & Arditi, 2004; Moomen, Qiao, Agbelie, Labi,
be routinely inspected every two years and the condition & Sinha, 2016). As a result, identifying the set of influencing
ratings of bridge components (e.g., bridge deck, superstruc- factors that can most adequately describe the deterioration
ture, and substructure) are recorded in the NBI database. process, and its best-fit functional form is time-consuming
The condition ratings are one-digit numbers that character- which typically needs prior engineering judgement with
ize the general condition of the entire component being trial-and-error many times.
rated in a scale from 0 to 9. A brief description on each Probabilistic models assume the deterioration of condi-
scale of condition rating used in NBI are reproduced in tion ratings as a stochastic process to account for the inher-
Table 1 for completeness. Additional recording items and ent uncertainties and variations due to unobserved
their descriptions can be found in the coding guide explanatory variables, the presence of measurement errors,
(FHWA, 1995). and the inherent stochasticity of the deterioration process
A time-dependent reliability analysis requires the know- (Madanat, Mishalani, & Ibrahim, 1995). Markovian methods
ledge of deterioration models which predict the future struc- are reported as the most common probabilistic model and
tural condition and form the basis for life-cycle monitoring have been extensively studied in modeling such deterior-
and maintenance planning. The condition ratings have been ation process (Cesare, Santamarina, Turkstra, & Vanmarcke,
utilized as the basis of bridge component deterioration mod- 1992; Frangopol, Kallen, & Van Noortwijk, 2004;
eling to assist with the BMS in optimizing maintenance, Morcous,2006; Ranjith, Setunge, Gravina, & Venkatesan,
rehabilitation, and replacement decisions. Existing 2013). Markovian methods assume the bridge deterioration
approaches for deterioration modeling of bridge component process as a Markov-Chain based stochastic process that
conditions may be classified into three categories: describes the condition deterioration with a probabilistic

CONTACT Yunfeng Zhang zyf@umd.edu Department of Civil & Environmental Engineering, University of Maryland, College Park, MD 20742, USA
ß 2020 Informa UK Limited, trading as Taylor & Francis Group
1448 H. LIU AND Y. ZHANG

Table 1. Descriptions of condition ratings used in the National Bridge Inventory (FHWA, 1995).
Condition Rating Condition Description
9 Excellent –
8 Very good No problems noted
7 Good Some minor problems
6 Satisfactory Structural elements show some minor deterioration
5 Fair All primary structural elements are sound but may have minor section loss, cracking, spalling, or scour
4 Poor Advanced section loss, deterioration, spalling, or scour
3 Serious Loss of section, deterioration, spalling or scour have seriously affected primary structural components. Local
failures are possible. Fatigue cracks in steel or shear cracks in concrete maybe present
2 Critical Advanced deterioration of primary structural elements. Fatigue cracks in steel or shear cracks in concrete may be
present or scour may have removed substructure support. Unless closely monitored it may be necessary to
close bridge until corrective action is taken
1 Imminent failure Major deterioration or section loss present in critical structural components or obvious vertical or horizontal
movement affecting structure stability. Bridge is closed to traffic but corrective action may put back in
light service
0 Failed Out of service – beyond corrective action

transition from one condition state to another during one levels of abstraction (Lecun et al., 2015). Among the deep
inspection interval. The challenge of applying the learning algorithms, Convolutional Neural Network
Markovian models for bridge deterioration modeling (CNN), have drawn growing interests in structural health
includes the first-order Markov chain that implies the future monitoring (SHM) by leveraging on recent advancement
condition depends only on the current condition state in computer vision technology (e.g., Cha, Choi, &
(Morcous, Rivard, & Hanna, 2002; Frangopol et al., 2004). B€uy€
uk€ozt€
urk, 2017; Dorafshan, Thomas, & Maguire, 2018;
As a generalization of Markov process models, Dynamic Kumar, Abraham, Jahanshahi, Iseley, & Starr, 2018;
Bayesian networks (DBNs) have also been employed as Zhang, Rajan, & Story, 2019; Zhang et al., 2017).
probabilistic models for reliability-based deterioration mod- Encouraging results have also been reported in image-
eling (Straub, 2009; Kosgodagan-Dalla Torre et al., 2017; driven or vibration-based structural condition assessment
Rafiq, Chryssanthopoulos, & Sathananthan, 2015; Yang & applications (Rafiei & Adeli, 2018; Gao & Mosalam, 2018;
Frangopol, 2018). DBNs are a special class of BNs that are Khodabandehlou, Pekcan, & Fadali, 2019; Liang, 2019; Liu
probabilistic models using directed acyclic graphs (DAG) to & Zhang, 2019a, 2019b). For highway bridge applications,
represent the joint probability mass function of a set of ran- CNN has also been applied to process bridge inspection
dom variables, with time-variant variables being assumed image data such as ground penetrating radar (GPR) and
with Markov property (Straub, 2009). The condition deteri- thermography for rebar localization or delamination detec-
oration is modeled as a stochastic process with a set of tion in concrete bridge decks (Dinh, Gucunski, & Duong,
model parameters, among which time-variant variables are 2018; Cheng, Shang, & Shen, 2019). One prominent fea-
usually assumed with Markov property. The DBN provides ture of the CNN model is its ability to represent high
an important tool to probabilistically assess the condition dimensional data abstractions for which conventional
and the reliability of structural elements at group levels, mathematical model may struggle to describe. This moti-
while the computation might become challenging with high- vates the present study of CNN for data mining of the
dimensional models or large number of random variables. NBI bridge inspection datasets.
AI models aim to interpret the deterioration of bridge This study investigates a CNN-based condition rating
component conditions by capturing data abstractions from data modeling method for predicting condition rating of
training dataset. The Artificial Neural Network (ANN) bridge components in future inspection using historical NBI
based methods are reported as the most common AI data. The model can be used as a data-driven condition rat-
method (Sobanjo, 1997; Huang, 2010; Kawamura, ing data modeling approach to assist with the decision-mak-
Miyamoto, Frangopol, & Kimura, 2003; Li & Burgue~ no, ing process of road authorities. A case study of Maryland
2010; Tokdemir, Ayvalik, & Mohammadi, 2000). For exam- and Delaware highway bridges using historical data
ples, Li and Burgue~ no (2010) conducted comparative studies (1992–2017) sourced from the NBI database has been per-
of several ANN methods for predicting bridge abutment formed to demonstrate the proposed technique. CNN mod-
condition ratings in the state of Michigan and their models els for three primary components of Maryland highway
achieved an averaged prediction accuracy near 73% in iden- bridges including the deck, superstructure, and substructure
tifying the true condition rating of damaged bridge abut- have been trained and validated. Optimization of model
ments (with condition rating
4). Huang (2010) analyzed parameters has been achieved through a parametric study.
historical maintenance and inspection data of concrete decks Research findings suggest that the deep learning model
in Wisconsin using ANN model with a reported accuracy of offers a promising tool as a data-driven condition forecast-
75% in classifying the true condition rating. ing approach for bridge components with demonstrated pre-
Deep learning is a machine learning technique that diction reliability. Bridge condition prediction results using
allows computational models comprised of multiple proc- the trained CNN models are also discussed to provide
essing layers to learn representation of data with multiple insights in the implementation.
 STRUCTURE AND INFRASTRUCTURE ENGINEERING 1449

Figure 1. Flowchart of CNN-model based bridge component condition rating prediction method.

Figure 2. Reorganization of raw NBI database (left) into feature-specific data tables (right).

2. Deep learning based condition rating forecasting condition forecasting derives from the deep learning model
method for bridge components trained with large number of data samples that best repre-
sents the actual condition deterioration process. As a case
2.1. Technical procedure
study to showcase the process, the historical records of
The proposed method for condition rating forecasting of Maryland and Delaware highway bridges retrieved from the
bridge components utilizes a deep CNN model which can NBI database (1992-2017) are used here as the training
be trained with historical bridge inspection data such as dataset for the CNN model.
those record items sourced from NBI. The bridge compo- The technical procedure of the proposed bridge condition
nents here refer to the three primary bridge components: forecasting method is illustrated in Figure 1. First, the selected
deck, superstructure, and substructure. The input data to raw NBI database are reorganized into a data repository con-
the CNN model can be either the latest one or multiple his- sists of feature specified data tables. The data table contains
torical inspection data including condition ratings, ages, and historical values of selected feature for each individual bridge,
other features. The output is defined as the condition rating which are sourced from NBI record items. The reorganization
predicted for the immediate next inspection. The forecasting is aimed at reducing the I/O time for preparing discrete data
model that can predict the component’s future condition samples, since raw NBI files are compiled on an annual basis
based on its natural deterioration process or under current with mixed features (see Figure 2). Next, the reorganized data
routine maintenance schedule is of the most interest, since repository is prepared into discrete data samples consistent
such forecasts can form the basis of cost-effective decisions with the format requirements and properly labeled (more
for major repairs, rehabilitation or replacement work. details in Section 3.1). Lastly, the prepared data samples are
To this end, an essential step of data pre-processing is randomly assigned to the training dataset and testing dataset
performed to screen the data samples and filter out uninter- for optimizing and validating the CNN model. Once validated,
ested outliers. The screening step adopted here is intended the trained CNN model is ready for use by engineers for
to discard those data samples with condition improvement. bridge condition deterioration modeling and future condition
Hence, the CNN model is trained to forecast the future con- rating forecasting.
dition of a bridge component by assuming there is no sig-
nificant condition improvement work in near future while
2.2. Deep learning model description
routine maintenance or minor repairs are still considered.
Such prediction is thus based on the CNN model trained 2.2.1. Convolutional neural network
using processed data samples from historical NBI records CNN is a special type of artificial neural networks that per-
and the model is optimized through an iteration process forms the convolution operation in at least one of its layers.
termed back-propagation. The accuracy of the bridge Examples of other computational layers in a CNN model
1450 H. LIU AND Y. ZHANG

Figure 3. Architecture of the CNN model for bridge component condition rating prediction (FC: fully connected; CR: condition rating).

include the pooling layer, the rectified linear unit (ReLU) and the sequential arrangement of the selected features
layer, and the fully connected (FC) layer. Details regarding need to be considered, requiring extra work in the
the mathematical operation of these layers can be found in architecture design. The 1x1 convolution is thus
references (see, e.g., Goodfellow, Bengio, & Courville, 2016); adopted to simplify the design process. The connections
Description of these layers is also given by Liu and Zhang between the features can be learned in the later stage of
(2019a). The design of the CNN architecture can be custom- the network (i.e., fully connected layers). The number
ized for specific task through experiments guided by moni- of filters (i.e., the kernel size) is designed by following
toring the validation error, such as the deep CNN models the common scheme of those sophisticated CNN mod-
reported with record-striking performance of image classifi- els such as VGG-16.
cation on the large-scale (order of millions) benchmark 2. The pooling layer which is designed to reduce the spa-
database of ImageNet (Deng et al., 2009), AlexNet tial size of representations is discarded to maintain the
(Krizhevsky, Sutskever, & Hinton, 2012), VGGNet simplicity of the structure. Springenberg, Dosovitskiy,
(Simonyan & Zisserman, 2014), and ResNet (He, Zhang, Brox, and Riedmiller (2014) suggested that a homoge-
Ren, & Sun, 2016). The common architecture of CNNs neous CNN solely consisting of convolutional layers
stacks a few convolution-ReLU layers, followed with a pool- can achieve the state-of-the-art performance without
ing layer, and repeats this pattern until it merges to the fully the need for max-pooling layers.
connected layers. 3. A dropout layer with a ratio of 0.5 (Srivastava, Hinton,
The structure of the CNN model adopted in this study is Krizhevsky, Sutskever, & Salakhutdinov, 2014) is added
designed based on the common scheme, as illustrated in to mitigate the overfitting effect.
Figure 3. The features of the adopted CNN model are high- 4. The CNN model is trained in a classification scheme,
lighted as follows: and the number of neuron units in the last FC layer is
set as N, which is equal to the number of possible con-
1. 1x1 (filter size) convolution (Lin, Chen, & Yan, 2013) is dition ratings in classification. In this study, N is set to
employed throughout all the convolutional layers in the be seven by considering the possible condition ratings
model to increase the representation power of the ranging from 3 to 9. The lowest condition rating con-
neural network as well as to simplify the architecture sidered in this study is 3 since very few samples have
design process. A key aspect of designing the CNN fil- been found to have a condition rating number below 3
ter (or the convolutional layer) is to capture the local in the considered NBI dataset.
connections by taking advantages of the properties of
natural signals (LeCun et al., 2015). For example, in 2.2.2. Cost function
natural signals such as images, local groups of values The cost function is the objective function that defines the
(i.e., pixels) are often highly correlated, forming dis- training target of the CNN model. In this study, the CNN
tinctive local motifs that can be easily detected (LeCun model is treated as a classification model and the training
et al., 2015). In this study, the input data matrix is target is to classify the future condition rating based on the
comprised of a list of selected features. Different fea- given input. The output (i.e., future condition rating) of the
tures in the matrix are believed to be connected but CNN model are thus discrete classes. Since bridge compo-
may not be “local” to each other. For instance, the traf- nent condition rating number is an integer number ranging
fic volume of the bridge is related to the number of from 0 to 9, it is natural to consider the condition forecast-
lanes and the GPS coordinates (i.e., location) of the ing as a classification problem with each class representing
bridge. However, the sequence of features (i.e., #1-2, one possible condition rating number. The cost function for
#7-8; see Table 2) are sparse, and a large filter (e.g., a classification model is usually formulated using the
1x8) may be needed. Consequently, both the filter size SoftMax with cross-entropy. The SoftMax function is used
 STRUCTURE AND INFRASTRUCTURE ENGINEERING 1451

Table 2. Features selected for deep learning model based bridge component condition forecasting.
Descriptive Information (Category) # Features (Influencing Factors) Data Source from NBI
Geographic Region (A) 1 GPS Latitude Degrees Item 16
2 GPS Longitude Degrees Item 17
Bridge Other Attributes (B) 3 Toll Item 20
4 Maintenance Responsibility Item 21
5 Functional Class of Inventory Rte. Item 26
6 Bridge Age Item 27, 90, 106
7 Lanes on Structure Item 28A
8 Average Daily Traffic (ADT) Item 29
9 Inspection Interval Item 90
10 Designated Inspection Frequency Item 91
11 Reconstruction Item 106
12 Average daily truck traffic (ADTT) Item 109
Structure Configuration (C) 13 Skew Item 34
14 Structural Material/Design Item 43A
15 Type of Design and/or Construction Item 43B
16 Number of Spans in Main Unit Item 45
17 Length of Maximum Span Item 48
18 Structure Length Item 49
19 Bridge Roadway Width Curb-To-Curb Item 51
20 Deck Structure Type Item 107
21 Type of Wearing Surface Item 108A
22 Type of Membrane Item 108B
23 Deck Protection Item 108C
Condition Rating 24 Deck Condition Rating (CR) Item 58
(D) or Superstructure CR or Item 59
or Substructure CR or Item 60

here to represent the probability distribution over C differ- view point, the trained CNN model under classification
ent classes (Goodfellow et al., 2016), as described below: scheme can be characterized as a probabilistic model based
2 3 on learned data distribution. Once the CNN model is
expðz1 Þ

 6 expðz2 Þ 7 trained and validated, the condition rating forecasting for a
1 6 7
Pmodel ^y i jxi , h ¼ PC 6 .. 7 (1) given data sample (xj ) is made by choosing the class (i.e.
expðzi Þ 4
i¼1 . 5
condition rating) with the highest probability as expressed
expðzC Þ by:

  
where Pmodel ^y i jxi , h represents the conditional probability Forecasting ¼ argmax½Pmodel ^y j jxj , h  (3)
distribution over C different classes based on given input
sample xi and model parameters h (including weights and
biases); zi (i ¼ 1,2, … ,C) denotes the activations in the pre-
ceding layer (e.g. a fully connected layer). It is noted that 2.2.3. Training algorithm
the probability distribution is a vector with a length of C The training of the CNN model is to search for the optimal
while
the sum  of this vector is equal to one. For instance, model parameters h that has the least cost function value.
Pmodel ^y i jxi , h ¼ ½0:1, 0:1, 0:5, 0:2, 0:1 may be the output of This can be achieved through an iteration process called
a SoftMax function. The cross-entropy loss is used to meas- gradient descent and its extensions. The generalized nota-
ure the dissimilarity between the conditional probability dis- tion of the gradient descent may be described as:
tribution of the training dataset X consisting of n data
oLðX, hÞ
samples and the distribution estimated by the CNN model hnew ¼ hold  g (4)
as given by: oh
where h denotes each parameter including weight and biases
1X n


LðX, hÞ ¼  yi ln Pmodel ^y i jxi , h (2) in the CNN model, the derivative or gradient is calculated
n i¼1 through the backpropagation method (Rumelhart, Hinton,
where
yi is the label of the data sample xi and the & Williams, 1986), and the learning rate (g) is a user-speci-
Pmodel ^y i jxi , h can be represented using the SoftMax func- fied parameter that defines the step size of each iteration.
tion in Equation (1). The label yi is an encoding vector with The value of the learning rate can be set in a variety ways.
a length of C, and the label for the ith class is determined by A common scheme is to set g initially as a relatively larger
assigning the ith value of the vector as one while all others step (e.g., g ¼ 0:01) to accelerate the training in the begin-
are zeros. ning and then periodically decrease the value to a lower
The loss function is taking the negative logarithm of the number (e.g., g ¼ g  0:5) to ensure that the step size is
model estimated likelihood. Since lnðxÞ 2 ð1, 0 for 8x 2 refined to achieve desired accuracy in the later stage.
½0, 1, the loss function is minimized towards zero. One extension of the gradient descent for enhanced
Minimizing the cross-entropy loss is equivalent to the max- training performance is the stochastic gradient descent
imum likelihood estimation of model parameters h to match (SGD) with momentum. The term of stochastic means the
the data distributions (Goodfellow et al., 2016). From this calculation of the cost function LðX, hÞ, which can be
1452 H. LIU AND Y. ZHANG

determined with Equation (2), is based on a randomly 3.1. Data preprocessing

selected sub-dataset X rather than the entire training set.
First, the raw data files containing the Maryland and
The sub-dataset X is called mini-batch, and the number of
Delaware highway bridges in the NBI database were
individual data samples in X is called the mini-batch size.
retrieved and reorganized into a data repository with the
The SGD with momentum introduces a velocity term in selected 24 features (or influencing factors; see Table 2 for
Equation (4) to guide the direction of parameter updating. details). It is noted that raw NBI database is organized per
The SGD with momentum is employed in this study as the bridge structure number, NBI Item ID, and calendar year,
training algorithm and details of the algorithm can be found which dramatically increases the I/O time for retrieving his-
in the reference by Goodfellow et al. (2016). The training torical values of a specific feature. Thus, the reorganization,
progress can be monitored by recording the value of the as illustrated in Figure 2, is performed by constructing data
cost function at each training iteration, and the training is tables for selected features and each table has its row repre-
considered to have converged if the cost curve reaches a senting an individual bridge (distinguished by the structure
value that cannot be further decreased, indicating the CNN number or Item 8 recorded in the data file submitted to
model is not being further optimized. The training results of FHWA in 2017) and its column representing the value his-
the CNN model reported in the following case study suggest tory of the feature sourced from calendar year 1992–2017.
that this algorithm has shown a satisfactory convergence For instance, there are 6,214 (5,335 in Maryland and 879 in
performance. Delaware) distinct structure numbers in the data file of 2017
and the data table has 6,214 rows  27 columns with the
first column as the structure number and the rest 26 col-
2.2.4. Model performance measurement umns as the corresponding feature value history from calen-
To quantitatively evaluate the CNN model performance in dar year 1992 to–2017.
forecasting, its performance metric (PM) is defined as the A cleansing procedure is also performed to discard
average prediction accuracy over samples with the current redundant data points for each feature table. It is noted that
condition rating from 3 to 9, as follows: the NBI records are published on a yearly basis while most
bridges are inspected every two years except for the cases
!
1 1X 9
ni described in Item 91 (e.g., after major collisions or earth-
PM ¼ (5) quakes), which results in redundant information in the
k 7 i¼3 Ni
retrieved data repository. For instance, a condition rating
may be duplicated in two adjacent years (columns) in the
where ni denotes the number of cases of correctly predicted data table marked by identical inspection date. Hence, each
samples with a current condition rating of i, and Ni is the feature table is cleansed by only storing data points with
total number of samples being predicted (i ¼ 3, … , 9; a total distinct inspection date.
of 7 possible condition ratings). The term k denotes that the
performance metric is also averaged from k trained models 3.2. Feature selection
using k-fold cross-validation. The k-fold cross-validation
refers to a model validation technique that random parti- The choice of the selected 24 features are primarily made
tions the original dataset to k subsets with equal size and based on the influencing factors for bridge condition ratings
employs k-1 of them for independent training of k models. observed in prior literature (Madanat et al., 1995; Chang,
Details about k-fold cross-validation may be found in other Maguire, & Sun, 2017; Ghonima, Schumacher,
literatures (see, e.g., Bishop, 2006; Goodfellow et al., 2016). Unnikrishnan, & Fleischhacker, 2018; Goyal et al., 2017;
In this study, the k is set as 5. In addition to the perform- Kim & Queiroz, 2017; Sanders & Zhang, 1994). For
ance metric, the output probabilities of trained CNN models example, a faster deterioration was found for bridges in
are also assessed to provide insights of the trained northern Indiana compared to those in southern Indiana,
which was attributed to the use of deicing salts in cold wea-
CNN models.
ther regions (Mishalani & Madanat, 2002). The bridge geo-
graphical region, which is represented by the Global
3. Case study of Maryland and Delaware Positioning System (GPS) Latitude (Item 16) and Longitude
Highway Bridges (Item 17) coordinates in this study, may implicitly account
for the influence of environmental factors such as the
The proposed deep learning model based method is first increased freeze-thaw cycles in colder regions.
used to build the condition forecasting models for the As detailed in Table 2, most features are directly sourced
bridge deck component of Maryland and Delaware highway from the NBI recording items; however, the bridge age is
bridges. Bridge data from the two states are combined to calculated by subtracting the year built (or the reconstruc-
augment the dataset size. The examination starts with a tion year if applicable) from the inspection year, and recon-
baseline model and follows with a parametric analysis for struction status is a binary number indicating whether a
optimal prediction. The condition forecasting method is reconstruction has been implemented and recorded in Item
then demonstrated for the other two bridge components – 106 with 0/1 for No/Yes. The selected features are also
superstructures and substructures. grouped into 4 categories based on the similarity of their
 STRUCTURE AND INFRASTRUCTURE ENGINEERING 1453

Table 3. Dataset size when using the 24 features from the latest m inspection
records.
Total # of samples
Bridge Component m¼1 m¼3 m¼5 m¼7 m ¼ 10 m ¼ 12
Deck 39891 31192 23293 15781 5821 2470
Superstructure 40045 31336 23427 15864 5849 2482
Substructure 40159 31427 23482 15928 5908 2506

descriptive information. The grouping of the selected fea-

tures is also aimed at reducing the number of possible per-
mutations of feature combination in subsequent
parametric study.

3.3. Dataset preparation

The organized data repository is thus prepared into discrete
data samples consistent with the designated input data for-
Figure 4. Sample learning curve of the CNN model.
mats and being properly labeled. The label is defined as
the actual condition rating of the bridge component in the
training samples occupying 80% portion while the remain-
immediate next inspection. The data sample uses the
ing 20% is allocated for the testing set. The total number of
selected features with their values from the last m successive
inspections (m  1) to populate the input matrix for the acquired data samples with considered matrix height (m)
CNN model. The matrix height represents the number of for the deck is listed in Table 3. It is noted that fewer data
distinct inspection records (indicated by different inspection samples are available when multiple-inspection records are
dates) and its width stands for the number of features con- used to populate the input matrix.
sidered. In the matrix, features in the same row are valued
from the same inspection record, and the row sequence is 3.4. Deep learning implementation
organized with the most recent record at bottom and the
earliest at top. The Deep Learning Toolbox (2019) is employed to train
Therefore, by using a different setting of input matrix, and validate the CNN model with the architecture shown in
the prediction by the well-trained CNN model either Figure 3. Accelerated Graphics Processing Units (GPU)
assumes that the future conditions are history-independent implementation is employed using an Nvidia GeForce Titan
and depend only on the feature values in the current X GPU card. The stochastic gradient descent with moment
(m ¼ 1) inspection record, or the future conditions are his- is chosen as the solution algorithm, with a mini-batch size
tory-dependent and depends on the multiple (m  2) feature of 1,024 and a moment coefficient of 0.95. The learning rate
records. Since very few data samples have a condition rating (g) employs a pieces-wise decayed setting that uses a rela-
lower than 3, the condition rating of 0, 1, 2, and 3 are tively large value to accelerate the training in the beginning
grouped together as 3 to indicate the bridge component is and the rate is periodically reduced by multiplying a decay
in failure condition requiring corrective action. factor to ensure that the learning rate is not too high when
Data samples with at least one feature value of NA (not the training nears the optimal point.
applicable) are discarded. Data samples with a label of con- In this study, the initial learning rate is 0.01 and the value is
dition rating greater than the rating in the current inspec- reduced after every 10 training epochs with a decay factor of
tion year are also discarded since the current deterioration 0.5. The total number of training iterations is set to be large
model does not consider significant rehabilitation or repair enough so that the training reaches an optimum; in this study,
work by assuming that the predicted condition rating can the maximum training epoch is set as 30. The total number of
only stay in its current value or drop to a lower value. The training iterations is equal to the training dataset size divided
prepared data samples are then normalized by Z-score by the mini-batch batch size and then times the maximum
expressed in Equation (6) for each individual feature: training epoch. Testing is performed after each training epoch.
xi li The convergence of well-trained deep CNN models can be
zi ¼ (6)
ri identified by monitoring their learning curves with a stable
where li and ri are the mean value and standard deviation of low training loss that cannot be further reduced. Sample learn-
corresponding feature xi over the entire discrete sample data- ing curve is shown in Figure 4.
set. The standardized dataset has a zero mean and a standard
deviation equal to one. The normalization can help to speed 3.5. Baseline model for Bridge Deck
up convergence, even when the features are not decorrelated
(LeCun, Bottou, Orr, & Muller, 1998; Ioffe & Szegedy, 2015). The proposed method is first validated with a baseline
Finally, the normalized data samples are randomly parti- model setup: using the 24 features (as listed in Table 2) of
tioned into the training and testing datasets, with the their values in the current inspection year (the size of input
1454 H. LIU AND Y. ZHANG

Table 4. The eight CNN models and corresponding input features.

Model # Feature Combination Model # Feature Combination
1 AþB þ CþD 5 AþD
2 AþB þ D 6 BþD
3 AþC þ D 7 CþD
4 BþC þ D 8 D

worth to mention that all the eight CNN models share the
same dataset used by the baseline model, from which the
corresponding data samples are adapted (or truncated)
according to the input features.
The PMs of the eight CNN models on the testing data-
sets yield similar values near 85%, with the Model #1 being
the best (85.4% of PM) and Models #5 and #8 being slightly
worse (still 85.2% for PM), suggesting that these additional
features are not significant contributor to the model predic-
tion accuracy. However, further investigation on model out-
put probabilities below shows that the eight models may not
Figure 5. Empirical data distribution of future condition ratings for decks with perform “equally” in terms of output probabilities even
a current condition rating of 7. when they achieved very close PM values in prediction.
It is interesting to note that Model #8 which uses only a
feature matrix is 1  24; see dataset size (m ¼ 1) in Table 3) single condition rating as the input achieved fairly accurate
to predict the condition rating of the bridge deck in the predictions, which suggests the condition rating is a primary
next inspection cycle. The performance metric (PM) of the influencing factor to the predictor in this study. One pos-
baseline model on the testing dataset is evaluated as 85.4% sible reason is that condition ratings of bridge decks do not
per Equation (5). The PM indicates the prediction capacity change within one inspection cycle in most inspection times.
of the trained model on the independent testing datasets. For example, in the processed dataset, there are respectively
The results indicate that the proposed deep learning model 100%, 96.5%, 94.0%, 92.1%, 90.9%, 69.4%, and 29.4% of
based method with the baseline model setup is fairly effect- deck inspection data samples remaining in the current con-
ive in condition forecasting for bridge decks, while a series dition rating (CCR) with a CCR value ranging from 3 to 9.
of parametric studies are also done to provide insights for The output probabilities from the trained CNN models are
the model prediction. also analyzed, and the focus is given to the term that reflects
the probability of decks to remain in the CCR within one
inspection cycle. This term will be referred as “remain-in-
3.6. Parametric analysis CCR” in the following analysis. The output probabilities
from Model #8 corresponding to the “remain-in-CCR” for
3.6.1. Input matrix width
decks with a CCR from 3 to 9 are found to closely match
The input matrix width represents the number of input fea-
previously mentioned percentages.
ture considered as the CNN model input. In practice, the
According to Equation (3), the predictions are assigned
contribution from different features to the condition deteri-
to the class (condition rating) of the highest probability and
oration of bridge components may vary in different ways;
a probability greater than 50% can guarantee that such class
thus, selecting the set of best-fit features is time-consuming has the maximum likelihood. Recalling the training object-
and may need trial-and-error of many times. One favorable ive, the CNN model is trained to learn data distribution
character of the deep learning model is its automated fea- through maximum likelihood estimation. Since Model #8
ture-mining, and in theory the best set of most relevant fea- uses one discrete variable as input, the data distribution can
tures can be learned by itself with no need for their explicit be empirically determined by calculating the percentages of
definition. This aspect is examined by observing the model samples belonging to each condition rating. For example,
performance with varied input matrix width (i.e., different the empirical data distribution of future condition ratings
feature selections). The four feature categories described in for decks with a CCR of 7 is plotted in Figure 5. A close
Table 2 are permutated with eight feature combinations as match can be found in Figure 6, which plots the distribution
listed in Table 4. of output probabilities from Model #8 for decks with a CCR
In addition to the baseline model (i.e., Model #1), seven of 7, predicting either “remain-in-CCR” (Figure 6a) or drop-
additional CNN models are established by adapting the ping to the rating 6 (Figure 6b).
input layer to the number of the selected input features, It is noted that the output probabilities from model #8
while the rest model architecture is kept unchanged. For for decks with the same CCR are constants, which means
example, when the combination of geographical region and decks with different features such as construction materials
the condition rating (model #5) is examined, the input layer and geographical regions are assigned the same probability.
is modified from 1  24 to 1  3, corresponding to the con- As a result, the prediction from model #8 is inadequate for
dition ratings, GPS latitude and longitude degrees. It is condition forecasting per individual bridge due to a lack of
 STRUCTURE AND INFRASTRUCTURE ENGINEERING 1455

90.91% 8.41%

Probability density

Probability density
90.63% 8.71%

Model output probability Model output probability

(a) (b)
90.91% 8.41%
Probability density

Probability density

Model output probability Model output probability

(c) (d)
90.91% 8.41%
Probability density

Probability density

Model output probability Model output probability

(e) (f)
Figure 6. Model estimated probability distribution to remain in rating 7 (or drop to rating 6 for b, d, f) in one inspection cycle for decks with a current condition
rating of 7: (a) Model #8, (c) Model #5, and (e) Model #1); drop to rating 6: (b) Model #8, (d) Model #5, and (f) Model #1.

resolution on bridge features. As a side-by-side comparison, of GPS coordinates. The wider dispersion indicates that the
the histograms of output probabilities from Models #1 and geographic region does have influences the bridge deck con-
#5 are also analyzed, with the same instances plotted in dition deterioration.
Figure 6. Comparing to Model #8, slightly wider distribu- On the other hand, Model #1 with 24 input features dis-
tions are observed in Model #5 with two additional features plays noticeably wider distribution than Models #5 and #8,
1456 H. LIU AND Y. ZHANG

Figure 7. Comparison of empirical cumulative distribution functions of the model output probability remaining in current condition rating (CCR) for bridge decks
with the CCR of: (a) 8; (b) 9.

The two-sample Kolmogorov–Smirnov (KS) test is intro-

duced here to offer a nonparametric mathematical tool that
statistically examines whether the two probability distribu-
tions are the same (Kolmogorov, 1933; Smirnov, 1948). The
KS test measures the maximum absolute difference between
the two cumulative distribution functions as follows:
D ¼ maxðjF1 ðxÞ  F2 ðxÞjÞ (7)
where F1(x) and F2(x) are the proportion of sample values
less than or equal to x.
When two models have been trained using the same pro-
cedure except that one model does not include the weak
feature in the input matrix, the null hypothesis (H0) is that
if the weak feature does not influence the model prediction,
the output probabilities from the two models shall be statis-
tically identical. The null hypothesis would be rejected if the
p-value is smaller than the significance level of a as:
PrðD>D jH0 Þ < a (8)
The p-value is calculated using the method by Marsaglia,
Figure 8. Matrix height effect on the cumulative distribution functions of the Tsang, and Wang (2003). Since observed deck samples will
model output probability remaining in current condition rating (CCR) for bridge most likely be assigned the same rating value as the CCR,
decks with the CCR of 8.
the output probabilities corresponding to “remain-in-CCR”
from the two models #1 and #4 are tested with the KS sta-
indicating that the number of input bridge features do affect tistics; and the results show that the null hypotheses are
model output probabilities. While expanding input matrix rejected at a significance level of 0.05 for decks with a CCR
by adding more features such as the GPS coordinates has from 4 to 8. For illustration, the discrepancy between the
been found to have insignificant influence on the model empirical cumulative distribution functions of the two
prediction accuracy, the distributions of model output prob- model outputs for decks with a CCR of 8 are shown in
ability in Figure 6 reveal that additional features do have Figure 7(a), and the KS statistic D is calculated as 0.054
some effect on the model outputs. This finding indicates with the p-value of 2.2E-4(<0.05, rejected).
two models with prediction accuracies close to each other Additionally, the converged empirical cumulative distri-
do not necessarily mean that two models are equal, since bution function for decks with a CCR of 9 from the two
their predictive probabilities may be quite different. models are also shown in Figure 7(b), and the KS statistic
Consequently, the eight models may not perform “equally” D is calculated as 0.027 with the p-value of 0.69 (>0.05,
in terms of output probabilities even when they achieved accepted). This finding suggests that the GPS coordinates
very close PM values in prediction. statistically influence the model predictions in terms of
 STRUCTURE AND INFRASTRUCTURE ENGINEERING 1457

show that the null hypotheses are rejected at a significance

level of 0.05 for decks with a CCR from 4 to 9. As an
example, the discrepancy on the empirical cumulative distri-
bution function from the two models for decks with a CCR
of 8 can be seen in Figure 8.

3.6.3. Dataset size effect

It is noted that the PM of model #9 is slightly different
from model #1 reported earlier, while the training procedure
of two models is identical except for their respective dataset
size (model #1 has a size nearly 2.5 times larger). This moti-
vates the investigation on the dataset size effect on the
model prediction. The distributions of model output prob-
ability regarding “remain-in-CCR” from the two models are
thus compared with the KS test. The results show that, with
the setting of m being 1, model predictions on bridge decks
with a CCR from 4 to 9 are statistically affected by the data-
set size (the null hypotheses are rejected at a significance
level of 0.05).
As an example, the discrepancy on the empirical cumula-
Figure 9. Dataset size effect on the cumulative distribution functions of the
model output probability remaining in current condition rating (CCR) for bridge tive distribution function from the two models for bridge
decks with the CCR of 8. decks with a CCR of 8 is shown in Figure 9. The finding
indicates that the model predicted data distribution may
model output probability for decks with a CCR from 4 to 8 vary with increased dataset size. As a result, a re-training
while have minor effect on decks with a CCR of 9. The con- may be recommended once large dataset becomes available.
verged cumulative distribution function from the two mod- Since this study only uses the Maryland and Delaware state
els also demonstrates that the CNN model can learn to highway bridge data, the findings (e.g., history-dependency
“ignore” minor features that do not significantly contribute and performance improvement) from the study of input
to the condition rating forecasting. matrix height effect are worth further study using expanded
dataset in the future.
3.6.2. Input matrix height effect
The matrix height m denotes the number of past inspection 3.7. Condition rating forecasting of bridge
records used for prediction. The total numbers of available superstructure and substructure
data samples in Table 3 show that the dataset size shrinks
to below 5000 when m is equal or greater than 10, and thus The proposed method is also validated on different bridge
considered values of m in this study are 1, 3, 5 and 7. The components. Similar procedures are adopted to prepare data
examination is to investigate how the performance of samples, except that Feature #24 uses corresponding condi-
trained CNN model varies with different settings of m. The tion ratings such as Item 59 (superstructure) and Item 60
comparisons are made based on prepared datasets contain- (substructure). The processed datasets of these two bridge
ing identical information from the current inspection while components, with different setting of m, have similar size
additional data information from preceding inspections vary comparing to the bridge deck in Table 3. Considering the
with different setting of m. This is achieved by first building auto-feature mining capability of the CNN model, this valid-
the control dataset with m equal to 7. When preparing data- ation study is based on the models using all 24 features.
set for other values of m (m ¼ 1,3,5), the control dataset is Similar to the bridge deck, the evaluated PMs increase
first copied and each sample is truncated by keeping the from 86.8% to 87.3% and from 85.1% to 85.4% for the
corresponding rows. This implementation can also maintain respective superstructure and substructures when m
prepared datasets with the same size, which may potentially increases from 1 to 7. It is noted that the comparison is
affect the model predictions as discussed later. made with the control dataset (m ¼ 7) prepared using the
Herein, the model with m equal to 1 is marked as Model method described in Section 3.6.2. Quick scanning of data
#9 to distinguish from model #1 since their dataset sizes are samples shows similar percentage patterns as the bridge
different. The evaluated PMs are respective 85.0%, 85.0%, deck that most of bridge superstructure and substructure
85.8%, 85.9% for models with m being 1, 3, 5, and 7, which stayed in the same condition in one inspection cycle; thus,
indicates that the matrix height marginally affects the model the model output probabilities from trained CNN models
prediction accuracy. A subsequent investigation is then con- are also analyzed with the KS test.
ducted by comparing model output probabilities between The test results show that model predictions on both
the two with m being 1 and 7. The focus is given to the superstructures and substructures with a CCR from 4 to 9
term regarding “remain-in-CCR” and the KS test results are also affected by the input matrix height m (the null
1458 H. LIU AND Y. ZHANG

hypotheses are rejected at a significance level of 0.05). This  Considering the auto-feature mining capability of the
may suggest that the future conditions of bridge superstruc- CNN model, the representation using all 24 selected fea-
ture and substructure are history dependent. However, fur- tures is believed to achieve the optimal predictions with
ther study is needed using expanded dataset since the adequate dataset size. For bridge decks, the evaluated
dataset size also shows effects for model predictions on PMs are 85.0%, 85.0%, 85.8%, 85.9% for models with m
these two bridge components with a CCR from 4 to 9 (the being 1, 3, 5, and 7 respectively, which indicates that the
null hypotheses are rejected at a significance level of 0.05). matrix height marginally affects the model prediction
accuracy. Similar to the bridge deck, the evaluated PMs
increase from 86.8% to 87.3% and from 85.1% to 85.4%
3.8. Discussion of the results
for the respective superstructure and substructures when
The CNN algorithm provides a data-driven method for m increases from 1 to 7.
bridge condition rating forecasting which has the potential  Future conditions of the three types of bridge primary
of adaptive model updating and is capable of individual components have been found to be history dependent
bridge case prediction (vs. group level for most probabilistic with the processed dataset, evidenced by the rejection of
or stochastic methods). Comparing to existing methods, the the null hypothesis at a significance level of 0.05.
advantage of the proposed method is the auto-feature min- However, the history-dependency needs further study
ing capability of the CNN model, which enables the model with expanded database since dataset size has also been
to automatically learn representations from training data of shown to have some effects on prediction results.
adequate size. Such advantage has been found to be very
convenient in processing bridge inspection data, in addition
to images or vibration signals reported in the references. In Disclosure statement
view of other promising results reported in machine learn- No potential conflict of interest was reported by the authors.
ing fields (e.g., LeCun et al., 2015), it is envisioned that the
deep learning models will find more applications in civil
infrastructure condition assessment and deterioration mod- References
eling with unsupervised training and more available data. Bishop C. M. (2006). Pattern Recognition and Machine Learning. New
Regarding its limitations, large amount of data is usually York: Springer.
desired to ensure successful implementation of CNN model Bolukbasi, M., Mohammadi, J., & Arditi, D. (2004). Estimating the
while the data quality is also critical due to its data-driven future condition of highway bridge components using national
bridge inventory data. Practice Periodical on Structural Design and
model nature. For example, condition rating data only indi-
Construction, 9(1), 16–25. doi:10.1061/(ASCE)1084-
cates a general condition of rated component based on vis- 0680(2004)9:1(16)
ual inspection which might introduce subjective error to the Cesare, M. A., Santamarina, C., Turkstra, C., & Vanmarcke, E. H.
data (Phares, Rolander, Graybeal, & Washer, 2001); thus, (1992). Modeling bridge deterioration with Markov chains. Journal
the trained CNN model in its present form only reflects the of Transportation Engineering, 118(6), 820–833. doi:10.1061/
inherent relationships in the training data and more (ASCE)0733-947X(1992)118:6(820)
research is needed for advanced data processing and object- Cha, Y. J., Choi, W., & B€ uy€
uk€ozt€
urk, O. (2017). Deep learning-based
crack damage detection using convolutional neural networks.
ive data collection such as nondestructive evaluation (NDE).
Computer-Aided Civil and Infrastructure Engineering, 32(5),
Data fusion with NDE data is desired to further improve 361–378. doi:10.1111/mice.12263
the prediction accuracy. Chang, M., Maguire, M., & Sun, Y. (2017). Framework for mitigating
human bias in selection of explanatory variables for bridge deterior-
ation modeling. Journal of Infrastructure Systems, 23(3), 04017002.
4. Conclusions doi:10.1061/(ASCE)IS.1943-555X.0000352
Chase, S. B., Small, E. P., & Nutakor, C. (1999). An in-depth analysis
A deep learning model based approach is developed for of the national bridge inventory database utilizing data mining, GIS
condition rating forecasting of three types of bridge compo- and advanced statistical methods. Transportation Research Circular,
nents including bridge deck, superstructure and substruc- 498, 1–17.
ture. Convolutional neural network (CNN) is adopted as the Cheng, C., Shang, Z., & Shen, Z. (2019). CNN-based deep architecture
deep learning model and trained for bridge condition fore- for reinforced concrete delamination segmentation through therm-
casting in the present study. A case study using NBI data of ography. arXiv preprint arXiv:1904.05509.
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009).
Maryland and Delaware highway bridges is carried out to ImageNet: A large-scale hierarchical image database. Proceedings of
demonstrate the proposed method. Specific findings are IEEE Conference on Computer Vision and Pattern Recognition,
summarized as follows: Miami, Florida, USA.
Dinh, K., Gucunski, N., & Duong, T. H. (2018). An algorithm for
 The CNN model can be trained to represent the high- automatic localization and detection of rebars from GPR data of
dimensional data abstractions such as multi-factors and concrete bridge decks. Automation in Construction, 89, 292–298.
doi:10.1016/j.autcon.2018.02.017
history-dependent effects on bridge components condi- Dorafshan, S., Thomas, R. J., & Maguire, M. (2018). Comparison of
tion deterioration. The trained CNN models can achieve deep convolutional neural networks and edge detectors for image-
prediction accuracies over 85% on the independent test- based crack detection in concrete. Construction and Building
ing datasets. Materials, 186, 1031–1045.
 STRUCTURE AND INFRASTRUCTURE ENGINEERING 1459

Federal Highway Administration (FHWA). (1995). Recording and cod- Liang, X. (2019). Image-based post-disaster inspection of reinforced
ing guide for the structure inventory and appraisal of the nation’s concrete bridge systems using deep learning with Bayesian optimiza-
bridges. Rep. No. FHWA-PD-96-001. tion. Computer-Aided Civil and Infrastructure Engineering, 34(5),
Frangopol, D. M., Kallen, M. J., & Van Noortwijk, J. M. (2004). 415–430. doi:10.1111/mice.12425
Probabilistic models for life-cycle performance of deteriorating Lin, M., Chen, Q., & Yan, S. (2013). Network in network. arXiv:
structures: review and future directions. Progress in Structural 1312.4400.
Engineering and Materials, 6(4), 197–212. doi:10.1002/pse.180 Liu, H., & Zhang, Y. (2019a). Image-driven structural steel damage
Frangopol, D. M., Kong, J. S., & Gharaibeh, E. S. (2001). Reliability- condition assessment method using deep learning algorithm.
based life-cycle management of highway bridges. Journal of Measurement, 133, 168–181. doi:10.1016/j.measurement.2018.09.081
Computing in Civil Engineering, 15(1), 27–34. doi:10.1061/ Liu, H., & Zhang, Y. (2019b). Deep learning-based brace damage
(ASCE)0887-3801(2001)15:1(27) detection for concentrically braced frame structures under seismic
Gao, Y., & Mosalam, K. M. (2018). Deep transfer learning for image- loadings. Advances in Structural Engineering, 22(16), 3473-3486. doi:
based structural damage recognition. Computer-Aided Civil and 10.1177/1369433219859389
Infrastructure Engineering, 33(9), 748–768. doi:10.1111/mice.12363 Madanat, S., Mishalani, R., & Ibrahim, W. H. W. (1995). Estimation of
Ghonima, O., Schumacher, T., Unnikrishnan, A., & Fleischhacker, A. infrastructure transition probabilities from condition rating data.
(2018). Advancing bridge technology, Task 10: Statistical analysis and Journal of Infrastructure Systems, 1(2), 120–125. doi:10.1061/
modeling of US concrete highway bridge deck performance – Internal (ASCE)1076-0342(1995)1:2(120)
Final Report. Civil and Environmental Engineering Faculty Marsaglia, G., Tsang, W. W., & Wang, J. (2003). Evaluating
Publications and Presentations, 443. Kolmogorov’s distribution. Journal of Statistical Software, 8(18), 1–4.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. doi:10.18637/jss.v008.i18
Cambridge, MA: MIT Press. Deep Learning Toolbox. (2019). MATLAB and Deep Learning Toolbox
Goyal, R., Whelan, M. J., & Cavalline, T. L. (2017). Characterising the Release 2019a. Natick, MA: The MathWorks, Inc.
effect of external factors on deterioration rates of bridge compo- Mishalani, R. G., & Madanat, S. M. (2002). Computation of infrastruc-
nents using multivariate proportional hazards regression. Structure ture transition probabilities using stochastic duration models.
and Infrastructure Engineering, 13(7), 894–905. doi:10.1080/ Journal of Infrastructure Systems, 8(4), 139–148. doi:10.1061/
15732479.2016.1217888 (ASCE)1076-0342(2002)8:4(139)
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for Moomen, M., Qiao, Y., Agbelie, B. R., Labi, S., & Sinha, K. C. (2016).
image recognition. In Proceedings of the IEEE conference on com- Bridge deterioration models to support Indiana’s bridge management
puter vision and pattern recognition, Las Vegas, Nevada, USA, June system (Joint Transportation Research Program Publication No.
2016. FHWA/IN/JTRP-2016/03). West Lafayette, IN: Purdue University.
Huang, Y. H. (2010). Artificial neural network model of bridge deteri- Morcous, G. (2006). Performance prediction of bridge deck systems
oration. Journal of Performance of Constructed Facilities, 24(6), using Markov chains. Journal of performance of Constructed
597–602. doi:10.1061/(ASCE)CF.1943-5509.0000124 Facilities, 20(2), 146–155. doi:10.1061/(ASCE)0887-
Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep 3828(2006)20:2(146)
network training by reducing internal covariate shift. arXiv: Morcous, G., Lounis, Z., & Mirza, M. S. (2003). Identification of envir-
1502.03167. onmental categories for Markovian deterioration models of bridge
Kawamura, K., Miyamoto, A., Frangopol, D. M., & Kimura, R. (2003). decks. Journal of Bridge Engineering, 8(6), 353–361. doi:10.1061/
Performance evaluation of concrete slabs of existing bridges using (ASCE)1084-0702(2003)8:6(353)
neural networks. Engineering Structures, 25(12), 1455–1477. doi:10. Morcous, G., Rivard, H., & Hanna, A. M. (2002). Modeling bridge
1016/S0141-0296(03)00112-3 deterioration using case-based reasoning. Journal of Infrastructure
Khodabandehlou, H., Pekcan, G., & Fadali, M. S. (2019). Vibration- Systems, 8(3), 86–95. doi:10.1061/(ASCE)1076-0342(2002)8:3(86)
based structural condition assessment using convolution neural net- Phares, B.M., Rolander, D.D., Graybeal, B., & Washer, G. (2001).
works. Structural Control and Health Monitoring, 26(2), e2308. Reliability of visual bridge inspection. Public Roads, 64, 22–29.
Kim, Y. J., & Queiroz, L. B. (2017). Big data for condition evaluation Rafiei, M. H., & Adeli, H. (2018). A novel unsupervised deep learning
of constructed bridges. Engineering Structures, 141, 217–227. doi:10. model for global and local health condition assessment of structures.
1016/j.engstruct.2017.03.028 Engineering Structures, 156, 598–607. doi:10.1016/j.engstruct.2017.10.070
Kolmogorov, A. (1933). Sulla determinazione empirica di una lgge di Rafiq, M. I., Chryssanthopoulos, M. K., & Sathananthan, S. (2015).
distribuzione. Istituto Italiano degli Attuari, 4, 83–91. Bridge condition modelling and prediction using dynamic Bayesian
Kosgodagan-Dalla Torre, A., Yeung, T. G., Morales, -Napoles, O., belief networks. Structure and Infrastructure Engineering, 11(1),
Castanier, B., Maljaars, J., & Courage, W. (2017). A two-dimension 38–50. doi:10.1080/15732479.2013.879319
dynamic Bayesian network for large-scale degradation modeling Ranjith, S., Setunge, S., Gravina, R., & Venkatesan, S. (2013).
with an application to a bridges network. Computer-Aided Civil and Deterioration prediction of timber bridge elements using the
Infrastructure Engineering, 32(8), 641–656. doi:10.1111/mice.12286 Markov chain. Journal of Performance of Constructed Facilities,
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classifi- 27(3), 319–325. doi:10.1061/(ASCE)CF.1943-5509.0000311
cation with deep convolutional neural networks. In Proceedings of Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning
Neural Information Processing Systems, Lake Tahoe, Nevada, USA, representations by back-propagating errors. Nature, 323(6088),
December 2012. 533–536. doi:10.1038/323533a0
Kumar, S. S., Abraham, D. M., Jahanshahi, M. R., Iseley, T., & Starr, J. Sanders, D. H., & Zhang, Y. J. (1994). Bridge deterioration models for
(2018). Automated defect classification in sewer closed circuit televi- states with small bridge inventories. Transportation Research Record,
sion inspections using deep convolutional neural networks. 1442, 101–109.
Automation in Construction, 91, 273–283. doi:10.1016/j.autcon.2018. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional net-
03.028 works for large-scale image recognition. arXiv:1409.1556.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, Smirnov, N. (1948). Table for estimating the goodness of fit of empir-
521(7553), 436–444. doi:10.1038/nature14539 ical distributions. The Annals of Mathematical Statistics, 19(2),
LeCun, Y., Bottou, L., Orr, G., & Muller, K. R. (1998). Efficient back- 279–281. doi:10.1214/aoms/1177730256
prop. Neural Networks: Tricks of the Trade. New York: Springer. Sobanjo, J. O. (1997). A neural network approach to modeling bridge
Li, Z., & Burgue~no, R. (2010). Using soft computing to analyze inspec- deterioration. In Proceedings of the 4th Congress on Computing in
tion results for bridge evaluation and management. Journal of Civil Engineering, ASCE, Reston, VA, pp. 623–626.
Bridge Engineering, 15(4), 430–438. doi:10.1061/(ASCE)BE.1943- Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014).
5592.0000072 Striving for simplicity: The all convolutional net. arXiv:1412.6806.
1460 H. LIU AND Y. ZHANG

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Yang, D. Y., & Frangopol, D. M. (2018). Probabilistic optimization
Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural framework for inspection/repair planning of fatigue-critical details
networks from overfitting. The Journal of Machine Learning using dynamic Bayesian networks. Computers & Structures, 198,
Research, 15(1), 1929–1958. 40–50. doi:10.1016/j.compstruc.2018.01.006
Straub, D. (2009). Stochastic modeling of deterioration processes Zhang, A., Wang, K. C. P., Li, B., Yang, E., Dai, X., Peng, Y., …
through dynamic Bayesian networks. Journal of Engineering Chen, C. (2017). Automated pixel-level pavement crack detection
Mechanics, 135(10), 1089–1099. doi:10.1061/(ASCE)EM.1943-7889. on 3D asphalt surfaces using a deep-learning network. Computer-
0000024 Aided Civil and Infrastructure Engineering, 32(10), 805–819. doi:10.
Tokdemir, O. B., Ayvalik, C., & Mohammadi, J. (2000). Prediction of 1111/mice.12297
highway bridge performance by artificial neural networks and genetic Zhang, X., Rajan, D., & Story, B. (2019). Concrete crack detection
algorithms. In Proceedings of the 17th International Symposium on using context-aware deep semantic segmentation network.
Automation and Robotics in Construction, Taipei, Taiwan, Computer-Aided Civil and Infrastructure Engineering, 34(11),
September 2000. doi:10.22260/ISARC2000/0066 951–971. doi:10.1111/mice.12477