DOI: 10.1111/mice.

12580

ORIGINAL ARTICLE

Uncertainty-assisted deep vision structural

health monitoring

Seyed Omid Sajedi Xiao Liang

Department of Civil, Structural and

Environmental Engineering, University at
Buffalo, the State University of New York,
Buffalo, NY 14260, USA
Correspondence
Xiao Liang, Department of Civil, Structural
and Environmental Engineering, Univer-
sity at Buffalo, the State University of New
York, Buffalo, NY.
Email: liangx@buffalo.edu
Abstract
Computer vision leveraging deep learning has achieved significant success in
the last decade. Despite the promising performance of the existing deep vision
inspection models, the extent of models’ reliability remains unknown. Structural
health monitoring (SHM) is a crucial task for the safety and sustainability of
structures, and thus, prediction mistakes can have fatal outcomes. In this paper,
we use Bayesian inference for deep vision SHM models where uncertainty can
be quantified using the Monte Carlo dropout sampling. Three independent case
studies for cracks, local damage identification, and bridge component detection
are investigated using Bayesian inference. Aside from better prediction results,
the two uncertainty metrics, variations in softmax probability and entropy, are
shown to have good correlations with misclassifications. However, modifying
the decision or triggering human intervention can be challenging based on raw
uncertainty outputs. Therefore, the concept of surrogate models is proposed
to develop the models for uncertainty-assisted segmentation and prediction
quality tagging. The former refines the segmentation mask and the latter is
used to trigger human interventions. The proposed framework can be applied
to future deep vision SHM frameworks to incorporate model uncertainty in the
inspection processes.

1 INTRODUCTION great promise in dealing with real-world images and

Visual inspections are an indispensable part of structural
health monitoring (SHM). Condition assessments can be
carried out periodically as a part of the maintenance pro-
gram or in near real-time after the extreme events (Sajedi
& Liang, 2019b, 2020a). Considering several factors such as
time–cost constraints, reliability, and life-safety concerns
of human-based inspections, there has been a growing
incentive for automation in SHM. Extracting useful
information from images is considered as a challenging
task, especially in the presence of noisy and complex
backgrounds. Deep learning algorithms have shown
more advanced ones are continuously being developed
(e.g., Chen, Girshick, He, & Dollár, 2019; Jégou, Drozdzal,
Vazquez, Romero, & Bengio, 2017). This progress has moti-
vated researchers to investigate the potential applications
of the machine and deep learning in civil engineering.
There have been efforts to apply these algorithms
in material properties prediction (Rafiei, Khushefati,
Demirboga, & Adeli, 2017), asphalt surface analysis
(Tong, Gao, Sha, Hu, & Li, 2018), recovering lost sensor
data (Oh, Glisic, Kim, & Park, 2019), early earthquake
warning systems (Rafiei & Adeli, 2017a), and construc-
tion management (Rafiei & Adeli, 2016, 2018a). More

© 2020 Computer-Aided Civil and Infrastructure Engineering

Comput Aided Civ Inf. 2020;1–17. wileyonlinelibrary.com/journal/mice 1

2 SAJEDI and LIANG
specifically, SHM research has significantly benefited
from utilizing deep learning in processing information
to assess structural conditions. This input information
can be from various sources such as vibration (Azimi
& Pekcan, 2019; Eltouny & Liang, 2020; Rafiei & Adeli,
2017b, 2018b; Sajedi & Liang, 2020b, 2020c), acoustic
emissions (Ebrahimkhanlou, Dubuc, & Salamone, 2019),
and strain measurements (Karypidis, Berrocal, Rempling,
Granath, & Simonsson, 2019). While being good indicators
of structural damage (Amezquita-Sanchez, Park, & Adeli,
2017), acquiring these types of records commonly requires
special instrumentations. Hence, their availability is com-
monly limited to specific case studies, simulations, and lab
experiments. This has made the generalization capability
of these data-driven models challenging, especially for
training deep learning models that require a substantial
number of observations. In contrast, images are relatively
more accessible both in terms of quality and quantity.
This is partially thanks to camera-equipped drones that
can easily capture images that are challenging to obtain
by a human inspector (e.g., Liu, Nie, Fan, & Liu, 2020).
Moreover, a significant amount of research is dedicated
to convert raw visual records into information that can be
utilized for the inspection and monitoring of structures
(Spencer, Hoskere, & Narazaki, 2019).
The vision-based algorithms are generally investigated
in two main categories: object detection and damage
classification. Identifying the structural components
has been done in forms of predicting objects’ bounding
boxes (Liang, 2019) and pixel-wise segmentation of the
whole scene (Narazaki et al., 2020). Information obtained
from these models can be used for further inspection
guidance. The second category (i.e., damage detection)
includes identifying various types of damage. Pavement
defects and road conditions have been studied using
different algorithms including probability generative
models and support vector machines (Ai, Jiang, Kei, &
Li, 2018), convolutional neural networks (CNNs) (Bang,
Park, Kim, & Kim, 2019), and recurrent neural networks
(A. Zhang et al., 2017). Identifying cracks have also been
the focus of several SHM studies using either bounding
boxes (Cha, Choi, & Büyüköztürk, 2017; Deng, Lu, & Lee
2020; Xue & Li, 2018) or semantic segmentation (Sajedi &
Liang, 2019a; Yang et al., 2018). Other types of structural
defects, such as delamination (Cha, Choi, Suh, Mah-
moudkhani, & Büyüköztürk, 2018), cavity (R. Li, Yuan,
Zhang, & Yuan, 2018; C. Zhang, Chang, & Jamshidi , 2019),
fatigue cracks (Hoskere, Narazaki, Hoang, & Spencer,
2018), and efflorescence (S. Li, Zhao, & Zhou, 2019),
or a subset of them are identified using deep learning
architectures.
By taking a closer look at this literature review, espe-
cially in the past three years, it is evident that deep
learning for visual inspections is becoming more trending
among the SHM research community. Artificial intel-
ligence may revolutionize SHM research and practice.
Nevertheless, certain facts should be kept in mind:
proposed models in the literature are data-driven, and
thus, their performance is highly dependent on the
quantity and quality of the training data. While there
have been efforts to improve the overall performance,
such as using transfer learning and data augmentation
(Gao & Mosalam, 2018), deep learning algorithms are not
mistake-free.
Erroneous predictions by vision models have caused
fatal accidents in the past where the side of a trailer vehicle
was mistakenly classified as the sky in a self-driving car
(McAllister et al., 2017). We believe that the consequences
of misclassifications in structural inspections could be far
more severe because SHM investigates structural damage
and safety directly. Therefore, it is necessary to have a
measure of the model’s confidence. This need is further
highlighted considering the much smaller size of image
datasets for civil engineering applications compared to
those of computer science like ImageNet (Deng et al.,
2009).
In this paper, we leverage the concept of deep Bayesian
neural networks (Gal & Ghahramani, 2015) for SHM
vision tasks. This model’s uncertainty output, alongside
the predictions, can be used to trigger human interven-
tions when the model uncertainty is high (e.g., monitoring
damage in a nuclear power plant). The following section
provides a brief underlying theory for Bayesian segmenta-
tion networks. In Section 3, we then explain the concept of
surrogate models as tools that aid in the interpretation of
uncertainty. Later, in Section 4, datasets for the three inde-
pendent case studies are briefly explained. In Section 5, a
sensitivity analysis is performed to provide insight on the
selection of dropout hyperparameters. The experimental
validations are given in Section 6 on the three case studies
while showing the applications of the two surrogate mod-
els. Finally, the paper is concluded by summarizing the key
points.
2 DEEP LEARNING FRAMEWORK
WITH BAYESIAN INFERENCE
This section gives a brief theoretical background on
Bayesian inference and how model uncertainty can be
quantified in deep segmentation models. Later, the deep
learning architecture used in this paper is explained.
SAJEDI and LIANG
2.1 Inference and uncertainty
quantification
Uncertainty should be an essential output of any dam-
age diagnostic system that is intended for automa-
tion. Knowing the confidence with which we can trust
the SHM model’s predictions is essential for decision-
making. Bayesian probability theory (Ahmadlou & Adeli,
2010; Koller & Friedman, 2009) offers a mathematically
grounded framework to reason about the uncertainty of
a data-driven model. However, such methods are compu-
tationally expensive, especially for real-time implementa-
tion in computer vision. Considering the importance of
model uncertainty on the one hand, and the challenge
of implementing Bayesian machine learning for computer
vision on the other hand, Gal and Ghahramani (2015) pro-
posed a delicate solution. They have showed that the use
of dropout operator in neural networks of arbitrary types
approximates Bayesian inference in deep Gaussian process
statistical model. In what follows, we provide a brief the-
oretical overview on how the uncertainty is dealt with in
the proposed approach.
CNNs are proven to be effective in dealing with image
data. However, the learnable parameters of kernels (W)
are deterministic values that are optimized by minimizing
a loss function calculated based on the raw image input
(X) and the desired output labels (Y). To reason about a
model’s uncertainty, W and subsequently the predicted
output probabilities should be treated as random vari-
ables. With that said, obtaining the posterior distribution
of convolution kernels given the existing labeled data,
𝑝(𝐖|𝐗, 𝐘), is of interest. This distribution is intractable
and cannot be analytically calculated (Gal & Ghahramani,
2015). Instead, an approximate variational distribution
(Graves, 2011), q(W), is used that is determined by
minimizing the Kullback–Leibler (KL) divergence as a
measure of similarity between this distribution and the
full posterior:
KL (𝑞(𝐖)||𝑝(𝐖|𝐗, 𝐘)) (1)
which can be expressed as the minimization of the
objective function (Lv ) with respect to the variational
parameters q(W):
𝐿𝑣 = − 𝑞(𝐖) log 𝑝(𝐘|𝐗, 𝐖)d𝐖 + KL (𝑞(𝐖)||𝑝(𝐖))
∫
(2)
By considering a Bernoulli distribution for setting
individual inputs before each convolution layer, and by
performing Monte Carlo integration, the objective cost in
(2) is equivalent to the training loss of the deep learning
model (Gal & Ghahramani, 2016):
3
𝑁
1 ∑𝑥 ∑𝐿 (
2 2
)
𝐿do = 𝐸(𝐘𝑖 , 𝐘̂ 𝑖 ) + 𝜆 ‖𝐖𝑖 ‖ + ‖𝐛𝑖 ‖ (3)
𝑁𝑥 𝑖=1 𝑖=1
where E(.,.) denotes the distance between predictions and
real output, which is captured using binary or categorical
cross-entropy in this paper, and the second term refers to
the L2 regularization of weights and bias terms (bi ). Nx is
the number of observations in the training set and 𝜆 is a
scaling hyperparameter.
Beyond the training phase, models need to be evaluated.
In terms of practical implementation, the main difference
between standard and Bayesian inference is in the way
dropout is used. With a prior probability, this operator
randomly sets a fraction of input elements into zero as a
way to reduce overfitting while training a neural network.
The standard dropout uses the weight averaging technique
(Srivastava, Hinton, Krizhevsky, Sutskever, & Salakhutdi-
nov, 2014) at test time such that the deep learning models
produce deterministic output. In Bayesian inference, the
output probabilities of classes, as intended, are random
variables. Therefore, the probability of a testing input
observation (x*) belonging to the class yi can be estimated
as:
𝑝(𝑦𝑖 |𝐱∗ , 𝐗, 𝐘) = 𝑝(𝑦𝑖 |𝐱∗ , 𝐗, 𝐖)𝑞(𝐖)d𝐖 (4)
∫
In pixel-wise image segmentation, a regular prediction
output (y) is a tensor of shape (height, width, Nb ) where
the last channel refers to the softmax probabilities of
class i ( 𝑆𝑖 , 𝑖 ∈ {1, 2, … , 𝑁b }). Feeding a specific input
image to a deep learning model with standard inference
will result in deterministic 𝑆𝑖 values, the maximum
of which is considered as the decision probability. In
contrast, activating dropout layers at the inference time
yields a prediction output where class probabilities can be
deemed to be random variables. For a single observation,
Ns Monte Carlo samples can be drawn to form a stacked
output tensor with the shape of (height, width, Nb , Ns ).
The expected value (mean) of 𝑆𝑖 samples is then used for
the inference, which is equivalent to the integration in
(4). Therefore, the probability of a pixel observation (y)
belonging to the class i can be expressed as:
𝑁
1 ∑s 𝑛
𝑝(𝑦 = 𝑖|𝐱∗ , 𝐗, 𝐘) ≈ E(𝑆𝑖 ) = 𝑆 (5)
𝑁s 𝑛=1 𝑖
This equation highlights the key difference between
the Bayesian and standard inferences. The latter directly
uses constant 𝑆𝑖 values that are obtained from a neural
network without dropout sampling, while the Bayesian
inference utilizes the mean of the Ns samples.
4 SAJEDI and LIANG

Two metrics are introduced to quantify model uncer-

tainty. Kendall and Gal (2017) proposed to use entropy (H)
as a measure of model epistemic uncertainty:

𝑁b
∑
𝐻(𝑝) = −𝑝𝑖 log(𝑝𝑖 ) (6)
𝑖=1

Moreover, class softmax variance can also be used as an

indicator of the model uncertainty for each class (Kendall,
Badrinarayanan, & Cipolla, 2015). This metric is obtained
by taking the variance of Monte Carlo samples ( 𝑆𝑖 ’s) with (b)
respect to each class. A relatively high variance in the soft-
max probabilities can indicate a model’s lack of confidence
in its predictions. In this paper, we use the class softmax
standard deviation (CSSD) for better numerical stability in
the input for the models that will be explained in Section 3. (c)

√
√
√ ∑𝑁s
1 (a)
𝐶𝑆𝑆𝐷𝑖 = √
2
[𝑆 𝑛 − E(𝑆𝑖 )] (7)
𝑁s − 1 𝑛=1 𝑖
FIGURE 1 FC-DenseNet building blocks

As the second overall measure of model uncertainty,

the mean of the per-class softmax standard deviation
(MCSSD) is used. In segmentation, entropy and MCSSD
can be obtained for all pixels. Therefore, the two measures
of uncertainty are presented for all pixels in the form of
a 2D tensor with the same dimension as that of the final
segmentation mask.
It should be emphasized that there is a fundamental
difference between softmax probabilities and model con-
fidence. The last softmax layer provides an approximation
of relative class probabilities rather than the absolute
model uncertainty that is captured using Monte Carlo
dropout. For example, a predicted label with a high soft-
max probability may also have a high model uncertainty
at the same time.
2.2 Deep learning architecture
In this paper, fully convolutional (FC) DenseNet is lever-
aged to perform Bayesian inference. This neural network
combines the underlying idea of U-Net (Ronneberger,
Fischer, & Brox, 2015) and DenseNets (Huang, Liu, Van
Der Maaten, & Weinberger, 2017). The architecture has
achieved state-of-the-art performance on benchmark
datasets for urban scene segmentation (Jégou et al.,
2017) and is considered among the most successful ones
for the given task. Similar to most image segmentation
algorithms, FC-DenseNet automatically extracts features
while reducing the spatial resolution of feature maps by
performing multiple pooling operations in the downsam-
pling path. The spatial resolution of the input is later
recovered in the upsampling path. The two paths are
connected with a bottleneck in between. What makes this
architecture unique is the presence of sophisticated con-
nectivity patterns where input from the previous layers is
concatenated with the extracted features maps. Moreover,
several skip connections help to recover fine-grained
information in the upsampling process.
The building blocks of FC-DenseNet are dense blocks
(DBs), transition up (TU), and transition down (TD) units.
The details of each unit are shown in Figure 1. DBs are
composed of modular stacked layers. Each module con-
tains a sequence of batch normalization, ReLU activation,
convolution, and dropout layers (Figure 1a). The output of
each module is then concatenated with its input. The first
convolution will extract 48 filters, while the number of
filters inside DB convolutions (growth rate) is 16. The TD
units are similar to a modular stacked layer, while it has
an additional max-pooling operation to reduce the spatial
resolution (Figure 1b). The number of filters in their cor-
responding 1 × 1 convolution is equal to the final output of
the previous DB. Finally, TU units are simply transposed
3 × 3 convolution (Long, Shelhamer, & Darrell, 2015).
In this paper, we investigate three different case studies.
The first dataset deals with binary crack detection. The
second one is dedicated to localizing damage. Finally, a
multiclass task of bridge component recognition is investi-
gated as the third case study. Given the different complex-
ity, the original architecture is tailored for computational
efficiency. Therefore, the number of modular stacked units
inside the DBs is modified. The details are given in Table 1.
SAJEDI and LIANG 5

TA B L E 1 Dense block configuration training and inference phases. For a single input image,
Block ID Number of modular stacked layers multiple Monte Carlo dropout samples are drawn to
DB-1, DB-11 2 generate a bin of softmax probabilities, from which the
DB-2, DB-10 3 prediction and uncertainty metrics can be obtained.
DB-3, DB-9 4 All three models are trained using Keras API (Chollet,
DB-4, DB-8 5
2015) with a batch size of 2 on computers equipped with
NVIDIA GTX 1070 and GTX 1080, each with 8 GB of
DB-5, DB-7 6
memory. Observations are randomly shuffled and 80%
DB-6 (Bottle neck) 8
are considered for training and the remaining are used
as the test set. Moreover, 20% of training observations are
held out as a validation set. The model with the lowest
The general structure of the adopted deep learning validation loss is selected for testing. It should be noted
architecture for Bayesian inference is illustrated in Fig- that Bayesian inference is used for both validation and test
ure 2. Recall that the dropout layers are active in both sets.

Input
Downsampling
First convolution

DB-1 C TD DB-2 C TD DB-3 C TD DB-4 C TD DB-5

TD
Skip connection

Skip connection

Skip connection

Skip connection
DB-i Dense block (i) DB-6
C Concatenation (Bottle neck)

TD Transition down
TU Transition up
TU

C
Last convolution

Softmax C DB-11 C TU DB-10 C TU DB-9 C TU DB-8 C TU DB-7

...
Upsampling

+ +

Softmax bin Prediction MCSSD Entropy

(Monte Carlo dropout samples)

FIGURE 2 Bayesian inference with FC-DenseNet architecture (parts of this figure are inspired by Jégou et al., 2017)
6 SAJEDI and LIANG
Based on our experiments, Nadam optimizer with
exponential weight decay (per epochs) yields a faster con-
vergence with a fewer number of epochs. L2 regularization
(Goodfellow, Bengio, & Courville, 2016) is also used for
better numerical stability and reducing the chances of
overfitting. All models described in this section are trained
with a maximum of 200 epochs and an early stopping of
15 epochs based on the validation loss. A learning rate of
1.0e-4 and an exponential decay rate of 0.9996 are also used
in training. Further details on dropout hyperparameters
are discussed in Section 5.
3 APPLICATIONS OF MODEL
UNCERTAINTY
Bayesian inference has two potential advantages for deep
vision SHM. First, it can potentially improve the overall
performance of a model. Second, the predictor is equipped
with a measure of uncertainty. The importance of this
output is crucial for the condition assessment of critical
structures such as nuclear facilities where missing a
simple defect may lead to catastrophic outcomes. Quan-
tification of uncertainty helps further minimize the risk
of using data-driven models for safety-critical inspections.
Being informed of the lack of confidence in a model’s pre-
dictions, a decision-maker can further explore the scene.
However, merely quantifying the uncertainty may not be
sufficient for a practical application in the SHM industry.
This insufficiency stems from two important factors.
First, the majority of the proposed deep learning frame-
works in the literature are designed/optimized to partially
automate the SHM process. Assigning a human operator
to monitor the uncertainty output and to modify the
predictions is not feasible for certain SHM tasks that
require processed information in real time. For example,
automatic guidance of unmanned aerial vehicles (UAVs)
for inspections (e.g., Liang, Zheng, & Zhang, 2018; Zheng,
Chen, & Liang, 2019) will not be automatic anymore if
it is constantly interrupted by manual instructions. Nev-
ertheless, inspecting hundreds of frames generated per
second is laborious and potentially impossible in a timely
manner.
The second challenge is the interpretation of uncer-
tainty. Identifying the prediction mistakes may not be
visually evident from the values expressed in Equations (6)
and (7). The upcoming case studies in the manuscript
demonstrate this complexity, especially for the last two
datasets that include real-world objects (such as trees,
rivers, cars, etc.) and distortions (e.g., presence of shadow,
different camera angles).
The classification problems in this paper are based on
semantic segmentation where the output prediction and
uncertainty are generated for thousands of pixels in a
single image. The incentive behind this selection is the
presence of relatively large uncertainty output that is both
intellectually challenging and laborious to process for
a human operator. To tackle this issue, we develop two
concept solutions that process the raw model uncertainty
and provide valuable information. The two are explained
in what follows.
3.1 Uncertainty-assisted segmentation
(UAS)
The uncertainty masks correlate well with the boundaries
in the regions of interest (i.e., cracks, local damage, and
bridge components). Moreover, compared with correct
predictions, a considerable portion of misclassified pixels
have a relatively high value of model uncertainty. These
correlations visually make sense because one can perform
a side-by-side comparison between the uncertainty mask
and the ground truth.
When exploring the field or performing an additional
expert analysis is not possible, model uncertainty is the
only source of information to potentially enhance the
predictions. In such circumstances, a human operator
may also face challenges in how to utilize the MCSSD
or entropy for a better estimate of real conditions. For
example, this utilization is very complicated (or even
impossible) for a human if the CNNs are used for
regression tasks such as depth estimation of pixels.
In this section, we propose a way regarding how to
benefit from the uncertainty mask without human inter-
vention. This goal is achieved by introducing a second
data-driven model to automatically interpret the MCSSD
and entropy masks. We call it a surrogate model because
identifying uncertainty rules for thousands of pixels,
which may or may not be correlated, is not directly
possible in an image. After training the original model
and calibrating the learnable parameters, the uncertainty
output is available. Therefore, it is desired to construct a
second input by concatenating the original image with
the mean softmax probabilities (E(Si )), CSSDi for all
classes, and entropy in stacked channels (see Figure 3).
The surrogate model is trained in a supervised manner
with the second input and the same labeled pixels as the
output. Different from the initial Bayesian segmentation,
the surrogate model has access to uncertainty output. It
is trained to learn the underlying mapping between the
second input (image data combined with early prediction
uncertainties) and the ground truth. This process is
denoted as uncertainty-assisted segmentation (UAS).
In this paper, for the sake of comparison, we use the
same architecture of the segmentation network to train
SAJEDI and LIANG
Bayesian Segmentation
Network
(B)
(A)
Raw input
(e.g., an image)
FIGURE 3 UAS surrogate model with the second input
the UAS surrogate model. The only difference is the
construction of uncertainty input data. This helps to
ensure that the improvement in the results is solely due
to the proposed novel input. Therefore, factors such as
an in-depth architecture or change in the selection of
hyperparameters are not a potential reason for the refined
predictions. Given this similarity, the second model is
also Bayesian. This assumption is appropriate for the
comparative studies. However, future work can focus on
optimizing the surrogate model itself for accuracy and
computational efficiency in real-time implementations.
3.2 Prediction quality tagging
The UAS refines the prediction of the original deep learn-
ing architecture. However, depending on the design of
the data-driven model and the quality of the dataset, such
improvements could be limited. In other words, chances
for mistakes (while reduced) still exist.
The original Bayesian deep learning model is trained
and validated based on hyperparameters that optimize
the performance given the existing data. As a result, the
majority of output predictions are often acceptable if the
model is trained properly. Nevertheless, a decision-maker
could be concerned about less common situations in
which a model has made mistakes. This issue is of great
importance, especially in cases where the deep learning
model is designed to localize damage in monitoring
critical infrastructure. For example, structural defects may
propagate and failure to detect damage in early stages
could lead to costly or unjustified repairs.
When achieving a perfect data-driven model is not
possible, the alternative is to equip the automation frame-
work with a system that triggers human intervention.
In this section, the concept of prediction quality tagging
7
(A): Mean Softmax Probabilities (E(Si))
(B): CSSDi for all classes
(C): Entropy
Second Input Surrogate Model
+ + +
(C)
Refined Prediction
(PQT) is proposed with the idea of making such warning
systems. PQT provides insights on the robustness of the
predictions based on the model uncertainty. Similar to
UAS, a surrogate model is developed that takes stacked
channels of softmax probabilities, MCSSD, entropy, and
additionally the predicted labels as input. The goal of
this PQT surrogate model is to label each output of the
original model based on the quality of predictions, which
is a prior metric selected by decision-makers. For example,
the PQT surrogate can be calibrated (trained) to accept or
reject an output based on a threshold of global prediction
accuracy among labeled pixels in a segmentation mask. As
a supervised learning task, the PQT labels are generated
for training, validation, and test sets.
A CNN architecture, inspired by VGG-16 (Simonyan
& Zisserman, 2014), is adopted for the PQT surrogate
(Figure 4). The architecture has been optimized for speed
and robustness. The network starts with four convolution
computational blocks (CBs). The 3D output tensor after
the last CB is then reshaped into a vector to be fed to a
sequence of fully connected blocks (FCBs). The surrogate
model ends with a sigmoid activation to predict the like-
lihood of PQT labels. Dropout with a probability of 40% is
used before the dense layers to alleviate overfitting. Leaky
ReLU activation is also utilized after convolution and
dense layers with a 0.05 negative slope coefficient. Further
details on the application of PQT are given in Section 6.2.
4 IMAGE DATASETS
Three different datasets are investigated regarding the
proposed model uncertainty for deep vision SHM. A
brief description of real-world images used for each case
study is given in what follows. Each dataset has specific
challenges that will be discussed later in the paper.
8 SAJEDI and LIANG
(a) CB (n): Convolution with n filters + Batch normalization + Leaky ReLU + Max pooling
(b) FCB (n): Dropout + Fully connected layer with n units + Leaky ReLU
(c) Dense (1): The last fully connected layer with 1 output unit
Reshape
CB (512)
CB (256)
CB (256)
CB (128)
CB(a) (64)
Second Input (similar to UAS + the prediction mask as an extra channel)
FIGURE 4 PQT surrogate model
4.1 Crack forest (I)
Binary segmentation of cracks is one of the well-studied
areas for automated SHM. For the first case study, the
crack forest dataset (Shi, Cui, Qi, Meng, & Chen, 2016)
is utilized. The dataset includes 118 color images with
a resolution of 320 × 480 pixels. The images are taken
by a camera-equipped smartphone with nearly constant
photography setup (e.g., distance, exposure, aperture). The
dataset reflects information on surface road conditions.
The ground truth is processed to include a binary mask
with two classes: crack and background.
4.2 Local structural damage (II)
The second case study is dedicated to localizing the
presence of damage. The dataset is obtained from Liang
(2019), which includes 436 images with a resolution of
430 × 400 pixels. The observations are obtained from
two main sources: laboratory experiments on reinforced
concrete columns and postearthquake damage records of
bridges. For computational efficiency, images are resized
to 215 × 200 pixels.
4.3 Bridge components (III)
For the third case study, we propose an example that deals
with object recognition rather than damage diagnosis. As
mentioned in the literature review, automatic identifica-
tion of different objects (e.g., structural components) is
crucial, especially in guiding UAVs for inspection. Bridges
are important parts of the civil infrastructure that will be
benefited from such camera-equipped UAV inspections.
To this end, the third dataset is composed of 236 images
Sigmoid
Rejected
or
FCB (64) Dense(c) (1) Accepted
FCB (256)
FCB (512)
FCB(b) (1024)
Prediction quality tag
of highway bridges. While originally used to identify
bridge columns with bounding boxes (Liang, 2019), bridge
components are pixel-wise labeled by the authors for
semantic segmentation.
5 SENSITIVITY ANALYSES ON
DROPOUT HYPERPARAMETERS
Dropout probability ratio (Pdo ) and the number of Monte
Carlo samples Ns are the two main hyperparameters
that affect the performance of Bayesian inference. For
example, without dropout (Pdo = 0), Bayesian inference is
meaningless. In contrast, very high dropping probabilities
will lower the convergence speed and increase the chances
of underfitting. Moreover, a proper selection of Ns for
Monte Carlo integration in (5) is important. Small sample
sizes could reduce the stability of the model and the
quality of output predictions. Nevertheless, the inference
time grows almost linearly with respect to Ns , which can
affect the real-time implementation of the deep learning
architecture. The focus of this section is to perform a
series of extensive sensitivity analyses to provide insight
on the selection of these hyperparameters, and to select
a final model for the three case studies. Therefore, the
results shown in this section are based on the validation
set. The overall performance for the unseen test sets will
be discussed later in Section 6.
5.1 Dropout probability (Pdo )
The effects of dropout probability are inspected by per-
forming a series of sensitivity analyses where 14 values
of Pdo (%) ∈ {1, 2, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 70},
are considered. The training process is accompanied by
SAJEDI and LIANG 9

Data set (I) Data set (II) Data set (III)

MCA (%)

MCA (%)

MCA (%)
(a) (b) (c)
MCA (%)

MCA (%)

MCA (%)
(d) (e) (f)

FIGURE 5 Sensitivity analysis plots for dropout hyperparameters

TA B L E 2 The selection of Pdo hyperparameter for the datasets

Dataset Image size Pdo (%) T20 (s) T50 (s)
I 320 × 480 20 1.31 3.11

MIoU (%)
II 215 × 200 5 0.38 0.94
III 224 × 224 15 0.44 1.06

the evaluation of the validation set per epoch because the

model with the lowest validation loss is used for further
testing. As a result, it is necessary to select Ns as a prior.
A second bin of Ns ∈ {10, 20, 50, 100, 200} is utilized to
address this issue. In total, 70 models are trained and
evaluated for each case study considering all the possible
combinations (Figure 5a–c). For each dataset, the model
with the highest mean class accuracy (MCA) is selected.
While the optimum value of Pdo depends on both the
deep learning architecture and the characteristics of the
dataset, we find that values between 5% and 20% yield
reasonable performance for the three datasets.
5.2 Monte Carlo sample size
To investigate the effects of sample size, the best models
obtained from Section 5.1 are considered for each dataset
(Table 2). The validation MCA values for different sample
sizes are compared in Figure 5d–f. Note that given the
stochastic nature of dropout operation, the reported MCA
values in this figure are the average of repeating the
sampling 10 times. The Bayesian inference data points
are also compared with the benchmark dashed line (i.e.,
the standard dropout) where the Bayesian. Moreover, the
average inference times per image for the sample size of
20 and 50 (denoted as T20 and T50 ) are given in Table 2.
Ns
FIGURE 6 Variations of MIoU for Dataset III
From this point forward, Ns = 50 is used to evaluate the
three models for the testing set. However, with a slight
compromise in the performance, one may utilize a smaller
sample for computational efficiency. Increasing Ns makes
the prediction output more stable. As an example given
in Figure 6, for the third dataset, this stability is shown by
whisker plots of the mean intersection over union (MIoU).
6 EXPERIMENTAL VALIDATIONS
The proposed concepts in Sections 2 and 3 are investigated
with more thorough analyses and visual examples are
given when necessary. This section is composed of two
parts. First, the testing performance is investigated for
different approaches including the surrogate model. In
the second part, an example is given to show how the PQT
concept can be useful for SHM.
10 SAJEDI and LIANG

TA B L E 3 Comparison of testing performance metrics for the binary crack detection case study (UW-MAP)
Class accuracy (%) IoU (%) F1-score (%)
Background Crack Average Background Crack Average Background Crack Average
No dropout 99.54 47.89 73.72 98.51 39.04 68.78 99.25 56.16 77.71
Standard dropout 99.44 62.69 81.07 98.70 49.06 73.88 99.35 65.83 82.59
Bayesian inference 99.50 62.56 81.03 98.76 50.10 74.43 99.37 66.75 83.06
Surrogate model 99.37 65.27 82.32 98.69 49.77 74.23 99.34 66.46 82.90
Note: Values with bold font refer to the strategy with the best performance.

6.1 Segmentation performance TA B L E 4 Change in testing performance metrics for different

Each of the following subsections includes an indepen-
dent comparative investigation on one of the datasets. The
testing performance of the Bayesian inference and the
corresponding surrogate models are presented. Addition-
ally, two other models are also included for comparison:
no dropout and standard dropout models. Having a total
of four models, it is possible to investigate the extent of
improvements with respect to different approaches. We
show that for the three case studies, Bayesian inference
and surrogate models have superior performance com-
pared with the one with standard dropout that utilizes the
common weight averaging technique.
6.1.1 Binary crack detection (I)
Crack segmentation is a challenging vision problem
because of the significant class imbalance. Images that
reflect a cracked surface are commonly dominated by
background pixels. In the current dataset, less than 2% of
the ground truth pixels are labeled as crack. As a result,
global accuracy (GA) is not a proper metric for this task.
For example, if one labels all pixels as background, GA will
be approximately 98%, while no cracks are successfully
detected.
The original implementation of FC-DenseNet utilizes
uniform weights (UW) in the loss function and maxi-
mum a-posteriori (MAP) decision rule, which takes the
maximum of softmax probabilities. A summary of results
is presented in Table 3. Considering MCA and crack
recall, the surrogate model has the highest performance.
However, the Bayesian model shows better IoU and
F1-score. Both models perform better than the one with
standard dropout. Some visual examples based on UWs
and MAP strategy are given in Figure 7. Because the
crack pixels comprise a low percentage of images, the
observations in Figure 7 are zoomed-in crops of 64 ×
96 pixels. This has been done for illustration purposes
only.
Eigen and Fergus (2015) proposed a median frequency
weight (MFW) assignment to help increase the MCA. In
strategies
Metric UW-MAP UW-ML MFW-MAP
Crack recall (%) 62.56 80.78 93.57
IoU (%) 50.10 37.95 31.36
F1-score (%) 66.75 55.01 47.75
Precision (%) 71.54 41.71 32.27
more recent work, Chan, Rottmann, Hüger, Schlicht, and
Gottschalk (2019) proposed modifying the decision rule
instead of adjusting the observation weights in the loss
function. This goal is achieved by weighing the posterior
class probabilities of crack and background classes with
their inverse frequency in the training data. As an alter-
native to MAP, the decision rule for this type of inference
is called the maximum likelihood (ML). Considering
the two previous strategies and UW-MAP, the variations
in the performance metrics are shown for the Bayesian
inference model in Table 4. For each strategy, the first
term relates to the way that weights are balanced and the
second indicates the decision rule to assign a label to each
pixel.
The results in Table 4 follow a trend that indicates a
trade-off between crack recall and other metrics. The
MFW-MAP model captures the highest number of cracks
by increasing more than 30% on the crack recall. However,
it is the most conservative by having the lowest precision.
In contrast, UW-MAP has the best performance regarding
the other three metrics. While it is ideal to have perfect
values for all metrics, the Bayesian UW-ML, with a
reasonable balance of all metrics, may be selected for
industrial applications. However, the choice of strategy
may differ depending on the application.
Aside from the improvements in the prediction results,
the main advantage of Bayesian inference for vision SHM
is the uncertainty output. The two model’s uncertainty
measures (MCSSD and entropy) are illustrated in Figure 7.
It can be observed that model uncertainty is relatively
high at the crack boundaries and, more importantly, the
model uncertainty correlates well with misclassifications.
For example, the model shows high uncertainty in iden-
tifying the background noises as crack. Moreover, both
SAJEDI and LIANG 11

Standard Bayesian Surrogate MCSSD MCSSD Entropy Entropy

Image Ground truth
dropout Inference Model Bayesian Surrogate Bayesian Surrogate

FIGURE 7 Sample testing observations for damage localization using Bayesian inference and UAS surrogate model

TA B L E 5 Comparison of testing performance metrics for the damage localization case study
Class accuracy (%) IoU (%) F1-score (%)
Background Damage Average Background Damage Average Background Damage Average
No dropout 92.14 66.26 79.20 86.12 48.07 67.10 92.54 64.93 78.74
Standard dropout 94.20 76.68 85.44 89.85 59.92 74.89 94.66 74.94 84.80
Bayesian inference 93.30 78.18 85.74 89.26 59.08 74.17 94.32 74.28 84.30
Surrogate model 93.98 78.12 86.05 89.90 60.54 75.22 94.68 75.42 85.05
Note: Values with bold font refer to the strategy with the best performance.

metrics are good indicators of model uncertainty while the surrogate model, which is the best approach. Some
being different in values and distribution. Considering detailed examples are provided in Figure 8. It is clear that
the correlation between mistakes and model uncertainty, the surrogate model has less uncertainty compared to the
entropy is more conservative. original Bayesian inference.

6.1.2 Damage localization (II) 6.1.3 Bridge component recognition

Unlike crack forest with close-up figures and minimum
background noise, the images in this dataset commonly
have complex backgrounds. This complexity is even ampli-
fied for lab experiments where the structural elements
are commonly covered with testing instruments. This
dataset is also labeled with a binary mask of two classes:
damage and background. The performance metrics for
this task are presented in Table 5. For all the considered
metrics, significant improvements are observed using
(III)
Different from the first two case studies, this one deals
with a multiclass segmentation problem. The assumed
structural components of interest for inspection are the
bridge column (pier), cap beam, abutment, foundation,
and superstructure. All the other miscellaneous objects
in the scene are labeled as background. An important
point about this dataset is that the abutment and foun-
dation are usually buried inside the ground or soil for
12
Standard Bayesian
Image Ground truth
dropout Inference
FIGURE 8 Sample testing observations for damage localization using Bayesian inference and UAS surrogate model
stabilization and are commonly less exposed. As a result,
the frequency of pixel observations for the foundation
and abutment is considerably less than that of the other
classes.
A summary of performance metrics is presented in
Table 6. It is evident that, compared with other com-
ponents, the less frequent classes of abutment and
foundation are detected with less accuracy. Similar to the
previous two case studies, the superior performance of
the Bayesian inference and UAS surrogate model over the
benchmark model is evident.
The benchmark has higher precision for the classes of
cap beam and foundation. However, this high precision
SAJEDI and LIANG
Surrogate MCSSD MCSSD Entropy Entropy
Model Bayesian Surrogate Bayesian Surrogate
is accompanied by a significantly smaller recall compared
with the Bayesian model. In this case, high precision does
not imply better performance. For instance, in a sample
test image, we observe that only 1.79% of the pixels are
correctly captured as a foundation by the no dropout
model with a 100% precision. Therefore, precision is
excluded from the results in Table 6. Instead, F1-score and
IoU are selected as better alternatives.
Sample testing observations for bridge component
segmentation are presented in Figure 9. Similar to the
other datasets, the presence of misclassification has
a good correlation with the model uncertainty. Yet,
there exist cases where the uncertainty is low while the
SAJEDI and LIANG 13

TA B L E 6 Comparison of testing performance metrics for bridge component recognition

Strategy Background Superstructure Column Cap beam Foundation Abutment Average

Class accuracy (%)
No dropout 95.44 87.88 86.90 19.47 1.79 22.13 52.27
Standard dropout 93.97 88.84 88.64 64.65 9.93 64.72 68.46
Bayesian inference 95.89 88.37 88.89 51.36 22.91 61.80 68.20
Surrogate model 95.45 89.03 88.63 65.36 24.34 62.71 70.92
IoU (%)
No dropout 81.54 70.73 69.80 18.98 1.79 21.75 44.10
Standard dropout 85.33 76.93 77.80 55.50 8.84 49.53 58.99
Bayesian inference 86.23 74.59 76.60 47.87 22.30 56.01 60.60
Surrogate model 87.20 76.94 78.20 55.47 23.23 57.44 63.08
F1-score (%)
No dropout 89.83 82.85 82.21 31.89 3.51 35.27 54.26
Standard dropout 92.09 86.96 87.51 71.38 16.24 66.25 70.07
Bayesian inference 92.61 85.45 86.75 64.75 36.47 71.80 72.97
Surrogate model 93.16 86.97 87.77 71.35 37.70 72.97 74.99
Note: Values with bold font refer to the strategy with the best performance.

segmentations are not accurate. For example, Figure 9(7)
illustrates a column in a laboratory setup. The column
is properly identified but the upper region of the image
is misclassified as the superstructure. While this is an
unlikely scene to happen for an outdoor inspection, it is
important to be aware of the cases where the model uncer-
tainty does not reflect mistakes. In this example, the model
shows confidence in predicting that the superstructure is
the closest class for the roof of a laboratory compared with
other possible classes in the dataset. One potential reason
for this type of inconsistency is the fact that no knowledge
about the existence of the laboratory roof is introduced in
data labeling. It is important to consider the limitations
of supervised learning caused by the lack of sufficient
information in the dataset. Such inconsistencies can be
alleviated by enlarging the training set for such observa-
tions and potentially increasing the number of possible
classes.
In the two previous binary segmentation problems,
despite the difference in the relative values, the two
uncertainty metrics follow a similar distribution. In this
multiclass dataset, noticeable differences between the
distribution patterns of entropy and MCSSD are observed.
Some of the examples in Figure 9 highlight this difference
where some pixels have relatively high entropy but low
MCSSD and vice versa. Observations like this will make
the incorporation of uncertainty for the decision-making
challenging. It is difficult or impossible for a human to
simultaneously interpret the entropy and MCSSD for all
pixels and modify the decision accordingly. This phe-
nomenon further justifies the adoption of a UAS model
that is explained in Section 3.1.
6.1.4 Discussion of results
The comparisons of testing metrics in Tables 3, 5, and 6 are
indicators of overall better performance for the Bayesian
and surrogate models. In all three datasets, IoU, F1-score,
and class accuracies are improved compared to the one
with standard dropout. This improvement is more signif-
icant for the third dataset with bridge components that
are less frequent and thus more difficult to recognize (i.e.,
abutment and foundation classes).
As mentioned in Section 3.1, the surrogate model is
Bayesian, and therefore, also outputs uncertainty. A side
by side comparison between the two MCSSD/entropy
masks is also presented in the given examples. The
color scale is normalized for each observation. It can be
observed that the surrogate model has less uncertainty
compared with the original Bayesian mask.
6.2 Validation of PQT surrogate model
One of the main topics of concern in this paper is the
reliability of deep vision models for automated SHM. After
observing the results in Section 6.1, one can notice that
despite the presence of a surrogate model, mistakes, while
reduced, still exist. This subsection focuses on an example
of the PQT concept mentioned earlier as a way to trigger
human interventions.
Defect classification is probably one of the most con-
cerning fields in SHM where mistakes of a data-driven
model can have grave consequences. Therefore, we select
the second dataset to validate the PQT concept. While
14 SAJEDI and LIANG

Standard Bayesian Surrogate MCSSD MCSSD Entropy Entropy

Image Ground truth
dropout Inference Model Bayesian Surrogate Bayesian Surrogate

7
Segmentation class dictionary
Background Superstructure Column Cap Beam Abutment Foundation

FIGURE 9 Sample testing observations for bridge component recognition using Bayesian inference and UAS surrogate model

different metrics can be used to trigger an alarm for human
interventions, we select the damage class accuracy (DCA)
as the metric for acceptance or rejection. A threshold of
75% is used to label each segmentation output based on the
quality of damage prediction. It is possible to calibrate the
PQT for other metrics such as GA or MCA with different
performance objectives.
The PQT surrogate model analyzes all the predictions
and their corresponding uncertainty mask, and labels
them based on the quality of prediction. This information
can be used to pass on a handful of problematic segmen-
tations to human agents for further examination. Being
equipped with a PQT surrogate, a damage detector com-
puter can use such labels to trigger human intervention if
necessary.
The testing results of the PQT surrogate proposed
in Section 3.2 are given in Figure 10. Considering the
rejection criteria, the model has 78.2% accuracy in label-
ing observations where mistakes often happen in the
proximity of the prior threshold for DCA. Nevertheless,
a decision-maker can adjust not only DCA but also the
decision rule for the PQT surrogate for more conservative
quality tagging. For example, instead of 50%, one may
utilize 60% and more output probability to accept the
prediction output. The importance of PQT surrogate is
further highlighted in Figures 10(3) and (6). Both images
have a high GA because the damaged area is relatively
small. While the original Bayesian model has missed
damage in both images, the PQT surrogate can be used to
send an alarm signal for further inspections.
SAJEDI and LIANG 15

Image Ground truth Prediction Mistakes MCSSD Entropy

Accepted DCA: 98.17

1

DCA: 46.96
2 Rejected

DCA: 0.0
3 Rejected

Accepted DCA: 90.59

4
DCA: 79.42

5 Rejected

DCA: 0.0
6 Rejected

FIGURE 10 Visual comparison between the initial and surrogate segmentation results

7 CONCLUSION In this paper, we leverage on deep Bayesian neural

Recent advances in computer vision and deep learn-
ing have had a major impact on vision-based struc-
tural inspections. Considering an ever-growing interest
in the application of deep learning models for SHM,
it is necessary to develop a framework that quanti-
fies the model’s confidence such that decision-makers
can make risk-informed decisions in different circums-
tances.
networks for vision-based structural inspections. The
novel frameworks proposed in this paper are validated
with three different datasets of real-world images, while
the performance is monitored for different metrics. It is
shown that Bayesian inference can further boost the over-
all performance while providing an uncertainty output
for the corresponding predictions. A series of sensitivity
analyses are performed to investigate the effects of dropout
hyperparameters on Bayesian inference.
16
Beyond quantification, it is shown that the interpreta-
tion of uncertainty output can be challenging to manually
analyze and integrate with an automated SHM system.
Therefore, the concept of surrogate models is introduced
to further benefit from uncertainty output. The UAS
method is proposed to automatically refine the prediction
results from the original Bayesian model. Moreover, PQT
is developed as a means to trigger human interventions in
an efficient manner.
The proposed methodology in this paper can be utilized
to equip the existing vision models for inspection and
monitoring with tools that can quantify and benefit from
output model uncertainty. In the absence of huge datasets
for training, Bayesian inference can be an effective tool to
make visual inspections using deep vision models more
reliable. The presented examples are the cornerstone to
further expand uncertainty in other related fields of SHM
for future research. For example, the uncertainty masks in
the crack segmentation can be used in the calculation of
crack width and heights in the postprocessing steps.
REFERENCES
Ahmadlou, M., & Adeli, H. (2010). Enhanced probabilistic neural
network with local decision circles: A robust classifier. Integrated
Computer-Aided Engineering, 17(3), 197–210.
Ai, D., Jiang, G., Kei, L. S., & Li, C. (2018). Automatic pixel-level pave-
ment crack detection using information of multi-scale neighbor-
hoods. IEEE Access, 6, 24452–24463.
Amezquita-Sanchez, J. P., Park, H. S., & Adeli, H. (2017). A novel
methodology for modal parameters identification of large smart
structures using MUSIC, empirical wavelet transform, and Hilbert
transform. Engineering Structures, 147, 148–159.
Azimi, M., & Pekcan, G. (2019). Structural health monitoring using
extremely-compressed data through deep learning. Computer-
Aided Civil and Infrastructure Engineering, 35(6), 597–614.
Bang, S., Park, S., Kim, H., & Kim, H. (2019). Encoder–decoder net-
work for pixel-level road crack detection in black-box images.
Computer-Aided Civil Infrastructure Engineering, 34(8), 713–727.
Cha, Y. J., Choi, W., & Büyüköztürk, O. (2017). Deep learning-based
crack damage detection using convolutional neural networks.
Computer-Aided Civil Infrastructure Engineering, 32(5), 361–378.
Cha, Y. J., Choi, W., Suh, G., Mahmoudkhani, S., & Büyüköztürk,
O. (2018). Autonomous structural visual inspection using region-
based deep learning for detecting multiple damage types.
Computer-Aided Civil Infrastructure Engineering, 33(9), 731–747.
Chan, R., Rottmann, M., Hüger, F., Schlicht, P., & Gottschalk,
H. (2019). Application of decision rules for handling class
imbalance in semantic segmentation. arXiv preprint arXiv:
.08394.
Chen, X., Girshick, R., He, K., & Dollár, P. (2019). Tensormask:
A foundation for dense object segmentation. arXiv preprint
arXiv:.12174.
Chollet, F. (2015). Keras. Retrieved from https://github.com/keras-
team/keras
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei–Fei, L (2009).
Imagenet: A large-scale hierarchical image database. In IEEE Con-
SAJEDI and LIANG
ference on Computer Vision and Pattern Recognition, 2009 (CVPR
2009). IEEE.
Deng, J., Lu, Y., & Lee, V. C.-S. (2020). Concrete crack detection with
handwriting script interferences using faster region-based convo-
lutional neural network. Computer-Aided Civil and Infrastructure
Engineering, 35(4), 373–388. https://doi.org/10.1111/mice.12497.
Ebrahimkhanlou, A., Dubuc, B., & Salamone, S. (2019). A general-
izable deep learning framework for localizing and characterizing
acoustic emission sources in riveted metallic panels. Mechanical
Systems and Signal Processing, 130, 248–272.
Eigen, D., & Fergus, R. (2015). Predicting depth, surface normals and
semantic labels with a common multi-scale convolutional archi-
tecture. In Proceedings of the IEEE International Conference on
Computer Vision.
Eltouny, K. A., & Liang, X. (2020). A nonparametric unsuper-
vised learning approach for structural damage detection. In 17th
World Conference on Earthquake Engineering (17WCEE), Sendai,
Japan.
Gal, Y., & Ghahramani, Z. (2015). Bayesian convolutional neural net-
works with Bernoulli approximate variational inference. arXiv
preprint arXiv:.02158.
Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approxima-
tion: Representing model uncertainty in deep learning. In Interna-
tional Conference on Machine Learning.
Gao, Y., & Mosalam, K. M. (2018). Deep transfer learning for image-
based structural damage recognition. Computer-Aided Civil and
Infrastructure Engineering, 33(9), 748–768.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Cam-
bridge, MA: MIT Press.
Graves, A. (2011). Practical variational inference for neural networks.
In Advances in Neural Information Processing Systems.
Hoskere, V., Narazaki, Y., Hoang, T., & Spencer, B. F., Jr (2018).
Vision-based structural inspection using multiscale deep convo-
lutional neural networks. arXiv preprint arXiv:1805.01055.
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q.
(2017). Densely connected convolutional networks. In Proceed-
ings of the IEEE Conference on Computer Vision and Pattern
Recognition.
Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., & Bengio, Y. (2017).
The one hundred layers tiramisu: Fully convolutional densenets
for semantic segmentation. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition Workshops.
Karypidis, D. F., Berrocal, C., Rempling, R., Granath, G., & Simons-
son, P. J. I. (2019). Structural health monitoring of RC structures
using optic fiber strain measurements: A deep learning approach.
In 2019 IABSE Congress. New York.
Kendall, A., Badrinarayanan, V., & Cipolla, R. (2015). Bayesian
segnet: Model uncertainty in deep convolutional encoder-
decoder architectures for scene understanding. arXiv preprint:
ArXiv:.1511.02680.
Kendall, A., & Gal, Y. (2017). What uncertainties do we need in
Bayesian deep learning for computer vision? In Advances in Neu-
ral Information Processing Systems.
Koller, D., & Friedman, N. (2009). Probabilistic graphical models:
Principles and techniques. Cambridge, MA: MIT Press.
Li, R., Yuan, Y., Zhang, W., & Yuan, Y. (2018). Unified vision-based
methodology for simultaneous concrete defect detection and
geolocalization. Computer-Aided Civil and Infrastructure Engi-
neering, 33(7), 527–544.
SAJEDI and LIANG
Li, S., Zhao, X., & Zhou, G. (2019). Automatic pixel-level multiple
damage detection of concrete structure using fully convolutional
network. Computer-Aided Civil Infrastructure Engineering, 34(7),
616–634.
Liang, X. (2019). Image-based post-disaster inspection of reinforced
concrete bridge systems using deep learning with Bayesian opti-
mization. Computer-Aided Civil Infrastructure Engineering, 34(5),
415–430.
Liang, X., Zheng, M., & Zhang, F. (2018). A scalable model-based
learning algorithm with application to UAVs. IEEE Control Sys-
tems Letters, 2(4), 839–844.
Liu, Y. F., Nie, X., Fan, J. S., & Liu, X. G. (2020). Image-based
crack assessment of bridge piers using unmanned aerial vehi-
cles and three-dimensional scene reconstruction. Computer-Aided
Civil Infrastructure Engineering, 35(5), 511–529. https://doi.org/10.
1111/mice.12501.
Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional net-
works for semantic segmentation. In Proceedings of the IEEE Con-
ference on Computer Vision and Pattern Recognition.
McAllister, R., Gal, Y., Kendall, A., Van Der Wilk, M., Shah,
A., Cipolla, R., & Weller, A. V. (2017). Concrete problems
for autonomous vehicle safety: Advantages of bayesian deep
learning. In International Joint Conferences on Artificial
Intelligence.
Narazaki, Y., Hoskere, V., Hoang, T. A., Fujino, Y., Sakurai, A., &
Spencer, B. F. Jr (2020). Vision-based automated bridge compo-
nent recognition with high-level scene consistency. Computer-
Aided Civil and Infrastructure Engineering, 35(5), 465–482.
https://doi.org/10.1111/mice.12505.
Oh, B. K., Glisic, B., Kim, Y., & Park, H. S. (2019). Convolutional neu-
ral network-based wind-induced response estimation model for
tall buildings. Computer-Aided Civil and Infrastructure Engineer-
ing, 34(10), 843–858.
Rafiei, M. H., & Adeli, H. (2016). A novel machine learning model for
estimation of sale prices of real estate units. Journal of Construc-
tion Engineering and Management, 142, 04015066.
Rafiei, M. H., & Adeli, H. (2017a). NEEWS: A novel earthquake
early warning model using neural dynamic classification and neu-
ral dynamic optimization. Soil Dynamics Earthquake Engineering,
100, 417–427.
Rafiei, M. H., & Adeli, H. (2017b). A novel machine learning-based
algorithm to detect damage in high-rise building structures. The
Structural Design of Tall and Special Buildings, 26(18), e1400.
Rafiei, M. H., & Adeli, H. (2018a). Novel machine-learning model for
estimating construction costs considering economic variables and
indexes. Journal of Construction Engineering and Management,
144(12), 04018106.
Rafiei, M. H., & Adeli, H. (2018b). A novel unsupervised deep learn-
ing model for global and local health condition assessment of
structures. Engineering Structures, 156, 598–607.
Rafiei, M. H., Khushefati, W. H., Demirboga, R., & Adeli, H. (2017).
Supervised deep restricted Boltzmann machine for estimation of
concrete. ACI Materials Journal, 114(2), 237–244.
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional
networks for biomedical image segmentation. In International
Conference on Medical Image Computing and Computer-Assisted
Intervention. Springer.
Sajedi, S. O., & Liang, X. (2019a). A convolutional cost-sensitive
crack localization algorithm for automated and reliable RC bridge
17
inspection. In 10th New York City Bridge Conference. New York
City.
Sajedi, S. O., & Liang, X. (2019b). Intensity-based feature selection for
near real-time damage diagnosis of building structures. In IABSE
Congress. New York City, The International Association for Bridge
and Structural Engineering.
Sajedi, S. O., & Liang, X. (2020a). A data-driven framework for near
real-time and robust damage diagnosis of building structures.
Structural Control & Health Monitoring, 27, e2488.
Sajedi, S. O., & Liang, X. (2020b). Deep Bayesian U-nets for efficient,
robust and reliable post-disaster damage localization. 17th World
Conference on Earthquake Engineering (17WCEE), Sendai, Japan.
Sajedi, S. O., & Liang, X. (2020c). Vibration-based semantic dam-
age segmentation for large-scale structural health monitoring.
Computer-Aided Civil and Infrastructure Engineering, 35(6), 579–
596.
Shi, Y., Cui, L., Qi, Z., Meng, F., & Chen, Z. (2016). Automatic
road crack detection using random structured forests. IEEE
Transactions on Intelligent Transportation Systems, 17(12), 3434–
3445.
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556.
Spencer, B. F., Hoskere, V., & Narazaki, Y. (2019). Advances in com-
puter vision-based civil infrastructure inspection and monitoring.
Engineering, 5(2), 199–222.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdi-
nov, R. (2014). Dropout: A simple way to prevent neural networks
from overfitting. The Journal of Machine Learning Research, 15(1),
1929–1958.
Tong, Z., Gao, J., Sha, A., Hu, L., & Li, S. (2018). Convolutional neural
network for asphalt pavement surface texture analysis. Computer-
Aided Civil Infrastructure Engineering, 33(12), 1056–1072.
Xue, Y., & Li, Y. (2018). A fast detection method via region-based fully
convolutional neural networks for shield tunnel lining defects.
Computer-Aided Civil Infrastructure Engineering, 33(8), 638–654.
Yang, X., Li, H., Yu, Y., Luo, X., Huang, T., & Yang, X. (2018). Auto-
matic pixel-level crack detection and measurement using fully
convolutional network. Computer-Aided Civil and Infrastructure
Engineering, 33(12), 1090–1109.
Zhang, A., Wang, K. C. P., Li, B., Yang, E., Dai, X., Peng, Y., . . . Chen,
C. (2017). Automated pixel-level pavement crack detection on 3d
asphalt surfaces using a deep-learning network. Computer-Aided
Civil and Infrastructure Engineering, 32(10), 805–819.
Zhang, C., Chang, C. C., & Jamshidi, M. (2019). Concrete bridge sur-
face damage detection using a single-stage detector. Computer-
Aided Civil Infrastructure Engineering, 35(4), 389–409.
Zheng, M., Chen, Z., & Liang, X. (2019). A preliminary study on a
physical model oriented learning algorithm with application to
UAVs. In ASME 2019 Dynamic Systems and Control Conference.
American Society of Mechanical Engineers Digital Collection.
How to cite this article: Sajedi SO, Liang X.
Uncertainty-assisted deep vision structural health
monitoring. Comput Aided Civ Inf. 2020; 1–17.
https://doi.org/10.1111/mice.12580