A Semi-Supervised Self-Training Method to Develop Assistive Intelligence for Segmenting Multiclass Bridge Elements from Inspection Videos
Abstract
Bridge inspection is an important step in preserving and rehabilitating transportation infrastructure for extending their service lives. The advancement of mobile robotic technology allows the rapid collection of a large amount of inspection video data. However, the data are mainly the images of complex scenes, wherein a bridge of various structural elements mix with a cluttered background. Assisting bridge inspectors in extracting structural elements of bridges from the big complex video data, and sorting them out by classes, will prepare inspectors for the element-wise inspection to determine the condition of bridges. This article is motivated to develop an assistive intelligence model for segmenting multiclass bridge elements from the inspection videos captured by an aerial inspection platform. With a small initial training dataset labeled by inspectors, a Mask Region-based Convolutional Neural Network pre-trained on a large public dataset was transferred to the new task of multiclass bridge element segmentation. Besides, the temporal coherence analysis attempts to recover false negatives and identify the weakness that the neural network can learn to improve. Furthermore, a semi-supervised self-training method was developed to engage experienced inspectors in refining the network iteratively. Quantitative and qualitative results from evaluating the developed deep neural network demonstrate that the proposed method can utilize a small amount of time and guidance from experienced inspectors (3.58 h for labeling 66 images) to build the network of excellent performance (91.8% precision, 93.6% recall, and 92.7% f1-score). Importantly, the article illustrates an approach to leveraging the domain knowledge and experiences of bridge professionals into computational intelligence models to efficiently adapt the models to varied bridges in the National Bridge Inventory.
Recommended Citation
M. M. Karim et al., "A Semi-Supervised Self-Training Method to Develop Assistive Intelligence for Segmenting Multiclass Bridge Elements from Inspection Videos," Structural Health Monitoring, SAGE Publications, Jan 2021.
The definitive version is available at https://doi.org/10.1177/14759217211010422
Department(s)
Civil, Architectural and Environmental Engineering
Second Department
Computer Science
Research Center/Lab(s)
INSPIRE - University Transportation Center
Keywords and Phrases
active learning; AI-human collaboration; bridge inspection; human-in-the-loop; multiclass segmentation; self-training; temporal coherence; Transfer learning
International Standard Serial Number (ISSN)
1475-9217; 1741-3168
Document Type
Article - Journal
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2021 SAGE Publications, All rights reserved.
Publication Date
01 Jan 2021
Comments
U.S. Department of Transportation, Grant 69A3551747126