Attention-based image captioning for structural health assessment of apartment buildings

Abstract

Automated visual assessment report generation in structural health monitoring (SHM) offers advantages for building inspections. However, current vision-based approaches that focus primarily on local surface detection cannot be directly used for inspection reports without further interpretation of the detected labels and coordinator metrics for an appropriate serviceability assessment. To address this gap, this paper presents an automated textual assessment framework for retrieving and generating linguistic descriptions of building component images. Six attention-based captioning methods were constructed based on convolutional neural network (Inception-V3, Xception, and ResNet50) and recurrent neural network (GRU, LSTM), and experimented via 7430 pairs of building component images and captions. The results indicated that the proposed methods had good predictive power and ResNet50-LSTM outperformed other methods with average precision, recall, and F1 scores of 0.84, 0.74, and 0.79, respectively. This paper highlights the potential of the image captioning approach for producing accurate and timely periodic structural assessment reports. © 2024 Elsevier B.V.