An Application of Cycle GAN for Creating Generated Real Training Images with 3D Excavator Pose Labels from a Synthetic Model

Abstract

3D excavator poses providing the motion information of the boom, arm, and bucket in 3D space support monitoring excavator activities for safety and productivity management in earthwork. Thus, previous studies have attempted to estimate 3D excavator poses using deep learning relying on the large data with high-quality annotations, which requires time-consuming and manual processes. To address this challenge, this study proposes cycle GAN to automatically create large generated real training images with 3D pose labels from synthetic images. The proposed model is trained on 800 pairs of synthetic and real images and evaluated through pre-trained ResNet50-based 3D pose estimations. The results reveal that 3D pose model trained on generated data, reaching 0.50 m key-point loss and 8.53-degree angle loss for testing on generated images, and 9.33-degree angle loss for testing on real images, yielded better results than model trained on synthetic data (i.e., 0.64 m, 15.18-degree, and 15.39-degree, respectively). This demonstrates the effectiveness of the proposed method for generating training images from synthetic images for 3D pose estimation. This 3D pose estimated from generated images enables construction managers to monitor excavator safety and productivity in the construction sites.