Implementation experiments on convolutional neural network training using synthetic images for 3D pose estimation of an excavator on real images

Abstract

Remote and descriptive visualization of spatio-temporal information of excavator activities may increase awareness about jobsite hazards and operational performance in earthwork operations. One of the emerging approaches to collect this information is to extract the 3D pose of an excavator from the video frames using a convolutional neural network (CNN). However, this method requires labeling the training datasets, which are difficult to prepare because of conditions unsuitable for installing the motion capture sensors. This study investigates the performance of a CNN for estimating the 3D pose when trained on a synthetic dataset. In particular, a kinematic constraint is proposed to update the model parameters efficiently during training. The results show that the proposed method estimated the 3D poses of a real excavator with an average pose error of 9.63°. Hence, the proposed data augmentation method could help address the training data issues and improves the learning of real data complexity.