Human Imitation Manipulator System Based on 2D Image Recognition

Abstract

This paper proposes a control system that uses deep learning to extract the positions of joints from shoulder to hand from 2D images, enabling a manipulator to mimic human movements. The proposed system utilizes a 2D camera to capture the appearance of a person as an image, and employs deep learning-based object recognition techniques to extract 3D coordinates of joints from the images. The extracted coordinates are then converted into vectors to obtain joint-specific rotation angles, which are subsequently used as input for controlling the manipulator. The simulation environment is implemented using ROS Gazebo and Moveit packages, while the actual robot control is conducted using Python and C++ for improved response speed. The functionality of the proposed system is validated through simulations and by employing a manipulator.