Real-time Environmental Virtualization Method for Telemanipulation of Dynamic Objects Using VR Interface

Abstract

With the evolution of robotic technology, the expansion of operations into challenging environments underscores the growing need for effective teleoperation systems. In such an environment, robots or machines can improve the efficiency and safety of tasks by delivering more detailed and accurate information to workers through virtual reality (VR). Current teleoperation systems have limitations in providing a comprehensive understanding of the work environment. Accordingly, this study proposes a technology that utilizes VR to provide a high level of telepresence to workers and enable intuitive control. To achieve this, we introduce a pre-generated computer-assisted design model for static objects beyond the viewing area of RGB-D cameras and a method to update the point cloud of the target objects, which are dynamic objects, in real-time. By incorporating this information, we created a 3D visual map and delivered it to the operator in real-time through HMD, enabling the operator to clearly recognize the robot's current location and surroundings. In addition, we introduced hand motion recognition through HMD viewpoints and VR controllers, allowing the operator to intuitively control the robot. These techniques can improve the efficiency and safety of remote work.