Offline Goal-Conditioned Model-Based Reinforcement Learning in Pixel-Based Environment

Abstract

Model-based Reinforcement Learning (RL), with its capacity to learn and utilize a dynamics model for planning, stands out for its enhanced data efficiency, particularly in robotics, compared to its model-free RL. Despite these advancements RL algorithms commonly face challenges related to data inefficiency and the complexity of formulating reward functions. To address this issues, offline RL emerges as a promising approach, training policies from preexisting data without online interactions. However, a preexisting data will inevitably fail to encompass the full state-action space, potentially causing significant extrapolation errors. Furthermore, offline RL requires fully labeled data, which can be costly to acquire in large quantities. Goal-conditioned RL utilizes reward-free data, which can alleviate limitations of offline RL. In this paper, we demonstrate how to combine offline goal-conditioned RL with model-based RL to solve complex tasks in robotics. Our evaluation using image observations indicates that our method could effectively tackle real world tasks.