Gameplanner: A Game-Theoretical Landmark Planning for Multi-Agent Goal-Conditioned Reinforcement Learning

Abstract

Off-policy multi-agent reinforcement learning in decentralized settings faces key problems of non-stationarity and sparse rewards. When applied to goal-conditioned tasks, conventional planners often lead to conflicts and deadlocks, as each agent plans optimally only for itself while treating others as dynamic obstacles. To solve these problems, we propose Gameplanner, a framework that coordinates agents using a game-theoretical planner for goal selection, starting point selection, and landmark selection. In these three games, each agent’s choice is treated as a strategy, and a mixed Nash Equilibrium (NE) is computed to determine a mutually stable selection. Moreover, concerning the payoff for the landmark selection game, Gameplanner grounds game-theoretical planning in learning through a Learning-Guided Payoff (LGP), which uses each agent’s learned critic values to construct the game’s payoff matrix. This ensures that game-theoretical decisions are guided by individual learning progress. We demonstrate the effectiveness of our method in the AntMaze goal-reaching environment, where Gameplanner increases the success rate and enables stable, independent learning without centralized communication.