Maximum Norm Minimization: A Single-Policy Multi-Objective Reinforcement Learning to Expansion of the Pareto Front

Abstract

In this paper, we propose Maximum Norm Minimization (MNM), a single-policy Multi-Objective Reinforcement Learning (MORL) algorithm to solve the multi-objective RL problem. The main objective of our MNM is to provide the Pareto optimal points constituting the Pareto front in the multi-objective space. First, MNM measures distances among the Pareto optimal points in the current Pareto front and then normalizes the distances based on maximum and minimum reward values for each objective in the multi-objective space. Second, MNM identifies the maximum norm, i.e., the maximum value of the normalized Pareto optimal distances. Then MNM seeks to find a new Pareto optimal point, which corresponds to the middle of the two Pareto optimal points constituting the maximum norm. By iterating these two processes, MNM is able to expand and densify the Pareto front with increasing summation of the Pareto front volumes and decreasing mean-squared distance of the Pareto optimal points. To validate the performance of MNM, we provide the experimental results of five complex robotic multi-objective environments. In particular, we compare the performance of MNM with those of other state-of-the-art methods in terms of the summation of volumes and the mean-squared distance of the Pareto optimal points.