Adaptive Tsallis Entropy Regularization for Efficient Reinforcement Learning

Abstract

In this paper, we present adaptive regularization using Tsallis entropy (ART) for efficient exploration in reinforcement learning (RL) problem. Tsallis entropy represents various types of entropy with an additional parameter called entropic index, which generalizes the Shannon-Gibbs entropy. Tsallis entropy is often utilized as a regularization of the policy function where the resulting policy becomes stochastic and balances exploration and exploitation. Previous work on using Tsallis entropy for exploration only considered the entropic index between zero and one, which is a limited set of entropies. However, in this paper, we extend it to wider range of entropic index and enable to employ all types of Tsallis entropy with a positive entropic index in RL problem. Furthermore, we propose the condition of the optimal entropic index which has the smallest regret bound among all positive entropic index. Using this condition, we can automatically determine the optimal entropic index without demanding a brute force search to find the proper regularization of policy. In the experiment, simply applying the ART to the existing RL methods leads to fast convergence and performance improvement. © 2022 IEEE.