Reinforcement Learning-Based Adaptive Critic Control Design for Leader-Followers Multi-Agent Systems in Zero-Sum Games

Abstract

This paper proposes the reinforcement learning (RL)-based adaptive critic controller for multi-agent systems with one leader and N followers. The overall problem is formulated by the zero-sum game framework, where the followers minimize while the leader maximizes the same cost to capture the formation problem between the leader and followers. The optimal value function is approximated using the RL approach, and an adaptive critic network with an additional stabilization term is employed to derive the weight update rule. By incorporating the additional stabilization term, the need for decomposition in the controller design is eliminated, and the analysis for the closed-loop system stability in the sense of uniformly ultimately boundedness is simplified. In addition, an asymmetric barrier Lyapunov function is utilized to effectively handle output constraints and enhance the control performance. The proposed RL-based adaptive controllers for the leader and followers do not require the exact modeling information, since it is estimated by the neural network-based identifier. The performance of the proposed method is demonstrated through various simulations, including the leader-follower vehicle tracking problem using the CarSim environment. Note to Practitioners - In practical multi-agent systems, controllers usually operate with a partially unknown system and constrained outputs. These constraints sometimes make it challenging to design stable control strategies. The proposed controller assumes that the system dynamics is partially unknown, and the constraints of the output and tracking error exist. These challenges commonly arise in real-world applications such as autonomous driving, cooperative robotics, and autonomous aerial vehicle (AAV) swarms. The proposed methods achieve these challenges through a neural network (NN)-based identifier, an asymmetric barrier Lyapunov function (ABLF) candidate. The NN-based identifier enables the inference of physical properties that are difficult to describe analytically. This improves reliability and makes the algorithm easier to implement in practical systems. The ABLF candidate also allows practitioners to ensure different upper and lower output boundaries. In addition, compared to complex backstepping procedures, the proposed method incorporates an additional stabilization term in the adaptive controller. Consequently, this simplifies the controller design and enables stable learning.