Reinforcement Learning-Based Multi-Agent Beam Tracking for Multi-RIS Hybrid Beamforming

Abstract

Reconfigurable intelligent surfaces (RIS) are emerging as a promising technology for next-generation wireless communications, capable of mitigating severe propagation attenuation, enhancing spectral efficiency, and expanding signal coverage. This paper focuses on online millimeter-wave (mmWave) beam tracking for multi-RIS-assisted hybrid beamforming systems. We develop two novel beam tracking algorithms based on multi-agent deep reinforcement learning (DRL): a multi-agent deep deterministic policy gradient (MADDPG)based algorithm for continuous-domain beam angle tracking and a multi-agent deep Q-network (MADQN)-based algorithm for codebook-based discrete-domain beam angle tracking. Both algorithms are designed to maximize the sum rate by jointly optimizing analog beamforming for the base station (BS) and reflection coefficients for multiple RISs in dynamic environments, leveraging historical information and without requiring current user position or channel information. After determining analog beamforming and RIS reflection coefficients, digital beamforming for the BS is constructed by estimating the end-to-end effective channel, which significantly reduces the overhead of channel estimation. Experimental results demonstrate that the proposed algorithms effectively adapt the analog beamformer and RIS reflection coefficients to account for user mobility, significantly outperforming existing benchmark schemes.