Option-Based Deep Reinforcement Learning for Topology Control of Power Systems

Abstract

In this paper, we propose an option-based deep reinforcement learning (DRL) algorithm called option-critic with long short-term memory (OC-LSTM), which combines the option-critic (OC) framework containing a hierarchical policy structure with a long short-term memory (LSTM) network, which makes full use of the powerful time-series feature extraction capability of LSTM networks and uses the OC framework to learn power system topology control policy of the power system. Specifically, in a complex and variable power system, the OC-LSTM extracts key power system state information through the LSTM network and uses the OC framework to define and optimize the high-level options and low-level action policy, which effectively reduces the dimensionality of the agent's topology control action space in the decision-making process. This combination improves the accuracy of topology control policies and effectively maintains the stability of the power system. The experimental results show that the OC-LSTM algorithm outperforms the benchmark DRL algorithm during training, with the ablation experiment further highlighting the effectiveness of LSTM in power system feature extraction. Additionally, the OC-LSTM algorithm enables stable operation of the IEEE 5-Bus, IEEE 14-Bus, and L2RPN WCCI 2020 power systems for 60 hours, all without human expert intervention.