Empowering artificial intelligence with homomorphic encryption for secure deep reinforcement learning

Abstract

Deep reinforcement learning (DRL) demonstrates significant potential in solving complex control and decision-making problems, but it may inadvertently expose sensitive, environment-specific information, raising privacy and security concerns for computer systems, humans and organizations. This work introduces a privacy-preserving framework using homomorphic encryption and advanced learning algorithms to secure DRL processes. Our framework enables the encryption of sensitive information, including states, actions and rewards, before sharing it with an untrusted processing platform. This encryption ensures data privacy, prevents unauthorized access and maintains compliance with data protection laws throughout the learning process. In addition, we develop innovative algorithms to efficiently handle a wide range of encrypted control tasks. Our core innovation is the homomorphic encryption-compatible Adam optimizer, which reparameterizes momentum values to bypass the need for high-degree polynomial approximations of inverse square roots on encrypted data. This adaptation, previously unexplored in homomorphic encryption-based ML research, enables stable and efficient training with adaptive learning rates in encrypted domains, addressing a critical bottleneck for privacy-preserving DRL with sparse rewards. Evaluations on standard DRL benchmarks demonstrate that our encrypted DRL performs comparably with its unencrypted counterpart (with a gap of less than 10%) and maintaining data confidentiality with homomorphic encryption. This work facilitates the integration of privacy-preserving DRL into real-world applications, addressing critical privacy concerns, and promoting the ethical advancement of artificial intelligence.