DoH Tunneling Detection System for Enterprise Network Using Deep Learning Technique

Abstract

In spite of protection mechanisms for Domain Name System (DNS), such as IP blacklist and DNS Firewall, DNS still has privacy issues in reality, since DNS is a plain-text protocol. Recently, to resolve this problem, an encrypted DNS, called DNS-over-HTTPS (DoH), has been developed, and is becoming more widespread. As the secured version of DNS, DoH guarantees privacy and security to prevent various attacks such as eavesdropping and manipulating DNS data by using the HTTPS protocol to encrypt the data between the DoH client and the DoH-based DNS resolver. DoH is one of the best security options for an enterprise network where more sensitive data protection is required. However, DoH may cause an unintended security breach, i.e., information leakage via malicious DoH tunneling. Since the DoH traffic is encrypted and indistinguishable from other HTTPS traffic, data hidden inside DoH packets can be easily leaked out of an enterprise network. Although some countermeasures to detect DoH tunneling attacks have been proposed, they still have limitations. Previous research used Supervised Machine Learning methods to detect DoH tunneling, which required a high volume of labeled data. In practice, collecting and labeling all of the data is an impossible task, especially in DoH, when all of the data are encrypted. Furthermore, Supervised Machine Learning methods rely heavily on human-engineered feature extraction, which makes classifying encrypted DoH traffic difficult. Furthermore, no previous research has mentioned a complete functional DoH detection applied to network infrastructure. Therefore, we propose a detection system for DoH tunneling attacks based on Transformer to detect a malicious DoH tunneling and build a fully functional DoH detection system that can be integrated with the security operation system of an enterprise network. The experiment results show a significant improvement compared with previous works.