Enhanced DGA botnet domain detection and family classification via n-gram analysis and Hellinger distance

Abstract

Bot masters spread malware to create botnets and use Domain Generation Algorithms (DGAs) to evade blacklist-based detection methods with numerous generated domains, posing a significant threat to network security. Since detection alone cannot halt malware operations, classifying DGA domains into their respective botnet families is essential for enabling targeted countermeasures and addressing vulnerabilities in infected systems. However, most existing approaches focus primarily on distinguishing DGA domains from legitimate ones and face challenges when classifying domains from DGA families with similar character distributions, highlighting the need for improved techniques. In response, we expand the focus to DGA family classification and conduct in-depth analyses using eXplainable Artificial Intelligence (XAI) techniques to explore the impact of n-grams on classification performance. These analyses reveal that n-gram preprocessing and Hellinger Distance (HD)-based features derived from n-gram probability distributions can significantly enhance classification performance. Building on these insights, we propose an integrated framework with two components, an N-gram-based Multi-scale One-Dimensional Convolutional Neural Network model (N-MODCNN) and a machine learning (ML) classifier utilizing HD features, for detecting and classifying DGA domains. N-MODCNN detects DGA domains from n-gram preprocessed inputs, and detected domains are classified into their respective botnet families by a soft ensemble approach that integrates predictions from N-MODCNN and the ML classifier, enabling robust and accurate classification. Experiments on recent public datasets show that our framework achieves up to 99% detection and classification accuracy. For families with similar character distributions, it achieves F1-scores exceeding 90%, representing improvements of up to 72 percentage points over existing methods.