Exploring the unseen: A transformer-based unknown traffic detection scheme with contextual feature representation

Abstract

Network traffic classification is vital for ensuring security, guaranteeing quality of service (QoS), and optimizing performance. Accurate classification of network traffic, particularly the detection of unknown traffic, becomes increasingly challenging in modern environments characterized by encrypted and dynamic traffic patterns. In this study, we propose a novel framework designed to address these challenges. The proposed method employs a bidirectional encoder representations from transformers (BERT)-based feature extraction model to capture contextual and discriminative features from packet bytes in traffic, followed by a feature verification model that computes similarity scores between packet classes to enable precise traffic classification. Even in dynamic situations where the unknown traffic ratio varies, our proposed adaptive algorithm can effectively detect unknown traffic by leveraging these similarity scores. We conduct extensive experiments on two benchmark datasets across various unknown traffic ratios and demonstrate that the proposed method outperforms state-of-the-art methods by a minimum of 4.55%p and a maximum of 32.04%p improvement in overall accuracy. © 2025 Elsevier B.V.