Latent Embedding-Based Isolation Forests for Out-of-Distribution Detection

Abstract

Non-cooperative communication contexts, such as spectrum surveillance and cognitive radio, increasingly rely on automatic modulation classification (AMC) for intelligent signal processing. However, AMC models without out-of-distribution (OOD) detection risk misclassifying unknown modulations with high confidence. Existing OOD detection methods, including autoencoders, operate under the assumption that the reconstruction loss of OOD inputs is smaller than that of ID inputs. This assumption does not always hold, and when it fails, detection may degrade. To address this, this paper proposes AE-iForest, which integrates an isolation forest (iForest) with the autoencoder's latent embeddings, serving as the feature space for OOD detection. In the proposed method, the iForest isolates OOD signals by recursively partitioning the latent feature space, producing fewer partitions (shorter path lengths) for OOD regions and more partitions for dense ID regions. For evaluation, we adopt a quantile-based thresholding rule on held-out ID samples to retain a fixed proportion of ID, while also considering threshold-free measures of separability. Experiments conducted under two scenarios demonstrate that the proposed method effectively addresses the limitations of reconstruction-loss-based approaches.