Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemble

Abstract

Out-of-distribution (OOD) detection aims todiscern outliers from the intended data distribution, which is crucial to maintaining high reliability and a good user experience. Most recentstudies in OOD detection utilize the information from a single representation that residesin the penultimate layer to determine whetherthe input is anomalous or not. Although sucha method is straightforward, the potential ofdiverse information in the intermediate layersis overlooked. In this paper, we propose a novelframework based on contrastive learning thatencourages intermediate features to learn layerspecialized representations and assembles themimplicitly into a single representation to absorbrich information in the pre-trained languagemodel. Extensive experiments in various intentclassification and OOD datasets demonstratethat our approach is significantly more effective than other works. The source code for ourmodel is available online.