SAFE: Unsupervised image feature extraction using self-attention based feature extraction network

Abstract

The ability to extract high-quality features from data is critical for machine learning applications. With the development of deep learning, various methods have been developed for image feature extraction, and unsupervised techniques have gained popularity due to their ability to operate without response variables. Autoencoders with encoder-decoder architectures are a common example of such techniques, but they are limited by a lack of proportional relationship between model reconstruction and encoder feature extraction performance. If the decoder is composed of multiple layers and mapping to a higher dimension is easier, the feature extraction performance of the encoder is likely to decrease. However, previous research has not adequately addressed this limitation. This study identifies the limitations of conventional unsupervised feature extraction techniques that utilize the encoder-decoder architecture, and proposes a novel feature extraction technique called SAFE, which utilizes a self-attention mechanism to eliminate decoder effects and improve the performance of encoder. To validate the effectiveness of the proposed model, we conducted experiments using diverse datasets (MNIST, Fashion MNIST, SVHN, and WM811K). The results of the experiments demonstrated that our proposed method exhibited, on average, 2%-10% higher performance in terms of accuracy and F-measure compared to the existing feature extraction techniques in the classification problem. While our research has limitations, specifically in its applicability only to the selection of image features, future studies should be undertaken to explore its potential application in various fields.