Improved xDeepFM with Single Value Decomposition and Attention Mechanism

Abstract

Due to the sheer volume and variety of data in industrial manufacturing, manually creating features can be costly. Therefore, appropriate feature processing methods become crucial. Most existing feature processing methods abstract feature engineering as a feature search problem, i.e., finding feature transformations that optimize model performance. However, for automated feature engineering, the number of searches and the number of transformation combinations are huge. Therefore, we use a factorization-based model that measures interactions in terms of vector products, automatically learns patterns of combined features, and generalizes them to unseen features. Prior to this paper, the DeepFM algorithm (which combines an FM model with a deep neural network model) and the xDeepFM algorithm (which proposes a novel Compressed Interaction Network (CIN) designed to make feature interactions explicit) were available. The LRCIN proposed in this paper focuses on improving the CIN network in the xDeepFM method, by introducing a low-rank approximation method in the CIN network to reduce the number of parameters, and adding an attention mechanism after the CIN to ensure the accuracy of the model. The experimental results show that our method can effectively reduce the time complexity of the model and improve the model accuracy to some extent.