Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning

Abstract

In this paper, we propose a simple, tricky method to improve sentence representation of unsupervised contrastive learning. Even though contrastive learning has achieved great performances in both visual representation learning (VRL) and sentence representation learning (SRL) fields, we focus on the fact that there is a gap between the characteristics and training dynamics of VRL and SRL. We first examine the role of temperature to bridge the gap between VRL and SRL, and find some temperature-dependent elements in SRL; i.e., a higher temperature causes overfitting of the uniformity while improving the alignment in the earlier phase of training. Then, we design a temperature cool-down technique based on this observation, which helps PLMs to be more suitable for contrastive learning via the preparation of uniform representation space. Our experimental results on widely-utilized benchmarks demonstrate the effectiveness and an extensibility of our method. Our code is publicly available at https://github.com/myngsooo/Cooldown.