Compress, Align, and Transfer: A new method for transferring pre-trained language models knowledge to CTC-based speech recognition

Abstract

Connectionist temporal classification (CTC) model is a leading approach for end-to-end (E2E) automatic speech recognition (ASR), known for its simplicity and fast speed, enabled by non-autoregressive decoding and conditional independence assumptions. However, they often struggle to model token sequence relationships accurately due to its underlying assumptions, leading to lower recognition performance compared to attention-based encoder–decoder (AED) and transducer. This issue becomes particularly pronounced when the training data is limited or model size is small, leading to frequent spelling errors and reduced overall accuracy. In this study, we propose a new distillation approach named “Compress, Align, and Transfer” (COMAT) aimed at enhancing CTC-based ASR systems. COMAT addresses these challenges by integrating knowledge from pre-trained language models (PLMs) into CTC-based ASR systems. Our method involves a compressing module that adjusts speech embeddings to condense with the length of PLM embeddings, enabling a more effective and direct knowledge transfer and a monotonic alignment search (MAS) to align for two different embeddings. COMAT not only preserves the rapid decoding benefits of CTC-based models but also significantly enhances their ability to model complex tokens by linking the CTC-based models and the linguistic depth of PLMs.