Language Model Personalization for Speech Recognition: A Clustered Federated Learning Approach with Adaptive Weight Average

Abstract

In the rapidly evolving field of automatic speech recognition (ASR), the push towards personalization has become a paramount concern. Text-only personalization, while advantageous for data collection and adaptable to text variations, can suffer from overfitting when using personal data and requires extensive data to mitigate this issue. Federated learning (FL) emerges as a solution, facilitating learning from diverse client models while preserving privacy. However, FL addresses the challenges posed by non independent and identically distributed (non-i.i.d) data, potentially leading to poor performance. We propose two approaches for language model personalization in ASR to address these issues. First, adaptive weighted average addresses the limitations of uniform weight average in the existing FL method by combining local language models into a global model. Second, clustered federated learning, based solely on model parameters, improves model stability without relying on information from the local domain. Both strategies aim to enhance personalization and reduce performance degradation, particularly in non-i.i.d scenarios within the FL.