Diagnosis-aware multitask fine-tuning of Whisper for dysarthric speech recognition

Abstract

AbstractIndividuals with dysarthria exhibit irregular speech patterns that vary by disease, significantly reducing the accuracy of conventional speech-recognition systems. Previous studies have typically focused on a single disease group or used aggregated data without accounting for inter-disease variation, thereby limiting disease-specific insights. In this study, fluency metrics were extracted from a Korean dysarthric speech corpus across three disease groups (stroke, cerebral palsy, and peripheral neuropathy) and the diseases were classified based on these features. The performance of the disease-specific speech-recognition models was evaluated using the weighted character error rate (Weighted-CER). Results showed that classification based on fluency metrics achieved 99% accuracy. The disease-specific models improved the CER by up to 18.34 and 1.05 percentage points compared with the Whisper–Small model and a model trained on the entire dataset, respectively. In terms of Weighted-CER, the error rate decreased by up to 15.27 and 1.49 percentage points, respectively. These findings indicate that disease-specific models can meaningfully enhance speech recognition and underscore the importance of developing speech-recognition systems that can adapt to individual speech characteristics in patients with dysarthria