Vector Field Decomposition-based Flow Matching for Zero-Shot Cross-Lingual Text-to-Speech

Abstract

Zero-shot text-to-speech (TTS) has recently achieved remarkable performance by leveraging a speech prompt instead of a speaker embedding, as it provides richer information. However, zero-shot cross-lingual tasks synthesize speech in multiple languages according to a given language ID, regardless of the language of the speech prompt. Consequently, the inherent language-specific characteristics of the speech prompt may conflict with the language ID, potentially affecting the accuracy of language representation in speech. Thus, we propose vector field decomposition-based flow matching that decomposes the vector field into speaker and language components. These components are trained to be activated in different frequency bins, as speaker and language identity are distributed across distinct frequency ranges in speech. This approach is particularly effective for cross-lingual TTS, as it minimizes conflicts between speech prompts and language IDs. As a result, the summation of the two components directly forms the vector field that represents the probability path from a Gaussian distribution to the target data distribution (e.g., mel spectrogram). Experimental results demonstrate that the proposed method outperforms the conventional method in terms of both subjective and objective evaluations.