F5-TTS

Despre SWivid (Shanghai Jiao Tong University et al.)

Fondat 2024 · Shanghai, China

F5-TTS was released in October 2024 by the SWivid research collective, an open-source group anchored at Shanghai Jiao Tong University with contributors from the X-LANCE Lab, Microsoft Research Asia and the Chinese University of Hong Kong. Lead authors Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Haian Lin, Wendi He, Xiaofei Wang, Tomoki Toda, Kai Yu and Xie Chen released F5-TTS under MIT licence on GitHub together with model weights and a Hugging Face demo. The work followed the earlier E2-TTS (Microsoft) paper and quickly became one of the most popular open-weights TTS models of 2024-2025, frequently cited as the best fully open alternative to ElevenLabs for English and Chinese voice cloning.

Arhitectură

Flow-matching non-autoregressive TTS with Diffusion Transformer (DiT)

F5-TTS (Fairy-tale Fast Flow Matching TTS) is a non-autoregressive text-to-speech system that replaces the encoder-decoder pipeline with a single Diffusion Transformer trained with conditional flow matching. Text is first converted to character tokens and zero-padded to the target mel-spectrogram length, then concatenated with a noisy mel reference for the target voice. A DiT backbone with ConvNeXt v2 blocks predicts the velocity field that maps Gaussian noise to a clean mel-spectrogram, which is then converted to a waveform via Vocos vocoder. Training data is the 100k-hour open Emilia dataset (multilingual long-form audio scraped from podcasts and audiobooks). Because there is no autoregressive decoder, F5-TTS achieves real-time factor below 0.2 on a single A10 GPU while keeping competitive WER with VALL-E and NaturalSpeech 3. The model is famous for very high-quality zero-shot voice cloning from a single 5-15 second reference clip.

Parametri

~330M (Base) and ~1.4B (Large)

Context

30 tokeni

Capabilități

• Zero-shot voice cloning from ~10 s of reference audio with no fine-tuning
• Non-autoregressive flow matching with real-time-factor < 0.2 on a single GPU
• Open weights under MIT licence, multiple checkpoints (Base, Small, Multilingual)
• English and Chinese out of the box; community fine-tunes for German, French, Japanese, Spanish
• Up to 30 s of generated audio per inference
• Speed control by adjusting the duration prompt
• Best for: research, on-device TTS, voice cloning prototypes, commercial products built on open weights

Antrenament & licență

Pretrained on the 100,000-hour open Emilia dataset of multilingual long-form audio with weakly supervised transcripts. Additional community fine-tunes use LibriSpeech, AISHELL-3 and bespoke audiobook collections.

Licență: Code and weights under MIT licence; commercial use permitted.

Teste de siguranță: No formal red-team report. Authors publish a model card recommending consent for voice cloning and a model-output watermark, which is optional.