F5-TTS

Про SWivid (Shanghai Jiao Tong University et al.)

Засновано 2024 · Shanghai, China

F5-TTS was released in October 2024 by the SWivid research collective, an open-source group anchored at Shanghai Jiao Tong University with contributors from the X-LANCE Lab, Microsoft Research Asia and the Chinese University of Hong Kong. Lead authors Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Haian Lin, Wendi He, Xiaofei Wang, Tomoki Toda, Kai Yu and Xie Chen released F5-TTS under MIT licence on GitHub together with model weights and a Hugging Face demo. The work followed the earlier E2-TTS (Microsoft) paper and quickly became one of the most popular open-weights TTS models of 2024-2025, frequently cited as the best fully open alternative to ElevenLabs for English and Chinese voice cloning.

Архітектура

Flow-matching non-autoregressive TTS with Diffusion Transformer (DiT)

F5-TTS (Fairy-tale Fast Flow Matching TTS) is a non-autoregressive text-to-speech system that replaces the encoder-decoder pipeline with a single Diffusion Transformer trained with conditional flow matching. Text is first converted to character tokens and zero-padded to the target mel-spectrogram length, then concatenated with a noisy mel reference for the target voice. A DiT backbone with ConvNeXt v2 blocks predicts the velocity field that maps Gaussian noise to a clean mel-spectrogram, which is then converted to a waveform via Vocos vocoder. Training data is the 100k-hour open Emilia dataset (multilingual long-form audio scraped from podcasts and audiobooks). Because there is no autoregressive decoder, F5-TTS achieves real-time factor below 0.2 on a single A10 GPU while keeping competitive WER with VALL-E and NaturalSpeech 3. The model is famous for very high-quality zero-shot voice cloning from a single 5-15 second reference clip.

Параметри

~330M (Base) and ~1.4B (Large)

Контекст

30 токенів

Можливості

• Zero-shot voice cloning from ~10 s of reference audio with no fine-tuning
• Non-autoregressive flow matching with real-time-factor < 0.2 on a single GPU
• Open weights under MIT licence, multiple checkpoints (Base, Small, Multilingual)
• English and Chinese out of the box; community fine-tunes for German, French, Japanese, Spanish
• Up to 30 s of generated audio per inference
• Speed control by adjusting the duration prompt
• Best for: research, on-device TTS, voice cloning prototypes, commercial products built on open weights

Навчання та ліцензія

Pretrained on the 100,000-hour open Emilia dataset of multilingual long-form audio with weakly supervised transcripts. Additional community fine-tunes use LibriSpeech, AISHELL-3 and bespoke audiobook collections.

Ліцензія: Code and weights under MIT licence; commercial use permitted.

Тестування безпеки: No formal red-team report. Authors publish a model card recommending consent for voice cloning and a model-output watermark, which is optional.