title: F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow
    Matching
summary: This paper introduces F5-TTS, a fully non-autoregressive text-to-speech
  system based on flow matching with Diffusion Transformer (DiT). Without
  requiring complex designs such as duration model, text encoder, and phoneme
  alignment, the text input is simply padded with filler tokens to the same
  length as input speech, and then the denoising is performed for speech
  generation, which was originally proved feasible by E2 TTS. However, the
  original design of E2 TTS makes it hard to follow due to its slow convergence
  and low robustness. To address these issues, we first model the input with
  ConvNeXt to refine the text representation, making it easy to align with the
  speech. We further propose an inference-time Sway Sampling strategy, which
  significantly improves our model's performance and efficiency. This sampling
  strategy for flow step can be easily applied to existing flow matching based
  models without retraining. Our design allows faster training and achieves an
  inference RTF of 0.15, which is greatly improved compared to state-of-the-art
  diffusion-based TTS models. Trained on a public 100K hours multilingual
  dataset, our Fairytaler Fakes Fluent and Faithful speech with Flow matching
  (F5-TTS) exhibits highly natural and expressive zero-shot ability, seamless
  code-switching capability, and speed control efficiency. Demo samples can be
  found at https://SWivid.github.io/F5-TTS. We release all code and checkpoints
  to promote community development.
paper_authors._id: 670761c9bbae2dc3843a4164
paper_authors.name: Yushen Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:54:29.316Z
paper_authors._id: 636916bb85fef3ca96e2ef73
paper_authors.avatarUrl: /avatars/3f36f3a02dd8c3bcd7f542b69bfa330e.svg
paper_authors.fullname: Yushen CHEN
paper_authors.type: user
paper_authors.user: SWivid
paper_authors._id: 670761c9bbae2dc3843a4165
paper_authors.name: Zhikang Niu
paper_authors._id: 670761c9bbae2dc3843a4166
paper_authors.name: Ziyang Ma
paper_authors._id: 670761c9bbae2dc3843a4167
paper_authors.name: Keqi Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:55:02.556Z
paper_authors._id: 6190e0a108a57f265f7f5246
paper_authors.avatarUrl: /avatars/f7dc279ea9e922a2684dd23d4aa3f757.svg
paper_authors.fullname: Keqi Deng
paper_authors.type: user
paper_authors.user: D-Keqi
paper_authors._id: 670761c9bbae2dc3843a4168
paper_authors.name: Chunhui Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:55:07.827Z
paper_authors._id: 66610c2ec629e0e089fc9cf9
paper_authors.avatarUrl: /avatars/b3a958de94ba93a8b21ae6d1011f0b45.svg
paper_authors.fullname: Chun-Hui Wang
paper_authors.type: user
paper_authors.user: ChunHuiWangFN
paper_authors._id: 670761c9bbae2dc3843a4169
paper_authors.name: Jian Zhao
paper_authors._id: 670761c9bbae2dc3843a416a
paper_authors.name: Kai Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:55:18.617Z
paper_authors._id: 66c04955780d735f176eb72f
paper_authors.avatarUrl: /avatars/10fb520fbaf028d80aa1fb17a1d17f06.svg
paper_authors.fullname: Kai Yu
paper_authors.type: user
paper_authors.user: kaiyu-hf
paper_authors._id: 670761c9bbae2dc3843a416b
paper_authors.name: Xie Chen
paper_title: F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow
    Matching
paper_summary: This paper introduces F5-TTS, a fully non-autoregressive text-to-speech
  system based on flow matching with Diffusion Transformer (DiT). Without
  requiring complex designs such as duration model, text encoder, and phoneme
  alignment, the text input is simply padded with filler tokens to the same
  length as input speech, and then the denoising is performed for speech
  generation, which was originally proved feasible by E2 TTS. However, the
  original design of E2 TTS makes it hard to follow due to its slow convergence
  and low robustness. To address these issues, we first model the input with
  ConvNeXt to refine the text representation, making it easy to align with the
  speech. We further propose an inference-time Sway Sampling strategy, which
  significantly improves our model's performance and efficiency. This sampling
  strategy for flow step can be easily applied to existing flow matching based
  models without retraining. Our design allows faster training and achieves an
  inference RTF of 0.15, which is greatly improved compared to state-of-the-art
  diffusion-based TTS models. Trained on a public 100K hours multilingual
  dataset, our Fairytaler Fakes Fluent and Faithful speech with Flow matching
  (F5-TTS) exhibits highly natural and expressive zero-shot ability, seamless
  code-switching capability, and speed control efficiency. Demo samples can be
  found at https://SWivid.github.io/F5-TTS. We release all code and checkpoints
  to promote community development.
paper_ai_keywords: flow matching Diffusion Transformer (DiT) non-autoregressive text-to-speech (TTS) ConvNeXt Sway Sampling flow step diffusion-based TTS real-time factor (RTF) multilingual dataset zero-shot ability code-switching capability speed control efficiency