title: Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like
summary: Laughter is one of the most expressive and natural aspects of human speech,
  conveying emotions, social cues, and humor. However, most text-to-speech (TTS)
  systems lack the ability to produce realistic and appropriate laughter sounds,
  limiting their applications and user experience. While there have been prior
  works to generate natural laughter, they fell short in terms of controlling the
  timing and variety of the laughter to be generated. In this work, we propose
  ELaTE, a zero-shot TTS that can generate natural laughing speech of any speaker
  based on a short audio prompt with precise control of laughter timing and
  expression. Specifically, ELaTE works on the audio prompt to mimic the voice
  characteristic, the text prompt to indicate the contents of the generated
  speech, and the input to control the laughter expression, which can be either
  the start and end times of laughter, or the additional audio prompt that
  contains laughter to be mimicked. We develop our model based on the foundation
  of conditional flow-matching-based zero-shot TTS, and fine-tune it with
  frame-level representation from a laughter detector as additional conditioning.
  With a simple scheme to mix small-scale laughter-conditioned data with
  large-scale pre-training data, we demonstrate that a pre-trained zero-shot TTS
  model can be readily fine-tuned to generate natural laughter with precise
  controllability, without losing any quality of the pre-trained zero-shot TTS
  model. Through the evaluations, we show that ELaTE can generate laughing speech
  with significantly higher quality and controllability compared to conventional
  models. See https://aka.ms/elate/ for demo samples.
paper_authors._id: 65caf7e211a80579a4db22cd
paper_authors.name: Naoyuki Kanda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:00:55.007Z
paper_authors._id: 64dbe5247f749b6e346663f5
paper_authors.avatarUrl: /avatars/d5c7f542c1fdb31eb49478afbb3dd08d.svg
paper_authors.fullname: Naoyuki Kanda
paper_authors.type: user
paper_authors.user: naoyuki82
paper_authors._id: 65caf7e211a80579a4db22ce
paper_authors.name: Xiaofei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-13T08:10:07.621Z
paper_authors._id: 64dc191bc307ee5369fbcb04
paper_authors.avatarUrl: /avatars/5a8a0db63a187e85d4ae2fff93a838f0.svg
paper_authors.fullname: Xiaofei Wang
paper_authors.type: user
paper_authors.user: xiaofei-wang
paper_authors._id: 65caf7e211a80579a4db22cf
paper_authors.name: Sefik Emre Eskimez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-15T10:13:05.453Z
paper_authors._id: 65cbadf9ab05dc46afbac58c
paper_authors.avatarUrl: /avatars/edf45a05c1dfafa35921a63d649573bd.svg
paper_authors.fullname: Sefik Emre Eskimez
paper_authors.type: user
paper_authors.user: emreeskimez
paper_authors._id: 65caf7e211a80579a4db22d0
paper_authors.name: Manthan Thakker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:01:08.379Z
paper_authors._id: 60615782575f3b3e6beb9077
paper_authors.avatarUrl: /avatars/f865572b493ef4802aa102d307953cd1.svg
paper_authors.fullname: Manthan Thakker
paper_authors.type: user
paper_authors.user: manthan40
paper_authors._id: 65caf7e211a80579a4db22d1
paper_authors.name: Hemin Yang
paper_authors._id: 65caf7e211a80579a4db22d2
paper_authors.name: Zirun Zhu
paper_authors._id: 65caf7e211a80579a4db22d3
paper_authors.name: Min Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:01:32.500Z
paper_authors._id: 649068256764aef550de5336
paper_authors.avatarUrl: /avatars/f08da7444642b36396ed88627e6e36d0.svg
paper_authors.fullname: Min Tang
paper_authors.type: user
paper_authors.user: minimalt
paper_authors._id: 65caf7e211a80579a4db22d4
paper_authors.name: Canrun Li
paper_authors._id: 65caf7e211a80579a4db22d5
paper_authors.name: Steven Tsai
paper_authors._id: 65caf7e211a80579a4db22d6
paper_authors.name: Zhen Xiao
paper_authors._id: 65caf7e211a80579a4db22d7
paper_authors.name: Yufei Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:02:18.257Z
paper_authors._id: 652a713ea21958f7db418195
paper_authors.avatarUrl: /avatars/23d9bc915990e0ee7742448c87bd77ca.svg
paper_authors.fullname: Yufei Xiao
paper_authors.type: user
paper_authors.user: Xyf4566
paper_authors._id: 65caf7e211a80579a4db22d8
paper_authors.name: Jinzhu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:02:26.308Z
paper_authors._id: 63e84032e02ee67e8e5922aa
paper_authors.avatarUrl: /avatars/0f82186101c264a68ab5201e932fabef.svg
paper_authors.fullname: Jinzhu Li
paper_authors.type: user
paper_authors.user: yxlijinzhu
paper_authors._id: 65caf7e211a80579a4db22d9
paper_authors.name: Yanqing Liu
paper_authors._id: 65caf7e211a80579a4db22da
paper_authors.name: Sheng Zhao
paper_authors._id: 65caf7e211a80579a4db22db
paper_authors.name: Michael Zeng
paper_title: Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like
paper_summary: Laughter is one of the most expressive and natural aspects of human speech,
  conveying emotions, social cues, and humor. However, most text-to-speech (TTS)
  systems lack the ability to produce realistic and appropriate laughter sounds,
  limiting their applications and user experience. While there have been prior
  works to generate natural laughter, they fell short in terms of controlling the
  timing and variety of the laughter to be generated. In this work, we propose
  ELaTE, a zero-shot TTS that can generate natural laughing speech of any speaker
  based on a short audio prompt with precise control of laughter timing and
  expression. Specifically, ELaTE works on the audio prompt to mimic the voice
  characteristic, the text prompt to indicate the contents of the generated
  speech, and the input to control the laughter expression, which can be either
  the start and end times of laughter, or the additional audio prompt that
  contains laughter to be mimicked. We develop our model based on the foundation
  of conditional flow-matching-based zero-shot TTS, and fine-tune it with
  frame-level representation from a laughter detector as additional conditioning.
  With a simple scheme to mix small-scale laughter-conditioned data with
  large-scale pre-training data, we demonstrate that a pre-trained zero-shot TTS
  model can be readily fine-tuned to generate natural laughter with precise
  controllability, without losing any quality of the pre-trained zero-shot TTS
  model. Through the evaluations, we show that ELaTE can generate laughing speech
  with significantly higher quality and controllability compared to conventional
  models. See https://aka.ms/elate/ for demo samples.
paper_ai_keywords: zero-shot TTS conditional flow-matching-based zero-shot TTS frame-level representation laughter detector laughter-conditioned data pre-training data fine-tuning