Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like
Laughter is one of the most expressive and natural aspects of human speech,
conveying emotions, social cues, and humor. However, most text-to-speech (TTS)
systems lack the ability to produce realistic and appropriate laughter sounds,
limiting their applications and user experience. While there have been prior
works to generate natural laughter, they fell short in terms of controlling the
timing and variety of the laughter to be generated. In this work, we propose
ELaTE, a zero-shot TTS that can generate natural laughing speech of any speaker
based on a short audio prompt with precise control of laughter timing and
expression. Specifically, ELaTE works on the audio prompt to mimic the voice
characteristic, the text prompt to indicate the contents of the generated
speech, and the input to control the laughter expression, which can be either
the start and end times of laughter, or the additional audio prompt that
contains laughter to be mimicked. We develop our model based on the foundation
of conditional flow-matching-based zero-shot TTS, and fine-tune it with
frame-level representation from a laughter detector as additional conditioning.
With a simple scheme to mix small-scale laughter-conditioned data with
large-scale pre-training data, we demonstrate that a pre-trained zero-shot TTS
model can be readily fine-tuned to generate natural laughter with precise
controllability, without losing any quality of the pre-trained zero-shot TTS
model. Through the evaluations, we show that ELaTE can generate laughing speech
with significantly higher quality and controllability compared to conventional
models. See https://aka.ms/elate/ for demo samples.
65caf7e211a80579a4db22cd
Naoyuki Kanda
admin_assigned
2024-02-13T11:00:55.007Z
64dbe5247f749b6e346663f5
/avatars/d5c7f542c1fdb31eb49478afbb3dd08d.svg
Naoyuki Kanda
user
naoyuki82
65caf7e211a80579a4db22ce
Xiaofei Wang
claimed_verified
2024-02-13T08:10:07.621Z
64dc191bc307ee5369fbcb04
/avatars/5a8a0db63a187e85d4ae2fff93a838f0.svg
Xiaofei Wang
user
xiaofei-wang
65caf7e211a80579a4db22cf
Sefik Emre Eskimez
claimed_verified
2024-08-15T10:13:05.453Z
65cbadf9ab05dc46afbac58c
/avatars/edf45a05c1dfafa35921a63d649573bd.svg
Sefik Emre Eskimez
user
emreeskimez
65caf7e211a80579a4db22d0
Manthan Thakker
admin_assigned
2024-02-13T11:01:08.379Z
60615782575f3b3e6beb9077
/avatars/f865572b493ef4802aa102d307953cd1.svg
Manthan Thakker
user
manthan40
65caf7e211a80579a4db22d1
Hemin Yang
65caf7e211a80579a4db22d2
Zirun Zhu
65caf7e211a80579a4db22d3
Min Tang
admin_assigned
2024-02-13T11:01:32.500Z
649068256764aef550de5336
/avatars/f08da7444642b36396ed88627e6e36d0.svg
Min Tang
user
minimalt
65caf7e211a80579a4db22d4
Canrun Li
65caf7e211a80579a4db22d5
Steven Tsai
65caf7e211a80579a4db22d6
Zhen Xiao
65caf7e211a80579a4db22d7
Yufei Xia
admin_assigned
2024-02-13T11:02:18.257Z
652a713ea21958f7db418195
/avatars/23d9bc915990e0ee7742448c87bd77ca.svg
Yufei Xiao
user
Xyf4566
65caf7e211a80579a4db22d8
Jinzhu Li
admin_assigned
2024-02-13T11:02:26.308Z
63e84032e02ee67e8e5922aa
/avatars/0f82186101c264a68ab5201e932fabef.svg
Jinzhu Li
user
yxlijinzhu
65caf7e211a80579a4db22d9
Yanqing Liu
65caf7e211a80579a4db22da
Sheng Zhao
65caf7e211a80579a4db22db
Michael Zeng
Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like
Laughter is one of the most expressive and natural aspects of human speech,
conveying emotions, social cues, and humor. However, most text-to-speech (TTS)
systems lack the ability to produce realistic and appropriate laughter sounds,
limiting their applications and user experience. While there have been prior
works to generate natural laughter, they fell short in terms of controlling the
timing and variety of the laughter to be generated. In this work, we propose
ELaTE, a zero-shot TTS that can generate natural laughing speech of any speaker
based on a short audio prompt with precise control of laughter timing and
expression. Specifically, ELaTE works on the audio prompt to mimic the voice
characteristic, the text prompt to indicate the contents of the generated
speech, and the input to control the laughter expression, which can be either
the start and end times of laughter, or the additional audio prompt that
contains laughter to be mimicked. We develop our model based on the foundation
of conditional flow-matching-based zero-shot TTS, and fine-tune it with
frame-level representation from a laughter detector as additional conditioning.
With a simple scheme to mix small-scale laughter-conditioned data with
large-scale pre-training data, we demonstrate that a pre-trained zero-shot TTS
model can be readily fine-tuned to generate natural laughter with precise
controllability, without losing any quality of the pre-trained zero-shot TTS
model. Through the evaluations, we show that ELaTE can generate laughing speech
with significantly higher quality and controllability compared to conventional
models. See https://aka.ms/elate/ for demo samples.
ELaTE is a zero-shot TTS that generates natural and controllable laughter in speech using a combination of conditional flow-matching and laughter detection.
zero-shot TTS conditional flow-matching laughter detection frame-level representation laughter conditioning