SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
Recent advancements in generative speech models based on audio-text prompts
have enabled remarkable innovations like high-quality zero-shot text-to-speech.
However, existing models still face limitations in handling diverse audio-text
speech generation tasks involving transforming input speech and processing
audio captured in adverse acoustic conditions. This paper introduces SpeechX, a
versatile speech generation model capable of zero-shot TTS and various speech
transformation tasks, dealing with both clean and noisy signals. SpeechX
combines neural codec language modeling with multi-task learning using
task-dependent prompting, enabling unified and extensible modeling and
providing a consistent way for leveraging textual input in speech enhancement
and transformation tasks. Experimental results show SpeechX's efficacy in
various tasks, including zero-shot TTS, noise suppression, target speaker
extraction, speech removal, and speech editing with or without background
noise, achieving comparable or superior performance to specialized models
across tasks. See https://aka.ms/speechx for demo samples.
64db155695da0aa95cfbc8d1
Xiaofei Wang
claimed_verified
2023-08-16T13:23:01.075Z
64dc191bc307ee5369fbcb04
/avatars/5a8a0db63a187e85d4ae2fff93a838f0.svg
Xiaofei Wang
user
xiaofei-wang
64db155695da0aa95cfbc8d2
Manthan Thakker
claimed_verified
2023-08-16T13:23:13.762Z
60615782575f3b3e6beb9077
/avatars/f865572b493ef4802aa102d307953cd1.svg
Manthan Thakker
user
manthan40
64db155695da0aa95cfbc8d3
Zhuo Chen
claimed_verified
2023-08-16T13:23:09.190Z
642635d22a423daad52bf3e2
/avatars/dbcb757428a1b9d275c116dd56852f98.svg
Zhuo Chen
user
pqpq123
64db155695da0aa95cfbc8d4
Naoyuki Kanda
claimed_verified
2023-08-16T13:23:05.092Z
64dbe5247f749b6e346663f5
/avatars/d5c7f542c1fdb31eb49478afbb3dd08d.svg
Naoyuki Kanda
user
naoyuki82
64db155695da0aa95cfbc8d5
Sefik Emre Eskimez
claimed_verified
2024-08-15T10:13:07.657Z
65cbadf9ab05dc46afbac58c
/avatars/edf45a05c1dfafa35921a63d649573bd.svg
Sefik Emre Eskimez
user
emreeskimez
64db155695da0aa95cfbc8d6
Sanyuan Chen
claimed_verified
2023-08-16T13:22:56.405Z
6465843ce27766e8921b9cab
/avatars/6017db24da593c9afb21edfcedcddfd4.svg
Sanyuan Chen
user
Sanyuan-Chen
64db155695da0aa95cfbc8d7
Min Tang
64db155695da0aa95cfbc8d8
Shujie Liu
claimed_verified
2023-08-16T13:22:51.706Z
64dc94f1bdac91d86834e13b
/avatars/264ebe3867121f56eefac6e1ab8cf17f.svg
Shujie Liu
user
j4ckl1u
64db155695da0aa95cfbc8d9
Jinyu Li
64db155695da0aa95cfbc8da
Takuya Yoshioka
claimed_verified
2023-08-16T13:22:46.950Z
64dc844f70446182be9ca4ed
/avatars/8fd58064644ef6ad28c2aed5b101c61f.svg
Takuya Yoshioka
user
ty274
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
Recent advancements in generative speech models based on audio-text prompts
have enabled remarkable innovations like high-quality zero-shot text-to-speech.
However, existing models still face limitations in handling diverse audio-text
speech generation tasks involving transforming input speech and processing
audio captured in adverse acoustic conditions. This paper introduces SpeechX, a
versatile speech generation model capable of zero-shot TTS and various speech
transformation tasks, dealing with both clean and noisy signals. SpeechX
combines neural codec language modeling with multi-task learning using
task-dependent prompting, enabling unified and extensible modeling and
providing a consistent way for leveraging textual input in speech enhancement
and transformation tasks. Experimental results show SpeechX's efficacy in
various tasks, including zero-shot TTS, noise suppression, target speaker
extraction, speech removal, and speech editing with or without background
noise, achieving comparable or superior performance to specialized models
across tasks. See https://aka.ms/speechx for demo samples.
SpeechX integrates neural codec language modeling and multi-task learning to enable versatile speech generation, transformation, and enhancement tasks in both clean and noisy conditions.
neural codec language modeling multi-task learning task-dependent prompting speech enhancement target speaker extraction speech removal speech editing