title: SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
summary: Recent advancements in generative speech models based on audio-text prompts
  have enabled remarkable innovations like high-quality zero-shot text-to-speech.
  However, existing models still face limitations in handling diverse audio-text
  speech generation tasks involving transforming input speech and processing
  audio captured in adverse acoustic conditions. This paper introduces SpeechX, a
  versatile speech generation model capable of zero-shot TTS and various speech
  transformation tasks, dealing with both clean and noisy signals. SpeechX
  combines neural codec language modeling with multi-task learning using
  task-dependent prompting, enabling unified and extensible modeling and
  providing a consistent way for leveraging textual input in speech enhancement
  and transformation tasks. Experimental results show SpeechX's efficacy in
  various tasks, including zero-shot TTS, noise suppression, target speaker
  extraction, speech removal, and speech editing with or without background
  noise, achieving comparable or superior performance to specialized models
  across tasks. See https://aka.ms/speechx for demo samples.
paper_authors._id: 64db155695da0aa95cfbc8d1
paper_authors.name: Xiaofei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-16T13:23:01.075Z
paper_authors._id: 64dc191bc307ee5369fbcb04
paper_authors.avatarUrl: /avatars/5a8a0db63a187e85d4ae2fff93a838f0.svg
paper_authors.fullname: Xiaofei Wang
paper_authors.type: user
paper_authors.user: xiaofei-wang
paper_authors._id: 64db155695da0aa95cfbc8d2
paper_authors.name: Manthan Thakker
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-16T13:23:13.762Z
paper_authors._id: 60615782575f3b3e6beb9077
paper_authors.avatarUrl: /avatars/f865572b493ef4802aa102d307953cd1.svg
paper_authors.fullname: Manthan Thakker
paper_authors.type: user
paper_authors.user: manthan40
paper_authors._id: 64db155695da0aa95cfbc8d3
paper_authors.name: Zhuo Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-16T13:23:09.190Z
paper_authors._id: 642635d22a423daad52bf3e2
paper_authors.avatarUrl: /avatars/dbcb757428a1b9d275c116dd56852f98.svg
paper_authors.fullname: Zhuo Chen
paper_authors.type: user
paper_authors.user: pqpq123
paper_authors._id: 64db155695da0aa95cfbc8d4
paper_authors.name: Naoyuki Kanda
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-16T13:23:05.092Z
paper_authors._id: 64dbe5247f749b6e346663f5
paper_authors.avatarUrl: /avatars/d5c7f542c1fdb31eb49478afbb3dd08d.svg
paper_authors.fullname: Naoyuki Kanda
paper_authors.type: user
paper_authors.user: naoyuki82
paper_authors._id: 64db155695da0aa95cfbc8d5
paper_authors.name: Sefik Emre Eskimez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-15T10:13:07.657Z
paper_authors._id: 65cbadf9ab05dc46afbac58c
paper_authors.avatarUrl: /avatars/edf45a05c1dfafa35921a63d649573bd.svg
paper_authors.fullname: Sefik Emre Eskimez
paper_authors.type: user
paper_authors.user: emreeskimez
paper_authors._id: 64db155695da0aa95cfbc8d6
paper_authors.name: Sanyuan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-16T13:22:56.405Z
paper_authors._id: 6465843ce27766e8921b9cab
paper_authors.avatarUrl: /avatars/6017db24da593c9afb21edfcedcddfd4.svg
paper_authors.fullname: Sanyuan Chen
paper_authors.type: user
paper_authors.user: Sanyuan-Chen
paper_authors._id: 64db155695da0aa95cfbc8d7
paper_authors.name: Min Tang
paper_authors._id: 64db155695da0aa95cfbc8d8
paper_authors.name: Shujie Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-16T13:22:51.706Z
paper_authors._id: 64dc94f1bdac91d86834e13b
paper_authors.avatarUrl: /avatars/264ebe3867121f56eefac6e1ab8cf17f.svg
paper_authors.fullname: Shujie Liu
paper_authors.type: user
paper_authors.user: j4ckl1u
paper_authors._id: 64db155695da0aa95cfbc8d9
paper_authors.name: Jinyu Li
paper_authors._id: 64db155695da0aa95cfbc8da
paper_authors.name: Takuya Yoshioka
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-16T13:22:46.950Z
paper_authors._id: 64dc844f70446182be9ca4ed
paper_authors.avatarUrl: /avatars/8fd58064644ef6ad28c2aed5b101c61f.svg
paper_authors.fullname: Takuya Yoshioka
paper_authors.type: user
paper_authors.user: ty274
paper_title: SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
paper_summary: Recent advancements in generative speech models based on audio-text prompts
  have enabled remarkable innovations like high-quality zero-shot text-to-speech.
  However, existing models still face limitations in handling diverse audio-text
  speech generation tasks involving transforming input speech and processing
  audio captured in adverse acoustic conditions. This paper introduces SpeechX, a
  versatile speech generation model capable of zero-shot TTS and various speech
  transformation tasks, dealing with both clean and noisy signals. SpeechX
  combines neural codec language modeling with multi-task learning using
  task-dependent prompting, enabling unified and extensible modeling and
  providing a consistent way for leveraging textual input in speech enhancement
  and transformation tasks. Experimental results show SpeechX's efficacy in
  various tasks, including zero-shot TTS, noise suppression, target speaker
  extraction, speech removal, and speech editing with or without background
  noise, achieving comparable or superior performance to specialized models
  across tasks. See https://aka.ms/speechx for demo samples.
paper_ai_keywords: neural codec language modeling multi-task learning task-dependent prompting speech enhancement target speaker extraction speech removal speech editing