title: Speech Slytherin: Examining the Performance and Efficiency of Mamba for
    Speech Separation, Recognition, and Synthesis
summary: It is too early to conclude that Mamba is a better alternative to
  transformers for speech before comparing Mamba with transformers in terms of
  both performance and efficiency in multiple speech-related tasks. To reach this
  conclusion, we propose and evaluate three models for three tasks: Mamba-TasNet
  for speech separation, ConMamba for speech recognition, and VALL-M for speech
  synthesis. We compare them with transformers of similar sizes in performance,
  memory, and speed. Our Mamba or Mamba-transformer hybrid models show comparable
  or higher performance than their transformer counterparts: Sepformer,
  Conformer, and VALL-E. They are more efficient than transformers in memory and
  speed for speech longer than a threshold duration, inversely related to the
  resolution of a speech token. Mamba for separation is the most efficient, and
  Mamba for recognition is the least. Further, we show that Mamba is not more
  efficient than transformer for speech shorter than the threshold duration and
  performs worse in models that require joint modeling of text and speech, such
  as cross or masked attention of two inputs. Therefore, we argue that the
  superiority of Mamba or transformer depends on particular problems and models.
  Code available at https://github.com/xi-j/Mamba-TasNet and
  https://github.com/xi-j/Mamba-ASR.
paper_authors._id: 6695c32430bd2a19adece3f5
paper_authors.name: Xilin Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:43:23.511Z
paper_authors._id: 6531a65daed617662c7f1007
paper_authors.avatarUrl: /avatars/ea2e504780dc40719f7501ab2c7d9c91.svg
paper_authors.fullname: Xilin Jiang
paper_authors.type: user
paper_authors.user: xi-j
paper_authors._id: 6695c32430bd2a19adece3f6
paper_authors.name: Yinghao Aaron Li
paper_authors._id: 6695c32430bd2a19adece3f7
paper_authors.name: Adrian Nicolas Florea
paper_authors._id: 6695c32430bd2a19adece3f8
paper_authors.name: Cong Han
paper_authors._id: 6695c32430bd2a19adece3f9
paper_authors.name: Nima Mesgarani
paper_title: Speech Slytherin: Examining the Performance and Efficiency of Mamba for
    Speech Separation, Recognition, and Synthesis
paper_summary: It is too early to conclude that Mamba is a better alternative to
  transformers for speech before comparing Mamba with transformers in terms of
  both performance and efficiency in multiple speech-related tasks. To reach this
  conclusion, we propose and evaluate three models for three tasks: Mamba-TasNet
  for speech separation, ConMamba for speech recognition, and VALL-M for speech
  synthesis. We compare them with transformers of similar sizes in performance,
  memory, and speed. Our Mamba or Mamba-transformer hybrid models show comparable
  or higher performance than their transformer counterparts: Sepformer,
  Conformer, and VALL-E. They are more efficient than transformers in memory and
  speed for speech longer than a threshold duration, inversely related to the
  resolution of a speech token. Mamba for separation is the most efficient, and
  Mamba for recognition is the least. Further, we show that Mamba is not more
  efficient than transformer for speech shorter than the threshold duration and
  performs worse in models that require joint modeling of text and speech, such
  as cross or masked attention of two inputs. Therefore, we argue that the
  superiority of Mamba or transformer depends on particular problems and models.
  Code available at https://github.com/xi-j/Mamba-TasNet and
  https://github.com/xi-j/Mamba-ASR.
paper_ai_summary: Mamba models, when compared with transformers of similar sizes, display comparable or higher performance in speech separation, recognition, and synthesis, but are more memory and speed efficient for longer speech durations.
paper_ai_keywords: Mamba-TasNet ConMamba VALL-M Sepformer Conformer VALL-E speech separation speech recognition speech synthesis cross-attention masked attention