Speech Slytherin: Examining the Performance and Efficiency of Mamba for
  Speech Separation, Recognition, and Synthesis
It is too early to conclude that Mamba is a better alternative to
transformers for speech before comparing Mamba with transformers in terms of
both performance and efficiency in multiple speech-related tasks. To reach this
conclusion, we propose and evaluate three models for three tasks: Mamba-TasNet
for speech separation, ConMamba for speech recognition, and VALL-M for speech
synthesis. We compare them with transformers of similar sizes in performance,
memory, and speed. Our Mamba or Mamba-transformer hybrid models show comparable
or higher performance than their transformer counterparts: Sepformer,
Conformer, and VALL-E. They are more efficient than transformers in memory and
speed for speech longer than a threshold duration, inversely related to the
resolution of a speech token. Mamba for separation is the most efficient, and
Mamba for recognition is the least. Further, we show that Mamba is not more
efficient than transformer for speech shorter than the threshold duration and
performs worse in models that require joint modeling of text and speech, such
as cross or masked attention of two inputs. Therefore, we argue that the
superiority of Mamba or transformer depends on particular problems and models.
Code available at https://github.com/xi-j/Mamba-TasNet and
https://github.com/xi-j/Mamba-ASR.
6695c32430bd2a19adece3f5
Xilin Jiang
claimed_verified
2025-02-26T08:43:23.511Z
6531a65daed617662c7f1007
/avatars/ea2e504780dc40719f7501ab2c7d9c91.svg
Xilin Jiang
user
xi-j
6695c32430bd2a19adece3f6
Yinghao Aaron Li
6695c32430bd2a19adece3f7
Adrian Nicolas Florea
6695c32430bd2a19adece3f8
Cong Han
6695c32430bd2a19adece3f9
Nima Mesgarani
Speech Slytherin: Examining the Performance and Efficiency of Mamba for
  Speech Separation, Recognition, and Synthesis
It is too early to conclude that Mamba is a better alternative to
transformers for speech before comparing Mamba with transformers in terms of
both performance and efficiency in multiple speech-related tasks. To reach this
conclusion, we propose and evaluate three models for three tasks: Mamba-TasNet
for speech separation, ConMamba for speech recognition, and VALL-M for speech
synthesis. We compare them with transformers of similar sizes in performance,
memory, and speed. Our Mamba or Mamba-transformer hybrid models show comparable
or higher performance than their transformer counterparts: Sepformer,
Conformer, and VALL-E. They are more efficient than transformers in memory and
speed for speech longer than a threshold duration, inversely related to the
resolution of a speech token. Mamba for separation is the most efficient, and
Mamba for recognition is the least. Further, we show that Mamba is not more
efficient than transformer for speech shorter than the threshold duration and
performs worse in models that require joint modeling of text and speech, such
as cross or masked attention of two inputs. Therefore, we argue that the
superiority of Mamba or transformer depends on particular problems and models.
Code available at https://github.com/xi-j/Mamba-TasNet and
https://github.com/xi-j/Mamba-ASR.
Mamba models, when compared with transformers of similar sizes, display comparable or higher performance in speech separation, recognition, and synthesis, but are more memory and speed efficient for longer speech durations.
Mamba-TasNet ConMamba VALL-M Sepformer Conformer VALL-E speech separation speech recognition speech synthesis cross-attention masked attention