Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors
  Approach
Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy
environments by integrating visual cues. While recent advances integrate Large
Language Models (LLMs) into AVSR, their high computational cost hinders
deployment in resource-constrained settings. To address this, we propose
Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of
Projectors (SMoP) module to scale model capacity without increasing inference
costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors,
Llama-SMoP enables the use of smaller LLMs while maintaining strong
performance. We explore three SMoP configurations and show that Llama-SMoP DEDR
(Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and
experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation
studies confirm its effectiveness in expert activation, scalability, and noise
robustness.
682f1664cd2f858ae3c8ec33
Umberto Cappellazzo
claimed_verified
2025-05-26T08:18:07.542Z
64903f017b630c141867877f
/avatars/3c7b248baed446ecc2b6adc2c444320d.svg
Umberto Cappellazzo
user
hisoka94
682f1664cd2f858ae3c8ec34
Minsu Kim
682f1664cd2f858ae3c8ec35
Stavros Petridis
682f1664cd2f858ae3c8ec36
Daniele Falavigna
682f1664cd2f858ae3c8ec37
Alessio Brutti
Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors
  Approach
Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy
environments by integrating visual cues. While recent advances integrate Large
Language Models (LLMs) into AVSR, their high computational cost hinders
deployment in resource-constrained settings. To address this, we propose
Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of
Projectors (SMoP) module to scale model capacity without increasing inference
costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors,
Llama-SMoP enables the use of smaller LLMs while maintaining strong
performance. We explore three SMoP configurations and show that Llama-SMoP DEDR
(Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and
experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation
studies confirm its effectiveness in expert activation, scalability, and noise
robustness.
Llama-SMoP, an efficient multimodal LLM incorporating Sparse Mixture of Projectors, enhances AVSR performance without increasing inference costs through modality-specific routers and experts.
Large Language Models (LLMs) Multimodal LLM Sparse Mixture of Projectors (SMoP) sparsely-gated mixture-of-experts (MoE) Disjoint-Experts Disjoint-Routers Audio-Visual Speech Recognition (AVSR) Automatic Speech Recognition (ASR) Visual Speech Recognition (VSR)